0.5 Sampling with automatic gain control  (Page 11/19)

The final step in implementing any solution is to check that the method behaves as desired, despite any simplifying assumptionsthat may have been made in its derivation. This may involve a detailed analysis ofthe resulting methodology, or it may involve simulations. Thorough testing would involve both analysis and simulationin a variety of settings that mimic, as closely as possible, the situations in which the method will be used.

Imagine being lost on a mountainside on a foggy night. Your goal is to get to the village which liesat the bottom of a valley below. Though you cannot see far, you can reach out and feelthe nearby ground. If you repeatedly step in the direction that heads downhill most steeply, you eventually reach a depression inwhich all directions lead up. If the contour of the land is smooth, and without any local depressions that can trap you,then you will eventually arrive at the village. The optimization procedure called “steepest descent”implements this scenario mathematically where the mountainside is defined by the “performance” function andthe optimal answer lies in the valley at the minimum value. Many standard communications algorithms (adaptive elements) can beviewed in this way.

An example of optimization: polynomial minimization

This first example is too simple to be of practical use, but it does show many of the ideas starkly. Suppose that the goal is tofind the value at which the polynomial

achieves its minimum value. Thus step (1) is set. As any calculus book will suggest, the direct way to find the minimum is totake the derivative, set it equal to zero, and solve for $x$ . Thus, $\frac{dJ\left(x\right)}{dx}=2x-4=0$ is solved when $x=2$ , which is indeed the value of $x$ where the parabola $J\left(x\right)$ reaches bottom. Sometimes (one might truthfully say “often”), however, such directapproaches are impossible. Maybe the derivative is just too complicated to solve (which can happen when the functions involved in $J\left(x\right)$ are extremely nonlinear). Or maybe the derivative of $J\left(x\right)$ cannot be calculated precisely from the available data, and instead must beestimated from a noisy data stream.

One alternative to the direct solution technique is an adaptive method called “steepest descent”(when the goal is to minimize), and called “hill climbing” (when the goal is to maximize).Steepest descent begins with an initial guess of the location of the minimum, evaluates which direction from this estimate is most steeply “downhill,”and then makes a new estimate along the downhill direction. Similarly, hill climbing begins with an initialguess of the location of the maximum, evaluates which direction climbs the most rapidly, and then makes a new estimate along theuphill direction. With luck, the new estimates are better than the old. The process repeats, hopefully getting closer to theoptimal location at each step. The key ingredient in this procedure is to recognize that the uphilldirection is defined by the gradient evaluated at the current location, while the downhill direction is the negative of this gradient.