<< Chapter < Page
  Signal theory   Page 1 / 1
Chapter >> Page >
Introduces general optimization theory in Hilbert Spaces

In the remainder of the course we will discuss optimization problems. In general, an optimization problem consists of picking the “best” signal according to some metric; the metric will be some functional f : X Y and the search will be over a set of interest D X , so that the problem can be written as

x ^ = arg max x D f ( x ) or x ^ = arg min x D f ( x ) .

We will need to extend the ideas of derivatives and gradients (which are used in optimization of single-variable real-valued functions) to arbitrary signal spaces where we can move in infinite directions on a set of interest.

Optimization examples. We wish to find the largest or smallest value of a functional f ( x ) , x X , over a set D X . For scalar-valued functions of scalar fields, the maximizer/minimizer is found by solving d f d x = 0 .

Directional derivatives

Assume that we have a metric function f : X Y and a set of interest D X . Navigating the “surface” of f to find a maximum or minimum requires for us to formulate a framework for derivatives.

Definition 1 Let x D X and h X be arbitrary. If the limit

δ f ( x ; h ) = lim α 0 1 α [ f ( x + α h ) - f ( x ) ]

exists, it is called Gâteaux differential of f at x with increment (or in the direction) h . If the limit exists for each h X , the transformation f is said to be Gâteaux differentiable at x . If f is Gâteaux differentiable at all x X , then it is called a Gâteaux differentiable functional .

This extends the concept of derivative to incorporate direction so it can be used for any signal space. Note that α needs to be sufficiently small so that x + α h D . Note also that for a fixed point x and variable direction h , the Gâteaux differential is a map from X to Y , i.e., δ f ( x ; · ) : X Y .

Fact 1 In the common case of Y = R ,

δ f ( x ; h ) = α f ( x + α h ) | α = 0 .

Example 1 Let H be a Hilbert space and L B ( H , H ) . Define the function f : H R by f ( x ) = L x , x . What is its Gâteaux differential? From the definition,

δ f ( x ; h ) = α ( L ( x + α h ) , x + α h ) | α = 0 .

We compute the derivative:

L ( x + α h ) , x + α h = L x , x + α L h , x + α L x , h + α 2 L h , h , α ( L ( x + α h ) , x + α h ) = L h , x + L x , h + 2 α L h , h .

Therefore,

δ f ( x ; h ) = L h , x + L x , h = L h , x + h , L x = h , L * x + h , L x , = h , ( L + L * ) x .

Unfortunately, the Gâbeaux differential does not satisfy our need to connect differentiability to continuity.

Definition 2 Let f : X Y be a transformation on D X . If for each x D and each direction h X there exists a function δ f ( x ; h ) : D × X Y that is linear and continuous with respect to h such that

lim h 0 f ( x + h ) - f ( x ) - δ f ( x ; h ) h = 0 ,

then f is said to be Fréchet differentiable at x and δ f ( x ; h ) is said to be the Fréchet differential of f at x with increment h .

One can intuitively see that there is a stronger connection between the common definition of a derivative (for functions R R ) and the Fréchet derivative. There are additional connections between the derivatives and their properties.

Lemma 1 If a function f is Fréchet differentiable then δ f ( x ; h ) is unique.

Lemma 2 If the Fréchet differential of f exists at x , then the Gâteaux differential of f exists at x and they are equal.

Lemma 3 If f defined on an open set D X has a Fréchet differential at x then f is continuous at x .

For a Fréchef-differentiable function, for any ϵ > 0 there exists a sufficiently small h X such that

f ( x + h ) - f ( x ) - δ f ( x ; h ) h < ϵ .

This in turn implies

f ( x + h ) - f ( x ) f ( x + h ) - f ( x ) - δ f ( x ; h ) + δ f ( x ; h ) ϵ h + δ f ( x ; · ) h , ( ϵ + δ f ( x ; · ) ) h ,

as δ f ( x ; h ) is a linear continuous functional on h , implying that it is bounded. Therefore, as h 0 , we have

lim h 0 f ( x + h ) - f ( x ) = 0 .

This implies that f is continuous at x .

Get Jobilize Job Search Mobile App in your pocket Now!

Get it on Google Play Download on the App Store Now




Source:  OpenStax, Signal theory. OpenStax CNX. Oct 18, 2013 Download for free at http://legacy.cnx.org/content/col11542/1.3
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Signal theory' conversation and receive update notifications?

Ask