2.2 Machine learning lecture 3 course notes  (Page 5/12)

Consider the following, which we'll call the primal optimization problem:

To solve it, we start by defining the generalized Lagrangian

Here, the ${\alpha }_i$ 's and ${\beta }_i$ 's are the Lagrange multipliers. Consider the quantity

Here, the “ $\mathcal{P}$ ” subscript stands for “primal.” Let some $w$ be given. If $w$ violates any of the primal constraints (i.e., if either $g_i\left(w\right)>0$ or $h_i\left(w\right)\ne 0$ for some $i$ ), then you should be able to verify that

Conversely, if the constraints are indeed satisfied for a particular value of $w$ , then ${\theta }_{\mathcal{P}}\left(w\right)=f\left(w\right)$ . Hence,

Thus, ${\theta }_{\mathcal{P}}$ takes the same value as the objective in our problem for all values of $w$ that satisfies the primal constraints, and is positive infinity if the constraints are violated. Hence, if weconsider the minimization problem

we see that it is the same problem (i.e., and has the same solutions as) our original, primal problem. For later use, we also define the optimal value of the objective tobe $p^{*}={min}_w{\theta }_{\mathcal{P}}\left(w\right)$ ; we call this the value of the primal problem.

Now, let's look at a slightly different problem. We define

Here, the “ $\mathcal{D}$ ” subscript stands for “dual.” Note also that whereas in the definition of ${\theta }_{\mathcal{P}}$ we were optimizing (maximizing) with respect to $\alpha ,\beta$ , here are are minimizing with respect to $w$ .

We can now pose the dual optimization problem:

This is exactly the same as our primal problem shown above, except that the order of the “ $max$ ” and the “ $min$ ” are now exchanged. We also define the optimal value of the dual problem's objective to be $d^{*}={max}_{\alpha ,\beta :{\alpha }_i\ge 0}{\theta }_{\mathcal{D}}\left(w\right)$ .

How are the primal and the dual problems related? It can easily be shown that

(You should convince yourself of this; this follows from the “ $maxmin$ ” of a function always being less than or equal to the “ $minmax$ .”) However, under certain conditions, we will have

so that we can solve the dual problem in lieu of the primal problem. Let's see what these conditions are.

Suppose $f$ and the $g_i$ 's are convex, When $f$ has a Hessian, then it is convex if and only if the Hessian is positive semi-definite. For instance, $f\left(w\right)=w^{T}w$ is convex; similarly, all linear (and affine) functions are also convex.(A function $f$ can also be convex without being differentiable, but we won't need those more general definitions of convexity here.) and the $h_i$ 's are affine. I.e., there exists $a_i$ , $b_i$ , so that $h_i\left(w\right)=a_i^{T}w+b_i$ . “Affine” means the same thing as linear, except that we also allow the extra intercept term $b_i$ . Suppose further that the constraints $g_i$ are (strictly) feasible; this means that there exists some $w$ so that $g_i\left(w\right)<0$ for all $i$ .

Under our above assumptions, there must exist $w^{*},{\alpha }^{*},{\beta }^{*}$ so that $w^{*}$ is the solution to the primal problem, ${\alpha }^{*},{\beta }^{*}$ are the solution to the dual problem, and moreover $p^{*}=d^{*}=\mathcal{L}(w^{*},{\alpha }^{*},{\beta }^{*})$ . Moreover, $w^{*},{\alpha }^{*}$ and ${\beta }^{*}$ satisfy the Karush-Kuhn-Tucker (KKT) conditions , which are as follows: