Machine learning lecture 1 course notes  (Page 4/13)

The normal equations

Gradient descent gives one way of minimizing $J$ . Let's discuss a second way of doing so, this time performing the minimization explicitlyand without resorting to an iterative algorithm. In this method, we will minimize $J$ by explicitly taking its derivatives with respect to the ${\theta }_j$ 's, and setting them to zero. To enable us to do this without having to write reamsof algebra and pages full of matrices of derivatives, let's introduce some notation for doing calculus with matrices.

Matrix derivatives

For a function $f:{\mathbb{R}}^{m\times n}\mapsto \mathbb{R}$ mapping from $m$ -by- $n$ matrices to the real numbers, we define the derivative of $f$ with respect to $A$ to be:

Thus, the gradient ${\nabla }_{A}f\left(A\right)$ is itself an $m$ -by- $n$ matrix, whose $(i,j)$ -element is $\partial f/\partial A_{ij}$ . For example, suppose $A=\left[\begin{array}{cc}{A}_{11}& A_{12}\\ A_{21}& A_{22}\end{array}\right]$ is a 2-by-2 matrix, and the function $f:{\mathbb{R}}^{2\times 2}\mapsto \mathbb{R}$ is given by

Here, $A_{ij}$ denotes the $(i,j)$ entry of the matrix $A$ . We then have

We also introduce the trace operator, written “ $\mathrm{tr}$ .” For an $n$ -by- $n$ (square) matrix $A$ , the trace of $A$ is defined to be the sum of its diagonal entries:

If $a$ is a real number (i.e., a 1-by-1 matrix), then $\mathrm{tr}a=a$ . (If you haven't seen this “operator notation” before, you shouldthink of the trace of $A$ as $\mathrm{tr}\left(A\right)$ , or as application of the “trace” function to the matrix $A$ . It's more commonly written without the parentheses, however.)

The trace operator has the property that for two matrices $A$ and $B$ such that $AB$ is square, we have that $\mathrm{tr}AB=\mathrm{tr}BA$ . (Check this yourself!) As corollaries of this, we also have, e.g.,

The following properties of the trace operator are also easily verified. Here, $A$ and $B$ are square matrices, and $a$ is a real number:

We now state without proof some facts of matrix derivatives (we won't need some of these until later this quarter).  [link] applies only to non-singular square matrices $A$ , where $\left|A\right|$ denotes the determinant of $A$ . We have:

To make our matrix notation more concrete, let us now explain in detail the meaning of the first of these equations. Suppose we have some fixedmatrix $B\in {\mathbb{R}}^{n\times m}$ . We can then define a function $f:{\mathbb{R}}^{m\times n}\mapsto \mathbb{R}$ according to $f\left(A\right)=\mathrm{tr}AB$ . Note that this definition makes sense, because if $A\in {\mathbb{R}}^{m\times n}$ , then $AB$ is a square matrix, and we can apply the trace operator to it; thus, $f$ does indeed map from ${\mathbb{R}}^{m\times n}$ to $\mathbb{R}$ . We can then apply our definition of matrix derivativesto find ${\nabla }_{A}f\left(A\right)$ , which will itself by an $m$ -by- $n$ matrix. [link] above states that the $(i,j)$ entry of this matrix will be given by the $(i,j)$ -entry of $B^T$ , or equivalently, by $B_{ji}$ .

The proofs of the first three equations in [link] are reasonably simple, and are left asan exercise to the reader. The fourth equation in [link] can be derived using the adjoint representation of the inverse of a matrix. If we define $A^{\text{'}}$ to be the matrix whose $(i,j)$ element is ${(-1)}^{i+j}$ times the determinant of the square matrix resulting from deleting row $i$ and column $j$ from $A$ , then it can be proved that $A^{-1}={\left(A^{\text{'}}\right)}^T/\left|A\right|$ . (You can check that this is consistent with the standard wayof finding $A^{-1}$ when $A$ is a 2-by-2 matrix. If you want to see a proof of this more general result, see an intermediate or advancedlinear algebra text, such as Charles Curtis, 1991, Linear Algebra , Springer.) This shows that $A^{\text{'}}=\left|A\right|{\left(A^{-1}\right)}^T$ . Also, the determinant of a matrix can be written $\left|A\right|={\sum }_j{A}_{ij}{A}_{ij}^{\text{'}}$ . Since ${\left(A^{\text{'}}\right)}_{ij}$ does not depend on $A_{ij}$ (as can be seen from its definition), this implies that $(\partial /\partial A_{ij})\left|A\right|=A_{ij}^{\text{'}}$ . Putting all this together shows the result.