2.9 Machine learning lecture 9 course notes  (Page 2/3)

If we were fitting a full, unconstrained, covariance matrix $\Sigma$ to data, it was necessary that $m\ge n+1$ in order for the maximum likelihood estimate of $\Sigma$ not to be singular. Under either of the two restrictions above, we may obtain non-singular $\Sigma$ when $m\ge 2$ .

However, restricting $\Sigma$ to be diagonal also means modeling the different coordinates $x_i$ , $x_j$ of the data as being uncorrelated and independent. Often, it would be nice to be able to capture some interesting correlation structure in the data. If we were to useeither of the restrictions on $\Sigma$ described above, we would therefore fail to do so. In this set of notes, we will describe the factor analysis model,which uses more parameters than the diagonal $\Sigma$ and captures some correlations in the data, but also without having to fit a full covariance matrix.

Marginals and conditionals of gaussians

Before describing factor analysis, we digress to talk about how to find conditional and marginal distributions of random variables with a joint multivariate Gaussian distribution.

Suppose we have a vector-valued random variable

where $x_1\in {\mathbb{R}}^r$ , $x_2\in {\mathbb{R}}^s$ , and $x\in {\mathbb{R}}^{r+s}$ . Suppose $x\sim \mathcal{N}(\mu ,\Sigma )$ , where

Here, ${\mu }_1\in {\mathbb{R}}^r$ , ${\mu }_2\in {\mathbb{R}}^s$ , ${\Sigma }_{11}\in {\mathbb{R}}^{r\times r}$ , ${\Sigma }_{12}\in {\mathbb{R}}^{r\times s}$ , and so on. Note that since covariance matrices are symmetric, ${\Sigma }_{12}={\Sigma }_{21}^T$ .

Under our assumptions, $x_1$ and $x_2$ are jointly multivariate Gaussian. What is the marginal distribution of $x_1$ ? It is not hard to see that $\mathrm{E}\left[x_1\right]={\mu }_1$ , and that $\mathrm{Cov}\left(x_1\right)=\mathrm{E}\left[(x_1-{\mu }_1)(x_1-{\mu }_1{)}_T\right]={\Sigma }_{11}$ . To see that the latteris true, note that by definition of the joint covariance of $x_1$ and $x_2$ , we have that

Matching the upper-left subblocks in the matrices in the second and the last lines above gives the result.

Since marginal distributions of Gaussians are themselves Gaussian, we therefore have that the marginal distribution of $x_1$ is given by $x_1\sim \mathcal{N}({\mu }_1,{\Sigma }_{11})$ .

Also, we can ask, what is the conditional distribution of $x_1$ given $x_2$ ? By referring to the definition of the multivariate Gaussian distribution, itcan be shown that $x_1|x_2\sim \mathcal{N}({\mu }_{1|2},{\Sigma }_{1|2})$ , where

When working with the factor analysis model in the next section, these formulas for finding conditional and marginal distributions of Gaussians will be very useful.

The factor analysis model

In the factor analysis model, we posit a joint distribution on $(x,z)$ as follows, where $z\in {\mathbb{R}}^k$ is a latent random variable:

Here, the parameters of our model are the vector $\mu \in {\mathbb{R}}^n$ , the matrix $\Lambda \in {\mathbb{R}}^{n\times k}$ , and the diagonal matrix $\Psi \in {\mathbb{R}}^{n\times n}$ . The value of $k$ is usually chosen to be smaller than $n$ .

Thus, we imagine that each datapoint $x^{\left(i\right)}$ is generated by sampling a $k$ dimension multivariate Gaussian $z^{\left(i\right)}$ . Then, it is mapped to a $k$ -dimensional affine space of ${\mathbb{R}}^n$ by computing $\mu +\Lambda z^{\left(i\right)}$ . Lastly, $x^{\left(i\right)}$ is generated by adding covariance $\Psi$ noise to $\mu +\Lambda z^{\left(i\right)}$ .

Equivalently (convince yourself that this is the case), we can therefore also define the factor analysis model according to

where $ϵ$ and $z$ are independent.