2.11 Machine learning lecture 11 course notes

Independent components analysis

Our next topic is Independent Components Analysis (ICA). Similar to PCA, this will find a new basis in which to represent our data. However, the goal is verydifferent.

As a motivating example, consider the “cocktail party problem.” Here, $n$ speakers are speaking simultaneously at a party, and any microphone placed in the room records only an overlapping combination of the $n$ speakers' voices. But let's say we have $n$ different microphones placed in the room, and because each microphone is a different distance from eachof the speakers, it records a different combination of the speakers' voices. Using these microphone recordings, can we separate out the original $n$ speakers' speech signals?

To formalize this problem, we imagine that there is some data $s\in {\mathbb{R}}^n$ that is generated via $n$ independent sources. What we observe is

where $A$ is an unknown square matrix called the mixing matrix . Repeated observations gives us a dataset $\{x^{\left(i\right)};i=1,...,m\}$ , and our goal is to recover the sources $s^{\left(i\right)}$ that had generated our data ( $x^{\left(i\right)}=A{s}^{\left(i\right)}$ ).

In our cocktail party problem, $s^{\left(i\right)}$ is an $n$ -dimensional vector, and $s_j^{\left(i\right)}$ is the sound that speaker $j$ was uttering at time $i$ . Also, $x^{\left(i\right)}$ in an $n$ -dimensional vector, and $x_j^{\left(i\right)}$ is the acoustic reading recorded by microphone $j$ at time $i$ .

Let $W=A^{-1}$ be the unmixing matrix. Our goal is to find $W$ , so that given our microphone recordings $x^{\left(i\right)}$ , we can recover the sources by computing $s^{\left(i\right)}=W{x}^{\left(i\right)}$ . For notational convenience, we also let $w_i^T$ denote the $i$ -th row of $W$ , so that

Thus, $w_i\in {\mathbb{R}}^n$ , and the $j$ -th source can be recovered by computing $s_j^{\left(i\right)}=w_j^T{x}^{\left(i\right)}$ .

Ica ambiguities

To what degree can $W=A^{-1}$ be recovered? If we have no prior knowledge about the sources and the mixing matrix, it is not hard to see that there are some inherent ambiguities in $A$ that are impossible to recover, given only the $x^{\left(i\right)}$ 's.

Specifically, let $P$ be any $n$ -by- $n$ permutation matrix. This means that each row and each column of $P$ has exactly one “1.” Here're some examples of permutation matrices:

If $z$ is a vector, then $Pz$ is another vector that's contains a permuted version of $z$ 's coordinates. Given only the $x^{\left(i\right)}$ 's, there will be no way to distinguish between $W$ and $PW$ . Specifically, the permutation of the original sources is ambiguous, which should be no surprise. Fortunately, thisdoes not matter for most applications.

Further, there is no way to recover the correct scaling of the $w_i$ 's. For instance, if $A$ were replaced with $2A$ , and every $s^{\left(i\right)}$ were replaced with $(0.5)s^{\left(i\right)}$ , then our observed $x^{\left(i\right)}=2A·(0.5)s^{\left(i\right)}$ would still be the same. More broadly, if a single column of $A$ were scaled by a factor of $\alpha$ , and the corresponding source were scaled by a factor of $1/\alpha$ , then there is again no way, given only the $x^{\left(i\right)}$ 's to determine that this had happened. Thus, we cannot recover the “correct” scaling of the sources. However, for the applications that we are concernedwith—including the cocktail party problem—this ambiguity also does not matter. Specifically, scaling a speaker's speech signal $s_j^{\left(i\right)}$ by some positive factor $\alpha$ affects only the volume of that speaker's speech. Also, sign changes do not matter, and $s_j^{\left(i\right)}$ and $-s_j^{\left(i\right)}$ sound identical when played on a speaker. Thus, if the $w_i$ found by an algorithm is scaled by any non-zero real number, the corresponding recovered source $s_i=w_i^{T}x$ will be scaled by the same factor; but this usually does not matter. (These comments also apply to ICA for the brain/MEGdata that we talked about in class.)