1.4 Machine learning lecture 5  (Page 8/15)

Student: Thank you.

Instructor (Andrew Ng) :Okay. Cool. So let’s talk a little bit about the advantages and disadvantages of using a generative learning algorithm, okay? So in the particular case of Gaussian discriminant analysis, we assume that X conditions on Y is Gaussian, and the argument I showed on the previous chalkboard, I didn’t prove it formally, but you can actually go back and prove it yourself is that if you assume X given Y is Gaussian, then that implies that when you plot Y given X, you find that – well, let me just write logistic posterior, okay?

And the argument I showed just now, which I didn’t prove; you can go home and prove it yourself, is that if you assume X given Y is Gaussian, then that implies that the posterior distribution or the form of PFY = 1 given X is going to be a logistic function, and it turns out this implication in the opposite direction does not hold true, okay?

In particular, it actually turns out – this is actually, kind of, cool. It turns out that if you’re seeing that X given Y = 1 is Hessian with parameter lambda 1, and X given Y = 0, is Hessian with parameter lambda 0. It turns out if you assumed this, then that also implies that PFY given X is logistic, okay?

So there are lots of assumptions on X given Y that will lead to PFY given X being logistic, and, therefore, this, the assumption that X given Y being Gaussian is the stronger assumption than the assumption that Y given X is logistic, okay? Because this implies this, right? That means that this is a stronger assumption than this because this, the logistic posterior holds whenever X given Y is Gaussian but not vice versa.

And so this leaves some of the tradeoffs between Gaussian discriminant analysis and logistic regression, right? Gaussian discriminant analysis makes a much stronger assumption that X given Y is Gaussian, and so when this assumption is true, when this assumption approximately holds, if you plot the data, and if X given Y is, indeed, approximately Gaussian, then if you make this assumption, explicit to the algorithm, then the algorithm will do better because it’s as if the algorithm is making use of more information about the data. The algorithm knows that the data is Gaussian, right? And so if the Gaussian assumption, you know, holds or roughly holds, then Gaussian discriminant analysis may do better than logistic regression.

If, conversely, if you’re actually not sure what X given Y is, then logistic regression, the discriminant algorithm may do better, and, in particular, use logistic regression, and maybe you see [inaudible]before the data was Gaussian, but it turns out the data was actually Poisson, right? Then logistic regression will still do perfectly fine because if the data were actually Poisson, then PFY = 1 given X will be logistic, and it’ll do perfectly fine, but if you assumed it was Gaussian, then the algorithm may go off and do something that’s not as good, okay?

So it turns out that – right. So it’s slightly different. It turns out the real advantage of generative learning algorithms is often that it requires less data, and, in particular, data is never really exactly Gaussian, right? Because data is often approximately Gaussian; it’s never exactly Gaussian.