1.4 Machine learning lecture 5  (Page 11/15)

So say I get a piece of email, and what I’ll do is I’ll then scan through this list of words, and wherever a certain word appears in my email, I’ll put a 1 there. So if a particular email has the word “aid” then that’s 1. You know, my email doesn’t have the words ausworth or aardvark, so it gets zeros. And again, a piece of email, they want me to buy something, CS229 doesn’t occur, and so on, okay? So this would be one way of creating a feature vector to represent a piece of email.

Now, let’s throw the generative model out for this. Actually, let’s use this. In other words, I want to model PFX given Y. The given Y = 0 or Y = 1, all right? And my feature vectors are going to be 0, 1 to the N. It’s going to be these split vectors, binary value vectors. They’re N dimensional. Where N may be on the order of, say, 50,000, if you have 50,000 words in your dictionary, which is not atypical. So values from – I don’t know, mid-thousands to tens of thousands is very typical for problems like these.

And, therefore, there two to the 50,000 possible values for X, right? So two to 50,000 possible bit vectors of length 50,000, and so one way to model this is the multinomial distribution, but because there are two to the 50,000 possible values for X, I would need two to the 50,000, but maybe -1 parameters, right? Because you have this sum to 1, right? So -1. And this is clearly way too many parameters to model using the multinomial distribution over all two to 50,000 possibilities.

So in a Naive Bayes algorithm, we’re going to make a very strong assumption on PFX given Y, and, in particular, I’m going to assume – let me just say what it’s called; then I’ll write out what it means. I’m going to assume that the XI’s are conditionally independent given Y, okay?

Let me say what this means. So I have that PFX1, X2, up to X50,000, right, given the Y. By the key rule of probability, this is PFX1 given Y times PFX2 given Y, X1 times PF – I’ll just put dot, dot, dot. I’ll just write 1, 1 × dot, dot, dot up to, you know, well – whatever. You get the idea, up to PFX50,000, okay?

So this is the chain were of probability. This always holds. I’ve not made any assumption yet, and now, we’re gonna meet what’s called the Naive Bayes assumption, or this assumption that X defies a conditionally independent given Y. Going to assume that – well, nothing changes for the first term, but I’m gonna assume that PFX3 given Y, X1 is equal to PFX2 given the Y. I’m gonna assume that that term’s equal to PFX3 given the Y, and so on, up to PFX50,000 given Y, okay? Or just written more compactly, means assume that PFX1, PFX50,000 given Y is the product from I = 1 to 50,000 or PFXI given the Y, okay?

And stating informally what this means is that I’m, sort of, assuming that – so unless you know the cost label Y, so long as you know whether this is spam or not spam, then knowing whether the word “A” appears in email does not affect the probability of whether the word “Ausworth” appears in the email, all right?