2.4 Machine learning lecture 5 course notes  (Page 4/4)

(The equation above assumes that $x_i$ and $y$ are binary-valued; more generally the summations would be over the domains of the variables.) The probabilities above $p(x_i,y)$ , $p\left(x_i\right)$ and $p\left(y\right)$ can all be estimated according to their empirical distributions on the training set.

To gain intuition about what this score does, note that the mutual information can also be expressed as a Kullback-Leibler (KL) divergence:

You'll get to play more with KL-divergence in Problem set #3, but informally, this gives a measure of how different the probabilitydistributions $p(x_i,y)$ and $p\left(x_i\right)p\left(y\right)$ are. If $x_i$ and $y$ are independent random variables, then we would have $p(x_i,y)=p\left(x_i\right)p\left(y\right)$ , and the KL-divergence between the two distributions will be zero. This is consistent with the idea if $x_i$ and $y$ are independent, then $x_i$ is clearly very “non-informative” about $y$ , and thus the score $S\left(i\right)$ should be small. Conversely, if $x_i$ is very “informative” about $y$ , then their mutual information $\mathrm{MI}(x_i,y)$ would be large.

One final detail: Now that you've ranked the features according to their scores $S\left(i\right)$ , how do you decide how many features $k$ to choose? Well, one standard way to do so is to use cross validation to select among the possible values of $k$ . For example, when applying naive Bayes to text classification—a problem where $n$ , the vocabulary size, is usually very large—using this method to select a feature subset oftenresults in increased classifier accuracy.

Bayesian statistics and regularization

In this section, we will talk about one more tool in our arsenal for our battle against overfitting.

At the beginning of the quarter, we talked about parameter fitting using maximum likelihood (ML), and chose our parameters according to

Throughout our subsequent discussions, we viewed $\theta$ as an unknown parameter of the world.This view of the $\theta$ as being constant-valued but unknown is taken in frequentist statistics. In the frequentist this view of the world, $\theta$ is not random—it just happens to be unknown—and it's our job to comeup with statistical procedures (such as maximum likelihood) to try to estimate this parameter.

An alternative way to approach our parameter estimation problems is to take the Bayesian view of the world, and think of $\theta$ as being a random variable whose value is unknown.In this approach, we would specify a prior distribution $p\left(\theta \right)$ on $\theta$ that expresses our “prior beliefs” about the parameters. Given a training set $S={\left\{(x^{\left(i\right)},y^{\left(i\right)})\right\}}_{i=1}^m$ , when we are asked to make a prediction on a new value of $x$ , we can then compute the posterior distribution on the parameters

In the equation above, $p\left(y^{\left(i\right)}\right|x^{\left(i\right)},\theta )$ comes from whatever model you're using for your learning problem. For example, if you are using Bayesian logisticregression, then you might choose $p\left(y^{\left(i\right)}\right|x^{\left(i\right)},\theta )=h_{\theta }{\left(x^{\left(i\right)}\right)}^{y^{\left(i\right)}}{(1-h_{\theta }\left(x^{\left(i\right)}\right))}^{(1-y^{\left(i\right)})}$ , where $h_{\theta }\left(x^{\left(i\right)}\right)=1/(1+exp(-{\theta }^T{x}^{\left(i\right)}))$ . Since we are now viewing $\theta$ as a random variable, it is okay to condition on it value, and write “ $p\left(y\right|x,\theta )$ ” instead of “ $p\left(y\right|x;\theta )$ .”

When we are given a new test example $x$ and asked to make it prediction on it, we can compute our posterior distribution on the class label using theposterior distribution on $\theta$ :

In the equation above, $p\left(\theta \right|S)$ comes from [link] . Thus, for example, if the goal is to the predict the expectedvalue of $y$ given $x$ , then we would output The integral below would be replaced by a summation if $y$ is discrete-valued.

The procedure that we've outlined here can be thought of as doing “fully Bayesian” prediction, where our prediction is computed by taking an average with respectto the posterior $p\left(\theta \right|S)$ over $\theta$ . Unfortunately, in general it is computationally very difficult to compute thisposterior distribution. This is because it requires taking integrals over the (usually high-dimensional) $\theta$ as in [link] , and this typically cannot be done in closed-form.

Thus, in practice we will instead approximate the posterior distribution for $\theta$ . One common approximation is to replace our posterior distribution for $\theta$ (as in [link] ) with a single point estimate. The MAP (maximum a posteriori) estimate for $\theta$ is given by

Note that this is the same formulas as for the ML (maximum likelihood) estimate for $\theta$ , except for the prior $p\left(\theta \right)$ term at the end.

In practical applications, a common choice for the prior $p\left(\theta \right)$ is to assume that $\theta \sim \mathcal{N}(0,{\tau }^{2}I)$ . Using this choice of prior, the fitted parameters ${\theta }_{\mathrm{MAP}}$ will have smaller norm than that selected by maximum likelihood.(See Problem Set #3.) In practice, this causes the Bayesian MAP estimate to be lesssusceptible to overfitting than the ML estimate of the parameters. For example, Bayesian logistic regression turns out to be an effective algorithm fortext classification, even though in text classification we usually have $n\gg m$ .