Machine learning lecture 1 course notes  (Page 8/13)

Logistic regression

We could approach the classification problem ignoring the fact that $y$ is discrete-valued, and use our old linear regression algorithm to try to predict $y$ given $x$ . However, it is easy to construct examples where this method performs very poorly.Intuitively, it also doesn't make sense for $h_{\theta }\left(x\right)$ to take values larger than 1 or smaller than 0 when we know that $y\in \{0,1\}$ .

To fix this, let's change the form for our hypotheses $h_{\theta }\left(x\right)$ . We will choose

where

is called the logistic function or the sigmoid function . Here is a plot showing $g\left(z\right)$ :

Notice that $g\left(z\right)$ tends towards 1 as $z\to \infty$ , and $g\left(z\right)$ tends towards 0 as $z\to -\infty$ . Moreover, g(z), and hence also $h\left(x\right)$ , is always bounded between 0 and 1. As before, we are keeping the conventionof letting $x_0=1$ , so that ${\theta }^{T}x={\theta }_0+{\sum }_{j=1}^n{\theta }_j{x}_j$ .

For now, let's take the choice of $g$ as given. Other functions that smoothly increase from 0 to 1 can also be used, but for a couple of reasons that we'llsee later (when we talk about GLMs, and when we talk about generative learning algorithms),the choice of the logistic function is a fairly natural one. Before moving on, here's a useful property of the derivative of the sigmoid function,which we write as $g^{\text{'}}$ :

So, given the logistic regression model, how do we fit $\theta$ for it? Following how we saw least squares regression could be derived as the maximum likelihood estimatorunder a set of assumptions, let's endow our classification model with a set of probabilistic assumptions,and then fit the parameters via maximum likelihood.

Let us assume that

Note that this can be written more compactly as

Assuming that the $m$ training examples were generated independently, we can then write down the likelihood of the parameters as

As before, it will be easier to maximize the log likelihood:

How do we maximize the likelihood? Similar to our derivation in the case of linear regression, we can use gradient ascent. Written in vectorialnotation, our updates will therefore be given by $\theta :=\theta +\alpha {\nabla }_{\theta }\ell \left(\theta \right)$ . (Note the positive rather than negative sign in the update formula, since we're maximizing, ratherthan minimizing, a function now.) Let's start by working with just one training example $(x,y)$ , and take derivatives to derive the stochastic gradient ascent rule:

Above, we used the fact that $g^{\text{'}}\left(z\right)=g\left(z\right)(1-g\left(z\right))$ . This therefore gives us the stochastic gradient ascent rule

If we compare this to the LMS update rule, we see that it looks identical; but this is not the same algorithm, because $h_{\theta }\left(x^{\left(i\right)}\right)$ is now defined as a non-linear function of ${\theta }^T{x}^{\left(i\right)}$ . Nonetheless, it's a little surprising that we end up with the same update rule for a rather differentalgorithm and learning problem. Is this coincidence, or is there a deeper reason behind this? We'll answer this when get get to GLM models. (See alsothe extra credit problem on Q3 of problem set 1.)