1.1 The maximum likelihood estimation method  (Page 2/2)

The pdf of the joint distribution shown in (1) is known as the likelihood function . If the sample were not independently drawn, the pdf of joint distribution could not be written in such a simple form because of the covariance among the members of the sample would not be equal to zero . The logarithm of this function (or as it is referred to, the log of the likelihood function) is given by the sum $$L\left(x_1,x_2,\dots ,x_n\right)=\ln f\left(x_1\right)+\ln f\left(x_2\right)+\cdots +\ln f\left(x_n\right)={\displaystyle \sum _{i=1}^n\ln f\left(x_i\right)}.$$ The maximum likelihood method involves choosing as estimators of the unknown parameters of the distribution the values that maximize the likelihood function. However, because the logarithm is a monotonically increasing function The function $g\left(y\right)$ is monotonically increasing for y if $g^{\prime }\left(y\right)>0.$ Because $\frac{d}{dx}\ln x=\frac{1}{x}>0\text{for}x>0,$ the logarithm function is monotonically increasing for positive values of $x.$ , maximizing the log of the likelihood function is equivalent to maximizing the likelihood function. The following example of this procedure illustrates how to derive ML estimators.

The ml estimator of the population mean and population variance.

Assume that $x~N\left(\mu ,{\sigma }^2\right).$ Consider a sample of size n drawn independently from this distribution. The likelihood function is the product of the pdf of each observation or:

Thus, the log of the likelihood function of this sample is $$L\left(x_1,x_2,\dots ,x_n\right)=-\frac{n\ln 2\pi }{2}-n\ln \sigma -\frac{{\displaystyle \sum _{i=1}^n{\left(x_i-\mu \right)}^2}}{2{\sigma }^2}.$$ In the ML method we want to find the estimators of the mean and variance, $\stackrel{⌢}{\mu }$ and $\stackrel{⌢}{\sigma }$ , that maximize the log of the likelihood function. Substituting in the parameter estimates into the log of the likelihood function gives our problem as:

Setting the derivatives of the log of the likelihood function with respect to $\stackrel{⌢}{\mu }$ and $\stackrel{⌢}{\sigma }$ equal to 0 gives:

Solving these two equations simultaneously gives:

Notice the fact that the estimator of the population mean is equal to the sample mean, a result that is the same as the one you found in your introductory statistics course. However, the unbiased estimator of the population variance used in that course is $$s^2=\frac{{\displaystyle \sum {\left(x_i-\stackrel{⌢}{\mu }\right)}^2}}{n-1}.$$

Thus, one of the common "problems" with using a ML estimator is that quite often they are biased estimators of a population parameter. On the other hand, under very general conditions ML estimators are consistent , are asymptotically efficient , and have an asymptotically normal distribution (these are desirable large sample size characteristics of potential estimators and are discussed in advanced statistics courses). Intuitively, what these concepts mean is that as the sample size increases the estimator becomes more precise (the variance becomes smaller and an bias disappears) and the distribution of the estimator approaches the normal distribution. The formal definitions of these terms involve advanced statistical concepts that are reported here only in the interest of completeness. An estimator $\left(\widehat{\theta }\right)$ of the parameter $\theta$ is consistent if and only if $p\lim \widehat{\theta }=\theta .$ This estimator has an asymptotically normal distribution if $\widehat{\theta }\stackrel{a}{\to }N\left(\theta ,{\left\{I\left(\theta \right)\right\}}^{-1}\right).$ An unbiased estimator is more efficient that another unbiased estimator if it has a smaller variance than the alternative estimator. An asymptotically efficient is an estimator whose mean square error tends to zero as the sample size increases. The mean square error (MSE) is defined to be $MSE\left(\widehat{\theta }\right)=E\left[{\left(\widehat{\theta }-\theta \right)}^2\right]=V\left(\widehat{\theta }\right)+{\left(Bias\left[\widehat{\theta }\right]\right)}^2.$ An estimator is asymptotically efficient if $\underset{n\to \infty }{\lim }MSE\left(\widehat{\theta }\right)=0.$ See any advanced statistics text or Statistical terminology for further information on these concepts.

Application of the ml method to regressions

The discussion above illustrates the basics of the ML method—you form the log of the likelihood function and then find the values of the parameter estimates that maximize this function. In most cases the maximization will not yield answers in closed form—that is, you cannot find a neat algebraic formula as we did for the population mean. However, you can use computer programs to search for the values of the parameter estimates that maximize this function. Thus, in most cases in advanced regression models you often will treat the ML method as a “black box” and not concern yourself with the estimation details. However, I illustrate one more example of the ML technique.

You can use the examples in this module as the basis of your understanding of the ML method. When you see that the ML method is used in a computer program, you can be fairly certain that the program uses one of the many optimizing subroutines to find the maximum of the log of the likelihood program. You can consult the help files with the computer program to see what underlying distribution is used to set up the log of the likelihood function. A concept related to the maximum likelihood estimation method worth exploring is the likelihood ratio test (see the module by Don Johnson entitled The Likelihood Ratio Test for an introduction to this key statistical test.)

Exercises

1. $f\left(x;\theta \right)=\left(\theta +1\right)x^{\theta }$ where $0\le x\le 1$ and $\theta >0.$
2. $f\left(x;\theta \right)=\theta e^{-\theta x}$ where $0\le x<\infty$ and $\theta >0.$