2.7 Sufficient statistics

Introduction

Sufficient statistics arise in nearly every aspect of statistical inference. It is important to understandthem before progressing to areas such as hypothesis testing and parameter estimation.

Suppose we observe an $N$ -dimensional random vector $X$ , characterized by the density or mass function $f(, x)$ , where $$ is a $p$ -dimensional vector of parameters to be estimated. The functional form of $f(x)$ is assumed known. The parameter $$ completely determines the distribution of $X$ . Conversely, a measurement $x$ of $X$ provides information about $$ through the probability law $f(, x)$ .

The $N$ -dimensional observation $X$ carries information about the $p$ -dimensional parameter vector $$ . If $p< N$ , one may ask the following question: Can we compress $x$ into a low-dimensional statistic without any loss of information?Does there exist some function $t=T(x)$ , where the dimension of $t$ is $M< N$ , such that $t$ carries all the useful information about $$ ?

If so, for the purpose of studying $$ we could discard the raw measurements $x$ and retain only the low-dimensional statistic $t$ . We call $t$ a sufficient statistic . The following definition captures this notion precisely:

Let $X_1,,X_M$ be a random sample, governed by the density or probability mass function $f(, x)$ . The statistic $T(x)$ is sufficient for $$ if the conditional distribution of $x$ , given $T(x)=t$ , is independent of $$ . Equivalently, the functional form of $f(, t, x)$ does not involve $$ .

1. Let $f(, x, t)$ denote the joint density or probability mass function on $(X,T(X\left)\right)$ . If $T(X)$ is a sufficient statistic for $$ , then

2. Given $t=T(x)$ , full knowledge of the measurement $x$ brings no additional information about $$ . Thus, we may discard $x$ and retain on the compressed statistic $t$ .

3. Any inference strategy based on $f(, x)$ may be replaced by a strategy based on $f(, t)$ .

Determining sufficient statistics

In the example above , we had to guess the sufficient statistic, and work out theconditional probability by hand. In general, this will be a tedious way to go about finding sufficientstatistics. Fortunately, spotting sufficient statistics can be made easier by the Fisher-Neyman Factorization Theorem .

Uses of sufficient statistics

Sufficient statistics have many uses in statistical inference problems. In hypothesis testing, the Likelihood Ratio Test can often be reduced to a sufficient statistic of the data. In parameter estimation, the Minimum Variance Unbiased Estimator of a parameter $$ can be characterized by sufficient statistics and the Rao-Blackwell Theorem .

Minimality and completeness

Minimal sufficient statistics are, roughly speaking, sufficient statistics that cannot becompressed any more without losing information about the unknown parameter. Completeness is a technical characterization of sufficient statistics that allows one toprove minimality. These topics are covered in detail in this module.

Further examples of sufficient statistics may be found in the module on the Fisher-Neyman Factorization Theorem .