In the applications of probability theory to real phenomena two results of the mathematical theory of probability play a conspicuous role. These results are known as the law of large numbers and central limit theorem. At this point in this book we have sufficient mathematical tools available to show how to apply these basic results. In Chapters 9 and 10 we develop the additional mathematical tools required to prove these theorems with a sufficient degree of generality.

A set of \(n\) observations \(X_{1}, X_{2}, \ldots, X_{n}\) are said to constitute a random sample of a random variable \(X\) if \(X_{1}, X_{2}, \ldots, X_{n}\) are independent random variables, identically distributed as \(X\) . Let

\[S_{n}=X_{1}+X_{2}+\cdots+X_{n}, \tag{5.1}\] 

be the sum of the observations. Their arithmetic mean

\[M_{n}=\frac{1}{n} S_{n}, \tag{5.2}\] 

is called the sample mean .

By (4.1), (4.6), and (4.7), we obtain the following expressions for the mean, variance, and moment-generating function of \(S_{n}\) and \(M_{n}\) , in terms of the mean, variance, and moment-generating function of \(X\) (assuming these exist):

\begin{align} E\left[S_{n}\right] & =n E[X], & \operatorname{Var}\left[S_{n}\right]=n \operatorname{Var}[X], & \quad \psi_{S_{n}}(t)=\left[\psi_{X}(t)\right]^{n}. \\ E\left[M_{n}\right] & =E[X], & \operatorname{Var}\left[M_{n}\right]=\frac{1}{n} \operatorname{Var}[X], & \quad \psi_{M_{n}}(t)=\left[\psi_{X}\left(\frac{t}{n}\right)\right]^{n}. \tag{5.3} \end{align} 

From (5.4) we obtain the striking fact that the variance of the sample mean \((1 / n) S_{n}\) tends to 0 as the sample size \(n\) tends to infinity. Now, by Chebyshev’s inequality, it follows that if a random variable has a small variance then it is approximately equal to its mean, in the sense that with probability close to 1 an observation of the random variable will yield an observed value approximately equal to the mean of the random variable; in particular, the probability is 0.99 that an observed value of the random variable is within 10 standard deviations of the mean of the random variable. We have thus established that the sample mean of a random sample \(X_{1}, X_{2}, \ldots, X_{n}\) of a random variable, with a probability that can be made as close to 1 as desired by taking a large enough sample, is approximately equal to the ensemble mean \(E[X]\) . This fact, known as the law of large numbers , was first established by Bernoulli in 1713 (see section 5 of Chapter 5). The validity of the law of large numbers is the mathematical expression of the fact that increasingly accurate measurements of a quantity (such as the length of a rod) are obtained by averaging an increasingly large number of observations of the value of the quantity. A precise mathematical statement and proof of the law of large numbers is given in Chapter 10.

However, even more can be proved about the sample mean than that it tends to be equal to the mean. One can approximately evaluate, for any interval about the mean, the probability that the sample mean will have an observed value in that interval, since the sample mean is approximately normally distributed. More generally, it may be shown that if \(S_{n}\) is the sum of independent identically distributed random variables \(X_{1}, X_{2}, \ldots, X_{n}\) , with finite means and variances then, for any real numbers \(a\begin{align} P\left[a \leq S_{n} \leq b\right] & =P\left[\frac{a-E\left[S_{n}\right]}{\sigma\left[S_{n}\right]} \leq \frac{S_{n}-E\left[S_{n}\right]}{\sigma\left[S_{n}\right]} \leq \frac{b-E\left[S_{n}\right]}{\sigma\left[S_{n}\right]}\right] \tag{5.5}\\[3mm] & =\Phi\left(\frac{b-E\left[S_{n}\right]}{\sigma\left[S_{n}\right]}\right)-\Phi\left(\frac{a-E\left[S_{n}\right]}{\sigma\left[S_{n}\right]}\right). \end{align} 

In words, (5.5) may be expressed as follows: the sum of a large number of independent identically distributed random variables with finite means and variances, nomalized to have mean zero and variance 1, is approximately normally distributed . Equation (5.5) represents a rough statement of one of the most important theorems of probability theory. In 1920 G. Polya gave this theorem the name “the central limit theorem of probability theory”. This name continues to be used today, although a more apt description would be “the normal convergence theorem”. The central limit theorem was first proved by De Moivre in 1733 for the case in which \(X_{1}, X_{2}, \ldots, X_{n}\) are Bernoulli random variables, so that \(S_{n}\) is then a binomial random variable. A proof of (5.5) in this case (with a continuity correction) was given in section 2 of Chapter 6. The determination of the exact conditions for the validity of (5.5) constituted the outstanding problem of probability theory from its beginning until the decade of the 1930’s. A precise mathematical statement and proof of the central limit theorem is given in Chapter 10 .

It may be of interest to outline the basic idea of the proof of (5.5), even though the mathematical tools are not at hand to justify the statements made. To prove (5.5) it suffices to prove that the moment-generating function

\[\psi_{n}(t)= E\left[e^{t\left(S_{n}-E\left[S_{n}\right]\right) / \sigma\left[S_{n}\right]}\right]=\left\{\psi_{X-E[X]}\left(\frac{t}{\sqrt{n} \sigma[X]}\right)\right\}^{n} \tag{5.6}\] 

satisfies for \(t\) in a neighborhood of 0

\[\lim _{n \rightarrow \infty} \log \psi_{n}(t)=\frac{t^{2}}{2}, \tag{5.7}\] 

in which \(t^{2} / 2\) is the logarithm of the moment-generating function of a random variable \(X\) , which is \(N(0,1)\) . Now, expanding in Taylor series,

\[\psi_{X-E[X]}(u)=1+\frac{1}{2} \sigma^{2}[X] u^{2}+A(u), \tag{5.8}\] 

where the remainder \(A(u)\) satisfies the condition \(\lim_{u \rightarrow 0} A(u) / u^{2}=0\) . Similarly, \(\log (1+v)=v+B(v)\) where \(\displaystyle\lim _{v \rightarrow 0} B(v) / v=0\) . Consequently one may show that for values of \(u\) sufficiently close to 0

\[\log \psi_{X-E[X]}(u)=\frac{1}{2} \sigma^{2}\left[X^{2}\right] u^{2}+C(u), \tag{5.9}\] 

where

\[\lim _{u \rightarrow 0} \frac{C(u)}{u^{2}}=0. \tag{5.10}\] 

It then follows that

\[\log \psi_{n}(t)=n \log \psi_{X-E[X]}\left(\frac{t}{\sqrt{n} \sigma[X]}\right)=\frac{t^{2}}{2}+n C\left(\frac{t}{\sqrt{n} \sigma[X]}\right), \tag{5.11}\] 

where

\[\lim _{n \rightarrow \infty} n C\left(\frac{t}{\sqrt{n} \sigma[X]}\right)=0. \tag{5.12}\] 

From (5.11) and (5.12) one obtains (5.7). Our heuristic outline of the proof of (5.5) is now complete.

Given any random variable \(X\) with finite mean and variance, we define its standardization , denoted by \(X^{*}\) , as the random variable

\[X^{*}=\frac{X-E[X]}{\sigma[X]}, \tag{5.13}\] 

The standardization \(X^{*}\) is a dimensionless random variable, with mean \(E\left[X^{*}\right]=0\) and variance \(\sigma^{2}\left[X^{*}\right]=1\) .

The central limit theorem of probability theory can now be formulated: The standardization \(\left(S_{n}\right)^{*}\) of the sum \(S_{n}\) of a large number \(n\) of independent and identically distributed random variables is approximately normally distributed . In Chapter 10 it is shown that this result may be considerably extended to include cases in which \(S_{n}\) is the sum of dependent nonidentically distributed random variables.

Exercises