The evaluation of expectations requires the use of operations of summation and integration for which completely routine methods are not available. We now discuss a method of evaluating the moments of a probability law, which, when available, requires the performance of only one summation or integration, after which all the moments of the probability law can be obtained by routine differentiation.
The moment-generating function of a probability law is a function \(\psi(\cdot)\) , defined for all real numbers \(t\) by
\[\psi(t)=E\left[e^{t x}\right]. \tag{3.1}\]
In words, \(\psi(t)\) is the expectation of the exponential function \(e^{i x}\) .
In the case of a discrete probability law, specified by a probability mass function \(p(\cdot)\) , the moment-generating function is given by
\[\psi(t)=\sum_{\substack{\text { over all points } x \text { such } \\ \text { that } p(x)>0}} e^{t x} p(x). \tag{3.2}\]
In the case of a continuous probability law, specified by a probability density function \(f(\cdot)\) , the moment-generating function is given by
\[\psi(t)=\int_{-\infty}^{\infty} e^{t x} f(x) dx. \tag{3.3}\]
Since, for fixed \(t\) , the integrand \(e^{t x}\) is a positive function of \(x\) , it follows that \(\psi(t)\) is either finite or infinite. We say that a probability law possesses a moment-generating function if there exists a positive number \(T\) such that \(\psi(t)\) is finite for \(|t| \leq T\) . It may then be shown that all moments of the probability law exist and may be expressed in terms of the successive derivatives at \(t=0\) of the moment-generating function [see (3.5) ]. We have already shown that there are probability laws without finite means. Consequently, probability laws that do not possess moment-generating functions also exist. It may be seen in Chapter 6 that for every probability law one can define a function, called the characteristic function, that always exists and can be used as a moment-generating function to obtain those moments that do exist.
If a moment-generating function \(\psi(t)\) exists for \(|t| \leq T\) (for some \(T>0\) ), then one may form its successive derivatives by successively differentiating under the integral or summation sign. Consequently, we obtain
\begin{align} \psi^{\prime}(t)&=\frac{d}{d t} \psi(t)=E\left[\frac{\partial}{\partial t} e^{t x}\right]=E\left[x e^{t x}\right] \\[2mm] \psi^{\prime \prime}(t)&=\frac{d^{2}}{d t^{2}} \psi(t)=E\left[\frac{\partial}{\partial t} x e^{t x}\right]=E\left[x^{2} e^{t x}\right] \tag{3.4} \\[2mm] \psi^{(3)}(t)&=\frac{d^{3}}{d t^{3}} \psi(t)=E\left[\frac{\partial}{\partial t} x^{2} e^{t x}\right]=E\left[x^{3} e^{t x}\right] \\[2mm] & \cdots \\[2mm] \psi^{(n)}(t)&=\frac{d^{n}}{d t^{n}} \psi(t)=E\left[x^{n} e^{t x}\right]. \end{align}
Letting \(t=0\) , we obtain \begin{align} \psi^{\prime}(0)& =E[x] \\[2mm] \psi^{\prime \prime}(0)& =E\left[x^{2}\right] \\[2mm] \psi^{(3)}(0)& =E\left[x^{3}\right] \tag{3.5} \\[2mm] & \ldots \\[2mm] \psi^{(n)}(0)& =E\left[x^{n}\right] \end{align}
If the moment-generating function \(\psi(t)\) is finite for \(|t| \leq T\) (for some \(T>0\) ), it then possesses a power-series expansion (valid for \(|t|
\[\psi(t)=1+E[x] t+E\left[x^{2}\right] \frac{t^{2}}{2 !}+\cdots+E\left[x^{n}\right] \frac{t^{n}}{n !}+\cdots. \tag{3.6}\]
To prove (3.6), use the definition of \(\psi(t)\) and the fact that
\[e^{t x}=1+x t+x^{2} \frac{t^{2}}{2 !}+\cdots+x^{n} \frac{t^{n}}{n !}+\cdots. \tag{3.7}\]
In view of (3.6), if one can readily obtain the power-series expansion of \(\psi(t)\) , then one can readily obtain the \(n\) th moment \(E\left[x^{n}\right]\) for any integer \(n\) , since \(E\left[x^{n}\right]\) is the coefficient of \(t^{n} / n\) ! in the power-series expansion of \(\psi(t)\) .
Example 3A . The Bernoulli probability law with parameter \(p\) has a moment-generating function for \(-\infty
\[\psi(t)=e^{t 0} \cdot q+e^{t 1} \cdot p=p e^{t}+q \tag{3.8}\]
with derivatives
\[\begin{array}{ll} \psi^{\prime}(t)=p e^{t}, & E[x]=\psi^{\prime}(0)=p, \tag{3.9}\\ \psi^{\prime \prime}(t)=p e^{t}, & E\left[x^{2}\right]=\psi^{\prime \prime}(0)=p. \end{array}\]
Example 3B . The binomial probability law with parameters \(n\) and \(p\) has a moment-generating function for \(-\infty
\[\psi(t)=\sum_{k=0}^{n} e^{t k} p(k)=\sum_{k=0}^{n}\left(\begin{array}{l} n \tag{3.10}\\ k \end{array}\right)\left(p e^{t}\right)^{k} q^{n-k}=\left(p e^{t}+q\right)^{n},\]
with derivatives
\begin{align} \psi^{\prime}(t) & =n p e^{t}\left(p e^{t}+q\right)^{n-1}, \quad E[x]=\psi^{\prime}(0)=n p, \\ \psi^{\prime \prime}(t) & =n p e^{t}\left(p e^{t}+q\right)^{n-1}+n(n-1) p^{2} e^{2 t}\left(p e^{t}+q\right)^{n-2}, \tag{3.11}\\ E\left[x^{2}\right] & =n p+n(n-1) p^{2}=n p q+n^{2} p^{2}. \end{align}
| Probability Law | Parameters | Probability Mass Function \(p(\cdot)\) | Mean \(m = E[x]\) | Variance \(\sigma^2 = E[x^2] - E^2[x]\) |
|---|---|---|---|---|
| Bernoulli | \(0 \leq p \leq 1\) | \(p(x) = \begin{cases} p & x = 1 \\ q & x = 0 \\ 0 & \text{otherwise} \end{cases}\) | \(p\) | \(pq\) |
| Binomial | \(n = 1, 2, \ldots\) \(0 \leq p \leq 1\) | \(p(x) = \begin{cases} \displaystyle \binom{n}{x} p^x q^{n-x} & x = 0, 1, 2, \ldots, n \\ 0 & \text{otherwise} \end{cases}\) | \(np\) | \(npq\) |
| Poisson | \(\lambda > 0\) | \(p(x) = \begin{cases} e^{-\lambda} \dfrac{\lambda^x}{x!} & x = 0, 1, 2, \ldots \\ 0 & \text{otherwise} \end{cases}\) | \(\lambda\) | \(\lambda\) |
| Geometric | \(0 \leq p \leq 1\) | \(p(x) = \begin{cases} pq^{x-1} & x = 1, 2, \ldots \\ 0 & \text{otherwise} \end{cases}\) | \(\dfrac{1}{p}\) | \(\dfrac{q}{p^2}\) |
| Negative binomial | \(r > 0\) \(0 \leq p \leq 1\) | \(p(x) = \begin{cases} \displaystyle \binom{r+x-1}{x} p^r q^x & x = 0, 1, 2, \ldots \\ \displaystyle \binom{-r}{x} p^r (-q)^x & x = 0, 1, 2, \ldots \\ 0 & \text{otherwise} \end{cases}\) | \(\dfrac{rq}{p} = rP \text{ if } P = \dfrac{q}{p}\) | \(\dfrac{rq}{p^2} = rPQ \text{ if } Q = \dfrac{1}{p}\) |
| Hypergeometric | \(N = 1, 2, \ldots\) \(n = 1, 2, \ldots, N\) \(p = 0, \dfrac{1}{N}, \dfrac{2}{N}, \ldots, 1\) | \(p(x) = \begin{cases} \dfrac{\displaystyle \binom{Np}{x} \binom{Nq}{n-x}}{\displaystyle \binom{N}{n}} & x = 0, 1, \ldots, n \\ 0 & \text{otherwise} \end{cases}\) | \(np\) | \(npq \dfrac{N - n}{N - 1}\) |
| Probability Law | Moment-Generating Function \(\psi(t) = E[e^{tx}]\) | Characteristic Function \(\phi(u) = E[e^{iux}]\) | Third Central Moment \(E[(x - E[x])^3]\) | Fourth Central Moment \(E[(x - E[x])^4]\) |
|---|---|---|---|---|
| Bernoulli | \(pe^t + q\) | \(pe^{iu} + q\) | \(pq(q - p)\) | \(3p^2q^2 + pq(1 - 6pq)\) |
| Binomial | \((pe^t + q)^n\) | \((pe^{iu} + q)^n\) | \(npq(q - p)\) | \(3n^2p^2q^2 + pqn(1 - 6pq)\) |
| Poisson | \(e^{\lambda(e^t - 1)}\) | \(e^{\lambda(e^{iu} - 1)}\) | \(\lambda\) | \(\lambda + 3\lambda^2\) |
| Geometric | \(\frac{pe^t}{1 - qe^t}\) | \(\frac{pe^{iu}}{1 - qe^{iu}}\) | \(\frac{q}{p^2}\left(1 + 2\frac{q}{p}\right)\) | \(\frac{q}{p^2}\left(1 + 9\frac{q}{p^2}\right)\) |
| Negative binomial | \(\left(\frac{p}{1 - qe^t}\right)^r = (Q - Pe^t)^{-r}\) | \(\left(\frac{p}{1 - qe^{iu}}\right)^r = (Q - Pe^{iu})^{-r}\) | \(\frac{rq}{p^2}\left(1 + 2\frac{q}{p}\right) = rPQ(Q + P)\) | \(\frac{rq}{p^2}\left(1 + (6 + 3r)\frac{q}{p^2}\right) = 3r^2p^2Q^2 + rPQ(1 + 6PQ)\) |
| Hypergeometric | see M. G. Kendall, Advanced Theory of Statistics , Charles Griffin, London, 1948, p. 127. | |||
| Probability Law | Parameters | Probability Density Function \(f(\cdot)\) | Mean \(m = E[x]\) | Variance \(\sigma^2 = E[x^2] - E^2[x]^2\) |
|---|---|---|---|---|
| Uniform over interval \(a\) to \(b\) | \(-\infty < a < b < \infty\) | \(f(x) = \begin{cases} \dfrac{1}{b-a} & a < x < b \\ 0 & \text{otherwise} \end{cases}\) | \(\dfrac{a + b}{2}\) | \(\dfrac{(b - a)^2}{12}\) |
| Normal | \(-\infty < m < \infty\) \(\sigma > 0\) | \(f(x) = \dfrac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2} \left(\dfrac{x - m}{\sigma}\right)^2}\) | \(m\) | \(\sigma^2\) |
| Exponential | \(\lambda > 0\) | \(f(x) = \begin{cases} \lambda e^{-\lambda x} & x > 0 \\ 0 & \text{otherwise} \end{cases}\) | \(\dfrac{1}{\lambda}\) | \(\dfrac{1}{\lambda^2}\) |
| Gamma | \(r > 0\) | \(f(x) = \begin{cases} \frac{\lambda}{\Gamma(r)} (\lambda x)^{r-1} e^{-\lambda x} & x > 0 \\ 0 & \text{otherwise} \end{cases}\) | \(\dfrac{r}{\lambda}\) | \(\dfrac{r}{\lambda^2}\) |
| Probability Law | Moment-Generating Function \(\psi(t) = E[e^{tx}]\) | Characteristic Function \(\phi(u) = E[e^{iux}]\) | Third Central Moment \(E[(x - E[x])^3]\) | Fourth Central Moment \(E[(x - E[x])^4]\) |
|---|---|---|---|---|
| Uniform over interval \(a\) to \(b\) | \(\dfrac{e^{tb} - e^{ta}}{t(b - a)}\) | \(\dfrac{e^{iub} - e^{iua}}{iu(b - a)}\) | \(0\) | \(\dfrac{(b - a)^4}{80}\) |
| Normal | \(e^{tm + \frac{1}{2}t^2\sigma^2}\) | \(e^{ium - \frac{1}{2}u^2\sigma^2}\) | \(0\) | \(3\sigma^4\) |
| Exponential | \(\dfrac{\lambda}{\lambda - t} = \left(1 - \dfrac{t}{\lambda}\right)^{-1}\) | \(\left(1 - \dfrac{iu}{\lambda}\right)^{-1}\) | \(\dfrac{2}{\lambda^3}\) | \(\dfrac{9}{\lambda^4}\) |
| Gamma | \(\left(1 - \frac{t}{\lambda}\right)^{-r}\) | \(\left(1 - \dfrac{iu}{\lambda}\right)^{-r}\) | \(\dfrac{2r}{\lambda^3}\) | \(\dfrac{6r + 3r^2}{\lambda^4}\) |
Example 3C . The Poisson probability law with parameter \(\lambda\) has a moment-generating function for all \(t\) .
\[\psi(t)=\sum_{k=0}^{\infty} e^{t k} p(k)=e^{-\lambda} \sum_{k=0}^{\infty} \frac{\left(\lambda e^{t}\right)^{k}}{k !}=e^{-\lambda} e^{\lambda e^{t}}=e^{\lambda\left(e^{t}-1\right)}, \tag{3.12}\] with derivatives \begin{align} \psi^{\prime}(t) & =e^{\lambda\left(e^{t}-1\right)} \lambda e^{t}, \quad E[x]=\psi^{\prime}(0)=\lambda, \\ \psi^{\prime \prime}(t) & =e^{\lambda\left(e^{t}-1\right)} \lambda^{2} e^{2 t}+\lambda e^{t} e^{\lambda\left(e^{t}-1\right)}, \quad E\left[x^{2}\right]=\psi^{\prime \prime}(0)=\lambda^{2}+\lambda. \tag{3.13} \end{align}
Consequently, the variance \(\sigma^{2}=E\left[x^{2}\right]-E^{2}[x]=\lambda\) . Thus for the Poisson probability law the mean and the variance are equal .
Example 3D . The geometric probability law with parameter \(p\) has a moment-generating function for \(t\) such that \(q e^{t}<1\) .
\[\psi(t)=\sum_{k=1}^{\infty} e^{t k} p(k)=p e^{t} \sum_{k=1}^{\infty}\left(q e^{t}\right)^{k-1}=p e^{t} \frac{1}{1-q e^{t}}. \tag{3.14}\]
From (3.14) one may show that the mean and variance of the geometric probability law are given by
\[m=E[x]=\frac{1}{p}, \quad \sigma^{2}=E\left[x^{2}\right]-E^{2}[x]=\frac{q}{p^{2}}. \tag{3.15}\]
Example 3E . The normal probability law with mean \(m\) and variance \(\sigma\) has a moment-generating function for \(-\infty
\begin{align} \psi(t) & =\frac{1}{\sqrt{2 \pi} \sigma} \int_{-\infty}^{\infty} e^{t x} e^{-1 / 2\left(\frac{x-m}{\sigma}\right)^{2}} d x=e^{m t} \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{t y \sigma} e^{-1 / 2 y^{2}} d y \tag{3.16}\\ & =e^{m t+(1 / 2) \sigma^{2} t^{2}} \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{-1 / 2(y-\sigma t)^{2}} d y=e^{m t+(1 / 2) \sigma^{2} t^{2}} . \end{align}
From (3.16) one may show that the central moments of the normal probability law are given by \begin{align} E\left[(x-m)^{n}\right] & =0 \quad \text { if } n=3,5, \ldots, \tag{3.17}\\ & =1 \cdot 3 \cdot 5 \cdots(n-1) \sigma^{n} \quad \text { if } n=2,4, \ldots. \end{align} An alternate method of deriving (3.17) is by use of (2.22) in Chapter 4.
Example 3F . The exponential probability law with parameter \(\lambda\) has a moment-generating function for \(t<\lambda\) .
\[\psi(t)=\lambda \int_{0}^{\infty} e^{t x} e^{-\lambda x} d x=\frac{\lambda}{\lambda-t}=\left(1-\frac{t}{\lambda}\right)^{-1}. \tag{3.18}\]
One may show from (3.18) that for the exponential probability law the mean \(m\) and the standard deviation \(\sigma\) are equal, and are equal to the reciprocal of the parameter \(\lambda\) .
Example 3G . The lifetime of a radioactive atom . It is shown in section 4 of Chapter 6 that the time between emissions of particles by a radioactive atom obeys an exponential probability law with parameter \(\lambda\) . By example 3F, the mean time between emissions is \(1 / \lambda\) . The time between emissions is called the lifetime of the atom. The half life of the atom is defined as the time \(T\) such that the probability is \(\frac{1}{2}\) that the lifetime will be greater than \(T\) . Since the probability that the lifetime will be greater than a given number \(t\) is \(e^{-\lambda t}\) , it follows that \(T\) is the solution of \(e^{-\lambda T}=\frac{1}{2}\) , or \(T=\log _{e} 2 / \lambda\) . In words, the half life \(T\) is equal to the mean \(1 / \lambda\) multiplied by \(\log _{e} 2\) .
Theoretical Exercises
3.1 . Generating function of moments about a point . Define the moment-generating function of a probability law about a point \(c\) as a function \(\psi_{c}(\cdot)\) defined for all real numbers \(t\) by \(\psi_{c}(t)=E\left[e^{l(x-c)}\right]\) . Show that \(\psi_{c}(t)\) may be obtained from \(\psi(t)\) by \(\psi_{c}(t)=e^{-c t} \psi(t)\) . The \(n\) th moment \(E\left[(x-c)^{n}\right]\) of the probability law about the point \(c\) is given by \(E\left[(x-c)^{n}\right]=\psi_{c}^{(n)}(0)\) and may be read off as the coefficient of \(t^{n} / n\) ! in the power-series expansion of \(\psi_{c}(t)\) .
3.2 . The factorial moment-generating function . \(\Phi(u)\) of a probability law is defined for all \(u\) such that \(|u|<1\) by \[\Phi(u)=E\left[(1+u)^{x}\right]=E\left[e^{x \log (1+u)}\right]=\psi[\log (1+u)].\] Its \(n\) th derivative evaluated at \(u=0\)
\[\Phi^{(n)}(0)=E[x(x-1) \cdots(x-n+1)]\]
is called the \(n\) th factorial moment of the probability law. From a knowledge of the first \(n\) factorial moments of a probability law one may obtain a knowledge of the first \(n\) moments of the probability law, and conversely. Thus, for example,
\[E[x(x-1)]=E\left[x^{2}\right]-E[x],\qquad E\left[x^{2}\right]=E[x(x-1)]+E[x].\tag{3.19}\]
Equation (3.19) was implicitly used in calculating certain second moments and variances in section 2. Show that the first \(n\) moments of two distinct probability laws coincide if and only if their first \(n\) factorial moments coincide.
Hint : Consult M. Kendall, The Advanced Theory of Statistics , Vol. I, Griffin, London, 1948, p. 58.
3.3 . The factorial moment-generating function of the probability law of the number of matches in the matching problem . The number of matches obtained by distributing, 1 to an urn, \(M\) balls, numbered 1 to \(M\) , among \(M\) urns, numbered 1 to \(M\) , has a probability law specified by the probability mass function
\begin{align} p(m) &= \begin{cases} \frac{1}{m!} \displaystyle\sum_{k=0}^{M-m} (-1)^k \frac{1}{k!}, & \text{for } m=0,1,2, \ldots, M \tag{3.20}\\[2mm] 0, & \text{otherwise.} \end{cases} \end{align}
Show that the corresponding moment-generating function may be written
\[\psi(t)=\sum_{m=0}^{M} e^{t m} p(m)=\sum_{r=0}^{M} \frac{1}{r !}\left(e^{t}-1\right)^{r}. \tag{3.21}\]
Consequently the factorial moment-generating function of the number of matches may be written
\[\Phi(u)=\sum_{r=0}^{M} \frac{1}{r !} u^{r}. \tag{3.22}\]
3.4 . The first \(M\) moments of the number of matches in the problem of matching \(M\) balls in \(M\) urns coincide with the first \(M\) moments of the Poisson probability law with parameter \(\lambda=1\) . Show that the factorial moment-generating function of the Poisson law with parameter \(\lambda\) is given by
\[\Phi_{P}(u)=e^{\lambda u}=\sum_{r=0}^{\infty} \frac{\lambda^{r}}{r !} u^{r}. \tag{3.23}\]
By comparing (3.22) and (3.23), it follows that the first \(M\) factorial moments, and, consequently, the first \(M\) moments of the probability law of the number of matches and the Poisson probability law with parameter 1, coincide.
Exercises
Compute the moment generating function, mean, and variance of the probability law specified by the probability density function, probability mass function, or distribution function given.
3.1 . \begin{align} &\text{(i)} \quad f(x) = \begin{cases} e^{-x}, & \text{for } x \geq 0 \\ 0, & \text{for } \text { elsewhere. } \end{cases} \\[2mm] &\text{(ii)} \quad f(x) = \begin{cases} e^{-(x-5)}, & \text{for } x \geq 5 \\ 0, & \text{for } \text { elsewhere. } \end{cases} \end{align}
Answer
(i) \(1 /(1-t)\) ; (ii) \(e^{5 t} /(1-t)\) .
3.2 . \begin{align} &\text{(i)} \quad f(x) = \begin{cases} \frac{1}{\sqrt{2 \pi x}} e^{-x / 2}, & \text{for } x > 0 \\ 0, & \text{elsewhere.} \end{cases} \\[2mm] &\text{(i)} \quad f(x) = \begin{cases} \frac{1}{4} x e^{-x / 2}, & \text{for } x > 0 \\ 0, & \text{elsewhere.} \end{cases} \end{align}
3.3 . \begin{align} &\text{(i)} \quad p(x) = \begin{cases} \frac{2}{3} (\frac{1}{3})^{x-1}, & \text{for } x=1, 2, \dots \\ 0, & \text{elsewhere.} \end{cases} \\[2mm] &\text{(i)} \quad f(x) = \begin{cases} e^{-2}\frac{2^{x}}{x!}, & \text{for } x=0, 1, \dots \\ 0, & \text{elsewhere.} \end{cases} \end{align}
Answer
(i) \(2 e^{t} /\left(3-e^{t}\right)\) ; (ii) \(e^{2\left(e^{t}-1\right)}\) .
3.4 . \begin{align} &\text{(i)} \quad F(x) = \Phi\left(\frac{x-2}{2}\right) \\[2mm] &\text{(ii)} \quad F(x) = \begin{cases} 0, & \text{for } x < 0 \tag{ii} \\ 1 - e^{-x / 5}, & \text{for } x \geq 0. \end{cases} \end{align}
3.5 . Find the mean, variance, third central moment, and fourth central moment of the number of matches when (i) 4 balls are distributed in 4 urns, 1 to an urn, (ii) 3 balls are distributed in 3 urns, 1 to an urn.
Answer
(i) \(1,1,1,4\) ; (ii) \(1,1,1,3\) .
3.6 . Find the factorial moment-generating function of the (i) binomial, (ii) Poisson, (iii) geometric probability laws and use it to obtain their means, variances, and third and fourth central moments.