Random variables, which arise as, or may be represented as, sums of other random variables, play an important role in probability theory. In this section we obtain formulas for the mean, mean square, variance, and moment-generating function of a sum of random variables.
Let \(X_{1}, X_{2}, \ldots, X_{n}\) be \(n\) jointly distributed random variables. Using the linearity properties of the expecration operation, we immediately obtain the following formulas for the mean, mean square, and variance of the sum:
\begin{align} E\left[\sum_{k=1}^{n} X_{k}\right] & =\sum_{k=1}^{n} E\left[X_{k}\right]; \tag{4.1}\\ E\left[\left(\sum_{k=1}^{n} X_{k}\right)^{2}\right] & =\sum_{k=1}^{n} E\left[X_{k}^{2}\right]+2 \sum_{k=1}^{n} \sum_{j=k+1}^{n} E\left[X_{k} X_{j}\right]; \tag{4.2}\\ \operatorname{Var}\left[\sum_{k=1}^{n} X_{k}\right] & =\sum_{k=1}^{n} \operatorname{Var}\left[X_{k}\right]+2 \sum_{k=1}^{n} \sum_{j=k+1}^{n} \operatorname{Cov}\left[X_{k}, X_{j}\right] . \tag{4.3} \end{align}
Equations (4.2) and (4.3) follow from the facts
\begin{align} \left(\sum_{k=1}^{n} X_{k}\right)^{2} & =\sum_{k=1}^{n} \sum_{j=1}^{n} X_{k} X_{j}=\sum_{k=1}^{n}\left(\sum_{j=1}^{k-1} X_{k} X_{j}+X_{k}^{2}+\sum_{j=k+1}^{n} X_{k} X_{j}\right), \tag{4.4}\\ \sum_{k=1}^{n} \sum_{j=1}^{k-1} X_{k} X_{j} & =\sum_{j=1}^{n} \sum_{k=j+1}^{n} X_{k} X_{j}=\sum_{k=1}^{n} \sum_{j=k+1}^{n} X_{k} X_{j}. \tag{4.5} \end{align}
Equation (4.3) simplifies considerably if the random variables \(X_{1}, X_{2}\) , \(\ldots, X_{n}\) are uncorrelated (by which is meant that \(\operatorname{Cov}\left[X_{k}, X_{j}\right]=0\) for every \(k \neq j\) ). Then the variance of the sum of the random variables is equal to the sum of the variances of the random variables; in symbols,
\[\operatorname{Var}\left[\sum_{k=1}^{n} X_{k}\right]=\sum_{k=1}^{n} \operatorname{Var}\left[X_{k}\right] \quad \text { if } \operatorname{Cov}\left[X_{k}, X_{j}\right]=0 \quad \text { for } k \neq j. \tag{4.6}\]
If the random variables \(X_{1}, X_{2}, \ldots, X_{n}\) are independent , then we may give a formula for the moment-generating function of their sum; for any real number \(t\)
\[\psi_{X_{1}+X_{2}+\cdots+X_{n}}(t)=\psi_{X_{1}}(t) \psi_{X_{2}}(t) \cdots \psi_{X_{n}}(t). \tag{4.7}\]
In words, the moment-generating function of the sum of independent random variables is equal to the product of their moment-generating functions . The importance of the moment-generating function in probability theory derives as much from the fact that (4.7) holds as from the fact that the moment-generating function may be used to compute moments. The proof of (4.7) follows immediately, once we rewrite (4.7) explicitly in terms of expectations:
\[E\left[e^{t\left(X_{1}+\cdots+x_{n}\right)}\right]=E\left[e^{t x_{1}}\right] \cdots E\left[e^{t X_{n}}\right]. \tag{$4.7^{\prime}$}\]
Equations (4.1)-(4.3) are useful for finding the mean and variance of a random variable \(Y\) (without knowing the probability law of \(Y\) ) if one can represent \(Y\) as a sum of random variables \(X_{1}, X_{2}, \ldots, X_{n}\) , the mean, variances, and covariances of which are known.
Example 4A. A binomial random variable as a sum . The number of successes in \(n\) independent repeated Bernoulli trials with probability \(p\) of success at each trial is a random variable. Let us denote it by \(S_{n}\) . It has been shown that \(S_{n}\) obeys a binomial probability law with parameters \(n\) and \(p\) . Consequently,
\[E\left[S_{n}\right]=n p, \quad \operatorname{Var}\left[S_{n}\right]=n p q, \quad \psi_{S_{n}}(t)=\left(p e^{t}+q\right)^{n}. \tag{4.8}\]
We now show that (4.8) is an immediate consequence of (4.1), (4.6), and (4.7). Define random variables \(X_{1}, X_{2}, \ldots, X_{n}\) by \(X_{k}=1\) or 0 , depending on whether the outcome of the \(k\) th trial is a success or a failure. One may verify that (i) \(S_{n}=X_{1}+X_{2}+\cdots+X_{n}\) ; (ii) \(X_{1}, \ldots, X_{n}\) are independent random variables; (iii) for \(k=1,2, \ldots, n, X_{k}\) is a Bernoulli random variable, with mean \(E\left[X_{k}\right]=p\) , variance \(\operatorname{Var}\left[X_{k}\right]=p q\) , and moment-generating function \(\psi_{X_{k}}(t)=p e^{t}+q\) . The desired conclusion may now be inferred.
Example 4B. A hypergeometric random variable as a sum. The number of white balls drawn in a sample of size \(n\) drawn without replacement from an urn containing \(N\) balls, of which \(a=N p\) are white, is a random variable. Let us denote it by \(S_{n}\) . It has been shown that \(S_{n}\) obeys a hypergeometric probability law. Consequently,
\[E\left[S_{n}\right]=n p, \quad \operatorname{Var}\left[S_{n}\right]=n p q \frac{N-n}{N-1}. \tag{4.9}\]
We now show that (4.9) can be derived by means of (4.1) and (4.3), without knowing the probability law of \(S_{n}\) . Define random variables \(X_{1}, X_{2}, \ldots, X_{n}: X_{k}=1\) or 0, depending on whether a white ball is or is not drawn on the \(k\) th draw. Verify that (i) \(S_{n}=X_{1}+X_{2}+\cdots+X_{n}\) ; (ii) for \(k=1,2, \ldots, n, X_{k}\) is a Bernoulli random variable, with mean \(E\left[X_{k}\right]=p\) and \(\operatorname{Var}\left[X_{k}\right]=p q\) . However, the random variables \(X_{1}, \ldots, X_{n}\) are not independent, and we need to compute their product moments \(E\left[X_{j} X_{k}\right]\) and covariances \(\operatorname{Cov}\left[X_{j}, X_{k}\right]\) for any \(j \neq k\) . Now, \(E\left[X_{j} X_{k}\right]=\) \(P\left[X_{j}=1, X_{k}=1\right]\) , so that \(E\left[X_{j} X_{k}\right]\) is equal to the probability that the balls drawn on the \(j\) th and \(k\) th draws are both white, which is equal to \([a(a-1)] /[N(N-1)]\) . Therefore,
\(\operatorname{Cov}\left[X_{j}, X_{k}\right]=E\left[X_{j} X_{k}\right]-E\left[X_{j}\right] E\left[X_{k}\right]=\frac{a(a-1)}{N(N-1)}-p^{2}=\frac{-p q}{N-1}\) .
Consequently, \[\operatorname{Var}\left[S_{n}\right]=n p q+n(n-1)\left(\frac{-p q}{N-1}\right)=n p q\left(1-\frac{n-1}{N-1}\right).\] The desired conclusions may now be inferred.
Example 4C. The number of occupied urns as a sum. If \(n\) distinguishable balls are distributed into \(M\) distinguishable urns in such a way that each ball is equally likely to go into any urn, what is the expected number of occupied urns?
Solution
For \(k=1,2, \ldots, M\) let \(X_{k}=1\) or 0, depending on whether the \(k\) th urn is or is not occupied. Then \(S=X_{1}+X_{2}+\cdots+X_{M}\) is the number of occupied urns, and \(E[S]\) the expected number of occupied urns. The probability that a given urn will be occupied is equal to \(1-[1-(1 / M)]^{n}\) . Therefore, \(E\left[X_{k}\right]=1-[1-(1 / M)]^{n}\) and \(E[S]=\) \(M\left\{1-[1-(1 / M)]^{n}\right\}\) .
Theoretical Exercises
4.1. Waiting times in coupon collecting . Assume that each pack of cigarettes of a certain brand contains one of a set of \(N\) cards and that these cards are distributed among the packs at random (assume that the number of packsavailable is infinite). Let \(S_{N}\) be the minimum number of packs that must be purchased in order to obtain a complete set of \(N\) cards. Show that \(E\left[S_{N}\right]=N \sum_{k=1}^{N}(1 / k)\) , which may be evaluated by using the formula (see \(H\) . Cramér, Mathematical Methods of Statistics , Princeton University Press, 1946, p. 125)
\[\sum_{k=1}^{N} \frac{1}{k}=0.57722+\log _{e} N+\frac{1}{2 N}+R_{N},\]
in which \(0
4.2 . Continuation of (4.1) . For \(r=1,2, \ldots, N\) let \(S_{r}\) be the minimum number of packs that must be purchased in order to obtain \(r\) different cards. Show that
\begin{align} E\left[S_{r}\right] & =N\left(\frac{1}{N}+\frac{1}{N-1}+\frac{1}{N-2}+\cdots+\frac{1}{N-r+1}\right) \\ \operatorname{Var}\left[S_{r}\right] & =N\left(\frac{1}{(N-1)^{2}}+\frac{2}{(N-2)^{2}}+\cdots+\frac{r-1}{(N-r+1)^{2}}\right). \end{align}
Show that approximately (for large \(N\) )
\[E\left[S_{r}\right] \doteq N \log \frac{N}{N-r+1}.\]
Show further that the moment-generating function of \(S_{r}\) is given by
\[\psi_{S_{r}}(t)=\prod_{k=0}^{r-1} \frac{(N-k) e^{t}}{\left(N-k e^{t}\right)}.\]
4.3. Continuation of (4.1) . For \(r\) preassigned cards let \(T_{r}\) be the minimum number of packs that must be purchased in order to obtain all \(r\) cards. Show that
\[E\left[T_{r}\right]=\sum_{k=1}^{r} \frac{N}{r-k+1}, \quad \operatorname{Var}\left[T_{r}\right]=\sum_{k=1}^{r} \frac{N(N-r+k-1)}{(r-k+1)^{2}}.\]
4.4 . The mean and variance of the number of matches . Let \(S_{M}\) be the number of matches obtained by distributing, 1 to an urn, \(M\) balls, numbered 1 to \(M\) , among \(M\) urns, numbered 1 to \(M\) . It was shown in theoretical exercise 3.3 of Chapter 5 that \(E\left[S_{M}\right]=1\) and \(\operatorname{Var}\left[S_{M}\right]=1\) . Show this, using the fact that \(S_{M}=X_{1}+\cdots+X_{M}\) , in which \(X_{k}=1\) or 0, depending on whether the \(k\) th urn does or does not contain ball number \(k\) . Hint: Show that \(\operatorname{Cov}\left[X_{j}, X_{k}\right]=(M-1) / M^{2}\) or \(1 / M^{2}(M-1)\) , depending on whether \(j=k\) or \(j \neq k\) .
4.5 . Show that if \(X_{1}, \ldots, X_{n}\) are independent random variables with zero means and finite fourth moments, then the third and fourth moments of the sum \(S_{n}=X_{1}+\cdots+X_{n}\) are given by
\[E\left[S_{n}^{3}\right]=\sum_{k=1}^{n} E\left[X_{k}^{3}\right], \quad E\left[S_{n}^{4}\right]=\sum_{k=1}^{n} E\left[X_{k}^{4}\right]+6 \sum_{k=1}^{n} E\left[X_{k}^{2}\right] \sum_{j=k+1}^{n} E\left[X_{j}^{2}\right] .\]
If the random variables \(X_{1}, \ldots, X_{n}\) are independent and identically distributed as a random variable \(X\) , then
\[E\left[S_{n}^{3}\right]=n E\left[X^{3}\right], \quad E\left[S_{n}^{4}\right]=n E\left[X^{4}\right]+3 n(n-1) E^{2}\left[X^{2}\right]\]
4.6 . Let \(X_{1}, X_{2}, \ldots, X_{n}\) be a random sample of a random variable \(X\) . Define the sample mean \(\bar{X}\) and the sample variance \(S^{2}\) by
\[\bar{X}=\frac{1}{n} \sum_{k=1}^{n} X_{k}, \quad S^{2}=\frac{1}{n-1} \sum_{k=1}^{n}\left(X_{k}-\bar{X}\right)^{2}.\]
(i) Show that \(E\left[S^{2}\right]=\sigma^{2}, \operatorname{Var}\left[S^{2}\right]=\left(\sigma^{4} / n\right)\left[\left(\mu_{4} / \sigma^{4}\right)-(n-3 / n-1)\right]\) , in which \(\sigma^{2}=\operatorname{Var}[X], \mu_{4}=E\left[(X-E[X])^{4}\right]\) . Hint : show that
\[\sum_{k=1}^{n}\left(X_{k}-E[X]\right)^{2}=\sum_{k=1}^{n}\left(X_{k}-\bar{X}\right)^{2}+n(\bar{X}-E[X])^{2} .\]
(ii) Show that \(\rho\left(X_{i}-\bar{X}, X_{j}-\bar{X}\right)=\frac{-1}{n-1}\) for \(i \neq j\) .
Exercises
4.1 . Let \(X_{1}, X_{2}\) , and \(X_{3}\) be independent normally distributed random variables, each with mean 1 and variance 3. Find \(P\left[X_{1}+X_{2}+X_{3}>0\right]\) .
Answer
0.8413.
4.2 . Consider a sequence of independent repeated Bernoulli trials in which the probability of success on any trial is \(p=\frac{5}{16}\) .
(i) Let \(S_{n}\) be the number of trials required to achieve the \(n\) th success. Find \(E\left[S_{n}\right]\) and \(\operatorname{Var}\left[S_{n}\right]\) .
Hint : Write \(S_{n}\) as a sum, \(S_{n}=X_{1}+\cdots+X_{n}\) , in which \(X_{k}\) is the number of trials between the \(k-1\) st and \(k\) th successes. The random variables \(X_{1}, \ldots, X_{n}\) are independent and identically distributed.
(ii) Let \(T_{n}\) be the number of failures encountered before the \(n\) th success is achieved. Find \(E\left[T_{n}\right]\) and \(\operatorname{Var}\left[T_{n}\right]\) .
4.3 . A fair coin is tossed \(n\) times. Let \(T_{n}\) be the number of times in the \(n\) tosses that a tail is followed by a head. Show that \(E\left[T_{n}\right]=(n-1) / 4\) and \(E\left[T_{n}^{2}\right]=\) \((n-1) / 4+[(n-2)(n-3)] / 16\) . Find Var \(\left[T_{n}\right]\) .
4.4 . A man with \(n\) keys wants to open his door. He tries the keys independently and at random. Let \(N_{n}\) be the number of trials required to open the door. Find \(E\left[N_{n}\right]\) and \(\operatorname{Var}\left[N_{n}\right]\) if (i) unsuccessful keys are not eliminated from further selections, (ii) if they are. Assume that exactly one of the keys can open the door.
In exercises 4.5 and 4.6 consider an item of equipment that is composed by assembling in a straight line 4 components of lengths \(X_{1}, X_{2}, X_{3}\) , and \(X_{4}\) , respectively. Let \(E\left[X_{1}\right]=20, E\left[X_{2}\right]=30, E\left[X_{3}\right]=40, E\left[X_{4}\right]=60\) .
4.5 . Assume \(\operatorname{Var}\left[X_{j}\right]=4\) for \(j=1, \ldots, 4\) .
(i) Find the mean and variance of the length \(L=X_{1}+X_{2}+X_{3}+X_{4}\) of the item if \(X_{1}, X_{2}, X_{3}\) , and \(X_{4}\) are uncorrelated.
(ii) Find the mean and variance of \(L\) if \(\rho\left(X_{j}, X_{k}\right)=0.2\) for \(1 \leq j
Answer
\(E[L]=150\) . (i) \(\operatorname{Var}[L]=16\) ; (ii) \(\operatorname{Var}[L]=25.6\) .
4.6 . Assume that \(\sigma\left[X_{j}\right]=(0.1) E\left[X_{j}\right]\) for \(j=1, \ldots, 4\) . Find the ratio \(E[L] / \sigma[L]\) , called the measurement signal-to-noise ratio of the length \(L\) (see section 6), for both cases considered in exercise 4.5.