An important tool in the study of the relationships that exist between two jointly distributed random variables, \(X\) and \(Y\) , is provided by the notion of conditional expectation. In section 3 of Chapter 7 the notion of the conditional distribution function \(F_{Y \mid X}(\cdot \mid x)\) of the random variable \(Y\) , given the random variable \(X\) , is defined. We now define the conditional mean of \(Y\) , given \(X\) , by

\[E[Y \mid X=x] = \left\{ \begin{array}{l} \displaystyle\int_{-\infty}^{\infty} y \, dF_{Y \mid X}(y \mid x) \tag{7.1} \\[5mm] \displaystyle\int_{-\infty}^{\infty} y f_{Y \mid X}(y \mid x) \, dy \\[5mm] \displaystyle\sum_{\substack{\text{over all } y \text{ such that} \\ p_{Y \mid X}(y \mid x) > 0}} y p_{Y \mid X}(y \mid x); \end{array} \right.\] 

the last two equations hold, respectively, in the cases in which \(F_{Y \mid X}(\cdot \mid x)\) is continuous or discrete. From a knowledge of the conditional mean of \(Y\) , given \(X\) , the value of the mean \(E[Y]\) may be obtained:

\[E[Y] = \begin{cases} \displaystyle\int_{-\infty}^{\infty} E[Y \mid X=x] \, dF_{X}(x) \tag{7.2} \\[5mm] \displaystyle\int_{-\infty}^{\infty} E[Y \mid X=x] f_{X}(x) \, dx \\[5mm] \displaystyle\sum_{\substack{\text{over all } x \text{ such that} \\ p_{X}(x) > 0}} E[Y \mid X=x] p_{X}(x) \end{cases}\] 

The conditional mean of one random variable, given another random variable, represents one possible answer to the problem of prediction . Suppose that a prospective father of height \(x_{1}\) wishes to predict the height of his unborn son. If the height of the son is regarded as a random variable \(X_{2}\) and the height \(x_{1}\) of the father is regarded as an observed value of a random variable \(X_{1}\) , then as the prediction of the son’s height we take the conditional mean \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) . The justification of this procedure is that the conditional mean \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) may be shown to have the property that

\begin{align} & E\left[\left(X_{2}-E\left[X_{2} \mid X_{1}=x_{1}\right]\right)^{2}\right] \tag{7.10} \\ & =\int_{-\infty}^{\infty} \int_{-\infty}^{\infty}\left(x_{2}-E\left[X_{2} \mid X_{1}=x_{1}\right]\right)^{2} f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right) d x_{1} d x_{2} \\ & \leq E\left[\left(X_{2}-g\left(X_{1}\right)\right)^{2}\right]=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty}\left[x_{2}-g\left(x_{1}\right)\right]^{2} f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right) d x_{1} d x_{2} \end{align} 

for any function \(g\left(x_{1}\right)\) for which the last written integral exists. In words, (7.10) is interpreted to mean that if \(X_{2}\) is to be predicted by a function \(g\left(X_{1}\right)\) of the random variable \(X_{1}\) then the conditional mean \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) has the smallest mean square error among all possible predictors \(g\left(X_{1}\right)\) .

From (7.7) it is seen that in the case in which the random variables are jointly normally distributed the problem of computing the conditional mean \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) may be reduced to that of computing the constants \(\alpha_{1}\) and \(\beta_{1}\) , for which one requires a knowledge only of the means, variances, and correlation coefficient of \(X_{1}\) and \(X_{2}\) . If these moments are not known, they must be estimated from observed data. The part of statistics concerned with the estimation of the parameters \(\alpha_{1}\) and \(\beta_{1}\) is called regression analysis .

It may happen that the joint probability law of the random variables \(X_{1}\) and \(X_{2}\) is unknown or is known but is such that the calculation of the conditional mean \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) is intractable. Suppose, however, that one knows the means, variances (assumed to be positive), and correlation coefficient of \(X_{1}\) and \(X_{2}\) . Then the prediction problem may be solved by forming the best linear predictor of \(X_{2}\) , given \(X_{1}\) , denoted by \(E^{*}\left[X_{2} \mid X_{1}=x_{1}\right]\) . The best linear predictor of \(X_{2}\) , given \(X_{1}\) , is defined as that linear function \(a+b X_{1}\) of the random variable \(X_{1}\) , that minimizes the mean square error of prediction \(E\left[\left(X_{2}-\left(a+b X_{1}\right)\right)^{2}\right]\) involved in the use of \(a+b X_{1}\) as a predictor of \(X_{2}\) . Now

\begin{align} -\frac{\partial}{\partial a} E\left[\left(X_{2}-\left(a+b X_{1}\right)\right)^{2}\right] & =2 E\left[X_{2}-\left(a+b X_{1}\right)\right] \tag{7.11} \\ -\frac{\partial}{\partial b} E\left[\left(X_{2}-\left(a+b X_{1}\right)\right)^{2}\right] & =2 E\left[\left(X_{2}-\left(a+b X_{1}\right)\right) X_{1}\right]. \end{align} 

Solving for the values of \(a\) and \(b\) , denoted by \(\alpha\) and \(\beta\) , at which these derivatives are equal to 0, one sees that \(\alpha\) and \(\beta\) satisfy the equations

\begin{align} \alpha+\beta E\left[X_{1}\right] & =E\left[X_{2}\right] \\ \alpha. E\left[X_{1}\right]+\beta E\left[X_{1}^{2}\right] & =E\left[X_{1} X_{2}\right] \tag{7.12} \end{align} 

Therefore, \(E^{*}\left[X_{2} \mid X_{1}=x_{1}\right]=\alpha+\beta x_{1}\) , in which

\[\alpha=E\left[X_{2}\right]-\beta E\left[X_{1}\right], \quad \beta=\frac{\operatorname{Cov}\left[X_{1}, X_{2}\right]}{\operatorname{Var}\left[X_{1}\right]}=\frac{\sigma\left[X_{2}\right]}{\sigma\left[X_{1}\right]} \rho\left(X_{1}, X_{2}\right). \tag{7.13}\] 

Comparing (7.7) and (7.13), one sees that the best linear predictor \(E^{*}\left[X_{2} \mid X_{1}=x_{1}\right]\) coincides with the best predictor, or conditional mean, \(E\left[X_{2} \mid X_{1}=x_{1}\right]\) , in the case in which the random variables \(X_{1}\) and \(X_{2}\) are jointly normally distributed.

We can readily compute the mean square error of prediction achieved with the use of the best linear predictor. We have

\begin{align} E\left[\left(X_{2}-E^{*}\left[X_{2} \mid X_{1}=x_{1}\right]\right)^{2}\right] & =E\left[\left\{\left(X_{2}-E\left[X_{2}\right]\right)-\beta\left(X_{1}-E\left[X_{1}\right]\right)\right\}^{2}\right] \tag{7.14} \\ & =\operatorname{Var}\left[X_{2}\right]+\beta^{2} \operatorname{Var}\left[X_{1}\right]-2 \beta \operatorname{Cov}\left[X_{2}, X_{1}\right] \\ & =\operatorname{Var}\left[X_{2}\right]-\frac{\operatorname{Cov}^{2}\left[X_{1}, X_{2}\right]}{\operatorname{Var}\left[X_{1}\right]} \\ & =\operatorname{Var}\left[X_{2}\right]\left\{1-\rho^{2}\left(X_{1}, X_{2}\right)\right\}. \end{align} 

From (7.14) one obtains the important conclusion that the closer the correlation between two random variables is to 1, the smaller the mean square error of prediction involved in predicting the value of one of the random variables from the value of the other. 

The Phenomenon of“Spurious” Correlation . Given three random variables \(U, V\) , and \(W\) , let \(X\) and \(Y\) be defined by

\[X=U+W, \quad Y=V+W \quad \text { or } \quad X=\frac{U}{W}, \quad Y=\frac{V}{W}, \tag{7.15}\] 

(or in some similar way) as functions of \(U, V\) , and \(W\) . The reader should be careful not to infer the existence of a correlation between \(U\) and \(V\) from the existence of a correlation between \(X\) and \(Y\) .

Theoretical Exercises

In the following exercises let \(X_{1}, X_{2}\) , and \(Y\) be jointly distributed random variables whose first and second moments are assumed known and whose variances are positive.

Exercises