Expectations of Jointly Distributed Random Variables

Consider two jointly distributed random variables $X_1$ and $X_2$ . The expectation $E_{X_1,X_2}\left[g(x_1,x_2)\right]$ of a function $g(x_1,x_2)$ of two real variables is defined as follows:

If the random variables $X_1$ and $X_2$ are jointly continuous, with joint probability density function $f_{X_1,X_2}(x_1,x_2)$ , then

E_{X_{1}, X_{2}}\left[g\left(x_{1}, x_{2}\right)\right]=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g\left(x_{1}, x_{2}\right) f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right) d x_{1} d x_{2}. \tag{2.1} 

If the random variables $X_1$ and $X_2$ are jointly discrete, with joint probability mass function $p_{{\mathrm{X}}_1,{\mathrm{X}}_2}(x_1,x_2)$ , then

E_{X_{1}, X_{2}}\left[g\left(x_{1}, x_{2}\right)\right]= \displaystyle\sum_{\substack{\text { over all }\left(x_{1}, x_{2}\right) \text { such } \\ \text { that } p_{X_{1}}, x_{2}\left(x_{1}, x_{2}\right)>0}} g\left(x_{1}, x_{2}\right) p_{X_{1}, x_{2}}\left(x_{1}, x_{2}\right). \tag{2.2} 

If the random variables $X_1$ and $X_2$ have joint distribution function $F_{X_1,X_2}(x_1,x_2)$ , then

E_{X_{1}, X_{2}}\left[g\left(x_{1}, x_{2}\right)\right]=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g\left(x_{1}, x_{2}\right) d F_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right), \tag{2.3} 

where the two-dimensional Stieltjes integral may be defined in a manner similar to that in which the one-dimensional Stieltjes integral was defined in section 6 of Chapter 5.

On the other hand, $g(X_1,X_2)$ is a random variable, with expectation

\begin{align} E[g(X_{1}, X_{2})] =\begin{cases} \displaystyle\int_{-\infty}^{\infty} y , dF_{g(X_{1}, X_{2})}(y), \tag{2.4} \\[4mm] \displaystyle\int_{-\infty}^{\infty} y f_{g(X_{1}, X_{2})}(y) , dy, \\[4mm] \displaystyle\sum_{\substack{\text{over all points} \; y \\ \text{where} \; p_{g(X_{1}, X_{2})}(y) > 0}} y , p_{g(X_{1}, X_{2})}(y). \end{cases} \end{align} 

depending on whether the probability law of $g(X_1,X_2)$ is specified by its distribution function, probability density function, or probability mass function.

It is a basic fact of probability theory that for any jointly distributed random variables $X_1$ and $X_2$ and any Borel function $g(x_1,x_2)$ 

E\left[g\left(X_{1}, X_{2}\right)\right]=E_{X_{1}, X_{2}}\left[g\left(x_{1}, x_{2}\right)\right], \tag{2.5} 

in the sense that if either of the expectations in (2.5) exists then so does the other, and the two are equal. A rigorous proof of (2.5) is beyond the scope of this book.

In view of (2.5) we have two ways of computing the expectation of a function of jointly distributed random variables. Equation (2.5) generalizes (1.5). Similarly, (1.11) may also be generalized.

Let $X_1,X_2$ , and $Y$ be random variables such that $Y=g_1(X_1,X_2)$ for some Borel function $g_1(x_1,x_2)$ . Then for any Borel function $g(\cdot )$ E[g(Y)]=E\left[g\left(g_{1}\left(X_{1}, X_{2}\right)\right)\right]. \tag{2.6} 

The most important property possessed by the operation of expectation of a random variable is its linearity property : if $X_1$ and $X_2$ are jointly distributed random variables with finite expectations $E\left[X_1\right]$ and $E\left[X_2\right]$ , then the sum $X_1+X_2$ has a finite expectation given by E\left[X_{1}+X_{2}\right]=E\left[X_{1}\right]+E\left[X_{2}\right]. \tag{2.7} Let us sketch a proof of (2.7) in the case that $X_1$ and $X_2$ are jointly continuous. The reader may gain some idea of how (2.7) is proved in general by consulting the proof of (6.22) in Chapter 2.

From (2.5) it follows that

E[X_{1} + X_{2}] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (x_{1} + x_{2}) f_{X_{1}, X_{2}}(x_{1}, x_{2}) , dx_{1} , dx_{2}. \tag{$2.7^{\prime}$} 

Now

\begin{align} & \int_{-\infty}^{\infty} d x_{1} x_{1} \int_{-\infty}^{\infty} d x_{2} f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right)=\int_{-\infty}^{\infty} dx_{1}\; x_{1} f_{X_{1}}\left(x_{1}\right)=E\left[X_{1}\right] \\ & \int_{-\infty}^{\infty} d x_{2} x_{2} \int_{-\infty}^{\infty} d x_{1} f_{X_{1}, X_{2}}\left(x_{1}, x_{2}\right)=\int_{-\infty}^{\infty} d x_{2}\;x_{2} f_{X_{2}}\left(x_{2}\right)=E\left[X_{2}\right]. \tag{___MATH_BLOCK_200___} \end{align} 

The integral on the right-hand side of \left(2.7^{\prime}\right) is equal to the sum of the integrals on the left-hand sides of \left(2.7^{\prime\prime}\right) . The proof of (2.7) is now complete.

The moments and moment-generating function of jointly distributed random variables are defined by a direct generalization of the definitions given for a single random variable. For any two nonnegative integers $n_1$ and $n_2$ we define \alpha_{n_{1}, n_{2}}=E\left[X_{1}^{n_{1}} X_{2}^{n_{2}}\right] \tag{2.8} as a moment of the jointly distributed random variables $X_1$ and $X_2$ . The sum $n_1+n_2$ is called the order of the moment. For the moments of orders 1 and 2 we have the following names; ${\alpha }_{1,0}$ and ${\alpha }_{0,1}$ are, respectively, the means of $X_1$ and $X_2$ , whereas ${\alpha }_{2,0}$ and ${\alpha }_{0,2}$ are, respectively, the mean squares of $X_1$ and $X_2$ . The moment ${\alpha }_{11}=E\left[X_1{X}_2\right]$ is called the product moment .

We next define the central moments of the random variables $X_1$ and $X_2$ . For any two nonnegative integers, $n_1$ and $n_2$ , we define \mu_{n_{1}, n_{2}}=E\left[\left(X_{1}-E\left[X_{1}\right]\right)^{n_{1}}\left(X_{2}-E\left[X_{2}\right]\right)^{n_{2}}\right] \tag{$2.8^{\prime}$} as a central moment of order $n_1+n_2$ . We are again particularly interested in the central moments of orders 1 and 2. The central moments ${\mu }_{1,0}$ and ${\mu }_{0,1}$ of order 1 both vanish, whereas ${\mu }_{2,0}$ and ${\mu }_{0,2}$ are, respectively, the variances of $X_1$ and $X_2$ . The central moment ${\mu }_{1,1}$ is called the covariance of the random variables $X_1$ and $X_2$ and is written $\mathrm{Cov}[X_1,X_2]$ ; in symbols, \operatorname{Cov}\left[X_{1}, X_{2}\right]=\mu_{1,1}=E\left[\left(X_{1}-E\left[X_{1}\right]\right)\left(X_{2}-E\left[X_{2}\right]\right)\right]. \tag{2.9} We leave it to the reader to prove that the covariance is equal to the product moment, minus the product of the means ; in symbols, \operatorname{Cov}\left[X_{1}, X_{2}\right]=E\left[X_{1} X_{2}\right]-E\left[X_{1}\right] E\left[X_{2}\right]. \tag{2.10} 

The covariance derives its importance from the role it plays in the basic formula for the variance of the sum of two random variables : \operatorname{Var}\left[X_{1}+X_{2}\right]=\operatorname{Var}\left[X_{1}\right]+\operatorname{Var}\left[X_{2}\right]+2 \operatorname{Cov}\left[X_{1}, X_{2}\right] \tag{2.11} 

To prove (2.11), we write \begin{align} \operatorname{Var}\left[X_{1}+X_{2}\right]= & E\left[\left(X_{1}+X_{2}\right)^{2}\right]-E^{2}\left[X_{1}+X_{2}\right] \\[2mm] = & E\left[X_{1}^{2}\right]-E^{2}\left[X_{1}\right]+E\left[X_{2}^{2}\right]-E^{2}\left[X_{2}\right] \\[2mm] & \quad\quad\quad +2\left(E\left[X_{1} X_{2}\right]-E\left[X_{1}\right] E\left[X_{2}\right]\right), \end{align} from which (2.11) follows by (1.8) and (2.10).

The joint moment-generating function is defined for any two real numbers, $t_1$ and $t_2$ , by $${\psi }_{X_1,X_2}(t_1,t_2)=E\left[e^{(t_1{X}_1+t_2{X}_2)}\right].$$ 

The moments can be read off from the power-series expansion of the moment-generating function, since formally \psi_{X_{1}, X_{2}}\left(t_{1}, t_{2}\right)=\sum_{n_{1}=0}^{\infty} \sum_{n_{2}=0}^{\infty} \frac{t_{1}^{n_{1}}}{n_{1} !} \frac{t_{2}^{n_{2}}}{n_{2} !} E\left[X_{1}^{n_{1}} X_{2}^{n_{2}}\right]. \tag{2.12} 

In particular, the means, variances, and covariance of $X_1$ and $X_2$ may be expressed in terms of the derivatives of the moment-generating function: \begin{align} E[X_1] &= \frac{\partial}{\partial t_1} \psi_{X_1, X_2}(0, 0), & E[X_2] &= \frac{\partial}{\partial t_2} \psi_{X_1, X_2}(0, 0), \tag{2.13} \\[4mm] E[X_1^2] &= \frac{\partial^2}{\partial t_1^2} \psi_{X_1, X_2}(0, 0), & E[X_2^2] &= \frac{\partial^2}{\partial t_2^2} \psi_{X_1, X_2}(0, 0). \tag{2.14} \\[4mm] E[X_1 X_2] &= \frac{\partial^2}{\partial t_1 \partial t_2} \psi_{X_1, X_2}(0, 0). \tag{2.15} \\[4mm] \text{Var}[X_1] &= \frac{\partial^2}{\partial t_1^2} \psi_{X_1 - m_1, X_2 - m_2}(0, 0), & \text{Var}[X_2] &= \frac{\partial^2}{\partial t_2^2} \psi_{X_1 - m_1, X_2 - m_2}(0, 0). \tag{2.16} \\[4mm] \text{Cov}[X_1, X_2] &= \frac{\partial^2}{\partial t_1 \partial t_2} \psi_{X_1 - m_1, X_2 - m_2}(0, 0). \tag{2.17} \end{align} in which $m_1=E\left[X_1\right],m_2=E\left[X_2\right]$ .

The foregoing notions may be extended to the case of $n$ jointly distributed random variables, $X_1,X_2,\dots ,X_n$ . For any Borel function $g(x_1,x_2,\dots ,x_n)$ of $n$ real variables, the expectation $E\left[g(X_1,X_2,\dots ,X_n)\right]$ of the random variable $g(X_1,X_2,\dots ,X_n)$ may be expressed in terms of the joint probability law of $X_1,X_2,\dots ,X_n$ .

If $X_1,X_2,\dots ,X_n$ are jointly continuous, with a joint probability density function $f_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)$ , it may be shown that

\begin{align} E\left[g\left(X_{1}, X_{2}, \ldots, X_{n}\right)\right] & =\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} g\left(x_{1}, x_{2}, \ldots, x_{n}\right) \tag{2.24} \\[3mm] & \times f_{X_{1}, X_{2}, \ldots, X_{n}}\left(x_{1}, x_{2}, \ldots, x_{n}\right) d x_{1} d x_{2} \cdots d x_{n}. \end{align} 

If $X_1,X_2,\dots ,X_n$ are jointly discrete, with a joint probability mass function $p_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)$ , it may be shown that

\begin{align} & \text { (2.25) } \quad E\left[g\left(X_1, X_2, \ldots, X_n\right)\right]= &\sum_{\substack{\text { over all }\left(x_1, x_2, \ldots, x_n\right) \text { such that } \\ p_{X_1, X_2, \ldots, X_n}\left(x_1, x_2, \ldots, x_n\right)>0}} g\left(x_1, x_2, \ldots, x_2, \ldots, x_n\right) p_{X_1, X_2, \ldots, X_n}\left(x_1, x_2, \ldots, x_n\right) \end{align}

The joint moment-generating function of $n$ jointly distributed random variables is defined by

\psi_{X_{1}, X_{2}, \ldots, X_{n}}\left(t_{1}, t_{2}, \ldots, t_{n}\right)=E\left[e^{\left(t_{1} X_{1}+t_{2} X_{2}+\cdots+t_{n} X_{n}\right)}\right]. \tag{2.26} 

It may also be proved that if $X_1,X_2,\dots ,X_n$ and $Y$ are random variables, such that $Y=g_1(X_1,X_2,\dots ,X_n)$ for some Borel function $g_1(x_1,x_2,\dots ,x_n)$ of $n$ real variables, then for any Borel function $g(\cdot )$ of one real variable

E[g(Y)]=E\left[g\left(g_{1}\left(X_{1}, X_{2}, \ldots, X_{n}\right)\right)\right]. \tag{2.27} 

Theoretical Exercises

Exercises