不相关与独立随机变量

The notion of independence of two random variables, $X_{1}$ and $X_{2}$ , is defined in section 6 of Chapter 7 . In this section we show how the notion of independence may be formulated in terms of expectations. At the same time, by a modification of the condition for independence of random variables, we are led to the notion of uncorrelated random variables.

We begin by considering the properties of expectations of products of random variables. Let $X_{1}$ and $X_{2}$ be jointly distributed random variables. By the linearity properties of the operation of taking expectations, it follows that for any two functions, $g_{1} (., .)$ and $g_{2} (., .)$ if the expectations on the right side of (3.1) exist. However, it is not true that a similar relation holds for products; namely, it is not true in general that $E [g_{1} (X_{1}, X_{2}) g_{2} (X_{1}, X_{2})] = E [g_{1} (X_{1}, X_{2})] E [g_{2} (X_{1}, X_{2})]$ . There is one special circumstance in which a relation similar to the foregoing is valid, namely, if the random variables $X_{1}$ and $X_{2}$ are independent and if the functions are functions of one variable only. More precisely, we have the following theorem:

Theorem 3A: If the random variables $X_{1}$ and $X_{2}$ are independent, then for any two Borel functions $g_{1} (\cdot)$ and $g_{2} (\cdot)$ of one real variable the product moment of $g_{1} (X_{1})$ and $g_{2} (X_{2})$ is equal to the product of their means; in symbols,

if the expectations on the right side of (3.2) exist.

To prove equation (3.2), it suffices to prove it in the form

since independence of $X_{1}$ and $X_{2}$ implies independence of $g (X_{1})$ and $g (X_{2})$ . We write out the proof of (3.3) only for the case of jointly continuous random variables. We have

Now suppose that we modify (3.2) and ask only that it hold for the functions $g_{1} (x) = x$ and $g_{2} (x) = x$ , so that

For reasons that are explained after (3.7), two random variables, $X_{1}$ and $X_{2}$ , which satisfy (3.4), are said to be uncorrelated. From (2.10) it follows that $X_{1}$ and $X_{2}$ satisfy (3.4) and therefore are uncorrelated if and only if

For uncorrelated random variables the formula given by (2.11) for the variance of the sum of two random variables becomes particularly elegant; the variance of the sum of two uncorrelated random variables is equal to the sum of their variances. Indeed,

if and only if $X_{1}$ and $X_{2}$ are uncorrelated.

Two random variables that are independent are uncorrelated, for if (3.2) holds then, a fortiori, (3.4) holds. The converse is not true in general; an example of two uncorrelated random variables that are not independent is given in theoretical exercise 3.2. In the important special case in which $X_{1}$ and $X_{2}$ are jointly normally distributed, it follows that they are independent if they are uncorrelated (see theoretical exercise 3.3).

The correlation coefficient $ρ (X_{1}, X_{2})$ of two jointly distributed random variables with finite positive variances is defined by

In view of (3.7) and (3.5), two random variables $X_{1}$ and $X_{2}$ are uncorrelated if and only if their correlation coefficient is zero.

The correlation coefficient provides a measure of how good a prediction of the value of one of the random variables can be formed on the basis of an observed value of the other. It is subsequently shown that

Further $ρ (X_{1}, X_{2}) = 1$ if and only if

and $ρ (X_{1}, X_{2}) = - 1$ if and only if

From (3.9) and (3.10) it follows that if the correlation coefficient equals 1 or -1 then there is perfect prediction; to a given value of one of the random variables there is one and only one value that the other random variable can assume. What is even more striking is that $ρ (X_{1}, X_{2}) = \pm 1$ if and only if $X_{1}$ and $X_{2}$ are linearly dependent.

That (3.8), (3.9), and (3.10) hold follows from the following important theorem.

Theorem 3B . For any two jointly distributed random variables, $X_{1}$ and $X_{2}$ , with finite second moments Further, equality holds in (3.11), that is, $E^{2} [X_{1} X_{2}] = E [X_{1}^{2}] E [X_{2}^{2}]$ if and only if, for some constant $t, X_{2} = t X_{1}$ , which means that the probability mass distributed over the $(x_{1}, x_{2})$ -plane by the joint probability law of the random variables is situated on the line $x_{2} = t x_{1}$ .

Applied to the random variables $X_{1} - E [X_{1}]$ and $X_{2} - E [X_{2}]$ , (3.11) states that

We prove (3.11) as follows. Define, for any real number $t, h (t) =$ $E [{(t X_{1} - X_{2})}^{2}] = t^{2} E [X_{1}^{2}] - 2 t E [X_{1} X_{2}] + E [X_{2}^{2}]$ . Clearly $h (t) \geq 0$ for all $t$ . Consequently, the quadratic equation $h (t) = 0$ has either no solutions or one solution. The equation $h (t) = 0$ has no solutions if and only if $E^{2} [X_{1} X_{2}] - E [X_{1}^{2}] E [X_{2}^{2}] < 0$ . It has exactly one solution if and only if $E^{2} [X_{1} X_{2}] = E [X_{1}^{2}] E [X_{2}^{2}]$ . From these facts one may immediately infer (3.11) and the sentence following it.

The inequalities given by (3.11) and (3.12) are usually referred to as Schwarz’s inequality or Cauchy’s inequality.

Conditions for Independence . It is important to note the difference between two random variables being independent and being uncorrelated. They are uncorrelated if and only if (3.4) holds. It may be shown that they are independent if and only if (3.2) holds for all functions $g_{1} (\cdot)$ and $g_{2} (\cdot)$ , for which the expectations in (3.2) exist. More generally, theorem $3 c$ can be proved.

Theorem 3c. Two jointly distributed random variables $X_{1}$ and $X_{2}$ are independent if and only if each of the following equivalent statements is true:

(i) Criterion in terms of probability functions. For any Borel sets $B_{1}$ and $B_{2}$ of real numbers, $P [X_{1}$ is in $B_{1}, X_{2}$ is in $B_{2}] = P [X_{1}$ is in $B_{1}] P [X_{2}$ is in $B_{2}]$ .

(ii) Criterion in terms of distribution functions. For any two real numbers, $x_{1}$ and $x_{2}, F_{X_{1}, X_{2}} (x_{1}, x_{2}) = F_{X_{1}} (x_{1}) F_{X_{2}} (x_{2})$ .

(iii) Criterion in terms of expectations. For any two Borel functions, $g_{1} (\cdot)$ and $g_{2} (\cdot), E [g_{1} (X_{1}) g_{2} (X_{2})] = E [g_{1} (X_{1})] E [g_{2} (X_{2})]$ if the expectations involved exist.

(iv) Criterion in terms of moment-generating functions (if they exist). For any two real numbers, $t_{1}$ and $t_{2}$ ,

Theoretical Exercises

3.1. The standard deviation has the properties of the operation of taking the absolute value of a number : show first that for any 2 real numbers, $x$ and $y, | x + y | \leq | x | + | y |, | | x | - | y | | \leq | x - y |$ .

Hint : Square both sides of the equations. Show next that for any 2 random variables, $X$ and $Y$ ,

Give an example to prove that the variance does not satisfy similar relationships.

3.2. Show that independent random variables are uncorrelated. Give an example to show that the converse is false.

Hint : Let $X = \sin 2 π U$ , $Y = \cos 2 π U$ , in which $U$ is uniformly distributed over the interval 0 to 1.

3.3. Prove that if $X_{1}$ and $X_{2}$ are jointly normally distributed random variables whose correlation coefficient vanishes then $X_{1}$ and $X_{2}$ are independent. Hint : Use example 2A .

3.4 . Let $α$ and $β$ be the values of $a$ and $b$ which minimize

$f (a, b) = E {| X_{2} - a - b X_{1} |}^{2} .$

Express $α, β$ , and $f (α, β)$ in terms of $ρ (X_{1}, X_{2})$ . The random variable $α + β X_{1}$ is called the best linear predictor of $X_{2}$ , given $X_{1}$ [see Section 7, in particular, (7.13) and (7.14)].

3.5. Prove that (3.9) and (3.10) hold under the conditions stated.

3.6. Let $X_{1}$ and $X_{2}$ be jointly distributed random variables possessing finite second moments. State conditions under which it is possible to find 2 uncorrelated random variables, $Y_{1}$ and $Y_{2}$ , which are linear combinations of $X_{1}$ and $X_{2}$ (that is, $Y_{1} = a_{11} X_{1} + a_{12} X_{2}$ and $Y_{2} = a_{21} X_{1} + a_{22} X_{2}$ for some constants $a_{11}, a_{12}, a_{21}, a_{22}$ and $Cov [Y_{1}, Y_{2}] = 0$ ).

3.7. Let $X$ and $Y$ be jointly normally distributed with mean 0, arbitrary variances, and correlation $ρ$ . Show that Hint : Consult H. Cramér, Mathematical Methods of Statistics , Princeton University Press, 1946, p. 290.

3.8. Suppose that $n$ tickets bear arbitrary numbers $x_{1}, x_{2}, \dots, x_{n}$ , which are not all the same. Suppose further that 2 of the tickets are selected at random without replacement. Show that the correlation coefficient $ρ$ between the numbers appearing on the 2 tickets is equal to $(- 1) / (n - 1)$ .

3.9. In an urn containing $N$ balls, a proportion $p$ is white and $q = 1 - p$ are black. A ball is drawn and its color noted. The ball drawn is then replaced, and $N r$ balls are added of the same color as the ball drawn. The process is repeated until $n$ balls have been drawn. For $j = 1, 2, \dots, n$ let $X_{j}$ be equal to 1 or 0, depending on whether the ball drawn on the $j$ th draw is white or black. Show that the correlation coefficient between $X_{i}$ and $X_{j}$ is equal to $r / (1 + r)$ . Note that the case $r = - 1 / N$ corresponds to sampling without replacement, and $r = 0$ corresponds to sampling with replacement.

Exercises

3.1. Consider 2 events $A$ and $B$ such that $P [A] = \frac{1}{4}, P [B ∣ A] = \frac{1}{2}, P [A ∣ B] == \frac{1}{4}$ . Define random variables $X$ and $Y : X = 1$ or 0, depending on whether the event $A$ has or has not occurred, and $Y = 1$ or 0, depending on whether the event $B$ has or has not occurred. Find $E [X], E [Y], Var [X], Var [Y]$ , $ρ (X, Y)$ . Are $X$ and $Y$ independent?

Answer

$E [X] = \frac{1}{4}, E [Y] = \frac{1}{2}, Var [X] = \frac{3}{16}, Var [Y] = \frac{1}{4}, ρ [X, Y] = 0; X$ and $Y$ are independent.

3.2. 考虑一个从装有4个编号为1至4的球的瓮中，进行有放回（无放回）抽取的容量为2的样本。设 $X_{1}$ 为样本中抽得数字的最小值， $X_{2}$ 为最大值。求 $ρ (X_{1}, X_{2})$ 。

3.3. 两枚均匀硬币，每枚的两面分别标有数字1和2，独立地抛掷。设 $X$ 表示所得两个数字之和， $Y$ 表示所得数字中的最大值。求 $X$ 与 $Y$ 之间的相关系数。

答案

$\sqrt{2 / 3}$ 。

3.4. 设 $U, V$ 和 $W$ 为具有相等方差且互不相关的随机变量。令 $X = U + V, Y = U + W$ 。求 $X$ 与 $Y$ 之间的相关系数。

3.5. 设 $X_{1}$ 和 $X_{2}$ 为互不相关的随机变量。用 $X_{1}$ 和 $X_{2}$ 的方差表示随机变量 $Y_{1} = X_{1} + X_{2}$ 与 $Y_{2} = X_{1} - X_{2}$ 之间的相关系数 $ρ (Y_{1}, Y_{2})$ 。

答案

$(σ_{1}^{2} - σ_{2}^{2}) / (σ_{3}^{2} + σ_{2}^{2})$ 。

3.6. 设 $X_{1}$ 和 $X_{2}$ 为互不相关的正态分布随机变量。求随机变量 $Y_{1} = X_{1}^{2}$ 与 $Y_{2} = X_{2}^{2}$ 之间的相关系数 $ρ (Y_{1}, Y_{2})$ 。

3.7. 考虑其联合矩母函数在练习2.6 中给出的随机变量。求 $ρ (X_{1}, X_{2})$ 。

答案

$4 a - 1$ 。

3.8. 考虑其联合矩母函数在练习2.7 中给出的随机变量。求 $ρ (X_{1}, X_{2})$ 。

3.9. 考虑其联合矩母函数在练习2.8 中给出的随机变量。求 $ρ (X_{1}, X_{2})$ 。

答案

$e^{- (a_{2} - a_{1})}$ 。

3.10. 考虑其联合矩母函数在练习2.9 中给出的随机变量。求 $ρ (X_{1}, X_{2})$ 。