The notions of independent and dependent events play a central role in probability theory. Certain relations, which recur again and again in probability problems, may be given a general formulation in terms of these notions. If the events \(A\) and \(B\) have the property that the conditional probability of \(B\) , given \(A\) , is equal to the unconditional probability of \(B\) , one intuitively feels that the event \(B\) is statistically independent of \(A\) , in the sense that the probability of \(B\) having occurred is not affected by the knowledge that \(A\) has occurred. We are thus led to the following formal definition.
Definition of an Event \(B\) Being Independent of an Event \(a\) Which Has Positive Probability. Let \(A\) and \(B\) be events defined on the same probability space \(S\) . Assume \(P[A]>0\) , so that \(P[B \mid A]\) is well defined.
The event \(B\) is said to be independent (or statistically independent) of the event \(A\) if the conditional probability of \(B\) , given \(A\) , is equal to the unconditional probability of \(B\) ; in symbols, \(B\) is independent of \(A\) if \[P[B \mid A]=P[B]. \tag{1.1}\]
Now suppose that both \(A\) and \(B\) have positive probability. Then both \(P[A \mid B]\) and \(P[B \mid A]\) are well defined, and from (4.6) of Chapter 2 it follows that \[P[A B]=P[B \mid A] P[A]=P[A \mid B] P[B]. \tag{1.2}\]
If \(B\) is independent of \(A\) , it then follows that \(A\) is independent of \(B\) , since from (1.1) and (1.2) it follows that \(P[A \mid B]=P[A]\) . It further follows from (1.1) and (1.2) that \[P[A B]=P[A] P[B]. \tag{1.3}\]
By means of (1.3) , a definition may be given of two events being independent, in which the two events play a symmetrical role.
Definition of Independent Events. Let \(A\) and \(B\) be events defined on the same probability space. The events \(A\) and \(B\) are said to be independent if (1.3) holds.
Example 1A . Consider the problem of drawing with replacement a sample of size 2 from an urn containing four white and two red balls. Let \(A\) denote the event that the first ball drawn is white and \(B\) , the event that the second ball drawn is white. By (2.5) , in Chapter 2, \(P[A B]=\left(\frac{4}{6}\right)^{2}\) , whereas \(P[A]=P[B]=\frac{4}{6}\) . In view of (1.3) , the events \(A\) and \(B\) are independent.
Two events that do not satisfy (1.3) are said to be dependent (although a more precise terminology would be nonindependent ). Clearly, to say that two events are dependent is not very informative, for two events, \(A\) and \(B\) , are dependent if and only if \(P[A B] \neq P[A] P[B]\) . However, it is possible to classify dependent events to a certain extent, and this is done later. (See section 5.)
It should be noted that two mutually exclusive events, \(A\) and \(B\) , are independent if and only if \(P[A] P[B]=0\) , which is so if and only if either \(A\) or \(B\) has probability zero.
Example 1B . Mutually exclusive events. Let a sample of size 2 be drawn from an urn containing six balls, of which four are white. Let \(C\) denote the event that exactly one of the balls drawn is white, and let \(D\) denote the event that both balls drawn are white. The events \(C\) and \(D\) are mutually exclusive and are not independent, whether the sample is drawn with or without replacement.
Example 1C . A paradox? Choose a summer day at random on which both the Dodgers and the Giants are playing baseball games. Let \(A\) be the event that the Dodgers win, and let \(B\) be the event that the Giants win. If the Dodgers and the Giants are not playing each other, then we may consider the events \(A\) and \(B\) as independent but not mutually exclusive. If the Giants and the Dodgers are playing each other, then we may consider the events \(A\) and \(B\) as mutually exclusive but not independent. To resolve this paradox, one need note only that the probability space on which the events \(A\) and \(B\) are defined is not the same in the two cases. (See example 2B.)
The notions of independent events and of conditional probability may be extended to more than two events. Suppose one has three events \(A, B\) , and \(C\) defined on a probability space. What are we to mean by the conditional probability of the event \(C\) , given that the events \(A\) and \(B\) have occurred, denoted by \(P[C \mid A, B]\) ? From the point of view of the frequency interpretation of probability, by \(P[C \mid A, B]\) we mean the fraction of occurrences of both \(A\) and \(B\) on which \(C\) also occurs. Consequently, we make the formal definition that \[P[C \mid A, B]=P[C \mid A B]=\frac{P[C A B]}{P[A B]}, \tag{1.4}\] if \(P[A B]>0\) ; \(P[C \mid A, B]\) is undefined if \(P[A B]=0\) .
Next, what do we mean by the statement that the event \(C\) is independent of the events \(A\) and \(B\) ? It would seem that we should mean that the conditional probability of \(C\) , given either \(A\) or \(B\) or the intersection \(A B\) , is equal to the unconditional probability of \(C\) . We therefore make the following formal definition.
The events \(A, B\) , and \(C\) , defined on the same probability space, are said to be independent (or statistically independent) if \[P[A B]=P[A] P[B], \quad P[A C]=P[A] P[C], \quad P[B C]=P[B] P[C] \tag{1.5}\] \[P[A B C]=P[A] P[B] P[C]. \tag{1.6}\]
If (1.5) and (1.6) hold, it then follows that (assuming that the events \(A\) , \(B, C, A B, A C, B C\) have positive probability, so that the conditional probabilities written below are well defined) \begin{align} & P[A \mid B, C]=P[A \mid B]=P[A \mid C]=P[A] \\ & P[B \mid A, C]=P[B \mid A]=P[B \mid C]=P[B] \tag{1.7} \\ & P[C \mid A, B]=P[C \mid A]=P[C \mid B]=P[C]. \end{align}
Conversely, if all the relations in (1.7) hold, then all the relations in (1.5) and (1.6) hold.
It is to be emphasized that (1.5) does not imply (1.6) , so that three events, \(A, B\) , and \(C\) , which are pairwise independent [in the sense that (1.5) holds], are not necessarily independent. To see this, consider the following example.
Example 1D . Pairwise independent events that are not independent. Let a ball be drawn from an urn containing four balls, numbered 1 to 4. Assume that \(S=\{1,2,3,4\}\) possesses equally likely descriptions. The events \(A=\{1,2\}, B=\{1,3\}\) , and \(C=\{1,4\}\) satisfy (1.5) but do not satisfy (1.6) . Indeed, \(P[C \mid A, B]=1 \neq \frac{1}{2}=P[C]=P[C \mid A]=P[C \mid B]\) . The reader may find it illuminating to explain in words why \(P[C \mid A, B]=\) 1.
Example 1E . The joint credibility of witnesses. Consider an automobile accident on a city street in which car I stops suddenly and is hit from behind by car II. Suppose that three persons, whom we call \(A^{\prime}, B^{\prime}\) , and \(C^{\prime}\) , witness the accident. Suppose the probability that each witness has correctly observed that car I stopped suddenly is estimated by having the witnesses observe a number of contrived incidents about which each is then questioned. Assume that it is found that \(A^{\prime}\) has probability 0.9 of stating that car I stopped suddenly, \(B^{\prime}\) has probability 0.8 of stating that car I stopped suddenly, and \(C^{\prime}\) has probability 0.7 of stating that car I stopped suddenly. Let \(A, B\) , and \(C\) denote, respectively, the events that persons \(A^{\prime}, B^{\prime}\) , and \(C^{\prime}\) will state that car I stopped suddenly. Assuming that \(A, B\) , and \(C\) are independent events, what is the probability that (i) \(A^{\prime}\) , \(B^{\prime}\) , and \(C^{\prime}\) will state that car I stopped suddenly, (ii) exactly two of them will state that car I stopped suddenly?
Solution
By independence, the probability \(P[A B C]\) that all three witnesses will state that car I stopped suddenly is given by \(P[A B C]=\) \(P[A] P[B] P[C]=(0.9)(0.8)(0.7)=0.504\) . It is subsequently shown that if \(A, B\) , and \(C\) are independent events then \(A, B\) , and \(C^{c}\) are independent events. Consequently, the probability that exactly two of the witnesses will state that car I stopped suddenly is given by \begin{align} P\left[A B C^{c} \cup A B^{\circ} C \cup A^{c} B C\right] &= P[A] P[B] P\left[C^{c}\right]+P[A] P\left[B^{c} \mid P[C]+P\left[A^{c}\right] P[B] P[C]\right. \\ & =(0.9)(0.8)(0.3)+(0.9)(0.2) (0.7)+(0.1)(0.8)(0.7) \\ & =0.398. \end{align}
The probability that at least two of the witnesses will state that car I stopped suddenly is \(0.504+0.398=0.902\) . It should be noted that the sample description space \(S\) on which the events \(A, B\) , and \(C\) are defined is the space of 3-tuples \(\left(z_{1}, z_{2}, z_{3}\right)\) in which \(z_{1}\) is equal to “yes” or “no,” depending on whether person \(A^{\prime}\) says that car I did or did not stop suddenly; components \(z_{2}\) and \(z_{3}\) are defined similarly with respect to persons \(B^{\prime}\) and \(C^{\prime}\) .
We next define the notions of independence and of conditional probability for \(n\) events \(A_{1}, A_{2}, \ldots, A_{n}\) .
We define the conditional probability of \(A_{n}\) , given that the events \(A_{1}\) , \(A_{2}, \ldots, A_{n-1}\) have occurred, denoted by \(P\left[A_{n} \mid A_{1}, A_{2}, \ldots, A_{n-1}\right]\) ; \begin{align} P\left[A_{n} \mid A_{1}, A_{2}, \ldots, A_{n-1}\right] & =P\left[A_{n} \mid A_{1} A_{2} \cdots A_{n-1}\right] \tag{1.8} \\ & =\frac{P\left[A_{1} A_{2} \cdots A_{n}\right]}{P\left[A_{1} A_{2} \cdots A_{n-1}\right]} \end{align} if \(P\left[A_{1} A_{2} \cdots A_{n-1}\right]>0\) .
We define the events \(A_{1}, A_{2}, \ldots, A_{n}\) as independent (or statistically independent) if for every choice of \(k\) integers \(i_{1}
Equation (1.9) implies that for any choice of integers \(i_{1}
We next consider families of independent events , for independent events never occur alone. Let \(\mathscr{A}\) and \(\mathscr{B}\) be two families of events; that is, \(\mathscr{A}\) and \(\mathscr{B}\) are sets whose members are events on some sample description space \(S\) .
Two families of events \(\mathscr{A}\) and \(\mathscr{B}\) are said to be independent if any two events \(A\) and \(B\) , selected from \(\mathscr{A}\) and \(\mathscr{B}\) , respectively, are independent. More generally, \(n\) families of events \(\left(\mathscr{A}_{1}, \mathscr{A}_{2}, \ldots, \mathscr{A}_{n}\right)\) are said to be independent if any set of \(n\) events \(A_{1}, A_{2}, \ldots, A_{n}\) (where \(A_{1}\) is selected from \(\mathscr{A}_{1}, A_{2}\) is selected from \(\mathscr{A}_{2}\) , and so on, until \(A_{n}\) is selected from \(\mathscr{A}_{n}\) ) is independent , in the sense that it satisfies the relation \[P\left[A_{1} A_{2} \cdots A_{n}\right]=P\left[A_{1}\right] P\left[A_{2}\right] \cdots P\left[A_{n}\right]. \tag{1.11}\]
As an illustration of the fact that independent events occur in families, let us consider two independent events, \(A\) and \(B\) , which are defined on a sample description space \(S\) . Define the families \(\mathscr{A}\) and \(\mathscr{B}\) by \[\mathscr{A}=\left\{A, A^{c}, S, \emptyset\right\}, \quad \mathscr{B}=\left\{B, B^{c}, S, \emptyset\right\}, \tag{1.12}\] so that \(\mathscr{A}\) consists of \(A\) , its complement \(A^{c}\) , the certain event \(S\) , and the impossible event \(\emptyset\) , and, similarly, \(\mathscr{B}\) consists of \(B, B^{c}, S\) , and \(\emptyset\) .
We now show that if the events \(A\) and \(B\) are independent then the families of events \(\mathscr{A}\) and \(\mathscr{B}\) defined by (1.12) are independent. In order to prove this assertion, we must verify the validity of (1.11) with \(n=2\) for each pair of events, one from each family, that may be chosen. Since each family has four members, there are sixteen such pairs. We verify (1.11) for only four of these pairs, namely \((A, B),\left(A, B^{c}\right),(A, S)\) , and \((A, \emptyset)\) , and leave to the reader the verification of (1.11) for the remaining twelve pairs. We have that \(A\) and \(B\) satisfy (1.11) by hypothesis. Next, we show that \(A\) and \(B^{c}\) satisfy (1.11) . By (5.2) of Chapter \(1, P\left[A B^{c}\right]=P[A]-P[A B]\) . Since, by hypothesis, \(P[A B]=P[A] P[B]\) , it follows that \[P\left[A B^{c}\right]=P[A](1-P[B])=P[A] P\left[B^{c}\right],\] for by (5.3) of Chapter 1 \(P\left[B^{c}\right]=1-P[B]\) . Next, \(A\) and \(S\) satisfy (1.11) , since \(A S=A\) and \(P[S]=1\) , so that \(P[A S]=P[A]=P[A] P[S]\) . Next, \(A\) and \(\emptyset\) satisfy (1.11) , since \(A \emptyset=\emptyset\) and \(P[\emptyset]=0\) , so that \(P[A \emptyset]=P[\emptyset]=\) \(P[A] P[\emptyset]=0\) .
More generally, by the same considerations, we may prove the following important theorem, which expresses (1.9) in a very concise form.
Theorem. Let \(A_{1}, A_{2}, \ldots, A_{n}\) be \(n\) events on a probability space. The events \(A_{1}, A_{2}, \ldots, A_{n}\) are independent if and only if the families of events \(\mathscr{A}_{1}=\left\{A_{1}, A_{1}^{c}, S, \emptyset\right\}, \quad \mathscr{A}_{2}=\left\{A_{2}, A_{2}^{c}, S, \emptyset\right\}, \ldots, \mathscr{A}_{n}=\left\{A_{n}, A_{n}^{c}, S, \emptyset\right\}\) are independent.
Theoretical Exercises
1.1 . Consider \(n\) independent events \(A_{1}, A_{2}, \ldots, A_{n}\) . Show that \[P\left[A_{1} \cup A_{2} \cup \cdots \cup A_{n}\right]=1-P\left[A_{1}^{c}\right] P\left[A_{2}^{c}\right] \cdots P\left[A_{n}^{c}\right].\] Consequently, obtain the probability that in 6 independent tosses of a fair die the number 3 will appear at least once. Answer : \(1-(5 / 6)^{6}\) .
1.2 . Let the events \(A_{1}, A_{2}, \ldots, A_{n}\) be independent and \(P\left[A_{i}\right]=p_{i}\) for \(i=1, \ldots, n\) . Let \(P_{0}\) be the probability that none of the events will occur. Show that \(P_{0}=\left(1-p_{1}\right)\left(1-p_{2}\right) \cdots\left(1-p_{n}\right)\) .
1.3 . Let the events \(A_{1}, A_{2}, \ldots, A_{n}\) be independent and have equal probability \(P\left[A_{i}\right]=p\) . Show that the probability that exactly \(k\) of the events will occur is (for \(k=0,1, \ldots, n\) ) \[\left(\begin{array}{l} n \tag{1.13} \\ k \end{array}\right) p^{k} q^{n-k}.\]
Hint : \(P\left[A_{1} \cdots A_{k} A_{k+1}^{c} \cdots A_{n}^{c}\right]=p^{k} q^{n-k}\) .
1.4 . The multiplicative rule for the probability of the intersection of \(n\) events \(A_{1}, A_{2}, \ldots, A_{n}\) . Show that, for \(n\) events for which \(P\left[A_{1} A_{2} \ldots A_{n-1}\right]>0\) , \[P\left[A_{1} A_{2} A_{3} \cdots A_{n}\right]= P\left[A_{1}\right] P\left[A_{2} \mid A_{1}\right] P\left[A_{3} \mid A_{1}, A_{2}\right] \cdots P\left[A_{n} \mid A_{1}, A_{2}, \ldots, A_{n-1}\right]\]
1.5 . Let \(A\) and \(B\) be independent events. In terms of \(P[A]\) and \(P[B]\) , express, for \(k=0,1,2\) , (i) \(P\) [exactly \(k\) of the events \(A\) and \(B\) will occur], (ii) \(P\) [at least \(k\) of the events \(A\) and \(B\) will occur], (iii) \(P\) [at most \(k\) of the events \(A\) and \(B\) will occur].
1.6 . Let \(A, B\) , and \(C\) be independent events. In terms of \(P[A], P[B]\) , and \(P[C]\) , express, for \(k=0,1,2,3\) , (i) \(P\) [exactly \(k\) of the events \(A, B, C\) will occur], (ii) \(P\) [at least \(k\) of the events \(A, B, C\) will occur], (iii) \(P\) [at most \(k\) of the events \(A, B, C\) will occur].
Exercises
1.1 . Let a sample of size 4 be drawn with replacement (without replacement) from an urn containing 6 balls, of which 4 are white. Let \(A\) denote the event that the ball drawn on the first draw is white, and let \(B\) denote the event that the ball drawn on the fourth draw is white. Are \(A\) and \(B\) independent? Prove your answers.
Answer
Yes, since \(P[A B]=\left(\frac{4}{6}\right)^{2}\) and \(P[A]=P[B]=\frac{4}{6}\)
(No, since \(P[A B]=\frac{2}{6}\) and \(P[A]=P[B]=\frac{4}{6}\) ).
1.2 . Let a sample of size 4 be drawn with replacement (without replacement) from an urn containing 6 balls, of which 4 are white. Let \(A\) denote the event that exactly 1 of the balls drawn on the first 2 draws is white. Let \(B\) be the event that the ball drawn on the fourth draw is white. Are \(A\) and \(B\) independent? Prove your answers.
1.3 . (Continuation of 1.2). Let \(A\) and \(B\) be as defined in exercise 1.2. Let \(C\) be the event that exactly 2 white balls are drawn in the 4 draws. Are \(A, B\) , and \(C\) independent? Are \(B\) and \(C\) independent? Prove your answers.
Answer
No.
1.4 . Consider example \(1 \mathrm{E}\) . Find the probability that (i) both \(A^{\prime}\) and \(B^{\prime}\) will state that car I stopped suddenly, (ii) neither \(A^{\prime}\) nor \(C^{\prime}\) will state that car I stopped suddenly, (iii) at least 1 of \(A^{\prime}, B^{\prime}\) , and \(C^{\prime}\) will state that car I stopped suddenly.
1.5 . A manufacturer of sports cars enters 3 drivers in a race. Let \(A_{1}\) be the event that driver 1 “shows” (that is, he is among the first 3 drivers in the race to cross the finish line), let \(A_{2}\) be the event that driver 2 shows, and let \(A_{3}\) be the event that driver 3 shows. Assume that the events \(A_{1}, A_{2}, A_{3}\) are independent and that \(P\left[A_{1}\right]=P\left[A_{2}\right]=P\left[A_{3}\right]=0.1\) . Compute the probability that (i) none of the drivers will show, (ii) at least 1 will show, (iii) at least 2 will show, (iv) all of them will show.
Answer
(i) 0.729; (ii) 0.271; (iii) 0.028; (iv) 0.001.
1.6 . Compute the probabilities asked for in exercise 1.5 under the assumption that \(P\left[A_{1}\right]=0.1, P\left[A_{2}\right]=0.2, P\left[A_{3}\right]=0.3\) .
1.7 . A manufacturer of sports cars enters \(n\) drivers in a race. For \(i=1, \ldots, n\) let \(A_{i}\) be the event that the ith driver shows (see exercise 1.5). Assume that the events \(A_{1}, \ldots, A_{n}\) are independent and have equal probability \(P\left[A_{i}\right]=p\) . Show that the probability that exactly \(k\) of the drivers will show is \(\left(\begin{array}{l}n \\ k\end{array}\right) p^{k} q^{n-k}\) for \(k=0,1, \ldots, n\) .
1.8 . Suppose you have to choose a team of 3 persons to enter a race. The rules of the race are that a team must consist of 3 people whose respective probabilities \(p_{1}, p_{2}, p_{3}\) of showing must add up to \(\frac{1}{2}\) ; that is, \(p_{1}+p_{2}+p_{3}=\frac{1}{2}\) . What probabilities of showing would you desire the members of your team to have in order to maximize the probability that at least 1 member of your team will show? (Assume independence.)
1.9 . Let \(A\) and \(B\) be 2 independent events such that the probability is \(\frac{1}{6}\) that they will occur simultaneously and \(\frac{1}{3}\) that neither of them will occur. Find \(P[A]\) and \(P[B]\) ; are \(P[A]\) and \(P[B]\) uniquely determined?
Answer
Possible values for \((P[A], P[B])\) are \(\left(\frac{1}{2}, \frac{1}{3}\right)\) and \(\left(\frac{1}{3}, \frac{1}{2}\right)\) .
1.10 . Let \(A\) and \(B\) be 2 independent events such that the probability is \(\frac{1}{6}\) that they will occur simultaneously and \(\frac{1}{3}\) that \(A\) will occur and \(B\) will not occur. Find \(P[A]\) and \(P[B]\) ; are \(P[A]\) and \(P[B]\) uniquely determined?