Consider the probability function \(P[\cdot]\) of a numerical-valued random phenomenon. The question arises concerning the convenient ways of stating the function without having actually to state the value of \(P[E]\) for every set of real numbers \(E\) . In general, to state the function \(P[\cdot]\) , as with any function, one has to enumerate all the members of the domain of the function \(P[\cdot]\) , and for each of these members of the domain one states the value of the function. In special circumstances (which fortunately cover most of the cases encountered in practice) more convenient methods are available.

For many probability functions there exists a function \(f(\cdot)\) , defined for all real numbers \(x\) , from which \(P[E]\) can, for any event \(E\) , be obtained by integration: \[P[E]=\int_{E} f(x) dx. \tag{2.1}\] 

Given a probability function \(P[\cdot]\) , which may be represented in the form of (2.1) in terms of some function \(f(\cdot)\) , we call the function \(f(\cdot)\) the probability density function of the probability function \(P[\cdot]\) , and we say that the probability function \(P[\cdot]\) is specified by the probability density function \(f(\cdot)\) .

A function \(f(\cdot)\) must have certain properties in order to be a probability density function. To begin with, it must be sufficiently well behaved as a function so that the integral ¹ in (2.1) is well defined. Next, letting \(E=R\) in (2.1) , \[1=P[R]=\int_{R} f(x) d x=\int_{-\infty}^{\infty} f(x) dx. \tag{2.2}\] 

It is necessary that \(f(\cdot)\) satisfy (2.2) ; in words, the integral of \(f(\cdot)\) from \(-\infty\) to \(\infty\) must be equal to 1.

Some typical probability density functions are illustrated in Fig.2A .

Example 2B . Computing probabilities from a probability density function . Let us consider again the numerical-valued random phenomenon, discussed in example 1A , that consists in observing the time one has to wait for a bus at a certain bus stop. Let us assume that the probability function \(P[\cdot]\) of this phenomenon may be expressed by (2.1) in terms of the function \(f(\cdot)\) , whose graph is sketched in Fig. 2B . An algebraic formula for \(f(\cdot)\) can be written as follows: \begin{align} f(x) =\begin{cases} 0, & \text { for } x<0 \tag{2.6} \\[2mm] \left(\frac{1}{9}\right)(x+1), & \text { for } 0 \leq x < 1 \\[2mm] \left(\frac{4}{9}\right)\left(x-\left(\frac{1}{2}\right)\right), & \text { for } 1 \leq x<\left(\frac{3}{2}\right) \\[2mm] \left(\frac{4}{9}\right)\left(\left(\frac{5}{2}\right)-x\right), & \text { for }\left(\frac{3}{2}\right) \leq x<2 \\[2mm] \left(\frac{1}{9}\right)(4-x), & \text { for } 2 \leq x<3 \\[2mm] \left(\frac{1}{9}\right), & \text { for } 3 \leq x<6 \\[2mm] 0 & \text { for } 6 \leq x \end{cases} \end{align} 

From (2.1) it follows that if \(A=\{x: 0 \leq x \leq 2\}\) and \(B=\{x: 1 \leq x \leq 3\}\) then \[\begin{gather} P[A]=\int_{0}^{2} f(x) d x=\frac{1}{2}, \quad P[B]=\int_{1}^{3} f(x) d x=\frac{1}{2}, \\ P[A B]=\int_{1}^{2} f(x) d x=\frac{1}{3}, \end{gather}\] which agree with the values assumed in example 1A .

For many probability functions there exists a function \(p(\cdot)\) , defined for all real numbers \(x\) , but with value \(p(x)\) equal to 0 for all \(x\) except for a finite or countably infinite set of values of \(x\) at which \(p(x)\) is positive, such that from \(p(\cdot)\) the value of \(P[E]\) can be obtained for any event \(E\) by summation: \[P[E]=\sum_{\substack{\text { over all } \\ \text { points } x \text { in } E \\ \text { such that } p(x)>0}} p(x) \tag{2.7}\] In order that the sum in (2.7) may be meaningful, it suffices to impose the condition [letting \(E=R\) in (2.7) ] that \[1=\sum_{\substack{\text { over all } \\ \text { points } x \text { in } \\ \text { such that } p(x)>0}} p(x) \tag{2.8}\] 

Given a probability function \(P[\cdot]\) , which may be represented in the form (2.7) , we call the function \(p(\cdot)\) the probability mass function of the probability function \(P[\cdot]\) , and we say that the probability function \(P[\cdot]\) is specified by the probability mass function \(p(\cdot)\) .

The terminology of “density function” and “mass function” comes from the following physical representation of the probability function \(P[\cdot]\) of a numerical-valued random phenomenon. We imagine that a unit mass of some substance is distributed over the real line in such a way that the amount of mass over any set \(B\) of real numbers is equal to \(P[B]\) . The distribution of substance possesses a density, to be denoted by \(f(x)\) , at the point \(x\) , if for any interval containing the point \(x\) of length \(h\) (where \(h\) is a sufficiently small number) the mass of substance attached to the interval is equal to \(h f(x)\) . The distribution of substance possesses a mass, to be denoted by \(p(x)\) , at the point \(x\) , if there is a positive amount \(p(x)\) of substance concentrated at the point.

We shall see in section 3 that a probability function \(P[\cdot]\) always possesses a probability density function and a probability mass function. Consequently, in order for a probability function to be specified by either its probability density function or its probability mass function, it is necessary (and, from a practical point of view, sufficient) that one of these functions vanish identically.

Exercises

Verify that each of the functions \(f(\cdot)\) , given in exercises \(2.1-2.5\) , is a probability density function (by showing that it satisfies (2.1) and (2.3) ) and sketch its graph. ³ Hint : use freely the facts developed in the appendix to this section.

Show that each of the functions \(p(\cdot)\) given in exercises 2.6 and 2.7 is a probability mass function [by showing that it satisfies (2.8) ], and sketch its graph. 
Hint : use freely the facts developed in the appendix to this section.

Appendix: The Evaluation of Integrals and Sums

If (2.1) and (2.7) are to be useful expressions for evaluating the probability of an event, then techniques must be available for evaluating sums and integrals. The purpose of this appendix is to state some of the notions and formulas with which the student should become familiar and to collect some important formulas that the reader should learn to use, even if he lacks the mathematical background to justify them.

To begin with, let us note the following principle. If a function is defined by different analytic expressions over various regions, then to evaluate an integral whose integrand is this function one must express the integral as a sum of integrals corresponding to the different regions of definition of the function . For example, consider the probability density function \(f(\cdot)\) defined by \begin{align} f(x) = \begin{cases} x, & \text{for } 0 < x < 1 \\[2mm] 2 - x, & \text{for } 1 < x < 2 \\[2mm] 0, & \text{elsewhere.} \end{cases} \end{align} To prove that \(f(\cdot)\) is a probability density function, we need to verify that (2.2) and (2.3) are satisfied. Clearly, (2.3) holds. Next, \begin{align} \int_{-\infty}^{\infty} f(x) d x & =\int_{0}^{2} f(x) d x+\int_{-\infty}^{0} f(x) d x+\int_{2}^{\infty} f(x) d x \\ & =\int_{0}^{1} f(x) d x+\int_{1}^{2} f(x) d x+0 \\ & =\left.\frac{x^{2}}{2}\right|_{0} ^{1}+\left.\left(2 x-\frac{x^{2}}{2}\right)\right|_{1} ^{2}=\frac{1}{2}+\left(2-\frac{3}{2}\right)=1, \end{align} and (2.2) has been shown to hold. It might be noted that the function \(f(\cdot)\) in (2.10) can be written somewhat more concisely in terms of the absolute value notation: \begin{align} f(x) = \begin{cases} 1 - |1 - x|, & \text{for } 0 \leq x \leq 2 \\[2mm] 0, & \text{otherwise} \end{cases} \end{align} Next, in order to check his command of the basic techniques of integration, the reader should verify that the following formulas hold: \[\int \frac{e^{x}}{\left(1+e^{x}\right)^{2}} d x=\frac{-1}{1+e^{x}}, \quad \int \frac{e^{x}}{1+e^{2 x}} d x=\tan ^{-1} e^{x}=\arctan e^{x},\] \[\int e^{-x-e^{-x}} d x=\int e^{-e^{-x}} e^{-x} d x=e^{-e^{-x}}. \tag{2.12}\] 

An important integration formula, obtained by integration by parts, is the following, for any real number \(t\) for which the integrals make sense: \[\int x^{t-1} e^{-x} \cdot d x=-x^{t-1} e^{-x}+(t-1) \int x^{t-2} e^{-x} d x \tag{2.13}\] Thus, for \(t=2\) we obtain \[\int x e^{-x} d x=-x e^{-x}+\int e^{-x} d x=-e^{-x}(x+1). \tag{2.14}\] 

We next consider the Gamma function \(\Gamma(\cdot)\) , which plays an important role in probability theory. It is defined for every \(t>0\) by \[\Gamma(t)=\int_{0}^{\infty} x^{t-1} e^{-x} dx. \tag{2.15}\] 

The Gamma function is a generalization of the factorial function in the following sense. From (2.13) it follows that \[\Gamma(t)=(t-1) \Gamma(t-1) . \tag{2.16}\] Therefore, for any integer \(r, 0 \leq r, \[\Gamma(t+1)=t \Gamma(t)=t(t-1) \cdots(t-r) \Gamma(t-r) . \tag{2.17}\] Since, clearly, \(\Gamma(1)=1\) , it follows that for any integer \(n \geq 0\) \[\Gamma(n+1)=n ! \tag{2.18}\] 

Next, it may be shown that for any integer \(n>0\) \[\Gamma\left(n+\frac{1}{2}\right)=\frac{1 \cdot 3 \cdot 5 \cdots(2 n-1)}{2^{n}} \sqrt{\pi}, \tag{2.19}\] which may be written for any even integer \(n\) \[\Gamma\left(\frac{n+1}{2}\right)=\frac{1 \cdot 3 \cdot 5 \cdots(n-1)}{2^{n / 2}} \sqrt{\pi}, \tag{2.20}\] since \[\Gamma\left(\frac{1}{2}\right)=\sqrt{\pi} . \tag{2.21}\] 

We prove (2.21) by showing that \(\Gamma\left(\frac{1}{2}\right)\) is equal to another integral of whose value we have need. In (2.15) , make the change of variable \(x=\frac{1}{2} y^{2}\) , and let \(t=(n+1) / 2\) . Then, for any integer, \(n=0,1, \ldots\) , we have the formula \[\Gamma\left(\frac{n+1}{2}\right)=\frac{1}{2^{(n-1) / 2}} \int_{0}^{\infty} y^{n} e^{-1 / 2 y^{2}} dy. \tag{2.22}\] 

In view of (2.22) , to establish (2.21) we need only show that \[\Gamma\left(\frac{1}{2}\right)=\sqrt{2} \int_{0}^{\infty} e^{-1 / 2 y^{2}} d y=\frac{1}{\sqrt{2}} \int_{-\infty}^{\infty} e^{-1 / 2 y^{2}} d y=\sqrt{\pi}. \tag{2.23}\] 

We prove (2.23) by proving the following basic formula; for any \(u>0\) \[\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{-\frac{1}{2} u y^{2}} d y=\frac{1}{\sqrt{u}}. \tag{2.24}\] 

Equation (2.24) may be derived as follows. Let \(I\) be the value of the integral in (2.24) . Then \(I^{2}\) is a product of two single integrals. By the theorem for the evaluation of double integrals, it then follows that \[I^{2}=\frac{1}{2 \pi} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \exp \left[-\frac{1}{2} u\left(x^{2}+y^{2}\right)\right] dx dy. \tag{2.25}\] 

We now evaluate the double integral in (2.25) by means of a change of variables to polar coordinates. Then \[I^{2}=\frac{1}{2 \pi} \int_{0}^{2 \pi} \int_{0}^{\infty} e^{-\frac{1}{2} u r^{2}} r d r d \theta=\int_{0}^{\infty} e^{-\frac{1}{2} u r^{2}} r d r=\frac{1}{u},\] so that \(I=1 / \sqrt{u}\) , which proves (2.24) .

For large values of \(t\) there is an important asymptotic formula for the Gamma function, which is known as Stirling’s formula . Taking \(t=n+1\) , in which \(n\) is a positive integer, this formula can be written \begin{align} \log n ! & =\left(n+\frac{1}{2}\right) \log n-n+\frac{1}{2} \log 2 \pi+\frac{r(n)}{12 n}, \\[2mm] n! & =\left(\frac{n}{e}\right)^{n} \sqrt{2 \pi n} e^{r(n) / 12 n}, \tag{2.26} \end{align} in which \(r(n)\) satisfies \(1-1 /(12 n+1). The proof of Stirling’s formula may be found in many books. A particularly clear derivation is given by H. Robbins, “A Remark on Stirling’s Formula”, American Mathematical Monthly , Vol. 62 (1955), pp. 26–29.

We next turn to the evaluation of sums and infinite sums . The major tool in the evaluation of infinite sums is Taylor’s theorem, which states that under certain conditions a function \(g(x)\) may be expanded in a power series: \[g(x)=\sum_{k=0}^{\infty} \frac{x^{k}}{k !} g^{(k)}(0), \tag{2.27}\] in which \(g^{(k)}(0)\) denotes the value at \(x=0\) of the \(k\) th derivative \(g^{(k)}(x)\) of \(g(x)\) . Letting \(g(x)=e^{x}\) , we obtain \[e^{x}=\sum_{k=0}^{\infty} \frac{x^{k}}{k !}=1+x+\frac{x^{2}}{2 !}+\cdots+\frac{x^{n}}{n !}+\cdots,-\infty

Take next \(g(x)=(1-x)^{n}\) , in which \(n=1,2, \ldots\) Clearly \begin{align} g^{(k)}(x) = \begin{cases} (-1)^{k}(n)_{k}(1 - x)^{n - k}, & \text{for } k = 0, 1, \ldots, n \\[2mm] 0, & \text{for } k > n. \end{cases} \end{align} 

Consequently, for \(n=1,2, \ldots\) \[(1-x)^{n}=\sum_{k=0}^{n}(-1)^{k}\left(\begin{array}{l} n \tag{2.30} \\ k \end{array}\right) x^{k}, \quad-\inftywhich is a special case of the binomial theorem. One may deduce the binomial theorem from (2.30) by setting \(x=(-b) / a\) .

We obtain an important generalization of the binomial theorem by taking \(g(x)=(1-x)^{t}\) , in which \(t\) is any real number. For any real number \(t\) and any integer \(k=1,2, \ldots\) define the binomial coefficient \begin{align} \binom{t}{k} = \begin{cases} \frac{t(t-1) \cdots (t-k+1)}{k!}, & \text{for } k = 1, 2, \ldots \tag{2.31} \\[2mm] 1, & \text{for } k = 0 \end{cases} \end{align} 

Note that for any positive number \(n\) \begin{align} \left(\begin{array}{c}-n \\ k\end{array}\right) & =(-1)^{k} \frac{n(n+1) \cdots(n+k-1)}{k !} \\ & =(-1)^{k}\left(\begin{array}{c}n+k-1 \\ k\end{array}\right). \tag{2.32} \end{align} 

By Taylor’s theorem, we obtain the important formula for all real numbers \(t\) and \(-1, \[(1-x)^{t}=\sum_{k=0}^{\infty}\left(\begin{array}{l} t \tag{2.33} \\ k \end{array}\right)(-x)^{k}.\] 

For the case of \(n\) positive we may write, in view of (2.32) , \[(1-x)^{-n}=\sum_{k=0}^{\infty}\left(\begin{array}{c} n+k-1 \tag{2.34} \\ k \end{array}\right) x^{k}, \quad|x|<1\] 

Equation (2.34) , with \(n=1\) , is the familiar formula for the sum of a geometric series: \[\sum_{k=0}^{\infty} x^{k}=1+x+x^{2}+\cdots+x^{n}+\cdots=\frac{1}{1-x}, \quad|x|<1 . \tag{2.35}\] 

Equation (2.34) with \(n=2\) and 3 yields the formulas \begin{align} & \sum_{k=0}^{\infty}(k+1) x^{k}=1+2 x+3 x^{2}+\cdots=\frac{1}{(1-x)^{2}}, \quad|x|<1, \tag{2.36} \\ & \sum_{k=0}^{\infty}(k+2)(k+1) x^{k}=\frac{2}{(1-x)^{3}}, \quad|x|<1 . \end{align} 

From (2.33) we may obtain another important formula. By a comparison of the coefficients of \(x^{n}\) on both sides of the equation \[(1+x)^{s}(1+x)^{t}=(1+x)^{s+t},\] we obtain for any real numbers \(s\) and \(t\) and any positive integer \(n\) \[\left(\begin{array}{l} s \tag{2.37} \\ 0 \end{array}\right)\left(\begin{array}{l} t \\ n \end{array}\right)+\left(\begin{array}{l} s \\ 1 \end{array}\right)\left(\begin{array}{c} t \\ n-1 \end{array}\right)+\cdots+\left(\begin{array}{l} s \\ n \end{array}\right)\left(\begin{array}{l} t \\ 0 \end{array}\right)=\left(\begin{array}{c} s+t \\ n \end{array}\right).\] 

If \(s\) and \(t\) are positive integers (2.37) could be verified by mathematical induction. A useful special case of (2.37) is when \(s=t=n\) ; we then obtain (5.13) of Chapter 2 .

Theoretical Exercises