{"loaderData":{"0-0-56":{"bookData":{"_id":"669feeab39e7e3d9dc7ad4c7","category_id":"630da3e25d2b6424645ccb5a","author_id":["66a5bda0ac6df2b414714fcf"],"book_name":"Elementary Theory of Matrices","book_image":"/books/HalmosMatricespng-1721757355679.png","book_description":"

Elementary Theory of Matrices by Paul Halmos is almost a concise and straightforward version of his famous book Finite-Dimensional Vector Spaces. Paul Halmos, known not only for his fundamental advances in various areas of mathematics but also for his exceptional expository skills, presents this short book to teach key topics. These include vector spaces and inner product spaces, linear transformations, spectral theory, dual spaces and tensor products, canonical forms, and limits of linear transformations.

","have_express_version":true,"is_published":false,"status":"ACTIVE","slug":"elementary-theory-of-matrices","buy_regular_link":"","buy_express_link":"","chapter_name_type":1,"created_at":"2024-07-22T14:01:51.431Z","__v":0,"updated_at":"2025-04-05T06:33:52.022Z","sequence":6,"chapter_data":[{"_id":"66a0200639e7e3d9dc7ad4f4","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Vector Spaces and Inner Product Spaces","sequence":1,"chapter_content":"

1. Real and complex vector spaces

Definition 1. A vector space, \\(\\mathcal{V}\\), is a set of elements, \\(x\\), \\(y\\), \\(z\\), etc., called vectors or points, which satisfies the following axioms.

1.1. To every pair, \\(x\\) and \\(y\\), of vectors in \\(\\mathcal{V}\\) there corresponds a vector \\(z\\), called the sum of \\(x\\) and \\(y\\), \\(z=x+y\\).

1.2. Addition is commutative: \\(x+y=y+x\\).

1.3. Addition is associative: \\((x+y)+z=x+(y+z)\\).

1.4. There exists in \\(\\mathcal{V}\\) a unique vector \\(0\\) such that for all \\(x\\) in \\(\\mathcal{V}\\), \\(x+0=x\\).

1.5. To every pair \\((a, x)\\), where \\(a\\) is a complex number and \\(x\\) is in \\(\\mathcal{V}\\), there corresponds a vector \\(y\\) in \\(\\mathcal{V}\\), called the product of \\(a\\) and \\(x\\), \\(y=a x\\), such that \\[ \\begin{aligned} a(x+y)&=ax+ay,\\\\ (a+b) x&=a x+b x,\\\\ a(b x)&=(a b) x,\\\\ 0 x&=0,\\\\ 1 x&=x. \\end{aligned} \\]

We shall refer to complex numbers as scalars or constants; later in these notes we shall discuss vector spaces in which the scalars may lie in an arbitrary field (not necessarily the field of complex numbers). To differentiate between the various cases we shall refer to the vector spaces just defined as complex vector spaces; if (1.5) is satisfied for real numbers, \\(a\\), we shall refer to real vector spaces. Even though throughout the first three chapters we shall be concerned with complex vector spaces only, it will be very helpful to keep in mind the best known example of real vector spaces, namely, two-dimensional real Euclidean space. This space, \\(E_{2}\\), is the set of all pairs, \\((a_{1}, a_{2})=x\\), of real numbers. Addition, zero, and scalar multiplication are defined by the formulas \\[ \\begin{aligned} (a_{1}, a_{2})+(b_{1}, b_{2})&=(a_{1}+b_{1}, a_{2}+b_{2}), \\\\ 0_{2}&=(0,0),\\\\ b(a_{1}, a_{2})&=(b a_{1}, b a_{2}). \\end{aligned} \\]

For any vector \\(x\\) we write, by definition, \\(-x=(-1) x\\); instead of \\(x+(-y)\\) we write \\(x-y\\). We observe that \\[ x-x=1 x+(-1) x=(1-1) x=0 x=0 \\] (so that the elements of \\(\\mathcal{V}\\) form under addition an Abelian group with \\(-x\\) playing the role of the inverse of \\(x\\)) and that for any scalar, \\(a\\), and an arbitrary \\(x\\),

\\[ a 0=a(0 x)=(a 0) x=0 x=0 \\]

(We comment on the fact that the symbol \\(0\\) is used in two different senses, once as a scalar and once as a vector. Later we shall even assign a third meaning to the symbol, but the relation between the various interpretations of it is such that no confusion should arise from this practice.)

2. Linear dependence and dimension

Definition 2. The vectors \\(x_{1}, x_{2}, \\ldots, x_{n}\\) in are linearly dependent if there exist \\(n\\) scalars, \\(a_{1}, a_{2}, \\ldots, a_{n}\\), not all zero, such that \\[ a_{1} x_{1}+a_{2} x_{2}+\\cdots+a_{n} x_{n}=0. \\]

If no such scalars, \\(a\\), exist, the vectors \\(x_{i}\\) (\\(i=1,2, \\ldots, n\\)) are linearly independent.

We shall assume throughout these notes, that there exists a positive integer \\(n\\) such that every set of more than \\(n\\) vectors is linearly dependent. If, moreover, a set of \\(n\\) linearly independent vectors actually exists, we shell say that the linear dimension of \\(\\mathcal{V}\\) is \\(n\\).

3. Inner products and inner product spaces

Two fundamental notions in Euclidean geometry, that occur already in the study of the space \\(E_{2}\\), are the notions of angle and length. In the following, we shall abstract the essential properties of these two notions and carry them over to abstract vector spaces. It turns out more convenient for our purposes to study the analog of not the angle between two vectors but of the cosine of this angle. Suppose then that \\(x=(a_{1}, a_{2})\\) and \\(y=(b_{1}, b_{2})\\) are two vectors in \\(E_{2}\\); we denote the angle between \\(x\\) (or \\(y\\)) and the positive real axis by \\(\\alpha\\) (or \\(\\beta\\)). Then the cosine of the angle between \\(x\\) and \\(y\\) becomes \\[ \\cos (\\alpha-\\beta)=\\cos \\alpha \\cos \\beta+\\sin \\alpha \\sin \\beta=\\frac{a_{1} b_{1}+a_{2} b_{2}}{|x||y|} \\] (where \\(|x|\\) denotes the distance of \\(x\\) from the origin, \\(|x|=(a_{1}^{2}+a_{2}^{2})^{1/2}\\)). It is the expression \\(a_{1} b_{1}+a_{2} b_{2}\\), which we shall denote by \\(\\langle x, y \\rangle\\), that is of interest to us. It is easy to verify that \\(\\langle x, y \\rangle\\), considered as a numerical valued function of the pair of vectors \\(x\\) and \\(y\\), is symmetric in \\(x\\) and \\(y\\), depends linearly on both \\(x\\) and \\(y\\), and has the property that \\(\\langle x, x \\rangle\\), which is the square of the distance between \\(x\\) and the origin, vanishes if and only if \\(x=0\\) and is otherwise positive.

The properties of \\(\\langle x, y \\rangle\\) are in one respect too special for our purposes. For consider the example of one dimensional complex Euclidean space: i.e., the set of all complex numbers \\(z\\). In this space, as well as in \\(E_{2}\\), the notions of angle and length are defined, but if the expression \\(\\langle x, y \\rangle\\), (which in case \\(|x|=|y|=1\\) is equal to the cosine of the angle between \\(x\\) and \\(y\\)), were linear in \\(x\\) and \\(y\\) then we should have \\[ \\langle i,-i \\rangle = -i^{2}\\langle 1, 1 \\rangle = 1, \\] (where \\(i=(-1)^{1 / 2}\\)). This would imply that the angle between \\(i\\) and \\(-i\\) is zero. It is easily verified that in fact the cosine of the angle between \\(x\\) and \\(y\\) (where \\(x\\) and \\(y\\) are two complex numbers of distance one from the origin) is the real part of \\(x \\bar{y}\\), where \\(\\bar{y}\\) denotes the complex conjugate of \\(y\\). In other words in this case the expression \\(x \\bar{y}\\), which is neither symmetric nor linear in \\(y\\), takes the place of the expression \\(xy\\) which might have been suggested by analogy with the situation in \\(E_{2}\\). Using the results of the last two paragraphs as a heuristic indication of what we should require in the general case we proceed to the formal definition.

Definition 3. An inner product, \\(\\langle x, y \\rangle\\), in a vector space, \\(\\mathcal{V}\\), is a complex numerically valued function of the ordered pair of vectors \\(x\\), \\(y\\), such that

3.1. \\(\\langle x, y \\rangle = \\overline{\\langle y, x \\rangle}\\),

3.2. \\(\\langle a_{1} x_{1}+a_{2} x_{2}, y \\rangle = a_{1}\\langle x_{1}, y \\rangle + a_{2}\\langle x_{2}, y \\rangle\\),

3.3. \\(\\langle x, x \\rangle \\geq 0\\); \\(\\langle x, x \\rangle = 0\\) implies \\(x=0\\).

An inner product space, \\(\\mathcal{U}\\), is a vector space in which an inner product is defined.

If \\(\\mathcal{U}\\) is an inner product space with the inner product \\(\\langle x, y \\rangle\\) then we have \\[ \\begin{aligned} \\langle x, b_{1} y_{1}+b_{2} y_{2} \\rangle &= \\overline{\\langle b_{1} y_{1}+b_{2} y_{2}, x \\rangle} \\\\ &=\\overline{b_{1}}\\,\\overline{\\langle y_{1}, x \\rangle}+\\overline{b_{2}}\\,\\overline{\\langle y_{2}, x \\rangle}\\\\ &=\\overline{b_{1}}\\langle x, y_{1}\\rangle + \\overline{b_{2}}\\langle x, y_{2}\\rangle. \\end{aligned} \\]

In analogy with the situation in Euclidean space we define in an inner product space the length, \\(|x|\\), of a vector \\(x\\), by the formula \\[|x|=\\langle x, x \\rangle^{1 / 2}.\\] We note that \\[ |a x|=|a||x|. \\]

4. Schwartz Inequality

We shall now prove for an inner product space the following inequality, known as Schwartz’ Inequality or Cauchy’s Inequality.

Theorem 1. \\[ |\\langle x, y \\rangle| \\leq |x||y|. \\]

Proof. Since \\[ \\begin{aligned} 0 &\\leq |x-y|^{2}\\\\ &=\\langle x-y, x-y \\rangle\\\\ &=\\langle x, x \\rangle - \\langle y, x \\rangle - \\langle x, y \\rangle + \\langle y, y \\rangle \\\\ & =|x|^{2}-\\Big[\\langle x, y \\rangle + \\overline{\\langle x, y \\rangle}\\Big]+|y|^{2}\\\\ &=|x|^{2}-2\\operatorname{Re}\\langle x, y \\rangle + |y|^{2}, \\end{aligned} \\] (where we use \\(\\operatorname{Re} z\\) and \\(\\operatorname{Im} z\\) to denote the real and imaginary parts of \\(z\\), respectively) it follows that \\[ \\operatorname{Re}\\langle x, y\\rangle \\leq \\frac{1}{2}\\big(|x|^{2}+|y|^{2}\\big). \\]

Since, furthermore, for any complex number \\(z\\), we can find a complex number \\(a\\) of absolute value one such that \\(az\\) is real and non-negative, we may find such a number \\(a\\) when \\(z = \\langle x, y \\rangle\\). Then \\[ |\\langle x, y \\rangle|=|a\\langle x, y\\rangle|=\\operatorname{Re}a\\langle x, y\\rangle = \\operatorname{Re}\\langle a x, y\\rangle. \\]

Applying the preceding inequality with \\(ax\\) in place of \\(x\\) we obtain \\[ \\begin{aligned} |\\langle x, y \\rangle|&=\\operatorname{Re}\\langle ax, y \\rangle\\\\ &\\leq \\frac{1}{2}\\left(|a x|^{2}+|y|^{2}\\right)\\\\ &=\\frac{1}{2}\\left(|x|^{2}+|y|^{2}\\right). \\end{aligned} \\]

Let \\(b\\) be a positive number for which \\(b^{2}=|y| /|x|\\) and apply the last inequality to the vectors \\(b x\\) and \\((1 / b) y\\) in place of \\(x\\) and \\(y\\). We obtain \\[ \\begin{aligned} |\\langle x, y \\rangle|&=\\left|\\left\\langle b x,\\Big(\\frac{1}{b}\\Big) y\\right\\rangle\\right|\\\\ &\\leq \\frac{1}{2}\\left(b^{2}|x|^{2}+\\Big(\\frac{1}{b}\\Big)^{2}|y|^{2}\\right)\\\\ &=\\frac{1}{2}\\big(|x||y|+|x||y|\\big), \\end{aligned} \\] and this is the desired result.■

5. Triangle Inequality in inner product spaces

A geometric interpretation and application of the Schwartz inequality is the following. For any two vectors \\(x\\) ind \\(y\\) in the inner product space \\(\\mathcal{U}\\) we may define the distance between \\(x\\) and \\(y\\) to be \\(|x-y|\\). It is clear that this distance is non-negative and symmetric in \\(x\\) and \\(y\\), and that it vanishes if and only if \\(x=y\\). It is precisely the Schwartz inequality which implies that this distance has the remaining property usually required of a distance function, namely that the triangle inequality, \\[ |x-z| \\leq |x-y|+|y-z|, \\] is valid. For we have \\[ \\begin{aligned} |x+y|^{2} &= \\langle x+y, x+y\\rangle\\\\ &= \\langle x, x \\rangle +\\langle x, y \\rangle + \\langle y, x \\rangle+ \\langle y, y\\rangle \\\\ &= |x|^{2}+2\\operatorname{Re}\\langle x, y \\rangle + |y|^{2}\\\\ &\\leq |x|^{2}+2|\\langle x, y \\rangle|+|y|^{2} \\\\ &\\leq |x|^{2}+2|x||y|+|y|^{2}\\\\ &= \\big(|x|+|y|\\big)^{2}, \\end{aligned} \\] so that \\(|x+y| \\leq |x|+|y|\\), and replacing \\(x\\) by \\(x-y\\) and \\(y\\) by \\(y-z\\), we obtain the triangle inequality.

6. \\(n\\)-dimensional complex Euclidean space

As an example of an inner product space, we consider \\(n\\)-dimensional complex Euclidean space, \\(\\overline{E}_n\\), which is the set of all ordered sets of \\(n\\) complex numbers. If \\(x=(a_{1},a_{2},\\ldots, a_{n})\\) and \\(y=(b_{1}, b_{2}, \\ldots, b_{n})\\) are two points of this space and \\(a\\) and \\(b\\) are complex numbers, we define \\(a x+b y\\) to be \\[ (a a_{1}+b b_{1}, a a_{2}+b b_{2}, \\ldots, a a_{n}+b b_{n}); \\] if in addition we define \\(0=(0,0, \\ldots, 0)\\) and \\[ \\langle x, y \\rangle = a_{1} \\bar{b}_{1}+a_{2} \\bar{b}_{2}+\\cdots+a_{n} \\bar{b}_{n}, \\] it is easy to verify that \\(\\overline{E}_n\\) is an inner product space. Applying the Schwartz inequality to a pair of vectors \\(x\\) and \\(y\\) in this space we get the arithmetic relation \\[ \\left|\\sum_{j=1}^{n} a_{j} \\bar{b}_{j}\\right|^{2} \\leq \\sum_{j=1}^{n}|a_{j}|^{2} \\sum_{j=1}^{n}|b_{j}|^{2}. \\]

Since in an arbitrary inner product space the expression \\(\\langle x, y \\rangle /|x||y|\\) plays the role of the cosine of the angle between \\(x\\) and \\(y\\), the Schwartz inequality can also be considered as a generalization of the fact that for real angles \\(t\\), \\(|\\cos t| \\leq 1\\).

7. Linear subspaces

Definition 4. A linear subspace ¹ (also called a vector subspace or simply a subspace) in a vector space \\(\\mathcal{V}\\) is a non-empty set, \\(M\\), of vectors in \\(\\mathcal{V}\\), which contains along with every pair of vectors \\(x\\) and \\(y\\) the vector \\(a x+b y\\), for arbitrary complex numbers \\(a\\) and \\(b\\).

It follows immediately from the definitions that a linear subspace in a vector space (or an inner product space) is a vector space (or an inner product space).

8. Orthonormal sets

Definition 5. For vectors \\(x\\) and \\(y\\) in an inner product space we say that \\(x\\) is orthogonal to \\(y\\), in symbols \\(x \\perp y\\), if \\(\\langle x, y \\rangle = 0\\). Two linear subspaces, \\(M\\) and \\(N\\), are orthogonal if every vector in each is orthogonal to every vector in the other. A set, \\(\\{x_{i}\\}\\), of vectors is an orthonormal set if \\(\\langle x_{i}, x_{j} \\rangle = \\delta_{ij}\\) (where \\(\\delta_{ij}\\) is the Kronecker symbol which is equal to \\(1\\) when \\(i=j\\) and is equal to \\(0\\) otherwise).

If \\(\\{x_i\\}\\) is a finite orthonormal set, then \\(\\sum_{i} a_{i} x_{i}=0\\) implies that \\[ a_{j}=\\sum_{i} a_{i}\\langle x_{i}, x_{j} \\rangle = 0, \\] so that an orthonormal set is linearly independent. Hence if \\(n\\) is the linear dimension of the space, no orthonormal set can have more than \\(n\\) elements; the maximal number of elements is the orthogonal dimension of the space.

If an orthonormal set is not contained in any larger orthonormal set it is complete.

It follows from the definition of the terms that the orthogonal dimension is not greater than the linear dimension.

9. Intersection and linear sum of linear subspaces

The intersection of any collection of linear subspaces is again a linear subspace. If \\(M\\) and \\(N\\) are linear subspaces we shall denote their intersection by \\(M \\cap N\\). If \\(K\\) is an arbitrary set of vectors there exist linear subspaces containing \\(K\\), (since the whole space is such), and we may form the intersection of all linear subspaces containing \\(K\\). This linear subspace is the linear subspace spanned by \\(K\\); if \\(M\\) and \\(N\\) are linear subspaces we shall denote the linear subspace spanned by the vectors of \\(M\\) and \\(N\\) by \\(M + N\\). \\(M+N\\) is the linear sum of \\(M\\) and \\(N\\).

10. Bessel’s Inequality

Theorem 2. If \\(\\{x_{i}\\}\\) is an orthonormal set, \\(x\\) is any vector, and \\(a_{i}=\\langle x, x_{i} \\rangle\\), then \\[ \\sum_{i}{|a_i|^{2}} \\leq |x|^{2}. \\]

Proof. Let \\(x^{*}\\) be the vector \\[ x^{*}=x-\\sum_{i} a_{i} x_{i}. \\] Then, since \\(|x^{*}| \\geq 0\\), we have \\[ \\begin{aligned} 0 &\\leq\\bigg\\langle x-\\sum_i a_i x_i, x-\\sum_i a_i x_i\\bigg\\rangle \\\\ &=\\langle x, x \\rangle - \\bigg\\langle x, \\sum_i a_i x_i\\bigg\\rangle - \\bigg\\langle \\sum_i a_i x_i, x\\bigg\\rangle + \\bigg\\langle\\sum_i a_i x_i, \\sum_i a_i x_i\\bigg\\rangle \\\\ &=\\langle x, x \\rangle -\\sum_i \\bar{a}_i\\langle x, x_i\\rangle - \\sum_i a_i\\langle x_i, x \\rangle + \\sum_i \\sum_j a_i \\bar{a}_j\\langle x_i, x_j \\rangle \\\\ &=\\langle x, x \\rangle - \\sum_i|a_i|^2-\\sum_i|a_i|^2+\\sum_i|a_i|^2\\\\ &=|x|^2-\\sum_i|a_i|^2, \\end{aligned} \\]and this is the desired result.■

Corollary 1. \\(x^*=x-\\sum_{i} a_{i} x_{i}\\) is orthogonal to each \\(x_{i}\\) and therefore to the linear subspace spanned by \\(\\{x_{i}\\}\\); a necessary and sufficient condition that \\(x\\) belong to this linear subspace is that \\(x=\\sum_{i}a_{i} x_{i}\\).

Proof. We have \\[ \\begin{aligned} \\langle x^{*}, x_{j} \\rangle &= \\langle x, x_{j} \\rangle - \\sum_{i} a_{i}\\langle x_{i}, x_{j} \\rangle\\\\ &=a_{j}-\\sum_{i} a_i \\delta_{i j}\\\\ &=0. \\end{aligned} \\] This proves the first part of the corollary. The sufficiency of the condition of the second part is obvious. If on the other hand, \\(x\\) belongs to the linear subspace spanned by \\(\\{x_{i}\\}\\) then \\(x^{*}\\), being a linear combination of \\(x\\) and the \\(x_{i}\\) must also belong to this linear subspace Since, however, \\(x^{*}\\) is orthogonal to every vector in this linear subspace it must be orthogonal to itself, i.e., \\(x^{*}=0\\), and the condition is necessary.■

11. Characterization of complete orthonormal sets

We shall now prove that the following five conditions on an orthonormal set \\(x_{i}\\) are equivalent to each other.

11.1. \\(\\{x_{i}\\}\\) is complete.

11.2. \\(\\langle x, x_{i} \\rangle = 0\\) for all \\(i\\) implies \\(x=0\\).

11.3. The linear subspace spanned by \\(\\{x_{i}\\}\\) is the whole space, \\(\\mathcal{U}\\).

11.4. For every \\(x\\) in \\(\\mathcal{U}\\), \\(x=\\sum_{i} a_i x_{i}\\), where \\(a_{i}=\\langle x, x_i \\rangle\\).

11.5. For every pair \\(x\\), \\(y\\) in \\(\\mathcal{U}\\) \\[ \\langle x, y \\rangle = \\sum_{i} a_{i} \\bar{b}_{i}, \\] where \\(a_i = \\langle x, x_{i} \\rangle\\) and \\(b_{i} = \\langle y, x_{i} \\rangle\\). (11.5 is Parseval’s identity.)

We shall prove this by establishing the following implication relations.

\\[ (11.1) \\implies (11.2) \\implies (11.3) \\implies (11.4) \\implies (11.5) \\implies (11.1) \\]

(11.1) \\(\\implies\\) (11.2): If there exists a vector \\(x \\neq 0\\) with \\(\\langle x, x_{i} \\rangle = 0\\) for all \\(i\\), then the set of vectors formed by \\(\\{x_{i}\\}\\) and \\(x /|x|\\) is an orthonormal set containing \\(\\{x_{i}\\}\\).

(11.2) \\(\\implies\\) (11.3): If \\(\\{x_{i}\\}\\) does not span the whole space then, by the corollary to Bessel’s inequality, there exists a vector \\(x\\) which is not of the form \\(\\sum_{i} a_{i} x_{i}\\). It follows from this corollary that the vector \\(x^*=x-\\sum_{i}\\langle x, x_{i} \\rangle x_{i}\\) is different from zero and is orthogonal to each \\(x_{i}\\).

(11.3) \\(\\implies\\) (11.4): This implication is a direct consequence of the corollary to Bessel’s inequality.

(11.4) \\(\\implies\\) (11.5): Assuming (11.4) we have \\(x=\\sum_i a_i x_{i}\\) and \\(y=\\sum_i b_{i} x_{i}\\), whence \\[ \\begin{aligned} \\langle x, y \\rangle &= \\bigg\\langle \\sum_{i} a_{i} x_{i}, \\sum_{j} b_{j} x_{j}\\bigg\\rangle\\\\ &= \\sum_{i} \\sum_{j} a_{i} b_{j}\\langle x_{i}, x_{j}\\rangle\\\\ &= \\sum_{i}a_i \\bar{b}_{i}. \\end{aligned} \\]

(11.5) \\(\\implies\\) (11.1): If \\(\\{x_{i}\\}\\) were contained in a larger orthonormal set, say if \\(x_{0}\\) is orthogonal to each \\(x_{i}\\), then taking \\(x=y=x_{0}\\) in (11.5) we obtain \\(|x_{0}|^{2}=0\\).

We observe that (11.5) with \\(x=y\\) is the natural generalization of the Pythagorean theorem.

12. Erhardt Schmidt orthogonalization process

Although we have proved several relations among properties of complete orthonormal sets, we have never yet established that such sets exist. The purpose of the construction that follows is to prove the existence of complete orthonormal sets.

Let \\(\\mathcal{U}\\) be an inner product space of linear dimension \\(n\\), and let \\(z_{1}\\), \\(z_{2}, \\ldots, z_{n}\\) be a set of linearly independent vectors. We define \\(x_{1}=z_{1} /|z_{1}|\\); then the set \\(\\{x_{1}\\}\\) consisting of the single vector \\(x_{1}\\) is an orthonormal set. We proceed by induction. Suppose that the vectors \\(x_{1}, x_{2}, \\ldots, x_{r}\\) have been defined so that they form an orthonormal set and so that \\(x_{i}\\) is a linear combination of \\(z_{1}, \\ldots, z_{1}\\) for \\(i=1,2, \\ldots, r\\). We write \\(y=c_{1} x_{1}+\\cdots+c_{r} x_{r}+z_{r+1}\\), and we determine the coefficients \\(c_{i}\\) so that \\(y\\) should be orthogonal to each \\(x_{i}\\), \\(i=1, \\ldots, r\\). Since \\(\\langle y, x_{i} \\rangle = c_{i} + \\langle z_{r+1}, x_{i} \\rangle\\), we may choose \\(c_{i}=-\\langle z_{r+1}, x_{i} \\rangle\\). Since \\(y\\) is a linear combination of \\(x_{1}, \\ldots, x_{r}\\) and \\(z_{r+1}\\) and is therefore a linear combination of \\(z_{1}, \\ldots, z_{r}, z_{r+1}\\) in such a way that the coefficient of \\(z_{r+1}\\) does not vanish (in fact it is equal to \\(1\\)), the linear independence of the \\(z_{i}\\) implies that \\(y \\neq 0\\). Hence we may define \\(x_{r+1}=y /|y|\\), and the set \\(x_{1}, \\ldots, x_{r}, x_{r+1}\\) will again satisfy our induction hypothesis. The vectors \\(x_{1}, \\ldots, x_{n}\\) so constructed are an orthonormal set which must be complete, for if it were contained in a larger orthonormal set then there would exist a set of more than \\(n\\) linearly independent vectors. This construction proves, therefore, that complete orthonormal sets do indeed exist and that the orthogonal dimension of \\(\\mathcal{U}\\) is equal to its linear dimension.

For a complete orthonormal set, we shall also use the terms ‘coordinate system,’ ‘frame of reference,’ and ‘orthogonal basis.’ By ‘basis’ or ‘linear basis’ we mean a set of \\(n\\) linearly independent vectors, where \\(n\\) is the dimension of the space. (Since we have proved the equality of the orthogonal and linear dimensions, we shall in the future not distinguish between them.) If \\(\\{x_{i}\\}\\) is a coordinate system then, by (11.4), every \\(x\\) can be written in the form \\(x=\\sum_{i} a_{i} x_{i}\\) where \\(a_{i} = \\langle x, x_{i} \\rangle\\). The numbers \\(a_{j}\\) are the coordinates of \\(x\\) with respect to \\(\\{x_{i}\\}\\).

13. The projection theorem

Theorem 3. Let \\(M\\) be a linear subspace in \\(\\mathcal{U}\\). Every vector \\(x\\) in \\(\\mathcal{U}\\) can be written in the form \\(x=y+z\\), where \\(y\\) is in \\(M\\) and \\(z\\) is orthogonal to every vector in \\(M\\), in one and only one way.

Proof. \\(M\\), being a linear subspace in an inner product space, is itself an inner product space, and we may find an orthonormal set \\(\\{x_{i}\\}\\) in \\(M\\) which is complete in \\(M\\). For any \\(x\\), write \\(y=\\sum_{i} a_{i} x_{i}\\), where \\(a_{i} = \\langle x, x_{i} \\rangle\\). Then clearly \\(y\\) is in \\(M\\) and by the corollary to Bessel’s inequality, \\(z=x-y\\) is orthogonal to every vector in \\(M\\). This establishes the existence of the representation; to prove uniqueness suppose that \\(x=y_{1}+z_{1}\\) and \\(x=y_{2}+z_{2}\\), where \\(y_{i}\\) and \\(z_{i}\\) (\\(i=1, 2\\)) have the properties of \\(y\\) and \\(z\\), respectively. Then we should have \\(y_{1}-y_{2}=z_{2}-z_{1}\\): i.e., a vector in \\(M\\) is equal to a vector which is orthogonal to \\(M\\). It follows that this vector is orthogonal to itself and is therefore zero, whence \\(y_{1}=y_{2}\\) and \\(z_{1}=z_{2}\\), as was to be proved.■

The vector \\(y\\) is the projection of \\(x\\) on \\(M\\); it is easy to verify that it is a generalization of the well known notion of perpendicular projection in Euclidean space. We shall investigate properties of projections more thoroughly in the next chapter.

14. Calculus of linear subspaces

We make some comments on the calculus of linear subspaces. If \\(M\\) is an arbitrary linear subspace, we denote by \\(M^\\perp\\) the set of all vectors orthogonal to \\(M\\). It is clear that \\(M^\\perp\\) is a linear subspace; we call \\(M^\\perp\\) the orthocomplement of \\(M\\).

The projection theorem proves that \\(M + M^\\perp\\), in the sense established in (9), is \\(\\mathcal{U}\\). If \\(M\\) and \\(N\\) are any two linear subspaces, \\(M \\cap N^\\perp\\) is a linear subspace. If \\(\\{x_{i}\\}\\) is a complete orthonormal set in \\(M\\) and \\(\\{y_{j}\\}\\) is a complete orthonormal set in \\(M^\\perp\\) then the set of vectors consisting of all \\(x_{i}\\) and all \\(y_{j}\\) is a complete orthonormal set in \\(\\mathcal{U}\\). If \\(M\\) is any linear subspace and \\(x\\) is a vector such that \\(x \\perp M^\\perp\\), (i.e., if \\(x\\) lies in \\(M^{\\perp\\perp}=\\left(M^\\perp\\right)^\\perp\\)) then \\(x\\) is in \\(M\\). For by the projection theorem, we may write \\(x=y+z\\), with \\(y\\) in \\(M\\) and \\(z\\) in \\(M^\\perp\\). Since an element, \\(y\\), of \\(M\\) is orthogonal to \\(M^\\perp\\), and is therefore in \\(M^{\\perp\\perp}\\), \\(x-y=z\\) must also be in \\(M^{\\perp\\perp}\\). Since \\(z\\), therefore, is both in \\(M^\\perp\\) and orthogonal to \\(M^\\perp\\), \\(z\\) must be zero. It follows that \\(M^{\\perp\\perp} = M\\).

15. Representation of linear functions

Of interest in analysis in general and in our study in particular is the study of linear functions on vector spaces.

Definition 6. A linear function \\(f(x)\\) is a (complex) numerically valued function of the vector \\(x\\) such that for any two vectors \\(x\\) and \\(y\\) and any two complex numbers \\(a\\) and \\(b\\), \\(f(a x+b y)=a f(x)+b f(y)\\).

(A geometric interpretation of linear functions is the fact that the set of all vectors \\(x\\) for which \\(f(x)=0\\), where \\(f(x)\\) is a linear function not identically zero, is a hyperplane in \\(\\mathcal{U}\\). Because two linear functions \\(f(x)\\) and \\(g(x)\\) related by \\(f(x)=a g(x)\\) for all \\(x\\), where \\(a\\) is any nonzero constant, correspond to the same hyperplane, it turns out a little more convenient to study the linear functions themselves and not the hyperplanes, that is, classes of linear functions defined by them. It is easy to verify, but we shall not make use of this fact in this form, that the hyperplanes of a vector space of dimension \\(n\\) are the linear subspaces of dimension \\(n-1\\). It is also easy to verify that the set of all linear functions defined on a vector space is itself a vector space; one interpretation of the theorem to be proved below is that the set of all linear functions on an inner product space is itself an inner product space.)

Theorem 4. If \\(f(x)\\) is a linear function in an inner product space \\(\\mathcal{U}\\), there exists a unique vector \\(y\\) in \\(\\mathcal{U}\\) such that for all \\(x\\), \\(f(x) = \\langle x, y \\rangle\\).

Proof. Since the theorem is obvious if \\({f}({x})\\) is identically zero, we may assume that this is not the case. Let \\(M\\) be the set of all vectors \\(x\\) for which \\(f(x)=0\\), and let \\(N = M^\\perp\\) be the orthocomplement of \\(M\\). Then \\(N\\) contains a vector \\(y_{0} \\neq 0\\) for which \\(f(y_{0}) \\neq 0\\); by multiplication by a suitable constant we may assume \\(f(y_{0})=1\\).

Consider now any vector \\(z\\) in \\(N\\). For any complex number \\(a\\), \\(z-ay_{0}\\) is also in \\(N\\) and we have \\[ f(z-a y_{0})=f(z)-a f(y_{0})= f(z)-a. \\]

If we choose \\(a=f(z)\\), then \\(f(z-ay_{0})=0\\), so that \\(z - ay_0\\) is in \\(M\\); since we have already seen that it must be in \\(N\\), it follows that \\(z-ay_{0}=0\\), or \\(z=f(z) y_{0}\\). (This shows that \\(N\\) is one dimensional and therefore that the dimension of \\(M\\) is \\(n-1\\).) We write \\(y=y_{0} /|y_{0}|^{2}\\), and if \\(x\\) is an arbitrary vector in \\(\\mathcal{U}\\) we use the projection theorem to write \\(x\\) in the form \\(x=w+z\\), with \\(w\\) in \\(M\\) and \\(z\\) in \\(N\\). Then \\[ \\begin{aligned} \\langle x, y \\rangle &= \\langle w+z, y \\rangle\\\\ &= \\langle z, y \\rangle\\\\ &=\\big\\langle f(z) y_{0}, y\\big\\rangle\\\\ &=f(z)\\big\\langle y_{0}, y_{0} /|y_{0}|^{2}\\big\\rangle\\\\ &=f(z); \\end{aligned} \\] and since \\(w\\) is in \\(M\\), \\(f(w)=0\\) so that \\(f(x)=f(w+z)=f(z)\\). This completes the proof of the existence of \\(y\\).

To prove uniqueness, suppose that for every \\(x\\), \\(\\langle x, y_{1} \\rangle = \\langle x, y_{2} \\rangle\\). Then we should have \\(\\langle x, y_{1}-y_{2}\\rangle = 0\\) for all \\(x\\): i.e., the vector \\(y_{1} - y_{2}\\) is orthogonal to every vector, and therefore in particular to itself, so that \\({y}_{1}={y}_{2}\\), as was to be proved.■

Also called a linear manifold. ↩︎

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"vector-spaces-and-inner-product-spaces","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]},{"_id":"66a03ba239e7e3d9dc7ad580","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Linear Transformations and Matrices","sequence":2,"chapter_content":"

1. Linear transformations in vector spaces

A linear transformation \\(A\\) in a vector space \\(\\mathcal{V}\\) is a correspondence which assigns to every vector \\(x\\) in \\(\\mathcal{V}\\) another vector \\(A x\\) in \\(\\mathcal{V}\\) in such a way that \\(A(ax + by)=aAx + bAy\\), for any two vectors \\(x\\) and \\(y\\) and any two complex numbers \\(a\\) and \\(b\\).

If \\(A\\) is any linear transformation and \\(a\\) any complex number, we define a linear transformation \\(B=aA\\) by \\(Bx = aAx\\). For any two linear transformations \\(A\\) and \\(B\\) we define their sum \\(C=A+B\\) by \\(C x=A x+B x\\) and their product \\(D=AB\\) by \\(D x=A(B x)\\). Two special linear transformation of interest are \\(0\\), defined by \\(0 x=0\\) for all \\(x\\), and \\(1\\), defined by \\(1 x=x\\) for all \\(x\\). If the correspondence between \\(x\\) and \\(y=Ax\\) happens to be one to one, in other words if every vector \\(y\\) can be written in the form \\(Ax\\) in one and only one way, we may define a transformation \\(B=A^{-1}\\) by \\(B y=x\\). It is easy to verify that \\(A^{-1}\\) is a linear transformation.

(The transformations we call linear are sometimes called homogeneous linear transformations: i.e., they have the property that \\(A0=0\\).)

We observe, without proof, that the following formulas are valid for linear transformations: the proofs of all of these facts are immediate from the definitions. \\[ \\begin{aligned} A+(B+C)&=(A+B)+C\\\\ A+B&=B+A\\\\ A(B C)&=(A B) C \\\\ A(B+C)&=A B+A C\\\\ (A+B) C&=A C+B C\\\\ a(A+B)&=a A+a B \\\\ (a+b) A&=a A+b A\\\\ a(b A)&=(a b) A\\\\ A+0&=A \\\\ A1&=1A=A\\\\ A A^{-1}&=A^{-1} A=1 \\end{aligned} \\]

The associative law enables us to define for every positive integer \\(n\\) the transformation \\(A^{n}\\) by the recursive definition \\(A^{n}=A A^{n-1}\\), \\(A^{1}=A\\). Although in general the multiplication of two transformations is not commutative, and in fact much of the difficulty and interest in the theory is due to this fact, for powers of one transformation we do have \\(A^n A^m = A^{n + m} = A^m A^n\\). If we make the convention that \\({A}^{0}=1\\) and if we define, in case \\(A^{-1}\\) exists, \\(A^{-n}\\), for any positive integer \\(n\\), by \\(A^{-n}=(A^{-1})^{n}\\) then the calculus of powers of a single linear transformation is exactly the same as in ordinary arithmetic. In accordance with this comment and the properties of addition and scalar multiplication, for every polynomial \\(p(z)=a_{0}+a_{1} z+\\cdots+a_{n} z^{n}\\) we may write \\(p(A)\\) as an abbreviation for the linear transformation \\(\\sum_{i=0}^{n} a_{i} A^{i}\\). These ideas will be very useful to us later.

We make also some more comments about \\({A}^{-1}\\), the inverse of \\(A\\). We observe first that a necessary and sufficient condition that \\(A\\) has an inverse, (in other words that be a one to one transformation), is that \\(Ax=0\\) imply \\(x=0\\). The necessity of this condition is obvious: we proceed to the proof of sufficiency. Since \\(A x=A y\\) implies \\(A(x-y)=0\\), it follows that \\(x=y\\), so that \\(A\\) is one to one as far as it goes. It remains to prove only that every vector has the form \\(A x\\). For this purpose let \\(x_{1}, x_{2}, \\ldots, x_{n}\\) be any \\(n\\) linearly independent vectors in \\(\\mathcal{V}\\): we claim that \\(A x_{1}, A x_{2}, \\ldots, A x_{n}\\) are also linearly independent. For if we had \\(\\sum_{i} c_{i} A x_{i}=0\\), then \\(A\\big(\\sum_{i} c_{i} x_{i}\\big)=0\\) whence \\(\\sum_{i} c_{i} x_{i}=0\\) and the linear independence of the \\(x_{i}\\) implies that \\(c_{i}=0\\) for all \\(i\\). Hence the set of all vectors of the form \\(A x\\), which is clearly a linear subspace, contains \\(n\\) linearly independent vectors and is therefore \\(n\\)-dimensional. It follows that it must coincide with \\(\\mathcal{V}\\).

We have already stated that \\(A A^{-1}=A^{-1} A=1\\). We claim now that these equations are characteristic of \\(A^{-1}\\): in other words if any linear transformation \\(B\\) exists for which \\(A B=1\\) then \\(B=A^{-1}\\). For

\\[ B=(A^{-1} A) B=A^{-1}(A B)=A^{-1} 1=A^{-1}. \\]

Similarly we could prove that if there is a \\(C\\) for which \\(C A=1\\) then \\(C=A^{-1}\\). It is an immediate consequence of this result that \\((A^{-1})^{-1}=A\\) and \\((A B)^{-1}=B^{-1} A^{-1}\\), and since \\(c A=(c 1) A\\), that \\((c A)^{-1}=c^{-1} A^{-1}\\).

2. The dual of a transformation

Let \\(A\\) be a linear transformation in an inner product space \\(\\mathcal{U}\\) and consider the expression \\(\\langle A x, y \\rangle\\). Since \\[ \\big\\langle A(a_{1} x_{1}+a_{2} x_{2}), y\\big\\rangle = a_{1}\\langle A x_{1}, y \\rangle + a_{2}\\langle A x_{2}, y \\rangle, \\] \\(\\langle A x, y \\rangle\\) is for each fixed \\(y\\) a linear function of \\(x\\). Hence, by the theorem of (I.15), there exists a uniquely determined vector, say \\(y^{*}\\), for which \\(\\langle Ax, y \\rangle = \\langle x, y^{*} \\rangle\\) for all \\(x\\). We denote the correspondence which assigns to every vector \\(y\\) the vector \\(y^*\\) as just defined by \\(B\\): \\(B y=y^*\\). We prove that \\(B\\) is a linear transformation. For if \\(\\langle A x, y_{1} \\rangle = \\langle x, y_{1}^{*} \\rangle\\) and \\(\\langle A x, y_{2} \\rangle = \\langle x, y_{2}^{*} \\rangle\\), where \\({y}_{1}^{*}=B{y}_{1}\\) and \\({y}_{2}^{*}=By_{2}\\), then by multiplying the two equations involving inner products by \\(\\bar{a}_{1}\\) and \\(\\bar{a}_{2}\\) respectively and adding, we obtain \\[ \\langle A x, a_{1} y_{1}+a_{2} y_{2} \\rangle = \\langle x, a_{1} y_{1}^{*}+a_{2} y_{2}^{*} \\rangle, \\] so that (using the uniqueness statement in (I.15)) \\[ B(a_{1} y_{1}+a_{2} y_{2})=a_{1} B y_{1}+a_{2} B y_{2}, \\] as was to be proved.

The process just described associates with every linear transformation \\(A\\) another linear transformation \\(B\\) which we shall denote by \\(A^{*}\\) and call the dual of \\(A\\) (This terminology is justified by the well known geometric language of duality: roughly speaking when \\(A\\) is applied in the space of vectors, \\(A^{*}\\) is applied in the dual space of hyperplanes.) The dual of \\(A\\) is uniquely characterized by the fundamental relation \\(\\langle Ax, y \\rangle = \\langle x, A^{*} y \\rangle\\). If we denote the dual of the dual of \\(A\\) by \\(A^{**}\\) then the relation

\\[ \\overline{\\langle y, A^{*} x \\rangle} = \\langle A^{*} x, y \\rangle = \\langle x, A^{**}y \\rangle = \\overline{\\langle A^{**}y, x \\rangle} \\]

implies (interchanging the roles of \\(x\\) and \\(y\\) and removing the conjugates) that for all \\(x\\) and \\(y\\) we have \\(\\langle A^{**}x, y \\rangle = \\langle x, A^{*} y \\rangle = \\langle A x, y \\rangle\\), so that \\(A^{**}=A\\).

The relation of the dual to the previously introduced operations (among linear transformations) of addition, multiplication, and scalar multiplication is completely described by the following identities: \\[ \\begin{aligned} (A+B)^{*}&=A^{*}+B^{*},\\\\ (A B)^{*}&=B^{*}{A}^{*},\\\\ (a A)^{*}&=\\bar{a}A^{*}. \\end{aligned} \\] The proofs of these identities will be found in the equations: \\[ \\begin{aligned} \\big\\langle (A + B)x, y\\big\\rangle &= \\langle A x, y \\rangle + \\langle B x, y \\rangle \\\\ &= \\langle x, A^{*} y \\rangle + \\langle x, B^{*} y \\rangle \\\\ &= \\big\\langle x, (A^{*}+B^{*}) y\\big\\rangle\\\\ \\langle A B x, y \\rangle &= \\langle B x, A^{*} y \\rangle = \\langle x, B^{*} A^{*} y\\rangle\\\\ \\langle a A x, y \\rangle &= a\\langle A x, y \\rangle = a\\langle x, A^{*} y \\rangle = \\langle x, \\bar{a}{A}^{*} y \\rangle \\end{aligned} \\]

We observe that \\(0^{*}=0\\) and \\(1^{*}=1\\), and that in case \\(A\\) has an inverse then \\(A^{*}\\) does also and \\((A^{*})^{-1}=(A^{-1})^{*}\\). The latter fact follows from the identity \\[ (A^{-1})^{*} A^{*}=\\big(A(A^{-1})\\big)^{*}=1^{*}=1. \\]

3. Matrices associated with linear transformations

Let \\(x_{1}, x_{2}, \\ldots, x_{n}\\) be an orthogonal basis in the inner product space \\(\\mathcal{U}\\) of dimension \\(n\\), and let \\(A\\) be a linear transformation on \\(\\mathcal{U}\\). Since for every \\(j\\), \\(j=1,2, \\ldots, n\\), \\(A x_{j}\\) is a vector in \\(\\mathcal{U}\\) it may, in virtue of (I.11.4), be written (uniquely) as a linear combination of the \\(x_{i}\\), say \\(A x_{j} = \\sum_{i} a_{i j} x_{i}\\). The set \\((a_{i j})\\) of \\(n^{2}\\) indexed complex numbers is a matrix; a matrix is usually written in the form of a square array, \\[ \\begin{aligned} \\begin{bmatrix} a_{11} & a_{12} & \\cdots & a_{1n}\\\\ a_{21} & a_{22} & \\cdots & a_{2n}\\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ a_{n1} & a_{n2} & \\cdots & a_{nn} \\end{bmatrix}. \\end{aligned} \\]

We shall consistently use the following notation. Capital Latin letters denote (as before) linear transformations, the corresponding lower case letters with double subscripts will be the elements of the corresponding matrix, and the capital letters in square brackets will stand for the matrix itself. When a linear transformation is distinguished by symbols, as \\(A^{*}\\) or \\(A^{\\prime}\\), the corresponding matrix elements and matrices will be denoted by \\(a_{i j}^{*}\\) or \\(a_{ij}^\\prime\\) and \\([A^*]\\) or \\([A^\\prime]\\) respectively. Two matrices \\([A]\\) and \\([B]\\) are equal if \\(a_{i j}=b_{i j}\\) for every \\(i\\) and \\(j\\).

With the aid of a fixed orthogonal basis \\(\\{x_{i}\\}\\) we have made correspond a matrix \\([A]\\) to every linear transformation \\(A\\): the correspondence is described by the relations \\(A x_{j}=\\sum_{i} a_{ij} x_i\\). We assert now that this correspondence is one to one. For let \\(A\\) be a linear transformation and let \\(x\\) be any vector. Then \\(x\\) is a linear combination of the vectors \\(x_{j}\\) of the basis, say \\(x=\\sum_{j} c_{j} x_{j}\\), and the linearity of \\(A\\) implies that \\[ A x=\\sum_{j} c_j Ax_{j}=\\sum_{j} c_{j} \\sum_{i} a_{i j} x_{i}=\\sum_{i} \\Big(\\sum_{j} a_{i j} c_{j}\\Big) x_{i}, \\]

so that the vector \\(x\\) whose \\(j\\)-th coordinate in the coordinate system \\(\\{x_i\\}\\) is \\(c_{j}\\) becomes, upon application of the transformation \\(A\\), the vector whose \\(i\\)-th coordinate in the same coordinate system is \\(\\sum_{j} a_{i j} c_{j}\\). Conversely if \\((a_{ij})\\) is any matrix we may define a transformation \\(A\\) by the formula \\[ A\\Big(\\sum_{j} c_{j} x_{j}\\Big)=\\sum_{i}\\Big(\\sum_{j} a_{ij} c_{j}\\Big) x_{i}; \\]

it is easy to verify that \\(A\\) is a linear transformation whose corresponding matrix is precisely \\([A]=(a_{ij})\\). We emphasize the fundamental fact that this one to one correspondence was set up by means of a particular coordinate system \\(\\{x_{i}\\}\\) and that as we pass from one coordinate system to another the same linear transformation may correspond to several matrices and one matrix may be the correspondent of several linear transformations. In fact the relation between the different matrices that may correspond to one linear transformation in various coordinate systems will be the object of study in much of what follows.

4. Isomorphism of matrices and transformations

Although the matrix associated with a linear transformation depends on a particular coordinate system, several properties of the correspondence between transformations and matrices are the same in all coordinate systems. In this section we study some of these properties. Throughout we assume that \\(\\{x_{i}\\}\\) is an arbitrary but fixed coordinate system and that the matrices we discuss are related to linear transformations by means of this system.

If \\(x=\\sum_{j} c_{j} x_{j}\\) and \\(k\\) is any complex number, then \\(k x=\\sum_{j} k c_{j} x_{j}\\), and \\(k A x=A k x=\\sum_{i}\\big(\\sum_{j} k a_{ij} c_{j}\\big) x_{i}\\), so that the matrix \\([kA]\\), corresponding to \\({kA}\\), has the elements \\({ka}_{ij}\\).

Similarly \\[ \\begin{aligned} (A+B)x &= A x+B x\\\\ &=\\sum_{i}\\Big(\\sum_{j} a_{ij} c_{j}\\Big) x_{i}+\\sum_{i}\\Big(\\sum_{j} b_{ij} c_{j}\\Big) x_{i} \\\\ &=\\sum_{i}\\Big(\\sum_{j}(a_{ij}+b_{ij}) c_{j}\\Big) x_{i}, \\end{aligned} \\] so that the matrix \\((A+B)\\) has elements \\((a_{i j}+b_{i j})\\).

Also \\[ \\begin{aligned} A B x&=A\\bigg(\\sum_{k}\\Big(\\sum_{j} b_{k j} c_{j}\\Big) x_{k}\\bigg)\\\\ &=\\sum_{i}\\Big(\\sum_{k} a_{i k} \\sum_{j} b_{k j} c_{j}\\Big) x_{i} \\\\ &=\\sum_{i}\\bigg(\\sum_{j}\\Big(\\sum_{k} a_{i k} b_{k j}\\Big) c_{j}\\bigg) x_{i}, \\end{aligned} \\] so that \\([A B]=\\big(\\sum_{k} a_{ik} b_{kj}\\big)\\).

Finally if \\(y=\\sum_{i} d_{i} x_{i}\\) then we have, (using Parseval’s identity, (I.11.5)),

\\[ \\begin{aligned} \\langle Ax, y \\rangle &= \\sum_i \\sum_j a_{ij} c_i \\bar{d}_j\\\\ &= \\sum_i c_i \\Big(\\sum_j a_{ij} \\bar{d}_j\\Big)\\\\ &= \\sum_i c_i \\overline{\\Big(\\sum_j b_{ji} d_j\\Big)}\\\\ &= \\langle x, By \\rangle, \\end{aligned} \\]

where \\(b_{ij}=\\bar{a}_{j i}\\).

We observe that for \\(A=0\\), \\(a_{ij}=0\\) and for \\(A=1\\), \\(a_{ij}=\\delta_{ij}\\).

A simple way of summing up the results of this section is the following. For a matrix \\([A]\\), (not for a linear transformation!) we define \\(k[A]=[B]\\) by \\(b_{ij}=k a_{ij}\\); and we define the conjugate transpose \\([C]=[{A}]^{*}\\) by \\(c_{ij}=\\bar{a}_{j i}\\). If moreover for any two matrices \\([A]\\) and \\([B]\\) we define their sum \\([C]=[A]+[B]\\) by \\(c_{ij}=a_{ij}+b_{ij}\\), and their product \\([D]=[A][B]\\) by \\(d_{ij}=\\sum_{k} a_{i k} b_{k j}\\), then our result is that the correspondence established by means of an arbitrary coordinate system between the set of all linear transformations of \\(\\mathcal{U}\\) and the set of square matrices of \\({n}\\) rows and \\({m}\\) columns is an isomorphism: i.e., it preserves addition, multiplication, and scalar multiplication, and makes the dual of a transformation correspond to the conjugate transpose of its matrix.

We return now to our general study of linear transformations without reference to any orthogonal basis. Periodically we shall stop to interpret our results in terms of the language and notation of matrices.

5. The forms \\(\\langle Ax, x\\rangle\\) and \\(\\langle Ax, y \\rangle\\)

With every linear transformation \\(A\\) We may associate the expressions \\(\\langle A x, x \\rangle\\) and \\(\\langle A x, y \\rangle\\). We may consider these as numerical valued functions, defined by means of \\(A\\), of a single vector \\(x\\) or of a pair of vectors \\(x\\) and \\(y\\), respectively. Properties of these functions are intimately connected with properties of the linear transformation \\(A\\): we shall study this subject in more detail later. At present we observe only two simple facts. First: if \\(\\langle A x, y \\rangle = 0\\) for all \\(x\\) and \\(y\\) then \\(A=0\\). For in particular we may choose \\(y=A x\\) and obtain \\(|A x|^{2}=0\\), whence \\(A x=0\\) for all \\(x\\). Second: if \\(\\langle A x, x \\rangle = 0\\) for all \\(x\\) then \\(A=0\\). The proof of this statement is less trivial: It depends on a standard technique called polarization. If \\(\\langle A x, x \\rangle = 0\\) for all \\(x\\) then for every pair of vectors \\(x\\) and \\(y\\) and every complex number \\(c\\) we have \\[ \\begin{aligned} 0&=\\big\\langle A(c x+y), (c x+y)\\big \\rangle\\\\ &=|c|^{2}\\langle A x, x \\rangle + \\langle A y, y \\rangle + c\\langle A x, y \\rangle + \\bar{c}\\langle A y, x \\rangle, \\end{aligned} \\] so that since the first two terms of the right member vanish we obtain \\[ c\\langle A x, y \\rangle +\\bar{c}\\langle A y, x \\rangle = 0. \\] Choosing first \\(c=1\\) and then \\(c=i=(-1)^{1 / 2}\\) we obtain the equations \\[ \\langle A x, y \\rangle + \\langle A y, x \\rangle = 0, \\quad i\\langle A x, y \\rangle - i\\langle A y, x \\rangle = 0. \\] Dividing the second equation by \\(i\\) and then forming the arithmetic mean of the two shows that \\(\\langle A x, y \\rangle = 0\\) for all \\(x\\) and \\(y\\), so that, by our first result, \\(A=0\\).

We observe that our second result is not true if we restrict ourselves to real spaces: the proof of course breaks down at our choice, \\(c=i\\). For example a \\(90^{\\circ}\\) rotation in the plane clearly has the property that it sends every vector \\(x\\) into a vector \\(Ax\\) which is orthogonal to \\(x\\).

6. Hermitian transformations

In many fundamental respects the algebraic system of all linear transformations on an inner product space \\(\\mathcal{U}\\) resembles the set of all complex numbers. In both systems the notions of addition, multiplication, \\(0\\) and \\(1\\) are defined and have similar properties; and, moreover, in both systems there is defined a conjugation (\\(A \\to A^{*}\\) and \\(z \\to\\bar{z}\\), respectively): i.e., an involutory conjugate automorphism of the system on itself. We shall use this analogy as a heuristic principle and we shall attempt to carry over to linear transformations some well known concepts of the complex domain.

When is a complex number real? Clearly a necessary and sufficient condition for the reality of \\(z\\) is that \\(z=\\bar{z}\\). We might accordingly (remembering that the analog of the complex conjugate for linear transformations is the dual) define a linear transformation to be real if \\(A=A^{*}\\). Actually linear transformations \\(A\\) for which \\(A=A^{*}\\) are called Hermitian. (Since the dual of a transformation is sometimes called its adjoint, Hermitian transformations are sometimes called self-adjoint. Other terms are symmetric and Hermitian symmetric. The reason for the latter terminology will appear later when we examine the corresponding matricial concept.) The following theorem shows that Hermitian transformations are tied up with reality in more ways than through the formal analogy that suggested their definition.

Theorem 1. A necessary and sufficient condition that \\(A\\) be Hermitian is that \\(\\langle A x, x \\rangle\\) be real for all \\(x\\).

Proof. For if \\(A=A^{*}\\), then \\[ \\langle A x, x \\rangle = \\langle x, A^{*} x \\rangle = \\langle x, Ax \\rangle = \\overline{\\langle Ax, x \\rangle}, \\] so that \\(\\langle A x, x \\rangle\\) is equal to its own conjugate and is therefore real. Conversely, if \\(\\langle A x, x \\rangle\\) is always real then \\[ \\langle A x, x \\rangle = \\overline{\\langle Ax, x \\rangle }=\\overline{\\langle x, A^{*} x \\rangle} = \\langle A^{*} x, x \\rangle, \\] so that \\(\\big\\langle (A-A^{*})x, x \\big\\rangle =0\\) for all \\(x\\), whence, by (II.5), \\(A - A^{*}=0\\).■

We remark that this theorem is also false in real spaces. For in the first place its proof depends on a lemma that is valid only in complex inner product spaces, and in the second place in a real space the reality of \\(\\langle Ax, x \\rangle\\) (in fact of \\(\\langle Ax, y \\rangle\\)) is a condition automatically satisfied by all \\(A\\), whereas the condition \\(A=A^{*}\\), or equivalently \\(\\langle A x, y \\rangle = \\langle x, A y \\rangle\\), need not be satisfied. It is not difficult to verify that the example given in (II.5) is a counter example to this theorem in real spaces.

7. Positive definite transformations

When is a complex number non-negative? Two equally natural necessary and sufficient conditions are that \\(z\\) may be written in the form \\(z=u^{2}\\) for some real \\(u\\) or that \\(z\\) may be written in the form \\(z=v \\bar{v}\\) for an arbitrary \\(v\\). Remembering also the fact that the Hermitian property of a transformation \\(A\\) can be described in terms of the function \\(\\langle A x, x \\rangle\\) we may consider any one of three conditions and attempt to use them as the definition of a transformation being non-negative.

7.1. \\(A = B^2\\), \\(B\\) Hermitian,

7.2. \\(A = C^* C\\), \\(C\\) arbitrary,

7.3. \\(\\langle Ax, x \\rangle \\geq 0\\) for all \\(x\\).

Before deciding on which one of these three conditions to use as definition we prove that the following implication relations hold: \\[ (7.1) \\implies (7.2) \\implies (7.3). \\]

For if \\(A=B^{2}\\) with a Hermitian \\(B\\), i.e., with a \\(B\\) for which \\(B^{*}=B\\), then \\(A=B B=B^{*} B\\). And if \\(A=C^{*} C\\) then \\[ \\langle A x, x \\rangle = \\langle C^{*} C x, x \\rangle = \\langle C x, C x \\rangle = |C x|^{2} \\geq 0. \\]

It is actually true that (7.3) implies (7.1), so that the three conditions are equivalent, but we shall not be able to prove this till much later. We adopt as our definition the third condition: a linear transformation \\(A\\) is non-negative, in symbols \\(A\\geq 0\\), if for all \\(x\\), \\(\\langle A x, x \\rangle \\geq 0\\).

(Non-negative transformations are usually called positive semidefinite. If \\(A\\geq 0\\) and \\(\\langle A x, x \\rangle = 0\\) implies \\(x=0\\) we call \\(A\\) positive definite.) It follows from the theorem in (II.6) that \\(A\\geq 0\\) implies that \\(A\\) is Hermitian. The transformations \\(0\\) and \\(1\\) are non-negative.

8. Algebraic combinations of Hermitian and definite transformation

We discuss the relation of the two concepts just defined to our preceding notions. If \\(A\\) and \\(B\\) are both Hermitian then, since \\((A+B)^{*}=A^{*}+B^{*}\\), \\(A+B\\) is also. If \\(A\\) is Hermitian then \\(cA\\) is Hermitian if and only if \\(c\\) is real. This follows immediately from the fact that \\((c A)^{*}=\\bar{c} A^{*}\\). If \\(A\\) and \\(B\\) are both Hermitian then \\(A B\\) is Hermitian if and only if \\(A B=B A\\). This is a consequence of the relations \\((A B)^{*}=B^*{A}^{*}=B A\\). Since \\((A^{-1})^{*}=(A^{*})^{-1}\\), \\(A^{-1}\\) is Hermitian if and only if \\(A\\) is. Similarly the relation \\((B^*{A B})^{*}=B^{*} A^{*} B\\) implies that \\(B^{*} AB\\) is Hermitian along with \\(A\\). A converse to the last statement is that if \\(B\\) has an inverse and \\(B^{*} A B\\) is Hermitian then so is \\(A\\). (We remark that both the direct and converse statements are valid for not necessarily Hermitian \\(B\\).) For if \\(B\\) has an inverse, every vector \\(x\\) may be written in the form \\(x=B y\\), and since \\(\\langle A x, x \\rangle = \\langle A B y, B y \\rangle = \\langle B^{*} A B y, y \\rangle\\), the reality of the last term for all \\(y\\) implies the reality of the first for all \\(x\\).

The formulas used in the proofs in the preceding paragraph prove also that \\(A+B\\) is non-negative if both \\(A\\) and \\(B\\) are, that if \\(A\\geq 0\\) then \\(cA\\) is non-negative if and only if \\({c}\\) is, that \\({B}^{*} {AB}\\) is non-negative along with \\(A\\), and that, conversely, if \\(B\\) has an inverse then the non-negativeness of \\(B^{*} AB\\) implies that of \\(A\\). It is true also that if \\(A\\) and \\(B\\) are non-negative and commutative then \\(AB\\) is non-negative, but we shall have to postpone the proof of this statement until later. Neither this statement nor the one concerning Hermitian transformations is true without the restriction of commutativity.

9. Matricial characterizations of Hermitian transformations

If \\(\\{x_{i}\\}\\) is any coordinate system in \\(\\mathcal{U}\\) then a necessary and sufficient condition that a linear transformation \\(A\\) be Hermitian is that the matrix \\([A]\\) corresponding to \\(A\\) in this coordinate system satisfy the equation \\([A]=[A]^{*}\\), or, in other words, that we have \\(a_{ij} = \\bar{a}_{ji}\\) for all \\(i\\) and \\(j\\). This explains, incidentally, why Hermitian transformations are sometimes called Hermitian symmetric. A similar matricial characterization of non-negative matrices is possible, but the conditions on the \\(a_{ij}\\) are much more complicated and since we shall not have any occasion to use them we do not enter on this subject here. We shall, however, refer to matrices (not linear transformations!) as Hermitian (or non-negative) if their associated linear transformations are Hermitian (or non-negative).

10. Unitary transformations

When does a complex number \\(z\\) have absolute value \\(1\\)? Clearly \\(\\bar{z}=z^{-1}\\) is a necessary and sufficient condition: guided by our heuristic principle we are led to consider linear transformations \\(U\\) for which \\(U^{*}=U^{-1}\\). Such transformations are called unitary. Concerning unitary transformations we prove the following two theorems,

Theorem 2. 10.1. A necessary and sufficient condition that \\(U\\) be unitary is that \\(\\langle Ux, Uy \\rangle = \\langle x, y \\rangle\\) for all \\(x\\) and \\(y\\).

10.2. A necessary and sufficient condition that \\(U\\) be unitary is that \\(|U x|=|x|\\) for all \\(x\\).

Proof. Since the condition of (10.1) is obviously stronger than that of (10.2), (i.e., if \\(\\langle Ux, Uy \\rangle = \\langle x, y \\rangle\\) then \\(\\langle Ux, Ux \\rangle = \\langle x, x \\rangle\\)) it will be sufficient to prove that the condition of (10.1) is necessary and that the condition of (10.2) is sufficient. If \\(|U x|=|x|\\) for all \\(x\\), then \\[ \\langle x, x \\rangle = \\langle U x, U x \\rangle = \\langle U^{*} U x, x \\rangle, \\] so that \\(\\big\\langle (U^*U - 1) x, x \\big\\rangle = 0\\) for all \\(x\\). It follows that \\(U^*U = 1\\), so that \\(U^{*}=U^{-1}\\). Conversely if \\(U\\) is unitary, so that \\(U^*U = 1\\), then for all \\(x\\) and \\(y\\), \\[ \\langle x, y \\rangle = \\langle U^{*} U x, y \\rangle = \\langle Ux, Uy \\rangle. \\]■

11. Automorphisms of inner product spaces

In any algebraic system, in particular in vector spaces and in inner product spaces, it is of interest to consider the automorphisms of the systems: i.e., to consider those one to one mappings of the system on itself which preserve all relations between elements of the system. The most general automorphism of a vector space is a one to one transformation that preserves addition and scalar multiplication: in other words it is an arbitrary linear transformation which has an Inverse. Of an automorphism \\(U\\) of an inner product space we should also require that it preserve inner product: i.e., that we have \\(\\langle Ux, Uy \\rangle = \\langle {x}, {y} \\rangle\\). But this, as we have seen, is equivalent to the requirement that \\(U\\) be unitary. Thus the two questions — ‘What linear transformations are the analogs of complex numbers of absolute value one?’ and ‘What are the most general automorphisms of inner product space?’ — have the same answer: unitary transformations. In tha next paragraph we shall see that unitary transformations furnish the answer to a third question also: ‘What happens to the matrix \\([A]\\) of the linear transformation \\(A\\) when we change coordinate systems?’

12. Change of basis in an inner product space

We start with the comment that a necessary and sufficient condition that a linear transformation \\(U\\) be unitary is that whenever \\(\\{x_{i}\\}\\) is a complete orthonormal set then so is \\(\\{U x_{i}\\}\\). For the condition is merely the statement that \\(\\langle Ux, Uy \\rangle = \\langle x, y \\rangle\\) for \\(x\\) and \\(y\\) lying in complete orthonormal set, and by linearity the condition extends to all \\(x\\) and \\(y\\). Suppose then that \\(\\{x_i^\\prime\\}\\) and \\(\\{x_i^{\\prime\\prime}\\}\\) are two coordinate systems (i.e., complete orthonormal sets). If \\(A\\) is an arbitrary linear transformation then throughout this paragraph we shall denote by \\([A]^\\prime\\) (or \\([A]^{\\prime\\prime}\\)) the matrix of \\(A\\) in the coordinate system \\(\\{x_i^\\prime\\}\\) (or \\(\\{x_{i}^{\\prime\\prime}\\}\\)). The matrices \\([A]^\\prime\\) and \\([A]^{\\prime \\prime}\\) are characterized, respectively, by the following two equations: \\[ Ax_j^\\prime = \\sum_i a_{ij}^\\prime x_i^\\prime, \\quad Ax_j^{\\prime \\prime}= \\sum_i a_{ij}^{\\prime \\prime} x_i^{\\prime \\prime}. \\]

Let us denote by \\(U\\) the linear transformation defined by the relations \\(Ux_i^\\prime = x_i^{\\prime\\prime}\\), and generally, \\(U\\big(\\sum_i c_i x_i^\\prime\\big) = \\sum_i c_i x_i^{\\prime\\prime}\\). It follows from the comment made as the beginning of this paragraph that \\(U\\) is unitary, and the second of tho two equations above implies that \\(AU^{-1} x_j^\\prime = U^{-1}\\big(\\sum_i a_{ij}^{\\prime\\prime} x_i^\\prime\\big)\\), or, in other words, that \\[ UAU^{-1} x_j^\\prime = \\sum_i a_{ij}^{\\prime\\prime} x_i^\\prime. \\]

But this means exactly that \\([UAU^{-1}]^\\prime = [A]^{\\prime\\prime}\\). Summing up: if \\([A]^\\prime\\) and \\([A]^{\\prime\\prime}\\) are two matrices corresponding to the same linear transformation in different coordinate systems, then there exists a unitary transformation \\(U\\) such that \\([U A U^{-1}] =[A]^{\\prime\\prime}\\), or, equivalently, there exists a unitary matrix \\([U]^\\prime\\) such that \\([U]^\\prime [A]^\\prime [U^{-1}]^\\prime = [A]^{\\prime\\prime}\\).

13. Matricial characterization of unitary transformations

If \\(U\\) is a unitary transformation and \\(\\{x_{i}\\}\\) a coordinate system in which the matrix of \\(U\\) is \\([U]\\), then it follows from the multiplication rule for matrices and the equation \\(UU^* = 1\\), that \\(\\sum_{k} u_{i k} \\bar{u}_{j k}=\\delta_{ij}\\). Clearly this condition characterizes \\([U]\\): i.e., it is necessary and sufficient in order that \\([U]\\) be unitary. We terminate, temporarily, our discussion of unitary transformations, and turn to the discussion of another special class of linear transformations that will be of great interest to us.

14. Orthogonal projections in an inner product space

We have seen (in I.13) that if \\(M\\) is a linear subspace then every vector \\(x\\) may be written, uniquely, in the form \\(x=y+z\\) with \\(y\\) in \\(M\\) and \\(z\\) in \\(M^{\\perp}\\). \\(y\\) is called the projection of \\(x\\) on \\(M\\). We consider the correspondence \\(P_{M}\\) which assigns to every vector \\(x\\) its projection \\(y=P_{M}{x}\\) on \\(M\\): \\(P_{M}\\) is called a projection transformation or simply a projection. On occasions when it is not necessary to denote the dependence of the projection on the linear subspace in terms of which it was defined we shall use the letters \\(E, F, \\ldots\\) for projections. We prove first that \\(E=P_{M}\\) is a linear transformation. For if \\(x_{i}=y_{i}+z_{i}\\); \\(i=1, 2\\), with \\(y_{i}\\) in \\(M\\) and \\(z_{i}\\) in \\(M^{\\perp}\\), then \\[ a_{1} x_{1}+a_{2} x_{2}=(a_{1} y_{1}+a_{2} y_{2})+(a_{1} z_{1}+a_{2} z_{2}), \\] and from the fact that \\(M\\) and \\(M^{\\perp}\\) are linear subspaces it follows therefore that \\[ E(a_{1} x_{1}+a_{2} x_{2})=a_{1} y_{1}+a_{2} y_{2}=a_{1} Ey_{1}+a_{2} E y_{2}. \\]

The following theorem gives a complete algebraic characterization of projections.

Theorem 3. If \\(E\\) is a projection then \\(E=E^{2}=E^{*}\\); conversely if \\(E\\) is a linear transformation for which \\(E=E^{2}=E^{*}\\) then \\(E=P_{M}\\) where \\(M\\) is the linear subspace of all vectors of the form \\(Ex\\).

Proof. If \\({E}={P}_M\\) and \\(x=y+z\\) with \\(y\\) in \\(M\\) and \\(z\\) in \\(M^{\\perp}\\), then \\(Ex=y\\). Since \\(y\\) has the representation \\(y=y+0\\), with \\(y\\) in \\(M\\) and \\(0\\) in \\(M^{\\perp}\\), and since this representation is unique, it follows that \\({Ey}={y}\\), whence \\({EEx} = {E}^{2} {x}={Ex}\\) for all \\({x}\\). To prove that \\(E={E}^{*}\\), let \\(x_{1}\\) and \\(x_{2}\\) be any two vectors and denote their projections on \\(M\\) (or on \\(M^{\\perp}\\)) by \\(y_{1}\\) and \\(y_{2}\\) (or \\(z_{1}\\) and \\(z_{2}\\)) respectively. Then \\[ \\begin{aligned} \\langle E x_{1}, x_{2} \\rangle &= \\langle y_{1}, x_{2} \\rangle\\\\ &= \\langle y_{1}, y_{2}+z_{2} \\rangle\\\\ &= \\langle y_{1}, y_{2} \\rangle\\\\ &= \\langle y_{1}+z_{1}, y_{2} \\rangle\\\\ &= \\langle x_{1}, E x_{2} \\rangle; \\end{aligned} \\] and this implies that \\({E}={E}^{*}\\).

Conversely suppose that \\({E}={E}^{2}={E}^{*}\\), and let \\(M\\) be the linear subspace of all vectors of the form \\(Ex\\). Since for any \\(x\\) we have \\(x=E x+(1-E) x\\) with \\(Ex\\) in \\(M\\), the proof of the theorem will be complete when we succeed in proving that for every \\(x\\), \\((1-{E}) x\\) is in \\(M^{\\perp}\\). But this follows from the equations \\[ \\begin{aligned} \\big\\langle E x,(1-E) y\\big\\rangle &= \\langle E x, y \\rangle - \\langle E x, E y \\rangle\\\\ &= \\langle E x, y \\rangle - \\langle E^{*} E x, y \\rangle \\\\ &= \\langle E x, y \\rangle - \\langle E^{2} x, y \\rangle\\\\ &= \\langle E x, y \\rangle - \\langle E x, y \\rangle\\\\ &= 0 \\end{aligned} \\] (We remark that the proof shows incidentally that \\(M^{\\perp}\\) consists precisely of all vectors of the form \\((1 - E)x\\).)■

In different words this theorem states that the characteristic properties of a projection are that it is Hermitian \\((E={E}^{*})\\) and idempotent \\((E=E^{2})\\). As a corollary of this theorem we obtain the fact that if \\(P_{M}=E\\) then \\(P_{M^{\\perp}}=1-E\\). Incidentally this theorem establishes a one to one correspondence between the class of all projections (idempotent and Hermitian transformations) and the class of all linear subspaces. In the following paragraphs we shall investigate this correspondence more closely and obtain conditions in order that certain algebraic combinations of projection transformations be themselves projections.

15. Products of projections

Theorem 4. A necessary and sufficient condition that the product \\(EF\\) of two projections, \\(E=P_{M}\\) and \\(F=P_{N}\\), be a projection is that \\(EF=FE\\); if this commutativity condition is satisfied then \\(EF = FE = P_{M \\cap N}\\).

Proof. If \\(EF\\) is a projection, along with \\(E\\) and \\(F\\), then \\[ EF = (EF)^{*} = F^{*} E^{*}={FE}, \\] so that \\(E\\) and \\(F\\) commute. (In fact we have already seen in (II.8) that if \\(A\\) and \\(B\\) are Hermitian then \\(A B\\) is Hermitian if and only if \\(A\\) and \\(B\\) commute.) Conversely if \\(EF = FE\\) then \\[ (EF)^2 = EFEF = EEFF = E^2 F^2 = EF \\] and \\[ ({EF})^{*}={F}^{*} {E}^{*}={FE}={EF}, \\] so that \\({EF}\\) is Hermitian and idempotent. Finally suppose that \\({E F}=P_{K}\\); we know from (II.14) that \\(K\\) is the set of all vectors of the form \\(EFx = FEx\\). Then every vector in \\({K}\\) is simultaneously of the form \\(Ey\\) and \\(Fz\\), so that \\(K\\) is contained in the intersection \\(M \\cap N\\). On the other hand if \\(x\\) is any vector in \\({M \\cap N}\\) then \\(F x=x\\) and \\(E x=x\\), whence \\(E F x=Ex=x\\), so that \\(x\\) is contained in \\(K\\). This proves that \\(K={M \\cap N}\\).■

16. Sums of projections

Theorem 5. A necessary and sufficient condition that the sum \\(E+F\\) of two projections, \\({E}=P_{M}\\) and \\(F=P_{N}\\), be a projection is that \\(E F=F E=0\\); if this condition is satisfied then \\(M\\) and \\(N\\) are orthogonal and \\(E+F=P_{M+N}\\).

Proof. If \\(E+F\\) is a projection then \\[ E+F=(E+F)^{2}=E+{EF}+{FE}+{F}, \\] whence \\({EF}+{FE}=0\\). Multiplying this equation first on the right and then on the left by \\(E\\) we obtain the equations \\[ {E FE}+{FE}=0, \\quad E F+E F E=0. \\] Upon subtraction it follows that \\({E F} - FE = 0\\), and this, combined with our original equation, yields \\(E F=F E=0\\). Conversely, if \\(EF = FE = 0\\), then

\\[ (E+F)^{2} = E + EF + FE + F = E + F, \\] so that \\(E+F\\) is idempotent; being the sum of two Hermitian transformations it is also Hermitian and therefore it is a projection. Finally suppose that \\(E+F=P_{K}\\). Since \\(P_{K}x=Ex+Fx\\) and since \\(E x\\) is in \\(M\\) and \\(F x\\) is in \\(N\\), it follows that \\(K\\) is contained in \\(M+N\\). On the other hand if \\(x\\) is any vector in \\(M+N\\) then \\(x\\) has the form \\(x=E y+F z\\), whence \\[ P_{K}x={E x}+{Fx}={Ey}+{EFz}+{FEy}+{Fz}={Ey}+{Fz}={x}, \\] (since \\({E F}=FE=0\\)), so that every \\(x\\) in \\(M+N\\) has the form \\(P_{K} x\\) and is therefore in \\(K\\). This proves that \\(K=M+N\\). In order to show that \\(M\\) and \\(N\\) are orthogonal we must show that \\(x\\) in \\(M\\) and \\(y\\) in \\(N\\) implies \\(\\langle x, y \\rangle = 0\\). This follows from the equations \\[ \\begin{aligned} \\langle x, y \\rangle &= \\langle E x, F y \\rangle\\\\ &= \\langle F^{*} {Ex}, y \\rangle\\\\ &= \\langle F E x, y \\rangle\\\\ &= \\langle 0, y \\rangle\\\\ &= 0. \\end{aligned} \\]■

17. Differences of projections

Theorem 6. A necessary and sufficient condition that the difference \\(E - F\\) of two projections, \\(E=P_{M}\\) and \\(F=P_{N}\\), be a projection is that \\(EF = FE = F\\); if this condition is satisfied then \\(N\\) is contained in \\(M\\) and \\(E - F = P_{M \\cap N^\\perp}\\).

Proof. A necessary and sufficient condition in order that \\(E - F\\) be a projection is that \\(1-(E-F)\\) be one: i.e., that \\((1-{E})+F\\) be a projection. According to (II.16) above, this is equivalent to \\((1-{E}) F=F(1-E)=0\\). If moreover, this condition is satisfied and if \\(E-F=P_{K}\\) then we know, still from (II.16) that \\(M^{\\perp}\\) and \\(N\\) are orthogonal, whence \\(M^{\\perp}\\) is contained in \\(N^\\perp\\), or, equivalently, \\(N\\) is contained in \\(M\\), and finally \\[ K=(M^\\perp \\cap N\\big)^\\perp = M \\cap N^\\perp. \\]■

18. Relation between projections and involutions

We conclude our discussion of projections by indicating their relation to certain other classes of transformations. We shall call a linear transformation \\(A\\) for which \\(A^{2}=1\\) an involution, or an involutory transformation. We assert now that if a linear transformation \\(U\\) has any two of the three properties—Hermitian, involutory, unitary—then it has the third.

(i). If \\(U=U^{*}\\) and \\(U^{2}=1\\) then \\(UU^{*}=1\\), so that \\(U^{*}=U^{-1}\\).

(ii). If \\(U=U^{*}\\) and \\(U^{*}=U^{-1}\\) then \\(U=U^{-1}\\) whence \\(U^{2}=1\\).

(iii). If \\(U^{*}=U^{-1}\\) and \\(U^{2}=1\\), then \\(U=U^{-1}=U^{*}\\).

Transformations having these three properties are related to projections through the following theorem.

Theorem 7. If two transformations \\(U\\) and \\(E\\) are related by the two (equivalent) conditions \\[ U=2E - 1, \\quad E=(1 / 2)(U+1), \\] then a necessary and sufficient condition that \\(E\\) be a projection is that \\(U\\) be Hermitian, unitary, and involutory.

Proof. If \\(E\\) is a projection then, since \\({E}^{*}=E\\) we must also have \\(U^{*}=U\\), and from \\(E^{2}=E\\) it follows that \\[ \\begin{aligned} U^{2}&=4 E^{2}-2E+1\\\\ &=4 E-2 E+1\\\\ &=2 E+1\\\\ &=U. \\end{aligned} \\] Conversely if \\(U\\) is Hermitian then \\(E\\) is, and if \\(U^{2}=1\\) then \\[ \\begin{aligned} E^{2}&=(1 / 4)(U^{2}+2 U+1)\\\\ &=(1 / 4)(2 U+2)\\\\ &=E. \\end{aligned} \\]■

19. The rank of a linear transformation

We conclude this chapter by a discussion of two important numerical invariants of linear transformations: the rank and the norm.

If \\(A\\) is any linear transformation we define two linear subspaces \\(M\\) and \\(N\\) as follows: \\(M\\) is the set of all vectors of the form \\(A x\\), and \\({N}\\) is the set of all vectors \\(x\\) for which \\(A x=0\\). Let \\(n\\) be the dimension of the vector space \\(\\mathcal{V}\\) under consideration, and let \\(r\\) and \\(s\\) be the dimensions of \\(M\\) and \\(N\\), respectively. We shall show that \\(r=n-s\\). The non-negative integer \\(r\\) is the rank of the linear transformation \\(A\\). Let \\(x_{1}, x_{2}, \\ldots, x_{s}\\) be a linear basis in \\(N\\) (which is of course not necessarily a coordinate system in the sense of this chapter, but merely a maximal linearly independent set in \\(N\\)), and let the vectors \\(x_{1}, x_{2}, \\ldots, x_{s}, x_{s+1}, \\ldots, x_{n}\\) be a linear basis in \\(\\mathcal{V}\\). Then every vector \\(x\\) in \\(\\mathcal{V}\\) may be written (uniquely) in the form \\(x=\\sum_{i} a_{i} x_{i}\\) and it follows that \\[ A x=\\sum_{i} a_{i} A x_{i}=\\sum_{i=s+1}^{n} a_{i} A x_{i}. \\]

In other words every vector of the form \\(Ax\\) is a linear combination of the \\(n - s\\) vectors \\(A x_{s+1}, \\ldots, A x_{n}\\), so that \\(r \\leq n-s\\). We shall prove that these \\(n-s\\) vectors are linearly independent, thereby proving that \\(r=n-s\\). If we had \\[ 0=\\sum_{i=s+1}^{n} c_{i} A x_{i}=A\\Big(\\sum_{i=s+1}^{n} c_{i} x_{i}\\Big) \\] then we should have \\(\\sum_{i=s+1}^{n} c_{i} x_{i}\\) belonging to \\(M\\). accordingly this vector would have an expression of the form \\(\\sum_{i=s+1}^{n} c_{i} x_{i}=-\\sum_{i=1}^{s} c_{i} x_{i}\\), so that \\(\\sum_{i=1}^{n} c_{i} x_{i}=0\\). The linear independence of the \\(x_{i}\\), \\(i = 1, 2, \\ldots,n\\), implies then that \\(c_{i}=0\\).

If for the moment we denote the dependence of \\(M\\) and \\(N\\) on \\(A\\) by writing \\(M=M_{A}\\) and \\(N=N_{A}\\), and if \\(B\\) is an arbitrary linear transformation which has an inverse, then it is very easy to characterize \\(N_{A B}\\) and \\(N_{B A}\\). For \\(A B x=0\\) if and only if \\(B x\\) lies in \\(N_{A}\\) and \\(B A x=0\\) if and only if \\(A x=0\\), i.e., if and only if \\(x\\) lies in \\(N_{A}\\). Since the image of \\(N_{A}\\) under \\(B^{-1}\\) has the same dimension as \\(N_{A}\\) it follows in particular that \\(A B\\) and \\(B A\\) have the same rank as \\(A\\). If \\(A\\) is a linear transformation in an inner product space \\(\\mathcal{U}\\) and if \\(x\\) is any vector in \\(N_{A}\\) then \\(0 = \\langle A x, y \\rangle = \\langle x, A^{*} y \\rangle\\) for all \\(y\\). In other words every vector of the form \\(A^{*} y\\) is orthogonal to \\(x\\), whence \\(M_{A^{*}} \\perp {N}_{A}\\). It follows that \\(M_{A^{*}}\\) is contained in \\(N_{A}^\\perp\\), whence, denoting the rank of \\(A^{*}\\) by \\(r^{*}\\), \\(r^{*} \\leq r\\). Since this is generally true we may apply this result to \\(A^{*}\\) obtaining \\(r \\leq r^{*}\\), so that \\(r=r^{*}\\). Finally we notice that if \\(A_{1}\\) and \\(A_{2}\\) are arbitrary linear transformations and \\(r_{1}\\) and \\(r_{2}\\) their ranks then, since \\({N}_{A_{2}}\\) is contained in \\({N}_{A_{1} A_{2}}\\), it follows that the rank \\(r\\) of \\({A_{1} A_{2}}\\) is \\(\\leq r_{2}\\). Applying our previous result on duals and denoting the rank of \\((A_{1} A_{2})^{*}=A_{2}^{*} A_{1}^{*}\\) by \\(r^{*}\\), we obtain \\(r=r^{*} \\leq r_{1}^{*}=r_{1}\\), and we obtain the result that the rank of a product of two transformations does not exceed the rank of either factor. (This is Sylvester’s law of nullity: the terminology arises from the fact that if \\(A\\) is a transformation of rank \\(r\\), \\(s=n-r\\) is called the nullity of \\(A\\).)

We observe that in virtue of (II.14) the rank of a projection \\(P_{M}\\) is the dimension of the linear subspace \\(M\\).

20. The norm of a linear transformation

In order to define the norm of a linear transformation \\(A\\) defined in an inner product space \\(\\mathcal{U}\\) of dimension \\(n\\) we prove first the following theorem.

Theorem 8. There exists a constant \\(K\\) such that for all \\(x\\), \\(|A x| \\leq K|x|\\).

Proof. Let \\(\\{x_{i}\\}\\) be an orthogonal basis in \\(\\mathcal{U}\\), and choose a constant \\(K_{0}\\) so that \\(|A x_{i}|<K_{0}\\) for all \\(i\\). Since an arbitrary vector \\(x\\) may be written in the form \\(x=\\sum_{i} c_{i} x_{i}\\), we have \\[ \\begin{aligned} |A x| &= \\bigg|A\\Big(\\sum_{i} c_{i} x_{i}\\Big)\\bigg|\\\\ &= \\bigg|\\sum_{i} c_{i} A x_{i}\\bigg|\\\\ &\\leq \\sum_{i}|c_{i}||A x_{i}|\\\\ &\\leq K_{0} \\sum_{i}|c_{i}| \\end{aligned} \\] whence it follows, remembering that \\(c_{i} = \\langle x, x_{i} \\rangle\\) and applying the Schwartz inequality, that \\[ |A x| \\leq K_{0} \\sum_{i}|x||x_{i}|=K_{0} \\sum_{i}|x|=n K_{0}|x|, \\] so that we may choose \\({K}={n}{K}_{0}\\).■

The least number \\({K}\\) with the property described in the theorem is called the norm of \\(A\\): more formally the norm of \\(A\\) can be defined as \\(|A|=\\sup (|Ax| / |x|)\\), with the supremum taken over all vectors \\(x \\neq 0\\).

21. Expressions for the norm of a transformation

Along with the norm \\(|A|\\) of the transformation \\(A\\) we may consider the following three constants: \\[ \\begin{aligned} & p=\\sup |A x| \\text { for } |x|=1, \\\\ & q=\\sup \\big(|\\langle A x, y \\rangle| /|x||y|\\big) \\text { for } x, y \\neq 0, \\\\ & r=\\sup |\\langle A x, y \\rangle| \\text { for } |x|=|y|=1. \\end{aligned} \\]

We shall prove that \\({p}={q}={r}=|{A}|\\). We first remark, however, that since \\[ |\\langle A x, y \\rangle| \\leq |A x||y| \\leq |A||x||y|, \\] the expressions defining \\(q\\) and \\(r\\) are bounded, and the suprema \\(q\\) and \\(r\\) are finite. It follows from this comment, and from the definitions of \\(p\\), \\(q\\), and \\(r\\), that \\(r \\leq q \\leq |A|\\), and we also have \\({p} \\leq |A|\\). Accordingly we will have proved the equality of all four constants involved if we succeed in proving that \\(p \\geq |A|\\), \\(q \\geq |A|\\), and \\(r \\geq q\\).

Since for any \\(x \\neq 0\\), \\(|A x| /|x|=|{A y}|\\), where \\(y=x /|x|\\), and since \\(|y|=1\\), it follows that \\(|A x| /|x| \\leq p\\), whence the supremum, \\(|A|\\), of the expression on the left is also \\(\\leq {p}\\).

If for any vector \\({x}\\) for which neither \\({x}=0\\) nor \\(A x=0\\) we write \\(y=A x\\), we obtain the equation \\[ |A x| /|x| = |\\langle A x, y \\rangle| /|x||y| \\leq q. \\] It follows that the supremum, \\(|A|\\), of the expression on the is \\(\\leq {q}\\).

If for any pair \\(x\\), \\(y\\) of non-vanishing vectors we define \\(x^{\\prime}=x /|x|\\) and \\(y^{\\prime}=y /|y|\\), then we obtain, since \\(|x|=|y^{\\prime}|=1\\), \\[ |\\langle A x, y \\rangle| /|x||y|=|\\langle A x^{\\prime}, y^{\\prime} \\rangle| \\leq r, \\] so that the supremum, \\(q\\), of the expression on the left is also \\(\\leq r\\).

The equation \\(\\langle A x, y \\rangle = \\langle x, A^{*} y \\rangle\\) implies immediately that \\(|A|=|A^{*}|\\). It is easy to verify that if \\(U\\) is unitary \\(|U|=1\\), and that if \\(E\\) is a projection, \\(E \\neq 0\\), \\(|E|=1\\).

22. Upper and lower bounds of a Hermitian transformation

For Hermitian transformations \\(A\\) we can find still another interesting expression for the norm, \\(|A|\\). If \\(c\\) is an arbitrary real number we shall write \\(A \\geq c\\) if the transformation \\(A - c1\\) is non-negative (Cf. II.7), and \\(A \\leq c\\) if \\(c1-A\\) is non-negative. In this sense we may write \\(-|A| \\leq A \\leq |A|\\), and we may define the upper bound, \\(\\beta\\), of the Hermitian transformation \\(A\\), to be the least (i.e., the infimum) of the numbers \\(c\\) for which \\(A \\leq c\\); similarly we define the lower bound, \\(\\alpha\\), of \\(A\\) to be the greatest (i.e., the supremum) of the numbers \\(c\\) for which \\(c \\leq A\\). In other words, remembering the definition of non-negative transformations, \\(\\alpha\\) and \\(\\beta\\) are, respectively, the infimum and the supremum of the set of all real numbers of the form \\(\\langle A x, x \\rangle /|x|^{2}\\). Our main result is that the upper and lower bounds of \\(A\\) are related to the norm, \\(|A|\\), of \\(A\\) by the relation \\[ |A|=\\max (|\\alpha|,|\\beta|)=k. \\] Since we have already observed that \\(-|A| \\leq A \\leq |A|\\), it follows that \\(-|A| \\leq \\alpha \\leq \\beta \\leq |A|\\), so that \\(k \\leq |A|\\). From the definition of \\(k\\) it follows that \\(-{k} \\leq A \\leq k\\), or, in other words, that the transformations \\(k1 - A\\) and \\(k1 + A\\) are non-negative. It follows from (II.8) that the transformations \\[ (k1 + A)^*(k1 - A)(k1 + A) = (k1 + A)(k1 - A)(k1 + A) \\] and \\[ (k1 - A)^*(k1 + A)(k1 - A) = (k1 - A)(k1 + A)(k1 - A) \\] are also non-negative. Hence so also is their sum, \\((2 k)(k^{2} 1-A^{2})\\). Since \\(k=0\\) implies \\(|A|=0\\), the theorem is trivial in this case; in any other case \\(k\\) is positive and we obtain therefore the result that \\((k^{2} 1-A^{2})\\) is a non-negative linear transformation. In other words \\[ \\begin{aligned} k^{2} |x|^{2}&=k^{2} \\langle x, x \\rangle\\\\ &> \\langle A^{2} x, x \\rangle\\\\ &= \\langle A x, A x \\rangle\\\\ &=|Ax|^{2}, \\end{aligned} \\] whence \\(k>|A|\\), and the proof of the theorem is complete. Since it is easy to prove, by the methods of the preceding paragraph, that \\[ \\sup |\\langle A x, x \\rangle| /|x|^{2}=\\sup |\\langle A x, x \\rangle|, \\] (where the first supremum is extended over all \\(x \\neq 0\\) and the second one over all \\(x\\) with \\(|x|=1\\)), our result is equivalent to the statement that for Hermitian transformations \\(A\\), \\(|A|=\\sup |\\langle A x, x \\rangle|\\) for \\(|x|=1\\).

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"linear-transformations-and-matrices","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]},{"_id":"66a048a539e7e3d9dc7ad831","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Spectral Theory of Normal Transformations","sequence":3,"chapter_content":"

1. Determinants

Throughout this chapter we shall work with a fixed inner product space \\(\\mathcal{U}\\) of dimension \\(n\\) in which an inner product \\(\\langle x, y \\rangle\\) is defined. Before starting on the subject of the title we make a comment on determinants. We assume as known the definition of the determinant of a matrix and its elementary properties. If \\(A\\) is an arbitrary linear transformation and \\([A]\\) its matrix in any coordinate system then its matrix in any other coordinate system is \\([U][A][U]^{-1}\\), where \\([U]\\) is a unitary matrix. Hence, if we denote by \\(D[A]\\) the determinant of \\([A]\\), \\(D[A]\\) is independent of the coordinate system chosen and depends only on the linear transformation \\(A\\). It is therefore proper to speak of the determinant of a linear transformation \\(A\\): we shall denote it by \\(D({A})\\). In matricial language we shall make use of the following properties of determinants.

The Laplace expansion. In particular we shall use the fact that if in the first row of a matrix \\([A]\\) only the first element is different from zero then the determinant of \\([A]\\) is the product of this first element and the determinant of the \\({n}-1\\) rowed square matrix obtained from \\([A]\\) by deleting the first row and first column.
\\(D[A]\\) is a homogeneous polynomial of degree \\(n\\) in its elements \\(a_{i j}\\).
A necessary and sufficient condition that the system of equations \\(\\sum_{j} a_{i j} c_{j}=0\\) have a solution different from \\(c_{1}=c_{2}=\\cdots=c_{n}=0\\) is that \\(D[A]=0\\). In the language of linear transformations the last fact can be stated as follows: a necessary and sufficient condition that there exists a vector \\({x} \\neq 0\\) for which \\(A x=0\\) is that \\(D(A)=0\\). Still in other words: \\(A\\) has an inverse if and only if \\(D(A) \\neq 0\\).

2. Invariant linear subspaces

All of the results of this chapter are based on the following very simple comment. If a linear subspace \\({M}\\) is invariant under the linear transformation \\(A\\), then \\(M^{\\perp}\\) is invariant under \\(A^{*}\\). (\\(M\\) is said to be invariant under \\(A\\) if \\(x\\) in \\(M\\) implies \\(A x\\) in \\(M\\): in symbols \\(AM \\subseteq M\\)). For if \\(x\\) is in \\(M\\) and \\(y\\) is in \\(M^{\\perp}\\) then we have \\(\\langle x, A^{*} y \\rangle = \\langle A x, y \\rangle = 0\\), since \\(A x\\) is in \\(M\\). Since this is true for all \\(x\\) in \\(M\\), \\(A^{*} y\\) is in \\(M^{\\perp}\\), as was to be proved.

3. Determining invariant linear subspaces of an arbitrary transformation

As a consequence of the result of (III.2) we have the following theorem.

Theorem 1. To any linear transformation \\(A\\) there correspond two sets of linear subspaces, \\(M_{i}\\) and \\(N_{i}\\), \\(i=1, 2, \\ldots, n\\), each invariant under \\(A\\), such that the dimension of \\(M_{i}\\) is \\(i\\), the dimension of \\(N_{i}\\) is \\(n-i+1\\), \\(M_{1} \\subseteq M_{2} \\subseteq \\cdots \\subseteq M_{n}\\), and \\(N_{1} \\supseteq N_{2} \\supseteq \\cdots \\supseteq N_{n}\\). (In particular of course \\(M_{n} = N_{1} = \\mathcal{U}\\).)

Proof. We proceed by induction on \\(n\\). For \\(n=1\\) the result is trivial; we assume that it has been proved for inner product spaces of dimension \\(\\leq {n}-1\\).

Suppose now that \\(\\mathcal{U}\\) is of dimension \\(n\\), and consider the expression \\(D(\\lambda)=D(A-\\lambda 1)\\) as a function of the complex variable \\(\\lambda\\), where \\(D(A)\\) denotes the determinant of \\(A\\). It follows from the discussion in (III.1) that \\(D(\\lambda)\\) is a polynomial in \\(\\lambda\\) of degree \\(n\\). We apply the fundamental theorem of algebra to obtain the result that for a suitable value of \\(\\lambda\\), say \\(\\lambda=\\lambda_{0}\\), \\(D(\\lambda_{0})=0\\). It follows that there exists a vector \\(x_{0} \\neq 0\\), such that \\((A-\\lambda_{0} 1) x_{0}=0\\), or, in other words, \\(A x_{0}=\\lambda_{0} x_{0}\\). We denote by \\(M_{1}\\) the one dimensional linear subspace of all vectors of the form \\(ax_0\\). Then \\(M_{1}\\) is invariant under \\(A\\); hence \\(M_{1}^\\perp\\) is invariant under \\(A^{*}\\). \\(M_{1}^\\perp\\) is an \\(n-1\\) dimensional inner product space and \\(A^{*}\\) is a linear transformation on this space. We may therefore use our induction hypothesis and conclude that we can find a set of linear subspaces \\(N_{i}^{*}\\), for \\(i=2, \\ldots, n\\), each invariant under \\(A^{*}\\), \\(N_{i}^{*}\\) having dimension \\(n-i+1\\), and \\(N_{2}^{*} \\supseteq \\cdots \\supseteq {N}_{n}^{*}\\). Returning to the entire \\(n\\)-dimensional space \\(\\mathcal{U}\\) we define the subspaces \\(M_{i}\\) for \\(i=2, \\ldots, n\\) by \\(M_{i}=M_{1}+(N_{2}^{*} \\cap N_{i+1}^{*\\perp})\\), where we make the convention that \\({N}_{{n}+1}\\) is the subspace consisting of \\((0)\\) alone. An application of (III.2) shows that \\(M_{i}\\) is invariant under \\(A\\), and it is clear that it has the required monotoneity and dimensionality properties. This proves the part of the theorem referring to the \\({M}_{i}\\): to prove the part concerned with the \\({N}_{i}\\), apply the result just proved to \\(A^{*}\\) to obtain the corresponding subspaces \\({M}_{i}^{*}\\), and set \\({N}_{1}=\\mathcal{U}\\), \\({N}_{i}=M_{i+1}^{*}\\) for \\(i = 2, \\ldots, n\\).■

4. Upper triangular and lower triangular forms of a matrix

The matricial interpretation of (III.3) is very useful. Let \\(A\\) be a linear transformation and find the subspaces \\(M_{i}\\) as described in (III.3). Let \\(x_1\\) be an arbitrary vector of length \\(1\\) in \\(M_{1}\\); for \\(i=2, \\ldots, n\\) choose \\(x_{i}\\) to be an arbitrary vector of length \\(1\\) in \\(M_{i} \\cap M_{i-1}^\\perp\\). It is clear that the \\(x_{i}\\) form a complete orthonormal set: let us examine the matrix of \\(A\\) in the coordinate system of the \\(x_{i}\\). Since \\(x_{j}\\) lies in \\(M_{j}\\) and \\(M_{j}\\) is invariant under \\(A\\), \\(A x_{j}\\) will also lie in \\(M_{j}\\). But the vectors of \\({M}_{j}\\), being orthogonal to the \\(x_{i}\\) with \\(i>{j}\\), are characterized by the fact that in their expansion in terms of the \\(x_{i}\\) the coefficients of the \\(x_{i}\\) with \\(i>j\\) all vanish. In other words if \\(A x_{j}=\\sum_{i} a_{i j} x_{i}\\), then for \\(i>{j}\\), \\(a_{i j}=0\\). The matrix \\([A]\\) in this coordinate system has the form \\[ \\begin{bmatrix} a_{11} & a_{12} & a_{13} & \\cdots & a_{1 n} \\\\ 0 & a_{22} & a_{23} & \\ldots & a_{2 n} \\\\ 0 & 0 & a_{33} & \\ldots & a_{3 n} \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & a_{n n} \\end{bmatrix}. \\]

We shall refer to this form as the upper triangular form of \\(A\\). Similarly, by considering the subspaces \\({N}_{i}\\) we can establish that in a suitable coordinate system the matrix of \\(A\\) becomes lower triangular: i.e., for \\(i<j\\), \\(a_{i j}=0\\).

5. Spectral representation of Hermitian transformations

In the case of Hermitian transformations (III.2) implies that whenever \\(M\\) is invariant under \\(A\\) so is \\(M^{\\perp}\\). Accordingly the results of (III.3) and (III.4) can be strengthened in this case, as follows.

Theorem 2. If \\(A\\) is a Hermitian transformation there exist pairwise orthogonal projections \\(E_{1}, E_{2}, \\ldots, E_{p}\\) whose sum is \\(1\\) (i.e., \\(E_{i} E_{j}=0\\) if \\(i \\neq j\\)) and pairwise different real numbers \\(a_{1}, a_{2}, \\ldots, a_{p}\\) such that \\(A=\\sum_{i} a_{i} E_{i}\\). This representation (the so-called spectral representation or spectral form of \\(A\\)) is unique: i.e., if \\(A=\\sum_{j} b_{j} F_{j}\\) where the \\(b_{j}\\) and \\(F_{j}\\) satisfies the conditions placed on the \\(a_{i}\\) and \\(E_{i}\\), then the number of \\(F\\)’s is the same as the number of \\(E\\)’s and after a suitable permutation of the indices, \\(E_{i}={F}_{i}\\) and \\(a_{i}=b_{i}\\) for \\(i = 1, 2, \\ldots, p\\).

(We note again that our original intuitive picture of Hermitian transformations as the analogs of real numbers is justified by the reality of the \\(a_{i}\\).)

Proof. We first prove that if for a Hermitian transformation \\(A\\) we have a vector \\(x\\) and a scalar \\(\\lambda\\) such that \\(A x=\\lambda x\\), \\(x \\neq 0\\), then \\(\\lambda\\) is real. This follows from the relation \\(\\langle A x, x \\rangle = \\langle \\lambda x, x \\rangle = \\lambda|x|^{2}\\), and the fact that both \\(\\langle A x, x \\rangle\\) and \\(|x|^{2}\\) are real.

We proceed again by induction on \\({n}\\). For \\({n}=1\\) we may take \\({p}=1\\) and \\(E_{1}=1\\). Suppose then that the theorem has been proved for all inner product spaces of dimension \\(\\leq n-1\\), and consider, as in the proof of (III.3), a number \\(\\lambda\\) and a vector \\(x \\neq 0\\), such that \\(A x=\\lambda x\\). Let \\(M_1\\) be the linear subspace of all vectors \\(y\\) for which \\(Ay =\\lambda y\\); write \\(a_{1}=\\lambda\\) and let \\(E_{1}\\) be the projection which projects on \\(M_{1}\\). We know already that \\(a_1\\) is real, and it is clear that \\(M_{1}\\) is invariant under \\(A\\). It follows that \\(M_{1}^\\perp\\) is invariant under \\(A\\), and we may apply our induction hypothesis to the inner product space \\(M_{1}^\\perp\\) (of dimension \\(\\leq {n}-1\\)). Since for \\(x\\) in \\(M_{1}^\\perp\\), \\((A-a_{1} E_{1}) x=A x\\), the result follows.

To prove uniqueness we observe first that if \\(F_{j}\\) are orthogonal projections of sum \\(1\\), \\(b_{j}\\) complex numbers, and \\(x\\) a vector \\(\\neq 0\\), then \\(\\big(\\sum_{j} b_{j} F_{j}\\big) x=0\\) implies that at least one \\(b_{j}=0\\). For otherwise we may consider the transformation \\(\\sum_{j} c_{j} F_{j}\\), with \\(c_{j}=b_{j}^{-1}\\), and an easy computation shows that this transformation is the inverse of the original one.

Suppose then that for some complex number \\(\\lambda\\) and vector \\({x} \\neq 0\\) we had \\(\\sum_{i} a_{i} E_{i} x=\\lambda x=\\sum_{i} \\lambda_{i} x\\). Then by the preceding paragraph we should have \\(\\lambda=a_{i}\\) for some \\(i\\). In other words the numbers \\(a_{i}\\) occurring in the spectral form are characterized by the fact that to each of them there corresponds an \\(x \\neq 0\\) for which \\(A x=a_{i} x\\). To prove that the \\(E_{i}\\) are also determined by \\(A\\) we note that if \\(Ax=a_{i} x\\) and \\(A y=a_{j} y\\); \\(i \\neq j\\), then \\(x\\) is orthogonal to \\(y\\). For \\[ a_{i}\\langle x, y \\rangle = \\langle A x, y \\rangle = \\langle x, A y \\rangle = a_{j}\\langle x, y \\rangle, \\] so that since \\(a_{i} \\neq a_{j}\\), \\(\\langle x, y \\rangle = 0\\). Consequently if \\(A x=a_{i}x\\). then \\(E_{j} x=0\\) for all \\(j \\neq i\\), so that \\(E_{i}\\) is characterized as the projection on precisely that linear subspace of vectors \\(x\\) for which \\(A x=a_{i} x\\).■

6. Reduction to principal axes

In the notation of the preceding section let \\(A=\\sum_{i} a_{i} E_{i}\\), and let \\({M}_{i}\\) be the range of \\({E}_{i}\\) (i.e., \\({M}_{i}\\) is the subspace on which \\({E}_{i}\\) projects). Then the \\(M_{i}\\) are mutually orthogonal linear subspaces which span \\(\\mathcal{U}\\). We may, accordingly, choose a complete orthonormal set in each \\({M}_{i}\\); the set of all vectors in all these orthonormal sets is a complete orthonormal set in \\(\\mathcal{U}\\). If we denote this set by \\(\\{x_{i}\\}\\) and compute the matrix of \\(A\\) in this coordinate system, we observe that \\(A x_{i}\\) is for each \\(i\\) a scalar multiple of \\(x_{i}\\), so that in the matrix \\([A]\\) every element, except possibly those on the main diagonal, is zero. Matrices with this property are called diagonal matrices: our result in (III.5) is equivalent to the statement that to every Hermitian transformation \\(A\\) there corresponds a coordinate system in which the matrix \\([A]\\) is a diagonal matrix. This reduction of an arbitrary Hermitian matrix to diagonal form, or, equivalently, of the quadratic form \\(\\langle Ax, x \\rangle\\) to a sum of squares, is known as the reduction to principal axes.

7. Terminology of spectral theory

Before we proceed to a more detailed study of the spectral form of Hermitian transformations we introduce some of the usual terminology of the spectral theory.

Definition 1. For an arbitrary linear transformation \\(A\\) the polynomial \\(D(\\lambda)=D(A-\\lambda 1)\\) of degree \\(n\\) is called the characteristic polynomial of \\(A\\), and the equation \\(D(\\lambda)=0\\) its characteristic equation. The roots of the characteristic equation are the eigenvalues of \\(A\\).

We have already made use of the fact that \\(\\lambda\\) is an eigenvalue of \\(A\\) if and only if there exists a non-zero vector \\(x\\) for which \\(A x=\\lambda x\\). The multiplicity of \\(\\lambda\\) as a root of the equation \\(D(\\lambda)=0\\) is called the multiplicity of the eigenvalue \\(\\lambda\\). The set of \\(n\\) eigenvalues \\(\\lambda_{1}, \\lambda_{2}, \\ldots, \\lambda_{n}\\), with multiplicities counted properly, is the spectrum of \\(A\\).

8. Linear combinations of orthogonal projections

In order to gain a better insight into the properties of the spectral representation we shall, in this section, study a slightly more general form. We assume that \\({E}_{1}, E_{2}, \\ldots, {E}_{p}\\) are pairwise orthogonal projections whose sum is \\(1\\), and we consider linear transformations of the form \\(A = \\sum_{i} a_{i} E_{i}\\), where, however, we no longer require that \\(A\\) be Hermitian (so that the \\(a_{i}\\) need not be real), nor that the \\(a_{i}\\) be all different. We shall study a number of elementary properties of such linear transformations \\(A\\).

8.1. If \\(A=\\sum_{i} a_{i} E_{i}\\), then \\[ \\begin{aligned} A^{2}&=\\Big(\\sum_{i} a_{i} E_{i}\\Big)\\Big(\\sum_{j} a_{j} E_{j}\\Big)\\\\ &=\\sum_{i} \\sum_{j} a_{i} a_{j} E_{i} E_{j}\\\\ &=\\sum_{i} a_{i}^{2} E_{i}, \\end{aligned} \\] (since \\(E_{i} E_{j}=E_{i}\\) or \\(0\\) according as \\(i=j\\) or \\(i \\neq j\\)). More generally if \\(p(\\lambda)\\) is an arbitrary polynomial, \\({p}(A)=\\sum_{i} p(a_{i}) E_{i}\\). If no \\(a_{i}\\) vanishes we prove similarly that \\(\\sum_{i} a_{i}^{-1} E_{i}\\) is the inverse of \\(A\\); and the relation \\[ \\begin{aligned} \\Big\\langle \\sum_{i} a_{i} E_{i} x, y \\Big\\rangle &= \\sum_{i} a_{i}\\langle E_{i} x, y \\rangle\\\\ &= \\sum_{i} a_{i}\\langle x, E_{i} y \\rangle\\\\ &= \\Big\\langle x, \\sum_{i} \\bar{a}_{i} E_{i} y \\Big\\rangle \\end{aligned} \\] shows that \\(A^{*}=\\sum_{i} \\bar{a}_{i} E_{i}\\). It follows immediately that \\(A\\) is Hermitian if and only if \\(a_i\\) is real, unitary if and only if \\(|a_{i}|=1\\), and idempotent if and only if \\(a_{i}=0\\) or \\(1\\): each of these conditions is supposed to hold for \\(i = 1, 2, \\ldots, {p}\\).

8.2. For any \\(x\\) we have \\[ \\begin{aligned} \\langle A x, x \\rangle &= \\Big\\langle \\sum_{i} a_{i} {E}_{i} x, x \\Big\\rangle\\\\ &= \\sum_{i} a_{i}\\langle E_{i} x, x \\rangle\\\\ &= \\sum_{i} a_{i}|E_i x|^{2}. \\end{aligned} \\]

It follows that if \\(a_{i} \\geq 0\\) for \\(i=1,2, \\ldots, p\\) then \\(A\\) is non-negative. Conversely if \\(A\\) is non-negative, we may choose \\(x \\neq 0\\) so that \\(E_{i} x=x\\), and for this \\(x\\) we have \\(\\langle A x, x \\rangle = a_{i}|x|^{2}\\), so that \\(a_{i} \\geq 0\\).

8.3. Denote by \\(a\\) the number \\(a=\\max (|a_{1}|,|a_{2}|, \\ldots,|a_{p}|)\\). We have, for any vector \\(x\\), \\[ \\begin{aligned} |A x|^{2} &= \\Big\\langle \\sum_i a_i E_i x, \\sum_{j} a_{j} E_{j} x \\Big\\rangle\\\\ &= \\sum_i \\sum_{j} a_{i} \\bar{a}_{j}\\langle E_{i} x, E_{j} x\\rangle \\\\ &= \\sum_i \\sum_{j} a_{i} \\bar{a}_{j}\\langle x, E_{i} E_{j} x \\rangle\\\\ &= \\sum_i|a_{i}|^{2}|E_{i} x|^{2} \\end{aligned} \\] so that \\(|A x|^{2} \\leq a|x|^{2}\\), or, in other words, \\(|A| \\leq a\\). If the maximum \\(a\\) is attained by \\(a_{i}\\), i.e., \\(|a_{i}|=a\\), then by choosing \\(x\\) so that \\({E}_{i} {x}={x}\\), as above, it follows that \\(|A|= a\\). Similarly it is easy to show that in case \\(A\\) is Hermitian, so that the \\(a_{i}\\) are real, the upper bound and the lower bound of \\(A\\) are given, respectively, by \\(\\max (a_{1}, a_{2}, \\ldots, a_{p})\\) and \\(\\min (a_{1}, a_{2}, \\ldots, a_{p})\\).

8.4. In this last section we suppose that the \\(a_{i}\\) are all different and real, so that \\(\\sum_i a_i E_i\\) is the spectral form of the Hermitian transformation \\(A\\). In this case we assert that a necessary and sufficient condition that \\(A B=B A\\) for a Hermitian \\(B\\) is that \\(E_{i} B=B E_{i}\\) for \\(i = 1, 2, \\ldots, p\\). The condition is clearly sufficient: we shall merely prove its necessity. We recall that \\(E_i\\) is characterized by the fact that \\(A x=a_{i} x\\) is equivalent to \\(E_{i} x=x\\). Hence if \\(A B=B A\\) and \\(E_{i} x=x\\) then \\[ A(B x)=B A x=B a_{i} x=a_{i} B x, \\] so that \\(E_{i} B x=B x\\). Since for an arbitrary \\(y\\), \\(x={E}_{i} y\\) does have the property that \\(E_{i} x=x\\), we have \\(E_{i} B E_{i} y={B}{E}_{i} y\\) for all \\(y\\), or, in other words, \\({E}_{i} {B}{E}_{i}={B}{E}_{i}\\). Taking the dual of this relation and remembering that \\(B\\) is Hermitian yields the desired result.

All of the results of this section are immediately translatable into matrix language: thus (8.1) asserts that a diagonal matrix is Hermitian, unitary, or idempotent, if and only if the diagonal elements are real, of absolute value one, or all zero or one, respectively.

9. Normal transformations and the Cartesian decomposition

Let \\(A\\) be an arbitrary linear transformation and write \\(B=(1 / 2)(A+A^{*})\\), \\(C=(1 / 2i)(A-A^{*})\\). It is immediately verified that \\(B\\) and \\(C\\) are Hermitian and \\(A=B+i C\\). We shall refer to this representation as the Cartesian decomposition of \\(A\\) and we call \\(B\\) and \\(C\\) the real and imaginary parts of \\(A\\). It might be conjectured that since every Hermitian transformation has a spectral form and every transformation is a linear combination of Hermitian transformations, that every linear transformation has a spectral form, or, equivalently, that every linear transformation can be realized as a diagonal matrix. It is, however, not in general true that the coordinate system which reduces \\([B]\\) to diagonal form will also reduce \\([C]\\) to diagonal form, or conversely. In fact a necessary and sufficient condition that \\(A\\) be realizable as a diagonal matrix, or, equivalently, that there exist mutually orthogonal projections, \\(E_i\\), of sum one, such that \\(A=\\sum_i a_i E_i\\) is that \\(B C=C B\\). (We remark that \\(B\\) and \\(C\\) are uniquely determined by \\(A\\), for if \\(A=B+iC\\) then \\(A^{*}=B-iC\\) and solving for \\(B\\) and \\(C\\) we obtain the original expressions for them.)

Proof of sufficiency. If \\(B C=C B\\), let \\(B=\\sum_i b_{i} F_{i}\\) and \\(C=\\sum_{j} c_{j} G_{j}\\) be the spectral representations of \\(B\\) and \\(C\\) respectively. From (III.8.4) it follows that \\(F_{i}\\) and \\(G_{j}\\) are commutative and therefore that \\(F_{i} G_{j}\\) is a projection. Since, moreover, \\(F_{i} G_{j} F_{r} G_{s}=F_{i} F_{r} G_{j} G_{s}=0\\) unless \\(i=r\\) and \\(j=s\\), the \\(F_{i} G_{j}\\) are pairwise orthogonal projections whose sum is one, with the additional property that \\(\\sum_j F_{i} G_{j}=F_{i}\\) and \\(\\sum_i F_{i} G_{j}=G_{j}\\). If we denote this set of projections in some order by \\(H_k\\), then we obtain representations of the form \\(B=\\sum_{k} b_{k}^{\\prime} H_{k}\\) and \\(C=\\sum_{k} c_{k}^\\prime {H}_{k}\\). (These are not, of course, the spectral forms of \\(B\\) and \\(C\\), since the \\(b_{k}^\\prime\\) and \\(c_{k}^{\\prime}\\) need not be pairwise different.) It follows that \\[ A=B+iC=\\sum_{k}(b_{k}^{\\prime}+ ic_{k}^{\\prime}) H_{k}, \\] as was to be proved.

Proof of necessity. Rather than proving necessity directly we prove that the condition \\(B C=C B\\) is equivalent to the condition \\(AA^*=A^*A\\). For if the latter condition is satisfied it follows from the definitions of \\(B\\) and \\(C\\) that the former one is; and conversely if \\(B C=C B\\) then \\(A=B+iC\\) and \\(A^{*}=B-iC\\) are commutative. That \\(A A^{*}=A^{*} A\\) is a necessary condition for the existence of a spectral form is clear: if \\(A=\\sum_i a_i E_i\\) then \\(A^{*}=\\sum_i \\bar{a}_{i} E_{i}\\).

Transformations \\(A\\) satisfying any one of the equivalent conditions discussed in this section are called normal. An arbitrary unitary transformation \\(U\\) is an example of a normal but not necessarily Hermitian transformation: since \\(U^{*}=U^{-1}\\), it is clear that \\(UU^{*}=U^{*} U\\) since both products are equal to the identity transformation.

10. Functions of normal transformations

If \\(A\\) is a normal transformation, let \\(A=\\sum_i a_i E_i\\) be the spectral form of \\(A\\): i.e., the \\(E_{i}\\) are pairwise orthogonal projections whose sum is one and the \\(a_{i}\\) are pairwise different complex numbers. Let \\(f(\\lambda)\\) be an arbitrary complex valued function of the complex variable \\(\\lambda\\), which is defined for \\(\\lambda=a_{i}\\), \\(i = 1, \\ldots, p\\). We define the symbol \\(f(A)\\) to be the transformation \\(f(A)=\\sum_i f(a_{i}) E_{i}\\). The discussion in (8.1) shows that if \\(f(\\lambda)\\) is a polynomial then this definition of \\(f(A)\\) agrees with the one used previously; if \\(f(\\lambda)=\\lambda^{-1}\\) then (assuming that \\(f(\\lambda)\\) is defined for all \\(a_{i}\\)) \\(f(A)=A^{-1}\\); and if \\(f(\\lambda)=\\bar{\\lambda}\\), then \\(f(A)=A^{*}\\). These statements imply that if \\(f(\\lambda)\\) is an arbitrary rational function of \\(\\lambda\\) and \\(\\bar{\\lambda}\\), we obtain \\(f(A)\\) by the replacements \\[ \\lambda \\to A, \\quad \\bar{\\lambda} \\to A^{*}, \\quad \\lambda^{-1} \\to A^{-1}, \\quad c \\to c1. \\]

The symbol \\(f(A)\\) is defined for much more general functions, however, and we shall in what follows make use of expressions of the type \\(e^{A}\\) and \\(A^{1 / 2}\\) without any further explanation. It is, however, worthwhile to observe that this procedure is only a notational advantage: it introduces nothing conceptually new. For we may write, for an arbitrary \\(f(\\lambda)\\), \\(f(a_{i})=b_{i}\\), and then we may find a polynomial, \\(p(\\lambda)\\) which at the finite set of different complex numbers \\(a_{i}\\) takes respectively the values \\(b_{i}\\). With this polynomial \\(p(\\lambda)\\) we have \\(f(A)=p(A)\\). Hence the class of transformations defined by \\(f(A)\\) is nothing essentially new: it only saves the trouble of constructing a polynomial \\(p(\\lambda)\\) to fit each special case.

11. The square root of a non-negative transformation

We recall that in (II.7) we mentioned three possible definitions of a non-negative transformation and adopted the weakest one, namely that \\(\\langle A x, x \\rangle \\geq 0\\) for all \\(x\\). The strongest of the three possible definitions was that we could write \\(A\\) in the form \\(A=B^{2}\\), with a Hermitian (real) \\(B\\): in other words that \\(A\\) has a Hermitian square root. It is now trivial to prove that the weak definition implies the strong one, so that they are equivalent. For let \\({A}\\) be non-negative in the sense that \\(\\langle Ax, x \\rangle\\) is non-negative for all \\(x\\). We know from (III.8.2) that in the spectral form \\(\\sum_i a_i E_i\\) of \\(A\\) the \\(a_{i}\\) must be non-negative. Hence if we consider the function \\(f(\\lambda)\\) defined for all non-negative real numbers \\(\\lambda\\) as the non-negative square root of \\(\\lambda\\), we may define \\(B=f(A)=\\sum_i f(a_{i}) E_i\\). It is clear that \\(B^{2}=A\\), and since the \\(f(a_{i})\\) are non-negative real numbers \\(B\\) is not only Hermitian but even non-negative.

At first glance it may seem hopeless to look for any uniqueness in the representation \\(A=B^{2}\\) since if we consider \\(B^{\\prime}=\\sum_i \\pm f(a_{i}) E_i\\), With arbitrary choice of sign in each place, we still have \\(A=(B^\\prime)^{2}\\), with \\(B^{\\prime}\\) Hermitian. The \\(B\\) we constructed, however, was non-negative and we can show this additional property guarantees uniqueness: in other words \\(A=(B^{\\prime})^{2}\\), \\(B^{\\prime} \\geq 0\\) implies \\(B^{\\prime}=B\\). For let \\(B^{\\prime}=\\sum_i b_{i} F_{i}\\) be the spectral form of \\(B^{\\prime}\\): the \\(b_{i}\\) are different non-negative numbers. It follows that \\[ (B^{\\prime})^{2}=\\sum_{j} b_{j} F_{j}=A=\\sum_i a_i E_i, \\] where the \\(F_{j}\\) are pairwise orthogonal projections of sum one and the \\(b_{j}^{2}\\) are pairwise different non-negative numbers. The uniqueness of the spectral form of \\(A\\) implies that the number of \\(F\\)’s is the same as the number of \\(E\\)’s and that after a suitable permutation of indices \\(b_{i}^{2}=a_{i}\\), whence \\(b_{i}=f(a_{i})\\), so that \\(B^{\\prime}=B\\).

12. Product of non-negative transformations

In (II.8) we stated the theorem that if \\(A\\) and \\(B\\) are commutative non-negative transformations then their product \\(AB = BA\\) is also non-negative. Using the results of the preceding section it is possible to give an easy proof of this statement. Let \\(A=\\sum_i a_{i} E_i\\) and \\(B=\\sum_{j} b_{j} F_{j}\\) be the spectral forms of \\(A\\) and \\(B\\) respectively and let \\(f(\\lambda)\\) be the same function (the non-negative square root) as we considered above. Then \\(f(A)\\) and \\(f(B)\\) are non-negative square roots of \\(A\\) and \\(B\\) respectively and since, in view of the comment in (III.10) they can also be written as polynomials in \\(A\\) and \\(B\\), the commutativity of \\(A\\) and \\(B\\) implies that of \\(f(A)\\) and \\(f(B)\\). (This fact is also easy to derive from the general commutativity theorem of (III.8.4).) Hence we have

\\[ \\begin{aligned} A B&=f(A)^{2} f(B)^{2}\\\\ &=f(A) f(A) \\, f(B) f(B)\\\\ &=f(A) f(B) \\, f(A) f(B)\\\\ &=\\big(f(A) f(B)\\big)^2. \\end{aligned} \\]

Since \\(f(A)\\) and \\(f(B)\\) are commutative Hermitian transformations their product is Hermitian, and therefore we have expressed \\(A B=B A\\) as the square of a Hermitian transformation. It follows that \\(A B=B A\\) is non-negative.

13. Commutative Hermitian transformations

We have often made use of the fact that if \\({C}\\) is a Hermitian transformation and \\(p(\\lambda)\\) and \\(q(\\lambda)\\) are any two polynomials then \\(A=p(C)\\) and \\(B=q(C)\\) are commutative. In this section we shall prove a converse of this result.

Theorem 3. If \\(A\\) and \\(B\\) are commutative Hermitian transformations then there exists a Hermitian transformation \\(C\\) and polynomials \\(p(\\lambda)\\) and \\(q(\\lambda)\\) such that \\(A=p(C)\\) and \\(B=q(C)\\). \\(C\\) can be written in the form \\(C=r(A, B)\\), where \\(r(\\lambda, \\mu)\\) is a polynomial in the two variables \\(\\lambda\\) and \\(\\mu\\).

(We observe that ordinarily an expression of the form \\(r(A, B)\\) would be meaningless, since in general \\(A\\) and \\(B\\) are not commutative, without some convention as to the order in which \\(A\\) and \\(B\\) are to be substituted for \\(\\lambda\\) and \\(\\mu\\). Since in the present case, however, \\(A\\) and \\(B\\) are commutative by hypothesis, this difficulty does not enter.)

Proof. Let \\(A=\\sum_i a_i E_i\\) and \\(B=\\sum_{j} b_{j} F_{j}\\) be the spectral forms of \\(A\\) and \\(B\\). It follows from (III.8.4) and the commutativity of \\(A\\) and \\(B\\) that \\(E_{i}\\) and \\(F_{j}\\) are commutative projections for every \\(i\\) and \\(j\\). Let the projections \\(E_{i} F_{j}\\), enumerated in some order, be \\(G_{k}\\). Then we have using the fact that \\(\\sum_{j} E_{i} F_{j}=E_{i}\\) and \\(\\sum_i E_{i} F_{j}=F_{j}\\), \\(A=\\sum_{k} a_{k}^{\\prime} G_{k}\\) and \\(B=\\sum_{k} b_{k}^\\prime G_{k}\\). Although it is true that the \\(G_{k}\\) are pairwise orthogonal projections of sum one, these representations are not the spectral forms of \\(A\\) and \\(B\\), since it need not be true that the \\(a_{k}^\\prime\\) (or the \\(b_k^\\prime\\)) are pairwise different. It is, however, true that for \\(h \\neq k\\) the pair (considered, say, as a point of the real Euclidean plane) \\((a_{h}^\\prime, b_{h}^\\prime)\\) is different from the pair \\((a_{k}^{\\prime}, b_{k}^{\\prime})\\). Hence we may find a polynomial \\(r(\\lambda, \\mu)\\) which at the finite set of different points \\((a_{k}^\\prime, b_{k}^\\prime)\\) assumes respectively, the pairwise different real values \\(c_{k}^{\\prime}\\): \\(r(a_{k}^{\\prime}, b_{k}^{\\prime}) = c_{k}^{\\prime}\\). We write \\(C=r(A, B)\\), and we proceed to prove that \\(C\\) has the required properties.

Since \\(A^{i}=\\sum_{k} (a_{k}^\\prime)^i {G}_{k}\\) and \\(B^{j}=\\sum_{k}(b_{k}^\\prime)^j {G}_{k}\\) it follows that \\(A^{i} B^{j}=\\sum_{k}(a_{k}^\\prime)^{i}(b_{k}^\\prime)^{j} G_k\\), and, more generally, for any polynomial \\(s(\\lambda, \\mu)\\), in two variables, \\(s(A, B)=\\sum_{k} s(a_{k}^\\prime, b_{k}^{\\prime}) G_{k}\\). Hence, in particular, \\(c=\\sum_k c_k^{\\prime} G_{k}\\). If we choose the polynomials \\(p(\\lambda)\\) and \\(q(\\lambda)\\) so that \\({p}(c_{k}^{\\prime})=a_{k}^{\\prime}\\) and \\({q}(c_{k}^{\\prime})=b_{k}^{\\prime}\\); we have \\(p(c)=A\\) and \\(q(C)=B\\), as was to be proved.■

This fact is a very useful one in the consideration of groups and algebras of matrices and their reducibility.

14. The correspondence \\(U=\\exp(2\\pi A)\\)

Since a unitary transformation \\(U\\) is normal it has a spectral representation \\(U=\\sum_{j} u_{j} E_{j}\\), where, of course, \\(|u_{j}|=1\\). Since we may write \\(u_{j}\\) in the form \\(u_{j}=\\exp (2 \\pi i a_{j})\\) with real numbers \\(a_{j}\\) (where \\(i=(-1)^{1 / 2}\\)), it follows that \\(U=\\exp (2 \\pi i A)\\), where \\(A\\) is the Hermitian transformation, \\(A=\\sum_j a_{j} E_{j}\\). Because of the periodicity of the exponential function we do not, of course, have uniqueness in this representation. Since, however, we may choose \\(a_{j}\\) so that \\(0 \\leq a_{j} <1\\), the corresponding \\(A\\) will have the property that its lower bound is non-negative, and its upper bound is \\(< 1\\), i.e. \\(0 \\leq A < 1\\), and it is easily verified that this additional condition determines a uniquely. The converse, of course, is clear: if \\(A\\) is any Hermitian transformation, \\(U=\\exp (2 \\pi i A)\\) is unitary. In other words there is a one to one correspondence between all unitary transformations and those Hermitian transformations which are bounded between \\(0\\) and \\(1\\), established by the formula \\(U=\\exp (2 \\pi i A)\\). This result is the analog of the same in the algebra of transformations, that every complex number \\(u\\) of absolute value one can be written, in one and only one way, in the form \\(u=\\exp (2 \\pi i a)\\), with \\(0 \\leq a < 1\\).

15. Cayley transform

The result of the preceding section was based on a one to one mapping between the unit circle in the complex plane and the half-open unit interval on the real line. Another mapping of real numbers into complex numbers of absolute value one is given by \\(u=f(a)\\), where \\(f(a)=\\dfrac{a+i}{a-i}\\). It is easy to verify that this mapping is one to one between all real numbers \\(a\\) and all complex numbers \\(u\\) of absolute value one with the exception of \\(u= 1\\). It follows that the mapping \\(U=f(A)\\), (whose inverse mapping, incidentally, is \\(A=g(U)\\), where \\(g(u)=i \\dfrac{u+1}{u-1}\\)), is a one to one correspondence between all Hermitian transformations \\(A\\) and those unitary transformations \\(U\\) which do not have the eigenvalue \\(1\\). Since the structure of unitary transformations is clear geometrically, this correspondence is often used to obtain information about Hermitian transformations that may be rather difficult to obtain in other ways. The unitary transformation \\(U\\) is said to be the Cayley transform of \\(A\\).

16. Polar decomposition

Every complex number \\(b\\) can be written in one and only one way in the form \\(b = pu = p\\exp(2\\pi i a)\\) where \\(p\\) is non-negative and \\(u\\) has absolute value one, or, equivalently, \\(a\\) lies between zero and one. We shall conclude this chapter by proving the analog, in the algebra of transformations, of this theorem. Although most of the results of this chapter are valid for normal transformations only, it is noteworthy that the theorem of this section is true in all generality, for arbitrary transformations \\(B\\).

Theorem 4. If \\(B\\) is an arbitrary linear transformation, there exists a non-negative transformation \\(P\\) and a unitary transformation \\(U\\) such that \\(B=P U\\). \\(P\\) is uniquely determined by \\(B\\); if \\(B\\) has an inverse, and only in this case, \\(U\\) is uniquely determined by \\(B\\).

Proof. Although it is not logically necessary we shall first give the proof in the case where \\(B\\) has an inverse: the proof in the other case is an obvious modification of this proof, which will give greater insight into the structure of arbitrary transformations.

Since the transformation \\({BB}^{*}\\) is non-negative, we can find a non-negative transformation \\(P\\) such that \\(P^{2}=BB^{*}\\). Write \\(V=B^{-1} P\\); since \\(B V=P\\), the theorem will be proved if we can prove that \\(V\\) is unitary, for then we may choose \\(U=V^{-1}\\). Since \\(V^{*}=P^{*}(B^{-1})^{*}=P(B^{*})^{-1}\\), the relations \\[ VV^{*}=B^{-1} {PP}({B}^{*})^{-1}=B^{-1} B B^{*}(B^{*})^{-1}=1 \\] imply that \\(V\\) is unitary. To prove uniqueness observe that \\(P U=P_{0} U_{0}\\) implies \\(U^{*} P=U_{0}^{*} P_{0}\\) and therefore \\(P UU^{*} P=P_{0} U_{0} U_{0}^{*} P_{0}\\) or, \\(P^{2}=P_{0}^{2}\\). Since a non-negative transformation has only one non-negative square root it follows that \\(P=P_{0}\\) and therefore (multiplying the relation \\(P U=P_{0} U_{0}\\) on the left by \\(P^{-1}\\)) \\(U=U_{0}\\). (\\(P^{-1}\\) exists since \\(B=P U\\) implies \\(P=B U^{-1}\\).)

We turn now to the general case where we do not assume that \\(B^{-1}\\) exists. We form \\(P\\) exactly the same way as in the preceding proof, that \\({P}^{2}={BB}^{*}\\), and we observe that for any vector \\({x}\\) we have \\[ \\begin{aligned} |P x|^{2} &= \\langle P x, P x \\rangle\\\\ &= \\langle P^{*} P x, x \\rangle \\\\ &= \\langle P^{2} x, x \\rangle\\\\ &= \\langle B B^{*} x, x \\rangle\\\\ &= \\langle B^{*} x, B^{*} x \\rangle\\\\ &=|B^{*} x|^{2}. \\end{aligned} \\] Hence if for every vector \\(y\\) in the linear subspace of vectors of the form \\(Px\\) we define \\(V y=B^{*} x\\), the transformation \\(V\\) is, wherever it is defined, length preserving. We must show that \\(V\\) is uniquely determined by the condition given: i.e., that \\({P}{x}^{\\prime}={P}{x}^{\\prime\\prime}\\) implies \\({B}^{*} {x}^{\\prime}=B^{*} x^{\\prime\\prime}\\). This is true since \\(P(x^{\\prime}-x^{\\prime\\prime})=0\\) is equivalent to \\(|P(x^{\\prime}-x^{\\prime\\prime})|=0\\) and this latter fact implies \\(|B^{*}(x^{\\prime}-x^{\\prime\\prime})|=0\\). If we define \\(V\\) on the orthogonal complement of the set of vectors of the form \\(Px\\) to be an arbitrary unitary transformation of this orthogonal complement into itself, the transformation \\(V\\), thus determined in all \\(\\mathcal{U}\\), is unitary and has the property that \\(VPx = B^{*} x\\) for all \\(x\\). Hence \\(B=P V^{*}\\) and the theorem follows upon setting \\(U=V^{*}\\). The uniqueness of \\(P\\) is proved the same way as in the case when \\(B^{-1}\\) exists, the extent of non-uniqueness of \\(U\\) is clear from the proof, in which at one place we were free to make an arbitrary choice for \\(V\\).■

We shall refer to the representation \\(B=P U\\) as the polar decomposition of \\(B\\). Applying the theorem just proved to \\(B^{*}\\) instead of \\(B\\) we obtain the result that every \\(B\\) can be written in the form \\(B={W P}\\) with a unitary \\(W\\) and a non-negative \\(P\\).

17. Polar conditions for normality

In terms of the polar decomposition we can state another necessary and sufficient condition that \\(B\\) be normal. If \\(B=P U\\) is the polar decomposition of \\(B\\), then \\(B\\) is normal if and only if \\(PU = UP\\). (Since \\(U\\) is not uniquely determined by \\(B\\), this statement is to be interpreted as follows: if for any decomposition of \\(B\\), \\(P\\) and \\(U\\) commute then \\(B\\) is normal and if \\(B\\) is normal then \\(P\\) and \\(U\\) commute.) Since \\(B^{*} B=U^{*} P^{2} U\\) and \\(B B^{*}=P^{2}\\), it is clear that \\(B\\) is normal if and only if \\(U\\) and \\(P^{2}\\) are commutative. If \\(U\\) and \\(P\\) commute then certainly \\(U\\) and \\(P^{2}\\) do, so that \\(B\\) is normal. If \\(B\\) is normal we have \\(U^{*} P^{2} U=P^{2}=A\\) (say), where \\(A\\) is non-negative; the relations \\(A=(U^{*}PU)^{2}\\) and \\(A = P^2\\) together with the uniqueness theorem for non-negative square roots, imply the commutativity of \\(U\\) and \\(P\\).

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"spectral-theory-of-normal-transformations","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]},{"_id":"66a0493339e7e3d9dc7ad844","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Dual Spaces and Tensor Products","sequence":4,"chapter_content":"

1. Transformations of rank one

Before beginning the proper subject matter of the present chapter we digress to a discussion of interest in itself whose results we shall need later. It follows easily from the spectral theory of normal transformations (or, equivalently, from the possibility of representing a normal transformation by a diagonal matrix) that every normal transformation is a sum of normal transformations of rank one; similarly every Hermitian (or non-negative) transformation is a sum of Hermitian (or non-negative) transformations of rank one. It becomes, therefore, of interest to investigate transformations of rank one.

Theorem 1. A necessary and sufficient condition that a linear transformation \\(A\\) has rank \\(1\\) is that in every coordinate system the matrix \\([A]=(a_{ij})\\) has the form \\(a_{ij}=b_{i} c_{j}\\).

Proof. If \\(A\\) has rank \\(1\\) then the set of all vectors of the form \\(Ax\\) is one dimensional, so that there exists a vector \\(x_{0}\\) with the property that \\(Ax\\) is for every \\(x\\) a constant multiple (depending on \\(x\\)) of \\(x_{0}\\): \\(A x=f(x) x_{0}\\). (It is easily verified that \\(f(x)\\) is a linear function of \\(x\\), but we shall not need this fact.) Now if \\(\\{x_{i}\\}\\) is a coordinate system the matrix \\([A]\\) is characterized by \\(A x_{j}=\\sum_{k} a_{k j} x_{k}\\), whence \\[ \\langle A x_{j}, x_{i} \\rangle = \\sum_{k} a_{k j}\\langle x_{k}, x_{i} \\rangle = a_{ij}, \\] so that \\[ a_{ij}=\\big\\langle f(x_{j}) x_{0}, x_{i} \\big\\rangle = f(x_{j})\\,\\langle x_{0}, x_{i} \\rangle = b_{i} c_{j}. \\]

Conversely if \\(a_{ij}=b_{i} c_{j}\\), we may find a linear function \\(f(x)\\) for which \\(f(x_{j})=c_{j}\\), and we may define \\(x_{0}=\\sum_i b_{i} x_{i}\\). The linear transformation \\(B\\) defined by \\(B x=f(x) x_{0}\\) is clearly of rank \\(1\\), and we have \\[ b_{ij} = \\langle B x_{j}, x_{i} \\rangle = b_{i} c_{j}=a_{ij}, \\] so that \\(B=A\\).■

If \\(A\\) has rank \\(1\\) and is Hermitian and if the matrix of \\(A\\) in some coordinate system is \\((b_{i} c_{j})\\), then we must have \\(b_{i} c_{j}=\\overline{b_{j} c_{i}}\\). If, for some \\(i\\), \\(b_{i}=0\\) and \\(c_{i} \\neq 0\\), then \\(\\bar{b}_{j}=(b_{i} / \\bar{c}_{i}) c_{j}=0\\) for all \\(j\\), whence \\(A=0\\). Since we assumed that the rank of \\(A\\) is \\(1\\), this is impossible and we can find an \\(i\\) for which \\(b_{i} c_{i} \\neq 0\\). Using this \\(i\\) the relation \\(\\bar{b}_{j}=(b_{i} / \\bar{c}_{i}) c_{j}\\) implies that \\(c_{j}=k \\bar{b}_{j}\\) with some constant \\(k\\) independent of \\({j}\\). Since the diagonal elements of a Hermitian matrix, i.e., the \\(\\langle A x_{i}, x_{i} \\rangle\\), are real, we can even conclude that \\(k\\) is real, so that in this case \\(a_{ij}\\) has the form \\(a_{ij}=k a_{i} \\bar{a}_{j}\\) with a real \\(k\\).

If \\(A\\) has rank \\(1\\) and is non-negative, then the discussion of the preceding paragraph applies and the fact that the diagonal elements of a non-negative matrix are non-negative implies that \\(k\\) is non-negative. In this case we may write \\({h}=+{k}^{1 / 2}\\) and the relation \\(k a_{i} \\bar{a}_{j} = (ha_i)\\overline{(ha_j)}\\) shows that \\(a_{ij}\\) has the form \\(a_{ij}=a_{i} \\bar{a}_{j}\\).

It is easy to see that the conditions given in the last two paragraphs are not only necessary but also sufficient. If \\(a_{ij}=k a_{i} \\bar{a}_{j}\\) then clearly \\(A\\) is Hermitian and has rank \\(1\\). If, moreover, \\(a_{ij}=a_{i} \\bar{a}_{j}\\) and \\(x=\\sum_i c_i x_{i}\\) then

\\[ \\begin{aligned} \\langle A x, x \\rangle &= \\sum_i \\sum_{j} c_{j} \\bar{c}_{i}\\langle A x_{j}, x_{i} \\rangle\\\\ &= \\sum_i \\sum_{j} c_{j} \\bar{c}_{i} a_{i} \\bar{a}_{j} \\\\ &= \\Big(\\sum_i a_{i} \\bar{c}_{i}\\Big) \\overline{\\Big(\\sum_{j} a_{j} \\bar{c}_{j}\\Big)}\\\\ &= \\Big|\\sum_i a_{i} \\bar{c}_{i}\\Big|^{2} \\\\ &\\geq 0 \\end{aligned} \\]

so that \\(A\\) is non-negative.

2. The Hadamard product of non-negative matrices

As a consequence of the preceding section it is very easy to prove a remarkable theorem on non-negative matrices, due to I. Schur.

Theorem 2. If \\(A\\) and \\(B\\) are non-negative linear transformations whose matrices in some coordinate system are \\((a_{ij})\\) and \\((b_{ij})\\) respectively then the linear transformation \\(C\\) whose matrix in this coordinate system is defined by \\(c_{ij}=a_{ij} b_{ij}\\) is also non-negative.

Proof. Since we may write both \\(A\\) and \\(B\\) as a sum of non-negative transformations of rank \\(1\\), \\(C\\) may be written as a sum of transformations the matrices of which are obtained from the matrices of two non-negative transformations of rank \\(1\\) in the same way as the matrix of \\(C\\) was obtained from the matrices of \\(A\\) and \\(B\\). Since a sum of non-negative transformations is non-negative, it is therefore sufficient to prove the theorem in the case where \\(A\\) and \\(B\\) both have rank \\(1\\). In this case \\(a_{ij}=a_{i} \\bar{a}_{j}\\), \\(b_{ij}=b_{i} \\bar{b}_{j}\\), and therefore \\(c_{ij}=c_{i} \\bar{c}_{j}\\), where \\(c_{i}=a_{i} b_{i}\\), whence it follows that \\(C\\) is non-negative (and has rank \\(1\\)).■

3. The dual space of a vector space

Definition 1. Let \\(\\mathcal{V}\\) be an arbitrary vector space; we denote by \\(\\mathcal{V}^{*}\\) the set of all linear functions defined on \\(\\mathcal{V}\\). If in \\(\\mathcal{V}^{*}\\), \\(0\\), \\(f+g\\), and \\(af\\) are defined by \\(f(x)=0\\), \\(f(x)+g(x)\\), and \\(af(x)\\) respectively, then \\(\\mathcal{V}^{*}\\) becomes a vector space: we call \\(\\mathcal{V}^{*}\\) the dual space of \\(\\mathcal{V}\\).

In the present chapter we shall discuss the theory of dual spaces. We call attention to the fact that all our definitions and theorems will be phrased without reference to any basis or coordinate system and that, although we shall make liberal use of bases, we use them only when that is unavoidable: namely in considerations of dimensionality, where bases enter by definition. Through out this chapter we shall mean by a basis a linear basis, i.e., a maximal set of linearly independent elements: in case \\(\\mathcal{V}\\) is an inner product space we shall in each case specify whether or not we need an orthogonal basis.

If \\(\\mathcal{V}\\) is of dimension \\(n\\) so is \\(\\mathcal{V}^{*}\\). For let \\(\\{x_{i}\\}\\) be a basis in \\(\\mathcal{V}\\). For each \\(i=1,2, \\ldots, n\\) we may define a linear function \\(f_{i}(x)\\) by the requirement that \\(f_{i}(x_{j})=\\delta_{ij}\\). Then \\(\\sum_{i} a_{i} f_{i}=0\\), i.e., \\(\\sum_i a_{i} f_{i}(x) \\equiv 0\\) implies \\[ 0=\\sum_i a_{i} f_{i}(x_{j})=\\sum_i a_{i} \\delta_{ij}=a_{j}, \\] so that the \\(f_{i}\\) are linearly independent. Moreover if \\(f\\) is arbitrary in \\(\\mathcal{V}^{*}\\) and \\(x=\\sum_{j} c_{j} x_{j}\\) in \\(\\mathcal{V}\\) then \\(f(x)=\\sum_{j} c_{j} f(x_{j})\\) and \\(f_{i}(x)=\\sum_{j} c_{j} f_{i}(x_{j})=c_{i}\\), so that \\(f(x)=\\sum_{j} f(x_{j}) f_{j}(x)\\). In other words \\(\\{f_{j}\\}\\) is a basis in \\(\\mathcal{V}^{*}\\), so that \\(\\mathcal{V}^{*}\\) has dimension \\(n\\).

Since \\(\\mathcal{V}\\) and \\(\\mathcal{V}^{*}\\) are both \\({n}\\) dimensional vector spaces it is possible, in many ways, to set up a one to one correspondence between them that preserves \\(0\\), sum, and scalar product: in other words \\(\\mathcal{V}\\) and \\(\\mathcal{V}^{*}\\) are isomorphic. These isomorphisms, however, are perfectly arbitrary and yield no information about the structure of vector spaces. If, however, we consider not \\(\\mathcal{V}^{*}\\) but its dual space, which we may denote by \\(\\mathcal{V}^{**}\\), (i.e., \\(\\mathcal{V}^{**}\\) is the set of all linear functions \\(X(f)\\) defined on \\(\\mathcal{V}^*\\)), then it is possible to set up a ‘natural’ isomorphism between \\(\\mathcal{V}\\) and \\(\\mathcal{V}^{**}\\).

Given any vector \\(x_{0}\\) in \\(\\mathcal{V}\\), we make correspond to it an element \\(X_{0}\\) in \\(\\mathcal{V}^{* *}\\) by defining, for every \\(f=f(x)\\) in \\(\\mathcal{V}^{*}\\), \\(X_{0}(f)=f(x_{0})\\). (It is easy to verify that \\(X_{0}=X_{0}(f)\\) is indeed a linear function of \\(f\\).) The correspondence \\(x_{0} \\to X_{0}\\) is linear: i.e., if \\(x_{1} \\to X_{1}\\), \\(x_{2} \\to X_{2}\\), and \\(a_{1} x_{1}+a_{2} x_{2} \\to X\\), then \\(X=a_{1} X_{1}+a_{2} X_{2}\\). For, by definition we have for each \\(f\\) in \\(\\mathcal{V}^{*}\\),

\\[ \\begin{aligned} X(f)&=f(a_{1} x_{1}+a_{2} x_{2})\\\\ &=a_{1} f(x_{1})+a_{2} f(x_{2})\\\\ &=a_{1} X_{1}(f)+a_{2} X_{2}(f). \\end{aligned} \\]

We now show that the correspondence \\(x \\to X\\) is one to one. If \\(x_{1}\\) and \\(x_{2}\\) correspond to the same \\(X\\), then we have for every \\(f\\) in \\(\\mathcal{V}^{*}\\), \\(f(x_{1})=f(x_{2})\\), or \\(f(x_{1}-x_{2})=0\\). If we introduce, as above, a basis \\(\\{y_{j}\\}\\) in \\(\\mathcal{V}\\) and corresponding linear functions \\(f_{i}(y)\\) in \\(\\mathcal{V}^{*}\\), defined by \\(f_{i}(y_{j})=\\delta_{ij}\\), then we see that \\(f(x)=0\\) for all \\(f\\), where \\(x=\\sum_{j} c_{j} y_{j}\\), implies in particular that \\(f_{i}(x)=c_{i}=0\\), so that \\(x=0\\). Hence \\(x_{1}\\) and \\(x_{2}\\) can correspond to the same \\(X\\) only if \\(x_{1}=x_{2}\\), as was to be proved. Finally we remark that every \\(X\\) in \\(\\mathcal{V}^{* *}\\) corresponds in this correspondence to some \\(x\\) in \\(\\mathcal{V}\\). The simplest proof of this fact is that since \\(\\mathcal{V}\\) and therefore \\(\\mathcal{V}^{*}\\) are \\({n}\\) dimensional vector spaces the dual space \\(\\mathcal{V}^{**}\\) of \\(\\mathcal{V}^{*}\\) is also \\({n}\\) dimensional. Hence if we can exhibit \\({n}\\) linearly independent elements of \\(\\mathcal{V}^{* *}\\) which do correspond to elements of \\(\\mathcal{V}\\) the desired result will follow. Let \\(\\{x_{i}\\}\\) be a basis in \\(\\mathcal{V}\\): then \\(X_{i}=X_{i}(f)=f(x_{i})\\) is a set of linearly independent elements, and therefore a basis, in \\(\\mathcal{V}^{* *}\\). For \\(\\sum_i a_{i} X_{i}=0\\), i.e., \\(\\sum_i a_{i} X_{i}(f)=0\\) for all \\(f\\), implies that

\\[ 0=\\sum_i a_{i} f(x_{i})=f\\Big(\\sum_i a_{i} x_{i}\\Big), \\]

whence, as above, \\(\\sum_i a_{i} x_{i}=0\\) and therefore \\(a_{i}=0\\).

Thus the correspondence \\(x \\to X\\) is an isomorphism, the so-called ‘natural isomorphism’, between \\(\\mathcal{V}\\) and \\(\\mathcal{V}^{* *}\\).

4. The dual space of an inner product space

The considerations of the preceding section apply, of course, to inner product spaces. In the case of inner product spaces, however, It is not necessary to go to \\(\\mathcal{U}^{**}\\): we shall establish a natural correspondence between \\(\\mathcal{U}\\) and \\(\\mathcal{U}^{*}\\).

Let \\(\\mathcal{U}\\) be an \\(n\\) dimensional inner product space and \\(\\mathcal{U}^{*}\\) its dual space. The theorem on the representation of linear functions (cf. (I.15)) shows that every \\(f=f(x)\\) in \\(\\mathcal{U}^{*}\\) has the form \\(f(x) = \\langle x, y \\rangle\\). This relation establishes a correspondence, which we already know to be one to one, between \\(f(x)\\) in \\(\\mathcal{U}^{*}\\) and \\(y\\) in \\(\\mathcal{U}\\). If in this correspondence \\(f_{i}\\) corresponds to \\(y_{i}\\), \\(i=1,2\\), then we have \\[ \\begin{aligned} f(x) &= a_{1} f_{1}(x)+a_{2} f_{2}(x)\\\\ &=a_{1} \\langle x, y_{1} \\rangle + a_{2} \\langle x, y_2 \\rangle\\\\ &=\\langle x, \\bar{a}_{1} y_{1}+\\bar{a}_{2} y_{2} \\rangle, \\end{aligned} \\] so that \\(f\\) corresponds to \\(y=\\bar{a}_{1} y_{1}+\\bar{a}_{2} y_{2}\\). Thus the correspondence \\(f(x) \\to y\\) is not an isomorphism but a conjugate isomorphism between \\(\\mathcal{U}^{*}\\) and \\(\\mathcal{U}\\).

This correspondence can also be used to define an inner product in \\(\\mathcal{U}^{*}\\). At first glance it might seem plausible to define \\(\\langle f_{1}, f_{2} \\rangle\\) to be \\(\\langle y_{1}, y_{2} \\rangle\\), where \\(f_{i}(x) = \\langle x, y_{i} \\rangle\\), but due to the fact that the correspondence is a conjugate isomorphism we have the relation

\\[ a\\langle f_{1}, f_{2} \\rangle \\neq \\langle a f_{1}, f_{2} \\rangle = \\bar{a}\\langle f_{1}, f_{2} \\rangle \\]

so that this definition does not satisfy the requirements of the definition of an inner product in (I.3). If, however, we define \\(\\langle f_{1}, f_{2} \\rangle = \\langle y_{2}, y_{1} \\rangle\\) then it is readily verified that \\(\\langle f_{1}, f_{2} \\rangle\\) is an inner product in \\(\\mathcal{U}^{*}\\) so that \\(\\mathcal{U}^{*}\\) is an inner product space.

5. Reflexivity of inner product spaces

If we apply the results of the preceding section not to the inner product space \\(\\mathcal{U}\\) but to its dual \\(\\mathcal{U}^{*}\\) we obtain a conjugate isomorphism between \\(\\mathcal{U}^{*}\\) and \\(\\mathcal{U}^{* *}\\). Thereby we have induced a one to one correspondence between \\(\\mathcal{U}\\) itself and \\(\\mathcal{U}^{* *}\\): it is readily verified (since the operation of conjugation is involutory) that this correspondence is an isomorphism. We now show that this isomorphism is the same as the natural isomorphism between \\(\\mathcal{U}\\) and \\(\\mathcal{U}^{*}\\) described in (IV.3). Let \\(y_{0}\\) be an arbitrary vector in \\(\\mathcal{U}\\); to it there corresponds an element \\(f_{0}(x) = \\langle x, y_{0} \\rangle\\) in \\(\\mathcal{U}^{*}\\); to this element, in turn, there corresponds the element \\(Y_{0}(f) = \\langle f, f_{0} \\rangle\\) in \\(\\mathcal{U}^{* *}\\). We must show that \\(Y_{0}(f)=f(y_{0})\\). Let \\(f=f(x)= \\langle x, y \\rangle\\) be an arbitrary element of \\(\\mathcal{U}^{*}\\); we have \\[ Y_{0}(f) = \\langle f, f_{0} \\rangle = \\langle y_{0}, y \\rangle = f(y_{0}), \\] as was to be proved.

6. Direct sum of vector spaces

Definition 2. If \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are arbitrary vector spaces we define the direct sum, \\(\\mathcal{W} = \\mathcal{U} \\oplus \\mathcal{V}\\) to be the set of all pairs \\((x, y)\\) with \\(x\\) in \\(\\mathcal{U}\\) and \\(y\\) in \\(\\mathcal{V}\\).

If in \\(\\mathcal{W}\\) we define \\[ \\begin{aligned} 0 &= (0, 0),\\\\ (x_{1}, y_{1})+(x_{2}, y_{2})&=(x_{1}+x_{2}, y_{1}+y_{2}),\\\\ a(x, {y})&=(a x, a y), \\end{aligned} \\] then \\(\\mathcal{W}\\) becomes a vector space. If, moreover, \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are inner product spaces we may define in \\(\\mathcal{W}\\), \\(\\big\\langle (x_{1}, y_{1}), (x_{2}, y_{2})\\big\\rangle = \\langle x_{1}, x_{2} \\rangle + \\langle y_{1}, y_{2} \\rangle\\) and \\(\\mathcal{W}\\) becomes thereby an inner product space. In fact although for vector spaces this definition yields something new, for inner product spaces it can be subsumed under the discussion of the projection theorem (I.13). In other words \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) can be thought of as two orthogonal linear subspaces in \\(\\mathcal{W}\\), and in an arbitrary inner product space a linear subspace and its orthogonal complement are a decomposition of the space into a direct sum.

If \\(A\\) and \\(B\\) are linear transformations in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively we may define a linear transformation \\(C\\), the direct sum of \\(A\\) and \\(B\\), in \\(\\mathcal{W}=\\mathcal{U} \\oplus \\mathcal{V}\\) by \\(C(x, y) = (A x, B y)\\). It is easy to discuss the matricial representation of \\(C\\), its relation to addition, multiplication, scalar multiplication, \\(0\\), \\(1\\), inverse, dual, etc. We omit this discussion here, and merely state without proof two propositions that will be useful to us later.

If \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) have dimensions \\(n\\) and \\(m\\) respectively the dimension of \\(\\mathcal{W}\\) is \\(n+m\\). If \\(\\{x_{i}\\}\\) and \\(\\{y_{j}\\}\\) are bases in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively then the totality of all vectors of either of the two forms \\((x_{i}, 0)\\) or \\((0, y_{j})\\) is a basis in \\(\\mathcal{W}\\). If the matrix of the direct sum transformation \\({C}\\) is computed in this basis it will have the form

\\[ \\begin{bmatrix} {[A]} & 0 \\\\ 0 & {[B]} \\end{bmatrix} \\]

where \\([A]\\) and \\([B]\\) are the matrices of \\(A\\) and \\(B\\) in the bases \\(\\{x_{i}\\}\\) and \\(\\{y_{j}\\}\\) respectively, and where the zeros represent rectangular blocks each element of which is zero.

The most general linear function \\(f(x, y)\\) on \\(\\mathcal{W} = \\mathcal{U} \\oplus \\mathcal{V}\\) is of the form \\(f(x, y)=g(x)+h(y)\\), where \\(g\\) and \\(h\\) are linear functions in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\). In other words the dual space of a direct sum is the direct sum of the dual spaces.

7. Tensor product of vector spaces

The main purpose of this chapter is to define for vector spaces (and inner product spaces) the notion of a tensor product. In other words if \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are given vector spaces we shall define for every vector \\(x\\) in \\(\\mathcal{U}\\) and \\(y\\) in \\(\\mathcal{V}\\) a product \\(x \\otimes y\\), which is to be an element of a suitable vector space, in such a way that \\(x \\otimes y\\) depends linearly on either variable if the other one is fixed and so that (in case \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are inner product spaces) we have \\[ \\langle x_{1} \\otimes y_{1}, x_{2} \\otimes y_{2} \\rangle = \\langle x_{1}, x_{2} \\rangle \\langle y_{1}, y_{2} \\rangle. \\]

In order to clarify the definition we shall give, we proceed heuristically on the basis of the proposition (ii) in the preceding section. If we denote the (as yet undefined) tensor product of \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) by \\(\\mathcal{W} = \\mathcal{U} \\otimes \\mathcal{V}\\), me may expect that \\(\\mathcal{W}^{*}=\\mathcal{U}^{*} \\otimes \\mathcal{V}^{*}\\). Since it is technically easier to do so, instead of defining \\(\\mathcal{W}\\) itself we shall instead define \\(\\mathcal{W}^{*}\\); we shall then write, by definition, \\(\\mathcal{W}=(\\mathcal{W}^{*})^{*}\\). Also we may expect that if \\(f(x)\\) and \\(g(y)\\) are linear functions in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively then it is their product, \\(f(x) g(y)\\), that should in some sense be the general element of \\(\\mathcal{W}^{*}\\). This product is a function \\(h(x, y)\\), defined for \\(x\\) in \\(\\mathcal{U}\\) and \\(y\\) in \\(\\mathcal{V}\\), with the property that for each fixed value of one variable it is a linear function of the other: in other words \\(h(x, y)\\) is a bilinear function of \\(x\\) and \\(y\\). This discussion is meant to motivate the formal work that we begin in the next paragraph.

Let \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) be vector spaces of dimensions \\(n\\) and \\(m\\) respectively; we denote by \\(\\mathcal{W}^{*}\\) the set of all bilinear functions \\(f(x, y)\\) defined for \\(x\\) in \\(\\mathcal{U}\\) and \\(y\\) in \\(\\mathcal{V}\\). Let \\(\\mathcal{W}\\) be the dual space of \\(\\mathcal{W}^{*}\\) (i.e., \\(\\mathcal{W}\\) is the set of all linear functions \\(z(f)\\) defined for \\(f\\) in \\(\\mathcal{W}^{*}\\)): we call \\(\\mathcal{W} = \\mathcal{U} \\otimes \\mathcal{V}\\) the tensor product of \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\). To every pair \\((x_{0}, y_{0})\\) of vectors with \\(x_{0}\\) in \\(\\mathcal{U}\\) and \\(y_{0}\\) in \\(\\mathcal{V}\\) we make correspond the element \\(z_{0}(f)\\) in \\(\\mathcal{W}\\) defined by \\(z_{0}(f)=f(x_{0}, y_{0})\\). (It is easy to verify that \\(z_{0}(f)\\) is a linear function of \\(f\\).) We write \\(z_{0}=x_{0} \\otimes y_{0}\\) and call \\(z_{0}\\) the tensor product of \\(x_{0}\\) and \\(y_{0}\\). We shall consistently use the notation \\(x\\) for vectors of \\(\\mathcal{U}\\), \\(y\\) for vectors of \\(\\mathcal{V}\\), and \\(z\\) for vectors of the vector space \\(\\mathcal{W}\\).

8. Dimension of a tensor product

We observe that the dimension of \\(\\mathcal{W}^{*}\\) is \\({nm}\\). For, exactly as in (IV.3) above, we may choose bases \\(\\{x_{i}\\}\\) and \\(\\{y_{j}\\}\\) in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively, and then we may find bilinear functions \\(f_{\\alpha \\beta}(x, y)\\) subject to the requirement that \\(f_{\\alpha \\beta}(x_{i}, y_{j})=\\delta_{i \\alpha} \\delta_{j \\beta}\\). It is then easy to show that the \\(f_{\\alpha \\beta}\\) are linearly independent and that every bilinear function is a linear combination of them.

We shall also need the fact that the elements \\(z_{ij}=x_{i} \\otimes y_{j}\\) of \\(\\mathcal{W}\\) are a basis in \\(\\mathcal{W}\\). According to the preceding paragraph we need only prove that they are linearly independent. If \\(\\sum_{ij} a_{ij} z_{ij}(f)=0\\) for all \\(f\\) then we should have, in particular, \\(\\sum_{ij} a_{ij} f_{\\alpha \\beta}(x_{i}, y_{j})=a_{\\alpha \\beta}=0\\) for all \\(\\alpha\\) and \\(\\beta\\), as was to be proved.

9. The dual of a tensor product

If \\(z_{1}=x_{1} \\otimes y_{1}\\) and \\(z_{2}=x_{2} \\otimes y_{2}\\) and if \\(z=(a_{1} x_{1}+a_{2} x_{2}) \\otimes y\\) then \\(z=a_{1} z_{1}+a_{2} z_{2}\\). For we have, for every bilinear function: \\(f(x, y)\\), \\[ \\begin{aligned} z(f)&=f(a_{1} x_{1}+a_{2} x_{2}, y)\\\\ &=a_{1} f(x_{1}, y)+a_{2} f(x_{2}, y)\\\\ &=a_{1} z_{1}(f)+a_{2} z_{2}(f). \\end{aligned} \\] Similarly we can show that \\[ x \\otimes (b_{1} y_{1}+b_{2} y_{2})=b_{1}(x \\otimes y_{1})+b_{2}(x \\otimes y), \\] so that \\(x \\otimes y\\) depends linearly on each of its factors when the other is held fixed. It follows from the preceding paragraph that every element in \\(\\mathcal{W}\\) is a sum of tensor products \\(x \\otimes y\\) (not necessarily uniquely). It is also easy to prove, using the bilinear character of \\(x \\otimes y\\), that every linear function of \\(z\\) (i.e., every element in the dual space of \\(\\mathcal{W}\\)) is a bilinear function of \\(x\\) and \\(y\\) and consequently a sum of products of the form \\(f(x) g(y)\\), where \\(f\\) and \\(g\\) are linear functions defined on \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively. Hence for general vector spaces our definition of tensor product fulfills the conditions (heuristically derived above) of our program. Before investigating the relation of tensor product spaces to linear transformations, we examine the situation in inner product spaces.

10. Tensor product of inner product spaces

If \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are inner product spaces the construction of the preceding sections applies unaltered: the only new problem is to introduce into the tensor product \\(\\mathcal{W} = \\mathcal{U} \\otimes \\mathcal{V}\\) an inner product related to the given inner products in the factor spaces in a suitable way. It is technically easier to define inner product not in \\(\\mathcal{W}\\) but in \\(\\mathcal{W}^{*}\\) and then apply the general theory of duals of inner product spaces to find a tensor product in \\(\\mathcal{W}\\).

If \\(f=f(x, y)\\) is any element of \\(\\mathcal{W}^*\\), \\(f\\) can be written as a sum of products of the form \\(g(x) h(y)\\), or, since \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) are inner product spaces, \\(f\\) can be written as a sum of expressions of the form \\(\\langle x, x_{0} \\rangle \\langle y, y_{0} \\rangle\\). Hence if \\(f^{\\prime}\\) and \\(f^{\\prime \\prime}\\) are any two elements of \\(\\mathcal{W}^{*}\\) we may write

\\[ f^{\\prime}(x, y)=\\sum_i\\langle x, x_{i}^{\\prime} \\rangle \\langle y, y_{i}^{\\prime} \\rangle \\quad \\text{ and } \\quad f^{\\prime \\prime}(x, y)=\\sum_{j}\\langle x, x_{j}^{\\prime\\prime}\\rangle \\langle y, y_{j}^{\\prime\\prime} \\rangle. \\]

We write, by definition, \\[ \\langle f^{\\prime}, f^{\\prime\\prime} \\rangle = \\sum_i \\sum_{j}\\langle x_{j}^{\\prime\\prime}, x_{i}^{\\prime} \\rangle \\langle y_{j}^{\\prime\\prime}, y_{i}^{\\prime} \\rangle. \\]

(The conjugate nature of the relation between vectors and linear functions again necessitates putting \\(x_{j}^{\\prime \\prime}\\) before \\(x_{i}^{\\prime}\\).) Before we can even start to prove that this definition fulfills the conditions of the definition of an inner product, we must prove that it defines \\(\\langle f^{\\prime}, {f}^{\\prime \\prime} \\rangle\\) independently of the representations as sums. To do this we observe that \\[ \\sum_j \\langle x_j^{\\prime\\prime}, x_i^\\prime \\rangle \\langle y_j^{\\prime\\prime}, y_i^\\prime \\rangle = \\overline{f^{\\prime\\prime}(x_i^\\prime, y_i^\\prime)}, \\] so that \\[ \\langle f^\\prime, f^{\\prime\\prime} \\rangle = \\sum_i \\overline{f^{\\prime\\prime}(x_i^\\prime, y_i^\\prime)}, \\] whence \\(\\langle f^{\\prime}, {f}^{\\prime \\prime} \\rangle\\) is independent of the particular representation of \\(f^{\\prime \\prime}\\). Since, moreover, in any given representations of \\(f^\\prime\\) and \\(f^{\\prime\\prime}\\), \\(\\langle f^{\\prime}, f^{\\prime\\prime} \\rangle = \\overline{\\langle f^{\\prime\\prime}, f^{\\prime} \\rangle}\\), it follows that \\(\\langle f^\\prime, f^{\\prime\\prime} \\rangle\\) is also independent of the representation of \\(f^{\\prime}\\).

It is easy to verify that the expression \\(\\langle f^\\prime, f^{\\prime\\prime} \\rangle\\) is linear in \\(f^\\prime\\), conjugate linear in \\({f}^{\\prime \\prime}\\), and Hermitian symmetric. It remains to prove that it is positive definite: i.e., that \\(\\langle f^{\\prime}, f^{\\prime} \\rangle \\geq 0\\) for all \\(f^{\\prime}\\), and that \\(\\langle f^{\\prime}, f^{\\prime} \\rangle = 0\\) if and only if \\(f^{\\prime}=0\\). This surprisingly, is not trivial: it requires Schur’s theorem, proved in (IV.2).

We have \\[ \\langle f^{\\prime}, f^{\\prime} \\rangle = \\sum_i^{p} \\sum_{j}^{p}\\langle x_{j}^{\\prime}, x_{i}^{\\prime} \\rangle \\langle y_{j}^{\\prime}, y_{i}^{\\prime} \\rangle. \\]

Let \\(a_{1}, a_{2}, \\ldots, a_{p}\\) be arbitrary complex numbers. Then \\[ \\begin{aligned} \\sum_i \\sum_{j}\\langle x_{j}^{\\prime}, x_{i}^{\\prime} \\rangle \\bar{a}_{i} a_{j} &= \\Big\\langle \\sum_{j} a_{j} x_{j}^{\\prime}, \\sum_i a_{i} x_{i}^{\\prime} \\Big\\rangle\\\\ &= \\Big|\\sum_i a_{i} x_{i}^{\\prime}\\Big|^{2}\\\\ &\\geq 0, \\end{aligned} \\] so that the matrix whose general element is \\(\\langle x_{j}^{\\prime}, x_{i}^{\\prime} \\rangle\\) is non-negative. Similarly we may show that the matrix whose general element is \\(\\langle y_{j}^{\\prime}, y_{i}^\\prime \\rangle\\) is non-negative; it follows from Schur’s theorem that the matrix whose general element is the product \\(\\langle x_{j}^{\\prime}, x_{i}^{\\prime} \\rangle \\langle y_{j}^{\\prime}, y_{i}^{\\prime} \\rangle\\) is also non-negative. Hence \\[ \\sum_i \\sum_{j}\\langle x_{j}^{\\prime}, x_{i}^{\\prime} \\rangle \\langle y_{j}^{\\prime}, y_{i}^{\\prime} \\rangle a_{i} \\bar{a}_{j} \\geq 0 \\] for every choice of the complex numbers \\((a_{1}, a_{2}, \\ldots, a_{p})\\): choosing \\(a_{i}=1\\) for all \\(i\\) proves that \\(\\langle f^{\\prime}, f^{\\prime} \\rangle \\geq 0\\).

In order to prove that \\(\\langle f^{\\prime}, f^{\\prime} \\rangle = 0\\) implies \\(f^{\\prime}=0\\) we proceed as follows. For the expression \\(\\langle f^{\\prime}, f^{\\prime} \\rangle\\), which now has all other properties of an inner product, we may prove the Schwartz inequality as in (I.4): \\[ |\\langle f^{\\prime}, f^{\\prime\\prime} \\rangle| \\leq \\Big(\\langle f^\\prime, f^{\\prime} \\rangle \\, \\langle f^{\\prime\\prime}, f^{\\prime\\prime} \\rangle \\Big)^{1 / 2}. \\]

It follows that the vanishing of \\(\\langle f^{\\prime}, f^{\\prime} \\rangle\\) implies the vanishing of \\(\\langle f^{\\prime}, f^{\\prime \\prime} \\rangle\\) for all \\(f^{\\prime \\prime}\\). Let \\(x^{\\prime \\prime}\\) and \\(y^{\\prime \\prime}\\) be arbitrary vectors and take, in particular, \\(f^{\\prime\\prime}=f^{\\prime\\prime}(x, y)= \\langle x, x^{\\prime\\prime} \\rangle \\langle y, y^{\\prime\\prime} \\rangle\\). The vanishing of \\(\\langle f^{\\prime}, f^{\\prime\\prime} \\rangle\\) implies that \\[ \\begin{aligned} 0 &= \\langle f^{\\prime}, f^{\\prime\\prime} \\rangle\\\\ &= \\sum_i \\langle x^{\\prime\\prime}, x_{i}^{\\prime} \\rangle \\langle y^{\\prime\\prime}, y_{i}^{\\prime} \\rangle\\\\ &= f^{\\prime}(x^{\\prime\\prime}, y^{\\prime\\prime}); \\end{aligned} \\] hence the vanishing of \\(\\langle f^{\\prime}, f^{\\prime\\prime} \\rangle\\) for all \\(f^{\\prime \\prime}\\) implies that \\(f^{\\prime}(x^{\\prime\\prime}, y^{\\prime \\prime})=0\\), for every pair \\(x^{\\prime \\prime}\\), \\(y^{\\prime \\prime}\\) of vectors, or, in other words, that \\(f^{\\prime}=0\\).

This concludes the introduction of an inner product in \\(\\mathcal{W}^{*}\\). Applying the results of (IV.4) we obtain an inner product in the dual space \\(\\mathcal{W}\\) of \\(\\mathcal{W}^{*}\\), so that \\(\\mathcal{W}\\) becomes an inner product space.

11. The inner product in a tensor product

It is now easy to prove that the inner product defined in \\(\\mathcal{W}\\) has the property that

\\[ \\langle x^{\\prime} \\otimes y^{\\prime}, x^{\\prime \\prime} \\otimes y^{\\prime \\prime} \\rangle = \\langle x^{\\prime}, x^{\\prime \\prime} \\rangle \\langle y^{\\prime}, y^{\\prime \\prime} \\rangle. \\]

We write \\[ z^{\\prime}=x^{\\prime} \\otimes y^{\\prime}, \\quad \\text{ and } \\quad z^{\\prime \\prime}=x^{\\prime \\prime} \\otimes y^{\\prime \\prime}, \\] and we define \\[ z_{0}^{\\prime}=z_{0}^{\\prime}(f)= \\langle f, f^{\\prime} \\rangle, \\qquad z_{0}^{\\prime\\prime}=z_{0}^{\\prime\\prime}(f)= \\langle f, f^{\\prime\\prime} \\rangle, \\] where \\(f^{\\prime}\\) and \\(f^{\\prime\\prime}\\) are the particular bilinear functions defined by \\[ f^{\\prime}(x, y)= \\langle x, x^{\\prime} \\rangle \\langle y, y^{\\prime} \\rangle \\quad \\text{and} \\quad f^{\\prime \\prime}(x, y)= \\langle x, x^{n} \\rangle \\langle y, y^{n} \\rangle. \\]

For an arbitrary \\(f=\\sum_{i} \\langle x, x_{i} \\rangle \\langle y, y_{i} \\rangle\\) we have \\[ \\begin{aligned} z_{0}^{\\prime}(f) &= \\sum_i \\langle x^{\\prime}, x_{i} \\rangle \\langle y^{\\prime}, y_{i} \\rangle\\\\ &= f(x^{\\prime}, y^{\\prime})\\\\ &= z^{\\prime}(f) \\end{aligned} \\] and \\[ \\begin{aligned} z_{0}^{\\prime\\prime}(f) &= \\sum_i \\langle x^{\\prime\\prime}, x_{i} \\rangle \\langle y^{\\prime\\prime}, y_{i} \\rangle\\\\ &= f(x^{\\prime\\prime}, y^{\\prime\\prime})\\\\ &= z^{\\prime\\prime}(f). \\end{aligned} \\] (This is similar to the proof in (IV.5) if the equality of the two natural correspondences between an inner product space and its second dual.) Hence we have, finally, \\[ \\langle z^{\\prime}, z^{\\prime\\prime} \\rangle = \\langle z_{0}^{\\prime}, z_{0}^{\\prime\\prime} \\rangle = \\langle f^{\\prime\\prime}, f^{\\prime} \\rangle = \\langle x^{\\prime}, x^{\\prime\\prime} \\rangle \\langle y^{\\prime}, y^{\\prime \\prime} \\rangle, \\] as was to be proved.

The last proved fact justifies the terminology of tensor product and describes completely the structure of \\(\\mathcal{W}\\) and its relation to \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\). It follows also that if \\(\\{x_{i}\\}\\) and \\(\\{y_{\\alpha}\\}\\) are orthogonal bases in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively then \\[ \\langle x_{i} \\otimes y_{\\alpha}, x_{j} \\otimes y_{\\beta} \\rangle = \\langle x_{i}, x_{j} \\rangle \\langle y_{\\alpha}, y_{\\beta} \\rangle = \\delta_{ij} \\delta_{\\alpha \\beta} \\] so that the \\(x_{i} \\otimes y_{\\alpha}\\) form an orthonormal set in \\(\\mathcal{W}\\). Since we have already seen that they form a maximal linearly independent set it follows that they are a complete orthonormal set, or an orthogonal basis, in \\(\\mathcal{W}\\).

12. Tensor product of transformations

We are now in a position to examine the relation of linear transformations to the theory of tensor products. If \\(A\\) and \\(B\\) are linear transformations defined in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively we define a linear transformation \\(C^{*}\\) in \\(\\mathcal{W}^{*}\\) by \\(C^{*} f(x, y)=f(A x, B y)\\), and then a linear transformation \\({C}\\) in \\(\\mathcal{W}\\) by \\({Cz}({f})=z(C^{*} f)\\). In brief: \\[ C\\Big(z\\big(f(x, y)\\big)\\Big) = z\\big(f(A x, B y)\\big). \\]

If we apply \\(C\\) to a particular \\(z\\) of the form \\(z=x \\otimes y\\) (i.e., \\(z(f)=f(x, y)\\)) we obtain \\[ C z=z\\big(f(A x, B y)\\big)= Ax \\otimes By. \\]

Since we have already remarked that every \\(z\\) is a sum of tensor products \\(x \\otimes y\\) the relation \\({Cz}={Ax} \\otimes {By}\\) completely characterizes \\(C\\). The linear transformation \\(C\\) in the space \\(\\mathcal{W}\\) is called the tensor product of the linear transformations \\(A\\) and \\(B\\), \\(C = A \\otimes B\\).

13. Kronecker products of matrices

Let \\(A\\) and \\(B\\) be linear transformations and \\(\\{x_{i}\\}\\) and \\(\\{y_{\\alpha}\\}\\) orthogonal bases in \\(\\mathcal{U}\\) and \\(\\mathcal{V}\\) respectively. We find the matrix of the linear transformation \\(C=A \\otimes B\\) in the orthogonal basis \\(\\{x_{i} \\otimes y_{\\alpha}\\}\\) of \\(\\mathcal{W}\\). Naturally the matrix depends on the way in which these \\(nm\\) vectors are ordered in a linear order: we suppose first that the order is the lexicographical one, i.e., \\[ x_1 \\otimes y_1, \\ldots, x_1 \\otimes y_m, \\quad x_2 \\otimes y_1, \\ldots, x_2 \\otimes y_m, \\quad \\ldots, \\quad x_n \\otimes y_1, \\ldots, x_n \\otimes y_m. \\]

We have \\[ \\begin{aligned} C(x_{j} \\otimes y_{\\beta}) &= A x_{j} \\otimes B y_{\\beta}\\\\ &= \\Big(\\sum_i a_{ij} x_{i}\\Big) \\otimes \\Big(\\sum_{\\alpha} b_{\\alpha \\beta} y_{\\alpha}\\Big)\\\\ &= a_{1j} b_{1\\beta}(x_{1} \\otimes y_{1}) + a_{1j} b_{2 \\beta}(x_{1} \\otimes y_{2}) + \\cdots + a_{1j} b_{m \\beta}(x_{1} \\otimes y_{m})+\\cdots \\end{aligned} \\]

so that the matrix of \\(C\\) has the form \\[ \\begin{bmatrix} a_{11}b_{11} & a_{11}b_{12} & \\cdots & a_{11}b_{1m} \\quad & a_{12}b_{11} & a_{12}b_{12} & \\cdots & a_{12}b_{1m} \\quad & \\cdots & \\cdots \\quad & a_{1n}b_{11} & a_{1n}b_{12} & \\cdots & a_{1n}b_{1m}\\\\ a_{11}b_{21} & a_{11}b_{22} & \\cdots & a_{11}b_{2m} \\quad & a_{12}b_{21} & a_{12}b_{22} & \\cdots & a_{12}b_{2m} \\quad & \\cdots & \\cdots \\quad & a_{1n}b_{21} & a_{1n}b_{22} & \\cdots & a_{1n}b_{2m}\\\\ \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\cdots & \\cdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots\\\\ a_{11}b_{m1} & a_{11}b_{m2} & \\cdots & a_{11}b_{mm} \\quad & a_{12}b_{m1} & a_{12}b_{m2} & \\cdots & a_{12}b_{mm} \\quad & \\cdots & \\cdots \\quad & a_{1n}b_{m1} & a_{1n}b_{m2} & \\cdots & a_{1n}b_{mm}\\\\ \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\,\\\\ a_{21}b_{11} & a_{21}b_{12} & \\cdots & a_{21}b_{1m} \\quad & a_{22}b_{11} & a_{22}b_{12} & \\cdots & a_{22}b_{1m} \\quad & \\cdots & \\cdots \\quad & a_{2n}b_{11} & a_{2n}b_{12} & \\cdots & a_{2n}b_{1m}\\\\ a_{21}b_{21} & a_{21}b_{22} & \\cdots & a_{21}b_{2m} \\quad & a_{22}b_{21} & a_{22}b_{22} & \\cdots & a_{22}b_{2m} \\quad & \\cdots & \\cdots \\quad & a_{2n}b_{21} & a_{2n}b_{22} & \\cdots & a_{2n}b_{2m}\\\\ \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\cdots & \\cdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots\\\\ a_{21}b_{m1} & a_{21}b_{m2} & \\cdots & a_{21}b_{mm} \\quad & a_{22}b_{m1} & a_{22}b_{m2} & \\cdots & a_{22}b_{mm} \\quad & \\cdots & \\cdots \\quad & a_{2n}b_{m1} & a_{2n}b_{m2} & \\cdots & a_{2n}b_{mm}\\\\ \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\,\\\\ \\vdots & \\vdots & \\vdots & \\vdots \\quad & \\vdots & \\vdots & \\vdots & \\vdots \\quad & \\ddots & \\ddots \\quad & \\vdots & \\vdots & \\vdots & \\vdots\\\\ \\vdots & \\vdots & \\vdots & \\vdots \\quad & \\vdots & \\vdots & \\vdots & \\vdots \\quad & \\ddots & \\ddots \\quad & \\vdots & \\vdots & \\vdots & \\vdots\\\\ \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\, & \\,\\\\ a_{n1}b_{11} & a_{n1}b_{12} & \\cdots & a_{n1}b_{1m} \\quad & a_{n2}b_{11} & a_{n2}b_{12} & \\cdots & a_{n2}b_{1m} \\quad & \\cdots & \\cdots \\quad & a_{nn}b_{11} & a_{nn}b_{12} & \\cdots & a_{nn}b_{1m}\\\\ a_{n1}b_{21} & a_{n1}b_{22} & \\cdots & a_{n1}b_{2m} \\quad & a_{n2}b_{21} & a_{n2}b_{22} & \\cdots & a_{n2}b_{2m} \\quad & \\cdots & \\cdots \\quad & a_{nn}b_{21} & a_{nn}b_{22} & \\cdots & a_{nn}b_{2m}\\\\ \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots \\quad & \\cdots & \\cdots \\quad & \\vdots & \\vdots & \\ddots & \\vdots\\\\ a_{n1}b_{m1} & a_{n1}b_{m2} & \\cdots & a_{n1}b_{mm} \\quad & a_{n2}b_{m1} & a_{n2}b_{m2} & \\cdots & a_{n2}b_{mm} \\quad & \\cdots & \\cdots \\quad & a_{nn}b_{m1} & a_{nn}b_{m2} & \\cdots & a_{nn}b_{mm}\\\\ \\end{bmatrix} \\] or, in a condensed notation whose meaning is clear, \\[ \\begin{bmatrix} a_{11}[B] & a_{12}[B] & \\cdots & a_{1 n}[B] \\\\ a_{21}[B] & a_{22}[B] & \\cdots & a_{2 n}[B] \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ a_{n 1}[B] & a_{n 2}[B] & \\cdots & a_{n n}[B] \\end{bmatrix}. \\]

If we had adopted, instead, the converse lexicographic ordering, i.e., \\[ x_1 \\otimes y_1, \\ldots, x_n \\otimes y_1, \\quad x_1 \\otimes y_2, \\ldots, x_n \\otimes y_2, \\quad \\ldots, \\quad x_1 \\otimes y_m, \\ldots, x_n \\otimes y_m. \\] we should have found the matrix of \\({C}\\) to be \\[ \\begin{bmatrix} b_{11}[A] & b_{12}[A] & \\ldots & b_{1 m}[A] \\\\ b_{21}[A] & b_{22}[A] & \\ldots & b_{2 m}[A] \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ b_{m 1}[A] & b_{m 2}[A] & \\ldots & b_{m m}[A] \\end{bmatrix}. \\]

The first of these two matrices is known as the Kronecker product, \\([A] \\otimes [B]\\), of \\([A]\\) and \\([B]\\) (in this order!); the second one is \\([B] \\otimes [A]\\). Since a permutation of the elements of an orthogonal basis is a trivial kind of change of basis (i.e. it is effected by a unitary transformation \\(U\\)) we obtain that \\[ \\big([A] \\otimes[B]\\big)=[U]\\big([B] \\otimes[A]\\big)[U]^{-1}. \\]

14. Properties of tensor product transformations

We now proceed to describe some of the elementary properties of tensor product transformations.

14.1. If \\(A=\\sum_i a_{i} A_{i}\\) and \\(B=\\sum_{j} b_{j} B_{j}\\) then \\(A \\otimes B=\\sum_i \\sum_{j} a_{i} b_{j}(A_{i} \\otimes B_{j})\\). For we have \\[ \\begin{aligned} (A \\otimes B)(x \\otimes y) &= Ax \\otimes By\\\\ &= \\Big(\\sum_i a_i A_i x\\Big) \\otimes \\Big(\\sum_j b_j B_j y\\Big)\\\\ &= \\sum_i \\sum_j a_i b_j (A_i \\otimes B_j)(x \\otimes y). \\end{aligned} \\]

14.2. If \\(A=A_{1} A_{2}\\) and \\(B=B_{1} B_{2}\\) then \\(A \\otimes B = (A_1 \\otimes B_1)(A_2 \\otimes B_2)\\). For \\[ \\begin{aligned} (A \\otimes B)(x \\otimes y) &= A x \\otimes B y\\\\ &= A_{1} A_{2} x \\otimes B_{1} B_{2} y\\\\ &= (A_{1} \\otimes B_{1})(A_{2} x \\otimes B_{2} y)\\\\ &= (A_{1} \\otimes B_{1})(A_{2} \\otimes B_{2})(x \\otimes y). \\end{aligned} \\] As immediate consequences of this result we obtain the formulas \\[ \\begin{aligned} (A \\otimes B) &= (A \\otimes 1)(1 \\otimes B) = (1 \\otimes B)(A \\otimes 1), \\\\ (A \\otimes B)^{-1} &= (A^{-1} \\otimes B^{-1}). \\end{aligned} \\]

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"dual-spaces-and-tensor-products","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]},{"_id":"66a0498e39e7e3d9dc7ad857","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Canonical Forms","sequence":5,"chapter_content":"

Throughout this chapter we shall be concerned with an \\(n\\) dimensional vector space \\(\\mathcal{V}\\). We shall not assume that it is a complex vector space: the scalar coefficients may lie in an arbitrary field. The treatment we shall give is taken from N. Jacobson’s Notes on Algebra.

1. The space of linear transformations

The set of all linear transformations defined on \\(\\mathcal{V}\\) is an \\(n^{2}\\) dimensional vector space. For if \\(\\{x_i\\}\\) is a basis in \\(\\mathcal{V}\\) we may define the linear transformations \\({E}_{ij}\\) in \\(\\mathcal{V}\\) by \\({E}_{ij} x_{k}=\\delta_{j k} x_{i}\\). (We have: \\[ E_{ij} E_{pq} x_k = \\delta_{qk} E_{ij} x_p = \\delta_{qk} \\delta_{jp} x_i, \\] so that \\(E_{ij} E_{pq} = \\delta_{jp} E_{iq}\\).) The \\({E}_{ij}\\) are linearly independent, for \\(\\sum_{ij}a_{ij} {E}_{ij} {x}=0\\) for all \\(x\\) implies \\[ \\begin{aligned} 0 &= \\sum_{ij} a_{ij} E_{ij} x_k\\\\ &= \\sum_{ij} a_{ij} \\delta_{jk} x_i\\\\ &= \\sum_i a_{ik} x_i, \\end{aligned} \\] whence \\(a_{ik} = 0\\) for all \\(i\\), \\(k\\). Moreover every linear transformation \\(A\\) is a linear combination of the \\(E_{ij}\\): if the matrix of \\(A\\) is \\((a_{ij})\\), so that \\(A x_{j}=\\sum_i a_{ij} x_{i}\\), then \\(A x_{j}=\\sum_i a_{ij} {E}_{ij} x_{j}\\), so that \\(A=\\sum_{ij}{a}_{ij} {E}_{ij}\\).

2. The minimal polynomial of a transformation

If \\(A\\) is an arbitrary linear transformation, the \\(n^{2}+1\\) linear transformations \\(1, A, A^{2}, \\ldots, A^{n^{2}}\\) are linearly dependent, from the result of the preceding section, so that there exists a polynomial \\(m(\\lambda)\\) of degree \\(\\leq n^{2}\\) for which \\(m(A)=0\\).

Definition 1. The polynomial \\(m(\\lambda)\\) of minimal degree \\(m\\) and leading coefficient \\(1\\) for which \\(m(A)=0\\) is the minimal polynomial of \\(A\\).

We observe that it is uniquely determined by the conditions we placed on it: if \\(f(\\lambda)\\) is such that \\(f(A)=0\\), we may divide \\(f(\\lambda)\\) by \\(m(\\lambda)\\) and obtain \\(f(\\lambda)=q(\\lambda) m(\\lambda)+r(\\lambda)\\), where the degree of \\(r(\\lambda)\\) is less than \\(m\\). From the relation \\(f(A)=q(A) m(A)+r(A)=0\\) we see that \\(r(A)=0\\), whence, by the properties of \\(m(\\lambda)\\), \\(r(\\lambda)\\) must be identically zero. Consequently \\(m(\\lambda)\\) divides any \\(f(\\lambda)\\) for which \\(f(\\lambda)=0\\): this proves that it is uniquely determined by our conditions. Moreover if \\(B={SA}{S}^{-1}\\), then \\(m(B)={S}m(A) S^{-1}=0\\), so that \\(m(\\lambda)\\) must be a divisor of the minimal polynomial of \\(B\\). Interchanging the roles of \\(A\\) and \\(B\\) we obtain the result that the minimal polynomials of two similar transformations (i.e., of two transformations \\(A\\) and \\(B\\) related by an equation of the form \\(B=S A S^{-1}\\)) are equal.

3. The order of a vector

Since for an arbitrary vector \\(x\\), the vectors \\(x, A x, A^{2} x, \\ldots, {A}^{n} x\\), being in number more than \\(n\\), are linearly dependent, there exists a polynomial \\(m_{x}(\\lambda)\\) for which \\(m_{x}(A) x=0\\). We verify easily, as in the previous section, that the requirements that \\(m_{x}(\\lambda)\\) have minimal degree and have leading coefficient one uniquely determine \\(m_{x}(\\lambda)\\): we call this polynomial the order of \\(x\\) (relative to \\(A\\)). Since, moreover, \\(m_{x}(\\lambda)\\) is a factor of every polynomial \\(f(\\lambda)\\) for which \\(f(A) x=0\\), the least common multiple (L.C.M.) of all \\({m}_{x}(\\lambda)\\) exists and is a factor of \\({m}(\\lambda)\\). If \\(\\{x_i\\}\\) is a basis in \\(\\mathcal{V}\\) and \\(m^{*}(\\lambda)= \\text{L.C.M.}\\, m_{x_{i}}(\\lambda)\\) then for every \\(x=\\sum_i a_{i} x_{i}\\) in \\(\\mathcal{V}\\) we have \\[ m^{*}(A) x=\\sum_i a_{i} m^{*}(A) x_{i}=0, \\] or \\(m^{*}(A)=0\\), so that \\(m^{*}(\\lambda)\\) is divisible by \\(m(\\lambda)\\). It follows that \\(m(\\lambda)\\) is exactly equal to \\(m^{*}(\\lambda)\\), or equivalently, to the L.C.M. of all \\(m_x(\\lambda)\\).

4. Quotient spaces

For an arbitrary linear subspace \\(M\\) in \\(\\mathcal{V}\\) we write \\(x_{1} \\equiv x_{2} \\,\\, (\\operatorname{mod} M)\\) if \\(x_{1}-x_{2}\\) is in \\(M\\). The set of all vectors \\(x\\) in \\(\\mathcal{V}\\), with the same meaning of addition and scalar multiplication as before, but with the notion of equality of two vectors redefined to mean their congruence \\(\\operatorname{mod} M\\), is a vector space: we denote it by \\(\\mathcal{V} / {M}\\). We prove that if the dimension of \\(M\\) is \\(r\\) then that of \\(\\mathcal{V} / M\\) is \\(n-r\\). For let \\(N\\) be an arbitrary linear subspace whose intersection with \\(M\\) consists of \\(0\\) alone and which together with \\(M\\) spans \\(\\mathcal{V}\\). If \\(x_{1}, x_{2}, \\ldots, x_{n-r}\\) is a basis in \\(N\\), then the \\(x_{i} \\operatorname{mod} M\\) are a basis in \\(\\mathcal{V} / M\\). If \\(\\sum_i a_{i} x_{i} \\equiv 0 \\,\\, (\\operatorname{mod} M)\\), i.e, if \\(\\sum_i a_{i} x_{i}\\) lies in \\(M\\), then, since \\(N \\cap M=\\{0\\}\\), we must have \\(\\sum_i a_{i} x_{i}=0\\), and the linear independence of the \\(x_{i}\\) implies that \\(a_{i}=0\\) for all \\(i\\). Hence the \\(x_{i} \\operatorname{mod} M\\) are linearly independent. To prove that they span \\(\\mathcal{V} / M\\) we observe that every \\(x\\) in \\(\\mathcal{V}\\) may be written in the form \\(x=y+z\\), with \\(y\\) in \\(M\\) and \\(z\\) in \\(N\\). In other words, since \\(x-z=y\\) is in \\({M}\\), every \\(x\\) is congruent, \\(\\operatorname{mod} {M}\\), to some \\({z}\\) in \\({N}\\), as was to be proved.

If \\(A\\) is a linear transformation which leaves \\(M\\) invariant, i.e., for which \\(x\\) in \\(M\\) implies \\(A x\\) in \\(M\\), then \\(x \\equiv y \\,\\, (\\operatorname{mod} M)\\) implies \\(A x \\equiv A y \\,\\, (\\operatorname{mod} M)\\), so that \\(A\\) induces a linear transformation in \\(\\mathcal{V} / M\\). The space \\(\\mathcal{V} / M\\) is the quotient space of \\(\\mathcal{V}\\) modulo \\(M\\) (or \\(\\mathcal{V}\\) by \\(M\\)).

5. Cyclic spaces

Definition 2. For any vector \\(x\\), and given linear transformation \\(A\\), we denote by \\(\\{x\\}\\) the linear subspace spanned by all vectors \\(x, A x, A^{2} x, A^{3} x, \\ldots\\): we call such linear subspaces cyclic.

If the order of \\(x\\) is \\(m(\\lambda)=\\lambda^{m}-\\sum_{i=0}^{m-1} a_{i} \\lambda^{i}\\), then \\(x, A x, \\ldots, A^{m-1} x\\) are linearly independent, but together with \\({A}^{m} x\\) they form a linearly dependent set, so that the dimension of \\(\\{x\\}\\) is \\(m\\). The matrix of \\(A\\) in the space \\(\\{x\\}\\) with respect to the basis \\(x, A x, \\ldots, A^{m-1} x\\) is \\[ \\begin{bmatrix} 0 & 1 & 0 & \\cdots & 0 \\\\ 0 & 0 & 1 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & 1 \\\\ a_{1} & a_{2} & a_{3} & \\cdots & a_{m} \\end{bmatrix}. \\]

We note that \\(m(\\lambda)\\) is the minimal polynomial of \\(A\\) in \\(\\{x\\}\\) and therefore depends only on the subspace \\(\\{x\\}\\) and not on the particular generator \\(x\\).

6. Vectors of maximal order

The fundamental lemma on which the work of this chapter is based is the following.

Lemma 1. Given any linear transformation \\(A\\), there exists a cyclic subspace \\(\\{x\\}\\) whose order is the minimal polynomial, \\(m(\\lambda)\\), of \\(A\\).

Proof. For let \\(\\{x_i\\}\\) be a basis in \\(\\mathcal{V}\\) and let \\(m_{i}(\\lambda)\\) be the order of \\(x_{i}\\). If \\[ m_{i}(\\lambda)=p_{1}(\\lambda)^{e_{i1}} \\ldots p_{r}(\\lambda)^{e_{ir}} \\] is the factorization of \\(m_{i}(\\lambda)\\) into powers of irreducible polynomials and if we write \\(e_{j}=\\max (e_{1j}, \\ldots, e_{n j})\\), for \\(j = 1, 2, \\ldots, r\\), then \\[ m(\\lambda) = \\text{L.C.M.} \\, m_{i}(\\lambda) = p_{1}(\\lambda)^{e_{1}} \\ldots p_{r}(\\lambda)^{e_{r}} \\] is the minimal polynomial of \\(A\\). We write \\[ y_{ij}=p_{1}(A)^{e_{i1}} \\ldots p_{j-1}(A)^{e_{i(j - 1)}} p_{j+1}(A)^{e_{i(j + 1)}} \\ldots p_{r}(A)^{e_{ir}}x_{i}; \\] then the order of \\(y_{ij}\\) is \\({p}(A)^{e_{ij}}\\). Hence among the \\({y}_{ij}\\) we may find vectors \\({y}_{1}, {y}_{2}, \\ldots {y}_{{r}}\\) whose orders are, respectively, \\({p}_{1}(\\lambda)^{e_{1}}, {p}_{2}(\\lambda)^{e_2}, \\ldots, {p}_{{r}}(\\lambda)^{e_r}\\). We shall prove that \\({y}={y}_{1}+{y}_{2}+\\cdots+{y}_{{r}}\\) has the order \\(m(\\lambda)\\).

For any polynomial \\(f(\\lambda)\\), denote by \\(N(f)\\) the linear subspace of vectors \\(x\\) for which \\(f(A) x=0\\). It is easy to verify that \\(N(f)\\) is invariant under \\({A}\\). We assert that if \\(f(\\lambda)\\) and \\(g(\\lambda)\\) are relatively prime then \\(N(f)\\) and \\(N(g)\\) have only \\(0\\) in common. For if \\(f(A) x=g(A) x=0\\) then \\(m_{x}(\\lambda)\\) must be a divisor of both \\(f(\\lambda)\\) and \\(g(\\lambda)\\), whence \\(m_{x}(\\lambda)=1\\), and \\(x=0\\). Hence, in our case, the intersection of any two of the subspaces \\[ N_{i} = N\\big(p_i(\\lambda)^{e_i}\\big) \\] consists of \\(0\\) alone, and the \\({N}_{i}\\) are invariant under \\(A\\). It follows (since \\(y_{i}\\) is in \\(N_{i}\\)) that \\[ f(A) y=f(A) y_{1}+f(A) y_{2}+\\cdots+f(A) y_{r}=0 \\] implies \\(f(A) y_{j}=0\\), so that \\(f(\\lambda)\\) must be divisible by \\(p_{j}(\\lambda)^{e_j}\\), \\(j=1, \\ldots, r\\), and therefore by \\(m(\\lambda)\\). This completes the proof of our lemma.■

Since the order of any vector is a polynomial of degree \\(\\leq n\\), we have proved that the degree of \\(m(\\lambda)\\) is \\(\\leq n\\): if it is equal to \\(n\\) then \\(\\mathcal{V}\\) itself is cyclic.

7. The rational canonical form

On the basis of the lemma of the preceding section we shall now prove that \\(\\mathcal{V}\\) is the direct sum of cyclic subspaces: i.e., that we can find vectors \\(x_{1}, \\ldots, x_{r}\\) of orders \\(m_{1}(\\lambda), \\ldots, m_{r}(\\lambda)\\) respectively (where \\(m_{i}(\\lambda)\\) is a polynomial of degree \\(m_{i}\\)) so that the vectors \\(A^{j} x_{i}\\), \\(i=1, \\ldots, r\\), \\(j=0,1, \\ldots, m_{i}-1\\), are a basis in \\(\\mathcal{V}\\).

Let \\(m_{1}(\\lambda)=m(\\lambda)\\) be the minimal polynomial of \\(A\\) and let \\(x_{1}\\) be a vector whose order is \\(m_{1}(\\lambda)\\). Let \\(m_{2}(\\lambda)\\) be the minimal polynomial of \\(A\\) in the quotient space \\(\\mathcal{V} /\\{x_1\\}\\), and let \\(x_{2}^\\prime\\) be a vector of this space whose order (\\(\\operatorname{mod} \\,\\{x_1\\}\\)) is \\(m_{2}(\\lambda)\\). This means that \\(m_{2}(A) x_{2}^{\\prime} \\equiv 0 \\,\\, (\\operatorname{mod} \\,\\{x_1\\})\\), i.e., that \\(m_{2}(A) x_{2}^{\\prime}\\) is in \\(\\{x_1\\}\\), so that \\(m_{2}(A) x_{2}^{\\prime}=h_{1}(A) x_{1}\\) for a suitable polynomial \\(h_{1}(\\lambda)\\).

Since \\(m_{1}(A) x_{2}^{\\prime}=0\\) (and, a fortiori, \\(m_{1}(A) x_{2}^{\\prime} \\equiv 0 \\,\\, (\\operatorname{mod} \\,\\{x_1\\})\\)) \\(m_{2}(\\lambda)\\) must be a factor of \\(m_{1}(\\lambda)\\): say \\(m_{1}(\\lambda)=n_{1}(\\lambda) m_{2}(\\lambda)\\). We have then \\[ n_{1}(A) h_{1}(A) x_{1}=n_{1}(A) m_{2}(A) x_{2}^{\\prime}=m_{1}(A) x_{2}^{\\prime}=0 \\] so that \\(n_{1}(\\lambda) h_{1}(\\lambda)\\) must be divisible by \\(m_{1}(\\lambda)\\): \\[ n_{1}(\\lambda) h_{1}(\\lambda)=k_{1}(\\lambda) m_{1}(\\lambda)=k_{1}(\\lambda) n_{1}(\\lambda) m_{2}(\\lambda). \\]

It follows (upon division by \\(n_{1}(\\lambda)\\)) that \\[ {h}_{1}(\\lambda)=k_{1}(\\lambda) m_{2}(\\lambda). \\]

We define \\(x_{2}=x_{2}^{\\prime}-k_{1}(A) x_{1}\\). Since \\(x_{2} \\equiv x_{2}^{\\prime} \\,\\, (\\operatorname{mod}\\,\\{x_1\\})\\) the order of \\(x_{2} \\, \\operatorname{mod} \\,\\{x_1\\}\\) is the same as that of \\(x_{2}^{\\prime}\\), i.e., it is \\(m_{2}(\\lambda)\\). We have moreover \\[ \\begin{aligned} m_{2}(A) x_{2} &= m_{2}(A) x_{2}^\\prime - k_{1}(A) m_{2}(A) x_{1}\\\\ &= h_{1}(A) x_{1}-h_{1}(A) x_{1}\\\\ &= 0. \\end{aligned} \\]

The intersection of the linear subspaces \\(\\{x_1\\}\\) and \\(\\{x_{2}\\}\\) consists of zero alone, for \\(f(A) x_{2}\\) in \\(\\{x_1\\}\\) implies that \\(m_{2}(\\lambda)\\) is a divisor of \\(f(\\lambda)\\), and therefore that \\(f(A) x_{2}=0\\).

The above constitutes the first step of an induction proof: we go on to indicate the second step.

Denote by \\(\\{x_{1}, x_{2}\\}\\) the linear subspace spanned together by \\(\\{x_1\\}\\) and \\(\\{x_{2}\\}\\) and consider the quotient space \\(\\mathcal{V} /\\{x_{1}, x_{2}\\}\\). Let \\(x_{3}^{\\prime}\\) be a vector whose order (\\(\\operatorname{mod} \\,\\{x_{1}, x_{2}\\}\\)) is the minimal polynomial, \\(m_{3}(\\lambda)\\), of \\(A\\) in this space. Then \\[ m_{3}(A) x_{3}^{\\prime}=h_{1}(A) x_{1}+h_{2}(A) x_{2}, \\] (where the auxiliary polynomial \\(h_{1}(\\lambda)\\) is not related to the one used above). Since \\(m_{2}(A) x_{3}^\\prime \\equiv 0 \\,\\, (\\operatorname{mod} \\,\\{x_1\\})\\) and therefore \\(\\operatorname{mod} \\,\\{x_{1}, x_{2}\\}\\), \\(m_{3}(\\lambda)\\) must divide \\(m_{2}(\\lambda)\\): \\[ m_{2}(\\lambda)=n_{2}(\\lambda) m_{3}(\\lambda). \\] Hence \\[ n_{2}(A) h_{2}(A) x_{2}=n_{2}(A) m_{3}(A) x_{3}^{\\prime}-n_{2}(A) h_{1}(A) x_{1} \\equiv 0 \\,\\, (\\operatorname{mod} \\,\\{x_1\\}), \\] so that \\[ n_{2}(\\lambda) h_{2}(\\lambda)=k_{2}(\\lambda) m_{2}(\\lambda)=k_{2}(\\lambda) n_{2}(\\lambda) m_{3}(\\lambda), \\] whence, finally, \\[ h_{2}(\\lambda)=k_{2}(\\lambda) m_{3}(\\lambda). \\]

Similarly, since \\(m_{1}(A) x_{3}^\\prime=0\\), \\(m_{1}(\\lambda)=n_{1}(\\lambda) m_{3}(\\lambda)\\), so that \\[ \\begin{aligned} n_{1}(A) h_{1}(A) &= n_{1}(A) m_{3}(A) x_{3}^\\prime - n_{1}(A) n_{2}(A) x_{2} \\\\ &= m_{1}(A) x_{2}-n_{1}(A) k_{2}(A) m_{3}(A) x_{2}\\\\ &= 0. \\end{aligned} \\] Consequently \\[ n_{1}(\\lambda) h_{1}(\\lambda)=k_{1}(\\lambda) m_{1}(\\lambda)=k_{1}(\\lambda) n_{1}(\\lambda) m_{3}(\\lambda), \\] so that \\[ {h}_{1}(\\lambda)=k_{1}(\\lambda) m_{3}(\\lambda). \\]

We write \\(x_{3}=x_{3}^{\\prime}-k_{1}(A) x_{1}-k_{2}(A) x_{2}\\); then \\(x_{3} \\equiv x_{3}^\\prime \\,\\, (\\operatorname{mod} \\,\\{x_{1}, x_{2}\\})\\) so that the order of \\(x_{3}\\) is still \\(m_{3}(\\lambda)\\); and moreover \\[ m_{3}(A) x_{3}=m_{3}(A) x_{3}^{\\prime}-m_{3}(A) k_{1}(A) x_{1}-m_{3}(A) k_{2}(A) x_{2}=0. \\]

It follows that the linear subspace \\(\\{x_{3}\\}\\) has only the vector \\(0\\) in common with \\(\\{x_{1}, x_{2}\\}\\), for \\(f(A) x_{3} \\equiv 0 \\,\\, (\\operatorname{mod} \\,\\{x_{1}, x_{2}\\})\\) implies that \\(f(\\lambda)\\) is divisible by \\(m_{3}(\\lambda)\\) and therefore that \\(f(A) x_{3}=0\\).

Proceeding in this way by induction we obtain finally the desired result. The polynomials \\(m_{i}(\\lambda)\\) (which incidentally have the property that each divides all preceding ones) are the invariant factors of \\(A\\).

Using the basis \\(A^{j} x_{i}\\), \\(i=1, \\ldots, r\\), \\(j=0,1, \\ldots, m_{i}-1\\), the matrix of \\(A\\) has the so-called rational canonical form: \\[ [A] = \\begin{bmatrix} {[A_{1}]} & 0 & \\cdots & 0 \\\\ 0 & {[A_{2}]} & \\cdots & 0 \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & \\cdots & {[A_{{r}}]} \\end{bmatrix} \\] where each \\([A_{i}]\\) has the form described in (V.5) above.

It is easily verified, using this form, that the characteristic polynomial, \\(f(\\lambda)\\), is the product of all the \\(m_{i}(\\lambda)\\) and is therefore divisible by \\(m_{1}(\\lambda)=m(\\lambda)\\). As a corollary, therefore, we obtain the Hamilton-Cayley theorem:

Theorem 1. For every \\(A\\), \\(f(A)=0\\), where \\(f(\\lambda)\\) is the characteristic polynomial of \\(A\\).

8. Elementary transformations

In the present section we consider matrices (not linear transformations) depending on a parameter \\(\\lambda\\).

Definition 3. Let \\(A(\\lambda)=\\big(a_{ij}(\\lambda)\\big)\\) be a matrix whose elements are polynomials in \\(\\lambda\\), and let \\(d_{j}\\big(A(\\lambda)\\big)\\) be the highest common factor (H.C.F.) of all \\(n - {j}\\) rowed minors of \\(A(\\lambda)\\). \\(d_{j}\\big(A(\\lambda)\\big)\\) is the \\(j\\)-th determinantal factor of \\(A(\\lambda)\\).

Two matrix polynomials, \\(A(\\lambda)\\) and \\(B(\\lambda)\\), are called equivalent if there exist non-singular matrix polynomials \\(P(\\lambda)\\) and \\(Q(\\lambda)\\) for which \\({B}(\\lambda)={P}(\\lambda) {A}(\\lambda) Q(\\lambda)\\). Since the rows (or columns) of \\({P}(\\lambda) A(\\lambda)\\) (or \\(A(\\lambda) Q(\\lambda)\\)) are linear combinations of the rows (or columns) of \\(A(\\lambda)\\), it follows that the determinantal factors of \\(B(\\lambda)\\) are linear combinations of those of \\(A(\\lambda)\\) and therefore that \\(d_{j}\\big(B(\\lambda)\\big)\\) is divisible by \\(d_{j}\\big(A(\\lambda)\\big)\\). Interchanging the roles of \\(A\\) and \\(B\\) we see that if \\(A(\\lambda)\\) is equivalent to \\(B(\\lambda)\\) then their determinantal factors are equal.

The matric polynomial \\(B(\\lambda)\\) is obtained from \\(A(\\lambda)\\) by elementary transformations if it is obtained by

the interchange of any two rows (or columns),
the multiplication of any row (or column) by a constant different from \\(0\\), or
the addition of the \\(u(\\lambda)\\)-fold of any row (or column) to any other row (or column), when \\(u(\\lambda)\\) is an arbitrary polynomial.

It is easily verified that the elementary transformations can be effected by suitable multiplication by non-singular matric polynomials, so that if \\(B(\\lambda)\\) is obtained from \\(A(\\lambda)\\) by elementary transformations then \\(B(\\lambda)\\) and \\(A(\\lambda)\\) are equivalent. In particular therefore \\(B(\\lambda)\\) and \\(A(\\lambda)\\) have the same determinantal factors.

9. Similarity of linear transformations

To apply the results of the preceding section to linear transformations, let \\(A\\), \\(B\\), and \\(S\\) be linear transformations connected by the equation \\(B=S A S^{-1}\\). Then in any coordinate system we have the matrix equation \\[ [S][A-\\lambda{1}][S]^{-1}=[B-\\lambda{1}], \\] so that the determinantal factors of \\([A-\\lambda{1}]\\) and \\([B-\\lambda 1]\\) are equal. In other words the determinantal factors are invariant under a change of basis. Hence if for any linear transformation \\(A\\) we want to compute the determinantal factors of \\(A-\\lambda 1\\), we may assume that the matrix of \\(A\\) is in rational canonical form. The generic block in the matrix of \\(A - \\lambda{1}\\) will then have the form \\[ \\begin{bmatrix} -\\lambda & 1 & 0 & \\cdots & 0 \\\\ 0 & -\\lambda & 1 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & 1 \\\\ a_{1} & a_{2} & a_{3} & \\cdots & a_{m}-\\lambda \\end{bmatrix}. \\]

Adding to the first column \\(\\lambda\\) times the second, \\(\\lambda^{2}\\) times the third, etc., we obtain \\[ \\begin{bmatrix} 0 & 1 & 0 & \\cdots & 0 \\\\ 0 & -\\lambda & 1 & \\cdots & 0 \\\\ 0 & 0 & -\\lambda & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ m(\\lambda) & a_{2} & a_{3} & \\cdots & a_{m}-\\lambda \\end{bmatrix}, \\] and by elementary transformations on the rows this becomes \\[ \\begin{bmatrix} 1 & 0 & 0 & \\cdots & 0 \\\\ 0 & 1 & 0 & \\cdots & 0 \\\\ 0 & 0 & 1 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & m(\\lambda) \\end{bmatrix}. \\]

After these transformations on each block, the matrix of \\(A-\\lambda 1\\) will have the form \\[ \\begin{bmatrix} 1 & \\cdots & 0 \\quad & 0 & \\cdots & 0\\\\ \\vdots & \\ddots & \\vdots \\quad & \\vdots & \\ddots & \\vdots\\\\ 0 & \\cdots & 1 \\quad & 0 & \\cdots & 0\\\\ \\, & \\, & \\, \\quad & \\, & \\, & \\, \\\\ 0 & \\cdots & 0 \\quad & m_1(\\lambda) & \\cdots & 0\\\\ \\vdots & \\ddots & \\vdots \\quad & \\vdots & \\ddots & \\vdots\\\\ 0 & \\cdots & 0 \\quad & 0 & \\cdots & m_r(\\lambda)\\\\ \\end{bmatrix}. \\]

It follows that for \\(j=0,1, \\ldots, r-1\\), \\[ d_{j}(A-\\lambda 1)=m_{j+1}(\\lambda)\\ldots m_{r}(\\lambda), \\] and for \\(j \\geq r\\), \\(d_{j}(A-\\lambda 1)=1\\). Hence for \\(j= 1, \\ldots, r\\), \\[ m_{j}(\\lambda)=d_{j-1}(A-\\lambda 1) \\big/ d_{j}(A-\\lambda 1), \\] which shows, incidentally, that the invariant factors \\(m_{j}(\\lambda)\\) are invariant under change of basis and that we may therefore speak of the invariant factors of a linear transformation. As an easy consequence of our results we obtain the fact that a necessary and sufficient condition that two linear transformations \\(A\\) and \\(B\\) be similar, i.e., that there exist a non-singular linear transformation \\(S\\) for which \\(B=S A S^{-1}\\), is that \\(A\\) and \\(B\\) have the same invariant factors. The necessity of this condition we have just proved; its sufficiency follows from the fact that if \\(A\\) and \\(B\\) have the same invariant factors then their matrices, in two suitably chosen coordinate systems, will have the same rational canonical form.

10. Elementary divisors

Let the minimal polynomial of \\(A\\) be \\(m(\\lambda)=p_{1}(\\lambda)^{e_1} \\ldots p_{s}(\\lambda)^{e_{s}}\\), and write \\(q_{i}(\\lambda)=m(\\lambda) / p_{i}(\\lambda)^{e_{i}}\\). Then the H.C.F. of all \\(q_{i}(\\lambda)\\) is \\(1\\), so that we may find polynomials \\(u_{i}(\\lambda)\\) for which \\[ u_{1}(\\lambda) q_{1}(\\lambda)+\\cdots+u_{s}(\\lambda) q_{s}(\\lambda)=1. \\]

Let \\(M_i\\) be the range of \\(q_{i}(A)\\), i.e., the linear subspace of all vectors of the form \\(q_{i}(A) x\\). If \\(x_{i}\\) is in \\(M_i\\) then \\[ p_{i}(A)^{e_i} x_i=0, \\] and if \\(x\\) is any vector then \\[ x=\\sum_i u_{i}(A) q_{i}(A) x=\\sum_i x_{i}, \\] where \\(x_{i}\\) is in \\(M_i\\). If \\(\\sum_i x_{i}=0\\) then \\[ q_{j}(A) x=\\sum_i q_{j}(A) x_{i}=q_{j}(A) x_{j}=0. \\] since \\(q_{i}(\\lambda)\\) and \\(p_{i}(\\lambda)^{e_{i}}\\) are relatively prime we may find polynomials \\(v_{i}(\\lambda)\\) and \\(w_{i}(\\lambda)\\) for which \\[ v_{i}(\\lambda) p_{i}(\\lambda)^{e_i}+ w_{i}(\\lambda) q_{i}(\\lambda)=1, \\] so that \\[ x_{i}=q_{i}(A) w_{i}(A) x_{i}+ v_i(A) p_{i}(A)^{e_i} x_{i}=0. \\]

To sum up: \\(\\mathcal{V}\\) is the direct sum of \\(M_{1}, \\ldots, M_s\\), i.e., the linear subspaces \\(M_{i}\\) span \\(\\mathcal{V}\\) and any two of them contain only the vector \\(0\\) in their intersection. Moreover the minimal polynomial of \\(A\\) in the invariant linear subspace \\(M_{i}\\) is exactly \\(p_{i}(\\lambda)^{e_i}\\).

If we apply the above considerations to each cyclic subspace of \\(A\\) that is exhibited by the rational canonical form we obtain an even finer decomposition of \\(\\mathcal{V}\\) into cyclic subspaces whose orders are the prime power factors of \\(m_{i}(\\lambda)\\). These orders are called the elementary divisors of \\(A\\). The elementary divisors are determined by the invariant factors: we show that the converse is true. If \\(\\mathcal{V}\\) is the direct sum of cyclic subspaces of orders \\(p_{i}^{\\prime}(\\lambda)^{f_{ij}}\\) which are powers of irreducible polynomials, and if the notation is so chosen that \\(f_{i1} \\geq f_{i2} \\geq \\cdots\\), then it is easy to verify that the \\(j\\)-th invariant factor, \\(m_{j}(\\lambda)\\), of \\(A\\) is \\[ m_{j}(\\lambda)=p_{1}^{\\prime}(\\lambda)^{f_{1j}} p_{2}(\\lambda)^{f_{2 j}}\\ldots \\]

From the unique factorization theorem it follows that the \\(p_{i}^\\prime(\\lambda)^{f_{ij}}\\) are the elementary divisors of \\(A\\), and therefore that the elementary divisors determine the invariant factors. It follows easily from our preceding results that a necessary and sufficient condition that two linear transformations be similar is that they have the same elementary divisors.

11. The classical canonical form

In preceding section we have decomposed \\(\\mathcal{V}\\) into a direct sum of cyclic subspaces and the decomposed each subspace into a direct sum of smaller cyclic subspaces whose orders are powers of irreducible polynomials. In the present section we shall indicate one more decomposition, of these last prime power spaces, which will not in general be into cyclic spaces.

Consider a cyclic space \\(\\{z\\}\\) of order \\(p(\\lambda)^{e}\\), where \\(p(\\lambda)=\\lambda^{c}-\\sum_{i=0}^{c-1} p_{i} \\lambda^{i}\\) is an irreducible polynomial. For \\(0 \\leq k < e\\), and \\(0 < {h} \\leq {c}\\), we write \\[ z_{k c + h} = p(A)^{e-k-1} {A}^{h-1} z. \\]

Since the degrees of the polynomials \\({p}(\\lambda)^{e-k-1} \\lambda^{{h}-1}\\) are all different and \\(< ce\\), the \\(z_{i}\\) are linearly independent and therefore form a basis for \\(\\{z\\}\\). In this basis the matrix of \\(A\\) has the form \\[ \\begin{bmatrix} {[P]} & [E] & 0 & \\cdots & 0 & 0 \\\\ 0 & {[P]} & {[E]} & \\cdots & 0 & 0 \\\\ 0 & 0 & {[P]} & \\cdots & 0 & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & [P] & [E]\\\\ 0 & 0 & 0 & \\cdots & 0 & {[P]} \\end{bmatrix} \\] where \\[ [P] = \\begin{bmatrix} 0 & 1 & 0 & \\cdots & 0 \\\\ 0 & 0 & 1 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & 1 \\\\ p_{0} & p_{1} & p_{2} & \\cdots & p_{c-1} \\end{bmatrix}, \\] \\[ [E] = \\begin{bmatrix} 0 & 0 & \\cdots & 1 \\\\ 0 & 0 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & \\cdots & 0 \\end{bmatrix}. \\]

In case the ground field is algebraically closed, so that the only irreducible polynomials are linear (i.e., \\(c = 1\\)), this matrix reduces to \\[ \\begin{bmatrix} \\lambda & 1 & 0 & \\cdots & 0 & 0 \\\\ 0 & \\lambda & 1 & \\cdots & 0 & 0 \\\\ 0 & 0 & \\lambda & \\cdots & 0 & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & \\lambda & 1\\\\ 0 & 0 & 0 & \\cdots & 0 & \\lambda \\end{bmatrix}. \\] The matrix of \\(A\\) in the entire space will then consist of a number of blocks of this type on the main diagonal: this form of the matrix of \\(A\\) is called the classical canonical (or Jordan) form.

As an application of these ideas we mention that (in the case of an algebraically closed field) a necessary and sufficient condition that a matrix be similar to a diagonal matrix is that the roots of the minimal equation (or the elementary divisors) be simple, i.e., have multiplicity one. For if these roots are simple then the blocks of the classical canonical form will consist of only one element each, so that no super diagonal \\(1\\)’s appear, and the matrix is diagonal. This proves the sufficiency of our conditions. To prove necessity we observe that for a diagonal matrix whose diagonal elements are \\(\\lambda_{i}\\) the product of factors \\((\\lambda-\\lambda_{i})\\) extended over a set of subscripts \\(i\\) in such a way as to contain every eigenvalue exactly once is precisely the minimal polynomial, which therefore has simple roots. As a particular application of this result we see that every idempotent linear transformation, which satisfies the equation \\(\\lambda^{2}-\\lambda=\\lambda(\\lambda-1)=0\\), may be realized by a diagonal matrix whose diagonal elements consist of only \\(0\\)’s and \\(1\\)’s.

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"canonical-forms","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]},{"_id":"66a049c739e7e3d9dc7ad86a","book_id":"669feeab39e7e3d9dc7ad4c7","chapter_name":"Limits of Linear Transformations","sequence":6,"chapter_content":"

1. Convergence of vectors

In this chapter we shall indicate the various possibilities for defining the concept of limit for sets of vectors or linear transformations, and we shall give a few of the many possible applications of these concepts.

If \\(\\mathcal{U}\\) is an \\(N\\)-dimensional inner product space, the two fundamental quantitive concepts—inner product, and length—suggest the following two definitions of convergence in \\(\\mathcal{U}\\).

Definition 1. We say that a sequence, \\(x_{n}\\), of vectors converges to the vector \\(x\\) in the strong sense if \\(|x_{n}-x| \\to 0\\) as \\(n \\to \\infty\\); and \\(x_{n} \\to x\\) in the weak sense if \\(\\langle x_{n}, y \\rangle \\to \\langle x, y \\rangle\\) for every \\(y\\).

It turns out that this distinction is merely pedantic in our finite dimensional case: the two notions of convergence are equivalent. For if \\(x_{n} \\stackrel{(s)}{\\to} x\\) (i.e., \\(x_{n}\\) converges strongly to \\(x\\)) then \\[ \\begin{aligned} |\\langle x_{n}, y \\rangle - \\langle x, y \\rangle| &= |\\langle x_{n}-x, y \\rangle|\\\\ &\\leq |x_{n}-x|\\cdot|y|\\\\ &\\to 0. \\end{aligned} \\] Conversely if \\(x_{n} \\stackrel{(w)}{\\to} x\\), and if \\(z_{1}, \\ldots, z_{N}\\) is an orthogonal basis in \\(\\mathcal{U}\\), \\(x_{n}=\\sum_i a_{i}^{n} z_{i}\\), \\(x=\\sum_i a_{i} z_{i}\\), then \\(a_{i}^{n} = \\langle x_{n}, z_{i} \\rangle\\) converges, as \\(n \\to \\infty\\), to \\(a_{i} = \\langle x, z_{i} \\rangle\\), for each \\(i=1, \\ldots, N\\). Consequently

\\[ \\begin{aligned} |x_{n}-x|^{2} &= \\Big|\\sum_i (a_{i}^{n}-a_{i}) z_{i}\\Big|^{2}\\\\ &= \\sum_i |a_{i}^{n}-a_{i}|^{2}\\\\ &\\to 0 \\end{aligned} \\] as \\(n \\to \\infty\\), as was to be proved.

Concerning the convergence (in either of the two equivalent senses) of vectors, we shall use without proof the following facts. The expression \\(ax + by\\) is a continuous function of all its arguments simultaneously, i.e., if \\(a_{n}\\) and \\(b_{n}\\) are sequences of complex numbers and \\(x_{n}\\) and \\(y_{n}\\) vectors, then \\[ a_{n} \\to a, \\quad b_{n} \\to b, \\quad x_{n} \\to x, \\quad y_{n} \\to y \\] implies that \\[ a_{n} x_{n} + b_{n} y_{n} \\to a x+b y. \\]

If \\(z_{i}\\) is an orthogonal basis in \\(\\mathcal{U}\\), \\(x_{n}=\\sum_i a_{i}^{n} z_{i}\\), \\(x=\\sum_i a_{i} z_{i}\\), then a necessary and sufficient condition for the convergence of \\(x_{n}\\) to \\(x\\) is that \\(a_{i}^{n} \\to a_{i}\\) for each \\(i\\). (Thus the notion of convergence here defined coincides with the usual one in \\(N\\)-dimensional complex Euclidean space.) Finally we shall assume as known the fact that the metric topology defined by the notion of convergence is complete: i.e., if \\(x_{n}\\) is a sequence of vectors for which \\(|x_{n}-x_{m}| \\to 0\\) as \\(n, m \\to \\infty\\), then there is a unique vector \\(x\\) such that \\(x_{n} \\to x\\) as \\(n \\to \\infty\\).

2. Convergence of linear transformations

The notion of convergence for linear transformations is apparently more complicated than that for vectors. Using the ideas of the preceding section, and remembering our discussion of the norm, \\(|A|\\), of a linear transformation \\(A\\), we are led to consider the following three definitions for the convergence of a sequence \\(A_{n}\\) of linear transformations to the linear transformation \\(A\\).

Definition 2. \\(A_{n} \\to A\\) in the uniform sense if \\(|A_{n}-A| \\to 0\\); in the strong sense if \\(A_{n} x \\to A x\\) in the strong sense for every vector \\(x\\); and in the weak sense if \\(A_{n} x \\to A x\\) in the weak sense.

Once more the distinctions are illusory: we shall prove that the three notions of convergence coincide. That weak and strong convergence are equivalent follows already from the results of the preceding section. If \\(A_{n} \\stackrel{(u)}{\\to} A\\), then for every \\(x\\) we have

\\[ \\begin{aligned} |A_{n} x-A x| &= |(A_{n}-A) x|\\\\ &\\leq |A_{n}-A| \\cdot |x|\\\\ &\\to 0, \\end{aligned} \\]

so that uniform convergence implies strong convergence. To prove the converse, let \\(x_{1}, \\ldots, x_{N}\\) be an orthogonal basis in \\(\\mathcal{U}\\), let \\(x=\\sum_i a_{i} x_{i}\\) be an arbitrary vector, and suppose that \\(A_{n} \\stackrel{(s)}{\\to} A\\). Then for any \\(\\varepsilon > 0\\) we may find an \\(n_{0}\\) such that for \\(n \\geq n_{0}\\) and \\(i=1, \\ldots, N\\) we have \\(|A_{n} x_{i}-A x_{i}|<\\varepsilon\\). It follows that \\[ \\begin{aligned} |(A_{n}-A) x| &= \\bigg|\\sum_i a_{i}({A}_{n}-A) x_{i}\\bigg|\\\\ &\\leq \\sum_i |a_{i}| \\cdot |(A_{n}-A) x_{i}|\\\\ &\\leq \\varepsilon N|x|, \\end{aligned} \\] so that for \\(n \\geq n_{0}\\), \\(|A_{n} - A| \\leq \\varepsilon N\\). This completes the proof of the equivalence of the three concepts, and yields also an easy proof of the fact that again our topology is complete. If \\(|A_{n}-A_{m}| \\to 0\\) as \\(n, m \\to \\infty\\), then for each \\(x\\), \\(|A_{n} x-A_{m} x| \\to 0\\). Hence to each \\(x\\) there corresponds a vector, which we may designate by \\(A x\\), such that \\(A_{n} x \\to A x\\). It is easy to verify that the correspondence from \\(x\\) to \\(A x\\) is given by a linear transformation \\(A\\) and that \\(A_{n} \\to A\\) in the strong sense. Consequently \\(A_{n} \\to A\\) in the uniform sense, and this proves our assertion concerning completeness.

If \\(x_{1}, \\ldots, x_{N}\\) is an orthogonal basis in which the matrices of \\(A_{n}\\) and \\(A\\) are \\((a_{ij}^{n})\\) and \\((a_{ij})\\) respectively, then a necessary and sufficient condition for the convergence of \\(A_{n} \\to A\\) (in any one of the three equivalent senses given above) is that \\(a_{ij}^{n} \\to a_{ij}\\) as \\(n \\to \\infty\\) for each \\(i, j=1, \\ldots, N\\). This follows from the fact that \\(a_{ij}^{n} = \\langle A_{n} x_{j}, x_{i} \\rangle\\) and the definition of weak convergence. Consequently the concepts of convergence we discussed reduce to the usual concept in \\({N}^{2}\\) dimensional complex or \\(2 {N}^{2}\\) dimensional real Euclidean space. It would be easy to use this fact in the proof of several statements given below: we prefer a slightly more circuitous route, which sheds more light on the concept of the norm of a transformation.

3. Inequalities on norms

In this section we fill a gap in our discussion of norms: our purpose is to prove the following four relations: \\[ \\begin{aligned} |A+B| &\\leq |A|+|B|,\\\\ |A B| &\\leq |A||B|,\\\\ |a A| &= |a||A|,\\\\ |A^{*}| &= |A|. \\end{aligned} \\]

All of the proofs are easy from the definition: we sketch the proofs for the product and the dual. We have \\[ \\begin{aligned} |(A B) x| &= |A(B x)|\\\\ &\\leq |A||B x|\\\\ &\\leq |A||B||x|. \\end{aligned} \\]

For duals we have \\[ \\begin{aligned} |\\langle A x, y \\rangle| &= |\\langle x, A^{*} y \\rangle|\\\\ &= |\\langle A^{*} y, x \\rangle|\\\\ &\\leq |A^{*} y| \\cdot |x|\\\\ &\\leq |A^{*}||x||y|. \\end{aligned} \\] It follows that \\(|A| \\leq |A^{*}|\\); by symmetry we obtain the desired result.

4. The continuity of the ring operations

\\(Ax\\) is a continuous function of \\(x\\); \\(|A|\\) is a continuous function of \\(A\\); \\(A+B\\), \\(A B\\), \\(a A\\), and \\(A^{*}\\) are continuous functions of all their arguments. Of these statements we shall prove only the ones concerning \\(|A|\\), \\(A B\\), and \\(A^{*}\\): the others are proved by very similar methods.

4.1. From the relations \\(|A_{n}| \\leq |A_{n}-{A}|+|A|\\) and \\(|A| \\leq |A-A_{n}| + |A_{n}|\\) we conclude that the absolute value of the difference \\(|A_{n}|-|A|\\) is dominated by \\(|A_{n}-A|\\). Since \\(A_{n} \\to A\\) means \\(|A_{n}-A| \\to 0\\), we see that the convergence of \\(A_{n}\\) to \\(A\\) implies \\(|A_{n}| \\to |A|\\).

In particular we note that \\(A_{n} \\to A\\) implies that \\(|A_{n}|\\) remains bounded.

4.2. If \\(A_{n} \\to A\\), \\(B_{n} \\to B\\), then

\\[ \\begin{aligned} |A_{n} B_{n}-A B| &= |A_{n} B_{n}-A_{n} B+A_{n} B-A B|\\\\ &\\leq |A_{n}| \\cdot |B_{n}-B| + |B| \\cdot |A_{n}-A|. \\end{aligned} \\]

Since \\(A_{n}\\) and \\(B\\) are bounded and the other factors converge to zero, we see that \\(A_{n} B_{n} \\to A B\\).

4.3. If \\(A_{n} \\to A\\) then for each \\(x\\) and \\(y\\), \\(\\langle A_{n} x, y \\rangle = \\langle x, A_{n}^{*} y \\rangle\\) converges to \\(\\langle A x, y \\rangle = \\langle x, A^{*} y \\rangle\\). Hence \\(A_{n}^{*}\\) converges to \\(A^{*}\\) in the weak and therefore in the uniform sense.

We shall now prove a few theorems about limits of special linear transformations by way of illustration of the general theory.

5. The mean ergodic theorem

Theorem 1. If \\(U\\) is a unitary transformation and \\(M\\) is a subspace of all vectors \\(x\\) for which \\(Ux=x\\), then \\[ \\frac{1}{n}(U+U^{2}+\\cdots+U^{n}) \\to E=P_{M}, \\]

where \\(P_{M}\\) is the projection on \\(M\\).

Proof. For let \\(U=\\sum_i u_{i} E_{i}\\) be the spectral form of \\(U\\); we have \\(|u_{i}|=1\\) and \\(U^{k}=\\sum_i u_{i}^{k} E_{i}\\). Moreover \\(E\\) is equal to one of the \\(E_{i}\\): that one for which \\(u_{i}=1\\). (we do not exclude the possibility that no \\(u_{i}=1\\), so that \\(M=\\{0\\}\\).) Hence we have \\[ \\frac{1}{n}(U+\\cdots+U^{n})=\\sum_i \\frac{1}{n}(u_{i}+\\cdots+u_{i}^{n}) E_{i}, \\] and it is sufficient to prove that the numbers \\(\\frac{1}{n}(u_i + \\cdots + u_i^n)\\) converges to \\(1\\) or \\(0\\) according as \\(u_i = 1\\) or \\(u_i \\neq 1\\). In case \\(u_{i}=1\\) this is obvious; In any other case we have \\[ \\begin{aligned} \\bigg|\\frac{1}{n}(u_{i}+\\cdots+u_{i}^{n})\\bigg| &= \\bigg|\\frac{1}{n} \\, \\frac{u_{i}^{n+1}-u_{i}}{u_{i}-1}\\bigg|\\\\ &\\leq \\frac{2}{n|u_{i}-1|}\\\\ &\\to 0. \\end{aligned} \\]■

6. Markoff matrices

To give our next application of the notion of convergence we forsake the theory of linear transformations and consider matrices, and we shall understand by the convergence of a sequence \\((a_{ij}^n)\\) of matrices to \\((a_{ij})\\) the numercial convergence \\(a_{ij}^n \\to a_{ij}\\) for each \\(i\\), \\(j\\).

Definition 3. A Markoff matrix is a matrix \\(P = (p_{ij})\\) for which \\(p_{ij} \\geq 0\\) and \\(\\sum_{j} p_{ij}=1\\).

Concerning Markoff matrices we need the following two lemmas.

Lemma 1. The eigenvalues of a Markoff matrix are in absolute value \\(\\leq 1\\).

Proof. Since the eigenvalues of \\((p_{ij})\\) and of \\((p_{j i})\\) are the same, it is sufficient to prove that \\(\\sum_i p_{ij} \\xi_{i} = \\lambda \\xi_j\\) implies \\(|\\lambda| \\leq 1\\). We have \\[ \\sum_i p_{ij}|\\xi_{i}| \\geq |\\lambda| \\cdot |\\xi_{j}| \\] and therefore \\[ \\sum_{j} \\sum_i p_{ij}|\\xi_{i}| \\geq |\\lambda| \\sum_{j}|\\xi_{j}|. \\] Since \\(\\sum_{j} p_{ij}=1\\), we have \\[ \\sum_i |\\xi_{i}| \\geq |\\lambda| \\cdot \\sum_{j} |\\xi_{j}| \\] and from this the desired conclusion is immediate.■

Lemma 2. The product of two Markoff matrices \\((p_{ij})\\) and \\((q_{i {j}})\\), is a Markoff matrix.

Proof. Write \\(r_{ij}=\\sum_{k} p_{i k} q_{k j}\\); it is clear that \\(r_{ij} \\geq 0\\). Also

\\[ \\sum_j r_{ij} = \\sum_k p_{ik} \\sum_j q_{kj}=\\sum_{k} p_{i k}=1, \\]

so that \\((r_{ij})\\) is a Markoff matrix.■

As a consequence of this lemma we obtain the fact that the totality of all elements in the infinite sequence \\(P, P^{2}, P^{3}, \\ldots\\) is bounded, and therefore that the same is true of the sequence of powers of \\(SPS^{-1}\\), where \\(S\\) is an arbitrary non-singular matrix. The last statement follow from the fact that the elements of \\((S^{-1})^{n}\\) are linear combinations, with coefficients which are elements of \\(S\\) and of \\(S^{-1}\\), of the elements of \\(P^{n}\\).

7. Limits of Markoff matrices

Our main theorem concerning Markoff matrices is the following:

Theorem 2. For an arbitrary Markoff matrix \\(P\\), \\((P+P^{2}+\\cdots+P^{n}) / n\\) converges to a limit as \\(n \\to \\infty\\); a necessary and sufficient condition that not only the averages, but also the powers \\(P^n\\) themselves converge to a limit is that \\(\\lambda=1\\) be the only eigenvalue of \\(P\\) of absolute value one.

Proof. To prove this result it is sufficient to prove it not for \\(P\\) but for \\(SPS^{-1}\\), with a suitably chosen \\(S\\). Since we may choose \\(S\\) so that \\(SPS^{-1}\\) is in the classical (Jordan) canonical form, we may restrict our attention to matrices of the form \\(A=\\lambda 1 + B\\), where \\[ B=\\begin{bmatrix} 0 & 1 & 0 & \\cdots & 0 \\\\ 0 & 0 & 1 & \\cdots & 0 \\\\ \\vdots & \\vdots & \\vdots & \\ddots & \\vdots \\\\ 0 & 0 & 0 & \\cdots & 1 \\\\ 0 & 0 & 0 & \\cdots & 0 \\end{bmatrix}. \\]

In this case we have \\[ A^{n}=(\\lambda{1}+B)^{n}=\\sum_{i=0}^{n}\\binom{n}{i} \\lambda^{n-i} B^{i}. \\] Hence, the general non-vanishing element in \\(A^{n}\\) is a number of the form \\(\\binom{n}{i} \\lambda^{n-i}\\), where \\(i\\) is a fixed positive integer depending only on the place of the element and independent of \\(n\\).

Since \\(\\lambda\\) is an eigenvalue of a Markoff matrix we know that \\(|\\lambda| \\leq 1\\). If \\(|\\lambda|<1\\) we have \\(\\big|\\binom{n}{i} \\lambda^{n-i}\\big| \\leq Kn^{i}|\\lambda|^{n}\\) for a suitable constant \\({K}\\), and therefore in this case not only the averages of the elements of \\(A, A^{2}, \\ldots, A^{n}\\), but even the elements themselves converge to zero.

If, however, \\(|\\lambda|=1\\), then \\(\\big|\\binom{n}{i} \\lambda^{n-i}\\big|=\\big|\\binom{n}{i}\\big| \\to \\infty\\), so that in this case the elements of \\(A^{n}\\) do not remain bounded. Since \\(A\\) is a transform of a Markoff matrix, this is impossible: the only possible conclusion is that the disturbing super diagonal \\(1\\)’s do not enter in this case, so that the boxes of the classical canonical form which correspond to eigenvalues of absolute value one are of size \\(1\\). In this case therefore the problem is reduced to the convergence properties of \\(\\lambda^{n}\\), where \\(|\\lambda|=1\\): we have already seen that the averages of this sequence always converge, to zero or one according as \\(\\lambda \\neq 1\\) or \\(\\lambda=1\\), and that the sequence itself converges if and only if \\(\\lambda \\neq 1\\).

This completes the proof of all our assertions: we see even that since the limit, after transforming by \\(S\\), is a diagonal matrix whose diagonal elements consist only of zeros and ones, the limit is in general idempotent.■

8. The exponential function

Our final example of limits is furnished by \\(\\exp (A)\\), where \\(A\\) is an arbitrary linear transformation in the inner product space.

Consider the transformations \\(B_{n}=\\sum_{i=0}^{n} \\frac{1}{i!} A^{i}\\); let \\(k\\) be the norm, \\(|A|\\), of the transformation \\(A\\). Since we have

\\[ |B_{n}-B_{m}|=\\bigg|\\sum_{i=m+1}^{n} \\frac{1}{i!} A^{i}\\bigg| \\leq \\sum_{i=m+1}^{n} \\frac{1}{i!} k^{i}, \\]

and since the dominant on the right, being a part of the power series for \\(\\exp(k)\\), converges to zero as \\(n, m \\to \\infty\\), it follows that the series \\(\\sum_{i=0} \\frac{1}{i!} A^{i}\\) converges, or, in other words that the \\(B_{n}\\) have a limit. This limit we define to be \\(\\exp (A)\\): we shall mention some of the elementary properties of this exponential function.

Since we may assume that \\(A\\) is a matrix in upper triangular form, so that the diagonal elements are the eigenvalues of \\(A\\), each occurring with its proper multiplicity, it follows that the eigenvalues of \\(\\exp(A)\\) are the exponentials of those of \\(A\\). From the consideration of the upper triangular form it follows also that the determinant of \\(\\exp (A)\\), i.e., \\(\\prod_{i=1}^{n} \\exp(\\lambda_{i})\\), where \\(\\lambda_{1}, \\ldots, \\lambda_{n}\\) are the diagonal elements of the upper triangular form of \\(A\\), is the same as \\(\\exp (\\lambda_{1}+\\cdots+\\lambda_{n})\\), i.e., the exponential of the trace of \\(A\\). This shows incidentally that \\(\\exp (A)\\) is never a singular transformation.

We conclude by studying the multiplication rule for exponentials. We suppose that \\(A\\) and \\(B\\) are commutative transformations. Since \\[ \\exp (A+B)-\\exp (A) \\exp (B) \\] is the limit of the expression \\[ \\begin{aligned} &\\sum_{p=0}^{n} \\frac{1}{p!}(A+B)^{p} - \\sum_{q=0}^{n} \\frac{1}{q!} A^{q} \\, \\sum_{r=0}^{n} \\frac{1}{r!} B^{r}\\\\ =& \\sum_{p=0}^{n} \\frac{1}{p!} \\sum_{s=0}^{p} \\binom{p}{s} A^{s} B^{p-s} - \\sum_{q=0}^{n} \\sum_{r=0}^{n} \\frac{1}{q!r!} A^{q} B^{r}, \\end{aligned} \\] we will have proved the multiplication rule when we have proved that this expression converges to zero as \\(n \\to \\infty\\). An easy verification yields the fact that for \\(q+r \\leq n\\), \\(A^{q} B^r\\) occur in both members with equal coefficients; the terms that do not cancel out are all in the right member and are equal to \\[ \\sum_{q} \\sum_{r} \\frac{1}{q!r!} A_{q} B^{r}, \\quad q+r>n, \\] the summation being extended over those values of \\(q, r \\leq n\\) for which \\(q+r>n\\). \\(q+r>n\\) implies that at least one of \\(q\\) and \\(r\\) is greater than the integer part of \\({n} / 2\\), the bound of this remainder is dominated by

\\[ \\begin{aligned} &\\sum_{q=0}^{\\infty} \\sum_{r=n / 2}^{\\infty} \\frac{1}{q!r!}|A|^{q}|B|^{r}+\\sum_{r=0}^{\\infty} \\sum_{q=n / 2}^{\\infty} \\frac{1}{q!r!}|A|^{q}|B|^{r} \\\\ =& \\Big(\\sum_{q=0}^{\\infty} \\frac{1}{q!}|A|^{q}\\Big) \\Big(\\sum_{r=n / 2}^{\\infty} \\frac{1}{r!}|B|^{r}\\Big) + \\Big(\\sum_{r=0}^{\\infty} \\frac{1}{r!}|B|^{r}\\Big) \\Big(\\sum_{q=n/2}^\\infty \\frac{1}{q!}|A|^{q}\\Big) \\\\ =& \\exp (|A|) \\,b_{n}+\\exp (|B|) \\,a_{n}, \\end{aligned} \\] where \\(a_n \\to 0\\) and \\(b_n \\to 0\\) as \\(n \\to \\infty\\).

","chapter_express_content":"","public_visibility":true,"book_slug":"elementary-theory-of-matrices","slug":"limits-of-linear-transformations","regular_authors":"","express_authors":"","created_at":"2024-07-22T14:01:51.434Z","__v":0,"allow_part_see":false,"header_data":[]}]},"seoMetadata":{"meta_title":"","meta_description":"","meta_keywords":"","meta_author":"","og_image":""},"error":null}},"actionData":null,"errors":null}

Table of Contents

1. Real and complex vector spaces

2. Linear dependence and dimension

3. Inner products and inner product spaces

4. Schwartz Inequality

5. Triangle Inequality in inner product spaces

6. \\(n\\)-dimensional complex Euclidean space

7. Linear subspaces

8. Orthonormal sets

9. Intersection and linear sum of linear subspaces

10. Bessel’s Inequality

11. Characterization of complete orthonormal sets

12. Erhardt Schmidt orthogonalization process

13. The projection theorem

14. Calculus of linear subspaces

15. Representation of linear functions

Table of Contents

1. Linear transformations in vector spaces

2. The dual of a transformation

3. Matrices associated with linear transformations

4. Isomorphism of matrices and transformations

5. The forms \\(\\langle Ax, x\\rangle\\) and \\(\\langle Ax, y \\rangle\\)

6. Hermitian transformations

7. Positive definite transformations

8. Algebraic combinations of Hermitian and definite transformation

9. Matricial characterizations of Hermitian transformations

10. Unitary transformations

11. Automorphisms of inner product spaces

12. Change of basis in an inner product space

13. Matricial characterization of unitary transformations

14. Orthogonal projections in an inner product space

15. Products of projections

16. Sums of projections

17. Differences of projections

18. Relation between projections and involutions

19. The rank of a linear transformation

20. The norm of a linear transformation

21. Expressions for the norm of a transformation

22. Upper and lower bounds of a Hermitian transformation

Table of Contents

1. Determinants

2. Invariant linear subspaces

3. Determining invariant linear subspaces of an arbitrary transformation

4. Upper triangular and lower triangular forms of a matrix

5. Spectral representation of Hermitian transformations

6. Reduction to principal axes

7. Terminology of spectral theory

8. Linear combinations of orthogonal projections

9. Normal transformations and the Cartesian decomposition

10. Functions of normal transformations

11. The square root of a non-negative transformation

12. Product of non-negative transformations

13. Commutative Hermitian transformations

14. The correspondence \\(U=\\exp(2\\pi A)\\)

15. Cayley transform

16. Polar decomposition

17. Polar conditions for normality

Table of Contents

1. Transformations of rank one

2. The Hadamard product of non-negative matrices

3. The dual space of a vector space

4. The dual space of an inner product space

5. Reflexivity of inner product spaces

6. Direct sum of vector spaces

7. Tensor product of vector spaces

8. Dimension of a tensor product

9. The dual of a tensor product

10. Tensor product of inner product spaces

11. The inner product in a tensor product

12. Tensor product of transformations

13. Kronecker products of matrices

14. Properties of tensor product transformations

Table of Contents

1. The space of linear transformations

2. The minimal polynomial of a transformation

3. The order of a vector

4. Quotient spaces

5. Cyclic spaces

6. Vectors of maximal order

7. The rational canonical form

Elementary Theory of Matrices

By Unknown Author

Table of Contents

footer_h_1

footer_social_media