The Method of Successive Approximations

Suppose for the moment that a solution $y(x)$ is known, which reduces to $y_0$ when $x=x_0$; this solution evidently satisfies the relation

$$y(x)=y_0+{\int }_{x_0}^{x}f\{t,w(t)\}dt.$$

This relation is, in reality, an integral equation,¹ involving the dependent variable under the integral sign. Let the function $y(x)$ be now regarded as unknown; the integral equation may then be solved by a method of successive approximation in the following manner.

Let $x$ lie in the interval $[x_0,x_0+h]$ and consider the sequence of functions $y_1(x),y_2(x),\dots ,y_n(x)$ defined as follows:

$$y_1(x)=y_0+{\int }_{x_0}^{x}f\{t,y_0\}dt,$$$$y_2(x)=y_0+{\int }_{x_0}^{x}f\{t,y_1(t)\}dt,$$$$⋮$$$$y_n(x)=y_0+{\int }_{x_0}^{x}f\{t,y_{n-1}(t)\}dt.$$

It will now be proved

(a) that, as $n$ increases indefinitely, the sequence of functions $y_n(x)$ tends to a limit which is a continuous function of $x$,

(b) that the limit-function satisfies the differential equation, and

(c) that the solution thus defined assumes the value $y_0$ when $x=x_0$ and is the only continuous solution which does so.

In the first place, it will be proved by induction that, when $x$ lies in the interval considered, $|y_n(x)-y_0|\le b$. Suppose then that $|y_{n-1}(x)-y_0|\le b$; it follows that $|f\{t,y_{n-1}(t)\}|\le M$, and consequently

$$|y_n(x)-y_0|\le {\int }_{x_0}^x|f\{t,y_{n-1}(t)\}|dt\le M(x-x_0)\le Mh\le b.$$

But evidently $|y_1(x)-y_0|\le b$; it is therefore true that

$$|y_n(x)-y_0|\le b,$$

for all values of $n$. It follows that $f\{x,y_n(x)\}\le M$ when .

It will now be proved, in a similar way, that

For suppose it to be true that, when $x_0\le x\le x_0+h$,

then

\begin{aligned} |y_n(x) - y_{n-1}(x)| &\leq \int_{x_0}^{x} |f\{t,\, y_{n-1}(t)\} - f\{t,\, y_{n-2}(t)\}|\,dt\\ &< \int_{x_0}^{x} K\,|y_{n-1}(t) - y_{n-2}(t)|\,dt, \end{aligned}

by the Lipschitz condition, so that

\begin{aligned} |y_n(x) - y_{n-1}(x)| &< \frac{MK^{n-1}}{(n-1)!} \int_{x_0}^{x} |t - x_0|^{n-1}\,dt \\ &= \frac{MK^{n-1}}{n!}\,|x - x_0|^n. \end{aligned}

But the inequality is clearly true when $n=1$, it is therefore true for all values of $n$. In the same way it can be proved to hold when $x_0-h\le x\le x_0$, it is therefore true for $|x-x_0|\le h$.

It follows that the series

$$y_0+\sum _{r=1}^{\mathrm{\infty }}\{y_r(x)-y_{r-1}(x)\}$$

is absolutely and uniformly convergent when $|x-x_0|\le h$ and moreover each term is a continuous function of $x$. But

$$y_n(x)=y_0+\sum _{r=1}^n\{y_r(x)-y_{r-1}(x)\};$$

consequently the limit-function

$$y(x)=\underset{n\to \mathrm{\infty }}{lim}{y}_n(x)$$

exists and is a continuous function of $x$ in the interval $(x_0-h,x_0+h)$.²

Now if it is true that

$$\underset{n\to \mathrm{\infty }}{lim}{y}_n(x)=y_0+\underset{n\to \mathrm{\infty }}{lim}{\int }_{x_0}^{x}f\{t,y_{n-1}(t)\}dt=y_0+{\int }_{x_0}^x\underset{n\to \mathrm{\infty }}{lim}f\{t,y_{n-1}(t)\}dt,$$

it will follow that $y(x)$ is a solution of the integral equation

$$y(x)=y_0+{\int }_{x_0}^{x}f\{t,y(t)\}dt.$$

That the inversion of the order of integration and procedure to the limit is legitimate may be proved as follows:

\begin{aligned} \left|\int_{x_0}^{x} [f\{t,\, y(t)\} - f\{t,\, y_{n-1}(t)\}]\,dt\right| &< K \int_{x_0}^{x} |y(t) - y_{n-1}(t)|\,dt \\ &< K\epsilon_n\,|x - x_0| < K\epsilon_n h, \end{aligned}

where ${ϵ}_n$ is independent of $x$ and tends to zero as $n$ tends to infinity.

The function $f\{t,y(t)\}$ is continuous in the interval ; consequently

$$\frac{dy(x)}{dx}=\frac{d}{dx}{\int }_{x_0}^{x}f\{t,y(t)\}dt=f\{x,y(x)\}.$$

The limit-function $y(x)$ therefore satisfies the differential equation; it also reduces to $y_0$ when $x$ assumes the value $x_0$.

It remains to prove that this solution $y(x)$ is unique. Suppose $Y(x)$ to be a solution distinct from $y(x)$, satisfying the initial condition $Y(x_0)=y_0$, and continuous in an interval (x_0,\, x_0 + h') where h' < h and h' is such that the condition

is satisfied for this interval. Then, since $Y(x)$ is a solution of the given equation, it satisfies the integral equation

$$Y(x)=y_0+{\int }_{x_0}^{x}f\{t,Y(t)\}dt,$$

and consequently

$$Y(x)-y_n(x)={\int }_{x_0}^x[f\{t,Y(t)\}-f\{t,y_{n-1}(t)\}]dt.$$

Let $n=1$, then

$$Y(x)-y_1(x)={\int }_{x_0}^x[f\{t,Y(t)\}-f\{t,y_0\}]dt,$$

and it follows from the Lipschitz condition that

Similarly, when $n=2$,

\begin{aligned} |Y(x) - y_2(x)| &< \int_{x_0}^{x} |f\{t,\, Y(t)\} - f\{t,\, y_1(t)\}|\,dt \\ &< K \int_{x_0}^{x} |Y(t) - y_1(t)|\,dt \\ &< K \int_{x_0}^{x} Kb(t - x_0)\,dt = \tfrac{1}{2} K^2 b (x - x_0)^2, \end{aligned}

and in general

whence

$$Y(x)=\underset{n\to \mathrm{\infty }}{lim}{y}_n(x)=y(x)$$

for all values of $x$ in the interval (x_0,\, x_0 + h'), and therefore the new solution is identical with the old. There is therefore one and only one continuous solution of the differential equation which satisfies the initial conditions.

3.2.1 Observations on the Method of Successive Approximation

The two main assumptions which were made regarding the behaviour of the function $f(x,y)$ in the domain $D$, namely the assumption of continuity and that of the Lipschitz condition are quite independent of one another. The question arises as to the necessity of these assumptions; it is therefore well to look a little more closely into them and to enquire whether or not they may be unduly restrictive.

In the first place, it will be seen that the continuity of $f(x,y)$ is not necessary for the existence of a continuous solution; in fact all that the previous investigation demands is that $f(x,y)$ be bounded, and that all integrals of the type

$${\int }_{x_0}^x|f\{t,y_n(t)\}|dt$$

exist. In particular, $f(x,y)$ may admit of a limited number of finite discontinuities.³

Thus, for instance, the differential equation

\begin{aligned} \frac{dy}{dx} &= y(1 - 2x) \text{ when } x > 0, \\ &= -y(2x - 1) \text{ when } x < 0 \end{aligned}

admits of a continuous solution satisfying the initial condition $y=1$ when $x=1$. This solution is

\begin{aligned} y &= e^{x - x^2} \text{ when } x \geq 0, \\ &= e^{x^2 - x} \text{ when } x \leq 0, \end{aligned}

and the solution is valid for all real values of $x$, moreover it is unique.

On the other hand, the Lipschitz condition, or a condition of a similar character, must be imposed in order to ensure the uniqueness of the solution. It is not difficult to construct an equation for which the Lipschitz condition is not satisfied, and which admits of more than one continuous solution fulfilling the initial conditions.⁴

Thus, for instance, in the equation

$$\frac{dy}{dx}=\sqrt{|y|},$$

the Lipschitz condition is violated in any region which includes the line $y=0$. The equation admits of two real continuous solutions satisfying the initial conditions $x=0$, $y=0$, viz.

\begin{aligned} (1^\circ)\quad y&=0\\ (2^\circ) \quad y &= \frac{1}{4}x^2\quad\text{when } x \geq 0,\\ &= -\frac{1}{4}x^2\quad \text{when } x \leq 0. \end{aligned}

Another example is given by the equation

$$\frac{dy}{dx}=f(x,y),$$

where

\begin{aligned} f(x, y) &= \frac{4x^3 y}{x^4 + y^2} &&\text{ when } x \text{ and } y \text{ are not both zero}, \\ &= 0 &&\text{ when } x = y = 0. \end{aligned}

It is easily proved that $f(x,y)$ is a continuous function of $x$ and $y$. On the other hand

$$f(x,Y)-f(x,y)=\frac{4x^3(x^4-yY)}{(x^4+y^2)(x^4+Y^2)}(Y-y).$$

If $y=p{x}^2$, $Y=q{x}^2$,

$$|f(x,Y)-f(x,y)|=4\left|\frac{1-pq}{(1+p^2)(1+q^2)}\right|\frac{|Y-y|}{|x|}$$

and therefore the Lipschitz condition is not satisfied throughout any region containing the origin.

The equation admits of the solution

$$y=c^2-\sqrt{(x^4+c^4)},$$

$c$ being an arbitrary real constant, and thus there is an infinity of solutions satisfying the initial conditions $x=0$, $y=0$.

The question has been placed on a firm basis by Osgood,⁵ who proved that, if $f(x,y)$ be continuous in the neighbourhood of $(x_0,y_0)$, there exists in general a one-fold infinity of solutions satisfying the initial conditions. These solutions lie entirely within the area bounded by two extremal solutions

$$y=Y_1(x),y=Y_2(x).$$

A necessary and sufficient condition that there be a unique solution is that $Y_1(x)$ and $Y_2(x)$ be identical. This is the case when the Lipschitz condition is satisfied, but it is also true when the Lipschitz condition is replaced by one or other of the less restrictive conditions

\begin{aligned} |f(x, Y) - f(x, y)| &< K_1\,|Y - y|\,\log\frac{1}{|Y - y|},\\ |f(x, Y) - f(x, y)| &< K_2\,|Y - y|\,\log\frac{1}{|Y - y|}\,\log\log\frac{1}{|Y - y|},\\ &\vdots \end{aligned}

in which $K_1,K_2,\dots$ are constants.

The constant $K$ which occurs in the Lipschitz condition determines, for any given value of $x$, the rapidity with which the comparison series

$$\sum \frac{M{K}^{n-1}}{n!}|x-x_0{|}^n$$

converges, and therefore gives an indication of the utility of the series

$$y_0+\sum _{r=1}^n\{y_r(x)-y_{r-1}(x)\}$$

as an approximation to the limit-function $y(x)$. Thus if $K$ were small, $y_n(x)$ would tend to the limit $y(x)$ more rapidly than if $K$ were large. Now in most cases occurring in practice $K$ is the upper bound of

$$\frac{\partial f(x,y)}{\partial y}$$

in the domain $D$. To make use of this fact, consider the family of curves

$$f(x,y)=C,$$

for all values of the constant $C$. The typical curve of this family is such that it intersects each integral curve in a point at which the gradient of the latter curve is $C$. For this reason the curves are known as the isoclinal lines.⁶ Let the isoclinal lines be plotted for a succession of discrete equally-spaced (e.g. integral) values of $C$, and let a line be drawn parallel to the $y$-axis. Then the intervals along this line in which the points of intersection with the isoclinal lines are densely packed correspond to large values of $K$, whereas those intervals in which the intersections are more widely spaced correspond to smaller values of $K$. This brings out the fact that the regions in which the method of successive approximations may most successfully be applied as a practical method of computation are those in which the isoclinal lines tend to run more or less parallel to the $y$-axis.⁷

The method of successive approximations leads to a solution which was shown to converge in the interval $|x-x_0|\le h$, where $h$ is the least of $a$ and $b/M$. But, as was remarked in passing, the assumption originally made that certain conditions are satisfied throughout the region $|x-x_0|\le a$, $|y-y_0|\le b$ was unnecessarily restrictive. If a region $|x-x_0|\le k$, $|y-y_0|\le M|x-x_0|$ can be found such that $f(x,y)$ satisfies the necessary conditions in that region, and $M$ is the upper bound of $|f(x,y)|$, then $k$ will certainly not be less, and may quite conceivably be greater, than $h$. Several writers have succeeded in thus extending the range in which the solution can be proved to converge, but no general method of determining the exact boundaries of the interval of convergence has yet been discovered.

3.2.2 Variation of the Initial Conditions

Let the given initial condition that $y=y_0$ when $x=x_0$ be replaced by the new condition $y=y_0+\eta$ when $x=x_0$, where $(x_0,y_0+\eta )$ is a point within the domain $D$ such that $|\eta |\le \delta$. Then, in place of the sequence of functions

$$y_1(x),y_2(x),\dots ,y_n(x),$$

as defined in § 3.2, there now arises the sequence

$$Y_1(x),Y_2(x),\dots ,Y_n(x),$$

defined as follows:

$$Y_1(x)=y_0+\eta +{\int }_{x_0}^{x}f\{t,y_0+\eta \}dt,$$$$Y_2(x)=y_0+\eta +{\int }_{x_0}^{x}f\{t,Y_1(t)\}dt,$$$$⋮$$$$Y_n(x)=y_0+\eta +{\int }_{x_0}^{x}f\{t,Y_{n-1}(t)\}dt.$$

The existence and uniqueness of the solution

$$Y(x)=\underset{n\to \mathrm{\infty }}{lim}{Y}_n(x)$$

then follow as before. Now

\begin{aligned} |Y_1(x) - y_1(x)| &\leq \delta + \int_{x_0}^{x} |f\{t,\, y_0 + \eta\} - f\{t,\, y_0\}|\,dt \\ &< \delta + K\delta\,|x - x_0|, \end{aligned}\begin{aligned} |Y_2(x) - y_2(x)| &\leq \delta + \int_{x_0}^{x} |f\{t,\, Y_1(t)\} - f\{t,\, y_1(t)\}|\,dt \\ &< \delta + K\delta\,|x - x_0| + \tfrac{1}{2}K^2\delta\,|x - x_0|^2, \end{aligned}

and, by induction,

\begin{aligned} |Y_n(x) - y_n(x)| &\leq \delta + K\delta\,|x - x_0| + \ldots + \frac{1}{n!}K^n\delta\,|x - x_0|^n \\ &< \delta e^{K|x - x_0|}, \end{aligned}

so that, in the limit,

$$|Y(x)-y(x)|\le \delta e^{K|x-x_0|}.$$

Consequently, when $|x-x_0|\le h$, the solution is uniformly continuous in the initial value $y_0$. To bring out this fact, it may be written in either of the forms

$$y(x,y_0)\text{and}y(x-x_0,y_0).$$

Moreover,

and consequently

$$\left|\frac{\partial y_n(x,y_0)}{\partial y_0}\right|\le 1+K|x-x_0|+\dots +\frac{1}{n!}{K}^n|x-x_0{|}^n,$$

from which it may be deduced that the series

$$\frac{\partial y(x,y_0)}{\partial y_0}=1+\sum _{n=1}^{\mathrm{\infty }}\frac{\partial \{y_n(x,y_0)-y_{n-1}(x,y_0)\}}{\partial y_0}$$

is absolutely and uniformly convergent. Therefore $y(x,y_0)$ is uniformly differentiable with respect to $y_0$ when $|x-x_0|\le h$.

A proof proceeding on similar lines to the above shows that if the differential equation involves a parameter $\lambda$, that is to say if

$$\frac{dy}{dx}=f(x,y;\lambda ),$$

where $f(x,y;\lambda )$ is single-valued and continuous and satisfies the Lipschitz condition uniformly in $D$ when ${\mathrm{\Lambda }}_1\le \lambda \le {\mathrm{\Lambda }}_2$, then the solution depends continuously upon $\lambda$, and in fact is uniformly differentiable with respect to $\lambda$ when $|x-x_0|\le h$.

3.2.3 Singular Points

A singular point may be defined as a point of the $(x,y)$-plane at which one or other of the conditions necessary for the establishment of the existence theorem ceases to hold. In fact if for the initial value-pair $(x_0,y_0)$ the solution

(a) is discontinuous, (b) is not unique, or (c) does not exist,

then the point $(x_0,y_0)$ is a singular point of the equation. As illustrations of the diverse ways in which the solutions of an equation may behave at or in the neighbourhood of a singular point, the following examples may be taken.

$$(1^{\circ })\frac{dy}{dx}=\frac{y}{x}.$$

The conditions requisite for the existence of a unique and continuous solution are fulfilled except in the neighbourhood of $x=0$. The solution corresponding to the initial value-pair $(x_0,y_0)$ is

$$y=\frac{y_0}{x_0}x,$$

when $x_0\ne 0$. If $x_0=0$ and $y_0\ne 0$, the solution reduces to $x=0$.

The only exceptional case is when $x_0=y_0=0$; the only singular point in the finite part of the $(x,y)$-plane is the origin. Now every integral-curve passes through the origin, which is a node of the integral-curves.

$$(2^{\circ })\frac{dy}{dx}=m\frac{y}{x}.$$

In this case also, the only singular point is the origin. To any other point $(x_0,y_0)$ corresponds the solution

$$y=y_0{\left(\frac{x}{x_0}\right)}^m.$$

The family of integral curves corresponding to all possible values of $(x_0,y_0)$ touch the $x$-axis at the origin if and the $y$-axis at the origin if $0 < m < 1$. Thus if , every integral-curve passes through the origin.

On the other hand, if , say $m=-p$, the solution is

$$y{x}^p=y_0{x}_0^p.$$

The family of integral-curves is asymptotic to the $x$- and $y$-axes. The degenerate curve

$$y{x}^p=0$$

passes through the origin, but no other integral-curve does so. The origin is a saddle-point, for in its neighbourhood the integral-curves resemble the contour lines around a mountain pass.

$$(3^{\circ })\frac{dy}{dx}=\frac{x+y}{x}.$$

The origin is the only singular point; to any other point $(x_0,y_0)$ corresponds the solution

$$y=\frac{y_0}{x_0}x+x\log \left|\frac{x}{x_0}\right|.$$

The origin is a node of the integral-curves.

$$(4^{\circ })\frac{dy}{dx}=-\frac{x}{y}.$$

The solution is, in general,

$$x^2+y^2=x_0^2+y_0^2.$$

No real integral-curve, except the degenerate curve $x^2+y^2=0$ passes through the origin, which is a focal point.

$$(5^{\circ })\frac{dy}{dx}=\frac{x+y}{x-y}.$$

This equation is most effectively dealt with by means of a transformation to polar co-ordinates

$$x=r\cos \theta ,y=r\sin \theta .$$

It then becomes

$$\frac{dr}{d\theta }=r;$$

the integral-curves are the family of logarithmic spirals

$$r=c{e}^{\theta }.$$

One curve of the family goes through each point of the plane except the origin. No integral-curve passes through the origin, which is a focal point of every curve of the family.

It will be noticed that all these examples are particular cases of the general form

$$\frac{dy}{dx}=\frac{ax+by}{cx+dy},$$

which may be integrated by the method of § 2.12. It will be found that, from the point of view of the behaviour of the integral-curves in the neighbourhood of the origin, the equation is of one or other of three main types according as

    I. ,

    II. ,

    III. $(b-c{)}^2+4ad=0$

In Case I the origin is a node if , and a saddle-point if ; in Case II the origin is a focal point, and in Case III a node.

Footnotes

1. Bôcher, Introduction to the Theory of Integral Equations; Whittaker and Watson, Modern Analysis, Chap. XI. ↩

2. Bromwich, Theory of Infinite Series, § 45. ↩

3. These may be discrete points or lines parallel to the $y$-axis; any other lines of discontinuity implying a violation of the Lipschitz condition throughout an interval of finite dimensions. Mie, Math. Ann. 43 (1893), p. 553, has shown that solutions exist whenever $f(x,y)$ is continuous in $x$ and discontinuous but integrable (in Riemann's sense) with respect to $x$. ↩

4. Peano, Math. Ann. 37 (1890), p. 182; Mie, loc. cit., ante; Perron, Math. Ann. 76 (1915), p. 471. ↩

5. Monatsh. Math. Phys. 9 (1898), p. 331. ↩

6. The term is due to Chrystal, see Wedderburn, Proc. Roy. Soc. Edin. 24 (1902), p. 400. ↩

7. Practical methods of approximate computation based upon the method of successive approximations have been devised by Severini, Rend. Ist. Lombard. (2) 31 (1898), pp. 657, 950; Cotton, C. R. Acad. Sc. Paris, 140 (1905), p. 494; 141 (1905), p. 177; 146 (1908), pp. 274, 510; Math. Ann. 81 (1908), p. 107. ↩