Lab | Markov processes

41.1 Markov processes

Key points

A stochastic process is said to be a Markov process if the conditional distribution $X_{u} | i_{t}$ given information up to time $t$ depends only on $x_{t}$ (41.2).

The forecast distribution of a random walk process $X_{t + Δ t} | i_{t}$ (36.25), where we recall that $i_{t}$ is the realized information as represented by the information set

I_{t} \equiv v e c (X_{t}, \dots, X_{0})

(41.1)

depends only on the most recent observation $x_{t}$ and discards every other piece of information.

In this section, we consider a larger class of probabilistic processes, namely discrete time Markov processes, that preserve such a useful feature by definition. They play an important role in quantitative finance because they can be simulated and hence forecasted in an efficient manner.

The below Table 41.3 summarizes (time-homogeneous) Markov processes across the four macro-categories highlighted in Table 35.1.

41.1.1 Theory

A probabilistic (35.11) discrete time (35.8) stochastic process ${X_{t}}_{t = 0, 1, \dots}$ (35.7) is said to be a Markov process if it satisfies the Markov property: the conditional distribution $X_{u} | i_{t}$ (3.14) of the process at time $u$ given we know the realization $I_{t} = i_{t}$ (41.1) of the process up to time $t$ depends only on the most recent position $x_{t}$ of the process at time $t$

X_{u} | i_{t} \sim f_{u, t} (x_{u} | x_{t}) .

(41.2)

The conditional pdf $f_{u, t} (x_{u} | x_{t})$ (3.14) depends on both start and end times $t, u$ and is known in literature as the transition density function associated with the Markov process ${X_{t}}_{t \in T}$ . It is the density of the probability that the process, starting from $x_{t}$ at time $t$ , will eventually be at $x_{u}$ at time $u$ .

Alert 41.1. We use (41.2) formally as definition of continuous time Markov processes as well.

Example 41.1. Let us consider the AR $(1)$ process $X_{t}$ (37.14) introduced as model for the $2$ years key rate of the yield curve. Assume the driving shock (37.15) Gaussian (37.93), in such a way that $X_{t}$ (37.14) is a Gaussian process (35.41) and the conditional distributions $X_{u} | i_{t}$ (3.14) can be computed from the linear forecast, as highlighted in (35.82)

X_{u} | i_{t} \sim N (E {X_{u} ∥ i_{t}}, C v {X_{u} ∥ i_{t}}) .

(41.3)

The linear forecast $E {X_{u} ∥ i_{t}}$ reads as in (37.50). The forecast error matrix $C v {X_{u} ∥ i_{t}}$ reads as in (37.53). Both follows because all (V)AR $(1)$ forecasts $X_{u} ∥ i_{t}$ depend only on $x_{t}$ (37.135). Summarizing

X_{u} | i_{t} \sim X_{u} | x_{t} \sim N (\frac{1 - b^{u - t}}{1 - b} a + b^{u - t} x_{t}, \frac{1 - b^{2 (u - t)}}{1 - b^{2}} σ^{2}),

(41.4)

therefore $X_{t}$ (35.43) satisfies the Markov property (41.2). In particular for $u = t + 1$

X_{t + 1} | i_{t} \sim X_{t + 1} | x_{t} \sim N (a + b x_{t}, σ^{2}) .

(41.5)

If the distribution (41.2) depends only of the lag $τ \equiv t - u$ , the process is a time-homogeneous Markov process

X_{t + τ} | i_{t} \sim f_{τ} (x_{t + τ} | x_{t}),

(41.6)

otherwise it is said to be a time-inhomogeneous Markov process.

Alert 41.2. Formally, any time-inhomogeneous Markov process (41.2) can be embedded in a time-homogeneous Markov process (41.6) by considering time as “state”

X_{t} \leftarrow {t, X_{t}} .

(41.7)

However, it is often more convenient to explicitly study time-inhomogeneous processes, as in the case of Markov chains (Section 53a.7.1).

Example 41.2. Examples of Markov processes.
We continue from Example 41.1. The conditional distribution (41.3) depends only on the lag $t - u$ , therefore the process $X_{t}$ (35.43) is a time-homogeneous Markov process (41.6).

Properties

As consequence of the Markov property (41.2), the transition density functions $f_{τ} (\cdot | \cdot)$ are all related each other by the Chapman-Kolmogorov equations [W]

f_{τ} (x_{t + τ} | x_{t}) = \int f_{τ - υ} (x_{t + τ} | x_{t + υ}) f_{υ} (x_{t + υ} | x_{t}) d x_{t + υ} .

(41.8)

The Chapman-Kolmogorov equations (41.8) imply that if time is discrete, we need to require the Markov property (41.6) for the unit step only 70.3

X_{t + 1} | i_{t} \sim f_{1} (x_{t + 1} | x_{t}) .

(41.9)

Moreover, the Chapman-Kolmogorov equations (41.8) implies that all the distributions $f_{τ} (\cdot | \cdot)$ are fully determined by the unit-step distribution $f_{1}$ (41.9). Therefore, the finite dimensional distributions (35.18) can all be computed starting from $f_{1}$ and the distribution of the initial state 70.3 .

Finally,the Chapman-Kolmogorov equations (41.8) yields the forecast distribution at the horizon $t + Δ t$

X_{t + Δ t} | i_{t} \sim f_{Δ t} (x_{t + Δ t} | x_{t}) .

(41.10)

Example 41.3. We continue from Example 41.2. Let us use the Chapman-Kolmogorov equation (41.8) to compute the forecast distribution $X_{t + 2} | i_{t}$ (41.10)

\begin{matrix} f_{2} (x_{t + 2} | x_{t}) & = \int f_{1} (x_{t + 2} | x_{t + 1}) f_{1} (x_{t + 1} | x_{t}) d x_{t + 1} & (41.11) = \int \frac{1}{\sqrt{2 π σ}} e^{- \frac{{(x_{t + 2} - a - b x_{t + 1})}_{t + 2}^{2}}{2 σ^{2}}} \frac{1}{\sqrt{2 π σ}} e^{- \frac{{(x_{t + 1} - a - b x_{t})}_{t + 1}^{2}}{2 σ^{2}}} d x_{1} = \frac{1}{\sqrt{2 π σ^{2} (1 + b^{2})}} e^{- \frac{1}{2 (1 + b^{2}) σ^{2}} {(x_{t + 2} - (1 + b) a - b^{2} x_{t})}_{t + 2}^{2}}, \end{matrix}

where we replaced $f_{1} (x_{t + 2} | x_{t + 1})$ , $f_{1} (x_{t + 1} | x_{t})$ with the normal pdf (49.5) with parameters specified by (41.5). The ensuing pdf identifies the below normal distribution (49.5)

f_{2} (x_{t + 2} | x_{t}) \sim N ((1 + b) a + b^{2} x_{t}, (1 + b^{2}) σ^{2}),

(41.12)

in accordance with (41.4) with $u = 2$ .

Example 41.4. Applications of the Chapman-Kolmogorov equation
In Example 41.11 we use the Chapman-Kolmogorov equation (41.8) as forecasting tool of credit ratings, which are modeled through discrete time, discrete state Markov processes, i.e. Markov chains (Section 41.2.1).

In practice, the forecast (36.25) is obtained via Monte Carlo simulations through “structural” representation (42.7). We refer to Section 42.1 for more details.

41.1.2 Relevant cases

In what follows we introduce some examples of Markov processes that are important in applications. We have already encountered some of them.

Random walk/VAR $(1)$

The random walk (36.6) is the first, fundamental example of Markov process (41.9).

More generally, any VAR $(1)$ (37.66) with i.i.d. shocks $ε_{t} \sim$ $f_{ε}$ (37.92) is a time-homogenous Markov process (41.6) with unit step distribution (41.9)

X_{t + 1} | i_{t} \sim f_{ε} (x_{t + 1} - b x_{t} - a),

(41.13)

from which we retrieve the Gaussian VAR $(1)$ (37.66)-(37.93) when the driving strong white noise is Gaussian. Note that VAR $(1)$ (37.66) driven by shocks that are only uncorrelated across time are not necessarily Markov processes (41.6).

The continuous time Markov processes (41.2) that are mostly used in practice are the continuous time counterpart of the random walk (36.6) and the Gaussian VAR $(1)$ , namely Levy processes Section 36.3.1 and the (multivariate) Ornstein-Uhlenbeck processes Section 37.6 respectively.

Example 41.5. Examples of Markov processes in applications.
Arguably the VAR $(1)$ process with normal shocks and its continuous time counterpart, the multivariate Ornstein-Uhlenbeck process are among the most popular models in financial applications. For instance, in Example 37.20 and Example 37.37 we use them to model the dynamics of a set of key rates of the yield curve (65.16).

Markov chains, namely Markov processes (41.2) taking value on a finite, discrete set of values, play a fundamental role in quantitative finance as main tool to model credit rating migrations. We study them in Section 41.2.1.

CIR process

From the Ornstein-Uhlenbeck process we can derive another mean-reverting, Markov process, useful to model a continuum-state process that can only be positive. Specifically, consider a zero-mean Ornstein-Uhlenbeck process $X_{t}$ (37.206). Then define $Y_{t} \equiv X_{t}^{2}$ . By applying Ito’s formula to $Y_{t}$ , we obtain the SDE of the square-root process (also called CIR process)

d Y_{t} = - ˜ θ (Y_{t} - ˜ μ) d t + ˜ σ \sqrt{Y_{t}} d B_{t},

(41.14)

where $˜ θ \equiv 2 θ$ , $˜ μ \equiv σ^{2} ∕ 2 θ$ , and $˜ σ \equiv 2 σ$ 70.5 . To ensure that (41.14) stays positive at all times, it is required that

2 ˜ θ ˜ μ \geq {˜ σ}^{2},

(41.15)

known also as Feller condition, see [Aquan-Assee, 2009].

The conditional distribution $X_{t + Δ t} | i_{t}$ , where $i_{t}$ is defined according to the notation convention (22), reads

Y_{t + Δ t} d = Y_{t} + c_{Δ t} Z,

(41.16)

where $c_{Δ t} \equiv (1 - e^{- ˜ θ Δ t}) {˜ σ}^{2} ∕ (4 ˜ θ)$ , and $Z$ has a non-central chi-squared distribution $χ^{2} (k, λ_{Δ t})$ [W], with degrees of freedom $k \equiv 4 ˜ θ ˜ μ ∕ {˜ σ}^{2}$ and shape parameter $λ_{Δ t} \equiv y_{t_{n o w}} e^{- ˜ θ Δ t} ∕ c_{Δ t}$ . In the long run $Δ t \to \infty$ , the square root process (41.14) converges to its stationary unconditional distribution, which is gamma [W]

Y_{\infty} \sim G a m m a (\frac{k}{2}, \frac{{˜ σ}^{2}}{2 ˜ θ}),

(41.17)

see [Brigo et al., 2009] and [W].

Example 41.6. The CIR process (41.16) is used in [Cox et al., 1985] to model interest rates. Hence the CIR process provides an alternative to the Vasicek model in Example 37.36. Furthermore, the CIR process (41.16) is used to model stochastic volatility, see Example 41.23.

To further generalize the Ornstein-Uhlenbeck process in a Markovian fashion we can also:

Use Levy processes instead of the Brownian motion to drive randomness, see [Barndorff-Nielsen and Shephard, 2001];
Consider the multivariate formulation (37.224).

X_{u} | i_{t} \sim f_{u, t} (x_{u} | x_{t}) .

X_{t + Δ t} | i_{t} \sim f_{Δ t} (x_{t + Δ t} - x_{t} - μ),

{X_{t}}_{t \in T} \sim f_{X} (\cdot)

T = Z,

{X_{t} \equiv ⎛ ⎜ ⎜ ⎝ \begin{matrix} X_{1, t} ⋮ X_{¯ n, t} \end{matrix} ⎞ ⎟ ⎟ ⎠}_{t \in T} where X_{t} (\cdot) : \begin{matrix} Ω \to R^{¯ n} ω \mapsto X_{t} (ω) \end{matrix},

f (x_{1}, \dots, x_{¯ n} | z_{1}, \dots, z_{¯ k})      f (x | z) \equiv \frac{f (x_{1}, \dots, x_{¯ n}, z_{1}, \dots, z_{¯ k})}{f (z_{1}, \dots, z_{¯ k})}      \frac{f (x, z)}{f (z)} .

I_{t} \equiv v e c (X_{t}, \dots, X_{0})

X_{t + 1} = a + b X_{t} + {˚ ε}_{t + 1},

{˚ ε}_{t} \sim w w n (0, σ^{2})

{˚ ε}_{t} \sim N (0, σ_{ε}^{2}) i.i.d.,

X_{t} \sim G P (μ_{X} (\cdot), k_{X} (\cdot, \cdot)) .

X_{P} | x_{G} \sim N (E {X_{P} ∥ x_{G}}, C v {X_{P} ∥ x_{G}}) .

E {X_{t} | | x_{0}} = b^{t} x_{0} + \frac{1 - b^{t}}{1 - b} a .

C v {X_{t} | | x_{0}} = \frac{1 - b^{2 t}}{1 - b^{2}} σ_{˚ ε}^{2} .

E {X_{u} ∥ x_{t}, \dots} = E {X_{u} ∥ x_{t}}, C v {X_{u} ∥ x_{t}, \dots} = C v {X_{u} ∥ x_{t}} .

X_{t} \sim G P ((\begin{matrix} 0.275 % 0.459 % \end{matrix})      μ_{X}, {(\begin{matrix} 0.971 & 0 0.0127 & 0.992 \end{matrix})}^{u - t} \times (\begin{matrix} {(0.095 %)}^{2} & {(0.150 %)}^{2} {(0.150 %)}^{2} & {(0.261 %)}^{2} \end{matrix})      k_{X}} .

X_{t + τ} | i_{t} \sim f_{τ} (x_{t + τ} | x_{t}),

X_{u} | i_{t} \sim N (E {X_{u} ∥ i_{t}}, C v {X_{u} ∥ i_{t}}) .

f_{τ} (x_{t + τ} | x_{t}) = \int f_{τ - υ} (x_{t + τ} | x_{t + υ}) f_{υ} (x_{t + υ} | x_{t}) d x_{t + υ} .

X_{t + 1} | i_{t} \sim f_{1} (x_{t + 1} | x_{t}) .

X_{P} \sim f_{X} (x_{n_{1}, t_{1}}, \dots, x_{n_{k}, t_{k}}) .

X_{t + Δ t} | i_{t} \sim f_{Δ t} (x_{t + Δ t} | x_{t}) .

f_{μ, σ^{2}}^{N} (x) = \frac{1}{σ \sqrt{2 π}} e^{- \frac{1}{2} {(\frac{x - μ}{σ})}^{2}} .

X_{t + 1} | i_{t} \sim X_{t + 1} | x_{t} \sim N (a + b x_{t}, σ^{2}) .

X_{u} | i_{t} \sim X_{u} | x_{t} \sim N (\frac{1 - b^{u - t}}{1 - b} a + b^{u - t} x_{t}, \frac{1 - b^{2 (u - t)}}{1 - b^{2}} σ^{2}),

X_{t + 1} = g (X_{t}, ε_{t + 1}),

X_{t + 1} = μ + X_{t} + ε_{t + 1} \Leftrightarrow Δ X_{t} = μ + ε_{t} \sim f (ε) i.i.d.,

X_{t + 1} = a + b X_{t} + {˚ ε}_{t + 1},

ε_{t} \sim f_{ε} i.i.d.

τ \to Y_{t} (τ) .

d X_{t} = - θ X_{t} d t + σ d B_{t} .

d Y_{t} = - ˜ θ (Y_{t} - ˜ μ) d t + ˜ σ \sqrt{Y_{t}} d B_{t},

T_{t} \equiv ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} (- \infty, t] & continuous time, inception at “infinitely remote past” {- \infty, \dots t - 1, t} & discrete time, inception at “infinitely remote past” [0, t] & continuous time, finite interval {0, \dots t - 1, t} & discrete time, inception at zero. \end{matrix}

Y_{t + Δ t} d = Y_{t} + c_{Δ t} Z,

d X_{t} = - (θ X_{t} - μ) d t + σ d B_{t},

	Discrete time	Continuous time

Probabilistic	$X_{t + 1} \| i_{t} \sim F (x_{t + 1} \| x_{t})$	$X_{t + τ} \| i_{t} \sim F_{τ} (x_{t + τ} \| x_{t})$
$\cup$
Gaussian	$X_{t + 1} \| i_{t} \sim N (a + b x_{t}, σ^{2})$	$d X_{t} = - (θ X_{t} - μ) d t + σ d B_{t}$ (MV)OU Section 37.6
$↕$
Mean-covariance	$X_{t + 1} = a + b X_{t} + {˚ ε}_{t + 1}$ (V)AR $(1)$ Section 37.3

41.1 Markov processes

41.1.1 Theory

Properties

41.1.2 Relevant cases

Random walk/VAR $(1)$

CIR process

Bibliography

External Links

Discussions

41.1 Markov processes

41.1.1 Theory

Properties

41.1.2 Relevant cases

Random walk/VAR(1)

CIR process

Bibliography

External Links

Discussions

Random walk/VAR $(1)$