Lab | Bayesian

24.5 Bayesian

Key points

The normal-inverse-Wishart model is a tractable model for the Bayesian estimation of location and dispersion of a strong white noise.
The estimation model is normal (24.39); the prior distribution of the parameters is normal-inverse-Wishart (24.45)-(24.46); the posterior distribution is also normal-inverse-Wishart (24.53)-(24.54), with updated parameters. The posterior predictive distribution of future instances of the strong white noise is Student t (24.62).

Bayesian estimation is discussed in full generality in Section 23.8; here we focus on the estimation of the distribution of i.i.d. variables, and in particular on one of the few analytically tractable yet non-trivial Bayesian models.

Bayesian estimation generalizes the parametric maximum likelihood approach ( Section 23.4), by modeling the unknown parameters as hidden variables.

As in classical estimation, the starting point of Bayesian estimation is an estimation model for information given hidden parameters, also known as likelihood (23.169), that is assumed true. When the data are realizations of a strong white noise (36.2), as in (23.1), such distribution is the product of the distribution of the variables

f (i | θ) i.i.d. = \prod_{t = 1}^{¯ t} f (ϵ_{t} | θ),

(24.38)

much like the maximum likelihood approach (23.60).

In the sequel, we specialize the general exponential model (23.207) to the case of normal i.i.d. variables, which belong to the exponential family (Section 49.7.1). In this case, the conjugate prior and posterior distribution (23.210) is normal-inverse-Wishart, and the predictive distribution of the future variable $ε_{t}$ , with $t > ¯ t$ , (23.214) is Student $t$ . We summarize the results in Table 24.8, and we proceed to discuss them.

24.5.1 Model and sample estimators

Let us assume that the i.i.d. variables are (conditionally) multivariate normal as in (24.12)

ε_{t} | μ, σ^{2} \sim N (μ, σ^{2}),

(24.39)

which is an exponential family distribution with natural parameters $θ$ (49.413), sufficient statistics $τ (ϵ_{t})$ (49.417) and log-partition function $ψ (\cdot)$ (49.421).

From the one-to-one correspondence between natural and standard parameters $θ \Leftrightarrow (μ, σ^{2})$ (49.413), the joint likelihood (23.207) follows from (24.16) and reads 70.35

\begin{matrix} f (i | θ) & = exp (¯ t (θ_{μ}^{'} {ˆ η}_{μ} + tr (θ_{σ} {ˆ η}_{σ}) - ψ (θ) + ln h)) & (24.40) = {(2 π)}^{- \frac{¯ ı ¯ t}{2}} det {(σ^{2})}^{- \frac{¯ t}{2}} exp (- \frac{¯ t}{2} [tr ({ˆ s}^{2} {(σ^{2})}^{- 1}) + {(ˆ m - μ)}^{'} {(σ^{2})}^{- 1} (ˆ m - μ)]), \end{matrix}

where $ˆ m$ is short notation for the historical sample mean (24.2) and ${ˆ s}^{2}$ is short for the historical sample covariance (24.4).

Note that, if we apply the sample mean and sample covariance estimators, $ˆ m$ (24.2) and ${ˆ s}^{2}$ (24.4), to a randomized time series of i.i.d. variables (23.2) distributed according to the normal model (24.39), we obtain two random variables $ˆ M | μ, σ^{2}$ and ${ˆ Σ}^{2} | μ, σ^{2}$ . More precisely, the conditional distribution of the sample mean estimator is 70.20

ˆ M | μ, σ^{2} \sim N (μ, \frac{1}{¯ t} σ^{2}),

(24.41)

and the conditional distribution of the sample covariance estimation is 70.21

{ˆ Σ}^{2} | μ, σ^{2} \sim W i s h a r t (¯ t - 1, \frac{1}{¯ t} σ^{2}) .

(24.42)

Moreover, the sample mean (24.41) and the sample covariance matrix ( 24.42) are independent random variables 70.19 . Notice that the distribution of the sample mean vector (24.41) generalizes the univariate case (27.41).

Example 24.12. Sample mean and sample covariance distribution

Consider a normal model (24.39) in the univariate case $¯ ı = 1$ . Figure 24.10 shows the joint and marginal distributions of sample mean (24.41) and (co)variance (24.42) as functions of the randomized time series $I$ .

24.5.2 Normal-inverse-Wishart prior distribution

Given the normal model (24.39), we assume that the natural parameters $θ$ follow the conjugate distribution (23.209)

Θ \sim C o n j (ψ (\cdot), η^{p r i}, ν^{p r i}),

(24.43)

where $ψ (\cdot)$ is given by (49.421);

η^{p r i} \equiv (\begin{matrix} η_{μ}^{p r i} v e c (η_{σ}^{p r i}) \end{matrix}) \equiv (\begin{matrix} μ^{p r i} v e c (σ^{2 p r i} + μ^{p r i} {(μ^{p r i})}^{'}) \end{matrix});

(24.44)

$σ^{2 p r i}$ is an $¯ ı \times ¯ ı$ symmetric and positive matrix; and $ν^{p r i}$ is a positive scalar.

Then, the location and dispersion parameters $(M, Σ^{2})$ of the conditionally normal i.i.d. variables (24.12), which are equivalent to the natural parameters $(M, Σ^{2}) \Leftrightarrow Θ$ , follow a normal-inverse-Wishart (NIW) distribution 70.11 . More precisely, $Σ^{2}$ follows an inverse-Wishart distribution (49.118)

{(Σ^{2})}^{- 1} \sim W i s h a r t (ν^{p r i}, \frac{1}{ν^{p r i}} {(σ^{2 p r i})}^{- 1});

(24.45)

and $M$ follows a multivariate normal distribution conditioning on $σ^{2}$

M | σ^{2} \sim N (μ^{p r i}, \frac{1}{ν^{p r i}} σ^{2}) .

(24.46)

Note that the general conjugate distribution (23.209) has only one parameter $ν^{p r i}$ to model the uncertainty of the prior. For the NIW distribution, however, we are able to specify another dispersion parameter $t^{p r i}$ for the location $M$ , i.e.

M | σ^{2} \sim N (μ^{p r i}, \frac{1}{t^{p r i}} σ^{2}),

(24.47)

and the posterior would still be the NIW, which we will show in the next section.

The above assumptions imply that the marginal distribution of $M$ is Student $t$ , see [Murphy, 2007]

M \sim t (μ^{p r i}, \frac{ν^{p r i}}{t^{p r i} (ν^{p r i} - ¯ ı + 1)} σ^{2 p r i}, ν^{p r i} - ¯ ı + 1) .

(24.48)

Hence the expectation (49.137) reads

E {M} = μ^{p r i},

(24.49)

and the covariance (49.154) reads

C v {M} = \frac{ν^{p r i}}{ν^{p r i} - 2} \frac{1}{t^{p r i}} σ^{2 p r i} .

(24.50)

Therefore $μ^{p r i}$ and $t^{p r i}$ represent respectively the statistician’s prior best guess and confidence for the location parameter.
The expectation and the covariance of the inverse-Wishart prior (24.45) read as in (49.121)-(49.122). From the expectation of a Wishart random variable (49.109) it follows that

E {{(Σ^{2})}^{- 1}} = {(σ^{2 p r i})}^{- 1},

(24.51)

and from the covariance (49.110) we have

C v {{(Σ^{2})}^{- 1}} = \frac{1}{ν^{p r i}} (I_{{¯ ı}^{2}}^{2} + K_{¯ ı, ¯ ı}) ({(σ^{2 p r i})}^{- 1} \otimes {(σ^{2 p r i})}^{- 1}) .

(24.52)

Hence we obtain that $σ^{2 p r i}$ and $ν^{p r i}$ respectively represent the statistician’s prior best guess and the confidence for the dispersion parameter.

Example 24.13. Normal-inverse-Wishart prior distribution

Consider a univariate i.i.d. variable, conditionally distributed as in (24.12), $ε_{t} | μ, σ^{2} \sim N (μ, σ^{2})$ with unknown location and dispersion parameters $(μ, σ^{2})$ . Then we suppose that ${(Σ^{2})}^{- 1}$ is univariate Wishart-distributed (24.45), or equivalently gamma (49.112), and $M | σ^{2}$ is univariate normally distributed (24.47).
In Figure 24.11 we show the joint prior distribution of $(M, Σ^{2})$ by varying the parameters $σ^{2 p r i}, ν^{p r i}, μ^{p r i}, t^{p r i}$ .

Example 24.14. Consider the GARCH residuals (53b.36) of $¯ n = 60$ stocks in the S&P 500.
We assign flexible probabilities by state-time conditioning on the VIX smoothed-scored log-return as in Example 23.4 and we compute the corresponding HFP covariance ${ˆ s}^{2}$ (24.3). Then, we shrink the HFP covariance to a correlation ${ˆ s}^{2} \leftarrow c o r r ({ˆ s}^{2})$ (10.22) to ensure that the GARCH residuals have unit variance as in (41.76).
We want to regularize the HFP correlation ${ˆ s}^{2} \leftarrow c o r r ({ˆ s}^{2})$ (10.22) so as to have as many null off-diagonal entries as possible ${[{(σ_{ε}^{2})}^{- 1}]}_{m, n}^{- 1} = 0$ (21.86) in order to obtain a parsimonious Gaussian Markov random field structure (E.11.31). Then, we perform Bayes estimation under the normal-inverse-Wishart model (Table 24.8) where we set $μ^{p r i} \equiv 0_{¯ n \times 1}, σ^{2 p r i} \equiv I_{¯ n \times ¯ n}$ and high confidence in the prior $t^{p r i}, ν^{p r i}$ .

24.5.3 Normal-inverse-Wishart posterior distribution

The NIW model (24.47)-(24.45) is conjugate of the normal likelihood (24.40) and thus the posterior (23.172) is also NIW 70.12 , with parameters that can be computed analytically.

The posterior distribution of ${(Σ^{2})}^{- 1}$ is Wishart

{(Σ^{2})}^{- 1} | i \sim W i s h a r t (ν^{p o s}, \frac{1}{ν^{p o s}} {(σ^{2 p o s})}^{- 1}),

(24.53)

and the conditional posterior distribution of $M$ is normal

M | σ^{2}, i \sim N (μ^{p o s}, \frac{1}{t^{p o s}} σ^{2}) .

(24.54)

As a result, according to (24.48), the unconditional posterior of $M$ is Student $t$ 70.18

M | i \sim t (μ^{p o s}, \frac{ν^{p o s}}{t^{p o s} (ν^{p o s} - ¯ ı + 1)} σ^{2 p o s}, ν^{p o s} - ¯ ı + 1) .

(24.55)

In the above equations, the posterior expectation $μ^{p o s}$ reads

μ^{p o s} = γ_{μ} μ^{p r i} + (1 - γ_{μ}) ˆ m,

(24.56)

where $ˆ m$ is the historical sample mean (24.2) and the parameter $γ_{μ} \in [0, 1]$ is defined using the confidence parameter for the location prior $t^{p r i}$ (24.47) and the sample size $¯ t$ as

γ_{μ} \equiv \frac{t^{p r i}}{t^{p r i} + ¯ t};

(24.57)

the posterior dispersion parameter $σ^{2 p o s}$ reads

σ^{2 p o s} = γ_{σ} σ^{2 p r i} + (1 - γ_{σ}) {ˆ s}^{2} + γ_{μ} (1 - γ_{σ}) (μ^{p r i} - ˆ m) {(μ^{p r i} - ˆ m)}^{'},

(24.58)

where ${ˆ s}^{2}$ is the historical sample covariance (24.4) and the parameter $γ_{σ} \in [0, 1]$ is defined using the confidence parameter for the dispersion prior $ν^{p r i}$ (24.45) and the sample size $¯ t$ as

γ_{σ} \equiv \frac{ν^{p r i}}{ν^{p r i} + ¯ t};

(24.59)

the parameter $t^{p o s}$ is

t^{p o s} = t^{p r i} + ¯ t,

(24.60)

and $ν^{p o s}$ is

ν^{p o s} = ν^{p r i} + ¯ t .

(24.61)

Refer to [Aitchison and Dunsmore, 1975] for more details.

The interpretation of the posterior parameters in (24.56)-(24.61) is the same as the prior parameters in (24.45)-(24.47), with the substitution $p o s \leftarrow p r i$ for all the parameters.

Example 24.15. Normal-inverse-Wishart posterior distribution

We continue from Example 24.13.
In Figure 24.12 we show three distributions:
- the joint prior distribution of $(M, Σ^{2})$ (24.45)-(24.46), with confidence parameters $(ν^{p r i}, t^{p r i})$
- the joint distribution of sample mean and sample variance $(ˆ M, {ˆ Σ}^{2})$ (24.41)-(24.42), with number of observations $¯ t$
- the joint posterior distribution of $(M, Σ^{2})$ (24.53)-(24.55).
As we let the confidence in the prior and the number of observations $¯ t$ vary, the posterior shrinks between the prior and the sample.
For a better visualization, we assume knowledge of the true parameters to plot the sample mean and sample variance pdf’s, keeping them fixed as the number of observations varies, similarly to what we do in Figure 24.10 for the frequentist analysis.

Example 24.16. We continue from Example 24.14. We compute the posterior distributions of the conditional location $M | i$ (24.53) and dispersion ${(Σ^{2})}^{- 1} | i$ (24.55).

24.5.4 Student t predictive distribution

From the normal model (24.12) and the normal-inverse-Wishart posterior (24.53)-(24.54), we can compute the posterior predictive distribution for the future variable $ε_{t}$ (23.214), which is $t$ -distributed 70.13

ε_{t} | (ϵ_{1}, \dots, ϵ_{¯ t}) \sim t (μ^{p o s}, \frac{t^{p o s} + 1}{t^{p o s}} σ^{2 p o s}, ν^{p o s}), t > ¯ t .

(24.62)

A word of reflection: the reader should not be puzzled that the i.i.d. variables $ε_{t}$ , which are supposed to be independent across time (36.2), appear to be dependent under the posterior predictive distribution (24.62).

Alert 24.1. The variables are only independent conditionally given the knowledge of the parameters $(μ, σ^{2})$ , as shown in Example 24.17.

Example 24.17. Consider two univariate i.i.d. normal variables

ε_{1} | μ d = ε_{2} | μ \sim N (μ, 1), independent

(24.63)

where the location parameter is a standard normal random variable

M \sim N (0, 1) .

(24.64)

Then the distribution of $ε_{2}$ conditional on $ε_{1} = ϵ_{1}$ reads 70.31

ε_{2} | ϵ_{1} \sim N (\frac{1}{2} ϵ_{1}, \frac{3}{2}) .

(24.65)

24.5.5 Classical equivalent, uncertainty and shrinkage

In our normal-inverse-Wishart posterior (24.53)-(24.54), the classical-equivalent (23.173) of the location parameter $M$ is $μcl_eq≡E{M}=μpos$ , as defined in (24.56). According to the shrinkage effect (23.178), if the length $¯ t$ of the time series $i \equiv {ϵ_{1}, \dots, ϵ_{¯ t}}$ is large, then the parameter $μ^{c l_e q}$ shrinks towards the sample mean $ˆ m$ ; instead, it shrinks towards the prior $μ^{p r i}$ when it is the confidence $t^{p r i}$ in the prior to be large

ˆ m large dataset ¯ t \leftarrow μ^{c l_e q} = μ^{p o s} high confidence t^{p r i} \to μ^{p r i} .

(24.66)

On the other hand, one possible classical-equivalent of the dispersion parameter $Σ^{2}$ is $σ^{2 c l_e q} \equiv σ^{2 p o s}$ as defined in (24.58), which satisfies $(σ2cl_eq)−1=E{(Σ2)−1}$ . Similar to the location, the parameter $σ^{2 c l_e q}$ shrinks towards the sample covariance ${ˆ s}^{2}$ if the length $¯ t$ of the time series is large; instead, it shrinks towards the prior $σ^{2 p r i}$ when the confidence $ν^{p r i}$ in the prior is large

{ˆ s}^{2} large dataset ¯ t \leftarrow σ^{2 c l_e q} = σ^{2 p o s} high confidence ν^{p r i} \to σ^{2 p r i} .

(24.67)

The classical-equivalent $σ^{2 c l_e q}$ has a slightly more complex expression when defined as the mode (23.174) of the posterior distribution 70.16 -70.14 .

When $t^{p r i} = ν^{p r i}$ , we recover the general shrinkage (23.213) for exponential family

(\begin{matrix} ˆ m v e c ({ˆ s}^{2} + ˆ m {ˆ m}^{'}) \end{matrix})      ˆ η large dataset ¯ t \leftarrow (\begin{matrix} μ^{p o s} v e c (σ^{2 p o s} + μ^{p o s} {(μ^{p o s})}^{'}) \end{matrix})      η^{p o s} high confidence ν^{p r i} \to (\begin{matrix} μ^{p r i} v e c (σ^{2 p r i} + μ^{p r i} {(μ^{p r i})}^{'}) \end{matrix})      η^{p r i} .

(24.68)

Example 24.18. In Figure 24.12 we highlight the shrinkage effects (24.66) and (24.67) of the classical equivalent.

Example 24.19. Bayesian estimation: stocks

We continue from Example 24.16. Our regularized estimate for the location and dispersion are the classical-equivalent parameters (23.173), respectively $μ^{c l_e q} = μ^{p o s}$ (24.56) and $σ^{2 c l_e q} = σ^{2 p o s}$ (24.58).
In the bottom plots Figure 24.13 we show:
-) the HFP correlation ${ˆ s}^{2}$ (24.3)-(10.22) and the corresponding inverse correlation ${({ˆ s}^{2})}^{- 1}$ (left);
-) the Bayes posterior $σ^{2 p o s}$ (24.58) and the corresponding inverse ${(σ^{2 p o s})}^{- 1}$ (right).
We observe that as the confidence in the prior $ν^{p r i}$ increases, the off-diagonal terms in the Bayes posterior $σ^{2 p o s}$ converge to $0$ as follows from the shrinkage effect (24.67).
In the top plot of Figure 24.13, we show the Gaussian Markov random field (21.85) structure of the i.i.d. variables (see Section 24.6.5), where the edges are missing if the absolute value of the corresponding entry of the inverse covariance falls below a given threshold (21.86). The more the confidence in the prior $ν^{p r i}$ increases, the more parsimonious the conditional independence (E.11.31) structure. Refer to Example 24.27 to compare with the glasso regularization.

The posterior of the expectation $M$ is $t$ -distributed (24.55) like the prior (24.48) and hence the uncertainty in the estimate of the expectation, as defined in (23.175), reads

s_{μ}^{2} = \frac{ν^{p o s}}{t^{p o s} (ν^{p o s} - 2)} σ^{2 p o s} .

(24.69)

It is immediate to verify that the uncertainty (24.69) shrinks to zero in the presence of a long time series or high confidence in the prior, according to the shrinkage effect (23.180).

On the other hand, the posterior of the covariance $Σ^{2}$ follows an inverse-Wishart distribution (24.53) like the prior (24.45) and then the uncertainty of the covariance estimate, as defined in (23.175), reads similarly to (24.52)

s_{σ^{2}}^{2} = \frac{1}{ν^{p o s}} (I_{{¯ ı}^{2}}^{2} + K_{¯ ı, ¯ ı}) ({(σ^{2 p o s})}^{- 1} \otimes {(σ^{2 p o s})}^{- 1}),

(24.70)

where $K_{¯ ı, ¯ ı}$ is the ${¯ ı}^{2} \times {¯ ı}^{2}$ commutation matrix (43.592). If we choose the modal square-dispersion (23.176) as uncertainty for the posterior distribution of $Σ^{2}$ we obtain different expressions 70.17 -70.15 . Again, it is immediate to verify that the uncertainty (24.70) and its alternative formulations shrink to zero in the presence of a long time series or high confidence in the prior, according to the shrinkage effect (23.180).

Example 24.20. In Figure 24.12 we highlight the shrinkage effect (23.180) for the uncertainty of location $s_{μ}^{2}$ (24.69) and of dispersion $s_{σ^{2}}^{2}$ (24.70).

ε_{t} | μ, σ^{2} \sim N (μ, σ^{2}),

{(Σ^{2})}^{- 1} \sim W i s h a r t (ν^{p r i}, \frac{1}{ν^{p r i}} {(σ^{2 p r i})}^{- 1});

M | σ^{2} \sim N (μ^{p r i}, \frac{1}{ν^{p r i}} σ^{2}) .

{(Σ^{2})}^{- 1} | i \sim W i s h a r t (ν^{p o s}, \frac{1}{ν^{p o s}} {(σ^{2 p o s})}^{- 1}),

M | σ^{2}, i \sim N (μ^{p o s}, \frac{1}{t^{p o s}} σ^{2}) .

ε_{t} | (ϵ_{1}, \dots, ϵ_{¯ t}) \sim t (μ^{p o s}, \frac{t^{p o s} + 1}{t^{p o s}} σ^{2 p o s}, ν^{p o s}), t > ¯ t .

I | θ \sim f (i | θ) .

ε_{t} \sim F (ε) i.i.d..

i \equiv (ϵ_{1}, \dots, ϵ_{t}, \dots, ϵ_{¯ t}) \equiv ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} ϵ_{1, 1} & ϵ_{1, t} & ϵ_{1, ¯ t} \cdot & \cdot & \cdot ϵ_{i, 1} & \dots & ϵ_{i, t} & \dots & ϵ_{i, ¯ t} \cdot & \cdot & \cdot ϵ_{¯ ı, 1} & ϵ_{¯ ı, t} & ϵ_{¯ ı, ¯ t} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠,

f_{θ} (i) = \prod_{t = 1}^{¯ t} f_{θ} (ϵ_{t}),

f (ϵ_{1}, \dots, ϵ_{¯ t} | θ) i.i.d. = \prod_{t = 1}^{¯ t} f (ϵ_{t} | θ) = e^{¯ t (θ^{'} ˆ η - ψ (θ) + ln ˆ h)},

Θ | i \sim C o n j (¯ t ψ (\cdot), ¯ t η^{p o s}, \frac{ν^{p o s}}{¯ t}) = C o n j (ψ (\cdot), η^{p o s}, ν^{p o s}),

\begin{matrix} f (ϵ_{t} | ϵ_{1}, \dots, ϵ_{¯ t}) & \equiv \int f (ϵ_{t} | θ) f_{ψ} (θ | i) d θ & (23.214) = \frac{g (η^{p o s}, ν^{p o s}) h (ϵ_{t})}{g (\frac{1}{1 + ν^{p o s}} τ (ϵ_{t}) + \frac{ν^{p o s}}{1 + ν^{p o s}} η^{p o s}, 1 + ν^{p o s})}, t > ¯ t . \end{matrix}

ε_{t} \sim N (μ, σ^{2}) .

θ \equiv (\begin{matrix} θ_{μ} v e c (θ_{σ}) \end{matrix}) \equiv ⎛ ⎝ \begin{matrix} {(σ^{2})}^{- 1} μ - \frac{1}{2} v e c ({(σ^{2})}^{- 1}) \end{matrix} ⎞ ⎠ .

τ (x) \equiv (\begin{matrix} τ_{μ} (x) τ_{σ} (x) \end{matrix}) \equiv (\begin{matrix} x v e c (x x^{'}) \end{matrix}) .

ψ (θ_{μ}, θ_{σ}) = - \frac{1}{4} θ_{μ}^{'} {(θ_{σ})}_{σ}^{- 1} θ_{μ} - \frac{1}{2} ln det (- 2 θ_{σ}) .

\begin{matrix} \sum_{t = 1}^{¯ t} p_{t} ln f_{θ} (ϵ_{t}) & = θ_{μ}^{'} {ˆ η}_{μ}^{H F P} + tr (θ_{σ} {ˆ η}_{σ}^{H F P}) - ψ (θ) + ln h & (24.16) = - \frac{¯ ı}{2} ln (2 π) - \frac{1}{2} ln (det (σ^{2})) - \frac{1}{2} [tr ({ˆ s}_{ε}^{2 H F P} {(σ^{2})}^{- 1}) + {({ˆ m}_{ε}^{H F P} - μ)}^{'} {(σ^{2})}^{- 1} ({ˆ m}_{ε}^{H F P} - μ)] . \end{matrix}

{ˆ m}_{ε}^{H i s t} \equiv {ˆ E}^{H i s t} {ε} = \frac{1}{¯ t} \sum_{t = 1}^{¯ t} ϵ_{t} .

\begin{matrix} {ˆ s}_{ε}^{2 H i s t} & \equiv {ˆ C v}^{H i s t} {ε} = \frac{1}{¯ t} \sum_{t = 1}^{¯ t} (ϵ_{t} - {ˆ m}_{ε}^{H i s t}) {(ϵ_{t} - {ˆ m}_{ε}^{H i s t})}_{t}^{'} & (24.4) = \frac{1}{¯ t} \sum_{t = 1}^{¯ t} ϵ_{t} ϵ_{t}^{'} - {ˆ m}_{ε}^{H i s t} {({ˆ m}_{ε}^{H i s t})}^{'} . \end{matrix}

I \equiv (ε_{1}, \dots, ε_{t}, \dots, ε_{¯ t}) \equiv ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} ε_{1, 1} & ε_{1, t} & ε_{1, ¯ t} \cdot & \cdot & \cdot ε_{i, 1} & \dots & ε_{i, t} & \dots & ε_{i, ¯ t} \cdot & \cdot & \cdot ε_{¯ ı, 1} & ε_{¯ ı, t} & ε_{¯ ı, ¯ t} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ .

ˆ M | μ, σ^{2} \sim N (μ, \frac{1}{¯ t} σ^{2}),

{ˆ Σ}^{2} | μ, σ^{2} \sim W i s h a r t (¯ t - 1, \frac{1}{¯ t} σ^{2}) .

{ˆ M}_{m e a n} | μ = e s_{m e a n} (I) | μ \equiv (\frac{1}{¯ t} \sum_{t = 1}^{¯ t} ε_{t}) | μ \sim N (μ, 1 ∕ ¯ t),

Θ \sim C o n j (¯ t ψ (\cdot), ¯ t η^{p r i}, \frac{ν^{p r i}}{¯ t}) = C o n j (ψ (\cdot), η^{p r i}, ν^{p r i}),

Σ^{2} \sim I n v W i s h a r t (ν, ψ^{2})

E {X} = μ, C v {X} = γ σ^{2},

C v {X} = \frac{ν}{ν - 2} σ^{2} .

E {Σ^{2}} = \frac{1}{ν - ¯ n - 1} ψ^{2},

C v {{[Σ^{2}]}_{m, n}^{2}, {[Σ^{2}]}_{p, q}^{2}} = \frac{2 {[ψ^{2}]}_{m, n}^{2} {[ψ^{2}]}_{p, q}^{2} + (ν - ¯ n - 1) ({[ψ^{2}]}_{m, p}^{2} {[ψ^{2}]}_{n, q}^{2} + {[ψ^{2}]}_{m, q}^{2} {[ψ^{2}]}_{n, p}^{2})}{(ν - ¯ n) {(ν - ¯ n - 1)}^{2} (ν - ¯ n - 3)} .

E {W^{2}} = ν σ^{2},

C v {W^{2}} = ν (I_{{¯ n}^{2}}^{2} + K_{¯ n, ¯ n}) (σ^{2} \otimes σ^{2}),

W i s h a r t (ν, σ^{2}) \Leftrightarrow G a m m a (ν, σ^{2}) \Leftrightarrow σ^{2} χ_{ν}^{2} .

M | σ^{2} \sim N (μ^{p r i}, \frac{1}{t^{p r i}} σ^{2}),

ϵ_{d, t} = \frac{x_{d, t} - x_{d, t - 1} - μ}{σ_{t - 1}},

\begin{matrix} {ˆ s}_{ε}^{2 H F P} & \equiv {ˆ C v}^{H F P} {ε} = \sum_{t = 1}^{¯ t} p_{t} (ϵ_{t} - {ˆ m}_{ε}^{H F P}) {(ϵ_{t} - {ˆ m}_{ε}^{H F P})}_{t}^{'} & (24.3) = \sum_{t = 1}^{¯ t} p_{t} ϵ_{t} ϵ_{t}^{'} - {ˆ m}_{ε}^{H F P} {({ˆ m}_{ε}^{H F P})}^{'} . \end{matrix}

c o r r (σ^{2}) \equiv D i a g (\frac{1_{¯ n \times 1}}{v o l (σ^{2})}) \times σ^{2} \times D i a g (\frac{1_{¯ n \times 1}}{v o l (σ^{2})}),

E {ε_{t}} = 0, S d {ε_{t}} = 1,

{[{(σ^{2})}^{- 1}]}_{m, n}^{- 1} = 0 \Leftrightarrow (m \sim n) \notin E .

(Y_{m} ⊥ ⊥ Y_{n}) | Y_{- {m, n}} \Leftrightarrow {[{(σ^{2})}^{- 1}]}_{m, n}^{- 1} = 0 .

\begin{matrix} f (i | θ) & = exp (¯ t (θ_{μ}^{'} {ˆ η}_{μ} + tr (θ_{σ} {ˆ η}_{σ}) - ψ (θ) + ln h)) & (24.40) = {(2 π)}^{- \frac{¯ ı ¯ t}{2}} det {(σ^{2})}^{- \frac{¯ t}{2}} exp (- \frac{¯ t}{2} [tr ({ˆ s}^{2} {(σ^{2})}^{- 1}) + {(ˆ m - μ)}^{'} {(σ^{2})}^{- 1} (ˆ m - μ)]), \end{matrix}

f^{p o s} (θ) \equiv f (θ | i) = \frac{f (i | θ) f^{p r i} (θ)}{\int f (i | ϑ) f^{p r i} (ϑ) d ϑ},

M \sim t (μ^{p r i}, \frac{ν^{p r i}}{t^{p r i} (ν^{p r i} - ¯ ı + 1)} σ^{2 p r i}, ν^{p r i} - ¯ ı + 1) .

μ^{p o s} = γ_{μ} μ^{p r i} + (1 - γ_{μ}) ˆ m,

ν^{p o s} = ν^{p r i} + ¯ t .

M | i \sim t (μ^{p o s}, \frac{ν^{p o s}}{t^{p o s} (ν^{p o s} - ¯ ı + 1)} σ^{2 p o s}, ν^{p o s} - ¯ ı + 1) .

{ˆ θ}^{m e a n} \equiv \int θ f^{p o s} (θ) d θ .

ˆ θ classical low confidence \leftarrow {ˆ θ}^{b a y e s} posterior high confidence \to θ^{p r i} prior .

σ^{2 p o s} = γ_{σ} σ^{2 p r i} + (1 - γ_{σ}) {ˆ s}^{2} + γ_{μ} (1 - γ_{σ}) (μ^{p r i} - ˆ m) {(μ^{p r i} - ˆ m)}^{'},

{ˆ θ}^{m p} \equiv argmax θ f^{p o s} (θ) .

Shrinkage: \frac{1}{¯ t} \sum_{t = 1}^{¯ t} τ (ϵ_{t}) (23.208) = ˆ η large dataset ¯ t \leftarrow η^{p o s} high confidence ν^{p r i} \to η^{p r i} .

ˆ m large dataset ¯ t \leftarrow μ^{c l_e q} = μ^{p o s} high confidence t^{p r i} \to μ^{p r i} .

{ˆ s}^{2} large dataset ¯ t \leftarrow σ^{2 c l_e q} = σ^{2 p o s} high confidence ν^{p r i} \to σ^{2 p r i} .

Y \sim N (μ, σ^{2}),

s_{θ}^{2} \equiv \int (θ - {ˆ θ}^{m e a n}) {(θ - {ˆ θ}^{m e a n})}^{'} f^{p o s} (θ) d θ .

s_{μ}^{2} = \frac{ν^{p o s}}{t^{p o s} (ν^{p o s} - 2)} σ^{2 p o s} .

s_{θ}^{2} high confidence \to 0 .

C v {{(Σ^{2})}^{- 1}} = \frac{1}{ν^{p r i}} (I_{{¯ ı}^{2}}^{2} + K_{¯ ı, ¯ ı}) ({(σ^{2 p r i})}^{- 1} \otimes {(σ^{2 p r i})}^{- 1}) .

K_{¯ ¯ ¯ m, ¯ n} \equiv \sum_{m, n} [δ_{¯ ¯ ¯ m \times 1}^{(m)} {(δ_{¯ n \times 1}^{(n)})}^{'}] \otimes [δ_{¯ n \times 1}^{(n)} {(δ_{¯ ¯ ¯ m \times 1}^{(m)})}^{'}] .

s_{θ}^{2} \equiv {(- \nabla_{θ, θ}^{2} ln f^{p o s} (θ) |_{θ = {ˆ θ}^{m p}}^{m p})}_{θ = {ˆ θ}^{m p}}^{- 1} .

s_{σ^{2}}^{2} = \frac{1}{ν^{p o s}} (I_{{¯ ı}^{2}}^{2} + K_{¯ ı, ¯ ı}) ({(σ^{2 p o s})}^{- 1} \otimes {(σ^{2 p o s})}^{- 1}),


Model (likelihood)	Normal	$ε_{t} \| μ, σ^{2} \sim N (μ, σ^{2})$
Prior	NIW	$⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} M \| σ^{2} \sim N (μ^{p r i}, \frac{1}{t^{p r i}} σ^{2}) {(Σ^{2})}^{- 1} \sim W i s h a r t (ν^{p r i}, \frac{1}{ν^{p r i}} {(σ^{2 p r i})}^{- 1}) \end{matrix}$
Posterior	NIW	$⎧ ⎨ ⎩ \begin{matrix} M \| σ^{2}, i \sim N (μ^{p o s}, \frac{1}{t^{p o s}} σ^{2}) {(Σ^{2})}^{- 1} \| i \sim W i s h a r t (ν^{p o s}, \frac{1}{ν^{p o s}} {(σ^{2 p o s})}^{- 1}) \end{matrix}$
Predictive	$t$	$ε_{t} \| (ϵ_{1}, \dots, ϵ_{¯ t}) \sim t (μ^{p o s}, \frac{t^{p o s} + 1}{t^{p o s}} σ^{2 p o s}, ν^{p o s}), t > ¯ t$

24.5 Bayesian

24.5.1 Model and sample estimators

24.5.2 Normal-inverse-Wishart prior distribution

24.5.3 Normal-inverse-Wishart posterior distribution

24.5.4 Student t predictive distribution

24.5.5 Classical equivalent, uncertainty and shrinkage

Bibliography

External Links

Discussions