Lab | Probabilistic regression

18.3 Probabilistic regression

Key points

Probabilistic regression is a supervised learning approach, where the goal is to approximate the true conditional distribution (18.147) of a continuous output (18.146) given an input.
In practice, discriminative regression models (18.159) are obtained from families of continuous distributions (18.153) parametrized via feature engineering (18.157) and are optimized by minimizing the cross-entropy (18.161).
Discriminative linear regression (18.165) admits a generative embedding (18.179) and is a special instance of generalized linear models (18.181).

In the below we discuss the issue of probabilistic prediction (17.74) for supervised regression. In this context

the output takes values in a continuum (17.6)
$X \in R^{¯ n};$ (18.146)
as we shall see, probabilistic statements (17.74) aim at guessing the conditional distribution of the output (17.3) given the input (17.4)
$¯ f (x | z) \approx f (x | z) .$ (18.147)

Alert 18.4. Probabilistic regression can be one of two classes: i) discriminative (17.82) or ii) generative (17.87), see Section 17.2.2. Here we focus mainly on the discriminative models, which are the most popular and computationally parsimonious, see e.g. Example 17.10.

Among the analytically tractable discriminative models, a special role is played by generalized linear models (Section 18.3.6) which include normal linear regression (18.181). From these follow immediate generalizations, and the more elaborate enhancements through the feature engineering techniques discussed in Chapter 7.

However, for most applications in risk and portfolio management, analytical discriminative models (17.82) are a luxury, as typically one needs to walk through all the steps of the “Checklist” (Parts IX-X-XI) to obtain a numerical predictive joint distribution of explanatory variables and target variables (17.1).

Example 18.16. In Example 17.14 we follow the steps of the “Checklist” to compute the predictive distribution for a call option return $X \equiv R_{t_{n o w} \to t_{h o r}}^{c a l l}$ (17.55) via kernel smoothing (23.8) on the S&P 500 index return $Z \equiv R_{t_{n o w} \to t_{h o r}}^{S & P}$ (17.56).

18.3.1 Error

In the supervised context, consider the probabilistic prediction problem (17.74), where the goal is to obtain a probabilistic prediction $¯ f (x | z)$ (19.179) for the conditional distribution $f (x | z)$ of the output $X$ (19.178).

To measure the goodness of the probabilistic prediction we use the logarithmic scoring rule (17.95), which we report here

s . r u l e (x, ¯ f) \equiv ln ¯ f (x) .

(18.148)

Then, following a discriminative approach (17.82), we evaluate the goodness of the prediction $¯ f (x) \equiv ¯ f (x | z)$ in terms of the ensuing cross-entropy (17.78), which here reads as in (17.83)

E^{f} {- ln ¯ f (X | Z)} = - \int ln ¯ f (x | z) f (x, z) d x d z,

(18.149)

where $f (x, z)$ denotes the true, unknown joint distribution of the inputs (17.4) and outputs (17.3). See more in Table 18.3.

18.3.2 Theoretical optimum

Let us define the theoretical optimum, that we would achieve by minimizing the error (18.149) if we knew the joint distribution $f_{X, Z}$ and if we had infinite computational power.

In the context of supervised point prediction (18.15), we consider arbitrary functions $χ$ that with any input $z$ (17.4) associate a point guess for the output $x$ (18.13). Similar, in the context of supervised probabilistic prediction, we consider arbitrary functions $¯ f (\cdot | \cdot)$ which associate any input $z$ (17.4) to a probabilistic guess (18.147), and hence a positive function which integrates to one

z \mapsto ¯ f (\cdot | z) .

(18.150)

Accordingly, let us perform the error minimization (18.149) across all the possible functions $¯ f (\cdot | \cdot)$ (18.150)

f^{*} (\cdot | \cdot) \equiv argmin ¯ f (\cdot | \cdot) E {- ln ¯ f (X | Z)},

(18.151)

and where we dropped the dependence of the expectation on the true joint distribution $f (x, z)$ for ease of notation.

The minimization (18.151) trivially yields as solution the true, unknown, conditional distribution of the output (17.3) given the input (17.4) 70.10

f^{*} (x | z) = f (x | z) .

(18.152)

18.3.3 Target parameters

In order to obtain reasonable, interpretable, analytically tractable optimizations and solutions, we proceed in two steps.

First, we choose a parametric map onto the space of distributions as in (18.150)

γ \in G \mapsto ¯ f (\cdot | γ) \in F,

(18.153)

where $γ$ is a vector of target parameters and the parametrization ensures that, while the parameters $γ$ span the set $G$ , the function $¯ f (\cdot | γ)$ remains a pdf (2.29)

F \equiv {f : R^{¯ n} \to R : f \geq 0, \int f (x) d x = 1} .

(18.154)

There exist several ways to specify a parametric map as in (18.153). In principle we can use any of the parametric families of (absolutely) continuous distributions discussed in Chapter 49. In practice, the specifications underlying the most popular approaches are summarized in the below.

Then, we postulate a model for the target parameters in (18.153) as a function of the inputs (17.4)

γ \equiv γ (z) \in G .

(18.155)

At this stage it is important to ensure that the output $γ (z)$ remains in the parameter domain $G$ in (18.153). This yields a viable model for the conditional distribution (18.152)-(18.154)

¯ f (x | z) \equiv ¯ f (x | γ (z)) .

(18.156)

18.3.4 Optimization in practice

Similar to least squares regression (Section 18.1.3), to build a probabilistic predictor (18.156) we proceed in two steps: first, we model suitably the dependence of the continuous, target parameters $γ_{ϑ} (z)$ (18.155) on the inputs; second, we determine the optimal predictor $¯ f (x | γ_{θ} (z))$ (18.156) via error minimization.

To model the target parameters $γ$ (18.155) we can leverage feature engineering techniques ( Chapter 7), applying the same results as for least squares regression (Section 18.1.3).

More precisely, at this stage, we can apply feature engineering to express the target parameters (18.155) as parametric functions $γ_{ϑ} (z)$ of the inputs as in Table 7.2. For example, we can parametrize the variables in several ways, e.g. via fixed basis, such as polynomials (7.25), adaptive basis, such as trees (7.34), or neural networks (7.47)

γ_{ϑ} (z) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} basis-linear & b ϕ (z) & ϑ = b trees & β_{Δ} ϕ_{Δ}^{1 h o t} (z) & ϑ = Δ neural networks & n n_{ϑ} (z) & ϑ = {b^{(1)}, \dots, b^{(¯ l)}} gradient boosting & γ_{ϑ^{(0)}}^{(0)} (z) - \sum_{l = 1}^{¯ l} α^{(l)} β_{Δ^{(l)}}^{(l)} ϕ_{Δ^{(l)}}^{1 h o t} (z) & ϑ = {ϑ^{(0)}, Δ^{(1)}, \dots, Δ^{(¯ l)}} \end{matrix}

(18.157)

where the parameters $ϑ$ belongs to a suitable domain $O$ such that ensures that the output $γ_{ϑ} (z)$ belongs to the domain $G$ in (18.153)

ϑ \in O \Rightarrow γ_{ϑ} (z) \in G .

(18.158)

By sequencing the parametric map (18.153) and the model of the target parameters (18.157), we obtain a parametric family of probabilistic predictors (18.156)

f_{ϑ} (x | z) \equiv ¯ f (x | γ_{ϑ} (z)) .

(18.159)

Then, similar to the theoretical counterpart (18.151), we evaluate the goodness of a predictor as in (18.159) through the cross-entropy (18.149)

e r r o r (ϑ) \equiv E {- ln ¯ f (X | γ_{ϑ} (Z))} .

(18.160)

Finally, the minimization (17.84) of the error (18.160) yields the discriminative regression

θ \equiv argmin ϑ \in O E {- ln ¯ f (X | γ_{ϑ} (Z))} .

(18.161)

Note how (18.159)-(18.160)-(18.161) are consistent with the basic principles of decision theory (Table 18.4).

18.3.5 Linear regression

Linear regression is one of the most popular model for discriminative regression. It can be specified either via i) discriminative approach or ii) generative approach, as we proceed to discuss.

Discriminative model

Let us consider a probabilistic (discriminative) model (18.159) where:

1.

the parametric map (18.153) is defined through the multivariate normal family (49.1)

¯ f (x | γ) \equiv f^{N} (x | μ, σ^{2}),

(18.162)

and hence the target parameters $γ$ in (18.153) are the location and dispersion ${μ, σ^{2}}$ ;

2.

the first set of target parameters

μ

are modeled as linear functions of fixed features

ϕ (z) \equiv {(ϕ_{1} (z), \dots, ϕ_{¯ l} (z))}_{1}^{'}

(7.21), such as affine (18.35) or polynomial (7.24)

μ (z) \equiv b ϕ (z),

(18.163)

where $b$ is an $¯ n \times ¯ l$ matrix; and the second set of target parameters $σ^{2}$ are modeled as a constant function

σ^{2} (z) \equiv s^{2} ≽ 0,

(18.164)

where $s^{2}$ is a symmetric (43.139) and positive (semi)definite (43.150) $¯ n \times ¯ n$ matrix.

This way, by sequencing the parametric map (18.162) and the models of the target parameters (18.163)-(18.164) as in the general discriminative modeling (18.159), we obtain the discriminative linear regression model

f_{b, s^{2}}^{2} (x | z) = f^{N} (x | b ϕ (z), s^{2}) .

(18.165)

By inputting the probabilistic model (18.165) in the cross-entropy (18.149) we obtain the error (18.160) 70.11

e r r o r (b, s^{2}) = E {{(X - b ϕ (Z))}^{'} {(s^{2})}^{- 1} (X - b ϕ (Z))} + ln det (s^{2}) .

(18.166)

Then the optimal probabilistic predictor solves (18.161)

(β, σ^{2}) \equiv argmin b, s^{2} ≽ 0 e r r o r (b, s^{2}),

(18.167)

which yield the optimal loadings 70.11

β = E {X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1},

(18.168)

and optimal dispersion parameters 70.11

σ^{2} = E {(X - β ϕ (Z)) {(X - β ϕ (Z))}^{'}} .

(18.169)

Refer to Section 25.2.2 for the empirical implementation.

Example 18.17. In Example 25.2 we fit the normal model (18.165) for the daily linear returns of $¯ n = 392$ stocks in the S&P 500 $X$ and $¯ k = 10$ sector indices $Z$ via maximum likelihood approach to perform probabilistic prediction in the stock market.

Similar to least squares regression (Section 18.1.4), the simplest instance of the discriminative linear regression (18.165) is obtained by setting the target parameters (18.163) simply as linear functions of affine features $ϕ^{a ff} (z)$ (18.35)

μ (z) \equiv b^{a ff} ϕ^{a ff} (z) = a + b z,

(18.170)

where $b^{a ff}$ is an $¯ n \times (¯ k + 1)$ matrix arranged as in (18.36).

In this special situation the linear regression prediction (18.165) becomes

f_{β^{a ff}, σ^{2}}^{a ff} (x | z) = f^{N} (x | β^{a ff} ϕ^{a ff} (z), σ^{2}),

(18.171)

where the optimal loadings matrix (18.168) reads 70.13

β^{a ff} = E {X ϕ^{a ff} {(Z)}^{'}} {(E {ϕ^{a ff} (Z) ϕ^{a ff} {(Z)}^{'}})}^{- 1} = (\begin{matrix} α_{X ∥ Z} & β_{X ∥ Z} \end{matrix}),

(18.172)

where $α_{X ∥ Z}$ are exactly the standard regression shifts (14.86) and $β_{X ∥ Z}$ are exactly the standard regression loadings (14.84) between the output $X$ (17.3) and the input $Z$ (17.4); and the dispersion matrix (18.169) reads as the regression residual covariance (14.100) 70.13

σ^{2} = C v {X} - C v {X, Z} {(C v {Z})}^{- 1} C v {Z, X} .

(18.173)

Note how in this special case the point prediction (17.49) stemming from (18.171) by using the expectation (17.100) is the same as the least-squares counterpart (18.34)-(18.37).

Similar results follow if the target parameters $μ (z)$ are modeled linearly (18.163) in features $ϕ (z)$ which include a constant $1$ and other non-constant functions $˜ ϕ (z)$ of the original inputs $z \equiv {(z_{1}, \dots, z_{¯ k})}_{1}^{'}$ (17.4)

ϕ (z) \equiv (\begin{matrix} 1 ˜ ϕ (z) \end{matrix}),

(18.174)

such as polynomials (7.23). Indeed, generalizing the affine case (18.172)-(18.173), here the optimal loadings matrix (18.168) reads as in the least-squares counterpart 70.17

β = E {X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1} = (\begin{matrix} α_{X ∥ ˜ ϕ (Z)} & β_{X ∥ ˜ ϕ (Z)} \end{matrix}),

(18.175)

where $α_{X ∥ ˜ ϕ (Z)}$ are the standard regression shifts (14.86) and $β_{X ∥ ˜ ϕ (Z)}$ are the standard regression loadings (14.84) between the output $X$ (17.3) and the non-constant features $˜ ϕ (Z)$ ; and the dispersion matrix (18.169) reads 70.17

σ^{2} = C v {X} - C v {X, ˜ ϕ (Z)} {(C v {˜ ϕ (Z)})}^{- 1} C v {˜ ϕ (Z), X} .

(18.176)

However bear in mind that here all the predictions are probabilistic and hence more informative than the (point) least-squares ones (18.31)-(18.41)-(18.42).

Generative embedding

A popular generative model (17.87) for probabilistic regression (18.147) is the jointly normal model (49.20)

f_{m_{{X, Z}}, s_{{X, Z}}^{2}} (x, z) \equiv f^{N} (x, z | (\begin{matrix} m_{X} m_{Z} \end{matrix})      m_{{X, Z}}, (\begin{matrix} s_{X}^{2} & s_{X, Z} s_{X, Z}^{'} & s_{Z}^{2} \end{matrix})      s_{{X, Z}}^{2}),

(18.177)

where $m_{{X, Z}}$ is a $(¯ n + ¯ k) \times 1$ vector and $s_{{X, Z}}^{2}$ is a a symmetric (43.139) and positive (semi)definite (43.150) $(¯ n + ¯ k) \times (¯ n + ¯ k)$ matrix. This is a generative embedding (17.93) for the linear regression (18.165) with affine features (18.35) 70.14

ϕ^{a ff} (z) \equiv (\begin{matrix} 1 z \end{matrix}) .

(18.178)

Alert 18.5. Furthermore, in this special case the conditional prediction (17.89) obtained by optimizing (17.88) the generative normal model (18.177) is the same as the optimal discriminative counterpart (18.171), or 70.14

f_{μ_{{X, Z}}, σ_{{X, Z}}^{2}} (x | z) = f_{β^{a ff}, σ^{2}}^{a ff} (x | z) .

(18.179)

Example 18.18. In Example 14.14 we consider a jointly normal model as in (18.177) for a univariate output and input $(X, Z)$ (14.79) and display the probabilistic prediction (17.1), or conditional distribution, which is still normal

¯ f (x | z) = f^{N} (x | μ_{X} + \frac{σ_{X} ϱ_{X, Z}}{σ_{Z}} (z - μ_{Z}), σ_{X}^{2} (1 - ϱ_{X, Z}^{2})),

(18.180)

as follows from the optimal parameters (18.172)-(18.173). Refer also to Figure 14.5 for a visualization.

Exponential family embedding

When the target parameters $σ^{2}$ (18.164) are exogenously specified and kept fixed (say $σ^{2} = I_{¯ n}$ ), the linear regression model (18.165), which here reads

f_{b} (x | z) = f^{N} (x | b ϕ (z), σ^{2}),

(18.181)

can be expressed in the form of an exponential family distribution $E x p (θ, τ (\cdot), h (\cdot))$ (49.395) as follows 70.19

f_{b} (x | z) = h (x) e^{{[σ^{- 1} b ϕ (z)]}^{'} τ (x) - ψ (σ^{- 1} b ϕ (z))},

(18.182)

where $σ = σ^{'}$ denotes the Riccati root (43.539) of $σ^{2}$ and

- the canonical parameters $θ \equiv {(θ_{1}, \dots, θ_{¯ n})}_{1}^{'}$ (49.404) are the target parameters in (18.153), which by construction (18.163) are linear functions of the features

θ (z) \equiv σ^{- 1} b ϕ (z);

(18.183)

- the sufficient statistics $τ (\cdot)$ are $τ (x) \equiv σ^{- 1} x$ (49.407);

- the base measure $h (\cdot)$ is the zero-mean normal pdf $h (x) \equiv f^{N} (x | 0, σ^{2})$ (49.4);

- the log-partition function (49.396) is quadratic (49.409)

ψ (θ) = \frac{1}{2} θ^{'} θ = \frac{1}{2} \sum_{n = 1}^{¯ n} θ_{n}^{2} .

(18.184)

In particular, since in this case the inverse link function $g^{- 1} (\cdot)$ (49.402) is the identity function (49.411), the model (18.182) can be equivalently specified in terms of expectation parameters $η$ (49.410)

η (z) \equiv σ^{- 1} b ϕ (z) = θ (z) .

(18.185)

18.3.6 Generalized linear models

We can relax the restrictive normal model (18.162) as in the standard linear regression (18.165).

More precisely, let us consider a probabilistic (discriminative) model (18.159) where:

1.

the parametric map (18.153) is defined through the exponential family distribution (49.395)

¯ f (x | θ) \equiv h (x) e^{θ^{'} τ (x) - ψ (θ)},

(18.186)

where $τ (x)$ and $h (x)$ are fixed sufficient statistics and base measure respectively, and hence the target parameters $γ$ in (18.153) are the canonical coordinates $θ$ ;

2.

the target parameters

θ

are modeled as linear functions of fixed features

ϕ (z) \equiv {(ϕ_{1} (z), \dots, ϕ_{¯ l} (z))}_{1}^{'}

(7.21), such as affine (18.35) or polynomial (7.24)

θ (z) \equiv b ϕ (z) .

(18.187)

This way, by sequencing the parametric map (18.186) and the model for the target parameters (18.187) as in the general discriminative modeling (18.159), we obtain the generalized linear models (GLM) [W]

f_{b} (x | z) = h (x) e^{{(b ϕ (z))}^{'} τ (x) - ψ (b ϕ (z))} .

(18.188)

Since the canonical parameters $θ$ are in one-to-one correspondence (49.402) with the expectation parameters $η$ (49.400), a generalized linear model (18.188) can be equivalently formulated by modeling the expectation parameters $η$ as linear functions of fixed features as in (18.187)

η (z) \equiv g^{- 1} (b ϕ (z)) .

(18.189)

where $g$ denotes the link function (49.402).

By inputting the probabilistic model (18.188) in the cross-entropy (18.149) we obtain the error (18.160)

e r r o r (b) = E {ψ (b ϕ (z)) - {(b ϕ (Z))}^{'} τ (X)},

(18.190)

where $ψ (\cdot)$ denotes the log-partition function (49.396).

Then the optimum predictor solves (18.161)

β \equiv argmin b e r r o r (b),

(18.191)

which is then addressed numerically in most of the cases [W].

Generalized linear models (18.188) are very flexible and cover a wide range of models [W], such as:

the linear regression model with fixed covariance $σ^{2}$ (18.181) 70.19 ;
the logistic regression model (19.216) when adapted to classification problems (17.9), see Section 19.3.6.

Finally, we can further extend GLMs (18.188) by including other shapes for the canonical coordinates (18.187), in that we can consider any of the parametrizations discussed in Chapter 7, see Table 7.2

θ (z) \equiv ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ \begin{matrix} trees & β_{Δ} ϕ_{Δ}^{1 h o t} (z) neural networks & n n_{ϑ} (z) gradient boosting & θ_{ϑ^{(0)}}^{(0)} (z) - \sum_{l = 1}^{¯ l} α^{(l)} β_{Δ^{(l)}}^{(l)} ϕ_{Δ^{(l)}}^{1 h o t} (z) \end{matrix}

(18.192)

18.3.7 Alternative generalizations

Similar to GLMs (18.188) we can easily extend the standard linear regression (18.165) by exploring other families of distributions.

For example, let us consider a probabilistic (discriminative) model (18.159) where:

1.

the parametric map (18.153) is defined through the multivariate Student

t

family (49.141)

¯ f (x | γ) \equiv f^{t} (x | μ, σ^{2}, ν),

(18.193)

where the degrees of freedom $ν$ are fixed, and hence the target parameters $γ$ in (18.153) are the location and dispersion ${μ, σ^{2}}$ ;

2.

the first set of target parameters

μ

are modeled as linear functions of fixed features

ϕ (z) \equiv {(ϕ_{1} (z), \dots, ϕ_{¯ l} (z))}_{1}^{'}

(7.21), such as affine (18.35) or polynomial (7.24)

μ (z) \equiv b ϕ (z),

(18.194)

where $b$ is an $¯ n \times ¯ l$ matrix; and the second set of target parameters $σ^{2}$ are modeled as a constant function

σ^{2} (z) \equiv s^{2} ≽ 0,

(18.195)

where $s^{2}$ is a symmetric (43.139) and positive (semi)definite (43.150) $¯ n \times ¯ n$ matrix.

This way, by sequencing the parametric map (18.193) and the models for the target parameters (18.194)-(18.195) as in the general discriminative modeling (18.159), we obtain another discriminative regression model (17.82)

f_{b, s^{2}}^{2} (x | z) = f^{t} (x | b ϕ (z), s^{2}, ν),

(18.196)

which is more suitable in applications, because it better fits the empirically observed fat tails in financial markets, see more in Section 25.2.3.

By inputting the probabilistic model (18.196) in the cross-entropy (18.149) we obtain the error (18.160) 70.3

e r r o r (b, s^{2}) = E {(ν + ¯ n) ln (1 + \frac{{(X - b ϕ (Z))}^{'} {(s^{2})}^{- 1} (X - b ϕ (Z))}{ν})} + ln det (s^{2}) .

(18.197)

Then the optimal probabilistic predictor solves (18.161)

(β, σ^{2}) \equiv argmin b, s^{2} ≽ 0 e r r o r (b, s^{2}),

(18.198)

which yield the optimal loadings 70.3

β = E {\frac{ν + ¯ n}{ν + {(X - β ϕ (Z))}^{'} {(σ^{2})}^{- 1} (X - β ϕ (Z))} X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1}

(18.199)

and optimal dispersion parameters 70.3

σ^{2} = E {\frac{ν + ¯ n}{ν + {(X - β ϕ (Z))}^{'} {(σ^{2})}^{- 1} (X - β ϕ (Z))} (X - β ϕ (Z)) {(X - β ϕ (Z))}^{'}} .

(18.200)

We remark that (18.199)-(18.200) provide the solution in terms of implicit equations. In practice, we must rely on recursive algorithms in order to approximate them numerically, see more in Section 25.2.3.

Furthermore, we recover the normal linear regression solutions (18.168)-(18.169) in the limit $ν \to + \infty$ 70.3 .

Alert 18.6. Notice that the Student $t$ regression model (18.196):
- in contrast to the linear regression (18.181), is not a GLM (18.188), because the Student $t$ distribution (49.141) does not belong to the exponential family (49.394), see [W];
- in contrast to the linear regression (18.177), does not admit a generative embedding (17.93) with a joint Student $t$ model (49.155), most notably because the dispersion parameter $s^{2}$ (18.200), representing the conditional covariance (3.87), does not depend on the inputs $z$ as in the generative counterpart (49.156)-(49.158).

Example 18.19. In Example 25.3 we fit the general Student $t$ model (18.196) with $ν = 4$ degrees of freedom for the daily linear returns of $¯ n = 392$ stocks in the S&P 500 $X$ and $¯ k = 10$ sector indices $Z$ via maximum likelihood approach to perform probabilistic prediction in the stock market.

¯ f (x | z) \approx f (x | z) .

X \in R^{¯ n};

f_{ϑ} (x | z) \equiv ¯ f (x | γ_{ϑ} (z)) .

γ \in G \mapsto ¯ f (\cdot | γ) \in F,

γ_{ϑ} (z) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} basis-linear & b ϕ (z) & ϑ = b trees & β_{Δ} ϕ_{Δ}^{1 h o t} (z) & ϑ = Δ neural networks & n n_{ϑ} (z) & ϑ = {b^{(1)}, \dots, b^{(¯ l)}} gradient boosting & γ_{ϑ^{(0)}}^{(0)} (z) - \sum_{l = 1}^{¯ l} α^{(l)} β_{Δ^{(l)}}^{(l)} ϕ_{Δ^{(l)}}^{1 h o t} (z) & ϑ = {ϑ^{(0)}, Δ^{(1)}, \dots, Δ^{(¯ l)}} \end{matrix}

θ \equiv argmin ϑ \in O E {- ln ¯ f (X | γ_{ϑ} (Z))} .

f_{b, s^{2}}^{2} (x | z) = f^{N} (x | b ϕ (z), s^{2}) .

f_{μ_{{X, Z}}, σ_{{X, Z}}^{2}} (x | z) = f_{β^{a ff}, σ^{2}}^{a ff} (x | z) .

f_{b} (x | z) = f^{N} (x | b ϕ (z), σ^{2}),

¯ f (x) \approx f (x) .

X \in R^{¯ n},

X \equiv ⎛ ⎜ ⎜ ⎝ \begin{matrix} X_{1} ⋮ X_{¯ n} \end{matrix} ⎞ ⎟ ⎟ ⎠,

Z \equiv ⎛ ⎜ ⎜ ⎝ \begin{matrix} Z_{1} ⋮ Z_{¯ k} \end{matrix} ⎞ ⎟ ⎟ ⎠,

¯ f (x) \equiv f_{ϑ} (x | z),

¯ f (x, z) \equiv f_{c} (x, z),

X \equiv R_{t_{n o w} \to t_{h o r}}^{c a l l} = \frac{V_{t_{h o r}}^{c a l l} (k^{s t r k}, t_{e n d})}{v_{t_{n o w}}^{c a l l} (k^{s t r k}, t_{e n d})} - 1;

p_{t} | z^{*} \equiv p e^{- {(| z_{t} - z^{*} | ∕ h)}_{t}^{γ}},

Z \equiv R_{t_{n o w} \to t_{h o r}}^{S & P} = \frac{V_{t_{h o r}}^{S & P}}{v_{t_{n o w}}^{S & P}} - 1 .

¯ p (x | z) \approx p (x | z) .

X \in {1, \dots, ¯ c};

s . r u l e (x, ¯ f) \equiv ln ¯ f (x) .

H (f | | ¯ f) \equiv E^{f} {- ln ¯ f (X)} .

e r r o r (ϑ, f) \equiv E^{f} {- ln f_{ϑ} (X | Z)} = - \int f (x, z) ln f_{ϑ} (x | z) d x d z,

E^{f} {- ln ¯ f (X | Z)} = - \int ln ¯ f (x | z) f (x, z) d x d z,

χ : z \mapsto ¯ ¯ ¯ x \equiv χ (z),

x \in R^{¯ n} .

z \mapsto ¯ f (\cdot | z) .

f^{*} (\cdot | \cdot) \equiv argmin ¯ f (\cdot | \cdot) E {- ln ¯ f (X | Z)},

f_{X} (x) \equiv \frac{\partial^{¯ n} F_{X} (x)}{\partial x_{1} \dots \partial x_{¯ n}} .

X \sim N (μ, σ^{2}),

f_{μ, σ^{2}, ν}^{t} (x) = \frac{Γ (\frac{ν + ¯ n}{2})}{Γ (\frac{ν}{2}) {(ν π)}^{\frac{¯ n}{2}}} det {(σ^{2})}^{- \frac{1}{2}} {(1 + \frac{{(x - μ)}^{'} {(σ^{2})}^{- 1} (x - μ)}{ν})}^{- \frac{ν + ¯ n}{2}} .

f^{*} (x | z) = f (x | z) .

F \equiv {f : R^{¯ n} \to R : f \geq 0, \int f (x) d x = 1} .

¯ f (x | z) \equiv ¯ f (x | γ (z)) .

γ \equiv γ (z) \in G .

p o l y_{b} (z) = b ϕ^{p o l y} (z),

t r e e_{Δ} (z) \equiv \sum_{l = 1}^{¯ l} β_{Δ}^{(l)} \times 1_{Δ z^{(l)}}^{(l)} (z) = β_{Δ} ϕ_{Δ}^{1 h o t} (z) .

n n_{ϑ} (z) \equiv (n e u r_{b^{(¯ l)}}^{(¯ l)} \circ \dots \circ n e u r_{b^{(1)}}^{(1)}) (z),

θ \equiv argmin ϑ e r r o r (ϑ, f) .

e r r o r (ϑ) \equiv E {- ln ¯ f (X | γ_{ϑ} (Z))} .

z \mapsto ϕ \equiv ϕ (z) \equiv ⎛ ⎜ ⎜ ⎝ \begin{matrix} ϕ_{1} (z) ⋮ ϕ_{¯ l} (z) \end{matrix} ⎞ ⎟ ⎟ ⎠,

ϕ^{a ff} (z) \equiv (\begin{matrix} 1 z \end{matrix});

ϕ^{p o l y} (z) \equiv ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ \begin{matrix} 1 \cdot z_{k} \cdot z_{l} z_{k} \cdot z_{l} z_{k} z_{m} \cdot \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ .

s = s^{'},

s^{2} ≽ 0 : x^{'} s^{2} x \geq 0 .

¯ f (x | γ) \equiv f^{N} (x | μ, σ^{2}),

μ (z) \equiv b ϕ (z),

σ^{2} (z) \equiv s^{2} ≽ 0,

b^{a ff} \equiv (\begin{matrix} a & b \end{matrix}) .

β = E {X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1},

⎛ ⎜ ⎝ \begin{matrix} α_{1} \cdot α_{¯ n} \end{matrix} ⎞ ⎟ ⎠ = ⎛ ⎜ ⎝ \begin{matrix} E {X_{1}} \cdot E {X_{¯ n}} \end{matrix} ⎞ ⎟ ⎠ - ⎛ ⎜ ⎝ \begin{matrix} C v {X_{1}, Z_{1}} & \dots & C v {X_{1}, Z_{¯ k}} \cdot & \cdot C v {X_{¯ n}, Z_{1}} & \dots & C v {X_{¯ n}, Z_{¯ k}} \end{matrix} ⎞ ⎟ ⎠ \times {⎛ ⎜ ⎝ \begin{matrix} V {Z_{1}} & \dots & C v {Z_{1}, Z_{¯ k}} \cdot & \cdot C v {Z_{¯ k}, Z_{1}} & \dots & V {Z_{¯ k}} \end{matrix} ⎞ ⎟ ⎠}^{- 1} \times ⎛ ⎜ ⎝ \begin{matrix} E {Z_{1}} \cdot E {Z_{¯ k}} \end{matrix} ⎞ ⎟ ⎠

⎛ ⎜ ⎜ ⎝ \begin{matrix} β_{1, 1} & \dots & β_{1, ¯ k} \cdot & \cdot β_{¯ n, 1} & \dots & β_{¯ n, ¯ k} \end{matrix} ⎞ ⎟ ⎟ ⎠ = ⎛ ⎜ ⎝ \begin{matrix} C v {X_{1}, Z_{1}} & \dots & C v {X_{1}, Z_{¯ k}} \cdot & \cdot C v {X_{¯ n}, Z_{1}} & \dots & C v {X_{¯ n}, Z_{¯ k}} \end{matrix} ⎞ ⎟ ⎠ \times {⎛ ⎜ ⎝ \begin{matrix} V {Z_{1}} & \dots & C v {Z_{1}, Z_{¯ k}} \cdot & \cdot C v {Z_{¯ k}, Z_{1}} & \dots & V {Z_{¯ k}} \end{matrix} ⎞ ⎟ ⎠}^{- 1}

σ^{2} = E {(X - β ϕ (Z)) {(X - β ϕ (Z))}^{'}} .

C v {˚ ε} = C v {X} - C v {X, Z} C v {Z}^{- 1} C v {Z, X} .

¯ ¯ ¯ x \approx x .

f_{β^{a ff}, σ^{2}}^{a ff} (x | z) = f^{N} (x | β^{a ff} ϕ^{a ff} (z), σ^{2}),

¯ f (x) \Rightarrow ¯ ¯ ¯ x \equiv E^{¯ f} {X} .

χ_{b^{a ff}}^{a ff} (z) \equiv b^{a ff} ϕ^{a ff} (z) = a + b z,

β^{a ff} = E {X ϕ^{a ff} {(Z)}^{'}} {(E {ϕ^{a ff} (Z) ϕ^{a ff} {(Z)}^{'}})}^{- 1} = (\begin{matrix} α_{X ∥ Z} & β_{X ∥ Z} \end{matrix}),

p o l y_{b} (z) \equiv b^{(0)} + \sum_{k = 1}^{¯ k} b_{k}^{(1)} z_{k} + \sum_{k_{1}, k_{2} = 1}^{¯ k} b_{k_{1}, k_{2}}^{(2)} z_{k_{1}} z_{k_{2}} + \dots + \sum_{k_{1}, \dots, k_{q} = 1}^{¯ k} b_{k_{1}, \dots, k_{q}}^{(q)} z_{k_{1}} \dots z_{k_{q}} + \dots .

β^{a ff} = E {X ϕ^{a ff} {(Z)}^{'}} {(E {ϕ^{a ff} (Z) ϕ^{a ff} {(Z)}^{'}})}^{- 1} = (\begin{matrix} α_{X ∥ Z} & β_{X ∥ Z} \end{matrix}),

σ^{2} = C v {X} - C v {X, Z} {(C v {Z})}^{- 1} C v {Z, X} .

χ_{b} (z) \equiv b ϕ (z) .

ϕ (z) \equiv (\begin{matrix} 1 ˜ ϕ (z) \end{matrix}),

β = E {X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1} = (\begin{matrix} α_{X ∥ ˜ ϕ (Z)} & β_{X ∥ ˜ ϕ (Z)} \end{matrix}),

(\begin{matrix} X Z \end{matrix}) \sim N ((\begin{matrix} μ_{X} μ_{Z} \end{matrix}), (\begin{matrix} σ_{X}^{2} & σ_{X, Z} σ_{X, Z}^{'} & σ_{Z}^{2} \end{matrix})),

f_{ϑ} (x | z) (17.89) \Leftarrow f_{c} (x, z) .

¯ f (x | z) \equiv f_{γ} (x | z) = \frac{f_{γ} (x, z)}{\int f_{γ} (x, z) d x} .

γ \equiv argmin c E^{f} {- ln f_{c} (X, Z)},

f_{m_{{X, Z}}, s_{{X, Z}}^{2}} (x, z) \equiv f^{N} (x, z | (\begin{matrix} m_{X} m_{Z} \end{matrix})      m_{{X, Z}}, (\begin{matrix} s_{X}^{2} & s_{X, Z} s_{X, Z}^{'} & s_{Z}^{2} \end{matrix})      s_{{X, Z}}^{2}),

(\begin{matrix} X Z \end{matrix}) \sim N ((\begin{matrix} μ_{X} μ_{Z} \end{matrix}), (\begin{matrix} σ_{X}^{2} & ϱ_{X, Z} σ_{X} σ_{Z} ϱ_{X, Z} σ_{X} σ_{Z} & σ_{Z}^{2} \end{matrix})) .

f_{θ} (x) = h (x) e^{θ^{'} τ (x) - ψ (θ)},

s^{2} = {(s_{R i c c})}_{R i c c}^{2} .

θ \equiv σ^{- 1} μ .

τ (x) \equiv σ^{- 1} x .

f_{μ, σ^{2}}^{N} (x) = {(2 π)}^{- \frac{¯ n}{2}} det {(σ^{2})}^{- \frac{1}{2}} e^{- \frac{1}{2} {(x - μ)}^{'} {(σ^{2})}^{- 1} (x - μ)} .

ψ (θ) \equiv ln (\int_{R^{¯ n}}^{¯ n} h (x) e^{θ^{'} τ (x)} d x) .

ψ (θ) = \frac{1}{2} θ^{'} θ .

θ = g (η) \equiv {(\nabla_{θ} ψ)}_{θ}^{- 1} (η) .

g (η) = η .

f_{b} (x | z) = h (x) e^{{[σ^{- 1} b ϕ (z)]}^{'} τ (x) - ψ (σ^{- 1} b ϕ (z))},

η = θ;

¯ f (x | θ) \equiv h (x) e^{θ^{'} τ (x) - ψ (θ)},

θ (z) \equiv b ϕ (z) .

η \equiv E_{θ} {τ (X)} = \nabla_{θ} ψ (θ);

f_{b} (x | z) = h (x) e^{{(b ϕ (z))}^{'} τ (x) - ψ (b ϕ (z))} .

p_{b} (\cdot | z) \equiv s o f t m a x ((\begin{matrix} b ϕ (z) 0 \end{matrix})) .

X \in {x^{(1)}, \dots, x^{(¯ c)}} \Leftrightarrow X \in {1, \dots, ¯ c},

X \sim t (μ, σ^{2}, ν),

¯ f (x | γ) \equiv f^{t} (x | μ, σ^{2}, ν),

μ (z) \equiv b ϕ (z),

σ^{2} (z) \equiv s^{2} ≽ 0,

f_{b, s^{2}}^{2} (x | z) = f^{t} (x | b ϕ (z), s^{2}, ν),

β = E {\frac{ν + ¯ n}{ν + {(X - β ϕ (Z))}^{'} {(σ^{2})}^{- 1} (X - β ϕ (Z))} X ϕ {(Z)}^{'}} {(E {ϕ (Z) ϕ {(Z)}^{'}})}^{- 1}

σ^{2} = E {\frac{ν + ¯ n}{ν + {(X - β ϕ (Z))}^{'} {(σ^{2})}^{- 1} (X - β ϕ (Z))} (X - β ϕ (Z)) {(X - β ϕ (Z))}^{'}} .

X \sim E x p (θ, τ (\cdot), h (\cdot))

(\begin{matrix} X Z \end{matrix}) \sim t ((\begin{matrix} μ_{X} μ_{Z} \end{matrix}), (\begin{matrix} σ_{X}^{2} & σ_{X, Z} σ_{X, Z}^{'} & σ_{Z}^{2} \end{matrix}), ν),

C v {X | z} \equiv \int_{R^{¯ n}}^{¯ n} (x - E {X | z}) {(x - E {X | z})}^{'} f (x | z) d x .

X | z \sim t (μ_{X | z}, σ_{X | z}^{2}, ν + ¯ k),

σ_{X | z}^{2} \equiv \frac{ν + {(z - μ_{Z})}_{Z}^{'} {(σ_{Z}^{2})}^{- 1} (z - μ_{Z})}{ν + ¯ k} (σ_{X}^{2} - σ_{X, Z} {(σ_{Z}^{2})}^{- 1} σ_{X, Z}^{'}) .

Model	Parametric map $¯ f (\cdot \| γ)$

Normal	$f^{N} (\cdot \| μ, σ^{2})$ (49.1)
Student $t$	$f^{t} (\cdot \| μ, σ^{2}, ν)$ (49.144)

18.3 Probabilistic regression

18.3.1 Error

18.3.2 Theoretical optimum

18.3.3 Target parameters

18.3.4 Optimization in practice

18.3.5 Linear regression

Discriminative model

Generative embedding

Exponential family embedding

18.3.6 Generalized linear models

18.3.7 Alternative generalizations

Bibliography

External Links

Discussions