Linear Time Series Models

Time series data are viewed as realizations of a stochastic process, with dependence across time.

A time series model specifies the dynamic evolution of this stochastic process over time.

\(f(\cdot)\) is referred to as the data-generating process (DGP).

• DGP is generally unknown and potentially complex.

• We focus on linear stationary models:
  • autoregressive (AR) models
  • moving average (MA) models
  • autoregressive moving average (ARMA) models

Algebraic Properties

The lag operator is a linear operator:

1. Lag of a constant: \[ L c = c. \]

2. Distributive (linearity): \[ (L^i + L^j) y_t = L^i y_t + L^j y_t = y_{t-i} + y_{t-j}. \]

3. Associative law of multiplication: \[ L^i L^j y_t = L^{i+j} y_t = y_{t-(i+j)}, \qquad L^0 y_t = y_t. \]

Lag Polynomials

Because these properties hold, lag operators can be manipulated algebraically.

A lag polynomial is written as \[ \phi(L) = 1 - \phi_1 L - \cdots - \phi_p L^p. \]

Applying it to \(y_t\): \[ \phi(L) y_t = y_t - \phi_1 y_{t-1} - \cdots - \phi_p y_{t-p}. \]

Mean and Intercept

If \(\{x_t\}\) has nonzero mean \(\mu\), the AR(\(p\)) model can be written as

\[ x_t - \mu = \phi_1 (x_{t-1} - \mu) + \cdots + \phi_p (x_{t-p} - \mu) + w_t. \]

Equivalently,

\[ x_t = \alpha + \phi_1 x_{t-1} + \cdots + \phi_p x_{t-p} + w_t, \]

where \(\alpha = \mu(1 - \phi_1 - \cdots - \phi_p)\).

Stability and Initial Conditions

To eliminate the dependence on the initial condition \(x_{t-k}\), we need \(\phi^k x_{t-k}\) to vanish as \(k \to \infty\).

If \(|\phi|<1\) and the initial condition is square-integrable, then

\[ \phi^k x_{t-k} \to 0 \quad \text{in mean square as } k\to\infty. \]

This gives an expression of \(x_t\) as a weighted sum of current and past shocks. \[ x_t = \sum_{j=0}^{\infty}\phi^j w_{t-j}. \]

Autocovariances

\[ \begin{align} \gamma(h) =& \mathrm{Cov}(x_{t+h},x_t) = \mathbb{E}\!\left[ \left(\sum_{j=0}^{\infty} \phi^j w_{t+h-j}\right) \left(\sum_{k=0}^{\infty} \phi^k w_{t-k}\right) \right]\\ &\;\;\vdots \\ =& \frac{\sigma_w^2\,\phi^h}{1-\phi^2}, \qquad h \ge 0, \end{align} \] and by symmetry, \[ \gamma(-h)=\gamma(h). \]

Autocorrelation

\[ \rho(h) = \frac{\gamma(h)}{\gamma(0)}=\phi^h, \qquad h\ge 0, \qquad \rho(-h)=\rho(h). \]

When \(|\phi|<1\), the AR(1) process is weakly stationary!

• In this case, backward iteration does not converge. The weights grow rather than decay, so \(x_t\) cannot be written as a weighted sum of past shocks.

Note: the AR equation describes an algebraic relation. Here, ``explosive’’ refers to failure of the implicit forward-time recursive construction. It does not mean that the equation itself admits no stationary solution.

• Instead, write:

\[ x_{t+1} = \phi x_t + w_{t+1}. \] Solving for \(x_t\) gives \[ x_t = \phi^{-1} x_{t+1} - \phi^{-1} w_{t+1}. \]

Iterating forward \(k\) steps yields \[ x_t = \phi^{-k} x_{t+k} - \sum_{j=1}^{k} \phi^{-j} w_{t+j}. \]

If \(|\phi|>1\), then \(|\phi^{-1}|<1\), so \[ \phi^{-k} x_{t+k} \to 0 \quad \text{in mean square as } k\to\infty, \] and we obtain the representation \[ x_t = - \sum_{j=1}^{\infty} \phi^{-j} w_{t+j}. \]

This representation depends on future shocks \(\{w_{t+1}, w_{t+2},\ldots\}\).

Hence, the explosive AR(1) process is noncausal.

Distributional equivalence

Define the causal AR(1) process \[ y_t = \phi^{-1} y_{t-1} + v_t, \] where \[ v_t \sim wn(0,\sigma_w^2 \phi^{-2}). \]

The processes \(\{x_t\}\) and \(\{y_t\}\) have the same finite-dimensional distributions.

Example: If \[ x_t = 2x_{t-1} + w_t, \qquad \sigma_w^2 = 1, \] then the equivalent causal process is \[ y_t = \tfrac12 y_{t-1} + v_t, \qquad \sigma_v^2 = \tfrac14. \]

These two processes are observationally equivalent.

Unlike AR processes, MA(\(q\)) processes are stationary for all parameter values,
since \(x_t\) is a finite linear combination of white noise.

Autocovariances

The autocovariance function is \[ \gamma(h)=\mathrm{Cov}(x_{t+h},x_t) = \begin{cases} (1+\theta^2)\sigma_w^2, & h=0,\\[4pt] \theta\sigma_w^2, & h=1,\\[4pt] 0, & |h|>1. \end{cases} \]

Autocorrelation

Therefore the ACF is \[ \rho(h)=\frac{\gamma(h)}{\gamma(0)} = \begin{cases} \dfrac{\theta}{1+\theta^2}, & h=\pm 1,\\[8pt] 0, & |h|>1. \end{cases} \]

Note that \(|\rho(1)|\le \frac12\) for all \(\theta\) (maximum at \(|\theta|=1\)).

• Moreover, the variance \(\gamma(0)=(1+\theta^2)\sigma_w^2\) is the same for the parameter pairs \((\sigma_w^2,\theta)\) and \((\sigma_w^2\theta^2,\;1/\theta)\).

• Hence, the same stochastic process admits two MA(1) representations: \[ x_t = w_t + \theta w_{t-1}, \qquad x_t = v_t + \frac{1}{\theta} v_{t-1}, \] with appropriately scaled white noise.

• This lack of uniqueness creates an identification problem!

Parameter Redundancy

• Multiplying both sides by any lag polynomial \(\eta(L)\) gives

\[ \eta(L)\phi(L)x_t = \eta(L)\theta(L) w_t, \] which implies the same stochastic process.

Example: Consider the white noise process \[ x_t = w_t. \]

Multiply both sides by \(\eta(L)=1-0.5L\): \[ (1-0.5L)x_t = (1-0.5L)w_t. \]

Rewriting, \[ x_t = 0.5x_{t-1} + w_t - 0.5w_{t-1}. \]

This looks like an ARMA(1,1) model!

• To rule out parameter redundancy, we require that \(\phi(z)\) and \(\theta(z)\) have no common factors.

Invertibility via the MA root condition

When \[ \theta(z)\neq 0 \quad \text{for all } |z|\le 1, \] the inverse \(\theta(L)^{-1}\) exists, and shocks can be recovered from past data: \[ w_t = \sum_{j=0}^{\infty} \pi_j x_{t-j}, \qquad \pi(z)=\sum_{j=0}^{\infty}\pi_j z^j=\frac{\phi(z)}{\theta(z)}. \]

AR–MA Duality

When both root conditions hold, the same stationary process admits both a MA(\(\infty\)) and an AR(\(\infty\)) representation.

Wold Decomposition

If \(\{x_t\}\) is weakly stationary and purely nondeterministic, then it admits a representation of the form \[ x_t = \sum_{j=0}^{\infty} \psi_j \, \varepsilon_{t-j}, \qquad \psi_0 = 1, \qquad \sum_{j=0}^{\infty} \psi_j^2 < \infty, \] where \(\{\varepsilon_t\}\) is a white-noise innovation sequence (mean \(0\), variance \(\sigma_\varepsilon^2\)).

Interpretation

• The MA(\(\infty\)) form is the canonical representation of a stationary, purely nondeterministic time series.
• All second-order dependence is encoded in the coefficients \(\{\psi_j\}\).

Connection to ARMA

• Causality in ARMA ensures the model is consistent with the Wold form (a one-sided MA(\(\infty\))).
• Invertibility is not required for Wold’s representation; it is an additional restriction used to identify shocks from the data.

Mean

\[ \mathbb{E}(x_t) = \sum_{j=0}^q \theta_j \mathbb{E}(w_{t-j}) = 0. \]

Autocovariance

For \(h \ge 0\), \[ \gamma(h) = \mathrm{Cov}(x_{t+h}, x_t) = \begin{cases} \sigma_w^2 \displaystyle\sum_{j=0}^{q-h} \theta_j \theta_{j+h}, & 0 \le h \le q, \\[10pt] 0, & h > q. \end{cases} \]

Autocorrelation

\[ \rho(h) = \frac{\gamma(h)}{\gamma(0)} = \begin{cases} \displaystyle \frac{\sum_{j=0}^{q-h} \theta_j \theta_{j+h}} {1 + \theta_1^2 + \cdots + \theta_q^2}, & 1 \le h \le q, \\[10pt] 0, & h > q. \end{cases} \]

The ACF cuts off after lag \(q\) — the defining signature of an MA(\(q\)) process.

Mean

\[ \mathbb{E}(x_t) = 0. \]

Autocovariances: first \(p+1\) values

Multiply the AR(\(p\)) equation by \(x_{t-h}\) and take expectations.

• For \(h=1,\ldots,p\):

\[ \gamma(h) = \phi_1\gamma(h-1)+\phi_2\gamma(h-2)+\cdots+\phi_p\gamma(p-h). \]

• For \(h=0\):

\[ \gamma(0) = \phi_1\gamma(1)+\cdots+\phi_p\gamma(p)+\sigma_w^2. \]

These \(p+1\) equations determine \[ \gamma(0),\gamma(1),\ldots,\gamma(p). \]

Autocovariances: recursion for larger lags

Once \(\gamma(0),\ldots,\gamma(p)\) are known, all higher autocovariances follow from \[ \gamma(h) = \phi_1\gamma(h-1)+\cdots+\phi_p\gamma(h-p), \qquad h>p. \]

The ACF of an AR(\(p\)) process decays gradually and does not cut off.

Autocovariance: initial values and recursion

Multiply the ARMA(\(p,q\)) equation by \(x_{t-h}\) and take expectations.

For \(h \ge 0\), \[ \gamma(h) = \phi_1 \gamma(h-1) + \cdots + \phi_p \gamma(h-p) + \sum_{j=0}^q \theta_j\,\mathrm{Cov}(w_{t-j}, x_{t-h}). \]

Initial values

The MA terms contribute only when \(h \le q\), since \[ \mathrm{Cov}(w_{t-j},x_{t-h}) = 0 \quad \text{for } h>j. \]

Therefore, the values \[ \gamma(0),\gamma(1),\ldots,\gamma(\max(p,q)) \] must be determined by solving a finite system of equations.

Recursion

Once these initial values are known, the MA terms vanish and the autocovariances satisfy the AR(\(p\)) recursion \[ \gamma(h)=\phi_1\gamma(h-1)+\cdots+\phi_p\gamma(h-p), \qquad h>\max(p,q). \]

MA terms determine short-run autocovariances; AR terms govern long-run decay of the ACF.

Definition (Partial Autocorrelation Function)

The partial autocorrelation function (PACF) is the sequence \[ \{\alpha(h)\}_{h\ge1}, \] where, for each lag \(h\), the partial autocorrelation \(\alpha(h)\) is defined as the correlation between the residuals obtained after linearly projecting \[ x_t \quad\text{and}\quad x_{t-h} \] on the intermediate lags \[ x_{t-1}, x_{t-2}, \ldots, x_{t-h+1}. \]

Equivalent regression characterization

Equivalently, \(\alpha(h)\) is the coefficient on \(x_{t-h}\) in the linear projection \[ x_t = \beta_1 x_{t-1}+\cdots+\beta_{h-1}x_{t-h+1} +\alpha(h)\,x_{t-h} +u_t. \]

FYI: this is basically Frisch–Waugh–Lovell theorem.

Next unit: Estimation and Inference for Stationary Time Series