Combining [[Autoregressive Model]] and [[Moving Average Model]] models into $\text{ARMA}(p,q)$ where $p$ defines the number of autoregressive lags and $q$ defines the number of lagged [[White Noise Model|White Noise]] terms. $ X_t=\phi_1X_{t-1}+\dots +\phi_pX_{t-p}+ W_t+ \theta_1W_{t-1}+\dots+\theta_qW_{t-q} $ For the following derivation of moments, consider a simpler $\mathrm{ARMA}(1,1)$ process of the form: $ r_t = \phi_0 + \phi_1 r_{t-1} + \epsilon_t- \theta \epsilon_{t-1} $ where: - $\epsilon$ is the white noise that follows $\mathcal N(0, \sigma^2)$ - $\lvert \phi_1 \rvert < 1$ to ensure [[Stationarity]] in the process - $\{\phi_0, \phi_1, \theta, \sigma^2\}$ are constants ## Expectation In a stationary process $\mathbb E[r_t]= \mu$. Thus, we simply write: $ \begin{align} \mu &= \mathbb E[\phi_0 + \phi_1 r_{t-1} + \epsilon_t- \theta \epsilon_{t-1}]\tag{1}\\[4pt] &=\phi_0+ \phi_1 \mathbb E[r_{t-1}]+\mathbb E[\epsilon]- \theta \mathbb E[\epsilon_{t-1}] \tag{2}\\[4pt] &=\phi_0+ \phi_1 \mu \tag{3}\\[4pt] \mu&=\frac{\phi_0}{1-\phi_1} \tag{4} \end{align} $ where: - (2) The expectation of white noise terms is zero. - (3) The expectation of $r_{t-1}$ is the same as for $r_t$ due to stationarity, and hence $\mu$. ## Variance For simplicity we assume $\phi_0=0$, which means that the process is centered (i.e. has zero mean). $ r_t = \phi_1 r_{t-1} + \epsilon_t- \theta \epsilon_{t-1} $ For such a process with zero mean, the [[Variance]] is equal to the second moment $\mathbb E[r_t^2]$. Hence we can write: $ \begin{align} \mathrm{Var}(r_t) &= \mathbb E\left[(\phi_1 r_{t-1} + \epsilon_t- \theta \epsilon_{t-1})^2\right]\tag{1}\\[6pt] &= \mathbb E\left[\phi_1^2 r_{t-1}^2 + \epsilon_t^2 + \theta^2 \epsilon_{t-1}^2 + 2 \,\phi_1 \, r_{t-1}\, \epsilon_t -2\phi_1r_{t-1} \theta \epsilon_{t-1}-2\epsilon_t\theta\epsilon_{t-1}\right]\tag{2}\\[6pt] &= \phi_1^2\mathbb E[r_{t-1}^2]+\mathbb E[\epsilon_t^2]+\theta^2 \mathbb E[\epsilon_{t-1}^2]+2\phi_1\mathbb E[r_{t-1}\epsilon_t]-2\phi_1\theta \mathbb E[r_{t-1} \epsilon_{t-1}] - 2\theta \mathbb E[\epsilon_t \epsilon_{t-1}]\tag{3}\\[6pt] &= \phi_1^2 \mathrm{Var}(r_t)+\sigma^2+\theta^2 \sigma^2-2\phi_1\theta \sigma^2 \tag{4} \end{align} $ In (4) multiple simplifications can be made: | Term | Explanation | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $\mathbb E[\epsilon_t^2]$ | Has zero mean for all $t$. Therefore its second moment is the variance, which by definition is $\sigma^2$. | | $\mathbb E[\epsilon_t \epsilon_{t-1}]$ | Represents the covariance between error terms, which by definition of the model are independent. Hence covariance is zero. | | $\mathbb E[r_{t-1}^2]$ | Represents the $\mathrm{Var}(r_{t-1})$ since the $\mathbb E[r_{t-1}]$ is zero. Due to stationarity the variance for all $t$ is the same. | | $\mathbb E[r_{t-1} \epsilon_t]$ | Represents the [[Covariance]] between the two [[Random Variable\|r.v's]] as both have zero mean. Since $e_t$ happens after $r_{t-1}$ the two terms are completely independent (i.e. covariance of zero). | | $\mathbb E[r_{t-1} \epsilon_{t-1}]$ | Since $r_{t-1}$ contains $\epsilon_{t-1}$ in its definition, they are not uncorrelated. More explicitly their covariance is $\sigma^2$. | Finally we can move over the variance from the right to the left side, and express it as follows: $ \begin{align} \mathrm{Var}(r_t) *(1-\phi_1^2) = \sigma^2+\theta^2 \sigma^2-2\phi_1\theta \sigma^2 \\[2pt] \mathrm{Var}(r_t) =\frac{\sigma^2(1 + \theta^2 - 2\phi_1 \theta)}{1-\phi_1^2} \end{align} $ ## Autocovariance We want to derive the 1-lag autocovariance $\gamma_1$ of the defined $\mathrm{ARMA}(1,1)$ process. Since the mean of $r$ for all $t$ is zero, we can write the covariance as the product of the expectation. $ \gamma_1 = \mathrm{Cov}(r_t, r_{t-1})=\mathbb E[r_tr_{t-1}]$ By filling in the recursion we can write: $ \begin{align} \gamma_1 &= \mathbb E[(\phi_1 r_{t-1} + \epsilon_t - \theta \epsilon_{t-1}) *r_{t-1}] \tag{1}\\[2pt] &=\mathbb E[\phi_1r_{t-1}^2+ \epsilon_tr_{t-1}- \theta \epsilon_{t-1}r_{t-1}] \tag{2}\\[2pt] &=\phi_1\mathbb E[r_{t-1}^2]+ \mathbb E[\epsilon_tr_{t-1}]- \theta \mathbb E[\epsilon_{t-1}r_{t-1}] \tag{3}\\[2pt] &=\phi_1 \mathrm{Var}(r_t)- \theta \sigma^2 \tag{4} \end{align} $ where: - (2) Multiply out the terms - (3) Apply [[Linearity of Expectations]] to write expectations separately - (4) Apply simplifications as above. Finally we fill in the derived definition of the variance, to get: $ \gamma_1 = \phi_1*\frac{\sigma^2(1 + \theta^2 - 2\phi_1 \theta)}{1-\phi_1^2} - \theta \sigma^2 $