The LMS estimator minimizes the [[Mean Squared Error]] ("MSE") by choosing the [[Conditional Expectation]] of the parameter $\Theta$ given the observed data $X$. It is expressed as: Estimator (random variable): $ \hat \Theta_{\mathrm{LMS}} =\mathbb{E}[ \Theta \vert X] $ Estimate (realized value): $ \hat \theta_{\mathrm{LMS}} =\mathbb{E}[ \Theta \vert X=x] $ ## Error Analysis The error term $\tilde \Theta$ quantifies the deviation between the estimate $\hat \Theta$ and the true parameter $\Theta$. $ \tilde \Theta = \hat \Theta - \Theta $ **Unconditional Error:** We can prove that the error $\tilde \Theta$ of the LMS estimator is unbiased (i.e. zero in expectation). $ \begin{align} \hat \Theta &= \mathbb{E}[\Theta \vert X] \tag{1}\\ \mathbb{E}[\hat \Theta] &= \mathbb{E}\big [ \mathbb{E}[\Theta \vert X] \big ] \tag{2}\\ \mathbb{E}[\hat \Theta] &= \mathbb{E}[\Theta] \tag{3}\\ \mathbb{E}[\hat \Theta] - \mathbb{E}[\Theta] &= 0 \tag{4}\\ \mathbb{E}[\tilde \Theta]&=0 \tag{5} \end{align} $ where: - (2) Putting expectation on both sides. - (3) The [[Law of Iterated Expectations]] transforms the expectation of a conditional expectation into an unconditional expectation. **Conditional Error:** Even conditionally, the expected error is unbiased. $ \begin {align} \mathbb{E}[\tilde \Theta \vert X=x] &= \mathbb{E}[\hat \Theta - \Theta \vert X=x] \tag{1}\\ &= \hat \Theta-\mathbb{E}[\Theta \vert X=x] \tag{2}\\ &= \hat \Theta - \hat \Theta \tag{3}\\[2pt] &=0 \tag{4} \end {align} $ where: - (2) The estimator $\hat \Theta$ turns into a constant when when $X=x$. Therefore $\hat \Theta$ can be pulled out of the expectation. - (3) The conditional expectation of $\Theta$ is the definition of the LMS estimator itself. **Covariance of Error:** To derive the covariance between the LMS estimator and the error is zero, we first need to find $\mathbb{E}[\tilde \Theta \hat \Theta]$. $ \begin{align} \mathbb E[\tilde \Theta \hat \Theta]&= \mathbb E\big[\mathbb E[\tilde \Theta \hat \Theta\vert X=x]\big] \tag{1} \\ &=\mathbb E\big[\hat \Theta * \mathbb E[\tilde \Theta \vert X=x]\big] \tag{2} \\ &=\mathbb E[\hat \Theta *0] \tag{3} \\[2pt] &=0 \tag{4} \end{align} $ where: - (1) Expressing the unconditional expectation as an expectation of all conditional expectations by law of iterated expectations. - (2) Pulling out $\hat \Theta$ of the inner expectation, as it is a constant. - (3) As established above the conditional expectation of the error $\tilde \Theta$ is zero. We apply the expanded form of [[Covariance#Expanded Form of Covariance|Covariance]], plugging in the values we have derived already. $ \mathrm{Cov}(\tilde \Theta, \hat \Theta) = \overbrace{\mathbb E[\tilde \Theta \hat \Theta]}^{=0} - \overbrace{\mathbb E[\tilde \Theta]}^{=0} * \mathbb{E}[\hat \Theta] = 0 $ **Variance of Error:** In the special case of the LMS estimation, the zero covariance is sufficient to express [[Variance of Sum of Random Variables#Special Case of Independence|Variance of Sum of Random Variables]] as sum of separate variances. $ \begin{align} \mathrm{Var}(\Theta) = \mathrm{Var}(\hat \Theta + \tilde \Theta) = \mathrm{Var}(\hat \Theta)+\mathrm{Var}(\tilde \Theta) \end{align} $ By definition the [[Variance]] of $\tilde \Theta$ is equal to the unconditional $\text{MSE}$. $ \begin{align} \mathrm{Var}(\tilde \Theta) &=\mathbb E \big [(\tilde \Theta - \mathbb E [\tilde \Theta])^2 \big] =\mathbb E [\tilde \Theta^2] \\ \mathrm{MSE} &= \mathbb{E}\big [(\Theta- \hat \Theta)^2 \big ] = \mathbb E [\tilde \Theta^2] \end{align} $ ## Conditional MSE The conditional MSE is the performance metric of the estimate given specific observations $X=x$. We will see that for the LMS estimator $\hat \Theta_{\text{LMS}}$ the conditional MSE is equal to the conditional [[Variance]]. $ \begin{align} \mathrm{MSE} &= \mathbb{E}\big [(\Theta- \hat \Theta_{\text{LMS}})^2 \vert X \big ] \tag{1} \\[6pt] &= \mathbb{E}\big [(\Theta- \hat \theta_{\text{LMS}})^2 \vert X=x \big ] \tag{2}\\[6pt] &= \mathbb{E}\big [(\Theta- \mathbb E[\Theta \vert X=x])^2 \vert X=x \big ] \tag{3}\\[6pt] &= \mathbb{E}\big [(\Theta- \mathbb E[\Theta])^2 \vert X=x \big ] \tag{4}\\[6pt] &= \mathrm{Var}(\Theta \vert X=x) \tag{5} \end{align} $ where: - (2) Once we made concrete observations $(X=x)$, we can replace $\hat \Theta_{\mathrm{LMS}}$ with $\hat \theta_{\mathrm{LMS}}$. - (3) Insert the definition of the estimator $\hat \theta_{\mathrm{LMS}}$. - (4) We can get rid of the additional conditioning, as the outer [[Expectation]] is already conditioning on $X=x$. To prove that the estimator $\hat \Theta_{\text{LMS}}$ yields the minimum possible MSE, we differentiate the MSE definition w.r.t. $\hat \theta$ and set it to zero. $ \begin{align} \mathrm{MSE} &= \mathbb{E}\big [(\Theta- \hat \Theta)^2 \vert X=x \big ] \tag{6}\\[4pt] &=\mathbb{E}[\Theta^2 \vert X=x]+ \mathbb{E}[-2 \hat \theta\Theta \vert X=x] + \hat \theta^2 \tag{7}\\[4pt] &=\mathbb{E}[\Theta^2 \vert X=x]-2 \hat \theta* \mathbb{E}[\Theta \vert X=x] + \hat \theta^2 \tag{8}\\[2pt] \frac{d}{d\hat \theta} \mathrm{MSE} &=-2 \mathbb{E}[\Theta \vert X=x] + 2\hat \theta \tag{9}\stackrel{!}{=} 0\\ \hat \theta &= \mathbb{E}[\Theta \vert X=x] \tag{10} \end{align} $ (6) Expanding the quadratic formula $(\Theta - \hat \Theta)^2$. ## Unconditional MSE The unconditional MSE evaluates the performance of the estimator before any observations are made. It is defined as the expectation of the conditional MSE. $ \begin{align} \mathrm{MSE} &= \mathbb{E}\big [(\Theta- \hat \Theta)^2 \big ] \\[4pt] &= \mathbb E\big [\mathbb E[\Theta - \hat \Theta]^2\vert X=x \big]\\[6pt] &= \mathbb E\big[\mathrm{Var}(\Theta \vert X=x) \big] \end{align} $ The $\hat \theta$ which leads to the minimum unconditional MSE, is found by taking the derivative w.r.t. $\hat \theta$. We see that MSE is minimized at $\mathbb E[\Theta]$, which is the definition of LMS estimator before seeing any data. $ \begin{align} \mathrm{MSE}&=\mathbb{E} \big [(\Theta- \hat \theta)^2 \big ] \\[6pt] &=\mathbb{E}[\Theta^2]-2\hat \theta* \mathbb{E}[\Theta] + \hat \theta^2 \\ \frac{d}{d\hat \theta} \mathrm{MSE}&= -2 \mathbb{E}[\Theta] + 2\hat \theta \stackrel{!}{=}0\\ \hat \theta &= \mathbb{E}[\Theta] \end{align} $ If we plugin our unconditional estimate $\mathbb E[\Theta]$ into the unconditional MSE formula, we see that it equals the variance of $\Theta$. $ \begin{align} \mathrm{MSE}&=\mathbb{E} \big [(\Theta- \hat \theta)^2 \big ] \\ &=\mathbb{E} \big [(\Theta- \mathbb{E}[\Theta])^2 \big ] \\ &= \mathrm{Var}(\Theta) \end{align} $