The [[Likelihood Functions|likelihood function]] $f_\theta(y)$ measures how likely the observed data $Y$ is, assuming it comes from a distribution with parameter(s) $\theta$. We denote $\ell(\theta)$ as the log-likelihood of a single observation. $ \ell(\theta)=\log \big(f_\theta(y)\big) $ ![[identities-of-log-likelihood.png|center|500]] The log-likelihood has two important properties (”identities”), when evaluated around the true parameter $\hat \theta$: - *Expectation identity:* The first derivative of $\ell(\theta)$ shows the slope of the function, i.e. how sensitive $\ell(\theta)$ is to changes in $\theta$. When evaluated around the true parameter $\theta^\star$, the [[Expectation]] of the slopes is $0$. $ \mathbb E\Big[\frac{\partial \ell}{\partial \theta}\Big]=0 $ - *Information identity:* When evaluated around the true parameter $\theta^\star$, the following two terms offset each other. $ \mathbb E\left[ \frac{\partial ^2 \ell}{\partial \theta ^2}\right] +\mathbb E\left[\left(\frac{\partial \ell }{\partial \theta }\right)^2\right]=0 $ | Notation | Name | Description | | ------------------------------------------------------------------------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $\mathbb E\left[ \frac{\partial ^2 \ell}{\partial \theta ^2}\right]$ | Curvature of function | The second derivative of $\ell(\theta)$ shows how curved the likelihood is around the evaluated $\theta^\star$. As $\ell(\theta)$ should have a [[Identify Convex and Concave Functions\|concave]] shape at this point, the expectation of it is negative. | | $\mathbb E\left[\left(\frac{\partial \ell }{\partial \theta }\right)^2\right]$ | Variability of slope | The squares of the slope show how much the slope varies around $\theta^\star$. As this is the expectation of squared terms, the expectation has to be positive. |