The [[Law of Large Numbers]] guarantees that the sample mean $\bar X_n$ converges to the true expectation $\mu$ of a [[Random Variable]] $X$. However, the LLN does not describe the shape of the distribution that emerges during this convergence. The Central Limit Theorem ("CLT") fills this gap, describing the asymptotic distribution (i.e. [[Modes of Convergence#Convergence in Distribution|Convergence in Distribution]]) of the sample mean or sample sum. **Assumptions:** - Random variables $X_i, \dots, X_n$ are [[Independence and Identical Distribution|i.i.d.]]. - Random variables have a finite mean $\mu$ and variance $\sigma^2$ - We only make asymptotic statements for $n \to \infty$. ## Behavior of Variance for Increasing $n$ Consider a sequence of i.i.d. r.v‘s $X_1, \dots, X_n$ where each $X_i$ has the same finite $\mu, \sigma^2$. We examine the behavior of the variance when $n \to \infty$. **Sample sum:** Variance approaches $\infty$ with increasing $n$. $ \mathrm{Var}(S_n) =\mathrm{Var}\left(\sum_{i=1}^n X_i\right) =n* \sigma^2 \to \lim_{n \to \infty} =\infty $ **Sample mean:** Variance approaches zero with increasing $n$. $ \mathrm{Var}(M_n) =\mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n X_i\right) =\frac{\sigma^2}{n} \to \lim_{n \to \infty} =0 $ **Scaled sample sum:** Dividing the sample sum by $\sqrt n$ keeps the variance constant for all $n$. $ \begin{align} \mathrm{Var} \left(\frac{S_n}{\sqrt{n}} \right)=\frac{1}{n} \mathrm{Var}(S_n)=\frac{n\sigma^2}{n} \to \lim_{n \to \infty} = \sigma^2 \end{align} $ ## Standardization for Normalization To transform the scaled sample sum into a standard [[Gaussian Distribution]] $\mathcal N(0,1)$, we need following adjustments: $ \begin{align} \lim_{n\to \infty}\frac{S_n}{\sqrt n} &=\mathcal N(S_n, \sigma^2) \tag{1}\\[4pt] \lim_{n\to \infty}\frac{S_n - n\mu}{\sqrt n}&=\mathcal N(0, \sigma^2) \tag{2}\\[4pt] \lim_{n\to \infty}\frac{S_n - n\mu}{\sigma\sqrt n}&=\mathcal N(0, 1) \tag{3} \end{align} $ where: - (1) *Raw scaled sum*. - (2) *Centered scaled sum:* Subtracting from the sample sum its expectation $\mathbb E[S_n]=n\mu$. This centers the Gaussian around zero. Subtractions and additions of constants do not affect the variance. - (3) *Standardized sum:* Dividing by the standard deviation scales the variance to $1$. This gives a r.v. $Z \sim \mathcal N(0,1)$. $ Z_n = \frac{S_n -n \mu}{\sigma\sqrt n} \xrightarrow[n \to \infty]{(d)} \mathcal N(0,1)$ This can also be expressed in terms of $\bar X_n$: $ \begin{align} Z_n =\frac{n\bar X_n - n\mu}{\sigma \sqrt n} &\xrightarrow[n \to \infty]{(d)} \mathcal N(0,1) \tag{1}\\[8pt] \frac{n}{\sqrt n}*\frac{\bar X_n - \mu}{\sigma} &\xrightarrow[n \to \infty]{(d)} \mathcal N(0,1) \tag{2}\\[8pt] \sqrt n*\frac{\bar X_n - \mu}{\sigma} &\xrightarrow[n \to \infty]{(d)} \mathcal N(0,1) \tag{3}\\[8pt] \end{align} $ where: - (1) Write $S_n$ as $n*\bar X_n$. - (2) Factor out all $n$ terms. - (3) The first fraction simplifies to $\sqrt n$. ## CLT Statement The CLT states that as $n \to \infty$, the [[Cumulative Density Function|CDF]] of the standardized r.v. $Z$ converges to the CDF of a standard normal distribution $\Phi(z)$. $ \lim_{n \to \infty} \mathbf P(Z_n \le z) = \Phi(z) $ > [!note:] > With some more assumptions the convergence of CDF’s can be extended to convergence in or PMFs’ as well. ## Relaxing Assumptions The CLT even holds after removing or weakening some assumptions. - $X_i$ do not need to be identically distributed (but require assumptions about $\mu, \sigma^2$). - $X_i$ can have weak dependence (not fully independent). - $n$ can be moderate. However it helps when the $X_i$ are already close to normal, or at least have symmetry around their mean, and only have a single mode (”unimodal”).