A numerical characteristic of a [[Random Variable]] are its moments. The [[Expectation]] and [[Variance]] are special cases of that. - The $k^{th}$ moment of $X$: $ \mathbb{E}[X^k] $ - The $k^{th}$ central moment of $X$: $ \mathbb{E}\big[(X-\mathbb{E}[X])^k \big] $ Moments can be calculated directly from their definition, but, even for moderate values of $k$ , this approach becomes cumbersome. The moment generating function (”MGF”) provides an easier way to generate moments. ## Definition The MGF is defined as the expectation of $e^{tX}$ where $X$ is the r.v. of interest, and $t$ is some value close to zero. $ M_X(t)=\mathbb E [e^{tX}], \quad \text{for } t \in \mathbb R $ To obtain the expectation we need to sum/integrate over all possible values of $X$ and multiply with the respective [[Probability Mass Function|PMF]] / [[Probability Density Function|PDF]] value. $ \begin{align} M_X(t) = \mathbb E [e^{tX}] &=\sum_x e^{tx} * p_X(x) && \text{(discrete)}\\[4pt] M_X(t) = \mathbb E [e^{tX}]&=\int_{-\infty}^{\infty} e^{tx} * f_X(x)*dx && \text{(continuous)} \end{align} $ If we want to find the $k^{th}$ moment of $X$, which is $\mathbb E[X^k]$, we need to take $k$-times the derivative of the MGF w.r.t. $t$ and evaluate at $t=0$. $ \mathbb E[X^k] = \frac{d^k}{dt^k}\Big[M_X(t)\Big]_{t=0} $ The following table states MGFs for the most common type of distributions. | Distribution | MGF | | ------------ | ------------------------------------------------------ | | Bernoulli | $pe^t+(1-p)$ | | Binomial | $\big(pe^t+(1-p)\big)^n$ | | Poisson | $\exp \left(\lambda*(e^t-1)\right)$ | | Gaussian | $\exp \left({\mu t + \dfrac 1 2 \sigma^2 t^2} \right)$ | | Exponential | $\frac{\lambda}{\lambda -t}, \, t < \lambda$ | ## Derivation In the following we show why this function is able to generate moments. $ \begin{align} e^x &= 1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+ \cdots \tag{1}\\[10pt] e^{tx} &= 1+ tx+\frac{t^2x^2}{2!}+\frac{t^3x^3}{3!}+ \cdots \tag{2}\\[10pt] \mathbb{E}[e^{tX}] &= \mathbb{E}[1]+ \mathbb{E}[tX]+ \mathbb{E}\Big[\frac{t^2X^2}{2!}\Big]+ \mathbb{E}\Big[\frac{t^3X^3}{3!}\Big]+ \cdots \tag{3}\\[10pt] \mathbb{E}[e^{tX}] &= 1+ t*\mathbb{E}[X]+ \frac{t^2}{2!}* \mathbb{E}[X^2]+ \frac{t^3}{3!}* \mathbb{E}[X^3]+ \cdots \tag{4} \end{align} $ where: - (1) The exponential $e^x$ can be decomposed into such a series. - (2) Adding a constant $t$ to the series. - (3) Taking the expectation on both sides, and splitting up separate expectations by [[Linearity of Expectations]]. - (4) Pulling out constants from the expectations. Taking the first derivative w.r.t. $t$.. $ \begin{align} \frac{d}{dt}\mathbb{E}[e^{tX}] \Big \lvert_{t=0} &=0+ \mathbb{E}[X]+ t*\mathbb{E}[X^2]+\frac{3 t^2}{3!}\mathbb{E}[X^3] + \cdots \Big \rvert_{t=0} \\[8pt] &=0+\mathbb{E}[X] + 0+ 0+ \cdots \\[8pt] &=\mathbb{E}[X] \end{align} $ Taking the second derivative w.r.t. $t$.. $ \begin{align} \frac{d^2}{dt^2}\mathbb{E}[e^{tX}] \Big \lvert_{t=0} &=0+ 0+\mathbb{E}[X^2]+\frac{6 t}{3!}\mathbb{E}[X^3] + \cdots \Big \rvert_{t=0} \\[8pt] &=0+0+\mathbb{E}[X^2] +0+ \cdots \\[8pt] &=\mathbb{E}[X^2] \end{align} $ >[!note:] >So the reason why exactly $\mathbb{E}[e^{tX}]$ generates the respective moment is because, with each $k$-th derivation the expectation term of $X^k$ does not carry any $t$, and is therefore not multiplied by $0$. Also when we set $t=0$, we can see the [[Probability Axioms#Axioms|Normalization axiom]] fulfilled. $ \begin{align} M_X(t) =\mathbb E[e^{tX}]=&\int_{-\infty}^{\infty}e^{tX} * f_X(x) \, dx \\[4pt] =\mathbb E[1] =& \int_{-\infty}^{\infty}1 * f_X(x) \, dx = 1 \end{align} $