Probability Distributions¶
Job: Encode laws of random variables; use densities, PMFs, and characteristic functions; state LLN/CLT precisely and identify standard parametric families.
Core¶
Law / distribution: pushforward \(\mu_X=\mathbb{P}\circ X^{-1}\) on \((\mathbb{R},\mathcal{B})\). CDF \(F_X(x)=\mathbb{P}(X\le x)\) càdlàg, determines \(\mu_X\).
Absolutely continuous: \(\mu_X\ll\mathrm{Leb}\) with density \(f_X\), \(F_X(x)=\int_{-\infty}^x f\). Discrete: \(\mu_X=\sum p_k\delta_{x_k}\). Singular continuous: e.g. Cantor distribution (CDF continuous, derivative \(0\) a.e., not abs. cont.).
Moments: \(\mathbb{E}[X^k]\) if integrable; MGF \(M(t)=\mathbb{E}[e^{tX}]\) in a neighborhood of \(0\) \(\Rightarrow\) moments determine the law in regular cases (Carleman condition). Characteristic function always exists: \(\varphi(t)=\mathbb{E}[e^{itX}]\).
| Family | Density / mass | Parameters | Key property |
|---|---|---|---|
| Bernoulli / Binomial | \(p\); \(\binom{n}{k}p^k(1-p)^{n-k}\) | \(p,n\) | sum of i.i.d. Bernoullis |
| Poisson(\(\lambda\)) | \(e^{-\lambda}\lambda^k/k!\) | \(\lambda\) | rare events; limit of Binomial |
| Geometric | \((1-p)^{k-1}p\) | \(p\) | memoryless on \(\mathbb{N}\) |
| Uniform\([a,b]\) | \(1/(b-a)\) | \(a,b\) | max-entropy on bounded interval |
| Exponential(\(\lambda\)) | \(\lambda e^{-\lambda x}1_{x>0}\) | \(\lambda\) | memoryless on \(\mathbb{R}_+\) |
| Gamma(\(k,\theta\)) | shape-scale form | \(k,\theta\) | sums of i.i.d. Exp |
| Normal(\(\mu,\sigma^2\)) | \((2\pi\sigma^2)^{-1/2}e^{-(x-\mu)^2/(2\sigma^2)}\) | \(\mu,\sigma\) | stable under convolution; max entropy given variance |
| \(\chi^2_n\), \(t_\nu\), \(F\) | derived from Gaussians | df | sampling distributions |
Weak LLN (Khintchine): \(X_i\) i.i.d. \(L^1\), \(\bar X_n\xrightarrow{\mathbb{P}}\mu\).
Strong LLN (Kolmogorov): same hypotheses \(\Rightarrow\) \(\bar X_n\to\mu\) a.s. (integrability is necessary and sufficient in the i.i.d. case).
CLT (Lindeberg–Lévy): \(X_i\) i.i.d., \(\mathbb{E}X_i=\mu\), \(\mathrm{Var}(X_i)=\sigma^2\in(0,\infty)\), then $$ \sqrt{n}\,\frac{\bar X_n-\mu}{\sigma}\ \Rightarrow\ \mathcal{N}(0,1) $$ in distribution. Proof sketch: Taylor expand \(\varphi\) near \(0\), use Lévy continuity. Lindeberg–Feller covers independent non-identical rows of triangular arrays.
Multivariate Gaussian: law determined by mean \(m\) and covariance \(\Sigma\); density when \(\Sigma\succ0\); affine images remain Gaussian; uncorrelated \(\Leftrightarrow\) independent for jointly Gaussian coordinates.
See it¶
Berry–Esseen: if \(\mathbb{E}|X_1|^3=\rho<\infty\), then $$ \sup_x\bigl|F_n(x)-\Phi(x)\bigr|\le\frac{C\rho}{\sigma^3\sqrt{n}} $$ for the normalized sum CDF \(F_n\) — quantitative CLT used in rigorous error bars.
Check¶
Does finite variance follow from the CLT conclusion for i.i.d. sums?
Domain of attraction of the normal law requires truncated second-moment conditions; pure i.i.d. with infinite variance can still, after different normalization, converge to stable laws with index \(\alpha<2\) — not to \(N(0,1)\) with \(\sqrt{n}\) scaling.
Prove memorylessness of \(\mathrm{Exp}(\lambda)\).
\(\mathbb{P}(X>s+t\mid X>s)=e^{-\lambda(s+t)}/e^{-\lambda s}=e^{-\lambda t}=\mathbb{P}(X>t)\). Characterization: continuous distributions on \(\mathbb{R}_+\) with this property are exponential.
Pitfalls¶
- Density is a Radon–Nikodym class (a.e. equivalence); point values of \(f\) are not probabilities.
- Convergence of densities \(\not\Leftrightarrow\) convergence in distribution without Scheffé / contiguity hypotheses.
- Heavy tails: sample means may fail LLN (stable laws with \(\alpha\le1\)).
Refs¶
Feller I–II; Durrett; Petrov, Limit Theorems of Probability Theory; Wainwright, high-dimensional statistics (concentration).