Statistics¶
Job: infer properties of unknown probability laws from data; organize descriptive summaries versus inferential guarantees under frequentist or Bayesian decision rules.
Core¶
Statistics is inverse probability with decision theory: given samples \(X_1,\ldots,X_n\sim P_\theta\) (or nonparametric \(P\)), produce estimators, tests, and credible/confidence sets with quantified error. Philosophical split (frequentist vs Bayesian) changes what probability quantifies, not the need for rigorous risk analysis.
| Layer | Content |
|---|---|
| Descriptive | empirical measures, location/scale, dependence |
| Estimation | MLE, method of moments; bias/variance; consistency |
| Testing / CIs | Neyman–Pearson; \(p\)-values; coverage |
| Bayesian | prior/posterior; MAP; credible sets; decision rules |
| Asymptotics | CLT, delta method, LAN/contiguity (pointer) |
| High-dim / nonparam | sparsity, regularization (pointer) |
Reading map¶
| Page | Focus |
|---|---|
| Descriptive statistics | empirical summaries, dependence, visual/moment structure |
| Inferential statistics | estimation, testing, CIs; frequentist vs Bayesian |
| Probability | sampling models, foundations, distributions |
| Combinatorics | counting for discrete probability models |
Dependency order¶
- Probability foundations + distributions
- Descriptive — understand data functionals
- Inferential — procedures with guarantees
- Decision theory / asymptotics / high-dimensional layers as needed
- Information theory for divergences and large deviations
Philosophical one-liners¶
| School | Probability means | Procedures optimize / control |
|---|---|---|
| Frequentist | long-run frequency under repeated sampling | error rates over the sample space (coverage, type I) |
| Bayesian | coherent degree of belief | posterior expected loss; priors encode information |
Both schools use the same likelihood algebra; they disagree on the status of \(\theta\) as random and on pre- vs post-data probability statements.
Where it appears¶
- Probability: sampling models, LLN/CLT engines
- Information theory: KL, Sanov, Chernoff–Stein
- Optimization: MLE / ERM as optimization problems
- Science / ML: experimental design, A/B tests, calibration
- ML mathematics: generalization bounds, nonparametric minimax estimation, double descent
Signature results¶
| Result | Content |
|---|---|
| Consistency of MLE | regular parametric models; identifiability |
| Cramér–Rao | \(\operatorname{Var}\hat\theta\ge 1/I(\theta)\) (unbiased, regular) |
| Neyman–Pearson | LRT most powerful for simple \(H_0\) vs \(H_1\) |
| Bernstein–von Mises | posterior \(\approx\) normal centered at efficient estimator (reg.) |
| Glivenko–Cantelli | \(\|F_n-F\|_\infty\to0\) a.s. (i.i.d.) |
See it¶
Likelihood ratio test. For nested regular models, \(-2\log\Lambda\Rightarrow\chi^2_{\Delta\mathrm{df}}\) (Wilks) under \(H_0\). Same algebra underlies model selection heuristics; finite-sample and boundary cases (e.g. mixture weights at 0) break the \(\chi^2\) limit — hypothesis sensitivity matters.
Check¶
What does “95% confidence” mean frequentist-ly?
The procedure covers the true \(\theta\) in \(\ge95\%\) of repeated samples (for the assumed model). It is not “\(P(\theta\in\mathrm{CI}\mid\mathrm{data})=0.95\)” unless a Bayesian posterior is specified.
Why can an unbiased estimator be inadmissible?
Risk \(R(\theta,\delta)=\mathbb{E}_\theta L(\theta,\delta)\): Stein phenomenon — in \(\dim\ge3\) for multivariate normal mean, James–Stein dominates the sample mean under squared error. Unbiasedness \(\neq\) optimality.
Pitfalls¶
- \(p\)-values are not posterior probabilities of \(H_0\).
- Multiple testing inflates family-wise error; need Bonferroni / FDR control.
- Model misspecification: sandwich variance, quasi-likelihood; “MLE” of wrong model.
- Confusing prediction intervals with parameter confidence intervals.
Refs¶
Casella–Berger; van der Vaart, Asymptotic Statistics; Lehmann–Romano; Gelman et al. (Bayesian); Wasserman, All of Statistics.
Next¶
Descriptive · Inferential · Probability · Information theory · Optimization