Inferential Statistics¶
Job: Estimate parameters and test hypotheses with frequentist risk control or Bayesian posterior decision theory; state efficiency and asymptotics.
Core¶
Model: observations \(X\sim P_\theta\), \(\theta\in\Theta\). Goal: infer \(\theta\) or functionals \(\psi(\theta)\).
Frequentist¶
Estimator \(\hat\theta(X)\). Risk \(R(\theta,\hat\theta)=\mathbb{E}_\theta[L(\theta,\hat\theta)]\). Unbiased: \(\mathbb{E}_\theta\hat\theta=\theta\).
Cramér–Rao: for unbiased estimators of \(\theta\in\mathbb{R}\) under regularity, $$ \mathrm{Var}\theta(\hat\theta)\ge\frac{1}{I(\theta)},\qquad I(\theta)=\mathbb{E}\theta[(\partial_\theta\log f_\theta(X))^2]. $$ MLE under standard regularity: \(\sqrt{n}(\hat\theta_n-\theta)\Rightarrow N(0,I(\theta)^{-1})\) (efficient).
Confidence set \(C(X)\) of level \(1-\alpha\): \(\inf_\theta\mathbb{P}_\theta(\theta\in C(X))\ge1-\alpha\). Duality with tests.
Hypothesis tests: \(H_0:\theta\in\Theta_0\) vs \(H_1:\theta\in\Theta_1\). Type I error \(\sup_{\Theta_0}\mathbb{P}_\theta(\mathrm{reject})\); power function \(\beta(\theta)\). Neyman–Pearson: LRT most powerful for simple vs simple. Wilks: \(2\log\Lambda\Rightarrow\chi^2_{\mathrm{df}}\) under nested regular models.
p-value: for nested tests, infimum of levels at which \(H_0\) is rejected; not \(\mathbb{P}(H_0\mid\mathrm{data})\).
Bayesian¶
Prior \(\pi(d\theta)\), likelihood \(f(x\mid\theta)\), posterior \(\pi(d\theta\mid x)\propto f(x\mid\theta)\pi(d\theta)\) (Bayes). Credible sets from posterior mass. Decision: minimize posterior expected loss. Bernstein–von Mises: posterior \(\approx N(\hat\theta_{\mathrm{MLE}},(nI)^{-1})\) under regularity — frequentist calibration of Bayesian intervals.
One-liners¶
| School | Probability means | Validity |
|---|---|---|
| Frequentist | long-run sampling | error rates over \(X\sim P_\theta\) |
| Bayesian | coherent belief + data | posterior rationality; prior sensitivity |
See it¶
i.i.d. \(N(\mu,1)\), prior \(\mu\sim N(m_0,v_0)\): posterior \(N(m_n,v_n)\) with $$ v_n^{-1}=v_0^{-1}+n,\qquad m_n=v_n(v_0^{-1}m_0+n\bar x). $$ As \(n\to\infty\), \(m_n\sim\bar x\), \(v_n\sim1/n\) — BvM in closed form.
Check¶
Does a 95% CI contain \(\theta\) with probability 0.95 given the observed data?
Not in the frequentist reading: the random set \(C(X)\) covers \(\theta\) in \(\ge95\%\) of samples; after observing \(X=x\), \(\theta\) is fixed and either in \(C(x)\) or not. Bayesian 95% credible intervals do carry posterior probability 0.95 under the model and prior.
Why can unbiasedness conflict with admissibility?
Stein phenomenon: for \(X\sim N(\theta,I_p)\), \(p\ge3\), the MLE \(X\) is inadmissible for \(\|\hat\theta-\theta\|^2\) risk; James–Stein shrinks toward a point and dominates \(X\) despite bias.
Pitfalls¶
- Multiple testing: unadjusted p-values inflate family-wise error; use Bonferroni, BH (FDR), etc.
- Model misspecification: sandwich covariance; Bayesian posteriors concentrate at KL projections (not "truth").
- Optional stopping / data-dependent analyses break naive frequentist guarantees without martingale corrections.
Refs¶
Lehmann–Romano, Testing Statistical Hypotheses; van der Vaart, Asymptotic Statistics; Schervish, Theory of Statistics; Ghosal–van der Vaart (Bayesian nonparametrics).