Probability — Foundations¶

Job: Set Kolmogorov's axioms; define random variables and expectation; construct conditional expectation as an $L^2$ (and $L^1$) projection; fix modes of convergence.

Core¶

Probability space $(\Omega,\mathcal{F},\mathbb{P})$: $\mathcal{F}$ a $\sigma$-algebra, $\mathbb{P}$ a measure with $\mathbb{P}(\Omega)=1$.

Kolmogorov axioms: $\mathbb{P}\ge0$, $\mathbb{P}(\Omega)=1$, countable additivity on pairwise disjoint events.

Random variable: measurable $X:(\Omega,\mathcal{F})\to(\mathbb{R},\mathcal{B}(\mathbb{R}))$. Law $\mu_X=\mathbb{P}\circ X^{-1}$. Expectation $\mathbb{E}[X]=\int_\Omega X\,d\mathbb{P}$ when integrable ($X\in L^1$).

Independence: $\sigma$-algebras $\mathcal{F}_i$ independent if $\mathbb{P}(\bigcap_{j=1}^k A_j)=\prod\mathbb{P}(A_j)$ for $A_j\in\mathcal{F}_{i_j}$ distinct indices. R.v.s independent if $\sigma(X_i)$ are.

Conditional probability: $\mathbb{P}(A\mid B)=\mathbb{P}(A\cap B)/\mathbb{P}(B)$ for $\mathbb{P}(B)>0$. $\sigma$-algebra version via conditional expectation of $1_A$.

Conditional expectation. For $X\in L^1(\mathcal{F})$ and sub-$\sigma$-algebra $\mathcal{G}\subset\mathcal{F}$, $\mathbb{E}[X\mid\mathcal{G}]$ is the a.s.-unique $\mathcal{G}$-measurable integrable r.v. with $$ \int_G \mathbb{E}[X\mid\mathcal{G}]\,d\mathbb{P}=\int_G X\,d\mathbb{P}\quad\forall G\in\mathcal{G}. $$ $L^2$ characterization: if $X\in L^2$, then $\mathbb{E}[X\mid\mathcal{G}]$ is the orthogonal projection of $X$ onto the closed subspace $L^2(\mathcal{G})\subset L^2(\mathcal{F})$. Tower property, pull-out, Jensen for conditional expectations follow.

Bayes / regular conditionals: on nice spaces (Polish), regular conditional distributions $\mathbb{P}(dy\mid x)$ exist; abstractly Radon–Nikodym on product measures.

Modes of convergence:

Mode	Definition
a.s.	$X_n\to X$ outside a null set
in prob.	$\mathbb{P}(
$L^p$	$\\|X_n-X\\|_p\to0$
in distribution	$\mathbb{E}[f(X_n)]\to\mathbb{E}[f(X)]$ all bounded continuous $f$

Implications: $L^p\Rightarrow$ in prob $\Rightarrow$ in distribution; a.s. $\Rightarrow$ in prob; converses need extra structure (UI for $L^1$, Skorokhod for a.s. representations).

Characteristic functions and uniqueness¶

Characteristic function $\varphi_X(t)=\mathbb{E}[e^{itX}]$ (Fourier transform of the law $\mu_X$). For $\mathbb{R}^d$: $\varphi_X(t)=\mathbb{E}[e^{i\langle t,X\rangle}]$.

Property	Statement
Always defined	$
Uniqueness	$\varphi_X=\varphi_Y$ on $\mathbb{R}^d$ $\Rightarrow$ $\mathcal{L}(X)=\mathcal{L}(Y)$ (Fourier inversion / Stone–Weierstrass on measures)
Inversion	if $\varphi\in L^1$, density $f(x)=(2\pi)^{-d}\int e^{-i\langle t,x\rangle}\varphi(t)\,dt$
Moments	if $\mathbb{E}
Independence	$X\perp Y$ $\Rightarrow$ $\varphi_{X+Y}=\varphi_X\varphi_Y$; converse true for the joint cf factorization

Lévy continuity theorem. For r.v.s on $\mathbb{R}^d$: if $\varphi_{X_n}(t)\to\psi(t)$ pointwise and $\psi$ is continuous at $0$, then $\psi=\varphi_X$ for some $X$ and $X_n\Rightarrow X$ in distribution. Conversely, weak convergence $\Rightarrow$ pointwise cf convergence. This is the standard route to CLTs (Lindeberg, etc.).

Tightness and Prokhorov¶

A family $\Pi$ of probability measures on a metric space $S$ is tight if for every $\varepsilon>0$ there is compact $K\subset S$ with $\mu(K)\ge 1-\varepsilon$ for all $\mu\in\Pi$. On $\mathbb{R}^d$: tight $\Leftrightarrow$ $\sup_{\mu\in\Pi}\mu(|x|>R)\to 0$ as $R\to\infty$.

Theorem (Prokhorov). On a Polish space $S$: - $\Pi$ tight $\Rightarrow$ $\Pi$ is relatively compact in the weak topology (every sequence has a weakly convergent subsequence of probabilities). - Conversely, if $S$ is Polish, relative compactness $\Rightarrow$ tightness.

Uses. Extract weakly convergent subsequences of laws (invariance principles, existence of Brownian motion via consistent marginals + tightness in $C[0,1]$ or $D[0,1]$); upgrade finite-dimensional convergence to process-level convergence. On $\mathbb{R}^d$, tightness of $\{X_n\}$ + convergence of $\varphi_{X_n}$ $\Rightarrow$ distributional convergence (combine Prokhorov + Lévy).

See it¶

Projection view: best mean-square predictor of $X$ given $\sigma(Y)$ is $\mathbb{E}[X\mid Y]=g(Y)$ for Borel $g$ (Doob–Dynkin). For jointly Gaussian $(X,Y)$, $g$ is affine — Kalman filter linear update is conditional expectation in the Gaussian model.

Check¶

Show that independence of $X,Y\in L^2$ implies $\mathbb{E}[X\mid Y]=\mathbb{E}X$ a.s.

For $G\in\sigma(Y)$, $\int_G X=\mathbb{E}[X1_G]=\mathbb{E}X\cdot\mathbb{E}[1_G]=\int_G \mathbb{E}X$ by independence. Uniqueness of conditional expectation $\Rightarrow$ claim.

Give a sequence with $X_n\to0$ in probability but not a.s.

Typewriter: on $[0,1]$ with Lebesgue, indicators of $[0,1]$, $[0,1/2]$, $[1/2,1]$, $[0,1/3]$, … Cycle through dyadic intervals. $\mathbb{P}(X_n=1)\to0$ but $X_n(\omega)=1$ i.o. a.s.

Why does continuity of the pointwise limit of $\varphi_{X_n}$ at $0$ matter in Lévy’s theorem?

Without continuity at $0$, mass can escape to infinity: e.g. $X_n=n$ has $\varphi_{X_n}(t)=e^{itn}\to 0$ for $t\not\in 2\pi\mathbb{Z}$ in an oscillatory sense, but laws are not tight and do not converge weakly to a probability on $\mathbb{R}$. Continuity of the limit at $0$ restores tightness on $\mathbb{R}^d$.

Pitfalls¶

$\mathbb{E}[X\mid Y=y]$ as a function of $y$ is only defined $\mu_Y$-a.e.; versions matter for continuous densities.
Pairwise independence $\not\Rightarrow$ mutual independence ($n\ge3$).
Conditional probability given null events requires regular conditionals / disintegrations — not the elementary ratio.
Tightness is metric-space dependent: relative compactness in $C[0,1]$ needs modulus-of-continuity controls, not just tight marginals.
Characteristic functions determine laws, but recovering densities needs integrability / smoothing of $\varphi$.

Refs¶

Durrett, Probability: Theory and Examples; Kallenberg, Foundations of Modern Probability; Billingsley, Probability and Measure.

Next¶

Distributions · Index · Measure theory · Stochastic processes · Statistics