Spiked Models, BBP, and RMT Applications to Statistics¶
Job: develop the spiked random matrix models (covariance / Wigner), the Baik–Ben Arous–Péché (BBP) phase transition, the deformed semicircle / free-convolution outlier computation, and the RMT applications to high-dimensional statistics (PCA, signal detection) — with the rank-1 spiked Wigner / the BBP / the PCA detection as the worked example.
Core¶
Spiked random matrix models¶
The spiked covariance model (the sample covariance matrix \(S=\frac1N XX^T\) with \(X=M+N\) a signal \(M\) of rank \(r\) plus noise \(N\), the Johnstone / spiked model). The spiked Wigner (the deformed Wigner \(H=H_0+\sum\theta_i v_iv_i^T\) with the spikes \(\theta_i\) and the rank-\(r\) perturbation).
| Spiked model | Definition | Phase transition |
|---|---|---|
| Spiked covariance | \(S=\frac1N XX^T\), \(X\) the signal+noise (rank-\(r\) signal) | the Marchenko–Pastur edge + the BBP outlier |
| Spiked Wigner (rank-\(r\)) | \(H=H_0+\sum_{i=1}^r\theta_i v_iv_i^T\) (rank-\(r\) spike) | the semicircle edge + the BBP outlier at \(\theta+\theta^{-1}\) |
| Rank-1 spiked Wigner | \(H=H_0+\theta vv^T\) (\(\theta\) the spike strength, \(v\) a unit vector) | the \(\theta_c=1\) BBP transition |
The BBP phase transition in detail¶
Theorem (BBP, Baik–Ben Arous–Péché). For the spiked Wigner \(H=H_0+\theta vv^T\) (rank-1 spike), the largest eigenvalue \(\lambda_{\max}\): - Subcritical (\(\theta<1\)): \(\lambda_{\max}\to 2\) (the semicircle edge) with Tracy–Widom fluctuations \(N^{2/3}(\lambda_{\max}-2)\to\mathrm{TW}_2\) (the spike is buried). - Supercritical (\(\theta>1\)): \(\lambda_{\max}\to\theta+\theta^{-1}\) (the outlier, the BBP location) with Gaussian fluctuations \(N^{1/2}(\lambda_{\max}-(\theta+\theta^{-1}))\to\mathcal{N}(0,\sigma^2(\theta))\) (the spike pops out). - Critical (\(\theta=1\)): a different transition distribution (the critical BBP, the deformed Tracy–Widom).
| BBP regime | Eigenvalue location | Fluctuations | Eigenvector overlap \(|\langle v_1,v\rangle|^2\) | |------------|---------------------|--------------|-----------------------------------------------| | Subcritical (\(\theta<1\)) | \(\lambda_{\max}\to 2\) (edge) | Tracy–Widom \(N^{2/3}\) | \(0\) (uncorrelated) | | Critical (\(\theta=1\)) | \(\lambda_{\max}\to 2\) (edge) | the deformed TW | the critical overlap | | Supercritical (\(\theta>1\)) | \(\lambda_{\max}\to\theta+\theta^{-1}\) (outlier) | Gaussian \(N^{1/2}\) | \(1-\theta^{-2}\) (aligned) |
RMT and high-dimensional statistics¶
The Tracy–Widom as the null distribution for the PCA (the principal component analysis, the largest eigenvalue of the sample covariance under the null = no signal, the Johnstone Tracy–Widom test). The spiked covariance / the signal detection (the BBP transition as the detectability threshold, the phase transition in the sparse PCA / the information-theoretic threshold).
| RMT / statistics | The RMT result |
|---|---|
| PCA null (no signal) | \(\lambda_{\max}\) follows the Tracy–Widom (the Johnstone test) |
| Signal detection | the BBP transition: \(\theta>\theta_c=1\) detected, \(\theta<\theta_c\) undetectable |
| Sparse PCA | the information-theoretic threshold (the BBP \(\theta_c\)) vs the computational threshold (the gap) |
| Spiked covariance | the Marchenko–Pastur edge + the BBP outlier (the spiked MP) |
The universality of spiked models and the deformed semicircle¶
The deformed semicircle law: the free convolution of the semicircle with the spike distribution (the law of \(H_0+\sum\theta_i v_iv_i^T\) via the free probability free convolution \(\boxplus\)). The outlier locations via the \(R\)-transform / the subordination: the free probability computation of the outlier locations (\(\theta+\theta^{-1}\) for the rank-1 Wigner spike) via the \(R\)-transform of the semicircle (the \(R\)-transform \(R_{\mathrm{sc}}(z)=z\), the subordination \(\omega(\theta)=\theta+R_{\mathrm{sc}}(\theta)=\theta+\theta\)... the outlier at \(\theta+1/\theta\)).
| Deformed semicircle | Description |
|---|---|
| Free convolution | \(\mu_{\mathrm{sc}}\boxplus\sum\delta_{\theta_i}\) (the deformed semicircle) |
| \(R\)-transform / subordination | \(R_{\mathrm{sc}}(z)=z\); the outlier at \(\theta+\theta^{-1}\) (the subordination) |
| BBP universality | the phase transition + the outlier locations universal (independent of the noise distribution) |
Other applications (pointers)¶
| RMT application | The connection |
|---|---|
| RMT and the Riemann zeta | the Montgomery–Dyson pair correlation; the Bogomolny–Keating / the Rudnick–Sarnak \(n\)-level correlations (the GUE hypothesis for the zeta zeros, analytic number theory) |
| RMT and quantum chaos | the Bohigas–Giannoni–Schmit conjecture (the chaotic spectra modeled by GUE / GOE, the quantum billiards) |
| Free probability / operator algebras | the limiting spectral distributions as the free probability / the asymptotic freeness — the bridge to NCG / operator algebras and free probability |
See it¶
BBP phase transition for the rank-1 spiked Wigner and the PCA signal detection. The spiked Wigner \(H=H_0+\theta vv^T\) (\(\theta\) the spike strength, \(v\) the signal direction). The largest eigenvalue: - Below \(\theta_c=1\): \(\lambda_{\max}=2+O(N^{-2/3})\) (Tracy–Widom, buried in the bulk — the signal is undetectable, the \(\lambda_{\max}\) follows the semicircle edge Tracy–Widom). - Above \(\theta_c=1\): \(\lambda_{\max}=\theta+\theta^{-1}+O(N^{-1/2})\) (the outlier, the Gaussian fluctuation — the signal is detected, the \(\lambda_{\max}\) pops out of the bulk).
The PCA test: under the null (no signal, \(\theta=0\)), \(\lambda_{\max}\) follows the Tracy–Widom (the Johnstone test); the signal is detected if \(\lambda_{\max}\) exceeds the Tracy–Widom threshold. The detectability threshold (the BBP) is \(\theta_c=1\) — below, the signal is buried (the PCA fails to detect, the \(\lambda_{\max}\) is Tracy–Widom); above, the signal pops out (the \(\lambda_{\max}\) is the outlier, the Gaussian). The eigenvector overlap \(|\langle v_1,v\rangle|^2\) (the top eigenvector \(v_1\) vs the spike \(v\)): \(0\) below \(\theta_c\) (the top eigenvector is uncorrelated with the signal), \(1-\theta^{-2}\) above (the top eigenvector aligns with the signal, the BBP eigenvector transition). The key point: the BBP transition is the rigorous RMT detectability threshold for the PCA, with the Tracy–Widom as the null and the Gaussian as the outlier distribution — the operational content of the spiked RMT for high-dimensional statistics.
Check¶
Show that the BBP outlier location \(\theta+\theta^{-1}\) is computed via the free-convolution \(R\)-transform, and identify the critical spike.
The spiked Wigner \(H=H_0+\theta vv^T\) has the limiting spectral distribution \(\mu_{\mathrm{sc}}\boxplus\delta_\theta\) (the free convolution of the semicircle with the point mass at \(\theta\), the asymptotic freeness of \(H_0\) and the rank-1 perturbation). The \(R\)-transform of the semicircle is \(R_{\mathrm{sc}}(z)=z\) (the free cumulants \(\kappa_n=\delta_{n,1}\)... actually \(\kappa_2=1\), the \(R(z)=z\)). The outlier location is the solution of the subordination equation \(\theta=\omega+R_{\mathrm{sc}}(\omega)^{-1}\)... the standard computation: the outlier \(\lambda_{\mathrm{out}}\) satisfies \(\theta=R_{\mathrm{sc}}(\lambda_{\mathrm{out}})+1/\lambda_{\mathrm{out}}\)... equivalently \(\lambda_{\mathrm{out}}=\theta+1/\theta\) for the rank-1 spike (the BBP location). The critical spike is \(\theta_c=1\) (where \(\theta+\theta^{-1}=2\) meets the semicircle edge — the outlier touches the edge at \(\theta=1\), the phase transition).
Why is the BBP detectability threshold \(\theta_c=1\) the information-theoretic threshold, and what is the computational gap for the sparse PCA?
The BBP threshold \(\theta_c=1\) is the information-theoretic threshold: above \(\theta_c\), the top eigenvalue pops out (the signal is detectable from the spectrum), below \(\theta_c\) the signal is buried (the spectrum is indistinguishable from the null). This is the information-theoretic detectability (the spectrum contains the signal). For the sparse PCA (the spike \(v\) is sparse, \(v\) has \(k\ll N\) nonzero entries), the computational threshold (the threshold for the polynomial-time algorithms to detect the signal) can be higher than the BBP information-theoretic threshold — the computational gap (the sparse PCA may be information-theoretically detectable but computationally hard, the Berthet–Rigollet / the Krauthgamer–Nadler threshold, the gap between the information-theoretic and the computational detectability).
Pitfalls¶
- The BBP transition is for rank-\(r\) spikes (the finite-rank perturbation; the full-rank / the deformed Wigner is the free convolution \(\boxplus\), the limiting distribution, not the outlier phase transition).
- The detectability threshold (the BBP) is the information-theoretic / the computational threshold (the sparse PCA may have a gap between the two — the information-theoretic detectability at \(\theta_c=1\), the computational detectability at a higher threshold for the sparse / structured spike).
- The deformed semicircle / the free convolution is the limiting spectral distribution (the macroscopic / the ESD), not the local statistics (the outlier is the macroscopic outlier; the Tracy–Widom is the local edge fluctuation, a different scale).
- The universality of the BBP holds for the Wigner noise (the finite variance / the moment matching), not for the heavy-tail / the sparse noise (the Lévy / the tree limits give different outliers).
- The Montgomery–Dyson / the zeta-GUE connection is conjectural — the Bogomolny–Keating / the Rudnick–Sarnak results are the partial rigorous evidence (the \(n\)-level correlations match the GUE in a range), not the full GUE hypothesis for the zeta zeros.
- The free probability is the limiting mechanism (the asymptotic freeness, the \(R\)-transform / the subordination), but the finite-\(N\) fluctuations are the RMT (the Tracy–Widom, the BBP Gaussian, the DBM), not the free probability directly — the free probability gives the macroscopic / the limiting distribution, the RMT the finite-\(N\) fluctuations.
Refs¶
- Baik–Ben Arous–Péché, "Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices" (Ann. Probab. 2005)
- Johnstone, "On the distribution of the largest eigenvalue in principal components analysis" (Ann. Stat. 2001)
- Capitaine–Donati–Martin, "Additive perturbations of an eigenvalue of a random matrix" (the free probability outlier computation)
- Péché, "Asymptotic results for the largest eigenvalue of spiked covariance matrices" (Berthet–Rigollet, the sparse PCA); Krauthgamer–Nadler, the computational threshold
Next¶
RMT index · Foundations · Universality & Log-Gases · Determinantal & β-Ensembles · Non-Hermitian & Circular · Free probability · FP / subordination — the BBP outlier via the subordination · Analytic number theory