4.5. Probability and Statistics (PST)

4.5. Probability and Statistics (PST)

This area covers the mathematical theory of uncertainty and inference, from the axiomatic foundations of probability to modern statistical methodology including Bayesian inference, high-dimensional statistics, and time series analysis.

Table 4.5: List of KUs in the Probability and Statistics area.

4.5.1. PST/Probability Axioms and Distributions  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Kolmogorov's axiomatic foundation of probability, random variables, distributions, and expectations.
Topics:
Core

  • Kolmogorov axioms, sigma-algebras, and probability spaces
  • Random variables: discrete and continuous distributions, CDF, PDF, and PMF
  • Expectation, variance, moments, and moment generating functions
  • Joint, marginal, and conditional distributions; independence and covariance
  • Standard families: Binomial, Poisson, Normal, Exponential, Gamma, and Beta

Learning Outcomes:
Core:

  1. State Kolmogorov's axioms and construct a probability space for a standard experiment [Familiarity]
  2. Compute expectations, variances, and moment generating functions for standard distributions [Usage]
  3. Determine joint and conditional distributions and assess independence of random variables [Assessment]

4.5.2. PST/Limit Theorems  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Laws of large numbers, central limit theorem, modes of convergence, and large deviations.
Topics:
Core

  • Modes of convergence: almost sure, in probability, in \(L^p\), and in distribution
  • Weak and strong laws of large numbers and their applications
  • Central Limit Theorem: statement, proof via characteristic functions, and Berry-Esseen bound
  • Characteristic functions and the continuity theorem (Lévy)
  • Large deviations: Cramér's theorem and the Gärtner-Ellis theorem

Learning Outcomes:
Core:

  1. Distinguish between modes of convergence of random variables and provide counterexamples showing their differences [Familiarity]
  2. Apply the Central Limit Theorem to approximate distributions of sums and means [Usage]
  3. Prove the strong law of large numbers and estimate large deviation probabilities using Cramér's theorem [Assessment]

4.5.3. PST/Stochastic Processes  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Markov chains, Poisson processes, Brownian motion, martingales, and stochastic calculus.
Topics:
Core

  • Discrete-time Markov chains: classification of states, stationary distributions, and ergodic theorem
  • Poisson processes: definition, properties, and generalizations (compound, inhomogeneous)
  • Brownian motion: definition, path properties, and the reflection principle
  • Martingales, stopping times, optional sampling theorem, and Doob's inequalities
  • Itô integral, Itô's formula, and stochastic differential equations

Learning Outcomes:
Core:

  1. Classify states of a Markov chain and determine its stationary distribution [Familiarity]
  2. Apply martingale theory and the optional sampling theorem to solve stopping time problems [Usage]
  3. Formulate and solve a stochastic differential equation using Itô's formula [Assessment]

4.5.4. PST/Estimation Theory  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Point and interval estimation: sufficiency, unbiasedness, efficiency, maximum likelihood, and asymptotic theory.
Topics:
Core

  • Sufficient statistics, the factorization theorem, and completeness
  • Unbiased estimators, the Cramér-Rao lower bound, and UMVUE
  • Maximum likelihood estimation: properties, consistency, and asymptotic normality
  • Confidence intervals: pivotal quantities, likelihood ratio, and bootstrap methods
  • Asymptotic theory: delta method, M-estimators, and the Fisher information matrix

Learning Outcomes:
Core:

  1. Explain the Cramér-Rao bound and identify when a UMVUE exists using the Lehmann-Scheffé theorem [Familiarity]
  2. Derive maximum likelihood estimators and compute their asymptotic variance [Usage]
  3. Construct confidence intervals using pivotal quantities and bootstrap resampling [Assessment]

4.5.5. PST/Hypothesis Testing  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Neyman-Pearson framework, uniformly most powerful tests, likelihood ratio tests, and multiple testing.
Topics:
Core

  • Neyman-Pearson lemma: most powerful tests for simple hypotheses
  • Uniformly most powerful (UMP) tests and monotone likelihood ratio families
  • Generalized likelihood ratio tests and Wilks' theorem
  • \(p\)-values, multiple testing, and false discovery rate (FDR) control
  • Nonparametric tests: sign test, Wilcoxon rank-sum, and Kolmogorov-Smirnov

Learning Outcomes:
Core:

  1. State the Neyman-Pearson lemma and explain the trade-off between type I and type II errors [Familiarity]
  2. Apply the likelihood ratio test and Wilks' theorem to composite hypothesis problems [Usage]
  3. Control the false discovery rate in a multiple testing scenario using the Benjamini-Hochberg procedure [Assessment]

4.5.6. PST/Bayesian Inference  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Bayesian paradigm: prior and posterior distributions, conjugate families, hierarchical models, and computational methods.
Topics:
Core

  • Bayes' theorem in continuous settings; prior and posterior distributions
  • Conjugate prior families: Beta-Binomial, Dirichlet-Multinomial, Normal-Normal
  • Hierarchical Bayesian models and empirical Bayes
  • Markov chain Monte Carlo (MCMC): Metropolis-Hastings and Gibbs sampling
  • Variational inference and the evidence lower bound (ELBO)

Learning Outcomes:
Core:

  1. Apply Bayes' theorem to update a prior with observed data and obtain a posterior distribution [Familiarity]
  2. Construct a hierarchical Bayesian model and derive its joint and conditional distributions [Usage]
  3. Implement Gibbs sampling or Metropolis-Hastings to approximate a posterior distribution [Assessment]

4.5.7. PST/High-Dimensional Statistics ↑ Back to top

Statistical theory in high dimensions: concentration inequalities, sparsity, LASSO, and random matrix theory.
Topics:
Non Core

  • Concentration inequalities: Markov, Chebyshev, Chernoff bounds, and sub-Gaussian variables
  • Sparse estimation: LASSO, ridge regression, and the restricted isometry property (RIP)
  • Random matrix theory: Wigner semicircle law and the Marchenko-Pastur distribution
  • Compressed sensing: exact recovery conditions and convex relaxation
  • Minimax lower bounds and the bias-variance trade-off in high dimensions

Learning Outcomes:
NonCore:

  1. Apply concentration inequalities to bound the deviation of sums of random variables [Familiarity]
  2. Analyze the statistical properties of LASSO estimators under sparsity and RIP conditions [Usage]
  3. Derive minimax lower bounds for estimation problems using information-theoretic methods [Assessment]

4.5.8. PST/Time Series Analysis  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Stationary processes, ARMA models, spectral analysis, state-space models, and forecasting.
Topics:
Core

  • Stationarity, autocovariance function, and the autocorrelation function (ACF)
  • ARMA models: identification, estimation (Yule-Walker, MLE), and diagnostics
  • Spectral density, the periodogram, and Wiener-Khinchin theorem
  • State-space models and the Kalman filter
  • ARIMA models, seasonal adjustment, and multi-step forecasting

Learning Outcomes:
Core:

  1. Identify stationarity and determine the order of an ARMA model from ACF and PACF plots [Familiarity]
  2. Fit ARIMA models to time series data, validate residuals, and produce forecasts [Usage]
  3. Apply the Kalman filter to estimate hidden states in a linear Gaussian state-space model [Assessment]

4.5.9. PST/Nonparametric Methods  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Distribution-free inference, rank-based tests, kernel density estimation, and nonparametric regression.
Topics:
Core

  • Empirical distribution function, the Glivenko-Cantelli theorem, and the bootstrap
  • Rank-based tests: Wilcoxon, Kruskal-Wallis, and Spearman correlation
  • Kernel density estimation: bandwidth selection, bias-variance trade-off
  • Nonparametric regression: Nadaraya-Watson estimator and smoothing splines
  • Permutation tests and randomization inference

Learning Outcomes:
Core:

  1. Explain the advantages and limitations of nonparametric methods compared to parametric alternatives [Familiarity]
  2. Apply kernel density estimation and select bandwidth using cross-validation [Usage]
  3. Conduct permutation and rank-based tests and interpret their results for inference without distributional assumptions [Assessment]

Spotted a typo, an outdated course, a broken link, or have a suggestion? Let us know.

Scan to open on your phone