7.61. Statistical Learning Theory (Elective)

7.61. Statistical Learning Theory (Elective)

Figure 7.61: Connection Map. MA355 Statistical Learning Theory

7.61.1. Justification ↑ Back to top

This course provides the theoretical foundations of statistical learning, covering the mathematical principles that support machine learning algorithms. It focuses on the theoretical analysis of generalization capability, model complexity, and performance guarantees for various learning methods. The course connects theory with practice through the study of generalization bounds, bias-variance tradeoff, and foundations of computational learning theory.

7.61.2. Generales Goals ↑ Back to top

  1. Understand the fundamentals of statistical learning theory and its probabilistic foundations.
  2. Analyze the bias-variance tradeoff in the selection and evaluation of machine learning models.
  3. Apply Vapnik-Chervonenkis (VC) theory to analyze model complexity and generalization capability.
  4. Develop and demonstrate generalization bounds for different learning algorithms.
  5. Connect theoretical principles with the practical implementation of statistical learning algorithms.

7.61.3. Contribution to Outcomes ↑ Back to top

ABET-1) An ability to identify, formulate, and solve broadly defined technical or scientific problems by applying knowledge of mathematics and science and/or technical topics to areas relevant to the discipline. (Usage)
ABET-3) An ability to develop and conduct experiments or test hypotheses, analyze and interpret data, and use scientific judgment to draw conclusions. (Usage)

7.61.4. Content ↑ Back to top

7.61.4.1. Foundations of Statistical Learning (20 hours) [Skills ABET-1,ABET-3] ↑ Back to top

Bibliography: (Vapnik, 1998; Hastie et al., 2009)

Topics

  1. Bias-variance tradeoff
  2. Vapnik-Chervonenkis (VC) theory
  3. Empirical risk minimization
  4. Generalization bounds
  5. Learning theory fundamentals

Learning Outcomes

  1. Analyze the bias-variance tradeoff in model selection [Assessment]
  2. Apply VC theory to analyze model complexity [Usage]
  3. Implement empirical risk minimization [Usage]
  4. Interpret generalization bounds [Familiarity]
7.61.4.2. Supervised Learning Methods (20 hours) [Skills ABET-1,ABET-3] ↑ Back to top

Bibliography: (Hastie et al., 2009; James et al., 2013)

Topics

  1. Linear models and extensions
  2. Support Vector Machines (SVM)
  3. Decision trees and ensemble methods
  4. Kernel methods
  5. Probabilistic supervised learning

Learning Outcomes

  1. Implement and analyze linear models [Usage]
  2. Apply SVM for classification and regression [Usage]
  3. Construct ensemble methods (bagging, boosting) [Usage]
  4. Use kernel methods for nonlinear problems [Usage]
7.61.4.3. Unsupervised Learning Methods (20 hours) [Skills ABET-1,ABET-3] ↑ Back to top

Bibliography: (Murphy, 2012; Bishop, 2006)

Topics

  1. Clustering algorithms (k-means, hierarchical)
  2. Dimensionality reduction (PCA, t-SNE)
  3. Density estimation
  4. Manifold learning
  5. Autoencoders and deep unsupervised learning

Learning Outcomes

  1. Apply clustering algorithms to data exploration [Usage]
  2. Implement PCA and other dimensionality reduction methods [Usage]
  3. Use density estimation for unsupervised learning [Usage]
  4. Analyze manifold learning techniques [Assessment]
7.61.4.4. Model Evaluation and Selection (20 hours) [Skills ABET-1,ABET-3] ↑ Back to top

Bibliography: (Hastie et al., 2009; James et al., 2013)

Topics

  1. Cross-validation strategies
  2. Performance metrics for classification and regression
  3. Model comparison and statistical tests
  4. Hyperparameter optimization
  5. Learning curves and diagnostic tools

Learning Outcomes

  1. Implement cross-validation for model assessment [Usage]
  2. Select appropriate performance metrics [Assessment]
  3. Compare models using statistical tests [Usage]
  4. Optimize hyperparameters using systematic methods [Usage]

7.61.5. Bibliography ↑ Back to top

Vapnik, V. N. (1998). Statistical Learning Theory. Wiley, 1st edition.

Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition.

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning: with Applications in R. Springer, 1st edition.

Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press, 1st edition.

Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer, 1st edition.

Spotted a typo, an outdated course, a broken link, or have a suggestion? Let us know.

Scan to open on your phone