2.6. Optimization and Machine Learning (OML)

2.6. Optimization and Machine Learning (OML)

This area covers the mathematical foundations and computational methods for optimization, with special emphasis on techniques relevant to machine learning. It includes convex and non-convex optimization, constrained optimization, large-scale optimization methods, and metaheuristics. The area bridges traditional optimization theory with modern machine learning applications, providing the tools needed to train complex models and solve large-scale optimization problems efficiently.

Table 2.6: List of KUs in the Optimization and Machine Learning area.

2.6.1. OML/Convex and Non-Convex Optimization  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Topics:
Core

  • Convex sets and functions
  • Optimality conditions
  • Gradient methods for convex optimization
  • Non-convex optimization challenges
  • Convergence analysis

Learning Outcomes:
Core:

  1. Characterize convex optimization problems [Assessment]
  2. Apply optimality conditions to verify solutions [Usage]
  3. Implement gradient-based methods for convex problems [Usage]
  4. Analyze challenges in non-convex optimization [Assessment]

2.6.2. OML/Constrained Optimization  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Topics:
Core

  • Linear programming (simplex method)
  • Integer programming
  • Lagrange multipliers and KKT conditions
  • Penalty and barrier methods
  • Duality theory

Learning Outcomes:
Core:

  1. Solve linear programming problems [Usage]
  2. Formulate and solve integer programming problems [Usage]
  3. Apply KKT conditions to constrained problems [Usage]
  4. Use penalty methods for constrained optimization [Usage]

2.6.3. OML/Optimization for Machine Learning  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Topics:
Core

  • Stochastic gradient descent
  • Momentum and adaptive methods (Adam, RMSProp)
  • Regularization and sparsity
  • Distributed optimization for ML
  • Second-order methods for ML

Learning Outcomes:
Core:

  1. Implement stochastic gradient descent variants [Usage]
  2. Apply adaptive optimization methods [Usage]
  3. Use regularization for model generalization [Usage]
  4. Design distributed optimization strategies [Usage]

2.6.4. OML/Large-Scale and Distributed Optimization  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Topics:
Core

  • First-order methods for large-scale problems
  • Coordinate descent and block methods
  • Consensus and distributed optimization
  • Federated learning optimization
  • Communication-efficient distributed optimization

Learning Outcomes:
Core:

  1. Apply first-order methods to large-scale problems [Usage]
  2. Implement coordinate descent algorithms [Usage]
  3. Design consensus-based distributed algorithms [Usage]
  4. Optimize federated learning systems [Usage]

2.6.5. OML/Metaheuristics and Bio-Inspired Optimization  (Core Tier1: 1 hr, Core Tier2: 1 hr) ↑ Back to top

Topics:
Core

  • Genetic algorithms and evolutionary strategies
  • Swarm intelligence (PSO, ant colony)
  • Simulated annealing
  • Tabu search
  • Hybrid metaheuristics

Learning Outcomes:
Core:

  1. Implement genetic algorithms [Usage]
  2. Apply swarm intelligence methods [Usage]
  3. Use simulated annealing for global optimization [Usage]
  4. Design hybrid metaheuristic approaches [Usage]

Spotted a typo, an outdated course, a broken link, or have a suggestion? Let us know.

Scan to open on your phone