Preface
Preliminary information for the blog post series on machine learning and optimization.
Collection-landing
Optimization Theory for Machine Learning
Dive into the mathematical underpinnings of optimization algorithms in machine learning. This series explores core theory, algorithm design, and connections to diverse mathematical fields, moving from fundamental concepts to advanced topics.
Introduction to Basic Mathematical Optimization
An introduction to the core ideas of mathematical optimization and their indispensable role in machine learning. We use linear regression to build intuition ...
Iterative Methods: Gradient-Free vs. Gradient-Based Optimization
An introduction to iterative optimization methods, differentiating between gradient-free and gradient-based approaches, their principles, pros, cons, and app...
Desirable Properties of Optimizers
A discussion of the key desirable properties for optimization algorithms in machine learning, covering effectiveness, efficiency, robustness, invariance, pra...
Speedrun of Common Gradient-Based ML Optimizers
A quick tour of popular gradient-based optimization algorithms in machine learning, detailing their mechanics and empirical performance characteristics.
Problem Formalization - First Principles and Modern Perspectives in ML Optimization
Establishing the mathematical framework for machine learning optimization, integrating foundational principles with modern generalization theory to define th...
Gradient Descent and Gradient Flow
Exploring the mathematical foundations of gradient descent, its continuous analogue gradient flow, and their connections.
Challenges of High-Dimensional Non-Convex Optimization in Deep Learning
Analyzing why non-convex, high-dimensional loss landscapes in deep learning defy classical optimization intuition yet remain optimizable.
Stochastic Gradient Descent: Noise as a Design Feature
How SGD's inherent randomness creates implicit regularization, escapes local minima, and shapes generalization - setting the stage for soft inductive biases.
Soft Inductive Biases: Improving Generalization
A deep dive into soft inductive biases, focusing on how regularization techniques and optimization dynamics guide machine learning models, particularly in de...
Adaptive Methods and Preconditioning: Reshaping the Landscape
Exploring how adaptive methods and preconditioning reshape optimization problems for faster convergence, from classical techniques to matrix-free innovations...
Momentum: Second-Order Gradient Flow ODE and a Multi-Step Method
Delving into the momentum optimization method, understanding its origins from both second-order 'heavy ball' dynamics and as a linear multi-step method for f...
Adam Through the Lens of Information Geometry: A Diagonal Fisher Approximation
Exploring Adam as an approximation to natural gradient descent using the diagonal empirical Fisher Information Matrix, and improvements proposed by FAdam.
Adam Optimizer: Online Learning of Updates and Efficacy with EMA
Exploring novel theoretical understandings of the Adam optimizer through the lens of Follow-The-Regularized-Leader for its updates, and the provable benefits...
Metrized Deep Learning: From Geometric Intuition to Principled Preconditioning with Muon and PolarGrad
Exploring how choosing the right norm for parameter spaces (like dimension-agnostic operator norms) and principled preconditioning (like PolarGrad) can revol...
Parameter-Free Optimization: Letting Algorithms Tune Themselves
Exploring optimization algorithms that dynamically adapt to the problem's structure, reducing or eliminating the need for manual hyperparameter tuning.