Differential Geometry – A Crash Course for Machine Learning

Why Differential Geometry for Machine Learning?

Modern machine learning models, especially deep neural networks, involve optimizing functions over very high-dimensional parameter spaces. These spaces often possess a rich geometric structure that Euclidean geometry alone cannot capture. Understanding this “shape” of the loss landscape is crucial for:

• Designing more efficient optimization algorithms (e.g., natural gradient).
• Analyzing the behavior of existing optimizers.
• Understanding challenges like saddle points, flat regions, and sharp minima.
• Characterizing the generalization properties of models.

Differential geometry provides the mathematical language and tools to study these curved spaces (manifolds) and the functions defined on them.

Goal of This Crash Course

This crash course aims to:

• Introduce the fundamental concepts of differential geometry: manifolds, tangent spaces, Riemannian metrics, connections, and curvature.
• Build intuition for these concepts, emphasizing their relevance to machine learning.
• Provide the necessary background to understand how geometric perspectives are used in advanced ML optimization theory (as discussed in the main optimization series).

This is a “crash course,” so our focus will be on breadth and intuition rather than deep theoretical proofs. We want to equip you with the vocabulary and core ideas to appreciate the geometric viewpoint in ML.

Prerequisites

To get the most out of this crash course, you should be comfortable with:

• Multivariable Calculus: Partial derivatives, gradients, Hessians, chain rule, line and surface integrals.
• Linear Algebra: Vector spaces, bases, linear transformations, matrices, determinants, eigenvalues/eigenvectors, inner products.
• Tensor Calculus (recommended): Basic understanding of tensors, index notation, and tensor operations (as covered in the prerequisite Tensor Calculus crash course).
• Mathematical Maturity: Comfort with abstract definitions and mathematical reasoning.

Course Outline

This crash course is divided into the following parts:

1. Part 1: Smooth Manifolds and Tangent Spaces – The Landscape of Parameters
   • What are manifolds? Why are they relevant to ML?
   • Tangent spaces: The space of possible “directions” (like gradients) at a point.
   • Vector fields: How directions change across the manifold.
2. Part 2: Riemannian Metrics and Geodesics – Measuring and Moving on Manifolds
   • Defining distances and angles on manifolds (Riemannian metrics).
   • The Fisher Information Metric: A natural geometry for statistical models.
   • Geodesics: The “straightest” paths on a manifold, and their connection to optimization.
3. Part 3: Curvature and Connections – How Manifolds Bend and Twist
   • Covariant derivatives and parallel transport: Differentiating consistently on curved spaces.
   • Curvature: Quantifying the “bending” of a manifold and its implications for loss landscapes (e.g., saddle points, minima).

We recommend going through these parts sequentially, as they build upon each other.