Lion-K CCWD: Corrected Cautious Weight Decay and Hyperparameter Transfer
Derivation of Lion-K with Corrected Cautious Weight Decay (CCWD) and transformation rules for hyperparameter transfer.
Post
Transformers as Constrained Optimization
Rewriting a pre-norm decoder-only transformer as a mixed-geometry constrained splitting scheme: RMSNorm as radial gauge fixing, attention as an entropy- or KL-constrained simplex solve, and residual branches as Euclidean trust-region steps.
Optimizers and ODEs
A continuous-time view of gradient-based optimization: starting from the observation that integrator choice matters in physics simulation, and transferring that insight to understand modern optimizers.
Frequency-Domain Muon for Conv Filters - Orthogonalizing the Operator
A technical note on extending Muon to orthogonalize convolution operators in the frequency domain, moving beyond simple reshaped weight projections.
Discrete Calculus
An introduction to Discrete Calculus, a theory for sums and differences of sequences as opposed to derivatives and integrals of functions in infinitesimal calculus.
Introduction to Stochastic Calculus
A beginner-friendly introduction to stochastic calculus, focusing on intuition and calculus-based derivations instead of heavy probability theory formalism.
Automatic Stochastic Differentiation
Introduction Calculus can be quite tedious when computed symbolically by hand. In many modern applications (for example, in machine learning), automatic differentiation is used to efficiently comp...
Deriving Reverse-Time Stochastic Differential Equations (SDEs)
A detailed derivation of the reverse-time stochastic differential equation used in Score-Based Generative Modeling.