OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan
OmniOpt provides a unified meta-pipeline and taxonomy to navigate the fragmented landscape of modern LLM optimizers.
How can we organize the fragmented landscape of over 100 modern LLM optimizers into a unified taxonomy and benchmark them systematically?
Optimizer selection for large-scale models has become a complex system-level design choice, yet the literature remains fragmented across hundreds of methods with inconsistent naming and evaluation protocols. The authors introduce a universal meta-pipeline that treats every optimizer as a sequence of five operational stages, allowing them to map diverse methods into a shared coordinate system based on where they intervene in the update process. This framework reveals that most optimizers are sparse modifications of a common template, enabling a systematic benchmark that exposes trade-offs in convergence, memory, and stability across different model scales and architectures.
Paper Primer
The paper organizes optimizers using a dual-dimension taxonomy: a methodological dimension based on the meta-pipeline stages, and an objective-oriented dimension that records measurable training effects. By using linear minimization oracles (LMOs) as a geometric primitive, the authors unify disparate update directions—such as sign-based, matrix-structured, and adaptive-moment methods—into a single four-axis decomposition.
Optimizer rankings are not global; they exhibit systematic crossings based on model scale, context length, and training domain.
The benchmark study across 60M to 1B parameter models and context lengths up to 32k tokens shows that no single optimizer dominates the multi-objective frontier. Performance trade-offs are highly protocol-sensitive, with aggressive state compression excelling under short contexts but degrading as input complexity grows.
Structured-matrix methods, such as Muon, provide the most stable transfer across architectures and tasks, though they impose a substantial per-step compute cost compared to element-wise adaptive methods like AdamW.
Why does the field need a meta-pipeline rather than just comparing final loss values?
Optimizer papers often name methods after local mechanisms (e.g., a new preconditioner or quantizer) without revealing if they are compositionally compatible or where they intervene in the update process. The meta-pipeline exposes these intervention sites, allowing researchers to reason about which mechanisms can be stacked and which are redundant.
What is the "identity-mapping principle" mentioned in the framework?
It is the observation that most modern optimizers perform nontrivial work in only one or two of the five pipeline stages, leaving the remaining stages as identity maps or standard defaults. This principle allows for a compact characterization of complex optimizers by identifying their primary mechanism family.
Abstract
OmniOpt surveys, categorizes, and benchmarks modern optimizers to guide systematic selection.
Optimizer selection for large‑scale models has become a system‑level decision limited by compute, memory, tuning budget, and task diversity, while over a hundred methods remain fragmented. OmniOpt addresses this by (1) modeling each optimizer update as a five‑stage meta‑pipeline, (2) unifying methods via norm‑constrained linear minimization oracles (LMOs), (3) introducing a dual‑dimension taxonomy that maps mechanisms to training objectives, and (4) benchmarking the full taxonomy across domains, scales, and regimes. The result is an operational coordinate system that lets researchers choose optimizers based on explicit mechanism and objective assumptions.
The Optimizer Selection Problem
We frame optimizer choice as a system‑level design decision and lay out the paper’s unifying premise.
Optimizers now determine whether a large‑scale training run succeeds, because they must both reduce loss and respect fixed budgets of compute, memory, bandwidth, and tuning effort. While Adam and AdamW dominate classic deep learning, newer LLM‑focused methods—sign‑based, matrix‑level, low‑rank, and sharpness‑aware—break the single‑template pattern, turning optimizer selection into a simultaneous empirical, mathematical, and engineering problem.
Choosing an optimizer is no longer a plug‑in tweak; it shapes the entire training system.
We propose a universal meta‑pipeline that isolates where each optimizer intervenes, an LMO‑driven four‑axis decomposition that explains update directions, and a dual‑dimension taxonomy that maps mechanisms to training objectives. The benchmark evaluates over a hundred optimizers across six effect objectives, revealing systematic trade‑offs rather than a single winner.
**Figure 1.** Overview of the proposed survey and benchmark framework for a wild range of optimizers. The paper first introduces a universal meta-pipeline, then develops an LMO-driven four-axis decomposition, builds a process-aligned methodological taxonomy and an effect-objective taxonomy, and finally connects both taxonomies to a large-scale benchmark study.
Prior surveys and benchmarks treat optimizers either as isolated update rules or as performance tables, missing how methods align with the update pipeline or how their effects vary with scale and protocol. By unifying the meta‑pipeline, four‑axis analysis, and dual‑dimension taxonomy, our work bridges these gaps and enables systematic comparison.
**Figure 2.** Evolution of optimizer design for deep learning and LLM training. The expanding coverage of T1–T5 mechanism lanes and the diagonal trend arrow indicate increasing method count and mechanism diversity, not a performance ranking or exact method counts.
Treat optimizer selection as a system‑level design choice: the meta‑pipeline tells you where an optimizer acts, and the taxonomy tells you which training objectives it will impact.
Optimization Framework and Notation
Foundational optimizer concepts and a taxonomy of LLM optimizer families.
LLM optimizers range from identity‑preconditioned SGD to matrix‑structured curvature approximations; the field coalesces around a diagonal‑preconditioned Adam baseline, while scalable second‑order methods remain the primary gap.
These optimizers apply the raw (or smoothed) stochastic gradient directly to the parameters, using a global learning‑rate scalar. They assume an identity preconditioner, i.e. the geometry of the parameter space is Euclidean. Because they store at most a momentum buffer, they incur negligible memory overhead and trivial per‑step compute, making them trivially scalable to any model size.
These optimizers augment the raw gradient with per‑parameter second‑moment statistics, yielding a diagonal preconditioner that rescales each coordinate individually. Adam (and its variant AdamW) dominate modern LLM training because they capture coordinate‑wise curvature cheaply while keeping memory and compute modest.
These approaches approximate the full curvature matrix (Hessian or Fisher) with structured factorizations (Kronecker, low‑rank, eigen‑bases) or stochastic estimators, enabling a matrix‑valued preconditioner. They aim to capture inter‑parameter interactions that diagonal methods miss, at the cost of higher memory and compute.
This table provides a list of mathematical symbols and their corresponding meanings used in the paper.
The Universal Meta-Pipeline
The universal meta‑pipeline isolates optimizer interventions, revealing that most methods tweak only one stage.
Most modern optimizers modify only one or two stages of the universal meta‑pipeline, leaving the rest as identity maps.
The optimizer step is broken into a fixed sequence of stages ($S_0$–$S_5$); each method plugs in a non‑trivial operator at one or more stages while the others remain identity.
How does this meta‑pipeline differ from the usual “compute gradient → apply update” description?
The usual description hides where a method intervenes. The meta‑pipeline makes every possible intervention explicit as a stage, so you can see whether a method changes the signal, the routing, the transformation, the state, the reconstruction, or the final scaling.
Using this vocabulary we can map every surveyed optimizer onto the pipeline and see which stages they actually touch.
AdamW follows the default routing (all parameters share a single element‑wise route) and leaves $S_1$ and $S_2$ as identities. Its novelty lies in $S_3$, where first‑ and second‑order exponential moving averages are maintained, and in $S_5$, where decoupled weight decay is applied after the adaptive scaling.
Muon separates two‑dimensional weight matrices from vector parameters in $S_1$, then applies Newton–Schulz spectral orthogonalization in $S_2$ to produce a gradient with unit singular values. The state evolution remains a standard momentum buffer, and reconstruction is identity.
GaLore identifies low‑rank subspaces for eligible tensors in $S_1$, projects gradients onto an $r$‑dimensional basis in $S_2$, runs a full‑space Adam update inside that subspace ($S_3$), and lifts the update back to full space in $S_4$. All other stages stay identity.
Lion keeps the standard element‑wise routing ($S_1$ identity) but replaces the gradient in $S_2$ with the sign of a momentum‑interpolated direction, yielding a binary update vector. State evolution stores only a first‑order momentum ($S_3$); reconstruction is identity.
SAM augments the training signal in $S_0$ by adding an adversarial perturbation computed from the current gradient, then proceeds with the default element‑wise pipeline. The sharpness‑aware correction is realized entirely in $S_0$ and the final scaling/clipping in $S_5$.
**Figure 3.** Universal meta-pipeline for one optimizer step. The training system provides a gradient or higher-order signal (S0), which the optimizer processes through five internal stages for each parameter group.
LMO Foundations and Axes
Four‑axis LMO decomposition unifies optimizer updates and highlights a missing geometry axis.
All modern optimizers can be expressed as a four‑axis LMO decomposition, revealing a common geometric core while many methods omit the geometry axis, limiting flexibility.
The update is built by (1) picking a space, (2) estimating a signal, (3) turning that signal into a direction via an LMO or preconditioner, and (4) writing the direction back with learning‑rate and optional tricks.
How does this four‑axis view differ from the traditional view that lumps geometry and preconditioning together?
Traditional formulations treat the geometry of the constraint set and the preconditioner as a single step; the four‑axis view separates them into Axis III, letting you replace an LMO with a Hessian‑based preconditioner (or vice‑versa) while keeping the surrounding pipeline unchanged.
This camp treats the Euclidean ball $D_2$ as the constraint set. The LMO returns the normalized gradient, which many optimizers use implicitly (e.g., SGD with gradient clipping or unit‑norm scaling). The survey reports that this geometry underlies most first‑order methods.
Here the constraint set is the $\ell_\infty$ ball $D_\infty$, whose vertices are sign patterns. The LMO yields the sign of the gradient, giving a direction that ignores magnitude entirely. This is the core of sign‑based optimizers.
When the update domain is a matrix, the spectral‑norm ball $D_{S_\infty}$ leads the LMO to return the polar factor $UV^\top$ of the gradient’s SVD. This yields a direction that respects matrix geometry.
Adam adapts the LMO constraint set each step: the box $D^{\text{Adam}}_t$ scales per‑coordinate by $|m_{t,i}|/\sqrt{v_{t,i}}$. The LMO then returns a direction that is the element‑wise ratio of first‑ and second‑moment estimates.
Optimizers such as K‑FAC approximate the curvature with a Kronecker‑structured preconditioner. In the four‑axis view, the preconditioner lives in Axis III, replacing the LMO with a matrix‑free linear solve.
Axis Definitions and Instantiations
We map every modern LLM optimizer onto four orthogonal design axes, exposing a shared structure and a missing systematic taxonomy.
Across modern LLM optimizers, design choices collapse onto four orthogonal axes: where the update lives, how the optimizer state is estimated, how that state is turned into a direction, and how the direction is applied. Most methods already respect the Axis II–III split, but the literature offers no unified map, forcing practitioners to guess which combination fits their hardware or task.
The SGD family (plain SGD and SGDM) stays entirely in the original parameter space, uses no auxiliary statistics beyond a simple momentum buffer, and generates directions by a plain ℓ₂‑ball LMO. Finalization is trivial—no weight decay or projection steps are added beyond the learning‑rate scaling.
Adam‑style optimizers maintain both first‑ and second‑moment estimates, interpret the second moment as a diagonal preconditioner, and view the update as an adaptive ℓ∞‑box LMO. Decoupled weight decay lives in the finalization stage, making AdamW the de‑facto standard for LLM fine‑tuning.
Methods that treat weight matrices as first‑class objects (Shampoo, MARS‑Shampoo, Muon, SOAP) operate on $\\mathbb{R}^{m\\times n}$, keep matrix‑shaped momentum, and use either a Kronecker‑factored metric or a spectral‑norm LMO. The finalization stage is usually empty, relying on the geometry operator to embed the preconditioned direction directly.
Low‑rank methods (GaLore, Fira, APOLLO, Conda) first project the full‑space momentum into a small subspace of rank $r\\ll\\min(m,n)$, run a standard Adam‑style update inside that subspace, and then lift the direction back. The projection basis may be learned (GaLore, Fira) or random/column‑wise (APOLLO, Conda). Finalization includes the projection‑back step.
Sign‑based optimizers replace the magnitude of the update with a fixed unit step in the sign direction (SignSGD) or a momentum‑scaled sign (Lion). Their LMO is a fixed $\\ell_\\infty$ ball, and the preconditioner is a diagonal matrix built from absolute gradients or momentum.
Dual-Dimension Taxonomy
Optimizers fall into five mechanism families whose trade‑offs are captured by six effect objectives.
The survey reveals a clean split: five primary mechanism families (T1–T5) each map to a distinct set of claimed training effects (O1–O6), enabling systematic benchmark design.
This family retains the Adam‑style per‑parameter update pipeline but broadens beyond vanilla Adam. All members compute first‑order moments, second‑order variance surrogates, or related statistics and apply a diagonal preconditioner independently to each parameter. Variants differ in decay schedules, bias‑correction tricks, iterate averaging, or global step‑size policies, yet the core operation remains an element‑wise scaling of the gradient.
Methods in this family treat weight tensors as matrices and modify the update space before applying any element‑wise scaling. They introduce non‑trivial routing (S1) and transformation (S2) stages such as orthogonalization, Kronecker‑factored preconditioning, or low‑rank subspace projection. The optimizer therefore operates on a reshaped or compressed representation of the gradient.
The defining operation is a sign‑map or other coarse direction encoder that discards magnitude information. Lion exemplifies this: it forms a signed direction from a momentum‑gradient interpolation and updates parameters using only the sign. This fundamentally changes the geometry of the update direction (Axis I) while keeping state storage minimal.
Family members target the optimizer‑state bottleneck directly. They compress, quantize, or share state across layers, rows, or blocks, often using low‑bit representations or factorized storage. The update computation itself stays unchanged; only the representation of moments, buffers, or auxiliary statistics is altered.
These optimizers wrap a base update with an additional geometric operation near the write‑back stage. Examples include sharpness‑aware perturbations (SAM), layer‑wise trust‑ratio scaling (LAMB), and post‑update filters that enforce consistency or sparsity. The base optimizer (often T1 or T3) remains intact; the wrapper supplies a second‑order or regularization signal.
**Table 6.** Dimension-B effect objectives and measurement sources.
The cross‑dimension matrix makes composition decisions explicit: orthogonal subspace projection (T2) pairs naturally with state quantization (T4), while post‑update filters (T5) can wrap any base family without interfering with its primary mechanism.
T1: Element-wise Adaptive Moment
Element‑wise adaptive‑moment optimizers preserve per‑parameter update topology while extending AdamW.
T1 optimizers dominate modern LLM training because they keep the update element‑wise while offering a rich menu of numerator, denominator, and outer‑control tweaks.
The survey partitions all optimizers into five families (T1‑T5) according to which slot of the universal meta‑pipeline they modify.
Why does the survey keep T1 separate from T2 if both modify the optimizer state?
Because T1 never mixes information across parameters—its update matrix stays diagonal—whereas T2 explicitly changes the basis or introduces off‑diagonal coupling, which fundamentally alters the topology of the update.
AdamW is the canonical element‑wise adaptive‑moment optimizer that decouples weight decay from the adaptive direction.
Why does AdamW separate weight decay from the adaptive denominator instead of folding it into $v_t$?
Because coupling decay to $v_t$ would make the regularization strength depend on the gradient variance, leading to unpredictable scaling; decoupling ensures a consistent $L_2$ penalty regardless of the adaptive preconditioner.
**Figure 1.** Taxonomy of T1-based optimizers, categorized into three groups: T1.1 (Scalar Bases & Direct Adam Variants), T1.2 (Multi-Timescale & Momentum Correction & Variance Reduction), and T1.3 (Iterate Averaging & Automatic Tuning).
**Figure 4.** Mechanism overview of T1 element-wise adaptive-moment and scalar control methods. The family is organized by the dominant intervention in the update: direct scalar or Adam-style variants, additional temporal or estimator-correction channels, and outer iterate or global-scale control. Boundary methods are shown only when they preserve scalar or element-wise update topology.
This camp keeps the update strictly coordinate‑wise. All modifications stay local to the AdamW template: they alter the numerator, denominator, bias correction, or weight‑decay handling without mixing information across parameters.
Optimizers in this camp enrich the signal that feeds the AdamW state. They introduce fast/slow EMA channels, variance‑reduced estimators, or alignment‑aware momentum corrections, all while retaining the diagonal preconditioner.
This camp leaves the inner AdamW update untouched and instead modifies the outer control loop: iterate averaging, schedule‑free learning‑rate inference, or dynamic step‑size switching.
T2: Matrix-Level Structural Methods
Matrix‑level optimizers reshape updates by exploiting full weight‑matrix geometry.
Matrix‑level structural methods (T2) replace coordinate‑wise adaptation with geometry‑aware transforms that act on entire weight matrices.
The T2 family operates on the two‑dimensional weight matrix $W_t\in\mathbb{R}^{m\times n}$ and its stochastic gradient $G_t$, allowing rows, columns, singular directions, or low‑rank subspaces to be manipulated directly.
This camp builds an orthogonal or polar direction from the matrix momentum and applies it as the update. The central method Muon flattens singular values while preserving singular vectors; extensions such as Dion, AdaMuon, and OrthoGrad add routing or adaptive scaling.
This camp estimates row‑ and column‑wise curvature (or Fisher) factors and preconditions the gradient with inverse‑root transforms. Shampoo and SOAP are the canonical representatives; Shampoo stores $L_t$ and $R_t$ statistics, while SOAP adds Adam‑style adaptation in the Shampoo eigenbasis.
This camp projects gradients into a low‑dimensional subspace, runs a standard adaptive optimizer there, and reconstructs a full‑matrix update. GaLore is the canonical LLM example; Fira and Alice extend the idea with residual pathways or structured‑Fisher bases.
**Figure 6.** Mechanism schematic for T2 matrix-level structural methods. The schematic summarizes the three matrix routes used in this survey: spectral direction selection, Kronecker or structured-Fisher preconditioning, and low-rank subspace projection. It is a taxonomy guide rather than an empirical ranking.
**Figure 7.** Taxonomy of matrix-level structural optimizers.
**Table 8.** Mechanism-informed effect assessment for T1 subclasses. The entries are design priors for benchmark planning, not final empirical conclusions.
T3: Discretization and Quantization
Discretizing update directions yields cheap state, fast steps, and distinct robustness trade‑offs.
T3 optimizers replace continuous magnitudes with sign‑like directions, cutting memory and compute while exposing new sensitivity to learning‑rate tuning.
This camp applies a hard sign map directly to the raw gradient (SignSGD) or to a momentum‑smoothed signal (Signum). The update discards magnitude entirely, transmitting only a 1‑bit direction per coordinate. Historically motivated by communication efficiency in distributed training, the methods expose the extreme limit of direction discretization.
Lion and its variants keep a single momentum buffer, interpolate it with the current gradient, and then apply a hard sign map. This preserves the sign‑direction core while re‑using AdamW‑style scaffolding (weight decay, learning‑rate schedule). The removal of the second‑moment buffer cuts persistent state roughly in half.
This camp keeps the sign‑direction backbone but relaxes the hard discontinuity or adds auxiliary channels. RLion replaces the step function with a bounded smooth map, FOCUS adds a trajectory‑attraction term, and Ano restores a limited magnitude channel while preserving the sign core. These hybrids aim to recover stability or expressivity lost in pure sign updates.
**Table 9.** Mechanism-informed effect assessment for T2 subclasses. The entries are design priors for benchmark planning, not empirical conclusions.
**Figure 8.** Mechanism schematic for T3 discretization and directional quantization. This schematic provides a taxonomy guide rather than an empirical ranking for the compact T3 family, grouping discretization and directional quantization mechanisms by their sign-direction generation methods.
T4 and T5 Families
State‑compression optimizers trade memory for approximation, while geometry‑aware optimizers reshape updates for stability.
T4 families shrink optimizer state (memory, bandwidth, lifetime) to enable larger models; T5 families impose curvature or geometric constraints to improve stability or generalization.
This camp groups methods that alter the representation, precision, or lifetime of the adaptive statistics used by Adam‑style optimizers. By compressing, sharing, quantizing, or consuming gradients early, they reduce the persistent memory footprint (often from $O(md)$ to $O(m\!+\!d)$ or lower) and free bandwidth for larger batch sizes or longer sequences. The survey reports four subclasses: row‑column factorization (AdaFactor, CAME), low‑bit quantization (8‑bit Adam, Q‑GaLore), block/layer‑level sharing (Adam‑mini, APOLLO, SM3, NovoGrad, Conda), and fused back‑prop update (LOMO, AdaLOMO). Each achieves memory reduction at the cost of approximation error, quantization noise, or altered gradient‑lifetime semantics.
This camp gathers optimizers that modify the geometry of the candidate update rather than the optimizer’s internal state. Methods either perturb the evaluation point (SAM family), estimate diagonal curvature to clip updates (Sophia, AdaHessian), filter or mask the base direction (Gradient Centralization, AdamP, C‑AdamW), or apply layer‑wise trust‑ratio scaling (LAMB, AGC). The goal is to improve stability, generalization, or large‑batch scaling by enforcing flatness, curvature‑aware step sizing, or norm‑consistent updates.
**Figure 10.** Mechanism schematic for T4 state compression and structural aggregation. The schematic organizes the family by the form of memory reduction: factored second-moment storage, low-bit state representation, shared adaptive statistics, and streaming gradient consumption.
**Figure 11.** Taxonomy of state-compressed and structurally aggregated optimizers.
**Figure 12.** Mechanism schematic for T5 curvature-aware and geometric regularization methods. The schematic organizes the family by where the geometric intervention enters the update: perturbed-gradient evaluation, diagonal-curvature estimation, post-update filtering, or layer-wise trust-region scaling.
**Figure 13.** Taxonomy of curvature-aware and geometry-regularized optimizers. The prefix C- denotes Cautious.
**Table 10.** Mechanism-informed effect assessment for T3 methods. The entries are design priors for benchmark planning, not empirical conclusions.
**Table 11.** Mechanism-informed effect assessment for T4 subclasses. The entries are design priors for benchmark planning, not empirical conclusions.