Research · MarkTechPost ·
Tilde Research Introduces Aurora: A Leverage-Aware Optimizer That Fixes a Hidden Neuron Death Problem in Muon
Tilde Research introduced Aurora, a leverage-aware optimizer for neural network training designed to address a flaw in Muon that can permanently deactivate a fraction of MLP neurons during training. The researchers also reported a 1.1B-parameter pretraining experiment and a new state-of-the-art result on one benchmark.