Muon preconditioning improves with structured gradient noise, not isotropic noise, according to a study of Muon in large language model training.
Read the original at arxiv.org→arXiv:2610.31728v1 Announce Type: new Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient...
Original headline: "When and How Does Muon Precondition? The Role of Structured Gradient Noise"
Coverage timeline
- Oct 26, 04:00 UTC arXiv cs.LG lead source When and How Does Muon Precondition? The Role of Structured Gradient Noise