Muon makes the first mixture-of-experts layer harder to balance
Read the original at arxiv.org→arXiv:2610.37914v1 Announce Type: new Abstract: Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after...
Original headline: "Muon Makes the First Mixture-of-Experts Layer Harder to Balance"
Coverage timeline
- Oct 30, 04:00 UTC arXiv cs.LG lead source Muon Makes the First Mixture-of-Experts Layer Harder to Balance