Z-Loss backward geometry in dense output heads and sparse routers
Read the original at arxiv.org→arXiv:2609.16179v1 Announce Type: new Abstract: Z-loss has been widely applied to the logits of language-model output heads and sparse mixture-of-experts routers. Z-loss constrains the softmax log-normalizers of...
Original headline: "Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers"
Coverage timeline
- Sep 16, 04:00 UTC arXiv cs.LG lead source Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers