Nested sparsity for scalable pretraining and continual learning
Read the original at arxiv.org→arXiv:2610.31818v1 Announce Type: new Abstract: Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model...
Original headline: "Nested Sparsity for Scalable Pretraining and Continual Learning"
Coverage timeline
- Oct 26, 04:00 UTC arXiv cs.LG lead source Nested Sparsity for Scalable Pretraining and Continual Learning