Converting dense models into sparse Mixture-of-Experts without pretraining from scratch; author experiments on Qwen/Qwen2.5-0 with RTX 4060.

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch. The Idea If you can turn a dense model into an MoE that only runs...