Small sparse MoE with 348M active parameters trained on ~11B tokens scores 85.5% on shuffled fact chains vs Qwen2.5-1.5B's 53.3%; evaluation uses GGUF and an eval set
Read the original at www.reddit.com→I've been recently working a small sparse MoE, and I've got interesting results I'd like strangers to invalidate. Tldr; I train small models and got surprising results and seek invalidation. It's not a general...
Original headline: "I trained a 348M-active MoE on ~11B tokens. It scores 85.5% vs Qwen2.5-1.5B's 53.3% on shuffled fact chains. GGUF + eval set — please try to break it."
Coverage timeline
- Nov 10, 17:11 UTC r/LocalLLaMA lead source I trained a 348M-active MoE on ~11B tokens. It scores 85.5% vs Qwen2.5-1.5B's 53.3% on shuffled fact chains. GGUF + eval set — please try to break it.