Trained a 21B MoE (3.36B active) from scratch on 70B tokens; with SuperBPE efficiency ≈ equivalent to 100B tokens, and a stable checkpoint released for continuation
Read the original at www.reddit.com→Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable...
Original headline: "I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)"
Coverage timeline
- Oct 28, 13:43 UTC r/LocalLLaMA lead source I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)