35B MoE runs at 20.5 tok/s on an iPhone without skipping experts, using Lookahead Expert Scheduling to load expert weights from flash storage for local execution
Read the original at www.reddit.com→I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real...
Original headline: "35B MoE running at 20.5 tok/s on an iPhone, without skipping experts"
Coverage timeline
- Oct 21, 13:12 UTC r/LocalLLaMA lead source 35B MoE running at 20.5 tok/s on an iPhone, without skipping experts