DSv4 inference on 32GB RAM achieves 70 tok/s prefill and 1.5 tok/s decode, with read speed as limiter and optimizations for sequential reads, pipelining, and speculative prefetch at prefill time.
Read the original at old.reddit.com→The past week I've been running DSv4 inference on my laptop by keeping everything RAM-resident except the MXFP4-experts (since expert pool is ~147GB and won't fit) TL;DR - read speed is the limiter more than the...
Original headline: "DSv4-0731 on 32gb RAM - 70tok/s prefill, 1.5tok/s decode"
Coverage timeline
- Aug 9, 09:53 UTC r/LocalLLaMA lead source DSv4-0731 on 32gb RAM - 70tok/s prefill, 1.5tok/s decode
- Aug 9, 10:14 UTC r/LocalLLaMA 300b on 32gb MoE-streaming findings + optimisations