DSv4-Flash optimized for 2x GH200 achieves 10,000 tok/s PP and >300 tok/s TG on SGLang
Read the original at old.reddit.com→There are some PRs to use and a nice trick to speed up PP on really longs contexts in my write up. Hope it helps! TL;DR: On this dual GH200 box, you build vLLM v0.26.0 from source, add the merged DSV4 cache-layout...
Original headline: "Optimised DSv4-Flash for 2x GH200: 10,000 tok/s PP, >300 tok/s TG on SGLang"
Coverage timeline
- Aug 4, 10:08 UTC r/LocalLLaMA lead source Optimised DSv4-Flash for 2x GH200: 10,000 tok/s PP, >300 tok/s TG on SGLang