LlamaAmpere v0.6 achieves ~13% TPS increase with 112 TPS average across 100k–105k tokens on 3 tasks; prompts process at ~1800 TPS and sustain 1000+ through first 100k, with prefill average ~1,300 TPS
Read the original at www.reddit.com→Hello everyone! As always, thank you for sharing you previous results (and bugs) on the last version. I am happy to report that LlamAmpere v0.6 is now out. The headline items for you (at temperature 1): a ~13% jump...
Original headline: "110+ TPS through 100K tokens with 27b on a single 3090, with 400k ctx"
Coverage timeline
- Oct 28, 14:33 UTC r/LocalLLaMA lead source 110+ TPS through 100K tokens with 27b on a single 3090, with 400k ctx