QFN Q_2 strata configs for 8gb vram and 32gb ram; 36 tps and 340 prefill 176128 context
Read the original at www.reddit.com→"--expert-cache", "2000", "--prefill", "auto", "--spec", "2", "--max-context", "176128", "--kv", "q4_0", "--resident-experts", "--kv-resident", "20480", "--pcie-frac", "0.00", "--spec-min-p", "0.70", "--adapt-every",...
Original headline: "QFN Q_2 strata configs for 8gb vram and 32gb ram. 36 tps | 340 prefill 176128 context."
Coverage timeline
- Oct 14, 11:54 UTC r/LocalLLaMA lead source QFN Q_2 strata configs for 8gb vram and 32gb ram. 36 tps | 340 prefill 176128 context.