Budget upgrade options for a 31B model: PCIe x1 bottleneck and GPU setup considerations for 16K+ context inference
Read the original at www.reddit.com→Looking for advice on upgrading my AI server on a budget. My goal is to run Gemma 4 31B Q5_K_M (~23GB) at 16K+ context (q8), achieving at least 15 t/s. I am aware of the 26B MoE, but I want the dense model after...
Original headline: "Budget upgrade for a 31B model? Dual GPUs over PCIe x1 or another approach?"
Coverage timeline
- Oct 20, 02:50 UTC r/LocalLLaMA lead source Budget upgrade for a 31B model? Dual GPUs over PCIe x1 or another approach?