Nvidia+AMD RPC offload increases context to 195k on 2x 4060 Tensor Parallel with no perf cost
Read the original at www.reddit.com→Mixing Nvidia+AMD hardware via RPC usually meant split-mode layer only. I managed to keep split-mode tensor working, and patch my llama-rpc to send the drafters over to my strix. That let me go from 150k -> 195k...
Original headline: "The space is moving quick. 2x 4060 Tensor Parallel main weights. RPC offload the drafters to amd strix. Got +2-3GB Vram for context at no perf cost."
Coverage timeline
- Oct 15, 16:43 UTC r/LocalLLaMA lead source The space is moving quick. 2x 4060 Tensor Parallel main weights. RPC offload the drafters to amd strix. Got +2-3GB Vram for context at no perf cost.