Qwen-flash-next Q4 with Expert cache on Llama.cpp setup; 19-21 t/s at 262K context using Qwen Flash Next Q4 model
Read the original at www.reddit.com→I just setup my server: 64GB DDR4 2133 4CH (Broadwell Xeon) 2 x RX 6800 16GB 1 x Rx 580 2048sp (16GB) I am using the Q4 of Qwen Flash Next, with the prompt: sudo ./llama-server -m ./models/Qwnflash180b-Q4.gguf --host...
Original headline: "Llama.cpp setup Qwen-flash-next Vulkan Q4 with Expert cache."
Coverage timeline
- Nov 1, 01:39 UTC r/LocalLLaMA lead source Llama.cpp setup Qwen-flash-next Vulkan Q4 with Expert cache.