MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts
Read the original at www.reddit.com→Looking for some assistance /ideation. I am running qwen flash next q4 in my Mac mini m5 64gb. QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd....
Original headline: "MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?"
Coverage timeline
- Oct 5, 10:50 UTC r/LocalLLaMA lead source MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?