moe-cache: LRFU/MQ-like hybrid cache, ClassLRU for llama.cpp improves decode and reduces uploads vs SLRU
Read the original at www.reddit.com→Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history. With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads....
Original headline: "moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp"
Coverage timeline
- Nov 6, 15:39 UTC r/LocalLLaMA lead source moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp