KV cache tricks for agents: prompt caching in llama.cpp stores memory-resident KV cache and ejects on prompt delta; used to run subagents when only one large model fits on GPU
Read the original at old.reddit.com→(Typing on my phone, apologies in advance for my “shorthand”.) I recently learned about prompt caching in llama.cpp. Basically, it’s a setting where your kv cache is stored in RAM as a copy and ejected on system...
Original headline: "What KV cache tricks do you guys do with your agents?"
Coverage timeline
- Aug 14, 12:10 UTC r/LocalLLaMA lead source What KV cache tricks do you guys do with your agents?