Where should the KV cache live? Placement across GPU, CPU, and SSD for long-lived sessions
Read the original at arxiv.org→arXiv:2609.16215v1 Announce Type: new Abstract: GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems...
Original headline: "Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions"
Coverage timeline
- Sep 16, 04:00 UTC arXiv cs.AI lead source Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions