FlashOffloading: low-latency KV cache offloading for long-context LLM serving
Read the original at arxiv.org→arXiv:2611.07542v1 Announce Type: new Abstract: In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the...
Original headline: "FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving"
Coverage timeline
- Nov 6, 05:00 UTC arXiv cs.AI lead source FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving