Llama.cpp: fix unified KV for multi-seq with flash_attn_ext_rows on CUDA
Read the original at www.reddit.com→Do you run multiple agents locally like I do? If so, this is a really nice speedup for the unified KV cache submitted by /u/jacek2023 [link] [comments]
Original headline: "llama + CUDA: flash_attn_ext_rows (fix unified kv for multi-seq) by am17an · Pull Request #29510 · ggml-org/llama.cpp"
Coverage timeline
- Nov 2, 16:41 UTC r/LocalLLaMA lead source llama + CUDA: flash_attn_ext_rows (fix unified kv for multi-seq) by am17an · Pull Request #29510 · ggml-org/llama.cpp