ggml-cpu/ops vectorizes flash-attention F16 to F32 conversion using F16C intrinsics; PR #26947
Read the original at old.reddit.com→Overview ggml_cpu_fp16_to_fp32 leverages hardware F16C intrinsics (AVX-512, AVX2, etc.), faster than the software-only ggml_fp16_to_fp32_row, bringing 17-31% gain in prompt processing rate for a smaller model like...
Original headline: "ggml-cpu/ops: vectorize flash-attention V-cache F16 to F32 conversion by jinzihao · Pull Request #26947 · ggml-org/llama.cpp"
Coverage timeline
- Aug 13, 09:22 UTC r/LocalLLaMA lead source ggml-cpu/ops: vectorize flash-attention V-cache F16 to F32 conversion by jinzihao · Pull Request #26947 · ggml-org/llama.cpp