FlashMask-3 speeds mask-aware distributed attention by compressing column-wise masks from O(n^2) to O(n) with four query-axis boundary vectors; implemented for Ampere GPUs.
Read the original at www.reddit.com→Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the...
Original headline: "[Paper] FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention"
Coverage timeline
- Oct 20, 14:02 UTC r/LocalLLaMA lead source [Paper] FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention