Expanding attention via a Taylor-series approach yields linear-time transformers with a fixed number of recurrent states
Read the original at arxiv.org→arXiv:2610.20637v1 Announce Type: new Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses...
Original headline: "Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States"
Coverage timeline
- Oct 16, 04:00 UTC arXiv cs.LG lead source Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States