Spectral outliers reveal dominant learned structure in transformer attention.
Read the original at arxiv.org→arXiv:2608.07921v1 Announce Type: new Abstract: We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of...
Original headline: "Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention"
Coverage timeline
- Aug 11, 04:00 UTC arXiv cs.LG lead source Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention