Attention predicts context rot in retrieval; longer inputs are not the main source of failure, says arXiv:2611.04169
Read the original at arxiv.org→arXiv:2611.04169v1 Announce Type: new Abstract: Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B...
Original headline: "Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval"
Coverage timeline
- Nov 4, 05:00 UTC arXiv cs.CL lead source Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval