Cracks in the foundation: four minor architectural decisions reduce long context extensibility across Olmo, Llama, and Qwen dense models
Read the original at arxiv.org→arXiv:2608.10296v1 Announce Type: new Abstract: One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the...
Original headline: "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension"
Coverage timeline
- Aug 12, 04:00 UTC arXiv cs.CL lead source Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension