Transformer MLP gate thresholds act as a reference for gate operation in mid-stack residual directions
Read the original at arxiv.org→arXiv:2609.31956v1 Announce Type: new Abstract: The corpus-mean direction of a transformer's residual stream is a component shared across all inputs, and is commonly removed by mean-centering before representational...
Original headline: "Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction"
Coverage timeline
- Sep 29, 04:00 UTC arXiv cs.CL lead source Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction