DeltaMomentum: a Key-Value based anisotropic momentum update via delta rule
Read the original at arxiv.org→arXiv:2608.19491v1 Announce Type: new Abstract: Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs...
Original headline: "DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule"
Coverage timeline
- Aug 21, 04:00 UTC arXiv cs.LG lead source DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule