A 21M-model with a 6.4B-parameter lookup table matches a 114M dense model and runs with the table on an SSD.
Read the original at www.reddit.com→I spent the last few weeks on a hobby research project and just made it public. The idea isn't new (product-key memory, Lample et al. 2019, and Meta's "Memory Layers at Scale"): give a model a huge table of learned...
Original headline: "I gave a 21M model a 6.4B-parameter lookup table. It matches a 114M dense model and runs with the table on an SSD (RX 9070)"
Coverage timeline
- Oct 6, 16:57 UTC r/LocalLLaMA lead source I gave a 21M model a 6.4B-parameter lookup table. It matches a 114M dense model and runs with the table on an SSD (RX 9070)