FactorEngram: factorized N-gram memory with basis-level gating for language models

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them...