Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
Read the original at arxiv.org→arXiv:2610.22428v1 Announce Type: new Abstract: In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation...
Coverage timeline
- Oct 19, 04:00 UTC arXiv cs.CL lead source Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models