Merge-based vocabulary extension stalls at the pre-tokenizer for Nepali, delaying English-preserving repair during continued pretraining
Read the original at arxiv.org→arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining...
Original headline: "Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining"
Coverage timeline
- Oct 27, 04:00 UTC arXiv cs.CL lead source Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining