AdaptTok-Indic: script-aware adaptive tokenization for Indic languages
Read the original at arxiv.org→arXiv:2610.33017v1 Announce Type: new Abstract: Multilingual tokenizers trained on English-dominated corpora often over-segment Indic languages, resulting in longer token sequences and increased tokenization...
Original headline: "AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages"
Coverage timeline
- Oct 27, 04:00 UTC arXiv cs.CL lead source AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages