Byte-level chunking for zero-shot transfer to low-resource languages with hierarchical byte-level architectures
Read the original at arxiv.org→arXiv:2608.27658v1 Announce Type: new Abstract: Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models...
Original headline: "When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages"
Coverage timeline
- Aug 31, 04:00 UTC arXiv cs.CL lead source When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages