Bharati: morphology-aware tokenizers for classical Indian languages with subword fertility analysis
Read the original at arxiv.org→arXiv:2607.23319v1 Announce Type: new Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce...
Original headline: "BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis"
Coverage timeline
- Jul 28, 04:00 UTC arXiv cs.CL lead source BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis