Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

Key points

• Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others.1
• We therefore ask whether tokenizer choice matters equally across languages, a question that the current literature leave unanswered.1
• Giving lower-resource languages a larger share of tokenizer-training data, however, does not unconditionally help those languages: both equal weighting and an allocation inverting the shares with respect to the language model training data increase their BPB, particularly when language-model training repeats data.1
• We find that the metrics quantifying these properties can be successfully used to predict downstream models' pairwise BPB rankings, suggesting a practical strategy for screening tokenizer candidates before training language models.1

Sources (1)

• [1]Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:29 PM

  We find that tokenizer choice matters more for languages with less language-model training data: across the 54 tokenizers, the standard deviation of a language's bits-per-byte (BPB) increases as its model training-data share decreases (Spearman rho = -0.52 over the 31 trained languages and -0.69 over the 28 written with word boundaries).

  Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others.

Extractive summary: sentences quoted from the sources.