Latent Core Tokenizer: Compress, but Meaningfully

Key points

• Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages.1
• LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary.1
• Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost.1
• Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery and how frequency is used to allocate the final vocabulary across languages.1

Sources (1)

• [1]Latent Core Tokenizer: Compress, but Meaningfully
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:32 PM

  We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction.

  Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages.

Extractive summary: sentences quoted from the sources.