Language Discrimination Improves Linguistic Learning in Multilingual Speech Models

Key points

• Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models.1
• Using a controlled English/French HuBERT setting, we test two interventions which strengthen language discrimination: an auxiliary language classifier and per-language k-means targets.1
• Across interventions, continuous-feature phone discrimination error (phone-ABX,↓) decreases from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%), while lexical performance (sWUGGY,↑) increases from 52.1% to 56.7% (monolingual: 58.5%) and prosodic performance (ProsAudit, lexical subtask,↑) from 68.9% to 72.9% (monolingual: 72.6%).1
• We introduce a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning).1

Sources (1)

• [1]Language Discrimination Improves Linguistic Learning in Multilingual Speech Models
  Apple Machine Learning Research · Oct 2, 12:00 AM

  We show that strengthening the model’s ability to discriminate languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-language sharing.

  Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models.

Extractive summary: sentences quoted from the sources.