Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation
Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures across languages are congruent and mutually mappable without loss of information.
ProofPaper ↗
Key points
- We formalize this failure using a usage-cloud framework, representing concepts as point sets of contextualized embeddings.
- We define $α$-unalignability as the impossibility of any mapping that simultaneously preserves lexical faithfulness (centroid correspondence) and structural faithfulness (local neighborhood topology).
- Behaviorally, we show that FLORES-200 translation failures are predicted by language family and resource class but not by script, and that LOBSTER reasoning scores vary by family.
- Mechanistically, we report a Representation-Intervention Gap (RIG) in a nine-model case study on Yami: the models' activations encode a regularity along which Yami groups with other low-resource and Austronesian languages, yet interventions on language-specific neurons show no demonstrated advantage over random masks: the regularity is visible but not usable by this intervention.
Sources (1)
- [1]Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark EvaluationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 01:10 PM
Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures across languages are congruent and mutually mappable without loss of information.
We formalize this failure using a usage-cloud framework, representing concepts as point sets of contextualized embeddings.
Extractive summary: sentences quoted from the sources.