Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs

Key points

• We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline.1
• The resulting language-specific head sets exhibit near-complete mutual exclusivity across the five languages, with a mean Jaccard similarity of only 0.017 for Llama 3.1 70B and 0.057 for Qwen 2.5 72B, revealing language-idiosyncratic circuits.1
• Ablating general BRH increases two-hop Negative Log-Likelihood (NLL) by 39-89x the random-head baseline, providing direct causal evidence of their role.1
• Together these results show that activation-level intervention alone can recover correct answers from cross-lingual reasoning failures.1

Sources (1)

• [1]Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 09:28 AM

  Multilingual LLMs answer the same multi-hop reasoning question across languages, but we lack a mechanistic account of whether they share an internal circuit.

  We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline.

Extractive summary: sentences quoted from the sources.