Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining

Key points

• Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge.1
• We study a transferable bridge through two complementary properties: global alignment with the text side, and local lexical manifold compatibility, where bridge embeddings remain close to the frozen LLM's input-embedding neighbourhoods.1
• We make this property measurable with a fixed, head-free, timestamp-free diagnostic that applies to any objective, and show that next-word prediction (NWP) and sentence-level contrastive pretraining do not fully capture token-level lexical compatibility.1
• On multilingual ASR and speech translation, LPR improves transfer, with the largest gains on translation and low-resource adaptation.1

Sources (1)

• [1]Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:19 AM

  We then introduce Local Prototype Reconstruction (LPR), a lightweight training-only regularizer that requires each aligned bridge token to be reconstructable from a small neighbourhood of frozen LLM token embeddings, with a hard single-prototype anchor as its limiting case.

  Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge.

Extractive summary: sentences quoted from the sources.