Steerspeech: Activation Steering For Emotion Control In Generated Speech

Key points

• Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training.1
• For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen.1
• To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens.1
• Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation.1

Sources (1)

• [1]Steerspeech: Activation Steering For Emotion Control In Generated Speech
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:56 PM

  We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations.

  Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training.

Extractive summary: sentences quoted from the sources.