Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods

Key points

• We introduce SteerScope, a two-axis, multi-dimensional evaluation suite that jointly characterizes steering outcomes and method properties through 15 metrics.1
• We score target efficacy and side effects on language quality, task capabilities, and safety and reliability, and further assess generalization and data dependence through steering-specific metrics for sample efficiency and sample sensitivity.1
• Under matched models, tasks, and evaluation protocols, we benchmark 23 methods spanning 4 families, including prompting, LoRA, and SFT as baseline methods, and release the suite as an extensible codebase.1
• We find that current activation steering methods do not yet surpass the Prompt Steering baseline in their overall balance between steering efficacy and side effects: across both model scales, no evaluated activation steering method achieves higher efficacy without incurring greater composite side effects.1

Sources (1)

• [1]Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:16 AM

  Activation steering provides a lightweight and flexible way to control large language model (LLM) behavior.

  We introduce SteerScope, a two-axis, multi-dimensional evaluation suite that jointly characterizes steering outcomes and method properties through 15 metrics.

Extractive summary: sentences quoted from the sources.