How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys

Key points

• First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack.1
• Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several.1
• Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%.1
• Code, preregistration and run outputs are released.1

Sources (1)

• [1]How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 08:55 PM

  Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists.

  First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack.

Extractive summary: sentences quoted from the sources.