Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Key points

• Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute.1
• One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts.1
• Using a construct validity framework that stipulates that an attribute must exist before a test can measure it, we start with HELM Safety's four datasets that might plausibly target harmful refusal, but find that three are saturated.1
• First, multidimensional item response theory modeling strongly suggests that HarmBench does not measure a singular attribute.1

Sources (1)

• [1]Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:46 PM

  Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles.

  Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute.

Extractive summary: sentences quoted from the sources.