Finding Blind Spots in AppWorld and WorkArena Task Verifiers

Key points

• We audit shipped AppWorld and WorkArena verifiers with source-informed mutation tests.1
• In AppWorld, duplicating a non-idempotent write creates an extra record while preserving every checked field value.1
• A cardinality patch applied to checker copies after the census makes all six cells fail while preserving valid controls.1
• Other checker-PASS cases are effect-correct degeneracies.1

Sources (1)

• [1]Finding Blind Spots in AppWorld and WorkArena Task Verifiers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 09:34 PM

  Execution-based task verifiers decide whether an agent succeeded.

  We audit shipped AppWorld and WorkArena verifiers with source-informed mutation tests.

Extractive summary: sentences quoted from the sources.