Robust Decentralized Fairness Auditing

Key points

• Emerging legislation requires large language models (LLMs) to be audited for compliance with regulatory standards, particularly fairness.1
• The fairness of the audited LLM is then estimated by aggregating all the vectors.1
• We show theoretically and empirically that even a single adversarial auditor in the network can steer this estimate by fabricating the vectors it sends, making an unfair LLM appear fair.1
• We implement Auditopus and compare it to robust aggregation baselines on two datasets with two pre-trained LLMs. Against an attacker that optimizes the vectors it sends to make the LLM appear fair, Auditopus reduces audit error by up to 78% on average relative to no defense and at least 62% relative to the robust aggregation baselines.1

Sources (1)

• [1]Robust Decentralized Fairness Auditing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:00 PM

  We propose Auditopus, a novel approach for robust decentralized fairness auditing.

  Emerging legislation requires large language models (LLMs) to be audited for compliance with regulatory standards, particularly fairness.

Extractive summary: sentences quoted from the sources.