Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

Key points

• We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts.1
• We run pre-registered studies on 1.7 million real requests (WildChat-1M, LMSYS-Chat-1M) with two cost/quality routers and a domain router, survey eleven systems' logging, and test post-processing defenses.1
• For RouteLLM at the 50% operating point, harassment and self-harm requests reach the strong model 19 points less often than comparable ones on prompts unseen in exploration, medical requests (exploratory: LLM labels failed their gate) 31 points less often on distinct prompts (both post hoc), and sexual requests 10 points more often (secondary); the other router's four are negative.1
• A post hoc exact per-user rate hides only even-prefix strong counts and forfeits most self-assessed routing value; it preserves odd-position decisions, from which a post hoc log attack reaches AUC 0.73 after 20 RouteLLM requests (exploratory).1

Sources (1)

• [1]Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:47 PM

  LLM routers pick a cheap or expensive model per request by its content, and many gateways and some cloud platforms can log that choice with content logging off.

  We measure this privacy channel beyond token counts, accounting for noisy labels and repeated prompts.

Extractive summary: sentences quoted from the sources.