Finding the Heads and the Neurons Responsible for Network Information Retrieval in Language Models

Key points

• At the head level the answer is yes, across five models spanning three architecture families: in every model, a small set of heads (1 to 9 out of 128 to 1152 candidates), found by causal ablation screening and tested for selectivity against matched negative and context-free controls, supports a detector with 99.5--100% held-out accuracy.1
• We then ask whether a head's responsibility concentrates into one neuron or stays spread across its dimensions; this is model-specific.1
• In another, the single clean head works as a whole (AUC = 1.000) but the best causally ranked neuron inside it does not (AUC = 0.665), so the responsibility there is spread across the head.1
• Causal head-finding for a specific network-information entity works across models and architectures; how far that finding can be pushed down to individual neurons varies, and needs to be checked for each model.1

Sources (1)

• [1]Finding the Heads and the Neurons Responsible for Network Information Retrieval in Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:51 AM

  We ask whether specific attention heads, and more finely specific neurons inside those heads, are responsible for recognizing that a language model's context contains network infrastructure information (a hostname paired with its IP address), and whether that responsibility can be validated causally rather than by correlation alone.

  At the head level the answer is yes, across five models spanning three architecture families: in every model, a small set of heads (1 to 9 out of 128 to 1152 candidates), found by causal ablation screening and tested for selectivity against matched negative and context-free controls, supports a detector with 99.5--100% held-out accuracy.

Extractive summary: sentences quoted from the sources.