Oriel — AI & technology worth noticing, every day.

Scoring Explanations Without Reading Them: Structural Shortcuts in Automated-Interpretability Evaluation Data

Paper · https://arxiv.org/abs/2610.37676

arXiv:2610.37676v1 Announce Type: new Abstract: Automated interpretability scores an explanation of a sparse-autoencoder (SAE) feature by asking a model to predict, token by token, where the feature fires on evaluation examples or to judge whether marked tokens fit it. We find that the evaluation data standard pipelines build can be scored without reading the explanation. On Neuronpedia's stored fuzzing runs for 500 random GPT-2 small features, 91.3% of activating examples are cropped with the peak on the centre token, and a rule that says "yes" when the centre token is marked reaches balanced accuracy 0.979, against 0.838 for the stored LLM scorers (recomputed from the stored per-item verdicts, on marks rebuilt from the pipeline's code); a rule reading only the mark count reaches 0.969. The cue changes form across pipelines: current delphi always marks the same slot in its fuzzing negatives, and full-context runs mark negatives with more tokens. OpenAI's simulation benchmark lists each neuron's top-activating sequences first; a rule that predicts activity on the first five scores 0.227 in pooled correlation against 0.160 for GPT-4's published simulator (post hoc). We give closed forms for what such rules can score; show that one scorer's simulation scores are higher on stored snippets than on natural windows but not shown to be reordered; and build windows that remove the position cue, derive but do not build negatives that remove the mark cues, and separate the sequence cue by decomposition. On natural-text windows cut at random offsets, the best fixed-offset prior shared by all features scores 0.015. On them, that scorer, a typed decision model (Jev), reaches a pooled correlation of 0.331 over 49 features, against 0.013 with shuffled explanations, and an open 8B simulator with the original read-out reads the explanation there (0.194 against 0.019). Scores should be reported beside the best explanation-free rule on the same items.

First seen by Oriel on .

AI summary

The paper finds that automated-interpretability benchmarks contain structural cues that explanation-free rules can exploit, and recommends reporting model scores against a simple baseline on the same items.

  • For 500 random GPT-2 small features, a rule marking the centre token achieves 0.979 balanced accuracy, compared with 0.838 for stored LLM scorers.
  • A rule using only mark count scores 0.969; other pipelines expose cues through fixed negative-mark positions or longer negative marks.
  • On OpenAI's simulation benchmark, predicting activity for the first five sequences scores 0.227 pooled correlation, versus 0.160 for GPT-4's published simulator.
  • On natural-text windows cut at random offsets, the best shared fixed-offset prior scores 0.015; Jev reaches 0.331 correlation over 49 features, versus 0.013 with shuffled explanations.
  • The authors build windows that remove the position cue and recommend comparing explanation-based scores with the strongest explanation-free rule on identical items.

#automated interpretability #sparse autoencoders #benchmark evaluation

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.37676v1https://arxiv.org/abs/2610.37676 Read discussion