← writingentry #003
Activation patching, slowly
What the normalised patching score actually measures, and three ways it quietly misleads.
Contents03 sections
In Reading the residual stream I gave activation patching one paragraph. It deserves more, because it is the tool that most often gets quoted as evidence, and the number it produces is easy to over-read.
The setup
Take two prompts that differ in exactly the way you care about: a clean prompt where the model gets the answer right, and a corrupted prompt where the relevant information has been swapped out. Pick a metric. The usual one is the logit difference between the correct answer and a plausible wrong answer :
Now run the model on , but at one site (a head output, an MLP output, the residual stream at some layer and position) replace the activation with the value it had on the clean run. Write the result as .
The normalised score
To compare sites, rescale so that the corrupted run sits at 0 and the clean run sits at 1:
A score near means patching that site changed nothing. A score near means that one activation was enough to bring the clean behaviour back.
Why it can leave
Nothing in the formula bounds it.1 If the patched activation pushes the correct logit further than the clean run ever did, . If it moves the model away from the clean answer, . Both happen in practice, and both are informative rather than errors.
Three ways the score misleads
- Sufficiency is not necessity. Patching in a clean activation tests whether it is enough to restore behaviour. It says nothing about whether the model needs it. For that you ablate: remove the activation from the clean run and see what breaks.
- The corruption defines the question. Swapping names asks “where does the model track which name is which?” Replacing names with random tokens asks something broader. The same model gives different heat maps for each, and neither is the “real” one.
- Distributed mechanisms hide. If ten heads each contribute a tenth of the effect, every single-head patch scores around and the map looks empty. Patch groups of components before concluding nothing is there.2
Derivation: why the last-layer, last-position residual patch always scores 1
At the final layer and final position, the residual stream is the only input to the unembedding. Patching the whole stream there copies everything the logits depend on, so
That makes it a free sanity check: if this cell is not exactly 1, the hooks are wired wrong.
A sanity-check harness
Before trusting any heat map, I now run the same three assertions:
def normalised_score(patched: float, clean: float, corrupted: float) -> float:
return (patched - corrupted) / (clean - corrupted)
assert abs(normalised_score(clean, clean, corrupted) - 1.0) < 1e-6 # full patch → 1
assert abs(normalised_score(corrupted, clean, corrupted)) < 1e-6 # no patch → 0
assert abs(clean - corrupted) > 1.0, "denominator too small to trust" # signal existsA patching score is a statement about this pair of prompts, this metric and this site. Quote it with all three attached.
Notes
-
It’s a ratio of two differences, and only the denominator is fixed by the clean and corrupted runs. The numerator depends on what the patch does, which can overshoot in either direction. ↑
-
Path patching, which restricts a patch to particular routes between components, is the usual next step when single-site patching comes back diffuse. ↑
comments
loading…
no replies yet. write one below, or select any sentence above to reply to it.
comments are read-only on this copy of the site (no comment server attached).