A
Member
A linear probe trains a simple classifier on internal activations to predict a property. High probe accuracy is often read as "the model represents this feature". That inference needs several controls.
Necessary checks:
1) Baselines. Compare against a probe on random features, on untrained or shuffled-label controls, and on a simple input-level classifier. If the input-level classifier does just as well, the probe tells you nothing about internal representation.
2) Train/test separation that respects data structure. Near-duplicate samples leaking across splits inflate accuracy.
3) Layer and token position. Report the full layer-wise curve, not the best layer, and state which token's activation was used.
4) Selectivity: probe for several related properties and check the classifier is not just picking up a correlated confound.
5) Causal follow-up. A probe shows correlation. To claim the feature is used, intervene (ablate, patch, or steer the direction) and measure the behavioural change.
Reporting template: dataset with splits and sizes, baseline scores, probe family and regularisation, layer-wise accuracy with confidence intervals, and the results of at least one causal intervention.
Common pitfalls: using a high-capacity probe (which can memorise the mapping itself), tuning on the test set, and treating a probe trained on one distribution as evidence about a different one.
Takeaway: probing is a hypothesis-generating tool. Interpretability claims get stronger only when a correlation result is followed by a controlled intervention.
Necessary checks:
1) Baselines. Compare against a probe on random features, on untrained or shuffled-label controls, and on a simple input-level classifier. If the input-level classifier does just as well, the probe tells you nothing about internal representation.
2) Train/test separation that respects data structure. Near-duplicate samples leaking across splits inflate accuracy.
3) Layer and token position. Report the full layer-wise curve, not the best layer, and state which token's activation was used.
4) Selectivity: probe for several related properties and check the classifier is not just picking up a correlated confound.
5) Causal follow-up. A probe shows correlation. To claim the feature is used, intervene (ablate, patch, or steer the direction) and measure the behavioural change.
Reporting template: dataset with splits and sizes, baseline scores, probe family and regularisation, layer-wise accuracy with confidence intervals, and the results of at least one causal intervention.
Common pitfalls: using a high-capacity probe (which can memorise the mapping itself), tuning on the test set, and treating a probe trained on one distribution as evidence about a different one.
Takeaway: probing is a hypothesis-generating tool. Interpretability claims get stronger only when a correlation result is followed by a controlled intervention.