Causal inference notes

Causal inference notes, part 1: pre-register your primary metric

Bar chart comparing eight test variants with confidence intervals

A/B tests are the closest most teams get to a controlled experiment. They are also easy to misread, mostly because of what happens after the test, not during it.

The garden of forking metrics

Track twenty independent metrics at a 5% significance level and the chance that at least one shows a “significant” change by luck alone is about 64%.1 Pick the winner afterwards and you have a story, not a result.

Bar chart comparing eight test variants with confidence intervals
Eight variants, one metric. Most intervals overlap; only the plum variant is clearly different from control.

A one-paragraph pre-registration

  1. The primary metric and the direction you expect.
  2. The minimum effect you would act on.
  3. The sample size and stopping rule.
  4. Two or three guardrail metrics that must not get worse.

If you did not write it down before the test, it is a hypothesis for the next one.

A rule we keep pinned above our desks
  1. 1 − 0.9520 ≈ 0.64, assuming independence. Real metrics are correlated, which helps a little but not enough. ↩