A/B tests are the closest most teams get to a controlled experiment. They are also easy to misread, mostly because of what happens after the test, not during it.
The garden of forking metrics
Track twenty independent metrics at a 5% significance level and the chance that at least one shows a “significant” change by luck alone is about 64%.1 Pick the winner afterwards and you have a story, not a result.

A one-paragraph pre-registration
- The primary metric and the direction you expect.
- The minimum effect you would act on.
- The sample size and stopping rule.
- Two or three guardrail metrics that must not get worse.
If you did not write it down before the test, it is a hypothesis for the next one.
A rule we keep pinned above our desks
- 1 − 0.9520 ≈ 0.64, assuming independence. Real metrics are correlated, which helps a little but not enough. ↩