Measuring what matters

Measuring what matters, part 1: proxies and their failure modes

Bar chart of effect sizes by cohort with error bars

Series: Measuring what matters

  1. Part 1: proxies and their failure modes
  2. Part 2: session length is a trap

Nobody wants “clicks”. We want people to find what they came for. Clicks are a proxy, and proxies fail in a handful of recurring ways.

Four failure modes

  1. Saturation: the proxy stops moving while the goal keeps improving.
  2. Inversion: more of the proxy means less of the goal (more searches because results got worse).
  3. Gaming: the proxy becomes a target and detaches from the goal.
  4. Heterogeneity: the proxy means different things for different groups.
Bar chart of effect sizes by cohort with error bars
Effect of the same change on the proxy, by cohort. Two cohorts move in the opposite direction.

The defence is not a better proxy but several, chosen so that their failure modes do not overlap, and a habit of checking them against qualitative evidence.1

  1. Interviews and support tickets are slow and noisy, but they rarely fail in the same direction as an engagement metric. ↩