Measuring what matters

Measuring what matters, part 2: session length is a trap

A right-skewed histogram of session lengths with a density curve

Series: Measuring what matters

  1. Part 1: proxies and their failure modes
  2. Part 2: session length is a trap

Session length is the classic inverted proxy: when a task gets harder, sessions get longer. Averages make this worse, because the distribution is heavily skewed.

A right-skewed histogram of session lengths with a density curve
Distribution of session length for 4,000 sampled sessions. The mean sits well to the right of the typical session.

Look at the shape first

6.4

mean minutes

4.9

median minutes

23.1

99th percentile

A handful of very long sessions drag the mean upward. When those sessions turn out to be people stuck in a broken checkout, “engagement is up” is the opposite of good news.

Better questions

  • Did people complete what they started?
  • How long did completion take, for those who completed?
  • Where did the long sessions spend their time?