12 Dec 21

Campbell's Law

NNGroup published this useful summary about the way focusing on metrics can be detrimental to teams, organizations, and their customers, especially when they're funneled into a single metric like in the North Star framework.

NN starts by arguing that the common saying "what can't be measured can't be managed" is a misquote, and that the actual quote is the polar opposite: "“It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”

Campbell’s law states that the more important a metric is in social decision making, the more likely it is to be manipulated.

There's a similar saying by another social scientist:

Godhart’s Law states that “When a measure becomes a target, it ceases to be a good measure”

  • Both statements argue that single metrics can't substitute for holistic, complicated human behavior.

  • When humans are put in a situation where a metric is of high importance, they are very likely to modify their behavior in order to improve the metric, but this doesn't mean they'll act in a way that corresponds to the underlying intention or problem the metric is trying to solve.

There are many examples of this:

  • Schools that are optimized to getting students in college fail to make them truly expand student's potentials for knowledge and growth.
  • Customer service agents "cheating" by put customers on mute instead of on hold to improve their SLA metrics.
  • Businesses can pressure or reward customers for leaving positive reviews. So they're more interested in inflated reviews than actual feedback and improving the customer experience.

What can we do

  • Recognize that all metrics are flawed
  • Recognize that all metrics are biased and reflect what's important to the people setting them.
  • Combine metrics together to get a fuller picture of the experience.
  • Never rely on quantitative metrics alone. Triangulate research methods
  • Treat data as a tool for decision making, but do now allow them to make the decisions for you.
About