Thanks for sharing, Jacob. It was super-interesting to learn how you think about this. Now I will do the academic thing and critique NPS' simplicity :)
If I could fix one thing it would be to add a variance component. A "focused 7" (all 7s) means something very different than a "broad 7" (half 10s, half 4s) and has very different implications for a manager trying to improve it. I can see how means are useful for understanding "between" variance (good Enterprise branches, bad Enterprise branches), but they are not so useful for a branch manager trying to improve their performance.
The variance point is a great one. I think the broader point (and even a positive lesson from the development of NPS) is that focusing on metrics other than a mean is a good thing. And often a single metric isn't enough - to your point, building a 2x2 chart of variance and mean might be a nice way for a manager to quickly identify branches that have bimodal distributions and warrant further scrutiny.
And more broadly, variance is definitely something that routinely gets ignored (outside of "statistical significance" talk) because it's hard to explain and somewhat unsatisfying to folks without a significant amount of statistics training. For whatever it's worth, I'm very familiar with this as I actually worked on a soccer project with an MLS team (with Jeff Chen) to figure out the variance component of a common soccer analytics metric.
What's particularly interesting about NPS is that people intuit that the deconstruction is of metric is important. So the popular advice I've seen (see the graphic at the top of the post) is to look at the % Promoters and % Detractors. In a sense, this is an informal way of looking at the variance of the score. In fact, you could imagine a manager doing a scatter plot of these two metrics would also be really informative and maybe easier to understand.
I think the reason for this decomposition is it makes it clear to managers how to improve NPS - increase promoters or decrease detractors. And separating things out does make quite clear what lever may be more helpful to your organization. For example, if your score is low because you don't have a lot of promoters, that's advice a manager can work with. If it's low because you have both a lot of promoters and a lot of detractors that would have a different remedy.
I do think this general model of summarizing measurements, like Likert scales, to provide ways for non-stats people to better understand metrics is a valuable one. Honestly, I think the sports analytics community is probably the best at this. I'll do a post on that in the future for sure.
Thanks for sharing, Jacob. It was super-interesting to learn how you think about this. Now I will do the academic thing and critique NPS' simplicity :)
If I could fix one thing it would be to add a variance component. A "focused 7" (all 7s) means something very different than a "broad 7" (half 10s, half 4s) and has very different implications for a manager trying to improve it. I can see how means are useful for understanding "between" variance (good Enterprise branches, bad Enterprise branches), but they are not so useful for a branch manager trying to improve their performance.
Thanks, Matt!
The variance point is a great one. I think the broader point (and even a positive lesson from the development of NPS) is that focusing on metrics other than a mean is a good thing. And often a single metric isn't enough - to your point, building a 2x2 chart of variance and mean might be a nice way for a manager to quickly identify branches that have bimodal distributions and warrant further scrutiny.
And more broadly, variance is definitely something that routinely gets ignored (outside of "statistical significance" talk) because it's hard to explain and somewhat unsatisfying to folks without a significant amount of statistics training. For whatever it's worth, I'm very familiar with this as I actually worked on a soccer project with an MLS team (with Jeff Chen) to figure out the variance component of a common soccer analytics metric.
What's particularly interesting about NPS is that people intuit that the deconstruction is of metric is important. So the popular advice I've seen (see the graphic at the top of the post) is to look at the % Promoters and % Detractors. In a sense, this is an informal way of looking at the variance of the score. In fact, you could imagine a manager doing a scatter plot of these two metrics would also be really informative and maybe easier to understand.
I think the reason for this decomposition is it makes it clear to managers how to improve NPS - increase promoters or decrease detractors. And separating things out does make quite clear what lever may be more helpful to your organization. For example, if your score is low because you don't have a lot of promoters, that's advice a manager can work with. If it's low because you have both a lot of promoters and a lot of detractors that would have a different remedy.
I do think this general model of summarizing measurements, like Likert scales, to provide ways for non-stats people to better understand metrics is a valuable one. Honestly, I think the sports analytics community is probably the best at this. I'll do a post on that in the future for sure.