More Thoughts On Measurements for Product Managers

I’m still thinking about how to measure product performance a customer-based world. Where I ended up yesterday was pretty much that there’s no alternative to using lifetime value, which specifically means calculating the incremental impact of each product sale on a customer’s future value. The main objection to this is the numbers will contain a great many estimates that will probably seem arbitrary, political or downright incomprehensible to most product managers. This violates one of the fundamental rules of management metrics, which is that managers should be judged on measures they can understand.

It also violates the rule that managers should be held accountable for things they can control. This is because the future value of the customer is affected by many factors other than that one product purchase. Managers would rightly feel they were being treated unfairly if the value assigned to their work was based largely on external elements.

One shouldn’t make too much of these issues. Although revenue is pretty easy to measure directly, any profit statement includes a fair number of allocated costs that are somewhat questionable. And even revenue figures will include estimated reserves for returns, bad debt and similar future losses. I suspect that few product managers could really explain how those calculations are made. Unless they suspect a major error (and that this error undervalues their performance), they are likely to just accept the figures provided. Lifetime value would ultimately work that way as well.

The “dependence on others” objection can also be overstated. In any large organization, many major revenue and cost drivers will be outside the product manager’s control. So they are used to that as well.

But, realistically, there is a big difference between being held responsible for profits on your own product’s sales—however those profits are measured—and profits on subsequent sales of other products. Both the fact that these are sales of other products and that they occur after the customer completes her experience with your product are problematic.

The best I can do right now is to suggest estimating the customer’s future behavior at the end of the product experience. This at least captures the customer’s state when they “left your hands”, so to speak, and before their intentions were affected by other activities. Even better, you could compare their expected behavior after the purchase with their expected behavior before the purchase, since any change is presumably due to their experience in between.

Of course, this immediately raises the question of when the product experience ends. Assuming they use the product after they buy it, its performance will affect their behavior at least as much as the purchase experience itself. I don’t have a specific answer for this; maybe you measure expected behavior in several places.

The other question this raises is where you get the estimates. In a rigorously tested environment, they could be based on firm data. When you find that environment, let me know. Here in the real world, you’ll probably be stuck measuring customer intentions with something like a net promoter score. Yes, I’m fully aware of the problems with such surveys, and still stand by my earlier criticisms. But if net promoter score is the best measure available, then that’s the one you use.

The point of measuring customer intentions after the purchase is simply to get product managers to think of affecting future behavior as part of their job. That’s what has to happen if they are going to help maximize lifetime value. Our real goal is to convert the net promoter scores into an expected future value stream, and thus report the change in expected lifetime value directly. But net promoter scores might actually be easier for them to grasp.

I’m still not thrilled with this solution, but it’s progress of a sort. At least it lets product managers manage something that is largely under their control, yet still orients them toward long term customer value. Those are the basic objectives.