This is a really good read about measuring product improvement.

TechCrunch

I was recently asked to spend some time with an early stage startup that has a revolutionary new product. I asked them if they thought they were making their product better. As with every other startup I’ve asked, they said yes.

Then I asked them, “How do you know?” Their answer was also pretty standard: they explained that they were adding new features, improving quality, and generally executing against the product roadmap. The features are a combination of requests from their early customers and vision-inspired guesses from the founders. Each month, their gross numbers—the total number of customers, total revenue, and total usage—move up and to the right. So, they said, they must be on the right track.

Then I asked them this question: what would happen to the company if the entire product development team took a month off and went on vacation? The sales staff would keep signing up new customers. The website would continue to get new traffic from word of mouth. Could they be sure that they wouldn’t—as a business—be making just as much “progress” as they claim to be making now?

In one scenario, they’ve been working overtime, putting in crazy hours, and in the other, they’d be on vacation. If both scenarios lead to the same result, how can the product development team claim to be making progress? To be doing effective work?