Measures – Will you get what you want?

by Stephen Shapiro

Measures - Will you get what you want?I recently spoke with a new client who shared with me their innovation measures. When I looked at their measurement system, I immediately saw flaws.

But before addressing these imperfections, let me first provide you my perspective on innovation measures.

In general, there are three types of measures associated with “challenge-based” innovation (be sure to read this article if you are unfamiliar with the concept of challenge-based innovation):

  1. Process Measures – These measure the activity associated with your challenges (e.g., 500 registered solvers, 40 submissions per challenge, 80 votes per challenge, etc)
  2. Solve-Rate Measures – These subjectively measure how well you solved your challenges (e.g., 82% of challenges were partially solved, 61% of challenges were completely solved, etc)
  3. Value Measures – These measure the actual value accrued (e.g., increased revenues by $25M, reduced costs by $35M, etc)

The last measure (value) is where the rubber meets the road. This is your ultimate goal. But sometimes, value realization can take years (or in the case of pharmaceutical companies, decades). Therefore, the second measure (solve-rate) is a good way to monitor progress with your program. But what about process measures?

Process measures are leading indicators that can be useful in measuring trends over time for things like community engagement, effectiveness of internal communications, and quality of challenges.

Let’s look at one common process measure: the number of ideas/solutions submitted for a given challenge. This was one of the measures that my new client used.

Imagine that you are using crowdsourcing to find a solution to a challenge. You post the challenge on your website or intranet. A month later you check to see how many responses you get. In this scenario…

Which is better:

  • Getting 100 ideas/solutions?
  • Getting only two ideas/solutions?

Most people intuitively think that 100 solutions is better than two. In fact, most organizations believe that more ideas equates to greater success. The reality is, however, that 100 is not necessarily better than two.

Let me re-frame the question…

Which is better:

  • Getting 100 ideas where only two of them were exactly what you needed and the other 98 were duds?
  • Getting two ideas where both were exactly what you needed?

Now the correct answer is a bit more obvious. In this situation, the latter is probably better. The amount of work needed to sift through the solutions is a lot less when you have only two submissions. Imagine if you received 10,000 ideas of which only two were good. You can see now that the effort to find the best solutions/ideas might be overwhelming.

Although activity is good, too many submissions can indicate that you have a poorly defined challenge. Therefore the ratio of good ideas to duds might be a more interesting measure.

The key is, make sure you understand the unintended consequences of your measurement system, especially when it comes to process measures. If done properly, process measures can help you drive higher solve rates (measure #2). And often, higher solve rates lead to greater value (measure #3) in the long run. But not always.

High solve rates with low value can also indicate problems with your innovation program:

  • Poor implementation – You are unable to convert solutions into finished products/services
  • Poor commercialization – Your solutions do not meet the needs of the market/customers and therefore do not generate revenue
  • Poor relevance – Your challenges, although solved, are not important enough to “move the needle” of the organization’s innovation efforts

Measures are important for helping tracking your innovation efforts. And they can help diagnose potential issues. But it is important to measure the right things.

There is an old expression: “You will get what you measure.”

But the bigger question is, “Will you get what you want?”

Don’t miss an article – Subscribe to our RSS feed and join our Continuous Innovation group!


Stephen ShapiroStephen Shapiro is the author of three books, a popular innovation speaker, and is the Chief Innovation Evangelist for Innocentive, the leader in Open Innovation.

No comments

Leave a Reply