Are we using the right research methods to evaluate innovation? Or do our methods themselves need significant reinvention? Behavioral Science suggests that the best way to understand how people respond to new ideas is to observe, and not ask.
Observing and collecting data in real time, and in realistic situations provides better insight into peoples’ future behavior than asking them about it. However, a lot of innovation research is still based on asking people what they think, often in highly unrealistic research environments.
Methods Matter. I believe that methods are an incredibly important part of the innovation process. Of course we need great ideas, but very few are going to make it to market without data to support them. We all want better success rates, but my bigger concern is that the wrong methods can also stifle brilliance before it gets a chance. Asking the wrong questions, or maybe the act of asking questions itself, risks throwing the baby out with the bathwater, and killing the really big ideas off before they have a chance.
We don’t know ourselves very well! One of many surprises as I explored Behavior Science is that we don’t really know ourselves very well! Our memories are surprisingly unreliable, we don’t really understand our own decision processes, and we are not very good at predicting our future behaviors (1,2). From the perspective of innovation research, this gap between what we think we will do, and what we actually do, makes asking people questions about how they will respond to our innovations very risky, as in many cases, they probably don’t know themselves.
Of course, some great real time observational methods exist today, which I’ll talk about later. I also know that it is easy for me to say “test under realistic situations’, but not always so easy to do, especially when dealing with cutting edge innovation. However, a lot of existing research still remains question based, and a lot of development work tends towards trendy web based questionnaires and neuro-marketing. I don’t believe these solve the problem. Instead, I’d love to see us focus more on developing methods that enable more watching, less asking, and that leverage technology to take research out of the lab and into the real world.
Let’s dig deeper into reasons for differences between our future and current selves, and how our bias for simple answers can compound this research challenge.
Context matters, a lot! Context has a huge impact on our behavior and decisions, although we are often blissfully unaware of it. For example, you may have heard “never go shopping when you are hungry”. This turns out to be good advice, and is backed up by numerous scientific studies that show hunger will cause us to buy significantly more calories. (e.g. 3) Unfortunately the influence of this situational hunger operates largely below our awareness, making it almost impossible for people to accurately predict it in a snack filled research environment. No matter how well a questionnaire is designed, or how accurately brainwaves are interpreted, people cannot accurately anticipate their ‘hungry behavior’ if they aren’t hungry.
Context has a huge impact on our decisions in general. Our mood, hunger, or how tired or stressed we are all impact our immediate goals, and hence our behavior. Has your morning self ever fully intended to go to the gym that evening, only to be thwarted by a more fatigued self later on the way home? This, or something similar, happens to all of us. However, if you asked our morning self if we were going to the gym, he would honestly say yes! Some context, like our mood, we bring with us. Other is supplied by our immediate surroundings, like the smell of fresh baked bread as we walk in a store, or even background music. And if you think the latter is a bit extreme, a study by North et al (4) showed that French music played in a store increased sales of French wine, whereas German music sold more German wine! The problem is that as individuals, we are mostly unaware of these subtle, but powerful influences on our behavior. So whether we are in a focus group, filling in a questionnaire, or in a neuroscience lab with electrodes glued to our head, what we say, or what we think, is being influenced by a very different context to the one that will influence our real world behavior.
Please don’t make me think about thinking. I mentioned in my previous blog that psychology teaches us that many everyday decisions occur below our awareness. Our lives are littered with useful little mental short cuts that help us navigate our world without having to consciously think about every decision (2,5,6). This poses a huge challenge for research, as by asking people questions, and/or but putting them in an unusual environment, we make them think a lot more about decisions than they typically do in the real world. This activates a whole different set of so called ‘high engagement’ decision processes that are based on deliberation and analysis, rather than the ‘low engagement’ short-cuts that often drive behavior in the real world. Effectively, question based research asks our conscious mind to second guess our ‘unconscious’ one, and without authentic contextual cues to help it.
Our Bodies matter! Our interaction with the physical world, or embodied cognition, also has a big impact on our low engagement decisions. A handle tells our ‘unconscious’ to pull open a door, or grab a package from a shelf (7), but physical context effects like size, distance, or just our mobility, color that interaction. Embodied and mobile cognition deserve a blog of their own. However, in many cases, taking research out of a lab, away from a video screen, and into the real world will eliminate this additional potential confound.
The Tyranny of Precision and the Illusion of Truth: We love numbers. However, pursuit of statistically valid data can lead to controlled conditions and artificial contexts that compound the challenges outlined above. But statistically clean data, it can create an illusion of truth. Reproducible doesn’t necessarily mean accurate, and every time we force a constraint on a panelist, or take a step away from reality, we change context and risk thinking about thinking, risking better numerical precision at the expense of predictive accuracy.
This doesn’t just apply to predictive research. We face similar challenges with backward looking research like product testing., as memory is also flawed and strongly impacted by context. More on that in a future blog.
Now, to be fair, how people respond to innovation will probably be a mix of low and high engagement thinking. Trying something very new will almost always evoke some level of evaluative thinking. However, even evaluative thinking is colored by context and emotion. Furthermore, even when dealing with the unfamiliar, its’ role is often less than we intuitively expect. Ironically, the more complex a decision, often the more likely we are to fall back on mental short cuts and simplified relative comparison, still drawing to some degree on familiarity and what feels right.
So how do we solve this? I don’t believe one technique is ever going to be perfect. Becoming a one trick research pony, and running around with a hammer looking for nails, while seductive as we gain expertise in a particular method, won’t solve the problem. However, using technology like smart phones, mini cameras, mobile eye tracking, and biometrics to collect real time data in real contexts, and to observe unobtrusively, where that is ethically acceptable, must be in the right direction. Also, techniques common in experimental psychology and some existing advertising research, that disguise exactly what question we are asking, are a great idea, while 3D printing can help with embodiment
There are a lot of conflicting numbers on success rates for new innovation, but I think most would agree it would be great to do better. To that end, I find the following rules of thumb helpful:
– Chose methods based on fit with task, rather than familiarity
– Get out of the Lab and make context as real as possible.
– Observe, don’t ask.
– Be unobtrusive
– Allow autonomy
– Don’t fall in love with numerical precision.
1. Dan Gilbert. Stumbing on Happiness (2007 Vintage Books)
2. Daniel Kahneman. Thinking fast and Slow (2011 Farrar, Straus and Giroux)
3. Dodd et al, (Int J Obes. 1977;1(1):43-7)
4. North, Hargreaves, & McKendrick (Nature, 390, 132, 1997)
5. Dan Ariely Predictably Irrational (2008), and The Upside of Irrationality (2010) HarperCollins
6. Gerd Gigerenzer. Gut Feelings (2008 Penguin)
7. Don Norman. The Design of Everyday Things (1988 Basic Books)
image credit: postcardsfromtheculdesac.com
Wait! Before you go…
Choose how you want the latest innovation content delivered to you:
- Daily — RSS Feed — Email — Twitter — Facebook — Linkedin Today
- Weekly — Email Newsletter — Free Magazine — Linkedin Group
A twenty-five year Procter & Gamble veteran, Pete has spent the last 8+ years applying insights from psychology and behavioral science to innovation, product design, and brand communication. He spent 17 years as a serial innovator, creating novel products, perfume delivery systems, cleaning technologies, devices and many other consumer-centric innovations, resulting in well over 100 granted or published patents. Find him at email@example.com