What are the top 3 causes of wrong customer insights?

(This is an excerpt from Chapter 1 of the book Why Consumers Don’t Buy: GO & STOP Signals)

Marketing mistakes are often attributed to wrong customer insights. The standard refrain is that mistakes happen because managers do not listen to the “voice of the customer.” However, the problem is not so simple. While not listening to the “voice of the customer” has often landed companies in trouble in the past, this does not seem to the main cause of consumer insight failures. We believe that consumer insight failures can be attributed to three major causes: (i) incorrect beliefs about consumer behavior, (ii) a hedgehogian approach to strategic decisions, and (iii) incorrect beliefs about market research.


Incorrect Model of the Mind

          Our mental models – that is, our beliefs about how things work – shape our thought processes. The accuracy of our predictions, inferences, and judgments about the world depend on the validity of our mental model of the world. If an astronomer’s beliefs about the solar system are incorrect, then his predictions about the eclipse are also likely to be incorrect. If an engineer’s beliefs about the properties of a material are incorrect, then his prediction about its tensile strength is also likely to be incorrect. In like vein, if a manager’s beliefs about how the human mind works are incorrect, then her predictions about consumer behavior are also likely to be incorrect.
The mental model of consumer behavior among MBAs and managers has been influenced by traditional economics model of rational consumer. The traditional literature in economics portrays consumers as homo economicus – rational and deliberative beings always making decisions to maximize their long term utility. At first blush, this parsimonious utility maximization model of consumer behavior seems reasonable. However, a more careful scrutiny reveals that when it comes to predicting consumer behavior the traditional utility maximization model and its extensions do not have much descriptive validity beyond some basic economic transactions. This is because the traditional economic models are more prescriptive than descriptive. That is, these models do not describe actual everyday consumer behavior; instead it characterizes how a “rational” consumer ought to behave. Mind you, in economics the word rationality is not used as most people use it in everyday parlance. This is rationality as defined by the mathematical models of economics. If you are a Star Trek fan, then think of Mr. Spock, the emotionless extraterrestrial humanoid from plant Vulcan who served as the first officer aboard Captain James T. Kirk’s space ship. Because he can exert complete control over his emotions and mind and he can perform complex computations in his mind in a jiffy; Spock comes very close to economists’ portrayal of a rational being. 
One doesn’t have to think for long to realize that ordinary earthlings do not behave like Spock. Therefore, their behaviors do not always conform to the tenets of rationality as prescribed in economics.
Here are some illustrative examples. One tenet of rationality is that a homo economicus should have consistent utility for money. For example, a rational consumer should always have more utility from $40 than from $20. So if a rational consumer finds a $20 discount attractive, then she should find a $40 discount even more attractive. But real-world consumers often violate this tenet of rationality. If you are like most consumers, then you would be delighted to get a $20 discount on a dress that is usually sold for $100, but would be considerably less excited by a $40 discount on an appliance that is usually sold for $2000. Although everybody knows that $40 can fetch twice the purchasing power, somehow $20 on $40 definitely seems more attractive than $40 on $2000. Clearly consumers don’t always value $40 more than $20; their valuations are influenced by somewhat arbitrary reference points in their minds.
Another tenet of rationality is that a homo economicus should have reasonably stable preferences for products. Only then can a homo economicus efficiently maximize utility. But this is seldom the case with real-world consumers. Real-world consumers’ preferences for products are hugely influenced by the salient cues in their immediate environment. They learn to spontaneously respond to the cues that they have previously seen in their environment, oftentimes without even being aware of such cues. For example, consumers often use prices and brand names as cues for quality. Brain scanning studies have shown that the same wine actually (i.e., physiologically) tastes better when it is priced at $25 than when it is priced at $5. The same energy drink can be actually less efficacious – provide less energy – when it is sold at half the price. For Coca Cola fans, the same cola – when served in two differently branded packs – seems tastier when it is branded as Coca Cola than when it is branded as Pepsi Cola. What these studies tell you is that, unlike the elusive homo economicus, real people do not have stable preferences. Consumers’ preferences, and by extension the utilities that underlie these preferences, are very labile.
If managers and public policy formulators work with the assumption that consumers are deliberative and emotionless utility maximizers, then their predictions about consumer behavior are likely to be way off the mark. Real-world consumers are quick thinkers, relying on fast and frugal heuristics, and emotionally sensitive beings, intrinsically motivated to avoid negative emotions and seek positive emotions.

Hedgehogian Thinking

Isiah Berlin, a British philosopher and thinker wrote an essay titled “The Hedgehog and the Fox” in 1953, in which he argued that influential thinkers can be divided into two categories: hedgehogs and foxes. This analogy was inspired by the ancient Greek warrior-poet Archilochus, who is reported to have noted that the fox knows many things; the hedgehog one great thing. Hedgehogs have one very effective way of dealing with adversity – it uses its sharp spines or quills to protect itself and inflict pain on its foe. When a hedgehog encounters a foe, it rolls itself into a ball such that its quills point outwards. Although the purpose of comparing a hedgehog to a fox is not very obvious, Berlin used this adage to argue that like hedgehogs, some people view the world through the lens of a single defining idea. In contrast, others, like foxes, draw on a wide variety of experiences and for them the world cannot be boiled down to a single idea.
Berlin’s ideas are very relevant for business managers. Mispredictions of consumer behavior—or incorrect consumer insights—caused by a hedgehogian view often result in misdirected business strategies. Managers often fall in love with one idea that seems powerful. They become obsessed with this seemingly powerful idea. They come to believe that this one particular idea will always beget successful marketing strategies. For example, some managers fervently believe that low prices and sales promotions are good for business. If they experience success with lowering prices and running sales promotions in one context, then they mindlessly try to implement the same strategy in all contexts without considering that consumer behavior varies across contexts. A sales promotion might help a pizza delivery chain, but it might completely backfire for a formal dining restaurant. As another example, some managers adopt a new packaging that is trendy because it has worked for others. So they start mindlessly adopting the new packaging in all categories. And some, like a recent New York Times article documented, are falling over each other to create “emporiums of cool” user experiences. Why? Because they saw some case studies suggesting that improving user experience improves financial bottom lines.
Instead of understanding the basic GO and STOP signals that driver behavior, when managers see consumer behavior through the lens of a single (often previously-successful) idea, it often leads to incorrect predictions about consumer behavior.

Not Understanding the Difference between Asking & Testing 

The third cause of incorrect prediction is incorrect beliefs about market research. Lamentably, many managers either do not test their consumer insights, or if they do, they use incorrect methodologies to test consumer insight. Managers often do not test consumer insights because they do not trust traditional market research. For instance, the technology marketing guru Steve Jobs is reported to have said "Some people say, 'Give customers what they want.' But that's not my approach. Our job is to figure out what they're going to want before they do. I think Henry Ford once said, 'If I'd asked customers what they wanted, they would have told me, "A faster horse!"' People don't know what they want until you show it to them. That's why I never rely on market research. Our task is to read things that are not yet on the page." Those are some strong words from one of the legendary marketers of this century. Are his words justified? Is market research unreliable?
The answer is both yes and no.
In a sense, Steve Jobs was wrong.
In some situations market research can provide very reliable answers. For repetitive and habitual behaviors, the traditional market research techniques that use past behavioral data to predict future behaviors can be very reliable. In the past few decades, market researchers and academics have developed impressive models to predict repetitive and habitual behaviors of consumers using their past behaviors. For example, John Little and Peter Gaudagni of Massachusetts Institute of Technology published a paper in 1983 which is now widely acknowledged as a classic. They demonstrated that if we know (a) your grocery purchase patterns for the past six months and the prices of coffee brands in your store during this period, and (b) next week’s prices and promotional schemes on all coffee brands in your regular grocery store, then we can predict with over 80% accuracy what brand of coffee you will buy on your next trip to grocery store. Over 80% prediction accuracy - isn’t that impressive?! In fact, the Nielsen research company uses behavioral data collected from retail stores to develop such models, and packaged goods giants such as Procter and Gamble, Unilever, and Colgate routinely use such models to predict how consumers will react to price changes and promotional schemes. Ignoring market research in such environs would be simply foolhardy.
In another sense, Steve Jobs was right.
However, traditional market research techniques are much less reliable when it comes to behaviors in novel contexts. Powerful innovations often entail changing the status quo – putting consumers in a new retail environment, offering them a new pack that they have never seen, or offering them a new-to-the-world product. In such instances marketers don’t have reliable past behavioral data to predict behavior. So it is not easy to predict how consumers will behave in such situations. For example, it is not easy to predict how many people will buy a 3-D TV using traditional research approaches. A 3-D TV entails a behavioral change – putting on a pair of 3-D glasses each time one switches on the TV. It is not easy to predict how many people will be willing to change their behavior. In a similar vein, it is not easy to predict how many people will buy the revolutionary personal transporter Segway using traditional research approaches. While we are not suggesting that market research ought to be entirely ignored in such consumption contexts, we do agree with Steve Jobs that traditional market research, as it is currently practiced, ought to be taken with a grain of salt.