I've been learning a lot from Danny Kahneman's great book Thinking Fast and Slow. Kahneman is the world's leading expert on human judgment and decision-making and the only non-economist to be awarded the Nobel Prize in Economics (he's a psychologist by training), so his insights and conclusions should be taken seriously. In Thinking Fast and Slow he collects them and explains them to the non-specialist. Read it and you won't look at the world the same way again. And you and the world will both be better off for it.
In particular, we'll all be better off if we stop placing so much weight on the intuitive judgments of 'experts' -- those who have risen to the top of their professions or hierarchies due, typically, to some combination of education, experience, tenure, previous success, and moxie. The business world, of course, is full of experts. We have financial analysts, product planners, business unit managers who hire people and assemble teams, marketing VPs, and an endless variety of consultants, pundits, and gurus.
And their 'expert' (read: intuitive) judgments - about whether a stock will rise, how a new ad campaign will go over, how a competitive battle will play out, which products will succeed in the marketplace, whom to hire and promote, and so on - should be received with great skepticism. In fact, they usually shouldn't be received at all.
This is because human experts are overconfident, inconsistent, and subject to a swarm of thoroughly documented biases, most of which they're not even aware of. What's more, in many cases they're making their confident predictions in areas where accurate predictions just aren't possible. If anyone tries to tell you what the price of gold will be in six months, or who's going to be on top of the high-tech industry in 2020, excuse yourself as quickly as possible.
The most common response to these truths, sadly, is a simple refusal to believe or act on them. Another is to acknowledge the limitations of experts, but to say that we don't really have any good alternatives to them.
Don't believe it. A great deal of excellent research (initiated by people like Paul Meehl and Robyn Dawes) shows how easy it is to beat the 'experts,' at least in those situations where an accurate prediction is possible (in other words, not the stock market or the future of the high-tech industry eight years hence). All you have to do is find a few measurable things that are correlated with what you're interested in -- for example, an extroverted personality is correlated with success as a salesperson -- measure them objectively, and use them to create an aggregate score. You don't even have to worry too much about how much weight to give each measure relative to the others when constructing the score; equal weighting works really well (a finding that astonishes me, given how many stats classes I've been in).
Of course, more sophisticated approaches can increase predictive power, and in our current age of big data more and more things are easy to measure; it used to be hard to hear the voice of the customer, for example; now it's echoing across the entire Internet. So there are fewer and fewer reasons all the time for relying on expert judgment. In the business world, decision-making based on intuition and gut feel should be driven out as ruthlessly as the Spanish Inquisition rooted out heretics (and for much better reasons).
The primary techniques for accomplishing this will not be torture and the auto-da-fé, but instead clear thinking and competition. Kahneman's book is an admirable example of such clear thinking. Smart companies and their leaders will read it and change themselves accordingly. The superior performance they'll enjoy as a result will eventually cause their rivals to adapt or die. Not immediately, and not quickly enough, but eventually.
This blog first appeared on Harvard Business Review on 2/8/2012.