Of experts, hedgehogs and foxes

Friday, 3rd March, 2006
Experts. They bob up on the television news and talk shows every night. Without the talking heads, ABC radio would find it impossible to fill in the hours. We watch them endlessly and hear them pontificate on what will happen to interest rates, to growth, to John Howard and George Bush, the likely developments in Iraq, China and Timbuctoo. On thousands upon thousands of websites around the world - including this one - people with more (and sometimes less) knowledge than me set out to explain what is happening and predict what will happen.
And what is all this expert opinion worth? Well, according to a recently published book by by Philip Tetlock, the Mitchell Professor of Leadership at the University of California, Berkeley, the answer is not much. After years of research, during which he worked with almost 300 experts known for commenting or offering advice on political and economic trends, Prof Tetlock concluded: when you compare the aggregate accuracy of experts to some very simple statistical models, you'll find that the experts are hard pressed to do better than very simple baselines.
On the cover of his book Prof. Tetlock has a fox and a hedgehog signifying his division of experts in to the famous Isaiah Berlin distinction of hedgehogs who know one big idea and confidently stick with it, applying it to any and all of various scenarios, and foxes who are more flexible and skeptical. In an interview about his book published on the TCS websitehe points to a tendency for experts to claim to know more than they actually do about the future. "The second thing you notice." says Prof Tetlock, "is that some experts are much more prone to be over-confident than are other experts. And that's where we get into our classification of experts in terms of their styles of reasoning as either hedgehogs or foxes."
Some further quotes from the interview give the flavour of his argument which has led to me ordering the book to read the whole thing:

One of the curious things about my experience in this study is that those experts who tend to be more doubtful that they could predict anything, were actually somewhat better at predicting. So modesty was actually a useful cue for accuracy in this context.
Foxes are more likely to believe that there's a certain amount of uncertainty and indeterminacy in the world; that the world just oscillates sometimes in violent unpredictable ways. And it doesn't really obey any set of laws whereas other schools of thought imply that there are some principals or laws or regulatory forces in history. It's not that foxes don't believe there are any regulatory forces in history. They tend to believe there are probably too many and they're often interacting in ways that make it very complicated to predict.
It raises the interesting question of why. And one possibility is that people aren't selecting experts primarily on the basis of the truth value of their pronouncements. People are selecting experts on other grounds and what might those other grounds be?
They're suggested in a lovely book Richard Posner wrote on public intellectuals. One is that the experts are providing what he called solidarity goods and the other is experts are providing entertainment goods. Now, a solidarity good is when it doesn't really matter whether the experts are right or wrong. The expert is affirming your values and the expert is doing a good job mashing the other side, affirming your side and making you feel good about your world view. So that would be a kind of a solidarity function for instance, better than a truth-seeking function. And an alternative function, a third function would be simple entertainment value.
One thing, and I think that's to a substantial degree in some of these television programs, there's an attraction to experts who are willing to make relatively extreme and therefore relatively interesting predictions. And one of my favorite examples is the various predictions experts have been making about Saudi Arabia over the last 15 plus years.
If you actually track that literature, you'll find that both in public and in private, experts have been predicting the disintegration of the Saudi regime, the Saudi monarchy pretty continuously. Now no doubt there is going to come a time when the Saudi monarchy does fall, in line with the broken clock theory, you're right at least twice a day. If you keep predicting the destruction of the Saudi monarchy, eventually you're going to be right.
Experts who predicted the disintegration of the Saudi regime have had a tendency to advance very interesting scenarios to bolster that prediction, so they can summon up very vivid images of Islamic colonels in Riyadh taking control and the new Saudi regime having an affinity for Osama Bin Laden and oil supplies being disrupted, gasoline prices going to $10 a gallon.
And there are all sorts of very vivid images they can summon up to catch people's attention and it simply makes much better television than an expert who comes on and says, well, yes, the pessimists are right that there are some sources of serious instability in Saudi Arabia and at some point, the Saudi regime is going to have to change its fundamental ways. But you know what, in the short to medium term, it's usually a pretty bad prediction to bet against authoritarian regimes like this, especially with authoritarian regimes that are sitting on top of huge wads of cash and have a very effective repressive police apparatus at their disposal.
A study against repressive government in the medium to short term tends to be a bad debt because governments control a lot of goodies, especially the Saudis and governments control a lot of sticks as well. So when you take into account the capacity to intimidate and the capacity to co-op, betting against a regime tends to be a bad debt.
So more generally what's going on here is experts who are entertaining tend to be those experts who attach high probabilities to low frequency events and there's a price to be paid for that.

There is a line among the fragments
of the Greek poet Archilochus which says:
“The fox knows many things, but
the hedgehog knows one big thing.”
Scholars have differed about the
correct interpretation of these dark
words, which may mean no mor
than that the fox, for all his cunning,
is defeated by the hedgehog’s one
defence. But, taken figuratively, the words can be made to yield
a sense in which they mark one
of the deepest differences which
divide writers and thinkers, and, it
may be, human beings in general.
For there exists a great chasm between those, on one side,
who relate everything to a single,
universal, organizing principle in
terms of which alone all that
they are and say has significance—
and, on the other side, those who
pursue many ends, often unrelated
and even contradictory.... Their
thought is scattered or diffused,
movingon many levels, seizing upon
the essence of a vast variety
of experiences and objects for
what they are in themselves,
without, consciously or
unconsciously, seeking to fit them into, or exclude
them from any one unchanging,
all-embracing, sometimes self-contradictory and incomplete,
at times fanatical, unitary inner
vision. The first kind of intellectual
and artistic personality belongs to
the hedgehogs, the second to the
foxes; and without insisting on a rigid
classification, we may, without too
much fear of contradiction, say that,
in this sense, Dante belongs to the
first category, Shakespeare to the second. 
Isaiah Berlin (b. 1909), British philosopher, critic.
The Hedgehog and the Fox,
ch. 1, Simon & Schuster (1957).