Who Cares About Forecast Accuracy?
by Robin Hanson
Reaction Essay
July 13th, 2011
Gardner and Tetlock note that while prediction is hard, we should be able to do better. For example, we could attend less to “hedgehogs” who know “one big thing” and whose forecasts are “beaten rather soundly even by the [random] chimp.” Yet we seem surprisingly uninterested in improving our forecasts:
Corporations and governments spend staggering amounts of money on forecasting, and one might think they would be keenly interested in determining the worth of their purchases and ensuring they are the very best available. But most aren't. They spend little or nothing analyzing the accuracy of forecasts and not much more on research to develop and compare forecasting methods. Some even persist in using forecasts that are manifestly unreliable. … This widespread lack of curiosity … is a phenomenon worthy of investigation.
I can confirm that this disinterest is real. For example, when I try to sell firms on internal prediction markets wherein employees forecast things like sales and project completion dates, such firms usually don’t doubt my claims that such forecasts are cheap and more accurate. Nevertheless, they usually aren’t interested.
TV weather forecasters are not usually chosen based on their forecast accuracy. Top business professors tell me that firms usually aren’t interested in doing randomized experiments to test their predictions about business methods. Furthermore, a well-connected reporter told me that a major DC-area media firm recently abandoned a large project to collect pundit prediction track records, supposedly because readers just aren’t interested.
Gardner and Tetlock are heartened to see a big research project testing new ways to aggregate forecasts, as that “holds out the promise of improving our ability to peer into the future.” But I can’t be much encouraged without a better understanding of our disinterest in using already well-tested and simpler methods. After all, a good diagnosis usually precedes a good prognosis.
So let me try to tackle this puzzle head on. Surprising disinterest in forecasting accuracy could be explained either by its costs being higher, or its benefits being lower, than we expect.
The costs of creating and monitoring forecast accuracy might be higher than we expect if in general thinking about times other than the present is harder than we expect. Most animals seem to focus almost entirely on reacting to current stimuli, as opposed to remembering the past or anticipating the future. We humans are proud that we attend more to the past and future, but perhaps this is still harder than we let on, and we flatter ourselves by thinking we attend more than we do.
The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers.
Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions.
Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy.
Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness.
Even in business, champions need to assemble supporting political coalitions to create and sustain large projects. As such coalitions are not lightly disbanded, they are reluctant to allow last minute forecast changes to threaten project support. It is often more important to assemble crowds of supporting “yes-men” to signal sufficient support, than it is to get accurate feedback and updates on project success. Also, since project failures are often followed by a search for scapegoats, project managers are reluctant to allow the creation of records showing that respected sources seriously questioned their project.
Often, managers can increase project effort by getting participants to see an intermediate chance of the project making important deadlines—the project is both likely to succeed, and to fail. Accurate estimates of the chances of making deadlines can undermine this impression management. Similarly, overconfident managers who promise more than they can deliver are often preferred, as they push teams harder when they fall behind and deliver more overall.
Even if disinterest in forecast accuracy is explained by forecasting being only a minor role for pundits, academics, and managers, might we still hope for reforms to encourage more accuracy? If there is hope, I think it mainly comes from the fact that we pretend to care more about forecast accuracy than we actually seem to care. We don’t need new forecasting methods so much as a new social equilibrium, one that makes forecast hypocrisy more visible to a wider audience, and so shames people into avoiding such hypocrisy.
Consider two analogies. First, there are cultures where few requests made of acquaintances are denied. Since it is rude to say “no” to a difficult request, people instead say “yes,” but then don’t actually deliver. In other cultures, it is worse to say “yes” but not deliver, because observers remember and think less of those who don’t deliver. The difference is less in the technology of remembering, and more in the social treatment of such memories.
A second analogy is that in some cultures people who schedule to meet at particular times actually show up over a wide range of surrounding times. While this might once have been reasonable given uncertain travel times and unreliable clocks, such practices continued long after these problems were solved. In other cultures, people show up close to scheduled meeting times, because observers remember and think less of those who are late.
In both of these cases, it isn’t enough to just have a way to remember behavior. A track record tech must be combined with a social equilibrium that punishes those with poor records, and thus encourages rivals and victims to collect and report records. The lesson I take for forecast accuracy is that it isn’t enough to devise ways to record forecast accuracy—we also need a new matching social respect for such records.
Might governments encourage a switch to more respect for forecast accuracy? Yes: by not explicitly discouraging it! Today, the simplest way to create forecast track records that get attention and respect is by making bets. In a bet, the parties work to define the disputed issue clearly enough to resolve later, and the bet payoff creates a clear record of who was right and wrong. Anti-gambling laws now discourage such bets—shouldn’t we at least eliminate this impediment to more respect for forecast accuracy records?
And once bets are legal we should go further, to revive our ancestors’ culture of respect for bets. It should be shameful to visibly disagree and yet evade a challenge to more clearly define the disagreement and bet a respectable amount on it. Why not let “put your money where your mouth is” be our forecast-accuracy-respecting motto?
[...] Posted on July 13, 2011 by Chris F. MasseThe Emperor of enterprise prediction markets is naked.Robin Hanson:I can confirm that this disinterest is real. For example, when I try to sell firms on internal [...]
[...] his response, Robin Hanson suggests that expert predictions aren’t really about trying to predict the future. They’re about signaling group affiliation and building authority. Experts make predictions [...]
[...] Who Cares About Forecast Accuracy? | Robin Hanson | Cato Unbound predictions more to signal your affiliation than to be accurate. no one cares about accurate predictions. let people bet, so they can at least profit from accuracy [...]
[...] legal in America, professor Hanson. — [AWARENESS] Posted on July 15, 2011 by Chris F. MasseRobin Hanson:A track record tech must be combined with a social equilibrium that punishes those with poor [...]
[...] frequently are more interested in predictions themselves than in the accuracy of the predictions. (Economist Robin Hanson had a terrific article about this at Cato Unbound last week. I highly recommend [...]
[...] Robin Hanson’s reaction essay, ‘Who Cares About Forecast Accuracy?’, is here. [...]
[...] Hanson’s reply brought to mind the scene in the movie A Few Good Men in which Jack Nicholson, the colonel in charge of Guantánamo, is badgered by a young idealistic attorney, Tom Cruise, in a murder investigation. In response to the attorney’s earnest entreaties that he just wants the “truth,” Nicholson launches into a tirade: the idealistic attorney has no inkling of the ugly trade-offs that the defense of liberty requires. He wouldn’t recognize, less tolerate, the truth if it stared him in the face (check?). [...]
[...] the Reaction Essay section, Robin Hanson addresses the puzzle of why forecasting remains so immune to accountability via – presumably easy [...]
[...] ESSAYS Who Cares about Forecast Accuracy? Robin Hanson argues that most people aren’t interested in the accuracy of predictions because [...]
[...] one. Robin Hanson wrote about the perversities of project planning and management in the Cato Unbound forum on expert forecasting: Even in business, champions need to assemble supporting political coalitions to create and sustain [...]