Fox-Hedging or Knowing: One Big Way to Know Many Things

It is hard to say which is more surprising, that anyone still argues that we can predict very little or that anyone believes expertise conveys reliable judgment. Each reflects a bad habit of mind that we should overcome. It is certainly true that predictive efforts, by whatever means, are far from perfect and so we can always come up with examples of failure. But a proper assessment of progress in predictive accuracy, as Gardner and Tetlock surely agree, requires that we compare the rate of success and failure across methods of prediction rather than picking only examples of failure (or success). How often, for instance, has The Economist been wrong or right in its annual forecasts compared to other forecasters? Knowing that they did poorly in 2011 or that they did well in some other selected year doesn’t help answer that question. That is why, as Gardner and Tetlock emphasize, predictive methods can best be evaluated through comparative tournaments.

Reliable prediction is so much a part of our daily lives that we don’t even notice it. Consider the insurance industry. At least since Johan de Witt (1625–1672) exploited the mathematics of probability and uncertainty, insurance companies have generally been profitable. Similarly, polling and other statistical methods for predicting elections are sufficiently accurate most of the time that we forget that these methods supplanted expert judgment decades ago. Models have replaced pundits as the means by which elections are predicted exactly because various (imperfect) statistical approaches routinely outperform expert prognostications. More recently, sophisticated game theory models have proven sufficiently predictive that they have become a mainstay of high-stakes government and business auctions such as bandwidth auctions. Game theory models have also found extensive use and well-documented predictive success on both sides of the Atlantic in helping to resolve major national security issues, labor-management disputes, and complex business problems. Are these methods perfect or omniscient? Certainly not! Are the marginal returns to knowledge over naïve methods (expert opinion; predicting that tomorrow will be just like today) substantial? I believe the evidence warrants an enthusiastic “Yes!” Nevertheless, despite the numerous successes in designing predictive methods, we appropriately focus on failures. After all, by studying failure methodically we are likely to make progress in eliminating some errors in the future.

Experts are an easy, although eminently justified, target for critiquing predictive accuracy. Their failure to outperform simple statistical algorithms should come as no surprise. Expertise has nothing to do with judgment or foresight. What makes an expert is the accumulation of an exceptional quantity of facts about some place or time. The idea that such expertise translates into reliable judgment rests on the false belief that knowing “the facts” is all that is necessary to draw correct inferences. This is but one form of the erroneous linkage of correlation to causation; a linkage at the heart of current data mining methods. It is even more so an example of confusing data (the facts) with a method for drawing inferences. Reliance on expert judgment ignores their personal beliefs as a noisy filter applied to the selection and utilization of facts. Consider, for instance, that Republicans, Democrats, and libertarians all know the same essential facts about the U.S. economy and all probably desire the same outcomes: low unemployment, low inflation, and high growth. The facts, however, do not lead experts to the same judgment about what to do to achieve the desired outcomes. That requires a theory and balanced evidence about what gets us from a distressed economy to a well-functioning one. Of course, lacking a common theory and biased by personal beliefs, the experts’ predictions will be widely scattered.

Good prediction—and this is my belief—comes from dependence on logic and evidence to draw inferences about the causal path from facts to outcomes. Unfortunately, government, business, and the media assume that expertise—knowing the history, culture, mores, and language of a place, for instance—is sufficient to anticipate the unfolding of events. Indeed, too often many of us dismiss approaches to prediction that require knowledge of statistical methods, mathematics, and systematic research design. We seem to prefer “wisdom” over science, even though the evidence shows that the application of the scientific method, with all of its demands, outperforms experts (remember Johan de Witt). The belief that area expertise, for instance, is sufficient to anticipate the future is, as Tetlock convincingly demonstrated, just plain false. If we hope to build reliable predictions about human behavior, whether in China, Cameroon, or Connecticut, then probably we must first harness facts to the systematic, repeated, transparent application of the same logic across connected families of problems. By doing so we can test alternative ways of thinking to uncover what works and what doesn’t in different circumstances. Here Gardner, Tetlock, and I could not agree more. Prediction tournaments are an essential ingredient to work out what the current limits are to improved knowledge and predictive accuracy. Of course, improvements in knowledge and accuracy will always be a moving target because technology, ideas, and subject adaptation will be ongoing.

Given what we know today and given the problems inherent in dealing with human interaction, what is a leading contender for making accurate, discriminating, useful predictions of complex human decisions? In good hedgehog mode I believe one top contender is applied game theory. Of course there are others but I am betting on game theory as the right place to invest effort. Why? Because game theory is the only method of which I am aware that explicitly compels us to address human adaptability. Gardner and Tetlock rightly note that people are “self-aware beings who see, think, talk, and attempt to predict each other’s behavior—and who are continually adapting to each other’s efforts to predict each other’s behavior, adding layer after layer of new calculations and new complexity.” This adaptation is what game theory jargon succinctly calls “endogenous choice.” Predicting human behavior means solving for endogenous choices while assessing uncertainty. It certainly isn’t easy but, as the example of bandwidth auctions helps clarify, game theorists are solving for human adaptability and uncertainty with some success. Indeed, I used game theoretic reasoning on May 5, 2010 to predict to a large investment group’s portfolio committee that Mubarak’s regime faced replacement, especially by the Muslim Brotherhood, in the coming year. That prediction did not rely on in-depth knowledge of Egyptian history and culture or on expert judgment but rather on a game theory model called selectorate theory and its implications for the concurrent occurrence of logically derived revolutionary triggers. Thus, while the desire for revolution had been present in Egypt (and elsewhere) for many years, logic suggested that the odds of success and the expected rewards for revolution were rising swiftly in 2010 in Egypt while the expected costs were not.

This is but one example that highlights what Nobel laureate Kenneth Arrow, who was quoted by Gardner and Tetlock, has said about game theory and prediction (referring, as it happens, to a specific model I developed for predicting policy decisions): “Bueno de Mesquita has demonstrated the power of using game theory and related assumptions of rational and self-seeking behavior in predicting the outcome of important political and legal processes.” Nice as his statement is for me personally, the broader point is that game theory in the hands of much better game theorists than I am has the potential to transform our ability to anticipate the consequences of alternative choices in many aspects of human interaction.

How can game theory be harnessed to achieve reliable prediction? Acting like a fox, I gather information from a wide variety of experts. They are asked only for specific current information (Who wants to influence a decision? What outcome do they currently advocate? How focused are they on the issue compared to other questions on their plate? How flexible are they about getting the outcome they advocate? And how much clout could they exert?). They are not asked to make judgments about what will happen. Then, acting as a hedgehog, I use that information as data with which to seed a dynamic applied game theory model. The model’s logic then produces not only specific predictions about the issues in question, but also a probability distribution around the predictions. The predictions are detailed and nuanced. They address not only what outcome is likely to arise, but also how each “player” will act, how they are likely to relate to other players over time, what they believe about each other, and much more. Methods like this are credited by the CIA, academic specialists and others, as being accurate about 90 percent of the time based on large-sample assessments. These methods have been subjected to peer review with predictions published well ahead of the outcome being known and with the issues forecast being important questions of their time with much controversy over how they were expected to be resolved. This is not so much a testament to any insight I may have had but rather to the virtue of combining the focus of the hedgehog with the breadth of the fox. When facts are harnessed by logic and evaluated through replicable tests of evidence, we progress toward better prediction.

We can all hope that government, academia, and the media will rally behind Gardner and Tetlock’s pursuit of systematic tests of alternative methods for predicting the future. Methodical tournaments of alternative methods surely will go a long way to advancing our understanding of how logic and evidence can convert mysteries into the known and knowable.

Also from this issue

Lead Essay

  • Dan Gardner and Philip E. Tetlock review the not-too-promising record of expert predictions of political and social phenomena. The truth remains that for all our social science, the world manages to surprise us far more often than not. Rather than giving up or simply declaring in favor of populism, however, they suggest several ways to improve expert predictions, including greater attention to styles of thinking as well as a “forecasting tournament” in which different methodologies will compete against one another to gain empirical data about the process. Still, they concede that our ability to predict the future will probably always be sharply limited.

Response Essays

  • Robin Hanson argues that most people aren’t interested in the accuracy of predictions because predictions often aren’t about knowing the future. They are about affiliating with an ideology or signaling one’s authority. The outcomes of predictions have nothing to do with either, of course, especially in the present. He suggests that one way to make predictions more accurate might be to lift both the social stigma and legal prohibitions against gambling. Unlike mere predictions, wagers carry real consequences for those who make them. Which, Hanson argues, they should.

  • John H. Cochrane offers a limited defense of the hedgehogs: Economics is full of uncertainty because the agents within the system are aware of the theories and possible actions of the other agents. Trying to capture all of them produces a hopeless muddle. Instead, what are needed are explanations of principle and the tendencies that arise all other things being equal. This calls for a hedgehoggy worldview after all. “Especially around policy debates,” he argues, “keeping the simple picture and a few basic principles in mind is the only hope.”

  • We should not be surprised when experts fail to predict the future, says Bruce Bueno de Mesquita. Expertise doesn’t mean good judgment; rather, expertise is an accumulation of many facts about a subject. That we commonly prefer the pronouncements of experts suggests a bias in favor of “wisdom” and against the scientific method. He argues that statistically rigorous game theory can do better by examining the beliefs and objectives of major players in a given situation, and he welcomes forecasting tournaments as a means of refining the method.