The Tests Are Underway

Robin Hanson’s comments intrigue me. He seems to believe that I am reluctant to subject my method to comparison to alternative methods and that I have not been open to doing so in the past. Nothing could be further from the truth. He does correctly note that academic journals are an open competition but, as he says, “the competition is not on the basis of forecast accuracy.” Well, that is true and not true.

Journal acceptance is presumably based on the quality of the work, including its added-value. Of course, it cannot be based on the accuracy of any forecasts contained in a paper under review since the point of forecasting is to address questions whose answers are not yet known. But here is where Hanson goes astray. As I argued in chapter 10 of The Predictioneer’s Game and in other publications as well (going back as far as the 1980s), researchers should dare to be embarrassed by putting their predictions in print before outcomes are known. Thus, it seems that the “competitors” who have not done so are the ones reluctant to face competition and not me. Peer review is a vehicle for daring to be embarrassed. I have an observable track record in journal publications open to scrutiny regarding accuracy; the bulk of “competitors” seem reluctant to subject themselves to just such evaluation. I urge them to do so.

Hanson is mistaken about me in other important ways. Contrary to the reluctance attributed to me, I and others have several publications testing alternative models against the same data in an open competition of ideas. This includes, for instance, the 1994 volume European Community Decision Making: Model, Applications and Comparisons and the 2006 volume Europe Decides; as well as articles with prominent prospect theorists in peer-reviewed journals, and numerous other published studies. Additionally, I participated in a competition organized by the Defense Threat Reduction Agency in 2003 in which approximately six different quantitative approaches to forecasting were applied to answering the same questions about al Qaeda tactics and targets over a two year window. It happens that my method “won” the competition while also being the least expensive project undertaken by the competitors. Finally, I have provided Philip Tetlock with dozens of studies by my undergraduate students who have enrolled in my course, Solving Foreign Crises, over the past several years in the hope that he will be able to evaluate their accuracy against alternative predictions within the confines of the IARPA project.

It is true that it can be costly to carry out predictions with my method. The data typically come from country or issue experts (who are asked for facts, not judgments) who must be paid for their time (my students manage to assemble data from information available on the web). But it is not true that the cost (in an academic setting) is large or that the model is not readily available for comparative testing. I have made my current model freely available online to bona fide faculty and students for course use or academic research use since 2009. My older model was made freely available online in association with my introductory international relations textbook starting around 1999. Hundreds of scholars and students have signed up for my current model and use it with some regularity. Several publications by others have already resulted from the use of that software. Anyone wishing to test other methods against mine for academic research purposes can register (it is free) to use the online software and conduct the very tests that Hanson calls for. That is why I have made the method available. So, yes I am proud of the accuracy achieved; I am eager for comparative testing; and will be more than happy if an alternative method proves better. That, after all, is how progress is made in the study of political decisionmaking.

Also from this issue

Lead Essay

  • Dan Gardner and Philip E. Tetlock review the not-too-promising record of expert predictions of political and social phenomena. The truth remains that for all our social science, the world manages to surprise us far more often than not. Rather than giving up or simply declaring in favor of populism, however, they suggest several ways to improve expert predictions, including greater attention to styles of thinking as well as a “forecasting tournament” in which different methodologies will compete against one another to gain empirical data about the process. Still, they concede that our ability to predict the future will probably always be sharply limited.

Response Essays

  • Robin Hanson argues that most people aren’t interested in the accuracy of predictions because predictions often aren’t about knowing the future. They are about affiliating with an ideology or signaling one’s authority. The outcomes of predictions have nothing to do with either, of course, especially in the present. He suggests that one way to make predictions more accurate might be to lift both the social stigma and legal prohibitions against gambling. Unlike mere predictions, wagers carry real consequences for those who make them. Which, Hanson argues, they should.

  • John H. Cochrane offers a limited defense of the hedgehogs: Economics is full of uncertainty because the agents within the system are aware of the theories and possible actions of the other agents. Trying to capture all of them produces a hopeless muddle. Instead, what are needed are explanations of principle and the tendencies that arise all other things being equal. This calls for a hedgehoggy worldview after all. “Especially around policy debates,” he argues, “keeping the simple picture and a few basic principles in mind is the only hope.”

  • We should not be surprised when experts fail to predict the future, says Bruce Bueno de Mesquita. Expertise doesn’t mean good judgment; rather, expertise is an accumulation of many facts about a subject. That we commonly prefer the pronouncements of experts suggests a bias in favor of “wisdom” and against the scientific method. He argues that statistically rigorous game theory can do better by examining the beliefs and objectives of major players in a given situation, and he welcomes forecasting tournaments as a means of refining the method.