Prediction Without Markets

Thursday, January 14, 2010
By Sharad Goel

photo-finish

In the 2008 Summer Olympics Usain Bolt ran 100 meters in 9.69 seconds, earning the gold medal and garnering the international attention that comes with being the “fastest man in the world.” While Bolt became a household name, his competitors didn’t fare nearly as well: far fewer people know that Richard Thompson and Walter Dix received silver and bronze, and I suspect that 8th place Darvis Patton is practically unknown outside the sprinting world. The 340 milliseconds that separated Bolt from Patton—the duration of a blink of the eye—was the difference between celebrity and obscurity. While a fascination with rank is perhaps justifiable for sports, such focus—let’s call it the Gold Medal Syndrome—is often problematic in statistical analysis.

Consider the case of prediction markets. In these markets, participants buy and sell securities that realize a value based on the occurrence of some future outcome, such as the result of an election, the box office revenue of an upcoming film, or the market share of a new product. For example, the day before the 2008 U.S. presidential election you could have paid $0.92 for a contract in the Iowa Electronic Markets that yielded $1 when Barack Obama won, implying a 92% market-estimated probability that Obama would win. There are compelling theoretical reasons to expect prediction markets to outperform all other available forecasting methods. As formalized by the efficient-market hypothesis, if there were some way to beat the market then at least one savvy trader would presumably exploit that advantage to make money; hence, market prices should update to eliminate any performance disparity.

Inspired by such theoretical arguments, and also by a growing body of empirical findings that show markets beat alternatives, several researchers have called for widespread application of prediction markets to real-world business strategy and policy development problems. In a 2007 Wall Street Journal op-ed, economists Robert Hahn and Paul Tetlock write:

“Imagine the president had a crystal ball to predict more accurately the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance—or even the consequences of a political settlement in Iraq on oil prices. Now, stop imagining: Such crystal balls [prediction markets] are within our grasp.”

The work on which these appeals are based, however, primarily addresses the relative ranking of prediction methods. By contrast, the magnitude of the differences in question has received much less attention, and as such, it remains unclear whether the performance improvement associated with prediction markets is meaningful from a practical perspective.

In a new study, Daniel Reeves, Duncan Watts, Dave Pennock and I compare the performance of prediction markets to conventional means of forecasting, namely polls and statistical models. Examining thousands of sporting and movie events, we find that the relative advantage of prediction markets is remarkably small. For example, the Las Vegas market for professional football is only 3% more accurate in predicting final game scores than a simple, three parameter statistical model, and the market is only 1% better than a poll of football enthusiasts. The plot below shows how the three methods perform on the complementary task of estimating the probability the home team wins.

mm_calibration

Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms. Nevertheless, it is certainly possible that there are forecasting applications where either the relative advantage of markets is larger, or that such differences in performance are consequential. Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

NB: Check out our paper for more details.

Illustration by Kelly Savage

Tags: , , ,

  • http://www.intrade.com Abacus Mike

    I think there is a paper by Prof Justin Wolfers that says political predictions from http://www.intrade.com (far bigger and more liquid than Iowa Electronic Markets) have only half the error % that the same predictions from Gallup Opinion Polls.

    If you Google “wolfers prediction market gallup” it is the first link.

  • Keith Adams

    From the conclusion:

    “Given that sports and entertainment markets are among the most mature and successful, our results challenge the view that prediction markets are substantively superior to alternative forecasting mechanisms.”

    The sentence makes it sound as though these “mature” markets provide the best possible environment for a prediction market. I think the opposite would be true; since luck plays such a huge role in sports and entertainment outcomes, there can be no true “insiders.” Prediction markets shine when someone has a secret, exploitable information edge, which is nearly impossible almost exclusively in sports and entertainment. Just because you worked on a movie and thought it sucked is not a reason to bet with high confidence that it will lose money; substitute your favorite sports analogy here….

  • Cameron Marlow

    Wait–you’re saying that there’s no inside information on horse races?! Then why would anyone bet on them? That’s preposterous.

    It’s true; I’m sure that the betting data was most easily available, but it would be great to see them replicate the study in some domain where insiders exist. Facebook IPO date?!

  • Prasad Chakka

    didn’t prediction markets prove not so useful even in political arena as well?

  • Ken Schmidt

    Good discussion. Is there an emerging counter movement to the recent social media craze? E.g.: http://on.wsj.com/5cqmWS

  • http://www.cam.cornell.edu/~sharad/ Sharad Goel

    @Keith: While it may be true that one shouldn’t reasonably expect sports and entertainment markets to outperform simple forecasting techniques, prediction market experts, including Cass Sunstein, Justin Wolfers, and Dave Pennock, have pointed to these domains as evidence that markets work. Particularly in sports, given the plethora of available data (e.g., individual player stats, pitching rotation, weather conditions, etc.) on which to base complex predictions, one would think experts, and hence markets, might also have a more substantial advantage. Finally, election markets—perhaps the most cited example of the power of prediction markets—exhibit the same phenomenon. In fact, the Iowa Electronic Markets (IEM) have been been outperformed by statistically adjusted polls [Erikson and Wlezien, 2008]—although, again, for me it’s not terribly important which method comes out on top, but rather that markets are comparable to alternatives.

    It may well be the case that prediction markets are most effective in situations where there are insiders who know what is going to happen, and who simply need to be incentivized to reveal their knowledge. Such instances, though, constitiute only a fraction of the applications where the promise of prediction markets is heralded. In particular, the examples of Hahn and Tetlock—the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance, and the consequences of a political settlement in Iraq on oil prices—do not fall into this category.

  • http://www.cam.cornell.edu/~sharad/ Sharad Goel

    @Abacus: Could you point me to the paper that you’re talking about? The paper I found (“Prediction Markets in Theory and Practice,” by Wolfers and Zitzewitz) states that Gallup polls had absolute error of 1.9% in predicting vote share on the eve of presidential elections, while the IEM had an error of 1.6%. I didn’t see the Intrade vs. Gallup comparison.

    Also, as Erikson and Wlezien (2008) argue, statistically adjusted polls perform better than the raw polling numbers.

  • Tom Brow

    “Particularly in sports, given the plethora of available data (e.g., individual player stats, pitching rotation, weather conditions, etc.) on which to base complex predictions, one would think experts, and hence markets, might also have a more substantial advantage.”

    I’d give even odds that a football expert can’t do much better with that fire hose of data than a run-of-the-mill “football enthusiast” can with her knowledge of the past few seasons’ outcomes. But who knows? It’d be neat if your paper plotted the performance of some kind of pool of experts alongside the market and poll methods. If the experts outperform the market, that makes a stronger argument that there exists inside information that the market fails to capture.

    “Finally, election markets—perhaps the most cited example of the power of prediction markets—exhibit the same phenomenon. In fact, the Iowa Electronic Markets (IEM) have been been outperformed by statistically adjusted polls [Erikson and Wlezien, 2008]…”

    Erikson and Wlezien make a good point that it’s not fair to compare market predictions against raw poll results on a single day. But it’s also not fair to select a statistical adjustment that makes polls perform better in the past five elections, and then use those same five elections to evaluate your adjusted poll’s performance. Naturally, your adjusted poll (and many other arbitrary transformations of the poll data) will outperform the market at predicting the past.

    To show that the poll predicts the future, we’ll have to wait decades for a statistically significant number of future elections, or the same statistical adjustment will have to be tested in domains other than US presidential elections.

    “In particular, the examples of Hahn and Tetlock—the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance, and the consequences of a political settlement in Iraq on oil prices—do not fall into this category [of situations where there are insiders].”

    I have to disagree. While there may not be “insiders” in the sense of being privy to secret knowledge, there are certainly people whose experience and domain knowledge will enhance their predictive acumen. If you could identify those people and poll them directly, that might give you the best prediction. But the true experts are hard to pick out, and that is where markets are alleged to offer an advantage.

    “Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

    That much is hard to debate!

  • http://www.cam.cornell.edu/~sharad/ Sharad Goel

    @Tom: Our results do in fact suggest that the firehose of sports data does little to improve predictions. If those data did lead to big improvements, one would expect that markets would also perform substantially better than our simple model (which we don’t see). Maybe that fact should have been “obvious” before we completed our study; regardless, prediction market experts have pointed to sports as a domain that showcases the power of markets, and it is that claim that we address.

    I agree that with the statistically corrected polls, one has to be careful not to overfit the models. Erikson and Wlezien, however, guard against this by only using data from past elections to predict the outcomes of future elections. While one could still conceivably overfit by testing a large number of models, the model they use seems quite natural.

    You’re right that I’m using “insider” too loosely. To clarify, I was alluding to situations (like the terrorist attack example given by Keith) where some small group of hard to identify individuals has crucial information. Such circumstances are quite different from the domains we analyze, and as such, I’m open to the possibility that the relative advantage of markets may be larger in those situations. By contrast, the policy examples of Hahn and Tetlock are much more similar to the sports and entertainment domains we consider. In all three domains, while there are certainly people who are more knowledgeable than others, the relevant information would seem to be much more widely distributed. In particular, I suspect that for policy predictions, a poll of academics, or perhaps estimates compiled by the Congressional Budget Office (CBO), would be about as accurate as a prediction market.

  • http://overcomingbias.com Robin Hanson

    Whoever said that every prediction market would always be more accurate than any other mechanism? I’d say they are more-accurate more often than they are less-accurate, compared to mechanisms with similar resources. And your plots at the bottom look like they are testing calibration, not accuracy.

  • http://ai.eecs.umich.edu/people/dreeves dreeves

    @Robin Hanson: It’s true that the relative advantage of prediction markets may be more pronounced in other domains. Indeed, when we first analyzed the football data we thought maybe football games are just particularly unpredictable. (Quoting ourselves from the paper: “it is possible that football remains a special case even in the domain of sports in that outcomes are dominated by hard to anticipant events—a hail Mary pass in the final minutes, for example, or an intercepted ball against the flow of play—for which there is relatively little real information on which to base sophisticated predictions.”)

    Prediction markets for baseball, we predicted, would perform better. Many more statistics are gathered for baseball and conventional wisdom has it that many variables like pitching rotation and recent batting performance of individual players need to be accounted for in predicting game outcomes. But again, simple statistical models that ignore all that do essentially as well as the market.

    So maybe sports in general are an exception? That’s why we decided to try another domain that prediction market advocates have pointed to to showcase their efficacy: predicting box office revenue for movies. Same story.

    And as we’ve been discussing in the comments above, for political elections it’s deja vu all over again.

    So I think you’re making a God of the Gaps argument here.

    As for the plots, we evaluate the prediction methods on RMSE and discrimination as well as calibration in the paper.

  • David K. Park

    Nice to have more data points showing that prediction markets are not superior to other alternative forecasting methods. All the examples in the paper have a “large” sample, however, if you had a relatively “small” amount of data, which forecasting model would you choose? Prediction markets or something else?

  • S. Arnesen

    Thank you for an interesting paper. As a political scientist I will comment on your discussion on political predictions:
    I would question the validity in analyzing entertainment and sports events to make claims about the accuracy of political elections. You indeed acknowledge this in the paper, but maintain your claim that this paper has relevance for political forecasts (by referring to other studies). I disagree with this approach.

    I think the dynamics of elections – which as a rule take place every four years or so – are much harder to capture by using statistical models than sports events are. They occur less frequently, and in very different contexts. What can the outcome of a presidential election in 1948 tell us about the outcome in 2012? A little bit, perhaps, but not much. Traders in prediction markets pay much more attention to the particular context every election takes place, and I believe this is one of the strengths of this method.

    Understanding why voters vote the way they do is very hard to capture, especially in advance(!). Wlezien and Erikson claim that polls adjusted with some structural variables are equally good or better than market predictions. That may be true, but if I remember correctly that analysis was made after the result was known, which is another playing field.

    I will gladly endorse another method if it outperforms prediciton markets in accuracy, but so far they have been the best tool in predicting the vote ex-ante. Once the market is up an running it is also cost-efficient for the organizer, and produces continuously updated predictions.

  • http://pancrit.org Chris Hibbert

    Top tier sports, national elections, and Hollywood releases are all arenas in which all the information one might analyze is already pretty much public. There are many methods for predicting these outcomes, and I wouldn’t argue that Prediction Markets have a huge advantage in these arenas. The markets where I expect PMs to have an advantage are where there are experts who, given an incentive, could share (or discover) information that’s not already public, and where you don’t already have an enormous crowd trying to figure out the answer. Certainly it’s fun to bet on your team or party, or to develop expertise on how the public will react to particular movies, but it’s not clear to me that we get better predictions in those areas.

    This is also one of my criticisms of the Servan-Schreiber paper. While I believe there are probably markets in which the availability of serious money to be won could attract people who’d be willing to spend research in order to get a better answer, NFL sports isn’t an arena where spending thousands of dollars will help you uncover facts that aren’t already in the mainstream media.

    When we talk about CEO markets, or product release dates, or market penetration numbers, we’re talking about markets in which the information isn’t already out there, and some people will spend time and effort to ferret out relevant facts for reputation (we see this often on Foresight Exchange) or money.

  • Pingback: Gates Hillman Prediction Market: The Movie: Oddhead Blog: Prediction Markets, Gambling, Electronic Commerce, Artificial Intelligence: David Pennock: Yahoo! Research

  • http://lumma.org/microwave Carl Lumma

    First of all, 3% is huge, and even 1% is significant on many tasks. Secondly, prediction markets have other important advantages, such as being harder to manipulate, and being self-funding.

  • Pingback: It’s Arab Spring, but is it Prediction Market Winter? » Oddhead Blog

  • Pingback: Interesting Links for 08-08-2017 | Made from Truth and Lies