This is also one of my criticisms of the Servan-Schreiber paper. While I believe there are probably markets in which the availability of serious money to be won could attract people who’d be willing to spend research in order to get a better answer, NFL sports isn’t an arena where spending thousands of dollars will help you uncover facts that aren’t already in the mainstream media.

When we talk about CEO markets, or product release dates, or market penetration numbers, we’re talking about markets in which the information isn’t already out there, and some people will spend time and effort to ferret out relevant facts for reputation (we see this often on Foresight Exchange) or money.

I think the dynamics of elections – which as a rule take place every four years or so – are much harder to capture by using statistical models than sports events are. They occur less frequently, and in very different contexts. What can the outcome of a presidential election in 1948 tell us about the outcome in 2012? A little bit, perhaps, but not much. Traders in prediction markets pay much more attention to the particular context every election takes place, and I believe this is one of the strengths of this method.

Understanding why voters vote the way they do is very hard to capture, especially in advance(!). Wlezien and Erikson claim that polls adjusted with some structural variables are equally good or better than market predictions. That may be true, but if I remember correctly that analysis was made after the result was known, which is another playing field.

I will gladly endorse another method if it outperforms prediciton markets in accuracy, but so far they have been the best tool in predicting the vote ex-ante. Once the market is up an running it is also cost-efficient for the organizer, and produces continuously updated predictions.

Prediction markets for baseball, we predicted, would perform better. Many more statistics are gathered for baseball and conventional wisdom has it that many variables like pitching rotation and recent batting performance of individual players need to be accounted for in predicting game outcomes. But again, simple statistical models that ignore all that do essentially as well as the market.

So maybe sports in general are an exception? That’s why we decided to try another domain that prediction market advocates have pointed to to showcase their efficacy: predicting box office revenue for movies. Same story.

And as we’ve been discussing in the comments above, for political elections it’s deja vu all over again.

So I think you’re making a God of the Gaps argument here.

As for the plots, we evaluate the prediction methods on RMSE and discrimination as well as calibration in the paper.

I agree that with the statistically corrected polls, one has to be careful not to overfit the models. Erikson and Wlezien, however, guard against this by only using data from past elections to predict the outcomes of future elections. While one could still conceivably overfit by testing a large number of models, the model they use seems quite natural.

You’re right that I’m using “insider” too loosely. To clarify, I was alluding to situations (like the terrorist attack example given by Keith) where some small group of hard to identify individuals has crucial information. Such circumstances are quite different from the domains we analyze, and as such, I’m open to the possibility that the relative advantage of markets may be larger in those situations. By contrast, the policy examples of Hahn and Tetlock are much more similar to the sports and entertainment domains we consider. In all three domains, while there are certainly people who are more knowledgeable than others, the relevant information would seem to be much more widely distributed. In particular, I suspect that for policy predictions, a poll of academics, or perhaps estimates compiled by the Congressional Budget Office (CBO), would be about as accurate as a prediction market.

I’d give even odds that a football expert can’t do much better with that fire hose of data than a run-of-the-mill “football enthusiast” can with her knowledge of the past few seasons’ outcomes. But who knows? It’d be neat if your paper plotted the performance of some kind of pool of experts alongside the market and poll methods. If the experts outperform the market, that makes a stronger argument that there exists inside information that the market fails to capture.

“Finally, election markets—perhaps the most cited example of the power of prediction markets—exhibit the same phenomenon. In fact, the Iowa Electronic Markets (IEM) have been been outperformed by statistically adjusted polls [Erikson and Wlezien, 2008]…”

Erikson and Wlezien make a good point that it’s not fair to compare market predictions against raw poll results on a single day. But it’s also not fair to select a statistical adjustment that makes polls perform better in the past five elections, and then use those same five elections to evaluate your adjusted poll’s performance. Naturally, your adjusted poll (and many other arbitrary transformations of the poll data) will outperform the market at predicting the past.

To show that the poll predicts the future, we’ll have to wait decades for a statistically significant number of future elections, or the same statistical adjustment will have to be tested in domains other than US presidential elections.

“In particular, the examples of Hahn and Tetlock—the impact of broader prescription coverage on the Medicare budget, the effect of more frequent audits on tax compliance, and the consequences of a political settlement in Iraq on oil prices—do not fall into this category [of situations where there are insiders].”

I have to disagree. While there may not be “insiders” in the sense of being privy to secret knowledge, there are certainly people whose experience and domain knowledge will enhance their predictive acumen. If you could identify those people and poll them directly, that might give you the best prediction. But the true experts are hard to pick out, and that is where markets are alleged to offer an advantage.

“Thus, while prediction markets may yet prove to be useful, it would seem the enthusiasm for their predictive prowess has outpaced the evidence.

That much is hard to debate!

Also, as Erikson and Wlezien (2008) argue, statistically adjusted polls perform better than the raw polling numbers.