Are You Overconfident?
We shall now find out if Messy Matters readers are smarter than Mechanical Turkers. For each of the questions below, provide a numerical range that you are 90% sure contains the correct answer. In particular, if you have “no idea” then give a very wide range; and if you happen to be quite certain then give a narrow range.
If you are perfectly calibrated then 90% of your intervals (no more, no less) should contain the right answers.
Please try your best, but don’t look up the answers!
(If the quiz doesn’t show up in your news reader, click through to the post on messymatters.com.)
We’ll discuss the results in the next post.
Quiz from Decision Traps by Russo and Shoemaker.
Image by Kelly Savage.
A reader disputes one of our answers: (Don’t follow these links till you take the quiz!)
> I am regular reader of your blog. I just tested my (over-)confidence and
> was surprised by your value for the deepest point in the ocean.
>
> Are you sure it’s correct? I found different values, e.g. a different
> one to yours here:
>
> http://www.marianatrench.com/mariana_trench-oceanography.htm
Thanks! We were using the number currently at http://en.wikipedia.org/wiki/Ocean which cites
http://www.rain.org/ocean/ocean-studies-challenger-deep-mariana-trench.html .
I’m now thinking this is the more definitive source, based on what it cites:
http://en.wikipedia.org/wiki/Mariana_Trench
(I guess I’ll edit http://en.wikipedia.org/wiki/Ocean to be consistent. [done])
The US Marines have a saying “When you’re 70% certain, act!”
So, the test checks for our default calibration. It’s hard to program oneself to operate at a pre-defined calibration level.
This was even harder (better?) for me as an European who is not so well versed in Imperial units (“in pounds? why not in euros?”).
This is an oldie but a goodie, dating back at least to the classic 1968 paper by Alpert and Raiffa. My own version is in section 4 of this article. More recently, there’s been some interesting literature on the idea of anchoring and adjustment.
Dr. Gelman, what was your number?
Tg:
I cheated. I gave nine intervals as (-infinity, +infinity) and one interval as (0,0). After all, the request was to be _perfectly calibrated_, not merely to be calibrated in expectation! This was the only way I could be sure of getting “90% of my intervals (no more, no less)” to contain the true values.
Dr. Gelman,
As an engineer, I’m embarrassed I didn’t do the same. Ugh.
Tg: Don’t feel embarrassed that you didn’t cheat like Andrew :). The point of the exercise is introspection, not “winning”. I guess this is a case in point of misaligned incentives!
This is the expected result, but I thought it would apply to others, but not to me, since I knew the literature. More proof education is not a cure ;)
I’m trying to think about this in the context of the Delphi forecasting technique, which asks people for their forecast and their 90% confidence interval. One method of combining forecasts is to combine by the inverse of the expected error variance; but that’s completely wrong if those with a narrow confidence interval aren’t the most knowledgeable, just the most overconfident.
[...] a 90% confidence band around the right answer. If you haven’t seen this done, you should go take and look and then read the rest of this blog [...]