Are You Overconfident?

Sunday, February 28, 2010
By dreeves

How to estimate the distance from the Earth to the Moon, if you're the Little Prince.

We shall now find out if Messy Matters readers are smarter than Mechanical Turkers. For each of the questions below, provide a numerical range that you are 90% sure contains the correct answer. In particular, if you have “no idea” then give a very wide range; and if you happen to be quite certain then give a narrow range.

If you are perfectly calibrated then 90% of your intervals (no more, no less) should contain the right answers.

Please try your best, but don’t look up the answers!

(If the quiz doesn’t show up in your news reader, click through to the post on messymatters.com.)

We’ll discuss the results in the next post.

PS: Here are the results.

Quiz from Decision Traps by Russo and Shoemaker. Image by Kelly Savage.

Tags: , , , , ,

  • http://ai.eecs.umich.edu/people/dreeves dreeves

    A reader disputes one of our answers: (Don’t follow these links till you take the quiz!)

    > I am regular reader of your blog. I just tested my (over-)confidence and
    > was surprised by your value for the deepest point in the ocean.
    >
    > Are you sure it’s correct? I found different values, e.g. a different
    > one to yours here:
    >
    > http://www.marianatrench.com/mariana_trench-oceanography.htm

    Thanks! We were using the number currently at http://en.wikipedia.org/wiki/Ocean which cites
    http://www.rain.org/ocean/ocean-studies-challenger-deep-mariana-trench.html .

    I’m now thinking this is the more definitive source, based on what it cites:
    http://en.wikipedia.org/wiki/Mariana_Trench

    (I guess I’ll edit http://en.wikipedia.org/wiki/Ocean to be consistent. [done])

  • http://stat.columbia.edu/~gelman/blog/ Aleks

    The US Marines have a saying “When you’re 70% certain, act!”

    So, the test checks for our default calibration. It’s hard to program oneself to operate at a pre-defined calibration level.

  • Otto

    This was even harder (better?) for me as an European who is not so well versed in Imperial units (“in pounds? why not in euros?”).

  • http://www.stat.columbia.edu/~gelman/blog/ Andrew Gelman

    This is an oldie but a goodie, dating back at least to the classic 1968 paper by Alpert and Raiffa. My own version is in section 4 of this article. More recently, there’s been some interesting literature on the idea of anchoring and adjustment.

  • tg

    Dr. Gelman, what was your number?

  • http://www.stat.columbia.edu/~gelman/blog/ Andrew Gelman

    Tg:

    I cheated. I gave nine intervals as (-infinity, +infinity) and one interval as (0,0). After all, the request was to be _perfectly calibrated_, not merely to be calibrated in expectation! This was the only way I could be sure of getting “90% of my intervals (no more, no less)” to contain the true values.

  • tg

    Dr. Gelman,

    As an engineer, I’m embarrassed I didn’t do the same. Ugh.

  • http://www.cam.cornell.edu/~sharad/ Sharad Goel

    Tg: Don’t feel embarrassed that you didn’t cheat like Andrew :). The point of the exercise is introspection, not “winning”. I guess this is a case in point of misaligned incentives!

  • http://mikekr.blogspot.com zbicyclist

    This is the expected result, but I thought it would apply to others, but not to me, since I knew the literature. More proof education is not a cure ;)

    I’m trying to think about this in the context of the Delphi forecasting technique, which asks people for their forecast and their 90% confidence interval. One method of combining forecasts is to combine by the inverse of the expected error variance; but that’s completely wrong if those with a narrow confidence interval aren’t the most knowledgeable, just the most overconfident.

  • Pingback: Cerebral Mastication » Blog Archive » I don’t even know how wrong I am!

  • coxy-the1st

    sorry i am not a numbers person but have this as a homework assignment for behavioral finance what did you put for best guess? The correct answer?

  • Joe

    There is a fundamental problem with these types of quiz games. I can win all the time by simply having 9 huge, absolutely safe intervals (Mozart was born between the Big Bang and today, etc.) and for a single question I give a blatantly wrong answer, like 1-2 nanometers for the Moon’s diameter.

  • http://beeminder.com Daniel Reeves

    It’s true. Andrew Gelman made that point here in the comments as well. I guess for this version you’re on your honor to not do that. You could mitigate that problem by asking the questions one at a time and not revealing how many will be coming. It’s actually a really interesting question how to make the incentives exactly right. I’ll leave that as an exercise for the reader!

  • Pingback: Estimation: why is it inaccurate and how can it be fixed? | NiNeTY-7