Comments on: Are You Overconfident?

By: Cerebral Mastication » Blog Archive » I don’t even know how wrong I am!

Mon, 12 Apr 2010 15:21:36 +0000

[...] a 90% confidence band around the right answer. If you haven’t seen this done, you should go take and look and then read the rest of this blog [...]

By: zbicyclist

zbicyclist — Fri, 09 Apr 2010 14:20:34 +0000

This is the expected result, but I thought it would apply to others, but not to me, since I knew the literature. More proof education is not a cure ;)

I’m trying to think about this in the context of the Delphi forecasting technique, which asks people for their forecast and their 90% confidence interval. One method of combining forecasts is to combine by the inverse of the expected error variance; but that’s completely wrong if those with a narrow confidence interval aren’t the most knowledgeable, just the most overconfident.

By: Sharad Goel

Sharad Goel — Mon, 15 Mar 2010 16:09:01 +0000

Tg: Don’t feel embarrassed that you didn’t cheat like Andrew :). The point of the exercise is introspection, not “winning”. I guess this is a case in point of misaligned incentives!

By: tg

tg — Mon, 15 Mar 2010 15:23:44 +0000

Dr. Gelman,

As an engineer, I’m embarrassed I didn’t do the same. Ugh.

By: Andrew Gelman

Andrew Gelman — Sun, 14 Mar 2010 20:59:13 +0000

Tg:

I cheated. I gave nine intervals as (-infinity, +infinity) and one interval as (0,0). After all, the request was to be _perfectly calibrated_, not merely to be calibrated in expectation! This was the only way I could be sure of getting “90% of my intervals (no more, no less)” to contain the true values.

By: tg

tg — Tue, 09 Mar 2010 22:19:46 +0000

Dr. Gelman, what was your number?

By: Andrew Gelman

Andrew Gelman — Sat, 06 Mar 2010 21:05:51 +0000

This is an oldie but a goodie, dating back at least to the classic 1968 paper by Alpert and Raiffa. My own version is in section 4 of this article. More recently, there's been some interesting literature on the idea of anchoring and adjustment.

By: Otto

Otto — Tue, 02 Mar 2010 10:54:19 +0000

This was even harder (better?) for me as an European who is not so well versed in Imperial units (“in pounds? why not in euros?”).

By: Aleks

Aleks — Mon, 01 Mar 2010 15:23:22 +0000

The US Marines have a saying “When you’re 70% certain, act!”

So, the test checks for our default calibration. It’s hard to program oneself to operate at a pre-defined calibration level.

By: dreeves

dreeves — Mon, 01 Mar 2010 14:20:26 +0000

A reader disputes one of our answers: (Don’t follow these links till you take the quiz!)

> I am regular reader of your blog. I just tested my (over-)confidence and
> was surprised by your value for the deepest point in the ocean.
>
> Are you sure it’s correct? I found different values, e.g. a different
> one to yours here:
>
> http://www.marianatrench.com/mariana_trench-oceanography.htm

Thanks! We were using the number currently at http://en.wikipedia.org/wiki/Ocean which cites
http://www.rain.org/ocean/ocean-studies-challenger-deep-mariana-trench.html .

I’m now thinking this is the more definitive source, based on what it cites:
http://en.wikipedia.org/wiki/Mariana_Trench

(I guess I’ll edit http://en.wikipedia.org/wiki/Ocean to be consistent. [done])