I’m trying to think about this in the context of the Delphi forecasting technique, which asks people for their forecast and their 90% confidence interval. One method of combining forecasts is to combine by the inverse of the expected error variance; but that’s completely wrong if those with a narrow confidence interval aren’t the most knowledgeable, just the most overconfident.

]]>As an engineer, I’m embarrassed I didn’t do the same. Ugh.

]]>I cheated. I gave nine intervals as (-infinity, +infinity) and one interval as (0,0). After all, the request was to be _perfectly calibrated_, not merely to be calibrated in expectation! This was the only way I could be sure of getting “90% of my intervals (no more, no less)” to contain the true values.

]]>