The effects of feedback and training on the performance of probability forecasters

An experiment examined the effects of outcome feedback and three types of performance feedback-calibration feedback, resolution feedback, and covariance feedback-on various aspects of the performance of probability forecasters. Subjects made 55 forecasts in each of four sessions, receiving feedback prior to making their forecasts in each of the last three sessions. The provision of calibration feedback was effective in improving both the calibration and overforecasting of probability forecasters, but the improvement was not gradual; it occurred in one step, between the second and third sessions. Simple outcome feedback had very little effect on forecasting performance. Neither resolution nor covariance feedback affected forecasters’ performances much differently than outcome feedback. However, unlike outcome feedback, the provision of performance feedback caused subjects to manage their use of the probability scale. Subjects switched from two-digit probabilities to one-digit probabilities, and those receiving calibration and resolution feedback also reduced the number of different probabilities they used.