Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs

From forecasting competitions to conditional value‐at‐risk requirements, the use of multiple quantile assessments is growing in practice. To evaluate them, we use a rule from the general class of proper scoring rules for a forecaster’s multiple quantiles of a single uncertain quantity of interest. The general rule is additive in the component scores. Each component contains a function that measures its quantile’s distance from the realization and weights its contribution to the overall score. To determine this function, we propose that the score of a group’s combined quantile should be better than that of a randomly selected forecaster’s quantile only when the forecasters bracket the realization (i.e., their quantiles do not fall on the same side of the realization). If a score satisfies this property, we say it is sensitive to bracketing. We characterize the class of proper scoring rules that is sensitive to bracketing when the decision maker uses a generalized average to combine forecasters’ quantiles. Finally, we show how weights can be set to match the payoffs in many important business contexts.