Pitfalls in simulation-based evaluation of forecast monitoring schemes

To evaluate the performance of a forecast monitoring scheme, forecasters have traditionally used a simulation-based estimator of some characteristic of the associated run length distribution. The most frequently cited performance measures are the average run length and the probability that the run length does not exceed a user-specified cutoff point. However, there is disagreement about the definition of run length that is appropriate in the context of forecasting. In this note we present the precise relationships between conflicting formulations both of the average run length and of the probability distribution function for the run length. The practical significance of each of these relationships is discussed. These results bear directly on the way in which simulation experiments should be designed and executed to compare alternative monitoring schemes.