A new methodology for evaluating incident detection algorithms

We present a novel, off-line approach for evaluating incident detection algorithms. Previous evaluations have focused on determining the detection rate versus false alarm rate curve – a process which we argue is inherently fraught with difficulties. Instead, we propose a cost-benefit analysis where cost mimics the real costs of implementing the algorithm and benefit is in terms of reduction in congestion. We argue that these quantities are of more practical interest than the traditional rates. Moreover, these costs, estimated on training data, can be used both as a mechanism to fine tune a single algorithm as well as a meaningful quantity for direct comparisons between different types of incident detection algorithms. We demonstrate our approach with a detailed example.