A decision analysis method for evaluating computer intrusion detection systems

This paper presents a decision analysis method for evaluating computer intrusion detection systems. The method integrates and extends receiver operating characteristic (ROC) and cost analysis methods to provide an expected cost metric. We demonstrate that both the ROC analysis and cost analysis methods are incomplete. Furthermore, we demonstrate how a decision tree can combine and extend the ROC and cost analysis methods to provide an expected cost metric that reflects the intrusion detection system's ROC curve, costs, and assessments of the hostility of the environment as summarized by the prior probability of intrusion. We further demonstrate how this method can be used to decide the optimal operating point on an intrusion detector's ROC curve, choose the best intrusion detection system, compare the value of one intrusion detection system with another's, determine the value of an intrusion detector over no detector, and determine how to adjust the operation of an intrusion detector to respond to changes in its environment. General results are given and the method is illustrated in several numerical examples that involve both hypothetical and real intrusion detection systems. We demonstrate that, contrary to common advice, the value of an intrusion detection system depends not only on its ROC curve, but also on various costs (such as those associated with making incorrect decisions about detection) and the hostility of the operating environment. Conclusions are drawn about the design and evaluation of intrusion detection systems and the role for decision analysis in that design and evaluation.