Statistical analysis of computational tests of algorithms and heuristics

Statistical analysis is a powerful tool to apply when evaluating the performance of computer implementations of algorithms and heuristics. Yet many computational studies in the literature do not use this tool to maximum effectiveness. This paper examines the types of data that arise in computational comparisons and presents appropriate techniques for analyzing such data sets. Case studies of computational tests from the open literature are re-evaluated using the proposed methods in order to illustrate the value of statistical analysis for gaining insight into the behavior of the tested algorithms.