System comparison procedures for automatic target recognition systems

Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves extensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the difference in two proportions as a sample size-reducing alternative to ranking and selection procedures and confidence intervals. An analysis of the test parameters leads to a practical methodology for implementing the Wald test for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferroni procedure for the multiple pairwise comparison of more than two systems. Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different configurations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing proportions with confidence intervals, particularly when data are expensive.