Measuring the quality of computer model performance

One of the critical validation issues is determining the quality of the performance of a computer model compared to human experts. This paper addresses that basic evaluation issue using a set of statistical methodologies that fall under the umbrella of ‘computer intensive statistics’, including enumeration and randomization. In some cases, computer intensive methods can be used to mitigate some of the limitations of small sample sizes, inherent in the comparison of systems to humans, since human expertise is limited. In addition, computer intensive methods make no distribution assumption, thus, the problems of the data not meeting the underlying parameter or distributional assumptions also can be mitigated. As a result, computer intensive methods can be used to supplement or replace traditional, distribution-based-methods. The evaluation data from two different studies is analyzed in detail. The investigation in one case study finds that the quality of the decision process is independent of outcome performance. The evaluation in the other study indicates that some parts of the system apparently have more impact on user’s judgements than other parts.