Verification of uncertain knowledge-based systems: An empirical verification approach

A number of different tests and approaches are developed to determine the existence of potential anomalies in rule-based systems that employ MYCIN uncertainty factors (weights). First, the distribution of weights is compared with other systems' distributions and weights are investigated as to their individual meanings, to determine whether any weights are unusual. Second, there is increasing evidence that people are not ‘good’ at developing weights on rules, building in symmetries and redundancies that signal ‘usual’ assumptions about the underlying probabilities. Accordingly, weight symmetries generated from rule pairs are analyzed to determine the existence of anomalies. Third, typically rule-based tools have been developed for application in specific domains, such as medicine. Unique aspects of those domains may limit application of the tools to other domains. Finally, ad hoc, rule-based approaches are suboptimal, and alternative formal probability approaches, such as Bayes' nets, more fully specify the probabilistic nature of knowledge. The paper is part of the empirical verification literature, where verification is done on an actual system and the system provides data that indicate the kinds of anomalies that can be expected. A case study is used to illustrate each of the verification tests and concerns.