Consistency and accuracy in decision aids: Experiments with four multiattribute systems

There have been a number of multiattribute decision aids developed to aid selection problems. Multiattribute value theory and the analytic hierarchy process are two commonly used techniques. Different systems can result in radically different conclusions if they inaccurately and inconsistently reflect the preference structure of decision makers, or if they are based on inappropriate theoretical models. This study examines the impact of the underlying theoretical model, the method in which preference information is elicited, and the structure of alternatives as influences on the results from using various decision aids. It was found that two systems based on the multiattribute value theory model were just as diverse in their conclusions as were results between analytic hierarchy process and the multiattribute value theory models. Therefore, accuracy of information reflecting decision maker preference is an important consideration. Feedback capable of assuring the decision maker that information provided is consistent is a necessary feature required of decision aids applied to selection problems. The study also found that the way in which information is elicited influenced the result more than did the underlying model. Exact numerical data for complex concepts such as attribute importance and alternative performance on attributes are not necessary, and elicitation procedures that are more natural for the user are likely to be more accurate.