Rank disagreement: A comparison of multi-criteria methodologies

A number of multi-criteria decision support techniques have emerged in recent years that use varying computational approaches to arrive at the most desirable solution and thereby ‘recommend’ a course of action. Decision makers who use the results of this analytic work should be assured that the computational schemes used by their supporting analysts or decision support software produce the appropriate solutions. The authors conducted a series of experiments that compared the top-ranked options resulting from the computational algorithms that support Multi-Attribute Value Theory (MAVT) and three methods that are reported in the literature that allow rank reversals, the change in rank order of two options when an unrelated option is added or deleted from the analysis: the Analytical Hierarchy Process (AHP), Percentaging and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). They also included a Fuzzy algorithm proposed by Yager to gauge its consistency with the other algorithms, even though it is not subject to rank reversals. These experiments demonstrated that the MAVT and AHP techniques, when provided with the same decision outcome data, very often identify the same alternative as ‘best’. The other techniques are noticeably less consistent with MAVT, the Fuzzy algorithm being the least consistent. The situations under which the most frequent and significant differences occurred were dependent upon the method. The results of the experiments indicate that other issues (e.g. the processes used for problem structuring and the elicitation of value weights) are likely to be of greater significance to problem outcome (based on our experience) than the choice between the computational algorithms of MAVT and AHP. The results cause concern about the use of the other methods.