Integrating evaluation methods into the DSS development process

Informal evaluations are an inherent part of developing decision support systems (DSSs) because judgment and uncertainty are an inherent part of the DSS development process. As a result, formal evaluation activities need to be capable of rigorously addressing the different information needs of different members of the sponsoring and development teams throughout the DSS development process. This paper addresses that need by discussing the applicability of three classes of evaluation methods to different stages in the DSS design and development process: subjective evaluation methods for obtaining users’ opinion regarding the DSS’s strengths and weaknesses; technical evaluation methods for ‘looking inside the black box’; and empirical evaluation methods for assessing the DSS’s impact on performance. In addition, the paper presents a decision making paradigm as a backdrop for considering the design to develop a DSS, and the role of evaluation with respect to this decision. The perspective throughout the paper is that evaluation represents the control mechanism in the DSS design and development process because it provides feedback that keeps the development effort on track.