How effective are decision analyses? Assessing decision process and group alignment effects

In this paper, we analyze approaches to assess the effectiveness of decision analyses. We develop an effectiveness framework, categorized in metrics to assess the quality of the decision process (‘process effectiveness’), the quality of the model results (‘output effectiveness’), and the quality of the long–term consequences of the analysis (‘outcome effectiveness’). With a focus on the first two dimensions, ‘process’ and ‘output,’ we introduce two new approaches to assess the effectiveness of decision analyses applied in organizations. First, a new process effectiveness approach serves to compare the effectiveness of decision analyses to existing decision processes. Second, we assess output effectiveness with a before/after preference measurement design, which aids to capture the alignment of group members quantitatively when using decision analysis. We used six case studies, implemented during an applied research project, to test the two measures.