An application of exploratory analysis: The weapon mix problem

Often, model-based military analysis consists of examining the effect of a policy within the context of plausible, best-estimate current or future scenarios. A great deal of time is spent precisely defining all of the data items that make up the scenario. Initial model runs are used to help ‘tune’ the data and establish the base-case. Concern about variations in the data leads to sensitivity analyses to better understand the impact on result measures. Unfortunately, these sensitivity analyses often produce troubling results: Changing the input conditions slightly causes some output values to change dramatically and erratically – and frequently in non-intuitive ways. Analysts and decisionmakers often interpret these outcomes as indicative of errors in the model or data, which sometimes leads to discrediting the model (and the analysis) altogether. In this paper, we show by example that the model and data may not be at fault; rather, the problem may be with how the model is used. In the example, we walk through a traditional analytic approach to the weapon mix problem, yielding the trouble just described. We then perform a different analysis on the same problem, model, and data based on a large number of computational experiments. We show that this kind of analysis provides deeper insights into the problem and greater information for the decisionmaker. We define this new kind of analysis as ‘Exploratory Analysis’, describe it more generally, and discuss its benefits to the decisionmaker and analyst. We also review the requirements for routine use of this methodology.