Performance measures for selection of metamodels to be used in simulation optimization

This paper points out the need for performance measures in the context of simulation optimization and suggests six such measures. Two of the measures are indications of absolute performance, whereas the other four are useful in assessing the relative performance of various candidate metamodels. The measures assess performance on three fronts: accuracy of placing optima in the correct location, fit to the response, and fit to the character of the surface (expressed in terms of the number of optima). Examples are given providing evidence of the measures' utility – one in a limited scenario deciding which of two competing metamodels to use as simulation optimization response surfaces vary, and the other in a scenario of a researcher developing a new, sequential optimization search procedure.