Validation of regression metamodels in simulation: Bootstrap approach

Simulation experiments are often analyzed through a linear regression model of their input/output data. Such an analysis yields a metamodel or response surface for the underlying simulation model. This metamodel can be validated through various statistics; this article studies (1) the coefficient of determination (R-square) for generalized least squares, and (2) a lack-of-fit F-statistic originally formulated by Rao, who assumed multivariate normality. To derive the distributions of these two validation statistics, this paper shows how to apply bootstrapping – without assuming normality. To illustrate the performance of these bootstrapped validation statistics, the paper uses Monte Carlo experiments with simple models. For these models (i) R-square is a conservative statistic (rejecting a valid metamodel relatively rarely), so its power is low; (ii) Rao's original statistic may reject a valid metamodel too often; (iii) bootstrapping Rao's statistic gives only slightly conservative results, so its power is relatively high.