A methodology for fitting and validating metamodels in simulation

This paper proposes a methodology that replaces the usual ad hoc approach to metamodeling. This methodology considers validation of a metamodel with respect to both the underlying simulation model and the problem entity. It distinguishes between fitting and validating a metamodel, and covers four types of goal: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. The methodology consists of a metamodeling process with 10 steps. This process includes classic design of experiments (DOE) and measuring fit through standard measures such as R-square and cross-validation statistics. The paper extends this DOE to stagewise DOE, and discusses several validation criteria, measures, and estimators. The methodology covers metamodels in general (including neural networks); it also gives a specific procedure for developing linear regression (including polynomial) metamodels for random simulation.