Two-stage versus sequential sample-size determination in regression analysis of simulation experiments

Many simulation models have responses with variances that vary with the inputs. Then consider the number of observations (simulation runs, replications) per combination of inputs. These numbers can be selected such that the variances of the average responses become approximately equal: system variants with high variability are simulated more often. These average responses can be analyzed through a regression (meta)model. The regression parameters can be estimated through Weighted Least Squares (WLS). WLS becomes identical to Ordinary Least Squares applied to the average responses. Because the variances are unknown, they are estimated by repeating the runs with different random numbers. The estimated variances yield the number of runs required to obtain approximately equal variances per average response. Two rules for selecting the required number of runs are presented, namely a two-stage and a sequential rule. The stopping rules are first formalized and analyzed; next they are further evaluated through several Monte Carlo experiments. Both rules yield confidence intervals for the estimated regression parameters that have the required coverage probabilities. The sequential rule demands more complicated computations to select the number of runs, but this rule saves simulation runs.