Validation of trace-driven simulation models: A novel regression test

Validation of trace-driven simulation models: A novel regression test

0.00 Avg rating0 Votes
Article ID: iaor19992087
Country: United States
Volume: 44
Issue: 6
Start Page Number: 812
End Page Number: 819
Publication Date: Jun 1998
Journal: Management Science
Authors: , ,
Keywords: statistics: regression, queues: theory
Abstract:

This paper argues that it is wrong to require that regressing the outputs of a trace-driven simulation on the observed real outcomes should give a 45° (unit slope) line through the origin (zero intercept). This note proposes instead an alternative requirement: the responses of the simulated and the real systems should have the same means and the same variances. To test statistically whether this requirement is satisfied, a novel procedure is derived: regress the differences between simulated and real responses on their associated sums, and test whether the resulting intercept and slope are both zero. This novel but simple test assumes identically, independently, and normally distributed outputs of the real system and the simulated system. The old and the new procedures are investigated in extensive Monte Carlo experiments that simulate M/M/1 queueing systems. The conclusions are: (i) the naive intuitive test rejects a valid simulation model substantially more often than the novel test does; (ii) the naive test shows ‘perverse’ behavior within a certain domain: the worse the simulation model, the higher its estimated probability of acceptance; and (iii) the novel test does not reject a valid simulation model too often (its type I error probability is correct), provided the queueing response is transformed appropriately to obtain (nearly) normal distributed responses.

Reviews

Required fields are marked *. Your email address will not be published.