Derivative estimates from simulation of continuous-time Markov chains

Countable-state, continuous-time Markov chains are often analyzed through simulation when simple analytical expressions are unavailable. Simulation is typically used to estimate costs or performance measures associated with the chain and also characteristics like state probabilities and mean passage times. This paper considers the problem of estimating derivatives of these types of quantities with respect to a parameter of the process. In particular, it considers the case where some or all transition rates depend on a parameter. The paper derives derivative estimates of the infinitesimal perturbation analysis type for Markov chains satisfying a simple condition, and urge that the condition has significant scope. The unbiasedness of these estimates may be surprising-a ‘naive’ estimator would fail in the present setting. What makes these estimates work is a special construction of specially structured parametric families of Markov chains. In addition to proving unbiasedness, the paper considers a variance reduction technique and makes comparisons with derivative estimates based on likelihood ratios.