Convergence of controlled models and finite-state approximation for discounted continuous-time Markov decision processes with constraints

In this paper we consider the convergence of a sequence $\{{\mathcal{M}}_n\}$ of the models of discounted continuous‐time constrained Markov decision processes (MDP) to the ‘limit’ one, denoted by ${\mathcal{M}}_{\infty }$ . For the models with denumerable states and unbounded transition rates, under reasonably mild conditions we prove that the (constrained) optimal policies and the optimal values of $\{{\mathcal{M}}_n\}$ converge to those of ${\mathcal{M}}_{\infty }$ , respectively, using a technique of occupation measures. As an application of the convergence result developed here, we show that an optimal policy and the optimal value for countable‐state continuous‐time MDP can be approximated by those of finite‐state continuous‐time MDP. Finally, we further illustrate such finite‐state approximation by solving numerically a controlled birth‐and‐death system and also give the corresponding error bound of the approximation.