Overtaking and almost-sure optimality for infinite horizon Markov Decision Processes

We consider infinite horizon optimal control of Markov chains on complete metric spaces. We employ the overtaking optimality criterion, which is either applied to the expected cost-now, or to the individual sample paths, yielding almost-sure optimality results. We use the existence of a solution pair (Φ(.)λ) to the optimality equation L Φ(x)=λ to establish and characterize optimal strategies. For finite state-spaces we derive sufficient, as well as necessary conditions for overtaking optimality.