Achieving target state-action frequencies in multichain average-reward Markov decision processes

In this paper we address a basic problem that arises naturally in average-reward Markov decision processes (MDPs) with constraints and/or nonstandard payoff criteria: Given a feasible state-action frequency vector (‘the target’), construct a policy whose state-action frequencies match those of the target vector. While it is well known that the solution to this problem cannot, in general, be found in the space of stationary randomized policies, we construct a solution that has ‘ultimately stationary’ structure: It consists of two stationary policies where the first one is used initially, and then the switch to the second one is made at a certain random switching time. The computational effort required to construct this solution is minimal. We also show that our problem can always be solved by a stationary policy if the original MDP is ‘extended’ by adding certain states and actions. The solution in the original MDP is obtained by mapping the solution in the extended MDP back to the original process.