Almost stationary epsilon-equilibria in zero-sum stochastic games

We show the existence of almost stationary epsilon-equilibria, for all epsilon > 0, in zero-sum stochastic games with finite state and action spaces. These are ε-equilibria with the property that, if neither player deviates, then stationary strategies are played forever with probability almost 1. The proof is based on the construction of specific stationary strategy pairs, with corresponding rewards equal to the value, which can be supplemented with history-dependent delta-optimal strategies, with small delta > 0, in order to obtain almost stationary epsilon-equilibria.