Priority index heuristic for multi-armed bandit problems with set-up costs and/or set-up time delays

A dynamic scheduling heuristic is derived to solve the multi-armed bandit problem (MABP) with set-up costs and/or time delays. While our heuristic is based on priority indices of the Gittins' type, the presence of set-up costs requires a couple of indices to be defined for each project of the MABP. Doubling the number of priority indices enables hysteretic buffers to be obtained in the scheduling diagram. In the presence of set-up costs, hysteresis zones are clearly mandatory to avoid high-frequency switchings between the various projects of the MABP. The performance of our general heuristic is tested on the class of deteriorating MABP for which the explicit optimal scheduling can be constructed.