A modeling framework for replacing medical therapies

A common application of Markov Decision Processes (MDPs) is to determine when to replace machines that stochastically degrade over time. Typical assumptions are that there exists an infinite supply of identical replacements, each of which, upon installation, immediately renews the system to the best state, from which stochastic deterioration proceeds. This paper considers a situation for which these assumptions no longer apply: the replacement of medical therapies so as to maximize a patient's expected lifetime (or quality-adjusted lifetime). For example, upon taking HIV therapy, levels of the virus gradually decrease; however, viral resistance accrues over time, leading to an eventual increase of viral levels. This prompts a switch in therapy; however, there are only a finite number of effective therapies. A model is presented that addresses these challenges to a stationary MDP framework and a general algorithm for scheduling therapies is discussed.