MARKOV MODEL EXAMPLE: ADVANCED PROSTATE CANCER
Figure 4.3 depicts a simplified model of the treatment of hormonenaive advanced prostate cancer with abiraterone acetate (AA). This model and its attendant data are drawn from Ramamurthy et al.’s study of the costs and projected benefits of AA versus docetaxel in the treatment of this malignancy, to which the reader is referred for the complete model and costeffectiveness analysis [5]. For this chapter, the model and data are simplified in favor of didactic value.
In Figure 4.3, states are represented for Stable on AA (denoted AAStable in the figure), AA with the side effect of fatigue (AAFatigue), progressive disease (PD), and Death (Dead). For clarity, arrows from states to themselves have not been drawn. The figure thus depicts the principal transitions in the model. The largest statetostate transition is from progressive disease to death. Table 4.8 contains the initial monthly cycle transition probability matrix for a 68yearold man (the median age in the clinical trials on which the model is based). This table is calculated from baseline estimates given by Ramamurthy et al.
Note that from AAStable, the most likely result after one month is to stay in AAStable, but there is a little greater than 2% chance of transitioning to PD. Additionally, there is a 0.3% chance of developing fatigue within the first six months of treatment, which would decrease quality of life for up to a year. A timedependent general population risk of death must also be added to the model. At 68 years of age, the annual risk of death is 1.83%, rising over time according to the Gompertz exponential function. At 84, the risk of death is 10%. Table 4.9 shows the experience over 12 monthly cycles of 10,000 men aged
FIGURE 4.3 Principal transitions in prostate cancer model. Transitions to same state (e.g., WellWell) are not shown.
TABLE 4.8
Transition Probability Matrix for the Prostate Cancer Model
AAStable 
AAFatigue 
PD 
Dead 

AAStablc 
0.9746 
0.0030 
0.0208 
0.0016 
AAFatiguc 
0 
0.9776 
0.0208 
0.0016 
PD 
0 
0.9795 
0.0205 

Dead 
0 
1 
68, treated with AA, according to the Markov model with the rising general death rate. In one month, 254 men have transitioned out of the AAStable state, with the greatest fraction to PD. The number of men in AAFatigue rises over the first six months of treatment, and then decreases as no more are added to this state and patients recover from treatmentinduced fatigue. The number of men in PD rises each month, but as fewer men are on AA treatment with each passing month, the increase lessens with each month. In fact, after 32 monthly
Expected State Membership of Markov Cohort Over 12 Monthly Cycles
TABLE 4.9
Month 
AAStable 
AAFatigue 
PD 
Dead 
0 
10,000 
0 
0 
0 
1 
9,746 
30 
208 
16 
2 
9,498 
59 
407 
36 
3 
9,257 
86 
598 
60 
4 
9,022 
112 
780 
87 
5 
8,793 
136 
954 
117 
6 
8,570 
159 
1,120 
151 
7 
8,393 
140 
1,278 
188 
8 
8,219 
123 
1,430 
228 
9 
8,047 
109 
1,574 
271 
10 
7,877 
96 
1,711 
316 
11 
7,710 
84 
1,842 
364 
12 
7,546 
74 
1,966 
414 
cycles (not shown), the number of men in the PD state peaks, then begins to decline as death due to progressive cancer and the general population mortality takes a greater toll.
Over an expected lifetime, the Markov model yields a membership of each state as shown in Figure 4.4. The AAStable cohort declines steadily as men transition to one of the other model states, and back transitions from AA Fatigue are never significant. AAFatigue rises over the first six months of the model, then declines to zero by Month 18. PD rises steadily to Month 32, and then falls slowly as patients die from their disease. The Dead state shows a typical sigmoid function, as initially few die, then the rate increases as more men reach PD, and then the rate slows as fewer men remain alive.
Baseline results from this model are presented in Table 4.10. Averaged over a cohort, the patient with advanced prostate cancer treated with AA can expect to live 32.9 months with stable disease. The overall 2% risk of fatigue, averaged across the entire cohort, adds 0.2 months per man in AAFatigue. A further 19.5 months are expected in PD, for an overall life expectancy of
52.6 months. Of course, no single patient has precisely this experience.
Sensitivity analysis (see Chapter 13) can be conducted on Markov transition probabilities, and modern software easily supports this. An Appendix to this chapter provides R code developed by the authors that can be used to replicate the model over the initial five years, and can be adapted to change parameters for sensitivity analysis or extend the model time [10].
Ramamurthy et al.’s more complete Markov formulation incorporates quality adjustments and a cost model, and is truncated to five years based on the
FIGURE 4.4 Natural history of prostate cancer example.
TABLE 4.10
Expected Results of Abiraterone Treatment Model
AAStable 
AAFatigue 
PD 
Total 

Expected months 
32.9 
0.2 
19.5 
52.6 
clinical trial results. Later chapters in this book will illustrate how costs and structural interventions can modify Markov and other stochastic models to generate sophisticated analyses of pharmacoeconomic problems.
Software Implementation
The modeler has a choice of different types of software to run a Markov model. TreeAge Pro is a specialized program for decision analysis with a full graphical user interface, where models are set up via “pointandclick” options. This interface makes it easy to work with, but the proprietary software limits transparency. It also may be challenging to set up especially complex models via a graphical interface, and model specifications are limited to the options available in the software [11]. Alternatively, the modeler could use a general purpose programming language to implement the model. Here we focus on R (R Foundation for Statistical Computing), but many other programming languages could be used, including MATLAB (The Math Works Inc.), Python, and SAS
(SAS Software). R is a freely available, open source statistical programming language, with a large library of usercontributed packages written for many types of analyses. If models are implemented with clear and welldocumented code, using open source programming languages can improve reproducibility and transparency, making it easier for modelers to share and fully understand one another’s work [1113]. Modelers can also borrow functionality from other packages, giving greater flexibility than what is available from specialized software. These advantages have led to an increase in the popularity of R for health economic evaluations [13]. One drawback of R is the learning curve for users unfamiliar with the language, especially for those with limited programming experience. Also, due to the nature of the usercontributed code libraries, there may be syntax differences between packages, and the amount of detail in the documentation and example code can vary, making some libraries easier to use than others.
Several R libraries contain useful implementations of Markov models, and more may be added in the future. The heemod (Health Economics Evaluation Modelling) package is particularly convenient, as it was specifically designed for costeffectiveness analysis. We used this library to run our example (code available online at https://github.com/BethHandorf/PharmacoeconomicsMar kovModels). Other libraries for discretetime Markov models are CTMCPack and markovchain. For a more thorough discussion on using R software for health economic decision modeling, see Jalal et al. [13].