Markov chains and semi-Markov models in time-to-event analysis

Abstract

A variety of statistical methods are available to investigators for analysis of time-to-event data, often referred to as survival analysis. Kaplan-Meier estimation and Cox proportional hazards regression are commonly employed tools but are not appropriate for all studies, particularly in the presence of competing risks and when multiple or recurrent outcomes are of interest. Markov chain models can accommodate censored data, competing risks (informative censoring), multiple outcomes, recurrent outcomes, frailty, and non-constant survival probabilities. Markov chain models, though often overlooked by investigators in time-to-event analysis, have long been used in clinical studies and have widespread application in other fields.

Background

The analysis of time-to-event data in human and animal studies presents several statistical challenges. In addition to the familiar problem of censored observations, there may be multiple types of failure under consideration (the “competing risk problem” [1]); clinically relevant outcomes other than failure may be observed during follow-up [2], including those that alter the risk of failure or can occur more than once [3,4]; and individual susceptibility to failure (i.e., frailty) may not be constant over time [5]. While traditional time-to-event analysis methods like Kaplan-Meier product-limit estimation and Cox proportional hazards regression are implemented easily and use censored data efficiently when the assumption of uninformative censoring holds, analyses involving informative censoring, multiple outcomes, or non-constant survival probabilities may be well suited for application of Markov processes [6]. A contemporary approach to the informative censoring problem in Cox regression involves a multivariate survival analysis [7].

Markov Processes

A Markov process is a stochastic process that describes the movement of an individual through a finite number of defined states, one (and only one) of which must contain the individual at any particular time. Possible movements among states may be depicted with a transition matrix or state diagram [2,3,6]. In order for the process to terminate, at least one of the states must be absorbing, i.e., individuals have zero probability of leaving the state once it has been entered. Death, for example, is an absorbing state used commonly in clinical studies, but it is also a well-known competing risk for clinical outcomes in studies of older persons [2,4]. Markov processes may be continuous or discrete as well as time-homogeneous or time-nonhomogeneous. The focus of this editorial will be discrete, time-homogeneous Markov processes called Markov chains.

Markov Chains

Markov chain models allow analysts to calculate the probability and rate (or intensity) of movement associated with each transition between states within a single observation cycle as well as the approximate number of cycles spent in a particular state. When observations are made at regular intervals, the number of cycles can be interpreted as time in a state. Time spent in all states prior to absorption can be summed to estimate the total survival time. Use of Markov chains requires two fundamental assumptions: (i) transition probabilities are constant over time (time homogeneity); and (ii) the probability of the next transition depends only on the current state (the first-order Markov property). These models are attractive for time-to-event analysis. They accommodate the simultaneous analysis of multiple events of interest and inclusion of competing risks through the states defined in the model, as well as consideration of individual frailty through subject-specific random effects [8,9].

Censored data, both right and left, are appropriate for Markov chains. In a Markov chain model, for example, an individual who never reaches an absorbing state (right-censored)—whether because the study observation is ongoing or the subject has withdrawn or been lost to follow up—can contribute information to the model regarding the transitions he or she did make, which is an advantage over traditional survival analysis methodology [6]. Because individuals are not required to enter the transition matrix in any particular state, left-censored data are also accommodated. Interval censoring is not formally accommodated in Markov chains, which assume that transitions take place only once per observation cycle, either at the beginning or the end. In reality, transitions make take place at any time, and multiple unobserved transitions may take place between cycle assessments. Approaches such as the half-cycle correction, where transitions are assumed to occur in the middle of the observation cycle [3], have been proposed to mitigate bias resulting from assuming that transitions take place only at the cycle’s beginning or end. If the clinical model and data structure support the assumption that all transitions are unidirectional (i.e., no reverse transitions are possible), a semi-Markov model, which is a special case of Markov chain where the time spent in the current state depends on both the prior and future adjoining states [10], could be considered for interval censored data [4,10].

Finally, unlike traditional time-to-event analysis where only one outcome is possible for each individual, Markov chains allow analysts to calculate survival times in multiple states. This is particularly attractive for studies of chronic diseases with well-defined phases, like cancer [11] and autoimmune diseases [12], where remission and recurrence are of interest in addition to overall survival, and dementia due to neurodegenerative disease, where pre-clinical and mildly symptomatic disease states are increasingly of interest to researchers working to identify treatments and prevention strategies [2,4]. As with traditional time-to-event analysis, survival curves may be estimated from model results [13]. Mean survival times may be inferred using matrix solution, Markov cohort simulation, or Markov Chain Monte Carlo simulation [3]. These calculations are more cumbersome, but still possible, when transition probability estimates are derived from covariate-adjusted regression models [14]. By contrast, semi-Markov models estimate mean survival times directly without the need for additional calculations [4].

Verifying Model Assumptions

The time homogeneity assumption can be assessed with a likelihood ratio test, and the first-order Markov property assumption can be examined with a chi-square test [6,15]. The time homogeneity assumption is often difficult to meet, particularly in studies of chronic disease where studies are years long, single observation cycles can span a year or more, and increasing age generally corresponds to greater risk of disease or death. However, this concern can be mitigated by data stratification (e.g., by age group or study period) or regression modeling, where the effect of covariates is included in the estimation of transition probabilities [16]. In regression, covariates may be either fixed or time-dependent.

Even when the fundamental model assumptions are met, application of the Markov chain model may still be unsuccessful. Data density, i.e., the observed frequency of each transition type, may be too sparse in some cells to implement the regression model. Sparse cells, where few events are observed, may lead to inaccurate estimation or prevent model convergence. In addition, there is no widely accepted goodness of fit test for the model.

Conclusion

Although Markov models have been used in clinical applications for over 60 years [17], incorporation of subject-specific random effects in Markov chains to account for individual propensity to make transitions is a relatively recent development [7]. However, inclusion of random effects makes estimation of the likelihood quite complex, and fitting such models can be time consuming. More importantly, their meaning must be carefully considered. Models that utilize tunnel states (i.e., non-absorbing states from which reverse transitions are not possible) [3], for example, complicate the use of random effects.

In closing, Markov chains are useful tools for survival analysis that allow for more nuanced modeling than is available in most standard time-to-event methods. While the focus of this editorial has been clinical studies, Markov chains have clear applications in diverse fields including labor research [18], finance [19], political science [20], chemical engineering [21], and demography [22]. However, while many journal readers and reviewers may readily comprehend the results from Markov models, they may lack familiarity with the underlying statistical assumptions, particularly in fields where the use of Markov models is not yet widespread. If so, they may neglect to challenge investigators to demonstrate that these assumptions are tenable. Given that improper use of Markov models may result in biased estimation, perhaps some standardization in the reporting of Markov model results and assumption verification is needed.