Hidden mover-stayer model for disease progression accounting for misclassified and partially observed diagnostic tests: Application to the natural history of human papillomavirus and cervical precancer

Abstract

Hidden Markov models (HMMs) have been proposed to model the natural history of diseases while accounting for misclassification in state identification. We introduce a discrete time HMM for human papillomavirus (HPV) and cervical precancer/cancer where the hidden and observed state spaces are defined by all possible combinations of HPV, cytology, and colposcopy results. Because the population of women undergoing cervical cancer screening is heterogeneous with respect to sexual behavior, and therefore risk of HPV acquisition and subsequent precancers, we use a mover-stayer mixture model that assumes a proportion of the population will stay in the healthy state and are not subject to disease progression. As each state is a combination of three distinct tests that characterize the cervix, partially observed data arise when at least one but not every test is observed. The standard forward-backward algorithm, used for evaluating the E-step within the E-M algorithm for maximum-likelihood estimation of HMMs, cannot incorporate time points with partially observed data. We propose a new forward-backward algorithm that considers all possible fully observed states that could have occurred across a participant’s follow-up visits. We apply our method to data from a large management trial for women with low-grade cervical abnormalities. Our simulation study found that our method has relatively little bias and out preforms simpler methods that resulted in larger bias.

1 |. INTRODUCTION

Markov models have previously been used to model the natural history of diseases.^1,2 With some diseases, such as cervical cancer, the disease state is assessed using multiple tests, each of which are subject to misclassification. When the results of one of these tests are missing, the disease state is only partially observed. Hidden Markov models (HMMs) have been proposed to account for misclassification in state identification³ with parameters estimated using a forward-backward algorithm. Adaptations to the standard forward-backward algorithmhavebeenproposedtohandlemissingobservations,⁴ but these approaches cannot be directly applied to partially observed data. This article proposes a maximum-likelihood approach for HMMs when the observed measurements are categorical and sometimes only partially observed. We extend our approach to also allow for the population to be a heterogeneous mixture of individuals susceptible and not susceptible to the disease (mover-stayer models⁵). These new methods are illustrated using data from a cervical cancer management trial.

Natural history models that describe the acquisition and clearance of human papillomavirus (HPV) infections in the cervix and the progression of persistently infected epithelium to cervical precancers/cancers are used to inform vaccination and screening strategies.^6–8 However, developing natural history models for HPV and cervical precancer/cancer poses multiple challenges. First, modeling strategies need to recognize that a portion of the population will not get infected with HPV and are therefore not susceptible to precancer/cancer progression. We propose a mover-stayer formulation⁵ for the underlying natural history to account for this heterogeneity. Second, population-level screening for cervical cancer entails multiple tests in sequential fashion—beginning with minimally invasive screening tests, such as testing for high-risk HPV types in the cervix and/or cytologic examination of cervical cells for abnormalities. Depending on the results of these tests, patients may then be referred to a colposcopic examination where areas suspicious of precancers are sampled using punch biopsies. As a result, any one of the three tests used to characterize the state of the cervix (HPV, cytology, or colposcopy) may be missing—either by design or by missing scheduled visits. Finally, while HPV tests have high accuracy, cytology tests are considerably less accurate⁹ and colposcopies, while more reliable than cytology, can still be insensitive for cervical precancers/cancers.^10,11

We propose a discrete-time HMM for HPV and cervical precancer/cancer in which the hidden and observed state spaces are defined by possible combinations of nonmissing HPV, cytology, and colposcopy test results. While other formulations of the hidden states are possible, we chose to mirror the observed states to allow for easy interpretability of the misclassification matrix. Most new HPV infections clear rapidly, but persistent HPV infections can cause precancerous lesions. To keep the Markov property of the model intact, we expand the state space beyond the positive-negative HPV test result to a trichotomous negative-positive-persistent scale that uses the HPV test result from the previous visit. We also treat the observed state as trichotomous as clinical management guidelines differ for newly positive vs persistent infections.¹² In our model, there are two scenarios that cause a partially observed state: (1) when any of the three current test results are missing or (2) when the current HPV result is positive but the previous HPV result is missing (ie, the observed HPV state could be either new or persistent). When the state is only partially observed, there exists a set of possible full observations that could have been observed.

Zucchini previously described how to handle fully missing observations,⁴ however, the handling of partially observed data is not straightforward. While fully missing observations are simply transitioned over, partially observed data require a reworking of the forward-backward algorithm. To properly account for partially observed data, each possible full observation borne out by a partial observation must be considered in the algorithm. Our proposed modification considers this set of all possible full observations by assuming that missingness of a test occurs at random (MAR) given the observed data. The MAR assumption is reasonable since a majority of the missing tests are by design, in that whether they are done depends on observed test results.

The model formulation is described in Section 2. In Section 3, we describe our novel version of the forward-backward algorithm for partially observed data. Section 4 illustrates the proposed method by fitting HMMs, with and without misclassification, to data from the ASCUS/LSIL Triage Study for Cervical Cancer (ALTS).¹³ In Section 5, we conducted simulation studies that compared three estimation approaches: (1) our proposed method with misclassification; (2) our proposed method without misclassification; and (3) standard methods using only fully observed measurements. Finally, we discuss our findings and implications for future work in Section 6.

2 |. MODEL FORMULATION

2.1 |. Partially observed data

We introduce the problem of partially missing measurements, followed by the full development of the model in Section 2.2. Let $X_{it}$ and $Y_{it}$ denote the latent (true) and observed (subject to misclassification) state, respectively, for the $t^{\text{th}}$ time point on the $i^{\text{th}}$ participant. Each state takes a value between 1 and $k$, and ${\mathbf{X}}_i=\left(X_{i1},X_{i2},\dots ,X_{in}\right)$ and ${\mathbf{Y}}_i=\left(Y_{i1},Y_{i2},\dots ,Y_{in}\right)$. We assume all participants have the same follow-up length $\left(n_i=n\right)$ and that all observations are equally spaced.

For the ALTS example each state is composed of three tests: HPV test (HPV), cytology, and colposcopy. HPV is 0 if the HPV test is negative, 1 if the test is newly positive, or 2 if the test has been positive for two or more tests in a row (persistence). Cytology is 0 if the cervical cytology result is normal, 1 if cytology is equivocal/low-grade, or 2 if cytology is HSIL/carcinoma. Likewise, colposcopy is 0 if the colposcopy is <CIN2 or 1 if CIN2+. Each state is represented as a tuple of the three tests, (HPV, cytology, colposcopy). ${\mathbf{Y}}_i$ is the observed result of the three tests with misclassification and ${\mathbf{X}}_i$ is the underlying disease status (eg, what the result would be without misclassification). In our example there are 18 possible states (3 × 3 × 2) and five equally spaced observations taken semiannually per individual.

${\mathbf{Y}}_i$ is considered to be partially observed when certain values of the state space are ruled out but at least two values are still possible. Suppose we know that $Y_{it}$ cannot be odd and $0<Y_{it}\le k$, we can then infer that the possible values for $Y_{it}$ are all even numbers less than or equal to k. Partially observed data occurs only in the observable data because the true state of the individual exists with or without observations. For estimation, we replace the partially observed data with the set of all possible full observations.

In our example, there are two scenarios that cause partially observed data. The first is when one or two of the three tests are missing. Since we have information about some but not all of the tests, there are multiple possibilities for what the fully observed data could be. The second case is when an individual tests HPV positive but we have no knowledge of their HPV test results 6 months prior and do not know if the HPV is either newly positive or persistent. Three possible participants’ test results can be seen in Figure 1.

FIGURE 1.

Participants are grouped by column, tests are grouped by row. Filled in circles represent fully observed data and open circles represent partially observed data. Participant one has partially observed data from a missed colposcopy at time 3. Participant two has partially observed data from an HPV positive test result at time one of the trial and a missed colposcopy at time two. Participant three has partially observed data due to a missed HPV test at time two and subsequent positive result at time three, and three missed colposcopies from times two to four. HPV, human papillomavirus

Define $Y_{it}^{\mathrm{*}}=\{Y_{it,1},Y_{it,2},\dots ,Y_{it,|Y_{it}^{\mathrm{*}}|}\}$ as the set of all fully observed possible states for $Y_{it}$ where $\left|Y_{it}^{\mathrm{*}}\right|$ is the size of set $Y_{it}^{\mathrm{*}}$. When the data is fully observed $Y_{it}^{\mathrm{*}}=Y_{it}$, however, when the data is partially observed, $Y_{it}^{\mathrm{*}}$ contains all possible fully observed states. For example, let $Y_{it}=(2,2,1)$, as all three tests were observed, $Y_{it}^{\mathrm{*}}=(2,2,1)$ as well. Now suppose that the colposcopy was not observed, so $Y_{it}=(2,2,-)$. Then, because a colposcopy can be either 0 or 1$,Y_{it}^{\mathrm{*}}=\{(2,2,0),(2,2,1)\}$.

Let ${\mathbf{W}}_i=\left(W_{i1},\dots ,W_{in}\right)$ be a vector of the fully observed state if $Y_{it}$ is not partially missing or what the fully observed state would be if $Y_{it}$ is partially missing (what the value would be if all tests were observed). It is important to note that for both possibilities, ${\mathbf{W}}_i$ is subject to misclassification as it corresponds to observable states. We can formally define ${\mathbf{W}}_i$ such that:

$$W_{it}=\left\{\begin{array}{ll}{Y}_{it}& \text{if}{Y}_{it}\text{is}\text{fully}\text{observed}\\ Y_{it,j}^{\mathrm{*}}& \text{if}{Y}_{it}\text{is}\text{partially}\text{observed},\end{array}\right.$$ (1)

where $Y_{it,j}^{\mathrm{*}}$ is the fully observed state if all tests had been done and refers to the $j^{\text{th}}$ element of the set $Y_{it}^{\mathrm{*}}$. Continuing the previous example, let $X_{it}=(2,2,1)$, $Y_{it}=(2,2,-)$, and $Y_{it}^{\mathrm{*}}=\{(2,2,0),(2,2,1)\}$. If a colposcopy were to be done (and therefore if $Y_{it}$ were to be fully observed) with no misclassification, then $W_{it}=(2,2,1)$; however, if there would have been a misclassification in the observation, then $W_{it}=(2,2,0)$.

We assume an individual cannot transition from a colposcopy CIN2+ to a colposcopy <CIN2 state. Of note, we censor individuals at the first observed CIN2+ as patients with observed CIN2+ were often treated. We also assume that the HPV tests have no misclassification^14,15 and colposcopies have a specificity of 1. The transitions allowed for in our model are visualized in Figure 2. We also assume that the three tests can be modeled as a first-order Markov chain and that the misclassification probabilities are independent over visits on the same individual. Denote the probability of the Markov chain starting in state $l_1$ by $r_{l_1}=P\left(X_{i1}=l_1\right)$, the transition from state $l_1$ to state $l_2$ as $p_{l_1{l}_2}=P\left(X_{it}=l_2|X_{i,t-1}=l_1\right)$, and the classification of state $l_1$ as state $l_2$ to be $M_{l_1{l}_2}=P\left(W_{it}=l_2|X_{it}=l_1\right)$.

FIGURE 2.

Possible state transitions by diagnostic test for hidden Markov model

The joint distribution of the latent and observed states can be calculated by multiplying the (1) probability of starting at the initial latent state, $r_{X_{i1}}$, the (2) transition probabilities for the latent states, $p_{X_{i,t-1}{X}_{it}}$, and the (3) probability of classifying the latent state as the observed state, $M_{X_{it}{W}_{it}}$. The complete-data joint distribution can be written as

$$P\left({\mathbf{X}}_i,{\mathbf{W}}_i\right)=r_{X_{i1}}\prod _{t=2}^n{p}_{X_{i,t-1}{X}_{it}}\prod _{t=1}^n{M}_{X_{it}{W}_{it}}.$$ (2)

In the next subsection, we incorporate population heterogeneity by using a mover-stayer model. Specifically, we incorporateatwogroupmixturewhereonegroupiscomposedofindividualswhohavenoHPVorcervicalabnormality,andthe other group is susceptible to cervical cancer progression. Similar models have been proposed for breast cancer screening without incorporating misclassification.¹⁶

2.2 |. Mover-Stayer

Let $\pi =P\left(Q_i=1\right)$ and $Q_i$ be an indicator of not being susceptible to HPV infection and cervical cancer progression. When $\pi =0$ this model reduces to a standard HMM, described in the previous subsection. If individual $i$ is a stayer, then they remain in the same latent state with probability one. In our example, we are only allowing stayers in the healthy (0,0,0) latent state. Therefore, if individual $i$ is a stayer, then ${\mathbf{X}}_i=\{((\mathrm{0,0},0),\dots (\mathrm{0,0},0)\}$. As being a stayer only effects the latent state, there are no further restrictions on classification. The joint distribution for individual $i$ with partially observed data and the mover-stayer component can be written as

$$f_i({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i)={\left(\pi \prod _{t=1}^n{M}_{0W_{it}}\right)}^{Q_i}\times {\left(\left(1-\pi \right)r_{X_{i1}}\prod _{t=1}^n{p}_{X_{i,t-1}{X}_{it}}\prod _{t=1}^n{M}_{X_{it}{W}_{it}}\right)}^{1-Q_i}.$$ (3)

If the individual is a stayer $\left(Q_i=1\right)$, then the probability is found by multiplying the classification probabilities for each observed data point given that the latent state is 0. If the individual is a mover $\left(Q_i=0\right)$, then the joint distribution is the same as for the HMM and reduces to Equation (2). Note that while we directly observe ${\mathbf{Y}}_i$, both ${\mathbf{X}}_i$ and ${\mathbf{Q}}_i$ are hidden and ${\mathbf{W}}_i$ is partially hidden.

The proposed approach incorporates potential misclassification in state identification. A special case of the approach is when there is no misclassification. This is a mover-stayer model with partially observed states, which in itself has not been considered before.

As a sensitivity analysis we examined the first-order Markov assumption as well. The mixture transition distribution model¹⁷ parametrizesasecond-orderMarkovchainwhileonlycreatingoneadditionalweightparameter.Oneassumption inherent to the model is that the transition matrix is the same for both the first- and second-order transition rates, with the weight parameter changing to account for the differences. To transition to a state at time $t$, the states at time $t-2$ and $t-1$ must now be taken into account. The effect of each state is considered separately, and the conditional probability is

$$P\left(X_{it}=l_3|X_{i,t-2}=l_1,X_{i,t-1}=l_2\right)={\lambda }_1{P}_{l_2{l}_3}+{\lambda }_2{P}_{l_1{l}_3},$$ (4)

where $l_1$, $l_2$, and $l_3$ all correspond to possible states. ${\lambda }_1$ is the weight parameter associated with the first lag, ${\lambda }_2$ is the weight parameter associated with second lag and ${\lambda }_1+{\lambda }_2=1$. $P_{l_1{l}_2}$, the transition rate from state $l_1$ to state $l_2$, can correspond to either a first- or second-order transition depending on which $\lambda$ it is multiplied by. Note that this model reduces to a first-order Markov chain when ${\lambda }_2=0$. Although only one more parameter is necessary, this considerably increases the computational complexity. Appendix A provides the details for this extension in full.

3 |. ESTIMATION

3.1 |. Complete data log likelihood

Here, we will introduce the procedure for estimation assuming no fully missing data but allow partially observed data. The method for dealing with fully missing data (ie, all tests are missing at a particular discrete visit) is similar to the method proposed by Zucchini et al⁴ and is given in Appendix B. Define the indicator variable $Z_{l_1}\left(X_{it}\right)$ as 1 if $l_1=X_{it}$ and 0 otherwise. We can then rewrite the complete-data joint distribution as

$$\begin{gathered} f_i\left({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i\right)={\left(\pi \prod _{t=1}^n\prod _{l_1=0}^k{M}_{0l_1}^{Z_0\left(X_{it}\right)Z_{l_1}\left(W_{it}\right)}\right)}^{Q_i} \\ \times \left(\left(1-\pi \right)\prod _{l_1=0}^k{r}_{l_1}^{Z_{l_1}\left(X_{il}\right)}\right. \\ \times \prod _{t=1}^n\prod _{l_1=0}^k\prod _{l_{2=0}}^k{p}_{l_1{l}_2}^{Z_{l_2}\left(X_{i-1}\right)Z_{l_2}\left(X_{it}\right)} \\ {\left.\times \prod _{t=1}^n\prod _{l_1=0}^k\prod _{l_2=0}^k{M}_{l_{l_1{l}_2}}^{Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)}\right)}^{1-Q_i}. \end{gathered}$$ (5)

Because ${\mathbf{X}}_i$, ${\mathbf{W}}_i$, and $Q_i$ are all latent, to find the maximum likelihood estimates we will maximize the marginal likelihood of the observed data, or ${\mathbf{Y}}_i^{\mathrm{*}}$’s, by evaluating the complete-data joint distribution at every possible value that the latent ${\mathbf{X}}_i$’s, ${\mathbf{W}}_i$ ‘s, and $Q_i$’s could take. Let $I$ denote the number of individuals. Therefore:

$$\begin{gathered} P({\mathbf{Y}}_1^{\mathrm{*}},{\mathbf{Y}}_2^{\mathrm{*}},\dots ,{\mathbf{Y}}_I^{\mathrm{*}})=\prod _{i=1}^{I}P({\mathbf{Y}}_i^{\mathrm{*}}) \\ \text{and} \\ P({\mathbf{Y}}_i^{\mathrm{*}})=\sum _{l_1=0}^k\dots \sum _{l_n=0}^k\sum _{w_1=0}^k\dots \sum _{w_n=0}^k\sum _{q=0}^1{f}_i\left(\left(l_1,\dots ,l_n\right),\left(w_1,\dots ,w_n\right),q\right), \end{gathered}$$ (6)

where $l_t$ is the latent state at time $t$ for individual $i$, $w_t$ is the observed, or would-be observed, state at time $t$ for individual $i$, and $q$ is an indicator for whether individual $i$ is a stayer. Although the summation of each (would be) observed state ($w_i$) is from 0 to $\mathrm{k}$, values that are not in the set of all possible fully observed states $Y_{it}^{\mathrm{*}}$ will have probabilities of 0 in the joint distribution. In the above joint distribution, $\left(l_1,\dots ,l_n\right)$ takes the place of the vector of latent states $\left({\mathbf{X}}_i\right)$ and $\left(w_1,\dots ,w_n\right)$ takes the place of the vector of possible fully observed states $\left({\mathbf{W}}_i\right)$. Although this likelihood can be directly maximized in theory, the number of terms makes the optimization intractable. Instead, we use the EM algorithm, iterating between an expectation and maximization step until the parameter estimates converge. The $\mathrm{E}$ step calculates the expected value of the complete data log likelihood as follows,

$$\begin{gathered} E\left[\sum _{i=1}^I\log f_i({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i)|{\mathbf{Y}}_i\right] \\ =E\left[Q_i|{\mathbf{Y}}_i\right]\left(\log \pi +\sum _{i=1}^I\sum _{t=1}^n\sum _{l_1=0}^{k}E\left[Z_0(X_{it})Z_{l_1}(W_{it})|{\mathbf{Y}}_i\right]\log M_{0W_{it}}\right) \\ +(1-E\left[Q_i|{\mathbf{Y}}_i\right])\{\log (1-\pi )+\sum _{i=1}^I\sum _{l_1=0}^{k}E\left[Z_{l_1}(X_{i1})|{\mathbf{Y}}_i\right]\log (r_{l_1}) \\ +\sum _{i=1}^I\sum _{t=2}^n\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}(X_{i,t-1})Z_{l_2}(X_{it})|{\mathbf{Y}}_i\right]\log (p_{l_1{l}_2}) \\ +\sum _{i=1}^I\sum _{t=1}^n\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}(X_{it})Z_{l_2}(W_{it})|{\mathbf{Y}}_i\right]\log (M_{l_1{l}_2})\}, \end{gathered}$$ (7)

where $E\left[Q_i|{\mathbf{Y}}_i\right]$$,E\left[Z_{l_1}\left(X_{i1}\right)|{\mathbf{Y}}_i\right]$, $E\left[Z_{l_1}\left(X_{i,t-1}\right)Z_{l_2}\left(X_{it}\right)|{\mathbf{Y}}_i\right]$, and $E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)|{\mathbf{Y}}_i\right]$ are calculated using an adaption of the forward-backward algorithm, explained in Section 3.4. The standard EM and forward-backward algorithm can be used when the complete set of tests are missing at a particular follow-up time. In this case, we simply apply the forward backward algorithm, removing the misclassification probabilities from the recursion steps associated with the missing measurements. However, this simple change in the forward backward algorithm does not work for partially missing data. In the next two subsections we propose an alternative recursion algorithm that specifically addresses the issue of partially missing data.

Considering the two alternative approaches mentioned in the introduction, the no misclassification method is a special case where the misclassification is ignored by setting $M_{l_1{l}_2}$ to 0 if $l_1\ne l_2$ and 1 otherwise. For the no partially missing method, all partially observed data is simply removed from the data and estimation proceeds as described.

3.2 |. Forward algorithm

To include the partially missing values in the forward algorithm we will consider every possible fully observed state. It is necessary to keep track of exactly which partially observed state is being used at each point for the E-step calculations. Define the forward quantity as:

$${\alpha }_{l_1}(it{)}_j=P(Y_{i1}^{\mathrm{*}},Y_{i2}^{\mathrm{*}},\dots ,Y_{it,j}^{\mathrm{*}},X_{it}=l_1|Q_i=0)$$ (8)

and is calculated by

$${\alpha }_{l_1}{(it)}_j=\{\begin{array}{ll}{r}_{l_1}{M}_{l_1{Y}_{i_j}^{*}}\hfill & \text{if}t=1\hfill \\ \sum _{l_0=1}^k\sum _{m=1}^{\left|Y_{i,t-1}^{*}\right|}{\alpha }_{l_0}{(i,t-1)}_m{p}_{l_0{l}_1}{M}_{l_1{Y}_{it,j}^{*}}\hfill & \text{if}t\ne 1.\hfill \end{array}$$ (9)

Note that the $j$ in ${\alpha }_{l_1}(it{)}_j$ indicates that the fully observed data point $Y_{it,j}^{\mathrm{*}}$ is used rather than (possibly partially observed) $Y_{it}^{\mathrm{*}}$. By calculating the forward quantity individually for each partially observed state, we will then be able to evaluate expectations needed to perform the E-step. This involves computing ${\alpha }_{l_1^{\mathrm{*}}}^{\mathrm{*}}(it)$ where

$${\alpha }_{l_1}^{\mathrm{*}}(it)=P(Y_{i1}^{\mathrm{*}},\dots ,Y_{it}^{\mathrm{*}},X_{it}=l_1|Q_i=0)=\sum _{j=1}^{\left|Y_{it}^{\mathrm{*}}\right|}{\alpha }_{l_1}(it{)}_j.$$ (10)

The additional computation time of our approach scales linearly with the size of the set of possible full observations that could have been observed.

3.3 |. Backward algorithm

The backward algorithm adaption is defined slightly differently than its forward counterpart. Because we do not need to keep track of each specific partially observed state for any future calculations, we can sum over all partially observed states. We define the backward quantity as:

$${\beta }_{l_1}(it)=P(Y_{i,t+1}^{\mathrm{*}},Y_{i,t+2}^{\mathrm{*}},\dots ,Y_{in}^{\mathrm{*}}|X_{it}=l_1,Q_i=0),$$ (11)

which is calculated by

$${\beta }_{l_1}(it)=\{\begin{array}{cc}1\hfill & \text{if}t=n\hfill \\ \sum _{l_2=0}^k{p}_{l_1{l}_2}\sum _{j=1}^{\left|Y_{i,t+1}^{\ast }\right|}{M}_{l_2{Y}_{i,t+1,j}^{\ast }}{\beta }_{l_2}(i,t+1)& \text{if}t<n.\hfill \end{array}$$ (12)

Contrasting with the forward step, the backward step does not individually calculate backward quantities for each possible fully observed state. Specifically, the backward probabilities ${\beta }_{l_1}(it)$ are evaluated by including an additional summation over partially observed states in the recursion equation. Our proposed adaptation takes additional time, when compared with the standard forward-backward, directly proportional to the size of the set of possible full observations that could have been observed, for partially observed data.

3.4 |. Calculating expectations

The following are the E-step expectations, calculated using ${\alpha }_{l_1}(it{)}_j$ and ${\beta }_{l_1}$ (it).

$$E\left[Q_i|{\mathbf{Y}}_i\right]=\frac{P\left(Q_i=1,{\mathbf{Y}}_i\right)}{P\left({\mathbf{Y}}_i\right)}=\frac{\pi \prod _{t=1}^n\sum _{j=1}^{\left|Y_{it}^{\mathrm{*}}\right|}P(Y_{it,j}^{\mathrm{*}}|X_{it}=0)}{P\left({\mathbf{Y}}_i\right)},$$ (13)

$$E\left[Z_{l_1}\left(X_{i1}\right)|{\mathbf{Y}}_i\right]=\frac{P\left(X_{i1}=l_1,{\mathbf{Y}}_i\right)}{P\left({\mathbf{Y}}_i\right)}=\frac{{\alpha }_{l_1}^{\mathrm{*}}(i1){\beta }_{l_1}(i1)}{P\left({\mathbf{Y}}_i\right)},$$ (14)

$$E\left[Z_{l_1}\left(X_{i,t-1}\right)Z_{l_2}\left(X_{it}\right)|{\mathbf{Y}}_i\right]=\frac{P\left(X_{i,t-1}=l_1,X_{it}=l_2,{\mathbf{Y}}_i\right)}{P\left({\mathbf{Y}}_i\right)}=\frac{{\alpha }_{l_1}^{\mathrm{*}}(i,t-1)p_{l_1{l}_2}\sum _{j=1}^{\left|Y_{it}^{\mathrm{*}}\right|}{M}_{l_2{Y}_{it,j}^{\mathrm{*}}}{\beta }_{l_2}(it)}{P\left({\mathbf{Y}}_i\right)},$$ (15)

$$\begin{gathered} E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)|{\mathbf{Y}}_i\right]=P\left(X_{it}=l_1|{\mathbf{Y}}_i\right)P\left(W_{it}=l_2|X_{it}=l_1,{\mathbf{Y}}_i\right) \\ =\frac{{\alpha }^{\mathrm{*}}{l}_1(it){\beta }_{l_1}(it)}{P\left({\mathbf{Y}}_i\right)}\times \frac{P(Y_{it,j}^{\mathrm{*}}=l_2|X_{it}=l_1)}{{\sum }_{m=1}^{|Y_{it}^{\mathrm{*}}|}P(Y_{it,m}^{\mathrm{*}}|X_{it}=l_1)}, \end{gathered}$$ (16)

where

$$P({\mathbf{Y}}_i)=P({\mathbf{Y}}_i|Q_i=1)+P({\mathbf{Y}}_i|Q_i=0)=\pi \prod _{t=1}^n\sum _{j=1}^{\left|Y_{it}^{\ast }\right|}P(Y_{it,j}^{\ast }|X_{it}=0)+(1-\pi )\sum _k^{l_1=0}{\alpha }_{l_1}^{\ast }(in).$$ (17)

4 |. APPLICATION

We illustrate our proposed method on data from the ALTS.13 The ALTS was conducted from 1996 to 2000 to evaluate management strategies for mildly abnormal and very common cytology results. Nonpregnant women, 18 years or older, with equivocal (n = 3844) or low-grade squamous intraepithelial lesions (LSIL) (n = 1572) cytology results within 6 months prior to enrollment were randomly assigned to one of three management arms: immediate colposcopy (IC), HPV triage, or conservative management (CM). We apply our methods to the larger population of patients referred for equivocal cytology results by fitting models that (1) account for misclassification and (2) assume no misclassification.

Women had colposcopy at enrollment if they were (1) randomized to the IC arm, (2) randomized to the HPV triage arm and were enrollment HPV positive or had an enrollment cytology result of HSIL/carcinoma, or (3) randomized to the CM arm and had an enrollment cytology result of HSIL/carcinoma. Regardless of randomization arm, patients in the study were screened biannually over 2 years with HPV and cytology. In addition, patients had colposcopies at postenrollment visits if cytology results were HSIL/carcinoma, and at the 2-year exit screen. Our HMM treats each of the biannual screens and any attendant colposcopies as discrete times spaced 6 months apart. As persistent HPV infections are a necessary precursor to cervical cancer, we classified HPV results as negative, new, or persistent for at least 6 months. The HPV result was missing for 5% of visits. For the 43% of HPV positives found at the enrollment visit or at a visit following a missing HPV test result, it is not known if the HPV infection is new or persistent. Cytology results were classified as normal, abnormal/low-grade, or HSIL/carcinoma and were rarely missing (0.3% of all visit). Colposcopy results were classified as cervical intraepithelial neoplasia grade 2+ (CIN2+, aka precancers) or <CIN2. For women who tested negative for both the HPV and cytology tests, we assumed colposcopies of <CIN2 as this is the common clinical assumption. Even after applying this assumption, colposcopy results were missing for 42% of visits, primarily by design. In addition, some results were missing due to skipped visits or lost to follow-up. In sum, the three tests were fully observed at 38% of visits, partially observed at 40% of visits, and completely unobserved at 22% of visits.

Our HMM allowed for certain state transitions (Figure 2) for HPV, cytology, and colposcopy as per the known natural history of HPV and cervical cancer.⁶ Most HPV infections clear quickly, but persistent HPV infections can cause cervical lesions that are detectable via cytology tests and colposcopies. While CIN2+ lesions can regress to <CIN2, we could not model this process using the data from ALTS as women with observed CIN2+ have their natural history interrupted by treatment. We allowed for a heterogeneous population of “movers,” women who may transitioned from state to state, and “stayers,” women whose latent states for HPV/cytology/colposcopy remained negative/normal/<CIN2 throughout the 2 years of the study. This mirrors the realities of population-wide cervical screening, which will always include some women who are not a risk for new HPV infections and subsequent precancer/cancers (eg. women in monogamous relationships).

In the model that accounts for misclassification, we fixed certain parameters in the misclassification matrix of the HMM to mirror the known behavior of each test. As HPV tests have high accuracy,^14,15 we assumed no misclassification of the HPV state. Because cytology results have poorer accuracy,⁹ we allowed misclassification of each cytology state. As colposcopy diagnoses in ALTS underwent extensive quality control, we assumed that CIN2+ was not misclassified. However, we allow for the possibility that a <CIN2 colposcopy result could be a misclassified CIN2+, which can occur if a CIN2+ lesion was not biopsied.

We fit a HMM to the ALTS dataset using our proposed method for partially observed data with and without misclassification (models A and B, respectively). To ensure the HMM arrived at a global maximum, we used multiple different starting parameters and confirmed that the algorithm estimated similar parameter values. In general, estimates allowing for misclassification had larger standard errors. We present some of the more precise and scientifically interesting estimates in Table 1. When testing the sensitivity of the first-order Markov assumption, λ₁ approached 1, indicating that the first-order Markov assumption holds. The second-order estimated parameters are also similar to the those of the first-order model.

TABLE 1.

Stayer, initial HPV, selected transition, and selected misclassification probabilities for ALTS using the proposed method

				Model A		Model B
				With misclassification		Without misclassification
				Est	Bootstrap SE	Est	Bootstrap SE
Stayers				0.059	0.032	0.017	0.011
Movers
	Initial state	New HPV		0.35	0.103	0.29	0.056
		Persistent HPV		0.22	0.083	0.25	0.056
	Transition probabilities	Previous state	Current state
		(0,0,0)	(0,0,0)	0.69	0.09	0.71	0.01
		(0,0,0)	(0,1,0)	0	0.04	0.14	0.01
		(0,0,0)	(1,0,0)	0.31	0.07	0.11	0.01
		(0,0,0)	(1,1,0)	0	0.02	0.04	0
		(1,0,0)	(0,0,0)	0.36	0.13	0.49	0.04
		(1,0,0)	(2,-,-)	0.37	0.12	0.45	0.04
		(2,0,0)	(0,0,0)	0.46	0.14	0.24	0.03
		(2,0,0)	(2,-,-)	0.54	0.17	0.73	0.05
		(2,1,0)	(-,-,1)	0.17	0.13	0.04	0.02
	Misclassification probabilities	True state	Observed state
		(2,1,1)	(2,0,0)	0	0.06
		(2,1,1)	(2,1,0)	0.5	0.21
		(2,1,1)	(2,2,0)	0.02	0.02
		(2,1,1)	(2,0,1)	0	0.04
		(2,1,1)	(2,1,1)	0.43	0.22
		(2,1,1)	(2,2,1)	0.05	0.02

Note: State tuples correspond to (HPV, cytology, colposcopy). HPV values are negative, new, or persistent (0–2), cytology values are normal, equivocal/low-grade, and HSIL/carcinoma (0–2), and colposcopy values are <CIN2 and CIN2+ (0–1). State (2,1,0) corresponds to HPV persistent, cytology equivocal/low-grade, and colposcopy <CIN2. (2,-,-) and (-,-,1) are marginal transition probabilities for being in the HPV-persistent state and in the CIN2+ state, respectively.

Abbreviation: HPV, human papillomavirus.

Bootstrap standard errors were calculated by sampling n = 3488 individuals, with replacement. Models A and B were then applied to the sampled data. This process was repeated 2500 times. Bootstrap standard errors were then calculated for each parameter by using the 2500 bootstrap estimates.

One question of interest for vaccination and screening strategies are the transition rates among healthy women. Model A estimated 5.9% (SE: 3.2%) of the population to be stayers, whereby these participants were not susceptible to HPV over the 2 year time period. Interestingly, assuming no misclassification (model B) resulted in an estimate of only 1.7% (SE: 1.1%) stayers, primarily because all women with equivocal/low-grade cytology results would be treated as having cervical changes. Overall, the percentage of stayers in ALTS are less than what one would expect to find in the general population due to their relatively young ages (75%+ of women were under age 35 at enrollment) and their equivocal cytology test entry criteria.

Among movers susceptible to HPV and subsequent cervical precancers, healthy women in ALTS were estimated to have 6-month transition probabilities of 69% (SE: 9%) and 31% (SE: 7%) for remaining in a healthy state vs acquiring a new HPV infection without developing cervical abnormalities, respectively. When no misclassification was assumed, these estimates were 71% (SE: 1%) and 11% (SE: 1%) with an 18% (SE: 1%) probability of transitioning to an equivocal/low-grade cervix state.

Because one cannot know whether HPV infections present at the first visit are new or persistent, standard methods typically cannot use these infections to estimate differential transition rates for new vs persistent infections. Our proposed method treats HPV-positives present at the first visit as partially observed (either new or persistent HPV) and subsequently estimate transition rates. Using models A and B, respectively, we estimated that 61% and 54% of women HPV-positive at the first visit were newly infected. Estimated transition probabilities for clearance and progression then differed depending on whether the infection was new or persistent. For example, under model A, HPV-positive women with normal cervices had probabilities of having their HPV infection persist 6-month later of 36% (SE: 13%) vs 54% (SE 17%) for new vs persistent infections, respectively.

In general, transition rate estimates for progressing to CIN2+ were higher and had greater standard errors when assuming misclassification of colposcopy results (model A) than when assuming no misclassification of colposcopy results (model B). For example, women in the state of persistently HPV-positive, equivocal/low-grade cytology, and <CIN2 colposcopy had a 17% (SE: 13%) vs 4% (SE: 2%) probability of transitioning to CIN2+ with models A and B, respectively.

Estimates for misclassification of cytology and colposcopy in model A had high standard errors but often also high estimates, indicating that misclassification should be accounted for. For example, women in the state of persistently HPV-positive, with equivocal/low-grade cytology, and CIN2 colposcopy had a 50% (SE: 21%) probability of being misclassified as persistently HPV-positive, with equivocal/low-grade cytology, and <CIN2 colposcopy. Our example illustrates that in applications, two competing priorities will need to be balanced—the incorporation of misclassification and the precise estimation of transition and classification probabilities for rare states.

5 |. SIMULATION

We compare the performance of the proposed method with misclassification (method A) and without misclassification (method B) with an alternative approach that applies standard methods for a HMM with a mover-stayer component after excluding partial observations (method C). Simulated latent data are generated from the maximum likelihood estimated proposed method with classification in Section 3 with the same amount of participants and follow-up length as in the ALTS data. Observed values after a positive colposcopy result are removed to mimic patient treatment and their subsequent removal from the trial. Partial missingness is introduced independently to HPV and cytology according to their global partial missing percentage in the data, 5.5% and 0.3%, respectively. Colposcopy partial missingness is conditional on HPV and cytology where the probabilities are calculated from the ALTS data. Full observations are then removed at the same rate as in the ALTS data. Finally, any new HPV positive result not immediately following an HPV negative result and any HPV persistent result not immediately following a new HPV positive or persistent result is considered to be partially observed as it is unsure if the HPV is newly acquired or persistent. Simulation results are based on 2500 simulated realizations.

Tables 2 to 4, show the simulation results for estimating the initial and mover-stayer proportion, transition, and misclassification probabilities, respectively. Discussing each method individually, the proposed approach with misclassification (method A) has good finite sample properties for estimating the parameters of all four components and results in unbiased estimation. The 95% confidence interval for the bias contains zero for every parameter and the empirical standard deviation (SD) remains insignificant for most states. The stayer proportion (Table 2) is accurately estimated with both extremely low bias and SD. Even for the CIN2+ classification probabilities (bottom half of Table 4), the proposed method has minimal bias (barring one state) and reasonable SDs.

TABLE 2.

Initial state probabilities and SE for simulations

				Model A			Model B			Model C
				With misclassification			Without misclassification			Without partial
	State	Truth	N	Bias	SE	MSE	Bias	SE	MSE	Bias	SE	MSE
Stayer		0.059	206	−0.002	0.016	0.0002	−0.041	0.008	0.0017	0.017	0.085	0.0076
Mover		0.941	3282	0.002	0.016	0.0002	0.041	0.008	0.0017	−0.017	0.085	0.0076
	(0,0,0)	0.11	361	−0.006	0.019	0.0004	0.159*	0.01	0.0252	0.028	0.104	0.0117
	(0,0,0)	0.235	770	−0.005	0.026	0.0007	−0.051*	0.007	0.0027	0.089	0.126	0.0238
	(0,1,0)	0.046	150	0.002	0.018	0.0003	−0.044*	0.001	0.0019	0.123	0.101	0.0253
	(0,1,0)	0.031	103	0.004	0.024	0.0006	0.022	0.022	0.001	−0.031*	0.003	0.001
	(0,2,0)	0.05	164	0.011	0.031	0.001	0.07	0.071	0.0099	−0.014	0.044	0.0021
	(0,2,0)	0.045	147	0.002	0.021	0.0004	−0.02*	0.005	0.0004	0.008	0.045	0.0021
	(1,0,0)	0.018	59	0.005	0.02	0.0004	0.056*	0.023	0.0036	−0.017*	0.007	0.0003
	(1,0,0)	0.112	368	0.022	0.037	0.0019	0.054	0.071	0.008	−0.078	0.042	0.0079
	(1,1,0)	0	0	0	0	0	0.003	0.005	0	0	0	0
	(1,1,0)	0.041	135	0.011	0.023	0.0006	−0.04*	0.001	0.0016	0.042	0.078	0.0077
	(1,2,0)	0	0	0	0	0	0.003	0.002	0	0	0	0
	(1,2,0)	0.003	11	0	0.004	0	−0.003*	0	0	0.007	0.025	0.0007
	(2,0,0)	0.075	245	−0.017	0.04	0.0018	−0.066*	0.005	0.0043	−0.06*	0.025	0.0041
	(2,0,0)	0.006	21	0.001	0.003	0	0.009	0.017	0.0004	0.016	0.026	0.0009
	(2,1,0)	0.138	452	−0.042	0.029	0.0026	−0.097*	0.006	0.0095	−0.037	0.053	0.0041
	(2,1,0)	0.04	131	0.005	0.03	0.0009	−0.036*	0.005	0.0013	−0.04*	0.001	0.0016
	(2,2,0)	0.05	165	0.008	0.036	0.0014	−0.021	0.017	0.0008	−0.038	0.025	0.0021
	(2,2,0)	0	0	0	0	0	0.001	0.005	0	0	0	0

Note: State tuples correspond to (HPV, cytology, colposcopy). State (2,1,0) corresponds to HPV 2, cytology 1, and colposcopy 0. Method A is the proposed method with misclassification, B is the proposed method without misclassification, and C is an HMM where the partially observed data are discarded.

^*indicates that the 95% CI for the estimated bias does not include zero. N is the average number of individuals with a particular state at time 1.

Abbreviations: HMM, hidden Markov model; HPV, human papillomavirus.

TABLE 4.

Classification probabilities and SE for simulations

				Method A			Method C
				With misclassification			Without partial
True state	Class. state	Truth	N	Bias	SE	MSE	Bias	SE	MSE
(0,0,0)	(0,0,0)	0.925	2988	−0.002	0.019	3e-04	0.011	0.121	0.0147
(0,1,0)	(0,1,0)	0.506	1324	−0.005	0.05	0.0024	−0.018	0.275	0.0759
(0,2,0)	(0,2,0)	0.025	13	−0.002	0.012	1e-04	−0.16	0.161	0.0511
(1,0,0)	(1,0,0)	0.901	572	−0.006	0.043	0.0018	0.099	0.202	0.0505
(1,1,0)	(1,1,0)	0.737	243	−0.008	0.075	0.0052	0.003	0.375	0.1397
(1,2,0)	(1,2,0)	0.484	71	−0.049	0.224	0.0507	0.136	0.458	0.2296
(2,0,0)	(2,0,0)	0.873	444	−0.014	0.065	0.0043	0.13	0.321	0.1197
(2,1,0)	(2,1,0)	0.952	799	−0.003	0.05	0.0025	0.034	0.179	0.0333
(2,2,0)	(2,2,0)	0.392	42	−0.044	0.156	0.0236	−0.407	0.298	0.2548
(0,0,1)	(0,0,1)	0.039	10	0.007	0.04	0.0016	−0.003	0.088	0.0076
(0,1,1)	(0,1,1)	0.016	41	0.002	0.008	1e-04	−0.007	0.023	6e-04
(0,2,1)	(0,2,1)	0	0	0	0	0	0	0	0
(1,0,1)	(1,0,1)	0.038	19	−0.012	0.058	0.0034	−0.018	0.131	0.0174
(1,1,1)	(1,1,1)	0	0	0	0	0	0	0	0
(1,2,1)	(1,2,1)	0.301	142	−0.154	0.144	0.0438	−0.429	0.283	0.2631
(2,0,1)	(2,0,1)	0.144	122	0.005	0.047	0.0022	0.031	0.068	0.0055
(2,1,1)	(2,1,1)	0.43	280	−0.016	0.155	0.024	0.126	0.251	0.0786
(2,2,1)	(2,2,1)	0.365	34	0.041	0.121	0.0144	−0.029	0.212	0.0457

Note: State tuples correspond to (HPV, cytology, colposcopy). State (2,1,0) corresponds to HPV 2, cytology 1, and colposcopy 0. Method A is the proposed method with misclassification and C is an HMM where the partially observed data are discarded.

^*indicates that the 95% CI for the estimated bias does not include zero. N is the average number of individuals with each state from time 1 to n.

Abbreviations: HMM, hidden Markov model; HPV, human papillomavirus.

Ignoring misclassification (method B) emulates a Markov chain and results in substantial bias and smaller standard errors for the initial and transition probabilities. This is especially noticeable for the transition rates (Table 3) where the SD is minimal and the bias is often comparable to the truth. Despite the small SD, this method still has a quite large MSE due to overwhelmingly large bias; for most estimates method B has a higher MSE than method A. In addition, the difference in relative size of the bias and SD means that most 95% CIs for the bias do not include zero (indicated by a *).

TABLE 3.

Six month transition rates and SE for simulations

				Model A			Model B			Model C
				With misclassification			Without misclassification			Without partial
Prev. state	Curr. state	Truth	N	Bias	SE	MSE	Bias	SE	MSE	Bias	SE	MSE
(0,0,0)	(0,0,0)	0.689	1199	0.021	0.052	0.0031	−0.02	0.011	0.0006	0.1	0.266	0.0804
(0,0,0)	(1,0,0)	0.306	532	−0.023	0.052	0.0031	0.194*	0.006	0.0377	−0.071	0.247	0.0658
(0,1,0)	(0,1,0)	0.675	1574	−0.014	0.041	0.0018	0.283*	0.02	0.0807	0.092	0.233	0.0628
(0,1,0)	(0,1,1)	0.193	450	0.017	0.053	0.003	0.192*	0.002	0.037	0.034	0.18	0.0335
(0,2,0)	(0,2,0)	0.795	367	−0.009	0.065	0.004	0.786*	0.022	0.6187	0.062	0.132	0.0211
(0,2,0)	(0,1,1)	0.163	75	0.008	0.063	0.0037	0.143*	0.056	0.0238	−0.061	0.137	0.0226
(1,0,0)	(2,0,0)	0.347	170	0.002	0.049	0.002	0.074*	0.027	0.006	−0.063	0.24	0.061
(1,0,0)	(0,1,1)	0.255	125	0	0.128	0.016	0.25*	0.007	0.0631	0.031	0.274	0.0765
(1,1,0)	(0,1,0)	0.472	143	0.015	0.085	0.0068	0.319*	0.032	0.1034	0.307	0.293	0.179
(1,1,0)	(2,2,0)	0.112	34	0.006	0.052	0.0024	0.087*	0.012	0.008	−0.215	0.32	0.1477
(1,2,0)	(0,1,0)	0.163	24	−0.006	0.114	0.0121	0.032	0.052	0.0039	0.158*	0.063	0.0297
(1,2,0)	(2,0,1)	0.188	28	0.016	0.083	0.0061	0.157*	0.033	0.0267	−0.016	0.306	0.0933
(2,0,0)	(0,0,0)	0.463	181	0.002	0.088	0.0071	0.179*	0.024	0.0329	−0.1	0.358	0.1385
(2,0,0)	(2,0,0)	0.505	198	0.005	0.095	0.0084	0.088*	0.033	0.0089	0.153	0.338	0.1383
(2,1,0)	(2,1,0)	0.363	284	−0.032	0.073	0.0061	−0.029	0.027	0.0015	0.116	0.229	0.0656
(2,1,0)	(0,1,1)	0.102	80	0.018	0.064	0.0043	0.1*	0.004	0.0102	0.067	0.115	0.0177
(2,2,0)	(2,1,0)	0.679	64	0	0.187	0.0329	0.101	0.097	0.0201	0.172	0.362	0.1582
(2,2,0)	(2,1,1)	0.126	12	−0.001	0.177	0.0308	0.065	0.065	0.0089	−0.086	0.308	0.1015

Note: State tuples correspond to (HPV, cytology, colposcopy). State (2,1,0) corresponds to HPV 2, cytology 1, and colposcopy 0. Method A is the proposed method with misclassification, B is the proposed method without misclassification, and C is an HMM where the partially observed data are discarded.

^*indicates that the 95% CI for the estimated bias does not include zero. N is the average number of individuals with a particular state for times 2 to n.

Abbreviations: HMM, hidden Markov model; HPV, human papillomavirus.

Discarding partially observed measurements (method C) results in nearly unbiased estimation, but also causes a substantial loss of precision. In Section 3, it was shown that one major drawback of method A is the high standard errors, and method C exacerbates this issue. As about 50% of all observable data is partially observed, roughly half of the available data are ignored in this method. For the classification rates (Table 4), this method has substantially larger SD than method A for every state. This is shown quantitatively as the median relative efficiency of the no partial method (method C) to the proposed method with misclassification (method A) is 0.12. Method C also relies on the assumption that the data is missing completely at random, and as we explicitly removed colposcopy results based on observed HPV and cytology, this assumption is violated.

In summary, although the proposed approach with misclassification (method A) results in a loss of efficiency relative to ignoring the misclassification error (method B), in terms of MSE method A outperforms methods B and C.

6 |. DISCUSSION

When the state-space is defined by multiple tests, partially observed data occurs when some tests are missing while others are observed. This is especially common when studying the natural history of a disease where the progression is checked by multiple diagnostic tests. HPV tests, cytology and colposcopies are used to screen women for cervical precancers in ALTS. If at least one but not all three tests are missing, the data point is considered partially observed and standard methods for HMMs cannot be applied. Given the large proportion of partially observed data in HPV cohort analyses, incorporating this type of incomplete data is very important.

This article presents a unique adaptation of the forward-backward algorithm that handles partial observations by considering the set of all possible full observations. The first proposed method incorporates misclassification of test results and population mixtures, where a subset of individuals move through different states and other individuals stay only in the healthy state. Interestingly, the problem of fitting Markov chain models (without misclassification) with a partially observed state space is in itself a computationally difficult problem. The proposed algorithm also solves this simpler problem. We apply these approaches to a dataset of women undergoing cervical cancer screening.

To properly account for partially observed data, we use the set of all possible full observations that could have been observed if no test was missing. To do this, we calculate forward probabilities for each possible full observation by considering each observed state or potentially observed state for totally and partially observed states, respectively. Although this leads to additional computations, the adaptation scales linearly with the size of the set of possible full observations. For example, a partially observed time point with five possible full observations requires four more calculations than one fully observed time point. Because our method only deals with partially observed data differently, our adaptation reduces to the standard forward-backward algorithm for fully observed data. Both partially and fully missing data is assumed to be MAR. We believe this to be the case as the data is missing based on prior observed test results, such as not needing a colposcopy based on HPV and cytology results. Through our simulation study, we found that our proposed method with misclassification more accurately estimates the parameters used to create the simulation data than the other approaches.

We applied these methods to 3488 women undergoing cervical cancer screening over 2 years in ALTS. Only 38% of ALTS visits were fully observed; 40% were partially observed and 22% were completely unobserved. Although the existing literature has shown that cytology and colposcopy can disagree, current models of natural history have primarily ignored measurement error and treat test results as the gold standard when estimating transition probabilities. Our application shows how transition probabilities can be estimated while incorporating misclassification, and indicates that the amount of misclassification cannot be ignored, as not accounting for misclassification will produce bias estimates. However, there are tradeoffs between unbiased estimation and large standard errors in models that incorporate misclassification. Approaches, such as ours, applied to larger datasets that also include HPV genotyping can further understanding of the natural history of this disease and inform strategies for primary and secondary prevention.

The mover-stayer model can incorporate heterogeneity in natural history beyond what would be expected from a standard Markov model. However, care should be given to over interpreting the probability of a mover on an individual basis. In particular, in studies with follow-up such as ALTS, we cannot attribute a stayer as someone who is unlikely to progress in any longer term sense.

A number of methodological extensions to this approach are possible. In settings that are more heterogeneous, covariates could be added to the parameters of the mover-stayer model. For highly irregular follow-up time, the discrete Markov chain can either be replaced by a Markov process,^18,19 or be sampled at finer intervals. Another possible direction is accounting for nonignorable missing data. Although unnecessary for our application, there are scenarios in which tests can be missing based on prior unobserved results, requiring a further adaptation of our method.

APPENDIX A. MIXTURE TRANSITION DISTRIBUTION

A.1. Complete data log likelihood

The complete data joint distribution for the mixture transition distribution model is

$$\begin{gathered} f_i\left({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i\right)={\left(\pi \prod _{t=1}^n\prod _{l_1=0}^k{M}_{0l_1}^{Z_0\left(X_{it}\right)Z_{l_1}\left(W_{it}\right)}\right)}^{Q_i} \\ \times \left(\left(1-\pi \right)\prod _{l_1=0}^k\prod _{l_2=0}^k{r}_{l_1{l}_2}^{Z_{l_1}\left(X_{i1}\right)Z_{l_2}\left(X_{i2}\right)}\right. \\ \times \prod _{t=1}^n\prod _{l_1=0}^k\prod _{l_2=0}^k\prod _{l_3=0}^k{\left({\lambda }_1{P}_{l_2{l}_3}+{\lambda }_2{P}_{l_1{l}_3}\right)}^{Z_{l_1}\left(X_{i,t-2}\right)Z_{l_2}\left(X_{i,t-1}\right)Z_{l_3}\left(X_{it}\right)} \\ {\left.\times \prod _{t=1}^n\prod _{l_1=0}^k\prod _{l_2=0}^k{M}_{l_1{l}_2}^{Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)}\right)}^{1-Q_i} \end{gathered}$$ (A1)

To accommodate the second-order Markov chain, the initial probabilities are a matrix describing the first two states rather than a vector. $r_{l_1{l}_2}$ refers to the probability of being in state $l_1$ at time $t=1$ and state $l_2$ at time $t=2$. As in Section 3.1, because ${\mathbf{X}}_i$, ${\mathbf{W}}_i$, and $Q_i$ are all latent we will use the EM algorithm to estimate the parameters. The complete data log likelihood is

$$\begin{gathered} E\left[\sum _{i=1}^I\log f_i\left({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i\right)|{\mathbf{Y}}_i\right] \\ =E\left[Q_i|{\mathbf{Y}}_i\right](\log \pi +\sum _{i=1}^I\sum _{t=1}^n\sum _{l_1=0}^{k}E\left[Z_0\left(X_{it}\right)Z_{l_1}\left(W_{it}\right)|{\mathbf{Y}}_i\right]\log M_{0W_{it}}) \\ +\left(1-E\left[Q_i|{\mathbf{Y}}_i\right]\right)\left(\log \left(1-\pi \right)+\sum _{i=1}^I\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}\left(X_{i1}\right)Z_{l_2}\left(X_{i2}\right)|{\mathbf{Y}}_i\right]\log \left(r_{l_1{l}_2}\right)\right. \\ \left.+\sum _{i=1}^I\sum _{t=3}^n\sum _{l_1=0}^k\sum _{l_2=0}^k\sum _{l_3=0}^{k}E\left[Z_{l_1}\left(X_{i,t-2}\right)Z_{l_2}\left(X_{i,t-1}\right)Z_{l_3}\left(X_{it}\right)\right)|{\mathbf{Y}}_i\right]\log \left({\lambda }_1{P}_{l_2{l}_3}+{\lambda }_2{P}_{l_1{l}_3}\right) \\ \left.+\sum _{i=1}^I\sum _{t=1}^n\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)|{\mathbf{Y}}_i\right]\log \left(M_{l_1{l}_2}\right)\right), \end{gathered}$$ (A2)

where $E\left[Q_i|{\mathbf{Y}}_i\right]$, $\left.E\left[Z_{l_1}\left(X_{i,t-2}\right)Z_{l_2}\left(X_{i,t-1}\right)Z_{l_3}\left(X_{it}\right)\right)|{\mathbf{Y}}_i\right]$, $E\left[Z_{l_1}\left(X_{i1}\right)Z_{l_2}\left(X_{i2}\right)|{\mathbf{Y}}_i\right]$, and $E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)|{\mathbf{Y}}_i\right]$ are calculated by an adaption of the forward-backward algorithm.

A.2. Second-order forward algorithm

The second-order forward algorithm is defined similarly to the first order in Section 3.2, however, now the forward quantity calculates the probability of being in specific latent states at time $t$ and $t-1$. The forward quantity is

$${\alpha }_{l_1{l}_2}(it{)}_j=P(Y_{i1}^{\mathrm{*}},Y_{i2}^{\mathrm{*}},\dots ,Y_{it,j}^{\mathrm{*}},X_{i,t-1}=l_1,X_{it}=l_2)$$ (A3)

and is calculated by

$$\begin{array}{c}{\alpha }_{l_1{l}_2}{(it)}_{j{j}^{\prime }}=\{\begin{array}{ll}{r}_{l_1{l}_2}{M}_{l_1{Y}_{ilj}}{M}_{l_2{Y}_{i2j^{\prime }}}\hfill & \text{if}t=2\hfill \end{array}\hfill \\ {\alpha }_{l_1{l}_2}{(it)}_j=\{\begin{array}{ll}\sum _{l_0=0}^k\sum _{m=1}^{\left|Y_{i,t-2}^{*}\right|}\sum _{m^{\prime }=1}^{\left|Y_{i,t-1}^{*}\right|}{\alpha }_{l_0{l}_1}{(i,t-1)}_{m{m}^{\prime }}({\lambda }_1{P}_{l_1{l}_2}+{\lambda }_2{P}_{l_0{l}_2})M_{l_2{Y}_{it,j}}\hfill & \text{if}t=3\hfill \\ \sum _{l_0=0}^k\sum _{m=1}^{\left|Y_{i,t-1}^{*}\right|}{\alpha }_{l_0{l}_1}{(i,t-1)}_m({\lambda }_1{P}_{l_1{l}_2}+{\lambda }_2{P}_{l_0{l}_2})M_{l_2{Y}_{it,j}}\hfill & \text{if}t>3.\hfill \end{array}\end{array}$$ (A4)

Unlike the first-order forward algorithm which starts at $t=1$, this algorithm is only defined at $t=2$ and later. For times $t>2$ the forward quantity can sum over all partially observed values for the state at time $t-1$ (while still keeping track of all partially observed values for the state at time $t$). When $t=2$, partially observed values for the first two states cannot be summed over, as they are necessary for later calculations. Therefore, the forward quantity at time $t=2$ is calculated slightly differently than at times $t>2$. Finally, define

$${\alpha }_{l_1{l}_2}(it{)}^{\mathrm{*}}=P(Y_{i1}^{\mathrm{*}},Y_{i2}^{\mathrm{*}},\dots ,Y_{it}^{\mathrm{*}},X_{i,t-1}=l_1,X_{it}=l_2)=\sum _{j=1}^{\left|Y_{it}^{\mathrm{*}}\right|}{\alpha }_{l_1{l}_2}(it{)}_j$$ (A5)

A.3. Second-order backward algorithm

The second-order backward algorithm, similarly to the second-order forward algorithm, includes two latent states in each backward quantity. The backward quantity is:

$${\beta }_{l_1{l}_2}(it)=P(Y_{i,t+1}^{\mathrm{*}},Y_{i,t+2}^{\mathrm{*}},\dots ,Y_{in}^{\mathrm{*}}|X_{i,t-1}=l_1,X_{it}=l_2)$$ (A6)

and is calculated by

$${\beta }_{l_1{l}_2}(it)=\{\begin{array}{ll}1\hfill & \text{if}t>n-2\hfill \\ \sum _{l_3}^k\sum _{j=1}^{\left|Y_{i,t+2}\right|}{\beta }_{l_2{l}_3}(i,t+1)({\lambda }_1{P}_{l_2{l}_3}+{\lambda }_2{P}_{l_1{l}_3})M_{l_3{Y}_{i,t+2,j}}\hfill & \text{if}t\le n-2.\hfill \end{array}$$ (A7)

A.4. Calculating expectations

The following are the E-step expectations, calculated using ${\alpha }_{l_1{l}_2}^{\mathrm{*}}$ (it) and ${\beta }_{l_1{l}_2}(it)$

$$E\left[Q_i|{\mathbf{Y}}_i\right]=\frac{P\left(Q_i=1,{\mathbf{Y}}_i\right)}{P\left({\mathbf{Y}}_i\right)}=\frac{\pi \prod _{t=1}^{n}P\left(Y_{it}|X_{it}=0\right)}{P\left({\mathbf{Y}}_i\right)},$$ (A8)

$$E\left[Z_{l_1}\left(X_{i1}\right)Z_{l_2}\left(X_{i2}\right)|{\mathbf{Y}}_i\right]=\frac{P\left(X_{i1}=l_1,X_{i2}=l_2,{\mathbf{Y}}_i\right)}{P\left({\mathbf{Y}}_i\right)}=\frac{{\alpha }_{l_1{l}_2}^{\mathrm{*}}(i2){\beta }_{l_1{l}_2}(i2)}{P\left({\mathbf{Y}}_i\right)},$$ (A9)

$$\begin{gathered} E[Z_{l_1}(X_{i,t-2})Z_{l_2}(X_{i,t-1})Z_{l_3}(X_{it}))|{\mathbf{Y}}_i]=\frac{P(X_{i,t-2}=l_1,X_{i,t-1}=l_2,X_{it}=l_3,{\mathbf{Y}}_i)}{P({\mathbf{Y}}_i)} \\ =\frac{{\alpha }_{l_1{l}_2}^{*}(i,t-1)({\lambda }_1{p}_{l_2{l}_3}+{\lambda }_2{p}_{l_1{l}_3})\sum _{j=1}^{\left|Y_{it}\right|}{M}_{l_3{Y}_{it,j}}{\beta }_{l_2{l}_3}(it)}{P({\mathbf{Y}}_i)}, \end{gathered}$$ (A10)

$$\begin{array}{ll}& E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{it}\right)|{\mathbf{Y}}_i\right]=P\left(X_{it}=l_1|{\mathbf{Y}}_i\right)P(W_{it}=l_2|X_{it}=l_1,{\mathbf{Y}}_i)\\ & =\left\{\begin{array}{ll}\frac{{\sum }_{l_0=0}^k{{\alpha }^{\mathrm{*}}}_{l_1{l}_0}(i2){\beta }_{l_1{l}_0}(i2)}{P\left({\mathbf{Y}}_i\right)}\times \frac{P({Y^{\mathrm{*}}}_{it,j}=l_2|X_{it}=l_1)}{{\sum }_{m=1}^{\left|Y_{it}^{\mathrm{*}}\right|}P({Y^{\mathrm{*}}}_{it,m}|X_{it}=l_1)}& \text{if}t=1\\ \frac{{\sum }_{l_0=0}^k{{\alpha }^{\mathrm{*}}}_{l_0{l}_1}(it){\beta }_{l_0{l}_1}(it)}{P\left({\mathbf{Y}}_i\right)}\times \frac{P({Y^{\mathrm{*}}}_{itj}=l_2|X_{it}=l_1)}{{\sum }_{m=1}^{\left|Y_{it}^{\mathrm{*}}\right|}P({Y^{\mathrm{*}}}_{it,m}|X_{it}=l_1)}& \text{if}t>1,\end{array}\right.\\ & \end{array}$$ (A11)

where

$$P({\mathbf{Y}}_i)=P({\mathbf{Y}}_i|Q_i=1)+P({\mathbf{Y}}_i|Q_i=0)=\pi \prod _{t=1}^{n}P(Y_{it}|X_{it}=0)+(1-\pi )\sum _{l_1=0}^k\sum _{l_2=0}^k{\alpha }_{l_1{l}_2}^{\ast }(in).$$ (A12)

APPENDIX B. COMPLETELY MISSING DATA

When the data is missing at random we can adapt the EM algorithm to deal with the missing values. We’ll treat the missing data as only missing in the observed states. Let ${\mathbf{Y}}_i=\left\{Y_{if(1)},\dots ,Y_{if(s)},\dots ,Y_{if\left(n_i\right)}\right\}$ be the vector of observed states for the $i^{\text{th}}$ individual where $f(s)$ denotes the $s^{\text{th}}$ nonmissing time point and $n_i$ is the number of nonmissing time points for individual $i$. Now, define ${\mathbf{W}}_i=\left\{W_{if(1)},\dots ,W_{if\left(n_i\right)}\right\}$ so that

$$W_{if(s)}=\left\{\begin{array}{ll}{Y}_{if(s)}& \text{if}{Y}_{if(s)}\text{is}\text{fully}\text{observed}\\ Y_{if(s{)}_j}^{\mathrm{*}}& \text{if}{Y}_{if(s)}\text{is}\text{partially}\text{observed},\end{array}\right.$$ (B1)

for a $j$ such that $1\le j\le \left|Y_{if(s)}^{\mathrm{*}}\right|$ and if $Y_{if(s)}^{\mathrm{*}}$ were to be fully observed, the full observation would be $Y_{if(s{)}_j}^{\mathrm{*}}$. The adjusted E-step is

$$\begin{gathered} E\left[\sum _{i=1}^I\log f_i\left({\mathbf{X}}_i,{\mathbf{W}}_i,Q_i\right)|{\mathbf{Y}}_i\right] \\ =E\left[Q_i|{\mathbf{Y}}_i\right]\left(\log \pi +\sum _{i=1}^I\sum _{t=1}^{n_i}\sum _{l_1=0}^{k}E\left[Z_0\left(X_{it}\right)Z_{l_1}\left(W_{if\left(t\right)}\right)|{\mathbf{Y}}_i\right]\log M_{0W_{it}}\right) \\ +\left(1-E\left[Q_i|{\mathbf{Y}}_i\right]\right)\left(\log \left(1-\pi \right)+\sum _{i=1}^I\sum _{l_1=0}^{k}E\left[Z_{l_1}\left(X_{i1}\right)|{\mathbf{Y}}_i\right]\log \left(r_{l_1}\right)\right. \\ +\sum _{i=1}^I\sum _{t=2}^n\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}\left(X_{i,t-1}\right)Z_{l_2}\left(X_{it}\right)|{\mathbf{Y}}_i\right]\log \left(p_{l_1{l}_2}\right) \\ \left.+\sum _{i=1}^I\sum _{t=1}^{n_i}\sum _{l_1=0}^k\sum _{l_2=0}^{k}E\left[Z_{l_1}\left(X_{it}\right)Z_{l_2}\left(W_{if\left(t\right)}\right)|{\mathbf{Y}}_i\right]\log \left(M_{l_1{l}_2}\right)\right) \end{gathered}$$ (B2)

To calculate the E-step with missing data the only change we need to make is to the forward-backward algorithm. Define ${\alpha }_{l_1}^{\mathrm{*}}(it)=P(Y_{if(1)}^{\mathrm{*}},\dots ,Y_{if\left(s^L\right)}^{\mathrm{*}},X_{it}=l_1)={\sum }_{j=1}^{\left|Y_{it}^{\mathrm{*}}\right|}{\alpha }_{l_1}(it{)}_j$ where $f\left(s^L\right)=\mathrm{m}\mathrm{a}\mathrm{x}(f(s)|f(s)\le t)$. Then we can calculate the forward algorithm as:

$${\alpha }_{l_1}{(it)}_j=\{\begin{array}{c}{r}_{l_1}\text{if}t=1\hfill \\ r_{l_1}{M}_{l_1{Y}_{i_j}}\text{if}t=f(1)=1\hfill \\ \sum _{l_0=0}^k\sum _{m=1}^{\left|Y_{i,t-1}^{\ast }\right|}{\alpha }_{l_0}{(i,t-1)}_m{p}_{l_0{l}_1}\text{if}f(s^L)<t\hfill \\ \sum _{l_0=0}^k\sum _{m=1}^{\left|Y_{i,t-1}^{\ast }\right|}{\alpha }_{l_0}{(i,t-1)}_m{p}_{l_0{l}_1}{M}_{l_1{Y}_{i,j}^{\ast }}^{\ast }\text{if}f(s^L)=t.\hfill \end{array}$$ (B3)

Also define ${\beta }_{l_1}(it)=P(Y_{if\left(s^U\right)}^{\mathrm{*}},\dots ,Y_{if\left(n_i\right)}^{\mathrm{*}}|X_{it}=l_1)$ where $f\left(s^U\right)=\mathrm{m}\mathrm{i}\mathrm{n}(f(s)|f(s)\ge t+1)$. Then the algorithm is calculated as,

$${\beta }_{l_1}(it)=\{\begin{array}{c}1\text{if}t\ge f(n_i)\hfill \\ \sum _{l_2=0}^k{p}_{l_1{l}_2}{\beta }_{l_2}(t+1)\text{if}f(s^U)>t+1\hfill \\ \sum _{l_2=0}^k{p}_{l_1{l}_2}\sum _{j=1}^{\left|Y_{i,t+1}^{\ast }\right|}{M}_{l_2{Y}_{i,t+1,j}}{\beta }_{l_2}(t+1)\text{if}f(s^U)=t+1.\hfill \end{array}$$ (B4)

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions. Code and instructions on how to install and use the proposed and the mixture transition distribution method is available at https://github.com/jordanaron22/PartiallyObservedHMM.