Parameter estimation in Cox models with missing failure indicators and the OPPERA study

Abstract

In a prospective cohort study, examining all participants for incidence of the condition of interest may be prohibitively expensive. For example, the “gold standard” for diagnosing temporomandibular disorder (TMD) is a physical examination by a trained clinician. In large studies, examining all participants in this manner is infeasible. Instead, it is common to use questionnaires to screen for incidence of TMD and perform the “gold standard” examination only on participants who screen positively. Unfortunately, some participants may leave the study before receiving the “gold standard” examination. Within the framework of survival analysis, this results in missing failure indicators. Motivated by the Orofacial Pain: Prospective Evaluation and Risk Assessment (OPPERA) study, a large cohort study of TMD, we propose a method for parameter estimation in survival models with missing failure indicators. We estimate the probability of being an incident case for those lacking a “gold standard” examination using logistic regression. These estimated probabilities are used to generate multiple imputations of case status for each missing examination that are combined with observed data in appropriate regression models. The variance introduced by the procedure is estimated using multiple imputation. The method can be used to estimate both regression coefficients in Cox proportional hazard models as well as incidence rates using Poisson regression. We simulate data with missing failure indicators and show that our method performs as well as or better than competing methods. Finally, we apply the proposed method to data from the OPPERA study.

1. Introduction

Time-to-event analyses are frequently conducted in medicine, actuarial science, and numerous other fields of applied science. There is a well-developed set of survival analysis methods implemented in standard software. Semi-parametric methods, such as the Cox proportional hazards model, allow robust estimation of the effects of covariates on the hazard function. However, these methods require the analyst to know the failure status of each participant, which may not always be available.

In some cases the outcome of interest may be difficult to ascertain. For example, in oncology studies, researchers may want to differentiate between deaths due to cancer and deaths due to car accidents or other unrelated causes. Investigators may easily record the mortality of all subjects, but it may be extremely difficult or costly to find out exactly why each subject died. One possible solution to this problem is delayed event adjudication [1]. This means that possible cases are not identified immediately but screened using simple methods that may have poor sensitivity or specificity. Later, the screened candidate cases are re-examined using a more precise, but also more costly and time-consuming, method to determine the true event status.

The study that motivates our work is Orofacial Pain: Prospective Evaluation and Risk Assessment (OPPERA), a prospective cohort study to identify risk factors for the onset of temporomandibular disorders (TMD). Each (initially TMD-free) OPPERA study participant was followed for a median of 2.8 years to identify cases of first-onset TMD. However, it was impractical to perform a physical examination on every participant. It would also have been inefficient given that most study participants did not develop the condition. Instead, this “gold standard” examination was performed only on participants with positive screens on a quarterly screening questionnaire that was designed to assess recent orofacial pain [2]. However, some participants with positive screens were lost to follow-up before receiving the “gold standard” examination. Thus a time-to-event analysis would have some participants with missing failure indicators.

Previous research indicates that when a subset of the failure indicators are missing, one can obtain more accurate estimates of the parameters of interest by using appropriate tools to estimate these missing values [1, 3, 4]. Cook and Kosorok [1] estimate parameters in Cox proportional hazard models with missing failure indicators by weighting observations according to their probability of being a true case. They show that the estimators are consistent and asymptotically normally distributed. However, the standard error of their proposed estimate cannot be easily obtained using existing software without bootstrapping. For the OPPERA data, a separate Cox model was calculated for each putative risk factor of interest, including approximately three thousand genetic markers. Consequently, applying this method to the OPPERA genetic data would be computationally intractable.

In the OPPERA study, the likelihood that a participant with a positive screen was examined was weakly associated with demographic variables such as gender, race, or socioeconomic status [2]. This indicated that the failure indicators in the OPPERA study were not missing completely at random (MCAR). Application of models that assume MCAR failure indicators may result in biased estimates of hazard ratios for covariates of interest. More importantly, a participant’s responses to their screening questions are predictive of whether or not they are an incident case of TMD. This setting presents statistical challenges, which require care in order to avoid bias and maintain efficiency. Additionally, incidence rate estimates are desired, and none of the methods currently available allow for estimation of the incidence rate. There is a clear need for new methodology to effectively answer the research questions of the OPPERA study.

In this paper, we propose a method for parameter and variance estimation in Cox regression models with missing failure indicators. The motivating data set is introduced in Section 2. We describe our method in Section 3. In Section 4, we report the results of simulations. Finally, in Section 5 we apply our method to the OPPERA study. We conclude with a discussion in Section 6.

2. Motivating Data Set: The OPPERA Study

OPPERA is a prospective cohort study designed to identify risk factors for first-onset TMD. A total of 3,263 initially TMD-free subjects were recruited at four study sites between 2006 and 2008. TMD status was confirmed by physical examination of the jaw joints and muscles using the Research Diagnostic Criteria for TMD [5], which is the gold standard for diagnosing TMD.

Upon enrollment in the study, each OPPERA participant was evaluated for a wide variety of possible risk factors for TMD, including psychological distress, previous history of painful conditions, and sensitivity to experimental pain. For a brief overview of the risk factors of interest in the OPPERA study, see Section S1 in the Supporting Information. See Ohrbach et al. [6], Fillingim et al. [7], Greenspan et al. [8], Maixner et al. [9], and Smith et al. [10] for a complete description of the baseline measures that were collected in OPPERA.

After enrollment, each participant was asked to complete questionnaires to evaluate recent orofacial pain once every three months. These questionnaires (hereafter referred to as “screeners”) evaluated the frequency and severity of pain in the orofacial region during the previous three months. The purpose of the screener was to identify participants who were likely to have recently developed TMD. For a complete description of the screener, see Slade et al. [11]. Participants with a positive screen were asked to undergo a follow-up physical examination by a clinical expert to diagnose presence or absence of TMD.

Of the 3,263 subjects, 2,737 filled out at least 1 screener, and the remaining 521 did not fill out any screeners. The total number of screeners was 26,666. There were 717 positive screeners, 486 (about 68%) of which were followed by a clinical examination. As reported in Bair et al. [2], case classifications made by one examiner (hereafter, “Examiner #4”) were deemed unreliable because the examiner diagnosed a much higher percentage of individuals with TMD compared to other examiners. We therefore set all of Examiner #4’s physical examination findings to be missing and imputed them using the methods in this paper. This left 404 positive screeners (56%) resulting in valid clinical exams.

3. Model

3.1. Notation and Assumptions

Assume there are n independent participants. For each participant i (i = 1, …, n), let C_i and T_i denote the potential times until censoring and failure, respectively, and let V_i = min(T_i, C_i), Δ_i = I(T_i ≤ C_i). Let Z_i a p × 1 vector of covariates measured at baseline and let X_i be a q × 1 vector of covariates measured at the time of the putative event. We assume the hazard for participant i follows a Cox proportional hazards model

$$\lambda (t|z_i)={\lambda }_0(t)\text{exp}(\beta \prime z_i)$$ (1)

where λ₀(t) is an unspecified baseline hazard function. Let ξ_i denote the indicator that Δ_i is observed. We observe (V_i, ξ_i) for i = 1, …, n and Δ_i when ξ_i = 1.

In the OPPERA study, V_i is the length of time for participant i between enrollment in the study and either of two events

1. a screener which resulted in a diagnosis of incident TMD

2. the last-completed screener before loss-to-follow-up.

Note that participants with a positive screen do not fill out additional screeners until they are examined, so V_i will be the time until the positive screen for a participant who has a positive screen but is never examined. If participant i had a positive screen and subsequently was diagnosed with TMD, then Δ_i = 1. If participant i either had a negative screen on the last quarterly screener before loss-to-follow up or a positive screen and was diagnosed to be free of TMD, then Δ_i = 0. If participant i had a positive screen on the last screener but was not examined, then Δ_i is missing and ξ_i = 0. The putative risk factors for TMD that were assessed at enrollment are denoted by the vector Z_i. Responses to the screener for participant i at time V_i are denoted by the vector X_i. For OPPERA, we also define Q_i = 1 if participant i has a positive screen on their final screener and Q_i = 0 otherwise.

We assume the failure indicators are missing at random (MAR) as follows:

$$P({\xi }_i=0|X_i,Z_i,V_i,{\mathrm{\Delta }}_i,Q_i=1)=P({\xi }_i=0|X_i,Z_i,V_i,Q_i=1)$$ (2)

In other words, the probability of having a missing failure indicator may depend on measured factors, but it does not depend on whether or not an event occurred. We will describe how to estimate the probability in (2) in Section 3.2 and then show how to use this estimate to impute the missing event indicators in Section 3.3.

3.2. Estimating Event Probabilities

We model the probability that participant i with a missing failure indicator is a case by a logistic regression model based on X_i and V_i:

$$P({\mathrm{\Delta }}_i=1|X_i,V_i,Z_i,{\xi }_i=0,Q_i=1)=\frac{\text{exp}(\alpha \prime X_i+\gamma \prime Z_i+\eta V_i)}{1+\text{exp}(\alpha \prime X_i+\gamma \prime Z_i+\eta V_i)}I(Q_i=1)$$ (3)

That is, we estimate the probability of examiner-diagnosed TMD in a participant who was not examined as intended. (Here I(x) denotes an indicator function.) The probability was estimated using the time between enrollment and their last positive screener as well as their answers on that screener. Then, for those individuals who had a positive screen on the last screener (i.e. those with Q_i = 1) and were not examined, the estimated probability of being a case is estimated by (3) with the parameters replaced by their respective estimates based on individuals who were examined.

Note that this also assumes that there is one observation per subject, which may not be the case in practice. For example, if some participants had a positive screen on more than one screener and are examined at least once, then we have multiple observations per participant. In that case, fitting a generalized linear mixed effects logistic regression model rather than a standard logistic regression model could account for correlations between the responses of the same participant. However, only a small number of participants in the OPPERA study were examined multiple times after positive screeners, so we simply discarded all but the most recent screener when analyzing the OPPERA data (thereby avoiding this problem of repeated observations).

3.3. Multiple Imputation

One popular method for handling missing data is multiple imputation. For a comprehensive review on multiple imputation, see Rubin [12]. Our imputation procedure is as follows:

1. Estimate the coefficients α, γ, and η in (3). We used a Bayesian model where α, γ, and η had a prior distribution that was Cauchy with center 0 and scale 2.5.

2. For each observation with a missing failure indicator, sample from the posterior distribution of α, γ, and η to obtain an estimate of the probability that an event occurred for each such observation.

3. Generate a Bernoulli random variable with success probability equal to the predicted probability found in step (2).

4. Combine the raw data and imputed data from step (3) to form a completed data set.

5. Fit the Cox proportional hazards model to the completed data set.

6. Record each parameter estimate β̂_j and covariance matrix Û_j.

7. Repeat steps (3)–(6) for a total of m times, where m is the desired number of imputations.

Next, we combine all of the estimates. The average parameter estimate is

$$\overline{\beta }=\frac{1}{m}\sum _{j=1}^m{\hat{\beta }}_j,$$ (4)

the within-imputation variance estimate is

$$Ū=\frac{1}{m}\sum _{j=1}^m{Û}_j,$$ (5)

and the between-imputation variance

$$\hat{B}=\frac{1}{m-1}\sum _{j=1}^m({\hat{\beta }}_j-\overline{\beta })({\hat{\beta }}_j-\overline{\beta })\prime .$$ (6)

Finally, the estimated covariance matrix is

$$\widehat{\mathrm{V}\mathrm{a}}\mathrm{r}(\overline{\beta })=Ū+(1+\frac{1}{m})\hat{B}.$$ (7)

It can be shown that $\overline{\beta }/\widehat{\mathrm{V}\mathrm{a}}\mathrm{r}(\overline{\beta })$ is approximately t distributed with degrees of freedom

$$(m-1)(1+\frac{mŪ}{(m+1)\hat{B}})$$ (8)

(7) and (8) can be used to compute confidence intervals for the multiply imputed parameter estimate β̄.

3.4. Estimation of Incidence

Previous sections of this paper described how to estimate hazard ratios in the presence of missing failure indicators. It may also be of interest to estimate incidence rates for the same event using Poisson regression instead of Cox regression. For example, one of the aims of the OPPERA study is to estimate the incidence rate of first-onset TMD.

In order to estimate incidence rates, we estimate the case probabilities as described previously based on participants who had a positive screen and were examined. Then we impute case status as described in Section 3.3 for those who had a positive screen but were not examined. However, in this case we fit Poisson regression models, rather than Cox models, to the completed data sets. Finally, we calculate the incidence rate based on the estimates of the regression coefficients in the Poisson model. Specifically, we use the data from imputation j to fit the model

$$\text{log}(E({\mathrm{\Delta }}_{ij}|X_i,Z_i,V_i))=\mu +\tau \prime X_i+\lambda \prime Z_i+\text{log}(V_i)$$ (9)

where Δ_ij denotes the j^th imputation for observation i, j = 1, …, m. We combine the m imputations using equation (4) and

$$\overline{\mu }=\frac{1}{m}\sum _{j=1}^m{\hat{\mu }}_j.$$ (10)

The estimated incidence rate for an individual with covariates X^* and Z^* is given by exp(μ̄ + τ̄X^* + λ̄Z^*). The variability of μ̄, τ̄, and λ̄ may be estimated using (7), and confidence intervals may be computed based on the t distribution using (8), as described previously.

4. Simulations

Data with missing failure indicators were simulated, and several possible methods were compared with respect to bias, coverage, and confidence interval width. Survival times for 1,000 individuals were generated with exponentially distributed failure times under a proportional hazards model with covariates as proposed by Bender et al. [13]. That is, the survival time for each individual was distributed according to (1) where λ₀(t) = 1 is the baseline hazard. For our simulations, Z_i was a single baseline covariate following a normal distribution with mean 2 and unit variance. In other words, conditional on Z_i, the failure times T_i followed an exponential distribution with hazard exp(β′Z_i) where β ∈ {−0.5, −1.5, −3}. The censoring times C_i followed an exponential distribution with mean 5 (corresponding to a hazard of exp(−log(5)) ≈ exp(−1.61)). This yielded about 35%, 75% and 90% censoring for β = −0.5, β = −1.5, and β = −3, respectively. We also defined Δ_i = I(T_i ≤ C_i). If Δ_i = 0, the implication is that the follow up period ended before the participant developed TMD, meaning that the observation was censored at time C_i.

Covariates are represented by Z_i, a risk factor for TMD measured at enrollment, and X_i, a measurement collected on the last screener. For each observation, a normally distributed covariate X_i1 was generated with mean Δ_i and standard deviation 0.3. In OPPERA, X_i represents a question on the screener evaluating some symptom of first-onset TMD, such as the frequency of jaw pain. This was used to generate Q_i = I(X_i > 0.5), an indicator of whether participant i screened positive on their last screener. Note that X_i depends on Δ_i, since participants who developed first-onset TMD are more likely to report symptoms on their screener, and Q_i depends on X_i, since the screener is positive if enough symptoms are reported. Also, ξ_i = I(Δ_i is observed) corresponds to the indicator of whether participant i came in for their clinical exam if Q_i = 1. In all simulations, δ_i was used as the failure indicator rather than Δ_i, where δ_i is defined as

$${\delta }_i=\{\begin{array}{cc}{\mathrm{\Delta }}_i\hfill & \text{if }{Q}_i=1\hfill \\ 0\hfill & \text{if }{Q}_i=0\hfill \end{array}$$

In other words, we set the failure indicator δ_i = 0 if the final screener was negative. This decision was made to reflect the fact that OPPERA participants who had a negative screen were not examined. Hence it is possible that some participants developed first-onset TMD but were never examined due to their final screener being negative. Thus, the simulations (incorrectly) treat these observations as censored.

We created missing failure indicators under the following classical missing data mechanisms of Rubin [14]:

1. The probability of having a missing failure indicator is independent of the data. This is known as missing completely at random (MCAR).

2. The probability of having a missing failure indicator depends on an observed covariate. This is known as missing at random (MAR).

3. The probability of having a missing failure indicator depends on the (potentially unobserved) failure indicator. This is known as missing not at random (MNAR).

Our method assumes that the data are MAR, which includes MCAR as a special case. Our simulations under MAR and MNAR parallel the study protocol in that failure indicators can only be missing for those with positive screeners. In other words, observations were potentially missing if and only if Q_i = 1. (Individuals with negative screeners have Q_i = 0 and are assumed to be censored. Those with positive screeners have Q_i = 1 and may have missing clinical examinations.) Details and results for MCAR and MNAR data are shown in Sections S2.2 and S2.4 in the Supporting Information. We also considered several simulation scenarios where the logistic regression model for predicting the failure indicator was misspecified; see Section S2.3 in the Supporting Information. For MAR data, we set failure indicators to be missing with probability

$$P({\xi }_i=0|X_i,Z_i,V_i,Q_i=1)=\frac{\text{exp}(-0.2-0.3Z_i+0.1V_i)}{1+\text{exp}(-0.2-0.3Z_i+0.1V_i)}$$ (11)

This resulted in approximately 50% of failure indicators being set to missing, which is consistent with the rate of missing failure indicators in the OPPERA study.

In each simulated data set, all observations with observed failure indicators who had a positive screen were used to fit a logistic regression model for case status with covariates Z_i, X_i and V_i. That is, using the complete data (i.e. observations with Q_i = 1 and ξ_i = 1), we fit the logistic regression model for the event probability conditional on Z_i, X_i, and V_i, namely

$$\text{logit}\{P({\mathrm{\Delta }}_i=1|X_i,Z_i,V_i,Q_i=1,{\xi }_i=1)\}=\alpha \prime X_i+\gamma \prime Z_i+\eta V_i$$ (12)

The estimated probabilities ${\hat{p}}_i=\frac{\text{exp}(\hat{\alpha }\prime X_i+\hat{\gamma }\prime Z_i+\hat{\eta }{V}_i)}{1+\text{exp}(\hat{\alpha }\prime X_i+\hat{\gamma }\prime Z_i+\hat{\eta }{V}_i)}$ were calculated for individuals with Q_i = 1 (where α̂, γ̂, and η̂ are drawn from their posterior distribution).

To evaluate the performance of our method, multiple imputation was employed to calculate 10 imputed estimates of β for each simulation as described in Section 3.3. For each observation i with Q_i = 1 and ξ_i = 0, we estimated failure indicators Δ̂_ij independently for each imputation j.

A Cox proportional hazards model was fit for each imputed data set, and the imputed estimates of the regression coefficient and their variances were recorded. These were aggregated using equations (4) and (7) to create confidence intervals for the multiple imputation estimates.

The performance of our method was compared with that of the method of Cook and Kosorok [1]. To obtain the estimates of Cook and Kosorok [1], for each simulated data set, we estimated the probabilities p̂_i that the (potentially unobserved) event for participant i is a true event, as described previously. We then fit a weighted Cox proportional hazards model to the data set with weights calculated as follows: Each observation with a missing failure indicator was deleted and replaced with two new observations. Each such pair of observations had the same failure time and covariates, but different failure indicators and weights. The first observation had weight p̂_i and Δ̂_i = 1, and the second observation had weight 1 − p̂_i and Δ̂_i = 0. Participants with fully observed data retained a single observation in the data set with unit weight. The estimated regression coefficient, β̂ was recorded.

The variance of this estimate was estimated by generating 1,000 bootstrap replicates of each simulated data set and refitting the model for each bootstrap replicate. A set of 1,000 subjects was selected at each bootstrap iteration by sampling from the data with replacement. For each bootstrap replicate, the estimated probability ${\hat{p}}_i^{*}$ that participant i is a true failure was calculated. These estimated ${\hat{p}}_i^{*}$ ’s were used to calculate a bootstrap estimate β̂^* of β using a weighted Cox model as described in the previous paragraph. The average parameter estimate, β̂̄ and percentile confidence intervals (β_0.025, β_0.975) were all recorded, where β_θ is the θ^th quantile among the 1,000 bootstrap replicates.

We also compared our method to the ideal situation in which the true values of Δ_i were observed for all observations (note that Δ_i was used instead of δ_i in this case), complete case analysis (meaning that we exclude from the data set all observations with missing failure indicators), and two ad hoc methods in which we treat the missing indicators either all as censored or all as failures. Results under the assumption of MAR are shown in Table 1. We estimated the bias of each method by calculating the mean difference between the estimated Cox regression coefficient and the true coefficient over the 1,000 simulations. We also calculated the mean width of the confidence intervals produced by each method over the 1,000 simulations. Similarly, we calculated the empirical coverage probability for the confidence intervals produced by each method by dividing the number of times that the confidence intervals contained the true value of the parameter by 1,000. We also report the Monte Carlo error for the coverage rate, which is the error in the empirical coverage probability due to conducting only a finite number of simulations (which would be $\sqrt{\alpha (1-\alpha )/n}$ for n simulations). Finally, the rate of missing information and the average running time of each method was computed.

Table 1.

Simulation Results for MAR

β*	Method	Bias	SE (Bias)	Width	SE (Width)	Coverage†	Running Time (s.)
−0.5	Full Data	−0.0008	0.0005	0.1666	0.0004	0.962	0.008
	Complete Case	0.0033	0.0007	0.2152	0.0004	0.955	0.007
	Treat all as Censored	0.1058	0.0007	0.2127	0.0004	0.514	0.007
	Treat all as Failures	0.0018	0.0005	0.1699	0.0004	0.964	0.008
	Cook & Kosorok	−0.0009	0.0005	0.1728	0.0004	0.959	22.0
	Multiple Imputation	−0.0003	0.0005	0.1721	0.0004	0.961	0.49
−1.5	Full Data	0.0047	0.0011	0.3176	0.0002	0.938	0.008
	Complete Case	−0.0558	0.0015	0.4317	0.0003	0.927	0.007
	Treat all as Censored	0.1241	0.0014	0.421	0.0003	0.767	0.007
	Treat all as Failures	0.0716	0.0011	0.3154	0.0002	0.841	0.007
	Cook & Kosorok	0.0052	0.0011	0.3399	0.0003	0.942	17.50
	Multiple Imputation	0.0082	0.0011	0.3353	0.0002	0.942	0.40
−3	Full Data	−0.0294	0.0025	0.7606	0.0009	0.945	0.007
	Complete Case	−0.2044	0.0036	1.0855	0.0017	0.918	0.008
	Treat all as Censored	0.0988	0.0034	1.0413	0.0015	0.92	0.008
	Treat all as Failures	0.5914	0.0025	0.6293	0.0006	0.085	0.008
	Cook & Kosorok	−0.0302	0.0029	0.9078	0.0017	0.94	17.33
	Multiple Imputation	−0.0042	0.0028	0.8556	0.0014	0.947	0.43

^*The rate of missing information is 0.017 when β = −0.5, 0.061 when β = −1.5, and 0.178 when β = −3.

^†The Monte Carlo error is 0.007.

All calculations were performed using R versions 3.0.2 running on a single core of a Dell C6100 server with a 2.93 GHz Intel processor. The function “mi.binary” in the “mi” R package was used to generate the imputed values of the missing failure indicators. The functions “boot” and “boot.ci” in the “boot” R package were used to calculate the bootstrap estimates of the standard error of the Cook and Kosorok [1] method. The Cox proportional hazard models were fit using the “coxph” function in the “survival” R package. The code used to perform the simulations (and analyze the OPPERA data) is available in the Supporting Information.

The empirical coverage probability of the confidence intervals produced by multiple imputation is close to the nominal level (0.95) in all simulations. Our multiple imputation method and the method of Cook and Kosorok [1] produced approximately unbiased estimates and valid confidence intervals in all the scenarios we considered. The estimates produced by the other methods showed a larger amount of bias and did not always achieve the desired coverage level. Our multiple imputation method also yielded the narrowest confidence intervals in each scenario. Although the method of Cook and Kosorok [1] produced confidence intervals that were only slightly wider, this indicates that our proposed method may have slightly greater power to detect true associations, particularly when the absolute value of β is large. Our proposed method also tended to have lower bias than the method of Cook and Kosorok [1] when the absolute value of β is large. The running time of our proposed method was also significantly less than the running time of the Cook and Kosorok [1] method. Moreover, for most parameter values, the coverage probabilities for the complete case and ad hoc methods were significantly different (p < 0.01) from the nominal rate.

In addition, we examined the performance of our proposed methods when we changed the logistic regression model for Δ_i. We investigate two additional types of models: one in which the model contained a variable unrelated to case status and another in which the model does not include one variable related to case status. As in the previous simulations, the failure times were generated by (1), censoring was exponential with mean 5, failure indicators were set to be missing completely at random or missing at random with probability given in equation (11), Z_i ~ N(2, 1), X_i1 ~ N(Δ_i, 0.3) and Q_i = I(Y_i2 > 0.5) for i = 1, …, n. We also generated X_i2 ~ N(0, 1) where Z_i, X_i1, X_i2 were mutually independent and X_i2 was independent of Δ_i and Q_i.

In the previous simulations, we fit the data to (12) with covariates Z_i and X_i = X_i1. The additional simulations instead used the covariates and parameters as follows:

1. X̃_i = {1, X_i1, X_i2}

2. X̃_i = 0.

That is, rather than fitting model (12) to the data, we modeled the case probability with

$$\text{logit}\{P({\mathrm{\Delta }}_i=1|X_i,Z_i,V_i,Q_i=1)\}=\tilde{\alpha }\prime {\tilde{X}}_i+\gamma Z_i+\eta V_i.$$ (13)

The results, which are shown in Section S2.3 in the Supporting Information, remained similar under both alternative models. This indicates that the proposed methods are robust to misspecification of the logistic regression model in some situations. Most notably, leaving out one covariate that was weakly related to case status did not markedly decrease the performance of the method.

We also performed some simulations where a random subset of the observations with Q_i = 0 were set to have missing failure indicators. The model to predict Δ_i was fitted using only the observations for which Q_i = 1, but the model was applied to all observations with missing failure indicators (including observations where Q_i = 0). The results are shown in Section S2.3 in the Supporting Information. In this case our method (as well as the Cook and Kosorok [1] method) produced reasonable results when the logistic regression model was specified correctly or when an extra covariate was included in the model. However, both methods performed poorly when an important covariate was missing from the logistic regression model.

Finally, we conducted simulations to evaluate the method’s ability to estimate incidence rates. A similar multiple imputation strategy was applied to Poisson regression. Our method produced estimates much closer to the true incidence rates than the complete case estimate. In fact, the complete case method underestimated incidence rates by as much as a factor of 3. See Section S2.5 in the Supporting Information for details.

5. Analysis of the OPPERA Study

In this section, we apply our method to estimate hazard ratios and incidence rates in the OPPERA study using m = 10 imputations.

5.1. Hazard Ratios

We applied our method to the OPPERA cohort to adjust for the effect of participants with missing clinical examinations. (Note that examinations for participants evaluated by Examiner #4 were also treated as missing.) First, we estimated the probability that a participant would be diagnosed as an incident case of TMD given a positive screener. Due to the rich body of information collected in each screener, we carefully selected a small number of predictor variables. Specifically, we fit a logistic regression model to predict the result of the clinical exam based on each item in the screener. As described previously, the regression coefficients were assumed to have a prior distribution that was Cauchy with center 0 and scale 2.5. All models were adjusted for study site.

The majority of the variables measured on the screener were not associated with the result of the clinical examination. The strongest predictor of being diagnosed with TMD was a count of non-specific orofacial symptoms (e.g stiffness, fatigue) in the previous three months. The time elapsed since enrollment and OPPERA study site were also important covariates, as shown in Bair et al. [2]. Several other possible predictors of being diagnosed with TMD were identified, but including these additional predictors in the model did not improve the predictive accuracy of the model and hence they were not included. (In general failure to include a relevant predictor variable when performing multiple imputation will produce greater error than including an irrelevant variable as evidenced by our simulations, so generally it is better to err on the side of including too many predictors rather than too few. However, in this case, our testing indicated that including additional variables did not improve the predictive accuracy of the model and in fact might actually decrease the accuracy. Hence, in this case we favored the more parsimonious model.)

Thus, we estimated the probability of being diagnosed with TMD based on the count of non-specific orofacial symptoms, time since enrollment, and OPPERA study site. This model was used to perform multiple imputation for those with no clinical examination. These imputed data sets were used to fit a series of Cox proportional hazards models to estimate the hazard ratio (and associated confidence interval and p-value) for each predictor using the methods described in Section 3.3. Examples of predictors include perceived stress, history of comorbid chronic pain conditions, and smoking status.

In addition, Bair et al. [2] examined univariate relationships between examination attendance and numerous possible predictor variables. Differences between examined and non-examined participants were small and most were not statistically significant. However, a few of the differences were statistically significant, indicating that the data were not MCAR, since MCAR requires that the probability of a missing observation does not depend on the data.

Table 2 shows the results of applying our method to a subset of the putative risk factors of TMD measured in OPPERA. Due to the large number of putative risk factors measured in OPPERA, we only report the results for a selected subset of the variables. All continuous variables were normalized to have mean 0 and standard deviation 1 prior to fitting the Cox models. (Thus, the hazard ratios for the continuous variables represent the hazard ratios corresponding to a one-standard deviation increase in the predictor variable.) In Table 2, all the quantitative sensory testing and psychosocial variables were continuous, while all of the clinical variables were dichotomous (and hence were not normalized). The small number of missing values in these predictor variables were (singly) imputed using the EM algorithm; see Greenspan et al. [8] or Fillingim et al. [7] for details. For a more detailed description of the OPPERA domains, see Section S1 in the Supporting Information, Maixner et al. [15], and Slade et al. [16].

Table 2.

Results from the OPPERA Study

Treat All MCIs as Censored					Multiple Imputation
	HR	LCL	UCL	P	HR	LCL	UCL	P
Clinical Variable
In the last month could not open mouth wide	3.26	1.83	5.84	<0.0001	2.35	1.39	3.96	0.0015
Has two or more comorbid chronic pain disorders	3.08	2.26	4.21	<0.0001	2.36	1.79	3.11	<0.0001
History of 5 respiratory conditions	1.38	1.01	1.87	0.0408	1.44	1.13	1.85	0.0040
Smoking: current	1.26	0.86	1.84	0.2403	1.48	1.07	2.04	0.0166
Smoking: former	1.87	1.22	2.87	0.0041	1.70	1.18	2.46	0.0045
One or more palpation tender points: right temporalis	1.83	1.32	2.52	0.0002	1.54	1.18	2.02	0.0018
One or more palpation tender points: left temporalis	1.60	1.14	2.25	0.0064	1.50	1.13	1.98	0.0045
One or more palpation tender points: right masseter	1.85	1.35	2.53	0.0001	1.69	1.31	2.17	<0.0001
One or more palpation tender points: left masseter	1.70	1.23	2.35	0.0013	1.50	1.15	1.97	0.0031
Quantitative Sensory Testing Variable
Pressure pain threshold: temporalis	1.26	1.07	1.49	0.0065	1.14	1.00	1.31	0.0466
Pressure pain threshold: masseter	1.23	1.04	1.45	0.0170	1.14	0.99	1.31	0.0674
Pressure pain threshold: TM joint	1.25	1.05	1.48	0.0106	1.15	1.01	1.32	0.0416
Mechanical pain aftersensation: 512mN probe, 15 s	1.23	1.09	1.38	0.0006	1.15	1.04	1.28	0.0071
Mechanical pain aftersensation: 512mN probe, 30 s	1.20	1.07	1.34	0.0020	1.12	1.02	1.24	0.0241
Psychosocial Variable
PILL Global Score	1.52	1.35	1.71	<0.0001	1.42	1.29	1.58	<0.0001
EPQ-R Neuroticism	1.39	1.21	1.60	<0.0001	1.25	1.11	1.42	0.0003
Trait Anxiety Inventory	1.43	1.25	1.64	<0.0001	1.34	1.19	1.52	<0.0001
Perceived Stress Scale	1.35	1.17	1.55	<0.0001	1.29	1.15	1.44	<0.0001
SCL 90R Somatization	1.44	1.31	1.58	<0.0001	1.40	1.29	1.51	<0.0001

HR, hazard ratio; LCL, lower confidence limit; UCL, upper confidence limit; TM, temporomandibular; PILL Pennebaker Inventory of Limbic Languidness; EPQ, Eysenck Personality Questionnaire; SCLR-90R, Symptom Checklist-90, Revised.

The rate of missing information varied slightly for each putative risk factor. The average rate of missing information was approximately 0.097. Compared to the unimputed results, which treated missing failure indicators as censored observations, imputation slightly reduced the hazard ratios for most of the psychosocial variables that were measured in OPPERA. For instance, Table 2 shows the (standardized) hazard ratios for the Pennebaker Inventory of Limbic Languidness (PILL) score, the neuroticism subscale of the Eysenck Personality Questionnaire (EPQ), the Spielberger Trait Anxiety Inventory score, the Perceived Stress Scale, and the somatization subscale of the Symptom Checklist-90, Revised (SCL-90R). In each case, the hazard ratios were reduced after imputation.

A similar pattern was observed after applying our imputation method to the measures of experimental pain sensitivity. The mechanical pain aftersensation ratings were strongly associated with first-onset TMD before imputation, but they were only weakly associated with first-onset TMD after imputation. The pressure pain algometer ratings were also more weakly associated with TMD after imputation (and one of three ratings in Table 2 was no longer significantly associated with first-onset TMD at the p < 0.05 level).

Interestingly, the hazard ratios for the presence of one or more palpation tender points at the temporalis and masseter muscles were also attenuated after imputation. These tender points were evaluated as part of the clinical examination using a different protocol than the quantitative sensory testing algometer pain ratings. However, both pain measures (algometer and palpation) were measured at the same facial locations. While the palpation ratings were more strongly associated with first-onset TMD than the algometer ratings both before and after imputation, it is interesting that different pain sensitivity measures using different protocols at the same anatomical location were both attenuated by imputation.

The effects of other clinical variables were also attenuated after imputation. For example, the hazard ratios associated with being unable to open one’s mouth wide in the past month and having two or more comorbid pain conditions were both noticeably attenuated after imputation. However, other clinical variables were more strongly associated with first-onset TMD after imputation. For example, having a history of respiratory illness was only weakly associated with first-onset TMD before imputation (HR=1.38, p=0.04), but the association was much stronger after imputation (HR=1.43, p=0.004). Also, being a current smoker was not significantly associated with first-onset TMD before imputation (HR=1.26, p=0.24) but was associated after imputation (HR=1.49, p=0.02).

5.2. Incidence Rates

In Table 3, the incidence rate of first-onset TMD was estimated using two different approaches. First, all missing failure indicators were treated as censored. Second, the multiple imputation method in this paper was used to estimate the incidence rate. The estimated TMD incidence rate using multiple imputation was 70% greater than the unimputed estimate. The estimated incidence rate increased by 70% for females and 87% for males. Estimated incidence rates for whites and Hispanics were 118% and 202% higher, respectively, with imputation. Thus, the incidence rate is likely to be underestimated without imputation.

Table 3.

Estimated TMD Incidence Rates With and Without Imputation

	No MI	MI	Percent Change
Overall	2.23	3.78	70%
Males	1.87	3.49	87%
Females	2.46	4.19	70%
White	1.70	3.70	118%
Black	4.20	5.70	36%
Hispanic	1.17	3.53	202%
Other	1.10	1.86	69%

Incidence rates are given in cases per 100 person-years.

6. Discussion

We have developed a computationally efficient method to adjust for missing failure indicators in time-to-event data using logistic regression and multiple imputation. Logistic regression is used to estimate the failure probability for participants with missing failure indicators. The missing values are imputed, and the standard errors are estimated using our multiple imputation method. This framework is important in studies where failure status may be measured in stages, which may lead to missing failure status indicators. This is a common occurrence in studies of diseases that are difficult or expensive to diagnose, such as TMD.

The present method is similar to the method of Magder and Hughes [17], who use an iterative procedure for parameter estimation based on the EM algorithm. Our assumption of MAR data renders their iterative method unnecessary. Other methods [18, 19, 20] depend on the MCAR assumption, which does not hold for the OPPERA study. Chen et al. [21] estimate Cox regression parameters using the EM algorithm and establish their consistency under basic regularity conditions, including missing at random (MAR) failure indicators. However, their approach depends on the assumptions of piecewise constant proportional hazard functions for the censoring time as well as for the failure time.

In each simulation scenario, our multiple imputation method produced the narrowest valid confidence intervals and no significant bias. In particular, the method of Cook and Kosorok [1] produced slightly wider confidence intervals in all but one of the simulations we considered. The differences were small, so the performance of the two methods appear to be comparable for most practical purposes. However, we believe that our method has several possible advantages over the method of Cook and Kosorok [1]. First, bootstrapping is much more intensive computationally than our multiple imputation approach. Calculating bootstrap confidence intervals generally requires at least 1000 bootstrap replicates [22], whereas as few as 10 imputed data sets may be sufficient for multiple imputation [23]. Although the difference in the computing time of the two methods is small for a single fitted model, many such models will be required in the course of the OPPERA study. OPPERA has already collected data on approximately three thousand genetic markers and has plans to collect data on approximately a million genetic markers in a genome-wide association study. Thus, at least three thousand (and potentially as many as a million) Cox models will need to be fit, and our proposed method may allow for a significant decrease in computing time. Moreover, our method can also be easily implemented in popular statistical software packages (such as SAS) without additional programming.

Additionally, our methodology may easily be extended to other models, such as Poisson regression. We conducted simulations (Table S9 in the Supporting Information) that showed that our proposed method can be used to estimate incidence rates using Poisson regression, which is one of the research aims of the OPPERA study. In particular, estimates of the failure rates were biased when missing failure indicators were treated as censored or when the complete case method was used, but they were unbiased when we employed the methodology in this paper.

Our method may yield increased bias and decreased coverage if the logistic regression model for predicting case status is inaccurate, as observed in the simulations in Section S2.3 in the Supporting Information. However, this would also be true for competing methods, including the method of Cook and Kosorok [1].

Our proposed also requires that the missing data be MAR. Although it is impossible to test this assumption directly, Bair et al. [2] showed that there were no significant differences between those who did and not attend their clinical examination with respect to a wide range of demographic variables and putative risk factors for TMD. Thus, the MAR assumption is reasonable for OPPERA. Furthermore, the results of the simulations described in Section S2.4 in the Supporting Information, show that our proposed method can produce valid results in some situations even if the MAR assumption is violated.

Also, our proposed method is only useful for imputing missing event failure indicators among participants who have positive screeners. If a participant develops first-onset TMD but still has a negative screener, such a participant will be treated as censored, and our method is unable to correct for this misclassification. The OPPERA screener was designed to have high sensitivity and modest specificity, so the number of false negative screens is expected to be low. (Indeed, OPPERA performed clinical examinations on a subset of the participants with negative screeners. Although analysis of this data is ongoing, preliminary results suggests that the false negative rate is less than 5%.) Thus, we expect that the small number of false negative screens will not meaningfully affect the results of our analysis. Also, note that under our simulation scenarios, we assumed that some failures were not observed due to a negative screener. Since our proposed method gave satisfactory results in these simulation scenarios, it appears that failing to observe some events due to negative screeners should not significantly bias the results.

In the OPPERA study, the hazard ratios associated with some variables were noticeably different after imputation. Although other results remained qualitatively unchanged, we note that even small changes in hazard ratios are important. In addition, estimated incidence rates were significantly increased after imputation. Since the results of OPPERA may become normative in the orofacial pain literature, precise calculation of the incidence rate of TMD and the hazard ratios associated with putative risk factors is important. Thus, imputation is recommended.