Abstract
Background
We sought to use data captured in the electronic health record (EHR) to develop and validate a prediction rule for virologic failure in patients being treated for HIV infection.
Methods
We used EHRs at two Boston tertiary care hospitals, Massachusetts General Hospital and Brigham and Women's Hospital, to identify HIV-infected patients who were virologically suppressed (HIV RNA ≤400 copies/mL) on antiretroviral therapy between 1/1/05 and 12/31/06. We used a multivariable logistic model with data from Massachusetts General Hospital to derive a one-year virologic failure prediction rule. The model was validated using data from the Brigham and Women's Hospital. We then simplified the scoring scheme to develop a clinical prediction rule.
Results
The one-year virologic failure prediction model, using data from 712 Massachusetts General Hospital patients, demonstrated good discrimination (c-statistic 0.78) and calibration (χ2 =6.6, p =0.58). The validation model, based on 362 Brigham and Women's Hospital patients, also showed good discrimination (c-statistic 0.79) and calibration (χ2 =1.9, p =0.93). The clinical prediction rule included seven predictors, Sub-optimal Adherence, CD4 count <100/μL, Drug and/or Alcohol Abuse, Heavily ART Experienced, Missed ≥1Appointment, Prior Virologic Failure, and Suppressed ≤12 months, and appropriately stratified patients in the validation dataset into low, medium and high risk groups, with one-year virologic failure rates of 3.0%, 13.0% and 28.6%.
Conclusions
A risk score based on seven variables available in the EHR predicts HIV virologic failure at one year and could be used for targeted interventions to improve outcomes in HIV disease.
Introduction
Advances in combination antiretroviral therapy (ART) in HIV disease have resulted in a decreased incidence of virologic failure and HIV-related morbidity and mortality[1, 2]. Despite this progress, virologic failure occurs in 10-20% of previously treatment-naïve subjects participating in antiretroviral clinical trials[3, 4]. In routine clinical practice, where monitoring of ART adherence, laboratory values and clinic follow-up and are generally less rigorous, failure rates are often higher[5-7]. This higher incidence of virologic failure, coupled with limited resources in clinical practice as compared to clinical trials, emphasizes the need for a simple prognostic tool to identify patients at increased risk of virologic failure. As is the case with diabetes, congestive heart failure, and other chronic diseases, a prediction rule to stratify HIV-positive patients' risk of future virologic failure could be used to target interventions and guide the allocation of resources[8-12].
Several prediction rules for progression to AIDS or death have been developed; however, most require complex formulas, or predict risk only at the time of ART initiation[5, 13-18]. The EuroSida research database, for example, has been used to develop a prognostic score for new AIDS diagnosis or death over the upcoming year, but this risk score is composed of 10 data elements and requires a web calculator to estimate risk[16]. Furthermore, these prediction rules assess CD4 count decline, HIV disease progression or death, rather than more proximal events, such as virologic failure, which if averted could prevent disease progression[17, 18].
We and others have demonstrated that data captured in the electronic health record (EHR) can be used to predict treatment outcomes in HIV clinic cohorts[7, 19, 20]. The objective of this study was to develop and validate a prediction rule that could be used to discriminate patient risk for virologic failure over the next year.
Methods
Study Design
We derived and validated a one-year virologic failure prediction model using retrospective data from the Massachusetts General Hospital (MGH) HIV clinic (derivation dataset) and the Brigham and Women's Hospital (BWH) HIV clinic (validation dataset). We included predictors identified in our prior analysis and considered psychosocial characteristics associated with poor HIV outcomes[21-24].
Derivation Dataset (MGH HIV clinic)
The MGH HIV clinic follows approximately 1,200 patients; patients are seen by fellows, residents and nurse practitioners who are supervised by attending physicians, or by attending physicians alone. The clinic uses an HIV-specific EHR, which has been shown to identify predictors of virologic failure[7].
Validation Dataset (BWH HIV clinic)
The BWH clinic provides HIV care to approximately 600 patients, who are followed by a separate staff of fellows, residents, nurse practitioners and attending physicians. Their HIV clinic uses the Longitudinal Medical Record, which was developed by Partners Health Care[25]. In contrast to the MGH, the BWH's EHR is not HIV-disease specific.
Inclusion Criteria
For both the derivation and validation datasets we included HIV-infected patients receiving ART with at least one arrived HIV clinic visit between January 1, 2005 and December 31, 2006 with evidence of virologic suppression on, or immediately preceding the arrived appointment by no more than 6 months, and 12 months of follow-up data. We defined viral suppression as an HIV RNA measurement ≤400 copies/mL, the highest lower limit of the assays used by either of the two hospitals (Roche PCR v1.5 assay at MGH (≤400 copies/mL for the standard and ≤50 copies/mL for the ultra-sensitive) and Bayer bDNA assay at BWH (≤75 copies/mL)). The study entry date was defined as the earliest date of a qualifying HIV clinic visit. Patients who did not require HIV therapy based on current treatment guidelines, patients without baseline EHR documentation, one-time HIV consults, or those with less than 12 months of follow-up data were excluded from the study. This study was approved by the Partners Human Research Committee.
Data Sources
Data for the derivation dataset included: age, gender, race, ethnicity, appointment history, CD4 count, HIV RNA, white blood cell count, and absolute neutrophil count, all of which were obtained electronically. Additional data were manually extracted from the EHR onto a data form by two extractors who were blinded to the study outcome. Manually extracted data included: current and past ART regimens, transfer of HIV care, active caretaker of a child or dependent adult, past or current drug and/or alcohol abuse, homelessness, past or current psychiatric diagnosis, and ART adherence at entry. To assess inter-extractor variability a 10% over-extraction was performed and compared. The analyses used 2.5 years of data, from January 1, 2005 to June 30, 2007.
The method of data extraction for the validation dataset was identical, except only those variables identified as significant in the final derivation model were extracted.
Adherence, ART History and Patient Characteristics
Adherence to ART was extracted from EHR clinic notes using the first reference to adherence in the following sequence: 1) study entry date, 2) the two most recent appointments prior to the entry date but within one year of entry date, or 3) the two appointments following the entry date within six months and prior to evidence of virologic failure. Sub-optimal adherence was extracted from the visit note and defined as <85% as percent or calculated based on number of missed doses ([prescribed – missed] / prescribed)[26]. In absence of a quantitative measure, adherence was based on descriptors. The following classification scheme was used: “fair”, “missed some doses”, “frequently misses doses”, and undocumented adherence status (“unknown”) were categorized as sub-optimal adherence, while “excellent”, “perfect”, “never misses”, “near-perfect”, “good”, “infrequently misses”, and “almost never misses,” were categorized as not sub-optimal adherence.
Prior antiretroviral history was extracted and initially categorized as: previously ART naïve, prior nucleoside reverse transcriptase inhibitor (NRTI) only, prior NRTI and non-nucleoside (NNRTI), prior NRTI and protease inhibitor (PI), or prior NRTI, NNRTI and PI. Current ART regimen was classified similarly with an additional category: current enfuvirtide or any investigational antiretroviral. Patient ART history was summarized as “Heavily ART Experienced,” defined as prior ART and either treatment with an NRTI, NNRTI, and PI prior to the current regimen, or current ART regimen containing enfuvirtide or an investigational antiretroviral.
Additional psychosocial characteristics were defined using the following criteria: alcohol and drug abuse was defined as inclusion in the EHR problem list, documentation of past or present use of illicit drugs, abuse of alcohol or prescription medications, or more than three drinks/day for women and four drinks/day for men[27]. Patients were considered to have a psychiatric condition if they were prescribed a psychiatric medication, were receiving psychiatric treatment, or had a psychiatric diagnosis in their EHR problem list. Caregiver was defined as a guardian or parent of a child <18 years of age or a dependent adult. Homelessness was defined as documentation of unstable housing or homelessness at study entry. Missed ≥1 Appointment was defined as one or more missed HIV clinic visits scheduled with a nurse practitioner or physician within the year prior to study entry[7].
Outcome Measurement
The primary study endpoint was virologic failure defined as either two consecutive HIV RNA measurements >400 copies/mL, or one HIV RNA measurement >400 copies/mL and no confirmatory test in the subsequent three months, where the first value >400 copies/mL occurred within 1 year of the entry date.
Statistical Analysis
Means and frequencies were used to summarize continuous and categorical data. Univariate and multivariable logistic regression analyses were used with the derivation data set to develop a 1-year virologic failure model. Variables identified as associated with virologic failure at p<0.10 in the univariate analyses were considered in the multivariable logistic regression model. The final model included all variables with p<0.05 in the multivariable model[28]. The c-statistic (area under the receiver operating characteristic (ROC) curve) was used to assess the model's discriminatory ability and the Hosmer-Lemeshow test was used to assess the model's calibration (goodness-of-fit)[28-30].
Validation of the final model consisted comparison of regression coefficients with a z-score to assess the need for recalibration, and comparison of the models' ability to stratify risk, discriminate (c-statistic), and accurately predict, calibration (Hosmer-Lemeshow)[31]. Z-scores were calculated using the equation:
z=(beta[derived]−beta[validation])/Sum(SEs),
where beta represents the predictors' regression coefficients from the derivation and validation models, and SE is the standard error for the regression coefficients.
Once the prediction model was validated, both weighted and simplified scoring schemes were elucidated. A simple risk score was calculated as the sum of predictors (one point per predictor)[32]. We further simplified the model by defining three risk categories from the derivation model to reflect low (<5%), medium, and high (>15%) one-year risk of virologic failure, and used these categories in a time-to-virologic failure analysis for patients in the derivation and validation data sets.
We also performed a Kaplan-Meier analysis, where patients were censored at the end of clinical follow-up or after 2.5 years. For this analysis we included all patients with less than a full year of follow-up. Differences between risk categories were compared using the Log-rank test. All statistical analyses were performed using SAS software (v9.1; SAS Institute Cary, NC).
Results
Study Cohorts
A total of 842 patients were identified with an arrived appointment at the MGH HIV clinic during the study period and an HIV RNA ≤400 copies/mL within the previous six months. Of these, 712 (85%) met the inclusion criteria for the derivation dataset. Eighty-two patients, 11.5% of the derivation cohort, experienced virologic failure during the first year of follow-up. Performing an identical search of the BWH database, 500 patients were identified; of these, 362 (72%) met inclusion criteria for the validation dataset. Thirty-eight patients, 10.5% of the validation cohort, experienced virologic failure in the first year.
The two cohorts were similar in terms of ART adherence, ART experience, CD4 counts, and missed appointments (Table 1). However, patients in the derivation cohort were more likely to be male, white, have past or current Drug and or Alcohol Abuse, Prior Virologic Failure, Psychiatric Condition, and Virologic Suppression <12 months, while patients in the validation cohort were more likely to be >40 years of age.
Table 1. Patient Characteristics.
| Patient Characteristics | Derivation (MGH) | Validation (BWH) | Differences | ||
|---|---|---|---|---|---|
| n | % of Total | N | % of Total | Chi Square (p) | |
| Total | 712 | 100.0 | 362 | 100.0 | - |
| Male | 558 | 78.4 | 222 | 61.3 | <0.0001 |
| Age >40 years | 476 | 66.9 | 273 | 75.4 | 0.004 |
| White, non-Hispanic | 416 | 58.4 | 108 | 29.8 | <0.0001 |
| Sub-optimal Adherence* | 185 | 26.0 | 81 | 22.4 | 0.20 |
| CD4 <100/μL | 29 | 4.1 | 13 | 3.6 | 0.70 |
| Drugs/Alcohol | 229 | 32.2 | 86 | 23.8 | 0.004 |
| Heavily ART experienced | 182 | 25.6 | 108 | 29.8 | 0.14 |
| Missed ≥1 Appointment | 277 | 38.9 | 122 | 33.7 | 0.095 |
| Prior Virologic Failure | 532 | 74.7 | 144 | 39.8 | <0.0001 |
| Psychiatric condition | 309 | 43.4 | 94 | 26.0 | <0.0001 |
| Suppressed <12 months | 306 | 43.0 | 112 | 30.9 | 0.0001 |
Definitions:
“Drugs/Alcohol” - current or past history of drugs or alcohol abuse.
“Missed ≥1 Appointments” in the previous year.
“Heavily ART Experienced”- prior nucleoside reverse transcriptase inhibitors, non-nucleoside reverse transcriptase inhibitors and protease inhibitors containing regimens prior to the current regimen, or a current enfuvirtide or investigational antiretroviral containing regimen.
For the derivation cohort, quantitative measures of adherence were used for 322 (45.2%) patients, qualitative descriptors for 245 (34.5%), and for 145 (20.4%) estimates of adherence were not available in the EHR.
Derivation Model (MGH Dataset)
In univariate analysis of the derivation dataset, seven patient characteristics were associated (p<0.10) with virologic failure over one year of follow-up: Sub-optimal adherence, CD4 count <100/μL, Drug and or Alcohol Abuse, Heavily ART Experienced, Missed ≥1 Appointment, Prior Virologic Failure, and Suppressed ≤12 months (Table 2). Variables identified in the univariate analysis were entered into a multivariable logistic regression model, and all seven maintained statistical significance (p<0.05, Table 3). The c-statistic for the derivation model was 0.78, indicating good discrimination, and the Hosmer-Lemeshow test showed excellent calibration (χ2 =6.6, p=0.58).
Table 2. Univariate Analysis (Derivation Dataset).
| Patient Characteristics | OR | 95% CI | p-value |
|---|---|---|---|
| Sub-optimal Adherence | 2.84 | 1.77, 4.55 | <0.0001 |
| CD4 count <100/μL | 4.47 | 2.00, 9.98 | 0.0003 |
| Drug and or Alcohol Abuse | 2.10 | 1.32, 3.35 | 0.002 |
| Heavily ART experienced | 2.59 | 1.62, 4.17 | <0.0001 |
| Missed ≥1 Appointment | 2.34 | 1.46, 3.72 | 0.0004 |
| Prior Virologic Failure | 10.3 | 3.21, 33.0 | <0.0001 |
| Suppressed <12 months | 3.49 | 2.12, 5.73 | <0.0001 |
| Age >40 years of age | 0.79 | 0.49, 1.28 | 0.34 |
| Absolute Neutrophil Count <1 | 0.68 | 0.09, 5.32 | 0.71 |
| Absolute Neutrophil Count 1 to 2 | 1.32 | 0.67, 2.60 | 0.43 |
| Absolute Neutrophil Count Unknown | 1.11 | 0.64, 1.94 | 0.71 |
| Caregiver | 0.56 | 0.26, 1.20 | 0.13 |
| HCV Positive | 1.53 | 0.86, 2.73 | 0.15 |
| Psychiatric condition | 1.08 | 0.68, 1.72 | 0.74 |
| Homeless | 1.80 | 0.50, 6.46 | 0.37 |
| Race/ethnicity: White | Ref | - | - |
| Race/ethnicity: Hispanic | 1.56 | 0.80, 3.05 | 0.19 |
| Race/ethnicity: Black | 1.05 | 0.58, 1.90 | 0.87 |
| Race/ethnicity: Other | 1.37 | 0.39, 4.81 | 0.63 |
| Race/ethnicity: Unknown | 2.41 | 0.85, 6.80 | 0.10 |
| Risk Factor: Heterosexual | Ref | - | - |
| Risk Factor: Blood | 0.59 | 0.07, 4.67 | 0.61 |
| Risk Factor: IDU | 1.60 | 0.77, 3.36 | 0.21 |
| Risk Factor: MSM | 1.40 | 0.78, 2.53 | 0.26 |
| Transferred HIV care | 0.99 | 0.57, 1.73 | 0.98 |
HIV Transmission Risk Factors: “MSM” – men who have sex with men, “IDU” – injection drug use. “Transferred HIV Care” within the last 12 months.
OR – odds ratio. CI – confidence intervals.
Table 3. Multivariable Analyses: Virologic Failure.
| Patient characteristics | Derivation Set (MGH, n = 712) | Validation Set (BWH, n = 362) | z-test* | ||||||
|---|---|---|---|---|---|---|---|---|---|
| VF Rate | OR | 95% CI | p-value | VF Rate | OR | 95% CI | p-value | p-value | |
| Sub-optimal Adherence | 20.5% | 2.77 | 1.67, 4.58 | <0.0001 | 23.5% | 3.77 | 1.80, 7.96 | 0.001 | 0.499 |
| CD4 count <100/μL | 34.5% | 2.98 | 1.20, 7.42 | 0.019 | 30.8% | 1.22 | 0.26, 5.62 | 0.803 | 0.325 |
| Drug and or Alcohol | 17.0% | 1.89 | 1.14, 3.12 | 0.013 | 17.4% | 1.78 | 0.80, 3.99 | 0.159 | 0.908 |
| Heavily ART experienced | 19.8% | 2.16 | 1.29, 3.60 | 0.003 | 16.7% | 2.00 | 0.92, 4.34 | 0.079 | 0.872 |
| Missed ≥1 Appointment | 17.0% | 1.72 | 1.04, 2.85 | 0.034 | 14.8% | 1.11 | 0.51, 2.45 | 0.788 | 0.361 |
| Prior Virologic Failure | 14.8% | 6.30 | 1.88, 21.07 | 0.003 | 18.8% | 2.61 | 1.14, 5.95 | 0.023 | 0.237 |
| Suppressed <12 months | 18.6% | 1.80 | 1.05, 3.10 | 0.034 | 19.6% | 2.22 | 1.04, 4.72 | 0.039 | 0.662 |
|
|
|
|
|||||||
| Hosmer-Lemeshow test | χ2 = 6.6, p = 0.58 | χ2 = 1.9, p = 0.93 | |||||||
| Discrimination | c-statistic = 0.78 | c-statistic = 0.79 | |||||||
z-test - compares the similarity of the two models' beta-coefficients for each predictor
Validation Model (BWH Dataset)
The selected predictors performed well for the validation dataset. The validation model showed good discrimination (c-statistic 0.79) and calibration (Hosmer-Lemeshow test χ2 =1.9, p =0.93; Table 3). For the validation dataset the model predicted an overall virologic failure rate of 7.4%, the actual virologic failure rate was 10.5% (Table 4).
Table 4. Derivation and Validation of Prediction Rule for Risk Score and Risk Categories.
| Risk Score | Derivation Set (MGH) | Validation Set (BWH) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Observed VF (Actual) | Model Predicted VF | Observed VF (Actual) | Model Predicted VF | |||||||||
| Total | Failed | Predicted | Total | Failed | Predicted | |||||||
| n | % | n | % | % | Std Dev | n | % | n | % | % | Std Dev | |
| 0 | 57 | 8.0 | 0 | 0 | 0.6 | 0 | 80 | 22.1 | 2 | 2.5 | 0.6 | 0 |
| 1 | 130 | 18.3 | 2 | 1.5 | 2.2 | 1.2 | 87 | 24.0 | 3 | 3.4 | 1.4 | 0.7 |
| 2 | 194 | 27.2 | 18 | 9.3 | 5.9 | 2.0 | 78 | 21.5 | 4 | 5.1 | 4.7 | 2.5 |
| 3 | 171 | 24.0 | 13 | 7.6 | 12.1 | 2.8 | 68 | 18.8 | 15 | 22.1 | 10.4 | 3.7 |
| 4 | 102 | 14.3 | 25 | 24.5 | 22.1 | 4.7 | 34 | 9.4 | 9 | 26.5 | 21.0 | 4.7 |
| 5 | 49 | 6.9 | 17 | 34.7 | 37.9 | 4.3 | 10 | 2.8 | 2 | 20.0 | 39.0 | 10.3 |
| 6 | 7 | 1.0 | 5 | 71.4 | 58 | 3.9 | 2 | 0.6 | 0 | 0.0 | 56.0 | 0 |
| 7 | 2 | 0.3 | 2 | 100 | 79.1 | 0 | 3 | 0.8 | 3 | 100 | 79.1 | 0 |
| Total Cohort | 712 | 100 | 82 | 11.5 | 11.5 | - | 362 | 100 | 38.0 | 10.5 | 7.4 | - |
| Model Discrimination | c-statistic | c-statistic | ||||||||||
|
|
|
|
||||||||||
| Full Model | 0.78 | 0.79 | ||||||||||
| Risk Score | 0.78 | 0.78 | ||||||||||
| Risk Category | 0.76 | 0.73 | ||||||||||
Risk Score and Risk Category
The simplified scoring schemes, risk score and risk category, also performed well in the derivation and validation datasets (Table 4). Risk scores were collapsed into low, medium and high risk categories to reflect <5%, 5-15% and >15% risk of virologic failure using the derivation model's predicted probabilities: low (0-1 predictors), medium (2-3 predictors) and high (≥4 predictors). In the validation dataset 167 patients (46.1% of the cohort) were identified as low risk, for which the model-predicted virologic failure rate was 1.0% and the observed rate 3.0%. One hundred and forty-six patients (40.3%) were identified as medium risk, for whom the predicted failure rate was 7.3% and the observed failure rate was 13.0%. Among the remaining 49 high risk (≥4 predictors) patients, 29.6% were predicted to have virologic failure and 28.6% were observed to fail (Figure 1). The c-statistic decreased only slightly using the simplified risk scoring: 0.78 for both datasets using the risk score, and 0.76 and 0.73 for the derivation and validation datasets using the risk categories (Figure 2).
Figure 1. One-Year VF Rate by Risk Categories.
One-year virologic failure rate and 95% confidence intervals (CI, shown by bars) by risk categories for the derivation and validation datasets.
Figure 2. Receiver-Operator Characteristic Curves.
Receiver-operator characteristic (ROC) curves for the derivation and validation datasets. The ability of the risk score and risk category to discriminate, c-statistic (area under the receiver operating characteristic curve), is shown to the right of each curve. For the derivation data set the positive predictive value for the high risk group was 30.6% and the negative predictive was 94.6%, these values for the validation dataset were 28.6% and 92.3%, respectively.
Survival Analysis
Using the simplified risk category, the one-year virologic failure prediction rule continued to stratify risk of virologic failure over 2.5 years of study data. Comparison of survival distributions for the three risk categories showed statistical significance for both the derivation and the validation datasets (both p<0.0001, Log-rank test; validation set shown in Figure 3). Comparing survival distributions for only those subjects who were still virologically suppressed at the end of one year, baseline risk categories continued to stratify patients' risk of virologic failure over the next 18 months (p<0.0001).
Figure 3. Time to Virologic Failure by Risk Group.
Time to virologic failure (HIV RNA >400 copies/mL) for the three risk categories using the validation dataset (Brigham and Women's Hospital). Number of patients included in the analyses at baseline and each 6 month interval are shown in the table. This includes patients with less than one full year of follow-up.
Discussion
We developed and validated a one-year virologic failure prediction rule using data from two demographically different HIV clinics. The prediction rule includes seven predictors, Sub-optimal Adherence, CD4 count <100/μL, Drug and or Alcohol Abuse, Heavily ART Experienced, Missed ≥1 Appointment, Prior Virologic Failure, and Suppressed ≤12 months, and it appropriately stratified patients risk of virologic failure over the next year. Using the simplified risk categories the prediction rule separated the validation cohort into three clinically relevant groups; low risk, with a 3.0% one-year virologic failure rate, medium risk, with a 13.0% one-year virologic failure rate, and high risk, with a 28.6% one-year virologic failure rate.
Despite the popularity of prediction rules in the literature, most are not widely used, in part due to their complexity[33]. This prediction, rule offers several advantages over other HIV-related prediction rules[13, 14, 16, 17]. First, it predicts the risk of virologic failure over one year, which typically precedes more serious HIV-related outcomes, thereby offering the potential for earlier interventions. In addition, the rule can be easily applied to heterogeneous populations at any time in the course of ART treatment, whereas other rules calculate risk from the initiation of ART, which is less applicable to clinical cohorts. The prediction relies on seven predictors, which are typically available in the electronic health record and do not require pre-ART records, such as nadir CD4, HIV RNA plateau and/or clinical stage as are used in other rules[14, 16, 17]. Further evidence for using current laboratory results comes from a study by May et al., which found that a CD4 count six months after starting ART was more predictive of long-term outcomes than nadir CD4 count[17]. Finally, the model supports simplified scoring schemes, both risk scores and risk categories.
The present study builds on prior analyses and shows that patient characteristics captured in the EHR can be used to predict patients' risk for virologic failure over the next year[5, 16, 20, 34-37]. In addition, we explored alternative models using different numbers and combinations of predictors (data not shown), however the current model was the most robust, with stable parameter estimates- arguing against over-fitting (data not shown). As with our earlier study, low CD4 count, short duration of virologic suppression, and a history of sub-optimal adherence, missed visits, and prior virologic failure, correlate with increased risk of virologic failure over the following year[7]. HIV transmission risk category was not significant in this analysis; however, it has been shown to be a predictor in other models of HIV disease progression[14]. This may in part be due to the inclusion of drug and alcohol abuse, missed appointments and ART adherence, which in this study were stronger risk factors for virologic failure.
In the validation process of this study, we found the model robust and able to support simplified scoring schemes. As expected, odds ratios for individual risk factors varied between the two models; however, these differences were small, and the regression coefficients for risk factors were similar (z-test, all p>0.2). Further, statistical significance for individual risk factors in the validation model is not essential, and lack of significance for some may be due in part to the small sample size. More important is the model's calibration and ability to discriminate patients by risk for virologic failure using the simplified scoring schemes; on both accounts the model performed well, with only a small decrease in discrimination.
The prediction rule offers several additional insights. We explored different ways to describe ART experience, but found that dichotomizing prior ART experience was efficient and had minimal impact on the model's discrimination (data not shown), although it may be necessary to modify this definition over time with the development of new drugs[38]. This analysis also suggests that the number of risks, rather than any specific risk factor, may be more important to identify patients at risk of virologic failure. The secondary analysis, time to virologic failure, shows that the risk categories are durable and continued to stratify risk over 2.5 year study follow-up.
There are several potential limitations to this study. First, not all predictors may be available in EHRs and as a result implementing the rule for a clinic cohort may require some manual extraction, whereas utilizing the rule for individual patients could be facilitated by additional history-taking. Second, while this and our prior study suggest that providers are able to identify patients with sub-optimal adherence, this must be examined in settings where clinicians have less HIV-experience[7]. Finally, these two clinics are relatively small, located in the same city, had similar virologic failure rates, therefore the prediction rule could be further strengthened by validation with larger and more geographically diverse cohorts. Recalibration of the model and risk categories may be necessary in some settings, especially if the one-year virologic failure rate is substantially different than that of the derivation and validation datasets[31, 39]. Future validations of the model with larger cohorts should also consider the impact of individual predictors on model discrimination[40].
In summary, we derived and validated a one-year HIV virologic failure prediction rule, which stratifies patient risk for virologic failure among those currently virologically suppressed on antiretrovirals. Clinical predictors of virologic failure include Sub-optimal Adherence, CD4 Count <100/μL, Drug and or Alcohol Abuse, Heavily ART Experienced, Missed ≥1 Appointment, Prior virologic failure, and Suppressed ≤12 months. The presence of ≤1, 2 to 3, or ≥4 of these predictors identified low, medium, and high risk groups. Ongoing studies are investigating the efficacy of intensive case management, adherence support calls, treatment buddies, and directly observed therapy. As these studies are completed our prediction rule will assist clinicians in directing focused interventions to the highest risk patients.
Acknowledgments
The authors thank the MGH Laboratory of Computer Science and the MGH and Brigham and Women's Hospital HIV Clinic patients and staff.
Support by: K01AI062435 (GKR), K24AI062476 (KAF), P30AI42851 (GKR, KAF) and K24DK080140 (JBM), and by the Massachusetts General Hospital Clinical Research Program.
Footnotes
Conflicts of interest: Dr. Robbins has received research support from Gilead Sciences, Schering-Plough, and consulting fees from Abbott Laboratories, Boehringer Ingelheim Pharmaceuticals and Tibotec. Dr. Sax reports receiving research support from GlaxoSmithKline, Pfizer and Merck Laboratories, and consulting fees from Abbott Laboratories, Bristol-Myers Squibb, Gilead Sciences, GlaxoSmithKline, Merck Laboratories and Tibotec. All other authors have no potential conflicts of interest to disclose.
References
- 1.Porter K, Babiker A, Bhaskaran K, et al. Determinants of survival following HIV-1 seroconversion after the introduction of HAART. Lancet. 2003;362:1267–74. doi: 10.1016/s0140-6736(03)14570-9. [DOI] [PubMed] [Google Scholar]
- 2.Palella FJ, Delaney K, Moorman A, et al. Declining morbidity and mortality among patients with advanced human immunodeficiency virus infection. HIV Outpatient Study Investigators. N Engl J Med. 1998;338:853–60. doi: 10.1056/NEJM199803263381301. [DOI] [PubMed] [Google Scholar]
- 3.Robbins GK, De Gruttola V, Shafer RW, et al. Comparison of sequential three-drug regimens as initial therapy for HIV-1 infection. N Engl J Med. 2003;339:2293–303. doi: 10.1056/NEJMoa030264. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Gulick RM, Ribaudo HJ, Shikuma CM, et al. Triple-nucleoside regimens versus efavirenz-containing regimens for the initial treatment of HIV-1 infection. N Engl J Med. 2004;350:1850–61. doi: 10.1056/NEJMoa031772. [DOI] [PubMed] [Google Scholar]
- 5.Lucas GM, Chaisson RE, Moore RD. Highly active antiretroviral therapy in a large urban clinic: risk factors for virologic failure and adverse drug reactions. Ann Intern Med. 1999;131:81–7. doi: 10.7326/0003-4819-131-2-199907200-00002. [DOI] [PubMed] [Google Scholar]
- 6.Mocroft A, Ruiz L, Reiss P, et al. Virological rebound after suppression on highly active antiretroviral therapy. AIDS. 2003;17:1741–51. doi: 10.1097/00002030-200308150-00003. [DOI] [PubMed] [Google Scholar]
- 7.Robbins GK, Daniels B, Zheng H, Chueh H, Meigs JB, Freedberg KA. Predictors of antiretroviral treatment failure in an urban HIV clinic. Journal of Acquired Immune Deficiency Syndromes: J Acquir Immune Defic Syndr. 2007;44:30–7. doi: 10.1097/01.qai.0000248351.10383.b7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97:1837–47. doi: 10.1161/01.cir.97.18.1837. [DOI] [PubMed] [Google Scholar]
- 9.Fortescue EB, Bates DW, Chertow GM. Predicting acute renal failure after coronary bypass surgery: cross-validation of two risk-stratification algorithms. Kidney Inter. 2000;57:2594–602. doi: 10.1046/j.1523-1755.2000.00119.x. [DOI] [PubMed] [Google Scholar]
- 10.Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285:2987–94. doi: 10.1001/jama.285.23.2987. [DOI] [PubMed] [Google Scholar]
- 11.Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA. 2003;290:2581–7. doi: 10.1001/jama.290.19.2581. [DOI] [PubMed] [Google Scholar]
- 12.Leung GM, Rainer TH, Lau FL, et al. A clinical prediction rule for diagnosing severe acute respiratory syndrome in the emergency department. Ann Intern Med. 2004;141:333–42. doi: 10.7326/0003-4819-141-5-200409070-00106. [DOI] [PubMed] [Google Scholar]
- 13.Mellors JW, Munoz A, Giorgi JV, et al. Plasma viral load and CD4+ lymphocytes as prognostic markers of HIV-1 infection. Ann Intern Med. 1997 Jun 15;126:946–54. doi: 10.7326/0003-4819-126-12-199706150-00003. [DOI] [PubMed] [Google Scholar]
- 14.Egger M, May M, Chene G, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet. 2002;13:119–29. doi: 10.1016/s0140-6736(02)09411-4. [DOI] [PubMed] [Google Scholar]
- 15.May M, Porter K, Sterne JA, Royston P, Egger M. Prognostic model for HIV-1 disease progression in patients starting antiretroviral therapy was validated using independent data. J Clin Epidemiol. 2005;58:1033–41. doi: 10.1016/j.jclinepi.2005.02.015. [DOI] [PubMed] [Google Scholar]
- 16.Mocroft A, Ledergerber B, Zilmer K, et al. Short-term clinical disease progression in HIV-1-positive patients taking combination antiretroviral therapy: the EuroSIDA risk-score. AIDS. 2007;21:1867–75. doi: 10.1097/QAD.0b013e328270b877. [DOI] [PubMed] [Google Scholar]
- 17.May M, Sterne JA, Sabin C, et al. Prognosis of HIV-1-infected patients up to 5 years after initiation of HAART: collaborative analysis of prospective studies. AIDS. 2007 May 31;21:1185–97. doi: 10.1097/QAD.0b013e328133f285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Johannessen A, Naman E, Ngowi BJ, et al. Predictors of mortality in HIV-infected patients starting antiretroviral therapy in a rural hospital in Tanzania. BMC Infect Dis. 2008;8:52. doi: 10.1186/1471-2334-8-52. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Sterling TR, Vlahov D, Astemborski J, Hoover DR, Margolick JB, Quinn TC. Initial plasma HIV-1 RNA levels and progression to AIDS in women and men. N Engl J Med. 2001;344:720–5. doi: 10.1056/NEJM200103083441003. [DOI] [PubMed] [Google Scholar]
- 20.Mugavero MJ, Lin HY, Allison JJ, et al. Racial Disparities in HIV Virologic Failure: Do Missed Visits Matter? J Acquir Immune Defic Syndr. 2009;50:100–8. doi: 10.1097/QAI.0b013e31818d5c37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Watts DH, Lambert J, Stiehm ER, et al. Progression of HIV disease among women following delivery. J Acquir Immune Defic Syndr. 2003;33:585–93. doi: 10.1097/00126334-200308150-00006. [DOI] [PubMed] [Google Scholar]
- 22.Conigliaro J, Gordon AJ, McGinnis KA, Rabeneck L, Justice AC. Veterans Aging Cohort 3-Site S. How harmful is hazardous alcohol use and abuse in HIV infection: do health care providers know who is at risk? J Acquir Immune Defic Syndr. 2003;33:521–5. doi: 10.1097/00126334-200308010-00014. [DOI] [PubMed] [Google Scholar]
- 23.Ironson G, O'Cleirigh C, Fletcher MA, et al. Psychosocial factors predict CD4 and viral load change in men and women with human immunodeficiency virus in the era of highly active antiretroviral treatment. Psychosom Med. 2005;67:1013–21. doi: 10.1097/01.psy.0000188569.58998.c8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Bardeguez AD, Lindsey JC, Shannon M, et al. Adherence to antiretrovirals among US women during and after pregnancy. Journal of Acquired Immune Deficiency Syndromes: J Acquir Immune Defic Syndr. 2008;48:408–17. doi: 10.1097/QAI.0b013e31817bbe80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Linder JA, Schnipper JL, Tsurikova R, Melnikas AJ, Volk LA, Middleton B. Barriers to electronic health record use during patient visits. AMIA Annu Symp Proc. 2006:499–503. [PMC free article] [PubMed] [Google Scholar]
- 26.Paterson DL, Swindells S, Mohr J, et al. Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Ann Intern Med. 2000;133:21–30. doi: 10.7326/0003-4819-133-1-200007040-00004. [DOI] [PubMed] [Google Scholar]
- 27.Dawson DA, Grant BF, Li TK. Quantifying the risks associated with exceeding recommended drinking limits. Alcohol Clin Exp Res. 2005;29:902–8. doi: 10.1097/01.alc.0000164544.45746.a7. [DOI] [PubMed] [Google Scholar]
- 28.Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 1997;277:488–94. [PubMed] [Google Scholar]
- 29.Xiong C, van Belle G, Miller JP, Morris JC. Measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups. Stat Med. 2006;25:1251–73. doi: 10.1002/sim.2433. [DOI] [PubMed] [Google Scholar]
- 30.Yang Y, Jin Z. Combining dependent tests to compare the diagnostic accuracies--a non-parametric approach. Stat Med. 2006;25:1239–50. doi: 10.1002/sim.2338. [DOI] [PubMed] [Google Scholar]
- 31.D'Agostino RB, Sr, Grundy S, Sullivan LM, Wilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180–7. doi: 10.1001/jama.286.2.180. [DOI] [PubMed] [Google Scholar]
- 32.Bandos AI, Rockette HE, Gur D. Incorporating utility-weights when comparing two diagnostic systems: a preliminary assessment. Acad Radiol. 2005;12:1293–300. doi: 10.1016/j.acra.2005.05.028. [DOI] [PubMed] [Google Scholar]
- 33.Liao L, Mark DB. Clinical prediction models: are we building better mousetraps? J Am Coll Card. 2003;42:851–3. doi: 10.1016/s0735-1097(03)00836-2. [DOI] [PubMed] [Google Scholar]
- 34.Howard AA, Arnsten JH, Lo Y, et al. A prospective study of adherence and viral load in a large multi-center cohort of HIV-infected women. AIDS. 2002;16:2175–82. doi: 10.1097/00002030-200211080-00010. [DOI] [PubMed] [Google Scholar]
- 35.Rastegar DA, Fingerhood MI, Jasinski DR. Highly active antiretroviral therapy outcomes in a primary care clinic. AIDS Care. 2003;15:231–7. doi: 10.1080/0954012031000068371. [DOI] [PubMed] [Google Scholar]
- 36.Giordano TP, Gifford AL, White AC, Jr, et al. Retention in care: a challenge to survival with HIV infection. Clin Infect Dis. 2007;44:1493–9. doi: 10.1086/516778. [DOI] [PubMed] [Google Scholar]
- 37.Hartzell JD, Spooner K, Howard R, Wegner S, Wortmann G. Race and mental health diagnosis are risk factors for highly active antiretroviral therapy failure in a military cohort despite equal access to care. J Acquir Immune Defic Syndr. 2007;44:411–6. doi: 10.1097/QAI.0b013e31802f83a6. [DOI] [PubMed] [Google Scholar]
- 38.Wilkin TJ, Taylor B, Olender S, Hammer SM. Advances in antiretroviral therapy. Top HIV Med. 2009;17:68–88. [PubMed] [Google Scholar]
- 39.Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epi. 2008;61:76–86. doi: 10.1016/j.jclinepi.2007.04.018. [DOI] [PubMed] [Google Scholar]
- 40.Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159:882–90. doi: 10.1093/aje/kwh101. [DOI] [PubMed] [Google Scholar]