Abstract
Missed HIV medical visits predict poor clinical outcomes. We sought to identify patients at high risk of missing visits. We analyzed 2002–2014 data from 6 large US HIV clinics. At each visit, we predicted the likelihood of missing the next scheduled visit using demographic, clinical, and patient-reported psychosocial variables. . Overall, 10,374 participants contributed 105,628 HIV visits. For 17% of visits, the next scheduled appointment was missed. The strongest predictor of a future missed visit was past-year missed visits. A model with only this predictor had area under the receiver operator curve=0.65; defining “high risk” as those with any past-year missed visits had 73% sensitivity and 51% specificity in correctly identifying a future missed visit. Inclusion of other clinical and psychosocial predictors only slightly improved performance. Past visit attendance can identify those at increased risk for future missed visits, allowing for proactive allocation of resources to those at greatest risk.
Keywords: HIV, retention in care, missed visits, appointment attendance, predictive models
Introduction
In recent years, considerable attention has focused on the HIV care continuum. Roughly half of persons diagnosed with HIV infection in the United States are not engaged in regular medical care [1]. Beyond the deleterious consequences to individual health, transmission from those diagnosed with HIV infection but not engaged in medical care accounts for an estimated 61% of new HIV cases [2]. Thus, improving HIV care engagement represents an enormous opportunity to maximize both the treatment and public health prevention benefits of antiretroviral therapy (ART).
A number of approaches to measuring engagement and retention in HIV care have been used [3], with no clear gold standard established [4]. A common quality indicator for care retention evaluates whether a patient has attended 2 or more HIV medical care visits over a 12-month period, with a minimum of 90 days between the attended visits.[3] Such quality indicators play a vital role from a health care policy perspective, but are not immediately available clinically in real time. In contrast, missed (“no show”) HIV medical care visits are captured in real time at the point of care and have consistently been associated with deleterious clinical outcomes, including mortality [5,6]. As such, missed clinic visits are immediately actionable by clinic staff. Proactive identification of patients at high risk of future no-shows and intervention to prevent drop-out holds promise as a cost-effective approach to address the critical continuum gap in HIV care engagement.
In this paper, we use demographic, clinical and behavioral data from a network of nationally distributed clinics in the US to develop a clinically relevant prediction model for identifying patients currently engaged in care who are at risk of missing their subsequent visit.
Methods
Data source
Data come from the Center for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS) observational clinical cohort. CNICS is a collaboration of eight large, geographically diverse, academically affiliated HIV primary care clinics [7]. Each clinic has established a database capturing information routinely collected by electronic health records and other institutional data systems. Nearly all patients consent to have their data captured. Data elements include demographic characteristics, HIV primary care visit attendance (including kept vs. no-show visits), clinical diagnoses, medications, and laboratory values. Data are de-identified and uploaded to a central CNICS repository on a quarterly basis. Data quality procedures have been previously described [7].
Most CNICS sites have also integrated brief patient-reported psychosocial surveys into routine clinical care. Patients complete psychosocial surveys at routine HIV primary care visits approximately every 6 months. Psychosocial domains assessed include depressive symptoms (Patient Health Questionnaire [PHQ]-9 [8]), anxiety/panic symptoms (PHQ-5 [9]), alcohol use (Alcohol Use Disorders Identification Test-Clinical [10,11]), substance use (Alcohol, Smoking, and Substance Involvement Screening Test [12,13]), and ART adherence (AIDS Clinical Trials Unit-4, Visual Analog Scale [14,15]).
Analysis sample
For this analysis, we included all CNICS patients with ≥2 attended HIV primary care appointments in 2002–2015 at 6 CNICS sites. Two sites were excluded that do not report missed visit data (n=1) or behavioral data (n=1). Primary analyses excluded the initial visit and were restricted to HIV primary care visits with a current psychosocial survey (“current” defined as that day or within the prior 6 months. This definition was selected to reflect both the typical schedule of psychosocial surveys at CNICS sites and a plausible frequency of such assessments in routine HIV clinical care. Secondary analyses that did not rely on psychosocial measures included all patients and visits.
Measures
Our primary aim was to develop a predictive model for whether a patient would attend vs. miss (no-show) a given scheduled HIV primary care appointment. Consistent with previous work in this area, scheduled appointments that were canceled by the patient or rescheduled by the clinic were excluded from consideration; appointments pro-actively canceled by the patient are assumed to reflect a different process (recognizing a conflict in advance and calling to cancel) than appointments for which the patient does not show without notification (understood to reflect either forgetting the appointment or behavioral constructs related to social norms and social responsibility) [3,4,16,17]. Attended vs. missed appointments were identified from institutional databases captured in CNICS.
Potential predictors of missed visits were defined a priori from three categories: demographic and contextual, clinical, and psychosocial characteristics. Demographic and contextual variables included site, age (time-updated), current gender, race/ethnicity (White non-Hispanic, Black non-Hispanic, Hispanic, or other), HIV risk group (male-to-male sex, injection drug use, heterosexual sex, or other), and health insurance type (private, public, or uninsured). Time-updated clinical characteristics defined for each kept appointment date included most recent CD4 count, most recent HIV RNA viral load, current ART status (taking vs. not taking ART), time in care at the CNICS site, and the number of missed visits and the missed visit proportion (MVP) in the 12 months prior to the appointment date. The MVP is calculated as the proportion of all scheduled HIV primary care visits in a period of time that were missed, after excluding canceled and rescheduled visits (i.e., missed visits divided by missed plus attended visits).[4] MVP was not calculated for visits in the first year. Time-updated psychosocial characteristics included PHQ-9 total depressive symptom severity score, PHQ-5 total anxiety/panic symptom score, AUDIT-C total alcohol use score, recent illicit drug use (any vs. none, not considering marijuana use), and ART adherence (any vs. no reported missed doses in the past month). Results were substantively unchanged in analyses considering two alternate definitions of ART adherence: no reported missed doses in the past two weeks, and no reported missed doses in the past week.
Analyses
The unit of analysis was each attended visit (thus, a patient could be represented in the analysis multiple times). For each attended visit, the outcome was defined as whether the subsequent scheduled visit was attended or a no-show. We began with bivariate comparisons of each potential predictor with the probability of no-show at the next visit. Henceforth, we refer to this probability as the “no-show risk” (a probability at the level of the scheduled appointment, not at the level of the individual). We evaluated the linearity assumption for each continuous or ordinal predictor, and used non-linear or categorical terms if indicated. We selected restricted cubic splines for age, CD4 count, and HIV RNA viral load (with an additional indicator term for suppressed [<75 copies/mL] vs. non-suppressed viral load) and categorical specifications for past-year MVP (0%, 1–25%, 26–50%, 51–99%, 100%), depressive severity (no [0–4], mild [5–9], moderate [10–14], moderately severe [15–19], or severe depressive symptoms [20–27]), anxiety/panic symptoms (PHQ-5 total score = 5 vs. <5), and AUDIT-C total alcohol use score (no drinking [0], non-hazardous drinking [1–3 for men and 1–2 for women], or hazardous drinking [≥4 for men and ≥3 for women]). We further combined ART status and adherence into a single 3-level categorical variable (not on ART, on ART and adherent, on ART and nonadherent).
We then fit a series of multivariable predictive logistic regression models, using the probability of no-show at the next visit as the outcome and including predictors in stages based on a priori considerations. We fit (1) a model with past-year number of missed visits as the sole predictor, (2) a model with past-year missed visits and demographic and clinical variables, and (3) a model with all predictors. Our primary analyses focused on the number rather than proportion of missed visits as a more readily calculable measure at the point of care. In secondary models, we restricted attention to those in CNICS care ≥1 year and fit models using the MVP rather than number of missed visits. We used a robust variance estimator to account for multiple visits per person.
To avoid overly optimistic performance estimates that can be induced by evaluating a predictive model in the same sample in which it was developed, we employed 10-fold cross-validation [18]. In this method, the predictive model is developed in 90% of the sample and its performance is evaluated in the remaining 10%; this process is repeated 10 times, each time excluding a different 10%, and the resulting estimates are averaged. Cross-validation is one of two methods of predictive model evaluation with the least bias and variance in simulations [18].
We compared the predictive ability of the different models of no-show risk by plotting receiver-operator characteristic (ROC) curves and comparing the area under the ROC curve (AUC). From each model, a predicted probability or risk score was calculated for each observation. To summarize each risk score’s predictive ability, we report sensitivity and specificity for one cutpoint. Since we expected that sensitivity would be more highly valued than specificity in applications of such a risk score but that moderate specificity would still be important, we selected the cutpoint that had the highest sensitivity while maintaining a specificity of ≥50%. We report bootstrapped 95% confidence intervals (CIs) based on 10,000 replicates. We further sought to categorize the risk score into three groups of low-risk, moderate-risk, and high-risk individuals so as to facilitate targeting interventions of different intensity. We also examined variation in risk score and risk stratification performance across sites.
Results
Description of sample
The primary analysis sample included 105,628 kept visits with current psychosocial data by 10,374 patients (Table 1). Patients contributed a median (interquartile range [IQR] of 7 (4–14) kept visits. The sample was mostly male, with half being White non-Hispanic, 29% being Black non-Hispanic, and 16% being Hispanic. The majority of patients had been in care ≥3 years, had CD4 counts ≥350, had viral loads <75 copies/mL, were on ART, and reported no missed doses in the past month. Approximately 31% of the sample reported moderate to severe depressive symptoms, 18% reported symptoms consistent with panic disorder, 22% reported hazardous alcohol consumption, and 16% reported current drug use. The secondary analysis sample, including all patients regardless of availability of psychosocial measures, included 543,787 kept visits by 21,928 patients and had similar demographic and clinical characteristics.
Table 1.
Characteristics of 10,374 patients in HIV primary care and of 105,628 kept appointments at 6 CNICS sites, and bivariate and multivariate association of each characteristic with no-show risk.
| Characteristic | Median (IQR) | Percent of sample (n=10,374 patients) |
No-show risk (n=105,628 appointments) |
||
|---|---|---|---|---|---|
| Odds ratio for no-show risk | |||||
| Bivariate | Multivariate | ||||
| Overall | 100% | 17% | |||
| Age | 46 (39–52) | ||||
| 18–24 | 2% | 29 % | 3.04 (2.64, 3.51) | 2.55 (2.21, 2.95) | |
| 25–34 | 14% | 24% | 2.53 (2.29, 2.80) | 2.03 (1.84, 2.24) | |
| 35–44 | 26% | 19% | 1.90 (1.73, 2.07) | 1.54 (1.42, 1.68) | |
| 45–54 | 39% | 15% | 1.45 (1.33, 1.58) | 1.27 (1.17, 1.37) | |
| 55+ | 19% | 11% | 1.00 (ref) | 1.00 (ref) | |
| Current gender | |||||
| Male | 84 % | 17 % | 1.00 (ref) | 1.00 (ref) | |
| Female | 16% | 19% | 1.20 (1.11, 1.30) | 0.88 (0.81, 0.96) | |
| Race/Ethnicity | |||||
| White non-Hispanic | 53 % | 15 % | 1.00 (ref) | 1.00 (ref) | |
| Black non-Hispanic | 29% | 21% | 1.59 (1.49, 1.70) | 1.39 (1.31, 1.48) | |
| Hispanic | 16% | 18% | 1.29 (1.18, 1.40) | 1.08 (1.00, 1.16) | |
| Other | 3% | 18% | 1.29 (1.10, 1.51) | 1.14 (1.00, 1.30) | |
| HI V infection risk group | |||||
| MSM (no IDU) | 64 % | 15 % | 1.00 (ref) | 1.00 (ref) | |
| IDU | 14% | 22% | 1.54 (1.43, 1.67) | 1.28 (1.18, 1.38) | |
| Heterosexual contact | 20% | 19% | 1.31 (1.22, 1.41) | 1.28 (1.18, 1.39) | |
| Other | 1% | 19% | 1.29 (1.05, 1.60) | 1.16 (0.96, 1.40) | |
| Health insurance type | |||||
| Private | 25 % | 14 % | 1.00 (ref) | 1.00 (ref) | |
| Public | 57% | 18% | 1.38 (1.28, 1.48) | 1.11 (1.04, 1.19) | |
| Uninsured | 17% | 20% | 1.59 (1.45, 1.74) | 1.23 (1.14, 1.33) | |
| Time in care at CNICS site (years) | 5 (2–9) | ||||
| <1 year | 16 % | 18 % | 1.22 (1.13, 1.33) | 0.96 (0.88, 1.04) | |
| 1–2 years | 22% | 18% | 1.23 (1.14, 1.33) | 0.88 (0.81, 0.95) | |
| 3–4 years | 15% | 17% | 1.16 (1.06, 1.27) | 0.93 (0.86, 1.01) | |
| 5–9 years | 26% | 17% | 1.16 (1.08, 1.25) | 0.95 (0.89, 1.02) | |
| 10+ years | 21% | 15% | 1.00 (ref) | 1.00 (ref) | |
| CD4 count (cells/mm3) | 506 (320–709) | 1.02 (0.94, 1.10) | |||
| <200 | 12 % | 21 % | 1.41 (1.31, 1.53) | ||
| 200–349 | 16% | 18% | 1.17 (1.10, 1.26) | 1.04 (0.98, 1.12) | |
| 350–499 | 20% | 18% | 1.14 (1.07, 1.20) | 1.05 (0.99, 1.11) | |
| 500+ | 51% | 16% | 1.00 (ref) | 1.00 (ref) | |
| HIV RNA viral load (copies/mL) |
<75(<75-<75) | 1.00 (ref) | |||
| <75 | 75 % | 15 % | 1.00 (ref) | ||
| 75–999 | 10% | 20% | 1.37 (1.29, 1.47) | 1.12 (1.05, 1.20) | |
| 1,000–9,999 | 5% | 23% | 1.69 (1.54, 1.84) | 1.24 (1.13, 1.36) | |
| 10,000–99,999 | 6% | 26% | 1.93 (1.78, 2.08) | 1.27 (1.16, 1.39) | |
| 100,000+ | 3% | 25% | 1.89 (1.70, 2.09) | 1.21 (1.07, 1.36) | |
| Antiretroviral status | 1.00 (ref) | ||||
| On ART, adherent | 55 % | 14 % | 1.00 (ref) | ||
| On ART, non-adherent | 31% | 21% | 1.61 (1.53, 1.69) | 1.27 (1.21, 1.33) | |
| Not on ART | 14% | 22% | 1.77 (1.66, 1.89) | 1.16 (1.08, 1.25) | |
| Number of missed visits, past year | 1.00 (ref) | ||||
| 0 | 1 (0–2) | 47 % | 10 % | 1.00 (ref) | |
| 1 | 21% | 18% | 1.94 (1.84, 2.04) | 1.75 (1.65, 1.85) | |
| 2 | 11% | 23% | 2.74 (2.57, 2.92) | 2.38 (2.22, 2.55) | |
| 3 | 6% | 26% | 3.25 (3.00, 3.51) | 2.70 (2.47, 2.95) | |
| 4+ | 14% | 31% | 4.10 (3.80, 4.41) | 3.42 (3.16, 3.70) | |
| Depressive symptoms | 6 (1–11) | 1.00 (ref) | |||
| None (0–4) | 45 % | 15 % | 1.00 (ref) | ||
| Mild (5–9) | 25% | 17% | 1.14 (1.08, 1.20) | 0.95 (0.89, 1.00) | |
| Moderate (10–14) | 14% | 20% | 1.36 (1.27, 1.45) | 1.03 (0.96, 1.10) | |
| Moderately severe (15–19) | 10% | 20% | 1.41 (1.30, 1.52) | 1.02 (0.94, 1.11) | |
| Severe(20–27) | 7% | 21% | 1.46 (1.32, 1.61) | 0.95 (0.85, 1.05) | |
| Anxiety symptoms | 0 (0–4) | 1.00 (ref) | |||
| None (0) | 66% | 16% | 1.00 (ref) | ||
| Some (1–4) | 16% | 18% | 1.16 (1.09, 1.24) | 1.05 (0.98, 1.12) | |
| Likely panic disorder (5) | 18% | 21% | 1.39 (1.31, 1.49) | 1.06 (0.98, 1.14) | |
| Alcohol use | 1(0–3) | 1.00 (ref) | |||
| None (0) | 39% | 16% | 1.00 (ref) | ||
| Non-hazardous (1–2/1–3)* | 38% | 17% | 1.02 (0.97, 1.08) | 1.04 (0.98, 1.09) | |
| Hazardous(≥3/≥4)* | 22% | 20% | 1.26 (1.18, 1.35) | 1.20 (1.13, 1.27) | |
| Current illicit drug use | 1.00 (ref) | ||||
| None | 84% | 15% | 1.00 (ref) | ||
| Any (non-marijuana) | 16% | 26% | 1.97 (1.85, 2.09) | 1.32 (1.24, 1.41) | |
Cutpoints for women and men, respectively MSM: Men who have sex with men. IDU: Injection drug use.
Bivariate associations with no-show risk
Of the attended appointments in the primary analysis sample, the overall no-show risk for the following scheduled appointment was 17% (Table 1). The most dramatic bivariate variation in no-show risk across sub-groups was for past-year missed visits. Patients who had missed no visits in the past year (MVP=0%) at the time of an attended visit had a 9% no-show risk for the next scheduled appointment, whereas patients who had missed 4 or more visits had a no-show risk for the next visit of 31%. Markedly higher no-show risk was also observed among younger patients, those with very high viral loads, and those reporting current drug use. Although less pronounced, the no-show risk was also higher for women, Black non-Hispanics, those with lower CD4 counts, those not on ART or reporting non-adherence, and those reporting higher depressive symptoms, likely panic disorder, and hazardous alcohol use. No show risk did not vary notably by time in CNICS care.
Predictive models
To predict no-show risk, a model that included past-year number of missed visits as the only predictor had an AUC (c-statistic) of 0.65 (95% CI 0.62–0.68) (Table 2, Model 1; Figure 1A). A cutpoint that classified those with ≥1 missed visit as “high risk” and those with 0 missed visits as “low risk” had 73% (69–77%) sensitivity and 51% (48–54%) specificity in correctly identifying those who would go on to miss their next scheduled appointment.
Table 2.
Comparison of predictive models of no-show risk among 10,374 patients in routine HIV primary care in the CNICS cohort.
| Model | Predictors | n | AUC1 | Sensitivity2 | Specificity3 |
|---|---|---|---|---|---|
| 1 | Past-year missed visits only | 105,622 | 0.65 (0.62, 0.68) |
73% (69%, 77%) |
51% (48%, 54%) |
| 2 | Past-year missed visits, demographic & clinical characteristics | 104,510 | 0.69 (0.67, 0.71) |
74% ( 70%, 78%) |
53% ( 44%, 62%) |
| 3 | Past-year missed visits, demographic, clinical, & psychosocial characteristics | 100,830 | 0.7 (0.67, 0.73) |
74% (71%, 77%) |
54% (44%, 64%) |
Reported estimates are means from 10-fold cross-validation.
Bootstrapped 95% confidence intervals for each estimate based on 200 replicates are presented in parentheses.
Area under the receiver-operator characteristic (ROC) curve. Also referred to as the c-statistic.
The probability that a patient who truly misses their next visit is classified as “high-risk” by the model.
The probability that a patient who truly attends their next visit is classified as “low-risk” by the model.
Figure 1.
Receiver-operator characteristic curves predicting no-show visits (A) using past-year missed visits only; (B) using all predictors.
In the same patients, a model that included a set of demographic and clinical predictors in addition to past-year missed visits had slightly improved AUC (0.69), sensitivity (74%), and specificity (53%; Table 2, Model 2). The further addition of a set of psychosocial predictors did not measurably change sensitivity but slightly improved AUC (0.70) and specificity (54%; Model 3; Figure 1B).
Models that employed past-year MVP rather than number of missed visits and that were stratified by time in CNICS care (≥1 year vs. <1 year) yielded substantively similar results. When using the secondary analysis sample of all patients regardless of psychosocial measure availability, the performance of the model with number of missed visits as the sole predictor was nearly identical to the primary analysis sample (AUC=0.65, sensitivity=76%, specificity=48%).
Risk stratification
Considering the predictive strength of past-year missed visits, the relatively minor additional predictive benefit of other demographic, clinical and psychosocial characteristics, and the likely advantages of a very simple risk stratification algorithm, we explored the utility of stratifying patients into low, moderate, and high-risk groups based only on past-year number of missed visits. Patients with no missed visits in the past year (“low risk”) comprised 47% of the sample overall (Figure 2A) and had only a 10% no-show risk at the subsequent appointment (Figure 2B). This group therefore accounted for only 27% of actual subsequent missed visits (Figure 2C). Patients with 1–2 missed visits in the past year (“moderate risk”) accounted for 32% of the sample, had a 20% no-show risk for the next appointment, and accounted for 37% of subsequent missed visits. Patients with >2 missed visits (“high risk”) accounted for only 20% of the sample but had a 30% no-show risk and therefore accounted for 36% of subsequent missed visits. Stratification by the missed visit proportion rather than number of missed visits was slightly more effective (Supplemental Figure 1), although we focus on number of missed visits due to ease of point-of-care calculation.
Figure 2.
Proportion of clinic population (A), no-show risk (B), and proportion of all no-show visits (C), by past-year missed visits.
Consistent with expectations, those classified in the “high risk” group based solely on past missed visits were on average of younger age, more likely to be uninsured, less likely to be on ART or to be adherent if on ART, less likely to be virally suppressed, more likely to have moderate to severe symptoms of depression and anxiety, and more likely to report current illicit drug use than those in the low or moderate risk groups. In a multivariable predictive model developed solely in the high risk group, these same factors generally remained independently predictive of future missed visits (with the exception of depression and anxiety symptoms), although the model only had moderate predictive ability within this subgroup (AUC=0.59; results not shown).
The 6 sites varied in no-show risk and the proportion of the clinic population that were classified as low, moderate, and high-risk (Supplemental Table 1). However, at each site the difference in risk between the low and high-risk groups was substantial, the high-risk group accounted for a disproportionate share of all missed visits, and AUCs were very similar (range: 0.63–0.66).
Discussion
In this large sample of patients engaged in routine HIV primary care from a geographically distributed network of HIV clinical sites, past HIV appointment attendance emerged as the most powerful predictor of future attendance. Patients with no missed visits in the past year had a one-in-ten chance of missing their next appointment, whereas patients who had missed more than two visits in the past year (but were in clinic on the index date of the analysis) had a nearly one-in-three chance of no-showing for their next appointment. This latter group, while comprising only one-fifth of the sample, accounted for over one-third of all subsequent missed visits.
These results have important implications for efforts to improve retention in care in real-world HIV practices. Many clinics recognize the importance of enhancing retention among their patients and employ a range of strategies from appointment reminders to intensive case management to help patients remain engaged in care [19,20]. Yet all clinics operate with constrained financial and staff resources that must be allocated strategically to address a wide range of patient care needs. This analysis suggests that among patients attending clinic on a given day, a relatively simple metric – the number of no-show appointments over the past year – can reasonably distinguish between those who have a low, moderate, or high risk of not showing up for their next scheduled appointment. Such risk stratification could greatly aid the efficient allocation of clinic resources, with the most intensive staff time targeted toward those at highest risk for not coming back. The finding that past visit attendance is a strong predictor of future visit attendance also suggests the possibility that short-term interventions to improve patients’ visit attendance, if successful in shifting patients’ overall propensity to attend visits, could have longer-term impacts.
We expected that measures of depression, anxiety, medication adherence, and alcohol and drug use measured through patient-reported surveys at CNICS sites would prove strong predictors of no-show risk. While several of these measures did indicate bivariable association with no-show risk, they offered little marginal benefit in predicting that risk over and above past-year number of missed visits. Similarly, demographic and clinical characteristics improved prediction only marginally. As such, characterizing visit attendance over the past year, rather than developing a more complex prediction rule with numerous inputs, represents a promising simple approach to risk stratification amenable for widespread implementation in HIV clinics. While electronic medical records make past visit attendance records increasingly accessible at point of care, clinics without ready access to this information could explore even simpler indicators such as whether the last visit was attended. However, once moderate- or high-risk patients are identified, proactive retention interventions might need to assess and address psychosocial or situational barriers on an individual basis.
Studies have demonstrated that missed clinic visits may tap into a different dimension of care retention than measures based on attended visits [20–22]. Missed visits have demonstrated significant prognostic value for mortality, in a dose-response manner, independent of whether an individual attends a minimum number of visits per year and is classified as “retained” on quality-of-care metrics.[6] Missed HIV clinic visits have been associated with delayed ART receipt, longer time to viral suppression, greater cumulative HIV burden, AIDS-defining illnesses and mortality [5,23]. In contrast to missed clinic visits, other measures of retention require an extended period of time to elapse (i.e., 12 months) before it can be known whether a patient has been retained, at which point there may be greater challenges locating patients, not to mention the months of missed opportunities to maintain antiretroviral treatment and proactively promote care retention. Missed clinic visits are immediately measured and actionable at the point of care, offering an opportunity for more expeditious intervention with individuals at increased risk of loss to care [3,20]. Our findings suggest there is an even earlier opportunity to intervene. Rather than reacting to a patient’s missed clinic visit, our data point toward proactive identification of those individuals currently in the clinic who are at greatest risk of missing their next visit, providing an opportunity to intervene in real time.
Strengths of this study include the large size, multiple sites, and demographic, clinical, and geographic diversity of the study population; the systematic capture of validated self-reported behavioral measures that are rarely routinely available in clinical cohorts; and the focus on no-show predictors potentially available in real time at the point of care. Despite geographic diversity, the analysis sample had a higher representation of White non-Hispanic and male individuals than the national HIV epidemic. Also, the no-show rate of 17% in this sample was lower than has been reported in several other recent studies [24–26]. possibly reflecting the inclusion criterion of having attended ≥2 clinic visits which may have selected for a more engaged population.
Our analysis focused on patients with ≥2 attended clinic visits to allow for measurement of past visit attendance. People living with HIV who are new to care experience unique challenges in establishing self-care and adherence behaviors while adjusting to a life-changing diagnosis [5,27]. Distinct prognostic models may be needed for these individuals to identify those at greatest risk of missing visits and dropping out of care. Similarly, those re-engaging after gaps in care may face unique challenges and require distinct risk prediction models. As with all risk scores, the performance of this approach should be externally validated, although confidence in the generalizability of our findings is strengthened by the comparable performance across the six sites in our sample. Although we did not formally divide our data into development and validation samples, the simplest model with number of missed visits did perform nearly identically in the primary sample of visits with current psychosocial measures (n=105,628) and the secondary sample of all visits (n=543,787). Finally, these results will only be applicable for clinics that keep track of missed visits; however, with the rapid movement toward electronic record-keeping systems, such information should continue to become increasingly available.
In summary, we found that the frequency of missed visits in the prior 12 months, whether as a proportion or count measure, was strongly predictive of the likelihood of missing the next clinic visit among people living with HIV in medical care. Because missed visits have been consistently associated with untoward HIV clinical outcomes and are a harbinger of becoming lost to care, the ability to identify persons at greatest risk to miss their next visit in real-time affords an opportunity for immediate, pro-active intervention, with considerable clinical and public health implications. Clinic-based strategies to risk-stratify patients based upon past visit attendance with targeted deployment of evidence-informed retention interventions should be evaluated in an effort to meaningfully address the largest gap on the HIV care continuum, an area of urgent need.
Supplementary Material
Key points.
A history of missed visits was a stronger predictor of future missed visits than demographic, clinical, or psychosocial characteristics. A simple classification scheme based on visit attendance history shows promise as an approach to proactively target patients at increased risk for failing to show for their next visit.
Acknowledgements:
BWP, SB, KC, HMC, EG, WCM, and MJM contributed to the acquisition of the data; BWP designed the analysis; BWP and MJM drafted the manuscript; BWP, AMB, SB, KC, SRC, HMC, EG, JK, WCM, and MJM assisted with the interpretation of the data and critically revised the manuscript for important intellectual content. All authors take responsibility for and approve the final version of the manuscript. We thank the National Institutes of Mental Health [grant number R01MH100970] and the National Institute of Allergy and Infectious Diseases [grant numbers R24AI067039 and P30 AI50410] for their support of this work.
Funding: This work was supported by the National Institutes of Mental Health [grant number R01MH100970 to BWP] and by the National Institute of Allergy and Infectious Disease [grant numbers R24AI067039, P30AI50410].
Footnotes
Conflicts of interest: BWP has received a speaking honorarium from MSD. KAC has been a scientific advisory board member for Roche and a community advisory board member for Gilead.
Previous presentation: A version of this work was presented at the 11th International Conference on HIV Treatment and Prevention Adherence, Miami FL, May 9–11, 2017.
References
- 1.Bradley H, Hall HI, Wolitski RJ, et al. Vital Signs: HIV diagnosis, care, and treatment among persons living with HIV--United States, 2011. MMWR Morb Mortal Wkly Rep. November 28 2014;63(47):1113–1117. [PMC free article] [PubMed] [Google Scholar]
- 2.Skarbinski J, Rosenberg E, Paz-Bailey G, et al. Human immunodeficiency virus transmission at each step of the care continuum in the United States. JAMA internal medicine. April 2015;175(4):588–596. [DOI] [PubMed] [Google Scholar]
- 3.Mugavero MJ, Davila JA, Nevin CR, Giordano TP. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS patient care and STDs. October 2010;24(10):607–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mugavero MJ, Westfall AO, Zinski A, et al. Measuring retention in HIV care: the elusive gold standard. J Acquir Immune Defic Syndr. December 15 2012;61(5):574–580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Mugavero MJ, Lin HY, Willig JH, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis. January 15 2009;48(2):248–256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Mugavero MJ, Westfall AO, Cole SR, et al. Beyond core indicators of retention in HIV care: missed clinic visits are independently associated with all-cause mortality. Clin Infect Dis. November 15 2014;59(10):1471–1479. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kitahata MM, Rodriguez B, Haubrich R, et al. Cohort profile: the Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol. October 2008;37(5):948–955. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. September 2001;16(9):606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. November 10 1999;282(18):1737–1744. [DOI] [PubMed] [Google Scholar]
- 10.Bradley KA, Bush KR, Epler AJ, et al. Two brief alcohol-screening tests From the Alcohol Use Disorders Identification Test (AUDIT): validation in a female Veterans Affairs patient population. Arch Intern Med. April 14 2003;163(7):821–829. [DOI] [PubMed] [Google Scholar]
- 11.Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med. September 14 1998;158(16):1789–1795. [DOI] [PubMed] [Google Scholar]
- 12.Newcombe DA, Humeniuk RE, Ali R. Validation of the World Health Organization Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): report of results from the Australian site. Drug Alcohol Rev. May 2005;24(3):217–226. [DOI] [PubMed] [Google Scholar]
- 13.WHO ASSIST Working Group. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction. September 2002;97(9):1183–1194. [DOI] [PubMed] [Google Scholar]
- 14.Chesney MA, Ickovics JR, Chambers DB, et al. Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (AACTG). AIDS Care. June 2000;12(3):255–266. [DOI] [PubMed] [Google Scholar]
- 15.Amico KR, Fisher WA, Cornman DH, et al. Visual analog scale of ART adherence: association with 3-day selfreport and adherence barriers. J Acquir Immune Defic Syndr. August 1 2006;42(4):455–459. [DOI] [PubMed] [Google Scholar]
- 16.Melnikow J, Kiefe C. Patient compliance and medical research: issues in methodology. J Gen Intern Med. February 1994;9(2):96–105. [DOI] [PubMed] [Google Scholar]
- 17.Martin S 98% of HBR Readers Love This Article: Businesses are just beginning to understand the power of “social norms”. Harvard Bus Rev. October 2012;90(10):23–25. [Google Scholar]
- 18.Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. August 2001;54(8):774–781. [DOI] [PubMed] [Google Scholar]
- 19.Gardner LI, Marks G, Craw JA, et al. A Low-Effort, Clinic-Wide Intervention Improves Attendance for HIV Primary Care. Clin. Infect. Dis 2012;55(8):1124–1134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gardner LI, Giordano TP, Marks G, et al. Enhanced personal contact with HIV patients improves retention in primary care: a randomized trial in 6 US HIV clinics. Clin. Infect. Dis 2014;59(5):725–734. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Mugavero MJ, Amico KR, Horn T, Thompson MA. The State of Engagement in HIV Care in the United States: From Cascade to Continuum to Control. Clin Infect Dis 2013;57(8):1164–1171. [DOI] [PubMed] [Google Scholar]
- 22.Thompson MA, Mugavero MJ, Amico KR, et al. Guidelines for Improving Entry Into and Retention in Care and Antiretroviral Adherence for Persons With HIV: Evidence-Based Recommendations From an International Association of Physicians in AIDS Care Panel. Ann Intern Med. 2012;156(11):817–294. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Giordano TP, Gifford AL, White AC, et al. Retention in Care: A Challenge to Survival with HIV Infection. Clin Infect Dis. 2007;44(11):1493–1499. [DOI] [PubMed] [Google Scholar]
- 24.Bofill L, Waldrop-Valverde D, Metsch L, Pereyra M, Kolber MA. Demographic and psychosocial factors associated with appointment attendance among HIV-positive outpatients. AIDS Care. October 2011;23(10):1219–1225. [DOI] [PubMed] [Google Scholar]
- 25.Nijhawan AE, Liang Y, Vysyaraju K, et al. Missed Initial Medical Visits: Predictors, Timing, and Implications for Retention in HIV Care. AIDS patient care and STDs. May 2017;31(5):213–221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Traeger L, O’Cleirigh C, Skeer MR, Mayer KH, Safren SA. Risk factors for missed HIV primary care visits among men who have sex with men. J Behav Med. October 2012;35(5):548–556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Fleishman JA, Yehia BR, Moore RD, Korthuis PT, Gebo KA. Establishment, Retention, and Loss to Follow-Up in Outpatient HIV Care. J Acquir Immune Defic Syndr. 2012;60(3):249–259. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.