Abstract
Objective:
To describe retention in HIV care based on various definitions of retention in the modern treatment era.
Design:
A cohort study of people enrolled in care at 7 mostly urban HIV clinics across the United States, 2010-2018.
Methods:
We estimated retention based on missed visits, kept visits, kept encounters (clinical visits, CD4 counts, and viral loads), and HIV labs. We contrasted risk factors for retention by different definitions and estimated odds ratios for of viral suppression and hazard ratios for mortality in 2 years immediately following the year in which retention was defined (the study year).
Results:
Across 108,171 person-years (N=21,481 people), in 71% of years people kept ≥75% of scheduled visits; in 78%, people kept ≥2 visits >90 days apart; in 74%, people had ≥2 HIV labs >90 days apart; and in 47%, people had no gaps >6 months in clinic visits. Missing >25% of scheduled visits despite attending ≥2 visits >90 days apart was associated with non-white non-Hispanic race/ethnicity, history of injection drug use, and prior AIDS diagnosis. In contrast, attending ≥75% of scheduled visits while not attending ≥2 visits >90 days apart was associated with male sex, white race, no injection drug use history, and no prior AIDS diagnosis. Subsequent viral non-suppression was more strongly associated with missed- than kept-visit measures of retention; 2-year mortality was only associated with failure to be retained by missed-visit measures.
Discussion:
Missed and kept-visit definitions of retention capture different constructs. Missed-visit measures are more strongly associated with poor HIV outcomes.
Keywords: Care continuum, HIV, Measurement error, Missed visits, Retention in care, Survival
Maintaining durable viral suppression in people living with HIV is critical to improving clinical outcomes and ending the HIV epidemic.[1, 2] One of the largest barriers to viral suppression is failing to be retained in HIV care.[3−5] People diagnosed with HIV but not in care account for 43% new HIV transmissions.[3] However, classifying patients as retained in care is challenging.
Retention in care has been defined[6, 7] based on missed visits (i.e. scheduled visits that were not attended without rescheduling)[8, 9] and kept visits (i.e., attended visits in a period of time). In a prior analysis of patients in 2008-2009, while all measures of retention were positively correlated, the correlation between visit adherence (a missed-visit measure) and attending ≥2 visit >90 days apart in a year (a kept-visit measure) was only 0.53.[7] Retention in a year by any definition was strongly associated with viral suppression ±120 days from the end of the year.[7]
Changes in expected visit frequency, particularly for people with stable viral suppression,[10] may mean that people who do not need frequent visits are misclassified as not retained based on kept-visit measures. Additionally, universal ART[11] could weaken the association between retention and viral suppression because even people who are not seen frequently may have access to medication.
Our goal in this analysis was to (1) estimate the prevalence of retention in HIV care based on various definitions; (2) describe differences in who is retained based on the definition of retention; and (3) estimate the association of different measures of retention with subsequent viral suppression and mortality in the modern treatment era.
METHODS
Study sample
The Center for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS) is a clinical cohort of patients enrolled in continuity HIV care (defined as attending ≥2 clinical visits within a 12-month interval) at any of 8 participating CFAR-affiliated HIV clinics in mostly urban settings across the United States (US) who consented to share their data.[12] Briefly, data abstracted from the electronic medical record include patient demographics, HIV acquisition risk factors, dates of scheduled clinical visits and whether patients kept those appointments, laboratory dates and values, and prescribed medications. CNICS patients are generally representative of people with HIV in the United States.[13]
Seven CNICS sites that submitted data on kept and missed clinic visits were included: University of Alabama at Birmingham; Case Western Reserve University; University of California San Diego; University of Washington; Fenway Health/Harvard University; University of North Carolina Chapel Hill; and Johns Hopkins University. For this analysis, we included patients with ≥1 viral load or CD4 count between 1 January 2009 and 31 December 2017 or ≥1 clinical visit between 1 January 2009 and 30 June 2018. At one site, we ended follow-up in 2016, based on cohort-level data availability.
The unit of analysis was a person-year – a calendar year in which we classified people as retained or not (the “study year”). We included person-years in our analyses based on encounters (kept HIV clinic visits, CD4 counts or HIV viral loads) in the “prior year” (study year-1). We considered outcomes (death and viral suppression) in the two years after the end of the study year (January 1, study year+1 to December 31, study year+2). People could contribute to analyses for ≥1 study year.
Study definitions
We defined a “clinic visit” as one that occurred in the HIV clinic with an HIV primary care provider to focus specifically on HIV-related medical care. “Scheduled” clinic visits excluded visits that were bumped or cancelled (since they could not be classified as kept or missed). Walk-in visits were classified as kept and counted in the number of scheduled clinic visits. An “encounter” was defined as a kept clinic visit, HIV viral load, or CD4 count.
The proportion of the cohort retained in a calendar year requires both a denominator (the people “expected” to be retained in care) and a numerator (the criteria used to define “retention”). For the denominator, we operationalized this expectation in four different ways. The first three, in order of increasing stringency (i.e., fewer people who meet criteria): (1) had ≥1 encounter (clinic visit OR HIV lab) in the prior year; (2) had ≥1 kept clinic visit in the prior year; and (3) had ≥1 kept clinic visit in the prior year AND ≥1 scheduled clinic visit in the study year. Requiring a scheduled clinic visit in the study year is perhaps the cleanest criteria for defining people as “expected” to be retained, but may not be operationalizable outside the clinic level (more on that in the Discussion). Finally, we considered (4) had ≥1 kept clinic visit in the study year (this denominator is used to estimate annual retention by the Health Resources and Services Administration (HRSA) although it relies on some circular logic where people “expected” to be seen were those who were seen). We restricted all denominators to people who survived to the end of the study year.
We considered eleven options for the numerator based on missed visits, kept visits, completed encounters, and HIV labs (CD4 cell counts and viral loads). For missed visits, we defined retention as (1) attending 100% of all scheduled visits during the calendar year, or (2) attending ≥75% of all scheduled visits. For kept visits, we defined retention as (3) attending ≥2 visits >90 days apart (the National Academy of Medicine (NAM) definition of retention);[14] (4) having no gaps in visits >6 months (>182 days); (5) having no gaps in visits >9 months (>273 days); or (6) having no gaps in visits >1 year (>365 days). For completed encounters, we defined retention as (7) having ≥2 encounters >90 days apart; (8) having ≥2 encounters, at least one of which was a clinic visit, >90 days apart (the HRSA definition of retention); (9) having no gaps in encounters >6 months; or (10) having no gaps in encounters >1 year. We also report the proportion retained based on (11) having ≥2 HIV labs >90 days apart (the CDC definition of retention). We provide an illustration of encounter patterns of several hypothetical people and their classification according to the denominator and numerator definitions above in Supplemental Figure 1.
Covariates
We considered the covariates as possible risk factors for non-retention and as possible confounders of the associations between retention and subsequent viral suppression and death. All time-varying (denoted by *) covariates were measured at the start of the study year. Covariates included: male sex at birth; age in years*(18-34, 35-49, ≥50); race/ethnicity (non-Hispanic white, non-Hispanic Black, Hispanic, and non-Hispanic other race); self-reported route(s) of HIV acquisition, (history of injection drug use prior to diagnosis (IDU), and someone with birth sex of male who has had sex with men (MSM; includes transgender women)); years since cohort enrollment*; prior AIDS-defining diagnosis*; ever initiated a combination ART regimen* (defined as ≥3 antiretroviral medications together); and prior viral suppression, if so and for how long*. Duration of prior viral suppression was defined, looking back from January 1 of the study year, as time (with no gaps >12 months) across which all of a patient’s viral load values were ≤400 copies/mL. (Although rare, some viral load tests during the study period used 400 as the detection limit.[15]) People with no viral load in the past month were classified as not suppressed. We adjusted all analyses for calendar year and CNICS site.
Statistical Analysis
We first described time between pairs of kept clinic visits for patients who returned for a follow-up visit within 18 months from the index visit (to exclude observations with no follow-up visit). We considered only visit pairs where the index visit was ≥18 months prior to the administrative censoring date to avoid biasing our estimates toward shorter visit intervals. To account for the fact that people with shorter intervals between attended clinic visits might contribute more visits to the analysis (and thus artificially shift the curves to the left), we weighted each observation by 1/ki, where ki was the number of visit pairs contributed by individual i = 1, …, N.
Next, we described the average yearly prevalence of retention according to different combinations of numerators and denominators described above. We omitted estimates for combinations where the numerator might be undefined (missed-visit measures for people with no scheduled clinic visits in the study year), where missing visit data might bias the numerator but not the denominator (retention based on kept visits when the denominator was ≥1 encounter in the prior year), and where there may have been insufficient follow-up time to meet the numerator definition (having a gap in visits or encounters of >9 or >12 months among people with ≥1 visit in the study year).
We calculated Spearman rank correlation statistics between different indicators of retention among person-years in which persons had ≥1 visit in the prior year and ≥1 scheduled visit in the study year (denominator definition #3; the only denominator for which all numerators were defined). We also calculated correlations that accounted for non-independence between observations within the same person;[16] all correlations were the same to within 0.03 (data not shown).
Next, we looked at risk factors for being unretained by one or both of a kept- (≥2 clinic visits >90 days apart; the NAM definition) or missed-visit definition of retention (≥75% visit attendance) using multinomial logistic regression. Odds ratios (ORs) from this model are conditional odds of being classified as unretained according to one or both measures versus the odds of being retained by both measures (the referent group), conditional on having one of the two outcome levels being compared.
Finally, we estimated the conditional odds of having a subsequent unsuppressed viral load or no viral load, relative to having a suppressed viral load, using multinomial logistic regression, and we estimated hazard ratios for the 2-year risk of death using Cox proportional hazards models. Deaths were ascertained through clinic sources and regular matches against the Social Security Death Index and National Death Index. The exposure was nominal categorical variable with levels: (0) retained by both definitions (reference group); (1) attended ≥75% of scheduled visits but did not attend ≥2 clinic visits >90 days apart (not retained by NAM definition); (2) attended ≥2 clinic visits >90 days apart but missed >25% of scheduled visits; and (3) not retained by either definition. We report both crude and covariate-adjusted estimates, where adjustment was undertaken through standardization with inverse probability of exposure weights.[17, 18]
Secondary analyses
Relative to being retained by both the NAM definition of retention (≥2 visits >90 days apart) and CDC definition of retention (≥2 HIV labs >90 days apart), we looked at risk factors for, and outcomes (viral non-suppression and death) subsequent to, 1) being retained according to the NAM definition only; 2) being retained by the CDC definition only; and 3) being retained by neither definition.
RESULTS
Study sample
From 2010-2018, 21,481 people contributed 108,171 person-years (study years) to this analysis. The median (interquartile range [IQR]) number of person-years contributed per person was 5 (2, 8). The sample was majority male (81%), with median age 44 years (IQR: 35, 51) at their first study year. Forty-one percent were non-Hispanic Black and 43% were non-Hispanic white. Sixty-one percent of people had MSM and 12% had IDU as an HIV acquisition risk factor. Across all person-years, median years in the cohort at the start of the study year was 6.0 (IQR: 2.6, 10.9) (Table 1).
Table 1.
Characteristics (number and percent, unless otherwise specified) of persons in the Center for AIDS Research Network of Integrated Clinical Studies (CNICS) cohort who had ≥1 viral load test, CD4 cell count, or clinical visit, 2010-2018, and person-years they were included in the denominator for estimating the proportion of the cohort that was retained in care
| Personsa | Person-years | |
|---|---|---|
| N | 21,481 | 108,171 |
|
| ||
| Male sex at birth | 17,385 (81) | 86,757 (80) |
| Age, yearsb | 44 (35, 51) | 48 (39, 54) |
| Race/Ethnicity | ||
| Black, non-Hispanic | 8,890 (41) | 44,779 (41) |
| White, non-Hispanic | 9,132 (43) | 46,749 (43) |
| Hispanic | 2,542 (12) | 12,607 (12) |
| Other, non-Hispanic | 917 (4) | 4,036 (4) |
| HIV acquisition risk | ||
| IDU | 2,567 (12) | 13,371 (12) |
| MSM | 13,207 (61) | 65,983 (61) |
| Calendar yearb | 2011 (2010, 2014) | 2014 (2012, 2016) |
| Prior AIDS diagnosis | 4,693 (22) | 28,926 (27) |
| ART-initiated | 18,578 (86) | 101,961 (94) |
| Years since cohort enrollmentb | 0.9 (0.5, 6.0) | 6.0 (2.6, 10.9) |
| Viral suppression | 15,788 (73) | 87,558 (81) |
| Years since most recent viral suppressionb | 0.7 (0.3, 2.2) | 2.3 (0.9, 5) |
Abbreviations: ART, antiretroviral therapy; CNICS, Center for AIDS Research Network of Integrated Clinical Studies; IDU, injection drug use; MSM, men who have sex with men
Time-varying covariates measured at the first person-year/observation per person
Median (interquartile range)
Of adjacent kept clinic visits that were ≤18 months apart: 86% of patients returned for a visit within 6 months, 94% within 9 months, and 97% within 12 months (Supplemental Figure 2). Median number of kept visits per person per year (among people who attended ≥1 visit) was 3 (IQR: 2, 4).
Estimates of retention
Among people with a visit in the prior year and ≥1 scheduled visit in the study year, 58% kept 100% of scheduled visits, 71% kept ≥75% of scheduled visits, 78% kept ≥2 visits >90 days apart (the NAM definition of retention), 82% had ≥2 encounters (at least one of which was a clinic visit) >90 days apart (the HRSA definition of retention), 74% had ≥2 HIV labs >90 days apart (the CDC definition of retention), and 47% had no gaps >6 months in clinic visits. Finally, the denominator mattered for measuring retention: percent retained by the NAM definition was 76% among people with ≥1 clinic visit in the prior year, 82% among people with ≥1 visit in the prior year and ≥1 scheduled visit in the study year, and 85% among people with ≥1 visit in the study year (Table 2). The proportion of people retained was fairly stable across calendar years included in this study (Supplemental Table 1).
Table 2.
Number (%) of person-years classified as retained, according to different definitions of retention, Centers for AIDS Research Network of Integrated Clinical Systems, 2010-2018
| Denominator |
||||
|---|---|---|---|---|
| Definition of retention (Numerator) | ≥1 clinic visit in prior yr, N=101,093 | ≥1 visit or lab in prior yr, N=105,575 | ≥1 visit in prior yr + ≥1 scheduled in study yr, N=93,899 | ≥1 visit in the study yr, N=96,260 |
| Missed-visit measures | ||||
|
| ||||
| 100% kept visits/scheduled | a | a | 54,124 (58) | 56,147 (58) |
| ≥75% kept visits/scheduled | a | a | 66,784 (71) | 69,651 (72) |
|
| ||||
| Kept-visit measures | ||||
|
| ||||
| ≥2 kept visit >90 days apartd | 73,134 (72) | b | 73,134 (78) | 76,704 (80) |
| No gaps >6 months, visits | 43,864 (43) | b | 43,864 (47) | 44,913 (47) |
| No gaps >9 months, visits | 74,308 (74) | b | 74,308 (79) | c |
| No gaps >1 year, visits | 85,989 (85) | b | 85,989 (92) | c |
|
| ||||
| Kept-encountere measures | ||||
|
| ||||
| ≥2 encounters >90 days apart | 77,878 (77) | 80,369 (76) | 77,565 (83) | 81,466 (85) |
| ≥2 encounters incl visit, >90 days apartf | 77,276 (76) | 79,105 (75) | 77,276 (82) | 81,466 (85) |
| No gaps >6 months, encounters | 51,345 (51) | 52,574 (50) | 51,204 (55) | 52,540 (55) |
| No gaps >1 year, encounters | 88,962 (88) | 92,044 (87) | 88,405 (94) | c |
|
| ||||
| Lab-based measure | ||||
|
| ||||
| ≥2 HIV labs >90 days apartg | 69,356 (69) | 71,605 (68) | 69,043 (74) | 72,348 (75) |
Numerator might be undefined (for people with no scheduled visits in the study year, there is no possibility of missing a visit)
Concern about the potential for missing visit data (for people who meet the denominator definition because they have labs only in the prior year)
Insufficient follow-up time to meet the numerator definition
National Academy of Medicine definition of retention
Encounters defined as kept HIV clinic visits, viral load tests, or CD4 cell counts
Health Resources and Services Administration definition of retention
CDC definition of retention
The two missed-visit measures were correlated with each other (Spearman correlation statistic=0.755), kept-visit measures were correlated with each other (0.416 to 0.748), kept-encounter measures were correlated with each other (0.411 to 0.970), and kept-visit measures were correlated with kept-encounter measures (0.353 to 0.897), but there was poor correlation between the missed-visit and the kept-visit or kept-encounter measures (−0.072 to 0.211). Additionally, lab-based measures were correlated with kept-visit and kept-encounter measures (0.453 to 0.783) but not with missed-visit measures (0.033 to 0.162). The correlation between the CDC lab-based measure and the NAM visit-based measure was 0.703 (Table 3).
Table 3.
Spearman rank correlation statistics between various measures of retention, among persons with ≥1 visit kept in prior year and ≥1 scheduled in study year, Centers for AIDS Research Network of Integrated Clinical Systems patients, 2010-2018
| Missed-visit measures |
Kept-visit measures |
Kept-encounter measuresa |
Lab-based measure |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% kept visits/scheduled | ≥75% kept visits/scheduled | ≥2 kept visit >90 days apartb | No gaps >6 months, visits | No gaps >9 months, visits | No gaps >1 year, visits | ≥2 visits or labs >90 days apart | ≥2 encounters, ≥1 visit >90 days apartc | No gaps >6 months, encounters | No gaps >1 year, encounters | ≥2 HIV labs >90 days apartd | |
| Missed-visit measures | |||||||||||
|
| |||||||||||
| 100% kept visits/scheduled | 1. | ||||||||||
| ≥75% kept visits/scheduled | 0.755 | 1. | |||||||||
|
| |||||||||||
| Kept-visit measures | |||||||||||
|
| |||||||||||
| ≥2 kept visit >90 days apartb | 0.018 | 0.183 | 1. | ||||||||
| No gaps >6 months, visits | 0.051 | 0.211 | 0.534 | 1. | |||||||
| No gaps >9 months, visits | 0.017 | 0.143 | 0.692 | 0.555 | 1. | ||||||
| No gaps >1 year, visits | −0.060 | 0.032 | 0.594 | 0.416 | 0.748 | 1. | |||||
|
| |||||||||||
| Kept-encounter measuresa | |||||||||||
|
| |||||||||||
| ≥2 encounters >90 days apart | 0.003 | 0.146 | 0.870 | 0.465 | 0.613 | 0.553 | 1. | ||||
| ≥2 encounters, ≥1 visit >90 days apartc | −0.009 | 0.138 | 0.897 | 0.480 | 0.643 | 0.598 | 0.970 | 1. | |||
| No gaps >6 months, encounters | 0.048 | 0.195 | 0.531 | 0.856 | 0.541 | 0.424 | 0.541 | 0.536 | 1. | ||
| No gaps >1 year, encounters | −0.072 | 0.007 | 0.526 | 0.353 | 0.635 | 0.849 | 0.587 | 0.570 | 0.411 | 1. | |
|
| |||||||||||
| Lab-based measures | |||||||||||
|
| |||||||||||
| ≥2 HIV labs >90 days apartd | 0.033 | 0.162 | 0.703 | 0.431 | 0.541 | 0.453 | 0.814 | 0.783 | 0.508 | 0.488 | 1. |
Encounters defined as kept HIV clinic visits, viral load tests, or CD4 cell counts
National Academy of Medicine definition of retention
Health Resources and Services Administration definition of retention
CDC definition of retention
Risk factors for not being retained
Among people with ≥1 clinic visit in the prior year and ≥1 scheduled visit in the study year, 61% had both ≥75% visit attendance and ≥2 kept visits >90 days apart; 10% attended ≥75% of scheduled visits but did not have ≥2 visits >90 days apart; 17% attended ≥2 visits >90 days apart but missed >25% of scheduled visits; and 12% were not retained by either measure. Not being retained was inversely associated with viral suppression just prior to the study year (adjusted conditional odds ratios (aOR) of 0.60-0.79), more time since achieving viral suppression (aOR=0.87-0.97 per year of suppression), older age (aOR=0.53 for people ≥50 years relative to 25-49 years; aOR=1.62 for people 18-34 years), and fewer years since cohort enrollment (aOR=1.01-1.02 per year). Relative to being retained by both measures, risk factors for not attending ≥2 visits >90 days apart (while not missing >25% of scheduled visits) included: male sex at birth (aOR=1.07); white, non-Hispanic race/ethnicity (reference group; aOR=0.73 for Black people, aOR=0.69 for Hispanic people); non-IDU HIV acquisition risk (aOR=0.79 for people with a history of IDU); and no prior AIDS diagnosis (aOR=0.77 for prior AIDS diagnosis). Conversely, risk factors for missing >25% of scheduled visit (while attending ≥2 visits >90 days apart) included: non-white race/ethnicity (aOR=1.95 for Black people, aOR=1.29 for Hispanic people); a history of IDU (aOR=1.95); non-MSM HIV acquisition risk (aOR=0.88 for MSM); and prior AIDS diagnosis (aOR=1.04; Table 4).
Table 4.
Conditional odds ratios (95% confidence intervals) for lack of retention in a calendar year according to missed (≥75% visit attendance) versus kept-visit (≥2 kept visit >90 days apart) retention definitions, among persons with ≥1 visit kept in prior year and ≥1 scheduled in study year (N=93,899), Centers for AIDS Research Network of Integrated Clinical Systems patients, 2010-2018
| Attended ≥75% of scheduled visits, not retained by kept-visit measurea | Retained by kept-visit measure, missed >25% of scheduled visitsa | Retained by neither definitiona | |
|---|---|---|---|
| Male sex at birth | 1.07 (0.97, 1.18) | 0.98 (0.91, 1.06) | 1.10 (1.01, 1.20) |
| Age | |||
| 18-34 years | 1.20 (1.12, 1.30) | 1.41 (1.31, 1.50) | 1.62 (1.51, 1.73) |
| 35-49 years | Ref | Ref | Ref |
| ≥50 years | 0.75 (0.71, 0.80) | 0.65 (0.62, 0.69) | 0.53 (0.50, 0.56) |
| Race/Ethnicity | |||
| White, non-Hispanic | Ref | Ref | Ref |
| Black, non-Hispanic | 0.73 (0.68, 0.78) | 1.95 (1.83, 2.09) | 1.53 (1.42, 1.64) |
| Hispanic | 0.69 (0.63, 0.76) | 1.29 (1.18, 1.41) | 1.05 (0.96, 1.16) |
| Other, non-Hispanic | 0.99 (0.86, 1.14) | 1.21 (1.05, 1.41) | 0.99 (0.85, 1.16) |
| HIV acquisition risk | |||
| IDU | 0.79 (0.71, 0.87) | 1.95 (1.81, 2.11) | 1.55 (1.42, 1.68) |
| MSM | 0.99 (0.92, 1.08) | 0.88 (0.82, 0.95) | 0.85 (0.78, 0.91) |
| Prior AIDS diagnosis | 0.77 (0.72, 0.83) | 1.04 (0.98, 1.11) | 0.97 (0.90, 1.03) |
| ART-initiated | 0.90 (0.80, 1.02) | 1.05 (0.95, 1.17) | 1.04 (0.93, 1.17) |
| Years since cohort enrollment | 1.02 (1.01, 1.03) | 1.01 (1.00, 1.01) | 1.01 (1.00, 1.02) |
| Viral suppression | 0.79 (0.73, 0.85) | 0.67 (0.63, 0.71) | 0.60 (0.56, 0.64) |
| Years since most recent viral suppression | 0.97 (0.96, 0.98) | 0.92 (0.91, 0.92) | 0.87 (0.86, 0.88) |
Abbreviations: ART, antiretroviral therapy; CNICS, Center for AIDS Research Network of Integrated Clinical Studies; IDU, injection drug use; MSM, men who have sex with men
Relative to being retained by both definitions (attending ≥75% of scheduled visits and attending ≥2 visits >90 days apart); conditional odds ratios are odds of having retention status in column header versus retained by both definitions, among person-years in one of those two groups; conditional odds ratios are adjusted for all other variables (rows) in the table, calendar year and CNICS site
Outcomes of retention
On average, 12% of subsequent, measured viral loads were unsuppressed. The 2-year odds of having unsuppressed viral load was higher among people defined as not-retained by both retention measures, in comparison to defined as retained by both retention measures (aOR=3.78, 95% confidence interval [CI]: 3.46, 4.13), followed by people who missed >25% of scheduled visits but attended ≥2 visits >90 days apart (aOR: 2.19, 95% CI: 2.02, 2.38), and then people who kept ≥75% of scheduled visits but failed to attend ≥2 visits >90 days apart (aOR: 1.54, 95% CI: 1.37, 1.72). Failure to be retained by either measure was similarly associated with an increased odds of having no follow-up viral load in the 2 years after the study year (aOR=1.49, 95% CI: 1.40, 1.58). However, having no follow-up viral load was more strongly associated with good visit attendance but not attending ≥2 visits >90 days apart (aOR=1.34, 95% CI: 1.27, 1.41) than with attending ≥2 visits >90 days apart but missing >25% of scheduled visits (aOR=1.17, 95% CI: 1.11, 1.22). Finally, average 2-year mortality after a given study year was 2.6%. Mortality was elevated only among people who missed ≥75% of scheduled visits (adjusted hazard ratio (aHR): 1.63, 95% CI: 1.44, 1.84) and was not associated with failure to attend ≥2 visits >90 days apart (Table 5).
Table 5.
Conditional odds ratiosa of viral non-suppression or lack of viral load measurement at the first subsequent viral load within 2 years (versus viral suppression) and hazard ratios for 2-year mortality according to whether, at the end of the baseline calendar year, people were retained by one or neither (versus both) missed- (attended ≥75% of scheduled visits) versus kept-visit (≥2 kept visit >90 days apart) retention definitions, among persons with ≥1 visit in prior year and ≥1 scheduled in study year, CNICS patients, 2010-2018
| Retained by both definitions | Attended ≥75% of scheduled visits, not retained by kept-visit measure | Retained by kept-visit measure, missed >25% of scheduled visits | Retained by neither definition | |
|---|---|---|---|---|
| Crude | ||||
|
| ||||
| Viral load >400 copies/mLa | 1. | 1.85 (1.66, 2.06) | 3.59 (3.33, 3.88) | 7.05 (6.51, 7.63) |
| No measured viral loada | 1. | 1.46 (1.39, 1.53) | 1.43 (1.37, 1.49) | 1.84 (1.75, 1.93) |
|
| ||||
| 2-year hazard of death | 1. | 0.74 (0.62, 0.89) | 1.73 (1.54, 1.93) | 1.12 (0.97, 1.29) |
|
| ||||
| Adjusted b | ||||
|
| ||||
| Viral load >400 copies/mLa | 1. | 1.54 (1.37, 1.72) | 2.19 (2.02, 2.38) | 3.78 (3.46, 4.13) |
| No measured viral loada | 1. | 1.34 (1.27, 1.41) | 1.17 (1.11, 1.22) | 1.49 (1.40, 1.58) |
|
| ||||
| 2-year hazard of death | 1. | 0.85 (0.69, 1.04) | 1.63 (1.44, 1.84) | 1.09 (0.91, 1.29) |
Relative to having a suppressed viral load; conditional odds ratios are odds of having an unsuppressed viral load versus a suppressed viral load or having no viral load versus a suppressed viral load, conditional on having an outcome in one of those two groups
Adjusted for sex, age, race/ethnicity, route of HIV acquisition (men who have sex with men and injection drug use), years since cohort enrollment, prior AIDS-defining diagnosis, prior initiation of ART, duration of ART, prior viral suppression, time spent with viral suppression, calendar year, and CNICS site
Secondary analyses
In secondary analyses, younger age was associated with being retained according to the CDC lab-based measure but not the NAM visit-based measure (aOR=1.56, 95% CI: 1.41, 1.72 for 18-34-year-olds relative to 35-49-year-olds; aOR=0.75, 95% CI: 0.69, 0.82 for persons ≥50 years); other variables were similarly associated with both measures of retention (Supplemental Table 2).
Finally, relative to being retained by both CDC and NAM definitions, failure to be retained by either or neither was associated with a higher odds of subsequent viral non-suppression and higher odds of not having a measured viral load. Odds of no subsequent viral load was higher following a year in which persons were retained by the NAM definition but not the CDC definition (aOR=1.20, 95% CI: 1.13, 1.28) than following a year in which persons were retained by the CDC definition but not the NAM definition (aOR=1.06, 95% CI: 0.99, 1.14). The 2-year hazard of death following years in which people were retained by neither definition was 0.81 times (95% CI: 0.72, 0.91) the hazard of death following years in which people were retained by both definitions. The hazard of death was slightly higher following years in which people were retained by one but not the other definitions although associations were not statistically significant (Supplemental Table 3).
DISCUSSION
In this large, representative sample of people with HIV in routine care the estimated proportion retained was highly variable depending on how retention was defined. Missed- and kept-visit measures of retention captured different phenomena. The correlation between the two measures was poor (even worse than was reported in a prior study of patients in care in 2008-2009).[7] People who attended ≥75% of scheduled visits but were not retained according to the NAM definition (≥2 visits >90 days apart) were more likely to be members of less traditionally vulnerable groups (male, non-Hispanic white, non-IDU, no prior AIDS diagnosis), and were less likely to die in the 2-years after the study year compared to people who missed scheduled visits, suggesting that “failure to be retained” by the kept visit measures might actually be an indicator (for at least some people) that frequent follow-ups are not deemed clinically necessary. In contrast, people who missed >25% of scheduled visits were more likely to be members of minoritized and socially vulnerable groups (non-white race/ethnicity, history of IDU), and had a high risk of subsequent viral non-suppression and death. Focusing only on kept-visit measures of retention would fail to identify people at high risk of a poor outcome. Wherever possible, people who are missing scheduled visits should be targeted to identify and address barriers to their engagement in care. Additionally, future studies aiming to identify correlates of retention might identify more clinically meaningful correlates by focusing on missed-visit definitions of retention.
One limitation of this analysis was our inability to distinguish between loss to clinic (but retained in HIV care elsewhere) and loss to care (and associated access to ART and other preventive monitoring and treatments).[19, 20] We might further undercount patient-clinician encounters because these data do not capture encounters that occur outside the context of a clinic visit (e.g., phone calls, text messages). These limitations might explain the relative strength of missed visits (which are completely captured) versus kept visits (which may be under-counted for people who seek care in other clinics) for predicting viral non-suppression and death. This is a limitation of most clinical cohort studies unless they are supplemented with additional data (e.g., linkage to surveillance data or tracing studies).[21−23] Out-migration can substantially bias estimates of retention in care when retention is based on kept visits.[24, 25] However, when loss to clinic is loss to care (not seeing an HIV provider at all), it may be more important than having missed some visits but still occasionally seeing an HIV provider. Finally, although missed clinic visits are independently associated with subsequent viral suppression and mortality, calculating visit adherence requires data on scheduled visits (even when not attended), which may only be available at the clinic-level. Monitoring missed-visit measures of retention may be infeasible in some settings (e.g., public health surveillance via laboratory tests or HRSA client-level reporting). There were no clear patterns regarding who would be classified as retained by kept-visit measures based on clinic visits versus HIV labs.
We would be bereft if we did not acknowledge the disparities in retention identified in this analysis. Non-white race/ethnicity, a history of IDU, not having MSM as an HIV risk, and a prior AIDS diagnosis were all associated with missed visits, and missed visits were, in turn, associated with subsequent poor clinical outcomes. Monitoring retention according to missed visits, might, therefore be thought of as an equity issue.
Different definitions of retention capture different constructs. Different definitions identified different people as not retained . Missed-visit measures of retention appear to be more strongly associated with subsequent viral suppression and mortality than measures based on kept visits. Where possible, future evaluations of the HIV care continuum should consider a nuanced spectrum of retention.
Supplementary Material
Funding:
This study was supported by grants from the National Institutes of Health (K01 AA028193, U01 DA036935, K24 AA027483, K08 MH118094, K01 AI131895, R24 AI067039, P30 AI027767, P30 AI027757, P30 AI036214, P30 AI027763, P30 AI036219, P30 AI094189, and P30 AI060354).
Conflicts of Interest:
PFR received funding from Gilead and Johnson & Johnson for unrelated research and consulting; all other authors have no conflicts of interest to disclose.
REFERENCES
- 1.Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, et al. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med 2011; 365(6):493–505. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV Epidemic: A Plan for the United States. JAMA : the journal of the American Medical Association 2019; 321(9):844–845. [DOI] [PubMed] [Google Scholar]
- 3.Li Z, Purcell DW, Sansom SL, Hayes D, Hall HI. Vital Signs: HIV Transmission Along the Continuum of Care - United States, 2016. MMWR Morbidity and mortality weekly report 2019; 68(11):267–272. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Skarbinski J, Rosenberg E, Paz-Bailey G, Hall HI, Rose CE, Viall AH, et al. Human immunodeficiency virus transmission at each step of the care continuum in the United States. JAMA internal medicine 2015; 175(4):588–596. [DOI] [PubMed] [Google Scholar]
- 5.Gray KM, Cohen SM, Hu X, Li J, Mermin J, Hall HI. Jurisdiction Level Differences in HIV Diagnosis, Retention in Care, and Viral Suppression in the United States. J Acquir Immune Defic Syndr 2014; 65(2):129–132. [DOI] [PubMed] [Google Scholar]
- 6.Mugavero MJ, Davila JA, Nevin CR, Giordano TP. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS 2010; 24(10):607–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mugavero MJ, Westfall AO, Zinski A, Davila J, Drainoni ML, Gardner LI, et al. Measuring retention in HIV care: the elusive gold standard. J Acquir Immune Defic Syndr 2012; 61(5):574–580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Mugavero MJ, Lin HY, Allison JJ, Giordano TP, Willig JH, Raper JL, et al. Racial disparities in HIV virologic failure: do missed visits matter? J Acquir Immune Defic Syndr 2009; 50(1):100–108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Mugavero MJ, Lin HY, Willig JH, Westfall AO, Ulett KB, Routman JS, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis 2009; 48(2):248–256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Thompson MA, Horberg MA, Agwu AL, Colasanti JA, Jain MK, Short WR, et al. Primary Care Guidance for Persons With Human Immunodeficiency Virus: 2020 Update by the HIV Medicine Association of the Infectious Diseases Society of America. Clinical Infectious Diseases 2020. [DOI] [PubMed] [Google Scholar]
- 11.Saag MS, Gandhi RT, Hoy JF, Landovitz RJ, Thompson MA, Sax PE, et al. Antiretroviral Drugs for Treatment and Prevention of HIV Infection in Adults: 2020 Recommendations of the International Antiviral Society–USA Panel. JAMA : the journal of the American Medical Association 2020; 324(16):1651–1669. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kitahata MM, Rodriguez B, Haubrich R, Boswell S, Mathews WC, Lederman MM, et al. Cohort profile: the Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol 2008; 37(5):948–955. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lesko CR, Cole SR, Hall HI, Westreich D, Miller WC, Eron JJ, et al. The effect of antiretroviral therapy on all-cause mortality, generalized to persons diagnosed with HIV in the USA, 2009–11. Int J Epidemiol 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Rebeiro PF, Horberg MA, Gange SJ, Gebo KA, Yehia BR, Brooks JT, et al. Strong agreement of nationally recommended retention measures from the Institute of Medicine and Department of Health and Human Services. PLoS One 2014; 9(11):e111772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lesko CR, Chander G, Moore RD, Lau B. Variation in estimated viral suppression associated with the definition of viral suppression used. Aids 2020; 34(10):1519–1526. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hamlett A, Ryan L, Serrano-Trespalacios P, Wolfinger R. Mixed models for assessing correlation in the presence of replication. J Air Waste Manag Assoc 2003; 53(4):442–450. [DOI] [PubMed] [Google Scholar]
- 17.Sato T, Matsuyama Y. Marginal structural models as a tool for standardization. Epidemiology 2003; 14(6):680–686. [DOI] [PubMed] [Google Scholar]
- 18.Cole SR, Hernan MA. Constructing Inverse Probability Weights for Marginal Structural Models. American Journal of Epidemiology 2008; 168(6):656–664. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Edwards JK, Cole SR, Westreich D, Moore R, Mathews C, Geng E, et al. Loss to Clinic and Five-Year Mortality among HIV-Infected Antiretroviral Therapy Initiators. PloS one 2014; 9(7):e102305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Edwards JK, Lesko CR, Herce ME, Murenzi G, Twizere C, Lelo P, et al. Gone but not lost: implications for estimating HIV care outcomes when loss to clinic is not loss to care. Epidemiology 2020; 31(4):570–577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ballif M, Christ B, Anderegg N, Chammartin F, Muhairwe J, Jefferys L, et al. Tracing people living with HIV who are lost to follow-up at ART programs in Southern Africa: A sampling-based cohort study in six countries. Clinical infectious diseases 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Geng EH, Odeny TA, Lyamuya R, Nakiwogga-Muwanga A, Diero L, Bwana M, et al. Retention in care and patient-reported reasons for undocumented transfer or stopping care among HIV-infected patients on antiretroviral therapy in Eastern Africa: application of a sampling-based approach. Clinical Infectious Diseases 2016; 62(7):935–944. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Rebeiro PF, Bakoyannis G, Musick BS, Braithwaite RS, Wools-Kaloustian KK, Nyandiko W, et al. An Observational Study of the Effect of Patient Outreach on Return to Care: The Earlier the Better. Journal of acquired immune deficiency syndromes (1999) 2017; 76(2):141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Dombrowski JC, Buskin SE, Bennett A, Thiede H, Golden MR. Use of multiple data sources and individual case investigation to refine surveillance-based estimates of the HIV care continuum. Journal of acquired immune deficiency syndromes (1999) 2014; 67(3):323–330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Buskin SE, Kent JB, Dombrowski JC, Golden MR. Migration Distorts Surveillance Estimates of Engagement in Care: Results of Public Health Investigations of Persons Who Appear to be Out of HIV Care. Sex Transm Dis 2014; 41(1):35–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.