Assessing the use of HIV surveillance data to help gauge patient retention-in-care

Abstract

Background

Improved retention-in-care may enhance health outcomes for people living with HIV/AIDS (PLWHA). While laboratory surveillance data may be used to gauge retention, no previous reports have compared surveillance lab vs. clinic visit-based measures of retention-in-care. We compared lab surveillance vs. clinic visit-based approaches for identifying retention status for PLWHA.

Methods

We examined 2011 patient visit data from the Ruth M. Rothstein CORE Center, Cook County's HIV clinic. We defined retained patients as those with visits every 6 months over 2 years and matched patients classified via visit data against HIV surveillance labs reported to the Chicago Department of Health. We determined the sensitivity, specificity and receiver operator characteristics of varying lab surveillance vs. clinic visit measures of retention.

Results

Of patients classified via clinic visit data, 91% of 1,714 in-care vs. 22% of 200 out-of-care patients met our most stringent surveillance based retention definition – having ≥ 2 viral load/CD4s performed 90 days apart reported by the same laboratory in 2011. Of surveillance lab-based definitions for retention, having ≥ 2 HIV viral load and/or CD4 values at least 3 months apart reported from the same facility possessed the best receiver operator parameters and the receiver operator characteristics curve comparing several surveillance lab vs. clinic-visit based retention measures had an area under the curve of 0.95.

Discussion

Our findings demonstrate that surveillance laboratory data can be used to assess retention-in-care for PLWHA. These data suggest that bi-directional data sharing between public health entities and care providers could advance re-engagement efforts.

Introduction

For people living with HIV/AIDS (PLWHA) effective linkage and retention-in-care improve rates of virologic suppression, and virologically suppressed PLWHA less readily transmit infection^1-7. It follows that poor rates of linkage and retention-in-care may contribute to the United States' relatively static HIV incidence rate⁸. Retention-in-care represents a key component of the HIV care continuum and improvements in retention rates may have significant impact on achieving National HIV/AIDS Strategy (NHAS) objectives⁹. As a reflection of the importance of retention in care, the NHAS has set a goal of increasing the percentage of Ryan White program clients' retained-in-care to 80% by 2015⁹.

Recent data demonstrate ample room for improvement with respect to retention-in-care for PLWHA in the US. Regardless of which retention-in-care indicator one considers, retention-in-care remains sub-optimal and below NHAS's 2015 retention-in-care goal. The Centers for Disease Control and Prevention (CDC) reports that, for those jurisdictions with mature HIV laboratory surveillance in place, only 51% of PLWHA qualify as retained-in-care, as defined as having ≥ 2 CD4 or HIV viral loads measured, at least three months apart in 2010¹⁰. Using a different measure, the proportion of Ryan White clients with ≥ two visits at least 90 days apart in 2010, the Health Resources Services Administration (HRSA) reports 76% of patients as retained-in-care¹¹. HRSA's HIV/AIDS Bureau (HAB) employs a different performance measure for retention-in-care, the proportion of patients with a visit in successive six month periods over 24 months. Employing this definition, HRSA HAB reported a mean rate of retention-in-care of 69% among reporting Ryan White grantees^12,13.

Given that numerous interventions have targeted improving retention-in-care, and considering that various agencies assess and report retention differently, with the work reported here, we sought to gain a better understanding of how different measures of retention-in-care relate to one another¹⁴. The National HIV Surveillance System (NHSS) represents a key program to measure quality-of-care for PLWHA. Given that that NHSS requires reporting of all CD4 and HIV viral loads, the CDC has used these data to as a surrogate marker for retention in HIV care. Conversely, HRSA has access to clinic visit data via Ryan White client level reporting.

With this report, the Chicago Developmental Center for AIDS Research (D-CFAR) partnered with the Chicago Department of Public Health (CDPH) to consider retention-in-care for a population of PLWHA receiving care at Chicago's largest ambulatory HIV clinic. For these patients, we compared clinic visit vs. NHSS surveillance lab-based measures of retention-in-care with the primary objective of assessing the validity of varying HIV surveillance lab-based definitions of retention. We believe that data presented here may help expand the applicability of HIV lab surveillance data for identifying out-of-care patients and support department of health efforts to improve patient retention-in-care

Methods

We obtained approval from the Cook County Health and Hospitals System (CCHHS), CDPH and Rush University Medical Center's Institutional Review Boards prior to initiating this research.

Setting

We drew our sample of patients with clinical visit-based retention-in-care data from patients receiving outpatient care at the Ruth M. Rothstein (RMR) CORE Center. The CCHHS affiliated RMR CORE Center provides ambulatory primary care to PLWHA in the Chicago eligible metropolitan area (EMA) without regard to patients' ability to pay. The RMR CORE Center cares for approximately 5500 PLWHA annually, or nearly one fifth of the Chicago EMA's PLWHA¹⁵.

Patients included

In our efforts to compare clinic visit vs. HIV laboratory surveillance-based measures of retention-in-care, we examined several populations of patients attending clinic at the RMR CORE Center. We first identified patients whom attended a primary care visit within the first 6 months of 2010. We used this group as our initial group from which we identified “in-care” patients. All patients had been diagnosed with HIV prior to January, 2010.

Clinic visit-based “in-care” cohort defined

We employed the HRSA HAB definition for retained-in-care, see Table 1. We chose this more stringent clinical visit-based definition for retention, as we planned to use this measure as our gold standard against we would assess HIV lab surveillance-based definitions for retention-in-care. Using this definition, we identified patients as truly retained-in-care for 2011 per this stringent HRSA HAB clinic visit-based retention definition.

Table 1. Clinic visit and laboratory surveillance-based definitions used to identify patients as retained-in-care vs. out-of-care.

Clinic visit-based definitions used for retention-in-care
“In-care” cohort:
Patients with at least 1 clinic visit in each 6 month of a 24-month period, from Jan, 2010 through Dec 31, 2011, with a minimum of 60 days between the first visit in the prior 6 months and the last visit in the subsequent 6 months.
“Out-of-care” cohort:
Patients whom had visited the clinic during the last half of 2010, but had no visits in 2011.
Varying lab surveillance-based definitions for retention-in-care
Definition 1:
Patients whom had ≥ 1 CD4 and/or HIV viral load reported to CDPH for 2011
Definition 2:
Patients whom had ≥ 2 CD4 and/or HIV viral loads done 90 days apart and reported to CDPH for 2011
Definition 3:
Patients whom had ≥ 2 CD4 and/or HIV viral loads 90 days a part performed by the same lab and reported to CDPH for 2011

Clinic visit-based “out-of-care cohort defined

To identify an “out-of-care” cohort we identified RMR CORE Center patients who had visited the clinic during the last half of 2010, but had no visits in 2011 (Table 1). We used this definition for several reasons. We presumed that if we had just chosen patients who had a visit in the first six months of 2010, but then did not meet the HRSA/HAB definition as retained, there may have been a significant subset of patients that missed a visit in the second half of 2010, but then had visits in 2011. If we had included this subset of patients as “out-of-care”, then many would still have HIV lab surveillance submitted related to their 2011 primary care visits – which would end up adversely, and somewhat artificially, impacting the accuracy of the lab surveillance measures of retention/non-retention. While it would have been optimal if we could have used two years of HIV surveillance data to match with a full 24-month clinic visit observation period, this was not possible as prior to 2011, the CD4/VL reporting to NHSS in Illinois was not well established, while collection of the 2012 lab surveillance data had not been completed at the time of our analysis.

Comparison of demographic and clinical traits for retained vs. non-retained patients

We employed basic descriptive statistics to report demographic and clinical characteristics for the “in-care” and “out-of-care” patients who visited the RMR CORE Center and our included in this analysis. We report on patients' age, gender, race, ethnicity and CD4 count and HIV viral loads. For this analysis, we drew CD4 and HIV viral load values from labs ordered at the RMR CORE Center and we used the chronologically latest 2011 values in calculating mean CD4 and proportion of patients with undetectable HIV viral loads (HIV viral load < 200 copies/ml). We used Chi-squared and Student's T-test, with two tailed p-values, to compare demographic and lab characteristics for patients considered retained-in-care vs. not retained-in-care, based on the clinic visit-based definition for retention detailed above.

We also carried out multivariable logistic regression in order to assess which clinical and demographic factors correlated with retention-in-care. For this analysis, we considered retention status as the dependent variable. We considered the following independent variables for inclusion in the multivariable model, and included factors with p-value < 0.2 on univariable analysis: gender (male vs. female); race (Black vs. non-Black); ethnicity (Hispanic vs. non-Hispanic); age, as a binary variable, comparing patients aged < 47 vs. ≥ 47 years old, as 47 was the median age of patients in our sample. We also considered HIV VL (< 200 vs. ≥ 200 copies/ml) as a binary variable and CD4 for inclusion in the multivariable model (Stata version 13, Statacorp, College Station, TX).

Cross matching clinic patients with CDPH's NHSS lab data

Illinois, inclusive of the Chicago EMA, represents one of the 19 jurisdictions with lab surveillance reporting of CD4 and HIV viral load values which the CDC considers as sufficiently mature and complete such that data from these areas can be included in nationally reported HIV surveillance summaries^8,10. For RMR CORE patients identified as “in-care” and “out-of-care”, as defined in the section above, we transmitted unique identifying information including first and last name and birthdate to the CDPH division of HIV/AIDS surveillance via a password protected and encrypted virtual private network. CDPH staff used these identifying data to match patients within their NHSS database. We excluded patients from further analysis if CDPH identified them as having: moved out of jurisdiction, died, not residing in the Chicago EMA or having received his/her HIV diagnosis in 2012.

For the remaining patients who matched between the RMR CORE clinic visit cohorts and the NHSS database, CDPH staff pulled all these patients 2011 CD4 and viral load data, as reported to them via electronic lab reporting systems.

Designation of retention-in-care status via HIV lab surveillance data

For RMR CORE Center patients which CDPH successfully matched to 2011 NHSS lab reporting data, CDPH surveillance staff assigned each patient a designation of being either in-care vs. out-of-care based on three related, but varied lab-based definitions, detailed in Table 1.

Assessing performance characteristics for lab-based retention measures

For the purposes of our analysis we considered the clinic visit-based retention-in-care status as the gold standard. We then assessed the sensitivity, specificity and receiver operator characteristics (ROC) using the three varying surveillance lab-based definitions for retention-in-care (see Table 1). ROC curves provide a graphic plot illustrating the performance of a binary classifier system (e.g. retained-in-care vs. not retained-in-care) as the discrimination threshold varies¹⁶. We created a ROC curve to assess the performance of varying surveillance lab-based definitions for retained-in-care vs. our selected gold standard clinic visit-based definition by plotting the sensitivity vs. 1- specificity. Diagnostic test evaluators consider the area under the ROC curve (AUC) as a global indicator of test accuracy, with AUC of 0.5 equating with a useless diagnostic test and an AUC of 1 representing the perfect diagnostic test. We present the sensitivity, specificity and ROC curve for HIV surveillance lab-based definitions' ability to designate patients as being retained-in-care.

Results

In 2011, 2464 out of 4181 (59%) patients whom made a primary care visit in the first six months of 2010 to the Ruth M. Rothstein CORE Center met the HRSA HAB clinic visit-based definition of retained-in-care over the subsequent 24 months. Conversely, 441 out of 4125 (11%) patients whom had visits in the last six months of 2010 then had no visits in 2011. Respectively, these patients constitute the clinic visit-based “in-care” and “out-of-care” patients whom we attempted to match with the CDPH NHSS laboratory database. Table 2 documents demographic and clinical traits of both the “in-care” and “out-of-care” RMR CORE Center patients.

Table 2. Demographic/clinical traits by clinic visit-based retention status.

	In-care N=2464	Out-of-care N=441	Univariable analysisp-value	Multivariable odd's ratio (95% CI)
Gender (%)
Male	657 (27)	100 (23)		1.35 (0.58 – 3.13)
Female	1805 (73)	338 (76)	p = 0.104	Referent
Transgender	2 (<1)	3 (1)		NA
Race (%)
Black	1519 (62)	307 (70)		1.10 (0.37 – 3.32)
White	878 (36)	117 (27)	p < 0.001
Asian	34 (1)	6 (1)		Referentb
Other/unknown	33 (1)	11 (2)
Ethnicity (%)
Hispanic	646 (26)	63 (14)	p < 0.001	0.11 (0.012 – 0.99)
Non-Hispanic	1818 (74)	378 (86)		Referent
Mean age (SD)	47 (4)	42 (12)	p < 0.001a	0.53 (0.23 – 1.22)c
Mean CD4(95% CI)	473(464-484)	510(409-610)	p = 0.751a	NAd
VL < 200 (%)	1962/2442 (80)	18/26 (69)	p = 0.157	0.42 (0.17 – 1.06)e

^aComparison via Student's T-test

^bBlack race compared to non-Black races as referent

^cAge considered as binary variable, with < 47 as the referent vs. ≥ 47 years of age

^dNot included in multivariable model since univariable p-value > 0.2

^eConsidered as binary variable with HIV VL > 200 copies as the referent category

As reported in Table 2, on univariable analysis, in-care vs out-of-care patients were older and more likely to identify as White and Hispanic. On multivariable logistic regression only Hispanic ethnicity associated with being “in-care” with an odd's ratio of 0.11 (95% CI 0.01 – 0.99, p = 0.049) vs the referent non-Hispanic ethnicity group. None of the other variables included in the multivariable regression model inclusive also of age, gender, race, and undetectable viral load status associated with being “in-care”.

One thousand nine hundred of 2,464 (77%) “in-care” vs. 284 of 441 (64%) “out-of-care” RMR CORE patients could be matched with CDPH's NHSS database (X²= 31.7, p < 0.001). CDPH could not match with the NHSS for patients receiving care at the RMR CORE Center whom resided outside the Chicago metropolitan statistical area, such as those patients living in suburban Cook County or Chicago's other surrounding collar counties (see Table 3).

Table 3. Results of match between RMR CORE in-care/out-of-care patients with CDPH NHSS database.

	Matched vs. non-matched of RMR CORE patients			Matched Patients
				Excluded from ROCc analysis			Included in ROC analysis
Clinic visit based retention status	Total	Non-matcheda (%)	Matched (%)	Deceased (%)	OOJd	DX =2012e	Living matched Chicago residents
Out-of-care patients	441	157 (36)b	284 (64)	29 (10)	55 (19)	0	200 (71)
In-care patients	2464	564 (23)	1,900 (77)	14 (1)	170 (9)	2 (<1)	1714 (90)
Total	2905	722 (25)	2184 (75)	43 (2)	225 (10)	2 (<1)	1914 (88)

^aNon-matched patients resided outside the Chicago metropolitan statistical area

^bRow percentages

^cROC = Receiver Operator Characteristics

^dOOJ = Out of Jurisdiction

^eWe excluded the 2 patients listed in the NHSS as diagnosed (DX) in 2012

Table 4 details the retention status classification based on surveillance labs for matched patients, whom we also designated as either “in-care” vs. “out-of-care” based on clinic visits. We report the sensitivity and specificity of the three varying HIV surveillance lab-based definitions for retention status compared to our clinic visit-based designation of retention status in Table 4. Using the sensitivities and specificities noted in Table 4, we plotted a ROC curve that graphically depicts the accuracy of the lab surveillance-based measures for correctly identifying patients as retained-in-care, compared to the clinic-visit based measure (see Figure 1). The area under the curve (AUC) for this ROC curve was 0.96, indicating that, given the definitions and parameters considered, the lab surveillance definitions selected can accurately identify patients as retained-in-care. Definition three, having ≥ two CD4 and/or HIV viral loads reported 90 days apart from the same lab represents the most accurate measure per the ROC curve, as indicated by its position located closest to the plot's upper left corner (Figure 1).

Table 4. Comparison of clinic visit vs. surveillance lab-based measures of retention.

	In-care vs. out-of-care based on clinic visit history
	In-care N=1714	Out-of-care N=200	Total N=1914	Performance characteristics
In-care vs. out-of-care based on lab surveillance-based definitions	N (%)	N (%)	N (%)	Sensitivitya(95% CI)	Specificityb(95% CI)
Definition 1: ≥ 1 lab reported, 2011
Yes (≥ 1 lab, 2011)	1692 (99)	93 (47)	1785 (93)	98.7%	53.5%
No (no labs reported, 2011)	22 (1)	107 (53)	129 (7)	(98.2 -99.2)	(46.6 – 60.4)
Definition 2: ≥ 2 labs reported, 2011
Yes (≥ 2 labs, 90 days apart)	1570 (92)	56 (28)	1626 (85)	91.6%	72.0%
No (< 2 labs, 90 days apart)	144 (8)	144 (72)	288 (15)	(90.3 – 92.9)	(69.9 – 74.1)
Definition 3: ≥ 2 labs from same lab, 2011
Yes (≥ 2 labs same lab)	1556 (91)	44 (22)	1600 (84)	90.8%	78.3%
No (< 2 labs, same lab)	158 (9)	156 (78)	314 (16)	(89.4 – 92.2)	(76.4 – 80.2)

^aSensitivity of lab measure to identify patient as retained-in-care, for patient deemed to be in-care based on clinic visit data.

^bSpecificity of lab measure to identify patient as out-of-care, for patient deemed as out-of-care based on clinic visit data.

Funding: This work was supported by a Center for AIDS Research supplemental grant administered by the District of Columbia Developmental Center for AIDS Research (PI Alan Greenberg, P30AI087714)

Figure 1.

ROC curve for surveillance lab-based measures of retention-in-care for clinically retained RMR CORE Center patients match to the National HIV Surveillance System database. ROC = Receiver Operator Characteristics; RMR = Ruth M. Rothstein; AUC = Area Under the Curve; surveillance lab based definitions illustrated here include, definition 1: ≥ 1 CD4 and/or HIV viral load reported in 2011, definition 2: ≥ 2 CD4 and/or HIV viral loads > 90 days apart reported in 2011, definition 3: ≥ 2 CD4 and/or HIV viral loads > 90 days apart reported from same lab in 2011.

Discussion

Both HRSA HAB and the NHAS have set retention-in-care goals, while the CDC regularly reports on lab-based retention-in-care based on NHSS data^10,13. With this analysis, we have attempted to clarify the relationship between clinic-visit and HIV surveillance lab-based measures of retention-in-care for PLWHA. In our assessment of the performance characteristics of three varying HIV surveillance lab-based definitions for retained-in-care, our ROC analysis suggested that our third definition – having had ≥ 2 CD4 and/or HIV viral loads measured by the same lab more than 90 days apart during a year – represented the best balance of sensitivity and specificity compared to our clinic visit-based definitions for retention-in-care. The high area under the ROC curve illustrating the relationship between retention status as determined by surveillance lab vs. clinic visits indicates the surveillance lab-based definitions may reasonably discriminate between retained vs. non-retained patients.

Our work has several important limitations. The lack of a true gold standard for retention status against which the surveillance lab-based definitions could be compared represents a key limitation. Patients in our “out-of-care” group based on clinic visits could have been in-care elsewhere, either within or outside of our jurisdiction. This may have adversely impacted the reported specificity of the surveillance lab-based definitions' ability to discriminate patients as being truly in-care vs. out-of-care.

Factors associated with the evolving maturity of the collection of HIV laboratory surveillance also limited our analysis. Because in Illinois, clinical laboratories did not come fully online with respect to routinely reporting all HIV viral load and CD4 values until 2011, and because, at the time of our analysis 2012 reporting had not been complete, we only had one year of surveillance lab data which we could compare with clinic-visit based data. Due to this limitation, we chose a non-standard clinic-visit-based “out-of-care” definition. We looked at patients who had a visit in the last half of 2010, but then no visit in 2011. This allowed us to compare a full year's time in which these “out-of-care” patients may have been receiving laboratory monitoring. We believe this adaptation allowed us to identify the most likely clinic visit-based “in-care” and “out-of'-care” patients against which we could most accurately compare surveillance lab-based retention measures.

Despite these limitations, we believe our analysis demonstrating HIV surveillance labs' ability to accurately discriminate the retention-in-care status of PLWHA has several implications. Notably, we demonstrate that meaningful, bi-directional data sharing between clinical care providers and the surveillance division of the local department of health can occur in a secure and confidential manner. Our finding that surveillance lab-based measures of retention-in-care relate well with clinic-visit based definitions may lend some reassurance to departments of health, and clinical care providers a like. We show that retention-in-care rates reported by departments of health and based on surveillance data have real world value with respect to patients' engagement with their medical home. The work presented here suggests that local departments of health can, with some confidence, use HIV lab surveillance data to identify lost-to-care patients and direct resources toward partnering with clinical care providers and/or AIDS Service Organizations (ASOs) to help re-engage these patients. We believe that such on-going and enhanced collaboration between HIV care providers, ASOs and the local and regional departments of health will facilitate improved retention-in-care for PLWHA.