Validation of an Abbreviated Version of the Denver HIV Risk Score for Prediction of HIV Infection in an Urban Emergency Department

Abstract

Objective

We sought to evaluate the performance of an abbreviated version of the Denver HIV Risk Score (DHRS) in two urban emergency departments (ED) with known high undiagnosed HIV prevalence.

Methods

We performed a secondary analysis of data collected prospectively between November 2005 and December 2009 as part of an ED-based non-targeted rapid HIV testing program from two sites. Demographics, past HIV testing history, IDU, and select high-risk sexual behaviors, including men who have sex with men, were collected by standardized interview. Information regarding receptive anal intercourse and vaginal intercourse were either not collected or collected inconsistently and where thus omitted from the model to create its abbreviated version.

Results

The study cohort included 15,184 patients with 114 (0.75%) newly diagnosed with HIV infection. HIV prevalence was 0.41% (95% CI: 0.21% – 0.71%) for those with a score <20, 0.29% (95% CI: 0.14% – 0.52%) with a score of 20–29, 0.65% (95% CI: 0.48% – 0.87%) with a score of 30–39, 2.38% (95% CI: 1.68% – 3.28%) with a score of 40–49, and 4.57% (95% CI: 2.09% – 8.67%) with a score ≥50. External validation resulted in good discrimination (area under the receiver operating characteristics curve = 0.75, 95% CI: 0.71–0.79). The calibration regression slope was 0.92 and its R² was 0.78.

Conclusions

An abbreviated version of the DHRS had comparable performance to that reported previously, offering a promising alternative strategy for HIV screening in the ED where limited sexual risk behavior information may be obtainable.

Introduction

Emergency Departments (EDs) are the most common site of missed opportunities to diagnose HIV infection in medical settings.¹ In 2001, the Centers for Disease Control and Prevention (CDC) revised their HIV testing guidelines and specifically highlighted EDs as the primary nontraditional clinical care settings for expanded HIV testing.² Under those guidelines, the strategy for HIV screening remained the same as that recommended in 1993,³ with risk-based screening (i.e., routinely asking patients about risks for HIV infection and offering confidential voluntary HIV counseling and testing for those at risk) except for those in high HIV/AIDS prevalence areas, where routine screening was recommended. In 2006, the CDC again revised their guidelines, recommending routine nontargeted (non-risk-based) opt-out HIV screening for all patients 13 to 64 years of age.⁴ Since then, the growing numbers of EDs have developed strategies to adopt HIV screening as part of their routine practice and reported a much higher rates of HIV positivity above the CDC threshold of 0.1% in those with systematic testing programs.⁵ Yet, the majority of EDs with systematic HIV screening programs still do not use nontargeted approaches, mainly because it is operationally challenging to implement.^6,7 Furthermore, it is costly for EDs to implement such screening programs,⁸ even with external public and private grant funding support.

Recently, a clinical prediction instrument, called the Denver HIV Risk Score (DHRS), was derived and validated in two relatively low HIV prevalent settings in Denver, Colorado and Cincinnati, Ohio. The purpose of the DHRS was to categorize patients into distinct risk groups with increasing probabilities of HIV infection in an effort to help inform routine HIV screening.⁹ In its original form, the DHRS includes the following eight characteristics: age, gender, race/ethnicity, sex with a male, vaginal intercourse, receptive anal intercourse, injection drug use (IDU), and a past HIV test. Several of these variables may be considered sensitive to both ED patients and providers, and thus are not always feasible to collect routinely in busy clinical settings, including EDs. Many were wondering if a modification of DHRS based on local availability of key elements might be applied and perform well in their own ED-based HIV testing programs. Therefore, we sought to evaluate the performance of an abbreviated version of the DHRS that excludes one and modifies two sexual behavioral variables in an inner-city ED located in a city with known high undiagnosed HIV prevalence.

Materials and Methods

Study Design

We performed a secondary analysis of data collected prospectively between November 2005 and December 2009 as part of a nontargeted rapid oral fluid HIV screening program from two sites: an inner-city ED with 60,000 adult visits per year (Site A) and an urban ED with 55,000 annual visits (Site B). The study was approved by the institutional review boards of the institutions.

Setting

Two urban EDs that are part of a single university health care system located in Baltimore, Maryland. Site A is an inner-city adult ED with 60,000 visits/year, whose population is socioeconomically disadvantaged, with > 75% African Americans, 15% prior or current injection drug users, and 11~12% HIV seroprevalence (with approximately 2.2% rate of new diagnosis in 2006 and 0.8% in 2009).^7,10 Site B is an urban adult and pediatric ED with 55,000 visits/year, which serves an ethnically and socio-economically diverse population, with 30–35% African Americans and high rates of sexually transmitted infections; overall rates of HIV at that site are not known, but rates of newly diagnosed HIV are approximately 0.3%.¹¹

Data Collection

Demographics (age, gender, race/ethnicity), past HIV testing history, IDU, and some sexual risk behaviors, including men who have sex with men, were collected by a standardized interview by HIV screening facilitators as part of the program. Elements in the DHRS explicitly collected using our standardized interview included: age, gender, race/ethnicity, sex with a male, injection drug use and HIV testing history. Information regarding receptive anal intercourse and vaginal intercourse, two of the sexual practice elements where are parts of the DHRS, was not explicitly collected at our site. Information regarding having sex with men was consistently collected for male or transgender patients throughout the study periods but was not collected for female patients in the early part of study period and was not consistently collected afterwards. HIV testing facilitators interviewed patients, recorded data on standardized data collection instruments, and entered data into a password-protected secure departmental website.

Data Analysis

The DHRS was retrospectively applied to the de-identified data set (Table 1).⁹ Males, females or transgender patients who stated having sex with men were assigned a score of +22 for the item “sex with a male”; the remaining subjects were assigned a score of 0. No score of “receptive anal intercourse” and “vaginal intercourse” was assigned to each subject since the information was not collected. Patients were grouped into risk score categories (<20, 20–29, 30–39, 40–49, ≥50) and HIV positivity rate within each group was reported as percentages with 95% CIs. Calibration of the abbreviated DHRS was assessed by plotting and comparing predicted HIV prevalence to observed HIV prevalence and reported as a slope and R² of the corresponding linear regression line using the same analytical approach in the original derivation study.⁹ Discrimination was assessed by constructing a receiver operating characteristics curve and calculating its area under the curve. Sensitivity analysis was performed by (1) treating all females having sex with a male because the variable “sex with a man” was not consistently collected for female patients (model 1) and (2) removing all three sexual questions in the original DHRS (model 2). Statistical analyses were performed using SAS, version 9.3 (SAS Institute, Inc., Cary North Carolina).

Table 1.

Denver HIV Risk Score and an abbreviated Denver HIV Risk Score.

Variables	Categories	Original Score	Modified Score*
Age	<22 or >60 years	0	0
	22–25 or 55–60 years	+4	+4
	26–32 or 47–54 years	+10	+10
	33–46 years	+12	+12
Gender	Female	0	0
	Male	+21	+21
Race/Ethnicity	Black	+9	+9
	Hispanic	+3	+3
	White	0	0
	Otherʃ	0	0
Sexual Practices	Sex with a man	+22	+22
	Vaginal intercourse	−10	NC†
	Receptive anal intercourse	+8	NC†
Injection Drug Practices	Injection Drug Use	+9	+9
HIV Testing	Having an HIV test in the past	−4	−4

^*In the modified abbreviated Denver HIV Risk Score applying for Site A and B, men or transgender who stated having sex with men and all females were assigned a score of 22 for the item “sex with a man”; the remaining subjects were assigned a score of 0. No score of “vaginal intercourse” and “receptive anal intercourse” was assigned to each subject.

^ʃRepresents Asian, American or Alaskan Native, Hawaiian, and Pacific Islander

^†NC: Not collected in Site A and Site B screening program

Results

During the study period, 15,184 ED patients were tested for HIV infection. Of these, 114 (0.75%) were newly diagnosed with HIV infection. The demographic and risk characteristics related to the DHRS are summarized in Table 2. The majority of patients were young adults under 40 years of age (50.7%), female (53.7%), African American (58.3%), and having been previously tested for HIV infection (61.6%).

Table 2.

Demographics and risk characteristics in 15,184 patients of two emergency departments (ED) who received a rapid HIV test via ED-based rapid HIV screening program

Characteristics	Categories	Number (%) N=15,184
Age	<22	1,614 (10.6)
	22–25	1,712 (11.3)
	26–32	2,354 (15.5)
	33–46	4,720 (31.1)
	47–54	2,837 (18.7)
	55–60	1,424 ( 9.4)
	61–64	523 ( 3.4)
Gender	Female	8,160 (53.7)
	Male	7,017 (46.2)
	Transgender	7 ( 0.1)
Race/Ethnicity	Black	8,844 (58.2)
	Hispanic	449 ( 3.0)
	White	5,331 (35.1)
	Other*	560 ( 3.7)
Sexual Practices	Men who have sex with men	210 ( 3.0)†
Injection Drug Practices	Injection drug use	1,085 ( 7.2)
HIV Testing	Having an HIV test in the past	9,353 (61.6)

^*Represents Asian, American or Alaskan Native, Hawaiian, Pacific Islander, race not specified, race unknown, or declined.

The median abbreviated DHRS was 31 (interquartile range: 22–37; range: −4–73). HIV prevalence was 0.41% (12/2961, 95% CI: 0.21% – 0.71%) for those with a score <20, 0.29% (10/3420, 95% CI: 0.14% – 0.52%) with a score of 20–29, 0.65% (46/7052, 95% CI: 0.48% – 0.87%) with a score of 30–39, 2.38% (37/1554, 95% CI: 1.68% – 3.28%) with a score of 40–49, and 4.57% (9/188, 95% CI: 2.09% – 8.67%) with a score ≥50 (Figure 1A). The calibration regression slope was 0.92 and its R² was 0.78 (Figure 2A), whereas the area under the receiver operating characteristics curve was 0.70 (95% CI: 0.65–0.75) (Figure 3A).

Figure 1.

Figure 1A, 1B & 1C. 1A: Prevalence of HIV infection within each risk score category of abbreviated Denver HIV Risk Score in 15,184 patients of two emergency departments (ED) who received a rapid HIV test via ED-based rapid HIV screening program; 1B: Sensitivity analysis by treating all females having sex with a male; 1C: Sensitivity analysis by excluding all sexual-related question

Bars, 95% confidence interval

For 1C, category of score of ≥50 was collapsed with score of 40–49 since only 21 patients had a score of 50 or more.

Figure 2.

Figure 2A, 2B & 2C: 2A: Calibration of abbreviated Denver HIV Risk Score to identify patients at risk of HIV infection, Baltimore 2005–2009; 2B: Sensitivity analysis by treating all females having sex with a male; 2C: Sensitivity analysis by excluding all sexual-related question

Figure 3.

Figure 3A, 3B & 3C. 3A: Discrimination of abbreviated Denver HIV Risk Score to identify patients at risk of HIV infection, Baltimore 2005–2009; 3B: Sensitivity analysis by treating all females having sex with a male; 3C: Sensitivity analysis by excluding all sexual-related question

In the sensitivity analysis, for model 1, we found that the median abbreviated DHRS was 33 (interquartile range: 29–38; range: 17–73). HIV prevalence was 0.0% (0/269, 95% CI: 0% – 1.20%) for those with a score <20, 0.13% (5/3830, 95% CI: 0.04% – 0.31%) with a score of 20–29, 0.59% (52/8812, 95% CI: 0.44% – 0.77%) with a score of 30–39, 2.33% (48/2063, 95% CI: 1.72% – 3.09%) with a score of 40–49, and 4.29% (9/210, 95% CI: 1.96% – 8.14%) with a score ≥50 (Figure 1B). The calibration regression slope was 1.02 and its R² was 0.73 (Figure 2B), whereas the area under the receiver operating characteristics curve was 0.75 (95% CI: 0.71–0.79) (Figure 3B). For model 2, we found that the median abbreviated DHRS was 19 (interquartile range: 10–31; range: −4–51). HIV prevalence was 0.4% (31/7708, 95% CI: 0.28% – 0.56%) for those with a score <20, 0.43% (11/2543, 95% CI: 0.23% – 0.75%) with a score of 20–29, 1.00% (39/3885, 95% CI: 0.73% – 1.36%) with a score of 30–39, 3.21% (33/1027, 95% CI: 2.26% – 4.43%) with a score of 40–49, and 0.0% (0/21, 95% CI: 0% – 13.29%) with a score ≥50 (or 3.15%, 95% CI: 2.21%, 4.34%, for those with a score ≥40)(Figure 1C). The calibration regression slope was 1.5 and its R² was 0.79 (Figure 2C), whereas the area under the receiver operating characteristics curve was 0.72 (95% CI: 0.67–0.76) (Figure 3C).

Discussion

The original DHRS, a clinical prediction tool for estimating the probability of patients being infected with HIV, provides a verified approach that could be used to target HIV screening in clinical practice.⁹ Our findings suggest that the abbreviated version of the DHRS performed fairly when applied retrospectively using data from two urban EDs in Baltimore, Maryland, and despite limited sexual risk behavior information. The original DHRS study reported that 74% and 63% of all patients diagnosed with HIV infection were in higher risk score groups (i.e. a score ≥30) in the derivation sample and validation sample, respectively.⁹ If the DHRS was applied in practice at our site (i.e. restrict testing to those with a score ≥30), our data demonstrated that 81% of newly-diagnosed HIV-infected cases detected using nontargeted screening would have been identified while testing only 58% of the patients in overall approximately 15,200 patients. The results from model 2 in our sensitivity analysis which was an even more limited form of DHRS by excluding all three sexual questions also demonstrated similar accuracy to detect patients at higher risk for HIV with an identification of 63% of undiagnosed cases while only testing only 33% of overall patients. This method may potentially provide a more efficient and economical alternative in the ED to identify undiagnosed HIV infections as compared nontargeted screening or other “universal” screening methods. A subsequent prospective, before-after study that used the original DHRS demonstrated that targeted screening using the DHRS was more strongly associated with identification of newly-diagnosed HIV infection than nontargeted screening in an acute care setting.¹²

Although our results externally validated the use of abbreviated version of DHRS, application of limited sexual risk behavior information seemed to be only slightly underestimated observed HIV seropositivity for those with higher risk scores in our population (i.e., lower predicted HIV seropositive than observed in higher risk score groups, slope=0.92), while results from the original validation of the DHRS showed almost perfect calibration between predicted and observed HIV prevalences.⁹ One possible explanation is that the sexual risk information is important in the prediction of HIV infection; however, another possible explanation is that our ED population and risks for HIV infection in our ED patients were different from those in Denver, Colorado or Cincinnati, Ohio. Evidence is that our EDs are located in Baltimore urban area where HIV seroprevalence is relatively higher than those in above 2 cities.^9,13 In addition, long-term and successful in needle exchange program in the city¹⁴ might also have impacted the importance of one risk score variable (i.e., IDU in the population). Periodic and site-specific modification of DHRS for the application in clinical settings should be carefully evaluated before its implementation.

Coupled with the Patient Protection and Affordable Care Act (PPACA) passed in March 2010¹⁵ and the announcement of a Grade “A” recommendation on screening for HIV in individuals aged 15–65 years from clinicians by the U.S. Preventive Services Task Force (USPSTF) in the spring of 2013,¹⁶ the cost of HIV testing, a high-value clinical preventive service, should and will be covered through no costs sharing to the beneficiaries. A dramatic increase in the number of ED patients tested in the near future is likely to be expected. Sustainability of ED-based HIV screening programs has become an major concern since the decrease in grant funding for screening programs from public health agencies as well as uncertainty of reimbursement for HIV screening in the ED.^17,18 It may be more sustainable for such resource-intensive public health initiative if information technology can be effectively integrated DHRS with ED electronic patient tracking systems and electronic medical record systems via novel computerized kiosk systems.¹⁹

Routinely offering HIV testing or nontargeted screening is appraised by many because it may remove exceptionalism and negative stigma around HIV testing.^20–22 Screening using DHRS could attract criticisms as a result of these reasons. However, most current ED-based nontargeted screening programs require substantial internal and external resources and therefore only test limited proportions of eligible ED patients while identifying only a small number of previously undiagnosed infections.²³ Large-scale prospective trials comparing nontargeted DHRS approach to different forms of non-targeted screening in the ED as well as some qualitative in-depth interview studies in ED patients, clinical providers, ED and hospital financial administrators, and public health policy decision makers are needed in order to better understand the utilization of DHRS in the ED.

This study has several limitations. First, information routinely collected from our ED-based rapid HIV screening program was not specifically collected for the external validation of the DHRS. Questions to obtain risk information when collected might be addressed or interpreted differently from those questions in the original derivation study. Second, patients might not disclose their risk information to our trained testing facilitators during the brief face-to-face interview. Consequently, the risk score obtained in our study might be underestimated, especially for those with high risk behaviors. The calibration curve from our analysis actually demonstrates this, i.e. ²⁴ the expected HIV seropositivity was slightly lower than the observed one in patients, especially in those with lower risk score (Figure 2A). Novel risk assessment via touch-screen kiosk or tablet could significantly reduce this type of information bias.²⁴ Third, the HIV screening programmatic data come from two geographically close EDs under the same hospital and health system. These two urban EDs are located in a high HIV/AIDS prevalence city where several large-scale HIV intervention programs have been implemented.²⁵ The local HIV epidemic is likely to be different in demographic and risk groups from other locales in U.S. Thus, one should be cautious when trying to generalize our findings to other EDs elsewhere. Another limitation is that in the sensitivity analysis all women were considered to have answered “yes” to having sex with males since the question was not asked consistently. Their actual risk is limited by the fact that the actual data was not collected for all female patients.

In conclusion, the DHRS possesses potential utilization in ED-based targeted screening program to efficiently identify previously undiagnosed HIV-infected patients, even only limited sensitive risk information available. Further integration with hospital electronic medical record system and novel deliver methods to remove exceptionalism and stigma such as kiosks are needed before it is widely implemented.