Validation of a Quantitative HIV Risk Prediction Tool Using a National HIV Testing Cohort

Abstract

Routine screening is recommended for HIV detection. HIV risk estimation remains important. Our goal was to validate the Denver HIV Risk Score (DHRS) using a national cohort from the CDC. Patients ≥13 years of age were included, 4,830,941 HIV tests were performed, and 0.6% newly-diagnosed infections were identified. Of all visits, 9% were very low risk (HIV prevalence = 0.20%); 27% low risk (HIV prevalence = 0.17%); 41% moderate risk (HIV prevalence = 0.39%); 17% high risk (HIV prevalence = 1.19%); and 6% very high risk (HIV prevalence = 3.57%). The DHRS accurately categorized patients into different HIV risk groups.

INTRODUCTION

In the United States, over 200,000 individuals are infected but remain undiagnosed with the human immunodeficiency virus (HIV).¹ Screening efforts are at the forefront of HIV prevention in the United States. A principal goal of the National HIV/AIDS Strategy is to reduce the proportion of patients living with undiagnosed HIV infection to 10% by 2015.² To improve identification of HIV-infected persons, both the Centers for Disease Control and Prevention (CDC) and the U.S. Preventive Services Task Force (USPSTF) recommend routine nontargeted HIV screening.^3,4

Numerous studies of nontargeted HIV screening have shown this approach to successfully identify patients with HIV infection, though the effectiveness of such large-scale screening has been judged as modest^5,6 while others have raised concerns about its costs and inefficiencies.^5,7,8 Risk-based HIV screening still remains a viable alternative to nontargeted HIV screening.^9,10 Although the concept of targeted HIV screening has existed for decades¹¹ and risk characteristics have been extensively studied,^9,12 specific targeted strategies remain largely undefined and have not been broadly evaluated in practice.^13–15

In 2012 the Denver HIV Risk Score (DHRS), an empirically-derived clinical prediction instrument, was developed to help clinicians estimate patients’ probabilities of being infected with HIV, thus informing HIV screening and prevention counseling.¹⁶ The DHRS consists of 6 demographics and risk behaviors variables that, when applied to a patient, results in a score; individuals with scores ≥30 are considered at increased risk for HIV infection (Supplemental Digital Content, Appendix). The external validity of the DHRS has not been assessed broadly and its validity in areas where HIV prevalence is high or where demographic and risk behavior characteristics differ is unknown.¹⁸ The goal of this study was to validate the DHRS using data from a national HIV testing cohort from the CDC. We hypothesized that HIV prevalence would significantly increase with DHRS values ≥30 and this relationship would be consistent across all geographic regions of the United States.

METHODS

Study Design

We performed a secondary analysis of the CDC’s National HIV Program Evaluation and Monitoring System (PEMS). PEMS is a national data reporting system of a standardized set of HIV prevention variables, secure web-based software, and a range of data collection support services. This study was approved by our institutional review board.

Population and Setting

We used HIV testing data that were collected from all CDC-funded HIV testing sites throughout the United States (except Massachusetts, North Dakota, Ohio, and Rhode Island) from January 1, 2008 through December 31, 2010. We included patients ≥13 years of age, with no other exclusions. Testing data were not restricted according to testing venue and included emergency departments (EDs), hospitals, outpatient clinics, sexually transmitted diseases and HIV counseling and testing sites, community-based organization (CBOs), blood banks, plasma centers, and correctional facilities.

Data Collection and Study Variables

Collection and reporting of PEMS data is required by health departments funded through CDC HIV prevention cooperative agreements. A standardized data collection instrument is completed by individuals providing testing services and is submitted through the Enhanced HIV/AIDS Reporting System (eHARS). The eHARS is a browser-based HIV/AIDS surveillance system deployed at all state health departments.

The dataset provided for this study included the following variables: patient birth year, gender, race/ethnicity, previous testing history and result, and risk characteristics including sex with a female, sex with a male, sex with a partner with known HIV infection, sex with a partner who injects drugs, sex with a male who has sex with other males, injection drug use and sharing of drug injection equipment, and date of HIV testing, geographic location of testing, and test results.

Outcomes

The primary outcome was newly-diagnosed HIV infection, defined by a confirmed HIV-positive test result and the patient self-reporting having not previously tested positive for HIV infection. The secondary outcome was all confirmed HIV-positive test results, defined as a positive HIV test result (i.e., with either conventional or rapid EIA) with confirmation by supplemental testing (e.g., Western Blot).

Statistical Analyses

Statistical analyses were performed using SAS 9.3 (SAS Institute, Inc., Cary, NC). Observations were assigned scores according to the DHRS and categorized into the following 5 DHRS groups: <20 (very low risk); 20–29 (low risk); 30–39 (moderate risk); 40–49 (high risk); and ≥50 (very high risk). Proportions are reported as percentages with 95% confidence intervals (CIs). Calibration is reported as predicted versus observed HIV prevalence, and discrimination is reported as the area under the receiver operating characteristics (ROC) curve. Unit of analysis was testing event unless otherwise specified. Additional analyses were performed stratifying by geographic region as defined by the United States Census Bureau (Supplemental Digital Content, Appendix).²⁰ Best-case and worst-case sensitivity analyses were also performed to estimate the effect of missing data on complete-case results (Supplemental Digital Content, Appendix).

RESULTS

During the three-year period, 4,830,941 tests were reported with complete data, representing the principal cohort for our analyses. Of these, the median age was 28 years (IQR: 22 – 40 years), 50% were male, 46% were Black, 33% were white or other, and 21% were Hispanic; additionally, 30,080 (0.6%) were newly-diagnosed with HIV infection.

Table 1 shows the distribution of the DHRS variables by primary and secondary outcomes. The prevalence of newly-diagnosed HIV infection within each of the 5 DHRS risk groups were: 0.20% (95% CI: 0.19% – 0.20%) [n=856/432,674]; 0.17% (95% CI: 0.16% – 0.17%) [n=2,168/1,312,427]; 0.39% (95% CI: 0.38% – 0.40%) [n=7,771/2,003,857]; 1.19% (95% CI: 1.16% – 1.21%) [n=9,617/811,501]; and 3.57% (95% CI: 3.50% – 3.65%) [n=9,668/270,482], respectively (Figure 1). When all confirmed HIV infections were considered, the DHRS performed similarly.

Table 1.

Risk score variables for the complete case cohort (N=4,830,941), stratified by HIV diagnosis and test result, CDC PEMS data, 2008 – 2010.

	Newly-Diagnosed HIV Infection				Confirmed HIV Test Result
	Yes		No		Positive		Negative
Refined DHRS Variables	n=30,080		n=4,800,861		n=44,513		n=4,786,428
Age
<22 or >60 years	3,926	(13.1)	1,163,132	(24.2)	5,240	(11.8)	1,161,818	(24.3)
22–25 or 55–60 years	5,888	(19.6)	1,017,206	(21.1)	8,029	(18.0)	1,015,065	(21.2)
26–32 or 47–54 years	10,419	(34.6)	1,477,305	(30.8)	15,493	(34.8)	1,472,231	(30.8)
33–46 years	9,847	(32.7)	1,143,218	(23.8)	15,751	(35.4)	1,137,314	(23.8)
Gender
Female	6,735	(22.4)	2,395,949	(49.9)	11,130	(25.0)	2,391,554	(50.0)
Male	23,345	(77.6)	2,404,912	(50.1)	33,383	(75.0)	2,394,874	(50.0)
Race/Ethnicity
Black	16,680	(55.5)	2,215,625	(46.2)	24,502	(55.0)	2,207,803	(46.1)
Hispanic	6,020	(20.0)	997,308	(20.8)	9,121	(20.5)	994,207	(20.8)
Other* or White	7,380	(24.5)	1,587,928	(33.1)	10,890	(24.5)	1,584,418	(33.1)
Sexual Practices
Sex with a male	20,176	(67.1)	2,544,240	(53.0)	27,930	(62.8)	2,536,486	(53.0)
Other Risks
Injection drug use	1,744	(5.8)	184,738	(3.9)	2,658	(6.0)	183,824	(3.8)
Past HIV test	22,039	(73.3)	3,305,774	(68.9)	36,100	(81.1)	3,291,713	(68.8)

Abbreviations: DHRS = Denver HIV Risk Score; CDC = Centers for Disease Control and Prevention; HIV = human immunodeficiency virus; PEMS = Program Evaluation and Monitoring System.

Percentages do not always add to 100% due to rounding error.

^*Represents American or Alaskan Native, Native Hawaiian, or non-Hawaiian Pacific Islander.

Figure 1.

Prevalence of human immunodeficiency virus (HIV) infection within each risk score category in the validation sample, newly identified HIV infections and all HIV infections, CDC PEMS data, 2008–2010. The refined Denver HIV Risk Score ranges from −4 to +73. Bars, 95% confidence interval.

The top 3 risk groups (scores ≥30) represented only 63% (n=3,085,840/4,830,941) of the cohort, yet 90% (n=27,056/30,080) of newly-diagnosed HIV infections, whereas the top 2 risk groups (scores ≥40) represented only 22% (n=1,081,983/4,830,941) of the cohort, yet 64% (n=19,285/30,080) of newly-diagnosed HIV infections. The DHRS demonstrated excellent calibration (regression slope: 1.09) and good discrimination (area under ROC curve: 0.77) (Supplemental Digital Content, Appendix).

DISCUSSION

In a large national HIV testing cohort, the DHRS accurately categorized individuals into different HIV risk groups, and a DHRS ≥30 identified individuals at significantly increased risk of being HIV-infected. It also demonstrated increasing HIV prevalence as the DHRS increased with the highest risk group having a DHRS ≥50. Results were similar across all geographic regions of the United States, strongly supporting the validity and generalizability of the DHRS for predicting HIV risk.

To our knowledge, the DHRS is the only instrument to quantify a patients probability of being infected with HIV, and was developed to help clinicians identify patients with HIV infection.¹⁶ Following its development, however, it was clear broader external validation was needed. The results of this study mirror the performance of the DHRS when first tested in ED populations from Cincinnati, Ohio¹⁶ and Baltimore, Maryland,²² thus confirming its fidelity as a valid prediction tool.

The CDC recommends non-risk-based HIV screening for adolescents and adults based,³ in part, on the concern that risk-based screening is less effective.^3,23 In 2013, the USPSTF affirmed the CDC’s recommendation, not because comparative effectiveness of different screening methods exists, but because individuals diagnosed and linked into care benefit from treatment while simultaneously reducing viral transmission.⁴ We believe the DHRS may be a valuable tool to help achieve goals established by the National HIV/AIDS Strategy,² especially when used in environments where HIV screening resources are limited or when repeat testing is warranted.

According to the DHRS, individuals 26 – 54 years of age, male, and Black or Hispanic, regardless of sexual orientation, risk behaviors, or prior HIV testing, should be considered at increased risk for HIV infection (DHRS ≥30). Our results support the notion that certain demographic groups, regardless of risk behaviors, should be routinely tested for HIV infection. This is consistent with recent expanded testing efforts by the CDC to target communities with high disease burdens.²⁴ Conversely, the DHRS identifies groups at such low risk for HIV infection as to not necessitate routine testing. We identified a threshold of ≥30 as having the best compromise between sensitivity and specificity; however, using different DHRS thresholds in different settings to inform screening practices may be warranted depending on the population being served and the HIV testing resources available.

The DHRS may also contribute meaningfully to current or future HIV screening paradigms. Nontargeted screening has been most widely studied in EDs with results indicating that a large proportion (nearly 80%) of eligible patients do not complete HIV testing.⁶ The DHRS may serve as an important adjunct to nontargeted screening by helping to quantify the risk of HIV infection for both clinicians and patients, thus helping to establish joint decision-making regarding consent for testing. Further, high-risk patients who know their risk may be more likely to accept testing, especially when involved in discussions with their clinician. Moreover, as the proportion of undiagnosed individuals in the United States becomes smaller (<10%), nontargeted screening will likely become relatively less effective and efficient when compared to more focused screening methods. Finally, it remains unknown whether certain clinical venues, especially non-traditional HIV testing settings like EDs and urgent cares, will be able to routinely provide nontargeted HIV screening, or whether a targeted approach is better suited for these sites.^25–27

Risk-based testing has historically failed because it was rooted in subjective assessment and without standardization. The DHRS is a simple, structured tool that if used as part of routine HIV screening may help identify most patients with undiagnosed HIV infection while conserving scarce public health resources. People at greatest risk for HIV infection, including those who are economically disadvantaged, men who have sex with men, Blacks, and Latinos, are disproportionately under-insured and rely heavily on publically-funded settings or EDs for HIV testing.²⁹ Without an insurance payer, the burden of paying for HIV testing falls to the institution offering the test or the cost shifts to other patients with a payer source, a liability many administrators are reticent to take on.^29,30 Although prior research supports the notion that “universal HIV testing” is cost effective from a societal perspective,^31,32 this means relatively little to administrators, clinicians, or public health officials as they plan annual budgets and seek funding needed to pay for large numbers of tests. Until a clearer understanding of how funding will be provided for HIV testing, the DHRS may be used to prioritize how and when HIV testing is offered in public settings. Future research will require comparative and cost effectiveness evaluations of routine risk-based HIV screening using the DHRS to non-risk-based HIV screening.

This study has limitations. The data were not specifically collected for purposes of validating the DHRS; as such, data may have been misclassified or missing. Given the size of the dataset from a diverse range of HIV testing venues and results of sensitivity analyses (Supplemental Digital Content, Appendix), we believe the effects of misclassification or missingness were small. The primary outcome included subjective reports by individuals of a new diagnosis and it is possible that in some instances these reports were incorrect; we specifically included our secondary outcome to help assess the impact of potential misclassification of the primary outcome. Also, data are reported at the test-level and not the individual-level; as such, it was not possible to link the results of repeat testing events for the same person. However, the definition of a confirmed HIV-positive testing event minimized this limitation for persons who were newly-identified because records for which there was a current HIV-positive test result and a history of a previous HIV-positive test were excluded from the definition of this outcome. Finally, data included in this study represented HIV testing funded by the CDC. Besides the four states that specifically declined participation, other non-CDC-funded HIV testing occurred that was not captured in the dataset and thus may have contributed to selection bias. Given the large number of observations, however, we believe our results are generalizable to all geographic regions as well as HIV testing venues in the United States.

In summary, the DHRS accurately categorized patients into groups with different HIV prevalences, and serves as a simple tool for quantifying risk and identifying individuals for HIV screening. The DHRS may contribute substantially to future HIV screening efforts, especially as the number of those with undiagnosed infection declines.