Abstract
We previously developed an electronic medical record-based algorithm for identifying patients at risk for HIV in the emergency department (ED). The aim of this study was to evaluate the performance of the HIV risk algorithm for identifying cisgender women with a pre-exposure prophylaxis (PrEP) indication. To retrospectively evaluate the HIV risk algorithm, we identified cisgender women with HIV diagnosed in the ED and retrospectively calculated the HIV risk algorithm output. To prospectively validate the algorithm, we surveyed cisgender women seeking care in the ED regarding behavioral risks for HIV. We prospectively determined whether the algorithm identified them as PrEP candidates. In the retrospective evaluation, 9.4% (2/21) of women with incident HIV infection were identified as at risk for HIV by the algorithm. In the prospective evaluation, 24% (59/245) of women who completed the survey had a PrEP indication based on self-report of behavioral risk factors for HIV. The sensitivity of the algorithm for identifying cisgender female PrEP candidates was 10%, and the specificity was 96%. PrEP indications missed by the electronic algorithm included condomless sex in a high HIV prevalence area, multiple sex partners, male partners who have sex with men, and recent bacterial sexually transmitted infections diagnosed at outside clinics. An electronic algorithm to identify PrEP candidates in the ED has low sensitivity for identifying cisgender women with PrEP indications. More research is needed to identify electronic data that can improve the algorithm sensitivity among cisgender women.
Keywords: pre-exposure prophylaxis, women, electronic medical records, algorithms
Introduction
Pre-exposure prophylaxis (PrEP) is highly effective for preventing HIV acquisition, but only a fraction of the individuals who could benefit from PrEP are receiving it.1 Many HIV-negative individuals who are at increased risk for HIV access medical care but are not prescribed PrEP, resulting in missed opportunities.2 Medical providers fail to prescribe PrEP for a variety of reasons, including lack of knowledge regarding PrEP indications and limited time to assess HIV risk among their patients.3,4
To increase PrEP prescription in clinical settings, recent research has focused on developing automated systems for identifying PrEP candidates using electronic medical record (EMR) data.5,6 Such automated systems utilize EMR-based algorithms to predict individual patients' risk for HIV acquisition and PrEP eligibility. These algorithms can assist medical providers in identifying patients who would benefit from HIV risk assessment, HIV prevention counseling, and PrEP.
To date, EMR-based algorithms for identifying PrEP candidates have shown good accuracy for identifying men who would benefit from PrEP but have performed suboptimally among cisgender women.6,7 For example, a recently developed machine-learning HIV risk algorithm failed to identify any cases of incident HIV infection among women in a validation cohort.7 The discrepancy in the performance of these algorithms for men versus women is especially concerning given that women are disproportionately under-represented among PrEP users.8
We previously developed and implemented an EMR-based algorithm to identify patients in the emergency department (ED) who are at risk for HIV and would benefit from PrEP.9 However, in the first year of implementation of this electronic HIV risk algorithm, only 7.5% of patients identified by the algorithm as potential PrEP candidates were women. As women account for 20% of new HIV infections in the United States, the algorithm identified fewer women than expected. Therefore, the aim of this study was to evaluate the sensitivity and specificity of the HIV risk algorithm among cisgender women in an effort to inform revisions needed to enhance prediction for women.
Materials and Methods
At our institution (a large urban academic medical center), an electronic HIV risk algorithm is embedded in the EMR and automatically calculated for all patients at the ED triage. Development of the HIV risk algorithm has been previously described.9 Briefly, the algorithm was developed by performing HIV risk assessment for patients testing negative for HIV in the ED. We created a logistic regression model by using EMR data available at the time of triage to model the outcome of having a PrEP indication, as defined by the United States Public Health Services (USPHS).10 The model was converted into an electronic risk score (HIV risk score) and incorporated into the EMR. The HIV risk score includes the following EMR data elements: male sex (7 points), chief complaint related to sexually transmitted infection (STI)-associated symptoms (6 points), age ≤20 years (13 points), age 21–24 years (8 points), positive STI in the previous 6 months (21 points), and man who has sex with men (21 points). For patients with an HIV risk score ≥16, an electronic alert is generated that triggers HIV prevention counseling and PrEP linkage as needed.
Retrospective evaluation of electronic HIV risk score
To retrospectively evaluate the performance of the HIV risk score, we calculated the sensitivity of the score for identifying women with incident HIV infection. We retrospectively reviewed the charts of all cisgender women with a new positive HIV test in the ED between January 1, 2011 and April 30, 2018. We then determined how many women with incident HIV infection were flagged by the algorithm as potential PrEP candidates (i.e., HIV risk score ≥16). Because the goal of the HIV risk score is to identify individuals as PrEP candidates before HIV seroconversion, we calculated the HIV risk score for previous ED visits as well as the index ED visit where individuals were diagnosed with HIV.
Prospective evaluation of electronic HIV risk score
To prospectively evaluate the performance of the HIV risk score, we surveyed HIV-negative cisgender women seeking care in the ED regarding behavioral risk factors associated with HIV acquisition.11 The survey was conducted between May 7, 2018 and August 31, 2018 at the University of Chicago Medicine (UCM). Based on self-reported behaviors, we determined whether survey participants had a PrEP indication based on the 2017 USPHS summary guidance PrEP criteria (i.e., HIV-positive sexual partner, recent bacterial STI, high number of sex partners in the past 6 months, history of inconsistent or no condom use in the past 6 months, and/or commercial sex work).10,12
We then calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the electronic HIV risk score for identifying cisgender female PrEP candidates using the USPHS PrEP criteria as a gold standard.
This study was approved by the University of Chicago Institutional Review Board. For the retrospective evaluation, a waiver of consent was granted. For the prospective evaluation, informed consent was obtained.
Results
Retrospective evaluation
Fifty-one cisgender women tested positive for HIV in the ED during the study period. Of these, 21 (41.2%) patients were newly diagnosed with HIV. Among the 21women with incident HIV infection, the median age was 38 years (interquartile range, 29–47) and 95.2% (20/21) were Black. Only one (1/21, 4.7%) cisgender woman with incident HIV infection had an HIV risk score ≥16 during the ED visit when they were diagnosed with HIV. None of the women had a history of a positive STI in the previous 6 months documented in the EMR, and only 19% (4/21) presented to the ED with a chief complaint related to STI symptoms.
Overall, 33.3% (7/21) of women with incident HIV infection had visited the UCM ED before the visit when they tested positive for HIV. Among these women, only one had an HIV risk score ≥16 at a previous ED visit. Taking into account all ED visits, the HIV risk algorithm had a sensitivity of 9.5%, that is, 2 out of 21 (9.5%) of cisgender women with incident HIV infection were identified by the algorithm.
Prospective evaluation
Two hundred forty-five cisgender women completed the survey in the ED. Table 1 shows demographics of survey participants. Twenty-four percent (59/245) of survey participants met the USPHS criteria for PrEP indication. Ten percent (6/59) of participants with a PrEP indication had an HIV risk score ≥16 (Table 2). Based on the prospective evaluation, the HIV risk algorithm had a sensitivity of 10%, a specificity of 96%, PPV of 43%, and NPV of 77% (Table 2). PrEP indications among participants who were not detected by the HIV risk algorithm included history of inconsistent condom use in a high HIV prevalence area (86.8%, 46/53), multiple sex partners in the previous 6 months (67.9%, 36/53), self-reported recent bacterial STI (9.4%, 5/53), and a male sex partner who has sex with other men (3.8%, 2/53).
Table 1.
Demographics of Survey Participants
| Characteristic | n (%) |
|---|---|
| n = 245 | |
| Age (years) | |
| Median (IQR) | 29 (24–34) |
| Race/ethnicity (not mutually exclusive) | |
| Black/African American | 221 (90.2%) |
| White | 14 (5.7%) |
| Hispanic or Latina | 12 (4.9%) |
| Asian | 5 (2.0%) |
| American Indian/Alaska Native | 4 (1.6%) |
| Other | 12 (4.9%) |
| Education level | |
| Some high school | 17 (6.9%) |
| High school or GED | 87 (35.5%) |
| Some college | 98 (40.0%) |
| Bachelor's degree | 24 (9.8%) |
| Graduate degree | 16 (6.5%) |
| Missing | 3 (1.2%) |
| Insurance | |
| Private insurance | 50 (20.4%) |
| Public insurance | 120 (49.0%) |
| None | 30 (12.2%) |
| Don't know/missing | 75 (30.6%) |
| Presenting for STI testing or treatment | |
| Yes | 25 (10.2%) |
| No | 213 (86.9%) |
| Missing | 8 (3.2%) |
| Had syphilis or gonorrhea in the past 6 months | |
| Yes | 8 (3.3%) |
| No | 237 (96.7%) |
| Number of vaginal sex partners | |
| 0 | 46 (18.8%) |
| 1 | 141 (57.6%) |
| ≥2 | 54 (22.0%) |
| Missing | 4 (1.6%) |
| Number of anal sex partners | |
| 0 | 195 (79.6%) |
| 1 | 24 (9.8%) |
| ≥2 | 3 (1.2%) |
| Missing | 23 (9.4%) |
GED, graduate equivalency degree; IQR, interquartile range; STI, sexually transmitted infection.
Table 2.
Results of Prospective Evaluation of Electronic HIV Risk Score
| PrEP indication | No PrEP indication | Total | Characteristic | |
|---|---|---|---|---|
| EMR HIV risk alert | 6 | 8 | 14 | PPV = 43% (6/14) |
| No EMR HIV risk alert | 53 | 178 | 231 | NPV = 77% (178/231) |
| Total | 59 | 186 | 245 | |
| Characteristic | Sensitivity = 10% (6/59) | Specificity = 96% (178/186) |
EMR, electronic medical record; NPV, negative predictive value; PPV, positive predictive value; PrEP, pre-exposure prophylaxis.
Discussion
EMR-based algorithms have the potential to increase PrEP prescriptions for at-risk patients by helping medical providers identify patients with PrEP indications. However, we found in both retrospective and prospective evaluations that the EMR-based HIV risk algorithm utilized at our institution has low sensitivity for identifying cisgender women with a PrEP indication. The EMR algorithm we evaluated is a simple point-based risk score based on a logistic regression model. However, other more complex HIV risk assessment algorithms, including machine-learning algorithms, have also performed poorly among women. Marcus et al. developed a machine-learning model to predict incident HIV infection by using EMR data.7 Their model correctly identified nearly half of the incident HIV cases among males but failed to identify any incident HIV cases among females. Krakower et al. also developed a machine-learning model to predict incident HIV infection by using data from a large ambulatory group's EMR.6 Although the majority of patients seen by the group were women, there were very few incident cases of HIV infection among women in their dataset, limiting the performance of their model among women.
This gender gap in the accuracy of EMR algorithms for detecting PrEP indications is particularly concerning given that cisgender women make up a disproportionately low percentage of PrEP users compared with men.13 The CDC estimates that ∼180,000 women in the United States are eligible for PrEP.10,14,15 However, in 2018, <9000 women were prescribed PrEP; this number represents <5% of women with a clinical indication for PrEP.13 In 2018, the PrEP-to-need ratio (defined as number of PrEP prescriptions divided by number of new HIV diagnoses) for women was less than a third of that for men (1.6 vs. 5.7), indicating a substantial disparity in PrEP use among women compared with their need.1
Common PrEP indications missed by the HIV risk algorithm were inconsistent condom use in a high HIV-prevalence area, multiple sex partners, and sex with male partners who have sex with other men. These factors are not often documented in structured fields of the EMR that are readily utilized by an EMR-based algorithm. The EMRs could be designed to encourage documentation of these risk factors in structured fields. Natural language processing (NLP) of unstructured text of clinical notes in the EMR also may be able to detect these and other nuanced risk factors for HIV acquisition. Indeed, Feller et al. found that an algorithm utilizing both structured fields and NLP of unstructured clinical notes to predict risk for HIV acquisition was more accurate than an algorithm using structured EMR data alone.16 The HIV risk algorithm also failed to detect women with recent bacterial STIs not diagnosed in our health care system. Data sharing between EMRs from different health systems and/or public health departments could potentially improve sensitivity for detecting STIs diagnosed at outside institutions.
This study has several limitations. We used USPHS summary criteria for PrEP as the gold standard for PrEP indications. However, these criteria may exclude some cisgender women who are at risk for HIV and who would benefit from PrEP.12 It is possible that some of the women who received “false positive” EMR HIV risk alerts (i.e., women with HIV risk scores ≥16 who do not meet USPHS summary criteria for a PrEP indication) actually would benefit from PrEP. If this is the case, then the PPV of the algorithm may be higher. In addition, we examined characteristics of an EMR HIV risk algorithm utilized in the ED at our institution. The algorithm was implemented in the ED because individuals who receive ED care are often disproportionately at risk for HIV and may have limited access to other health care settings where they could receive PrEP.17 However, our findings may not be generalizable for other alerts or in other care settings.
Conclusions
In summary, we found that an electronic algorithm to identify patients in the ED at risk for HIV who may benefit from PrEP had low sensitivity but high specificity for identifying cisgender women with a PrEP indication. More research is needed to identify additional EMR data elements that can improve the sensitivity of EMR-based algorithms for identifying cisgender women who would benefit from PrEP.
Acknowledgments
The authors thank the study participants who generously shared their time and thoughts.
Authors' Contributions
J.P.R. and L.R.H. conceived of the study and obtained funding. J.S., R.N.B., A.K.J., E.E.F., J.P.R., and L.R.H. participated in study design. A.B., M.C., and R.N.B. collected data. E.E.F. performed data analysis. J.P.R., E.E.F., A.K.J., and L.R.H. interpreted results. J.P.R. drafted the article, and all other authors critically revised the article for important intellectual content.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
This work was supported by an administrative supplement to the Third Coast Center for AIDS Research (CFAR), an NIH-funded program (P30 AI117943).
References
- 1. Siegler AJ, Mehta CC, Mouhanna F, et al. . Policy- and county-level associations with HIV pre-exposure prophylaxis use, the United States, 2018. Ann Epidemiol 2020;45:24–31.e3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Smith DK, Chang MH, Duffus WA, Okoye S, Weissman S. Missed opportunities to prescribe preexposure prophylaxis in South Carolina, 2013–2016. Clin Infect Dis 2019;68:37–42 [DOI] [PubMed] [Google Scholar]
- 3. St. Vil NM, Przybyla S, LaValley S. Barriers and facilitators to initiating PrEP conversations: Perspectives and experiences of health care providers. J HIV AIDS Soc Serv 2019;18:166–179 [Google Scholar]
- 4. Petroll AE, Walsh JL, Owczarzak JL, McAuliffe TL, Bogart LM, Kelly JA. PrEP Awareness, familiarity, comfort, and prescribing experience among US primary care providers and HIV specialists. AIDS Behav 2017;21:1256–1267 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Marcus JL, Sewell WC, Balzer LB, Krakower DS. Artificial intelligence and machine learning for HIV prevention: Emerging approaches to ending the epidemic. Curr HIV/AIDS Rep 2020;17:171–179 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Krakower DS, Gruber S, Hsu K, et al. . Development and validation of an automated HIV prediction algorithm to identify candidates for pre-exposure prophylaxis: A modelling study. Lancet HIV 2019;6:e696–e704 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Marcus JL, Hurley LB, Krakower DS, Alexeeff S, Silverberg MJ, Volk JE. Use of electronic health record data and machine learning to identify candidates for HIV pre-exposure prophylaxis: A modelling study. Lancet HIV 2019;6:e688–e695 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Siegler AJ, Mouhanna F, Giler RM, et al. . The prevalence of pre-exposure prophylaxis use and the pre-exposure prophylaxis-to-need ratio in the fourth quarter of 2017, United States. Ann Epidemiol 2018;28:841–849 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Ridgway JP, Almirol EA, Bender A, et al. . Which patients in the emergency department should receive preexposure prophylaxis? Implementation of a predictive analytics approach. AIDS Patient Care STDs 2018;32:202–207 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Centers for Disease Control and Prevention. US Public Health Service: Preexposure Prophylaxis for the Prevention of HIV Infection in the United States—2017 Update: A Clinical Practice Guideline. 2017. Available at: https://www.cdc.gov/hiv/pdf/risk/prep/cdc-hiv-prep-guidelines-2017.pdf (Last accessed October15, 2020).
- 11. Hirschhorn LR, Brown RN, Friedman EE, et al. . Black cisgender women's PrEP knowledge, attitudes, preferences, and experience in Chicago. J Acquir Immune Defic Syndr 2020;84:497–507 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Calabrese SK, Willie TC, Galvao RW, et al. . Current US guidelines for prescribing HIV pre-exposure prophylaxis (PrEP) disqualify many women who are at risk and motivated to use PrEP. J Acquir Immune Defic Syndr 2019;81:395–405 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. AIDSVu: Sullivan PS, Woodyatt C, Koski C, et al. A data visualization and dissemination resource to support HIV prevention and care at the local level: analysis and uses of the AIDSVu public data resource. J Med Internet Res 2020;22:e23173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Smith D, Van Handel M, Wolitski R, et al. . Vital signs: Estimated percentages and numbers of adults with indications for preexposure prophylaxis to prevent HIV acquisition—United States, 2015. MMWR Morb Mortal Wkly Rep 2015;64:1291–1295 [DOI] [PubMed] [Google Scholar]
- 15. Smith DK, Van Handel M, Grey J. Estimates of adults with indications for HIV pre-exposure prophylaxis by jurisdiction, transmission risk group, and race/ethnicity, United States, 2015. Ann Epidemiol 2018;28:850–857.e9. [DOI] [PubMed] [Google Scholar]
- 16. Feller DJ, Zucker J, Yin MT, Gordon P, Elhadad N. Using clinical notes and natural language processing for automated HIV risk assessment. J Acquir Immune Defic Syndr 2018;77:160–166 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Hansoti B, Kelen GD, Quinn TC, et al. . A systematic review of emergency department based HIV testing and linkage to care initiatives in low resource settings. PLoS One 2017;12:e0187443. [DOI] [PMC free article] [PubMed] [Google Scholar]