Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Mar 1.
Published in final edited form as: AIDS. 2020 Mar 1;34(3):447–458. doi: 10.1097/QAD.0000000000002455

The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six U.S. cities.

Emanuel KREBS 1, Xiao ZANG 1,2, Benjamin ENNS 1, Jeong E MIN 1, Czarina N BEHRENDS 3, Carlos DEL RIO 4, Julia C DOMBROWSKI 5, Daniel J FEASTER 6, Kelly A GEBO 7, Matthew GOLDEN 5, Brandon DL MARSHALL 8, Lisa R METSCH 9, Bruce R SCHACKMAN 3, Steven SHOPTAW 10, Steffanie A STRATHDEE 11, Bohdan NOSYK 1,2, localized economic modeling study group
PMCID: PMC7046093  NIHMSID: NIHMS1066367  PMID: 31794521

Abstract

Objective:

Effective interventions to reduce the public health burden of HIV/AIDS can vary in their ability to deliver value at different levels of scale and in different epidemiological contexts. Our objective was to determine the cost-effectiveness of HIV treatment and prevention interventions implemented at previously-documented scales of delivery in six US cities with diverse HIV microepidemics.

Design:

Dynamic HIV transmission model-based cost-effectiveness analysis.

Methods:

We identified and estimated previously-documented scale of delivery and costs for 16 evidence-based interventions from the US CDC’s Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. Using a model calibrated for Atlanta, Baltimore, Los Angeles, Miami, New York City and Seattle, we estimated averted HIV infections, quality-adjusted life years (QALY) gained and incremental cost-effectiveness ratios (healthcare perspective; 3% discount rate, 2018$US), for each intervention and city (10-year implementation) compared to the status quo over a 20-year time horizon.

Results:

Increased HIV testing was cost-saving or cost-effective across cities. Targeted preexposure prophylaxis for high-risk men who have sex with men was cost-saving in Miami and cost-effective in Atlanta ($6,123/QALY), Baltimore ($18,333/QALY) and Los Angeles ($86,117/QALY). Interventions designed to improve antiretroviral therapy initiation provided greater value than other treatment engagement interventions. No single intervention was projected to reduce HIV incidence by more than 10.1% in any city.

Conclusions:

Combination implementation strategies should be tailored to local epidemiological contexts to provide the most value. Complementary strategies addressing factors hindering access to HIV care will be necessary to meet targets for HIV elimination in the US.

Keywords: HIV, localized HIV microepidemics, interventions, implementation, costeffectiveness, dynamic HIV transmission model

Introduction

The President of the United States recently announced the intention to eliminate the domestic HIV epidemic within 10 years[1]. To achieve this ambitious goal, healthcare providers and public health departments will need to overcome political, legal and structural barriers, and make efficient use of current and future funding[2]. A number of efficacious biomedical, behavioral and structural interventions are available; however, there is a paucity of evidence on real-world implementation of many of these interventions[3], including the population base reached, their adoption across diverse care delivery settings and how long they are sustained[4].

This paucity of evidence challenges decisions on how interventions should be implemented to make the best use of available funding[4], which are further complicated by the fact that the HIV epidemic in the United States is a collection of distinct regional microepidemics, dispersed predominantly across large urban centers[5]. Anderson et al. (2014) demonstrated that a regionally-focused public health response to HIV can provide substantially greater public health benefits compared to a uniform, national strategy for the same investment level[6]. The heterogeneity of HIV microepidemics across the United States suggests that focused, locally-oriented strategies in treating and preventing HIV will be required to end the HIV epidemic.

More than ever, simulation modeling is playing a critical role in priority setting for HIV treatment and prevention[7]. Dynamic HIV transmission models can provide a unified framework to quantify the health and economic value of different strategies to address the HIV epidemic while accounting for microepidemic context and the synergistic effects of different combinations of public health interventions[8]. The context in which healthcare services are delivered can influence the cost-effectiveness of interventions[9] and dynamic HIV transmission models using best-available localized data that capture the heterogeneity across settings are uniquely positioned to offer guidance on contextually efficient strategies to implement[10].

Ending the HIV epidemic will require an understanding of the population-level impact of HIV interventions, as they may vary substantially in their ability to deliver value at different levels of scale and in different microepidemics. Our objective was to determine the cost-effectiveness of HIV treatment and prevention interventions, offered at previously documented levels of scale in six US cities with diverse HIV microepidemics.

Methods

Model Description

We adapted and calibrated a previously published dynamic, compartmental HIV transmission model[11, 12] to replicate the city-level HIV microepidemic for six US cities: Atlanta; Baltimore; Los Angeles (LA); Miami (Dade County); New York City (NYC); and Seattle (King County). The model tracked individuals susceptible to HIV infection through the course of infection, diagnosis, treatment with antiretroviral therapy (ART) and ART dropout. In each city, the adult population 15–64 was partitioned by sex at birth, HIV risk group (men who have sex with men [MSM], people who inject drugs [PWID], MSM who inject drugs [MWID] and heterosexuals [HET]), race/ethnicity (black/African American, Hispanic/Latinx and non-Hispanic white/others) and sexual risk behavior level (high- vs. low-risk). HIV transmission occurred through heterosexual contact, homosexual contact, and the sharing of injection equipment. We assumed assortative and proportional sexual partnership mixing by race/ethnicity and sexual risk behavior level, respectively [1316]. City-specific sexual risk behaviors by race/ethnicity were derived from the US Centers for Disease Control and Prevention’s National Behavioral Health Survey (NHBS) [17, 18] for high-risk MSM and MWID and region-specific estimates by race/ethnicity from the National Survey of Family Growth were used for the other subgroups [19]. We also used city-specific injection risk behavior evidence from NHBS and assumed proportional mixing among PWID (i.e., individuals who share many injections were more likely to select a partner who also shares many injections). Consistent with other dynamic transmission models, the rate of HIV transmission was dependent on the distribution of people living with HIV across states of care engagement and disease progression and was also impacted by receipt of ART [2022], preexposure prophylaxis (PrEP)[23] and medication for opioid use disorder (MOUD)[24], access to syringe service programs (SSP)[25] and the decreased number of sexual partners following diagnosis[26].

We derived MSM population estimates by multiplying city-level male population estimates from census data by county- or CBSA-specific MSM proportions, and the size of the PWID population by multiplying race/ethnicity-stratified total population numbers by the most recent gender-weighted, race/ethnicity-specific prevalence estimates for each city [13, 27]. Given the uncertainty in population sizes for MWID, we derived population estimates by taking the average of the proportion of MSM that inject drugs and the proportion of male PWID that have sex with men[13, 1618, 28]. Finally, based on the best available evidence, we assumed that 72.7% of PWID and MWID had an opioid use disorder[29].

The model also captured heterogeneity across risk and ethnic groups in maturation (e.g., rates at which individuals age out of the model) and mortality and the disparities in accessing health and prevention services, including HIV testing, ART, SSP, MOUD and targeted PrEP for high-risk MSM. Our evidence synthesis[13] and calibration process are documented elsewhere[16].

Model calibration and validation

For each city, we calibrated the model to match HIV prevalence, new diagnoses and deaths (2012–2015), stratified by sex, race/ethnicity, and HIV risk group (17 targets total), and validated against external incidence estimates[16]. We projected HIV microepidemic trajectories accounting for official population growth and demographic shifts in each city over a 20-year time horizon (2020–2040) to serve as the basis of comparison for individual interventions[30]. In the projections, all health services were held at their 2015 levels except for PrEP which was held at 2017 levels to account for its recent rapid growth in uptake among MSM.

Interventions Assessed

Evidence-based interventions were selected from the US Centers for Disease Control and Prevention ‘Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention’[31] and from the recently published literature. We included interventions with established effectiveness data and promising scalability within four specific domains: HIV prevention programs (SSP, MOUD and PrEP); HIV testing; ART engagement (ART initiation and retention); and ART re-engagement (re-initiation). Finally, we ranked the quality of the evidence informing each intervention using the Oxford Centre for Evidence-based Medicine – Levels of Evidence scale[32].

We used the Reach Effectiveness Adoption Implementation Maintenance framework[33] to define four components for the implementation of each intervention: (i) scale of delivery; (ii) population-level impact; (iii) period over which the intervention delivery is sustained; and (iv) costs of implementation, delivery and sustainment. Supplemental Table 1 http://links.lww.com/QAD/B592 describes definitions and assumptions.

Scale

Scale refers to the proportion of a target population that is provided with an intervention. We defined the scale of delivery for each HIV prevention programs as the annual rate of expanded access, or additional scale-up, estimated using the best-available program-specific evidence. In contrast, the scale of delivery for HIV testing and care interventions was defined as the product of setting-specific reach and adoption for each intervention i, target population j and healthcare setting k,

Scaleij=Reachijk×Adoptionik.

where reach is defined as the participation rate in a given intervention, conditional on: (a) the probability an individual will access services in setting k and (b) the probability the individual will accept the intervention being delivered. Adoption is defined as the proportion of a healthcare setting that actually delivers the intervention. Consequently, a variety of combinations of reach and adoption can result in a given scale of delivery. We focused our attention on identifying the best publicly-available data to inform the scale of delivery range for each intervention i, in target population j, and healthcare setting k.

Population Impact

The population-level impact of HIV prevention programs enters the model by reducing the probability of HIV transmission. Specifically, MOUD and SSP reduce the number of risky injections with shared equipment and targeted PrEP among high-risk MSM reduces the probability of HIV acquisition via both sexual contact and the sharing of injection equipment[16].

In contrast, each HIV testing and care intervention affects model parameters dictating transitions between health states (compartments). The population-level impact is defined as the product of the scale of delivery and the effectiveness of the intervention,

Population-level Impacti=Scaleij×Effectivenessi.

We assumed the impact for all interventions to be additive to existing service levels for each city[30].

Implementation and Sustainment

We defined the implementation period as an 18-month linear scale-up from status quo service levels up to the scale of delivery defined for each intervention. Further, we assumed proportional scale-up across risk and ethnic groups, implying higher scale of delivery following implementation for groups receiving greater service levels at baseline, thus accounting for underlying structural barriers to healthcare access. The population impact of an intervention was then held constant throughout the sustainment period, reaching 10 years to match the goals of the “End the HIV Epidemic” initiative[1].

Intervention Costs

The costs attributable to each intervention included costs of implementation, delivery and sustainment. If applicable, implementation and sustainment costs accrued during the first 18 months and the following period up to 10 years, respectively, and delivery costs accrued throughout the 10-year delivery period. We derived cost components using assumptions about personnel caseloads and patient volumes specific to each healthcare setting.

The costs attributable to implementation were specific to the setting in which the intervention was delivered, as were sustainment costs. The costs of delivery were specific to each intervention as were direct material costs, if applicable. Lastly, we assumed that costs of implementation and sustainment were constant across different scales of delivery and we assumed no public health intervention costs for the status quo.

Evidence Verification

We validated the selection of evidence informing interventions and implementation modeling via a two-part survey delivered to a scientific advisory committee comprised of local content experts from each city. Committee members were asked to (i) rate their confidence in the proposed interventions; (ii) identify additional interventions being considered by public health departments; (iii) rate their confidence in the evidence sources used to derive the scale of delivery, and (iv) identify additional sources to estimate the scale of delivery. Committee members were then asked to rate their confidence in the ranges for reach, adoption and scale used for the implementation of each intervention. Mean responses were synthesized and used to inform final scale of delivery ranges.

Detailed information on the sources, assumptions and calculations used to derive the reach, adoption and scale of delivery as well as the evidence verification process are provided in the supplemental material., http://links.lww.com/QAD/B592.

Cost-effectiveness analysis

Model-projected outcomes included quality-adjusted life-years (QALYs), total costs (2018 USD) and new HIV infections. We considered a 20-year time horizon (2020–2040) to capture the long-term individual benefits of ART and 2nd-order transmission effects. The analysis conformed to best practice guidelines on cost-effectiveness analyses and was conducted from the healthcare sector perspective[34]. Both costs and QALYs were reported using a 3% annual discount rate[35]. We estimated incremental cost-effectiveness ratios (ICERs) as the incremental cost per QALY gained for each individual intervention compared to the status quo scenario maintaining current service levels. Although no explicit threshold exists in the US[34], we defined cost-effective interventions as those with an ICER below $100,000/QALY. We indicated interventions as cost-saving in instances where projected costs were lower and effectiveness (measured as QALYs) was higher compared to the status quo.

We performed probabilistic sensitivity analysis to evaluate the extent of parameter uncertainty for each intervention. For each city, we used the 2,000 best-fitting calibrated parameter sets from 10,000 calibration runs, sampling all non-calibrated parameters simultaneously from distributions that were previously developed for each model parameter[13].

Results

We included three HIV prevention programs and 10 HIV testing, ART engagement and ART re-engagement interventions in our analysis (Table 1). Evidence for the majority of the interventions (n=7) was derived from high-quality randomized control trials. We synthesized evidence from 11 peer-reviewed publications, 12 public health and surveillance reports and 3 publicly available data sets to estimate ranges for the scale of delivery for each intervention (Table 2, Fig. 1). We considered expanded access to MOUD for both buprenorphine and methadone, opt-out HIV testing in both primary care and in emergency departments and care coordination for increased ART retention among all individuals and targeted to individuals with CD4<200 cells per μL. As a result, our analysis included the implementation of 16 interventions.

Table 1.

HIV prevention programs and care interventions selected for inclusion in our analysis.

Intervention Description Source Effectiveness (95% CI) Study Setting Target Population Study Design Study Size Study Location Evidence Level*
HIV prevention programs
Syringe service program (SSP) Reduced risk of HIV infection with increased access to clean injection equipment. Aspinall et al. 2014 Int J Epi 0.42 (0.22, 0.81) SSP PWID Meta-analysis n = 6 USA: 3; Other: 3 2a
Expanded access to medication for opioid use disorder (MOUD) Reduced risk of HIV infection with decreased number of injections. MacArthur et al. 2012 BMJ 0.46 (0.32, 0.67) Primary Care & OTP PWID Meta-analysis n = 9 USA: 4; Other: 5 2a
Expanded access to full-time pre-exposure prophylaxis (PrEP) Reduced risk of HIV infection associated with protective level adherence (≥4 doses/week). Liu 2016 JAMA Intern Med 0.60 (0.56, 0.62) Primary Care High-risk MSM Cohort study (DEMO) n = 557 MIA; SF; DC 1b
HIV Testing
Opt-out testing Conduct a routine rapid HIV test unless explicitly declined. Montoya et al. 2016 The BMJ 1.28 (1.24, 1.31) Hospital General RCT n = 4,800 SF General Hospital 1b
EMR testing offer reminder Integrate the routine offering of testing to patients with no EMR record of HIV test or at high-risk. Felsen et al. 2017 JAIDS 2.78 (2.62, 2.96) Hospital General Quasi-exp. pre/post n = 55,553 Bronx, NY 2b
Nurse-initiated rapid testing Increase testing with nurse-initiated screening and rapid HIV testing. Anaya et al. 2008 J Gen Intern Med 5.27 (3.11, 8.90) Primary Care General RCT n = 251 V.A. Healthcare System 2b
Integrated rapid testing Increase testing with on-site rapid HIV testing in drug treatment centers. Metsch et al. 2012 Am J Pub H 4.52 (3.57, 5.72) DTP PWID RCT (CTN 0032) n = 1,281 12 US communities 1b
ART engagement
Individual case management for ART initiation Increase ART initiation with up to five contacts with a case manager over a 90-day period. Gardner et al. 2005 AIDS 1.41 (1.10, 1.60) HIV clinics General RCT (ARTAS) n = 316 ATL; BAL; LA; MIA 1b
Individual care coordination for ART retention Increase ART retention with comprehensive care coordination program. Robertson et al. 2018 Am J Epi 1.10 (1.07, 1.13) HIV clinics General Pre/post^ n = 6,812 New York City 2b
EMR alert of suboptimal ART engagement Decrease ART drop-out with interactive alerts providing clinical information and mechanism for follow-up appointments and tests. Robbins et al. 2012 Ann Int Med 0.69 (0.53, 0.90) HIV clinics General RCT (FastTrack) n = 1,011 Massachusetts General Hospital 1b
RAPID ART initiation Immediate ART initiation with multicomponent intervention for recent or late diagnoses. Pilcher et al. 2017 JAIDS 1.32 (1.23, 1.54)** HIV clinics General Cohort study n = 86 SF General Hospital 3b
ART re-engagement
Enhanced personal contact Increase ART re-initiation with face-to-face meeting and continuous contact over the year. Gardner et al. 2014 Clin Infect Dis 1.22 (1.09, 1.36) HIV clinics General RCT n = 1,838 6 US cities 1b
Re-linkage program Increase ART re-initiation with outreach intervention using clinical and surveillance data. Bove et al. 2015 JAIDS 1.70 (1.20, 2.30) HIV clinics General Cohort study n = 753 Seattle, WA 2b

PWID: Pepople who inject drugs; MSM: Men who have sex with men; PrEP: Pre-exposure prophylaxis; EMR: Electronic medical records; V.A.: US Department of Veteran’s Affairs;

RAPID: Rapid ART Program for Individuals with an HIV Diagnosis; ATL: Atlanta; BAL: Baltimore; LA: Los Angeles; MIA: Miami; SF: San Francisco, WA: Washington State; DC: Washington, D.C.

*

Levels of evidence adapted from Oxford Centre for Evidence-based Medicine – Levels of Evidence: 1a - Systematic review of RCTs; 1b - Individual high-quality RCT; 2a - Systematic review of cohort studies; 2b - Individual cohort study or quasi-experimental study; 3a - Systematic review of case-control studies; 3b - Individual case-control study; 4 - Case series; 5-Expert opinion.

Effectiveness defined as efficacy for 4 doses/week [96% (90%, 99%); Anderson 2012 Sci Transl Med] X protective level adherence [62.5% (associated with taking ≥4 doses/week); Liu 2016 JAMA Int Med]

^

Study with contemporaneous surveillance registry–based comparison group

**

Range does not represent the 95% confidence interval (CI); derivation of range values are detailed in the supplement.

Table 2.

Description of the implementation, reach, adoption, and scale of delivery of HIV prevention programs and HIV testing and care interventions.

Intervention Implementation Reach* (Setting-specific) Reach Adoption (Setting-specific) Adoption Scale of Delivery** (95% CI)
HIV prevention programs Expanded access
Syringe service program (SSP) Expanded access to clean injection equipment for the prevention of parenteral HIV transmission for people who inject drugs (PWID). - - - - - - - - 11% (7%, 15%)
Medication for opioid use disorder (MOUD) with buprenorphine Expanded access to office-based medication for opioid use disorder with buprenorphine for PWID. - - - - - - - - 14% (10%, 17%)
Medication for opioid use disorder (MOUD) with methadone Expanded access to opioid treatment program-based medication for opioid use disorder with methadone for PWID. - - - - - - - - 12% (2%, 17%)
Targeted PrEP for high-risk MSM and MWID Expanded access to pre-exposure prophylaxis (PrEP) targeted to high-risk men who have sex with men (MSM) and high-risk MSM who inject drugs (MWID)§. - - - - - - - - 68% (55%, 80%)
HIV Testing Reach X Adoption
Opt-out HIV testing in ER Individuals visiting an hospital emergency room (ER) receive a routine HIV test unless they decline. % with ER visit L12M^^ 11%−29% % visits with testing 19%−26% 3%−6% (2%, 7%)
Opt-out HIV testing in primary care Individuals visiting a primary care provider receive a routine HIV test unless they decline. % seeing Dr. L12M^ 59%−94% % visits with testing 32%−54% 25%−40% (19%, 51%)
EMR testing offer reminder Individuals visiting an hospital emergency room with a certified EMR are routinely offered an HIV test. % with ER visit L12M^^ 11%−29% % with certified EMR 97%−100% 11%−29% (9%, 36%)
Nurse-initiated rapid testing Individuals visiting a physician or health care professional are routinely offered an HIV test. % seeing Dr. L12M^ 10%−16% % visits with testing 32%−54% 4%−7% (3%, 9%)
MOUD integrated rapid testing PWID receiving medication for opioid use disorder are offered on-site rapid HIV testing from their prescriber. % accepting intervention 54% % prescribers implementing 14%−40% 17% (8%, 22%)
ART engagement
Individual case management for ART initiation HIV clinics offer the intervention to ART-naive people living with HIV (PLHIV) to increase the probability of ART initiation. % accepting intervention 86% % clinics implementing 60%−71% 57% (52%, 62%)
Individual care coordination for ART retention HIV clinics offer the intervention to eligible PLHIV currently receiving ART to reduce the probability of ART drop-out. % PLHIV eligible for RWHAP 38%−75% % RWHAP clinics offering intervention 20%−33% 10%−20% (8%, 25%)
Individual care coordination for ART retention, targeted HIV clinics offer the intervention to PLHIV with CD4<200 cells per^L currently receiving ART to reduce the probability of ART drop-out. % initiating ART with CD4<200 46%−70% % clinics implementing 60%−71% 30%−46% (27%, 50%)
EMR alert of suboptimal ART engagement PLHIV receiving ART are reminded of care engagement information to reduce the probability of ART drop-out when visiting their physician. % receiving HIV care L12M 60%−91% % with certified EMR 69%−86% 42%−78% (29%, 84%)
RAPID ART initiation HIV clinics offer the intervention for same-day ART initiation to newly diagnosed individuals to increase the probability of immediate ART initiation. % PLHIV linked to care 46%−93% % clinics implementing 60%−71% 30%−61% (27%, 66%)
ART re-engagement
Enhanced personal contact HIV clinics offer the intervention to individuals who have imperfect ART engagement to increase the probability of ART re-initiation. % successfully enrolled 69% % clinics implementing 60%−71% 45% (41%, 49%)
Re-linkage program HIV clinics implement the program and successfully contact a proportion who have dropped-out of ART to increase the probability of ART re-initiation. % successfully contacted 24% % RWHAP funded clinics 29%−40% 8% (7%, 10%)
*

ART: antiretroviral therapy; EMR: Electronic medical records; MOUD: Medication for opioid use disorder. Reach also includes proportion of patients accepting, when applicable.

**

Scale of delivery is defined as Reach X Adoption, with the exception of HIV prevention programs where it is defined as the annual rate of expanded access.

^

Any visits to a medical doctor or health professional stratified by gender, race/ethnicity, and region.

^^

Any visit to a hospital emergency room stratified by gender, race/ethnicity, and region.

State-specific.

Stratified by gender, race/ethnicity, and region.

§

We assume that 25% of MSM are indicated for PrEP in accordance with CDC guidelines.

Fig. 1. Previously-documented rate of annual expanded access for HIV prevention programs and previously-documented scale of delivery for HIV testing and care interventions.

Fig. 1.

* Targeted PrEP for high-risk men who have sex with men (MSM) and high-risk MSM who inject drugs (MWID).

ART: Antiretroviral therapy; SSP: Syringe service program; PrEP: pre-exposure prophylaxis; MOUD: Medication for opioid use disorder; BUP: Buprenorphine; ER: Emergency Department; PC: Primary care; EMR: Electronic medical records.

Levels of evidence adapted from Oxford Centre for Evidence-based Medicine – Levels of: 1a - Systematic review of RCTs; 1b - Individual high-quality RCT; 2a - Systematic review of cohort studies; 2b - Individual cohort study or quasi-experimental study; 3a - Systematic review of case-control studies; 3b - Individual case-control study; 4 - Case series; 5-Expert opinion.

Among HIV prevention programs, we found expanding SSP above their current levels to prevent HIV transmission to be cost-saving in Miami and cost-effective in Atlanta and LA (Fig 2). Expanded access to MOUD was cost-effective across all cities, ranging from $20,173/QALY for MOUD with methadone in Miami to $40,916/QALY for office-based MOUD with buprenorphine in NYC. Expansion of targeted PrEP for high-risk MSM and MWID was found to be cost-saving in Miami and cost-effective in Atlanta, Baltimore and LA.

Fig. 2. Incremental costs, QALYs and incremental cost-effectiveness ratios (ICER) resulting from previously-documented expanded access to HIV prevention programs and HIV testing and care interventions implemented at previously- documented scale of delivery in six U.S. cities.

Fig. 2.

Results presented are for the 2020–2040 study period with expanded access to HIV prevention programs and implementation of HIV testing and care interventions sustained over a 10-year period (mean of the 2,000 run conducted in the probabilistic sensitivity analysis; 95% credible intervals are presented in Supplemental Table 8 http://links.lww.com/QAD/B592). The vertical gridline in the 3rd row indicates the $100,000/QALY threshold for interventions to be considered cost-effective. CS: Cost-saving; QALYs: quality-adjusted life-years; ART: Antiretroviral therapy; SSP: Syringe service program; PrEP: pre-exposure prophylaxis; MOUD: Medication for opioid use disorder; BUP: Buprenorphine; ER: Emergency Department; PC: Primary care; EMR: Electronic medical records; ART initiation: Case management intervention to increase ART initiation; ART retention: Care coordination to increase ART retention; EMR alert: EMR alert of suboptimal ART engagement.

* Targeted PrEP for high-risk men who have sex with men (MSM) and high-risk MSM who inject drugs (MWID).

Expanding HIV testing interventions, including MOUD integrated rapid testing, was found to be cost-saving or cost-effective in every city. Total incremental cost savings for general population HIV testing interventions ranged from $1.2 million for opt-out testing in the emergency departments in Baltimore, to $235.6 million saved for electronic medical record testing reminders in LA over the 20-year study time horizon.

Interventions designed to improve ART initiation provided greater value within each city, with few exceptions, compared to either ART engagement interventions to prevent drop-out or ART re-engagement interventions. Increasing both case management to improve ART initiation and RAPID ART resulted in ICERs below the $100,000/QALY threshold in all cities besides Atlanta; however, the ART initiation intervention provided the most value in Atlanta among ART engagement and re-engagement interventions ($106,509/QALY) as it did in LA ($55,495/QALY), Miami ($61,159/QALY) and NYC ($31,696/QALY). Comparatively, RAPID ART provided the most value in Baltimore ($23,926/QALY) and Seattle ($65,340/QALY). The scale-up of the ART retention intervention targeted to individuals with CD4<200 was cost-effective in Baltimore ($80,028/QALY), LA ($61,493/QALY), Miami ($84,559/QALY) and NYC ($69,266/QALY), and the electronic medical record reminder for suboptimal ART engagement was cost-effective in LA ($70,473/QALY), Miami ($92,948/QALY), NYC ($86,785/QALY) and Seattle ($96,551/QALY). Finally, ART re-engagement interventions were cost-effective at a $100,000/QALY threshold in all cities but Atlanta.

We estimated that the maximum incidence reduction over 20 years compared to the status quo would result from enhancing general population HIV testing in all cities but Miami where expanded access to targeted PrEP for high-risk MSM and MWID would have the greatest impact on the epidemic (Fig. 3). Specifically, the electronic medical record testing reminder would reduce incidence the most in Atlanta (7.6%), LA (6.6%), NYC (7.8%) and Seattle (7.6%), and both nurse-initiated testing and targeted PrEP would reduce incidence equivalently in Baltimore (10.0%). Targeted PrEP would result in the greatest incidence reduction for Miami (10.1%), and relatively large reductions in Atlanta (6.0%), LA (3.4%), NYC (7.5%), and Seattle (5.3%). Expanding SSP and MOUD had relatively small population-level impacts on the percentage of averted infections across cities. Detailed results from the probabilistic sensitivity analysis can be found in Supplemental Table 8. http://links.lww.com/QAD/B592

Fig. 3. Percentage of total averted infections resulting from previously-documented expanded access to HIV prevention programs and HIV testing and care interventions implemented at previously-documented scale of delivery in six U.S. cities.

Fig. 3.

Results presented are for the 2020–2040 study period with expanded access to HIV prevention programs and implementation of HIV testing and care interventions sustained over a 10-year period (mean of the 2,000 run conducted in the probabilistic sensitivity analysis; 95% credible intervals are presented in the Supplemental Table 8, http://links.lww.com/QAD/B592). QALYs: quality-adjusted life-years; ART: Antiretroviral therapy; SSP: Syringe service program; PrEP: pre-exposure prophylaxis; MOUD: Medication for opioid use disorder; BUP: Buprenorphine; ER: Emergency Department; PC: Primary care; EMR: Electronic medical records; ART initiation: Case management intervention to increase ART initiation; ART retention: Care coordination to increase ART retention; EMR alert: EMR alert of suboptimal ART engagement.

* Targeted PrEP for high-risk men who have sex with men (MSM) and high-risk MSM who inject drugs (MWID).

Discussion

We determined the incremental cost-effectiveness of introducing or increasing the scale of delivery of a set of evidence-based HIV treatment and prevention interventions in six U.S. cities with diverse microepidemics. Increased HIV testing was found to be cost-saving or cost-effective across cities, despite extensive epidemiological and structural differences in their public health responses to HIV[5]. In contrast, the value provided by expanded access to HIV prevention programs or ART engagement and re-engagement interventions was dependent on local context, highlighting fundamental differences in access to care across settings[5]. As no single intervention was predicted to avert more than 10% of projected new HIV infections, our findings emphasize the need for targeted, locally-oriented combination implementation strategies to reach the ambitious goal of ending the HIV epidemic[1].

We reiterate that our analysis considered only increments in service provision – that is, additional scale-up beyond existing service levels. Our implementation scientific approach to estimating scale and implementation costs was, to our knowledge, a novel application, providing a more concrete assessment of the impact of these interventions within different settings. Costeffectiveness analyses of nearly each of the individual interventions have been done before using a similar modeling approach, but typically at a national level (or in a single setting) without consideration of the effects of scale of delivery combined with heterogeneity in local structural and epidemiological context. While we only considered interventions directly affecting HIV-related outcomes, limited scale-up of delivery for some interventions reflect the realities of constraints in healthcare access. For instance, the disparity in access and quality of care for people living with HIV in Atlanta, located in a state that did not expand Medicaid under the Affordable Care Act, is reflected in the relatively lower value provided by expanding ART engagement and re-engagement interventions in this city. Our findings, consistent with prior evidence that found greater ART dropout for people living with HIV that are from the South[36], further underline the need for multifaceted public health strategies to overcome social and structural barriers to care.

While nationwide expansion of HIV testing in the United States has previously been found to be cost-effective (but not cost-saving)[11, 3739], testing levels were typically based on national guidelines, without accounting for current service levels. Though precise figures are notably absent[13], our analysis accounted for estimated differences in testing rates across cities, demographic and HIV risk groups[16] over an extended time horizon, demonstrating greater value than some, but not all[40, 41] prior applications. Recommendations from the US CDC and the US Preventive Services Task Force currently recommend opt-out HIV testing for adults in all healthcare settings and annual screening for individuals at high risk of HIV infections[42, 43]. Large administrative claims database studies of Medicaid and commercially insured patients aged 13 to 64 found only 4.3% and 2.8%, respectively, had received at least one HIV test in 2012[44]. Furthermore, HIV tests were conducted in only 0.4% of all emergency department visits in the United States in 2015[45]. Taken with our results suggesting the cost-effectiveness of HIV testing scale-up in all cities, this evidence indicates that interventions designed to increase HIV testing should be included in any combination implementation strategy.

The value provided by expanded access to SSP and targeted PrEP for high-risk MSM was found to be highly dependent on underlying service levels and the local epidemiological context. Syringe distribution expansion was cost-saving (Miami) or highly cost-effective (Atlanta, LA) when existing coverage was low but additional expansion of SSP services in well-resourced cities that have already experienced the substantial public health benefits of high SSP coverage (Baltimore, NYC, Seattle)[46] provided less value. Though expanding access to PrEP for high-risk MSM reduced incidence across all cities (from 3.4% in LA to 10.1% in Miami), we estimated that targeted PrEP provided good value for money in cities with relatively lower coverage and higher rates of HIV incidence among high-risk MSM (Atlanta, Baltimore, LA, Miami) but provided less value in NYC and Seattle where coverage levels are relatively higher. These findings are consistent with research indicating decreasing marginal impact of increased coverage at high levels of PrEP uptake[47]. In contrast, given the reduced risk of mortality, expansion of MOUD to PWID with an opioid use disorder was found to be cost-effective across cities regardless of existing coverage levels, even though the population-level impact on HIV incidence was relatively low.

Our analysis had several limitations. First, we imposed simplifying assumptions for risk behaviors associated with transmission of HIV rather than more complex network structures that may better approximate sexual and injection networks. Nevertheless, these assumptions were consistent with the availability of best-quality evidence and supported simulation of the focal HIV microepidemics with a high degree of precision[16]. Second, we made linear assumptions about the implementation and impact of each intervention and their costs. A different characterization of the returns to scale and the increasing or decreasing marginal cost of an intervention may influence results; however, the form of these functions is unknown and, to the best of our knowledge, there is no evidence available to inform them[48]. Third, we did not consider all possible interventions for all individuals at risk of HIV. Emerging evidence in HIV treatment and care delivery, which may comprise interventions for vulnerable populations, longer-acting ART and PrEP formulations or efficient delivery of self-testing for HIV, present opportunities for future expansion. Furthermore, real-world implementation data to inform reach, adoption and scale for individual interventions are limited[4] and we did not explicitly consider interventions that could improve the various implementation steps for each intervention. Fourth, results of expanded access to HIV prevention programs were highly dependent on local context and underestimation of existing service levels could influence our findings; however, service levels were derived using the best publicly available data[13] and we assessed the robustness of our results with probabilistic sensitivity analysis. Lastly, we only considered HIV prevention benefits associated with access to MOUD and SSP. Because of the reduction in the risk of transmission of HCV associated with MOUD and SSP, expansion of these programs could deliver greater overall health benefits at lower incremental costs than our results indicate.

This study demonstrates that combination implementation strategies for HIV should be tailored to local epidemiological contexts in order to provide the most value; however, complementary public health strategies addressing factors hindering access to HIV care will be needed to maximize the impact of these strategies and meet the newly-established targets for HIV elimination in the United States.

Supplementary Material

Supplement

Acknowledgements

EK and BN conceptualized the study and wrote the first draft of the article. EK and BE contributed to the evidence synthesis and contributed to manuscript development. EK and XZ assisted with analyses. XZ, CNB, CDR, JD, DJF, KAG, MG, BDLM, LM, BRS, SS, SAS, and BN aided in the interpretation of results and provided critical revisions to the manuscript. BN secured funding for the study. All authors approved the final draft.

This work was supported by the National Institutes on Drug Abuse (NIDA grant no. R01DA041747 to Dr. Nosyk). Dr. Schackman received additional support from the Center for Health Economics of Treatment Interventions for Substance Use Disorder, HCV, and HIV (NIDA grant P30DA040500). Dr. Strathdee is supported by a NIDA Method to Extend Research in Time (MERIT) award (R37DA019829). The funders had no direct role in the conduct of the analysis or the decision to submit the manuscript.

Footnotes

Conflicts of Interest and Source of Funding

EK, XZ, BE, JEM, CNB, CDR, DJF, KAG, MG, BDLM, LM, BRS, SS, SAS, and BN declare no competing interests. JCD has participated in research supported by grants to the University of Washington from Hologic. Funding for the study described in this publication was provided by Simon Fraser. Dr. Gebo is also a paid consultant to Simon Fraser. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict of interest policies.

References

  • 1.Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV epidemic: a plan for the United States. JAMA 2019; 321(9):844–845. [DOI] [PubMed] [Google Scholar]
  • 2.del Rio Carlos, Armstrong Wendy S, Curran James, Can the United States Achieve Human Immunodeficiency Virus Epidemic Control? A New Initiative Offers Hope, Clinical Infectious Diseases, Volume 69, Issue 8, 15 October 2019, Pages 1434–1435, 10.1093/cid/ciz155 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Chang LW, Serwadda D, Quinn TC, Wawer MJ, Gray RH, Reynolds SJ. Combination implementation for HIV prevention: moving from clinical trial evidence to population-level effects. Lancet Infect Dis 2013; 13(1):65–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Glasgow RE, Eckstein ET, ElZarrad MK. Implementation science perspectives and opportunities for HIV/AIDS research: integrating science, practice, and policy. JAIDS Journal of Acquired Immune Deficiency Syndromes 2013; 63:S26–S31. [DOI] [PubMed] [Google Scholar]
  • 5.Panagiotoglou D, Olding M, Enns B, Feaster D, Del Rio C, Metsch L, et al. Building the case for localized approaches to HIV: structural conditions and health system capacity to address the HIV/AIDS epidemic in six US cities. AIDS and Behavior 2018; 22(9):3071–3082. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Anderson SJ, Cherutich P, Kilonzo N, Cremin I, Fecht D, Kimanga D, et al. Maximising the effect of combination HIV prevention through prioritisation of the people and places in greatest need: a modelling study. Lancet 2014; 384(9939):249–256. [DOI] [PubMed] [Google Scholar]
  • 7.Jacobsen M, Walensky R. Modeling and cost-effectiveness in HIV prevention. Curr HIV/AIDS Rep 2016; 13:64–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Garnett G, Cousens S, Hallett T, Steketee R, Walker N. Mathematical models in the evaluation of health programmes. Lancet 2011; 378(9790):515–525. [DOI] [PubMed] [Google Scholar]
  • 9.Mishra S, Mountain E, Pickles M, Vickerman P, Shastri S, Gilks C, et al. Exploring the population-level impact of antiretroviral treatment: the influence of baseline intervention context. Aids 2014; 28:S61–S72. [DOI] [PubMed] [Google Scholar]
  • 10.Schackman BR. Implementation science for the prevention and treatment of HIV/AIDS. Journal of acquired immune deficiency syndromes (1999) 2010; 55(Suppl 1):S27. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Long EF, Brandeau ML, Owens DK. The Cost-Effectiveness and Population Outcomes of Expanded HIV Screening and Antiretroviral Treatment in the United States. Annals of Internal Medicine 2010; 153(12):778-+. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Nosyk B, Min JE, Lima VD, Hogg RS, Montaner JS. Cost-effectiveness of population-level expansion of highly active antiretroviral treatment for HIV in British Columbia, Canada: a modelling study. The lancet HIV 2015; 2(9):00127–00127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Krebs E, Enns B, Wang L, Zang X, Panagiotoglou D, Del Rio C, et al. Developing a dynamic HIV transmission model for 6 U.S. cities: An evidence synthesis. PLOS ONE 2019; 14(5):e0217559. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Newman MEJ. Mixing patterns in networks. Phys Rev E 2003; 67(2). [DOI] [PubMed] [Google Scholar]
  • 15.Sutton AJ, House T, Hope VD, Ncube F, Wiessing L, Kretzschmar M. Modelling HIV in the injecting drug user population and the male homosexual population in a developed country context. Epidemics-Neth 2012; 4(1):48–56. [DOI] [PubMed] [Google Scholar]
  • 16.Zang Z, Krebs E, Min JE, Pandya A, Marshall BD, Shackman BR, Behrends C, Feaster DJ, Nosyk B. Development and calibration of a dynamic HIV transmission model for 6 US cities. Med Dec Making. 2020. January; 40(1); 3–16 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Centers for Disease Control and Prevention. HIV Infection, Risk, Prevention, and Testing Behaviors among Persons Who Inject Drugs—National HIV Behavioral Surveillance: Injection Drug Use, 20 U.S. Cities, 2012 HIV Surveillance Special Report 11. Revised edition In; 2015. [Google Scholar]
  • 18.Centers for Disease Control and Prevention. HIV Infection Risk, Prevention, and Testing Behaviors among Men Who Have Sex With Men—National HIV Behavioral Surveillance, 20 U.S. Cities, 2014. HIV Surveillance Special Report 15. In; 2016. [Google Scholar]
  • 19.Centers for Disease Control and Prevention. Public use data file documentation. 2011–2013. National Survey of Family Growth. User’s guide. In. Hyattsville, Maryland: Centers for Disease Control and Prevention, National Center for Health Science; 2014. [Google Scholar]
  • 20.Baggaley RF, White RG, Hollingsworth TD, Boily MC. Heterosexual HIV-1 Infectiousness and Antiretroviral Use Systematic Review of Prospective Studies of Discordant Couples. Epidemiology 2013; 24(1):110–121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Cohen M, Chen Y, McCauley M, Gamble T, Hosseinipour M, Kumarasamy N, et al. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med 2011; 365(6):493–505. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Nosyk B, Zang X, Min JE, Krebs E, Lima VD, Milloy MJ, et al. Relative effects of antiretroviral therapy and harm reduction initiatives on HIV incidence in British Columbia, Canada, 1996–2013: a modelling study. The Lancet HIV 2017; 4(7):E303–E310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Choopanya K, Martin M, Suntharasamai P, Sangkum U, Mock PA, Leethochawalit M, et al. Antiretroviral prophylaxis for HIV infection in injecting drug users in Bangkok, Thailand (the Bangkok Tenofovir Study): a randomised, double-blind, placebo-controlled phase 3 trial. The Lancet 2013; 381(9883):2083–2090. [DOI] [PubMed] [Google Scholar]
  • 24.MacArthur GJ, Minozzi S, Martin N, Vickerman P, Deren S, Bruneau J, et al. Opiate substitution treatment and HIV transmission in people who inject drugs: systematic review and meta-analysis. Bmj 2012; 345:e5945. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Aspinall EJ, Nambiar D, Goldberg DJ, Hickman M, Weir A, Van Velzen E, et al. Are needle and syringe programmes associated with a reduction in HIV transmission among people who inject drugs: a systematic review and meta-analysis. Int J Epidemiol 2014; 18(11):2144–2155. [DOI] [PubMed] [Google Scholar]
  • 26.Marks G, Crepaz N, Senterfitt JW, Janssen RS. Meta-analysis of high-risk sexual behavior in persons aware and unaware they are infected with HIV in the United States - Implications for HIV prevention programs. Jaids-J Acq Imm Def 2005; 39(4):446–453. [DOI] [PubMed] [Google Scholar]
  • 27.Tempalski B, Pouget E, Cleland C, Brady J, Cooper H, Hall H, et al. Trends in the population prevalence of people who inject drugs in US metropolitan areas 1992–2007. PLoS One 2013; 8(6):e64789. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Grey JA, Bernstein KT, Sullivan PS, Purcell DW, Chesson HW, Gift TL, et al. Estimating the Population Sizes of Men Who Have Sex With Men in US States and Counties Using Data From the American Community Survey. JMIR Public Health and Surveillance 2016; 2(1):e14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Degenhardt L, Peacock A, Colledge S, Leung J, Grebely J, Vickerman P, et al. Global prevalence of injecting drug use and sociodemographic characteristics and prevalence of HIV, HBV, and HCV in people who inject drugs: a multistage systematic review. The Lancet Global Health 2017; 5(12):e1192–e1207. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Nosyk B, Zang X, Krebs E, Behrends C, Del Rio C, Dombrowski J, Feaster DJ, Golden M, Granich R, Marshall BDL, Mehta SH, Metsch L, Schackman BR, Shoptaw S, Strathdee SA. Ending the epidemic in America will not happen if the status quo continues: modeled projections for HIV incidence in 6 US cities. Clin Infect Dis. 2019. December; 69(12): 2195–2198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Centers for Disease Control and Prevention (CDC). Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. In; 2018.
  • 32.Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes B, et al. Oxford Centre for Evidence-based Medicine - Levels of Evidence (March 2009). In. Edited by Howick J,Oxford, UK: Centre for Evidence-Based Medicine; 2009. [Google Scholar]
  • 33.Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American journal of public health 1999; 89(9):1322–1327. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine. JAMA 2016; 316(10):1093. [DOI] [PubMed] [Google Scholar]
  • 35.Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS)—explanation and elaboration: a report of the ispor health economic evaluation publication guidelines good reporting practices task force. Value Health 2013; 16(2):231–250. [DOI] [PubMed] [Google Scholar]
  • 36.Wang L, Krebs E, Min JE, Mathews WC, Nijhawan A, Somboonwit C, et al. Combined estimation of disease progression and retention on antiretroviral therapy among treated individuals with HIV in the USA: a modelling study. The Lancet HIV 2019; 6(8):e531–e539. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Paltiel AD, Weinstein MC, Kimmel AD, Seage III GR, Losina E, Zhang H, et al. Expanded screening for HIV in the United States—an analysis of cost-effectiveness. New England Journal of Medicine 2005; 352(6):586–595. [DOI] [PubMed] [Google Scholar]
  • 38.Sanders GD, Bayoumi AM, Sundaram V, Bilir SP, Neukermans CP, Rydzak CE, et al. Costeffectiveness of screening for HIV in the era of highly active antiretroviral therapy. New England Journal of Medicine 2005; 352(6):570–585. [DOI] [PubMed] [Google Scholar]
  • 39.Borre ED, Hyle EP, Paltiel AD, Neilan AM, Sax PE, Freedberg KA, et al. The clinical and economic impact of attaining National HIV/AIDS Strategy treatment targets in the United States. The Journal of infectious diseases 2017; 216(7):798–807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Lasry A, Sansom SL, Hicks KA, Uzunangelov V. Allocating HIV prevention funds in the United States: recommendations from an optimization model. PloS one 2012; 7(6):e37545. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Hutchinson AB, Farnham PG, Sansom SL, Yaylali E, Mermin JH. Cost-effectiveness of frequent HIV testing of high-risk populations in the United States. Journal of acquired immune deficiency syndromes (1999) 2016; 71(3):323. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Branson BM, Handsfield HH, Lampe MA, Janssen RS, Taylor AW, Lyss SB, et al. Revised recommendations for HIV testing of adults, adolescents, and pregnant women in healthcare settings. Morbidity and Mortality Weekly Report: Recommendations and Reports 2006; 55(14):1.-CE-4. [PubMed] [Google Scholar]
  • 43.Moyer VA. Screening for HIV: US preventive services task force recommendation statement. Annals of internal medicine 2013; 159(1):51–60. [DOI] [PubMed] [Google Scholar]
  • 44.Dietz PM, Van Handel M, Wang H, Peters PJ, Zhang J, Viall A, et al. HIV testing among outpatients with Medicaid and commercial insurance. PloS one 2015; 10(12):e0144965. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Rui P, Kang K. National Hospital Ambulatory Medical Care Survey: Emergency Department Summary Tables. In: US Department of Health and Human Services, National Center for Health …; 2015. [Google Scholar]
  • 46.Ruiz MS, O’Rourke A, Allen ST, Holtgrave DR, Metzger D, Benitez J, et al. Using Interrupted Time Series Analysis to Measure the Impact of Legalized Syringe Exchange on HIV Diagnoses in Baltimore and Philadelphia. JAIDS Journal of Acquired Immune Deficiency Syndromes 2019; 82:S148–S154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Jenness SM, Goodreau SM, Rosenberg E, Beylerian EN, Hoover KW, Smith DK, et al. Impact of the Centers for Disease Control’s HIV preexposure prophylaxis guidelines for men who have sex with men in the United States. The Journal of infectious diseases 2016; 214(12):1800–1807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Brandeau ML, Zaric GS, De Angelis V. Improved allocation of HIV prevention resources: using information about prevention program production functions. Health Care Manag Sci 2005; 8(1):19–28. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement

RESOURCES