Prioritization of Evidence-Based and Evidence-Informed Interventions for Retention in Medical Care for Persons with HIV

Abstract

Up to 50% of those diagnosed with HIV in the U.S. are not retained in medical care. Care retention provides opportunity to monitor benefits of HIV therapy and enable viral suppression. To increase retention, there is a need to prioritize best practices appropriate for translation and dissemination for real-world implementation. Eighteen interventions from CDC’s Compendium of Evidence-Based Interventions were scored using the RE-AIM framework to determine those most suitable for dissemination. A CDC Division of HIV Prevention workgroup developed a RE-AIM scale with emphasis on the Implementation and Maintenance dimensions and less emphasis on the Efficacy dimension since all 18 interventions were already identified as evidence-based or evidence-informed. Raters referenced primary efficacy publications and scores were averaged for a ranked RE-AIM score for interventions. Of 18 interventions, four included care linkage and 7 included viral suppression outcomes. Interventions received between 20.6 and 35.3 points (45 maximum). Scores were converted into a percentage of the total possible with ranges between 45.8 and 78.4%. Top four interventions were ARTAS (78.4%); Routine Screening for HIV (RUSH) (73.2%); Optn4Life (67.4%) and Virology Fast Track (65.9%). All four scored high on Implementation and Maintenance dimensions. Select items within the scale were applicable to health equity, covering topics such as reaching under-served focus populations and acceptability to that population. Navigation-enhanced Case Management (NAV) scored highest on the health equity subscale. RE-AIM prioritization scores will inform dissemination and translation efforts, help clinical staff select feasible interventions for implementation, and support sustainability for those interventions.

Introduction

Persons with HIV (PWH) must enter and remain in medical care to obtain the benefits of antiretroviral therapy (ART) [1, 2]. Retention in care provides the opportunity to monitor responses to HIV therapy, enable viral suppression [3-5], and deliver ancillary services [1, 6]. The Health Resources and Services Administration (HRSA) recommends a follow-up visit at least once every 3–6 months after initiating HIV care separated by at least 90 days [7]. Lack of health insurance, stigma, depression, competing life activities [8], geographic distance from providers and lack of transportation [9], homelessness [10], lack of acceptance of HIV infection status [11], physical and psychiatric illness [12], and substance abuse issues [13] are barriers to consistent medical care. During 2019 in 45 jurisdictions, 57.8% of persons were retained in HIV medical care [14]. Patients not retained in care have a reduced likelihood of ART initiation [2, 15], are at increased risk of transmitting HIV [1, 16], and are less likely to be medication adherent, increasing the risk for viral drug resistance [17]. Such patients have reduced CD4 counts; are at greater risk of opportunistic infections [18]; are at greater risk of disease progression; and have increased mortality risk [1, 19, 20]. In sum, retention in HIV care is essential for improving HIV outcomes [21, 22].

Best practices have been identified to improve HIV care outcomes such as linkage, retention, re-engagement, and viral suppression [23-25]. To address the need to increase retention in medical care, dissemination of best practices is needed. Select HIV prevention interventions have been disseminated into public health practice [26], but often healthcare interventions found to be best practices are not widely translated, disseminated, and implemented under real-world conditions [27-29]. Determining the most efficacious interventions may not be the only or most appropriate standard to apply [30]. Although efficacious, some interventions may be too labor intensive, expensive, and demanding of clients. Identifying the best practices most appropriate for local implementation is needed to provide informed consumer choice for HIV clinical service organizations. Such interventions would be efficacious but also most appropriate for clinic resources and populations served.

Given that HIV care retention rates are reduced for several populations and the potential need to tailor interventions to unique population needs, the most effective interventions may vary across clinics or settings [14]. Additionally, varied resource availability and service delivery models might influence the effectiveness of a given intervention or require mid-course corrections and adaptations [31]. These factors must be considered along with intervention efficacy. Thus, a method for prioritizing the retention in care interventions is needed. The aim of this paper is to test such a prioritization method on evidence-based and evidence-informed interventions for retention in HIV care to provide guidance on which of these interventions may be most appropriate for implementation under real-world conditions.

Methods

The study sample consisted of the best practices for retention in care from the Linkage to, Retention in, and Re-engagement in HIV Care (LRC) chapter of CDC’s Prevention Research Synthesis (PRS) Project’s Compendium of Evidence-Based Interventions and Best Practices [25, 32]. Best practices include evidence-based interventions (EBIs) and evidence-informed interventions (EIs). EBIs are tested with a comparison group (e.g., randomized controlled trials) and are considered to have strong evidence of efficacy whereas EIs are interventions that are tested with one-group, pre-post research designs or small sample sizes and have sufficient evidence of efficacy.

To date, 18 U.S.-based best practices (Table 1) have been identified to improve retention in care. Eight are EBIs and 10 are EIs. Four also have evidence for linking PWH who have a recent HIV diagnosis to medical care and seven for improving viral suppression. Four of the interventions identified by PRS; ARTAS [33], HIV Care Coordination/Steps to Care [34], Stay Connected [35], Enhanced Personal Contact [36], have previously been disseminated by the CDC Division of HIV Prevention (DHP) into practice.

Table 1.

RE-AIM Scores, Equity Subscale Scores and Ranks, Intervention characteristics identified during scoring, and data sources

Intervention name	Reach Score (Max = 5)	Efficacy Score (Max = 8)	Adoption score (Max = 8)	Implementation score (Max = 15)	Maintenance score (Max = 9)	RE-AIM total score/ rank (Max = 45)	% Total Score	Health equity subscale score/rank (Max = 26)
ARTAS [33]	4.7	5.7	5.7	12.7	6.7	35.3 ranked 1st	78.4%	21 ranked 2nd (tied with OPTn4Life)
Features identified during scoring: Case managers used strength-based counseling to link recently diagnosed HIV-positive clients to care. Intervention is not complex and can be easily implemented by many HIV prevention providers. Training and technical assistance is readily available. Majority of clients were linked by second session. Intervention was evidence-based for both linkage and retention in care
Data Sources: Primary outcome paper, coder familiarity with the program
Routine Universal Screening for HIV (RUSH) [38]	4	6	5.3	11.7	6	33 ranked 2nd	73.2%	20.3 ranked 4th
Features identified during scoring: Provides screening, opt-out testing, and immediate linkage to care for emergency dept patients who test positive, whether newly or previously diagnosed. Case managers accompany those who test positive to treatment intake where ART may begin. Rapid access to ART on same day as testing resulted in high scores. Intervention was evidence-informed for linkage, retention in care, and viral suppression
Data Sources: Primary outcome paper
OPTn4Life [40]	5	5	8	9.3	3	30.3 ranked 3rd	67.4%	21 ranked 2nd (tied with ARTAS)
Features identified during scoring: A multi-featured mobile app provides appointment/medication reminders, virtual tele-heath visits, help with health goals, 2-way messaging, lab results (viral load and CD4) and HIV information. Menu of client-centered features would make this app of high interest to other clinical providers. Intervention was evidence-informed for 2 outcomes: retention in care and viral suppression
Data Sources: Primary outcome paper
Virology fast track [41]	5	4	5	10.7	5	29.7 ranked 4th	65.9%	17.7 ranked 8th
Features identified during scoring: Electronic medical record (EMR) alerts of sub-optimal follow-up, virologic failure, and new laboratory toxicities. Alerts providers on EMR home page, patient specific EMRs, and emails. Allows providers to request follow-up appointments and lab tests. Comprehensive clinical tool with broad generalizability to other clinics and providers. Intervention was evidence-based for retention
Data Sources: Primary outcome paper
Navigation-Enhanced Case Management (NAV) [42]	4.7	6.7	7.7	6.3	3.3	28.6 ranked 5th	63.6%	21.6 ranked 1st
Features identified during scoring: Case managers provide discharge planning/navigation upon release from incarceration for adults living with HIV who have previously used/currently using substances. Adaptation of evidence-based intervention for newly released incarcerated persons disseminated by CDC. Training/technical assistance available. Identified as evidence-based intervention for both linkage and retention
Data Sources: Primary outcome paper, coder familiarity with the program
Strength Through Livin’ Empowered (STYLE) [47]	5	6	6.3	6.3	3.7	27.3 ranked 6th	60.7%	20 ranked 5th
Features identified during scoring: Social marketing and outreach to YMSM of color linking newly diagnosed and those lost to care to social and medical services including support groups and counseling. Culturally designed to reach recently diagnosed or lost-to-care HIV-positive African American and Hispanic/Latino young men who have sex with men. The intervention was identified as evidence-informed for retention in care
Data Sources: Primary outcome paper
HIV Care Coord/Steps to Care [34]	5	4.3	6.7	7.7	3.3	27 ranked 7th	59.9%	19 ranked 6th
Features identified during scoring: Case management/navigation integrated into coordinated care team that provides ART adherence, appointment reminders, outreach for missed appointments, and transportation. Focus is on recently diagnosed with HIV who are at high risk for, or have a history of, suboptimal HIV care outcomes. Online training, web resources, technical assistance provided by CDC. Intervention is evidence-informed for retention, evidence-based for viral suppression
Data Sources: Primary outcome paper, coder familiarity with the program
weCARE [44, 45]	5	3.7	7.3	7.7	2.7	26.3 ranked 8th	58.5%	18.3 ranked 7th
Features identified during scoring: Social media messages around HIV continuum of care topics followed by cyberhealth educator-initiated conversations. Focus on hard-to-reach racially and ethnically diverse young men who have sex with men (MSM) and transgender women living with HIV. Use of available social media platforms was strength but labor-intensive implementation for cyber counselor to manage theory-based messages and personalized communication. Intervention was evidence-informed for retention in care/ viral suppression
Data Sources: Primary outcome paper, follow-up papers
PositiveLinks [48]	5	5	6.7	6.7	2.3	25.7 ranked 9th	57.%	17.3 ranked 9th
Features identified during scoring: Smartphone app for patients with tailored education including appointment reminders, daily queries about medication adherence, a community message board, and stress reduction techniques. Smartphones given to all clients would be challenge for implementation/maintenance and generalizability. The intervention was evidence-informed for retention and viral suppression
Data Sources: Primary outcome paper
Project nGAGE [49]	5	4	6.7	6.7	3	25.3 ranked 10th	56.2%	17 ranked 10th
Features identified during scoring: Social network approach for both retention and medication adherence for young Black men who have sex with men. Counselor meets for 90 min with client and support confidant, develops care plan, followed by 4 booster calls. Intervention involved complex implementation restricted to those clients willing to involve personal confidant in their treatment plan, limiting reach. The intervention was evidence-based for retention
Data Sources: Primary outcome paper
Link LA [50]	5	5	5.3	6.7	2.3	24.3 ranked 12th	54.1%	16.3 ranked 12th
Features identified during scoring: 12-session, 24-week intervention for HIV positive men and transgender women in jail and after release. Intervention delivered by lay peer navigators emphasizes retention in care and medication adherence. Intervention requires considerable access to correctional facilities, is labor intensive, a challenge to maintain over time, and thus lacks generalizability. The intervention was evidence-based for linkage, retention, and viral suppression
Data Sources: Primary outcome paper
Stay Connected [35]	3.7	1.7	7	7.3	4.7	24.3 ranked 12th	54.%	14 ranked 15th
Features identified during scoring: Clinic-wide intervention provides brochures, posters, and brief supportive messages from all staff on the importance of retention in care. This was low complexity and low effort intervention. Training and technical assistance is available. The intervention was evidence-informed for retention
Data Sources: Primary outcome paper, coder familiarity with the program
Enhanced Personal Contact [36]	4	4	5	8.3	3	24.3 ranked 12th	54.4	15.4 ranked 14th
Features identified during scoring: Trained clinic interventionist paired with patients for face to face and phone contacts and reminders 7 and 2 days before appointments. No shows get a call after 24 h. This was a low complexity and low effort intervention. Training and technical assistance is available. Intervention was evidence-based for retention
Data Sources: Primary outcome paper, coder familiarity with the program
Financial incentives [51, 52]	5	4	2	10	2	23 ranked 14th	51.1%	11 ranked 18th
Features identified during scoring: Financial incentives to HIV positive persons newly diagnosed and new to treatment or treatment experienced for linkage to care, retention in care, and viral suppression. Intervention was evidence-based for retention in care only. The intervention strategy was considered potentially costly and not feasible for HIV clinical and prevention providers
Data Sources: Primary outcome paper, follow-up papers
Clinic-based Buprenorphine Treatment (BPU) [43]	5	3.7	6	5.3	2.7	22.7 ranked 15th	50.3%	16.7 ranked 11th
Features identified during scoring: Integrate BPU Tx into primary care for opioid dependent HIV positive patients. Includes counseling, urine screens, and initial direct observation followed by take-home privileges. The intervention is designed for a narrowly defined focus population and clinicians would need to be fully educated on buprenorphine treatment combined with HV treatment. Intervention was evidence-based for retention
Data Sources: Primary outcome paper
Bilingual/Bicultural Care Team [53]	4.7	4	6	4	3.7	22.3 ranked 16th	49.5%	16 ranked 13th
Features identified during scoring: Three-person bilingual care team of nurse practitioner, case manager, and peer navigator offer linguistically/culturally appropriate patient education, referrals, home visits for Hispanic/Latino clinic patients. The intervention would be best implemented in geographic areas with higher levels of Hispanic/Latino patients to make the intervention feasible and cost effective. Intervention was evidence-informed for retention
Data Sources: Primary outcome paper
Patient-centered HIV Care Model (PCHCM) [54—57]	4	3	4.7	7.3	3	22 ranked 17th	48.7%	12.3 ranked 17th
Features identified during scoring: Integrates community-based HIV specialized pharmacists with HIV clinic medical providers. This allows sharing of medical records and ongoing progress on medication action plan developed by pharmacists, physicians, and patients. Intervention raised concern about reimbursement for pharmacists for participation in a complex intervention involving much data transfer from clinic to pharmacy. Intervention was evidence-informed for retention and viral suppression
Data Sources: Primary outcome paper, follow-up papers
Centralized HIV Services/Coordinated Care [46]	4.3	2.7	4.3	6.3	3	20.6 ranked 18th	45.8%	13 ranked 16th
Features identified during scoring: multi-disciplinary youth clinic focused social workers and case managers use motivational interviewing to teach health navigation skills and encourage HIV disease management. Focus population was young African American and Hispanic/Latino HIV clinic patients aged 13–23 years. The intervention would be best implemented by a clinic with a substantial number of young racial and ethnic minority HIV clinic patients. Motivational interviewing technique would require considerable training, quality control, and continued training for all newly hired clinic staff, making this intervention potentially costly. Intervention was evidence-informed for retention
Data Sources: Primary outcome paper

A multidisciplinary workgroup from CDC’s DHP was formed to guide the process of developing a scale for scoring and conducting the reviews of the 18 interventions. The RE-AIM (Reach, Efficacy, Adoption, Implementation, Maintenance) model [27, 28, 30] provided the framework used to determine the interventions most suitable for national dissemination. Members of the workgroup developed operational definitions of the RE-AIM dimensions to support the prioritization of evidence-based and evidence-informed interventions. To test the definitions, the initial RE-AIM-informed prioritization instrument was piloted with two interventions (ARTAS [33] and Stay Connected [35]). Reviewers used a three-item scale (High evidence, Moderate evidence, No evidence) to assess the level of evidence for the dimension in the article; the point value ranged from 0 (no evidence) to 2 (high evidence). Missing or non-reported data was scored as zero points. The results from the pilot reviews led to clarifications being made and additional questions being added to the RE-AIM informed prioritization instrument. These two interventions were then re-scored using the final version of the instrument.

The remaining interventions were independently reviewed and scored using the final instrument. Each dimension was scored based on specific assessment criteria as shown in Table 2. There were eleven subject matter expert (SME) raters who were then assigned into groups of three raters per intervention. The eleven raters were assigned to approximately 5 interventions each to equally distribute the SMEs. The 3-person rater groups discussed their individual scores to ensure common understanding of the intervention.

Table 2.

RE-AIM Dimensions assessed across 18 interventions for retention in medical care for HIV positive persons

RE-AIM dimension	Assessment criteria	Points available	Total/max 45 points
Reach	Access to those most at risk	2
	Recruitment and retention strategy	1
	Reported level of exposure, participation, completion of intervention	2
	REACH subscale		5
Efficacy	EBI or EI	2
	Clinical significance	2
	Lack of negative effects	0
	Multiple beneficial outcomes	2
	Durability of effects	1
	Immediacy of effects	1
	Efficacy subscale		8
Adoption	Acceptability to clients	3
	Acceptability to implementers	3
	Appropriateness for clients	2
	Adoption subscale		8
Implementation	Adaptability	4
	Generalizability	2
	Cost	2
	Training and technical assistance to implementing staff	4
	Feasibility/complexity	3
	Implementation subscale		15
Maintenance	Training and technical assistance available for new hired staff	3
	Feasibility/complexity	3
	Generalizability	1
	Sustainability	1
	Operationalized/standard of care	1
	Maintenance subscale		9

To make our scoring process less subjective, we developed ‘scoring pointers’ that guide the scoring process for most of our instrument’s questions. Examples were provided in most of these scoring pointers to further inform the scoring process and decrease subjectivity. Our SME panels had a discussion, question by question, on each intervention and SMEs were allowed to change an initial score if another panel member was able to provide evidence as to why a particular score for a particular question was warranted.

Each of the 3-person rater groups presented their scores during larger workgroup sessions and discussed the rationale for scores. The scores from the three raters were averaged and then summed to develop a final prioritization score for each intervention. The interventions were then ranked by their final score (see Table 1).

Various sources of information were used to evaluate the selected interventions on the RE-AIM dimensions. At a minimum, all raters referenced the primary efficacy publication and a detailed overview of the intervention from the PRS Compendium. The latter document included a description of the intervention as well as a summary of the evaluation study and results. Some raters read additional articles that were published after the original efficacy publication to locate information not included in the primary publication. Most often, this additional reading provided insight into the availability of training and technical assistance, cost analysis, or maintenance and sustainability criteria.

Raters scored criteria outlined in the RE-AIM informed prioritization instrument in one of two ways: (1) based on explicit information contained in the reference documents (primarily in the areas of Reach and Effectiveness) and (2) based on subject matter expertise of the independent raters (primarily in the areas of Adoption, Implementation, and Maintenance). Such expertise typically was gained from firsthand experience with the selected intervention or similar interventions. Some of the criteria drawing on subject matter expertise involved a degree of subjectivity and called on the raters to provide their professional opinion based on an understanding of the resource needs and intensity of the intervention.

The scoring of intervention costs provides an example of the scoring process. We used the following scoring rules. Points were given for demonstrated cost–benefit or cost-effectiveness. Points were also given for any reporting of costs that would potentially allow for replication of an intervention, and any outcomes in addition to the primary outcomes of retention in medical care, that would indicate cost-saving by demonstrating multiple outcomes across the HIV prevention and care continuum. Whereas some interventions may not report costs, they may report the number of staff required for implementation, the professional qualifications or experience of intervention staff, equipment needed for implementation, space requirements, and electronic equipment needed for implementation. Reviewers also assessed whether these additional resources were reasonable or costly for agencies that would implement the intervention under real-world conditions. An intervention could receive up to 5 points on the four cost questions.

The RE-AIM framework is based on the premise that evidence for successful implementation under real-world conditions must include broad dimensions in addition to the efficacy of the research [30]. The ultimate impact of the intervention is due to the combined effects of these dimensions. A review of all the selected interventions would then inform translation and dissemination as well as support the sustainability of interventions that are disseminated into the field of practice. The RE-AIM dimensions are operationalized and summarized in Table 2 and specific dimension criteria are provided below.

RE-AIM Dimensions

Dimension 1: Reach

Reach is the absolute number, proportion, and representativeness of individuals willing to participate in an intervention [37]. We constructed a Reach subscale for those at increased risk of dropping out of care compared to others in a general clinic population. We addressed representativeness of the intervention through assessment of coverage of the focus population. Coverage was further assessed by looking at the proportion of clients/patients who were exposed to intervention materials, participated in intervention activities, and completed the intervention content. Within a clinic population the absolute number of patients and the proportion that participated in the study was typically reported. Potential points for the Reach dimension had a range of 0 to 5. (See Table 2).

Dimension 2: Efficacy

Efficacy is the impact of an intervention on outcomes, including potential negative effects [37]. We assessed efficacy of an intervention on six domains: level of evidence, clinical significance, negative effects, multiple benefits, durability of effects, and immediacy of effects. For level of evidence, we gave one-point higher scores to EBIs because they are more rigorously tested than EIs. We assessed clinical significance for the primary outcome of retention in care by examining the magnitude of effect and the width of the confidence intervals. For example, we gave a moderate score of one point to interventions with large effects when the confidence intervals were wide. An intervention with large effects and narrow confidence intervals could earn 2 points. Level of evidence was determined for other relevant outcomes such as linkage to care and viral suppression [38]. Because the 18 interventions used differing criteria for retention, with differing follow-up time periods, the 3-person scoring team discussed each intervention’s outcomes to make judgements about effect size and width of confidence intervals.

Each intervention study was also reviewed for any negative effects. Negative effects were effects in the opposite direction of the intervention’s stated intentions. Interventions with multiple additional stated outcomes could lose points if the outcomes were in the opposite direction of the research objectives and statistically significant. Null effects were not considered negative effects and were scored as zero points. Negative effects and adverse events were scored as −1 each. Interventions that had multiple effects other than retention such as increased linkage to care or reduced viral load were given additional points. Because follow-up periods vary for different study protocols, we looked at whether intervention effects were detected 12 months or more after intervention delivery. This time frame served as our measure of durability of effects. Immediacy of effects was scored based on whether the researchers reported effects less than or equal to 30 days after initial intervention delivery. This measure was less applicable to retention in care since longer periods of continuous care are desirable. Potential points for the Efficacy dimension had a range of −2 to 8 points.

Dimension 3: Adoption

Adoption is the absolute number, proportion, and representativeness of settings and intervention agents who are willing to initiate a program [37]. Adoption was assessed on two domains: acceptability and appropriateness. Acceptability was further reviewed for acceptability of the intervention for the clients and acceptability for staff from the implementing agency. Appropriateness was assessed by looking for formative evaluation efforts to tailor intervention content to the focus population prior to initiation of the study. Potential points for the Adoption dimension had a range of 0 to 8.

Dimension 4: Implementation

Implementation refers to the intervention agents’ fidelity to the various dimensions of an intervention’s protocol. This dimension includes consistency of delivery as intended and the time and cost of the intervention [37]. Implementation was assessed on five domains: adaptability, costs, availability of training and technical assistance, intervention complexity, and appropriateness for current contexts. Adaptability was assessed by determining whether the intervention could be adapted for new focus populations or new venues. Training and technical assistance may be provided to implementing staff during an efficacy study and points were given if there was documentation that such training and technical assistance was provided to ensure fidelity and to serve as resources for later dissemination purposes. The complexity of the intervention was also assessed with points given for interventions that were logistically easy and convenient. Appropriateness of implementation under current political and social contexts was also assessed. Potential points for the Implementation dimension had a range of 20 points (− 5 to 15).

Dimension 5: Maintenance

Maintenance is the extent to which a program or policy becomes institutionalized or part of routine organizational practices [37]. Maintenance was assessed on six domains: ongoing training and technical assistance, complexity of maintaining the intervention, resources needed for maintenance, implementation challenges in a changing social and political environment, sustained fidelity of intervention activities, and whether the intervention became institutionalized and the standard of care. Many agencies have turnover of staff and new staff require training and technical assistance. Interventions are best maintained if activities become part of everyday culture and norms of the implementing organization. Potential points for the Maintenance dimension had a range of 14 points (− 4 to 9).

Adaptation of RE-AIM

We adapted the RE-AIM approach which weighs each dimension equally [30, 37]. For example, because all the 18 interventions were already identified as EBIs or EIs, we focused on aspects of efficacy such as durability which is not typically assessed by the PRS. Thus, the Efficacy dimension was allotted 8 points. The dimensions of Implementation and Maintenance were essential to the process of prioritizing interventions for translation, dissemination, and sustainability for DHP and thus most points fell to these dimensions with Implementation getting a maximum of 15 points, and Maintenance getting a maximum of 9 points. The maximum possible score on the RE-AIM informed prioritized instrument was 45 points.

RE-AIM questions may be developed that reflect both a RE-AIM dimension and a health equity concern. The questions for our RE-AIM assessment were often developed with health equity as a focus. The HIV Implementation Outcomes Operationalization Guide [39] was reviewed by our group which confirmed that health equity questions could be integrated and included in the 5 RE-AIM dimensions. The REAIM questions around health equity included topics such as whether those vulnerable populations most in need of retention in care were included in the study, whether appropriate methods were used to locate and recruit vulnerable populations into the intervention, whether the intervention content was acceptable for the focus population, whether formative evaluation ensured the appropriateness of intervention materials and activities, whether the intervention was adaptable so that it might reach other persons with HIV-risk factors, whether those patients at risk for discontinuation of treatment were retained in the intervention, whether the intervention was appropriate for the context and lived experiences of patients, and whether the intervention implementation and maintenance costs seemed reasonable.

Results

Table 1 contains the prioritization scores for 18 interventions reported by each of the five dimensions. Total average scores were converted into a percentage of the total possible score for ease of comparison across articles, with ranges between 45.8 and 78.4% of the total possible score.

Eight (44.4%) interventions were designated as EBIs for retention and 10 (55.6%) interventions were designated as EIs. Four (22.2%) included linkage to care, and 7 (38.9%) included viral suppression outcomes.

Interventions included in the analysis received between 20.6 and 35.3 points. When looking at the total prioritization scores, the top 4 interventions were ARTAS [33] (78.4%), a strengths-based case management intervention; RUSH [38] (73.2%), a case management intervention for persons visiting emergency rooms; Optn4Life [40] (67.4%), a mobile health intervention that sends appointment and medication reminders to PWH and Virology Fast Track [41] (65.9%), an electronic medical record provider-alert approach. These interventions were more likely to receive high scores on Implementation and Maintenance (see Table 1).

In our separate analysis of the RE-AIM questions that had a health equity focus, the NAV [42] intervention, and the Clinic-based Buprenorphine Treatment (BPU) [43] intervention moved up four ranks on the equity sub-scale when compared to the RE-AIM scale. Both interventions have a specific focus population that experience considerable health care inequalities, and all intervention activities were tailored to the unique needs of the focus population. NAV was designed specifically for persons with HIV recently released from jail and BPU was designed to combine HIV care with buprenorphine administration for persons with HIV who were dependent on opioids. The intervention designs were both acceptable and appropriate for the focus populations and were reasonable in terms of costs and complexity to implement and maintain.

None of the 18 interventions scored the highest potential number of points on all RE-AIM dimensions. All 18 interventions were determined to be best practices (i.e., either evidence-based or -informed) for retention in care by the PRS Project and thus there was less variation in the Efficacy/Effectiveness dimension. Three interventions had tied scores for RE-AIM and two interventions had tied scores for the Equity subscale. Below is a description of a high-, medium-, and low-scoring intervention. The interventions that scored the lowest on the RE-AIM instrument are either evidence-based or evidence-informed and in no way should be considered defective or inappropriate for implementation.

ARTAS [33], the top total RE-AIM scoring intervention, did not obtain the highest tier scores in Reach with 8 interventions scoring higher, Efficacy with 3 interventions scoring higher, and Adoption with 8 interventions scoring higher. However, ARTAS was the highest total RE-AIM scoring intervention because the intervention did have the highest tier scores in Implementation and Maintenance where the most potential points could be awarded. The review panel noted that this intervention, delivered often by social workers, used a strengths-based approach familiar to most social workers/case managers/navigators and could easily be maintained by an implementing agency. The review panel noted that ARTAS had broad applicability to a range of populations with HIV risk factors and implementing venues. There was readily available training and technical assistance, and the panel determined that the intervention was relatively easy and convenient to implement and maintain.

The intervention weCare [44, 45] is an example of an intervention that scored in the medium level of the RE-AIM assessment. The intervention used social media to improve linkage to and retention in care and additional health outcomes among racially and ethnically diverse MSM, ages 13–34, living with HIV. The intervention used social media with which young MSM are familiar, such as text messaging, mobile applications (apps), and Facebook. The intervention had a maximum score on the Reach dimension since the intervention focused on persons at increased risk for HIV infection and transmission, demonstrated sound methods for recruiting and retaining the focus population, and had high levels of participation and completion of the intervention. The intervention had a high score on the Efficacy dimension since the intervention demonstrated reduction in viral load in addition to retention in care. The weCare intervention also demonstrated effects beyond 12 months. The intervention lost points due to study design which resulted in the intervention being determined to be evidence-informed rather than evidence-based. The researchers used community-based participatory research (CBPR) strategies that included considerable input from community members, community-based organization staff, and healthcare/clinical providers. This CBPR approach resulted in high scores in the Adoption dimension for acceptability and appropriateness of intervention activities for both the focus population and the implementing agencies. On the Implementation dimension, weCare scored points for the adaptability of the intervention methodology, the responsiveness to community context, and the ability for the intervention to be replicated by others due to the clear description of all intervention activities in the outcome paper. Moderate scores were obtained however due to lack of cost data, the need for training and technical assistance for other agencies that may attempt to replicate the approach, and reviewers’ consensus that the intervention was not logistically easy to implement. On the Maintenance dimension, weCare scored in the medium range since CBPR formative techniques would need to be continued to keep the intervention culturally current. Reviewers pointed out that training and technical assistance would be needed to maintain consistent delivery of the intervention. Maintaining the intervention was judged to be logistically challenging, especially if implementation resources are reduced after the end of the study period.

Centralized HIV Services [46] was the intervention that scored lowest on our RE-AIM assessments. Due to research design, the intervention was designated as evidence-informed rather than evidence-based. The intervention effects however were impressive with the African American and Hispanic/Latino youth, ages 13–23, exhibited a 34% increase in medical visit constancy when youth were served by adolescent medical specialists, case managers who specialize in youth services, and when additional youth-focused support was provided. This demonstrates that even the intervention that scored lowest on our RE-AIM scale substantially increased retention in care. The intervention scored lowest due to research design and the requirement of adolescent—specialized medical, nursing, and case management staff as well as social and support services that were specifically designed for ethnic and racial minority youth. The intervention would require a clinic to have a substantial caseload of adolescent patients to increase feasibility of implementation. Also, the intervention staff used motivational interviewing with study participants. The reviewers for this intervention were concerned that this technique requires in-depth training and technical assistance, and thus is not easily implemented by new clinic employees unless training and technical assistance on motivational interviewing is ongoing. The intervention lost points in both implementation and maintenance due to the issues with sustainability of motivational interviewing.

Discussion

The current assessment ranked PRS best practices for retention in medical care for PWH using an instrument based on the RE-AIM framework. The RE-AIM framework provided the DHP workgroup with clear dimensions and adaptable scoring criteria that allowed for thorough assessment and ranking for a range of interventions.

Several factors that were not fully accounted for in the intervention score may influence the effectiveness or appropriateness of the intervention for a given clinic or organization. First, the 18 interventions focused on a diverse range of populations, including, for example, men who have sex with men (MSM), economically marginalized populations, persons who were recently released from incarceration, and young racial/ethnic minority populations. Therefore, the most effective or appropriate intervention for a given organization or clinic may be dependent on the specific focus population served. Prioritization and subsequent selection for implementation of interventions should consider the topscoring interventions for each of a variety of key populations on which a clinic may wish to focus retention efforts. The health equity subscale within the RE-AIM assessment also may provide additional information when selecting an intervention for implementation.

The intervention studies also used a variety of retention in care measures such as the number of visits during a defined follow-up period, the interval between visits, or the number or percentage of kept or missed appointments. Some studies used more rigorous criteria to define retention (e.g., at least 3 visits during a 12-month period vs. 2 visits during a 12-month period). Interventions that were demonstrated to be efficacious under the most rigorous criteria for successful retention in care might receive a higher level of recommendation than those with less rigorous retention criteria.

Many of the interventions were shown to be efficacious for multiple outcomes in addition to retention in care, such as linkage to care or viral suppression. Cost-savings might be achieved by implementing interventions that improve outcomes at multiple stages of the continuum of care.

Finally, the interventions used a broad range of strategies to improve retention in care, including paying incentives, incorporating rapid access to ART, quickly linking patients from emergency departments to HIV treatment services, integrating drug treatment with HIV clinical services, providing patient navigation services, offering culturally and linguistically appropriate clinical and social services, incorporating community pharmacists into the HIV care team, conducting social marketing and peer outreach, linking a patient with a supportive partner within their social network, and improving provider-patient communication with supportive comments to patients at clinic visits. The level of investment of these strategies and the resources needed to successfully implement them are highly variable. High investment interventions, such as those that require system or policy changes or many dedicated staff to implement, may not be appropriate in all settings. Those interventions with higher scores in Adoption, Implementation, and Maintenance may have more public health impact and therefore are a better investment. Issues around adaptation of interventions should be prioritized in the context of local resources and needs to ensure maximum impact.

Because of the high variability in focus population and intervention methodology we suggest the development of an intervention selection guide for use by practitioners as the best course of action for national dissemination. This guide would discuss the various methodologies that might be used for a retention in care program. Such a guide would also address health equity by identifying those interventions that demonstrated efficacy with various vulnerable focus populations such as Spanish speaking persons, adolescents, persons with substance use disorders, persons who are incarcerated, Black/African American MSM, and Black/African American transgender women.

Limitations

This report has several limitations. First, the 18 best practices reviewed were identified in the PRS Compendium. It is possible that more recent retention in care interventions were excluded from this review that may have been added as best practices before the publication of this report. Second, most reviewed intervention studies did not explicitly reference dimensions of the RE-AIM framework, though many discernable RE-AIM dimensions were reported. Under-reported dimensions could decrease score validity. For example, journal reporting requirements dictate authors provide an appropriate description of the study population. Thus, information necessary to evaluate reach was readily available to raters. Information on intervention maintenance greater than six months after the conclusion of the intervention was not often reported in the reviewed manuscripts. For some interventions, more than one publication was reviewed which may have impacted the final score. Interventions may have subsequent publications not evaluated by the raters in this report.

Third, the RE-AIM informed prioritization instrument, adapted from RE-AIM dimensions, was developed, and implemented by employees of CDC’s DHP. Subjectivity was reduced by having discussions by the three reviewers for common understanding of the intervention and consistent scoring. It is possible that evaluators such as HIV-clinic staff would have considered other criteria when developing the instrument and rating the interventions. Fourth, DHP currently disseminates, including funding, for four of the interventions, leading to potential scoring bias. Fifth, each intervention study was independently evaluated by a different combination of raters creating the potential for inconsistent scoring. This limitation was potentially mitigated by discussions of intervention scores with the larger group of all raters, thus building consensus around each dimension of the instrument. Sixth, raters scored intervention costs; however, no items or scales were developed or included that would allow detection of societal costs such as any costs incurred by patients to participate in the intervention research. Seventh, we found the RE-AIM framework challenging for this body of interventions since the various study designs used by the researchers did not often indicate the representativeness of individuals who were willing to participate in an intervention. Likewise, interventions that focused on a subset of patients, such as young Black or African American MSM, made it more difficult to determine the actual denominator when making judgements about representativeness. New patients in a clinic may have only received a partial dosage of an intervention that was well in progress when the patient entered treatment. Studies that distinguished between current established patients at the beginning of an intervention compared to new patients were helpful in assessing representativeness. Eighth, interviews with primary authors were not included in our initial methodology. Contacting authors may offer additional information relevant to making determinations as to those interventions best suited for real-world implementation and for addressing health equity.

Conclusions

Clinics and agencies that wish to implement an intervention to increase retention in medical care may: (1) consider the focus population that the clinic serves and select an intervention shown to be efficacious with that population or a similar population, (2) consider whether the intervention should also address linkage to care and select an intervention that was efficacious for linkage as well as retention, (3) consider selecting an intervention that achieved viral suppression in addition to retention, (4) consider selecting an intervention that is suitable for the resources and culture of the clinic that will be implementing the intervention or that may be adapted to local context, (5) consider the intervention agents (e.g., all clinical staff, pharmacists, peers with HIV, navigators, social workers) and whether the clinic has access to persons who fall into these categories, (6) consider a lower investment (time, resources, complexity) intervention before attempting a complex, high investment intervention, and (7) consider implementing a combination of intervention strategies to increase desired effects/outcomes. Prioritization scores can help clinical staff select the strongest interventions that address the needs of their population served by clinics seeking best-fit interventions.