Abstract
This overview of reviews summarizes the evidence from systematic reviews (SR) on the effectiveness of antiretroviral therapy (ART) adherence interventions for people with HIV (PWH) and descriptively compares adherence interventions among key populations. Relevant articles published during 1996–2017 were identified by comprehensive searches of CDC’s HIV/acquired immunodeficiency syndrome (AIDS) Prevention Research Synthesis Database and manual searches. Included SRs examined primary interventions intended to improve ART adherence, focused on PWH, and assessed medication adherence or biologic outcomes (e.g., viral load). We synthesized the qualitative data and used the Assessment of Multiple Systematic Reviews (AMSTAR) for quality assessment. Forty-one SRs met inclusion criteria. Average quality was high. SRs that evaluated text-messaging interventions (n = 9) consistently reported statistically significant improvements in adherence and biologic outcomes. Other ART adherence strategies [e.g., behavioral, directly administered antiretroviral therapy (DAART)] reported improvements, but did not report significant effects for both outcomes, or intervention effects that did not persist postintervention. In the review focused on people who inject drugs (n = 1), DAART alone or in combination with medication-assisted therapy improved both outcomes. In SRs focused on children or adolescents aged <18 years (n = 5), regimen-related and hospital-based DAART improved biologic outcomes. ART adherence interventions (e.g., text-messaging) improved adherence and biologic outcomes; however, results differed for other intervention strategies, populations, and outcomes. Because few SRs reported evidence for populations at high risk (e.g., men who have sex with men), the results are not generalizable to all PWH. Future implementation studies are needed to examine medication adherence interventions in specific populations and address the identified gaps.
Keywords: medication adherence, HIV, systematic review, intervention
Introduction
Since the advent of combination antiretroviral therapy (ART) to treat HIV disease more than two decades ago, a great deal of research has identified effective methods for attaining optimal adherence to ART among people with HIV (PWH). Sustained adherence to ART is necessary to improve clinical outcomes and decrease HIV transmission risk.1 Nonadherence to ART, often due to regimen-specific barriers, psychosocial (e.g., mental health), and structural issues (e.g., lack of housing or health insurance),2–4 is associated with treatment resistance, increased viral load, and increased mortality.5–7
Evidence of effective ART adherence strategies remains mixed, possibly because of differences in adherence measures, populations assessed, intervention outcomes, and intervention strategies. Given the many strategies for improving adherence, it is important to determine which are most effective, and for whom. Identification of effective medication adherence interventions and the needs of specific populations will improve PWHs ability to maintain medication adherence, the “plus,” to reaching the 90–90-90 UNAIDS global challenge.8
The abundance of primary research studies on ART adherence resulted in a body of systematic reviews (SRs). Therefore, an overview of reviews was an ideal approach to examining effective strategies to improve ART adherence while exploring measurement, population, and outcome differences. The two overviews of reviews9,10 examining ART adherence interventions have limitations. Mbuagbaw et al.10 reported favorable findings for mobile-phone text messaging; however, they limited their focus to text messaging. Fu et al.9 examined patient compliance with medication adherence protocols used in ART, but only qualitatively described the findings of 10 SRs.
Examining a larger body of evidence allowed us to summarize the effectiveness of adherence strategies, describe adherence interventions among specific populations, and summarize research gaps.
Methods
We conducted an overview of reviews and used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement (Supplementary Appendix SA1) as a guideline to organize the review.11 We reviewed qualitative SRs and quantitative meta-analyses, published during 1996–2017, that evaluated ART adherence interventions among PWH and assessed adherence [Medication Event Monitoring System (MEMS), pharmacy refill, pill count, or self-report] or biologic (HIV viral load, HIV-1 RNA, or CD4+ count) outcomes. We excluded meta-analyses of individual patient data from studies without a comprehensive systematic search, conference abstracts, overview of reviews, and SRs only examining medication adherence to pre- or postexposure prophylaxis (i.e., PrEP or PEP) or medication adherence for other chronic diseases.
A comprehensive search of the CDC HIV/AIDS Prevention Research Synthesis (PRS) database was performed to identify SRs focused on HIV medication adherence during 1996–2017.12 By the end of January 2018, the PRS database had amassed ∼ 87,000 records related to HIV, AIDS, or sexually transmitted infection prevention research literature.13 Five comprehensive automated searches for HIV prevention literature, primary studies, and SRs are implemented annually using the following databases: MEDLINE (OVID), EMBASE (OVID), PsycINFO (OVID), CAB Global Health (OVID), CINAHL (EBSCOhost), and Sociological Abstracts (Pro- Quest)13 (Supplementary Appendix SA2).
Each automated search was developed in the MEDLINE (OVID) database by using indexing and keyword terms cross-referenced with Boolean logic and no language limits. The final search was tailored to the other databases’ proprietary indexing systems. Supplementary searches for the PRS database include a quarterly manual search of 52 journals (list available from the PRS website), publication alerts, various online nonindexed databases (e.g., Google Scholar, Scopus), gray literature sites (e.g., NY Academy of Medicine), electronic mail lists, listservs, and reference lists from relevant HIV behavioral prevention research literature.
For our review, the librarian searched the PRS database using the PRS registry coding criteria, key words, and index terms. (For the complete PRS database query, see Supplementary Appendix SA3)
Two independent reviewers reviewed the citations (title and abstract) and then used Distiller SR, version 2 (Evidence Partners, Ottawa, Canada)14 to review the full text. Data extracted from SRs that met inclusion criteria were study characteristics (e.g., adherence measurement, intervention strategy, location, and review type), sample characteristics (e.g., target population), medication adherence outcome and measurement tool, summary findings (quantitative or qualitative), study quality, and research gaps. Relevant adherence outcomes included self-report, electronic drug monitors (e.g., MEMSCap), pill count, and pharmacy refill. Relevant adherence-related biologic outcomes included HIV viral load and CD4+ count.
Because inconsistencies in reports of SRs of intervention strategies precluded quantitative analysis, we provide descriptive summaries of findings and research gaps. In addition, intervention effectiveness among key populations (e.g., children and persons who inject drugs) noted in SRs and for some populations in primary studies was descriptively summarized. The 11-item Assessment of Multiple Systematic Reviews (AMSTAR) tool15 was used to assess the quality of each included systematic review. Responses to AMSTAR comprise “Yes,” “No,” “Cannot answer,” or “Not applicable.” We tallied the total number of “Yes” responses for an overall quality score of 0–11. Scores were categorized as high (8–11 points), medium (5–7), and low (0–4) quality.16
Results
Of 578 citations, 350 were excluded at title and abstract level; 187 were excluded after full-text review. A total of 41 studies met our inclusion criteria (Fig. 1). (For study characteristics, AMSTAR quality scores, and research gaps, see Supplementary Appendix SA4)
FIG. 1.
PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Of the 41 SRs, 22 evaluated a single intervention strategy [e.g., behavioral, directly administered antiretroviral therapy (DAART), or pharmaceutical], 12 evaluated any ART adherence strategy (multiple strategy reviews), and 7 were focused on a key population (i.e., youth, persons who use drugs, and persons with co-occurring mental illness). Of the 22 assessed strategies, 9 were based on technology (computer, internet, and mobile phone text messaging); 4 were behavioral (based on motivational interviewing and cognitive behavioral therapy); 2 were based on DAART; 3 were based on patient support and education; 2 were based on pharmacist- related care (e.g., education for medication self-management, follow-up of pharmaceutical therapy); and 2 were based on dosing regimen.
Of the seven SRs focused on key populations, five focused on children or young people aged <24 years, one focused on people who inject drugs, and one focused on persons with co-occurring mental illness.
The majority of the SRs contained primary studies conducted in either the US or non-US countries (n = 31), included randomized controlled trial (RCT) and non-RCT primary studies (n = 21), qualitatively summarized the evidence (n = 26), and were rated high quality (n = 28). Fifteen reviews were meta-analyses.
We present the evidence for each intervention strategy from all reviews with the quality score and research gaps in the following sections. In addition, we qualitatively summarize the existing literature to identify intervention strategies that are effective at improving medication adherence.
Effectiveness, by strategy
Technology-based interventions (computer, phone, text messaging, website, electronic monitoring devices) (n = 9). Six qualitative reviews17–20 and three meta-analyses21–23 examined technology-based medication adherence interventions.
Two meta-analyses21,23 examined text messaging interventions and one examined telephone-delivered interventions (voice landline and mobile telephone).21 Improvements were observed regardless of outcome assessed and length of text messages. One review21 reported a significant effect of text messaging (via mobile phone or pager) for adherence [odds ratio (OR) = 1.39; 95% confidence interval (CI): 1.18— 1.64]. Larger effects were observed when text messages were not sent daily, supported bidirectional communication, included personalized content, and were timed to match participants’ ART dosing. The second meta-analysis23 included the same two studies, which were captured by Finitsis et al.21 and similarly reported favorable effects of text messaging (mobile phone only). Any weekly text messaging [risk ratio (RR) = 0.78; 95% CI: 0.68–0.89] and short weekly text messaging were associated with lower rates of nonadherence. One additional meta-analysis22 that examined telephone- delivered interventions did not observe improvements in adherence [n = 3; standardized mean difference (SMD) = 0.49; 95% CI: –1.12 to 2.11].
A qualitative brief review of 13 RCT studies published during 2013–201517 reported that mobile phone messages using interactive texts and intensive phone-delivered counseling improved adherence and biologic outcomes. Internet website interventions (n = 3) did not result in consistently significant improvements in adherence or biologic outcomes. Of the studies of software tools, four reported significant improvements in both outcomes. A second review24 that examined mobile health (mHealth) interventions in men who have sex with men (MSM) included two studies that measured adherence or biologic outcomes. Because the proportion of MSM included in the two studies was minimal (4% and 41%), we classified Muessig et al.24 as a technology review, not a key population review. mHealth interventions (two-way text messages) were found to improve self-report adherence and the proportion of participants who were virally suppressed. Claborn et al.18 focused on computer-delivered interventions, but because of poor study designs and small samples did not draw conclusions. Two qualitative reviews examined various technology-delivered interventions.19,25 Telephone and text messaging interventions resulted in promising improvements in adherence and biologic outcomes. However, results for electronic reminder devices such as alarms and computer-delivered interventions did not result in consistently statistically significant improvements. The last review20 qualitatively examined the effectiveness of electronic reminder devices. When the devices were implemented as a stand-alone adherence strategy, four of the eight studies reported improvements in adherence, and five of seven studies reported improvements in adherence when other adherence strategies were combined.
In summary, meta-analyses of mobile phone text-messaging interventions reported significant improvements in medication adherence and biologic outcomes. On the contrary, voice telephone-delivered interventions did not improve adherence. Qualitative evidence of results from other forms of technology (e.g., computer, internet) was mixed because studies varied in the technology examined and the populations assessed. AMSTAR quality scores ranged from moderate to high. Research gaps included the need to optimize the effectiveness of text messaging interventions. This includes larger studies with sufficient power to consistently detect intervention effects. In addition, cost- effectiveness data are needed for interventions that target key populations (e.g., youth, women, substance users, and sex workers) and are used in low-resource settings, as are longitudinal studies with longer follow-up.
Behavioral interventions (n=4).
Four SRs26–29 examined behavioral interventions such as cognitive behavioral therapy and motivational interviewing. A meta-analysis26 of behavioral interventions (e.g., cognitive behavioral therapy, stress management, or adherence counseling) reported significant effects in self-report adherence outcomes (OR = 1.50; 95% CI: 1.16–1.94) and borderline significant effects for viral load (OR = 1.25; 95% CI: 0.99–1.59). Intervention effects were larger in studies that used objective (e.g., MEMS) (OR = 1.70; 95% CI: 1.22–2.37) versus self-reported (OR = 1.39; 95% CI: 0.92–1.13) measures of adherence and larger for studies with longer recall periods (2–4 weeks) compared with shorter recall periods (≦7 days) (QB = 3.97; p < 0.05). A second meta-analysis29 examining the benefit of electronic medication monitoring-informed counseling also observed improvements on adherence (SMD = 0.51, 95% CI: 31–0.71) and viral load (OR: 1.35, 95% CI: 1.12–1.63). However, the benefits of monitoring-informed counseling did not persist when monitoring-informed counseling was the only component of the intervention. In multi-component interventions, intervention effects were larger in studies that included behavioral, cognitive behavioral, or motivational counseling (Q = 11.89, p = 0.0006). One qualitative SR27 of motivational interviewing interventions reported improved adherence and biologic outcomes; however, effects were either not significant or did not persist 6 months after baseline. A second qualitative review28 reported favorable adherence outcomes from behavioral counseling and mobile phone or text messaging.
Overall, the evidence for behavioral interventions was favorable for adherence outcomes and mixed for biologic outcomes. AMSTAR scores ranged from moderate to high quality. Research gaps included a lack of targeted interventions in women, structural and provider–patient interventions, high-quality studies with larger sample sizes, cost-effectiveness studies, common measures of adherence, and studies conducted in low-resource settings and clinical settings.
DAART (n = 2).
Two meta-analyses30,31 examined DAART, an intervention in which a health care worker or other health professional observes while a patient swallows the medication. One meta-analysis30 of RCTs reported that DAART did not improve self-reported adherence assessed by missed pills (RR = 1.04; 95% CI: 0.91–1.20; p = 0.55) or viral suppression (RR = 1.02; 95% CI: 0.98–1.06; p = 0.29). In a secondary analysis by Ford et al.,30 DAART improved viral suppression among drug users and homeless populations (RR = 1.31; 95% CI: 1.00–1.71; p = 0.05), but not in the general population (RR = 0.96; 95% CI: 0–75.9; p = 0.24). A second meta- analysis,31 which included RCTs and non-RCTs, reported positive effects for adherence (RR = 1.17; 95% CI: 1.03–1.32) and viral suppression (RR = 1.24; 95% CI: 1.081–41). Intervention effects for viral suppression remained significant when only non-RCTs (RR = 1.32; 95% CI: 1.11–1.58) were examined, but not when RCTs (RR = 1.18; 95% CI: 0.99–1.42) were included in the analysis. This review did not find a statistically significant moderator of the intervention effect estimate, potentially due to low statistical power. According to durability analysis of assessments of intervention effectiveness at least 1 month postintervention, intervention effects did not persist (RR = 0.95; 95% CI: 0.86–1.05).
Overall, DAART interventions were effective for drug users and homeless persons, but durability analysis in general populations found that the effects diminished postintervention. Intervention effects were observed in the meta-analysis examining RCTs and non-RCTs, but not in the meta-analysis limited to RCTs. Study quality of both SRs was high. Research gaps included targeted interventions with finite duration and specific populations, as well as cost-effectiveness and large-scale trials.
Patient support and education interventions (n=3).
Two SRs28,29 and one meta-analysis32 assessed patient support and education. The first qualitative review, which examined ART adherence interventions implemented by community health workers, reported improvements in 13 of 16 studies that measured HIV viral load, CD4+ counts, or both. The most effective interventions were peer education and DAART. The second qualitative review, which assessed interventions implemented by health care professionals, reported improved ART adherence in 10 of 19 studies, but reported inconsistent results for biologic outcomes. Included studies that observed significant findings on one virologic outcome also observed nonsignificant findings for a different outcome or time point, which might also explain the mixed results. Intervention effectiveness also differed according to intervention strategy and population. For example, the most effective patient support and education interventions were those administered to individuals (instead of groups) and focused on practical medication management skills. The least effective interventions were those focused on marginalized populations.
The meta-analysis by Nachega et al.32 examined the effect of community versus facility-based interventions for clinically stable patients with HIV in low- and middle-income countries (LMIC) to improve antiretroviral adherence and viral suppression. No statistically significant difference was observed in optimal ART adherence (RR = 1.02, 95% CI 0.99–1.04) or viral suppression (RR = 1.00, 95% CI 0.98–1.03) between participants enrolled in a community-based ART and facility-based ART program.
Overall, community- and facility-based interventions implemented in LMIC resulted in comparable adherence and biologic outcomes. Interventions administered by community health workers improved biologic outcomes. The results of patient support and education interventions delivered by health care professionals (e.g., nurse, pharmacist) varied according to intervention strategy and population. The quality of the Kenya et al.33 review was medium; the quality of both the Nachega et al.32 and Rueda et al.34 reviews was high. Research gaps included the need for targeted interventions, studies with a larger sample size, and studies that observe improvements in both behavioral and biologic outcomes rather than one or the other.
Pharmacist-related interventions (n = 2).
Two reviews35,36 examined pharmacist-related interventions. Pharmacists may educate patients in medication self-management, follow-up on pharmaceutical therapy, or provide collaborative care for depression. A recent meta-analysis35 examined the effects of pharmacist-provided direct care or pharmacist-assisted pharmacologic therapy. The reported improvements in adherence outcomes (OR = 1.47; 95% CI: 0.81–2.65) and virologic suppression (OR = 1.95; 95% CI: 0.61–6.25) were not significant. A qualitative review36 of pharmacist-related interventions examined the pharmacist’s role in HIV care: ART adherence and viral load improved. The results of studies that assessed CD4+ count were less favorable: CD4+ count improved in two of seven studies in which pharmacists played a central role, but not in studies in which pharmacists played a peripheral role.
Findings were favorable for adherence and viral load outcomes; however, the effects, when quantitatively synthesized, were not significant. The quality of both reviews was high. Research gaps included RCTs with larger samples, cost-effectiveness studies, and interventions in which pharmacists play an expanded or a central role.
Dosing regimen (n = 2).
Two quantitative reviews37,38 compared the effectiveness of differing ART dosing regimens for adherence and biologic outcomes. Nachega et al.37 compared the effectiveness of once- versus twice-daily ART dosing regimen on adherence and biologic outcomes. Adherence was higher for once-daily regimens than for twice- daily regimens (weighted mean difference = 2.55%; 95% CI: 1.23–3.87; p = 0.0002); however, dosing regimens did not affect viral suppression (RR = 1.01; 95% CI: 0.98–1.03; p = 0.57). Increases in adherence were associated with reductions in pill burden for both regimens (Spearman correlation = –0.45; 95% CI: –0.67 to –0.15; p = 0.004); however, in stratified analysis, adherence was significantly associated only with pill burden for twice-daily regimens. In addition, adherence declined over time for twice daily (Spearman correlation = –0.41; 95% CI: –0.64 to –0.11; p = 0.01), but not once-daily regimens (Spearman correlation = –0.37; 95% CI: 0.70–0.09; p = 0.11). Clay et al.38 compared fixed-dose single-tablet regimen (STR) and multiple tablet fixed-dose regimen (MTR) impact on adherence and biologic outcomes. Patients on STR were more adherent than MTR of any frequency (OR: 2.37, 95% CI: 1.68–3.35), twice-daily MTR (OR: 2.53, 95% CI: 1.13–5.66), and once-daily MTR (OR: 1.81, 95% CI: 1.15–2.84). In addition, the relative risk for viral load suppression at 48 weeks was higher (RR: 1.09,95% CI: 1.04–1.15, p = 0.0003). Changes in CD4+ count at 48 weeks were comparable between STR and MTR (SMD: –0.01, 95% CI: –0.14 to 0.11).
Overall, for adherence outcomes, once-daily regimens were more effective than twice-daily regimens; however, both regimens had similar effects on viral load. STRs were also more effective than multiple tablet regimens on adherence, as well as viral suppression. The AMSTAR quality score for both reviews37 was high. The research gap was the need to investigate the virologic impact of switching regimens (e.g., from once-daily single-tablet regimens to once- daily multi-tablets).
Multiple-strategy SRs (n = 12).
Twelve reviews37,39–49 (8 qualitative and 4 quantitative) assessed the effectiveness of medication adherence interventions. One meta-analysis44 of ART medication adherence interventions in sub-Saharan Africa reported that text-messaging interventions (n = 2 studies) significantly reduced the risk of nonadherence (risk difference = –0.10; 95% CI: –0.17 to –0.03), but DAART interventions (n = 2 studies) did not improve viral load (RR = 1.03; 95% CI: 0.78–1.36).
A network meta-analysis (an analytic comparison of intervention studies that compares interventions both directly and indirectly across studies) of intervention studies in Africa37 found that weekly text messaging (OR = 1.65; 95% CI: 1.25–2.18), counseling and text messaging combined (OR = 2.07; 95% CI: 1.22–3.53), usual standard of care plus intensified adherence counseling (OR = 1.46; 95% CI: 1.061–98), and enhanced standard of care plus treatment support (OR = 1.83; 95% CI: 1.36–2.45) improved adherence outcomes when compared with usual standard of care. Similarly, a network meta-analysis of ART adherence interventions conducted in all study settings (global network) and LMIC49 found that text messaging compared to standard of care improved adherence. Only cognitive behavioral therapy [OR = 1.46, 95% credible intervals (CrI): 1.05–2.12] and treatment supporter interventions (e.g., peer-based support or medication managers) (OR = 1.25, 95% CrI: 1.01–1.71) implemented in global network improved viral suppression, but no intervention strategy observed a statistically significant improvement for viral suppression in LMIC. Moderator analysis in the global network suggested intervention effects wane over time for adherence (coefficient estimate on the log-odds scale –0.43, 95% CrI: –0.75 to 0.11) and viral suppression (coefficient estimate on the log-odds scale –0 to 48, 95% CrI: –0.84 to –0.12). The last meta-analysis41 evaluated adherence interventions that included routine viral load monitoring of patients with elevated viral loads. Five primary studies that examined the proportion of virally suppressed patients reported a pooled estimate of 70.5% (95% CI: 56.6–84.4). Three remaining primary studies reported that the intervention resulted in declines in mean viral load.
Eight qualitative reviews reported favorable intervention effects for a single outcome, but few studies reported consistent improvements both in adherence and biologic outcomes. For example, one of the largest and most comprehensive reviews46 examined 49 studies and reported that 27 improved at least one adherence outcome (e.g., MEMS or pill count) but that only 16 studies improved biologic outcomes. Even fewer studies (n = 10) improved both adherence and biologic outcomes. Charania et al.43 identified 10 evidence-based interventions that resulted in improved medication adherence; however, only one reported improved adherence and biologic outcomes. According to the efficacy criteria of the PRS Compendium, the evidence presented for the included interventions was “good” (www.cdc.gov/hiv/dhap/prb/prs/efficacy/ma/criteria/index.html).
Overall, in quantitative multiple-strategy reviews, text messaging, intensive counseling, and enhanced standard of care interventions were effective when compared with usual care. When qualitatively synthesized, ART adherence interventions did not consistently improve both adherence and biologic outcomes, but at least one outcome improved due to the intervention. AMSTAR quality scores for the 13 SRs were 5–11. Common research gaps included cost- effectiveness studies, standard criterion measures of adherence, interventions with long-term follow-up (>1 year), and high-quality interventions (e.g., RCTs) in targeted populations and settings with high rates of HIV.
Effectiveness, by population
Children and young persons (n=5). Five qualitative SRs50–54 of ART adherence interventions among children and young persons (aged ≦24 years) reported improvements in adherence and biologic outcomes, but the intervention effects were not consistently statistically significant. All five reviews reported that regimen-related intervention studies improved virologic outcomes. Arrivillaga et al.,50 who examined comprehensive and adherence-only interventions, reported that adherence-only medical interventions [e.g., insertion of gastrointestinal (GI) tube, dosing regimen] improved adherence and viral load. Comprehensive interventions improved adherence and viral load, especially among children. In a separate review,52 hospital-based DAART among children improved virologic outcomes, but the improvements decreased as time after discharge increased. Medication diaries used by caregivers did not improve ART adherence or biologic outcomes among youth.51 In addition, a qualitative review53 reported improved adherence and biologic outcomes for DAART, insertion of GI tube, and educational programs. The review also found home nursing visits marginally improved adherence, and one-time interventions without ongoing education were insufficient at improving adherence or biologic outcomes. Finally, Shaw et al.54 qualitatively summarized interventions specifically targeted to adolescents and young adults (aged 13–24 years). The majority of the included studies (n = 5) were repeated measures within-group designs that reported descriptive findings with no statistical tests. Of the between-group studies (n = 5), technology-based interventions (e.g., computer, internet, cell phone calls, or text messaging) or interventions implemented in nonadherent youth were most favorable.
Four reviews reported that regimen-related interventions—DAART and GI tube insertion—improved virologic outcomes. Technology-based interventions improved adherence and biologic outcomes among adolescents and young adults. Comprehensive interventions were especially effective among children or young persons with complex needs. However, one review53 found no evidence for medication diaries and a decline in the effectiveness of hospital-based DAART after discharge for virologic outcomes (AMSTAR scores: 6–10). Research gaps included RCTs with larger sample sizes and a need for consistent measures of adherence.
People who use drugs (n = 1).
One qualitative SR55 examined interventions to improve adherence among people who use drugs (PWUD). The review55 examined 45 intervention studies that contained ART adherence, virologic, and immunologic outcomes. Studies that used DAART alone or in combination with medication-assisted therapy (MAT) programs observed the strongest intervention effects. The included studies revealed consistent improvements in adherence, viral load, and increased CD4+ counts at short-term follow-ups. Less consistent data supported other interventions, although some studies revealed significant short-term improvement of outcomes in nurse-delivered multi-component and contingency management (behavioral therapy, including positive reinforcement) interventions. None of the interventions, including those using DAART, demonstrated long-term treatment outcomes postintervention. Although DAART supported adherence and viral suppression among PWUD, adherence diminished after DAART was terminated. More research is needed to examine the importance of booster sessions, ART-naive PWUDs, and the long-term evaluation of interventions.
Overall, short-term interventions that included DAART alone or in combination with MAT improved ART adherence and biologic outcomes. Besides the Binford et al.55 review, eight additional reviews19,27,31,33,44,47,48,56 captured primary studies that included PWUD. Most of these primary studies (∼80%) were included in the Binford et al.55 review. Four primary studies,57–60 not included in the Binford et al.55 review, implemented various medication adherence interventions (e.g., DAART and patient education) and reported significant improvements in adherence and viral load.
Persons living with co-occurring mental illness (n = 1).
One qualitative review,61 which focused on interventions for persons with co-occurring mental illness, included four primary studies that assessed adherence-based interventions. Two of the four studies were RCTs and reported intervention effects on adherence as well as depression. Three of the included studies measured adherence, but only one study reported improved adherence (measured by MEMS cap) up to 12 months postintervention. One study did not report statistically significant improvements in adherence or viral load.
Overall, short-term interventions among persons with co-occurring mental illnesses improved adherence outcomes, but no evidence of effectiveness on viral load was reported. The AMSTAR quality score for this review was poor. Research gaps identified in this review included the need for more research on PWH with co-occurring mental illness and explanatory models of adherence that account for psychiatric illnesses.
Populations identified from primary studies.
Additional evidence on specific populations was provided by primary studies in the included SRs. Populations that were assessed by these primary studies comprised incarcerated persons and treatment-naive persons. Limited evidence was available specifically on women, low-income populations, persons who have fallen out of care, and MSM, although these groups may be represented in clinic-based studies.
Incarcerated populations.
Two SRs31,47 included primary studies that targeted incarcerated persons. In Simoni et al.,47 three primary studies62–64 examining DAART interventions in incarcerated persons observed improvements in self-reported adherence and viral load compared with controls. A second SR by Hart et al.31 only included one study of incarcerated persons,62 which was also captured by Simoni et al.47
Treatment-naive populations.
Treatment-naive populations were included in four SRs.22,31,47,48 Simoni et al.47 assessed three uncontrolled pilot DAART studies targeting treatment-naive persons and reported that the majority adhered to prescribed doses and were virally suppressed (HIV-1 RNA). Hart et al.31 provided stratified data comparing the proportion of the population that was ART naive (<50% vs. ≥50%), but reported no difference (p = 0.42) in DAART effectiveness between studies with individuals <50% ART naive (RR = 1.33; 95% CI: 1.03–1.72; p < 0.0001) versus ≥50% ART naive (RR = 1.12; 95% CI: 1.04–1.21; p = 0.24). Two additional reviews examined telephone-delivered22 and multi-component48 adherence interventions among ART- naive persons. Results were favorable for adherence measured by self-report and MEMS, but not for biologic outcomes.
Discussion
Our overview summarizes the evidence from SRs and meta-analyses on effective strategies to improve ART adherence, intervention effectiveness among key populations, and research gaps. The following are our findings:
Text messaging interventions consistently improved medication adherence and biologic outcomes.
- In key populations, structured interventions, which use well-defined and planned methods to ensure that patients receive their medication, resulted in favorable effects on adherence and biologic outcomes:
- Regimen-related, DAART, and comprehensive interventions were effective among children and young people.
- DAART alone or in combination with MAT programs was most effective among persons who use drugs.
Our findings agree with those of a recent network meta-analysis (not included in this review49) that short message service and text-messaging interventions were superior to standard of care on improving ART adherence. We also found that structured interventions worked well in populations of children, young people, and persons who use drugs. We did not find sufficient evidence that other strategies (based on behavior, pharmacists, technology, or patient education) consistently improved adherence or biologic outcomes. This finding does not indicate these other strategies are ineffective; rather there is a lack of sufficient evidence to determine the effectiveness for them.
Study variability in outcome-assessment measures made it difficult to summarize the evidence on medication adherence interventions. Medication adherence is commonly assessed by subjective self-report and objective adherence measures such as electronic monitoring devices (e.g., MEMS), pill count, refill records, as well as objective biologic measures, including CD4+ count and viral load (HIV-1 RNA). The majority of the included reviews reported both subjective and objective measures; however, few reviews reported statistically significant or consistent (in the case of qualitative reviews) intervention effects on both subjective or objective adherence measures and objective biologic measures. In addition, no reviews compared whether results differed based on subjective adherence measures (e.g., survey) and objective adherence measures (pill count or MEMS cap).
Multiple SRs focused on children or young people.50–52 The evidence among children demonstrated evidence for structured interventions as noted and was promising for other strategies; however, only qualitative synthesis methods were used because the reviews reported differing adherence- outcome measures and differing intervention strategies. All reviews on youth were published before 2013. More evidence from primary studies on effectiveness is likely to be available in this population (e.g., high-risk young persons), but not summarized in published reviews.
Research gaps
Common research gaps, regardless of intervention strategy or population, were identified in the included SRs. Many reviews were based on evidence from studies with small samples (<100), which can affect study quality. Therefore, reviews identified a need for larger controlled primary studies. Other common research gaps include a lack of studies with longer follow-up periods and a lack of cost-effectiveness analyses. Many authors identified a need for culturally tailored interventions for populations at high risk. Although multiple SRs focused on key populations (e.g., children, PWUD, persons with co-occurring mental illness), few reviews included studies focused on MSM, women, pregnant women, prisoners, sex workers, persons with memory impairment, young people with a recent HIV diagnosis, as well as various age groups and ethnicities. Most of the included reviews were focused on multiple intervention strategies rather than a single strategy (e.g., a technology-based or pharmaceutical intervention). The lack of reviews on specific strategies might be due to the lack of published high-quality primary studies that met the rigorous inclusion criteria of the included reviews. Finally, evidence is needed to demonstrate that improved adherence translates into improved biologic outcomes, which can be affected by the adherence measurement and timing of such measurement.
Limitations
We could not quantitatively synthesize evidence because most of the reviews were qualitative.
The various methods of measuring adherence and viral suppression made it difficult to quantitatively summarize the data across all reviews.
A common barrier expressed by the included reviews was the lack of targeted interventions in key populations (e.g., women, MSM, and PWUD) with high rates of HIV. However, many population descriptions were too sparse to identify key populations. In addition, the paucity of data reported by authors did not permit comparison of intervention strategies among populations.
Conclusions and Future Directions
The evidence from SRs of medication adherence interventions demonstrated that text-messaging interventions consistently reported improved adherence and biologic outcomes. On the contrary, behavioral, pharmacist-related, and technology-based intervention strategies had mixed evidence. Our examination of the effectiveness of ART interventions among specific populations (e.g., children, PWUD) revealed that structured interventions (e.g., DAART) resulted in favorable adherence and biologic outcomes. Data on other populations (e.g., MSM) were sparse.
Future implementation studies, which examine medication adherence interventions in certain populations, and their dissemination, are needed. Specifically, future studies might determine which populations respond to current adherence strategies and then prioritize the populations in need of more specifically tailored interventions. Future studies might also address the identified gaps (e.g., need for longer follow-up periods and cost-effectiveness studies). Lessons learned from ART adherence interventions will have important implications for future studies examining PrEP adherence interventions in populations at risk for HIV. As PWH continue to live longer, medication adherence will remain important to ensure that they attain durable viral suppression—for their own health and to prevent transmission to others.
Supplementary Material
Acknowledgments
We thank members of the Prevention Research Synthesis team for their contribution to the PRS database, including Jeff Becasen, Mary Mullins, and Emiko Kamitani. The work of Leslie W. Ross was supported by the Emory University Rollins Earn and Learn (REAL) internship program.
Footnotes
Author Disclosure Statement
No competing financial interests exist.
Disclaimer: Findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
References
- 1.CDC. Recommendations for HIV Prevention with Adults and Adolescents with HIV in the United States, 2014. Atlanta, GA: Centers for Disease Control & Prevention, 2014. [Google Scholar]
- 2.AIDSInfo. Guidelines for the use of antiretroviral agents in HIV-1-infected adults and adolescents. AIDS Info website, 2010. Available at: https://aidsinfo.nih.gov/contentfiles/Ivguidelines/adultandadolescentgl.pdf [Google Scholar]
- 3.Chesney MA. Factors affecting adherence to antiretroviral therapy. Clin Infect Dis 2000;30 Suppl 2:S171–S176. [DOI] [PubMed] [Google Scholar]
- 4.Atkinson MJ, Petrozzino JJ. An evidence-based review of treatment-related determinants of patients’ nonadherence to HIV medications. AIDS Patient Care STDs 2009;23:903–914. [DOI] [PubMed] [Google Scholar]
- 5.Gardner EM, Sharma S, Peng G, et al. Differential adherence to combination antiretroviral therapy is associated with vi- rological failure with resistance. AIDS 2008;22:75–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Maggiolo F, Airoldi M, Kleinloog HD, et al. Effect of adherence to HAART on virologic outcome and on the selection of resistance-conferring mutations in NNRTI- or PI-treated patients. HIV Clin Trials 2007;8:282–292. [DOI] [PubMed] [Google Scholar]
- 7.Lima VD, Harrigan R, Bangsberg DR, et al. The combined effect of modern highly active antiretroviral therapy regimens and adherence on mortality over time. J Acquir Immune Defic Syndr 2009;50:529–536. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Corless IB, Hoyt AJ, Tyer-Viola L, et al. 90–90-90-Plus: Maintaining adherence to antiretroviral therapies. AIDS Patient Care STDS 2017;31:227–236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fu L, Hu Y, Lu HZ. Overviews of reviews on patient compliance with medication protocols used in highly active antiretroviral therapy. Int J Nurs Sci 2015;2:61–65. [Google Scholar]
- 10.Mbuagbaw L, Mursleen S, Lytvyn L, et al. Mobile phone text messaging interventions for HIV and other chronic diseases: An overview of systematic reviews and framework for evidence transfer. BMC Health Serv Res 2015;15: 33.31–33.16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015;4:1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Lyles CM, Crepaz N, Herbst JH, et al. Evidence-based HIV behavioral prevention from the perspective of the CDC’s HIV/AIDS Prevention Research Synthesis Team. AIDS Educ Prev 2006;18(4 Suppl A):21–31. [DOI] [PubMed] [Google Scholar]
- 13.DeLuca JB, Mullins MM, Lyles CM, et al. Developing a comprehensive search strategy for evidence based systematic reviews. Evid Based Libr Inf Pract 2008;3:3–32. [Google Scholar]
- 14.Evidence Partners. DistillerSR (Version 2) [Software] [computer program]. 2016. Available at: https://v2.systematic-review.ca
- 15.Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Flodgren G, Eccles MP, Shepperd S, et al. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev 2011:Cd009255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Amico KR. Evidence for technology interventions to promote ART adherence in adult populations: A review of the literature 2012–2015. Curr HIV/AIDS Rep 2015;12:441–450. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Claborn KR, Fernandez A, Wray T, et al. Computer-based HIV adherence promotion interventions: A systematic review. Transl Behav Med 2015;5:294–306. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Pellowski JA, Kalichman SC. Recent advances (2011–2012) in technology-delivered interventions for people living with HIV. Curr HIV/AIDS Rep 2012;9:326–334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Wise J, Operario D. Use of electronic reminder devices to improve adherence to antiretroviral therapy: A systematic review. AIDS Patient Care STDs 2008;22:495–504. [DOI] [PubMed] [Google Scholar]
- 21.Finitsis DJ, Pellowski JA, Johnson BT. Text message intervention designs to promote adherence to antiretroviral therapy (ART): A meta-analysis of randomized controlled trials. PLoS One 2014;9:e88166. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Gentry S, van-Velthoven MH, Tudor Car L, et al. Telephone Delivered Interventions for Reducing Morbidity and Mortality in People with HIV Infection. Oxford, England: John Wiley, 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Horvath T, Azman H, Kennedy GE, et al. Mobile Phone Text Messaging for Promoting Adherence to Antiretroviral Therapy in Patients with HIV Infection. Oxford, UK: Wiley, 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Muessig KE, Grand S, Horvath KJ, et al. Recent mobile health interventions to support medication adherence among HIV-positive MSM. Curr Opin HIV AIDS 2017;12: 432–441. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Lima IC, Galvao MT, Alexandre Hde O, et al. Information and communication technologies for adherence to antiretroviral treatment in adults with HIV/AIDS. Int J Med Inform 2016;92:54–61. [DOI] [PubMed] [Google Scholar]
- 26.Simoni JM, Pearson CR, Pantalone DW, et al. Efficacy of interventions in improving highly active antiretroviral therapy adherence and HIV-1 RNA viral load. A meta-analytic review of randomized controlled trials. J Acquir Immune Defic Syndr 2006;43 Suppl 1:S23–S35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Hill S, Kavookjian J. Motivational interviewing as a behavioral intervention to increase HAART adherence in patients who are HIV-positive: A systematic review of the literature. AIDS Care 2012;24:583–592. [DOI] [PubMed] [Google Scholar]
- 28.Robbins RN, Spector AY, Mellins CA, et al. Optimizing ART adherence: Update for HIV treatment and prevention. Curr HIV/AIDS Rep 2014;11:423–433. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Langebeek N, Nieuwkerk P. Electronic medication monitoring-informed counseling to improve adherence to combination anti-retroviral therapy and virologic treatment outcomes: A meta-analysis. Front Public Health 2015;3: 139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Ford N, Nachega JB, Engel ME, et al. Directly observed antiretroviral therapy: A systematic review and meta-analysis of randomised clinical trials. Lancet 2009;374: 2064–2071. [DOI] [PubMed] [Google Scholar]
- 31.Hart JE, Jeon CY, Ivers LC, et al. Effect of directly observed therapy for highly active antiretroviral therapy on virologic, immunologic, and adherence outcomes: A meta-analysis and systematic review. J Acquir Immune Defic Syndr 2010;54:167–179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Nachega JB, Adetokunboh O, Uthman OA, et al. Community-based interventions to improve and sustain antiretroviral therapy adherence, retention in HIV care and clinical outcomes in low- and middle-income countries for achieving the UNAIDS 90–90-90 targets. Curr HIV/AIDS Rep 2016;13:241–255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Kenya S, Chida N, Symes S, et al. Can community health workers improve adherence to highly active antiretroviral therapy in the USA? A review of the literature. HIV Med 2011;12:525–534. [DOI] [PubMed] [Google Scholar]
- 34.Rueda S, Park-Wyllie LY, Bayoumi AM, et al. Patient Support and Education for Promoting Adherence to Highly Active Antiretroviral Therapy for HIV/AIDS. Chichester, UK: The Cochrane Library/John Wiley & Sons, 2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Rocha BS, Silveira MP, Moraes CG, et al. Pharmaceutical interventions in antiretroviral therapy: Systematic review and meta-analysis of randomized clinical trials. J Clin Pharm Ther 2015;40:251–258. [DOI] [PubMed] [Google Scholar]
- 36.Saberi P, Dong BJ, Johnson MO, et al. The impact of HIV clinical pharmacists on HIV treatment outcomes: A systematic review. Patient Prefer Adherence 2012;6:297–322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Nachega JB, Parienti JJ, Uthman OA, et al. Lower pill burden and once-daily antiretroviral treatment regimens for HIV infection: A meta-analysis of randomized controlled trials. Clin Infect Dis 2014;58:1297–1307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Clay PG, Nag S, Graham CM, et al. Meta-analysis of studies comparing single and multi-tablet fixed dose combination HIV treatment regimens. Medicine 2015;94:e1677. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Fogarty L, Roter D, Larson S, et al. Patient adherence to HIV medication regimens: A review of published and abstract reports [comment]. Patient Educ Counsel 2002;46: 93–108. [DOI] [PubMed] [Google Scholar]
- 40.Barnighausen T, Chaiyachati K, Chimbindi N, et al. Interventions to increase antiretroviral adherence in sub-Saharan Africa: A systematic review of evaluation studies. Lancet Infect Dis 2011;11:942–951. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Bonner K, Mezochow A, Roberts T, et al. Viral load monitoring as a tool to reinforce adherence: A systematic review. J Acquir Immune Defic Syndr 2013;64:74–78. [DOI] [PubMed] [Google Scholar]
- 42.Chaiyachati KH, Ogbuoji O, Price M, et al. Interventions to improve adherence to antiretroviral therapy: A rapid systematic review. AIDS 2014;28(Suppl. 2):S187–S204. [DOI] [PubMed] [Google Scholar]
- 43.Charania MR, Marshall KJ, Lyles CM, et al. Identification of evidence-based interventions for promoting HIV medication adherence: Findings from a systematic review of U.S.-based studies, 1996–2011. AIDS Behav 2014;18:646–660. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Mathes T, Pieper D, Antoine SL, et al. Adherence-enhancing interventions for highly active antiretroviral therapy in HIV-infected patients—A systematic review. HIV Med 2013;14:583–595. [DOI] [PubMed] [Google Scholar]
- 45.Mathews T, Antoine SL, Pieper D. Adherence-enhancing interventions for active antiretroviral therapy in sub-Saharan Africa: A systematic review and meta-analysis. Sex Health 2014;11:230–239. [DOI] [PubMed] [Google Scholar]
- 46.Mbuagbaw L, Sivaramalingam B, Navarro T, et al. Interventions for enhancing adherence to antiretroviral therapy (ART): A systematic review of high quality studies. AIDS Patient Care STDs 2015;29:248–266. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Simoni JM, Frick PA, Pantalone DW, et al. Antiretroviral adherence interventions: A review of current literature and ongoing studies. Topics HIV Med 2003;11:185–198. [PubMed] [Google Scholar]
- 48.Yang Y State of the science: The efficacy of a multi-component intervention for ART adherence among people living with HIV. J Assoc Nurses AIDS Care 2014;25:297–308. [DOI] [PubMed] [Google Scholar]
- 49.Kanters S, Park JJ, Chan K, et al. Interventions to improve adherence to antiretroviral therapy: A systematic review and network meta-analysis. Lancet HIV 2017;4:e31–e40. [DOI] [PubMed] [Google Scholar]
- 50.Arrivillaga M, Martucci V, Hoyos PA, et al. Adherence among children and young people living with HIV/AIDS: A systematic review of medication and comprehensive interventions. Vulnerable Children Youth Stud 2013;8: 321–337. [Google Scholar]
- 51.Bain-Brickley D, Butler LM, Kennedy GE, et al. Interventions to Improve Adherence to Antiretroviral Therapy in Children with HIV Infection. London, England: John Wiley & Sons, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Reisner SL, Mimiaga MJ, Skeer M, et al. A review of HIV antiretroviral adherence and intervention studies among HIV-infected youth. Topics HIV Med 2009;17:14–25. [PMC free article] [PubMed] [Google Scholar]
- 53.Simoni JM, Montgomery A, Martin E, et al. Adherence to antiretroviral therapy for pediatric HIV infection: A qualitative systematic review with recommendations for research and clinical management. Pediatrics 2007;119: e1371–e1383. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Shaw S, Amico KR. Antiretroviral therapy adherence enhancing interventions for adolescents and young adults 13–24 years of age: A review of the evidence base. J Acquir Immune Defic Syndr 2016;72:387–399. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Binford MC, Kahana SY, Altice FL. A systematic review of antiretroviral adherence interventions for HIV-infected people who use drugs. Curr HIV/AIDS Rep 2012;9:287–312. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Simoni JM, Pearson CR, Pantalone DW, et al. Efficacy of interventions in improving highly active antiretroviral therapy adherence and HIV-1 RNA viral load: A meta-analytic review of randomized controlled trials. J Acquir Immune Defic Syndr 2006;43(Suppl. 1):S23–S35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.McCance-Katz EF, Gourevitch MN, Arnsten J, et al. Modified directly observed therapy (MDOT) for injection drug users with HIV disease. Am J Addict 2002;11:271–278. [DOI] [PubMed] [Google Scholar]
- 58.Smith-Rohrberg D, Mezger J, Walton M, et al. Impact of enhanced services on virologic outcomes in a directly administered antiretroviral therapy trial for HIV-infected drug users. J Acquir Immune Defic Syndr 2006;43 Suppl 1:S48–S53. [DOI] [PubMed] [Google Scholar]
- 59.Pradier C, Bentz L, Spire B, et al. Efficacy of an educational and counseling intervention on adherence to highly active antiretroviral therapy: French prospective controlled study. HIV Clin Trials 2003;4:121–131. [DOI] [PubMed] [Google Scholar]
- 60.Rotheram-Borus MJ, Swendeman D, Comulada WS, et al. Prevention for substance-using HIV-positive young people: Telephone and in-person delivery. J Acquir Immune Defic Syndr 2004;37 Suppl 2:S68–S77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Palmer NB, Basinski JR, Uldall KK. Psychiatric illness, access and adherence to HAART: A brief review of recent findings and implications for care. HIV Ther 2010;4:215–230. [Google Scholar]
- 62.Babudieri S, Aceti A, D’Offizi GP, et al. Directly observed therapy to treat HIV infection in prisoners. JAMA 2000; 284:179–180. [DOI] [PubMed] [Google Scholar]
- 63.Kirkland LR, Fischl MA, Tashima KT, et al. Response to lamivudine-zidovudine plus abacavir twice daily in antiretroviral-naive, incarcerated patients with HIV infection taking directly observed treatment. Clin Infect Dis 2002;34:511–518. [DOI] [PubMed] [Google Scholar]
- 64.Lanzafame M, Trevenzoli M, Cattelan AM, et al. Directly observed therapy in HIV therapy: A realistic perspective? J Acquir Immune Defic Syndr 2000;25:200–201. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.