Skip to main content
Translational Behavioral Medicine logoLink to Translational Behavioral Medicine
. 2015 Jun 3;5(3):294–306. doi: 10.1007/s13142-015-0317-0

Computer-based HIV adherence promotion interventions: a systematic review

Translation Behavioral Medicine

Kasey R Claborn 1,3,, Anne Fernandez 1, Tyler Wray 1, Susan Ramsey 2
PMCID: PMC4537461  PMID: 26327935

Abstract

Researchers have instituted a range of methodologies to increase access to HIV adherence interventions. This article reviews studies published through January 2014 utilizing computer-based delivery of such interventions to persons living with HIV. A systematic review of five databases identified ten studies (three RCTs, three pilot studies, three feasibility studies, and one single-group trial) that met the inclusion criteria. Descriptions of the interventions’ content and characteristics are included. Interventions varied widely in terms of program structure, theoretical framework, and content. Only six studies reported medication adherence outcomes. Of these, four (five RCTS and one single group pre-post test) reported significant improvement in adherence using various measures, and two approached significance. Results suggest that computer-delivered adherence interventions are feasible and acceptable among both HIV-positive adolescents and adults. Definitive conclusions regarding clinical impact cannot be drawn due to the small number of adequately powered randomized trials in this review. Additional randomized controlled research is needed to draw inferences regarding intervention efficacy.

Keywords: Computer-based intervention, HIV, Adherence, eHealth

INTRODUCTION

The CDC estimates that 1.1 million people living in the USA are infected with HIV [1], with approximately 50,000 new infections per year [2]. Antiretroviral therapy (ARV) is highly effective and is allowing persons living with HIV (PLWH) to have longer, healthier lives [3, 4]. ARV decreases the replication of the virus in an infected person’s blood. Viral load refers to the amount of HIV in the blood and is measured by the number of copies of HIV per milliliter of blood. Viral suppression occurs when the copies of HIV are at very low levels in the blood, which reduces the likelihood of transmitting the virus to other people and decreases morbidity among the infected individual. Despite the effectiveness of ARV, only a quarter of those with HIV are keeping the virus under control successfully [5]. Medication non-adherence is a significant contributor to the low rate of successful viral suppression. A recent meta-analysis found that only an estimated 59 % of participants in North American studies were adherent at a commonly accepted minimal threshold for successful viral suppression [6]. Without a high level of adherence, the HIV-infected individual is at greater risk for unsuccessful viral suppression, disease progression, and shortened lifespan [79]. In addition, low levels of adherence increase the risk of infecting others and contribute to the development of treatment-resistant strains of HIV [7, 8].

Given the significant public health problem presented by poor adherence to ARV, a great deal of research has been devoted to improving adherence. Interventions have been developed to address this significant public health problem, with most studies showing some degree of success at improving adherence [9]. These interventions have become increasingly complex, oftentimes entailing intensive individual counseling over multiple sessions, expert facilitated group sessions, and automated text messaging programs [10]. Despite the demonstrated efficacy of ARV adherence interventions, implementation into real-world clinical settings has been severely limited by the resources required to initiate and maintain the interventions [10]. These interventions require significant staff time, training, and ongoing supervision that are simply not feasible in most HIV clinical settings [10].

In the USA, we have a relative abundance of resources, yet still have difficulty implementing evidence-based ARV improvement interventions. Health-care providers are often faced with time constraints and lack of support and resources within the organizational structure [11, 12]. Furthermore, providers often receive inadequate training in behavioral interventions and, consequently, have demonstrated low self-efficacy in the delivery of these interventions [11, 13]. To implement interventions with fidelity to evidence-based approaches, additional training and supervision are necessary. Resource-limited countries experience additional barriers to implementation of behavioral interventions. These barriers include HIV-related stigma present in the community and among health-care workers, socioeconomic factors, negative perceptions and attitudes about ARV, and lack of knowledge [1417]. Rigorous implementation studies in HIV adherence interventions are lacking; however, global themes limiting adoption of these interventions appear to be due to infrastructure deficits and diffusion of knowledge within organizational structures [18]. The use of technology has the potential to overcome these barriers and lead to widespread adoption of efficacious interventions [19].

As a result of the need for more readily disseminable, low-cost interventions, there has been interest in the development of behavioral intervention technologies (BITs) to promote ARV adherence [20]. A myriad of technology devices have been developed and tested for health behavior change including the use of pagers, smartphones, text messaging, videos, and computer-based programs [21]. BITs have the potential to improve dissemination and implementation of evidence-based treatments, both domestically and abroad. Further, they may extend access to treatment, promote treatment engagement, and build on skills for adherence. Finally, BITs could expand interventions to rural settings, serve as an inexpensive option for supplementary and/or primary care, and promote adherence among hard to reach populations who are overly represented among PLWH (e.g., low socioeconomic status, homeless, pregnant women, sex workers, and substance users) and frequently fall out of care [19, 20].

BITs should match the usage pattern of the target population. The heterogeneity of HIV-infected populations reduces the generalizability of use of all BIT modalities. For example, over 90 % of people have access to cellular phones, but fewer use smartphone devices, and oftentimes, among lower income populations, service may be disconnected at certain times throughout the month [22]. This limits the general usability of mobile health applications and text messaging interventions. Further, the “digital divide” remains a concern for use of BITs among some generational cohorts, socioeconomic strata, and ethnic/racial minorities [23]. These populations are overly represented in HIV clinics; however, there remains a dearth in the literature as to the use of such tools among high risk groups [23, 24].

Consequently, technology-delivered interventions should use a modality that is generalizable to such groups. These challenging populations may be better served in the clinic setting by delivering BITs in an environment that allows for assistance from staff. Further, capitalizing on existing clinic resources will reduce the lag time between intervention development and widespread implementation. Desktop computers are widely available in clinics within the USA. Additionally, populations who are less digitally savvy may feel more comfortable and have a higher level of familiarity with desktop computers compared to smartphones or tablet devices [25]. The use of desktop computer-delivered interventions may be more readily implemented within existing HIV clinic structures.

The primary aim of this article is to examine the state of the science regarding existing desktop computer- and web-based HIV adherence interventions within the empirical literature. Specifically, we aim to: (a) synthesize the content and theoretical frameworks of existing computerized adherence interventions and (b) describe the structure and tailoring methods of each intervention. Additionally, this study seeks to describe the current limitations of this growing body of research and to consider future directions to promote development of eHealth technologies as a method of improving health among PLWH.

METHODS

Search strategy and selection of studies

We systematically reviewed published studies that examined computer- and web-based interventions for adherence among PLWH by applying the PRISMA criteria [26]. The PRISMA statement provides a set of guidelines to be used when conducting and reporting systematic reviews (see Appendix). Searches were conducted within five databases including PsycINFO, PubMed, CINAHL, Medline, and Web of Science through January 2015. The search was conducted using the following terms to ascertain relevant articles: (“computer” OR internet” OR “web” OR “electronic” OR “mobile”) AND (“HIV”) AND (“ARV” OR “antiretroviral” OR “adherence”). Articles were included if they met the following a priori criteria: (a) the target population for the study was PLWH, (b) the study consisted of a behavioral intervention targeting medication adherence, and (c) a computer was the method of intervention delivery. Given the small number of randomized controlled trials available to date, the search was expanded to include non-randomized studies (e.g., pre–post-experimental designs). Studies were excluded if they were not available in the English language. No date range parameters were included in the search terms. Articles were searched through January 1st, 2015, and the resulting articles date back to 2011.

Coding of computer-based intervention characteristics

We defined computer-based interventions as a desktop computer-delivered intervention that could be completed either onsite at a clinic or at the participant’s home. The content of each intervention was categorized into three broad categories (adherence building modules, self-efficacy modules, and other health-related behavior modules) and divided into components within those categories (see Table 3). Three researchers independently coded the content of the interventions, and details of each study were independently coded by three researchers. Operational definitions were provided in a codebook to ensure consistent and accurate categorization throughout the coding process. Participant characteristics included (a) age, (b) proportion of women in the sample, (c) race/ethnicity, (d) sample type, and (e) country in which the study was conducted. Design characteristics included (a) type of design (e.g., randomized controlled trial), (b) type of control group (e.g., standard care), (c) recruitment site (e.g., HIV clinic), (d) measure of adherence (e.g., viral load, self-report), and (e) follow-up assessment timeframe. Intervention characteristics included (a) theoretical framework that guided the intervention, (b) number and length of sessions, (c) rate of attrition from baseline to follow-up, (d) inclusion of multimedia, (e) delivery setting (e.g., clinic, home), (f) adherence building components, (g) self-efficacy promotion components, and (h) inclusion of other health behavior components (e.g., depression). Satisfactory inter-coder reliability was established (k = 0.81). A code was considered a match if coders had an exact match on 149 variables of the data extraction. Any disparities in judgment that emerged during the coding process were resolved through discussion. We contacted the authors and asked them to provide additional information regarding program content that was not included in the original article. Of the nine authors contacted, eight responded and supplied the requested information. The author who did not respond to the original email was re-contacted.

Table 3.

Coded intervention components by study

Intervention feature Claborn et al. [27] Côté et al. [28] Fisher et al. ([29]2011) Hersch et al. [30] Horvath et al. [31] Naar-King et al. [32] Ownby et al. [33] Remien et al. [34] Shegog et al. [35]
Adherence-related modules
Education General ARV education Feedback for adherence HIV and ARV education General ARV education HIV and ARV education ARV education HIV and ARV education HIV and ARV education Yes
Pill-taking strategies Self-cuing and adaptive cognitions No Taking difficult pills Self-cuing and adaptive cognitions No No Self-cuing strategies Yes Yes
Reminder devices Describes alarms No Describes devices No Provided reminder No Describes devices Yes Describes devices
Scheduling doses Yes Yes Yes Yes Yes No Yes Yes Yes
Analysis of missed doses Assessment of missed doses Log book – identify undesired effects Assessment of missed doses No No Assess barriers of adherence and how to overcome No Assess barriers of adherence and how to overcome Assess barriers of adherence and how to overcome
Side effects Yes Yes Yes Yes No No No Yes Yes
Patient-provider communication strategies Cognitive techniques Active listening, expressing emotions Asking providers’ questions Assertiveness techniques No No No Asking providers’ questions Yes
Skills building modules
Addressing HIV-related myths No No Yes No No No Yes Yes Yes
Positive reinforcement Self-reinforcement Self-reinforcement Personalized reward certificate Self-reinforcement No Self-reinforcement No Self-reinforcement Verbal/text-based reinforcement
Problem-solving skills Articulate goals, identify barriers, make a plan Decision rules Learn to identify, understand and overcome barriers Identify barriers and develop a plan to overcome barriers No Identify barriers and solutions to overcome barriers No Define the problem, decide on a goal, develop a plan Yes
Building social support No Identify and mobilize support resources No Yes Support from online forum No Enlisting social support to help with medications Social support network tree, and “support buddy” Yes
Self-awareness Yes Yes Yes Yes Yes Yes No Yes Yes
Other health-related barriers to adherence
Stress management No Cognitive techniques, meditation Stress reduction strategies Self-assessment, strategies No No No No Role model video shows impact on adherence
Depression No No No No No No Resources No Role model video
Alcohol and drug use No No Effects of alcohol and drug use Self-assessment, tips for reducing No No Resources Assessment and referral No
List of resources No Healthcare and community resources Mental health and housing assistance Links to national organizations and websites Links to HIV-related websites No No No Yes

Assessment of risk of bias of studies

Two researchers independently assessed risk of bias in the included randomized controlled trials (n = 6). Reliability among coders was assessed on a pilot sample of articles. Discrepancies in coding were discussed with a third author. Risk of bias was assessed for the included randomized controlled trial as outlined by Higgins and colleagues [36].

RESULTS

Results of the systematic literature search are displayed in Fig. 1. A total of 870 articles were identified after duplicate references were deleted (n = 1021). Articles were excluded if: (a) the article was unavailable in English (n = 0), (b) the study did not target ARV adherence (n = 369), (e) the article was a review or and meta-analysis (n = 272), (c) the study did not test a behavioral intervention (n = 145), and (d) the intervention was not delivered by a computer (n = 56) (see Fig. 1 for study flow diagram). A total of 10 studies [2735, 37] were included in the final review (see Table 1). Five journals were represented among the fields of medicine and psychology. Table 1 provides a description of the study designs and methodology.

Fig 1.

Fig 1

Study flow diagram

Table 1.

Characteristics of included studies

Randomized controlled trials Non-randomized controlled trials
Claborn et al. [27] Côté et al. [28] Fisher et al. [29] Hersch et al. [30] Horvath et al. [31] Naar-King et al. [32] Outlaw et al. [37] Ownby et al. [33] Remien et al. [34] Shegog et al. [35]
Design RCT, pilot Online RCT RCT RCT RCT, pilot RCT, pilot Feasibility Single group Feasibility Feasibility
Sample N = 97 (nc = 50, nt = 47) N = 232 N = 594 (nc = 304, nt = 290) N = 168 (nc = 79, nt = 89) N = 123 (nc = 57, nt = 67) N = 76 (nc = 40, nt = 36) N = 10 N = 124 N = 27 N = 10
Adherence criterion <95 % adherence in past 30 days Prescribed ARV for at least 6 months Prescribed ARV at study inception Detectable VL (defined as >48) <100 % adherence in past 30 days Initiating ARV Initiating ARV None Nonadherent None
Location USA Canada USA USA USA USA USA USA South Africa USA
Type of control group TAU List of websites Assessment control Wait-list control TAU Active nutrition N/A N/A N/A N/A
Framework used CBT SCT IMB CBT IMB MET MET IMB SAT MET
Recruitment site Outpatient clinic Online ads, referrals Outpatient clinic Outpatient clinic Online ads Outpatient clinic Outpatient clinic Outpatient clinic Outpatient clinic Outpatient clinic
Attrition rate baseline to FUP 24 % N/A 4 % 3 % 10 % 8 % 0 % 12 % 15 % 0 %
Adherence measure Self-report Self-report Self-report, VL Self-report, VL Self-report Self-report, VL None MEMS None Self-report
Age group (M) >18 (44) >18 (N/A) >18 (47) >18 (46) >18 (43) 16-24 (20) 18-24 (20) >18 (47) >18 (37) 14-22 (17)
% Women 16.5 % N/A 39 % 21 % 0 % 19.7 % 20 % 29 % 74 % 84 %

N number of participants, nc number of participants in control condition, nt number of participants in treatment condition, ARV antiretroviral therapy, VL viral load, TAU treatment as usual, CBT cognitive behavioral therapy, SCT social cognitive theory, IMB information motivation behavior model, SAT social action theory, MET motivational enhancement therapy

Methodological quality of studies

The risk of bias assessment revealed minor methodological issues with some studies included in this review. Figure 2 provides an overview of the quality of the randomized controlled trials included in this review. These results should take into consideration that half of the articles coded were pilot studies. The process of randomization was unclear for two studies, and allocation concealment was not clearly reported for three studies. Several studies used computerized methods for randomization and assessment procedures which limited risk of bias. The articles reviewed lacked of reporting of power calculations, and only three studies reported effect sizes. Further, the studies lacked consensus in defining ARV adherence which limits the ability to compare study outcomes among the RCTS.

Fig 2.

Fig 2

Risk of bias summary for included randomized controlled trials

Program structure

The interventions exhibited variability in structure and method of delivery (see Table 2). All of the interventions were tailored to PLWH; however, the target population in three studies was youth [32, 35, 37], one study targeted adult MSM [31], and the remaining six targeted adults [2730, 33, 34]. A majority of the studies had participants complete the intervention within the clinic setting (n = 6) [27, 29, 3234, 37], whereas three studies [28, 30, 31] allowed participants to complete the intervention at home, and one study [35] required participants to complete sessions in both locations. One group of interventions [27, 30, 33, 35] consisted of a single session (n = 4). The remaining six studies varied in number of sessions (range 2–18), with two studies [29, 31] allowing participants to choose the number of sessions to complete. Finally, one study [34] used the computerized intervention in conjunction with face-to-face delivery. The six session interactive multimedia program was designed to increase treatment fidelity among lay adherence counselors to the South African health policy for adherence counseling.

Table 2.

Coded intervention features by study

Intervention feature Claborn et al. [27] Côté et al. [28] Fisher et al. [29] Hersch et al. [30] Horvath et al. [31] Naar-King et al. [32] Outlaw et al. [37] Ownby et al. [33] Remien et al. [34] Shegog et al. [35]
Dissemination channel Desktop Internet Desktop Internet Internet Desktop Desktop Touchscreen Laptop Desktop
Delivery setting Clinic Home Clinic Home Home Clinic Clinic Clinic Clinic Botha
Standardized vs. tailored Standardized Tailored Tailored Standardized Standardized Tailored Tailored Tailored Tailored Tailored
Length of Intervention (min) 33 min 80–120 min 26 min (mean) 100–200 (min) Varied 30 min/session 30 min/session 60 min 45 min/session Varied up to 1 h
Number of sessions 1 4 Varied (mode = 6) 1 Varied 2 2 1 6 1
Time between sessions N/A 1 week At least 1 month N/A Daily 1 month 1 month N/A 1 week N/A
Audio narration Yes Yes Yes Yes No Yes Yes Yes Yes Yes
Knowledge check Quiz No Yes No No No No Yes Yes Yes
Videos Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Inclusion of face-to-face sessions No No No No No No No No Yes No

aParticipants primarily accessed the program via hospital computers during their clinic visits; however, there were two participants who completed the program during a home visit using laptop computers on which the application was installed

Intervention content

The content of the interventions differed from each other. Table 3 lists the content components most commonly included in the interventions. The interventions varied in theoretical orientation. Two studies [27, 30] were developed on cognitive behavior therapy principles, three studies [29, 30, 33] utilized the information-motivation-behavior model, and two studies [32, 35] were developed within a motivational interviewing/motivational enhancement framework. The remaining two studies were developed based on either Bandura’s social cognitive theory [28] or social action theory [34]. The content of the interventions fell broadly within three categories: adherence building, improving self-efficacy, and other health-related behaviors. These categories were further subdivided to examine specific components within each group.

Tailored versus standardized interventions

Although all of the interventions were developed with a specific target group in mind, six programs [28, 29, 3235] were tailored to the unique needs of individuals instead of providing a standardized intervention for the group. Cote and colleagues [28] defined tailored interventions as “change strategies intended for a given person based on their specific characteristics, identified beforehand through an evaluation” (p. 2). Tailored interventions are expected to contribute to higher rates of engagement in the intervention and more significant change in the target behavior. For the purposes of this review, tailored was defined as an intervention tailored to unique characteristics of the participants (e.g., gender, adherence level) and standardized was defined as the same intervention modules given to all participants.

Among the six tailored interventions, participants completed questions either prior to or throughout the course of the intervention that allowed intervention tailoring to address the participant’s individual needs. The methods of tailoring varied among the studies. Table 4 describes the tailoring methodology of each relevant intervention. In general, the intensity of tailoring varied across the studies. One study [33] provided automated responses using the participants’ name and branched to different content based on participant responses. Other studies (n = 4) [28, 29, 34, 35] provided targeted interventions based on participant responses regarding adherence-related barriers and then provided activities to reinforce positive behavior change. Another group of studies (n = 3) [25, 29, 32] allowed participants autonomy over module completion, allowing participants to choose to skip components of the intervention or choose modules to complete from a menu of options.

Table 4.

Method of tailoring selected interventions

Article Individualized assessment Description of tailoring methodology
Côté et al. [28] Prior to randomization A virtual nurse delivers tailored teaching based on the degree of adherence (>95 %, 85–94 %, or <85 %) and provides feedback and positive reinforcement on the participant’s personal style and methods and on the acquired skills. The program is personalized as a function of their needs and characteristics.
Fisher et al. [29] During program; prior to adherence modules Participants were offered targeted adherence promotion strategies that addressed individualized barriers. Then participants selected an intervention activity from the list of suggestions, engaged in the activity, and chose an adherence-related goal.
Naar-King et al. [32] During program; prior to adherence modules Participants choose 1 of 7 avatars, are routed through arms of the program based on their ratings of importance and confidence and choices for goal setting, receive personalized feedback and ARV information based on their recent medical information and response to an ARV questionnaire, may choose to read through the intervention or have an audio narrator; and have the choice to skip informational components and pick among a menu of options for goal setting.
Ownby et al. [33] Throughout the course of the program Provided automated responses using the participant’s name; assesses participant learning throughout the intervention and branches to different content based on participant response.
Remien et al. [34] Throughout the course of the program Participants met with an in-person adherence counselor throughout each session who guided the participant through the computerized modules. Participants entered information into the program including social support network and treatment regimen. Then participants choose an adherence barrier from a menu of options and create an action plan.
Shegog et al. [35] Prior to initiating the program Intervention activities are tailored on reported missed doses and blood counts, and psychosocial factors of perceived importance, self-efficacy, and intentions regarding adherence. A “risk profile” is developed and activity clusters are made available to the participant based on the profile.

ARV antiretroviral therapy

Adherence-related modules

The content within the adherence building modules was categorized into seven components. While all of the studies included general education regarding adherence to HIV medications, there was variability in the degree to which additional adherence-related modules were included. For instance, some addressed skills building components aimed at medication-related barriers to adherence (see Table 3), including pill taking strategies (n = 6), use of reminder devices (n = 6), scheduling doses (n = 8), and coping with side effects (n = 7). Seven of the interventions included an activity that allowed participants to conduct a functional analysis of previously missed doses. In this context, a functional analysis was defined as the identification of variables that influence the occurrence or maintenance of nonadherence among PLWH. A majority of the interventions (n = 6) included a module targeting strategies to improve communication between the participant and healthcare providers.

Skills building modules

Intervention content in relation to improving adherence self-efficacy varied considerably across studies. Most studies (n = 7) incorporated problem-solving skills throughout the intervention and also addressed strategies for building a strong social support network (n = 6). Eight studies utilized components that promote increased self-awareness towards adherence-related behaviors. Only four studies reported incorporating components that addressed HIV-related myths, primarily surrounding medications. Finally, seven of the studies included components of positive reinforcement for adherence-related behaviors. The provision of positive reinforcement varied in two forms: (1) participants were instructed on how to provide positive reinforcement for future adherence behavior and (2) participants were given positive reinforcement from the intervention narrator based on responses to assessment questions.

Other health-related barriers to adherence

The remaining intervention components fell within the broad category of other health-related behaviors. Several interventions (n = 6) addressed mental health barriers such as depression (n = 2), stress management (n = 4), and alcohol and drug use (n = 4). Several studies provided additional skills building components including cognitive restructuring of maladaptive thoughts (n = 2) and visual imagery exercises (n = 3). Five studies addressed structural barriers regarding adherence, which included transportation concerns and access to treatment and medications. Finally, five studies provided participants with information about additional resources in general for PLWH.

Intervention outcomes on adherence-related measures

Only eight [2733, 35] of the ten studies included in the review measured adherence. The method of measuring adherence varied across studies. A majority of the studies measured adherence via self-report, while only two studies [30, 33] utilized Medication Event Monitoring System (MEMS). MEMS devices are electronic pill bottle caps with a pressure-activated chip which records the time and date that the bottle is opened. This more objective measure of medication adherence is currently considered gold standard; however, it is often modified by self-reported adherence and, consequently, is a less objective measure of adherence [38] (ADD CITE). Table 5 displays study outcomes for adherence. As regards to self-reported adherence, only two studies [29, 30] found statistically significant improvement in adherence from baseline to follow-up; however, several additional studies [27, 32] “approached” significance. Of the two studies utilizing MEMS data, only one [30] demonstrated statistically significant improvements in adherence. Three studies [29, 30, 32] also collected viral load data. Of these three studies, only one [37] demonstrated a statistically significant change in viral load using a cutoff of >400; however, analysis at a more stringent cutoff (viral load >48), which is typically indicative of having an undetectable viral load, was nonsignificant. Taken together, these initial studies have demonstrated promise with regard to improving adherence by utilizing computer-based delivery methods. However, evidence of efficacy for these interventions is limited, and larger randomized controlled trials are needed to address the limitations noted.

Table 5.

Study outcomes on adherence measures

Article Intervention title Main outcome Measurement p value
Claborn et al. [27] eLifeSteps 1. Adherence 1. Self-report .056a
Fisher et al. [29] LifeWindows 1. Adherence
2.Viral load
1.Self-report
2.Chart review
.024b
NS
Hersch et al. [30] Life Steps for Managing Medications and Stress 1.Adherence
2.Viral load
1.MEMS
1.Self-report
2.Viral load
.042a
.07
.024c
Horvath et al. [31] Thrive with Me 1.Adherence Self-report .43a
Naar-King et al. [32] Motivational Enhancement System for Adherence 1.Adherence
2.Viral Load
1.Self-report
2.Viral load
<.05d
NS
Ownby et al. [33] Not available 1.Adherence MEMS .07e

ACTG Adherence Clinical Trials Group self-report questionnaire, MEMS Medication Event Monitoring System, NS not significant

aMeasured change in adherence between baseline and follow-up

bBased on a 70 % adherence cutoff. The p value was .05 for a 90 % adherence cutoff

cBased on viral load cutoff of >400. Analysis with viral load >48 was nonsignificant

dMeasured difference in 7-day adherence recall between groups at 6-month follow-up

eBased on 95 % adherence cutoff score. The p value was .04 for 85 % adherence cutoff

DISCUSSION AND CONCLUSIONS

Antiretroviral therapy (ARV) can be highly effective [8]. However, too often, inadequate adherence impedes successful viral suppression [3], increasing the risk of disease progression and HIV-related morbidity and mortality [46], increasing the risk of infecting others [8], and contributing to the development of treatment resistant strains of HIV [7]. While efficacious face-to-face interventions have been developed to improve ARV adherence, dissemination and implementation of these interventions into real-world clinical settings have been severely limited by the resources required to sustain them [10]. As a result, some researchers have turned to the development of computer-delivered adherence interventions [21]. This article reviewed published studies utilizing a computer-based delivery mechanism of HIV adherence interventions. To date, ten computer-based HIV adherence interventions have been developed and published in the literature. These interventions remain in the early stages of testing, with a majority of the published studies in the feasibility and pilot testing phase. Many of these studies are underpowered to reach statistically significant changes in adherence over time due to the study design and small sample size. As such, generating inferences about the efficacy or clinical impact (e.g., cost-effectiveness) of computer-based interventions based on these reports may be premature.

Components of existing computerized interventions

This review found significant variability in the structure and theoretical framework of computer-based adherence interventions. This finding may reflect the limited knowledge in understanding which treatment components significantly improve adherence-related behaviors and maintain these gains over time. In contrast to traditional BITs, the interventions (n = 7) reviewed in this study were primarily delivered in the clinic setting instead of offsite. Programs varied in intervention dose, ranging from a single session up to 18 sessions (see Table 2). Session length varied from 30 min to over 2 h. Although single-session approaches are more easily implemented, it appears that an increased dose may be needed to increase long-term effectiveness. All of the interventions reported in this review were developed using sound theoretical frameworks for behavior change including cognitive behavior theory, social cognitive theory, information-motivation-behavior model, social action theory, and motivational enhancement theory. Only two studies [27, 30] adapted an empirically supported face-to-face adherence intervention (Life Steps) [39] into a computer-delivered format. Most programs included an audio interviewer to guide the participant throughout the program. Audio narration is an important component of these programs considering the literacy concerns of the target population. Additionally, narration may increase program engagement. Half of the interventions incorporated a review of important concepts and knowledge check (e.g., quiz). One program used the computerized program as an adjunct to a counselor-facilitated face-to-face intervention [34]. Combined BITs with in-person sessions may facilitate implementation of these interventions within the clinic setting, while maintaining important characteristics of the therapeutic alliance that are associated with behavior change (e.g., rapport) and provide a stronger dose of the intervention over time.

Three themes emerged in regard to program content including adherence-related, skills building, and other health-related barriers to adherence modules (e.g., stress management, negative affect, and substance misuse). Specific adherence skills building components addressed medication-taking strategies, analyzing patterns associated with missed doses, problem solving skills, building social support, and improving communication with providers. All of the themes represent individual-level characteristics aimed at improving adherence. This may be a limitation of existing adherence interventions considering that maintaining optimal levels of ARV adherence is a lifelong process that requires lifestyle changes for the individual. Interventions that expand upon the ecological framework and address important interpersonal (e.g., incorporating primary partners) and community factors (e.g., access to healthcare, stigma, poverty) may facilitate the maintenance of gains in adherence and other healthy behaviors over time.

Limitations of computer-delivered interventions for adherence

This work is still in its infancy, with the vast majority of extant literature aimed at demonstrating feasibility and piloting of the intervention. Our review of the literature yielded six studies of computerized interventions to improve adherence that included some measure of ARV adherence outcome [27, 2933]. Taken together, these studies demonstrate that computerized adherence interventions show promise. However, consistent with the very early nature of the work in this area, studies have methodological limitations including small sample sizes, self-report as the sole measure of adherence, a short follow-up window, and lack of a control condition that equated for computer interaction time. These results warrant larger randomized controlled trials addressing the current limitations of the literature.

This review provides insight into content and intervention design considerations for computerized interventions. First, the content related to adherence skills building appears to be primarily based upon the IMB model incorporating educational components, motivational enhancement methods, and developing behavioral skills such as incorporating the regimen into daily routine, pill-taking and reminder strategies, and improving communication skills with healthcare providers. As regards to promoting adherence self-efficacy, most interventions teach participants self-reinforcement behaviors for achieving adherence-related goals and develop problem-solving skills. Although a majority of interventions include a social skills component, only two emphasized enlisting a partner to assist with medication taking. Second, it appears that providing targeted interventions may be a more efficient and effective method of promoting medication adherence when compared to less directed, social networking intervention methods. Finally, attrition rates in the studies reviewed were much lower than in face-to-face intervention studies. However, intervention retention was problematic in the more intensive, multi-session computerized interventions [29, 30].

A number of important limitations to the current review should be acknowledged. First, this review focused exclusively on interventions that were delivered and received on a personal (desktop) computer. A number of other adherence-related programs are in various stages of development that utilize other available technologies, including paging systems, SMS messaging, and mobile applications [38, 40, 41]. These interventions differ from those reviewed in that they are intended to deliver treatment components both within the clinic setting and beyond, often with the goal of being incorporated into the daily lives of patients. As such, this review focused primarily on clinic-initiated/facilitated, desktop computer-delivered interventions in order to explore tools that are more readily available to health-care providers and could be integrated readily into current clinic practices. Second, the small pool of systematic reports that is available on computer-delivered interventions for HIV-related medication adherence limited conclusions that could be drawn and, as such, narrowed the scope of this review. Although this limitation is likely due in large part to the novelty of both adherence interventions and technology-based dissemination, well-designed, rigorous tests of intervention efficacy are urgently needed. Given the lag between data collection and publication, additional reports of intervention efficacy are likely to become available in the coming years. Another limitation is that the “gray literature” (e.g., conference abstracts and proceedings, these/dissertations, and registered trials) in the databases was not searched, which may have limited the yield of relevant findings. Nevertheless, this review highlights the current state of the published research literature and emphasizes the need for publication and dissemination of these tools to the public.

Future directions

Future studies should examine dose response to these interventions via longitudinal follow-up. Further, these interventions may be aided with brief, supportive follow-up interventions utilizing text messaging, phone calls, or other technology-based mechanisms in an effort to improve outcomes and maintain behavior changes over a longer timeframe. The addition of these components may encourage generalization of skills through clinical contacts occurring in real-world situations outside of the clinical setting. People living with HIV represent a diverse group of people; therefore, interventions tailored to specific populations (e.g., MSM, substance users, and women) may address barriers and adherence concerns specific to the unique needs of these individuals. Additionally, no gold standard protocol exists for interventions targeting adherence. Dismantling studies may provide further insight into the components that are most helpful at improving adherence among PLWH. Cost-effectiveness studies are needed as well in order to identify the extent to which implementation of these interventions are preferable in terms of economic costs, effects, and utility when compared to usual care. Future studies should be designed with regard to evaluation of implementation and sustainability of the intervention within the clinic setting. It is important to note that computerized interventions are not subject to fidelity checks; consequently, without examining the actual electronic content of each intervention, it is not possible to know if the reported components of these programs adequately represent what they were intended to represent. Finally, only a few studies reported effect sizes for treatment outcomes. It is strongly recommended that future studies include effect sizes in reporting of results [42].

Acknowledgments

This was an investigator-initiated study supported in part by grant number T32 AA007459 from the National Institute on Alcohol Abuse and Alcoholism at the National Institutes of Health. The funders played no role in the design, conduct or analysis of the study, nor in the interpretation and reporting of the study findings. All authors had full access to all of the data in the study (including statistical reports and tables) and can take responsibility for the integrity of the data and the accuracy of the data analysis.

Conflict of Interest

Kasey Claborn, Anne Fernandez, Tyler Wray, and Susan Ramsey declare that they have no conflict of interest.

Appendix

PRISMA CHECKLIST Section/topic # Checklist item Reported on page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both. 1
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. 2
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known. 2-4
Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). 4
METHODS
Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. N/A
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. 5
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. 5
Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. 5
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). 5
Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. 5
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. 5-6
Risk of bias in individual studies 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. 6
Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). 5-6
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each meta-analysis. N/A
Section/topic # Checklist item Reported on page #
Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). 6
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. N/A
RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. 6-7
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Table 1
Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12). 7, Figure 2
Results of individual studies 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot. Table 5
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. N/A
Risk of bias across studies 22 Present results of any assessment of risk of bias across studies (see Item 15). Figure 2
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). N/A
DISCUSSION
Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers). 11-13
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). 14
Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. 14-15
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. 16

Footnotes

Implications

Practice: Computer-delivered interventions can be an efficient and cost-effective method of improving patient treatment knowledge and self-efficacy.

Policy: Resources are needed to increase research on development and testing of eHealth interventions and examination of implementation strategies and sustainability potential.

Research: eHealth interventions need to be rigorously evaluated and include examination of dose–response, cost-effectiveness, and fidelity to treatment protocols to promote future sustainable practice within the clinic setting.

References

  • 1.CDC. Monitoring selected national HIV prevention and care objectives by using HIV surveillance data-United States and 6 U.S. dependent areas-2010. HIV Surveillance Supplemental Report 2012.
  • 2.Hall H, Frazier E, Skarbinski J, et al. Differences in human immunodeficiency virus care and treatment among subpopulations in the United States. JAMA Internal Medicine. 2013;173(14):1337–1344. doi: 10.1001/jamainternmed.2013.6841. [DOI] [PubMed] [Google Scholar]
  • 3.Ortego C, Huedo-Medina T, Vejo J, et al. Adherence to highly active antiretroviral therapy (HAART): a meta-analysis. AIDS and Behavior. 2011;15(7):1381–1396. doi: 10.1007/s10461-011-9942-x. [DOI] [PubMed] [Google Scholar]
  • 4.Paterson D, Swindells S, Singh N, et al. Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Annals of Internal Medicine. 2000;133(1):21–30. doi: 10.7326/0003-4819-133-1-200007040-00004. [DOI] [PubMed] [Google Scholar]
  • 5.Bangsberg D, Perry S, Moss A, et al. Non-adherence to highly active antiretroviral therapy predicts progression to AIDS. AIDS (London, England) 2001;15(9):1181–1183. doi: 10.1097/00002030-200106150-00015. [DOI] [PubMed] [Google Scholar]
  • 6.Lima V, Harrigan R, Montaner J, et al. The combined effect of modern highly active antiretroviral therapy regimens and adherence on mortality over time. Journal of Acquired Immune Deficiency Syndromes. 2009;50(5):529–536. doi: 10.1097/QAI.0b013e31819675e9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Sethi A, Celentano D, Gange S, Moore R, Gallant J. Association between adherence to antiretroviral therapy and human immunodeficiency virus drug resistance. Clinical Infectious Diseases: An Official Publication of the Infectious Diseases Society of America. 2003;37(8):1112–1118. doi: 10.1086/378301. [DOI] [PubMed] [Google Scholar]
  • 8.Cohen M, Chen Y, Fleming T, et al. Prevention of HIV-1 infection with early antiretroviral therapy. The New England Journal of Medicine. 2011;365(6):493–505. doi: 10.1056/NEJMoa1105243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Simoni J, Pearson C, Pantalone D, Marks G, Crepaz N. Efficacy of interventions in improving highly active antiretroviral therapy adherence and HIV-1 RNA viral load: a meta-analytic review of randomized controlled trials. Journal of Acquired Immune Deficiency Syndromes. 2006;43(Suppl1):S23–S35. doi: 10.1097/01.qai.0000248342.05438.52. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Simoni J, Amico K, Smith L, Nelson K. Antiretroviral adherence interventions: translating research findings to the real world clinic. Current HIV/AIDS Reports. 2010;7(1):44–51. doi: 10.1007/s11904-009-0037-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Chen H, Grimley D, Waithaka Y, Aban I, Hu J, Bachmann L. A process evaluation of the implementation of a computer-based, health provider-delivered HIV-prevention intervention for HIV-positive men who have sex with men in the primary care setting. AIDS Care. 2008;20(1):51–60. doi: 10.1080/09540120701449104. [DOI] [PubMed] [Google Scholar]
  • 12.Goldstein M, Guise B, Ruggiero L, Raciti M, Abrams D. Clinical psychiatry for medical students [e-book] Philadelphia: J B Lippincott Company; 1990. Behavioral medicine strategies for medical patients; pp. 609–629. [Google Scholar]
  • 13.Whitlock E, Orleans C, Pender N, Allan J. Evaluating primary care behavioral counseling interventions: an evidence-based approach. American Journal of Preventive Medicine. 2002;22(4):267–284. doi: 10.1016/S0749-3797(02)00415-4. [DOI] [PubMed] [Google Scholar]
  • 14.Dewing S, Mathews C, Schaay N, Cloete A, Louw J, Simbayi L. Behaviour change counselling for ARV adherence support within primary health care facilities in the Western Cape, South Africa. AIDS and Behavior. 2012;16(5):1286–1294. doi: 10.1007/s10461-011-0059-z. [DOI] [PubMed] [Google Scholar]
  • 15.Mepham S, Zondi Z, Mbuyazi A, Mkhwanazi N, Newell M. Challenges in PMTCT antiretroviral adherence in northern KwaZulu-Natal, South Africa. AIDS Care. 2011;23(6):741–747. doi: 10.1080/09540121.2010.516341. [DOI] [PubMed] [Google Scholar]
  • 16.Grant E, Logie D, Masura M, Gorman D, Murray S. Factors facilitating and challenging access and adherence to antiretroviral therapy in a township in the Zambian Copperbelt: a qualitative study. AIDS Care. 2008;20(10):1155–1160. doi: 10.1080/09540120701854634. [DOI] [PubMed] [Google Scholar]
  • 17.Murray L, Semrau K, Bolton P, et al. Barriers to acceptance and adherence of antiretroviral therapy in urban Zambian women: a qualitative study. AIDS Care. 2009;21(1):78–86. doi: 10.1080/09540120802032643. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Rotheram-Borus M, Swendeman D, Chovnick G. The past, present, and future of HIV prevention: integrating behavioral, biomedical, and structural intervention strategies for the next generation of HIV prevention. Annual Review of Clinical Psychology [serial online] 2009;5:143–167. doi: 10.1146/annurev.clinpsy.032408.153530. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Swendeman D, Rotheram-Borus M. Innovation in sexually transmitted disease and HIV prevention: internet and mobile phone delivery vehicles for global diffusion. Current Opinion in Psychiatry. 2010;23(2):139–144. doi: 10.1097/YCO.0b013e328336656a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Pellowski J, Kalichman S. Recent advances (2011–2012) in technology-delivered interventions for people living with HIV. Current HIV/AIDS Reports. 2012;9(4):326–334. doi: 10.1007/s11904-012-0133-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Mohr DC, Burns M, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. General Hospital Psychiatry. 2013;35(4):332–338. doi: 10.1016/j.genhosppsych.2013.03.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.The Pew Charitable Trust. Mobile Technology Fact Sheet. 2014. Available from: http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/
  • 23.Chiasson M, Hirshfield S, Rietmeijer C. HIV prevention and care in the digital age. Journal of Acquired Immune Deficiency Syndromes. 2010;55(Suppl 2):S94–S97. doi: 10.1097/QAI.0b013e3181fcb878. [DOI] [PubMed] [Google Scholar]
  • 24.Catalani C, Philbrick W, Fraser H, Mechael P, Israelski D. mHealth for HIV treatment & prevention: a systematic review of the literature. The Open AIDS Journal. 2013;7:17–41. doi: 10.2174/1874613620130812003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Van Volkom M, Stapley J, Amaturo V. Revisiting the digital divide: generational differences in technology use in everyday life. North American Journal of Psychology. 2014;16(3):557–574. [Google Scholar]
  • 26.Moher D, Liberati A, Tetzlaff J, Altman D. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. International Journal of Surgery (London, England) 2010;8(5):336–341. doi: 10.1016/j.ijsu.2010.02.007. [DOI] [PubMed] [Google Scholar]
  • 27.Claborn K, Leffingwell T, Miller M, Meier E, Stephens J. Pilot study examining the efficacy of an electronic intervention to promote HIV medication adherence. AIDS Care. 2014;26(3):404–409. doi: 10.1080/09540121.2013.824534. [DOI] [PubMed] [Google Scholar]
  • 28.Côté J, Godin G, Fadel G, et al. Evaluation of a real-time virtual intervention to empower persons living with HIV to use therapy self-management: study protocol for an online randomized controlled trial. Trials. 2012;13:187. doi: 10.1186/1745-6215-13-187. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Fisher J, Amico K, Friedland G, et al. Computer-based intervention in HIV clinical care setting improves antiretroviral adherence: the LifeWindows Project. AIDS and Behavior. 2011;15(8):1635–1646. doi: 10.1007/s10461-011-9926-x. [DOI] [PubMed] [Google Scholar]
  • 30.Hersch R, Cook R, Spencer J, et al. Test of a web-based program to improve adherence to HIV medications. AIDS and Behavior. 2013;17(9):2963–2976. doi: 10.1007/s10461-013-0535-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Horvath K, Oakes J, Simoni J, et al. Feasibility, acceptability and preliminary efficacy of an online peer-to-peer social support ART adherence intervention. AIDS and Behavior. 2013;17(6):2031–2044. doi: 10.1007/s10461-013-0469-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Naar-King S, Outlaw A, Ondersma S, et al. Motivational Enhancement System for Adherence (MESA): pilot randomized trial of a brief computer-delivered prevention intervention for youth initiating antiretroviral treatment. Journal of Pediatric Psychology. 2013;38(6):638–648. doi: 10.1093/jpepsy/jss132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Ownby R, Waldrop-Valverde D, Caballero J, Jacobs R. Baseline medication adherence and response to an electronically delivered health literacy intervention targeting adherence. Neurobehavioral HIV Medicine. 2012;4:113–121. doi: 10.2147/NBHIV.S36549. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Remien R, Mellins C, Stein D, et al. Masivukeni: development of a multimedia based antiretroviral therapy adherence intervention for counselors and patients in South Africa. AIDS and Behavior. 2013;17(6):1979–1991. doi: 10.1007/s10461-013-0438-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Shegog R, Markham C, Leonard A, Bui T, Paul M. ' + CLICK': pilot of a web-based training program to enhance ART adherence among HIV-positive youth. AIDS Care. 2012;24(3):310–318. doi: 10.1080/09540121.2011.608788. [DOI] [PubMed] [Google Scholar]
  • 36.Higgins J, Altman D, Sterne J, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ (Clinical Research Ed.) 2011;343:d5928. doi: 10.1136/bmj.d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Outlaw A, Naar-King S, Merlo L, et al. The initial feasibility of a computer-based motivational intervention for adherence for youth newly recommended to start antiretroviral treatment. AIDS Care. 2014;26(1):130–135. doi: 10.1080/09540121.2013.813624. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Pop-Eleches C, Thirumurthy H, Bangsberg D, et al. Mobile phone technologies improve adherence to antiretroviral treatment in a resource-limited setting: a randomized controlled trial of text message reminders. Aids. 2011;25(6):825–834. doi: 10.1097/QAD.0b013e32834380c1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Safren S, Otto M, Worth J. Life-steps: applying cognitive behavioral therapy to HIV medication adherence. Cognitive and Behavioral Practice [serial online]. Fal 1999 1999;6(4):332–341.
  • 40.Cole-Lewis H, Kershaw T. Text messaging as a tool for behavior change in disease prevention and management. Epidemiologic Reviews. 2010;32(1):56–69. doi: 10.1093/epirev/mxq004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Thirumurthy H, Lester RT. M-health for health behaviour change in resource-limited settings: applications to HIV care and beyond. Bulletin of the World Health Organization. 2012;90(5):390–392. doi: 10.2471/BLT.11.099317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Faulkner C, Fidler F, Cumming G. The value of RCT evidence depends on the quality of statistical analysis. Behaviour Research and Therapy. 2008;46(2):270–281. doi: 10.1016/j.brat.2007.12.001. [DOI] [PubMed] [Google Scholar]

Articles from Translational Behavioral Medicine are provided here courtesy of Oxford University Press

RESOURCES