Implementation of eHealth interventions across the HIV care cascade: A review of recent research

Abstract

Purpose of Review:

We review recent implementation science focusing on eHealth interventions to improve outcomes along the HIV care cascade. We highlight several gaps in the eHealth implementation literature and propose areas for future study.

Recent Findings:

We identified 17 studies conducted in North America, Europe, and Sub-Saharan Africa assessing the acceptability, appropriateness, adoption, cost, feasibility, fidelity, penetration, or sustainability of eHealth interventions targeting the HIV care cascade. Most interventions used SMS messages to improve cascade outcomes. Feasibility research has demonstrated the importance of adaptability for intervention scale-up and delivery. Key gaps in the literature remain related to predictors of the adoption of eHealth interventions by health facilities and staff. In addition, no studies explored sustainability and few used theoretical frameworks for implementation research or validated measures of implementation outcomes. We propose next steps for the future of eHealth implementation research to inform the delivery, scale-up, and maintenance of eHealth interventions in the real world.

Introduction

Electronic health (eHealth) interventions, including desktop computer-, tablet-, and mobile phone-based (mHealth) programs, have enormous potential to support health systems in achieving the UNAIDS 90-90-90 HIV treatment targets, especially in low- and middle-income countries (LMICs). An estimated 96% of the world’s population has access to a mobile phone, including 70% of individuals in the lowest income group, and at least half have access to the internet through their phones [1, 2]. Two billion people possess app-enabled smart phones [3]. Functioning desktop computers, tablets, and other technologies are increasingly available at health facilities, even in low-resource settings [4].

eHealth interventions leveraging these technology platforms can reach patients across the HIV care cascade, offering novel, affordable approaches to increase rates of HIV testing, diagnosis, linkage to care, treatment retention, treatment adherence, and viral suppression [5]. For example, SMS interventions are highly effective at improving antiretroviral therapy (ART) adherence, and recent studies have demonstrated that weekly reminder texts and interactive patient interfaces improve patient adherence by 28% when compared with the standard of care [6]. As a result of these findings, the WHO now recommends SMS adherence promotion interventions in their ART therapy guidelines [7]. Although most HIV-related eHealth interventions target patient behaviors, a growing literature suggests that the same technologies may also offer substantial benefit to HIV care providers and health facilities, especially in low-resource settings, by improving provider-patient and provider-provider communication, increasing rates of service delivery, and ensuring fidelity to treatment protocols [8].

While eHealth interventions have shown promise in improving HIV care and treatment outcomes, the population-level effects of these approaches will be limited until they are delivered at scale, sustained over time, and available to those who need them most [9]. Unfortunately, few eHealth interventions move beyond the pilot study phases [10]. Pilot, efficacy, and effectiveness studies are critical steps towards successful implementation, but even the most efficacious interventions will have little impact if there has been inadequate consideration of how they will be implemented or insufficient evidence to convince key policymakers and stakeholders of their affordability and scalability [11, 12].

Implementation science (IS) is an emerging research paradigm that uses a range of rigorous, interdisciplinary methods to inform delivery of efficacious interventions in real-world settings. It is distinguished from other fields by its focus on deliberate actions to introduce or change the delivery of new technologies, practices, and services [13, 14]. These are known as implementation strategies: intentional actions or approaches available to an implementer, like identifying a local champion or providing financial incentives to providers [15, 16]. IS examines the proximal implementation outcomes of such actions - including acceptability, appropriateness, adoption, costs, feasibility, fidelity, penetration, and sustainability [17] - alongside intermediate service delivery and more distal health outcomes. Implementation researchers use unique theoretical frameworks to guide the translation of research into practice (e.g., the Canadian Institutes of Health Research Model of Knowledge Translation [18]); study the determinants of implementation success (e.g., the Consolidated Framework for Implementation Research [19]); and evaluate the impact of implementation (e.g., the RE-AIM framework [20]) [21]. They also experiment with intervention characteristics and implementation strategies, adopting so-called effectiveness-implementation hybrid trial designs that either test intervention effects on service delivery or health outcomes while observing implementation outcomes (Type 1), test both intervention and implementation strategy (Type 2), or test implementation strategy effects on implementation outcomes while observing service delivery or health outcomes (Type 3) [22].

IS methods can maximize the public health impact of HIV-related eHealth interventions by helping researchers identify and test implementation strategies, develop new approaches to improve their interventions in different settings, and explore the causal mechanisms between intervention and impact. Our objectives are to review recent implementation research on eHealth interventions targeting the HIV care cascade and suggest pathways for future research.

Methods

We searched PubMed from 1 January 2015 to 28 February 2018 for recent English language peer-reviewed publications, using combinations of keywords: (mHealth or eHealth) and HIV and (“implementation science” or “implementation outcome” or “d&I” or “dissemination and implementation” or acceptability or adoption or appropriateness or feasibility or fidelity or cost or penetration or sustainability). We identified and screened 93 results. We excluded studies that did not: 1) evaluate the implementation of an eHealth or mHealth interventions targeting the HIV care cascade and 2) assess at least one implementation outcome as specified by Proctor et al. (2011) [17]. We also reviewed the reference lists of included articles for other studies meeting our inclusion criteria.

Narrative Review

We include 17 studies in our review (Table 1) [23–39]. These evaluated the implementation of 15 different interventions, as two studies evaluated the Get Connected! intervention [23, 34], and two evaluated the WelTel intervention [26, 30]. Four interventions primarily targeted testing and diagnosis [23, 25, 34, 38, 37], 4 targeted linkage to and retention in care [24, 28, 31, 36], and 7 targeted treatment adherence and viral suppression [26, 27, 29, 30, 32, 33, 35, 39]. Eight interventions used SMS technology [24–27, 30, 31, 33, 35, 39], 4 used smartphone- or tablet-based applications [29, 36–38], 3 used desktop computers [23, 25, 34, 38], 1 used phone-based chat and instant messaging [28], and 1 used Medication Event Monitoring System (MEMS) caps [32]. The majority of studies (11, 64.7%) evaluated implementation in LMICs [24, 25, 27, 28, 30, 31, 33, 35, 37–39]. We organize our summary of recent HIV-related eHealth implementation research by the first level of the HIV care cascade targeted by each intervention.

Table 1:

Included studies, by year of publication and author (n=17)

First Author Citation	Year	Country	Intervention	Technology	Level	Cascade Step	Study Design	Implementation Outcomes	Key Findings
Bauermeister [23]	2015	United States	Get Connected!	Desktop	Pa	1	RCT	Ac	High patient acceptability; tailored intervention may be more acceptable than locator-only intervention.
Dryden- Peterson [24]	2015	Botswana	Tokafatso	SMS	Pr	2	RCT; stepped wedge	Ac, C	SMS platform and provision of airtime appreciated by providers. Cost of $ 1.98 per CD4 result delivered vs. $2.73 with standard delivery methods.
Finocchario- Kessler [25]	2015	Kenya	HIT System	Deskstop, SMS	Pr, Pa	1, 2	Cohort; prospective	Ad, c, Fe, P	High reach, good effectiveness, adopted by health facilities at different levels, implementation varied depending on health facility, can be maintained by integrating into national EMR system
Murray [26]	2015	Canada	WelTel	SMS	Pa	3	Qualitative; FGDs, interviews	Ac, Ap, Fe	Providers felt intervention built relationships with patients, streamlined existing mHealth interventions, and dealt with privacy issues. Increase in workload offset by benefits.
Georgette [27]	2016	South Africa	Not named	SMS	Pa	3	Qualitative; interviews	Ac	Patients felt intervention was acceptable and helpful, most would recommend to friends. Two-way communication desired by most.
Henwood [28]	2016	South Africa	Khaya HIV Positive	Instant message	Pa	2, 3	Qualitative; surveys, FGDs	Ac	Moderate patient acceptability. Cost and need for anonymity were problems. Usage limited by preference for other platforms, logistical obstacles, and loss of interest.
Kurth [29]	2016	United States	CARE+ Spanish	Tablet	Pa	3	RCT; qualitative; interviews; FGDs	Ac, Ap	High patient acceptability; easy to use, ensured privacy and confidentiality. Providers felt intervention was useful, but could not replace provider-patient relationship.
Bardosh [30]	2017	Kenya, Canada	WelTel	SMS	Pa	3	Qualitative; case study	Ac, Ad, Ap, Fe	Positive influence on “culture of care” at health facilities. Scale-up viewed as precarious and uncertain process. Challenges: juggling different interests, determining appropriate financing pathways, maintaining network growth, packaging intervention for impact and relevance.
Campbell [31]	2017	Uganda	Not named	SMS	Pa	2	Qualitative; interviews	Ac	SMS language, phone characteristics, and experience with similar technologies contributed to ease of use. Perceived usefulness because system augmented HIV care services and improved access to social support. Importance of confidentiality, disclosure, and stigma. Barriers and facilitators downstream from the intervention impacted target outcome.
de Bruin [32]	2017	Nether lan ds	AIMS	MEMS caps	Pr, Pa	3	RCT; mathematical model	C	Using MEMS caps data to create patient adherence reports was cost-effective. This finding was robust in sensitivity analyses.
Hirsch- Moverman [33]	2017	Lesotho	START	SMS	Pa	3	Qualitative; interviews	Ac	High patient acceptability of intervention and provision of airtime. Some did not perceive SMS usefulness. Providers used phones for text and phone calls. Electricity and technical difficulties were barriers for some.
Horvath [34]	2017	United States	Get Connected!	Desktop	Pa	1	Qualitative; surveys	Ac	Few differences in patient acceptability by intervention type or eHealth literacy, related to information and system quality.
Patel [35]	2017	Kenya	N/A	SMS	Pa	3	Mathematical model	C	SMS adherence interventions cost-effective, especially with retention improvements. ICER sensitive to intervention costs, effectiveness, and cohort CD4 count.
Wray [36]	2017	United States	eTEST	Smartphone, Bluetooth	Pa	2	Qualitative; interviews	Ac, Fe	eTEST system detected HBST use in nearly all cases. Timing, method, and duration of follow-up were appropriate and helpful.
Bull [37]	2018	T anzania	T-HIT	Tablet	Pr	1, 2	RCT	Ac, Ad, Fe	High acceptability feasibility. Perceived as useful. More antenatal visits during the pilot intervention compared with paper logs.
Gbadamosi [38]	2018	Nigeria	Vitira Health	Desktop, Smartcard, Smartphone	Pr, Pa	1	Qualitative; observation	Fe	Could be used to store encrypted HIV results and display prenatal test results to skilled birth attendants without internet connectivity.
Ronen [39]	2018	Kenya	Mobile WAChX	SMS	Pa	3	Qualitative; FGDs	Ac	Half of participants supported overtly HIV- related terms, wanting detailed educational messages about ART and PMTCT. Concerns about confidentiality. Acceptability of HIV-related content depends on disclosure status and others’ access to phone.

Abbreviations: SMS = Short message service; Pr = Provider; Pa = Patient; 1 = HIV testing and diagnosis; 2 = Linkage to and retention in care; 3 = Adherence to antiretroviral medication and viral suppression; FGDs = Focus group discussions; RCT = Randomized controlled trial; Ac = Acceptability, Ap = Appropriateness, Ad = Adoption, C = Costs, Fe = Feasibility, Fi = Fidelity, P = Penetration, S = Sustainability

HIV Testing and Diagnosis

Two recent studies have evaluated the implementation of Get Connected!, which is a web-based eHealth intervention designed to promote and reinforce HIV testing among high-risk young men who have sex with men in Michigan, United States. Get Connected! is a website that can tailor the display of information and imagery to the specific characteristics (e.g., age, ethnicity, sexual identity, relationship status, HIV testing history) of individual patients. In the first study, Bauermeister et al. (2015) used a randomized controlled trial to pilot the intervention, assessing post-intervention patient-level acceptability [23]. They determined that the intervention was highly acceptable, and more participants in the intervention group sought HIV testing after reviewing the website compared with participants in the control condition who received access to an online directory of healthcare providers. In the second study, Horvath et al. (2017) reanalyzed the data from the trial to determine whether patient-level acceptability was associated with baseline eHealth literacy [34]. They hypothesized that participants with low competency with computers and the internet may have difficulty navigating the intervention interface, and would be less likely to internalize the information presented. However, they identified few differences in patient acceptability by eHealth literacy.

Several recent studies have also examined the implementation of facility-level eHealth interventions aiming to increase rates of infant HIV testing and diagnosis. In Kenya, researchers designed a web-based intervention (the HITSystem) that automatically alerts providers and lab technicians when infants are due for early HIV diagnosis, and sends SMS messages to mothers’ mobile phones when results are ready or they are scheduled to come in for follow-up. Following a pilot efficacy study, Finocchario-Kessler et al. (2015) used the RE-AIM framework to evaluate the impact of the real-world implementation of the HITSystem across ten health facilities in urban and peri-urban areas of central and western Kenya [25]. The investigators collected data at the patient, provider, and facility levels to track intervention reach among mothers, adoption of the intervention by health facilities and staff, fidelity of the recommendations and message delivery, and cost over time. They found that once the intervention was implemented, its uptake was nearly universal at health facilities, and the program was able to reach the majority of women with high literacy levels [25]. Adaptability was critical to the adoption of the HITSystem: by responding to and integrating user feedback from facility staff, implementers were able to increase feelings of ownership of the intervention at the facility level and adjust to changes in national care guidelines. However, one of the challenges to maintenance of the HITSystem was staff turnover at the different health facilities, which created disruption in service delivery [25].

While interventions like Get Connected! and the HITSystem can improve rates of HIV testing, others are needed to increase the availability of test results at point-of-care, especially in low-resource settings. In Nigeria, Gbadamosi et al. (2018) developed the Vitira Health platform to ensure skilled birth attendants have access to antenatal HIV test results [38]. Vitira Health combines a web-based patient information database, patient-held smartcards storing encrypted health data, and a mobile app allowing health care workers to scan and view patient data stored on the smartcards. Gbadamosi et al. convened an interdisciplinary group of experts to identify and develop the core aspects of the platform, with the goal of maximizing affordability, ease of use, security, and scalability, and then deployed the platform at four facilities to assess feasibility via observation and informal participant feedback. Preliminary results from 19 skilled birth attendants showed that the health workers were able to be trained in scanning the smartcard and viewing the participants’ data, suggesting that the program is feasible for broader implementation. However, it will be important for future studies to measure patient and provider acceptability, cost, and sustainability [38].

Robust data systems for patient monitoring are also critical to promote early engagement in testing, track patients through the care cascade, and prevent loss to follow-up. In Tanzania, Bull et al. (2018) developed a tablet-based data collection system (the Tanzania Health Information Technology [T-HIT] system) to capture and manage HIV testing and other pregnancy-related health data within a reporting dashboard. This dashboard includes fields for HIV testing history, blood type, tuberculosis diagnosis, and medications, and is integrated with alerts and reminders to improve continuity and quality of service delivery [37]. The investigators conducted a rigorous cluster randomized controlled trial to assess the preliminary efficacy and feasibility of the T-HIT system and found that providers in intervention facilities readily adopted and used the T-HIT system to track HIV testing data. They also observed an increase in reporting of antenatal visits in the T-HIT system compared to the traditional paper-based system (1594 antenatal visits recorded in T-HIT vs. 879 recorded on paper over the same period at the same facilities), highlighting T-HIT’s potential to improve antenatal care for women at risk of HIV [37]. These results indicate the intervention is feasible, though further study will be necessary to assess provider acceptability.

Linkage to and Retention in Care

Several eHealth interventions have been developed and tested to improve linkage to HIV care and retention in care, including “smart” home-based testing and text notification services. Home-based self-testing can be more convenient and confidential than facility-based HIV testing, and it has demonstrated high uptake in low-resource settings and among men who have sex with men in the United States [40, 41]. However, a key limitation of home-based testing is the lack of post-test follow-up and referral [42]. Wray et al. (2017) conducted interviews with high-risk participants to inform the development of a smartphone app and Bluetooth low energy beacon system (the eTEST system) to monitor self-testing kit use and trigger follow-up counselling and referrals by phone [36]. Following development of the system, the research team conducted a usability study to assess intervention feasibility and participant acceptability through follow-up interviews. In qualitative interviews with 10 participants, most felt the intervention was appropriate and helpful, and the vast majority reported that the follow-up phone calls were appropriate in length (they lasted an average of 10 minutes) and timing (within 24 hours of HIV testing). However, a few participants reported that they would have liked to receive a phone call even sooner after receiving their test results (e.g., within one hour of testing), and future research with the eTEST system will need to explore phone call timing particularly for those who have never tested before or who have tested positive [36].

SMS text message delivery of test results from laboratories to the point-of-care also may reduce time to ART initiation, especially in regions with limited options for transport, though provider acceptability will ultimately determine whether such interventions are scalable in these settings [43]. In Botswana, Dryden-Petersen et al. (2015) used a stepped wedge cluster randomized controlled trial to demonstrate the effectiveness of an SMS intervention to deliver CD4 test results from the central laboratory to peripheral clinics, with the goal of improving rates of treatment initiation [24]. They also tracked expenditure to estimate the cost of implementation and interviewed providers to assess acceptability. They found that the SMS intervention cost less per test result delivered than standard delivery methods ($1.98 vs. $2.73 per result), and that providers appreciated the SMS platform and the provision of airtime to contact patients. In Uganda, Campbell et al. (2017) used qualitative interviews to explore the technology-related attitudes and behaviors of participants of a similar study of an SMS intervention to improve patient linkage to care. The intervention and their semi-structured interview guides were informed by an established theoretical model for technology acceptance (the Technology Acceptance Model, TAM) [31]. Participants perceived the intervention to be useful and easy to use, believing that it improved their experience of HIV-related care, and emphasized the importance of SMS message confidentiality to avoid accidental HIV status disclosure. The authors used their findings to propose a revised theoretical framework for examining factors related to technology acceptance in resource-limited settings; this model adds several constructs to the TAM, including “technology literacy,” “transportation costs,” “clinic capacity,” and “program funding,” which are important for the scale-up of eHealth interventions in low literacy populations [31].

Achieving ART Adherence and Viral Suppression

Several recent studies have evaluated the implementation of SMS adherence interventions to assess their patient-level acceptability. In South Africa, Georgette et al. (2017) implemented a weekly SMS adherence support programs for 88 adult patients on ART [27]. Most participants (77%) reported that the program helped them remember their clinic appointments, and this response was particularly true for men and individuals who had already disclosed their HIV status outside of their homes. While 3 (3.4%) individuals reported that the SMS program led to unintentional HIV status disclosure, 81 (92%) of participants said that they would recommend the program to a friend (including all 3 individuals with unintentional disclosure). Future iterations of this intervention could also consider incorporating two-way SMS and features to customize the timing of SMS delivery. In Lesotho, an SMS adherence support and clinic appointment reminder program (START) used code words in their medication reminder messages to protect patient confidentiality [33]. According to qualitative in-depth interviews, participants generally preferred to receive messages daily (rather than weekly), and patients and providers almost universally felt supported by the intervention.

In addition, two studies have evaluated the implementation of the WelTel SMS adherence intervention, which uses simple text messages of “How are you?” to connect with patients and promote treatment adherence. The first study, by Murray et al. (2015), used focus group discussions and interviews to assess provider perceptions of intervention acceptability, appropriateness, and feasibility in British Columbia, Canada [26]. Providers felt that the intervention built relationships with patients, streamlined existing outreach efforts, and reduced privacy issues related to using personal phones to contact patients. While their initial workload increased as a result of intervention delivery, the benefits to both patients and providers helped to offset this, and providers felt that “if this is really successful [the workload] would even out over time [26].” Building off of this work, Bardosh et al. (2017) adopted a comparative qualitative case study design to explore multi-level contextual factors (e.g., organizational culture, resource constraints, policy environment) affecting the implementation of the WelTel intervention across several projects in Kenya and Canada [30]. They used key informant interviews with stakeholders to assess patient and provider acceptability, intervention appropriateness, relative adoption across different clinic types and settings, and the feasibility of scale-up to new contexts. They identified organizational- and intervention-level characteristics that facilitate or impede successful implementation, including the capacity of local management and the adaptability of the technology itself. Importantly, they also noted specific implementation strategies that appear to speed and strengthen the adoption of WelTel.

As these SMS ART adherence interventions are being tested and implemented, one important area of debate is the content and phrasing of the messages themselves; there is currently no consensus on how to structure these messages, and there is significant variation in messages across interventions and contexts. Ronen et al. (2017) nested a formative study into an ongoing randomized trial of an SMS-based prevention of mother-to-child transmission (PMTCT) adherence intervention in Kenya, conducting focus groups with HIV-infected pregnant and post-partum women with the goal of assessing preferred HIV-related message content [39]. Some participants were comfortable with texts overtly related to HIV, hoping for detailed educational messages, though many were concerned about confidentiality and the risk of unintentional status disclosure. The authors used their results to develop messages for the parent trial, deciding to allow participants to choose between overt and covert HIV-related texts, and found that two-thirds of participants opted for overt messages.

Another barrier to wide-scale implementation of SMS ART adherence interventions has been the lack of evidence on cost and cost-effectiveness, without which funders and policymakers are unlikely to invest in scale-up. Patel et al. (2017) combined effectiveness estimates from two SMS adherence effectiveness trials in Kenya with data from an East African cohort of patients living with HIV [35]. The investigators developed an individual-level microsimulation model to estimate the incremental costs and improvements to functional health status associated with these interventions [35]. Their results suggest that SMS adherence interventions in Kenya meet WHO standards for cost-effectiveness (base case estimate of $1037/QALY), and that this cost-effectiveness would be maintained once Kenya moved to a universal test and treat approach to HIV treatment.

eHealth interventions using tablets and MEMS caps have also been used to improve ART adherence. The CARE+ tablet-based education and information intervention (grounded in the TAM theoretical model) was tested for usability and effectiveness at improving ART adherence and HIV clinical outcomes among Spanish-speaking patients in New York City [29]. Kurth et al. (2016) found that the tablet program was user-friendly and acceptable to participants, and resulted in increased ART adherence and reduced viral load, though these differences were not statistically significant when compared with the control group [29]. Meanwhile, MEMS caps can provide feedback on the date and time of pill bottle openings, assisting patients in understanding their own pill-taking behaviors. The Adherence Improving self-Management Strategy (AIMS) study randomized 224 HIV patients in the Netherlands to the standard of care or an intervention arm, which included patient printouts of their MEMS data [32]. Nurse counselors used the MEMS adherence reports to describe optimal patterns of adherence, tailor counseling messages, and set adherence goals with each participant in the intervention arm. In addition to assessing intervention effectiveness, the researchers examined cost-effectiveness, estimating that AIMS reduced lifetime societal costs by €592 per patient [32]. Future research should explore the feasibility and cost of a similar approach to improve ART adherence in low-resource settings [9].

Discussion

IS has a growing role in supporting the development and scale-up of eHealth interventions for HIV care and treatment, though significant gaps remain. Recent studies have concentrated on assessing patient- and provider-level acceptability (the perception that the intervention is agreeable, satisfactory, or confers relative advantage) [30, 31, 39, 36, 37] and on exploring their feasibility (whether the intervention is suitable for everyday use, practicable, or fits with provider workflow) [30, 36–38]. Patient acceptability often relates to convenience or concerns about risks to confidentiality from the use of an HIV-related eHealth intervention, while provider acceptability derives from ease of use and the ability to integrate an intervention into current workflow. Adaptability of these interventions appears to be a key ingredient in feasibility, especially when considering future scale-up. Few studies measured provider- or facility-level adoption (early uptake or intent to try) or penetration (spread within an eligible population or level of institutionalization) of an HIV-related eHealth intervention [25, 30, 37]; more research is necessary to understand the factors associated with the successful diffusion and uptake of these interventions by health facilities, staff, and patients. Appropriateness (the pre-adoption perception of practicability, fit, or relevance), fidelity (whether the core components of an intervention were implemented as intended), and cost were rarely evaluated in our sample of studies; sustainability was never evaluated. Appropriateness is important because patients, providers, and facilities are unlikely to adopt an intervention if they do not perceive its utility and compatibility. Fidelity is critical to ensuring that successful outcomes can be repeated across settings [44]. Evidence of cost and cost-effectiveness is essential to justify scale-up given limited resources. Finally, we must study and promote sustainability (the extent an intervention can be maintained, routinized, or institutionalized by a provider or facility) to ensure we do not waste investments in startup and scale-up [45, 46]. These concerns are not trivial: many HIV-related eHealth interventions have low-tech alternatives (e.g., paper medical records) that may be sufficient or could be improved at lower cost than delivering a new high-tech intervention.

Theoretical frameworks, rigorous study designs, and reliable, valid measures enhance the generalizability and replicability of implementation research [47, 48], though only two of our included studies explicitly incorporated conceptual or theoretical frameworks for implementation research [25, 30], and none used validated quantitative measures of implementation outcomes like acceptability, feasibility, or appropriateness. Given the nascent nature of the eHealth and IS fields, this gap is not surprising. Moreover, much eHealth development is taking place in low- and middle-income countries, whereas essentially all currently available theoretical frameworks and validated measures for implementation outcomes were developed in high-income countries [21, 49]. These frameworks and measures will need to be adapted, expanded, or re-developed for ease of use and relevance in low-resource settings. In particular, use of standardized implementation outcome measures will promote the comparability of findings across studies, accelerating knowledge production in the field and easing the translation of findings into practice. Standardized measures are increasingly available [49]. For example, three new 4-item measures of acceptability, feasibility, and appropriateness have recently demonstrated promising psychometric properties [50], and a 40-item measure of program sustainability has been adapted for use in LMIC settings [51, 52].

Few eHealth interventions for HIV have been implemented beyond pilot studies or efficacy trials and none have achieved widespread usage. This suggests that we must expand and deepen our scope of inquiry to consider how we will deliver, scale, and maintain these interventions in the real world. To date we have focused on measuring levels of acceptability, feasibility, and adoption, alongside intervention effects on service delivery and patient health outcomes. The next step is to more comprehensively assess implementation outcomes like appropriateness, fidelity, cost, and sustainability, using standardized measures where possible, and examine patient-, provider- or facility-level predictors of those outcomes. For example, which staff and clinic characteristics predict rate of adoption of the HITSystem in Kenya? What is the perceived sustainability of WelTel in Kenya versus Canada? After that, the next step to experiment with intervention structure or implementation strategy and observe subsequent changes in relevant implementation, service delivery, and/or patient health outcomes - perhaps adopting a Type 1, 2, or 3 effectiveness-implementation hybrid trial design [22]. For example, do financial incentives promote the adoption of the T-HIT system by providers in Tanzania? Are patients using eTEST-enabled HIV self-tests more likely to link to care if follow-up calls occur within one hour of testing, or within one day? These questions are hypothetical and serve merely as examples. In practice, researchers should partner with policymakers, implementers, and end-users to identify priority implementation research questions and design impactful studies. Such research will have the potential to promote the translation of findings into practice, ensuring that we deliver on the promise of eHealth interventions to help meet the 90-90-90 targets worldwide.