Skip to main content
The Cochrane Database of Systematic Reviews logoLink to The Cochrane Database of Systematic Reviews
. 2021 Jul 27;2021(7):CD012944. doi: 10.1002/14651858.CD012944.pub2

Decision‐support tools via mobile devices to improve quality of care in primary healthcare settings

Smisha Agarwal 1,, Claire Glenton 2, Tigest Tamrat 3, Nicholas Henschke 4, Nicola Maayan 5, Marita S Fønhus 2, Garrett L Mehl 3, Simon Lewin 2,6
Editor: Cochrane Effective Practice and Organisation of Care Group
PMCID: PMC8406991  PMID: 34314020

Abstract

Background

The ubiquity of mobile devices has made it possible for clinical decision‐support systems (CDSS) to become available to healthcare providers on handheld devices at the point‐of‐care, including in low‐ and middle‐income countries. The use of CDSS by providers can potentially improve adherence to treatment protocols and patient outcomes. However, the evidence on the effect of the use of CDSS on mobile devices needs to be synthesized. This review was carried out to support a World Health Organization (WHO) guideline that aimed to inform investments on the use of decision‐support tools on digital devices to strengthen primary healthcare.

Objectives

To assess the effects of digital clinical decision‐support systems (CDSS) accessible via mobile devices by primary healthcare providers in the context of primary care settings.

Search methods

We searched CENTRAL, MEDLINE, Embase, Global Index Medicus, POPLINE, and two trial registries from 1 January 2000 to 9 October 2020. We conducted a grey literature search using mHealthevidence.org and issued a call for papers through popular digital health communities of practice. Finally, we conducted citation searches of included studies.

Selection criteria

Study design: we included randomized trials, including full‐text studies, conference abstracts, and unpublished data irrespective of publication status or language of publication. 

Types of participants: we included studies of all cadres of healthcare providers, including lay health workers and other individuals (administrative, managerial, and supervisory staff) involved in the delivery of primary healthcare services using clinical decision‐support tools; and studies of clients or patients receiving care from primary healthcare providers using digital decision‐support tools.

Types of interventions: we included studies comparing digital CDSS accessible via mobile devices with non‐digital CDSS or no intervention, in the context of primary care. CDSS could include clinical protocols, checklists, and other job‐aids which supported risk prioritization of patients. Mobile devices included mobile phones of any type (but not analogue landline telephones), as well as tablets, personal digital assistants, and smartphones. We excluded studies where digital CDSS were used on laptops or integrated with electronic medical records or other types of longitudinal tracking of clients.

Data collection and analysis

A machine learning classifier that gave each record a probability score of being a randomized trial screened all search results. Two review authors screened titles and abstracts of studies with more than 10% probability of being a randomized trial, and one review author screened those with less than 10% probability of being a randomized trial. We followed standard methodological procedures expected by Cochrane and the Effective Practice and Organisation of Care group. We used the GRADE approach to assess the certainty of the evidence for the most important outcomes.

Main results

Eight randomized trials across varying healthcare contexts in the USA,. India, China, Guatemala, Ghana, and Kenya, met our inclusion criteria. A range of healthcare providers (facility and community‐based, formally trained, and lay workers) used digital CDSS. Care was provided for the management of specific conditions such as cardiovascular disease, gastrointestinal risk assessment, and maternal and child health. The certainty of evidence ranged from very low to moderate, and we often downgraded evidence for risk of bias and imprecision.

We are uncertain of the effect of this intervention on providers' adherence to recommended practice due to the very low certainty evidence (2 studies, 185 participants). The effect of the intervention on patients' and clients' health behaviours such as smoking and treatment adherence is mixed, with substantial variation across outcomes for similar types of behaviour (2 studies, 2262 participants). The intervention probably makes little or no difference to smoking rates among people at risk of cardiovascular disease but probably increases other types of desired behaviour among patients, such as adherence to treatment. The effect of the intervention on patients'/clients' health status and well‐being  is also mixed (5 studies, 69,767 participants). It probably makes little or no difference to some types of health outcomes, but we are uncertain about other health outcomes, including maternal and neonatal deaths, due to very low‐certainty evidence. The intervention may slightly improve patient or client acceptability and satisfaction (1 study, 187 participants). We found no studies that reported the time between the presentation of an illness and appropriate management, provider acceptability or satisfaction, resource use, or unintended consequences.

Authors' conclusions

We are uncertain about the effectiveness of mobile phone‐based decision‐support tools on several outcomes, including adherence to recommended practice. None of the studies had a quality of care framework and focused only on specific health areas.   We need well‐designed research that takes a systems lens to assess these issues.

Plain language summary

Effect of decision‐support tools on mobile phones on primary health care

What was the aim of this review?

In this Cochrane Review, we aimed to find out if primary (community) healthcare workers using decision‐support tools on mobile phones or other mobile devices give better quality care. We looked for studies where researchers compared a decision‐support tool used on mobile phones to routine practice where there may be no guidance or some guidance in a paper format. We searched for studies conducted from 1 January 2000 to 9 October 2020. We found eight studies.

Key messages

We do not know if decision‐support tools used on mobile devices make primary healthcare workers better at following recommended practice. The evidence is not clear about the effects of these tools on patients' and clients' behaviour and on their health. We need more and better research to assess these issues.

What was studied in the review?

In many settings, patients receive low‐quality care. This is often because they live in poor or rural settings with few healthcare workers, or because healthcare workers do not have enough supplies, equipment, or proper training. Healthcare workers may struggle to stay up‐to‐date or may not have enough time to make the right decisions, which can result in poor quality of care for patients.

Decision‐support tools may help address some of these problems. A decision‐support tool helps the healthcare worker think through what he or she knows about the patient. The tool then helps guide the healthcare worker to the right decision for that patient. Designing decision‐support tools that can be used on mobile phones or other mobile devices such as tablets and personal digital assistants (PDAs) can make these tools easier to use and keep up‐to‐date.

The main aim of our review was to find out if healthcare workers using decision‐support tools on mobile phones give better healthcare than healthcare workers using decision‐support tools that are not on mobile phones or that use no decision‐support tools. We looked at the use of these tools in primary healthcare settings only.

What were the main results of the review?

We found eight relevant studies. Three studies were carried out in the USA and five studies in India, China, Guatemala, Ghana, and Kenya. These studies showed that when primary healthcare workers use decision‐support tools on mobile phones:

– we do not know if they are better at following recommended clinical practice, because the quality of this evidence was very low;

– there was no clear pattern of a positive or negative effect on patients' or clients' behaviour and on their health;

– this may slightly improve patients’ satisfaction with medical information;

– we do not know if this approach led primary healthcare workers to manage people’s health issues more quickly because we found no studies that measured this. We also found no studies that explored the effect on healthcare worker satisfaction, resource use, or whether this approach had any unintended consequences (e.g. harms).

How up‐to‐date is this review?

We searched for studies published up to October 2020.

Summary of findings

Summary of findings 1. Mobile clinical decision‐support system compared to standard care in primary healthcare settings: summary.

Mobile clinical decision‐support system compared to standard care in primary healthcare settings: summary
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers
Setting: primary healthcare settings (China, India, Ghana, Guatemala, Kenya, USA)
Intervention: mobile clinical decision‐support system
Comparison: standard care or no intervention (standard care could be providers using PDA with decision rules about a non‐intervention‐related health area, provider training and decision‐support tools on paper, paper‐based information booklet on management and follow‐up of people with diabetes, or usual care that did not involve any additional follow‐up)
Outcomes Effects of mobile clinical decision‐support systems (Number of studies, participants)) Certainty of the evidence (GRADE)
Providers' adherence to recommended practices, guidelines, or protocol Uncertain – the certainty of the evidence was very low (2 studies, 185 participants)a,b ⊕⊝⊝⊝
Very low
Time between presentation and appropriate management Uncertain – no direct evidence identified No evidence
Patients' or clients' health behaviour Probably makes little or no difference to the numbers of smokers among people with high cardiovascular disease risk (1 study, 2086 participants)c ⊕⊕⊕⊝
Moderate
Probably increases the number of people taking their antihypertensive medication (1 study, 2086 participants)c ⊕⊕⊕⊝
Moderate
May increase the number of people with high cardiovascular disease risk‐taking prescribed aspirin (1 study, 2086 participants)c but may make little or no difference to medication adherence among people with poorly controlled diabetes (1 study, 185 participants)d ⊕⊕⊝⊝
Low
Patients' or clients' health status and well‐being Probably makes little or no difference to systolic blood pressure among people with high cardiovascular disease risk (1 study, 2086 participants)c or to the number of women giving birth in a hospital (1 study, 799 participants)e ⊕⊕⊕⊝
Moderate
May make little or no difference to HbA1c levels among people with poorly controlled diabetes (1 study, 185 participants),d to the number of people with hyperlipidaemia reaching LDL cholesterol goals (1 study, 875 participants)f ⊕⊕⊝⊝
Low
Uncertain of the effect on maternal deaths and neonatal deaths (2 studies, 66,630 participants),e,g and a some other maternal health outcomes (1 study, 799 participants)e the certainty of evidence was very low ⊕⊝⊝⊝
Very low
Patient or client acceptability and satisfaction May improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (1 study, 187 participants)d ⊕⊕⊝⊝
Low
Providers' acceptability and satisfaction Uncertain – no direct evidence identified No evidence
Resource use Uncertain – no direct evidence identified No evidence
Unintended consequences Uncertain – no direct evidence identified No evidence
HbA1c: glycated haemoglobin; LDL: low‐density lipoprotein; PDA: personal digital assistant.

Background

The use of clinical decision‐support tools on mobile devices may help primary healthcare providers, including frontline healthcare workers, to improve the quality of services provided. The provision of appropriate, evidence‐based, quality healthcare services is a concern of global policy makers.

Description of the condition

 There is widespread recognition that the quality of healthcare services varies widely, and is often suboptimal across healthcare systems (Moja 2014WHO 2019WHO 2020). In primary healthcare, despite the availability of knowledge, and specific diagnostic, treatment, and management protocols, there is often a discrepancy between the knowledge and the application. This 'know‐do' gap in the quality of healthcare has been widely cited as a key barrier to improving healthcare outcomes (Blank 2013Mohanan 2015). A range of systemic factors contribute to the deficiencies and challenges in delivering high‐quality evidence‐based healthcare services.

Globally, the World Health Organization (WHO) projects a shortfall of 12.9 million healthcare providers by 2035 (WHO 2020). Having the right type of healthcare provider, at the right time, in the right place, continues to be a challenge worldwide. Clustering of health personnel in capital cities and other urban areas, and out‐migration from low‐ and middle‐income countries (LMICs) to high‐income countries (HICs), further diminishes the number of healthcare providers available in rural areas (Dussault 2006). Especially in LMICs, the lack of trained primary healthcare providers has prompted policy makers to explore a shift of key tasks from higher to lower cadres of health workers (Baker 2007Lehmann 2008WHO 2016). While the transition of vital primary healthcare services to a lower cadre of workers is feasible, it requires ongoing training support to ensure that service quality and safety standards are maintained (Rednick 2014WHO 2016).

In other clinical settings, even when an adequate number of providers with the right training are available, the quality of care is variable. Busy healthcare providers may struggle to stay abreast of current evidence and apply it consistently. They may also lack information on alternate risk‐reduction approaches, and be ensure which treatments work best (Kocher 2010; Middleton 2016). Time‐constraints may result in the omission of essential information for counselling the patient, and long work hours may result in increased errors (Bright 2012; Sutton 2020).

Standardized protocols, which provide critical information at the point‐of‐care, support decision‐making and guide healthcare providers through the process of diagnosis and management. They can introduce efficiencies into the system, optimize the time with the client, and improve the overall quality of services (Bright 2012Mickan 2014Sutton 2020).

Description of the intervention

Several challenges encountered in clinical practice could benefit from using clinical decision‐support systems (CDSS). One definition of a CDSS is "any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient‐specific assessments or recommendations that are then presented to clinicians for consideration" (Kawamoto 2005). The increasing ubiquity and affordability of smartphones and tablets has made it possible for clinical decision‐support tools to become available to healthcare providers on handheld devices at the point‐of‐care, and broadened the definition to include clinical guidelines and protocols that might be available in a digital format. Mobile decision‐support tools can potentially address some of the challenges faced by many healthcare systems regarding adequate training of healthcare providers, shifting clinical tasks from clinicians to frontline health workers with limited formal training, and improving the quality of preventive, diagnostic, and treatment care across a range of health issues (Carter 2019Orton 2018).

Since the mid 2000s, personal digital assistants (PDAs), and other wireless mobile devices, such as smartphones and tablets are increasingly being used at the point‐of‐care to obtain evidence and guidance on clinical conditions, do necessary calculations for drugs, and access other medical information (Orton 2018Richardson 2019Yau 2019). They also support more advanced CDSSs linked to a comprehensive patient database (Divall 2013). Where before, in most low‐income settings, only basic features, such as voice calls (or interactive voice response (IVR)) and short message service (SMS) were commonplace, the pace of growth in mobile technology increasingly allows for a range of functionality (e.g. low‐cost access to the internet, high‐quality cameras for still and video footage, applications stored on‐device, preloaded audio or video clips and images, global positioning service (GPS), and the potential to connect additional sensors and devices) that allows for the development of very sophisticated point‐of‐care decision‐support systems, even in low‐resource settings (Orton 2018).

Mobile CDSS may vary in the range of functionality and applications to improve diagnoses; facilitate evidence‐based screening, counselling, and treatment; and improve workflow efficiencies.

Broadly, CDSS may serve the following functions:

  • guide the healthcare provider through process algorithms using 'if…then… rules, based on evidence‐based clinical protocols;

  • provide the healthcare provider with a checklist, based on clinical protocols;

  • provide step‐by‐step guidance to screen clients by health status or risk status, possibly using models based on machine learning, where mathematical functions might be used to classify patients into risk groups.

The adoption behaviours and effectiveness of CDSS may vary based on the function it is intended to serve. CDSS may be integrated with electronic health records or stand alone. For the purpose of this review, we focused only on mobile decision‐support tools that were not integrated with an electronic health record or management system, or were integrated with such a system but could be used independent of it.

How the intervention might work

Approaches to using mobile clinical decision‐support tools vary substantially across countries and contexts, in part, depending on the availability of technological and healthcare infrastructure, and the costs of mobile devices and data packages. In addition to the costs and infrastructure, the level of sophistication of a CDSS would also depend on the complexity of the disease, the purpose of the CDSS (e.g. screening alone, or screening integrated with risk assessment and counselling), and the capacity of the healthcare providers to adopt such systems.

At the most basic level, a decision‐support system might comprise of a process algorithm, and transfer a paper‐based protocol into a digital format. For example, a series of digital 'If…then…' logic‐guided questions may be used to assess appropriate contraceptive choices for a client, based on demographic information and preferences. By inputting client data in a systematic way, a decision‐support tool might be used to identify and prioritize clients into risk groups. Additional point‐of care support mechanisms, such as automated algorithmic instructions that prompt healthcare workers to follow certain guidelines, and provide tailored counselling messages and recommendations, might be added to such a system. For example, in addition to the assessment for an appropriate contraceptive, the decision‐support tool may provide a number of recommendations, and prompt the healthcare worker to discuss risks for each contraceptive method, and care and follow‐up for the contraceptive method chosen by the client. At each step, the provider might be required to check off a counselling item that has been discussed, before the system moves to the next set of questions and recommendations. In theory, such a system would promote comprehensive evidence‐based counselling, and improve the overall quality of care provided by the healthcare worker.

In HICs, CDSS is typically used in clinical settings by trained healthcare providers or medical students. In LMICs, decision‐support tools are used for both in‐clinic and community‐based outreach visits, by a range of healthcare providers, including clinicians, midwives, and lay healthcare workers.

Why it is important to do this review

The use of decision‐support tools for clinical decision‐making has been well‐established, and is supported by some emerging evidence. Bright 2012 conducted a systematic review of 148 randomized trials on the effectiveness of CDSS in 2012. They reported that the use of CDSS was associated with significant improvements in a range of morbidity outcomes, healthcare process measures related to performing preventive services, diagnostic testing, and improved adherence to treatment regimens (Bright 2012). However, since the early 2010s, clinical decision‐support tools have transitioned from being operated on stationary computer systems to wireless mobile devices, which provide additional opportunities for point‐of‐care assessments, diagnoses, and management. Furthermore, most healthcare systems in LMICs, especially in rural areas, do not have the required infrastructure for computerized CDSS (Richardson 2019; Yau 2019). The use of these tools on wireless digital devices makes them accessible to healthcare providers in LMICs, which was not possible previously. Despite the substantial investments and global interest in using mobile digital devices to support clinical decision‐making, specific evidence on the effectiveness of such interventions on clinical and public health practice is limited.

Two systematic reviews assessed whether the use of handheld computers, primarily PDAs, improved access to information and supported point‐of‐care clinical decision‐making (Divall 2013Mickan 2014). Compared to paper resources, the reviews suggested that using handheld computers improved access to information, adherence to clinical guidelines, appropriate diagnostic decision making, and data collection quality (Divall 2013). This review will build on the existing studies, to include the use of other mobile devices, such as smartphones and tablets, which are the most current forms of handheld digital devices, especially in LMICs. One review assessed the feasibility of, and barriers to, using digital point‐of‐care decision‐support tools by healthcare providers in Africa (Adepoju 2017). Based on largely descriptive and observational studies, conducted in seven sub‐Saharan African countries, the review concluded that healthcare providers found mobile decision‐support tools useful; however, they expressed concerns about altered workflows and increased workloads. The review identified technical and infrastructural support, and adequate training, as key barriers to adopting clinical decision‐support tools in the sub‐Saharan African context.

Digital, mobile, wireless technologies provide an innovative and accessible platform to accelerate health services and improve quality of care for some of the most difficult‐to‐reach populations. Given the recent emergence of such technologies for health, there is considerable demand from ministries of health, donors, and decision makers for evidence‐based guidance to invest in such technologies. In response to this global need, the WHO is developing guidelines to inform investments on digital health approaches. This review constitutes one of 11 reviews on the effectiveness of digital health interventions that will be used directly to inform these WHO guidelines (WHO 2019).

Objectives

To assess the effects of digital clinical decision‐support systems (CDSS) accessible via mobile devices by primary healthcare providers in the context of primary care settings.

Methods

Criteria for considering studies for this review

Types of studies

We included randomized controlled trials (RCTs), available as full‐text studies, conference abstracts, and unpublished data. We included studies regardless of their publication status and language of publication.

Types of participants

  • All cadres of healthcare providers (i.e. professionals, paraprofessionals, and lay health workers) providing healthcare services to patients, using digital, clinical decision‐support tools in the context of a primary care setting.

  • Other individuals or groups involved in the delivery of primary healthcare services, including administrative staff, managerial, and supervisory staff, who may or may not have been based in a primary healthcare facility or in the community but must have been involved in supporting the delivery of primary healthcare services using digital, clinical decision‐support tools.

  • Clients or patients receiving care from primary healthcare providers who were using decision‐support tools.

We included participants regardless of their location, professional status, condition, or demographic factors, such as age.

Types of interventions

We included studies that compared digital, clinical decision‐support tools accessible via mobile devices with non‐digital decision‐support tools, or no intervention, in the context of primary care. We included studies in which digital, decision‐support tools were developed for use primarily on a mobile device, and were used by health workers for the purpose of service delivery, to follow clinical protocols, guide service delivery using checklists and job aids, or prioritize clients by risk or other health status in a primary healthcare setting.

Mobile devices were mobile phones of any type (but not analogue landline telephones), as well as tablets, PDAs, and smartphones. We included studies if a mobile device was used, and the tool was intended to be used in a mobile state. For example, if websites or other applications were used, they should have been optimized for use on a mobile device, and healthcare workers should have been trained to use the mobile device. We included studies in which a laptop was used as a tablet, with applications customized for such use.

Primary healthcare services were a combination of the following:

  • the first contact point of healthcare (Awofeso 2004), including care delivered at an individual level, community level, or both (Muldoon 2006), by individual healthcare providers or teams of providers, and intended to provide and co‐ordinate care in settings where people work and live, or provide continuity of care (Muldoon 2006);

  • any healthcare that prevented illness, promoted health, was therapeutic, or was rehabilitative (Global Health Watch 2011).

The intervention may have been implemented in public or private healthcare facilities, in the community, or the homes of the patients. We included studies in any country.

The comparisons for this review were:

  • digital decision‐support tools accessible via mobile device compared to non‐mobile and non‐digital decision‐support tools (e.g. a mobile job‐aid versus a paper job‐aid);

  • digital decision‐support tools accessible via mobile device compared to standard practice (i.e. non‐digital intervention or no intervention).

We excluded:

  • studies in which the use of the digital decision‐support tool was dependent on its integration with an electronic medical record or other types of client health‐tracking tools;

  • studies in which the use of the decision‐support tool was primarily for the purpose of training alone, and did not involve direct service delivery;

  • studies in which digital decision support was conducted on stationary computers or laptops alone;

  • studies that compared one type of mobile decision support with another type of mobile decision support;

  • studies in which patients used the digital decision‐support systems;

  • pilot and feasibility studies (pilot study was defined as "a version of the main study that is run in miniature to test whether the components of the main study can all work together," and feasibility study was defined as "pieces of research done before a main study" (Arain 2010)).

We included studies in which digital decision‐support tools were delivered as part of a wider package (such as sending messages to the client or provider, supporting the provider in prioritizing clients, etc.), if the decision‐support tool was the major component of the intervention.

Types of outcome measures

Primary outcomes
  • Providers' adherence to recommended practices, guidelines, or protocols (e.g. providing the service at the recommended time, referral as recommended, screening and prioritizing as recommended).

  • Time between presentation and appropriate management, including time for referrals and service linkages.

  • Patients' or clients' health behaviour.

  • Patients' or clients' health status and well‐being, assessed through validated measures, if available.

  • Patient or client acceptability and satisfaction with the intervention, assessed through validated measures, if available.

  • Provider acceptability and satisfaction with the intervention, assessed through validated measures, if available.

  • Resource use (e.g. human resources and time, training, supplies, and equipment).

  • Unintended consequences that resulted in an adverse effect of the intervention (these could have included misreading or misinterpretation of the data; transmission of inaccurate data, e.g. incorrect underlying algorithms or clinical protocols; loss of verbal or non‐verbal communication cues; decreased direct engagement with patient; issues of privacy and disclosure; loss (including theft) or misuse of device (in cases in which health workers were given the phones or tablets); interrupted workflow due to infrastructural constraints for battery recharging and network coverage; impacts on equity; disruptions on the delivery of health services, unforeseen ill‐effects on patient outcomes).

Secondary outcomes

None.

Search methods for identification of studies

We restricted the search from 2000. This was based on the increased availability and penetration of mobile devices in LMICs starting in 2000 (International Telecommunications Union 2015).

Electronic searches

An independent information specialist developed the search strategies in consultation with the review authors.

We searched the following databases for primary studies from 2000.

  • Cochrane Central Register of Controlled Trials (CENTRAL), Issue 9, 2020, the Cochrane Library.

  • MEDLINE and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily, Ovid (searched 9 October 2020).

  • Embase 1974 to 2020 week 40, Ovid (searched 9 October 2020).

  • Global Index Medicus/Global Health Library, WHO (searched 9 October 2020).

  • POPLINE, K4Health (searched 5 August 2019).

Search strategies were comprised of keywords and controlled vocabulary terms. We applied no limits on language. We used a modified version of the Cochrane Highly Sensitive Search Strategy to identify randomized trials (Lefebvre 2011). All search strategies used are provided in Appendix 1.

Searching other resources

We searched for ongoing trials in the following trial registries:

  • WHO ICTRP (International Clinical Trials Registry Platform; www.who.int/ictrp) (searched 5 August 2019);

  • US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov (www.clinicaltrials.gov) (searched 9 October 2020).

We searched Epistemonikos (www.epistemonikos.org) for relevant systematic reviews and potentially eligible primary studies. In addition, the WHO issued a call for papers through popular digital health communities of practice, such as the Global Digital Health Network and Implementing Best Practices, to identify additional primary studies and grey literature.

Grey literature

We searched mhealthevidence.org for grey literature to 2018. The search portal for mhealthevidence.org is more limited, therefore, we reviewed the titles and abstracts of all contributed content not referenced in MEDLINE Ovid. This database was discontinued after 2018.

We reviewed reference lists of all included studies and relevant systematic reviews for additional potentially eligible primary studies. We contacted authors of included studies and reviews to clarify reported published information and to seek unpublished data. We allowed a response time of up to one month from authors who were contacted.

Data collection and analysis

Selection of studies

A core team of two review authors (NH, NM), with assistance where necessary from an additional review author (SA), were responsible for the selection of studies. We downloaded all titles and abstracts retrieved by electronic searching to a reference management database and remove duplicates (DistillerSR). We used a machine learning classifier that is able to assign a probability score that a given record described, or did not describe, a randomized trial. It was built based on 280,000 titles and abstracts from Embase, which have been manually labelled by the Cochrane Crowd (Wallace 2017).

We processed all the search results through the classifier. Two review authors independently screened the titles and abstracts of studies with a 10% probability or greater of being a randomized trial; one review author screened those with a less than 10% probability of being a randomized trial.

We retrieved the full‐text study reports and publications of studies that are screened and included. Two review authors (NH, NM) independently screened the full texts to identify studies to include, and recorded reasons for excluding ineligible studies. We resolved any disagreements through discussion, or if required, we consulted a third review author (SA).

We listed studies that initially appeared to meet the inclusion criteria, but were excluded after full‐text review, in the Characteristics of excluded studies table. We collated multiple reports of the same study, so that each study rather than each report was the unit of interest in the review. We provided any information we could obtain about ongoing studies in the Characteristics of ongoing studies table. We recorded the selection process in sufficient detail to complete a PRISMA flow diagram (Liberati 2009).

Data extraction and management

We modified the Effective Practice and Organisation of Care (EPOC) group standard data collection form and adapted it for our study characteristics and outcome data (EPOC 2017a). We identified key characteristics of the intervention for extraction based on the mHealth Evidence Review and Assessment (mERA) guidelines (Agarwal 2016). We piloted the form on at least one study in the review. Two review authors (NH, NM) independently extracted the following study characteristics from the included studies.

  • General information: title, reference details, author contact details, publication type, funding source, conflicts of interest of study authors.

  • Population and setting: country, geographical location (rural, urban, or peri‐urban, defined as outskirts of urban areas), healthcare setting (e.g. facility‐based, home‐based).

  • Methods: function of the intervention, study design, unit of allocation, duration of participation.

  • Participant characteristics: type of healthcare worker (function, age, length of training), description of clients serviced by the healthcare worker, description of any other participants in the intervention, withdrawal.

  • Interventions: intervention purpose, components, type of technology (hardware and software characteristics) and model of delivery, type of mobile device(s) used (smartphone, tablet, feature phone, basic phone, laptops), phone ownership, content of the intervention, health provider training, interoperability, compliance with national guidelines, data security, comparison, fidelity assessment, duration of intervention.

  • Outcomes: primary and other outcomes specified and collected, time points reported, adverse events, results of any subgroup analyses.

Two review authors (NM, NH) independently extracted outcome data from the included studies. We noted in the Characteristics of included studies table if outcome data were reported in an unusable way. We resolved disagreements by consensus, or by involving a third review author (SA).

Assessment of risk of bias in included studies

Two review authors (NH, NM) independently assessed risk of bias for each included study, using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2017), and the guidance from the EPOC group for assessing randomized trials (EPOC 2017b). We resolved any disagreements by discussion, or by involving a third review author (SA). We assessed the risk of bias according to the following domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, similarity of baseline characteristics, and any other bias. We assessed incomplete outcome data separately for different outcomes.

We judged each potential source of bias as high, low, or unclear and provided a quote from the study report, together with a justification for our judgement in the risk of bias table. We summarized the risk of bias judgments across different studies for each of the domains listed. We considered blinding separately for different key outcomes where necessary. Where information on risk of bias related to unpublished data, or correspondence with a trialist, we noted this in the risk of bias table. We did not exclude studies on the grounds of their risk of bias, but clearly reported the risk of bias when presenting the results of the studies. When considering treatment effects, we took into account the risk of bias for the studies that contributed to that outcome (Guyatt 2008).

We conducted the review according to the published protocol (Agarwal 2018), and reported any deviations from it in the Differences between protocol and review section of the systematic review.

Measures of treatment effect

We analyzed the treatment effects in the individual trials using Review Manager 2014. We estimated the effect of the intervention using risk ratio (RRs) with 95% confidence intervals (CI) for dichotomous data, and mean difference (MD; where studies used the same scale), or standardized mean difference (SMD; when studies used different scales) with 95% CIs for continuous data (Higgins 2017). We ensured that an increase in scores for continuous outcomes could be interpreted in the same way for each outcome, explained the direction to the reader, and reported where the directions were reversed, if this was necessary.

Unit of analysis issues

For cluster‐randomized trials that did not account adequately for the effects of clustering on the effect estimate, we planned to adjust the data prior to entry into the meta‐analysis to avoid unit‐of‐analysis errors. If insufficient information was available to reanalyze the results, we contacted the authors of the primary paper to request the necessary data. If these data were not available, we planned to report the effect estimates without the CIs or P values. However, as no formal meta‐analysis was carried out, due to the heterogeneity across studies reporting similar outcomes, data adjustment from cluster RCTs was not performed.

For cross‐over trials, we prioritised the inclusion of data that was collected before the cross‐over occurred. Where a unidirectional cross‐over occurred (i.e. only one group crossed‐over) this was accounted for in the GRADE rating for each outcome as a potential source of indirectness.

Dealing with missing data

We contacted investigators to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study was identified as abstract only). If this was not possible or we were unable to get in touch with the investigators, we reported the data as missing, noted this in the risk of bias tables, and did not attempt to impute the missing values. There were no adjustments to the analyses to account for adverse events as none of the studies reported adverse events.

Assessment of heterogeneity

We examined heterogeneity by visual inspection of forest plots, as well as using the I² statistic to measure heterogeneity among the trials in each analysis.

Assessment of reporting biases

We planned to assess reporting bias by creating and examining funnel plots if we were able to pool more than 10 trials within a comparison (Sterne 2011). However, only five studies were included, and no pooling was possible.

Data synthesis

Where intervention characteristics and outcome measures were similar across included studies (i.e. if the treatments, participants, and the underlying clinical question were similar enough for pooling to make sense), we planned to conduct a meta‐analysis to estimate an overall effect size. If analyses, adjusted for potential confounders, were reported for either dichotomous or continuous outcomes, we reported these estimates of effect from the primary analysis. In cases where the adjusted analyses for dichotomous outcomes were reported using odds ratios (OR) and not RRs, we planned to convert OR to RR before including the result in a meta‐analysis using the guidance of the Cochrane Handbook for Systematic Reviews of Interventions McKenzie 2021). As no meta‐analysis was performed, we reported adjusted OR directly from the included studies.

We planned to note where reported data were skewed and consider the implication of this; however, there were no skewed data.

The included studies were not similar enough to allow meta‐analysis. Therefore, we reported the results in a narrative format.

Guyatt 2008Schünemann 2017EPOC 2017cGRADEpro GDTEPOC 2017c

Subgroup analysis and investigation of heterogeneity

We had planned to perform subgroup analysis to assess the variation in the delivery of the intervention across different population groups, interventions, or setting characteristics, if possible. We had planned to conduct subgroup analyses only if a sufficient number of trials were available to make statistically significant comparisons between groups. However, we did not identify a sufficient number of trials, and therefore, did not conduct any subgroup analyses.

Sensitivity analysis

We had planned to perform sensitivity analyses to assess the robustness of our conclusions and explore its impact on effect sizes. We had planned to restrict any meta‐analysis to published studies only, and remove studies from any meta‐analyses that had a high risk of bias, based on the risk of bias assessment. However, we did not conduct any sensitivity analyses as there were insufficient studies to conduct meta‐analysis.

Summary of findings and assessment of the certainty of the evidence

Two review authors independently assessed the certainty of the evidence (high, moderate, low, or very low) using the five GRADE considerations (risk of bias, consistency of effect, imprecision, indirectness, and publication bias; Guyatt 2008). We used the methods and recommendations described in the Cochrane Handbook for Systematic Reviews of interventions (Schünemann 2017), and the EPOC worksheets (EPOC 2017c), and used GRADEpro GDT software (GRADEpro GDT). We resolved disagreements on certainty ratings by discussion, and provided justification for decisions to downgrade or upgrade the ratings using footnotes in the table, and made comments to aid readers' understanding of the review, where necessary. We used plain language statements to report these findings in the review (EPOC 2017c).

We created summary of findings tables for the main comparisons for the following outcomes:

  • providers' adherence to recommended practice, guidelines, or protocols;

  • time between presentation and appropriate management;

  • patients' or clients' health behaviour;

  • patients' or clients' health status and well‐being;

  • patient or client acceptability and satisfaction with the intervention;

  • provider acceptability and satisfaction with the intervention;

  • resource use;

  • unintended consequences that result in an adverse effect of the intervention.

We drew conclusions about the certainty of the evidence within the text of the review. Outcomes for the main summary of findings tables were selected based on whether the indicators were validated, globally accepted, or considered to be of clinical/public health importance.

Results

Description of studies

We retrieved 7777 unique records for title and abstract screening after removing duplicates, and shortlisted 431 records for full‐text screening. Of these, we identified eight randomized trials that were eligible for this review. We included seven of these in our quantitative analysis.

Results of the search

 We described the most pertinent excluded studies in the Characteristics of excluded studies table. There are three studies are awaiting assessment and one ongoing study. The study selection process is summarized in Figure 1.

1.

1

Study flow diagram.

Included studies

Details of the characteristics of the study population, intervention strategies, and outcomes are presented in the Characteristics of included studies table and Table 2.

1. Description of digital interventions employed by the included studies.
Study ID* Type of technology Intervention description Phone ownership Provider training Compliance with guidelines
Mobile device Software
Amoakoh 2019 Basic (non‐smart) mobile phone None mCDMSI consisted of 4 components: phone calls (voice), text messaging (SMS), access to the internet (data) and access to an USSD that provided protocols for management of obstetric and neonatal emergencies. Midwives were given individual‐use phones, and health facilities had a shared phone Health workers were trained before program initiation and once during successive monitoring visits Ghana's Safe Motherhood Protocol
Berner 2006 PDA Palm Operating System 4.01 CDSS (MedDecide) on PDA, used to support non‐steroidal anti‐inflammatory drug (NSAIDs)‐related GI risk assessment and treatment recommendations. PDA provided to physicians 1 × 30‐minute training session with demonstration and instruction on use of all programs Based on published evidence‐based literature
Eaton 2011 Computer kiosk and PDA Interactive CDSS software Interactive decision support software on a PDA using data from patients regarding their risk factors for CHD to improve adherence to guidelines PDA provided to physicians 1‐hour academic detailing session + 4 additional sessions on clinical guidelines and use of CDSS tools National Cholesterol Education Program guidelines
Gautham 2015 Smartphones (android, iOS, Symbiam); windows mobile 6.5 cell phone Gui deVue was used for mobile media‐rich interactive guideline system Media‐rich, mobile phone‐based clinical guidance system for management of fevers, diarrhoeas, and respiratory problems. Unclear 2‐day training programme in guideline‐based care and in use of the mHealth system WHO Integrated Management of Childhood and Adult illnesses
Heisler 2014 Tablets with 3G access iDecide Personally tailored, interactive diabetes medication decision aid designed for CHWs to deliver information and treatment to individuals with diabetes and low health literacy. Tablets were provided 80 hours of initial training in motivational interviewing‐based communication and diabetes self‐management support, with 4–8 hours of booster training annually AHRQ Guides ("Pills for Type 2 Diabetes" and "Premixed Insulin for Type 2 Diabetes")
Keane 2018 Smartphones (android, iOS, Symbiam, tablets Commcare (IMAM app) App provided step‐by‐step guidance on assessment, treatment, and referral of children for malnutrition. Smartphones were provided 3‐day training on the use of the tablets/phones Not reported
Martinez 2018 Samsung S3 mini smartphone (android) with peripheral sensor devices Customized android‐based app The app allowed collection of maternal and perinatal symptoms and clinical signs, maternal vital signs and the fetal heart rate. Using these data, TBAs were guided through detection of complications and refer pregnant women to health facilities. Smartphones were provided 4‐day training led by study nurses to review medical concepts on perinatal complications and use of smart phone Not reported
Tian 2015 Smartphones (android, iOS, Symbiam) Customized android‐based app The decision‐support component assisted the CHWs on the cardiovascular disease management of their patients based on patients’ medical history, lifestyle, blood pressure and medication. Smartphones were provided Initial 1‐day systematic training. Refresher training every 3–4 months during the implementation Simplified international and national guidelines on cardiovascular disease management
* None of the studies reported on data security and interoperability

CDSS: clinical decision‐support system; PDA: personal digital assistant; WHO: World Health Organization.

Locations and populations

We identified eight randomized trials that fulfilled the inclusion criteria (Amoakoh 2019Berner 2006Eaton 2011Gautham 2015Heisler 2014Keane 2018Martinez 2018Tian 2015). Three trials were conducted in the USA (Berner 2006Eaton 2011Heisler 2014), with two of the three conducted in urban clinics (Berner 2006Heisler 2014). Three studies were conducted in rural areas: one in India (Gautham 2015), one in China and India (Tian 2015), and one in Guatemala (Martinez 2018). Two studies were conducted in Ghana (Amoakoh 2019), and Kenya (Keane 2018). Of the three studies in the USA, one was conducted with internal medicine residents and standardized patients trained to convey specific health conditions, in a university outpatient setting (Berner 2006); and two were conducted in primary care settings: one with community health workers in Detroit (Heisler 2014), and one with physicians in New England (Eaton 2011).

The five studies from LMICs enrolled a range of healthcare providers to deliver the CDSS intervention: community health workers (Tian 2015), male and female rural health providers with varying levels of training and experience (Gautham 2015), traditional birth attendants (TBA; Martinez 2018), and primary health facility‐based providers (Amoakoh 2019Keane 2018).

Intervention strategies

The interventions aimed to improve provider adherence to the treatment protocol across a range of health areas: integrated management of childhood and adult illnesses for management of fevers (IMCI/IMAI), diarrhoea and respiratory problems; risk assessment, counselling, and treatment of non‐communicable diseases including diabetes management, and cardiovascular disease; and gastrointestinal (GI) risk assessment.

The CDSS tool tested by Gautham 2015 used mobile media‐rich interactive guidelines (mMRIGs) with audio, images, and video to support protocol compliance for IMCI/IMAI by rural health providers. 

In Eaton 2011, physicians received a PDA‐based decision‐support tool to improve adherence to National Cholesterol Education Program Guidelines. The CDSS algorithm helped determine the patient's lipid diagnosis (low‐density lipoprotein (LDL) dominant, isolated low high‐density lipoprotein (HDL) level, triglyceride dominant, mixed lipid disorder, and atherogenic dyslipidaemia), calculated the LDL and non‐HDL cholesterol goals (when appropriate), made recommendations regarding therapeutic lifestyle management, provided optimal dosage of lipid‐lowering drugs tailored to the patient's risk factor status, and provided an interactive shared decision‐making page for physicians to discuss lowering lipid values, and other coronary heart disease (CHD) risk factor management. 

Another intervention targeted at cardiovascular health was tested in community‐based settings in India and China (Tian 2015). 

Community health workers used a smartphone‐based android application consisting of prompts to gather data on patient's medical history, new symptoms and diagnoses, medication usage, current lifestyle habits, and blood pressure, and provide guidance for prescribing any of the target antihypertensive medications. iDecide is a personally tailored, interactive diabetes medication decision aid for use by community health workers in a community health centre has four core components (Heisler 2014): 1. animations on the effect of diabetes on how glucose is processed in the body and how different medication classes, foods, and physical activity affect blood sugar; 2. pictographs showing participants' own risk of diabetes complications (tailored based upon their baseline glycated haemoglobin (HbA1c)); 3. participants review their current diabetes medications and barriers to taking medications; 4. prompts participants to set goals and develop specific action plans.

Berner 2006 examined the effect of a personal digital assistant‐based CDSS tool, used to support non‐steroidal anti‐inflammatory drug (NSAIDs)‐related GI risk assessment and treatment recommendations. All participants received some rules on a PDA that could potentially apply to the standardized patients' complaints; however, only the 34 participants who were randomized to the intervention arm received the rules for GI risk assessment when prescribing NSAIDs.

Three studies targeted interventions at improving maternal and neonatal outcomes. 

Martinez 2018 provided TBAs in Guatemala with an android‐based platform to collect demographic data, maternal and perinatal symptoms, and maternal vital signs using a range of peripheral sensor devices (pulse oximeter, hand‐held one‐dimensional Doppler ultrasound, self‐inflating oscillometric blood pressure cuff). Based on the symptoms, TBAs could check a list of common maternal and perinatal complications presented as pictures, which then triggered automatic communication with the on‐call clinical team by voice call or a text message. 

In Keane 2018, facility workers were provided with a smartphone/tablet‐based app that provided step‐by‐step guidance on assessment, treatment, and referral of children for malnutrition. The app provided treatment protocols, counselling messages, and calculated z‐scores for assessment of malnutrition and provision of ready‐to‐use therapeutic food sachets. The data were uploaded to a cloud server where they could be used for program management. 

Amoakoh 2019 used a four‐component decision‐support intervention in Ghana comprised of phone calls, text messages, access to data services, and to unstructured supplementary service data (USSD). The USSD allowed health workers to access obstetric and neonatal clinical management protocols, and for two‐way communication between frontline midwives and health facility health workers. Health workers also received monthly reminders about the availability of these protocols.

Outcomes

Outcomes were reported as follows.

Providers' adherence to recommended practices, guidelines, or protocols

Two trials reported this outcome: Berner 2006 reported the proportion of cases per physician with unsafe practices, and proportion of cases per physician with key GI risk factors recorded. Gautham 2015 reported protocol compliance for diagnoses and management of adult and child fevers, diarrhoea, and respiratory problems by rural healthcare providers.

Time between presentation and appropriate management

We found no studies that reported time between presentation and appropriate management.

Patients' and clients' health behaviour

Heisler 2014 reported antihyperglycaemic medication decisional conflict (at three months of follow‐up), diabetes self‐care efficacy, change in HbA1c, and medication adherence on a scale of 1 to 100 over three months of follow‐up. Tian 2015 reported health behaviours associated with cardiovascular health‐proportion of current smokers (at one year of follow‐up), high‐risk people taking aspirin in the past month, and self‐reported use of antihypertensive medication for 25 or more days in the past month.

Patients' or clients' health status and well‐being

Six trials reported this outcome. Eaton 2011 reported the proportion of patients reaching LDL‐cholesterol goals and non‐HDL‐cholesterol goals at one year of follow‐up after the intervention. These outcomes were presented by low, medium, and high cardiovascular risk status; however, the data were not presented in a way that could be analyzed. Heisler 2014 reported diabetes distress at three months of follow‐up. Tian 2015 reported health outcomes associated with cardiovascular health (i.e. change in mean systolic pressure, proportion of high‐risk people hospitalized in the last year). These outcomes were measured with one year of follow‐up after the intervention. Martinez 2018 reported several maternal health outcomes including monthly emergency facility referral rate by type of maternal/perinatal complication (labour progression abnormality, hypertensive disorder of pregnancy, haemorrhage, premature labour, fetal cardiac abnormality, suspected neonatal sepsis, neonatal respiratory compromise, preterm newborn), and number of maternal and neonatal deaths. Amoakoh 2019 reported the total number of neonatal deaths (death occurring from birth to 28 days) from all deliveries over an 18‐month period, and the total number of maternal deaths among all antenatal care attendants at health facilities. These outcome definitions differed from globally accepted definitions of neonatal mortality rate (the number of neonatal deaths per 1000 live births) and maternal mortality rate (the number of maternal deaths in a given period per 100,000 women of reproductive age) or maternal mortality ratio (number of maternal deaths per 100,000 live births). Keane 2018 measured the number of neonatal deaths, maternal deaths, and the proportion of children who were cured of malnutrition, defaulted treatment, or died. The data were unclearly reported and, therefore, not included in the analysis.

Patients' or clients' acceptability and satisfaction

Heisler 2014 measured satisfaction with clarity of medication information and satisfaction with helpfulness of medication information.

Provider acceptability and satisfaction

One study reported this outcome (Gautham 2015). However, the outcome was reported for the intervention arm only. Provider acceptability and satisfaction was measured through a series of questions on the level of comfort with the system, willingness to continue using the system, being able to remember the steps without the system, usefulness of combination of media, willingness to recommend the system, wish for more health conditions to be included in the system, ease of use of the system, and helpfulness of the system in following guidelines.

Resource use

No studies reported resource use.

Unintended consequences

No studies reported unintended consequences.

Excluded studies

We excluded 431 full‐text articles with reasons: 68 did not meet the study design criteria; interventions in 124 articles did not meet the definition for a CDSS; 63 studies were not conducted in a primary care setting; interventions in 86 studies were also integrated with an electronic medical record; 20 intervention studies did not involve direct patient care; interventions in 12 studies were not accessible primarily by a mobile phone; in 10 studies the comparison arm did not meet inclusion criteria; in 30 studies the intervention was not used by a healthcare provider; and in five studies the digital intervention was not core to the main intervention. 

We described the most pertinent excluded studies in the Characteristics of excluded studies table.

Studies awaiting assessment

Three studies are awaiting assessment (Keitel 2017Khan 2020de Molina‐Férnandez 2019Studies awaiting classification table).

Ongoing studies

We found one ongoing study (NCT03311399Characteristics of ongoing studies table).

Risk of bias in included studies

We used Cochrane's tool for assessing the risk of bias in each individual study (presented in Figure 2 and Figure 3), which are summarized in the Characteristics of included studies table.

2.

2

Risk of bias graph: review authors' judgements about each risk of bias item presented as percentages across all included studies.

3.

3

Risk of bias summary: review authors' judgements about each risk of bias item for each included study.

Allocation

Random sequence generation (selection bias)

Four studies had adequate sequence generation methods where participants were randomly assigned to the intervention or control group using a computerized random number generator or algorithm (Amoakoh 2019Berner 2006Heisler 2014Tian 2015). Four studies had unclear risk (Eaton 2011Gautham 2015Keane 2018Martinez 2018).

Allocation concealment (selection bias)

Three studies specified methods for allocation concealment where an independent study staff generated the randomization pattern (Amoakoh 2019Martinez 2018Tian 2015). Five studies did not describe methods for allocation concealment (Berner 2006Eaton 2011Gautham 2015Heisler 2014Keane 2018).

Blinding

In six studies, there was no blinding of participants to the intervention as the intervention required overt interaction with participants (i.e. healthcare providers in the intervention group were given a mobile device with a specific program, and the comparison group had usual care which did not involve the use of a mobile device) (Amoakoh 2019Eaton 2011Gautham 2015Heisler 2014Martinez 2018Tian 2015). Two studies made no clear mention about blinding of participants and personnel to the intervention (Berner 2006Keane 2018).

Two studies had adequate methods for blinding of outcome assessment where data assessors were blinded to the allocation throughout the study (Berner 2006Heisler 2014). Three studies had high risk of detection bias as adherence to treatment protocols by providers was assessed through direct observation of the study participants, or no clear protocol for outcome blinding was implemented (Eaton 2011Gautham 2015Martinez 2018). Three studies had unclear risk of detection bias (Amoakoh 2019Keane 2018Tian 2015).

Incomplete outcome data

Five studies reported low attrition, where the reasons for attrition seemed unrelated to the outcomes (Berner 2006), losses were balanced across groups (Eaton 2011Heisler 2014), or where the analysis included all the participants who were randomized to the intervention (Martinez 2018Tian 2015). Three studies had unclear reporting on attrition (Amoakoh 2019Gautham 2015Keane 2018).

Selective reporting

Three studies had adequate reporting of outcomes where all outcomes were reported (Berner 2006Eaton 2011Tian 2015), three studies had a high risk of reporting bias (Amoakoh 2019Gautham 2015Heisler 2014), and two studies had an unclear risk of selective reporting (Keane 2018Martinez 2018).

Other potential sources of bias

In Berner 2006 and Keane 2018, we were uncertain of additional sources of bias, as reporting was insufficient to assess this. In Gautham 2015, the authors acknowledged that there may have been contamination between the groups as the participants had opportunities to work together. One of the authors of the Gautham 2015 study declared a conflict of interest resulting from part ownership in a private company for commercialization of products used in the study. In Heisler 2014, there were some baseline imbalances between treatment and comparison groups. More participants in the treatment group had completed high school, were less likely to have difficulty with written healthcare information, and were more confident in filling out medical paperwork. However, the authors accounted for these imbalances in the analyses. In Amoakoh 2019, baseline differences in neonatal mortality risk factors were unaccounted for between the study arms. The authors also reported contamination across the intervention and comparison areas due to referral of women in comparison areas to intervention area health facilities.

Effects of interventions

See: Table 1

See: Table 1Table 3Table 4.

2. Summary of findings 2: mobile clinical decision‐support system compared to standard care in primary healthcare settings: details I.

Mobile clinical decision‐support system compared to standard care in primary healthcare settings: details I
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers
Setting: primary healthcare settings (China, Ghana, Guatemala, India, Kenya, USA)
Intervention: mobile clinical decision‐support system
Comparison: standard care or no intervention (providers using PDA with decision rules about a non‐intervention‐related health area; provider training and decision‐support tools on paper; paper‐based information booklet on management and follow‐up of people with diabetes; or usual care that did not involve any additional follow‐up)
Outcomes Anticipated absolute effects (95% CI) Relative effect(95% CI) № of participants(studies) Certainty of the evidence(GRADE) What happens?
Standard care Mobile clinical decision‐support system
Providers' adherence to recommended practices, guidelines, or protocols
Providers' adherence to recommended practices 1 study assessed a digital decision‐support tool for non‐steroidal anti‐inflammatory drug prescribing safety. The mean proportion of providers with unsafe prescriptions was 0.23 in the intervention group and 0.45 in the comparison group. The proportion of providers following recommended practice was 0.58 in the intervention group and 0.45 in the comparison group. 
1 study assessed a digital decision‐support tool for management of fevers, diarrhoeas and respiratory problems by rural providers. For female patients, mean protocol compliance was 63.34% in the intervention group and 69% in the comparison group. For male patients, compliance was 53.59% in the intervention group and 71.12% in the comparison group. 
185
(2 RCTs)
India, USA
(Berner 2006Gautham 2015)
⊕⊝⊝⊝
Very lowa,b We are uncertain of the effect of this approach on providers’ adherence to recommended clinical practice because the certainty of this evidence was very low.
(Both studies reported incomplete data.)
Time between presentation and appropriate management
Time between presentation and appropriate management No studies reported this outcome  
Patients' or clients' acceptability and satisfaction
Client satisfaction with clarity of medication information
(1–100 scale, where higher is better)
Mean satisfaction was 82.6 points on a 1–100 scale Mean satisfaction was 9.2 points higher (95% CI 0.97 higher to 17.43 higher) on a 1–100 scale 187
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowc,d This approach may improve the satisfaction with clarity of medication information among people with poorly controlled diabetes.
Client satisfaction with helpfulness of medication information
(1–100 scale, where higher is better)
Mean satisfaction was 87.6 points on a 1–100 scale Mean satisfaction was 11.3 higher (95% CI 3.28 higher to 19.32 higher) 187
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowc,d This approach may improve the satisfaction with helpfulness of medication information among people with poorly controlled diabetes.
Provider acceptability and satisfaction
Providers' acceptability/satisfaction No studies reported this outcome  
Resource use
Resource use No studies reported this outcome  
Unintended consequences
Unintended consequences No studies reported this outcome  
CI: confidence interval; LDL: low‐density lipoprotein; MD: mean difference; OR: odds ratio; RCT: randomized controlled trial; RR: risk ratio; SD: standard deviation.
GRADE Working Group grades of evidence
High certainty: we are very confident that the true effect lies close to that of the estimate of the effect. 
Moderate certainty: we are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. 
Low certainty: our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect. 
Very low certainty: we have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect.

aDowngraded one level due to risk of bias: unclear random sequence generation and allocation concealment. Blinding of participants was not possible given the intervention.
bDowngraded two levels due to very serious imprecision: standard errors or confidence intervals for the outcomes were not reported.
cDowngraded one level due to risk of bias: unclear allocation concealment and selective outcome reporting.
dDowngraded one level due to imprecision due to small sample size.

3. Summary of findings 2: mobile clinical decision‐support system compared to standard care in primary healthcare settings: details II.

Mobile clinical decision‐support system compared to standard care in primary healthcare settings: details II
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers
Setting: primary healthcare settings (China, Ghana, Guatemala, India, Kenya, USA)
Intervention: mobile clinical decision‐support system
Comparison: standard care or no intervention (providers using PDA with decision rules about a non‐intervention‐related health area; provider training and decision‐support tools on paper; paper‐based information booklet on management and follow‐up of people with diabetes; or usual care that did not involve any additional follow‐up)
Outcomes Anticipated absolute effects (95% CI) Relative effect(95% CI) № of participants(studies) Certainty of the evidence(GRADE) What happens
Standard care Mobile clinical decision‐support system
Patients' or clients' health behaviour
Antihyperglycaemic medication decisional conflict at 3 months of follow‐up Mean 72.3 (SD 13.2) Mean 70.9 (SD 13.7) MD 2.60 (−6.76 to 2.16) 176
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowa,b The intervention may make little or no difference to decisional conflict.c
Diabetes care self‐efficacy Mean 80 (SD 16.6) Mean 83.3 (SD 19.5) MD 3.30 (−0.95 to 7.55) 176
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowa,b The intervention may make little or no difference to diabetes care self‐efficacy.c
Adherence – high‐risk people taking aspirin in the last month at 1 year of follow‐up 22 per 1000 206 per 1000
(95% CI 134 to 317) RR 9.30
(6.05 to 14.28) 2086
(1 RCT)
India and China
(Tian 2015)
⊕⊕⊝⊝
Lowa,d This approach may increase the number of people with high cardiovascular disease risk taking their aspirin.
Adherence – self‐reported use of community healthcare workers prescribed antihypertensive medication for ≥ 25 days in the past month at 1 year of follow‐up 94 per 1000 362 per 1000
(95% CI 295 to 447) RR 3.86
(3.14 to 4.76) 2086
(1 RCT)
India and China
(Tian 2015)
⊕⊕⊕⊝
Moderated This approach probably increases the number of people taking their antihypertensive medication.
Adherence – medication adherence at 3 months of follow‐up
(1–100 scale, where higher is better)
Mean medication adherence was 90.5 points on a 1–100 scale Mean medication adherence was 2.3 points lower (95% CI 6.76 lower to 2.16 higher) on a 1–100 scale 176
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowa,d This approach may make little or no difference to medication adherence among people with poorly controlled diabetes.c
Current smoker at 1 year of follow‐up 363 per 1000 374 per 1000
(95% CI 334 to 421) RR 1.03
(0.92 to 1.16) 2086
(1 RCT)
India and China
(Tian 2015)
⊕⊕⊕⊝
Moderated This approach probably makes little or no difference to the number of smokers among people with high cardiovascular disease risk.
Patients'/clients' health status and well‐being
Diabetes distress at 3 months of follow‐up Mean 66.6 (SD 30.7) Mean 76.9 (SD 22.3) MD 15.7 (8.24 to 23.16) 176
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowa,b The intervention may have a small positive effect on diabetes distress.c
 
Number of women who had a caesarean section 27 per 374 38 per 425 RR 1.24 (0.77 to 1.99) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to abnormal progression of labour 36 per 374 51 per 425 RR 1.25 (0.83 to 1.87) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to a hypertensive disorder 4 per 374 15 per 425 RR 3.3 (1.1 to 9.86) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to haemorrhage 10 per 374 9 per 425 RR 0.79 (0.33 to 1.93) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to premature labour 6 per 374 5 per 425 RR 0.73 (0.23 to 2.38) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to fetal cardiac abnormality 4 per 374 4 per 425 RR 0.88 (0.22 to 3.49) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to suspected sepsis 5 per 374 4 per 425 RR 0.70 (0.19 to 2.60) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to respiratory compromise 2 per 374 5 per 425 RR 2.20 (0.43 to 11.27) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Number of women who had an emergency referral due to a premature newborn 0 per 374 4 per 425 RR 7.92 (0.43 to 146.67) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowe,f,m We are uncertain about the effects on this outcome because the certainty of this evidence was very low.
Haemoglobin a1c (HbA1c)
(controlled HbA1c is typically < 7.5 or 7 (depending on risk factors))
Mean HbA1c was 7.9% Mean HbA1c was 0.1% lower (95% CI 0.3 lower to 0.18 higher) 176
(1 RCT)
USA
(Heisler 2014)
⊕⊕⊝⊝
Lowa,b This approach may make little or no difference to HbA1c levels among people with poorly controlled diabetes.
Mean systolic blood pressure
(target systolic blood pressure is typically < 140 mmHg)
Mean systolic blood pressure was 152.3 mmHg Mean systolic blood pressure was 2.8 mmHg lower (95% CI 5.09 lower to 0.51 lower) 2086
(1 RCT)
India and China
(Tian 2015)
⊕⊕⊕⊝
Moderateb This approach probably makes little or no difference to the systolic blood pressure among people with high cardiovascular disease risk.
People reaching LDL‐cholesterol goal 74% 74%
  875
(1 RCT)
USA
(Eaton 2011)
⊕⊕⊝⊝
Lowg,h This approach may make little or no difference to the number of people with hyperlipidaemia reaching LDL‐cholesterol goals.
Maternal deaths 2.6 per 1000 4.6 per1000
(0 to 50)
RR 1.76 (0.16 to 19.33 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊝⊝⊝
Very lowi,j,m We are uncertain about the effects of this approach on maternal deaths because the certainty of this evidence was very low.
Hospital delivery at 12 months of follow‐up 77 per 1000 98 per 1000
(62 to 154)
RR 1.27 (0.81 to 2.00) 799
(1 RCT)
Guatemala
(Martinez 2018)
⊕⊕⊝⊝
Lowa,i,m This approach may increase the number of hospital deliveries, but the 95% CI included both a decrease and an increase in hospital deliveries.
Neonatal deaths 1 study reported no difference between the intervention and comparison groups in neonatal deaths (OR 6.25, 95% CI 0.76 to 51). 
1 study reported increased odds of neonatal deaths in the intervention group compared to the comparison group (OR 2.09, 95% CI 1.00 to 4.37)
66,630
(2 RCTs)
Guatemala (Martinez 2018), Ghana (Amoakoh 2019)
⊕⊝⊝⊝
Very lowk,l We are uncertain about the effects of this approach on neonatal deaths because the certainty of this evidence was very low. Results could not be pooled as only 1 of the 2 studies presented estimates adjusted for clustering.
CI: confidence interval; HbA1c: glycated haemoglobin; LDL: low‐density lipoprotein; MD: mean difference; PDA: personal digital assistant; RCT: randomized controlled trial; RR: risk ratio; SD: standard deviation.
GRADE Working Group grades of evidence
High certainty: we are very confident that the true effect lies close to that of the estimate of the effect.
Moderate certainty: we are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.
Low certainty: our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect.
Very low certainty: we have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect.

aDowngraded one level due to imprecision due to few events.
bDowngraded one level due to risk of bias: unclear allocation concealment and selective outcome reporting.
cOutcome measures scaled from 0 to 100 with higher numbers indicating more positive outcomes.
dDowngraded one level due to risk of bias due to lack of blinding of participants and unclear blinding of outcome assessment.
eDowngraded one level due to lack of blinding of participants and outcomes, and unclear random sequence generation and selective reporting.
fDowngraded one level due to imprecision: very few events. 
gDowngraded one level due to risk of bias: unclear random sequence generation and allocation concealment; participants, providers, and outcome assessors not blinded.
hDowngraded one level due to imprecision: unclear number of participants in either group for analysis, only proportions reported.
iDowngraded one level due to lack of blinding of participants, personnel, and outcomes data; and unclear sequence generation and reporting bias.
jDowngraded two levels due to very serious imprecision due to few events and wide confidence intervals that encompassed both a large benefit and a large harm associated with the intervention.
kDowngraded two levels due to very serious risk of bias: both studies had lack of blinding of participants, personnel, and outcomes data; and one study had unclear sequence generation and reporting bias.
lDowngraded one level due to spurious effects from confounding as one study reported potential differences between groups on baseline neonatal mortality risk factors.

mDowngraded one level due to indirectness: potential for contamination as the control group providers also received 5 months of intervention.

Mobile clinical decision‐support system compared to standard care in primary healthcare settings

Providers' adherence to recommended practices, guidelines, or protocols

We are uncertain of the effect of this intervention on providers' adherence to recommended practice (very low‐certainty evidence). 

Two studies reported this outcome, measured as proportion of providers with unsafe practices, and proportion of providers who were compliant.Because of variations in outcome measurement across the studies, as well as incomplete reporting of data, meta‐analyses of these results was not feasible.One study assessed effectiveness of providers' use of a digital decision‐support tool on NSAID prescribing safety (Berner 2006). It reported the mean proportion of providers with unsafe prescriptions at 0.23 (31 providers) for the intervention group and 0.45 (28 providers) for the comparison group. The proportion of providers following recommended practice was 0.58 (31 participants) for the intervention group and 0.45 (28 participants) for the comparison group. Another study assessed the use of a digital decision‐support tool for management of fevers, diarrhoeas, and respiratory problems by rural providers (Gautham 2015). For female patients, mean protocol compliance was 63.34% (8 providers, 38 female patients) for the intervention group and 69% (8 providers, 43 female patients), and for male patients, mean protocol compliance was 53.59% (8 providers, 27 male patients) for the intervention group and 71.12% (8 providers, 18 male patients).

Time between presentation and appropriate management

No studies reported time between presentation and appropriate management.

Patients' or clients' health behaviour

The intervention probably makes little or no difference to the numbers of smokers among people with high cardiovascular disease risk (moderate‐certainty evidence), probably increases the number of people taking their antihypertensive medication (moderate‐certainty evidence), and may increase the number of people with high cardiovascular disease risk taking prescribed aspirin. It may make little or no difference to medication adherence among people with poorly controlled diabetes (low‐certainty evidence).

Use of mobile‐based clinical decision‐support tools probably leads to little or no change in proportion of clients who reporting being current smokers at one‐year follow‐up (RR 1.03, 95% CI 0.92 to 1.16; moderate‐certainty evidence) (Tian 2015). At one‐year follow‐up, use of the tools by community health workers for counselling may increase the number of high‐risk individuals taking preventive aspirin in the last month (RR 9.30, 95% CI 6.05 to 14.28; low‐certainty evidence) and probably increases the number of people using of antihypertensive medication for 20 days or more in the past month (RR 3.86, 95% CI 3.14 to 4.76; moderate‐certainty evidence) (Tian 2015). Contrary to these findings, the effect of clinical decision‐support tools by providers may makelittle or no difference to medication adherence among people with poorly controlled diabetes at three months of follow‐up. The MD in medication adherence was 2.3 points lower in the intervention group (95% CI −6.76 to 2.16; low‐certainty evidence) (Heisler 2014). These differences in the findings of the studies may be explained by differences in theirsettings: in Tian 2015, lay health workers delivered the intervention in a community‐based setting in India and China and, in Heisler 2014, trained physicians delivered the intervention in a primary care facility in the USA.

Patients' or clients' health status and well‐being

The use of mobile‐based clinical decision‐support tools probably makes little or no difference to systolic blood pressure level among people with high cardiovascular disease risk or to the number of women giving birth in a hospital (moderate‐certainty evidence). The intervention may make little or no difference to HbA1c levels among people with poorly controlled diabetes, or to  the number of people with hyperlipidaemia reaching LDL‐cholesterol goals (low‐certainty evidence). . We are uncertain of the effect on maternal deaths and neonatal deaths, and a number of other maternal health outcomes (very low‐certainty evidence).

Across the studies that reported the effect of mobile‐based clinical decision‐support tools by providers on health outcomes, these may make little or no difference to HbA1c, mean systolic blood pressure, hospital delivery over the 12‐month study period, and maternal death (HbA1c: −0.1%, 95% CI −0.3 to 0.18; low‐certainty evidence; Heisler 2014; mean systolic blood pressure: −2.8 mmHg, 95% CI −5.09 to 0.51; moderate‐certainty evidence; Tian 2015; hospital delivery over the 12‐month study period: RR 1.27, 95% CI 0.81 to 2.00; low‐certainty evidence; Eaton 2011; maternal death: RR 1.76, 95% CI 0.16 to 19.33; very low‐certainty evidence; Martinez 2018). Two studies reported the effects of mobile‐based clinical decision‐support tools on neonatal deaths but we are uncertain about these effects because the certainty of this evidence was very low (Amoakoh 2019Martinez 2018). Martinez 2018 reported that there may be no difference in neonatal deaths between groups (OR 6.25, 95% CI 0.76 to 51.00), whereas Amoakoh 2019 reported that there may be an increase in neonatal deaths in the intervention group (OR 2.09, 95% CI 1.00 to 4.37) (very low‐certainty evidence). One study reported that there may be little or difference in the proportion of patients who achieved LDL‐ cholesterol goals after one year of intervention (low‐certainty evidence) (Eaton 2011).

The studies reported several other patient/client health outcomes (see additional summary of findings table in Table 4). Heisler 2014 reported that the intervention may have a small positive effect on diabetes distress at three months of follow‐up (MD 15.7, 95% CI 8.24 to 23.16;) ( low‐certainty evidence). Martinez 2018 reported the effect of the intervention on the number of women who had a caesarean section, and who had an emergency referral due to the following conditions: abnormal progression of labour, hypertensive disorders, haemorrhage, premature labour, fetal cardiac abnormality, suspected neonatal sepsis, respiratory compromise, and premature newborn.

Patient or client acceptability and satisfaction with the intervention

The intervention may improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (low‐certainty evidence). Client satisfaction with health information provided by the healthcare provider was measured on a scale of 1 to 100. Heisler 2014 reported that there may be a 9.2‐point increase (95% CI 0.97 to 17.43) in client satisfaction with clarity of medication information, and an 11.3‐point increase (95% CI 3.28 to 19.32) in client satisfaction with helpfulness of medication information.

Provider acceptability and satisfaction with the intervention

No studies reported providers' acceptability and satisfaction with the intervention. Tian 2015 included this outcome but reported provider acceptability of the intervention for the intervention group only.

Resource use

No studies reported resource use (e.g. human resources and time, training, supplies, and equipment).

Unintended consequences

No studies reported unintended consequences.

Discussion

Summary of main results

Eight RCTs met our inclusion criteria. Three trials were conducted in the USA. The remaining trials were conducted in India (one trial), India and China (one), Guatemala (one), Ghana (one), and Kenya (one). We are uncertain of the effect of this intervention on providers’ adherence to recommended practice due to the very low certainty of evidence (2 studies, 185 participants). The effect of the intervention on patients' and clients' health behaviour was mixed. It probably makes little or no difference to some types of behaviour but probably increases other types of desired behaviour (2 studies, 2262 participants) There is large heterogeneity in the evidence for similar types of behaviour. For example, the evidence suggests improvements in 'adherence to medications' for certain conditions, and not for other conditions.

The effect of the intervention on patients'/clients' health status and well‐being was also mixed (5 studies, 69,767 participants). It probably makes little or no difference to some types of health outcomes, but we are uncertain about other health outcomes, including maternal and neonatal deaths, due to very low‐certainty evidence. The intervention may improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (1 study, 187 participants). Most of this evidence comes from single studies, some of which had few participants. No studies reported the time between presentation of an illness and appropriate management, provider acceptability or satisfaction, resource use, or unintended consequences.

Overall completeness and applicability of evidence

None of the included studies evaluated the impact of the intervention on the time between presentation of an illness and appropriate management, provider acceptability/satisfaction, resource use, or unintended consequences. For outcomes that were reported, the studies were often small and few in number.

The interventions in the included studies were heterogeneous in nature. The interventions were implemented across HICs and low‐income countries; in facility‐ and community‐based contexts; and by providers with varying degrees of training and experience including community health workers, internal medicine residents, and primary care physicians. Additionally, across the eight studies, the CDSS interventions had a range of functionalities applied across specific health areas including antenatal and postnatal care counselling, cardiovascular risk assessment and treatment, IMCI/IMAI and adult illnesses, and counselling and support for people with diabetes. None of the studies looked at the use of CDSS across a range of health care areas that healthcare personnel, especially in LMICs may typically deal with. For example,  in most primary health care settings, the same health care personnel may deal with a diverse range of healthcare issues including chronic disease and infectious disease management, and caring for pregnant women and children. The CDSS interventions evaluated by the included studies focused on a singular healthcare area‐ this would not capture the challenges that are faced at a systemic level where healthcare workers are required to juggle services across a range of priorities. Should healthcare workers have CDSS tools across all priority health care areas? If so, what are the unique training and implementation challenges that might be faced when services are digitally integrated? What would be the effectiveness of an integrated intervention? The review does not provide insights on these critical questions. Further research needs to take into account the wider health care system context in order to identify how CDSS should be designed in a way such that they can be responsive to the priority population needs.

The contextual range of these studies reflects the widespread use of these tools and suggests that the results may be applicable across a wide range of settings. However, the findings were difficult to generalize to a specific practice and the effectiveness of CDSS may vary based on the type of healthcare provider (lay health worker versus professional provider), baseline knowledge, health condition, and setting. There were also variations in control groups across the studies, which made us uncertain about how much variation in the outcomes was due to the intervention versus other factors. None of the studies used quality of care framework. Rather, studies focused on specific outcomes that, albeit informative on the effects of the intervention, failed to portray a reasonably complete picture on the quality of care. For example, it is remarkable that no studies reported on health workers satisfaction or unintended consequences.  This review cannot provide evidence on what would the intervention effects be across a more comprehensive set of quality of care dimensions; hence caution has to be exerted in interpreting the beneficial findings reported in this review.

Quality of the evidence

The certainty of the evidence was very low for the outcome providers' adherence to recommended protocols due to risk of bias and concerns about serious imprecision. The certainty of evidence for patients'/clients' outcomes ranged from low to moderate owing to imprecision due to few events and risk of bias. The certainty of the evidence for patients'/clients' health status ranged from moderate to very low. The results on mean systolic blood pressure were downgraded due concerns about risk of bias. Other outcomes were downgraded due to concerns about risk of bias, as well as serious imprecision resulting from very few events and lack of accounting for clustering of participants.

Potential biases in the review process

We utilized a comprehensive search strategy, inclusive of certain databases with unpublished studies, independent assessment of study eligibility and risk of bias, and independent data extraction. The focus of this review was on CDSS that were primarily accessed by and optimized for mobile usage. One potential area of bias is in the assessment of whether the intervention was optimized for mobile usage, especially where reporting on the details of the intervention was inadequate. Furthermore, it is conceivable that the authors have mischaracterized the components and functionalities of the intervention, especially in the case of complex interventions. Evidence for all outcomes came from single studies and it was not possible to explore the impact of all the sources of heterogeneity.

Agreements and disagreements with other studies or reviews

We did not identify any reviews that address the same objectives as this review. Reviews that have looked at the effects of stationary CDSS in clinical settings found that CDSS was effective in improving clinical performance, preventive care, and provider performance (Bright 2012Hunt 1998Jaspers 2011Kawamoto 2005Varghese 2018). Given that most of the existing reviews focus largely on computerized stationary CDSS in hospital‐based and HIC settings, the implementation considerations and subsequent effects of the interventions are different from the studies included in our review. Like this review, other reviews were limited by methodological limitations, heterogeneous interventions and comparison groups, and multiple outcomes. 

Reviews on the effectiveness of clinical decision‐support tools have different objectives than this review. . Most reviews focused primarily on computerized CDSS, and not on the use of mobile‐phone based CDSS (Bright 2012Caballero‐Ruiz 2017Hunt 1998Jaspers 2011Kaplan 2001Kawamoto 2005Kilsdonk 2017Syrowatka 2016). Other reviews may not have specifically focused on CDSS solely and included all digital interventions (Adepoju 2017Brenner 2016Carter 2019Mishra 2019). Several reviews were published in the 1990s or 2000s (Hunt 1998Kaplan 2001Kawamoto 2005), and were primarily focused on the use of CDSS in hospital‐based settings (Bright 2012Hunt 1998Jaspers 2011Kaplan 2001Martínez‐Pérez 2014Sutton 2020Varghese 2018) and high‐income settings (Dreesens 2019). Other reviews focused on the use of CDSS among community health workers in LMIC settings only (Agarwal 2015Mishra 2019). A few reviews have focused primarily on specific outcomes such as feasibility and acceptability of the CDSS (Jaspers 2011Kaplan 2001Kawamoto 2005Kilsdonk 2017). Reviews also focused on specific health conditions such as cancer (Baptista 2018Mazo 2020Tong 2021), maternal and child health care (Caballero‐Ruiz 2017Carter 2019), non‐communicable diseases (Mishra 2019), and cardiovascular disease (Njie 2015). Given the different objectives as well as inclusion/exclusion criteria of these reviews, their results cannot be directly compared to the results of our review. 

Authors' conclusions

Implications for practice.

Recent analysis suggests that improved health service coverage without a minimum level of quality significantly limits effectiveness (Kruk 2018). While clinical decision‐support system (CDSS) tools may have potential to influence the quality of care provision, the study provides limited evidence about the efficacy of CDSS on health outcomes. At the same time, the review does not provide any suggestion of deleterious effects due to the use of CDSS.

One linked Cochrane Review has synthesized qualitative research on health workers' perceptions and experiences of using mHealth technologies to deliver primary health services (Odendaal 2020). The following implications for practice have been taken from that review.

Health systems questions
  • Will health workers be part of the planning, implementation, and evaluation processes of mobile health programmes? Will their views be sought, and their perspectives taken at each stage of the programme?

  • To what extent is political buy‐in from health ministries required, and achieved, for the successful implementation of the mobile health programme?

  • Has a proper assessment been made on whether health workers' use of mobile devices is adding to or alleviating their workload? How will the extra workload that may occur, be accommodated for?

Technical and infrastructural questions
  • Does your setting have the necessary infrastructural and technological capacity to support the level of sophistication intended by the intervention? For example, is there sufficient electricity supply and electricity coverage, network capacity, technical support, and vendors to purchase phone credit or data for the level of intervention that you intend to implement? Have you considered how these might vary by region?

  • Are the devices being used in the intervention sufficiently sophisticated for the level of intervention being planned, and are these devices replaceable or reparable within your setting? Have you considered who will repair them, and who will cover the costs?

  • When planning mHealth programmes, has the number of staff who have access to mobile devices been taken into account?

  • Has adequate provision been made for health workers to have enough phone credit and data, without having to use their own resources?

  • Is there a strategy to integrate the mobile health platform within existing electronic health information systems?

Health worker training and skills
  • Has the programme management budgeted for adequate training of initial staff, refresher training, and in‐service training for new staff members?

  • What is the level of digital literacy among those health workers who will implement the intervention, as well as managers and supervisors who will support them? What further interventions are needed to ensure adequate skill level is present at the beginning of the intervention and maintained over the course of the intervention?

  • Has the programme management identified 'champions' among the workers whom they can call upon to assist those struggling with the devices?

  • When the device allows the health worker to screen and diagnose clients, are they clinically equipped to respond appropriately to the results of the screening and diagnosing? Are they able to explain the results to the patient?

  • Is there a system in place to allow staff who dislike, or who lack digital literacy to use mobile devices, to continue with standard practice, such as a paper‐based system for recording work?

Implications for research.

More, well‐designed research needs to be undertaken to understand the effectiveness of CDSS in improving quality of health services. This review identified some limitations that future trialists should consider. First, larger well‐designed trials of effect of mobile clinical decision‐support tools on health outcomes are needed. Ideally, these trials should be conducted beyond the pilot stage, after the intervention is 'bedded in' and refined. These trials should also measure outcomes that have not been measured to date, namely unintentional consequences, and resource use/cost‐effectiveness of the intervention. Second, CDSS tool may have a range of functionality (e.g. checklists, targeted two‐way communication with a provider though a text message or an application). Reporting on the details of the intervention can help delineate the specific functions driving the health impact. Third, trialists should also consider possible adverse effects (e.g. could use of a mobile device detract from delivery of care?) and how such an intervention might affect inequities by gender or access to mobile networks. Fourth, in several contexts, randomized trials may not be feasible or appropriate given the limitations in implementing such interventions. Alternate study designs, such as longitudinal or cohort studies should be explored. For CDSS interventions that are known to be efficacious based on results from well‐designed trials, it might be necessary to conduct implementation science studies that assess adaptations of the intervention to a given context. These will help provide insights on cost‐benefit, cost‐utility, and process improvement. Fifth, it is important the future studies consider the effects of CDSS interventions that are integrated with the broader health system, and address a range of health conditions and processes that health care personnel routinely deal with in their clinical practice. Lastly, CDSS interventions need to be evaluated using a quality of care framework. None of the studies reported on patient satisfaction, which is core to adopted and sustainability of such interventions. 

What's new

Date Event Description
28 July 2021 Amended A comment was removed from the text of the review. 

History

Protocol first published: Issue 2, 2018
Review first published: Issue 7, 2021

Acknowledgements

We acknowledge the help and support of National Institute for Health Research (NIHR), via Cochrane Infrastructure funding to the Effective Practice and Organisation of Care (EPOC) Group. The views and opinions expressed herein are those of the review authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, National Health Service, or the Department of Health.

We are grateful to the Guideline Development Group of the Digital Health Guidelines for their constructive feedback in formulating the guiding questions for this systematic review. We thank Marit Johansen and John Eyers for executing the search strategies for the review. Cochrane Response, an evidence services unit operated by Cochrane, provided screening and data extraction services for this review. 

We thank the following peer reviewers for their review and feedback: Xavier Bosch‐Capblanch (contact editor), Jhon Camacho, Stella Maria O'Brien, Andrew Hutchings, Caroline Free, Paul Miller, Daniela Gonçalves Bradley, Jennifer Hilgart, and Chris Cooper (EPOC managing editor of the review).

Appendices

Appendix 1. Search strategies

Cochrane Central Register of Controlled Trials (CENTRAL) Issue 9, 2020, Cochrane Library

ID Search Hits
#1 MeSH descriptor: [Cell Phone] this term only 674
#2 MeSH descriptor: [Smartphone] this term only 384
#3 MeSH descriptor: [MP3‐Player] this term only 21
#4 MeSH descriptor: [Computers, Handheld] this term only 270
#5 ((cell* or mobile*) near/1 (phone* or telephone* or technolog* or device*)):ti,ab,kw 4321
#6 (handheld or hand‐held):ti,ab,kw 2296
#7 (smartphone* or smart‐phone* or cellphone* or mobiles):ti,ab,kw 4090
#8 ((personal near/1 digital) or (PDA near/3 (device* or assistant*)) or MP3 player* or MP4 player*):ti,ab,kw 320
#9 (samsung or nokia):ti,ab,kw 161
#10 (windows near/3 (mobile* or phone*)):ti,ab,kw 6
#11 android:ti,ab,kw 633
#12 (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*):ti,ab,kw 1103
#13 (tablet* near/3 (device* or computer*)):ti,ab,kw 854
#14 MeSH descriptor: [Telemedicine] this term only 2079
#15 MeSH descriptor: [Webcasts as Topic] this term only 24
#16 MeSH descriptor: [Text Messaging] this term only 848
#17 MeSH descriptor: [Telenursing] this term only 31
#18 (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health):ti,ab,kw 4477
#19 (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*):ti,ab,kw 6291
#20 (webcast* or web‐cast*):ti,ab,kw 40
#21 (((text* or short or voice or multimedia or multi‐media or electronic or instant) near/1 messag*) or instant messenger) .ti,ab,kw 65
#22 (texting or texted or texter* or ((sms or mms) near (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP):ti,ab,kw 2850
#23 (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*):ti,ab,kw 1069
#24 MeSH descriptor: [Mobile Applications] this term only 628
#25 "mobile app*":ti,ab,kw 658
#26 MeSH descriptor: [Reminder Systems] this term only 933
#27 (remind* near/3 (text* or system* or messag*)):ti,ab,kw 2143
#28 MeSH descriptor: [Medical Informatics] this term only 74
#29 MeSH descriptor: [Medical Informatics Applications] this term only 23
#30 MeSH descriptor: [Nursing Informatics] this term only 7
#31 MeSH descriptor: [Public Health Informatics] this term only 1
#32 ((medical or clinical or health or healthcare or nurs*) near/3 informatics):ti,ab,kw 325
#33 MeSH descriptor: [Computer‐Assisted Instruction] this term only 1214
#34 ((interactive or computer‐assisted) near/1 (tutor* or technolog* or learn* or instruct* or software or communication)):ti,ab,kw 1605
#35 {or #1‐#34} 26207
#36 MeSH descriptor: [Decision Support Systems, Clinical] this term only 370
#37 MeSH descriptor: [Decision Making] this term only 2085
#38 MeSH descriptor: [Decision Support Techniques] this term only 800
#39 MeSH descriptor: [Diagnosis, Computer‐Assisted] this term only 682
#40 MeSH descriptor: [Decision Support Systems, Management] this term only 8
#41 MeSH descriptor: [Expert Systems] this term only 58
#42 MeSH descriptor: [Point‐of‐Care Systems] this term only 423
#43 MeSH descriptor: [Guideline Adherence] this term only 1044
#44 MeSH descriptor: [Clinical Protocols] this term only 4725
#45 MeSH descriptor: [Checklist] this term only 265
#46 MeSH descriptor: [Therapy, Computer‐Assisted] this term only 1319
#47 MeSH descriptor: [Drug Therapy, Computer‐Assisted] this term only 148
#48 MeSH descriptor: [Electronic Prescribing] explode all trees 22
#49 MeSH descriptor: [Clinical Laboratory Information Systems] this term only 8
#50 MeSH descriptor: [Clinical Pharmacy Information Systems] this term only 21
#51 ((decision* near/3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*"):ti,ab,kw 20360
#52 ((therap* or prescrib* or prescript* or diagnos*) near/2 (computer* or digital or electronic)):ti,ab,kw 3968
#53 ((guideline* or protocol*) near/4 (adher* or comply or complian* or observ*) or checklist*):ti,ab,kw 14974
#54 {or #36‐#53} 42794
#55 #35 and #54 3065
#56 #55 with Cochrane Library publication date Between Aug 2019 and Oct 2020, in Trials 670
#57 #55 with Publication Year from 2019 to 2020, in Trials 605
#58 #56 or #57 787

 

MEDLINE and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily from 1946, Ovid (searched 9 October 2020)

# Searches Results
1 Decision Support Systems, Clinical/ 8089
2 Decision Making/ 96106
3 Decision Support Techniques/ 20573
4 Diagnosis, Computer‐assisted/ 22696
5 Decision Support Systems, Management/ 958
6 Expert Systems/ 3410
7 Point‐of‐Care Systems/ 13132
8 Guideline Adherence/ or Clinical Protocols/ or Checklist/ 66164
9 Therapy, Computer‐Assisted/ 6813
10 "Drug Therapy, Computer‐Assisted"/ 1672
11 Electronic Prescribing/ 1037
12 Clinical Laboratory Information Systems/ or Clinical Pharmacy Information Systems/ 3221
13 ((decision* adj3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*").ti,ab,kw. 210661
14 ((therap* or prescrib* or prescript* or diagnos*) adj2 (computer* or digital or electronic)).ti,ab,kw. 9621
15 (((guideline* or protocol*) adj4 (adher* or comply or complian* or observ*)) or checklist*).ti,ab,kw. 68556
16 or/1‐15 448888
17 Cell Phones/ 8631
18 Smartphone/ 4724
19 MP3‐Player/ 185
20 Computers, Handheld/ 3641
21 ((cell* or mobile*) adj1 (phone* or telephone* or technolog* or device*)).ti,ab,kw. 21876
22 (handheld or hand‐held).ti,ab,kw. 13682
23 (smartphone* or smart‐phone* or cellphone* or mobiles).ti,ab,kw. 15681
24 ((personal adj1 digital) or (PDA adj3 (device* or assistant*)) or MP3 player* or MP4 player*).ti,ab,kw. 1453
25 (samsung or nokia).ti,ab,kw. 1390
26 (windows adj3 (mobile* or phone*)).ti,ab,kw. 59
27 android.ti,ab,kw. 2932
28 (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*).ti,ab,kw. 3232
29 (tablet* adj3 (device* or computer*)).ti,ab,kw. 1902
30 Telemedicine/ 24150
31 Webcasts as topic/ 348
32 Text Messaging/ 3016
33 Telenursing/ 220
34 (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health).ti,ab,kw. 35112
35 (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*).ti,ab,kw. 22715
36 (webcast* or web‐cast*).ti,ab,kw. 273
37 (((text* or short or voice or multimedia or multi‐media or electronic or instant) adj1 messag*) or instant messenger).ti,ab,kw. 6317
38 (texting or texted or texter* or ((sms or mms) adj (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP).ti,ab,kw. 3803
39 (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*).ti,ab,kw. 9991
40 Mobile Applications/ 6339
41 "mobile app*".ti,ab,kw. 6135
42 Reminder Systems/ 3500
43 (remind* adj3 (text* or system* or messag*)).ti,ab,kw. 2014
44 Medical informatics/ or Medical informatics applications/ 14349
45 Nursing informatics/ or Public health informatics/ 2679
46 ((medical or clinical or health or healthcare or nurs*) adj3 informatics).ti,ab,kw. 6089
47 Computer‐Assisted Instruction/ 11906
48 ((interactive or computer‐assisted) adj1 (tutor* or technolog* or learn* or instruct* or software or communication)).ti,ab,kw. 2733
49 or/17‐48 165200
50 randomized controlled trial.pt. 514593
51 controlled clinical trial.pt. 93873
52 randomized.ab. 522812
53 placebo.ab. 219198
54 drug therapy.fs. 2240479
55 randomly.ab. 366157
56 trial.ab. 551010
57 groups.ab. 2221881
58 or/50‐57 4969223
59 exp animals/ not humans.sh. 4741227
60 58 not 59 4336186
61 16 and 49 and 60 3807
62 (201809* or 201810* or 201811* or 201812* or 2019*).dt,dp,ed,ep,yr. 2829843
63 61 and 62 881
64 (201908* or 201909* or 201910* or 201911* or 201912* or 2020*).dt,dp,ed,ep,yr. 2597293
65 61 and 64 795

 

Embase from 1974, Ovid (searched 9 October 2020)

# Searches Results
1 decision support system/ or clinical decision support system/ 25781
2 decision making/ or clinical decision making/ or medical decision making/ or shared decision making/ 369459
3 computer assisted diagnosis/ or computer assisted drug therapy/ or computer assisted therapy/ 44301
4 expert system/ 5417
5 "point of care system"/ 2227
6 protocol compliance/ or checklist/ 39173
7 clinical protocol/ 100201
8 computer assisted diagnosis/ 39288
9 computer assisted drug therapy/ 914
10 computer assisted therapy/ 4644
11 electronic prescribing/ 3077
12 laboratory information system/ 1067
13 ((decision* adj3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*").ti,ab,kw. 281691
14 ((therap* or prescrib* or prescript* or diagnos*) adj2 (computer* or digital or electronic)).ti,ab,kw. 14412
15 (((guideline* or protocol*) adj4 (adher* or comply or complian* or observ*)) or checklist*).ti,ab,kw. 94744
16 or/1‐15 767076
17 mobile phone/ or smartphone/ 29884
18 mp3 player/ 201
19 ((cell* or mobile*) adj1 (phone* or telephone* or technolog* or device*)).ti,ab,kw. 25726
20 (handheld or hand‐held).ti,ab,kw. 17743
21 (smartphone* or smart‐phone* or cellphone* or mobiles).ti,ab,kw. 18968
22 ((personal adj1 digital) or (PDA adj3 (device* or assistant*)) or MP3 player* or MP4 player*).ti,ab,kw. 1896
23 (samsung or nokia).ti,ab,kw. 2356
24 (windows adj3 (mobile* or phone*)).ti,ab,kw. 80
25 android.ti,ab,kw. 4192
26 (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*).ti,ab,kw. 5455
27 (tablet* adj3 (device* or computer*)).ti,ab,kw. 2684
28 telemedicine/ or telecardiology/ or teleconsultation/ or teledermatology/ or telediagnosis/ or telemonitoring/ or telepathology/ or telepsychiatry/ or teleradiotherapy/ or telesurgery/ or teletherapy/ 41005
29 webcast/ 358
30 text messaging/ 5206
31 telenursing/ 283
32 (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health).ti,ab,kw. 42203
33 (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*).ti,ab,kw. 28014
34 (webcast* or web‐cast*).ti,ab,kw. 439
35 (((text* or short or voice or multimedia or multi‐media or electronic or instant) adj1 messag*) or instant messenger).ti,ab,kw. 7599
36 (texting or texted or texter* or ((sms or mms) adj (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP).ti,ab,kw. 5157
37 (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*).ti,ab,kw. 12598
38 mobile application/ 11922
39 "mobile app*".ti,ab,kw. 6398
40 reminder system/ 2660
41 (remind* adj3 (text* or system* or messag*)).ti,ab,kw. 2768
42 medical informatics/ 20670
43 nursing informatics/ 1559
44 ((medical or clinical or health or healthcare or nurs*) adj3 informatics).ti,ab,kw. 9375
45 teaching/ 91921
46 ((interactive or computer‐assisted) adj1 (tutor* or technolog* or learn* or instruct* or software or communication)).ti,ab,kw. 3797
47 or/17‐46 292926
48 crossover procedure/ 64524
49 double blind procedure/ 176364
50 randomized controlled trial/ 622476
51 single‐blind procedure/ 40351
52 random$.tw. 1580230
53 factorial$.tw. 39052
54 (crossover$ or cross over$ or cross‐over$).tw. 108663
55 placebo$.tw. 313560
56 (doubl$ adj blind$).tw. 212917
57 (singl$ adj blind$).tw. 25656
58 assign$.tw. 403779
59 allocat$.tw. 157754
60 volunteer$.tw. 261957
61 or/48‐60 2386666
62 16 and 47 and 61 4242
63 limit 62 to embase 1994
64 (201809* or 201810* or 201811* or 201812* or 2019*).dd. 787410
65 ("2018" or "2019").yr. 3234006
66 64 or 65 3400778
67 63 and 66 477
68 (201908* or 201909* or 201910* or 201911* or 201912* or 2020*).dd. 917313
69 ("2019" or "2020").yr. 2969423
70 68 or 69 3101761
71 63 and 70 497
72 ("201932" or "201933" or "201934" or "201935" or "201936" or "201937" or "201938" or "201939" or 20194* or 20195* or 2020*).em. 2405021
73 63 and 72 292
74 71 or 73 528

 

POPLINE, K4Health (searched 5August 2019)

All Fields:

((decision* AND (make OR makes OR making OR made OR support* OR algorithm* OR aid OR aids OR app OR apps OR application* OR technique*)) OR "expert system*" OR checklist* OR job‐aid* OR "job aid*")

OR

Keyword:

DECISION MAKING

AND

All Fields:

((cell OR cellular OR mobile) AND (phone OR phones OR telephone OR telephones OR technology OR technologies OR device OR devices)) OR smartphone OR smartphones OR smart‐phone OR smart‐phones OR cellphone OR cellphones OR mobiles OR mhealth OR m‐health OR "mobile health" OR ehealth OR e‐health OR "electronic health" OR telemedicine OR tele‐medicine OR telehealth OR tele‐health OR telecare OR tele‐care OR telenursing OR tele‐nursing OR telepsychiatry OR tele‐psychiatry OR telemonitor OR telemonitoring OR tele‐monitor OR tele‐monitoring OR teleconsult OR teleconsulting OR tele‐consult OR tele‐consulting OR telecounsel OR telecounseling OR tele‐counsel OR tele‐counseling OR telecoach OR telecoaching OR tele‐coach OR tele‐coaching OR videoconference OR videoconferences OR videoconferencing OR video‐conference OR video‐conferences OR video‐conferencing OR webcast OR webcasts OR  webcasting OR web‐cast OR web‐casts OR web‐casting OR ((text OR texts OR texting OR short OR voice OR multimedia OR multi‐media OR electronic OR instant) AND (message OR messages OR messaging)) OR "instant messenger" OR texting OR texted OR texter OR texters OR ((sms OR mms) AND (service OR services OR message OR messages OR messaging)) OR "interactive voice response" OR "interactive voice responses" OR ivr OR "voice call" OR "voice calls" OR callback OR "voice over internet" OR voip OR "mobile app" OR "mobile apps" OR "mobile application" OR "mobile applications" OR "social media" OR ((medical OR clinical OR health OR healthcare OR nurse OR nurses OR nursing) AND informatics)

OR

Keyword:

TEXT MESSAGING OR MOBILE DEVICES OR INFORMATION COMMUNICATION TECHNOLOGY OR CELLULAR PHONE

AND

Keyword:

QUANTITATIVE RESEARCH OR RESEARCH METHODOLOGY OR CLINICAL TRIALS OR CONTROL GROUPS

OR

All Fields:

(randomised OR randomized OR "randomly allocated" OR "random allocation" OR "controlled trial" OR "control group" OR "control groups" OR trial)

Global Index Medicus, WHO (searched 9 October 2020)

Search fields used: Title, abstract, subject

("cell phones" OR smartphone OR mp3‐player OR "Computers, Handheld" OR telemedicine OR videoconferencing OR "Text Messaging" OR telenursing OR "Mobile Applications" OR "Reminder Systems" OR "Electronic Mail" OR "Medical Informatics" OR "Nursing Informatics" OR "Public Health Informatics" OR multimedia OR hypermedia OR blogging OR "cell phone" OR "cellular phone" OR "cellular phones" OR "mobile phone" OR "mobile phones" OR "mobile devices" OR "mobile devices" OR smartphones OR smart‐phone OR smart‐phones OR cellphone OR cellphones) AND (app OR apps OR application* OR "decision technique" OR "decision techniques" OR checklist* OR "expert system" OR "expert systems" OR job‐aid OR job‐aids OR "job aid" OR "job‐aids" OR "decision support system" OR "decision support systems" OR "decision making" OR "decision support technique" OR "decision support techniques" OR "expert system" OR "expert systems" OR "point‐of‐care system" OR "point‐of‐care systems" OR "guideline adherence" OR "clinical protocol" OR "clinical protocols" OR checklist*) AND ("Controlled Clinical Trials, Randomized" OR "Controlled Clinical Trials as Topic" OR "Controlled Clinical Trial" OR "Clinical Trial" OR randomised OR randomized OR "randomly allocated" OR "random allocation" OR "controlled trial" OR "control group" OR "control groups" OR trial)

International Clinical Trials Registry Platform (ICTRP), WHO (searched 5 August 2019)

Two separate strategies. Used advanced search, with recruitment status: All

Strategy 1:

Title: decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or job aid

AND

Intervention: mobile device OR mobiles OR smartphone OR phone OR cellphone

Strategy 2:

Title: mobile device OR mobiles OR smartphone OR phone OR cellphone

AND

Intervention: decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or job aid

ClinicalTrials.gov, National Institutes of Health (searched 9 October 2020)

Other Terms: (decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or "job aid") AND ("mobile device" OR "mobile devices" OR mobiles OR smartphone OR smartphones OR "smart phone")

With time‐limit: first posted from 8 May 2019 to 9 October 2020

Data and analyses

Comparison 1. Mobile clinical decision support compared to standard care for patients.

Outcome or subgroup title No. of studies No. of participants Statistical method Effect size
1.1 Providers' adherence to recommended practices, guidelines, or protocols 2   Other data No numeric data
1.2 Patients' health status and well‐being (dichotomous outcomes) 2   Risk Ratio (M‐H, Random, 95% CI) Subtotals only
1.2.1 Emergency referrals at 7 months of follow‐up 1 463 Risk Ratio (M‐H, Random, 95% CI) 1.49 [1.09, 2.04]
1.2.2 High‐risk people taking aspirin in the last month at 1 year of follow‐up 1 2086 Risk Ratio (M‐H, Random, 95% CI) 9.30 [6.05, 14.28]
1.2.3 Self‐reported use of community healthcare workers‐prescribed antihypertensive medication for ≥ 25 days in the past month at 1 year of follow‐up 1 2086 Risk Ratio (M‐H, Random, 95% CI) 3.86 [3.14, 4.76]
1.2.4 Successful referrals at 7months of follow‐up 1 123 Risk Ratio (M‐H, Random, 95% CI) 0.97 [0.87, 1.08]
1.2.5 Hospital delivery at 12 months of follow‐up 1 799 Risk Ratio (M‐H, Random, 95% CI) 1.27 [0.81, 2.00]
1.2.6 Number of women who had a caesarean section 1 799 Risk Ratio (M‐H, Random, 95% CI) 1.24 [0.77, 1.99]
1.2.7 Number of women who had emergency referral due to abnormal progression of labour 1 799 Risk Ratio (M‐H, Random, 95% CI) 1.25 [0.83, 1.87]
1.2.8 Number of women who had emergency referral due to a hypertensive disorder 1 799 Risk Ratio (M‐H, Random, 95% CI) 3.30 [1.10, 9.86]
1.2.9 Number of women who had an emergency referral due to haemorrhage 1 799 Risk Ratio (M‐H, Random, 95% CI) 0.79 [0.33, 1.93]
1.2.10 Number of women who had emergency referral due to premature labour 1 799 Risk Ratio (M‐H, Random, 95% CI) 0.73 [0.23, 2.38]
1.2.11 Number of women who had emergency referral due to fetal cardiac abnormality 1 799 Risk Ratio (M‐H, Random, 95% CI) 0.88 [0.22, 3.49]
1.2.12 Number of women who had emergency referral due to suspected sepsis 1 799 Risk Ratio (M‐H, Random, 95% CI) 0.70 [0.19, 2.60]
1.2.13 Number of women who had emergency referral due to respiratory compromise 1 799 Risk Ratio (M‐H, Random, 95% CI) 2.20 [0.43, 11.27]
1.2.14 Number of women who had emergency referral due to a preterm newborn 1 799 Risk Ratio (M‐H, Random, 95% CI) 7.92 [0.43, 146.67]
1.3 Patients' health status and well‐being (continuous outcomes) 1   Mean Difference (IV, Random, 95% CI) Subtotals only
1.3.1 Diabetes care self‐efficacy 1 176 Mean Difference (IV, Random, 95% CI) 3.30 [‐0.95, 7.55]
1.3.2 Antihyperglycaemic medication decisional conflict at 3 months of follow‐up 1 175 Mean Difference (IV, Random, 95% CI) 2.60 [‐2.07, 7.27]
1.3.3 Diabetes distress at 3 months of follow‐up 1 176 Mean Difference (IV, Random, 95% CI) 15.70 [8.24, 23.16]
1.3.4 Medication adherence at 3 months of follow‐up 1 176 Mean Difference (IV, Random, 95% CI) ‐2.30 [‐6.76, 2.16]
1.4 Patients' health status and well‐being (dichotomous undesirable outcome) 2   Risk Ratio (M‐H, Random, 95% CI) Subtotals only
1.4.1 Current smoker at 1 year of follow‐up 1 2086 Risk Ratio (M‐H, Random, 95% CI) 1.03 [0.92, 1.16]
1.4.2 Maternal deaths 1 799 Risk Ratio (M‐H, Random, 95% CI) 1.76 [0.16, 19.33]
1.5 Patients' health status and well‐being (continuous undesirable outcomes) 2   Mean Difference (IV, Random, 95% CI) Subtotals only
1.5.1 Mean systolic blood pressure (mmHg) 1 2086 Mean Difference (IV, Random, 95% CI) ‐2.80 [‐5.09, ‐0.51]
1.5.2 Haemoglobin a1c 1 175 Mean Difference (IV, Random, 95% CI) ‐0.10 [‐0.38, 0.18]
1.6 Patients' health status and well‐being (neonatal deaths) 2   Odds Ratio (IV, Random, 95% CI) Subtotals only
1.7 Patients' acceptability and satisfaction 1   Mean Difference (IV, Random, 95% CI) Subtotals only
1.7.1 Satisfaction with helpfulness of medication information 1 176 Mean Difference (IV, Random, 95% CI) 11.30 [3.28, 19.32]
1.7.2 Satisfaction with clarity of medication information 1 176 Mean Difference (IV, Random, 95% CI) 9.20 [0.97, 17.43]
1.8 Providers' acceptability and satisfaction 1   Other data No numeric data

1.1. Analysis.

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 1: Providers' adherence to recommended practices, guidelines, or protocols

Providers' adherence to recommended practices, guidelines, or protocols
Study Outcome Findings Comments
Berner 2006 1. Proportion of cases per physician with unsafe prescriptions
2. Proportion of cases per physician with key risk factor recorded
1. Intervention group (IG): 0.23
Control group (CG): 0.45
2. IG:0.58
CG: 0.45
IG: 31 participants
CG: 28 participants
Incomplete outcome data reported. Standard errors not reported.
Gautham 2015 1. Mean protocol compliance for female patients
2. Mean protocol compliance for male patients
1. IG: 69
CG: 63.34
2. IG: 71.12
CG: 53.59
1. IG: 8 providers, 38 patients
CG: 8 providers, 43 patients
2. IG: 8 providers, 27 patients
CG: 8 providers, 18 patients
Incomplete outcome data reported. Standard errors not reported.

1.2. Analysis.

1.2

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 2: Patients' health status and well‐being (dichotomous outcomes)

1.3. Analysis.

1.3

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 3: Patients' health status and well‐being (continuous outcomes)

1.4. Analysis.

1.4

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 4: Patients' health status and well‐being (dichotomous undesirable outcome)

1.5. Analysis.

1.5

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 5: Patients' health status and well‐being (continuous undesirable outcomes)

1.6. Analysis.

1.6

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 6: Patients' health status and well‐being (neonatal deaths)

1.7. Analysis.

1.7

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 7: Patients' acceptability and satisfaction

1.8. Analysis.

Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 8: Providers' acceptability and satisfaction

Providers' acceptability and satisfaction
Study Outcome Finding Comments
Tian 2015 1. Providers' level of comfort using the system
2. Providers' willingness to continue using the system
3. Providers' wish for more health conditions to be included in the system
4. Helpfullness of the system in enabling provider to follow guidelines
5. Ease of use of the system
6. Being able to remember steps without the system
7. Providers' willingness to recommend the system
1. 7/8
2. 7/8
3. 8/8
4. 6/8
5. 7/8
6. 0/8
7. 7/8
Outcomes measured after 2 months of using the system. Outcomes reported only for the intervention group. Incomplete data.

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Amoakoh 2019.

Study characteristics
Methods Aim: to evaluate the utilization and effect of the mCDMSI on institutional neonatal mortality in the Eastern Region of Ghana.
Study design: cluster randomized controlled trial
Country: Ghana
Setting: CHPS compounds and maternity homes, HCs, and hospitals in the Eastern Region
Cluster features: district located in the Eastern Region of Ghana; expected deliveries of ≥ 1100/year for the year 2014 for a district
Recruitment: within the randomized clusters, all health facilities that conducted deliveries in the year preceding the start of the intervention (2014) were recruited into this study.
Study duration: 18 months
Study dates: August 2015 to April 2017
Participants Inclusion criteria: women who delivered babies in hospitals in the study clusters for the 18‐month intervention period
Sample size: 8 intervention (74 facilities); 8 control (102 facilities) clusters; 65,831 deliveries in total
Age, mean: intervention 27.1 (SD 6.4) years; control 27.3 (SD 6.3) years for women delivering during the study
Sex: 100% female
Interventions Intervention: text messaging of standard protocols for maternal and neonatal care to front‐line healthcare providers in the region
Content: mCDMSI consisted of 4 components: phone calls (voice), text messaging (SMS), access to the internet (data) and access to an USSD that provided protocols for management of obstetric and neonatal emergencies in response to selection from a short code drop‐down menu
Control: no mobile phones or access to emergency protocols via USSD
Co‐interventions: not reported
Outcomes Institutional neonatal mortality including deaths of babies admitted from birth and those (re)admitted from home; utilization of the mCDMSI for clinical decision‐making
Outcome assessment timepoints: 18 months
Notes Funding: Netherlands Foundation for Scientific Research – WOTRO, Science for Global Development; Utrecht University
Conflicts of interest: none declared
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Low risk Used a randomization scheme of permuted blocks to randomize the 16 districts equally to the 2‐armed programme (control and intervention).
Allocation concealment (selection bias) Low risk Randomization performed by an independent data analyst in order to achieve comparability and avoid selection bias.
Blinding of participants and personnel (performance bias)
all outcomes High risk Due to the nature of this intervention, blinding was not feasible.
Blinding of outcome assessment (detection bias)
all outcomes Unclear risk Data were extracted from the district health information management system‐2 (DHIMS‐2) database.
Incomplete outcome data (attrition bias)
all outcomes Unclear risk Data that were available in the DHIMS‐2 and captured as aggregate per health facility were the number of neonatal deaths and the number of deliveries. Detailed information regarding each delivery captured in the DHIMS‐2 was limited to hospital deliveries, and further limited to peri‐partum maternal data.
Selective reporting (reporting bias) High risk Some outcomes listed in the clinical trial registry not reported in results.
Other bias High risk Likely to be baseline differences in neonatal mortality risk factors between study groups (in discussion).

Berner 2006.

Study characteristics
Methods Aim: to evaluate the effectiveness of a PDA‐based CDSS on NSAID prescribing safety in the outpatient setting.
Study design: randomized controlled trial
Country: USA
Setting: university‐based clinic
Recruitment: participants recruited from a pool of 105 internal medicine residents
Study duration: baseline phase approximately 6 months and follow‐up phase 8 months
Study dates: not reported
Participants Inclusion criteria: internal medicine residents assigned to an urban university‐based, resident‐staffed clinic
Sample size: 68 (intervention 34; control 34)
Age, mean: intervention 27.35 (SD 2.18) years; control 28.57 (SD 2.28) years
Sex: intervention 26% female; control 29% female
Interventions Intervention: CDSS (MedDecide) on PDA, the focus was a clinical prediction rule to assess NSAID‐related gastrointestinal risk and provide real‐time treatment recommendations based on the patient's risk.
Content: 19 rules representing diagnostic, risk assessment, and treatment recommendations. Participants were randomized to receive 1 of 6 different sets of these rules for their PDA, with each set containing 14 rules. Only the participants who were randomized to the intervention group received, within their set of 14 rules, the rule for gastrointestinal risk assessment when prescribing NSAIDs.
Control: clinical prediction rules that did not include the rule for gastrointestinal risk.
Co‐interventions: other clinical prediction rules on PDA (e.g. Epocrates, Medcalc, Medmath, and a breast cancer risk calculator).
Outcomes Patient safety; safe prescribing (i.e. duration and dose) of NSAIDs; presence of key risk factor
Outcome assessment timepoints: 8 months of follow‐up
Notes To evaluate outcomes, 4 standardized patient cases were used (13 trained standardized patients) and participants saw ≥ 1 standardized patient during both baseline and follow‐up.
Funding: supported in part by the Agency for Healthcare Research and Quality
Conflicts of interest: not reported
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Low risk Computerized random number generator that ensured an equal number of participants in each group.
Allocation concealment (selection bias) Unclear risk Not reported in text.
Blinding of participants and personnel (performance bias)
all outcomes Unclear risk Participants were 'blinded to the specific outcome of interest;' however, it was unclear whether standardized patients were blinded to whether they were seeing an intervention or control physician.
Blinding of outcome assessment (detection bias)
all outcomes Low risk Patient safety outcomes were determined by a review of documentation from standardized patient encounters. 5 physicians with experience in health services research constituted the outcomes committee. Each chart was independently reviewed by 2 clinicians who were blinded to participant, timing (baseline or follow‐up), and group (intervention or control).
Incomplete outcome data (attrition bias)
all outcomes Low risk Attrition was low: 3/34 intervention; 6/34 control. The reasons for attrition seem unlikely to be related to outcomes.
Selective reporting (reporting bias) Low risk No obvious missing outcomes or data. No protocol or trial registry was detected for the study.
Other bias Unclear risk No evidence of additional sources of bias, but reporting insufficient to be certain. For example, there may have been contamination between intervention and control participants.

Eaton 2011.

Study characteristics
Methods Aim: to determine whether an intervention based on patient activation and a physician decision‐support tool was more effective than usual care for improving adherence to National Cholesterol Education Program guidelines.
Study design: cluster randomized controlled trial
Country: USA
Setting: 30 primary care practices
Cluster features: 30 primary care practices in southeastern New England
Recruitment: not reported
Study duration: 2 years
Study dates: June 2003 to May 2005
Participants Inclusion criteria: not reported
Sample size: 30 practices, 55 physicians, 4105 patients
Age, mean: intervention 54.0 (SD 1.1) years; control 52.3 (SD 1.1) years
Sex: intervention 60.3% female; control 58.2% female
Interventions Intervention: interactive decision software on a PDA using data from a patient activation tool, HeartAge, in which patients answered questions regarding their risk factors for CHD (age, sex, blood pressure, total cholesterol, HDL‐cholesterol levels, diabetes).
Content: determined the patient's lipid diagnosis (LDL dominant, isolated low HDL level, triglyceride dominant, mixed lipid disorder, and atherogenic dyslipidaemia), calculated the ATP III LDL and non‐HDL cholesterol goals (when appropriate), made recommendations regarding therapeutic lifestyle management, provided optimal dosage of lipid‐lowering drugs tailored to the patient's risk factor status to meet the ATP III goals, and provided an interactive shared decision making page for physicians to discuss lowering‐lipid values in the context of HeartAge, absolute and relative risks, and other CHD risk factor management.
Control: received a PDA but without the decision‐support tool and had minimal further contact to mimic usual care.
Co‐interventions: not reported.
Outcomes LDL‐cholesterol values, non‐HDL‐cholesterol values; ATP III Cholesterol Guideline Adherence; how the tool affected physician decision making
Outcome assessment timepoints: 1 year
Notes Funding: not reported
Conflicts of interest: HeartAge is trademarked by Memorial Hospital of Rhode Island with Drs Eaton and Ahern as co‐developers.
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Unclear risk Method of randomization not reported.
Allocation concealment (selection bias) Unclear risk Method of allocation concealment not reported.
Blinding of participants and personnel (performance bias)
all outcomes High risk Not blinded.
Blinding of outcome assessment (detection bias)
all outcomes High risk Not blinded unless there was a disagreement.
Quote: "Any disagreement on the major outcome, LDL cholesterol values, and non‐HDL cholesterol values was reviewed by one of the investigators (D.R.P., C.B.E.) who were blinded to the physician and practice, and a final decision was made."
Incomplete outcome data (attrition bias)
all outcomes Low risk 78 patients in the control group were excluded from analysis because of death, pregnancy, or they left the practice. 56 patients in the intervention group were excluded from analysis because of death, pregnancy, or they left the practice.
Losses to follow‐up were low and balanced across groups.
Selective reporting (reporting bias) Low risk All outcomes reported.
Other bias Low risk No other obvious source of bias.

Gautham 2015.

Study characteristics
Methods Aim: to measure changes in protocol compliance by health workers in their everyday work settings, assess the usability and acceptability of the mobile application with health workers in the intervention group and obtain patient feedback on health workers' use of the mobile system during treatment.
Study design: cluster randomized controlled trial
Country: India
Setting: 2 neighbouring districts, Sivagangai and Dharmapuri, in rural Tamil Nadu, a South Indian state. At 1 of these sites (Thirupathur in the Sivagangai district) the participants consisted of 8 independent informal healthcare providers based in villages. In the other site, the participants linked with a non‐governmental hospital (Tribal Health Initiative hospital) in a tribal area called Sittilingi in the district of Dharmapuri.
Cluster features: 16 RHPs
Recruitment: the RHPs in Sivagangai had been part of a previous technology‐enabled distance learning project developed by 1 of the co‐authors. The 2 different RHP groups were selected because the authors assumed that they would provide valuable additional lessons on the contribution of local contexts in the way the application was finally used and its sustainability and success.
Study duration: 2 months
Study dates: not reported
Participants Inclusion criteria: not reported
Sample size: 16 RHPs, 126 patients
Age, mean: independent practitioners 48.75 years; auxiliaries attached to tribal hospital 24.75 years
Sex: RHPs 50% female
Interventions Intervention: media‐rich, mobile phone‐based clinical guidance system for management of fevers, diarrhoeas, and respiratory problems.
Content: 2 validated clinical guidelines were adapted for implementation in the decision‐support tool. These were the WHO's IMCI and IMAI.
Control: RHPs given only the phone plus a set of paper guidelines to use in the field. The application was installed on their phones after the field testing was over.
Co‐interventions: not reported.
Outcomes Protocol compliance (as per clinical guidelines); acceptability; usability
Outcome assessment timepoints: 2 months
Notes Funding: Information Society Innovation Fund (ISIF) Asia allocated in a competitive process to Garhwal Community Development and Welfare Society (GCDWS) in 2010. ISIF Asia operates through a partnership between the International Development Research Centre of Canada, the Swedish International Development Agency – SIDA and the Asia Pacific Network Information Centre (APNIC), with sponsorship from the Internet Society and the Dot Asia Organization.
Conflicts of interest: the second author (M Sriram Iyengar) is part owner of a company incorporated in the USA for commercialization of the GV technology.
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Unclear risk No information given on sequence generation. Participants consisted of 16 RHPs, 8 in each of 2 locations.
Quote: "In each location, four RHPs were randomly assigned to the experimental group and four to the control group; thus, there were eight RHPs in each group."
Allocation concealment (selection bias) Unclear risk No information given on allocation concealment. Participants consisted of 16 RHPs, 8 in each of 2 locations.
Quote: "In each location, four RHPs were randomly assigned to the experimental group and four to the control group; thus, there were eight RHPs in each group."
Blinding of participants and personnel (performance bias)
all outcomes High risk It is clear that the providers and patients were not blind to allocation. It would be impossible to blind the RHPs and difficult, although not impossible, to blind the patients.
Blinding of outcome assessment (detection bias)
all outcomes High risk Outcome assessors not blinded to allocation assessment. Outcome of protocol compliance was assessed by direct observation of participants who would be using their phones if in the intervention group. The usability/acceptability outcomes were only assessed in the intervention group.
Incomplete outcome data (attrition bias)
all outcomes Unclear risk It appears that all 16 providers completed the intervention period. There was no mention of any attrition. There was an under‐recruitment in some categories of the planned number of patients, which would have led to some estimates being based on fewer patients than planned.
Selective reporting (reporting bias) High risk It was not reported how many participants were included in the usability and acceptability results.
Other bias High risk The authors acknowledged that there may have been contamination between the groups in 1 district as both groups worked together. There was also a conflict of interest as 1 author is part owner of a company incorporated in the USA for commercialization of the GV technology used in developing mMRIGs and deploying them on mobile phones.

Heisler 2014.

Study characteristics
Methods Aim: to evaluate whether more sophisticated tailored, interactive e‐Health tools increase the effectiveness of CHW outreach with underserved patients compared to when they relied on printed educational materials alone.
Study design: randomized controlled trial
Country: USA
Setting: community HC in Detroit, USA
Recruitment: potentially eligible participants were identified from a computer‐generated list of the CHASS patients.
Study duration: 3 months
Study dates: September 2011 to September 2013
Participants Inclusion criteria: people with physician‐diagnosed type 2 diabetes; HbA1c > 7.5% in the prior 6 months or expressed concerns about current diabetes medications during the screening assessment
Sample size: 188 (intervention 93; control 95)
Age, mean: intervention 51 (SD 8.6); control 52 (SD 9.4)
Sex: intervention 76% female; control 66% female
Interventions Intervention: iDecide is a personally tailored, interactive diabetes medication decision aid designed for CHWs to deliver on tablet computers with 3G access to African American and Latino adults with diabetes and low health literacy.
Content: iDecide uses the same content as the Agency for Health Care Quality Guides ("Pills for Type 2 Diabetes" and "Premixed Insulin for Type 2 Diabetes").
Control: all participants received an initial 1‐to‐1, face‐to‐face session with a CHW and a copy of the printed materials to take home. The sessions using printed materials lasted approximately 1.5 hours.
Co‐interventions: not reported.
Outcomes Antihyperglycaemic medication decisional conflict; knowledge and beliefs about antihyperglycaemic medications; satisfaction with medication information (clarity and helpfulness); improvements in diabetes care self‐efficacy; diabetes distress; medication adherence
Outcome assessment timepoints: 3 months
Notes Funding: Agency for Health Care Quality and Research and the National Institute of Diabetes and Digestive and Kidney Diseases.
Conflicts of interest: no conflict of interest or financial disclosures.
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Low risk Quote: "participants … were randomized by the computer program using a random sequence algorithm into one of two study arms."
Allocation concealment (selection bias) Unclear risk No report of allocation concealment.
Blinding of participants and personnel (performance bias)
all outcomes High risk The nature of the intervention meant that it was impossible to blind the CHWs and patients.
Blinding of outcome assessment (detection bias)
all outcomes Low risk Quote: "data assessors remained blinded to group assignment throughout the study."
Incomplete outcome data (attrition bias)
all outcomes Low risk Attrition was low and balanced between groups. The intervention group lost 6/95 participants and the control group lost 6/97 participants. Most attrition was due to researchers being unable to reach participants and it is unlikely that this was related to the intervention. The authors used various methods to impute missing data and they reported that there were no significant changes in results.
Selective reporting (reporting bias) High risk The study was registered on ClinicalTrials.gov as NCT01427660. 1 of the primary outcomes in the study report (medication decisional conflict) was given as the primary outcome on the ClinicalTrials.gov entry. However, the secondary outcomes in the ClinicalTrials.gov entry were given as "self‐reported medication adherence will be assessed through three well‐validated measures. We will assess changes in anti‐hyperglycemic medication dosages and/or numbers of medications by patient report and medical record review." There only appears to be 1 measurement of medication adherence in the trial report. There were no data regarding changes to medication dosages. Therefore, there was a high risk of bias for selective outcome reporting.
Other bias Unclear risk There were some baseline imbalances.
Quote: "More participants randomized to iDecide had completed high school (61%) than those randomized to the printed materials group (35 %, p<0.001). Patients in the iDecide group were also less likely to have difficulty with written healthcare information (p=0.03) and were more likely to be confident filling out medical paperwork (p=0.003)."
Comment: it is likely that these factors would have an impact on the outcomes measured. The authors adjusted for these imbalances in their own analysis.

Keane 2018.

Study characteristics
Methods Aim: to evaluate a mobile health app to support the IMAM.
Study design: cluster randomized controlled trial
Country: Kenya
Setting: health facilities from three subcounties in Wajir
Cluster features: 40 health facilities from 3 subcounties
Recruitment: not reported
Study duration: 11 months
Study dates: November 2015 to October 2016
Participants Inclusion criteria: not reported
Sample size: 40 health facilities (20 interventions; 20 control), individuals not reported
Age: not reported
Sex: not reported
Interventions Intervention: IMAM app used on tablets or mobile phones provided health workers with simple, step‐by‐step guidance on the assessment, treatment, or referral of children visiting the IMAM programme.
Content: treatment protocol, counselling messages and return dates, and calculated z‐scores and numbers of ready‐to‐use therapeutic food sachets needed. It also recorded each child's information, making child follow‐up and defaulter tracing easier. Data were regularly uploaded to the 'cloud,' which enabled the provision of live and accurate data for county‐level management.
Control: paper‐based child treatment data from registers.
Co‐interventions: not reported.
Outcomes Neonatal deaths; maternal deaths; proportion of children cured, defaulted, died, or not cured when they exited the IMAM programme
Outcome assessment timepoints: 1 year
Notes Unclear if this is a peer‐reviewed journal, limited reporting of study methodology and results.
Funding: not reported
Conflicts of interest: not reported
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Unclear risk No information reported.
Allocation concealment (selection bias) Unclear risk No information reported.
Blinding of participants and personnel (performance bias)
all outcomes Unclear risk No information reported.
Blinding of outcome assessment (detection bias)
all outcomes Unclear risk No information reported.
Incomplete outcome data (attrition bias)
all outcomes Unclear risk No information reported.
Selective reporting (reporting bias) Unclear risk No information reported.
Other bias Unclear risk No information reported.

Martinez 2018.

Study characteristics
Methods Aim: to characterize baseline rates of complication detection and facility‐level referral by TBAs in rural Guatemala, and to evaluate the impact of the mHealth system on these rates.
Study design: randomized controlled trial
Country: Guatemala
Setting: Maya Health Alliance, a Guatemalan primary healthcare organization with a clinical centre in Tecpán
Recruitment: a list of 150 TBAs from the study area was produced in collaboration with local health officials.
Study duration: 12 months
Study dates: January 2015 to February 2018
Participants Inclusion criteria: midwives working in the Tecpán municipality, who had attended ≥ 5 deliveries per year in the previous 5 years, and who held a valid license to practice issued by local health authorities
Sample size: 44 TBAs randomized (intervention 23; control 21)
Age, mean: intervention 47 (range 40–55) years; control 51 (range 43–55) years
Sex: 100% female
Interventions Intervention: mHealth decision‐support system to improve maternal and perinatal complication detection and referral rates to facility‐level care by TBAs.
Content: the smartphone application allowed collection of simple demographics; maternal and perinatal symptoms and clinical signs; maternal vital signs (pulse, oxygen saturation, systolic and diastolic blood pressure); and the fetal heart rate. While using the application, TBAs were guided through a pictographic list of common maternal and perinatal complications grouped by visit type.
Control: usual care.
Co‐interventions: not reported.
Outcomes Number of monthly referrals to facility‐level care; proportion of referrals that were completed; adverse events
Outcome assessment timepoints: 7 months
Notes Funding: Eunice Kennedy Shriver National Institute of Child Health and Human Development and The Fogarty International Center at the National Institute of Health.
Conflicts of interest: no competing interests declared.
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Unclear risk Computer‐based randomization process. However, when > 1 TBA resided/practiced within the same settlement, they were allocated to the same study group.
Allocation concealment (selection bias) Low risk The allocation and assignment procedures were performed by a study author prior to meeting participants and not otherwise involved in the recruitment of participants or daily conduct of field work.
Blinding of participants and personnel (performance bias)
all outcomes High risk Due to the pragmatic nature of a trial involving access to mHealth technology, TBAs, pregnant participants, or study personnel could not be blinded to allocation.
Blinding of outcome assessment (detection bias)
all outcomes High risk Due to the pragmatic nature of a trial involving access to mHealth technology, TBAs, pregnant participants, or study personnel could not be blinded to allocation.
Incomplete outcome data (attrition bias)
all outcomes Low risk 2 TBAs in the intervention group (9%) and 1 in the comparison group (5%) discontinued their participation in the study but continued to report outcomes data, allowing for inclusion in the intention‐to‐treat analysis.
Selective reporting (reporting bias) Unclear risk All relevant outcomes reported, protocol checked.
Other bias Low risk No other bias apparent.

Tian 2015.

Study characteristics
Methods Aim: to develop and evaluate a simplified yet guideline‐based multifaceted intervention program for cardiovascular management among people with high cardiovascular risk delivered by CHWs with the aid of a mobile technology‐based EDSS in rural China and India.
Study design: cluster randomized controlled trial
Country: China and India
Setting: community based, either village clinics or patient homes
Cluster features: 27 villages from 15 townships (the administrative unit managing the village) in 2 counties of Tibet, China and 20 villages from 1 tehsil (an administrative unit for a group of villages) in Haryana State, India
Recruitment: not reported
Study duration: 1 year
Study dates: January 2012 to March 2014
Participants Inclusion criteria: residents in the participating villages and having high cardiovascular risk (aged ≥ 40 years with self‐reported history of CHD, stroke, diabetes mellitus, systolic blood pressure ≥ 160 mmHg, or a combination of these)
Sample size: 2086 participants, number of CHWs not reported
Age, mean: intervention 59.7 (SD 11.7); control 60.4 (SD 11.8)
Sex: intervention 65.4% female; control 66.8% female
Interventions Intervention: an electronic decision‐support component to assist the CHWs on the follow‐up and management of their high‐risk patients.
Content: prompts regarding the patient's medical history, new conditions, medication usage, current lifestyle habits, blood pressure, and the appropriateness for prescribing any of the target medications.
Control: usual cardiovascular management programmes continued without additional intervention.
Co‐interventions: not reported.
Outcomes Proportion of patient‐reported antihypertensive medication use; proportion of high‐risk people taking aspirin; systolic blood pressures of high‐risk people; proportion of current smokers; proportion of high‐risk people aware of the harms of a high‐salt diet; proportion of high‐risk people receiving monthly follow‐ups from the CHWs; proportion of high‐risk people hospitalized.
Outcome assessment timepoints: 1 year
Notes Funding: US National Heart, Lung, and Blood Institute, National Institutes of Health, Department of Health and Human Services; UnitedHealth Group Chronic Disease Initiative
Conflicts of interest: none reported
Risk of bias
Bias Authors' judgement Support for judgement
Random sequence generation (selection bias) Low risk Study‐independent staff generated the randomization pattern through a central computerized process.
Allocation concealment (selection bias) Low risk Study‐independent staff generated the randomization pattern through a central computerized process.
Blinding of participants and personnel (performance bias)
all outcomes High risk No blinding as CDSS compared to usual care.
Blinding of outcome assessment (detection bias)
all outcomes Unclear risk Quote: "The outcomes were assessed with data collected during baseline and post intervention surveys from all high‐risk individuals in both intervention and control villages in a standardized manner."
Incomplete outcome data (attrition bias)
all outcomes Low risk Intention‐to‐treat analysis including all randomized participants.
Selective reporting (reporting bias) Low risk All relevant outcomes reported.
Other bias Low risk No other bias.

ATP: Adult Treatment Panel; CDSS: clinical decision‐support system; CHASS: Community Health and Social Services Center; CHD: coronary heart disease; CHPS: community‐based Health Planning and Services; CHW: community health worker; EDDS: electronic decision‐support system; HC: health centre; HDL: high‐density lipoprotein; IMAI: Integrated Management of Adult Illnesses; IMAM: integrated management of acute malnutrition; MCI: Integrated Management of Childhood Illnesses; LDL: low‐density lipoprotein; mCDMSI: mHealth clinical decision‐making support intervention; NSAID: non‐steroidal anti‐inflammatory drug; PDA: personal digital assistant; RHP: rural health provider; SD: standard deviation; SMS: short message service; TBA: traditional birth attendant; USSD: unstructured supplementary service data; WHO: World Health Organization.

Characteristics of excluded studies [ordered by study ID]

Study Reason for exclusion
Arts 2017 Dependent on integration with electronic medical records/health tracking tools.
Caballero‐Ruiz 2017 Not used by healthcare provider.
Carroll 2012 Not accessible by or primarily used by mobile.
Carroll 2018 Dependent on integration with electronic medical records/health tracking tools.
Field 2009 Not primary care setting.
Forrest 2013 Dependent on integration with electronic medical records/health tracking tools.
Gill 2011 Dependent on integration with electronic medical records/health tracking tools.
Gong 2019 Decision support not major component of intervention.
Gulliford 2019 Dependent on integration with electronic medical records/health tracking tools.
Holbrook 2011 Dependent on integration with electronic medical records/health tracking tools.
Karlsson 2017 Dependent on integration with electronic medical records/health tracking tools.
Linder 2009 Dependent on integration with electronic medical records/health tracking tools.
Litvin 2013 Dependent on integration with electronic medical records/health tracking tools.
Mann 2014 Not accessible by or primarily used by mobile.
McGinn 2013 Dependent on integration with electronic medical records/health tracking tools.
Melnick 2019 Dependent on integration with electronic medical records/health tracking tools.
Morgan 2011 Not primary care setting.
Nam 2012 Not primary care setting.
Nendaz 2010 Not primary care setting.
Peiris 2019 Dependent on integration with electronic medical records/health tracking tools.
Prabhakaran 2019 Dependent on integration with electronic medical records/health tracking tools.
Schaeffer 2019 Not a randomized controlled trial.
Sturkenboom 2008 Dependent on integration with electronic medical records/health tracking tools.
Tajmir 2017 Not primary care setting.
Tamblyn 2008 Not used by healthcare provider.
Tebb 2019 Dependent on integration with electronic medical records/health tracking tools.
Wright 2012 Dependent on integration with electronic medical records/health tracking tools.

Characteristics of studies awaiting classification [ordered by study ID]

de Molina‐Férnandez 2019.

Methods Randomized controlled trial
Participants 708 women (intervention group 854; control group 854)
Interventions Contraceptive counselling
Outcomes Adherence to treatment; decisional conflict; satisfaction with the counsellor or clinician; test of knowledge prior to and after fieldwork
Notes The SHARECONTRACEPT tool, developed by previous phases of this project, is available at: decisionscompartides.gencat.cat/en/decidir-sobre/anticoncepcio_hormonal/

Keitel 2017.

Methods Randomized controlled trial
Participants Children aged 2–59 months presenting with acute febrile illness to 9 outpatient clinics in Dar es Salaam, Tanzania
Interventions e‐POCT is a novel electronic algorithm based on current evidence; it guides clinicians through the entire consultation and recommends treatment based on a few clinical signs and POCT results, some performed in all patients (malaria rapid diagnostic test, haemoglobin, oximeter) and others in selected subgroups only (C‐reactive protein, procalcitonin, glucometer)
Outcomes Proportion of clinical failures; proportion with antibiotics prescribed on day 0; primary referrals; and severe adverse events
Notes  

Khan 2020.

Methods Cluster randomized controlled trial
Participants Patients aged ≥ 2 months with uncomplicated acute diarrhoea.
Interventions Electronic decision support (rehydration calculator).
Outcomes Rate of intravenous fluid ordered; intravenous fluid volumes; antibiotics and zinc ordered; clinical course; adverse events
Notes  

e‐POCT: electronic point‐of‐care test; POCT: point‐of‐care test.

Characteristics of ongoing studies [ordered by study ID]

NCT03311399.

Study name Using mHealth technology to identify and refer surgical site infections in Rwanda
Methods Randomized controlled trial
Participants Women undergoing caesarean‐section surgery at a rural hospital in Rwanda
Interventions Surgical site infection screening protocol, delivered by community health workers equipped with mHealth support
Outcomes Number of women with surgical site infection returning to care
Starting date 15 March 2017
Contact information Robert Riviello, Brigham and Women's Hospital, Boston, Massachusetts, USA
Notes  

Differences between protocol and review

We added 'Patient health behaviour' as a separate outcome from 'client health status.'

We did not use network meta‐analysis to statistically infer a non‐digital/non‐mobile comparison group by comparing a study that compared digital intervention to non‐digital intervention with another study that compared one type of a digital intervention with another type of a digital intervention, as the interventions and outcomes were not sufficiently homogeneous to be grouped. We identified no studies that were sufficiently similar to combine the outcomes and therefore, did not perform any meta‐analyses. Given the small number of trials included in the review, we did not assess possible publication biases. Given the few included studies, we did not perform any additional subgroup analyses to assess the variation in the delivery of the intervention across different groups or characteristics.

Contributions of authors

Conceiving the protocol: CG, TT, SL, SA, GM, MF.

Designing the protocol: CG, TT, SL, SA, NM, NH, MF, GM.

Co‐ordinating the review: SA, CG, SL.

Writing the review: SA, CG.

Providing general advice on the review: CG, SL.

Securing funding for the review: GM.

All review authors approved the text

Publication history 

  • Review submitted to the editorial base for peer review whislt update searches were updated and incorporated: 10/09/2020

  • Review returned to authors following peer review: 09/02/2021

  • Revised review received by the editorial base: 15/04/2021

  • Review returned to authors following editorial comments and copy edit: 14/06/2021

  • Review submitted for publication: 29/06/2021

  • Review accepted for publication: 21/07/21

Contributions of the editorial base

EPOC managing editor: Co‐ordinated the editorial process and contributed to peer review (Chris Cooper)

EPOC internal referee (EPOC): peer‐review (Daniela Gonçalves Bradley)

Information specialist (EPOC): provided comments on the search approach (Paul Miller).

Sources of support

Internal sources

  • No sources of support provided

External sources

  • This work was funded by the UNDP‐UNFPA‐UNICEF‐WHO‐World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), a cosponsored program executed by the World Health Organization (WHO), Switzerland

Declarations of interest

SA: The author was commissioned by the WHO to conduct this review.

CG: I am an editor for Cochrane Effective Practice and Organisation of Care. I was not involved in the editorial process for this review.

TT: Works for the WHO.

NH: Since June 2016, I have been employed by Cochrane Response, an evidence services unit operated by Cochrane. Cochrane Response was contracted by the WHO to produce this review. 

MS: None known. 

GM: Owns stock in Apple computers.

NM: I previously worked for Enhanced Reviews Ltd, a company that conducts systematic reviews, mostly for the public sector. Since June 2016, I have been employed by Cochrane Response, an evidence services unit operated by Cochrane. Cochrane Response was contracted by the World Health Organization to produce this review.

MF: The author was commissioned by the WHO to conduct this review.

GM: Works for the WHO.

SL: I am the Joint Co‐ordinating Editor for Cochrane Effective Practice and Organisation of Care. I was not involved in the editorial process for this review.

Edited (no change to conclusions)

References

References to studies included in this review

Amoakoh 2019 {published data only}

  1. Amoakoh HB, Klipstein-Grobusch K, Agyepong IA, Zuithoff NP, Amoakoh-Coleman M, Kayode GA, et al. The effect of an mHealth clinical decision-making support system on neonatal mortality in a low resource setting: a cluster-randomized controlled trial. EClinicalMedicine 2019;12:31-42. [DOI] [PMC free article] [PubMed] [Google Scholar]

Berner 2006 {published data only}

  1. Berner ES, Houston TK, Ray MN, Allison JJ, Heudebert GR, Chatham WW, et al. Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial. Journal of the American Medical Informatics Association 2006;13(2):171-9. [DOI] [PMC free article] [PubMed] [Google Scholar]

Eaton 2011 {published data only}

  1. Eaton CB, Parker DR, Borkan J, McMurray J, Roberts MB, Lu B, et al. Translating cholesterol guidelines into primary care practice: a multimodal cluster randomized trial. Annals of Family Medicine 2011;9(6):528-37. [DOI] [PMC free article] [PubMed] [Google Scholar]

Gautham 2015 {published data only}

  1. Gautham M, Iyengar MS, Johnson CW. Mobile phone–based clinical guidance for rural health providers in India. Health informatics journal 2015;21(4):253-66. [DOI] [PubMed] [Google Scholar]

Heisler 2014 {published data only}

  1. Heisler M, Choi H, Palmisano G, Mase R, Richardson C, Fagerlin A, et al. Comparison of community health worker–led diabetes medication decision-making support for low-income Latino and African American adults with diabetes using e-health tools versus print materials: a randomized, controlled trial. Annals of Internal Medicine 2014;161(10 Suppl):S13-22. [DOI] [PMC free article] [PubMed] [Google Scholar]

Keane 2018 {published data only}

  1. Frank T, Keane E, Roschnik N, Emary C, O'Leary M, Snyder L. Developing a mobile healthapp to manage acute malnutrition: a five-country experience. Field Exchange 2017;54:7. [Google Scholar]
  2. Keane E, Roschnik N, Chui J, Osman IA, Osman HM. Evaluation of mobile application to support the treatment of acutely malnourished children in Wajir county, Kenya. Field Exchange 2018;57:61. [Google Scholar]

Martinez 2018 {published data only}

  1. Martinez B, Ixen EC, Hall-Clifford R, Juarez M, Miller AC, Francis A, et al. mHealth intervention to improve the continuum of maternal and perinatal care in rural Guatemala: a pragmatic, randomized controlled feasibility trial. Reproductive health 2018;15(1):120. [DOI] [PMC free article] [PubMed] [Google Scholar]

Tian 2015 {published data only}

  1. Tian M, Ajay V, Dunzhu D, Hameed S, Li X, Liu Z, et al. A cluster-randomized controlled trial of a simplified multifaceted management program for individuals at high cardiovascular risk (SimCard Trial) in Rural Tibet, China, and Haryana, India. Circulation 2015;17:115. [DOI] [PMC free article] [PubMed] [Google Scholar]

References to studies excluded from this review

Arts 2017 {published data only}

  1. Arts DL, Abu-Hanna A, Medlock SK, Weert HC. Effectiveness and usage of a decision support system to improve stroke prevention in general practice: a cluster randomized controlled trial. PloS One 2017;12(2):e0170974. [DOI] [PMC free article] [PubMed] [Google Scholar]

Caballero‐Ruiz 2017 {published data only}

  1. Caballero-Ruiz E, García-Sáez G, Rigla M, Villaplana M, Pons B, Hernando ME. A web-based clinical decision support system for gestational diabetes: automatic diet prescription and detection of insulin needs. International Journal of Medical Informatics 2017;102:35-49. [DOI] [PubMed] [Google Scholar]

Carroll 2012 {published data only}

  1. Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A randomized controlled trial of screening for maternal depression with a clinical decision support system. Journal of the American Medical Informatics Association 2012;20(2):311-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

Carroll 2018 {published data only}

  1. Carroll JK, Pulver G, Dickinson LM, Pace WD, Vassalotti JA, Kimminau KS, et al. Effect of 2 clinical decision support strategies on chronic kidney disease outcomes in primary care: a cluster randomized trial. JAMA Network Open 2018;1(6):e183377. [DOI] [PMC free article] [PubMed] [Google Scholar]

Field 2009 {published data only}

  1. Field TS, Rochon P, Lee M, Gavendo L, Baril JL, Gurwitz JH. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. Journal of the American Medical Informatics Association 2009;16(4):480-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

Forrest 2013 {published data only}

  1. Forrest CB, Fiks AG, Bailey LC, Localio R, Grundmeier RW, Richards T, et al. Improving adherence to otitis media guidelines with clinical decision support and physician feedback. Pediatrics 2013;131(4):e1071-81. [DOI] [PubMed] [Google Scholar]

Gill 2011 {published data only}

  1. Gill JM, Mainous AG, Koopman RJ, Player MS, Everett CJ, Chen YX, et al. Impact of EHR-based clinical decision support on adherence to guidelines for patients on NSAIDs: a randomized controlled trial. Annals of Family Medicine 2011;9(1):22-30. [DOI] [PMC free article] [PubMed] [Google Scholar]

Gong 2019 {published data only}

  1. Gong E, Gu W, Sun C, Turner EL, Zhou Y, Li Z, et al. System-integrated technology-enabled model of care to improve the health of stroke patients in rural China: protocol for SINEMA – a cluster-randomized controlled trial. American Heart Journal 2019;207:27-39. [DOI] [PubMed] [Google Scholar]

Gulliford 2019 {published data only}

  1. Gulliford MC, Prevost AT, Charlton J, Juszczyk D, Soames J, McDermott L, et al. Effectiveness and safety of electronically delivered prescribing feedback and decision support on antibiotic use for respiratory illness in primary care: REDUCE cluster randomised trial. BMJ 2019;364:l236. [DOI] [PMC free article] [PubMed] [Google Scholar]

Holbrook 2011 {published data only}

  1. Holbrook A, Pullenayegum E, Thabane L, Troyan S, Foster G, Keshavjee K, et al. Shared electronic vascular risk decision support in primary care: Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE III) randomized trial. Archives of Internal Medicine 2011;171(19):1736-44. [DOI] [PubMed] [Google Scholar]

Karlsson 2017 {published data only}

  1. Karlsson LO, Nilsson S, Charitakis E, Bång M, Johansson G, Nilsson L, et al. Clinical decision support for stroke prevention in atrial fibrillation (CDS-AF): rationale and design of a cluster randomized trial in the primary care setting. American Heart Journal 2017;187:45-52. [DOI] [PubMed] [Google Scholar]

Linder 2009 {published data only}

  1. Linder J, Schnipper J, Tsurikova R, Yu T, Volk L, Melnikas A, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Journal of Innovation in Health Informatics 2009;17(4):231-40. [DOI] [PubMed] [Google Scholar]

Litvin 2013 {published data only}

  1. Litvin CB, Ornstein SM, Wessell AM, Nemeth LS, Nietert PJ. Use of an electronic health record clinical decision support tool to improve antibiotic prescribing for acute respiratory infections: the ABX-TRIP study. Journal of General Internal Medicine 2013;28(6):810-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

Mann 2014 {published data only}

  1. Mann D, Knaus M, McCullagh L, Sofianou A, Rosen L, McGinn T, et al. Measures of user experience in a streptococcal pharyngitis and pneumonia clinical decision support tools. Applied Clinical Informatics 2014;5(03):824-35. [DOI] [PMC free article] [PubMed] [Google Scholar]

McGinn 2013 {published data only}

  1. McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Internal Medicine 2013;173(17):1584-91. [DOI] [PubMed] [Google Scholar]

Melnick 2019 {published data only}

  1. Melnick ER, Jeffery MM, Dziura JD, Mao JA, Hess EP, Platts-Mills TF, et al. User-centred clinical decision support to implement emergency department-initiated buprenorphine for opioid use disorder: protocol for the pragmatic group randomised EMBED trial. BMJ Open 2019;9(5):e028488. [DOI] [PMC free article] [PubMed] [Google Scholar]

Morgan 2011 {published data only}

  1. Morgan MB, Branstetter BF 4th, Clark C, House J, Baker D, Harnsberger HR. Just-in-time radiologist decision support: the importance of PACS-integrated workflow. Journal of the American College of Radiology 2011;8(7):497-500. [DOI] [PubMed] [Google Scholar]

Nam 2012 {published data only}

  1. Nam HS, Cha M‐J, Kim YD, Kim EH, Park E, Lee HS, et al. Use of a handheld, computerized device as a decision support tool for stroke classification. European Journal of Neurology 2012;19(3):426-30. [DOI] [PubMed] [Google Scholar]

Nendaz 2010 {published data only}

  1. Nendaz MR, Chopard P, Lovis C, Kucher N, Asmis LM, Dörffler J, et al. Adequacy of venous thromboprophylaxis in acutely ill medical patients (IMPART): multisite comparison of different clinical decision support systems. Journal of Thrombosis and Haemostasis 2010;8(6):1230-4. [DOI] [PubMed] [Google Scholar]

Peiris 2019 {published data only}

  1. Peiris D, Praveen D, Mogulluru K, Ameer MA, Raghu A, Li Q, et al. SMARThealth India: a stepped-wedge, cluster randomised controlled trial of a community health worker managed mobile health intervention for people assessed at high cardiovascular disease risk in rural India. PloS One 2019;14(3):e0213708. [DOI] [PMC free article] [PubMed] [Google Scholar]

Prabhakaran 2019 {published data only}

  1. Prabhakaran D, Jha D, Prieto-Merino D, Roy A, Singh K, Ajay VS, et al. Effectiveness of an mHealth-based electronic decision support system for integrated management of chronic conditions in primary care: the mWellcare cluster-randomized controlled trial. Circulation 2019;139(3):380-91. [DOI] [PubMed] [Google Scholar]

Schaeffer 2019 {published data only}

  1. Schaeffer LE, Ahmed S, Rahman M, Whelan R, Rahman S, Roy AD, et al. Development and evaluation of a mobile application for case management of small and sick newborns in Bangladesh. BMC Medical Informatics and Decision Making 2019;19(1):116. [DOI] [PMC free article] [PubMed] [Google Scholar]

Sturkenboom 2008 {published data only}

  1. Wyk JT, Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, Lei J. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment. Circulation 2008;117(3):371-8. [DOI] [PubMed] [Google Scholar]

Tajmir 2017 {published data only}

  1. Tajmir S, Raja AS, Ip IK, Andruchow J, Silveira P, Smith S, et al. Impact of clinical decision support on radiography for acute ankle injuries: a randomized trial. Western Journal of Emergency Medicine 2017;18(3):487. [DOI] [PMC free article] [PubMed] [Google Scholar]

Tamblyn 2008 {published data only}

  1. Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. Journal of the American Medical Informatics Association 2008;15(4):430-8. [DOI] [PMC free article] [PubMed] [Google Scholar]

Tebb 2019 {published data only}

  1. Tebb KP, Trieu SL, Rico R, Renteria R, Rodriguez F, Puffer M. A mobile health contraception decision support intervention for Latina adolescents: implementation evaluation for use in school-based health centers. JMIR mHealth and uHealth 2019;7(3):e11163. [DOI] [PMC free article] [PubMed] [Google Scholar]

Wright 2012 {published data only}

  1. Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, McLoughlin KS, et al. Improving completeness of electronic problem lists through clinical decision support: a randomized, controlled trial. Journal of the American Medical Informatics Association 2012;19(4):555-61. [DOI] [PMC free article] [PubMed] [Google Scholar]

References to studies awaiting assessment

de Molina‐Férnandez 2019 {published data only}

  1. Molina-Férnandez MI, Raigal-Aran L, De la Flor-Lopez M, Prata P, Font-Jimenez I, Valls-Fonayet F, et al. The effectiveness of a digital shared decision-making tool in hormonal contraception during clinical assessment: study protocol of a randomized controlled trial in Spain. BMC Public Health 2019;19(1):1-8. [DOI] [PMC free article] [PubMed] [Google Scholar]

Keitel 2017 {published data only}

  1. Keitel K, Kagoro F, Samaka J, Masimba J, Said Z, Temba H, et al. A novel electronic algorithm using host biomarker point-of-care tests for the management of febrile illnesses in Tanzanian children (e-POCT): a randomized, controlled non-inferiority trial. PLoS Medicine 2017;14(10):e1002411. [DOI: 10.1371/journal.pmed.1002411] [PMID: ] [DOI] [PMC free article] [PubMed] [Google Scholar]

Khan 2020 {published data only}

  1. Khan AI, Mack JA, Salimuzzaman M, Zion MI, Sujon H, Ball RL, et al. Electronic decision support and diarrhoeal disease guideline adherence (mHDM): a cluster randomised controlled trial. Lancet Digital Health 2020;2(5):e250-8. [CLINICALTRIALS.GOV: NCT03154229] [DOI] [PMC free article] [PubMed] [Google Scholar]

References to ongoing studies

NCT03311399 {published data only}

  1. NCT03311399. Using mHealth technology to identify and refer surgical site infections in Rwanda. clinicaltrials.gov/ct2/show/NCT03311399 (first received 17 October 2017). [CLINICALTRIALS.GOV: NCT03311399]

Additional references

Adepoju 2017

  1. Adepoju IO, Albersen BJ, De Brouwere V, Roosmalen J, Zweekhorst M. mHealth for clinical decision-making in Sub-Saharan Africa: a scoping review. JMIR mHealth and uHealth 2017;5(3):e38. [DOI] [PMC free article] [PubMed] [Google Scholar]

Agarwal 2015

  1. Agarwal S, Perry HB, Long L-A, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Tropical Medicine & International Health 2015;20(8):1003-14. [DOI] [PMC free article] [PubMed] [Google Scholar]

Agarwal 2016

  1. Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: Mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174. [DOI] [PubMed] [Google Scholar]

Arain 2010

  1. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology 2010;10(1):67. [DOI] [PMC free article] [PubMed] [Google Scholar]

Awofeso 2004

  1. Awofeso N. What is the difference between 'primary care' and 'primary healthcare'? Quality in Primary Care 2004;12(2):93-4. [Google Scholar]

Baker 2007

  1. Baker B, Benton D, Friedman E, Russell A. Systems support for task-shifting to community health workers. chwcentral.org/sites/default/files/Systems%20Support%20for%20Task-Shifting%20to%20Community%20Health%20Workers.pdf (accessed 2 October 2017).

Baptista 2018

  1. Baptista S, Sampaio ET, Heleno B, Azevedo LF, Martins C. Web-based versus usual care and other formats of decision aids to support prostate cancer screening decisions: systematic review and meta-analysis. Journal of Medical Internet Research 2018;20(6):e9070. [DOI] [PMC free article] [PubMed] [Google Scholar]

Blank 2013

  1. Blank A, Prytherch H, Kaltschmidt H, Krings A, Sukums F, Mensah N, et al. "Quality of prenatal and maternal care: bridging the know-do gap" (QUALMAT study): an electronic clinical decision support system for rural Sub-Saharan Africa. BMC Medical Informatics and Decision Making 2013;13(1):44. [DOI] [PMC free article] [PubMed] [Google Scholar]

Brenner 2016

  1. Brenner SK, Kaushal R, Grinspan Z, Joyce C, Kim I, Allard RJ, et al. Effects of health information technology on patient outcomes: a systematic review. Journal of the American Medical Informatics Association 2016;23(5):1016-36. [DOI] [PMC free article] [PubMed] [Google Scholar]

Bright 2012

  1. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Annals of Internal Medicine 2012;157(1):29-43. [DOI] [PubMed] [Google Scholar]

Carter 2019

  1. Carter J, Sandall J, Shennan AH, Tribe RM. Mobile phone apps for clinical decision support in pregnancy: a scoping review. BMC Medical Informatics and DEcision Making 2019;19:1. [DOI] [PMC free article] [PubMed] [Google Scholar]

DistillerSR [Computer program]

  1. DistillerSR. Ottawa (Canada): Evidence Partners, (accessed 15 December 2017).

Divall 2013

  1. Divall P, Camosso-Stefinovic J, Baker R. The use of personal digital assistants in clinical decision making by health care professionals: a systematic review. Health Informatics Journal 2013;19(1):16-28. [DOI] [PubMed] [Google Scholar]

Dreesens 2019

  1. Dreesens D, Kremer L, Weijden T. The Dutch chaos case: a scoping review of knowledge and decision support tools available to clinicians in the Netherlands. Health Policy 2019;123(12):1288-97. [DOI] [PubMed] [Google Scholar]

Dussault 2006

  1. Dussault G, Franceschini MC. Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Human Resources for Health 2006;4(1):12. [DOI] [PMC free article] [PubMed] [Google Scholar]

EPOC 2017a

  1. Cochrane Effective Practice and Organisation of Care (EPOC). Data collection form. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).

EPOC 2017b

  1. Cochrane Effective Practice and Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).

EPOC 2017c

  1. Cochrane Effective Practice and Organisation of Care (EPOC). EPOC worksheets for preparing a 'Summary of findings' table using GRADE. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).

Global Health Watch 2011

  1. Global Health Watch. Primary health care: a review and critical appraisal of its revitalization. www.ghwatch.org/sites/www.ghwatch.org/files/B1_0.pdf (accessed 2 October 2017).

GRADEpro GDT [Computer program]

  1. McMaster University (developed by Evidence Prime) GRADEpro GDT. Version (accessed 15 December 2017). Hamilton (ON): McMaster University (developed by Evidence Prime), 2015. Available at gradepro.org.

Guyatt 2008

  1. Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al, GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336(7650):924-6. [DOI] [PMC free article] [PubMed] [Google Scholar]

Higgins 2017

  1. Higgins JP, Savović J, Page MJ, Elbers RG, Sterne JA. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). Cochrane, 2017. Available from: handbook: training.cochrane.org/handbook/archive/v5.2.

Hunt 1998

  1. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280(15):1339:46. [DOI] [PubMed] [Google Scholar]

International Telecommunications Union 2015

  1. International Telecommunications Union. Information and Communication Technology (ICT) facts and figures. www.itu.int/en/ITU-D/Statistics/Documents/facts/ICTFactsFigures2015.pdf (accessed 2 October 2017).

Jaspers 2011

  1. Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association 2011;18(3):327-34. [DOI] [PMC free article] [PubMed] [Google Scholar]

Kaplan 2001

  1. Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. International Journal of Medical Informatics 2001;64(1):15-37. [DOI] [PubMed] [Google Scholar]

Kawamoto 2005

  1. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494):765. [DOI] [PMC free article] [PubMed] [Google Scholar]

Kilsdonk 2017

  1. Kilsdonk E, Peute LW, Jaspers MW. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. International Journal of Medical Informatics 2017;1(98):56-64. [DOI] [PubMed] [Google Scholar]

Kocher 2010

  1. Kocher R, Emanuel EJ, DeParle NA. The Affordable Care Act and the future of clinical medicine: the opportunities and challenges. Annals of Internal Medicine 2010;153(8):536-9. [DOI] [PubMed] [Google Scholar]

Kruk 2018

  1. Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Global Health Commission 2018;6:e1196. [DOI: ] [DOI] [PMC free article] [PubMed] [Google Scholar]

Lefebvre 2011

  1. Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editor(s). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available at: handbook: training.cochrane.org/handbook/archive/v5.1/.

Lehmann 2008

  1. Lehmann U, Dieleman M, Martineau T. Staffing remote rural areas in middle- and low-income countries: a literature review of attraction and retention. BMC Health Services Research 2008;8(1):19. [DOI] [PMC free article] [PubMed] [Google Scholar]

Liberati 2009

  1. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine 2009;6(7):e1000100. [DOI] [PMC free article] [PubMed] [Google Scholar]

Martínez‐Pérez 2014

  1. Martínez-Pérez B, la Torre-Díez I, López-Coronado M, Sainz-de-Abajo B, Robles M, García-Gómez JM. Mobile clinical decision support systems and applications: a literature and commercial review. Journal of Medical Systems 2014;38(1):4. [DOI] [PubMed] [Google Scholar]

Mazo 2020

  1. Mazo C, Kearns C, Mooney C, Gallagher WM. Clinical decision support systems in breast cancer: a systematic review. Cancers 2020;12(2):369. [DOI] [PMC free article] [PubMed] [Google Scholar]

McKenzie 2021

  1. McKenzie JE, Brennan SE. Synthesizing and presenting findings using other methods. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editors(s). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 . Cochrane,  February 2021. [Google Scholar]

Mickan 2014

  1. Mickan S, Atherton H, Roberts NW, Heneghan C, Tilson JK. Use of handheld computers in clinical practice: a systematic review. BMC Medical Informatics and Decision Making 2014;14(1):56. [DOI] [PMC free article] [PubMed] [Google Scholar]

Middleton 2016

  1. Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearbook of Medical Informatics 2016;25(S01):S103-16. [DOI] [PMC free article] [PubMed] [Google Scholar]

Mishra 2019

  1. Mishra SR, Lygidakis C, Neupane D, Gyawali B, Uwizihiwe JP, Virani SS, et al. Combating non-communicable diseases: potentials and challenges for community health workers in a digital age: a narrative review of the literature. Health Policy and Planning 2019;34(1):55-66. [DOI] [PubMed] [Google Scholar]

Mohanan 2015

  1. Mohanan M, Vera-Hernández M, Das V, Giardili S, Goldhaber-Fiebert JD, Rabin TL, et al. The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India. JAMA Pediatrics 2015;169(4):349-57. [DOI] [PMC free article] [PubMed] [Google Scholar]

Moja 2014

  1. Moja L, Kwag KH, Lytras T, Bertizzolo L, Pecoraro V, Rigon G, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. American Journal of Public Health 2014;104(12):e12-22. [DOI] [PMC free article] [PubMed] [Google Scholar]

Muldoon 2006

  1. Muldoon LK, Hogg WE, Levitt M. Primary care (PC) and primary health care (PHC): what is the difference? Canadian Journal of Public Health/Revue Canadienne de Santé Publique 2006;409(5):409-11. [DOI] [PMC free article] [PubMed] [Google Scholar]

Njie 2015

  1. Njie GJ, Proia KK, Thota AB, Finnie RK, Hopkins DP, Banks SM, et al. Clinical decision support systems and prevention: a community guide cardiovascular disease systematic review. American Journal of Preventive Medicine 2015;49(5):784-95. [DOI] [PMC free article] [PubMed] [Google Scholar]

Odendaal 2020

  1. Odendaal WA, Watkins JA, Leon N, Goudge J, Griffiths F, Tomlinson M, et al. Health workers' perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews 2020, Issue 1. Art. No: CD011942. [DOI: 10.1002/14651858.CD011942.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

Orton 2018

  1. Orton M, Agarwal S, Muhoza P, Vasudevan L, Vu A. Strengthening delivery of health services using digital devices. Global Health: Science and Practice 2018;6(1):S61-71. [DOI] [PMC free article] [PubMed] [Google Scholar]

Rednick 2014

  1. Rednick C, Dini HS, Long LA. The current state of CHW training programs in Sub-Saharan Africa and South Asia: what we know, what we don't know, and what we need to do. 1millionhealthworkers.org/files/2013/01/1mCHW_mPowering_LitReview_Formatted.compressed.pdf (accessed 2 October 2017).

Review Manager 2014 [Computer program]

  1. Nordic Cochrane Centre, The Cochrane Collaboration Review Manager 5 (RevMan 5). Version 5.3. Copenhagen: Nordic Cochrane Centre, The Cochrane Collaboration, 2014.

Richardson 2019

  1. Richardson KM, Fouquet SD, Kerns E, McCulloh RJ. Impact of mobile device-based clinical decision support tool on guideline adherence and mental workload. Academic Pediatrics 2019;19(7):828-34. [DOI] [PMC free article] [PubMed] [Google Scholar]

Schünemann 2017

  1. Schünemann HJ, Oxman AD, Higgins JP, Vist GE, Glasziou P, Akl E, et al, on behalf of the Cochrane GRADEing Methods Group and the Cochrane Statistical Methods Group. Chapter 11: Completing 'Summary of findings' tables and grading the confidence in or quality of the evidence. In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). Cochrane, 2017. Available from: handbook: training.cochrane.org/handbook/archive/v5.2.

Sterne 2011

  1. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002. [DOI: 10.1136/bmj.d4002] [DOI] [PubMed] [Google Scholar]

Sutton 2020

  1. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, strategies for success. npj Digital Medicine 2020;3:17. [DOI] [PMC free article] [PubMed] [Google Scholar]

Syrowatka 2016

  1. Syrowatka A, Krömker D, Meguerditchian AN, Tamblyn R. Features of computer-based decision aids: systematic review, thematic synthesis, and meta-analyses. Journal of Medical Internet Research 2016;18(1):e20. [DOI] [PMC free article] [PubMed] [Google Scholar]

Tong 2021

  1. Tong G, Geng Q, Wang D, Liu T. Web-based decision aids for cancer clinical decisions: a systematic review and meta-analysis. Support Care Cancer 2021 Apr 8 [Epub ahead of print]. [DOI: ] [DOI] [PubMed]

Varghese 2018

  1. Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review. Journal of the American Medical Informatics Association 2018;25(5):593-602. [DOI] [PMC free article] [PubMed] [Google Scholar]

Wallace 2017

  1. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. Journal of the American Medical Informatics Association 2017;24(6):1165-8. [DOI] [PMC free article] [PubMed] [Google Scholar]

WHO 2016

  1. World Health Organization. Health workforce requirements for universal health coverage and sustainable development goals. apps.who.int/iris/bitstream/handle/10665/250330/9789241511407-eng.pdf (accessed 25 March 2021).

WHO 2019

  1. World Health Organization. WHO guideline: recommendations on digital interventions for health systems strengthening, 2019. www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/ (accessed 9 June 2021). [PubMed]

WHO 2020

  1. World Health Organization. Global strategy on human resources for health: Workforce 2030. www.who.int/hrh/resources/global_strategy_workforce2030_14_print.pdf (accessed 25 March 2021).

Yau 2019

  1. Yau M, Timmerman V, Zwarenstein M, Mayers P, Cornick RV, Bateman E, et al. e-PC101: an electronic clinical decision support tool developed in South Africa for primary care in low-income and middle-income countries. BMJ Global Health 2019;3:e001093. [DOI] [PMC free article] [PubMed] [Google Scholar]

References to other published versions of this review

Agarwal 2018

  1. Agarwal S, Tamrat T, Glenton C, Lewin S, Henschke N, Maayan N, et al. Decision‐support tools via mobile devices to improve quality of care in primary healthcare settings. Cochrane Database of Systematic Reviews 2018, Issue 2. Art. No: CD012944. [DOI: 10.1002/14651858.CD012944] [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from The Cochrane Database of Systematic Reviews are provided here courtesy of Wiley

RESOURCES