Abstract
Background
The ubiquity of mobile devices has made it possible for clinical decision‐support systems (CDSS) to become available to healthcare providers on handheld devices at the point‐of‐care, including in low‐ and middle‐income countries. The use of CDSS by providers can potentially improve adherence to treatment protocols and patient outcomes. However, the evidence on the effect of the use of CDSS on mobile devices needs to be synthesized. This review was carried out to support a World Health Organization (WHO) guideline that aimed to inform investments on the use of decision‐support tools on digital devices to strengthen primary healthcare.
Objectives
To assess the effects of digital clinical decision‐support systems (CDSS) accessible via mobile devices by primary healthcare providers in the context of primary care settings.
Search methods
We searched CENTRAL, MEDLINE, Embase, Global Index Medicus, POPLINE, and two trial registries from 1 January 2000 to 9 October 2020. We conducted a grey literature search using mHealthevidence.org and issued a call for papers through popular digital health communities of practice. Finally, we conducted citation searches of included studies.
Selection criteria
Study design: we included randomized trials, including full‐text studies, conference abstracts, and unpublished data irrespective of publication status or language of publication.
Types of participants: we included studies of all cadres of healthcare providers, including lay health workers and other individuals (administrative, managerial, and supervisory staff) involved in the delivery of primary healthcare services using clinical decision‐support tools; and studies of clients or patients receiving care from primary healthcare providers using digital decision‐support tools.
Types of interventions: we included studies comparing digital CDSS accessible via mobile devices with non‐digital CDSS or no intervention, in the context of primary care. CDSS could include clinical protocols, checklists, and other job‐aids which supported risk prioritization of patients. Mobile devices included mobile phones of any type (but not analogue landline telephones), as well as tablets, personal digital assistants, and smartphones. We excluded studies where digital CDSS were used on laptops or integrated with electronic medical records or other types of longitudinal tracking of clients.
Data collection and analysis
A machine learning classifier that gave each record a probability score of being a randomized trial screened all search results. Two review authors screened titles and abstracts of studies with more than 10% probability of being a randomized trial, and one review author screened those with less than 10% probability of being a randomized trial. We followed standard methodological procedures expected by Cochrane and the Effective Practice and Organisation of Care group. We used the GRADE approach to assess the certainty of the evidence for the most important outcomes.
Main results
Eight randomized trials across varying healthcare contexts in the USA,. India, China, Guatemala, Ghana, and Kenya, met our inclusion criteria. A range of healthcare providers (facility and community‐based, formally trained, and lay workers) used digital CDSS. Care was provided for the management of specific conditions such as cardiovascular disease, gastrointestinal risk assessment, and maternal and child health. The certainty of evidence ranged from very low to moderate, and we often downgraded evidence for risk of bias and imprecision.
We are uncertain of the effect of this intervention on providers' adherence to recommended practice due to the very low certainty evidence (2 studies, 185 participants). The effect of the intervention on patients' and clients' health behaviours such as smoking and treatment adherence is mixed, with substantial variation across outcomes for similar types of behaviour (2 studies, 2262 participants). The intervention probably makes little or no difference to smoking rates among people at risk of cardiovascular disease but probably increases other types of desired behaviour among patients, such as adherence to treatment. The effect of the intervention on patients'/clients' health status and well‐being is also mixed (5 studies, 69,767 participants). It probably makes little or no difference to some types of health outcomes, but we are uncertain about other health outcomes, including maternal and neonatal deaths, due to very low‐certainty evidence. The intervention may slightly improve patient or client acceptability and satisfaction (1 study, 187 participants). We found no studies that reported the time between the presentation of an illness and appropriate management, provider acceptability or satisfaction, resource use, or unintended consequences.
Authors' conclusions
We are uncertain about the effectiveness of mobile phone‐based decision‐support tools on several outcomes, including adherence to recommended practice. None of the studies had a quality of care framework and focused only on specific health areas. We need well‐designed research that takes a systems lens to assess these issues.
Plain language summary
Effect of decision‐support tools on mobile phones on primary health care
What was the aim of this review?
In this Cochrane Review, we aimed to find out if primary (community) healthcare workers using decision‐support tools on mobile phones or other mobile devices give better quality care. We looked for studies where researchers compared a decision‐support tool used on mobile phones to routine practice where there may be no guidance or some guidance in a paper format. We searched for studies conducted from 1 January 2000 to 9 October 2020. We found eight studies.
Key messages
We do not know if decision‐support tools used on mobile devices make primary healthcare workers better at following recommended practice. The evidence is not clear about the effects of these tools on patients' and clients' behaviour and on their health. We need more and better research to assess these issues.
What was studied in the review?
In many settings, patients receive low‐quality care. This is often because they live in poor or rural settings with few healthcare workers, or because healthcare workers do not have enough supplies, equipment, or proper training. Healthcare workers may struggle to stay up‐to‐date or may not have enough time to make the right decisions, which can result in poor quality of care for patients.
Decision‐support tools may help address some of these problems. A decision‐support tool helps the healthcare worker think through what he or she knows about the patient. The tool then helps guide the healthcare worker to the right decision for that patient. Designing decision‐support tools that can be used on mobile phones or other mobile devices such as tablets and personal digital assistants (PDAs) can make these tools easier to use and keep up‐to‐date.
The main aim of our review was to find out if healthcare workers using decision‐support tools on mobile phones give better healthcare than healthcare workers using decision‐support tools that are not on mobile phones or that use no decision‐support tools. We looked at the use of these tools in primary healthcare settings only.
What were the main results of the review?
We found eight relevant studies. Three studies were carried out in the USA and five studies in India, China, Guatemala, Ghana, and Kenya. These studies showed that when primary healthcare workers use decision‐support tools on mobile phones:
– we do not know if they are better at following recommended clinical practice, because the quality of this evidence was very low;
– there was no clear pattern of a positive or negative effect on patients' or clients' behaviour and on their health;
– this may slightly improve patients’ satisfaction with medical information;
– we do not know if this approach led primary healthcare workers to manage people’s health issues more quickly because we found no studies that measured this. We also found no studies that explored the effect on healthcare worker satisfaction, resource use, or whether this approach had any unintended consequences (e.g. harms).
How up‐to‐date is this review?
We searched for studies published up to October 2020.
Summary of findings
Summary of findings 1. Mobile clinical decision‐support system compared to standard care in primary healthcare settings: summary.
Mobile clinical decision‐support system compared to standard care in primary healthcare settings: summary | ||
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers Setting: primary healthcare settings (China, India, Ghana, Guatemala, Kenya, USA) Intervention: mobile clinical decision‐support system Comparison: standard care or no intervention (standard care could be providers using PDA with decision rules about a non‐intervention‐related health area, provider training and decision‐support tools on paper, paper‐based information booklet on management and follow‐up of people with diabetes, or usual care that did not involve any additional follow‐up) | ||
Outcomes | Effects of mobile clinical decision‐support systems (Number of studies, participants)) | Certainty of the evidence (GRADE) |
Providers' adherence to recommended practices, guidelines, or protocol | Uncertain – the certainty of the evidence was very low (2 studies, 185 participants)a,b | ⊕⊝⊝⊝ Very low |
Time between presentation and appropriate management | Uncertain – no direct evidence identified | No evidence |
Patients' or clients' health behaviour | Probably makes little or no difference to the numbers of smokers among people with high cardiovascular disease risk (1 study, 2086 participants)c | ⊕⊕⊕⊝ Moderate |
Probably increases the number of people taking their antihypertensive medication (1 study, 2086 participants)c | ⊕⊕⊕⊝ Moderate | |
May increase the number of people with high cardiovascular disease risk‐taking prescribed aspirin (1 study, 2086 participants)c but may make little or no difference to medication adherence among people with poorly controlled diabetes (1 study, 185 participants)d | ⊕⊕⊝⊝ Low | |
Patients' or clients' health status and well‐being | Probably makes little or no difference to systolic blood pressure among people with high cardiovascular disease risk (1 study, 2086 participants)c or to the number of women giving birth in a hospital (1 study, 799 participants)e | ⊕⊕⊕⊝ Moderate |
May make little or no difference to HbA1c levels among people with poorly controlled diabetes (1 study, 185 participants),d to the number of people with hyperlipidaemia reaching LDL cholesterol goals (1 study, 875 participants)f | ⊕⊕⊝⊝ Low | |
Uncertain of the effect on maternal deaths and neonatal deaths (2 studies, 66,630 participants),e,g and a some other maternal health outcomes (1 study, 799 participants)e– the certainty of evidence was very low | ⊕⊝⊝⊝ Very low | |
Patient or client acceptability and satisfaction | May improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (1 study, 187 participants)d | ⊕⊕⊝⊝ Low |
Providers' acceptability and satisfaction | Uncertain – no direct evidence identified | No evidence |
Resource use | Uncertain – no direct evidence identified | No evidence |
Unintended consequences | Uncertain – no direct evidence identified | No evidence |
HbA1c: glycated haemoglobin; LDL: low‐density lipoprotein; PDA: personal digital assistant. |
aBerner 2006. bGautham 2015. cTian 2015. dHeisler 2014. eMartinez 2018. fEaton 2011. gAmoakoh 2019.
Background
The use of clinical decision‐support tools on mobile devices may help primary healthcare providers, including frontline healthcare workers, to improve the quality of services provided. The provision of appropriate, evidence‐based, quality healthcare services is a concern of global policy makers.
Description of the condition
There is widespread recognition that the quality of healthcare services varies widely, and is often suboptimal across healthcare systems (Moja 2014; WHO 2019; WHO 2020). In primary healthcare, despite the availability of knowledge, and specific diagnostic, treatment, and management protocols, there is often a discrepancy between the knowledge and the application. This 'know‐do' gap in the quality of healthcare has been widely cited as a key barrier to improving healthcare outcomes (Blank 2013; Mohanan 2015). A range of systemic factors contribute to the deficiencies and challenges in delivering high‐quality evidence‐based healthcare services.
Globally, the World Health Organization (WHO) projects a shortfall of 12.9 million healthcare providers by 2035 (WHO 2020). Having the right type of healthcare provider, at the right time, in the right place, continues to be a challenge worldwide. Clustering of health personnel in capital cities and other urban areas, and out‐migration from low‐ and middle‐income countries (LMICs) to high‐income countries (HICs), further diminishes the number of healthcare providers available in rural areas (Dussault 2006). Especially in LMICs, the lack of trained primary healthcare providers has prompted policy makers to explore a shift of key tasks from higher to lower cadres of health workers (Baker 2007; Lehmann 2008; WHO 2016). While the transition of vital primary healthcare services to a lower cadre of workers is feasible, it requires ongoing training support to ensure that service quality and safety standards are maintained (Rednick 2014; WHO 2016).
In other clinical settings, even when an adequate number of providers with the right training are available, the quality of care is variable. Busy healthcare providers may struggle to stay abreast of current evidence and apply it consistently. They may also lack information on alternate risk‐reduction approaches, and be ensure which treatments work best (Kocher 2010; Middleton 2016). Time‐constraints may result in the omission of essential information for counselling the patient, and long work hours may result in increased errors (Bright 2012; Sutton 2020).
Standardized protocols, which provide critical information at the point‐of‐care, support decision‐making and guide healthcare providers through the process of diagnosis and management. They can introduce efficiencies into the system, optimize the time with the client, and improve the overall quality of services (Bright 2012; Mickan 2014; Sutton 2020).
Description of the intervention
Several challenges encountered in clinical practice could benefit from using clinical decision‐support systems (CDSS). One definition of a CDSS is "any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient‐specific assessments or recommendations that are then presented to clinicians for consideration" (Kawamoto 2005). The increasing ubiquity and affordability of smartphones and tablets has made it possible for clinical decision‐support tools to become available to healthcare providers on handheld devices at the point‐of‐care, and broadened the definition to include clinical guidelines and protocols that might be available in a digital format. Mobile decision‐support tools can potentially address some of the challenges faced by many healthcare systems regarding adequate training of healthcare providers, shifting clinical tasks from clinicians to frontline health workers with limited formal training, and improving the quality of preventive, diagnostic, and treatment care across a range of health issues (Carter 2019; Orton 2018).
Since the mid 2000s, personal digital assistants (PDAs), and other wireless mobile devices, such as smartphones and tablets are increasingly being used at the point‐of‐care to obtain evidence and guidance on clinical conditions, do necessary calculations for drugs, and access other medical information (Orton 2018; Richardson 2019; Yau 2019). They also support more advanced CDSSs linked to a comprehensive patient database (Divall 2013). Where before, in most low‐income settings, only basic features, such as voice calls (or interactive voice response (IVR)) and short message service (SMS) were commonplace, the pace of growth in mobile technology increasingly allows for a range of functionality (e.g. low‐cost access to the internet, high‐quality cameras for still and video footage, applications stored on‐device, preloaded audio or video clips and images, global positioning service (GPS), and the potential to connect additional sensors and devices) that allows for the development of very sophisticated point‐of‐care decision‐support systems, even in low‐resource settings (Orton 2018).
Mobile CDSS may vary in the range of functionality and applications to improve diagnoses; facilitate evidence‐based screening, counselling, and treatment; and improve workflow efficiencies.
Broadly, CDSS may serve the following functions:
guide the healthcare provider through process algorithms using 'if…then… rules, based on evidence‐based clinical protocols;
provide the healthcare provider with a checklist, based on clinical protocols;
provide step‐by‐step guidance to screen clients by health status or risk status, possibly using models based on machine learning, where mathematical functions might be used to classify patients into risk groups.
The adoption behaviours and effectiveness of CDSS may vary based on the function it is intended to serve. CDSS may be integrated with electronic health records or stand alone. For the purpose of this review, we focused only on mobile decision‐support tools that were not integrated with an electronic health record or management system, or were integrated with such a system but could be used independent of it.
How the intervention might work
Approaches to using mobile clinical decision‐support tools vary substantially across countries and contexts, in part, depending on the availability of technological and healthcare infrastructure, and the costs of mobile devices and data packages. In addition to the costs and infrastructure, the level of sophistication of a CDSS would also depend on the complexity of the disease, the purpose of the CDSS (e.g. screening alone, or screening integrated with risk assessment and counselling), and the capacity of the healthcare providers to adopt such systems.
At the most basic level, a decision‐support system might comprise of a process algorithm, and transfer a paper‐based protocol into a digital format. For example, a series of digital 'If…then…' logic‐guided questions may be used to assess appropriate contraceptive choices for a client, based on demographic information and preferences. By inputting client data in a systematic way, a decision‐support tool might be used to identify and prioritize clients into risk groups. Additional point‐of care support mechanisms, such as automated algorithmic instructions that prompt healthcare workers to follow certain guidelines, and provide tailored counselling messages and recommendations, might be added to such a system. For example, in addition to the assessment for an appropriate contraceptive, the decision‐support tool may provide a number of recommendations, and prompt the healthcare worker to discuss risks for each contraceptive method, and care and follow‐up for the contraceptive method chosen by the client. At each step, the provider might be required to check off a counselling item that has been discussed, before the system moves to the next set of questions and recommendations. In theory, such a system would promote comprehensive evidence‐based counselling, and improve the overall quality of care provided by the healthcare worker.
In HICs, CDSS is typically used in clinical settings by trained healthcare providers or medical students. In LMICs, decision‐support tools are used for both in‐clinic and community‐based outreach visits, by a range of healthcare providers, including clinicians, midwives, and lay healthcare workers.
Why it is important to do this review
The use of decision‐support tools for clinical decision‐making has been well‐established, and is supported by some emerging evidence. Bright 2012 conducted a systematic review of 148 randomized trials on the effectiveness of CDSS in 2012. They reported that the use of CDSS was associated with significant improvements in a range of morbidity outcomes, healthcare process measures related to performing preventive services, diagnostic testing, and improved adherence to treatment regimens (Bright 2012). However, since the early 2010s, clinical decision‐support tools have transitioned from being operated on stationary computer systems to wireless mobile devices, which provide additional opportunities for point‐of‐care assessments, diagnoses, and management. Furthermore, most healthcare systems in LMICs, especially in rural areas, do not have the required infrastructure for computerized CDSS (Richardson 2019; Yau 2019). The use of these tools on wireless digital devices makes them accessible to healthcare providers in LMICs, which was not possible previously. Despite the substantial investments and global interest in using mobile digital devices to support clinical decision‐making, specific evidence on the effectiveness of such interventions on clinical and public health practice is limited.
Two systematic reviews assessed whether the use of handheld computers, primarily PDAs, improved access to information and supported point‐of‐care clinical decision‐making (Divall 2013; Mickan 2014). Compared to paper resources, the reviews suggested that using handheld computers improved access to information, adherence to clinical guidelines, appropriate diagnostic decision making, and data collection quality (Divall 2013). This review will build on the existing studies, to include the use of other mobile devices, such as smartphones and tablets, which are the most current forms of handheld digital devices, especially in LMICs. One review assessed the feasibility of, and barriers to, using digital point‐of‐care decision‐support tools by healthcare providers in Africa (Adepoju 2017). Based on largely descriptive and observational studies, conducted in seven sub‐Saharan African countries, the review concluded that healthcare providers found mobile decision‐support tools useful; however, they expressed concerns about altered workflows and increased workloads. The review identified technical and infrastructural support, and adequate training, as key barriers to adopting clinical decision‐support tools in the sub‐Saharan African context.
Digital, mobile, wireless technologies provide an innovative and accessible platform to accelerate health services and improve quality of care for some of the most difficult‐to‐reach populations. Given the recent emergence of such technologies for health, there is considerable demand from ministries of health, donors, and decision makers for evidence‐based guidance to invest in such technologies. In response to this global need, the WHO is developing guidelines to inform investments on digital health approaches. This review constitutes one of 11 reviews on the effectiveness of digital health interventions that will be used directly to inform these WHO guidelines (WHO 2019).
Objectives
To assess the effects of digital clinical decision‐support systems (CDSS) accessible via mobile devices by primary healthcare providers in the context of primary care settings.
Methods
Criteria for considering studies for this review
Types of studies
We included randomized controlled trials (RCTs), available as full‐text studies, conference abstracts, and unpublished data. We included studies regardless of their publication status and language of publication.
Types of participants
All cadres of healthcare providers (i.e. professionals, paraprofessionals, and lay health workers) providing healthcare services to patients, using digital, clinical decision‐support tools in the context of a primary care setting.
Other individuals or groups involved in the delivery of primary healthcare services, including administrative staff, managerial, and supervisory staff, who may or may not have been based in a primary healthcare facility or in the community but must have been involved in supporting the delivery of primary healthcare services using digital, clinical decision‐support tools.
Clients or patients receiving care from primary healthcare providers who were using decision‐support tools.
We included participants regardless of their location, professional status, condition, or demographic factors, such as age.
Types of interventions
We included studies that compared digital, clinical decision‐support tools accessible via mobile devices with non‐digital decision‐support tools, or no intervention, in the context of primary care. We included studies in which digital, decision‐support tools were developed for use primarily on a mobile device, and were used by health workers for the purpose of service delivery, to follow clinical protocols, guide service delivery using checklists and job aids, or prioritize clients by risk or other health status in a primary healthcare setting.
Mobile devices were mobile phones of any type (but not analogue landline telephones), as well as tablets, PDAs, and smartphones. We included studies if a mobile device was used, and the tool was intended to be used in a mobile state. For example, if websites or other applications were used, they should have been optimized for use on a mobile device, and healthcare workers should have been trained to use the mobile device. We included studies in which a laptop was used as a tablet, with applications customized for such use.
Primary healthcare services were a combination of the following:
the first contact point of healthcare (Awofeso 2004), including care delivered at an individual level, community level, or both (Muldoon 2006), by individual healthcare providers or teams of providers, and intended to provide and co‐ordinate care in settings where people work and live, or provide continuity of care (Muldoon 2006);
any healthcare that prevented illness, promoted health, was therapeutic, or was rehabilitative (Global Health Watch 2011).
The intervention may have been implemented in public or private healthcare facilities, in the community, or the homes of the patients. We included studies in any country.
The comparisons for this review were:
digital decision‐support tools accessible via mobile device compared to non‐mobile and non‐digital decision‐support tools (e.g. a mobile job‐aid versus a paper job‐aid);
digital decision‐support tools accessible via mobile device compared to standard practice (i.e. non‐digital intervention or no intervention).
We excluded:
studies in which the use of the digital decision‐support tool was dependent on its integration with an electronic medical record or other types of client health‐tracking tools;
studies in which the use of the decision‐support tool was primarily for the purpose of training alone, and did not involve direct service delivery;
studies in which digital decision support was conducted on stationary computers or laptops alone;
studies that compared one type of mobile decision support with another type of mobile decision support;
studies in which patients used the digital decision‐support systems;
pilot and feasibility studies (pilot study was defined as "a version of the main study that is run in miniature to test whether the components of the main study can all work together," and feasibility study was defined as "pieces of research done before a main study" (Arain 2010)).
We included studies in which digital decision‐support tools were delivered as part of a wider package (such as sending messages to the client or provider, supporting the provider in prioritizing clients, etc.), if the decision‐support tool was the major component of the intervention.
Types of outcome measures
Primary outcomes
Providers' adherence to recommended practices, guidelines, or protocols (e.g. providing the service at the recommended time, referral as recommended, screening and prioritizing as recommended).
Time between presentation and appropriate management, including time for referrals and service linkages.
Patients' or clients' health behaviour.
Patients' or clients' health status and well‐being, assessed through validated measures, if available.
Patient or client acceptability and satisfaction with the intervention, assessed through validated measures, if available.
Provider acceptability and satisfaction with the intervention, assessed through validated measures, if available.
Resource use (e.g. human resources and time, training, supplies, and equipment).
Unintended consequences that resulted in an adverse effect of the intervention (these could have included misreading or misinterpretation of the data; transmission of inaccurate data, e.g. incorrect underlying algorithms or clinical protocols; loss of verbal or non‐verbal communication cues; decreased direct engagement with patient; issues of privacy and disclosure; loss (including theft) or misuse of device (in cases in which health workers were given the phones or tablets); interrupted workflow due to infrastructural constraints for battery recharging and network coverage; impacts on equity; disruptions on the delivery of health services, unforeseen ill‐effects on patient outcomes).
Secondary outcomes
None.
Search methods for identification of studies
We restricted the search from 2000. This was based on the increased availability and penetration of mobile devices in LMICs starting in 2000 (International Telecommunications Union 2015).
Electronic searches
An independent information specialist developed the search strategies in consultation with the review authors.
We searched the following databases for primary studies from 2000.
Cochrane Central Register of Controlled Trials (CENTRAL), Issue 9, 2020, the Cochrane Library.
MEDLINE and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily, Ovid (searched 9 October 2020).
Embase 1974 to 2020 week 40, Ovid (searched 9 October 2020).
Global Index Medicus/Global Health Library, WHO (searched 9 October 2020).
POPLINE, K4Health (searched 5 August 2019).
Search strategies were comprised of keywords and controlled vocabulary terms. We applied no limits on language. We used a modified version of the Cochrane Highly Sensitive Search Strategy to identify randomized trials (Lefebvre 2011). All search strategies used are provided in Appendix 1.
Searching other resources
We searched for ongoing trials in the following trial registries:
WHO ICTRP (International Clinical Trials Registry Platform; www.who.int/ictrp) (searched 5 August 2019);
US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov (www.clinicaltrials.gov) (searched 9 October 2020).
We searched Epistemonikos (www.epistemonikos.org) for relevant systematic reviews and potentially eligible primary studies. In addition, the WHO issued a call for papers through popular digital health communities of practice, such as the Global Digital Health Network and Implementing Best Practices, to identify additional primary studies and grey literature.
Grey literature
We searched mhealthevidence.org for grey literature to 2018. The search portal for mhealthevidence.org is more limited, therefore, we reviewed the titles and abstracts of all contributed content not referenced in MEDLINE Ovid. This database was discontinued after 2018.
We reviewed reference lists of all included studies and relevant systematic reviews for additional potentially eligible primary studies. We contacted authors of included studies and reviews to clarify reported published information and to seek unpublished data. We allowed a response time of up to one month from authors who were contacted.
Data collection and analysis
Selection of studies
A core team of two review authors (NH, NM), with assistance where necessary from an additional review author (SA), were responsible for the selection of studies. We downloaded all titles and abstracts retrieved by electronic searching to a reference management database and remove duplicates (DistillerSR). We used a machine learning classifier that is able to assign a probability score that a given record described, or did not describe, a randomized trial. It was built based on 280,000 titles and abstracts from Embase, which have been manually labelled by the Cochrane Crowd (Wallace 2017).
We processed all the search results through the classifier. Two review authors independently screened the titles and abstracts of studies with a 10% probability or greater of being a randomized trial; one review author screened those with a less than 10% probability of being a randomized trial.
We retrieved the full‐text study reports and publications of studies that are screened and included. Two review authors (NH, NM) independently screened the full texts to identify studies to include, and recorded reasons for excluding ineligible studies. We resolved any disagreements through discussion, or if required, we consulted a third review author (SA).
We listed studies that initially appeared to meet the inclusion criteria, but were excluded after full‐text review, in the Characteristics of excluded studies table. We collated multiple reports of the same study, so that each study rather than each report was the unit of interest in the review. We provided any information we could obtain about ongoing studies in the Characteristics of ongoing studies table. We recorded the selection process in sufficient detail to complete a PRISMA flow diagram (Liberati 2009).
Data extraction and management
We modified the Effective Practice and Organisation of Care (EPOC) group standard data collection form and adapted it for our study characteristics and outcome data (EPOC 2017a). We identified key characteristics of the intervention for extraction based on the mHealth Evidence Review and Assessment (mERA) guidelines (Agarwal 2016). We piloted the form on at least one study in the review. Two review authors (NH, NM) independently extracted the following study characteristics from the included studies.
General information: title, reference details, author contact details, publication type, funding source, conflicts of interest of study authors.
Population and setting: country, geographical location (rural, urban, or peri‐urban, defined as outskirts of urban areas), healthcare setting (e.g. facility‐based, home‐based).
Methods: function of the intervention, study design, unit of allocation, duration of participation.
Participant characteristics: type of healthcare worker (function, age, length of training), description of clients serviced by the healthcare worker, description of any other participants in the intervention, withdrawal.
Interventions: intervention purpose, components, type of technology (hardware and software characteristics) and model of delivery, type of mobile device(s) used (smartphone, tablet, feature phone, basic phone, laptops), phone ownership, content of the intervention, health provider training, interoperability, compliance with national guidelines, data security, comparison, fidelity assessment, duration of intervention.
Outcomes: primary and other outcomes specified and collected, time points reported, adverse events, results of any subgroup analyses.
Two review authors (NM, NH) independently extracted outcome data from the included studies. We noted in the Characteristics of included studies table if outcome data were reported in an unusable way. We resolved disagreements by consensus, or by involving a third review author (SA).
Assessment of risk of bias in included studies
Two review authors (NH, NM) independently assessed risk of bias for each included study, using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2017), and the guidance from the EPOC group for assessing randomized trials (EPOC 2017b). We resolved any disagreements by discussion, or by involving a third review author (SA). We assessed the risk of bias according to the following domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, similarity of baseline characteristics, and any other bias. We assessed incomplete outcome data separately for different outcomes.
We judged each potential source of bias as high, low, or unclear and provided a quote from the study report, together with a justification for our judgement in the risk of bias table. We summarized the risk of bias judgments across different studies for each of the domains listed. We considered blinding separately for different key outcomes where necessary. Where information on risk of bias related to unpublished data, or correspondence with a trialist, we noted this in the risk of bias table. We did not exclude studies on the grounds of their risk of bias, but clearly reported the risk of bias when presenting the results of the studies. When considering treatment effects, we took into account the risk of bias for the studies that contributed to that outcome (Guyatt 2008).
We conducted the review according to the published protocol (Agarwal 2018), and reported any deviations from it in the Differences between protocol and review section of the systematic review.
Measures of treatment effect
We analyzed the treatment effects in the individual trials using Review Manager 2014. We estimated the effect of the intervention using risk ratio (RRs) with 95% confidence intervals (CI) for dichotomous data, and mean difference (MD; where studies used the same scale), or standardized mean difference (SMD; when studies used different scales) with 95% CIs for continuous data (Higgins 2017). We ensured that an increase in scores for continuous outcomes could be interpreted in the same way for each outcome, explained the direction to the reader, and reported where the directions were reversed, if this was necessary.
Unit of analysis issues
For cluster‐randomized trials that did not account adequately for the effects of clustering on the effect estimate, we planned to adjust the data prior to entry into the meta‐analysis to avoid unit‐of‐analysis errors. If insufficient information was available to reanalyze the results, we contacted the authors of the primary paper to request the necessary data. If these data were not available, we planned to report the effect estimates without the CIs or P values. However, as no formal meta‐analysis was carried out, due to the heterogeneity across studies reporting similar outcomes, data adjustment from cluster RCTs was not performed.
For cross‐over trials, we prioritised the inclusion of data that was collected before the cross‐over occurred. Where a unidirectional cross‐over occurred (i.e. only one group crossed‐over) this was accounted for in the GRADE rating for each outcome as a potential source of indirectness.
Dealing with missing data
We contacted investigators to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study was identified as abstract only). If this was not possible or we were unable to get in touch with the investigators, we reported the data as missing, noted this in the risk of bias tables, and did not attempt to impute the missing values. There were no adjustments to the analyses to account for adverse events as none of the studies reported adverse events.
Assessment of heterogeneity
We examined heterogeneity by visual inspection of forest plots, as well as using the I² statistic to measure heterogeneity among the trials in each analysis.
Assessment of reporting biases
We planned to assess reporting bias by creating and examining funnel plots if we were able to pool more than 10 trials within a comparison (Sterne 2011). However, only five studies were included, and no pooling was possible.
Data synthesis
Where intervention characteristics and outcome measures were similar across included studies (i.e. if the treatments, participants, and the underlying clinical question were similar enough for pooling to make sense), we planned to conduct a meta‐analysis to estimate an overall effect size. If analyses, adjusted for potential confounders, were reported for either dichotomous or continuous outcomes, we reported these estimates of effect from the primary analysis. In cases where the adjusted analyses for dichotomous outcomes were reported using odds ratios (OR) and not RRs, we planned to convert OR to RR before including the result in a meta‐analysis using the guidance of the Cochrane Handbook for Systematic Reviews of Interventions McKenzie 2021). As no meta‐analysis was performed, we reported adjusted OR directly from the included studies.
We planned to note where reported data were skewed and consider the implication of this; however, there were no skewed data.
The included studies were not similar enough to allow meta‐analysis. Therefore, we reported the results in a narrative format.
Subgroup analysis and investigation of heterogeneity
We had planned to perform subgroup analysis to assess the variation in the delivery of the intervention across different population groups, interventions, or setting characteristics, if possible. We had planned to conduct subgroup analyses only if a sufficient number of trials were available to make statistically significant comparisons between groups. However, we did not identify a sufficient number of trials, and therefore, did not conduct any subgroup analyses.
Sensitivity analysis
We had planned to perform sensitivity analyses to assess the robustness of our conclusions and explore its impact on effect sizes. We had planned to restrict any meta‐analysis to published studies only, and remove studies from any meta‐analyses that had a high risk of bias, based on the risk of bias assessment. However, we did not conduct any sensitivity analyses as there were insufficient studies to conduct meta‐analysis.
Summary of findings and assessment of the certainty of the evidence
Two review authors independently assessed the certainty of the evidence (high, moderate, low, or very low) using the five GRADE considerations (risk of bias, consistency of effect, imprecision, indirectness, and publication bias; Guyatt 2008). We used the methods and recommendations described in the Cochrane Handbook for Systematic Reviews of interventions (Schünemann 2017), and the EPOC worksheets (EPOC 2017c), and used GRADEpro GDT software (GRADEpro GDT). We resolved disagreements on certainty ratings by discussion, and provided justification for decisions to downgrade or upgrade the ratings using footnotes in the table, and made comments to aid readers' understanding of the review, where necessary. We used plain language statements to report these findings in the review (EPOC 2017c).
We created summary of findings tables for the main comparisons for the following outcomes:
providers' adherence to recommended practice, guidelines, or protocols;
time between presentation and appropriate management;
patients' or clients' health behaviour;
patients' or clients' health status and well‐being;
patient or client acceptability and satisfaction with the intervention;
provider acceptability and satisfaction with the intervention;
resource use;
unintended consequences that result in an adverse effect of the intervention.
We drew conclusions about the certainty of the evidence within the text of the review. Outcomes for the main summary of findings tables were selected based on whether the indicators were validated, globally accepted, or considered to be of clinical/public health importance.
Results
Description of studies
We retrieved 7777 unique records for title and abstract screening after removing duplicates, and shortlisted 431 records for full‐text screening. Of these, we identified eight randomized trials that were eligible for this review. We included seven of these in our quantitative analysis.
Results of the search
We described the most pertinent excluded studies in the Characteristics of excluded studies table. There are three studies are awaiting assessment and one ongoing study. The study selection process is summarized in Figure 1.
1.
Study flow diagram.
Included studies
Details of the characteristics of the study population, intervention strategies, and outcomes are presented in the Characteristics of included studies table and Table 2.
1. Description of digital interventions employed by the included studies.
Study ID* | Type of technology | Intervention description | Phone ownership | Provider training | Compliance with guidelines | |
Mobile device | Software | |||||
Amoakoh 2019 | Basic (non‐smart) mobile phone | None | mCDMSI consisted of 4 components: phone calls (voice), text messaging (SMS), access to the internet (data) and access to an USSD that provided protocols for management of obstetric and neonatal emergencies. | Midwives were given individual‐use phones, and health facilities had a shared phone | Health workers were trained before program initiation and once during successive monitoring visits | Ghana's Safe Motherhood Protocol |
Berner 2006 | PDA | Palm Operating System 4.01 | CDSS (MedDecide) on PDA, used to support non‐steroidal anti‐inflammatory drug (NSAIDs)‐related GI risk assessment and treatment recommendations. | PDA provided to physicians | 1 × 30‐minute training session with demonstration and instruction on use of all programs | Based on published evidence‐based literature |
Eaton 2011 | Computer kiosk and PDA | Interactive CDSS software | Interactive decision support software on a PDA using data from patients regarding their risk factors for CHD to improve adherence to guidelines | PDA provided to physicians | 1‐hour academic detailing session + 4 additional sessions on clinical guidelines and use of CDSS tools | National Cholesterol Education Program guidelines |
Gautham 2015 | Smartphones (android, iOS, Symbiam); windows mobile 6.5 cell phone | Gui deVue was used for mobile media‐rich interactive guideline system | Media‐rich, mobile phone‐based clinical guidance system for management of fevers, diarrhoeas, and respiratory problems. | Unclear | 2‐day training programme in guideline‐based care and in use of the mHealth system | WHO Integrated Management of Childhood and Adult illnesses |
Heisler 2014 | Tablets with 3G access | iDecide | Personally tailored, interactive diabetes medication decision aid designed for CHWs to deliver information and treatment to individuals with diabetes and low health literacy. | Tablets were provided | 80 hours of initial training in motivational interviewing‐based communication and diabetes self‐management support, with 4–8 hours of booster training annually | AHRQ Guides ("Pills for Type 2 Diabetes" and "Premixed Insulin for Type 2 Diabetes") |
Keane 2018 | Smartphones (android, iOS, Symbiam, tablets | Commcare (IMAM app) | App provided step‐by‐step guidance on assessment, treatment, and referral of children for malnutrition. | Smartphones were provided | 3‐day training on the use of the tablets/phones | Not reported |
Martinez 2018 | Samsung S3 mini smartphone (android) with peripheral sensor devices | Customized android‐based app | The app allowed collection of maternal and perinatal symptoms and clinical signs, maternal vital signs and the fetal heart rate. Using these data, TBAs were guided through detection of complications and refer pregnant women to health facilities. | Smartphones were provided | 4‐day training led by study nurses to review medical concepts on perinatal complications and use of smart phone | Not reported |
Tian 2015 | Smartphones (android, iOS, Symbiam) | Customized android‐based app | The decision‐support component assisted the CHWs on the cardiovascular disease management of their patients based on patients’ medical history, lifestyle, blood pressure and medication. | Smartphones were provided | Initial 1‐day systematic training. Refresher training every 3–4 months during the implementation | Simplified international and national guidelines on cardiovascular disease management |
* None of the studies reported on data security and interoperability |
CDSS: clinical decision‐support system; PDA: personal digital assistant; WHO: World Health Organization.
Locations and populations
We identified eight randomized trials that fulfilled the inclusion criteria (Amoakoh 2019; Berner 2006; Eaton 2011; Gautham 2015; Heisler 2014; Keane 2018; Martinez 2018; Tian 2015). Three trials were conducted in the USA (Berner 2006; Eaton 2011; Heisler 2014), with two of the three conducted in urban clinics (Berner 2006; Heisler 2014). Three studies were conducted in rural areas: one in India (Gautham 2015), one in China and India (Tian 2015), and one in Guatemala (Martinez 2018). Two studies were conducted in Ghana (Amoakoh 2019), and Kenya (Keane 2018). Of the three studies in the USA, one was conducted with internal medicine residents and standardized patients trained to convey specific health conditions, in a university outpatient setting (Berner 2006); and two were conducted in primary care settings: one with community health workers in Detroit (Heisler 2014), and one with physicians in New England (Eaton 2011).
The five studies from LMICs enrolled a range of healthcare providers to deliver the CDSS intervention: community health workers (Tian 2015), male and female rural health providers with varying levels of training and experience (Gautham 2015), traditional birth attendants (TBA; Martinez 2018), and primary health facility‐based providers (Amoakoh 2019; Keane 2018).
Intervention strategies
The interventions aimed to improve provider adherence to the treatment protocol across a range of health areas: integrated management of childhood and adult illnesses for management of fevers (IMCI/IMAI), diarrhoea and respiratory problems; risk assessment, counselling, and treatment of non‐communicable diseases including diabetes management, and cardiovascular disease; and gastrointestinal (GI) risk assessment.
The CDSS tool tested by Gautham 2015 used mobile media‐rich interactive guidelines (mMRIGs) with audio, images, and video to support protocol compliance for IMCI/IMAI by rural health providers.
In Eaton 2011, physicians received a PDA‐based decision‐support tool to improve adherence to National Cholesterol Education Program Guidelines. The CDSS algorithm helped determine the patient's lipid diagnosis (low‐density lipoprotein (LDL) dominant, isolated low high‐density lipoprotein (HDL) level, triglyceride dominant, mixed lipid disorder, and atherogenic dyslipidaemia), calculated the LDL and non‐HDL cholesterol goals (when appropriate), made recommendations regarding therapeutic lifestyle management, provided optimal dosage of lipid‐lowering drugs tailored to the patient's risk factor status, and provided an interactive shared decision‐making page for physicians to discuss lowering lipid values, and other coronary heart disease (CHD) risk factor management.
Another intervention targeted at cardiovascular health was tested in community‐based settings in India and China (Tian 2015).
Community health workers used a smartphone‐based android application consisting of prompts to gather data on patient's medical history, new symptoms and diagnoses, medication usage, current lifestyle habits, and blood pressure, and provide guidance for prescribing any of the target antihypertensive medications. iDecide is a personally tailored, interactive diabetes medication decision aid for use by community health workers in a community health centre has four core components (Heisler 2014): 1. animations on the effect of diabetes on how glucose is processed in the body and how different medication classes, foods, and physical activity affect blood sugar; 2. pictographs showing participants' own risk of diabetes complications (tailored based upon their baseline glycated haemoglobin (HbA1c)); 3. participants review their current diabetes medications and barriers to taking medications; 4. prompts participants to set goals and develop specific action plans.
Berner 2006 examined the effect of a personal digital assistant‐based CDSS tool, used to support non‐steroidal anti‐inflammatory drug (NSAIDs)‐related GI risk assessment and treatment recommendations. All participants received some rules on a PDA that could potentially apply to the standardized patients' complaints; however, only the 34 participants who were randomized to the intervention arm received the rules for GI risk assessment when prescribing NSAIDs.
Three studies targeted interventions at improving maternal and neonatal outcomes.
Martinez 2018 provided TBAs in Guatemala with an android‐based platform to collect demographic data, maternal and perinatal symptoms, and maternal vital signs using a range of peripheral sensor devices (pulse oximeter, hand‐held one‐dimensional Doppler ultrasound, self‐inflating oscillometric blood pressure cuff). Based on the symptoms, TBAs could check a list of common maternal and perinatal complications presented as pictures, which then triggered automatic communication with the on‐call clinical team by voice call or a text message.
In Keane 2018, facility workers were provided with a smartphone/tablet‐based app that provided step‐by‐step guidance on assessment, treatment, and referral of children for malnutrition. The app provided treatment protocols, counselling messages, and calculated z‐scores for assessment of malnutrition and provision of ready‐to‐use therapeutic food sachets. The data were uploaded to a cloud server where they could be used for program management.
Amoakoh 2019 used a four‐component decision‐support intervention in Ghana comprised of phone calls, text messages, access to data services, and to unstructured supplementary service data (USSD). The USSD allowed health workers to access obstetric and neonatal clinical management protocols, and for two‐way communication between frontline midwives and health facility health workers. Health workers also received monthly reminders about the availability of these protocols.
Outcomes
Outcomes were reported as follows.
Providers' adherence to recommended practices, guidelines, or protocols
Two trials reported this outcome: Berner 2006 reported the proportion of cases per physician with unsafe practices, and proportion of cases per physician with key GI risk factors recorded. Gautham 2015 reported protocol compliance for diagnoses and management of adult and child fevers, diarrhoea, and respiratory problems by rural healthcare providers.
Time between presentation and appropriate management
We found no studies that reported time between presentation and appropriate management.
Patients' and clients' health behaviour
Heisler 2014 reported antihyperglycaemic medication decisional conflict (at three months of follow‐up), diabetes self‐care efficacy, change in HbA1c, and medication adherence on a scale of 1 to 100 over three months of follow‐up. Tian 2015 reported health behaviours associated with cardiovascular health‐proportion of current smokers (at one year of follow‐up), high‐risk people taking aspirin in the past month, and self‐reported use of antihypertensive medication for 25 or more days in the past month.
Patients' or clients' health status and well‐being
Six trials reported this outcome. Eaton 2011 reported the proportion of patients reaching LDL‐cholesterol goals and non‐HDL‐cholesterol goals at one year of follow‐up after the intervention. These outcomes were presented by low, medium, and high cardiovascular risk status; however, the data were not presented in a way that could be analyzed. Heisler 2014 reported diabetes distress at three months of follow‐up. Tian 2015 reported health outcomes associated with cardiovascular health (i.e. change in mean systolic pressure, proportion of high‐risk people hospitalized in the last year). These outcomes were measured with one year of follow‐up after the intervention. Martinez 2018 reported several maternal health outcomes including monthly emergency facility referral rate by type of maternal/perinatal complication (labour progression abnormality, hypertensive disorder of pregnancy, haemorrhage, premature labour, fetal cardiac abnormality, suspected neonatal sepsis, neonatal respiratory compromise, preterm newborn), and number of maternal and neonatal deaths. Amoakoh 2019 reported the total number of neonatal deaths (death occurring from birth to 28 days) from all deliveries over an 18‐month period, and the total number of maternal deaths among all antenatal care attendants at health facilities. These outcome definitions differed from globally accepted definitions of neonatal mortality rate (the number of neonatal deaths per 1000 live births) and maternal mortality rate (the number of maternal deaths in a given period per 100,000 women of reproductive age) or maternal mortality ratio (number of maternal deaths per 100,000 live births). Keane 2018 measured the number of neonatal deaths, maternal deaths, and the proportion of children who were cured of malnutrition, defaulted treatment, or died. The data were unclearly reported and, therefore, not included in the analysis.
Patients' or clients' acceptability and satisfaction
Heisler 2014 measured satisfaction with clarity of medication information and satisfaction with helpfulness of medication information.
Provider acceptability and satisfaction
One study reported this outcome (Gautham 2015). However, the outcome was reported for the intervention arm only. Provider acceptability and satisfaction was measured through a series of questions on the level of comfort with the system, willingness to continue using the system, being able to remember the steps without the system, usefulness of combination of media, willingness to recommend the system, wish for more health conditions to be included in the system, ease of use of the system, and helpfulness of the system in following guidelines.
Resource use
No studies reported resource use.
Unintended consequences
No studies reported unintended consequences.
Excluded studies
We excluded 431 full‐text articles with reasons: 68 did not meet the study design criteria; interventions in 124 articles did not meet the definition for a CDSS; 63 studies were not conducted in a primary care setting; interventions in 86 studies were also integrated with an electronic medical record; 20 intervention studies did not involve direct patient care; interventions in 12 studies were not accessible primarily by a mobile phone; in 10 studies the comparison arm did not meet inclusion criteria; in 30 studies the intervention was not used by a healthcare provider; and in five studies the digital intervention was not core to the main intervention.
We described the most pertinent excluded studies in the Characteristics of excluded studies table.
Studies awaiting assessment
Three studies are awaiting assessment (Keitel 2017; Khan 2020; de Molina‐Férnandez 2019; Studies awaiting classification table).
Ongoing studies
We found one ongoing study (NCT03311399; Characteristics of ongoing studies table).
Risk of bias in included studies
We used Cochrane's tool for assessing the risk of bias in each individual study (presented in Figure 2 and Figure 3), which are summarized in the Characteristics of included studies table.
2.
Risk of bias graph: review authors' judgements about each risk of bias item presented as percentages across all included studies.
3.
Risk of bias summary: review authors' judgements about each risk of bias item for each included study.
Allocation
Random sequence generation (selection bias)
Four studies had adequate sequence generation methods where participants were randomly assigned to the intervention or control group using a computerized random number generator or algorithm (Amoakoh 2019; Berner 2006; Heisler 2014; Tian 2015). Four studies had unclear risk (Eaton 2011; Gautham 2015; Keane 2018; Martinez 2018).
Allocation concealment (selection bias)
Three studies specified methods for allocation concealment where an independent study staff generated the randomization pattern (Amoakoh 2019; Martinez 2018; Tian 2015). Five studies did not describe methods for allocation concealment (Berner 2006; Eaton 2011; Gautham 2015; Heisler 2014; Keane 2018).
Blinding
In six studies, there was no blinding of participants to the intervention as the intervention required overt interaction with participants (i.e. healthcare providers in the intervention group were given a mobile device with a specific program, and the comparison group had usual care which did not involve the use of a mobile device) (Amoakoh 2019; Eaton 2011; Gautham 2015; Heisler 2014; Martinez 2018; Tian 2015). Two studies made no clear mention about blinding of participants and personnel to the intervention (Berner 2006; Keane 2018).
Two studies had adequate methods for blinding of outcome assessment where data assessors were blinded to the allocation throughout the study (Berner 2006; Heisler 2014). Three studies had high risk of detection bias as adherence to treatment protocols by providers was assessed through direct observation of the study participants, or no clear protocol for outcome blinding was implemented (Eaton 2011; Gautham 2015; Martinez 2018). Three studies had unclear risk of detection bias (Amoakoh 2019; Keane 2018; Tian 2015).
Incomplete outcome data
Five studies reported low attrition, where the reasons for attrition seemed unrelated to the outcomes (Berner 2006), losses were balanced across groups (Eaton 2011; Heisler 2014), or where the analysis included all the participants who were randomized to the intervention (Martinez 2018; Tian 2015). Three studies had unclear reporting on attrition (Amoakoh 2019; Gautham 2015; Keane 2018).
Selective reporting
Three studies had adequate reporting of outcomes where all outcomes were reported (Berner 2006; Eaton 2011; Tian 2015), three studies had a high risk of reporting bias (Amoakoh 2019; Gautham 2015; Heisler 2014), and two studies had an unclear risk of selective reporting (Keane 2018; Martinez 2018).
Other potential sources of bias
In Berner 2006 and Keane 2018, we were uncertain of additional sources of bias, as reporting was insufficient to assess this. In Gautham 2015, the authors acknowledged that there may have been contamination between the groups as the participants had opportunities to work together. One of the authors of the Gautham 2015 study declared a conflict of interest resulting from part ownership in a private company for commercialization of products used in the study. In Heisler 2014, there were some baseline imbalances between treatment and comparison groups. More participants in the treatment group had completed high school, were less likely to have difficulty with written healthcare information, and were more confident in filling out medical paperwork. However, the authors accounted for these imbalances in the analyses. In Amoakoh 2019, baseline differences in neonatal mortality risk factors were unaccounted for between the study arms. The authors also reported contamination across the intervention and comparison areas due to referral of women in comparison areas to intervention area health facilities.
Effects of interventions
See: Table 1
See: Table 1; Table 3; Table 4.
2. Summary of findings 2: mobile clinical decision‐support system compared to standard care in primary healthcare settings: details I.
Mobile clinical decision‐support system compared to standard care in primary healthcare settings: details I | |||||||
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers Setting: primary healthcare settings (China, Ghana, Guatemala, India, Kenya, USA) Intervention: mobile clinical decision‐support system Comparison: standard care or no intervention (providers using PDA with decision rules about a non‐intervention‐related health area; provider training and decision‐support tools on paper; paper‐based information booklet on management and follow‐up of people with diabetes; or usual care that did not involve any additional follow‐up) | |||||||
Outcomes | Anticipated absolute effects (95% CI) | Relative effect (95% CI) | № of participants (studies) | Certainty of the evidence (GRADE) | What happens? | ||
Standard care | Mobile clinical decision‐support system | ||||||
Providers' adherence to recommended practices, guidelines, or protocols | |||||||
Providers' adherence to recommended practices | 1 study assessed a digital decision‐support tool for non‐steroidal anti‐inflammatory drug prescribing safety. The mean proportion of providers with unsafe prescriptions was 0.23 in the intervention group and 0.45 in the comparison group. The proportion of providers following recommended practice was 0.58 in the intervention group and 0.45 in the comparison group. 1 study assessed a digital decision‐support tool for management of fevers, diarrhoeas and respiratory problems by rural providers. For female patients, mean protocol compliance was 63.34% in the intervention group and 69% in the comparison group. For male patients, compliance was 53.59% in the intervention group and 71.12% in the comparison group. |
— | 185 (2 RCTs) India, USA (Berner 2006; Gautham 2015) |
⊕⊝⊝⊝ Very lowa,b | We are uncertain of the effect of this approach on providers’ adherence to recommended clinical practice because the certainty of this evidence was very low. (Both studies reported incomplete data.) |
||
Time between presentation and appropriate management | |||||||
Time between presentation and appropriate management | No studies reported this outcome | — | — | — | |||
Patients' or clients' acceptability and satisfaction | |||||||
Client satisfaction with clarity of medication information (1–100 scale, where higher is better) |
Mean satisfaction was 82.6 points on a 1–100 scale | Mean satisfaction was 9.2 points higher (95% CI 0.97 higher to 17.43 higher) on a 1–100 scale | — | 187
(1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowc,d | This approach may improve the satisfaction with clarity of medication information among people with poorly controlled diabetes. | |
Client satisfaction with helpfulness of medication information (1–100 scale, where higher is better) |
Mean satisfaction was 87.6 points on a 1–100 scale | Mean satisfaction was 11.3 higher (95% CI 3.28 higher to 19.32 higher) | — | 187
(1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowc,d | This approach may improve the satisfaction with helpfulness of medication information among people with poorly controlled diabetes. | |
Provider acceptability and satisfaction | |||||||
Providers' acceptability/satisfaction | No studies reported this outcome | — | — | — | |||
Resource use | |||||||
Resource use | No studies reported this outcome | — | — | — | |||
Unintended consequences | |||||||
Unintended consequences | No studies reported this outcome | — | — | — | |||
CI: confidence interval; LDL: low‐density lipoprotein; MD: mean difference; OR: odds ratio; RCT: randomized controlled trial; RR: risk ratio; SD: standard deviation. | |||||||
GRADE Working Group grades of evidence High certainty: we are very confident that the true effect lies close to that of the estimate of the effect. Moderate certainty: we are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. Low certainty: our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect. Very low certainty: we have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect. |
aDowngraded one level due to risk of bias: unclear random sequence generation and allocation concealment. Blinding of participants was not possible given the intervention. bDowngraded two levels due to very serious imprecision: standard errors or confidence intervals for the outcomes were not reported. cDowngraded one level due to risk of bias: unclear allocation concealment and selective outcome reporting. dDowngraded one level due to imprecision due to small sample size.
3. Summary of findings 2: mobile clinical decision‐support system compared to standard care in primary healthcare settings: details II.
Mobile clinical decision‐support system compared to standard care in primary healthcare settings: details II | ||||||
Patient or population: healthcare providers using clinical decision‐support tools and patients receiving care from such providers Setting: primary healthcare settings (China, Ghana, Guatemala, India, Kenya, USA) Intervention: mobile clinical decision‐support system Comparison: standard care or no intervention (providers using PDA with decision rules about a non‐intervention‐related health area; provider training and decision‐support tools on paper; paper‐based information booklet on management and follow‐up of people with diabetes; or usual care that did not involve any additional follow‐up) | ||||||
Outcomes | Anticipated absolute effects (95% CI) | Relative effect (95% CI) | № of participants (studies) | Certainty of the evidence (GRADE) | What happens | |
Standard care | Mobile clinical decision‐support system | |||||
Patients' or clients' health behaviour | ||||||
Antihyperglycaemic medication decisional conflict at 3 months of follow‐up | Mean 72.3 (SD 13.2) | Mean 70.9 (SD 13.7) | MD 2.60 (−6.76 to 2.16) | 176 (1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowa,b | The intervention may make little or no difference to decisional conflict.c |
Diabetes care self‐efficacy | Mean 80 (SD 16.6) | Mean 83.3 (SD 19.5) | MD 3.30 (−0.95 to 7.55) | 176 (1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowa,b | The intervention may make little or no difference to diabetes care self‐efficacy.c |
Adherence – high‐risk people taking aspirin in the last month at 1 year of follow‐up | 22 per 1000 | 206 per 1000 (95% CI 134 to 317) | RR 9.30 (6.05 to 14.28) | 2086
(1 RCT) India and China (Tian 2015) |
⊕⊕⊝⊝ Lowa,d | This approach may increase the number of people with high cardiovascular disease risk taking their aspirin. |
Adherence – self‐reported use of community healthcare workers prescribed antihypertensive medication for ≥ 25 days in the past month at 1 year of follow‐up | 94 per 1000 | 362 per 1000 (95% CI 295 to 447) | RR 3.86 (3.14 to 4.76) | 2086
(1 RCT) India and China (Tian 2015) |
⊕⊕⊕⊝ Moderated | This approach probably increases the number of people taking their antihypertensive medication. |
Adherence – medication adherence at 3 months of follow‐up (1–100 scale, where higher is better) |
Mean medication adherence was 90.5 points on a 1–100 scale | Mean medication adherence was 2.3 points lower (95% CI 6.76 lower to 2.16 higher) on a 1–100 scale | — | 176
(1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowa,d | This approach may make little or no difference to medication adherence among people with poorly controlled diabetes.c |
Current smoker at 1 year of follow‐up | 363 per 1000 | 374 per 1000 (95% CI 334 to 421) | RR 1.03 (0.92 to 1.16) | 2086
(1 RCT) India and China (Tian 2015) |
⊕⊕⊕⊝ Moderated | This approach probably makes little or no difference to the number of smokers among people with high cardiovascular disease risk. |
Patients'/clients' health status and well‐being | ||||||
Diabetes distress at 3 months of follow‐up | Mean 66.6 (SD 30.7) | Mean 76.9 (SD 22.3) | MD 15.7 (8.24 to 23.16) | 176 (1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowa,b | The intervention may have a small positive effect on diabetes distress.c |
Number of women who had a caesarean section | 27 per 374 | 38 per 425 | RR 1.24 (0.77 to 1.99) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to abnormal progression of labour | 36 per 374 | 51 per 425 | RR 1.25 (0.83 to 1.87) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to a hypertensive disorder | 4 per 374 | 15 per 425 | RR 3.3 (1.1 to 9.86) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to haemorrhage | 10 per 374 | 9 per 425 | RR 0.79 (0.33 to 1.93) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to premature labour | 6 per 374 | 5 per 425 | RR 0.73 (0.23 to 2.38) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to fetal cardiac abnormality | 4 per 374 | 4 per 425 | RR 0.88 (0.22 to 3.49) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to suspected sepsis | 5 per 374 | 4 per 425 | RR 0.70 (0.19 to 2.60) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to respiratory compromise | 2 per 374 | 5 per 425 | RR 2.20 (0.43 to 11.27) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Number of women who had an emergency referral due to a premature newborn | 0 per 374 | 4 per 425 | RR 7.92 (0.43 to 146.67) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowe,f,m | We are uncertain about the effects on this outcome because the certainty of this evidence was very low. |
Haemoglobin a1c (HbA1c) (controlled HbA1c is typically < 7.5 or 7 (depending on risk factors)) |
Mean HbA1c was 7.9% | Mean HbA1c was 0.1% lower (95% CI 0.3 lower to 0.18 higher) | — | 176 (1 RCT) USA (Heisler 2014) |
⊕⊕⊝⊝ Lowa,b | This approach may make little or no difference to HbA1c levels among people with poorly controlled diabetes. |
Mean systolic blood pressure (target systolic blood pressure is typically < 140 mmHg) |
Mean systolic blood pressure was 152.3 mmHg | Mean systolic blood pressure was 2.8 mmHg lower (95% CI 5.09 lower to 0.51 lower) | — | 2086
(1 RCT) India and China (Tian 2015) |
⊕⊕⊕⊝ Moderateb | This approach probably makes little or no difference to the systolic blood pressure among people with high cardiovascular disease risk. |
People reaching LDL‐cholesterol goal | 74% | 74% | — | 875
(1 RCT) USA (Eaton 2011) |
⊕⊕⊝⊝ Lowg,h | This approach may make little or no difference to the number of people with hyperlipidaemia reaching LDL‐cholesterol goals. |
Maternal deaths | 2.6 per 1000 |
4.6 per1000 (0 to 50) |
RR 1.76 (0.16 to 19.33 | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊝⊝⊝ Very lowi,j,m | We are uncertain about the effects of this approach on maternal deaths because the certainty of this evidence was very low. |
Hospital delivery at 12 months of follow‐up | 77 per 1000 |
98 per 1000 (62 to 154) |
RR 1.27 (0.81 to 2.00) | 799 (1 RCT) Guatemala (Martinez 2018) |
⊕⊕⊝⊝ Lowa,i,m | This approach may increase the number of hospital deliveries, but the 95% CI included both a decrease and an increase in hospital deliveries. |
Neonatal deaths | 1 study reported no difference between the intervention and comparison groups in neonatal deaths (OR 6.25, 95% CI 0.76 to 51). 1 study reported increased odds of neonatal deaths in the intervention group compared to the comparison group (OR 2.09, 95% CI 1.00 to 4.37) |
— | 66,630 (2 RCTs) Guatemala (Martinez 2018), Ghana (Amoakoh 2019) |
⊕⊝⊝⊝ Very lowk,l | We are uncertain about the effects of this approach on neonatal deaths because the certainty of this evidence was very low. Results could not be pooled as only 1 of the 2 studies presented estimates adjusted for clustering. | |
CI: confidence interval; HbA1c: glycated haemoglobin; LDL: low‐density lipoprotein; MD: mean difference; PDA: personal digital assistant; RCT: randomized controlled trial; RR: risk ratio; SD: standard deviation. | ||||||
GRADE Working Group grades of evidence High certainty: we are very confident that the true effect lies close to that of the estimate of the effect. Moderate certainty: we are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different. Low certainty: our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect. Very low certainty: we have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect. |
aDowngraded one level due to imprecision due to few events. bDowngraded one level due to risk of bias: unclear allocation concealment and selective outcome reporting. cOutcome measures scaled from 0 to 100 with higher numbers indicating more positive outcomes. dDowngraded one level due to risk of bias due to lack of blinding of participants and unclear blinding of outcome assessment. eDowngraded one level due to lack of blinding of participants and outcomes, and unclear random sequence generation and selective reporting. fDowngraded one level due to imprecision: very few events. gDowngraded one level due to risk of bias: unclear random sequence generation and allocation concealment; participants, providers, and outcome assessors not blinded. hDowngraded one level due to imprecision: unclear number of participants in either group for analysis, only proportions reported. iDowngraded one level due to lack of blinding of participants, personnel, and outcomes data; and unclear sequence generation and reporting bias. jDowngraded two levels due to very serious imprecision due to few events and wide confidence intervals that encompassed both a large benefit and a large harm associated with the intervention. kDowngraded two levels due to very serious risk of bias: both studies had lack of blinding of participants, personnel, and outcomes data; and one study had unclear sequence generation and reporting bias. lDowngraded one level due to spurious effects from confounding as one study reported potential differences between groups on baseline neonatal mortality risk factors.
mDowngraded one level due to indirectness: potential for contamination as the control group providers also received 5 months of intervention.
Mobile clinical decision‐support system compared to standard care in primary healthcare settings
Providers' adherence to recommended practices, guidelines, or protocols
We are uncertain of the effect of this intervention on providers' adherence to recommended practice (very low‐certainty evidence).
Two studies reported this outcome, measured as proportion of providers with unsafe practices, and proportion of providers who were compliant.Because of variations in outcome measurement across the studies, as well as incomplete reporting of data, meta‐analyses of these results was not feasible.One study assessed effectiveness of providers' use of a digital decision‐support tool on NSAID prescribing safety (Berner 2006). It reported the mean proportion of providers with unsafe prescriptions at 0.23 (31 providers) for the intervention group and 0.45 (28 providers) for the comparison group. The proportion of providers following recommended practice was 0.58 (31 participants) for the intervention group and 0.45 (28 participants) for the comparison group. Another study assessed the use of a digital decision‐support tool for management of fevers, diarrhoeas, and respiratory problems by rural providers (Gautham 2015). For female patients, mean protocol compliance was 63.34% (8 providers, 38 female patients) for the intervention group and 69% (8 providers, 43 female patients), and for male patients, mean protocol compliance was 53.59% (8 providers, 27 male patients) for the intervention group and 71.12% (8 providers, 18 male patients).
Time between presentation and appropriate management
No studies reported time between presentation and appropriate management.
Patients' or clients' health behaviour
The intervention probably makes little or no difference to the numbers of smokers among people with high cardiovascular disease risk (moderate‐certainty evidence), probably increases the number of people taking their antihypertensive medication (moderate‐certainty evidence), and may increase the number of people with high cardiovascular disease risk taking prescribed aspirin. It may make little or no difference to medication adherence among people with poorly controlled diabetes (low‐certainty evidence).
Use of mobile‐based clinical decision‐support tools probably leads to little or no change in proportion of clients who reporting being current smokers at one‐year follow‐up (RR 1.03, 95% CI 0.92 to 1.16; moderate‐certainty evidence) (Tian 2015). At one‐year follow‐up, use of the tools by community health workers for counselling may increase the number of high‐risk individuals taking preventive aspirin in the last month (RR 9.30, 95% CI 6.05 to 14.28; low‐certainty evidence) and probably increases the number of people using of antihypertensive medication for 20 days or more in the past month (RR 3.86, 95% CI 3.14 to 4.76; moderate‐certainty evidence) (Tian 2015). Contrary to these findings, the effect of clinical decision‐support tools by providers may makelittle or no difference to medication adherence among people with poorly controlled diabetes at three months of follow‐up. The MD in medication adherence was 2.3 points lower in the intervention group (95% CI −6.76 to 2.16; low‐certainty evidence) (Heisler 2014). These differences in the findings of the studies may be explained by differences in theirsettings: in Tian 2015, lay health workers delivered the intervention in a community‐based setting in India and China and, in Heisler 2014, trained physicians delivered the intervention in a primary care facility in the USA.
Patients' or clients' health status and well‐being
The use of mobile‐based clinical decision‐support tools probably makes little or no difference to systolic blood pressure level among people with high cardiovascular disease risk or to the number of women giving birth in a hospital (moderate‐certainty evidence). The intervention may make little or no difference to HbA1c levels among people with poorly controlled diabetes, or to the number of people with hyperlipidaemia reaching LDL‐cholesterol goals (low‐certainty evidence). . We are uncertain of the effect on maternal deaths and neonatal deaths, and a number of other maternal health outcomes (very low‐certainty evidence).
Across the studies that reported the effect of mobile‐based clinical decision‐support tools by providers on health outcomes, these may make little or no difference to HbA1c, mean systolic blood pressure, hospital delivery over the 12‐month study period, and maternal death (HbA1c: −0.1%, 95% CI −0.3 to 0.18; low‐certainty evidence; Heisler 2014; mean systolic blood pressure: −2.8 mmHg, 95% CI −5.09 to 0.51; moderate‐certainty evidence; Tian 2015; hospital delivery over the 12‐month study period: RR 1.27, 95% CI 0.81 to 2.00; low‐certainty evidence; Eaton 2011; maternal death: RR 1.76, 95% CI 0.16 to 19.33; very low‐certainty evidence; Martinez 2018). Two studies reported the effects of mobile‐based clinical decision‐support tools on neonatal deaths but we are uncertain about these effects because the certainty of this evidence was very low (Amoakoh 2019; Martinez 2018). Martinez 2018 reported that there may be no difference in neonatal deaths between groups (OR 6.25, 95% CI 0.76 to 51.00), whereas Amoakoh 2019 reported that there may be an increase in neonatal deaths in the intervention group (OR 2.09, 95% CI 1.00 to 4.37) (very low‐certainty evidence). One study reported that there may be little or difference in the proportion of patients who achieved LDL‐ cholesterol goals after one year of intervention (low‐certainty evidence) (Eaton 2011).
The studies reported several other patient/client health outcomes (see additional summary of findings table in Table 4). Heisler 2014 reported that the intervention may have a small positive effect on diabetes distress at three months of follow‐up (MD 15.7, 95% CI 8.24 to 23.16;) ( low‐certainty evidence). Martinez 2018 reported the effect of the intervention on the number of women who had a caesarean section, and who had an emergency referral due to the following conditions: abnormal progression of labour, hypertensive disorders, haemorrhage, premature labour, fetal cardiac abnormality, suspected neonatal sepsis, respiratory compromise, and premature newborn.
Patient or client acceptability and satisfaction with the intervention
The intervention may improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (low‐certainty evidence). Client satisfaction with health information provided by the healthcare provider was measured on a scale of 1 to 100. Heisler 2014 reported that there may be a 9.2‐point increase (95% CI 0.97 to 17.43) in client satisfaction with clarity of medication information, and an 11.3‐point increase (95% CI 3.28 to 19.32) in client satisfaction with helpfulness of medication information.
Provider acceptability and satisfaction with the intervention
No studies reported providers' acceptability and satisfaction with the intervention. Tian 2015 included this outcome but reported provider acceptability of the intervention for the intervention group only.
Resource use
No studies reported resource use (e.g. human resources and time, training, supplies, and equipment).
Unintended consequences
No studies reported unintended consequences.
Discussion
Summary of main results
Eight RCTs met our inclusion criteria. Three trials were conducted in the USA. The remaining trials were conducted in India (one trial), India and China (one), Guatemala (one), Ghana (one), and Kenya (one). We are uncertain of the effect of this intervention on providers’ adherence to recommended practice due to the very low certainty of evidence (2 studies, 185 participants). The effect of the intervention on patients' and clients' health behaviour was mixed. It probably makes little or no difference to some types of behaviour but probably increases other types of desired behaviour (2 studies, 2262 participants) There is large heterogeneity in the evidence for similar types of behaviour. For example, the evidence suggests improvements in 'adherence to medications' for certain conditions, and not for other conditions.
The effect of the intervention on patients'/clients' health status and well‐being was also mixed (5 studies, 69,767 participants). It probably makes little or no difference to some types of health outcomes, but we are uncertain about other health outcomes, including maternal and neonatal deaths, due to very low‐certainty evidence. The intervention may improve satisfaction with the clarity or helpfulness of medication information among people with poorly controlled diabetes (1 study, 187 participants). Most of this evidence comes from single studies, some of which had few participants. No studies reported the time between presentation of an illness and appropriate management, provider acceptability or satisfaction, resource use, or unintended consequences.
Overall completeness and applicability of evidence
None of the included studies evaluated the impact of the intervention on the time between presentation of an illness and appropriate management, provider acceptability/satisfaction, resource use, or unintended consequences. For outcomes that were reported, the studies were often small and few in number.
The interventions in the included studies were heterogeneous in nature. The interventions were implemented across HICs and low‐income countries; in facility‐ and community‐based contexts; and by providers with varying degrees of training and experience including community health workers, internal medicine residents, and primary care physicians. Additionally, across the eight studies, the CDSS interventions had a range of functionalities applied across specific health areas including antenatal and postnatal care counselling, cardiovascular risk assessment and treatment, IMCI/IMAI and adult illnesses, and counselling and support for people with diabetes. None of the studies looked at the use of CDSS across a range of health care areas that healthcare personnel, especially in LMICs may typically deal with. For example, in most primary health care settings, the same health care personnel may deal with a diverse range of healthcare issues including chronic disease and infectious disease management, and caring for pregnant women and children. The CDSS interventions evaluated by the included studies focused on a singular healthcare area‐ this would not capture the challenges that are faced at a systemic level where healthcare workers are required to juggle services across a range of priorities. Should healthcare workers have CDSS tools across all priority health care areas? If so, what are the unique training and implementation challenges that might be faced when services are digitally integrated? What would be the effectiveness of an integrated intervention? The review does not provide insights on these critical questions. Further research needs to take into account the wider health care system context in order to identify how CDSS should be designed in a way such that they can be responsive to the priority population needs.
The contextual range of these studies reflects the widespread use of these tools and suggests that the results may be applicable across a wide range of settings. However, the findings were difficult to generalize to a specific practice and the effectiveness of CDSS may vary based on the type of healthcare provider (lay health worker versus professional provider), baseline knowledge, health condition, and setting. There were also variations in control groups across the studies, which made us uncertain about how much variation in the outcomes was due to the intervention versus other factors. None of the studies used quality of care framework. Rather, studies focused on specific outcomes that, albeit informative on the effects of the intervention, failed to portray a reasonably complete picture on the quality of care. For example, it is remarkable that no studies reported on health workers satisfaction or unintended consequences. This review cannot provide evidence on what would the intervention effects be across a more comprehensive set of quality of care dimensions; hence caution has to be exerted in interpreting the beneficial findings reported in this review.
Quality of the evidence
The certainty of the evidence was very low for the outcome providers' adherence to recommended protocols due to risk of bias and concerns about serious imprecision. The certainty of evidence for patients'/clients' outcomes ranged from low to moderate owing to imprecision due to few events and risk of bias. The certainty of the evidence for patients'/clients' health status ranged from moderate to very low. The results on mean systolic blood pressure were downgraded due concerns about risk of bias. Other outcomes were downgraded due to concerns about risk of bias, as well as serious imprecision resulting from very few events and lack of accounting for clustering of participants.
Potential biases in the review process
We utilized a comprehensive search strategy, inclusive of certain databases with unpublished studies, independent assessment of study eligibility and risk of bias, and independent data extraction. The focus of this review was on CDSS that were primarily accessed by and optimized for mobile usage. One potential area of bias is in the assessment of whether the intervention was optimized for mobile usage, especially where reporting on the details of the intervention was inadequate. Furthermore, it is conceivable that the authors have mischaracterized the components and functionalities of the intervention, especially in the case of complex interventions. Evidence for all outcomes came from single studies and it was not possible to explore the impact of all the sources of heterogeneity.
Agreements and disagreements with other studies or reviews
We did not identify any reviews that address the same objectives as this review. Reviews that have looked at the effects of stationary CDSS in clinical settings found that CDSS was effective in improving clinical performance, preventive care, and provider performance (Bright 2012; Hunt 1998; Jaspers 2011; Kawamoto 2005; Varghese 2018). Given that most of the existing reviews focus largely on computerized stationary CDSS in hospital‐based and HIC settings, the implementation considerations and subsequent effects of the interventions are different from the studies included in our review. Like this review, other reviews were limited by methodological limitations, heterogeneous interventions and comparison groups, and multiple outcomes.
Reviews on the effectiveness of clinical decision‐support tools have different objectives than this review. . Most reviews focused primarily on computerized CDSS, and not on the use of mobile‐phone based CDSS (Bright 2012; Caballero‐Ruiz 2017; Hunt 1998; Jaspers 2011; Kaplan 2001; Kawamoto 2005; Kilsdonk 2017; Syrowatka 2016). Other reviews may not have specifically focused on CDSS solely and included all digital interventions (Adepoju 2017; Brenner 2016; Carter 2019; Mishra 2019). Several reviews were published in the 1990s or 2000s (Hunt 1998; Kaplan 2001; Kawamoto 2005), and were primarily focused on the use of CDSS in hospital‐based settings (Bright 2012; Hunt 1998; Jaspers 2011; Kaplan 2001; Martínez‐Pérez 2014; Sutton 2020; Varghese 2018) and high‐income settings (Dreesens 2019). Other reviews focused on the use of CDSS among community health workers in LMIC settings only (Agarwal 2015; Mishra 2019). A few reviews have focused primarily on specific outcomes such as feasibility and acceptability of the CDSS (Jaspers 2011; Kaplan 2001; Kawamoto 2005; Kilsdonk 2017). Reviews also focused on specific health conditions such as cancer (Baptista 2018; Mazo 2020; Tong 2021), maternal and child health care (Caballero‐Ruiz 2017; Carter 2019), non‐communicable diseases (Mishra 2019), and cardiovascular disease (Njie 2015). Given the different objectives as well as inclusion/exclusion criteria of these reviews, their results cannot be directly compared to the results of our review.
Authors' conclusions
Implications for practice.
Recent analysis suggests that improved health service coverage without a minimum level of quality significantly limits effectiveness (Kruk 2018). While clinical decision‐support system (CDSS) tools may have potential to influence the quality of care provision, the study provides limited evidence about the efficacy of CDSS on health outcomes. At the same time, the review does not provide any suggestion of deleterious effects due to the use of CDSS.
One linked Cochrane Review has synthesized qualitative research on health workers' perceptions and experiences of using mHealth technologies to deliver primary health services (Odendaal 2020). The following implications for practice have been taken from that review.
Health systems questions
Will health workers be part of the planning, implementation, and evaluation processes of mobile health programmes? Will their views be sought, and their perspectives taken at each stage of the programme?
To what extent is political buy‐in from health ministries required, and achieved, for the successful implementation of the mobile health programme?
Has a proper assessment been made on whether health workers' use of mobile devices is adding to or alleviating their workload? How will the extra workload that may occur, be accommodated for?
Technical and infrastructural questions
Does your setting have the necessary infrastructural and technological capacity to support the level of sophistication intended by the intervention? For example, is there sufficient electricity supply and electricity coverage, network capacity, technical support, and vendors to purchase phone credit or data for the level of intervention that you intend to implement? Have you considered how these might vary by region?
Are the devices being used in the intervention sufficiently sophisticated for the level of intervention being planned, and are these devices replaceable or reparable within your setting? Have you considered who will repair them, and who will cover the costs?
When planning mHealth programmes, has the number of staff who have access to mobile devices been taken into account?
Has adequate provision been made for health workers to have enough phone credit and data, without having to use their own resources?
Is there a strategy to integrate the mobile health platform within existing electronic health information systems?
Health worker training and skills
Has the programme management budgeted for adequate training of initial staff, refresher training, and in‐service training for new staff members?
What is the level of digital literacy among those health workers who will implement the intervention, as well as managers and supervisors who will support them? What further interventions are needed to ensure adequate skill level is present at the beginning of the intervention and maintained over the course of the intervention?
Has the programme management identified 'champions' among the workers whom they can call upon to assist those struggling with the devices?
When the device allows the health worker to screen and diagnose clients, are they clinically equipped to respond appropriately to the results of the screening and diagnosing? Are they able to explain the results to the patient?
Is there a system in place to allow staff who dislike, or who lack digital literacy to use mobile devices, to continue with standard practice, such as a paper‐based system for recording work?
Implications for research.
More, well‐designed research needs to be undertaken to understand the effectiveness of CDSS in improving quality of health services. This review identified some limitations that future trialists should consider. First, larger well‐designed trials of effect of mobile clinical decision‐support tools on health outcomes are needed. Ideally, these trials should be conducted beyond the pilot stage, after the intervention is 'bedded in' and refined. These trials should also measure outcomes that have not been measured to date, namely unintentional consequences, and resource use/cost‐effectiveness of the intervention. Second, CDSS tool may have a range of functionality (e.g. checklists, targeted two‐way communication with a provider though a text message or an application). Reporting on the details of the intervention can help delineate the specific functions driving the health impact. Third, trialists should also consider possible adverse effects (e.g. could use of a mobile device detract from delivery of care?) and how such an intervention might affect inequities by gender or access to mobile networks. Fourth, in several contexts, randomized trials may not be feasible or appropriate given the limitations in implementing such interventions. Alternate study designs, such as longitudinal or cohort studies should be explored. For CDSS interventions that are known to be efficacious based on results from well‐designed trials, it might be necessary to conduct implementation science studies that assess adaptations of the intervention to a given context. These will help provide insights on cost‐benefit, cost‐utility, and process improvement. Fifth, it is important the future studies consider the effects of CDSS interventions that are integrated with the broader health system, and address a range of health conditions and processes that health care personnel routinely deal with in their clinical practice. Lastly, CDSS interventions need to be evaluated using a quality of care framework. None of the studies reported on patient satisfaction, which is core to adopted and sustainability of such interventions.
What's new
Date | Event | Description |
---|---|---|
28 July 2021 | Amended | A comment was removed from the text of the review. |
History
Protocol first published: Issue 2, 2018 Review first published: Issue 7, 2021
Acknowledgements
We acknowledge the help and support of National Institute for Health Research (NIHR), via Cochrane Infrastructure funding to the Effective Practice and Organisation of Care (EPOC) Group. The views and opinions expressed herein are those of the review authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, National Health Service, or the Department of Health.
We are grateful to the Guideline Development Group of the Digital Health Guidelines for their constructive feedback in formulating the guiding questions for this systematic review. We thank Marit Johansen and John Eyers for executing the search strategies for the review. Cochrane Response, an evidence services unit operated by Cochrane, provided screening and data extraction services for this review.
We thank the following peer reviewers for their review and feedback: Xavier Bosch‐Capblanch (contact editor), Jhon Camacho, Stella Maria O'Brien, Andrew Hutchings, Caroline Free, Paul Miller, Daniela Gonçalves Bradley, Jennifer Hilgart, and Chris Cooper (EPOC managing editor of the review).
Appendices
Appendix 1. Search strategies
Cochrane Central Register of Controlled Trials (CENTRAL) Issue 9, 2020, Cochrane Library
ID | Search | Hits |
#1 | MeSH descriptor: [Cell Phone] this term only | 674 |
#2 | MeSH descriptor: [Smartphone] this term only | 384 |
#3 | MeSH descriptor: [MP3‐Player] this term only | 21 |
#4 | MeSH descriptor: [Computers, Handheld] this term only | 270 |
#5 | ((cell* or mobile*) near/1 (phone* or telephone* or technolog* or device*)):ti,ab,kw | 4321 |
#6 | (handheld or hand‐held):ti,ab,kw | 2296 |
#7 | (smartphone* or smart‐phone* or cellphone* or mobiles):ti,ab,kw | 4090 |
#8 | ((personal near/1 digital) or (PDA near/3 (device* or assistant*)) or MP3 player* or MP4 player*):ti,ab,kw | 320 |
#9 | (samsung or nokia):ti,ab,kw | 161 |
#10 | (windows near/3 (mobile* or phone*)):ti,ab,kw | 6 |
#11 | android:ti,ab,kw | 633 |
#12 | (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*):ti,ab,kw | 1103 |
#13 | (tablet* near/3 (device* or computer*)):ti,ab,kw | 854 |
#14 | MeSH descriptor: [Telemedicine] this term only | 2079 |
#15 | MeSH descriptor: [Webcasts as Topic] this term only | 24 |
#16 | MeSH descriptor: [Text Messaging] this term only | 848 |
#17 | MeSH descriptor: [Telenursing] this term only | 31 |
#18 | (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health):ti,ab,kw | 4477 |
#19 | (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*):ti,ab,kw | 6291 |
#20 | (webcast* or web‐cast*):ti,ab,kw | 40 |
#21 | (((text* or short or voice or multimedia or multi‐media or electronic or instant) near/1 messag*) or instant messenger) .ti,ab,kw | 65 |
#22 | (texting or texted or texter* or ((sms or mms) near (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP):ti,ab,kw | 2850 |
#23 | (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*):ti,ab,kw | 1069 |
#24 | MeSH descriptor: [Mobile Applications] this term only | 628 |
#25 | "mobile app*":ti,ab,kw | 658 |
#26 | MeSH descriptor: [Reminder Systems] this term only | 933 |
#27 | (remind* near/3 (text* or system* or messag*)):ti,ab,kw | 2143 |
#28 | MeSH descriptor: [Medical Informatics] this term only | 74 |
#29 | MeSH descriptor: [Medical Informatics Applications] this term only | 23 |
#30 | MeSH descriptor: [Nursing Informatics] this term only | 7 |
#31 | MeSH descriptor: [Public Health Informatics] this term only | 1 |
#32 | ((medical or clinical or health or healthcare or nurs*) near/3 informatics):ti,ab,kw | 325 |
#33 | MeSH descriptor: [Computer‐Assisted Instruction] this term only | 1214 |
#34 | ((interactive or computer‐assisted) near/1 (tutor* or technolog* or learn* or instruct* or software or communication)):ti,ab,kw | 1605 |
#35 | {or #1‐#34} | 26207 |
#36 | MeSH descriptor: [Decision Support Systems, Clinical] this term only | 370 |
#37 | MeSH descriptor: [Decision Making] this term only | 2085 |
#38 | MeSH descriptor: [Decision Support Techniques] this term only | 800 |
#39 | MeSH descriptor: [Diagnosis, Computer‐Assisted] this term only | 682 |
#40 | MeSH descriptor: [Decision Support Systems, Management] this term only | 8 |
#41 | MeSH descriptor: [Expert Systems] this term only | 58 |
#42 | MeSH descriptor: [Point‐of‐Care Systems] this term only | 423 |
#43 | MeSH descriptor: [Guideline Adherence] this term only | 1044 |
#44 | MeSH descriptor: [Clinical Protocols] this term only | 4725 |
#45 | MeSH descriptor: [Checklist] this term only | 265 |
#46 | MeSH descriptor: [Therapy, Computer‐Assisted] this term only | 1319 |
#47 | MeSH descriptor: [Drug Therapy, Computer‐Assisted] this term only | 148 |
#48 | MeSH descriptor: [Electronic Prescribing] explode all trees | 22 |
#49 | MeSH descriptor: [Clinical Laboratory Information Systems] this term only | 8 |
#50 | MeSH descriptor: [Clinical Pharmacy Information Systems] this term only | 21 |
#51 | ((decision* near/3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*"):ti,ab,kw | 20360 |
#52 | ((therap* or prescrib* or prescript* or diagnos*) near/2 (computer* or digital or electronic)):ti,ab,kw | 3968 |
#53 | ((guideline* or protocol*) near/4 (adher* or comply or complian* or observ*) or checklist*):ti,ab,kw | 14974 |
#54 | {or #36‐#53} | 42794 |
#55 | #35 and #54 | 3065 |
#56 | #55 with Cochrane Library publication date Between Aug 2019 and Oct 2020, in Trials | 670 |
#57 | #55 with Publication Year from 2019 to 2020, in Trials | 605 |
#58 | #56 or #57 | 787 |
MEDLINE and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily from 1946, Ovid (searched 9 October 2020)
# | Searches | Results |
1 | Decision Support Systems, Clinical/ | 8089 |
2 | Decision Making/ | 96106 |
3 | Decision Support Techniques/ | 20573 |
4 | Diagnosis, Computer‐assisted/ | 22696 |
5 | Decision Support Systems, Management/ | 958 |
6 | Expert Systems/ | 3410 |
7 | Point‐of‐Care Systems/ | 13132 |
8 | Guideline Adherence/ or Clinical Protocols/ or Checklist/ | 66164 |
9 | Therapy, Computer‐Assisted/ | 6813 |
10 | "Drug Therapy, Computer‐Assisted"/ | 1672 |
11 | Electronic Prescribing/ | 1037 |
12 | Clinical Laboratory Information Systems/ or Clinical Pharmacy Information Systems/ | 3221 |
13 | ((decision* adj3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*").ti,ab,kw. | 210661 |
14 | ((therap* or prescrib* or prescript* or diagnos*) adj2 (computer* or digital or electronic)).ti,ab,kw. | 9621 |
15 | (((guideline* or protocol*) adj4 (adher* or comply or complian* or observ*)) or checklist*).ti,ab,kw. | 68556 |
16 | or/1‐15 | 448888 |
17 | Cell Phones/ | 8631 |
18 | Smartphone/ | 4724 |
19 | MP3‐Player/ | 185 |
20 | Computers, Handheld/ | 3641 |
21 | ((cell* or mobile*) adj1 (phone* or telephone* or technolog* or device*)).ti,ab,kw. | 21876 |
22 | (handheld or hand‐held).ti,ab,kw. | 13682 |
23 | (smartphone* or smart‐phone* or cellphone* or mobiles).ti,ab,kw. | 15681 |
24 | ((personal adj1 digital) or (PDA adj3 (device* or assistant*)) or MP3 player* or MP4 player*).ti,ab,kw. | 1453 |
25 | (samsung or nokia).ti,ab,kw. | 1390 |
26 | (windows adj3 (mobile* or phone*)).ti,ab,kw. | 59 |
27 | android.ti,ab,kw. | 2932 |
28 | (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*).ti,ab,kw. | 3232 |
29 | (tablet* adj3 (device* or computer*)).ti,ab,kw. | 1902 |
30 | Telemedicine/ | 24150 |
31 | Webcasts as topic/ | 348 |
32 | Text Messaging/ | 3016 |
33 | Telenursing/ | 220 |
34 | (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health).ti,ab,kw. | 35112 |
35 | (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*).ti,ab,kw. | 22715 |
36 | (webcast* or web‐cast*).ti,ab,kw. | 273 |
37 | (((text* or short or voice or multimedia or multi‐media or electronic or instant) adj1 messag*) or instant messenger).ti,ab,kw. | 6317 |
38 | (texting or texted or texter* or ((sms or mms) adj (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP).ti,ab,kw. | 3803 |
39 | (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*).ti,ab,kw. | 9991 |
40 | Mobile Applications/ | 6339 |
41 | "mobile app*".ti,ab,kw. | 6135 |
42 | Reminder Systems/ | 3500 |
43 | (remind* adj3 (text* or system* or messag*)).ti,ab,kw. | 2014 |
44 | Medical informatics/ or Medical informatics applications/ | 14349 |
45 | Nursing informatics/ or Public health informatics/ | 2679 |
46 | ((medical or clinical or health or healthcare or nurs*) adj3 informatics).ti,ab,kw. | 6089 |
47 | Computer‐Assisted Instruction/ | 11906 |
48 | ((interactive or computer‐assisted) adj1 (tutor* or technolog* or learn* or instruct* or software or communication)).ti,ab,kw. | 2733 |
49 | or/17‐48 | 165200 |
50 | randomized controlled trial.pt. | 514593 |
51 | controlled clinical trial.pt. | 93873 |
52 | randomized.ab. | 522812 |
53 | placebo.ab. | 219198 |
54 | drug therapy.fs. | 2240479 |
55 | randomly.ab. | 366157 |
56 | trial.ab. | 551010 |
57 | groups.ab. | 2221881 |
58 | or/50‐57 | 4969223 |
59 | exp animals/ not humans.sh. | 4741227 |
60 | 58 not 59 | 4336186 |
61 | 16 and 49 and 60 | 3807 |
62 | (201809* or 201810* or 201811* or 201812* or 2019*).dt,dp,ed,ep,yr. | 2829843 |
63 | 61 and 62 | 881 |
64 | (201908* or 201909* or 201910* or 201911* or 201912* or 2020*).dt,dp,ed,ep,yr. | 2597293 |
65 | 61 and 64 | 795 |
Embase from 1974, Ovid (searched 9 October 2020)
# | Searches | Results |
1 | decision support system/ or clinical decision support system/ | 25781 |
2 | decision making/ or clinical decision making/ or medical decision making/ or shared decision making/ | 369459 |
3 | computer assisted diagnosis/ or computer assisted drug therapy/ or computer assisted therapy/ | 44301 |
4 | expert system/ | 5417 |
5 | "point of care system"/ | 2227 |
6 | protocol compliance/ or checklist/ | 39173 |
7 | clinical protocol/ | 100201 |
8 | computer assisted diagnosis/ | 39288 |
9 | computer assisted drug therapy/ | 914 |
10 | computer assisted therapy/ | 4644 |
11 | electronic prescribing/ | 3077 |
12 | laboratory information system/ | 1067 |
13 | ((decision* adj3 (make or makes or making or made or support* or algorithm* or aid or aids or app or apps or application* or technique*)) or expert system* or job‐aid* or "job aid*").ti,ab,kw. | 281691 |
14 | ((therap* or prescrib* or prescript* or diagnos*) adj2 (computer* or digital or electronic)).ti,ab,kw. | 14412 |
15 | (((guideline* or protocol*) adj4 (adher* or comply or complian* or observ*)) or checklist*).ti,ab,kw. | 94744 |
16 | or/1‐15 | 767076 |
17 | mobile phone/ or smartphone/ | 29884 |
18 | mp3 player/ | 201 |
19 | ((cell* or mobile*) adj1 (phone* or telephone* or technolog* or device*)).ti,ab,kw. | 25726 |
20 | (handheld or hand‐held).ti,ab,kw. | 17743 |
21 | (smartphone* or smart‐phone* or cellphone* or mobiles).ti,ab,kw. | 18968 |
22 | ((personal adj1 digital) or (PDA adj3 (device* or assistant*)) or MP3 player* or MP4 player*).ti,ab,kw. | 1896 |
23 | (samsung or nokia).ti,ab,kw. | 2356 |
24 | (windows adj3 (mobile* or phone*)).ti,ab,kw. | 80 |
25 | android.ti,ab,kw. | 4192 |
26 | (ipad* or i‐pad* or ipod* or i‐pod* or iphone* or i‐phone*).ti,ab,kw. | 5455 |
27 | (tablet* adj3 (device* or computer*)).ti,ab,kw. | 2684 |
28 | telemedicine/ or telecardiology/ or teleconsultation/ or teledermatology/ or telediagnosis/ or telemonitoring/ or telepathology/ or telepsychiatry/ or teleradiotherapy/ or telesurgery/ or teletherapy/ | 41005 |
29 | webcast/ | 358 |
30 | text messaging/ | 5206 |
31 | telenursing/ | 283 |
32 | (mhealth or m‐health or "mobile health" or ehealth or e‐health or "electronic health" or "digital health" or uhealth or u‐health).ti,ab,kw. | 42203 |
33 | (telemedicine or tele‐medicine or telehealth or tele‐health or telecare or tele‐care or telenursing or tele‐nursing or telepsychiatry or tele‐psychiatry or telemonitor* or tele‐monitor* or teleconsult* or tele‐consult* or telecounsel* or tele‐counsel* or telecoach* or tele‐coach*).ti,ab,kw. | 28014 |
34 | (webcast* or web‐cast*).ti,ab,kw. | 439 |
35 | (((text* or short or voice or multimedia or multi‐media or electronic or instant) adj1 messag*) or instant messenger).ti,ab,kw. | 7599 |
36 | (texting or texted or texter* or ((sms or mms) adj (service* or messag*)) or interactive voice response* or IVR or voice call* or callback* or voice over internet or VOIP).ti,ab,kw. | 5157 |
37 | (Facebook or Twitter or Whatsapp* or Skyp* or YouTube or "You Tube" or Google Hangout*).ti,ab,kw. | 12598 |
38 | mobile application/ | 11922 |
39 | "mobile app*".ti,ab,kw. | 6398 |
40 | reminder system/ | 2660 |
41 | (remind* adj3 (text* or system* or messag*)).ti,ab,kw. | 2768 |
42 | medical informatics/ | 20670 |
43 | nursing informatics/ | 1559 |
44 | ((medical or clinical or health or healthcare or nurs*) adj3 informatics).ti,ab,kw. | 9375 |
45 | teaching/ | 91921 |
46 | ((interactive or computer‐assisted) adj1 (tutor* or technolog* or learn* or instruct* or software or communication)).ti,ab,kw. | 3797 |
47 | or/17‐46 | 292926 |
48 | crossover procedure/ | 64524 |
49 | double blind procedure/ | 176364 |
50 | randomized controlled trial/ | 622476 |
51 | single‐blind procedure/ | 40351 |
52 | random$.tw. | 1580230 |
53 | factorial$.tw. | 39052 |
54 | (crossover$ or cross over$ or cross‐over$).tw. | 108663 |
55 | placebo$.tw. | 313560 |
56 | (doubl$ adj blind$).tw. | 212917 |
57 | (singl$ adj blind$).tw. | 25656 |
58 | assign$.tw. | 403779 |
59 | allocat$.tw. | 157754 |
60 | volunteer$.tw. | 261957 |
61 | or/48‐60 | 2386666 |
62 | 16 and 47 and 61 | 4242 |
63 | limit 62 to embase | 1994 |
64 | (201809* or 201810* or 201811* or 201812* or 2019*).dd. | 787410 |
65 | ("2018" or "2019").yr. | 3234006 |
66 | 64 or 65 | 3400778 |
67 | 63 and 66 | 477 |
68 | (201908* or 201909* or 201910* or 201911* or 201912* or 2020*).dd. | 917313 |
69 | ("2019" or "2020").yr. | 2969423 |
70 | 68 or 69 | 3101761 |
71 | 63 and 70 | 497 |
72 | ("201932" or "201933" or "201934" or "201935" or "201936" or "201937" or "201938" or "201939" or 20194* or 20195* or 2020*).em. | 2405021 |
73 | 63 and 72 | 292 |
74 | 71 or 73 | 528 |
POPLINE, K4Health (searched 5August 2019)
All Fields:
((decision* AND (make OR makes OR making OR made OR support* OR algorithm* OR aid OR aids OR app OR apps OR application* OR technique*)) OR "expert system*" OR checklist* OR job‐aid* OR "job aid*")
OR
Keyword:
DECISION MAKING
AND
All Fields:
((cell OR cellular OR mobile) AND (phone OR phones OR telephone OR telephones OR technology OR technologies OR device OR devices)) OR smartphone OR smartphones OR smart‐phone OR smart‐phones OR cellphone OR cellphones OR mobiles OR mhealth OR m‐health OR "mobile health" OR ehealth OR e‐health OR "electronic health" OR telemedicine OR tele‐medicine OR telehealth OR tele‐health OR telecare OR tele‐care OR telenursing OR tele‐nursing OR telepsychiatry OR tele‐psychiatry OR telemonitor OR telemonitoring OR tele‐monitor OR tele‐monitoring OR teleconsult OR teleconsulting OR tele‐consult OR tele‐consulting OR telecounsel OR telecounseling OR tele‐counsel OR tele‐counseling OR telecoach OR telecoaching OR tele‐coach OR tele‐coaching OR videoconference OR videoconferences OR videoconferencing OR video‐conference OR video‐conferences OR video‐conferencing OR webcast OR webcasts OR webcasting OR web‐cast OR web‐casts OR web‐casting OR ((text OR texts OR texting OR short OR voice OR multimedia OR multi‐media OR electronic OR instant) AND (message OR messages OR messaging)) OR "instant messenger" OR texting OR texted OR texter OR texters OR ((sms OR mms) AND (service OR services OR message OR messages OR messaging)) OR "interactive voice response" OR "interactive voice responses" OR ivr OR "voice call" OR "voice calls" OR callback OR "voice over internet" OR voip OR "mobile app" OR "mobile apps" OR "mobile application" OR "mobile applications" OR "social media" OR ((medical OR clinical OR health OR healthcare OR nurse OR nurses OR nursing) AND informatics)
OR
Keyword:
TEXT MESSAGING OR MOBILE DEVICES OR INFORMATION COMMUNICATION TECHNOLOGY OR CELLULAR PHONE
AND
Keyword:
QUANTITATIVE RESEARCH OR RESEARCH METHODOLOGY OR CLINICAL TRIALS OR CONTROL GROUPS
OR
All Fields:
(randomised OR randomized OR "randomly allocated" OR "random allocation" OR "controlled trial" OR "control group" OR "control groups" OR trial)
Global Index Medicus, WHO (searched 9 October 2020)
Search fields used: Title, abstract, subject
("cell phones" OR smartphone OR mp3‐player OR "Computers, Handheld" OR telemedicine OR videoconferencing OR "Text Messaging" OR telenursing OR "Mobile Applications" OR "Reminder Systems" OR "Electronic Mail" OR "Medical Informatics" OR "Nursing Informatics" OR "Public Health Informatics" OR multimedia OR hypermedia OR blogging OR "cell phone" OR "cellular phone" OR "cellular phones" OR "mobile phone" OR "mobile phones" OR "mobile devices" OR "mobile devices" OR smartphones OR smart‐phone OR smart‐phones OR cellphone OR cellphones) AND (app OR apps OR application* OR "decision technique" OR "decision techniques" OR checklist* OR "expert system" OR "expert systems" OR job‐aid OR job‐aids OR "job aid" OR "job‐aids" OR "decision support system" OR "decision support systems" OR "decision making" OR "decision support technique" OR "decision support techniques" OR "expert system" OR "expert systems" OR "point‐of‐care system" OR "point‐of‐care systems" OR "guideline adherence" OR "clinical protocol" OR "clinical protocols" OR checklist*) AND ("Controlled Clinical Trials, Randomized" OR "Controlled Clinical Trials as Topic" OR "Controlled Clinical Trial" OR "Clinical Trial" OR randomised OR randomized OR "randomly allocated" OR "random allocation" OR "controlled trial" OR "control group" OR "control groups" OR trial)
International Clinical Trials Registry Platform (ICTRP), WHO (searched 5 August 2019)
Two separate strategies. Used advanced search, with recruitment status: All
Strategy 1:
Title: decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or job aid
AND
Intervention: mobile device OR mobiles OR smartphone OR phone OR cellphone
Strategy 2:
Title: mobile device OR mobiles OR smartphone OR phone OR cellphone
AND
Intervention: decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or job aid
ClinicalTrials.gov, National Institutes of Health (searched 9 October 2020)
Other Terms: (decision OR decisions OR decision‐making OR checklist OR checklists OR job‐aid or "job aid") AND ("mobile device" OR "mobile devices" OR mobiles OR smartphone OR smartphones OR "smart phone")
With time‐limit: first posted from 8 May 2019 to 9 October 2020
Data and analyses
Comparison 1. Mobile clinical decision support compared to standard care for patients.
Outcome or subgroup title | No. of studies | No. of participants | Statistical method | Effect size |
---|---|---|---|---|
1.1 Providers' adherence to recommended practices, guidelines, or protocols | 2 | Other data | No numeric data | |
1.2 Patients' health status and well‐being (dichotomous outcomes) | 2 | Risk Ratio (M‐H, Random, 95% CI) | Subtotals only | |
1.2.1 Emergency referrals at 7 months of follow‐up | 1 | 463 | Risk Ratio (M‐H, Random, 95% CI) | 1.49 [1.09, 2.04] |
1.2.2 High‐risk people taking aspirin in the last month at 1 year of follow‐up | 1 | 2086 | Risk Ratio (M‐H, Random, 95% CI) | 9.30 [6.05, 14.28] |
1.2.3 Self‐reported use of community healthcare workers‐prescribed antihypertensive medication for ≥ 25 days in the past month at 1 year of follow‐up | 1 | 2086 | Risk Ratio (M‐H, Random, 95% CI) | 3.86 [3.14, 4.76] |
1.2.4 Successful referrals at 7months of follow‐up | 1 | 123 | Risk Ratio (M‐H, Random, 95% CI) | 0.97 [0.87, 1.08] |
1.2.5 Hospital delivery at 12 months of follow‐up | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 1.27 [0.81, 2.00] |
1.2.6 Number of women who had a caesarean section | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 1.24 [0.77, 1.99] |
1.2.7 Number of women who had emergency referral due to abnormal progression of labour | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 1.25 [0.83, 1.87] |
1.2.8 Number of women who had emergency referral due to a hypertensive disorder | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 3.30 [1.10, 9.86] |
1.2.9 Number of women who had an emergency referral due to haemorrhage | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 0.79 [0.33, 1.93] |
1.2.10 Number of women who had emergency referral due to premature labour | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 0.73 [0.23, 2.38] |
1.2.11 Number of women who had emergency referral due to fetal cardiac abnormality | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 0.88 [0.22, 3.49] |
1.2.12 Number of women who had emergency referral due to suspected sepsis | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 0.70 [0.19, 2.60] |
1.2.13 Number of women who had emergency referral due to respiratory compromise | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 2.20 [0.43, 11.27] |
1.2.14 Number of women who had emergency referral due to a preterm newborn | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 7.92 [0.43, 146.67] |
1.3 Patients' health status and well‐being (continuous outcomes) | 1 | Mean Difference (IV, Random, 95% CI) | Subtotals only | |
1.3.1 Diabetes care self‐efficacy | 1 | 176 | Mean Difference (IV, Random, 95% CI) | 3.30 [‐0.95, 7.55] |
1.3.2 Antihyperglycaemic medication decisional conflict at 3 months of follow‐up | 1 | 175 | Mean Difference (IV, Random, 95% CI) | 2.60 [‐2.07, 7.27] |
1.3.3 Diabetes distress at 3 months of follow‐up | 1 | 176 | Mean Difference (IV, Random, 95% CI) | 15.70 [8.24, 23.16] |
1.3.4 Medication adherence at 3 months of follow‐up | 1 | 176 | Mean Difference (IV, Random, 95% CI) | ‐2.30 [‐6.76, 2.16] |
1.4 Patients' health status and well‐being (dichotomous undesirable outcome) | 2 | Risk Ratio (M‐H, Random, 95% CI) | Subtotals only | |
1.4.1 Current smoker at 1 year of follow‐up | 1 | 2086 | Risk Ratio (M‐H, Random, 95% CI) | 1.03 [0.92, 1.16] |
1.4.2 Maternal deaths | 1 | 799 | Risk Ratio (M‐H, Random, 95% CI) | 1.76 [0.16, 19.33] |
1.5 Patients' health status and well‐being (continuous undesirable outcomes) | 2 | Mean Difference (IV, Random, 95% CI) | Subtotals only | |
1.5.1 Mean systolic blood pressure (mmHg) | 1 | 2086 | Mean Difference (IV, Random, 95% CI) | ‐2.80 [‐5.09, ‐0.51] |
1.5.2 Haemoglobin a1c | 1 | 175 | Mean Difference (IV, Random, 95% CI) | ‐0.10 [‐0.38, 0.18] |
1.6 Patients' health status and well‐being (neonatal deaths) | 2 | Odds Ratio (IV, Random, 95% CI) | Subtotals only | |
1.7 Patients' acceptability and satisfaction | 1 | Mean Difference (IV, Random, 95% CI) | Subtotals only | |
1.7.1 Satisfaction with helpfulness of medication information | 1 | 176 | Mean Difference (IV, Random, 95% CI) | 11.30 [3.28, 19.32] |
1.7.2 Satisfaction with clarity of medication information | 1 | 176 | Mean Difference (IV, Random, 95% CI) | 9.20 [0.97, 17.43] |
1.8 Providers' acceptability and satisfaction | 1 | Other data | No numeric data |
1.1. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 1: Providers' adherence to recommended practices, guidelines, or protocols
Providers' adherence to recommended practices, guidelines, or protocols | |||
Study | Outcome | Findings | Comments |
Berner 2006 | 1. Proportion of cases per physician with unsafe prescriptions 2. Proportion of cases per physician with key risk factor recorded |
1. Intervention group (IG): 0.23 Control group (CG): 0.45 2. IG:0.58 CG: 0.45 |
IG: 31 participants CG: 28 participants Incomplete outcome data reported. Standard errors not reported. |
Gautham 2015 | 1. Mean protocol compliance for female patients 2. Mean protocol compliance for male patients |
1. IG: 69 CG: 63.34 2. IG: 71.12 CG: 53.59 |
1. IG: 8 providers, 38 patients CG: 8 providers, 43 patients 2. IG: 8 providers, 27 patients CG: 8 providers, 18 patients Incomplete outcome data reported. Standard errors not reported. |
1.2. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 2: Patients' health status and well‐being (dichotomous outcomes)
1.3. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 3: Patients' health status and well‐being (continuous outcomes)
1.4. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 4: Patients' health status and well‐being (dichotomous undesirable outcome)
1.5. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 5: Patients' health status and well‐being (continuous undesirable outcomes)
1.6. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 6: Patients' health status and well‐being (neonatal deaths)
1.7. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 7: Patients' acceptability and satisfaction
1.8. Analysis.
Comparison 1: Mobile clinical decision support compared to standard care for patients, Outcome 8: Providers' acceptability and satisfaction
Providers' acceptability and satisfaction | |||
Study | Outcome | Finding | Comments |
Tian 2015 | 1. Providers' level of comfort using the system 2. Providers' willingness to continue using the system 3. Providers' wish for more health conditions to be included in the system 4. Helpfullness of the system in enabling provider to follow guidelines 5. Ease of use of the system 6. Being able to remember steps without the system 7. Providers' willingness to recommend the system |
1. 7/8 2. 7/8 3. 8/8 4. 6/8 5. 7/8 6. 0/8 7. 7/8 |
Outcomes measured after 2 months of using the system. Outcomes reported only for the intervention group. Incomplete data. |
Characteristics of studies
Characteristics of included studies [ordered by study ID]
Amoakoh 2019.
Study characteristics | ||
Methods |
Aim: to evaluate the utilization and effect of the mCDMSI on institutional neonatal mortality in the Eastern Region of Ghana. Study design: cluster randomized controlled trial Country: Ghana Setting: CHPS compounds and maternity homes, HCs, and hospitals in the Eastern Region Cluster features: district located in the Eastern Region of Ghana; expected deliveries of ≥ 1100/year for the year 2014 for a district Recruitment: within the randomized clusters, all health facilities that conducted deliveries in the year preceding the start of the intervention (2014) were recruited into this study. Study duration: 18 months Study dates: August 2015 to April 2017 |
|
Participants |
Inclusion criteria: women who delivered babies in hospitals in the study clusters for the 18‐month intervention period Sample size: 8 intervention (74 facilities); 8 control (102 facilities) clusters; 65,831 deliveries in total Age, mean: intervention 27.1 (SD 6.4) years; control 27.3 (SD 6.3) years for women delivering during the study Sex: 100% female |
|
Interventions |
Intervention: text messaging of standard protocols for maternal and neonatal care to front‐line healthcare providers in the region Content: mCDMSI consisted of 4 components: phone calls (voice), text messaging (SMS), access to the internet (data) and access to an USSD that provided protocols for management of obstetric and neonatal emergencies in response to selection from a short code drop‐down menu Control: no mobile phones or access to emergency protocols via USSD Co‐interventions: not reported |
|
Outcomes | Institutional neonatal mortality including deaths of babies admitted from birth and those (re)admitted from home; utilization of the mCDMSI for clinical decision‐making Outcome assessment timepoints: 18 months |
|
Notes |
Funding: Netherlands Foundation for Scientific Research – WOTRO, Science for Global Development; Utrecht University Conflicts of interest: none declared |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Low risk | Used a randomization scheme of permuted blocks to randomize the 16 districts equally to the 2‐armed programme (control and intervention). |
Allocation concealment (selection bias) | Low risk | Randomization performed by an independent data analyst in order to achieve comparability and avoid selection bias. |
Blinding of participants and personnel (performance bias) all outcomes | High risk | Due to the nature of this intervention, blinding was not feasible. |
Blinding of outcome assessment (detection bias) all outcomes | Unclear risk | Data were extracted from the district health information management system‐2 (DHIMS‐2) database. |
Incomplete outcome data (attrition bias) all outcomes | Unclear risk | Data that were available in the DHIMS‐2 and captured as aggregate per health facility were the number of neonatal deaths and the number of deliveries. Detailed information regarding each delivery captured in the DHIMS‐2 was limited to hospital deliveries, and further limited to peri‐partum maternal data. |
Selective reporting (reporting bias) | High risk | Some outcomes listed in the clinical trial registry not reported in results. |
Other bias | High risk | Likely to be baseline differences in neonatal mortality risk factors between study groups (in discussion). |
Berner 2006.
Study characteristics | ||
Methods |
Aim: to evaluate the effectiveness of a PDA‐based CDSS on NSAID prescribing safety in the outpatient setting. Study design: randomized controlled trial Country: USA Setting: university‐based clinic Recruitment: participants recruited from a pool of 105 internal medicine residents Study duration: baseline phase approximately 6 months and follow‐up phase 8 months Study dates: not reported |
|
Participants |
Inclusion criteria: internal medicine residents assigned to an urban university‐based, resident‐staffed clinic Sample size: 68 (intervention 34; control 34) Age, mean: intervention 27.35 (SD 2.18) years; control 28.57 (SD 2.28) years Sex: intervention 26% female; control 29% female |
|
Interventions |
Intervention: CDSS (MedDecide) on PDA, the focus was a clinical prediction rule to assess NSAID‐related gastrointestinal risk and provide real‐time treatment recommendations based on the patient's risk. Content: 19 rules representing diagnostic, risk assessment, and treatment recommendations. Participants were randomized to receive 1 of 6 different sets of these rules for their PDA, with each set containing 14 rules. Only the participants who were randomized to the intervention group received, within their set of 14 rules, the rule for gastrointestinal risk assessment when prescribing NSAIDs. Control: clinical prediction rules that did not include the rule for gastrointestinal risk. Co‐interventions: other clinical prediction rules on PDA (e.g. Epocrates, Medcalc, Medmath, and a breast cancer risk calculator). |
|
Outcomes | Patient safety; safe prescribing (i.e. duration and dose) of NSAIDs; presence of key risk factor Outcome assessment timepoints: 8 months of follow‐up |
|
Notes | To evaluate outcomes, 4 standardized patient cases were used (13 trained standardized patients) and participants saw ≥ 1 standardized patient during both baseline and follow‐up. Funding: supported in part by the Agency for Healthcare Research and Quality Conflicts of interest: not reported |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Low risk | Computerized random number generator that ensured an equal number of participants in each group. |
Allocation concealment (selection bias) | Unclear risk | Not reported in text. |
Blinding of participants and personnel (performance bias) all outcomes | Unclear risk | Participants were 'blinded to the specific outcome of interest;' however, it was unclear whether standardized patients were blinded to whether they were seeing an intervention or control physician. |
Blinding of outcome assessment (detection bias) all outcomes | Low risk | Patient safety outcomes were determined by a review of documentation from standardized patient encounters. 5 physicians with experience in health services research constituted the outcomes committee. Each chart was independently reviewed by 2 clinicians who were blinded to participant, timing (baseline or follow‐up), and group (intervention or control). |
Incomplete outcome data (attrition bias) all outcomes | Low risk | Attrition was low: 3/34 intervention; 6/34 control. The reasons for attrition seem unlikely to be related to outcomes. |
Selective reporting (reporting bias) | Low risk | No obvious missing outcomes or data. No protocol or trial registry was detected for the study. |
Other bias | Unclear risk | No evidence of additional sources of bias, but reporting insufficient to be certain. For example, there may have been contamination between intervention and control participants. |
Eaton 2011.
Study characteristics | ||
Methods |
Aim: to determine whether an intervention based on patient activation and a physician decision‐support tool was more effective than usual care for improving adherence to National Cholesterol Education Program guidelines. Study design: cluster randomized controlled trial Country: USA Setting: 30 primary care practices Cluster features: 30 primary care practices in southeastern New England Recruitment: not reported Study duration: 2 years Study dates: June 2003 to May 2005 |
|
Participants |
Inclusion criteria: not reported Sample size: 30 practices, 55 physicians, 4105 patients Age, mean: intervention 54.0 (SD 1.1) years; control 52.3 (SD 1.1) years Sex: intervention 60.3% female; control 58.2% female |
|
Interventions |
Intervention: interactive decision software on a PDA using data from a patient activation tool, HeartAge, in which patients answered questions regarding their risk factors for CHD (age, sex, blood pressure, total cholesterol, HDL‐cholesterol levels, diabetes). Content: determined the patient's lipid diagnosis (LDL dominant, isolated low HDL level, triglyceride dominant, mixed lipid disorder, and atherogenic dyslipidaemia), calculated the ATP III LDL and non‐HDL cholesterol goals (when appropriate), made recommendations regarding therapeutic lifestyle management, provided optimal dosage of lipid‐lowering drugs tailored to the patient's risk factor status to meet the ATP III goals, and provided an interactive shared decision making page for physicians to discuss lowering‐lipid values in the context of HeartAge, absolute and relative risks, and other CHD risk factor management. Control: received a PDA but without the decision‐support tool and had minimal further contact to mimic usual care. Co‐interventions: not reported. |
|
Outcomes | LDL‐cholesterol values, non‐HDL‐cholesterol values; ATP III Cholesterol Guideline Adherence; how the tool affected physician decision making Outcome assessment timepoints: 1 year |
|
Notes |
Funding: not reported Conflicts of interest: HeartAge is trademarked by Memorial Hospital of Rhode Island with Drs Eaton and Ahern as co‐developers. |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Unclear risk | Method of randomization not reported. |
Allocation concealment (selection bias) | Unclear risk | Method of allocation concealment not reported. |
Blinding of participants and personnel (performance bias) all outcomes | High risk | Not blinded. |
Blinding of outcome assessment (detection bias) all outcomes | High risk | Not blinded unless there was a disagreement. Quote: "Any disagreement on the major outcome, LDL cholesterol values, and non‐HDL cholesterol values was reviewed by one of the investigators (D.R.P., C.B.E.) who were blinded to the physician and practice, and a final decision was made." |
Incomplete outcome data (attrition bias) all outcomes | Low risk | 78 patients in the control group were excluded from analysis because of death, pregnancy, or they left the practice. 56 patients in the intervention group were excluded from analysis because of death, pregnancy, or they left the practice. Losses to follow‐up were low and balanced across groups. |
Selective reporting (reporting bias) | Low risk | All outcomes reported. |
Other bias | Low risk | No other obvious source of bias. |
Gautham 2015.
Study characteristics | ||
Methods |
Aim: to measure changes in protocol compliance by health workers in their everyday work settings, assess the usability and acceptability of the mobile application with health workers in the intervention group and obtain patient feedback on health workers' use of the mobile system during treatment. Study design: cluster randomized controlled trial Country: India Setting: 2 neighbouring districts, Sivagangai and Dharmapuri, in rural Tamil Nadu, a South Indian state. At 1 of these sites (Thirupathur in the Sivagangai district) the participants consisted of 8 independent informal healthcare providers based in villages. In the other site, the participants linked with a non‐governmental hospital (Tribal Health Initiative hospital) in a tribal area called Sittilingi in the district of Dharmapuri. Cluster features: 16 RHPs Recruitment: the RHPs in Sivagangai had been part of a previous technology‐enabled distance learning project developed by 1 of the co‐authors. The 2 different RHP groups were selected because the authors assumed that they would provide valuable additional lessons on the contribution of local contexts in the way the application was finally used and its sustainability and success. Study duration: 2 months Study dates: not reported |
|
Participants |
Inclusion criteria: not reported Sample size: 16 RHPs, 126 patients Age, mean: independent practitioners 48.75 years; auxiliaries attached to tribal hospital 24.75 years Sex: RHPs 50% female |
|
Interventions |
Intervention: media‐rich, mobile phone‐based clinical guidance system for management of fevers, diarrhoeas, and respiratory problems. Content: 2 validated clinical guidelines were adapted for implementation in the decision‐support tool. These were the WHO's IMCI and IMAI. Control: RHPs given only the phone plus a set of paper guidelines to use in the field. The application was installed on their phones after the field testing was over. Co‐interventions: not reported. |
|
Outcomes | Protocol compliance (as per clinical guidelines); acceptability; usability Outcome assessment timepoints: 2 months |
|
Notes |
Funding: Information Society Innovation Fund (ISIF) Asia allocated in a competitive process to Garhwal Community Development and Welfare Society (GCDWS) in 2010. ISIF Asia operates through a partnership between the International Development Research Centre of Canada, the Swedish International Development Agency – SIDA and the Asia Pacific Network Information Centre (APNIC), with sponsorship from the Internet Society and the Dot Asia Organization. Conflicts of interest: the second author (M Sriram Iyengar) is part owner of a company incorporated in the USA for commercialization of the GV technology. |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Unclear risk | No information given on sequence generation. Participants consisted of 16 RHPs, 8 in each of 2 locations. Quote: "In each location, four RHPs were randomly assigned to the experimental group and four to the control group; thus, there were eight RHPs in each group." |
Allocation concealment (selection bias) | Unclear risk | No information given on allocation concealment. Participants consisted of 16 RHPs, 8 in each of 2 locations. Quote: "In each location, four RHPs were randomly assigned to the experimental group and four to the control group; thus, there were eight RHPs in each group." |
Blinding of participants and personnel (performance bias) all outcomes | High risk | It is clear that the providers and patients were not blind to allocation. It would be impossible to blind the RHPs and difficult, although not impossible, to blind the patients. |
Blinding of outcome assessment (detection bias) all outcomes | High risk | Outcome assessors not blinded to allocation assessment. Outcome of protocol compliance was assessed by direct observation of participants who would be using their phones if in the intervention group. The usability/acceptability outcomes were only assessed in the intervention group. |
Incomplete outcome data (attrition bias) all outcomes | Unclear risk | It appears that all 16 providers completed the intervention period. There was no mention of any attrition. There was an under‐recruitment in some categories of the planned number of patients, which would have led to some estimates being based on fewer patients than planned. |
Selective reporting (reporting bias) | High risk | It was not reported how many participants were included in the usability and acceptability results. |
Other bias | High risk | The authors acknowledged that there may have been contamination between the groups in 1 district as both groups worked together. There was also a conflict of interest as 1 author is part owner of a company incorporated in the USA for commercialization of the GV technology used in developing mMRIGs and deploying them on mobile phones. |
Heisler 2014.
Study characteristics | ||
Methods |
Aim: to evaluate whether more sophisticated tailored, interactive e‐Health tools increase the effectiveness of CHW outreach with underserved patients compared to when they relied on printed educational materials alone. Study design: randomized controlled trial Country: USA Setting: community HC in Detroit, USA Recruitment: potentially eligible participants were identified from a computer‐generated list of the CHASS patients. Study duration: 3 months Study dates: September 2011 to September 2013 |
|
Participants |
Inclusion criteria: people with physician‐diagnosed type 2 diabetes; HbA1c > 7.5% in the prior 6 months or expressed concerns about current diabetes medications during the screening assessment Sample size: 188 (intervention 93; control 95) Age, mean: intervention 51 (SD 8.6); control 52 (SD 9.4) Sex: intervention 76% female; control 66% female |
|
Interventions |
Intervention: iDecide is a personally tailored, interactive diabetes medication decision aid designed for CHWs to deliver on tablet computers with 3G access to African American and Latino adults with diabetes and low health literacy. Content: iDecide uses the same content as the Agency for Health Care Quality Guides ("Pills for Type 2 Diabetes" and "Premixed Insulin for Type 2 Diabetes"). Control: all participants received an initial 1‐to‐1, face‐to‐face session with a CHW and a copy of the printed materials to take home. The sessions using printed materials lasted approximately 1.5 hours. Co‐interventions: not reported. |
|
Outcomes | Antihyperglycaemic medication decisional conflict; knowledge and beliefs about antihyperglycaemic medications; satisfaction with medication information (clarity and helpfulness); improvements in diabetes care self‐efficacy; diabetes distress; medication adherence Outcome assessment timepoints: 3 months |
|
Notes |
Funding: Agency for Health Care Quality and Research and the National Institute of Diabetes and Digestive and Kidney Diseases. Conflicts of interest: no conflict of interest or financial disclosures. |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Low risk | Quote: "participants … were randomized by the computer program using a random sequence algorithm into one of two study arms." |
Allocation concealment (selection bias) | Unclear risk | No report of allocation concealment. |
Blinding of participants and personnel (performance bias) all outcomes | High risk | The nature of the intervention meant that it was impossible to blind the CHWs and patients. |
Blinding of outcome assessment (detection bias) all outcomes | Low risk | Quote: "data assessors remained blinded to group assignment throughout the study." |
Incomplete outcome data (attrition bias) all outcomes | Low risk | Attrition was low and balanced between groups. The intervention group lost 6/95 participants and the control group lost 6/97 participants. Most attrition was due to researchers being unable to reach participants and it is unlikely that this was related to the intervention. The authors used various methods to impute missing data and they reported that there were no significant changes in results. |
Selective reporting (reporting bias) | High risk | The study was registered on ClinicalTrials.gov as NCT01427660. 1 of the primary outcomes in the study report (medication decisional conflict) was given as the primary outcome on the ClinicalTrials.gov entry. However, the secondary outcomes in the ClinicalTrials.gov entry were given as "self‐reported medication adherence will be assessed through three well‐validated measures. We will assess changes in anti‐hyperglycemic medication dosages and/or numbers of medications by patient report and medical record review." There only appears to be 1 measurement of medication adherence in the trial report. There were no data regarding changes to medication dosages. Therefore, there was a high risk of bias for selective outcome reporting. |
Other bias | Unclear risk | There were some baseline imbalances. Quote: "More participants randomized to iDecide had completed high school (61%) than those randomized to the printed materials group (35 %, p<0.001). Patients in the iDecide group were also less likely to have difficulty with written healthcare information (p=0.03) and were more likely to be confident filling out medical paperwork (p=0.003)." Comment: it is likely that these factors would have an impact on the outcomes measured. The authors adjusted for these imbalances in their own analysis. |
Keane 2018.
Study characteristics | ||
Methods |
Aim: to evaluate a mobile health app to support the IMAM. Study design: cluster randomized controlled trial Country: Kenya Setting: health facilities from three subcounties in Wajir Cluster features: 40 health facilities from 3 subcounties Recruitment: not reported Study duration: 11 months Study dates: November 2015 to October 2016 |
|
Participants |
Inclusion criteria: not reported Sample size: 40 health facilities (20 interventions; 20 control), individuals not reported Age: not reported Sex: not reported |
|
Interventions |
Intervention: IMAM app used on tablets or mobile phones provided health workers with simple, step‐by‐step guidance on the assessment, treatment, or referral of children visiting the IMAM programme. Content: treatment protocol, counselling messages and return dates, and calculated z‐scores and numbers of ready‐to‐use therapeutic food sachets needed. It also recorded each child's information, making child follow‐up and defaulter tracing easier. Data were regularly uploaded to the 'cloud,' which enabled the provision of live and accurate data for county‐level management. Control: paper‐based child treatment data from registers. Co‐interventions: not reported. |
|
Outcomes | Neonatal deaths; maternal deaths; proportion of children cured, defaulted, died, or not cured when they exited the IMAM programme Outcome assessment timepoints: 1 year |
|
Notes | Unclear if this is a peer‐reviewed journal, limited reporting of study methodology and results. Funding: not reported Conflicts of interest: not reported |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Unclear risk | No information reported. |
Allocation concealment (selection bias) | Unclear risk | No information reported. |
Blinding of participants and personnel (performance bias) all outcomes | Unclear risk | No information reported. |
Blinding of outcome assessment (detection bias) all outcomes | Unclear risk | No information reported. |
Incomplete outcome data (attrition bias) all outcomes | Unclear risk | No information reported. |
Selective reporting (reporting bias) | Unclear risk | No information reported. |
Other bias | Unclear risk | No information reported. |
Martinez 2018.
Study characteristics | ||
Methods |
Aim: to characterize baseline rates of complication detection and facility‐level referral by TBAs in rural Guatemala, and to evaluate the impact of the mHealth system on these rates. Study design: randomized controlled trial Country: Guatemala Setting: Maya Health Alliance, a Guatemalan primary healthcare organization with a clinical centre in Tecpán Recruitment: a list of 150 TBAs from the study area was produced in collaboration with local health officials. Study duration: 12 months Study dates: January 2015 to February 2018 |
|
Participants |
Inclusion criteria: midwives working in the Tecpán municipality, who had attended ≥ 5 deliveries per year in the previous 5 years, and who held a valid license to practice issued by local health authorities Sample size: 44 TBAs randomized (intervention 23; control 21) Age, mean: intervention 47 (range 40–55) years; control 51 (range 43–55) years Sex: 100% female |
|
Interventions |
Intervention: mHealth decision‐support system to improve maternal and perinatal complication detection and referral rates to facility‐level care by TBAs. Content: the smartphone application allowed collection of simple demographics; maternal and perinatal symptoms and clinical signs; maternal vital signs (pulse, oxygen saturation, systolic and diastolic blood pressure); and the fetal heart rate. While using the application, TBAs were guided through a pictographic list of common maternal and perinatal complications grouped by visit type. Control: usual care. Co‐interventions: not reported. |
|
Outcomes | Number of monthly referrals to facility‐level care; proportion of referrals that were completed; adverse events Outcome assessment timepoints: 7 months |
|
Notes |
Funding: Eunice Kennedy Shriver National Institute of Child Health and Human Development and The Fogarty International Center at the National Institute of Health. Conflicts of interest: no competing interests declared. |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Unclear risk | Computer‐based randomization process. However, when > 1 TBA resided/practiced within the same settlement, they were allocated to the same study group. |
Allocation concealment (selection bias) | Low risk | The allocation and assignment procedures were performed by a study author prior to meeting participants and not otherwise involved in the recruitment of participants or daily conduct of field work. |
Blinding of participants and personnel (performance bias) all outcomes | High risk | Due to the pragmatic nature of a trial involving access to mHealth technology, TBAs, pregnant participants, or study personnel could not be blinded to allocation. |
Blinding of outcome assessment (detection bias) all outcomes | High risk | Due to the pragmatic nature of a trial involving access to mHealth technology, TBAs, pregnant participants, or study personnel could not be blinded to allocation. |
Incomplete outcome data (attrition bias) all outcomes | Low risk | 2 TBAs in the intervention group (9%) and 1 in the comparison group (5%) discontinued their participation in the study but continued to report outcomes data, allowing for inclusion in the intention‐to‐treat analysis. |
Selective reporting (reporting bias) | Unclear risk | All relevant outcomes reported, protocol checked. |
Other bias | Low risk | No other bias apparent. |
Tian 2015.
Study characteristics | ||
Methods |
Aim: to develop and evaluate a simplified yet guideline‐based multifaceted intervention program for cardiovascular management among people with high cardiovascular risk delivered by CHWs with the aid of a mobile technology‐based EDSS in rural China and India. Study design: cluster randomized controlled trial Country: China and India Setting: community based, either village clinics or patient homes Cluster features: 27 villages from 15 townships (the administrative unit managing the village) in 2 counties of Tibet, China and 20 villages from 1 tehsil (an administrative unit for a group of villages) in Haryana State, India Recruitment: not reported Study duration: 1 year Study dates: January 2012 to March 2014 |
|
Participants |
Inclusion criteria: residents in the participating villages and having high cardiovascular risk (aged ≥ 40 years with self‐reported history of CHD, stroke, diabetes mellitus, systolic blood pressure ≥ 160 mmHg, or a combination of these) Sample size: 2086 participants, number of CHWs not reported Age, mean: intervention 59.7 (SD 11.7); control 60.4 (SD 11.8) Sex: intervention 65.4% female; control 66.8% female |
|
Interventions |
Intervention: an electronic decision‐support component to assist the CHWs on the follow‐up and management of their high‐risk patients. Content: prompts regarding the patient's medical history, new conditions, medication usage, current lifestyle habits, blood pressure, and the appropriateness for prescribing any of the target medications. Control: usual cardiovascular management programmes continued without additional intervention. Co‐interventions: not reported. |
|
Outcomes | Proportion of patient‐reported antihypertensive medication use; proportion of high‐risk people taking aspirin; systolic blood pressures of high‐risk people; proportion of current smokers; proportion of high‐risk people aware of the harms of a high‐salt diet; proportion of high‐risk people receiving monthly follow‐ups from the CHWs; proportion of high‐risk people hospitalized. Outcome assessment timepoints: 1 year |
|
Notes |
Funding: US National Heart, Lung, and Blood Institute, National Institutes of Health, Department of Health and Human Services; UnitedHealth Group Chronic Disease Initiative Conflicts of interest: none reported |
|
Risk of bias | ||
Bias | Authors' judgement | Support for judgement |
Random sequence generation (selection bias) | Low risk | Study‐independent staff generated the randomization pattern through a central computerized process. |
Allocation concealment (selection bias) | Low risk | Study‐independent staff generated the randomization pattern through a central computerized process. |
Blinding of participants and personnel (performance bias) all outcomes | High risk | No blinding as CDSS compared to usual care. |
Blinding of outcome assessment (detection bias) all outcomes | Unclear risk | Quote: "The outcomes were assessed with data collected during baseline and post intervention surveys from all high‐risk individuals in both intervention and control villages in a standardized manner." |
Incomplete outcome data (attrition bias) all outcomes | Low risk | Intention‐to‐treat analysis including all randomized participants. |
Selective reporting (reporting bias) | Low risk | All relevant outcomes reported. |
Other bias | Low risk | No other bias. |
ATP: Adult Treatment Panel; CDSS: clinical decision‐support system; CHASS: Community Health and Social Services Center; CHD: coronary heart disease; CHPS: community‐based Health Planning and Services; CHW: community health worker; EDDS: electronic decision‐support system; HC: health centre; HDL: high‐density lipoprotein; IMAI: Integrated Management of Adult Illnesses; IMAM: integrated management of acute malnutrition; MCI: Integrated Management of Childhood Illnesses; LDL: low‐density lipoprotein; mCDMSI: mHealth clinical decision‐making support intervention; NSAID: non‐steroidal anti‐inflammatory drug; PDA: personal digital assistant; RHP: rural health provider; SD: standard deviation; SMS: short message service; TBA: traditional birth attendant; USSD: unstructured supplementary service data; WHO: World Health Organization.
Characteristics of excluded studies [ordered by study ID]
Study | Reason for exclusion |
---|---|
Arts 2017 | Dependent on integration with electronic medical records/health tracking tools. |
Caballero‐Ruiz 2017 | Not used by healthcare provider. |
Carroll 2012 | Not accessible by or primarily used by mobile. |
Carroll 2018 | Dependent on integration with electronic medical records/health tracking tools. |
Field 2009 | Not primary care setting. |
Forrest 2013 | Dependent on integration with electronic medical records/health tracking tools. |
Gill 2011 | Dependent on integration with electronic medical records/health tracking tools. |
Gong 2019 | Decision support not major component of intervention. |
Gulliford 2019 | Dependent on integration with electronic medical records/health tracking tools. |
Holbrook 2011 | Dependent on integration with electronic medical records/health tracking tools. |
Karlsson 2017 | Dependent on integration with electronic medical records/health tracking tools. |
Linder 2009 | Dependent on integration with electronic medical records/health tracking tools. |
Litvin 2013 | Dependent on integration with electronic medical records/health tracking tools. |
Mann 2014 | Not accessible by or primarily used by mobile. |
McGinn 2013 | Dependent on integration with electronic medical records/health tracking tools. |
Melnick 2019 | Dependent on integration with electronic medical records/health tracking tools. |
Morgan 2011 | Not primary care setting. |
Nam 2012 | Not primary care setting. |
Nendaz 2010 | Not primary care setting. |
Peiris 2019 | Dependent on integration with electronic medical records/health tracking tools. |
Prabhakaran 2019 | Dependent on integration with electronic medical records/health tracking tools. |
Schaeffer 2019 | Not a randomized controlled trial. |
Sturkenboom 2008 | Dependent on integration with electronic medical records/health tracking tools. |
Tajmir 2017 | Not primary care setting. |
Tamblyn 2008 | Not used by healthcare provider. |
Tebb 2019 | Dependent on integration with electronic medical records/health tracking tools. |
Wright 2012 | Dependent on integration with electronic medical records/health tracking tools. |
Characteristics of studies awaiting classification [ordered by study ID]
de Molina‐Férnandez 2019.
Methods | Randomized controlled trial |
Participants | 708 women (intervention group 854; control group 854) |
Interventions | Contraceptive counselling |
Outcomes | Adherence to treatment; decisional conflict; satisfaction with the counsellor or clinician; test of knowledge prior to and after fieldwork |
Notes | The SHARECONTRACEPT tool, developed by previous phases of this project, is available at: decisionscompartides.gencat.cat/en/decidir-sobre/anticoncepcio_hormonal/ |
Keitel 2017.
Methods | Randomized controlled trial |
Participants | Children aged 2–59 months presenting with acute febrile illness to 9 outpatient clinics in Dar es Salaam, Tanzania |
Interventions | e‐POCT is a novel electronic algorithm based on current evidence; it guides clinicians through the entire consultation and recommends treatment based on a few clinical signs and POCT results, some performed in all patients (malaria rapid diagnostic test, haemoglobin, oximeter) and others in selected subgroups only (C‐reactive protein, procalcitonin, glucometer) |
Outcomes | Proportion of clinical failures; proportion with antibiotics prescribed on day 0; primary referrals; and severe adverse events |
Notes |
Khan 2020.
Methods | Cluster randomized controlled trial |
Participants | Patients aged ≥ 2 months with uncomplicated acute diarrhoea. |
Interventions | Electronic decision support (rehydration calculator). |
Outcomes | Rate of intravenous fluid ordered; intravenous fluid volumes; antibiotics and zinc ordered; clinical course; adverse events |
Notes |
e‐POCT: electronic point‐of‐care test; POCT: point‐of‐care test.
Characteristics of ongoing studies [ordered by study ID]
NCT03311399.
Study name | Using mHealth technology to identify and refer surgical site infections in Rwanda |
Methods | Randomized controlled trial |
Participants | Women undergoing caesarean‐section surgery at a rural hospital in Rwanda |
Interventions | Surgical site infection screening protocol, delivered by community health workers equipped with mHealth support |
Outcomes | Number of women with surgical site infection returning to care |
Starting date | 15 March 2017 |
Contact information | Robert Riviello, Brigham and Women's Hospital, Boston, Massachusetts, USA |
Notes |
Differences between protocol and review
We added 'Patient health behaviour' as a separate outcome from 'client health status.'
We did not use network meta‐analysis to statistically infer a non‐digital/non‐mobile comparison group by comparing a study that compared digital intervention to non‐digital intervention with another study that compared one type of a digital intervention with another type of a digital intervention, as the interventions and outcomes were not sufficiently homogeneous to be grouped. We identified no studies that were sufficiently similar to combine the outcomes and therefore, did not perform any meta‐analyses. Given the small number of trials included in the review, we did not assess possible publication biases. Given the few included studies, we did not perform any additional subgroup analyses to assess the variation in the delivery of the intervention across different groups or characteristics.
Contributions of authors
Conceiving the protocol: CG, TT, SL, SA, GM, MF.
Designing the protocol: CG, TT, SL, SA, NM, NH, MF, GM.
Co‐ordinating the review: SA, CG, SL.
Writing the review: SA, CG.
Providing general advice on the review: CG, SL.
Securing funding for the review: GM.
All review authors approved the text
Publication history
Review submitted to the editorial base for peer review whislt update searches were updated and incorporated: 10/09/2020
Review returned to authors following peer review: 09/02/2021
Revised review received by the editorial base: 15/04/2021
Review returned to authors following editorial comments and copy edit: 14/06/2021
Review submitted for publication: 29/06/2021
Review accepted for publication: 21/07/21
Contributions of the editorial base
EPOC managing editor: Co‐ordinated the editorial process and contributed to peer review (Chris Cooper)
EPOC internal referee (EPOC): peer‐review (Daniela Gonçalves Bradley)
Information specialist (EPOC): provided comments on the search approach (Paul Miller).
Sources of support
Internal sources
No sources of support provided
External sources
This work was funded by the UNDP‐UNFPA‐UNICEF‐WHO‐World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), a cosponsored program executed by the World Health Organization (WHO), Switzerland
Declarations of interest
SA: The author was commissioned by the WHO to conduct this review.
CG: I am an editor for Cochrane Effective Practice and Organisation of Care. I was not involved in the editorial process for this review.
TT: Works for the WHO.
NH: Since June 2016, I have been employed by Cochrane Response, an evidence services unit operated by Cochrane. Cochrane Response was contracted by the WHO to produce this review.
MS: None known.
GM: Owns stock in Apple computers.
NM: I previously worked for Enhanced Reviews Ltd, a company that conducts systematic reviews, mostly for the public sector. Since June 2016, I have been employed by Cochrane Response, an evidence services unit operated by Cochrane. Cochrane Response was contracted by the World Health Organization to produce this review.
MF: The author was commissioned by the WHO to conduct this review.
GM: Works for the WHO.
SL: I am the Joint Co‐ordinating Editor for Cochrane Effective Practice and Organisation of Care. I was not involved in the editorial process for this review.
Edited (no change to conclusions)
References
References to studies included in this review
Amoakoh 2019 {published data only}
- Amoakoh HB, Klipstein-Grobusch K, Agyepong IA, Zuithoff NP, Amoakoh-Coleman M, Kayode GA, et al. The effect of an mHealth clinical decision-making support system on neonatal mortality in a low resource setting: a cluster-randomized controlled trial. EClinicalMedicine 2019;12:31-42. [DOI] [PMC free article] [PubMed] [Google Scholar]
Berner 2006 {published data only}
- Berner ES, Houston TK, Ray MN, Allison JJ, Heudebert GR, Chatham WW, et al. Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial. Journal of the American Medical Informatics Association 2006;13(2):171-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
Eaton 2011 {published data only}
- Eaton CB, Parker DR, Borkan J, McMurray J, Roberts MB, Lu B, et al. Translating cholesterol guidelines into primary care practice: a multimodal cluster randomized trial. Annals of Family Medicine 2011;9(6):528-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gautham 2015 {published data only}
- Gautham M, Iyengar MS, Johnson CW. Mobile phone–based clinical guidance for rural health providers in India. Health informatics journal 2015;21(4):253-66. [DOI] [PubMed] [Google Scholar]
Heisler 2014 {published data only}
- Heisler M, Choi H, Palmisano G, Mase R, Richardson C, Fagerlin A, et al. Comparison of community health worker–led diabetes medication decision-making support for low-income Latino and African American adults with diabetes using e-health tools versus print materials: a randomized, controlled trial. Annals of Internal Medicine 2014;161(10 Suppl):S13-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
Keane 2018 {published data only}
- Frank T, Keane E, Roschnik N, Emary C, O'Leary M, Snyder L. Developing a mobile healthapp to manage acute malnutrition: a five-country experience. Field Exchange 2017;54:7. [Google Scholar]
- Keane E, Roschnik N, Chui J, Osman IA, Osman HM. Evaluation of mobile application to support the treatment of acutely malnourished children in Wajir county, Kenya. Field Exchange 2018;57:61. [Google Scholar]
Martinez 2018 {published data only}
- Martinez B, Ixen EC, Hall-Clifford R, Juarez M, Miller AC, Francis A, et al. mHealth intervention to improve the continuum of maternal and perinatal care in rural Guatemala: a pragmatic, randomized controlled feasibility trial. Reproductive health 2018;15(1):120. [DOI] [PMC free article] [PubMed] [Google Scholar]
Tian 2015 {published data only}
- Tian M, Ajay V, Dunzhu D, Hameed S, Li X, Liu Z, et al. A cluster-randomized controlled trial of a simplified multifaceted management program for individuals at high cardiovascular risk (SimCard Trial) in Rural Tibet, China, and Haryana, India. Circulation 2015;17:115. [DOI] [PMC free article] [PubMed] [Google Scholar]
References to studies excluded from this review
Arts 2017 {published data only}
- Arts DL, Abu-Hanna A, Medlock SK, Weert HC. Effectiveness and usage of a decision support system to improve stroke prevention in general practice: a cluster randomized controlled trial. PloS One 2017;12(2):e0170974. [DOI] [PMC free article] [PubMed] [Google Scholar]
Caballero‐Ruiz 2017 {published data only}
- Caballero-Ruiz E, García-Sáez G, Rigla M, Villaplana M, Pons B, Hernando ME. A web-based clinical decision support system for gestational diabetes: automatic diet prescription and detection of insulin needs. International Journal of Medical Informatics 2017;102:35-49. [DOI] [PubMed] [Google Scholar]
Carroll 2012 {published data only}
- Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A randomized controlled trial of screening for maternal depression with a clinical decision support system. Journal of the American Medical Informatics Association 2012;20(2):311-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Carroll 2018 {published data only}
- Carroll JK, Pulver G, Dickinson LM, Pace WD, Vassalotti JA, Kimminau KS, et al. Effect of 2 clinical decision support strategies on chronic kidney disease outcomes in primary care: a cluster randomized trial. JAMA Network Open 2018;1(6):e183377. [DOI] [PMC free article] [PubMed] [Google Scholar]
Field 2009 {published data only}
- Field TS, Rochon P, Lee M, Gavendo L, Baril JL, Gurwitz JH. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. Journal of the American Medical Informatics Association 2009;16(4):480-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
Forrest 2013 {published data only}
- Forrest CB, Fiks AG, Bailey LC, Localio R, Grundmeier RW, Richards T, et al. Improving adherence to otitis media guidelines with clinical decision support and physician feedback. Pediatrics 2013;131(4):e1071-81. [DOI] [PubMed] [Google Scholar]
Gill 2011 {published data only}
- Gill JM, Mainous AG, Koopman RJ, Player MS, Everett CJ, Chen YX, et al. Impact of EHR-based clinical decision support on adherence to guidelines for patients on NSAIDs: a randomized controlled trial. Annals of Family Medicine 2011;9(1):22-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gong 2019 {published data only}
- Gong E, Gu W, Sun C, Turner EL, Zhou Y, Li Z, et al. System-integrated technology-enabled model of care to improve the health of stroke patients in rural China: protocol for SINEMA – a cluster-randomized controlled trial. American Heart Journal 2019;207:27-39. [DOI] [PubMed] [Google Scholar]
Gulliford 2019 {published data only}
- Gulliford MC, Prevost AT, Charlton J, Juszczyk D, Soames J, McDermott L, et al. Effectiveness and safety of electronically delivered prescribing feedback and decision support on antibiotic use for respiratory illness in primary care: REDUCE cluster randomised trial. BMJ 2019;364:l236. [DOI] [PMC free article] [PubMed] [Google Scholar]
Holbrook 2011 {published data only}
- Holbrook A, Pullenayegum E, Thabane L, Troyan S, Foster G, Keshavjee K, et al. Shared electronic vascular risk decision support in primary care: Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE III) randomized trial. Archives of Internal Medicine 2011;171(19):1736-44. [DOI] [PubMed] [Google Scholar]
Karlsson 2017 {published data only}
- Karlsson LO, Nilsson S, Charitakis E, Bång M, Johansson G, Nilsson L, et al. Clinical decision support for stroke prevention in atrial fibrillation (CDS-AF): rationale and design of a cluster randomized trial in the primary care setting. American Heart Journal 2017;187:45-52. [DOI] [PubMed] [Google Scholar]
Linder 2009 {published data only}
- Linder J, Schnipper J, Tsurikova R, Yu T, Volk L, Melnikas A, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Journal of Innovation in Health Informatics 2009;17(4):231-40. [DOI] [PubMed] [Google Scholar]
Litvin 2013 {published data only}
- Litvin CB, Ornstein SM, Wessell AM, Nemeth LS, Nietert PJ. Use of an electronic health record clinical decision support tool to improve antibiotic prescribing for acute respiratory infections: the ABX-TRIP study. Journal of General Internal Medicine 2013;28(6):810-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Mann 2014 {published data only}
- Mann D, Knaus M, McCullagh L, Sofianou A, Rosen L, McGinn T, et al. Measures of user experience in a streptococcal pharyngitis and pneumonia clinical decision support tools. Applied Clinical Informatics 2014;5(03):824-35. [DOI] [PMC free article] [PubMed] [Google Scholar]
McGinn 2013 {published data only}
- McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Internal Medicine 2013;173(17):1584-91. [DOI] [PubMed] [Google Scholar]
Melnick 2019 {published data only}
- Melnick ER, Jeffery MM, Dziura JD, Mao JA, Hess EP, Platts-Mills TF, et al. User-centred clinical decision support to implement emergency department-initiated buprenorphine for opioid use disorder: protocol for the pragmatic group randomised EMBED trial. BMJ Open 2019;9(5):e028488. [DOI] [PMC free article] [PubMed] [Google Scholar]
Morgan 2011 {published data only}
- Morgan MB, Branstetter BF 4th, Clark C, House J, Baker D, Harnsberger HR. Just-in-time radiologist decision support: the importance of PACS-integrated workflow. Journal of the American College of Radiology 2011;8(7):497-500. [DOI] [PubMed] [Google Scholar]
Nam 2012 {published data only}
- Nam HS, Cha M‐J, Kim YD, Kim EH, Park E, Lee HS, et al. Use of a handheld, computerized device as a decision support tool for stroke classification. European Journal of Neurology 2012;19(3):426-30. [DOI] [PubMed] [Google Scholar]
Nendaz 2010 {published data only}
- Nendaz MR, Chopard P, Lovis C, Kucher N, Asmis LM, Dörffler J, et al. Adequacy of venous thromboprophylaxis in acutely ill medical patients (IMPART): multisite comparison of different clinical decision support systems. Journal of Thrombosis and Haemostasis 2010;8(6):1230-4. [DOI] [PubMed] [Google Scholar]
Peiris 2019 {published data only}
- Peiris D, Praveen D, Mogulluru K, Ameer MA, Raghu A, Li Q, et al. SMARThealth India: a stepped-wedge, cluster randomised controlled trial of a community health worker managed mobile health intervention for people assessed at high cardiovascular disease risk in rural India. PloS One 2019;14(3):e0213708. [DOI] [PMC free article] [PubMed] [Google Scholar]
Prabhakaran 2019 {published data only}
- Prabhakaran D, Jha D, Prieto-Merino D, Roy A, Singh K, Ajay VS, et al. Effectiveness of an mHealth-based electronic decision support system for integrated management of chronic conditions in primary care: the mWellcare cluster-randomized controlled trial. Circulation 2019;139(3):380-91. [DOI] [PubMed] [Google Scholar]
Schaeffer 2019 {published data only}
- Schaeffer LE, Ahmed S, Rahman M, Whelan R, Rahman S, Roy AD, et al. Development and evaluation of a mobile application for case management of small and sick newborns in Bangladesh. BMC Medical Informatics and Decision Making 2019;19(1):116. [DOI] [PMC free article] [PubMed] [Google Scholar]
Sturkenboom 2008 {published data only}
- Wyk JT, Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, Lei J. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment. Circulation 2008;117(3):371-8. [DOI] [PubMed] [Google Scholar]
Tajmir 2017 {published data only}
- Tajmir S, Raja AS, Ip IK, Andruchow J, Silveira P, Smith S, et al. Impact of clinical decision support on radiography for acute ankle injuries: a randomized trial. Western Journal of Emergency Medicine 2017;18(3):487. [DOI] [PMC free article] [PubMed] [Google Scholar]
Tamblyn 2008 {published data only}
- Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. Journal of the American Medical Informatics Association 2008;15(4):430-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Tebb 2019 {published data only}
- Tebb KP, Trieu SL, Rico R, Renteria R, Rodriguez F, Puffer M. A mobile health contraception decision support intervention for Latina adolescents: implementation evaluation for use in school-based health centers. JMIR mHealth and uHealth 2019;7(3):e11163. [DOI] [PMC free article] [PubMed] [Google Scholar]
Wright 2012 {published data only}
- Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, McLoughlin KS, et al. Improving completeness of electronic problem lists through clinical decision support: a randomized, controlled trial. Journal of the American Medical Informatics Association 2012;19(4):555-61. [DOI] [PMC free article] [PubMed] [Google Scholar]
References to studies awaiting assessment
de Molina‐Férnandez 2019 {published data only}
- Molina-Férnandez MI, Raigal-Aran L, De la Flor-Lopez M, Prata P, Font-Jimenez I, Valls-Fonayet F, et al. The effectiveness of a digital shared decision-making tool in hormonal contraception during clinical assessment: study protocol of a randomized controlled trial in Spain. BMC Public Health 2019;19(1):1-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Keitel 2017 {published data only}
- Keitel K, Kagoro F, Samaka J, Masimba J, Said Z, Temba H, et al. A novel electronic algorithm using host biomarker point-of-care tests for the management of febrile illnesses in Tanzanian children (e-POCT): a randomized, controlled non-inferiority trial. PLoS Medicine 2017;14(10):e1002411. [DOI: 10.1371/journal.pmed.1002411] [PMID: ] [DOI] [PMC free article] [PubMed] [Google Scholar]
Khan 2020 {published data only}
- Khan AI, Mack JA, Salimuzzaman M, Zion MI, Sujon H, Ball RL, et al. Electronic decision support and diarrhoeal disease guideline adherence (mHDM): a cluster randomised controlled trial. Lancet Digital Health 2020;2(5):e250-8. [CLINICALTRIALS.GOV: NCT03154229] [DOI] [PMC free article] [PubMed] [Google Scholar]
References to ongoing studies
NCT03311399 {published data only}
- NCT03311399. Using mHealth technology to identify and refer surgical site infections in Rwanda. clinicaltrials.gov/ct2/show/NCT03311399 (first received 17 October 2017). [CLINICALTRIALS.GOV: NCT03311399]
Additional references
Adepoju 2017
- Adepoju IO, Albersen BJ, De Brouwere V, Roosmalen J, Zweekhorst M. mHealth for clinical decision-making in Sub-Saharan Africa: a scoping review. JMIR mHealth and uHealth 2017;5(3):e38. [DOI] [PMC free article] [PubMed] [Google Scholar]
Agarwal 2015
- Agarwal S, Perry HB, Long L-A, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Tropical Medicine & International Health 2015;20(8):1003-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
Agarwal 2016
- Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: Mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174. [DOI] [PubMed] [Google Scholar]
Arain 2010
- Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology 2010;10(1):67. [DOI] [PMC free article] [PubMed] [Google Scholar]
Awofeso 2004
- Awofeso N. What is the difference between 'primary care' and 'primary healthcare'? Quality in Primary Care 2004;12(2):93-4. [Google Scholar]
Baker 2007
- Baker B, Benton D, Friedman E, Russell A. Systems support for task-shifting to community health workers. chwcentral.org/sites/default/files/Systems%20Support%20for%20Task-Shifting%20to%20Community%20Health%20Workers.pdf (accessed 2 October 2017).
Baptista 2018
- Baptista S, Sampaio ET, Heleno B, Azevedo LF, Martins C. Web-based versus usual care and other formats of decision aids to support prostate cancer screening decisions: systematic review and meta-analysis. Journal of Medical Internet Research 2018;20(6):e9070. [DOI] [PMC free article] [PubMed] [Google Scholar]
Blank 2013
- Blank A, Prytherch H, Kaltschmidt H, Krings A, Sukums F, Mensah N, et al. "Quality of prenatal and maternal care: bridging the know-do gap" (QUALMAT study): an electronic clinical decision support system for rural Sub-Saharan Africa. BMC Medical Informatics and Decision Making 2013;13(1):44. [DOI] [PMC free article] [PubMed] [Google Scholar]
Brenner 2016
- Brenner SK, Kaushal R, Grinspan Z, Joyce C, Kim I, Allard RJ, et al. Effects of health information technology on patient outcomes: a systematic review. Journal of the American Medical Informatics Association 2016;23(5):1016-36. [DOI] [PMC free article] [PubMed] [Google Scholar]
Bright 2012
- Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Annals of Internal Medicine 2012;157(1):29-43. [DOI] [PubMed] [Google Scholar]
Carter 2019
- Carter J, Sandall J, Shennan AH, Tribe RM. Mobile phone apps for clinical decision support in pregnancy: a scoping review. BMC Medical Informatics and DEcision Making 2019;19:1. [DOI] [PMC free article] [PubMed] [Google Scholar]
DistillerSR [Computer program]
- DistillerSR. Ottawa (Canada): Evidence Partners, (accessed 15 December 2017).
Divall 2013
- Divall P, Camosso-Stefinovic J, Baker R. The use of personal digital assistants in clinical decision making by health care professionals: a systematic review. Health Informatics Journal 2013;19(1):16-28. [DOI] [PubMed] [Google Scholar]
Dreesens 2019
- Dreesens D, Kremer L, Weijden T. The Dutch chaos case: a scoping review of knowledge and decision support tools available to clinicians in the Netherlands. Health Policy 2019;123(12):1288-97. [DOI] [PubMed] [Google Scholar]
Dussault 2006
- Dussault G, Franceschini MC. Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Human Resources for Health 2006;4(1):12. [DOI] [PMC free article] [PubMed] [Google Scholar]
EPOC 2017a
- Cochrane Effective Practice and Organisation of Care (EPOC). Data collection form. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).
EPOC 2017b
- Cochrane Effective Practice and Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).
EPOC 2017c
- Cochrane Effective Practice and Organisation of Care (EPOC). EPOC worksheets for preparing a 'Summary of findings' table using GRADE. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed prior to 9 June 2021).
Global Health Watch 2011
- Global Health Watch. Primary health care: a review and critical appraisal of its revitalization. www.ghwatch.org/sites/www.ghwatch.org/files/B1_0.pdf (accessed 2 October 2017).
GRADEpro GDT [Computer program]
- McMaster University (developed by Evidence Prime) GRADEpro GDT. Version (accessed 15 December 2017). Hamilton (ON): McMaster University (developed by Evidence Prime), 2015. Available at gradepro.org.
Guyatt 2008
- Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al, GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336(7650):924-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Higgins 2017
- Higgins JP, Savović J, Page MJ, Elbers RG, Sterne JA. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). Cochrane, 2017. Available from: handbook: training.cochrane.org/handbook/archive/v5.2.
Hunt 1998
- Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280(15):1339:46. [DOI] [PubMed] [Google Scholar]
International Telecommunications Union 2015
- International Telecommunications Union. Information and Communication Technology (ICT) facts and figures. www.itu.int/en/ITU-D/Statistics/Documents/facts/ICTFactsFigures2015.pdf (accessed 2 October 2017).
Jaspers 2011
- Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association 2011;18(3):327-34. [DOI] [PMC free article] [PubMed] [Google Scholar]
Kaplan 2001
- Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. International Journal of Medical Informatics 2001;64(1):15-37. [DOI] [PubMed] [Google Scholar]
Kawamoto 2005
- Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494):765. [DOI] [PMC free article] [PubMed] [Google Scholar]
Kilsdonk 2017
- Kilsdonk E, Peute LW, Jaspers MW. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. International Journal of Medical Informatics 2017;1(98):56-64. [DOI] [PubMed] [Google Scholar]
Kocher 2010
- Kocher R, Emanuel EJ, DeParle NA. The Affordable Care Act and the future of clinical medicine: the opportunities and challenges. Annals of Internal Medicine 2010;153(8):536-9. [DOI] [PubMed] [Google Scholar]
Kruk 2018
- Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Global Health Commission 2018;6:e1196. [DOI: ] [DOI] [PMC free article] [PubMed] [Google Scholar]
Lefebvre 2011
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editor(s). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available at: handbook: training.cochrane.org/handbook/archive/v5.1/.
Lehmann 2008
- Lehmann U, Dieleman M, Martineau T. Staffing remote rural areas in middle- and low-income countries: a literature review of attraction and retention. BMC Health Services Research 2008;8(1):19. [DOI] [PMC free article] [PubMed] [Google Scholar]
Liberati 2009
- Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine 2009;6(7):e1000100. [DOI] [PMC free article] [PubMed] [Google Scholar]
Martínez‐Pérez 2014
- Martínez-Pérez B, la Torre-Díez I, López-Coronado M, Sainz-de-Abajo B, Robles M, García-Gómez JM. Mobile clinical decision support systems and applications: a literature and commercial review. Journal of Medical Systems 2014;38(1):4. [DOI] [PubMed] [Google Scholar]
Mazo 2020
- Mazo C, Kearns C, Mooney C, Gallagher WM. Clinical decision support systems in breast cancer: a systematic review. Cancers 2020;12(2):369. [DOI] [PMC free article] [PubMed] [Google Scholar]
McKenzie 2021
- McKenzie JE, Brennan SE. Synthesizing and presenting findings using other methods. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editors(s). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 . Cochrane, February 2021. [Google Scholar]
Mickan 2014
- Mickan S, Atherton H, Roberts NW, Heneghan C, Tilson JK. Use of handheld computers in clinical practice: a systematic review. BMC Medical Informatics and Decision Making 2014;14(1):56. [DOI] [PMC free article] [PubMed] [Google Scholar]
Middleton 2016
- Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearbook of Medical Informatics 2016;25(S01):S103-16. [DOI] [PMC free article] [PubMed] [Google Scholar]
Mishra 2019
- Mishra SR, Lygidakis C, Neupane D, Gyawali B, Uwizihiwe JP, Virani SS, et al. Combating non-communicable diseases: potentials and challenges for community health workers in a digital age: a narrative review of the literature. Health Policy and Planning 2019;34(1):55-66. [DOI] [PubMed] [Google Scholar]
Mohanan 2015
- Mohanan M, Vera-Hernández M, Das V, Giardili S, Goldhaber-Fiebert JD, Rabin TL, et al. The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India. JAMA Pediatrics 2015;169(4):349-57. [DOI] [PMC free article] [PubMed] [Google Scholar]
Moja 2014
- Moja L, Kwag KH, Lytras T, Bertizzolo L, Pecoraro V, Rigon G, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. American Journal of Public Health 2014;104(12):e12-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
Muldoon 2006
- Muldoon LK, Hogg WE, Levitt M. Primary care (PC) and primary health care (PHC): what is the difference? Canadian Journal of Public Health/Revue Canadienne de Santé Publique 2006;409(5):409-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
Njie 2015
- Njie GJ, Proia KK, Thota AB, Finnie RK, Hopkins DP, Banks SM, et al. Clinical decision support systems and prevention: a community guide cardiovascular disease systematic review. American Journal of Preventive Medicine 2015;49(5):784-95. [DOI] [PMC free article] [PubMed] [Google Scholar]
Odendaal 2020
- Odendaal WA, Watkins JA, Leon N, Goudge J, Griffiths F, Tomlinson M, et al. Health workers' perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews 2020, Issue 1. Art. No: CD011942. [DOI: 10.1002/14651858.CD011942.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]
Orton 2018
- Orton M, Agarwal S, Muhoza P, Vasudevan L, Vu A. Strengthening delivery of health services using digital devices. Global Health: Science and Practice 2018;6(1):S61-71. [DOI] [PMC free article] [PubMed] [Google Scholar]
Rednick 2014
- Rednick C, Dini HS, Long LA. The current state of CHW training programs in Sub-Saharan Africa and South Asia: what we know, what we don't know, and what we need to do. 1millionhealthworkers.org/files/2013/01/1mCHW_mPowering_LitReview_Formatted.compressed.pdf (accessed 2 October 2017).
Review Manager 2014 [Computer program]
- Nordic Cochrane Centre, The Cochrane Collaboration Review Manager 5 (RevMan 5). Version 5.3. Copenhagen: Nordic Cochrane Centre, The Cochrane Collaboration, 2014.
Richardson 2019
- Richardson KM, Fouquet SD, Kerns E, McCulloh RJ. Impact of mobile device-based clinical decision support tool on guideline adherence and mental workload. Academic Pediatrics 2019;19(7):828-34. [DOI] [PMC free article] [PubMed] [Google Scholar]
Schünemann 2017
- Schünemann HJ, Oxman AD, Higgins JP, Vist GE, Glasziou P, Akl E, et al, on behalf of the Cochrane GRADEing Methods Group and the Cochrane Statistical Methods Group. Chapter 11: Completing 'Summary of findings' tables and grading the confidence in or quality of the evidence. In: Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). Cochrane, 2017. Available from: handbook: training.cochrane.org/handbook/archive/v5.2.
Sterne 2011
- Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002. [DOI: 10.1136/bmj.d4002] [DOI] [PubMed] [Google Scholar]
Sutton 2020
- Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, strategies for success. npj Digital Medicine 2020;3:17. [DOI] [PMC free article] [PubMed] [Google Scholar]
Syrowatka 2016
- Syrowatka A, Krömker D, Meguerditchian AN, Tamblyn R. Features of computer-based decision aids: systematic review, thematic synthesis, and meta-analyses. Journal of Medical Internet Research 2016;18(1):e20. [DOI] [PMC free article] [PubMed] [Google Scholar]
Tong 2021
Varghese 2018
- Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review. Journal of the American Medical Informatics Association 2018;25(5):593-602. [DOI] [PMC free article] [PubMed] [Google Scholar]
Wallace 2017
- Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. Journal of the American Medical Informatics Association 2017;24(6):1165-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
WHO 2016
- World Health Organization. Health workforce requirements for universal health coverage and sustainable development goals. apps.who.int/iris/bitstream/handle/10665/250330/9789241511407-eng.pdf (accessed 25 March 2021).
WHO 2019
- World Health Organization. WHO guideline: recommendations on digital interventions for health systems strengthening, 2019. www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en/ (accessed 9 June 2021). [PubMed]
WHO 2020
- World Health Organization. Global strategy on human resources for health: Workforce 2030. www.who.int/hrh/resources/global_strategy_workforce2030_14_print.pdf (accessed 25 March 2021).
Yau 2019
- Yau M, Timmerman V, Zwarenstein M, Mayers P, Cornick RV, Bateman E, et al. e-PC101: an electronic clinical decision support tool developed in South Africa for primary care in low-income and middle-income countries. BMJ Global Health 2019;3:e001093. [DOI] [PMC free article] [PubMed] [Google Scholar]
References to other published versions of this review
Agarwal 2018
- Agarwal S, Tamrat T, Glenton C, Lewin S, Henschke N, Maayan N, et al. Decision‐support tools via mobile devices to improve quality of care in primary healthcare settings. Cochrane Database of Systematic Reviews 2018, Issue 2. Art. No: CD012944. [DOI: 10.1002/14651858.CD012944] [DOI] [PMC free article] [PubMed] [Google Scholar]