Effects of computerised clinical decision support systems (CDSS) on nursing and allied health professional performance and patient outcomes

Objectives

This is a protocol for a Cochrane Review (intervention). The objectives are as follows:

Main objective: to assess the effects of computerised clinical decision support systems (CDSS) use on nurses' and allied health professionals’ performance and patient outcomes, by comparing those who use CDSS and those who do not use CDSS.

Secondary objective: to assess the resource use associated with CDSS used by nurses and allied health professionals.

Background

In modern health systems, nurses and allied health professionals (AHPs) are responsible for clinical judgements and decisions that commit financial, human and technical resources to patient care. For example, a community nurse or podiatrist may have to judge if a housebound patient with a diabetic foot ulcer that is increasingly red, painful and with obvious exudate is displaying signs and symptoms of infection and decide on possible courses of action. AHPs such as speech and language therapists, occupational therapists and physiotherapists often act as “gatekeepers”, using their judgement to assess need for, and access to, specialist rehabilitative equipment and budgets. Alongside the development and growth of digital technology to enable healthcare delivery generally (Duggal 2018) (such as electronic health records, or e‐rostering systems), computerised clinical decision support systems (CDSS) have the potential to provide tailored, evidence‐based, advice to nurses and AHPs to inform their clinical decision making (Dunn‐Lopez 2016; Middleton 2016).

Description of the condition

Modern nursing and AHP roles in multi‐disciplinary healthcare delivery can involve decisions and judgements about screening, diagnosis, management and preventive care (Higgs 2008). Decisions and judgements made by nurses and AHPs sometimes vary – even for similar patients. Some of this variation is unwarranted (Chang 2002). Clinical decision making involves managing uncertainty, including uncertainty arising from imperfect or missing research evidence, varying patient and carers' preferences, and conflicting guidance and information (Thompson 2004). Since Eddy’s seminal work on variations in practice in the 1980s, researchers have known that clinicians’ experience of uncertainty in decision making and variations in practice are linked conceptually and empirically (Eddy 1984). To manage uncertainties when faced with unaided decision making, nurses and AHPs — like all clinicians — use cognitive shortcuts or heuristics (Cioffi 1997). These heuristics can be useful but are also associated with systematic biases and errors (Cioffi 1997; Mannion 2014; Harenčárová 2017).

Description of the intervention

Computerised clinical decision support systems are software‐ or computer‐based technologies that offer patient‐specific recommendations based on research, expert opinion, machine learning/artificial intelligence or combinations of these, designed to influence the clinical decision making of health professionals (Sim 2001; Sutton 2020). CDSS access patient information from practitioners, healthcare staff, patient’s manual data entry, queries of electronic medical records before research or expert knowledge: this information is evaluated to provide computer‐generated assessments or recommendations delivered to the clinician via a computer, tablet, phone or electronic medical record. The clinician can then choose whether to use the computer‐generated recommendations. Examples of when decision support is used by nurses and AHPs include: assessing fall risk and preventative behaviours (Lytle 2015); pressure ulcer management (Khong 2015); selecting interventions for managing musculoskeletal disorders (Gross 2016); screening for language disorders in children (Ruiz 2014); depression screening (Mahabir 2014); and choices within clinical pathways for primary care, such as triage and prioritisation (NHS 2020).

Computerised clinical decision support systems come in knowledge‐based and non‐knowledge‐based forms (Berner 2007). Knowledge‐based CDSS use logical “IF‐THEN‐ELSE” rules to evaluate information that has been entered by a clinician or drawn from an electronic health record and then matched to a computerised knowledge base (in many cases, this consists of expert opinion or national/international clinical practice guidelines) to provide assessments (management options or probabilities) or actionable recommendations or outputs (Cresswell 2012). These forms of CDSS automate information gathering and provide advice in line with guidelines. Examples of this type of CDSS are: drug prescription/ alert tools (Kuperman 2007) and emergency and out‐of‐hour telephone calls used for triaging patients (Randell 2007). Non‐knowledge‐based CDSS use machine learning and artificial intelligence rather than logical rules to support clinicians’ decision making (Berner 2007). Typical examples of this type of CDSS are predictive risk models for assessing disease prognosis (Rahbari 2011).

Computerised clinical decision support systems can be stand‐alone or integrated into digital infrastructure in health systems such as electronic health records (EHRs) or computerised physician order entry (CPOE). They can be hosted via a desktop/laptop, tablet or smartphone, and can be web‐based on a local system or as a phone or computer “app”. CDSS may present information on host devices or via an integrated EHR/CPOE system.

How the intervention might work

The focus for decision support systems are the judgements and decisions that inform the processes of care delivery. Thus, for a clinical decision support system to “work” it must be used by nurses and AHPs to systematically improve the quality of their decision‐based care or management, and contribute to reducing inappropriate variations in processes of care or management.

Computerised clinical decision support systems aim to provide high‐quality, relevant and useful information for decision makers, at the points it is needed (Sutton 2020). CDSS help nurses and AHPs make more effective decisions by facilitating the combination of CDSS‐generated information with a clinician’s professional knowledge. They supplement the clinical decision making of nurses and AHPs, rather than replace it. CDSS are often used to encourage concordance with clinical guidelines, to encourage standardisation of care and reduce unwarranted variations in practice (Coiera 2006). Examples of the kinds of decisions that might be supported by CDSS include the following (Andersson 2014; Dunn‐Lopez 2016).

• Recognising patient deterioration. CDSS can increase situational awareness, and facilitate relevant clinical information gathering and application of research‐based information, local or national guidance to assessment.

• Proposing a course of action for patients with conditions that merit the application of clinical guidelines. CDSS improve the consistency of judgements, adherence to guideline recommendations and reduction in (unwarranted) variation.

• Triaging patients and prioritisation decisions (for example, in emergency and pre‐hospital settings). As well as providing guideline‐based recommendations, CDSS might improve judgement reliability and simplify choices by reducing “noise” in the situation and amplifying appropriate “signals” — such as risk or “red flag” signs and symptoms — to encourage more accurate and reliable prioritisation decisions and fewer adverse events.

Recommendations from CDSS cannot improve decision making unless they are implemented and the systems are used by nurses and AHPs (Osheroff 2012 ). Studies suggest CDSS implementation and use of decision support systems by nurses and AHPs is rarely straightforward and can be suboptimal (Randell 2007; Dowding 2009; Porter 2018). 

Computerised clinical decision support systems can create potential harms as well as benefits, for example: disruption of work and workflow; alert fatigue; clinician deskilling; or the application of poor‐quality CDSS knowledge, logic or algorithms (Sutton 2020). CDSS may rely on clinicians’ computer literacy, something that is highly variable in the healthcare workforce. Systems can generate opportunity costs for clinicians and those who maintain and support technology in health systems (Sutton 2020). CDSS can also widen existing inequalities in access to high‐quality care. For example, implementing CDSS only in prestigious teaching hospitals may disadvantage those patients who are less likely to access such hospitals (Langhorne 2020).

There are three main mechanisms by which CDSS might improve processes of decision‐based care in nurses and AHPs. These are: combining high‐quality or novel information with clinicians' knowledge; promoting appropriate management/treatment choices, and accuracy of predictions or diagnoses; and successful implementation and use by clinicians. 

Why it is important to do this review

Numerous syntheses, including reviews of reviews (Lau 2010), have examined research evidence for computerised decision support. These vary in their focus: some assess the effects of CDSS on a broad range of healthcare provider processes and outcomes (Lau 2010; Jaspers 2011; Kwan 2020Leon 2020), whereas others focus on specific healthcare functions (such as lipid control (Aspry 2013)) or specific healthcare domains, such as primary (Bryan 2008) or neonatal care (Tan 2005). There are also reviews on CDSS characteristics that encourage adoption and use (Kawamoto 2005; Van de Velde 2018), and the economics of investing in decision support systems (Bassi 2013). These syntheses suggest that the effects of CDSS on processes and outcomes of care are mixed. CDSS improve some processes of care, such as prescribing, test ordering, documentation and — importantly for nurses and AHPs — guideline concordance (Kwan 2020), but improvements are modest. A recent high‐quality systematic review suggests 5.8% absolute improvements in patients receiving recommended care may be possible, but with widely variable and significant reported effects (Kwan 2020). In many reviews of the effects of CDSS, providers and clinicians are treated as largely homogeneous, when in fact they constitute different groups of professionals with distinct knowledge (that interacts with CDSS‐generated information), work (and associated decisions that provide the focus for CDSS) and training in decision making.

Reviews that examine the effect of CDSS on outcomes of care highlight the uncertainty of the evidence (Garg 2005; Kwan 2020) but also conflate healthcare professional groups. This ignores the possibility that differing professional group members of multi‐disciplinary teams make a unique contribution to the overall effects reported; an example would be nurses providing lipid control advice in a bigger team (Aspry 2013). The mediating impact of individual professions on the effect of CDSS may be as important as characteristics such as periodic performance feedback or providing decision support results to patients (Kawamoto 2005).

In a “qualitative meta‐analysis”, Rahimi 2008 highlighted the need for professional groups in the team to be convinced of worth of the CDSS, and the health information provided, if they were to work collaboratively to deliver care. The potential tension between CDSS‐generated digital knowledge and the experiential knowledge and information that is most valued, and therefore used, by nurse users of a national CDSS system was highlighted by Hanlon and colleagues (Hanlon 2016). The mediating effect of individual professionals on CDSS is proposed as a possible mechanism for the lack of effects on processes and outcomes seen in trials of lipid control (Aspry 2013). Whilst nurses or AHPs, or both, may act as effect modifiers in primary studies included in syntheses, their unique contribution, work, and decisions are often unaccounted for in reviews.

Differences in training, culture, organisational processes and other cognitive and socio‐technical variables have been hypothesised as explanatory factors explaining variation in CDSS outcomes (Coiera 2006). The separate professional groups in the teams that characterise modern healthcare delivery have unique and separate professional preparation and qualifications, cultures and managerial and organisational structures that reflect historical differences in structural power, the division of labour in health care and professional and socio‐economic status (ONS 2020;Nancarrow 2005).   

Reviews of varying types have examined the effect of CDSS on nurses’ performance and outcomes (Anderson 2008; Lee 2013; Piscotty 2014; Dunn‐Lopez 2016). We are aware of a single systematic review of studies that meets Cochrane Effective Practice and Organisation of Care (EPOC) review criteria for included study designs (Randell 2007). This 14‐year‐old review found the effects of CDSS on nursing processes of care were equivocal and outcomes uncertain. It highlighted issues such as the influence of users, and methodological characteristics such as units of randomisation in trials of CDSS; and the review called for more research into the uses of CDSS (Randell 2007). To the best of our knowledge, no systematic review has examined the effects of CDSS on AHP performance and outcomes.

Medical, nursing and AHP roles often overlap and evaluations of CDSS for nurses and AHPs reflect this. Randell 2007 focused on studies comparing CDSS‐supported decisions made by nurses with non‐CDSS‐supported decisions made by doctors, in the area of anticoagulation therapy and first‐contact triage. CDSS‐informed decisions are a function, however, not just of the CDSS system itself (its data, knowledge base, inference engine, decision rules and algorithms) but also the decision maker’s knowledge and decision‐related expertise (Sintchenko 2006 ). Important differences exist between nurses, AHPs and doctors. These can be socioeconomic (doctors generally are afforded higher professional status, which is reflected in areas such as decisional autonomy), educational (doctors have a university degree in most health systems, whilst a wholly graduate nursing and AHP workforce does not exist in all countries), and knowledge‐based (variations in the knowledge underpinning professional practice and decision making) (Thompson 2017). In a systematic review of role substitution in primary care, decision‐making processes (length of time taken, amounts of information requested) was a key difference between doctors and nurse practitioners (Laurant 2018). CDSS must work with decision processes, rather than against them, for successful adoption (Kawamoto 2005; Randell 2007; Dowding 2009). Thus, the impact of potential differences in decision making and CDSS effectiveness due to differences in work, individual and professional characteristics makes this review necessary.

Objectives

Main objective: to assess the effects of computerised clinical decision support systems (CDSS) use on nurses' and allied health professionals’ performance and patient outcomes, by comparing those who use CDSS and those who do not use CDSS.

Secondary objective: to assess the resource use associated with CDSS used by nurses and allied health professionals.

Methods

Criteria for considering studies for this review

Types of studies

We will consider for inclusion any studies that meet Cochrane Effective Practice and Organisation of Care Group (EPOC) definition of a randomised controlled trial; that is, an experimental study in which people are allocated to different interventions using methods that are random (EPOC 2017a).

Types of participants

Participants will be:

• nurses (including midwives, nurse assistants, health or clinical assistants);

• allied health professionals (AHPs), which includes: art therapists, drama therapists, music therapists, chiropodists/podiatrists, dietitians, occupational therapists, anaesthesiologists, orthoptists, osteopaths, paramedics, physiotherapists, prosthetists and orthotists, radiographers and speech and language therapists).

We will include those who are qualified and those in postgraduate training, whether they work in primary or secondary healthcare settings.

Types of interventions

We will seek to identify randomised controlled trials comparing CDSS‐aided clinical decisions to routine decisions  (i.e. those unaided by CDSS) made by nurses or AHPs. Computerised clinical decision support is defined as digitised job aids that combine an individual’s health information with the healthcare provider’s knowledge and clinical protocols in order to assist healthcare providers in making diagnosis and treatment decisions (WHO 2019) and improve clinical decision‐making. An example is UpToDate (www.uptodate.com). CDSS must provide patient‐specific advice to nurses or AHPs, or both (Haynes 2010). We will not examine the effects of computerised systems that summarise patient information only; non‐computerised clinical decision support systems (such as paper versions of flowcharts); those that provide electronic access to information only (e.g. in‐ward access to online databases of research abstracts, such as MEDLINE); or systems producing recommendations for groups of patients or populations.

The comparator is usual care (or non‐use of CDSS), defined as clinical practice where clinical decision making is unsupported by CDSS. Studies assessing clinicians’ CDSS‐aided diagnostic performance against a defined reference standard alone will not be included.

Types of outcome measures

Primary outcomes

The primary outcomes for the review reflect improved decision making by nurses and AHPs using CDSS.

Processes of care outcomes:

• adherence to guidelines (for example, the proportion or numbers of guideline recommendations followed);

• improved decision‐specific performance (for example, improved diagnostic accuracy or risk prediction). 

Patient care outcomes:

• improvement in CDSS‐targeted outcomes (for example, numbers of falls for systems trying to reduce falls, mean blood glucose level or time in target range for systems targeting diabetes management);

• CDSS safety (for example, numbers of adverse events associated with CDSS use).  

Implementation outcomes:

• degree of use (for example, the proportion of decisions informed by CDSS);

• satisfaction with CDSS;

• decisional conflict, as measured by scales such as the Decisional Conflict Scale (Légaré 2012).

Secondary outcomes

The secondary outcomes of the review are indicators of resource use associated with CDSS:

• length of hospital stay in days;

• number of outpatient attendances;

• direct medical resource use.

• Ratios of incremental cost and incremental effects

Search methods for identification of studies

Electronic searches

We will search the following electronic databases from their inception to present:

• MEDLINE and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Daily (Ovid) (1946 to date of search);

• MEDLINE(Ovid) (1946 to date of search);

• Embase Classic and Embase (Ovid) (1947 to date of search);

• Health Management Information Consortium (HMIC) (Ovid) (1983 to date of search);

• Cumulative Index to Nursing and Allied Health Literature (CINAHL) (Ebsco) (1981 to date of search);

• Cochrane Central Register of Controlled Trials (Wiley);

• Cochrane Database of Systematic Reviews (Wiley);

• Social Sciences Citation Index Expanded (Clarivate) (1900 to date of search).

The search will not be limited by language. We will search using terms and related synonyms associated with two main concepts:

• computerised clinical decision support systems;

• clinicians (AHPs and nurses).

See Appendix 1 for the MEDLINE search strategy as an example.

Searching other resources

We will search the following trial registries: 

• World Health Organization International Clinical Trials Registry Platform (WHO ICTRP; www.who.int/ictrp), to the date of the search;

• US National Institutes of Health Ongoing Trials Register, ClinicalTrials.gov (www.clinicaltrials.gov), to the date of the search).

We will search the following sources of grey literature:

• Health Services Research Projects in Progress (HSRPROJ);

• Conference Proceedings Citation Index: Social Science;

• ProQuest Dissertations, Theses, Abstracts and Indexes;

• OpenGrey (www.opengrey.eu), to the date of the search;

• OpenClinical (www.Openclinical.org);

• Agency for Healthcare Research and Quality (AHRQ; www.ahrq.gov), to the date of the search.

We will also review reference lists of all included studies and relevant systematic reviews for additional potentially eligible primary studies. We will contact authors of included studies/reviews to clarify reported published information and to seek unpublished results and data.

Data collection and analysis

Selection of studies

All titles and abstracts will be moved to a reference management database and duplicates removed. We will use the Covidence review production toolkit (www.covidence.org) to manage screening, extraction and organising of the review and ensure efficient production. Six review authors (AMK, CT, TM, RR, HY and HK) will independently screen titles and abstracts for inclusion. Full‐text study reports/publications will be retrieved and two review authors (CT and TM) will independently screen the full‐text papers and identify studies for inclusion and exclusion. Reasons for exclusion of the ineligible studies will be recorded. We will resolve any disagreement through discussion or, if required, we will consult a third review author (RR).

Studies initially appearing to meet the inclusion criteria but that we later exclude will be listed in the 'Characteristics of excluded studies' table. Multiple reports of the same study will be collated so that each study, rather than each study report, is the unit of interest in the review. Any information we can obtain about ongoing studies will be provided and our selection process will be recorded using a PRISMA flow diagram (Liberati 2009).

Data extraction and management

We will use a revised version of the standard EPOC data collection form, adapted for study characteristics and reported outcomes (EPOC 2017b). The form will be piloted on at least one study in the review. Two review authors (CT and TM) will independently extract data using the following categories.

1. Study methods: trial design; number of study sites; location and setting (primary care, acute care, critical care).

2. Trial participants: patients (and their underlying medical conditions, for example people with diabetes) and professionals. For professionals, we will extract variables known to impact decision performance, i.e. their age, educational attainment, gender, general and specific clinical experience. The generalist or specialist nature of professionals involved will also be extracted (for example, general community nurses or specialist tissue viability nurses).

3. CDSS interventions and associated comparison group: means of delivering recommendations (for example, personal computer, laptop, mobile phone); underlying uncertainty the CDSS is targeting (diagnosis, management/treatment, both); knowledge‐based system (for example, digital version of flowchart‐type rules mapped to a guideline) or non‐knowledge‐based system (machine learning) or some hybrid of the two; control group comparability (similar professionals, patients and time frames). Because the theoretical basis for CDSS can impact on usability and effectiveness, we will also extract information on the theoretical basis of any of the CDSS reported: for example, Bayesian versus frequentist approaches to probability; incorporation of any psychological de‑biasing (for example, contextualised performance feedback) alongside CDSS‐derived recommendations.

4. Outcomes: primary outcomes (changes in processes of care; patient outcomes including adverse effects; and implementation outcomes) and secondary outcomes (resource use; and information on costs and any economic evaluation for incorporation in a brief economic commentary in the review's discussion).

5. Equity: CDSS have the potential to advantage/disadvantage patients, for example, by using recommendations to improve access to services or social capital or restrict resource allocation. We will extract data on characteristics that stratify health opportunities and outcomes, in line with PROGRESS‐Plus (https://methods.cochrane.org/equity/projects/evidence-equity/progress-plus).

Two review authors (CT, TM) will independently extract outcome data from included studies. We will note in the 'Characteristics of included studies' table if outcome data were reported in an unusable way. We will resolve disagreements by consensus or by involving a third review author (RR).

Assessment of risk of bias in included studies

Two review authors (CT and TM) will independently assess risk of bias for each study using the criteria outlined in the Cochrane Handbook for Systematic Reviewsof Interventions (Higgins 2019), and the guidance from Cochrane EPOC (EPOC 2017c). We will resolve any disagreements by discussion or by involving a third review author (RR). We will assess the risk of bias according to the following domains:

1. random sequence generation;

2. allocation concealment;

3. blinding of participants and personnel;

4. blinding of outcome assessment;

5. incomplete outcome data;

6. selective outcome reporting;

7. baseline outcomes measurement;

8. baseline characteristics;

9. other bias.

We will judge each study as being at high, low, or unclear risk of bias for the domains listed above, and we will provide a quote from the study report together with a justification for our judgement in the 'Risk of bias' table. 'Risk of bias' judgements will be summarised across different studies for each of the domains listed. An overall 'Risk of bias' assessment (high, moderate or low) for each included study, using the approach suggested in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2019), will be undertaken. Studies with a low risk of bias in all key domains, or where it seems unlikely that bias will seriously alter the results, will be considered as having a low risk of bias. Studies where risk of bias in at least one domain was unclear or judged to have some bias that could plausibly raise doubts about the conclusions, will be considered as having an unclear risk of bias. We will consider studies with a high risk of bias in at least one domain, or judged to have serious bias that decreases the certainty of the conclusions, to have a high risk of bias.

We will consider blinding separately for different key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all‐cause mortality may be very different than for a patient‐reported pain scale). Where information on risk of bias relates to unpublished data or correspondence with a trialist, this will be noted in the 'Risk of bias' table. We will not exclude studies solely on the grounds of risk of bias but will clearly report the risk of bias when presenting the results of the studies.

When considering treatment effects, we will take into account the risk of bias for the studies that contribute to that outcome.

The review will be conducted according to this protocol and we will report any deviations from it in the 'Differences between protocol and review' section of the systematic review.

Measures of treatment effect

We will estimate the effect of the intervention using risk ratio/risk difference for dichotomous data, together with the appropriate associated 95% confidence interval (CI); and mean difference or standardised mean difference for continuous data, together with the 95% appropriate associated CI (Higgins 2019) . We will ensure that an increase in scores for continuous outcomes can be interpreted in the same way for each outcome, and we will explain the direction to the reader and report where the directions were reversed, if this was necessary.

Unit of analysis issues

For cluster‐randomised trials, only results based on cluster‐adjusted analysis will be extracted, or re‐analysis will be conducted if such results are not reported by authors.

Dealing with missing data

Study investigators will be contacted in order to verify key study characteristics and obtain missing outcome data where possible (e.g. when a study is identified as abstract only). We will try to compute missing summary data from other reported statistics. Whenever it is not possible to obtain data, we will report the level of missing data and consider how that might impact the certainty of the evidence.

Assessment of heterogeneity

If we find enough studies where we judge the participants, interventions, comparisons and outcomes to be sufficiently similar, we will conduct a meta‐analysis (Borenstein 2009). We will use the I² statistic to measure heterogeneity among the trials in each analysis. We will deem there to be low heterogeneity where I² is between 0% and 25%, medium heterogeneity where I² is between 25% to 75%, and high heterogeneity where  I² is above 75% (Higgins 2019). If we identify substantial heterogeneity, we will explore it by pre‐specified subgroup analysis. Where studies report variables that relate to professional knowledge, levels of education (a known factor in non‐medical professional performance and outcomes) or specific skills training associated with judgement or decision making in practitioners, then we will examine the association (where possible) between variable and CDSS outcomes reported.

Assessment of reporting biases

We will attempt to contact study authors to ask them to provide missing outcome data. Where this is not possible, and the missing data are thought to introduce serious bias, we will explore the impact of including such studies in the overall assessment of results. If we can pool more than 10 trials, we will create and examine a funnel plot to explore possible publication biases, interpreting the results with caution (Sterne 2011).

Data synthesis

Meta‐analysis will be conducted if it will be meaningful i.e. if the treatments, participants, and the underlying clinical question are similar enough for pooling to make sense (Borenstein 2009). For dichotomous outcomes, we will generate a pooled risk ratio with associated 95% CIs. For continuous outcomes where the variable of interest has been measured using the same instrument across studies, we will generate a pooled mean difference with 95% CIs. For continuous outcomes where different instruments have been used to measure a common phenomenon (e.g. change in professional performance, change in functional improvement) we will combine outcome data using standardised mean difference (SMD) with 95% CIs. For time‐to‐event data, we will pool estimates of hazard ratios and 95% CIs. We will use the generic inverse variance method when synthesising data.

Estimates from cluster‐randomised trials will be incorporated into meta‐analyses using methods from published guidance in order to take account of any clustering effect (Higgins 2019). Where meta‐analysis of reported data is not possible, our approach will be informed by the Synthesis without Meta‐Analysis (SWiM) reporting guideline for data synthesis (Campbell 2020).

We will develop a brief economic commentary based on current methods guidelines (methods.cochrane.org/economics) to summarise the availability and principal findings of trial‐based and model‐based full economic evaluations (cost‐effectiveness analyses, cost‐utility analyses, cost‐benefit analyses)* that compare CDSS use by nurses and AHPs to appropriate control groups (healthcare professionals not using CDSS). This commentary will focus on the extent to which principal findings of eligible economic evaluations indicate that an intervention might be judged favourably (or unfavourably) from an economic perspective, when implemented in different settings.

* a definition of these terms can be found in the Glossary and a fuller explanation is provided in the supplementary material on the Campbell and Cochrane Economics Methods Group website.

Where they exist, we will summarise economic costs and outcomes according to GRADE recommendations for presenting economic data (Brunetti 2013). The perspective adopted will depend, in part, on the perspectives of the underlying studies, but where possible we will adopt the perspective of the practitioners making clinical decisions (i.e. at the hospital or service level). Target outcome measures will, where possible, include incremental cost‐effectiveness ratios (ICERs) and cost‐benefit ratios.

Subgroup analysis and investigation of heterogeneity

Where sufficient numbers of comparisons for similar outcomes across studies exist, heterogeneity of results will be explored graphically across studies. The visual analyses will be supplemented with multivariate statistical analyses (meta‐regression), if appropriate, to examine how the size of observed effects are related to the specified explanatory factors. The following factors will be considered for subgroup analysis:

1. care setting (primary care versus hospital);

2. country of health system and economic status (for example, low‐ and middle‐income countries versus high‐income economies (WB 2020);

3. type of care (promotion, treatment, rehabilitation);

4. health professionals involved (nurses versus AHPs; generalists versus specialists);

5. type of intervention (customized checklist versus a machine learning risk prediction algorithm).

We will conduct subgroup analyses for all primary and secondary outcomes (see Types of outcome measures).

Sensitivity analysis

Subject to having sufficient data, we will undertake sensitivity analyses in which we will restrict inclusion of studies to those with low risk of bias overall. Estimates will be compared to those from meta‐analyses of all studies to determine the robustness of pooled estimates of effect. In order to assess the presence of publication bias, we will present funnel plots for meta‐analyses comprising 10 studies or more (Sterne 2016).

Stakeholder consultation and involvement

The initial impetus for the bid came from discussions with governors and service user representatives from Leeds Partnership NHS Foundation Trust. CT was a non‐executive director involved with the introduction of technology to help assessment by community mental health nurses.

Because end users of the review will be professionals and commissioners as well as patients and the public, we have set up an advisory group composed of people with specialist knowledge of digital health in a UK NHS context. One of our co‐authors, Alison Ledward, is a public member of the UK National Institute for Health Research Health Services and Delivery Research (NIHR HS&DR) Researcher‐Led Panel and has helped to shape our protocol. She has helped to ensure we maintain our focus on the effects of CDSS on patient outcomes and experiences and determining whether CDSS help nurses and AHPs make better decisions for patients.

Summary of findings and assessment of the certainty of the evidence

Two review authors will independently assess the certainty of the evidence (high, moderate, low, or very low) using the five GRADE considerations (risk of bias, consistency of effect, imprecision, indirectness and publication bias) (Guyatt 2008). We will use the methods and recommendations described in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2019), and the EPOC worksheets (EPOC 2017d); and we will use GRADEpro GDT software (GRADEpro 2015). We will resolve disagreements on certainty ratings by discussion and provide justification for decisions to downgrade or upgrade the ratings using footnotes in the table. We will also make comments to aid readers' understanding of the review where necessary. We will use plain language statements to report these findings in the review (EPOC 2017d).

Findings will be summarised in a 'Summary of findings' table(s) for the main comparison(s). This will include the most important outcomes (effects on process of care; patient outcomes; adverse events; implementation and resource use; see Appendix 2). If during the review process, we become aware of an important outcome that we failed to list in our planned 'Summary of findings' table(s), we will include the relevant outcome and explain the reasons for this in the section 'Differences between protocol and review'. 

We will consider whether there is any additional outcome information that was not able to be incorporated into meta‐analyses and note this in the comments and state if it supports or contradicts the information from the meta‐analyses. If it is not possible to meta‐analyse the data, we will summarise the results in the text.

History

Protocol first published: Issue 3, 2021

Notes

This protocol is based on standard text and guidance provided by Cochrane Effective Practice and Organisation of Care (EPOC).

Appendices

Appendix 1. Ovid MEDLINE(R) and Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations search strategy

1 exp Decision Making/ (193943)

2 decision support techniques/ (19497)

3 (decision* adj2 making).ti,ab,kf. (131157)

4 (decision* adj2 support*).ti,ab,kf. (19287)

5 (decision* adj2 aid*).ti,ab,kf. (5447)

6 or/1‐5(314107)

7 exp Computers/ (76693)

8 exp information systems/ (216711)

9 exp Informatics/ (494358)

10 Internet/ (70418)

11 Software/ (105263)

12 Cell Phones/ (8064)

13 smart phone/ (3504)

14 Mobile Applications/ (4780)

15 exp Telemedicine/ (26297)

16 Medical Records Systems, Computerized/ (18968)

17 exp Electronic Health Records/ (18305)

18 computer*.ti,ab,kf. (288702)

19 electronic*.ti,ab,kf. (244945)

20 (internet or web or online or on‐line).ti,ab,kf. (252396)

21 (software or computer program*).ti,ab,kf. (166693)

22 (automate* or automation).ti,ab,kf. (121258)

23 (pda or pdas).ti,ab.(11378)

24 personal digital assistant*.ti,ab,kf. (989)

25 (app or apps).ti,ab,kf. (25389)

26 (application* adj2 mobile*).ti,ab,kf. (3005)

27 (iPad* or iPhone* or smartphone* or smart phone* or smart device*).ti,ab,kf. (11965)

28 (tablet adj2 (pc or device* or comput*)).ti,ab,kf. (1296)

29 (telehealth or telecare or telemedicine).ti,ab.(12930)

30 or/7‐29(1468638)

31 6 and 30(56294)

32 exp Decision Making, Computer‐Assisted/ (139027)

33 Decision Support Systems, Clinical/ (7501)

34 (computer assisted adj2 (decision* or diagnos* or therap*)).ti,ab,kf.(897)

35 (computer aided adj2 (decision* or diagnos* or therap*)).ti,ab,kf.(2824)

36 (decision adj2 support adj2 (system* or tool*)).ti,ab,kf.(7845)

37 (decision making adj2 (system* or tool*)).ti,ab,kf.(2131)

38 Expert Systems/ (3369)

39 (expert adj2 system*).ti,ab,kf.(3355)

40 Reminder Systems/ (3316)

41 ((computer* or electronic* or CDSS) adj2 (reminder* or alert*)).ti,ab,kf.(1088)

42 reminder system*.ti,ab,kf.(733)

43 Medical Order Entry Systems/ (2160)

44 ((computer* or electronic*) adj2 order entry).ti,ab,kf.(1711)

45 CPOE.ti,ab,kf.(1049)

46 or/32‐45(162485)

47 31 or 46 [All computerised clinical decision support systems terms](206028)

48 Allied Health Personnel/ (11522)

49 Allied Health Occupations/ (562)

50 Physical Therapist Assistants/ (9)

51 Physical Therapy Specialty/ (2743)

52 Speech and language therapists/ (2930)

53 Occupational Therapy/ (12858)

54 Nutritionists/ (1082)

55 dietetics/ (7640)

56 Music therapists/(3585)

57 Anesthesiologists/(1078)

58 Orthoptist*.mp. (259)

59 Chiropodist*.mp.(122)

60 Podiatrist*.mp. (729)

61 exp Osteopaths/(317)

62 prosthetist*.mp. (319)

63 Orthotist*.mp. (176)

64 Radiographer*.mp.(1369)

65 Art therapist*.mp. (69)

66 Drama therapist*.mp (1)

67 (allied adj2 health adj2 (profession* or worker* or personnel or occupation* or staff)).ti,ab,kf.(2920)

68 ((physical or occupational or language or speech or physio* or music or art or drama) adj2 therap*).ti,ab,kf.(41089)

69 (Operating adj2 department adj2 practitioner*).ti,ab,kf.(7461)

70 physiotherapist*.ti,ab,kf.(7461)

71 dietetic*.ti,ab,kf.(7508)

72 dietitian*.ti,ab,kf.(5897)

73 nutritionist*.ti,ab,kf.(2650)

74 orthoptist*.ti,ab,kf. (257)

75 chiropodist*.ti,ab,kf. (122)

76 podiatrist*.ti,ab,kf. (727)

77 osteopath*.ti,ab,kf. (4923)

78 prosthetist*.ti,ab,kf. (318)

79 orthotist*.ti,ab,kf. (176)

80 radiographer*.ti,ab,kf. (1359)

81 Patient care team/ (63695)

82 ((multidisciplinary or interdisciplinary or multiprofessional or interprofessional) adj2 team*).ti,ab,kf.(25065)

83 Emergency Medical Technicians/ (5620)

84 Emergency Medical Services/ (41446)

85 Ambulances/ (5950)

86 Air Ambulances/ (2664)

87 paramedic*.ti,ab,kf.(7515)

88 HEMS.ti,ab,kf.(646)

89 ems.ti,ab,kf.(11526)

90 emt.ti,ab,kf.(19240)

91 prehospital.ti,ab,kf.(11319)

92 pre‐hospital.ti,ab,kf.(4080)

93 first responder*.ti,ab,kf.(1945)

94 emergency medical technician*.ti,ab,kf.(997)

95 emergency services.ti,ab,kf.(3475)

96 ambulance*.ti,ab,kf.(9924)

97 field triage.ti,ab,kf.(244)

98 out‐of‐hospital.ti,ab,kf.(9756)

99 (nurse or nurses or nursing).mp.(703803)

100 Midwifery/ (18767)

101 (midwif* or midwiv*).ti,ab,kf.(23372)

102 or/48‐101 [specified allied health professionals or nurses or midwives] (959079)

103 47 and 102 [All computerised clinical decision support systems and specified allied health professionals or nurses or midwives] (8353)

Appendix 2. 'Summary of findings' table draft

Summary of findings: the effect of CDSS on nurses' and AHPs' performance and patient health outcomes

Population: nurses and AHPs (both qualified and in training) Setting: primary, secondary, and tertiary care Intervention: any computerised clinical decision support system (CDSS) used to aid decision making Comparison: usual care
Outcomes	Anticipated absolute effects (95% CI)		Relative effect	Number of studies	Certainty of the evidence (GRADE)	comments
	Risk with usual care	Risk with CDSS
Process of care
Guideline adherence
CDSS‐target decision‐specific performance
Patient outcomes
CDSS‐targeted outcomes (e.g. number of falls; time in target range)
Safety
Implementation
Degree of usage
User satisfaction
Decisional conflict
Resource use

Contributions of authors

Conceiving the protocol: CT
Designing the protocol: TM
Co‐ordinating the protocol: TM
Designing search strategies: DA
Writing the protocol: TM, CT, RR, AMK, KB, HY, DA, AL, SS, HK
Providing general advice on the protocol: CT
Securing funding for the protocol: CT, AMK, KB, RR, HY, DA, AL
Performing previous work that was the foundation of the current study: CT, RR
Planned clinical data analysis (synthesis): TM, CT
Planned economic data analysis (synthesis): HY, KB

Contributions of the editorial base 

EPOC managing editor: Provided one round of review and contributed to six subsequent rounds of review. Co‐ordinated the editorial process (Chris Cooper)

Contact editor (EPOC): contributed six rounds of review and approved the protocol for publication (Xavier Bosch‐Capblanch);

Information specialist (EPOC): provided peer review of the search strategy and comments on the search approach (Paul Miller); and

Co‐ordinating editor (EPOC): contributed three rounds of review (Sasha Shepperd). 

Publication history 

Publication history

• Review proposal form received: 15/11/2019

• Editorial feedback provided: 16/12/2019

• Review proposal form returned to EPOC: 24/01/2020

• Editorial feedback provided: 24/03/2020

• Title registered: 27/03/2020

• Protocol received: 16/05/2020

• Protocol returned: 26/05/2020

• Revised protocol received: 29/05/2020

• Protocol returned: 24/06/2020

• Revised protocol received: 10/07/2020

• Protocol returned following external peer review: 26/10/2020 

• Revised protocol received: 05/11/2020

• Protocol returned: 10/11/2020

• Revised protocol received: 13/11/2020

• Protocol returned: 07/11/2020

• Revised protocol received: 10/01/2021

• Protocol returned: 29/01/2021

• Protocol accepted for publication: 03/02/2021

Sources of support

Internal sources

• Universities of Leeds and Bradford , UK

  The review team were supported by their respective University departments

External sources

• National Institute for Health Research (NIHR), UK

  AMK is an NIHR Senior Investigator; the review team were part of a wider non‐Cochrane synthesis of mixed evidence for CDSS effects on nursing and allied health professionals (NIHR award NIHR127926) 

Declarations of interest

• Teumzghi Mebrahtu: the University of Leeds, which employs Teumzghi Mebrahtu, has received grant funding from the National institute for Health Research (NIHR) to support his work on this review (fundingawards.nihr.ac.uk/award/NIHR127926).

• Carl Thompson: is the principal investigator and holder of the research grant from the National institute for Health Research (NIHR)  which is funding  this review (fundingawards.nihr.ac.uk/award/NIHR127926). 

• Rebecca Randell: is a co‐investigator of a research grant from the National institute for Health Research (NIHR)  part of which is funding work on this review (fundingawards.nihr.ac.uk/award/NIHR127926).

• Anne‐Maree Keenan:  is a co‐investigator of a research grant from the National institute for Health Research (NIHR)  part of which is funding work on this review (fundingawards.nihr.ac.uk/award/NIHR127926).

• Karen Bloor: is a co‐investigator of a research grant from the National institute for Health Research (NIHR) part of which is funding work on this review (fundingawards.nihr.ac.uk/award/NIHR127926).

• Deirdre Andre: is a co‐investigator of a research grant from the National Institute for Health Research (NIHR) part of which is funding work on this review (https://fundingawards.nihr.ac.uk/award/NIHR127926).

• Huiqin Yang: is a co‐investigator of a research grant from the National institute for Health Research (NIHR) part of which is funding work on this review (fundingawards.nihr.ac.uk/award/NIHR127926) .

• Sarah Skyrme: The research grant held by Carl Thompson (fundingawards.nihr.ac.uk/award/NIHR127926) includes funding to support Sarah Skryme’s work on this review.

• Alison Ledward:  is a co‐investigator of a research grant from the National institute for Health Research (NIHR)  part of which is funding work on this review (fundingawards.nihr.ac.uk/award/NIHR127926).

• Henry King: the University of Leeds, which employs Henry King, has received grant funding from the National institute for Health Research (NIHR) to support his work on this review fundingawards.nihr.ac.uk/award/NIHR127926).

New