Abstract
Background
Early detection of diseases in primary care is crucial for timely treatment and better outcomes. Complex care demands and limited resources can make early detection challenging. Clinical decision support systems (CDSS) aim to improve the diagnostic process. However, barriers to implementation have so far prevented their effective use.
This systematic review aimed to identify barriers for the implementation of CDSS for disease detection in primary care and use this to develop recommendations for implementation.
Methods
We searched MEDLINE, EMBASE, Scopus, Web of Science and Cochrane databases. Included studies reported barriers to the implementation of CDSS for the detection of undiagnosed, prevalent diseases in primary care. Two independent researchers undertook screening and data extraction. The QuADS tool was used for quality assessment. Data on barriers and facilitators were synthesised using an inductive-deductive approach based on the Theoretical Domains Framework. This was used to identify solutions via the Behaviour Change Wheel.
Results
10498 titles and abstracts were screened, and 768 full texts were assessed. We included 99 studies describing 85 tools, mostly in high-income countries. Most studies (66, 66.7%) applied qualitative methods and described CDSS implemented in pilot studies (64, 64.7%). Included studies had very limited stakeholder involvement or theoretical underpinning.
We identified 2563 unique barriers and facilitators to implementation. Barriers were spread across the Theoretical Domains Framework including technical and workflow implementation issues at practice level, wider healthcare system issues, problems with the usability of systems, PCPs’ and patients’ attitudes and beliefs, a lack of skills and knowledge, and social barriers.
Implementation recommendations for development teams involve selecting appropriate diagnostic challenges for CDSS, ensuring usability, engaging stakeholders and testing CDSS prior to implementation. Primary care teams need to clarify responsibilities, provide training and support patients. Underlying barriers across healthcare systems will need to be addressed at policy level.
Conclusions
The range and scale of the barriers and complexity of recommendations highlight implementation challenges for CDSS in primary care. Although recommendations can be used to improve implementation, our findings emphasise the need to carefully reflect on the feasibility of CDSS in primary care at the point of design and development.
The systematic review was preregistered using PROSPERO (CRD42024517054): https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=517054
Supplementary Information
The online version contains supplementary material available at 10.1186/s13012-025-01445-4.
Keywords: Clinical decision support, Implementation, Primary care, Design and development, Policy, Behaviour change wheel, Theoretical domains framework, Systematic review
Contributions to the literature.
Efforts have been made to develop electronic systems that help primary care staff make decisions when detecting and diagnosing diseases, but primary care staff are often not able to use the systems as intended.
This review summarises the evidence for why electronic systems that support diagnostic decision-making have been under-utilised, including practical issues, attitudes and beliefs, and problems across wider healthcare systems.
The findings underpin new recommendations for developers, primary care teams, commissioners and policymakers to improve the use of electronic decision support systems in primary care, or to develop alternatives if systems do not seem feasible.
Background
In many healthcare systems, primary care acts as the first point of contact for patients, holding a gatekeeping role for diagnosis and referral [1–3]. Early detection and accurate diagnosis in primary care are crucial for timely intervention and management but can be challenging [4, 5]. Patients can present with non-specific symptoms that could be attributed to multiple conditions with varying incidence [3, 6]. Primary care professionals (PCPs) have to consider differential diagnoses, risk factors, prevalence of potential conditions and comorbidities [7]. PCPs face these increasingly complex care demands despite workforce shortages and limited consultation time [8, 9]. They need to decide which patients to refer for specialist investigation without overwhelming secondary care’s limited resources or causing undue patient anxiety and overdiagnosis [10]. There is often a lack of clear guidelines, causing PCPs to use individual strategies based on their clinical judgement [11, 12]. Although clinical judgement based on knowledge and experience plays a key role in diagnostic skills, it might also be subject to bias [13, 14].
Clinical decision support systems (CDSS) have been developed to help PCPs manage information and inform decisions [15]. They cover a wide range of decision-making supporting triage, diagnostics, and referral, as well as management options for a range of conditions [16, 17]. Computerised CDSS can facilitate information gathering and retrieval [16, 18]. The use of CDSS and their reminder functions have also been reported to have positive effects on the screening for and detection of chronic diseases in primary care [19].
Despite potential benefits of diagnostic CDSS, barriers in primary care settings have so far prevented full implementation [20, 21]. Low uptake and use of CDSS is common across diagnostic settings and conditions. For example, Price et al. [22] found that cancer-specific diagnostic CDSS were used in only 16.7% of primary care practices in the UK. In a trial of an automated test ordering and feedback system, Bindels et al. [23] reported that only 4% of test-ordering recommendations were addressed. The trial could not be completed due to high drop-out of general practitioners (GPs) who were supposed to use the system. Similarly, Rubin et al. [24] reported major implementation issues due to interoperability problems and lack of fit with existing workflows in a pilot trial. Even after successful initial implementation, other studies have found that the commitment to, and use of, CDSS decreases over time [25, 26].
Addressing barriers is crucial as low uptake can reduce the effectiveness of CDSS [27, 28]. Several reviews have been published regarding CDSS use in primary care, reporting that implementation of CDSS for detection may have failed due to a variety of behavioural, organisational, and regulatory factors [29]. However, the reviews were limited to specific populations (Chen et al. [30]), conditions and study designs (Bradley et al. [31]), and outcomes (Fletcher et al. [32]), CDSS already in use (Meunier et al. [33]) and lacked a theoretic framework. Further, the field of CDSS in primary care is rapidly evolving due to software advances with many new publications around new technologies such as machine learning since these reviews.
In their meta-analysis concerning improvements in care, Kwan et al. [28] argued that current literature still provides little guidance to identify circumstances under which CDSS might be effective. Identifying and consequently addressing a comprehensive set of barriers and facilitators using an implementation framework is vital to develop specific implementation strategies. One of the most researched and frequently used frameworks of implementation, especially of healthcare professional behaviour, is the Theoretical Domains Framework [34]. The TDF’s 14 domains include physical and psychological skills, cognitions, attitudes and beliefs as well as social and environmental factors [35]. One of the key advantages of this framework is that it is aligned with intervention functions which target specific behavioural determinants. Intervention functions can be augmented by policy categories, e.g., changing guidelines, fiscal measures, regulation, and legislation [36]. Hence, the TDF could help to identify implementation problems regarding uptake and use of CDSS and guide the development of feasible recommendations.
The aim of this systematic review was to develop recommendations for the implementation of computerised CDSS for disease detection in primary care by reviewing the most up-to-date literature, mapping the barriers and facilitators to a comprehensive implementation framework.
Methods
This systematic review is reported in line with the PRISMA 2020 statement. A checklist can be found in the supplementary materials.
Data sources and search strategy
With the assistance of a librarian, we developed a search strategy using search terms from implementation frameworks [35, 37] and similar searches related to the implementation, uptake and sustainable use of CDSS in existing reviews [31–33].
A search for research articles irrespective of date of publication was conducted on 05 October 2023 and updated on 20 August 2024 using terms around primary care settings, CDSS MeSH terms, barriers and facilitators, and detection of disease in the following databases: Medline (searched through Ovid), EMBASE, Scopus, Web of Science and Cochrane. The search strategy was adapted to suit each database after an initial scoping search to assess the search strategy’s applicability for the current review. An overview is provided in the supplementary materials.
Inclusion and exclusion criteria
References were included in the systematic review if they met the following inclusion criteria based on the PICOS (population, intervention, comparison, outcomes and study type) scheme [38]:
Population
Studies were included if they were conducted in primary care settings with PCPs, other primary care staff, patients and caregivers.
Intervention
CDSS for detection of a prevalent, but previously undiagnosed disease were defined as any computerised, digital or electronic CDSS designed to assess an individual’s disease risk and provide recommendations to support decisions for the diagnostic process or triage, including all potential chronic or health diagnoses. We excluded CDSS assessing the future risk of health conditions or events (e.g., stroke or suicide risk) or used for preventive purposes only, as recommendations from these CDSS would differ from diagnostic systems. As PCPs are responsible for generating diagnostic hypotheses and making decisions around referrals, references were excluded if they described purely patient-facing CDSS or symptom checkers.
Comparison
Not applicable.
Outcomes
Articles had to report one or more barrier to, or facilitator of, the implementation or continued use of CDSS. We were not restrictive to the types of barriers and facilitators and included articles covering both specific CDSS as well as hypothetical scenarios. We included all records regardless of the country of origin to capture potential barriers and facilitators from different healthcare systems.
Study type
We included all primary research. Articles were excluded if they were unobtainable in full-text format (e.g., conference abstracts). We excluded reviews or meta-analyses from the systematic review but scanned references of relevant reviews (N = 79) to identify additional relevant records during the initial search.
Procedure and analysis
After deleting duplicates, all titles and abstracts were independently screened for eligibility by two authors (CD and AA). Discrepancies were discussed between reviewers and resolved by a third reviewer (TS) if necessary. Full-text screening was conducted by two independent authors (CD and AA) and discrepancies were resolved by discussion. Inter-rater reliability was calculated regarding inclusion decisions for both the abstract (Cohen’s Kappa = 0.88) and full-text review (Cohen’s Kappa = 0.60). We used Covidence for title and abstract screening, full-text review and data extraction. After the search and study selection, data was extracted independently by two authors (CD and AA/AP) using a coding sheet (see supplementary materials).
For mapping barriers and facilitators to the TDF, we used a combination of a deductive approach (coding into pre-defined domains) and inductive analysis (identification of subthemes). One author (CD) assigned a short summarising phrase to all barriers and facilitators (both qualitative and quantitative data). These initial phrases were coded under the TDF domains. Initially, we used all 14 domains for coding, however, no statements were assigned to the Optimism domain. We report Goals and Intention together because a clear distinction was hindered as some studies regarded hypothetical CDSS. The coding was checked by a second author (SES) and discrepancies were resolved by discussion. If necessary, a barrier or facilitator was coded into more than one domain. Subsequently, subthemes were generated within the TDF domains and discussed with the group of authors to reach consensus. Barriers and facilitators were often direct opposites, so we focus on barriers in the first part of the results.
We had intended to also apply the Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies (NASSS) framework [37], but initial coding revealed a lack of fit: one NASSS category (3B, demand-side values), would include around 40% of all statements, whereas others would rarely be used. Category 5 seemed insufficient in differentiating wider system issues.
Development of recommendations
We developed recommendations for the implementation of CDSS by linking the identified TDF domains to established interventions and policy categories using the Behaviour Change Wheel [BCW; 36]. Intervention functions (e.g., training and education, incentivisation and persuasion [36]) were used to frame recommendations for design and development, the primary care setting, and healthcare policies. We used facilitators from the literature as well as behaviour change techniques to elaborate recommendations further and adding detailed specifications. Strategies for healthcare policy were based on policy categories.
Critical appraisal/risk of bias assessment
Two reviewers (CD and AA/AP) independently conducted quality appraisals for the included articles using the Quality Assessment with Diverse Studies [39]. The tool is comprised of 13 common criteria that are rated on a scale from 0 (no mention at all) to 3 (complete). It is not recommended to calculate an overall score but discuss quality assessment findings narratively for the different criteria.
Patient and public involvement
Two patient and public representatives contributed to a lay summary of the review protocol and discussed the review with the researchers at early stages of protocol development. After conducting interim analyses, we discussed the themes to sense check the interpretation and discuss implications from a patient perspective.
Results
Description of research studies and CDSS
A total of 99 studies [24–26, 40–135] were included in the review (Fig. 1; list of references in supplementary materials) describing barriers to the implementation of 85 different CDSS. CDSS that were evaluated in more than one study were at different stages of development and therefore described differently across studies.
Fig. 1.
Flowchart. CDSS = Clinical Decision Support System; PCP: Primary Care Provider
Nearly all (n = 94) studies [24, 26, 40–45, 47–70, 72–83, 85–134]) reported a CDSS that provided a risk assessment of a prevalent, yet undiagnosed condition. Studies also included CDSS with additional preventive functions [40, 54, 58, 61, 67, 83, 88, 101, 106, 112, 117–119, 126, 130, 132], covered CDSS for test-ordering [25, 45, 46, 51, 52, 54, 55, 59, 67, 71, 74, 76, 79, 84, 87, 94, 97, 103, 105, 107, 108, 110–112, 118, 119, 126, 130, 135], referral [40, 42, 43, 47, 51, 52, 56, 61, 63, 74, 76, 80, 82, 85, 92–94, 96, 99, 101, 102, 105, 118, 120, 121, 123, 124, 129–131, 134], treatment recommendations [40, 43, 50, 54, 60–62, 65, 67, 69, 72, 73, 76, 81–83, 85, 88, 91, 93, 96, 102, 103, 105, 107, 108, 112, 116, 117, 124, 128–130, 133], and safety-netting/follow-up [43, 47, 53, 57, 61, 81, 86, 95, 102, 107, 108, 129, 130]. CDSS that did not include a risk assessment for a current condition focused on best practice test-ordering recommendations. CDSS were mainly (n = 87 studies) for use by GPs and equivalent staff [24–26, 41–57, 59–72, 74–76, 78, 81–97, 99–119, 121, 122, 124–126, 128–130, 133–135], as opposed to CDSS developed for use of other PCPs (e.g., tele-triage staff, optometrists and pharmacists [40, 58, 73, 77, 79, 80, 98, 120, 123, 127, 131, 132]). Five studies included CDSS with additional patient-facing functions [59, 62, 66, 102, 116].
Sixty-six studies [24, 25, 40–45, 47–49, 54, 56–72, 78–81, 84, 87, 88, 91, 94, 95, 97, 98, 101, 103–109, 111–113, 115–118, 120, 121, 124, 127, 129–134] used qualitative designs to assess barriers and facilitators to CDSS use, of which 14 were part of simulation studies, usability testing or ethnographic observations [56, 64, 80, 87, 94, 95, 104, 107–109, 111, 112, 121, 127]. Eighteen papers reported survey results [52, 53, 55, 73, 74, 77, 82, 83, 89, 102, 110, 119, 122, 123, 125, 126, 128, 135], and 12 used mixed-methods [46, 50, 51, 75, 76, 85, 86, 90, 92, 96, 100, 114]. Results were reported as part of RCTs in two studies [26, 99] and in one case report [93]. Most studies (n = 64) described CDSS that were evaluated as part of a pilot study [24, 25, 40, 41, 43–46, 51, 52, 54–57, 59, 60, 62–66, 69, 70, 72, 75, 76, 78, 81–92, 95, 96, 98, 99, 102–104, 106–109, 111–113, 116, 118–121, 123, 126, 128, 132–134]. Nineteen studies reported hypothetical CDSS [48, 49, 58, 68, 71, 73, 74, 77, 97, 100, 105, 110, 114, 115, 122, 125, 129, 130, 135], while three evaluated CDSS in a large-scale trial [26, 61, 124] and 13 described CDSS that were implemented in routine care [42, 47, 50, 53, 67, 79, 80, 93, 94, 101, 117, 127, 131].
Studies were mainly conducted in English-speaking countries including the US (n = 28) [26, 44, 45, 51–53, 62, 74, 78, 83, 84, 87, 88, 90, 94, 95, 97, 106–108, 110–112, 114, 118, 119, 126, 134], the UK (n = 16) [24, 42, 47, 49, 55, 58, 60, 63, 76, 82, 100, 101, 104, 105, 120, 128], Australia/New Zealand (n = 12) [56, 57, 59, 77, 81, 93, 102, 103, 117, 121, 132, 135] and Canada (n = 6) [25, 71, 72, 92, 96, 129]. Twelve studies were conducted in low-to-middle income countries (LMICs; [40, 43, 50, 61, 65, 69, 85, 91, 99, 116, 124, 133]).
Details of the included studies grouped by conditions are presented in Table 1, with details about the CDSS summarised in Table 2.
Table 1.
Overview of research studies
| Author/Year | Description of CDSS | Country | Income level classi-fication | Aim of study | Methodology for investigating barriers and facilitators | Dates of study data collection | Study setting | Study participants | Number of participants in study |
|---|---|---|---|---|---|---|---|---|---|
| Cancer | |||||||||
| Cunich 2011 [59] | ALProst to support PSA test decision-making, avoid over-investigation for prostate cancers | Australia | HIC | Develop and pilot a decision support tool for prostate cancer screening | Qualitative | n/s | Metropolitan GP practices | GPs | 10 |
| Saleem 2005 [112] | Screening & Surveillance App to improve quality of care by reducing reliance of providers on their memories and by presenting accepted clinical guidelines at point of care | US | HIC | Determine barriers and facilitators to the effective use of CDSS | Ethnographic observations, qualitative | January to June 2004 | VA Medical Centre ambulatory clinics | Intake/triage nurses, providers (physicians, residents, nurse practitioners, physician assistants) | 90 |
| Saleem 2011 [111] | Screening & Surveillance App to support physicians in tracking and managing colorectal cancer screening and surveillance | US | HIC | Understand whether design changes to CDSS result in improved usability, workload/workflow integration | Laboratory simulation experiment, qualitative | n/s | VA Medical Centre ambulatory clinics | Primary care physicians, nurse practitioners | 12 |
| Militello 2014 [94] | Screening & Surveillance App to support physicians in tracking and managing colorectal cancer screening and surveillance | US | HIC | Analyse use of CDSS and workflow variations | Ethnographic observations | 2004 to 2008 | VA Medical Centre ambulatory clinics | Administrators, clinical champions, technical support staff | 220 |
| Chiang 2015 [56] | QCancer to provide specific risks of multiple cancers | Australia | HIC | Investigate challenges to use of implementation of CDSS in primary care | Simulation study, qualitative | n/s | University-affiliated primary care practice-based research network | GPs | 15 |
| Carney 2015 [53] | To screen for breast, cervical, and colorectal cancer, and follow-up | US | HIC | Examine the relationship between health IT and under-resourced settings | Survey | January to March 2014 | Federally funded HRSA community health centres | Community health centre directors, officials, providers, and general staff | 44 |
| Dikomitis 2015 [63] | eRATs to aid recognition of potential lung or colorectal cancer symptoms | UK | HIC | Obtain views from GPs who piloted the electronic risk assessment tools on workflow integration and widespread use | Qualitative | February to March 2012 | GP practices | GPs | 23 |
| Schroy 2015 [114] | To estimate an individual’s probability of having advanced colorectal neoplasia at the present time | US | HIC | Assess the receptivity to the use of CDSS | Mixed-methods | January 2010 to November 2011 | Large urban academic safety-net institution | Primary care physicians and nurse practitioners | 66 |
| Militello 2016 [95] | Screening & Surveillance App to support physicians in tracking and managing colorectal cancer screening and surveillance | US | HIC | Employ a decision-centred design conceptual framework to screening CDSS | Ethnographic observation, qualitative | n/s | VA Medical Centre ambulatory clinics | Primary care providers | 34 |
| Porat 2016 [105] | TRANSFoRm project to support GPs cognitive requirements in Lung cancer, myeloma, colorectal cancer diagnostic process | UK | HIC | Elicit user requirements for the design of a prototype CDSS | Qualitative | n/s | Primary care practices | GPs | 43 |
| Saraiva 2016 [113] | To assist in the diagnosis of gastrointestinal cancers; anal, colorectal, oesophagus, and stomach | Brazil | MIC | Present a case-based reasoning-first rule-based reasoning-last based CDSS | Qualitative | n/s | n/s | GPs | 5 |
| Stevens 2016 [119] | Colorectal cancer dashboard to support primary care providers and nurses in an interdisciplinary approach in managing colorectal cancer screening | US | HIC | Assess and evaluate primary care providers'knowledge and understanding of clinical decision supports | Pre-post-implementation survey | n/s | Ambulatory Care Specialist Clinic | Health care physicians, nurse practitioners and practice nurses | 5 |
| Engelen 2017 [66] | To support in clarifying and communicating preference in prostate cancer screening | Belgium | HIC | Identify and address issues around the use of decision support tools | Qualitative | October 2013 | GP practices, social clubs and societies | GPs and male patients | 59 |
| Porat 2017 [104] | TRANSFoRm project to support GPs cognitive requirements in Lung cancer, myeloma, colorectal cancer diagnostic process | UK | HIC | Examine the usability and acceptability of a diagnostic DSS prototype integrated with the EHR | Simulation study, qualitative | n/s | University-affiliated GP practices | GPs | 46 |
| Pannebakker 2019 [101] | Electronic 7PCL to help identify patients at risk of malignant melanoma for early referral and investigation | UK | HIC | Understand GP and patient perspectives on implementation and usefulness of CDSS | Qualitative | August 2016 to January 2017 | GP practices | GPs, patients | 28 |
| Rim 2019 [110] | Multiple tools to support discussions regarding PSA testing for prostate cancer screening with patients | US | HIC | Examine providers’ perspectives on use of CDSS | Survey | June to July 2016 | Web-based survey | Primary care practitioners and nurse practitioners | 1256 |
| Akanuwe 2021 [42] | QCancer to provide specific risks of multiple cancers | UK | HIC | Explore barriers and facilitators to the implementation of a CDSS | Qualitative | September 2014 to September 2015 | GP practices | Patients, primary care physicians and nurses | 36 |
| Rubin 2021 [24] | Macmillan eCDS tool to assess risk for oesophago-gastric cancer | UK | HIC | Feasibility of a trial of CDSS for suspected cancer | Qualitative | November 2015 to December 2017 | General practices in Clinical Research Networks | GPs | 9 |
| Chima 2022 [57] | Future Health Today to identify patients at risk of an undiagnosed cancer, identify follow-up actions | Australia | HIC | Explore usefulness and feasibility of a novel QI tool | Qualitative | February to March 2021 | GP practices | GPs, practice nurses, practice managers, consumer | 28 |
| Lowery 2022 [90] | Decision-Precision to help PCPs tailor lung cancer screening discussion to patient’s risk factor profile | US | HIC | Test two strategies for implementing a prediction-based CDSS | Mixed-methods | February 2017 to March 2019 | Veterans’ Affairs Medical Centres | Screening coordinators, primary care practitioners | n/s |
| Black 2023 [47] | C the Signs to identify patients at risk of multiple cancers at an early stage | UK | HIC | Understand the impact of CDSS on diagnosis and safety netting | Qualitative | November 2020 to June 2022 | Primary care practices within Cancer Alliance | GPs, practice nurse, administrative staff | 21 |
| Frisinger 2023 [68] | To evaluate if a skin lesion is likely to be a malignant melanoma and in need of further analysis and treatment | Sweden | HIC | Investigate stakeholder perceptions of recently developed CDSS | Qualitative | n/s | Public and private primary healthcare organisations | Regional managers/chief medical officers, service providing organization managers and doctors | 16 |
| Harper 2023 [74] | To improve prostate cancer screening | US | HIC | Understand clinician attitudes towards CDSS-assisted PSA decision-making | Survey | November 2016 to November 2017 | Primary care clinics in large academic health system | Primary care physician and advanced practice provider | 59 |
| Sturrock 2023 [120] | Head and Neck Cancer Risk Calculator (HaNC-RC V2) to support decision-making around referral in HNC | UK | HIC | Explore potential use of CDSS in pharmacists’ referral | Qualitative | July 2021 to August 2022 | Community pharmacies | Community pharmacists | 17 |
| Carlsson 2024a [52] | To improve guideline-concordant practice in PSA prostate screening | US | HIC | Pilot test an electronic health record-embedded decision support tool to facilitate PSA screening discussions | Survey | Stated end date April 2020 | Primary care practices and community health centres | Primary care practitioners | 10 |
|
Carlsson 2024b [51] |
Tool for SDM and PSA screening | US | HIC | Assess PCP’s attitudes, perceptions and feasibility of implementing CDS | Mixed-methods | September 2018 | Primary care | Primary care practitioners | 10 |
| Helenason 2024 [75] | AI-based dermalyser decision support system to support the early detection of cutaneous melanoma | Sweden | HIC | Assess feasibility of an AI-based CDSS for cutaneous melanoma | Mixed-methods | March 2021 to May 2021 | Primary healthcare centres | Primary care physicians | 15 |
| Morgan 2024 [97] | EHR-based lung cancer screening decision aids | US | HIC | Learn about [PCPs'] thoughts, preferences, and perceived barriers and facilitators regarding the design and implementation of an EHR-based LCS DA | Qualitative | October 2020 to February 2021 | General internal medicine outpatient clinics | Primary care practitioners | 15 |
| Skurla 2024 [118] | DecisionPrecision (DP) tool designed to support personalised SDM for lung cancer screening | US | HIC | Examine how clinicians react to using the encounter-based decision tool (DP) to support personalised SDM | Qualitative | June 2018 to January 2019 | Veteran affairs sites | Clinicians | 96 |
| Respiratory diseases | |||||||||
| Li 2012 [87] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia in patients presenting with breathlessness | US | HIC | Improve the usability of a CDS prototype by describing phases of evaluation prior to widespread deployment of CDSS | Simulation study, qualitative | n/s | Ambulatory care clinic associated with the hosting academic institution | Primary care providers | 16 |
| McCullagh 2014 [26] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia in patients presenting with breathlessness | US | HIC | Measure factors of adoption and factors impacting sustained use of CDSS | RCT | November 2010 to November 2011 | Two large urban ambulatory primary care practices | Primary care residents | 70 |
| Ginsburg 2016 [69] | IMCI tool/mPneumonia to help frontline health workers adhere to IMCI guidelines and differentiate wheezing illnesses from pneumonia | Ghana | LMIC | Understand the feasibility, usability and acceptability of CDSS | Qualitative | July to September 2014 | Sub-district health facilities | Community health nurses and officers, health assistants, midwives, caregivers | 72 |
| Richardson 2017 [108] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia in patients presenting with breathlessness | US | HIC | Understand the barriers to and facilitators of CDSS | Qualitative, observational usability study | n/s | Large academic health care centre | Primary care providers | 12 |
| Richardson 2019 [107] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia in patients presenting with breathlessness | US | HIC | Further understand the barriers to and facilitators of meaningful CDS tool usage in real clinical context | Qualitative, observational usability study | January 2017 | Large academic health care centre | Primary care providers | 3 |
| Ellington 2021 [65] | IMCI tool/mPneumonia to help frontline health workers adhere to IMCI guidelines and differentiate wheezing illnesses from pneumonia | Uganda | LIC | Understand determinants of successful CDSS implementation | Qualitative | January 2020 | Peri-urban and rural health centres | Healthcare professionals | 28 |
| Sunjaya 2022 [121] | to support Asthma/COPD diagnosis in patients with breathlessness | Australia | HIC | Present results of design thinking approach developing a CDSS | Observational usability study | n/s | Primary care practices from varying local health districts | GPs | 9 |
| Canny 2023 [49] | Hypothetical tool to calculate the likelihood of asthma | UK | HIC | Understand patient views on CDSS for asthma diagnosis | Qualitative | October 2020 to January 2021 | GP practices | Individuals with asthma and parents of children with asthma | 17 |
| Daines 2024 [60] | Asthma Diagnosis Clinical Decision Support System prototype to improve asthma diagnosis | UK | HIC | Understand clinician views and determine the barriers and facilitators for CDSS use | Qualitative | 1 May 2020 to 31 August 2020 | Practices from clinical research networks (CRNs) | GPs, nurse | 16 |
| Mental health | |||||||||
| Turner 2003 [128] | To assist practitioners in diagnosing and managing dementia | UK | HIC | examine the potential usefulness of a CDSS to assist practitioners in diagnosing and managing dementia | Survey | n/s | Primary care practices | GPs, primary care nurses | 97 |
| Tewari 2017 [124] | SMART Mental Health to support screening by ASHAs, and clinical diagnosis and management by primary care doctors of depression, anxiety, and increased suicide risk | India | LMIC | Development and evaluation of a multifaceted intervention including CDSS | Qualitative | January to February 2016 | Two rural sites | Positively screened community members, ASHAs, project field staff | 78 |
| Dannenberg 2019 [62] | Option Grid™ to outline common approaches to managing depression | US | HIC | Gather input from end-users to inform the subsequent development of an electronic CDSS | Qualitative | October 2016 to April 2017 | Family Medicine and General Internal Medicine Clinic | Clinicians, consumers with and without depression | 32 |
| Morgan 2019 [96] | Primary Care – Dementia Assessment & Treatment Algorithm*TM to assess, diagnose, initially manage, and monitor dementia in primary care | Canada | HIC | Describe the development, adaptation, and implementation phases of a Rural PHC Model for Dementia | Mixed-methods | 2015 to 2017 | Predominantly rural health services | Health care providers: physicians, nurse practitioner, occupational therapist, home care nurses | 133 |
| Parker 2020 [102] | Youth StepCare to help GPs identify and treat anxiety and depression in youth patients | Australia | HIC | Assess the feasibility and acceptability of CDSS | Survey | August 2018 to January 2019 | Practices affiliated with Primary Health Networks | GPs and practice staff | 11 |
| Radovic 2021 [106] | Screening Wizard (SW) to guide PCPs in responding to positive depression and suicidality screens in adolescents, facilitate shared decision-making, increase uptake of depression treatment | US | HIC | Describe multi-stakeholder perspectives to understand the acceptability and potential barriers to CDSS implementation | Qualitative | n/s | Primary care clinics of paediatric practice-based research network | Adolescents with history of depression, parents, healthcare providers (physicians, nurse practitioners, other practice staff) | 40 |
| Shannon 2021 [116] | “Detection and Integrated Care for Depression and Alcohol Use in Primary Care” (DIADA) project to help doctors use the screening results for depression and alcohol disorder to make diagnoses and initiate care | Colombia | MIC | Add experience to implementation research, gain understanding that could improve success of the overall project | Qualitative | n/s | Urban primary care site | Patients and primary care staff (physicians and administrators) | 18 |
| Daniel 2023 [61] | SMART Mental Health to support screening, clinical diagnosis and management of depression, anxiety, and increased suicide risk | India | LMIC | Test initial feasibility and acceptability of CDSS, understand barriers to mental health services | Qualitative | March to May 2019 | Two villages within 4 km of respective primary healthcare centre | Community members, project field staff, ASHAs, primary care physicians | 53 |
| Luitel 2023 [91] | Mobile app-based clinical guideline to support primary HCWs in detection and management of mental health conditions | Nepal | LMIC | Present the perceptions and experiences of primary healthcare workers in using the app | Qualitative | February to March 2022 | Primary healthcare services (primary health care centres, health posts, urban health clinics and community health units) | Primary healthcare workers (health assistants, auxiliary health workers, medical officers) | 15 |
| Eye, ears, nose and throat diseases | |||||||||
| afKlercker 1998 [41] | To support decision-making regarding ear, nose, and throat (ENT) diseases primarily in nurses'telephone consultations and for QM recording the findings | Sweden | HIC | Assess how the introduction of new technology is accepted and what means are available to influence this | Qualitative | n/s | Rural primary care centre | Physician, nurses, nurses'aid | 5 |
| López 2017 [89] | OphthalDSS to help in diagnosis, offering educative content about pathologies indicating anterior segment ocular diseases | Spain | HIC | Develop, implement and evaluate CDSS for primary care physicians not specialized in ophthalmology | Survey | n/s | Primary care practices | Primary care physicians | 50 |
| Zazove 2017 [134] | EAR-PC study to identify and prompt audiology referrals for patients who were at high risk for hearing loss | US | HIC | Explore GPs’ views on CDSS to improve identification of individuals at risk | Qualitative | February to October 2015 | University-based family medicine | Clinicians | 23 |
| Gunasekeran 2022 [73] | Multiple hypothetical tools to provide provisional diabetic retinopathy (DR), glaucoma, age-related macular degeneration (AMD), and cataract diagnoses based on areas of interest in ophthalmic images to develop a treatment plan | Multinational, 70 countries | Range | Evaluate the acceptance and perception of AI applications among ophthalmologists | Survey | March 2020 to March 2021 | Online through ophthalmology associations | Ophthalmologists, primary eye care providers; primary care providers with eye care services | 1176 |
| Ho 2022 [77] | To detect major retinal disease with high diagnostic accuracy | Australia | HIC | Identify optometrists'attitudes towards the use of AI in clinical practice | Survey | February to June 2021 | Online | Optometrists | 133 |
| Constantin 2023 [58] | SCONe repository to capture retinal images by optometrists to exploit using AI to enable earlier diagnosis | UK | HIC | Better understand optometrists'needs and constraints in using AI in primary eye care | Qualitative | March to August 2021 | Optometry and ophthalmology practices | Optometrists and ophthalmologists | 23 |
| Oremule 2024 [100] | Hypothetical tool for AI clinical decision support in hearing health | UK | HIC | Examining the understanding, experience and attitudes of healthcare professionals towards AI technologies | Mixed-methods | March to May 2023 | Online survey in UK | Audiologists, ENT specialists, general practitioners | 93 |
| Cardiovascular diseases | |||||||||
| Kharbanda 2015 [83] | TeenBP to improve clinician recognition of elevated blood pressure and incident hypertension in children and adolescents to provide recommendations while adhering to national guidelines | US | HIC | Describe the development and piloting of CDSS to improve the quality of care delivered to children and adolescents | Survey | November 2013 to July 2014 | Community-based, primary care clinic located in the metropolitan region | GPs, nurse practitioner, physician assistant, paediatrician | 14 |
| Bangash 2020 [44] | To define individuals as possible familial hypercholeste-rolemia cases | US | HIC | Use an implementation science framework to develop a CDSS for familial hypercholeste-rolemia | Qualitative | November 2018 to October 2019 | University clinic campus | Primary care physicians, specialist physicians | 13 |
| Barry 2022 [45] | ECG AI-guided screening tool to improve diagnosis of left ventricular EF | US | HIC | Analysis of provider reflections and suggestions for effective clinical adoption of AI-enabled tools | Qualitative | December 2019 to February 2020 | Mayo Clinic | Physicians, physician assistants, nurse practitioners | 29 |
| Ho 2023 [78] | Machine learning (ML) algorithms to improve the detection of peripheral arterial disease (PAD) | US | HIC | Evaluate physician- and patient-elicited barriers and facilitators to the implementation of an ML-based PAD screening tool | Qualitative vignette | September 2021 to May 2022 | University’s Divisions of Primary Care and Population Health, Vascular Medicine, and Cardiology | Physicians, cardiovascular specialists, patients | 26 |
| Kling 2024 [84] | EHR-based alternative alert (AA) providing CDS within EHR for stress ECGs and stress echos ordering relative to stress tests | US | HIC | Investigate the impact of the initiative on ordering of stress ECGs relative to all exercise stress tests and explore PCP’s and cardiologists’ perceptions | Qualitative | July to August 2021 | Outpatient PC and cardiology clinic | Primary care practitioners, cardiologists | 24 |
| Pregnancy-related | |||||||||
| Abejirinde 2018 [40] | Bliss4Midwives to enable non-invasive point-of-care screening for pre-eclampsia, gestational diabetes and anaemia in pregnancy | Ghana | LMIC | Explore the experiences of women exposed to CDSS and how CDSS affected provider relationships and service utilisation | Qualitative | June 2016 to April 2017 | Predominantly rural health facilities | Pregnant women, midwives, community health nurses | 30 |
| Wright 2020 [132] | MatCHAT to screen for antenatal and postnatal depression as well as anxiety, substance use and partner violence | New Zealand | HIC | Investigate whether CDSS might support early detection and appropriate management, assess feasibility, utility and acceptability | Qualitative | July 2017 to February 2018 | Metropolitan hospital | Community midwives | 5 |
| Nagraj 2023 [99] | SMARThealth Pregnancy to identify pregnant women with anaemia, hypertensive disorders in pregnancy and gestational diabetes | India | LMIC | Co-design and evaluate a theory-informed complex intervention and implementation strategies | RCT | October 2019 to December 2020 | Primary healthcare clusters | Community health workers including GPs, auxiliary nurse midwives and ASHAs, pregnant women | 258 |
| Tegenaw 2024 [123] | CDSS delivers automated clinical pathways and computer assisted pruning and selection for antenatal, pregnant patient and post-natal care | Ethiopia | LIC | Evaluate the user acceptance of a CDSS in LRSs | Survey | August 2022 to January 2023 | Primary health care centres | Healthcare professionals | 7 |
| Pain | |||||||||
| Peiris 2014 [103] | Web-based CDSS to support Australian general practitioners (GPs) to diagnose and manage back pain | Australia | HIC | Determine whether a Web-based CDSS had the potential to support GPs to diagnose and manage back pain, to identify barriers and enablers to uptake | Qualitative | June 2012 to May 2013 | GP practices | GPs | 20 |
| Maghsoud-Lou 2017 [92] | Spinal Problem E-Referral (SPER) system to standardize and streamline the spinal injury referral process | Canada | HIC | Evaluate the accuracy of CDSS in terms of determining patient severity; Evaluate the general usability | Mixed-methods | n/s | Neurosurgery clinic | Clinicians | 12 |
| Guenter 2019 [72] | To improve management of neuropathic pain | Canada | HIC | Improve the adherence to CPG recommendations in primary care for the diagnosis and treatment of neuropathic pain through CDSS | Qualitative | n/s | University-based department of family medicine | GPs, residents, nurse practitioners | 118 |
| Infectious diseases | |||||||||
| Chadwick 2021 [55] | BBV_TP1 to identify individuals at higher risk of undiagnosed blood-borne virus infection | UK | HIC | Evaluate a prototype application designed to prompt in real-time | Cohort study, survey | March to November 2019 | GP practices | GPs, nurse practitioners, other clinicians; patients | 154 |
| McGlynn 2023 [93] | HealthPathways help GPs assess, manage, and refer COVID-19 cases in the format of pathways; provide localised, clinical information for patient consultation | Australia | HIC | Review team’s programme management responses, identify key factors for scale and spread | Case report | January 2020 to January 2022 | Major public hospitals, smaller hospitals, community health centres, major metropolitan hospital | n/a | n/a |
| Chronic kidney disease | |||||||||
| Litvin 2016 [88] | To improve the identification and management of CKD | US | HIC | Assess the impact of CDSS tools on primary care CKD clinical quality measures; to describe facilitators and barriers | Qualitative | September 2012 to September 2014 | National primary care practice-based research network | Providers and clinical staff members | 40 |
| Hunter 2024 [81] | Future health today (FHT) to identify primary care patients at risk of, or with undiagnosed or untreated chronic kidney disease | Australia | HIC | Analyse and improve the implementation of FHT in practices | Qualitative | July 2020 to April 2021 | General practice | GP, general practice nurse, general practice manager | 30 |
| Other | |||||||||
| Jones 2014 [82] | MoM Local Pathways to aid GPs management decisions in suspected allergy | UK | HIC | Develop and implement an online decision pathway | Survey | June 2013 | Referral management centres in NHS Trust | GPs | 110 |
| Tseng 2023 [126] | EHR clinical decision to identify patients due for diabetes screening | US | HIC | Report the development, implementation, and assessment of the QI program to identify patients due for diabetes screening and thus increase screening rates | Pre-post survey | April 2021 to May 2022 | Suburban, academically affiliated facility | Physicians | 17 |
| Cano 2024 [50] | SkinNTDs app to assist in diagnosis and management of skin NTDs | Ghana and Kenya | LMIC | Assess engagement, functionality, aesthetics and information quality of SkinNTDs app | Mixed-methods | December 2022 to April 2023 | Primary care | Frontline healthcare workers | 60 (17 in qualitative part) |
| Multiple conditions | |||||||||
| Timpka 1989 [125] | Hypertext system under development to provide knowledge transfer and decision support | Sweden | HIC | Analyse factors for successful CDSS development to uncover problematic general issues | Survey | n/s | Healthcare centres (some with research and development activities) | GPs and residents | 189 |
| Ridderikhoff 1997 [109] | Medical diagnostic decision support system covering broad areas of medical diagnostics and terminology | Netherlands | HIC | Design acceptable medical diagnostic decision support system (DDSS) | Simulation study, qualitative | n/s | Institute for Family Medicine | GPs | 20 |
| Dupuits 1998 [64] | HIOS/HIOS + to assist GPs in (diagnostic) decision-making | Netherlands | HIC | Describe development and evaluation of 2 CDSS | Usability testing, qualitative | n/s | GP practices | GPs | 3 |
| Bindels 2003 [46] | GRIF automated feedback system to stimulate adherence to practice guidelines in diagnostic test-ordering | Netherlands | HIC | Investigate experiences of GPs with CDSS system and reasons for non-acceptance of recommendations | Mixed-methods | n/s | GP practices affiliated with University Hospital | GPs | 24 |
| Holmström 2007 [79] | Symptoms, Advice, Measure; Ped’s Advice to offer triage recommend-dations and self-care advice | Sweden | HIC | Explore use of CDSS for telenursing from user perspective | Qualitative | 2004 to 2005 | Call centre in mid-sized town | Tele-nurses | 12 |
| Varonen 2008 [130] | Evidence-Based Medicine electronic Decision Support (EBMeDS) to provide patient-specific decision support | Finland | HIC | Identify potential barriers and facilitators to implementing CDSS from user perspective | Qualitative | October to December 2005 | Network of the Centre for Pharmacotherapy Development | Physicians | 39 |
| Curry 2011 [25] | To reduce inappropriate imaging requests | Canada | HIC | Assess clinical guideline adherence for diagnostic imaging and acceptance of electronic decision support | Qualitative | 36 weeks | Family Medicine Clinic | Physicians | 16 |
| Kortteisto 2012 [86] | Evidence-Based Medicine electronic Decision Support (EBMeDS) to provide patient-specific decision support | Finland | HIC | Describe reasons for using CDSS to improve application within workflow | Mixed-methods | July 2009 to February 2011 | Primary healthcare centre | GPs, nurses, other healthcare staff | 48 |
| Henderson 2013 [76] | Isabel to aid clinicians in differential diagnosis | UK | HIC | Elicit users’ views on usability of CDSS, determine uptake and impact on decision-making and patient management | Mixed-methods | n/s | GP practices | GPs | 7–10* |
| Shibl 2013 [117] | To enhance users’ decision-making process, resulting in more informed decisions | Australia | HIC | Develop and preliminarily test theoretical model for CDSS acceptance | Qualitative | n/s | n/s | GPs | 37 |
| Zhuang 2013 [135] | To support GPs'decision strategies related to pathology test ordering | Australia | HIC | Develop CDSS framework, investigate and understand the current practice of decision makers and their CDSS requirements | Survey | n/s | Online, Royal Australian College of General Practice (RACGP) | GPs | 85 |
| Griffith 2014 [71] | Hypothetical tool to promote more appropriate diagnostic imaging ordering practices | Canada | HIC | Understand physicians’ diagnostic imaging ordering to develop CDSS | Qualitative | n/s | Canadian jurisdiction | Non-radiologist physicians | 12 |
| Knoble 2015 [85] | Smartphone app of Integrated Management of Childhood Illnesses (IMCI) to support diagnostic decision-making | Nepal | LMIC | Determine patients’ and healthcare workers’ acceptance, usage and reasons for use of CDSS | Mixed-methods | n/s | Rural healthcare facilities | Patients and healthcare workers | 505 |
| Murdoch 2015 [98] | Odyssey to support telephone triage for same-day appointments in primary care | UK | HIC | Understand how nurses deploy and integrate CDSS in tele-triage for same-day appointments | Qualitative | June 2012 | GP practices | Nurses | 4 |
| Turnbull 2017 [127] | NHS Pathways to minimise risk by standardising and monitoring triage practice | UK | HIC | Examine how call-handlers manage, experience and respond to risk | Ethnographic observation, qualitative | n/s | Call centres integrated with General Practice-led urgent care centres | Call-handlers, clinical advisers (nurses or paramedics) and organisational managers | 47 |
| Yau 2018 [133] | e-PC101 to support diagnosis, screening and management of common presenting symptoms and chronic conditions | South Africa | MIC | Digitise a CDSS and describe lessons learned for its development, evaluation and implementation | Qualitative | 2-month period in 2014 | Primary care clinics | Primary care nurses | 17 |
| Holmström 2019 [80] | Rådgivningsstödet (RGS) Webb to aid TN triage function, assure consistency, and enhance patient safety | Sweden | HIC | Describe factors influencing use of CDSS among tele-nurses | Observation, qualitative | September to October 2017 | Primary healthcare centres | Tele-nurses | 6 |
| Wouters 2020 [131] | Netherlands Triage Standard (NTS) to standardise and increase the accuracy and reliability of nurses’ urgency triage decisions | Netherlands | HIC | Understand role of CDSS in decision-making and clinical reasoning in telephone triage | Qualitative | July 2016 to July 2018 | Out-of-hours services for primary care | Tele-nurses | 24 |
| Ford 2021 [67] | Multiple tools to identify patients in need of review, create alerts to remind GP to perform tasks, indicate a patient’s risk of disease | UK | HIC | Optimise the design of future CDSS by identifying useful and problematic aspects | Qualitative | n/s | GP practices | GPs | 11 |
| Buck 2022 [48] | Multiple hypothetical tools to free up physicians’ time, ensure stronger physician–patient relationship, reduce diagnostic errors | Germany | HIC | Investigate GP’s attitudes toward AI-enabled systems in diagnosis | Qualitative | March to May 2020 | GP practices | GPs | 18 |
| Gonçalves 2022 [70] | ISMiHealth tool to guide GPs through computer prompts about screening for migrants | Spain | HIC | Evaluate GPs’ views on the acceptability, adaptability, and feasibility of multi-disease screening CDSS | Qualitative | March to June 2019 | Primary healthcare centres | GPs | 29 |
| Tabla 2022 [122] | Hypothetical tool to support diagnostic or therapeutic decisions | France | HIC | Identify criteria making CDSS desirable | Survey | n/s | Primary care setting | GPs | 126 |
| Alwadhi 2023 [43] | E-IMNCI provides evidence-based clinical management decisions to reduce preventable deaths due to childhood illness | India | LMIC | Develop and assess the feasibility and acceptability of E-IMNCI | Qualitative | October 2020 to May 2021 | Health facilities | Auxiliary nurse midwives ANMs, caregivers/mothers | 103 |
| Schütze 2023 [115] | ‘Smart physician portal for patients with unclear disease’ (SATURN) to make a diagnosis in cases of diagnostic uncertainty | Germany | HIC | Determine user requirements of new CDSS | Qualitative | March to June 2022 | Primary care setting | GPs | 5 |
| Upshaw 2023 [129] | Hypothetical tool to enable proactive care and triage; reduce physician burnout and reclaim time spent with patients | Canada | HIC | Potentiate the benefit of AI in primary care by engaging key stakeholders in guiding its application | Qualitative | September to October 2020 | Primary care setting | Patients, primary care providers, health system leaders | 48 |
| Carter 2024 [54] | Health Catch-UP! for infectious disease and selected non-communicable disease screening and catch-up vaccination | UK | HIC | Evaluate the CDSS | Qualitative | September 2021 to March 2022 | Primary care practices | GPs, practice nurses, healthcare care assistant, administrative staff and patients | 72 |
ASHA Accredited Social Health Activist, CDSS Clinical Decision Support System, GP General Practitioner, PC Primary Care, PCP Primary Care Professionals, VA Veterans Affairs
HIC High Income Country, MIC Middle Income Country, LMIC Low-to-Middle Income Country, LIC Low Income Country
n/s not specified
Table 2.
Overview of CDSS
| Author/Year | Description of CDSS | Role in diagnostic pathway as described in study | Stage of development at time of study | Users of CDSS | Target population of CDSS | Trigger of CDSS | Output of CDSS | CDSS integrated in EHR? | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Prevention | Current risk | Test-ordering | Referral | Treatment | Follow-up | Not developed | Pilot study | Large scale trial | Implemented | |||||||
| Cancer | ||||||||||||||||
| Saleem 2005 [112] | Screening & Surveillance App to improve quality of care by reducing reliance of providers on their memories and by presenting accepted clinical guidelines at point of care | X | X | X | X | X | GPs, practice nurses, physician assistants | Primary care patients at VA centres | Passive clinical reminder when patient chart is opened | List of clinical reminders of evidence-based guidelines | Yes | |||||
| Cunich 2011 [59] | ALProst to support PSA test decision-making, avoid over-investigation for prostate cancers | X | X | X | GPs, patients | Men eligible for PSA test | Voluntary use | Indication which option regarding PSA testing is best for the user | No | |||||||
| Saleem 2011 [111] | Screening & Surveillance App to support physicians in tracking and managing colorectal cancer screening and surveillance | X | X | X | GPs | Patients over 40/between 50 and 75 | Clinical reminder | Clinical reminders and test recommendations | Yes | |||||||
| Militello 2014 [94] | Range of clinical reminders, Screening & Surveillance App to support colorectal cancer screening | X | X | X | X | GPs, practice nurses | Primary care patients | n/s | Clinical reminders and test recommendations | Yes | ||||||
| Carney 2015 [53] | To screen for breast, cervical, and colorectal cancer, and follow-up | X | X | X | GPs, practice nurses, lab and pharmacy workers | Primary care patients | n/s | Screening recommendation | No | |||||||
| Chiang 2015 [56] | QCancer to provide specific risks of multiple cancers | X | X | X | GPs | Primary care patients | Voluntary use | Risk estimates for 10 undiagnosed cancers, information on guideline-concordant pathways | No | |||||||
| Dikomitis 2015 [63] | eRATs to aid recognition of potential lung or colorectal cancer symptoms | X | X | X | GPs | Primary care patients | Voluntary use, additional log-on | On-screen prompts, calculated cancer risk, audit tables of patients | No | |||||||
| Schroy 2015 [114] | To estimate an individual’s probability of having advanced colorectal neoplasia at the present time | X | X | GPs | Primary care patients aged 50 years and older | n/s | Individual’s current probability of having advanced colorectal neoplasia in risk categories | No | ||||||||
| Militello 2016 [95] | Screening & Surveillance App to support physicians in tracking and managing colorectal cancer screening and surveillance | X | X | X | GPs | Primary care patients | n/s | Visualisation of recommendations, link to guidelines | No | |||||||
| Porat 2016 [105] | TRANSFoRm project to support GPs cognitive requirements in Lung cancer, myeloma, colorectal cancer diagnostic process | X | X | X | X | X | GPs | Primary care patients | as soon as the GP enters current health complaint | List of potential diagnoses updated according to coded input; Proposed examinations and investigations | Yes | |||||
| Saraiva 2016 [113] | To assist in the diagnosis of gastrointestinal cancers; anal, colorectal, oesophagus, and stomach | X | X | GPs | Primary care patients | Voluntary use | Risk of undiagnosed prevalent cancer | No | ||||||||
| Stevens 2016 [119] | Colorectal cancer dashboard to support primary care providers and nurses in an interdisciplinary approach in managing colorectal cancer screening | X | X | X | X | GPs, practice nurses | Primary care patients aged 50 years and older | Voluntary use | Report based on a provider's list of active patients who need a colorectal cancer screening protocol | No | ||||||
| Engelen 2017 [66] | To support in clarifying and communicating preference in prostate cancer screening | X | X | GPs, patients | Men | Voluntary use (available as booklet and website) | n/s | No | ||||||||
| Porat 2017 [104] | TRANSFoRm project to support GPs cognitive requirements in Lung cancer, myeloma, colorectal cancer diagnostic process | X | X | GPs | Primary care patients | as soon as the GP enters the reason for encounter and subsequently starts coding signs and symptoms | List of possible diagnoses, updated according to user input, checklist of associated signs and symptoms | Yes | ||||||||
| Pannebakker 2019 [101] | Electronic 7PCL to help identify patients at risk of malignant melanoma for early referral and investigation | X | X | X | X | GPs | Primary care patients | Voluntary use | Risk estimate of melanoma (high/low) and referral recommendation | Yes | ||||||
| Rim 2019 [110] | Multiple tools to support discussions regarding PSA testing for prostate cancer screening with patients | X | X | X | GPs | Men | n/s | Benefits and harms of PSA testing for prostate cancer screening | No | |||||||
| Akanuwe 2021 [42] | QCancer for cancer risk assessment | X | X | X | GPs | Primary care patients | Voluntary use | Cancer risk assessment | Yes | |||||||
| Rubin 2021 [24] | Macmillan eCDS tool to assess risk for oesophago-gastric cancer | X | X | X | GPs | Primary care patients aged 55 years and older | Voluntary use | Risk for oesophago-gastric cancer, referral recommendations | Yes | |||||||
| Chima 2022 [57] | Future Health Today to identify patients at risk of an undiagnosed cancer, identify follow-up actions | X | X | X | GPs, practice nurses, practice managers | Primary care patients | Upon opening patient's medical record | Prompt during a consultation, recommend-dation for investigation/follow-up on test results | Yes | |||||||
| Lowery 2022 [90] | Decision-Precision to help PCPs tailor lung cancer screening discussion to patient’s risk factor profile | X | X | GPs | Primary care patients | Voluntary use | Individualized quantitative risk assessment of screening trade-offs, patient-friendly summary, and graphics | Yes | ||||||||
| Black 2023 [47] | C the Signs to identify patients at risk of multiple cancers at an early stage | X | X | X | X | GPs, practice nurses, adminis-trative staff | Primary care patients | Voluntary use | Risk assessment, pre-filled referral forms, patient education material, dashboard for patient tracking | Yes | ||||||
| Frisinger 2023 [68] | To evaluate if a skin lesion is likely to be a malignant melanoma and in need of further analysis and treatment | X | X | GPs | Primary care patients | Combined with dermato-scope | Risk assessment of lesion being melanoma | No | ||||||||
| Harper 2023 [74] | CDS-assisted decision-making for PSA-based prostate cancer screening | X | X | X | X | GPs, practice nurses | Men | n/s | Recommendation for screening, referral, and test ordering | n/s | ||||||
| Sturrock 2023 [120] | Head and Neck Cancer Risk Calculator (HaNC-RC V2) to support decision-making around referral in HNC | X | X | X | Community pharmacists | Patients in community pharmacy | Voluntary use | Probability value and referral recommendation | No | |||||||
| Carlsson 2024a [52] | To improve guideline-concordant practice in PSA prostate screening | X | X | X | X | GPs | Men | Based on patient eligibility (male, sex and age) | Guideline for risk stratified screening algorithm, tool for SDM, 3 best practice advisories, module to set automated reminders | Yes | ||||||
| Carlsson 2024b [51] | Tool for SDM and PSA screening | X | X | X | X | GPs | Men in primary care with highest risk of aggressive prostate cancer | n/s | Guideline for risk stratified screening algorithm, tool for SDM, 3 best practice advisories, module to set automated reminders | Yes | ||||||
| Helenason 2024 [75] | AI-based dermalyser decision support system to support the early detection of cutaneous melanoma | X | X | GPs | Patients with skin lesion | Taking dermoscopic photo and upload into AI | Two tailed clinical decision supportive statement, either as: “Melanoma cannot be excluded” or “No signs of Melanoma” | No | ||||||||
| Morgan 2024 [97] | EHR-based lung cancer screening decision aids | X | X | X | GPs | Primary care patients | n/s | Personalised screening recommendation | Yes | |||||||
| Skurla 2024 [118] | DecisionPrecision (DP) tool designed to support personalised SDM for lung cancer screening | X | X | X | X | X | GPs | Primary care patients | n/s | Risk factor inputs, patient’s screening eligibility status and patients risk of dying from lung cancer in the next 6 years | No | |||||
| Respiratory diseases | ||||||||||||||||
| Li 2012 [87] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia | X | X | X | GPs | Primary care patients | When provider enters documentation that matches keywords in the algorithm criteria | Alerts, calculated risk of patient having the disease, smartset for ordering, progress note, prewritten patient instructions | Yes | |||||||
| McCullagh 2014 [26] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia | X | X | GPs | Primary care patients | When entering documentation that matches keywords in the algorithm criteria | Risk assessment and alert for streptococcal pharyngitis/pneumonia, documentation in progress note | Yes | ||||||||
| Ginsburg 2016 [69] | IMCI tool/mPneumonia to improve HCPs’ ability to diagnose, classify, and manage childhood pneumonia | X | X | X | GPs, practice nurses | Children at primary health facilities | Voluntary use | Partially automated respiratory rate counter, final diagnosis | No | |||||||
| Richardson 2017 [108] | iCPR2 to support identification of Streptococcal pharyngitis/pneumonia | X | X | X | X | X | GPs | Primary care patients | tool is triggered by a reason for visit of sore throat, cough, or upper respiratory tract infection | Calculated risk score, smart-set for ordering, documentation for progress notes, laboratory orders, prescription orders, diagnoses, patient instructions | Yes | |||||
| Richardson 2019 [107] | iCPR2 to evaluate the risk of GAS pharyngitis in patients presenting with sore throat and the risk of pneumonia in patients presenting with cough or upper respiratory tract infection | X | X | X | X | X | GPs; practice nurses | Primary care patients | tools were triggered by a reason for visit of sore throat, cough, or upper respiratory tract infection | Calculated risk score, smart-set for ordering, documentation for progress notes, laboratory orders, prescription orders, diagnoses, patient instructions, and level of service | Yes | |||||
| Ellington 2021 [65] | IMCI tool/ALRITE to help frontline health workers adhere to IMCI guidelines and differentiate wheezing illnesses from pneumonia | X | X | X | Frontline healthcare workers, nurses | Children under 5 years of age at primary health facilities | Voluntary use | Partially automated respiratory rate counter, educational videos and adapted respiratory assessment score, final diagnosis | No | |||||||
| Sunjaya 2022 [121] | to support Asthma/COPD diagnosis in patients with breathlessness | X | X | X | GPs | Primary care patients | Voluntary use | Final probabilities of differential diagnoses, option to view more differentials, diagnostic test suggestions | Yes | |||||||
| Canny 2023 [49] | Hypothetical tool to calculate the likelihood of asthma | X | X | GPs | Primary care patients | n/s | Probability of asthma | n/s | ||||||||
| Daines 2024 [60] | Asthma Diagnosis Clinical Decision Support System prototype to improve asthma diagnosis | X | X | X | GPs, Practice nurses | Primary care patients | n/s | Probability of asthma, suggestions for next steps to confirm/refute a diagnosis of asthma | No | |||||||
| Mental health | ||||||||||||||||
| Turner 2003 [128] | To assist practitioners in diagnosing and managing dementia | X | X | X | GPs; Practice nurses | Primary care patients | a symptom or problem triggers the diagnostic protocol, over-ride option | Decision support for dementia diagnosis; diagnostic review; dementia management; carer needs assessment; and dementia problems | Yes | |||||||
| Tewari 2017 [124] | SMART Mental Health to support mental health screening by ASHAs, and clinical diagnosis and management by primary care doctors (two separate EDSS) | X | X | X | X | GPs, ASHAs | Community members | Voluntary use by ASHAs | Referral recommendations based on traffic light system | No | ||||||
| Dannenberg 2019 [62] | Option Grid™ to outline common approaches to managing depression | X | X | X | Patients in clinic waiting room, clinicians | Primary care patients | Voluntary use by patients in clinic waiting room | Risk assessment regarding suffering from depression, information about management | No | |||||||
| Morgan 2019 [96] | Primary Care – Dementia Assessment & Treatment Algorithm*TM to assess, diagnose, initially manage, and monitor dementia in primary care | X | X | X | X | GPs; Practice nurses | Primary care patients | n/s | Visit flow sheets, education manual, and care pathways/algorithms | Yes | ||||||
| Parker 2020 [102] | Youth StepCare to help GPs identify and treat anxiety and depression in youth patients | X | X | X | X | X | GPs, practice staff, patients in waiting room | Primary care patients aged 14 to 17 years | Voluntary use | Symptom scores and clinical recommendations for treatment and monitoring | No | |||||
| Radovic 2021 [106] | Screening Wizard (SW) to guide PCPs in responding to positive depression and suicidality screens in adolescents; to facilitate SDM, increase uptake of depression treatment | X | X | X | GPs, adolescents, and their parents | Adolescent primary care patients | Voluntary use, tablet | Report of screening results that highlights adolescent-parent discrepancies for GP, accompanying patient handouts | No | |||||||
| Shannon 2021 [116] | “Detection and Integrated Care for Depression and Alcohol Use in Primary Care” (DIADA) project to help doctors use the screening results for depression and alcohol disorder to make diagnoses and initiate care | X | X | X | GPs, admin staff and patients | Adult primary care patients | Voluntary use | Diagnostic and treatment recommend-dations | No | |||||||
| Daniel 2023 [61] | SMART Mental Health to support screening, clinical diagnosis and management of depression, anxiety, and increased suicide risk | X | X | X | X | X | X | GPs, ASHAs | Community members | Voluntary use by ASHAs | Risk assessment with inbuilt follow-up questions based on traffic light system to prioritize patients, ability to connect with mental health specialist | No | ||||
| Luitel 2023 [91] | Mobile app-based clinical guideline to support primary HCWs in detection and management of mental health conditions | X | X | X | Health assistants, auxiliary health workers, medical officers | Children and adolescents | Voluntary use | Provide provisional mental health condition and management | No | |||||||
| Eye, ear, nose and throat diseases | ||||||||||||||||
| afKlercker 1998 [41] | To support decision-making regarding ear, nose, and throat (ENT) diseases primarily in nurses'telephone consultations and for QM recording the findings | X | X | GPs, practice nurses | Primary care patients | Mostly used in nurses'telephone consultations | n/s | No | ||||||||
| López 2017 [89] | OphthalDSS to help in diagnosis, offering educative content about pathologies indicating anterior segment ocular diseases | X | X | GPs not specialized in ophthal-mology, medical students | Primary care patients | Voluntary use, smartphones app | Pathology in the app that could match with the symptoms, guide to disease information | No | ||||||||
| Zazove 2017 [134] | EAR-PC study to identify and prompt audiology referrals for patients 55 and older who were at high risk for hearing loss | X | X | X | GPs | Primary care patients aged 55 and older | n/s | Model best practice alerts with 4 responses: put prevalent HL on problem list; patient declined testing; Ignored; referral to audiology | Yes | |||||||
| Gunasekeran 2022 [73] | Multiple hypothetical tools to provide provisional diabetic retinopathy (DR), glaucoma, age-related macular degeneration (AMD), and cataract diagnoses based on areas of interest in ophthalmic images to develop a treatment plan | X | X | X | Eye care practitioners, ophthal-mologists | Patients/public using eye care services | n/s | n/s | n/s | |||||||
| Ho 2022 [77] | To detect major retinal disease with high diagnostic accuracy | X | X | Optometrists | Patients/public using optometry services | n/s | Likely diagnosis from an OCT scan at the point-of-care; or AI is run overnight to process OCT scan and provide a second opinion | n/s | ||||||||
| Constantin 2023 [58] | SCONe repository to capture retinal images by optometrists to exploit using AI to enable earlier diagnosis | X | X | X | Optometrists | Patients/public using optometry services | n/s | Patient risk for developing or having eye diseases | No | |||||||
| Oremule 2024 [100] | Hypothetical tool for AI clinical decision support in hearing health | X | X | GPs, audiologists, ENT specialists, ENT specialty doctors and associate specialists in the UK | Primary care patients and visiting specialists | n/s | Diagnostic support and management recommendations | n/s | ||||||||
| Cardiovascular diseases | ||||||||||||||||
| Kharbanda 2015 [83] | TeenBP to improve clinician recognition of elevated blood pressure and incident hypertension in children and adolescents to provide recommendations while adhering to national guidelines | X | X | X | X | GPs, practice nurses, rooming staff | Children and adolescents in primary care | initiated after a BP is entered into the EHR | BP percentiles; Review of prior BPs/diagnoses to identify cases with incident HTN; Review of medications, BMI, and comorbidities; patient specific orders, weblinks to guideline, patient education material | Yes | ||||||
| Bangash 2020 [44] | To identify individuals as possible familial hypercholeste-rolemia cases | X | X | GPs, specialist physicians | Patients registered with clinic | First prototype passive reminder, second prototype inbox message upon logging into EHR | Alert of possible FH cases | Yes | ||||||||
| Barry 2022 [45] | ECG AI-guided screening tool to improve diagnosis of left ventricular EF | X | X | X | GPs, practice nurses, physician assistants | Primary care patients | n/s | Recommendation report including suggestion for TTE, brief description of the AI algorithm and phone number to call for additional information | No | |||||||
| Ho 2023 [78] | Machine learning (ML) algorithms to improve the detection of peripheral arterial disease (PAD) | X | X | GPs | Primary care patients | n/s | Summarised patient data, AI PAD prediction, recommendation whether to screen for PAD | No | ||||||||
| Kling 2024 [84] | EHR-based alternative alert (AA) providing CDS within EHR for stress ECGs and stress echos ordering relative to stress tests | X | X | GPs, cardiologists | Primary care and cardiology patients | Clinicians queue up a stress echo order in EHR for patient with no contraindication for stress ECG | Noninteractive decision tree to guide the selection of an appropriate stress test and included links to applicable references | Yes | ||||||||
| Pregnancy-related | ||||||||||||||||
| Abejirinde 2018 [40] | Bliss4Midwives to enable non-invasive point-of-care screening for pre-eclampsia, gestational diabetes and anaemia in pregnancy | X | X | X | X | X | Midwives | Pregnant women in rural areas | Voluntary use, smartphones app | Traffic light system indicates risk category or referral urgency; prompts for the health worker on counselling and treatment | No | |||||
| Wright 2020 [132] | MatCHAT to screen for antenatal and postnatal depression as well as anxiety, substance use and partner violence | X | X | X | Community midwives | Antenatal and postnatal women | Voluntary use | Overview of screening results for shared decision-making | No | |||||||
| Nagraj 2023 [99] | SMARThealth Pregnancy to identify pregnant women with anaemia, hypertensive disorders in pregnancy and gestational diabetes | X | X | X | GPs, ASHAs | Pregnant women in rural areas | Voluntary use, smartphones app | Referral and counselling advice using traffic light system; highlighted missing ANC practices | No | |||||||
| Tegenaw 2024 [123] | CDSS delivers automated clinical pathways and computer assisted pruning and selection for antenatal, pregnant patient and post-natal care | X | X | X | Practice nurses, midwives | Patients attending antenatal and postnatal care | Entering measured symptoms | The ranked table provides evidence for assessing various factors, such as symptoms, findings, urgency, CP, CP frequency, accuracy, and prior and posterior probability, to facilitate evidence-based decision-making by the user | Yes | |||||||
| Pain | ||||||||||||||||
| Peiris 2014 [103] | Web-based CDSS to support Australian general practitioners (GPs) to diagnose and manage back pain | X | X | X | X | GPs | Primary care patients | Voluntary use | Likely diagnosis, required investigations, treatment plan, review, common questions and concerns | No | ||||||
| Maghsoud-Lou 2017 [92] | Spinal Problem E-Referral (SPER) system to standardize and streamline the spinal injury referral process | X | X | X | GPs | Primary care patients | Voluntary use | Standardized spinal injury information collection templates; Assessment of condition severity and triage/treatment recommendations; complete referral letter | No | |||||||
| Guenter 2019 [72] | To improve management of neuropathic pain | X | X | X | GPs, practice nurses | Patients at Veterans Affairs (VA) family medicine clinics | n/s | Recommendations for managing neuropathic pain; links to materials that could support self-management | Yes | |||||||
| Infectious diseases | ||||||||||||||||
| Chadwick 2021 [55] | BBV_TP1 to identify individuals at higher risk of undiagnosed blood-borne virus infection | X | X | X | GPs | Primary care patients | Prompt during consultation in case of risk factors/suspect test results | Soft prompt indicating risk and advising BBV testing, or hard prompt requiring a response | Yes | |||||||
| McGlynn 2023 [93] |
HealthPathways help GPs assess, manage, and refer COVID-19 cases in the format of pathways; provide localised, clinical information for patient consultation |
X | X | X | X | GPs | Primary care patients | n/s | Up-to-date clinical and referral information guiding the clinician to the most appropriate level of care; key information about local services | No | ||||||
| Chronic kidney disease | ||||||||||||||||
| Litvin 2016 [88] | To improve the identification and management of CKD | X | X | X | X | GPs | Primary care patients | embedded into general or disease-specific progress note templates at the point of care | Prompts for at least annual testing in patients with HTN, DM; prompts for diagnosis and management | Yes | ||||||
| Hunter 2024 [81] | Future health today (FHT) to identify primary care patients at risk of, or with undiagnosed or untreated chronic disease | X | X | X | X | GPs, Practice nurses | Primary care patients with or without a CKD diagnosis | Extracts data from general practice’s EMR each night and provides recommendation when a patient record is opened in EMR | POC prompt and web-based dashboard provides recommendation links to relevant guidelines and resources | Yes | ||||||
| Other | ||||||||||||||||
| Jones 2014 [82] | MoM Local Pathways to aid GPs management decisions in suspected allergy | X | X | X | X | GPs | Primary care patients | n/s | Referral recommendations | Yes | ||||||
| Tseng 2023 [126] | EHR clinical decision to identify patients due for diabetes screening | X | X | X | X | GPs, other practice staff | Primary care patients aged 35 to 70 years | During the encounter, in the EHR | Alerts physicians by creating a flag in the patient chart during the encounter to order the screening test | Yes | ||||||
| Cano 2024 [50] | SkinNTDs app to assist in diagnosis and management of skin NTDs | X | X | X | Frontline healthcare workers | Primary care patients | Smartphone app | Identifying signs and symptoms, providing more information about these diseases | No | |||||||
| Multiple conditions | ||||||||||||||||
| Timpka 1989 [125] | Hypertext system under development to provide knowledge transfer and decision support | X | X | GPs | Primary care patients | n/s | Knowledge base for diagnosis support with 500 basic disease descriptions; therapeutic recommendations | n/s | ||||||||
| Ridderikhoff 1997 [109] | Medical diagnostic decision support system covering broad areas of medical diagnostics and terminology | X | X | GPs | Primary care patients | n/s | Differential diagnosis, 76 disease descriptions with 60–200 items per disease | No | ||||||||
| Dupuits 1998 [64] | HIOS/HIOS + to assist GPs in (diagnostic) decision-making | X | X | GPs | Primary care patients | Upon entering specific information about signs/symptoms, examination data, drugs, test results and diagnosis | Advice, information and reports based on practitioner input | Yes | ||||||||
| Bindels 2003 [46] | GRIF automated feedback system to stimulate adherence to practice guidelines in diagnostic test-ordering | X | X | GPs | Primary care patients | When entering medical patient data and desired diagnostic tests into electronic order entry form | Five types of test-ordering recommendations | No | ||||||||
| Holmström 2007 [79] | Symptoms, Advice, Measure; Ped’s Advice to offer triage recommend-dations and self-care advice | X | X | X | Tele-nurses | Members of public calling health service line | Mandatory use in every call | Triage recommendations/self-care advice | No | |||||||
| Varonen 2008 [130] | Evidence-Based Medicine electronic Decision Support (EBMeDS) to provide patient-specific decision support | X | X | X | X | X | X | X | GPs, Practice nurses, physio-therapists, psychologists | Primary care patients | At the time of opening the record, entering a new diagnosis, or prescribing a drug | Reminders, alerts or prompts | No | |||
| Curry 2011 [25] | To reduce inappropriate imaging requests | X | X | GPs | Primary care patients | Mouse click triggered the CPOE with embedded clinical support | Recommendation for a more appropriate/no imaging test | Yes | ||||||||
| Kortteisto 2012 [86] | Evidence-Based Medicine electronic Decision Support (EBMeDS) to provide patient-specific decision support | X | X | X | GPs, Practice nurses, physio-therapists, psychologists | Primary care patients | at the time of opening the record, entering a new diagnosis, or prescribing a drug | Drug alerts, reminders, guideline links, virtual health check/patient-specific decision support messages | Yes | |||||||
| Henderson 2013 [76] | Isabel to aid clinicians in differential diagnosis | X | X | X | X | X | GPs, nurse practitioners | Primary care patients | Voluntary use | List of 10 possible diagnoses based on the clinical features | No | |||||
| Shibl 2013 [117] | To enhance users’ decision-making process, resulting in more informed decisions | X | X | X | X | GPs | Primary care patients | n/s | n/s | No | ||||||
| Zhuang 2013 [135] | To support GPs'decision strategies related to pathology test ordering | X | X | GPs | Primary care patients | n/s | Suggestions regarding pathology request | No | ||||||||
| Griffith 2014 [71] | Hypothetical tool to promote more appropriate diagnostic imaging ordering practices | X | X | GPs and other physicians | Wide range of patients including primary care | n/s | n/s | n/s | ||||||||
| Knoble 2015 [85] | Smartphone app of Integrated Management of Childhood Illnesses (IMCI) to support diagnostic decision-making | X | X | X | X | GPs, practice nurses | Primary care patients in rural areas | Voluntary use, smartphones app | Provisional diagnosis and management advice; differential diagnosis list | No | ||||||
| Murdoch 2015 [98] | Odyssey to support telephone triage for same-day appointments in primary care | X | X | Practice nurses | Callers to primary care practices and their proxies | When nurse selects key symptom to launch the CDSS | Triage recommendations/self-care advice | No | ||||||||
| Turnbull 2017 [127] | NHS Pathways to minimise risk by standardising and monitoring triage practice | X | X | Non-clinical tele-triage staff | Members of public calling NHS 111 | When staff takes call | Triage recommendations | No | ||||||||
| Yau 2018 [133] | e-PC101 to support diagnosis, screening and management of common presenting symptoms and chronic conditions | X | X | X | GPs, practice nurses | Adult ambulatory patients in LMIC settings | Voluntary use, tablet/smartphone app | 2300 diagnostic, screening, and management recommendations | No | |||||||
| Holmström 2019 [80] | Rådgivningsstödet (RGS) Webb to aid TN triage function, assure consistency, and enhance patient safety | X | X | X | Tele-nurses | Members of public calling health service line | n/s | Advisory support, including medical information documentation, weblinks | No | |||||||
| Wouters 2020 [131] | Netherlands Triage Standard (NTS) to standardise and increase the accuracy and reliability of nurses’ urgency triage decisions | X | X | X | Tele-nurses | Members of public calling out-of-hours primary care centres | When staff takes call | Triage recommendations with six possible urgency levels | No | |||||||
| Ford 2021 [67] | Multiple tools to identify patients in need of review, create alerts to remind GP to perform tasks, indicate a patient’s risk of disease | X | X | X | X | X | GPs | Primary care patients | Some require activation by a clinician while others run automatically when patient record is opened | Prescribing-based alerts, prompts to record information, identification of patients in need for review, test alerts, diagnostic and prognostic risk assessment | n/s | |||||
| Buck 2022 [48] | Multiple hypothetical tools to free up physicians’ time, ensure stronger physician–patient relationship, reduce diagnostic errors | X | X | GPs | Primary care patients | n/s | n/s | No | ||||||||
| Gonçalves 2022 [70] | ISMiHealth tool to guide GPs through computer prompts about screening for migrants | X | X | GPs | Migrant population | Automatic prompts based on migrant characteristics attending PHC centres for any reasons | Prompts with screening recommendations | Yes | ||||||||
| Tabla 2022 [122] | Hypothetical tool to support diagnostic or therapeutic decisions | X | X | GPs | Primary care patients | Voluntary use, devices supported by AI | Computerised test interpretation | n/s | ||||||||
| Alwadhi 2023 [43] | E-IMNCI provides evidence-based clinical management decisions to reduce preventable deaths due to childhood illness | X | X | X | X | X | Auxiliary nurse midwives, medical officers | Children in primary care | Automatically creates and maintains a patient file for each child managed at the health centre | Learning resources, logistic management tool, helps in registration, assessment, classification, identification of treatment, follow up and referral of patients | No | |||||
| Schütze 2023 [115] | ‘Smart physician portal for patients with unclear disease’ (SATURN) to make a diagnosis in cases of diagnostic uncertainty | X | X | GPs | Primary care patients | n/s | Diagnosis with a statistical probability | No | ||||||||
| Upshaw 2023 [129] | Hypothetical tool to enable proactive care and triage; reduce physician burnout and reclaim time spent with patients | X | X | X | X | X | GPs, practice nurses | Primary care patients | n/s | AI-supported decision support | Yes | |||||
| Carter 2024 [54] | Health Catch-UP! for infectious disease and selected non-communicable disease screening and catch-up vaccination | X | X | X | X | X | GPs, practice nurses | Migrant patients in primary care | Primary healthcare professional input 6 key demographic variables – age, sex, BMI, country of origin, ethnicity and date of entry to the UK | Single “pop-up” or prompt which summarises the guideline-recommended screening blood tests and vaccines individualised to that patient | Yes | |||||
n/s not specified
Quality assessment
Most studies gave a clear and detailed statement on their research aims, had an appropriate study design, adequate format and content of data collection tools, and an appropriate method of analysis. The description of the research setting and target population, adequacy of sampling, rationale of data collection tools, description of data collection procedures, provision of recruitment data and the method of analysis was mostly rated as basic or moderate. Evidence of research stakeholder involvement was limited to CDSS users being part of the sample in most cases, with few studies addressing multiple stakeholders or engaging them in the development of CDSS and interpretation of findings. Most studies provided a very limited to basic discussion of strengths and limitations. Finally, the justification for the selected analytic method and theoretical or conceptual underpinning of the research was generally very limited. The appraisal of all QuADS criteria is presented in Supplementary Table 1.
Qualitative synthesis of barriers
We identified 2563 unique statements that were assigned to TDF categories. As facilitators were often the direct opposite of barriers, we only present barriers in the results and Table 3. An overview table including facilitators that were used to develop recommendations is provided in Supplementary Table 2.
Table 3.
Barriers to CDSS implementation and use for disease detection in primary care
| Domaina | Subthemes | Barriers decreasing use | Studies reporting barriers/opposite facilitators |
|---|---|---|---|
| Environmental context and resources (595) | Time constraints, time-intensive (vs time-saving) CDSS and high workload (153) |
Primary care context: - High patient volumes - Lack of staff/high staff turnover - Competing priorities/Main responsibility is to ensure appropriate care Busy patients/caregivers CDSS taking extra time: - Training - Data entry - Documentation and administration - Consultation and communication of results |
[24, 25, 40–46, 48, 49, 51, 52, 54, 55, 57–69, 72, 73, 75, 76, 78–82, 84–86, 88, 90, 91, 93, 94, 96, 97, 100, 104, 105, 107–109, 112, 114–119, 122, 124, 125, 127–130, 132–134] |
| Integration into workflow and care process (143) |
Lack of integration into workflow: - Logistic reasons/lack of infrastructure - Lack of software access for locum GPs - Interruptions in busy schedules - Monosymptomatic/too structured CDSS - Individual patient’s other medical priorities - Lack of needed admin support - Challenges in consultation - Even small changes to routine processes needed for CDSS implementation are hard to achieve - Unforeseen events that alter workflow, e.g., vaccination programmes for COVID-19 - Misalignment of implementation programmes targeting ideal implementation with existing, real-life workflows - Inadequate workarounds |
[24, 25, 40–42, 44–48, 50, 51, 54, 56, 57, 60, 61, 63, 65, 67–69, 72, 74–76, 78–81, 83, 84, 86–88, 90, 91, 94, 96–105, 107, 108, 111, 112, 114–118, 122, 125, 127–130, 132–135] | |
| Technical issues and integration into IT systems (122) |
- Technical difficulties slowing down the computer/system crashes - Lack of integration into existing IT systems and other CDSS - Problem of duplicate systems for monitoring patients - Poor technology infrastructure - Data security issues, data loss - Lack of functionality - Multiple competing tools |
[24–26, 40, 41, 44–48, 50, 51, 54, 56, 60–65, 67–69, 72, 73, 75–78, 80, 81, 86–88, 91, 94–97, 103–106, 111, 112, 115–117, 121, 123, 130, 132–134] | |
| Increasing demands in healthcare, integration into primary and secondary care systems and governance (102) |
- Contradictory pressures in primary care, e.g., COVID-19 pandemic - Increasing demands and complexities leading to lower capacity for innovation - Changing recommendations - Inflexible regulations - Ineffective insurance systems - Lack of follow-up care (especially in LMIC) - Lack of procuration, governance, support and external collaborations - Lack of CDSS availability in smaller practices |
[24, 42, 45, 47, 48, 50, 51, 53, 54, 58, 61, 63, 65, 68, 69, 73, 75, 78–81, 86, 88, 91–93, 95, 96, 99, 101, 112, 115, 116, 118, 120, 122, 124, 127, 128, 132–134] | |
| Financial issues and physical resources (44) |
- Difficulties securing financial support - Lack of resources to implement CDSS, especially in LMIC - Lack of resources to follow recommendations (therapy options/vaccines), especially in LMIC - Inequalities and lack of access in LMIC - Concerns about additional costs for more testing |
[45, 48, 50, 54, 58, 65, 68, 69, 73, 76, 81, 84, 91, 93, 94, 96, 100, 116, 118, 124, 125, 129, 130, 132–134] | |
| Medicolegal concerns or defence (19) | - Concerns about liability and medicolegal repercussions | [47, 48, 63, 68, 73, 75, 84, 98, 100, 101, 106, 114, 121, 123, 129] | |
| Potential of pre-consultation CDSS (6) | - Providers sceptical about patients using CDSS without guidance | [56, 62, 97, 122] | |
| Mandatory vs voluntary use (6) | - Voluntary use reducing motivation | [47, 48, 68, 76, 81, 85] | |
| Memory, attention and decision processes (410) | Usability and navigation within CDSS (231) |
Low usability - Unclear, technical, long, irrelevant, repeated recommendations - Difficulties finding follow-up actions/shortcuts - Manual data input - Unclear outputs/suboptimal presentation of results - Large masses of text - Too many open windows required - Lack of information from patient records - Too many clicks - Physical size of the application too small (tablets) - Lack of default options - Difficulties re-entering system or saving data - Lack of options for multiple lines of enquiries - Hard stops/dead ends in CDSS where no further action is possible - Free-text not picked up - Alerts not resolvable - Lack of interactivity - Restrictive answer options - Lack of customisation |
[24, 41, 43, 44, 46–52, 56–62, 64, 65, 67, 69, 72, 75, 78–81, 85–89, 91–93, 95–99, 101, 103–105, 107–109, 111, 112, 114–118, 121, 123, 124, 127, 130–133] |
| Preferences for alerts and prompts (48) |
- Too demanding or easily missed alerts - Prompt frequency too high or low - Irrelevant alerts at the point-of-care |
[44, 50, 51, 55, 63, 65, 67, 76, 81, 83, 84, 86, 87, 97, 101, 107, 108, 111, 112, 119–121, 130, 134] | |
| Cognitive overload (39) |
- Prompt fatigue and information overload - Multiple CDSS or multiple open sites, difficult to find log-in, delayed email reminders after PCP closed CDSS - Unmanageable lists of patients to assess/refer created by CDSS |
[44, 45, 47, 48, 51, 52, 54, 56, 63, 67, 75, 80, 81, 84, 86, 87, 95, 103, 104, 111, 112, 118, 129, 130, 134] | |
| Lack of efficiency of CDSS in supporting initial assessment (43) |
- Information not coded - Relevant information not considered - Inaccurate patient answers - Missing data entry options - CDSS asking for many (irrelevant) details, causing long initial assessments |
[46–48, 54, 60, 65, 68–70, 75, 79–81, 84, 87, 91, 92, 99, 105, 106, 109, 112, 115, 118, 121, 127, 129] | |
| Lack of coherent way to track, monitor and follow-up on patients (41) |
- Hard to track patient lists/follow-up tests - Missing coding options - Alternative ways for documentation used interfering with coherent monitoring and tracking of patients within CDSS |
[41, 47, 51, 52, 58, 62, 65, 70, 79, 81, 85, 88, 89, 91, 92, 94, 95, 97, 102, 105, 107, 108, 112, 115, 116, 121, 122, 124, 130, 134] | |
| Cognitive inflexibility of provider (8) |
- CDSS only used if conditions are suspected means PCPs selecting cases for CDSS use - Working hypothesis leads to tinkering with systems |
[46, 47, 52, 101, 105, 128, 131] | |
| Beliefs about capabilities (253) | CDSS capability/trust in results (98) |
- Lack of validation and evidence-base - Unclear to users what feeds into risk assessment - Potential bias if AI trained on different cohort, belief that AI not sufficiently mature - Low provider beliefs in accuracy - Patients requiring doctors’ agreement |
[24, 25, 40–42, 44–46, 48, 50, 55, 56, 60, 62, 63, 67–69, 72, 74, 75, 77, 78, 81, 84–86, 88–93, 95, 100, 104, 106–108, 111, 112, 114, 115, 117, 119, 121, 127, 129, 130, 134, 135] |
| Provider capability to complete the task with/without the CDSS (96) |
- Perception that providers are better than computers because of their experience, clinical judgement and human skills - Providers fear they would be seen as less capable using CDSS - Seemingly easy to handle conditions - Lack of confidence to communicate and manage condition after risk assessment |
[24, 26, 40–42, 45–48, 50–52, 54, 56, 57, 61–63, 65–69, 75–81, 91, 95, 98–103, 106, 108, 109, 112, 117, 121, 122, 124, 125, 127, 129–131, 134] | |
| Patient capabilities affecting use of the CDSS (33) |
- Lack of patient health literacy, computer/AI literacy and language skills - Difficulties understanding numerical outputs - Low patient education levels and age |
[40, 49–51, 57, 59, 61, 62, 66, 79, 80, 98, 100, 106, 114, 116, 124, 129, 131, 132] | |
| Provider confidence to use CDSS (26) | - Lack of technical skills and computer literacy (related to profession/age) | [24, 40, 48, 56, 57, 60, 67, 76, 79, 81, 86, 91, 94, 100, 104, 109, 117, 120, 124, 125, 133, 134] | |
| Skills (166) | Need for training/familiarity (102) |
- Lack of (sufficient) training and support - Difficulties remembering instructions |
[25, 41, 42, 45, 47, 50, 54, 56–58, 61, 63, 65–69, 72, 73, 75, 76, 78, 80, 85, 86, 88, 90, 91, 93, 94, 96, 97, 99–101, 104, 109, 116, 117, 120, 121, 124, 127–130, 133, 134] |
| Ease of use/required skills (35) |
- If CDSS too easy, might be boring - Complex and non-intuitive CDSS are used less |
[41, 43, 44, 48, 52, 54, 55, 59, 60, 64, 68, 69, 75, 79, 84–86, 90, 94, 101, 104, 108, 111, 120, 121, 123, 124, 130] | |
| Use of the CDSS increases or impedes skill development (29) | - Perception of risk of clinical deskilling/becoming over-reliant, especially in HICs | [40, 47, 48, 50, 57, 58, 61, 69, 72, 80, 81, 85, 98–101, 115, 117, 122, 123, 129, 130] | |
| Knowledge (138) | Use of the CDSS provides patient education (43) | - Lack of patient education resources within CDSS | [44, 48, 49, 51, 52, 57, 58, 60–62, 65, 66, 69, 80, 88, 97, 103, 106–108, 111, 115, 116, 118, 121, 124] |
| Use of the CDSS to increase provider knowledge (19) |
- CDSS not designed to support knowledge increase - CDSS not providing reasoning for diagnosis or details about treatment options |
[40, 46, 58, 60, 63, 70, 75, 78, 81, 84, 91, 99, 118, 124] | |
| Provider knowledge about technology (19) | - Lack of awareness about existing CDSS and technological advances | [44, 45, 48, 50, 51, 73, 82, 87, 88, 100, 101] | |
| Fit of CDSS with guidelines/best practice (20) |
- CDSS conflicting with guidelines - Changing or confusing guidelines - Lack of familiarity with a guideline |
[42, 45, 46, 51, 57, 60, 63, 64, 72, 80, 84, 91, 93, 101, 103, 118] | |
| Providers’ baseline knowledge about condition and evidence base of CDSS (30) |
- Lack of familiarity with a condition and how to treat - PCPs attributed a lack of skills/knowledge to time constraints to avoid personal responsibility for CDSS use |
[44, 45, 48, 50, 51, 54, 57, 75, 78, 81, 84, 87, 105, 113, 118, 128, 130, 134] | |
| Patients’ baseline knowledge about condition (7) |
- Lack of awareness about condition - Inadequate beliefs and mistrust in treatment such as vaccination |
[49, 54, 57, 61, 78, 124] | |
| Social influences (249) | Impact on patient-provider relationship (107) | - Fear that CDSS endanger trust and communication in strong patient-provider relationship built over time | [40, 43, 45–49, 51, 54, 57–60, 62, 65–68, 74, 78–80, 84–87, 90, 91, 93–95, 97, 100–110, 112, 115, 116, 118, 122, 125, 128–130, 132, 134] |
| Patient expectations and priorities (39) |
- Difficulties to address varying patient attitudes and expectations in one CDSS - Patients expecting personal conversations and assessments rather than use of CDSS - Patients having multiple reasons to see GP and expecting solutions for all of them, but not all can be addressed by CDSS at the same time - Ordering tests to meet patient expectations even if not recommended by CDSS |
[24, 40, 44, 46, 48, 51, 54, 57, 59, 61–63, 66, 68, 69, 76, 77, 80, 81, 85, 86, 88, 91, 96, 100, 101, 106, 114, 117, 127, 134] | |
| Stakeholder involvement for successful implementation (82) |
- Unclear communication - Communities being hesitant due to stigma |
[40, 41, 44, 45, 47, 50, 56, 60, 61, 65, 67–69, 73, 75, 76, 78, 81, 86, 91, 93, 96, 99–102, 112, 116, 117, 119, 122, 124, 125, 127, 131, 134] | |
| Stigma causing reluctance to use CDSS/disclose symptoms (21) |
- Lack of symptom disclosure due to stigma - Risk of increasing stigma - Higher priority assigned to physical rather than mental conditions by PCPs |
[54, 61, 62, 91, 96, 106, 116, 124, 134] | |
| Social/professional role and identity (94) | Roles/responsibilities regarding CDSS in primary care (46) |
- CDSS not appropriate for all PCPs - Lack of ownership/Belief CDSS are not part of core responsibilities - Roles and responsibilities regarding CDSS not agreed (including admin and support staff) - PCPs not seeing management of respective condition as their responsibility |
[42, 44, 45, 47, 48, 56–58, 60, 65–67, 73, 76, 78, 79, 81, 86, 93, 99, 102, 106, 112, 118, 122, 129, 130, 134] |
| Professional autonomy (29) |
- Perceptions of CDSS undermining professional role - Existential threat, fear of being replaced by AI |
[25, 46, 48, 51, 60, 62, 67, 74, 81, 100, 103, 105, 106, 118] | |
| Effect on professional reputation (10) | - Perceived negative effects of CDSS on professional reputation, especially in HICs | [40, 45, 58, 61, 84, 91, 99] | |
| Flexibility of CDSS to allow for individual approach (9) |
- Different styles and approaches to symptom assessment and diagnosis cannot be covered by single CDSS - CDSS not flexible enough to cover different approaches in communicating diagnosis |
[44, 60, 63, 87, 103–105, 127, 134] | |
| Emotion (117) | Emotion caused by use of the CDSS (content) (42) |
- Too confronting outputs - Too frequent/inappropriate alerts - Specific conditions associated with fear (e.g., cancer) - Confusion about outputs - PCPs’ fear that patients might not be able to cope with results |
[24, 42, 45, 47–49, 56–59, 62, 67, 69, 79–81, 84, 87, 88, 90, 91, 99, 101–104, 106, 122, 124, 126, 130] |
| Emotion caused by implementation issues/technology (33) |
- Technical issues leading to frustration (especially if early experience) - Concerns and fear about data security issues/potential misuse of data |
[24, 40, 41, 44–48, 54, 67, 68, 78, 86, 94, 95, 97, 104, 107, 112, 116, 123, 129, 132, 134] | |
| Providers’ prospective emotions (24) |
- Scepticism - Anxiety - Threat - Feeling uncomfortable |
[42, 45, 47, 48, 58, 62, 68, 69, 72, 77, 78, 86, 96, 100, 101, 109, 117, 118, 125, 129, 134] | |
| Patients’ prospective emotions (18) |
- Fear - Uncertainty or confusion - Feeling suspicious during consultations - Use of term “artificial intelligence” as fear-inducing |
[40, 62, 69, 78, 85, 91, 106, 133] | |
| Goals/Intention (192) | Understanding benefit/purpose (97) |
- Confusion about the purpose - Lack of additional benefit if targeted condition “straight-forward” to assess - Doubt about potential benefits of screening - Need to suspect specific condition to start CDSS - Lack of fit with local needs - CDSS seen as peripheral to tasks/pure research or training CDSS - Reported high value of the CDSS is contradicted by behaviour of the CDSS not being used |
[24, 25, 40, 44, 48, 49, 52, 56–58, 60–63, 66–68, 72, 73, 75–78, 81–83, 86, 87, 89–91, 93–97, 99–101, 103, 104, 107, 109, 115, 116, 118, 120, 121, 123–125, 129, 130, 133, 134] |
| General attitudes and willingness to use CDSS (60) |
- Lack of motivation/interest - Forgetting CDSS indicating a lack of motivation - Pre-conceived perceptions of CDSS being complicated |
[25, 41, 44, 46, 50, 55, 57, 58, 61, 65, 68, 69, 75, 76, 78, 81–88, 94, 97, 100, 102, 104, 106, 110, 113, 114, 116, 118, 119, 121, 123, 125, 133] | |
| Change resistance vs openness for change (35) |
- General aversion for the use of technology and changes in primary care practice - Preference for traditional consultations and processes - Aversion higher in older and male clinicians, and higher in resource-abundant settings |
[45, 48, 51, 61, 63, 65, 68, 69, 73, 76–78, 81, 86, 100, 102, 106, 110, 116, 118, 130, 132, 135] | |
| Beliefs about consequences (331) | Effect of CDSS on decision-making process (164) |
Beliefs about a lack of consequence or even negative consequences: - Referral decisions made regardless of CDSS - Lack of congruency with own assessment - CDSS too simple, inaccurate, irrelevant, not including all information - CDSS interfering with thought processes, not aligning with needs or not providing guidance - Concerns about CDSS introducing bias in diagnostic decision-making |
[24, 25, 40, 42, 43, 47, 48, 50–52, 54, 56–60, 62, 64–67, 69, 74–76, 79–81, 84–91, 93, 95, 99–109, 112–115, 117, 118, 120–123, 126, 129–135] |
| Effectiveness of CDSS for patient outcomes and experiences (115) |
Concerns about a lack of accuracy or quality of medical information - Wrong or too general diagnoses - Overdiagnosis or false negative results and mistreatment - Harm/patient safety issues - Concerns that recommendations cannot be followed due to limited resources or a mismatch with patient needs - Delayed care for patients with more urgent needs Concerns about further marginalisation of vulnerable groups Misuse of data (e.g., by insurance companies) Danger of CDSS being used by unqualified staff |
[40, 42–50, 54, 56, 58, 60–62, 65, 67–70, 73, 74, 77, 78, 80, 81, 84, 85, 87, 88, 91, 94, 95, 98–100, 102, 104–106, 109, 111, 116, 118, 122, 124, 127, 129, 130, 132, 133] | |
| Effect of CDSS on diagnostic testing (52) |
- CDSS not streamlining diagnostic processes - Unnecessary or wrong testing - Lack of perceived effectiveness on test-ordering |
[25, 40, 42, 44, 46, 47, 50, 51, 54, 57, 65, 66, 68, 71, 74, 76, 81, 84–86, 92, 95, 99, 101, 103, 104, 122, 123, 126, 130, 132, 135] | |
| Reinforcement (11) | Feedback and incentives (11) |
- Lack of tailored feedback whether diagnostic decisions are adequate - Competing incentive programmes - Repeated negative experience and difficulty solving clinical reminders |
[46, 50, 58, 61, 64, 78, 81, 88, 101, 107] |
| Behavioural regulation (7) |
- Developing new habit to simply dismiss alerts - Decrease of use over longer time periods - Being “stuck” in traditional ways of decision-making, especially older PCPs |
[26, 60, 67, 80, 81, 118] |
aFrequencies of barriers and facilitators in line with the TDF framework are presented to illustrate how often the role of each TDF domain was investigated
Environmental context and resources
The most investigated TDF domain was Environmental context and resources. Subthemes included time constraints and high workload preventing effective implementation and engagement with CDSS. There were multiple challenges to integrating CDSS into PCPs’ workflow, for example, lack of infrastructure, interruptions in busy schedules or inflexible CDSS. Even small changes to routine, e.g., regarding documentation and follow-up, were difficult to achieve, highlighting the need for flexible, brief and customisable CDSS. Technical issues and a lack of integration into existing IT systems resulted in frustration and lower uptake. The context of multiple CDSS being used in a single setting appeared to act as a barrier by increasing mental load through multiple alerts, as well as technical difficulties. Pre-consultation information collection through patient-facing features were discussed as a potentially time-saving solution, but PCPs remained sceptical about their validity, and some caregivers/patients did not have sufficient time to engage.
Wider system issues included alterations to clinical guidelines, contradictory pressures in referring patients early while protecting secondary care resources, ineffective regulations, and insurance systems that led to low capacity for innovation. PCPs had medicolegal concerns if they did not follow CDSS recommendations, or if CDSS provided inappropriate recommendations. Financial issues and resource shortages regarding staff and IT were a barrier to CDSS use, especially in LMICs.
Memory, attention and decision processes
This domain reflected how the usability and navigation of CDSS required or affected the user’s attention, memory and decision-making process. Most CDSS presented a risk of increased cognitive load due to inadequate prompts. This resulted in a preference for passive, clear, and non-intrusive alerts. Confirmation bias (sticking to an early diagnostic hypothesis) or easy-to-miss alerts potentially reduced CDSS effectiveness. A lack of accurate information gathering during initial assessment, or alternative documentation systems such as paper records made CDSS inadequate or redundant.
Beliefs about capabilities
There was provider scepticism towards the capability of CDSS to generate accurate recommendations as the evidence base for CDSS development was often unclear, which resulted in mistrust. This was the case for classic risk assessment algorithms and newer artificial intelligence methods. Whilst PCPs were generally reluctant to use CDSS as they saw their experience and clinical judgement as superior or feared being seen as less capable by using CDSS, patients (especially in LMICs) perceived healthcare professionals as more capable when using a CDSS. A lack of patients’ health literacy or language skills were a barrier to PCPs’ use of CDSS as it could impair symptom assessment, particularly in telephone triage contexts. Lastly, low provider confidence in using CDSS in general and lack of digital literacy reduced CDSS use.
Skills
A need for continued staff training was identified regarding both use of CDSS and managing conditions (details in Table 3). In LMICs, providers such as Accredited Social Health Activists (ASHAs) or midwives reported that they believed CDSS helped improve their skills to recognise health conditions, whereas highly skilled clinicians from high-income countries feared a reduction in PCPs’ skills and clinical judgement if they relied on CDSS too much.
Knowledge
Barriers in the Knowledge domain included a lack of awareness about existing CDSS and technical advances. CDSS recommendations sometimes conflicted with existing clinical guidelines, especially in fields with changing recommendations, e.g., screening for prostate cancer. Providers’ previous knowledge about the condition either facilitated their use of CDSS if they knew how to follow-up on CDSS recommendations, or it reduced the perceived need for a CDSS. Patients’ beliefs about their condition could be a barrier as this affected their trust in assessment and treatment. The combination of CDSS with tailored patient education materials encouraged use as it facilitated the communication of results.
Social influences
Some providers and patients perceived CDSS as a danger to the fundamental patient-provider relationship. Patients presenting with multiple issues or expecting to be treated in a certain way reduced CDSS use. Stigma around certain conditions, especially mental health conditions, led to lower engagement with CDSS as patients were hesitant to discuss symptoms. Lack of stakeholder engagement and community involvement led to lower uptake.
Social/professional role and identity
PCPs understood themselves to be highly skilled, and to have a central role in patient care; they saw their clinical judgement as either crucial for CDSS use or superior to CDSS. Some PCPs perceived CDSS as an existential threat to these skills and their role, which led to a reduction in their intention to use CDSS. A lack of ownership or unclear responsibilities around CDSS implementation and managing the target condition hindered use. CDSS were not flexible enough to cover individual approaches to assessment and consultation. In LMICs however, some PCPs experienced positive effects on their professional reputation and standing within communities when they used CDSS.
Emotion
PCPs emotions towards CDSS could range from excitement to anxiety about CDSS. Patients’ fear or uncertainty regarding technology or a severity of diagnosis affected PCPs acceptance of CDSS as they perceived dealing with patients’ emotion around CDSS as an additional task. CDSS outputs could trigger negative emotions if they were perceived as too confronting (e.g., red warning sign), and if PCPs lacked the confidence to deal with them during their consultation. This was especially true for serious conditions or conditions with negative preconceptions such as cancer, but also if the CDSS used AI methods.
Goals and intention
Confusion about the purpose and doubts about the potential benefit of CDSS hindered implementation. Some studies also reported a general resistance to change, with negative attitudes towards the future of AI and other technical solutions that hindered use, especially in older PCPs.
Beliefs about consequences
Some providers did not believe that CDSS could positively influence decision-making or believed that implementation of CDSS would lead to additional testing and thus increased costs in settings with limited resources. Concerns about accuracy were related to beliefs about negative consequences such as false negative results or overdiagnosis that potentially impacted patient safety.
Recommendations for the implementation of CDSS
By linking the identified barriers and facilitators to the Behaviour Change Wheel (BCW) framework, we specified implementation recommendations for design and development teams, primary care teams, as well as commissioners and policymakers (see Fig. 2). The latter span across local, regional and national levels, including trust leaders, professional associations, primary care networks, governments, and health agencies.
Fig. 2.
Recommendations for the design and implementation of CDSS in primary care
For CDSS Design and development teams, recommendations include: integrating CDSS within existing Electronic Health Record (EHR) and IT systems; aligning CDSS with current practice and complex workflow; ensuring front-end usability by developing simple, intuitive and attractive CDSS; supporting adaptability; designing CDSS for appropriate diagnostic challenges; engaging stakeholders early and throughout the developmental process; testing and evaluating CDSS adequately prior to implementation, and communicating the value of CDSS for patients and providers and actively addressing concerns. Primary care teams should clarify team responsibilities, ensure CDSS training and education, and engage and support patients.
Commissioners and policymakers should provide a strategy to incorporate sustainable digital solutions in healthcare systems with clear governance and quality control mechanisms. They should clarify liability and medicolegal practices. Guidance on the use of evidence-based CDSS should be provided endorsing CDSS use. In terms of fiscal policies, commissioners and policy makers should provide adequate funding for staff and sufficient IT resources to facilitate the development, integration, use and maintenance of CDSS.
Discussion
Summary
This systematic review synthesised existing evidence from 99 studies on barriers and facilitators to the implementation of computerised CDSS for disease detection in primary care and mapped these to a comprehensive implementation framework to develop recommendations for successful implementation. The qualitative synthesis revealed a multitude of complex barriers across a range of domains, including practical implementation issues in the context of primary care, problems with the usability of systems, PCPs’ and patients’ attitudes and beliefs about efficacy, risks or negative outcomes of CDSS, a lack of skills and knowledge regarding technology and target conditions, as well as social barriers. The range and scale of the barriers highlights the complex challenges of implementation.
Strengths and limitations
Whilst there are many strengths to this review including use of a comprehensive search strategy and implementation framework, there are several limitations that may affect the reported barriers and quality of the developed recommendations. As with other reviews, this review is limited by the quality and data covered in the included studies. The number of identified studies and the resulting amount of data made it challenging to conduct in-depth analyses to compare studies from different settings or focusing on different conditions. As primary care staff varies across contexts (e.g., ASHAs or community midwives were more responsible for CDSS use in LMICs), future research should address barriers that might be specific to different primary care staff groups or specific conditions. Results indicate that practical barriers such as lack of (IT) infrastructure might be more relevant in LMICs, whereas PCPs in HICs seem more sceptical about the value of CDSS.
Implementation frameworks such as the NASSS framework [37] and the GUIDES checklist [136] emphasise the importance of sustainability, stepwise implementation and continuous improvement. However, only a few studies included in the current review addressed the implementation of a system at different stages of the development process (e.g., [94, 95, 111, 112] or [26, 87, 107, 108]) or addressed CDSS use over time [25, 102, 121]. It is not known whether that is due to a lack of research beyond the initial or planned introduction of a new CDSS or that most CDSS simply do not reach the stage of implementation in routine care. It could also be argued that there is bias in the published literature, with research focussing on CDSS when there is an issue with implementation, thus factors documenting successful implementation may be less well represented. On the other hand, negative findings might be less likely to be published, leading to our review underestimating barriers. A limitation is therefore that we did not search for unpublished literature.
We report the stage of development for each CDSS, but we did not stratify findings from studies reporting on hypothetical, pilot and routinely used CDSS. Anticipated barriers and facilitators identified for hypothetical scenarios may not be demonstrated when a hypothetical CDSS is actually developed and implemented. Instead, different and potentially unanticipated factors may emerge. Implementation in pilot studies might differ from use in routine care if more support can be offered when CDSS are still prototypes. Further stratification of results may highlight potential differences in barriers and facilitators. However, we believe that our recommendations remain relevant for all scenarios, as we derived them from a range of studies covering the whole development and implementation process. In their previous review, Meunier et al. [33] excluded CDSS at concept or prototype stage. Our review especially adds to the understanding of human factors by providing a more detailed account of barriers and facilitators around perceived consequences for quality of diagnosis and care, intentions to use CDSS, as well as prospective emotions.
Finally, we report frequencies of how many statements were assigned to TDF domains and subthemes. Whilst these likely indicate the most salient barriers, this should be interpreted with caution, especially as the theoretical or conceptual underpinning of the included studies was very limited. This might have led to an imbalance about which barriers and facilitators were investigated and thus represented in this review. As highlighted by the large number of statements in the Environmental context and resources domain, practical issues in integrating CDSS at practice level are among the most relevant barriers. However, they might mask underlying issues that were not explored in depth due to a lack of theoretical underpinning.
Comparison with existing literature
This review is not the first to identify barriers to the implementation of CDSS. For example, Bradley et al. [31] conducted a qualitative review on GPs'attitudes towards cancer-specific CDSS, showing that GPs had concerns about consequences for their clinical judgement. Chen et al. [30] reviewed facilitators including perceived usefulness to provide relevant knowledge and structured care, confidence built through training, as well as functional technical features and external factors such as allocated personnel and technical support. They identified poor workflow integration as a barrier along with limited applicability of CDSS in multimorbidity. More recently, Fletcher et al.’s scoping review [32] of healthcare professionals’ perceptions of CDSS implementation on their workflow found that CDSS can impact cognitive load and can cause alert fatigue. Meunier et al. [33] quantified barriers and facilitators to the implementation of CDSS in primary care using the HOT-fit framework, mirroring findings that increased workload is a crucial barrier. However, our current review covers a wider range of implementation challenges through the application of a well-established implementation framework [35], includes a more comprehensive range of studies on CDSS in primary care across healthcare settings and conditions, and uses the findings to develop practical recommendations.
The complexity of the implementation challenge at hand is also reflected in the recommendations, developed for different groups of stakeholders. While clear evidence-based recommendations have been made for design and development teams and primary care teams, support from commissioners and policymakers is just as crucial for successful CDSS implementation. For example, liability and medicolegal practice need to be clarified by legislative bodies, professional associations need to provide guidance and endorse CDSS use, and governments need to provide adequate funding supporting new digital strategies. Previous guidance for developing complex healthcare interventions show the need to understand the context and wider system throughout the development process [37, 137]. This systematic review highlights the scale of structural issues that are linked to sociopolitical decisions and healthcare system funding. Considering the complexity of the tasks that are needed to implement CDSS successfully, current implementation efforts might not be adequate [138]. This is in line with literature describing failed trials [24, 46] as well as substantial barriers to CDSS routine use even after commissioning in practice [42, 47, 101].
Our patient and public representatives reflected on emerging results and agreed that successful CDSS development and implementation might exceed existing resources within the current context of primary care. They were also concerned whether primary care records provided sufficient information to make a preliminary diagnosis. One representative suggested to focus on how digital systems could save time and effort for both patients and PCPs in the diagnostic process, and to clarify benefits for use.
Practice and policy implications
This review provides evidence-based recommendations that underpin successful implementation of CDSS. One clear message is that to provide feasible and effective decision support, one must consider the complex context of primary care. Most CDSS described in this review presuppose a single condition and address the prevalence of this target condition in a binary question, e.g., depression “yes” or “no”. This requires users to have a diagnosis in mind before using a CDSS. The diagnostic approach in primary care is more complex and probabilistic than this binary model assumes, starting with a range of possible diagnoses and progressively narrowing them down according to their probabilities [139]. There are early feasibility evaluations of CDSS providing differential diagnosis from the 1980s and 1990s, e.g., providing a knowledge base with basic disease descriptions and recommendations [109, 125] or building diagnostic models [64]. Yet computing technology was often too limited to account for the complex nature of decision-making, leading to unsatisfactory results [76]. More recently, ADA Dx was developed to accelerate rare disease diagnosis [140], prompting next steps in the assessment and offering probabilities for each diagnosis. Although this used more complex computing, the system was deemed not commercially viable as it addressed rare diseases.
CDSS typically utilise already existing data in the clinical record, rather than prompting the clinician to seek further confirming or disconfirming information, or to conduct further investigations. Hence, many CDSS fail to acknowledge that decision-making in primary care is a dynamic process. Previous studies have not sufficiently examined and described consequences for workflow [141], although providing automatic decision support as part of PCPs’ workflow has been found to be an important facilitator [142, 143]. Future efforts to develop and implement CDSS need to improve CDSS congruency with the diagnostic approach of PCPs. CDSS need to elicit additional information before or during consultations and need to use more sophisticated technology to support more complex and dynamic decision-making. Researchers need to lead conversations with stakeholders including patients, PCPs, developers, commissioners and policymakers about feasible alternatives outside of the consultation. For example, pre-consultation collection of patient information using apps has been found to improve diagnostic accuracy in emergency healthcare settings [144]. This is in line with our recommendation to integrate patient-facing and primary care team components along the primary care pathway to enable easy assessment, documenting and monitoring. As current IT infrastructure is falling behind possibilities of modern technology, first steps in making CDSS more usable for PCPs could focus on administrative tasks identifying patients for review or facilitating referrals between primary and secondary care [145].
If CDSS are fully integrated at the point of decision-making in PCPs’ workflow, they could become mandatory to manage referrals and further investigations and thus create a more unified approach, as discussed in Black et al. [47]. However, mandatory integration at the point of care, e.g., as a gateway for referral, would need to be carefully prepared, as illustrated in our recommendations. Design and development teams would need to ensure technical and adequate workflow integration prior to implementation. Additionally, liability and medicolegal practice need to be clarified as summarised in the recommendations for policymakers.
Some CDSS we identified in this review have focused on guiding decision-making for test-ordering rather than risk assessment to target a specific point of care [25, 46, 71, 84, 135]. Nevertheless, practical barriers such as a lack of system usability, and lack of fit with test-ordering guidelines still posed significant barriers in these studies. Again, this indicates the need for development teams to work with commissioners and policymakers on macro-levels. Teams need to ensure that CDSS reflect current guidelines and establish opportunities to adapt CDSS in case of changing guidelines.
Conclusions
This systematic review identified a wide range of interrelated barriers concerning the uptake and use of CDSS for early disease detection in primary care, ranging from IT and workflow barriers within primary care to wider system issues. Recommendations for development teams, primary care teams, commissioners and policymakers highlight the complexity of the implementation task. Although recommendations can be used to improve implementation of CDSS, our findings emphasise the need to carefully reflect on the feasibility of CDSS in primary care, starting at the point of design and development. Primary care teams and patients need to be closely involved in all steps of the development and implementation of CDSS.
Supplementary Information
Acknowledgements
The authors would like to thank Paula Funnell for her support in developing the search strategy. We would like to acknowledge Yo Green and David Holden who supported this manuscript as patient and public representatives.
Author contributions
CD contributed to conception and design, title and abstract screening, full-text review, data extraction, analysis and interpretation, and drafting the manuscript. FW and TR were involved in the conception and design, interpretation of the data, and revision of the manuscript. AA worked on title and abstract screening, full-text review, data extraction and analysis, and revision of the manuscript. AP contributed to the data extraction and analysis. TS contributed to design and title and abstract screening. GR contributed to writing and revising manuscript. SES contributed to conception and design, analysis and interpretation of the data, and revised the manuscript. All authors read and approved the final manuscript.
Funding
This study was funded by a Cancer Research UK programme grant ('CANDETECT: Accelerating detection of upper gastro-intestinal (UGI) cancers using a multi-cancer early detection platform in primary care'. EDDPGM-May22\100002). SES is supported by Barts Charity (G-001520; MRC&U0036).
Data availability
The datasets generated and/or analysed during the current study are available in the Open Science Framework repository, https://osf.io/ejkhc.
Declarations
Ethics approval and consent to participate.
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Sripa P, Hayhoe B, Garg P, Majeed A, Greenfield G. Impact of GP gatekeeping on quality of care, and health outcomes, use, and expenditure: a systematic review. Br J Gen Pract. 2019;69(682):e294–303. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Greenfield G, Foley K, Majeed A. Rethinking primary care’s gatekeeper role. BMJ. 2016;354:i4803. [DOI] [PubMed] [Google Scholar]
- 3.Singh H, Schiff GD, Graber ML, Onakpoya I, Thompson MJ. The global burden of diagnostic errors in primary care. BMJ Qual Saf. 2017;26(6):484–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Holtedahl K. Challenges in early diagnosis of cancer: the fast track. Scand J Prim Health Care. 2020;38(3):251–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Morgan E, De Lima B, Aleksandrova T, Sanders L, Eckstrom E. Overcoming Barriers to Early Dementia Diagnosis and Management in Primary Care. J Gen Intern Med. 2021;36(8):2486–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kostopoulou O, Delaney BC, Munro CW. Diagnostic difficulty and error in primary care–a systematic review. Fam Pract. 2008;25(6):400–13 [DOI] [PubMed] [Google Scholar]
- 7.Knight AW, Senior TP. The common problem of rare disease in general practice. Med J Aust. 2006;185(2):82–3. [DOI] [PubMed] [Google Scholar]
- 8.Roland M, Everington S. Tackling the crisis in general practice. BMJ. 2016;352:i942. [DOI] [PubMed] [Google Scholar]
- 9.Hobbs FDR, Bankhead C, Mukhtar T, Stevens S, Perera-Salazar R, Holt T, et al. Clinical workload in UK primary care: a retrospective analysis of 100 million consultations in England, 2007–14. The Lancet. 2016;387(10035):2323–30 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hardy V, Yue A, Archer S, Merriel SWD, Thompson M, Emery J, et al. Role of primary care physician factors on diagnostic testing and referral decisions for symptoms of possible cancer: a systematic review. BMJ Open. 2022;12(1):e053732. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Evans J, Ziebland S, Macartney JI, Bankhead CR, Rose PW, Nicholson BD. GPs’ understanding and practice of safety netting for potential cancer presentations: a qualitative study in primary care. Br J Gen Pract. 2018;68(672):e505–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Tompson A, Nicholson BD, Ziebland S, Evans J, Bankhead C. Quality improvements of safety-netting guidelines for cancer in UK primary care: insights from a qualitative interview study of GPs. Br J Gen Pract. 2019;69(689):e819–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Friedemann Smith C, Duncombe S, Fleming S, Hirst Y, Black G, Bankhead C, et al. What features are considered important for electronic safety-netting tools to UK general practice staff? An interview and Delphi consensus study. BJGP Open. 2023;7(3):BJGPO.2022.0163. [DOI] [PMC free article] [PubMed]
- 14.Royce CS, Hayes MM, Schwartzstein RM. Teaching Critical Thinking: A Case for Instruction in Cognitive Biases to Reduce Diagnostic Errors and Improve Patient Safety. Acad Med. 2019;94(2):187–94. [DOI] [PubMed] [Google Scholar]
- 15.Chen W, Howard K, Gorham G, O’Bryan CM, Coffey P, Balasubramanya B, et al. Design, effectiveness, and economic outcomes of contemporary chronic disease clinical decision support systems: a systematic review and meta-analysis. J Am Med Inform Assoc. 2022;29(10):1757–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digital Medicine. 2020;3(1):17 [DOI] [PMC free article] [PubMed]
- 17.Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of Clinical Decision-Support Systems A Systematic Review. Ann Intern Med. 2012;157(1):29-U77. [DOI] [PubMed] [Google Scholar]
- 18.Kilsdonk E, Peute LW, Riezebos RJ, Kremer LC, Jaspers MWM. Uncovering healthcare practitioners’ information processing using the think-aloud method: From paper-based guideline to clinical decision support system. Int J Med Informatics. 2016;86:10–9. [DOI] [PubMed] [Google Scholar]
- 19.Harada T, Miyagami T, Kunitomo K, Shimizu T. Clinical Decision Support Systems for Diagnosis in Primary Care: A Scoping Review. Int J Environ Res Public Health. 2021;18(16):8435. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Berner ES. Diagnostic decision support systems: why aren’t they used more and what can we do about it? AMIA Annu Symp Proc. 2006;2006:1167–8 [PMC free article] [PubMed] [Google Scholar]
- 21.Segal MM, Rahm AK, Hulse NC, Wood G, Williams JL, Feldman L, et al. Experience with Integrating Diagnostic Decision Support Software with Electronic Health Records: Benefits versus Risks of Information Sharing. EGEMS (Wash DC). 2017;5(1):23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Price S, Spencer A, Medina-Lara A, Hamilton W. Availability and use of cancer decision-support tools: a cross-sectional survey of UK primary care. Br J Gen Pract. 2019;69(684):e437–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bindels R, Hasman A, van Wersch JW, Talmon J, Winkens RA. Evaluation of an automated test ordering and feedback system for general practitioners in daily practice. Int J Med Informatics. 2004;73(9):705–12. [DOI] [PubMed] [Google Scholar]
- 24.Rubin G, Walter FM, Emery J, Hamilton W, Hoare Z, Howse J, et al. Electronic clinical decision support tool for assessing stomach symptoms in primary care (ECASS): a feasibility study. BMJ Open. 2021;11(3):e041795. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Curry L, Reed MH. Electronic decision support for diagnostic imaging in a primary care setting. J Am Med Inform Assoc. 2011;18(3):267–70 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.McCullagh L, Mann D, Rosen L, Kannry J, McGinn T. Longitudinal adoption rates of complex decision support tools in primary care. Evid Based Med. 2014;19(6):204–9. [DOI] [PubMed] [Google Scholar]
- 27.Sarkar U, Samal L. How effective are clinical decision support systems? BMJ. 2020;370:m3499. [DOI] [PubMed] [Google Scholar]
- 28.Kwan JL, Lo L, Ferguson J, Goldberg H, Diaz-Martinez JP, Tomlinson G, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ. 2020;370:m3216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Mcinerney CD, Scott BC, Johnson OA. Are Regulations Safe? Reflections From Developing a Digital Cancer Decision-Support Tool. JCO Clinical Cancer Informatics. 2021;5:353–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Chen W, O’Bryan CM, Gorham G, Howard K, Balasubramanya B, Coffey P, et al. Barriers and enablers to implementing and using clinical decision support systems for chronic diseases: a qualitative systematic review and meta-aggregation. Implementation Science Communications. 2022;3(1):81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Bradley PT, Hall N, Maniatopoulos G, Neal RD, Paleri V, Wilkes S. Factors shaping the implementation and use of Clinical Cancer Decision Tools by GPs in primary care: a qualitative framework synthesis. BMJ Open. 2021;11(2):e043338. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Fletcher E, Burns A, Wiering B, Lavu D, Shephard E, Hamilton W, et al. Workload and workflow implications associated with the use of electronic clinical decision support tools used by health professionals in general practice: a scoping review. BMC Primary Care. 2023;24(1):23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Meunier P-Y, Raynaud C, Guimaraes E, Gueyffier F, Letrilliart L. Barriers and Facilitators to the Use of Clinical Decision Support Systems in Primary Care: A Mixed-Methods Systematic Review. The Annals of Family Medicine. 2023;21(1):57–69. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Michie S, Van Stralen MM, West R. The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation Science. 2017;12(1):77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Michie S, Atkins L, West R. The Behaviour Change Wheel: A Guide to Designing Interventions. Sutton, UK: Silverback Publishing; 2014. [Google Scholar]
- 37.Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J Med Internet Res. 2017;19(11): e367. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14(1):579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Harrison R, Jones B, Gardner P, Lawton R. Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies. BMC Health Serv Res. 2021;21(1):144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Abejirinde IOO, Douwes R, Bardají A, Abugnaba-Abanga R, Zweekhorst M, van Roosmalen J, et al. Pregnant women’s experiences with an integrated diagnostic and decision support device for antenatal care in Ghana. BMC Pregnancy and Childbirth. 2018;18(1):209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.af Klercker T, Zetraeus S. Dilemmas in introducing World Wide Web-based information technology in primary care: a focus group study. Family Practice. 1998;15(3):205–210. [DOI] [PubMed]
- 42.Akanuwe JNA, Black S, Owen S, Siriwardena AN. Barriers and facilitators to implementing a cancer risk assessment tool (QCancer) in primary care: a qualitative study. Primary Health Care Research and Development. 2021;22:e51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Alwadhi V, Bajpayee D, Kumar N, Mohanty JS, Mukherji K, Saboth PK, et al. E-IMNCI: a novel clinical diagnostic support system approach to strengthen effectiveness and quality of IMNCI implementation in India. BMJ open quality. 2023;12(Suppl 3):e001857. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Bangash H, Pencille L, Gundelach JH, Makkawy A, Sutton J, Makkawy L, et al. An Implementation Science Framework to Develop a Clinical Decision Support Tool for Familial Hypercholesterolemia. J Pers Med. 2020;10(3):67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Barry B, Zhu X, Behnken E, Inselman J, Schaepe K, McCoy R, et al. Provider Perspectives on Artificial Intelligence-Guided Screening for Low Ejection Fraction in Primary Care: Qualitative Study. JMIR AI. 2022;1(1):e41940. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Bindels R, Hasman A, Derickx M, van Wersch JWJ, Winkens RAG. User satisfaction with a real-time automated feedback system for general practitioners: a quantitative and qualitative study. Int J Qual Health Care. 2003;15(6):501–8. [DOI] [PubMed] [Google Scholar]
- 47.Black GB, Machen S, Parker-Deeks S, Cronin A, Chung DN. Using an electronic safety netting tool designed to improve safety with respect to cancer referral in primary care: a qualitative service evaluation using rapid appraisal methods. Bmj Open Quality. 2023;12(3):e002354. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Buck C, Doctor E, Hennrich J, Jöhnk J, Eymann T. General Practitioners’ Attitudes Toward Artificial Intelligence-Enabled Systems: Interview Study. J Med Internet Res. 2022;24(1):e28916. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Canny A, Donaghy E, Murray V, Campbell L, Stonham C, Bush A, et al. Patient views on asthma diagnosis and how a clinical decision support system could help: A qualitative study. Health Expect. 2023;26(1):307–17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Cano M, Ruiz-Postigo JA, Macharia P, Amoako YA, Phillips RO, Kinyeru E, et al. Evaluating the World Health Organization’s SkinNTDs App as a Training Tool for Skin Neglected Tropical Diseases in Ghana and Kenya: Cross-Sectional Study. J Med Internet Res. 2024;26:e51628. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Carlsson SV, Preston M, Vickers A, Malhotra D, Ehdaie B, Healey M, et al. Provider Perceptions of an Electronic Health Record Prostate Cancer Screening Tool. Appl Clin Inform. 2024;15(2):282–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Carlsson SV, Preston MA, Vickers A, Malhotra D, Ehdaie B, Healey MJ, et al. A Provider-Facing Decision Support Tool for Prostate Cancer Screening in Primary Care: A Pilot Study. Appl Clin Inform. 2024;15(02):274–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Carney TJ, Weaver M, McDaniel AM, Jones J, Haggstrom DA. Organizational factors influencing the use of clinical decision support for improving cancer screening within community health centers. E-Health and Telemedicine: Concepts, Methodologies, Tools, and Applications. 2015;1:118–48. [Google Scholar]
- 54.Carter J, Goldsmith LP, Knights F, Deal A, Jayakumar S, Crawshaw AF, et al. Health Catch-UP!: a realist evaluation of an innovative multi-disease screening and vaccination tool in UK primary care for at-risk migrant patients. BMC Med. 2024;22(1):497. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Chadwick D, Forbes G, Lawrence C, Lorrimer S, van Schaik P. Using an electronic health record alert to prompt blood-borne virus testing in primary care. AIDS. 2021;35(11):1845–50. [DOI] [PubMed] [Google Scholar]
- 56.Chiang PPC, Glance D, Walker J, Walter FM, Emery JD. Implementing a QCancer risk tool into general practice consultations: an exploratory study using simulated consultations with Australian general practitioners. Br J Cancer. 2015;112:S77–83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Chima S, Martinez-Gutierrez J, Hunter B, Manski-Nankervis JA, Emery J. Optimization of a Quality Improvement Tool for Cancer Diagnosis in Primary Care: Qualitative Study. Jmir Form Res. 2022;6(8):e39277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Constantin A, Atkinson M, Bernabeu MO, Buckmaster F, Dhillon B, McTrusty A, et al. Optometrists’ Perspectives Regarding Artificial Intelligence Aids and Contributing Retinal Images to a Repository: Web-Based Interview Study. Jmir Human Factors. 2023;10:e40887. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Cunich M, Salkeld G, Dowie J, Henderson J, Bayram C, Britt H, et al. Integrating Evidence and Individual Preferences Using a Web-Based Multi-Criteria Decision Analytic Tool An Application to Prostate Cancer Screening. Patient-Patient Centered Outcomes Research. 2011;4(3):153–62. [DOI] [PubMed] [Google Scholar]
- 60.Daines L, Donaghy E, Canny A, Murray V, Campbell L, Stonham C, et al. Clinician views on how clinical decision support systems can help diagnose asthma in primary care: a qualitative study. J Asthma. 2024;61(4):377–85. [DOI] [PubMed] [Google Scholar]
- 61.Daniel M, Kaur A, Mukherjee A, Bhattacharya A, Tewari A, Sagar R, et al. The systematic medical appraisal, referral and treatment (SMART) mental health programme: formative research informing a cluster randomized controlled trial. SSM - Mental Health. 2023;3:100223. [Google Scholar]
- 62.Dannenberg MD, Bienvenida JCM, Bruce ML, Nguyen T, Hinn M, Matthews J, et al. End-user views of an electronic encounter decision aid linked to routine depression screening. Patient Educ Couns. 2019;102(3):555–63. [DOI] [PubMed] [Google Scholar]
- 63.Dikomitis L, Green T, Macleod U. Embedding electronic decision-support tools for suspected cancer in primary care: a qualitative study of GPs’ experiences. Primary Health Care Research and Development. 2015;16(6):548–55. [DOI] [PubMed] [Google Scholar]
- 64.Dupuits F, Hasman A, Pop P. Computer-based assistance in family medicine. Comput Methods Programs Biomed. 1998;55(1):39–50. [DOI] [PubMed] [Google Scholar]
- 65.Ellington LE, Najjingo I, Rosenfeld M, Stout JW, Farquhar SA, Vashistha A, et al. Health workers’ perspectives of a mobile health tool to improve diagnosis and management of paediatric acute respiratory illnesses in Uganda: a qualitative study. BMJ Open. 2021;11(7):e049708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Engelen A, Vanderhaegen J, Van Poppel H, Van Audenhove C. The use of decision aids on early detection of prostate cancer: views of men and general practitioners. Health Expect. 2017;20(2):221–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Ford E, Edelman N, Somers L, Shrewsbury D, Levy ML, van Marwijk H, et al. Barriers and facilitators to the adoption of electronic clinical decision support systems: a qualitative interview study with UK general practitioners. BMC Med Inform Decis Mak. 2021;21(1):193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Frisinger A, Papachristou P. The voice of healthcare: introducing digital decision support systems into clinical practice-a qualitative study. BMC Primary Care. 2023;24(1):67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Ginsburg AS, Agyemang CT, Ambler G, Delarosa J, Brunette W, Levari S, et al. MPneumonia, an innovation for diagnosing and treating childhood pneumonia in low-resource settings: A feasibility, usability and acceptability study in Ghana. PLOS ONE. 2016;11(10):e0165201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Gonçalves AQ, Sequeira-Aymar E, Martin CA, Dalmau RM, Cruz A, Evangelidou S, et al. Usefulness and practicality of a multidisease screening programme targeting migrant patients in primary care in Spain: a qualitative study of general practitioners. BMJ Open. 2022;12(11):e065645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Griffith JK, Borycki EM, Kushniruk AW. Diagnostic imaging ordering practices: physician perspectives and implications for decision support. Healthcare quarterly (Toronto, Ont). 2014;17(2):62–70. [DOI] [PubMed] [Google Scholar]
- 72.Guenter D, Abouzahra M, Schabort I, Radhakrishnan A, Nair K, Orr S, et al. Design Process and Utilization of a Novel Clinical Decision Support System for Neuropathic Pain in Primary Care: Mixed Methods Observational Study. JMIR Medical Informatics. 2019;7(3):e14141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.Gunasekeran DV, Zheng F, Lim GYS, Chong CCY, Zhang S, Ng WY, et al. Acceptance and Perception of Artificial Intelligence Usability in Eye Care (APPRAISE) for Ophthalmologists: A Multinational Perspective. Front Med. 2022;9:875242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74.Harper J, Hunt T, Choudry M, Kapron AL, Cooney KA, Martin C, et al. Clinician interest in clinical decision support for PSA-based prostate cancer screening. Urologic Oncology-Seminars and Original Investigations. 2023;41(3):145.e17-145.e23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 75.Helenason J, Ekström C, Falk M, Papachristou P. Exploring the feasibility of an artificial intelligence based clinical decision support system for cutaneous melanoma detection in primary care–a mixed method study. Scand J Prim Health Care. 2024;42(1):51–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Henderson EJ, Rubin GP. The utility of an online diagnostic decision support system (Isabel) in general practice: a process evaluation. JRSM Short Reports. 2013;4(5):31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 77.Ho S, Doig GS, Ly A. Attitudes of optometrists towards artificial intelligence for the diagnosis of retinal disease: A cross-sectional mail-out survey. Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists). 2022;42(6):1170–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 78.Ho V, Brown Johnson C, Ghanzouri I, Amal S, Asch S, Ross E. Physician- and Patient-Elicited Barriers and Facilitators to Implementation of a Machine Learning-Based Screening Tool for Peripheral Arterial Disease: Preimplementation Study With Physician and Patient Stakeholders. JMIR cardio. 2023;7: e44732. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 79.Holmström IK. Decision aid software programs in telenursing: not used as intended?: Experiences of Swedish telenurses. Nurs Health Sci. 2007;9(1):21–6. [DOI] [PubMed] [Google Scholar]
- 80.Holmström IK, Gustafsson S, Wesstrom J, Skoglund K. Telephone nurses’ use of a decision support system: An observational study. Nurs Health Sci. 2019;21(4):501–7. [DOI] [PubMed] [Google Scholar]
- 81.Hunter B, Davidson S, Lumsden N, Chima S, Gutierrez JM, Emery J, et al. Optimising a clinical decision support tool to improve chronic kidney disease management in general practice. BMC primary care. 2024;25(1):220. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 82.Jones RB, Ashurst EJ, Jones D, Gude R, Kaminski E. Development and implementation of a decision pathway for general practitioners for the management or referral of suspected allergy. J Public Health Res. 2014;3(2):54–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 83.Kharbanda EO, Nordin JD, Sinaiko AR, Ekstrom HL, Stultz JM, Sherwood NE, et al. TeenBP: Development and Piloting of an EHR-Linked Clinical Decision Support System to Improve Recognition of Hypertension in Adolescents. EGEMS (Washington, DC). 2015;3(2):1142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Kling SMR, Kalwani NM, Winget M, Gupta K, Saliba-Gustafsson EA, Baratta J, et al. An initiative to promote value-based stress test selection in primary care and cardiology clinics: A mixed methods evaluation. J Eval Clin Pract. 2024;30(1):107–18. [DOI] [PubMed] [Google Scholar]
- 85.Knoble SJ, Bhusal MR. Electronic diagnostic algorithms to assist mid-level health care workers in Nepal: A mixed-method exploratory study. Int J Med Informatics. 2015;84(5):334–40. [DOI] [PubMed] [Google Scholar]
- 86.Kortteisto T, Komulainen J, Mäkelä M, Kunnamo I, Kaila M. Clinical decision support must be useful, functional is not enough: A qualitative study of computer-based clinical decision support in primary care. BMC Health Services Research. 2012;12(1). [DOI] [PMC free article] [PubMed]
- 87.Li AC, Kannry JL, Kushniruk A, Chrimes D, McGinn TG, Edonyabo D, et al. Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support. Int J Med Informatics. 2012;81(11):761–72. [DOI] [PubMed] [Google Scholar]
- 88.Litvin CB, Hyer JM, Ornstein SM. Use of Clinical Decision Support to Improve Primary Care Identification and Management of Chronic Kidney Disease (CKD). J Am Board Fam Med. 2016;29(5):604–12. [DOI] [PubMed] [Google Scholar]
- 89.López MM, Díez ID, Jimeno JCP, López-Coronado M. mHealth App for iOS to Help in Diagnostic Decision in Ophthalmology to Primary Care Physicians. Journal of Medical Systems. 2017;41(5):81. [DOI] [PubMed]
- 90.Lowery J, Fagerlin A, Larkin AR, Wiener RS, Skurla SE, Caverly TJ. Implementation of a Web-Based Tool for Shared Decision-making in Lung Cancer Screening: Mixed Methods Quality Improvement Evaluation. Jmir Human Factors. 2022;9(2). [DOI] [PMC free article] [PubMed]
- 91.Luitel NP, Neupane V, Lamichhane B, Koirala GP, Gautam K, Karki E, et al. Experience of primary healthcare workers in using the mobile app-based WHO mhGAP intervention guide in detection and treatment of people with mental disorders: A qualitative study in Nepal. SSM - Mental Health. 2023;4. [DOI] [PMC free article] [PubMed]
- 92.Maghsoud-Lou E, Christie S, Abidi SR, Abidi SSR. Protocol-Driven Decision Support within e-Referral Systems to Streamline Patient Consultation, Triaging and Referrals from Primary Care to Specialist Clinics. Journal of Medical Systems. 2017;41(9). [DOI] [PubMed]
- 93.McGlynn A, She EN, Bennett P, Liaw ST, Jackson T, Harris-Roxas B. Exploring the spread and scale of a web-based clinical decision support portal in Sydney, Australia, during COVID-19: a case study. Journal of Integrated Care. 2023.
- 94.Militello LG, Arbuckle NB, Saleem JJ, Patterson E, Flanagan M, Haggstrom D, et al. Sources of variation in primary care clinical workflow: Implications for the design of cognitive support. Health Informatics J. 2014;20(1):35–49. [DOI] [PubMed] [Google Scholar]
- 95.Militello LG, Saleem JJ, Borders MR, Sushereba CE, Haverkamp D, Wolf SP, et al. Designing Colorectal Cancer Screening Decision Support: A Cognitive Engineering Enterprise. Journal of Cognitive Engineering and Decision Making. 2016;10(1):74–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 96.Morgan D, Kosteniuk J, Seitz D, O'Connell ME, Kirk A, Stewart NJ, et al. A five-step approach for developing and implementing a Rural Primary Health Care Model for Dementia: a community-academic partnership. Primary Health Care Research and Development. 2019;20. [DOI] [PMC free article] [PubMed]
- 97.Morgan O, Schnur JB, Diefenbach MA, Kale MS. Physician Preferences for an Electronic Lung Cancer Screening Decision Aid. American Journal of Managed Care. 2024;30:SP445-SP451. [DOI] [PMC free article] [PubMed]
- 98.Murdoch J, Barnes R, Pooler J, Lattimer V, Fletcher E, Campbell JL. The impact of using computer decision-support software in primary care nurse-led telephone triage: Interactional dilemmas and conversational consequences. Soc Sci Med. 2015;126:36–47. [DOI] [PubMed] [Google Scholar]
- 99.Nagraj S, Kennedy S, Jha V, Norton R, Hinton L, Billot L, et al. A Mobile Clinical Decision Support System for High-Risk Pregnant Women in Rural India (SMARThealth Pregnancy): pilot Cluster Randomized Controlled Trial. Jmir Formative Research. 2023;7: e44362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 100.Oremule B, Saunders GH, Kulk K, d’Elia A, Bruce IA. Understanding, experience, and attitudes towards artificial intelligence technologies for clinical decision support in hearing health: a mixed-methods survey of healthcare professionals in the UK. Journal of Laryngology and Otology. 2024. [DOI] [PMC free article] [PubMed]
- 101.Pannebakker MM, Mills K, Johnson M, Emery JD, Walter FM. Understanding implementation and usefulness of electronic clinical decision support (eCDS) for melanoma in English primary care: A qualitative investigation. BJGP Open. 2019;3(1). [DOI] [PMC free article] [PubMed]
- 102.Parker BL, Achilles MR, Subotic-Kerry M, O’Dea B. Youth StepCare: a pilot study of an online screening and recommendations service for depression and anxiety among youth patients in general practice. BMC Fam Pract. 2020;21(1):2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 103.Peiris D, Williams C, Holbrook R, Lindner R, Reeve J, Das A, et al. A web-based clinical decision support tool for primary health care management of back pain: development and mixed methods evaluation. JMIR Research Protocols. 2014;3(2): e17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Porat T, Delaney B, Kostopoulou O. The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients. Bmc Medical Informatics and Decision Making. 2017;17. [DOI] [PMC free article] [PubMed]
- 105.Porat T, Kostopoulou O, Woolley A, Delaney BC. Eliciting User Decision Requirements for Designing Computerized Diagnostic Support for Family Physicians. Journal of Cognitive Engineering and Decision Making. 2016;10(1):57–73. [Google Scholar]
- 106.Radovic A, Anderson N, Hamm M, George-Milford B, Fascetti C, Engster S, et al. Perceived Facilitators of and Barriers to Implementation of a Decision Support Tool for Adolescent Depression and Suicidality Screening: Focus Group and Interview Study. Jmir Mental Health. 2021;8(9). [DOI] [PMC free article] [PubMed]
- 107.Richardson S, Feldstein D, McGinn T, Park LS, Khan S, Hess R, et al. Live Usability Testing of Two Complex Clinical Decision Support Tools: Observational Study. JMIR Hum Factors. 2019;6(2): e12471. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 108.Richardson S, Mishuris R, O’Connell A, Feldstein D, Hess R, Smith P, et al. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Informatics. 2017;106:1–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 109.Ridderikhoff J, vanHerk E. A diagnostic support system in general practice: Is it feasible? Int J Med Informatics. 1997;45(3):133–43 [DOI] [PubMed] [Google Scholar]
- 110.Rim SH, Hall IJ, Massetti GM, Thomas CC, Li J, Richardson LC. Primary Care Providers’ Intended Use of Decision Aids for Prostate-Specific Antigen Testing for Prostate Cancer Screening. Journal of cancer education : the official journal of the American Association for Cancer Education. 2019;34(4):666–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 111.Saleem JJ, Haggstrom DA, Militello LG, Flanagan M, Kiess CL, Arbuckle N, et al. Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation. BMC Med Inform Decis Mak. 2011;11:74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 112.Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc. 2005;12(4):438–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 113.Saraiva R, Perkusich M, Silva L, Almeida H, Siebra C, Perkusich A. Early diagnosis of gastrointestinal cancer by using case-based and rule-based reasoning. Expert Syst Appl. 2016;61:192–202. [Google Scholar]
- 114.Schroy PC, Caron SE, Sherman BJ, Heeren TC, Battaglia TA. Risk assessment and clinical decision making for colorectal cancer screening. Health Expect. 2015;18(5):1327–38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 115.Schütze D, Holtz S, Neff MC, Kohler SM, Schaaf J, Frischen LS, et al. Requirements analysis for an AI-based clinical decision support system for general practitioners: a user-centered design process. Bmc Medical Informatics and Decision Making. 2023;23(1). [DOI] [PMC free article] [PubMed]
- 116.Shannon CL, Bartels SM, Cepeda M, Castro S, Cubillos L, Suárez-Obando F, et al. Perspectives on the Implementation of Screening and Treatment for Depression and Alcohol Use Disorder in Primary Care in Colombia. Community Ment Health J. 2021;57(8):1579–87. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 117.Shibl R, Lawley M, Debuse J. Factors influencing decision support system acceptance. Decis Support Syst. 2013;54(2):953–61. [Google Scholar]
- 118.Skurla SE, Leishman NJ, Fagerlin A, Wiener RS, Lowery J, Caverly TJ. Clinician Perceptions on Using Decision Tools to Support Prediction-Based Shared Decision Making for Lung Cancer Screening. MDM Policy and Practice. 2024;9(1). [DOI] [PMC free article] [PubMed]
- 119.Stevens AM, Eardley DL. Evaluation of a technology-enabled tool to improve colorectal cancer screening. Online J Nurs Inform. 2016;20(1):1. [Google Scholar]
- 120.Sturrock A, Bissett SM, Carrozzo M, Lish R, Howe D, Mountain S, et al. Qualitative interview study exploring the early identification and referral of patients with suspected head and neck cancer by community pharmacists in England. BMJ Open. 2023;13(3). [DOI] [PMC free article] [PubMed]
- 121.Sunjaya AP, Martin A, Jenkins C. A Design Thinking Approach to Developing a Clinical Decision Support System for Breathlessness in Primary Care. Studies in health technology and informatics. 2022;290:839–43. [DOI] [PubMed] [Google Scholar]
- 122.Tabla S, Calafiore M, Legrand B, Descamps A, Andre C, Rochoy M, et al. Artificial Intelligence and Clinical Decision Support Systems or Automated Interpreters: What Characteristics Are Expected by French General Practitioners? Studies in health technology and informatics. 2022;290:887–91. [DOI] [PubMed] [Google Scholar]
- 123.Tegenaw GS, Sori DA, Teklemariam GK, Verbeke F, Cornelis J, Jansen B. Evaluation of a Computer-Aided Clinical Decision Support System for Point-of-Care Use in Low-Resource Primary Care Settings: Acceptability Evaluation Study. JMIR Hum Factors. 2024;11(11):e47631. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 124.Tewari A, Kallakuri S, Devarapalli S, Jha V, Patel A, Maulik PK. Process evaluation of the systematic medical appraisal, referral and treatment (SMART) mental health project in rural India. BMC Psychiatry. 2017;17(1):385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 125.Timpka T. Introducing hypertext in primary health care: A study on the feasibility of decision support for practitioners. Comput Methods Programs Biomed. 1989;29(1):1–13. [DOI] [PubMed] [Google Scholar]
- 126.Tseng E, Hsu YJ, Nigrin C, Clark JM, Marsteller JA, Maruthur NM. Improving Diabetes Screening in the Primary Care Clinic. Jt Comm J Qual Patient Saf. 2023;49(12):698–705. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 127.Turnbull J, Prichard J, Pope C, Brook S, Rowsell A. Risk work in NHS 111: the everyday work of managing risk in telephone assessment using a computer decision support system. Health Risk Soc. 2017;19(3–4):189–208. [Google Scholar]
- 128.Turner S, Iliffe S, Downs M, Bryans M, Wilcock J, Austin T. Decision support software for dementia diagnosis and management in primary care: relevance and potential. Aging Ment Health. 2003;7(1):28–33. [DOI] [PubMed] [Google Scholar]
- 129.Upshaw TL, Craig-Neil A, Macklin J, Gray CS, Chan TCY, Gibson J, et al. Priorities for Artificial Intelligence Applications in Primary Care: A Canadian Deliberative Dialogue with Patients, Providers, and Health System Leaders. J Am Board Fam Med. 2023;36(2):210–20. [DOI] [PubMed] [Google Scholar]
- 130.Varonen H, Kortteisto T, Kaila M, Grp EBS. What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract. 2008;25(3):162–7. [DOI] [PubMed] [Google Scholar]
- 131.Wouters LT, Zwart DL, Erkelens DC, Huijsmans M, Hoes AW, Damoiseaux RA, et al. Tinkering and overruling the computer decision support system: Working strategies of telephone triage nurses who assess the urgency of callers suspected of having an acute cardiac event. J Clin Nurs. 2020;29(7–8):1175–86 [DOI] [PubMed] [Google Scholar]
- 132.Wright T, Young K, Darragh M, Corter A, Soosay I, Goodyear-Smith F. Perinatal e-screening and clinical decision support: the Maternity Case-finding Help Assessment Tool (MatCHAT). J Prim Health Care. 2020;12(3):265–71. [DOI] [PubMed] [Google Scholar]
- 133.Yau M, Timmerman V, Zwarenstein M, Mayers P, Cornick RV, Bateman E, et al. e-PC101: an electronic clinical decision support tool developed in South Africa for primary care in low-income and middle-income countries. BMJ Glob Health. 2018;3(Suppl 5): e001093. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 134.Zazove P, McKee M, Schleicher L, Green L, Kileny P, Rapai M, et al. To act or not to act: responses to electronic health record prompts by family medicine clinicians. Journal of the American Medical Informatics Association : JAMIA. 2017;24(2):275–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 135.Zhuang ZY, Wilkin CL, Ceglowski A. A framework for an intelligent decision support system: A case in pathology test ordering. Decis Support Syst. 2013;55(2):476–87. [Google Scholar]
- 136.Van de Velde S, Kunnamo I, Roshanov P, Kortteisto T, Aertgeerts B, Vandvik PO, et al. The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support. Implement Sci. 2018;13(1):86. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 137.O’Cathain A, Croot L, Duncan E, Rousseau N, Sworn K, Turner KM, et al. Guidance on how to develop complex interventions to improve health and healthcare. BMJ Open. 2019;9(8):e029954. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 138.Greenhalgh T, Maylor H, Shaw S, Wherton J, Papoutsi C, Betton V, et al. The NASSS-CAT Tools for Understanding, Guiding, Monitoring, and Researching Technology Implementation Projects in Health and Social Care: Protocol for an Evaluation Study in Real-World Settings. JMIR Research Protocols. 2020;9(5):e16861. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 139.Heneghan C, Glasziou P, Thompson M, Rose P, Balla J, Lasserson D, et al. Diagnostic strategies used in primary care. BMJ. 2009;338:b946. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 140.Ronicke S, Hirsch MC, Türk E, Larionov K, Tientcheu D, Wagner AD. Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study. Orphanet J Rare Dis. 2019;14(1):69. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 141.Roshanov PS, You JJ, Dhaliwal J, Koff D, Mackay JA, Weise-Kelly L, et al. Can computerized clinical decision support systems improve practitioners’ diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review. Implement Sci. 2011;6(1):88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 142.Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 143.Van de Velde S, Heselmans A, Delvaux N, Brandt L, Marco-Ruiz L, Spitaels D, Cloetens H, Kortteisto T, Roshanov P, Kunnamo I, Aertgeerts B. A systematic review of trials evaluating success factors of interventions with computerised clinical decision support. Implement sci. 2018;13:1114. 10.1186/s13012-018-0790-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 144.Faqar-Uz-Zaman SF, Anantharajah L, Baumartz P, Sobotta P, Filmann N, Zmuc D, et al. The Diagnostic Efficacy of an App-based Diagnostic Health Care Application in the Emergency Room: eRadaR-Trial. A prospective, Double-blinded, Observational Study. Ann Surg. 2022;276(5):935-942 [DOI] [PubMed]
- 145.Bradley SH. Forget about replacing doctors with AI—just get our computers to work. BMJ. 2024;385: q1171. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets generated and/or analysed during the current study are available in the Open Science Framework repository, https://osf.io/ejkhc.