Skip to main content
PMC home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2017 Sep 23;25(5):593–602. doi: 10.1093/jamia/ocx100

Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review

Julian Varghese 1,✉, Maren Kleine 2, Sophia Isabella Gessner 1, Sarah Sandmann 1, Martin Dugas 1,3
PMCID: PMC7646949  PMID: 29036406

Abstract

Objectives

To systematically classify the clinical impact of computerized clinical decision support systems (CDSSs) in inpatient care.

Materials and Methods

Medline, Cochrane Trials, and Cochrane Reviews were searched for CDSS studies that assessed patient outcomes in inpatient settings. For each study, 2 physicians independently mapped patient outcome effects to a predefined medical effect score to assess the clinical impact of reported outcome effects. Disagreements were measured by using weighted kappa and solved by consensus. An example set of promising disease entities was generated based on medical effect scores and risk of bias assessment. To summarize technical characteristics of the systems, reported input variables and algorithm types were extracted as well.

Results

Seventy studies were included. Five (7%) reported reduced mortality, 16 (23%) reduced life-threatening events, and 28 (40%) reduced non–life-threatening events, 20 (29%) had no significant impact on patient outcomes, and 1 showed a negative effect (weighted κ: 0.72, P < .001). Six of 24 disease entity settings showed high effect scores with medium or low risk of bias: blood glucose management, blood transfusion management, physiologic deterioration prevention, pressure ulcer prevention, acute kidney injury prevention, and venous thromboembolism prophylaxis. Most of the implemented algorithms (72%) were rule-based. Reported input variables are shared as standardized models on a metadata repository.

Discussion and Conclusion

Most of the included CDSS studies were associated with positive patient outcomes effects but with substantial differences regarding the clinical impact. A subset of 6 disease entities could be filtered in which CDSS should be given special consideration at sites where computer-assisted decision-making is deemed to be underutilized.

Registration number on PROSPERO: CRD42016049946.

Keywords: clinical decision support systems, medical order entry systems, reminder systems, outcome and process assessment

BACKGROUND

Computerized clinical decision support systems (CDSSs) are designed to aid clinical decision-making using individual patient characteristics to generate health-related recommendations.1

CDSS implementations to improve patient care are increasingly being studied in clinical trials.1–7 For instance, Brenner et al.2 conducted a recent systematic review of CDSSs in all health care settings and reported mixed findings (15/40 studies with positive patient outcome effects, 25/40 with nonsignificant effects).

Recent meta-analyses of CDSS effects on patient outcomes are limited to randomized controlled studies, due to methodological issues such as spurious precision, confounding, and potentially stronger selective reporting biases in nonrandomized studies.4

Bright et al.3 and Moja et al.5 conducted a meta-analysis applying fixed-effects models and random-effects models on randomized controlled CDSS studies. Both concluded that there was low to moderate evidence for reduced morbidity as a pooled patient outcome and low evidence for reduced mortality. Moja et al.5 mentioned that high heterogeneity of outcome effects might depend on the disease entity. Therefore, a filtered list of disease entities, in which CDSS succeeded more frequently, would be helpful to identify promising candidate use cases for implementation in clinical practice.

While a meta-analysis is applied on a single common outcome or a pooled outcome such as morbidity, a qualitative approach would be capable of weighting different outcome effects according to clinical importance perceived by physicians.

The following 2 unanswered questions formed the rationale for this qualitative review.

Are there specific disease entities for which CDSSs succeeded more frequently?

Kawamoto et al.1 and Roshanov et al.6 identified contextual success factors of CDSS implementation to improve patient care. For instance, systems more likely to succeed provided advice for patients in addition to practitioners, or required practitioners to supply a reason for overriding CDSS advice.

While those factors should be considered for CDSS implementation in general, another factor that is not analyzed is the targeted disease entity.

The disease entity represents the medical condition or event for which the CDSS intervention is intended to support prevention, diagnosis, or treatment. It can be a crucial factor associated with success or failure of CDSS implementation, since different disease entities are associated with differences regarding patient characteristics, health providers’ backgrounds, clinical workflows in hospital routines, or any other disease-specific requirements of a CDSS.

This review aims to filter CDSS studies by their targeted disease and where they mainly reported on patient outcome improvements. Those disease entities represent clinical use cases where CDSS implementation could benefit patient care.

To what extent are the reported outcome effects clinically important?

Clinical importance varied among different patient outcomes that were evaluated. Patient outcomes of different clinical importance can be assessed (1) among different CDSS studies or (2) within the same CDSS study.

To provide an example of the first category: A CDSS that was applied in a study that showed a reduction in the incidence of postoperative nausea (outcome 1) may be of less clinical importance than a system that was applied in a study for the prophylaxis of venous thromboembolism (VTE) that showed a reduction in the incidence of pulmonary embolisms (outcome 2), one of the most preventable causes of in-hospital death.8

To provide an example for the second category: The CDSS for VTE prophylaxis is associated with several outcome effects. It could reduce the incidence of deep vein thrombosis (outcome 1) or pulmonary embolism (outcome 2). Again, outcome 2 is more closely related to mortality. A CDSS study that showed a significant improvement in both outcomes should be rated higher in terms of clinical importance than a study that only showed a reduction in deep vein thrombosis.

Of course, individual risk of bias assessment based on study design characteristics should be taken into account when comparing outcomes of different studies.

To our knowledge, different levels of clinical importance in CDSS study outcomes have not been assessed previously. This study aimed to close this gap by applying a predefined medical effect score to classify and summarize all outcome findings of each CDSS study.

We aimed to identify specific disease entities most impacted by CDSS and to assess the significance of their clinical impact by conducting a physician-driven review of recent CDSS studies that had a CDSS as the main study intervention, evaluated at least 1 patient outcome measurement, and included a control group that represented the usual care with no components of the CDSS involved. No further restrictions on study participants, interventions, comparisons, outcomes, or study design were applied.

Risk of bias assessment was performed to account for different study quality characteristics. The goal was to identify disease entities in which CDSS studies showed high medical effect and moderate risk of bias and therefore should be carefully considered by clinicians and IT experts for implementation at hospital sites.

MATERIALS AND METHODS

General requirements

The review was designed to meet the 27-item checklist “Preferred Reporting Items for Systematic Reviews and Meta-Analyses,”9 a tool to assess reporting quality. The a priori design of this study has been registered on the PROSPERO register for systematic reviews (no. CRD42016049946).10

Data sources and study selection

Definition of CDSS

To our knowledge, there is not a standard definition by which CDSS systems can be clearly distinguished from other similar health IT functionalities. Instead, some health IT systems share a certain degree of CDSS functionality. A definition by Kawamoto et al.,1 which is frequently used in the literature, will be used for this review:

“A CDSS is any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration.”

Search strategy

The search strategy was developed by an information specialist and is based on 2 recent systematic reviews2,7 on health information systems. The strategy was adapted to focus on terms relevant for CDSS and patient outcome assessments. Medline, Cochrane Trials, and Cochrane Reviews were searched for relevant studies. Studies suggested by expert opinion collected at Medical Informatics Conference 201511 and Medical Informatics Europe Conference 20161,2 were also considered. All studies had to be published between January 2005 and April 2016. The rationale of choosing that start date was to focus on recently developed CDSSs working in an environment with mature health information technology tools such as electronic medical records.

Articles referenced in all included articles and reviews were considered for further inclusion. Search terms applied on PubMed are provided in Figure 1.

Figure 1.

Figure 1.

Open in a new tab

Search terms used on PubMed.

Two search phases were required for the entire screening. The first started in July 2015 (articles screened were published between January 2005 and July 2015), and a second updated search started in April 2016 (articles screened were published between July 2015 and April 2016). Table 1 lists all study selection criteria, which were applied to every study within the search results.

Table 1.

Selection criteria for identifying CDSS studies

Inclusion Exclusion

  • CDSS fulfills definition and is used as the main study intervention component

  • CDSS is applied in an inpatient or intensive care setting

  • CDSS is used within an existing clinical routine workflow operating on real patient data

  • Full-text articles published between 2005 and April 2016

  • Study evaluates at least 1 patient outcome with existing control group

  • Control group has to represent the usual care with no components of the CDSS being involved

  • No English full text available

  • Study participants were test patients, or CDSS was used solely by medical students

  • Same CDSS for the same disease domain by the same site (only the latest study was considered)

  • Studies only evaluating guideline adherence or compliance without an evaluation of at least 1 patient outcome
Open in a new tab

Title and abstract screening was conducted by 1 reviewer (JV), full-text screening by 2 independent reviewers (JV, MK). Disagreements were resolved by consensus.

Conference proceedings were also considered if listed within the sources.

Data extraction and synthesis

Evaluating study quality and fostering future reimplementation

Details on study participants, interventions, outcome results, study design, and further study characteristics such as disease entity of interest, health care setting (eg, study country, single or multiple hospitals involved), patient sample sizes, sample size calculations, statistical methods for baseline analysis, and confounder adjustments were extracted independently by 2 reviewers (JV, MK) from every included study wherever reported. A data abstraction form was piloted and finalized to standardize all necessary data extraction items and is available in Supplementary Table S1. Principal study designs were: (1) randomized controlled, (2) nonrandomized interventional or pre/post with at least 1 prospective study arm, and (3) purely retrospective as retrospective cohorts or case-control studies. Randomization was distinguished by: (1) patient, (2) health care professional, and (3) site or ward levels. Based on the availability of reported characteristics, 2 biostatisticians (MK, SS) used a standardized approach recommended by the Agency for Healthcare Research and Quality,13 which was also used by a recent systematic CDSS review by Bright et al.3 to independently evaluate overall risk of bias (low, medium, high)13 for each study. This approach was applicable to all principal study designs.

Criteria for risk of bias assessment and justification comments are provided in Supplementary Table S2. Subsequently, the ratings of both biostatisticians were compared, and any differences were resolved by consensus.

To facilitate technical reimplementation and linkage with existing electronic medical record systems, we extracted necessary input variables of all CDSSs and established interoperable input data models according to Operational Data Model,14 a format supported by the Food and Drug Administration and the European Medicines Agency for the exchange of metadata in clinical trials. Those models are provided on our metadata repository,15 a registered European information infrastructure, and can be downloaded in various formats for reuse in different hospital information systems. This way, patient-related input variables that were used within the included CDSS studies and were crucial for information processing and the logics of the systems are defined in a harmonized and structured way and can be reused for future implementation. Additionally, the algorithm of each CDSS, if described in the original articles, was classified by a major inference technique16 (eg, rule-based system using if-then logic, Bayesian inference, or neural networks).

Medical effect score and generation of an example set of promising disease entities

Two physicians (JV, SIG) used the extracted data from the previous step and the original full-text articles to independently rate the overall medical effect of each study on a categorical point scale. For each study, they independently summarized all reported outcome findings with 1 medical effect score. The score represents the physician-perceived impact on the patient’s overall medical condition. Similar to the American Society of Anesthesiologists’ classification17 to rate patients’ preoperative health, the score both physicians used assessed the relatedness to life-threatening conditions or mortality as a key indicator. By definition, studies showing significantly reduced mortality received the highest rating. Table 2 shows the definitions of the categorical point scale that was used to assess the medical effect of every outcome event (eg, reduction of the incidence of pulmonary embolisms from 10% to 7%) in 1 study.

Table 2.

Medical effect scores represented by a categorical scale

5: Mortality reduction Mortality was significantly reduced in the CDSS intervention group.
4: Strong positive effect No effect on mortality was measured, but patient outcome events with immediate life-threatening potential (eg, adverse reactions or forms of morbidity) were reduced.
3: Medium positive effect Patient outcome events with no immediate life-threatening potential were reduced. Patients suffering from those events would require nonurgent treatment.
2: Light positive effect Patient outcome events with no immediate life-threatening potential were reduced. Patients suffering from those events would not necessarily require treatment.
1: No significant effect No significant effect on patient’s medical condition was measured. A potential benefit for the patient’s medical condition was unclear or not expected.
Negative effect Patient outcome event had a negative effect on the patient’s medical condition.
Open in a new tab

The definitions of the medical effect scores had been prespecified prior to the start of the medical assessment of each study and were not changed during or after the assessment.

All ratings except for 1 required statistical significance. If a study provided multiple positive or nonsignificant outcome events, the outcome event with the highest rating (indicating the most clinically important event) was considered. If the outcome events were contrary (at least 1 positive and at least 1 negative outcome event) within 1 study, a rating of 1 was given. If 1 outcome event was negative and all others were nonsignificant, a negative effect was summarized.

This scoring did not favor the type of outcome (primary vs secondary), though some studies may have been powered for the primary outcome. The rationale for this approach was to avoid neglecting potentially important secondary outcomes and to conduct our medical evaluation regardless of the CDSS study investigator’s choice of the most important outcome.

The scores that were given by the 2 physicians were then compared to measure interrater reliability based on weighted kappa statistics. Details of the calculations are provided in Supplementary Table S3. Score differences were then resolved by consensus.

Generation of an example set of promising disease entities should take into account the relative frequency of studies with a positive medical effect and moderate risk of bias within the disease entity. The following definitions were used: A disease entity was defined to be common if it was represented in at least 2 included CDSS studies. A common disease entity was defined to be promising if the majority (>50%) of CDSS studies within that disease entity had (1) at least 3 points or greater (medium positive effect or more) on the medical effect score and (2) medium or low risk of bias in statistical assessment.

RESULTS

Search selection

After removal of duplicates, 19 590 articles were screened, of which 3109 were rejected by titular review, 16 346 by abstract review, and 69 by full-text review. Four additional articles were identified from other reviews and expert opinions, resulting in a final set of 70 CDSS studies (Figure 2). The full list of included and excluded articles can be found via supplementary link S7.

Figure 2.

Figure 2.

Open in a new tab

CDSS study selection.

CDSS study characteristics

Nonrandomized studies with a pre/post analysis or cohort studies with prospective data collection as a principal study design were most common (47%, n = 33), followed by randomized controlled studies (28%, n = 20) and purely retrospective ones (25%, n = 18). An overall increase in published CDSS studies can be observed over time (2005: 4 studies; 2015: 11 studies), which is mainly associated with the increase in nonrandomized prospective studies; see Supplementary Figure S4 for details. An overview of the different study designs with risk of bias assessment is provided in Supplementary Figure S5. Fifty-one studies (73%) implemented CDSSs in single hospitals, while the rest managed to run CDSSs in multiple hospitals. Regarding the inference types of all CDSSs, most of them were rule-based systems18 (72%), followed by Bayesian inference networks19 (6%). The rest of the systems (22%) did not provide any description of their algorithms. Full details on required study designs, patient and disease characteristics, study results, and risk of bias assessments are provided in Supplementary Table S1.

Electronic input data models of each CDSS are provided on our metadata repository for technical reimplementation in various data formats.20Figure 3 exemplifies an extract of a standardized form that represents the input of the CDSS in the study by Mitchell et al.,21 which used a comprehensive set of input variables to determine indication of VTE prophylaxis.

Figure 3.

Figure 3.

Open in a new tab

Extract of a standardized electronic input form that lists contraindications of VTE prophylaxis. Detailed information, such as variable definitions and semantic codes, is available in different formats for reuse in different medical information systems.

Medical effect scores and promising disease entities

There was substantial22 interrater reliability (weighted κ: 0.72, P < .0001) between the 2 physicians who independently assigned medical effect scores; see Figure 4 for differences of specific rating values.

Figure 4.

Figure 4.

Open in a new tab

Differences of given medical effects scores between the 2 physicians (overall weighted κ: 0.72).

Of 70 included studies, 25 (36%) reported on mortality. Five were associated with significant mortality reduction. Sixteen studies (23%) showed a strong positive effect by reducing life-threatening events. Twenty (29%) showed a medium positive effect by reducing non–life-threatening events that required medical treatment. Eight studies (11%) showed a light positive effect by reducing non–life-threatening events that did not necessarily require medical treatment. Twenty studies (29%) showed no or nonsignificant effect on overall medical condition. One study implemented a blood glucose management system, which had a negative impact as a result of increased hypoglycemic events.

Figure 5 provides an overview of all included studies and the proportions of medical effect score values and risk of bias assessments.

Figure 5.

Figure 5.

Open in a new tab

Overview of (A) medical effect scores and (B) proportion of risk of bias for all 70 CDSS studies; eg, 5 studies were evaluated with a score of 5 (mortality reduction), and 3 of those 5 studies (60%) had a low risk of bias.

Twenty-four disease entities were identified. Twelve of them were common disease entities (at least n = 2 studies reporting). Figure 6 breaks down Figure 5 to show results (medical effect scores and risk of bias) specifically for the 12 common disease entities.

Figure 6.

Figure 6.

Open in a new tab

Specifies the results from Figure 5 for the common disease entities (≥2 studies); eg, 15 studies dealt with venous thromboembolism prophylaxis, and 8 showed a strong positive medical effect, of which approximately one-third (3/8) were rated as low risk of bias. Disease entities with a bold frame were identified as promising according to previous definition.

Based on medical effect and risk of bias assessment, 6 disease entities were identified as promising: (1) blood glucose management (n = 14 studies), (2) blood transfusion management (n = 5), (3) physiologic deterioration prevention/physiologic surveillance (n = 4), (4) pressure ulcer prevention (n = 2), (5) acute kidney injury prevention (n = 2), and (6) VTE prophylaxis (n = 15). For the 6 promising disease entities, Table 3 lists results of medical effect scores with justification details. The full list of all study characteristics (including participants, interventions, comparisons, outcomes details) and results of medical effect score evaluation, risk of bias assessment, and disease entity assignment is available in Supplementary Table S1.

Table 3.

List of promising disease entities

Clinical scope, disease entity Medical effect score Risk of bias Study reference Study design Study duration Patient sample size Medical score consensus justification
Blood glucose management (n = 14) 3 Low 23 2 26 months 891

  • Negative: increase of hypoglycemic events due to insulin overdose.

  • 1: No significant patient outcome effect. Mean glucose values might have been changed, but patient proportion within glucose target range barely changed.

  • 3: Increase of patient proportion within blood glucose target range.

  • 4: Reduction of severe hypoglycemia (<40 mg/dl) or severe hyperglycemia (>200 mg/dl).
1 Low 24 1 72 hours 20
3 High 25 1 9 hours 40
3 Medium 26 3 1.5 years 438
3 Low 27 1 1 month 128
3 Low 28 1 72 hours 18
4 Medium 29 2 1 year 1373
3 Medium 30 2 11 months 667
3 Low 31 1 9 months 300
1 Low 32 2 7 years 2040
3 Low 33 2 1.5 years 197
Neg. Low 34 1 2 years 2648
1 Medium 35 2 12 months 3189
4 Low 36 2 4 years 22 990
Blood transfusion management (n = 5) 1 High 37 3 2 years 2200

  • 1: No significant patient outcome effect.

  • 2: Slight but significant reduction of sickle hemoglobin percentage.

  • 3: Reduction of surgical wound infections.

  • 5: Hospital mortality or 30-day mortality was reduced.
2 High 38 3 9 years 28
5 Low 39 2 6 years 14 150
3 Medium 40 2 2 years 1509
5 Low 41 3 2 years 12 590
Physiologic deterioration prevention, physiologic surveillance (n = 4) 3 High 42 2 Unknown 74

  • 3: Increase of patient time being in MAP target range. But average MAP value increase was low to moderate.42

  • Reduction of critical events during spinal analgesia in orthopedic surgery.

  • However, exposure to critical events was short term.43

  • 5: Hospital mortality was reduced.
3 Medium 43 1 2 years 150
5 Medium 44 2 3 months 12 881
5 Medium 45 3 2 years 50 000
Pressure ulcer prevention (n = 2) 3 Low 46 2 1 year 1214 3: Incidence of pressure ulcers was reduced.
3 Low 47 2 2 years 18 483
Nephrotoxic agents and acute kidney injury prevention (n = 2) 4 Medium 48 2 1 year 463 4: Reduction in the rate of contrast-induced acute kidney injury48 or preventable adverse events for patient with renal impairment.49
4 Medium 49 2 5.5 years 1590
VTE prophylaxis (n = 15) 4 Low 50 1 4 years 2506

  • 1: No significant patient outcome effect.

  • 3: Reduction of VTE as deep vein thrombosis. Adverse reactions as hemorrhages were not significantly increased.

  • 4: Same as 3. Additionally, the number of pulmonary embolisms was reduced.
4 Low 51 2 2 years 12 000
1 Medium 52 1 90 days 2506
4 Medium 53 2 2 years 38 647
1 Medium 54 2 1 year 800
3 Medium 55 2 3 years 3285
3 Medium 56 3 3 years 1599
4 Medium 57 3 3 years 223 062
4 Low 21 2 2 years 5238
4 Medium 58 3 2 months 1942
1 Medium 59 1 13 weeks 15 736
1 Medium 60 3 2 years 7278
4 Medium 61 2 5 years 45 046
4 Medium 62 2 1 year 812
3 Medium 63 3 10 years 2591
Open in a new tab

Refer to Supplementary Table S1 for full details of study design and study outcome characteristics.

Study design: 1 = randomized controlled trial, 2 = nonrandomized with 1 prospective study arm, 3 = nonrandomized, purely retrospective.

MEC = medical effect score; VTE = venous thromboembolism; MAP = mean arterial pressure.

DISCUSSION

The increase in CDSS studies noted in this review confirms the findings by Brenner et al.2 that the overall number of CDSS studies is increasing per year, while the number of randomized controlled trials (RCTs) remains consistently low.

Though RCTs represent the gold standard as a principal study design, this review found evidence that some RCTs had even higher risk of bias than nonrandomized studies as a result of small sample sizes, contamination effects, or the lack of adjustment for confounders. Future RCTs should address those issues.

The exemplary set of promising disease entities was identified based on the procedures, definitions, and criteria established for this review. More or less restrictive thresholds could be adapted to change the set of studies analyzed and the promising disease entities identified.

A crucially important aspect of CDSS is under what circumstances it requires regulatory approval as a medical device. Though all studies received study approval, only 3 of them reported medical device approval. For dissemination of CDSS from a study context to daily routine practice, approval as a medical device is inevitable for systems that directly affect clinical processes or patient outcomes.64,65

Since the majority of studies (73%) report on single hospital implementations, further studies should consider CDSSs within multicenter hospitals, to account for both organizational site-specific factors and technical diversities regarding hospital information systems.

Most of the implemented systems were rule-based and applied simple if-then logic. This underlines the potential ease of technical reimplementation and traceability.

CDSSs are sociotechnical systems. Ascertaining sociotechnical CDSS success factors was not the scope of our analysis. Information on those factors was rarely reported in the original study articles. Rhoshanov et al.6 sought to identify key sociotechnical success factors based on meta-regression of RCTs and summarized 3 factors that could increase the odds of success: (1) systems that require practitioners to provide reasons when overriding advice, (2) systems that provide advice concurrently to patients and practitioners, and (3) systems developed by the authors. However, those results are not supported by our observations. For the 6 promising disease entities, we could not identify a disease-independent common rule for why some systems succeeded and some did not. Our review was restricted to studies that evaluated at least 1 patient outcome in inpatient settings, while Rhoshanov et al.6 also included studies that did not evaluate patient outcomes, but processes of care in outpatient settings. It is known that most of the CDSS studies evaluate processes of care but not patient outcomes.7 Thus, the aforementioned factors may rather adhere to benefits for clinical processes and/or outpatient settings.

Based on the 2 disease entity settings with the most published CDSS studies (VTE prophylaxis and blood glucose management), we would like to conclude with 2 key sociotechnical characteristics that we believe are beneficial regarding the clinical impact of CDSS.

  • 1. Existence of a comprehensive and reasonable disease-related knowledge base

Some disease entities are associated with medical events where risk prediction is easily inferable by a set of well-known evidence-based patient risk variables and formalized rule sets or algorithms. Ideally, these variables are accessible from the patient’s medical record at the point of care with sufficient data quality. Disease entities or medical events for which knowledge of risk prediction is not available or is hard to formalize for machine readability pose a challenge. In addition, this knowledge base (input variables and inferences) should output alerts or recommendations of high specificity to reduce “alert fatigue” and should be customizable according to local requirements or new knowledge input.66

  • 2. Integration with all relevant hospital stakeholders

Although it is known that CDSSs can foster guideline adherence,3 some hospital sites may already have high adherence to officially recommended guidelines. Therefore, implementation should be carefully considered among local clinicians, quality management experts, and IT staff. When planning to implement a CDSS, system testing and validation67 should be considered by IT staff and all system users. Extensive risk analysis as to what could potentially go wrong with the system or what would be the worst case for patients should be elaborated and concluded with appropriate risk-mitigating measures. For instance, an insulin dosing system that could potentially suggest overdosing could be coupled with a hypoglycemia bundle to prevent potentially life-threatening hypoglycemic events.36

Before running the CDSS in the real-world setting, on-site promotion, education, and training of system users could raise the awareness of the system and the disease entity.68

The full list of noticeable observations of sociotechnical characteristics of CDSSs, with an aim to explain why some CDSSs may have succeeded or failed within this review, is provided under S6 in the supplement and should be taken into account when reimplementing CDSSs at other hospital sites.

Strengths and limitations

As with all reviews, selective outcome reporting or publication bias is a given, since positive outcome effects are more likely to be published than nonsignificant or negative ones.69 Additionally, we relied on the publication standards of the journals to rule out studies with financial dependencies between authors and vendors of CDSSs that were assessed.

To analyze a comprehensive set of CDSS studies that were recently published (since 2005), we included studies with heterogeneous study design and could not focus only on high-quality RCTs. The benefit of this search strategy was a larger sample size of published CDSS studies, and therefore the ability to remain focused on recently developed CDSSs working in the context of modern electronic health care systems. Thus, older studies (eg, from the 1990s) with CDSSs having limited access to structured electronic medical records were excluded. The downside is limited comparability of study outcomes resulting from different study designs. Similar to Brenner et al.,2 a standardized assessment of individual study characteristics was taken into account. To our knowledge, this is the first review with an assessment that assigned individual patient outcome to a medical effect score regarding the patient’s overall medical condition. This complements existing meta-analyses of CDSS studies that are limited to single outcome analyses of randomized controlled studies.

The medical effect score enabled clinical weighting of different outcome effects of different studies. The strength of this scoring is that it allows clinical comparisons of intervention systems across different disease entities, since different outcome events were categorized according to a predefined concept of clinical importance. One limitation of the score is that in case of multiple outcomes, per-study differences in outcome type (primary vs secondary outcome) were not taken into account. However, only 41 of 70 included studies (59%) defined a primary outcome, and only 28 studies (40%) had patient outcome as the primary outcome.

For those 28 studies, one could have chosen only the primary outcome, especially when the outcomes were mixed. However, the primary outcome is the measure that a study investigator considers to be most relevant to the study and might not represent the most important outcome of clinical significance. It is problematic if only the primary outcome is chosen and secondary outcomes, which could indicate harmful events, are neglected. As an example, Kalfon et al.34 studied a CDSS that had no apparent effect on mortality (primary outcome), but led to significant increases in severe hypoglycemia (secondary outcome). In this case, our scoring methodology evaluated a negative outcome summary, which is more cautious. As a counterexample, in studies with positive and nonsignificant outcome events, our scoring would choose the outcome event with the highest rating (indicating the clinically most important event). This could overrate studies that only had a positive secondary outcome but a less positive or neutral primary outcome. There was only one study40 with this characteristic, which would have received a rating of 2 instead of our rating of 3.

Studies with contrary outcomes (positive and negative effects) were not found in this review.

Due to the lack of standardized medical effect scores, we could not reuse validated scores or instruments to weight different medical outcome effects. Weighted κ statistics (>0.7) showed high agreement between 2 independent physicians, which underlines the reproducibility of our results if different physicians were to use the score.

Outcome measurements such as quality of life and mental health contribute substantially to a patient’s overall health, but these were not reported in the included CDSS studies.

CONCLUSION

Though most (70%) of the included CDSS studies reported positive patient outcome effects, there are substantial differences regarding the clinical importance of improved patient outcomes. Among different clinical use cases of CDSS implementation, a small exemplary subset of disease entities could be identified in which CDSSs mostly improved patient outcomes by preventing significantly harmful events. Those disease entities should be given special consideration at sites where computer-assisted decision-making is underutilized.

Declarations

Competing interests

The authors declare that they have no competing interests.

Funding

This work is funded by the German Ministry for Education and Research, grant ID: 01ZZ1602B.

Authors’ contributions

All authors made significant contributions to the manuscript. JV developed the design of the systematic review and was involved in the data screening and extraction with MK, conducted the medical evaluation of the included studies, and wrote the manuscript. SIG was involved in the medical evaluation of the included studies. MK and SS conducted the risk of bias analysis. MD supervised and guided the project. All authors provided critical revision and approved the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Supplementary Material

Supplementary Data
Click here for additional data file. (383.5KB, zip)

References

  • 1. Kawamoto K, Houlihan C, Balas E, Lobach D. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Brenner S, Kaushal R, Grinspan Z et al. , Effects of health information technology on patient outcomes: a systematic review. J Am Med Inform Assoc. 2016;235:1016–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Bright T, Wong A, Dhurjati R et al. , Effect of clinical decision-support systems. Ann Intern Med. 2012;1571:29–43. [DOI] [PubMed] [Google Scholar]
  • 4. Ioannidis J, Patsopoulos N, Rothstein H. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008;3377658:1413–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Moja L, Kwag K, Lytras TBL et al. , Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health. 2014;10412:e12–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Roshanov P, Fernandes N, Wilczynski J et al. , Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 2013;346:f657. [DOI] [PubMed] [Google Scholar]
  • 7. Jones S, Rudin R, Perry T, Shekelle P. Health information technology: an updated systematic review with a focus on meaningful use. Ann Int Med. 2014;1601:48–54. [DOI] [PubMed] [Google Scholar]
  • 8. Lindblad B, Sternby N, Bergqvist D. Incidence of venous thromboembolism verified by necropsy over 30 years. BMJ. 1991;3026778:709–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Moher D, Liberati A, Tetzlaff J, Altman D. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Ann Int Med. 2009;1514:264–69. [DOI] [PubMed] [Google Scholar]
  • 10. Varghese J, Kleine M, Gessner S, Sandmann S, Dugas M. Effect of clinical decision support systems on patient outcomes in inpatient settings: a systematic review. PROSPERO 2016:CRD42016049946. www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42016049946. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Sarkar I, Georgiou A, de Azevedo Marques P. MEDINFO 2015: eHealth-enabled Health. IOS Press: Studies in Health Technology and Informatics; 2015. ISBN: 978-1-61499-563-0 (print). [Google Scholar]
  • 12. Hoerbst A, Hackl W, de Keizer N, Prokosch H, Hercigonja-Szekeres M, de Lusignan S. Exploring Complexity in Health: An Interdisciplinary Systems Approach. IOS Press: Studies in Health Technology and Informatics; 2016:978-1-61499-677-4. [Google Scholar]
  • 13. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2011. [PubMed] [Google Scholar]
  • 14. Clinical Data Interchange Standards Consortium. Operational Data Model. www.cdisc.org/odm. Accessed October 2016. [Google Scholar]
  • 15. Dugas M, Neuhaus P, Meidt A et al. , Portal of medical data models: information infrastructure for medical research and healthcare. Database. 2016;2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Yu X, Kacprzyk J. Applied Decision Support with Soft Computing. Berlin, London: Springer; 2003. [Google Scholar]
  • 17. Mohamed D. American Society of Anesthesiologists physical status classification. Indian J Anaesthesiol. 2011;552:111–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Oniésko A, Lucas P, Druzdzel M. Comparison of rule-based and Bayesian network approaches in medical diagnostic systems. Artificial Intell Med. 2001;2101:283–92. [Google Scholar]
  • 19. DJV M. Probable networks and plausible predictions – a review of practical Bayesian methods for supervised neural networks. Network: Comput Neural Syst. 2009;63:469–505. [Google Scholar]
  • 20. Institute of Medical Informatics. Medical Data Models for CDSS Input. https://medical-data-models.org/welcome/search?title=%22Input+Data+Elements+CDSS%22. Accessed October 2016. [Google Scholar]
  • 21. Mitchell JD, Collen JF, Petteys S, Holley AB. A simple reminder system improves venous thromboembolism prophylaxis rates and reduces thrombotic events for hospitalized patients. J Thrombosis Haemostasis. 2012;102:236–43. [DOI] [PubMed] [Google Scholar]
  • 22. McHugh M. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;223:276–82. [PMC free article] [PubMed] [Google Scholar]
  • 23. Thomas AN, Marchant AE, Ogden MC, Collin S. Implementation of a tight glycaemic control protocol using a web-based insulin dose calculator. Anaesthesia. 2005;6011:1093–100. [DOI] [PubMed] [Google Scholar]
  • 24. Cordingley JJ, Vlasselaers D, Dormand NC et al. , Intensive insulin therapy: enhanced Model Predictive Control algorithm versus standard care. Intensive Care Med. 2009;351:123–28. [DOI] [PubMed] [Google Scholar]
  • 25. Saager L, Collins GL, Burnside B et al. , A randomized study in diabetic patients undergoing cardiac surgery comparing computer-guided glucose management with a standard sliding scale protocol. J Cardiothoracic Vasc Anesthesia. 2008;223:377–382. [DOI] [PubMed] [Google Scholar]
  • 26. Guerra YS, Das K, Antonopoulos P et al. , Computerized physician order entry–based hyperglycemia inpatient protocol and glycemic outcomes: the CPOE-HIP study. Endocrine Practice. 2010;163:389–97. [DOI] [PubMed] [Google Scholar]
  • 27. Wexler DJ, Shrader P, Burns SM, Cagliero E. Effectiveness of a computerized insulin order template in general medical inpatients with type 2 diabetes: a cluster randomized trial. Diabetes Care. 2010;3310:2181–83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Mann EA, Jones JA, Wolf SE, Wade CE. Computer decision support software safely improves glycemic control in the burn intensive care unit: a randomized controlled clinical study. J Burn Care Res. 2011;322:246–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Meyfroidt G, Wouters P, de Becker W, Cottem D, van den Berghe G. Impact of a computer-generated alert system on the quality of tight glycemic control. Intensive Care Med. 2011;377:1151–57. [DOI] [PubMed] [Google Scholar]
  • 30. Lipton JA, Barendse RJ, Schinkel AFL, Akkerhuis KM, Simoons ML, Sijbrands EJG. Impact of an alerting clinical decision support system for glucose control on protocol compliance and glycemic control in the intensive cardiac care unit. Diabetes Technol Therapeutics. 2011;133:343–49. [DOI] [PubMed] [Google Scholar]
  • 31. Dumont C, Bourguignon C. Effect of a computerized insulin dose calculator on the process of glycemic control. Am J Critical Care. 2012;212:106–15. [DOI] [PubMed] [Google Scholar]
  • 32. Maat B, Rademaker CMA, Oostveen MI, Krediet TG, Egberts TCG, Bollen CW. The effect of a computerized prescribing and calculating system on hypo- and hyperglycemias and on prescribing time efficiency in neonatal intensive care patients. J Parenteral Enteral Nutr. 2013;371:85–91. [DOI] [PubMed] [Google Scholar]
  • 33. Saur NM, Kongable GL, Holewinski S, O'Brien K, Nasraway JSA. Software-guided insulin dosing: tight glycemic control and decreased glycemic derangements in critically ill patients. Mayo Clinic Proc. 2013;889:920–29. [DOI] [PubMed] [Google Scholar]
  • 34. Kalfon P, Giraudeau B, Ichai C et al. , Tight computerized versus conventional glucose control in the ICU: a randomized controlled trial. Intensive Care Med. 2014;402:171–81. [DOI] [PubMed] [Google Scholar]
  • 35. Nair BG, Grunzweig K, Peterson GN et al. , Intraoperative blood glucose management: impact of a real-time decision support system on adherence to institutional protocol. J Clin Monitoring Comput. 2016;303:301–12. [DOI] [PubMed] [Google Scholar]
  • 36. Maynard G, Kulasa K, Ramos P et al. , Impact of a hypoglycemia reduction bundle and a systems approach to inpatient glycemic management. Endocrine Pract. 2015;214:355–67. [DOI] [PubMed] [Google Scholar]
  • 37. Fernandez Perez ER, Winters JL, Gajic O. The addition of decision support into computerized physician order entry reduces red blood cell transfusion resource utilization in the intensive care unit. Am J Hematol. 2007;827:631–33. [DOI] [PubMed] [Google Scholar]
  • 38. McCrory MC, Strouse JJ, Takemoto CM, Easley RB. Computerized physician order entry improves compliance with a manual exchange transfusion protocol in the pediatric intensive care unit. J Pediatric Hematol Oncol. 2014;362:143–47. [DOI] [PubMed] [Google Scholar]
  • 39. Goodnough LT, Maggio P, Hadhazy E et al. , Restrictive blood transfusion practices are associated with improved patient outcomes. Transfusion. 2014;54(10 Pt 2):2753–59. [DOI] [PubMed] [Google Scholar]
  • 40. Razavi SA, Carter AB, Puskas JD, Gregg SR, Aziz IF, Buchman TG. Reduced red blood cell transfusion in cardiothoracic surgery after implementation of a novel clinical decision support tool. J Am College Surgeons. 2014;2195:1028–36. [DOI] [PubMed] [Google Scholar]
  • 41. Loftus TJ, Spratling L, Stone BA, Xiao L, Jacofsky DJ. A patient blood management program in prosthetic joint arthroplasty decreases blood use and improves outcomes. J Arthroplasty. 2016;311:11–14. [DOI] [PubMed] [Google Scholar]
  • 42. Giuliano KK, Jahrsdoerfer M, Case J, Drew T, Raber G. The role of clinical decision support tools to reduce blood pressure variability in critically ill patients receiving vasopressor support. Comput Inform Nursing. 2012;304:204–09. [DOI] [PubMed] [Google Scholar]
  • 43. Zaouter C, Wehbe M, Cyr S et al. , Use of a decision support system improves the management of hemodynamic and respiratory events in orthopedic patients under propofol sedation and spinal analgesia: a randomized trial. J Clin Monitoring Comput. 2014;281:41–47. [DOI] [PubMed] [Google Scholar]
  • 44. Evans RS, Kuttler KG, Simpson KJ et al. , Automated detection of physiologic deterioration in hospitalized patients. J Am Med Inform Assoc. 2015;222:350–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Schmidt PE, Meredith P, Prytherch DR et al. , Impact of introducing an electronic physiological surveillance system on hospital mortality. BMJ Qual Saf. 2015;241:10–20 [DOI] [PubMed] [Google Scholar]
  • 46. Cho I, Park I, Kim E, Lee E, Bates DW. Using EHR data to predict hospital-acquired pressure ulcers: a prospective study of a Bayesian Network model. Int J Med Inform. 2013;8211:1059–67. [DOI] [PubMed] [Google Scholar]
  • 47. Sebastian-Viana T, Losa-Iglesias M, Gonzalez-Ruiz JM, Lema-Lorenzo I, Nunez-Crespo FJ, Salvadores Fuentes P. Reduction in the incidence of pressure ulcers upon implementation of a reminder system for health-care providers. Appl Nursing Res. 2016;29:107–12. [DOI] [PubMed] [Google Scholar]
  • 48. Cho A, Lee JE, Yoon JY et al. , Effect of an electronic alert on risk of contrast-induced acute kidney injury in hospitalized patients undergoing computed tomography. Am J Kidney Dis. 2012;601:74–81. [DOI] [PubMed] [Google Scholar]
  • 49. Leung AA, Schiff G, Keohane C et al. , Impact of vendor computerized physician order entry on patients with renal impairment in community hospitals. J Hosp Med. 2013;810:545–52. [DOI] [PubMed] [Google Scholar]
  • 50. Kucher N, Koo S, Quiroz R et al. , Electronic alerts to prevent venous thromboembolism among hospitalized patients. New Engl J Med. 2005;35210:969–77. [DOI] [PubMed] [Google Scholar]
  • 51. Lecumberri R, Marques M, Diaz-Navarlaz MT et al. , Maintained effectiveness of an electronic alert system to prevent venous thromboembolism among hospitalized patients. Thromb Haemostasis. 2008;1004:699–704. [DOI] [PubMed] [Google Scholar]
  • 52. Piazza G, Goldhaber SZ. Computerized decision support for the cardiovascular clinician: applications for venous thromboembolism prevention and beyond. Circulation. 2009;12012:1133–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53. Galanter WL, Thambi M, Rosencranz H et al. , Effects of clinical decision support on venous thromboembolism risk assessment, prophylaxis, and prevention at a university teaching hospital. Am J Health-Sys Pharm. 2010;6715:1265–73. [DOI] [PubMed] [Google Scholar]
  • 54. Novis SJ, Havelka GE, Ostrowski D et al. , Prevention of thromboembolic events in surgical patients through the creation and implementation of a computerized risk assessment program. J Vasc Surgery. 2010;513:648–54. [DOI] [PubMed] [Google Scholar]
  • 55. Maynard GA, Morris TA, Jenkins IH et al. , Optimizing prevention of hospital-acquired venous thromboembolism (VTE): prospective validation of a VTE risk assessment model. J Hosp Med. 2010;51:10–18. [DOI] [PubMed] [Google Scholar]
  • 56. Haut ER, Lau BD, Kraenzlin FS et al. , Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg. 2012;14710:901–07. [DOI] [PubMed] [Google Scholar]
  • 57. Umscheid CA, Hanish A, Chittams J, Weiner MG, Hecht TEH. Effectiveness of a novel and scalable clinical decision support intervention to improve venous thromboembolism prophylaxis: a quasi-experimental study. BMC Med Inform Decis Making. 2012;12:92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Zeidan AM, Streiff MB, Lau BD et al. , Impact of a venous thromboembolism prophylaxis smart order set: improved compliance, fewer events. Am J Hematol. 2013;887:545–49. [DOI] [PubMed] [Google Scholar]
  • 59. Beeler PE, Eschmann E, Schumacher A, Studt JD, Amann-Vesti B, Blaser J. Impact of electronic reminders on venous thromboprophylaxis after admissions and transfers. J Am Med Inform Assoc. 2014;21(e2):e297–303. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60. Lin SW, Kang WY, Lin DT et al. , Comparison of warfarin therapy clinical outcomes following implementation of an automated mobile phone–based critical laboratory value text alert system. BMC Med Genomics. 2014;7(Suppl 1):S13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61. Amland RC, Dean BB, Yu H et al. , Computerized clinical decision support to prevent venous thromboembolism among hospitalized patients: proximal outcomes from a multiyear quality improvement project. J Healthcare Qual. 2015;374:221–31. [DOI] [PubMed] [Google Scholar]
  • 62. Nazarenko GI, Kleymenova EB, Payushik SA, Otdelenov VA, Sychev DA, Yashina LP. Decision support systems in clinical practice: the case of venous thromboembolism prevention. Int J Risk Safety Med. 2015;27(Suppl 1):S104–05. [DOI] [PubMed] [Google Scholar]
  • 63. Woller SC, Stevens SM, Towner S et al. , Computerized clinical decision support improves warfarin management and decreases recurrent venous thromboembolism. Clin Appl Thrombosis/Hemostasis. 2015;213:197–203. [DOI] [PubMed] [Google Scholar]
  • 64. European Commission. Guidelines on the Qualification and Classification of Stand alone Software used in Healthcare within the Regulatory. MEDDEV 2.1/6 2012. [Google Scholar]
  • 65. Karnik K. FDA regulation of clinical decision support software. J Law Biosci. 2014;12:202–08. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66. Sittig D, Wright A, Osheroff J et al. , Grand challenges in clinical decision support. J Biomed Inform. 2008;412:387–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67. Al-Hyari A, Al-Taee A, Al-Taee M. Clinical decision support system for diagnosis and management of chronic renal failure. 2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT). 2013. [Google Scholar]
  • 68. Ballard DW, Vemula R, Chettipally UK et al. , Optimizing clinical decision support in the electronic health record. Clinical characteristics associated with the use of a decision tool for disposition of ed patients with pulmonary embolism. Appl Clin Inform 2016;73:883–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69. Hopewell S, Loudon K, Clarke M. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;211:MR000006. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Data
Click here for additional data file. (383.5KB, zip)

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES