Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Jun 1.
Published in final edited form as: Anesthesiology. 2014 Jun;120(6):1339–1353. doi: 10.1097/ALN.0000000000000251

Effect of a Cognitive Aid on Adherence to Perioperative Assessment and Management Guidelines for the Cardiac Evaluation of Non-Cardiac Surgical Patients

William R Hand 1,✉, Kathryn H Bridges 2, Marjorie P Stiegler 3, Randall M Schell 4, Amy N DiLorenzo 5, Jesse M Ehrenfeld 6, Paul J Nietert 7, Matthew D McEvoy 8
PMCID: PMC4108481  NIHMSID: NIHMS592198  PMID: 24705442

Abstract

BACKGROUND

The 2007 American College of Cardiologists/American Heart Association Guidelines on Perioperative Cardiac Evaluation and Care for Noncardiac Surgery is the standard for perioperative cardiac evaluation. Recent work has shown residents and anesthesiologists do not apply these guidelines when tested. This research hypothesized that a decision support tool would improve adherence to this consensus guideline.

METHODS

Anesthesiology residents at 4 training programs participated in an unblinded prospective randomized cross-over trial in which they completed two tests covering clinical scenarios. One quiz was completed from memory and one with the aid of an electronic decision support tool. Performance was evaluated by overall score (% correct), number of incorrect answers with possibly increased cost or risk of care, and the amount of time required to complete the quizzes both with and without the cognitive aid. The primary outcome was the proportion of correct responses attributable to the use of the decision support tool.

RESULTS

All anesthesiology residents at four institutions were recruited and 111 residents participated. Use of the decision support tool resulted in a 25% improvement in adherence to guidelines compared to memory alone (p<0.0001), and participants made 77% fewer incorrect responses that would have resulted in increased costs. Use of the tool was associated with a 3.4-minute increase in time to complete the test (p<0.001).

CONCLUSIONS

Use of an electronic decision support tool significantly improved adherence to the guidelines as compared to memory alone. The decision support tool also prevented inappropriate management steps possibly associated with increased healthcare costs.

Introduction

Postoperative cardiac complications are a major source of morbidity, mortality, and cost in the perioperative period.1 To preserve patient safety and reduce unnecessary testing, the American College of Cardiology (ACC) and the American Heart Association (AHA) have published consensus guidelines for preoperative cardiac evaluation and management of non-cardiac surgical patients.2 The main purpose of these guidelines is to aid clinicians in performing risk stratification and appropriate cardiac evaluation of patients having intermediate to high-risk non-cardiac surgery.

Appropriately applied, these guidelines are intended to accomplish several goals. First, the guidelines identify patients needing additional preoperative assessment via diagnostic tests and imaging techniques (e.g. stress test or echocardiogram). Second, the guidelines identify patients who may benefit from the institution or continuance of preoperative pharmacologic management, such as beta-blockers for targeted perioperative heart rate control.3 Third, the guidelines may facilitate the informed consent process, helping physicians quantify perioperative risk. Taken together, these guidelines are intended to improve patient safety and use of resources.

While these guidelines have been accepted as standard practice parameters for preoperative cardiac evaluation by the American Society of Anesthesiologists, recent studies have demonstrated that anesthesiologists and anesthesiology residents often fail to follow these guidelines when assessed with multiple-choice questions (MCQ).4,5 Reasons for poor adherence to established guidelines appear to be multifactorial, with several general categories of barriers having been identified.6 Barriers include inadequate physician knowledge (lack of familiarity), physician attitudes about the guidelines (lack of efficacy, outcome expectancy, agreement, or motivation due to previous practice), and behavior (communication issues between patient and practitioner, characteristics of each guideline, and environmental factors effecting the marginal effort to follow guidelines).6 The authors of the two previous studies in this specific area recommended evaluation of decision support tools (DSTs) as a future direction to improve adherence to published guidelines.4,5

DSTs often improve adherence to published guidelines and in many instances improve clinical outcomes, although this has not been universally true.7–25 However, most research on clinical decision support has focused upon patient management after initial assessment and diagnosis or through mandatory alerts for providers, such as reminders for intraoperative antibiotic prophylaxis, postoperative nausea and vomiting prophylaxis, or proper use of alarms for separate from cardiopulmonary bypass.26–34 In contrast, the use of DSTs for patient assessment/diagnosis and the subsequent application of evidence-based protocols has yet to be rigorously tested.26 Furthermore, we are unaware of any studies that have investigated the effect of DSTs on adherence to guidelines for the preoperative assessment and planning for non-cardiac surgical patients. Accordingly, we tested the hypothesis that an electronic DST would improve assessment of patient status and subsequent adherence to the ACC/AHA 2007 guidelines on perioperative cardiovascular evaluation and care for non-cardiac surgery among anesthesiology residents as compared to memory alone.

Materials and Methods

Four anesthesiology training programs participated in this prospective randomized trial with crossover design that was completed between February and May 2013. After investigational review board approval was granted at all institutions (Charleston, SC, Nashville, TN, Lexington, KY, Chapel Hill, NC), all residents (total 211) were recruited via electronic communication, with 111 (52.6%) self-selecting to consent to participate (see Figure 1). As no baseline performance measure was known for the study population, an a priori power analysis was not performed. The recruited sample size (n=111) provided 80% power to detect very subtle increases (~0.3 standard deviation units) in the primary outcome, the proportion of correct responses attributable to the use of the cognitive aid, assuming 2-sided hypothesis testing and an alpha level of 0.05. Secondary outcomes, described in the last paragraph of the methods section included the performance of test-takers related to the types of errors and efficiency with which they completed the tests. Multi-site compliance and logistics were regulated in the Department of Anesthesia and Perioperative Medicine at the Medical University of South Carolina (MUSC). Randomization was done using computer generated number generation and was assigned by MUSC as described below. The programs that participated in the study were MUSC (Charleston, SC), Vanderbilt University Medical Center (VUMC, Nashville, TN), University of North Carolina (UNC, Chapel Hill, NC), and University of Kentucky (UK, Lexington, KY).

Figure 1.

Figure 1

Open in a new tab

Group Assignments

Groups A-D were randomly assigned to complete either Quiz 1 or 2 first with or without the decision support tool. Participants in each group subsequently completed the remaining quiz with or without the decision support tool to ensure both quizzes and both testing conditions were evaluable. *One resident from Group C was paged for an emergency during testing. His incomplete results were removed from analysis. DST = decision support tool

Demographic information was collected on participants including year in training, gender, and race. Participants were then randomly assigned to groups A-D by a statistician with no personal experience with any of the participants. Group assignment A-D determined the order of quizzes and the availability of the DST for each quiz (see Figure 1). Differences between groups’ demographics were assessed via chi-square tests.

The DST containing the AHA/ACC evaluation and management guidelines was presented in electronic form on either an Apple iPad or iPhone (Apple, Inc., Cupertino, CA). A subset of the authors (MDM, WRH) designed the logic of the DST based directly upon the AHA/ACC guidelines. An iOS programmer created the executable application. Various screenshots of the DST can be seen in Appendix 1, which demonstrate how a user would navigate the ACC/AHA guidelines using the DST. Participants had no prior exposure or formal training to the DST before this study. The DST was distributed electronically to participants on the day of testing and the software expired immediately after use so that participants tested on subsequent days would not have confounding access to the DST prior to their testing session.

A single, optional, online didactic lecture provided a short introduction to the ACC/AHA perioperative cardiac guidelines and an introduction to the DST. The same online training was used at all study sites. This didactic was intentionally brief (22 minutes), as its goal was orientation to the use of the DST rather than extensive education and review of the ACC/AHA guidelines. The goal of keeping this brief was to test the true capacity of residents to apply guidelines based on their existing functional knowledge of these published guidelines, rather than testing them immediately after an extensive review, as was used in two prior studies.4,5 Orientation was performed a minimum of two weeks prior to implementation of the cognitive aid at all sites to prevent score inflation associated with very recent exposure. Testing was performed with local participation in a standardized (classroom) environment, which included supervision from trained study personnel who administered the test and gave access to the DST according to group assignment. All participants were excused from active clinical duties and asked to silence all pagers except hospital-wide mayday pagers when appropriate. Testing of participants was accomplished within a 10-day period at all sites.

To build on the recent works by Vigoda and colleagues, MCQ stems were created with a correct answer representing each evaluation path and terminal endpoint of the ACC/AHA perioperative cardiac evaluation algorithm.4,5 Although there is no defined guideline for the design and assessment of MCQs, our process was similar to that previously described.4,5 The clinical stems and corresponding MCQ answer options were assessed for construct and content validity by evaluating whether they actually represented an adequate test of knowledge of the ACC/AHA guidelines under consideration, both in the specifics of each question being asked and in the scope of the questions as a whole concerning the information contained in the guidelines. This was done by having a cohort of anesthesiology faculty from 3 institutions who are actively involved in preoperative assessment take the quizzes and assess the manner in which the content of the quiz reflected the content of the guidelines and adequately tested the scope of knowledge represented in the guidelines. This procedure followed a modified Delphi technique method, and as such the quizzes went through several iterative changes that incorporated faculty input until no further improvements to the questions were suggested. As the grading of the quiz was objective with correct answers agreed upon by the group of faculty that evaluated the quiz, there was no reliability assessment of grading. There was also no reliability measure of participant performance as each quiz was only taken once and under different testing conditions (i.e. memory alone v. cognitive aid). The stems and MCQs can be reviewed in Appendix 2.

The response options were also categorized into four groups: correct, incorrect and associated with increased risk to the patient, incorrect and associated with increased cost to the patient or medical system, and incorrect and associated with both increased risk and cost. The categorization of incorrect responses was created by two authors (WRH and MDM) and subsequently validated by the remainder of authors. Categorization was based on the intervention each answer option would require (i.e. obtaining an unnecessary electrocardiogram posed no risk to a patient needing non-emergent surgery but would increase cost and therefore would be categorized as incorrect and associated with increased cost to the patient or medical system). Participants were instructed to select the single best answer for each question from memory or as directed by the DST depending on the testing condition (with or without DST). The answers given to each item were recorded in a database along with the duration required to complete each nine-question quiz.

Statistical Analyses

Generalized linear mixed models (GLMMs) were used to evaluate the data because participants took 2 separate quizzes under two testing environments (with and without DST). 35 The generalized linear mixed models (GLMMs) were used to determine the independent effect of the use of the DST on overall scores, on individual quiz item scores, and on the duration of time spent taking the quizzes. These models adjusted for the potential influence of the specific quiz being administered (Quiz 1 vs. 2), the order in which the quiz was taken (first vs. second), the presence or absence of the DST, and the study site (MUSC, UNC, UK, VUMC). By including these covariates, the GLMMs provided the opportunity for us to conduct F-tests on each of these, to determine whether any of them had any independent association with the participants’ scores and quiz durations, and they provided, via the intraclass correlation coefficient, a means of estimating the reliability between subjects’ scores on the 2 quizzes. Interaction terms between the use of the DST and quiz order were assessed and included in the final models when they were moderately significant (p<0.10) because of the potential for the DST to be differentially beneficial when taking the first versus second quiz (i.e. to account for a possible learning effect from use of the DST first). Overall quiz scores, durations, and number of incorrect responses that would result in increased healthcare costs were treated as continuous variables with a linear link function in their GLMM whereas individual quiz item scores were treated as binary variables with logit link functions in their GLMM. Random participant effects were used in the GLMMs to account for correlation between scores within the same participants. For continuous measures, adjusted mean scores (i.e. scores that adjusted for order, site, and quiz effects) were estimated along with their 95% confidence intervals using the GLMMs. For binary responses, the use of the DST was quantified using adjusted probabilities of correct responses, along with odds ratios and their 95% confidence intervals. For each quiz, we also investigated internal consistency using Cronbach’s alpha. Analyses were conducted using SAS v9.3 (SAS Institute Inc, Cary, NC) Proc MIXED, Proc GLIMMIX, and the Proc CORR procedures, and p-values <0.05 were considered statistically significant.

Results

A total of 111 residents (of 211 recruited) participated among the four sites (MUSC 37, UK 33, UNC 25, and VUMC 16). One participant at MUSC consented to participate and finished 5/18 questions before being paged for an emergent intubation; his results were removed from the dataset. All data sets were complete with no ambiguity in answer selection or other potential source of error apparent to the authors. Of the 111 residents participating, 36 (32%) were post-graduate year (PGY) 4 or higher, 29 (26%) were PGY-3, 30 (27%) were PGY-2, and 16 (14%) were PGY-1 or interns (see Table 1 for demographic breakdown by group). Based on gender, race (Caucasian vs. non-Caucasian), or PGY, there were no significant differences between the four groups. Cronbach’s alpha coefficients were calculated to be 0.66 for Quiz 1 and 0.65 for Quiz 2, indicating a moderate level of internal consistency for each.

Table 1.

Demographics

Group A B C D Total
N 31 26 26 28 111
Male 22 19 18 19 78 (70%)
Caucasian 27 21 22 25 95 (86%)
PGY-1 4 4 4 4 16 (14%)
PGY-2 9 6 7 8 30 (27%)
PGY-3 9 7 5 8 29 (26%)
PGY-4 9 9 10 8 36 (32%)
Open in a new tab

Demographics: the number of participants in groups A-D is shown (N). Further detail includes the number of male, Caucasian, and PGY 1–4 participants.

N = Number, PGY = Post -graduate year

Table 2 summarizes the findings of the GLMM for participants’ overall scores. The intraclass correlation coefficient was estimated to be 0.26, indicating that subjects’ scores on Quiz 1 tended to be moderately correlated with their scores on Quiz 2. The scores on the quizzes (Quiz 1 and 2) were comparable (p>0.05), validating that there was not a significant difference in difficulty of the quiz. However, there was a clear order effect, meaning that subjects tended to score better on the second quiz when compared with the first, regardless of whether it was Quiz 1 or 2. The results indicated that the DST had a statistically significant impact on both the first (p<0.0001) and second (p<0.0001) quiz scores, and the DST benefit was significantly more pronounced (p<0.01) when used during the first quiz as compared to being used during the second quiz, as seen in Table 2. On average, the DST resulted in participants scoring 2.9 points higher when the DST was employed during their first quiz and 1.6 points higher when employed during their second quiz, which represents an absolute improvement in performance of 32% and 18%, respectively, and a relative improvement of 60.4% and 23.9%, respectively. The DST by quiz interaction was not significant, providing some evidence of the DST’s generalizability. There was no significant (p=0.98) difference in scores across the 4 sites (see Table 2). PGY also had no significant effect on performance, except that subjects with more years of training were significantly less likely (p=0.04) to have an incorrect response that would increase risk to the patient. The odds of an incorrect response that increased risk to the patient declined by 31% with each additional year of training (odds ratio = 0.69, 95% CI = 0.48 to 0.99).

Table 2.

Results of the generalized linear mixed model analyses assessing the impact of cognitive aid on participants’ overall scores.

Effect Adjusted Mean Score (Number Correct) 95% Confidence Interval P-Value**
Quiz 1 6.9 6.7 to 7.2 0.74
Quiz 2 6.9 6.6 to 7.1
Site
 UK 6.9 6.6 to 7.3 0.98
 UNC 6.9 6.5 to 7.3
 MUSC 6.8 6.5 to 7.2
 VUMC 6.9 6.4 to 7.4
First Quiz
 Without Cognitive Aid 4.8 4.5 to 5.2
 With Cognitive Aid 7.7 7.4 to 8.1
 Impact of Cognitive Aid† 2.9* 2.4 to 3.4 <0.0001
Second Quiz
 Without Cognitive Aid 6.7 6.4 to 7.1
 With Cognitive Aid 8.3 8.0 to 8.7
 Impact of Cognitive Aid† 1.6* 1.1 to 2.1 <0.0001
Open in a new tab
†

Impact of cognitive aid is defined as the difference between the scores with and without the cognitive aid.

*

P < 0.01 comparing impact of cognitive aid on first quiz compared with impact of cognitive aid on second quiz.

**

Obtained via F-tests based on results from the generalized linear mixed models

MUSC = Medical University of South Carolina, UK = University of Kentucky, UNC = University of North Carolina, VUMC = Vanderbilt University Medical Center

Table 3 summarizes the findings of the GLMM assessing the impact of the factors associated with the duration of time it took participants to take their quizzes. Durations were comparable on the 2 quizzes, and participants’ second quizzes took less time, on average, than their first quizzes. Use of the cognitive aid did increase the quiz duration on both their first (p<0.0001) and second (p<0.001) quizzes, with the added duration being moderately longer on the first quiz compared with their second (4.4 vs. 2.4 added minutes, p=0.06). There were differences in the quiz durations by site, with participants taking longer at UK and MUSC (10.9 and 11.0 minutes) than at UNC and VUMC (9.0 minutes for both).

Table 3.

Results of the analyses assessing the impact of cognitive aid on participants’ quiz durations.

Effect Adjusted Quiz Duration (in minutes) 95% Confidence Interval P-Value*
Quiz 1 9.8 9.1 to 10.4 0.30
Quiz 2 10.2 9.5 to 10.8
Site
 UK 10.9 10.0 to 11.8 <0.01
 UNC 9.0 8.0 to 10.1
 MUSC 11.0 10.1 to 11.9
 VUMC 9.0 7.6 to 10.3
First Quiz
 Without Cognitive Aid 8.5 7.7 to 9.4
 With Cognitive Aid 12.9 12.0 to 13.9
 Impact of Cognitive Aid† 4.4 3.1 to 5.6 <0.0001
Second Quiz
 Without Cognitive Aid 8.0 7.1 to 8.9
 With Cognitive Aid 10.4 9.6 to 11.3
 Impact of Cognitive Aid† 2.4 1.2 to 3.7 <0.001
Open in a new tab
†

Impact of cognitive aid is defined as the difference between the scores with and without the cognitive aid.

*

Obtained via F-tests based on results from the generalized linear mixed models

MUSC = Medical University of South Carolina, UK = University of Kentucky, UNC = University of North Carolina, VUMC = Vanderbilt University Medical Center

Table 4 summarizes the findings of the GLMMS assessing the impact of the cognitive aid on each of the individual quiz items. The cognitive aid was associated with a statistically significant improvement of a correct responses for scenario question numbers 1, 3, 4, 7, 8, and 9, even after adjusting for quiz type, quiz order, and study site.

Table 4.

Results of GLMMs assessing the impact of the cognitive aid on individual questions

Quiz Item Number Adjusted* Probability of Responding Correctly Odds Ratio and 95% Confidence Interval P-Value**
1: Without Cognitive Aid 55.3% reference
1: With Cognitive Aid 94.6% 14.1 (6.6 to 30.2) <0.0001
2: Without Cognitive Aid 83.3% reference
2: With Cognitive Aid 89.9% 1.8 (0.9 to 3.6) 0.11
3: Without Cognitive Aid 64.0% reference
3: With Cognitive Aid 93.3% 7.8 (3.4 to 18.1) <0.0001
4: Without Cognitive Aid 66.4% reference
4: With Cognitive Aid 92.2% 6.0 (2.5 to 14.0) <0.0001
5: Without Cognitive Aid 89.3% reference
5: With Cognitive Aid 93.8% 1.8 (0.7 to 4.7) 0.20
6: Without Cognitive Aid 73.1%† reference
6: With Cognitive Aid 99.1%† 38.8 (0.0 to 61,436) 0.33
7: Without Cognitive Aid 38.4% reference
7: With Cognitive Aid 79.6% 6.3 (3.8 to 10.3) <0.0001
8: Without Cognitive Aid 66.5% reference
8: With Cognitive Aid 93.9% 7.7 (2.7 to 22.1) <0.001
9: Without Cognitive Aid 48.0% reference
9: With Cognitive Aid 88.4% 8.3 (4.0 to 17.1) <0.0001
Open in a new tab
*

Probability of correct responses were adjusted for quiz type (1 vs. 2), order (first vs. second), study site (UK, UNC, MUSC, Vanderbilt), and participant effects using generalized linear mixed models (GLMMs).

**

Obtained via F-tests based on results from the generalized linear mixed models

†

On quiz item number six in Quiz 2, all participants responded correctly (with and without the cognitive aid), and this resulted in non-estimable parameters within the GLMM; thus the results only reflect the impact of the cognitive aid within Quiz 1.

GLMM = Generalized linear mixed model

Use of the DST also had statistically significant impact on the number of incorrect quiz responses that were deemed to increase healthcare costs, but not on incorrect responses that would increase risk to patients. Overall there were 468 incorrect selections. Of the incorrect selections, 387 (83% of incorrect) were assigned to the “increased cost” category, 38 (8%) to the “increased risk” category, and 43 (9%) to the “increased cost and risk” category. Table 5 highlights the fact that the impact of the DST on quiz items healthcare costs was statistically significant (p<0.0001) for both quizzes. Interestingly, the impact was greater if the DST was employed during the participants’ first quiz when compared with their second (p<0.001).

Table 5.

Results of GLMMs assessing the impact of the cognitive aid on incorrect responses that would result in increased cost.

Study Outcome Adjusted* Mean (Number Incorrect) 95% Confidence Interval P-Value**
Incorrect Responses that Increase Costs
 First Quiz
  Without Cognitive Aid 4.0 3.7 to 4.4
  With Cognitive Aid 1.0 0.6 to 1.3
  Impact of Cognitive Aid† −3.1 −3.6 to −2.6 <0.0001
 Second Quiz
  Without Cognitive Aid 2.1 1.8 to 2.5
  With Cognitive Aid 0.5 0.2 to 0.9
  Impact of Cognitive Aid† −1.6 −2.1 to −1.1 <0.0001
Open in a new tab
*

Mean number of incorrect responses were adjusted for quiz type (1 vs. 2), order (first vs. second), study site (UK, UNC, MUSC, Vanderbilt), and participant effects using generalized linear mixed models.

**

Obtained via F-tests based on results from the generalized linear mixed models

†

Impact of cognitive aid is defined as the difference between the scores with and without the cognitive aid.

GLMM = Generalized linear mixed model

Discussion

Our results demonstrate three important findings. First, the use of an electronic DST increased adherence to the 2007 ACC/AHA guidelines. Second, use of the electronic DST reduced management errors associated with increased costs, such as ordering unnecessary tests and consults. Third, use of the DST resulted in a longer time to complete the test.

The electronic, smart-phone based DST improved absolute performance, as measured by adherence to the ACC/AHA preoperative assessment algorithm, by an average of 25% as compared to performance from memory alone. This result was consistent across residents at all years of training at four different institutions. In specific, the DST provided a statistically significant benefit for 6 of the 9 MCQ scenarios (see Table 4). The recent works by Vigoda et al highlight the inability of residents and practicing anesthesiologists to apply the ACC/AHA Guidelines on Perioperative Cardiovascular Evaluation and Care for Noncardiac Surgery.4,5 Our results among residents taking the first test without the assistance of DST correctly answered an average of 4.8/9 (53%) questions. Similar to Vigoda’s studies, our rate of error beckons for a mechanism of improvement—and the improvement is seen with our simple DST. Under the same first-quiz conditions, the DST group correctly answered 7.7/9 (86%) questions. The efficacy of the DST is consistent with other treatment algorithms that have been assessed with DSTs with excellent uptake and success.19–24, 36–38 Perhaps the reason DSTs were adopted for Advanced Cardiac Life Support is related to the high-stakes and high-stress clinical environments in which code scenarios usually occur, but this research advocates the use of a DST in the calm and routine preoperative care setting. The authors propose that better adherence to published guidelines may, in fact, reduce the number of high-stakes events being managed with a DST due to higher quality care.

The DST provided statistically significant improvement for six of the nine possible scenarios when evaluating performance on both Quiz 1 and 2. Scenario 6 did not reach statistical significance because all respondents correctly answered this question on Quiz 2, creating non-estimable parameter for the GLMM; we remain encouraged by the absolute improvement despite the mathematical nuance created by the lack of an estimable error rate. That said, the composite percentages of 73% vs. 99% for testing without and with DST, respectively, certainly appear to show benefits consistent with the other scenarios showing statistically significant improvement. For clinical scenarios number 2 and 5, statistical significance was not reached despite the fact that the scores with DST were amongst the highest for any scenario. We are encouraged that the absolute percentages were higher with the DST and believe with a large sample size we would have had the power to demonstrate a statistically significant improvement. The high scores for these scenarios in the memory-alone group were similar to these in the DST group.

The second finding of interest is that 83% of the incorrect options selected by participants would increase the cost of care either to the patient directly or the medical system providing care to the patient (e.g. requesting an unwarranted electrocardiogram or choosing to “delay until blood pressure controlled”). During the design of this study the authors associated each possible incorrect MCQ option with either increasing the cost, risk, or both cost and risk to the patient. Participants testing with or without the DST did well avoiding selections that increased risk to the patient; however, 83% of incorrect selections included decisions to over-test or delay procedures at an increased cost to the patient and healthcare system. When using the DST, participants made 77% fewer incorrect responses that would have resulted in increased costs compared to answering from memory alone (3.1/4 for Quiz 1 and 1.6/2.1 for Quiz 2).

The third finding of interest is that participants required an average of 3.4 minutes (204s) longer to complete the quizzes when using the DST. This equates to a mere 23 additional seconds per question (204 seconds ÷ 9 questions) required to improve adherence to an important guideline in perioperative medicine. Hospitals face the demand to deliver higher quality care with greater efficiency than ever before. However, we posit that the investment of fractionally more time using a DST would be more than offset by the correct application of best-practice guideline. We also expect that efficiency using this DST would improve with familiarity—as previously described, the residents had almost no prior exposure to the DST software in order to prevent any excessive learning effect that would have confounded the “from memory” test scores. These 23 seconds may be seen as an investment in quality as they portended improved adherence to the guideline in question. In comparison, “the average surgical ‘time out’ at VUMC consumes 57 seconds (average of last 172,000 electronically observed time outs), and the ‘time out’ has been credited with reducing surgical site and procedure errors.” (Jesse Ehrenfeld, MD, Associate Professor, Department of Anesthesiology, Vanderbilt University, Nashville, TN) Several studies have shown deviation from published guidelines increases morbidity and overall hospital cost. As any DST is considered for adoption, one must recognize that implementation requires planning and education as there is often hidden work associated with new technologies.39–41 With adequate education, orientation to a DST, and rapid practitioner adoption, the authors believe the benefits shown in this study could be magnified.

A limitation is that we evaluated performance of anesthesiologists-in-training, not practicing physicians. Improved performance by non-residents cannot be inferred from this research, despite similarly poor performance by practicing physicians demonstrated by Vigoda et al.4,5 Additionally, while the results of this study appear generalizable among several institutions, the generalizability of the use of an electronic DST beyond the scope of the ACC/AHA guidelines cannot be made at this time. Future research needs to address whether similar tools will be of benefit for other perioperative management guidelines. 42–44 It would be intuitive to assume that any such DST could improve adherence to guidelines, but we and others have shown that simply having guidelines in front of the clinician does not guarantee adherence.9,45,46 Specific work is needed to investigate the human factors in each application. In addition, the quizzes used in this study were not created using rigorous psychometric methods; given their moderate degree of consistency (Cronbach’s alpha coefficients), it is possible that the DST might perform differently (better or worse) if different assessment tools were used.

We found that the use of the DST was associated with more work per simulated patient encounter. Future research in this domain of implementation science needs to address whether this differential amount of time exists in simulated encounters with human standardized patients and in clinical settings. Our design could not determine the effect that order had beyond a simple description of percent correct. For example, we were unable to identify which type of errors might be prevented by a test taker who did or did not have previous exposure to the DST. We, therefore, are unable to make specific statements about prospective benefits of DST use.

Our small sample size prevented us from making statements about each question stem independently, and resulted in the conclusion that the DST improved performance on only 6 of 9 MCQ stems. Sample size was estimated based on overall percent correct and an improvement with the DST. We did not perform a reliability assessment (e.g. test-retest) in order to test whether the grade achieved precisely represents a stable description of participant knowledge. Finally, we did not achieve perfect adherence to and application of the AHA/ACC Guidelines under consideration, even though the DST contained all of the proper logic for such application. This failure is likely multifactorial: participant error, lack of attention, lack of knowledge (e.g. what constitutes an emergent case), or lack of familiarity with the DST itself. The design of the study prevented substantial practice navigating the DST in order to maintain a valid baseline for scores without DST assistance.

In summary, we demonstrated that a simple DST improved resident adherence to the ACC/AHA guidelines on perioperative cardiovascular evaluation and care for noncardiac surgery evaluated via multiple-choice questions with clinical stems. There was a significant improvement in overall performance of residents when using the DST. Residents selected fewer incorrect options in 6 of 9 scenarios and primarily reduced incorrect selections that would increase the cost of medical care. As anesthesiologists strive to practice evidence-based medicine, these results indicate that it may be time to embrace decision support technology during the preoperative assessment and planning phase of patient care. Future research needs to address the implementation of such tools in the clinical setting.

Acknowledgments

Funding:

Foundation for Anesthesia Education and Research (FAER) (Rochester, MN), Research in Education Grant (PI: McEvoy). This project was also supported by the South Carolina Clinical & Translational Research Institute, Medical University of South Carolina’s Clinical and Translational Institute and National Institute of Health/National Center for Advancing Translational Sciences (Bethesda, MD) Grant Number UL1TR000062 and National Center for Research Resources Award Number UL1RR029882 (Charleston, SC).

Appendix 1

Various screenshots from the decision support tool as seen by a user. The screenshots demonstrate both user interface (questions to answer) and specific endpoints of the preoperative testing algorithm. The actual decision support tool utilizes a touchscreen interface and only one screen is visible at any time.

Appendix 1.

Appendix 1

Appendix 1

Appendix 1

Appendix 1

Appendix 1

Open in a new tab

Screenshots from Decision Support Tool

Appendix 2

Multiple Choice Questions. Quizzes A and B. The scenario descriptions and multiple-choice questions were presented with and without the decision support tool available and in assigned order to each participant.

P # _____ Cognitive Aid YES NO Time Start: ___:____
Quiz A Order: 1st 2nd Time Finish: ___:____
Open in a new tab

  1. 65 year-old male with 40 pack-year tobacco history, obesity, and anxiety disorder was recently diagnosed with laryngeal cancer presenting for radical neck dissection. He does no physical activity due to social anxiety. His father died of heart disease at 45. The patient’s pre-operative vital signs are T: 36.5 HR: 82 BP: 155/72 RR: 16.

    Multiple choice:

    1. Proceed with planned surgery.

    2. Refer for treadmill cardiac stress test.

    3. Need ECG before I can make a recommendation.

    4. Delay surgery until BP is < 140/70.

    5. Refer for cardiac consultation.

  2. 61 year-old female with controlled hypertension, Insulin dependent diabetes (Hbg A1c 9.9%), and gastro-esophageal reflux disease is scheduled for metatarsal tendon release for bilateral “hammer toes”. Glucose ranges from 110–230 mg/dL. She works as a waitress and walks to and from work, worsening her foot pain. The patient’s pre-operative vital signs are T: 36.5 HR: 78 BP: 124/82 RR: 14.

    Multiple choice:

    1. Proceed to surgery.

    2. Order a stress test (dobutamine echo, exercise, or nuclear imaging).

    3. Start beta blocker therapy and delay surgery until heart rate adequately controlled.

    4. Delay surgery until Hgb A1c < 7%.

    5. Patient needs ECG before I can make recommendation.

  3. 72 year-old male with a history of poorly controlled diabetes on insulin therapy, CHF, TIA with residual weakness on his left side, hypertension, and a 50 pack-year history of smoking presents for elective repair of 6.2 cm abdominal aortic aneurysm. He his able to walk only one block at a time because of left sided weakness. The patient’s pre-operative vital signs are T: 36.5 HR: 98 BP: 165/85 RR: 18.

    Multiple choice:

    1. Proceed to planned surgery.

    2. Patient needs to have PFTs before proceeding.

    3. Consider testing if it will change management.

    4. Delay surgery overnight to collect records from outside providers.

    5. Obtain immediate cardiology consultation.

  4. 71 year-old male with poorly controlled Insulin dependent diabetes and hypertension which led to renal dysfunction and peripheral neuropathy describes his activity as being limited to dressing, brushing his teeth, and clicking the television remote control. His preoperative creatinine is 2.3 mg/dL. Colonoscopy revealed colon cancer and you are evaluating him for a colectomy in five days. The patient’s pre-operative vital signs are T: 36.5 HR: 82 BP: 144/78 RR: 14.

    Multiple choice:

    1. Stress test is needed before proceeding to OR.

    2. Start beta-blockers and delay surgery one month.

    3. Refer for 2-D echocardiogram.

    4. Need ECG before I can make recommendation.

    5. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

  5. 68 year-old female with poorly controlled diabetes and hypertension presents after an automobile accident with an open femur fracture and cold right foot. The patient is stable and answers your questions. She is unable to do more than her daily chores without becoming short of breath and had a stroke two years ago. The patient’s pre-operative vital signs are T: 36.5 HR: 98 BP: 156/55 RR: 14.

    Multiple choice:

    1. Obtain immediate cardiology consultation.

    2. Patient needs to have 2-D echocardiogram before I make recommendation.

    3. Need ECG before I make recommendation.

    4. Consider non-invasive testing if it would change management.

    5. Proceed to planned surgery.

  6. 62 year-old male with 40 pack-year smoking history and ischemic stroke two years ago presents for an elective femoral-popliteal bypass due to claudication. He can walk several blocks but is ultimately limited by leg pain. The patient’s pre-operative vital signs are T: 36.5 HR: 78 BP: 178/82 RR: 12.

    Multiple choice:

    1. Refer for cardiology consultation.

    2. Need ECG before I can make recommendation.

    3. Delay surgery until BP <140/70.

    4. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

    5. Stress test is required before going to OR.

  7. 66 year-old male active smoker (1ppd x20 years) with severe osteoarthritis is scheduled for total knee arthroplasty. He suffered an MI 10 weeks ago but did not require percutaneous intervention and is completing rigirous cardiac rehabilitation despite knee pain. He was discharged with beta-blocker, statin, and ace inhibitor. He is compliant with medications and endorses only knee pain as limitation to exertion. The patient’s pre-operative vital signs are T: 36.5 HR: 62 BP: 118/59 RR: 14.

    Multiple choice:

    1. Surgery should be delayed at least 3 months after a myocardial infarction.

    2. Patient should have a stress test (dobutamine echo, exercise, or nuclear image) before surgery.

    3. Proceed with planned surgery.

    4. Surgery should be delayed at least 6 months after a myocardial infarction.

    5. Need ECG and pulmonary function tests before I can make a recommendation.

  8. 72 year-old male with history of calf claudication who has Type 2 diabetes requiring insulin therapy, hypertension and chronic renal insufficiency (Cr 2.2 mg/dL) is scheduled for femoral-popliteal bypass. He endorses worsening chest pain and shortness of breath while doing household chores. The patient’s pre-operative vital signs are T: 36.5 HR: 55 BP: 145/85 RR: 18.

    Multiple choice:

    1. Patient needs a cardiac catheterization immediately.

    2. Patient should have a stress test, beta-blockade, and cardiology consult prior to surgery.

    3. Proceed with surgery if today’s ECG is unchanged from ECG performed 5 months ago.

    4. Proceed with planned surgery with HR control.

    5. Proceed with planned surgery.

  9. 68 year-old male presents with poorly controlled diabetes on insulin therapy, hypertension, chronic kidney dysfunction with creatinine of 2.1, congestive heart failure, and 40 pack-year history of smoking. He is unable to walk more then 3 blocks secondary to fatigue and some knee pain. He is presenting for total knee replacement. The patient’s pre-operative vital signs are T: 36.5 HR: 98 BP: 165/85 RR: 18.

    Multiple choice:

    1. Proceed to planned surgery.

    2. Patient needs to have PFTs before I make recommendation.

    3. Need ECG before I make recommendation.

    4. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

    5. Obtain immediate cardiology consultation.

P # _____ Cognitive Aid YES NO Time Start: ___:____
Quiz B Order: 1st 2nd Time Finish: ___:____
Open in a new tab

  1. 66 year-old female with uncontrolled diabetes with diabetic nephropathy (creatinine of 2.1), hypertension, TIA two years ago with no residual symptoms, and a symptomatic pituitary adenoma presents for elective transphenoidal resection of pituitary tumor. She manages her ADLs but does not leave her residence due to fatigue. The patient’s pre-operative vital signs are T: 36.5 HR: 98 BP: 165/85 RR: 18.

    Multiple choice:

    1. Proceed to planned surgery.

    2. Patient needs to have 2-D echocardiogram before I make recommendation.

    3. Need ECG before I make recommendation.

    4. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

    5. Obtain immediate cardiology consultation.

  2. 67 year-old female with uncontrolled diabetes, hypertension, and dialysis-dependent renal disease presents for same-day evaluation from dialysis clinic. She has developed a clot and infection in her A-V fistula, which has since developed subcutaneous crepitus. Patient is posted for exploration and revision. She has poor functional capacity and her last stress test (last week) was consistent with inducible ischemia. Patient has scheduled an appointment with her cardiologist in one week to discuss intervention. The patient’s pre-operative vital signs are T: 36.5 HR: 128 BP: 156/55 RR: 14.

    Multiple choice:

    1. Obtain immediate cardiology consultation.

    2. Patient needs to have 2-D echocardiogram before I make recommendation.

    3. Need ECG before I make recommendation.

    4. Consider non-invasive testing if it would change management.

    5. Proceed to surgery.

  3. 70 year-old female who had a myocardial infarction 6 years ago presents for preoperative evaluation for nephrectomy to treat renal cell carcinoma. She has chronic renal insufficiency (Cr 2.5 mg/dL), hypertension, and indicates she has to rest after walking two blocks on flat ground but symptoms resolve quickly with rest. The patient’s pre-operative vital signs are T: 36.5 HR: 82 BP: 144/78 RR: 14.

    Multiple choice:

    1. PFTs needed before proceeding to OR.

    2. Start beta-blockers and delay surgery four weeks.

    3. Refer for 2-D echocardiogram.

    4. Need ECG before I can make recommendation.

    5. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

  4. 66 year-old female with hypertension and hypothyroidism presents for evaluation before her scheduled lumbar laminectomy and fusion. Her leg pain prevents her from standing and she is not sure what exertion she is capable of. Patient takes a statin for hypercholesterolemia and thyroid hormone replacement. The patient’s pre-operative vital signs are T: 36.5 HR: 82 BP: 125/72 RR: 16.

    Multiple choice:

    1. Proceed with planned surgery.

    2. Refer for dobutamine stress test due to inactivity.

    3. Need ECG before I can make a recommendation.

    4. Delay surgery until BP is < 140/70.

    5. Refer for cardiac consultation.

  5. 66 year-old female with hypertension, chronic renal insufficiency (Cr 2.2 mg/dL), and Type 2 diabetes endorses fatigue while folding laundry and new chest pressure walking to mailbox on flat land. The patient presents for elective knee arthroplasty for degenerative joint disease. The patient’s pre-operative vital signs are T: 36.5 HR: 55 BP: 145/85 RR: 18.

    Multiple choice:

    1. Patient needs a cardiac catheterization immediately.

    2. Patient should have a stress test, beta blockade, and cardiology consult prior to surgery.

    3. Proceed with surgery if today’s ECG is unchanged from ECG performed 5 months ago.

    4. Proceed with planned surgery with HR control.

    5. Proceed with planned surgery.

  6. 64 year-old female with poorly controlled Type 2 diabetes (Hbg A1c 10.1%), controlled hypertension, and gout presents for laparoscopic cholecystectomy. Blood glucose ranges from 110–230 mg/dL and she says she is able to keep up with her peers in a twice-per-week aerobics class. The patient’s pre-operative vital signs are T: 36.5 HR: 78 BP: 124/82 RR: 14.

    Multiple choice:

    1. Proceed to surgery.

    2. Order a stress test (dobutamine echo, exercise, or nuclear imaging).

    3. Start beta blocker therapy and delay surgery until heart rate adequately controlled.

    4. Delay surgery until Hgb A1c < 7%.

    5. Patient needs ECG before I can make recommendation.

  7. 70 year-old female with long smoking history (1ppd) presents for lumbar laminectomy due to radicular pain unsuccessfully treated by steroid injections. After her myocardial infarction two and a half months ago, she was medically treated with beta-blocker, ace inhibitor, and statin. She swims 20 minutes without stopping every day. The patient’s pre-operative vital signs are T: 36.5 HR: 62 BP: 118/59 RR: 14.

    Multiple choice:

    1. Surgery should be delayed at least 3 months after a myocardial infarction.

    2. Patient should have a stress test (dobutamine echo, exercise, or nuclear image) before surgery.

    3. Proceed with planned surgery.

    4. Surgery should be delayed at least 6 months after a myocardial infarction.

    5. Need ECG and pulmonary function tests before I can make a recommendation.

  8. 74 year-old female with controlled hypertension, uncontrolled Type 2 diabetes, and peripheral vascular disease presents for evaluation before having a newly diagnosed 5.6cm abdominal aortic aneurysm repair. Her exertional capacity is limited to light housework before fatiguing. The patient’s pre-operative vital signs are T: 36.5 HR: 78 BP: 178/82 RR: 12.

    Multiple choice:

    1. Refer for cardiology consultation.

    2. Need ECG before I can make recommendation.

    3. Delay surgery until BP <140/70.

    4. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

    5. Stress test is required before going to OR.

  9. 74 year-old male who suffered an ischemic stroke two years ago presents with stable calf claudication after 2 blocks that limits exertion, uncontrolled hypertension, and uncontrolled Insulin dependent diabetes is scheduled for elective ileo-femoral bypass. Patient states he doesn’t think he has any heart issues and hasn’t visited a cardiologist since his stroke. The patient’s pre-operative vital signs are T: 36.5 HR: 98 BP: 165/85 RR: 18.

    Multiple choice:

    1. Proceed to planned surgery.

    2. Patient needs to have PFTs before I make recommendation.

    3. Proceed to surgery with heart rate control or consider noninvasive testing if it will change management.

    4. Delay surgery overnight to collect records from outside providers.

    5. Obtain immediate cardiology consultation.

Footnotes

Conflict of Interest: The authors declare no competing interests.

Contributor Information

William R. Hand, Email: handw@musc.edu, Department of Anesthesia & Perioperative Medicine, Medical University of South Carolina, 167 Ashley Avenue, Suite 301, Charleston, South Carolina 29425, 843-792-5442, 843-792-9314 (fax).

Kathryn H Bridges, Department of Anesthesia & Perioperative Medicine, Medical University of South Carolina, Charleston, South Carolina.

Marjorie P Stiegler, Department of Anesthesiology, University of North Carolina, Chapel Hill, Chapel Hill, North Carolina.

Randall M Schell, Department of Anesthesiology, University of Kentucky, Lexington, Kentucky.

Amy N DiLorenzo, Department of Anesthesiology, University of Kentucky, Lexington, Kentucky.

Jesse M Ehrenfeld, Departments of Anesthesiology, Surgery, and Biomedical Inforamtics, Vanderbilt University Medical Center, Nashville, Tennessee.

Paul J Nietert, Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina.

Matthew D McEvoy, Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee.

References

  • 1.Fleischmann KE, Goldman L, Young B, Lee TH. Association between cardiac and noncardiac complications in patients undergoing noncardiac surgery: Outcomes and effects on length of stay. Am J Med. 2003;115:515–20. doi: 10.1016/s0002-9343(03)00474-1. [DOI] [PubMed] [Google Scholar]
  • 2.Fleisher LA, Beckman JA, Brown KA, Calkins H, Chaikof E, Fleischmann KE, Freeman WK, Froehlich JB, Kasper EK, Kersten JR, Riegel B, Robb JF. ACC/AHA 2007 guidelines on perioperative cardiovascular evaluation and care for noncardiac surgery: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines on Perioperative Cardiovascular Evaluation for Noncardiac Surgery) J Am Coll Cardiol. 2007;50:e159–241. doi: 10.1016/j.jacc.2007.09.003. [DOI] [PubMed] [Google Scholar]
  • 3.Fleischmann KE, Beckman JA, Buller CE, Calkins H, Fleisher LA, Freeman WK, Froehlich JB, Kasper EK, Kersten JR, Robb JF, Valentine RJ. 2009 ACCF/AHA focused update on perioperative beta blockade: A report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation. 2009;120:2123–51. doi: 10.1161/CIRCULATIONAHA.109.192689. [DOI] [PubMed] [Google Scholar]
  • 4.Vigoda MM, Sweitzer B, Miljkovic N, Arheart KL, Messinger S, Cadniotti KC, Lubarsky DA. 2007 American College of Cardiology/American Heart Association (ACC/AHA) Guidelines on perioperative cardiac evaluation are usually incorrectly applied by anesthesiology residents evaluating simulated patients. Anesth Analg. 2011;112:940–9. doi: 10.1213/ANE.0b013e31820a1457. [DOI] [PubMed] [Google Scholar]
  • 5.Vigoda MM, Behrens V, Miljkovic N, Arheart KL, Lubarsky DA, Dutton RP. Perioperative cardiac evaluation of simulated patients by practicing anesthesiologists is not consistent with 2007 ACC/AHA guidelines. J Clin Anes. 2012;24:446–55. doi: 10.1016/j.jclinane.2011.11.007. [DOI] [PubMed] [Google Scholar]
  • 6.Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PC, Rubin HR. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458–65. doi: 10.1001/jama.282.15.1458. [DOI] [PubMed] [Google Scholar]
  • 7.Sintchenko V, Coiera E, Iredell JR, Gilbert GL. Comparative Impact of Guidelines, Clinical Data, and Decision Support on Prescribing Decisions: An Interactive Web Experiment with Simulated Cases. J Am Med Inform Assoc. 2004;11:71–7. doi: 10.1197/jamia.M1166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Litvin CB, Ornstein SM, Wessell AM, Nemeth LS, Nietert PJ. Adoption of a clinical decision support system to promote judicious use of antibiotics for acute respiratory infections in primary care. Int J of Med Inform. 2012;81:521–6. doi: 10.1016/j.ijmedinf.2012.03.002. [DOI] [PubMed] [Google Scholar]
  • 9.Milani RV, Lavie CJ, Dornelles AC. The impact of achieving perfect care in acute coronary syndrome: The role of computer assisted decision support. Am Heart J. 2012;164:29–34. doi: 10.1016/j.ahj.2012.04.004. [DOI] [PubMed] [Google Scholar]
  • 10.Haut ER, Lau BD, Kraenzlin FS, Hobson DB, Kraus PS, Carolan HT, Haider AH, Holzmueller CG, Efron DT, Pronovost PJ, Streiff MB. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch of Surg. 2012;147:901–7. doi: 10.1001/archsurg.2012.2024. [DOI] [PubMed] [Google Scholar]
  • 11.Milani RV, Lavie CJ, Dornelles AC. The impact of achieving perfect care in acute coronary syndrome: The role of computer assisted decision support. Am Heart J. 2012;164:29–34. doi: 10.1016/j.ahj.2012.04.004. [DOI] [PubMed] [Google Scholar]
  • 12.Dechant LM. UA/NSTEMI: Are you following the latest guidelines? Nursing. 2012;42:26–33. doi: 10.1097/01.NURSE.0000418611.61100.ca. quiz 34. [DOI] [PubMed] [Google Scholar]
  • 13.Lodewijckx C, Sermeus W, Vanhaecht K, Panella M, Deneckere S, Leigheb F, Decramer M. Inhospital management of COPD exacerbations: A systematic review of the literature with regard to adherence to international guidelines. J Eval Clin Pract. 2009;15:1101–10. doi: 10.1111/j.1365-2753.2009.01305.x. [DOI] [PubMed] [Google Scholar]
  • 14.Loomba RS, Arora R. ST elevation myocardial infarction guidelines today: A systematic review exploring updated ACC/AHA STEMI guidelines and their applications. Am J Ther. 2009;16:e7–e13. doi: 10.1097/MJT.0b013e31818d40df. [DOI] [PubMed] [Google Scholar]
  • 15.O’Donnell C, Verbeek R. Guidelines for STEMI. CMAJ. 2005;172:1425–6. doi: 10.1503/cmaj.1041729. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sykes PK. Prevention and management of postoperative delirium among older patients on an orthopedic surgical unit: A best practice implementation project. J Nurs Care Qual. 2012;27:146–53. doi: 10.1097/NCQ.0b013e31823f8573. [DOI] [PubMed] [Google Scholar]
  • 17.Asche CV, Leader S, Plauschinat C, Raparla S, Yan M, Ye X, Young D. Adherence to current guidelines for chronic obstructive pulmonary disease (COPD) among patients treated with combination of long-acting bronchodilators or inhaled corticosteroids. Int J Chron Obstruct Pulmon Dis. 2012;7:201–9. doi: 10.2147/COPD.S25805. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Calvin JE, Shanbhag S, Avery E, Kane J, Richardson D, Powell L. Adherence to evidence-based guidelines for heart failure in physicians and their patients: Lessons from the Heart Failure Adherence Retention Trial (HART) Congest Heart Fail. 2012;18:73–8. doi: 10.1111/j.1751-7133.2011.00263.x. [DOI] [PubMed] [Google Scholar]
  • 19.Haut ER, Lau BD, Kraenzlin FS, Hobson DB, Kraus PS, Carolan HT, Haider AH, Holzmueller CG, Efron DT, Pronovost PJ, Streiff MB. Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma. Arch Surg. 2012;147:901–7. doi: 10.1001/archsurg.2012.2024. [DOI] [PubMed] [Google Scholar]
  • 20.Kalla K, Christ G, Karnik R, Malzer R, Norman G, Prachar H, Schreiber W, Unger G, Glogar HD, Kaff A, Laggner AN, Maurer G, Mlczoch J, Slany J, Weber HS, Huber K Vienna STEMI Registry Group. Implementation of guidelines improves the standard of care: The Viennese registry on reperfusion strategies in ST-elevation myocardial infarction (Vienna STEMI registry) Circulation. 2006;113:2398–405. doi: 10.1161/CIRCULATIONAHA.105.586198. [DOI] [PubMed] [Google Scholar]
  • 21.Mangin D. Adherence to evidence-based guidelines is the key to improved health outcomes for general practice patients: NO. J Prim Health Care. 2012;4:158–60. [PubMed] [Google Scholar]
  • 22.Vause J. Adherence to evidence-based guidelines is the key to improved health outcomes for general practice patients: YES. J Prim Health Care. 2012;4:156–8. [PubMed] [Google Scholar]
  • 23.Okelo SO, Butz AM, Sharma R, Diette GB, Pitts SI, King TM, Linn ST, Reuben M, Chelladurai Y, Robinson KA. Interventions to Modify Health Care Provider Adherence to Asthma Guidelines: A Systematic Review. Pediatrics. 2013;132:517–34. doi: 10.1542/peds.2013-0779. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, Mann DM. Efficacy of an Evidence-Based Clinical Decision Support in Primary Care Practices: A Randomized Clinical Trial. JAMA Intern Med. 2013;173:1584–91. doi: 10.1001/jamainternmed.2013.8980. [DOI] [PubMed] [Google Scholar]
  • 25.Flynn D, Ford GA, Stobbart L, Rodgers H, Murtagh MJ, Thomson RG. A review of decision support, risk communication and patient information tools for thrombolytic treatment in acute stroke: Lessons for tool developers. BMC Health Serv Res. 2013;13:225. doi: 10.1186/1472-6963-13-225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.El-Kareh R, Hasan O, Schiff GD. Use of health information technology to reduce diagnostic errors. BMJ Qual Saf. 2013;22(Suppl 2):ii40–51. doi: 10.1136/bmjqs-2013-001884. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Wanderer JP, Ehrenfeld JM. Clinical Decision Support for Perioperative Information Management Systems. Semin Cardiothorac Vasc Anesth. 2013;17:288–93. doi: 10.1177/1089253213490078. [DOI] [PubMed] [Google Scholar]
  • 28.Wanderer JP, Sandberg WS, Ehrenfeld JM. Real-time alerts and reminders using information systems. Anesthesiol Clin. 2011;29:389–96. doi: 10.1016/j.anclin.2011.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.St Jacques P, Sanders N, Patel N, Talbot TR, Deshpande JK, Higgins M. Improving timely surgical antibiotic prophylaxis redosing administration using computerized record prompts. Surg Infect (Larchmt) 2005;6:215–21. doi: 10.1089/sur.2005.6.215. [DOI] [PubMed] [Google Scholar]
  • 30.O’Reilly M, Talsma A, VanRiper S, Kheterpal S, Burney R. An anesthesia information system designed to provide physician-specific feedback improves timely administration of prophylactic antibiotics. Anesth Analg. 2006;103:908–12. doi: 10.1213/01.ane.0000237272.77090.a2. [DOI] [PubMed] [Google Scholar]
  • 31.Wax DB, Beilin Y, Levin M, Chadha N, Krol M, Reich DL. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg. 2007;104:1462–6. doi: 10.1213/01.ane.0000263043.56372.5f. [DOI] [PubMed] [Google Scholar]
  • 32.Kooij FO, Klok T, Hollmann MW, Kal JE. Decision support increases guideline adherence for prescribing postoperative nausea and vomiting prophylaxis. Anesth Analg. 2008;106:893–8. doi: 10.1213/ane.0b013e31816194fb. [DOI] [PubMed] [Google Scholar]
  • 33.Eden A, Pizov R, Toderis L, Kantor G, Perel A. The impact of an electronic reminder on the use of alarms after separation from cardiopulmonary bypass. Anesth Analg. 2009;108:1203–8. doi: 10.1213/ane.0b013e3181984ef9. [DOI] [PubMed] [Google Scholar]
  • 34.Zilberberg MD, Chaudhari P, Nathanson BH, Campbell RS, Emons MF, Fiske S, Hays HD, Shorr AF. Development and validation of a bedside risk score for MRSA among patients hospitalized with complicated skin and skin structure infections. BMC Infect Dis. 2012;11;12:154. doi: 10.1186/1471-2334-12-154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.McCulloch C, Searle SR. Generalized, Linear, and Mixed Models. New York: John Wiley & Sons; 2001. pp. 28–57. [Google Scholar]
  • 36.Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AA. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–9. doi: 10.1056/NEJMsa0810119. [DOI] [PubMed] [Google Scholar]
  • 37.van Klei WA, Hoff RG, van Aarnhem EE, Simmermacher RK, Regli LP, Kappen TH, van Wolfswinkel L, Kalkman CJ, Buhre WF, Peelen LM. Effects of the introduction of the WHO “Surgical Safety Checklist” on in-hospital mortality: A cohort study. Ann Surg. 2012;255:44–9. doi: 10.1097/SLA.0b013e31823779ae. [DOI] [PubMed] [Google Scholar]
  • 38.Mills PD, DeRosier JM, Neily J, McKnight SD, Weeks WB, Bagian JP. A cognitive aid for cardiac arrest: You can’t use it if you don’t know about it. Jt Comm J Qual Saf. 2004;30:488–96. doi: 10.1016/s1549-3741(04)30057-2. [DOI] [PubMed] [Google Scholar]
  • 39.Novak LL. Finding hidden sources of new work from BCMA implementation: The value of an organizational routines perspective. AMIA Annual Symposium proceedings/AMIA Symposium AMIA Symposium. 2012;2012:673–80. [PMC free article] [PubMed] [Google Scholar]
  • 40.Wears RL, Berg M. Computer Technology and Clinical Work: Still Waiting for Godot. JAMA. 2005;293:1261–3. doi: 10.1001/jama.293.10.1261. [DOI] [PubMed] [Google Scholar]
  • 41.Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;13:742–52. doi: 10.7326/0003-4819-144-10-200605160-00125. [DOI] [PubMed] [Google Scholar]
  • 42.Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA Checklist Improves Trainee Performance During a Simulated Episode of Local Anesthetic Toxicity. Reg Anesth Pain Med. 2012;37:8–21. doi: 10.1097/AAP.0b013e31823d825a. [DOI] [PubMed] [Google Scholar]
  • 43.Practice Alert for the Perioperative Management of Patients with Coronary Artery Stents: A Report by the American Society of Anesthesiologists Committee on Standards and Practice Parameters. Anesthesiology. 2009;110:22–3. doi: 10.1097/ALN.0b013e3181928913. [DOI] [PubMed] [Google Scholar]
  • 44.Douketis JD, Spyropoulos AC, Spencer FA, Mayr M, Jaffer AK, Eckman MH, Dunn AS, Kunz R. Perioperative management of antithrombotic therapy: Antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141(2 Suppl):e326S–e350S. doi: 10.1378/chest.11-2298. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Arriaga AF, Bader AM, Wong JM, Lipsitz SR, Berry WR, Ziewacz JE, Hepner DL, Boorman DJ, Pozner CN, Smink DS, Gawande AA. Simulation-Based Trial of Surgical-Crisis Checklists. N Engl J Med. 2013;368:246–53. doi: 10.1056/NEJMsa1204720. [DOI] [PubMed] [Google Scholar]
  • 46.Ziewacz JE, Arriaga AF, Bader AM, Berry WR, Edmondson L, Wong JM, Lipsitz SR, Hepner DL, Peyre S, Nelson S, Boorman DJ, Smink DS, Ashley SW, Gawande AA. Crisis Checklists for the Operating Room: Development and Pilot Testing. J Am Coll Surg. 2011;213:212–7. doi: 10.1016/j.jamcollsurg.2011.04.031. [DOI] [PubMed] [Google Scholar]

RESOURCES