Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Jun 1.
Published in final edited form as: Am J Surg. 2014 Oct 13;209(6):977–984. doi: 10.1016/j.amjsurg.2014.06.037

Acceptability of the Decision Support for Safer Surgery Tool

Wynne E Norton a, Patrick W Hosokawa b, William G Henderson b, Eric T Volckmann c, Joyce Pell c, Majed G Tomeh d, Robert E Glasgow c, Sung-Joon Min b, Leigh A Neumayer c, Mary T Hawn e
PMCID: PMC4395521  NIHMSID: NIHMS641551  PMID: 25457241

Abstract

Background

We examined providers' perceptions of the Decision Support for Safer Surgery (DS3) tool which provided preoperative patient-level risk estimates of postoperative adverse events.

Methods

The DS3 tool was evaluated at two academic medical centers. During the validation study, surgeons provided usefulness ratings of the DS3 tool for each patient prior to surgery. At the end of the study, providers' perceptions of the DS3 tool were assessed via questionnaire. Data were analyzed using descriptive statistics and independent samples t-tests.

Results

During the trial, twenty-three surgeons completed usefulness ratings of the DS3 tool for 1,006 patients. Surgeons rated the tool as Very Useful or Moderately Useful in 251 (25%) of the cases, Neutral in 469 (46.6%) of the cases, and Moderately Un-Useful or Not Useful in 286 (28.4%) cases. At the end of the trial, thirty-two providers completed the questionnaire; perceptions were relatively neutral, although several aspects were rated quite favorably.

Conclusions

The DS3 tool may be most useful for achieving particular tasks (e.g., training novice surgeons; increasing patient engagement) or encouraging specific processes (e.g., team-based care) in surgical care settings.

Major surgical complications are associated with significant increases in perioperative and late mortality,1 prolonged length of stay,2 and marked increases in hospital costs.3 Significant efforts to reduce surgical complications have been made recently, including (but not limited to) the National Surgical Quality Improvement Program (NSQIP),4,5 an outcomes-driven quality improvement program initially developed in the Veterans Health Administration (VHA) and also implemented in the private sector with sponsorship from the American College of Surgeons (ACS).6-9 While implementation of NSQIP has demonstrated improvements in both quality of care and surgical outcomes,4 the benefits of NSQIP are limited by its reliance on risk reporting and adjustment through the use of retrospective data after the operations have been completed vs. risk prediction and mitigation through the use of prospective data prior to the operations being performed.

In an effort to enhance NSQIP capability, the Decision Support for Safer Surgery (DS3) tool was developed to provide preoperative patient-level estimates of risk of postoperative adverse events to surgical care providers.10 The objective of the original study was to prospectively assess the predictive validity of the DS3 tool by comparing risk estimates provided by statistical models to risk estimates provided by experienced surgeons for 30-day postoperative mortality, overall morbidity, and a range of postoperative complications (e.g., cardiac, pulmonary, thromboembolic, renal, and surgical site infection). The prospective observational cohort study included a diverse group of 1,791 general surgery patients from two large academic medical centers (University of Alabama at Birmingham [UAB] and University of Utah [Utah]) who were operated on between June 2010 and January 2012. Prior to each enrolled patient's surgical procedure, attending surgeons provided patient-level estimates of postoperative morbidity and mortality. In addition, a research assistant entered risk data, including patient-level demographics, general medical conditions (e.g., functional status, weight, height, BMI and ASA class), comorbidities, and operative variables, into the DS3 web-based software system to generate patient-level estimates of postoperative morbidity and mortality based on developed statistical prediction models.10 The statistical model estimates provided via the DS3 tool performed as well as experienced surgeons in predicting postoperative adverse events across a range of diverse patients and surgical procedures.11 Moreover, correlations between model estimates and surgeon estimates of postoperative adverse events were statistically significant for each outcome category (p<0.0001).11 n sum, the DS3 tool has the potential to improve quality of care and patient outcomes by systematically identifying high-risk patients and, importantly, allowing for steps to reduce perioperative morbidity and mortality.

Although research is needed to demonstrate the impact of the DS3 tool on care processes and patient outcomes, steps can be taken early-on to design the tool with dissemination in mind12 in order to bridge the anticipated research-to-practice gap. In an effort to accelerate the potential integration of this tool into practice settings, secondary, exploratory aims of the study, described herein, were to (1) describe attending surgeons' usefulness ratings of the risk estimates provided in the DS3 tool and (2) assess surgical care team members' (e.g., surgeons, nurses, anesthesiologists, information technologists and clinic support staff) perceptions of the DS3 tool. As described below, the first aim included 23 surgeons' ratings of the usefulness of the DS3 tool conducted in vivo as part of the original trial. The second aim involved a brief survey administered at the end of the original trial to 32 surgical care team members to assess their perceptions of the DS3 tool.

Methods

DS3 Usefulness Ratings

Attending general surgeons from two academic medical centers (UAB and Utah) were asked to complete a risk assessment for each patient for a list of postoperative adverse events prior to seeing the risk assessment from the statistical model built into the DS3 tool. Surgeons provided a probability assessment for each adverse outcome (e.g., 5%, 10%, etc.) and rated the patient's risk assessment for each adverse outcome as low, average, or high. Usefulness of the statistical model risk assessment provided to surgeons was assessed by a single-item (i.e., ‘Please rate the usefulness of this risk assessment’) administered via paper-and-pencil and measured on a 5-point Likert-scale (1 = Not Useful, 2 = Moderately Un-Useful, 3 = Neutral, 4 = Moderately Useful, 5 = Very Useful).

DS3 Questionnaire

Toward the end of the prospective observational study, a paper-and-pencil questionnaire was administered at each site following a study team debriefing and facilitated discussion. Individuals who completed the questionnaire included surgeons who had been involved in the prospective study (i.e., provided risk estimates for patients' postoperative adverse events) and other key members of the surgical care team (e.g., nurses, anesthesiologists, information technologists and clinic support staff), all of whom were identified as key stakeholders by the lead site investigator. Immediately following the debriefing meeting and facilitated discussion, all surgical care team members (e.g., surgeons, nurses, anesthesiologists, etc.) completed a paper-and-pencil questionnaire to assess their attitudes toward the risk estimates provided by the DS3 tool, described below.

Demographics

Demographic items included age, gender, and race/ethnicity. Participants were asked to indicate their current professional role (i.e., surgeon, nurse, anesthesiologist, administrator, information technologist, clinic support staff, or other [open-ended response option]) and years of experience in their field. A single-item was used to assess participants' prior experience with and/or knowledge of the DS3 tool and measured on a 5-point Likert scale (1 = None, 5 = A Great Deal). Categorical or continuous response options were provided, as appropriate.

Perceptions of DS3 Tool

Participants' perceptions of the DS3 tool were assessed by 4-subscales: (1) General Attitudes toward the DS3 Tool, (2) Information Technology Aspects of DS3 Tool, (3) Impact of DS3 Tool on Practitioners' Workflow, and (4) Impact of DS3 Tool on Clinical Care Setting. Consistent with the Diffusion of Innovations theory,13 items for each subscale were constructed to assess attributes of innovations that can facilitate or impede their adoption and use, including its relative advantage (i.e., degree to which an innovation is perceived as better than the idea it supersedes), complexity (i.e., degree to which an innovation is perceived as difficult to understand or use), compatibility (i.e., degree to which an innovation is perceived as being consistent with the existing values, past experiences, and needs of potential adopters), observability (i.e., degree to which the results of an innovation are visible to others) and trialability (i.e., degree to which an innovation may be experimented with on a limited basis).13,14 We also assessed participants' attitudes toward the DS3 tool effectiveness (i.e., performance), cost (i.e., money or other resources) and simplicity (i.e., inverse of complexity) based on our review of the literature.15-17

Each item was assessed on a 5-point Likert scale (1 = Strongly Disagree, 5 = Strongly Agree). In addition to presenting descriptives for each individual item, we also created a summed score of each subscale, with higher scores indicating more positive attitudes toward the DS3 tool. General Attitudes toward DS3 Tool was assessed on a 16-item subscale, which demonstrated reasonable reliability (α = 0.71). Information Technology Aspects of DS3 Tool was assessed on a 7-item subscale, which demonstrated moderate reliability (α = 0.63). Impact of DS3 Tool on Practitioners' Work was assessed on a 7-item subscale. The scale demonstrated reasonable reliability (α = 0.62). Finally, Impact of DS3 Tool on Clinical Care Setting was assessed on a 6-item subscale and demonstrated relatively poor reliability (α = 0.56).

The study was approved by the Institutional Review Boards at the University of Colorado Anschutz Medical Campus, the University of Utah, the University of Alabama at Birmingham, and the New England IRB for QCMetrix, Inc.

Data Analysis

DS3 Tool Usefulness Ratings

Usefulness ratings provided by attending surgeons were summarized by frequency distributions. The relationships between usefulness ratings and patient risk level and complexity of the operation were tested using independent samples t-tests.

DS3 Questionnaire

Descriptive statistics were used to characterize the overall sample in terms of demographics. Associations between participant characteristics (e.g., gender, prior experience with the DS3 tool) and attitudes towards the tool were examined via correlational analysis. Descriptive statistics were used to characterize responses to individual items on each of the four subscales assessing participant's attitudes toward the DS3 tool. Inter-item correlations and scale reliability were examined within subscales; average scores were computed for scales after reverse coding items (where appropriate). Independent sample t-tests were used to examine differences in responses between surgeons and non-surgeons (i.e., nurse, anesthesiologist, administrator, information technologist, clinic support staff, or other) for individual items as well as average subscale scores. For simplicity of presentation, the data from the two participating sites (i.e., University of Utah and University of Alabama at Birmingham) were combined for analysis. All analyses were conducted in SPSS v.17.

Results

DS3 Usefulness Ratings

Twenty-three attending surgeons from the two participating sites completed usefulness ratings on the statistical model estimates provided by the DS3 tool of the likelihood of the patient having postoperative complications. These usefulness ratings were completed for 1,006 patients (average per surgeon, 43.7 ratings, range 1-112). In 28 (2.8%) of the cases, the attending surgeon rated the risk estimates provided by the DS3 tool as very useful, 223 (22.2%) as moderately useful, 469 (46.6%) as neutral, 126 (12.5%) as moderately un-useful, and 160 (15.9%) as not useful. Table I displays the relationship between the surgeon's usefulness ratings and the risk level of the patient as judged by the surgeon and statistical model for mortality and overall morbidity, as well as a measure of the complexity of the patient's operation. Surgeons assessed the usefulness of the model estimates as “neutral” more often for patients at a lower risk level and having lower complexity operations. However, surgeons tended to equally assess the models as useful and not useful for patients at higher risk level and having higher complexity operations.

Table I. Patient risk level and complexity of the operation by usefulness rating.

Usefulness Response N Mean Surgeon Risk Estimate Mean Model Risk Estimate Mean Work RVU
Mortality Morbidity Mortality Morbidity
Not Useful (1, 2) 285 0.51% 8.29% 0.54% 9.63% 17.77
Neutral (3) 468 0.44% 4.77% 0.29% 5.70% 13.74
Useful (4, 5) 249 0.79% 8.72% 0.68% 10.69% 18.24
Difference 1-2 vs. 3 0.1904 <.0001 0.0015 <.0001 <.0001
Difference 3 vs. 4-5 <.0001 <.0001 <.0001 <.0001 <.0001
Difference 1-2 vs. 4-5 0.0007 0.6099 0.2355 0.2553 0.6895
Open in a new tab

Note. Usefulness response options: 1 = Not Useful, 2 = Moderately Un-useful, 3 = Neutral, 4 = Moderately Useful, 5 = Very Useful. Independent samples t-tests were used to compare mean differences across usefulness ratings on patient risk level and complexity of the operation.

DS3 Questionnaire

Table II summarizes demographic characteristics of the 32 individuals who partially or fully completed the questionnaire after the study completion, with approximately equal numbers from UAB (n = 14) and Utah (n = 18). Demographics of the questionnaire participants were male (n = 17, 54.8%), White (n = 26, 89.7%), and less than or equal to 39 years old (n = 13, 44.8%). The largest group of participants was surgeons (n = 14, 48.3%), with fewer numbers of nurses (n = 4, 13.8%), anesthesiologists (n = 2, 6.9%), administrative personnel (n = 5, 17.2%), or other (n = 4, 13.8%, including ‘Critical Care’ (n = 1), ‘Clerical’ (n = 1), ‘Quality,’ (n = 1) and ‘no response provided’ (n = 1). Prior experience with and/or knowledge of the DS3 tool was distributed as follows: None/Little (n = 12, 38.7%), Somewhat (n = 7, 20.7%), and Much/A Great Deal (n = 12, 38.7%). Correlations between participant characteristics and attitudes were non-significant, albeit likely a reflection of the small sample size rather than lack of any true association between variables.

Table II.

Demographic characteristics of UAB (N = 14) and Utah (N = 18) participants completing the DS3 Tool Attitudes Questionnaire (N = 32).

Variable Response Options UAB, N (%) Utah, N (%) Overall, N (%)*
Gender Male 7 (50%) 10 (58.8%) 17 (54.8%)
Female 7 (50%) 7 (41.2%) 14 (45.2%)
Race/Ethnicity White 11 (84.6%) 15 (93.8%) 26 (89.7%)
Asian 1 (7.7%) 1 (6.3%) 2 (6.9%)
Other 1 (7.7%) 0 (0%) 1 (3.4%)
Age <39 6 (46.2%) 7 (43.8%) 13 (44.8%)
40-49 5 (38.5%) 4 (25%) 9 (31%)
>50 2 (15.4%) 5 (31.3%) 7 (24.1%)
Professional Role Surgeon 6 (46.2%) 8 (50%) 14 (48.3%)
Nurse 3 (23.1%) 1 (6.3%) 4 (13.8%)
Anesthesiologist 0 (0%) 2 (12.5%) 2 (6.9%)
Administrator 2 (15.4%) 3 (18.8%) 5 (17.2%)
Other** 2 (15.4%) 2 (12.5%) 4 (13.8%)
Prior Experience with DS3 Tool None/Little 7 (50%) 5 (29.4%) 12 (38.7%)
Somewhat 3 (21.4%) 4 (23.5%) 7 (20.7)
Much/Great Deal 4 (28.6%) 8 (47.1%) 12 (38.7%)
Open in a new tab
*

Numbers do not always add to N = 32 or 100% due to missing data or rounding.

**

Open-ended response option available for ‘Other.’ Responses included Critical Care (n = 1; UAB), Clerical (n = 1; Utah), Quality (n = 1; Utah), and no additional response provided (n = 1; UAB).

General Attitudes toward the DS3 Tool

As displayed in Table III, participants' average score on the 16-item subscale was 3.01 (SD = 0.58), indicating overall neutrality in attitudes toward the DS3 system across the 16 subscale items. However, some of the individual items were quite favorable toward the DS3 tool, including that it would be best embedded in the EMR (M: 4.26, SD: 0.57), that it has the potential to improve quality and safety of surgery (4.22, 0.60), that it seems to capture all of the risk factors (4.13, 0.57), and that it would be fairly easy to understand and use after training (3.90, 0.47). No statistically significant differences (p≤0.05) were observed between surgeons and non-surgeons on the average score or on any of the individual items.

Table III. DS3 Tool Questionnaire Descriptives (N = 32).
Subscale/Items Overall M (SD) Surgeons M (SD) Non-Surgeons M (SD) p-value
A. General Attitudes toward DS3 Tool
1. The DS3 tool would be best embedded in the EMR (vs. stand-alone product). 4.26 (0.57) 4.21 (0.69) 4.33 (0.48) 0.64
2. The DS3 tool has the potential to improve quality and safety of surgical care. 4.22 (0.60) 4.21 (0.69) 4.27 (0.59) 0.82
3. The DS3 does NOT capture all risk factors for all procedures.1 4.13 (0.57) 4.36 (0.49) 4.00 (0.57) 0.09
4. The DS3 tool would be easy to understand and use after receiving training. 3.90 (0.47) 3.79 (0.42) 4.07 (0.45) 0.09
5. Risk prediction from the DS3 will improve patient care. 3.81 (0.83) 3.71 (1.06) 3.87 (0.64) 0.64
6. The DS3 does NOT account for intra-operative events which may influence postoperative risk. 1 3.69 (0.66) 3.71 (0.61) 3.77 (0.72) 0.83
7. Not all variables in the DS3 are available at the time of pre-op consultation (e.g., ASA).1 3.57 (1.04) 3.43 (1.01) 3.71 (1.13) 0.49
8. The DS3 tool would be more effective than what we are currently using in my hospital. 3.53 (0.80) 3.43 (0.93) 3.53 (0.64) 0.72
9. The DS3 tool is applicable to my patient population. 3.37 (0.89) 3.14 (1.02) 3.57 (0.75) 0.22
10. The DS3 does NOT adequately assess interaction between risk factors and surgical procedure (i.e. high risk patient/low risk procedure vs. low risk patient/high risk procedure).1 3.17 (0.75) 3.43 (0.75) 3.00 (0.70) 0.14
11. It takes too much time to enter data in the DS3 tool.1 3.16 (0.68) 3.07 (0.47) 3.27 (0.88) 0.47
12. The DS3 can easily be incorporated into routine clinical care. 3.10 (0.94) 3.14 (0.86) 2.93 (1.03) 0.56
13. The DS3 tool requires too many resources.1 3.03 (0.71) 3.00 (0.67) 3.14 (0.77) 0.60
14. The DS3 tool is too complex.1 2.90 (0.66) 2.93 (0.61) 2.86 (0.77) 0.78
15. It will be difficult to identify who is responsible for entering data into the DS3 tool. 1 2.84 (0.77) 2.93 (0.82) 2.80 (0.77) 0.66
16. Risk prediction from the DS3 is available in a timely fashion to allow for intervention. 2.67 (0.88) 2.71 (0.82) 2.57 (0.93) 0.67
Average score on General Attitudes2 subscale 3.01 (0.58) 3.09 (0.31) 3.04 (0.45) 0.72
B. Information Technology Aspects of DS3 Tool
1. The DS3 tool could be readily adapted for different electronic systems of care. 3.88 (0.76) 3.85 (0.55) 3.91 (0.94) 0.84
2. If clinic personnel wanted to use the DS3 tool, I would implement it in the care setting. 3.81 (0.69) 3.64 (0.74) 4.00 (0.63) 0.21
3. The DS3 tool would require a lot of updating and troubleshooting.1 3.29 (0.89) 3.43 (0.75) 3.25 (1.05) 0.62
4. The DS3 tool could be easily implemented within my facility's electronic infrastructure. 3.18 (1.05) 2.86 (1.16) 3.33 (0.77) 0.24
5. The DS3 tool would make my job easier. 3.15 (0.71) 2.86 (0.66) 3.42 (0.66) 0.04*
6. Integrating the DS3 tool into the care setting is NOT my responsibility.1 2.70 (0.95) 2.93 (0.91) 2.55 (0.93) 0.31
7. The DS3 tool would take too much time to integrate into the system.1 2.68 (0.81) 2.71 (0.46) 2.75 (1.05) 0.91
Average score on Information Technology2 subscale 3.22 (0.52) 3.12 (0.32) 3.30 (0.72) 0.39
C. Impact of DS3 Tool on Practitioners' Work
1. The DS3 tool will enhance discussions I have with patients regarding informed consent. 4.00 (0.83) 4.00 (0.78) 4.00 (0.95) 1.00
2. The DS3 tool will improve the quality of care I provide. 3.36 (0.82) 3.14 (0.94) 3.58 (0.66) 0.19
3. I would recommend the DS3 tool to my other colleagues at other hospitals. 3.40 (0.57) 2.36 (0.49) 2.82 (0.60) 0.86
4. I intend to adopt and use the DS3 tool once it becomes available. 3.28 (0.54) 3.36 (0.63) 3.10 (0.31) 0.25
5. The DS3 tool will enable me to accomplish tasks more quickly. 2.70 (0.72) 2.29 (0.46) 3.08 (0.66) 0.002**
6. Decision-support tools interfere with my ability to provide high quality care.1 2.62 (0.63) 3.64 (0.49) 3.18 (0.60) 0.04*
7. The DS3 tool will NOT help me identify high-risk patients.1 2.50 (1.03) 2.50 (0.85) 2.36 (1.20) 0.74
Average score on Impact of DS3 Tool on Work2 subscale 3.05 (0.95) 3.32 (0.45) 2.94 (1.10) 0.24
D. Impact of DS3 Tool on Clinical Care Setting
1. Reducing surgical complications is a high priority for my practice. 4.58 (0.70) 4.86 (0.36) 4.36 (0.80) 0.05*
2. The DS3 tool will be a good education training tool for residents. 4.11 (0.49) 4.07 (0.47) 4.15 (0.55) 0.68
3. The DS3 tool will improve allocation of resources and appropriate level of care post-operatively (e.g., ICU, step down, overnight observation vs. outpatient). 3.81 (0.68) 3.79 (0.80) 3.92 (0.51) 0.63
4. The DS3 tool is compatible with existing beliefs and practices at my facility. 3.69 (0.67) 3.86 (0.53) 3.50 (0.79) 0.18
5. There would be high demand for the DS3 tool within the surgical clinics at my facility. 3.04 (0.44) 3.14 (0.53) 2.91 (0.30) 0.20
6. The DS3 tool will NOT improve coding and capture of co-morbidities in my clinic.1 2.54 (0.81) 2.57 (0.85) 2.50 (0.79) 0.82
Average score on Impact of DS3 Tool on Clinic2 subscale 3.58 (0.69) 3.85 (0.38) 3.39 (0.80) 0.06
Open in a new tab

Note. 1 = Strongly Disagree, 2 = Disagree, 3 = Neither Agree nor Disagree, 4 = Agree, 5 = Strongly Agree.

*

p≤0.05

**

p≤0.01

1

Items were reverse scored prior to creation of average score.

2

Higher score indicates more positive attitudes.

Information Technology Aspects of DS3 Tool

As displayed in Table III, participants' average score on the 7-item subscale was 3.22 (SD = 0.52). Favorable items about the DS3 tool across both surgeons and non-surgeons included that it could be readily adapted for different electronic systems (3.88, 0.76) and that, if clinic personnel wanted to use it, it should be implemented in the care setting (3.81, 0.69). One significant difference was observed between surgeons and non-surgeons: non-surgeons reported that the DS3 tool would make their job easier more so than surgeons (3.42 vs. 2.86, p=0.04). No statistically significant differences were observed between surgeons and non-surgeons on the average score.

Impact of DS3 Tool on Practitioners' Work

As displayed in Table III, participants' average score on the 7-item subscale was 3.05 (SD = 0.95). Individual items favorable to the DS3 tool across both surgeons and non-surgeons included that the tool will enhance discussions with patients regarding informed consent for surgery (4.00, 0.83), participants would recommend the DS3 tool to other colleagues (3.40, 0.57), and that the DS3 tool will improve quality of care (3.36, 0.82). There were two statistically significant differences between surgeons and non-surgeons: surgeons were less agreeable with the idea that the DS3 tool would enable them to accomplish tasks more quickly (2.29, 0.46 for surgeons vs. 3.08, 0.66 for non-surgeons, p=0.002) and surgeons were more agreeable with the idea that the DS3 tool would improve ability to provide high quality care (3.64, 0.49 for surgeons vs. 3.18, 0.60 for non-surgeons, p=0.04). No statistically significant differences were observed between surgeons and non-surgeons on the average score.

Impact of DS3 Tool on Clinical Care Setting

As displayed in Table III, participants' average score on the 6-item subscale was 3.58 (SD = .69). Individual items that were favorable to the DS3 tool across both surgeons and non-surgeons included that reducing surgical complications is a high priority (4.58, 0.70), the DS3 tool will be a good education training tool for residents (4.11, 0.49), the DS3 tool will improve allocation of resources and appropriate levels of postoperative care (3.81, 0.68), and that the DS3 tool is compatible with existing beliefs and practices at the site (3.69, 0.67). There was one statistically significant difference observed between surgeons and non-surgeons: surgeons were more agreeable with the statement that reducing surgical complications is a high priority for their practice than non-surgeons (4.86, 0.36 for surgeons vs. 4.36, 0.80 for non-surgeons, p=0.05).

On the average score of impact of DS3 tool on clinical care settings, surgeons reported more positive attitudes than non-surgeons (3.85 vs. 3.39), although the p-value did not reach statistical significance (p = .06).

Discussion

The current study explored surgical care team members' perceptions of the of the DS3 tool, a system designed to provide preoperative patient-level risk estimates of postoperative adverse events. Across all patient cases (N = 1,006), attending surgeons most often rated the usefulness of the risk estimates produced by the DS3 tool as neutral (46.6%), followed by moderately un-useful or not useful (28.4%) and very useful or moderately useful (25%). Surgeons were more likely to rate the tool as more useful when patients were at a higher estimated risk for mortality and morbidity. At the debriefing following completion of the prospective parent trial, a similar pattern of results was found for surgical care team members' responses to the questionnaire assessing their perceptions of the DS3 tool, although several specific items were rated more favorably than others. Several differences in perceptions of the DS3 tool were also observed between surgeons and non-surgeons. Overall, results suggest that surgical care team members held relatively neutral to slightly positive attitudes toward the DS3 tool.

These findings identify specific aspects of the DS3 tool that may facilitate or impede integration into other surgical care settings, pending supportive data demonstrating the impact of the DS3 tool on improving care processes and patient outcomes. Aspects of the DS3 tool that were rated less favorably by surgical care team members will need to be addressed by revising or refining the tool (as applicable) and/or deploying tailored implementation strategies (e.g. external facilitation; interactive training) to help overcome some of the barriers toward use in surgical care settings. Conversely, aspects of the DS3 tool that were rated more favorably can be highlighted in informational sessions to increase interest and demand (i.e. ‘pull’) for the tool among potential end-users.

In addition to data presented above, participants also had suggestions for how the DS3 tool could improve team-based care processes throughout the perioperative period. During the study debriefing meetings, we conducted facilitated group discussions with team members (i.e., surgeon, nurse, anesthesiologist, administrator, information technologist, clinic support staff, or other; data not shown). While risk estimates produced by the DS3 tool were felt to be important, many care team members wanted the tool to provide specific suggestions or guidance for action in preoperative (e.g., patient counseling, delay surgery), operative (e.g., additional support staff), and/or post-operative phases (e.g., placement; additional monitoring; extended hospital stay) that corresponded to a patient's risk for post-operative complications. In contrast to many existing CDSSs or tools that are designed for use by a single provider, participants noted that risk estimates provided by the DS3 tool would be useful for the entire perioperative care team. With increased interest and importance of interprofessional collaboration18 and team-based care,19 future research will need to better understand how the DS3 tool and other CDSSs interact with the entire care team to either alter care processes or improve communication that leads to improved patient outcomes.

Limitations of the study should be noted. As is common across many areas of implementation research,20 the present study lacked validated measures to assess surgical care team members' perceptions of the DS3 tool. Although constructs and items were informed by relevant theory13,15 and validated measures in other health areas,21,22 future work is needed to develop validated measures to assess surgical care team members' perceptions of decision-support tools, and to examine whether more positive attitudes leads to increased likelihood of adoption of such tools. Findings are likely to reflect issues faced in other academic medical centers but may not generalize to other types of care centers (e.g., VHA).

Clinical decision support systems have the potential to improve care processes and practitioner performance across a range of clinical procedures, care settings and health conditions.23-26 Such systems, however, need to be assessed for feasibility and acceptability—and subsequently augmented and improved, as appropriate—in order to provide guidance on where, how and with whom they should be used, recognizing that not all CDSSs should be widely disseminated and/or implemented.27,28 Indeed, findings from the present study suggest that the DS3 tool in its current form may be most useful in specific (vs. all) care settings, such as those that have a large number of novice (vs. experienced) surgeons as well as those focused on cultivating a team-based (vs. segmented) approach towards perioperative care. Moreover, assessing perceived usefulness and perceptions of CDSSs can also help identify potential barriers for integrating such tools into a wider range of surgical care settings (pending supportive data of the impact of such tools on care processes and patient outcomes), and can inform the selection of targeted implementation strategies to increase adoption. This exploratory study assessed the feasibility and acceptability of the DS3 tool in its current form as a strategy for identifying the context in which the tool may be most useful (e.g., training novice surgeons; facilitating team-based care processes; increasing patient engagement) as well as future efforts to increase its adoption and use in surgical care settings, pending supportive data on its impact on care processes and patient outcomes.

Acknowledgments

This work was funded by grant #5R44NR010653 from the National Institute of Nursing Research Small Business Innovation Grant. Dr. Tomeh is the CEO of QCMetrix, Inc., and has potential financial interest in the development of the decision support model discussed in this article. Mr. Hosokawa and Dr. Norton received consulting fees from QCMetrix, Inc.

Footnotes

Disclosure Information: All other authors have no conflicts of interest to declare.

Author's Contribution: Norton, Hosokawa, Henderson, Volckmann, Pell, Tomeh, Glasgow, Min, Neumeyer and Hawn contributed to the study conception and design, data collection, writing the article, and critical revision of the article. Henderson, Tomeh, Glasgow, Neumeyer and Hawn contributed to obtaining funding for the study. Norton, Hosokawa, Henderson, Hawn and Pell contributed to data analysis and interpretation of study findings.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.Khuri SF, Henderson WG, DePalma RG, et al. Determinants of long-term survival after major surgery and the adverse effect of postoperative complications. Ann Surg. 2005 Sep;242(3):326–341. doi: 10.1097/01.sla.0000179621.33268.83. discussion 341-323. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Collins TC, Daley J, Henderson WH, Khuri SF. Risk factors for prolonged length of stay after major elective surgery. Ann Surg. 1999 Aug;230(2):251–259. doi: 10.1097/00000658-199908000-00016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Dimick JB, Chen SL, Taheri PA, Henderson WG, Khuri SF, Campbell DA., Jr Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program. J Am Coll Surg. 2004 Oct;199(4):531–537. doi: 10.1016/j.jamcollsurg.2004.05.276. [DOI] [PubMed] [Google Scholar]
  • 4.Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg. 1998 Oct;228(4):491–507. doi: 10.1097/00000658-199810000-00006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Daley J, Forbes MG, Young GJ, et al. Validating risk-adjusted surgical outcomes: site visit assessment of process and structure. National VA Surgical Risk Study. J Am Coll Surg. 1997 Oct;185(4):341–351. [PubMed] [Google Scholar]
  • 6.Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg. 2010;44:251–267. doi: 10.1016/j.yasu.2010.05.003. [DOI] [PubMed] [Google Scholar]
  • 7.Raval MV, Cohen ME, Ingraham AM, et al. Improving American College of Surgeons National Surgical Quality Improvement Program risk adjustment: incorporation of a novel procedure risk score. J Am Coll Surg. 2010 Dec;211(6):715–723. doi: 10.1016/j.jamcollsurg.2010.07.021. [DOI] [PubMed] [Google Scholar]
  • 8.Fink AS, Campbell DA, Jr, Mentzer RM, Jr, et al. The National Surgical Quality Improvement Program in non-veterans administration hospitals: initial demonstration of feasibility. Ann Surg. 2002 Sep;236(3):344–353. doi: 10.1097/00000658-200209000-00011. discussion 353-344. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Raval MV, Bentrem DJ, Eskandari MK, et al. The role of Surgical Champions in the American College of Surgeons National Surgical Quality Improvement Program--a national survey. J Surg Res. 2011 Mar;166(1):e15–25. doi: 10.1016/j.jss.2010.10.036. [DOI] [PubMed] [Google Scholar]
  • 10.Richman JS, Hosokawa PW, Min SJ, et al. Toward prospective identification of high-risk surgical patients. Am Surg. 2012 Jul;78(7):755–760. [PubMed] [Google Scholar]
  • 11.Glasgow RE, Hawn MT, Hosokawa PW, et al. Comparison of prospective risk estimates for postoperative complications: human vs computer model. J Am Coll Surg. 2014;218(2):237–245. doi: 10.1016/j.jamcollsurg.2013.10.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Klesges LM, Estabrooks PA, Dzewaltowski DA, Bull SS, Glasgow RE. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med. 2005 Apr;29(Suppl):66–75. doi: 10.1207/s15324796abm2902s_10. [DOI] [PubMed] [Google Scholar]
  • 13.Rogers EM. Diffusion of Innovations. Fifth. New York: Free Press; 2003. [Google Scholar]
  • 14.Dearing JW. Evolution of diffusion and dissemination theory. J Public Health Manag Pract. 2008 Mar-Apr;14(2):99–108. doi: 10.1097/01.PHH.0000311886.98627.b7. [DOI] [PubMed] [Google Scholar]
  • 15.Dearing JA. Measurement of innovation attributes. Denver: Center for Health Dissemination and Implementation Research; 2007. [Google Scholar]
  • 16.Dearing JW. Applying Diffusion of Innovation Theory to Intervention Development. Res Soc Work Pract. 2009 Sep 1;19(5):503–518. doi: 10.1177/1049731509335569. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Kortteisto T, Komulainen J, Makela M, Kunnamo I, Kaila M. Clinical decision support must be useful, functional is not enough: a qualitative study of computer-based clinical decision support in primary care. BMC Health Serv Res. 2012;12:349. doi: 10.1186/1472-6963-12-349. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Zwarenstein M, Goldman J, Reeves S. Interprofessional collaboration: effects of practice-based interventions on professional practice and healthcare outcomes. 2009 Cochrane Database Syst Rev;(3):CD000072. doi: 10.1002/14651858.CD000072.pub2. [DOI] [PubMed] [Google Scholar]
  • 19.Wynia MK, Von Kohorn I, Mitchell PH. Challenges at the intersection of team-based and patient-centered health care: insights from an IOM working group. JAMA. 2012 Oct 3;308(13):1327–1328. doi: 10.1001/jama.2012.12601. [DOI] [PubMed] [Google Scholar]
  • 20.Proctor E, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor E, editors. Dissemination and Implementation Research in Health. New York, NY: Oxford University Press Inc.; 2012. pp. 261–280. [Google Scholar]
  • 21.Aarons GA, Cafri G, Lugo L, Sawitzky A. Expanding the domains of attitudes towards evidence-based practice: the evidence based practice attitude scale-50. Adm Policy Ment Health. 2012 Sep;39(5):331–340. doi: 10.1007/s10488-010-0302-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS) Ment Health Serv Res. 2004 Jun;6(2):61–74. doi: 10.1023/b:mhsr.0000024351.12294.65. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005 Mar 9;293(10):1223–1238. doi: 10.1001/jama.293.10.1223. [DOI] [PubMed] [Google Scholar]
  • 24.Bright TJ, Wong A, Dhurjati R, et al. Effect of Clinical Decision-Support Systems: A Systematic Review. Ann Intern Med. 2012 Apr 23; doi: 10.7326/0003-4819-157-1-201207030-00450. [DOI] [PubMed] [Google Scholar]
  • 25.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998 Oct 21;280(15):1339–1346. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
  • 26.Roshanov PS, Misra S, Gerstein HC, et al. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6:92. doi: 10.1186/1748-5908-6-92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi: 10.1136/bmj.38398.500764.8F. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–530. doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES