Eye of the beholder: Risk calculators and barriers to adoption in surgical trainees

Abstract

Background:

Accurate risk assessment before surgery is complex and hampered by behavioral factors. Underutilized risk-based decision-support tools may counteract these barriers. The purpose of this study was to identify perceptions of and barriers to the use of surgical risk-assessment tools and assess the importance of data framing as a barrier to adoption in surgical trainees.

Methods:

We distributed a survey and risk assessment activity to surgical trainees at four training institutions. The primary outcomes of this study were descriptive risk assessment practices currently performed by residents, identifiable influences and obstacles to adoption, and the variability of preference sets when comparing modified System Usability Scores of a current risk calculator to a purpose-built calculator revision. Risk calculator comparison responses were compared with simple and multivariable regression to identify predictors for preferentiality.

Results:

We collected responses from 124 surgical residents (39% response rate). Participants endorsed familiarity with direct verbal communication (100%), sketch diagrams (87%), and brochures (59%). The most contemporary risk communication frameworks, such as best-worst case scenario framing (38%), case-specific risk calculators (43%), and all-procedure calculators (52%) were the least familiar. Usage favored traditional models of communication with only 26% of residents regularly using a strategy other than direct verbal discussion or anatomic sketch diagrams. Barriers limiting routine use included lack of electronic and clinical workflow integration. The mean modified System Usability Scores domain scores were widely dispersed for all domains, and no domain demonstrated one calculator’s superiority over another.

Conclusion:

Risk assessment tools are underutilized by trainees. Of importance, preference sets of clinicians appear to be unpredictable and may benefit more from a customizable, bespoke approach.

Introduction

Accurate surgical risk assessment is complex and hampered by psychologic and cultural factors for both patients and surgeons. Although patients have difficulty understanding all-cause risks incurred with surgery,^1-3 new work has also shown that surgeons have limited ability to utilize their knowledge and experience to provide a patient-specific risk assessment at the bedside. Experienced surgeons tend to overemphasize idiosyncratic factors specific to their practices;^4-6 and surgical trainees tend to systematically overestimate risks for complex surgical patients.^7-10 These risk assessment limitations may have clinical implications. For example, the costs of inappropriate therapeutic decisions have been estimated to increase total costs of care by 20%.¹¹

Appropriate risk assessment may improve decision-making, but many common risk assessment tools are typically one-dimensional. The literature is filled with nomograms and other risk-prediction tools, but patients expect holistic approaches to the uncertainty after surgery and want to understand their comprehensive risk profile.¹² Newer tools, such as spectrum-based best-case and worst \-case diagrams and all-outcome risk calculators, have been developed to address these limitations.¹³ One of the largest implementations of risk-based decision support is the American College of Surgeons’ National Surgical Quality Improvement Program’s (NSQIP’s) Risk Calculator (RC). This tool has been shown to forecast more consistent and empirically valid assessments of complications for a broad range of general surgery operations.^9,14 Enhanced use of RCs for patient communication, preoperative risk profiling, and targeted interventions may improve the care received by patients—particularly those with surgically complex disease or high-risk traits.^8,15

For a surgical trainee to master enhanced risk counseling with patients, they must understand the complications of surgery and effectively communicate with patients. The American Board of Surgery has recognized these precepts in the current Milestone Project, delineating core competencies of surgical eduaction.¹⁶ Risk calculators and other forms of decision support offer important opportunities for residents to receive real-time feedback and assistance as their knowledge and communication abilities mature during the course of their residencies.¹⁷ However, no studies have reported the effective use of all-procedure risk calculators as part of training institution practice. The purpose of this study was to identify barriers to routine use of all-procedure RCs and assess the relative importance of current data framing as a potential barrier to adoption in surgical. We hypothesized that ease of access, workflow integration, and user-specific preferences would be major barriers to increased adoption of modern RCs. Identifying such obstacles to implementation would provide a roadmap for further RC development and innovation.

Methods

Study population and recruitment

We recruited residents from four US-based general surgery residency programs, including categorical residents, designated preliminary residents, and nondesignated preliminary residents. Each residency program sent two sequential recruitment E-mails to its own residents, with responses collected via a self-directed, internet-based questionnaire (REDCap, Nashville, TN). Responses and individual completion performance were masked from program leadership at participating institutions. The Johns Hopkins Medicine Institutional Review Board (Baltimore, MD) evaluated this study design and deemed it exempt from review.

Alternative risk calculator design

We created an alternative risk calculator (ARC) with the intent of addressing implementation hurdles observed at our own institution. Specifically, we aimed to achieve the following: (1) increased ease of use, (2) enhanced prioritization of outcomes, and (3) more detailed comparability between patient-specific risks and procedure-specific base cases (described later in this report). We used an iterative, purpose-driven design methodology and serially incorporated views of study authors and informal pilot testing with medical students. Because the underlying model for the NSQIP RC is proprietary and not shared publicly,^18-20 we adopted the underlying algorithm for a previously described surgical RC that had been validated against the NSQIP RC.²¹ We constructed a Web-based user interface (shinyapps.io, RStudio, Boston, MA) with real-time data visualization based on patient-specific factors meant to simulate a customized alternative to the existing NSQIP RC. Screenshots of this user interface are included as (eText 1).

This study intended to explore the benefit of various data visualizations in addressing the needs of various populations rather than the global superiority of one variant. We intentionally avoided demonstrating that one particular data visualization approach was superior to another. Therefore, we randomly renamed the NSQIP RC and the ARC as Risk Calculator #1 or Risk Calculator #2. The reported results intentionally conceal each calculator’s identity to its assigned pseudonym.

Data collection and survey design

Part 1. Perceptions survey

All participants first completed a demographic questionnaire that included additional quantitative Likert-style and qualitative unstructured-response questions on one’s typical risk-assessment practices. Specific components of risk assessment investigated were frequency of risk counseling, decision-making aid use and familiarity, and perceived burdens for increased use of existing RCs (eText 2). We also assessed respondents’ general risk assessment knowledge as it related to common general surgery procedures and because early trainees frequently overestimate and underestimate surgical risk compared with prediction models.¹⁰

Part 2. Calculator-assisted risk prediction

After the demographic questionnaire, we instructed participants to complete two Web-based clinical vignettes related to the need for surgical counseling. We selected cholecystectomy and colectomy as the operative events to allow for a baseline familiarity across all residency years. In both activities, participants first completed a hypothetical risk assessment with a preoperative consultation with minimal patient-specific risk factors and then completed a second, high-comorbidity variant of the vignette (eText 3). We presented participants with the vignette and then asked them to use a provided RC to gather information that they would then use as part of a risk-counseling discussion with the patient. No data were collected to evaluate how participants would structure these conversations because the focus on this part of the study was one’s conceptualization of his or her risk assessment, not the accuracy of the risk estimate.

We clustered randomized participants by program, with half of the programs completing the vignette activity with the NSQIP RC first and then the ARC; the other half of participants completed the vignette activity with the ARC first and then the NSQIP RC. Participants were informed before starting the exercise that they would ultimately use two RCs with each vignette and be asked to compare the usability of each to achieve their risk counseling goals.

After performing the vignette-based task, participants completed a modified System Usability Scale (mSUS). The original System Usability Scale provides a global subjective assessment of usability with a ten-item attitude Likert scale.^22,23 We rephrased the original scale to apply it in a direct comparison context between the NSQIP RC and the ARC as described earlier in this report (modified scale in eText 4). Then, we recentered scoring for each mSUS domain on a visual analog scale to reflect a score of −50 to indicate strong preference for Risk Calculator #1, 0 being neutral between both calculators, and +50 to indicate strong preference for Risk Calculator #2.

Data analysis

We performed descriptive statistics to characterize surgical residents’ knowledge, attitudes, and practices with respect to preoperative risk counseling and associated decision aids. Qualitative responses were grouped thematically by two authors (I.L. and A.R.) with discrepancies reviewed via author group discussion. Univariate analysis comparing resident-specific factors across institution was conducted using the Fisher exact test, Student t test, Wilcoxon rank sum test, and one-way analysis of variance, as appropriate. We then performed further exploratory analysis, using simple and multivariable linear regression of mSUS scales on participants’ demographic variability, familiarity, and use of surgical RCs to identify predictors of mSUS variation. In the interest of avoiding a Simpson’s paradox, where mixed effects obscured a variable’s relevance, we intended to pursue multivariable regression regardless of statistical significance observed in univariate analysis.²⁴ We considered P < .05 statistically significant. All statistical analysis was performed using Stata IC 15.0 (StataCorp, College Station, TX).

Results

Characteristics of respondents

We invited 320 residents to participate in this simulated risk-counseling exercise, with 124 (39%) completing at least the initial questionnaire and 100 (31%) completing the questionnaire and both vignettes. A total of 57% of respondents were male, with a median age of 29 years (interquartile range: 28–32). In total, 55% of respondents were interns or junior residents. The majority (81%) of respondents had at least two or more counseling discussions per week where surgical risk was discussed. The most common category of “typical risk of any surgical complication” for the operations each resident most commonly performed was “6–25%” (56%). Of note, most respondents correctly predicted the complication risk category of routine laparoscopic cholecystectomy and laparoscopic colectomy 85% and 59% of the time, respectively. These demographic characteristics by home institution are reported in Table 1.

Table 1.

Respondents’ demographics by institution, n = 124 respondents ^*.

	Institution A	Institution B	Institution C	Institution D	P
N	47	42	20	15
Female	20 (41%)	23 (55%)	5 (25%)	6 (40%)	.160
Age (y, median, IQR)	29 (28–32)	29 (27–32)	30.5 (29.5–32.5)	29 (28–31)	.345
Training level†					.636
Intern	11 (33%)	12 (29%)	4 (20%)	4 (20%)
Junior resident (PGY 2–3)	6 (18%)	14 (33%)	6 (30%)	5 (33%)
Research resident	10 (30%)	8 (19%)	4 (20%)	1 (7%)
Senior resident (PGY 4–5)	4 (12%)	8 (19%)	6 (30%)	6 (40%)
Fellow	2 (6%)	0	0	0
Average surgical risk of your patients					.006
< 5%	12 (24%)	21 (50%)	13 (65%)	6 (40%)
6%−25%	34 (69%)	21 (50%)	7 (35%)	9 (60%)
> 25%	3 (6%)	0	0	0
Risk assessment correct‡					.143
Laparoscopic cholecystectomy	37 (76%)	37 (88%)	17 (85%)	15 (100%)
Laparoscopic colectomy	31 (63%)	22 (52%)	12 (60%)	9 (60%)	.754

IQR, interquartile range; PGY, postgraduate year.

^*Reported as Number (%), unless otherwise indicated.

^†Missing training level data for 30% of responses at Institution A attributed to elective anonymity.

^‡Correct is determined by the base procedure risk, as reported by the NSQIP Surgical Risk Calculator.

Familiarity and usage of risk-communication strategies

Participants endorsed familiarity with several risk and decision-making tools, including direct verbal communication (100%), sketch diagrams (87%), and patient education brochures (59%). In contrast, surgical residents were less familiar with the most contemporary risk-communication frameworks, such as best-worst case scenario framing (38%),¹³ case-specific risk calculators (eg, Online Society of Thoracic Surgeons Risk Calculator; 43%), and all-procedure RCs (eg, NSQIP RC; 52%). Usage also favored traditional models of risk communication, with only 26% of residents using a communication strategy other than direct verbal discussion or anatomic sketch diagrams in their most recent 20 patient encounters. Although respondents rated RCs as a “Like” or “Strongly Like” risk-communication strategy 48% of the time, RCs were reported as being used more than half the time by only 17% of respondents (Fig. 1).

Fig. 1.

Comparative proportion of respondents indicating familiarity, favorability, and utilization. Six queried risk-assessment and counseling strategies are listed along the x-axis of the bar graph. The y-axis represents the proportion of residents in agreement for each question. Familiarity (“Yes,” black), favorability (“Like” or “Strongly Like,” white), and utilization (“Use More than Half the Time” or “Almost always use,” gray) are plotted side by side for comparison.

Barriers to risk-communication practice implementation

Key factors for adoption of risk-communication practices cited by residents included clinical tradition or defaults (93%), evidence of effectiveness (89%), and familiarity (97%). When asked specifically what implementation barriers individuals or their institutions faced to more widespread utilization of all-procedure RCs, participants cited lack of electronic medical record integration or phone-based application availability and obtaining adequate patient specificity as the most significant factors (Table 2).

Table 2.

Qualitative responses assessing current barriers to practical use of all-procedure risk calculators^*.

	Number of individuals citing barrier
Lack of health information technology integration	39
“Making it more available, accessible on computers, apps, etc.”
“Need to integrate into EMR (electronic medical record)”
“Inconvenient to access it. Not readily available to me on the spot.”
Lack of specificity	18
“...hard to get the risk calculator to be very specific to the case I’m doing, and if it doesn’t match, it’s hard for me to adjust.”
“It does not have certain procedures included in its list.”
“Patients here are usually the exception and not the norm.”
Lack of familiarity	11
“Not aware of the calculator.”
“Not sure how to access it.”
Efficiency cost	11
“Not enough time in the day.”
“It is way too slow and cumbersome to enter information.”
Lack of perceived benefit	11
“I do not use it except in situations where patients are very high-risk.”
“I don’t think the exact percentage risk is all that helpful.”
Difficulty of interpreting risks and communicating to patients	8
“Patients have a limited understanding of what these percentages mean.”
“Difficult to explain meaningfulness of this data to patients.”
Lack of evidence	6
“Database possess too many limitations to effectively and accurately risk stratify”
“Not evidence-based (to communicate risk) and has proven to be an ineffective means to communicate risk.”

NSQIP RC provided in prompt as example of “all-procedure risk calculator.” Multiple responses allowed.

^*89 respondents.

Comparison of NSQIP RC with customized ARC

The mean mSUS domain scores were widely dispersed for all domains (Fig. 2), and no domain demonstrated one calculator’s superiority over the other. No difference was observed in using the two RCs for finding information for either the low-complexity (Vignette 1) or high-complexity (Vignette 2) scenarios. As seen in Fig. 2, scores were not normally distributed but followed a tri-modal pattern with the greatest frequency of preferences declared at either the two poles or closely to the indifference midline.

Fig. 2.

Scatter plot of each modified System Usability Score domain comparing the NSQIP RC to an alternative risk calculator. Each pole of the mSUS scores represents a total preference for each respective calculator, with the midpoint of the spectrum representing total indifference for the two (identification of poles intentionally masked). Mean ( orange ), −2 standard deviations ( red ) , and +2 standard deviations ( green ) scores are marked for each domain. Domain scores were averaged across a low-complexity and a high-complexity vignette.

When we performed univariable regression on several a priori-defined factors potentially affecting RC user preference (Table 1), no identifiable variable was found to be statistically significant. When results were repeated, using multivariable regression and controlling for others factors, there was still no observed statistically relevant difference among potential predictors of calculator preference.

Discussion

We sought to examine current perspectives on risk assessment practices among surgical trainees and the role of data visualization as a potential obstacle to further adoption and utilization of existing risk-assessment tools. We demonstrated that residents have favorable views of many risk assessment tools but tend to underutilize them because of perceived barriers to implementation. Furthermore, we found that alternating how data were presented to these respondents varied their preference in a nonpredictable fashion.

Surgeons often make clinical decisions in a manner inconsistent with the best available predictive tools,^4,10,25,26 and risk-assessment tools may help reconcile these differences.^6,9 Anecdotally, the implementation of risk assessment tools in surgical decision-making has been mixed, but no studies have demonstrated their use in routine practice. In our own efforts to use to an all-procedure risk-assessment calculator at our institution, early complaints from experienced surgeons, as well as surgical trainees, focused on the usability limitations of existing platforms and the lack of alignment between risk data output and patient decision-making.

These uptake and implementation hurdles are not unique to surgery. Behaviorists have recognized for decades the relationship between how data are framed and how they are interpreted.^27-30 In healthcare, patients given numerically equivalent information change their preferences when the information is presented differently.^29,31-33 The effect of framing on the decision-making of clinicians has also been studied in a limited context. Some studies focus on clinician decision-making when presented with published evidence framed in different ways. The existing literature suggests that additional training and experience are not protective against such framing influences.^34,35 For the surgeon, we believe that data visualization is a form of framing effect, and we were unable to find any study that examines the influence of data visualization on critical decision-making.

Within this context, this study reaffirmed that residents are routinely involved in the risk-counseling process with patients. Al though the number of “counseling discussions” performed by residents may sound high, pilot testing of the study found that many residents would informally discuss risks and benefits of a procedure even if already performed by a supervising attending surgeon. Given that such informal encounters may be equally as important for a patient’s decision to proceed, we included such events when considering a resident’s total counseling activity as reported in the results. We also demonstrated that many residents provided incorrect answers when asked to estimate the risks of routine general surgery procedures. One potential source of these errors is the lack of familiarity with existing tools that can support one’s own knowledge and decision-making. For example, existing risk assessment and counseling tools were only familiar to 36%−52% of respondents, depending on the instrument. These findings highlight the need for greater education on the use of decision-support tools and improved assessment of learner needs that may lead to better dissemination and utilization of existing decision aids.

This study also highlighted a gap between risk assessment tool familiarity and risk assessment practices. More than half of residents were familiar with risk assessment tools like RCs, but they were used less than 20% of the time despite residents endorsing a favorable view of these tools. Barriers to increased implementation include lack of integration in existing healthcare technologies and disrupting existing workflows. Residents found a lack of specificity, benefit, or existing evidence to endorse further utilization. These findings highlight the importance of assessing user needs to close the research-to-practice gap.^36,37 Further use of focused, mixed methodologies may assist in understanding and addressing technologic integration and how the user interacts with data. These findings reinforce that the barriers we hypothesized demonstrate an intrinsic importance for respondents as well.

We also assessed a potential barrier to the usability of commonly utilized existing risk assessment and counseling instruments. In other words, is the lack of formal risk assessment practices more limited by the lack of data or how those data are provided for clinical decision-makers? To answer this question, we compared usability of the existing NSQIP RC with the ARC, a customized calculator and alternative display format using similar predictive modeling. Not only was there no statistical superiority between the two formats, but we identified wide divergence in individuals’ preferences for data visualization. Most important, these differences could not be predicted by any demographic characteristic of the resident respondent when examined through univariable and multivariable regression on RC preference. We interpret these findings to mean that data visualization preferences are highly individualized and unpredictable. These findings support previous calls for risk calculator model transparency, interoperability, and multistakeholder engagement.^18,19 These findings also highlight the need for implementation and dissemination research to assess and better utilize evidence-based tools. Such investigations may highlight which existing risk prediction platforms would be more attractive to surgeons. These models should then be incorporated into existing workflows rather than remaining isolated components of the care process. For example, integrated clinical decision support tools already exist for quality improvement efforts, such as venous thromboembolism prophylaxis³⁸ and inpatient insulin dosing for diabetic patients.³⁹ A similar paradigm is feasible for risk assessment and counseling tools. These earlier quality improvement programs have also demonstrated that, although such electronic medical record tasks may increase work requirements of front-line surgical trainees and staff, the deleterious effects of additional work burden can be mitigated by closed-loop feedback and aligning provider incentives with patient care needs.¹⁷ We have also observed a relationship between risk assessment tool use and trainee engagement in surgical education settings, such as morbidity and mortality conference.¹⁰ Ultimately, future research is needed that accounts for the needs of relevant stakeholders (eg, residents and program directors), with any intervention accounting for relevant process and balance measures for implementation. Balancing these domains will aid in intervention dissemination and long-term sustainability.

Ultimately, such enhanced framing and data-visualization efforts represent the first of many implementation steps on the path from inception to widespread adoption of decision support devices. There has been an exceptional output of high-quality healthcare services research in recent years that created the robust and diverse risk assessment tools and surgical outcome prediction models existing today.^40-43 However, there is still a research-to-practice gap between knowing what maximizes healthcare quality and what is delivered in practice. We must not only identify the best interventions but also ensure that they are effectively delivered in clinical and community practice. This latter-stage development is the focus of dissemination and implementation research and building this knowledge base is imperative to get the best return on investment in all facets of research. One example that has worked well at our institution in a variety of settings is the Translating Evidence into Practice framework.⁴⁴ This approach focuses on identifying areas of need (ie, surgical risk assessment) and then developing quality improvement interventions, with bedside providers that address these quality needs in a way that is also compatible with daily practice concerns. Another opportunity that is ideal for the surgical education realm is a train-the-trainer model where risk assessment and counseling education is targeted to current surgeons teaching residents to help them model best practices to a rotating host of resident trainees.⁴⁵ In our experience, and as seen in this study, experimenting with various modalities such as these to enhance interoperability and workflow integration have been central to successful implementation strategies. We believe these findings support increased access to and partnerships with existing surgical risk calculators to allow individualized customization and assessment of an alternative means of utilization. Some of the most widely used currently prohibit external manipulation, which may limit their widespread use.

This study is not without its limitations. The small sample size statistically limits interpretability of comparisons between RCs. However, we explicitly did not wish to demonstrate that one specific RC was uniquely superior to another. Instead, we used these usability comparisons to identify subpopulations of respondents who were drawn to various aspects of the RC, highlighting that one monolithic format limits the attractiveness for the user population. Similarly, the overall response rate for this study was only 39% of invited resident participants. Given the constraints on residents’ time, we were not surprised that participation came in just under typically reported physician surveys (50%−60% response rates) and above public Internet-based research questionnaires (approximately 33% response rate).^46,47

In conclusion, we found in this study that many residents know about risk assessment tools but tend to underutilize them even though knowledge of risk assessment and effective patient counseling are critical components of modern surgical training. Key barriers to further use appear to be the need for enhanced electronic integration into healthcare information technology systems and clinical workflows. It is important to note that clinicians’ preference sets appear to be unpredictable and may benefit more from a customizable, bespoke approach to risk-assessment tools rather than a one-size-fits-all model. Finally, implementation of risk-assessment and counseling strategies must take into consideration these postdevelopment, pre-adoption design issues to be successful.