Abstract
Background:
This study aims to quantitatively assess use of the NSQIP surgical risk calculator (NSRC) in contemporary surgical practice and to identify barriers to use and potential interventions that might increase use.
Materials and methods:
We performed a cross-sectional study of surgeons at seven institutions. The primary outcomes were self-reported application of the calculator in general clinical practice and specific clinical scenarios as well as reported barriers to use.
Results:
In our sample of 99 surgeons (49.7% response rate), 73.7% reported use of the NSRC in the past month. Approximately half (51.9%) of respondents reported infrequent NSRC use (<20% of preoperative discussions), while 14.3% used it in ≥40% of preoperative assessments. Reported use was higher in nonelective cases (30.2% vs 11.1%) and in patients who were ≥65 years old (37.1% vs 13.0%), functionally dependent (41.2% vs 6.6%), or with surrogate consent (39.9% vs 20.4%). NSRC use was not associated with training status or years in practice. Respondents identified a lack of influence on the decision to pursue surgery as well as concerns regarding the calculator’s accuracy as barriers to use. Surgeons suggested improving integration to workflow and better education as strategies to increase NSRC use.
Conclusions:
Many surgeons reported use of the NSRC, but few used it frequently. Surgeons reported more frequent use in nonelective cases and frail patients, suggesting the calculator is of greater utility for high-risk patients. Surgeons raised concerns about perceived accuracy and suggested additional education as well as integration of the calculator into the electronic health record.
Keywords: NSQIP surgical risk calculator, High-risk patients, Risk assessment, Surgical decision-making
Introduction
Counseling patients regarding the risks and benefits of a procedural intervention is a key component of the informed consent process [1]. Many tools have been created to provide evidence-based estimates of the likelihood of certain procedural complications [2,3]. These tools have been shown to increase patient knowledge, improve doctor-patient communication, and reduce internal conflict for patients in their decision-making process [4,5]. However, they are often underutilized [6-8]. Several barriers to widespread use have been identified, including lack of integration with the electronic health record (EHR), perceived lack of benefit in individual practice, and medicolegal concerns [6,9,10].
In 2013, the National Surgical Quality Improvement Program (NSQIP) published a surgical risk calculator (SRC). The NSQIP SRC (NSRC) incorporates approximately twenty clinical factors and was designed to provide patient-specific risk estimates of postoperative complications and discharge disposition to guide both surgical decision-making and informed consent [11,12]. Since its creation, the NSRC has been validated in different patient populations and clinical situations [13-17]. This literature suggests that the accuracy of the NSRC may vary based on patient population and surgical subspecialty [18-23].
While the NRSC is a valuable resource, important knowledge gaps remain with respect to its implementation. Use of the NSRC by surgeons in preoperative surgical assessments and informed consent discussions as well as barriers to more widespread use are beginning to be explored but are still uncertain [10]. Understanding the surgeon experience with the NSRC may increase its impact. In this context, there were three objectives for the current study: first, determine how often the NSRC is used by surgeons in preoperative assessments; second, understand barriers to incorporating the NSRC into clinical practice; and third, identify strategies that may increase use of the NSRC.
Methods
We performed a mixed-methods survey study to evaluate use of the NSRC by surgeons in clinical practice. This study was reviewed by the Yale University Institutional Review Board and determined to be exempt.
Survey design
We developed a 26-item instrument (Appendix A) comprised of three parts corresponding to each of the three study objectives. Initially, the items in this instrument were written by authors SM, SA, CG, and RB. This first draft of the survey then underwent pilot testing with surgical attendings and trainees. During cognitive interviews, authors KS, TJ, LK, and JF were asked to speak their thoughts aloud as they completed the survey to ensure that the items were being interpreted as intended and captured the scientific intent of the survey item [24]. The instrument was revised iteratively during this process.
The finalized items were then formated and inputted into Qualtrics software (Qualtrics LLC, Provo, Utah). The final instrument took approximately four minutes to complete. No changes to the instrument were made after to this survey after distribution to the study participants.
Survey content
In Part 1, respondents estimated the percentage of preoperative patient encounters in the last month during which they used the NSRC. They estimated their use with specific patient populations and in specific clinical situations. Part 2 included questions with structured answer choices as well as questions that allowed for written responses. Respondents selected barriers which they had encountered in clinical practice from a provided list. They were also given the opportunity to describe any additional barriers they had encountered. Part 3 similarly contained questions with structured answer choices and others that allowed for written responses. Participants selected interventions that might increase their use of the NSRC from a provided list and offered additional through free-text responses. In addition to these objective-directed questions, participants provided demographic data.
Participants
The instrument was distributed to attending surgeons who practiced acute care surgery or colorectal surgery and to PGY4 (post-grad year) and PGY5 general surgery residents from August 2021 through September 2021. We included attending surgeons in these two specialties as they perform a wide range of operations across the spectrum of surgical complexity in both elective and nonelective settings and because they operate on patients with a range of ages, comorbidities, and physiologic compromise. We felt that this was an ideal population to capture the heterogeneity of typical surgical practice. Senior general surgery residents were included as they represented trainees who were familiar with the risks and benefits as well as the technical aspects of the operations for which they were evaluating patients.
A convenience sample of seven academic institutions was selected for distribution. Site representatives at each institution compiled distribution lists and email addresses of all surgeons practicing acute care surgery, colorectal surgery, or who were PG4 or PGY5 general surgery residents. Survey links were emailed directly to surgeons who were identified as eligible participants using Qualtrics software.
One week prior to sending the survey link, all identified surgeons received an introductory email from their respective site representative which described the study and asked for their participation. The following week, surgeons received an individualized email with a survey link. At two- and four-week intervals, those surgeons who had not completed the instrument were sent follow up emails. Final data were collected six weeks after distribution of the survey links. All responses were anonymous and we were unable to see which surgeons had responded and which hadn’t.
Statistical analysis
This project utilized both quantitative and qualitative analyses. Quantitative descriptive statistics were employed to analyze data from structured answer choices. Overall use of the NSRC was evaluated, followed by use in specific clinical situations and patient populations. We also assessed whether respondent demographics were associated with NSRC use. Paired-samples t-tests, Chi-squared tests, and Pearson correlation analyses were performed using STATA SE software (StataCorp, College Station, Texas). All tests were two-tailed with significance established at p<0.05.
Qualitative data from the written responses underwent content analysis by authors SM and RB [25]. Core elements were identified from the written responses and offered insight into shared experiences of barriers to use of the NSRC as well as ideas to increase future use.
Results
Response rate and overall NSRC use
We received 99 completed surveys from a total of 199 surveys which were distributed (response rate 49.7%). Attending surgeons were more likely to respond than residents (55.0% vs 43.2%). Acute care surgeons were more likely to respond than colorectal surgeons (62.8% vs 36.4%) (Table 1). Residents reported performing a higher proportion of elective cases than attending surgeons. The majority of both residents and attending surgeons were members of the American College of Surgeons (ACS) and reported practicing in university-based hospitals. Only one respondent did not have previous knowledge of the NSRC.
Table 1.
Demographics of respondents.
| Residents | Attending Surgeons |
|
|---|---|---|
| N | 38 | 61 |
| Response rate | 43.2% (38 of 88) | 55.0% (61 of 111) |
| Acute care surgery | – | 49 (62.8%) |
| Colorectal surgery | – | 12 (36.4%) |
| Years in practice | ||
| <5 | – | 13 (21.3%) |
| 5–9 | – | 10 (16.4%) |
| 10–14 | – | 17 (27.9%) |
| >15 | – | 18 (29.5%) |
| Median percent of cases in past year that were elective [IQR] | 70 [50,82] | 25 [12,75] |
| Member of American College of Surgeons | 29 (76.3%) | 57 (93.4%) |
| Academic affiliation of Hospital | ||
| University-based | 31 (81.6%) | 51 (83.6%) |
| University-affiliated | 3 (7.9%) | 5 (8.2%) |
| Independent | 0 | 2 (3.3%) |
IQR: Inter-quartile range
In total, 73.7% of respondents (n = 73) reported using the NSRC in the past month. Of these 73 individuals, approximately half (49.3%) reported use of the calculator in less than 20% of preoperative discussions (Fig. 1) while 15.1% of respondents reported use in ≥40% of cases. Based on the observed distribution of data, we defined reported use in >40% of cases to be regular use.
Fig. 1.
Overall use of NSRC for all respondents.
NSRC use in specific patient populations and clinical situations
Reported use of the NSRC was more frequent for older patients (≥65 years) compared to younger patients (mean percentage of cases for which NSRC was reportedly used: 37.1% vs 13.0%, p<0.001) (Fig. 2a). NSRC use was reported more frequently in nonelective than in elective cases (30.2% vs 11.1%, p<0.001), for patients who were functionally dependent before surgery vs non-dependent (41.2% vs 6.6%, p<0.001), and for cases with surrogate consent vs patient consent (39.9% vs 20.4%, p<0.001).
Fig. 2.
a. Average use of NSRC in specific patient populations and clinical situations, b. Frequency of use of NSRC in specific patient populations and clinical situations.
Regular use was reported more frequently for older patients 65 (41.4% of respondents) as compared to younger patients (7.4% of respondents) (Fig. 2b). More respondents reported regular use in nonelective cases than elective cases (31.4% vs 8.5%) and for patients who were functionally dependent before surgery compared to those who were functionally independent (56.1% vs 1.6%). Regular use was reported more frequently for patients with surrogate consent compared to patient consent (49.2% vs 23.1%).
NSRC use based on surgeon demographics
There was no difference in reported use of the NSRC between residents and attending surgeons (p = 0.49) (Fig. 3a). There was no difference based on ACS membership (p = 0.52) or based on the academic affiliation of the hospital at which the respondent operated (p = 0.12). For the attending surgeon cohort, reported NSRC use was not associated with the number of years in clinical practice (p = 0.44). Acute care surgeons reported use of the NSRC in 22.8% of cases while colorectal surgeons reported use in 12.3% of cases (p = 0.08).
Fig. 3.
a. Average use of NSRC by provider, b. Frequency of NSRC use by provider.
Similar rates of regular use were reported by attending surgeons and residents (41.4% vs 44.0%) (Fig. 3b). Regular use was also similar between acute care surgeons and colorectal surgeons (17.1% vs 12.5%).
Barriers to current NSRC use
Both attending surgeons and residents described the lack of the NRSC’s perceived impact on both the surgeon’s and patient’s decision to pursue surgery as the most frequently encountered barrier to use (Fig. 4a). The remaining barriers, including another provider completing the preoperative workup, the length of time required for data input, concern about inaccurate estimates, the accessibility of the NSRC, and use of a different surgical risk calculator were endorsed less frequently.
Fig. 4.
a. Barriers to use of NSQIP calculator, b. Interventions to improve use of NSQIP calculator.
Content analysis of the written responses about barriers to use of the NSRC identified two common concerns: its limited impact on clinical practice and its accuracy in certain patient populations. One attending surgeon explained the lack of utility for patients with limited treatment options, saying “the tool is useless for most of my patients. I find it useful only when there is a viable non-operative treatment for the patient.” A resident mentioned “I find it rare that patients want to hear what the algorithms have to say. The NSQIP calculator becomes a nice line to include in the note for medico legal reasons but I don’t think ultimately guides a surgeon or patient’s decision.”
There were also concerns about the accuracy of the NSRC in certain patient populations and clinical situations. One attending surgeon stated that the NSRC was “not an accurate predictor in tertiary center emergency patients (extreme outliers of risk) which is a large portion of my practice.” Another attending surgeon didn’t trust the NSRC in patients with cirrhosis, saying “I tried the calculator for a cirrhotic with a strangulated hernia requiring resection and the calculator said his risk was lower than the usual risk. It’s hard for me to trust that calculator.”
Strategies to increase NSRC use
Attending surgeons identified improving accuracy as the intervention most likely to increase use of the NSRC, followed by integration into the EHR and adding measures related to disease severity (Fig. 4b). Residents felt similarly about integration with the EHR and improving the accuracy of the calculator, but thought that creation of a smartphone app was most likely to increase use of the NSRC.
Two common themes emerged in the content analysis of the written responses regarding how to increase use of the NSRC: further education for surgeons on how to incorporate the calculator into their clinical practice and integration of the calculator into the EHR. Statements from two residents suggest that currently, there is a knowledge gap in how to apply the estimates provided by the NSRC in clinical practice. One resident explained that the NSRC “does not apply to goals of care focus.” while another proposed that “many cases are more straight forward and using a risk calculator will not impact a decision to go to the OR [operating room].” An attending surgeon identified this need for additional education as they suggested to “educate the surgeons how to interpret the numbers and how to convey that information to patients and families.”
Many surgeons believed that integration of the NSRC into the EHR would increase use. One attending surgeon suggested that “a trigger or reminder in the EMR [electronic medical record]” would prompt surgeons to apply the calculator in cases where they wouldn’t have thought to use it and also to remind those who forget to use it in relevant settings. Another attending surgeon explained that having the NSRC as part of the EHR would facilitate “ease of access” as compared to the current need to use additional software. A third attending surgeon proposed “an automatic fill in of the metrics that shows the calculated score and allows the surgeon to make adjustments” in order to allow for more efficient data entry.
Discussion
In this mixed-methods survey study of 99 surgeons at seven academic institutions, over 70% of surgeons reported using the NSRC at least once in the past month, yet fewer than 15% of surgeons reported use of the NSRC in ≥40% of cases. The NSRC was incorporated more frequently for patients with greater clinical uncertainty regarding outcome (nonelective procedures) as well as for those who were deemed to be at higher baseline risk (older, functionally dependent, or required a surrogate decision-maker). The major barrier to use of the NSRC was the lack of perceived importance to clinical decision-making. Potential strategies to increase the use of NSRC use included better accessibility and enhanced accuracy. Taken together, these findings suggest incorporation of the NSRC into the preoperative surgical assessment was variable and largely based on patient complexity, surgeons mistrust the NSRC to provide accurate risk stratification of outcomes, and more can be done to improve the implementation of the NSRC into clinical surgical practice.
This study indicates that the NSRC was used more frequently in the assessment of patients who were older, frail, and for whom nonelective surgery was being considered. These vulnerable patients represent a cohort for whom estimation of risk for postoperative complications can be quite difficult [26-35]. Tools like the NSRC can improve communication surrounding treatment options and expectations between patients and their surgeons [36]. They can also facilitate advance care planning for patients considering high-risk surgery [37]. The value of the NSRC for these patients is clear, in that it provides an evidence-based risk assessment that can serve as the foundation of the conversation between a patient and their surgeon about goals and expectations for their operation.
Concern about the perceived accuracy of the NSRC was emphasized by surgeons as a barrier to use. These concerns focused on the ability of the NSRC to estimate risks for patients considering nonelective surgery. While multiple studies have validated the predictive ability of the NSRC in emergency operations, [16,38-40] others have shown that the risk estimates are less accurate in the emergent setting than in the elective setting. Lubitz et al. showed that the NSRC was less accurate in predicting complications after emergency colorectal surgery than after elective colorectal surgery, [23] while Hyder et al. showed that NSRC estimates were less accurate in emergency cases with low predicted mortality than in elective cases with low predicted mortality [38]. There are reports of risk overestimations [39,41] and underestimations [42] for certain postoperative complications, depending on the clinical context. It should also be noted that the NSRC allows for surgeons to adjust the risk estimated through the ‘Surgeon Adjustment of Risks’ tool if they feel that there are important patient characteristics which the NSRC has not taken into account [43]. In light of the results of this study and the existing literature, we should consider the accuracy of the NSRC in the emergent setting to be an area for potential improvement.
A group of surgeons expressed that the NSRC was not applicable to their particular patient population. Several of these surgeons felt that if the NSRC was not being used specifically to decide whether a patient should undergo surgery, it was not useful. The NSRC can be valuable in its ability to guide both surgeons and patients as each considers surgery as a treatment option, but its utility is not limited to this one decision process. Prior research has established the role that decision aids can play in helping patients to feel more knowledgeable about their condition and treatment options, more clear about their values, and ultimately, to feel more satisfied in the decisions that they make about their care [4]. Hurley reported that two-thirds of orthopedic patients who were exposed to decision aids received treatments that were aligned with their choices [44]. Thus, the utility of the NSRC may extend beyond the decision to pursue surgery and may be helpful in facilitating the alignment of patient- and surgeon-expectations, ultimately increasing patient satisfaction with their medical decisions.
These less intuitive uses for the NSRC underscore that additional education for surgeons is vital to more widespread implementation. By offering additional educational content, the ACS can provide insight into ways that the NSRC can help ensure care is aligned with patient goals and expectations. This material can also explain how to use features like the Surgeon Adjustment of Risks to improve the risk estimates offered by the NSRC. These materials might be posted as a written document or a video module on the NSRC webpage.
Incorporation of the NSRC into the EHR emerged in our study as a strategy that would help to increase use. Creation of a smartphone application was suggested as well. The benefits of technology in the use of other decision aids is well established. Kaner et al. demonstrated a decrease in consultation times required for the electronic versions of three different decision aids [45]. Staszewska et al. reviewed the impact of integrating decision aids for seriously ill patients into the EHR, and noted improvement in several shared decision-making metrics [46]. The results of our study suggest that use of the NSRC may increase with technological integration in ways similar to other decision aids [47].
Limitations
This study has limitations. First, while the survey invitation did not mention the NSRC or other risk calculators, it did explain that the project was focused on tools used to facilitate the informed consent process. Thus, our sample may have been subject to non-response bias [48] in that those surgeons who did not respond may have used the NSRC less frequently. As a survey-based study, this study is also subject to other survey biases including attrition bias, response bias, selection bias, and volunteer bias [49]. In addition, our data were collected from seven academic institutions. Though this collection of institutions does include hospitals of varying sizes, these findings may not be fully representative of academic or community practice. We hope to broaden the scope of this project to include all types of practices, including those based in the community as well as hospitals that do not participate in NSQIP. In addition, we hope to include surgeons from specialties other than acute care surgery and colorectal surgery. These additional settings and specialties represent crucial populations of surgeons who are not well-represented in this study. Finally, we didn’t collect data from patients in this study, and as such, we were not able to comment on barriers to use or ways to increase use of the NSRC from the patient perspective.
Conclusions
Nearly 3 in 4 surgeons sampled reported using the NSRC at least once in the past month, however, fewer than 1 in 7 used it in more then 40% of cases. Surgeons were more likely to report applying the NSRC to nonelective cases and frail patients, suggesting that the calculator is thought to be of greater utility for high-risk patients. Barriers to use of the NSRC include a lack of influence on surgical decision-making and concern about inaccuracy of the predictions. Increased use of the calculator may be facilitated by integration into the EHR and additional education for surgeons on how to integrate the NSRC into clinical practice.
Supplementary Material
Funding
At the time this research was conducted, Dr. Miller’s time was funded by the Yale National Clinician Scholars Program and by Clinical and Translational Science Award number TL1 TR001864 from the National Center for Advancing Translational Science.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Footnotes
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Supplementary materials
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.sipas.2023.100173.
Disclosures/Conflicts of interest
Neither this group nor its members have any disclosures or conflicts of interest to disclose.
References
- [1].Ledercq WKG, Keulers BJ, Scheltinga MRM, Spauwen PHM, van der Wilt GJ. A review of surgical informed consent: past, present, and future. A quest to help patients make better decisions. World J Surg 2010;34(7):1406–15. 10.1007/s00268-010-0542-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [2].Knops AM, Legemate DA, Goossens A, Bossuyt PMM, Ubbink DT. Decision aids for patients facing a surgical treatment decision: a systematic review and meta-analysis. Ann Surg 2013;257(5):860–6. 10.1097/SLA.0b013e3182864fd6. [DOI] [PubMed] [Google Scholar]
- [3].Sepucha K, Atlas SJ, Chang Y, et al. Patient decision aids improve decision quality and patient experience and reduce surgical rates in routine orthopaedic care: a prospective cohort study. JBJS 2017;99(15):1253–60. 10.2106/JBJS.16.01045. [DOI] [PubMed] [Google Scholar]
- [4].Stacey D, Légaré F, Lewis K, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2017;2017(4). 10.1002/14651858.CD001431.pub5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Sacks GD, Dawes AJ, Ettner SL, et al. Impact of a risk calculator on risk perception and surgical decision making. Ann Surg 2016;264(6):889–95. 10.1097/SLA.0000000000001750. [DOI] [PubMed] [Google Scholar]
- [6].Leeds IL, Rosenblum AJ, Wise PE, et al. Eye of the beholder: risk calculators and barriers to adoption in surgical trainees. Surg U S 2018;164(5):1117–23. 10.1016/j.surg.2018.07.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Aasen: Systematic review of preoperative risk discussion… - Google Scholar. Accessed April 20, 2023. https://scholar.google.com/scholar_lookup?title=Systematic%20Review%20of%20Preoperative%20Risk%20Discussion%20in%20Practice&journal=J%20Surg%20Educ.&doi=10.1016%2Fj.jsurg.2020.02.008&volume=77&issue=4&pages=911-20&publication_year=2020&author=Aasen%2CDM&author=Wiesen%2CBM&author=Singh%2CAB.
- [8].Lambert-Kerzner A, Ford KL, Hammermeister KE, Henderson WG, Bronsert MR, Meguid RA. Assessment of attitudes towards future implementation of the “Surgical Risk Preoperative Assessment System” (SURPAS) tool: a pilot survey among patients, surgeons, and hospital administrators. Patient Saf Surg 2018;12(1):12. 10.1186/s13037-018-0159-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Bunzli S, Nelson E, Scott A, French S, Choong P, Dowsey M. Barriers and facilitators to orthopaedic surgeons’ uptake of decision aids for total knee arthroplasty: a qualitative study. BMJ Open 2017;7(11):e018614. 10.1136/bmjopen-2017-018614. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Pradhan N, Dyas AR, Bronsert MR, et al. Attitudes about use of preoperative risk assessment tools: a survey of surgeons and surgical residents in an academic health system. Patient Saf Surg 2022;16(1):13. 10.1186/s13037-022-00320-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].FAQ - ACS Risk Calculator. Accessed January 18, 2021. https://riskcalculator.facs.org/RiskCalculator/faq.html.
- [12].About - ACS Risk Calculator. Accessed January 18, 2021. https://riskcalculator.facs.org/RiskCalculator/about.html.
- [13].Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217(5). 10.1016/j.jamcollsurg.2013.07.385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [14].Cohen ME, Liu Y, Ko CY, Hall BL. An examination of american college of surgeons NSQIP surgical risk calculator accuracy. J Am Coll Surg 2017;224(5). 10.1016/j.jamcollsurg.2016.12.057. 787–795.e1. [DOI] [PubMed] [Google Scholar]
- [15].Burgess JR, Smith B, Britt R, Weireter L, Polk T. Predicting postoperative complications for acute care surgery patients using the ACS NSQIP surgical risk calculator. Am Surg 2017;83(7):733–8. 10.1177/000313481708300730. [DOI] [PubMed] [Google Scholar]
- [16].Scotton G, Del Zotto G, Bernardi L, et al. Is the ACS-NSQIP risk calculator accurate in predicting adverse postoperative outcomes in the emergency setting? An Italian single-center preliminary study. World J Surg 2020;44(11):3710–9. 10.1007/s00268-020-05705-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [17].Yap MKC, Ang KF, Gonzales-Porciuncula LA, Esposo E. Validation of the American college of surgeons risk calculator for preoperative risk stratification. Heart Asia 2018;10(2):e010993. 10.1136/heartasia-2017-010993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Kuy SR, Romero RAL. Decreasing 30-day surgical mortality in a VA Medical Center utilizing the ACS NSQIP Surgical Risk Calculator. J Surg Res 2017;215:28–33. 10.1016/j.jss.2017.03.030. [DOI] [PubMed] [Google Scholar]
- [19].Shaker S, Rivard C, Nahum R, Vogel RI, Teoh D. The American College of Surgeon’s surgical risk calculator’s ability to predict disposition in older gynecologic oncology patients undergoing laparotomy. J Geriatr Oncol 2019;10(4):618–22. 10.1016/j.jgo.2019.02.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [20].Armstrong EA, Beal EW, Lopez-Aguiar AG, et al. Evaluating the ACS-NSQIP Risk Calculator in Primary GI Neuroendocrine Tumor: results from the United States Neuroendocrine Tumor Study Group. [PMC free article] [PubMed] [Google Scholar]
- [21].Golden DL, Ata A, Kusupati V, et al. Predicting Postoperative Complications after Acute Care Surgery: how Accurate Is the ACS NSQIP Surgical Risk Calculator? [PubMed] [Google Scholar]
- [22].Margolick J, Wiseman SM. Risk of major complications following thyroidectomy and parathyroidectomy: utility of the NSQIP surgical risk calculator. Am J Surg 2018;215(5):936–41. 10.1016/j.amjsurg.2018.01.006. [DOI] [PubMed] [Google Scholar]
- [23].Lubitz AL, Chan E, Zarif D, et al. American college of surgeons NSQIP risk calculator accuracy for emergent and elective colorectal operations. J Am Coll Surg 2017;225(5):601–11. 10.1016/j.jamcollsurg.2017.07.1069. [DOI] [PubMed] [Google Scholar]
- [24].Brasel K, Haider A, Haukoos J. Practical guide to survey research. JAMA Surg 2020;155(4):351–2. 10.1001/jamasurg.2019.4401. [DOI] [PubMed] [Google Scholar]
- [25].HOLSTI OR. Content analysis for the social sciences and humanities. Read MA Addison-Wesley Content Anal 1969. Published online Accessed January 5, 2022, https://ci.nii.ac.jp/naid/10007743499/. [Google Scholar]
- [26].Bleicher RJ, Abrahamse P, Hawley ST, Katz SJ, Morrow M. The influence of age on the breast surgery decision-making process. Ann Surg Oncol 2008;15(3):854–62. 10.1245/s10434-007-9708-x. [DOI] [PubMed] [Google Scholar]
- [27].Bernstein AD, Parsonnet V. Bedside estimation of risk as an aid for decision-making in cardiac surgery. Ann Thorac Surg 2000;69(3):823–8. 10.1016/S0003-4975(99)01424-1. [DOI] [PubMed] [Google Scholar]
- [28].Richardson JD, Cocanour CS, Kern JA, et al. Perioperative risk assessment in elderly and high-risk patients1 1No competing interests declared. J Am Coll Surg 2004;199(1):133–46. 10.1016/j.jamcollsurg.2004.02.023. [DOI] [PubMed] [Google Scholar]
- [29].Zhang Y, Ma L, Wang T, et al. Protocol for evaluation of perioperative risk in patients aged over 75 years: aged patient perioperative longitudinal evaluation-multidisciplinary trial (APPLE-MDT study). BMC Geriatr 2021;21(1):14. 10.1186/s12877-020-01956-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [30].Gao S, Hendrie HC, Hall KS, Hui S. The relationships between age, sex, and the incidence of dementia and Alzheimer disease: a meta-analysis. Arch Gen Psychiatry 1998;55(9):809–15. 10.1001/archpsyc.55.9.809. [DOI] [PubMed] [Google Scholar]
- [31].Podcasy JL, Epperson CN. Considering sex and gender in Alzheimer disease and other dementias. Dialogues Clin Neurosci 2016;18(4):437–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [32].Fukuda N, Wada J, Niki M, Sugiyama Y, Mushiake H. Factors predicting mortality in emergency abdominal surgery in the elderly. World J Emerg Surg 2012;7(1):12. 10.1186/1749-7922-7-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [33].Becher RD, Wyk BV, Leo-Summers L, Desai MM, Gill TM. The incidence and cumulative risk of major surgery in older persons in the United States. Ann Surg 2021. 10.1097/SLA.0000000000005077. Published online December 16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [34].Raymont V, Bingley W, Buchanan A, et al. Prevalence of mental incapacity in medical inpatients and associated risk factors: cross-sectional study. Lancet 2004;364(9443):1421–7. 10.1016/S0140-6736(04)17224-3. [DOI] [PubMed] [Google Scholar]
- [35].Silveira MJ, Kim SYH, Langa KM. Advance directives and outcomes of surrogate decision making before death. N Engl J Med 2010;362(13):1211–8. 10.1056/NEJMsa0907901. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [36].Whelan T, Levine M, Willan A, et al. Effect of a decision aid on knowledge and treatment decision making for breast cancer surgerya randomized trial. JAMA 2004;292(4):435–41. 10.1001/jama.292.4.435. [DOI] [PubMed] [Google Scholar]
- [37].Schuster AL, Aslakson RA, Bridges JF. Creating an advance-care-planning decision aid for high-risk surgery: a qualitative study. BMC Palliat Care 2014;13(1):32. 10.1186/1472-684X-13-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [38].Hyder JA, Reznor G, Wakeam E, Nguyen LL, Lipsitz SR, Havens JM. Risk prediction accuracy differs for emergency versus elective cases in the ACS-NSQIP. Ann Surg 2016;264(6):959–65. 10.1097/SLA.0000000000001558. [DOI] [PubMed] [Google Scholar]
- [39].Long AM, Hildreth AN, Davis PT, Ur R, Badger AT, Miller PR. Evaluation of the performance of ACS NSQIP surgical risk calculator in emergency general surgery patients. Am Surg 2020;86(2):83–9. 10.1177/000313482008600214. [DOI] [PubMed] [Google Scholar]
- [40].Parkin CJ, Moritz P, Kirkland O, Glover A. What is the accuracy of the ACS-NSQIP surgical risk calculator in emergency abdominal surgery? A meta-analysis. J Surg Res 2021;268:300–7. 10.1016/j.jss.2021.07.009. [DOI] [PubMed] [Google Scholar]
- [41].Liu Y, Cohen ME, Hall BL, Ko CY, Bilimoria KY. Evaluation and enhancement of calibration in the American college of surgeons NSQIP surgical risk calculator. J Am Coll Surg 2016;223(2):231–9. 10.1016/j.jamcollsurg.2016.03.040. [DOI] [PubMed] [Google Scholar]
- [42].Kongwibulwut M, Chiang K, Lee JM, et al. Life after 90: predictors of mortality and performance of the ACS-NSQIP risk calculator in 4,724 nonagenarian patients undergoing emergency general surgery. J Trauma Acute Care Surg 2019;86(5):853–7. 10.1097/TA.0000000000002219. [DOI] [PubMed] [Google Scholar]
- [43].Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217(5). 10.1016/j.jamcollsurg.2013.07.385. 833–842.e3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [44].Hurley VB. Are patients really getting what they want? The routine implementation of decision aids for patients with hip or knee osteoarthritis in the high value healthcare collaborative and alignment between patient treatment choice and receipt. J Eval Clin Pract 2021;27(6):1207–15. 10.1111/jep.13570. [DOI] [PubMed] [Google Scholar]
- [45].Kaner E, Heaven B, Rapley T, et al. Medical communication and technology: a video-based process study of the use of decision aids in primary care consultations. BMC Med Inform Decis Mak 2007;7(1):2. 10.1186/1472-6947-7-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [46].Staszewska A, Zaki P, Lee J. Computerized decision aids for shared decision making in serious illness: systematic review. JMIR Med Inform 2017;5(4):e6405. 10.2196/medinform.6405. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [47].O’Donnell S, Cranney A, Jacobsen MJ, Graham ID, O’Connor AM, Tugwell P. Understanding and overcoming the barriers of implementing patient decision aids in clinical practice*. J Eval Clin Pract 2006;12(2):174–81. 10.1111/j.1365-2753.2006.00613.x. [DOI] [PubMed] [Google Scholar]
- [48].Berg N. Non-Response bias. Social Science Research Network; 2005. Accessed February 21, 2022, https://papers.ssrn.com/abstract=1691967. [Google Scholar]
- [49].Sedgwick P. Questionnaire surveys: sources of bias. BMJ 2013;347:f5265. 10.1136/bmj.f5265. [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.