Abstract
Background.
The American College of Surgeons Commission on Cancer’s (CoC) new operative standards for breast cancer, melanoma, and colon cancer surgeries will require that surgeons provide synoptic documentation of essential oncologic elements within operative reports. Prior to designing and implementing an electronic tool to support synoptic reporting, we evaluated current documentation practices at our institution to understand baseline concordance with these standards.
Methods.
Applicable procedures performed between 1 January 2018 and 31 December 2018 were included. Two independent reviewers evaluated sequential operative notes, up to a total of 100 notes, for documentation of required elements. Complete concordance (CC) was defined as explicit documentation of all required CoC elements. Mean percentage CC and surgeon-specific CC were calculated for each procedure. Interrater reliability was assessed via Cohen’s kappa statistic.
Results.
For sentinel lymph node biopsy, mean CC was 66% (n = 100), with surgeon-specific CC ranging from 6 to 100%, and for axillary dissection, mean CC was 12% (n = 89) and surgeon-specific CC ranged from 0 to 47%. The single surgeon performing melanoma wide local excision had a mean CC of 98% (n = 100). For colon resections, mean CC was 69% (n = 96) and surgeon-specific CC ranged from 39 to 94%. Kappa scores were 0.77, 0.78, −0.15, and 0.78, respectively.
Conclusions.
We identified heterogeneity in current documentation practices. In our cohort, rates of baseline concordance varied across surgeons and procedures. Currently, documentation elements are interspersed within the operative report, posing challenges to chart abstraction with resulting imperfect interrater reliability. This presents an exciting opportunity to innovate and improve compliance by introducing an electronic synoptic documentation tool.
The goal of evidence-based care is to improve patient outcomes. In surgical oncology, adherence to evidence-based standards is associated with improved perioperative morbidity and long-term survival benefits.1,2 These standard practices rely on documentation as a vehicle for communication between collaborating teams of multidisciplinary clinicians across longitudinal episodes of oncologic care. Operative documentation, in particular, conveys crucial information about the conduct and elements of oncologic procedures, which contribute to prognostication, treatment planning, surveillance, and other clinical decision making. Documentation also serves as a record for assessing compliance with standards of care.
While free-text narrative documentation has been a common style for clinical and operative documentation, templated synoptic operative reports have become increasingly prevalent. Their use has been shown to capture operative data more completely and consistently,3 and to increase the reliability of documentation of critical elements in several cancer procedures, such as for colon and rectal cancer.4–7 Using synoptic operative report templates also improves documentation of quality indicators and increases interrater reliability.7 Looking to other fields of medicine, synoptic reports in breast and rectal cancer pathology improved the description of critical elements and communication of relevant data.8,9 This evidence supports the growing consensus on the utility of synoptic reporting in surgical practice, documentation, and quality assessment and improvement.5,6,10–12
Further affirming the importance of consistent operative reporting, in 2019 the American College of Surgeons Commission on Cancer (CoC) released their Optimal Resources for Cancer Care: 2020 Standards manual,13 which outlines specific intraoperative elements for several oncologic procedures, including sentinel node biopsy and axillary dissection for breast cancers of epithelial origin, wide local excision for primary cutaneous melanoma, and curative resection for colon cancer.14 The new standards will require that these essential elements be documented in a synoptic ‘element’ and ‘response’ format. While the required elements must at least be included in the text of the operative note in synoptic format, the CoC and its affiliate, the Cancer Surgery Standards Program, have developed comprehensive synoptic operative reporting templates for implementation in the electronic health record (EHR) or through third-party vendors. Institutions can also opt to create their own electronic tool. By 2022, centers desiring ongoing CoC accreditation will be required to submit their plan for achieving compliance with the standards, and by 2023, compliance rates of ≥70% will be required for accreditation.15
As our institution prepared to adapt our local practices to adhere to CoC guidelines, we sought to assess our current operative documentation practices. To that end, we performed a retrospective review of operative notes for the applicable procedures to determine baseline rates of concordance with the new standards as a comparison for future performance. These data will inform the creation and implementation of an electronic synoptic operative reporting tool within our health system.
METHODS
Setting
This work was conducted at the University of Pennsylvania Health System, a quaternary academic health system that houses the Abramson Cancer Center, a CoC-accredited cancer center. We focused on operations performed at the three Philadelphia hospitals: The Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Penn Presbyterian Medical Center. All three hospitals utilize PennChart, the University of Pennsylvania Heath System’s installation of Epic (Epic Systems Corporation, Verona, WI, USA), as the integrated inpatient, outpatient, and perioperative EHR. Perioperative activities and documentation are supported by the Epic OpTime module.
Commission on Cancer Standards
Standards 5.3–5.6 in the Optimal Resources for Cancer Care: 2020 Standards manual specify the patient criteria, measures of compliance, elements, and responses required for synoptic operative reporting (electronic supplementary Tables 1 and 2).13 Our study was conducted prior to the February 2021 update to the standards and therefore evaluated concordance based on the language of the original version of the 2020 standards published in 2019 (electronic supplementary Table 1). Standard 5.3, for breast cancer sentinel node biopsy, required that surgeons document on five elements: the substrate utilized for sentinel node biopsy (in either the non-neoadjuvant or neoadjuvant setting) and whether all abnormal nodes (colored, radioactive, palpably suspicious, or clipped) were removed. Standard 5.4, for breast cancer axillary dissection, contained four elements: description of the boundaries of resection, whether the long thoracic and thoracodorsal nerves were preserved, whether intercostobrachial nerves were preserved if identified, and whether level III nodes were removed; standard 5.5, for primary cutaneous melanoma wide local excision, contained three elements: Breslow thickness, clinical excision margin, and whether dissection was carried down to the fascia; and standard 5.6, for colon cancer curative resection, evaluated two elements: tumor location and extent of lymphovascular resection.
Study Cohort and Data Collection
Operative procedures to which CoC standards will apply were identified for inclusion (Fig. 1). A 12-month period (1 January 2018 to 31 December 2018) was reviewed to allow for a minimum of 2 years of follow-up and enable future correlation with institutional tumor registry data for survival outcomes. Procedures were identified by surgeon and Current Procedural Terminology (CPT) codes in Epic OpTime. Procedure-specific CPT codes included sentinel node biopsy (38525, 38792); axillary dissection (38745, 38740); melanoma (11600–04, 11606, 11620–24, 11626, 11640–44, 11646, 17311–15); and colon resection (44140–41, 44143–47, 44150–51, 44155–58, 44160, 44204–08, 44210, 44213).
FIG. 1.
Study methods schematic. Surgeries performed at the Hospital of the University of Pennsylvania, Pennsylvania Hospital, and Penn Presbyterian Hospital from 1 January 2018 to 31 December 2018 were reviewed. From these, search criteria were further refined by surgeon and CPT code as described in the schematic. CPT Current Procedural Terminology
Operative reports for cases outside the scope of the CoC criteria (e.g. colon resections for non-cancer pathologies such as diverticulitis) were excluded from review. The remaining reports were processed sequentially until 100 had been reviewed (sentinel node biopsy, wide local excision, colon resection) or the full 2018 procedure list was exhausted (axillary dissection; n = 89). The individuals reviewing operative reports were patient-facing providers, including one surgical resident, one trained clinical research coordinator, and one medical student, all with >2 years of experience with chart abstraction. Group training was performed at study initiation to standardize abstraction and familiarize the reviewers with the required elements.
Two independent reviewers manually evaluated the entire body of each final operative note to identify explicit documentation of each required element and mode of dictation (verbal telephone or EHR template). Reviewers then determined complete concordance (CC) for each individual operative note. For a note to have CC, it had to explicitly document all elements required by the relevant CoC standard as specified in the original version of the 2020 standards published in 2019. Therefore, CC would suggest that the reports would be compliant with the standards, after they go into effect.
Study data were collected using REDCap electronic data capture tools hosted at the University of Pennsylvania.16,17 This project was reviewed and determined to qualify as quality improvement by the University of Pennsylvania’s Institutional Review Board.
Statistical Analyses
Statistical analysis was performed. The concordance scores were averaged between the two reviewers. Mean percentage concordance scores were calculated for each required operative element, and overall mean percentage CC and surgeon-specific percentage CC were calculated for each procedure. Of note, when calculating CC metrics for colon resections, we excluded four reports as per the original CoC Standards document, which allowed for exclusion in situations such as ‘metastatic disease, poor functional status, advanced age, or significant comorbidities.’ The reasons for exclusion of the four reports were: reoperation for recurrent malignancy, malignant large bowel obstruction in the setting of metastatic disease, and two operations for endoscopically unresectable polyps (including one reoperation for a polyp in the setting of previously resected colon cancer).
Interrater reliability was determined using Cohen’s kappa statistic (possible range −1 to +1). The degree of agreement between observers was categorized according to established conventions (kappa:<0 = disagreement; 0–0.20 = none; 0.21–0.39 = minimal; 0.40–0.59 = weak; 0.60–0.79 = moderate; 0.80–0.90 = strong; and >0.90 = almost perfect).18 The association between dictation mode and CC was compared using Pearson’s Chi-square test. Data analyses were performed using R (The R Foundation for Statistical Computing, Vienna, Austria; 2016).
RESULTS
Standard 5.3: Breast Sentinel Node Biopsy
For sentinel node biopsy, 100 reports by five surgeons were reviewed. Dictation mode was verbal telephone dictation for one surgeon and EHR template for four surgeons. Overall mean CC was 66%, with surgeon-specific CC ranging from 6–100% (Fig. 2a). All surgeons consistently reported the substrate used (dye, radiotracer, or clips) (100%). Nearly all operative reports explicitly documented the removal of palpably suspicious (overall: 98.5%; surgeon-specific: 95–100%) or clipped (overall: 99%; surgeon-specific: 95–100%) nodes. However, documentation of the removal of all colored/dye-filled (overall: 67%; surgeon-specific: 8–100%) or radioactive nodes (overall: 69%; surgeon-specific: 6–100%) was less consistent. Cohen’s kappa statistic of 0.77 demonstrated ‘moderate’ agreement between reviewers.
FIG. 2.
Surgeon-specific percentage complete concordance for (a) breast cancer sentinel node biopsy; (b) breast cancer axillary dissection; (c) primary cutaneous melanoma wide local excision; and (d) curative colon cancer resection. Bar shading denotes whether surgeon dictation mode was verbal telephone or EHR template. Note that for panel (b), the dictation mode for the four surgeons with 0% CC was verbal telephone for one surgeon and EHR templates for three surgeons. EHR electronic health record, CC complete concordance
Standard 5.4: Breast Axillary Dissection
For axillary dissection, 89 reports by seven surgeons were reviewed. Dictation mode was verbal telephone for one surgeon and EHR template for six surgeons. Overall mean CC was 12%, with surgeon-specific CC ranging from 0–47% (Fig. 2b). Almost all surgeons consistently documented preservation of the long thoracic and thoracodorsal nerves (overall: 98%; surgeon-specific: 96–100%). A majority appropriately documented the boundaries of resection (overall: 87%; surgeon-specific: 8–100%) and explicitly documented attempts to spare the intercostobrachial nerve (overall: 78%; surgeon-specific: 0–100%); however, there was marked variation between surgeons. The most common missing element was the status of level III nodes, with explicit reporting of this data in only 30% of operative notes (surgeon-specific: 0–100%). Cohen’s kappa statistic of 0.78 demonstrated ‘moderate’ agreement between reviewers.
Standard 5.5: Wide Local Excision for Primary Cutaneous Melanoma
Only one surgeon performed wide local excision for primary cutaneous melanoma at our institution during the period reviewed (Fig. 2c). This surgeon used verbal telephone dictation. Across 100 reviewed operative reports, explicit documentation of required elements was nearly complete and resulted in almost perfect mean CC (98%). One reviewer found one note to be non-concordant due to failure to specify Breslow thickness, while the other reviewer found three notes to be non-concordant due to failure to document whether dissection was carried down to the fascia. A negative kappa statistic of −0.15 was consistent with ‘slight disagreement’ between observers.18 However, there was 96% agreement in the concordance rating between the two reviewers, with no record rated non-concordant by both reviewers. The Kappa statistic may have therefore been confounded by the class imbalance from the very low number of non-concordant notes and may have decreased utility in this scenario.
Standard 5.6: Colon Resection
For curative colon resection, 100 reports by five surgeons were reviewed. Dictation mode was verbal telephone for four surgeons and EHR template for one surgeon. After four reports were excluded as allowed by the CoC standard, the overall mean CC for the remaining 96 notes was 69%, with surgeon-specific concordance ranging from 39–94% (Fig. 2d). While all reviewed reports recorded tumor location, the 31% of notes that were non-concordant failed to explicitly document the extent of lymphovascular resection. A kappa statistic of 0.78 demonstrated ‘moderate’ agreement between reviewers.
Dictation Mode
Across the four surgery types, six surgeons used verbal telephone dictation to generate 222 reports, with a mean CC of 81.5%, and seven surgeons used EHR templates to generate 163 reports, with a mean CC of 36.8%. The association between dictation mode and rate of CC was assessed for the three surgical procedures (sentinel node biopsy, axillary dissection, colon resection) where both telephone and EHR dictation were employed. In analysis by individual surgical procedure, verbal telephone dictation was significantly associated with CC only for sentinel node biopsy operative reports (Table 1). For axillary dissection and colon resection, the dictation mode was not associated with CC. A range of low and high concordance rates were observed among surgeons using both dictation modes (Fig. 2).
TABLE 1.
Association between dictation mode and complete concordance. Rates of concordance were assessed for the three procedures (sentinel node biopsy, axillary dissection, and curative colon resection) for which both dictation modes were employed. Association was determined using Pearson’s Chi-square test
| Variable | Concordant [n (%)] | Nonconcordant [n (%)] | n | Chi-square statistic (df) | P-value | |
|---|---|---|---|---|---|---|
|
| ||||||
| Standard 5.3: Sentinel Node Biopsy | ||||||
| Dictation mode | Verbal telephone | 23.5 (97.9) | 0.5 (2.1) | 24 | 12.53 (1) | 0.0004 |
| EHR template | 42.5 (55.9) | 33.5 (44.1) | 76 | |||
| Standard 5.4: Axillary Dissection | ||||||
| Dictation mode | Verbal telephone | 0.0 (0.0) | 12.0 (100.0) | 12 | 0.7762 (1) | 0.3783 |
| EHR template | 10.5 (13.6) | 66.5 (86.4) | 77 | |||
| Standard 5.6: Colon Resection | ||||||
| Dictation mode | Verbal telephone | 59.5 (69.2) | 26.5 (30.8) | 86 | 3.11e-30 (1) | 1 |
| EHR template | 7.0 (70.0) | 3.0 (30.0) | 10 | |||
df degrees of freedom, EHR electronic health record.
DISCUSSION
The goal of this study was to establish a baseline for operative reporting in anticipation of the implementation of the CoC operative standards. We identified substantial heterogeneity in current surgical documentation practices. We also found imperfect interrater reliability resulting from retrospective manual data extraction. One of the strengths of this study is that it evaluates multiple surgeons of differing subspecialties across several procedure types and dictation modes. Our findings demonstrate that heterogeneity and variability in documentation practices and concordance with standards is not limited to one particular surgical specialty, surgical procedure, or mode of dictation. These findings suggest that current documentation practices imperfectly capture significant oncologic data and are subject to variable interpretation.
Our findings corroborate the literature demonstrating variability in completeness of operative reporting.19 Specifically, we found overall mean CC to range from 12% in breast axillary dissection to 98% in melanoma wide local excision. Previous studies, using a different definition for completeness, have reported 14–100% completeness for non-standardized operative reports for laparoscopic cholecystectomy.3 In another work assessing operative report completeness based on the National Accreditation Program for Rectal Cancer’s key elements, narrative operative reports were found to have only 39% completeness.4 Both of these studies demonstrated improved completeness of reporting after the introduction of synoptic operative reports, with the former also noting improved interrater reliability.
Traditional methods of narrative operative reporting pose several barriers to easy assessment of the quality of operative practices.10 Specifically, manual review of narrative operative reports is time-intensive and is subject to reviewer interpretation.3 Additionally, as our study and others have found, a traditional narrative operative report may incompletely document important information about the conduct of surgical procedures.20 Structured synoptic reports, in which data are recorded in an ‘element’ and ‘response’ binary format, may overcome some of these challenges of traditional operative reporting. Interestingly, we found that telephone dictation was actually associated with higher rates of concordance with documentation standards than EHR templates for sentinel node biopsy, but not for axillary dissection or colon resection. These data may reflect surgeon-specific factors, as each surgeon utilized one dictation mode. Alternatively, the use of personal EHR templates that are not designed according to the forthcoming CoC standards may lead to consistent non-concordance.
While the CoC has developed comprehensive synoptic operative reporting templates for implementation directly in the EMR or through third-party vendors, institutions may also opt to create their own tools following the CoC synoptic reporting structure.13 In considering tool design, we can leverage EHRs methods for recording discrete data from templated text within documentation.21 Thus, beyond mere textual recording of elements in synoptic format, the introduction of these electronic tools enables prospective capture of CoC data elements along with other data of interest. Saving the data in a discrete format also facilitates automated report generation for compliance analyses.22 In addition to supporting compliance, these data can be repurposed for other uses, such as for research, quality improvement initiatives, practice dashboards, and surgeon self-assessment.23,24 The potential downstream yield arising from the initial structured documentation effort may provide further incentives for surgeons to adopt these electronic tools. Optimizing acceptance and adoption of these valuable tools will also depend on a user-centered design that incorporates surgeon perspectives, preferences, and concerns.5,6,10 Future investigations will include designing and piloting these tools, with a focus on prioritizing usability and integration into surgeon workflows, and subsequently studying the impact on documentation compliance.
This study has several limitations. First, as a retrospective review, we were unable to conclude whether the reported rates of non-concordance reflect failure to perform or failure to report the required critical steps. Second, this assessment was performed at a single urban academic institution and therefore may not be generalizable to all practice settings. Our results suggest that the adoption of synoptic format with discrete data elements may ensure compliance with emerging standards. One of the challenges in the use of operative elements as quality metrics in cancer care is that this may prove challenging in community healthcare centers and other settings where resources to adopt these changes may not be as readily available. Our review does capture both telephone-dictated and EHR template-generated operative notes, thus demonstrating the limitations of the documentation strategies most commonly utilized in both academic and community practices. Third, we report imperfect interrater reliability across all reviewed procedures, which could introduce inaccuracy in the final reported concordance rates. While an important limitation to this study, this variation—along with the labor- and time-intensive nature of manual human abstraction—also highlights the real challenges of reliable data extraction from natural language text, further emphasizing the potential benefits of prospective data capture using synoptic operative reporting templates. Fourth, the reviewers in our study were patient-facing providers with experience in chart abstraction rather than formally trained cancer registrars. Since operative reports are viewed and used most frequently by clinical providers without specific cancer registrar training, we feel that this approach most closely approximated real-world conditions and makes our results maximally generalizable. However, future studies could explore whether formally trained cancer registrars might demonstrate different results, or identify specific critical elements that are likely to be overlooked.
Finally, we must consider that documentation templates and tools alone may not be sufficient for improving care standardization and adherence with best practices, with the ultimate goal of improving patient outcomes. Such tools rely on timely revision of guidelines to reflect the current state of scientific knowledge and updated standards.25,26 Furthermore, written materials and other educational media and programs are important for effective dissemination of best practices. Moreover, implementing electronic synoptic operative reporting tools that prospectively record synoptic data will enable further secondary uses of these data to support compliance—from automated report cards to real-time dashboards. These will bring us ever closer to supporting surgeon self-assessment and adoption of standards in real-time, thus harnessing the potential EHR tools and data for quality improvement and research.
CONCLUSIONS
Our assessment of baseline reporting practices for several surgical oncology procedures demonstrates the significant limitations of both standard operative notes and standard data abstraction methods in cancer surgery. We demonstrate heterogeneity in current documentation practices with varying rates of observed surgeon concordance with the new CoC operative standards. Imperfect interrater reliability highlights challenges to chart abstraction resulting from required data being interspersed within standard operative reports. Therefore, we identify significant potential value in the adoption of synoptic operative reporting templates, modeled after the upcoming CoC standards, to facilitate operative reporting compliance. This presents an exciting opportunity to introduce an innovative electronic synoptic documentation tool that will support compliance while also enabling prospective data capture and more robust secondary uses of these data.
Supplementary Material
Supplementary Information The online version contains supplementary material available at https://doi.org/10.1245/s10434-021-10766-9.
FUNDING
Heather Wachtel received financial support from the National Center for Advancing Translational Sciences of the National Institutes of Health, KL2-TR001879.
Footnotes
DISCLOSURE Jacqueline M. Soegaard Ballester, Kristin E. Goodsell, Jae P. Ermer, Giorgos C. Karakousis, John T. Miura, Nicole M. Saur, Najjia N. Mahmoud, Ari Brooks, Julia C. Tchou, Peter E. Gabriel, Lawrence N. Shulman, and Heather Wachtel have no related commercial conflicts of interests to disclose.
REFERENCES
- 1.Shulman LN, Browner AE, Palis BE, et al. Compliance with cancer quality measures over time and their association with survival outcomes: the Commission on Cancer’s experience with the quality measure requiring at least 12 regional lymph nodes to be removed and analyzed with colon cancer resections. Ann Surg Oncol. 2019;26(6):1613–21. 10.1235/s10434-019-07323-w. [DOI] [PubMed] [Google Scholar]
- 2.Zhao B, Tsai C, Hunt KK, Blair SL. Adherence to surgical and oncologic standards improves survival in breast cancer patients. J Surg Oncol. 2019;120(2):148–59. 10.1002/jso.25506. [DOI] [PubMed] [Google Scholar]
- 3.Harvey A, Zhang H, Nixon J, Brown CJ. Comparison of data extraction from standardized versus traditional narrative operative reports for database-related research and quality control. Surgery. 2007;141(6):708–14. 10.1016/j.surg.2007.01.022. [DOI] [PubMed] [Google Scholar]
- 4.Kanters AE, Vu JV, Schuman AD, et al. Completeness of operative reports for rectal cancer surgery. Am J Surg. 2020;220(1):165–9. 10.1016/j.amjsurg.2019.09.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bidwell SS, Merrell SB, Poles G, Morris AM. Implementation of a synoptic operative report for rectal cancer. Dis Colon Rectum. 2020;63(2):190–9. 10.1097/DCR.0000000000001518. [DOI] [PubMed] [Google Scholar]
- 6.Park J, Pillarisetty VG, Brennan MF, et al. Electronic synoptic operative reporting: assessing the reliability and completeness of synoptic reports for pancreatic resection. J Am Coll Surg. 2010;211(3):308–15. 10.1016/j.jamcollsurg.2010.05.008. [DOI] [PubMed] [Google Scholar]
- 7.Maniar RL, Hochman DJ, Wirtzfeld DA, et al. Documentation of quality of care data for colon cancer surgery: comparison of synoptic and dictated operative reports. Ann Surg Oncol. 2014;21(11):3592–7. 10.1245/s10434-014-3741-3. [DOI] [PubMed] [Google Scholar]
- 8.King S, Dimech M, Johnstone S. Structured pathology reporting improves the macroscopic assessment of rectal tumour resection specimens. Pathology. 2016;48(4):349–52. 10.1016/j.pathol.2016.03.003. [DOI] [PubMed] [Google Scholar]
- 9.Sluijter CE, van Lonkhuijzen LRCW, HJ van Slooten, Nagtegaal ID, Overbeek LIH. The effects of implementing synoptic pathology reporting in cancer diagnosis: a systematic review. Virchows Archiv. 2016;468(6):639–49. 10.1007/s00428-016-1935-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Stogryn S, Hardy KM, Abou-Setta AM, Clouston KM, Metcalfe J, Vergis AS. Advancement in the quality of operative documentation: a systematic review and meta-analysis of synoptic versus narrative operative reporting. Am J Surg. 2019;218(3):624–30. 10.1016/j.amjsurg.2019.05.003. [DOI] [PubMed] [Google Scholar]
- 11.Eryigit Ö, van de Graaf FW, Lange JF. A systematic review on the synoptic operative report versus the narrative operative report in surgery. World J Surg. 2019;43(9):2175–85. 10.1007/s00268-019-05017-8. [DOI] [PubMed] [Google Scholar]
- 12.Temple WJ, Francis WP, Tamano E, Dabbs K, Mack LA, Fields A. Synoptic surgical reporting for breast cancer surgery: an innovation in knowledge translation. Am J Surg. 2010;199(6):770–5. 10.1016/j.amjsurg.2009.07.037. [DOI] [PubMed] [Google Scholar]
- 13.American College of Surgeons. Optimal Resources for Cancer Care: 2020 Standards and Resources. Available at: https://www.facs.org/Quality-Programs/Cancer/CoC/standards/2020. Accessed 25 Feb 2021. [Google Scholar]
- 14.American College of Surgeons. Operative Standards for Cancer Surgery. Available at: https://www.facs.org/Quality-Programs/Cancer/acs-crp/oscs. Accessed 25 Feb 2021. [Google Scholar]
- 15.American College of Surgeons. Operative Standards Implementation. Available at: https://www.facs.org/Quality-Programs/Cancer/CoC/standards/2020/operative-standards/implementation. Accessed 25 Feb 2021. [Google Scholar]
- 16.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. 10.1016/j.jbi.2019.103208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276–82. 10.11613/bm.2012.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Seng SS, Chang JH, Yoo J, et al. Reporting and abstracting variability in technical standards for breast cancer operations. J Surg Res. 2020;253(909):79–85. 10.1016/j.jss.2020.03.041. [DOI] [PubMed] [Google Scholar]
- 20.Eng JL, Baliski CR, McGahan C, Cai E. Completeness of breast cancer operative reports in a community care setting. Breast. 2017;35:91–7. 10.1016/j.breast.2017.06.042. [DOI] [PubMed] [Google Scholar]
- 21.Fuller S PS1–45: Get smart: finding structured data using SmartTools in clarity. Clin Med Res. 2014;12(1–2):95–95. 10.3121/cmr.2014.1250.ps1-45. [DOI] [Google Scholar]
- 22.Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13(6):395–405. 10.1038/nrg3208. [DOI] [PubMed] [Google Scholar]
- 23.Payne TH. The electronic health record as a catalyst for quality improvement in patient care. Heart. 2016;102(22):1782–7. 10.1136/heartjnl-2015-308724. [DOI] [PubMed] [Google Scholar]
- 24.Zhao J, Forsythe R, Langerman A, Melton GB, Schneider DF, Jackson GP. The value of the surgeon informatician. J Surg Res. 2020;252:264–71. 10.1016/j.jss.2020.04.003. [DOI] [PubMed] [Google Scholar]
- 25.Martínez García L, Sanabria AJ, García Alvarez E, et al. The validity of recommendations from clinical guidelines: a survival analysis. CMAJ. 2014;186(16):1211–9. 10.1503/cmaj.140547. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Clark E, Donovan EF, Schoettker P. From outdated to updated, keeping clinical guidelines valid. Int J Qual Health Care. 2006;18(3):165–6. 10.1093/intqhc/mzl007. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.