ABSTRACT
Background
Surgical education and skill assessments are important in improving surgical skills. However, instructors' comments tend to be complex and unorganized, with varying content and categories. This study aimed to develop a natural language processing (NLP) model to automatically classify feedback comments on surgical procedures and evaluate the performance of the developed model. An NLP model was used to automatically analyze the tendency of differences in feedback comments for each surgeon's skill.
Methods
Evaluation comments provided to 1813 applicants seeking endoscopic surgical skill qualification for colorectal cases were used. As a ground truth for model training, comments were manually categorized into five competencies—tissue handling, psychomotor skills, dissection quality, exposure quality, and efficiency—and two emotional polarities—positive or negative. The competency and emotional polarity percentages were compared between applicants who did and did not qualify. An automatic feedback comment classification model based on NLP was constructed and verified. Feedback comments on surgical videos were analyzed using an NLP model.
Results
Training data comprised 5846 sentences from 175 cases. The overall accuracies of the comment analysis task were 0.75 and 0.94 for competency and emotional polarity classifications, respectively. The remaining 1526 cases were analyzed using the NLP model. The proportion of positive polarity was significantly higher in the qualified surgeon group than in the unqualified surgeon group.
Conclusions
The NLP model developed to evaluate feedback comments related to surgical skill efficiently organized feedback comments on surgical procedures and may contribute to enhanced skill evaluation and surgical education.
Keywords: colorectal neoplasms, educational status, feedback, natural language processing, surgeons
1. Introduction
Surgical skill is an important factor affecting morbidity and mortality [1, 2, 3], and surgical education and skill assessments play important roles in the improvement of surgical skill. Although many Likert‐type skill assessment tools have been created as objective measures to evaluate surgical skill [4, 5], in actual surgical practice, instructors provide verbal comments to trainees rather than using skill assessment tools. These comments often tend to be complex and unorganized and include a variety of content and categories. Thus, automatic categorization and organization of instructors' comments can make surgical education more efficient and contribute to surgical skill improvement.
Natural language processing (NLP) is a form of machine learning that allows artificial intelligence to understand text [6]. NLP has been applied in medicine, for example, to extract information about symptoms and incident reports from free descriptions in electronic medical records [7, 8]. In the field of surgery, NLP has been used to extract postoperative complications from electronic medical records [9]. Several studies have reported the use of machine learning to classify comments on surgical procedures and compare the effectiveness of rating systems based on comments [10, 11, 12].
The Japan Society for Endoscopic Surgery has launched the Endoscopic Surgical Skill Qualification System (ESSQS) system [13]. Surgery videos are submitted to the society to qualify for technical certification, and detailed feedback comments on surgical techniques, categorized by evaluation category, are returned for all videos. NLP analysis of these voluminous comments may highlight what categories are important in the technical evaluation, as well as the differences in trends between successful and unsuccessful applicants.
In this study, we aimed to develop an NLP model that can automatically classify expert feedback comments into surgical skill assessment categories and emotional polarities, including positive and negative assessments. We then evaluated the performance of the NLP model to classify feedback comments on surgical procedures. We also compared feedback comments for applicants who did and did not qualify to identify differences between these surgeon groups.
2. Materials and Methods
2.1. ESSQS
In Japan, the ESSQS was established to maintain and improve the quality of laparoscopic surgery [13]. The ESSQS score is based on a Likert‐type evaluation, with 70 out of 100 points being the threshold for passing. Surgeons only receive accreditation if their technique is good enough for them to act as instructors.
In addition to being scored on the basis of ESSQS scoring criteria for unedited surgical videos submitted for review, the applicants also receive an approximately 200‐word comment from the evaluator who reviews the video. While the score is a summative evaluation that ensures objectivity, the comments represent specific feedback that the experts would like to convey to each applicant and are the focus of the present study (Appendix S1).
2.2. Dataset
The dataset consisted of evaluation comments in the ESSQS for colorectal cases from 2016 to 2020. On the basis of the conventions of machine learning, 175/1701 cases (approximately 10% of all ESSQS evaluations in the dataset) were used to generate the annotation data in this study (Figure 1). The cases for annotation were randomly selected from the passed and failed cohorts such that the ratio of passed to failed applicants was similar to that of the actual examination results; these cases were used as training data for the construction of the comment analysis model. The quantity and quality of evaluation comments varied among the cases. We attempted to minimize these effects by extracting a sufficient quantity of cases. All cases, except those used for manual annotation, were analyzed using the comment analysis model.
FIGURE 1.
Extraction of annotation data. The process for extraction of the data used for manual annotation is presented. ESSQS, Endoscopic Surgical Skill Qualification System.
2.3. Manual Annotation
We used the extracted 10% of the comments mentioned above (Figure 1). First, the comments were separated by periods or commas, and any sentence with clearly separate meanings connected by a conjunction was divided into two sentences. The titles of the surgical processes and evaluation items used by the evaluators were excluded. This process was conducted to create individual sentences that maintained a certain degree of consistency for evaluation.
Five practicing surgeons classified the evaluation sentences based on the following five competencies: tissue handling, psychomotor skills, dissection quality, exposure quality, and efficiency. Surgeons included two gastroenterology surgeons and three gynecology surgeons. According to Igaki et al. among the five surgical competencies, tissue handling, psychomotor skills, and efficiency are basic surgical skills, whereas dissection and exposure qualities represent advanced surgical skills [14]. The surgeons categorized organ‐specific technical assessments, technical assessments of surgeon autonomy that did not fit into the five competencies, and non‐technical evaluations into the “Others” category (Appendix S2). When multiple competencies were included in a sentence, each was counted separately. The “Others” category was excluded from the analysis.
The same five surgeons classified the emotional polarity of each sentence as positive, negative, or neutral. In terms of negativity, this classification refers to technical aspects that need improvement, and it should not be interpreted as emotional disapproval. It is identified for educational purposes. If a sentence included elements with both positive and negative polarities, it was classified as the latter. Sentences classified as showing neutral polarity were excluded from the analysis. The prepared training data were double‐checked by another surgeon and corrected if necessary. Discrepancies in the data were resolved by consensus among all five surgeons (Appendix S3).
After manual annotation, the trend of competency classification between the groups of passed and failed applicants was compared, along with the positive: negative ratio in the emotional polarity classification. Odds ratios (ORs) were calculated for the ratio of positive comments received by the passed and failed applicants.
2.4. Natural Language Processing Model Development
Using deep learning with the annotation data from the previous section, two comment analysis models were constructed: a five‐category competency model and an emotional polarity model. Pre‐processing involved normalization of the half‐width and full‐width characters. As with the manual annotation, the titles of the surgical processes and evaluation items used by the evaluator were excluded. We used an NLP model called a bidirectional encoder representation from transformers (BERT; Google Inc., Mountain View, CA, USA), a transformer‐based language model pre‐trained on a large amount of text data. It has been reported to be capable of understanding context by considering the words before and after a word and can be fine‐tuned for various NLP tasks such as question answering and sentiment analysis [15]. Details of the model hyperparameters are shown in Appendix S4.
2.5. Evaluation Metrics
The accuracy of the comment analysis model was evaluated using 10‐fold cross‐validation and was presented in terms of recall, precision, F1 score, and overall accuracy. The formula for each indicator is as follows:
where TP, FP, TN, and FN represent true‐positive, false‐positive, true‐negative, and false‐negative results, respectively.
Accuracy was calculated to compute the percentage of agreement between the surgeon's classification and the model's classification.
2.6. Natural Language Processing‐Based Surgical Skill Assessment
The comment analysis model was used to categorize the comments provided in the ESSQS based on competency and emotional polarity.
Trends in competency classification and the percentages of negative and positive polarities were compared between the passed and failed applicants. We excluded the sentences categorized as “Others” from the competency model and those categorized as neutral from the emotional polarity model.
2.7. Statistical Analysis
Statistical analyses were performed using the chi‐square test and Fisher's exact test, and statistical significance was set at p < 0.05. All statistical analyses were performed using EZR, version 1.41 (Saitama Medical Center, Jichi Medical University, Japan), a modified version of the R commander that includes statistical functions frequently used in biostatistics [16].
ORs were calculated for the ratio of positive comments received by the passed and failed applicants.
2.8. Ethics
The study protocol was reviewed and approved by the Ethics Committee of the National Cancer Center Hospital East (registration no. 2020‐315). This study conformed to the provisions of the Declaration of Helsinki, 1964 (revised in Brazil in 2013). The Institutional Review Board approved the use of JSES review results for this study. Consent was obtained from both applicants and reviewers for the secondary use of the ESSQS data and comments.
3. Results
3.1. Number of Data Points After Category Classification
Among the 1813 applicants from 2016 to 2020 who underwent ESSQS evaluations for the colorectal region, the median pass rate was 28% (508/1813). Consent for secondary use was obtained from the applicants and reviewers in 1701 cases. For the ESSQS pass rate, the ratio of passed to failed applicants was 2:5. Of these, 175 cases (approximately 10%) were selected (passed: 50, failed: 125). Two reviewers evaluated and commented on each case, yielding 350 comments (Figure 1). These 350 comments contained 5846 sentences, of which 4876 sentences were included after removing duplicates. The number of sentences per case was 30.8 ± 14.2 for all cases, 25.9 ± 11.3 for the passed group, and 32.7 ± 14.7 for the failed group. The data were classified into five competencies and Others, resulting in 6211 data points. The number of data points for Others was 1685; thus, the number of data points for the five competencies was 4526 (Appendix S5).
3.2. Comparison of Five Competencies Between Passed and Failed Applicants
A comparison of the pass/fail percentages for each of the five competencies using manual annotation is shown in Table 1. No significant difference was observed in the percentages of competency classifications between passed and failed applicants for any item.
TABLE 1.
Percentages of the five competency categories classified by pass/fail status in manual annotation and in the comment analysis model.
| Competencies | Manual annotation | Comment analysis model | ||
|---|---|---|---|---|
| Pass | Fail | Pass | Fail | |
| Total number of comments | 1287 (100.0%) | 3239 (100.0%) | 12 053 (100.0%) | 33 056 (100.0%) |
| Tissue handling | 305 (23.7%) | 808 (24.9%) | 3118 (25.9%) | 9176 (27.8%) |
| Psychomotor skill | 70 (5.4%) | 118 (3.6%) | 453 (3.8%) | 1044 (3.2%) |
| Dissection quality | 433 (33.6%) | 997 (30.8%) | 4060 (33.7%) | 10 634 (32.2%) |
| Exposure quality | 383 (29.8%) | 1069 (33.0%) | 3607 (29.9%) | 10 211 (30.9%) |
| Efficiency | 96 (7.5%) | 247 (7.6%) | 815 (6.8%) | 1991 (6.0%) |
3.3. Comparison of Emotional Polarity Between Passed and Failed Applicants
The emotional polarity‐classification ratios and ORs by manual annotation are shown in Table 2. In terms of the percentage of positive comments for passed and failed applicants, the percentage of positive comments was significantly higher for those who passed (p < 0.005). The percentage of neutral comments was very small for all competencies.
TABLE 2.
Percentages of emotional polarity classification and odds ratios per competency classified by pass/fail status in manual annotation.
| Manual annotation a | |||||||
|---|---|---|---|---|---|---|---|
| Pass | Fail | ||||||
| Co | Positive | Negative | Neutral | Positive | Negative | Neutral | OR |
| Total | 297/1038 (28.6) |
722/1038 (69.5) |
19/1038 (1.9) | 190/3169 (5.9) | 2945/3169 (91.9) | 34/3169 (2.2) | 6.4 |
| TH | 44 (16.5) | 217 (81.6) | 5 (1.9) | 22 (2.7) | 803 (96.9) | 4 (0.5) | 7.4 |
| PS | 21 (33.3) | 40 (63.5) | 2 (3.2) | 17 (12.5) | 119 (87.5) | 0 (0) | 3.6 |
| DQ | 130 (35.4) | 227 (61.9) | 10 (2.7) | 86 (8.8) | 878 (89.7) | 15 (1.5) | 5.8 |
| EQ | 76 (24.4) | 235 (75.3) | 1 (0.3) | 61 (6.1) | 929 (92.6) | 13 (1.3) | 4.9 |
| Ef | 26 (32.5) | 53 (66.3) | 1 (1.3) | 4 (1.8) | 216 (97.3) | 2 (0.9) | 26.4 |
Abbreviations: Co, competency; DQ, dissection quality; Ef, efficiency; EQ, exposure quality; OR, odds ratio; PS, psychomotor skill; TH, tissue handling.
Odds ratios for each of the five competencies in the manual annotation results are presented, and the emotional polarity‐classification ratios of the comments categorized as pass/fail are compared.
3.4. Performance of the Natural Language Processing Model
The training data consisted of 175 cases, 350 comments, and 4876 sentences, while the test data consisted of 491 sentences. The results of the 10‐fold cross‐validation for assessment of the accuracy of the competency and emotional polarity classification models are listed in Table 3. Overall accuracy was 0.75 for the categories and 0.94 for the emotional polarity classification.
TABLE 3.
Accuracy of the competency and emotional polarity classification model.
| Categories | Precision | Recall | F1 score | Overall accuracy |
|---|---|---|---|---|
| Competency | ||||
| Tissue handling | 0.84 | 0.75 | 0.79 | 0.75 |
| Psychomotor skill | 0.57 | 0.67 | 0.61 | |
| Dissection quality | 0.78 | 0.72 | 0.74 | |
| Exposure quality | 0.83 | 0.80 | 0.81 | |
| Efficiency | 0.48 | 0.59 | 0.53 | |
| Others | 0.83 | 0.69 | 0.75 | |
| Emotional polarity | ||||
| Positive | 0.92 | 0.85 | 0.87 | 0.94 |
| Negative | 0.98 | 0.95 | 0.96 | |
| Neutral | 0.68 | 0.77 | 0.72 | |
3.5. Natural Language Processing‐Based Comment Classification
After excluding the 175 cases used for training, the remaining 1526 cases included 3052 comments, which were classified by the comment analysis model. These data were obtained from ESSQS evaluations performed by 73 raters. The total number of sentences used in this analysis was 45 109 (passed: 12 053, failed: 33 056).
The percentage comparison between passed/failed applicants for each of the five competencies using the proposed model is shown in Table 1. Similar to manual annotation, no significant difference was observed in the percentage of competencies between passed and failed applicants.
The emotional polarity‐classification ratios and ORs are shown in Table 4. The highest odds ratio was for efficiency; as with manual annotation, the percentage of positive comments was higher in the group of passed applicants, and the two groups showed a significant difference (p < 0.005).
TABLE 4.
Percentages of emotional polarity classification and odds ratios per competency classified by pass/fail status in the comment analysis model.
| Comment analysis model a | |||||||
|---|---|---|---|---|---|---|---|
| Pass | Fail | ||||||
| Co | Positive | Negative | Neutral | Positive | Negative | Neutral | OR |
| Total | 2607/12 041 (21.7) | 9274/12 041 (77) |
160/12 041 (0.6) |
2389/33 039 (7.2) | 30 291/33 039 (91.7) | 359/33 039 (1.1) | 3.6 |
| TH | 406 (13) | 2660 (85.3) | 50 (1.6) | 407 (4.4) | 8657 (94.3) | 107 (1.2) | 3.2 |
| PS | 106 (23.4) | 339 (74.8) | 8 (1.8) | 154 (14.8) | 867 (83) | 21 (2.0) | 1.8 |
| DQ | 1188 (29.3) | 2802 (69) | 63 (1.6) | 1093 (10.3) | 9380 (88.2) | 153 (1.4) | 3.6 |
| EQ | 672 (18.6) | 2902 (80.5) | 30 (0.8) | 569 (5.6) | 9577 (93.8) | 63 (0.6) | 3.9 |
| Ef | 235 (28.8) | 571 (70.1) | 9 (1.1) | 166 (8.3) | 1810 (90.9) | 15 (0.8) | 4.5 |
Abbreviations: Co, competency; DQ, dissection quality; Ef, efficiency; EQ, exposure quality; OR, odds ratio; PS, psychomotor skill; TH, tissue handling.
The comment analysis model classified 5 years of comments into each of the five competencies and compared the emotional polarity classification ratios between passed and failed applicants to show the odds ratios.
4. Discussion
In this study, we developed an NLP‐based model that categorizes feedback comments along with surgical skill assessment tools for surgical procedures in surgical videos according to competence and emotional polarity. This model allows us to automatically classify comments into categories and further quantify emotional polarity. There was no difference in the distribution of competence between qualified and non‐qualified participants in the comments, but there was a significant difference in the percentages of emotional polarity. These findings represent a fundamental step toward enhancing the efficiency of surgical learning and education. Quantifying comments using NLP is a first step toward providing a new method of surgical skill assessment that does not require a manual scoring process according to the assessment tool's criteria. By building a model using some of the training data, we were able to automatically analyze a large amount of verbal data for surgical skill evaluation comments.
Surgical skill evaluation is essential for surgical education. Mainstream surgical skill evaluation is currently based on Likert scales [4], error scores [5], and other evaluation approaches, such as video‐based assessments [17]. In the actual clinical setting of surgery, evaluation of surgical skills using evaluation sheets is rarely performed because of the time required for evaluation and the complexity of the evaluation method, and most education is based on intraoperative and postoperative comments and debriefing. In this situation, systematic and automatic categorization of feedback comments will also provide feedback for the evaluator, not only the practitioner, thereby improving the quality of the evaluator.
In the previous study, Ötleş et al. tested the accuracy of a rough automatic classification of surgical training feedback comments as specific or general, encouraging or not encouraging, and corrective or not corrective using NLP. The highest mean accuracy was 0.64; however, they mentioned the small sample size and the limitations of NLP's performance [10]. We set up a more specific and simpler comment classification based on the skills meta‐competencies, wherein conventional surgical skill evaluation tools were decomposed and reintegrated by surgical skill and classified with clear definitions to minimize duplication [14]. Thus, it is clear and useful for feedback on surgical techniques. The NLP accuracy also achieved an overall accuracy of 0.75 for the categories and 0.94 for the emotional polarity. The comment analysis model created based on this categorization enabled a clearer surgical skill evaluation, and the accuracy of the comment analysis model achieved a certain level of precision because relatively short single sentences were used as the training data and targets of analysis. We believe that automatically dividing advice comments by the five competencies will make it easier for practitioners to know which competency they are being advised on and contribute to learning efficiency.
For emotional polarity classification, the percentage of positive comments was significantly higher in those who passed than in those who failed. However, negative comments accounted for 9274/12041 (77% of the total comments), even among those who passed (Table 4). This could be attributed to the fact that the error‐checking method tended to highlight areas for improvement rather than praising aspects that were being performed correctly. Observational clinical human reliability analysis is a representative error‐checking tool for technical evaluation [18], and assessments using this tool have shown a relationship between the number of errors and the surgeon's level of proficiency. People tend to do more checking for errors than finding the good when evaluating. Thus, the difference between those who passed and those who failed is in the lower percentage of negative comments. The biggest difference between those who passed and those who failed seems to be the efficiency of the surgery. Basic technical skills, such as tissue handling or psychomotor skills, can be improved through repetitive practice and external guidance, but efficiency reflects deeper cognitive and strategic abilities. Efficient surgery requires planning, prediction, and an understanding of the overall surgical process, and these skills cannot be acquired passively but must be cultivated through active thinking and experience. This may explain why certified practitioners demonstrate a more structured and intentional surgical approach, resulting in fewer negative comments regarding inefficiency. Furthermore, the qualification criteria of the ESSQS emphasize not only technical safety but also the ability to teach others. Efficiency is closely linked to reproducibility and the clarity of instruction, making it a critical differentiator in surgical evaluation. On the other hand, while non‐qualified participants may carefully execute each step, the absence of a consistent surgical strategy leads to feedback focused on redundant maneuvers, unnecessary time loss, and inefficiency. These results suggest that improving efficiency requires not only technical skills but also conceptual understanding, and that promoting strategic thinking in surgical training may contribute to improved certification outcomes.
In addition to these findings, the NLP model offers several practical insights and educational benefits. First, trainees can better understand their own weaknesses by tracking which competency areas attract negative feedback, enabling more targeted and personalized training. Second, by visualizing trends in competency‐ and emotion‐based comment distributions, evaluators can identify which competencies are most frequently addressed and how feedback sentiment changes over time. Third, the model allows institutions to analyze the feedback styles of individual evaluators, thereby promoting more balanced and constructive evaluation practices. Fourth, by applying the model longitudinally, surgical educators can track skill development over time, supporting curriculum planning and trainee progress monitoring. Finally, large‐scale analysis of feedback data may help refine surgical training programs and evaluation criteria, contributing to data‐driven improvements in surgical education.
This study had several limitations. First, this study is at a proof‐of‐concept stage and has not yet been linked to clinical outcomes. Although it may become a fundamental technology to improve the quality and efficiency of surgical education, the feasibility of skill assessment has not been demonstrated at this stage. Currently, our model streamlines the interpretation of existing feedback but does not reduce the evaluator's burden. Second, all the comments used in this study were feedback comments based on ESSQS surgical skill evaluation criteria on intraoperative videos collected from Japanese facilities. Therefore, validation using comments from international video datasets is desirable. Speech recognition technology is required to classify the evaluator's comments in real time using a comment analysis model to apply to intraoperative surgical education. Constructing a model that can automatically classify intraoperative verbal comments is a challenge that needs to be addressed in a future study.
5. Conclusions
We constructed an NLP‐based model that automatically classified expert comments on the basis of surgical skill evaluation competencies and emotional polarity. The model can be divided into five competencies and two polarities with relatively high accuracy, making it easy to understand the differences in the tendency of feedback comments for each surgical skill. In the future, this technology may be used for automatic skill evaluation and may be applied to developing more constructive and efficient surgical education.
Author Contributions
Shusaku Iwai: writing – original draft, data curation. Shin Takenaka: project administration, writing – original draft, methodology. Daichi Kitaguchi: writing – review and editing. Hiroki Matsuzaki: software, formal analysis. Kei Nakajima: data curation, investigation. Yuya Takabe: investigation. Nobuyoshi Takeshita: supervision. Masaaki Ito: conceptualization, project administration.
Disclosure
The authors have nothing to report.
Ethics Statement
The study protocol was reviewed and approved by the Ethics Committee of the National Cancer Center Hospital East (registration no. 2020–315). This study conformed to the provisions of the Declaration of Helsinki, 1964 (revised in Brazil in 2013). The Institutional Review Board approved the use of JSES review results for this study.
Consent
Consent was obtained from both applicants and reviewers for the secondary use of the ESSQS data and comments.
Conflicts of Interest
The authors declare no conflicts of interest.
Supporting information
Appendix S1: All comments for one candidate.
Appendix S2: Definitions of the five skill meta‐competencies (partially modified).
Appendix S3: Example comment.
Appendix S4: Hyperparameters of BERT.
Appendix S5: Extraction of annotation data.
Acknowledgments
We gratefully acknowledge the work of the past and present members of our laboratory. We would like to express our deep gratitude to the Japanese Society of Endoscopic Surgery for providing the data used in this study. We acknowledge Editage for providing English language editing. This research was supported by the Japan Agency for Medical Research and Development (AMED) under grant number JP24vk0124007h0002.
Funding: This work was supported by Japan Agency for Medical Research and Development (AMED) under (Grant JP21vk0124001).
We categorized evaluators' comments on surgical skill competency into five competencies and two polarities and used this data to train a natural language processing (NLP) model to evaluate surgical skills using feedback comments. This model seems feasible to evaluate trainees' surgical skills. This technology could improve the efficiency of surgical education.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author, [Masaaki Ito], upon reasonable request.
References
- 1. Aoyama S., Inoue Y., Ohki T., Itabashi M., and Yamamoto M., “Usefulness of the Endoscopic Surgical Skill Qualification System in Laparoscopic Colorectal Surgery: Short‐Term Outcomes: A Single‐Center and Retrospective Analysis,” BMC Surgery 19, no. 1 (2019): 90, 10.1186/s12893-019-0528-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Birkmeyer J. D., Finks J. F., O'Reilly A., et al., “Surgical Skill and Complication Rates After Bariatric Surgery,” New England Journal of Medicine 369, no. 15 (2013): 1434–1442, 10.1056/NEJMsa1300625. [DOI] [PubMed] [Google Scholar]
- 3. Stulberg J. J., Huang R., Kreutzer L., et al., “Association Between Surgeon Technical Skills and Patient Outcomes,” JAMA Surgery 155, no. 10 (2020): 960–968, 10.1001/jamasurg.2020.3007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Martin J. A., Regehr G., Reznick R., et al., “Objective Structured Assessment of Technical Skill (OSATS) for Surgical Residents,” British Journal of Surgery 84, no. 2 (1997): 273–278, 10.1046/j.1365-2168.1997.02502.x. [DOI] [PubMed] [Google Scholar]
- 5. Vassiliou M. C., Feldman L. S., Andrew C. G., et al., “A Global Assessment Tool for Evaluation of Intraoperative Laparoscopic Skills,” American Journal of Surgery 190, no. 1 (2005): 107–113, 10.1016/j.amjsurg.2005.04.004. [DOI] [PubMed] [Google Scholar]
- 6. Koleck T. A., Dreisbach C., Bourne P. E., and Bakken S., “Natural Language Processing of Symptoms Documented in Free‐Text Narratives of Electronic Health Records: A Systematic Review,” Journal of the American Medical Informatics Association 26, no. 4 (2019): 364–379, 10.1093/jamia/ocy173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Young I. J. B., Luz S., and Lone N., “A Systematic Review of Natural Language Processing for Classification Tasks in the Field of Incident Reporting and Adverse Event Analysis,” International Journal of Medical Informatics 132 (2019): 103971, 10.1016/j.ijmedinf.2019.103971. [DOI] [PubMed] [Google Scholar]
- 8. Chary M., Parikh S., Manini A. F., Boyer E. W., and Radeos M., “A Review of Natural Language Processing in Medical Education,” Western Journal of Emergency Medicine 20, no. 1 (2019): 78–86, 10.5811/westjem.2018.11.39725. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Mellia J. A., Basta M. N., Toyoda Y., et al., “Natural Language Processing in Surgery: A Systematic Review and Meta‐Analysis,” Annals of Surgery 273, no. 5 (2021): 900–908, 10.1097/SLA.0000000000004419. [DOI] [PubMed] [Google Scholar]
- 10. Ötleş E., Kendrick D. E., Solano Q. P., et al., “Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies,” Academic Medicine 96, no. 10 (2021): 1457–1460, 10.1097/ACM.0000000000004153. [DOI] [PubMed] [Google Scholar]
- 11. Ahle S. L., Eskender M., Schuller M., et al., “The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace‐Based Assessments,” Annals of Surgery 275, no. 3 (2022): 617–620, 10.1097/SLA.0000000000003907. [DOI] [PubMed] [Google Scholar]
- 12. Stahl C. C., Jung S. A., Rosser A. A., et al., “Natural Language Processing and Entrustable Professional Activity Text Feedback in Surgery: A Machine Learning Model of Resident Autonomy,” American Journal of Surgery 221, no. 3 (2021): 369–375, 10.1016/j.amjsurg.2020.11.044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Kimura T., Mori T., Konishi F., and Kitajima M., “Endoscopic Surgical Skill Qualification System in Japan: Five Years of Experience in the Gastrointestinal Field,” Asian Journal of Endoscopic Surgery 3, no. 2 (2010): 66–70, 10.1111/j.1758-5910.2009.00034.x. [DOI] [Google Scholar]
- 14. Igaki T., Takenaka S., Watanabe Y., et al., “Universal Meta‐Competencies of Operative Performances: A Literature Review and Qualitative Synthesis,” Surgical Endoscopy 37, no. 1 (2023): 835–845, 10.1007/s00464-022-09573-4. [DOI] [PubMed] [Google Scholar]
- 15. Devlin J., Chang M. W., Lee K., and Toutanova K., “BERT: Pre‐Training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Association for Computational Linguistics, 2019), 4171–4186, 10.18653/v1/N19-1423. [DOI] [Google Scholar]
- 16. Kanda Y., “Investigation of the Freely Available Easy‐To‐Use Software ‘EZR’ for Medical Statistics,” Bone Marrow Transplantation 48, no. 3 (2013): 452–458, 10.1038/bmt.2012.244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Tang B. and Cuschieri A., “Objective Assessment of Surgical Operative Performance by Observational Clinical Human Reliability Analysis (OCHRA): A Systematic Review,” Surgical Endoscopy 34, no. 4 (2020): 1492–1508, 10.1007/s00464-019-07365-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Aggarwal R., Moorthy K., and Darzi A., “Laparoscopic Skills Training and Assessment,” British Journal of Surgery 91, no. 11 (2004): 1549–1558, 10.1002/bjs.4816. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix S1: All comments for one candidate.
Appendix S2: Definitions of the five skill meta‐competencies (partially modified).
Appendix S3: Example comment.
Appendix S4: Hyperparameters of BERT.
Appendix S5: Extraction of annotation data.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author, [Masaaki Ito], upon reasonable request.