Applications of Natural Language Processing in Otolaryngology: A Scoping Review

ABSTRACT

Objective

To review the current literature on the applications of natural language processing (NLP) within the field of otolaryngology.

Data Sources

MEDLINE, EMBASE, SCOPUS, Cochrane Library, Web of Science, and CINAHL.

Methods

The preferred reporting Items for systematic reviews and meta‐analyzes extension for scoping reviews checklist was followed. Databases were searched from the date of inception up to Dec 26, 2023. Original articles on the application of language‐based models to otolaryngology patient care and research, regardless of publication date, were included. The studies were classified under the 2011 Oxford CEBM levels of evidence.

Results

One‐hundred sixty‐six papers with a median publication year of 2024 (range 1982, 2024) were included. Sixty‐one percent (102/166) of studies used ChatGPT and were published in 2023 or 2024. Sixty studies used NLP for clinical education and decision support, 42 for patient education, 14 for electronic medical record improvement, 5 for triaging, 4 for trainee education, 4 for patient monitoring, 3 for telemedicine, and 1 for medical translation. For research, 37 studies used NLP for extraction, classification, or analysis of data, 17 for thematic analysis, 5 for evaluating scientific reporting, and 4 for manuscript preparation.

Conclusion

The role of NLP in otolaryngology is evolving, with ChatGPT passing OHNS board simulations, though its clinical application requires improvement. NLP shows potential in patient education and post‐treatment monitoring. NLP is effective at extracting data from unstructured or large data sets. There is limited research on NLP in trainee education and administrative tasks. Guidelines for NLP use in research are critical.

One‐hundred sixty‐six papers with a median publication year of 2024 (range 1982, 2024) were included. Sixty‐one percent (102/166) of studies used ChatGPT and were published in 2023 or 2024. Sixty studies used NLP for clinical education and decision support, 42 for patient education, 14 for electronic medical record improvement, 5 for triaging, 4 for trainee education, 4 for patient monitoring, 3 for telemedicine, and 1 for medical translation. For research, 37 studies used NLP for extraction, classification, or analysis of data, 17 for thematic analysis, 5 for evaluating scientific reporting, and 4 for manuscript preparation.

1. Introduction

Artificial Intelligence (AI) has grown rapidly in healthcare with applications ranging from diagnostics, prognostics, to treatment planning [1, 2, 3]. Amidst the broad spectrum of AI applications, Natural Language Processing (NLP) has been relatively uncharted in healthcare until recently. NLP is a specialized branch of AI that focuses on understanding, interpreting, and generating human language in an intelligent and meaningful way.

Large language models (LLMs) are novel algorithms based on the transformer architecture proposed in 2017 that enable training on massive datasets [4]. LLMs have recently been used to power NLP applications, leading to significant improvements. Different LLMs optimize NLP for different tasks. For example, Generative Pre‐Trained Transformer (GPT) is optimized for generating coherent and relevant text, and Bidirectional Encoder Representations from Transformers (BERT) for understanding and interpreting language. The release of ChatGPT (OpenAI, November 2022) marked a significant NLP milestone, achieving the fastest growth in consumer app history by reaching 100 million users in 2 months [5]. NLP is still only emerging in healthcare and promises a wide range of future applications.

Otolaryngologists must often balance patient‐centered care with research and teaching [6, 7]. As clinicians, they must navigate triaging, ensuring timely diagnosis and treatment, patient education, documentation, monitoring, and follow‐up. Scholarly activities include grant writing, data extraction and analysis, as well as manuscript preparation. As educators, otolaryngologists give lectures and provide student evaluations on top of clinical supervision and teaching. NLP technology emerges as a promising adjunct to enhance efficiency across these domains to aid otolaryngologists in fulfilling their varied responsibilities.

Interest in the application of NLP to otolaryngology has surged since the introduction of large language models [8]. This study aims to review the current literature, shedding light on the existing applications of NLP within otolaryngology.

2. Materials and Methods

This study explores the application of NLP tools in otolaryngology for enhancing traditional workflows. Reporting adhered to the Preferred Reporting Items for Systematic Reviews and Meta‐analyzes Extension for Scoping Reviews (PRISMA‐ScR) statement, and the methodology was published on Open Science Framework (DOI: 10.17605/OSF.IO/QR26G) [9].

2.1. Search Strategy

The MEDLINE, EMBASE, SCOPUS, Cochrane Library, Web of Science, and CINAHL databases were searched on December 26th, 2023 using a search strategy peer‐reviewed by a medical librarian under the Peer Review of Electronic Search Strategies criteria [10]. The medical subject headings “otolaryngology”, “natural language processing”, “otorhinolaryngology”, “neurotology”, and “head and neck cancer” with relevant keywords were used (Supporting Information S1).

2.2. Study Inclusion and Exclusion Criteria

Original full‐text articles utilizing NLP for otolaryngology or head and neck (H&N) cancer patient care and research, regardless of publication date, were included. Reviews on the topic of NLP in otolaryngology as well as basic science, text and opinion, and supplement articles were excluded. Irrelevant, non‐English articles, and articles where the full text could not be found were also excluded.

2.3. Selection and Appraisal of Studies

Search results were uploaded to the Covidence (Melbourne, Australia) platform, which automatically removed duplicates. Two reviewers (NB, BM) screened studies using Covidence, with disagreements resolved by AA (Figure 1). NB and BM then performed a full‐text review. Discrepancies were settled through consensus by all three reviewers.

FIGURE 1.

Summary flowchart of the outcomes of the identification and screening process. Flow chart format has been adapted from the PRISMA 2020 flow diagram for new systematic reviews [11]. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

2.4. Data Extraction

Data were extracted in duplicate from the included studies by NB and BM. Discrepancies were settled by consensus. The extraction template included: Study type, level of evidence, country, objective, AI used, application, subspecialty, sample size, methodology, and outcomes.

2.5. Level of Evidence

The 2011 Oxford Centre for Evidence‐Based Medicine's levels of evidence were assigned to each study as an indicator of the potential bias in each study [12].

3. Results

A total of 3224 studies were initially identified (Figure 1). After duplicate removal (n = 2219) and abstract screening of 1005 studies, 274 studies were included in the full‐text review. One hundred sixty‐six studies were ultimately included [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178].

Study characteristics are summarized in Table S1. The majority of studies were performance evaluations (n = 118), case series (n = 14), or cross‐sectional (n = 9). The median publication year was 2024 (range 1982, 2024) (Figure 2). The majority of studies were from the USA (n = 71), China (n = 16), Italy (n = 12), or Germany (n = 9) (Table 1). H&N oncology was the most common subspecialty (n = 54), followed by rhinology (n = 15) and neurotology (n = 9) (Table 2). Many studies did not focus on a subspecialty of OHNS (n = 78).

FIGURE 2.

Histogram of the year of publication distribution (binwidth = 1 year).

TABLE 1.

Number of studies by country.

Country	Number of studies
USA	71
China	16
Italy	12
Germany	9
France	7
Turkey	7
UK	6
Australia	4
Belgium	4
Canada	4
Poland	4
Denmark	3
South Korea	3
India	2
Japan	2
Spain	2
Brazil	1
Finland	1
Israel	1
Jordan	1
Lebanon	1
Netherlands	1
Portugal	1
Saudi Arabia	1
Taiwan	1
Thailand	1

TABLE 2.

Number of studies investigating the respective subspecialty.

Subspecialty	Number of studies
Not specified/OHNS overall	81
Head and neck cancer	54
Rhinology	15
Neurotology	9
Pediatrics	5
Laryngology	5
Facial plastics	3

One hundred twenty studies utilized NLP in a clinical context, while 46 applied it for research purposes. Figure 3 shows the breakdown of the specific applications. Table S2 contains a brief description of study methods and outcomes.

FIGURE 3.

Number of studies by specific applications of NLP for Clinical and Research Applications. A single study may contain multiple applications. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

No meta‐analysis was performed in this study due to the heterogeneity of the methodologies of the included studies.

3.1. Large Language Models

Sixty‐one percent (101/166) of studies included the use of ChatGPT, all of which were published in 2023 or 2024. ChatGPT‐4 was used in 33% (55/166) of studies, ChatGPT‐3.5 in 31% (52/166), ChatGPT‐3 in 1% (2/166), and 4% (6/166) did not specify the version. Twenty‐five studies utilized other large language models [28, 36, 47, 49, 52, 60, 61, 82, 84, 88, 90, 91, 97, 107, 108, 109, 115, 118, 124, 144, 146, 148, 155, 158, 168]. BERT (Google Research, CA, USA) [60, 61, 88, 91, 107, 109, 115, 124, 144, 155] and Gemini (Google DeepMind, London, UK) [28, 36, 49, 52, 84, 97, 146, 148, 158, 168] were each included in 10 studies, Copilot (Microsoft, Redmond, WA, USA) in 5 [82, 97, 108, 146, 158], and Claude (Anthropic, San Francisco, CA, USA) [47, 49] and Llama (Meta, Menlo Park, CA, USA) in 2 [90, 118].

3.2. Level of Evidence

Level of evidence was found as follows: Level 1 (n = 1), Level 2 (n = 2), Level 3 (n = 2), Level 4 (n = 25), and Level 5 (n = 1). The majority of studies were not classifiable as per Oxford level of evidence (n = 134).

3.3. Clinical Uses

3.3.1. Patient Education and Monitoring

There were 42 studies investigating NLP tools for patient education [31, 34, 35, 39, 40, 42, 44, 51, 56, 62, 63, 64, 70, 76, 77, 94, 95, 97, 98, 99, 100, 101, 104, 106, 111, 113, 122, 123, 126, 128, 146, 153, 157, 159, 160, 161, 165, 168, 169, 174, 176, 178]. Accuracy of responses to patient questions was evaluated in 34 studies, ranging from 69.2% to 98.3% over 13 studies [31, 40, 62, 63, 64, 94, 98, 100, 101, 123, 128, 161, 174], while 21 studies used scales [34, 39, 42, 44, 70, 76, 77, 99, 104, 106, 111, 113, 122, 146, 157, 159, 165, 168, 169, 176, 178]. Four studies utilized the DISCERN instrument to characterize the quality of answers, ranging from 35.83 to 50.7 [44, 99, 104, 159]. There were three studies that assessed the safety of LLM responses, finding that 0%–5% of answers were potentially harmful [34, 168, 174].

Nineteen studies investigated the readability of LLM responses [31, 44, 51, 76, 95, 97, 98, 99, 100, 104, 106, 113, 123, 126, 146, 159, 160, 165, 169]. Of these, 16 studies [31, 44, 95, 97, 99, 100, 104, 106, 113, 123, 126, 146, 159, 160, 169] graded readability of LLM outputs using the Flesch–Kincaid Reading Ease Scale (FRES) and 15 studies [31, 44, 76, 95, 97, 98, 99, 100, 104, 106, 126, 146, 159, 160, 169] used the Flesch–Kincaid Grade Level (FKGL) score ranging from 25 to 77.27 and 5.3 to 14.2, respectively. Five studies utilized the Simple Measure of Gobbledygook score to evaluate readability, ranging from 9.9–30.9 [95, 99, 104, 123, 159]. Eight studies prompted ChatGPT to give responses with increased readability, finding improvements of 13.3–30.5 in FRES and 1.5–5.2 in FKGL scores [51, 76, 95, 98, 113, 126, 159, 160]. Three of these studies [95, 126, 160] assessed ChatGPT's ability to adapt patient materials to Grade 5–6 reading levels, finding FKGL score reductions of 3.2–5.23 points, resulting in final FKGL scores between 5.3 and 7.6, meeting their goal in two studies [126, 160].

Three studies compared LLMs, finding that ChatGPT outperformed Gemini (Bard) and Copilot (Bing) in accuracy and quality of responses, but Copilot had the best readability [97, 146, 168]. Bard was found to be less safe than ChatGPT [168].

Four studies applied NLP to outpatient monitoring [21, 56, 162, 163]. Ma et al. 2021 showed that automated health chats for H&N cancer patients undergoing radiotherapy captured some adverse effects more effectively than physicians. Two studies utilized NLP to analyze sentiment and emotions surrounding treatment for patients with H&N cancers [162, 163].

3.3.2. Electronic Medical Record Improvement

There were 14 studies where NLP was used in applications that could improve electronic medical records (EMR) and documentation [48, 58, 83, 87, 96, 105, 117, 121, 135, 136, 137, 150, 151, 167]. Four studies evaluated NLP models for documentation tasks [83, 96, 121, 150]. Two studies automatically generated operative notes from intraoperatively spoken keywords, demonstrating time savings but mixed accuracy [83, 121]. Four studies implemented NLP for EMR information retrieval tasks such as retrieving diagnoses, past treatments, and post‐operative complications, demonstrating mixed results for accuracy [48, 87, 117, 137]. Another four studies evaluated accuracy for classifying diagnostic imaging findings, ranging from 83% to 96.5% [58, 135, 136, 167].

3.3.3. Triaging and Patient Classification

Three studies used NLP for patient triaging [37, 52, 61] finding accuracies of 53.8% and 76.7% and an F1 of 0.76 for various categorization tasks, whereas 2 used NLP for classifying patients' symptoms to the appropriate medical department with accuracies of 75.5% and 93.5% [60, 131].

3.3.4. Clinician Education and Decision Support

Of sixty studies evaluating NLP responses to clinical questions, 57 utilized ChatGPT [16, 23, 25, 27, 28, 32, 33, 36, 42, 45, 46, 47, 49, 50, 52, 53, 65, 66, 67, 68, 72, 73, 74, 75, 78, 79, 80, 81, 82, 84, 85, 90, 92, 93, 94, 102, 103, 108, 110, 111, 114, 116, 122, 127, 128, 129, 139, 140, 141, 148, 149, 152, 154, 158, 173, 176, 177]. Forty studies reported accuracies ranging from 5.1% to 100% (median 73.5) [16, 23, 25, 27, 28, 32, 36, 45, 49, 50, 52, 53, 65, 66, 67, 72, 78, 79, 80, 81, 82, 84, 85, 92, 94, 102, 103, 110, 116, 125, 128, 129, 139, 140, 141, 152, 154, 158, 171, 177] whereas 16 studies utilized various scales, predominantly Likert [42, 46, 47, 68, 74, 75, 90, 93, 111, 114, 122, 127, 148, 149, 173, 176]. ChatGPT's accuracies for treatment recommendations were 25%–85.3% among 16 studies [25, 45, 47, 65, 67, 72, 75, 78, 79, 93, 103, 110, 111, 139, 148, 177]. ChatGPT 4 outperformed Gemini Advanced and Llama 2 in recommending treatments and was comparable to Claude 3 [47, 103, 148]. Of 23 studies evaluating diagnostic capabilities, 17 reported accuracies of 5.1%–100%, with 15 achieving 62.5%–100% [16, 27, 32, 33, 36, 46, 47, 52, 57, 72, 78, 79, 84, 92, 93, 116, 125, 127, 139, 152, 158, 171, 177]. In diagnostic workup comparisons, ChatGPT outperformed Gemini (Bard) in two studies, matched its performance in another two, but was outperformed by Claude 3 in a study by Schmidl et al. [36, 52, 84, 158]. Twenty‐seven studies evaluated responses to general questions with accuracies from 37.3% to 87.2% [23, 27, 28, 42, 49, 50, 53, 66, 68, 74, 80, 81, 82, 85, 90, 94, 102, 108, 122, 128, 129, 140, 141, 149, 154, 173, 176]. Of these, 12 studies evaluated ChatGPT's responses to board‐exam questions, finding accuracies of 47.3%–86% for ChatGPT‐4 and 45.2%–57% for older models [23, 28, 49, 50, 66, 74, 81, 85, 140, 141, 154, 160]. Buhr 2024 compared ChatGPT4 to Claude 2 and Bard (Gemini), finding that ChatGPT4 outperformed its counterparts by 7.3% and 26.9%, respectively. Long et al. showed ChatGPT4's 75% accuracy on the Royal College of Physicians and Surgeons of Canada Otolaryngology questions and later developed ChatENT, a derivative trained on OHNS references, which outperformed ChatGPT4 with 80% on American Board and 87% on RCPSC sample questions [66].

Four studies utilized NLP for trainee selection [91, 118, 132, 134]. Two studies used NLP to create clusters of residency and fellowship applicants based on personal statements and recommendation letters, which could predict success in the match [91, 134]. Halagur et al. demonstrated demographic biases of ChatGPT in residency selection, while Farlow et al. did not find gender bias in ChatGPT generated reference letters.

3.4. Research Uses

3.4.1. Data Extraction and Analysis for Research

Fourteen studies used NLP to extract information from clinical notes [38, 55, 86, 87, 115, 124, 136, 142, 145, 147, 155, 156, 170, 175]. Two of eleven studies reported accuracies at 83%–84% [136, 156], while the remainder ranged from 94% to 100% [38, 55, 87, 142, 145, 147, 155, 170, 175]. Three studies implemented NLP into databases, including searching EMRs to identify patients who meet inclusion criteria, extracting data from a public database, and indexing biospecimens in a biobank [17, 29, 54]. Five studies used NLP to extract or analyze information from literature [13, 14, 15, 112, 172]. Seven studies utilized BERT for extraction [88, 107, 109, 115, 124, 144, 155].

Seventeen studies used NLP for thematic analysis [15, 22, 24, 43, 86, 88, 89, 107, 109, 112, 119, 120, 133, 138, 143, 144, 164]. Of these, nine studies utilized NLP to analyze the sentiment of OHNS patient posts on forums and social media platforms [22, 24, 43, 86, 88, 107, 138, 143, 164]. Three studies used NLP to analyze letters of recommendation for fellowship applications, revealing gender‐based variations [119, 120, 133].

3.4.2. Manuscript Preparation

Four studies explored NLP assistance in scientific writing [26, 41, 71, 130]. Three studies evaluated ChatGPT's provision of scientific references with accuracies of 73%–87%, compared to earlier versions (ChatGPT 3 and 3.5) with accuracies ranging from 10% to 60% [41, 71, 130]. Lechien et al. utilized ChatGPT 4 for writing assistance where it detected 50% of errors and proposed acceptable corrections for 84% [26].

3.4.3. Evaluation of Scientific Reporting

Five studies examined the use of NLP tools for the evaluation of scientific reporting within otolaryngology [18, 19, 20, 30, 69]. Dang and Hanba evaluated ChatGPT 3.5's ability to assess methodological quality, while Alohali et al. developed an NLP model to predict impact of otology manuscripts. Koroleva et al. created an NLP algorithm with a 97.8% F‐measure for detecting distorted research interpretations. Safranek et al. demonstrated ChatGPT's utility in automating ethical compliance checks, and Carnino et al. reported increased AI‐generated text in 143 JAMA Otolaryngology papers after ChatGPT's release.

4. Discussion

To the author's knowledge, this study represents the most comprehensive review of NLP in the otolaryngology literature. Our findings highlight a surge of NLP in OHNS, with 94% of publications between 2020 and 2024 and the earliest in 1982 by Sagar et al. The majority of studies were US‐based, focused on H&N oncology, and were in clinical settings.

4.1. Patient Education and Monitoring

NLP tools can enhance patient education by tailoring information to factors like culture, language, and health literacy [179, 180]. With OHNS patients forgetting up to 80% of provider education, chatbots could provide ongoing support, but they must ensure accuracy, safety, and appropriate knowledge levels [181, 182, 183]. Our study found LLM response accuracy varied, with some as low as 69%, underscoring their limitations in patient education [98]. While most errors were safe, up to 5% were potentially harmful, and most responses exceeded the NIH‐recommended grade 6 reading level, even with prompt engineering [34, 51, 76, 95, 98, 113, 126, 159, 160]. However, LLM responses improved significantly when prompted and demonstrated utility in improving the readability of existing patient material [51, 76, 95, 98, 113, 126, 159, 160].

Patient interactions with chatbots can be utilized in an outpatient setting during or post treatment to clarify concerns and questions and can reduce unnecessary hospital visits and detect more complications than physicians, as suggested by the findings of Ma et al. [56]. Additionally, chatbots have the potential to detect complications earlier, provide more timely advice, and reduce in‐person wait times, thereby improving outcomes. Accuracy, reliability, and user confidence must continue to be improved and validated with ongoing research in utilizing NLP for patient education and monitoring. More research is needed to compare LLMs for patient education.

4.2. Electronic Medical Record Improvement

Time‐consuming documentation is a major source of physician burnout [184]. While NLP tools are widely studied as scribes in patient care, evidence of their effectiveness in otolaryngology remains limited [185]. Using NLP for surgical charting is unique, requiring adherence to routine steps, procedure‐specific lexicon, and intraoperative charting to enhance efficiency and accuracy. Our findings suggest further refinements are needed before implementation. Integrating NLP into EMR systems not only promises to alleviate the burden of documentation but also to enhance the accessibility of critical patient information. The few studies on this topic demonstrated mixed reliability of NLP for this task. The literature also shows that NLP can aid in labeling results with impressive accuracy, useful for auditing reports and highlighting critical findings to prevent oversight. However, this application should complement, not replace, clinical judgment.

4.3. Triaging and Patient Classification

NLP has the potential to effectively and objectively triage patients, enabling centralized systems to efficiently match patients to physicians and resources, thereby reducing wait times. While promising, the results for both triaging and classifying patients' symptoms to the appropriate medical departments demonstrated the need for significant improvement before use in a clinical setting.

4.4. Clinician Education and Decision Support

NLP tools utilizing LLMs have the potential to efficiently tailor information and provide precise training‐level adjusted answers in time‐constrained clinical or educational settings. Overall, the accuracies of LLM treatment recommendations, diagnostic workups, or responses to OHNS questions were found to be varied between studies. This may be explained by inter‐study methodological heterogeneity and the rapid evolution of LLM models. Additionally, the language used to interact with LLMs may dramatically change the reliability, as demonstrated by Noda et al. 2024, where ChatGPT 4 had a 2‐fold increase in accuracy when prompted in English versus Japanese [141]. Limitations around language can have far‐reaching implications for resource‐limited and non‐English settings. Only a small subset of studies compared different LLMs in this domain, with the vast majority of studies utilizing ChatGPT [36, 47, 49, 52, 84, 103, 148, 158].

NLP tools may streamline the selection process for postgraduate training programs and enable more comprehensive review by predicting the most suitable applicants. Studies have shown that NLP can identify applicants suited for training programs, but further research is needed to validate its use and prevent bias, as noted by Halagur et al. [91, 134]. Moreover, while NLP has been assessed for clinical education and selection processes, studies specifically for teaching medical students and residents are currently limited.

4.5. Data Extraction and Analysis for Research

NLP can extract vast sets of data that were previously too labor‐intensive by manual methods, enabling research previously unfeasible. Our results indicate that NLP tools demonstrate extraction accuracies comparable to humans over large datasets [38, 55, 87, 142, 145, 147, 155, 170, 175, 186, 187]. This is highlighted by Ali et al. [175], where 2,184,309 items of information were extracted from histopathology reports in 22.7 min, with an estimated accuracy of 99%. It is important to note that accuracies may vary by letter type, and it is prudent to utilize a verification set when automatically extracting information using NLP tools [188]. These results highlight the potential of integrating NLP into diverse databases, including EMRs, academic repositories, and biobanks, to streamline searches and data extraction [13, 14, 15, 17, 29, 54, 112, 172].

NLP can enhance data analysis through tasks such as data cleaning, statistical analysis, and thematic analysis. Thematic analysis uses topic modeling and sentiment analysis to create themes among unstructured data, making it a valuable tool for qualitative studies that traditionally require extensive human effort. Two main research applications of thematic analysis seen in the literature include analyzing patient posts on forums and social media, and examining recommendation letters for demographic biases. NLP has the ability to reveal subtle insights that might be missed by human reviewers on a large scale, saving resources.

4.6. Manuscript Preparation

NLP can aid in many facets of scientific dissemination, including writing support. Using NLP for writing assistance has demonstrated significant benefits, particularly for non‐native English speakers [130]. Its perceived value among scholars is evidenced by the rising prevalence of suspected AI‐generated content in peer‐reviewed otolaryngology literature [30]. While there is also growing enthusiasm for using NLP to assist with citation creation, our findings highlight the limitations of NLP in both writing support and citation generation. Despite the potential for NLP to aid in the communication of scientific results, developing guidelines in the use of AI platforms is imperative to ensure that academic content is generated by authors as opposed to chatbots.

4.7. Evaluation of Scientific Reporting

NLP‐based evaluation of scientific reporting tools can supplement conventional review processes to uncover insights that might elude human reviewers and improve objectivity. In particular, NLP‐driven algorithms capable of predicting impact may provide data‐driven insights that could ensure robust methodology, results, and guide editorial decisions. Moreover, NLP can provide an automated safeguard for ensuring the integrity of scientific communication. While current research has evaluated it for the use of identifying misleading findings, ensuring correct conflicts of interest, and assessing for AI use in published works, other potential applications may include identifying unusual patterns in data, citation verification, and fact checking. More research is required to evaluate performance as well as to apply to non‐manuscript scientific communication.

4.8. Level of Evidence

Most studies did not fit the Oxford CEBM 2011 evidence hierarchy (n = 134), mainly consisting of performance evaluation (n = 119), qualitative (n = 9), bibliometric analysis (n = 4), and methods (n = 3). Among classifiable studies, 26 were lower evidence levels (4 or 5) and 5 higher (1–3), highlighting the field's emerging nature.

4.9. Limitations

Due to the rapidly evolving research in this domain, this study may have missed articles published after our search date, August 30 2024. Additionally, the Oxford CEBM levels of evidence do not apply to a significant portion of these studies, limiting our assessment of potential bias. Most studies are of low evidence, and methodologies were very heterogeneous; therefore, limiting the conclusions about NLP performance that can be drawn from this review. The ethical implications of NLP in OHNS were not reviewed in this study.

4.10. Future Implications

The use of NLP in otolaryngology without a governance framework poses significant ethical risks, including patient safety concerns, data integrity issues, and potential privacy violations, particularly with the use of LLM's [189]. Without proper oversight, AI‐generated outputs may contain inaccuracies or biases that could mislead clinicians and patients, compromise research findings, and exacerbate healthcare disparities. Additionally, the misuse of patient data for AI training without transparent consent processes raises concerns about privacy and intellectual property. To ensure ethical compliance, AI tools in OHNS must be rigorously validated, used alongside clinical expertise, and governed by clear protocols that prioritize patient welfare, privacy, and scientific integrity.

Future research should validate the use of AI scribes in the otolaryngology context, and more work is needed for surgical documentation. Guidelines around NLP in scientific reporting, particularly manuscript preparation, are required. Further research is needed to validate NLP's role in manuscript preparation, including areas like text, table, figure, and citation generation. Research on NLP in trainee education is sparse, particularly regarding its trustworthiness and relevance to learning objectives. Future work to implement NLP tools for search and data extraction in databases would be beneficial as there is evidence of impressive accuracies for these tasks. Finally, NLP's potential in resource‐limited settings should be a focus of future work, taking into consideration the ramifications on performance if a non‐English language is used. Considering ChatGPT's declining market share compared to LLMs such as Claude, Llama, and Gemini, it is crucial that more efforts are made to incorporate these alternatives [190].

5. Conclusions

NLP in otolaryngology is rapidly growing, with a broad range of applications, but most research is nascent. Although LLMs can pass the OHNS board exam simulations, they are not yet reliable for clinical use. NLP shows promise in enhancing patient education, streamlining documentation, improving data management for efficiency and care, and triaging. It also offers potential in early complication detection for better outcomes. NLP is currently accurate and efficient in data extraction from diverse sources and aiding in qualitative analysis. Guidelines for NLP use in research and its application in administration and trainee education warrant further study.

Ethics Statement

The authors have nothing to report.

Consent

All authors consent for publication.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1. Supporting Information S1: Exact Search Strategy.

Table S1. Summary of Study Characteristics. Google Bard had been renamed to Gemini and Microsoft Bing Chat to Copilot.

Table S2. Summary of study methods and outcomes.

Banyi N., Ma B., Amanian A., Bur A., and Abdalkhani A., “Applications of Natural Language Processing in Otolaryngology: A Scoping Review,” The Laryngoscope 135, no. 9 (2025): 3049–3063, 10.1002/lary.32198.

Data Availability Statement

All data generated during the current study are available from the corresponding author on reasonable request.