A Systematic Review of Natural Language Processing Methods and Applications in Thyroidology

Abstract

This study aimed to review the application of natural language processing (NLP) in thyroid-related conditions and to summarize current challenges and potential future directions. We performed a systematic search of databases for studies describing NLP applications in thyroid conditions published in English between January 1, 2012 and November 4, 2022. In addition, we used a snowballing technique to identify studies missed in the initial search or published after our search timeline until April 1, 2023. For included studies, we extracted the NLP method (eg, rule-based, machine learning, deep learning, or hybrid), NLP application (eg, identification, classification, and automation), thyroid condition (eg, thyroid cancer, thyroid nodule, and functional or autoimmune disease), data source (eg, electronic health records, health forums, medical literature databases, or genomic databases), performance metrics, and stages of development. We identified 24 eligible NLP studies focusing on thyroid-related conditions. Deep learning-based methods were the most common (38%), followed by rule-based (21%), and traditional machine learning (21%) methods. Thyroid nodules (54%) and thyroid cancer (29%) were the primary conditions under investigation. Electronic health records were the dominant data source (17/24, 71%), with imaging reports being the most frequently used (15/17, 88%). There is increasing interest in NLP applications for thyroid-related studies, mostly addressing thyroid nodules and using deep learning-based methodologies with limited external validation. However, none of the reviewed NLP applications have reached clinical practice. Several limitations, including inconsistent clinical documentation and model portability, need to be addressed to promote the evaluation and implementation of NLP applications to support patient care in thyroidology.

Article Highlights.

• •NLP has the potential to streamline healthcare data retrieval, reducing laborious tasks for stakeholders.
• •Despite its promising applications in research and patient care, our review revealed limited NLP exploration in thyroidology.
• •The most investigated domain was thyroid nodules, primarily using electronic health records with radiology reports as the primary data source.
• •Traditional challenges in NLP exploration and implementation include language diversity, contextual comprehension, and model portability.
• •Large language models have been recently explored and offer solutions to previously NLP encountered challenges.

Artificial intelligence (AI) aims to achieve human-like intelligence through entities (eg, machines), capable of processing and performing actions akin to human behavior.¹ In addition, machine learning stands as a rapidly evolving field of computer science that seeks to train machines using data sets to perform time-consuming tasks that typically require human cognitive abilities. Its potential to solve real-world challenges across various domains, including health care, has led to increased research into its uses.² By integrating different AI modalities into the decision-making process of physicians, this technology has the potential to enhance the field of medicine by improving the accuracy of diagnostic tests, streamlining provider workflow, enabling better disease and therapeutic monitoring, and resulting in better patient outcomes.^3,4

Natural language processing (NLP) represents the intersection between linguistics and AI, analyzing text and speech to achieve human-like language understanding.⁵ Despite its more than 50-year history, the growing applicability in medicine has sparked significant interest in these technologies, particularly by overcoming previous limitations through advancements in machine and deep learning.^5,6 In health care, NLP models have primarily been used to identify and extract information from unstructured data from the electronic health records (EHRs).⁷ The implementation of EHRs has led to exponential growth in the volume of health care data over the past 2 decades. However, only around 20% of EHR data are estimated to be structured as diagnostic or billing codes and simple clinical variables like vital signs or laboratory results. In contrast, the bulk of data in EHRs are unstructured in free text, like clinical notes or diagnostic reports, making their use in research time-consuming and challenging.⁸ To address this challenge, NLP techniques have been developed to efficiently extract and standardize diverse medical data from textual sources, encompassing elements like history and physical examinations, laboratory findings, diagnostic reports, and treatment records.⁹ For example, in radiology, NLP has been used to identify specific features of interest within imaging reports.¹⁰ Similarly, in the field of oncology, it has been leveraged to extract and categorize information from pathology reports, thereby supporting the staging and prediction of outcomes for various types of cancer.¹¹

Thyroid disorders are highly prevalent among the general population. Recent studies have demonstrated the effectiveness of various NLP systems in extracting relevant information from EHRs within the field of thyroidology.¹² These NLP-driven approaches have shown promise in enhancing and validating diagnostic and prognostic tools for various thyroid pathologies, including functional thyroid disorders, thyroid nodules, and thyroid cancer.^4,13 Despite these promising developments, the existing literature lacks a comprehensive overview. Such an overview could prove invaluable for clinicians, researchers, and other stakeholders interested in understanding how NLP can be applied to improve the care of patients with thyroid diseases. This study was designed to fill this gap by systematically reviewing the applications of NLP in thyroid-related conditions. We also aimed to summarize the current challenges and provide insights into future perspectives in this burgeoning field.

Methods

This systematic review was conducted based on a priori established protocol, available in PROSPERO (CRD42022375085). This article is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines (PRISMA).¹⁴

Data Sources and Search Strategy

A comprehensive search was conducted in multiple scientific databases for articles published between January 2012 and November 2022. Databases included Ovid MEDLINE, Epub Ahead of Print, In-Process & Other Non-Indexed Citations; Daily; Ovid EMBASE; Ovid Cochrane Central Register of Controlled Trials; Ovid Cochrane Database of Systematic Reviews; and Scopus. The search strategy for references using NLP in patients with thyroid conditions was designed and conducted by an experienced librarian (L.P.). Subsequently, a snowballing technique was applied to expand our search by identifying additional potentially relevant references that might have been overlooked during the initial search or published after our search timeline, until April 2023. The full search strategy is described in Supplemental Material (available online at https://www.mcpdigitalhealth.org/).

Eligibility Criteria

We included studies describing empirical research about the application of NLP methods for any thyroid-related condition. Studies were excluded for the following reasons: nonempirical works, such as editorials, systematic reviews, and commentaries; not performed in humans; or not published or available in English. In cases of multiple studies by the same research group using the same data set, we prioritized based on the publication date (including the most recent and excluding the earlier ones).

Study Selection Process

All titles, abstracts, and full texts were independently screened using standardized and piloted criteria through the Distiller Systematic Review software by pairs of reviewers (D.S.T., C.S.J., M.D., and R.L.-T.). For each stage, we conducted pilots to calibrate judgments among the reviewers. Disagreements between 2 reviewers were resolved by consensus.

Data Collection and Analyses

For all included studies, we extracted the following variables: general study characteristics, including year, country, and population demographic characteristics; and NLP applications or tasks (eg, identification, classification, or automation). Notably, certain publications involved multiple NLP applications, which were categorized accordingly. Furthermore, we noted the thyroid conditions (eg, thyroid cancer, thyroid nodule, or functional or autoimmune disease), data source (eg, EHR, online health forums or social media, medical literature databases, or genomic databases), evaluated data set, applied NLP methods (eg, rule-based, machine learning, deep learning, or hybrid), performance metrics (eg, accuracy, sensitivity [SN], and positive predictive value [PPV]), and stages of development of the models (eg, training, validation, or testing). In cases in which the data were unavailable, we abstained from extraction.

Results were summarized using frequencies and percentages for categorical variables. The exploratory nature of the research question, the heterogeneity of methodology, and the lack of consensus on the quality assessment approach for the NLP models prevented a quantitative meta-analysis and the evaluation of the risk of bias. Instead, data were analyzed using a meta-narrative approach.¹⁵

Results

Study Selection and Characteristics

A total of 2111 potentially relevant references were identified and retrieved through the electronic database search. After screening the titles and abstracts, 2007 studies were excluded. During the subsequent full-text screening, 86 studies were excluded, primarily for either not describing applied NLP methods or not focusing on thyroid conditions. Ultimately, 18 articles met our eligibility criteria and were included in the review. Using the snowballing technique, we identified and added 6 relevant studies to our analysis. Consequently, 24 articles were finally included in our study (Figure 1).

Figure 1.

PRISMA-P flow diagram depicting the study selection process. NLP, natural language processing; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analysis.

An increasing trend in yearly NLP-based thyroid publications was observed, especially in the past 3 years, with more than half of the included articles falling within this timeframe (15/24, 62.5%). Most studies were conducted in the United States (12/24, 50%), followed by China (7/24, 29%) and South Korea (3/24, 12%). In addition, demographic characteristics were reported in approximately one-third of the studies included (9/24, 37%) (Table).16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38

Table.

Baseline Characteristics of the Studies Analyzed in This Systematic Review

Reference, year	Study design	Demographic informationa	Thyroid condition	Data source	Cohort	AI methods and tasks
Canton et al,16 2021; USA	Single center	Age, gender, ethnicity	Thyroid nodules	EHR	2289 records	DL
	Retrospective			Radiology reports		Identification
Chen et al,17 2017; China	Single center	NA	Thyroid nodules	EHR	13,592 records	DL
	Retrospective			Radiology reports		Classification
Chen et al,18 2018; China	Single center	NA	Thyroid nodules	EHR	6116 records	DL
	Retrospective			Radiology reports		Classification
Chen et al,19 2020; USA	Multicenter	NA	Thyroid nodules	EHR	247 records	RB
	Retrospective			Radiology reports		Identification, automation
Chen et al,20 2022; China	NA	NA	Thyroid nodules	MLD	5770 references	ML
	Retrospective			PubMed		Automation
Dedhia et al,21 2022; USA	Multicenter	NA	Thyroid nodules	EHR	1612 records	Hybrid
	Retrospective			Radiology reports		Identification, automation
Drake et al,22 2019; USA	Multicenter	Age, gender	Thyroid nodules	EHR	51,907 records	RB
	Retrospective			Radiology reports		Identification
Grani et al,23 2021; Italy	NA	NA	Functional or autoimmune disease	SM	27,525 entries	ML
	Retrospective			Medicitalia.it		Classification
Kongburan et al,24 2016; Thailand	NA	NA	Thyroid cancer	MLD	500 references	ML
	Retrospective			PubMed		Automation
Lian et al,25 2023; USA	Single center	NA	Thyroid cancer	EHR	500 records	DL
	Retrospective			Interviews	100 patient sample	Classification
Luft et al,26 2019; USA	Single center	Age, gender, ethnicity	Functional or autoimmune disease	EHR	84,000 records	RBb
	Retrospective			Patient charts	1319 patient sample	Identification
Miao et al,27 2020; China	Single center	Age, gender	Thyroid nodules	EHR	1290 records	NA
	Retrospective			Radiology and pathology reports		Identification
Park and Hong,28 2018; South Korea	NA	Age, gender	Functional or autoimmune disease	SM	1768 entries	ML
	Retrospective			WebMD		Classification
Park et al,29 2021; South Korea	Multicenter	Age, gender	Thyroid cancer	EHR	308 records	RB
	Retrospective			Radiology reports	220 patient sample	Automation
Pathak et al,30 2023; USA	Multicenter	NA	Thyroid nodules	EHR	490 records	DL
	Retrospective			Radiology reports		Identification
Santos et al,31 2021; USA	Single center	Age, gender, ethnicity	Thyroid nodules	EHR	1132 records	DL
	Retrospective and prospective			Radiology reports		Identification, automation
Short et al,32 2022; USA	Single center	Age, gender, ethnicity, comorbidities	Thyroid nodules	EHR	13,385 records	DL
	Retrospective			Radiology reports		Identification, automation
Yoo et al,12 2022; South Korea	Multicenter	Age, gender	Thyroid cancer	EHR	108,372 records	RB
	Retrospective			Radiology and pathology reports		Classification
Zhang et al,33 2021; China	NA	NA	Thyroid cancer	MLD	34,692 references	ML
	Retrospective			PubMed		Automation
Zhang et al,34 2022; China	Multicenter	NA	Thyroid cancer	EHR	788,129 records	DL
	Retrospective			Radiology reports		Automation
Zhang et al,35 2023; USA	Multicenter	NA	Thyroid nodules	EHR	565 records	Hybrid
	Retrospective			Radiology and pathology reports	471 patient sample	Identification, automation
Zheng et al,36 2020; USA	NA	NA	Functional or autoimmune disease	GD	772,394 records	NA
	Retrospective			BioVU DNA biobank	84,821 patient sample	Identification
Zhou et al,37 2021; USA	NA	NA	Thyroid cancer	GD	NA	NA
	Retrospective			ClinGen, NCCN guidelines, OMIM, Genetics Home Reference, GeneCards, and Gene-NCBI		Automation
Zou et al,38 2021; China	Single center	NA	Thyroid nodules	EHR	5328 records	DL
	Retrospective			Radiology reports		Classification

DL, deep learning; EHR, electronic health records; GD, genetic database; ML, machine learning; MLD, medical literature database; NA, not applicable or reported; RB, ruled-based; SM, social media.

^aAnalysis from studies in which demographic information from data sets were reported.

^bNot specified, however presumed to be the most likely implemented method.

Data Sources

Most studies included EHR as the retrieved source (17/24, 71%), followed by medical literature databases (3/24, 13%) such as PubMed, genomic databases (2/24, 8%), and online health forums or social media (2/24, 8%) (Figure 2). Within the studies that used data from EHRs, radiologic reports emerged as the most used source, constituting 88% (15/17) of the cases. Among these, thyroid ultrasound data were the prevailing imaging modality, used in 73% (11/15) of instances, while computer tomography scan reports were used in 20% (3/15) of cases. Three studies used both radiology and pathology reports.^12,27,35 The studies were presented with varying data set sizes. Of the 24 studies, 11 (45%) used sample sizes of less than 5000 records, and 9 of 24 (38%) studies used data sets with more than 10,000 records.

Figure 2.

Sankey flow diagram depicting the distribution of thyroid. DL, deep learning; EHR, electronic health records; F or AD, functional and autoimmune disease; GD, genetic database; ML, machine learning; MLD, medical literature database; NA, not applicable or reported; RB, ruled-based; SM, social media.

NLP Application in Thyroid Conditions

We found that thyroid nodule was the most investigated domain (13/24, 54%), followed by thyroid cancer (7/24, 29%), and functional and autoimmune disease (4/24, 17%). Supplemental Table 1 (available online at https://www.mcpdigitalhealth.org/) describes an in-depth analysis of the NLP methods used on various data sources to achieve distinct tasks within the thyroid conditions Supplemental Table 2.

NLP Applications in Thyroid Nodules

Several NLP algorithms were developed and trained on large data sets to classify nodules as benign or malignant based on reported radiologic features (accuracy, 86%-88%; SN, 84%-92%; and PPV, 94%).^18,38 In addition, other NLP models aimed at identifying granular radiology characteristics from ultrasound reports (accuracy, 77%-98%; SN, 85%-98%; and PPV:,74%-98%).^21,30,31

Incidental thyroid findings are often observed in nonthyroid-related images, leading to increased detection of nodules and a potential thyroid cancer diagnosis. Drake et al²² used an NLP algorithm to evaluate the prevalence of incidental findings across various imaging modalities. In addition, Canton et al¹⁶ developed a highly accurate model for detecting thyroid lesions in imaging studies frequently used during trauma assessments in emergency department settings (SN, 90%; specificity, 95.3%).

The Thyroid Imaging, Reporting, and Data System (TI-RADS) is commonly used to ensure consistency in reporting systems for thyroid nodule characteristics.^39,40 Chen et al¹⁹ developed a model to capture the crucial missing elements of TI-RADS. Moreover, Short et al³² engineered an NLP pipeline tailored to identify radiologic reports that align with the criteria for follow-up assessments. Their model exhibited promising accuracy (96.5%), SN (92.1%), and specificity (96%). Furthermore, Santos et al³¹ introduced a model designed to integrate TI-RADS reports with patient demographic information and comorbidities. Significantly, this model underwent prospective validation (accuracy, 0.89; F1 score, 0.99) and testing at an external facility (accuracy, 0.85; F1 score, 0.94).³¹ Finally, Zhang et al³⁵ crafted a multistep model capable of integrating data from thyroid images, pathology reports, and radiology reports, achieving a commendable accuracy rate of 83%.

NLP Applications in Thyroid Cancer

Lian et al²⁵ developed an NLP pipeline to measure and categorize health-related quality of life based on narrative interviews with patients who underwent surgical treatment (area under the curve, 0.76; accuracy, 70.09; SN, 70.02%; and PPV, 70.20%).²⁵ In addition, Yoo et al¹² developed an NLP algorithm to determine thyroid cancer diagnosis and stage based on retrospective information from medical records, specifically using surgical pathology and whole-body scan reports (SN, 100%; PPV, 100%). Another application was able to retrieve information from online publications and genomic databases to identify genes contributing to nonmedullary thyroid cancer, encompassing 95%-97% of all thyroid cancers. The study by Zhou et al³⁷ used an NLP algorithm to identify nonmedullary thyroid cancer–related genes from online databases.

NLP Applications in Functional and Autoimmune Disease

Grani et al²³ used an NLP pipeline to collect data from text messages in an online open medical forum to analyze patient experiences of living with hypothyroidism and concerns regarding their medication. Similarly, Park and Hong²⁸ aimed to uncover patients’ perspectives on thyroid hormone replacement therapy from online health forums (WebMD) and determine its impact on treatment satisfaction. In another study, Zheng et al³⁶ developed an NLP-based tool, capable of extracting clinical characteristics of patients with hypothyroidism from EHRs based on phenotypes collected from an array of medical resources, with an accuracy rate of over 97%.³⁶ Finally, Luft et al²⁶ used NLP to extract clinical features from EHRs from a cohort of pediatric patients with mood and anxiety disorders and correlated them with abnormal thyroid stimulating hormone levels.

Discussion

The digitalization of health care has aimed to improve patient care, streamline health care processes, and revolutionize clinical and health care delivery research.⁴¹ NLP can retrieve large volumes of narrative data and transform it into computable elements for downstream analyses, which was previously limited by labor-intensive and time-consuming manual extraction by human annotators.⁴² As in every area of health care, the potential applications and benefits of NLP in thyroid diseases are enormous. For instance, we noted an increasing number of NLP-based thyroid publications, especially in the past 3 years, with more than half of the included articles falling within this timeframe. To our knowledge, this is the first systematic review of NLP applications in the context of thyroidology. Our systematic review identified that, although still limited, NLP is already being used to study thyroid diseases, particularly thyroid nodules, and thyroid cancer, which were the focus of 54% and 29% of the studies included in our review, respectively.

Studies have applied NLP to extract thyroid nodule features from radiology reports leveraging state-of-the-art deep learning models.³⁰ Furthermore, although some models focused on basic tasks, such as detecting the presence of thyroid incidentalomas in computed tomography, magnetic resonance imaging, or ultrasound reports,^16,22 more sophisticated models were capable of determining whether the reported incidentaloma met the criteria for further ultrasound evaluation and additionally tracked the completion of such evaluations.³² Compared with traditional manual extraction of unstructured data, these models effectively harnessed large volumes of data, facilitating observational research, promoting quality improvement initiatives, enabling standardization of unstructured documentation, and developing real-time predictive tools for clinical care. In addition, NLP models were able to build large-volume data sets by extracting features from different free text sources, including diagnostic and pathology reports and clinician documentation.^12,21 Furthermore, some models standardized unstructured data from multiple institutions to generate multicenter data sets.^{12,19,21,29,34}

Although most models used EHR data, other valuable data sources have been explored. Zheng et al³⁶ used NLP-extracted phenotypes from several online resources. Others used NLP tools to facilitate abstract screening for systematic literature reviews on thyroid cancer genetic associations or even automatically analyzing and integrating findings from previously published literature.²⁴ These initiatives allowed the researchers to use data more efficiently and comprehensively from the exponentially growing body of literature and genetic repositories. In addition, Park and Hong²⁸ used NLP on social media posts to identify issues related to thyroid hormone replacement from patient medication reviews. This type of tool can provide a more complete picture of the patient experience, including their emotional and social responses to their condition and treatments, which might not be efficiently captured in clinical records, to facilitate the identification of themes for discussion during shared decision-making encounters and characterize elements to be considered on future treatment or quality-of-life studies.

Notwithstanding the promising results of NLP, none of the applications included in this review are readily available for use in clinical practice. Notably, among the 24 studies identified, only 1 adopted a prospective design,³¹ and 2 undertook validation of their NLP models in an external health care setting.^31,34 The road to incorporating these NLP interventions into routine research or clinical practice is riddled with several challenges that need careful consideration and concerted efforts to surmount. Specifically in the thyroid field, we hypothesize that the uptake of NLP methods is associated with the complexity of the thyroid-related domains, variations in language expression and reporting styles, completeness and accuracy of clinical documentation (ie, data on patient-specific concerns, complaints, or severity of symptoms depends on the accuracy of providers’ documentation), semantic (ie, misspellings, abbreviations, acronyms, or synonyms), and context (ie, it is challenging to create algorithms that can appropriately extract chronologic descriptions or simultaneous references in situations like a thyroid ultrasound report that includes several nodules), which can affect the NLP outcome, decrease the performance of the algorithm when applied to different institutions, and limit the portability and scalability of the interventions.^7,21,34,43 In addition, data sources need to be representative of the population to avoid the incorporation of inequities and social bias into the models.⁴⁴ Finally, using NLP methods requires high optimization for the local environment and extensive domain knowledge, which can be expensive, and stakeholders’ lack of financial resources or prioritization could halt their implementation. Thus, efficacy and cost-effectiveness trials are important to demonstrate the value of the intervention and facilitate its adoption.⁴¹

The examination of NLP studies within the domain of thyroidology, as presented in this review, has brought to light a pronounced divergence in NLP methods. Deep learning emerged as the preferred NLP method, with some studies applying pretrained large language models,^16,25,30,34 followed by rule-based approaches. However, it is noteworthy that some studies refrained from providing explicit specifications regarding the used NLP methods.

In contemplating the future of NLP within the domain of thyroid nodule and cancer management, it is evident that enhancing the reporting system methods and integrating models, particularly with the framework of large language models, will significantly expand the role of NLP. This integration marks a shift toward more sophisticated, efficient, and versatile applications of NLP in thyroid nodule and cancer management.

We acknowledge several limitations that should be considered when evaluating our results. The heterogeneity observed in methodology, outcomes, and performance metrics across the included studies made it unfeasible to consistently report the results of the individual studies and to conduct a meta-analysis. In addition, we must recognize the potential for publication bias, which could lead to an overrepresentation of favorable results. However, despite these limitations, it is crucial to highlight the strengths of our systematic review. We adhered diligently to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement guidelines, ensuring a methodical and standardized approach to our review process. Furthermore, we successfully identified many publications that met our rigorous inclusion criteria. Consequently, our findings provide a comprehensive and robust characterization of the current landscape of NLP in thyroidology, holding valuable implications for clinicians, researchers, and other stakeholders invested in this field of study.

Conclusion

The utilization of NLP within the realm of thyroidology exhibits a growing interest and holds the potential for advancing both research and patient care, while mitigating the burden placed on healthcare system stakeholders. However, it is noteworthy that the domains of interest within thyroidology and the NLP methodologies used remain somewhat restricted. Consequently, ample opportunity exists for further exploration, encompassing the untapped potential of NLP applications in various thyroid conditions that have yet to be investigated.

Potential Competing Interests

Drs Brito and Soto Jacome were supported by the National Cancer Institute of the National Institutes of Health under Award Number R37CA272473. N.S.O. was supported by the National Cancer Institute of the National Institutes of Health under Award Number K08CA248972. Dr Wu was supported by Patient-Centered Outcomes Research Institute (PCORI) under Award Number ME-2018C3-14754 and National Institute on Aging under Award Number R56AG069880. The other authors report no competing interests.