Artificial intelligence-assisted risk stratification of thyroid nodules with atypia of undetermined significance

Abstract

Objective

Given the limitations of conventional approaches in managing indeterminate thyroid nodules, there remains an unmet need for non-invasive assistant tools to improve risk stratification. This study aimed to evaluate the clinical applicability of an artificial intelligence (AI) model for thyroid nodules with atypia of undetermined significance (AUS) cytology.

Methods

We retrospectively reviewed patients who underwent fine-needle aspiration (FNA) for thyroid nodules between January 2019 and December 2020 across five medical institutions in Korea. Nodules initially diagnosed as AUS and later confirmed as benign or malignant were included. A previously developed deep learning-based AI model, AI-Thyroid, was employed to provide binary classifications (benign or malignant) and malignancy risk estimates.

Results

A total of 165 thyroid nodules were analyzed. The median (interquartile range) longest diameter was 1.30 cm (0.80–2.10), and the malignancy rate of the cohort was 39%. In binary classification tasks, the model achieved a sensitivity of 0.91 and a negative predictive value of 0.87. The area under the curve (AUC) based on estimated malignancy risk was 0.75 (95% confidence interval (CI): 0.68–0.83), and the AUC derived from K-TIRADS categories 2–5 was 0.76 (95% CI: 0.69–0.83), indicating comparable diagnostic accuracy with the traditional scoring system. Subgroup analyses demonstrated that the model achieved a sensitivity of 98% in nodules smaller than 1.5 cm.

Conclusion

AI-assisted ultrasound analysis offers supplementary diagnostic information for thyroid nodules with AUS cytology. Its high sensitivity and negative predictive value may assist clinicians in decision-making processes, particularly for small, low-risk thyroid nodules.

Introduction

Pathological confirmation through tissue biopsy is essential for diagnosing cancer. In evaluating thyroid nodules suspected of malignancy, fine-needle aspiration (FNA) cytology serves as the primary diagnostic approach, given the superficial anatomical location of the thyroid gland. While histopathological examination of surgical specimens remains the definitive gold standard for confirming thyroid cancer, FNA cytology is widely used in clinical practice because it is minimally invasive and carries a low risk of damage to surrounding structures or scarring (1). However, indeterminate cytology, including unsatisfactory specimens and atypia of undetermined significance (AUS), poses a clinical challenge and comprises approximately 20% of FNA results (2). Gray zones that cannot be definitively diagnosed as benign or malignant nodules, despite adequate sampling, are classified as AUS, and they correspond to category III within the Bethesda System for Reporting Thyroid Cytopathology, which is a standardized reporting method that divides thyroid nodules into six categories (3).

Recent guidelines for patients with AUS nodules recommended re-examination, molecular testing, or diagnostic surgery (4, 5). However, repeat FNA may still yield inconclusive results, while contributing to additional anxiety and procedural discomfort on patients. Although molecular testing can improve diagnostic accuracy, its use in routine clinical practice is limited by high costs and restricted insurance coverage. Diagnostic surgery may result in unnecessary thyroidectomy, posing risks related to anesthesia, potential complications, and scarring, thereby causing both physical and psychological burdens to patients. For these reasons, decision making following an AUS diagnosis remains a complex challenge for clinicians, and there is a continued demand for reliable, non-invasive diagnostic tools to support clinical judgment. One promising strategy is the application of advanced ultrasound image analysis to improve risk stratification (6).

With recent advances in artificial intelligence (AI), particularly deep learning, automated interpretation of medical images has become feasible, demonstrating high accuracy across various clinical fields (7). In thyroid ultrasonography, AI models trained on large-scale datasets have been developed, with some reports indicating diagnostic performance comparable to or exceeding that of human experts (8, 9). Among these, AI-Thyroid is a deep learning model developed using multicenter data from Korea to assess the malignancy risk of thyroid nodules based on static ultrasound images (10). This study aims to evaluate the diagnostic performance of AI-Thyroid for thyroid nodules reported as AUS on cytology, in order to determine whether it can serve as a complementary diagnostic tool to existing ultrasound risk stratification systems for nodules with indeterminate cytology.

Materials and methods

Study subjects

This study screened patients who underwent FNA for the evaluation of thyroid nodules between January 2019 and December 2020 at five university-based hospitals located in Seoul, Korea (Fig. 1). Board-certified physicians in internal medicine, otolaryngology, general surgery, and radiology, specializing in thyroid diseases, performed ultrasound examinations and FNA. We included indeterminate nodules with initial FNA results identified as AUS that were confirmed as benign or malignant after repeat FNA, core needle biopsy, or surgery. Both initial cytologic interpretations and final diagnoses were classified according to the 2017 Bethesda System for Reporting Thyroid Cytopathology. Cases without a definitive final diagnosis (including loss to follow-up) or with unevaluable ultrasound images – such as non-nodular inflammatory lesions, totally calcified nodules, or inadequate image quality due to incomplete image acquisition or overlap with adjacent nodules – were excluded.

Figure 1.

Study flow. This multicenter retrospective study enrolled patients from three data sources. Group 1 included patients identified from the Catholic Medical Center Clinical Data Warehouse, covering Seoul St. Mary’s Hospital, Yeouido St. Mary’s Hospital, and Eunpyeong St. Mary’s Hospital. Group 2 consisted of patients from Hanyang University Hospital. Group 3 comprised patients from Kangbuk Samsung Hospital. AUS, atypia of undetermined significance; FNA, fine-needle aspiration.

Data from Yeouido St. Mary’s Hospital, Seoul St. Mary’s Hospital, and Eunpyeong St. Mary’s Hospital were collected through the clinical data warehouse, a research-purpose anonymized database of the Catholic Medical Center, and the study protocol was approved by the Institutional Review Board (IRB) of Yeouido St. Mary’s Hospital (IRB number SC23WNDI0235). The protocol of this study was also reviewed and approved by the IRBs of Kangbuk Samsung Hospital (IRB number KBSMC 2023-10-033-001) and Hanyang University Hospital (IRB number HYU 2023-12-044).

Data processing

The ultrasound images used in this study were collected retrospectively through the review of the electronic medical record system. One thyroid nodule was selected from each patient, and two sonographic images, from a transverse view and a longitudinal view, taken for the evaluation of the longest diameters in three directions, were extracted from each nodule. Based on the examiner’s interpretation, three investigators (JK, HAL, and MK) conducted a secondary review to confirm the nodule size defined as the longest diameter of the nodule, image features, and classification according to the Korean Thyroid Imaging Reporting and Data System (K-TIRADS) (11). Accordingly, inter-observer agreement was assessed between the index assessment and the initial secondary review, whereas intra-observer agreement was assessed between the initial and repeat secondary reviews after a washout period, with blinding to prior assessment results.

The images were preprocessed by one physician (JK). The captured two-dimensional images were cropped into square shapes of various sizes to encompass the margin of nodule. In cases where the nodule was too asymmetric or large and the entire margin could not be covered, the part of the image that was most suspected of malignancy on ultrasound (solid portion, the most hypoechoic part of the lesion, or punctate echogenic foci) was included with a part of nodule margin.

Artificial intelligence model

We used AI-Thyroid, a publicly available web-based model developed through a previous multicenter study in Korea (https://us.cdss.co.kr/). This model was trained on 19,711 captured two-dimensional ultrasound images from 6,163 patients collected across multiple medical institutions, making it the largest dataset used in the development of a thyroid ultrasound AI model in Korea (10). The prediction output of the model includes a binary classification (benign or malignant) and a malignancy risk score expressed as a percentage. In our analysis, two ultrasound images – one transverse and one longitudinal – were used per nodule (Fig. 2). Each nodule was therefore analyzed twice, and for the final analysis, the higher of the two malignancy risk scores was adopted.

Figure 2.

Representative case of a benign nodule with discrepant results between transverse and longitudinal image analyses. Each nodule was analyzed in both transverse and longitudinal views, and the result with the higher predicted malignancy rate was used for the final classification. The overall discrepancy rate between the two views was 18.6%. (A) Transverse image. (B) Longitudinal image.

Statistical analysis

Continuous variables were expressed as medians with interquartile ranges (IQRs), and categorical variables were expressed as frequencies and proportions. Weighted Cohen’s kappa statistics with quadratic weights were used to assess inter-observer and intra-observer agreement for K-TIRADS classification, accounting for the ordinal nature of the categories. Model performance was assessed in terms of sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and the area under the receiver operating characteristic (ROC) curve (AUC), each accompanied by 95% confidence intervals (CIs). Comparisons of AUCs were conducted using DeLong’s test. A confusion matrix was constructed to examine classification performance by comparing predicted with observed outcomes. McNemar’s test was employed to determine whether disagreement patterns within the confusion matrix were statistically significant, and the Mann–Whitney U test was applied to evaluate differences in the distribution of continuous malignancy risk scores between benign and malignant groups. All analyses were performed using R software (version 4.3.3; R Foundation for Statistical Computing, Austria).

Results

Baseline characteristics of the study cohort

A total of 165 patients from five participating institutions were included in the analysis, with one thyroid nodule per patient. The median age was 53 years, and 71% of the patients were female. The median nodule size was 1.30 cm, and the overall malignancy rate was 39%. Final diagnoses were established by surgery in 79 patients (47.9%), core needle biopsy in 51 patients (30.9%), and repeat FNA in 35 patients (21.2%). For human interpretation of K-TIRADS classification, weighted κ values were 0.90 (95% CI: 0.75–0.97) for inter-observer agreement and 0.82 (95% CI: 0.67–0.97) for intra-observer agreement. The baseline characteristics of the study cohort are summarized in Table 1.

Table 1.

Baseline characteristics of 165 patients with atypia of undetermined significance (AUS) thyroid nodules. The values are presented as median IQR or as n (%), as appropriate. Final diagnoses were established by surgery, core needle biopsy, or repeat fine-needle aspiration.

Characteristics	Values
Age, years	53 (41–62)
Sex
Male	46 (27.8)
Female	119 (72.1)
Final diagnostic method
Surgery	79 (47.9)
Core needle biopsy	51 (30.9)
Repeat fine-needle aspiration	35 (21.2)
Final diagnosis
Benign	101 (61.2)
Malignancy	64 (38.8)
Nodule size, cm	1.30 (0.80–2.10)
K-TIRADS
2	7 (4.2)
3	51 (30.9)
4	58 (35.2)
5	49 (29.7)
US characteristics of thyroid nodules
Solid composition	141 (85.5)
Hypoechogenicity	101 (61.2)
Non-parallel	28 (16.8)
Irregular margin	37 (22.2)
Punctate echogenic foci	40 (24.0)

IQR, interquartile range; K-TIRADS, Korean Thyroid Imaging Reporting and Data System; US, ultrasound.

Diagnostic performance of the AI model for AUS thyroid nodules

Table 2 summarizes the diagnostic performance metrics, showing that the AUC for the malignancy risk percentage of AI-Thyroid was 0.75 (95% CI: 0.68–0.83), and the AUC for K-TIRADS categories 2–5 was 0.76 (95% CI: 0.69–0.83), with no statistically significant difference, indicating comparable diagnostic accuracy between the AI model and the traditional scoring system. In binary prediction, the model demonstrated a higher sensitivity (0.91 vs 0.53) and negative predictive value (0.87 vs 0.74) compared with K-TIRADS. Subgroup analyses based on nodule size revealed a notably high sensitivity of 0.98 for nodules smaller than 1.5 cm. Institution-specific analyses demonstrated minimal variability in diagnostic outcomes, suggesting robust performance of the AI model across different examiners and ultrasound equipment.

Table 2.

Diagnostic performance of AI-Thyroid by nodule size and institution. Overall model accuracy and subgroup accuracies were compared with those of K-TIRADS as the reference using DeLong’s test, with P-values presented.

Groups	SN	SP	PPV	NPV	Accuracy (95% CI)	AUC (95% CI)	P
Total cohort, n = 165
K-TIRADS	0.53	0.85	0.69	0.74	0.72 (0.65–0.79)	0.76 (0.69–0.83)	Ref
AI-Thyroid	0.91	0.39	0.48	0.87	0.58 (0.51–0.66)	0.75 (0.68–0.83)	0.891
Subgroup analysis
Size of nodule
<1.5 cm, n = 90	0.98	0.23	0.61	0.90	0.64 (0.54–0.74)	0.76 (0.66–0.86)	0.981
≥1.5 cm, n = 75	0.64	0.49	0.23	0.86	0.52 (0.40–0.64)	0.61 (0.45–0.76)	0.084
Institution
Group 1, n = 54	0.91	0.38	0.50	0.86	0.59 (0.45–0.72)	0.72 (0.59–0.86)	0.662
Group 2, n = 62	0.94	0.41	0.36	0.95	0.56 (0.42–0.68)	0.82 (0.71–0.93)	0.367
Group 3, n = 49	0.88	0.35	0.61	0.73	0.63 (0.48–0.77)	0.75 (0.62–0.89)	0.956

SN, sensitivity; SP, specificity; ref, reference; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the ROC curve; CI, confidence interval; Ref, reference; K-TIRADS, Korean Thyroid Imaging Reporting and Data System.

Diagnostic outcomes and malignancy risk distribution

The confusion matrices for the model and K-TIRADS in classifying AUS nodules are shown in Table 3. Based on these results, additional diagnostic testing could potentially be deferred for nodules predicted as benign by the AI model (45 of 167 nodules, 27%). Within this subset, however, six nodules (13.3%) were ultimately found to be malignant (Table 4). These false-negative cases generally exhibited low-suspicion features according to conventional imaging classifications. The predicted malignancy risk was 62.29% (IQR: 32.50–82.57) for nodules classified as benign and 85.04% (IQR: 72.29–93.57) for those classified as malignant. The overall distribution of malignancy risk is illustrated in Fig. 3. Based on AUC analysis, the optimal cutoff value for malignancy prediction was identified as 69.9% (95% CI: 65.3–82.8%).

Table 3.

Confusion matrices comparing AI-Thyroid and K-TIRADS in AUS thyroid nodules.

Test results	Final diagnosis		P
	Benign nodule	Malignancy
K-TIRADS
Low/intermediate (2 + 3, 4)	86	30	0.037*
High (5)	15	34
AI-Thyroid
Benign	39	6	<0.001*
Malignant	62	58
Predicted risk of malignancy (%)	62.29 (32.50–82.57)	85.04 (72.29–93.57)	<0.001†

^*McNemar’s test was used to assess whether the pattern of disagreement between predictions and actual outcomes in the confusion matrix was statistically significant.

^† The Mann–Whitney U test was applied to compare the distribution of continuous scores (risk values) between benign and malignant groups.

Table 4.

Individual patient characteristics of malignant cases misclassified as benign by the AI model

Case	Age	Sex	K-TIRADS	Size (cm)	Imaging features on ultrasound
Case 1	61	M	Low (3)	1.3	Isoechoic solid component without suspicious features
Case 2	68	F	Low (3)	1.8	Isoechoic, partially cystic nodule with solid component, without suspicious ultrasound features
Case 3	30	M	Low (3)	2.5	Isoechoic solid component without suspicious features
Case 4	37	F	Low (3)	1.6	Isoechoic solid component without suspicious features
Case 5	59	F	Low (3)	2.2	Isoechoic solid component without suspicious features
Case 6	75	F	High (5)	4.1	Solid and hypoechoic nodule with non-parallel orientation

Figure 3.

Distribution of AI-predicted malignancy risk in AUS nodules. The histogram shows the number of patients across predicted malignancy risk scores ranging from 0 to 100% in 10% intervals. The binary classification threshold for the AI model was set at 50%, while the optimal cutoff value derived from the receiver operating characteristic (ROC) analysis was 69.9% (95% CI: 65.3–82.8), which is indicated in the figure.

Discussion

We aimed to investigate whether a deep learning model utilizing ultrasound images could provide additional diagnostic value for thyroid nodules initially classified as AUS on cytology. The AI model demonstrated performance comparable to that of conventional imaging interpretation systems and maintained consistent accuracy despite variability in ultrasound image quality due to differences in equipment and examiners. Notably, the model we used in this study exhibited a high sensitivity and negative predictive value, suggesting potential clinical utility in a rule-out strategy.

Indeterminate thyroid nodules, including those categorized as AUS (Bethesda III) and follicular neoplasm/suspicious for follicular neoplasm (Bethesda IV), pose a diagnostic challenge due to their inconclusive cytologic features with variable malignancy risk. Accurate risk stratification in this group is essential to avoid overtreatment while ensuring timely detection of malignancy. In this study, we focused specifically on AUS nodules because Bethesda IV nodules have distinct cytological and sonographic features, which may influence AI performance. Moreover, current clinical guidelines recommend diagnostic surgery for Bethesda IV nodules regardless of imaging findings, limiting the additional clinical decision-making value that an AI-based imaging tool could provide in this subgroup (5).

AUS nodules show wide variability in architectural and nuclear atypia, leading to substantial differences in malignancy risk assessment (12). The latest Bethesda update reports malignancy rates of 13–30% (13), whereas Korean surgical cohorts have shown higher rates (24.5–55.2%) (14, 15, 16), likely reflecting preferential selection of nodules with suspicious ultrasound features or marked cytologic atypia. In our multicenter cohort, the malignancy rate was 38.8%, determined through repeated FNA, core needle biopsy, and diagnostic surgery – an approach reflecting real-world practice. These findings confirm that AUS nodules carry a meaningful malignancy risk and underscore the need for reliable, non-invasive adjunctive tools, such as AI-based ultrasound interpretation, to improve diagnostic accuracy in this challenging cytologic category.

In previous studies, AI models have generally been reported to achieve accuracies exceeding 80% (10, 17). In fact, the AUC was 0.92 in the external validation test set in a study conducted by the developers of AI-Thyroid. However, in the cohort of the present study, the model showed a relatively lower predictive power with an AUC of 0.75, and the diagnostic performance of the traditional ultrasound-based system was similarly reported to be 0.76. These results were consistently confirmed in both K-TIRADS and ACR-TIRADS for nodules with AUS cytology in other studies (14, 18). Taken together, these findings suggest that the relatively low accuracy observed in our study is not unique to this model but rather reflects the inherent diagnostic challenges of AUS nodules.

In the current study, the AI-Thyroid model demonstrated a modest overall accuracy of 0.58, despite achieving an AUC of 0.75. The binary classification was determined using a 50% malignancy risk threshold; however, the optimal cutoff based on AUC analysis was approximately 70% (Fig. 3). This discrepancy between the set threshold and the optimal cutoff suggests that the present configuration of the model is intentionally oriented toward maximizing sensitivity in real-world clinical settings, where the primary goal may be to avoid missing malignancies rather than to achieve the highest possible overall accuracy. When AI tools are implemented as screening instruments to support clinical decision making, prioritizing higher sensitivity may be advantageous, as it allows a greater proportion of benign nodules to be safely ruled out.

The model achieved a sensitivity of 0.91 in this multicenter cohort of patients with AUS-diagnosed thyroid nodules, supporting its potential role as a complementary tool in clinical practice. Deferring additional diagnostic interventions for nodules predicted as benign by AI-Thyroid could have reduced additional biopsies by 27% (45 of 165 nodules) in this cohort. In diagnostic strategies, sensitivity and specificity exhibit a well-recognized trade-off relationship, in which increasing sensitivity often leads to a decrease in specificity (19). As a result, nodules classified as malignant by the AI model may include a proportion of false-positive cases. In this context, AI-based malignancy predictions should be interpreted cautiously and should not be used as a trigger for intensified diagnostic evaluation.

Of the nodules predicted as benign, six were ultimately malignant (Table 4). Most were assigned low K-TIRADS scores, reaffirming that imaging-based assessment alone cannot serve as a definitive diagnostic tool and that pathologic confirmation remains essential for the diagnosis of thyroid nodules. Notably, the only missed high-suspicion nodule in the false-negative cases measured 4.1 cm. This observation may partly reflect technical constraints of the AI model. Because the model analyzes ultrasound images using a fixed square input format, very large or markedly elongated nodules may not be fully encompassed within the analyzed region, potentially limiting feature extraction – such as assessment of margin shape or contour irregularity. Such image-cropping constraints may contribute to reduced diagnostic performance in larger nodules and help explain the size-dependent differences observed in this study. In the revised 2021 K-TIRADS, a 1.5 cm biopsy threshold can be applied regardless of detailed ultrasound characteristics (11), which was used as the criterion for subgroup analysis. In nodules <1.5 cm, the model achieved a sensitivity of 98% (Table 3), suggesting that the added diagnostic value of this AI model may be greatest in smaller, low-risk nodules.

This model operates in a semi-automated manner, meaning that human experts are required throughout the entire process – including proper ultrasound acquisition, operation of the AI model, and interpretation of its results (20). Such expert involvement may have contributed to the high sensitivity observed in this study. In fact, the current model may produce variable results depending on the scope and location of image cropping, underscoring the need for caution in its application (Supplementary Fig. S1 (see section on Supplementary materials given at the end of the article)). In addition, we adopted the higher malignancy risk score from the two analyses to mitigate view-dependent variability. Two ultrasound images – one in the transverse view and one in the longitudinal view, each including the longest axis for nodule size evaluation – were used. The discrepancy rate between binary predictions from the two views was 18.6% (Fig. 2). Although the AUC tended to be slightly higher in the longitudinal view, the difference was not statistically significant (Supplementary Fig. S2). We adopted the higher malignancy risk score from the two analyses, which may have further contributed to the high sensitivity observed in this study. Recent studies have investigated multi-view algorithms that simultaneously analyze two or more images of the same nodule (21, 22). Furthermore, approaches are being developed to improve predictive performance by integrating multi-modality data – such as computed tomography or genetic testing – alongside ultrasound images (23, 24).

To improve specificity, additional training on calcified nodules may be required, which could in turn improve the overall diagnostic accuracy of the current model. Calcification patterns – such as rim calcification, macrocalcification, and total calcification – are relatively common in thyroid nodules and can present a diagnostic challenge, as their ultrasound appearance may overlap between benign and malignant lesions. The developers previously reported that ultrasound images of totally calcified nodules were excluded from model training (10). We also observed notable errors in the recognition of colloid nodules and nodules with rim calcification (Supplementary Fig. S3), both of which can be misinterpreted as suspicious or non-suspicious depending on subtle imaging features. Addressing these gaps through targeted learning that incorporates a diverse range of calcification patterns could improve the model’s ability to correctly identify benign calcified nodules, thereby enhancing specificity and overall performance across a broader spectrum of thyroid nodule presentations.

This study has several limitations. While a single AI platform was applied in this analysis, the findings may not be directly generalizable to other AI algorithms. Diagnostic performance can vary depending on differences in training data composition, preprocessing techniques, or model architecture. In addition, the cohort size of the present study was relatively modest, and the findings should be interpreted as a preliminary validation exploring the feasibility of AI-based ultrasound assessment in thyroid nodules diagnosed as AUS. The study population comprised only patients with benign thyroid nodules or papillary thyroid carcinoma, while uncommon nodular lesions – such as those with extensive calcification, abscesses, or rare malignancies – were excluded. Nevertheless, given that the papillary type accounts for approximately 97% of thyroid cancer cases in Korea (25), the model is likely to be broadly applicable in clinical practice. Lastly, because this study was based on retrospective data with confirmed diagnoses, the possibility of bias cannot be entirely excluded. A prospective study with a longitudinal follow-up would therefore be valuable to assess the model’s clinical utility.

In conclusion, the AI-assisted tool using static ultrasound images demonstrated strong performance in evaluating thyroid nodules with AUS cytology and may serve as a useful complement to existing risk stratification systems. Further prospective studies across heterogeneous clinical environments are needed to assess generalizability and optimize its integration into real-world practice.

Declaration of interest

The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the work reported.

Funding

This work was supported by the Korean Thyroid Association Clinical Research Award (2023) and by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Korean government (Ministry of Education, RS-2023-00245534, and Ministry of Science and ICT, RS-2025-24683847).

Author contribution statement

JK and D-JL conceived and designed the study and drafted/critically revised the manuscript for important intellectual content. JL, JH, OK, K-HB, CMS, HAL, HK, IY, and MK acquired, analyzed, and interpreted the data. All authors discussed the results, contributed to manuscript revisions, and approved the final version.