The future of thyroid nodule risk stratification

Introduction

Over the last two decades the use of sonographic risk stratification systems (SRSS) has become an integral part in the evaluation of patients with thyroid nodules.^1–4 The aim of these systems is to standardize the evaluation of thyroid cancer risk according to ultrasound features, in order to individualize the management of patients with thyroid nodules.^1–3,5,6 Clinical evidence and practice guidelines further support the routine use of risk stratification, as it can decrease the number of unnecessary thyroid biopsies and improve reporting of thyroid cancer risk while avoiding missed diagnoses of clinically relevant thyroid cancer.^7–10

More recently, artificial intelligence (AI) methodologies have been applied to improve thyroid cancer risk stratification and the care of patients with thyroid nodules with promising results.^11,12 This article aims to review ultrasound features, SRSS and current clinical evidence incorporating AI in the diagnostic evaluation of patients with thyroid nodules.

Association of thyroid nodules ultrasound features and thyroid cancer

Certain individual US features are specific for malignancy while others are associated with benign histology.^1,2,13 US features of benign and malignant nodules may overlap, limiting their diagnostic properties as a single “rule-in” or “rule-out” test for thyroid cancer diagnosis. However, they are extremely useful to stratify the risk for thyroid malignancy.^1,2,13 Suspicious features include micro-calcifications, local invasion, taller-than-wide shape, irregular/infiltrative margins, and markedly reduced echogenicity. However, the absence of a halo, ill-defined margins, solid composition, and vascularity, are less specific.^14–16 Cystic nodules and spongiform nodules have a high diagnostic odds ratio to predict benign nodules.¹⁷ While a solitary suspicious finding may be associated with variable rates of cancer, the accumulation of suspicious findings significantly increases the malignancy risk. Indeed, a retrospective study of 1658 thyroid nodules measuring ≥ 1 cm showed in a univariate analysis that there was a significant association of malignancy with a single suspicious ultrasound feature and on multivariate analysis the malignancy risk was further increased with an escalating number of high risk sonographic features.¹⁸

Table 1 summarizes thyroid ultrasound features, their associated histologic findings and diagnostic odds ratios to predict benign or malignant thyroid nodules.^2,13,15,19

Table 1:

Thyroid ultrasound features, definitions, and diagnostic odds ratio

US feature	Definition	Descriptors	Histological background	Diagnostic Odds Ratio (DOR)13
Composition	Proportion of solid tissue and fluid in a nodule	Solid Predominantly solid Predominantly cystic Cystic	Hyperplastic nodules have abundant colloid which appears cystic on sonography. Neoplasms can undergo cystic degeneration and have cystic areas.	Solid (DOR-M) (4.45, 2.63–7.5) Cystic (DOR-B) (6.78, 2.26–20.3) Spongiform (DOR-B) (12, 0.61–234.3)
Echogenicity	Brightness relative to normal thyroid parenchyma	Hyperechoic Isoechoic Hypoechoic Very hypoechoic	Classic PTC and MTC produce less acoustic interfaces than micro-follicles and may appear hypoechoic	Hypoechoic (DOR-M) (4.5, 3.2–6.4) Isoechoic (DOR-B) (3.6, 2–6.3)
Borders	Interface between nodule and surrounding thyroid parenchyma	Smooth Ill-defined Irregular Infiltrative Extra-thyroidal extension	Infiltrative or lobular borders may be concerning for invasive thyroid carcinoma. Hyperplastic nodules may have ill-defined borders due to similar echogenicity to surrounding parenchyma, which may not be suspicious for malignancy	Infiltrative margins (DOR-M) (6.89, 3.35–14.1)
Shape	Evaluated in the axial plane by comparing the height (“tallness”) and width of a nodule measured parallel and perpendicular to the ultrasound beam	Taller-than-wide Wider-than-tall	Disproportionate growth in anteroposterior dimension is considered an aggressive growth pattern	Taller-than-wide (DOR-M) (11.14, 6.6–18.9)
Echogenic foci	Hyperechoic foci	Large comet tail artifacts Macrocalcifications Peripheral rim calcifications Punctate echogenic foci	Large comet tail artifacts are associated with colloid. Punctuate echogenic foci have been associated with Psammoma bodies, which may be associated with PTC.	Internal microcalcifications (DOR-M) (6.78, 4.48–10.24)

DOR-M, diagnostic odds ratio to predict malignant nodules and DOR-B, diagnostic odds ratio to predict benign nodules, presented as estimate and 95% Confidence Interval. PTC, papillary thyroid cancer; MTC, medullary thyroid cancer.

Contemporary thyroid ultrasound risk stratification systems - overview

Numerous SRSS exist with the goal of standardizing the evaluation of patients with thyroid nodules.^1,3,20,21 Fundamentally, these systems aim to separate thyroid nodules according to thyroid cancer risk and propose a nodule size at which biopsy should be recommended, according to the estimated risk for thyroid cancer.^1,3,5,21

Evaluation of individual thyroid ultrasound features provides insight into the nature of the underlying lesion and its estimated cancer risk. Moreover, the presence of ultrasound features suspicious for malignancy are rarely independent of each other.¹⁵ For this reason, establishing sonographic patterns to aid the management of patients with thyroid nodules is based on the concept that sonographic features are present concurrently. While pattern-based SRSS are associated with improved inter-observer variability in assigning risk classification¹, these systems^1,3,22 have the disadvantage that 5–17% of nodules are non-classifiable.^23,24 Therefore, point based systems, such as the American College of Radiology Thyroid Imaging Reporting & Data System (ACR TIRADS) have the advantage of being inclusive of all nodule types.² However, classification of nodules using the ACR TIRADS is limited by interobserver variability in assigning scores for each feature and can be time-consuming in the setting of classifying multiple nodules.²

Table 2 summarizes contemporary thyroid nodule SRSS, their diagnostic performance and clinical implications.^{1–3,5,7,21,23,25,26}

Table 2:

Diagnostic performance of SRSS

SRSS	System type	Categories	Biopsy cut-off	POM5	Unnecessary biopsy rate7 (95% CI)	Missed malignancy rate23 %	Sensitivity25 (95% CI)	Specificity25 (95% CI)	Likelihood ratio for positive results (LR+)25 (95% CI)	DOR25 (95% CI)	Inter-observer agreement26
ATA	Ultrasound pattern based	Benign	No FNA	0	51% (44–58)	4.1%	87% (75–94)	31% (24–40)	1.2 (1.0–1.4)	3.1 (1.3–7.1)	0.5 (0.4–0.6)
		Very low risk	> 2cm or no FNA	<3%
		Low risk	1.5 cm	5–10%
		Intermediate risk	≥ 1 cm	10–20%
		High Risk	≥ 1 cm	70–90%
ACR-TIRADS	Weighted point-based system	Benign (TIRADS-1)	No FNA	<2%	25% (22–29)	2.2%	74% (61–83)	64% (56–70)	1.9 (1.6–2.3)	4.9 (3.1–7.7)	0.5 (0.4–0.6)
		Not suspicious (TIRADS-2)	No FNA	<2%
		Mildly suspicious (TIRADS-3)	≥ 2.5 cm	5%
		Moderately suspicious (TIRADS-4)	≥ 1.5 cm	5–20%
		Highly suspicious (TIRADS-5)	≥ 1 cm	>20%
ETA	Ultrasound pattern based	EU-TIRADS 1 - normal	-	0	38% (16–66)	3.2%	54% (51–57)	53% (51–55)	1.4 (1.0–1.8)	2.2 (0.9–5.1)	0.6 (0.5–0.6)
		EU-TIRADS 2- benign	No FNA	0
		EU-TIRADS 3- low risk	> 2 cm	2–4%
		EU-TIRADS 4- intermediate risk	> 1.5 cm	6–17%
		EU-TIRADS 5- high risk	> 1 cm	26–87%
AACE/ ACE/ AME	Ultrasound pattern based	Low risk	> 2 cm, increase in size, clinical features	1%	N/A	2.9%	74% (71–78)	53 (51–55)	1.5 (1.1–2.1)	3.1 (1.0–9.4)	0.4 (0.4–0.6)
		Intermediate risk	> 2 cm	5–15%
		High risk	≥ 1 cm	50–90%
K-TIRADS	Ultrasound pattern based	K-TIRADS 1 – no nodule	NA	-	55% (42–67)	3.5%	86 (73–94)	28 (20–38)	1.2 (1.0–1.4)	2.5 (1.1–5.5)	0.5 (0.4–0.6)
		K-TIRADS 2 – benign	≥ 2 cm	<3%
		K-TIRADS 3 – low suspicion	≥ 1.5 cm	3–15%
		K-TIRADS 4 – Intermediate suspicion	≥ 1 cm	15–50%
		K-TIRADS 5 – High suspicion	≥ 1 cm (>0.5 cm selective)	>60%

AACE, American Association of Clinical Endocrinologists; ACE, American College of Endocrinology; ACR, American College of Radiology; AME, Associazione Medici Endocrinologi (AME); ATA, American Thyroid Association; CI, confidence interval; DOR, diagnostic odds ratio; ETA, European Thyroid Association, ETA; EU-TIRADS, European Thyroid Imaging Reporting and Data System; FNA, fine needle aspiration; K-TIRADS, Korean Thyroid Imaging Reporting and Data System; LR, likelihood ratio; TIRADS, - Thyroid Imaging Reporting and Data System; N/A, not available; POM, prevalence of malignancy

Clinical benefits of thyroid nodule risk stratification

The use of thyroid nodule SRSS has revolutionized the management of patients with thyroid nodules by 1) improving the quality of thyroid ultrasound assessment and communication between clinicians, 2) potentially decreasing the number of unnecessary biopsies and 3) using thyroid cancer risk to personalize patient care.

While the details included in thyroid and neck ultrasound reports are variable, SRSS provide a means to standardize reporting and communication among clinicians. For example, a retrospective study of ultrasound reports of 478 thyroid nodules, found that most reports did not include the size of the dominant nodule (71%), comment on suspicious features (91%) or the presence of suspicious lymph nodes (83%), with 46% of the reports lacking a description of malignancy risk.²⁷ Multiple studies have shown a positive effect of implementation of thyroid nodule risk stratification in the quality of the ultrasound report, with increased inclusion of ultrasound features of interest, thyroid cancer risk and suggestions of management strategies to address this risk.^10,28,29

Moreover, better reporting of thyroid ultrasound features and cancer risk translates to improved clinical care. For example, observational data suggest that strict implementation of recommendations for thyroid biopsy according to the different SRSS can result in decreased rates of unnecessary biopsies.^5,7,23,30 However, most commonly, these studies consider nodules with benign cytology as unnecessary. This is a rather narrow definition, given that the necessity and appropriateness of a thyroid biopsy depends on additional factors beyond thyroid nodule risk and size. ^5,6,31

Comparisons of current thyroid nodule risk stratification systems

Castellana et al performed a systematic review and meta-analysis to compare five thyroid nodule SRSS and found that American Thyroid Association (ATA) guidelines had the highest pooled sensitivity and ACR-TIRADS the highest pooled specificity.²⁵ Moreover, when performing a direct comparison analysis, the diagnostic odds ratio for selecting patients for thyroid biopsy was higher for ACR-TIRADS than for ATA (5.6 vs 2.9 p=0.002) or Korean Society of Thyroid Radiology’s Thyroid Imaging Reporting and Data System (K-TIRADS) (4.5 vs 2.5, p=0.002). The review included 12 studies and 18,750 thyroid nodules.

Kim et al performed a systematic review of 8 articles including 13,092 nodules and compared unnecessary biopsy rates among four thyroid nodule SRSS.⁷ ACR-TIRADS had the lowest unnecessary biopsy rates (25%). However, low unnecessary biopsy rates may exist at the expense of higher missed malignancy rates.^5,32 ACR-TIRADS is associated with the higher proportion of malignant nodules that do not receive a recommendation for biopsy; however, for nodules that do not meet biopsy criteria but carry a relevant risk of malignancy, active surveillance instead of biopsy is recommended, potentially decreasing the risk of a significant adverse outcome.³³ Overall, SRSS improve the prediction of benign or malignant disease, with high risk categories having a strong association to malignant or suspicious histology.^20,23,25,34

Size threshold

Current guidelines offer variable size cutoffs according to risk stratification category, with a lower threshold for biopsy as the risk for thyroid cancer increases, although limited clinical evidence guides the selection of these thresholds.^1–3,21 In fact, the selection of size thresholds for thyroid nodule fine needle aspiration (FNA) has remained controversial given that 1) the association of thyroid nodule size and malignancy risk is not linear (but could result in higher risk for false negative cytology) and 2) size only partly correlates with thyroid cancer prognosis.^5,35–38 Machens et al reported increased cumulative risk for distant metastases from papillary thyroid cancer and follicular thyroid cancer above 2 cm.³⁹ Based on this, ACR-TIRADS selects a slightly higher threshold for biopsy in the lower risk categories of 2.5 cm, after considering the reported discrepancy of pathologic tumor size and estimated sonographic size.²

Ha et al evaluated the effect of size thresholds using the current biopsy indications for ATA, K-TIRADS, and ACR-TIRADS, in 3323 consecutive thyroid nodules. Overall, the sensitivity for detecting malignancy was higher for ATA and K-TIRADS compared to ACR-TIRADS at 89.6% and 94.5%, versus 74.7%, respectively. ACR-TIRADS, on the other hand, was associated with higher specificity (67.3%) compared to ATA (33.2%) and K-TIRADS (26.4%). This improved specificity with ACR-TIRADS translated to the lowest rate of unnecessary FNA at 25.3%, in comparison with ATA (51.7%) and K-TIRADS (56.9%). However, if a higher threshold for thyroid biopsy of 1.5 cm for intermediate risk thyroid nodules (instead of the existing 1 cm threshold) and of 2.5 cm for low suspicion thyroid nodules (as opposed to the existing 1.5 cm threshold) were used, the specificity and rates of unnecessary biopsies for this ‘revised’ ATA system would be similar to those for ACR-TIRADS, highlighting the effects of size thresholds on the performance of these SRSS.^5,40,41

Limitations of thyroid nodule risk stratification

The clinical evidence supporting the prevalence of malignancy associated with the different thyroid cancer risk categories is mostly derived from observational studies that are enriched with malignant cases, which can lead to over-estimation of the diagnostic performance.^5,42 In addition, SRSS have been validated mostly among thyroid nodules potentially harboring PTC, however the diagnostic performance for patients with medullary or follicular thyroid carcinoma might differ.^25,43 Another limitation when using risk stratification in practice is the inability to classify all nodules (when using a pattern-based system), with an unknown percentage of these nodules harboring malignancy.^34,43,44

Structured evaluation and report of thyroid ultrasounds requires expertise and availability of clinical time, both potential barriers to implementation.¹⁰ Similarly, reproducibility of ultrasound assessments is pivotal, yet most of the clinical evidence reporting moderate interobserver agreement is derived from small group of expert thyroid examiners. ^26,45–48

Finally, another potential unintended consequence of the implementation of thyroid nodule risk stratification is the over-reliance in the radiology report and recommendations (of observation and/or biopsy), that may not take into consideration the clinical situation of the patient.⁶

Clinical factors and risk stratification

Because thyroid nodules are such a common clinical entity, individual risk stratification based on patient characteristics is an important component of the evaluation. Various clinical and demographic factors should be assessed to further refine the malignancy risk and aid in the next steps of clinical management.

Age at diagnosis of a nodule influences the risk that any nodule is cancerous; younger age is associated with an increased malignancy risk.^49,50 In a single center study of 6391 adult patients with thyroid nodules, the risk of malignancy was highest (22.9%) in the group aged 20–30 years, and declined by 2.2% per year until the risk reached a plateau of 12.6% at age 60. It is important to note, however, that in spite of the overall lower likelihood of malignancy in the older patients, there was a greater risk of a more aggressive histologic phenotype when a cancer was identified.⁴⁹

Sex also plays an important role in risk stratification of patients with thyroid nodules. Although not consistently reported,^51,52 a greater risk of malignancy in those with a thyroid nodule has been suggested in men compared to women.⁵³ In the largest reported experience with thyroid nodules, a single-study center of 20,001 consecutive thyroid nodules determined that the odds ratio for malignancy in men was 1.7 (1.5–1.9, CI, p<0.0001), compared to women.⁵⁴

While the etiology of most thyroid malignancies is unclear, a small proportion of differentiated cancers may be attributable to an inherited condition. Because thyroid cancer may be seen in up to 10–15% of the population at autopsy,^55,56 the occurrence of two cases of thyroid cancer among first degree relatives in most families is attributable to chance, rather than a genetic predisposition.⁵⁷ However, the identification of nonmedullary thyroid cancer in three or more first-degree relatives is associated with a 96% chance of an inherited condition.⁵⁷

Exposure to radiation in childhood is a well-recognized cause of differentiated thyroid carcinoma and the impact may be seen for decades after therapy.^58–60 Additionally, those patients exposed during childhood or adolescence to ionizing radiation from a nuclear fallout are at increased risk of malignancy.⁶¹ Consequently, clinicians should consider these risk factors in the decision-making process of whether to perform FNA.¹

Another important consideration in the evaluation of a patient with a thyroid nodule is the presence of comorbid conditions. It is well recognized that thyroid cancers are often associated with an indolent clinical course. If the patient has a low-risk thyroid nodule in the setting of another, more aggressive primary malignancy or uncontrolled chronic condition, consideration may be given to postponing the evaluation of an identified thyroid nodule. A recent study examined the likelihood of identification of a clinically significant thyroid cancer (defined as anaplastic, medullary, or poorly differentiated carcinoma, or the presence of distant metastases) in a cohort of 1129 patients over the age of 70 years. They found only 17 (1.5%) significant risk thyroid cancers (SRTC) among 2527 nodules, all of which were readily identifiable by initial imaging and cytology. During a median follow up of four years, there were only 10 (0.9%) thyroid cancer deaths, all of which occurred in patients with SRTC. Interestingly, they found that among all other patients, there were 160 deaths (14.4%), and there was a more than two-fold increased risk of death among those with a separate non-thyroidal malignancy or coronary artery disease at the time of nodule evaluation. The authors concluded that the evaluation of thyroid nodules in patients over the age of 70 years without high risk imaging or cytologic findings should be tempered in the setting of significant comorbid illness.⁶²

Artificial intelligence and thyroid nodule risk stratification

Background and definition of artificial intelligence

AI refers to the imitation of human abilities such as learning and problem solving by computer systems.^63–65 Although the idea of AI is not new, the application to enhance clinical care has expanded recently given the availability of computers that are fast and able to analyze large datasets.^65,66 Within AI, machine learning is the use of mathematical operations to solve problems and for systems to learn from the evaluation of data over time.^65,67 Machine learning algorithms can be classified according to how the data is provided and whether the outcome of interest is labeled during the development of the model.^65,67,68 Specifically, in thyroid nodule risk stratification, the use of images in which the ultrasound features of interest have been manually extracted by a human operator (e.g. echogenicity, composition) is considered conventional machine learning. On the other hand, if thyroid nodule images are analyzed as raw data (pixels) by the computer model, this is considered deep learning. Similarly, if the nodule is labeled as thyroid cancer or a benign nodule during the model development, this is considered supervised machine learning where the goal is for the system to learn to classify/predict the desired outcome (e.g. malignancy). Alternatively, if the thyroid nodule is not labeled for the outcome of interest (thyroid cancer or benign), this is considered unsupervised learning and the goal is the identification of new clusters/groups (Figure 1).^67–69

Figure 1.

Structure of conventional machine learning and deep learning (according to human or computer model extraction of the images or pixels) for thyroid nodule risk stratification. Supervised machine learning as during development classification/prediction of thyroid cancer/benign is provided to the model.

A particular modality of interest for imaging-based diagnosis is computer-aided diagnosis (CAD). These methods are based on machine learning models for the evaluation of images and can be used to complement and enhance the human operator assessment.⁷⁰ Augmented intelligence refers to AI models that seek to support human cognition, instead of replacing it. In addition, models of “human in the loop” refer to models in which the outputs from machine learning algorithms require the review from human operators.⁶⁴

Overview of how AI can improve ultrasound risk stratification

Thyroid nodule risk stratification allows clinicians to estimate the risk for thyroid cancer and personalize the care of patients with thyroid nodules.^6,31 However, concerns in terms of thyroid nodule ultrasound feature assessment reproducibility, time requirement for these assessments and overall precision of the current frameworks can limit the clinical benefits.^10,33,71 The use of AI can help resolve these concerns by: 1) improving our frameworks for thyroid nodule risk stratification and 2) increasing the automation of thyroid nodule risk stratification.^11,69,72 In addition, these systems can be used in areas in which experts in thyroid ultrasonography are not available, possibly improving the quality of care across different healthcare systems.⁷²

1). Improving frameworks for thyroid nodule risk stratification

The use of machine learning algorithms offers an opportunity to re-evaluate the weight of the ultrasound features that we currently use for thyroid cancer risk stratification and to increase the number of variables that are evaluated.¹¹

For example, Zhang et al aimed to develop a machine learning model for the diagnosis of thyroid nodules, based on the evaluation of 11 ultrasound features that were manually extracted and real-time elastography.⁷³ Their model was developed using a sample of 2064 nodules with histological diagnosis of thyroid cancer in 36%. The diagnostic properties were compared to a radiologist. The machine learning model was able to integrate information from the provided variables, with higher weight given to the presence of calcification, hypoechoic halo and elastography.⁷³ In fact, the random forest algorithm had a higher area under the curve than the radiologist (AUC = 0.92 [95% confidence interval (CI) 0.90–0.95] vs. 0.83 [95% CI 0.82–0.85]).⁷³ Interestingly, size was included as a potential variable and was given lower importance by the machine-learning model, highlighting the need to understand potential biases in the datasets used to train machine-learning models and the importance of collaboration between expert clinicians and machine-learning data scientists. The evidence that links thyroid nodule size and malignancy risk is somewhat controversial and instead, this variable may provide more prognostic information.^{5,35,36,39,74}

Similarly, the study by Wildman-Tobriner et al, aimed to optimize the diagnostic performance of ACR-TIRADS using AI.⁷⁵ A dataset of 1425 biopsy proven thyroid nodules (malignancy rate of 11%) was evaluated for the five ACR-TIRADS components and a genetic AI algorithm used to optimize the classification.⁷⁵ The optimized AI ACR-TIRADS, assigned new weights to 8 features and for 6 the new system assigned 0 points. The optimized AI ACR- TIRADS had similar receiver operator curves and sensitivity, but higher specificity than the radiologists using the original ACR-TIRADS.⁷⁵

Multiple additional studies have explored the use of AI to improve our understanding of what ultrasound features are associated with thyroid cancer risk, what weight should be assigned to each of the features and taking advantage of the ability of the algorithms to integrate variables in order to provide a more accurate and precise estimate of thyroid cancer.^11,12 Future studies, should include large datasets of patients with thyroid nodules, with different risk for thyroid cancer and representative of the populations of interest, in order to take full advantage of AI methods and enhancing thyroid cancer risk estimation.

2). Increasing automation of thyroid nodule risk stratification

Another barrier to routine implementation of thyroid nodule risk stratification is the need for accurate and reproducible assessment of thyroid nodule features.^33,71 These assessments have shown that highly trained evaluators and time are important considerations that can affect reproducibility, given that ultrasound assessment is an operator-dependent task.^10,48 AI algorithms could help solve this clinical gap by supporting highly reproducible, fast and automated thyroid nodule risk stratification.

Xu et al, conducted a systematic review that included 19 studies (4781 nodules) assessing the diagnostic performance of machine learning algorithms for the diagnosis of thyroid nodules.⁷⁶ Only the results from the external validation cohorts were included in the analysis. There were 6 studies that evaluated conventional machine learning and 13 that evaluated deep learning models. Overall, the studies were considered to be at moderate risk of bias and the prevalence of thyroid cancer cases was high.⁷⁶ The conventional machine learning models had a sensitivity of 0.86 (95% CI 0.79-.92), specificity 0.85 (95% CI 0.77–0.91), and diagnostic odds ratio (DOR) 37.4 (95% CI 24.9–56.2). Models based on deep learning had a sensitivity of 0.89 (95% CI 0.81–0.93), specificity 0.84 (95% CI 0.75–0.90), DOR 40.9 (95% CI 18.1–92.1). The performance of deep learning based algorithms was comparable to the diagnostic performance of radiologists [DOR 40.1 (95%CI 15.6–103.3) vs DOR 44.9 (95%CI 30.7–65.6), respectively].⁷⁶ These findings are consistent with the systematic review by Chambara et al, that evaluated 14 studies of CAD models for thyroid nodules, demonstrating comparable diagnostic performance of the AI models to human evaluators.⁷⁰ These reviews suggest good diagnostic performance of machine learning algorithms for the identification of thyroid cancer.⁷⁶

The study by Li et al, was a large multicenter study conducted in China including >300,000 images from 17 627 patients with thyroid cancer and 25 325 patients with benign thyroid nodules.⁶⁹ The area under the curve in each external validation site was 0.91 (95% CI .87–0·96) and 0.91 (95% CI 0.89–0.93). Moreover, the sensitivity of the machine learning model was 84.3% vs 92.9% (p=0.048) in one site and 84.7% vs 89% (p=0.25), while the specificity was 86.9% vs 56.1% (p<0·0001) and 87.8% vs 68.6% in the second site (p<0·0001), when compared to radiologists.⁶⁹

Thomas et al, developed a similarity algorithm for thyroid nodule risk stratification and used heat maps to attempt to explain the algorithm output. Heatmaps are visual representation that can help identify the areas (in an image in this case) that were used to support the output of the AI model.⁷⁷ Their development cohort included 2025 images from 482 patients and the overall malignancy prevalence was 32%.⁷⁷ In this model, the physician is always “in the loop” as they select the images of interest and are presented with the final output that they can accept/reject. The similarity model was used to classify test images and demonstrated an accuracy of 81.5%.⁷⁷ Interestingly, the heatmap was not able to completely explain the rationale for the model outputs.⁷⁷

This body of evidence suggests that machine learning algorithms can enhance thyroid nodule risk stratification by analyzing thyroid nodule images without the need for expert extraction of ultrasound features.

Lessons learned and next steps for the use of AI for thyroid nodule risk stratification

The ultimate goal of applying AI methods to improve thyroid nodule risk stratification is to improve the care of patients with thyroid nodules (Figure 2). The preliminary evidence presented suggests reasonable diagnostic performance of machine learning algorithms. However, a few concerns arise. Similar to the existent thyroid nodule diagnostic literature, the datasets that are used to train these models suffer from bias related to: high prevalence of thyroid cancer (improves diagnostic performance), over-representation of papillary thyroid cancer, variable gold standards and inappropriate exclusions.^5,11,42 These biases cannot be completely overcome by large datasets or by advanced machine learning algorithms and limit the validity and generalizability of findings.⁷⁸

Figure 2.

Potential applications of AI to improve thyroid nodule risk stratifications, areas that need to be addressed in future studies in order to achieve the goal of improved clinical outcomes.

A large number of studies have focused on deep learning algorithms, but our understanding of the features and/or rationale used by the algorithm to reach conclusions continues to be poorly understood for most models.¹² This “black box” and limited interpretability significantly hinder the reliability, trust and adoption of these methods in practice.⁷⁹

Similarly, as is the case for other areas of AI for diagnosis based on imaging, there is paucity of randomized, controlled trials focused not only on diagnostic performance but that also assess effects on clinical care and outcomes (i.e. increased reproducibility, accuracy/discussion of thyroid cancer risk, unnecessary biopsies). Although most models have focused on providing a dichotomous answer of benign/malignant classification for thyroid nodules, models that provide an estimate of thyroid cancer risk can also assist and support personalization of care for patients with thyroid nodules.⁷⁹ Moreover, the possible unintended consequences of implementation of these models in clinical practice have rarely been assessed (i.e. Clinician/patient distrust, legal implications).^64,68

Summary

Ultrasound risk stratification is an essential tool in the management of patients with thyroid nodules. The application of AI to thyroid nodule risk stratification can help improve the diagnostic accuracy of current SRSS and the reproducibility of assessments. The results of preliminary studies using AI to enhance thyroid nodule risk stratification are promising, supporting the need for future research that moves from diagnostic performance to effects on clinical outcomes.

Key points.

• Sonographic risk stratification of thyroid nodules is the first step to personalize the care of patients with thyroid nodules

• Multiple systems are available for sonographic risk stratification of thyroid nodules, each associated with different advantages and limitations

• The use of artificial intelligence to improve thyroid cancer risk estimation and assessment reproducibility is promising

• Future studies using artificial intelligence should focus on improving patient outcomes and use rigorous scientific methods

Synopsis.

Clinical evidence supports the association of ultrasound features with benign or malignant thyroid nodules and serves as the basis for sonographic stratification of thyroid nodules, according to an estimated thyroid cancer risk. Contemporary guidelines recommend management strategies according to thyroid cancer risk, thyroid nodule size and the clinical scenario. Yet, reproducible and accurate thyroid nodule risk stratification requires expertise, time and understanding of the weight different ultrasound features have on thyroid cancer risk. The application of artificial intelligence to overcome these limitations is promising and has the potential to improve the care of patients with thyroid nodules.

Clinics Care Points –

• The use of risk stratification systems in the evaluation of thyroid nodules can lead to a decrease of unnecessary thyroid biopsies without missed diagnosis of clinically relevant thyroid cancer

• Reproducible and accurate risk stratification of thyroid nodules can be time-consuming and requires expertise, which represent potential limitations for widespread implementation

• Deep learning models for thyroid nodule risk stratification have shown good diagnostic properties for thyroid cancer, yet their impact on patient care requires further evaluation