Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners

Abstract

Purpose

Computer-aided diagnosis (CAD) may improve interobserver agreement in the risk stratification of thyroid nodules. This study aims to evaluate the performance of the Korean Thyroid Imaging Reporting and Data System (K-TIRADS) classification as estimated by an expert radiologist, a senior resident, a medical student, and a CAD system, as well as the interobserver agreement among them.

Methods

Between July 2016 and 2018, 107 nodules (size 5–40 mm, 27 malignant) were classified according to the K-TIRADS by an expert radiologist and CAD software. A third-year resident and a medical student with basic imaging training, both blinded to previous findings, retrospectively estimated the K-TIRADS classification. The diagnostic performance was calculated, including sensitivity, specificity, positive and negative predictive values, and the area under the receiver operating characteristic curve.

Results

The CAD system and the expert achieved a sensitivity of 70.37% (95% CI 49.82–86.25%) and 81.48% (61.92–93.7%) and a specificity of 87.50% (78.21–93.84%) and 88.75% (79.72–94.72%), respectively. The specificity of the student was significantly lower (76.25% [65.42–85.05%], p = 0.02).

Conclusion

In our opinion, the CAD evaluation of thyroid nodules stratification risk has a potential role in a didactic field and does not play a real and effective role in the clinical field, where not only images but also specialistic medical practice is fundamental to achieve a diagnosis based on family history, genetics, lab tests, and so on. The CAD system may be useful for less experienced operators as its specificity was significantly higher.

Introduction

Thyroid nodules are commonly found during imaging of the neck [1, 2], but only a small proportion of these lesions subsequently prove to be clinically significant [3]. Nowadays, neck ultrasonography (US) is used to guide decisions regarding fine-needle aspiration (FNA) cytology or serial follow-up. Nodules should be carefully selected for FNA biopsy [4, 5] because of the vast number of subjects concerned, the potentially inconclusive results (both non-diagnostic [6] and indeterminate), and the risk of overdiagnosis of low-risk cancers [7]. However, while some US features are associated with nodule malignancy, the diagnostic accuracy is limited, and substantial interobserver variability has been documented [8–19].

Several classification systems that combine various US findings have been developed to estimate the likelihood of malignancy and select nodules for FNA biopsy. The application of these systems, which are endorsed by international scientific societies [1, 20–23], has proven to reduce interobserver variability, but there is room for further improvement [24]. To this end, using artificial intelligence, CAD has been proven to differentiate malignant from benign nodules, with an accuracy rate similar to expert radiologists [25–28] and may also reduce intra- and interobserver variability.

We performed a prospective analysis of sonographic examinations to evaluate the diagnostic performance of the K-TIRADS classification as estimated by an expert attending radiologist, a senior resident, a medical student, and a CAD system (S-Detect™), as well as the interobserver agreement among them. Furthermore, we evaluated the interobserver agreement between S-Detect™ and the expert radiologist in the evaluation of single sonographic features.

Methods

Cases

Between July 2016 and July 2018, 555 nodules were consecutively examined at the Head and Neck Radiology Unit of Policlinico Umberto I, “Sapienza” University of Rome (Italy), by a single radiologist. Patients were enrolled if they had no more than three thyroid nodules. Target nodules were submitted to cytological examination or thyroidectomy. The exclusion criteria were cystic lesions, nodules smaller than 5 mm (to avoid misinterpretations of the pathology report), examinations performed on an appliance unequipped with CAD software, or cytopathology provided by external services. The original examination was performed by a single attending radiologist with 18 years of experience in thyroid imaging using a Samsung RS80 system equipped with a 7–15 MHz linear probe and the S-Detect™ software. All nodules were characterized in terms of size, shape, margins, composition (solid, mixed, or cystic content), echogenicity, calcifications, hyperechoic foci, vascularity, and extrathyroidal extension. The operator then arranged the nodules according to the K-TIRADS [23] classification. Afterward, the S-Detect™ software was used to automatically determine the shape, composition, echogenicity, and margins. Other features, such as the presence of calcifications, stiffness according to elastosonography, and color Doppler vascular pattern, had to be input manually. The S-Detect™ system is semi-automatic since it requires the operator to select a region of interest (ROI), and then it performs segmentation and recognition of the nodule boundaries. If the process was incorrectly performed, the operator has to repeat it to obtain a correct recognition of the nodule.

All images were stored in a picture archiving and communication system (PACS) for retrospective analysis. A resident with three years of experience and a medical student with basic thyroid imaging training, both blinded to all clinical, pathological, and S-Detect™ findings, provided their estimation of the K-TIRADS classification for each nodule.

Reference standard

A composite reference standard was used: the final histology for nodules undergoing thyroid surgery and FNA cytology for patients with benign cytology findings. In this latter case, a nodule stability confirmed by at least a 12-months follow-up was also required. FNA specimens were prepared as direct smears and assessed by two expert cytopathologists (V. A. and D. B.) in accordance with national guidelines [29].

Statistical analysis

The classifications (K-TIRADS 2–3 were considered as test negative and K-TIRADS 4–5 as positive) were compared with the reference standard to estimate the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic (AUROC) curve, each with 95% confidence intervals (CIs) of each classification. Interobserver agreement was assessed with Krippendorff’s alpha [30] for analyses involving ordinal data and more than two observers and with Cohen’s kappa for analyses involving dichotomous variables and two observers. Values less than 0.20 were considered indicative of slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–1.00, near-perfect agreement [31]. Sensitivity and specificity were compared using McNemar’s test [32]. Data were analyzed using the IBM SPSS Statistics package, version 25.0 (IBM Corp., Armonk, New York, United States). AUROC was computed and compared using the easyROC package [33]. Institutional Review Board approval and informed consent was obtained from all subjects.

Results

The final cohort was composed of 76 patients (56 females and 20 males; mean age 55 years; Fig. 1) and a total of 107 nodules (size range 5–40 mm). Among them, 27 were malignant (25.2%; 17 papillary thyroid cancers, six follicular variant papillary thyroid cancers, three medullary thyroid cancers, and one poorly differentiated carcinoma), and 80 were benign (74.8%; 22 histologically confirmed and 58 diagnosed based on cytology reports and follow-ups). The sensitivity, specificity, predictive values, and AUROC are reported in Table 1.

Fig. 1.

Inclusion of the thyroid nodules in the final analysis

Table 1.

Sensitivity, specificity, predictive values, and area under the receiver operating characteristics curve of the K-TIRADS system evaluated by S-Detect™ and three clinicians (expert attending radiologist, resident, and medical student)

	Sensitivity	Specificity	PPV	NPV	AUROC
S-Detect™	70.37% (49.82–86.25%)	87.50%‡ (78.21–93.84%)	65.52% (45.67–82.06%)	89.74% (80.79–95.47%)	0.79 (0.69–0.88)
Expert	81.48% (61.92–93.7%)	88.75%† (79.72–94.72%)	70.97% (51.96–85.78%)	93.42% (85.31–97.83%)	0.85 (0.77–0.93)
Resident	74.07% (53.72–88.89%)	85.00%* (75.26–92.0%)	62.50% (43.69–78.9%)	90.67% (81.71–96.16%)	0.80 (0.7–0.89)
Student	70.37% (49.82–86.25%)	76.25% (65.42–85.05%)	50.00% (33.38–66.62%)	88.41% (78.43–94.86%)	0.73 (0.63–0.83)

*The specificity of the student is significantly lower than that of the expert and S-Detect™ (*p = 0.16 vs. the resident; ^†p = 0.02 vs. the expert; ^‡p = 0.022 vs. S-Detect™); no significant difference was reported between S-Detect™ and the more experienced examiners

The overall discriminant ability of S-Detect™ and the three examiners was not statistically different. However, the sensitivity of the expert radiologist seems to be better than S-Detect™ (81.5% vs. 70.4%; p = 0.25). The expert missed five malignancies, while S-Detect™ missed eight (similar to the beginner medical student).

The S-Detect™ software had the greatest diagnostic agreement with the expert radiologist, whereas agreement decreased with less experienced examiners. This was confirmed both for the dichotomic classification (suspicious vs. not suspicious; Table 2) and for the complete K-TIRADS classification (Table 3). A substantial agreement exists between the experienced radiologist and S-Detect™ in the assessment of single US features (Table 4). However, this result may be biased because of the use of the software by the senior radiologist alone (who personally performed the ROI selection). To obtain a correct segmentation and recognition of the nodule boundaries, the examiner had to perform the ROI selection procedure for a median of three times (range 1–5 times) before achieving a correct segmentation of the nodule.

Table 2.

Agreement in dichotomic TIRADS classification (K-TIRADS 2–3 vs. 4–5) between S-Detect™ and the three examiners (Cohen Kappa coefficient ± Standard error)

	S-Detect™	Expert	Resident	Student
S-Detect™	–	0.815 ± 0.063	0.748 ± 0.071	0.634 ± 0.079
Expert	0.815 ± 0.063	–	0.888 ± 0.049	0.723 ± 0.071
Resident	0.748 ± 0.071	0.888 ± 0.049	–	0.788 ± 0.063
Student	0.634 ± 0.079	0.723 ± 0.071	0.788 ± 0.063	–

Table 3.

Agreement in TIRADS classification between S-Detect™ and the three examiners (Krippendorff’s alpha)

	Agreement with S-Detect™ (Krippendorff’s alpha)
All three examiners	0.79 (0.73–0.85)
Expert radiologist	0.84 (0.75–0.92)
Resident	0.77 (0.66–0.87)
Medical student	0.69 (0.57–0.80)

Table 4.

Agreement in single sonographic features between S-Detect™ and the expert radiologist (Krippendorff’s alpha)

	Agreement between expert radiologist and S-Detect™ (Krippendorff’s alpha)
Composition	0.79 (0.65–0.93)
Echogenicity	0.71 (0.59–0.81)
Margins	0.74 (0.58–0.90)
Shape	0.71 (0.38–0.94)

Discussion

In the last years, neck ultrasonography has become the cornerstone of the diagnostic algorithm of thyroid nodules, as well as ultrasound elastography [34, 35]. Its main drawbacks are the inadequate predictive values of single US features and the well-known operator dependency. Efforts have been made to improve the discriminative power of ultrasound and reduce interobserver variability through the use of image analysis [19], machine learning, and CAD systems [27, 28, 36]. Several scientific societies have proposed sonographic classification systems to help clinicians categorize and report US features of thyroid nodules. Their application, however, is time-consuming and requires specific training [24].

S-Detect™ is commercially available and has been clinically validated in previous studies [27, 28]. Choi and colleagues [26] tested S-Detect™ on 102 nodules (89 patients) and reported that a radiologist with 20 years of experience had better specificity (94.9% vs. 74.9%, p = 0.002) and discriminant power (AUROC 0.92 vs. 0.83, p = 0.021) than the software, though the sensitivity was similar. Similar figures were reported by Gao and colleagues [37]. Yoo et al. [38] also compared the diagnostic performance of a radiologist with 10 years of experience before and after using the S-Detect™ software as a “second opinion.” After S-Detect™ support, the sensitivity of the examiner increased (92% vs. 84%), but a slight reduction in specificity (85.1% vs. 95.5%) and PPV (82.1% vs. 93.3%) occurred. Jeong et al. [39] studied the application of a CAD system by radiologists with different levels of experience and found that diagnostic performance varied according to operator experience; in particular, the sensitivity ranged from 70.5 to 88.6%. Overall, a meta-analysis confirmed that the sensitivity of the available CAD systems is similar to that of experienced radiologists, with lower specificity and diagnostic odds ratio [40]. Another recent study tested the diagnostic performance of a newer version of the S-Detect™ software. The authors concluded that current systems had limited specificity in the diagnosis of thyroid cancer and that, even if the new version aims to recognize calcifications, this evaluation is not accurate [41].

In our cohort, S-Detect™, a CAD system designed to simplify the scoring and reporting of thyroid nodules, achieved a sensitivity and specificity which were statistically comparable to those obtained by an expert radiologist or a senior resident. Furthermore, it outperformed the less experienced examiner. However, S-Detect™ required manual input of some features (including the presence of microcalcification, which is a crucial finding) and multiple attempts for the correct segmentation of the lesion: the selection of the ROI has a great influence on the final evaluation.

Our study has some limitations. First, this was a relatively small and select cohort of thyroid nodules, all of which had already been selected for FNA biopsy or surgery. This is reflected in the high malignancy rate (25.2%). Second, the reference standard used may have caused false-negative results, even if they are uncommon. On the contrary, all malignancies were histologically confirmed. This study was conducted in a single center, and all US examinations were performed by a radiologist with extensive experience, who was also involved in multiple research programs related to TIRADS systems [4, 18, 24, 42, 43]. This may impact the applicability of our findings to other settings. Also, the software was directly used by the senior radiologist alone.

Conclusion

S-Detect™ had a diagnostic performance similar to that of an experienced radiologist. While it does not provide a clinical advantage to the expert clinician, it may be a useful tool for the less experienced operator as its specificity was significantly higher. It may also be used for training purposes as an aid in recognizing suspicious US features and expediting learning of the TIRADS scoring process and its practical application.

Abbreviations

AUROC

Area under the Receiver Operating Characteristic curve

CAD

Computer-aided diagnosis

CI

Confidence interval

FNA

Fine-needle aspiration

NPV

Negative predictive value

PACS

Picture archiving and communication system

PPV

Positive predictive value

ROI

Region of interest

TI-RADS

Thyroid imaging, reporting, and data system

US

Ultrasonography

Author contributions

MLDP, PG, PP, GO, DF, MC and VC participated in the conception and design of the study; VC collected personal classification and S-Detect data; MLDP and GA performed a subjective classification of nodule images; GA, ET, VR collected clinical data and linked nodule data with final histology reports; VA and DB provided cytology data; MB, VDA, FF, and LG provided patients’ data and performed surgeries. DF and GG performed the statistical analysis and were the major contributors in writing the manuscript. All authors read and approved the final manuscript.

Funding

This work received no specific funding.

Compliance with ethical standards

Conflict of interest

Vito Cantisani lectured for Bracco, Samsung, Canon. The other authors declare that they have no conflict of interest.

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1975 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.