Is TIRADS a practical and accurate system for use in daily clinical practice?

Abstract

Aim:

To assess the positive predictive value (PPV) and inter-observer agreement of Thyroid Imaging Reporting and Data System (TIRADS) as described by Kwak et al.

Materials and Methods:

This was a prospective study wherein ultrasound was performed by two radiologists on patients with thyroid nodules >1 cm. The third radiologist interpreted archived images. Ultrasound features and TIRADS category were compared with cytology and surgical histopathology. PPV was calculated for all readers’ combined assessment. Inter-observer agreement was calculated using linear weighted kappa.

Results:

A total of 238 patients with 272 nodules of mean size 2.9 ± 1.7 cm were included. PPV for malignancy was 6.6%, 32%, 36%, 64%, 59%, and 91% for TIRADS 2, 3, 4a, 4b, 4c, and 5 categories, respectively. Inter-observer agreement was substantial [kappa (k) = 0.61-0.80] for assessment of nodule echogenicity, margins, calcification, and shape and good (k = 0.570, P < 0.001) for assessment of composition of the thyroid nodules. Overall agreement between observers was substantial for assigning TIRADS category [multi-rater weighted kappa coefficient (wt k) = 0.721, P < 0.001].

Conclusions:

TIRADS is a simple and practical method of assessing thyroid nodules with high PPV and good inter-observer agreement.

Introduction

A palpable thyroid swelling is a common disorder and almost 12% of adult Asian Indians have been shown to have a palpable nodule in a recent population-based study.[1] When patients were assessed by ultrasound, the prevalence of a thyroid nodule was as high as 80% among children in iodine-deficient parts of India.[2] However, the incidence of thyroid cancer is low (1-1.8 per 100,000).[3] Ultrasound is a widely accepted imaging modality for the initial assessment of thyroid nodules. There are well-established ultrasound findings that differentiate benign and malignant thyroid nodules,[4,5,6,7,8,9,10,11] and there are several classification systems which categorize thyroid nodules according to the risk of cancer.[8,12,13,14,15,16] Of the many classification systems that have been described, Thyroid Imaging Reporting and Data System (TIRADS) described by Kwak et al.[15] is a relatively simple system which can be easily adopted, just like Breast Imaging-Reporting and Data System (BIRADS) which has been successfully used for several years to assess breast lesions.[17] We aimed at assessing the positive predictive value (PPV) and the inter-observer variability of TIRADS for the ultrasound features of thyroid nodules as described by Kwak et al.[15]

Materials and Methods

Study population

This study was approved by the institutional review board (IRB Number: 7796/12) and conducted in the Department of Radiology of a 2800-bedded tertiary care teaching hospital in southern India from January to November 2012. For assessing the PPV of TIRADS, a sample size of approximately 270 nodules was estimated with an expected sensitivity of 85% and specificity of 80%, 5% margin of error, and 95% confidence interval. For assessing inter-reader reliability, a sample of size of approximately 250 nodules was required to test the sample for population agreement of kappa (k = 0.80 vs. K = 0.90), with a prevalence of 12% disease in the population, at 80% power and 5% margin of error with a two-sided test.

Patients with solitary thyroid nodule and dominant nodules of a multi-nodular goiter clinically with a maximum size in excess of 1 cm were included in the study in order to make the correlation with histology easier. Out of 526 patients who underwent thyroid sonology during this period, 307 consecutive patients with a total of 346 nodules satisfied the inclusion criteria and participated in the study. Patients were included for final analysis if they had (a) fine needle aspiration cytology (FNAC) result showing benign or malignant lesion or (b) undergone surgery for treatment of thyroid nodule with initial FNAC reporting as nodule suspicious for malignancy, indeterminate or inadequate sampling. Informed consent was obtained from all participants prior to entry in the study. Among 307 patients who participated in the study, 69 patients (9 males, 60 females) with 74 nodules were excluded from the final analysis. They had non-diagnostic FNAC results such as inadequate cytology (n = 22), indeterminate (n = 14), follicular neoplasm (n = 9), suspicious for malignancy (n = 5), and did not undergo surgery or FNAC (n = 24). Two hundred and thirty-eight patients (68 males and 170 females) with 272 nodules were included for the final analysis. The mean age was 42.7 ± 13.7 years with a range of 16–82 years. The mean age of male patients was 45.6 ± 15.6 years (range 18–82 years) and the mean age of female patients was 41.5 ± 12.7 years (range 17–78 years). Out of 238 patients, 207 patients had a single nodule, 28 patients had two nodules, and 3 patients had three nodules. The mean nodule size was 2.9 ± 1.7 cm (range 1-9.2 cm).

Thyroid ultrasound

Ultrasound of the neck and thyroid gland was performed in two ultrasound machines (ACUSON S2000^™, Siemens and ACUSON Antares^™, Siemens health care, Erlangen, Germany) using a high-frequency probe (7-11 MHz). A conventional and compound ultrasound was performed in all patients. Ultrasonography was performed by two radiologists with 8 years (A. C. – reader 1) and 4 years (A. K. – reader 2) of experience, one after the other, and they were blinded with regards to the findings of the other radiologist. Ultrasound features assessed for each nodule were composition (solid, cystic, mixed), echogenicity (hyperechoic, isoechoic, hypoechoic, markedly hypoechoic), margins (well defined with or without halo sign, microlobulated, ill-defined, irregular), presence of calcification (microcalcification, macrocalcification), and shape of the nodule (round, oval). Figures 1–5 show examples of each ultrasound descriptor of the nodule. Nodules with >75% solid component were labeled as solid; cystic nodules had no solid components and mixed nodules had both solid and cystic areas with solid component constitution <75% of the size of the lesion. For mixed lesions, echogenicity, margin, shape, and presence of calcification were assessed for the solid component. Echogenicity was described in comparison with the thyroid gland and the strap muscles. The lesion was considered hyperechoic if the echogenicity was more than that of thyroid gland, isoechoic if the echogenicity was equal to that of thyroid gland, hypoechoic if the echogenicity was equal to that of strap muscle, and markedly hypoechoic if the echogenicity was lower than that of strap muscle. Hypoechoic smooth rim around the nodule was considered as a positive halo sign. Presence of short cycle undulations of more than three along the margin of the nodule was considered as microlobulated margin. Spiculated margins were considered as irregular, while fuzzy margins were considered ill-defined. Calcification that measured less than 1 mm was defined as microcalcification and calcification more than 1 mm was labeled as macrocalcification. The shape was described as round or “taller than wide” if the anteroposterior dimension was equal to or greater than the transverse dimension, and a nodule which was “wider rather than” tall was described as an oval nodule. Findings that were considered in favor of a malignancy were hypoechoic or markedly hypoechoic in echogenicity; irregular, microlobulated, or ill-defined margins; presence of microcalcification; and round shape. Based on the number of features suspicious for malignancy, TIRADS category was assigned to the nodules by both the observers, as described by Kwak et al.[15] Completely cystic thyroid nodule, nodules with comet tail artifacts, and spongiform thyroid nodules were assigned TIRADS category 2. Solid, oval, well-defined, isoechoic nodules were assigned TIRADS category 3. Nodules were assigned TIRADS categories 4a, 4b, 4c, and 5 if they had one, two, three, and more than three suspicious ultrasound features, respectively. Presence and absence of significant neck nodes (hypoechoic, round nodes with calcification or necrosis irrespective of their size) and the pattern of vascularity of the nodules were also documented, though they were not a part of TIRADS.

Figure 1 (A-C).

Ultrasound image showing examples of (A) cystic (B) solid and (C) mixed composition of thyroid nodules

Figure 5 (A and B).

Ultrasound image showing examples of (A) oval and (B) taller than wide shaped thyroid nodules

Figure 2 (A-D).

Ultrasound image showing examples of (A) hyperechoic (B) isoechoic (C) hypoechoic and (D) markedly hypoechoic thyroid nodules

Figure 3 (A-D).

Ultrasound image showing examples of (A) well-defined (B) microlobulated (C) ill-defined and (D) irregular thyroid nodules

Figure 4 (A and B).

Ultrasound image showing examples of (A) microcalcification and (B) macrocalcification in thyroid nodules

A third radiologist with 3 years of experience (P. P. – reader 3) retrospectively reviewed the images of the thyroid stored in the picture archiving and communication system (PACS) (GE centricity PACS, GE Healthcare, 2000) specifications and company, year) with documented ultrasound features and a TIRADS final assessment category for each thyroid nodule, and was blinded to the findings of the other two radiologists, in addition to being blinded to the FNAC and histopathology reports. A common consensus was arrived after discussion among each other for nodules with a discrepancy in the interpretation of the findings or the TIRADS category assigned.

Thyroid FNAC

FNAC was performed after the thyroid ultrasound and the time period between the ultrasound and the FNAC was from 12 h to 2 days. An FNAC was performed by the surgeon for solid palpable thyroid nodules. In patients with a dominant nodule of a multi-nodular goiter, the nodules to be subjected for FNAC were marked by the radiologist. For mixed solid and cystic lesions, cystic lesions, and nonpalpable thyroid nodules with suspicious ultrasound features, the radiologist performed ultrasound-guided FNAC. FNAC was performed using a 23 gauge needle attached to a 5 ml syringe. Two to three aspirations were performed on each nodule. Cytology smears were prepared on three to six slides. Slides were fixed immediately in 95% alcohol and stained with Papanicolaou stain. A cytology technician was available to prepare the slides and to confirm the adequacy of the specimen. The cytology was reported by cytopathologists according to the Bethesda system for reporting thyroid cytology.[15,18] Bethesda class II (benign) and Bethesda class VI (malignant) were considered as diagnostic cytology reports and included for final analysis. The rest of the categories (Bethesda class I – inadequate; Bethesda class III – atypical cells or follicular cells of indeterminate significance; Bethesda class IV – follicular neoplasm; and Bethesda class V – suspicious for malignancy) were considered non-diagnostic and were included in the final analysis only if surgical histopathology was available. Our positive hit rate was 74.5% (n = 210) and the rest were inadequate (n = 72). Among the inadequate thyroid FNACs, 7 (15.2%), 23 (23%), 17 (39.5%), 5 (20.8%), 9 (32.1%), and 11 (26.8%) nodules were TIRADS 2, 3, 4a, 4b, 4c, and 5, respectively. FNAC from 6 (17.1%), 53 (26.8%), and 13 (26.5%) nodules with cystic, solid, mixed solid, and cystic composition, respectively, were inadequate.

The imaging findings and the TIRADS category were compared with FNAC and surgical histopathology. Patients with non-diagnostic FNAC not being planned for surgery and showing nodules of TIRADS 3 or less on USG had a clinical follow-up and ultrasound thyroid every 6 months. A nodule in which temporal stability was demonstrated with ultrasound follow-up at 1 year follow-up was considered benign even if FNAC was non-diagnostic and the patient did not undergo surgery. A total of nine nodules (six cystic and three hyperechoic) were considered benign using the temporal stability of ultrasound appearance of the nodules and no features suspicious for malignancy as a criterion for inclusion. Of these, four nodules (three hyperechoic and one cystic nodule) had non-diagnostic cytology (Bethesda class I – inadequate). Rest of the five nodules that were all cystic neither had cytology nor surgery, but were unchanged at follow-up ultrasound performed 12 months after initial presentation.

Statistical methods

IBM SPSS Analytics 16.0 software (Chicago, IL, USA) and SAS/STAT software, version 7.3 (Copyright^© SAS Institute Inc., Cary, NC, USA) were used for statistical analysis. The PPV and likelihood ratio for malignancy for ultrasound features of thyroid nodules and final assessment categories was determined by using data from the assessments of all readers combined. Receiver operating characteristic (ROC) curve was used to assess the diagnostic performance of TIRADS. Inter-rater reliability was measured using the quadratic weighted kappa statistics, and reported using a bootstrapped, bias-corrected method with 95% confidence interval (Reichenheim, 2004; Carpenter and Bithell, 2000). Kappa is scaled such that “zero” is the amount of agreement that would be expected by chance and “one” indicates perfect agreement. Kappa was interpreted according to the guidelines laid by Landis and Koch.[19]

Results

Figure 6[20] shows the study group. Out of 272 nodules (right lobe- 134, isthmus- 11, left lobe- 127) in 238 patients, 154 nodules were benign (119 nodules in females, 35 nodules in males) and 118 nodules were malignant (75 nodules in females, 43 nodules in males). Malignancy was more common among male patients presenting with a thyroid nodule (P = 0.01). There was no significant difference in the mean age of patients with benign (mean age was 42.8 ± 12.7 years) and malignant thyroid nodules (mean age was 43.8 ± 13.8 years) (P = 0.657). There was no significant difference in the age of male and female patients with benign nodules (P = 0.712). However, among patients with malignant thyroid nodules, men (mean age was 47.7 ± 18.1 years) were significantly older than women (mean age was 40.5 ± 12 years) (P = 0.01). The size of benign nodules (mean of 4.2 ± 2.7 cm, range 1.2-9.2 cm) was significantly more than that of malignant nodules (mean of 2.3 ± 1.9 cm, range 1-6.7 cm) (P = 0.03). A total of 168 (56 benign and 112 malignant) nodules were treated with surgery. Table 1[20] gives the histopathology of these nodules.

Figure 6.

Study group

Table 1.

Histopathology of 168 nodules treated with surgery

The frequency of ultrasound findings in thyroid nodules according to TIRADS descriptors is given in Table 2.[20] Vascularity was assessed in 182 nodules; 40 had central vascularity and the rest showed peripheral vascularity. Ten (0.3%) benign nodules and 30 (16.4%) malignant nodules showed central vascularity; 95 (52.1%) benign nodules and 47 (25.8%) malignant nodules showed peripheral vascularity. There were neck nodes along with 45 (16.5%) nodules. Five (3.2%) benign nodules had associated significant neck nodes and 40 (35%) malignant nodules had significant neck nodes.

Table 2.

Frequency of ultrasound features of thyroid nodules according to TIRADS descriptors

The PPV and diagnostic performance of TIRADS

Table 3[20] summarizes the TIRADS category of the nodules, cytology diagnosis, and surgical histopathology results of patients who underwent surgery. Table 4[20] shows the distribution of benign and malignant nodules in each of the TIRADS categories with the PPV and positive likelihood ratio. The diagnostic performance of TIRADS considering categories 4a, 4b, 4c, and 5 as malignant and categories 2 and 3 as benign is as follows: Sensitivity = 72%, specificity = 68.8%, PPV = 63.9%, negative predictive value (NPV) = 76.2%, and accuracy = 70.2%. The diagnostic performance of TIRADS considering categories 4b, 4c, and 5 as malignant and categories 2, 3, and 4a as benign is as follows: Sensitivity = 60.2%, specificity = 85.1%, PPV = 75.5%, NPV = 73.6%, and accuracy = 74.2%. ROC curve drawn to assess the diagnostic performance of TIRADS compared to conclusive FNAC or histopathology showed an area under the curve of 0.761 [95% Confidence Interval (CI): 0.703-0.819; P < 0. 001].

Table 3.

Summary of ultrasound TIRADS category and surgical histopathology for patients who underwent surgery (n=168)

Table 4.

Distribution of benign and malignant thyroid nodules in each TIRADS category with positive predictive value for malignancy, positive likelihood ratio (LR+)

Inter-observer agreement for ultrasound features of thyroid nodules and TIRADS

Inter-observer agreement for reproducibility of ultrasound features of thyroid nodules was substantial for nodule echogenicity (kappa = 0.691), margins (0.728), calcification (kappa = 0.791), and shape (0.626), while the agreement was good for assessment of nodule composition (kappa = 0.570). Overall agreement between observers for assigning TIRADS category was substantial (multi-rater weighted kappa coefficient (wt k) = 0.721, P < 0.001). But there was considerable variation in the inter-observer agreement for each category of TIRADS. There was substantial agreement between observers for assigning TIRADS categories 4c and 5 (kappa = 0.685 and 0.712, respectively), but poor agreement between observers for assigning TIRADS categories 4a (kappa = 0.201) and 4b (kappa = 0.179). There was good agreement for rest of the categories.

Discussion

Many previous studies have proven the usefulness of ultrasound evaluation of thyroid nodules and its ability to differentiate benign from malignant nodules.[5,6,7,8,9,10,11,12,13,14] Several classification systems have been proposed to stratify the risk of malignancy in thyroid nodules.[7,8,12,13,14,15,16] Most of them are complex using several ultrasound features and formulae which are not easy to use in daily practice, especially in a tertiary care teaching setup where examiners of varying experience perform ultrasound scans. Of all the systems, the classification proposed by Kwak et al.[15] is simple and similar to BIRADS system which has been in use for many years and is familiar to most radiologists. Therefore, we aimed to assess the PPV and inter-observer agreement of TIRADS as proposed by Kwak et al.[15]

A study by Kim et al. previously reported that hypoechogenicity, marked hypoechogenicity, microlobulated or irregular margins, microcalcification, and taller than wide shape were the ultrasound features which best predicted the chance of malignancy in thyroid nodules. These features were considered to be suspicious for malignancy in our study and as proposed by Kwak et al.,[15] we assigned the TIRADS categories to the thyroid nodules depending on the number of these features seen. The PPV for malignancy was 6.6%, 32%, 36%, 64%, 59%, and 91% for TIRADS 2, 3, 4a, 4b, 4c, and 5 categories, respectively. However, Kwak et al.[15] describes 0%, 1.7%, 3.3%, 9.2%, 44.4–72.4%, and 87.5% risk of malignancy for TIRADS 2, 3, 4a, 4b, 4c, and 5 categories, respectively. However, in a subsequent retrospective validation study, the authors found a 7.3% and 8.3-96.6% risk of malignancy in TIRADS 3 and TIRADS 4-5 categories of thyroid nodules.[14]

Table 5[20] shows the comparison rate of malignancy according to various reporting systems. There was improvement in the PPV (from 64% to 75%) and specificity (from 69% to 85.5%) when TIRADS category 4a nodules were reassigned to category 3; however, sensitivity of TIRADS reduced (from 72% to 60%). In the actual clinical setting, it may be practical to follow-up patients with just one suspicious feature and indeterminate cytology than subjecting them to surgery.

Table 5.

Comparison of diagnostic performance of the various ultrasound classification systems available to assess thyroid nodules

Two observers performed real-time thyroid ultrasound and the third radiologist interpreted the images from the PACS. We used weighted kappa statistics to assess the agreement between observers for ultrasound features of thyroid nodules and TIRADS categories, since we were dealing with ordinal variables (categorical variables with more than two categories). We also chose this method of analysis over simple kappa statistics in order to remove the influence of clinically unimportant differences among the observers.

Overall agreement between the observers was substantial for assessment of nodule echogenicity, margins, calcification, and shape, while agreement was good for assessment of composition of the thyroid nodules. Overall agreement between observers was substantial for assigning TIRADS category to the thyroid nodules. However, inter-observer agreement for each category of TIRADS varied, with substantial agreement for TIRADS categories 4c and 5 and poor agreement for TIRADS 4a and 4b categories.

Our study has several limitations. Firstly, the experience of the observers was not uniform and the observers were not given prior training on the use of TIRADS terminologies. These could have influenced the inter-observer agreement. However, this is what is likely to happen in practice when TIRADS is administered to a large population of radiologists with varying experience. Secondly, being a tertiary care referral center, the percentage (32%) of patients undergoing thyroid surgery after thyroid ultrasound was quiet high and large number of nodules included in the study was malignant. This would have caused a selection bias. Thirdly, of the 112 malignant thyroid nodules, 34 (30.3%) were a follicular variant of papillary cancers which have a relatively benign ultrasound appearance [Figure 7]. Tuberculosis of the thyroid gland is uncommon, but ultrasound appearance can closely mimic malignancy [Figure 8]. These conditions would have influenced the diagnostic performance of TIRADS. Fourthly, false-positive and false-negative cytology may have had an impact on the results obtained. Fifthly, ultrasound-guided FNAC was performed in only 30% of nodules which were either predominantly cystic, mixed solid, and cystic nodules or were small and had suspicious features on ultrasound. Rest of the nodules underwent blind FNAC by surgeons of varying experience. This could have contributed to the high number of inadequate FNACs. Sixthly, as also described by Kwak et al.,[14,15] breast cancer and thyroid cancer are very different in terms of the prevalence, disease severity, clinical course, and temporal progression. We are not sure if we can apply similar systems for both these entities.

Figure 7 (A and B).

Ultrasound of two different patients: (A) follicular adenoma and (B) follicular variant of papillary carcinoma. Both these nodules were assigned TIRADS category 3 by all three observers

Figure 8 (A-D).

Ultrasound of 32-year-old male patient with neck swelling showed (A-C) bilateral necrotic and markedly hypoechoic masses in the thyroid gland and (D) necrotic cervical nodes. Biopsy revealed tuberculosis. All observers assigned TIRAD category 5 for these nodules

In conclusion, the PPV for malignancy was high for TIRADS category 5 and 4c nodules. Reassigning TIRADS category 4a nodules as TIRADS 3 will improve the PPV and specificity of TIRADS. Overall agreement between observers for assigning TIRADS category was substantial. Thus, TIRADS is a simple and practical method of assessing thyroid nodules and can be used in practice.

Financial support and sponsorship

Fluid research grant of Christian Medical College, Vellore. (IRB Number: 7796/12).

Conflicts of interest

There are no conflicts of interest.