The Impact of Benign Gene Expression Classifier Test Results on the Endocrinologist–Patient Decision to Operate on Patients with Thyroid Nodules with Indeterminate Fine-Needle Aspiration Cytopathology

Abstract

Background

Seventy-five percent of thyroid nodules with indeterminate fine-needle aspiration (FNA) cytology are found to be benign postoperatively. A novel genomic test, the Afirma^® gene expression classifier (AGEC), has been available for clinical use in the United States, since late 2010. In 2010, two modest-sized validation studies showed that the AGEC could identify a benign gene expression signature in indeterminate cytology thyroid FNA samples with a negative predictive value >95%. The objective of this study was to evaluate how the AGEC impacted the joint decision of the endocrinologist and patient to operate when FNA cytology was indeterminate, but the AGEC reading of the nodule was benign.

Methods

In this cross-sectional cohort study, data were contributed retrospectively by 51 endocrinologists at 21 practice sites that had previously obtained ≥3 benign AGEC readings in ≥1 cm nodules with indeterminate FNA cytology readings. Information regarding demographic data, nodule size and location, decision to operate, surgery type (hemithyroidectomy [HT] or total thyroidectomy [TT]), and reason for recommending surgery was retrospectively collected.

Results

Compared to a 74% previous historical rate of surgery for cytologically indeterminate nodules, the operative rate fell to 7.6% during the period that AGEC were obtained in the clinical practices, a highly significant reduction in the decision to operate (p<0.001). The rate of surgery on cytologically indeterminate nodules that were benign by the AGEC reading did not differ from the historically reported rate of operation on cytologically benign nodules (p=0.41). The four primary reasons reported by the physicians for operating on nodules with a benign AGEC reading, in descending order: large nodule size (46.4%), symptomatic nodules (25.0%), rapidly growing nodules (10.7%), or a second suspicious or malignant nodule in the same patient (10.7%). These reasons are concordant with those typically given for operation on cytologically benign nodules.

Conclusions

In a substantial group of medical practices, obtaining an AGEC test in patients with cytologically indeterminate nodules was associated with a striking reduction in the rate of diagnostic thyroidectomy. Approximately, one surgery was avoided for every two AGEC tests run on thyroid FNAs with indeterminate cytology.

Introduction

Thyroid nodules are common, with population-based screening studies identifying palpable nodules on clinical examination in 5%–7% of adults and ultrasound detection in up to 50% of women and 20% of men over age 50 (1). Thyroid cancer, increasing in incidence more rapidly than any other cancer type, usually presents as a thyroid nodule (2). However, the vast majority of thyroid nodules, whether palpable or incidentally discovered, are benign. Historically, a mere 5% of evaluated nodules ultimately prove to be malignant based on autopsy findings (3), and even though selection of nodules for biopsy based on suspicious ultrasound features enriches the yield of malignant fine-needle aspirations (FNAs), the percentage that are malignant remains only 10%–15% (4). With such a low pretest probability of malignancy, the challenge for any test is to sustain a sufficiently high sensitivity that cancers are not missed, while simultaneously achieving a sufficiently high negative predictive value (NPV) to avoid surgery in the majority of patients whose nodules are benign and nonthreatening.

Several professional organizations, including the American Association of Clinical Endocrinologists (AACE) and the American Thyroid Association (ATA), have published guidelines for the evaluation of thyroid nodules, all of which endorse a similar multistep strategy: clinical assessment, measurement of thyrotropin, ultrasound evaluation, and biopsy of nodules selected according to size and ultrasound characteristics (5,6). Fine-needle aspiration biopsy (FNAB) is a safe and simple outpatient procedure that yields cellular material (either liquid or smears) suitable for cytological analysis (7). When the biopsy material contains sufficient cellularity for adequate assessment, cytology yields an accurate and reliable diagnosis of a benign nodule in the majority of cases (72%; range 62%–85%). While some centers have reported false negative rates for cytology as low as 1%, a recent meta-review reported a 6% risk of malignancy (range 2%–18%) on operated cytologically benign thyroid nodules with a performance that varied considerably between academic and community-based centers (8,9). Approximately, 5% (range 1%–8%) of nodules are cytologically malignant (typically papillary thyroid carcinoma [PTC]), with a positive predictive value of 97% (93%–100%).

Unfortunately, 15%–30% of biopsied nodules exhibit indeterminate cytology, defined here as diagnostic subtypes consistent with The Bethesda System for Reporting Thyroid Cytology of: Atypia of Undetermined Significance (AUS) or Follicular Lesion of Undetermined Significance (FLUS); (Suspicious for) Follicular/Hürthle Cell Neoplasm; and Suspicious for Malignancy (6,10). Current guidelines recommend surgical resection for these nodules to permit adequate pathological evaluation, but only 15%–30% of these nodules prove to be malignant (usually either follicular variant of PTC, or follicular carcinoma), while 70%–85% prove benign (11,12). Consequently, most thyroid surgery for asymptomatic thyroid nodules is actually performed as a diagnostic rather than a therapeutic procedure.

The Afirma^® gene expression classifier (AGEC) is a proprietary diagnostic test developed by Veracyte, Inc. (South San Francisco, CA) for the preoperative identification of benign thyroid nodules whose cytology is indeterminate. Testing is offered through a sole source, Clinical Laboratory Improvement Amendments (CLIA)—certified reference laboratory. The assay classifies nodules as either benign (>95% NPV) or suspicious for malignancy (>50% risk for malignancy). The risk of malignancy for a thyroid nodule with a benign AGEC diagnosis is reported to be comparable to that of an operated nodule with a benign cytopathology diagnosis (13,14). With a benign AGEC result, observation or ultrasound follow-up could be recommended in lieu of thyroid surgery, avoiding unnecessary surgery and reducing healthcare costs (15).

According to a recent meta-review of published pathology series in the United States, cytologically benign thyroid nodules are resected in 9% of cases (range 3%–16%) (8) based on a variety of clinical factors. We hypothesized that cytologically indeterminate nodules that have a benign reading by AGEC will also sometimes be resected, at a rate similar to that seen with cytologically benign findings. We further hypothesized that a decision to operate would depend on a number of clinical findings similar to the commonly reported reasons for operating on cytologically benign nodules specifically: large nodule size; suspicious ultrasound characteristic(s); presence of other thyroid nodules; family history of thyroid cancer; rapid nodule growth; local compressive symptoms; presence of suspicious lymph node(s); history of external radiation; patient preference; and/or other factors. However, there are no published data on the impact of such an AGEC result on patient and physician decision-making in clinical practice. The goal of this study, therefore, was to determine whether the finding of a benign AGEC result changed physician–patient decision-making regarding diagnostic surgery or continued observation in a clinical practice setting.

Methods

Study design

This retrospective, multicenter study was designed by the principal investigators (D.S.D., B.M.) and the sponsor, Veracyte, Inc. Data collection and statistical analysis were provided by the sponsor (J.C.D., L.F., G.C.K., R.B.L.), with oversight by the academic coinvestigators. The study protocol was approved by a central, as well as by institution-specific, investigational review boards (IRBs). The approving IRBs found the study to be of minimal patient risk, and thus granted a waiver of informed consent for patients whose data were collected for study analysis. The coinvestigators (D.S.D., J.P.K., B.M.) had full access to all study data and analyses.

Endocrinology practices that ordered AGEC tests on patients having a thyroid FNAB, and for which the AGEC reading was benign on three or more patients during the data collection period (September 2011 through March 2012), were invited to participate. The minimum number of three patients with benign AGEC results per site was established to ensure blinding of results. The practice site identified consecutive eligible patients who had undergone FNAB with indeterminate cytology and for whom benign AGEC results had been reported. The endocrinology practice had the option to request from the clinical laboratory, in writing, copies of their eligible patient reports to ease the task of data collection. These reports were used for reference to aid in the identification of patients with benign AGEC results and completion of case report form data; no patient identifying information was recorded nor provided to the clinical study group.

Patient selection

Eligible patients were age 21 years or older and had one or more thyroid nodules of 1 cm in diameter or greater confirmed by ultrasound. Patients were included in the study if their nodules yielded a cytopathology indeterminate result and the AGEC diagnosis from the same nodule was reported as benign. Indeterminate cytopathology diagnosis was determined by either a single, centralized, high-volume cytopathology group (Thyroid Cytopathology Partners [TCP]; Austin, Texas), or by cytopathology groups at respective academic centers. Practice sites individually collected, compiled, and submitted study data to the clinical study group.

Data collection

After verifying patient eligibility, practice sites recorded patient demographic information, nodule size and location, and whether thyroid surgery was recommended. If surgery was performed, surgery type (hemithyroidectomy [HT] or near total thyroidectomy [TT]) and the reason for recommending surgery were reported by physicians on a study case report form. The form was prepopulated with the likely reasons for recommending surgery, which included: large nodule; symptomatic nodule; rapid nodule growth; suspicious lymph node; history of external radiation to the neck; family history of thyroid cancer; multiple endocrine neoplasia (e.g., clinically suspicious for MEN1, MEN2); ultrasound characteristics suspicious; cytopathology indeterminate, but suspicious for malignancy; prior indeterminate cytopathology and benign gene expression classifier test in same nodule; other nodule(s) suspicious for malignancy or malignant; patient anxiety; cosmesis; and other causes as defined by the participating endocrinologist.

Statistical analysis

Statistical analysis was performed using R software, version 2.13. Study surgical rates were compared to values reported in the literature via exact binomial tests. Surgical rates within the study for academic and commercial practices were tested for equality via a chi-squared test. p values<0.05 were considered significant. Confidence intervals [CI] for proportions are reported as 95% two-sided exact binomial CI.

Results

A total of 51 endocrinologists (46 community-based, 5 academic) at 21 endocrinology practices in 11 states contributed data to the study. Cytopathology diagnosis was performed primarily by a centralized cytopathology service (341 samples, 86.3%), while the remainder were performed at academic centers (Table 1). Data collection occurred from September 2011 through March 2012. We analyzed data on 368 patients (395 nodules). In the entire cohort, most patients were women (n=301, 82%) with median age of 55 years (range 21–86) and median nodule size of 2.4 cm (range 1.0–7.0 cm) (Table 2). Surgery was performed on a minority of cytologically indeterminate, but AGEC benign patients (n=28, 7.6% [CI 5.1–10.8], p-value<0.001) when compared to the 74% (weighted mean; range 63%–96%) rate previously reported for surgery on cytologically indeterminate nodules (8), (Fig. 1). The primary reasons that surgery was recommended on AGEC benign patients were for those with large (n=13, 46.4%), symptomatic (n=7, 25.0%), or rapidly growing (n=3, 10.7%) nodules as well as for patients with a second suspicious or malignant nodule (n=3, 10.7%). In one case, surgery was elected because of patient preference (n=1, 3.6%), despite a benign AGEC result (Fig. 2). Both HT (n=19, 67.9%) and TT (n=8, 28.6%) were recommended (Table 3 and Fig. 3), although the reasons for type of surgery were not collected. The rate of ultrasound observation in lieu of surgery (92.4%) was not significantly different from that previously reported (91%) (8) for cytopathology benign FNAs (p=0.41 [CI 89.2–94.9]).

Table 1.

Endocrinology Practice and Cytopathology Diagnostic Group Characteristics

Practice site, n (%)
Academic	1 (4.8%)
Community-based	20 (95.2%)
Cytopathology diagnostic group, n (%)
Centralized cytopathology service	341 (86.3%)
Academic cytopathology service	54 (13.7%)
Total thyroid nodules, N (%)	395 (100%)

Table 2.

Patient Demographics

Gender, n (%)
Female	301 (81.8%)
Male	67 (18.2%)
Ethnicity, n (%)
Caucasian	244 (66.3%)
African American	18 (4.9%)
Asian/Pacific Islander	9 (2.4%)
Hispanic	12 (3.3%)
Other	6 (1.6%)
Unknown	79 (21.5%)
Total patients, N (%)	368 (100%)
Age, median (range)	55 (21–86)
Nodule size (cm), median (range)	2.4 (1.0–7.0)

FIG. 1.

Comparison of the frequency of referral to surgery for patients with cytologically indeterminate fine-needle aspirations (FNAs) under the traditional clinical paradigm, and the frequency of referral to surgery for patients with cytologically indeterminate FNAs using benign Afirma^® gene expression classifier results. The difference is significant with p<0.001.

FIG. 2.

Counts and proportions of 42 reasons provided for referral to surgery for the 28 surgical referrals in this study. Other reasons provided for surgical referral included abnormal gland appearance (n=1), abnormal physical exam (n=1), family history of thyroid cancer (n=1), enlarging nodules (n=1), Grave's disease (n=1), Hashimoto's thyroiditis (n=1), patient history of breast cancer (n=1), multiple other large nodules (n=1), election of preventative surgery (n=1), patient preference (n=1), and symptoms not otherwise specified (n=2).

Table 3.

Nonsurgery Management Versus Surgery Recommended when Cytology is Indeterminate, but Gene Expression Classifier Result is Benign

	n (%)	[95% CI]
Nonsurgery management	340 (92.4%)	[89.2%–94.9%]
Surgery performed	28 (7.6%)	[5.12%–10.8%]
Reason surgery recommended
Large nodule	13 (46.4%)	[27.5%–66.1%]
Local pressure symptoms	7 (25.0%)	[10.7%–44.9%]
Rapid nodule growth	3 (10.7%)	[2.27%–28.2%]
Other suspicious nodule	3 (10.7%)	[2.27%–28.2%]
Suspicious by cytology	2 (7.1%)	[0.9%–23.5%]
Suspicious by ultrasound	2 (7.1%)	[0.9%–23.5%]
Other	12 (42.9%)	[24.5%–62.8%]
Type of surgery recommended
Hemithyroidectomy	19 (67.9%)	[47.6%–84.1%]
Total thyroidectomy	8 (28.6%)	[13.2%–48.7%]
Unknown	1 (3.6%)	[0.1%–18.3%]

FIG. 3.

Type of surgery recommended for the 28 surgical referrals in this study.

Discussion

This retrospective study is the first published report assessing the impact on routine clinical practice of the Afirma thyroid genomic diagnostic test, designed to reclassify thyroid nodules with indeterminate FNA cytopathology preoperatively as either benign or suspicious. The study was not designed to further evaluate the performance characteristics of the AGEC, which is commercially available and in clinical use in several centers. Rather, we sought to determine whether the information provided by the AGEC led to a decision to avoid surgery in patients whose nodules were found to exhibit a benign AGEC profile.

Although the null hypothesis was that endocrinologists and patients would not find the benign AGEC (with a NPV of 95%) sufficiently reassuring to avoid a diagnostic thyroidectomy, in fact, we found a striking reduction in the physician–patient decision to operate on cytologically indeterminate thyroid nodules, whose AGEC was reported to be benign. Compared to a 74% historical rate of surgery for cytologically indeterminate nodules, the operative rate was a mere 7.6% in those nodules with a benign AGEC result in this study, a reduction that is highly statistically significant. The remaining pool of patients with AGEC suspicious results are enriched for malignancy, since about half of the benign nodules were identified by genomic testing.

The demographics of the patients and nodules in this study were typical of patients whose nodules are evaluated and treated, with a 4:1 ratio of women to men, an average age of 55, and an average nodule size of 2 cm (16). Because in all community-based practices, the endocrinologist had ordered the AGEC as an auto-reflex when cytopathology was indeterminate, the nodules sampled are very likely to reflect the range of characteristics typically encountered in a clinical practice. Although the three indeterminate cytology subtypes were not individually reported for the purposes of this study, the indeterminate cytopathology subtype, Suspicious for Malignancy, was only given twice as a reason for the decision to operate on a cytologically indeterminate, but AGEC benign nodule, even though it was listed as a prepopulated reporting option on the physician-completed case report form. Nevertheless, we suspect that most patients with cytology Suspicious for Malignancy results undergo surgery, given the relatively high (>60%) risk of carcinoma, and concerns that FNA passes for cytology sampling may have picked up cancer cells missed by the FNA passes for molecular test sampling.

No significant difference was evident in the rate of referral to surgery between academic (2 of 52 patients, 3.85%) and community-based (26 of 316 patients, 8.23%) practices (p=0.41 [95% CI on the difference between rates by practice type: −2.78%–11.54%]).

The rate of surgery on cytologically indeterminate, AGEC benign nodules did not differ from the historically reported rate of operation on cytologically benign nodules (p=0.41) (8). Furthermore, the four primary reasons given for operating on AGEC benign nodules (Fig. 3), in descending order: large nodule size, symptomatic nodules, rapidly growing nodules, or a second suspicious or malignant nodule in the same patient. These reasons are concordant with those typically given for operation on cytologically benign nodules (17).

Interestingly, when surgery was performed, the ratio of HT to TT was >2:1. This is the inverse of the 25-year trend toward TT, with the ratio of HT to TT dropping to 1:2 in a report based upon U.S. population-based data (18). Consequently, in addition to reducing the number of diagnostic surgeries, an AGEC benign result seems to modify the type of surgery toward HT. This may prove to be an additional source of cost savings, since TT may be as much as twice as expensive as a lobectomy (15).

False negative errors in cytological assessment may result from the FNA sampling error, which may also be a cause of a false negative AGEC result. Malignant thyroid nodules can be heterogeneous, within which, portions exhibit genomic changes, while other areas do not (19). Concern has been expressed that the FNA sampling error might be a particular problem in large nodules, leading some authors to recommend resection of even cytologically benign nodules >4 cm, though not all studies have confirmed this need for a greater index of suspicion (17,20). Both ATA and AACE guidelines recommend clinical and sonographic follow-up of cytologically benign nodules, and we emphasize similarly careful follow-up of molecularly benign nodules, because long-term follow-up studies of AGEC benign nodules have not yet been reported. In clinical practice, we advise such patients to contact us if they experience a change in clinical status (hoarseness, difficulty swallowing, or visible enlargement of the nodule), and to return (even in the absence of symptoms) for repeat clinical and ultrasound evaluation in 6 to 12 months.

A statistical modeling study has shown the AGEC to be cost effective, while also improving quality of life, primarily through a reduction in diagnostic surgery, surgical complications, and the need for life-long thyroid hormone replacement (15). Using the information provided by the current study, we can now directly assess the cost–effectiveness of this novel diagnostic test. It has been previously reported that for the first consecutive 2667 thyroid FNAs tested commercially, 15.5% were cytologically indeterminate, and 53% of resulted tests were AGEC benign (21). Subsequent to that report, and as of the conclusion of this study at end March 2012, 11,968 thyroid FNAs had been received by Veracyte, 16% of which were cytologically indeterminate. Of the 2040 AGEC tests performed (either reflexed from TCP cytopathology or ordered directly following local cytopathology at academic centers), 52.3% were molecularly benign (22). This excludes the 215 (10.5% of the total) FNA samples, which had inadequate RNA yield or quality. Because the current study demonstrated that a benign AGEC result led to avoidance of surgery 92.4% of the time, it appears that one thyroid surgery can be avoided for every two AGEC tests that are run. With fully burdened (including complications, and blending HT and TT costs) thyroidectomy costs of ∼$12,000 at 2010 Medicare rates, and including an assumed reimbursement by Medicare of $3200 for the AGEC, the use of the test should result in a significant reduction in total health care expenditure of about $2600 per patient tested. This figure is somewhat higher than the $1453 savings suggested by the statistical modeling study (15), largely the result of the more dramatic reduction in surgical rates seen in this study. The savings in the elderly population may be higher still, because it has been shown that 15% of Medicare-aged patients are rehospitalized within 1 month of thyroidectomy (23). This figure may be partially offset if AGEC suspicious results increase the rate of operation on cytologically indeterminate nodules above the historical 74%, though this study did not address that issue. Furthermore, the conclusions of this study depend on nodules that are AGEC benign not acquiring suspicious characteristics requiring surgery over a longer time frame, which should be the subject of longer term follow-up studies.

This study investigated the decision-making of endocrinologists experienced in managing the entire continuum of care of patients with thyroid nodules and cancer. We believe the findings will be widely applicable, since the study included 51 endocrinologists practicing in 11 states, representing local practice patterns across a wide geographic range. However, it is not clear to what extent, if any, physician decision-making might differ for primary care physicians or other nonendocrinologist specialists.

In conclusion, this study confirms the clinical utility (24) of a novel genomic test. Following a cytology indeterminate result, patient and physician decision-making regarding diagnostic thyroid surgery was effectively modified based on the results of a genomic test whose NPV for a benign nodule exceeds 95%. At the time of this study, clinical decisions were made based on two modest-sized clinical validation studies. After this study was completed, a large prospective validation study of the AGEC was published showing a similar overall NPV of 94%–95% for AUS/FLUS and Follicular/Hürthle Cell Neoplasm subtypes (25). The NPV for cytology that proved Suspicious for Malignancy was lower, however (85%), indicating that the AGEC should not be used to avoid a surgery in this cytological subtype, although it may be helpful in the decision to consider lobectomy versus TT. Empirically, the endocrinologists made recommendations similar to those made following a benign cytopathology diagnosis. The magnitude of the reduction in diagnostic surgeries on cytologically indeterminate nodules was ten-fold, from 74% historically to 7.6% when the AGEC result was benign. Despite the small, but finite residual malignancy risk, patients and physicians seem comfortable substituting clinical and sonographic follow-up for diagnostic surgery in a large majority of cases with a benign molecular diagnostic test.

Disclosure Statement

This investigation was funded by a research grant provided to the institutions of D.S.D. and J.P.K. by Veracyte, Inc.

J.C.D., L.F., G.C.K., and R.B.L. are employees of Veracyte, Inc.

B.M. has no competing financial interests.