Variation in Diagnostic Performance among Radiologists at Screening CT Colonography

We observed relatively consistent performance among radiologists interpreting results of CT colonographic examinations, especially when compared against reported data for optical colonoscopy, including a difference in the range of detection of adenomas and advanced neoplasia of less than twofold.

Abstract

Purpose:

To assess the variation in diagnostic performance among radiologists at screening computed tomographic (CT) colonography.

Materials and Methods:

In this HIPAA-compliant, institutional review board–approved study, 6866 asymptomatic adults underwent first-time CT colonographic screening at a single center between January 2005 and November 2011. Results of examinations were interpreted by one of eight board-certified abdominal radiologists (mean number of CT colonographic studies per reader, 858; range, 131–2202). Findings at CT colonography and subsequent colonoscopy were recorded, and key measures of diagnostic performance, including adenoma and advanced neoplasia detection rate, were compared among the radiologists.

Results:

The overall prevalence of histopathologically confirmed advanced neoplasia was 3.6% and did not differ significantly among radiologists (range, 2.4%–4.4%; P = .067; P = .395 when one outlier was excluded). Overall, 19.5% of polyps detected at CT colonography proved to be advanced neoplasia and did not differ significantly among radiologists (range, 14.4%–23.2%; P = .223). The overall per-polyp endoscopic confirmation rate was 93.5%, ranging from 80.0% to 97.6% among radiologists (P = .585). The overall percentage of nondiagnostic CT colonographic examinations was 0.7% and was consistent among radiologists (range, 0.3%–1.1%; P = .509).

Conclusion:

Consistent performance for adenoma and advanced neoplasia detection, as well as other clinically relevant end points, were observed among radiologists at CT colonographic screening.

© RSNA, 2013

Introduction

According to long-standing World Health Organization guidelines, several criteria exist for evaluation of the quality of screening procedures, including validity, reliability, cost, and acceptance (1). With respect to these characteristics, computed tomographic (CT) colonography has been well-studied and has been found to be a valid screening test for colorectal cancer and other advanced neoplasia (2–8), as well as a test demonstrating both cost-effectiveness (9,10) and a high degree of acceptance among patients (11,12). However, because of a lack of third-party coverage and other issues, widespread adoption of CT colonography as a screening modality has yet to occur in the United States.

Beyond limited data from clinical trials (3), one important characteristic that has not been studied is the reliability among readers of CT colonographic images in clinical practice with respect to clinically relevant end points, including adenoma detection rate, which was recently validated as a quality target for colonoscopy (13). Studies of optical colonoscopy and flexible sigmoidoscopy have shown wide variation in detection of colorectal neoplasia related to the operator-dependent differences in colonoscopic technique (14–17). Thus, the actual clinical performance of colonoscopy may fall short of the rates reported in the literature. Although the CT colonographic technique is well standardized and image acquisition is not operator dependent, important differences in clinically relevant lesion detection may still exist. Our purpose was to assess the variation in diagnostic performance among radiologists at screening CT colonography.

Materials and Methods

Study Design

This Health Insurance Portability and Accountability Act–compliant retrospective study was approved by our institutional review board (University of Wisconsin School of Medicine and Public Health, Madison, Wis). The need for signed informed consent was waived. From January 2005 through November 2011, we prospectively enrolled 7629 consecutive individuals referred from general medical practice for a first-time CT colonographic examination for the purpose of colorectal cancer screening. The results of each examination were interpreted by one of 11 abdominal radiologists. We excluded patients with a history of colorectal cancer, inflammatory bowel disease, polyposis syndromes, and colorectal surgery. We also excluded patients who were suspected of having symptomatic colorectal cancer and those who were referred following incomplete colonoscopy. To allow meaningful comparison of performance among radiologists, we included only the results of CT colonographic screening examinations interpreted by the eight radiologists who read more than 100 total screening CT colonographic examinations during the study interval and excluded those read by the three radiologists who had interpreted fewer than 100 examinations. A total of 6866 asymptomatic patients comprised the screening CT colonography study population (Fig 1).

Figure 1:

Flow diagram of study cohort. CTC = CT colonography.

A positive CT colonographic examination was defined as one in which any polyp of 6 mm or larger in size was found. A negative CT colonographic examination was defined as one in which the findings did not meet the criterion for a positive examination, including an examination with only isolated diminutive (<6-mm) polyps. In accordance with the CT Colonography Reporting and Data System (18), patients with any large polyps (polyps ≥ 10 mm in size) or more than two small polyps (polyps 6–9 mm in size) found at CT colonography were referred for colonoscopy with polypectomy. Patients with one or two small polyps found at CT colonography were given the option of either colonoscopy with polypectomy or CT colonography polyp surveillance. Patients with a negative examination result at CT colonography received a recommendation to undergo routine screening in 5 years. For calculation of endoscopic referral rates, patients were considered to be referred for colonoscopy if the interpreting radiologist recommended colonoscopy following CT colonography, regardless of whether the patient actually underwent colonoscopy. An examination was classified as nondiagnostic if at least one colonic segment could not be visualized on any series of images because of poor preparation, poor distention, or a combination of the two, thereby preventing exclusion of large polyps. This corresponds to category C0 in the CT Colonography Reporting and Data System classification (18).

The results of colonoscopy and polypectomy, as well as the final histopathologic result at histopathologic analysis, were recorded for all patients who were undergoing colonoscopy. Concordance was recorded for the CT colonographic findings of patients who were undergoing colonoscopy with polypectomy by using colonoscopic findings as the reference standard. In discordant cases where a CT colonographic finding was not confirmed, the finding was considered to be a CT colonographic false-positive result unless additional imaging or repeat colonoscopy demonstrated that the lesion was missed at initial colonoscopy.

CT Colonographic Technique

The CT colonographic technique used in our screening program has been described in detail elsewhere (19). Briefly, patients undergo a bowel preparation protocol beginning 1 day prior to CT colonography consisting of a cathartic cleansing agent (sodium phosphate prior to 2008 and magnesium citrate thereafter); polyethylene glycol was substituted as needed in a small number of patients. Contrast material tagging of residual fluid and fecal material was achieved with 2.1% wt/vol barium and diatrizoate meglumine and diatrizoate sodium (Gastrografin; Bracco Diagnostics, Princeton, NJ). During the CT colonographic examination, colonic insufflation was achieved and maintained throughout image acquisition by using automated continuous carbon dioxide delivered through a rectal catheter. Patients were routinely scanned in both supine and prone positions, with decubitus positioning as needed. Images were acquired with 8- to 64-section multidetector CT scanners by using 1.25-mm collimation, 1-mm reconstruction interval, 120 kVp, and either a fixed tube current–time product (50–75 mAs) or tube-current modulation (range, 30–300 mA). The radiologist’s interpretation of CT colonographic examination results was performed by using both three-dimensional reconstructions for initial polyp detection and two-dimensional cross-sectional images for secondary detection and polyp confirmation (4,6,19,20). A single CT colonographic software system (V3D; Viatronix, Stony Brook, NY) was used during the entire study period.

Interpreting Radiologists

Eight radiologists met the inclusion criterion of having interpreted the results of at least 100 CT colonographic examinations during the study interval, with a mean of 858 examinations per reader and a range of 131–2202 (Table 1). All radiologists were board certified, with an average of 12 years in practice by the conclusion of the study (range, 2–31 years). Six of eight had completed a fellowship in abdominal radiology, and three of those had dedicated instruction in CT colonography, including a focused continuing medical education course and teaching file review of at least 50 cases. Six of eight worked solely in the subspecialty area of academic abdominal imaging, while two had a more general practice. Two of the radiologists have developed and conducted a CT colonographic training course for other radiologists. Regardless of prior training, the experience level of each radiologist included in this study is best represented by the reported number of CT colonographic case interpreted during the course of this study.

Table 1.

Demographic Characteristics of Study Cohort according to Radiologist

^*Data are means ± standard deviations, except where specified otherwise.

Statistical Analysis

The Pearson χ² test was used to test for differences in categorical variables. The Student t test and analysis of variance were used, where appropriate, to test for differences in continuous variables. A two-tailed P value of less than .05 was used as the criterion for a significant difference. A priori power analysis for χ² testing utilizing appropriate degrees of freedom revealed that to achieve a statistical power of 0.99 for a conventionally small effect size of 0.10 and a standard acceptable limit of α error of .05, a sample size of approximately 2900 patients would need to be enrolled. This is less than half of our actual study population and allows us to meaningfully fail to reject the null hypothesis when comparison among our radiologists shows no difference for a given categorical variable.

A multiple regression analysis (JMP 98.0 version 8.0; SAS, Cary, NC), including patient age, patient sex, and interpreting radiologist as independent variables, was used to predict the rate of advanced neoplasia for patients with small polyps who were undergoing CT colonographic surveillance and to generate an adjusted advanced neoplasia rate for each radiologist. P values were calculated for each radiologist, with a second P value calculated after the exclusion of any significant outlier in the group.

Results

The demographic characteristics of the CT colonographic patient population are listed in Table 1. The mean age and male-to-female ratio for the entire population were 56.7 years ± 7.2 (standard deviation) and 3175:3691, respectively. There was no significant difference for these characteristics between each radiologist’s cohort.

Of 6866 individuals screened, the overall prevalence of histopathologically confirmed advanced neoplasia was 3.6% (250 of 6866), with no significant difference seen among radiologists (range, 2.4%–4.4%; P = .067) (Table 2). The overall prevalence for any nondiminutive adenoma (including advanced neoplasia) was 6.4% (440 of 6866), again with no significant difference seen among radiologists (range, 3.8%–7.4%; P = .322). These values with 95% confidence intervals for each radiologist are depicted in Figure 2. The overall rate of nondiagnostic studies was 0.7% (51 of 6866), with no significant difference seen among radiologists (range, 0.3%–1.1%; P = .509).

Table 2.

Per-Patient and Per-Polyp Measures according to Radiologist

^*Numbers in parentheses are percentages. Percentages were calculated on the basis of the total number of patients for each radiologist.

^†Numbers in parentheses are percentages. Percentages were calculated on the basis of all polyps in patients who underwent polypectomy for each radiologist.

Figure 2:

Percentage of patients with adenomas and advanced neoplasia according to radiologist. Bubble size represents relative case load. Error bars = 95% confidence intervals.

Overall, 19.5% (300 of 1541) of all polyps found at CT colonography by radiologists were ultimately proved at histopathologic analysis to be advanced neoplasms, with a range among radiologists of 14.4%–23.2% (P = .223). The overall per-polyp endoscopic confirmation rate (positive predictive value) was 93.5% (1043 of 1116), ranging from 80.0% to 97.6% among radiologists (P = .585), and the corresponding overall per-polyp false-positive rate was 6.5%. The overall prevalence of histopathologically confirmed hyperplastic polyps was 2.1% (33 of 1571), with a range among radiologists of 0.4%–3.4% (P < .001). Among all radiologists, the mean percentage of patients with a positive examination was 14.6% (1001 of 6866), with a range among radiologists of 10.3%–21.0% (P < .001). Among all radiologists, the overall per-patient endoscopic false-positive rate was 5.2% (35 of 677), with a range among radiologists of 0%–16.7% (P = .535).

Of 1001 patients with a positive result of screening CT colonography, 59.1% (592 of 1001) were referred for colonoscopy with polypectomy, and 40.9% (409 of 1001) entered CT colonographic polyp surveillance. There was no significant difference observed among radiologists in the colonoscopy referral rate for patients with a positive result of CT colonography (range, 33.3%–63.5%; P = .421), although there was a significant difference observed among radiologists in overall colonoscopy referral rate (mean, 8.6%; range, 4.6%–11.9%; P < .001). Of 592 total patients with positive results who were referred for colonoscopy, 569 (96.1%) actually completed the procedure, with no significant difference among radiologists with respect to completion rate (range, 86.2%–100.0%; P = .118). Sixteen patients (2.7%) did not undergo colonoscopy because the patient declined further testing (n = 11), the patient died of other causes before colonoscopy could be performed (n = 3), or the patient had a medical condition that precluded colonoscopy (n = 2). As of this writing, seven patients (1.2%) are still awaiting colonoscopy. Of the 409 patients who elected to undergo polyp surveillance, 108 (26.4%) ultimately underwent colonoscopy, while 301 (73.6%) remained in surveillance.

The range of advanced neoplasia detection rates after multivariate adjustment for small polyps in patients undergoing CT colonographic surveillance was 2.8%–5.1% (P = .035). Individual advanced neoplasia rates before and after this adjustment are listed in Table 3. After excluding the one performance outlier (radiologist F), the differences in advanced neoplasia detection among radiologists before (P = .395) and after (P = .484) multivariate adjustment were not significant.

Table 3.

Advanced Neoplasia Rates before and after Adjustment with Multivariate Analysis

^*When radiologist F is excluded, the total number of patients is 5437.

^†Numbers in parentheses are 95% confidence intervals.

^‡P values indicate a significant difference.

Discussion

Numerous studies have established CT colonography as a valid (2–8) and cost-effective (9,10) colorectal cancer screening test that is well-accepted by patients (11,12). The present study offers evidence that CT colonographic performance is also highly reproducible among radiologists with respect to clinically significant end points. Our results from a large screening cohort of 6866 patients demonstrate a relatively narrow range in advanced neoplasia detection rate among radiologists who were interpreting results from screening CT colonography. The difference among radiologists in advanced neoplasia detection rate was less than twofold and not significant (P = .067). In addition, there was a similar difference (less than twofold) among radiologists in the detection of all adenomas (P = .322).

In contrast, the variation in adenoma detection rate among gastroenterologists performing screening optical colonoscopy is substantially larger, ranging from three- to sixfold or more (15–17). In their study of 2053 patients, Barclay et al (15) reported a sevenfold difference in advanced adenoma detection among gastroenterologists within a single respected practice. The comparatively low variation in advanced neoplasia detection rate and adenoma detection rates seen among radiologists who are interpreting CT colonographic results has important implications for diagnostic performance in the broader clinical setting. Because adenoma detection rate has recently been validated as a clinical surrogate marker for diagnostic performance (13), the minimal variation among radiologists suggests that CT colonography represents a reproducible, consistent screening examination. This affects dissemination into clinical practice, as quality performance is more likely to be maintained.

Multiple regression analysis performed to correct for the presence of subcentimeter polyps in patients undergoing CT colonographic surveillance increased the estimated advanced neoplasia rate in this screening cohort to 4.2%, which is slightly higher than the rate previously reported for optical colonoscopic screening within our generally healthy patient population (3.4%) (4) and slightly lower than the pooled rate seen among other less healthy cohorts (5.6%) (21). The overall range of advanced neoplasia detection rates remained less than twofold (2.8%–5.1%) after this adjustment. The results of this analysis also confirm the consistency among seven of eight radiologists (P = .484) in regard to the advanced neoplasia detection rate, while one radiologist detected significantly fewer advanced neoplasms. This finding confirmed a trend seen in the initial analysis of the advanced neoplasia detection rate, where exclusion of this reader resulted in an increase in the P value from .067 to .395.

Two key factors in CT colonographic practice account for the lower total adenoma yield relative to that observed at screening colonoscopy: the accepted practice of not reporting isolated diminutive lesions of less than 6 mm in size, which accounts for most of the difference, and offering patients the option of CT colonographic surveillance for small polyps of 6–9 mm (17). Most of the harvested polyps at screening colonoscopy are subcentimeter, of which most are diminutive (22). In terms of both clinical relevance and cost-effectiveness, attempted universal removal of all nonadvanced subcentimeter polyps (whether tubular adenomas or nonneoplastic lesions) appears to be a relatively costly and inefficient strategy (4,10,18,23). Primary CT colonography approaches screening with a different philosophy that seeks to greatly reduce the number of polypectomies involving very-low-risk lesions while simultaneously preserving the advanced neoplasia yield. That only 2% of our CT colonographic screening cohort had histopathologically confirmed hyperplastic lesions, compared with 21% for screening colonoscopy in the series by Barclay et al (15) underscores this difference in paradigm. CT colonography, however, concentrates the number of large hyperplastic and serrated polyps, largely ignoring the common diminutive lesions. The high percentage of advanced neoplasms per resected polyp in our CT colonographic screening program compared with screening colonoscopy has important implications for containing costs and complications.

The differences in variation among practitioners that exist between CT colonography and colonoscopy likely reflect intrinsic differences in the nature of the two screening modalities. With respect to colonoscopy, it is well recognized that technical expertise is required to navigate the colonoscope to the cecum and that mucosal visualization, especially in areas behind folds and along the inner curvature of the colon, is dependent on the skill of the colonoscopist (24,25). Essentially, the colonoscopy “data set” is being simultaneously generated and interpreted extempore by each individual colonoscopist. For CT colonography, the situation is markedly different. Once the colon is prepared and distended, the patient is scanned with a defined CT protocol—obviating any operator-dependent skill—to obtain the static data set, which is then subsequently interpreted (and further reviewed, if needed).

The low variation among radiologists in the nondiagnostic examination rate further supports the reproducible nature of CT colonography. We report a nondiagnostic CT colonography rate of 0.7%, with a narrow range among individual radiologists of 0.3%–1.1% (P = .509). For colonoscopy, researchers in one large multicenter screening study in more than 3000 individuals of 50–75 years of age reported a fourfold higher colonoscopy failure rate of 2.8%, with a range among centers of 0.7%–6.7% (26). Investigators with other colonoscopy experiences have reported much higher rates of failure to intubate the cecum (27). Besides inadequate bowel preparation, colonoscopy can fail for anatomic reasons, such as increased colonic tortuosity or redundancy or inability of the patient to tolerate sedation. CT colonography does not require sedation, leaving inadequate bowel preparation and luminal distention as the primary reasons for CT colonography failure. With the use of oral contrast material tagging, preparation failure for CT colonography is rare.

This study had several potential limitations. Because CT colonographic screening is dependent on patient referral from primary care physicians, there was possible sample bias that could limit generalization of results. However, because our primary aim was analysis of variation among CT colonographic readers, any sample bias would have been evenly distributed among radiologists. There was also possible selection bias because the distribution of patients among interpreting radiologists was not randomized. However, there was no significant difference in the demographic factors of the study patients among radiologists. In addition, all examinations were scheduled in advance without prior knowledge of which radiologist would be interpreting the results of CT colonographic examinations on any given day. In this study, all radiologists were from a single academic center, and consequently, our results may not translate to other screening programs but may nevertheless serve as a reference. Given the clinical nature of this study, patients with negative results of CT colonographic examinations did not routinely undergo polyptectomy, and consequently, no histopathologic data are available for these patients. Last, because all radiologists in this study interpreted examinations by using a combination of three-dimensional reconstructions (primarily for polyp detection) and two-dimensional images (for polyp confirmation and secondary detection) (4,6,19,20), our results may not generalize to centers where a primary two-dimensional approach is utilized.

Our data have important implications for patient care and quality assurance. The ability to track the metrics we have presented and retrospectively review the source data for each CT colonographic examination should allow for identification and correction of sources of variation, including suboptimal interpretation strategies used by individual radiologists. For example, although the advanced neoplasia rate did not vary significantly, there was a significant difference among radiologists in the percentage of patients with positive results (ie, at least one polyp ≥ 6 mm). This suggests that the difference in the rate of cases with positive results at CT colonography is largely composed of patients with clinically nonsignificant, nonneoplastic lesions and is supported by the significant difference seen among radiologists in the detection of hyperplastic polyps (range, 0.4%–3.4%; P < .001). Ostensibly, changes in interpretation strategy or philosophy may allow radiologists with higher rates of positive results to decrease the number of low-risk lesions called, which would lead to fewer instances of unnecessary follow-up colonoscopy. Further research should allow for standardization of best practices, perhaps following in the footsteps of mammography, which has seen great success in the implementation of a lexicon describing findings in addition to other reliability-improving measures (28,29).

In conclusion, we observed consistent performance among radiologists who are interpreting results of CT colonographic examinations, especially when compared against reported data for optical colonoscopy, including a difference in the range of detection of adenomas and advanced neoplasia of less than twofold. This finding is important because reliability in performance is a cornerstone of any successful screening test, and, when combined with the already-demonstrated validity, cost-effectiveness, and patient acceptance, it provides further evidence that CT colonography would be successful if widely implemented as an option for colorectal cancer screening.

Advances in Knowledge.

• • In more than 6866 screening CT colonographic examinations performed at a single center, there was no significant difference seen in advanced neoplasia detection rate among eight radiologists who interpreted results of screening CT colonography (overall rate, 3.6%; range among radiologists, 2.4%–4.4%).

• • The overall per-polyp endoscopic confirmation rate (positive predictive value) was 93.5%, with no significant difference seen among the eight radiologists (range, 80.0%–97.6%); the false-positive rate among polyps for which polypectomy was performed was 6.5%.

• • The overall rate of nondiagnostic CT colonographic examinations was 0.7%, with no significant difference seen among radiologists (range, 0.3%–1.1%).

Implications for Patient Care.

• • Our findings suggest that CT colonography has the potential to be a highly reliable screening test for colorectal cancer.

• • Tracking the metrics we have presented should allow for quality assurance and improvement with respect to CT colonography.

Disclosures of Conflicts of Interest: B.D.P. No relevant conflicts of interest to disclose. D.H.K. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: received consultancy fees from Viatronix and Digital Artforms, and is co-founder of VirtuoCTC. Other relationships: none to disclose. C.H. No relevant conflicts of interest to disclose. A.R. No relevant conflicts of interest to disclose. E.S.B. No relevant conflicts of interest to disclose. P.J.P. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: received consultancy fees from Viatronix, Check-Cap, Bracco, and iCAD and is co-founder of VirtuoCTC. Other relationships: none to disclose.