Interreader Variability in Semantic Annotation of Microvascular Invasion in Hepatocellular Carcinoma on Contrast-enhanced Triphasic CT Images

Abstract

Purpose

To evaluate interreader agreement in annotating semantic features on preoperative CT images to predict microvascular invasion (MVI) in patients with hepatocellular carcinoma (HCC).

Materials and Methods

Preoperative, contrast material–enhanced triphasic CT studies from 89 patients (median age, 64 years; age range, 36–85 years; 70 men) who underwent hepatic resection between 2008 and 2017 for a solitary HCC were reviewed. Three radiologists annotated CT images obtained during the arterial and portal venous phases, independently and in consensus, with features associated with MVI reported by other investigators. The assessed factors were the presence or absence of discrete internal arteries, hypoattenuating halo, tumor-liver difference, peritumoral enhancement, and tumor margin. Testing also included previously proposed MVI signatures: radiogenomic venous invasion (RVI) and two-trait predictor of venous invasion (TTPVI), using single-reader and consensus annotations. Cohen (two-reader) and Fleiss (three-reader) κ and the bootstrap method were used to analyze interreader agreement and differences in model performance, respectively.

Results

Of HCCs assessed, 32.6% (29 of 89) had MVI at histopathologic findings. Two-reader agreement, as assessed by pairwise Cohen κ statistics, varied as a function of feature and imaging phase, ranging from 0.02 to 0.6; three-reader Fleiss κ varied from −0.17 to 0.56. For RVI and TTPVI, the best single-reader performance had sensitivity and specificity of 52% and 77% and 67% and 74%, respectively. In consensus, the sensitivity and specificity for the RVI and TTPVI signatures were 59% and 67% and 70% and 62%, respectively.

Conclusion

Interreader variability in semantic feature annotation remains a challenge and affects the reproducibility of predictive models for preoperative detection of MVI in HCC.

Keywords: Computer Applications-Detection/Diagnosis, Informatics, Liver, Oncology

Supplemental material is available for this article.

© RSNA, 2020

Summary

Although models for noninvasive detection of microvascular invasion in hepatocellular carcinoma on CT images could provide critical preoperative information, interreader variability in semantic annotation remains a challenge for validation of such models.

Key Points

• ■ Recent research has focused on linking descriptive semantic features, observed at contrast-enhanced multiphasic CT or MRI, with the presence of microvascular invasion, a feature closely associated with early recurrence and death in patients with hepatocellular carcinoma (HCC).

• ■ Discordance and variability exist among radiologists for identification and characterization of these semantic features, underscoring the known limitations of using human-observed features for predictive modeling.

• ■ Interreader variability for semantic feature annotation for HCC remains a challenge and affects reproducibility of previously proposed predictive models for preoperative detection of microvascular invasion.

Introduction

The incidence of hepatocellular carcinoma (HCC) continues to rise predominantly as a result of cirrhosis from nonalcoholic steatohepatitis and hepatitis C (1,2). Surgical resection or liver transplantation can be potentially curative (3,4); nevertheless, 40% of patients who underwent resections and 20% of patients who underwent a transplant have biologically aggressive HCCs at explant pathologic examination that result in early recurrence and death (4–6). Microvascular invasion (MVI) is a critical histopathologic feature associated with early recurrence and death (7,8). Because needle biopsy sampling is inadequate (9), the current practice relies on gross tumor diameter and number (Milan criteria), measured at preoperative imaging, to predict tumor biology (10). However, imaging often under- or overestimates tumor size (11), limiting the definitive diagnosis of MVI to pathologic confirmation of the surgical specimen. Recent research has focused on linking descriptive radiographic semantic features of biologically aggressive HCCs categorized using a fixed vocabulary (12–14). For example, Banerjee et al showed an association between MVI and the presence of internal arteries, absence of a hypoattenuating halo, or a sharp transition in attenuation between the tumor and adjacent liver parenchyma observed on contrast material–enhanced cross-sectional images (13), described as the radiogenomic venous invasion (RVI) trait. Renzulli et al combined newly defined semantic imaging features with the radiogenomic algorithm described previously by Segal et al to predict the presence of MVI (12,14) (two-trait predictor of venous invasion [TTPVI]). Although the results show promise, these studies remain to be validated in independent cohorts and across multiple centers, each with a heterogeneous population and variable imaging protocols. It is conceivable that unequivocal identification and categorization of these descriptive features by individual radiologists could be problematic and lead to discordance (15,16).

The purpose of this study was to measure the ability of radiologists to identify and characterize the semantic features described previously by Banerjee et al (13) and Renzulli et al (14) in an independent cohort. To accomplish this, we studied the interreader agreement for individual semantic features previously described by these investigators. Our secondary aim was to investigate the diagnostic accuracy of predicting MVI using the RVI imaging traits described by Banerjee et al (13), as well as the TTPVI combined with other features, such as tumor margins described by Renzulli et al (14), and to test the reproducibility of these models in an independent cohort imaged with diverse imaging parameters and annotated by different radiologists, mimicking a real-world scenario.

Materials and Methods

Study Design

This retrospective study was approved by the institutional review boards of participating institutions, and all data were handled in compliance with the Health Insurance Portability and Accountability Act. Patient data were accrued from a tertiary transplant center and an affiliated Veterans Affairs medical center. Medical records of adult patients (age ≥ 18 years) who underwent surgical resection between February 2008 and June 2017 for an untreated, solitary liver mass suspected to be HCC were reviewed (n = 180). Within this cohort, patients were excluded if (a) final pathologic diagnosis included cholangiocarcinoma, mixed HCC and cholangiocarcinoma, or rare variants, such as fibrolamellar HCC (n = 18); (b) the patient underwent contrast-enhanced MRI (n = 28); (c) the patient was imaged outside the 3-month preoperative window (n = 10); or (d) if suboptimal imaging (slice thickness > 3 mm, poor contrast enhancement, and/or single-phase imaging) occurred (n = 35). Thus, all included patients had a confirmed diagnosis of HCC with preoperative, triphasic, contrast-enhanced CT with slice thickness of 3 mm or less for both the arterial and portal venous phases obtained within 3 months prior to surgery. The final cohort consisted of 89 patients (median age, 64 years; age range, 36–85 years; 70 men). A small subset (26 patients) was used for pilot quantitative analysis; since then, the cohort was expanded to whatever our total number was, the number of radiologists was increased, and a center was added (Veterans Affairs) (15).

CT Imaging

Contrast-enhanced CT examinations were performed with CT scanners from Discovery GE (n = 52) (GE Healthcare, Piscataway, NJ), Siemens (n = 36) (Siemens Healthcare, Germany), or Toshiba (n = 1) (Toshiba Medical Systems, Otawara, Tochigi Prefecture, Japan), with a single collimation width in the range of 0.6–1.25 mm, slice thickness in the range of 0.625–3 mm, tube voltage between 80 and 140 kVp, and tube current between 90 and 789 mA. Triphasic CT images were obtained from patients using a single-energy source and an automated bolus-tracking technique with arterial phase images and portal venous phase images captured at 30 and 70 seconds, respectively, after contrast agent injection. Image acquisition and reconstruction details are provided in Table E1 (supplement).

Semantic Features and Interreader Agreement

Triphasic CT images in the arterial and portal venous phase were retrieved from the participating institution’s picture archiving and communication system. Images were viewed and annotated independently on ePAD, a freely available quantitative imaging informatics platform (17), by three radiologists (B.P., A.K., and N.K, with 10, 4, and 15 years of experience, respectively), henceforth referred to as reader 1, reader 2, and reader 3, in no particular order. Images acquired during the arterial and portal venous phase were viewed in parallel using a liver window and level setting (window width: 150 HU, window level: 30 HU). On the basis of the concepts reported by Banerjee et al and Renzulli et al (13,14), the following semantic features associated with MVI were annotated in the arterial and the portal venous phases: presence of discrete internal arteries in the arterial phase, presence of discrete internal arteries in the portal venous phase, peritumoral enhancement (defined as a detectable portion of peritumoral tissue that enhances during the arterial phase and becomes isointense to background liver on the portal venous phase [14]), and the presence of a hypoattenuating halo on the portal venous phase. In the absence of a hypoattenuating halo on the portal venous phase, the readers were asked to additionally annotate the presence or absence of a tumor-liver difference, defined as the presence of a focal or circumferential sharp transition between the tumor and the adjacent liver parenchyma (13). Finally, readers were asked to characterize the tumor border on the portal venous phase as smooth, nodular focal, nodular crescent, multinodular, or infiltrative. All readers were provided with the published text and supplementary data from Banerjee et al and Renzulli et al for reference, including published images that illustrated each feature.

A consensus panel with all three readers convened 180 days after the last independent annotation was recorded. Immediately prior to the consensus annotation, feature definition and description were briefly discussed among the three readers along with a visual review of published illustrative images to ensure agreement. During consensus annotation, the readers resolved cases of disagreements by discussion. All readers were blinded to the clinical history, pathologic data, and patient outcome throughout the study.

MVI Decision Tree Classification

The models used to detect MVI in the Banerjee et al and Renzulli et al studies were evaluated using the annotations created by each reader independently and consensus annotations. Results for both, single reader for each model and consensus for each model, are reported herein. To evaluate the RVI signature introduced by Banerjee et al, three semantic features, observed in the portal venous phase, were used to develop a predictive model: internal arteries, hypoattenuating halo, and tumor-liver difference in the absence of a hypoattenuating halo (13). Similarly, to evaluate the TTPVI signature proposed by Renzulli et al, two semantic features were used to create a model: internal arteries on the arterial phase and hypoattenuating halos on the portal venous phase (14). Renzulli et al further modified their TTPVI model by incrementally adding tumor margins (smooth vs nonsmooth on the portal venous phase), peritumoral enhancement, and finally a model that combined all: TTPVI, margins, and peritumoral enhancement. Figure 1 shows the decision trees for each of the five models. Each model was evaluated based on the annotations of each of the three single readers, as well as based on the consensus panel annotation.

Figure 1:

Schematic representation of the decision tree models published by, A, Banerjee et al (13) and by, B–E, Renzulli et al (14). mVI = microvascular invasion, RVI = radiogenomic venous invasion, TTPVI = two-trait predictor of venous invasion.

Statistical Methods

To measure interreader variability, we used Cohen κ applied to annotations of each pair of readers, and Fleiss κ to calculate interreader variability across the three readers. Interreader agreement was defined as follows for Cohen κ and Fleiss κ: less than 0.21 poor agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and 0.81 to 1.00 excellent agreement. κ confidence intervals (CIs) were calculated using Stata kappaetc module (Stata, College Station, Tex). Area under the receiver operating characteristic curve (AUC) was used to evaluate the predictive models. Model CIs were calculated using the bootstrap method; differences between consensus and single-reader models were tested using the DeLong AUC comparison method implemented using the R package “nsROC” (R Foundation for Statistical Computing; https://www.r-project.org/) (18). To correct for repeated testing, we used the Bonferroni method. P values < .003 were considered statistically significant.

Results

To determine the ability of radiologists to identify and characterize semantic features of HCC described by Banerjee et al and Ranzulli et al, CT image data from 89 patients (median age, 64 years; 70 men) were assessed. Demographic data and tumor characteristics of the patients are described in Table 1. The mean tumor diameter, measured in the axial plane was 6 cm (median 4.2 cm, standard deviation 4.61 cm, interquartile range 3–7.5 cm). Twenty-nine patients had MVI based on the postoperative surgical pathology reports.

Table 1:

Baseline Characteristics and Patient Demographics

Semantic Features and Interreader Agreement

The three readers assessed six different features of HCC across arterial phase and portal venous phase images. Two features were annotated on arterial phase images: (a) discrete internal arteries and (b) peritumoral enhancement. Four features were annotated on portal venous phase images: (a) discrete internal arteries, (b) hypoattenuating halo, (c) tumor margin, and (d) tumor-liver difference. Two-reader agreement, as assessed by pairwise Cohen κ statistics, varied as a function of feature and imaging phase, ranging from 0.02 (tumor-liver difference on portal venous phase) to 0.6 (discrete internal arteries on arterial phase); three-reader Fleiss κ varied from −0.17 (tumor-liver difference on portal venous phase) to 0.56 (discrete internal arteries on arterial phase) (Table 2). Overall, the data demonstrated poor-to-moderate interreader agreement with wide variation in the concordance for individual features. Examples of the annotated features are found in Figure 2.

Table 2:

Pairwise (Cohen κ) and Three-Reader (Fleiss κ) Agreement

Figure 2:

Axial images from contrast-enhanced CT that demonstrate semantic features with high and low concordance between readers. In general, the presence of internal arteries had high concordance between readers. This is demonstrated in, A, by a large, right-lobed hepatocellular carcinoma (HCC) with internal arteries (arrow) visualized during the arterial phase of triphasic CT in a 63-year-old man. B, Similarly, the presence of a hypoattenuating halo circumscribing a large segment IV HCC in the portal venous phase (arrow) in a 46-year-old woman had high concordance between readers. C–H, Semantic features with variable concordance. C, Axial CT image of a liver segment VIII HCC in a 63-year-old man, which illustrates the presence of the liver-tumor difference feature (arrow) captured on the portal venous phase that was identified by all readers. D, A case with high reader disagreement on the presence of the tumor-liver difference feature (arrow) in a 66-year-old male patient with a segment IV HCC. Likewise, E and F, illustrate good interreader agreement on the presence of peritumoral enhancement (arrow), E, at the arterial phase surrounding a segment IV HCC that became isointense (arrow), F, at the portal venous phase in a 74-year-old man. G, Arterial phase and, H, portal venous phase demonstrate low interreader agreement for the presence of peritumoral enhancement (arrow) in a 70-year-old male patient with a liver segment VIII HCC.

Single-Reader Models of MVI Prediction

We next assessed performance for each reader when the Banerjee et al and Renzulli et al models were applied to our data set. The sensitivity, specificity, positive predictive value, and negative predictive values for each model across all three readers and the consensus reading are shown in Table 3. Of the three readers, models evaluated using reader 3 annotations came closest to that published by Banerjee et al and Renzulli et al. For the three-feature RVI model introduced by Banerjee et al, reader 3 had an accuracy of 61 of 89 (69%; 95% CI: 56%, 84%), with a 46 of 89 (52%; 95% CI: 31%, 74%) sensitivity, and a 69 of 89 (77%; 95% CI: 66%, 85%) specificity. Similarly, for the TTPVI model introduced by Renzulli et al, reader 3 reported an AUC of 0.74 (95% CI: 0.47, 0.88), with a 60 of 89 (67%; 95% CI: 33%, 99%) sensitivity and a 66 of 89 (74%; 95% CI: 53%, 98%) specificity, again lower than the AUC of 0.85 reported by Renzulli et al. Using the annotation from reader 3, the AUC for the model based on tumor margins (smooth vs nonsmooth margins) was 0.64 (95% CI: −0.43, 0.78) with 56 of 89 (63%; 95% CI: 33%, 99%) sensitivity and 50 of 89 (56%; 95% CI: 37%, 75%) specificity. Results from reader 2 came closest to that of published data for both peritumoral enhancement alone with 20 of 89 (22%; 95% CI: 0%, 50%) sensitivity, 85 of 89 (95%; 95% CI: 86%, 100%) specificity, and an AUC of 0.65 (95% CI: 0.54, 0.74) compared with the published AUC of 0.76 and the combined model: 10 of 89 (11%; 95% CI: 0%, 33%) sensitivity, 87 of 89 (98%; 95% CI: 92%, 100%) specificity, and an AUC of 0.62 (95% CI: 0.54, 0.68) compared with the published AUC of 0.90. Figure 3 shows receiver operating characteristic curves for all models.

Table 3:

Single-Reader and Consensus Model Performance for the Evaluated Models

Figure 3:

Receiver operating characteristic curves of the five models. A, TTPVI proposed by Renzulli et al (14). B, Single-feature MVI marker using tumor margin. C, Single-feature MVI marker using peritumoral enhancement. D, Combined four-feature model combining features from A, B, and C. E, RVI model proposed by Banerjee et al (13) which uses hypoattenuating halo, internal arteries, and tumor-liver difference features on the portal venous phase. AUROC = area under the receiver operating characteristic curve, MVI = microvascular invasion, RVI = radiogenomic venous invasion, TTPVI = two-trait predictor of venous invasion.

Consensus Model of MVI Prediction

Following evaluation of the models using individual readers, we then sought to investigate the performance of a consensus panel for feature categorization, followed by retesting the performance of our consensus data on the predictive models of Banerjee et al and Renzulli et al. To identify how the consensus reading differed from the individual readings, we calculated the number of times each reader changed the annotation of a feature during the consensus review (Table 4). The overall trend was of reader 1 and reader 2 converging their annotation to that of reader 3 for most features.

Table 4:

Individual Reader Changes during Consensus Review

For the three-feature RVI model, there was a marginal improvement in sensitivity using consensus data when compared with a single reader, 59% (95% CI: 37%, 87%) versus 52% (95% CI: 31%, 74%), respectively, but a decrease in specificity, 67% (95% CI: 51%, 78%) versus 77% (95% CI: 66%, 85%), respectively, and an AUC of 0.67 (95% CI: 0.46, 0.73). Similarly, for the combined Renzulli et al model, consensus data again demonstrated no improvement in the sensitivity or specificity and an AUC of 0.60 (95% CI: 0.53, 0.69). The sensitivity, specificity, and the AUC for TTPVI, nonsmooth tumor margins, and peritumoral enhancement, all individual components of the combined Renzulli et al model, were as follows: sensitivity 70% (95% CI: 43%, 100%), 70% (95% CI: 50%, 100%), and 11% (95% CI: 0%, 40%), respectively; specificity 62% (95% CI: 50%, 75%), 69% (95% CI: 52%, 91%), and 93% (95% CI: 83%, 100%), respectively; and AUC of 0.70 (95% CI: 0.56, 0.86), 0.73 (95% CI: 0.57, 0.86), and 0.59 (95% CI: 0.51, 0.69), respectively.

Comparison of Single-Reader Model to Consensus Model

For the RVI model, only reader 1 was significantly different from consensus (P = .002). Whereas for the TTPVI model, only reader 2 was significantly different from consensus (P = .002). For tumor margin, all single results were significantly different from consensus (P < .001), highlighting the difficulty in characterizing tumor margins for soft-tissue tumors on the portal venous phase. The confusion matrices for the tumor margin model are shown in Table E2 (supplement). Finally, for the combined model by Renzulli et al and peritumoral enhancement on the portal venous phase, no statistically significant differences were found between the consensus model and any of the single readers.

Discussion

The overall objective of this study was to determine the ability of different radiologists to characterize and identify semantic features of MVI in HCC. In this work, we evaluated five previously published models (13,14) for classifying MVI using semantic features annotated during the arterial and portal venous phases of triphasic contrast-enhanced CT, a standard imaging study for HCC diagnosis. In our analysis, we demonstrated a low-to-moderate interreader agreement for the features included in the models. The results of this study illustrated the limitations of predictive models for MVI based on descriptive traits identified and categorized by individual observers.

For model evaluation and measuring the effect of interreader variability on predictive performance, we used both a single-reader and a consensus panel approach. However, as observed previously, the results using the consensus model approach were similar to that of a single-reader approach. The wide CIs reflected difficulty in annotating model features and/or the heterogeneity within the cohort. HCCs are often large, visually heterogeneous, and with ambiguous margins. Thus, we observed that, for each independent reader, consistent identification of features was a challenge because each individual reader’s interpretation was subjectively different. However, in both single-reader and consensus settings, the results from Renzulli et al and Banerjee et al could not be reproduced, indicating the inherent difficulty in identifying and characterizing the features. Despite our best efforts, our instruction on feature characteristics was confined to the descriptive text and the limited number of printed radiographic images published in the two studies. These training constraints therefore limit reproducibility at other institutions and in turn, limit the effectiveness of these predictive models. Finally, scanners and associated imaging parameters vary and could affect visual appearance, and consequently human annotation of semantic features, and thus affect reproducibility.

There is a profound need to exploit computational tools to help characterize the tumor biology of HCC. With a hazard ratio of up to 4, MVI is currently the most important predictor of recurrence and overall survival (7,19). Current practice relies on gross tumor diameter and number (Milan criteria), measured on preoperative images as a surrogate marker for prognosis (4,10,20). Imaging, however, understages or overstages 40% of HCC (11) and even when accurate, 30% of HCCs that are within Milan criteria are biologically aggressive whereas 50% of HCCs outside of criteria are not (7,21–23). To address these limitations, recent research has focused on linking descriptive radiologic semantic features of biologically aggressive HCCs, categorized using a fixed vocabulary, to its genomic profile (12–14,24). In these studies, the classification models were derived from homogeneous cohorts with tightly controlled imaging parameters. Differences in annotations among readers were resolved by consensus, which is not applicable in clinical practice. Thus, we suggest that these currently published semantic feature–based classification models should be expanded further and tested with larger cohorts before being applied in a clinical setting. Development of other types of models that do not have these limitations or have them to a lesser extent could potentially substitute or augment semantic-based models. As demonstrated in previous studies of HCC (15,16) and other cancers (25–28), quantitative radiomic features can be extracted from a region of interest using semiautomated techniques that minimize training requirements and variations in radiologist interpretation. Because HCC are visibly heterogeneous, radiomic features, which measure quantities undetectable by the human eye (29), have the potential to surpass the performance of semantic features, as demonstrated by some recent studies of HCC (15,30–32).

Our study had some limitations, including its retrospective nature. First, the models we evaluated were based on the set of semantic features proposed in two seminal studies (13,14). Other semantic features that have higher association of MVI or that can be easily described and/or visualized in published data may exist but were not evaluated in this study. Second, our study examined the performance of three radiologists in a limited number of patients from a heterogeneous cohort and with varying imaging parameters, factors that could affect identification and classification of semantic features. Third, our study was limited to triphasic CT imaging. Whereas CT is widely used for diagnosis and management of HCC, the exclusion of patients with MRI findings further limited the data set. Finally, the images used in our study were obtained using a range of scanning parameters and devices. Although this variability in images is common in a clinical setting and is important to evaluate, studies with controlled imaging parameters could provide additional value. Future prospective studies in larger cohorts obtained on similar scanners and reproducible image acquisition technique may be an alternate solution to improving performance.

In summary, use of algorithms that combine semantic features of tumors at triphasic CT to predict MVI may have limitations because of interreader variability, variations in patient characteristics among clinical cohorts, and variations among scanning equipment and settings. The best predictive single feature (14), peritumoral enhancement, which reflects pathophysiology associated with HCC, had poor performance in our study. Our work demonstrated the importance of additional research that is needed before clinical adoption of semantic feature models for a noninvasive diagnosis of MVI using standard-of-care imaging such as triphasic CT.