Inter-observer Agreement on the Assessment of Relative Liver Lesion Signal Intensity on Hepatobiliary Phase Imaging with Gadoxetate (Gd-EOB-DTPA)

Abstract

Purpose

To assess the inter-observer agreement on the qualitative and quantitative evaluation of relative signal intensity of liver lesions on delayed hepatobiliary phase (HBP) MRI with gadoxetate (Gd-EOB-DTPA).

Methods

105 patients with liver lesions, who had delayed HPB MRI using gadoxetate were reviewed retrospectively. For each patient, 4 readers (2 fellows in training and 2 attending radiologists) qualitatively assessed the relative SI of the largest representative lesion on a 5 point scale, and quantitatively measured the relative SI of the lesion to adjacent liver parenchyma using region of interests (ROI). Intra-class correlation (ICC) and kappa statistics with quadratic weights (k) analysis, and maximally selected rank statistic were performed.

Results

Substantial agreement between fellows (k=0.719; ICC=0.705) and almost perfect agreement between attending radiologists (k=0.853; ICC=0.849), was found for both qualitative and quantitative assessment of relative SI on delayed HPB imaging. A cut-off ratio to differentiate between hypointense and iso- to hyperintense lesions by ROI was calculated to be 0.90.

Conclusion

Inter-observer agreement of liver lesion relative SI on delayed HBP imaging is high and may improve with radiologist experience. A cut-off ratio of relative SI at 0.90 may be useful to quantitatively distinguish hypointense from iso- to hyperintense liver lesions.

Introduction

Gadoxetate (gadoxetic acid; Gd-EOB-DTPA) is a widely used gadolinium based contrast agent in liver MR imaging, given its utility in detecting and diagnosing liver tumors [1-8]. It combines the properties of extracellular fluid contrast agents for early dynamic phase imaging and of hepatocyte specific contrast agents for delayed hepatobiliary phase (HBP) imaging. Several studies support the use of delayed HBP imaging to help differentiate between benign and malignant or high risk liver lesions [9-13]. Delayed HBP imaging is often used to distinguish between focal nodular hyperplasia (FNH) and hepatocellular adenoma (HCA), two benign liver lesions with distinct clinical managements because of the risk of malignancy and hemorrhage with HCA [14]. FNH and HCA are often distinguished by their differential uptake of gadoxetate on delayed HBP, as FNH is typically iso- or hyperintense in signal intensity relative to adjacent liver parenchyma, and HCA are commonly hypointense [9-13]. Hepatocellular carcinoma (HCC) demonstrates more variable uptake of gadoxetate on delayed HBP, with the majority being hypo-or isointense in signal intensity, and a minority demonstrating hyperintense signal [15-17]. In contrast to FNH and HCC, non-hepatocellular liver lesions such as liver metastases are almost always hypointense in signal intensity compared to liver parenchyma [6].

Characterizing liver lesions as benign or high risk/malignant tumors with gadoxetate thus relies on the assessment of relative signal intensities with delayed HBP imaging. In routine clinical practice, however, lesions that are similar in signal intensity to liver parenchyma can be interpreted as either hypointense or isointense by different radiologists. It is increasingly recognized that interobserver variability plays an important role in the practice of radiology[18]. For example, a recent study shows the potential limitation of using scoring systems to diagnose HCC, and the limitations in assessing relative signal intensity of liver lesions for the assessment of washout appearance [19]. To date, no study has addressed the potential interobserver variability in assessing liver lesions on delayed HBP with gadoxetate. Few studies have also proposed a quantitative cut-off ratio for distinguishing between hypointense and iso- to hyperintense liver lesions [10, 11, 20].

Thus, the purpose of this study was to assess inter-observer agreement on the qualitative and quantitative evaluation of the relative signal intensity (RelSI) of liver lesions on delayed HBP with gadoxetate.

Materials and Methods

Institutional review board approval with waived informed consent was obtained for this retrospective study.

Patient selection

All patients who underwent more than one liver MRI examination with intravenous contrast between January 2011 and May 2013 were identified through a search of our institution's Picture Archiving and Communication System (PACS) database (Centricity, GE Healthcare Integrated Imaging Solutions). A total of 213 patients had more than one MRI examination performed, in which at least one was performed with gadoxetate (Gd-EOB-DTPA, Bayer Healthcare, Whippany, NJ). Among the 213 patients, 108 patients were excluded because of the absence of a liver lesion on MRI (n=38), prior embolization or radio-frequency ablation (n=64), absence of hepatobiliary phase imaging performed (n=1), severely poor scan image quality (n=3), resection of the liver lesion (n=1) and indeterminate pathology (n=1). A total of 105 patients (59 females, 46 males, mean age 52 years) were included in our study. For each lesion, histopathological confirmation was used whenever possible, either by biopsy or resection pathologic reports. For lesions where histology was not available, diagnosis was made based on their imaging characteristics alone. Among the 105 patients included in the study, 50 patients underwent at least two liver MRI with gadoxetate an average of 5.4 months apart. The remainder had multiple MR examinations, only one of which was performed with gadoxetate.

MR Imaging Protocol

MR imaging was performed on 1.5 (110 cases) or 3.0 (45 cases) Tesla MR scanners (GE HDxt, MR450 or MR750, GE Healthcare, Milwaukee, WI) using a phased-array torso coil. All patients received a standard intravenous bolus dose of 10 mL of gadoxetate disodium (Eovist, Bayer Healthcare, NJ, USA) followed by 20 mL of saline flush.

The delayed HBP images were acquired using 3D T1-weighted gradient-recalled echo (GRE) sequence, with TR/TE 3.8-4.5/1.7-2.1 ms, flip angle = 12 degrees, 3 to 6 mm slice thickness, 29 to 48 cm FOV and 224 × 128 to 320 × 224 matrix size. The delay between the injection and acquisition of the delayed HBP images were recorded.

Image Analysis

The single largest representative liver lesion on delayed HBP imaging was selected from each patient by two radiologists in consensus, who were not part of the 4 readers below. The liver lesion signal intensity (SI) relative to the adjacent liver parenchyma was assessed by 4 separate readers (2 fellows in training and 2 attending radiologists with at least 6 years of post abdominal imaging fellowship experience).

Qualitative analysis – Each radiologist reviewed the MR images on PACS. Using a 5 point scale, each reader assessed the SI of the representative liver lesion relative to the adjacent liver parenchyma: 1 = very hypointense, 2 = mildly hypointense, 3 = isointense, 4 = mildly hyperintense, and 5 = very hyperintense (Fig. 1). In lesions that demonstrated heterogeneous SI, the readers assigned a single value corresponding to the predominant SI.

Fig. 1.

Liver lesions are characterized by its signal intensities as (A) very hypointense; (B) mildly hypointense; (C) isointense and (D) hyperintense.

Quantitative analysis - Each reader measured the SI of the representative liver lesion and the adjacent liver parenchyma, using the circular region of interest (ROI) tool on PACS. To measure the SI of the liver lesion (SI_lesion), each reader was instructed to draw an ROI in a representative slice of the lesion, avoiding the border for volume averaging effect. SI of the background liver parenchyma (SI_liver) was measured on the same slice, by placing the ROI on the liver parenchyma adjacent to the liver lesion, avoiding blood vessels and artifacts. The relative SI ratio was calculated as: RelSI = SI_lesion/SI_liver.

Statistics

The RelSI median and range values were calculated in all patients, as well as in hypointense and iso- to hyperintense lesions for each reader. The intra-class correlations (ICC) were evaluated and kappa statistics with quadratic weights (k) were estimated to assess the agreement between fellows, between attending radiologists, and between the first and second MRIs in patients who had two 2 MRIs performed with gadoxetate [21]. Bland-Altman plot was used to show the agreement between readers as well as between the first and second MRIs in patients with 2 MRIs. In addition, a maximally selected rank statistic approach was applied to identify a cut-off ratio of RelSI to differentiate between hypointense and iso- to hyperintense lesions [22]. All statistical analyses were performed in software package SAS 9.2 (SAS Institute Inc., Cary, NC, USA), and R version 2.13 (The R Foundation for Statistical Computing).

Results

Study Population

In the 105 patients selected for our study, the liver lesions (mean size 2.2 cm, range 0.8-9.4 cm) included 62 metastases, 22 FNH, 8 adenomas, 5 cholangiocarcinomas and 6 hemangiomas. Two additional lesions were considered to be either adenoma or FNH by pathology. Forty-nine of the liver lesions have pathological diagnosis via FNA (n=5), core biopsy (n=27) and resection (n=17), with the remainder 56 liver lesions were diagnosed by their imaging characteristics alone. The mean time from injection to hepatobiliary phase image acquisition is 20 minutes 7 seconds ± 5 minutes 27 seconds.

Qualitative Analysis

The result of the qualitative assessment of the SI of the liver lesions is summarized in Table 1. There was substantial agreement between fellows, k = 0.719 (95% CI 0.588, 0.849), and almost perfect agreement between attending radiologists, k = 0.853 (95% CI 0.780, 0.926). In the 50 patients who had 2 separate MRIs with gadoxetate, there was almost perfect inter-scan agreement for the qualitative assessment of relSI across all readers (Table 2).

Table 1. Agreement on Qualitative Assessment of SI (n=105).

Fellow 1	Fellow 2
Score	1	2	3	4	5
1	42	18	2	1	0
2	4	13	2	0	0
3	3	2	8	1	0
4	0	0	2	5	2
5	0	0	0	0	0
Attending Radiologist 1	Attending Radiologist 2
Score	1	2	3	4	5
1	64	4	2	0	0
2	6	4	1	0	0
3	1	2	9	1	0
4	0	1	2	4	0
5	0	0	0	4	0

Weighted Kappa (95%CI) between fellows = 0.719 (0.588,0.849); attending radiologists = 0.853 (0.780,0.926)

Table 2. Agreement Between First MRI Score and Second MRI score (n=50).

Fellow 1	Second MRI Score
First MRI Score	1	2	3	4
1	26	2	0	0
2	3	9	0	0
3	0	1	4	0
4	0	0	1	4
Fellow 2	Second MRI Score
First MRI Score	1	2	3	4
1	27	2	0	0
2	4	8	0	0
3	0	0	6	1
4	0	0	0	2
Attending Radiologist 1	Second MRI Score
First MRI Score	1	2	3	4
1	34	2	0	0
2	3	3	0	0
3	0	0	1	1
4	0	0	2	4
Attending Radiologist 2	Second MRI Score
First MRI Score	1	2	3	4
1	33	0	1	0
2	4	5	0	0
3	0	0	2	1
4	0	0	1	3

Weighted Kappa (95%CI) fellow 1= 0.926 (0.866,0.986); fellow 2 = 0.912 (0.841,0.983); attending radiologist 1 = 0.921 (0.863,0.979); attending radiologist 2 = 0.885 (0.775,0.996);

Quantitative Analysis

The result of the quantitative assessment of the RelSI of the liver lesions is summarized in Fig. 2. There was substantial agreement between fellows, ICC = 0.705 (95% CI 0.594, 0.790) and almost perfect agreement among attending radiologists, ICC = 0.849 (95% CI 0.786, 0.895).

Fig. 2.

Bland-Altman plot for RelSI between fellow 1 (F1) and fellow 2 (F2), and between attending radiologist 1 (R1) and attending radiologist 2 (R2).

The median RelSI of hypointense liver lesions and iso- to hyperintense liver lesions for attending radiologists are summarized in Table 3. Based on the agreement among attending radiologists, a consensus optimal cut-off ratio of RelSI to differentiate between hypointense and iso- to hyperintense lesions was calculated to be 0.90.

Table 3. Quantitative Assessment of RelSI Median (Range) (n=105).

	All Patients	Hypointense (SI=1-2)	Isointense to Hyperintense (SI=3-5)	Cut-off Ratio
Fellow 1	0.59 (0.14,1.49)	0.54 (0.14,1.14)	1.05 (0.57,1.49)	0.8791
Fellow 2	0.63 (0.19,1.90)	0.57 (0.19,1.90)	1.04 (0.79,1.58)	0.9573
Attending Radiologist 1	0.57 (0.15,1.33)	0.53 (0.15,0.88)	1.07 (0.85,1.33)	0.8297
Attending Radiologist 2	0.61 (0.15,1.23)	0.54 (0.15,1.00)	0.99 (0.82,1.23)	0.9352

Optimal cut-off ratio based on pooled Attending Radiologists 1 & 2 readings is 0.90.

In the 50 patients who had 2 MRIs, there was almost perfect inter-scan agreement for the quantitative assessment of RelSI for both attending radiologists and one of the fellows, and substantial inter-scan agreement for the other fellow (Table 4 and Fig. 3).

Table 4. Inter-scan Agreement on Quantitative Assessment (RelSI) (n=50).

	First MRI median (range)	Second MRI median (range)	ICC (95%CI)
Fellow 1	0.59 (0.16,1.15)	0.59 (0.20,1.33)	0.856 (0.760,0.915)
Fellow 2	0.61 (0.19,1.90)	0.60 (0.19,1.50)	0.715 (0.548,0.827)
Attending Radiologist 1	0.57 (0.15,1.15)	0.58 (0.20,1.29)	0.878 (0.795,0.929)
Attending Radiologist 2	0.56 (0.15,1.23)	0.56 (0.19,1.31)	0.862 (0.769,0.919)

Fig. 3.

Bland-Altman plot for RelSI from the first and second MRIs, by fellow 1 (F1), fellow 2 (F2), attending radiologist 1 (R1) and attending radiologist 2 (R2).

Discussion

MRI with gadoxetate is increasingly used for the characterization of focal liver lesions and delayed HBP imaging is often helpful to characterize indeterminate liver lesions [1-8]. However, the high sensitivity and specificity of gadoxetate liver MRI may depend on the reproducible distinction between hypointense and iso- to hyperintense liver lesions on delayed HPB by individual radiologists. To date, the interobserver agreement in the assessment of relative SI of liver lesion on delayed HBP imaging has not been evaluated. A recent study on interobserver agreement during the application of different imaging criteria for HCC shows only a moderate agreement for ‘washout appearance’, which is a similar exercise in evaluating relative SI of liver lesions compared to background parenchyma. In contrast, our results show that both qualitative and quantitative evaluations of liver lesion relative SI on delayed HBP is substantial to almost perfect between fellows in training and attending radiologists, respectively. The higher agreement among attending radiologists suggest that this agreement improves with radiologist experience. In addition, among the patients who had 2 separate gadoxetate enhanced MRIs, there was almost perfect agreement on both the qualitative and quantitative assessment of the liver lesions SI on delayed HPB.

Many studies have reported the benefit of hepatobiliary specific contrast agent, such as gadoxetate, in diagnosing and differentiating FNH from HCA [9-13]. A study by Grazioli et al with 82 patients, found that 91.2% of FNH lesions were iso- or hyperintense while 93% of HCA were hypointense to the surrounding liver parenchyma on delayed HBP [10]. While the distinction between FNH and adenoma is important, a radiologist is often faced with an indeterminate hypervascular liver lesion where the differential diagnosis extends to HCC and other malignant liver lesions. Using iso- or hyperintense SI in the delayed HBP (with gadobenate dimeglumine) as indicator of lesion benignity, Morana et al found the sensitivity, specificity, accuracy, positive predictive and negative predictive value for benign lesion identification were 96.6%, 87.6%, 91.4%, 85.1% and 97.3% respectively [12]. Together, these studies highlight the importance of a radiologist's interpretation of a liver lesion's relative signal intensity on delayed HBP phase.

Few studies have suggested a quantitative cut-off ratio for a liver lesion RelSI to distinguish hypointense from iso/ hyperintense liver lesions [10]. In our study, a cut-off ratio of 0.90 RelSI was found to distinguish hypointense from iso- to hyperintense liver lesions. This finding is similar to a threshold suggested by Grazioli et al, which found an optimal cut-off value of 0.87 to differentiate FNH from adenoma based on their appearances on delayed HBP, with 91.2% of FNH being iso- or hyperintense and 93% of adenoma being hypointense [10]. Our cut-off ratio of 0.90 is slightly below that previously suggested by Mohajer et al, which characterized liver lesions as hypointense if RelSI <0.95, isointense if lesions were ≥ 0.95 and ≤ 1.05, and hyperintense if RelSI > 1.05 [11]. Additional prospective studies will be needed to further validate our findings. Reporting cut-off ratios in RelSI may also be important in future research studies evaluating a reader's ability to prospectively characterize benign from malignant liver lesions, or evaluating hepatocellular carcinoma with varying degrees of gadoxetate uptake on delayed HBP.

There are several limitations to our studies. As a retrospective study, we did not control the number of liver lesions for each diagnosis (FNH, adenoma, etc) included for analysis. We had a bias towards metastatic disease to the liver, because of our institution referral patterns. A second limitation is that many liver lesions lacked pathologic confirmation. However, as we sought to measure interobserver agreement in the assessment of relative SI on delayed HBP imaging, an accurate diagnosis of our lesion was not a goal of our study. A third limitation is that no patients with HCC were included in our study, another consequence of the referral patterns at our institution, which does not perform liver transplantations. Thus, the findings from our study cannot be extrapolated to the interpretation of HCC on delayed HBP, which are known to have variable uptake of gadoxetate [15-17]. A fourth limitation relates to the imaging parameters used at our institution. Our delayed HBP imaging is performed using a 3D T1 weighted GRE sequence, with a flip angle of 12 degrees. Institutions using different flip angles may alter the relative SI of liver lesions due to changes in T1 weighting. Therefore, an optimal cut-off ratio may need to be determined for a variety of delayed HBP imaging protocols. Finally, we did not specify what size ROI needed to be used by the interpreting radiologist, or how to place the ROI in heterogeneous lesions. However, we thought leaving the exact ROI size and placement to the discretion of the radiologist would more closely mimic actual clinical practice.

In conclusion, inter-observer agreement on qualitative and quantitative assessment of relative liver lesion signal intensity on hepatobiliary phase imaging was substantial to almost perfect among radiologists, and may improve with experience. Our study has shown that both qualitative assessments and quantitative assessments had high inter-scan agreement. A cut-off ratio of 0.90 for a lesion's SI relative to liver parenchyma may be useful to distinguish between hypointense and iso- to hyperintense liver lesions; however this number requires further validation in future studies.