Does the clinical information play a role in the magnetic resonance diagnostic confidence analysis of ovarian and deep endometriosis?

Abstract

Objective:

Some recent studies have explored how the experience in the observers change their performance in the endometriosis detection using MRI but the effects of the clinical information remains uncertain. The purpose of this study was to assess the effect of the clinical information in the diagnostic confidence in the MRI diagnosis of endometriosis.

Methods and materials:

Institutional Review Board was obtained. This study is compliant to STARD method. 80 patients (mean age 32 years; range 19 – 46 years) who had undergone MRI study and surgery for suspected endometriosis were retrospectively evaluated. MRI exams were performed with a 1.5 T scanner and the following five locations were assessed: ovary, anterior compartment, vaginal fornix, utero-sacral ligaments, and Rectum\Sigmoid\Pouch of Douglas. Data sets were evaluated twice on a 5-point scale by four radiologists with different level of expertise; the first time blinded to the clinical information and the second time, after 3 months together with the clinical chart. Statistical analysis included receiver operating characteristics curve analysis, the Cohen weighted test and sensitivity, specificity, positive predictive value, negative predictive value, accuracy, LR+ and LR–.

Results:

A total of 140 localization of endometriosis (47 endometriomas and 93 endometriotic nodules) were found. The pairwise comparison demonstrated that in all cases the presence of clinical information improved the Az value. The concordance analysis indicated a mixed pattern from modest agreement (weighted κ value 0.556 for anterior compartment) to excellent agreement values (weighted κ value 0.867 for ovarian endometriomas).

Conclusion:

The results of our study suggest that clinical information is useful in diagnosing endometriosis in general anterior compartment, but not in other locations. Less experienced radiologists (resident) may benefit from it at utero-sacral ligaments or Rectum\Sigmoid\Pouch of Douglas.

Advances in knowledge:

In this era of sometimes indiscriminate use of diagnostic methods, it is important to emphasis the context for interpretation of diagnostic results. Our paper confirms that clinical information is useful in diagnosing endometriosis.

Introduction

Endometriosis is defined as the presence of endometrial-like tissue outside of the uterus.^1,2 This pathology affects 6–10% of females in childbearing age,³ up to 50% of females with infertility and up 50–60% of females and teenage girls with pelvic pain.^4,5 Usually the pelvic pain due to endometriosis is chronic and is associated with dyspareunia, dysmenorrhea and lower abdominal pain.⁶ The pain can occur unpredictably and intermittently throughout the menstrual cycle or it can be continuous.⁷

Nowadays, imaging offers the opportunity of identify and stage the presence of endometriosis and its severity non-invasively. In particular, transvaginal ultrasound (TVUS) and MRI has been established as a reference standard for the pre-operative endometriosis study.^8–11 Because of its lower cost, transvaginal ultrasonography is preferred over MRI in the diagnosis of endometriosis.³

In the MRI, recent studies have been explored suggesting how the experience in the observers change their performance in terms of sensitivity and specificity as well as the variation of the diagnostic accuracy according to the localization of the endometriosis.^12,13 Usually such kind of studies are performed with one or more readers without knowledge of clinical information so they can evaluate the sign completely blind by focusing their attention to the “finding”.

This approach is far to the routinely diagnostic practice where the radiologists have access to the clinical information and they usually assess the clinical records during the reading of an exam. Therefore, the purpose of this study is to assess the effect of the clinical information in the diagnostic confidence in the MRI diagnosis of endometriosis.

Methods and materials

This is a retrospective study, compliant to STARD initiative.¹⁴ For retrospective research, our Institutional Review Board does not require its approval or informed consent. However, written informed consent was obtained from each patient treated in the Department of Gynaecology for review of their records, files, and images for research and educational purposes and all patients read and signed an informed consent form prior to perform the MRI study.

Patients were selected from our database of in-patients who underwent surgery for suspected endometriosis from March 2009 to December 2011 and that performed MRI within 1 month before the surgery. The patients selected were 80 (mean age 32; range 19–46 years). Clinical presentation was characterized by the presence of one or more of the following symptoms: dysmenorrhea (n = 51), infertility (n = 21), lower abdominal pain (n = 12), pain with defecation (n = 8), hypermenorrhea (n = 4), and abnormal genital bleeding (n = 3).

Part of our population was included in previously published study (n = 41).¹³

MRI technique

MRI exams were conducted using a Gyroscan 1.5 T superconducting magnet (Philips, Best, Netherlands) according to a previously published technique¹³ . In summary, the MRI exam protocol for patients suspected having endometriosis was performed in the following steps: before each MR procedure, patients were asked to perform an enema and about 15 min before the examination, 20 mg of hyoscine butylbromide (Buscopan; Boehringer Ingelheim, Higashine, Japan) was administered intramuscularly in order to reduce intestinal peristalsis.

T ₂ weighted sequences (repetition time 4000 ms, echo time 90 ms, matrix 256 × 512, three signal acquired 4- to 5-mm-section thickness, field of view, 32 cm) in axial, sagittal and coronal planes are performed followed by T ₁ weighted sequence [repetition time (TR) 500 ms, echo time (TE) 14 ms] without fat suppression and fat suppressed in axial plane before and after intravenous injection of gadolinium contrast media (of 0.1 ml per Kg of body weight of Gadobutrol [Gadovist (Gd-DO3A-butrol); Bayer^AG, Berlin, Germany]) were also performed. For the exams, a phased array surface coil is used and anterior and posterior saturation bands covering subcutaneous fat are placed in order to eliminate respiratory phase ghosting artifacts.

Anatomic localization of the lesions

According to the previous work by Saba et al, ¹³ we considered these five localization of endometriosis: [1] endometriosis in the ovary or endometrioma. [2] Endometriosis of the uterosacral ligaments and the “torus uterinus”. [3] Endometriosis of the vaginal fornix. [4] Endometriosis of the anterior compartment (bladder and ureters) [5] Endometriosis of the Rectum\Sigmoid\Pouch of Douglas (including the recto-vaginal septum and Douglas pouch).

MRI analysis

MRI data analysis was performed in two different phases. In the first phase, the readers reviewed the data set for the presence of endometriosis, blinded to the clinical data; they did not have access to any information of the patients and also the patient’s age was blinded from the viewer of the workstation. The second phase was performed after 3 months by the same readers, in order to eliminate the “memory effect”, and they assessed the data set for the presence of endometriosis; in this second phase they were allowed to analyse all the clinical records of the patients (with the exclusion of the surgical results and TV-US reports).

Data sets were reviewed by four readers with different level of expertise (LS—expert, radiologist with 13 years experience in gynaecological MRI; AB—intermediate, radiologist with 9 years experience in gynaecological MRI; GL—intermediate radiologist with 8 years experience in gynaecological MRI; FS – junior radiologist with 3 years experience in gynaecological MRI). The readers were not informed about the prevalence of endometriosis in the analysed cohort as well as the prevalence of pathology in the five locations.

The readers analysed the data sets independently and they were asked to express their assessment for the presence of endometriosis in the five locations in a 5-degree scale in a similar method suggested by Park et al ¹⁵: 1 (certainly absent), 2 (probably absent), 3 (uncertain), 4 (probably present), 5 (certainly present)

Surgery and pathologic examinations

In this work, the reference standard was considered the surgery results and we considered the nodule of endometriosis present when one of these three criteria was present¹⁶: (1) resected lesion with hystopathological demonstration of endometriosis; (2) visualization of a deep pelvic lesion of endometriosis associated with only fibrosis at the biopsy; (3) presence of complete cul-de-sac obliteration secondary to endometriosis.

Statistical Analysis

Continuous data were described as the mean value ± standard deviation (SD) and compared using the unpaired Student's t-test. Receiver operating characteristics (ROC) curve analysis was also performed and the areas under the ROC (Az) were calculated. Sensitivity, specificity, positive predictive value and negative predictive value, accuracy, LR + and LR- were calculated. Finally, the agreement was calculated with the Cohen weighted test. A values <0 was considered a poor agreement, 0–0.2 slight agreement; 0.21–0.4 fair agreement; 0.41–0.6 moderate agreement; 0.61–0.8 substantial agreement and 0.81–1 almost perfect agreement. A p-value <0.05 was regarded to indicate statistical significance association and all values were calculated using a two-tailed significance level. Statistical analysis was performed with the SPSS 13.0 statistical package (SPSS Inc, Chicago, IL). Graphics were plotted with MedCalc 15.0 software (MedCalc, Mariakerke, Belgium).

Results

General results

In this study, 36 females were treated by using open laparotomy whereas 44 by using laparoscopy; a total of 140 localization of endometriosis were found. In particular, 47 endometriomas and 93 endometriotic nodules were found [23 in the utero-sacral ligament (USL), 24 in the vaginal fornix (VF), 17 in the anterior compartment (AC) and 29 in the Rectum\Sigmoid\Pouch of Douglas (RSD)]. In 10 cases other pathological conditions were found: 5 cases of hydrosalpinx, 1 of pyosalpinx and 1 cases of Brenner tumor.

ROC curve analysis

The ROC curve analysis values and the pairwise comparison between the Phase 1 and Phase 2are summarised in Table 1 and in the Figures 1–5. The pairwise comparison demonstrated that in all cases the presence of clinical information (Phase 2) improved the Az value. We also found that the best diagnostic confidence was obtained by the readers in the detection of ovarian endometriomas and in the identification of endometriosis nodules in the RSD localization.

Table 1.

Comparison between the ROC curves

		Az Phase 1	Az Phase 2	DBA	SE	95% CI	p- value
Ovary	Reader 1	0.948	0.956	0.00782	0.0054	−0.00275 to 0.0184	0.1471
	Reader 2	0.898	0.916	0.0181	0.00997	−0.00140 to 0.0377	0.0788
	Reader 3	0.885	0.910	0.0247	0.0137	−0.00210 to 0.0515	0,0809
	Reader 4	0.881	0.894	0.0876	0.00558	−0.00217 to 0.0197	0.1161
USL	Reader 1	0.886	0.903	0.0169	0.0112	−0.00477 to 0.0385	0.1265
	Reader 2	0.846	0.869	0.032	0.0144	0.00172 to 0.0582	0.0934
	Reader 3	0.795	0.836	0.0409	0.0194	0.00292 to 0.0788	0.0348*
	Reader 4	0.757	0.814	0.057	0.0222	0.00292 to 0.0788	0.0104*
VF	Reader 1	0.771	0.789	0.0179	0.028	−0.0371 to 0.0729	0.523
	Reader 2	0.736	0.758	0.0212	0.0349	−0.0473 to 0.0897	0.554
	Reader 3	0.628	0.651	0.0234	0.0297	−0.0348 to 0.0816	0.439
	Reader 4	0.641	0.685	0.0446	0.0362	−0.0263 to 0.116	0.2179
AC	Reader 1	0.891	0.943	0.0524	0.0221	0.00379 to 0.101	0.0326*
	Reader 2	0.804	0.859	0.0551	0.0265	0.00315 to 0.107	0.0377*
	Reader 3	0.821	0.886	0.0643	0.0291	0.00734 to 0.121	0.0269*
	Reader 4	0.767	0.842	0.0754	0.0325	0.0116 to 0.139	0.0206*
RSD	Reader 1	0.928	0.948	0.0202	0.0107	−0.0077 to 0.0412	0.159
	Reader 2	0.820	0.857	0.0368	0.0197	−0.00178 to 0.0754	0.087
	Reader 3	0.805	0.829	0.0235	0.012	0.00389 to 0.0470	0.092
	Reader 4	0.775	0.826	0.054	0.0211	0.0127 to 0.0953	0.0103*

AC, anterior compartment; ROC, receiver operating characteristic; RSD, Rectum\Sigmoid\Pouch of Douglas; USL, utero-sacral ligaments; VF, vaginal fornix.

Figure 1.

ROC curve analysis in the ovarian location where Phase 1 and Phase 2corresponding to for the four readers are given. ROC, receiver operating characteristic.

Figure 2.

ROC curve analysis in the utero-sacral ligament location where Phase 1 and Phase 2 corresponding to for the four readers are given. ROC, receiver operating characteristic.

Figure 3.

ROC curve analysis in the vaginal fornix location where Phase 1 and Phase 2 corresponding to for the four readers are given. ROC, receiver operating characteristic.

Figure 4.

ROC curve analysis in the anterior location where Phase 1 and Phase 2 corresponding to for the four readers are given. ROC, receiver operating characteristic.

Figure 5.

ROC curve analysis in the Rectum\Sigmoid\Pouch of Douglas location where Phase 1 and Phase 2 corresponding to for the four readers are given. ROC, receiver operating characteristic.

Cohen kappa analysis

The concordance analysis indicated a mixed pattern from modest agreement (weighted κ value 0.556 for anterior compartment) to excellent agreement values (weighted κ value 0.867 for ovarian endometriomas)

Analysis of the lesions

In Table 2 are summarized the number of the lesions of endometriosis identified by the readers in the Phase 1 and in the Phase 2. In both phases, the Reader 1 found the highest number of lesions but the effect of the clinical information changes for readers with a different level of expertise. Moreover, when we analysed the trends according to the locations we found that the clinical information increase the number of the potential affected sites in the ovaries an anterior compartment (129 vs 146 and 76 vs 85) but a reduction in the nodules suggested by the readers as pathological in the USL and RSD locations.

Table 2.

Number of the lesions identified by the readers—Class 4 and 5—between parenthesis the TP value is given

Localization	Reader 1		Reader 2		Reader 3		Reader 4		Global values (average)
	Phase 1	Phase 2	Phase 1	Phase 2	Phase 1	Phase 2	Phase 1	Phase 2	Phase 1	Phase 2
Ovary (n = 47)	34 (34)	38 (38)	31 (30)	33 (32)	32 (31)	36 (35)	32 (31)	35 (34)	129 (126)	142 (139)
Utero-sacral ligament (n = 23)	20 (16)	19 (16)	23 (16)	19 (16)	18 (12)	15 (12)	19 (12)	15 (12)	80 (60)	68 (60)
Vaginal fornix (n = 24)	20 (13)	21 (14)	17 (12)	17 (12)	23 (10)	23 (10)	23 (10)	23 (11)	83 (45)	84 (47)
Anterior compartment (n = 17)	23 (12)	25 (15)	20 (10)	20 (12)	18 (10)	21 (13)	15 (7)	19 (12)	76 (39)	85 (52)
Rectum\Sigmoid\Pouch of Douglas (n = 29)	24 (22)	23 (22)	17 (14)	15 (14)	25 (19)	22 (19)	29 (19)	22 (19)	95 (74)	82 (74)
Global values (n = 140)	121 (97)	126 (105)	108 (82)	104 (87)	116 (83)	117 (89)	118 (79)	114 (88)

TP, true positive.

We assessed also the percentage of the uncertain classification of findings for the readers, according to the different locations and the results are summarized in Figure 6.

Figure 6.

Boxplots showing the percentage of uncertain findings (class 3) in Phase 1 and Phase 2 in the different localizations. RSD, Rectum\Sigmoid\Pouch of Douglas.

Discussion

It is a well known concept that the clinical information is important in the diagnostic flow chart and more on the process of imaging diagnosis, because radiologist is not a simple “image viewer” but a physician with a dedicated training in the image interpretation. From the synthesis of the image information and clinical background of the patient a correct diagnosis can be performed, in particular in challenging cases.

However, the real effect that the clinical information determines to the radiologist’s performance is an unknown realm, not at all explored in the radiological literature. Even if in the common practice it is routine that the radiologist read the clinical chart of a patients and sometimes he/she visits him/her, the common approach in the scientific studies is to search the “finding” by blinding the readers to the clinical patient’s information in order to not introduce bias.

We therefore tested the effect of the clinical information on a group of readers with different experience in order to assess their performance and effect and how the patient’s background may change the reader’s prospective. We selected, as topic to be studied, the endometriosis where multiple locations are involved and the clinical features of the patients may suggest the involvement of a specific organ.¹⁷

In our analysis, the surgery and histopathology detected 140 locations of endometriosis (47 endometriomas and 93 nodules involving other locations) for an average value of 1.75 endometriomas/nodules of endometriosis for each patient. These data are similar to previously published paper by Saba et al.¹³ The prevalence of endometriosis for each location was not high, compared with other previously published papers.^{10,13,18–20}

In the ROC curve analysis, the pairwise comparison showed that in all cases the clinical information improved the Az value. In particular, in the anterior compartment for all the four readers there was a statistically significant difference between the Az values. In the other locations there were no statistically significant differences with the exception of the Reader 4 (junior radiologist with 3 years of experience) where the Az showed a statistically significant improvement also for the USL and RSD locations and for the Reader 3 who showed a statistically significant improvement in the Az of the USL locations. Moreover, we found that the best diagnostic confidence was obtained by the readers in the detection of ovarian endometriomas and in the identification of endometriosis nodules in the RSD location. These findings suggest that there are locations where the diagnostic accuracy is challenging like the AC and the clinical features of the patients help to obtain a correct diagnosis, whereas there are other locations where this effect is reduced. These data are concordant with previously published papers that showed that the endometriosis affecting vaginal fornix or the endometrioma had the best sensitivity whereas when the AC, USL or intestines are involved the sensitivity is reduced.^18–20

By analysing the number of the lesions endometrioma/endometriosis we found that in both phases, the Reader 1 identified the highest number of lesions and showed the best false-positive score (n = 97 and n = 105 for the Phase 1 and Phase 2 respectively). It is interesting to observe that the effect of the clinical information plays a different effect in Reader 1 (best experienced radiologist) where it was found an increase of the lesions identified (121 vs 126) with a similar trend in the false-positive cases (97 vs 105), whereas for the Reader 4 (junior radiologist) there was a reduction of the identified lesions (118 vs 114) but an increase in the false positive cases (79 vs 88). This result suggests that the use of the clinical information is different for readers with a different level of expertise.

Even more interesting is the analysis of the trends according to the locations where it was found that the clinical information increase the number of the potential affected sites in the ovaries of an AC (129 vs 146 and 76 vs 85) with an increase in the true-positive cases, in particular in the AC (39 vs 52). The clinical information determines a reduction in the nodules suggested by the readers as pathological in the USL and RSD locations (80 vs 68 and 95 vs 82 respectively) with the true-positive values that remain exactly the same (60 vs 60 and 74 vs 74). No significant differences between Phase 1 and Phase 2 in the VF locations were found. This finding suggest that the clinical information plays a different role according to the anatomical site explored

When the percentage of the uncertain classification was assessed, it was found that the level of uncertain findings corresponding to Reader 4 was high (with percentages up to the 43.2% in the Phase 1 for the anterior compartment) but there was a strong effect played by the clinical information with a reduction of the uncertain cases. Only in the RSD location there was an increase of the Class 3 (from 27.2 to 30.9%). For the other readers there were mixed patterns.

The concordance analysis indicated a mixed pattern from modest agreement (weighted κ value 0.556 for anterior compartment) to an excellent agreement values (weighted κ value 0.867 for ovarian endometriomas). These findings are concordant with the other observations because confirm that in those cases where the clinical effect plays a role there is a reduction of the concordance.

In this study, we calculated also the diagnostic performance and found that the location where the clinical information plays a lesser role is the detection of endometrioma where there are no significant differences by considering all the four readers performance. However, with the use of the clinical information, there is an increase in the diagnostic accuracy in the other locations, in particular corresponding to reader four where the accuracy in USL location goes from 60.5 to 69.1% while in the VF location from 51.2 to 63%.

We are aware that in this study there are some limitations. First, the laparotomy was performed in 36 out of 80 patients and this fact may determine that some nodules could have been missed in the laparoscopic analysis. Second, in this study the surgery was regarded as the reference standard but it is possible that for some locations, MRI works better than surgery such as in those cases where severe condition of fibrosis/adhesions determines a limitation in the access. Third, the “learning curve” can determine another bias, in particular corresponding to Reader 4, who had 3 years of experience, and therefore, the increase in the diagnostic accuracy can be due to partially increase in the expertise and not simply due to the clinical information.

In conclusion, the results of our study suggest that the clinical information changes the diagnostic confidence of the readers with different level of entity depending on the reader’s expertise and the location of the involved site. This information confirms that a specific training is necessary for those locations where the diagnosis is more challenging. Similar studies should be performed for other pathologies in order to better design the training of the radiologist.