Abstract
Background and purpose
Clinical history is known to influence interpretation of a wide range of radiologic examinations. We sought to evaluate the influence of the clinical history on MRI interpretation of optic neuropathy.
Materials and methods
107 consecutive orbital MRI scans were retrospectively reviewed by three neuroradiologists. The readers independently evaluated the coronal STIR sequence for optic nerve hyperintensity and/or atrophy (yes/no) and the coronal post-contrast T1WI for optic nerve enhancement (yes/no). Readers initially evaluated the cases blinded to the clinical history. Following a two week washout period, readers again evaluated the cases with the clinical history provided. Inter-reader and reader-clinical radiologist agreement was assessed using Cohen's simple kappa coefficient.
Results
Intra-reader agreement, without and with provision of clinical history, was 0.564–0.716 on STIR and 0.270–0.495 on post-contrast T1WI. Inter-reader agreement was overall fair-moderate. On post-contrast T1WI, inter-reader agreement was significantly higher when the clinical history was provided (p = 0.001). Reader-clinical radiologist agreement improved with provision of the clinical history to the readers on both the STIR and post-contrast T1WI sequences.
Conclusions
In the MRI assessment of optic neuropathy, only modest levels of inter-reader agreement were achieved, even after provision of clinical history. Provision of clinical history improved inter-reader agreement, especially when assessing for optic nerve enhancement. These findings confirm the subjective nature of orbital MRI interpretation in cases of optic neuropathy, and point to the importance of an accurate clinical history. Of note, the accuracy of orbital MRI in the context of optic neuropathy was not assessed, and would require further investigation.
Keywords: Health sciences, Medicine
1. Introduction
Contrast-enhanced orbital MRI is a useful diagnostic modality in the workup of patients with optic neuropathy. MRI findings may increase diagnostic confidence in clinically suspected cases of optic neuropathy, clarify the diagnosis in patients with an atypical clinical course, and exclude mimicking orbital pathologies [1]. Moreover, MRI findings can aid in assessing both disease severity and prognosis [2, 3].
The MRI findings in optic neuropathy reflect the underlying pathologic process. Specifically, inflammation, demyelination, and axonal loss can be demonstrated by various imaging characteristics [4, 5, 6]. Increased signal intensity on T2WI seen in and around lesions results from inflammation-induced edema or ischemia [7, 8]. Abnormal contrast enhancement is seen in active inflammatory or demyelinating lesions as a result of breakdown of the blood-nerve barrier [1, 7]. Segmental atrophy of the nerve in chronic lesions is the result of axonal loss [9].
Unfortunately, the diagnostic accuracy of MRI in the assessment of optic neuropathy can be hindered by various anatomical, technical, and clinical factors. The small caliber of the optic nerve poses a challenge due to limits in MRI spatial resolution. Identification of changes in nerve size requires perception of very small absolute fluctuations in caliber despite a large relative percentage change. Physiologic perineural vascular enhancement can occasionally be mistaken for abnormal nerve enhancement. Perineural orbital fat may induce chemical shift artifacts which obfuscate findings [10]. Partial volume effects between the nerve and adjacent CSF of the optic nerve sheath may also produce false positive findings on T2-weighted sequences [3].
We posit that, due to these potential limitations, MRI interpretation of optic neuropathy is subjective and potentially inconsistent between different radiologists. Moreover, we hypothesize that MRI assessment is significantly influenced by the provided clinical history. The aim in the present study was to determine interreader agreement in the MRI assessment of optic neuropathy, and also to assess the impact of the clinical history on radiologists’ assessment of optic neuropathy when interpreting orbital MRI examinations.
2. Materials and methods
Institutional review board approval was obtained for this HIPAA compliant retrospective study. We searched our institution’s PACS for the last 200 orbital MRI examinations performed regardless of indication. Exclusion criteria were: 1) patient less than 18 years of age; 2) MRI study without coronal STIR or coronal T1WI post-contrast fat saturated orbital sequences; 3) obvious orbital mass lesion that easily explained optic neuropathy. In total, 107 examinations were included.
Imaging was performed on 1.5 and 3T MRI scanners at our institution. The following parameters were most frequently utilized: Coronal STIR: TR/TE/TI = 4000/60/200msec, FOV = 180 cm2, slice thickness = 4 mm, gap = 0.4 mm, matrix = 320 × 240, NEX = 1; Post-contrast T1WI with fat saturation: TR/TE = 700/10msec, FOV = 180 cm2, slice thickness = 4 mm, gap = 0.4 mm, matrix = 380 × 269, NEX = 1. Post-contrast imaging was obtained after intravenous administration of 0.1 ml/kg of Gadabutrol (Bayer Healthcare, Whippany, NJ).
Three CAQ-certified neuroradiologists with three, four, and nine years of experience were asked to independently review only the coronal STIR and coronal post-contrast T1WI sequences (part of the standard imaging protocol at our institution) for imaging signs of optic neuropathy. The coronal STIR sequence was graded for optic nerve hyperintensity and/or atrophy (yes/no), and the coronal post-contrast T1WI was graded for optic nerve enhancement (yes/no). Rating of hyperintensity and atrophy on the STIR sequence was grouped to capture the presence or absence of features of optic neuropathy rather than the specific imaging findings. The readers initially graded the cases blinded to all clinical history. Following a two week washout period, readers again graded the cases with the original clinical history provided on the study’s requisition. Cases during the second read, with clinical history, were presented in randomized order to prevent recall bias. Each study’s dictated report was scored utilizing the same criteria, thus providing data from the original interpreting clinical radiologist with clinical history. The clinical radiologists from our institution who originally interpreted each study were all CAQ-certified.
2.1. Statistical methods
The kappa (κ) coefficient was used to assess agreement between results provided for the same laterality of the same subject between: (1) each reader using each sequence with and without knowledge of clinical history; (2) two different readers; and (3) the clinical radiologist and each reader. Kappa is interpreted as an indication of agreement less than by chance when less than zero, as slight agreement when 0 ≤ κ ≤ 0.2, as fair agreement when 0.2 < κ ≤ 0.4, as moderate agreement when 0.4 < κ ≤ 0.6, and as substantial agreement when κ > 0.6 [11].
For each sequence, logistic regression for correlated data was used to compare results derived with and without knowledge of clinical history in terms of the percentage of times each pair of readers provided concordant results when evaluating the same eye using the same sequence.
All statistical tests were conducted at the two-sided 5% significance level, using SAS 9.3 software (SAS Institute, Cary, NC).
3. Results
3.1. Patients
107 patients (38 male, 69 female) with a mean age of 49 years (SD = 20.0, range = 18–91) were included in the study. The most frequent indications referrers provided for orbital MRI evaluation included evaluation for optic neuropathy (n = 29), diplopia or nystagmus (n = 22), perceptual abnormalities (i.e. blurriness or seeing stars) (n = 19), vision loss (n = 16), eye pain (n = 13), potential orbital mass (n = 12), or headache (n = 10). Less common indications included evaluation for infection, trauma, or other cranial nerve deficit. 21 patients (19.6%) were referred with two or more complaints. Only 9 cases specifically mentioned a concern for demyelinating disease such as multiple sclerosis or neuromyelitis optica. When the clinical indication was for optic neuropathy, laterality was provided for the left optic nerve in 7 cases, the right optic nerve in 15 cases, bilaterally in 1 case, and not provided in 6 cases.
The original interpreting radiologist deemed the study positive on STIR in 21 cases (19.6%) and on T1WI in 7 cases (6.5%). 6 cases (5.6%) were deemed positive on both sequences. The clinical history provided a concerning laterality in 10 of the cases deemed positive by the original interpreting radiologist. The laterality of the positive finding matched that of the provided clinical history 9 times.
3.2. Comparing an individual reader’s results with and without knowledge of clinical history
Intra-reader agreement was assessed on each sequence before and after provision of clinical history and is presented in Table 1. On STIR imaging, agreement ranged from κ = 0.564 to 0.716. On post-contrast T1WI, agreement ranged from κ = 0.270 to 0.495.
Table 1.
Agreement between an individual reader’s results with and without knowledge of clinical history (κ).
| Reader | STIR | T1 |
|---|---|---|
| 1 | 0.564 | 0.270 |
| 2 | 0.610 | 0.483 |
| 3 | 0.716 | 0.495 |
| Overall | 0.635 | 0.409 |
3.3. Comparing readers to the clinical radiologist
Agreement between each reader and the clinical radiologist was assessed before and after provision of clinical history to the readers and is presented in Table 2. On STIR imaging, agreement between all readers and the clinical radiologist increased when the clinical history was provided to the readers. On post-contrast T1WI, agreement between two out of three readers and the clinical radiologist increased when the clinical history was provided to the readers, and slightly decreased between one of the readers and the clinical radiologist. On STIR imaging, agreement ranged from κ = 0.400 to 0.424 without history and κ = 442 to 686 with history. On post-contrast T1WI, agreement ranged from κ = 0.174 to 0.395 without history and κ = 0.151 to 0.686 with history.
Table 2.
Agreement between the clinical radiologist and each reader using each sequence with and without knowledge of clinical history (κ).
| Reader | Sequence | Without history | With history |
|---|---|---|---|
| 1 | STIR | 0.4 | 0.686 |
| 1 | T1 | 0.261 | 0.686 |
| 2 | STIR | 0.424 | 0.442 |
| 2 | T1 | 0.395 | 0.604 |
| 3 | STIR | 0.407 | 0.642 |
| 3 | T1 | 0.174 | 0.151 |
| Overall | STIR | 0.411 | 0.577 |
| Overall | T1 | 0.255 | 0.459 |
3.4. Comparing reader pairs with and without knowledge of clinical history
Agreement between readers, before and after provision of clinical history, was assessed for each sequence. The overall kappa scores between reader-pairs increased when the readers were provided the clinical history, both when evaluating the STIR sequence (κ = 0.427 without history; κ = 0.505 with history) and the post-contrast T1WI (κ = 0.271 without history; κ = 0.336 with history).
Table 3 presents the number and percentage of concordant results between reader pairs per sequence, without and with provision of the clinical history. For all reader pairs, when assessing the post-contrast T1WI, concordance increased when provided the clinical history. Overall agreement between reader pairs when assessing the post-contrast T1WI was significantly higher when the clinical history was provided than without the history (p = 0.001).
Table 3.
The percentage (proportion) of times two readers provided concordant results when assessing the same eye using the same sequence and the p value to assess whether concordance between two readers was different in the presence versus the absence of clinical history.
| STIR |
T1 |
|||||
|---|---|---|---|---|---|---|
| Readers | Without History | With History | P Value | Without History | With History | P Value |
| 1,2 | 81.3% (174/214) | 79.0% (169/214) | 0.527 | 88.8% (190/214) | 96.7% (207/214) | 0.006 |
| 1,3 | 83.2% (178/214) | 89.7% (192/214) | 0.036 | 82.7% (177/214) | 92.1% (197/214) | 0.005 |
| 2,3 | 77.6% (166/214) | 79.9% (171/214) | 0.522 | 89.3% (191/214) | 93.5% (200/214) | 0.060 |
| Overall | 80.7% (518/642) | 82.9% (532/642) | 0.397 | 86.9% (558/642) | 94.1% (604/642) | 0.001 |
Fig. 1 shows representative images of a case in which interpretation was altered after provision of clinical history.
Fig. 1.
Coronal post-contrast T1WI in a 58 year old male with double vision for two years. When evaluating the MRI without knowledge of the clinical history, two of the readers independently interpreted the left optic nerve as enhancing (arrow). After provision of the clinical history, all three readers (and the clinical radiologist) independently determined that there was no abnormal optic nerve enhancement.
4. Discussion
The results of our study indicate the subjective nature of orbital MRI interpretation with regard to imaging of optic neuropathy and the presence of at least a partial influence of clinical history on diagnostic reproducibility. The fluctuation of intra-observer and inter-observer agreement before and after provision of clinical history serves as evidence that, in this controlled experimental environment, detection of imaging findings is not accomplished in isolation. Rather, it is an integrated conclusion based upon accompanying clinical information. Imaging diagnoses of optic neuropathy seem to be clinically formulated in consideration of patient demographics, clinical presentation, and the likelihood that imaging findings are truly positive findings.
Previous investigations into the influence of clinical history date as far back as 1981, where Doubilet et al. showed its positive impact upon chest radiograph assessment [12]. Subsequently, numerous other investigators have looked into the topic. Loy and Irwig’s systematic review of 16 studies, 15 of which assessed radiologic tests, summarized those researchers’ findings of improved diagnostic sensitivity and accuracy with provision of history [13].
The field of neuroimaging has not received as much attention to this matter. McNeil et al. showed positive effects of history on overall interpretation of head CT studies [14]. Mullins et al. more specifically demonstrated the positive effects of clinical history on detection of acute infarcts in their retrospective review of concordance of the electronic health record indication and radiologic report findings [15]. However, to our knowledge, the influence of clinical history on the MRI interpretation of optic neuropathy has not been previously reported. Only one other reference in our literature search pertained to reader agreement with regard to optic nerve imaging; Rizzo et al. presented their experience with reader concordance in the assessment of 24 cases of optic nerve atrophy, demonstrating significant variability in agreement as well as low sensitivity for assessing optic nerve atrophy [16].
Our results indicate significantly increased reader concordance on the post-contrast T1WI when provided clinical history. Readers also more closely matched their assessment of the optic nerves with that of the original interpreting clinical radiologist on both imaging sequences after provision of clinical history. This finding suggests that clinical history influences readers to produce a more “real-world” interpretation of imaging findings. Our results elaborate upon the suggestion of recent guidelines set forth by an international consensus group detailing the import of communicating clinical history on MRI requisitions in cases of MS. They suggest that adequate knowledge of patient information optimizes the utility of the examination. They recommend indicating the clinical question being addressed and including relevant clinical history, current disease-modifying therapy, and date and place of any previous MR imaging [17].
Interestingly, our data generally demonstrated fair-moderate levels of concordance between the readers, even after the provision of clinical history to the readers. Our results are consistent with a prior study by Rizzo et al. who reported agreement between three neuroradiologists in the assessment of optic nerve atrophy on the STIR sequence [16]. They reported concordance levels of between 61.4–82.3%. Together, these results appear to indicate the subjectivity and diagnostic limitations inherent to MRI evaluation of the optic nerves.
While our aim was to identify the presence of an effect of clinical history on MRI interpretation, we did not seek to define the accuracy of orbital MRI in cases of optic neuropathy. In measuring agreement between readers, we measured the reliability of the technique as opposed to its validity. Determining the accuracy of MRI in assessing optic neuropathy is complicated by the lack of gold-standard pathological diagnosis in nearly all such clinical cases [18]. And while the provision of clinical history appears to improve reader reliability in the assessment of optic neuropathy on MRI, it does not necessarily improve accuracy in all cases. For example, in some cases of optic neuropathy, MRI may show enhancement before the appearance of clinical symptomatology [19]. Additionally, nerve enhancement may occasionally exhibit a protracted course, beyond the period of acute symptomatology [20]. In such cases, overreliance on the clinical history may influence radiologists towards an inaccurate interpretation of the imaging findings.
Our study is subject to several potential limitations. It has a retrospective design, although effort was made to minimize bias by analyzing consecutive cases. Despite our diligent attempt to minimize bias, results should be interpreted with caution as they have not yet been validated in a prospective fashion or using active clinical cases. Study examinations were acquired from multiple different MRI scanners of varying field strengths (between 1.5 T and 3 T). However, controlling for scanner variability with the use of a single MRI machine and technique may have limited the generalizability of our results. Limited clinical histories provided to readers from order requisitions could be further elaborated with details from the electronic medical record to, perhaps, render greater diagnostic accuracy and thus reader agreement. Finally, interpreting MRI examinations in our study is not akin to reading cases clinically [13]; the artificial environment and lack of actual patient-care pressure may have influenced readers to complete the task of reading 107 cases without attention to management implications.
5. Conclusions
In summary, we assessed reader agreement before and after the provision of clinical history to determine its influence on MRI interpretation of optic neuropathy. Provision of clinical history significantly increased inter-reader agreement with respect to optic nerve enhancement and increased agreement between the readers and the clinical radiologist. Our results emphasize the subjectivity of orbital MRI interpretation in cases of optic neuropathy. Further investigation is required to elucidate the exact effects of history on imaging interpretation of the optic nerves.
Declarations
Author contribution statement
Neil K Bansal, Mari Hagiwara, Maria J Borja, James Babb, Sohil H Patel: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper
Funding statement
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Competing interest statement
The authors declare no conflict of interest.
Additional information
No additional information is available for this paper.
References
- 1.Kolappan M., Henderson A.P., Jenkins T.M. Assessing structure and function of the afferent visual pathway in multiple sclerosis and associated optic neuritis. J. Neurol. 2009;256:305–319. doi: 10.1007/s00415-009-0123-z. [DOI] [PubMed] [Google Scholar]
- 2.Beck R.W., Trobe J.D., Moke P.S., Gal R.L., Xing D., Bhatti M.T., Brodsky M.C., Buckley E.G., Chrousos G.A., Corbett J., Eggenberger E., Goodwin J.A., Katz B., Kaufman D.I., Keltner J.L., Kupersmith M.J., Miller N.R., Nazarian S., Orengo-Nania S., Savino P.J., Shults W.T., Smith C.H., Wall M. Optic Neuritis Study Group. High- and low-risk profiles for the development of multiple sclerosis within 10 years after optic neuritis: experience of the optic neuritis treatment trial. Arch Ophthalmol. 2003;121:944–949. doi: 10.1001/archopht.121.7.944. [DOI] [PubMed] [Google Scholar]
- 3.Aiken A.H., Mukherjee P., Green A.J. MR imaging of optic neuropathy with extended echo-train acquisition fluid-attenuated inversion recovery. AJNR Am. J. Neuroradiol. 2011;32:301–305. doi: 10.3174/ajnr.A2391. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Miller D.H., Barkhof F., Frank J.A. Measurement of atrophy in multiple sclerosis: pathological basis, methodological aspects and clinical relevance. Brain. 2002;125:1676–1695. doi: 10.1093/brain/awf177. [DOI] [PubMed] [Google Scholar]
- 5.Rocca M.A., Hickman S.J., Bo L. Imaging the optic nerve in multiple sclerosis. Mult. scler. 2005;11:537–541. doi: 10.1191/1352458505ms1213oa. [DOI] [PubMed] [Google Scholar]
- 6.Boretius S., Gadjanski I., Demmer I. MRI of optic neuritis in a rat model. Neuroimage. 2008;41:323–334. doi: 10.1016/j.neuroimage.2008.02.021. [DOI] [PubMed] [Google Scholar]
- 7.Berg S., Kaschka I., Utz K.S. Baseline magnetic resonance imaging of the optic nerve provides limited predictive information on short-term recovery after acute optic neuritis. PLoS One. 2015;10:e0113961. doi: 10.1371/journal.pone.0113961. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kupersmith M.J., Alban T., Zeiffer B. Contrast-enhanced MRI in acute optic neuritis: relationship to visual performance. Brain. 2002;125:812–822. doi: 10.1093/brain/awf087. [DOI] [PubMed] [Google Scholar]
- 9.Francisco P.B. Lorena G-V: Diagnosis Approach of Optic Neuritis. J. Neurol. Neurophysiol. 2015;06:345. [Google Scholar]
- 10.Johnson G., Miller D.H., MacManus D. STIR sequences in NMR imaging of the optic nerve. Neuroradiology. 1987;29:238–245. doi: 10.1007/BF00451760. [DOI] [PubMed] [Google Scholar]
- 11.Viera A.J., Garrett J.M. Understanding interobserver agreement: the kappa statistic. Fam. med. 2005;37:360–363. [PubMed] [Google Scholar]
- 12.Doubilet P., Herman P.G. Interpretation of radiographs: effect of clinical history. Am. J. Roentgenol. 1981;137:1055–1058. doi: 10.2214/ajr.137.5.1055. [DOI] [PubMed] [Google Scholar]
- 13.Loy C., Irwig L. Accuracy of diagnostic tests read with and without clinical information: a systematic review. JAMA. 2004;292:1602–1609. doi: 10.1001/jama.292.13.1602. [DOI] [PubMed] [Google Scholar]
- 14.McNeil B.J., Hanley J.A., Funkenstein H.H. Paired receiver operating characteristic curves and the effect of history on radiographic interpretation. CT of the head as a case study. Radiology. 1983;149:75–77. doi: 10.1148/radiology.149.1.6611955. [DOI] [PubMed] [Google Scholar]
- 15.Mullins M.E., Lev M.H., Schellingerhout D. Influence of Availability of Clinical History on Detection of Early Stroke Using Unenhanced CT and Diffusion-Weighted MR Imaging. Am. J. Roentgenol. 2002;179:223–228. doi: 10.2214/ajr.179.1.1790223. [DOI] [PubMed] [Google Scholar]
- 16.Rizzo J.F., III. Neuroimaging of the Optic Nerve. NANOS Annual Meeting 2008: Diagnostic Neuroimaging. 2008:347–356. [Google Scholar]
- 17.Traboulsee A., Simon J.H., Stone L. Revised Recommendations of the Consortium of MS Centers Task Force for a Standardized MRI Protocol and Clinical Guidelines for the Diagnosis and Follow-Up of Multiple Sclerosis. AJNR Am. J. Neuroradiol. 2016;37:394–401. doi: 10.3174/ajnr.A4539. originally published online on November 12, 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Crewson P.E. Reader Agreement Studies. Am. J. Roentgenol. 2005;184:1391–1397. doi: 10.2214/ajr.184.5.01841391. [DOI] [PubMed] [Google Scholar]
- 19.Guy J., Mancuso A., Quisling R.G. Gadolinium-DTPA-enhanced magnetic resonance imaging in optic neuropathies. Ophthalmology. 1990;97:592–599. discussion 599-600. [PubMed] [Google Scholar]
- 20.Hickman S.J., Toosy A.T., Miszkiel K.A. Visual recovery following acute optic neuritis–a clinical, electrophysiological and magnetic resonance imaging study. J. Neurol. 2004;251:996–1005. doi: 10.1007/s00415-004-0477-1. [DOI] [PubMed] [Google Scholar]