Second-opinion Interpretations of Neuroimaging Studies by Oncologic Neuroradiologists Can Help Reduce Errors in Cancer Care

Abstract

Background

The purpose of this study was to investigate the utility and clinical impact of second-opinion interpretations of outside neuroimaging studies by oncologic neuroradiologists at an NCI-designated cancer center.

Methods

We performed a retrospective analysis of initial outside and second-opinion radiology reports from 300 CT and MRI neuroimaging studies and identified cases with discrepancies between the two reports. An adult neuro-oncologist, pediatric neuro-oncologist, and head and neck surgeon reviewed each pair of discrepant reports based on their area of expertise, patient age, and type of study performed. The clinicians were blinded to the origin of each report and recorded whether the differences in the reports would have led to a change in patient management and/or disease staging. Histopathologic analysis, clinical assessment, and/or minimum 3-month imaging follow-up served as the reference standards to establish which of the 2 reports were correct.

Results

Among the 283 cases that met our study criteria, there were 55 neuroimaging studies with disagreements (19%) between the initial outside report and second-opinion interpretation. Patient management and/or disease stage would have been altered in 42 of 283 cases (15%) based on report differences as determined by the 2 neuro-oncologists and surgeon participating in the study. Sufficient follow-up was available in 35 of 42 cases (83%). The second-opinion interpretation was correct 100% of the time (35/35).

Conclusion

Second-opinion interpretations of neuroimaging studies by subspecialized oncologic neuroradiologists provide added value by reducing error and optimizing the care of cancer patients.

Introduction

There has been a recent emphasis on reducing health care costs in the United States by adopting a value-based health care delivery system.¹ The goal of this approach is to simultaneously reduce costs and provide patients with higher quality of care that leads to better outcomes.² Quality care is believed to be less expensive because it is more efficient and less wasteful. In an ideal system, the right care should always be provided at the right time.

Two sources of increased cost in health care are medical errors and cancer care.^3–6 Medical errors cost the United States $19.5 billion³ in 2008 and estimates of cancer care costs in the United States are projected to be $173 billion in 2020, which represents a 39% increase from 2010.⁷ By extension, medical errors that occur in the context of treating cancer patients are likely very expensive.⁸

Cancer patients frequently undergo neuroimaging examinations for initial diagnosis, staging, assessment of treatment response, and surveillance of their malignancies. The results of these examinations help shape how patients are counseled and treated. Neuroimaging interpretation errors can therefore lead to inappropriate or potentially even harmful care and can result in increased financial costs to the patient, treating institution, and health care system in general. Patients frequently present to our high volume academic cancer center with neuroimaging examinations that have been performed and interpreted elsewhere. Many of these undergo a formal reinterpretation by one of our oncologic neuroradiologists. The purpose of this study was to investigate the clinical impact of the second-opinion interpretations as assessed by treating physicians from the departments of neurology, pediatrics, and surgery (head and neck service). We hypothesized that second-opinion interpretations would be more accurate and provide added value by reducing errors in cancer patient staging and/or management.

Methods

This retrospective study was performed in accordance with the Health Insurance Portability and Accountability Act and after local Institutional Review Board approval, including waiver of informed consent.

Eligibility

A retrospective search of our institutional clinical database was performed to identify patients fulfilling the following inclusion criteria: biopsy proven cancer, CT or MR neuroimaging examinations that had been performed and initially interpreted at an outside institution, and second-opinion neuroimaging interpretations documented in an official report between November 2012 and April 2013 by oncologic neuroradiologists from our institution participating in this study. This search resulted in 300 consecutive neuroimaging studies with both initial and second-opinion interpretations. We excluded 17 of these studies because the second-opinion reader had the advantage of additional clinical information (e.g., biopsy and/or surgery results) and/or imaging exams (e.g., prior or intervening studies) not available to the initial reader at the time of interpretation. We did not exclude any cases because of potential advantages available to the initial reader because this information was not obtainable. This left 283 pairs of reports eligible for inclusion in the study.

Data Analysis and Interpretation

The reports from the outside institutions will be referred to as initial reports in this article, whereas the reports from our institution will be referred to as second-opinion reports. These 2 reports (1 outside and 1 second-opinion) will be referred to as the “report pairs”. Each of the 283 report pairs were first independently reviewed by 2 attending neuroradiologists certified by the American Board of Radiology and with a Certificate of Added Qualification in neuroradiology. The 2 neuroradiologists retrospectively reviewing the report pairs did not reinterpret the images. The contents of the report pairs officially documented in the medical record between November 2012 and April 2013 were compared in 2015. In consensus, the report pairs were then divided into 2 groups: those with disagreements and those with no disagreements. The nature of the disagreement was further classified: as either a difference in interpretation of an imaging finding made by both the initial and second-opinion readers, or as a difference in the detection of a finding made by one reader but not the other.

When reports were discrepant, three board-certified and fellowship trained attending physicians from the departments of neurology (A.M.O, 14 years experience), pediatrics (Y.K. 15 years), and surgery (head and neck service; I.G., 10 years) reviewed each pair of discrepant reports (without associated images) based on their area of expertise, age of the patient, and type of imaging study performed. Thus, for example, the pediatric neuro-oncologist was responsible for the reports pertaining to patients 18 years of age or younger and the head and neck surgeon reviewed the CT neck and MRI reports of adult or pediatric patients with head and neck cancer. The discrepant reports were presented to the clinicians as de-identified written summaries of the key imaging findings for each report along with the patient histories that were similarly available to both the radiologist who issued the initial report and the oncologic neuroradiologist who issued the second-opinion report. All three clinicians were blinded to the origin of each report throughout the study. The written summaries were created by a neuroradiologist participating in the study with 5 years of experience interpreting oncologic neuroimaging examinations. For every pair of reports, each clinician recorded whether differences between the reports were clinically important. This was defined as a report discrepancy that would have altered any 1 or more of the following: 1) disease stage of the patient; 2) recommended treatment approach; 3) how the patient was counseled. For all cases where the recommended treatment approach would have been altered, the oncologists and head and neck surgeon recorded their preferred treatment strategy and any additional tests that would be required based on each pair of reports and the provided clinical histories.

When clinically important differences between the initial and second-opinion reports were identified, the accuracy of the second-opinion report was assessed by histopathology, clinical assessment, and/or a minimum 3-month imaging follow-up as the reference standard from careful chart reviews by the lead study investigator (V.H.).

Statistical Analysis

Confidence intervals (CIs) were estimated using the Wilson score interval with continuity correction.⁹ The programming language R (version 3.1.1; R Foundation for Statistical Computing, Vienna, Austria) was used to perform all statistical computations.

Results

A total of 252 unique patients were included in the study (127 males and 125 females). The median patient age was 60 years (range: 6 months to 92 years). The 283 outside neuroimaging examinations were performed at 154 different institutions. Two hundred and thirty-seven (84%) of the 283 studies were initially interpreted by radiologists at private community hospitals or outpatient radiology facilities and 46 (16%) were initially read by radiologists at academic tertiary care facilities. The median time interval between initial reports and second-opinion reports was 17 days (range: 0–165 days).

Two hundred and twenty-four (79%) of 283 neuroimaging studies were MRIs. Spine (n=104; 37%) and brain (n=99; 35%) MRIs were the most commonly performed examinations in this study. The remaining twenty-one MRIs (7%) were of the neck (n=8), brachial plexus (n=4), orbits (n=4), sella (n=2), petrous bones (n=2), and mandible (n=1). Fifty-nine (21%) of 283 neuroimaging studies were CT scans. CT scans of the brain (n=30; 11%) and neck soft tissues (n=17; 6%) were the most commonly performed CT examinations in this study. The remaining twelve CT scans (4%) were of the paranasal sinuses (n=6), spine (n=4), temporal bones (n=1), and cervical arteries (CT neck angiogram; n=1).

Initial and Second-opinion Report Disagreements

In the preliminary review performed by two neuroradiologists, for 228/283 neuroimaging studies (81%; 95% CI: 76–85) there was no disagreement between the initial and second-opinion reports and correspondingly for 55/283 (19%; 95% CI: 15–24) studies there was disagreement between the reports. Of the 55 cases with discrepant reports, 87% were MRIs (48/55) and 13% were CT scans (7/55). There were 29/283 disagreements (10%; 95% CI: 7–14) based on the detection of abnormalities by one reader but not the other (for example, the presence of brain metastases or osseous metastases) and 26/283 disagreements (9%; 95% CI: 6–13) based on differences in the interpretation of abnormalities detected by both readers (for example, whether an area of contrast enhancement in the brain or marrow signal in the spine was benign or malignant). Representative examples are shown in Figures 1 and 2. Of the 55 cases with discrepant reports, 84% (46/55) were initially read by radiologists at private community hospitals or outpatient radiology facilities and 16% (9/55) were initially read by radiologists at academic tertiary care facilities.

Figure 1.

Axial T1-weighted MR images acquired before (a) and after (b) the administration of intravenous contrast for a 65-year-old female patient with lung cancer. The post-contrast images demonstrate abnormal enhancement along the cerebellar folia suspicious for leptomeningeal carcinomatosis (arrows). The presence of leptomeningeal disease was not mentioned in the initial report but was noted in the second-opinion report and subsequently confirmed by lumbar puncture and cerebrospinal fluid analysis.

Figure 2.

Sagittal short-TI inversion recovery (STIR) (a) and T1-weighted (b) MR images acquired for a 76-year-old male patient with lung cancer and back pain. The presence of a compression deformity involving the T8 vertebral body with linear T1 hypointensity (arrow) and band-like edema was interpreted as benign after second-opinion review given the absence of other suspicious spine lesions, background T1 hyperintense post-radiation marrow changes, no associated epidural/paravertebral soft tissue mass and no involvement of the posterior elements. The initial report concluded the T8 vertebral body demonstrated findings consistent with metastatic disease. A benign fracture was confirmed with serial follow-up imaging.

There were 48/55 cases (87%) from adults with discrepant reports (none with active head and neck cancer) and these report pairs were further reviewed by the adult neuro-oncologist participating in this study. There were 5/55 (9%) pediatric cases that were reviewed by a pediatric neuro-oncologist and 2/55 (4%) cases from adults with head and neck cancer reviewed by a head and neck surgeon. The head and neck surgeon also reviewed the discrepant reports from a pediatric patient (6 year old female) with a left neck rhabdomyosarcoma and separately reached the same conclusions about how management would be altered as the pediatric neuro-oncologist who initially reviewed the case. There was no overlap in review for the remaining cases.

Analysis of Report Pairs with Disagreements

Of the 55 cases with discrepant reports reviewed by either one of the 2 neuro-oncologists (adult and pediatric) or head and neck surgeon participating in the study, 42 of the discrepancies were considered clinically important (76%; 95% CI: 64–86). Therefore, patient management (recommended treatment approach and/or patient counseling) and/or disease stage would have changed in 42 of 283 cases (15%; 95% CI: 11–20) overall. The characteristics of the patients with clinically important differences in their initial and second-opinion reports are summarized in Table 1 and the indications for their neuroimaging studies are summarized in Table 2. When excluding the 3 cases where only patient counseling would have been altered, there were 39/283 cases (14%; 95% CI: 10–18) with changes in recommended treatment approach and/or patient stage based on report differences. More specific information regarding the proposed changes in treatment approach is summarized in Table 3.

Table 1.

Characteristics of Patients with Clinically Important Report Discrepancies

Patients	N=42 (%)
Age (years)
Median	69
Range	2 – 84
Male	22 (52%)
Female	20 (48%)
Known cancer
Lung	10
Breast	8
Prostate	8
Melanoma	2
Panreatic	1
Multiple myeloma	1
B-cell lymphoma	1
Nasopharyngeal	1
Ampullary	1
Urothelial	1
Glioblastoma Multiforme	1
Neuroblastoma	1
Diffuse Intrinsic Pontine Glioma (DIPG)	1
No established cancer diagnosis	5

Table 2.

Neuroimaging Indications for Patients with Clinically Important Report Discrepancies

Type of CT/MRI	Indication for study	N	Total	%
Spine	Back pain	10	17	40
	Staging- assess for bone metastases	5
	Urinary incontinence and back pain	1
	Follow-up bone metastases	1
Brain	Staging- assess for brain metastases	9	21	50
	Headache	4
	Follow-up brain metastases	2
	Seizure	1
	New speech difficulty	1
	Altered mental status	1
	Right arm numbness	1
	Follow-up glioblastoma	1
	Follow-up DIPG	1
Neck	Evaluate neck mass	2	4	10
	Sore throat and vocal cord paralysis	2

Table 3.

Change in Treatment Approach Based on Second-opinion Interpretations

Change in Treatment Approach (Outside report → Second-opinion interpretation)	Number of Patients
Whole brain radiation (WBRT) or hospice → Follow-up with imaging	3
Routine follow-up → Restaging work-up and change in chemotherapy	10
Routine follow-up → Stroke work-up and repeat MRI in 1 month	1
Restaging work-up → Routine follow-up	5
Lumbar puncture → Routine follow-up	1
Routine follow-up → Stereotactic radiosurgery (SRS)	1
Follow-up with imaging → Begin cancer work-up	1
WBRT → WBRT plus chemotherapy and restaging work-up	1
PET-CT and infectious disease work-up → Routine follow-up	1
PET-CT and possible radiation therapy → Routine follow-up	1
Antibiotics → PET-CT and restaging work-up	1
Echocardiogram and repeat MRI in 1 month → WBRT	1
Restaging work-up → Pain management	1
Intracranial radiation therapy → Chemotherapy or hospice	1
Routine follow-up → Lumbar puncture and possible ventricular shunt	1
PET-CT and biopsy → Change in chemotherapy	1
SRS → WBRT	1
Routine follow-up → Radiation therapy to the spine	2
Chemotherapy → Acquire brain MRI prior to therapy	1
Infection work-up → CT larynx and laryngoscopy for cancer staging	1
Total laryngectomy → Nonsurgical management	1
Neurosurgery consult for hydrocephalus → Change chemotherapy	1
Change chemotherapy → Routine follow-up	1

Abnormalities considered suspicious for malignancy by the initial reader were interpreted as benign by oncologic neuroradiologists providing second-opinion reports for 11/42 (26%; 95% CI: 15–41) cases with clinically important report differences. Second-opinion review suggested the presence of metastases (in patients with known and unknown primary tumors) or increased size/extent of primary tumors (brain and neck) not mentioned in the initial report for 26/42 (62%; 95% CI: 47–75) cases. For the remaining 5/42 (12%; 95% CI: 5–25) report pairs with clinically important differences, the second-opinion review provided a more specific diagnosis (n = 1; brain metastases instead of infection versus brain metastases), suggested disease stability instead of progression (n=2), suggested hydrocephalus instead of no hydrocephalus (n=1), and suggested a pathologic fracture of the T1 vertebral body spinous process instead of no pathologic fracture (n=1) in a patient with metastatic lung cancer and upper thoracic back pain.

Histopathology, clinical assessment, and/or a minimum 3-month imaging follow-up were available for 35/42 (83%) of cases with clinically important report differences and were utilized to determine that the second-opinion interpretation was correct 100% of the time (35/35). For the remaining 7/42 (17%) cases, the patients either succumbed to their disease within a short time frame after imaging (less than 3 months) or were treated at another institution and therefore follow-up clinical or imaging data were unavailable or insufficient to clearly determine whether the initial reports or second-opinion reports were more accurate.

Discussion

Our study demonstrates that second-opinion interpretations of outside neuroimaging studies by oncologic neuroradiologists at an NCI-designated cancer center are more accurate and provide added value by reducing errors in cancer patient staging and management. These findings add to the existing literature demonstrating that reinterpretation of imaging studies by subspecialty radiologists in different fields, such as pediatric imaging, emergency radiology, and neuroradiology, has a positive impact on patient care.^10–14 Our findings are also in agreement with studies from tertiary cancer centers concluding that reinterpretation by radiologists specialized in oncologic imaging has a clinically important effect on the accuracy of cancer patient staging, prognostication, and management.^15–19 The rates of clinically important or major discrepancies between reports in these studies range from 1.2% to 40%. A direct comparison of this data to our 15% rate of clinically important discrepancies is difficult because of inherent differences between subspecialties, variability in the types and mix of cases reviewed, and variances in how clinical importance was determined. One of the strengths of our study was the participation of physicians who make patient treatment decisions and having them determine if and how patient care would have been altered in a clinically important manner. In a larger but otherwise fairly comparable neuroimaging study conducted at an academic medical center there were 1,977 of 4,534 cases from the neoplastic category submitted for second-opinion interpretation.¹³ The rates of clinically important discrepancies for tumors of the brain, head and neck, and spine, respectively, were 6.7% (88/1322), 13.2% (55/416), and 5.0% (12/239). The overall rate of clinically important discrepancies for oncologic neuroimaging studies was 7.8% (155/1977). The lower rate of clinically important discrepancies compared to our study may be in part related to differences in sample size. Another potential reason is that our cases may have been more complex and therefore more difficult for the outside reader to interpret accurately. A third factor is the intensive expertise of our oncologic neuroradiologists derived from repeated exposure to a high volume of common and uncommon entities that cannot be replicated at most other institutions. This could have conceivably resulted in the detection of more subtle findings that would ordinarily have been missed.

A second-opinion neuroimaging study performed at a multidisciplinary cancer center found that reinterpretation of images from patients with known or presumed head and neck cancer would have altered treatment in 55 of 136 cases (40%).¹⁷ Similar to our study, the accuracy of the second-opinion report was 100% and confirmed by histopathologic, clinical, and/or radiologic follow-up. A more recent study found the recommended management changed in 38% (36/94) of patients with head and neck cancer after the second-opinion report with an accuracy of 93%.²⁰ The high rate of discrepancy in these two studies is not surprising because many radiologists have greater difficulty with head and neck cases (especially those with cancer) than brain and spine imaging. Our study had a much smaller number of submitted head and neck exams compared to those of the brain and spine. A major contributing factor is that many outside head and neck studies are reviewed and presented at our Head and Neck multidisciplinary disease management team meetings without a formal report being issued. It is highly likely that if these were formally reinterpreted and included in this study our clinically important discrepancy rate would be even higher than 15%.

We are currently considering implementation of a policy change that would require official inside interpretation of all outside neuroimaging studies. This is fairly common practice within many pathology departments, including our own, and has been adopted by some radiology departments. Even without such a policy, the numbers of outside studies that are being submitted for second-opinion reports has been steadily increasing at our institution. In the 2015 calendar year, there were 2,046 outside neuroimaging studies that were officially reinterpreted, compared to 835 in 2011 and 1,422 in 2013. Extrapolation of our data suggests that patient management and/or staging would potentially change in approximately 307 (15%) of these cases.

Applied more broadly, this information is highly relevant as cancer remains a leading cause of death²¹ and there are currently widespread concerns about the quality and costs of cancer care in the United States.^5,22,23 In an effort to address these issues, certain initiatives are already in progress. For example, United HealthCare has bound payments in cancer to quality measurement²⁴ and Anthem has implemented programs where oncologists are paid a bonus for utilizing certain treatment approaches but not others.²⁵ During this era of cost containment and quality improvement in cancer care, the value of second-opinion radiology interpretations by subspecialists must continue to be measured and demonstrated, especially to policy makers and third party payers who may question the utility of this practice. Our study strengthens the existing evidence that second-opinion interpretations by oncologic radiologists help to maximize the quality of care that cancer patients receive by enabling more appropriate counseling, more accurate staging, and more optimal treatment decisions. The reduction in errors provided by second-opinion interpretations also potentially helps to contain downstream costs that might be generated from treatment decisions based on incorrect data. We did not identify a cost analysis study of this kind in the literature for radiology. Notably, a report from the pathology department at another academic cancer center has demonstrated that second opinion interpretation of pathologic specimens from breast cancer patients helped reduce health care costs.²⁶

It is important to mention that our study is not without limitations. One limitation is that not all outside neuroimaging studies are submitted for official second-opinion interpretation at our institution, potentially resulting in a selection bias. The decision to obtain a second-opinion review is at the discretion of the physician who provides care for the patient at our institution. It is possible that some outside studies were submitted for reinterpretation because of inconsistencies between the initial report and the clinical presentation of the patient. This type of selection bias is likely minimal because the majority of discrepancies consisted of imaging abnormalities that were unlikely to be detected clinically. Nonetheless, our rate of clinically important discrepancies may have been lower if all outside examinations were routinely submitted for review. Conversely, inclusion of more neck studies would probably have raised the rate of discrepancies, as discussed previously. Ultimately, it is difficult to be certain what the discrepancy rate would be in these hypothetical scenarios. Another limitation is that we were not able to evaluate if the second-opinion reports did actually change patient management. The distinction between clinically important and unimportant discrepancies can be subjective. We tried to address this concern by recruiting experienced clinicians to evaluate all the reports with discrepancies. A third limitation is that we did not have sufficient follow up for 7 cases with clinically important discrepancies. This additional data may potentially have reduced the 100% accuracy rate of the second-opinion reports.

In summary, our results demonstrate that the reinterpretation of outside neuroimaging studies by subspecialized oncologic neuroradiologists leads to error reduction in the management of cancer patients and therefore improves the quality of their care. Larger and prospective studies are needed to validate these results and should include an assessment of the cost implications of this practice.