Impact of an Information Technology-Enabled Initiative on the Quality of Prostate Multiparametric MRI Reports

Abstract

Rationale and Objectives

Assess the impact of implementing a structured report template and a computer-aided diagnosis (CAD) tool on the quality of prostate multiparametric MRI (mp-MRI) reports.

Materials and Methods

Institutional Review Board approval was obtained for this HIPAA-compliant study performed at an academic medical center. The study cohort included all prostate mp-MRI reports (n=385) finalized 6 months before and after implementation of a structured report template and a CAD tool (collectively the IT tools) integrated into the PACS workstation. Primary outcome measure was quality of prostate mp-MRI reports. An expert panel of our institution’s subspecialty trained abdominal radiologists defined prostate mp-MRI report quality as optimal, satisfactory or unsatisfactory based on documentation of 9 variables. Reports were reviewed to extract the predefined quality variables and determine whether the IT tools were used to create each report. Chi-square and Student’s t-tests were used to compare report quality before and after implementation of IT tools.

Results

The overall proportion of optimal or satisfactory reports increased from 29.8% (47/158) to 53.3% (121/227) (p<0.001) after implementing the IT tools. While the proportion of optimal or satisfactory reports increased among reports generated using at least one of the IT tools (47/158=[29.8%] vs. 105/161=[65.2%]; p<0.001), there was no change in quality among reports generated without use of the IT tools (47/158=[29.8%] vs. 16/66=[24.2%]; p=0.404).

Conclusion

The use of a structured template and CAD tool improved the quality of prostate mp-MRI reports compared to free-text report format and subjective measurement of contrast enhancement kinetic curve.

Introduction

Multiparametric magnetic resonance imaging (mp-MRI) is a well-established, non-invasive modality for prostate cancer diagnosis(1,2), staging(3,4), biopsy targeting(5,6) and treatment planning(7). However, variation in what results are reported and how they are presented remains a barrier to widespread adoption of mp-MRI in clinical practice(8,9).

Measuring the quality of radiology reports is a complex task(10,11), particularly due to lack of report standardization and objective key performance indicators(12). Free-text reports are typically not standardized, thus varying in content and format. Structured report templates have been developed to provide a consistent format and promote use of standard terminology(13). Compared to conventional, free-text reports, in addition to improving communication of test results, structured reports allow information to be retrieved and reused more easily(13). However, evidence of the impact of structured templates on the quality of radiology reports compared to free-text format is inconsistent(13,14).

To address the challenge of prostate mp-MRI report content and format variability, recommendations for a standardized method for interpretation and reporting were published by the European Association of Urology (EAU)(15). Also, a structured lexicon for classification – the Prostate Imaging Reporting and Data System (PI-RADS) – was proposed by the European Society of Urology Radiology (ESUR)(16), which is currently working with a scientific cooperation group to develop an international PI-RADS version(17).

The EAU and ESUR guidelines recommend standard content including interpretation of MRI contrast enhancement kinetics pattern. Reporting of prostate gland and tumor contrast enhancement kinetics, when conducted manually, is time-consuming, subjective, and associated with a high number of false-positive results(18). Computer-aided diagnosis (CAD) tools aim to provide a fast, reproducible, quantitative, and standardized computer analysis of image data, and have been used in the assessment of various diseases across a wide range of imaging modalities(19). Use of CAD tools in MRI automates standard curve pattern interpretation of contrast enhancement kinetics and has been shown to be valuable in the evaluation of breast cancer(18). However, its effectiveness in standardizing and improving prostate mp-MRI reports has yet to be proven(20,21).

In the era of accountable care, a national collaboration between the Congress and various organizations including the Centers for Medicare and Medicaid Services, the American College of Radiology and the National Quality Forum was established to develop radiology performance measures(22). However, there are no current national standard performance measures for prostate mp-MRI reporting. In the absence of national standard performance measures, a recent white paper from the American College of Radiology recommends that institutions continuously evaluate and improve their practice using their own performance measures(22). Thus the purpose of our study was to assess the impact of an information technology-enabled quality improvement intervention on prostate mp-MRI reports.

Materials and Methods

Study Design and Setting

Institutional Review Board approval was obtained for this Health Insurance Portability and Accountability Act–compliant, before and after study performed at a quaternary care, academic medical center. The requirement for informed consent was waived. All MRI examinations of the prostate were performed on 3 Tesla MRI scanners using a standard protocol - endorectal coil, axial T2, coronal T2, sagital T2, axial diffusion weighted imaging (500 & 1400 B value) axial fast spoiled gradient echo T1, pre-contrast axial fat saturation T1, axial fat saturation T1 dynamic contrast enhancement (40 phase) with contrast, axial gradient echo fat saturation T1 including abdomen and pelvis, glucagon given just prior to imaging. Prostate mp-MRI studies were interpreted initially by a trainee (resident or fellow) who generated a preliminary report. The images were subsequently reviewed by a faculty abdominal radiology subspecialist and a final report was generated. Reports were dictated and signed using a speech recognition system (PowerScribe Version 4.8, Nuance Communications, Inc., Burlington, MA).

Intervention

The quality improvement intervention consisted of training, implementation and support for two information technology (IT) tools, a structured report template and a CAD tool. A prostate mp-MRI structured report template (supplemental material, eFigure 1) was created and integrated into the institution’s speech recognition system, and a commercially-available CAD tool (VersaVue Enterprise 3.2.1, iCAD Inc., Nashua, NH) was integrated into the picture archiving and communication system (PACS) workstation. The details of the intervention, supported by the Director of the abdominal imaging and intervention division [SGS], were discussed at a divisional staff meeting before implementation. However, the use of the IT tools by the interpreting radiologist was voluntary. Prostate mp-MRI exams were uploaded to the CAD tool by the MRI technologist after the exam was completed. Faculty radiologists or trainees could access the pre-installed CAD tool software application at each PACS workstation at the time of report generation.

A one-hour instructional program was given to all abdominal radiology faculty and trainees, consisting of instructions on how to generate a report using the structured template and how to use the CAD tool for pharmacokinetics measurement, followed by a one-on-one hands-on training session. Additional support and training was provided to faculty and trainees as requested or needed.

Study Population

From the radiology administrative data repository, we identified all prostate mp-MRI reports generated 6 months before and after implementation of the intervention on February 1, 2013. We excluded 31 reports for prostate MRI procedures that were performed for reasons other than prostate cancer evaluation, such as post-prostatectomy, or if the prostate was not visible due to involvement by external neoplasm. The remaining 385 prostate mp-MRI reports finalized between August 1, 2012 and July 31, 2013 were included in the study cohort.

Outcome Measures

The primary outcome measure was the quality of prostate mp-MRI reports, categorized as optimal, satisfactory or unsatisfactory as predefined by our institution’s expert panel of subspecialty trained abdominal radiologists, based on documentation of 9 quality variables (prostate size/volume, focal lesion size, restricted diffusion, contrast enhancement, T1 intensity, T2 intensity, focal lesion location divided by prostate segmented anatomy, local invasion, and lymph node involvement) in each report (Table 1). A single physician investigator [PCS] reviewed each prostate mp-MRI report to abstract each quality variable defined by the expert panel. Optimal reports documented all quality variables (Table 1). Standard reports documented all standard quality variables. Reports lacking documentation of one or more variables required for satisfactory report quality were categorized as unsatisfactory. Our secondary outcomes were the documentation of each individual variable, the frequency of use of the IT tools, the radiologist’s adoption of the structured report template and CAD over time, and the change in quality of prostate mp-MRI reports over time.

Table 1.

Categorization of the quality of prostate mp-MRI reports

	Report quality categories
Quality variablesa	Optimal	Satisfactory	Unsatisfactory
Prostate size	Volume	3 planes or volume	<3 planes
Focal lesion size	3 planes or volume	≥ 2 planes	<2 planes
Restricted diffusion	ADC value	Documented	Not documented
Contrast enhancement	Curve or qualityb	Documented	Not documented
T1 intensity/hemorrhage	Documented	Not documented	Not documented
T2 intensity	Documented	Documented	Not documented
Focal lesion segmental locationc	Documented	Documented	Not documented
Local invasiond	Documented	Documented	Not documented
Lymph nodes involvement	Documented	Documented	Not documented

Notes; ADC= apparent diffusion coefficient

^aAs defined by an expert panel of our institution’s subspecialty trained abdominal radiologists

^bCurve or early/late enhancement or wash-in/wash-out description

^cFocal lesion location divided by prostate segmented anatomy to include: Left/right; Peripheral/Central; Apex/Mid/Base

^dExtracapsular extension; seminal vesicles; neurovascular bundle (NVB)

Data collection

Using a chart review tool, each prostate mp-MRI report was reviewed by a non-radiologist physician research fellow [PCS] to abstract and record the following variables: serum prostate-specific antigen value, biopsy-proven prostate cancer prior to the exam, Gleason histopathological grading, documentation of each one of the 9 discrete quality variables, use of the structured report template, and use of CAD documented in the report. Patient demographic data was extracted from the institution database warehouse, and radiology faculty and trainee data (i.e., gender and level of experience) was obtained through the departmental credentialing database.

Statistical Analysis and Sample Size

Chi-square was used to assess whether the proportion of optimal or satisfactory quality reports changed after the intervention. At a sample size of 188, we have 99% power at an alpha of 0.05 to detect a 10% increase in the proportion of such reports from a 58% baseline(12). To assess whether the report’s documentation of patient-related characteristics improved, Chi-square was used for categorical variables (eg, prostate-specific antigen value documented or not; focal lesion present or not) and Student’s t-test was used for continuous variables (eg, patient age [years]; prostate-specific antigen [nanogram per mililiter]). Cochran-Armitage test was used for trend analysis. All statistical analyses were performed using commercially available software (JMP Pro 10, SAS Institute, Cary, NC). A p-value of <0.05 was considered statistically significant.

Results

Patient and Report Characteristics

The final study cohort included 385 prostate mp-MRIs reports from 372 unique patients. There were no significant differences in patient-related characteristics derived from the reports between the before (n=158) and after (n=227) cohorts (Table 2). Preliminary reports were generated by 45 different trainees, 59.7% (230/385) by abdominal radiology fellows and 34.3% (132/385) by residents. During the study period, prostate MRI exams were interpreted by 10 abdominal radiology subspecialty-trained faculty, 17 abdominal radiology subspecialty fellows, and 28 diagnostic radiology residents. Only 6.0% (23/385) of reports were interpreted by faculty alone. Final reports were generated by 10 different faculty whose level of experience ranged from 0 to 24 years (mean [years] ± standard deviation [SD], 8.2 ± 7.4).

Table 2.

Patient-related characteristics based on 385 prostate multiparametric MRI reports

Characteristic	Before (n=158)	After (n=227)	p-value
Age (mean [years] ± standard deviation [SD]) a	64.2 ± 8.6	63.3 ± 9.1	0.293
Prostate-specific antigen (PSA) documented b	125/158 (79.1)	194/227 (85.5)	0.104
PSA (mean [ng/mL] ± SD) a	8.1 ± 12.2	8.2 ± 7.4	0.880
Biopsy proven cancer documented b	140/158 (88.6)	187/227 (82.4)	0.093
Gleason documented b	105/158 (66.5)	159/227 (70.0)	0.456
Gleason (mean ± SD) a	6.88 ± 0.9	6.83 ± 1.0	0.668
Focal lesion present b	121/152 (79.6)	170/224 (75.9)	0.398

Notes: Data in parenthesis are percentages.

ng/mL = nanograms/milliliters

cc= cubic centimeter

^a= p-values obtained using Student t test

^b= p-values obtained using Chi square

Structured Template and CAD Tool Utilization

After the quality improvement intervention, a total of 70.9% (161/227) reports were generated using at least one of the IT tools, among which 49.7% (80/161) used both, 42.9% (69/161) used the structured template alone, and 7.5% (12/161) used CAD alone. The use of CAD was more frequently documented among reports generated using the structured template than reports that did not use the template (80/149 [53.7%] vs. 12/78 [15.4%]; p<0.001). We also observed an increasing use of the CAD tool (p<0.001) and the structured template (p<0.001) over time (Figure 1). Similarly, the proportion of optimal or satisfactory quality prostate mp-MRI reports increased over time (p<0.001) (Figure 2).

Figure 1.

Percentage of reports generated using structured report template or computer-aided diagnosis (CAD) tool among 385 prostate multiparametric MRI reports over time

Figure 2.

Percentage of optimal or satisfactory quality reports among 385 prostate multiparametric MRI reports over time, after implementation of a structured report template and computer-aided diagnosis tool

Report Quality

The total proportion of optimal or satisfactory reports increased significantly from 29.8% (47/158) to 53.3% (121/227) (p<0.001) between the before and after intervention periods. Although the proportion of optimal or satisfactory reports increased significantly among reports generated using the structured template and/or CAD tool (47/158 [29.8%] vs. 105/161 [65.2%]; p<0.001), there was no change in the quality of reports generated after implementation that did not use at least one of the IT tools (47/158 [29.8%] vs. 16/66 [24.2%]; p=0.404) (Figure 3). While no reports generated prior to the intervention were categorized as optimal, 14.1% (32/227) of reports were categorized as optimal after the intervention, among which 93.8% (30/32) used at least one of the tools.

Figure 3.

Proportion of optimal or satisfactory quality reports among 385 prostate multiparametric MRI reports before and after implementation of a structured report template and computer-aided diagnosis (CAD) tool

Four of the nine quality variables were significantly more likely to be documented in reports following the intervention: focal lesion size (in at least 2 planes [p=0.003]; in 3 planes or volume [p<0.001]), restricted diffusion (ADC value [p<0.001]), contrast enhancement (documentation [p=0.019]; curve/quality description [p<0.001]), and local invasion (p<0.001) (Table 3). Before the intervention, reports rarely included ADC value (0.6%) or contrast enhancement curve or quality (2.5%), while more than 1/3 of reports generated after the intervention described these components (p<0.001).

Table 3.

Documentation of quality variables before and after the implementation of a structured report template and a computer-aided diagnosis tool

Quality variablesa	Before (n=158)	After (n=227)	p-value
Prostate size in 3 planes or volume	155 (99.4)	227 (100)	0.227
Prostate volume	149 (94.3)	222 (97.8)	0.072
Focal lesion presence documented	152 (96.2)	224 (98.7)	0.114
Focal lesion segmental locationb	117 (74.1)	166 (73.1)	0.840
Focal lesion sizec
≥ 2 planes	79 (65.3)	137 (80.6)	0.003
3 planes or volume	24 (19.8)	73 (42.9)	<0.001
Restricted diffusion
Documented	139 (88.0)	202 (89.0)	0.759
ADC value	1 (0.63)	100 (44.1)	<0.001
Contrast enhancement
Documented	111 (70.3)	183 (80.6)	0.019
Curve/qualityd	4 (2.5)	87 (38.3)	<0.001
T2 intensity	152 (96.2)	214 (94.3)	0.390
T1 intensity	78 (49.4)	115 (50.7)	0.803
Local invasione	131 (82.9)	213 (93.8)	<0.001
Lymph nodes involvement	156 (100)	155 (99.6)	0.404

Notes: Data in parenthesis are percentages. Values in bold are statistically significant.

^aAs defined by an expert panel of our institution’s subspecialty trained abdominal radiologists

^bFocal lesion location divided by prostate segmented anatomy to include: Left/right; Peripheral/Central; Apex/Mid/Base

^cDenominators were 121 (before) and 170 (after) since focal lesion was not always present

^dCurve or initial/late or wash-in/wash-out description

^eExtracapsular extension; seminal vesicles; neurovascular bundle (NVB)

Discussion

Although a single study showed targeted prostate biopsy accuracy improvement following implementation of a prostate mp-MRI structured template(23), to the best of our knowledge, this is the first study to use a set of report quality variables to evaluate the effect of implementing a structured report template and CAD tool on the quality of prostate mp-MRI reports. Unsatisfactory documentation of quality variables in radiology reports has been previously observed in patients with suspected stroke, and the use of informatics tools has been proposed as a mechanism for improvement(12). We found that the proportion of optimal or satisfactory prostate mp-MRI reports significantly increased after implementation and use of a structured report template and a CAD tool to generate reports.

Non-standard radiology report language has been associated with poor diagnostic certainty(24). Structured radiology reports are preferred by referring physicians(25) and promote adherence to practice guidelines(26). However, when compared to free-text reports, evidence regarding the impact of a structured format on the content of reports is inconsistent(14,27). These conflicting results may be due to differences in the formatting, content and/or implementation of the structured template. For instance, similar to our study, Schwatz et al. identified improvement in report quality when using structured templates that were derived by the institution’s staff committee(27), while Johnson et al. reported no impact when using a template that was developed by a third-party vendor(14).

Report documentation of focal lesion size, ADC value, contrast enhancement, and local invasion documentation significantly improved in our study. The last three of these components are essential for PI-RADS classification, and the documentation of all four components is recommended by the EAU(15,16). We suspect that the features responsible for the observed increase in optimal or satisfactory quality reports were enforcement of key clinical and radiological information input by the structured template, and objective measurement of contrast enhancement kinetics by CAD. It has been shown that the use of CAD can significantly improve the ability of less-experienced radiologists to distinguish benign from malignant prostate lesions(28), but the impact of CAD on report quality has not been evaluated.

We also observed that adoption of the IT tools, and their impact on report quality, was not immediate and occurred over several months. This phenomenon has been previously observed in the radiology setting(29) and should be taken into account during quality improvement initiatives using informatics tools. According to healthcare administrators, the most important success factors for the implementation of change initiatives are alignment of culture and values between individuals and departments; organization’s mission, vision and values; operational activities within the organization and definition of how processes should work; and engagement of people(30).

Although the quality of prostate mp-MRI reports improved over time, less than 30% were categorized as optimal quality after implementation, and after 6 months, more than 25% were still unsatisfactory. These results suggest that voluntary adoption of IT tools to improve report quality, even when supported by departmental leadership and embedded in radiologists’ workflow, may be inconsistent and may take months. Potential explanations include a gradual learning curve, resistance to behavior change, or lack of evidence regarding benefits of adopting the IT tool. In a recent study, rapid and sustained improvement in documentation of teaching physician attestation statement in radiology reports was demonstrated when use of IT tools were combined with timely and effective physician feedback(31). Future studies are needed to assess impact of various quality improvement strategies for accelerated and consistent adoption and use of IT tools on report quality. Such strategies may include automated monitoring of report content through the use of natural language processing (NLP) tools, feedback to radiologists using quality dashboards, education targeting quality variables awareness, financial incentives, and mandatory use of the IT tools.

Our study has a number of limitations. First, the CAD software was not always easily accessible as integrated workflow due to its perceived time-consuming launch from the PACS workstation or to technologist inconsistency in manually uploading the exam into the CAD software. However, by three months after implementation, radiologists were using it in approximately 50% of their interpretations. Second, our quality variables and report quality categorization scheme were derived by abdominal radiologist subspecialists but did not have the input of other specialties such as urologists, oncologists, or radiation therapists. We have, however, had overall positive feedback from our referring colleagues since the introduction of the template, and PI-RADS components, generated by an international multidisciplinary committee, were considered in the generation of our template. Third, the design of our quality improvement intervention limits our ability to separately evaluate the effectiveness of each individual component. Finally, our data cannot be used to assess the clinical impact of structured templates and CAD implementation, such as changes in care management or patient outcomes. These could be the subject of future studies.

Conclusions

In summary, a quality improvement intervention consisting of implementation of a structured report template and CAD tool significantly improved the quality of prostate mp-MRI reports. However, the voluntary use of the IT tools resulted in their inconsistent adoption over several months. In addition, use of IT tools did not optimize quality of all reports. Future studies are needed to assess the impact of various quality improvement strategies for accelerated and consistent adoption and use of IT tools on report quality and their impact on patient care and outcomes.