Efficacy of a Checklist-Style Structured Radiology Reporting Template in Reducing Resident Misses on Cervical Spine Computed Tomography Examinations

Abstract

The increasing use of medical checklists to promote patient safety raises the question of their utility in diagnostic radiology. This study evaluates the efficacy of a checklist-style reporting template in reducing resident misses on cervical spine CT examinations. A checklist-style reporting template for cervical spine CTs was created at our institution and mandated for resident preliminary reports. Ten months after implementation of the template, we performed a retrospective cohort study comparing rates of emergent pathology missed on reports generated with and without the checklist-style reporting template. In 1,832 reports generated without using the checklist-style template, 25 (17.6 %) out of 142 emergent findings were missed. In 1,081 reports generated using the checklist-style template, 13 (11.9 %) out of 109 emergent findings were missed. The decrease in missed pathology was not statistically significant (p = 0.21). However, larger differences were noted in the detection of emergent non-fracture findings, with 17 (28.3 %) out of 60 findings missed on reports without use of the checklist template and 5 (9.3 %) out of 54 findings missed on reports using the checklist template, representing a statistically significant decrease in missed non-fracture findings (p = 0.01). The use of a checklist-style structured reporting template resulted in a statistically significant decrease in missed non-fracture findings on cervical spine CTs. The lack of statistically significant change in missed fractures was expected given that residents’ search patterns naturally include fracture detection. Our findings suggest that the use of checklists in structured reporting may increase diagnostic accuracy.

Introduction

Radiology reports have historically been generated by using free-text dictations, in which the radiologist dictates in a narrative style without a standardized order or format. The content of radiology reports generated in this manner has been shown to be error-prone and often unclear to referring physicians [1–5]. As other fields of medicine have achieved quality improvement by reducing variability within their practices, there has been an effort to reduce variability in the field of radiology by shifting from conventional free-text reporting to structured reporting [6, 7].

Structured reporting is a term that can encompass a variety of data entry and report generation techniques. Essential attributes of structured reports include a standardized lexicon, predetermined data elements, and consistent format and organization. Structured reports typically contain headings and paragraphs that distinguish the basic elements of the report, and frequently contain subheadings to delineate the anatomic areas evaluated [8]. Many benefits have been attributed to structured reporting techniques, including increased clarity of reports, decreased report turnaround times, improved quality and consistency of reports, and increased satisfaction with reports among referring physicians [9–13].

While there are a variety of structured reporting formats, many structured reports take the form of a checklist, with a list of anatomic areas or findings thought to be part of a thorough search pattern for a particular study. The increasing use of checklists in other fields of medicine and the ease with which checklists can be incorporated into structured reporting templates raise the question of their utility in improving structured reporting practices. After Pronovost’s widely publicized study in which a simple five-item central-line safety checklist resulted in startling reductions in catheter-related infections [14], a growing body of research has emerged supporting the efficacy of even the simplest of checklists in various areas of medicine [15–20]. However, few studies have evaluated the use of checklist-style structured reporting for improving diagnostic accuracy in radiology.

In this study, a checklist-style structured reporting template for cervical spine CT examinations was introduced at a tertiary care hospital with a level-one trauma center. We then evaluated the efficacy of the checklist-style template in reducing missed pathology in resident preliminary reports for cervical spine CTs.

Methods

Our study was performed with an institutional review board exemption and is compliant with the requirements of the Health Insurance Portability and Accountability Act. Informed consent was waived.

This study was conducted within a 25-resident radiology residency program based at a 1,076-bed hospital with a level-one trauma center. Workflow within the hospital’s radiology and emergency departments is frequently assisted by full preliminary reports conveying imaging interpretations rendered by radiology residents, with the majority of preliminary reports generated during on-call hours when there is limited coverage by attending radiologists. Preliminary reports are later reviewed and finalized when there is availability of an attending radiologist, with additional comments added to convey any discrepancies between the interpretations of the resident and attending radiologist. When there are discrepancies which may affect patient management, these findings are communicated to the covering physician. If the patient has been discharged from the emergency department (ED), the patient is recalled to the ED and an electronic patient care reporting system (Med-Media EMStat™, Harrisburg, Pennsylvania) is used to document these resident misses.

A checklist-style structured reporting template for cervical spine CTs was created by residents and attending neuroradiologists within this residency program in December 2010. Contents of the template were decided upon after review of discrepancies between resident preliminary reports and attending final reports for cervical spine CTs over a 2-year period, with more frequently missed pathology chosen for inclusion in the checklist (Fig. 1). Radiological Society of North America (RSNA) best-practices radiology report templates were also referenced during the creation of the template [21].

Fig. 1.

Checklist-style reporting template implemented at the study institution. Brackets indicate fill-in fields, with text within brackets representing default text

Prior to the creation of this checklist template, preliminary and final reports for cervical spine CTs were generated without a standardized structured reporting system. The structure of preliminary reports were left to the discretion of the reporting radiology resident or attending radiologist, with some reports generated using free-text dictation while other reports were generated using a variety of non-checklist templates which—as a collective—lacked the standardized content, format, language, or organization typical of structured reporting systems. Once the cervical spine checklist template was introduced to residents within the program, its use was mandated for all resident preliminary reports on cervical spine CTs ordered from the emergency department. The checklist template was added to each resident’s user profile autotext lists in the department’s speech recognition reporting system (Nuance Powerscribe 360, Burlington, Massachusetts). The mandated use of this checklist template represented the department’s first attempt at standardized structured reporting outside of breast imaging reports.

A retrospective review of the institution’s radiologic database was subsequently performed, evaluating all cervical spine CT reports generated during a 28-month period from July 2009 to October 2011. All CTs were performed on 64-slice multidetector scanners (Toshiba Aquilion 64, Tokyo, Japan or Siemens Somatom Sensation 64, Munich, Germany). Sagittal, axial, and coronal images in soft tissue and bone algorithms were obtained for all CTs. Only reports generated prior to our decision to perform this study—and thus prior to resident knowledge of this study—were included.

We identified 2,913 cervical spine CTs ordered in the emergency department with preliminary resident reports during this time period. One thousand eighty-one reports generated between December 2010 and October 2011 using the checklist-style reporting template were compared with 1,832 reports generated between July 2009 and March 2011 using free-text or non-checklist structured reporting templates. Both study periods included the first 3 months of an academic year (July to September), when the call pool is typically comprised of more inexperienced residents. The time period from December 2010 to March 2011 was included in both the pre and posttest groups due to a transition period when there was incomplete compliance with use of the template; however, only reports using the checklist-style template during this transition period were included in the posttest group and only reports using free text or other templates were included in the pretest group.

Preliminary resident findings later confirmed by a certificate of added qualifications (CAQ)-certified attending neuroradiologist and resident misses later detected by a CAQ-certified attending neuroradiologist were counted. Only findings considered relevant to emergency department management were considered, which included all acute fractures, all causes of severe spinal canal stenosis, prevertebral soft tissue swelling or masses, abnormal spinal cord or extradural densities, intracranial hemorrhage, and emergent lung findings (consolidation, pneumothorax, or pleural effusions). In studies with multiple findings, each finding was counted separately, with the exception of studies with multiple fractures, which were counted as one fracture finding. In one study with multiple fractures, there was one fracture which was identified and one fracture which was missed in the preliminary report, and for this study, both an identified and missed finding were counted. Findings which were also identified on other studies (e.g., intracranial hemorrhage detected on concurrently performed head CT or pneumothoraces detected on concurrently performed chest CT or radiograph) were not counted.

Emergency department radiology recall numbers for other neuroradiology studies during the study period were also analyzed to assess for overall changes in neuroradiology reporting accuracy by residents within the call pool. Data for resident misses requiring patient recall were obtained from the hospital’s electronic patient care reporting system, and we analyzed the frequency of such resident misses on head and lumbar spine CTs, which were being reported without the use of checklist-style reporting templates within our institution. The number and rates of missed findings on head and lumbar spine CTs were counted for the pretest period from July 2009 to December 2010 (before the institution of our cervical spine CT template) and the posttest period from January 2011 to October 2011 (after the institution of our template). Statistical comparisons were then performed to assess for statistically significant differences in resident accuracy on neuroradiology studies during the pre and posttest periods.

Differences in the rate of missed emergent findings on cervical spine CTs before and after the institution of checklist-style reports were compared by use of chi-square testing. Differences in rates of missed emergent non-fracture findings, missed emergent prevertebral soft tissue and spinal canal findings, and rates of missed findings requiring patient recalls on head and lumbar spine CTs were also compared by use of chi-square testing. Differences in rates of missed lung findings were compared by use of Fisher’s exact test due to the presence of a zero cell. Statistical software (StatCorp Stata 12.1, College Station, Texas) was used for the analysis. A p value of 0.05 or less was considered to indicate statistical significance.

Results

Following the creation of the checklist-style template in December 2010, the use of the template was mandated for all preliminary resident dictations for cervical spine CTs ordered in the emergency department. In December 2010, 45 (49.5 %) out of 91 preliminary dictations were compliant with the mandated use of the checklist-style template. From January 2011 through March 2011, 277 (93.6 %) out of 296 preliminary dictations used the checklist-style template. From April 2011 to the end of our study period, compliance with the checklist-style template was greater than 99 %.

In 1,832 reports generated without use of the checklist-style reporting template, 25 (17.6 %) out of 142 emergent findings were missed by on-call residents. In 1,081 reports generated with the checklist-style template, 13 (11.9 %) out of 109 emergent findings were missed by on-call residents. The decrease in missed emergent findings after implementation of the checklist-style template was not statistically significant (p = 0.21). Fractures accounted for the majority of emergent findings. There was an increase in frequency of missed fractures after institution of the checklist template, with 8 (9.8 %) out of 82 fractures missed on reports without use of the checklist-style templates and 8 (14.5 %) out of 55 fractures missed on reports with the checklist-style templates. The increase in missed fractures was not statistically significant (p = 0.39).

Larger differences were noted in the detection of emergent non-fracture findings, with 17 (28.3 %) out of 60 findings missed on reports without use of checklist-style templates and 5 (9.3 %) out of 54 findings missed on reports using checklist-style templates, corresponding to a statistically significant decrease in rates of missed emergent non-fracture findings (p = 0.01). For emergent lung findings, 7 (14.9 %) out of 47 findings were missed on reports without use of the checklist-style templates and 0 (0 %) out of 27 findings were missed on reports using checklist-style templates, representing a statistically significant decrease in rates of missed emergent lung findings (p = 0.04). For emergent prevertebral and spinal canal findings, 8 (44.4 %) out of 18 findings were missed on reports without use of the checklist-style templates and 4 (18.2 %) out of 22 findings were missed on reports using checklist-style templates, but the decrease did not meet criteria for statistical significance (p = 0.07). There were only two cases with emergent intracranial findings on cervical spine CTs without concurrently performed head CTs during the pretest period, so analysis of statistics for intracranial findings was not performed.

In comparisons of patient recall rates during the study periods before and after the institution of the cervical spine CT checklist-style template, 24 (0.15 %) out of 15,273 resident-generated head CT reports during the pretest time period had a missed finding requiring an emergency department patient recall, compared with 26 (0.32 %) out of 8,148 head CTs during the posttest time period. The increase in ED recall rates for head CTs during the posttest period was statistically significant (p = 0.01). For lumbar spine CTs, 3 (0.78 %) out of 384 studies performed during the pretest period required a patient recall, compared with 2 (1.0 %) out of 193 studies during the posttest period. The increase in ED recall rates for lumbar spine CTs was not statistically significant (p = 0.76). No checklist-style templates or other changes in reporting practices were instituted for head or lumbar spine CTs during the study period (Fig. 2).

Fig. 2.

Graph depicting the rate of resident misses before and after the implementation of the checklist-style structured reporting template

Discussion

The use of checklists in medicine has been widely supported, with research in various specialties demonstrating their efficacy in achieving safety goals ranging from decreased catheter-related bloodstream infections to decreased surgical morbidity and mortality [14–20]. Studies evaluating structured radiology reports—many of which have a checklist format—have attributed a wide range of benefits ranging from increased clarity to increased satisfaction among referring physicians [9–13]. However, relatively little has been published regarding the efficacy of checklists in improving the diagnostic accuracy of radiologic studies, particularly among radiologists-in-training. The one study we are aware of that has addressed this question found a decrease in accuracy and completeness scores in resident reports for a preselected set of 25 brain MR imaging cases when using a structured reporting software system when compared with free-text dictation [22]. However, the study was limited by sparse training of resident subjects on the structured reporting software system and performance of the study outside of the clinical setting, without subject blinding. Our study involved the long-standing use of a checklist-style structured reporting template in a clinical setting, with evaluation of its efficacy in decreasing resident misses on preliminary cervical spine CT reports.

The results of this study suggested that checklist-style structured reporting templates may improve the accuracy and thoroughness of resident reports. The use of checklists in standardized reporting templates for cervical spine CTs resulted in an observed reduction in the number of misses by on-call residents. Although the decrease in overall emergent missed pathology was of uncertain significance (p = 0.21), there was a statistically significant decrease in the number of missed emergent non-fracture findings (p = 0.01). A particularly large reduction was noted in emergent lung findings (p = 0.04), with the included lung apices often overlooked by residents on cervical spine studies prior to the checklist-style template. A statistically significant decrease in overall emergent missed pathology also may have been achieved with a larger sample size.

The reduction in missed emergent non-fracture findings—but not missed fractures—was expected, given that residents’ search patterns on cervical spine CTs naturally include fracture detection, particularly at a level-one trauma center. A checklist-style reporting template with one field addressing fractures was unlikely to improve the accuracy of fracture detection in this setting. The rate of missed fractures actually increased after the institution of the checklist-style template, but the increase was not statistically significant (p = 0.39).

Given the suggestion by some authors that structured reporting systems may be distracting and possibly even detrimental to the accuracy of a radiologists’ normal search habits [8, 23], we were initially concerned that the increase in missed fractures—although statistically insignificant—might suggest a negative effect of the checklist-style templates on resident search patterns for fractures. However, analysis of patient recall numbers from our electronic patient care reporting system suggested that this finding was more likely secondary to an overall decline in accuracy of neuroradiology reporting by the group of on-call residents during the posttest period (after the institution of the cervical spine CT checklist template). There was a statistically significant increase in the rate of missed findings requiring patient recalls on head CTs during the posttest period (p = 0.01). There was also an increase in the rate of missed findings requiring patient recalls on lumbar spine CTs during the posttest period, although this increase was not statistically significant (p = 0.76). Checklist-style structured reporting templates were not instituted for either head or lumbar spine CTs, and the increased rate of missed findings was thought to be secondary to differences in residents within the call pools during the pre and posttest periods. These findings also supported the validity of the statistically significant decrease in missed emergent non-fracture findings on cervical spine CTs during the posttest period (Table 1).

Table 1.

Summary of results

p values and odds ratios for resident missed findings
Variable	Odds ratio (95 % CI)	p value
All emergent findings	0.63 (0.31–1.31)	0.21
Fractures	1.57 (0.55–4.48)	0.39
Emergent non-fracture findings	0.26 (0.09–0.76)	0.01
Lung findings	0.10 (0.01–1.79)	0.04
Prevertebral/spinal canal findings	0.28 (0.07–1.16)	0.07

There were several limitations to this study. First, it was performed with a small group of residents from a single radiology residency program, with only 12–18 residents within the call pool at any given time. Second, the composition of the resident call pool naturally changed with each academic year, with new residents joining the call pool each July. With our sample of reports without the use of checklist-style templates generated between July 2009 to March 2011, and our sample of reports with checklist-style templates generated between December 2010 and October 2011, there were changes in the composition of resident subjects reading cervical spine CTs both within and between the pre and posttest periods. Consequently, it is difficult to exclude the possibility that differences in rates of missed findings on cervical spine CTs were a result of having different residents reading the studies rather than being a result of the new checklist-style reporting template. We attempted to assess for differences in the accuracy of residents generating reports during the pre and posttest time periods by analyzing patient recall rates for head and lumbar CTs. However, this method fails to account for multiple variables, including, but not limited to, differences in the rates of positive findings during the two study groups and differences in the number of missed findings on studies for admitted patients, for whom patient recall was unnecessary. Third, it is difficult to exclude the Hawthorne effect in this study [24]. Although this was a retrospective study and residents were not aware of the intent to perform this study at the time of CT reporting, residents were still aware that the residency program director was monitoring the usage of the checklist-style reporting template, which may have increased the thoroughness of resident search patterns during the posttest time period. Finally, this was a study of only a single specific checklist-style template, designed after review of missed findings within our residency program, and the results may not be generalizable to all checklist-style reporting templates.

Conclusion

Our study suggested an improvement in the accuracy of reports generated with a checklist-style reporting template. There was a reduction in resident missed findings after the institution of a checklist-style structured reporting template for cervical spine CTs, particularly with emergent non-fracture findings, an expected result given that resident search patterns naturally include fracture detection. However, there were many limitations to this study, and further research on this topic is warranted to determine the generalizability of our findings.