Effect of clinical decision support on documented guideline adherence for head CT in emergency department patients with mild traumatic brain injury

Abstract

Imaging utilization in emergency departments (EDs) has increased significantly. More than half of the 1.2 million patients with mild traumatic brain injury (MTBI) presenting to US EDs receive head CT. While evidence-based guidelines can help emergency clinicians decide whether to obtain head CT in these patients, adoption of these guidelines has been highly variable. Promulgation of imaging efficiency guidelines by the National Quality Forum has intensified the need for performance reporting, but measuring adherence to these imaging guidelines currently requires labor-intensive and potentially inaccurate manual chart review. We implemented clinical decision support (CDS) based on published evidence to guide emergency clinicians towards appropriate head CT use in patients with MTBI and automated data capture needed for unambiguous guideline adherence metrics. Implementation of the CDS was associated with a 56% relative increase in documented adherence to evidence-based guidelines for imaging in ED patients with MTBI.

Background

The use of CT in the emergency department (ED) increased sixfold from 1995 to 2007.¹ Although the rate of growth may have slowed recently, the absolute numbers of such examinations are large.² CT imaging provides timely and efficient diagnoses for trauma and complex diseases, but inappropriate use contributes to waste and potential patient harm from unnecessary radiation exposure.^{3 4} A number of evidence-based guidelines exist for appropriate CT use,^5–7 and policymakers, including the Centers for Medicare and Medicaid Services, are developing reporting measures for imaging efficiency to assess physician adherence to such evidence, which may eventually be used for reimbursement.⁸

Patients with mild traumatic brain injury (MTBI) are those who have had traumatically induced disruption of brain function, manifested by at least one of the following: loss of consciousness, amnesia, alteration of mental state, or a focal neurologic deficit that may or may not be transient.⁹ Further, patients defined as having MTBI cannot have had loss of consciousness for >30 min, an initial Glasgow Coma Scale¹⁰ <13 (range 3–15), or post-traumatic amnesia >24 h.⁹

MTBI accounts for more than 1.2 million visits to US EDs annually, with 63% resulting in head CTs.¹¹ While up to 15% of ED patients with MTBI have an acute finding on CT, <1% require neurosurgical intervention.¹² Although evidence-based guidelines may help emergency physicians decide whether to obtain a head CT for the evaluation of MTBI,^{5 6 13 14} their adoption into practice has been highly variable. A survey of US emergency physicians found that only 30% were aware of one well-validated guideline; only 12% reported using it clinically.¹⁵ One-third of emergency physicians cited forgetting guideline details in clinical practice.¹⁶

In 2008, the American College of Emergency Physicians published a clinical policy on neuroimaging and decision-making for adult head trauma in the ED.¹⁷ Subsequently, the National Quality Forum endorsed similar consensus standards in 2012 to promote quality and efficiency in medical imaging delivery.¹⁸ In 2013, the Choosing Wisely campaign disseminated recommendations to reduce unnecessary testing, including head CT imaging for MTBI based on validated decision rules.¹⁹ Despite these attempts, it has been difficult to promote guidelines and measure adherence. Current methods to assess guideline adherence require labor-intensive manual chart review to collect granular clinical data.

Clinical decision support (CDS) integrated with computerized provider order entry (CPOE) has been shown to decrease utilization, and improve yield, of ED CT imaging.²⁰ CDS implementation has been mandated by the Health Information Technology for Economic and Clinical Health Act and federal meaningful use requirements as a tool to promote evidence-based practice.^21–23 Therefore, we aimed to determine the impact of an electronic CDS tool, based on validated evidence, designed to guide emergency clinician decision-making for use of head CT for patients with MTBI. We hypothesized that health information technology in the form of CPOE-integrated CDS can be used as a tool to measure and improve documented adherence to evidence-based guidelines.

Methods

Setting and population

The requirement to obtain informed consent was waived by the institutional review board for this Health Insurance Portability And Accountability Act (HIPAA)-compliant, prospective study, performed in the ED of a 793-bed, urban, academic level 1 trauma center. We evaluated the 27-month periods prior and subsequent to the month when a CDS tool, based on the New Orleans Criteria, the Canadian CT Head Rule, and the CT in Head Injury Patients Prediction Rule, was implemented.^5–7 Using the same inclusion and exclusion criteria as in these decision rules, we included all ED patients for whom clinicians ordered CTs for MTBI during baseline (1 August 2007 to 31 October 2009) and post-intervention (1 December 2009 to 29 February 2012) periods. As a level 1 trauma center, our ED has multi-body part trauma CT combinations, which were excluded from the MTBI CDS.

CDS intervention

All imaging orders at our institution are placed via a web-based CPOE system (Percipio; Medicalis Corp, San Francisco, California, USA). Implementation details have been described previously.²⁴ The intervention consisted of electronic CDS, launched when a head CT for MTBI was ordered. The requesting clinician was required to input data into the CDS specifically designed to address the evidence-based criteria needed to provide justification for imaging (figure 1). If the order met the criteria for at least one guideline based on the inputted data, no CDS recommendations were made and the request proceeded. For orders not meeting the criteria for all three guidelines, the CDS recommended not obtaining CT imaging and presented the clinician with references (figure 2). The clinician could ignore the advice and proceed with imaging, or cancel the order.

Figure 1.

Screenshots of electronic clinical decision support data collection screens.

Figure 2.

Electronic clinical decision support output for non-adherent studies.

The development of this CDS required the harmonization of the multiple validated prediction rules for MTBI in order to maximize sensitivity. Harmonization involved reviewing the three decision rules for overlap, and incorporating risk factors from each of the rules into CDS as guideline-adherent indications for performing head CT. In cases where similar criteria existed between the rules, we chose the most sensitive. For example, both the New Orleans Criteria⁶ and the CT in Head Injury Patients Prediction Rule⁷ identify age above 60 years as a risk factor, whereas the Canadian CT Head Rule⁵ identifies age 65 or greater as a risk factor; in this case, performing a head CT for patients above 60 years old was considered guideline adherent. Addressing head CT appropriateness by each of these rules would have created complexity in clinical input and would have been cumbersome for clinicians to remember and document had it not been incorporated into a CPOE-integrated CDS tool.

Data collection

We included a random sample of 200 head CTs on patients with MTBI from each of the baseline and intervention periods. For the baseline period, we initially identified all patients who had a trauma-related head CT based on information entered into the CPOE system. From this list of patients, we subsequently performed random chart reviews until we identified 200 patients with MTBI, based on inclusion criteria listed above. Any visits that were not unambiguously associated with MTBI presentation were excluded. The intervention period's CDS implementation allowed identification of patients with MTBI without the need for chart review. Two attending physician abstractors reviewed the electronic medical records (EMRs) of the patients associated with these orders to determine documented guideline adherence for head CT evaluation in patients with MTBI. Differences were reconciled via discussion and consensus. In the baseline period, the abstractors conducted a structured implicit review of clinician notes in the EMR. If the notes lacked sufficient documentation to determine adherence, the CT was considered non-adherent. As the post-intervention period included the granular clinical data collected from the CDS tool, documented guideline adherence was measured directly from data input into the CDS, without EMR abstraction. To evaluate the accuracy of the information entered by clinicians in the CDS, we manually reviewed a sample of clinician notes and calculated the concordance rates between data in the EMR and that entered into CDS.

Outcome measures and statistical analyses

The primary outcome measure was documented adherence to evidence-based guidelines for use of head CT for ED patients with MTBI, assessed in the random sample of head CTs. The sample size was powered to detect a 15% effect size (power=0.8, α=0.05) with an estimated baseline adherence rate of 40%. This resulted in a desired sample size of 400 records (200 in each group) for the primary outcome. The secondary outcome measure was concordance of adherence documentation between the CDS tool and the clinical note in the EMR. While the CDS tool enforced granular data collection, sufficient comparable data were often lacking in the EMR clinical note. We assumed that, if at least one hard sign to justify head CT imaging for MTBI was documented in the clinical note, then the imaging order was adherent. Analyses were performed using Microsoft Excel 2008 and JMP Pro V.10. χ² tests with proportional analyses were used to assess baseline and post-intervention differences. A two-tailed p value of <0.05 was considered to be significant.

Results

Patient population

Of 249 014 ED visits during the study period, 19 726 (7.9%) were associated with a head CT (for any indication). Before CDS implementation, CPOE contained imaging indications for trauma but did not contain granular data to unambiguously indicate MTBI, therefore we could not determine the rate of head CTs specifically for MTBI. The mean age of patients receiving head CTs was significantly lower in the baseline period (57.8 years, 95% CI 57.4 to 58.2) than the post-intervention period (60.0 years, 95% CI 59.6 to 60.4) (p<0.05), but there was no significant difference in the proportion of men and women receiving head CTs (45.4% and 46.5% of CTs were performed in men during the baseline and post-intervention periods respectively, p=0.11).

Documented guideline adherence and concordance

During the baseline period, documented guideline adherence, as assessed by manual chart review, was 49.0% (98/200). After CDS implementation, documented guideline adherence, evaluated using data entered into the CDS, increased to 76.5% (153/200). These 27.5% absolute and 56.1% relative effect sizes were significant (p<0.001).

The overall concordance for documented guideline adherence between manual chart review and electronic CDS data entry was 70% (table 1) among 50 charts reviewed. Thirty-five concordant studies (70%) were adherent by both CDS and manual review. Ten studies (20%) were non-adherent by CDS and lacked adequate documentation by manual review to determine adherence, thereby categorizing all 10 as non-adherent. An additional five studies (10%) were discordant; of these, four met adherence criteria by manual review but not CDS, while one met adherence criteria by CDS but lacked adequate documentation by manual chart review.

Table 1.

Concordance of manual clinician note review compared with electronic clinical decision support (CDS) in determining guideline adherence (n=50)

	Manual clinician note review
Electronic CDS chart review	Adherent	Not adherent	Incomplete data	Total
Adherent	35	0	1	36
Not adherent	4	0	10	14
Incomplete data	0	0	0	0
Total	39	0	11	50

Discussion

Integrating CDS into a CPOE system improved documented guideline adherence for use of head CT in evaluating ED patients with MTBI. Compared with baseline, documented guideline adherence after CDS implementation improved by an absolute 27.5%.

Documenting clinical information is critical for improving healthcare delivery, communication among caregivers, reimbursement, patient safety, and, increasingly, measuring individual clinician and health system performance against established best practices. Many public forums report guideline adherence as a quality measure for the purpose of comparison.^{19 25} Although clinical practice guidelines have been promoted to improve quality of care,^26–29 accurately measuring adherence to guideline-based metrics can be difficult and potentially lead to erroneous conclusions.^{30 31} A recent study assessing referral information for chest CT pulmonary embolus imaging requests found that documented relevant clinical information on requests was lacking and only 0.4% of requests contained adequate documentation to calculate a Wells score.³² The use of structured CDS can provide documentation of clinical attributes needed for unambiguous assessment of guideline adherence, which we found lacking in prose-form clinician notes. An alternate approach may be to create structured clinician notes to enforce granular data capture; however, such initiatives have not been broadly accepted or adopted in EMRs.^{33 34}

In the baseline period, the vast majority of studies characterized as non-adherent actually lacked the granular level of documentation necessary to allow determination of adherence. In particular, lack of documentation of specific variables did not equate to the lack of symptomology; if a clinician did not document the absence of a clinical finding, it is not possible to determine if the clinician did not evaluate the finding or whether it was truly absent. The CDS served as a prompt for the clinician to document information that may show the imaging order to be guideline-adherent. Similarly, eliciting comprehensive relevant clinical information by the CDS improves the data the radiologist receives, potentially enhancing the quality of the image interpretation.

Manual chart review is a resource-intensive process frequently fraught with incomplete documentation. Although the CDS tool automated and enforced collection of granular data, providing the clarity necessary to assess each imaging request's adherence to guidelines, it did increase the workflow burden (additional screens and mouse clicks were required to submit a head CT order) on the ordering ED clinician. In this study, we did not measure these additional time requirements or clinician satisfaction.

The CDS tool launches on the basis of clinician-entered data into the CPOE system. While this scenario allows clinicians to enter erroneous data to bypass intrusive CDS screens, previous literature indicates that clinicians largely adhere to CDS implementation without attempting to ‘game’ the system.³⁵ Accordingly, we found only 10% of CDS-entered data to be discordant with manual review.

We designed our CDS tool to incorporate the leading evidence at the time of implementation, which incorporated an amalgamation of three validated prediction rules. While our initial aim was to maximize sensitivity and prevent patient harm, we likely sacrificed specificity. Our approach led to clinician buy-in from both departments of radiology and emergency medicine and serves as a platform for iterative improvement.

Limitations

Our study was performed in a single academic setting with an established history of CPOE, CDS, and systematic quality improvement efforts, making its generalizability uncertain. However, as federal mandates for integrated EMRs increase, more commercial vendor systems are including tools such as CDS that could be similarly employed to enable documentation and measurement of national quality measures.

This study was not powered to measure CDS impact on utilization or yield of head CT for patients with MTBI. While previous studies have shown improvements in both metrics, this study's purpose was to assess documentation capabilities of a CDS tool for measuring adherence to established practice guidelines. Moreover, while documented adherence to evidence improved after CDS, we cannot assess the appropriateness of CT examinations performed. However, the documentation of care is measurable and increasingly a method used to assess physician performance.^{8 25}

Finally, we were unable to identify reasons why physicians decided to over-ride the CDS and order a non-guideline-adherent head CT. Furthermore, we were unable to assess the clinical validity of physicians’ over-ride of the CDS recommendations from chart review, as accurately recreating the context of the clinical judgment from the medical record is subjective. Subsequent iterations of CDS implemented at our facility require physicians to provide a reason for over-riding CDS recommendations, which will aid assessment of appropriateness in future studies.

Conclusion

Implementing CDS significantly increased documented adherence to published evidence for imaging in ED patients with MTBI. The CDS tool also provided an efficient, unambiguous method for retrieving the data needed to compute adherence performance metrics, which can support iterative quality improvement projects. However, even after CDS implementation, one-fourth of CTs in ED patients with MTBI remained inconsistent with evidence, suggesting that further opportunities for performance improvement remain.