Abstract
Introduction
While only 15–20% of patients with foot and ankle injuries presenting to urgent care centers have clinically significant fractures, most undergo radiography. We examined the impact of electronic point-of-care clinical decision support (CDS) on adherence to the Ottawa Ankle Rules (OAR), as well as use and yield of foot and ankle radiographs in patients with acute ankle injury.
Methods
We obtained institutional review board approval for this randomized controlled study performed April 18, 2012—December 15, 2013. All ordering providers credentialed at an urgent care affiliated with a quaternary care academic hospital were randomized to either receive or not receive CDS, based on the OAR and integrated into the physician order-entry system, with feedback at the time of imaging order. If the patient met OAR low-risk criteria, providers were advised against imaging and could either cancel the order or ignore the alert. We identified patients with foot and ankle complaints via ICD-9 billing codes and electronic health records and radiology reports reviewed for those who were eligible. Chi-square was used to compare adherence to the OAR (primary outcome), radiography utilization rate and radiography yield of foot and ankle imaging (secondary outcomes) between the intervention and control groups.
Results
Of 14,642 patients seen at urgent care during the study period, 613 (4.2%, representing 632 visits) presented with acute ankle injury and were eligible for application of the OAR; 374 (59.2%) of these were seen by control-group providers. In the intervention group, CDS adherence was higher for both ankle (239/258=92.6% vs. 231/374=61.8%, p=0.02) and foot radiography (209/258=81.0% vs. 238/374=63.6%; p<0.01). However, ankle radiography use was higher in the intervention group (166/258=64.3% vs. 183/374=48.9%; p<0.01), while foot radiography use (141/258=54.6% vs. 202/374=54.0%; p=0.95) was not. Radiography yield was also higher in the intervention group (26/307=8.5% vs. 18/385=4.7%; p=0.04).
Conclusion
Clinical decision support, previously demonstrated to improve guideline adherence for high-cost imaging, can also improve guideline adherence for radiography – as demonstrated by increased OAR adherence and increased imaging yield.
INTRODUCTION
Patients with foot and ankle pain often present to emergency departments (ED) and urgent care centers, accounting for nearly 2.8 million visits in 2010 (7.6% of all injury visits).1 Despite this frequency, clinically significant fractures are only found in 15–20% of cases.2 Validated, high quality, evidence-based guidelines for imaging patients with suspected ankle fracture (the Ottawa Ankle Rules [OAR]),3,4,2 have been available for almost 20 years. However, their widespread adoption into practice has been suboptimal. In 2001, 96% of United States (U.S.) physicians reported familiarity with the rules; however, only 31% reported using them “always” or “most of the time.”5 In the same study, Canadian physicians reported using the rules 89% of the time. However, despite this, an analysis in Ontario showed that ED foot and ankle radiography still increased 1% annually from 2001–2007.6
The federal Health Information Technology and Economic Health (HITECH) Act of 2009 aims to improve quality of healthcare and reduce waste through meaningful use of health IT, including a major focus on clinical decision support (CDS).7,8 Imaging CDS and CDS-enabled interventions have been reported to improve adherence to evidence9,10,11 and to reduce unnecessary imaging and increase its yield.12–16,17,18 Imaging CDS is most effective when based on high-quality evidence and embedded in provider workflow.19 However, most prior reports have focused on impact of CDS on “high cost” imaging (e.g., computed tomography, magnetic resonance imaging) with sparse use of a randomized controlled study design.20 Despite the emphasis on high-cost imaging, low-cost imaging examinations are the most common diagnostic imaging examinations performed in the U.S. and may be overused21,22, expose the patient to unnecessary ionizing radiation,23,24, may result in longer length of stay in the ED,25,26 and result in incidental or ambiguous findings that lead to additional high-cost imaging.27 However, it remains unclear whether CDS will have a similar impact in low-cost as in high-cost imaging.
Thus, the purpose of this study was to assess the impact of CDS on radiography for acute ankle and foot injuries. More specifically, we evaluated the impact of a CDS tool on physician-documented adherence to the OAR in the evaluation of acute ankle injury in the urgent care setting. We chose the urgent care setting, as such centers are typically designed to handle relatively low acuity injuries (e.g., acute ankle and foot injuries). We hypothesized that such CDS, integrated into provider workflow, would increase adherence to the OAR.
METHODS
Study Setting and Subjects
We obtained institutional review board approval for this Health Insurance Portability and Accountability Act-compliant randomized controlled study, performed between April 18, 2012, and December 15, 2013, at an urgent care center affiliated with a quaternary care, academic hospital. All providers (medical doctors and physician assistants) credentialed at the urgent care center, stratified by title, were randomized to either receive the CDS intervention at the time of ordering a foot or ankle radiograph (intervention group) or not (control group). Providers who began working at the urgent care center after the randomization period were excluded from the study.
Population Health Research Capsule.
What do we already know about this issue?
Clinical decision support (CDS) has been effective for improving the appropriateness of high-cost imaging, but its effect on low-cost imaging remains unclear.
What was the research question?
What was the impact of randomized CDS on adherence to the Ottawa Ankle Rules (OAR)?
What was the major finding of the study?
While adherence to the OAR increased with CDS, use of ankle radiographs was also higher in the CDS group.
How does this improve population health?
Evidence-based CDS can be successfully implemented for both low- and high-cost imaging in the ED.
Data Collection
Although providers were prospectively randomized, we collected data retrospectively. Data were captured concurrently with patient care, including in the CDS system for providers randomized to receive it. Therefore, we waited until all study data accrued and then collected it. Using International Classification of Diseases, Ninth Revision (ICD-9) codes, we queried the billing database for all unique patients presenting to the urgent care center during the study period with a discharge diagnosis code for a foot or ankle complaint (719.47, 824.x, 825.x, 826.x, 829.x, 837.0, 838.0, 845.x, 924.2x, and 928.2x; see Supplemental eTable 1). A subsequent review of each patient’s electronic health record (EHR) was performed using an explicit chart review data collection form. Data collected included patient age, chief complaint, mechanism of injury, presence of any of the exclusion criteria from the OAR (injury greater than 10 days prior to presentation, altered level of consciousness, multiple trauma, pregnancy, penetrating or open wounds, patients with isolated superficial skin complaints, and patients < 18 years old), tenderness over the malleoli or midfoot, and radiography utilization (eTable 2). We reviewed radiology reports to determine whether a fracture was noted and, if so, what type. Patients with chief complaints not pertaining to the foot or ankle were excluded. After chart review, patients with one or more OAR exclusion criterion were removed from the study cohort. We included only the first patient visit for each trauma episode; patient visits for re-assessment of the same ankle or foot injury were excluded.
Intervention
Providers were clustered into two subsets based on their titles (physician [MD] or physician assistant [PA]). In April 2012, providers in each subset were randomly assigned to the intervention or control groups using a random number generator function. Intervention group urgent care providers were exposed to CDS based on the OAR integrated into the computerized physician order entry (CPOE) system (Percipio; Medicalis, San Francisco CA). The CDS intervention consisted of four successive screens to capture data to determine the utility of the study according to the OAR (Figure 1A–D), and one educational screen (Figure 1E). If the utility of the study was low, the provider was exposed to the educational screen informing him/her of such, and s/he could either cancel or proceed with the imaging order.
Figure 1A–E.
Clinical decision support screens for the Ottawa ankle rule integrated in the computerized physician order-entry system.
ED, emergency department
To determine whether any differences observed in adherence were simply a result of “gaming” the system, which we defined as inaccurate data entry into the CPOE system to avoid potentially onerous CDS interactions, one investigator (IKI) performed manual chart reviews of 158 randomly sampled charts in the CDS group, based on power calculation with alpha of 0.05, power of 0.8, and confidence interval of 15%. We calculated the concordance and discordance rates between data in the physician note and data entered into the order entry screen (electronic orders consisting of the data entered in CPOE and CDS). Visits are concordant when the data in the visit note and the CPOE and/or CDS system matched (based on adherence to the evidence-based guideline). They are discordant when data are conflicting. If data entered in the CPOE and/or CDS system did not have corresponding entry in the physician note, we considered the visit to have incomplete documentation in the physician note rather than as discordant.
Outcome Measures
The primary outcome measure was adherence to the OAR, defined as the number of eligible patients with acute ankle and foot injuries whose workup was consistent with the OAR, whether or not they were imaged. We calculated the adherence rate for each group.
Secondary outcome measures were radiography utilization and fracture rates (radiography yield) among patients with acute ankle injuries. Ankle fracture, foot fracture, and clinically significant fracture rates were calculated. We defined clinically significant fractures as those with fracture displacement greater than 3 mm2–4. Radiography yield was calculated as the number of foot (or ankle) radiographs that detected a clinically significant fracture divided by the total number of foot (or ankle) radiographs in each group. We calculated radiography utilization rate as the number of foot (or ankle) radiographs performed divided by the total number of patient visits for acute ankle injury in each group.
Power calculation based on the rate of adherence to OAR estimated that a sample size of 334 patient visits (167 in each arm) was needed to detect a relative difference of 15% between the intervention and control groups (baseline adherence of 0.510,14, alpha = .05, power = 0.8).
Statistical Analysis
We descriptively analyzed provider demographics. Chi-square was used to compare adherence to the OAR, radiography utilization rate and radiography yield of foot and ankle imaging between the intervention and control groups. We considered a two-tailed p-value of <0.05 statistically significant. All analyses were conducted using JMP Pro 11.0 (SAS Institute, Cary, NC).
RESULTS
Provider and Patient Characteristics
A total of 66 providers were randomized to either the intervention (n=32) or control (n=34) groups; 26 of them (10 intervention, 16 control) saw eligible patients during the study period (Table 1). There were 22,982 total visits (14,642 unique patients) to the urgent care center during the 20-month study period. Of these, 988 patients were identified by the ICD-9 codes pertaining to acute ankle and foot injuries, representing 6.7% of all patients seen. We excluded 356 visits (1.5%) identified by ICD-9 codes as foot and ankle related: 177 visits were not associated with an ankle injury, 7 were multi-trauma, 26 were isolated injuries to the skin, 13 were referred with radiographs, 81 were injuries that happened more than 10 days previously, 44 were reassessment of the same injury, and 8 were associated with pregnant patients. After applying exclusion criteria, 613 patients (4.2% of all patients) representing 632 patient visits (2.7% of all patient visits) were clinically eligible for application of the OAR, of which 258 patient visits (40.8% of eligible visits) were seen by a provider in the intervention group.
Table 1.
Provider characteristics by group and total in study of efficacy of clinical decision support, based on the Ottawa Ankle Rules, to curb unnecessary foot and ankle radiography.
| CDS | Control | Overall | ||||
|---|---|---|---|---|---|---|
|
|
|
|
||||
| N | % | N | % | N | % | |
| Initial randomization | ||||||
| MD | 23 | 24 | 47 | |||
| Physician assistant | 9 | 10 | 19 | |||
| Total randomized | 32 | 34 | 66 | 100.0% | ||
| Providers who saw patients* | ||||||
| MD | 4 | 12 | 16 | |||
| Physician assistant | 6 | 4 | 10 | |||
| Total at end of study | 10 | 38.5% | 16 | 61.5% | 26 | 100.0% |
| Patient distribution | ||||||
| MD | 162 | 265 | 427 | |||
| Physician assistant | 96 | 109 | 205 | |||
| Total patients seen | 258 | 40.8% | 374 | 59.2% | 632 | 100.0% |
| Provider characteristics | ||||||
| Average years of experience | 13 | 16 | ||||
| MD | 19 | 18.5 | ||||
| PA | 10 | 9.3 | ||||
| Gender (% male) | 50% | 56% | ||||
| MD | 75% | 75% | ||||
| PA | 43% | 0% | ||||
CDS, clinical decision support.
Providers randomized to each group who actually evaluated enrolled patients during the study.
Discordance between EHR and order requisition was 0% in the control and 1.3% in the intervention group while concordance was 48% and 10.7%, respectively. In the remaining cases, EHR documentation of discrete clinical information entered in CPOE was incomplete to unambiguously assess adherence to OAR.
Adherence to Evidence-based Guidelines
Rate of adherence to the OAR was higher for ankle radiography (92.6% vs. 61.8%, p=0.015) and foot radiography (81.0% vs. 63.6%, p<0.001) in the intervention group as compared to the control group (Table 2).
Table 2.
Adherence to Ottawa Ankle Rules (OAR) and radiography use by group.
| Adherence | CDS intervention group | Control group | |||||
|---|---|---|---|---|---|---|---|
|
|
|
|
|||||
| Workups consistent with OAR | N | Adherence | Workups consistent with OAR | N | Adherence | p-value | |
| Ankle | 239 | 258 | 92.6% | 231 | 374 | 61.8% | 0.0155* |
| Foot | 209 | 258 | 81.0% | 238 | 374 | 63.6% | 0.0001* |
| Use | CDS intervention group | Control group | |||||
| No. exams performed | Patients seen | Use | No. exams performed | Patients seen | Use | p-value | |
| Ankle only | 101 | 258 | 39.1% | 124 | 374 | 33.2% | 0.1375 |
| Foot only | 76 | 258 | 29.5% | 143 | 374 | 38.2% | 0.0194 |
| Ankle and Foot | 65 | 258 | 25.2% | 59 | 374 | 15.8% | 0.0039 |
| No radiography | 17 | 258 | 6.6% | 47 | 374 | 12.6% | 0.0134 |
| Total Ankle | 166 | 258 | 64.3% | 183 | 374 | 48.9% | 0.0002* |
| Total Foot | 141 | 258 | 54.6% | 202 | 374 | 54.0% | 0.95 |
CDS, clinical decision support; OAR, Ottawa Ankle Rules.
Values are statistically significant.
Radiography Use
Ankle radiography use was higher in the intervention group (64.3% vs. 48.9%, p<0.001), but foot radiography use (54.6% vs. 54.0%, p=0.950) was not significantly different (Table 3). Ankle and foot radiography was performed in 25.2% of patient visits in the intervention group compared to 15.8% in the controls (p<0.01. Only 6.6% of patient visits in the intervention group had no radiography, compared to 12.6% in the control group (p=0.0136)
Table 3.
Radiography combinations by group.
| Clinical decision support intervention group | Control group | ||||||
|---|---|---|---|---|---|---|---|
|
|
|
||||||
| No. exams performed | Patients seen | % | No. of exams performed | Patients seen | % | p-value | |
| Ankle only | 101 | 258 | 39.1% | 124 | 374 | 33.2% | 0.1375 |
| Foot only | 76 | 258 | 29.5% | 143 | 374 | 38.2% | 0.0194 |
| Ankle and foot | 65 | 258 | 25.2% | 59 | 374 | 15.8% | 0.0039 |
| None | 17 | 258 | 6.6% | 47 | 374 | 12.6% | 0.0134 |
Fracture Prevalence
Prevalence of clinically significant fractures in the study was 8.7% (44/632), 2.1-fold higher in the intervention group (10.1% vs. 4.8%, p<0.02). Significant ankle fractures were 1.9-fold higher in the intervention group (4.7% vs. 2.4%, p=0.122) while significant foot fractures were 2.3-fold higher in the intervention group (5.4% vs. 2.4%, p<0.05).
Prevalence of all fractures in the study was 13.3% (84/632), twofold higher in the intervention group (19.0% vs. 9.4%, p<0.01). A total of 48 ankle fractures were identified in the cohort (7.59%). The prevalence rate of all ankle fractures was 1.9-fold higher in the intervention group (10.5% vs. 5.6%, p<0.02). A total of 36 foot fractures (5.70%) were identified in the cohort. The prevalence rate of all foot fractures was 2.3-fold higher in the intervention group (8.5% vs. 3.7%, p<0.01; Table 4).
Table 4.
Radiography yield and fractures per patient visit of ankle and foot radiography by group.
| CDS intervention group | Control | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|||||||||||
| Fractures | Total exams | Yield | # Patient visits | Fractures per Patient visit | Fractures | Total exams | Yield | # Patient visits | Fractures per patient visit | Yield p-values | Patient visit p-values | |
| Clinically significant fractures | 26 | 307 | 8.7% | 258 | 10.1% | 18 | 385 | 4.7% | 374 | 4.8% | 0.0421 | 0.0165 |
| Ankle | 12 | 166 | 7.2% | 258 | 4.7% | 9 | 183 | 4.9% | 374 | 2.4% | 0.364 | 0.122 |
| Foot | 14 | 141 | 9.9% | 258 | 5.4% | 9 | 202 | 4.5% | 374 | 2.4% | 0.0461 | 0.0463 |
| Avulsion fractures | 23 | 307 | 7.5% | 258 | 8.9% | 17 | 385 | 4.4% | 374 | 4.5% | 0.0849 | 0.0266 |
| Ankle | 15 | 166 | 9.0% | 258 | 5.8% | 12 | 183 | 6.6% | 374 | 3.2% | 0.389 | 0.111 |
| Foot | 8 | 141 | 5.7% | 258 | 3.1% | 5 | 202 | 2.5% | 374 | 1.3% | 0.127 | 0.125 |
| All fractures | 49 | 307 | 16.0% | 258 | 19.0% | 35 | 385 | 9.1% | 374 | 9.4% | 0.0060 | 0.0004 |
| Ankle | 27 | 166 | 16.2% | 258 | 10.5% | 21 | 183 | 11.5% | 374 | 5.6% | 0.195 | 0.0237 |
| Foot | 22 | 141 | 15.6% | 258 | 8.5% | 14 | 202 | 6.9% | 374 | 3.7% | 0.0099 | 0.0108 |
CDS, clinical decision support.
Radiography Yield
For clinically significant fractures, the radiography yield was 1.8-fold higher in the intervention group (overall 26/307=8.5% vs. 18/385=4.7%, p=0.0421). Foot radiography yield was 2.2-fold higher (14/141=9.9% vs. 9/202=4.5%, p=0.0461). Ankle radiography yield was 1.5-fold higher but did not reach significance (12/166=7.2% vs. 9/183=4.9%, p=0.354).
For all fractures, the radiography yield was 1.8-fold higher in the intervention group (overall 49/307=16.0% vs. 35/385=9.1%, p=0.0060). Foot radiography yield was 2.3-fold higher (22/141=15.6% vs. 14/202=6.9%, p=0.0099). Ankle radiography yield was 1.4-fold higher but did not reach significance (27/166=16.2% vs. 21/183=11.5%, p=0.195; Table 4).
DISCUSSION
Foot and ankle radiography represent low-cost, high-volume tests that – when used inappropriately – create waste, unnecessary radiation exposure, and likely increased lengths of stay in the ED/urgent care center.4 Similar to concerns regarding inappropriate use of high-cost imaging, if radiography imaging results are ambiguous or if incidental findings are discovered, potentially unnecessary downstream diagnostic and therapeutic procedures (with their associated costs and risks) may result.
We found that the implementation of a CDS tool at an urgent care center resulted in a significant increase in documented adherence to OAR, improving adherence to 93% for ankle and 81% for foot radiography for acute ankle injuries. Although we did not quantitatively compare the effort required to implement and sustain OAR deployment, associated data capture, and unambiguous calculation of adherence to OAR when using CDS compared to paper-based interventions, the relative ease of performing our experiment may encourage implementation of other decision rules using CDS infrastructure. Moreover, prior reports highlight the need for chart review when using paper forms to complete data extraction from the patient’s chart as nearly 23% of data capture forms were incomplete.4,28 When using CDS, providers were required to complete entry of required data elements to place orders for imaging. Quality improvement strategies using CDS may thus provide near real-time measure of provider’s adoption of evidence without the need for time-intensive retrospective chart review.
We also found that implementation of CDS reduced unnecessary foot radiography. Although the prevalence of clinically significant foot fractures was 2.3-fold higher in the CDS group, foot radiography use was similar to the control group, resulting in a higher foot radiography yield in the CDS group.
Our findings suggest overutilization of radiography even post CDS. Prior studies have reported approximately 16% prevalence of significant fractures, and 4% prevalence of avulsion fractures in an ED cohort of patients with acute ankle injuries4,28. The intervention group in our urgent care had a 10% prevalence of significant fractures (with a nearly equal proportion of avulsion injuries), reflecting the diagnosis of less severe injuries compared to the ED. However, despite these less severe injuries, patients in our CDS group were imaged more frequently than prior reports. Stiell et al. reported that 20% of patients were spared imaging, with 10% having both ankle and foot imaging after implementation of OAR. Conversely, in our CDS cohort, only 6.6% of patients were spared imaging and 25% had both ankle and foot imaging (see eTable 4 for comparisons to previously published data). In our intervention group, 241 patients were imaged to identify 12 clinically significant ankle and another 14 clinically significant foot fractures. The overuse of imaging was even more dramatic in the control group: 327 patients were imaged to diagnose nine clinically significant ankle and another nine clinically significant foot fractures.
Our results suggest that despite the existence of a well-known, validated decision rule, use of radiography for the evaluation of acute ankle trauma in the urgent care setting is suboptimal. Moreover, we found that although implementation of CDS based on OAR resulted in modest improvement in use of radiography in these patients, radiography use was not optimized. This overuse of imaging may be due to a number of factors. Patients’ preferences for imaging may have been a contributing factor when evidence-based guidelines were not followed; while patients are becoming aware of the risks of high cost high-radiation imaging, extremity radiographs carry a much less negative connotation. In addition, the OAR may have been suboptimally applied. Future studies would be needed to assess whether additional teaching, to both patients and providers, on use of OAR might reduce unnecessary utilization of radiography in patients with acute ankle injuries in urgent care centers.
The lower concordance between CDS-documented clinical attributes and the physician note found in the CDS group is expected. CDS required explicit documentation of relevant discrete clinical attributes, a capability absent in narrative documentation in the physician notes. This limitation of physician notes highlights the shortcomings of some current strategies for data collection (which rely on automated data extraction strategies from EHRs) as information may not be well documented and will thus often not be discoverable.
LIMITATIONS
There were a number of limitations to our study. We were unable to assess impact of OAR embedded in CDS on use and yield of radiography for evaluation of ankle fractures. The prevalence of significant fractures of ankle and foot differed significantly (near twofold) between control and intervention groups. Thus, our observed higher imaging yield of significant fractures, and higher use of radiography in the intervention group may simply reflect the higher prevalence of fractures in the intervention group. This in turn suggests that our randomization might not have been effective, with the intervention group consisting of providers evaluating patients with more significant fractures. An alternative explanation would be that a substantial number of significant fractures were missed in the control group, a very unlikely scenario, as 90% of the patients enrolled in the study were imaged, and we found no missed fractures re-presenting to the urgent care center. Secondly, we randomized on an intent-to-treat basis. As all clinicians were randomized based on being credentialed to practice at the urgent care, not every physician and PA may have worked there during the study period. Thirdly, our data was all obtained from a single site and thus may not be generalizable. Finally, we did not train our providers in interpreting OAR, which may have contributed to overuse of radiography.
CONCLUSION
We found that implementation of the Ottawa Ankle Rules embedded in clinical decision support significantly improved documented adherence to the OAR, and modestly improved use of foot radiography. The relative ease of implementation, data capture, and unambiguous measurement of provider adherence to OAR, without the need for time-consuming chart review, suggests CDS can efficiently deliver complex imaging-related decision rules embedded in provider workflow. Despite more than 20 years of experience with OAR,4,28, we found radiography likely remains overused in patients with acute ankle injury in urgent care centers. Future studies would be needed to assess whether additional training about the Ottawa Ankle Rules for providers and patients, or more stringent CDS-enabled interventions, can help reduce unnecessary radiography in these patients.
Supplementary Information
Footnotes
Section Editor: Judith Knoll, DO
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.
REFERENCES
- 1.National Hospital Ambulatory Medical Care Survey: 2010. Emergency Department Summary Tables. 2016. Available at: http://www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/2010_ed_web_tables.pdf.
- 2.Bachmann LM, Kolb E, Koller MT, et al. Accuracy of Ottawa ankle rules to exclude fractures of the ankle and mid-foot: systematic review. BMJ. 2003;326:417. doi: 10.1136/bmj.326.7386.417. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Stiell IG, Greenberg GH, McKnight RD, et al. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med. 1992;21(4):384–90. doi: 10.1016/s0196-0644(05)82656-3. [DOI] [PubMed] [Google Scholar]
- 4.Stiell IG, McKnight R, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271(11):827–32. [PubMed] [Google Scholar]
- 5.Graham ID, Stiell IG, Laupacis a, et al. Awareness and use of the Ottawa ankle and knee rules in 5 countries: can publication alone be enough to change practice? Ann Emerg Med. 2001;37(3):259–66. doi: 10.1067/mem.2001.113506. [DOI] [PubMed] [Google Scholar]
- 6.Dowdall H, Gee M, Brison RJ, et al. Utilization of radiographs for the diagnosis of ankle fractures in Kingston, Ontario, Canada. Acad Emerg Med. 2011;18(5):555–8. doi: 10.1111/j.1553-2712.2011.01058.x. [DOI] [PubMed] [Google Scholar]
- 7.Blumenthal D, Tavenner M. The ‘meaningful use’ regulation for electronic health records. N Engl J Med. 2010;363:501–4. doi: 10.1056/NEJMp1006114. [DOI] [PubMed] [Google Scholar]
- 8.Health Information Technology for Economic and Clinical Health (HITECH) 2009. Available at: https://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/understanding/coveredentities/hitechact.pdf.
- 9.Raja AS, Gupta A, Ip IK, et al. The use of decision support to measure documented adherence to a national imaging quality measure. Acad Radiol. 2014;21(3):378–83. doi: 10.1016/j.acra.2013.10.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Gupta A, Ip IK, Raja AS, et al. Effect of clinical decision support on documented guideline adherence for head CT in emergency department patients with mild traumatic brain injury. J Am Med Inform Assoc. 2014;21(e2):e347–51. doi: 10.1136/amiajnl-2013-002536. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ip IK, Gershanik EF, Schneider LI, et al. Impact of IT-Enabled Intervention on MRI Use for Back Pain. Am J Med. 2014;127(6):512–8. doi: 10.1016/j.amjmed.2014.01.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Vartanians VM, Sistrom CL, Weilburg JB, et al. Increasing the Appropriateness of Outpatient Imaging: Effects of a Barrier to Ordering Low-Yield Examinations. Radiology. 2010;255:842–9. doi: 10.1148/radiol.10091228. [DOI] [PubMed] [Google Scholar]
- 13.Blackmore CC, Mecklenburg RS, Kaplan GS. Effectiveness of clinical decision support in controlling Inappropriate Imaging. J Am Coll Radiol. 2011;8(1):19–25. doi: 10.1016/j.jacr.2010.07.009. [DOI] [PubMed] [Google Scholar]
- 14.Raja AS, Ip IK, Prevedello LM, et al. Effect of computerized clinical decision support on the use and yield of CT pulmonary angiography in the emergency department. Radiology. 2012;262(2):468–74. doi: 10.1148/radiol.11110951. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Wasser EJ, Prevedello LM, Sodickson A, et al. Impact of a real-time computerized duplicate alert system on the utilization of computed tomography. JAMA Intern Med. 2013;173(11):1024–6. doi: 10.1001/jamainternmed.2013.543. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ip IK, Schneider L, Seltzer S, et al. Impact of provider-led, technology-enabled radiology management program on Imaging. Am J Med. 2013;126(8):687–92. doi: 10.1016/j.amjmed.2012.11.034. [DOI] [PubMed] [Google Scholar]
- 17.Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–30. doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ip IK, Khorasani R. In: Evidence-Based Neuroimaging Diagnosis and Treatment. Medina LS, Sanelli PC, Jarvik JG, editors. Springer; New York: 2013. [Accessed Aug 1, 2016]. pp. 49–62. Available at: http://link.springer.com/referenceworkentry/10.1007/978-1-4614-3320-0_5. [Google Scholar]
- 19.Khorasani R, Hentel K, Langlotz, et al. Ten commandments for effective clinical decision support for imaging: enabling evidence-based practice to improve quality and reduce waste. AJR Am J Roentgenol. 2014;203(5):945–51. doi: 10.2214/AJR.14.13134. [DOI] [PubMed] [Google Scholar]
- 20.Raja AS, Ip IK, Dunne RM, et al. Effects of performance feedback reports on adherence to evidence-based guidelines in use of CT for evaluation of pulmonary embolism in the emergency department: a randomized trial. AJR Am J Roentgenol. 2015;205(5):936–40. doi: 10.2214/AJR.15.14677. [DOI] [PubMed] [Google Scholar]
- 21.Rao VM, Levin DC, Parker L, et al. Trends in utilization rates of the various imaging modalities in emergency departments: Nationwide Medicare Data From 2000 to 2008. J Am Coll Radiol. 2011;8(10):706–9. doi: 10.1016/j.jacr.2011.04.004. [DOI] [PubMed] [Google Scholar]
- 22.Bhargavan M, Sunshine JH. Utilization of radiology services in the United States: levels and trends in modalities, regions, and populations. Radiology. 2005;234(3):824–32. doi: 10.1148/radiol.2343031536. [DOI] [PubMed] [Google Scholar]
- 23.Amis ES, Jr, Butler PF, Applegate KE, et al. American College of Radiology White Paper on Radiation Dose in Medicine. J Am Coll Radiol. 2007;4(5):272–84. doi: 10.1016/j.jacr.2007.03.002. [DOI] [PubMed] [Google Scholar]
- 24.Lin EC. Radiation risk from medical imaging. Mayo Clin Proc. 2010;85(12):1142–6. doi: 10.4065/mcp.2010.0260. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003;5(3):155–61. doi: 10.1017/s1481803500006539. [DOI] [PubMed] [Google Scholar]
- 26.Mace SE, Stephens C, Amato I, et al. A prospective, multicenter study of factors affecting the emergency department length of stay of pediatric patients: does the diagnosis, especially psychiatric diagnosis, matter? Ann Emerg Med. 2013;62(4):S69–70. [Google Scholar]
- 27.Black WC. Advances in radiology and the real versus apparent effects of early diagnosis. Eur J Radiol. 1998;27(2):116–122. doi: 10.1016/s0720-048x(97)00160-5. [DOI] [PubMed] [Google Scholar]
- 28.Stiell I, Wells G, Laupacis, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ. 1995;311(7005):594–7. doi: 10.1136/bmj.311.7005.594. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.