Audit filters for improving processes of care and clinical outcomes in trauma systems

Abstract

Background

Traumatic injuries represent a considerable public health burden with significant personal and societal costs. The care of the severely injured patient in a trauma system progresses along a continuum that includes numerous interventions being provided by a multidisciplinary group of healthcare personnel. Despite the recent emphasis on quality of care in medicine, there has been little research to direct trauma clinicians and administrators on how optimally to monitor and improve upon the quality of care delivered within a trauma system. Audit filters are one mechanism for improving quality of care and are defined as specific clinical processes or outcomes of care that, when they occur, represent unfavorable deviations from an established norm and which prompt review and feedback. Although audit filters are widely utilized for performance improvement in trauma systems they have not been subjected to systematic review of their effectiveness.

Objectives

To determine the effectiveness of using audit filters for improving processes of care and clinical outcomes in trauma systems.

Search methods

Our search strategy included an electronic search of the Cochrane Injuries Group Specialized Register, the Cochrane EPOC Group Specialized Register, CENTRAL (The Cochrane Library 2008, Issue 4), MEDLINE, PubMed, EMBASE, CINAHL, and ISI Web of Science: (SCI‐EXPANDED and CPCI‐S). We handsearched the Journal of Trauma, Injury, Annals of Emergency Medicine, Academic Emergency Medicine, and Injury Prevention. We searched two clinical trial registries: 1) The World Health Organization International Clinical Trials Registry Platform and, 2) Clinical Trials.gov. We also contacted content experts for further articles. The most recent electronic search was completed in December 2008 and the handsearch was completed up to February 2009.

Selection criteria

We searched for randomized controlled trials, controlled clinical trials, controlled before‐and‐after studies, and interrupted time series studies that used audit filters as an intervention for improving processes of care, morbidity, or mortality for severely injured patients.

Data collection and analysis

Two authors independently screened the search results, applied inclusion criteria, and extracted data.

Main results

There were no studies identified that met the inclusion criteria for this review.

Authors' conclusions

We were unable to identify any studies of sufficient methodological quality to draw conclusions regarding the effectiveness of audit filters as a performance improvement intervention in trauma systems. Future research using rigorous study designs should focus on the relative effectiveness of audit filters in comparison to alternative quality improvement strategies at improving processes of care, functional outcomes, and mortality for injured patients.

Plain language summary

Audit filters for improving trauma care

Major injuries are a significant cause of death and decreased quality of life worldwide. Previous research has shown that when severely injured patients are managed in an organized system of care that includes treatment by paramedics, transportation to a hospital that has specialist doctors available to treat their injuries, and additional health personnel to help them to rehabilitate, patients are more likely to survive and suffer fewer disabilities.

There are a number of ways that the quality of care provided to injured patients in a trauma system can be determined and improved. Trauma audit filters are descriptions of specific actions that should be taken, timeframes in which tests or treatments should be provided, or outcomes that are expected to occur in injured patients. Reviewing the charts of patients whose care deviates from that described by an audit filter and providing feedback to the clinicians involved in that patient's care is expected to provide a means of correcting errors and improving future performance.

Our study tried to determine how effective audit filters are at improving trauma care.

We were unable to find any studies to include in the review. More studies are needed to determine if audit filters are effective in improving care and, if they are, how effective they are in comparison to other quality improvement strategies.

Background

Patient safety and quality in the healthcare setting has become an increasingly important topic in clinical research (Auerbach 2007). Spurred by the publication of two landmark reports by the Institute of Medicine, To Err Is Human and Crossing the Quality Chasm (Kohn 1999; Hurtado 2001), much research has focused on quality improvement in health care at the system level. Despite this interest in promoting performance improvement in clinical medicine, recent reports have called into question the wisdom of implementing quality improvement programs without evidence of their benefits (Auerbach 2007). Berwick 2008 has suggested that careful consideration be given to the local context, as well as costs and potential harms of an improvement program, before it is implemented.

In most countries, traumatic injuries rank as the leading cause of death in the first four decades of life (CIHI 2007) and are a major cause of potential years of life lost (Nathens 2000). Major trauma patients experience a 20% mortality rate overall, and many survivors are left with permanent disability (Stiell 2008).

In response to this significant public health concern, co‐ordinated systems of care for severely injured patients were developed, initially in the United States (Mullins 1999), and subsequently in numerous countries including South Africa (Goosen 2003), Malaysia (Sethi 2007), New Zealand (Civil 2001), and Canada (Kortbeek 2003). The Trauma Association of Canada (TAC 2007) has defined a trauma system as a preplanned, organized, and co‐ordinated injury control effort in a defined geographic area, which:

• is publicly funded and administered;

• provides comprehensive injury surveillance and prevention programs;

• delivers the full spectrum of trauma care, including rapid access to emergency medical services, transport to an appropriate level of care, acute medical care, and rehabilitation;

• engages in research, training and performance improvement;

• is involved in disaster preparedness.

In the past decade, several studies have emerged that have demonstrated the mortality benefits of trauma systems. In the United States of America, states with trauma systems have a 9% lower crude mortality rate, and a 17% lower motor vehicle accident mortality rate than states without organized systems of trauma care (Nathens 2000). Similarly, in Australia, the introduction of a trauma system for the state of Victoria led to a mortality reduction of 38% over a five‐year period (Adjusted Odds Ratio = 0.62; 95% CI 0.48 to 0.80) (Cameron 2008).

Despite these benefits, recent studies suggest that there is variation in outcomes in the trauma center component of the system. For instance, Shafi 2008 found that in a sample of 58 high‐volume U.S. trauma centers, nearly one out of five hospitals performed worse than expected and almost one in four performed significantly better than would have been expected for their patient mix. The reasons for this variation in outcomes are not clear, nor is the impact of other components of the trauma system. These findings stress the need for a reliable means of identifying and improving the quality of care delivered within trauma systems.

Trauma system performance can be monitored in several ways: crude outcomes, risk‐adjusted outcomes, and 'preventable death rates' (MacKenzie 2006). Performance improvement is monitored in relation to these variables, and is also tracked directly with 'audit and feedback' that often employs 'audit filters' created by the American College of Surgeons Committee on Trauma (ACSCOT) (ACS 2006). Audit filters (also known as 'quality indicators') can be conceptualized as descriptions of specific clinical processes or outcomes of care that, when they occur, represent unfavorable deviations from an established norm. In a sense, audit filters represent 'sentinel' events in patient care (such as delays in performance of key tests or treatments, or unexpected deaths) which are associated with poor outcomes and/or sub‐optimal care. When such a 'fall out' event occurs it ideally should trigger review, discussion and, if appropriate, correction and individual practitioner feedback.

Although originally derived from both research and expert consensus opinion, there has been some criticism of the ACSCOT filters as a performance improvement strategy. For instance, one author has suggested that the filters do not properly account for individual patient injury profiles or recent advances in clinical care, and that they may not be directly linked to meaningful clinical or patient‐centered outcomes (Willis 2007; Willis 2008). There has also been a concern that in an established trauma system, with high patient volumes, these filters may yield few instances of unjustifiable fallouts and few opportunities for quality improvement (Cryer 1996). Finally, a recent Cochrane Review found that audit and feedback at the individual healthcare practitioner level was only minimally effective for improving clinical practice and patient outcomes (unless there is poor baseline adherence to recommended practice) (Jamtvedt 2006).

Why it is important to do this review

Traumatic injuries represent a considerable public health burden with significant personal and societal costs. The care of the severely injured patient in a trauma system progresses along a continuum that includes numerous interventions being provided by a multidisciplinary group of healthcare personnel. Despite the recent emphasis on quality of care in medicine, there has been little research to direct trauma clinicians and administrators on how optimally to monitor and improve upon the quality of care delivered within a trauma system. Audit filters are widely used for performance improvement in trauma systems, but have not been subjected to systematic review of their effectiveness.

Objectives

To determine the effectiveness of using audit filters for improving processes of care and clinical outcomes in trauma systems.

Methods

Criteria for considering studies for this review

Types of studies

We included data from randomized controlled trials, controlled clinical trials, controlled before‐and‐after studies, and interrupted time series studies. A summary of the observational study design types and inclusion criteria is provided in Table 1.

1. Description and inclusion criteria for observational studies (EPOC 2008).

Study design	Description	Specific inclusion criteria
Controlled clinical trial	A trial in which participants were: a) definitely assigned prospectively to 1 or 2 (or more) alternative forms of health care using a quasi‐random allocation method; or b) possibly assigned prospectively to 1 or 2 (or more) alternative forms of health care using a process of random or quasi‐random allocation
Controlled before‐and‐after study	A study involving the inclusion of experimental and control groups by a non‐random process and inclusion of baseline period of assessment of main outcomes	1) Contemporaneous data collection for pre‐ and post‐intervention groups for intervention and control sites 2) Appropriate control site. Control site is comparable in terms of reimbursement system, level of care, setting of care, and academic status. 3) Minimum of 2 sites for each of the intervention and control groups
Interrupted time series study	A study demonstrating a change in trend for an outcome that is attributable to a particular intervention	1) Clearly defined point in time when the intervention occurred 2) Minimum of 3 data points both before and after the intervention occurred

Studies performed in general or specific populations were eligible for inclusion.

Types of participants

Trauma patients were defined as persons sustaining one or more severe injuries. Participants included male and female patients of any age cared for by healthcare professionals following a traumatic injury in an organized system of trauma care. A trauma system is defined as a preplanned, organized, and co‐ordinated injury control effort in a defined geographic area (TAC 2007).

Types of interventions

The intervention of interest was the use of audit filters in a trauma system as a quality assurance initiative. Studies comparing the use or non‐use of audit filters between systems or within a single system were considered for inclusion. Also, studies comparing audit filters with another intervention were also eligible. Audit filters were operationally defined as descriptions of specific clinical processes or outcomes of care that represent unfavorable deviations from an established norm and are used for the purposes of identifying patients at risk of a poor outcome and enhancing quality of care via correction, discussion, or feedback.

Types of outcome measures

Primary outcomes

Primary outcomes of interest were measures of mortality and morbidity (e.g. discharge disability, quality of life, delay in return to work, etc.).

Secondary outcomes

Secondary outcomes of interest were measures of processes of care (e.g. time to performance of a key diagnostic test or to definitive treatment of an injury).

Search methods for identification of studies

There were no restrictions on publication status, language, or country of publication.

Electronic searches

We searched the following electronic databases:

• Cochrane Injuries Group Specialized Register (searched 8 December 2008)

• Cochrane EPOC Group Specialized Register (searched 16 January 2009)

• CENTRAL (The Cochrane Library 2008, Issue 4)

• MEDLINE 1950 to November 2008 (Week 3)

• EMBASE 1980 to November 2008 (Week 49)

• CINAHL 1982 to December 2008

• ISI Web of Science: Science Citation Index Expanded (SCI‐EXPANDED) 1970 to December 2008, Conference Proceedings Citation Index‐Science (CPCI‐S) 1990 to December 2008

• PubMed (last 6 months; searched 9 December 2008).

Full search strategies for the databases searched are presented in Appendix 1.

Searching other resources

Reference lists

We reviewed the reference lists of relevant studies for other potentially important studies.

Handsearches

We handsearched tables of contents of the following journals:

• Journal of Trauma: Injury, Infection, and Critical Care (1998 to February 2009)

• Injury: International Journal of the Care of the Injured (1998 to January 2009)

• Annals of Emergency Medicine (1998 to February 2009)

• Academic Emergency Medicine (1998 to February 2009)

• Injury Prevention (1998 to February 2009)

Clinical trial registries

We searched for unpublished studies using the following online clinical trial registries using the keywords "trauma", "audit", "system", "injury", and "performance":

• The World Health Organization International Clinical Trials Registry Platform (http://www.who.int/ictrp/en);

• Clinical Trials.gov (http://clinicaltrials.gov).

Personal contact

We contacted five experts with expertise in trauma systems in order to identify other potentially important articles. These authors were identified as experts based on their history of publishing research on trauma systems.

Data collection and analysis

Selection of studies

CE and DH independently reviewed the titles, abstracts, and descriptor terms of all articles identified in the electronic searches and discarded irrelevant articles.

CE and DH independently applied the inclusion criteria to the full text version of each citation. If disagreement occurred, a third author (LD), contributed to the discussion to determine consensus.

Authors had access to the journal name, the authors, and their affiliated institution for each publication, as hiding this information has uncertain benefit in protecting against bias (Berlin 1997).

We used Endnote version 10.0 (Thomson Research Software, Carlsbad, CA) to manage the citation information.

Data extraction and management

The methodology described in the protocol for this review was that two authors (CE and DH) would independently extract the following information from included studies.

• Source: study ID, reviewing author ID, citation and contact details.

• Eligibility: confirmation of inclusion criteria, reasons for exclusion.

• Methods: study design, date of study, total study duration including follow‐up time period, features of: sequence generation, allocation concealment, blinding, unit of allocation, unit of analysis, power calculation.

• Participants: intervention and control trauma system: trauma center description, location, academic status, patient profile, experience, pre‐hospital care provided, referring hospitals, baseline performance measures; patients: age, gender, severity of injury, mechanism of injury.

• Interventions: number of intervention groups, details of which audit filters used and how results were used for system improvement, description of degree of control group contamination, protection from secular changes.

• Outcomes: definitions of outcomes, units of measurement, methods of data analysis, risk‐adjustment undertaken, when the data were collected, when the data were reported, reliability of primary outcome measures.

• Results: number of participants allocated to each intervention group, for each outcome of interest: sample size, missing participants, outcomes in natural units, effect estimate with confidence interval and P value, subgroup analyses undertaken.

• Miscellaneous: funding source, ethics board approval, key conclusions, relevant author comments, references to other relevant studies, correspondence required, other comments by review team.

We would have collected these data on a standardized paper data extraction form. A third author (WP) would have reviewed discrepancies in data collection.

As no studies meeting our inclusion criteria were identified, the following steps were not undertaken. They will be reserved for future updated versions of the review.

Assessment of risk of bias in included studies

Randomized controlled trials and controlled clinical trials

We would have assessed randomized controlled trials and controlled clinical trials using the Cochrane Collaboration's 'Risk of bias' assessment tool (Higgins 2008, Section 8.5). Specific sources of bias would have included: sequence generation, allocation concealment, blinding of participants, personnel and outcome assessors, incomplete outcome data, selective outcome reporting, and 'other' identified concerns about sources of bias, such as baseline imbalance and protection against contamination (EPOC 2008a). Authors' judgments regarding risk of bias would have been either "yes" (low risk of bias), "no" (high risk of bias), or "unclear" (if insufficient information is available) as is customary for this tool.

In the event that cluster‐randomized trials were included, the additional sources of bias that would have been assessed include (Higgins 2008, Section 16.3.2):

• recruitment bias;

• baseline imbalance in either clusters or individuals;

• loss of clusters;

• incorrect analysis;

• comparability with individually‐randomized trials.

Controlled before‐and‐after studies

We would have assessed controlled before‐and‐after studies for risk of bias based on criteria developed by the EPOC group (EPOC 2008a). We would have assessed seven criteria and made judgment using the "yes", "no", or "unclear" category system described above.

• Baseline measurement of trauma system performance.

• Similarity of comparator control trauma systems to intervention systems.

• Blinded assessment of primary outcomes.

• Protection against contamination.

• Reliable primary outcome measures.

• Follow up of healthcare professionals (if relevant).

• Follow up of patients > 80%.

Interrupted time series studies

We would have assessed interrupted time series studies for risk of bias based on criteria developed by the EPOC group (EPOC 2008a). We would have assessed eight criteria and made judgment using the "yes", "no", or "unclear" category system described above.

• Protection against secular changes.

• Data were analyzed appropriately (ARIMA models or time series regression with serial correlation testing).

• Reason for the number of points pre‐ and post‐intervention are given.

• Shape of the intervention effect was specified.

• Protection against detection bias.

• Blinded assessment of primary outcome.

• Completeness of data set (> 80% of participants or episodes of care are included).

• Reliable primary outcome measure.

We would have presented assessments in 'Risk of bias' tables, stratified by study design.

Measures of treatment effect

Randomized controlled trials and controlled clinical trials

We would have recorded outcomes in natural units for each outcome in every study group. For each comparison, we would have reported the baseline and post‐intervention differences between control and study groups in natural units. For ordinal data, we would have documented measures of effect (risk ratios) along with 95% confidence intervals and corresponding P values.

Controlled before‐and‐after studies

We would have recorded outcomes in natural units for each outcome in every study group. We would have presented pre‐and post‐intervention means along with the post‐intervention absolute change, the post‐intervention relative percentage change, the absolute change from baseline, and the difference in absolute change from baseline as per EPOC guidelines (EPOC 2008b).

Interrupted time series studies

We would have reported outcomes in natural units for each major outcome. Consistent with EPOC guidelines (EPOC 2008b), additional information that would have been presented would include:

• the number of time points pre‐ and post‐intervention;

• the number of patients in the whole series;

• the time interval between points;

• pre‐ and post‐intervention means;

• absolute changes in outcomes, reported in natural units;

• relative percentage change;

• the model used and statistical significance of any findings.

We would have used Review Manager 5.0 (RevMan) for all data management purposes (RevMan 2008).

Unit of analysis issues

Potential units of analysis included individual trauma patients, clusters of patients within trauma settings and pre‐hospital and in‐hospital systems, and measures of patient outcomes within geographic and other administrative units. We would have stratified presentation of the results according to these varying units of analysis.

Had cluster‐randomized controlled trials been identified, we would have reviewed the method of analysis used by the study authors to determine whether the data were correctly analyzed. We would have considered data to have been analyzed correctly if either: 1) the analysis was conducted at the same level as the allocation (i.e. at the 'cluster' level) or, if 2) the analysis was conducted at the level of the individual, but appropriate statistical correction for the clustering methodology was performed (Higgins 2008, Section 16.3.3). We would have entered effect estimates and their standard errors from correct analysis of cluster‐randomized trials into RevMan and, if statistical homogeneity existed, conducted meta‐analyses using the generic inverse‐variance method (Higgins 2008, Section 16.3.3).

If the data analysis for a cluster‐randomized trial was determined to have been performed incorrectly, and if sufficient information was available, we would have performed an "approximately correct analysis" (Higgins 2008, Section 16.3.3).

Dealing with missing data

We would have clearly documented incomplete data on the data extraction form and contacted study authors to obtain important missing information.

Assessment of heterogeneity

A preliminary review of the relevant literature suggested that there was much variation in study design, participants, interventions, comparisons, and outcomes.

If studies were identified and there was not clear evidence of clinical heterogeneity, we would have assessed statistical heterogeneity using the Chi² test (with a P value < 0.1 suggesting significant variation in effect estimates beyond chance) and quantified using the I² statistic (with an I² value > 50% representing substantial heterogeneity precluding meta‐analysis) (Higgins 2008, Section 9.5.2).

Assessment of reporting biases

We would have compared the methods section of each article with the results reported to ensure that there had been consistent reporting of all outcomes of interest. Where important results (positive or negative) were not described in adequate detail, we planned to contact the authors of the respective studies for clarification. In instances where clearly important outcomes were not reported (e.g. mortality), we would have contacted authors to explore whether determining those outcomes was a part of the original protocol. If a sufficient number of studies had been identified, we would have used a funnel plot to explore for potential reporting biases.

Data synthesis

We would have stratified studies according to their design and performed meta‐analysis within these strata if the data from included studies were sufficiently homogeneous for participants, interventions, comparisons, and outcomes to provide a meaningful summary of the results.

Subgroup analysis and investigation of heterogeneity

If sufficient data were available, we planned to conduct subgroup analysis using meta‐regression for the following variables: study design, trauma system location (urban versus rural), trauma system patient population size or catchment area, age, trauma system experience, and academic status of trauma center.

Sensitivity analysis

If the data had been available, we would have performed sensitivity analyses for crude to adjusted estimates of treatment effects, as well as published to unpublished study results.

Results

Description of studies

Results of the search

We screened the titles of 741 studies to identify eligible trials. Of these studies, 42 abstracts were appraised and five complete articles were examined in detail.

Included studies

We identified no studies that met our inclusion criteria.

Excluded studies

The five studies that are reported in the 'Characteristics of excluded studies' table most closely addressed our research question. The two studies by Chadbunchachai (Chadbunchachai 2001 and Chadbunchachai 2003) prospectively evaluated the efficacy of trauma audit filters and key performance indicators, respectively, in altering the preventable mortality rate in a single Thailand hospital. These studies were both excluded as neither study included sufficient pre‐ or post‐intervention data points or clearly defined a point in time when the respective interventions were implemented. The Eckstein 1999 study, which used a specific performance improvement program to reduce paramedic on‐scene times for penetrating trauma patients was excluded as there was only one pre‐intervention data point and the authors, when contacted, were unable to provide additional data points. The Ruchholtz 2004 study was excluded as it addressed the utility of the trauma registry itself at improving performance in trauma care (rather than audit filters) and was a retrospective design. Willis 2007 did not meet our inclusion criteria as it is a narrative review of trauma audit filters.

Risk of bias in included studies

As no studies met the inclusion criteria, we were not able to consider the risk of bias present.

Effects of interventions

Effects of interventions were not determined as there were no included studies.

Discussion

Despite an extensive electronic search yielding 741 results, a review of tables of contents from five journals of relevance to our research question, review of multiple clinical trial registries, and direct communication with several content experts, we were unable to identify any studies that addressed our specific research question and also met our inclusion criteria. The comprehensiveness of our search suggests that there is, at present, no documented evidence to support or refute the use of audit filters as a means of improving processes of care and clinical outcomes in an established system of trauma care. This finding suggests an urgent need to design a clinical trial to address this question so that administrators of trauma care will know whether efforts to collect audit filter data will generate improvements in clinical care, or whether an alternative quality improvement strategy should be investigated.

The three studies which came closest to meeting our inclusion criteria were those of Chadbunchachai 2001, Chadbunchachai 2003, and Eckstein 1999. Chadbunchachai 2001 compared rates of preventable death at Khon Kaen Regional Hospital in the two years before and the six months following the implementation of a set of audit filters. They found that the preventable death rate at their institution declined from 2.7% to 2.0% following the initiation of the audit filters. As our inclusion criteria for this study design required a minimum of three data points before and an additional three after the implementation of a set of audit filters, this study did not meet our inclusion criteria. This study also specified a two‐month period of audit filter implementation, which we feel is not clear enough to be confident of when the post‐intervention phase of the study began.

Chadbunchachai 2003 reported on the revision of their previous set of audit filters (subsequently termed "key performance indicators"), the re‐implementation of these indicators, and the change in preventable death rate. Following this set of revisions, the preventable death rate declined from 2.0% to 1.3%. This study could also not be included as there were not a sufficient number of data measurements either before or after the key performance indicators were integrated into the trauma system.

Eckstein 1999 developed a focused audit and feedback program at the Los Angeles County/USC Medical Center to reduce the amount of time paramedics spent on‐scene resuscitating victims of penetrating trauma. Cases involving on‐scene times greater than 20 minutes were reviewed and, if the delayed transportation appeared unjustified, constructive feedback was provided to the paramedics involved in that particular case in order to prevent recurrence. The rates of prolonged on‐scene time decreased from 4.1% prior to the program's inception to 1.5% four years later. This study also could not be included as there were not a sufficient number of measurements of the on‐scene times prior to the implementation of the performance improvement program.

Authors' conclusions

Implications for practice.

Organized systems of trauma care have been shown to improve survival for severely injured patients (Nathens 2000; Cameron 2008). Despite this, it is not yet clear which aspects of the system contribute to the reduction in mortality, by how much, and how it is that they do it. Our study builds on these questions and suggests that there is another unknown in the trauma system literature: "do our currently accepted strategies for improving the quality of care that we deliver to injured patients actually work?"

We were unable to identify any studies of sufficient methodological rigor to address the question of whether trauma audit filters are effective in improving the processes by which care is rendered to critically injured patients, or whether they provide a means for reducing mortality rates at the system level. This is a very important gap in the literature. A recent study from Australia by Willis 2008investigated the relationship between 14 audit filters and clinical outcomes. Only three of the filters (abdominal surgery > 24 hours after arrival, blunt compound tibial fracture treatment > eight hours after arrival, and non‐operative management of femoral diaphyseal fracture) were associated with an increased risk for mortality, while only three others were linked to increased lengths of hospital stay (cranial surgery > 24 hours after arrival, abdominal surgery > 24 hours after arrival, and patients developing venous thromboemboli or decubitus ulcers). These findings challenge the underlying assumption with audit filters; that they are reflective of quality of care provided and/or are determinants of clinical outcomes. These findings, when taken in the context of our review, suggest that our current mechanism for identifying shortcomings in clinical care and building upon them in order to improve patient care is based on little evidence. With the recent academic, political, and public interest in patient safety and quality of care there is an urgent need to determine how to best monitor and improve on the care delivered within trauma systems.

Implications for research.

Our study suggests several areas for future research. First, methodologically sound studies addressing the question of whether audit filters are an effective and economical method of improving processes of care and clinical outcomes in trauma systems are needed. Such studies may be difficult to carry out using a randomized trial design given that audit filters are routinely used in most trauma systems already, and have generally been accepted as a useful quality assurance tool by system administrators. Thus, finding a temporally‐matched comparable control group for such a study may prove difficult. Such a study would need to be 'cluster randomized' by trauma system and the data analyzed accordingly.

For these reasons, an observational study design (ideally with a control group) such as those included in our search strategy may be a more practical means of addressing our primary study question. A controlled before‐and‐after study design, which would involve measuring outcomes in an 'experimental trauma system' (which implements audit filters) with a 'control trauma system' (which does not implement audit filters) before and after the intervention might be ideal. A variation of the controlled before‐and‐after study design was used in the the Ontario Pre‐hospital Advanced Life Support (OPALS) major trauma study (Stiell 2008). This study compared system‐wide trauma outcomes before and after the implementation of a pre‐hospital advanced life support program in Ontario, Canada. The study found no improvement in outcomes related to the intervention, but was limited by the lack of a separate, temporally‐matched comparator 'control trauma system' without the advanced life support intervention. This limitation would also likely apply a controlled before‐and‐after study addressing our research question because it would be difficult to identify a comparable control trauma system that was not already using audit filters.

An alternative approach would be the interrupted time series study design. Using this methodology, multiple outcome observations (for example, processes of care, morbidity, and mortality) would be collected over consecutive years. Audit filters would then be introduced at a clearly defined point in time, and observations would again be repeated over consecutive years. Such a study would ideally include a 'control trauma system' for which outcomes are collected along the same timeline, but which does not implement audit filters. The effectiveness of audit filters could then be determined by comparing the trends in outcomes between the pre‐ and post‐intervention phases of the study (EPOC 2008c). Chadbunchachai 2003, Chadbunchachai 2001, and Eckstein 1999 all used uncontrolled interrupted‐time‐series designs for their respective studies, but were excluded as each had an insufficient number of pre‐intervention data points to provide a reliable sense of the pre‐intervention trend in outcomes.

Future research should also compare the relative and/or additive effectiveness of trauma audit filters with other quality improvement strategies. Recent initiatives have included video‐auditing of trauma resuscitation (Fitzgerald 2006), daily rounding checklists for the trauma intensive care unit (DuBose 2008), and national survival risk ratios (Bergeron 2007). Future studies that assess the effectiveness of quality improvement strategies for trauma care should include both mortality and morbidity outcome measures (Glance 2004).

Appendices

Appendix 1. Search strategy

Injuries Specialized Register (searched 8 December 2008)

((Trauma or emergency or intensiv*) and (care or ward or nurs* or medic* or physician* or surgery or treatment* or service* or unit* or centre* or center*)) and ((quality and indicator*) or (audit and filter*) or "medical audit" or (program and evaluat*))

EPOC Specialized Register (searched 16 January 2009)

Search 1

1. [triage or critical care or emergency nursing or emergency medicine or trauma or traumatology or intensive care or emergency medical services or emergency unit(s) or emergency room(s) or emergency service(s) or emergency medicine or emergency care or emergency ward(s)] SEARCHED IN ALL FIELDS. 

2. (Medical audit or audit filter or audit filters or audit filtre or audit filters or outcome assessment or process assessment or program evaluation or quality indicators [SEARCHED IN ALL FIELDS] OR Quality or evaluation or assessment [SEARCHED IN TITLE and DESCRIPTORS/MeSH])

3. 1 and 2

Search 2  

Medical audit or audit filter or audit filters [SEARCHED IN ALL FIELDS]

CENTRAL (The Cochrane Library 2008, Issue 4)

1. MeSH descriptor Triage explode all trees 
 2. MeSH descriptor Critical Care explode all trees 
 3. MeSH descriptor Emergency Nursing explode all trees 
 4. MeSH descriptor Emergency Medicine explode all trees 
 5. MeSH descriptor Trauma centres explode all trees 
 6. MeSH descriptor Traumatology explode all trees 
 7. MeSH descriptor Intensive Care Units explode all trees 
 8. MeSH descriptor Emergency Medical Services explode all trees 
 9. (Trauma or emergency or intensiv*) near3 (care or ward or nurs* or medic* or physician* or surgery or treatment* or service* or unit* or centre* or centre*) 
 10. (#1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9) 
 11. MeSH descriptor Wounds and Injuries explode all trees with qualifier: MO 
 12. (#10 AND #11) 
 13. MeSH descriptor Outcome Assessment (Health Care) explode all trees 
 14. MeSH descriptor Process Assessment (Health Care) explode all trees 
 15. MeSH descriptor Medical Audit explode all trees 
 16. MeSH descriptor Program Evaluation explode all trees 
 17. MeSH descriptor Quality Indicators, Health Care explode all trees 
 18. (quality near3 indicator*) near5 (health‐care or healthcare) 
 19. audit near3 filter* 
 20. (#13 OR #14 OR #15 OR #16 OR #17 OR #18 OR #19) 
 21. (#12 AND #20)

MEDLINE 1950 to November 2008 (Week 3)

1. exp Triage/ 
 2. exp Critical Care/ 
 3. exp Emergency Nursing/ 
 4. exp Emergency Medicine/ 
 5. exp Trauma centres/ 
 6. exp Traumatology/ 
 7. exp Intensive Care Units/ 
 8. exp Emergency medical services/ 
 9. ((Trauma or emergency or intensiv*) adj3 (care or ward or nurs* or medic* or physician* or surgery or treatment* or service* or unit* or centre* or centre*)).ab,ti. 
 10. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 
 11. standards.fs. 
 12. 10 and 11 
 13. exp "Wounds and Injuries"/mo [Mortality] 
 14. 12 or 13 
 15. exp *"outcome assessment (health care)"/mt [methods] 
 16. exp *"process assessment (health care)"/mt [methods] 
 17. exp Medical Audit/mt [Methods] 
 18. exp Program Evaluation/mt [Methods] 
 19. exp *quality indicators, health care/(quality adj3 indicator* adj5 health?care).ab,ti. 
 20. (audit adj3 filter*).ab,ti.exp 
 21. Medical Audit/mt [Methods] 
 22. 15 or 16 or 17 or 18 or 19 or 20 or 21 or 22 
 23. 14 and 23

EMBASE 1980 to November 2008 (Week 49)

1. exp emergency health service/ 
 2. exp Emergency Medicine/ 
 3. exp Emergency Surgery/ 
 4. exp Emergency Treatment/ 
 5. exp Emergency Nursing/ 
 6. exp Emergency Care/ 
 7. exp Emergency/ 
 8. exp Emergency Ward/ 
 9. exp Emergency Physician/ 
 10. exp Traumatology/ 
 11. exp intensive care unit/ 
 12. ((Trauma or emergency or intensiv*) adj3 (care or ward or nurs* or medic* or physician* or surgery or treatment*or service* or unit* or centre* or centre*)).ab,ti. 
 13. or/1‐12 
 14. exp injury/ 
 15. exp wound/ 
 16. (injur* or wound* or trauma*).ab,ti. 
 17. 14 or 15 or 16 
 18. exp mortality/ 
 19. 17 and 18 
 20. exp Outcome Assessment/ 
 21. exp Medical Audit/ 
 22. exp quality control/ 
 23. exp total quality management/ 
 24. exp Performance Measurement System/ 
 25. exp Health Care Quality/ 
 26. (quality adj3 indicator* adj5 health?care).ab,ti. 
 27. (audit adj3 filter*).ab,ti. 
 28. or/20‐27 
 29. exp standard/ 
 30. exp gold standard/ 
 31. exp Professional Standard/ 
 32. 29 or 30 or 31 
 33. 13 and 19 and 28 and 32

CINAHL 1982 to December 2008

1. MH “triage” 
 2. MH “critical care” 
 3. MH “critical care nursing” 
 4. MH “emergency nursing” 
 5. MH “emergency medicine” 
 6. MH “trauma centres” 
 7. MH "Emergency Medical Services+" 
 8. MH "Intensive Care Units+" 
 9. MH "Trauma Nursing" 
 10. TI ( Trauma or emergency or intensiv* ) and TI ( care or ward or nurs* or medic* or physician* or surgery or treatment* or service* or unit* or centre* or centre* ) 
 11. S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 
 12. (MH "Wounds and Injuries+") 
 13. TI (injur* or wound* or trauma*) or AB (injur* or wound* or trauma*) 
 14. S12 or S13 
 15. S11 and S14 
 16. (MH "Nursing Audit") 
 17. MH "Quality of Care Research" 
 18. (MH "Quality Assessment") 
 19. (MH "Clinical Indicators") 
 20. quality N3 indicator* 
 21. audit N3 filter* 
 22. S16 or S17 or S18 or S19 or S20 or S21 
 23. S15 and S22

ISI Web of Science: Science Citation Index Expanded (SCI‐EXPANDED) 1970 to 6 December 2008, Conference Proceedings Citation Index ‐ Science (CPCI‐S) 1990 to 6 December 2008

Topic=((Trauma or emergency or intensiv*) and (care or ward or nurs* or medic* or physician* or surgery or treatment* or service* or unit* or centre* or center*)) AND Topic=((injur* or wound* or trauma*) and mortality) AND Topic=((quality and indicator*) or (audit and filter*) or "medical audit" or (program and evaluat*) or (quality and control*))

PubMed (last six months; searched 9 December 2008) 

#1    (Trauma or traumas or emergency or intensiv*) and (care or ward or nursing or medical or physician* or surgery or treatment* or service* or unit or units or centre or centres or center or centers*) 
 #2    (injury or injuries or injured or wound or wounds or wounding or wounded or trauma or traumas) and mortality 
 #3    #1 AND #2 
 #4    quality assessment or quality indicator* or quality control or clinical assessment 
 #5    program evaluat* 
 #6    nursing audit* 
 #7    medical audit* 
 #8    audit filter* 
 #9    #4 OR #5 OR #6 OR #7 OR #8 
 #10 #3 AND #9

Characteristics of studies

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Chadbunchachai 2001	This is an interrupted time series study evaluating rates of errors in trauma treatment, rates of such errors contributing to mortality, and preventable death rates at a single Thai hospital after the revision and re‐implementation of a set of trauma audit filters. This study was excluded as there were an insufficient number of pre‐ and post‐intervention data points and there was not a clearly defined point in time at which the revised audit filters were implemented.
Chadbunchachai 2003	This is an interrupted time series study comparing preventable death rates at a single Thai hospital following the revision and expansion of a set of existing trauma audit filters. This study was excluded as there were not enough pre‐ or post‐intervention data points.
Eckstein 1999	This is an interrupted time series study evaluating the efficacy of a performance improvement program at decreasing the on‐scene time for paramedics responding to calls involving a patient sustaining a penetrating mechanism of trauma. The study was excluded because there was only one pre‐intervention data point available.
Ruchholtz 2004	This is a retrospective review of prospectively collected data from a German trauma registry in which the authors demonstrate improved performance in trauma care (as measured with quality indicators) as a result of feedback provided from registry data. This study was excluded for several reasons, the most important of which being that the intervention being studied is the trauma registry itself and not audit filters.
Willis 2007	This study is a narrative review of issues regarding the use of quality indicators as a quality assurance tool in trauma care. It was excluded because it does not fit the inclusion criteria for this review.

Contributions of authors

All authors were involved in the study design. CE and DH performed the literature review. CE wrote the draft manuscript, which was edited by DH, WP, and LD.

Declarations of interest

None known.

New