Abstract
Background
Reporting of adverse clinical events is thought to be an effective method of improving the safety of healthcare. Underreporting of these adverse events is often said to occur with consequence of missing of opportunities to learn from these incidents. A clinical incident can be defined as any occurrence which is not consistent with the routine care of the patient or the routine operation of the institution.
Objectives
To assess the effects of interventions designed to increase clinical incident reporting in healthcare settings.
Search methods
We searched the the following databases: Cochrane Effective Practice and Organisation of Care Group Specialised Register, the Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE (OVID), EMBASE (OVID), CINAHL (EBSCO), Social Science Citation Index and Science Citation Index (Web of Knowledge), Healthstar (OVID), INSPEC, DHSS-DATA, SIGLE, ISI Conference Proceedings, Web of Science Conference Proceedings Citation Index (Science), Database of Abstracts of Reviews of Effectiveness (DARE).
Selection criteria
Randomised controlled trials (RCT), controlled before-after studies (CBA) and interrupted time series (ITS) of interventions designed to increase clinical incident reporting in healthcare.
Data collection and analysis
At least two review authors assessed the eligibility of potentially relevant studies, extracted the data and assessed the quality of included studies.
Main results
Four studies (one CBA and three ITS studies) met our inclusion criteria and were included in the review. The CBA study showed a significant improvement in incident reporting rates after the introduction of the new reporting system. Just one of the ITS studies showed a statistically significant improved effectiveness of the new reporting system from nine months. The other two studies reported no statistically significant improvements.
Authors’ conclusions
Because of the limitations of the studies it is not possible to draw conclusions for clinical practice. Anyone introducing a system into practice should give careful consideration to conducting an evaluation using a robust design.
Medical Subject Headings (MeSH): Medical Errors [prevention & control], Risk Management [organization & administration; *standards; utilization]
MeSH check words: Humans
BACKGROUND
There has been increasing interest in safety within healthcare over the last 10 to 20 years. The catalyst for this was the Harvard Medical Practice study of hospital inpatients in 1991 (Brennan 1991) that suggested an adverse event rate of nearly 10%. A number of similar reports from across the world were subsequently published but were only really of concern to healthcare regulators. It was not until 1999 that public attention (and media interest) was first drawn to the importance and magnitude of the issue of patient harm from healthcare systems. The seminal publication was ‘To err is human’ (Kohn 2000) from the U.S. Institute of Medicine, which estimated that medical errors kill between 44,000 and 98,000 people a year in American hospitals. This has been interpreted as healthcare systems killing more people annually than car accidents or as the equivalent of three jumbo jets crashing every two days (Leap 1994). It has been estimated that in the USA, the total national costs of preventable adverse events are between USD 17 and USD 29 billion (Thomas 1999).
Although these error figures have been challenged (McDonald 2000), it is often reported that errors in medical care are on a different scale from error tolerated elsewhere. A clearer picture of the epidemiology of error is emerging and it is becoming clear that providing safe and effective care requires not only expert clinicians, but also well designed care processes and organizational supports (Elwyn 2005). So recognition that errors are often the results of poorly designed systems and encouraging everyone to identify and learn from errors are equally important in promoting a culture of safety.
Various terms are used in describing errors in medical care and it is likely that different studies will use different definitions and any analysis must address this. There is no agreed definition of what does or does not constitute a clinical incident. One of the most common definitions was developed by the Hospital for Sick Children in Toronto and it is used by many hospitals in the UK: ‘a medical error is any occurrence which is not consistent with the routine care of the patient or the routine operation of the institution’. They can be broadly divided into adverse events and near misses. According to ‘Organisation with a memory’ (NHS 2000) an adverse event is ‘An event or omission arising during clinical care and causing physical and psychological injury to a patient’. These are, therefore, clinical incidents in which the patient does come to harm. The same group defines near misses as a ‘situation in which an event or omission or a sequence of events or omissions arising during clinical care fails to develop further whether or not as the result of compensating actions, thus preventing injury to the patient’. Near misses are important because they can indicate ‘weak areas’ in medical care and thus warn of future harm to patients.
Why it is important to do this review
The patient safety literature contains a number of reports of systems that can be used for the reporting of clinical incidents. Some authors have suggested certain attributes that they feel represent elements of reporting systems likely to make them more useful to users (Leape 2002). Leape suggests successful systems should occur in a safe non-punitive environment, be simple, timely and inexpensive. However, the effectiveness of such systems in promoting adverse event recording is not clear.
OBJECTIVES
To determine the effectiveness of interventions that aim to increase the rate of clinical incident reporting in healthcare systems.
METHODS
Criteria for considering studies for this review
Types of studies
We considered randomised controlled trials (RCTs) non randomised controlled trials (NRCTs), controlled before-after studies (CBAs) and interrupted time series analyses (ITS) that met the quality criteria used by the Cochrane Effective Practice and Organisation of Care Group (EPOC 2011). CBAs were only included if they had at least two intervention and two control sites. ITS were eligible if they had a clearly defined point in time when the intervention occurred and three data collection points before and after the intervention to take into account secular trends and auto-correlation among measurements over time (Ramsay 2003).
Types of participants
Any public or private healthcare organisation including qualified healthcare professionals who deal directly with patients.
Types of interventions
We included any intervention aimed at increasing clinical incident reporting, for example,comparing an established system with a modified one; or the introduction of a formal system in the place of an informal system. Included systems could be either mandatory or voluntary, paper-based or electronic.
Types of outcome measures
Main outcomes: Any objective measure of the rate of clinical incident reporting of adverse events or near misses as defined by the authors.
Other outcomes: Improved patient safety (e.g. lowering of mortality rates); reduced resource utilization (e.g. reduced length of stay) or other available patient outcome data where reported; user (practitioner and institution) satisfaction.
Search methods for identification of studies
Electronic searches
We identified primary studies using the following bibliographic databases, sources, and methods. We identified related systematic reviews by searching the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effectiveness (DARE), and the databases listed below. Search strategies were developed by F. Beyer. Databases were searched from database inception date to March 2012 with the exception of HealthStar, INSPEC, DHSS-DATA, and SIGLE which were searched in September 2007. SIGLE was not searched in 2012 because it is no longer updated; HealthStar, INSPEC, and DHSS-DATA were omitted from the 2012 search due to the low relevance of the 2007 results. We did not apply language or date limits. Results were restricted by methodological filters to identify RCT and non-RCT designs. Strategies are available in Appendix 1..
Databases
MEDLINE, OVID (1950-, In-Process and other non-indexed citations)
EMBASE, OVID (1947-)
The Cochrane Central Register of Controlled Trials (CENTRAL), Issue 3, 2012 Wiley
CINAHL (Cumulative Index to Nursing and Allied Health Literature), EbscoHost (1980-)
EPOC Group, Specialised Register, Reference Manager
Science Citation Index (Web of Knowledge) (1970-)
Social Science Citation Index (Web of Knowedge)
ISI Conference Proceedings
Web of Science Conference Proceedings Citation Index- Science (Web of Knowledge)
HealthStar (OVID) (2007-)
Inspec
Department of Health and Social Security (DHSS-DATA)
SIGLE (System for Information on Grey Literature in Europe)
Searching other resources
Reviewed reference lists of relevant systematic reviews and studies.
Contacted researchers with expertise relevant to the review topic or EPOC interventions.
Data collection and analysis
Selection of studies
We downloaded all titles and abstracts retrieved by electronic searching to the reference management database EndNote, and removed duplicates. At least two of the review authors (EP, MPE, SF, GR) independently examined the remaining references. We excluded those studies which clearly did not meet the inclusion criteria and obtained copies of the full text of potentially relevant references. At least two of the review authors (EP, MES, SF, GR) independently assessed the eligibility of retrieved papers.
Data extraction and management
We designed a modified version of the EPOC Data Collection Checklist (EPOC 2011). At least two of the authors (EP, GF, SF, GR) independently extracted the data using the modified checklist. We used the same criteria as those outlined in the Cochrane Handbook for Systematic Reviews of Interventions to evaluate data (Higgins 2008) to evaluate data and we solved any disagreement by discussion and the involvement of an arbitrator (MPE) as necessary.
Assessment of risk of bias in included studies
We assessed the risk of bias of studies eligible for the review using the criteria suggested by EPOC (EPOC 2011). RCTs, CCTs and CBAs were assessed for generation of allocation sequence, concealment of allocation, baseline outcome measurements, baseline characteristics, incomplete outcome data, blinding of outcome assessor, protection against contamination, selective outcome reporting and other risks of bias. For ITS designs we also assessed the independence of the intervention from other changes, the prespecified shape of the intervention and if the intervention was unlikely to affect data collection. We resolved any disagreement by discussion and the involvement of an arbitrator (MPE) as necessary.
Measures of treatment effect
For each study, we reported data in natural units. Where baseline results were available from RCTs, NRCTs and CBAs, we reported pre-intervention and post-intervention means or proportions for both study and control groups as the unadjusted and adjusted (for any baseline imbalance) values.
For ITS studies we reported the main outcomes in natural units and two effect sizes: the change in the level of outcome immediately after the introduction of the intervention and the change in the slopes of the regression lines. Both of these estimates were necessary for interpreting the results of each comparison. For example, there could have been no change in the level immediately after the intervention, but there could have been a significant change in slope. We also reported level effects for six months and yearly post-intervention points within the post-intervention phase.
We presented the results for all comparisons using a standard method of presentation where possible. For comparisons of RCTs, CCTs, CBAs we reported (separately for each study design): median effect size across included studies; inter-quartile ranges of effect sizes across included studies; and the range of effect sizes across included studies.
Data synthesis
Meta-analysis was not carried out. Instead a narrative results summary was produced. In three of the included studies (Bilimoria 2009; Dixon 2002; Stump 2000) data on incident reporting rate were re-analysed as a time series. We used the Review Manager 5 (RevMan) (RevMan 2011) to present and synthesise the data.
Subgroup analysis and investigation of heterogeneity
No sub-group or sensitivity analyses were performed. Too few studies were identified to explore heterogeneity.
RESULTS
Description of studies
See: Characteristics of included studies; Characteristics of excluded studies.
Results of the search
Figure 1 shows the study PRISMA flow chart (Liberati 2009). The search strategy led to the identification of 2153 records. After the independent examination by the review authors, we retrieved 25 articles potentially eligible for the review. After the full text assessment, four studies met the inclusion criteria. A description of retrieved studies and reasons for their exclusion are presented in the “Characteristics of excluded studies” table. The four studies which met the inclusion criteria are reported in detail in the “Characteristics of included studies” table.
Figure 1.
An update search ran the 20 March 2012 yielded an additional 2701 citations (after 883 duplicates were removed). Of these studies 2681 were judged as non relevant and directly excluded, leaving 20 abstracts/studies for closer scrutiny.Of these 20 studies 13 were excluded with reasons (see Characteristics of excluded studies table) and seven studies, for which no full text was retrieved, were added to the studies awaiting classification list.
Included studies
Of the four included studies three were re-analysed as ITS (Bilimoria 2009; Dixon 2002; Stump 2000) and one was a CBA (Evans 2007). The 3 ITS studies were all conducted in USA, and the CBA study was conducted in Australia.
Targeted behaviour
The aim of all the included studies was to improve systems of medical incident reporting. Stump et al (Stump 2000) aimed to increase the number and quality of voluntary reports to uncover system weaknesses and shorten the turnaround time from occurrence to analysis to ultimately implement corrective actions. Dixon et al (Dixon 2002) introduced a web-based system to standardize and enhance incident reporting. Evans (Evans 2007) aimed to improve incident reporting rates and change the type of incident reported. Bilimoria et al (Bilimoria 2009) aimed to track adverse events and near-miss events to establish an automated method to identify patterns of events and to assess the reporting behavior of physicians.
Participants and settings
The setting in the Stump study (Stump 2000) was Yale-New Haven Hospital, a 944-bed, tertiary-care teaching hospital affiliated with Yale University. Dixon et al (Dixon 2002) was set in a group of four administratively linked hospitals in Dallas, Texas. Evans et al (Evans 2007) involved the nursing staff working in 20 (10 intervention and 10 control) units in four major cities and two regional hospitals of South Australia. Bilimoria (Bilimoria 2009) was set in one surgery department in a metropolitan tertiary care centre in the USA.
Description of the intervention
Stump 2000
A standardized, non-punitive system with improvement efforts focused on the medication-use system, competency assessment and reporting incentives. To foster the non-punitive culture, the term “medication-use variance” was used instead of “clinical error” because it is free of negative connotations. Key elements of the system were as follows:
Voluntary: participants were encouraged and taught how to use the new system but whether they chose to use the system was up to them.
Standardised: the previous system had a number of different medication error systems e.g. for chemotherapy, a whole hospital incident reporting system and an internal pharmacy system. The new system centralised all medication error reports to the pharmacy department.
Non-punitive: the previous process had been punitive i.e. ‘corrective’ (e.g. counselling or remedial training) or involving disciplinary action. The new system focused on errors in the ‘system’ as well as general ‘competency assessment’ if necessary.
Anonymous: a paper-based system was used to ensure this.
Timely: the intention was that most reports would be received within 24 hours.
Paper based: although the reports were written, rather than use a narrative style a ‘check-box’ report was devised. This prompted the user for ‘key data’ that had been devised by a committee.
Near misses data: one of the aims of the system was to capture this type of data which had never been reported under the previous system.
Education: during implementation of the form, staff conducted in-service training programs on the need for and use of the new reporting system.
Feedback: the paper described the move from the previous system where those who reviewed the data were ‘far from the frontline’ to the new system which involved staff at ‘grassroots level’ in reviewing data and in planning improvements.
The new system was first pilot-tested from May to September 1998 in the hospital pharmacy and three patient care areas (details not given in paper). The implementation in the whole hospital started in March 1999.
Dixon 2002
A web-based “patient occurrence system” not limited to medical error. Key elements of the system were as follows
Electronic reporting: the hospitals existing Intranet was used. This system was available to all employees using the normal secure systems (username, password).
Confidential: using the system described above it was not possible to be anonymous but the participants were reassured that it was confidential. There was also a ‘hotline’ system available where staff could ring a number and report incidents anonymously.
Standardised: the electronic form was designed to capture information felt to be important. The data were collected in code form to make analysis easier.
Educational programme in tandem: an educational programme was implemented to cover all staff and all shifts. These discussed the need for reporting systems as well as how to fill the forms in.
Timely: the electronic systems were immediate unlike the previous paper based systems which could be significantly delayed.
The author described the system going ‘live’ in July 2000. Although the paper discussed a phase II and III, these were related to access of the reported events (by administrators) rather than changes that would have been obvious to the user. The corresponding author was contacted and confirmed that the electronic system was implemented from 5 July 2000.
Evans 2007
An intervention package comprising intense education, a range of reporting options, a change in report management and enhanced feedback. The components of the intervention were as follows:
Education: development of a manual to improve knowledge of reportable incidents and education sessions to explain the purpose of the study.
Reduce fear of reporting: processes were redesigned and anonymous reports were validated and managed only by the patient safety manager in each institution.
Reduce reporting burden: a one-page report replaced the three-page form.
Improve feedback: four newsletters were distributed to inform staff about incidents reported during the study period.
The intervention started from June to August 2003 with units implementing the intervention over a 40-week period.
Bilimoria 2009
An online morbidity, mortality and anonymous near-misses reporting system. For this review we were interested in one component of the intervention - the introduction of a weekly reminder to report complications and near-miss events. The weekly reminders were introduced in January 2007.
Outcomes
In all of the included studies the “incident reporting rate” was measured. Moreover in Dixon study the authors also measured also the “time to report” and in Evans’ study the “change in type of incident” was reported.
Excluded studies
Thirty-four studies were excluded after full copies of the papers were obtained and scrutinized. Reasons for exclusion are reported in the “Characteristics of excluded studies” table.
Risk of bias in included studies
The risk of bias of included studies is described in the risk of bias table within the Characteristics of included studies table. All the included studies are at high risk of bias.
None of the three ITS (Bilimoria 2009; Dixon 2002; Stump 2000) gave rational explanation for the shape of the intervention effect; although as we were re-analysing them, this was difficult to assess. It was also unclear whether the outcomes could have been influenced by other events during the study period. One of the criteria for assessing the risk of bias of studies with an ITS design pertains to the fact that the intervention should be unlikely to affect data collection; in all our ITS studies the intervention itself represented a change in the way data were collected. Moreover, none of the ITS studies had a control group.
The CBA study presented inadequate generation and concealment of the allocation sequence. As stated in the study, there was also heterogeneity between the control and intervention units at baseline for characteristics and outcome measures.
Effects of interventions
See: Summary of findings for the main comparison
Due to the heterogeneity of institutions, personnel, interventions and study design in included studies, we did not combine the results quantitatively.
Incident reporting rate
Stump 2000
Before the introduction of the new reporting system there was a small decrease of about 2.5 reported events each quarter which was non-significant (pre-slope= −2.516, P=0.588). The number of reported events ranged from 42 to 49 events per quarter prior to the intervention. The mean number of reported events increased at three and six months after the start of the intervention, ranging from 112 to 276 events per quarter, but this improvement was not statistically significant (n=26.4, P=0.491 and n=72.2, P=0.067 respectively). There was a statistically significant increase of the mean number of reported events at 9 (n=117.9, P=0.010), 12 (n=163.6, P=0.003) and 15 months (n=209.3, P=0.002).
Dixon 2002
Before the intervention there was a very small increase of about one reported events per-month which was nonsignificant (pre-slope=0.99, P=0.596). The mean increase in the number of reported events at one, three, six, nine and twelve months after the introduction of the new system ranged between 31 and 34 events per month, but none of these increases was statistically significant.
Evans 2007
Despite matching by type of unit and hospital location, there was considerable heterogeneity between the control and intervention units at baseline. Owing to the large size of the major city intervention intensive care unit (ICU), there were twice as many ICU beds in the intervention units (42 beds) as there were in the control units (21 beds), a significant difference (Chi2 (1) = 11.19, P=0.001). At baseline, there were significantly more reports made in inpatient intervention units and in emergency department (ED) control units, and there were more anonymous reports generated in control units. Compared with control units, the intervention resulted in an absolute increase of 60.3 reports/10,000 occupied bed days (OBDs) (95% CI 23.8 to 96.8 reports/10 000 OBDs) in inpatient areas (P=0.001), 39.5 reports/10 000 ED attendances (95% CI 17.0 to 62.0 reports/10 000 ED attendances; P=0.001) and 20.2 anonymous reports/10,000 ED attendances and OBDs combined (95% CI 12.6 to 27.8 reports/10 000 ED attendances and OBDs combined; P=0.001). Within inpatient areas, the most significant improvement occurred in medical units, with an additional 84.5 reports/10 000 OBDs (95% CI 24.9 to 144.1 reports/10,000 OBDs). The intervention was not able to demonstrate improved reporting in the ICUs. There was heterogeneity between individual units, with rates in medical and surgical intervention units (n = 6) ranging from 113 to 431 reports/10,000 OBDs.
Bilimoria 2009
Before introduction of the reminder system, there was a very small increase in the number of reported events each month which was statistically significant (pre-slope = 0.50, P = 0.008). After the introduction of the reminder system at one, three, six and eight months there was a statistically significant decrease in the mean number of reporting events (respectively four, five, six and six events).
DISCUSSION
Summary of main results
Four studies (one CBA and three ITS studies) met our inclusion criteria and were included in this review (Bilimoria 2009; Dixon 2002; Evans 2007; Stump 2000). The CBA study (Evans 2007) showed a significant improvement in incident reporting rates after the introduction of the new reporting system. Just one of the ITS studies (Stump 2000) showed a statistically significant effectiveness of the new reporting system from nine months. The other two studies (Bilimoria 2009; Dixon 2002) reported no statistically significant improvements; one showed statistically significant decreases in recording (Bilimoria 2009).
Overall completeness and applicability of evidence
The retrieved evidence is incomplete and of limited applicability. With only four studies investigating the introduction of quite different, complex multiple interventions it is difficult to draw any clear conclusions about either their effectiveness or their generalisability.
The studies reported a limited range of outcome measures. In all of the included studies the ‘incident reporting rate’ was measured. Moreover in Dixon’s study the authors measured also the ‘time to report’ and in Evans’ study the ‘change in type of incident’ was reported, but neither study identified any direct or indirect costs of implementing their systems. The point of increasing incident reporting is to make healthcare safer (that is for reports to identify harm or potential harm that can be avoided), but none of the included studies directly explored this dimension. Stump et al describe a number of medication errors that were identified and improved following the new reporting system, but numerical data were not reported. The other studies did not mention improvements in patient safety after implementation of their system.
Quality of the evidence
The evidence that we identified has to be regarded as sparse and susceptible to bias. The re analysed ITS scored badly on the ITS risk of bias criteria at least in part because the authors never intended it to be analysed as an ITS. The CBA was scored as unclear on several of the risk of bias criteria.
Potential biases in the review process
The searches were performed at two points in time and by two different review teams and due to lack of information on the first search we could not complete a PRISMA flow chart that covered the whole search period. If both searches had been done by the same team it is possible different studies could have been identified. There is also the risk of publication bias, but as to too few studies were identified for inclusion in this review, we could not assess it.
Agreements and disagreements with other studies or reviews
We are not aware of any other systematic reviews on this topic.
AUTHORS’ CONCLUSIONS
Implications for practice
Because of the limitations of the studies it is not possible to draw conclusions for clinical practice. Facilities, practices or individuals introducing a system into practice should give careful consideration to conducting an evaluation using a robust design.
Implications for research
RCTs are likely to be challenging to conduct in this area. Finding sufficient units to randomise and suitable control groups will be challenging. However, time series designs can lend themselves well to this type of intervention and should be designed as robustly as possible.
Whichever study design is used, there are a number of specific factors that need to be taken into account. Standardised definitions of clinical incidents (within and ideally between trials) should be used. The features of the reporting methods should be clearly described, including mechanism (electronic or manual), anonymity, accessibility, and ease of input. An appropriate range of outcome measures should be reported including views of users (feedback and user friendliness) and appropriate measures of resource use (to allow robust economic analyses).
PLAIN LANGUAGE SUMMARY.
Are interventions to improve clinical incidence reporting effective?
A clinical incident is “any occurrence which is not consistent with the routine care of the patient or the routine operation of a healthcare institution”. To prevent clinical incidents, it is important to promote a safety culture among clinicians, healthcare professionals and patients, but also to identify and learn more from errors in order to improve the care process and the organisational support within healthcare institutions. Therefore, it is important to review the effectiveness of incident reporting systems aimed at increasing the number of clinical incident reported by health professionals.
We searched the scientific literature for studies that evaluated the effectiveness of interventions to increase clinical incident reporting in healthcare. We found four studies using different types of designs and which had several methodological shortcomings. The studies used different interventions and their results are heterogeneous: in two studies there was a significant increase in the reporting rate of clinical incidents.
The results of this review show that rigorous evidence to demonstrate the effectiveness of this type of intervention is lacking, so it is not possible to draw conclusions for clinical practice.
ACKNOWLEDGEMENTS
The generous funding from NIHR is gratefully acknowledged. We would also like to acknowledge the assistance of Fiona Beyer in developing the search strategies.
SOURCES OF SUPPORT
Internal sources
Newcastle University, UK.
University of Modena and Reggio Emilia, Italy.
External sources
National Institute of Health Research, UK.
Appendix 1. Search Strategies
MEDLINE (OVID) and Cochrane Library (Wiley)
exp Risk Management/
exp Medical Errors/
exp safety/
Medication Errors/
or/1-4
disclosure/
reporting.tw.
6 or 7
5 and 8
Incident Report$.tw.
adverse event report$.tw.
voluntary report$.tw.
mandatory report$.tw.
error report$.tw.
or/10-14
9 or 15
combined with EPOC methods filter for MEDLINE
992 results 2007-date
EMBASE (OVID)
exp Risk Management/
exp Medical Error/
exp safety/
Medication Error/
or/1-4
(reporting or disclosure).tw.
5 and 6
(Incident Report$ or adverse event report$ or voluntary report$ or mandatory report$ or error report$).tw.
7 or 8
combined with EPOC methods filter for EMBASE
923 results 2007-date
CINAHL
Risk Management [NT Incident reports]
(MH “Risk Management+”)
(MH “Medication Errors”)
(MH “Safety+”)
1 or 2 or 3
(MH “Truth Disclosure”)
TI reporting or AB reporting
5 or 6
4 and 7
TI (Incident Report* or adverse event report* or voluntary report* or mandatory report* or error report*) OR AB (Incident Report* or adverse event report* or voluntary report* or mandatory report* or error report*)
8 or 9
combined with EPOC methods filter for CINAHL
316 results 2007-date
WoK (SCI, SSCI, Conference Proceedings)
TS=(medical SAME error*) OR TS=(medication SAME error*) OR (TS=“risk management”)
TS=disclosure OR TS=reporting
2 AND 1
TS=(“Incident Report*”) OR TS=(“adverse event report*”) OR TS=(“voluntary report*”) OR TS=(“mandatory report*”) OR TS=(“error report*”)
4 OR 3
TS=(random* or factorial* or controlled or evaluat* or trial* or experiment* or study or studies or design or crossover* or cross-over* or placebo* or assign* or volunteer* or intervention* or effect* or compar* or chang* or impact*) or TS=(“cross over”) or TS=(“time series”)
5 and 6
SUMMARY OF FINDINGS FOR THE MAIN COMPARISON
| Modified system compared with an established on for increasing clinical incident reporting | |||
| Patient or population: healthcare professionals | |||
| Settings: public or private healthcare organisation | |||
| Intervention: intervention aimed at increasing clinical incident reporting with the introduction of a modified system of data collection | |||
| Comparison: established system | |||
| Outcomes | No of Participants (studies) | Quality of the evidence (GRADE) | Comments |
| Incident reporting rate | 4 studies | ⊕○○○ very low |
Evidence incomplete and of limited applicability. With only four studies investigating the introduction of quite different, complex multiple interventions it is difficult to draw any clear conclusions about either their effectiveness, or their generalisability |
GRADE Working Group grades of evidence
High quality: Further research is very unlikely to change our confidence in the estimate of effect.
Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low quality: We are very uncertain about the estimate.
CHARACTERISTICS OF STUDIES
Characteristics of included studies [ordered by study ID]
| Methods | Re-analysed as an ITS design. One site, no control Weekly reminder introduced in January 2007 until August 2007 |
|
| Participants | Surgery department in a metropolitan tertiary care centre in the USA | |
| Interventions | Online morbidity, mortality and anonymous near misses reporting system. For this review we were interested in one component of the intervention - the introduction of a weekly reminder to report complications and near-misses events | |
| Outcomes | Incident reporting rate | |
| Notes | ||
| Risk of bias | ||
| Bias | Authors’ judgement | Support for judgement |
| Incomplete outcome data (attrition bias) All outcomes |
Unclear risk | Not specified in the paper |
| Selective reporting (reporting bias) | Unclear risk | The outcome are not mentioned in the method section |
| Other bias | Low risk | No evidence of other risk of bias |
| Other changes | Unclear risk | Not specified in the paper |
| Shape of the intervention | High risk | The intervention started in July 2005 and the analysis is from September 2005 to August 2007 |
| Data collection | Low risk | The intervention itself included a change in the data collection method |
| Allocated intervention | Low risk | Objective outcome measure |
| Methods | Re-analysed as an ITS design. One site, no control Intervention implemented from July 2000 |
|
| Participants | Four administratively linked hospitals in Dallas, Texas | |
| Interventions | Web-based “patient occurrence system” not limited to medical error | |
| Outcomes | Incident reporting rate | |
| Notes | ||
| Risk of bias | ||
| Bias | Authors’ judgement | Support for judgement |
| Incomplete outcome data (attrition bias) All outcomes |
Unclear risk | Not specified in the paper |
| Selective reporting (reporting bias) | Low risk | All relevant outcome reported |
| Other bias | Low risk | No evidence of other risk of bias |
| Other changes | Unclear risk | Not specified in the paper |
| Shape of the intervention | Unclear risk | It is not clear what is the point of analysis |
| Data collection | Low risk | The intervention itself included a change in the data collection method |
| Allocated intervention | Low risk | Objective outcome measure |
| Methods | CBA design Intervention run from June to August 2003, with units implementing it over a 40-week period |
|
| Participants | Nursing staff working in 20 units in four major cities and two regional hospitals of South Australia | |
| Interventions | Intense education, a range of reporting options, changes in report management and enhanced feedback | |
| Outcomes | Incident reporting rate | |
| Notes | ||
| Risk of bias | ||
| Bias | Authors’ judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Non random allocation |
| Allocation concealment (selection bias) | Unclear risk | Not specified in the paper |
| Incomplete outcome data (attrition bias) All outcomes |
Unclear risk | Not specified in the paper |
| Selective reporting (reporting bias) | Low risk | All outcomes analysed. |
| Other bias | Low risk | None identified. |
| Allocated intervention | Low risk | Objective outcome data |
| Baseline outcome measurement similar? | High risk | See Table 2 |
| Baseline characteristics similar? | High risk | See Table 1 |
| Protection against contamination? | Low risk | No evidence of contamination |
| Methods | Re-analysed as an ITS design. One site, no control |
|
| Participants | A 944-bed, tertiary-care teaching hospital affiliated with Yale University (USA) | |
| Interventions | A standardised, non-punitive system focused on the medication-use system, competency assessment and reporting incentives | |
| Outcomes | Incident reporting rate | |
| Notes | ||
| Risk of bias | ||
| Bias | Authors’ judgement | Support for judgement |
| Incomplete outcome data (attrition bias) All outcomes |
Unclear risk | Not specified in the paper |
| Selective reporting (reporting bias) | Low risk | Only one primary outcome which was reported |
| Other bias | Low risk | No evidence of other risk of bias |
| Other changes | Unclear risk | Not specified in the paper |
| Shape of the intervention | Unclear risk | The type of analysis is not mentioned in the paper |
| Data collection | Low risk | The intervention itself included a change in the data collection method |
| Allocated intervention | Low risk | Objective outcome measure |
CBA - controlled before-after studies
ITS - interrupted time series analyses
Characteristics of excluded studies [ordered by study ID]
| Study | Reason for exclusion |
|---|---|
| Abstoss 2011 | No baseline data. |
| Antonacci 2008 | Not experimental |
| Askeland 2008 | One-site CBA |
| Auerbach 2012 | Not about improving incidence reporting. No outcome of interest |
| Blaya 2010 | Not about increasing clinical incidence reporting. |
| Boyer 2009 | No introduction of a new system |
| Charpentier 2011 | Not about how to increase clinical incident reporting. |
| Costello 2007 | One-site CBA |
| Duckers 2009 | A systematic review. |
| Force 2006 | Not enough data points to be included as an ITS |
| Fukuda 2008 | One-site CBA |
| Grant 2007 | One-site CBA |
| Hall 2010 | A systematic review. |
| Haller 2007 | Not experimental |
| Jansma 2011 | CBA with only one intervention and one control site. |
| King 2010 | A review paper. |
| Kivlahan 2002 | Not experimental |
| Lehmann 2007 | About drugs harm only |
| McGraw 2011 | Not about increasing the clinical incidence reporting. |
| Mick 2007 | Not experimental |
| Munson 2010 | No baseline data. |
| Norden-Hagg 2012 | Uncontrolled before and after study that could not be re-analysed as a time series |
| Park 2008 | One-site CBA |
| Perez Blanco 2009 | Not experimental |
| Relihan 2009 | Not experimental |
| Taylor 2007 | One-site CBA |
| Tuttle 2004 | Not enough data points to be included as an ITS |
| Wagner 2008 | Not on improving incident reporting rate |
| Walsh 2008 | No data on reporting, only on identifed errors in database |
| Weingart 2009 | No introduction of a new system |
| Welsh 1996 | No clearly defined point in time when the intervention occurred |
| Zingg 2007 | Not experimental |
| Zwart 2011a | Uncontrolled before and after study that was no re-analysable as a time series |
| Zwart 2011b | Uncontrolled before and after study that was no re-analysable as a time series |
CBA - controlled before-after studies
ITS - interrupted time series analyses
DATA AND ANALYSES
This review has no analyses.
HISTORY
Protocol first published: Issue 1, 2006
Review first published: Issue 8, 2012
DIFFERENCES BETWEEN PROTOCOL AND REVIEW
We didn’t evaluate the cost of the intervention.
Footnotes
DECLARATIONS OF INTEREST
None known
References to studies included in this review
- Bilimoria 2009 {published data only} .Blimoria KY, Kmiecik TE, DaRosa DA, Halverson A, Eskandari MK, Bell RH, Jr, et al. Development of an online morbidity, mortality and near-miss reporting system to identify patterns of adverse events in surgical patients. Archives of Surgery. 2009;144(4):305–11. doi: 10.1001/archsurg.2009.5. [DOI] [PubMed] [Google Scholar]
- Dixon 2002 {published data only} .Dixon JF. Going paperless with custom-built web-based patient occurrence reporting. Journal on Quality Improvement. 2002;28(7):387–95. doi: 10.1016/s1070-3241(02)28038-4. [DOI] [PubMed] [Google Scholar]
- Evans 2007 {published data only} .Evans SM, Smith BJ, Estherman A, Runciman WB, Maddern G, Stead K, et al. Evaluation of an intervention aimed at improving voluntary incident reporting in hospitals. Quality & Safety in Health Care. 2007;16:169–75. doi: 10.1136/qshc.2006.019349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stump 2000 {published data only} .Stump LS. Re-engineering the medication error-reporting process: removing the blame and improving the system. American Journal of Health-System Pharmacy. 2000;57(Suppl 4):S10–7. doi: 10.1093/ajhp/57.suppl_4.S10. [DOI] [PubMed] [Google Scholar]
References to studies excluded from this review
- Abstoss 2011 {published data only} .Abstoss KM, Shaw BE, Owens TA, Juno JL, Commiskey EL, Niedner MF. Increasing medication error reporting rates while reducing harm through simultaneous cultural and system-level interventions in an intensive care unit. BMJ Quality & Safety. 2011;20(11):914–22. doi: 10.1136/bmjqs.2010.047233. [DOI] [PubMed] [Google Scholar]
- Antonacci 2008 {published data only} .Antonacci AC, Lam S, Lavarias V, Homel P, Eavey RD. Benchmarking surgical incident reports using a database and a triage system to reduce adverse outcomes. Archives of Surgery. 2008;143(12):1192–7. doi: 10.1001/archsurg.143.12.1192. [DOI] [PubMed] [Google Scholar]
- Askeland 2008 {published data only} .Askeland RW, McGrane S, Levitt JS, Dane SK, Greene DL, Vandeberg JA, et al. Improving transfusion safety: implementation of a comprehensive computerized bar code-based tracking system for detecting and preventing errors. Transfusion. 2008;48(7):1308–17. doi: 10.1111/j.1537-2995.2008.01668.x. [DOI] [PubMed] [Google Scholar]
- Auerbach 2012 {published data only} .Auerbach AD, Sehgal NL, Blegen MA, Maselli J, Alldredge BK, Vittinghoff E, Wachter RM. Effects of a multicentre teamwork and communication programme on patient outcomes: results from the Triad for Optimal Patient Safety (TOPS) project. BMJ Quality & Safety. 2012;21(2):118–26. doi: 10.1136/bmjqs-2011-000311. [DOI] [PubMed] [Google Scholar]
- Blaya 2010 {published data only} .Blaya JA, Shin SS, Yale G, Suarez C, Asencios L, Contreras C, et al. Electronic laboratory system reduces errors in National Tuberculosis Program: a cluster randomized controlled trial. International Journal of Tuberculosis & Lung Disease. 2010;14(8):1009–15. [PMC free article] [PubMed] [Google Scholar]
- Boyer 2009 {published data only} .Boyer R, McPherson ML, Deshpande G, Smith SW. Improving medication error reporting in hospice care. American Journal of Hospice & Palliative Care. 2009;26(5):361–7. doi: 10.1177/1049909109335145. [DOI] [PubMed] [Google Scholar]
- Charpentier 2011 {published data only} .Charpentier C, Chevalier N, Rajezakowski S, Penavayre M, Chenevier D. Evaluation of a computerized system for medication errors reporting. International Journal of Clinical Pharmacy. 2011;33(2):357. [Google Scholar]
- Costello 2007 {published data only} .Costello JL, Torowicz DL, Yeh TS. Effects of a pharmacist-led pediatrics medication safety team on medication-error reporting. American Journal of Health System Pharmacy. 2007;64(13):1422–6. doi: 10.2146/ajhp060296. [DOI] [PubMed] [Google Scholar]
- Duckers 2009 {published data only} .Duckers M, Faber M, Cruijsberg J, Grol R, Schoonhoven L, Wensing M. Safety and risk management interventions in hospitals: a systematic review of the literature. Medical Care Research & Review. 2009;66(6):90S–119S. doi: 10.1177/1077558709345870. Suppl. [DOI] [PubMed] [Google Scholar]
- Force 2006 {published data only} .Force MV, Deering L, Hubbe J, Andersen M, Hagemann B, Cooper-Hahn M, Peters W. Effective strategies to increase reporting of medication errors in hospitals. The Journal of Nursing Administration. 2006;36(1):34–41. doi: 10.1097/00005110-200601000-00009. [DOI] [PubMed] [Google Scholar]
- Fukuda 2008 {published data only} .Fukuda H, Imanaka Y, Hirose M, Hayashida K. Evaluation of the impact of patient safety activities on the number of voluntary incident reports at teaching hospitals in Japan. Value in Health. 2008;Vol. 11(issue 3):A52. [Google Scholar]
- Grant 2007 {published data only} .Grant MJ, Larsen GY. Effect of an anonymous reporting system on near-miss and harmful medical error reporting in a pediatric intensive care unit. Journal of Nursing Care Quality. 2007;22(3):213–21. doi: 10.1097/01.NCQ.0000277777.35395.e0. [DOI] [PubMed] [Google Scholar]
- Hall 2010 {published data only} .Hall J, Peat M, Birks Y, Golder S, Entwistle V, Gilbody S, et al. Effectiveness of interventions designed to promote patient involvement to enhance safety: a systematic review. Quality & Safety in Health Care. 2010;19(5):1–7. doi: 10.1136/qshc.2009.032748. [DOI] [PubMed] [Google Scholar]
- Haller 2007 {published data only} .Haller G, Myles PS, Stoelwinder J, Langley M, Anderson H, McNeil J. Integrating incident reporting into an electronic patient record system. Journal of American Medical Informatics Association. 2007;14(2):175–81. doi: 10.1197/jamia.M2196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jansma 2011 {published data only} .Jansma JD, Wagner CK, Reinier W, Bijnen AB. Effects on incident reporting after educating residents in patient safety: a controlled study. BMC Health Services Research. 2011;11:335. doi: 10.1186/1472-6963-11-335. [DOI] [PMC free article] [PubMed] [Google Scholar]
- King 2010 {published data only} .King A, Daniels J, Lim J, Cochrane DD, Taylor A, Ansermino JM. Time to listen: a review of methods to solicit patient reports of adverse events. Quality & Safety in Health Care. 2010;19(2):148–157. doi: 10.1136/qshc.2008.030114. [DOI] [PubMed] [Google Scholar]
- Kivlahan 2002 {published data only} .Kivlahan C, Sangster W, Nelson K, Buddenbaum J, Lobenstein K. Developing a comprehensive electronic adverse event reporting system in an academic health center. The Joint Commission Journal on Quality Improvement. 2002;28(11):583–94. doi: 10.1016/s1070-3241(02)28062-1. [DOI] [PubMed] [Google Scholar]
- Lehmann 2007 {published data only} .Lehmann DF, Page N, Kirschman K, Sedore A, Guharoy R, Medicis J, Ploutz-Snyder R, Weinstock RS, Duggan DB. Every error a treasure: improving medication use with a nonpunitive reporting system. The Joint Commission Journal on Quality and Patient Safety. 2007;33(7):401–7. doi: 10.1016/s1553-7250(07)33046-8. [DOI] [PubMed] [Google Scholar]
- McGraw 2011 {published data only} .McGraw C, Topping C. The District Nursing Clinical Error Reduction Programme. British Journal of Community Nursing. 2011;16(1):135–40. doi: 10.12968/bjcn.2011.16.1.35. [DOI] [PubMed] [Google Scholar]
- Mick 2007 {published data only} .Mick JM, Wood GL, Massey RL. The good catch pilot program: increasing potential error reporting. The Journal of Nursing Administration. 2007;37(11):499–503. doi: 10.1097/01.NNA.0000295611.40441.1b. [DOI] [PubMed] [Google Scholar]
- Munson 2010 {published data only} .Munson GW, Prabhakar S, Francis DL. Patient self-report cards to track post-endoscopy complications: The Mayo Clinic Rochester experience. Gastroenterology. 2010;1:S323. [Google Scholar]
- Norden-Hagg 2012 {published data only} .Norden-Hagg A, Kettis-Lindblad A, Ring L, Kalvemark-Sporrong S. Experiences of a nationwide web-based system: reporting dispensing errors in Swedish pharmacies. International Journal of Pharmacy Practice. 2012;20(1):25–32. doi: 10.1111/j.2042-7174.2011.00159.x. [DOI] [PubMed] [Google Scholar]
- Park 2008 {published data only} .Park CS, Kim TB, Kim SL, Kim JY, Yang KA, Bae YJ, et al. The use of an electronic medical record system for mandatory reporting of drug hypersensitivity reactions has been shown to improve the management of patients in the university hospital in Korea. Pharmacoepidemiology and Drug Safety. 2008;17(9):919–25. doi: 10.1002/pds.1612. [DOI] [PubMed] [Google Scholar]
- Perez Blanco 2009 {published data only} .Perez Blanco V, Rubio Gomez I, Alarcon Gascuena P, Mateos Rubio J, Herradon Cano M, Delgado Garcia A. Implementation of a form for adverse effect notification: results for the 1st year. Revista de Calidad Asistential. 2009;24(1):3–10. doi: 10.1016/S1134-282X(09)70069-7. [DOI] [PubMed] [Google Scholar]
- Relihan 2009 {published data only} .Relihan E, Silke B, O’Grady F. Internally-developed electronic reporting system for medication errors. Irish Medical Journal. 2009;102(7):223–4. [PubMed] [Google Scholar]
- Taylor 2007 {published data only} .Taylor JA, Brownstein D, Klein EJ, Strandjord TP. Evaluation of an anonymous system to report medical errors in pediatric inpatients. Journal of Hospital Medicine. 2007;2(4):226–33. doi: 10.1002/jhm.208. [DOI] [PubMed] [Google Scholar]
- Tuttle 2004 {published data only} .Tuttle D, Holloway R, Baird T, Sheehan B, Skelton WK. Electronic reporting to improve patient safety. Quality and Safety in Health Care. 2004;13(4):281–86. doi: 10.1136/qshc.2003.009100. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wagner 2008 {published data only} .Wagner LM, Capezuti E, Clark PC, Parmelee PA, Ouslander JG. Use of a falls incident reporting system to improve care process documentation in nursing homes. Quality and Safety in Health Care. 2008;17(2):104–8. doi: 10.1136/qshc.2007.022947. [DOI] [PubMed] [Google Scholar]
- Walsh 2008 {published data only} .Walsh KE, Landrigan CP, Adams WG, Vinci RJ, Chessare JB, Cooper MR, et al. Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatrics. 2008;121(3):e421–7. doi: 10.1542/peds.2007-0220. [DOI] [PubMed] [Google Scholar]
- Weingart 2009 {published data only} .Weingart SN, Price J, Duncombe D, Connor M, Conley K, Conlin GJ, et al. Enhancing safety reporting in adult ambulatory oncology with a clinician champion: a practice innovation. Journal of Nursing Care Quality. 2009;24(3):203–10. doi: 10.1097/NCQ.0b013e318195168d. [DOI] [PubMed] [Google Scholar]
- Welsh 1996 {published data only} .Welsh CH, Pedot R, Anderson RJ. Use of morning report to enhance adverse event detection. Journal of General Internal Medicine. 1996;11(8):454–60. doi: 10.1007/BF02599039. [DOI] [PubMed] [Google Scholar]
- Zingg 2007 {published data only} .Zingg U, Zala-Mezo E, Kuenzli B, Maron C, Licht A, Metzger U, Platz A. Implementation and evaluation of the Critical Incident Reporting System (CIRS) in general surgery. British Journal of Surgery. 2007;Vol. 94(issue 6):772–3. [Google Scholar]
- Zwart 2011a {published data only} .Zwart DLM, Van Rensen ELJ, Kalkman CJ, Verheij TJM. Central or local incident reporting? A comparative study in Dutch GP out-of-hours services. British Journal of General Practice. 2011;61(584):183–7. doi: 10.3399/bjgp11X561168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zwart 2011b {published data only} .Zwart LM, Steerneman AHM, van Rensen ELJ, Kalkman CJ, Verheij TJM. Feasibility of centre-based incident reporting in primary healthcare: the SPIEGEL study. BMJ Quality & Safety. 2011;20(2):121–7. doi: 10.1136/bmjqs.2009.033472. [DOI] [PubMed] [Google Scholar]
Additional references
- Brennan 1991 .Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study. New England Journal of Medicine. 1991;324(6):370–6. doi: 10.1056/NEJM199102073240604. [DOI] [PubMed] [Google Scholar]
- Elwyn 2005 .Elwyn G, Corrigan JM. The patient safety story. BMJ. 2005;331(7512):302–4. doi: 10.1136/bmj.38562.690104.43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- EPOC 2011 .Effective Practice and Organisation of Care Group EPOC. Website. http://epoc.cochrane.org/epoc-resources-review-authors.
- Higgins 2008 .Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Intervention. John Wiley & Sons; Chichester: 2008. [Google Scholar]
- Kohn 2000 .Kohn LT, Corrigan J, Donaldson MS. Institute of Medicine (U.S.) Vol. 6. Committee on Quality of Health Care in America; 2000. To err is human: building a safer health system. [PubMed] [Google Scholar]
- Leap 1994 .Leape LL. Error in medicine. JAMA. 1994;272(23):1851–7. [PubMed] [Google Scholar]
- Leape 2002 .Leape LL. Reporting of adverse events. New England Journal of Medicine. 2002;347(20):1633–8. doi: 10.1056/NEJMNEJMhpr011493. [DOI] [PubMed] [Google Scholar]
- Liberati 2009 .Liberati A, Altman DG, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):1–27. doi: 10.1371/journal.pmed.1000100. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McDonald 2000 .McDonald CJ, Weiner M, et al. Deaths due to medical errors are exaggerated in Institute of Medicine report. JAMA. 2000;284(1):93–5. doi: 10.1001/jama.284.1.93. [DOI] [PubMed] [Google Scholar]
- NHS 2000 .NHS An organisation with a memory: report of an expert group on learning from adverse events in the NHS. Department of Health Chief Medical Officer of Health (UK) 2000 [Google Scholar]
- Ramsay 2003 .Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE. Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. International Journal of Technology Assessment in Health Care. 2003;19(4):613–23. doi: 10.1017/s0266462303000576. [DOI] [PubMed] [Google Scholar]
- RevMan 2011 .The Nordic Cochrane Centre . Review Manager 5.1. The Cochrane Collaboration; Copenhagen: 2011. [Google Scholar]
- Thomas 1999 .Thomas EJ, Studdert DM, Newhouse JP, Zbar BI, Howard KM, Williams EJ, Brennan TA. Costs of medical injuries in Utah and Colorado. Inquiry. 1999;36(3):255–64. [PubMed] [Google Scholar]
- * Indicates the major publication for the study