National and Institutional Trends in Adverse Events Over Time: A Systematic Review and Meta-analysis of Longitudinal Retrospective Patient Record Review Studies

Supplemental digital content is available in the text.

Objective

This study aimed to determine if the implementation of large-scale patient safety initiatives have been successful in reducing overall and preventable adverse event rates in hospital inpatients.

Design

The design used in this study was systematic review and meta-analysis.

Data Resources

We followed our published protocol (PROSPERO [CRD42019140058]) and searched the following databases: PubMed, CINAHL, PsycINFO, Cochrane Library, and Embase from inception to February 2020. The reference lists of eligible studies were also searched.

Eligibility

All longitudinal retrospective record review studies that examined adverse event rates before and after the introduction of patient safety initiatives in hospital inpatients were included.

Data Extraction

Data extraction, quality, and risk of bias assessment were carried out by 2 independent reviewers. Information on study design, setting, demographics, interventions, and safety outcome measures was extracted.

Results

A total of 3894 articles were screened, and 7 articles met the eligibility criteria for our systematic review with 5 of these providing sufficient information for inclusion in the meta-analysis. The degree of heterogeneity was high among studies. The meta-analysis demonstrated a minimal risk reduction in overall adverse event rates of 0.017 (95% confidence interval, 0.002–0.032) when the lower-quality studies were excluded, with one adverse event being prevented for every 59 hospital admissions.

Conclusions

These findings are significant when the large numbers of admissions to a hospital every year are considered. Given the low numbers of large-scale implementation studies, there is a need for more research on the effectiveness of patient safety initiatives to further assess the impact of such initiatives on adverse events.

A primary responsibility of health care providers is to “first do no harm,” a principle highlighted in the seminal study on adverse events “To Err Is Human.” Published in 1999, it estimated that 98,000 inpatients in the United States die of adverse events each year.¹

Several articles have evaluated the impact of specific patient safety interventions on reducing specific adverse events and preventable deaths in hospital. Targeted interventions have been shown to decrease the adverse event of interest, for example, multicomponent interventions to reduce falls risk,² interventions to prevent delirium,³ an early warning system to mitigate the mortality from cardiopulmonary arrest,⁴ pharmacist reconciliation to reduce medication-related adverse events,⁵ and surgical checklists to minimize infection and perioperative mortality rates.⁶ Although there is a large amount of evidence for a range of patient safety initiatives, it remains unclear whether the aggregate efforts of such initiatives cause improvements in overall adverse event rates at a national or institutional level.

The Organisation for Economic Co-operation and Development’s policy brief on the economic impact of patient safety states that “the cost of failure dwarfs the investment required to implement effective [adverse event] prevention” and emphasizes the use of evidence-based initiatives to reduce adverse events and the importance of monitoring such efforts over time.⁷ This systematic review and meta-analysis aim to determine if evidence has been translated into practice and whether health care services and institutions have been successful in reducing adverse event rates as a result. The objectives of this systematic review and meta-analysis were to determine if the implementation of patient safety initiatives on a large-scale has been successful in reducing overall adverse event rates and preventable adverse event rates in hospital settings.

METHODS

The study was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis group guidelines.⁸ The study protocol was registered with PROSPERO (CRD42019140058).

Scope of the Review

Studies included compared overall adverse event rates before and after a patient safety initiative was introduced at an institutional or national level.

Data Sources

An electronic search of the PubMed, CINAHL, PsycoINFO, Cochrane and Embase databases from inception to June 2019 (and later updated to include results up to February 2020) was conducted.

All fields were searched using terms related to adverse events and hospital inpatient population. To identify studies describing adverse events, the search terms’ “adverse event,” “medical error,” “iatrogenic disease,” “patient safety,” “critical incident,” “undesirable outcome,” “clinical incident,” “iatrogenic injury,” and “sentinel event’ were combined. Search terms to identify inpatients combined “inpatient,” “hospital patient,” and “hospitalization,” and terms to identify chart review combined “chart review” and “record review.” The terms “Harvard Medical Practice” and “Trigger Tool” were also combined, as these terms capture methodologies commonly used to collect adverse event data. The search strategy is shown in Appendix 1 (http://links.lww.com/JPS/A368) and was translated for other databases mentioned, as appropriate. Review articles, conference abstracts, gray literature, and dissertations were excluded. The search was supplemented with a manual search of the bibliography of eligible studies.

Study Selection

Inclusion Criteria

All longitudinal studies examining adverse event rates before and after the implementation of patient safety initiatives in the inpatient population were included. Retrospective chart review studies have been shown to be effective in detecting adverse events and are currently considered the “gold standard” for identifying adverse event rates.⁹ The outcome measures of interest for the meta-analysis and systematic review were a change in the prevalence rates of adverse events after an intervention. Where exact data for this outcome were not described, the authors were contacted and this information was requested.

Exclusion Criteria

We excluded studies that took place in the community, primary health care services, and nursing home settings only and studies that took place in multiple patient settings from which information on inpatients could not be disaggregated. Studies examining specific patient populations (e.g., dialysis patients only) and studies evaluating specific types of adverse events (e.g., diagnostic adverse events only) were excluded. However, we did not exclude studies that included such subgroups as part of the general inpatient population studied. Eligibility for inclusion/exclusion is shown in Appendix 2 (http://links.lww.com/JPS/A368).

Data Extraction and Analysis

Search results were exported to Endnote¹⁰ and then to Covidence software,¹¹ which allowed for the automated removal of duplicate articles. Two reviewers (W.C. and B.L.) independently screened the titles and abstracts of all the records identified and then reviewed the full texts of potentially eligible publications. Any disagreements were resolved by discussion between the 2 reviewers. A third reviewer (N.R.) was available to resolve any issues in the case of no consensus. The following information was extracted: setting, sample size, description of the intervention, overall and preventable adverse event rates before intervention, overall and preventable adverse event rates after the intervention, and adverse event outcomes.

Quality Assessment and Risk of Bias

The Crowe Critical Appraisal Tool (CCAT) was used to determine study quality.¹² This tool was used by 2 reviewers (W.C. and B.L.) independently to form a score out of a maximum of 40. This tool is designed to be applied to a wide range of study designs, including observational and interventional studies, and has undergone testing for reliability and validity.^13,14 Each research design is appraised on its own merits and “not relative to some preconceived notion of hierarchy.” There is therefore no criterion standard, and scores assigned do not equate to prespecification of “poor,” “moderate,” and “high.”¹² Instead, articles are ranked relative to each other.

The final score assigned was the average of the 2 appraiser scores (scores containing 0.5 were rounded up or down based on consensus). The intraclass correlation coefficient was used to evaluate consistency in quality scores between raters. For example, <0.5 indicates poor; 0.5–0.749, moderate; 0.75–0.9, good; and >0.9, excellent.¹⁵ The data were then stratified by tertiles of the quality score (i.e., first tertile, lower quality; second tertile, medium quality; and third tertile, higher quality). This method has been validated previously.¹⁶

The CCAT also assesses the degree of bias. Each article was initially assessed for risk of bias using the Cochrane Collaboration’s tool for assessing the risk of bias.¹⁷ This assessment informed the overall CCAT quality score assessment. A risk of bias summary figure that presents all judgments and a cross-tabulation of studies was generated on Cochrane’s Review Manager (RevMan V5.3).¹⁸

Data Analysis and Synthesis

A systematic synthesis was carried out on the eligible studies describing the safety initiatives and their impact. Where a confidence interval (CI) or P value is omitted in this description, it is due to the information not being present in the original article. Authors of eligible articles reported their findings using a range of unit measures (Appendix 3, http://links.lww.com/JPS/A368). Each measurement was translated into a single unit: risk difference using the Review Manager software. A meta-analysis describing risk difference rates was conducted based on data extracted. A meta-analysis was carried out using the Review Manager software and calculated using a Mantel-Haenszel random-effects model with 95% CIs. A sensitivity analysis was performed by repeating the meta-analysis in the absence of the first tertile (lower quality) studies. Numbers needed to treat (NNTs) were generated from the risk difference.

Heterogeneity was measured using the I² statistic facilitated by the Review Manager software. The ranges of I² (heterogeneity) were classified in accordance with the Cochrane handbook.¹⁹ Funnel plots were constructed using the Review Manager software, and Egger test was computed on Stata V.16²⁰ to help determine publication bias. The intraclass correlation coefficient was calculated using the Stata software.

RESULTS

A total of 4719 studies were identified by searching the databases, of which 825 were identified by the Convidence software as duplicate entries. There were 3894 studies that were screened by abstract and title. The texts of 44 articles underwent full-text review, and of these, 36 were excluded (Fig. 1). During the full-text review, 2 high-profile studies were excluded. The initial study by Baines et al²¹ was excluded, as it stated that the findings did not capture the impact of the patient safety intervention, the study by Benning et al²² was excluded because it only examined patients 65 years and older with respiratory conditions. No additional publications were retrieved by hand searching the references of eligible articles. One further study was excluded after personal correspondence with the author revealed that the study did not assess the impact of the described patient safety intervention (i.e., the patient safety interventions were implemented before the data collection had commenced).²³ Seven articles were included for the systematic review. Five of the 7 studies provided data for inclusion in the meta-analysis.^24–28

FIGURE 1.

Flow diagram of search results.

Characteristics of Included Studies

The studies included in this systematic review were based in Spain,²⁸ Sweden,^27,29 Italy,³⁰ the United States,²⁶ Norway,²⁵ and the Netherlands²⁴ (Table 1). The numbers of records reviewed in each study ranged from 960²⁹ to 64,917.²⁷ One study measured adverse event prevalence at 3 points in time (i.e., point prevalence).²⁴ The other 6 studies measured adverse event rates at monthly^26–29 or bimonthly^25,30 intervals over the duration of the intervention. The mean duration of the studies was 4.2 years (range, 3–6 years). All studies excluded psychiatric patients, and only one study included pediatric patients²⁴ (but excluded children younger than 1 year). Although all studies examined both medical and surgical patients, 4 of the studies^25–27,30 examined obstetric cases also (Appendix 4, http://links.lww.com/JPS/A368). Of the 7 articles included, 6 used the Global Trigger Tool method^25–30 and 1 used the Harvard Medical Practice method²⁴ of adverse event detection.

TABLE 1.

Description of Study Country, Setting, Sample Size, Frequency of Review, and Duration of the Review

Author (Year)	Country	Setting	Sample Size (Records Reviewed)	Method of Chart Review	Frequency of Review	Duration of Review	Duration of Review, y
Baines et al (2015)24	The Netherlands	2004: 21 hospitals, 2008: 22 hospitals, 2012: 20 hospitals	15,997	HMPS	On 3 occasions	Point prevalence: 2004, 2008, 2011/2012	4
Deilkås et al (2015)25	Norway	18–19 public hospitals and 5 private hospitals	40,851	GTT	Bimonthly review	2010–2013	4
Garrett et al (2013)26	United States	25 hospitals	17,295	GTT	Monthly review	2009–2011	3
Mortaro et al (2017)30	Italy	Single hospital	1320	GTT	Bimonthly review	2009–2014	5
Nilsson et al (2018)27	Sweden	63 hospitals	64,917	GTT	Monthly review	2013–2016	4
Rutberg et al (2014)29	Sweden	Single university hospital	960	GTT	Monthly review	2011–2013	3
Suarez et al (2014)28	Spain	Single hospital	1440	GTT	Monthly review	2009–2012	4

GTT, Global Trigger Tool; HMPS, Harvard Medical Practise Study.

Risk of Bias

In all studies, random allocation to the preintervention or postintervention group would not be possible, and it would not have been possible to blind the chart reviewers, thus introducing allocation and detection bias. Two^29,30 of the 7 articles did not document exact data on outcomes. Instead, they stated that the findings were nonsignificant. This may suggest a reporting bias in these articles. Appendix 5 (http://links.lww.com/JPS/A368) presents the risk of bias summary and the risk of bias graph reflecting the judgments described previously.

Quality Score

The average CCAT quality score was 30.1/40 (range, 27.5–37). Interrater agreement between reviewers was “good,” with an intraclass coefficient of 0.84.¹⁵ No article presented a flowchart. Four articles were deemed of poorer quality for not assessing ethical considerations^26,28–30 and 2 for not clearly defining their intervention.^26,30 Limited information on the intervention introduced was a key distinguishing factor of studies that were classified as being of lower quality.

Table 2 illustrates how each reviewer scored the article and the overall CCAT score agreed upon by the reviewers. The articles were then ranked 1 to 7 and inserted into the appropriate quality tertile as described earlier.

TABLE 2.

Individual Reviewers’ Quality Assessment Score, Overall Quality Assessment Score, and Ranking and Tertile Range of Quality Assessment Stop

Author (Year)	Crowe Assessment (Overall Score)	WC Crowe Score	BL Crowe Score	Rank	Tertiles
Baines et al (2015)24	37	38	36	1	3rd
Rutberg et al (2014)29	32	32	33	2	3rd
Nilsson et al (2018)27	31	31	31	3	2nd
Deilkås et al (2015)25	30	30	29	4	2nd
Mortaro et al (2017)30	28	28	30	5	1st
Garrett et al (2013)26	27	29	26	6	1st
Suarez et al (2014)28	26	26	29	7	1st

The Global Trigger Tool and the Harvard Medical Practise Study methodologies were used to determine the adverse event rates.

BL, Brian Li; WC, Warren Connolly.

Description of Patient Safety Interventions

A controlled before-and-after study by Deilkås et al²⁵ examined the effectiveness of the patient safety initiative “In Safe Hands.” which targeted the prevention of pressure ulcers, catheter-related urinary tract infections, central-line infections, overdose, deaths after discharge from an institution, falls, and postoperative infections during a 4-year period.³¹ The initiative was successful in decreasing overall adverse events rates by −3.1% (95% CI, −5.2% to −1.1%) from 16.1% (95% CI, 14.6% to 17.5%) in 2011 to 13% (95% CI, 11.7% to 14.2%) in 2013.

Nilsson et al²⁷ focused on the Swedish Government’s patient safety initiative for 4 years, which targeted medication-related adverse events, prevention of antibiotic resistance, and reduction of hospital-acquired infections. The authors concluded that the initiative resulted in a decrease in the proportion of admissions with adverse events classified as hospital-acquired infections, pneumonia, ventilator-associated pneumonia, and urinary bladder distension. It resulted in an overall statistically significant decrease in adverse event rates from 13.1% (95% CI, 12.7%–13.6%) of admissions having one or more adverse event in 2013 to 11.4% (95% CI, 10.9%–12.0%) in 2016.

Suarez et al²⁸ introduced initiatives targeting falls, lower bed height, pressure ulcers, medication errors, hospital-acquired infections, catheter-related infections, and promoting surgical safety improvement (surgical checklists) and patient safety awareness training. Adverse events decreased by 22.6% (risk reduction, 0.8; 95% CI, 0.66–0.97; P = 0.02), and severe adverse events also fell by more than half (risk reduction, 0.48; 95% CI, 0.24–0.96; P = 0.04).

Mortaro et al³⁰ stated that “appropriate improvement initiatives” were developed based on the findings of regular audits (but the scope of such initiatives was not described). The authors stated that adverse event rates did not show a substantial reduction during the entire study period (but these figures were not presented).

Rutberg et al²⁹ conducted a multifaceted initiative targeting infection control in relation to bladder catheterization, central venous line infections, education on hospital-acquired infections, and use of antibiotics. They also provided education on hand hygiene and rapid response teams for the early detection and treatment of sepsis. The authors stated that they did not see any reduction in the rate of adverse events.

Baines et al²⁴ evaluated the success of the national safety program “Patient Harm, Work Safely” as well as the national implementation of surgical checklists. The program involved preventing postoperative wound infections, early treatment for critically ill patients, prevention of renal failure from iodinated contrast agents, prevention of medication-related adverse events, prevention of line sepsis, safe patient transfer, and screening of vulnerable elderly patients (falls, poor nutrition, delirium). Overall adjusted adverse event rates remained similar (6% in 2008 [95% CI, 4.9%–7.3%] versus 5.7% in 2011/2012 [95% CI, 7.4%–6.2%], P = 0.68), as did preventable adverse event rates (2.0% [95% CI, 1.5%–2.8%] in 2008 versus 1.4% [95% CI, 0.9%–2.0%; P = 0.1] in 2011/2012).

Garrett et al²⁶ initiated system-wide collaborative improvement projects for glycemic management and pressure ulcers and found a significant reduction in overall adverse events (98 adverse events per 1000 patient days in the first 6 months of 2009 compared with 67 adverse events per 1000 patient days in the last 6 months of 2011).

Meta-analysis

Two studies (Rutberg et al²⁹ and Mortaro et al³⁰) could not be included in our meta-analysis, as they did not report sufficient before-and-after intervention adverse event data. The authors of these studies were contacted directly for these data, but we were unable to obtain further information. Therefore, 5 studies^24–28 were included in the meta-analysis. Four studies^25–28 reported an overall decrease in adverse event rates, and one study reported no statistically significant change in adverse event rates.²⁴ One study²⁴ provided weighted prevalence rates based on the study’s sampling frame, and we used these weighted rates for the meta-analysis.

The total number of records included in the meta-analysis was 69,062. The pooled risk difference in adverse event rates between the baseline and postintervention time periods across all included studies was 0.022 (95% CI, 0.017–0.027) in favor of the intervention (Fig. 2). There was a high level of heterogeneity (I² = 84% [P < 0.001]). The meta-analysis was repeated excluding the 2 lower-quality studies^26,28; in this analysis, the risk difference was 0.017 (95% CI, 0.002–0.032). When the average risk reduction was translated to NNTs, we estimated that one adverse event for every 59 patients admitted was prevented. Two studies (both higher quality) presented data on preventable adverse event rates.^24,27 In this meta-analysis, the total number of records was 40,588, and the pooled risk difference for preventable adverse events was 0.011 (95% CI, 0.0001–0.021) in favor of the intervention (Fig. 2; I² = 85% [P < 0.01]).

FIGURE 2.

Forest plot of the effect of patient safety initiatives on overall adverse events including all studies (A) and overall adverse events excluding lower-quality studies (B) and preventable adverse events (C). Diamond represents the pooled estimate of risk difference, calculated using the Mantel-Haenszel random effects model and 95% CIs. The squares represent study weighting, and horizontal bars represent 95% CIs.

Publication Bias

We found relatively few large-scale longitudinal studies examining adverse events. This is despite the fact that there are many point-prevalence studies from a wide range of countries including Ireland,³² Canada,³³ Australia,³⁴ New Zealand,³⁵ Mexico,³⁶ Brazil,³⁷ France,³⁸ Sweden,³⁹ Spain,²⁸ Belgium,⁴¹ Denmark,⁴² Korea,⁴³ and Norway.⁴⁴ Both the asymmetrical funnel plot (Fig. 3) and significant Egger test (P = 0.02) corroborate the reviewer’s judgment that publication and reporting bias were present.

FIGURE 3.

Funnel plot of studies used in meta-analysis.

DISCUSSION

We conducted a systematic review and meta-analysis to understand the average impact of large-scale patient safety initiatives on adverse event rates. This meta-analysis demonstrated a minimal reduction in adverse event rates. When studies of lower quality were excluded, we determine that the average risk reduction was 0.017, with one adverse event being prevented for every 59 hospital admissions (when this figure is converted to NNTs). These equate to considerable numbers when the large numbers of admissions to a hospital every year are considered.

Implications

We believe that the figures calculated in our meta-analysis may provide a realistic target for adverse event reduction. Some authors encourage a target of “zero harm” as set out in the more recent (2018) publication of Zero Harm: How to Achieve Patient and Workforce Safety in Healthcare,⁴⁵ whereas others have disputed the feasibility and highlighted the potential harms of such a target.⁴⁶ In addition to being demoralizing for health care staff, setting the unrealistic goal of absolute safety may result in unintended harm. For example, the use of standardized protocols has been successful in reducing hospital-acquired infections and improving outcomes of conditions that involve time-sensitive care such as stroke and myocardial infarction treatment.⁴⁷ However, “managerial gaming” of similar protocols to achieve targets has been shown to be responsible for discharging patients too early, data miscoding, the unnecessary admission of patients, and making patients wait in ambulances.^48–50 Furthermore, models that offer financial incentives for reducing adverse events or punishment for reporting adverse events may ultimately lead to adverse events being covered up with the resultant loss of learning.⁴⁶ Setting realistic targets is essential for avoiding such outcomes.

Large-scale patient safety initiatives are costly and resource-intensive and need to be audited for effectiveness, like any clinical intervention.⁵¹ The cost-effectiveness of adverse event prevention for many interventions have been demonstrated individually (e.g., pressure ulcers,⁵² medication-related adverse events,⁵³ health care–associated infections,⁵⁴ and falls⁵⁵). Although each study has used different models to determine the cost-effectiveness of an intervention, the models ultimately indicate the savings made by implementing a patient safety initiative relative to standard practice. However, it is debatable whether health care services have achieved these savings, based on our findings.

Future Research

This systematic review and meta-analysis described the patient safety initiatives of individual studies and stated the impact of the initiatives on adverse event rates. This may help guide health care services when developing patient safety initiatives. We acknowledge that quality improvement initiatives usually involve a planning, implementation, audit, and action phase in a cyclical process. The Plan, Do, Study, Act (used in the United Kingdom’s National Health Service) model is an example of such a model, which provides a framework for developing, testing, and implementing changes.⁵⁶ We call for the inclusion of a description of such a framework in studies examining adverse events, thus providing detail that would further illustrate how changes were made and could offer insights that may be transferable to other health care services.

Strengths and Limitations

This systematic review was performed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidance and is the first systematic review and meta-analysis of peer-reviewed literature examining overall changes in adverse event rates due to patient safety initiatives at a national or institutional level. We adopted a rigorous approach to the appraisal of bias, quality, analysis, and reporting of interventions.

As with any systematic review, we may have missed relevant studies despite an extensive search. Because of the small numbers of longitudinal adverse event studies, it was not possible to carry out a subgroup analysis or determine the implications of initiatives for vulnerable groups, such as those 65 years and older, multimorbid patients, and frail patients. The small number of studies also restricted our sensitivity analysis, which aimed to explore the effects of bias and study quality. The level of heterogeneity was high owing to the wide variation in safety initiatives, settings, study size, duration, and adverse event detection methods used. Two studies^29,30 could not be included in our meta-analysis because they did not report sufficient data. Both studies stated that there was no statistically significant change in the adverse event rate after the intervention. It is likely that studies yielding neutral or negative results are less likely to be reported and published, resulting in bias. This may have resulted in an overestimation of our calculated effect size.

Retrospective record reviews are limited by the quality of the clinical notes, reviewers being unable to determine the exact cause and effect, and varying degrees of internal and external validity.⁵¹ The interrater reliability of this methodology has been reported as moderate to substantial.⁵⁷ Overall, the evidence suggests that such retrospective record reviews do not capture the full degree and quantity of adverse events.^51,58

CONCLUSIONS

In conclusion, we have presented a systematic review and meta-analysis that highlights the limitations of capturing the impact of large-scale quality improvement initiatives. Particular strategies may have had a positive impact on overall and preventable adverse events. The modest reduction in adverse events presented in this study is far from the target of “zero harm” set out by some authors.⁴⁵ However, our findings may provide a more realistic target for health care providers when implementing patient safety initiatives.