Experiences with global trigger tool reviews in five Danish hospitals: an implementation study

Abstract

Objectives

To describe experiences with the implementation of global trigger tool (GTT) reviews in five Danish hospitals and to suggest ways to improve the performance of GTT review teams.

Design

Retrospective observational study.

Setting

The measurement and monitoring of harms are crucial to campaigns to improve the safety of patients. Increasingly, teams use the GTT to review patient records and measure harms in English and non-English-speaking countries. Meanwhile, it is not clear as to how the method performs in such diverse settings.

Participants

Review teams from five Danish pilot hospitals of the national Danish Safer Hospital Programme.

Primary and secondary outcome measures

We collected harm rates, background and anecdotal information and reported patient safety incidents (PSIs) from five pilot hospitals currently participating in the Danish Safer Hospital Programme. Experienced reviewers categorised harms by type. We plotted harm rates as run-charts and applied rules for the detection of patterns of non-random variation.

Results

The hospitals differed in size but had similar patient populations and activity. PSIs varied between 3 and 12 per 1000 patient-days. The average harm rate for all hospitals was 60 per 1000 patient-days ranging from 34 to 84. The percentage of harmed patients was 25 and ranged from 18 to 33. Overall, 96% of harms were temporary. Infections, pressure ulcers procedure-related and gastrointestinal problems were common. Teams reported differences in training and review procedures such as the role of the secondary reviewer.

Conclusions

We found substantial variation in harm rates. Differences in training, review procedures and documentation in patient records probably contributed to these variations. Training reviewers as teams, specifying the roles of the different reviewers, training records and a database for findings of reviews may improve the application of the GTT.

Article summary.

Article focus

• To describe experiences with the implementation of global trigger tool (GTT) reviews in five Danish hospitals.

• To suggest ways to improve the performance of GTT review teams.

Key messages

• Differences in training, review processes and documentation can contribute to differences GTT results.

• Systematic training of review teams, standardisation of review processes and feedback on results should improve the performance of review teams.

Strengths and limitations of this study

• The study supplies detailed contextual information that is relevant for the implementation of the GTT that increasingly is being used to monitor the safety in hospitals.

• The observational study design dose not allow for conclusions on causality and quantification of the contributions of different factors with relevance for the implementation of GTT reviews.

Background

Patients run a high risk of being harmed during hospital admissions. Adverse events occur in up to 10% of hospitalisations and can cause death, permanent or temporary disability.¹ For patients and healthcare workers, these harms and the underlying flaws of their healthcare systems that permit them to happen are deeply upsetting and completely unacceptable.

To improve the safety of patients, national and regional campaigns have been carried through^2–5 or are ongoing.⁶ Improvements have been achieved in some areas such as reductions of catheter-related blood stream infections.⁷ However, system-wide progress is slow⁸ and improvements are often limited to particular medical conditions or institutions. Indeed, a recent study⁹ from the state of North Carolina, an active participant in large-scale patient safety initiatives, concluded that overall rates of harm during 2002–2007 were not reduced. Thus, the challenge to improve the safety of patients in hospitals remains and specific and sensitive measures of harms are needed to assess and monitor the effects of changes to make hospitals safer.

In Denmark, the Operation Life campaign during 2006–2008 focused on patient safety in intensive care and during surgery. An estimated 1654 fewer than expected patients died in the Danish population of 5.5 million during the campaign.¹⁰ In 2010, another campaign the Safer Hospital Programme (http://www.sikkerpatient.dk/fagfolk/patientsikkert-sygehus.aspx) was launched at five pilot hospitals to reduce mortality by 15% and harms by 30% through the implementation of 12 care bundles. The hospitals are required to measure and report harms.

Meanwhile, a gold standard for the measurement of harms does not exist. Methods like voluntary reports only detect a small fraction of harms,¹¹ chart reviews have low inter-rater reliability¹² and are very time consuming and so are direct observations of healthcare processes.¹³ Studies comparing different methods of harm detection have found very little overlap of the detected harms.¹⁴ Therefore, complete estimates of the incidence of harms probably require the combination of different methods. Meanwhile, such an approach is time consuming and results are often delayed, which is unsuitable for patient safety campaigns in which frequent and regular measurements of harms are needed to evaluate and monitor the effects of interventions and organisational changes.

The global trigger tool (GTT) has been developed for the purpose of monitoring harms at low cost.¹⁵ Harm in this context is defined as an ‘Unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalisation, or that results in death’.¹⁶ Thus, the tool measures factual harm to patients, while errors not leading to harm, near errors and errors of omission are not included. A GTT review is a trigger-based chart audit of closed patient charts. Two reviewers, usually nurses or pharmacists, each review a limited number of randomly chosen charts with a given set of 56 triggers or “hints” of errors. The finding of such a hint triggers an investigation into whether and, if so, how severely, a patient actually has been harmed. Review time is limited to 20 min per admission. Finally, the two reviewers compare their conclusions and a supervisor, usually a physician, qualifies the number and severity of harms and decides in cases of disagreement. The number of harms is then expressed as a rate, for example, harms per 1000 bed-days. It has been suggested that GTT teams need a limited amount of training and practice to achieve good levels of reliability to identify harms.^16–18 The feasibility of the method invites for rapid adoption in healthcare systems around the world where practical ways to measure harms are much in demand. Nevertheless, experiences with the GTT in non-English-speaking countries are limited. Thus, careful calibration of the instrument and the review team that uses it is warranted to avoid evaluating the safety performance of hospitals with imprecise measurements.

A team of Danish experts translated the GTT to Danish¹⁹ from the English and a Swedish version.²⁰ The tool was tested in four hospitals in different health regions.²¹ The harm rate in these hospitals was around 20 per 1000 bed-days. A recent report of harms to Danish patients with cancer found a rate of 68 per 1000 bed-days.²² Notwithstanding these variable rates, policy makers advocate the widespread²³ implementation of GTT reviews in Danish hospitals. Meanwhile in our opinion, it is not sufficiently clear as to how the tool performs in the hands of Danish review teams.

The aim of this study was to describe experiences with the GTT in five Danish hospitals and suggest ways to improve the performance of GTT review teams and thus contribute to the accurate measurements of harms.

Methods

In this retrospective observational study, we present harm rates as measured by the GTT at the five hospitals participating in the Safer Hospital Programme in Denmark. The hospitals had used the GTT for 18 months from January 2010 until June 2011. The project managers at the hospitals supplied tabulated data on the size, activity and patient populations of the five hospitals from the year 2010. All data were collected between August and December 2011.

CvP, the project manager at one of the five hospitals and a consultant in pulmonary medicine, and AMK, a registered nurse and experienced GTT reviewer, interviewed members of the GTT teams and project managers on the telephone. The interviews comprised questions on the training of the GTT teams, team composition, roles of the team members and review processes. Moreover, they asked open questions about unexpected observations and changes. The GTT teams supplied lists of the recorded harms from their reviews.

To give an impression of the safety culture at the participating hospitals, we present the number and severity of reported patient safety incidents (PSIs) in 2010. JA, a physician by training who works in the Danish Society for Patient Safety, gathered these data from the Danish Patient Safety Database (www.dpsd.dk) for voluntary reporting of PSIs. Risk managers at the five hospitals classified the severity of PSIs in the Danish Safety Database into mild, moderate, severe and fatal. In Denmark, reporting of PSIs is mandatory, confidential and sanction free for healthcare personnel.

GTT review

The Danish translation of the GTT toolkit was the reference for the review teams.¹⁹ Teams were to review a random sample of 10 admissions twice a month. Closed admissions of patients of at least 18 years of age and of at least 24 h duration were eligible. The date of discharge was the index date. The GTT teams should review all available information from the admissions, that is, physician and nurse notes, medication orders and history as well as results of diagnostic tests. Each primary reviewer should review each record. Afterwards, the primary reviewers should meet to discuss harms and come to consensus. Finally, the primary reviewers should meet with the supervising physician to present their findings for approval and severity classification. Triggers, harms and the severity of harms should be recorded on standardised work sheets, then transferred to spreadsheets and stored locally. The team should classify the severity of harms according to the National Council for Medication Error Reporting and Prevention Index. The five hospitals only register harm rates in a shared database other data from GTT reviews are stored locally.

Data analysis

We calculated the monthly harms per 1000 bed days, harms per 100 admissions and the percentage of harmed patients. The harm rate was then plotted on a run chart and analysed using four run chart rules for the detection of patterns of non-random variation such as shifts or trends.²⁴ We collected and managed the data in Microsoft Excel V.2003 and produced the statistical analysis and graphs in R Statistical Software V.2.13.1.

For the purpose of this study, two nurses from the GTT team of one of the hospitals who had done over 400 GTT reviews each retrospectively categorised harms found at all five hospitals into categories also used by Classen et al.²⁵ They added gastrointestinal complications and pressure ulcers as categories because these were common types of harm (figure 1). Each harm was assigned to only one category.

Figure 1.

Harms by type. The dot plots show the relative distribution of harms by type. VTE, venous thromboembolism.

The regional ethical committee deemed an ethical review of the study unnecessary.

Results

Background data of the five pilot hospitals

The five hospitals, one in each Danish health region, vary in size but all hospitals have departments of internal medicine, orthopaedic and general surgery as well as obstetrics and gynaecology. All hospitals use electronic patient records, but to varying degree parts of the documentation, such as nursing notes, are on paper. The populations of patients at the five hospitals were similar with regard to age and gender distribution. There was a fourfold difference in reporting of PSIs among hospitals (table 1). Meanwhile, the distributions of PSIs by consequence were almost identical (figure 2).

Table 1.

Background information on the five hospitals (2010)

	Hilleroed	Horsens	Kolding	Naestved	Thy-Mors
Discharges	60 098	30 377	27 526	28 677	11 836
Average patient age (years)	55	57	53	59	58
Percent of females	62	59	61	55	55
Patient-days	231 978	108 060	90 710	113 353	49 711
Outpatient visits	262 547	212 899	124 184	184 374	65 165
Employees	3163	1367	1507	1668	689
Hospital standardised mortality rate	95	97	96	112	100
Reported safety incidents	2736	365	923	1182	223
Reported patient safety incidents per 1000 patient-days	12	3	10	10	4

Figure 2.

Patient safety incidents by consequence. The dot plots show the relative distribution of patient safety incidents by consequence as reported to the Danish national database. Categories minor and moderate represent no and temporary harms, major permanent harms. Overall, 96% of the incidents were temporary.

Experiences with the implementation of the GTT

Reviewing patient files with the GTT was new to three of the five teams. Prior to the Safer Hospital Programme, Naestved and Hilleroed Hospital had been using the GTT for reviewing patient records for 1 and 2 years, respectively.

In May 2010, the GTT team from each of the five hospitals participated in a 7 h training session with experts in the method from Denmark and the Institute of Health Care Improvement (IHI). The session included an introduction to the Danish GTT manual, frontal teaching, review of three training records per team and plenary discussions of the findings (table 2). Only the review team at Hilleroed had in 2008 received a similar training. All teams used nurses as primary reviewers and physicians as supervisors. All teams received on-site expert coaching with reviews of 10 or more records, up to three times. The expert coach was a physician who was trained by experts from the IHI. Furthermore, all teams participated in two full-day network seminars during the study period. All teams started reviewing patient records for the measurement of their baseline in May 2010 and retrospectively reviewed records from January to May 2010.

Table 2.

Characteristics and review procedures of global trigger tool teams at five hospitals

	Hilleroed	Horsens	Kolding	Naestved	Thy-Mors
Team characteristics
Number of physicians	2	1	1	2	1
Number of nurses	3	3	4 later 3	4	3
Number of changes in team	0	1 (physician)	1	0	2 (nurses)
Review intervals	Twice per month	Monthly	Monthly (two half days)	Monthly	Variable
Training
Hours of training	14*	7	7	7	7
Site visits by Danish expert (days)	1	3	1	4	2
Complete team present during site visit	−	−	+	−	−f
Number of records reviewed together with expert	<10	>10	>10	>10	>10
Review procedures
Whole team meets for reviews	+	+	Since Jan. 2011	−	−
Physician acts as judge (J) in cases of disagreement or reviewer (R) based on triggers	J	R	J	J	J
Records entirely electronic	−	+	+	−	+
Dedicated person responsible to find records	+	−	+	+	−
Secretary plots triggers and harms	−	−	−	−	+

*Team also trained by national expert in 2008.

The compositions of teams changed between zero (Hilleroed and Naestved) and three times (Thy-Mors) during the study period. Review intervals at the hospitals varied between 2 weeks, 1 month and irregular. Complete teams reviewed together and compared findings at two, later three hospitals. The role of the physician in the review team varied from only judging cases where the primary reviewers were in doubt or disagreed (Hilleroed) to identifying harms based on triggers found by primary reviewers (Horsens). Table 2 shows differences in review procedures at the hospitals.

Anecdotal information about GTT reviews

At Naestved, the team sampled 24 records (in case some were incomplete) each month and sorted them in the order of the date of the admission and reviewed the first 20 records. Thus, the sample became biased towards admissions in the earlier part of the month. Moreover, the team initially reviewed only admissions to the last department of a hospital admission. Thus, they did not find harms that, for example, occurred during an admission to the intensive care unit earlier during the hospital stay. These errors were accidentally discovered during a site visit and the team did a new review for the period. The team at Kolding discovered after 3 months that their sampling procedure excluded admissions that had an appointment for ambulatory follow-up after surgery, and they decided to discard the first 3 months from their baseline.

GTT findings

In total, 687 adverse events were identified in 11 487 patient-days, that is, the overall average harm rate was 60 per 1000 patient-days. The monthly harm rate ranged from 34 to 84 harms per 1000 patient-days (figure 3). The harm rates at all five hospitals showed only random variation.²³ The overall numbers of harms per 100 admissions were at Thy Mors 45, Naestved 24, Kolding 30, Horsens 43 and Hilleroed 54. The percentage of harmed patients was 25, ranging from 18 (Horsens) to 33 (Hilleroed) (figure 4).

Figure 3.

Rates of harm. The run charts show monthly rates of harms measured with the global trigger tool. The curve shows the harm rate expressed as the number of adverse events per 1000 patient-days. The horizontal line is the median harm rate.

Figure 4.

Harmed patients. The run charts show the percentages of harmed patients measured with the global trigger tool. The horizontal line is the median percentage of harms.

Overall, 96% of harms were temporary (grades E and F). However, the severity distribution varied between hospitals (figure 5). Notably, the hospital with the highest harm rate (Hilleroed) also had the highest proportion of grade E harms.

Figure 5.

Harms by severity. The dot plots show the relative distribution of severity of harms in categories E—I, where E and F are temporary, G—H permanent harms and I death. Overall, 96% of harms were temporary.

Five hundred and fifty-three harms (80%) were recorded with sufficient detail in the hospital datasheets to categorise them by type. The proportion of harms without description varied: 5% (Kolding and Hilleroed), 12% (Thy-Mors and Horsens) and 45% (Naestved). Common types of harm were infections, procedure-related, pressure ulcers and gastrointestinal problems (figure 1).

Discussion

We observed marked differences in the harm rates and types identified by GTT review teams in five Danish public hospitals. The GTT is not designed to compare hospitals but we were surprised by the magnitude of the variations. Therefore, we designed this study to identify factors that contributed to the differences. The hospitals, their patient populations, structures and activity levels were similar but we found differences in the training, the review procedures and the experience of the review teams.

Other studies have also found variation of harms across hospitals. Naessens et al¹⁸ in a study of 1138 admissions to three academic health centres in three states of the USA found a variation of harms by hospital between 23.1% and 37.9% of admissions. In a study of surgical harms, the variation was between 5% and 35%,²⁶ Sharek et al²⁷ observed harm rates between 0.18 and 1.28 per patient in 749 admissions to 15 newborn intensive care units in the USA and Canada and Resar et al²⁸ in 62 intensive care units in the USA measured between 3.2 and 27.36 harms per 100 days. Thus, significant variations of GTT findings seem to be common.

Several factors could explain the variation in rates of harm. First, there can be differences in the safety of the clinical processes at the hospitals. However, it seems unlikely that such differences should cause as much as a 2.5-fold variation of harm rates given the similar patient populations at the five hospitals and the homogeneity of the Danish healthcare system in general. We even found that the hospital with the highest mean age and the highest hospital standardised mortality rate had the lowest rate of harms. Second, the documentation of triggers and harms probably varies across hospitals. High PSI reporting rates are generally considered a sign of a mature safety culture rather than of poor safety and one could speculate that staff is more likely to document harms in a hospital with such a culture. Also different types of patient records (electronic and paper) and differences in layout and presentation could influence the results of the reviews. We do not have data to explore these questions but we certainly cannot exclude an influence on the different harm rates across hospitals. Third, differences in the training and the experience of the review teams influence the subjective process of judging harms in any record review.¹² The team that found the most harms was the most experienced review team and had attended two training seminars. This team was also stable and reviewed regularly with the whole team present twice a month. Finally, the teams conducted the review processes in slightly different ways. Most importantly, the roles of nurses and physicians varied. The role of the physician in the review team that found the highest harm rate was to judge in cases of disagreement while physicians in the other teams themselves identified harms. We assume that nurses are more prone to register harms of lower severity, while physicians might consider them insignificant. This interpretation is supported by our finding that the variation of harm rates was greatest in the least severe category ‘E’.

Thus, in our opinion, the experience of the GTT teams and the way they perform the reviews strongly contributed to the differences in harm rates in the five hospitals. Moreover, differences in the documentation of harms in the patient records seem to influence the number of harms the GTT team can find. We did not expect these factors to be so important because the GTT was implemented according to current recommendations^{15 17} and was guided by some of their authors. Moreover, all the teams had attended a GTT network meeting with national and international experts and received site visits by a national expert. However, these precautions, it seems, were not sufficient to secure at standardised reviewing process at the hospitals. Thus, users of the GTT and its results, healthcare personnel, administrators, payers or the public, should be aware of the challenges of the implementation of the method and allow for sufficient training and evaluation of the results. Our findings also stress that GTT findings should guide hospitals in their efforts to improve patient safety but the results should not be used to compare hospitals.

Strengths, limitations and further research

The strength of this study is its relevance for the implementation of the GTT that increasingly is being used to monitor the safety in hospitals. Our contextual data are detailed and thus practical. The limitations are inherent to the observational nature of the study that prevents conclusions on causal links between the variations of harm rates and the observed differences in team training and review processes. For the same reason, we cannot quantify the contribution of the different factors. Nevertheless, the findings are plausible and fit with the recommendations for the use of the method.¹⁵

Further research should address how teams’ reviewing experience and training influence team performance and how team training can be optimised. Moreover, studies should investigate the influence of changes of documentation and presentation of information in patient charts and the use of the types of harm for improving patient safety.

In conclusion, differences in training, review processes and documentation probably have contributed to variations in rates of harm as measured by the GTT. Thus, healthcare staff and policy makers should be aware of the need for systematic training of the review teams and standardisation of the review process when implementing the GTT in new settings. These factors are related to the implementation of the GTT reviews and are not inherent to the method as such. Our study has implications for the implementation of the method in other settings and we suggest considering the following interventions to improve the implementation of the GTT in new settings:

• Secure that the review team is trained as a team.

• Specify the roles of the reviewers during the reviews to avoid overestimation/underestimation of especially harms of lesser severity depending on professional background.

• Test review teams’ abilities to find harms with a set of training charts to estimate their ‘sensitivity’ before routine monitoring is instituted.

• Define a minimum number of patient charts that the team should have reviewed before monitoring harms routinely.

• Perform reviews with all team members present.

• Ensure a structured review process, that is, a space where the team can work without interruptions, regular time intervals between reviews to keep team ‘in shape’.

• Implement a common database with individual patient data to allow for re-examination of reviewed charts to avoid problems such as sampling errors.