Description of the development and validation of the Canadian Paediatric Trigger Tool

Abstract

Objective

To describe the process of developing and validating the Canadian Association of Paediatric Health Centres Trigger Tool (CPTT).

Methods

Five existing trigger tools were consolidated with duplicate triggers eliminated. After a risk analysis and modified Delphi process, the tool was reduced from 94 to 47 triggers. Feasibility of use was tested, reviewing 40 charts in three hospitals. For validation, charts were randomly selected across four age groups, half medical/half surgical diagnoses, from six paediatric academic health sciences centres. 591 charts were reviewed by six nurses (for triggers and adverse events (AEs)) and three physicians (for AEs only). The incidence of trigger- and AE-positive charts was documented, and the sensitivity and specificity of the tool to identify charts with AEs were determined. Identification of AEs by nurses and physicians was compared. The positive predictive value (PPV) of each trigger was calculated and the ratio of false- to true-positive AE predictors analysed for each trigger.

Results

Nurses rated the CPTT easy to use and identified triggers in 61.1% (361/591; 95% CI 57.2 to 65.0) of patient charts; physicians identified AEs in 15.1% (89/ 591, 95% CI 0.23 to 0.43). Over a third of patients with AEs were neonates. The sensitivity and specificity were 0.88 and 0.44, respectively. Nurse and physician AE assessments correlated poorly. The PPV for each trigger ranged from 0 to 88.3%. Triggers with a false/true-positive ratio of >0.7 were eliminated, resulting in the final 35-trigger CPTT.

Conclusions

The CPTT is the first validated, comprehensive trigger tool available to detect AEs in children hospitalised in acute care facilities.

Introduction

International data demonstrate that the incidence of adverse events (AEs) in hospitalised adults ranges from 2.9% to 16.6%.^1–12 Physiological and developmental differences in children could make them more vulnerable to harm, yet no data from methodologically similar studies in full paediatric populations exist. Published AE rates in children range from 0.2 to 154 per 10 000 discharge records,¹³ and 1 to 2.96 per 100 discharges.^{14 15} Chart reviews targeting specific paediatric populations and specific types of AEs reveal much higher rates.^16–18 Identifying a composite paediatric AE rate has been hampered by the absence of a single tool suitable for examining AEs across paediatric populations and events.

No single method can identify all harm associated with patient care. Concurrent use of complementary methods in real time and retrospectively for AE detection has been recommended.^19–24 Trigger tool (TT) methodology, however, is considered the best single tool, given its tiered approach and sensitivity to AE detection. It can be customised to diverse settings and, used consistently, can accurately measure harm over time.²⁵ The use of screening criteria to identify charts with possible harm followed by an in-depth review for actual harm has been used for decades, including the landmark Harvard Medical Practice Study.^1–11 More recently, the Institute for Healthcare Improvement (IHI) has recommended use of their Global Trigger Tool (IHI-GTT) for local quality-improvement activities.²⁶ Although focused paediatric TTs have been described, there remains no TT validated to detect AEs across all age groups of hospitalised children and youth.

There are growing efforts to improve patient safety in paediatrics, and quantification of the burden of iatrogenic harm in children could catalyse awareness and stimulate changes in paediatric practice and healthcare policy, as occurred following publication of adult AE rates. The Patient Safety Collaborative of the Canadian Association of Paediatric Health Centres established a working group to develop a paediatric TT (CPTT) to detect AEs in children hospitalised in Canada.²⁷ The goal was to develop a reliable and robust tool that could be used for both local quality-improvement activities and research into the rate, incidence, and factors contributing to AEs across a large population. The rigorous processes used to develop, test the feasibility of use of and validate the CPTT in detecting AEs are outlined in this paper.

Patients and methods

The early development stages of the CPTT have been previously reported.²⁷ Briefly, five TTs identified through a detailed literature review and personal communications were adapted to the modular format of the IHI-GTT. The Care, Medication, Surgical and Intensive care modules were retained, and two modules added: Laboratory and Other. Screening criteria from all other TTs were mapped against this list, duplicate triggers were removed, and additional triggers recommended and approved by consensus were added.

Each trigger in the resulting 94-trigger tool was evaluated for risk priority, frequency and detectability. Triggers above a pre-established cut-off or mean risk score were retained. Using a modified Delphi process, the remaining triggers were evaluated by a team of clinical paediatric patient safety experts who eliminated half, resulting in the preliminary 47-trigger CPTT.

Pilot testing of the CPTT

The CPTT was piloted in three acute care paediatric settings in Alberta: a stand-alone paediatric unit in a community hospital, a paediatric hospital-within-a-hospital and a regional paediatric centre. At each site, five medical and five surgical charts were selected for audit from fiscal year 2004 from each of three age groups: 29–365 days, 366 days to 5 years and over 5 years; 10 charts were randomly selected from the fourth age group of 0–28 days.

Two nurses visited the hospitals between June and August 2005, and independently reviewed all 40 charts at each hospital using the draft trigger tool. Each nurse had undergone detailed chart abstraction training as carried out for the Canadian Adverse Event Study,¹ and reliability was tested using a set of standard charts. Reviewers were asked to record any difficulties with interpretation or application of trigger definitions with particular emphasis on potential duplication or impracticality. Nurses were also asked to identify any triggers they thought should be considered for exclusion by the Research Team. Their experience was that the tool could be used to assess the charts, and no triggers were identified that could be eliminated. The κ statistic for the measurement of inter-rater reliability (IRR) for triggers in 120 charts reviewed was 0.81, and the validation study was commenced.

Definitions

Based on the definition used in previous national studies of AEs, an AE was defined as an unintended injury or complication that results in disability at the time of discharge, death, prolonged hospital stay or subsequent hospitalisation, and is caused by healthcare management.^{1 4 6 7 10 11} The AE had to occur within a 3-month period before or during the index admission and be detected during or within the 3-month period before or after the admission. ‘Healthcare management’ included the actions of individual hospital staff, as well as the broader systems and care processes, and included acts of omission and commission. To qualify as an AE, healthcare management must have received a causation rating of at least 4 on a six-point scale, that is, greater than 50% likelihood of being caused by healthcare management (box 1).

Box 1. Six-point scale for physician determination of causation².

1. Virtually no evidence for management causation.

2. Slight to modest evidence for management causation.

3. Management causation not likely; less than 50/50 but close call.

4. Management causation more likely; more than 50/50 but close call.

5. Moderate to strong evidence for management causation.

6. Virtually certain evidence for management causation.

The tool was validated by separate nurse and physician reviews of the same charts of children hospitalised in six academic paediatric health sciences centres (IWK Hospital Halifax; Children's Hospital of Winnipeg; Alberta Children's Hospital Calgary; Stollery Children's Hospital Edmonton; Children's Hospital of Eastern Ontario Ottawa; SickKids Hospital Toronto) during the fiscal year 2005/2006. Validation included the use of the TT on a sample of paediatric patient charts and the assessment of those charts to determine if AEs were present.

Training for nurses and physicians

Two nurses from each of the six participating institutions, and three physicians from the validation study team, with at least 10 years' academic paediatric centre experience, attended a 2-day training session at the University of Toronto. A customised study training manual and standard set of blinded paediatric hospital charts were used. Following training, nurses and physicians each reviewed 20 standardised charts, and the IRR was calculated.

Data collection and IRR calculation

At each hospital, two nurses reviewed a total of 100 selected charts for evidence of triggers. Nurses determined the presence of an AE based on the presence of an injury and harm associated with healthcare management and then assigned a National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) classification of harm for each event (table 1). Blinded to the nurses' results, one of the three physicians reviewed the same 100 charts at two sites and classified all injuries as AEs or not, using the standard definition of AE used in other studies (table 1). Ten per cent of charts were reabstracted, and the IRR for the six nurse teams calculated.

Table 1.

Distribution of nurse and physician assessment of adverse events (AEs) as compared with the nurses' classification of harm using the National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) scale

NCC-MERP harm category	Definition of category of harm	NCC-MERP	AE*	AE*
		RN review	RN review	MD review
No harm	No evidence of harm	442	2	34
E	Temporary harm to the patient requiring intervention	80	38	22
F	Temporary harm to the patient requiring initial or prolonged hospitalisation	56	41	25
G	Permanent harm	2	1	0
H	Intervention required to sustain life	7	7	4
I	Death	4	4	4
	Total	591	93	89

For each category of harm as defined by the National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) classification system, we note the number of cases in that category identified as having an AE by the nurse and the physician reviewers. No harm: nurse reviewers designated 442 cases as being in the NCC-MERP category of ‘No Harm.’ The nurses identified two, and the physician reviewers identified 34 of the 442 cases as having had an AE. Category E: of the 80 ‘E’ category cases, 47.5% and 27.5% were designated as an AE by the nurse and physician reviewers, respectively. Category F: of the 56 ‘F’ category cases, 73% and 46.6% were designated as an AE by the nurse and physician reviewers, respectively. Category G: the nurse reviewers identified one of the two ‘G’ category cases as an AE, whereas the physicians did not designate either as an AE. Category H: of the seven ‘H’ category cases, four were designated as AEs by the nurses and physician reviewers respectively. Category I: there was complete agreement on the deaths.

^*AE is defined as an injury, associated with a disability and caused by healthcare management.

Validation study

Biostatistical consultation determined that 600 charts from six paediatric healthcare centres would enable the reliability and validity of the tool to be established with a high degree of confidence (α 0.05; β 0.10). A standardised chart selection algorithm was applied to the Discharge Abstract Database for all paediatric separations in participating hospitals for fiscal 2006. One hundred charts plus a 20% oversampling (n=120) to accommodate for missing charts, stratified by age group as described above, were selected for review from those identified by the algorithm at each site.

Charts of children receiving all levels of inpatient care were eligible for inclusion. However, patients were excluded if they were 18 years of age and older or their most responsible diagnosis at discharge was obstetrical or psychiatric. For patients with more than one hospitalisation during fiscal year 2006, one stay was randomly selected to be reviewed for the study. No more than 100 eligible charts were reviewed at any single centre.

Analysis

Descriptive statistics were used for the trigger frequency and the number of AEs identified. The validity of the CPTT was judged by its sensitivity, that is, the percentage of trigger-positive charts that physicians judged to have an AE; and specificity, that is, the percentage trigger-negative charts that had no AE according to physician review. The positive predictive value (PPV) of each trigger, and the sensitivity and specificity of the overall tool were also determined.

Ethics

Research Ethics Board approval was obtained at each participating centre as well as the University of Toronto, and provincial legislation regarding privacy and protection of evidence was respected.

Results of validation testing

After training for the validation study, the κ statistic for IRR for nurse-identified triggers and physician-identified AEs was 0.62 and 0.57, respectively, and 0.67 for the IRR for nurses at each participating site. Five hundred and ninety-one of the 600 charts reviewed by nurses were available for physician review; the others were misfiled or on the clinical wards where the patient was currently hospitalised. Of the 591 charts reviewed by both nurses and physicians, 361 (61.1%; 95% CI 57.2 to 65.0) had at least one trigger: 148 (41%) had one, 145 (40.2%) had two, three or four, and 59 (16.3%) had five or more triggers.

Physicians identified 340 injuries in 180 of the 591 charts. In 89 charts, the injuries met the criteria for AEs, for an AE rate of 15.1% (95% CI 12.2% to 17.1%). The distribution of AEs by age and gender is displayed in table 2. Of the 89 patients with an AE, 37 (41.6%) were designated as resulting from acts of commission, 34 (38.2%) from acts of omission and six events (6.7%) from both; and 12 (13.5%) were not categorised.

Table 2.

Characteristics and distribution of patients with adverse events (AEs)

Patient characteristics	Overall	Admission age-group categories				p Level
		0–28 days	29–365 days	>1–5 years	>5 years
Age
No of patients and percentage of overall AEs	89 (100%)	33 (37.1%)	21 (23.6%)	17 (19.1%)	18 (20.2%)	0.006
Percentage of age group with AE		22.0%	14.2%	14.8%	10.1%	ns
95th percentile CI for percentage of age group		15.4–28.6	8.6–19.8	8.3–21.3	5.7–14.5	ns
Gender
Percentage of males with AE (95th percentile CI)	16% (12.4% to 20.4%)					ns
Percentage of females with AE (95th percentile CI)	13% (9.3% to 17.5%)					ns

In the nurses' review, only 78 (87.6%) of the 89 AE patient charts were identified as having one or more triggers (table 3).

The sensitivity of the Canadian Association of Paediatric Health Centres Trigger Tool was 0.88 (95% CI 0.79 to 0.94) and the specificity 0.44 (95% CI 0.39 to 0.48).

Further review of the 11 (12.4%) AE charts designated by the nurses as trigger-negative revealed that the trigger: ‘Other: Any other undesirable outcomes not covered above’ could have been applied to nine charts.

Table 3.

Sensitivity and specificity of the 47-Trigger Canadian Paediatric Trigger Tool

Trigger	Adverse event		Total
	Yes	No
Yes	78	283	361
No	11	219	230
Total	89	502	591

Sensitivity=0.88 (95% CI 0.79 to 0.94). Specificity=0.44 (95% CI 0.39 to 0.48).

Nurses' assessment of harm and AEs varied by the classification method used, and also differed from physicians' assessment. MDs found AEs in 34 charts that nurses assigned ‘No Harm’; otherwise, nurses were more likely to identify AEs in charts at all levels of harm severity (table 1). Nurses and physicians agreed on the presence of an AE in 489 (82.7%) of the charts for a κ of 0.34 (95% CI 0.23 to 0.43).

The PPV (table 4) and ratio of false-positive to true positive predictors of an AE for each trigger were calculated. The PPV ranged from 0 to 83.3. Based on these results, and in order to improve the efficiency of the CPTT, 11 triggers were removed: eight with a low predictive value and ratio of >0.7, and three that were never selected. A fourth unselected trigger, ‘wrong site surgery,’ was retained, and the two readmission triggers merged (C03 and C16). These revisions resulted in the final 35-trigger CPTT (table 5).

Table 4.

Forty-seven screening criteria of the preliminary Canadian Association of Paediatric Health Centres Trigger Tool (CPTT)^* and positive predictive values for each trigger

Care module		Positive predictive value
C01	Transfusion/use of blood products	42.7
C02	Any code or arrest	62.5
C03‡	Unplanned admission (including readmission) as a result of any healthcare management within the 3 months prior to or after discharge from the index hospitalisation	26.3
C04	Infection of any kind	54.1
C05	In-hospital stroke	33.3
C06	Transfer to higher level of care	45.8
C07	Catheter infiltration/burn	35.3
C08§	Wrong maternal breast milk	ns†
C09	Complication related to central venous catheter	61.1
C10	Necrotising enterocolitis	66.7
C11	Cranial imaging in infants <3 months	38.8
C12	Extreme temperature: ≤35.0°C or ≥41°C	35.5
C13	Intubation/reintubation/accidental extubation	51.5
C14	Unexpected death	80.0
C15	Emergent C-section delivery (neonate only)	40.0
C16‡	Unplanned admission (including readmission) as a result of any healthcare management within the 12 months prior to the index admission	23.8
C17§	Unplanned transfer to another acute care hospital (excluding transfers for tests, procedures or specialised care not available at referring hospital)	ns†
C18§	Inappropriate discharge to home/inadequate discharge plan for Index Hospitalisation (excluding ‘against medical advice’)	20.0
C19	Dissatisfaction with care documented in the medical record and/or evidence of complaint lodged (including documented complaint, conflict between patient/family and staff, discharged against medical advice/documentation or correspondence indicating litigation, either contemplated or actual	30.0
ICU module
I01	Readmission to ICU	33.3
I02	In-unit procedure	66.7
I03	Failed intubation	83.3
Laboratory module
L01	Abrupt drop >25% Hgb or Hct	48.8
L02§	Leucopenia	18.8
	Age <1 month WBC <5000/mm e3 (5.0×10 e9/l)
	Age 1–23 months WBC <4000/mm e3 (4.0×10 e9/l)
	Age 2–18 years WBC <3000/mm e3 (3.0×10 e9/l)
L03	Platelet count < 50 000/mm e3 (50.0×10 e9/l)	55.6
L04	PTT>100 s or INR >6	66.7
L05	d-dimer (positive by local lab normal)	62.5
L06§	Glucose <2.8 mmol/l (50 mg/dl)	25.0
L07	Sodium: 120 mmol/l >Na >150 mmol/l	46.2
L08	Potassium: 3.0 mmol/l >K+ >6.0 mmol/l	35.6
L09	Rising BUN/creat >2×baseline	68.5
L10	Hypoxia: O2 Sat <75%	48.1
L11	Positive blood culture	64.3
L12	Gentamicin: (except CF patients); trough >2 mg/l or peak >10 mg/l	44.0
Medication module
M01	Vitamin K (excluding newborns)	28.6
M02§	Benadryl (diphenhydramine): for symptoms of allergic reaction	18.2
M03§	Narcan (naloxone)	0.0
M04§	Antiemetic use (for treatment of symptoms)	20.0
M05§	Sodium or calcium polystyrene (kayexalate, resonium)	ns†
M06	Heparin or low-molecular-weight heparin	52.4
Surgical module
S01	Unplanned or return to surgery	77.8
S02	Intraoperative intravenous epinephrine, norepinephrine, naloxone, flumazenil	63.6
S03§	Pathology report normal/unrelated	15.0
S04§	Insertion of arterial or central venous line during surgery (excluding CV surgery)	25.0
S05	Removal/injury or repair of organ	40.0
S06¶	Wrong site/wrong procedure/wrong patient	NS†
Other module
O01	Any other undesirable outcome not covered above	31.4

All other triggers were retained in the final 35-trigger CPTT.

^*The positive predictive value and ultimate disposition of each trigger are also shown.

^†Screening criterion (trigger) was never selected by nurse reviewers in this sample of 600 cases.

^‡Screening criterion C16 merged with C03.

^§Screening criterion (trigger) with low positive predictive value or not selected removed from CPTT.

^¶Wrong site/wrong procedure/wrong patient criterion: a rare event but always associated with an AE and therefore retained despite NS designation.

ICU, intensive care unit; Hgb, haemoglobin; Hct, haematocrit; WBC, While blood count; PTT, partial thromboplastin time; INR, international normalised ratio; BUN, blood urea nitrogen; creat, creatinine; CF, cystic fibrosis; CV, cardiovascular.

Table 5.

35-Trigger Canadian Paediatric Trigger Tool

Care module
C 01	Transfusion/use of blood products
C 02	Any code or arrest (successfully resuscitated)
C 03	Unplanned admission (including readmission) as a result of any healthcare management within the 3 months prior to OR after discharge from the index hospitalisation
C 04	Infection of any kind
C 05	In hospital stroke
C 06	Transfer to higher level of care
C 07	Catheter infiltration/burn
C 08	Complication related to central venous catheter
C 09	Necrotising enterocolitis
C 10	Cranial imaging in infants ≤3 months
C 11	Extreme temperature: ≥35°C (35°C) or ≤40°C (40°C)
C 12	Intubation/reintubation/accidental extubation
C 13	Emergent C-section delivery (neonate only)
C 14	Dissatisfaction with care documented in the medical record and/or evidence of complaint lodged (including documented complaint, conflict between patient/family and staff, discharged against medical advice/documentation or correspondence indicating litigation, either contemplated or actual
C 15	Unexpected death
ICU module
I 01	Readmission to ICU
I 02	In-unit procedure
I 03	Failed extubation
Laboratory module
L 01	Abrupt drop ≥25% Hgb or Hct
L 02	Platelet count < 50 000/mm3 (50×10 e9 l)
L 03	PTT>100 s or INR>6
L 04	D-dimer (positive by local lab normal)
L 05	Sodium: 120 mmol/l>Na>150 mmol/l
L 06	Potassium: 3.0 mmol/l>K+>6.0 mmol/l
L 07	Rising BUN/creat >2×baseline
L 08	Hypoxia: O2 Sat <75%
L 09	Positive blood culture
L 10	Gentamicin/tobramycin: (except CF patients) trough <2 mg/l or peak >10 mg/l
Medication module
M 01	Vitamin K (excluding newborns)
M 02	Heparin or low-molecular-weight heparin
Surgical module
S 01	Unplanned or return to surgery
S 02	Intraoperative intravenous epinephrine, norepinephrine, naloxone, flumazenil
S 03	Removal/injury or repair of organ
S 04	Wrong site/wrong procedure/wrong patient
Other module
O1	Other/any other undesirable outcomes not covered above

ICU, intensive care unit; Hgb, haemoglobin; Hct, haematocrit; WBC, While blood count; PTT, partial thromboplastin time; INR, international normalised ratio; BUN, blood urea nitrogen; creat, creatinine; CF, cystic fibrosis; CV, cardiovascular.

Discussion

Several agencies, including the Agency for Healthcare Research and Quality, have argued that indicators needed to be developed for specific populations, including paediatrics.²⁸ We concluded that a robust TT, validated for paediatric use, was necessary for evaluating AEs and conducting a national paediatric study in Canada. Unlike the IHI-GTT where individual modules were validated separately, the CPTT has been validated for use as a single tool for all paediatric populations.

Similar to the other national studies of AEs, use of our tool calls for a two-stage review process: first, nurses use the trigger tool to identify charts likely to have an AE, followed by physicians who review the triggered charts for AEs. However, in order to establish the sensitivity of the nurse review, all charts in this study were reviewed independently for triggers (by nurses) and for AEs (by physicians and nurses). This process is the first of its kind to establish a false-negative rate for triggered charts, by identifying patients with AEs whose charts were not triggered. The number of false-negative charts may be reduced through education highlighting use of the ‘other’ trigger by the nurses. Nurse and physician review of all charts for AEs also demonstrated differences between these groups in designating AEs. Nurses and physicians assessed the charts using somewhat different methods. Nurses reviewed charts using the CPTT and assessed all charts in terms of the NCC-MERP Harm Scale and the presence of AEs,²⁹ whereas physicians reviewed all charts for AEs only. These different methods mean their results are not strictly comparable. However, the physicians read each case from admission to discharge, and based on these findings, it appears that physicians may be better qualified to assess the presence of AEs. This difference between nurse and physician assessment of AEs is consistent with previous findings,^{30 31} but these findings should be explored further.

There are inherent limitations in TT methodology. The utility of any chart review is subject to the quality of the documentation.³² Like other screening tests, TTs have a high sensitivity and relatively low specificity. However, their purpose is to cull charts unlikely to have an AE, leaving behind for physician review only those charts with a high probability of having an AE. The IRR of identifying AEs through chart review is variable; previous studies reported κ scores ranging from −0.077 to 0.66.^{1–9 16 19 32–35} Combining ratings from more than one physician^{19 36} and monitoring reviewer performance during the study with personal feedback may improve IRR.³⁰ However, a recent report suggests that a review process involving two physicians per chart is no more reliable than only one in detecting AEs.³⁶ Additional training that includes a consensus approach on the number of AEs in the training charts has been used successfully.³⁵ Finally, previous national studies of AEs have focused on those characterised by significant harm, that is NCC-MERP categories F–I, underestimating the total burden of harm by disregarding events with temporary harm requiring intervention only (category E events). Although achieving high IRR for category E can be challenging,³⁵ capturing these events in future studies will result in a more comprehensive view of the nature of iatrogenic injury, and permit comparisons with recent studies of AEs in children and youth that have used the expanded definition.^{17 18}

Despite these limitations, the use of a broad paediatric population sample to validate the CPTT permits its use in other acute care paediatric institutions. With its final refinements, the validated CPTT offers the first comprehensive evidence-based tool for assessing harm in hospitalised children and youth.

Conclusion

The CPTT is the first valid and reliable TT for detecting harm in children and youth of all ages hospitalised in acute care. This 35-trigger tool is reliable and robust, and can be used in quality-improvement initiatives and for more rigorous research agendas.

Future research should focus on improving the efficiency of the CPTT and investigating the differences between nurse and physician assessments of AEs. The CPTT is likely to be of interest to researchers wishing to comprehensively study the epidemiology and burden of iatrogenic harm in hospitalised children and youth, using a methodology similar to the previous national studies of AEs in hospitalised adults. Such studies will enhance current efforts to raise the profile of paediatric patient safety issues.