Development and evaluation of trigger tools to identify pediatric blood management errors

Abstract

Background

Pediatric Patient Blood Management (PBM) programs require continuous surveillance of errors and near misses. However, most PBM programs rely on passive surveillance methods. Our objective was to develop and evaluate a set of automated trigger tools for active surveillance of pediatric PBM errors.

Materials and methods

We used the Rand-UCLA method with an expert panel of pediatric transfusion medicine specialists to identify and prioritize candidate trigger tools for all transfused blood products. We then iteratively developed automated queries of electronic health record (EHR) data for the highest priority triggers. Two physicians manually reviewed a subset of cases meeting trigger tool criteria and estimated each trigger tool’s positive predictive value (PPV). We then estimated the rate of PBM errors, whether they reached the patient, and adverse events for each trigger tool across four years in a single pediatric health system.

Results

We identified 28 potential triggers for pediatric PBM errors and developed 5 automated trigger tools (positive patient identification, missing irradiation, unwashed products despite prior anaphylaxis, transfusion lasting >4 hours, over-transfusion by volume). The PPV for ordering errors ranged from 38–100%. The most frequently detected near miss event reaching patients was first transfusions without positive patient identification (estimate 303, 95% CI: 288–318 per year). The only adverse events detected were from over-transfusions by volume, including 4 adverse events detected on manual review that had not been reported in passive surveillance systems.

Discussion

It is feasible to automatically detect pediatric PBM errors using existing data captured in the EHR that enable active surveillance systems. Over-transfusions may be one of the most frequent causes of harm in the pediatric environment.

INTRODUCTION

Patient Blood Management (PBM) is a complex process involving multiple stakeholders (e.g., ordering providers, blood bank technologists, nurses) completing many difficult tasks (assessing indications, selecting product, volume/dosing, rate of administration, special processing, etc.) and monitoring many outcomes¹. Children are at a higher risk of adverse outcomes from PBM errors than adults due to their small size and requirements for special processing (e.g., risk of immunocompromise among pre-term and term infants)^2–4. Health information technology and other PBM interventions in education, auditing, and feedback related to PBM have demonstrated better adherence to evidence-based practice, patient outcomes, and reduced costs^5–9. However, many of these interventions are resource-intensive and are limited to specific use cases¹⁰.

PBM intervention design priorities should be guided by the most burdensome error types, frequency, and severity. Currently, our understanding of blood product safety errors is driven primarily by incident reports^11,12. However, active surveillance of all medical errors on a general pediatric unit found that incident reports identified only 5% of all medical errors and only 6% of all adverse events¹³. In contrast with incident reports, trigger tools identify a larger number of errors, allow burden comparisons across error types, and can provide automated data feeds with minimal ongoing resource utilization for “Plan, Do, Study, Act” cycles^14–20. While scrutinizing near misses in PBM is not novel, to our knowledge these efforts have relied previously on incident reports, whereas automated electronic screening tools could move PBM from passive to active surveillance and improve overall safety.

In this study, we hypothesized that PBM errors are detectable via EHR phenotypes (defined, computer-interpretable criteria that can be automatically applied to a population) and identify errors that are missed in passive surveillance. We first used a modified Delphi method with subject matter experts to define and prioritize a set of PBM metrics. We then developed a series of automated trigger tools for high priority metrics using EHR queries and validated their performance through a retrospective cross-sectional analysis of all blood transfusions orders and administrations at a single pediatric health system from July 2018–June 2022. Finally, we estimated medical error and adverse event frequency for each automated trigger.

MATERIAL AND METHODS

Candidate trigger list

We first conducted a review of the PBM literature to identify candidate triggers for possible inclusion^2,3,11,21,22. Where possible, we spoke with the authors of the original papers. We supplemented this list with an analysis of blood product-related incident reports at the primary site for this work, a large urban academic pediatric health system in the Southeastern United States in which the primary blood bank serves 2 freestanding children’s hospitals as well as high volume hematology and oncology clinics blood and marrow transplant program, and solid organ transplants (e.g., heart and liver) with over 10,000 blood transfusions per year. From this review, we initially compiled 24 unique candidate triggers and evidence supporting the clinical practice.

Expert stakeholder panel

We convened an expert stakeholder panel consisting of 7 pediatric hematologists and transfusion specialists across 6 institutions responsible for management of pediatric blood banks and program evaluation. Using the RAND/UCLA Appropriateness method²³, we initially asked all experts to independently rate on a 9-point Likert scale the severity of the error type, preventability, quality of the evidence supporting the best practice, and estimated feasibility of capturing the error reliably in an automated fashion. Experts were allowed to ask questions to the central team and clarifications deemed useful were sent out to the whole panel. Experts were also allowed to add additional candidate triggers. A summary of preliminary ratings was sent to each panelist including the average scores for each trigger in each domain and the variance across panelists. The full group was convened for a synchronous virtual discussion (via video conference) facilitated by one author (EWO) focused first on the triggers with highest average scores and second on triggers with the greatest variance. Minor adjustments to trigger definitions and new triggers were added based on this discussion. At the end of the discussion, panelists again independently assessed the severity, preventability, evidence quality, and feasibility of this final trigger set on a 9-point Likert scale. The final prioritized trigger set was based on the average of scores for each trigger with each domain treated equally.

Trigger tool development

Two authors with EHR querying expertise (SK and EWO) assessed the prioritized trigger list for feasibility of automation within our health system. We supplemented the prioritized trigger list with high priority blood product safety metrics at the primary site. EHR queries were developed using the Epic Systems^© Clarity database, which contains all data on blood product orders and administrations within an enterprise-wide implementation of the EHR. For all queries, we excluded blood product orders and administrations associated with massive transfusion protocol, extracorporeal membrane oxygenation, emergency release, and priming of hemodialysis machines.

Validation

For each trigger tool, we reviewed a minimum of 16 examples to estimate the positive predictive value (PPV) based on an estimated power of 80% to detect PPV ≥15% if the true PPV was ≥40%. Each example was reviewed by a pediatric hospital medicine attending (EWO) and at least one board-certified pediatric transfusion medicine specialist (MRR or CDJ). Each review consisted of (1) estimation of query accuracy and low effort adjustments to improve, (2) determination of an ordering error, defined by the statement “assuming all actions were carried out exactly as ordered, would there have existed any gaps in the standard of care?” (3) determination if the error reached the patient (e.g., in a patient with DiGeorge syndrome where packed red blood cells (PRBCs) were not ordered to be irradiated, but nonetheless irradiated PRBCs were given would NOT count as having reached the patient), and (4) identification of adverse events, defined using the NCC MERP criteria of D or higher²³.

Frequency of trigger events

After completing validation, we ran queries at the primary site from July 1, 2018 through June 30, 2022. We reported the estimated number of trigger events and, where appropriate, the frequency as a proportion of the appropriate denominator.

Statistical analysis

Descriptive statistics were calculated for validation and frequency of each trigger. Standard errors were calculated for candidate trigger ratings by the expert stakeholder panel in Microsoft Excel (Redmond, Washington). Proportion confidence intervals for the estimated annual errors reaching the patient and causing adverse events were calculated using χ² tests using R version 3.5.1 (Vienna, Austria)²⁴.

Ethical considerations and reporting guidelines

This study was approved by the Emory University IRB (STUDY 00000956).

RESULTS

Trigger tool prioritization

We identified 27 unique trigger tools, which were prioritized by the expert stakeholder panel (Figure 1). The highest priority trigger tool was ensuring appropriate patient identification with the type and screen–either through a barcoded sample or two separately collected samples– with an average score of 8.9 on a 9 point Likert scale. Two trigger tools had an average score ≥8 across the 4 domains, while 6 had an average score from 7–8, 10 had an average score from 6–7, and 11 had an average score <6.

Figure 1. Pediatric Blood Management trigger tools: results of the modified Delphi (Rand/UCLA) exercise with the expert stakeholder panel.

Ratings from pediatric transfusion experts on the severity, preventability, quality of evidence, and feasibility of proposed trigger tools on a 9-point Likert scale.

*Hemodynamic instability defined as: two most recent documented blood pressures prior to transfusion within normal range for age; not on vasopressors; not on extracorporeal membrane oxygenation; no massive transfusion protocol; not emergency O-negative transfusion.

**Increased respiratory support defined as: new oxygen requirement after transfusion; intubated within 12 hrs of transfusion; mean airway pressure increase >20% within 12 hrs of transfusion.

Final metric definitions

We developed computer-interpretable, automatable definitions for 5 trigger tools (Table I) including:

Table I.

Validation results and frequency for 5 trigger tools

Trigger short name	Trigger description	Positive predictive value # meeting criteria/# manually reviewed (%)			Frequency
		Ordering error	Reached patient	Adverse event	Definition	Rate (%)	Estimated reach patient (events/yr)	Estimated adverse events/yr
Positive patient ID	In absence of barcoding, patients with no prior ABO type in the system SHOULD have two type and screens ordered and drawn to be sure blood group is correct prior to first transfusion	17/17 (100)	17/17 (100)	0/17 (0)	Proportion of first transfusions for the patient in the primary site health system that were flagged by the trigger tool	1,210/5,285 (22.9)	303 (288–318)	0 (0–1)
Missing irradiation	Patients with evidence of T-cell dysfunction SHOULD receive irradiated PRBCs and platelets	6/16 (38)	0/6 (0)	---	Proportion of PRBC transfusion orders that SHOULD have been irradiated that were flagged by the trigger toola	1,176/24,405 (4.8)	0 (0–1)	0 (0–1)
Unwashed products despite prior anaphylaxis	Patients with history of anaphylaxis to blood transfusion SHOULD receive blood products at lower risk of subsequent reaction (washed, plasma volume reduced [ PVR], using platelet additive solution [ PAS] platelets, or from IgA deficient donor [ for plasma])	12/17 (71)	4/12 (33)	0/4 (0)	Proportion of PRBC or platelet transfusions that SHOULD have been washed that were flagged by the trigger tool	369/500 (74)	22 (18–26)	0 (0–1)
Transfuse in < 4 hours	Blood product aliquots SHOULD each be administered in < 4 hours	---	12/16 (75)	0/12 (0)	Proportion of blood product administrations where the start and stop times are > 255 minutes apart and < 330 minutes apart (to account for missed documentation)	590/66,796 (0.8)	111 (101–122)	0 (0–1)
Overtransfusions b	Outside of emergencies, patients should not receive excessc PRBCs between hemoglobin checks	11/16 (69)	11/11 (100)	4/11 (44)	Proportion of transfusion blocksd that were flagged by the trigger tool	77/32,219 (0.2)	14 (10–18)	6 (4–9)

^aIn the primary site health system, all platelets are irradiated. Therefore platelet transfusion orders were not included in the frequency calculation for trigger 2.

^bOvertransfusions were not included in the original candidate trigger list for the expert stakeholder panel; rather, it was selected due to safety events in the primary site health system.

^cTransfusion blocks were defined as the time period between two hemoglobin results during which a patient received PRBCs ( with a maximum block time of 2 weeks). For example, if a hemoglobin resulted at 09:00, the patient then received 100 mL of PRBCs from 10:00 to 13:00 and another 100 mL from 15:00 to 18:00, and another hemoglobin was checked and resulted at 20:00, the transfusion block would be the time period from 09:00 to 20:00 with a total of 200 mL PRBCs given.

^d“Excess PRBCs” were defined as transfusion blocks in which the patient ( 1) received > 20 mL/kg of PRBCs, ( 2) had hemoglobin increase by > 3, and ( 3) final hemoglobin > 3 above baseline ( defined as the average of ≥10 prior hemoglobins) or if no baseline available then final hemoglobin ≥13 for patients with sickle cell disease or ≥15 for all other patients.

1. Positive patient identification - In the absence of barcoding, patients with no prior ABO type in the system SHOULD have two type and screens ordered and drawn to be sure blood group is correct prior to first transfusion25 scanning a patient’s wristband barcode before pretransfusion sample collection.

2. Missing irradiation - Patients with evidence of T-cell dysfunction SHOULD receive irradiated PRBCs and platelets²⁶.

3. Unwashed products despite prior anaphylaxis - Patients with history of anaphylaxis to blood transfusion SHOULD receive blood products at lower risk of subsequent reaction (washed, plasma volume reduced [PVR], using platelet additive solution [PAS] platelets, or from IgA deficient donor [for plasma])^27,28.

4. Transfusions lasting >4 hours - Blood product aliquots SHOULD each be administered in <4 hours³⁰.

5. Over-transfusions by volume - Outside of emergencies, patients should not receive excess PRBCs between hemoglobin (Hb) checks. Using the “transfusion block” (defined as mL/kg of PRBCs between Hb checks) as the unit of analysis, a transfusion block with an over-transfusion met the following criteria:

   1. ≥20 mL/kg since the last Hb check and

   2. Hb increased by ≥3

   3. if the baseline Hb is known (≥10 prior Hb checks in the system), the ending Hb is ≥3 above the baseline; if unknown ending Hb ≥13 (sickle cell disease patient) or Hb ≥15 (any patients) where baseline is defined as the average of all prior Hb checks^30,31.

Validation

For each of the 5 trigger tools for which we developed automated queries, we achieved a PPV for an ordering error or worse ranging from 38 to 100%, with all but one trigger having PPV ≥70% (Table I). The proportion of ordering errors that reached the patient ranged from 0 to 100%. Specifically, missing orders for irradiated blood products were common, but we did not find any cases where non-irradiated blood products reached the patient when necessary, likely due to mechanisms within the blood bank to identify patients with T-cell deficiency independent of orders. Among the examples manually validated, we did not find any adverse events except for over-transfusions (trigger 28). Of note, among the 6 adverse events detected in our review of trigger 28,4 (67%) had not been previously identified in our incident report system.

Frequency of trigger events

We estimated the frequency of each trigger event across 4 years in the primary site health system (Table I). Inadequate positive patient ID (trigger 1) occurred in 22.9% of first transfusions in our health system; we estimate that this event would reach 303 patients per year (1,210 triggers × 100% ordering error rate × 100% reached patient/4 years), but did not find any adverse events in our validation sample. Similarly, while we estimate 294 ordering errors per year associated with missing irradiation requests (trigger 2), we did not find any examples that reached the patient. For unwashed products to patients with prior anaphylaxis (trigger 3), we estimate 22 errors per year would have reached the patient (369 triggers × 71% ordering error rate × 33% reached patient/4 years), but we did not detect any adverse events. For blood products administered over >4 hours (trigger 4), we estimate 103 errors per year reaching patients (590 triggers × 75% reached patient/4 years), but we saw no adverse events. Finally, we estimate 14 over-transfusion events would reach patients per year (77 triggers × 69% ordering error rate × 100% reached patient/4 years), resulting in an estimated 6 adverse events per year (estimated 14 reached patient per year × 44% adverse event rate).

DISCUSSION

Effective PBM requires an ongoing assessment of errors and near misses to enable continuous improvement. Pediatric PBM is uniquely challenging in this regard because error types may be more complicated to define (e.g., over-transfusions) and the evidence base is less well established with fewer randomized controlled trials³². Our approach focused on identifying the most important and burdensome pediatric PBM error types in the setting of imperfect data. We therefore leveraged a combination of expert opinion, automatable definitions, and manual review of specific cases to provide a foundation for active surveillance of pediatric PBM errors. We identified a high frequency of errors through novel, automated trigger tools that were not reported in passive surveillance systems. The most commonly detected adverse events were PRBC over-transfusions by volume administered, using a metric based on transfusion blocks (the volume per kg of administration between Hb checks), which can be life threatening³⁰. This finding correlated with local experience at our primary health system where over-transfusions were a prioritized safety problem for the organization. In fact, our automated methods identified previously undetected adverse events associated with over-transfusions. Nonetheless, automated identification of over-transfusions remains challenging due to problems of inadequate sensitivity (e.g., incorrectly documented over-transfusions) and specificity (e.g., large volumes given appropriately in the setting of emergencies or unclear evidence).

While over-transfusions were the most frequently identified adverse events, we found frequent near misses in other domains that likely merit further intervention before adverse events are detected. We estimate that a large number of patients may have received their first blood transfusion in the primary health system without positive patient identification, risking wrong patient errors with potentially dangerous effects. Similarly, many blood product orders did not appropriately ask for irradiation or washed products, although backup systems in the blood bank appear to have mitigated most of those errors from reaching patients. Finally, many patients had blood transfusions documented beyond the recommended maximum of 4 hours, though this may reflect some documentation errors. Further investigation of additional triggers identified by our expert panel would likely yield further insights into pediatric PBM failures.

Clinical decision support (CDS) and other systems interventions have demonstrated the ability to reduce PBM errors in adults and children^9,33–35. However most such improvement efforts focus on and develop metrics for a single error type at a time. Passive surveillance systems such as hospital incident reports¹¹ capture a broader set of error categories, but the extra effort required for providers to enter these means they often do not include near misses and may misrepresent the distribution of adverse events and near misses^15,37,38, leading to mis-targeting of interventions by pediatric PBM practitioners. Active surveillance systems for medical errors likely yield the most useful information for targeting and evaluating interventions, yet such systems are generally expensive and require substantial human resources to maintain¹³. By contrast, trigger tools can help bridge the gap between the benefits of active surveillance and its resource requirements; trigger tools for pediatric safety¹⁴, medication errors³⁸, diagnostic error³⁹, and surgical adverse events^40,41 among many others. To our knowledge, this study describes the first suite of trigger tools focused on pediatric PBM and provides a framework for development of a more comprehensive PBM active surveillance program.

Limitations

While pediatric PBM experts from multiple centers were involved in development and prioritization of a trigger tool list, the automated tools themselves were developed and evaluated in a single center. This limits their generalizability both in terms of accuracy (e.g., different documentation patterns at other centers may invalidate the trigger tools) and assessment of the highest burden PBM errors (e.g., over-transfusions may be unique to the primary health system; other safety measures implemented in that system such as interventions focused on positive patient identification may conceal challenges that loom larger at other centers). Additionally, the trigger tool candidates were limited by the existing evidence in pediatric PBM. As new evidence develops, these proposed error detection methods may no longer be valid or may change in priority. Finally, due to practical constraints we did not assess inter-rater reliability during manual review for detection of errors. Instead, we relied on consensus of two physicians including at least 1 with pediatric transfusion medicine training. However, if assessments of harm were unreliable, this could mean the trigger tool accuracy was falsely inflated (e.g., due to a Hawthorne effect) or falsely low (e.g., due to biases in how error rates may reflect on the primary health system’s existing PBM program).

CONCLUSIONS

Through automated trigger tools, it is feasible to detect pediatric PBM errors, near misses, and adverse events that are missed in passive surveillance. An efficient process to accurately identify PBM errors, give more frequent feedback to clinical and improvement teams can inform prioritization of interventions, develop real-time EHR based decision support and accelerated guide iterative improvement efforts through plan-do-study-act cycles.