Informatics Tools for the Development of Action-Oriented Triggers for Outpatient Adverse Drug Events

Abstract

Background:

Trigger tools are an important development in the identification and reduction of adverse drug events (ADEs). Most previously published triggers are simple, consisting of one or two conditions. Simple logic may lead to alerts for conditions not caused by a drug or already treated by the provider.

Methods:

We created a knowledge-encoding tool to develop outpatient ADE triggers to more specifically identify harm caused by a drug and which require further clinical intervention. The tool presented the user with data on similar triggers from the literature and a series of fields to facilitate the creation of algorithms based on epidemiological principles.

Results:

Using this tool, we created 23 triggers that addressed 55 high-harm outpatient drugs and ADEs.

Conclusion:

Informatics tools can facilitate the design of clinically rich triggers. More investigation is needed to determine whether the performance characteristics of clinically rich triggers are better than those of simple triggers.

Background

Adverse drug event detection continues to be an important objective of patient safety research. Improvements in the ability to accurately identify adverse drug events (ADEs) include computerized triggers or algorithms that use electronic patient data to identify patterns consistent with a possible ADE¹. The concurrent or real-time evaluation of triggered alerts has been used to guide clinical interventions to prevent emerging ADEs or mitigate actual ADEs^2–4. These action-oriented triggers have been popular with clinicians at several sites^{5, 6}.

A typical trigger consists of one or two logical steps such as a lab value threshold or the combination of a lab value threshold and an active prescription². These types of triggers have moderate positive predictive value on the order of 0.23% to 0.59%^6–9.

When investigating a triggered alert a clinician applies additional criteria to make a determination of iatrogenic harm¹⁰. Some of these criteria include alternative explanations, reasonable timing, and indications that the team has addressed the problem. Searching for information relevant to these criteria takes extra attention and time of the clinician. We hypothesized that it would be possible to create a new type of computerized trigger that was rich in clinical knowledge and more efficient at identifying opportunities to intervene in ADEs.

This paper reports on one aspect of an AHRQ contract to develop action-oriented ADE triggers for the outpatient setting. Few triggers target outpatient ADEs, despite high prevalence rates and excess utilization from preventable emergency room visits and hospitalizations^11–13. Instead of merely adapting ADE triggers developed for the inpatient setting, we chose to test the concept of encoding clinically rich triggers for outpatient ADEs. We describe our process of developing an informatics tool to encode clinical logic into trigger development. We also present the outpatient ADE triggers resulting from this process.

Methods

The outpatient ADE trigger development project was funded by the Agency for Healthcare Research and Quality (AHRQ). A team of researchers from the Veterans Health Administration (VA), Boston Medical Center (BMC) and Intermountain Healthcare developed a set of triggers to detect ambulatory adverse events (AEs)¹⁴. The AHRQ study capitalizes on the availability of electronic ambulatory data at each institution, as well as the clinical and trigger development expertise of the research team. While the goal of the project is to develop AE triggers for the outpatient setting, this paper focuses only on the ADE triggers.

The AHRQ project began with a review of the literature on triggers designed to detect AEs, including ADEs, in the inpatient and outpatient setting¹⁵. The review captured all of the triggers identified in the literature, and highlighted types and causes of ADEs associated with significant harm or high frequency in the outpatient setting. The project team’s clinical experts used the literature on triggers and prevalence of causal agents and ADEs to determine gaps in trigger development and to prioritize outpatient trigger design. Priority was given to triggers that could easily be translated into the outpatient setting, and to the development of triggers where gaps existed in the literature.

A knowledge-encoding tool was created using Microsoft (MS) Access to incorporate results from the trigger literature along with clinical and trigger development expertise into a set of fields to build rules. Trigger information, including established rules, test characteristics (sensitivity, specificity, and positive predictive value (PPV)), ADE prevalence, and harm were collected systematically from the literature into the literature form. A separate form linked to the literature form captured clinical input to develop trigger rules. The structure of the database supported broad analysis of existing research and highlighted areas for future trigger development.

The knowledge-encoding tool was designed to enable trigger developers to build off advancements in the field. The literature form included a field for action-oriented logic, purposefully crafted logic to maximize the identification of imminent or actual iatrogenic ADEs in which the clinical team could intervene to prevent or ameliorate the event. The form also captured information on whether the trigger was scoped to address any ADE versus a specific ADE.

Several fields were included to fully articulate existing triggers in the literature: ADE(s) targeted; cause of the ADE; and data sources used by the trigger. In addition to literature on triggers, the knowledge-encoding tool had a form to input literature on significant causes of patient harm. Areas that were previously unaddressed by triggers were easily discerned by comparing prevalence information for ADE causes and events and triggers from the literature.

The knowledge-encoding tool used a separate MS Access form, the clinical input form, to develop outpatient trigger rules. The clinical input form was designed to capture and code clinical knowledge based on epidemiological principles of ADE causality¹⁶.

Clinicians using the knowledge-encoding tool had access to the information captured from the literature to specifically take into consideration prior research when compiling triggers. The clinical input form mapped to results from the literature based on AE cause. When the clinician picked the ADE cause, similar triggers documented in the literature form appeared on the right hand side of the screen. Information on existing triggers included the effect associated with the trigger, the trigger rule and the average PPV (see Figure 1).

Figure 1.

Screenshot of the knowledge-encoding tool’s clinical input form and results for similar triggers in the literature

The elements required to design a trigger using the clinical input form include the effect, the cause (there can be many effects and/or many causes), and the algorithm (see Table 1). The creator must also specify the data source(s) that support the algorithm. More complex triggers also present information on timing and how different data sources can be used to identify specific effects.

Table 1.

Description of the clinical input form fields

Field Name	Explanation
Effect	The ADE targeted. Linked to ADE targeted on the literature form and includes a write-in option. Can select up to three effects
Cause	Cause of the ADE. Linked to causes on the literature form and includes a write-in option. Can select only one cause
Synthesize the Trigger	The outline of the trigger rule. May not incorporate all the detail captured in the clinical input form
Interventionist Trigger	Check box to indicate trigger uses action-oriented logic and can be acted on by clinicians to prevent or ameliorate the event
Category	Type of information from patient data, e.g. weight gain, classified as a trend
Data Source	Forms of patient data used in the algorithm such as laboratory orders
Susceptibilities/Resistance to AE	Patients at high or low risk for the ADE, e.g. patients with chronic kidney disease at high risk for opiod ADE
Markers of Other Explanations	Patient information that would lead to similar effects based on causes that are not adverse
Timing that Impacts Likelihood of Causation	Patient information occurring prior to effect that increases or decreases risk of ADE from specific ADE cause, such as initiating new drug regimen
Markers of Unrecognized Iatrogenic Harm	Patient information that shows evidence of an ADE but does not document a medical intervention
Markers of Recognized Harm	Patient information consistent with a medical reaction to an ADE, such as administration of an antidote
Sequences of Markers	Sequences of markers or trends in patient information that increase or decrease the likelihood that effect results from cause or that ADE is serious

Since the knowledge-encoding tool described the data source and the average PPV of similar triggers from the literature, upon development of outpatient triggers, researchers on the AHRQ project could prioritize the triggers based on how well they lined up with electronic data within each health system and how likely they were to perform well. This list of ADE triggers, developed through the knowledge-encoding tool, was presented to a Delphi panel of trigger experts to identify final triggers for testing.

The final list of ADE triggers to test was determined through a modified Delphi approach¹⁷. A panel of eleven trigger experts participated in three rounds of consensus development conducted through email. Panelists ranked triggers on utility for patient or system-level interventions based on a scale of 1–9 (1 is most useful). The research team chose to test ADE triggers with patient or system-level utility rankings of three or lower by at least nine of the panelists based on Round 3 results.

Results

Three members of the research team populated the literature section of the database with more than 900 AE triggers from 110 research articles. Results of the literature review are available in a separate manuscript¹⁵. Clinician researchers developed 23 outpatient ADE triggers using the clinical input form. The triggers addressed 55 prevalent and harmful outpatient drugs and ADEs.

The knowledge-encoding tool allowed clinicians to supply detailed information on the causal agents and effects associated with a particular ADE trigger. However, the trigger rule did not always incorporate all the detail encoded in the tool. The example in Table 2 shows which fields populated in the knowledge-encoding tool were also included in the trigger rule. Delphi panelists rated the 23 ADE triggers and six triggers met our criteria for baseline assessment (see Table 3). Each of the six triggers includes action-oriented logic.

Table 2.

Example of the construction of an outpatient ADE triggers using the knowledge-encoding tool

ADE Trigger Description	Criteria Used in Trigger Design	Encoded In Tool	Incorporated In Rule
Trigger: Warf Goal: Detect rapid or excessive anticoagulation to prevent bleed AE Targeted: Hemorrhagic event AE Cause: Hematologic agents Rule: [Started on warfarin within 14 days AND (INR>3.0 AND INR increased by 1 within 2 days) AND no repeat INR within 2 days] OR (Started in warfarin longer than 14 days prior AND INR>4 AND no repeat INR within 2 weeks) OR (INR>6 AND no repeat INR within 2 days) Note: INR stands for International Normalized Ratio	Action-oriented Logic	✓	✓
	Susceptibilities/ Resistance to AE	✓
	Markers of Other Explanations
	Timing that Impacts Likelihood of Causation
	Markers of Unrecognized Iatrogenic Harm	✓	✓
	Markers of Recognized Harm	✓	✓
	Sequences of Markers that Impact the Likelihood that Effect Results from Specific Cause or that AE is Serious	✓	✓

Table 3.

List of outpatient ADE triggers eligible for baseline assessment

Trigger	Trigger Logic
Creat	[New order or increase in (direct GFR reducer OR volume reducer OR nephrotoxin) within (1–5 days OR 1 day to last creatinine measure) AND (No new trimethoprim within 5 days)] AND (No decrease in any meds above since the last creatinine measure OR no repeat order for creatinine) AND (>25% reduction in creatinine clearance since initiation or increase of above med AND resulting creatinine clearance < 50)
Klow	Use of potassium reducer AND [K <3.0 OR (K < 3.5 AND K decreased by >15%) versus previous measurement] AND (No new potassium raiser OR decreased potassium reducer) within 5 days of triggering potassium result
Khigh	[(K+>5.5 and up by >10% since last measurement) OR (K+>6.0)] AND (Potassium raiser active OR Potassium reducer discontinued 1 day to 4 weeks days prior) AND No new potassium reducer OR decrease in potassium raiser within 5 days of triggering result
BMT	(On bone-marrow-toxic drug with a course more than 2 weeks AND No chemotherapy within 2 weeks) AND [(WBCs<2,500 AND decrease from before course by more than 2,000) OR (WBCs<2,000 AND decrease from before course by more than 1,000) OR (Platelets<50k AND decrease by 75k within 1 week)] AND (no repeat CBC OR no decrease in drug) within 5 days of triggering result
Warf	[Started on warfarin within 14 days AND (INR>3.0 AND INR increased by 1 within 2 days) AND no repeat INR within 2 days] OR (Started in warfarin longer than 14 days prior AND INR>4 AND no repeat INR within 2 weeks) OR (INR>6 AND no repeat INR within 2 days) Note: INR stands for International Normalized Ratio
Gork	Active prescription of sedative hypnotic including anticholinergic AND Subsequent diagnosis of (dementia, fall, delirium)

Discussion

The goal of trigger development is to efficiently and effectively detect AEs to prevent patient harm in real-time and to aid in system-wide surveillance that can lead to process improvement¹. The incorporation of clinical logic into triggers designed to work with electronic patient data may reduce the time needed to follow up on activated triggers, while keeping false negatives to a minimum.

One of the strengths of the knowledge-encoding tool is that the input fields for developing the trigger reflect the logic behind the causality assessment of the ADE. As the clinician described the causes and effects, they could see similar triggers from the literature on the right side of the screen. The inclusion of this information supports the consistency requirement in the Bradford-Hill criteria for causality¹⁸. The knowledge-encoding tool also employed fields related to specificity and temporality, which are both critical to establishing causality. Clinicians using the tool processed the research evidence and applied experience to judge the various scenarios in which a patient’s information would be consistent or inconsistent with the causal pathway between agent and effect.

Research and clinical experience enable chart reviewers to discern causality when identifying AEs. In addition to increasing the accuracy of the triggers, the incorporation of clinical logic may also increase the reliability of trigger evaluation. Chart review reliability is typically poor but improves with increased detail in the abstraction process¹⁹. The knowledge-encoding tool increases the specificity of the trigger thereby eliminating some of the decision-making required by reviewers and improving the reliability of ADE identification.

The clinical input obtained through the knowledge-encoding tool also allows users to understand how the trigger can be applied to improve patient safety. The field on action-oriented logic and the description of how to identify an iatrogenic event leads to triggers that can be used to identify immediate patient harm and intervene as appropriate. Non-interventionist triggers can be applied retrospectively to support system-wide efforts to improve processes of care.

The construction of triggers is limited by the extent to which patient data is available electronically. While the epidemiological perspective may be well understood, the data may not be available to apply this type of reasoning in trigger rules. Specifically, electronic patient notes prove quite difficult to search without human chart review. Advancements in natural language processing and other text-searching tools increase the capacity of trigger rules to include electronic notes in trigger detection^{3, 20}. In the future, triggers may be designed to detect a wider range of AEs using detailed notes data.

A limitation of this research is that the outpatient triggers developed with the knowledge-encoding tool have not been tested. The next stage of the AHRQ trigger project is a baseline assessment of the six ADE triggers using structured chart review by trained pharmacist abstractors. A sample of trigger positive and trigger negative cases from VA, BMC and Intermountain Healthcare patient data will be reviewed. Rates of ADEs detected as well as specificity and sensitivity will be measured. Trigger algorithms will be modified to obtain optimal sensitivity and specificity in identifying ADEs.

Future research should explore how triggers can capture additional information related to the epidemiology of ADE development. This information can be used to optimize the balance between specificity and sensitivity. As triggers become more effective in detecting ADEs, researchers should also consider other factors affecting system-wide trigger adoption, including data availability and cost of implementation.