Validation of a Reduced Set of High-Performance Triggers for Identifying Patient Safety Incidents with Harm in Primary Care: TriggerPrim Project

Abstract

Objective

The aim of the study was to construct and validate a reduced set of high-performance triggers for identifying adverse events (AEs) via electronic medical records (EMRs) review in primary care (PC).

Methods

This was a cross-sectional descriptive study for validating a diagnostic test. The study included all 262 PC centers of Madrid region (Spain). Patients were older than 18 years who attended their PC center over the last quarter of 2018. The randomized sample was n = 1797. Main measurements were as follows: (a) presence of each of 19 specific computer-identified triggers in the EMR and (b) occurrence of an AE. To collect data, EMR review was conducted by 3 doctor-nurse teams. Triggers with statistically significant odds ratios for identifying AEs were selected for the final set after adjusting for age and sex using logistic regression.

Results

The sensitivity (SS) and specificity (SP) for the selected triggers were: ≥3 appointments in a week at the PC center (SS = 32.3% [95% confidence interval {CI}, 22.8%–41.8%]; SP = 92.8% [95% CI, 91.6%–94.0%]); hospital admission (SS = 19.4% [95% CI, 11.4%–27.4%]; SP = 97.2% [95% CI, 96.4%–98.0%]); hospital emergency department visit (SS = 31.2% [95% CI, 21.8%–40.6%]; SP = 90.8% [95% CI, 89.4%–92.2%]); major opioids prescription (SS = 2.2% [95% CI, 0.0%–5.2%]; SP = 99.8% [95% CI, 99.6%–100%]); and chronic benzodiazepine treatment in patients 75 years or older (SS = 14.0% [95% CI, 6.9%–21.1%]; SP = 95.5% [95% CI, 94.5%–96.5%]).

The following values were obtained in the validation of this trigger set (the occurrence of at least one of these triggers in the EMR): SS = 60.2% (95% CI, 50.2%–70.1%), SP = 80.8% (95% CI, 78.8%–82.6%), positive predictive value = 14.6% (95% CI, 11.0%–18.1%), negative predictive value = 97.4% (95% CI, 96.5%–98.2%), positive likelihood ratio = 3.13 (95% CI, 2.3–4.2), and negative likelihood ratio = 0.49 (95% CI, 0.3–0.7).

Conclusions

The set containing the 5 selected triggers almost triples the efficiency of EMR review in detecting AEs. This suggests that this set is easily implementable and of great utility in risk-management practice.

Knowing patient safety incidents with harm, termed adverse events (AEs), is essential for their prevention.¹ Electronic medical record (EMR) review is one of the most used learning methods to accomplish this,^2,3 although it requires to be complemented with information obtained with other tools (e.g., reporting systems, safety indicators).⁴ Given the low prevalence of AEs in primary care (PC), with only 1 to 3 AEs per hundred records reviewed,^5,6 trigger tools have been designed to make EMR review more efficient for both researchers (measurements) and clinicians (learning). The Institute for Healthcare Improvement (IHI) developed the Global Trigger Tool^7,8 for detecting any type of AEs in hospitals and the Outpatient Adverse Event Trigger Tool⁹ for ambulatory patients later in 2006. The National Health Service Education for Scotland (NES) also devised a specific adaptation of the latter for the PC setting in 2010.¹⁰

Numerous descriptive studies about trigger tools have been performed in hospitals. However, the research conducted on global AEs in PC is more limited.^11–13 Trigger tools have been reported as an effective method for improving patient safety,^14–16 although few practical experiences of use by clinicians are published,¹⁷ including an extensive one in Scotland.^18,19 This is likely due to practical difficulties, such as the long time required to do it.²⁰ The present work aimed to reduce the required time of this methodology by identifying a reduced set of triggers with high positive predictive value (PPV). Several studies have evaluated a reduced set of triggers,^11–13 although those were not validated as a specific set.

Conversely, studies conducted in PC mostly focus on the usefulness of trigger tools for identifying specific AEs, which are mainly related to medication^21–23 or diagnosis.^24,25 Furthermore, the majority of the referred studies measured the presence of AEs in records with triggers and therefore only calculated the predictive ability of the triggers in EMRs that contain at least one of such events. The frequency and characteristics of AEs undetected by triggers are unknown in these studies, therefore jeopardizing the complementary role of EMR for knowing AEs, which highlights the need for validating such triggers sets.

Validation studies are scarce and most of them are performed in the hospital setting. These studies also measure undetected AEs in EMRs that do not contain triggers.^26–28 Validation studies can also be found in PC where, even if AEs are measured in EMRs with and without triggers, only a subsample of them is assessed.^21,23 The authors could only find one study²⁹ that measured the AEs that were not detected using the trigger tool methodology in the overall sample. That study, which also selected and validated a reduced trigger set, reported having detected only one AE in an EMR without triggers, likely due to the small sample size.

The present study aimed at constructing and validating a reduced set of high-performance triggers in identifying AEs via EMR review in PC.

METHODS

An observational, descriptive, cross-sectional study was designed for validating a diagnostic test via EMR review. The study was conducted over the last trimester of 2018 in the 262 PC centers of the Madrid region (Spain), which serve a population of more than 6.5 million citizens.

Study Subjects

The eligible population was subjects 18 years or older at the baseline who received health care at any of the included PC centers during the study period (N = 2,743,719).

Based on previous research,²⁹ the sensitivity and specificity were expected to be approximately 50% and 80%, respectively. The sample size was calculated to obtain sensitivity and specificity values with an absolute precision of 8%. Considering confidence intervals (CIs) of 95% and 5% of missing data, the estimated sample size was 2000 EMRs. The study sample was obtained by simple randomization.

Study Variables

Test to be assessed was the presence of a trigger in the EMR.

For the selection of eligible triggers, an initial list was obtained through bibliographic review. Triggers proposed by the IHI⁹ and NES¹⁰ were incorporated, as well as others suggested in the literature.¹² Of these, only specific computer-identified triggers were selected to guarantee their subsequent implementation and viability, as pointed out in other studies.^21–25 For that, the following requirements were established: (1) data had to be accessible from electronic records; (2) data had to be normalized and exploitable; and (3) the registry normalization had to match the trigger definition. Subsequently, those triggers deemed nonapplicable in the health system where the study was conducted were discarded using the nominal group technique with 15 experts from the research team. Finally, 19 triggers were included in the study (the initial list of triggers, both included and discarded, is shown in Table 1).

TABLE 1.

List of Triggers: Initial and Included/Excluded

Triggers Included in the Study
Change of assigned healthcare professional9,13
≥ Three appointments in a week at the PC center9–12,29
INR > 59–13
Ictus/TIA8
Pressure ulcer8
Record of adverse reaction to medication10,11,29
Bone fracture ≥75 y old (McKay 2013)
Hospital admission9–13,29*
Readmission to hospital <30 d12
Hospital emergency department visit9–12,29
≥6 active prescriptions9,11,13,29
Major opioid prescription12
Chronic benzodiazepine treatment in patients ≥75 y old12
Inappropriate medication regime ≥75 y old (McKay 2013)†
Prescription of NSAIDs + ACEi/ARBII + diuretics in patients ≥65 y old (Margham 2018)
Glycemia <50 mg/dL9,11,13
Hemoglobin <9 g/dL9–13
Glomerular filtration (estimated) < 30 mL/min9–13,29
Potassium <3 or >6 mmol/L9,11,13

Excluded Triggers	Requisite Not Met (Cause for Exclusion)
	Access to Electronic Record	Normalized and Exploitable Data	Record Matches Definition	Applicable Health System
Newly diagnosed with cancer9		x	x
Surgical procedure9,13	x
Complaint or claim9,13		x
Medication discontinuation9–11,29			x
Permanent home nurse9				x
High-priority code10				x

Bibliographic references that recommend each trigger were recorded.

*“Hospital discharge” during the study period was recorded.

†Medications as per the BEERS and STAR/STOP criteria were recorded.

ACEi, angiotensin-converting enzyme inhibitors; ARBs, angiotensin II receptor blockers; INR, international normalized ratio; NSAIDs, nonsteroid anti-inflammatory drugs; TIA, transient ischemic accident.

Reference criterion: Presence of an AE in the EMR.

A search for AEs was performed in all the sampled EMRs, whether containing any of the included triggers. The presence of an AE was defined as the identification of any AE in the EMR over the study period, according to the World Health Organization definition for AEs.¹ To facilitate the reviewers’ work, the 6-point scale from the APEAS study⁶ was used, and a score of ≥2 points was considered to be indicative of the presence of an AE. The severity of the AE, based on the classification by the National Reporting and Learning System of the National Health Service (NHS),³⁰ was also recorded whenever present. The avoidability and nature of AEs were also recorded.

Finally, sociodemographic variables such as age and sex were also collected, as well as the time required for the EMR review.

Data Collection

The presence of a trigger was determined via automated retrieval from electronic records. The searched sources were as follows: the Madrid PC medical records database (AP-Madrid), where each patient has a single EMR and every EMR is linked to a single patient; the unified prescription system (MUP); the strategic basic-information database for health settings (CIBELES); and the minimum basic data set (CMBD), a clinical and administrative database of obligatory compliance for the hospitals of the Spanish National Health System. Data extraction was performed using Oracle SQL Developer software and Access (Microsoft).

All EMRs, both containing triggers or not, were reviewed to identify AEs. The profile of reviewers varies widely among the existing studies.¹³ In this work, the reviewers were organized into 3 teams, each comprising a physician and a nurse with expertise in patient safety. In addition, they received specific training on the trigger tool methodology and were provided with a procedures manual detailing the steps to follow during the review of EMRs.

Each of the 3 teams functioned as a single reviewer. The interrater agreement between the three was analyzed for the variable “presence of an AE” in a subsample of 60 EMRs. The calculated Fleiss κ was 0.512 (95% CI, 0.508–0.516), which is along the lines of that reported in previous research.¹³ To improve the interrater concordance between teams, weekly consensus meetings were established during the EMR review process. Each team presented the cases giving rise to doubts and a criterion was agreed upon by the 3 teams. The agreed criteria were then adopted by all teams to clarify such concerns and to be applied when dealing with similar cases.

Data Analysis

The prevalence of AEs over the study period was calculated. In addition, the prevalence of each of the 19 assessed triggers was obtained, as well as sensitivity, specificity, PPV, and negative predictive value (NPV), all with their corresponding 95% CIs.

To construct the set of triggers with the best performance in detecting AEs, a multivariate analysis was conducted using a logistic regression model. The dependent variable was the presence of an AE and the independent variables were the presence of each trigger, age, and sex. The resulting set, which was termed TriggerPrim, comprised those triggers with a statistically significant odds ratio (OR). An EMR was classified as TriggerPrim+ whenever one trigger from the set was present. Otherwise, the EMR was considered as TriggerPrim−.

Values of sensitivity, specificity, PPV, NPV, and positive and negative likelihood ratios were calculated for the TriggerPrim set, with their corresponding 95% CIs. The severity of AEs that went undetected using the TriggerPrim set was also evaluated.

Following law requirements, all the data extracted via EMR review were anonymized, so the patient’s identity was definitively separated from personal data.

SPSS version 26.0 (IBM Corp, Armonk, New York) statistical software was used for all analyses.

RESULTS

Complete data were obtained from 1797 EMRs. Figure 1 shows the flowchart of the EMR selection process.

FIGURE 1.

Flowchart of the EMR selection process.

A total of 1080 triggers were found in 673 EMRs and 97 AEs were identified in 93 EMRs. Of note, all sampled EMRs, both with and without triggers, were reviewed for AE detection. Table 2 displays the EMR distribution by age and sex for the overall sample and also disaggregated by EMRs with and without AEs as well as with or without any of the 19 evaluated triggers.

TABLE 2.

Characteristics of Patients, Globally and by Presence or Absence of Adverse Events and Any of the Evaluated Triggers

	n (%)
	Global (n = 1797)	No Adverse Event (n = 1704)	≥1 Adverse Event (n = 93)	P	No Trigger (n = 1124)	≥1 Trigger (n = 673)	P
Sex
Women	1053 (58.6%)	991 (58.2%)	62 (66.7%)	0.105	636 (56.6%)	417 (62%)	0.025
Men	744 (41.4%)	713 (41.8%)	31 (33.3%)		488 (43.4%)	256 (38%)
Age
18–30 y	228 (12.7%)	222 (13.0%)	6 (6.5%)		168 (14.9%)	60 (8.9%)
31–45 y	416 (23.1%)	399 (23.4%)	17 (18.3%)		308 (27.4%)	108 (16.0%)
46–65 y	590 (32.8%)	561 (32.9%)	29 (31.2%)	0.000	431 (38.3%)	159 (23.6%)		0.000
66–75 y	300 (16.7%)	287 (16.8%)	13 (14.0%)		166 (14.8%)	134 (19.9%)
≥75 y	263 (14.6%)	235 (13.8%)	28 (30.1%)		51 (4.5%)	212 (31.5%)

The overall prevalence of AEs was 5.18% (95% CI, 4.16%–6.20%). Avoidable AEs were 71.3% (95% CI, 62.1%–80.5%) of all the recorded AEs. In terms of severity, 7.4% (95% CI, 2.1%–12.7%) were classified as severe and 31.9% (95% CI, 22.4%–41.4%) as moderate. The most frequent types were prescription errors (28.7%; 95% CI, 19.5%–37.9%), drug administration errors by patients (17.0%; 95% CI, 9.4%–24.6%), and diagnostic errors (11.7%; 95% CI, 5.2%–18.2%).

Table 3 shows the values of the prevalence, sensitivity, specificity, PPV, and NPV calculated for all the assessed triggers, with their corresponding 95% CIs.

TABLE 3.

Prevalence, Sensitivity, Specificity, and Positive and Negative Predictive Values of Each Evaluated Trigger

	Prevalence		Sensitivity	Specificity	PPV	NPV
Trigger	n (%)	95% CI	% (95% CI)	% (95% CI)	% (95% CI)	% (95% CI)
Change of assigned healthcare worker	78 (4.3%)	(3.4%–5.2%)	5.4% (0.8%–10.0%)	95.7% (94.7%–96.7%)	6.4% (1.0%–11.8%)	94.9% (93.9%–95.9%)
≥3 appointments in 1 wk at PC center	152 (8.5%)	(7.2%–9.8%)	32.3% (22.8%–41.8%)	92.8% (91.6%–94.0%)	19.7% (13.4%–26%)	96.2% (95.3%–97.1%)
INR > 5	4 (0.2%)	(0%–0.4%)	4.3% (0.2%–8.4%)	100% (99.9%–100%)	100% (21.2%–100%)	95.0% (94.0%–96.0%)
Ictus/TIA	1 (0.1%)	(0%–0.2%)	0% (0%–0.1%)	99.9% (99.75%–100%)	0% (0%–87.1%)	94.8% (93.8%–95.8%)
Pressure ulcer	1 (0.1%)	(0%–0.2%)	0% (0%–0.1%)	99.9% (99.75%–100%)	0% (0%–87.1%)	94.8% (93.8%–95.8%)
Record of adverse reaction to medication	1 (0.1%)	(0%–0.2%)	0% (0%–0.1%)	99.9% (99.75%–100%)	0% (0%–87.1%)	94.8% (93.8%–95.8%)
Bone fracture in patients ≥75 y old	1 (0.1%)	(0%–0.2%)	1.1% (0.0%–3.2%)	100% (99.9%–100%)	100% (17.2%–100%)	94.9% (93.9%–95.9%)
Hospital admission	65 (3.6%)	(2.7%–4.5%)	19.4% (11.4%–27.4%)	97.2% (96.4%–98.0%)	27.7% (16.8%–38.6%)	95.7% (94.7%–96.7%)
Readmission to hospital within 30 d	3 (0.2%)	(0%–0.4%)	2.2% (0.0%–5.2%)	99.9% (99.75%–100%)	66.7% (13.4%–100%)	94.9% (93.9%–95.9%)
Hospital emergency department visits	186 (10.4%)	(9.0%–11.8%)	31.2% (21.8%–40.6%)	90.8% (89.4%–92.2%)	15.6% (10.4%–20.8%)	96.0% (95.0%–97.0%)
≥6 active prescriptions	294 (16.4%)	(14.7%–18.1%)	29.0% (19.8%–38.2%)	84.3% (82.6%–86.0%)	9.2% (5.9%–12.5%)	95.6% (94.6%–96.6%)
Major opioids prescription	6 (0.3%)	(0%–0.6%)	2.2% (0.0%–5.2%)	99.8% (99.6%–100%)	33.3% (0.0%–71.0%)	94.9% (93.9%–95.9%)
Chronic benzodiazepines treatment in patients ≥75 y old	89 (5.0%)	(4.0%–6.0%)	14% (6.9%–21.1%)	95.5% (94.5%–96.5%)	14.6% (7.3%–21.9%)	95.3% (94.3%–96.3%)
Inappropriate medication in patients ≥75 y old	199 (11.1%)	(9.6%–12.6%)	20.4% (12.2%–28.6%)	89.4% (87.9%–90.9%)	9.5% (5.4%–13.6%)	95.4% (94.4%–96.4%)

Five of the triggers included in this study were not present in any EMR: prescription of nonsteroid anti-inflammatory drugs + angiotensin-converting enzyme inhibitors/angiotensin II receptor blockers + diuretics in patients ≥65 years of age; glycemia <50 mg/dL; hemoglobin <9 g/dL; glomerular filtration (estimated) < 30 mL/min; and potassium <3 or >6 mmol/L.

INR, international normalized ratio; TIA, transient ischemic attack.

Those triggers with a significant OR in the multivariate analysis were selected for the TriggerPrim set. The 5 triggers that were finally included in the TriggerPrim set were as follows: prescription of major opioids (OR, 6.69; 95% CI, 1.02–43.71), ≥3 appointments in a week at the PC center (OR, 3.91; 95% CI, 2.33–6.56), hospital admission (OR, 3.52; 95% CI, 1.76–7.02), hospital emergency department visit (OR, 2.77; 95% CI, 1.6–4.78), and chronic treatment with benzodiazepines in patients 75 years or older (OR, 2.61; 95% CI, 1.25–5.45) (Fig. 2).

FIGURE 2.

Triggers associated with the presence of an adverse event in the medical record, adjusted for age and sex. TriggerPrim set. Nagelkerke R² = 0.162; Hosmer-Lemeshow test P = 0.741.

Fifty-six AEs were detected in 387 EMRs classified as TriggerPrim+. Table 4 shows the sensitivity, specificity, predictive values, and likelihood ratios of the TriggerPrim set for identifying AEs. The calculated positive and negative likelihood ratios were 3.13 (95% CI, 2.29–4.28) and 0.49 (95% CI, 0.34–0.71), respectively.

TABLE 4.

Sensitivity, Specificity, Predictive Values, and Likelihood Ratios of the TriggerPrim Set for Identifying AEs

Sensitivity	Specificity	PPV	NPV	PLR	NLR
60.2% (95% CI, 50.2%–70.1%)	80.8% (95% CI, 78.8%–82.6%)	14.6% (95% CI, 11.0%– 18.1%)	97.4% (95% CI, 96.5%– 98.2%)	3.13 (95% CI, 2.3–4.2)	0.49 (95% CI, 0.3–0.7)

NLR, negative likelihood ratio; PLR, positive likelihood ratio.

Seven severe AEs were found in the overall sample, of which only one was present in an EMR classified as TriggerPrim−. This severe AE in a patient with chronic sequelae (pain, altered sensitivity, and functional impairment) was detected in PC but secondary to arthrodesis of the vertebrae performed in a hospital, with no data suggesting that it was an avoidable AE.

The mean review time for EMRs classified as TriggerPrim+ was 5.62 (95% CI, 5.46–5.78) minutes, yielding 38.7 minutes spent per AE found.

DISCUSSION

Main Findings

This study constructed the TriggerPrim set incorporating the 5 triggers with the best predictive capacity for identifying AEs in PC. This reduced set was composed of the following triggers: ≥3 appointments in 1 week at the PC center, hospital admission, hospital emergency department visit, prescription of major opioids, and chronic benzodiazepine treatment in patients 75 years or older. Furthermore, the TriggerPrim set was validated. The values obtained for the positive and negative likelihood ratios were 3.13 and 0.49, respectively, which suggest significant utility for identifying AEs via EMR review. The PPV of the set was estimated to be 14.6% versus 5.18% of global AE prevalence, which means that the TriggerPrim tool triples the probability of identifying AEs via EMR review. On the other hand, the absence of triggers included in the TriggerPrim set nearly halves the probability of detecting an AE in the EMR (NPV = 97.4%).

According to published data,^11–13,29 the manual reviewing of records without any trigger tool might take 90 to 150 minutes per detected AE versus 50 to 80 minutes using “learning model” trigger methods for clinicians (as recommended by the IHI or NHS). Consequently, trigger tools significantly reduce the time spent per detected AE. Specifically, the time spent using the TriggerPrim tool was half that with a usual trigger method (38 versus 50–80 minutes per detected AE, respectively) and approximately one third of the time spent when not using any trigger tool.

Interpretation in the Context of Current Literature

The majority of studies assessed the presence of an AE only in EMRs with a trigger,^{11–13,22,24,25} and hence the NPV could not be estimated. Similar to Eggleton and Dovey,²⁹ the present work analyzed EMRs both with and without triggers, which allowed calculating both the PPV and NPV, as well as sensitivity, specificity, and likelihood ratios. All these values served to perform a validation of the included triggers.

In the analysis of triggers conducted individually, those related to abnormal analytical results were not detected in any of the sampled EMRs, except for the trigger “international normalized ratio > 5.” A very low prevalence was also observed for several other triggers: ictus/transient ischemic attack, pressure ulcer, bone fracture in patients 75 years or older, and adverse reaction to medication. The low prevalence of all the mentioned triggers implies their limited capability for enhancing the performance of the EMR review process.

All the evaluated triggers showed high specificity and low prevalence values. Figures for sensitivity varied notably among them, with the most prevalent triggers showing the highest sensitivity. No published data were found with which to compare these findings. The only study found that examined the presence of AEs in all EMRs both with and without triggers²⁹ did not express the results as sensitivity and specificity of each trigger. The remaining studies evaluated AEs in only EMRs with triggers, so these outcomes could not be calculated.

The studies found presented their outcomes by reporting the PPV for each trigger individually. The PPVs estimated in this work were slightly higher than those reported in other research,^{11–13,21,22} except for the triggers “hospital emergency department visit” and “≥6 active prescriptions,” which were measured based on very diverse criteria in previous studies. The larger number of AEs in EMRs with triggers this study identified could be due to the definition of AE adopted, the detection capacity of the reviewers, or the used methodology.

On their own, none of the assessed triggers displayed high values of both sensitivity and specificity or high figures of both PPV and NPV. This implies that no trigger on its own shows good performance in identifying AEs via EMR review.

Former studies have approached the selection of a reduced set of triggers.^11,13,22,29 However, selecting a suitable set of triggers that are useful in EMR review for the identification of AEs is complex. Other studies that did not calculate NPVs and performed manual trigger searches sought to reach a balance between PPV and feasibility by consensus. On the contrary, the present study was capable of estimating NPVs and all the evaluated triggers were available by computerized data extraction. Therefore, statistical-epidemiological criteria were followed. Because finding a correct balance between PPV and NPV is challenging, the researchers decided to select those triggers with a statistically significant OR in a logistic regression model, adjusted by age and sex.

The 5 triggers that finally formed the TriggerPrim set were as follows: ≥3 appointments in 1 week at the PC center, hospital admission, hospital emergency department visit, prescription of major opioids, and chronic benzodiazepine treatment in patients 75 years or older. The first three were related to the volume of medical care use and the occurrence of AEs was closely related to the frequency of use of health services. Therefore, these 3 variables might seem to be surrogates and not triggers. However, these specific triggers are not the cause but the consequence of many AEs, namely, the AE causes the patient to attend the center on multiple occasions or the emergency department or get hospitalized. Nevertheless, triggers are useful for increasing the likelihood of detecting an AE in an EMR, irrespective of being the cause, consequence, or merely an association. This made these 3 triggers be defined as such by the IHI, NES, and other research.^9–13,29

The sensitivity for the TriggerPrim set was slightly superior to that reported by Honigman et al²¹ and Eggleton and Dovey²⁹ (60.2% versus 58% and 51%, respectively), while the estimated specificity (81%) was similar in both studies. These values indicate a high performance of the TriggerPrim tool in identifying AEs. The PPV of the set was 14.6% (over a one-trimester period of EMR review), which was superior to the 9% reported by de Wet et al,¹¹ 7.5% by Honigman et al,²¹ or 1.4% to 14.6% in a review by Davis et al.¹³ A study by Singh et al²² obtained a higher PPV (25%) but it corresponded to a 1-year review period. These data again point to the high performance of the TriggerPrim set. The prevalence and characteristics of the AEs briefly described in the results section had been previously published in detail.³¹

This reduced set containing only 5 triggers triples the efficiency of the EMR review methodology for identifying AEs. This implies that by reviewing only EMRs classified as TriggerPrim+, an AE will be found in every six EMRs, compared with only one AE in every 20 EMRs. Furthermore, the time spent by the reviewers per AE found was half of that used with the usual trigger tool “learning model.” In addition, by not reviewing those EMRs classified as TriggerPrim−, only 2 AEs in a hundred EMRs will go undetected, none of which would be an avoidable severe event.

Strengths and Limitations

Among the strengths of this study is that a limited set of triggers was selected that is useful for improving EMR review, instead of merely conducting a description of the predictive value of each trigger. Furthermore, this selection was made using a reproducible methodology and the set was validated. Another strength is that it assesses all types of AEs in all patients instead of focusing on specific events, such as medication errors or high-risk patients. Using specific computer-identified triggers adds value to this tool by making its automatic implementation viable. The large sample size and multicentric design that reach a complete geographic region are also strengths to note.

In terms of limitations, underreporting is a potentially existing bias, inherent to the methodology of EMR review, as already observed in comparable studies.

The estimated Fleiss κ value is considered acceptable for this type of study that sought all types of AEs in all patients, which points to a good interrater agreement. In addition, weekly consensus meetings were held throughout the fieldwork to improve concordance between reviewers.

The study period was set for the last quarter of 2018. However, because of the lengthy process of obtaining EMR access authorizations, data extraction, and recruitment of reviewers, followed by the COVID-19 pandemic, the fieldwork could not be conducted until 2021. This resulted in losses of 10.4% of the initial sample, because of the lack of access to EMR for 2 different reasons. The transfer of residence outside the Madrid region, a reason not directly related to the occurrence of an AE or trigger, accounted for more than half of the cases. Nevertheless, no differences were observed when comparing the characteristics of these patients to those included in the final sample. The remaining missing data (4.6% of the initial sample) correspond to patients deceased between enrollment and the fieldwork. The proportion of subjects 75 years or older was larger compared with the initial sample (45.8% versus 14.6%, respectively), potentially entailing a small underestimation of the prevalence of both AEs and triggers, which were higher in this aged-population segment. However, the fact that both figures were underestimated to a similar extent reduces the likelihood of a bias in the correlation between triggers and AEs.

The TriggerPrim set was validated using the same data with which it was constructed. However, this study did not aim to design a predictive model but a tool that served to increase the likelihood of finding an AE in EMRs. Therefore, the adopted design was the validation of a diagnostic test. The standard method for such study design consists of constructing and validating the diagnostic test using the same data set, similarly to the present work and other research.^{21,23,26–29}

Implications for Policy, Practice, and Research

The practical implementation of a trigger tool in EMR review entails identifying AEs, learning from them, and taking corrective measures.³² The main implication for the implementation of the TriggerPrim tool in clinical practice is that it makes EMR review a cost-effective method for identifying AEs. Moreover, as the tool seeks specific computer-identified triggers, obtaining a sample of EMRs that contain any of the triggers from the TriggerPrim set is easy and can even be automated. In addition, the detection of these triggers in the EMRs does not require any review time as it is computerized. For example, as proposed in the methodology by the NHS,¹⁰ a Patient Safety Unit or a PC team reviewing 50 EMRs classified as TriggerPrim+ would identify 7 to 8 AEs to learn from; in contrast, only 1 in 50 EMRs classified as TriggerPrim− would contain an undetected AE, which would not be an unavoidable severe event. The average review time of less than 6 minutes per EMR means that a 10-member team, for example, divided into 5 teams of reviewers, could review all 50 EMRs in 1 hour. The time spent per detected AE would double that with the regular trigger tool learning model and triple it without any trigger tool. This suggests that the tool can be incorporated into work routines as a periodic task (e.g., quarterly), making of TriggerPrim a permanent-learning tool, as suggested by de Wet.¹⁸

The study period was set prior to the COVID-19 pandemic, which has changed the healthcare model, entailing new risks for patient safety. Further research should be conducted to identify newly emerged triggers and types of AE. Although the approach of this study aimed at enhancing the performance of EMR review, research on predictive models that include comorbidity and use of a health system would be desirable to validate algorithms that trigger alarms in specific patients. Because the developed tool can be implemented in practice, its implementation should also be evaluated.

CONCLUSIONS

The present study constructed a set of triggers, termed TriggerPrim, incorporating only those with the highest positive and negative predictive values. The set contains the following triggers: ≥3 appointments in a week at the PC center, hospital admission, hospital emergency department visit, prescription of major opioids, and chronic benzodiazepine treatment in patients 75 years or older. A validation of the TriggerPrim set was conducted, finding the set to triple the probability of identifying an AE via EMR review (1 AE in 6 EMRs) and revealing a NPV of 97.4%. Furthermore, TriggerPrim halves the time spent in EMR review per detected AE compared with usual trigger methods. Finally, this tool could be implemented in patient safety units or PC teams, establishing an adequate periodicity and using it as a normalized system for learning and improvement.