The Mental Health Trigger Tool: Development and Testing of a Specialized Trigger Tool for Mental Health Settings

Abstract

Objective

Tools generally used in measuring patient safety incidents in general healthcare settings are not considered suitable for mental health settings. The aim of this study was to develop and evaluate a specialized trigger tool for mental health settings that could detect both traditionally defined adverse events (AEs) and other mental health–related patient safety incidents (MHPSIs).

Methods

We first defined and categorized AEs and MHPSIs based on existing literature and then developed a trigger list, initially consisting of 50 items, which was subsequently reduced to 25 items after a pilot study. We then explored the properties of this final 25-item trigger tool, the Mental Health Trigger Tool (MHTT), through a retrospective review of 515 patient records with a two-stage review process similar to Global Trigger Tool methodology. We used findings of an alternative method of review which consisted of page-to-page reviews of patient records in the analysis of properties of MHTT.

Results

Using the MHTT, at least one AE was identified in 98 patient records (19%) and at least one MHPSI was identified in 58 patient records (11%). The MHTT had a sensitivity of 98.6% and its specificity was 100%. The probability of finding an AE/MHPSI when any trigger was detected in a patient record with MHTT was 33.8% and that of individual triggers ranged from 0% to 100%.

Conclusions

The MHTT may offer an effective, practical, and easy-to-use method in identifying and measuring safety incidents in mental health settings.

A number of influential studies have drawn public attention to safety in healthcare settings.^1,2 An important measure of patient safety is the rate of adverse events (AEs). The Institute of Healthcare Improvement (IHI) defines an AE as an “unintended physical injury resulting from or contributed by medical care that requires additional monitoring, treatment or hospitalization or that results in death.”³ The AE rate in general acute care hospitals range from 3.7% to 33.2% depending on the methodology used for its measurement.^1,2,4

Global Trigger Tool (GTT), developed by the IHI, is a practical and resource-efficient method for measuring AE and available evidence suggests that the GTT may detect up to 10 times more AEs compared with more traditional methods such as voluntary reporting.^3,4 The GTT follows “trigger tool methodology,” which involves time-limited retrospective reviews of randomly selected patient records repeated at regular intervals. The GTT helps to detect AEs by identifying the presence of triggers, defined as “an occurrence, prompt or flag found on review of the medical record that ‘triggers’ further investigation to determine the presence or absence of an AE.”³

Patient Safety in Mental Health Settings

Although some AEs such as “patient fall” and adverse drug events (ADEs) may be common to both general healthcare and mental health settings, AEs related to surgical or other invasive procedures are unlikely to occur in mental health settings.^5–7 A higher rate of ADEs, in particular, extrapyramidal adverse effects, is seen in psychiatric settings compared with medical or surgical settings.^7–9 With regard to hospital-acquired infections, psychiatric units are more similar to long-term residential facilities and these may sometimes be related to personal hygiene issues.¹⁰

Another distinct feature of mental health settings is that certain important safety incidents involving patients may be related to their own behaviors or the behavior of others. For example, disruptive and aggressive behaviors, self-harming behaviors and absconding or missing incidents were some of the most commonly reported patient safety incidents in mental health settings in the United Kingdom.¹¹ These types of incidents are considered as important markers of patient safety in inpatient psychiatric settings and are the results of a combination of factors such as patient’s mental condition and effectiveness of clinical interventions.^11–14 However, determining the role of medical care in contributing to such incidents can be difficult.¹² These incidents may not result in physical harm to the patient involved in the incident, a prerequisite to be classified as AEs. Similar incidents may be rare in general healthcare settings, and they may not be identified as AEs because their description may not conform to the traditional definition of AEs.^1–3

Lack of clarity in definition of patient safety incidents and subsequent lack of systematic studies have resulted in significant knowledge gap in understanding patient safety issues in mental health settings.⁵ A recent work done by Marcus et al¹⁵ has attempted to systematically define patient safety events in inpatient psychiatry settings, broadly categorizing them to AEs and medical errors. Adverse events have been further classified into nondrug AEs (self-harm, assault, sexual contact, patient fall, other injuries) and ADEs. Medical errors are subclassified into medication errors and nondrug errors.

Tools currently available for measuring AEs in general healthcare settings may not be suitable for use in mental health settings. With GTT, it is clearly stated in its monograph by the IHI that inpatient psychiatric patients should be excluded because the tool’s triggers are not defined for this population.³ Although the IHI has developed an ADE trigger tool for use in mental health settings, it does not address nondrug-related AEs or other safety incidents relevant to mental health settings.¹⁶ Another important methodology used in the study of AEs in healthcare settings is the Harvard Medical Practice Study’s methodology.¹ Although several major studies have used this methodology, it may also not be suitable for mental health settings as evidenced by the exclusion of psychiatric patients from those studies.²

Currently, there are no published tools that could examine both traditionally defined AEs and other mental health–related patient safety incidents. Organizations such as National Patient Safety Agency in the United Kingdom, which gather and analyze data on patient safety incidents in mental health settings, do it through incident reporting systems, a methodology affected by significant underreporting.^11,17 Unlike in general healthcare settings, studies in mental health settings often do not examine the full range of patient safety incidents. For example, there are several studies that focus exclusively on ADEs in psychiatry.^7,9 Other studies, most of them relying on incident reports as source of data, focus on either aggressive or self-injurious behaviors and sometimes exclusively on accidents in mental health settings.^6,18–20

Further to the widespread use of the GTT, there have been several successful attempts in developing trigger tools for use in various healthcare settings.^21–24 However, similar efforts have been lacking in mental health. This article outlines the development and evaluation of a specialized trigger tool, the Mental Health Trigger Tool (MHTT), based on the principles of the IHI’s GTT and trigger tool methodology, aimed at detecting and measuring both traditionally defined AEs and other patient safety incidents relevant to mental health settings via retrospective record/chart reviews.

METHODS

The study was conducted at the Institute of Mental Health, a tertiary psychiatric hospital in Singapore. Ethical approval for the study was obtained through the Institutional and Cluster Review Boards. The study team consisted of 26 members including four staff psychiatrists, five senior residents, six senior pharmacists, and 11 senior nurses. All members were employees of the Institute of Mental Health with experience in mental health settings ranging from 3 to 30 years.

Patient Safety Incidents and Their Severity

Safety incidents that were classifiable as AEs according the definition of IHI were separately identified from patient safety incidents that were relevant to mental health settings but did not conform to the IHI’s definition of an AE.³ We identified the latter as the mental health–related patient safety incidents (MHPSIs).

Adverse Events

A list of potential AEs that are clinically relevant to psychiatric settings was prepared by the team through literature review and these items fell into four main categories (Table 1).^6,7,9,16,20 The severity of an AE was rated according to National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) Index, which rates ADEs to patients on a five-point scale: E, temporary harm to patient and required intervention; F, temporary harm to patient and required initial or prolonged hospitalization; G, permanent patient harm; H, intervention required to sustain life; and I, patient death.²⁵

TABLE 1.

List of AEs and MHPSIs

List of AEs

i) Electroconvulsive therapy–related AE: AEs that are specifically related to use of ECT including that related to anesthesia.

ii) Hospital-acquired infection: any infections that had onset after the admission to the hospital.

iii) ADEs: any clinically significant adverse reactions to medications that were identified for the first time during inpatient stay.

iv) Patient injury: any physical injury sustained during inpatient stay, for example, as result of accidents, falls, assaults by others, self-harm, or that related to mechanical or physical restraints.

List of MHPSIs

i) Aggressive behavior: any attempt or an incident of physically aggressive behavior (including sexual assault), toward other person(s) or aggressive behavior toward property. The act must be of a severity that additional clinical interventions are required to manage the patient (the aggressor), such as increased observation, use of rapid tranquilization, restraint or seclusion or transfer to a psychiatric intensive care unit or medical facility. Verbal aggression or incidents of aggression that do not require any additional clinical interventions are not included.

ii) Self-harm and suicide: any attempt or an incident of self-harm or suicide. The act must be of a severity that additional clinical interventions are required to manage the patient, such as increased observation, use of rapid tranquilization, restraint or seclusion or transfer to a psychiatric intensive care unit or medical facility. Verbal reports of ideas of self-harm or suicide or incidents that do not require any additional clinical interventions are not included.

iii) Victim of assault: an incident in which the patient is a victim of a physical or sexual assault by other(s). The act must involve physical contact between the aggressor and the victim. All incidents irrespective of whether the victims sustained a physical injury or not are to be included.

iv) Absconding: An incident of unauthorized absence by a patient from the inpatient unit irrespective of the length of the period of absence.

Mental Health–Related Patient Safety Incidents

Further to a literature review, four specific patient safety incidents that were well known as important markers of patient safety in inpatient psychiatric settings and the focus of several major initiatives in understanding and improving patient safety in mental health settings across the world were selected for inclusion in the study (Table 1).^{5,11–14,18,19} If the patient sustained a physical injury during the incidents, this was included under the AE category, “patient injury,” and the severity of the physical injury was rated using NCC-MERP index.²⁵

Initial Set of Triggers and Pilot Study

An initial set of triggers that could potentially identify AEs and MHPSIs in mental health settings was selected through a series of focus group discussions, expert consultations, including that with IHI, and review of both GTT and ADE trigger tool.^3,16 Of 80 triggers identified initially, 50 triggers were selected through a modified Delphi method, which was then used to conduct a pilot study.²⁶ A random sample of 135 discharged patient records was selected and a two-stage review process, similar to the GTT methodology, was conducted.³ Based on the findings of this pilot trial and feedback from reviewers, the trigger list was modified by eliminating triggers that identified similar AEs/MHPSIs or that were practically difficult to identify or not associated with any AEs/MHPSIs. This resulted in the final version of MHTT with a list of 25 triggers with the following four main categories: general care triggers, laboratory triggers, medication-related triggers, and behavior-related triggers (Table 2).

TABLE 2.

List of Triggers in the Final 25-Item Version of MHTT

Trigger Group	Trigger Items
General care	G1 Transfer to general hospital/medical ward/referral for medical consultation
	G2 Fall or reports of patient injuries, e.g., bruise/hematoma
	G3 Initiation or increase in frequency of monitoring of physical parameters
Laboratory	L1 Imaging studies, e.g., x-ray or CT scan
	L2 WBC <3.0 or neutrophils <1.5
	L3 Serum creatinine kinase
	L4 Raised serum prolactin (>500 mIU/L)
Medication related	M1 Abrupt reduction in dose or discontinuation of medications
	M2 Antimicrobials*
	M3 Use of rectal suppository/enema or severe constipation*
	M4 Anticholinergics*
	M5 Anti-histamines (except those given at night)*
	M6 Analgesics/anti-inflammatory drugs*
	M7 Antihyperlipidemic drugs*
	M8 Thyroxine*
	M9 Antidiabetic medications*
	M10 Rapid tranquilization
	M11 Dyskinesia/abnormal movements of limbs or body
	M12 Propranolol*
Behavior related	B1 Reports of attempted self-harm/suicide or increased observation of patient
	B2 Disturbed/aggressive/violent behavior or physical aggression by patient
	B3 Physically or sexually assaulted by others
	B4 Transfer to higher level of care in psychiatry (HDPCU [psychiatric ICU] or DAV ward)
	B5 Restraint or seclusion use
	B6 Reports of absconding or missing from hospital

*Only medications started after admission are included as triggers.

WBC indicates white blood cells.

Evaluation Study of the 25-Item MHTT

Sample Size

In the absence of comparable studies in Singapore or internationally, the following method was used in reaching the required sample size estimation: using a single proportion formula with an estimated prevalence rate of 20% of an AE/MHPSI with a precision rate of 0.04%, the estimated sample size was calculated as 385. However, we decided to increase the sample size to minimum of 500 to account for missing data and to enable further analysis of tool’s properties. Our sample size was similar to another study on validation of a trigger tool.²¹

Patient Records

Records of inpatients discharged from January 01, 2014, to June 30, 2014, were screened and selected using predefined inclusion and exclusion criteria (Table 3).

TABLE 3.

Inclusion and Exclusion Criteria

Inclusion criteria

1. Patients discharged from hospital at least 1 mo before the review 2. Age group 21–90 y 3. Latest admission period >24 h and <90 d

Exclusion criteria

1. Incomplete case records without discharge summary or discharge coding 2. Duration of admission period <24 h or >90 d 3. Patient is admitted to hospital at the time of collection of patient record.

Mental Health Trigger Tool

The 25-item MHTT developed by the study team was used for the study. An instruction manual containing the description of each trigger, the likely location of each trigger in the patient record, and the potential AE or MHPSI that each trigger may be associated with, was prepared to guide the reviewers.

Training Sessions for Study Team

First phase of training consisted of interactive sessions on GTT and trigger tool methodology. In the second phase of training, the focus was on MHTT with interactive lectures. This phase of training also consisted of a mock review sessions with each reviewer reviewing at least two patient records using the MHTT.

Reviewers and Review Process

Patient records were reviewed through a two-stage process. In the first stage, each patient record was independently reviewed by two nonphysician reviewers (pharmacist or nurse) using MHTT. They recorded presence or absence of triggers and/or AEs or MHPSIs. Reviewers were instructed to limit the review to a maximum of 20 minutes using a stop watch. In the second stage, a physician reviewer went through findings of each primary reviewer and confirmed presence of triggers and/or AEs/MHPSIs by cross-checking with the information from relevant parts of each patient record. The findings recorded by the physician reviewers were considered final and were used for analysis.

Alternative Method of Review of Patient Records

Ideally, calculation of sensitivity and specificity require comparison of outcomes identified by a tool with a robust independent method, the “criterion standard.” In our study, in the absence of an acceptable “criterion standard” method to identify AEs/MHPSIs retrospectively to allow comparison with MHTT findings, an alternative method of review of patient records was designed by the study team. This alternative review method consisted of unfocused in-depth manual review of each patient record by a physician reviewer, reading every page of the paper and electronic records. For this review, physician reviewers were instructed to actively look directly for any AEs or MHPSIs and it did not involve identification of triggers. No time limit was applied to these reviews as the emphasis was to ensure that AE/MHPSI possibly missed by MHTT method, if any, was correctly identified. For the purpose of analysis, we made the assumption that the alternative method of review, as the “criterion standard,” would detect any AE/MHPSI that was already identified by the MHTT method, and therefore, only those patient records without AE/MHPSI as per the MHTT method were reviewed by this alternative method. Reviewing only proportion of patient records using another method to analyze sensitivity and specificity of a screening tool in healthcare has been reported previously.²⁷

Statistical Analysis

The MHTT method of review (“the test”) consists of two steps. The first step is the detection of triggers, which is then followed by a second step consisting of identifying whether the patient record has an AE/MHPSI. For the calculation of sensitivity and specificity of the MHTT in comparison with the alternative method of review, we considered the MHTT method as one “test” (with two steps) and deemed it as “test positive” when the MHTT method found an AE/MHPSI in a patient record and “test negative” when it did not find AE/MHPSI.

With the MHTT method, the probability of finding an AE or MHPSI when one or more triggers were detected in a patient record was calculated as the proportion of “trigger-positive” patient records with AE/MHPSI. The probability of finding an AE or MHPSI in the presence of individual triggers was calculated as the number of times a specific trigger identified an AE/MHPSI (i.e., directly linked to the AE/MHPSI identified) divided by the total number of times that trigger was found.

The Cohen κ coefficient was used as a measure of interrater agreement between primary reviewers and each primary reviewer and physician reviewer for triggers as well as AE/MHPSI.

Analysis was performed with PASW Statistics for Windows, version 18.0 (SPSS Inc, Chicago, IL).

RESULTS

A total of 515 patient records were reviewed using the MHTT. Overall, 59% (n = 305) of patients were male and the mean age of the study population was 46.9 years (range = 21–89 y). The mean and median lengths of stay of patients were 15.7 days (range = 1–89) and 12 days, respectively.

A total of 1202 triggers were identified in 515 patient records with an average of 2.3 triggers per patient record (range = 0–11). The top three triggers detected (in descending order of frequency) were as follows: M1, Abrupt reduction in dose or discontinuation of medications (n = 157, 30.5%); M10, Rapid tranquilization (n = 155, 30.1%); and G3, Initiation or increase in frequency of monitoring of physical parameters (n = 146, 28.3%).

Using the MHTT, one or more AEs were identified in 98 patient records (19%) and one or more MHPSIs were identified in 58 patient records (11%). The alternative method of review conducted in those patient records without AE and/or MHPSI according to the MHTT review method identified two additional patient records with AE and/or MHPSI (one patient record had one AE and the other had one AE and one MHPSI). The details are given in Figure 1.

FIGURE 1.

Results of review of patient records using the MHTT.

Analysis of the overall performance of the trigger tool compared with the findings of the alternative method of review showed that the MHTT’s sensitivity was 98.6% and specificity was 100% (there were two false-negative cases with the MHTT method). The probability of finding an AE/MHPSI when any trigger was detected in a patient record with MHTT was 33.8% and that of individual triggers ranged from 0% (L3, Serum creatinine test) to 100% (L2, white blood cells <3.0/neutrophils <1.5) (Table 4).

TABLE 4.

Proportion of Individual Triggers That Were Directly Linked to AE/MHPSI

Trigger	No. Patient Records Where the Trigger Was Linked to AE/MHPSI	Total No. Patient Records With the Trigger	Probability (%) of Finding an AE/MHPSI When the Trigger Is Present
B3	6	6	100
B6	3	3	100
L2	1	1	100
B2	49	51	96
M11	19	24	79
M2	25	43	58
M12	5	9	56
M4	23	44	52
B4	7	14	50
G2	12	27	44
G1	18	63	29
M10	44	155	28
B5	39	139	28
M1	43	157	27
L4	1	4	25
L1	10	51	20
G3	22	146	15
M6	14	110	13
B1	3	25	12
M7	2	19	11
M3	5	52	10
M5	3	39	8
L3	0	8	0
M9	0	12	0

Interrater Reliability

κ for agreement on presence of triggers between primary reviewers was 0.505, which indicated moderate agreement. κ for agreement between each primary reviewer and physician reviewer on presence of triggers ranged from 0.646 to 0.658, indicating moderate agreement. Agreement on the presence of any AE/MHPSI between primary reviewers had a κ value of 0.355 (fair agreement) and that between each primary reviewer and physician reviewer ranged from 0.537 to 0.573 (moderate agreement).

Additional Information on AEs and MHPSIs

Adverse Event Data

A total of 98 patient records (19%) reviewed using the MHTT methodology had 110 AEs altogether because some records had more than one AE. Two additional patient records with AE (one AE each) were identified using the alternative method of review. The number of AEs per 1000 patient-days was 13.9. Most of the AEs were of mild severity; severity of harm related to AE as classified using NCC-MERP was as follows: category E, 88% (n = 99); category F, 11% (n = 12); category G, 0%, category H, 0%; and category I, 1% (n = 1).

There were 67 cases of ADEs and most of them (38 of 67) were extra pyramidal adverse effects related to antipsychotic medications. Only one AE related to electroconvulsive therapy (ECT) was identified. There were 25 cases of hospital-acquired infections, of which seven were skin infections. Of the 19 incidents of “patient injury,” seven were related to falls, three were related to physical restraints, and one was reported as an accidental injury. Two incidences of injury were classified as unexplained. The rest of cases of “patient injury” (n = 6) were related to MHPSIs.

The MHPSI Data

Using the MHTT method, a total of 58 patient records had MHPSIs. Overall, 67 MHPSIs were detected because some patient records had more than one MHPSI. One additional patient record with MHPSI was identified using the alternative review method. Overall, 11% of patients reviewed experienced at least one MHPSI. The number of MHPSI per 1000 patient-days was 8.4. Aggressive behaviors toward property or other patients or staff constituted majority (56 of 67 incidents) of MHPSIs. There were three incidences of attempted self-harm or attempted suicide and six cases were victims of assaults by other patients. There were three incidences of absconding. Of the six incidents involving physical injury to the patients, three were incidents where the patients were victims of assaults by other patients; one was related to an act of self-harm, whereas the rest two were injuries to the perpetrators of aggressive behaviors. In all cases, physical injuries sustained were considered to be of mild severity (category E as per NCC-MERP).

DISCUSSION

The challenges in defining, categorizing, and identifying patient safety incidents in mental health settings have been recognized as a major obstacle to conduct systematic research in this field.^5,11,12,15 In the process of developing MHTT, we have attempted to address some of these challenges by first defining and separating the patient safety incidents into (a) those that conformed to the traditionally defined AEs and (b) MHPSIs. We then developed an inclusive trigger tool, based on the trigger tool methodology that is able to identify and measure both AEs and MHPSIs through time-limited retrospective review of patient records, comparable with the GTT in general healthcare settings. The overall performance of the MHTT, with high sensitivity and specificity, is consistent with a valid screening tool in detecting AEs and MHPSIs in mental health settings. With MHTT, the proportion of “trigger-positive” patient records with AE/MHPSI was 33.8%, which indicate that there is one-in-three chance of finding an AE/MHPSI in a patient record if it is positive for at least one trigger listed in MHTT. Our results on interrater agreement between reviewers are comparable with other studies on trigger tools with better agreement between reviewers for triggers compared with that for AEs.²⁸

By separating safety incidents identified by MHTT into traditionally defined AEs and those primarily related to mental health (MHPSIs), as we have done in our study, hospitals and health authorities may be able to achieve a clearer perspective on the range of patient safety incidents relevant to mental health settings. This information can then be helpful in planning and implementation of appropriate interventions. It would also allow meaningful comparisons between AEs that occur in mental health settings and general healthcare settings as well as comparisons between MHPSIs occurring in exclusive mental healthcare settings and those occurring in psychiatric units in general hospitals.

In comparing our study findings on AEs with general healthcare settings, we found that our AE rate of just more than 19% was lower than that found in studies in general hospitals. For example, Good et al²⁹ in their study of AEs using GTT in acute care hospitals found that 25% of the 2369 admissions reviewed had at least one AE that was acquired during admission period and the AE rate rose to 39.8% of admissions when all AEs identified during admission were included. The number of AEs/1000 patient-days in our study was 13.9 compared with 33.2 AEs/1000 patient-days in a Swedish study and 60 AEs/1000 patient-days in a Danish hospital study.^8,30

Similarly, comparisons of MHPSIs could be made between our study and studies done in other mental health settings. In our study, of all patient records reviewed, 11% had at least one MHPSI and the majority were aggressive behaviors. Aggression directed toward psychiatric staff and other patients is a significant problem in psychiatric facilities throughout the world and a review of studies of incidence of violence in psychiatric inpatients across various countries reported that the mean violence rate ranged from 16.06% in Germany to 41.73% in the United Kingdom.^11,18,31

There are several limitations to our study. We used a modified Delphi method in the initial stages of selection of the triggers and this method has limitations in terms of reliability and validity. The alternative method we used in our study may have its own reliability issues, and in addition, we did not review all patient records using this method. Our interrater reliability findings, although comparable to similar trigger tools used in healthcare, probably indicate issues related to selection and training of reviewers. Although multiple training sessions were conducted before the study, we did not conduct a formal assessment of interrater reliability between the reviewers before the study. As with any retrospective studies on patient records, reliability of results of our study may have been affected by the quality and completeness of documentation. Finally, our study was limited to one institution and a multicenter study would have increased our confidence in the generalizability of the tool.

CONCLUSIONS

Despite the limitations, the MHTT can be a practical and easy-to-use tool in understanding and measuring a variety of patient safety incidents in mental health settings. The current version of MHTT is designed for use in inpatient mental health settings and may need further modification for use in outpatient settings. Further studies are recommended with MHTT in other mental health settings.