Using Structured Telephone Follow-up Assessments to Improve Suicide-related Adverse Event Detection

Abstract

Adverse event (AE) detection and reporting practices were compared during the first phase of the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE), a suicide intervention study. Data were collected using a combination of chart reviews and structured telephone follow-up assessments post-enrollment. Beyond chart reviews, structured telephone follow-up assessments identified 45% of the total AEs in our study. Notably, detection of suicide attempts significantly varied by approach with 53 (18%) detected by chart review, 173 (59%) by structured telephone follow-up assessments, and 69 (23%) marked as duplicates. Findings provide support for utilizing multiple methods for more robust AE detection in suicide research.

Accurate adverse event (AE) detection is an issue affecting all areas of clinical research, yet most studies addressing AE reporting methods target drug-related or non-mental health-related research (e.g., Ehrenpreis, Sifuentes, Ehrenpreis, Smith, & Marshall, 2012; Gupta, 2008). One clinical area with particular concern for accurate AE detection and reporting practices is suicide research. With more than one million suicides per year worldwide (WHO, 2011) and over 30,000 per year in the US alone (CDC, 2009), facilitating suicide research is essential. However, there are only about 30 active studies assessing psychosocial and pharmacological interventions that may benefit individuals at risk for suicidal behavior (Oquendo et al., 2011). One reason for the lack of empirical research on suicidal behaviors is the perceived liability, risks, and inadequate training in monitoring and treating suicidal crises (Pearson, Stanley, King, & Fisher, 2001). Currently, clinical researchers primarily rely on chart reviews for AE detection (Gilbert, Lowenstein, Koziol-McLain, Barta, & Steiner, 1996). However, researchers suggest that using a variety of methods to detect AEs could produce more accurate AE detection (Murff, Patel, Hripcsak, & Bates, 2003). The current study aims to test whether combining chart reviews with structured telephone follow-up assessments could improve AE detection in clinical research, specifically suicide research.

Two common measures of AEs, the Agency for Healthcare Research and Quality’s Patient Safety Indicators (PSIs; Agency for Healthcare Research and Quality (AHRQ), 2003) and the Utah/Missouri Adverse Event Classification (a superset of Patient Safety Indicators; Masheter & Houghland, 2005) involve an automated review of discharge codes to detect AEs. However, automated measures like these have been critiqued as neither sensitive nor specific enough to correctly identify adverse events (e.g., West, Weeks, & Bagian). This concern prompted the search for more direct approaches to measuring AEs, which resulted in the use of “triggers” to identify AEs, the Institute for Healthcare Improvement’s (IHI) Global Trigger Tool for Measuring AEs (IHI, 2003). The Global Trigger Tool involves review of patient charts by two or three employees (e.g., nurses, pharmacists) who are trained to systematically review charts by looking at discharge codes, discharge summaries, medications, lab results, operation records, nursing notes, physician progress notes, and other notes or comments to determine whether there is a “trigger” in the chart. An example of a trigger would be a notation for a medication stop order or an abnormal lab result. Any notation of a trigger leads to further investigation into whether an AE occurred and how severe the event was (Classen et al., 2011; IHI, 2003).

The Global Trigger Tool has many strengths including use of multiple triggers and standardized methodology that allows for reliable detection of AEs in hospital settings (Resar, 2008), yet even these improved methods have limitations associated with relying solely on chart reviews for AE detection, namely underreporting of AEs. A recent study evaluating a software tool designed to assess laboratory results for identification of AEs found improvements in accuracy of AE detection beyond chart reviews. An additional 17% of serious adverse events (SAEs) were detected that were missed in chart review (Niland et al., 2012). Findings like these suggest that chart review alone may not be an adequate method of AE detection. For that reason, empirical data are needed to determine whether implementing multiple AE data collection methods may improve AE detection in suicide research.

Method

Study Participants

The following sections summarize the AE detection and reporting methods used during the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) study (U01 MH088278; Boudreaux, Camargo, Miller). ED-SAFE is a quasi-experimental clinical trial that includes participation from eight general medical emergency departments (EDs) across the United States (see Boudreaux et al., 2013 for full description of the study). Eligible participants are ED patients aged 18 years or older with thoughts of killing themselves in the past week or an actual, aborted, or interrupted attempt to kill oneself in the past week. ED-SAFE consists of three phases of data collection: (1) Treatment as Usual, (2) Universal Screening, and (3) Intervention. The current study focuses on the first phase, Treatment as Usual where data were collected on usual practices in the ED.

Overall, there were 497 subjects enrolled in the first phase of the ED-SAFE study. Most were White (77%), female (56%), and never married (52%). The median age was 38 years with most enrolled subjects reporting active suicidal ideation at baseline (63%). Of the total 497 enrolled subjects, 493 (99%) had a completed 6-month chart review and 492 (99%) had a completed 12-month review. Of the 497 subjects who reached the due date for all five of their structured telephone follow-up assessments, 367 (74%) missed fewer than 2 assessments.

AE terminology

For our study, an AE is defined as an unfavorable and unintended symptom, disease, or outcome temporally associated with the use of a medical or behavioral treatment or intervention regardless of whether it is considered related to the treatment or intervention. SAEs are any untoward medical occurrence that results in death or the immediate risk of death, hospitalization or prolonging of an existing ED visit or hospitalization, or persistent or significant disability/incapacity (Food and Drug Administration, 2009). See Table 1 for events reported as AEs and SAEs during our suicide intervention study.

Table 1.

Listing of Adverse Events and Serious Adverse Events documented during the current study

Adverse Event	Non-suicidal self injury (NSSI) Evidence of coercion to participate Distress during assessment Access of confidential information by a non-authorized person
Serious Adverse Event	Death, for any reason Suicide attempt, including actual, aborted, or interrupted attempts Emergency department (ED) visit (potentially suicide related) Inpatient hospitalization (potentially suicide related) Police/EMS called by Boys Town crisis counselor

We consider suicide attempts (actual, aborted, or interrupted) and suicide completion as expected SAEs, events that may be reasonably anticipated to occur as a result of the study procedure and are described in the consent form. Each AE is graded in terms of severity (Non-serious, Severe, Life-threatening or disabling, or Fatal) and relation to study intervention (Definite, Probable, Possible, Unlikely, or Unrelated). In general, our definitions of suicidal ideation and behavior conform to the Columbia Classification Algorithm for Suicide Assessment (C-CASA) that was developed to systematize definitions of suicidal behavior (Posner, Oquendo, Gould, Stanley, & Davies, 2007).

AE detection and reporting

We are using a multi-method approach for collecting AE data during all three phases. Our AE reporting methods were informed by the National Institutes of Health (NIH) guideline (1999) addressing reporting AEs to institutional review boards for NIH supported multicenter clinical trials. Multiple data sources help minimize limitations associated with not having access to electronic databases from other healthcare systems in the region of the site. In phase 1, AE data were collected from two primary ED-SAFE data sources: (1) chart reviews completed by site research staff, and (2) structured telephone follow-up assessments completed by a call center.

For the chart review component, all prospectively enrolled subjects are followed for 12 months using patient medical records from the healthcare system with which the enrolling site is affiliated. Structured chart reviews take place 6 months and 12 months after subject enrollment. Subjects who are reported to have died (for any reason), or who are not reached at the 12-month follow-up call trigger an official death registry review (e.g., state/county vital statistics registries). This enables us to confirm reported deaths and identify deaths among those who would otherwise be lost to follow-up. A death can be detected during telephone follow-up assessments (e.g., emergency contact notifies call center during call) or chart review (e.g., documented in patient’s medical record). Although an AE report is submitted as soon as a death is discovered, a death certificate review provides additional details needed to finalize the AE report. Official death certificate reviews are documented in the chart review database.

Phone interviewers from the University of Massachusetts Medical School (UMMS) Office of Survey Research call center conduct structured telephone follow-up assessments for this study. During assessments, phone interviewers make sure that they have the following information readily available: 1) Contact information for on-call crisis counselor; 2) Contact information for the subject; and 3) Emergency contact numbers for the subject. During the initial part of the assessment, the interviewer confirms the subject’s telephone number and current location, so that this information will be readily available to provide to the crisis counselor if needed.

ED-SAFE contracted with Boys Town National Hotline (2013) to ensure that there would always be a mental health counselor on call during the follow-up assessments. To ensure subject safety through the course of the study, individuals who reported active suicidal ideation or an attempt during a call are transferred to a crisis counselor. An AE report triggered by an on-call crisis counselor would be due to a study subject reporting active suicidal ideation or behavior that requires the counselor to call the police or EMS.

Subjects are transferred to a mental health counselor at Boys Town in the following four types of situations: (1) Subject is currently suicidal; (2) Subject made a recent suicide attempt without seeking health care; (3) Interviewer encounters any other situation where he/she believes that the subject is at imminent risk of hurting him/herself or others, such as the subject being extremely distressed during the call or the subject makes an innuendo suggesting he/she is going to kill him/herself; and (4) Interviewer encounters any other situation where the subject appears to need additional resources, but the subject does not appear to be suicidal.

The thresholds for calling Boys Town are incorporated into the programming of the Computer Assisted Telephone Interview (CATI) protocol. This standardized the decision regarding mental health counseling and reduced complications introduced by relying on human judgment. Embedded in the follow-up survey are items from the Columbia Suicide Severity Rating Scale (CSSRS, Posner et al., 2011) intended to identify suicidal ideation severity and intensity, as well as attempt lethality. If a subject responds in a specific way to a certain question, for example, indicating active suicidal ideation with “yes,” the CATI programming will prompt the interviewer to call the on-call crisis counselor at the end of the follow-up assessment. The interviewer also could transfer the subject to a crisis clinician during the call if the interviewer felt the subject was in imminent danger.

AEs are triggered by the Boys Town crisis counselor according to the following protocol and were considered reportable SAEs if an actively suicidal subject hung up or was disconnected prior to the transfer and the subject did not answer the crisis counselor’s call. In such cases, the crisis counselor will continue to try to reach the subject for up to 3 additional attempts over 60 minutes. If the crisis counselor was unable to reach the subject directly or through emergency contact provided, the crisis counselor would contact the local police/emergency medical services (EMS). Standardized reporting forms for AEs triggered by Boys Town are transferred from Boys Town to the ED-SAFE data coordinating center on the business day following the AE.

Recording and managing the data

Chart review and Boys Town data are entered into a web-based data management system, Research Electronic Data Capture (REDCap; Vanderbilt, Nashville, TN). REDCap is a secure, web-based application for building and managing online surveys and databases. REDCap provides automated export procedures for seamless data downloads to Excel and common statistical packages (SPSS, SAS, Stata, R), as well as a built-in project calendar, a scheduling module, ad hoc reporting tools, and advanced features, such as branching logic, file uploading, and calculated fields (https://arcsapps.umassmed.edu/redcap/).

Telephone follow-up interviewers entered data into Confirmit (Oslo, Norway, 1996), a survey software system, to administer the telephone follow-up interviews. The Confirmit system includes the following security measures to ensure the highest levels of data security and compliance with regulatory standards: SSL encryption, data at rest encryption, and high security module for the sFTP server (FIPS Level 2 certified).

We designed an automated program to compile data from all of the ED-SAFE data sources (including REDCap and Confirmit) into one composite database, which we call ED-CORE. We use the data in ED-CORE to filter out a pre-determined set of variables related to AEs in the database that trigger the creation of a separate AE data file. This file is then imported into a graphical user interface (GUI) designed to allow for review by the ED-SAFE Safety Officer, who works at the Emergency Medicine Network (EMNet) Coordinating Center based at Massachusetts General Hospital. At this stage, the ED-SAFE Safety Officer manually reviews the data provided for incomplete records or duplicates. Any duplicates found are marked as duplicates, so they are not reported or counted toward the overall AE total. The remaining active cases then are reviewed to determine the AE type (e.g., non-suicidal self-injury (NSSI), hospitalization, suicide attempt), outcome, suicide status change (e.g., going from suicidal ideation at baseline to suicide attempt at the 6-week follow-up assessment; applicable for suicide attempts reported during telephone follow-up assessments only), and the AE narrative, which incorporates qualitative data provided by the site (chart reviews) or the phone interviewer (structured telephone follow-up assessment). Any mention of intentional self-harm ideation or behavior was considered an adverse event for our study purposes. Completed AE reports for all SAEs are then distributed within 72 hours to the participating sites, the ED-SAFE Steering Committee, and NIMH/DSMB representatives. In cases of SAEs reported on Fridays, submission of reports within 96 hours is permissible. All non-serious AE reports are reported on a monthly basis.

Results

To test whether inclusion of a second data source for AE detection was beneficial, we examined AE data collected during the first phase of ED-SAFE (TAU) using chart reviews and structured telephone follow-up assessments. Although there were only two chart review timeframes (6- and 12-month) versus five for the telephone follow-up assessment (6-, 12-, 24-, 36-, and 52-weeks), comparisons of the two methods involved comparing events detected during comparable timeframes (see Figure 1).

Figure 1.

Comparison of review timeframes for structured chart reviews and structured telephone follow-up assessments

Among the AEs detected during the TAU Phase, there was some overlap due to the different AE detection methods. Originally 1,871 AEs were detected by both chart reviews and structured telephone follow-up assessments with 200 cases (11%) marked as duplicates (AEs detected by both chart review and telephone follow-up assessment or detected multiple times by the same source (e.g., subject mistakenly reported event at both 6- and 12-week assessment)). Including data from both chart reviews and telephone follow-up assessments, we were able to detect a total of 1,671 independent AEs during the TAU phase (956 SAEs).

When we considered data from the 6- and 12- month chart reviews alone, we found 836 AEs (496 SAEs) independent of those detected during the structured telephone follow-up assessments. The majority of these SAEs were ED visits (n=249; 74%) and hospitalizations (n=194; 42%) related or potentially related to suicide. Only 18% (n=53) of the total 226 suicide attempts were detected during 6- and 12- month chart reviews (see Table 2).

Table 2.

Comparison of adverse event data collected during structured chart reviews and structured telephone follow-up assessments.

	Combined chart review and telephone follow-up	Duplicate cases*		6- and 12-month chart review data only		Telephone follow-up assessment data only
		n	%	n	%	n	%
Total AEs	1871	200	11%	836	45%	835	45%
Total SAEs	1148	192	17%	496	43%	460	40%
Total Suicide attempts	295	69	23%	53	18%	173	59%
Total ED visits (suicide or potentially suicide related)	338	23	7%	249	74%	66	20%
Total hospitalizations (suicide or potentially suicide related)	457	56	12%	194	42%	207	45%
Total deaths	24	15	63%	0	0%	9	38%
Total suicidal crises (Police/EMS called)	6	1	17%	1	17%	4	67%

Abbreviations: AE, adverse event; SAE, serious adverse event, ED, emergency department; EMS, emergency medical services.

^*Duplicate cases were events detected by both chart review and telephone follow-up assessment or repeated events (e.g., subject incorrectly reported event at 6- and 12-week follow-up assessment)

Evaluation of the AE data from the structured telephone follow-up assessments revealed that 835 AEs (460 SAEs) were detected that did not overlap with AE data from the chart reviews (see Table 1). Telephone follow-up assessments identified almost half of the total AEs (835/1,671 = 45%). While chart reviews detected a large proportion of suicide or potentially suicide related ED visits (74% vs. 20%; difference = 54%; 95%CI 43–65, p<0.001), telephone follow-up assessments detected a greater proportion of suicide attempts (59% vs. 18%; difference = 41; 95%CI 28–54, p<0.001). Even though 43% of all suicide or potentially suicide related events could be detected by chart review alone, including a second detection method, such as telephone follow-up assessments, accounted for an additional 40% of all suicide or potentially suicide related events detected during the treatment as usual phase of our suicide intervention trial.

Discussion

The current study presents an integrated approach to AE detection that improves upon current detection and reporting practices. Past literature reviews demonstrate that clinical research has relied heavily on chart reviews for AE detection (e.g., Murff et al., 2003). Our findings demonstrate that additional methods, such as structured telephone follow-up assessments may provide a more robust picture of AEs in suicide research. Our study found that 43% of all suicide or potentially suicide related events could be detected by chart review alone; however, including a second detection method, such as telephone follow-up assessments, accounted for another 40% of all suicide or potentially suicide related events.

We anticipated that structured telephone follow-up assessments would improve the AE detection rate because many of the survey questions were directly related to AEs, but we did not anticipate that more than half of the suicide attempts reported would be detected during these telephone follow-up assessments. We also did not anticipate that telephone follow-up assessments would almost double the unique AEs detected during our suicide intervention study. This could be attributed to chart reviews being limited to events occurring at the enrolling site, while telephone follow-up assessments gathered data on events occurring at all locations (e.g., home, hospital, social events) during the specified study timeframe. Regardless, these findings provide additional support for prior claims that multi-methods are an important consideration for improving current AE detection practices in suicide research (Naessens et al., 2009; Olsen et al., 2007).

To our knowledge, this is the first multi-method study of AE detection in clinical suicide research. Combining chart reviews and structured telephone follow-up assessments is advantageous because it allowed for detection of suicide-related events occurring both at and outside of the enrolling hospital. Although telephone follow-up assessments independently detected several AEs beyond chart reviews in the current study, there are certain limitations associated with introducing additional methods. The most important being increased demand for resources, specifically time and cost. However, standardization of the chart review and telephone follow-up assessment materials as well as training and a detailed manual, helped reduce completion time for these tasks. Single method approaches may lead to underreporting of AEs, which could jeopardize patient safety and the clinical research (e.g., Classen et al., 2011). Telephone follow-up assessments allowed us to detect additional suicide-related AEs when many of these events would have gone undetected because they did not receive treatment for the event (e.g., suicide attempts with no treatment received, non-suicidal self injury (NSSI) reports). All methods have limitations, but the over-reliance on any one method can limit the study outcomes, as well as methods directed at patient safety, such as AE detection (Naessens et al., 2009; Pope & Mays, 1993). Empirical evidence, like that from the current study, is critical for developing a systematic method for detecting, documenting, and reporting AEs and SAEs to protect both patient and researcher.

We recognize that a major concern in devising a solution for reporting SAEs is imprecise taxonomy, since the choice of terms has implications for how the SAE is handled both in reporting requirements and decisions to modify or suspend the intervention for patient protection. Suicide researchers are often left with the question of how suicide-related SAEs should be classified when suicide can be both a SAE and a study outcome. Pearson et al. (2001) note that even in studies that compare TAU to a given intervention, methodological complications are introduced. Increased monitoring of suicidal subjects adds standardization to behavior that is not standardized. Our study implemented a structured protocol for AE detection and reporting that targeted at-risk suicide behaviors aimed at generating minimal interference with “real world” behavior. We were able to balance ethical concerns by providing live referrals to a crisis hotline for actively suicidal patients, but maintained more natural conditions by solely documenting and reporting historical AEs and SAEs to the relevant study committees and IRBs. Using a structured set of guidelines for AE detection and reporting both improved the rate of detection and decreased the overall reporting burden by removing non-study relevant AEs.

Although these findings provide empirical data on AE detection and reporting for suicide research, our findings are preliminary and additional investigation is needed. For example, it would be beneficial to examine whether the current study methods are applicable to other suicide-related studies. In addition, future research should investigate which methods detect the most relevant AEs in other settings (e.g., school-based research) or for other populations (e.g., general public versus ED). To assuage both the concern for subject safety and AE reporting burden, generating empirically based methodology is critical.

Limitations

Use of chart reviews may be a limitation of this study, specifically in documenting and interpreting the data. We tried to mitigate these limitations by designing a detailed protocol for data collection and analysis, implementing standardized abstractor training, and using REDCap for data capture. REDCap improves data quality by using required fields, branching logic, and validation loops.

An additional concern for the chart review component of our study was that we limited chart reviews to events occurring within the enrolling hospital. If subjects presented to other EDs or hospitals, we were unable to account for these events which may underrepresent the total number of AEs for our study population. Nevertheless, we were able to address our study aim of determining the added benefit of structured telephone follow-up assessments in detecting AEs during a suicide intervention study.

Another limitation is the use of self-reporting to collect data during the phone interviews. However, the literature suggests that client self-reports are often found to be accurate in suicide research (Yigletu et al., 2004). In addition, our follow-up telephone interviews were standardized, which have been shown to elicit a more complete and objective assessment of suicidal behaviors (Asnis et al., 1994).

Conclusions

Although rising suicide rates (AFSP, 2012) and the Joint Commission’s safety goals (Joint Commission, 2008) emphasize the urgent need for suicide research, anecdotal evidence suggests that fear of inadequate or overwhelming AE reporting requirements may be stopping researchers from pursuing this topic. Prior work supports the use of multiple AE data collection methods in patient safety settings (Naessens et al., 2009), yet there is still a lack of comprehensive scientific guidelines for detecting AEs, reporting SAEs, and analyzing the impact of these data for human subjects protection in suicide research (Oquendo et al., 2011).

To date there are no existing data comparing AE detection or analysis methods in suicide research. The current paper demonstrates that a combination of chart reviews and structured telephone assessments provided a more robust approach to AE detection of suicide-related events. Although chart reviews certainly contributed to the overall AE detection rate, telephone follow-up assessments identified approximately half of the AEs detected during our suicide intervention trial. Most notably, telephone follow-up assessments detected three times the number of suicide attempts when compared to chart reviews. Our findings provide a better understanding of how implementation of multi-methods may improve AE detection in the context of suicide research. Using these findings as guidelines will increase our knowledge about AE detection, reporting, and analysis and facilitate the development and improvement of research on suicidal populations.