Implementation Barriers Encountered During a Universal Suicide Screening Program in Pediatric Emergency Departments

Abstract

Objective:

Because understanding barriers to universal suicide risk screening in pediatric emergency departments (PEDs) may both improve identification and management of suicidal behaviors and ideation, this study assessed barriers to a quality improvement initiative examining the use of a novel computerized adaptive test (CAT), the Kiddie-CAT, in two PEDs.

Methods:

Research assistants (RAs) trained in Rapid Assessment Procedures-Informed Clinical Ethnography methods documented barriers related to the environment, individuals, and workflow as encountered during screening shifts, categorizing the barriers’ impacts as either general to a screening shift or related to screening an individual youth/caregiver dyad. Using thematic content analysis, investigators further categorized barriers based on type (e.g. workflow, language/comprehension, clinician attitudes/behaviors) and relationship to the limited integration of this initiative into clinical protocols. Reasons for refusal and descriptive data on barriers are also reported.

Results:

Individual screen barriers were most often related to workflow (22.9%) and youth/caregiver language/comprehension challenges (28%). Similarly, workflow issues accounted for 48.2% of all general shift barriers. However, many of these barriers were related to the limited integration of the initiative, as RAs rather than clinical staff conducted the screening.

Conclusions:

Although this study was limited by a lack of complete integration into clinical protocols and was complicated by the COVID-19 pandemic impacts on PEDs, the findings suggest that considerable attention needs to be directed both to physician education and to workflow issues that could impede universal screening efforts.

Introduction:

Youth suicide is the second and third leading cause of death for youth in the US ages 10–14 and 15–24. In 2020, suicide accounted for a total of 6,643 deaths across these age ranges.¹ In 2021, the American Academy of Pediatrics, American Academy of Child and Adolescent Psychiatry and Children’s Hospital Association released a joint statement declaring a national emergency regarding child and adolescent mental health (MH), making specific note of the troubling rates of youth suicide.²

Emergency departments (EDs) serve as the primary contact with the health care system for many youth³ and recently, have seen dramatic increases in visits for MH concerns, including those related to suicidality.^4–6 Almost 50% of youth who die by suicide visit an ED in the year prior to death and 15% have an ED encounter in the four weeks prior.⁷ Youth with public or no insurance and those from low-income families are particularly likely to rely on the ED for their health care,⁸ and youth with public insurance who die by suicide are significantly more likely to have visited an ED in the previous 6 months compared to their non-suicidal peers.⁹

Consequently, EDs have been identified as a key setting for the identification and management of youth MH challenges and suicide in particular.¹⁰ However, despite mounting evidence suggesting that universal suicide screening in pediatric EDs (PEDs) is beneficial and feasible, guidelines and recommendations for screening vary.^11–15 Further, although US-based studies of universal suicide screening initiatives in PEDs mention challenges such as screening tool limitations, workflow disruptions, and inadequate workforce training and report highly variable participation and positive screen rates, just three of eleven studies in a recent systematic review included prospective, systematic data collection on barriers related to implementation.¹⁶ Two of the three focused on patient perceptions exclusively, while the other focused on provider perspectives. A subsequent literature review specifically examining barriers to screening identified just nine studies. These studies focused heavily on youth and caregiver attitudes, which were found to be largely supportive of screening. Conversely, three studies examining provider attitudes identified more ambivalence towards screening, with providers often citing a lack of confidence in their training, ability to screen and intervene.¹⁷ However, without systematic evaluations examining the implementation of screening protocols, it remains unclear if these are real or perceived barriers.

To document barriers to universal youth suicide screening in PEDs, data on barriers were collected as part of a quality improvement (QI) initiative designed to assess challenges associated with the implementation of a novel computerized adaptive test (CAT), the Kiddie-CAT (K-CAT). To guide the comprehensive documentation of possible barriers, data were collected on the elements highlighted in the Implementation Phase of the Exploration, Preparation, Implementation, Sustainment (EPIS) model.¹⁸

Method:

Study Design and Setting

This QI initiative took place in one public and one private hospital PED. Both serve as training sites for an academic medical center located in a major metropolitan area of the Northeastern United States. Neither hospital has MH specialists directly employed in the PED. However, both have 24/7 access to MH clinicians through a consultation-liaison (CL) team in the private hospital and a specialized psychiatric PED in the public hospital.

This initiative took place over two phases during which research associates (RAs) conducted screenings with youth and caregiver dyads. RAs staffed an average of four 6-hour shifts per week across settings. During the baseline phase (August 2021-March 2022), physicians only received K-CAT screening results for patients who scored severe on one of the three diagnostic categories assessed (anxiety, depression, and suicidality) and/or who received suicide warnings (SW). Physicians were instructed by PED leadership to follow existing protocol for psychiatric concerns during this phase.

The second phase followed a one-hour physician training covering information about youth suicide, the K-CAT, and a study-generated disposition protocol. For each dyad screened during this phase (April 2022-June 2022), physicians received a K-CAT score report form, a one-page outline of a disposition protocol linked to K-CAT results, and if indicated, a community referral list to be provided to the youth and caregiver.

Study Population

Eligible participants included all patients 7–17 years old with the following exclusions: a medically severe concern (i.e., an emergency severity index score of 1), behavior incompatible with screening (e.g. severe agitation, psychosis, intoxication), limited-to-no verbal language, and/or not fluent in either English or Spanish. Those who arrived without a caregiver and/or previously participated in this project were also excluded. The screening segment of the study was classified as a clinical QI effort by the NYU Grossman School of Medicine IRB. Although formal consent was not required, all youth/caregivers received an information sheet summarizing the project, risks, and benefits. Caregivers then approved participation for themselves and/or their youth by checking a box following the information sheet. Participating youth checked a box of their own to document their understanding and agreement to participate.

Screening Measures

The K-CAT is an electronically administered multidimensional item response theory-based tool validated for youth ages 7–17, offered in English and Spanish, with strong psychometrics.¹⁹ Results indicating a categorical severity level (normal, mild, moderate, or severe) for anxiety and depression and the presence/absence of a SW were used in this study.

The following measures were also included in the screening battery: a sociodemographic questionnaire, a COVID questionnaire, the Autism Spectrum Quotient-10 (caregiver-report only), and the NIH Toolbox Picture Vocabulary Test (PVT; youth only). All measures were administered in REDCap, except for the PVT, which is collected and scored in the NIH Toolbox app.

Data Collection

Screeners were trained in Rapid Assessment Procedures-Informed Clinical Ethnography (RAPICE) methods prior to the start of this project. As participant observers, screeners kept a log of barriers throughout each shift, which included factors related to the environment, individuals, and workflow that may have impacted the overall course of the shift (“general shift barriers”) or an individual screen (“individual screen barriers”). Information collected covered multiple domains suggested by the inner context of the Implementation Phase of the EPIS model, including organizational characteristics, individual characteristics, and innovation fit for the service (PEDs), the providers, and the patients. Additionally, screeners described reasons for ineligibility and/or refusal for all dyads presenting to the PEDs who were not screened. The reasons for refusal were determined a priori and coded at the time of collection. Youth sex at birth and age data were retrieved from the PED board by RAs at the time of refusal for refusals only. Barriers data were logged in REDCap at the end of each shift.

Data Analysis

Log entries were exported to Excel and analyzed using thematic content analysis.²⁰ First, the entries were reviewed by the coding team, which included one research coordinator, one clinical psychologist/researcher, and one senior MH services researcher, to familiarize themselves with the data and identify preliminary topics and themes. One author divided the entries into distinct meaning units, segments of text ranging from a few words to multiple sentences. The coding team then used the immersion-crystallization approach to develop categories and finalized themes through an iterative process of constant comparison.²¹ The first author coded all meaning units and the senior and second author separately coded 10% each. Agreement was 98%. All disagreements were reviewed and discussed until coders reached consensus.

General shift barriers were broken into five categories: workflow (e.g. issues locating or approaching dyads), clinical team attitudes or behaviors (e.g. clinician confusion regarding protocol), environment (e.g. noise causing distraction/distress), technology (e.g. WiFi issues), and other. Individual screen barriers included eight categories: technology, workflow, youth and caregiver disposition (e.g. attitudes, behaviors, and competing physical health concerns), youth and caregiver language/comprehension (e.g. RAs asked to define a word), clinical team attitudes or behaviors, and other.

All barriers were further characterized as either generalizable challenges, issues related to the limited integration of the initiative(e.g. administration of additional research measures, use of RAs to conduct screening), or undetermined. For example, language and comprehension issues on the KCAT were considered generalizable while language and comprehension issues encountered during completion of research measures were categorized as related to limited integration. Barriers were classified as undetermined if it was not clear if it would have occurred had the screening process been better integrated (e.g. interruptions to screening by clinical staff, challenges feeding back results to physicians). Research leadership also documented barriers encountered throughout implementation; these observations are descriptively reported.

Chi-squared tests were used to compare participation rate and patient demographics.

Results:

Participants

1777 youth and caregiver dyads presented to the PEDs during shifts covered by RAs throughout this initiative. Of these, 1438 dyads were ineligible. Reasons for ineligibility are shown in Figure 1. Youth and/or caregivers from 241 eligible dyads agreed to screening while 100 eligible dyads refused screening (overall participation rate 70.7%). Reasons for refusals are shown in Table 1.

Figure 1:

Flow diagram of youth and caregiver dyads presenting to the PEDs

Table 1:

Youth and caregiver reasons for refusal to complete suicide risk, depression, and anxiety screening

Refusal Reason	Caregivers (N=100)	Youth (N=100)
Too focused on presenting concern or just not interested	32	27
Has existing mental health care	7	8
Concern regarding appropriateness of screening	4	2
Declining with youth	29	NA
Declining with caregiver	NA	45
Unknown/Other	28	18

Dyads with youth designated female at birth were more likely to participate (Χ²[1]=5.55, p=0.018) and 47.3% of participants were female. There were no statistically significant differences in participation by age (M participant 12.6). Participants were most often White (22.0%) and self-identified as Latino or Hispanic (53.9%).

Individual Screen and General Shift Barriers

Individual screen barriers were reported for 100 (41.5%) dyads screened; of these, 63% encountered just one barrier, 25% had two, and 12% had two or more. Fifty-six general shift barriers were recorded. Barriers are reported in Table 2.

Table 2:

Individual screen barriers and general shift barriers to the implementation of universal suicide risk screening in PEDs

	Generalizable N(%)	Related to Limited Integration N(%)	Undetermined if Generalizable N(%)	Total N(%)
Individual Screen Barrier Category
Technology	5 (3.2)	7 (4.5)	9 (5.7)	21 (13.4)
Workflow	4 (2.5)	3 (1.9)	29 (18.5)	36 (22.9)
Youth Disposition	12 (7.6)	3 (1.9)	0	15 (9.6)
Caregiver Disposition	11 (7)	1 (0.6)	1 (0.6)	13 (8.3)
Youth Language and Comprehension	16 (10.2)	9 (5.7)	0	25 (15.9)
Caregiver Language and Comprehension	10 (6.4)	9 (5.7)	0	19 (12.1)
Clinical Team Attitudes or Behavior	2 (1.3)	0	11 (7.0)	13 (8.3)
Other	12 (7.6)	3 (1.9)	0	15 (9.6)
Total	72 (45.9)	35 (22.3)	50 (31.8)	157 (100.0)
General Shift Barrier Category
Workflow	2 (3.6)	0	25 (44.6)	27 (48.2)
Clinical Team Attitudes or Behaviors	0	0	9 (16.1)	9 (16.1)
Environment	15 (26.8)	0	0	15 (26.8)
Technology	1 (1.8)	1 (1.8)	2 (3.6)	4 (7.1)
Other	0	0	1 (1.8)	1 (1.8)
Total	18 (32.1)	1 (1.8)	37 (66.1)	56 (100.0)

Organizational Characteristics:

Under existing PED protocols, nursing staff screen youth starting at 10 years old using the Columbia Suicide Severity Rating Scale (C-SSRS). Given this and the fact that the primary goal of this initiative was to evaluate the use of a novel instrument (the K-CAT) and an enhanced disposition protocol in the PEDs, RAs from the research team conducted screening. While using RAs avoided placing additional burden on clinical staff, it introduced organizational barriers. New to the setting and not considered part of the clinical team, RAs encountered challenges identifying and locating patients, communicating results to physicians in a timely manner, and conducting recruitment and screening between other clinical activities (e.g. nursing checks, specialty consultations). Additionally, although these hospitals have available MH specialists, these clinicians are not collocated in the PED. Physicians expressed that consultation wait times and potentially lengthy evaluations were sources of concern as they could disrupt patient flow.

PED Leadership:

The research team included three PED physicians: the director of the division of pediatric emergency services (who oversees attendings and trainees in both PEDs), the clinical site chief of the public hospital’s PED, and one attending physician. As champions and PED leadership, these physicians identified and addressed multiple barriers throughout the screening initiative. For example, prior to the start of Phase 2, they highlighted physicians’ lack of existing materials to provide to youth scoring in the moderate range on anxiety and depression, for whom the disposition protocol did not suggest consultation. This prompted the development of a referral list, which was incorporated into the disposition protocol as a packet to be sent home with dyads. Leadership also worked to develop scripts in the electronic medical record system to facilitate the process of documenting K-CAT results in patient medical records.

Service Environment:

The COVID-19 pandemic had a significant impact on PED patient characteristics (e.g. higher acuity and lower patient flow were experienced during waves) and workflows (e.g. changing isolation protocols).

Interdepartmental Networks:

Clinical leadership from the child and adolescent psychiatry teams at both hospitals participated in the iterative process of protocol development. Due to limited CL team availability during nights, screening shifts were limited to daytime hours in the private hospital. Though concerns arose early on regarding an increase in consultations called by the PEDs, these calls remained manageable. In barriers data collection, RAs noted that psychiatry teams in both hospitals expressed that PED physicians provided inadequate context for calls in multiple instances when consultations were requested as the result of a K-CAT screen.

Innovation Fit (Development and Adaptation):

28% of all individual screen barriers were related to youth or caregiver language/comprehension issues. Though some of these (N=18, 40.9%) were attributable to the research battery, RAs noted several words or phrases that were frequently misunderstood by dyads completing the K-CAT. Notably, some younger participants (typically 7–8-year-old patients) struggled to understand the word “suicide.” The K-CAT developers collected feedback from the research team and made modifications to both the English and Spanish versions of the K-CAT.

The K-CAT-linked disposition protocol was developed with input from a team of child psychologists, psychiatrists, and PED physicians using recommendations from the literature and building on existing protocols in both hospitals. However, physicians raised concerns regarding burden, confidence, and existing training in the Phase 2 training as well as through surveys.²² Modifications were therefore made to the enhanced disposition protocol to simplify interpretation and follow-up. While the original disposition protocol included recommendations based on both the suicide severity category and the presence/absence of a SW, the final version used only the SW result.

Discussion:

Given the importance of universal screening for youth MH concerns in PEDs, understanding the barriers to screening initiatives is essential. Previous studies have pointed to a lack of budgetary and staffing support as well as provider training and confidence as major challenges to implementing universal MH screening initiatives in PEDs.²³ Results from this initiative on clinician training and attitudes were reported in a separate publication but point to similar concerns.²² Ultimately, both staffing and budgetary concerns were addressed by external funding from a NIH research grant that supported screening equipment, access to the K-CAT and RAs to staff the screening. However, external funding for screening efforts is not usually available.

The lack of full integration of this screening initiative into the PEDs’ workflows was a significant limitation to and source of barriers in this study. Research team members are far less familiar with the setting and less integrated in workflow than PED staff. Importantly, using RAs introduced additional barriers which may not be applicable to usual PED screening efforts. For example, although the majority of barriers were related to workflow and language/comprehension challenges, many examples of these, such as an RA having trouble locating a patient (considered an integration-related workflow barrier) or confusion with language on a research measure (an integration-related language/comprehension barrier), may not be applicable to other screening initiatives.

Further, this study took place during the COVID-19 pandemic in an area heavily affected by the virus, which resulted in significant burden on PEDs and may have impacted barriers identified. And, as reported in Blinded et al, physicians involved in this initiative reported particular weaknesses in managing suicide risk.²² As a result, adjustments to the disposition protocol were made to reduce the responsibility of PED physicians in managing screening results. This was facilitated in large part by the availability of consultation psychiatry services in both hospitals. While the involvement of MH specialists may be ideal, a recent survey of PED leaders found that only 66% of PEDs have 24/7 access to a MH specialist.²⁴ Although the need for improving physician training and confidence in managing suicide risk remains clear, additional data on barriers as perceived by the PED clinicians were limited.

Data on reasons for refusal were also limited and often vague. Caregivers (32%) and youth (27%) often expressed general disinterest in screening or prioritization of their presenting concern. However, it is worth noting that nurses typically administered the C-SSRS during triage. Thus, dyads may have felt comfortable declining screening with the K-CAT knowing clinicians had another source of information regarding suicide risk. Further, few youth (2%) and caregivers (4%) expressed stigma or fear around discussing MH and suicide. These findings are promising and confirm previous findings on youth and caregiver attitudes towards screening.^10,25,26

The K-CAT’s major benefits include being fast and easy to administer, the ability to assess several diagnostic domains, and providing immediate, electronic scoring. Partnership with the tool developers allowed RAs to provide feedback on the K-CAT based on generalizable language/comprehension issues, which accounted for 16.6% of all individual screen barriers. Multiple changes were made to K-CAT language during this time, including removing double negatives, clarifying the time frame being asked about, and defining “suicide.” Finally, although not implemented in this screening effort, the K-CAT can be integrated into electronic medical records such that results are directly fed to physicians and reported in patient charts. This feature could likely circumvent some of the workflow challenges.

Barriers encountered during this universal suicide screening initiative suggest that considerable attention needs to be directed both to physician education and to workflow issues that could impede universal screening efforts. Given that suicide is a major cause of death for youth, understanding the barriers to implementing routine universal suicide screening in EDs should be a priority.