Adolescent suicide risk screening: A secondary analysis of the SHIELD randomized clinical trial

Abstract

Objective:

To evaluate the effectiveness of adolescent suicide risk screening to increase initiation of mental health services via a secondary analysis using data from the SHIELD randomized clinical trial, which evaluated school-based screening for major depressive disorder (MDD).

Study Design:

Students in 14 Pennsylvania high schools were randomized by grade to either: usual school practice of targeted referral for behavior raising concern for suicide risk, or universal screening using the Patient Health Questionnaire-9 (PHQ-9) with any response >0 to item #9 regarding suicide risk considered positive. Students identified in either arm were referred to the Student Assistance Program (SAP), which is mandated in all Pennsylvania schools. SAP determined follow-up. Study groups were compared using mixed effects logistic regression.

Results:

Participants included 12,909 students with 6,473 (50.1%) randomized to universal screening. Students were 46% female and 43% Hispanic or non-Hispanic Black. Adolescents in the universal screening arm had 7.1 times higher odds (95% CI 5.7–8.8) of being identified as at risk for suicide, 7.8 times higher odds (95% CI 4.6–13.1) of follow-up needs, and 4.0 times higher odds (95% CI 2.0–7.9) of initiating mental health treatment.

Conclusions:

Although the PHQ-9 is a MDD screening tool, its use in universal screening increased identification and treatment initiation for adolescents at risk for suicide. This confirms the value of universal screening, and suggests a suicide-specific risk assessment would have even greater impact on treatment initiation for identified youth.

Suicide has been identified as “an increasingly prominent public health issue.”^1–3 Youth suicide risk has been further exacerbated by the COVID-19 pandemic.⁴ Data suggests adolescents who screen at risk for suicide are more likely to engage with treatment.⁵ Yet, in 2014, when the US Preventive Services Task Force (USPSTF) reviewed available data, there was insufficient evidence to support adolescent suicide risk screening in primary care.^6–7 Among the research gaps, the USPSTF called for research to better understand the benefits and harms of targeted versus general screening and the utility of incorporating technology into large-scale screening studies.⁶

Increased risk of suicide is associated with greater severity of major depressive disorder (MDD), and the USPSTF endorses screening for adolescent MDD in primary care.^8,9 MDD screening tools such as the Patient Health Questionnaire-9 (PHQ-9) and PHQ adolescent version (PHQ-A) include a brief assessment of suicide risk with item #9, which asks, “…how often have you been bothered by…thoughts that you would be better off dead or of hurting yourself in some way?”^7,10–11 Studies of suicide risk assessment embedded within a MDD screen such as the PHQ-9 find positive responses are associated with increased suicide risk, though negative responses do not reliably correlate with low suicide risk.^12–15

In the context of the Screening in High Schools to Identify, Evaluate and Lower Depression (SHIELD) study, a randomized clinical trial (RCT) using the PHQ-9 to evaluate the effectiveness of school-based screening for adolescent MDD,^16–17 addressing responses to item #9 regarding suicide risk was the question that placed the greatest burden upon schools for urgent/emergent student follow-up. The PHQ-9 was selected over the PHQ-A for use in SHIELD, as the PHQ-A includes two additional unscored items regarding suicide risk that were anticipated to be complex to address in a large-scale screening.¹² As positive responses to item #9 on the PHQ-9 are associated with increased suicide risk,^12–15 we sought to use data from the SHIELD study to determine if it strengthens the case for universal suicide risk screening in schools. Specifically, we sought to evaluate whether the PHQ-9 increased identification and treatment initiation among those with increased suicide risk, and begin to address identified gaps in the USPSTF statement. If students with increased suicide risk identified via the PHQ-9 demonstrate greater treatment initiation, effectiveness would be anticipated to improve further with the use of a validated suicide risk screening tool.

Methods

The SHIELD study protocol and primary analysis have been previously published.^16–17 This study was approved by the Penn State College of Medicine Institutional Review Board. In brief, the 14 participating schools included a sample of both rural and urban schools with diverse student demographics.¹⁷

The two study arms in the RCT were targeted screening (control arm) and universal screening (intervention arm). Targeted screening is the current school process. Students at risk for suicide are identified based on active presentation with or disclosure of self-harm or suicidal ideation. For example, suicidal intent may be directly observed or overheard and shared with a school staff member, or a peer may be privy to information and in turn alert school staff. All schools have existing school protocols to manage these scenarios. Thus, a subset of these students may be “targeted” for referral and further screening or assessment prompted by their behavior, but students in this arm are not preemptively queried regarding suicide risk. This is compared with the universal screening intervention in which all students, regardless of behavior or symptoms, completed the PHQ-9 with the goal of identifying students at risk for suicide based on a positive response to question #9.

Half of the participating schools were randomized such that students in 9^th and 11^th grades received targeted screening (and 10^th and 12^th universal screening). The other half were randomized such that students in 10^th and 12^th grades received targeted screening (and 9^th and 11^th universal screening). Given that many school courses, programs and screenings are grade-specific (e.g. 9^th grade English or 11^th grade hearing screens), our program’s adherence to this screening approach was practical for participating schools.

In advance of screening, parents received a letter detailing the project and the screening instrument, the PHQ-9, and were offered the opportunity to opt their student out of the study.^16–17 Students in the universal screening arm could also decline the PHQ-9 at the time of screening. The PHQ-9 is a well-established, validated screening tool for the diagnosis of adolescent MDD.¹⁰ As noted above, PHQ item #9 asks about thoughts of self-harm, with a sensitivity of 87.6% and specificity of 66.1% compared with use of a specific suicide risk assessment tool among adults.¹⁸ Response options range from 0 points (“not at all”) to 3 points (“nearly every day”).¹² In this analysis, a response of 1 to 3 to item #9 was considered a positive screen.

Students in the universal screening arm (intervention) were administered the PHQ-9 electronically via ten study-purchased iPads during the school day by the study team. This was planned in the classroom or in a designated space, e.g., school library or auditorium, with direct entry into REDCap (Research Electronic Data Capture), a secure, web-based application designed to support research studies.^17,19 Students with a positive response to item #9 triggered an immediate electronic notification to research staff, who reported the information by student study identification number to predesignated school staff on the screening day for follow-up by the crisis team or the school Student Assistance Program (SAP) as detailed below.^16–17 As research staff only had access to the deidentified student study number, all subsequent follow-up was conducted by school staff who maintained the list tying student study number to identifiable information.^16–17

The targeted screening arm (control) followed the current school practice. If a student was determined at risk for suicide during the school year, a referral was either made to the school crisis team (for immediate threats to student safety) or to the SAP.^20–21 Specifically, SAP referral records had to include one of the following referral reasons: self-harm, suicidal ideation, past thoughts of hurting self or crisis. In Pennsylvania public schools, SAP is the primary process to address the needs of students who display any “barrier to learning”. Pennsylvania SAP is jointly overseen by the Pennsylvania Network for Student Assistance Services (PNSAS). PNSAS is a collaboration between Pennsylvania’s Department of Drug and Alcohol Programs, Department of Education (PDE) and Department of Human Services.²⁰ SAP is mandated in all Pennsylvania schools and endorsed by the Substance Abuse and Mental Health Services Administration.^20,22 The SAP team consists of a group of trained professionals that utilize a systematic four-step process to gather and review data on observable behaviors/symptoms prompting the referral.²⁰ SAP does not diagnose behavioral health disorders. Following review, students confirmed to require further intervention are referred for follow-up school and/or community-based services, which may be based on screening or assessment by a SAP liaison from a community behavioral health agency.²⁰ SAP liaisons comprise a varied group of professionals from county mental health and drug and alcohol agencies with expertise in mental health, substance use and general counseling services. SAP liaisons support school teams through technical assistance, consultation, and by conducting student behavioral health screenings and level of care assessments. SAP liaisons may also assist with school crisis response, prevention and support groups.²³

Students in the universal screening arm who exhibited behaviors raising concern for suicide risk separate from the PHQ-9 screening could be referred via the same process as for the targeted screening arm to SAP.

The main outcome was the percentage of students with identified suicide risk that warranted further evaluation by SAP and subsequently initiated treatment. Information on treatment initiation was based on the Pennsylvania Department of Education 4092 form, which is completed for each student referred to SAP.²⁴ To meet this outcome, students had to meet three criteria:

The first criterion was to be identified at risk for suicide: PHQ-9 response >0 on item #9 or behaviors prompting SAP referral for the following reasons: self-harm, suicidal ideation or free text “other” response of “past thoughts of hurting self” or “self-harm ideation/crisis”.

The second criterion was to be confirmed in need of services: A “yes” response to the question “was SAP referral warranted?” as part of SAP workflow.²⁴ The SAP team had to confirm the identified PHQ result or behavior/symptoms warranted further evaluation, which is typically done via further data collection (e.g., conversation with the student and/or parent, behavior checklists, etc.). For students with a positive PHQ-9, the SAP referral reason had to be consistent with the reasons listed above (for identification). Specifically, students could not meet criteria with a positive PHQ-9 followed by a SAP referral for reasons separate from suicide risk (e.g. academic concern). If this field in the workflow was incomplete, but services were recommended,¹⁷ students were considered to meet this criterion. Also, as noted above, students in immediate danger would be referred to crisis intervention versus SAP.

The final criterion was to have initiated at least one recommended SAP treatment/service: This includes primary or secondary school-based, community-based or SAP liaison services recommended during the SAP process. The list of services is the same as those used in analysis of the primary outcome.^{17, 24} In cases where services were recommended, but it was unclear based on the school records if treatment initiation occurred, students did not meet criterion 3. Students already in treatment did not meet criteria for initiation, though a sensitivity analysis was conducted to further evaluate this as detailed below.

Statistical analysis

Statistical analysis was conducted using the intent-to-treat principle. The primary outcome was an indicator of whether a student met all 3 criteria (initiated). The first and second criteria (identified and confirmed) were also analyzed. The statistical analysis compared universal to targeted screening and was conducted using mixed effects logistic regression.^17,25 Models contained a fixed effect for randomized group and a random effect for school. Odds ratios and 95% confidence intervals were reported.

We conducted two sensitivity analyses with respect to students who failed to meet the confirmed or initiated criteria because they were already in treatment. In the first sensitivity analysis, under the argument that these individuals were already known cases, we excluded these students entirely. In the second sensitivity analysis, with the argument that detection of suicide risk is essential even for students already in treatment,²⁶ we assumed these students met each criteria. We refit the statistical models under both scenarios.

Finally, we evaluated use of the first 8 questions of the PHQ-9 without inclusion of item #9 to simulate the Patient Health Questionnaire-8 (PHQ-8), and compared responses on item #9 of the PHQ-9 to positive scores on the PHQ-8.²⁷ For the simulated PHQ-8, a positive score was defined as ≥10 because it contained one fewer question. Only students who had assented and completed the PHQ-9 were included in this analysis.

Results

Participants

Participant and school demographics have been previously described.¹⁷ In brief, of 13,171 students from 14 schools eligible for study inclusion, 262 (2.0%) opted out. Thus, 12,909 students were included in the analysis, with 6,436 (49.9%) randomized to targeted screening and 6,473 (50.1%) randomized to universal screening. Students were 46.1% female, 45.2% non-Hispanic white, 22.4% non-Hispanic Black, 20.8% Hispanic, and 11.5% Multiracial/other. Median age was 16 (range 13–21 years). Demographics were comparable between study arms.

Of the 14 schools, seven were classified as urban, with a median size of 370 students. Eight of the 14 schools, primarily due to a larger student body size, completed universal screening over multiple days, ranging from two to ten days.¹⁷

There were 172 students (1.3%) who did not assent to the PHQ-9 at the time of screening. Among students in the universal arm, 1,941 did not receive the PHQ-9 screen primarily due to COVID-19 school closures and student absences/tardiness. In addition, 459 (7.1%) students in the targeted screening arm were erroneously offered universal PHQ-9 screening. Thus, a total of 4,819 students assented and completed the PHQ-9 (4,360 in the universal screening arm and 459 in the targeted screening arm).¹⁷

Main outcome

A total of 718 students (5.6%) met our criteria for being identified as at risk for suicide, including 622 in the universal screening arm and 96 in the targeted screening arm (Table I). Students confirmed to be in need of follow-up via SAP included 143 in the universal screening arm and 16 in the targeted screening arm, and students meeting the main outcome of treatment initiation included 47 in the universal screening arm and 10 in the targeted screening arm. Results of the fitted regression models indicated students in the universal screening arm had 7.1 (95% CI 5.7–8.8) times higher odds of being identified as at risk for suicide, 7.8 (95% CI 4.6–13.1) times higher odds of being confirmed in need follow-up, and 4.0 (95% CI 2.0–7.9) times higher odds of initiating recommended treatment/services compared with students in the targeted screening arm (Table II).

Table I.

Percentage of students with each outcome in the targeted and universal screening arms

	Targeted screening (N=6436)	Universal screening (N=6473)	Total (N=12909)
Identified*, N (%)	96 (1.5%)	622 (9.6%)	718 (5.6%)
Confirmed*, N (%)	16 (0.2%)	143 (2.2%)	159 (1.2%)
Initiated*, N (%)a	10 (0.2%)	47 (0.7%)	57 (0.4%)

^*identified as at risk for suicide: positive Patient Health Questionnaire-9 (response >0 on item #9) or behavior prompting referral to Student Assistance Program (SAP) for suicide risk; confirmed: SAP does not diagnose, but to meet criteria, SAP must confirm identified behavior/symptoms warrant further evaluation; initiated: participated in at least one SAP recommended treatment/service, e.g., follow-up mental health supports

Table II.

Mixed effects logistic regression model evaluating odds of identified, confirmed and initiated^* for students in the universal and targeted screening arms

Outcomes	Randomized group	OR (95% CI)	p-value	Variance of random school effect (SE)
Identified	Universal screening Targeted screening (ref)	7.1 (5.7–8.8) 1	<0.001	0.08 (0.05)
Confirmed	Universal screening Targeted screening (ref)	7.8 (4.6–13.1) 1	<0.001	2.4 (1.2)
Initiated	Universal screening Targeted screening (ref)	4.0 (2.0–7.9) 1	<0.001	1.6 (1.0)

^*identified as at risk for suicide: positive Patient Health Questionnaire-9 (response >0 on item 9) or behavior prompting referral to Student Assistance Program (SAP) for suicide risk; confirmed: SAP does not diagnose, but to meet criteria, SAP must confirm identified behavior/symptoms warrant further evaluation; initiated: participated in at least one SAP recommended treatment/service, e.g., follow-up mental health supports

OR: Odds ratio; CI: Confidence interval; SE: Standard error

There were 559 students who met criteria 1 (identified), who did not meet criteria 2 (confirmed). This was primarily due to referral to an outside agency with no further information available via SAP (35.4%), or that the student met with a counselor, social worker (suicide liaison) or other school staff with no further documentation by SAP (38.1%). There were 102 students who met criteria 2 (confirmed) who did not meet criteria 3 (initiated). The main reasons were parent refusal (43.1%), the student was already in treatment (40.2%), or student refusal (10.8%).

Sensitivity analysis

A sensitivity analysis focused on students who were already in treatment/services (Table III). First, these 63 students were excluded entirely, leaving a sample size of 12,846. The resulting odds of treatment initiation were similar to the initial analysis (OR 4.1, 95% CI: 2.0–8.1). Second, students who were already in treatment were assumed to meet the confirmed and initiated criteria. Under this assumption, 181 total students met the confirmed criteria (OR 9.2; 95% CI 5.5–15.4) and 85 students met the initiated criteria (OR 7.2; 95% CI 3.7–13.8).

Table III.

Sensitivity analysis considering both the exclusion and inclusion of students already in treatment

Categories	Excluded students in treatment (OR [95% CIs])	Assumed in treatment met criteria (OR [95% CIs])
Identified*	6.5 (5.2–8.1)	7.1 (5.7–8.8)
Confirmed*	5.8 (3.4–9.9)	9.2 (5.5–15.4)
Initiated*	4.1 (2.0–8.1)	7.2 (3.7–13.8)

^*identified as at risk for suicide: positive Patient Health Questionnaire-9 (response >0 on item 9) or behavior prompting referral to Student Assistance Program (SAP) for suicide risk; confirmed: SAP does not diagnose, but to meet criteria, SAP must confirm identified behavior/symptoms warrant further evaluation; initiated: participated in at least one SAP recommended treatment/service, e.g., follow-up mental health support.

OR: Odds ratio; CI: Confidence interval

Consideration of the PHQ-8

Finally, results of the PHQ-8 can be simulated among N=4,819 students (37.3%) who assented and completed the PHQ-9 (in either treatment group). Of students with a positive score on item #9, 205 students (30.2%) would have had a negative overall score on the PHQ-8. These students are potentially “at risk” of being missed had the PHQ-8 screen alone been used (Table IV).

Table IV.

Sensitivity analysis considering both the exclusion and inclusion of students already in treatment

	Item 9 Score of 0 (N=4,100)	Item 9 Score of >0 (N=688)	Total (N=4,819)
Overall PHQ-8 score*
Missing	70	10	111
Negative (<10)	3,498(86.8%)	205(30.2%)	3,703(78.7%)
Positive (≥10)	532(13.2%)	473(69.8%)	1,005(21.3%)

^*As the PHQ-8 includes one less question, the overall score distribution changes, and scoring was adjusted so a positive result was ≥10

Discussion

This secondary analysis of data from the SHIELD RCT demonstrates that universal screening in the school setting with the PHQ-9 helps to identify students at risk for suicide, and increases treatment initiation. Analyses considering removal of the suicide risk question and use of the PHQ-8 demonstrated a substantial number of at-risk students would be missed.

We recognize that the use of a suicide-specific assessment tool (e.g., Ask Suicide Screening Questions [ASQ], Columbia Suicide Screen) would likely further increase the number of students at-risk for suicide who are accurately identified.^28,29 ASQ had a sensitivity of 100%, specificity of 88% and negative predictive value of 100% for identifying elevated suicide risk among adolescents in primary care.²⁶ About one-third of adolescents who screen negative on item #9 of the PHQ were deemed at risk based on the ASQ.¹³ However, the PHQ-9 was used to minimize burden to the schools, and our analysis demonstrated the use of the PHQ-8 versus the PHQ-9 would have missed another 30% of at risk students.

Gould et al. conducted a longitudinal study of 317 students identified at risk for suicide via school-based screening. At the 2-year follow-up, among 78 students who were newly referred to services at the time of screening, 69% had initiated treatment.⁵ Our findings similarly demonstrated the increased effectiveness of universal school-based suicide risk screening on treatment initiation with the benefit of a large and diverse sample. There may have been additional cases that initiated treatment, but our criteria required verification of treatment initiation based on SAP records, and SAP is not the sole referral pathway for suicidal students.²¹ All students who screened positive for suicide risk with the PHQ-9 were immediately reported to schools on the same day,^16–17 but students could be referred to crisis intervention or to outside services, which were not subsequently tracked by school SAP teams.

Balancing screening benefits with the burden of implementation on school staff is an ongoing challenge, as multiple steps are necessary to further evaluate students who screen positive.³⁰ The overall identification rate through universal screening was 9.6%. Prior work considering varying thresholds on the Columbia Suicide Screen found a high threshold algorithm identified students at elevated risk for suicide. However, the lower threshold algorithm had the added benefit of identifying students with other mental health concerns that present barriers to academic success (e.g. anxiety, depression) who would benefit from treatment.³⁰ As discussed in the context of the original RCT, schools that partnered with the research team were confident in their ability to manage an increased number of student referrals, and are the appropriate target for this type of intervention.¹⁷ Even among these schools, several set a threshold number of students identified at risk for suicide that could be managed in a given day by available staff. Once that threshold was reached, the research team returned for screening on a subsequent day. Thus, generalizability of these results is limited to schools with adequate capacity to appropriately manage identified at-risk students, in this case approximately 10% of those screened. Of note, some schools worked with SAP liaisons to ensure extra staff was available on screening days, suggesting the added value of this practice to enhance school-community partnerships. This may be an effective approach to ensure timely screening follow-up and compensate for increased workload to school staff.

The study results are especially timely considering the 2014 USPSTF statement on suicide risk screening in primary care is currently under review.³¹ Although done in a school versus a primary care setting, the study results support the effectiveness of suicide risk screening to increase treatment initiation for identified adolescents. The findings directly respond to the USPSTF call for studies to address research gaps by examining targeted versus general screening.⁶ In addition, in the same section on research needs and gaps, the USPSTF highlighted incorporating technology into large scale screening studies.⁶ As detailed in the methods, technology was used to alert research staff to students at risk for suicide in real time, a process that could be replicated in a primary care setting.

The primary limitation of the study was that as a pragmatic clinical effectiveness trial, it could not impose the constraints of an RCT run at an academic institution to optimize fidelity (e.g. study staff could not directly access students or parents, and assessment measures subsequent to PHQ-9 screening in the universal arm were solely selected by the schools). As mentioned above, the PHQ-9 is not a suicide assessment tool, and SAP is not the sole management pathway for students identified at risk for suicide. The data were also limited by what schools could provide, and for students referred to crisis intervention or outside services, this often meant SAP had no further record of their disposition or treatment, potentially leading to missing instances where students had initiated treatment. Even when measured accurately, treatment initiation does not guarantee sustained engagement or response. Therefore, we cannot comment on the clinical impact of screening beyond initiation of services.

Despite these data limitations, which bias findings towards the null, screening increased identification of students at risk for suicide and treatment initiation. Future work should aim to measure the functional improvements in students identified by school-based screenings, as well as the benefits of interventions designed to enhance treatment utilization as well as disease detection. School staff and students were not blinded to study arm, which may have influenced management of positive screens. Finally, this secondary analysis was also limited in that subgroups became too small to allow accurate analysis by race/ethnicity, sex and rural/urban location.

Abbreviations:

USPSTF

US Preventive Services Task Force

MDD

major depressive disorder

PHQ-9

Patient Health Questionnaire-9

PHQ-A

Patient Health Questionnaire-Adolescent Version

SHIELD

Screening in High Schools to Identify, Evaluate and Lower Depression

RCT

randomized clinical trial

SAP

Student Assistance Program

PHQ-8

Patient Health Questionnaire-8