Suicidality Screening: Ensuring Providers Do Not Miss Adolescents at High Risk through Clinical Decision Support

Abstract

Background

Major depression in adolescents is recognized as a serious psychiatric illness and a risk factor for suicide. It is important for providers to screen for depression and suicidality and have clinical decision support (CDS) for evaluation, risk assessment, and initial management. Our efforts focused on leveraging the electronic health record (EHR) to ensure that high-risk adolescents received appropriate interventions before leaving the office.

Objectives

This study aims to improve notification of positive adolescent depression and suicide risk screening results. Subaims include (1) designing and optimizing CDS alerts for severe depression and/or acute lethality; and (2) increasing visibility for different EHR users to avoid the near misses.

Methods

We implemented several iterations of an interruptive alert after a positive mental health screening for patients aged 12 to 20 years seen in primary care and subspecialty rheumatology clinics between 2016 and 2023. Screening started with the Patient Health Questionnaire (PHQ) and Ask Suicide-Screening Questions (ASQ) on paper, and later via tablets.

Results

There were 76,919 visits during the study period, of which 50,502 (66%) completed PHQ screening. Rates of PHQ screening increased throughout the study period. Of the ASQ screenings completed, alerts were fired for 2,355 nonacute positive and 411 acute positive. In most cases, all alerts fired were more than once. Providers and clinicians receiving alerts (physicians, nurse practitioners, social workers, and psychologists) acknowledged 63 to 96% of alerts and cancelled 6 to 70% of alerts. We revised the initial alert identifying lethality to include separate alerts for major depressive symptoms and suicide risk. Iterations also included an icon on the clinic schedule and incorporated guidance on appropriate actions.

Conclusion

By creating multiple types of alerting within different workflows, we improved notification and increased next-step visibility, leveraging decision support for patients screening positive for depression and suicidality.

Background and Significance

Major depression in adolescents is recognized as a serious psychiatric illness. It is a significant risk factor not only for suicide but also for academic failure, substance abuse, and early pregnancy. 1 2 Multiple organizations recommend routine depression screening in children starting at 12 years of age. 1 3 In 2021, the American Academy of Pediatrics, American Academy of Child and Adolescent Psychiatry, and the Children's Hospital Association declared a National State of Emergency in child and adolescent mental health. 4 With the high prevalence of depression—20% by age 20, 5 it is not only important for clinics to screen for depression and suicidality, but screening efforts must include plans for evaluation, risk assessment, and initial management after a positive screen and before the patient leaves the office. 6 7 Despite the known association of preceding health care utilization and suicide attempts, screening adolescents for depression and suicide risk has always been a challenge. 8 Time constraints, lack of resources, and lack of adequate training in medical management of depression and suicide risk are barriers to adequately addressing mental health screening concerns for providers. 1 Although prior algorithms have attempted to identify those at high risk of suicidality, prediction models have been suboptimal and rely on both structured and unstructured data. 9 10

While embedded clinical decision support (CDS) alerts within the electronic health record (EHR) are an ideal way to assist with screening, interruptive alerts can lead to increased workflow interruptions, added distractions, and low acceptance. 11 12 Therefore, it is necessary to leverage the EHR to provide alerts that are succinct and contain CDS for maximum effectiveness.

Objectives

In this paper, we describe improving provider notification after positive adolescent depression and suicide risk screening results. Subaims include (1) designing and optimizing CDS alerts and clinician workflows for severe depression and/or acute lethality; and (2) increasing visibility for different EHR users to avoid near misses.

Methods

Setting

This study included patients at a large, pediatric quaternary care academic medical center with both primary care and pediatric specialties at Nationwide Children's Hospital in central Ohio from 2016 to 2023. During the study period, the primary care network expanded from 11 to 13 clinics with approximately 100 providers, 20 social workers, and an increase from 4 to 8 psychologists embedded within the clinics. The multidisciplinary rheumatology lupus team works out of a single clinic and includes pediatric rheumatologists and fellows, a clinical psychologist, and a social worker.

Participants

Patients between the ages of 12 and 21 years of age at the time of their visit, seen in a primary care clinic with a visit type of “New Patient” or “Well Child,” or in a rheumatology clinic with a diagnosis of systemic lupus erythematosus, were included. Patients in primary care visits were considered part of the “primary care cohort,” and those seen in rheumatology were considered part of the “subspecialty cohort.” We included the Rheumatology Lupus Clinic because this multidisciplinary clinic has social workers and psychologists along with a team focused on improving mental health outcomes. See Supplementary Appendix—Diagnosis Identification (available in the online version only) for additional details.

Visit Characteristics

We extracted the following data from the EHR through structured reports: date of visit, age, race, and ethnicity. Problem list and visit diagnosis were identified. We defined the coronavirus disease 2019 (COVID-19) pandemic period as visits between March 11, 2020, and May 11, 2023. 13

Mental Health Screening

The initial Patient Health Questionnaire (PHQ) depression screen combined screening for depression with suicide risk and self-harm risk ( Supplementary Appendix—Mental Health [available in the online version only]). We refined the screening process to use a modified version of the PHQ to assess for depressive symptoms only and added the Ask Suicide-Screening Questions (ASQ) to screen separately for suicide risk. A PHQ total score of 10 or above has a sensitivity of 88% and a specificity of 88% for major depression. 14 The ASQ is a series of five questions about suicide risk. A “yes” answer to any of the first four questions is considered a non-acute positive screen. A “yes” answer to question 5 is considered an acute positive screen and indicates a patient at immediate and imminent risk for suicidal behavior. 15

Provider Alerts

We implemented the first iteration of our interruptive alert within primary care in September 2020, targeting providers who opened the chart of a patient who endorsed suicidality on the PHQ and/or ASQ ( Table 1 ). In January 2022, the second iteration of the alert provided clearer guidance on next steps for positive PHQ screen and positive ASQ screen in two separate alerts. A separate alert was created for nursing staff to be aware of patients at high risk for acute suicidality. The third iteration of the alert was implemented in August 2023 and triggered not just for suicidality but for PHQ scores ≥10. 14 16 17

Table 1. Alert iterations.

Iteration	Alerts	Gaps/issues	How issues were addressed
Baseline	Answers to the questions and score for PHQ are populated in the EHR into discrete fields	Scores and answers appeared outside the provider workflow There was no alert for high-risk scores or concerns for suicidality, which would require immediate action	Scores and answers appeared within the provider note and workflow Pop-up alert for high-risk scores triggered on opening the patient chart Implementation of ASQ to assess acute versus non-acute suicidality risk
1 (September 2020–December 2022)	Single alert for ASQ and PHQ Only triggers for suicidality risk	Unclear on which screening was positive and next appropriate action Did not clearly identify children with moderate depression Nursing staff was not notified of suicidality risk If chart was not opened during encounter or was opened prior to questionnaire being filed, there was no alert for clinic teams	Two separate alerts were created for ASQ and PHQ Clear guidance provided on next steps for different risk levels Separate nursing alert was created
2 (January 2022–July 2023)	Separate alerts for PHQ and ASQ ASQ had two alerts to indicate non-acute or acute suicidality. Only one ASQ alert is displayed to the user Nursing alert created for acute suicidality on ASQ Clinic note guides providers on appropriate action	PHQ alerts can be silenced by any user by acknowledging next steps, which would make it not trigger for anyone else	Modified to alert each unique user for abnormal PHQ
3 (August 2023–Present)	Alert for PHQ trigger by scores of ≥10 for moderate depression for each unique user ASQ alerts remain separate Schedule column is more alert and visible for all clinic staff Silent alert to rheumatology clinic social worker (rheumatology only)	–	–

Abbreviations: ASQ, Ask Suicide-Screening Questions; EHR, electronic health record; PHQ, Patient Health Questionnaire.

Rheumatology alerts for positive PHQ screens began in February 2022, with a positive alert defined as a PHQ score of ≥10 and/or positive screening for lethality. ASQ alerts began in 2023 and aligned with the existing workflow within primary care.

Alert Measurement

For every encounter where an alert is fired, the unique alert record is defined at the encounter level. After the initial iteration, unique alert records exist for both the PHQ and the ASQ. We also evaluated the alert firings, which are versions of the alert record that could display multiple times per encounter. Providers could offer alert feedback in real time by selecting thumbs up/down icons and entering comments.

Statistical Analysis

Descriptive statistics are reported. Univariate logistic regression clustered by encounter evaluated the association between alert cancellation and clinical characteristics. Physician is the comparator for clinical roles; suicide screening is the comparator for depression screening; and rheumatology clinic is the comparator for the primary care clinic. Variables with p  < 0.2 in univariate analysis were included in multivariate analysis. Analysis was completed using Stata 16.0 (College Station, Texas, United States).

Human Subject Research Approval

This study was approved by the Nationwide Children's Hospital Institutional Review Board, STUDY00003743.

Results

Alert Design and Iteration

At baseline, the scores from the mental health screenings populated into the EHR, but this was outside of the provider workflow with no alerts for high-risk scores or guidance on next steps. Through several revisions, we refined alerts to include adding screening scores within the provider workflow, flagging high-risk scores within provider notes, and pop-up alerts including CDS guiding next steps depending on risk level. Initial iterations limited the alerts to certain providers, but gradually this was expanded to include other members of the care team to ensure safety precautions could be immediately implemented for acutely suicidal patients ( Table 1 ). Interruptive alerts for providers began in 2020. Initial alerts, termed Iteration 1, only fired once per provider after opening the chart of patients screening positive for self-harm or suicide risk from either the PHQ or ASQ without consideration of the total PHQ score. These alerts were displayed to providers, trainees, psychologists, and social workers. Pitfalls from this approach included the exclusion of nursing staff in the alert system, and it was unclear from the alert which screening test was positive.

By January 2022, the team modified the alert to provide greater clarification by separating the alert for PHQ from the ASQ (Iteration 2) for users when they opened the patient's chart. This change was quickly communicated to providers, which patients were screened positive for severe depression versus increased suicide risk or both. In addition, the team added CDS to help providers handle each scenario, including a link to review responses as needed and to initiate a risk assessment. The team created two separate alerts for a positive ASQ: one for a non-acute positive screen and another for an acute positive screen. Both alerts triggered upon opening the chart. The alert for a non-acute positive screen recommended a risk assessment and creation, revision, or review of a safety plan. The alert for an acute positive response recommended 1:1 supervision until completion of a risk assessment ( Fig. 1A ). However, ASQ alerts would fire for any “yes” responses or if a patient did not answer one of the questions. Additionally, if a patient ever endorsed a previous suicide attempt on the ASQ, this question remained positive on future screenings, and the alert would fire. The team also created a separate nursing alert for an acute positive screen to allow faster initiation of 1:1 supervision to enhance patient safety. All alerts had to be acknowledged before additional actions in the chart could proceed, and providers were required to discretely document their interpretation of the ASQ screening, such as negative, non-acute positive, and acute positive.

Fig. 1.

( A ) Acute positive lethality screening for Ask Suicide-Screening Questions (ASQ). ©2024 Epic Systems Corporation. ( B ) Alert on provider schedule for high-risk screenings. ©2024 Epic Systems Corporation. ( C ) Primary Care provider notes with highlighting in high-risk patients. ©2024 Epic Systems Corporation. PHQ, Patient Health Questionnaire. (data presented in the figure are imaginary)

Further modification in August 2023 included alerting for PHQ scores of ≥10 to identify patients at risk for moderate to severe depression, regardless of suicide risk screening results. The team also added an icon to the schedule column ( Fig. 1B ), which was visible on clinic computers and clinic bed boards. We enhanced the primary care provider note to reinforce the identification of patients with a positive screen ( Fig. 1C ). If the PHQ score was abnormal, a highlighted banner appeared in the note writing section of the EHR. An abnormal score also triggered the highlighting of several headers of the note writing section to prompt the provider that action was recommended in these areas. These areas included options for referrals, counseling provided in the clinic, recommendations for medical treatment, and follow-up. Primary care and subspecialists also added automatically generated text based on PHQ scores within provider notes, including additional guidance to alert providers of abnormal screening requiring intervention and appropriate actions.

Depression and Suicidality Alert Analysis

There were 76,919 visits during the study period, of which 50,502 (66%) completed PHQ screening ( Supplementary Table S1 [available in the online version only]). Rates of PHQ screening increased throughout the study period. More than half of the questionnaires were completed during the COVID-19 pandemic; visit diagnoses of anxiety and depression during the pandemic were 3.9% and 5.0%, respectively, and outside of the pandemic were 2.4% and 3.4%, respectively, both p  < 0.01. Prior diagnosis of anxiety (19% vs. 6%) and depression (17% vs. 5%) were more common in the subspecialty clinic than the primary care clinic, both p  < 0.01. Alerts within the EHR successfully identified adolescents screening positive for moderate to severe depressive symptoms and/or increased risk of suicide after a series of revisions and expansions.

Depression and suicidality alerts were deployed throughout all iterations for all categories of depression ( Supplementary Table S2 [available in the online version only]), including when there was no PHQ score recorded. There were 18 non-acute and 10 acute positive ASQ alerts with no PHQ documentation. One encounter had both a non-acute and acute positive alert fire.

Of the ASQ screenings completed, alerts were fired for 2,355 non-acute positive and 411 acute positive. Of the 2,355 non-acute positive alerts, provider evaluation of the patient classified responses as 1,174 (50%) non-acute positive, including reclassifying 902 (38%) and 74 (3%) alerts as negative or acute positive screens, respectively. There were 205 (9%) non-acute positive screens where the provider did not document the severity.

Of the 411 acute positive alerts, provider evaluation of the patient classified responses as 65 (15%) acute positive, including reclassifying 148 (36%) and 165 (40%) of alerts as negative or non-acute positive, respectively. There were 33 (8%) acute positive screens where the provider did not document the severity.

All alerts fired more than once in most cases ( Table 2 ), and 5,986 (7.8%) of visits had one or more alerts. Sixty-three percent to 96% of alerts were acknowledged, and 6% to 70% of alerts were cancelled. The alerts were most frequently displayed to providers, social workers, and psychologists. Despite having interruptive alerts, we had some near misses in our center due to providers opening charts before the patient completed the screening or after the visit was complete; however, we were unable to easily or accurately quantify these occurrences. To our knowledge, there were at least three instances where a patient had a positive screen and left the clinic without a safety plan, including one patient who left prior to being seen by the provider, and two whose screening was not recognized in real time. In-line alert feedback was provided with 14 “thumbs up,” 8 “thumbs down,” and 5 comments, including “Next steps are in place.” When the link to view responses is clicked, the entire alert goes away and cannot be referenced after reviewing responses. The alert looks extremely severe, eye-catching, then goes away unexpectedly “when the link is selected, then doesn't return,” “can't be removed,” “alert,” and “helpful.”

Table 2. Alert characteristics.

	Iteration 1: PHQ + ASQ Combined Alert ( N  = 1,560)	Iteration 2: PHQ only ( N  = 2,924)	Iteration 2: ASQ alerts only ( N  = 2,647)	Iteration 3: PHQ only + score ≥10 ( N  = 840)	Iteration 3: ASQ alerts only ( N  = 118)
Total unique alerts fired per visit
1	477 (30%)	733 (25%)	581 (22%)	197 (23%)	8 (7%)
2–5	788 (51%)	1,445 (49%)	1,418 (54%)	475 (57%)	109 (92%)
6–10	266 (17%)	614 (21%)	533 (20%)	148 (18%)	1 (1%)
>10	29 (2%)	132 (5%)	115 (4%)	20 (2%)	0 (0%)
Outcome per alert firing a
Accept	2 (0.1%)	457 (16%)	502 (19%)	38 (5%)	2 (2%)
Acknowledge	1,266 (81%)	2,064 (71%)	1,812 (69%)	528 (63%)	113 (96%)
Link to results	333 (18%)	717 (25%)	416 (16%)	146 (17%)	21 (18%)
Cancel	1,010 (65%)	1,973 (68%)	1,684 (63%)	584 (70%)	7 (6%)
Alert displayed by provider type a
Fellow	13 (1%)	97 (3%)	81 (2%)	19 (2%)	4 (3%)
Nurse Practitioner	384 (25%)	528 (18%)	480 (18%)	168 (20%)	20 (17%)
Physician	1,037 (66%)	2,011 (69%)	1,895 (72%)	638 (76%)	94 (80%)
Psychologist	299 (19%)	808 (28%)	762 (29%)	76 (9%)	32 (27%)
Resident	45 (3%)	304 (10%)	283 (11%)	135 (16%)	22 (19%)
Social Work	660 (42%)	1,296 (44%)	1,211 (46%)	99 (12%)	93 (79%)

Abbreviations: ASQ, Ask Suicide-Screening Questions; PHQ, Patient Health Questionnaire.

^aNot mutually exclusive; multiple provider types could see the alert for each encounter, and alert could be displayed to the same provider multiple times.

In univariate analysis, there was a statistically significant association with alert cancellations among social workers, depression screening alerts, and encounters within primary care, whereas psychologists, trainees, nurse practitioners, and the second and third alert iterations were negatively associated with alert cancellations ( Table 3 ). In multivariate analysis, social workers, depression screening alerts, and encounters within primary care continued to be statistically significant with alert cancellations, and the third alert iteration was negatively associated with alert cancellation.

Table 3. Univariate and multivariate association of alert cancellation with clinical characteristics.

	Univariate		Multivariate
Covariate	Odds ratio (95% CI)	p -value	Odds ratio (95% CI)	p -value
Clinical role
Physician	Reference	–	Reference	–
Psychologist	0.49 (0.45, 0.55)	<0.01	0.50 (0.46, 0.55)	<0.01
Social worker	1.15 (1.07, 1.24)	<0.01	1.14 (1.06, 1.23)	<0.01
Resident	0.70 (0.62, 0.79)	<0.01	0.71 (0.63, 0.80)	<0.01
Fellow	0.37 (0.28, 0.51)	<0.01	0.40 (0.29, 0.54)	<0.01
Nurse practitioner	0.85 (0.77, 0.94)	<0.01	0.85 (0.77, 0.94)	<0.01
Depression screening alert	1.20 (1.15, 1.25)	<0.01	1.21 (1.17, 1.26)	<0.01
Primary care clinic	4.21 (2.85, 6.23)	<0.01	3.41 (2.29, 5.09)	<0.01
Alert iteration
1	Reference	–	Reference	–
2	0.87 (0.81, 0.94)	<0.01	1.01 (0.95, 1.10)	0.65
3	0.83 (0.75, 0.93)	<0.01	0.87 (0.77, 0.97)	0.01

Discussion

This study describes how we improved notification of positive adolescent depression and suicide risk screening results, including separating depression and suicidality alerts, providing clear guidance for next steps, and providing different alerts in multiple formats in pediatric primary care and subspecialty clinics. Our study demonstrates the need for refining alert mechanisms over multiple iterations to accurately identify adolescents at risk for depression and suicidality and provide timely interventions. In the first iteration, we had one single interruptive alert which fired once for both PHQ and ASQ, only when patient endorsed suicide risk or self-harm, and was triggered by opening the patient chart. This alert failed for several reasons captured by the Five Rights of Clinical Decision Support, 18 which are a common set of guiding principles when designing support: (1) The chart was opened before the questionnaires were completed and the alert never fired (wrong time); (2) the chart was opened after patient left the clinic and was not evaluated in real time (wrong time); and (3) if the chart was opened by any provider/clinician type and alert was cancelled before action was taken, it did not fire again during the encounter (potentially wrong person or wrong time). Additionally, we used a depression screening tool to evaluate suicidality (wrong information). To mitigate the above failures, in our second iteration, with feedback from our providers, nurses, psychologists, and social workers, we mapped out various workflows and scenarios that would prevent any future near misses. We separated the alerts for PHQ and ASQ positive screens. The alert for a non-acute positive ASQ needed to be acknowledged and required initiation of a risk assessment and safety plan. Patients with an acute positive ASQ also required initiation of 1:1 supervision. We expanded these alerts to fire for each provider type until acknowledged. Our third iteration incorporated the PHQ score to identify those patients at risk for moderate to severe depression and further identify and differentiate depression and suicide risks. With this iteration, we added three additional layers of alerting to the workflow: (1) schedule column icon which showed for all care team members outside of the standard interruptive alert; (2) highlighted text within the primary care provider notes appeared after a positive PHQ screen; (3) silent alert to the clinic social worker through in-basket messaging (rheumatology only).

This study also demonstrates that CDS must often be multifaceted and redundant to include the variety of workflows occurring in a clinical setting, particularly when addressing a potentially life-threatening condition such as suicide. Although this redundancy raises the concern for alert fatigue, our alert governance group has striven to limit interruptive alerts to those that significantly affect patient health and safety, which this alert clearly does. Interruptive Our Practice Advisories of any kind were only generated in 7% of visits to these departments. There is a need for multiple layers of alerting at various points in the workflow involving the entire care team to ensure that no patient who has a critically positive behavioral health screen leaves the clinic without an appropriate intervention. Prior studies have neither identified the optimal number of interventions to display nor which providers should be notified of the screening outcomes. Our workflow alerts multiple providers and care team members with a varying amount of information to keep the patient safe. The visibility of positive screening for all team members became a highly effective way to identify patients endorsing moderate to severe depression and/or acute suicidality, regardless of whether they accessed the patient's chart directly. This cross-disciplinary awareness outside of entering the patient chart helped to prevent near misses.

Our study should be interpreted with several limitations and challenges. Some of the challenges in this series of interventions included receiving timely feedback from the care team, changing workflows, securing buy-in from all stakeholders, multiple new builds, and providing updated education after each new change. It took 1 to 2 years between alerts for iterations to occur. Although creating multiple alerts involved more members of the care team providing greater visibility after a positive screen, one potential downside was that the alert kept firing even after one provider acknowledged it, or if the provider was doing further assessment and did not specify their next steps on the alert.

Limitations in data collection include the inability to track any depression screenings or scores that were entered via free text or other personalized documentation tools in the provider notes; however, this occurred early in the study period before all workflows were standardized across all clinics. We also deployed screenings in highly engaged clinics with strong informatics teams; universal screening is not yet the standard of care across the entire institution. Depression and suicide risk screening can be intimidating, as many clinics may not have access to psychologists and social workers to provide an additional layer of support, as described by Kansara et al. 19 However, increased alerts and CDS can be customized to leverage available community resources in each unique practice setting.

Conclusion

We were able to successfully screen adolescents for depression and suicidality and provide timely awareness to all care team members through a suite of interventions including separate alerts for positive suicide and depression screens, alerting all providers until risk was acknowledged, and integrating multiple layers of safety into clinical workflows through clinical documentation and patient schedule visibility. Notably, redundancy of alerts was necessary to ensure screening was not missed.

Clinical Relevance Statement

This work demonstrates the importance of creating multiple types of alerting to improve notification of high-risk patients to different members of the clinical care team. We created alerts that accounted for different workflows, including alerts on the schedule, documentation in notes, and interruptive alerts when opening the chart. In addition, it is important to stratify the risk levels for each alert and provide clear guidance on next steps.

Multiple-Choice Questions

1. Which of the following is a key benefit of CDS systems?

   1. They are usually more accurate than humans and completely eliminate the need for provider/clinician input.

   2. They can auto-prescribe medications based on the symptoms and diagnosis.

   3. They provide timely information, to help inform patient care and can improve the speed and accuracy of clinical decision-making.

   4. They decrease the need for specialty referrals and hence decrease health care utilization costs.

   Correct Answer : The correct answer is option c. When CDS tools are designed to provide the right information at the right time in the workflow to the right person, they help increase the speed and accuracy of the clinical team by quickly parsing large amounts of data and identifying potential problems. CDS tools facilitate evidence-based decision-making and enhance patient safety.

2. What is one of the reasons for CDS failure?

   1. Accurate data collection

   2. User-friendly interface design

   3. Involving end users right before implementation

   4. Decreasing the number of interruptive alerts

   Correct Answer : The correct answer is option c. For a successful CDS implementation, end users should be involved in all aspects of the CDS design, testing, and implementation. Even after implementation, it is important to obtain periodic feedback and adjust the CDS as needed.

Supplementary Appendix

Mental Health Screening Administration

In primary care, the Patient Health Questionnaire modified for Adolescents 20 21 (PHQ-9A) screening for depression via paper began in 2013. The questionnaire included the standard nine PHQ questions and four additional questions assessing symptom duration, impact on daily functioning, and previous suicide risk or self-harm. Early in the study period, depression and suicide screening were administered at the provider's discretion. Providers scored questionnaires manually and entered results non-discretely into the electronic health record (EHR) clinic notes. In 2016, some providers began transcribing screening results into a standardized data field. By summer 2019, universal screening across all primary care clinics began for all adolescent patients at preventive care visits. Electronic screening on tablets was deployed in December 2019, allowing automatic integration of screening results to the EHR. We initially utilized the PHQ-9A, which includes a question about suicide risk and self-harm. In August 2020, we added suicide-specific screening with the Ask Suicide-Screening Questions (ASQ). In August 2023, we changed our depression screen to the PHQ-8, which does not include any suicide risk questions, and continued to screen for suicide risk with the ASQ. 15 22 All paper and electronic questionnaires were written in English. For patients whose preferred language is not English, nursing staff verbally administered screenings with a medical interpreter and recorded responses in the EHR. In rheumatology, PHQ-9A screening began in 2017 for patients with a diagnosis of systemic lupus erythematosus annually by leveraging EHR reporting tools. The registration team gave the patient a paper questionnaire upon check-in, and clinic staff manually entered results into the EHR. In February 2022, clinic staff began administering questionnaires via tablets given directly to the patient at check-in. Questionnaires were then auto-assigned to every clinic encounter in November 2022. By 2023, ASQ and PHQ-8 screens were auto-assigned and completed via tablet for every relevant rheumatology clinic encounter.

Mental Health Measurement

In an effort to standardize reporting across multiple PHQ measures, PHQ-8 scores were calculated and reported for all patients, regardless of which PHQ questionnaire was completed. PHQ-8 contains all the questions in the PHQ-9 except one. The questions necessary to calculate a numeric score for the PHQ screen are the same regardless of which version is used. Depression severity was defined by PHQ score: 0 to 4, none or minimal; 5 to 9, mild; 10 to 14, moderate; 15 to 19, moderately severe; >20, severe. 21 The ASQ is a four-item questionnaire with binary response options. A positive screen is defined as a “yes” response to any of the four questions. If a patient has a positive screen, an additional (fifth) question asking about immediate suicide risk is asked. An ASQ screen may be considered negative (zero “yes” responses), non-acute positive (any “yes” response to questions 1–4), or an acute positive (“yes” response to question 5 indicating an immediate risk).

Diagnosis Identification

The following diagnostic codes were used: anxiety (ICD-10-CM F40 or F41), depression (F32 or F33), attention deficit hyperactivity disorder (F90), mood disorder (F43), autism (F84), and systemic lupus erythematosus (M32). The coronavirus disease 2019 (COVID-19) pandemic period was defined as visits between March 11, 2020, and May 11, 2023. 13

Supplementary Table S1. Baseline encounter characteristics of completed Patient Health Questionnaire screening.

	Total ( N  = 50,502)	Primary care ( N  = 50,031)	Subspecialty clinic ( N  = 471)
Number of encounters	50,502	50,031	471
Unique patients	33,297	33,193	115
Visits per patient, median [IQR]	1 [1, 2]	1 [1, 2]	4 [2, 6]
Race
American Indian	48 (0.1%)	48 (0.1%)	0 (0%)
Asian	2,601 (5%)	2,540 (5%)	61 (13%)
Black	27,953 (55%)	27,811 (56%)	142 (30%)
Pacific Islander	150 (0.3%)	148 (0.3%)	2 (1%)
White	16,433 (33%)	16,192 (32%)	241 (51%)
Multirace	3,043 (6%)	3,018 (6%)	25 (5%)
Unknown	274 (0.6%)	274 (0.6%)	0 (0%)
Hispanic ethnicity	9,575 (19%)	9,514 (19%)	61 (12%)
Encounters by year
2016	273 (1%)	273 (1%)	–
2017	1,667 (3%)	1,661 (3%)	6 (1%)
2018	1,528 (3%)	1,475 (3%)	53 (11%)
2019	593 (1%)	542 (1%)	51 (11%)
2020	6,065 (12%)	6,039 (12%)	26 (6%)
2021	9,425 (19%)	9,395 (19%)	30 (6%)
2022	14,546 (29%)	14,402 (29%)	144 (31%)
2023	16,405 (32%)	16,244 (32%)	161 (34%)
COVID-19	34,259 (68%)	34,008 (68%)	251 (53%)
Age in years, median [IQR]	14 [13, 16]	14 [13, 16]	17 [15, 19]
Language
English	32,271 (64%)	31,854 (64%)	417 (89%)
Spanish	8,219 (16%)	8,180 (16%)	39 (8%)
Nepali	1,506 (3%)	1,505 (3%)	1 (1%)
Somali	49,96 (10%)	4,995 (10%)	1 (1%)
Other	3,510 (7%)	3,497 (7%)	13 (3%)
Depression severity
None	35,778 (71%)	35,450 (71%)	328 (70%)
Mild	9,519 (19%)	9,434 (19%)	85 (18%)
Moderate	3,640 (7%)	3,602 (7%)	38 (8%)
Moderately severe	1,246 (2%)	1,232 (2%)	14 (3%)
Severe	319 (1%)	313 (1%)	6 (1%)
Suicide severity a
None	26,711 (92%)	26,571 (92%)	140 (87%)
Non-acute positive	2,098 (7%)	2,078 (7%)	20 (12%)
Acute positive	175 (1%)	174 (1%)	1 (1%)
Problem list diagnosis
Anxiety	2,731 (5%)	2,443 (5%)	288 (61%)
Depression	2,629 (5%)	2421 (4%)	208 (44%)
ADHD	6,16 (1%)	603 (1%)	13 (3%)
Mood disorder	1,281 (3%)	1,257 (3%)	24 (5%)
Autism	238 (1%)	232 (1%)	6 (1%)
Visit diagnosis
Anxiety	2,375 (5%)	2,270 (5%)	105 (22%)
Depression	3,209 (6%)	3,130 (6%)	79 (17%)
ADHD	2,278 (5%)	2,276 (5%)	2 (1%)
Mood disorder	1,379 (3%)	1,367 (3%)	12 (3%)
Autism	606 (1%)	605 (1%)	1 (1%)

Abbreviations: ADHD, attention deficit hyperactivity disorder; COVID, coronavirus disease 2019; IQR, interquartile range; PHQ, Patient Health Questionnaire.

^aLimited to 28,984 visits.

Supplementary Table S2. Alert deployments by depression category.

PHQ category	Total number of encounters ( N  = 50,502)	No alerts available ( N  = 44,764)	Iteration 1: PHQ + ASQ combined alert ( N  = 1,560)	Iteration 2: PHQ only ( N  = 2,924)	Iteration 2: ASQ alerts only ( N  = 2,647)	Iteration 3: PHQ only + score ≥10 ( N  = 840)	Iteration 3: ASQ alerts only ( N  = 118)
None	35,778	34,252 (76%)	373 (24%)	753 (26%)	612 (23%)	72 (9%)	21 (18%)
Mild	9,519	8,004 (18%)	449 (29%)	892 (31%)	834 (31%)	93 (11%)	28 (24%)
Moderate	3,640	1,971 (4%)	415 (27%)	774 (26%)	721 (27%)	456 (54%)	34 (29%)
Moderate–Severe	1,246	461 (1%)	245 (16%)	365 (12%)	338 (13%)	168 (20%)	23 (19%)
Severe	319	76 (0.1%)	68 (4%)	127 (4%)	119 (5%)	46 (5%)	7 (6%)
No PHQ recorded	–	–	10 (1%)	13 (1%)	23 (1%)	5 (1%)	5 (4%)

Abbreviations: ASQ, Ask Suicide-Screening Questions; PHQ, Patient Health Questionnaire.

Funding Statement

Funding This project was supported, in part, by The Ohio State University Clinical and Translational Science Institute (CTSI) under Grant Number UL1TR002733. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.