Suicide Screening in Primary Care: Use of an Electronic Screener to Assess Suicidality and Improve Provider Follow-Up for Adolescents

Abstract

Purpose:

The purpose of this study was to assess the feasibility of using an existing computer decision support system to screen adolescent patients for suicidality and provide follow-up guidance to clinicians in a primary care setting. Predictors of patient endorsement of suicidality and provider documentation of follow-up were examined.

Methods:

A prospective cohort study was conducted to examine the implementation of a CDSS that screened adolescent patients for suicidality and provided follow-up recommendations to providers. The intervention was implemented for patients aged 12–20 years in two primary care clinics in Indianapolis, Indiana.

Results:

The sample included 2,134 adolescent patients (51% female; 60% black; mean age =14.6 years [standard deviation = 2.1]). Just over 6% of patients screened positive for suicidality. A positive endorsement of suicidality was more common among patients who were female, depressed, and seen by an adolescent‒medicine board-certified provider as opposed to general pediatric provider. Providers documented follow-up action for 83% of patients who screened positive for suicidality. Documentation of follow-up action was correlated with clinic site and Hispanic race. The majority of patients who endorsed suicidality (71%) were deemed not actively suicidal after assessment by their provider.

Conclusions:

Incorporating adolescent suicide screening and provider follow-up guidance into an existing computer decision support system in primary care is feasible and well utilized by providers. Female gender and depressive symptoms are consistently associated with suicidality among adolescents, although not all suicidal adolescents are depressed. Universal use of a multi-item suicide screener that assesses recency might more effectively identify suicidal adolescents.

Suicide is the second leading cause of death among adolescents aged 10 to 19, with over 2,200 suicide-related deaths recorded in that age group in 2014 [1]. Moreover, the rate of adolescent suicide shows no signs of decreasing. From 2007 to 2015, the suicide rate among males aged 15 to 19 increased 31%, and among females, it doubled, reaching the highest rate recorded for the period 1975–2015 [2]. In 2015, about 9% of high school students (grades 9–12) reported attempting suicide and about 3% reported having made an attempt that required medical intervention [3]. Other suicide-related behaviors such as suicidal ideation and making a plan to commit suicide are also increasingly prevalent among US adolescents. In 2015, over 17% of high school students reported having seriously considered attempting suicide and over 14% reported having made a plan to commit suicide [3].

Most adolescents visit a primary care provider at least once a year [4]. Although some adolescents are not engaged in primary care [5], the primary care visit represents an opportunity to conduct suicide screening and intervention for those who are. However, most primary care providers do not screen adolescents for suicide risk, perhaps because of suicide being a low base rate event [6,7], a lack of formal psychiatric training, and/or a general uncomfortableness screening adolescent patients for suicide risk [6]. Regardless of the reason, as many as 83% of adolescent suicide attempters are not identified as such by their primary care providers [6]. One method of increasing rates of suicide risk screening among adolescents is to standardize the screening process. Interventions that introduce standardized suicide risk screening questions into already-existing electronic medical record questionnaires have demonstrated feasibility [8] and achieved increased rates of screening for, detection of, and referral for suicidality [9] among adolescent patient populations. Although several studies have examined the feasibility of suicide screening in primary care, none has examined how electronic suicide risk screening can also be used to inform specific provider follow-up actions.

Despite evidence that increased screening practices lead to increased rates of detection and treatment, recommendations for adolescent suicide screening in primary care are inconsistent. The American Academy of Pediatrics and the American Medical Association recommend annual suicide screening for adolescents in primary care [10–13]. The US Preventative Services Task Force, however, concluded that there is insufficient evidence to recommend universal screening for suicide risk among asymptomatic adolescents and adults [14]. The present study assesses the use of a computerized clinical decision support system (CDSS) to screen adolescents for suicide risk, deliver follow-up recommendations to the provider, and document actual provider follow-up actions in a primary care setting.

Methods

CHICA system overview

The Child Health Improvement through Computer Automation (CHICA) system is a CDSS that integrates electronic medical record (EMR) system data, previsit screening data, and correlative provider responses from previous visits to generate appropriate follow-up recommendations, tools, and provider prompts. When a patient checks into a clinic, the CHICA system automatically generates a previsit screener form (PSF) based on information in the patient’s EMR including age, developmental stage, current and previous medical conditions, and known risk factors for morbidity. The PSF is administered on an electronic tablet and is completed before the provider encounter. It consists of two parts: (1) a form for the nurse to record vitals, height, and weight; and (2) a 20-item patient questionnaire on a variety of health-related topics such as diet, depression, sexual behaviors, and substance use.

A provider worksheet (PWS) is generated based on patient responses to PSF items. Partially completed PSFs still generate a PWS. The PWS is printed and given to the provider for consultation during the encounter. It consists of six prompts, each identifying a health need based on patient responses to the PSF questionnaire or information in the EMR. The prompts are prioritized by the CHICA system based on national clinical guidelines and a decision analytic algorithm [15]. Each prompt consists of an explanation of the health need followed by corresponding action items, each with a checkbox, that allow the provider to document data, procedures, prescriptions, referrals, and other actions that might take place during the encounter. Figure 1 shows the provider prompt that is generated when a patient endorses suicidality. Providers may select more than one follow-up option for the suicide prompt; in fact, the first action (“high risk for suicide”) is followed by an arrow to indicate that an additional follow-up action is warranted.

Figure 1.

Example physician prompt—suicide.

When a provider responds to worksheet prompts, the form is scanned and uploaded by clinic staff after the patient encounter. The CHICA system analyzes provider responses using optimal mark and character recognition to detect which action items were taken by the provider and then records the appropriate actions in a database. Together, the PSF and PWS provide screening and correlative follow-up options for providers. More detailed information about CHICA including rule processing, development of Arden rules, data storage, and implementation can be found in previous publications [16–19].

Study design and screening process

Adolescents aged 12–20 who presented to their pediatric primary care clinic for an annual (nonsick) or sick visit were selected to participate in the controlled trial. The suicide screening tool was implemented in two primary care federally qualified health center clinics that utilize CHICA and are part of an urban, Midwest county hospital system (Eskenazi Health). Providers were primarily trained in pediatrics, family medicine, and combined internal medicine and pediatrics, with some having completed subspecialty fellowship training in adolescent medicine. Many clinic visits occurred during adolescent-specific clinic times. Institutional review board approval was received from the local university.

Study variables

Suicide risk was assessed on the PSF using a single question based on American Academy of Pediatrics (AAP) and American Medical Association recommendations [10,11,13]: “Have you ever seriously thought about killing yourself, made a plan, or actually tried to kill yourself?” In addition to suicide risk, depression was also assessed as part of the PSF using the Patient Health Questionnaire-2 (PHQ-2) [20] as an initial screen. This instrument consists of two questions assessing anhedonia and depressed mood over the past 2 weeks. If the adolescent responded “yes” to one or both questions in the PHQ-2, the Patient Health Questionnaire-9 (PHQ-9), a longer nine-item screening tool for depression [21], was automatically administered. The PHQ-2 and PHQ-9 have sensitivities of 74% and 96% and specificities of 75% and 82%, respectively [20,21].

Adolescent substance use was assessed according to the AAP’s Bright Futures guidelines [22], which recommend a two-part screening process. The first part consists of a three-item prescreener, which asks the patient about drinking alcohol, smoking marijuana or hashish, and using anything else to get high during the past 12 months. A positive response to any item triggers a longer six-item measure called the CRAFFT [22,23]. For the purposes of this study, patients who answered “yes” to any of the three prescreener questions about substance abuse were categorized as having used substances in the past 12 months.

Chart abstraction

Data indicated by providers on PWS prompts were captured by the CHICA system through optimal mark recognition. However, in some cases, the provider did not indicate a response to a PWS suicide prompt. In cases where a patient screened positive for suicide and no provider action was documented on the PWS, a chart review was conducted by trained research assistants to review manually entered provider notes to find indications of follow-up action. Thirty percent of cases included in the chart abstraction were audited by a second research assistant with a pooled kappa of κ =.79, which indicates substantial agreement [24]. Data acquired during the chart abstraction were incorporated into the CHICA-generated dataset.

Analyses

Descriptive statistics (means and standard deviations [SD], frequencies) were calculated for patient demographic characteristics, clinic site, provider type (general pediatric or adolescent−medicine board-certified [AMBC]), endorsement of suicidality, endorsement of substance use, and endorsement of depressive symptoms on the PHQ-2. Descriptive statistics were also calculated for provider follow-up actions at the patient’s first visit with suicide risk endorsement. For never-suicidal patients, provider follow-up actions were drawn from the first overall visit during the study time frame.

We then calculated the prevalence of suicidality and described provider follow-up treatment practices. Fisher’s exact tests were performed to determine the extent to which certain demographic and clinical factors were associated with patient endorsement of suicidality. Lastly, two predictive models were conducted to model (1) adolescent endorsement of suicidality on the entire patient population (N = 2,130) and (2) provider follow-up practices on patients who screened positive for suicidality (n = 131). Logistic regression analysis was performed for model (1). Mixed-effects logistic regression analysis was performed for model (2) to account for the clustering among patients’ follow-up practices within providers. For both models, we performed internal validation by splitting the data into training and validation datasets. We then used receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) to assess the validity of the model. The model was fit using the training data set and then assessed by applying the model to the validation data set.

Results

Our sample population included 2,134 adolescents (51% female; 60% black; mean age = 14.6 years [SD = 2.1]) who presented to a primary care provider for either a sick visit or nonsick visit during the study period and who completed a PSF. Table 1 contains demographic information for the overall patient sample as well as for the subset that endorsed suicidality. Just over 6% of patients (n = 131) endorsed suicidality. Table 2 shows the results of a univariate analysis of patient demographic and clinical factors on suicidality. According to these results, patients who were female, patients who reported depressive symptoms, and patients who were seen by an AMBC provider (as opposed to a general pediatric provider) were more likely to report suicidality. Substance use was also associated with suicidality but with borderline significance. There were no known suicide attempts during the study period.

Table 1.

Patient demographic and clinical factors

	All patients N = 2,130a		Patients who endorsed suicidality n = 131
	n	%	n	%
Race
Black	1285	60.3	92	70.2
Hispanic	305	14.3	13	9.9
Other	374	17.5	15	11.5
White	169	7.9	11	8.4
Gender
Female	1088	51.1	101	77.1
Male	1042	48.9	30	22.9
Mean age (SD)	14.6 (2.1)		14.8 (1.9)
Insurance
Public	1134	53.2	79	60.3
Private	103	4.8	3	2.3
Self-pay	148	7.0	6	4.6
Other	746	35.0	43	32.8
Clinic type
General	1803	84.6	103	78.6
Adolescent	328	15.4	28	21.4

SD = standard deviation.

^aFour patients excluded because of missing information on gender.

Table 2.

Univariate analysis of patient and clinical factors on suicidality

	% whoendorsed suicidality	Odds ratio(95% CI)	p Value
Gender
Female	9.3	3.45 (2.27, 5.24)	<.01
Male	2.9	—
Race
Black	7.2	1.11 (.58, 2.11)	.08
Hispanic	4.3	.64 (.28, 1.46)
White	6.5	—
Other	4.0	.61 (.27, 1.35)
Age group
Early (12–14)	5.4	—	.20
Middle (15–17)	7.4	1.39 (.96, 2.01)
Late (18–21)	5.5	1.02 (.53, 1.98)
Clinic site
Clinic A	6.6	—	.24
Clinic B	5.4	.80 (.55, 1.17)
Provider type
General pediatric	5.7	—	.05
Adolescent medicine	8.5	1.54 (1.00, 1.16)
Substance use
Yes	9.4	1.67 (.96, 2.88)	.07
No	5.9	—
Depressive symptoms (PHQ-2)
Yes	22.4	15.41 (10.13, 23.44)	<.01
No	1.8	—

CI = confidence interval; PHQ-2 = Patient Health Questionnaire-2.

Initially, 32% (n = 42) of records with a positive suicide screen were lacking documentation of provider follow-up action on the PWS. Subsequent chart review and visual review of the PWS, however, decreased this number by 20 records. Of those 20 records, five were found on chart review to indicate provider action on the PWS that had not been recorded in the CHICA system because of either a misplaced check mark or a scanning error. The remaining 15 were found to indicate follow-up action by the provider in the provider note, which is manually entered into the chart and not part of the automated CHICA system. In the end, providers failed to document follow-up for 22 adolescents who endorsed suicidality (17%).

Provider follow-up actions to positive suicide screens are detailed in Table 3. Of the 131 patients who endorsed suicidality, 22 (17%) were lacking documentation of provider follow-up and 93 (71%) were deemed “not suicidal” after assessment by the provider. In the remaining 16 patients (12%) who endorsed suicidality, there were 13 instances of the provider selecting “high risk for suicide,” 10 instances of immediate hospitalization and psychiatric evaluation, 9 referrals to a crisis center, and 12 referrals to a suicide prevention handout. Importantly, providers could select more than one follow-up action for any given patient. Overall, providers documented follow-up for 83% of patients who endorsed suicidality. Clinic site and Hispanic race were both strongly correlated with documentation of provider follow-up as all Hispanic patients and all patients seen at clinic site B received documented follow-up from their provider.

Table 3.

Provider follow-up for adolescents who endorsed suicidality

Follow-up items	n (%)a
Deemed high risk for suicide	13 (9.9)
Immediate hospitalization and psychiatric evaluation	10 (7.6)
Referred to crisis center	9 (6.9)
Referred to suicide prevention handout	12 (9.2)
Deemed not suicidal	93 (71.0)
No documented follow-up	22 (16.8)

^aSum of percentages exceeds 100% because more than one follow-up action was indicated for some patients.

A multivariable logistic regression analysis was conducted to model predictors of suicidality on the entire patient population (N = 2,130). Variables included in this analysis were the same as those included in the univariate analysis. Significant predictors of suicidality included female gender (odds ratio [OR] 3.17 (1.81, 5.53); p <.01) and depressive symptoms as measured by the PHQ-2 (OR 16.66 [9.66, 28.71]; p <.01). A second mixed-effects multivariable logistic regression analysis was conducted to model predictors of provider follow-up on patients who endorsed suicidality (n = 131). Variables included gender, race, age group, provider type, substance use, and depression. Clinic site was excluded from this analysis because it separated the outcome variable as mentioned previously. According to this model, there were no significant predictors of provider follow-up. For both predictive models, AUCs were calculated to summarize their diagnostic accuracy. An AUC of.5 indicates no discrimination and 1.0 indicates perfect discrimination [25]. For the first model, AUC was.86 for the training dataset and.84 for the validation dataset. For the second model, the AUC was.74 for the training dataset and.72 for the validation dataset.

Discussion

Suicide is a significant source of morbidity and mortality among adolescents [13]. Primary care is an ideal setting in which to identify and intervene with suicidal adolescents. Consistent with prior work, this study demonstrates implementation of a suicide screening algorithm into an existing CDSS to screen a large number of adolescents for suicidality as part of routine primary care. Although other studies have examined predictors of adolescent suicidality in community samples, the present study contributes to the literature in that it is one of only two studies to examine predictors of positive suicide screening in primary care settings.

Epidemiology of adolescent suicidality

The rate of adolescent suicidality in the present study (6%) was similar to the rates in previous studies, suggesting that the suicide screening procedures in this study were effective in identifying adolescents with a history of suicidality. Two previous studies reported slightly lower rates of suicidality than does the present study. Using face-to-face provider screening, Wintersteen [9] found that 3.6% of all adolescents endorsed lifetime suicidal thoughts or behaviors, but reported no differences based on demographic characteristics. Using computerized assessments, Husky et al. [26] found that 4.6% of adolescents reported current suicidal thoughts or previous suicide attempts, with no significant differences based on gender, age, or race. In contrast, Gardner, et al. [8] identified higher rates, with 14% of primary care adolescents reporting suicidal thoughts in the past month on tablet-based screenings.

In national surveys, adolescents recruited from community settings have typically reported higher rates of suicidality than those screened in primary care settings In 2015, the Youth Risk Behavior Survey [3] found that 17% of adolescents reported seriously considering attempting suicide in the past 12 months, and the National Comorbidity Survey [27] found that 12% of youth endorsed lifetime suicidal ideation. One possible explanation for a difference in rates of adolescent suicidality between community and clinical settings is that community surveys are often anonymous and participants may perceive that there will be no consequences for reporting suicidality. In a clinical setting, however, patients understand that their provider will see their responses, which may decrease the likelihood of them endorsing suicidality [9]. Another possible explanation is that many highrisk adolescents are not engaged in primary care [28].

Screening and follow-up

According to AAP guidelines [13], adolescents who report ever considering suicide on a screening tool should be assessed by a healthcare provider. We demonstrated that electronic patient screening followed by action prompts for health-care providers is both feasible and well utilized. Providers documented follow-up for 83% of adolescent patients who endorsed suicidality. Of those patients, the majority (71%) were deemed not suicidal. Providers failed to document follow-up in 17% of patients with a positive endorsement of suicidality. However, lack of documentation does not necessarily mean the patient did not receive follow-up attention. Upon chart review, documentation of follow-up action was found elsewhere in the chart for almost half of records that were initially missing it. Even so, lack of documented follow-up is both a clinical and a legal problem with implications beyond the scope of this discussion (see Ref. [29] for a thorough discussion of the ethical issues of CDSS). In an effort to increase compliance with follow-up guidelines, study investigators implemented an emergency response system by which clinic directors received an e-mail if no follow-up action was documented within 24 hours of an encounter involving a positive suicide screen. This system was recently implemented, and the authors cannot speak to its effectiveness at this time.

Risk factors and comorbidity

Depression is widely recognized as a risk factor for suicide [30], with many studies establishing suicide and depression as co-outcomes [31,32] or using depressive symptoms as a selection criterion for suicide risk screening [33–37]. Many providers may view suicidality as a problem that occurs only in individuals with psychiatric disorders [9]. The current study identified depression as a strong predictor of suicidality; however, our results suggest that suicide risk and depression do not always co-occur. In fact, 24% of adolescents who endorsed suicidality did not report depressive symptoms and one of those adolescents was deemed currently suicidal by their provider. Thus, using depressive symptoms as a selection criterion for suicide screening is generally warranted but may result in missed opportunities for identification, intervention, and treatment of suicidality. Primary care providers should consider universal, independent screenings for both depression and suicidality.

Substance use is another risk factor often associated with suicidality in adolescent populations. In the present study, substance use was weakly associated with suicidality (p =.07) in univariate analysis, but other studies report stronger correlations. More specifically, substance abuse disorder diagnoses [38], alcohol use [27,34,39], marijuana use [39], and illicit drug use [27] have all been associated with an increased risk for suicidality in adolescent populations. Adolescents with dual diagnoses of depression and substance use are at an even higher risk for suicide [36]. Based on this information, there is a strong case for primary care providers to screen for both depression and substance use among adolescent patients.

Limitations

One limitation of the current study is that is does not assess recency of suicidality. A multi-item screening tool that assesses recency could provide more detailed information about the urgency and nature of suicide risk, allowing for more tailored follow-up recommendations for the provider. However, the desire for more information must be balanced with clinic-flow and time constraints. The single-item tool used in this study is the item recommended by Bright Futures [40], and as there is a need to assess multiple risk areas in primary care, a single-item measure of suicide is feasible to implement in the primary care setting. A strength of CDSS is that a general single-item measure can be administered universally in primary care, and a brief, multi-item follow-up screen can then be administered to further define risk and guide provider action.

Several adolescents who endorsed suicidality appeared to receive no follow-up from their provider as suggested by a lack of documentation on the PWS. Although chart reviews were helpful in identifying documentation of provider follow-up outside of the CHICA system, it cannot be determined in these cases whether CDSS and associated prompts influenced the provider’s behavior. Additionally, this study could not assess whether patients actually benefitted from suicide screening procedures or provider follow-up. To assess whether patients benefited from provider follow-up would require that data be extracted from visits outside of the study encounter, which was defined for each patient as their first visit during the study time frame when they endorsed suicidality, or for patients who never endorsed suicidality, their first visit during the study time frame. This limits our knowledge of both positive and negative outcomes that might have occurred outside of the study encounter.

Finally, it is impossible to know whether the adolescent patient or their caregiver completed the PSF. Although clinic staff were instructed to hand the tablet computer to the patient and advise that the patient complete the questionnaire, it is possible that a caregiver completed it for the patient or with their help. Allowing a caregiver to be the informant would likely lead to lower rates of reported suicide, depression, and substance use on the screening instrument. We recommend that future studies include a question that identifies the informant as either the patient or someone else.

The present study demonstrated that using a CDSS framework to implement universal suicide screening in a pediatric primary care setting is feasible. Of the total sample, 6% of adolescent patients screened positive for lifetime suicidality, with 10% of those patients determined to be at high risk for attempting suicide after provider assessment. The majority of patients who screened positive for suicidality received follow-up from their provider. Depressive symptoms commonly co-occurred with endorsement of lifetime suicidality; however, one quarter of patients who endorsed lifetime suicidality did not report depressive symptoms, and one of those patients was deemed at high risk for suicide after provider assessment. This suggests that it may be beneficial to screen separately and independently for depression and suicide in primary care settings. Further research should examine whether integrating a multiple-item suicide screening tool into CDSS would be comparably feasible and even more effective in identifying adolescents who are currently experiencing suicidality. A multiple-item measure would also allow providers to better assess the adolescent’s level of suicide risk and tailor follow-up care accordingly.

IMPLICATIONS AND CONTRIBUTION.

Computer decision support systems can be used effectively in primary care to screen adolescents for suicidality and provide follow-up recommendations for providers. Female gender, younger age, and depressive symptoms were associated with lifetime suicidality among adolescents, although not all suicidal adolescents reported depressive symptoms.