Abstract
Background
Caring Contacts can effectively reduce suicide ideation, attempts, and death. In published clinical trials, Caring Contacts were sent by someone who knew the recipient. At scale, Caring Contacts programs rarely introduce the recipient and sender. It is not known whether receiving Caring Contacts from someone unknown is as effective as messages from someone the recipient has met.
Methods
Pragmatic randomized controlled trial comparing Caring Contacts with (CC+) versus without an introductory phone call (CC). Recruitment occurred January-July 2021, with outcomes assessed at 6 months. Participants were primary care patients or healthcare providers/staff reporting adverse mental health outcomes on a qualifying survey. Participants were sent 11 standardized caring text messages over 6 months; when participants replied, they received personalized unscripted responses. CC+ calls were semi-structured. The primary outcome was loneliness (NIH Toolkit).
Results
Participants included 331 patients (mean [SD] age: 45.5 [16.4], 78.9% female) and 335 healthcare providers/staff (mean [SD] age: 40.9 [11.8], 86.6% female). There were no significant differences in loneliness at 6 months by treatment arm in either stratum. In patients, mean (SD) loneliness was 61.9 (10.7) in CC, and 60.8 (10.3) in CC+, adjusted mean difference of −1.0 (95% CI: −3.0, 1.0); p-value=0.31. In providers/staff, mean (SD) loneliness was 61.2 (11) in CC, and 61.3 (11.1) in CC+, adjusted mean difference of 0.2 (95% CI: −1.8, 2.2); p-value=0.83.
Limitations
Study population was 93% white which may limit generalizability.
Conclusions
Including an initial phone call added operational complexity without significantly improving the effectiveness of a Caring Contacts program.
Keywords: Caring Contacts, Loneliness, Depression, Suicide, Mental Health, Brief Contact Intervention
Introduction
Background
The COVID-19 pandemic began amid a high prevalence of adverse mental health conditions.1 The lived experience of the pandemic exacerbated social isolation, loneliness, and mental health disorders.2,3 During the first year of the pandemic, 41% of US adults reported adverse mental health conditions, with healthcare providers and staff at disproportionately high risk.4 This study compared the effectiveness of two versions of a Caring Contacts intervention in patients and healthcare providers and staff experiencing mental distress during the COVID-19 pandemic to determine whether knowing the person sending caring messages is necessary to optimize reductions in loneliness, depression, and suicidal ideation and behavior.
Caring Contacts is one of the only brief interventions with demonstrated effectiveness in reducing suicidal ideation, attempts, and death in randomized clinical trials.5–10 Caring Contacts is recommended as part of standard suicide prevention care by the Joint Commission, 11 the US Department of Veterans Affairs (VA), 12 the US Department of Defense, 12 and the National Action Alliance for Suicide Prevention.13 Caring Contacts involves brief, non-demanding caring messages sent via letters, 5,6 postcards, 7,8 or text messages9. Published trials sent Caring Contacts from someone who knew the recipient, but in practice messages are often sent from someone unknown to the recipient. 14,15 It is unknown whether the intervention is more effective if the recipient knows the person sending the caring messages.
Objective & Hypothesis
The study compared the effectiveness of Caring Contacts sent following an introductory phone call (CC+) versus Caring Contacts without an introductory call (CC). The authors hypothesized that CC+ would improve loneliness, depression, and suicidal ideation and behavior compared to CC. Loneliness was selected as the primary outcome because it is a hypothesized mechanism of the causal pathway for Caring Contacts.
Methods
Design
This study is a pragmatic randomized controlled comparative effectiveness trial, with a parallel design and 1:1 assignment stratified by patients, and healthcare providers/staff. Study objectives, interventions, and outcomes pertain to the individual level.
The St. Luke’s Health System Institutional Review Board approved and oversaw the study. A Data and Safety Monitoring Board monitored the safety and scientific integrity of the trial. The research protocol and statistical analysis plan are available at ClinicalTrials.gov.
Setting
St. Luke’s Health System (St. Luke’s) in Idaho is a private, regional not-for-profit health system that employs over 16,000 healthcare providers and staff and had 1.87 million clinic visits in 2021. All data for this study were collected virtually.
Study Population
Participants were St. Luke’s providers/staff or primary care patients, recruited based on severity of mental distress reported through an initial survey. Eligibility criteria were intentionally broad for the initial survey and included: proficiency in English, willingness to provide informed consent; for providers/staff: (a) 18+ years old and (b) a current provider or employee at St. Luke’s; for patients: (a) ≥12years old, (b) completed primary care visit in last 12 months; and (c) current mobile electronic health record account user. Inclusion criteria for the clinical trial included (1) moderate or high score for loneliness, suicidal ideation, psychological stress, anxiety, and/or depression2 on the initial survey; and (2) access to a phone with call and texting capabilities.
Study Procedures
Individuals were invited to participate in the initial survey via email (providers/staff) or mobile electronic health record message (patients). REDCap (Research Electronic Data Capture) was used for surveys and informed consent. 16,17 Eligible survey participants were contacted by text message to schedule a study enrollment call; informed consent was conducted over the phone written consent in REDCap. Randomization was stratified by stratum and the assigned treatment was displayed on individuals’ informed consent forms. Trial participants completed a pre-intervention baseline survey immediately following consent. Two non-clinician follow-up specialists at the Idaho Crisis and Suicide Hotline (Hotline) were trained on the Caring Contacts intervention and sent caring texts, monitored and responded to incoming text messages and completed introductory phone calls with CC+ participants. Outcomes surveys were sent at 6 months via text or email per participant preference.
Description of Interventions
Participants in both intervention arms received medical and behavioral health services including medications and psychosocial treatment as usual.
Caring Contacts (CC)
The CC intervention comprised 11 standardized caring texts sent over 6 months according to the following schedule: weeks 1, 2, 3, 4, 6, 8, 10, 12, 16, 20, and 24, plus a birthday text if their birthday occurred during follow-up. The schedule and content of outgoing texts was uniform across participants and was developed in consultation with a local lived experience with suicide advisory board. A HIPAA-compliant program called Mosio was used for calls and texting. Participants could choose whether to reply to the caring texts. Hotline follow-up specialists reviewed and responded to incoming text messages with unscripted individually tailored replies, similar to previous text-based Caring Contacts studies.9 Replies adhered to the principles of Caring Contacts (non-demanding, unconditional care) except in instances of suicidal crisis when follow-up specialists asked questions to assess safety.
Caring Contacts with Introductory Call (CC+)
CC+ was the same as CC, plus one introductory phone call to connect the participant and follow-up specialist before texting began. Phone calls were unscripted but generally included: an introduction; a safety/wellbeing check; a discussion of the participants’ mental health and contributing stressors; a review of relevant resources; and a description of the caring text messages. Caring texts were initiated without a phone call if the call was not completed within 2 weeks of enrollment.
Fidelity Monitoring
The Mosio texting platform facilitated real-time oversight of call and texting activity, tracking incoming and outgoing texts with dates and timestamps. Study staff reviewed the content of call notes and text messages and monitored the date and time of outgoing texts to ensure fidelity with the research protocol.
Measures
Primary Outcome
The primary outcome was loneliness. Loneliness is a well-established risk factor for suicide, 18,19 depression, 20–23 stress, 22,24 and anxiety. 20,23 The authors hypothesized that by reducing loneliness, Caring Contacts may also reduce depression, stress, anxiety, and suicidality. Loneliness was measured at baseline and at 6 months using the NIH Toolbox Social Relationship Scales Emotion Battery Loneliness measure, which is validated and psychometrically sound.25 The loneliness measure comprises five items (adults) or seven items (adolescents) rated on a Likert scale. Responses were used to calculate a raw score which was converted to a t-score.
Secondary Outcomes
Secondary outcomes were measured at baseline and 6 months. Suicidal ideation and behavior were assessed using the 6-item Columbia Suicide Severity Rating Scale (C-SSRS) self-assessment screener.26–28 Depression was assessed using the Patient Health Questionnaire-9 (PHQ-9).29,30 Perceived Burdensomeness and thwarted belongingness were measured using the 15-item Interpersonal Needs Questionnaire.19,31
Exploratory Outcomes
Stress, 32 anxiety, 33 alcohol use, marijuana use, illicit drug use,34 and use of outpatient mental healthcare services were exploratory outcomes.
Power & Sample Size
For each stratum, a sample of 330 participants (165 in each arm) was planned to provide 80% power to detect a difference of 5 units in the loneliness outcome, assuming at least 70% of participants would be retained. The minimum clinically important difference (MCID) was defined a priori as 5 T-score units on the loneliness scale.
Statistical Analysis
The analysis population included all randomized, consented participants who completed the baseline survey. Data were analyzed using an intention to treat protocol. Linear regression with robust standard errors was used to determine whether mean loneliness differed between intervention arms, adjusted for the baseline score as a precision variable. The comparative effectiveness of the interventions was modeled separately by stratum. With two active treatment arms, this study’s results show the difference in effectiveness between the interventions, not the efficacy of either intervention compared to an inactive control. Hot deck multiple imputation was used for missing outcome data; missing 6-month primary and secondary outcomes were sampled from complete cases in the same treatment arm, baseline loneliness tertile, and stratum. Twenty complete data sets were imputed, and results were combined across imputations using Rubin’s rule. 35
For secondary outcomes, linear regression was used to estimate the difference in means. Robust/sandwich standard errors were used to allow for departures of the observed standard errors from classic model assumptions, such as heteroskedasticity. Inference is based on a two-sided significance threshold of 0.05; confidence intervals are correspondingly 95%. Analyses were performed in R version 4.1.2 (R Foundation for Statistical Computing). Post-hoc sensitivity analyses used Chi-squared tests for binary outcomes and the Mann-Whitney test for ordinal outcomes.
Randomization & Masking
Randomization occurred at the individual level and was stratified. The study statistician generated a random list of treatment assignments for each stratum with varying block sizes. REDCap pulled the next treatment assignment from the list at the time of randomization. The list was concealed from the study staff conducting enrollment.
This trial was single masked, with most members of the study team including the lead statistician masked to treatment assignment for individuals and aggregate data by treatment arm. Participants were aware of the treatment arm to which they were assigned but were unaware of the alternative treatment arm. Masking interventionists or study participants to the assigned intervention was not feasible due to the nature of the intervention.
Role of the Funder
This study was funded by the Patient-Centered Outcomes Research Institute® (PCORI). PCORI had no role in study design, data collection, data analysis, data interpretation, or writing of this article.
Results
Demographic & Baseline Characteristics
3,646 individuals completed the initial survey. Of those, 331 patients and 335 healthcare providers/staff were enrolled in the clinical trial. Table 1 describes participant demographics and baseline characteristics by stratum and treatment arm. Participants were mostly female (82.7%), white (93.2%), and non-Hispanic (91.3%), with a mean age of 43.1 years (SD: 14.4). Less than 1% of the study population was under age 18. Unemployment was common (40%) among patients. Baseline clinical characteristics are reported in Table 2 and Table 3 and were similar across treatment arms within strata.
Table 1:
Demographic Characteristics of MHAPPS Trial Participants by Strata and Study Arm
| Patients | Providers/Staff | All Participants | ||||
|---|---|---|---|---|---|---|
| CC N=166 |
CC+ N=165 |
CC N=168 |
CC+ N=167 |
CC N=334 |
CC+ N=332 |
|
| Sex | ||||||
| Female | 130 (78.3%) | 131 (79.4%) | 140 (83.3%) | 150 (89.8%) | 270 (80.8%) | 281 (84.6%) |
| Ethnicity | ||||||
| Hispanic or Latino | 9 (5.4%) | 11 (6.7%) | 12 (7.1%) | 19 (11.4%) | 21 (6.3%) | 30 (9%) |
| Not Hispanic or Latino | 156 (94%) | 154 (93.3%) | 153 (91.1%) | 145 (86.8%) | 309 (92.5%) | 299 (90.1%) |
| Unknown | 1 (0.6%) | 0 (0%) | 3 (1.8%) | 3 (1.8%) | 4 (1.2%) | 3 (0.9%) |
| Age | ||||||
| Mean (SD) | 44.3 (17) | 46.7 (15.7) | 41.6 (12.2) | 40.1 (11.4) | 43 (14.8) | 43.4 (14.1) |
| Median (min, ma) | 43 (12,89) | 46 (17,83) | 39 (20,69) | 39 (21,72) | 41 (12,89) | 42 (17,83) |
| Race | ||||||
| American Indian/Alaska Native | 2 (1.2%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (0.6%) | 0 (0%) |
| Asian | 3 (1.8%) | 1 (0.6%) | 1 (0.6%) | 2 (1.2%) | 4 (1.2%) | 3 (0.9%) |
| Black or African American | 2 (1.2%) | 1 (0.6%) | 1 (0.6%) | 0 (0%) | 3 (0.9%) | 1 (0.3%) |
| Native Hawaiian or Pacific Islander | 1 (0.6%) | 0 (0%) | 1 (0.6%) | 0 (0%) | 2 (0.6%) | 0 (0%) |
| Caucasian or White | 153 (92.2%) | 156 (94.5%) | 155 (92.3%) | 157 (94%) | 308 (92.2%) | 313 (94.3%) |
| Other or multiple | 4 (2.4%) | 4 (2.4%) | 7 (4.2%) | 5 (3%) | 11 (3.3%) | 9 (2.7%) |
| Unknown | 1 (0.6%) | 3 (1.8%) | 3 (1.8%) | 3 (1.8%) | 4 (1.2%) | 6 (1.8%) |
| Employed at baseline | 101 (60.8%) | 96 (58.2%) | 168 (100%) | 166 (99.4%) | 269 (80.5%) | 262 (78.9%) |
| Healthcare employee type | ||||||
| Providers (physicians & advanced practice providers) | -- | -- | 8 (4.8%) | 8 (4.8%) | -- | -- |
| Pharmacy, patient access specialists, lab, specialists, technicians, medical assistants | -- | -- | 26 (15.5%) | 20 (12%) | -- | -- |
| Trades & food service | -- | -- | 1 (0.6%) | 2 (1.2%) | -- | -- |
| Nursing | -- | -- | 57 (33.9%) | 67 (40.1%) | -- | -- |
| Office work & educators | -- | -- | 62 (36.9%) | 60 (35.9%) | -- | -- |
| Senior management | -- | -- | 2 (1.2%) | 2 (1.2%) | -- | -- |
| Social support services | -- | -- | 11 (6.5%) | 6 (3.6%) | -- | -- |
| Other | -- | -- | 1 (0.6%) | 1 (0.6%) | -- | -- |
Table 2:
MHAPPS Trial Baseline Clinical Characteristics and Primary & Secondary Outcomes at 6 Month Follow-up
| Patients | Providers/Staff | |||||||
|---|---|---|---|---|---|---|---|---|
|
| ||||||||
| CC | CC+ | Adjusted Mean Difference (95% CI) | p-value | CC | CC+ | Adjusted Mean Difference (95% CI) | p-value | |
| N=166 | N=165 | N=168 | N=167 | |||||
| Loneliness (NIH Loneliness Scale) a | ||||||||
| Baseline loneliness | ||||||||
| Mean (SD) | 64.4 (9.9) | 64.4 (10.3) | 62.6 (9.3) | 62.5 (9.5) | ||||
| Moderate/high loneliness, No. (%) | 132 (79.5%) | 128 (77.6%) | 125 (74.4%) | 121 (72.5%) | ||||
| Follow-up Loneliness | ||||||||
| Mean (SD) | 61.9 (10.7) | 60.8 (10.3) | −1.0 (−3.0, 1.0) | 0.31 | 61.2 (11) | 61.3 (11.1) | 0.2 (−1.8, 2.2) | 0.83 |
| Moderate/high loneliness, No. (%) | 98 (59%) | 87 (52.7%) | 103 (61.3%) | 98 (58.7%) | ||||
|
| ||||||||
| Suicidal Ideation & Behavior (C-SSRS) b | ||||||||
| Baseline suicidal ideation/behavior (SI) | ||||||||
| Mean (SD) | 0.5 (1.3) | 0.4 (1.1) | 0.3 (0.9) | 0.4 (1.1) | ||||
| Any suicidal ideation, No. (%) | 39 (23.5%) | 39 (23.6%) | 28 (16.7%) | 26 (15.6%) | ||||
| Moderate/high SI, No. (%) | 12 (7.2%) | 9 (5.5%) | 7 (4.2%) | 9 (5.4%) | ||||
| Follow-up suicidal ideation/behavior (SI) | ||||||||
| Mean (SD) | 0.4 (0.9) | 0.6 (1.3) | 0.2 (0.002, 0.5) | 0.048 | 0.5 (1.2) | 0.4 (1.1) | −0.2 (−0.4, 0.1) | 0.2 |
| Any suicidal ideation, No. (%) | 40 (24.1%) | 46 (27.9%) | 41 (24.4%) | 32 (19.2%) | ||||
| Moderate/high SI, No. (%) | 11 (6.6%) | 15 (9.1%) | 18 (10.7%) | 11 (6.6%) | ||||
|
| ||||||||
| Perceived Burdensomeness (PB) (INQ-15) c | ||||||||
| Baseline Perceived Burdensomeness (PB) | ||||||||
| Mean (SD) | 10.1 (6.7) | 9.2 (5.4) | 7.9 (3.6) | 8.0 (4.9) | ||||
| Moderate/high PB, No. (%) | 16 (9.6%) | 12 (7.3%) | 7 (4.2%) | 5 (3.0%) | ||||
| Follow-up Perceived Burdensomeness | ||||||||
| Mean (SD) | 9.6 (6.8) | 9.5 (6.7) | 0.5 (−0.6, 1.6) | 0.38 | 8.4 (5.1) | 8.5 (5.1) | 0.1 (−0.8, 1.0) | 0.81 |
| Moderate/high PB, No. (%) | 13 (7.8%) | 15 (9.1%) | 10 (6.0%) | 7 (4.2%) | ||||
|
| ||||||||
| Thwarted Belongingness (TB) (INQ-15) c | ||||||||
| Baseline Thwarted Belongingness (TB) | ||||||||
| Mean (SD) | 31.1 (11.1) | 30.3 (11.4) | 31.4 (10.0) | 30.4 (11.8) | ||||
| Moderate/high TB, No. (%) | 71 (42.8%) | 62 (37.6%) | 65 (38.7%) | 62 (37.1%) | ||||
| Follow-up Thwarted Belongingness | ||||||||
| Mean (SD) | 29.3 (12.2) | 27.6 (12.3) | −0.4 (−2.6, 1.8) | 0.73 | 27.8 (12.3) | 28.6 (12.4) | 1.2 (−0.8, 3.2) | 0.24 |
| Moderate/high TB, No. (%) | 62 (37.3%) | 47 (28.5%) | 47 (28.0%) | 58 (34.7%) | ||||
|
| ||||||||
| Depression (PHQ-9) d | ||||||||
| Baseline Depression | ||||||||
| Mean (SD) | 10.5 (6.1) | 10.0 (6.0) | 8.4 (5) | 8.6 (5.4) | ||||
| Moderate/high depression, No. (%) | 87 (52.4%) | 82 (49.7%) | 66 (39.3%) | 66 (39.5%) | ||||
| Follow-up Depression | ||||||||
| Mean (SD) | 8.6 (6.2) | 7.9 (5.8) | −0.4 (−1.5, 0.7) | 0.45 | 7.7 (5.1) | 8.1 (5.4) | 0.3 (−0.7, 1.3) | 0.51 |
| Moderate/high depression, No. (%) | 65 (39.2%) | 52 (31.5%) | 59 (35.1%) | 62 (37.1%) | ||||
Note: bolded text indicates results at 6-month follow-up.
The NIH Toolbox Social Relationship Scale for Loneliness is a validated method for measuring loneliness. Participants rate items on a 5-point scale, with options ranging from never (1) to always (5). This creates a raw score, which is then converted to a t-score, with higher scores indicating greater levels of loneliness. T-score of 50 represents the mean of the US general population (based on the 2010 Census) and 10 T-score units represents one standard deviation. A T-score of 60.7 or more (adults) or 60 or more (youth) indicates moderate to high levels of loneliness.
The Columbia Suicide Severity Rating Scale (C-SSRS) 6-item screener (self-assessment lifetime-recent (baseline) and since last visit (6 months) for Primary Care settings versions will be used). The C-SSRS is a validated tool to assess suicidality. C-SSRS score is determined based on the highest question number to which the participant responds “yes”. For example, a score of 5 would be assigned to a participant who responded “yes” to Question 5 and any or all preceding questions. Higher scores are indicative of greater risk for suicide. C-SSRS screener scores range from 0 to 6, with higher scores indicative of higher suicide risk. Scores of 3 and higher indicate moderate to high risk for suicide.
The Interpersonal Needs Questionnaire (INQ) was developed from the Interpersonal Theory of Suicide and the 15-item version (INQ15) is a validated tool to measure both perceived burdensomeness (PB) (6 items, scores range from 6 to 42) and thwarted belongingness (TB) (9 items; scores range from 9 to 63). Individuals provide a self-report response to each item ranging from 1 (Not at all true for me) to 7 (Very true for me). The appropriate items are reverse coded, and items are summed to calculate the TB and PB subscale scores with higher scores indicating greater TB and PB. Clinical Cutoff scores predicting desire for death (moderate/high as reported in table above) are 35 or greater for TB and 19 or greater for PB. Clinical cutoff scores predicting desire for suicide are 50 or greater for TB and 30 or greater for PB.
The PHQ-9 is a widely used and validated tool to screen for depression in primary care and other non-psychiatric settings. The tool is composed of 9 questions each with a response of 0–3 which generate a score from 0–27 with higher scores indicating a greater degree of depression. Scores are categorized in the following manner: a score of 5–9 is considered minimal depression, 10–14 is considered mild major, 15–19 is moderate major, and ≥20 is severe major.
Table 3:
MHAPPS Trial Exploratory and Safety Outcomes & Baseline Clinical Characteristics
| Patients | Providers/Staff | |||||||
|---|---|---|---|---|---|---|---|---|
| CC | CC+ | Adjusted Mean Difference (95% CI) | p-value | CC | CC+ | Adjusted Mean Difference (95% CI) | p-value | |
| Anxiety (GAD-7) a | ||||||||
| Baseline Anxiety | N=166 | N=165 | N=168 | N=167 | ||||
| Mean (SD) | 9.3 (5.3) | 8.8 (5.3) | 7.8 (4.4) | 8.4 (5) | ||||
| Moderate/high anxiety, No. (%) | 62 (37.3%) | 61 (37.0%) | 44 (26.2%) | 51 (30.5%) | ||||
| Follow-up Anxiety | N=165 | N=159 | N=165 | N=166 | ||||
| Mean (SD) | 7.9 (5.6) | 7.3 (5) | −0.3 (−1.3, 0.7) | 0.54 | 6.8 (4.7) | 8.2 (5.1) | 1.0 (0.1, 2.0) | 0.04 |
| Moderate/high anxiety, No. (%) | 44 (26.5%) | 39 (23.6%) | 38 (22.6%) | 49 (29.3%) | ||||
|
| ||||||||
| Stress (NIH Perceived Stress Scale) b | ||||||||
| Baseline Stress | N=166 | N=165 | N=168 | N=167 | ||||
| Mean (SD) | 60.7 (8.7) | 59.6 (8.6) | 58.5 (7.3) | 59.5 (7.7) | ||||
| Moderate/high stress, No. (%) | 98 (59%) | 85 (51.5%) | 80 (47.6%) | 94 (56.3%) | ||||
| Follow-up Perceived Stress | N=165 | N=159 | N=165 | N=166 | ||||
| Mean (SD) | 56.5 (10.7) | 55.3 (10.1) | 55.1 (9.2) | 57.6 (9.6) | ||||
| Moderate/high stress, No. (%) | 64 (38.6%) | 45 (27.3%) | 59 (35.1%) | 70 (41.9%) | ||||
| Change from baseline | −4.2 (9.3) | −4.4 (8) | −0.2 (−2.1, 1.7) | 0.84 | −3.3 (10.3) | −1.9 (9.9) | 1.5 (−0.7, 3.7) | 0.18 |
|
| ||||||||
| Recent mental health treatment c | N=164 | N=158 | N=164 | N=166 | ||||
| At follow-up, n (%) | 119 (71.7%) | 105 (63.6%) | −4.5% (−15.4%, 6.3%) | 0.41 | 124 (73.8%) | 123 (73.7%) | 5.5% (−5.2%, 16.2%) | 0.31 |
|
| ||||||||
| Increased substance use at follow-up d | ||||||||
| Increased Tobacco use, n/N (%) | 23/163 (14.1%) | 13/152 (8.6%) | −5.6% (−12.5%, 1.4%) | 0.12 | 16/164 (9.8%) | 6/164 (3.7%) | −6.1% (−11.5%, −0.7%) | 0.03 |
| Increased alcohol use, n/N (%) | 42/162 (25.9%) | 38/153 (24.8%) | −1.1% (−10.7%, 8.5%) | 0.83 | 52/164 (31.7%) | 55/162 (34%) | 2.2% (−8.0%, 12.5%) | 0.67 |
| Increased marijuana use, n/N (%) | 15/156 (9.6%) | 19/152 (12.5%) | 2.9% (−4.1%, 9.9%) | 0.42 | 17/156 (10.9%) | 14/159 (8.8%) | −2.1% (−8.7%, 4.5%) | 0.53 |
| Increased illicit drug use, n/N (%) | 7/159 (4.4%) | 3/157 (1.9%) | −2.5% (−6.3%, 1.4%) | 0.2 | 3/159 (1.9%) | 2/165 (1.2%) | −0.7% (−3.4%, 2.0%) | 0.62 |
|
| ||||||||
| Safety Outcomes e | N=166 | N=165 | N=168 | N=167 | ||||
| Any death, n (%) | 0 (0%) | 0 (0%) | 1 (0.6%) | 0 (0%) | ||||
| Death by suicide, n (%) | 0 (0%) | 0 (0%) | 1 (0.6%) | 0 (0%) | ||||
| Attempted suicide, n (%) | 0 (0%) | 0 (0%) | 2 (1.2%) | 0 (0%) | ||||
| Interrupted/aborted suicide attempt, n (%) | 0 (0%) | 1 (0.6%) | 3 (1.8%) | 3 (1.8%) | ||||
Symptoms of anxiety were assessed using the GAD-7, a brief self-report scale frequently used in the identification of Generalized Anxiety Disorder. The tool is composed of seven items, which are rated 0–3 to generate a score from 0–21. Higher scores indicate a greater severity of generalized anxiety symptoms. Scores of 11 and higher indicate moderate to high anxiety.
The National Institute of Health (NIH) Toolbox Self-Efficacy Scales Perceived Stress measure is a validated tool to measure the stress and coping resources of an individual. The measure is comprised of ten items which are scored and granted a t-score specific to adult or adolescent participants. Higher t-scores are indicative of higher levels of stress. T-score of 50 represents the mean of the US general population (based on the 2010 Census) and 10 T-score units represents one standard deviation. A T-score of 60.5 or more for adults or 60.8 or more in adolescents indicates moderate to high levels of stress.
Any counseling, therapy, or other mental health treatment in the last 6 months.
Number of participants reporting increased alcohol, tobacco, marijuana, and illicit drug use, measured by questions adapted from Youth Risk Behavior Survey posed in relation to timing of COVID-19 pandemic.
Three elements of the full version of the Columbia Suicide Severity Rating Scale (C-SSRS) (lifetime-recent (baseline) and since last visit (6 months)) were complied to create a Suicide Attempts Survey. These measured self-reported aborted or self-interrupted suicide attempts, interrupted suicide attempts, and actual suicide attempts. Non-lethal self-harm and lethal means used for attempts or completions were also collected. These elements were not included in the C-SSRS score but were compared across the intervention groups. Medical records were used to measure suicide completion.
Timing of the study and COVID-19 Pandemic
The study took place from January 2021 – January 2022. Recruitment occurred between January 18, 2021, and July 6, 2021. Participants were followed for 6 months. All outcome assessments were completed by January 6, 2022. During the outcome assessment period, Idaho was severely impacted by the Delta variant of COVID-19. All health systems in Idaho operated under crisis standards of care from September – November, 2021.
Participant Flow
Figure 1 is the participant flow diagram. Retention was high with over 98% of participants completing the 6-month outcome survey.
Fig. 1.
MHAPPD trial participant flow diagram.
Primary Outcome
There was no significant difference in loneliness by treatment arm in either patients (−1.0 difference, 95% CI: (−3.0, 1.0), p=0.31) or providers/staff (0.2 difference, 95% CI: (−1.8, 2.2), p=0.83). Including an introductory phone call did not significantly improve loneliness among Caring Contacts recipients with mental distress over a 6-month period. Results for loneliness are summarized in Table 2.
Secondary Outcomes
There was a statistically significant difference in suicidality by treatment arm in patients, but the magnitude of difference was small (0.2 difference, 95% CI: (0.002, 0.5), p=0.048). Post hoc sensitivity analyses were conducted in the patient cohort to further explore this finding. A Mann-Whitney test did not find a difference in C-SSRS between intervention arms (p=0.38), and a Chi-squared test of the difference in proportions of participants with a moderate or high C-SSRS score was not significant (p=0.53). There was no significant difference in suicidality in providers/staff (−0.2 difference, 95% CI: (− 0.4, 0.1), p=0.20). There was no significant difference in depression by treatment arm in patients (−0.4 difference, 95% CI: (−1.5, 0.7), p=0.45) or providers/staff (0.3 difference, 95% CI: (−0.7, 1.3), p=0.51). Neither perceived burdensomeness nor thwarted belongingness differed significantly by treatment arm in either stratum. Results for secondary outcomes are summarized in Table 2.
Other Exploratory Outcomes
Anxiety, stress, recent mental health treatment, increased substance use, and safety outcomes including death, death by suicide, and attempted suicide were assessed as exploratory outcomes. Results are summarized in Table 3. Anxiety was statistically significantly higher among CC+ recipients compared to CC in providers/staff (1.0 difference, 95% CI: (0.1, 2.0), p=0.04) but there was no significant difference among patients (−0.3 difference, 95% CI: (−1.3, 0.7), p=0.54). In a post-hoc sensitivity analysis, a Chi-squared test did not find a difference by study arm in the proportion of employees reporting moderate or high levels of anxiety (p=0.22). No other exploratory outcomes differed significantly by treatment arm.
Participant Satisfaction & Engagement
Participant satisfaction with the interventions was high and similar by treatment arm and stratum. Of CC+ participants who reported receiving a phone call, at least 97% in each stratum were strongly or somewhat satisfied with the phone call. Most participants agreed strongly or somewhat agreed that the caring text messages were helpful, and they were glad to receive them (86.7% in CC and 86.7% in CC+ (providers/staff)) and (92.4% in CC and 90.7% in CC+(patients)). Table 4 summarizes the number of CC+ recipients who received an initial phone call and the number of incoming and outgoing text messages by intervention arm and stratum.
Table 4:
MHAPPS Trial Summary of Ingoing and Outgoing Text Messages
| Patients | Providers/Staff | ||||
|---|---|---|---|---|---|
| CC N=166 |
CC+ N=165 |
CC N=168 |
CC+ N=167 |
||
| Initial call received | No. (%) | -- | 159 (96.4%) | -- | 157 (94%) |
| Number of text messages | mean (SD) | 17.8 (6.3) | 20.7 (5.8) | 15.3 (4.5) | 19 (5.3) |
| sent from Hotline | median (Q1, Q3) | 16 (13, 20.8) | 20 (16, 24) | 14 (12, 17) | 18 (15, 22) |
| Number of text messages | mean (SD) | 9.9 (9.7) | 11.4 (7.5) | 6.2 (6.4) | 9.9 (7.4) |
| sent from participant | median (Q1, Q3) | 8 (3.2, 13) | 11 (6, 15) | 4 (2, 8) | 9 (5, 14) |
Heterogeneity of Treatment Effects
The treatment effect for loneliness was analyzed for sub-groups (specified a priori) based on: age; patient-facing vs. non-patient-facing healthcare employees; baseline suicide risk; ethnicity; sex at birth; gender identity; sexuality; and rural vs. urban residence. Subgroup analysis results are summarized in Supplement table 1. The treatment effect varied significantly based on age category (p-value for the interaction: 0.03). Older adults (50+ years) were lonelier in CC+ compared to CC, though the difference was not statistically significant (2.1 difference, 95% CI: (−0.04, 4.3)). Loneliness was lower (but not statistically significant) in CC+ compared to CC in 12–24 year olds and 25–49 year olds. The treatment effect also varied significantly based on ethnicity (p-value for the interaction: 0.01). Hispanic participants reported significantly lower loneliness in CC+ compared to CC (−7.3 difference, 95% CI: (−12.4, −2.1). While the magnitude of difference was large, there were few Hispanic participants, and this finding should be considered hypothesis-generating.
Discussion
Summary of Results
We did not find evidence that including an introductory phone call significantly improves clinical outcomes or participant satisfaction in a two-way text message Caring Contacts intervention for patients and healthcare providers and staff with mental distress. There were two statistically significant findings: (1) suicidal ideation and behavior was higher among patients receiving CC+ compared to CC, and (2) anxiety was higher among providers/staff receiving CC+ compared to CC. In the context of the overall null findings of the trial across multiple mental health outcomes, the inconsistency of these findings across strata, and the non-significant findings from the post hoc sensitivity analyses, we believe there is insufficient evidence to conclude that there is any clinically meaningful difference in mental health outcomes between intervention arms.
Results in Context
This is the first study to examine Caring Contacts for participants not recruited for suicidality, the first to include loneliness as a primary outcome, and the first to provide Caring Contacts to health care providers and staff. Previously published efficacy trials have compared usual care to usual care plus Caring Contacts,5–10 whereas this comparative effectiveness trial was designed to determine whether caring texts sent by someone the recipient has met improved clinical outcomes or satisfaction compared to caring texts sent by someone unknown to the participant. No such difference was found. In prior studies, Caring Contacts were sent by a clinician who was part of the recipient’s care team. Having a single phone call to connect with an otherwise unknown follow-up specialist is not equivalent to having an established relationship with a caregiver. Additional research is needed to determine whether Caring Contacts from an unknown sender are as effective as messages from a known member of the care team.
While most clinical outcomes improved over time, suicidal ideation increased in both cohorts and intervention arms during the trial. The increase in suicidal ideation was statistically significant in providers/staff (mean change of 0.15, 95% CI: (0.02, 0.27), p=0.02) but not among patients (mean change of 0.02, 95% CI: (−0.11, 0.16), p=0.73). Without an inactive control arm, it is unknown how high rates of suicidal ideation may have been without Caring Contacts. The higher levels of suicidal ideation reported in the follow-up period may reflect regression to the mean or a cohort effect related to the Delta surge of COVID-19 and implementation of crisis standards of care. The increased suicidality was not consistent with changes in the other mental health outcomes, nor was it explained by participant satisfaction, which was very high, consistent with other Caring Contacts research.
Lessons for Scale-Up of Caring Contacts
The results from this study may inform the design and implementation of text message-based Caring Contacts interventions. Removing the requirement to meet or speak with the person to send caring texts makes the intervention easier to deliver at scale, reducing costs and operational complexity.
All states have at least one local suicide and crisis hotline. Health systems can feasibly partner with hotlines to deliver two-way Caring Contacts even in rural or low-resource settings. The nationwide rollout of 988 expands the role of state crisis and suicide prevention hotlines, with an emphasis on providing longer-term follow-up support. Caring Contacts is an evidence-based follow-up model that Hotlines can deliver effectively.
Staffing for two-way text message programs is a commonly cited barrier to scaling Caring Contacts. Because suicide prevention hotlines are typically staffed 24–7, they are well positioned to monitor and respond to incoming text messages. Our intervention team comprised two follow-up specialists tasked with calls and text replies for 666 participants. They communicated their working hours so participants knew when to expect text replies. Overnight and on weekends, Hotline phone room supervisors monitored incoming texts and could respond in true emergencies when participants were at imminent risk for suicide (none occurred during the trial). Other incoming texts waited for response until the follow-up specialists were working.
External Validity
This was a pragmatic clinical trial, conducted in a non-academic regional healthcare system with a local hotline delivering interventions – a model that could be realistically replicated. The study population was mostly female (83%), white (93%), and non-Hispanic (91%), and findings may not be generalizable to populations with different demographics. Few minors enrolled in the trial and the study cannot draw conclusions about adolescents.
Strengths
This trial demonstrated the feasibility of delivering an evidence-based mental health intervention in a largely rural state during a public health crisis without adding work to the healthcare system. The study successfully retained 98% of participants. The trial addresses an important gap in implementation science for Caring Contacts, demonstrating that it may not be necessary to introduce the Caring Contacts author and the recipient.
Limitations
This study has several limitations. A longer follow-up period would have allowed assessment of longer-term outcomes. Selection bias was likely present in (1) the patient sample, as 40% of enrolled patients were unemployed; and (2) the physicians and clinical providers, as they were about half as likely as other cadres of healthcare workers to participate in the study. This is the first time Caring Contacts has been rigorously evaluated in non-suicidal individuals, and without an inactive control arm conclusions about the efficacy of the intervention in this population cannot be made. With no inactive control arm, it is impossible to know whether changes in mental health observed in both intervention arms were due to regression to the mean or a cohort effect related to lived experience of the pandemic, which surged during the follow-up period, versus the effectiveness of the interventions. This is a limitation of all comparative effectiveness trials lacking inactive control arms.
Future Research
While this trial suggests that offering Caring Contacts to healthcare providers, staff, and patients experiencing mental distress is acceptable and feasible, additional research is needed to determine the efficacy of Caring Contacts to reduce mental distress in non-suicidal individuals compared to an inactive control arm. Determining whether results of this study would differ for Hispanic, and/or non-white populations is critical. Research is also needed to understand whether messages from a peer or clinical staff who have treated the patient would be received differently than those from someone unknown. It is unknown whether the findings from this study would be consistent in populations of suicidal individuals. Finally, research is needed to assess the comparative effectiveness of one-way versus two-way Caring Contacts, as the version being scaled often involves one-way communication, in which participants cannot interact with the person supporting them. 14,15
Conclusions
Adding an introductory phone call did not significantly improve the effectiveness of a Caring Contacts intervention in a mostly White, mostly female population of healthcare providers, staff, and patients experiencing mental distress. Delivering the interventions in partnership with the state Crisis and Suicide Hotline facilitated provision of evidence-based mental health support during the peak of the COVID-19 pandemic without straining the healthcare system. Similar health system-community based partnerships could be replicated to deliver two-way text message Caring Contacts interventions elsewhere, including in settings with limited resources.
Supplementary Material
Highlights.
Evidence supports use of Caring Contacts to reduce suicidal ideation and behavior
Important questions remain regarding how to deliver the intervention
Introducing Caring Contacts recipients to the sender may not be necessary
Outcomes were similar whether participants knew the person texting them or not
Eliminating introductions could reduce the cost and complexity of the intervention
Acknowledgements
Research reported in this publication was funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (HIS-2018C3-14695). The information presented in this publication is solely the responsibility of the authors and does not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
Data and Safety Monitoring was provided by the Institute of Translational Health Sciences at the University of Washington, supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Study data were collected and managed using REDCap electronic data capture tools3 hosted at the Institute of Translational Health Sciences. REDCap (Research Electronic Data Capture) is a secure, web-based application designed to support data capture for research studies, providing: 1) an intuitive interface for validated data entry; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for importing data from external sources.
REDCap at ITHS is supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319.
We gratefully acknowledge the support of our team at the Patient-Centered Outcomes Research Institute (PCORI), our research, regulatory, and compliance coordinators, our follow-up specialists at the Idaho Crisis and Suicide Hotline, and the whole St. Luke’s team, including Executive Leaders, Communications, the Research Department and IRB, and the Behavioral Health Service Line. Finally, we wish to recognize the contributions of members of our People with Lived Experience with Suicide Advisory Board, and our participants, who prioritized participating in this study during an exceptionally challenging time.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Trial Registration
The Mental Health Among Patients, Providers, and Staff (MHAPPS) Trial is registered at ClinicalTrials.gov (https://clinicaltrials.gov/ct2/show/NCT04700137), with the following identifier: NCT04700137. The full trial protocol can be accessed at the link above.
Declaration of Interests
The authors declare no competing interests.
Moderate or high risk was defined as follows: NIH Toolkit Loneliness raw score of 13 or greater for adults or 16 or greater for adolescents; or C-SSRS score of 3 or greater; or GAD7 score of 11 or greater; or PHQ9 score of 10 or greater; or NIH Toolkit Stress raw score of 31 or greater for adults or 33 or greater for adolescents.
Paul A. Harris, Robert Taylor, Robert Thielke, Jonathon Payne, Nathaniel Gonzalez, Jose G. Conde, Research electronic data capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009 Apr;42(2):377–81.
References
- 1.Substance Abuse and Mental Health Services Administration, (SAMHSA). Key substance use and mental health indicators in the united states: Results from the 2017 national survey on drug use and health. HHS Publication No. SMA 18-5068, NSDUH Series H-53 2018. Available from: https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHFFR2017/NSDUHFFR2017.pdf. Accessed July 21, 2022.
- 2.Vindegaard N, Benros ME. COVID-19 pandemic and mental health consequences: Systematic review of the current evidence. Brain Behav Immun 2020. doi: S0889-1591(20)30954-5 [pii]. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Czeisler MÉ. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic — united states, june 24–30, 2020. MMWR Morb Mortal Wkly Rep 2020;69. https://www.cdc.gov/mmwr/volumes/69/wr/mm6932a1.htm. Accessed Nov 18, 2020. doi: 10.15585/mmwr.mm6932a1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Czeisler ME, Lane RI, Petrosky E, et al. Mental health, substance use, and suicide ideation during the COVID-19 pandemic-united states, june 24-30, 2020. MMWR Morb Mortal Wkly Rep 2020;69:1049–1057. 10.15585/mmwr.mm6932a1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Motto JA. Suicide prevention for high-risk persons who refuse treatment. Suicide and Life Threatening Behavior 1976;6(4):223–30. [PubMed] [Google Scholar]
- 6.Motto JA, Bostrom AG. A randomized controlled trial of postcrisis suicide prevention. Psychiatric Services 2001;52(6):828–833. 10.1176/appi.ps.52.6.828. doi: 10.1176/appi.ps.52.6.828. [DOI] [PubMed] [Google Scholar]
- 7.Carter GL, Clover K, Whyte IM, Dawson AH, Este CD. Postcards from the EDge project: Randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning. BMJ 2005;331(7520):805–807. 10.1136/bmj.38579.455266.E0. doi: 10.1136/bmj.38579.455266.E0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Hassanian-Moghaddam H, Sarjami S, Kolahi A, Carter GL. Postcards in persia: Randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-poisoning. British journal of psychiatry 2018;198(4):309–316. 10.1192/bjp.bp.109.067199. doi: 10.1192/bjp.bp.109.067199. [DOI] [PubMed] [Google Scholar]
- 9.Comtois KA, Kerbrat AH, DeCou CR, et al. Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention: A randomized clinical trial. JAMA Psychiatry 2019;76(5):474–483. https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,url,uid&db=mdc&AN=30758491&site=ehost-live&scope=site. doi: 10.1001/jamapsychiatry.2018.4530. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Skopp Nancy A., Smolnski Derek J., Bush Nigel E., Beech Erin H., Workman Don E., Edwards-Stewart Amanda, Belsher Bradley E Caring contacts for suicide prevention: A systematic review and meta-analysis. Psychological Services 2022. 10.1037/ser0000645. [DOI] [PubMed] [Google Scholar]
- 11.Detecting and treating suicide ideation in all settings. Sentinel event alert 2016(56):1–7. https://search.proquest.com/docview/1768560921. [PubMed]
- 12.The Assessment and Management of Suicide Risk Work Group, Office of Safety and Value, VA / Office of Evidence-Based Practice, US Army Medical Command. VA/DOD clinical practice guidelines for the assessment and management of patients at risk for suicide https://www.healthquality.va.gov/guidelines/MH/srb/VADoDSuicideRiskFullCPGFinal5088919.pdf. Updated 2019. Accessed April 16, 2019.
- 13.National Action Alliance for Suicide Prevention: Transforming Health Systems Initiative Work Group. Recommended standard care for people with suicide risk: Making health care suicide safe 2018. [Google Scholar]
- 14.Reger Mark A, Lauver Mary G, Manchester C, Abraham Traci H, Landes Sara J, Garrido Melissa M, Griffin Cameron, Woods Jack A, Strombotne Kiersten L, Hughes Gregory. Development of the veterans crisis line caring letters suicide prevention intervention. Health Services Research 2022;57(Supplement 1):45–52. doi: 10.1111/1475-6773.13985. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.McCabe D, Gischlar C. Maryland department of health launches ‘MD mind health’: Texting initiative aims to reach young people, rural communities and general public. Maryland Department of Health Newsroom July 8, 2020. Available from: https://health.maryland.gov/newsroom/Pages/Maryland-Department-of-Health-launches-%E2%80%98MD-Mind-Health%E2%80%99.aspx. [Google Scholar]
- 16.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42(2):377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform 2019;95. doi: 10.1016/j.jbi.2019.103208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Milner A, Spittal MJ, Kapur N, Witt K, Pirkis J, Carter G. Mechanisms of brief contact interventions in clinical populations: A systematic review. BMC psychiatry 2016;16(1):194. 10.1186/s12888-016-0896-4. doi: 10.1186/s12888-016-0896-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Van Orden KA, Cukrowicz KC, Witte TK, Joiner TE. Thwarted belongingness and perceived burdensomeness: Construct validity and psychometric properties of the interpersonal needs questionnaire. Psychol Assess 2012;24(1):197–215. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3377972/. Accessed Jul 15, 2020. doi: 10.1037/a0025358. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Moeller RW, Seehuus M. Loneliness as a mediator for college students’ social skills and experiences of depression and anxiety. Journal of Adolescence 2019;73:1–13. 10.1016/j.adolescence.2019.03.006. doi: 10.1016/j.adolescence.2019.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cacioppo John T, Hughes Mary Elizabeth, Waite Linda J, Hawkley Louise C, Thisted Ronald A. Loneliness as a specific risk factor for depressive symptoms: Cross-sectional and longitudinal analyses. Psychol Aging 2006;21(1):140–51. [DOI] [PubMed] [Google Scholar]
- 22.Yanguas J, Pinazo-Henandis S, Tarazona-Santabalbina FJ. The complexity of loneliness. Acta biomedica : Atenei Parmensis 2018;89(2):302–314. https://www.ncbi.nlm.nih.gov/pubmed/29957768. doi: 10.23750/abm.v89i2.7404. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Loades ME, Chatburn E, Higson-Sweeney N, et al. Rapid systematic review: The impact of social isolation and loneliness on the mental health of children and adolescents in the context of COVID-19. Journal of the American Academy of Child & Adolescent Psychiatry 2020. http://www.sciencedirect.com/science/article/pii/S0890856720303373. Accessed July 8, 2020. doi: 10.1016/j.jaac.2020.05.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Wang J, Lloyd-Evans B, Giacco D, et al. Social isolation in mental health: A conceptual and methodological review. Soc Psychiatry Psychiatr Epidemiol 2017;52(12):1451–1461. 10.1007/s00127-017-1446-1. doi: 10.1007/s00127-017-1446-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, Nowinski CJ. NIH toolbox for assessment of neurological and behavioral function. Neurology 2013;80(Issue 11, Supplement 3):S2–S6. 10.1212/wnl.0b013e3182872e5f. doi: 10.1212/wnl.0b013e3182872e5f. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Posner K, Brent D, Lucas C, et al. Columbia-suicide severity rating scale (C-SSRS) baseline/screening version The Research Foundation for Mental Hygiene, Inc. 2009. https://cssrs.columbia.edu/wp-content/uploads/C-SSRS1-14-09-BaselineScreening.pdf. [Google Scholar]
- 27.Posner K, Brent D, Lucas C, et al. Columbia-suicide severity rating scale (C-SSRS) screener - since last contact – communities and healthcare The Research Foundation for Mental Hygiene, Inc. 2010. https://cssrs.columbia.edu/wp-content/uploads/C-SSRS_Pediatric-SLC_11.14.16.pdf. [Google Scholar]
- 28.The Columbia Lighthouse Project / Center for Suicide Risk Assessment. The columbia suicide severity rating scale (C-SSRS) supporting evidence. Columbia Lighthouse Project Website 2018. https://cssrs.columbia.edu/wp-content/uploads/CSSRS_Supporting-Evidence_Book_2018-10-10.pdf.
- 29.Manea L, Gilbody S, McMillan D. A diagnostic meta-analysis of the patient health questionnaire-9 (PHQ-9) algorithm scoring method as a screen for depression. Gen Hosp Psychiatry 2015;37(1):67–75. https://pubmed.ncbi.nlm.nih.gov/25439733/. Accessed Aug 11, 2020. doi: 10.1016/j.genhosppsych.2014.09.009. [DOI] [PubMed] [Google Scholar]
- 30.Arroll Bruce, Goodyear-Smith Felicity, Crengle Susan, Gunn Jane, Kerse Ngaire, Fishman Tana, Falloon Karen, Hatcher Simon. Validation of PHQ-2 and PHQ-9 to screen for major depression in the primary care population. Annals of Family Medicine 2010;8(4):348–353. https://www.clinicalkey.es/playcontent/1-s2.0-S154417091060056X. doi: 10.1370/afm.1139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Hill RM, Rey Y, Marin CE, Sharp C, Green KL, Pettit JW. Evaluating the interpersonal needs questionnaire: Comparison of the reliability, factor structure, and predictive validity across five versions. Suicide and Life-Threatening Behavior 2015;45(3):302–314. 10.1111/sltb.12129. Accessed Aug 10, 2020. doi: 10.1111/sltb.12129. [DOI] [PubMed] [Google Scholar]
- 32.Salsman JM, Butt Z, Pilkonis PA, et al. Emotion assessment using the NIH toolbox. Neurology 2013;80(Issue 11, Supplement 3):S76–S86. 10.1212/wnl.0b013e3182872e11. doi: 10.1212/wnl.0b013e3182872e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med 2006;166(10):1092–1097. Accessed Jul 15, 2020. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
- 34.Centers for Disease Control and Prevention, (CDC). 2019 standard high school state and local youth risk behavior survey. Youth Risk Behavior Surveillance System 2019. [Google Scholar]
- 35.Rubin D Multiple imputation for nonresponse in surveys New York: John Wiley & Sons; 1987. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.