A Social Media Based Examination of the Effects of Counseling Recommendations After Student Deaths on College Campuses

Abstract

Student deaths on college campuses, whether brought about by a suicide or an uncontrollable incident, have serious repercussions for the mental wellbeing of students. Consequently, many campus administrators implement post-crisis intervention measures to promote student-centric mental health support. Information about these measures, which we refer to as “counseling recommendations”, are often shared via electronic channels, including social media. However, the current ability to assess the effects of these recommendations on post-crisis psychological states is limited. We propose a causal analysis framework to examine the effects of these counseling recommendations after student deaths. We leverage a dataset from 174 Reddit campus communities and ~400M posts of ~350K users. Then we employ statistical modeling and natural language analysis to quantify the psychosocial shifts in behavioral, cognitive, and affective expression of grief in individuals who are “exposed” to (comment on) the counseling recommendations, compared to that in a matched control cohort. Drawing on crisis and psychology research, we find that the exposed individuals show greater grief, psycholinguistic, and social expressiveness, providing evidence of a healing response to crisis and thereby positive psychological effects of the counseling recommendations. We discuss the implications of our work in supporting post-crisis rehabilitation and intervention efforts on college campuses.

Introduction

College campuses are close-knit, largely geographically collocated communities where a crisis event can have a profound negative impact on the overall wellbeing of the campus community (Swan and Hamilton 2017). One such crisis that is frequently encountered is the death of a student. Recent statistics report that two in every 1000 U.S. college students die every year, because of accidental, suicidal, and acute and chronic illness reasons (Turner, Leno, and Keller 2013). Among these, campus suicides have almost tripled within the last fifty years, and about 18% of undergraduates and 5% of graduate students have had lifetime thoughts of attempting a suicide (Collegian 2017). These alarming statistics not only hint at the strains of campus and academic life, every such tragic incident also has widespread repercussions by affecting the general psychological wellbeing of the campus (Wrenn 1999). In fact, some of the most dangerous consequences of such crises include “copycat suicides” (when student suicides come in clusters due to social contagion effects) and mental health challenges like post-traumatic stress disorder. Given students already underutilize mental health care resources due to social stigma, lack of awareness, and the pressures of academic life (Eisenberg, Golberstein, and Gollust 2007), unanticipated crises like student deaths bring additional challenges to the mental health amelioration efforts on campuses.

Crisis events on college campuses, such as student deaths, therefore, underscore the necessity to reinforce existing intervention programs or undertake new initiatives toward reducing the psychological effects of the crisis in the student community (Blanco et al. 2008). A common approach adopted by campus administrators involves public communication and outreach, promoting information about various student-centric support, coping resources, and counseling services. Given the pervasive use of web-based technologies in the college student demography (Pew 2016), these recommendations are often shared via email and social media, also because such communication channels bear the potential to provide a common, stigma-free platform to comment and discuss about the event itself, as well as to grieve and cope. Figure 1 shows an excerpt of one such post shared by a campus administration on Reddit. In this paper, we refer to such posts as “counseling recommendations.”

Figure 1:

An excerpt of a counseling recommendation post shared on r/gatech following the death of a Georgia Tech student.

However, significant methodological gaps exist in measuring the effectiveness of these post-crisis interventional recommendations shared by campus officials (Schwartz and Whitaker 1990). These range from a reliance on retrospective self-reports, to the difficulty in causally determining the link between exposure to these recommendations and the psychological states of students following a crisis (DeStefano, Mellott, and Petersen 2001).

Our Work.

We address the above gaps in examining the efficacy of counseling recommendations following a crisis, in the specific context of student death incidents on college campuses, targeting two innovations. First, we use unobtrusively gathered social media data of college Reddit communities, where these recommendations are shared by campus officials. Social media helps us track individuals who engage with these recommendations and what effects they have on their psychological states. Then, as a second innovation, we develop a causal analysis framework that statistically models the shifts in psychological states characterizing individuals who are exposed to these recommendations, and those in a control group. As indicators of these changes, drawing from natural language analysis (word embeddings), the crisis literature, and psychological theories like the “grief work hypothesis” (Schut 1999), we develop the following categories of measures: a) affective changes, specifically around the expression of grief (we model a new “grief lexicon”), b) behavioral changes, and c) cognitive changes.

Focusing on a dataset of ~400M posts and ~350K users spanning 174 college communities on Reddit, our findings show that, compared to baseline scenarios, in the aftermath of student death incidents, individuals who are actively exposed to the recommendation (via commentary) tend to show statistically significant shifts in their psychosocial attributes compared to a matched control cohort who do not engage with the recommendations in the same manner. Examining these changes, we find that the exposed group demonstrates greater expressivity of grief, shows signals of social integration and diversity in interactions, and exhibits improved cognitive processing as well as linguistic and stylistic complexity. We situate our findings in the crisis and mental health literatures that associate such shifts with a healing response, which in turn are indicative of benefits to one’s psychological state. Our work thus provides the first large-scale, (social media) data-driven study of the effects of post-crisis counseling interventions. We conclude by discussing the implications of our work in supporting improved mental health policy decisions with social media, following a devastating crisis in a community.

Privacy and Ethics.

Given the sensitive nature of this work, despite working with public de-identified data, we are committed to securing the privacy of individuals and the campuses included in our dataset. We do not report any information that can identify a specific person or a student death.

Related Work

Crisis and Mental Health Interventions.

According to the social amplification theory of risk, crises affect the psychological, social, institutional, and cultural normalcy of life among the exposed individuals and their close ones (Kasperson et al. 1988). Crisis intervention teams regularly undertake assignments to tackle and prevent mental health problems in the aftermath of crises (Reijneveld et al. 2003). For example, grief being a natural response to intense sadness and distress that ensues many crises, particularly, the death of someone close, working with the framework of “grief work hypothesis” (Schut 1999), psychologists often recommend trauma and bereavement intervention therapies to overcome the emotional upheaval of loss (Cable 1996; Saltzman et al. 2001). However, several studies in psychology have argued about the effectiveness of such outreaching interventions. Some observed that routine referral to counseling resources following loss lowered anxiety, supported coping and regaining self-esteem, and enabled the individuals relate better and look to the future (Currier, Neimeyer, and Berman 2008). In contrast, prior research also found that the majority of bereaved people are resilient enough to adapt without the need of counselors and therapists, questioning whether the intervention measures at all have beneficial effects post-crisis (Bonanno 2004; Jordan and Neimeyer 2003).

In addition to this apparent dichotomy regarding the effects of post-crisis interventions, commonly adopted methods, like surveys on mental health service utilization further suffer from limitations. They do not capture the short-term dynamics and context of the situation–critical during a crisis, are prone to retrospective recall bias (Tourangeau, Rips, and Rasinski 2000), and suffer from compliance, implementation, and scalability issues (Scollon et al. 2009). Specifically after a student death, employing a psychological assessment survey that asks delicate questions is difficult due to the sensitivity of the situation (De Choudhury et al. 2014).

A sound study design examining the effects of post-crisis counseling interventions should include the possibility to differentiate natural change due to coping and resilience from changes attributable to the interventions themselves (Schut and Stroebe 2010). Further, to establish whether an intervention has benefits for an individual’s psychological state, requires a comparison between an intervention and a non-intervention control group. However, so far, such studies have been severely limited due to the challenge in collecting adequate pre- and post-intervention data. Our work addresses these gaps by: 1) appropriating a naturalistic source of data before and after student death crises—social media; and 2) using causal inference techniques, that can infer the effects of exposure to counseling recommendations that are shared after such crises on college campuses.

Social Media, Crisis, and Mental Health.

Several studies have demonstrated that analyzing language can help us understand psychological states relating to an individual’s mental health (Pennebaker and Chung 2007). In recent years, linguistic patterns observed on social media have been examined in the context of inferring and eventually improving wellbeing (De Choudhury et al. 2013). Complementarily, the crisis literature has also found promising evidence of supporting the potential of web and social media language in better understanding the impacts of natural and man-made disasters (Mark et al. 2012; De Choudhury, Monroy-Hernandez, and Mark 2014).

Contextually related to our problem, Brubaker et al. (2012) and Glasgow et al. (2014) analyzed social media data to understand community grieving following personal and societal tragic events. Specific to college communities, Saha and De Choudhury recently examined the evolution of stress following a gun violence incident on campus (Saha and De Choudhury 2017). This rich body of work motivates our choice of social media as a data source and a “passive sensor” to examine the psychosocial changes that ensue student deaths on college campuses, and to what extent students are affected by exposure to outreaching intervention means like counseling recommendations.

These studies have, however, largely employed correlational techniques, and although they are very insightful, causal approaches are critical to tease out specific influences on one’s psychological state that are attributable to a treatment of interest, in our case, it being counseling recommendations. Recently, researchers have drawn on the causal literature to study the impacts of social support and online community participations in helping weight loss (Cunha, Weber, and Pappa 2017) and reducing suicidal risk (De Choudhury and Kiciman 2017). Our adoption of a causal inference framework to assess the psychological effects of counseling recommendations advances these investigations in a new, unexplored context.

Data

For our study, we use Reddit as our data source. Reddit (reddit.com) is a social news aggregation and discussion website consisting of diverse communities, known as “subreddits”, which offer demographical, topical, or interest-specific discussion boards. Subreddits dedicated to colleges are widely prevalent and provide a common portal for students on the same campus to discuss and share about a variety of issues related to their personal, social, and academic life. Bagroy et al. (2017) demonstrated that campus subreddit data well represents the campus population for over 100 U.S. colleges and may be utilized as a reliable source of data for inferring mental wellbeing.

To collect data, with the help of the websites “US News” (usnews.com which lists the top U.S. universities) and “SnoopSnoo” (snoopsnoo.com which groups subreddits into several categories, one of which is “Universities and Colleges”), we first compiled a list of 174 college subreddits. The largest subreddits on this list, based on subscriber count, include r/UIUC, r/berkeley, r/gatech, and r/UTAustin, which had 13K-19K subscribers as of January 2018.

Counseling Recommendations (CR) Dataset.

Next, starting with a seed list of generic and campus-specific keywords, we first used an iterative snowballing technique to build a list of search queries to identify counseling recommendation posts in our 174 subreddits: 1) Generic Keywords are related to death and counseling, such as “death”, “suicide”, “counsel*”, “rip”, “therapy”. This list also includes phrases related to email, and positions of responsibility, like “email”, “email dean”, “president”, 2) Campus-specific Keywords are specific to a campus, which we compiled by consulting the official college websites to obtain names of the campus administrators (e.g., president or dean) and the counseling body. Using these keywords, we queried Reddit’s search interface for counseling recommendation posts, and manually inspected the returned posts for correctness in terms of our definition of counseling recommendations. This gave us 88 counseling recommendation posts across 46 subreddits, which we denote as the CR dataset.

Baseline Datasets.

Additionally, for our research goal—quantifying the psychosocial changes attributable to the counseling recommendations following student death events instead of other hidden factors (e.g., changes associated with active participation in any content shared by campus officials, exposure to content around non-crisis events, or general interest in counseling related content), we consider three other baseline datasets (ref. Table 1).

Table 1:

Datasets on student death (SD) and counseling recommendation (C).

	SD	¬SD
C	CR	B2
¬C	B3	B1

Baseline Dataset B₁ includes announcements from campus officials unassociated with a crisis (student death) event and without any pointers to counseling or support resources. E.g., B₁ contains posts about non-crisis/non-critical campus events, and appointments or resignations of officials.

Baseline Dataset B₂ consists of campus announcements unassociated with a student death but points to counseling services. E.g., it includes counseling recommendations that are either routine, or about socio-political issues and policies (e.g., immigration).

Baseline Dataset B₃ includes posts that are campus announcements acknowledging a student death but without pointers to counseling information.

We acquired these datasets employing similar technique as in the case of CR posts—identifying keywords iteratively (e.g., “sexual”, “violence”, “immigration”, “policy”, or “student affairs”), querying and manually inspecting the correctness of returned posts. Eventually, B₁ had 229 posts, B₂ had 30 posts, and B₃ had 1 post across the 46 subreddits in which at least one CR post was present.

Next, using nested queries on the cloud platform, Google BigQuery which hosts an entire archive of Reddit data (Bagroy et al. 2017), we obtained the usernames of those users who commented on the CR, B₁, B₂, and B₃ posts. We also collected these users’ historical archives (or”timelines”) with all posts. Our paper uses “posts” as one term for posts and comments, unless specified otherwise. Additionally, we collected similar data of 358,871 other users (378,381,052 timeline posts), who posted on the campus subreddits, outside of the CR, B₁, B₂, and B₃ posts. As a measure to restrict our corpus among those individuals who belong to the same campus per subreddit, we further pruned our dataset of any users who posted on more than one campus subreddit. Finally, we identified 842 users and 3,167,266 timeline posts for the CR dataset, 2,215 users and 6,818,873 timeline posts for the B₁ dataset, 321 users and 1,231,784 timeline posts for B₂, and no users in B₃.

Matching

Our ultimate goal is to quantify to what extent a counseling recommendation shared on Reddit following a student death incident impacts the psychological states of individuals who are exposed to it. Answering this question necessitates testing for causality in order to eliminate any confounds associated with the observed effects (that is, psychosocial changes of individuals) following a post-crisis intervention (that is, a counseling recommendation shared by campus official after a student death incident). Causal analysis is also important because the observed effects could simply be a result of the passing of time, or of people’s ability to heal and cope with the crisis and gain resilience, and therefore may have little to do with the counseling recommendations. Therefore, the crux of our approach is to tease apart the effects that are attributable to the counseling recommendations instead of other psychosocial changes that follow crisis events.

Ideally, such problems are tackled using Randomized Controlled Trials (RCTs). However, given our data is observational and an RCT is impractical and unethical in our specific context involving crises (student deaths) and psychological states of individuals, we adopt a causal analysis framework based on statistical matching, which “simulates” an RCT by controlling for as many observed covariates as possible (Imbens and Rubin 2015). In our case, given the scale of our large dataset (~400M posts from ~350K users) and the high dimensionality of the covariates along which we intend to match the users, we adopt a two-tier approach that optimizes for computational efficiency. This includes: 1) Propensity score matching, conditioned on offline and online behaviors of users, and 2) Mahalanobis distance computation, measured on the linguistic attributes of user posts. Both of these matching techniques are widely adopted in the causal inference literature (Rosenbaum and Rubin 1985).

Defining Treatment and Control Groups

Any causal inference framework involves first defining a “treatment”, and then constructing cohorts which would constitute “treatment” and “control” groups. In our problem, treatment constitutes exposure to a counseling recommendation. We operationalize it as commenting on a Reddit post that is a counseling recommendation. We note that while commentary is a limited way to identify CR exposure and lurkers may also be considered exposed, it is a high precision method (that is, the commenting individuals were definitely exposed to the counseling recommendation) and is readily measurable from our data. We adopt this definition of treatment for all posts in our CR and baseline datasets (B₁, B₂, and B₃).

Next, causal analysis literature (Rosenbaum and Rubin 1985) recommends that effects can be appropriately inferred on “treated” users only when we do not observe comparable results for another randomized group of “control” users under similar setting. Accordingly, for each of the datasets, we categorize two groups of users based on the above defined treatment – 1) Treatment group who were commenters in their respective CR, B₁, B₂, or B₃ posts, and were active on Reddit before and after it, 2) Control group as a subset of all other users in the same subreddit, where each member is a statistical match of one from the Treatment group.

Statistical Matching Approach

Our matching strategy controls for a variety of covariates such that the effects (psychosocial changes) are examined between two groups of users showing similar overall offline and online behavioral and linguistic patterns. 1) First, assuming that our user pool consists primarily of college students as shown in prior work (Saha and De Choudhury 2017), we control for users within the same campus subreddit. This mostly accounts for any offline behavioral changes attributable to regional, seasonal, academic calendar, or other local factors. For online behavioral patterns, we include as covariates the number of comments and posts, ‘karma’, and tenure on the platform—similar covariates were used in recent work (Chandrasekharan et al. 2017). 2) Second, controlling for the linguistic attributes, we use the 50 categories given in the Linguistic Inquiry and Word Count (LIWC) lexicon as covariates in our matching model. These categories span across affective, cognitive, lexical, stylistic, and social attributes (Chung and Pennebaker 2007). Next, for each dataset CR, B₁, B₂, B₃, our two-tier matching framework proceeds as follows:

1) In the first step, with the offline (subreddit participation) and online behavioral covariates introduced above, we trained logistic regression classifiers estimating the propensity to receive a treatment, called propensity scores (p). For every Treatment (Tr_i) user and their exposure date, we matched on users commenting on the same subreddit with at least one post before and after that exposure date. Next, we obtained the top k (k = 3) most similar users per (Tr_i) user, conditioning to a maximum caliper distance (c) (with α = 0.2), i.e., ∣ Tr_i(p)–¬Tr_i(p) ∣≤ c, where c = α*σ_pooled (σ_pooled is the pooled standard deviation, and α ≤ 0.2 is recommended for “tight matching”). 2) In the second step of matching, per Tr_i user, we identified the most similar user (Ct_i) among the top k users, based on the 50 LIWC lexical categories as covariates and adopting the Mahalanobis distance metric (Rosenbaum and Rubin 1985). Finally, we obtained 821 matched pairs in the CR and 1,754 and 295 in the B₁ and B₂ datasets respectively. Note that, since B₃ had no user, we did not include it in our approach and analyses.

Assessing Balance between Groups

In order to ensure that our matching techniques eliminated any imbalance of covariates, we used effect size (Cohen’s d) metric to quantify the differences in the Treatment and the Control groups across each of the covariates. This was performed for the CR dataset as well as the baseline datasets B₁ and B₂. Lower values of Cohen’s d imply better similarity between the groups, and a value lower than 0.2 indicates “small” differences between the groups (Cohen 1992). Overall, we found that the two-tier matching approach significantly improves covariate imbalance by over 35%, 9%, and 61% after the addition of the lexical covariates in the three datasets CR, B₁, and B₂ respectively (see Figure 2). This justifies the choice of our matching approach that optimizes for computational efficiency, at the same time controls for behavioral and linguistic differences across the Treatment and Control groups in the CR, B₁, B₂ datasets.

Figure 2:

Cohen’s d for balance between Treatment and Control groups. Absolute values are mean-aggregated per type.

Validating Temporal Confounds.

We also assessed the likelihood of temporal differences in activities of our matched cohorts. For example, it could be possible that one group posts at a higher frequency than the other, which would distort the time-aggregated analysis of effects (i.e., psychosocial changes) we observe across them after the student death events. For this purpose, we compared the z-scores of number of words shared by Treatment and Control individuals Before and After the CR (or B₁, B₂) posts. Quantifying the standardized variation around the mean value of a distribution, z-scores, that do not rely on absolute values, estimate the relative changes in a time series. Using paired two-tail (t-tests, we find that the daily z-scores for our Treatment and Control groups in any of the CR, B₁, B₂ datasets show no statistically significant differences (p > 0.05, see Table 2), revealing that temporal confounds are unlikely in our ensuing analysis.

Table 2:

Mean z-scores of the number of words posted Before and After exposure for the Treatment and Control Groups.

Group→	Treatment		Control		t-test
Dataset↓	Before	After	Before	After	t	p
Main (CR)	0.13	0.02	−0.16	0.21	0.97
Baseline (B1)	−0.41	0.38	−0.57	0.43	1.07
Baseline (B2)	−0.05	0.02	−0.14	−0.04	0.52

Measuring Efficacy

Now, we present the measures via which we quantify the psychological effects of counseling recommendations. Our measures are based on the three core psychosocial constructs elucidated in the psychology literature: a) Affective, b) Behavioral, and c) Cognitive attributes (Breckler 1984). Inspired from the widely adopted “difference in difference” technique in the causal-inference research (Abadie 2005), we estimate the effects of counseling recommendations in terms of the changes corresponding to all our psychosocial measures in the Treatment and Control groups Before and After the date of a specific CR, B₁, or B₂ post.

Affective Changes

Researchers have demonstrated affective variability in individuals following crisis events (Mark et al. 2012). Our work models affect from the perspective of “grief”. Grief is a “response” and a mix of conflicting feelings and a wide range of strong emotions (James and Friedman 2009). When someone dies, alongside bringing shock, disbelief, and numbness, it leaves friends and relatives feeling lost, anxious, depressed, or physically unwell. Grief is the process by which we adjust to the death of someone close (Saltzman et al. 2001; Wrenn 1999). A rich body of literature in psychology, by way of the “grief work hypothesis” (Schut 1999) therefore has identified the coping and healing benefits of grieving (Cable 1996), which in turn are associated with achieving timely resilience and return to normalcy and day-to-day activities following crises. Thus examining grief as a measure of psychological change following CR exposure is extremely relevant in our setting.

While prior work has developed methods to identify affective attributes like mood, emotion, and sentiment (De Choudhury, Counts, and Gamon 2012; Saha et al. 2017), currently, there are no computational means to infer grief from language. Moreover, due to the complexity of grief as an affective construct (note the definition above), gathering high quality ground truth is challenging. Furthermore, in assessing psychosocial changes among individuals particularly in response to an environmental stimulus (such as crisis), psychology literature and theories advocate a grounded representation of affect, comprising of not only the commonly used valence (pleasantness dimension), but also the intensity of affect, known as activation. To address these challenges, and to obtain a theoretically valid assessment of grief around the sharing of counseling recommendations, we employ a novel open vocabulary approach of 1) building a grief lexicon; and 2) mapping the words in the grief lexicon to two affective dimensions, valence and activation, drawing on the established Russell circumplex model of affect (Posner, Russell, and Peterson 2005).

Building a Grief Lexicon.

To build a grief lexicon, we adopted an open-vocabulary based transfer learning approach. Transfer learning approaches have been employed recently in social media studies of health, wherein the dataset under question did not contain labeled data on a target variable of interest (Saha and De Choudhury 2017). In our approach, we leveraged data from 15 subreddits around the topic of grief, such as r/grief, r/GriefSupport, or r/bereavement, where people engage in sharing their sorrow and grieve about the loss of their loved ones. From these subreddits, we obtained over 50K posts (D_G), based on the archives available on Google’s Big Query. Additionally, we obtained a generic Reddit corpus, D_R of posts unrelated to any grief or mental health issues, also used in prior work (Bagroy, Kumaraguru, and De Choudhury 2017).

Thereafter, we extracted all n-grams (n = 2) from the above two datasets D_G and D_R, along with their tf-idf scores. Then, we used Log Likelihood Ratio (LLR) measures to obtain a ranked list of most distinguishing n-grams across the two corpuses. LLR for an n-gram is determined by calculating the logarithm (base 2) of the ratio of its two probabilities, following add-1 smoothing. Based on the LLR measures, when an n-gram is comparably frequent in both the datasets, its LLR is close to 0; it is < 0, when the n-gram is more frequent in D_G, and > 0 for the opposite. Among the 4,714 n-grams exhibiting negative LLR, we obtained a list of those 50% of n-grams with the most negative values—we used median as the measure of central tendency here. These 2,357 n-grams with a big negative skew in LLR are most distinctive of D_G, and we refer to them as the “Grief Lexicon”, L_G. Table 3 reports a sample of the top 30 of these n-grams ranked on their tf-idf scores.

Table 3:

Top 30 n-grams used discriminatingly in reddit grief communities. These n-grams were obtained by ranking their Log Likelihood Ratio (LLR) measures with generic non-mental health communities (− 1 ≤ LLR < 0), tf-idf values are scaled at 10⁻².

Word	tf-idf	Word	tf-idf	Word	tf-idf
thank		loved		help
sorry		husband		memories
loss		support		feelings
remove		passed		easier
hope		hugs		miss
lost		beautiful		son
grief		sharing		peace
death		glad		cancer
died		suicide		comfort
pain		heart		sucks

Modeling the Affective Dimensions of Grief.

Next, to characterize the valence and activation dimensions of words in the above grief lexicon based on the circumplex model, we employed the widely used word embedding technique to derive latent semantic relatedness between words (Mikolov et al. 2013) and the Affective Norms for English Words (ANEW) lexicon (Nielsen 2011). ANEW is an affect dictionary, curated after extensive and rigorous psychometric studies, containing a list of over 1,000 affect categories and their quantified measures of valence and activation. Prior research has successfully used ANEW to understand expression of mood and affect (De Choudhury, Counts, and Gamon 2012).

For every affect category in ANEW, we obtained its vector representation in a 300 dimensional word-embedding space using the word2vec model (pre-trained on Google News dataset of ~100B words). Within the word-vector space, semantic similarity between any two words can be estimated with cosine similarity, using which we mapped all the n-grams in our grief lexicon (L_G) to the most similar ANEW category (if any, threshold = 0.69 (Rekabsaz, Lupu, and Hanbury 2017)) and obtained their valence and activation values. Accordingly, 2,357 n-grams from our grief lexicon were mapped to 459 unique ANEW categories. With their valence and activation values as coordinates on an x-y frame and tf-idf as the magnitudes, we modeled our grief lexicon in the two-dimensional circumplex space of affect (see Figure 3). We find that expressions across a range of valence and activation values occur frequently in grief, e.g., “kind”, “inspire”, “love”, “anger”, “sad”, “afraid”, and so on. This aligns with the definition of grief (James and Friedman 2009), and justifies our lexically induced open-data strategy of modeling grief in the circumplex model of affect.

Figure 3:

Weighted distribution of affect categories (ANEW) for grief lexicon on Russel’s circumplex model. Top ANEW categories and their standardized tf-idf ([0, 1]) are labeled.

Characterizing Treatment & Control with Grief.

With the above grief lexicon and its 2-dimensional affective model, we quantify the affective expression of grief in the Treatment and Control groups around the date of the CR, B₁, or B₂ posts in their respective datasets. Specifically, within each of these groups, we obtain all the n-grams and their tf-idf values before and after the date of post. Applying the same word-vector based similarity metric described above, we map these n-grams to the most similar grief word and its valence and activation value. Then, we compute the mean percentage change of valence and activation of grief in our Treatment and Control groups in our datasets.

Behavioral Changes

Next, we measure psychosocial changes in behavior around the date of counseling recommendation posts. In the psychology, mental health, and crisis literatures (De Choudhury, Monroy-Hernandez, and Mark 2014), many behaviors including changes in social functioning and shift of interests can be indicative of an individual’s changing psychological trajectory. We are interested in observing the following changes as effects of exposure to counseling recommendations: Does the user become more active on Reddit, indicating improved extroversion? Do they participate in more subreddits, indicating a diversity of interests and interactions? Do they involve themselves in more discussion threads on Reddit, indicating social engagement? Inspired from prior work (Wise, Hamman, and Thorson 2006), we answer these questions with three metrics, a) activity, or frequency of posting, b) interaction diversity, that is, number of unique subreddits they participate in, and c) interactivity, given by computing the number of comments to post ratio.

Cognitive Changes

Literature in psychology identifies cognitive attributes as another indicator of an individual’s psychological state (Bandura 1993) —an uptick in wellbeing is known to be associated with reduced cognitive impairment and improved perceptual processing. Further, psycholinguistics literature has revealed the association of linguistic structural and stylistic patterns in written communication with cognition (Pennebaker and Chung 2007). Borrowing from prior work (Ernala et al. 2017), we adopt the following techniques to examine cognitive changes through linguistic syntax, structure, and stylistic vocabulary usage:

Coleman-Liau Index (CLI)

is a measure of linguistic structure and provides a readability assessment test based on character and word structure within a sentence (Pitler and Nenkova 2008). This measure approximates a U.S. grade level required to understand the content, and can be calculated with the formula: CLI = 0.0588L – 0.296S – 15.8, where L is the average number of letters per 100 words and S equals the average number of sentences per 100 words.

Complexity and Repeatability

are syntactic measures that indicate an individual’s cognitive state in the form of planning, execution, and memory, and are in turn, linked to psychological states (Ernala et al. 2017). We quantify complexity as the average length of words per sentence, and repeatability as the normalized occurrence of non-unique words.

LIWC.

Linguistic Inquiry and Word Count (LIWC) is a well-validated lexicon that groups words into psycholinguistic categories (Pennebaker and Chung 2007). We specifically focus on the normalized occurrences of Cognition & Perception, Linguistic Style, and Social Context categories.

Results

We present our results starting with an overview comparing the differences between the changes in Before and After samples per dataset, CR, B₁, and B₂. To evaluate statistical significance of these differences, we conducted Welch’s t-test, and adjusted the p-values using False Discovery Rate (FDR) correction. Table 5 gives a summary. We find that for most of the measures, the Treatment and Control groups in B₁ and B₂ show no statistically significant differences in the Before and After periods, but that all other measures barring one (Activity) show significant differences in the Treatment and Control groups in the CR dataset. This dataset also shows revealing changes in magnitude for the Treatment group, for example – a) for affect, grief expression significantly increases, b) for behavior, increased social engagement, interactiveness, and diversity of interests, and c) for cognition, improved cognitive and linguistic processing.

Table 5:

Comparing the mean percentage difference between Before and After periods in the Treatment (Tr) and Control (Ct) groups. Bar lengths represent relative and numbers denote absolute magnitudes. Blank entries convey no statistical significance.

Data→	CR		B1		B2
Measure↓	Δ Tr	ΔCt	ΔTr	ΔCt	ΔTr	ΔCt
Affective Changes
Grief: Activation			–	–	–	–
Grief: Valence			–	–	–	–
Behavioral Changes
Activity	–	–	–	–	–	–
Interaction Diversity					–	–
Interactivity			–	–	–	–
Cognitive Changes
Readability
Complexity
Repeatability
Linguistic Style			–	–	–	–
Cognition & Perception			–	–	–	–
Social Context			–	–	–	–

Several studies in psychology and the crisis literature have associated greater expressivity whether in terms of the positivity or intensity of emotionality, bereavement and grief expression, or language with an improvement in their psychological wellbeing status (Klein and Boals 2001). Situating our results within these studies, we observe that compared to baseline scenarios, counseling recommendations following student deaths are succeeded by effects indicative of improved wellbeing. In the following subsections we discuss in further detail our results on the CR dataset.

Affective Changes

First, we examine the affective changes that characterize the Treatment group’s exposure to counseling recommendation. Employing the circumplex representation of grief words, we find that grief expressions increase considerably (15% for valence, 9% for activation) in Treatment as compared to a marginal (−1%) decrease in Control (t = 2.68, p < 0.05). Figure 4 plots these changes from the Before to the After period on the same circumplex model, where larger circles indicate greater differences for those corresponding grief expression. A closer look at Figure 4(a) reveals that higher differences are more prominent in the cases where a specific grief expression increased in the After period. These expressions which show significant changes, belong to all four quadrants in the circumplex model, such as “friend”, “hope”, “sad”, and “lost”. In contrast, although drawn on the same scale, large circles are scarce in Figure 4(b), suggesting minimal changes in grief expression in the Control group. This observation affirms that individuals exposed to counseling recommendations in the CR dataset become more expressive from an affective perspective, and this affective expression illustrates grieving as a positive psychological response to the crisis (Pennebaker and Chung 2007).

Figure 4:

Differences in grief words (from the proposed grief lexicon) in the Treatment and Control groups, plotted on Russel’s circumplex model of affect. The radius of the circles are proportional to the mean differences in occurrences of the grief words between the Before and After periods around the date of CR post.

Behavioral Changes

We find that counseling recommendations are associated with no significant differences in terms of a user’s posting frequency (activity). An alternative interpretation of this finding backs our causal analysis that, despite all users continuing usual social media activity before and after the exposure to the CR post, the outcome varies for the Treatment and Control groups for “every” other measure.

Next, Figure 5 shows the behavioral changes in users around the date of sharing of the CR posts. For interaction diversity, that is, the measure of a user’s engagement across multiple communities, we find similar changes in the Treatment and Control group, the former being marginally higher by 1% (t = 4.0, p < 0.05). However for interactivity, a major increase by 29% occurs in the Treatment cohort, as compared to a small −1% change in Control (t = 4.1, p < 0.05). These measures support positive social functioning effects of CR posts, in turn known to have coping benefits following loss of someone close (Pennebaker and Chung 2007).

Figure 5:

Distribution of differences of interactivity (comments to posts ratio) and interaction diversity (unique subreddits).

Cognitive Changes

Readability.

Within the Treatment group in the CR dataset, we find a mean increase of 14% in the Coleman-Liau Index (CLI) following exposure to the counseling recommendations. Although this number is close to the changes in Control group (11%), we observe statistically significant differences (t = −81, p < 0.05) between the two groups. Since both groups of users were statistically matched on their overall linguistic usage, and are alike in their educational qualification (college students), a comparable overall increase in readability is unsurprising, especially because this measure typically increases with writing over the years (Pitler and Nenkova 2008). To illustrate this observation further, we obtained the probability density function (with Gaussian kernel) of CLI in the Before and After periods of exposure to CR posts, for the Treatment and Control cohorts (Figure 6). This figure shows that the distribution of the CLI measure changed considerably for the Treatment group, and no such effect is observable in the Control group. Specifically, the variance of distribution in Treatment cohort reduced substantially by 90% (σ decreased from 6.1 to 1.9) after CR post exposure. Increased readability of written speech is known to indicate better control over the train of thought, better coherence in expressing ideas, and better discourse organization (Thorndyke 1977). That such increases manifest in the Treatment group after exposure to CR posts further indicate psychological effects around improved wellbeing.

Figure 6:

Distribution of Readability (CLI) in the Treatment (left) and Control groups (right), Before and After the CR post.

Repeatability and Complexity.

Figure 7 shows the After and Before differences in linguistic repeatability and complexity in the Treatment and Control groups following exposure to CR posts. For repeatability, the figure reveals that a greater fraction of Treatment users show negative and nearzero changes (Mdn_Treatment = −2 vs. Mdn_Control = 8), that is, their linguistic repeatability decreases. In addition to statistically significant differences (t = 11.3, p < 0.05), we find that while repeatability decreases by 3% for Treatment users, it increases by 9% for Control users. For complexity, Treatment users demonstrate over 80% increase compared to the Control users (1.3% vs. 0.7%). Although numerically the change is small, statistical significance tests (t = 18.6, p < 0.05) show compared to a linguistically matched Control population, the Treatment users show a greater increase in the usage of longer words. Mental health challenges can manifest in the form of poverty of speech, are accompanied by a reduction in syntactic complexity, and an impairment in syntactic comprehension (Ernala et al. 2017). Such tendencies typically result from an overall cognitive deficit, difficulty concentrating, distraction, or a preference for expressing simpler ideas. As repeatability and complexity capture such syntactic attributes in Reddit posts, reduction in repeatability and increase in complexity following CR post exposure are, therefore, indicative of positive psychological changes in the Treatment cohort.

Figure 7:

Distribution of differences in repeatability and complexity in Treatment and Control groups.

Cognition & Perception, Linguistic Style, Social Context.

Finally, analyzing the normalized occurrences of LIWC categories for linguistic style, cognition, and social context, we observe interesting patterns. Figure 8 shows the variability (95% confidence interval) of differences for statistically significant LIWC categories. We find that for all of the categories, the Treatment dataset shows significantly higher variability than the Control. As all of these plots lie on the positive y-axis, we further infer that levels of cognitive measures increased following exposure to the CR posts.

Figure 8:

Differences in cognitive measures between the Treatment and Control groups following CR exposure, based on usage of LIWC categories. The vertical lines represents 95% confidence interval range, and the dot shows the mean. Statistical significance is reported based on Welch t-test. p-values are adjusted using FDR correction (*p < 0.05, * * 0.001 < p < 0.01, * * *p < 0.001).

We find that cognitive measures, such as “causation”, “cognitive mechanics” and “tentativeness” significantly increase after the exposure to CR posts. Per prior work, this indicates an improvement in an individual’s cognitive functioning (Pennebaker and Chung 2007). Additionally, greater usage of “negation”, and words relating to “feel” and “percept” indicate greater perceptual expressiveness, known to be associated with first-hand accounts of real world happenings, events, and experiences (Brubaker et al. 2012).

Likewise, within linguistic style measures, we find revealing changes, such as pronouns (1^st, 2^nd, and 3^rd) and temporal attributes increase considerably (mean difference=~5) in the Treatment dataset. Both psycholinguistics and crisis literature note that 1^st person and past tense usage relate with narrating personal or collective experiences of upheavals, which seems likely in our case (Mark et al. 2012). Prior work also notes higher usage of 2^nd person pronouns in the aftermath of crises and 3^rd person pronoun use is associated with the language of adaptive and coping related health benefits following crises. Further, the increased usage of lexical density features such as “adverbs”, “articles”, and “quantifiers” indicate that Treatment users express via more complex narratives (Chung and Pennebaker 2007)—a signal of better psychological health (Ernala et al. 2017). Among the social context measures, treated users use more “family” and “friends” words. Based on prior work, this is a known behavior for individuals coping with grief and trauma, and reference to socialization has therapeutic benefits for an individual’s psychological state (Seeman 1996).

Discussion and Conclusion

Summary.

We demonstrate that, with a novel causal analysis framework and unobtrusively gathered social media data, it is possible to quantify, to what extent exposure to counseling recommendations following a student death on a college campus positively impacts an individual’s psychological state. Our work, therefore, bears the potential to complement existing techniques of assessing the effectiveness of intervention measures deployed after crises. In this way, we advance the growing body of research in social media and health, opening up new avenues of addressing health challenges by employing social media as a mechanism of supportive mental health and crisis intervention delivery.

Using a Reddit dataset of 174 campus communities and ~400M posts from ~350K users, we observe statistically significant psychosocial (affective, behavioral, cognitive) effects of exposure to counseling recommendations on the treated population as compared to a statistically matched control cohort. In assessing these psychosocial effects, our causal inference framework allowed us to account for behavioral and linguistic covariates across the treatment and control groups, also eliminating confounds due to temporal variability in their Reddit activity. Further, by comparing against baseline scenarios, our approach reveals that the observed effects were characteristic of the specific context of student death related crises, instead of other latent factors.

A contribution of our work is a “grief lexicon” and a transfer learning based methodology to build it. Drawing on recent advances in computational linguistics research, we expanded a validated affect dictionary with word embeddings and employed it on public social media data. Our technique can be used in other social media and health research that involves extracting domain-specific information, but where ground truth data is limited and unlabeled data is plenty.

Implications.

Our findings provide support for the “grief work hypothesis” (Schut 1999), that situates grief counseling and therapy as a way for working through loss. In our treatment group, following student death incidents, we find evidence of greater affective expressivity of grief, greater desire for social connectedness and diversity in interactions, improved cognitive and perceptual processing, and emergent linguistic and stylistic complexity. Based on psychology and crisis literatures around the healing and coping benefits of grieving (James and Friedman 2009), our results indicate that exposure to counseling recommendations on social media after crisis events, signals effects associated with positive benefits for one’s psychological state.

We believe our findings are not only useful in helping gauge whether sharing counseling recommendations on social media are at all effective, but also can support crisis rehabilitation efforts on college campuses. Campus officials can utilize the outcomes of our work as a way to identify individuals who are not benefiting from these counseling recommendations. This can help them employ other proactive intervention measures to support their mental health. Broadly, our work can inform campus policy decisions around mental health outreach. Our work also sheds light into the role of communication technologies like social media, in supporting these efforts, both during crises as well as to tackle college student mental health challenges.

Limitations And Future Work.

While our findings are indicative of the positive benefits of exposure to counseling recommendations, we cannot make broad claims about the efficacy of these recommendations in improving the mental wellbeing of the entire college campus. Our findings are limited to only those individuals who chose to explicitly engage with the CR posts via Reddit commentary. Thus our observations suffer from a self-selection bias. It is possible that students were exposed to these information via alternative means (e.g., word-of-mouth) and that some availed counseling services independent of exposure to such post-crisis outreach. Since these information are not observable to us, our results should be interpreted with caution. Multi-prong data gathering approaches used in prior crisis informatics work (De Choudhury et al. 2014) are a potential solution.

Our results do not indicate whether the individuals who engaged with the counseling recommendations actually availed counseling services. We cannot be certain if the positive psychosocial shifts we see are a consequence of some form of therapy or other measures they adopted to cope with the impacts of the events. Nevertheless, our causal analysis does indicate positive effects on psychological wellbeing in the treatment cohort compared to a control. This suggests that irrespective of the mechanisms of counseling or support adopted, exposure to counseling recommendations on social media, largely yields positive psychological outcomes.

Another limitation of the work is the lack of data on a true control that encompasses student death incidents in campuses without any shared counseling recommendation. While creation of such a true baseline is ethically questionable (debarring some students from help resources while some others benefit from it), future work can investigate other means to create an appropriate control through partnerships with student health services on a campus.

Finally, in the individuals exposed to the counseling recommendations, it is promising to see signs of healing and coping, which in turn indicate that they might be returning to normalcy and achieving resilience in the aftermath of the student death incidents. However, in the absence of ground truth clinical assessments, we cannot claim that these psychosocial shifts imply clinically meaningful changes in the mental health of the exposed individuals. Future work can augment our analyses with self-reported or counseling service utilization data to assess the post-crisis clinical efficacy of the counseling recommendations in college campuses.