Application of sentence-level text analysis: The role of emotion in an experimental learning intervention

Abstract

This registered study aimed at testing the role of emotion in the intervention effect of an experimental intervention study in academic settings. Previous analyses of the National Study of the Learning Mindset (Yeager et al., 2019) showed that in a randomized controlled trial, high school students who were given the growth mindset intervention had, on average higher GPA than did students in the control condition. Previous analyses also showed that school achievement levels moderated the intervention effect. This study applied a sentence-level text analysis strategy to detect participants’ attentional focus in five emotional dimensions (valence, arousal, dominance/control, approach-avoidant, and uncertainty) across three writing prompts students wrote during the intervention. Linear mixed models were conducted to test if emotional dimension scores computed using the text analysis predicted a higher intervention effect (i.e., higher post-intervention GPA given pre-intervention GPA). The moderating role of school achievement levels was also examined. The results of this study have implications on the possibility of applying text analysis strategies on open-ended questions in interventions or experimental studies to examine the role of the emotion-attentional focus of participants during intervention or experimental studies on the intervention or experimental outcomes, especially those that are conducted in academic settings.

Human cognitive processes are tied to emotion (Pekrun, 2006; Tyng, Amin, Saad, & Malik, 2017). Emotion was found to be related to attention and engagement (Ballenghein, Megalakaki, & Baccino, 2019; Meinhardt & Pekrun, 2003; Yiend, Barnicot, Williams, & Fox, 2018), perception (Pourtois, Schettino, & Vuilleumier, 2013), reward motivation (Padmala, Sambuco, & Pessoa, 2019), and memory (Schweizer et al., 2019). Recent evidence showed that emotion also plays a role in learning, such as learning in academic settings (Mayer, 2019), and multimedia learning (Um, Plass, Hayward, & Homer, 2012). To expand our knowledge on the role of emotion in experimental settings and learning intervention, this study examined the relationship between emotion and learning outcome in a large-scale, e-learning intervention study in educational settings. Specifically, this study uses data from the National Study of the Learning Mindset (NSLM) to explore whether the use of affectively connotative lexicon (as measured along five emotional dimensions) detected in students’ writing during the intervention study were associated with the effectiveness of a computer-based learning mindset intervention. A text analysis approach will be used to score the emotional dimensions reflected in students’ writing.

Emotion and Learning in Experimental/Intervention Studies in Academic Settings

In this study, emotion is conceptually defined as consisting of “a feeling state/process that motivates and organizes cognition and action… [and] antecedent cognitive appraisals and ongoing cognition… and may motivate approach or avoidant behavior, exercise control/regulation of responses” (Izard, 2010, p. 397). In this study, emotion is operationalized as multidimensional, including valence, arousal/activation, dominance/control (VAD), uncertainty, and approach-avoidance, all of which are commonly included in classic and large-scale cross-cultural studies on emotion and emotion semantics (e.g., Fontaine, Scherer, Roesch, & Ellsworth, 2007; Jackson et al., 2019).

According to the Control-Value Theory (CVT) of achievement emotions (Pekrun, Frenzel, Goetz, & Perry, 2007; Pekrun & Perry, 2014), the emotion experienced in a learning or academic setting (i.e. emotions tied to achievement situations or achievement emotion) may contribute to one’s learning outcomes or academic performance (Artino & Pekrun, 2014; Duffy, Lajoie, Pekrun, & Lachapelle, 2018; Peterson, Brown, & Jun, 2015). The theory posits that learners’ emotion comes from their judgment or appraisal of the value of the task and their control over the task, resulting in emotion including two main dimensions: Valence (evaluation of pleasantness, or positive vs. negative feelings), and activation/arousal (physiologically activating or deactivating emotion) (Pekrun, Goetz, Titz, & Perry, 2002; Pekrun & Perry, 2014). Generally, increased appraisal of positive value and control, and increased positive valence are associated with positive outcomes in academic settings (Goetz, Pekrun, Hall, & Haag, 2006; Pekrun, Goetz, Frenzel, & Barchfeld, 2011; Pekrun, Lichtenfeld, Marsh, Murayama, & Goetz, 2017; Pekrun & Perry, 2014), experimental studies (Heidig, Müller, & Reichelt, 2014; Münchow & Bannert, 2019; Park, Knörzer, Plass, & Brünken, 2015; Plass, Heidig, Hayward, Homer, & Um, 2014), and in intervention/training programs (Duffy et al., 2018). Results for the second dimension, arousal were mixed. While some studies found that emotional arousal directly predicted performance (e.g., Chung, Cheon, & Lee, 2015) and memory (Otani, Libkuman, Widner, & Graves, 2007) in academic settings, other studies found that emotional arousal may impair memory functioning and learning outcomes (e.g., Madan, Fujiwara, Caplan, & Sommer, 2017).

Another model of emotional dimensions, the Valence-Arousal-Dominance (VAD) model (or Pleasure-Arousal-Dominance model, PAD) aligns with the CVT. The VAD highlights that the three fundamental emotion dimensions are valence, arousal, and dominance/control (Russell & Mehrabian, 1977). According to the VAD (and consistent with CVT), arousal is the degree of alertness and engagement people experience. Positive activating emotion includes joy and surprise, and negative activating emotion includes anger, disgust, anxiety, and fear, while sadness is conceptualized as a negative deactivating emotion (Frijda, 1986; Goudbeek & Scherer, 2010; Russell & Mehrabian, 1977). Recent studies found similar mapping of basic emotion to the VAD model (e.g., Fontaine et al., 2007). Similar to the CVT, VAD sees dominance/control as an important determinant of emotion. There is only a slight difference between the two theories: While CVT places control as an antecedent of emotional valence and arousal, the VAD includes control as a part of the emotion appraisal process (consistent with the emotion definition in this study). Such feeling of control was found to map to basic emotion of joy (+), anger (+), anticipation (+), sad (−), fear (−), and anxiety (−) (Fontaine et al., 2007; Goudbeek & Scherer, 2010; Russell & Mehrabian, 1977). Research on learning and the perception of control found that control was predictive of learning outcomes (Goetz et al., 2006). Although not directly referring to control emotion, many studies examining students’ perceived academic control or how participants “felt” (Hall, 2008, p. 1134) about their control in academic settings found that such cognitive appraisal of control predicts academic success (e.g., Hall, 2008; Perry, Hladkyj, Pekrun, Clifton, & Chipperfield, 2005; Respondek, Seufert, Hamm, & Nett, 2020; Stark, Brünken, & Park, 2018). Synthesizing the CVT and the VAD, this study explored the relationship between the three emotional dimensions (V-A-D) and the intervention effect (Research Questions – RQ 1 – 3). It was hypothesized that valance (RQ1 or hypothesis 1) and dominance/control detected in students’ writing (RQ3 or hypothesis 3) positively predicted the intervention effect, moderated by school achievement levels (moderation explained further below). The relationship between arousal and intervention effect is less clear because of the mixed research findings, so the research question for arousal detected in students’ writing and the intervention effect (RQ 2) was exploratory rather than confirmatory.

Another relatively less studied emotional dimension is uncertainty or unpredictability. Although accounting for less variance than valence and arousal in dimension reduction analyses of emotion language structures (Fontaine et al., 2007; Jackson et al., 2019), uncertainty is still regarded as an important emotion appraisal process (Ellsworth, 2003; Fontaine et al., 2007). Uncertainty is defined as one’s appraisal of novelty and familiarity (Ellsworth, 2003), and is mapped to basic emotion of anticipation (+), surprise (+), fear (+), disgust (−), anger (−), and joy (−) (Smith & Ellsworth, 1985). However, it is unclear how uncertainty connects to academic experiences. The increasing feeling of uncertainty (operationalized as the lack of information in learning settings) was found to predict higher curiosity and more positive learning outcome than certainty (Campion, Martins, & Wilhelm, 2009; Lamnina & Chase, 2019; Loibl & Rummel, 2014), although uncertain emotion (unexpected and uncertain) was found to relate to higher negative affect (Lamnina & Chase, 2019). In animal research, uncertain stimuli (tones and shocks) were found to contribute to learned helplessness (Overmier & Wielkiewicz, 1983). In humans, uncertain stimuli (tones) were associated with higher anxiety and poorer behavioral adjustment and performance in gain-loss tasks (Nelson, Kessel, Jackson, & Hajcak, 2016). Uncertainty is also suggesting to be disruptive, preventing individuals to attend to cues that help them effectively cope with the negative situation (Grupe & Nitschke, 2013). Since previous findings on uncertainty and learning are mixed, this study explored (but did not conduct confirmatory testing on) the relations between uncertainty detected in students’ writing and the learning intervention effect (RQ 4).

A final emotion dimension examined in this study is approach-avoidance which is another fundamental emotion process (Elliot, Eder, & Harmon-Jones, 2013). Approach and avoidance are motivational directions (or goal orientations) that are closely linked to emotion (Elliot et al., 2013; Harmon-Jones, Harmon-Jones, & Summerell, 2017), with an approach defined as striving for and eagerly approaching a desirable state and avoidance defined as eagerly moving away from undesirable state (Scholer, Cornwell, & Higgins, 2019). Approach-avoidance is similar to emotional arousal because both may activate or deactivate behaviors, although researchers still see the two as distinctive concepts (Harmon-Jones et al., 2017). Approach-motivated emotion used to be linked only to positive emotion (approaching rewards, e.g., joy), and avoidance-motivated emotion used to be linked only to negative emotion (avoiding punishment, e.g., fear, disgust) (Cacioppo, Gardner, & Berntson, 1999; Russell & Carroll, 1999); however, studies in the past decades have suggested that approach-motivated emotion also involves negative affect, anger (Carver & Harmon-Jones, 2009; Harmon-Jones, Harmon-Jones, & Price, 2013; Marsh, Ambady, & Kleck, 2005). In terms of the link between approach-avoidance and learning or academic success, studies generally support that approach-orientation plays a positive role whereas avoidance-orientation plays a negative role (Elliot, 2020; Kirikkanat & Soyer, 2018; Lüftenegger et al., 2016). Approach (vs. avoidance) is also seen as more effective in academic settings as it encourages individuals to make plans and manage tasks (e.g., self-regulated learning) (Bartels, Magun-Jackson, & Kemp, 2009; Elliot & Moller, 2003). This study tested whether approach-avoidance detected in texts predicted the intervention effect (RQ 5). It was hypothesized that approach-avoidance detected in students’ writing (i.e. higher approach emotion) predicted a higher intervention effect.

Table 1 summarized the emotion dimensions, conceptual definitions, and RQs, and confirmatory/exploratory nature of each RQ examined in this study. This study extended previous findings on emotion and learning to explore whether emotion detected in students’ writing (by chosen lexicons and text analysis algorithm) during the intervention study was associated with the degree of intervention effect through secondary data analysis of an effective computer-based learning intervention in a national sample in the United States (for the original intervention study, see Yeager et al., 2019).

Table 1.

Emotion dimensions examined in this study.

Emotion dimensions	Conceptual Definitions	Lexicon	Research Questions (RQ1 – RQ5)
Valence	Positive and negative feelings, evaluation-pleasantness (Pekrun & Perry, 2014; Russell & Mehrabian, 1977)	hash_sentiment_jockers_rinkers (Rinker, 2018), inclusive of NRC and other lexicons	Does valence detected in texts using the lexicon predict the degree of intervention effect? (RQ1)*
Arousal	Degree of alertness and engagement (Frijda, 1986) that is often related to physiological activation (heart rate) or sympathetic arousal (Fontaine et al., 2007; Pekrun, 2006).	NRC VAD	Does arousal detected in texts using the lexicon predict the degree of intervention effect? (RQ2)
Control	Sense of power or control over the eliciting event; feelings of power or weak/submission (Osgood, May, & Miron, 1975)	NRC VAD	Does control emotion detected in texts using the lexicon predict the degree of intervention effect? (RQ3)*
Uncertainty	Uncertainty-certainty (uc-c) is defined as the appraisals of novelty and unpredictability (Ellsworth, 2003)	Generated from NRC emotion using lexicons for anticipation (uc), surprise (uc), fear (uc), disgust (c), anger (c), and joy (c) (Smith & Ellsworth, 1985)	Does uncertainty emotion detected in text using the lexicon predict the degree of intervention effect? (RQ4)
Approach-Avoidance	Motivational directions (or goal orientations); approach (ap) defined as striving for and eagerly approaching a desirable state and avoidance (av) defined as eagerly moving away from undesirable state (Elliot et al., 2013; Scholer et al., 2019)	Generated from NRC emotion using lexicons for joy (ap), anger (ap), fear (av), and disgust (av) (Cacioppo et al., 1999; Elliot et al., 2013; Marsh et al., 2005; Russell & Carroll, 1999)	Does approach-avoidant emotion detected in text using the lexicon predict the degree of intervention effect? (RQ5)*

^*Only RQ1, 3 and 5 contain confirmatory hypotheses such that higher scores in the emotional dimensions (detected by text analysis) predicted higher intervention effect. The other dimensions (RQ2 and 4) are exploratory research questions because of mixed research findings in the literature. An added research question (RQ 6) examined the confirmatory hypothesis that there will be a higher number of significant relationships between emotional dimension scores and the intervention effect in the scores from WP2 and 3 than WP1.

Automated Text Analysis of Emotion

Emotion is reflected not only through one’s non-verbal communication, but also one’s verbal communication, that is, language (for a recent review on language and emotion, see Lindquist, Gendron, Satpute, & Lindquist, 2018). The foundation of automated text analysis (particularly a word frequency approach) is the assumption that emotion, or to be more exact, the attentional focus of the writers can be determined from narratives (Bestgen, 1994; Dyer, 1983; Tausczik & Pennebaker, 2010). Attentional focus is one’s “priorities, intentions, and thoughts” (Tausczik & Pennebaker, 2010, p. 30). It reflects one’s selection of experiences (Croft & Cruse, 2004) and internal feelings, and how one processes the environmental stimuli (Tausczik & Pennebaker, 2010). The examination of attentional focus using a linguistic approach is based on the idea that “different words in a semantic frame or domain focus our attention on the different elements in the frame” (Croft & Cruse, 2004, p. 47). This approach of understanding writers’ attentional focus using text analysis is termed as “word as attention” (Boyd & Schwartz, 2021, p. 25). This word as attention approach examines word frequencies to estimate individuals’ attention to different domains (e.g., examining frequencies of emotional words to estimate writers’ attention to different emotional dimensions). Despite some drawbacks in such an approach (e.g., explorative, descriptive nature), word as attention approach allows extension of language analysis to different research domains, such as politics, personality, and in this study, intervention/educational studies (see Boyd & Schwartz, 2021 for a review of the approach). This study was particularly interested in examining students’ attentional focus to affectively-connotative concepts (measured using the lexicons tied to the five emotional dimensions defined above) and how this emotion-attentional focus correlated with the intervention effect.

One of the most common methods in “word as attention”, automated text analysis is the user-defined dictionary (UDD) method (E. E. Chen & Wojcik, 2016). In the UDD approach, dictionaries (or lexicons) are developed based on existing common dictionaries and prior research (e.g., Pennebaker, Boyd, Jordan, & Blackburn, 2015). Among all UDD methods, the most common tool used by psychological researchers is Linguistic Inquiry Word Count (LIWC; Tausczik & Pennebaker, 2010) which helps users identify the frequency of words that reflect different psychological categories. LIWC is considered a valid tool to look up psychological aspects of texts (Jones, Wojcik, Sweeting, & Silver, 2016). It is also an easy-to-use tool for psychologists as it does not require users to have any computer programing knowledge. An alternative to word-level text analysis like LIWC is a sentence-level text analysis. While methods like LIWC break down unstructured text by identifying frequencies of words, sentence-level text analysis breaks down unstructured text into sentences (by identifying punctuations). This study did not aim at replacing the word-level method with a sentence-level method or demonstrating whether sentence-level is superior to word-level analysis. This study hoped to provide an alternative to existing text analysis for research that needs to consider words before and after the target words through the use of a convenient R package.

Lexicon for Emotion Words

In UDD text analysis, the key is to select a lexicon that is appropriate to the study context. Table 1 summarized the lexicons that are used in this study. Most text analysis of emotion words focuses on valence alone; therefore, there are a number of valence lexicons available, such as AFINN (Nielsen, 2011), bing (Hu & Liu, 2004), and NRC (Mohammad & Turney, 2013). LIWC also contains a lexicon for positive and negative emotion (Pennebaker et al., 2015). These dictionaries used different methods to score the valence of words. Recently, computer scientists noticed the limitation of independent lexicons and started to develop a lexicon that combines these three common dictionaries (e.g., Jockers, 2015). This study employed one of the most recent developments, hash_sentiment_jockers_rinker lexicon (Rinker, 2018b) that combines Jockers’ (2015) wrapper for the three common lexicons, as well as the SentiWord lexicon (Baccianella, Esuli, & Sebastiani, 2010) and Rinker’s augmented version of the bing (Hu & Liu, 2004) lexicon. The final improved version of this valence lexicon consisted of 11,710 English words and phrases that are scored according to the polarity of valence from −1 (negative) to +1 (positive).

For emotion arousal and dominance/control, there are two main options. One was developed and expanded from the Affective Norms for English Words (ANEW) VAD lexicon by asking a large Amazon Turk sample to rate a subset of the words (Warriner, Kuperman, & Brysbaert, 2013). The scores of each word in the lexicon come from the mean ratings of the Amazon Turk (MTurk) sample. Therefore, researchers can extract the average ratings from their sample and use them as scores for calculating valence, arousal, and dominance. The final updated lexicon contains 14,000 words. Another (more recent) lexicon is the NRC VAD lexicon, containing not only the aforementioned valence lexicon but also a lexicon for arousal and dominance (Mohammad, 2018). The NRC VAD lexicon includes words from the NRC Emotion lexicon (valence, see above), ANEW, the updated ANEW by Warriner et al. (2013), and a few other sets, totaling 20,007 words. The scores in NRC VAD were generated in a similar way as Warriner et al. (2013); however, NRC used a wider sample of native-English speakers on a crowdsourcing platform, CrowdFlower, instead of MTurk. Also, instead of asking participants to rate each term, they gave participants a set of four randomly selected terms (4-tuple) and asked them to choose the best and the worst level of valence (V), arousal (A), and dominance per 4-tuple (D; i.e., best-worst sets/BWS). Participants were allowed to do as many or as few 4-tuples as they wanted. Using the total of 778,085 BWS obtained, V/A/D scores were computed by counting how many times a word was nominated as the best minus how many times the word was nominated as the worst and then transforming into a score between 0 (lowest V/A/D) and 1 (highest V/A/D). The NRC VAD showed good split-half reliability and generally lower correlations among the VAD dimensions (i.e. discriminant validity) than Warriner et al. (2013) lexicon. In this study, the NRC arousal and dominance lexicon were used.

In addition to these developed lexicons on arousal, one may also use existing studies in affective science and put together an arousal or dominance lexicon from basic emotion. For example, studies show that joy, anger, fear, surprise, and anxiety reflect high arousal emotions, whereas sadness and disgust are low arousal emotions (e.g., Fontaine et al., 2007; Goudbeek & Scherer, 2010). Similarly, dominance is represented by joy, anger, anticipation, whereas submissive (low dominance) is represented by sadness, fear, and anxiety (Fontaine et al., 2007; Goudbeek & Scherer, 2010). However, because there are already reliable and valid arousal and dominance lexicons available, existing validated lexicons that include more words (i.e., NRC VAD) are preferred.

Unlike VAD, other emotional dimensions have received less attention in lexicon development. Therefore, putting together a lexicon using a basic emotion approach is probably the only option (another option is to develop a new lexicon, which is beyond the scope of this study). The most popular basic emotion lexicon is the NRC emotion lexicon (Mohammad & Turney, 2010). It consists of eight emotion types, including anger, fear, anticipation, trust, surprise, sadness, joy, and disgust (Mohammad & Turney, 2010, 2013). Other lexicons, such as the aforementioned AFINN and bing only detect valence, but not discrete emotion types. For uncertainty, a review of the literature revealed that uncertainty is represented by anticipation, surprise, and fear, and certainty associated with anger, disgust, and joy (Fontaine et al., 2007; Smith & Ellsworth, 1985; Tiedens & Linton, 2001). For approach-avoidance, the emotions of joy and anger were suggested to be approach motivation, and fear and disgust were suggested to be avoidant motivation (Elliot et al., 2013; Harmon-Jones et al., 2017; Marsh et al., 2005; Russell & Carroll, 1999).

Valence shifters, Double Negative, and More

In sentence-level text analysis for emotion, it is important to identify sentence structure that may alter the meaning of an emotional word. For example, valence shifters, words that change or enhance the direction of a sentence, are key information that tells researchers the actual meaning of the text and thus need to be considered by a text analysis algorithm (Polanyi & Zaenen, 2006). For example, the word “never” in “I am never satisfied” changes the valence of the sentence from positive to negative). Other than negation (e.g., “never” and “not”), valence shifters can also be intensifier/amplifier (e.g., “really” and “very”), downtoner/de-amplifier (e.g., “barely” and “rarely”), or adversative conjunction (e.g., “but” and “however”). Therefore, in the text analysis algorithm, researchers may include a lexicon for these valence shifters (e.g., hash_valence_shifters; Rinker, 2018) in the final computation of emotional dimension scores.

Adding to the complexity of valence shifters is the issue of the location of valence shifters, the effect of punctuations, and the number of valence shifters. Regarding the location of valence shifters, some valence shifters may appear right before the word of interests (“I am not happy”), while others may precede by more than one position (e.g. “I do not feel good”, “not that I like her…”). Valence shifters may also appear after the word of interest (e.g., “I fear no man.”). In addition, punctuation may change the role of valence shifters in a sentence. For example, for the sentence “I am happy, not sad”, the valence shifters “not” should be tied to “sad”, rather than “happy” which is located before the comma. Also, a sentence may have more than one valence shifters. Specifically, double negative in a sentence means positive valence (e.g., “I did not say I did not like her”, with two negators near the targeted polarized word “like”). These issues can be categorized as the n-gram problem (Jurafsky & Martin, 2019). The “n” in n-gram refers to the number of words to be considered in each instance of text analysis. To analyze n-gram, researchers first have to tokenize the text. For example, bigram tokenization breaks down texts into two-word pairs for analysis (e.g., “I am”, “am not”, “not happy”). Then, using a valence shifter lexicon, valence shifters (negator, amplifier, de-amplifier, or adversative conjunction) can be detected and the score of the valence reflected in the same sentence can be adjusted. The score should be adjusted based on the combinations of valence shifters in a sentence, as well as the position of punctuation (Open Data Science, 2018; Raja, 2017). Finally, computer science research has suggested that English-language text analysis should consider at least trigrams (e.g., two words before and one word after the target emotional word, or three words before and no word after the target word) (Yu, Shang, Hsu, Castellanos, & Han, 2016). Therefore, an improved algorithm should be able to 1) search for valence shifters surrounding the target word of interests (e.g., at least three words surrounding the target emotional word), 2) consider more than one valence shifters (e.g., double-negative), and 3) consider the effect of punctuation in valence shifters.

Sentimentr

This study introduced an innovative text analysis that has a great potential for psychological research, sentimentr (Rinker, 2019). Sentimentr was originally developed to examine valence; however, the developer has recently added a wrapper to analyze basic emotion (via the function emotion_by, see below). With some modifications, this study further applied sentimentr in analyzing other emotion dimensions. In reviewing various recent developments in text analysis in R (syuzhet, Rsentiment, sentimentr, and SentimentAnalysis), Naldi (2019) suggested that although all packages contain a good lexicon and the flexibility to update lexicons, only sentimentr accurately considered valence shifters which is a very critical issue in text analysis (see the section above on valence shifters). Sentimentr also addresses the valence shifters problems discussed earlier by 1) examining six words surrounding the target emotional word (four before and two after), 2) considering multiple valence shifters (e.g., double negative), and 3) considering the effect of punctuation in valence shifters (i.e., using punctuation to separate the effect of valence shifters).

Although there is a steep learning curve in learning sentimentr (just as other programming-based analyses), sentimentr contains a very powerful algorithm that has already taken into account the language issues mentioned above. Therefore, users do not have to write detailed source code. Its default setting can also be easily changed to update lexicons. In addition, sentimentr uses the R-programming language, which is already popular in psychology. Therefore, researchers do not need to learn another programming language to use sentimentr.

Sentimentr has mostly been applied by computer scientists in studying big data, such as consumers’ online reviews (Kawate & Patil, 2017), emotion that YouTubers convey (Kleinberg, Mozes, & van der Vegt, 2018), and online polls (Kassraie, Modirshanechi, & Aghajan, 2017). Although sentimentr has not been used in mainstream psychology, it has been applied to many areas of research involving the study of emotion, such as public opinions in business and management (Ikoro, Sharmina, Malik, & Batista-Navarro, 2018; Sinha, Choudhury, & Agrawal, 2014), political science (Sanders, 2018) and even predicting stock trend from sentiments reflected in the market and social media (J. Chen et al., 2015; Samuel & Kretinin, 2018).

In sentimentr, the valence score is computed by first considering the polarity of words and by default, four words preceding and two words after the polarity word (i.e. words that contain valence or polar cluster c_ijl; [1.1]). Then, valence shifters, punctuations, and their positions in the cluster are considered, resulting in the sentence-level valence scores (s_ij; [1.2]).

$$c_{ijl}=\sum \left(\left(1+w_{amp}+w_{deamp}\right)\ast \left(w_{ijk}^p\right)\ast (-1{)}^{2+w_{neg}}\right)$$ [1.1]

$$valenc{e}_{s_{ij}}=\frac{\sum c_{ijl}}{\sqrt{n_{w_{i,j\dots n}}}}$$ [1.2]

where w_ijk^p is the polarity score of a polar word (i.e. words that contain valence) within the k^th word in the sentence j within paragraph i (s_ij). The average sentence-level sentiment is computed by taking into account the weighted number of the amplifiers (w_amp) and de-amplifiers (w_deamp), multiplied by the polarity scores of the words and the functions (odd vs. even) of the negators (w_neg), by the square root of the number of words in a sentence (n_w). For details of the algorithm and its source code, see https://github.com/trinker/sentimentr.

Similarly, arousal (arousal_pi) and control/dominance (control_pi) are computed by detecting arousal and control emotion words and assigning the arousal scores ($w_{ijk}^a$) and control/dominance ($w_{ijk}^c$) scores with the NRC VAD lexicon (formulas 2 and 3 respectively). The valence shifters used in the analyses of arousal and control emotion are the same as the ones used in the analysis of valence (i.e., w_amp, w_deamp, w_neg).

$$arousa{l}_{sij}=\frac{\sum \left(\left(1+w_{amp}+w_{deamp}\right)\ast \left(w_{ijk}^a\right)\ast (-1{)}^{2+w_{neg}}\right)}{\sqrt{n_{w_{i,j\dots n}}}}$$ [2]

$$contro{l}_{sij}=\frac{\sum \left(\left(1+w_{amp}+w_{deamp}\right)\ast \left(w_{ijk}^d\right)\ast (-1{)}^{2+w_{neg}}\right)}{\sqrt{n_{w_{i,j\dots n}}}}$$ [3]

For the two dimensions that employ basic emotion lexicons, uncertainty-certainty and approach-avoidant, words relating to the emotion at each end of the dimension are detected and counted (e.g., anger is detected for certainty, disgust is detected for avoidant, see Table 1). The emotion_by function in sentimentr (Rinker, 2019) allows the easy computation of the rate of occurrences of non-negated emotional words within a sentence. This approach first screens out any negated emotional words and counts only those words that do not contain negation within a cluster (i.e. in this study, 4 words preceding and 2 words following the emotional words). For example, to compute the rate of uncertainty in a sentence, emotion_by counts the number of non-negated approach words (n_uncertaintyj⁺), divided by the number of words in a sentence (n_w) such that a score of 0 represents no occurrences of non-negated emotional words (i.e. approach emotion) in a sentence, and 1 represents that all words in the sentence contain non-negated emotional words. A single score for each emotional dimension (uncertainty – certainty s_ij and approach – avoidant s_ij) is obtained by subtraction:

$$uncertainty-certainty{s}_{ij}=\frac{n_{uncertaint{y}_j}{}^{+}-n_{certaint{y}_j}{}^{+}}{n_{w_j}}$$ [4]

$$approach-avoidant{s}_{ij}=\frac{n_{aproac{h}_j}{}^{+}-n_{avoidan{t}_j}{}^{+}}{n_{w_j}}$$ [5]

Current Study and Research Questions

This study used the National Study of Learning Mindsets (NSLM) data. NSLM is the largest-ever randomized controlled trial of an online growth mindset intervention program in the U.S. in k-12 settings (Yeager et al., 2019). Using the text analysis equations above, scores for each emotional dimension were computed based on students’ writings during the learning intervention study. This secondary registered report used students’ writing in response to three writing prompts as a proxy measure for the emotional state of the mind or the emotion-attentional focus (Boyd & Schwartz, 2021; Tausczik & Pennebaker, 2010) of students during the study. In other words, using text analysis strategies, this study examined students’ scope of attention while responding to each writing prompt during three different stages of interventions. This study was particularly interested in the affectively connotative concepts (measured along five emotional dimensions using established lexicons) being attended during each writing prompt. The primary goal of this study was to investigate whether students’ emotional attentional foci are associated with the intervention effect, moderated by school achievement levels. The five specific research questions were listed in Table 1. Since in the original study, the intervention effect was found to be the largest in low and medium achieving schools rather than high achieving school (Yeager et al., 2019), the variable of school achievement level was added to the hypothesis testing as a moderator to examine whether school achievement levels moderated the relations between emotional dimensions and intervention outcome. To enhance the reproducibility of the method used in this paper, all the steps of text analysis for emotion were explained in the method section. R programming codes used in this study can be found on https://osf.io/xtdc3.

Method

All procedures and analytic methods were registered during Stage 1 of the Registered Report. A 10% random sample from the complete dataset was provided by the data gatekeeper of the NSLM dataset to validate the automated text analysis method, evaluate the feasibility of the proposed methods, and establish proof of concept. Both the pilot study and the main study (i.e., analyses of the 90% remaining sample) followed the same set of procedures and measures described in this section. All measures, manipulations, and exclusions in the study are disclosed, as well as the method of determining the final sample size (with particular reference to whether the collection was continued after data analysis).

Sample

Data Source and Original Study Design

The data used in this study comes from the NSLM dataset (doi:10.3886/ICPSR37353.v1). The testing of the effectiveness of the Mindset intervention was pre-registered and later showed that the intervention program was effective in increasing students’ GPA and growth mindset (Yeager et al., 2019). The NSLM data were collected from 9^th graders during the 2015-2016 academic year using a probability sampling strategy (i.e., stratified random sampling) stratifying school achievement levels (low, medium, high) and minority composition (i.e., percentage of Black/African-American or Hispanic/Latino/Latina students). Based on the stratification plan, 139 schools were randomly selected without replacement among approximately 12,000 public high schools in the United States. Among them, 76 schools participated in the study, but only 65 schools provided students’ records (e.g., GPA). Two schools requested to remove their data for public use; therefore, a total of 63 schools were included in the final sample. The final dataset contained 21,364 participants. Despite that some schools did not respond, Tipton generalizability index for comparing experimental samples and populations (.98) showed that the representativeness of the sample was not compromised (Tipton, 2014; Yeager et al., 2019).

During the study, participating 9^th graders were randomly assigned to either the intervention condition or the control condition through a computer algorithm provided by a third-party technology vendor. The intervention introduced ideas and examples of the growth mindset and the application of the growth mindset to the purpose of life. Two intervention sessions were administered 21 to 27 days apart. In the first session, students in the intervention condition “read an article on scientific evidence for neural plasticity [and] complete writing exercises to internalize article’s lessons]; in the second session, students in the intervention “learn more about how students/celebrities have put a growth mindset into practice [and] complete writing exercises” (Yeager et al., 2019, p. s10). The control condition was similar to the intervention condition, except that participants read ideas and examples of the brain functions. In addition to the intervention or control materials, students also filled out a survey consisting of various measures on mindset, and social and academic variables. (For detailed methodology, see Dweck & Yeager, 2019; Yeager et al., 2019, 2016).

Pilot Sample

The pilot sample was a random sample of 10% data (N = 2,269) provided by the data gatekeeper. Since only students who were in the intervention group (i.e., growth mindset intervention) were asked to write their reflection of the intervention, only students in the intervention group were included in the current study (N = 834). Missing data were excluded from the analysis. Students whose pre-intervention GPA (n = 77), post-intervention GPA (n = 19) or both pre- and post-intervention GPA (n = 173) was not available were excluded. Fifty-two students did not write any reflection (among them 33 also had missing GPA data). Generalized linear model results showed that students who did not write any reflection were not different from those who wrote reflection by pre-intervention GPA (B = −.19, S.E. = .17, t = −1.11, p = .267), post-intervention GPA (B = −.23, S.E. = .15, t = −1.50, p = .133), or sex (B = .18, S.E. = .33, t = .54, p = .590). Similarly, students missing pre-intervention GPA were not different from those without missing GPA by sex (B = .29, S.E. = .27, z = 1.09, p = .28), or post-intervention GPA (B = −.14, S.E. = .13, z = −1.08, p = .279). Students missing post-intervention GPA were not different by sex (B = −.31, S.E. = .38, z = −.82, p = .414) or pre-intervention GPA (B = −.36, S.E. = .23, z = −1.57, p = .116).

The final sample of the pilot data consisted of 550 participants from 61 schools across the United States, and a total of 58,826 words in 3,680 sentences for analysis across three writing prompts. About half of the participants were female (n_female = 234, 48.1%; n_male = 252, 51.9%). Sensitivity power analysis using G*Power 3.1.9.2 was conducted on the pilot sample using an alpha significance criterion of .05 (two-tailed), a power criterion of .90, N = 550 to test the minimum effect size for t-tests for differences between two dependent means. Results showed that the minimum effect size (Cohen’s d) is .14 or, converting to partial R used in this study (Ruscio, 2008), the minimum effect size is partial R = .07. Results of the pilot sample were included in the supplementary document (sections 3.1 - 3.5).

Main Study Sample

The same exclusion criteria for the pilot data were used in screening the main study sample. Participants were excluded if 1) they were in the control condition (i.e. did not write reflections of the intervention), and if 2) any of their written reflection, or pre-test and post-test GPAs were missing in the data. The main study sample was the remaining data, consisting of a total of 6,886 student participants in the treatment condition (who were not included in the pilot sample). Missing data excluded from the study included students whose pre-intervention GPA (n = 573), post-intervention GPA (n = 194) or both pre- and post-intervention GPA (n = 1,681) were missing. A total of 114 students did not write any reflection (among them 43 also had missing GPA data). Generalized linear model results showed that students who did not write any reflection had significantly lower pre-intervention GPA (B = −.81, S.E. = .11, z = −7.33, p < .001), and post-intervention GPA (B = −.75, S.E. = .11, z = −6.88, p < .001) than had students who wrote reflection. Male students (n = 51) were more likely than female students (n = 25) to have missing data on the writing reflection prompts (B = .72, S.E. = .25, z = 2.92, p = .004). For missing GPA, sex was not predictive of missing pre-intervention GPA (B = −.07, S.E. = .10, z = −.77, p = .444), or post-intervention GPA (B = .18, S.E. = .13, z = 1.42, p = .155). Students with missing data on pre-intervention GPA tended to have a lower post-intervention GPA (B = −.13, S.E. = .05, t = −2.65, p = .008), and vice versa, i.e., students with missing data on post-intervention GPA tended to have a lower pre-intervention GPA (B = −.58, S.E. = .07, t = −7.98, p < .001).

The final sample of the pilot data consisted of 4,372 participants from 63 schools across the United States, and a total of 481,624 words in 20,624 sentences for analysis across three writing prompts. About half of the participants were female (n_female = 1,851, 50%; n_male = 1837, 50%). Sensitivity power analysis using G*Power 3.1.9.2 was conducted for the main sample using an alpha significance criterion of .05 (two-tailed), a power criterion of .90, N = 4,372 to test the minimum effect size for t-tests for differences between two dependent means. Results showed that the minimum effect size (Cohen’s d) is .05 or converting to partial R used in this study (Ruscio, 2008), the minimum effect size was partial R = .03.

Measures

Writing Prompts

Students were asked to write their thoughts at three different time points: At the beginning of the first intervention session (before main intervention materials are presented), at the end of the first session, and during the second intervention session. The two intervention sections were 21-27 days apart (Yeager et al., 2019). The writing prompts were used to check students’ engagement with the growth mindset intervention in the original study through counting the number of words students wrote (for detail, see supplementary information of the original study Yeager et al., 2019). The original study did not examine the contents of students’ writing; however, the secondary analysis of the writing prompts to analyze students’ emotion in this study context was approved by NSLM. The detailed instructions of the three writing prompts were included in the supplementary file. In the main study data, there were a total of 481,624 words spread across WP1 (142,957), WP2 (216,827), and WP3 (121,840) words. Among them, there was a total of 10,691 unique tokens. At sentence level, there were a total of 7,602 (WP1), 13,679 (WP2), and 8,188 (WP3) sentences.

Scores of Emotional Dimensions

All writing prompts are analyzed in the same way; specifically, texts are cleaned and for each emotional dimension, a score is computed for each sentence in each writing prompt using the formula listed in the introduction. Because of the concerns of the violation of independence in text analysis (Winter, 2020), instead of averaging or summing sentence scores to generate participants’ scores, raw sentence scores will be used in computing the predictive models in linear mixed models (see data analysis plan). All scores are on a continuous scale. Scores computed by sentiment_by range from −1 to 1, and scores computed by emotion_by range from 0 to 1. To maintain consistency in the report of the emotional dimensions, all scores computed by sentiment_by were rescaled to 0 to 1 using rescale in package scales without affecting their distribution.

Learning outcome – GPA

The learning outcome used in this study was the same as the original NSLM intervention study (Yeager et al., 2019). Grade-point averages (GPAs) were the average scores of the core courses GPA (i.e. mathematics, English/language arts, science, and social studies) and were administrative data reported by the schools. The GPA was calculated by an external company, MDRC (Zhu, Garcia, Boxer, Wadhera, & Alonzo, 2019). Raw score GPA was ranged from 1.00 to 4.30 (pre-intervention GPA: M = 2.79, SD = .98; post-intervention GPA: M = 2.59, SD = 1.07). GPA was collected before the intervention and the year after the intervention. Specifically, pre-intervention GPA was the 8^th grade GPA if the intervention was conducted in semester 1 (fall), or 9^th grade semester 1 if the intervention was conducted in semester 2 (spring; i.e., most recent GPA at the time of the intervention/writing prompts). Post-intervention GPA was the 9^th grade fall and spring GPA if the intervention was conducted in the fall, or 9^th grade spring for students who took the intervention in the spring (i.e., end of the school year but after the intervention/writing prompts). To account for school variations in school grading norm, all GPA used in the analysis was centered by school means and standard deviations in the overall sample. The GPA used in this study and the calculation was the same as the intervention effect computed in Yeager et al. (2019).

School Achievement Level

School achievement level was computed by the investigators of the original NSLM dataset (see Yeager et al., 2019). Specifically, based on composite PSAT scores, state standardized tests, and AP participation and scores, each school is assigned a category of low achievement (i.e. below 25^th percentile), medium achievement (i.e. 25^th to 75^th percentile), and high achievement (i.e. above 75^th percentile). Since the NSLM intervention treatment effect was found to be moderated by school achievement level (Yeager et al., 2019), in this study, school achievement level was added as a covariate and the interaction term school achievement level x valence x survey phase is added. In the main study, 678 students (16%) coming from 12 different schools belonged to the low achievement category, while 2,509 students (57%) coming from 36 schools and 1,185 students (27%) coming from 15 different schools belonged to the middle and high achievement category respectively.

Survey Weights

Weights were provided by the NSLM data manager and were previously computed by the survey firm and were used in previous analyses of the intervention effects (Yeager et al., 2019).

Data Treatment Procedure

Text Cleaning

The first step was to prepare the text for accurate analysis. First, all texts (stored in the variable x) were converted to lower case using the sapply(x, tolower) function in base r. Then, using the R package textclean (Rinker, 2018a), common symbols were replaced with text (e.g., “@” are replaced by “at”, “%” are replaced by percent and “#” are replaced by “number”, and “1^st” to “first”). Contractions were expanded (e.g., from “don’t” to “do not”). Any remaining unconverted special characters were removed from the texts. Only punctuations were kept in the texts, as punctuation is crucial in breaking sentences for sentence-level sentiment analysis. Specifically, sentimentr considers pause locations (commas, colons, and semicolons specifically) and uses pause locations to calculate the boundary of a word cluster surrounding a polarized word. Punctuations that are used to end a sentence (period, exclamation marks, and question marks) are used to identify sentences within paragraphs. Typos and non-interpretable words were screened using the package hunspell (Ooms, 2018). The hunspell package in R (Ooms, 2018) enables researchers to input their texts (in a data frame) and screen the texts using the default dictionary. Any words that do not exist in the dictionary are stored in a list object. These included names (e.g., places, specific terms) and misspelled words/random letters (names of the participants were removed by the NSLM data manager, but any remaining names in the text were identified with hunspell). A total of 3,840 counts of non-interpretable words (typos and names) were identified out of the total 481,624 words across the three WPs (i.e., 0.7%). When counting only unique tokens, the total unique non-interpretable words were 3,218 out of a total of 10,691 unique tokens (i.e., 30%). The package mgsub (Ewing, 2019) was then used to remove names and replace any misspelled words.

Modalities

Modalities or modal verbs may change the meaning of the sentence, e.g., “it could be better” is certainly not as positive as “it is better”. Therefore, modal verbs are added to the valence shifters used in sentimentr when computing valence scores. There is no clear agreement on the treatment of modalities (Gelbukh, 2015; Liu, Yu, Liu, & Chen, 2014; Schulder, Wiegand, & Ruppenhofer, 2020). Therefore, only a limited amount of modal verbs that have empirical findings to support are added to the valence shifters based on recent literature (e.g., Jang & Kim, 2017). Specifically, “can/could/would/may/might” are added as de-amplifier, and “can/could/would/may/might not” remained as negator in the valence shifters list in sentimentr.

Data Analysis Plan

Both the pilot analysis (Stage 1) and the main study data analysis (Stage 2) followed the same data analysis plan. Results of the pilot analysis were included in the supplementary document (sections 3.1 - 3.5).

Descriptive statistics

Descriptive statistics (mean and standard deviation) for all emotional dimension scores were reported. Then, frequencies of top emotional words for each writing prompt were computed. Since approach-avoidance and certainty-uncertainty scores were generated based on basic emotion, top word frequencies for basic emotion were reported.

Analysis plan for Research Questions 1 – 5 (i.e., effects of each emotional dimension on the intervention effect)

Then, predictive modeling of each of the research questions/hypotheses listed in Table 1 was tested using separate linear mixed models estimating the fixed effect of the emotional dimension and accounting for by-subject variability:

$$y={\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_3+{\beta }_4{x}_1{x}_2+{\beta }_5{x}_2{x}_3+{\beta }_6{x}_1{x}_3+{\beta }_7{x}_1{x}_2{x}_3+{\beta }_8{x}_4+b_1{z}_1+\epsilon$$ [6]

where y is the intervention effect (GPA), β₁ and x₁ are the fixed-effect coefficient and regressor of the emotional dimension (valence, arousal, etc.), β₂ and x₂ are the fixed-effect coefficient and regressor of the survey phase (pre vs. post-intervention), β₃ and x₃ are the fixed-effect coefficient and regressor of school achievement level, b₁ and z₁ are the random-effect coefficient and regressor for individual participants, and ε is the error term. In addition, because students were prompted by the writing prompt when writing their responses, it is possible that the words mentioned in the writing prompt may have primed students to use those terms more frequently. Therefore, their emotional dimension scores (attentional focus) may be affected. To control for the effect of the writing prompt, emotional scores of students’ usage of writing prompt words were controlled (β₈x₄). Finally, survey weights (see measures) were used when computing the model. The results were computed using the package lme4 (Bates, Mächler, Bolker, & Walker, 2015) and emmeans (Lenth, 2019).

To determine whether each emotional dimension was related to the intervention effect, the coefficients of the interaction terms involving survey phase and valence and t-values were computed (i.e., mainly the interaction terms emotional dimension x survey phase and emotional dimension x survey phase x school achievement level). A 95% confidence interval was used. Effect sizes for fixed effects were calculated using semi-partial R, a recommended method for computing effect sizes for generalized linear mixed models (Jaeger, Edwards, Das, & Sen, 2017). The R package, r2glmm were used to calculate all effect sizes (Jaeger, 2017). Finally, sensitivity analyses were conducted if the coefficients of the interaction terms x₁x₂ or x₁x₂x₃ was significant. In this case, the package emmeans (Lenth, 2019) was used to see if each school's achievement levels (low, medium, and high) were significantly different in the relations between the emotional dimension and the intervention effects (i.e., Pre-post GPA).

Outlier Treatment Plan

After the linear mixed model is computed, outliers of the model were detected following the suggestions of Nieuwenhuis, te Grotenhuis, and Pelzer (2012), and tests were conducted to see if the model without outliers is significantly different from the model with outliers. Specifically, the R package is specifically designed for mixed model outlier detection, influence.ME (Nieuwenhuis et al., 2012) was used to calculate Cook’s Distance for each of the testing variables. Cook’s distance greater than 4/n, where n is the number of grouping, was considered to be influential outliers (Van der Meer, Te Grotenhuis, & Pelzer, 2010). If outliers are present, sigtest function is used to test whether the removal of outliers results in significant changes in the significance levels of the testing variables in the model (i.e. whether the absence of outliers yields a significantly different result). A cut-off t-value of -1.96 was used (Nieuwenhuis et al., 2012). If the model without outliers was found to be significantly different from the model with outliers, the model without outliers will be recalculated and the findings would be presented. In the main study sample, a conservative Cook’s distance was used such that any Cook’s distance greater than (4/7602) was considered influential outliers (i.e., n = 7,602, the smallest sample size across all three writing prompts, resulting in the highest Cook’s distance cut-off).

Results

Descriptive statistics

Descriptive statistics (mean and standard deviation) for all emotional dimension scores were reported in Table 2. Frequencies of top emotional words for each writing prompt were presented in the supplementary document. Specifically, most frequent words for valence, arousal, and dominance/control were reported separately (supplementary document section 4.1). Since approach-avoidance and certainty-uncertainty scores were generated based on basic emotion, top word frequencies for basic emotion were reported (supplementary document section 4.2).

Table 2.

Means and standard deviations of emotional dimension scores obtained through text analyses. Results are organized by sentences across the three writing prompts (WPs).

Emotional dimensions	WP1	WP2	WP3
	Sentence n = 7,602	Sentence n = 13,679	Sentence n = 8,188
By sentences	M (SD)	M (SD)	M (SD)
Valence	.46 (.09)	.40 (.08)	.52 (.10)
Arousal	.65 (.09)	.49 (.08)	.56 (.10)
Control/Dominance	.53 (.11)	.42 (.08)	.34 (.10)
Uncertainty-Certainty	.52 (.08)	.35 (.06)	.51 (.07)
Approach-Avoidance	.43 (.08)	.41 (.06)	.42 (.07)

Note: For easier understanding and comparison across variables, the variables presented in this table are scaled to a (0, 1) scale.

Linear Mixed Model Analyses for Research Questions 1-5

A summary of the results for the main study were presented as a summary in Table 3. Tables presenting detailed estimates and effect sizes for individual tests were presented in the supplementary file (Section 4.4). Again, the estimates relevant to our research questions are mainly the interaction term emotional dimension x survey (x₁x₂) phase and the three-way interaction of emotional dimension x survey phase x school achievement level (x₁x₂x₃). For Research Question 1 (valence), only WP2 showed significant effect. Particularly, sensitivity analyses showed that contrary to the hypothesis, for WP2, valence negatively predicted the intervention effect in low school achievement (b = −.48, S.E. = .15, z = −3.16, p = .002). However, consistent with the hypothesis, valence positively predicted the intervention effect in middle school achievement (b = .16, S.E. = .06, z = −2.61, p = .009). For students in high school achievement group, valence did not predict intervention effect (b = .00, S.E. = .10, z = −0.03, p = .977).

Table 3.

Summary of linear mixed model fixed effect results of the effect of emotional dimensions on students’ pre- and post-intervention GPA. Only major predictors relating to the research questions are shown here. Detailed results for each of the model can be found in the supplementary file.

	VAL		ARS		CRT		UNC		ApAv
Writing prompt 1	B	SE	B	SE	B	SE	B	SE	B	SE
Main effect of emotional dimension (x1)	−0.06	0.19	0.04	0.22	0.14	0.15	0.11	0.18	−0.25	0.22
Interaction effect of the emotional dimension and survey phase (Post-intervention) (x1x2)	0.23	0.20	0.07	0.23	−0.07	0.16	−0.18	0.19	0.33	0.23
Interaction effect of the emotional dimension and survey phase (Post-intervention) with school achievement level (x1x2x3)
School Achievement Level: Medium	−0.19	0.22	−0.06	0.25	0.08	0.17	0.28	0.21	−0.56*	0.25
School Achievement Level: High	0.26	0.25	0.09	0.28	0.23	0.20	0.20	0.25	−0.15	0.30
Writing prompt 2
Main effect of emotional dimension (x1)	0.23#	0.12	0.38	0.12	0.26*	0.13	0.30#	0.18	−0.03	0.17
Interaction effect of the emotional dimension and survey phase (Post-intervention) (x1x2)	−0.48**	0.15	−0.70***	0.14	−0.48**	0.16	−0.52*	0.21	−0.02	0.21
Interaction effect of the emotional dimension and survey phase (Post-intervention) with school achievement level (x1x2x3)
School Achievement Level: Medium	0.64***	0.16	0.68***	0.15	0.53**	0.17	0.54*	0.23	0.09	0.23
School Achievement Level: High	0.48**	0.18	0.64***	0.18	0.48*	0.19	0.74**	0.27	−0.17	0.26
Writing prompt 3
Main effect of emotional dimension (x1)	−0.18	0.16	0.10	0.17	0.30	0.19	0.16	0.20	−0.70**	0.24
Interaction effect of the emotional dimension and survey phase (Post-intervention) (x1x2)	0.38	0.17	−0.14	0.18	−0.24	0.19	−0.41	0.23	1.67***	0.27
Interaction effect of the emotional dimension and survey phase (Post-intervention) with school achievement level (x1x2x3)
School Achievement Level: Medium	−0.17	0.19	−0.02	0.20	0.25	0.21	0.28	0.25	−1.57***	0.29
School Achievement Level: High	−0.10	0.22	0.31	0.23	0.41	0.24	0.53#	0.29	−1.75***	0.33

Note: VAL = emotional valence, ARS = arousal, CRT = control, ApAv = approach-avoidant, UNC = uncertainty-certainty

^#p < .10

^*p < .05

^**p < .01

^***p < .001

For arousal (Research Question 2) and control/dominance (Research Question 3), again, significant findings were only observed in WP2. Both arousal (b = −.26, S.E. = .06, z = −4.16, p < .001) and control (b = −.14, S.E. = .07, z = −2.14, p = .033) negatively predicted the intervention effect regardless of school achievement levels. When breaking down into the three school achievement levels, arousal negatively predicted the intervention effect in low school achievement level (b = −.70, S.E. = .14, z = −4.96, p < .001), but did not predict the intervention effect in middle (b = −.02, S.E. = .06, z = −0.35, p = .724) or high school achievement levels (b = −.06, S.E. = .11, z = −0.58, p = .559). For control, contrary to the hypothesis, control negatively predicted the intervention effect regardless of school achievement levels (b = −.14, S.E. = .07, z = −2.14, p = .033). The negative significant effect was only found in low school achievement level (b = −.48, S.E. = .16, z = −3.03, p = .002), but not in middle (b = .06, S.E. = .06, z = −0.88, p = .380) or high school achievement level (b = .00, S.E. = .10, z = −0.02, p = .987).

For uncertainty-certainty (Research Question 4), significant results were observed in WP2 only. Specifically, in WP2, uncertainty negatively predicted the intervention effect only in low school achievement level (b = −.52, S.E. = .21, z = −2.43, p = .015), but not in middle (b = .02, S.E. = .09, z = −0.22, p = .823) or high school achievement levels (b = .22, S.E. = .16, z = −1.39, p = .165). It is worth noting that for uncertainty-certainty in WP3, the interaction term was marginally significant (b = 0.53, S.E. = 0.29, t = 1.82, p = .069). A sensitivity analysis was performed to explore the pattern. Results showed that similar to the results in WP2, the predictive relationship between uncertainty-certainty and intervention effect was more negative in low achievement schools (b = −0.41, S.E. = .23, z = −1.81, p = .071) than in middle (b = −0.14, S.E. = .10, z = −1.31, p = .189) or high achievement schools (b = 0.11, S.E. = .18, z = 0.64, p = .524).

For approach-avoidant (Research Question 5), significant results were observed in WP1 and WP3. Sensitivity analyses revealed that as hypothesized, approach-avoidant predicted intervention effect negatively for middle achievement school (b = −0.24, S.E. = .10, z = −2.54, p = .011) such that higher approach scores predicted lower intervention effect. However, for low (b = −.33, S.E. = .23, z = −1.45, p = .148) and high achievement school (b = .19, S.E. = .19, z = 0.99, p = .323), approach-avoidant showed a non-significant positive trend between approach-avoidant and the intervention effect. For WP3, approach-avoidant predicted positive intervention effect only significantly in low school achievement (b = 1.67, S.E. = .27, z = 6.30, p <.001) such that the higher students’ approach scores, the higher the intervention effect. However, approach-avoidant did not predict intervention effect in the middle (b = 0.10, S.E. = .12, z = 0.87, p = .397) or high achievement school (b = 0.08, S.E. = .20, z = 0.42, p = .677).

Finally, across the three writing prompts (WP), WP1 (completed before presenting any intervention materials) yielded almost no significant results. On the contrary, WP2 (completed after the first intervention session) yielded the most significant results.

Exploratory analyses

Since valence and control were found in an opposite direction than hypothesized, for exploratory purposes, interactions among valence, arousal, and control on the intervention effect were examined using the same linear mixed model methods but without considering school achievement level (to avoid doing a 4-way interaction and generating overcomplicated findings). Results showed that valence and arousal interacted to predict the intervention effect (b = −1.71, S.E. = .52, t = −3.28, p = .001). Sensitivity analysis showed that valence positively predicted the intervention outcome only when arousal is low (−1SD) (b = 0.17, S.E. = .07, z = −2.53, p = .011), and did not predict intervention outcome when arousal is high (+1SD) or at the mean level. The interaction between arousal and control were also significant (b = −2.03, S.E. = .52, t = −3.93, p < .001). Particularly, arousal played an antagonistic role, that is, when arousal is low (−1SD), control predicted intervention effects positively (b = .15, S.E. = .06, z = 2.25, p = .024), but when arousal is high (+1SD), control predicted intervention effects negatively (b = −.18, S.E. = .07, z = −2.70, p = .007).

Outliers treatment

The R package influence.ME (Nieuwenhuis et al., 2012) was used to calculate Cook’s Distance for each of the testing variables. Cook’s distance greater than (4/7602) was considered influential outliers (i.e., n = 7,602, the smallest sample size across all three writing prompts, resulting in the highest Cook’s distance cut-off). The number of outliers and significant outliers identified was presented in the supplementary material section 4.5. Significant outliers were identified using the sigtest() function in the influence.ME package (Nieuwenhuis et al., 2012). A datapoint was determined to be influential when removing it resulted in a change in the statistical significance of the tested variables in the linear mixed model (Nieuwenhuis et al., 2012). For the main study sample, despite having some outliers identified, none were significant outliers. The (lack of) impact of the outliers was further confirmed by re-running the linear mixed model with all outliers removed. The results showed similar patterns and the statistical significance of the variables tested in the model remained unchanged. The findings reported in the result session were based on the full sample with no outliers removed.

Discussion

The purpose of this registered secondary analysis was to test the role of five emotional dimensions on the intervention effect of the National Study of the Learning Mindset (NSLM) by analyzing the texts students wrote in three open-ended writing prompts (written across three time points during the study). Specifically, text analyses were used to detect students’ scope of attention or the emotion-attentional focus (Boyd & Schwartz, 2021; Tausczik & Pennebaker, 2010) while responding to each writing prompt during three different stages of interventions. Five research questions were asked, covering five different aspects of emotion. Each research question dealt with whether emotional dimension scores predicted the intervention effect (i.e., pre-post GPA change), moderated by the three school achievement levels.

Results showed that arousal, control, and uncertainty negatively predicted students’ intervention effect, but only in students coming from schools with a low achievement level. On the other hand, high scores on the approach-avoidant spectrum (i.e., high approach scores) positively predicted the intervention effect, but again, only in students coming from a low school achievement level. The results for valence were mixed. Valence positively predicted the intervention effect in middle school achievement but negatively predicted the intervention effect in low school achievement.

It was originally hypothesized that valence, control, and approach-avoidant would positively predict the intervention effect (no confirmatory hypotheses were made for arousal and uncertainty). Only the hypothesis for approach-avoidant was supported. Control, on the other hand, showed opposite results than expected. Control was defined in this study conceptually as a feeling of power and operationally as joy (+), anger (+), anticipation (+), sad (−), fear (−), and anxiety (−) (Fontaine et al., 2007; Goudbeek & Scherer, 2010; Russell & Mehrabian, 1977). It is possible that anger may have contributed to the negative predictive relationship between control and intervention effect. Further, similar to valence, joy might have played a negative role in the intervention effect for students from low achievement school level, contributing to the overall negative relationship. Another contradictory finding was valence. Valence was expected to play a positive role in students; however, this only happened in students in middle school achievement level. Students’ valence negatively predicted the intervention effect in low school achievement level. Therefore, it is possible that valence could play a very different role in academic outcomes depending on students’ backgrounds. However, exploratory analyses showed that valence may interact with arousal to predict the intervention effect. Specifically, positive valence may help students learn the intervention materials, but only when their arousal level is low. This is consistent with recent studies that emphasized the interacting role of valence and arousal (Chung et al., 2015; Schweizer et al., 2019).

As expected, WP1 (pre-intervention prompt) did not generate many significant findings. On the other hand, WP2 (after the first intervention session) generated significant findings in four out of five emotional dimensions. WP2 is also the richest data, containing the highest numbers of sentences and words. Another finding worth noting is that most of the results were found in low achievement schools, meaning that emotional focus of students in low achievement schools predicted the intervention effect the most. On the other hand, no significant findings were found in high-achieving schools at all. This may be because the intervention effect was found to be smallest in high-achieving schools (Yeager et al., 2019). Further, schools with high achievements may already have extensive resources to help students to succeed; therefore, this intervention may not have contributed much to these students’ GPA improvement (Yeager et al., 2019). High achievement schools may also have cultivated positive emotional learning environments and may have taught ideas similar to the growth mindset; therefore, students’ emotional focus at the time of the intervention (i.e., exposure to familiar materials) may not produce an effect. On the contrary, students in low achievement schools might be exposed to the ideas of the growth mindset intervention for the very first time. Thus, their emotional focus when interacting with the intervention materials might have determined whether they internalized the materials and whether the intervention contributed to their academic achievements in the following semesters. Specifically, the results from this study suggested that for students in low achievement schools, having approach emotional focus (i.e., defined as approaching a desirable state) during the learning process predicted better learning outcomes. The lack of control/dominance (i.e., lack of sense of power, feeling submission) and the lack of arousal emotional focus (i.e., low alertness/physiological arousal) during the learning process seemed to predict better (not worse) learning outcomes in students in low achievement schools.

As with any research studies, this study is limited in several ways. This study used existing data. Students’ emotional experiences were not directly measured in the original NSLM. Instead, the open-ended questions (used originally to check compliance) were used to serve as an indicator of students’ emotional experiences and they did not directly ask students to report how they were feeling. This is based on the assumption that emotion can be reflected in one’s language (which is also the foundation of emotional analysis in natural language processing) (Bestgen, 1994; Dyer, 1983). It is possible that students’ writing may jointly reflect (confound with) their level of compliance and engagement during the intervention, language skills, or academic performance. Another assumption is that how students felt during the study (i.e., at any time point when they sat in front of the computer completing the study) would be expressed as they write their responses to the questions. The benefit of this method is that it avoided the self-reporting nature of emotion measurements.

Another limitation is that text analysis in behavioral sciences is work-in-progress (Boyd & Schwartz, 2021), and naturally, there are limitations in the text analysis approach chosen in this study. Delimitation is necessary to clarify the scope of this approach. First, texts are not independent (Winter, 2020). Each sentence represents an idea, and the next sentence depends on the ideas expressed in previous sentences. For example, in WP2, students were asked to imagine a future student that was struggling. They were then asked to use the things they learned in the intervention to encourage them. Therefore, it is possible that students started with negatively valenced statements (reflecting the initial state of anxiety of the imagined 9^th grader) and then moved to positively valenced statements showing encouragement. One way to resolve potential dependency between sentences is to apply the linear mixed model approach used in this study to analyze sentences separately and then account for the varying slopes for each participant (Winter, 2020). This avoids averaging texts written by a participant and taking away the unique valence of each sentence. Despite this adjustment, the approach used in this study cannot be used to understand the change of tones from one sentence to another (i.e., examining the frequency of emotional dimensions reflected in texts, not change in emotional degree within a person). Second, the algorithm and text processing approach in this study did not differentiate the use of formal/regular expression versus idiomatic expression, slang, or metaphor. Also, only five modal verbs are considered. Some of these issues can be addressed easily in the text cleaning stage using lexicons for idioms and modal auxiliaries to code the frequency of use, or simply add these words to the emotion lexicons or valence shifters lexicons and provide them with a score for each emotional dimension. However, to implement these strategies, a clear evidence-based mechanism needs to be identified. In this study, only limited number of modal verbs are addressed, and idiomatic expression, slangs, and metaphor are not accounted for by the algorithm. Third, pronunciations in this study are only used to identify sentences and setting boundaries for a word cluster. The use of exclamation marks, question marks, or commas in hypotaxis is not considered in the present algorithm and text processing procedure.

Finally, narratives are communicative and social in nature, containing an experiencer and a target recipient. In this study, students responded to a prompt asking them to think of a time they feel challenged (WP1), write a letter to a future 9^th grader based on what they learned in the intervention (WP2), and how they can use their stronger brain to address an issue they care (WP3). Therefore, we could not equate the text as the students’ experiences of emotion in the intervention or learning processes. Instead, the scores computed from the texts merely reflect the attentional focus of the students during that hour of intervention (Tausczik & Pennebaker, 2010). There are merits and pitfalls of this “Words as Attention” approach (Boyd & Schwartz, 2021), yet the “words as attention” approach can still serve as a way for psychologists to understand the description of correlational relationships (e.g., in this study, the relationship between emotional dimensions detected in text and intervention effect). Also, publishing in-progress studies are crucial to the future development of text analysis application in psychology. This study added to existing studies by exploring how the analysis of five different emotional dimensions can be done with a single R package, sentimentr.

Despite the limitations, this study made a step forward to introduce text analysis of emotional dimensions in intervention and experimental studies in social psychology. Potential values of this study include 1) the application of an R-based text analysis method for researchers who want to include open-ended questions in their experimental design with large samples, 2) the exploration of emotional dimensions (not only valence) in text analysis, and 3) results that demonstrate the data pattern of how emotional dimensions (detected through text analysis) may be associated with an experimental study outcome.

Highlights.

• Methods and analytic plans are registered.

• Automated text analysis is used to detect emotion-attentional focus of students in a national intervention.

• Results showed that five emotional dimensions detected via text analyses predicted the intervention effect, moderated by school achievement levels.