Abstract
The reliability of online mood manipulations is potentially undermined by participants’ noncompliance behavior, e.g., skipping a part of the experiment or switching between web pages during the mood manipulation. The goal of the current research is to investigate (1) whether and how mood manipulations are threatened by noncompliance behavior, (2) whether it is confounded with the induced mood state as predicted by Affect Regulation Theory, and (3) what measures can be taken to control for the noncompliance. In two online-experiments, noncompliance behavior was assessed during the mood manipulation with movie clips by tracking interruptions of watching and page switches. The results support the affect regulation hypothesis demonstrating that people confronted with negative emotional content interrupted watching the video and switched between pages more often than people with positive content. Methodologically, this causes a threat to the internal validity of internet-based mood manipulation studies. To decrease the risk of noncompliance, the current study recommends to block skipping a part of the mood manipulation, detect page focus events and measure the time people stay on a page.
Keyword: Psychology
1. Introduction
The development of the Internet has opened a new way to conduct psychological experiments. An online study enables fast data collection, gives access to diverse population groups, extends the sample size and saves laboratory resources. However, it also introduces new challenges such as the lack of supervision of participants and control over the presentation of stimuli (Gureckis et al., 2016; Reips, 2002). The validity of a web study may be undermined by noncompliance behavior, e.g., skipping a part of the experiment or switching between web pages during the mood manipulation. As more researchers use online tools to elicit emotions, it is critical to develop techniques for detecting and preventing cases of noncompliance. The current research suggests such a method and assesses how large the effect of noncompliance during a mood induction procedure is and whether it depends on the emotional state.
The present experimental study employs short videos as a mood induction method, which is one of the most efficient and widely used means to induce an emotion both in a laboratory (Westermann et al., 1996) and in an online experiment (Ferrer et al., 2015; Gilman et al., 2017). Previous research on online mood manipulation by movies hardly addressed the problem of the noncompliance behavior. One simple assessment is to ask people whether they closed their eyes or looked away during the video (Rottenberg et al., 2007). In another online study, participants’ engagement was assessed by open-ended accuracy questions about film content (Gilman et al., 2017). However, participants can guess the right option or write a vague answer, so the questionnaire does not ensure the reliability of the mood induction. Moreover, people who recognize the movie that they watched before can skip the mood manipulation, but answer correctly on the accuracy question.
The attempts to measure noncompliance online were carried out using other mood induction methods such as texts and images. Göritz (2007) categorized the noncompliance as “invalid datasets” that comprised participants who interrupted the study, did not answer some questions or answered incorrectly, or took suspiciously more or less time than it was required to process the texts or images. Although the time variable was used to detect the noncompliance, it is difficult to justify a specific discriminatory benchmark since people may have different speeds of text reading. However, the use of movie clips for mood induction provides a more participant-independent time threshold, which is the length of the clip itself.
The noncompliance with a mood induction procedure is related to the problems of dropout and failure to follow instructions (Galesic, 2006; Reips, 2002). The dropouts, or participants who interrupt the experiment, are usually identified by missing data and excluded from analysis (Crump et al., 2013). The correct understanding of instructions can be verified by a catch task where a participant is asked to provide a specific response that is different from the default one, e.g. to click a small circle at the bottom of the screen instead of answering questions on the page (Oppenheimer et al., 2009). On the other hand, participants who do not comply with a mood induction procedure can read and understand instructions but skip a part of the study and continue until the end, so that their data seem to be complete. It is therefore possible that the filtering of the sample by missing data or correct responses in the catch task may not eliminate the cases of noncompliance. Increasing the sample size as a strategy to enhance the statistical power also does not help alleviate the problem as it increases the probability of noncompliance behavior and does not reduce the risk of confounding the noncompliance with an experimental condition (Arechar et al., 2018).
The findings of research on quality control in online experiments can be used to improve the methodology of mood manipulation. In particular, multitasking can be a problem if a participant does not pay enough attention to the task and switches to another task. Studies on multitasking have shown that switching between tasks can take place at intervals of several minutes (Judd and Kennedy, 2011; Yeykelis et al., 2014). In addition, emotional factors play a role in the task-switching behavior: Yeykelis et al. (2014) discovered that people experienced an increase in arousal prior to switching to different content. Gould, Cox, and Brumby (2016) have shown that switching negatively affects performance in a crowdsourcing task, but interventions such as pop-up messages can reduce the frequency of task switching. However, further research is required to find out how noncompliance depends on the emotional content of mood manipulation.
1.1. Residence time as a measure of the noncompliance behavior
We propose to use the duration of a person's stay on a web page with the movie clip as a measure of the noncompliance. The opportunity to interrupt the video and proceed to the next part of the experiment depends on the type of the survey software and the design of the study. The implementation of sophisticated algorithms that show a "continue" button only when the video is ended is possible but requires programming skills. The default option of many survey services (e.g., LimeSurvey, Unipark) is to embed a video into the web page together with a continue button. Therefore, one of the purposes of the current experimental study is to analyze whether people will use the option to proceed without watching the movie clip until the end.
Even if the continue button can be hidden until the video is over, there are no algorithms that prevent a user from switching to another web page, which might decrease the mood manipulation effect (if people do not watch the video) or even reverse it (if people encounter another mood eliciting content). However, the browser can detect the moment of page defocusing and record the time people spend out of the page focus; e.g., the PageFocus algorithm for identifying cheating in online tests (Diedenhofen and Musch, 2017).
In summary, the noncompliance behavior measured by the residence time can be affected by an early quitting or by temporary leave of a page. Both forms represent avoidance behavior, as participants withdraw from the part of the experiment that induces emotions.
1.2. The role of emotions in noncompliance behavior
The emotional content of a movie clip might activate emotion regulation responses (Gross, 1998). From the perspective of affect regulation theories, people in a negative mood engage in activities because of their mood-lifting consequences, but people in a positive mood avoid situations with mood-threatening consequences (Andrade, 2005, but for a critique see Wegener and Petty, 1994). Therefore, we expect people to try to evade a negative video to improve their mood, but stay engaged with a positive video to maintain their mood. In terms of experimental outcomes, people in a negative mood condition should therefore show a higher level of noncompliance than in a positive mood condition.
The noncompliance also mediates the relationship between the induction procedure and changes of mood. Since skipping a part of negative mood induction video occurs because people want to avoid feeling negative emotions, the noncompliance behavior should attenuate the reported negative emotions at the end of the experiment in the negative mood group. However, in the positive mood group, the reason to avoid the mood manipulation would be different. People might perceive the present environment as mood threatening (e.g., boring or annoying) and escape from the mood manipulation situation to protect their good mood. Therefore, the reported positive emotions at the end should not be attenuated by the noncompliance behavior.
To summarize, the purpose of the current research is to investigate (1) whether and how mood manipulations are threatened by noncompliance behavior, (2) whether it is confounded with the to-be-induced mood state, and (3) what measures can be taken to control for the noncompliance. To achieve these goals, we conducted two online-experiments where noncompliance behavior was measured during the mood manipulation with movie clips.
2. Methods
2.1. Experiment 1
2.1.1. Participants
We recorded 472 cases of participation in the experiment (mood group: 257 positive (54%), 215 negative (46%)). One hundred and twenty-six records were filtered out based on page reloads, repeated participation, leaving the experiment before the end, and reported problems with the video (see Table 1). The exclusion rate was not different between mood conditions, χ2 (1, N = 472) = 0.08, p = .77. The final sample comprised 346 participants (see Table 2).
Table 1.
Number of excluded participants in Experiment 1.
| Negative | Positive | Overall | |
|---|---|---|---|
| Beginning N | 215 | 257 | 472 |
| Page reloads | 17 | 17 | 34 |
| Repeated participation | 2 | 2 | 4 |
| Did not answer the intention question | 0 | 0 | 0 |
| Left at the first mood question | 2 | 8 | 10 |
| Left during the movie | 18 | 23 | 41 |
| Left at the second mood question | 2 | 5 | 7 |
| Left at the emotions questionnaire | 13 | 8 | 21 |
| Left at the debriefing questions | 1 | 4 | 5 |
| Left on the final page with emails | 1 | 1 | 2 |
| Problems with video | 0 | 2 | 2 |
| Final N | 159 | 187 | 346 |
Table 2.
Sample characteristics.
| Mood condition | Sample size | Mean age (SD) | Age range | Females | German | English |
|---|---|---|---|---|---|---|
| Negative | 159 | 25.87 (8.70) | 17–73 | 120 | 98 | 61 |
| Positive | 187 | 26.12 (10.54) | 17–74 | 144 | 111 | 76 |
| All | 346 | 26.01 (9.72) | 17–74 | 264 | 209 | 137 |
2.2. Materials
Mood induction. Positive mood was induced by amusement emotions since amusement is often correlated with happiness but different from other emotions (Gilman et al., 2017). We used the scene with Scrat chasing nuts from the movie “Ice Age: Continental Drift” (2012, 2:30 min). Negative mood was elicited by showing the scene with the death of Mufasa from the movie “Lion King” (1994, 3:10 min) (Rottenberg et al., 2007). We chose sad emotions to induce negative mood since sadness has a decreased level of arousal (Bonanno et al., 2008) and is not associated directly with avoidance motivation as other high-arousal negative states of anxiety, fear or disgust. Both movie clips were validated in our previous study (Shevchenko and Bröder, 2018). The video was played in the frame of 720 × 576 pixels in the center of the screen, the control panel with information about the video length was hidden, and the right mouse button was disabled. The button to start or pause the video was located at the top of the screen.
Mood manipulation check. Before and after watching the video, a mood manipulation check was done by asking participants about their current mood using a general question (“How would you characterize your present emotional state?”) on a visual analog scale from “very depressed” to “very elated” recording answers from 0 to 500. The numerical values were not displayed to participants, but were scaled into a range from 0 to 100 for further analysis. The change of mood score was calculated by subtracting the mood score before the video from the mood score after the video. Additionally, after the second general mood question following the video, participants rated their current state on ten different emotion scales of the Positive and Negative Affective Schedule (PANAS, Watson and Clark, 1999). We did not present this questionnaire before watching the video not to elicit mood regulation thoughts or demand effects. The visual analog scale of the PANAS had five labels that were displayed at the top of the screen to help participants mark their responses: “very slightly or not at all”, “a little”, “moderately”, “quite a bit”, and “very much”, and the answers were internally recorded in the range from 0 to 500. Later, the answers were transformed into the range between 1 and 5 and averaged for the scales of positive (happy, joyful, delighted, and interested) and negative emotions (downhearted, gloomy, sad, distressed, and angry). One additional item was used to control for arousal (relaxed).
Behavioral measures of noncompliance. Two events were monitored during the mood manipulation: the interruption of the procedure with the continue button and temporary leaving of the web page. For both measures, the browser recorded timestamps of events such as opening the page, starting the video, leaving the web page (“out of focus” event), returning to the web page (“in the focus” event), and clicking the continue button. The interruption was detected if the continue button was pressed before the end of the video clip and the residence time on the web page was shorter than the length of the video. The case of temporary leaving was registered if there was at least one out-of-focus event indicating that a participant had performed an activity (e.g., a mouse click) outside the web page.
2.2.1. Design
The experiment was conducted online on the website of the first author1. The participants were recruited from the subject pool of students at the University of Mannheim. The link to the study was also published online in the list of Internet experiments2. The participation was encouraged by a lottery in which five randomly selected participants won a 20 Euro voucher for an online shop. The mood was manipulated in a between-subjects design, so one group of participants watched a sad movie clip, the other one watched a funny one. Two forms of noncompliance behavior that affected the residence time were measured: the interruption of the video before its end by clicking the continue button and temporary leaving of the page. The mood manipulation check was done by comparing the mood before and after watching the movie clip. The study was approved by the Mannheim University Ethics Committee.
2.2.2. Procedure
Participants read the general information about the study and the consent form. They were told that they had to complete the study and answer all questions in order to take part in the lottery with a chance to win a 20 Euro voucher. After filling in demographic data, participants rated their current emotional state. On the next page with the video, they were instructed to click the play button to start the clip which would play for around 3 min. Participants were asked to watch the video until the end, and then press the continue button at the bottom of the page. After the video, the general mood rating question and a more precise questionnaire with ten different emotions followed. After that, participants had to answer whether they watched the video before, watched the video until the end and were distracted by other activities while watching the video. If they were distracted, they were asked to describe what they were doing. On the last page, participants answered where they took part in the study (at home, at work, at university, in transport, or other) and who was with them (they were alone, with strangers, friends, relatives, or other). They also could leave an email address for the lottery and write their comments about the study.
2.3. Results
One hundred and three participants (30%) did not comply with the instructions of the mood manipulation. They temporarily left the page (n = 49, 14%), interrupted the video (n = 47, 14%), or temporarily left the page and then interrupted the video (n = 7, 2%).
2.3.1. Interruption of the mood induction procedure
The number of video interruptions was higher in the negative mood condition (n = 33, 21%) than in the positive mood condition (n = 21, 11%), χ2 (1, N = 346) = 5.92, p = .015, φ = 0.13 (see Table 3).
Table 3.
Interruptions in the mood groups. Row percentages are written in parentheses.
| Interruption | No interruption | Total | |
|---|---|---|---|
| Negative | 33 (21%) | 126 (79%) | 159 |
| Positive | 21 (11%) | 166 (89%) | 187 |
| Total | 54 | 292 | 346 |
A 2 × 2 between-subjects ANOVA was applied to analyze the effect of the mood manipulation and interruption of the video on the change in mood. The dependent variable was the difference between the mood score before and after the video, which was measured by a general question about the current emotional state. Since 10 participants did not answer the question, the analysis was done for 336 participants. The raw values were scaled to a range from 0 to 100 to facilitate the reading of the results, so that the difference score varied between -100 and 100. A negative value represented a decrease in mood, and a positive value reflected an improvement in mood (see Fig. 1). The analysis showed both main effects of the mood manipulation, F (1, 332) = 138.84, MSE = 43720, p < .001, η2 = 0.29, and the interruption, F (1, 332) = 6.06, MSE = 2355, p = .014, η2 = 0.013. Importantly, there was a significant interaction between the manipulation and the interruption, F (1, 332) = 7.48, MSE = 2355, p = .007, η2 = 0.015 (see Table 4). A post-hoc Tukey test showed that in the negative mood condition people who did not interrupt the video had a greater decline in mood than people who interrupted it, p < .01. In the positive mood condition, there were no significant differences between people who interrupted and who watched the whole video, p = .95.
Fig. 1.
The changes in mood dependent on the mood condition and the interruption of the video (N = 336). The group means and 95% CI are shown. A negative score indicates that participants declined in mood, a positive score shows that they improved their mood.
Table 4.
The mood changes for the groups who interrupted and did not interrupt the video in the negative and positive mood manipulation conditions. The asterisk (*) displays significant differences between the groups (p < 0.01).
| Negative mood manipulation |
Positive mood manipulation |
All participants |
||||
|---|---|---|---|---|---|---|
| N | Mean (SD) | N | Mean (SD) | N | Mean (SD) | |
| No interruption | 122 | -20.82* (19.90) | 164 | 4.96 (14.73) | 286 | -6.04 (21.34) |
| Interruption | 31 | -7.82* (25.39) | 19 | 2.67 (10.23) | 50 | -3.83 (21.44) |
| All participants | 153 | -18.18 (21.68) | 183 | 4.72 (14.33) | 336 | -5.71 (21.34) |
We conducted a 2 × 2 between-subjects ANOVA analysis with the emotions measured by the PANAS after the video3. For negative emotions, the main effect of mood manipulation was significant, F (1, 327) = 89.87, MSE = 70.15, p < .001, η2 = 0.21, the main effect of the interruption was not significant, F (1, 327) = 0.03, MSE = 0.03, p = .86, η2 < 0.001, but the interaction effect was significant, F (1, 327) = 4.14, MSE = 3.23, p = .043, η2 = 0.01. For positive emotions, the main effect of mood manipulation was significant, F (1, 327) = 94.23, MSE = 66.43, p < .001, η2 = 0.22, but the main effect of the interruption, F (1, 327) = 2.83, MSE = 1.99, p = .09, η2 = 0.007, and the interaction effect were not significant, F (1, 327) = 0.89, MSE = 0.62, p = .35, η2 = 0.002. For arousal, the main effect of mood manipulation was significant, F (1, 327) = 24.94, MSE = 29.12, p < .001, η2 = 0.07, but the main effect of the interruption, F (1, 327) = 0.21, MSE = 0.24, p = .65, η2 = 0.001, and the interaction effect were not significant, F (1, 327) = 2.31, MSE = 2.70, p = .13, η2 = 0.007 (see Table 5 for descriptive statistics).
Table 5.
The PANAS scores after the video in different groups (Experiment 1).
| Negative mood manipulation |
Positive mood manipulation |
All participants |
|||||
|---|---|---|---|---|---|---|---|
| N | Mean (SD) | N | Mean (SD) | N | Mean (SD) | ||
| Negative scale | No interruption | 122 | 2.86 (0.87) | 160 | 1.85 (0.78) | 282 | 2.29 (0.96) |
| Interruption | 30 | 2.65 (1.28) | 19 | 2.21 (1.01) | 49 | 2.48 (1.19) | |
| All participants | 152 | 2.82 (0.97) | 179 | 1.89 (0.81) | 331 | 2.32 (1.00) | |
| Positive scale | No interruption | 122 | 2.24 (0.71) | 160 | 3.20 (0.88) | 282 | 2.79 (0.94) |
| Interruption | 30 | 2.57 (1.05) | 19 | 3.27 (0.92) | 49 | 2.84 (1.05) | |
| All participants | 152 | 2.31 (0.79) | 179 | 3.21 (0.88) | 331 | 2.79 (0.95) | |
| Arousal | No interruption | 122 | 3.28 (1.04) | 160 | 2.60 (1.09) | 282 | 2.90 (1.12) |
| Interruption | 30 | 2.99 (1.19) | 19 | 2.83 (1.06) | 49 | 2.92 (1.13) | |
| All participants | 152 | 3.22 (1.07) | 179 | 2.63 (1.09) | 331 | 2.90 (1.12) | |
Since the length of the movie clips was different (3 min 10 s in the negative and 2 min 30 s in the positive condition), the potential concern may be that people in the negative mood condition skipped the video more often because it was longer. Therefore, we analyzed the interruption time to find out whether the length of the movie clips played a role in the noncompliance behavior. The majority of participants (n = 51, 95%) interrupted the video earlier than 2 min 31 s, i.e. before the end of the shorter video clip. We repeated the analysis without people who interrupted the video later (n = 3) and replicated the original results – the participants in the negative mood condition (n = 30, 19%) more often interrupted the video than in the positive mood condition (n = 21, 11%), χ2 (2, N = 343) = 4.30, p < .05, φ = 0.11. Additionally, we used a t-test to compare the interruption time between the mood manipulation conditions. Since the interruption time had a right skewed distribution (Range = 3–189 s, Mdn = 24 s, M = 50.31 s, SD = 52.85 s), we log-transformed the values. There were no differences in the interruption time between the positive (Range = 3–149 s, Mdn = 23 s, M = 48.95 s, SD = 50.60 s) and the negative group (Range = 3–189 s, Mdn = 25 s, M = 51.18 s, SD = 54.99 s), t (52) = 0.34, p = .73, d = 0.096.
2.3.2. Temporary leave
Out of 346 participants, 56 (16%) temporarily left the web page with the video at least once. There were no differences between the number of people who did it in the positive (n = 31) and in the negative mood condition (n = 25), χ2 (1, N = 346) = 0.05, p = .83, φ = 0.01.
Regarding the changes in mood measured before and after the video, a 2 × 2 ANOVA with main effects of the mood condition and temporary leave showed that only the effect of the mood condition was significant, F (1, 332) = 134.54, MSE = 43720, p < .001, η2 = 0.29; Mneg = -18.18, SDneg = 21.68 and Mpos = 4.72, SDpos = 14.33. Neither the effect of the leave nor the interaction between the mood condition and the leave were significant, F (1, 332) = 0.41, MSE = 135, p = .52, η2 = 0.001, and F (1, 332) = 2.44, MSE = 794, p = .12, η2 = 0.005.
The leaving time had a right skewed distribution (Range = 0.71–958.40 s, M = 69.44 s, Mdn = 11.61 s, SD = 142.56 s), therefore we log-transformed the values to compare the leaving time between the mood manipulation conditions. There were no differences in the leaving time between the positive (Range = 1.40–958.40 s, Mdn = 13.51 s, M = 57.11 s, SD = 172.92 s) and the negative group (Range = 0.71–405.41 s, Mdn = 10.57 s, M = 79.38 s, SD = 94.41 s), t (54) = -0.59, p = .56, d = -0.16.
2.3.3. Self-report
We report on the exploratory analysis of the self-report data, as these data can provide additional insights into what can distract people during the experiment and what can lead to noncompliance. In the group of participants who interrupted the video, the percent of people who watched the video before (n = 35, 66%) was higher than in the group who did not interrupt the video (n = 143, 49%), χ2 (1, N = 345) = 5.23, p = .022, φ = 0.124. Prior viewing could be a potential confounding variable in the experiment, since more participants have seen the movie of the negative (n = 127, 80%) than the positive mood manipulation condition (n = 51, 27%), χ2 (1, N = 345) = 96.71, p < .001, φ = 0.53. To test for confounding, we performed a logistic regression where the dependent variable was the video interruption, and independent variables were mood manipulation condition, prior viewing and their interaction. The effect of the mood manipulation remained significant, b = -1.20, p = 0.024, the effect of the prior viewing was not significant, b = -0.17, p = 0.72, and their interaction was not significant, b = 1.19, p = .077. In the group of people who had already watched the movie, people interrupted the video equally likely in the negative (n = 25, 20%) and in the positive mood manipulation condition (n = 10, 20%), χ2 (1, N = 178) = 0.00001, p = .99, φ = 0.0009. However, in the group of people who had not watched the movie before, people in in the negative mood condition (n = 7, 23%) interrupted the video more often than in the positive condition (n = 11, 8 %), χ2 (1, N = 167) = 5.51, p = .018, φ = 0.18. Additionally, the prior viewing was not related to the change in mood, as the main effect of the prior viewing, F (1, 331) = 2.13, MSE = 692.68, p = .15, η2 = 0.005, and its interaction with the mood manipulation did not significantly affect changes in mood, F (1, 331) = 0.88, MSE = 285.94, p = .34, η2 = 0.002.
Regarding the question whether participants watched the video until the end, 284 people (98%), who did not interrupt the video, answered positively, whereas in the group of people who interrupted the video, 42 participants (79%) answered positively (see Table 6). Additionally, there was no statistically significant differences in the report of being distracted during the video between the group who interrupted the video (n = 8, 15%) and the group who did not interrupt the video (n = 22, 8%), p = .074. Among the reasons to be distracted, the most popular answers were related to other people (n = 9), using phone (n = 5), and food-related activities such eating, drinking or cooking (n = 5). Regarding the questions about physical and social context of participation, the majority of participants (n = 234, 70%) reported to be at home alone during the experiment5 (see Table 7).
Table 6.
Number of people who answered positively on the control questions in Experiment 1.
| Question | Stayed on the web page | Interrupted the video | Left the web page at least once | Left the page at least once and interrupted the video |
|---|---|---|---|---|
| N | 243 | 47 | 49 | 7 |
| “Have you seen this video before?” | 116 (48%) | 29 (62%) | 27 (55%) | 6 (86%) |
| “Have you watched the video until the end?” | 236 (97%) | 35 (74%) | 48 (98%) | 7 (100%) |
| “Were you distracted by other activities while watching the video?” | 18 (7%) | 6 (13%) | 4 (8%) | 2 (29%) |
| Any answer was given to the question “If you were distracted, please describe what you were doing.” | 22 (9%) | 3 (6%) | 4 (8%) | 2 (29%) |
Table 7.
Physical and social context of participation in Experiment 1.
| Alone | Friends | Relatives | Strangers | Other | Total | |
|---|---|---|---|---|---|---|
| Home | 243 | 9 | 33 | 3 | 6 | 294 |
| University | 11 | 9 | 0 | 12 | 3 | 35 |
| Transport | 1 | 0 | 0 | 1 | 0 | 2 |
| Work | 7 | 0 | 0 | 0 | 1 | 8 |
| Other | 1 | 4 | 0 | 1 | 1 | 7 |
| Total | 263 | 22 | 33 | 17 | 11 |
2.4. Discussion
In Experiment 1, 30% of the participants who finished the experiment either interrupted the video or temporarily left the video page. Whereas the temporary leave was relatively short and did not influence mood ratings, interruption of the video was different between mood conditions and affected the mood ratings afterward. The results support the affect regulation hypothesis so that people confronted with the negative emotional content interrupted watching the video more often than people with the positive content. Additionally, the people in the negative mood condition who interrupted watching the video were less affected by the mood manipulation, while there were no differences for the positive mood condition.
Although the video length was different between positive and negative conditions, it did not substantially affect the interruption rate. Perhaps, the length of the video should be somewhat longer to observe the higher interruption at later phases of the experiment, e.g. as a result of the experienced burden (Galesic, 2006). Nevertheless, we used video clips of similar length in Experiment 2 to rule out potential confounding of the mood manipulation with the time in the experiment.
The mood manipulation was overall successful, as people in the negative mood manipulation condition had more negative and less positive emotions after the video than people in the positive condition. According to the changes in mood before and after the video, the size of the mood manipulation effect was larger for the negative (20% decrease) than for the positive mood condition (5% increase). That result is in line with previous research finding that negative emotions are easier to elicit than positive ones (Ferrer et al., 2015). However, we did not observe any ceiling effect for the positive mood score – people in the positive mood condition significantly increased their mood as well.
Reviewing the self-report data, we found that more people interrupted the video they had seen before. This could be the result of a strategy to minimize time in the experiment if participants already knew the movie plot. The prior viewing had no effect on mood changes, which contradicts the finding that the prior viewing can intensify the experienced emotions (Gross and Levenson, 1995). The majority of participants did not admit that they interrupted the video, which implies that the self-report data can be an unreliable indicator of compliance. The reported distraction was not related to the interruption of the video, suggesting that the distraction is a more random event occurring during the experiment, but the noncompliance is a participant's decision to dismiss the experimental instructions. Finally, a relatively small percentage of users (15%) conducted the study outside their home, but mobile use was not associated with the noncompliance or mood manipulation results.
Experiment 1 showed that the differences between negative and positive conditions were more prominent in interrupting the mood manipulation procedure by clicking the continue button than in the temporary leave of the web page. However, both forms of noncompliance may be caused by the same avoidance motivation. Therefore, we expected that the prevention of video interruption (e.g., by hiding the continue button until the end of the video) would increase the temporary leave rate and the effect of mood manipulation on the leave. To investigate this hypothesis, we conducted Experiment 2 where participants could not use the continue button before the end of the video.
3. Methods
3.1. Experiment 2
3.1.1. Participants
Two hundred and eight cases (mood group: 104 positive (50%), 104 negative (50%)) of participation were registered. Filtering was applied to 28 records (see Table 8) leaving 180 participants for the analysis (see Table 9).
Table 8.
Number of excluded participants in Experiment 2.
| Negative | Positive | Overall | |
|---|---|---|---|
| Beginning N | 104 | 104 | 208 |
| Page reloads | 3 | 2 | 5 |
| Repeated participation | 3 | 3 | 6 |
| Did not answer the intention question | 3 | 0 | 3 |
| Left at the first mood question | 2 | 1 | 3 |
| Left during the movie | 4 | 6 | 10 |
| Left at the second mood question | 0 | 0 | 0 |
| Left at the emotions questionnaire | 0 | 1 | 1 |
| Left at the accuracy questions about the video | 0 | 0 | 0 |
| Left at the debriefing questions | 0 | 0 | 0 |
| Left on the final page with emails | 0 | 0 | 0 |
| Problems with video | 0 | 0 | 0 |
| Final N | 89 | 91 | 180 |
Table 9.
Sample characteristics.
| Mood condition | Sample size | Mean age (SD) | Age range | Females |
|---|---|---|---|---|
| Negative | 89 | 21 (3.1) | 18–51 | 73 |
| Positive | 91 | 21 (4.2) | 18–34 | 84 |
| All | 180 | 21 (3.7) | 18–51 | 157 |
4. Materials
Mood induction. Positive mood was manipulated by the fragment with the character played by Ben Stiller fighting with a dog in the movie “There's something about Mary” (1998, 3:25 min) (Gilman et al., 2017). Negative mood was induced by the scene with a man losing his wife in a car accident from the movie “Return to me” (2000, 3:25 min) (Rottenberg et al., 2007).
Mood manipulation check. As in Experiment 1, participants answered a general question about their current mood (“How would you characterize your present emotional state?”) both before and after watching the video. Additionally, they rated their emotions on ten different scales of the PANAS after the video (Watson and Clark, 1999).
4.1. Design and procedure
The design and procedure of the experiment were similar to Experiment 1 except the continue button on the web page with the video was hidden until the end of the video. Therefore, only one form of noncompliance behavior, that is a temporary leave of the page, was registered. Participants were also asked whether they experienced any technical problems while watching the video. Additionally, three accuracy questions about the content of the video were added to examine whether they can distinguish noncompliance behavior. The participants were recruited from the subject pool of students at the University of Mannheim.
5. Results
5.1. Temporary leave
Thirty-seven participants (21%) temporarily left the web page during the video. The number of participants was higher in the negative mood condition (n = 25, 28%) than in the positive mood condition (n = 12, 13%), χ2 (1, N = 180) = 6.12, p = .013, φ = 0.18 (see Table 10).
Table 10.
The temporary leave in the mood groups. Row percentages are written in parentheses.
| Stayed on the page | Left the page at least once | Total | |
|---|---|---|---|
| Negative | 64 (72%) | 26 (28%) | 89 |
| Positive | 79 (87%) | 12 (13%) | 91 |
| Total | 143 | 37 | 180 |
As an exploratory analysis, we compared the time spent out of the focus of the web page between mood manipulation conditions. Since dependent variables were not normally distributed, a nonparametric Mann-Whitney U-test was applied. The length of missing time (in seconds) was not different between negative and positive mood manipulation conditions, W = 139, p = .88, r = 0.03. However, there was a tendency for participants in the negative condition (Mdn = 29.81 s) to leave the web page later than the participants in the positive condition (Mdn = 7.67 s), W = 204, p = .08, r = 0.29 (see Fig. 2).
Fig. 2.
The time (in seconds) spent out of the focus of the web page with the video. The time of video play is on the horizontal axis with two vertical bars indicating the beginning and the end of the video. The lines represent the time intervals of different participants when they were outside the web page.
A 2 × 2 between-subjects ANOVA was applied to analyze the effect of the mood manipulation and temporary leave on the change in the mood score that was measured before and after the video. Since two participants did not answer the question, the analysis was done for 178 participants. The analysis showed a main effect of the mood manipulation, F (1, 174) = 170.02, MSE = 50188, p < .001, η2 = 0.49, but there were neither effects of temporary leave, F (1, 174) = 2.34, MSE = 691, p = .13, η2 = 0.007, nor the interaction between the mood condition and temporary leave, F (1, 174) = 0.14, MSE = 42, p = .71, η2 < 0.001 (see Table 11).
Table 11.
The mood changes for the groups who stayed on the page and left the page at least once in the negative and positive mood manipulation conditions.
| Negative mood manipulation |
Positive mood manipulation |
All participants |
||||
|---|---|---|---|---|---|---|
| N | Mean (SD) | N | Mean (SD) | N | Mean (SD) | |
| Stayed on the page | 63 | -26.44 (19.70) | 79 | 8.34 (15.67) | 142 | -7.09 (24.64) |
| Left the page at least once | 24 | -20.51 (16.81) | 12 | 11.73 (12.29) | 36 | -9.76 (21.70) |
| All participants | 87 | -24.81 (19.04) | 91 | 8.78 (15.25) | 178 | -7.63 (24.04) |
We also conducted the ANOVA analysis for emotions measured by the PANAS after the video6. For negative emotions, the effect of mood manipulation was significant, F (1, 174) = 68.29, MSE = 39.07, p < .001, η2 = 0.281, but neither the effect of temporary leave, F (1, 174) = 0.005, MSE = 0.003, p = .94, η2 < 0.001, nor the interaction was significant, F (1, 174) = 0.50, MSE = 0.29, p = .48, η2 = 0.002. For positive emotions, the effect of mood manipulation was significant, F (1, 174) = 39.28, MSE = 22.20, p < .001, η2 = 0.183, but neither the effect of temporary leave, F (1, 174) = 0.01, MSE = 0.005, p = .92, η2 < 0.001, nor the interaction was significant, F (1, 174) = 0.88, MSE = 0.50, p = .35, η2 = 0.004. For arousal, there was no significant effect of mood manipulation, F (1, 174) = 1.66, MSE = 1.76, p = .20, η2 = 0.009, no significant effect of temporary leave, F (1, 174) = 0.03, MSE = 0.04, p = .85, η2 < 0.001, nor the interaction, F (1, 174) = 3.07, MSE = 3.25, p = .08, η2 = 0.004 (see Table 12).
Table 12.
The PANAS scores after the video in different groups (Experiment 2).
| Negative mood manipulation |
Positive mood manipulation |
All participants |
|||||
|---|---|---|---|---|---|---|---|
| N | Mean (SD) | N | Mean (SD) | N | Mean (SD) | ||
| Negative scale | Did not leave | 64 | 2.79 (0.84) | 79 | 1.89 (0.70) | 143 | 2.29 (0.89) |
| Left the page | 23 | 2.88 (0.69) | 12 | 1.77 (0.77) | 35 | 2.50 (0.88) | |
| All participants | 87 | 2.81 (0.80) | 91 | 1.88 (0.71) | 178 | 2.33 (0.89) | |
| Positive scale | Did not leave | 64 | 2.34 (0.73) | 79 | 3.10 (0.81) | 143 | 2.76 (0.86) |
| Left the page | 23 | 2.46 (0.60) | 12 | 2.94 (0.76) | 35 | 2.63 (0.69) | |
| All participants | 87 | 2.37 (0.69) | 91 | 3.08 (0.80) | 178 | 2.73 (0.83) | |
| Arousal | Did not leave | 64 | 3.10 (1.06) | 79 | 2.76 (1.00) | 143 | 2.91 (1.04) |
| Left the page | 23 | 2.79 (1.08) | 12 | 3.17 (0.96) | 35 | 2.92 (1.02) | |
| All participants | 87 | 3.02 (1.07) | 91 | 2.82 (1.00) | 178 | 2.91 (1.03) | |
5.2. Self-report
Nobody in the negative mood manipulation condition had seen the movie before, and 18 people in the positive condition (20%) had previously seen the movie (see Table 13). Prior viewing was not related to the temporary leave of the web page, χ2 (1, N = 180) = 0.19, p = .66, φ = 0.03, and did not affect the change in the mood score, F (1, 175) = 1.18, MSE = 347.90, p = .28, η2 = 0.003. Everybody reported to watch the video until the end, but four people (one in the group who temporary left the page) reported being distracted by other activities during the video. Among the reasons of distraction, three causes were related to other people and one to the music outside. Four people (all of them are in the group who temporarily left the page, two in the positive and two in the negative condition) reported to have technical problems during the video. The majority of participants (n = 139, 77%) reported to be at home alone during the experiment7 (see Table 14). Concerning the accuracy questions, the number of people who gave at least one wrong answer in three accuracy questions was not different in the group who stayed on the web page (n = 7, 5%) and who temporarily left the web page (n = 3, 8%), χ2 (1, N = 180) = 0.58, p = .45, φ = 0.06.
Table 13.
Number of people who answered positively on the control questions in Experiment 2.
| Question | The participants who stayed on the webpage | The participants who left the web page at least once |
|---|---|---|
| N | 143 | 37 |
| “Have you seen this video before?” | 15 (10%) | 3 (8%) |
| “Have you watched the video until the end?” | 143 (100%) | 37 (100%) |
| “Have you experienced any technical problems while watching the video?” | 0 (0%) | 4 (11%) |
| “Were you distracted by other activities while watching the video?” | 3 (2%) | 1 (3%) |
| Any answer was given to the question “If you were distracted, please describe what you were doing.” | 3 (2%) | 1 (3%) |
Table 14.
Physical and social context of participation in Experiment 2.
| Alone | Friends | Relatives | Strangers | Other | Total | |
|---|---|---|---|---|---|---|
| Home | 139 | 4 | 7 | 0 | 0 | 150 |
| University | 10 | 8 | 0 | 5 | 3 | 26 |
| Transport | 0 | 0 | 1 | 1 | 0 | 2 |
| Work | 1 | 0 | 0 | 0 | 0 | 1 |
| Other | 0 | 0 | 0 | 1 | 0 | 1 |
| Total | 150 | 12 | 8 | 7 | 3 |
6. Discussion
The results of Experiment 2 supported the idea that people in the negative mood condition avoid the video more often than people in the positive mood condition. This result is in line with the affect regulation theory that expects people to regulate their emotions by escaping from the situation with negative emotional content. The finding conceptually replicates Experiment 1, which showed that people in the negative mood condition interrupted the video by clicking the continue button more often than people in the positive condition. Since Experiment 2 was void of the opportunity to interrupt the video, participants who wanted to withdraw from the negative emotional content used the chance to switch between web pages.
Additional explorative analysis showed that the participants in the negative mood condition were leaving the webpage at a later moment than in the positive condition. This might be related to the fact that the emotionally adverse event in the negative condition video (injured wife of the main hero in a hospital) appeared only at 20th second, whereas the positive condition video did not have any negative episodes and could be recognized from the beginning as an excerpt from a comedy.
Although there were differences between conditions, the temporary leave did not affect the mood ratings as the interruption of the video did in Experiment 1. The possible reasons are that the leave was not long enough to avoid the mood manipulation or switching to another web page was not sufficient to restore the previous mood.
The answers to accuracy questions were not related to objective measures of the residence time on the web page. However, the accuracy questions could filter people who completely missed the video or, perhaps, did not understand the instructions. As in Experiment 1, asking participants whether they were distracted during the task could not guarantee the reliable answer. The motivation to get credit points or participate in the lottery might prompt participants to give socially desirable responses. Also, the participants might not recognize their switches back and forth between web pages during the video as a distraction that they should report.
6.1. General discussion
The current research investigated noncompliance behavior in an online mood manipulation with movie clips. The filtering applied to participants with incomplete data is not enough to tackle the noncompliance behavior. There is a proportion of participants (30% in Experiment 1 and 21 % in Experiment 2), not traceable with standard measures (e.g., accuracy questions), who do not follow the mood-manipulation instruction: they interrupt the video by using the continue button or switch between web pages during the video. This proportion might differ depending on the sample characteristics. Our experiments used a university sample of students, who participated for credit points. However, the more heterogeneous and less motivated sample might result in more cases of noncompliance, as indicated by the higher dropout rates with less committed samples (Reips, 2002).
Our research shows that the noncompliance behavior depends on the emotional content of the mood manipulation. Supporting the affect regulation theoretical account, participants preferred to avoid negative emotional stimuli more often than positive ones. The finding is in line with previous research that has demonstrated attentional deployment strategy of mood regulation, that is to look away from negative stimuli (van Reekum et al., 2007). In an online study, there are other possible strategies to avoid the mood manipulation, which are contingent on the design of an online-study, i.e., accessibility of the continue button. We recommend hiding the continue button until the end of a mood manipulation to prevent participants from interrupting the procedure. In case if the continue button cannot be removed, the residence time on the web page can be measured and used as a control variable in the analysis. Additionally, the inclusion of the time of the web page staying in focus into the analysis is an exciting avenue for future research. Contrary to a laboratory experiment, where a participant is confined to the use of experimental software, online experiments are executed in the web-browser environment where participants can transition between multiple windows or tabs.
The results of our study contribute to a more general field of research such as the use of paradata in a survey. Paradata refers to additional information about participants' behavior that is collected during the survey, such as response times, response rates or mouse movements (for a review, see Kreuter, 2013). The emotional state of respondents may play a role in their interaction with the survey. Moreover, if some of the questions evoke emotional reactions, this may change the response style during the survey. Therefore, several survey questions can be used to assess the participants' emotional state in order to determine the relationship between their mood and the noncompliance. Additionally, researchers can use paradata to infer the participants’ emotional state, which is an exciting avenue for further research. As our study showed, a longer leaving time in an online study may correlate with negative emotions experienced at the moment. Another challenge is the collection of paradata from the mobile user group. Further research should test the browser focus events on mobile devices and develop algorithms for their detection.
In conclusion, people might regulate their emotions by avoiding certain mood eliciting content on the Internet which poses a threat to the internal validity of Internet-based mood manipulation studies. To decrease these risks, future research should adopt the following techniques: preventing participants from skipping a part of the mood manipulation, detecting page focus events, and measuring the time people stay on a page.
Declarations
Author contribution statement
Y. Shevchenko: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed materials, analysis tools or data; Wrote the paper.
A. Bröder: Conceived and designed the experiments; Wrote the paper.
Funding statement
Yury Shevchenko was supported by a research grant of the Center for Doctoral Studies in Social and Behavioral Sciences (CDSS) at the University of Mannheim.
Competing interest statement
The authors declare no conflict of interest.
Additional information
No additional information is available for this paper.
Footnotes
The link to the experiments https://yuryshevchenko.com/online-study/emotional-movies/.
Psychological Research on the Net https://psych.hanover.edu/research/exponnet.html.
The number of people for the analysis was 331, since five people did not complete the PANAS questionnaire.
The number of people for the analysis was 345, since one participant did not answer on the question about watching the movie before.
We thank a reviewer who pointed out the possibility that the participants had used mobile phones, which could affect the noncompliance rate. To address this concern, we conducted an exploratory analysis of whether the use of mobile phones influenced the interruption rate. We identified the mobile phone users based on user agent information sent from the user's browser to our server. Although this information may be forged, we did not expect our participants to do so intentionally. We categorized users with the agent information containing the tag “Mobile” as mobile users (n = 52, 15%). There was no relation between the use of mobile phone and the video interruption, χ2 (1, N = 346) = 0.002, p = .96, φ = 0.003, as mobile users (n = 46, 16%) interrupted the video as often as the other participants (n = 8, 16%). We also used a 2 × 2 ANOVA to analyze the effect of the mobile phone use, the mood manipulation condition and their interaction on the changes in mood. The effect the mood manipulation remained significant, F (1, 332) = 133.70, MSE = 43720, p < .001, η2 = 0.28, but neither the use of mobile phone, F (1, 332) = 0.58, MSE = 191, p = .45, η2 = 0.001, nor the interaction effect were significant, F (1, 332) = 0.16, MSE = 53, p = .69, η2 < 0.001. Regarding the temporary leave, there were no leave events in the group of mobile users, which can be related to the behavior of the participants, but also to technical difficulties in detecting the focus event in a mobile browser.
The number of people for the analysis was 178, since two people did not complete the PANAS questionnaire.
We analyzed the use of mobile phones in the same way as we did in Experiment 1. In general, 21 participants (22%) were categorized as mobile users. There was a relation between the use of mobile phone and the temporary leave of the video, χ2 (1, N = 180) = 9.87, p = .002, φ = 0.23, as mobile users (n = 1, 3%) left the video less often than the other participants (n = 36, 26%). We applied a 2 × 2 ANOVA to analyze the effect of the mobile phone use, the mood manipulation condition and their interaction on the changes in mood. The effect the mood manipulation remained significant, F (1, 174) = 170.72, MSE = 50187.98, p < .001, η2 = 0.49, but neither the use of mobile phone, F (1, 174) = 0.44, MSE = 130.45, p = .51, η2 = 0.001, nor the interaction effect were significant, F (1, 174) = 2.77, MSE = 814.05, p = .10, η2 = 0.008.
References
- Andrade E.B. Behavioral consequences of affect: combining evaluative and regulatory mechanisms. J. Consum. Res. 2005;32(3):355–362. [Google Scholar]
- Arechar A.A., Gächter S., Molleman L. Conducting interactive experiments online. Experiment. Econ. 2018 doi: 10.1007/s10683-017-9527-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bonanno G.A., Goorin L., Coifman K.G. Sadness and grief. In: Lewis M., Haviland-Jones J.M., Barrett L.F., editors. Handbook of Emotions. Guilford Press; 2008. pp. 797–810. [Google Scholar]
- Crump M.J.C., McDonnell J.V., Gureckis T.M. Evaluating amazon’s mechanical Turk as a tool for experimental behavioral research. PLoS One. 2013;8(3) doi: 10.1371/journal.pone.0057410. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Diedenhofen B., Musch J. PageFocus: using paradata to detect and prevent cheating on online achievement tests. Behav. Res. Methods. 2017;49(4):1444–1459. doi: 10.3758/s13428-016-0800-7. [DOI] [PubMed] [Google Scholar]
- Ferrer R.A., Grenen E.G., Taber J.M. Effectiveness of internet-based affect induction procedures: a systematic review and meta-analysis. Emotion. 2015;15(6):752–762. doi: 10.1037/emo0000035. [DOI] [PubMed] [Google Scholar]
- Galesic M. Dropouts on the web: effects of Interest and burden experienced during an online survey. J. Off. Stat. 2006 [Google Scholar]
- Gilman T.L., Shaheen R., Nylocks K.M., Halachoff D., Chapman J., Flynn J.J. A film set for the elicitation of emotion in research: a comprehensive catalog derived from four decades of investigation. Behav. Res. Methods. 2017;49(6):2061–2082. doi: 10.3758/s13428-016-0842-x. [DOI] [PubMed] [Google Scholar]
- Göritz A.S. The Induction of Mood via the WWW. Motiva. Emot. 2007 [Google Scholar]
- Gould S.J.J., Cox A.L., Brumby D.P. Diminished control in crowdsourcing. ACM Trans. Comput. Hum. Interact. 2016;23(3):1–29. [Google Scholar]
- Gross J.J. The emerging field of emotion regulation: an integrative review. Rev. Gen. Psychol. 1998;2(3):271–299. [Google Scholar]
- Gross J.J., Levenson R.W. Emotion elicitation using films. Cognit. Emot. 1995;9(1):87–108. [Google Scholar]
- Gureckis T.M., Martin J., McDonnell J., Rich A.S., Markant D., Coenen A. psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods. 2016;48(3):829–842. doi: 10.3758/s13428-015-0642-8. [DOI] [PubMed] [Google Scholar]
- Judd T., Kennedy G. Measurement and evidence of computer-based task switching and multitasking by “Net Generation” students. Comput. Educ. 2011;56(3):625–631. [Google Scholar]
- Kreuter F., editor. Improving Surveys with Paradata: Analytic Uses of Process Information. Wiley; Hoboken, New Jersey: 2013. [Google Scholar]
- Oppenheimer D.M., Meyvis T., Davidenko N. Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 2009 [Google Scholar]
- Reips U.D. Standards for internet-based experimenting. Exp. Psychol. 2002;49(4):243–256. doi: 10.1026//1618-3169.49.4.243. [DOI] [PubMed] [Google Scholar]
- Rottenberg J., Ray R.D., Gross J.J. Emotion elicitation using films. Handbook of emotion elicitation and assessment. In: Coan J.A., Allen J.J.B., editors. Handbook of Emotion Elicitation and Assessment. Oxford University Press; 2007. pp. 2–28. [Google Scholar]
- Shevchenko Y., Bröder A. The effect of mood on integration of information in a multi-attribute decision task. Acta Psychol. 2018;185:136–145. doi: 10.1016/j.actpsy.2018.02.007. [DOI] [PubMed] [Google Scholar]
- van Reekum C.M., Johnstone T., Urry H.L., Thurow M.E., Schaefer H.S., Alexander A.L., Davidson R.J. Gaze fixations predict brain activation during the voluntary regulation of picture-induced negative affect. Neuroimage. 2007;36(3):1041–1055. doi: 10.1016/j.neuroimage.2007.03.052. [DOI] [PubMed] [Google Scholar]
- Watson D., Clark L. Iowa Research Online; 1999. The PANAS-X Manual for the Positive and Negative Affect Schedule-Expanded Form. [Google Scholar]
- Wegener D.T., Petty R.E. Mood management across affective states: the hedonic contingency hypothesis. J. Personal. Soc. Psychol. 1994;66(6):1034–1048. doi: 10.1037//0022-3514.66.6.1034. [DOI] [PubMed] [Google Scholar]
- Westermann R., Spies K., Stahl G., Hesse F.W. Relative effectiveness and validity of mood induction procedures: a meta-analysis. Eur. J. Soc. Psychol. 1996;26(4):557–580. [Google Scholar]
- Yeykelis L., Cummings J.J., Reeves B. Multitasking on a single device: arousal and the frequency, anticipation, and prediction of switching between media content on a computer. J. Commun. 2014;64(1):167–192. [Google Scholar]