The Question of Violent Video Games and Aggression: Testing Statistical and Methodological Issues of Null Effects Using Data From an Open‐Access Case Study

ABSTRACT

While numerous meta‐analyses, reviews, and task forces from various scientific bodies have linked violent media use to aggression, some studies report null effects and claim that such a relationship is non‐existent. Several scholars have noted that potential methodological and statistical errors could explain failures to replicate these effects. One recent failure to replicate established violent video game effects (Przybylski and Weinstein 2019) has made its data set publicly available, thereby offering a unique opportunity to examine the hypothesis that methodological and statistical problems underlie some replication failures. The present study re‐examined the original results from Przybylski and Weinstein (2019) using more appropriate analyses and replicated those results with recalculated, corrected, and more theoretically appropriate variables. The first part examines issues within the original study, including problems with the measure used to assess aggression, statistical control, and the measurement of exposure to video game violence (VGV). The second part created a more standard measure of VGV exposure to test whether the null result stemmed from this measurement issue. Overall, results demonstrate that conceptual misunderstandings of aggression, poor measures regarding both aggression and video game violence exposure, and inappropriate statistical procedures contributed to the initially reported null results. Furthermore, by using improved materials (including a more sophisticated coding scheme to assess exposure to violent video games) and sound statistical analysis (correcting for overcontrol), the data replicate the long‐established relationship between playing violent video games and aggressive behavior.

Concern about what people are exposed to in the media is nothing new, as media effects research dates back to the 1900s (e.g., De Tarde ¹⁹¹²), with one primary question for scholars being whether exposure to violent video media increases aggressive behavior.

Findings showing a relationship between violent media exposure and increased aggression are some of the most replicated and well‐supported effects in the psychological literature (e.g., Anderson et al. ²⁰⁰³, ²⁰¹⁰; Cantor ²⁰⁰⁰; Comstock and Scharrer ²⁰⁰³; Mathur and VanderWeele ²⁰¹⁹; Prescott et al. ²⁰¹⁸; Wood et al. ¹⁹⁹¹)—with effects being demonstrated in short‐term experiments, longitudinal cross‐sectional surveys, and even in neurological imaging. Several key theories of aggressive behavior have been applied to the media's effects on aggression, explaining its multifaceted influence on human psychology (e.g., social learning, scripts, cognitive neoassociation, social information processing, excitation transfer, and desensitization). This multidisciplinary approach to understanding the impact of violent media exposure is illustrated by the General Aggression Model, which demonstrates how external and social stimuli interact with internal psychological processes to influence the likelihood of aggressive behavior (e.g., Anderson and Bushman ²⁰¹⁸). The entire body of media violence research literature, including numerous meta‐analyses and reviews by scientific bodies tasked with evaluating the quality of the research, consistently concludes that the observed positive relationship is supported by the empirical evidence and theory ¹ .

1. Null Effects, Replication Crises, and Debate

As in any research domain, a small proportion of media violence studies report null effects. In response to these failures to replicate, some scholars have expressed criticism around the established relationship between aggression and media violence exposure, with some claiming that it is non‐existent or illusory (e.g., Ferguson and Kilburn ²⁰⁰⁹, ²⁰¹⁰; Hilgard et al. ²⁰¹⁷). In response to this “debate,” several scholars have noted numerous flaws in some null‐result studies (as well as noting that in a robust research field on a relatively small effect, some studies will naturally find null results).

1.1. Possible Causes for Discrepant Findings

If the vast majority of media violence scholars and comprehensive meta‐analyses find overwhelming support for the basic media violence/aggression link and the underlying causal processes, then why do others report in their empirical studies and meta‐analyses that there is indeed no such effect? Researchers have identified multiple explanations for null effect findings, including the simple fact that, by chance alone, one should expect a few studies with nonsignificant effects. Other potential issues include poor methodology, weak measurement of key variables, inappropriate statistical analyses, researchers' intent, and conceptual misunderstandings (Anderson and Bushman 2023; Bushman and Anderson ²⁰²³; Kim et al. ²⁰²¹). Here we outline a few of these identified concerns.

1.2. Inappropriate or Excessive Covariates

One common mistake found in studies that claim to demonstrate a null effect is that the researchers inappropriately control for shared variance from their dependent variables. For instance, several correlational studies have partialled out trait aggression (e.g., Przybylski and Weinstein ²⁰¹⁹) from their outcome variable of aggression, essentially partialling out the dependent variable from itself. Others have partialled out trait variables that theoretically mediate the media violence to aggression link, such as anger and hostility (e.g., Ferguson et al. ²⁰¹⁴). Using such inappropriate “controls” renders the test of the media violence/aggression link conceptually meaningless (Wright and Tokunaga 2023).

Other studies control for so many variables (relative to sample size) that they inappropriately deflate the target effect in question. For example, this is a concern in correlational studies such as DeCamp and Ferguson (2017), which “controlled” for all the following variables: violent media propensity, parental attachment, youth disclosure, parental enforcement, yelling in the home, violence in the home, having been hit by an adult, gender, race and ethnicity, and poverty. Excessive covariates are particularly problematic when the control variables include measures that are essentially duplicate or overlapping measures of the predictor variable, the dependent variable, or mediating variables. Common statistical principles also reveal that even if the controlled variables are not of these three types, too many predictors relative to sample size invalidates the statistical test of the focal hypothesis.

1.3. Poor Measures of Aggression and Media Violence

Another common error in media violence research is using poor measures of key variables, such as using too few items to assess aggressive behavior or media violence exposure reliably, or using measures that do not assess the underlying conceptual construct—for example, using a trait‐measure of aggression as the dependent variable is inappropriate in a short‐term experimental study (e.g., Teng et al. ²⁰¹¹). This can create issues in the generalization of results because trait aggression measures help assess the overall patterns of aggression in a person's lifespan, which are appropriate to ask in cross‐sectional and longitudinal designs. However, in experimental designs assessing the influence of media violence exposure on “state” aggression (i.e., actions, cognitions, and emotions “in the moment”), trait aggression questions would not be appropriate outcome variables.

2. Present Research

The present research utilized a publicly available data set from a recent video game and aggression study to investigate whether the null effect reported in that study was due to methodological and/or statistical errors. Additionally, the data set used by the original authors is particularly advantageous for testing the hypotheses concerning methodological and statistical shortcomings of some null effects studies theorized by reviews such as Anderson and Bushman (2023) and Kim et al. (2021). For example, the data set is large, reducing the probabilities of type 1 or type 2 errors when the correct analyses are performed. The data set also includes multiple assessments of the relevant constructs, allowing for various types of tests with differing constructions of key variables, such as multiple assessments of “aggression” and multiple methods for computing media violence exposure.

We describe it as two “studies” to indicate the different approaches to the sets of analyses conducted. “Study 1” used the originally published materials and data to explore the issues in the original claims and analyses, and to test for the effect of overcontrol in the original regression model. “Study 2” created a new standardized measure of exposure to violent video games from the original data set to test the replicability of our findings from “Study 1.”

There are three main areas of concern in the original article:

• 1.The conduct problems subscale of the Strengths and Difficulties Questionnaire (SDQ) has not been historically used as a measure of aggression and does not appear to be an appropriate measurement for that construct (Goodman et al. 2000).
• 2.The measure of violent video game exposure is potentially weak. The authors reported that they used an expert coder to rate whether the games reported by participants contained violent content. To do so, they used the European Union and North American Ratings Boards (the PEGI and ESRB ratings systems, respectively) to code whether each game listed by participants had violent content or not (0 = no violent content, 1 = violent content). However, there are numerous concerns with using content ratings boards to assess violence and aggression presented in games (e.g., Handsley and Warburton ²⁰²¹).
• 3.The statistical methodology included inappropriate covariates. Specifically, the analyses treat the Buss Perry Aggression Questionnaire (BPAQ) as a statistical control for “trait aggression” in this cross‐sectional study. We hypothesize that the authors may have effectively partialled out the conceptual dependent variable (aggressive behavior) by including the BPAQ as a covariate.

3. Methods

3.1. Design

This study has been preregistered on Open Science Framework (OSF): https://osf.io/ney9t/. Additionally, the data and materials used in the original study were also preregistered on OSF and can be accessed here: https://osf.io/rkw6z/. Przybylski and Weinstein (2019) used a cross‐sectional design to test the hypothesis “that time spent playing violent video games is positively associated with adolescents' everyday behavioural aggression (p. 3).” All original data were gathered via surveys from parents/caregivers and participants' self‐reports.

4. Przybylski and Weinstein (2019) Participants

4.1. Drop‐Outs and Missing Data

The article does not report any missing data, nor does it address how missing data were handled in the analyses. As we will discuss, the nature of how participants were recruited for the survey may explain the absence of missing data, as they were recruited to complete this additional survey after participating in an in‐person research project.

4.2. Demographics

One thousand four participants and an equal number of caregivers were recruited to participate using “A participant profile of adolescents aged 14 and 15 years living in England, Scotland and Wales…based on 2011 United Kingdom Census data (p. 4).” The authors disclose that they partnered with a research center that had previously used these types of profiles in its recruitment. Details concerning profile construction, recruitment materials, the description of the study's consent form, and specific instructions to participants were not included in the manuscript or the OSF materials. ²

Of the 1004 participants, all were either 14 (n = 497) or 15 (n = 507) years of age, and most identified as white (91.9%). The participant pool consisted of 540 males, 461 females, and three individuals who identified as having another gender identity.

5. Przybylski and Weinstein (2019) Materials

5.1. Aggression

5.1.1. Strength and Difficulties Questionnaire

The dependent variable of aggressive behavior used by Przybylski and Weinstein (2019) was the conduct problems (CP) subscale of the Strength and Difficulties Questionnaire (SDQ), a survey used to evaluate psychological adjustment in children and adolescents (Goodman 1997; Van Roy et al. ²⁰⁰⁸; Goodman et al. ²⁰⁰⁰). The SDQ is typically given to caregivers (such as parents) to answer questions about the behaviors of their children on a one to three scale (1 = Not True, 2 = Somewhat True, 3 = Certainly True). However, even though Przybylski and Weinstein (2019) claim that this measure has been used “to measure aggressive behaviour and interpersonal aggression (p. 4),” we have been unable to identify literature that suggests the SDQ as a standard assessment of aggression.

The SDQ was not created to measure aggressive behavior explicitly, nor was the conduct problems subscale. The SDQ functions as a way for caregivers (such as parents or teachers) to report behaviors in children to identify potential psychological issues (see Goodman et al. ²⁰⁰⁰). The studies cited by Przybylski and Weinstein for the use of this scale focused on broader behavioral issues (Goodman et al. 2000; Matsuishi et al. ²⁰⁰⁸; Becker et al. ²⁰⁰⁴), and none specifically assessed aggression in adolescents. Additionally, the conduct problems subscale is not explicitly focused on aggression, as only one of its five items assesses aggressive behavior (“Often gets into physical fights”). In short, this outcome variable is at best a weak measure of the conceptual dependent variable of aggressive behavior.

5.1.2. Buss‐Perry Aggression Questionnaire

The Buss‐Perry Aggression Questionnaire (BPAQ) is a self‐report survey meant to measure four aspects of aggressive personality. Two concern aggressive behavior (physical and verbal); the other two assess potential mediating variables (hostility and anger) (Buss and Perry 1992). The original scale has 29 items. Przybylski and Weinstein (2019; see Appendix B) used a 12‐item short form, with three items for each subscale. The two behavior subscales are the most valid measures of aggressive behavior included in this study; they could have been used to test the focal hypothesis. Nonetheless, these short‐form measures are also statistically weak, as they are based on only three items for each subscale.

5.2. Violent Video Game Exposure

To create a variable measuring exposure to violent video games (VGV), Przybylski and Weinstein (2019) used information from five items: Item 1 asked participants if they played video games (“yes/no”). If they answered “yes” to item 1, Item 2 asked participants to name their three most played games. Item 3 asked what video game system participants engaged with most often. Item four asked if they played primarily single‐player or multi‐player games. Item 5 was a 9‐point scale that asked how frequently each player reported playing the listed games in hours per day (1 = none at all, 9 = 7+ hours a day). The information provided was “used by a coder who was blind to the purpose of the study to find the game's entry on the Pan European Game Information (PEGI) website for confirmatory hypothesis testing (p. 6).”

The games reported by the participants had to be matched by the expert coder in the data set because many of the game names reported by participants were vague and ambiguous. For example, participants often reported playing Call of Duty, but the franchise comprises over 20 games, some of which have different parental ratings than others. It is unclear how the coder identified which version of the game to use for coding violent content, as per the PEGI website, for each participant who listed only that generic name. After choosing a specific version of the game, the coder assigned a new code of one if the PEGI badge indicated the game included violent content, or zero if the game had no violent content badge. The coder also assessed the age rating of the game (a 1–6 scale, where 1 = 3+ and 6 = 18+). Finally, the authors multiplied the 1–0 presence of violence code for each game by the participants' 9‐point rating of time spent playing that game. This was done for each of the three games reported, and then those three scores were combined to create a variable of time spent playing violent video games. This is a reasonable way to calculate a video game violence exposure score if the PEGI rating system accurately codes for violence and the coder's decision regarding which game title (and version) was intended by the P's typed name of their three most frequently played games (Busching et al. 2015).

There are numerous considerations in creating a composite measure of video game play and exposure to violent content in those games. One primary concern with the present study is the use of only one coder. A second is reliance on the PEGI rating system to code violent content (0 = no, 1 = yes). A third is the use of a dichotomous coding of games as either violent or nonviolent; games differ considerably in the amount of violent content. Several studies have demonstrated a clear misunderstanding of aggression and violence in the context of ratings boards' evaluations of violent content in video games (e.g., Gentile ²⁰⁰⁸; Handsley and Warburton ²⁰²¹; Walsh and Gentile ²⁰⁰¹). Thus, the VGV measure could be a weak predictor variable due to the flaws in its construction, potentially leading to a suppressed effect of the relationship between violent video game play and aggressive behavior.

In addition, there were inconsistencies and errors within the coded data set. For instance, we discovered that some games that did have violent content badges (such as Clash of Clans, Civilization, The Witcher, No Man's Sky, and The Legend of Zelda) were incorrectly coded as not having violent content. Several times, the same game (e.g., The Legend of Zelda: Breath of the Wild) was coded as violent by the coder in some instances but as non‐violent in others. All of these considerations led to a concern that the coding of violent video game play may not have been adequate to sensitively and accurately test the focal research question.

6. Our Study 1 Questions and Results

6.1. Remove BPAQ: Inappropriate Control Variables

As noted earlier, including BPAQ subscales is an example of using inappropriate covariates, or overcontrol. The original article reported that time spent playing violent video games was not a significant predictor of “aggressive behavior” (SDQ—conduct problems), both for the linear (β = 0.01, t(741) = 0.40, p = 0.69) or parabolic (i.e., quadratic) (β = 0.03, t(740) = 0.49, p = 0.62) video game violence exposure terms, when both gender and 4 BPAQ subscales were statistically controlled. However, the BPAQ includes measures of the conceptual dependent variable (aggressive behavior) and of theoretically meaningful mediating variables (anger and hostility). What is the effect of removing the BPAQ subscales from the regression analysis on the association between violent game exposure and Conduct Problems? ³

The first row of results in Table 1 (Model 0) is the video game violence exposure effect (VGV) reported in the original article. As can be seen in Model 1 of Table 1, removing BPAQ scores from the regression equation yielded a significant linear effect of time spent playing violent video games on the original measure of Conduct Problems (β = 0.08, t(745) = 2.14, p = 0.03). This VGV beta is eight times larger than the corresponding beta when BPAQ aggressive personality is inappropriately overcontrolled. Interestingly, even this poor measure of aggressive behavior yielded a significant effect of video game violence, once the overcontrol problem was fixed.  ⁴

Table 1.

Effects of poor aggressive behavior measure, dependent‐variable overcontrol, and mediating‐variable overcontrol on the violent video game effect on aggressive behavior, Gender in all models.

Model	Non‐gender control variables	Dependent variable	PWvgv B	PWvgv β	t(741–745)	p ≦
PW(2019)
0	4 BPAQ subscales	SDQ‐CP	0.01	0.01	0.40	0.69
Test VGV effects on DVs without BPAQ “controls”
1	None	SDQ‐CP	0.07	0.08	2.14*	0.03
2	None	BPAQ	0.15	0.09	2.34*	0.02
3	None	BPAQp	0.05	0.10	2.79*	0.01
4	None	BPAQp+v	0.04	0.09	2.56*	0.01
5	None	TPA	0.07	0.10	2.63*	0.01
Test VGV effects on DVs with “controls”
6	BPAQ	SDQ‐CP	0.01	0.01	0.40	0.69
9	Trait Anger, Trait Hostility	BPAQp	0.02	0.04	1.66	0.10
7	Trait Anger, Trait Hostility	BPAQp+v	0.02	0.02	1.26	0.21
8	Trait Anger, Trait Hostility	TPA	0.03	0.04	1.64	0.10

Note: Independent Variable: PWvgv = original Przybylski and Weinstein linear video game violence measure. BPAQ: Buss Perry Aggression Questionnaire subscales—Trait Anger, Trait Hostility, Trait Verbal Aggression (v), Trait Physical Aggression (p). SDQ‐CP: Strength and Difficulties Questionnaire conduct problems subscale. TPA = 3 Buss‐Perry physical aggressive behavior items and 1 CP aggressive behavior item.

^* p < 0.05. N = 748.

6.2. Using Better Aggressive Behavior Measures as the Dependent Variable

What is the effect of using more appropriate dependent variable measures of aggression, from the original data set? There are four obvious choices of better indicators of aggressive behavior: (a) trait aggressive personality (BPAQ), a composite of all 12 BPAQ items; (b) trait aggressive behavior (BPAQp+v), a composite of the 6 BPAQ trait physical and verbal aggression items; (c) trait physical aggression (TPA), a composite of the three BPAQ physical aggression items and the one CP physical aggression item; and (d) BPAQp, the three‐item BPAQ trait physical aggression subscale. ⁵

6.2.1. BPAQ Total: Trait Aggressive Personality

The 12 BPAQ short‐form items were averaged to create a composite score of aggressive behavior (M = 2.44, SD = 1.01, a = 0.95). As shown in Model 2, playing violent video games was positively associated with total aggressive personality. This replicates our finding for SDQ‐CP and contradicts the conclusions of Przybylski and Weinstein (2019) ⁶ .

6.2.2. BPAQp + V: Trait Aggression

This DV combined the trait physical and trait verbal subscales of the BPAQ (6 items, M = 2.38, SD = 1.05, a = 0.91). As shown in Model 3, this yielded a significant positive linear relationship between violent video game play and trait aggressive behavior. Despite having fewer items than the 12‐item BPAQ score, the effect was slightly larger for this purely behavioral measure.

6.2.3. TPA: Trait Physical Aggression #1

We created a measure combining the three‐trait physical aggression subscale of the BPAQ with the SDQ item “Often fights with other children and bullies them” (all four items were standardized before creating this composite, a = 0.84) ⁷ . Once again, the linear effect of playing violent video games was positive and statistically significant. As can be seen in Model 4, this effect was slightly larger than in previous models.

6.2.4. BPAQp: Trait Physical Aggression #2

This measure includes only the three trait physical aggression items from the BPAQ. As can be seen in Model 5, the VGV effect on aggression was significant and the largest of the five measures.

6.3. Effects of Including Theory‐Based Mediator Variables

As noted earlier, trait anger and trait hostility have been theorized and validated as mediating variables in the media violence to aggressive behavior effect. Therefore, partialling out their effects before testing the residual relation between habitual media violence exposure and trait aggressive behavior should reduce that relation. Doing so is inappropriate if one is testing the hypothesis that media violence exposure is positively related to aggressive behavior, rather than testing the mediation model. The remaining models in Table 1 demonstrate how “controlling” for these mediating variables reduced the video game violence effect on aggressive behavior to nonsignificance, consistent with theory.

7. Our Study 2 Questions and Results

We created two new measures of VGV to see whether they changed the observed effects on aggressive behavior. One involved having two trained coders assess the violent content of participants' reported video games on a 7‐point scale. The other corrected errors in the original article's dichotomous PEGI ratings.

7.1. Expert Raters' Measure of Video Game Violence Exposure

We hypothesized that a new VGV measure (AAvgv) based on expert raters' continuous judgments of violent content, rather than based on a binary code of the PEGI violent content badge, would yield a significant association between violent game time and aggressive behavior. We expected that both the zero‐order correlations and the size of the regression effect would be at least slightly bigger for the new AAvgv score than the original PWvgv score, or the corrected CORvgv score (discussed later).

To test this, two trained undergraduate research assistants (RAs) coded the video games reported by the participants, after watching gameplay of the games on websites such as Twitch and YouTube. To assess training, RAs were instructed on how to complete the task and asked to do a small sample of cases until they achieved a reliable level of agreement. The RAs coded the games on a 1–7 scale, “How Often Do Characters Try to Physically Injure Players/Characters in the Game?” (1 = Never, 7 = All the Time). Overall, the raters demonstrated good inter‐rater reliability between them for all three games, Game 1 (N = 493, M = 8.38, SD = 4.65, a = 0.87), Game 2 (N = 505, M = 8.30, SD = 4.59, a = 0.89), Game 3 (N = 306, M = 9.07, SD = 4.48, a = 0.87). The two scores were averaged for each game.

These averaged scores were multiplied by the reported time spent playing the game: “About how many hours a day do you usually play?” Items were scored on a 9‐point Likert scale (1 = none at all, 9 = about 7 h a day). This created a time spent playing violent games score for each of the three games, and those were averaged together to create a new total score of time spent playing violent games: AAvgv (for correlations of these variables, see supplemental materials on OSF). Table 2 shows the results of these tests with gender included as a covariate in the regression models with CP, Trait Physical, BPAQp+y, and TPA as individual predictors ⁸ .

Table 2.

Effects of new AAvgv measure on Aggression, Gender in the model.

Model	Non‐gender control variables	Dependent variable	AAvgv B	AAvgv β	t(740‐744)	p =
Test VGV effects on DVs without BPAQ “controls”
1	None	SDQ‐CP	0.01	0.04	1.00	0.32
2	None	BPAQ	0.04	0.09	2.26*	0.02
3	None	BPAQp	0.01	0.09	2.44*	0.02
4	None	BPAQp+v	0.03	0.10	2.57*	0.01
5	None	TPA	0.02	0.08	2.17*	0.03
Test VGV effects on DVs with “controls”
6	BPAQ	SDQ‐CP	−0.01	−0.03	−0.88	0.38
7	Trait Anger, Trait Hostility	BPAQp	0.01	0.03	1.29	0.20
8	Trait Anger, Trait Hostility	BPAQp+v	0.01	0.03	1.73	0.08
9	Trait Anger, Trait Hostility	TPA	0.01	0.03	1.04	0.30

Note: Independent Variable: PWvgv = original Przybylski and Weinstein linear video game violence measure. BPAQ: Buss Perry Aggression Questionnaire subscales—Trait Anger, Trait Hostility, Trait Verbal Aggression (v), Trait Physical Aggression (p). SDQ‐CP: Strength and Difficulties Questionnaire conduct problems subscale. TPA = 3 Buss‐Perry physical aggressive behavior items and 1 CP aggressive behavior item.

^* p < 0.05. N = 748.

7.1.1. Effects Without BPAQ Overcontrol Problem

The first five models tested the associations between our new VGV measure and key dependent measures with only gender as a covariate. As shown in the first five models in Table 2, the SDQ‐CP was not significantly correlated with VGV, but as expected, the other four measures of aggression were. Interestingly, the largest effect size was with the BPAQ physical + verbal aggressive behavior.

7.1.2. Effects of BPAQ Overcontrol

The second set of models in Table 2 tested the VGV effects with various BPAQ measures as covariates. None of these inappropriately overcontrolled models yielded a significant unique VGV effect. Of course, the fact that adding BPAQ trait anger and trait hostility as covariates (Models 7, 8, & 9) reduced the aforementioned significant VGV effects on aggressive behavior supports the theoretical model that views trait anger and trait hostility as mediating variables in the VGV‐aggressive behavior association.

7.2. Corrected Dichotomous PEGI Coding Scheme of Video Game Violence Exposure

As noted earlier, we discovered that the original research team's dichotomous PEGI coding variable contained mistakes. In short, the original violence video game variable (which we had previously named PWvgv) contained some errors in the dichotomous coding of whether the game contained violence or not. A research assistant was tasked with recoding the list of games to indicate the presence of the violence badge in the PEGI ratings (0 = the badge is not present, 1 = the violence badge is present). They also coded the game's PEGI and ESRB age ratings. For mobile games not rated by either, the RA based their ratings on the Google Play Store.

For each game and each participant, a difference score was calculated from both the new coder and the original coder. Thus, a score of zero indicates a match between the original and our new dichotomous score, whereas a 1 or −1 indicates a mismatch. There were 97 mismatches ⁹ , 18 for game one, 56 for game two, and 23 for game three. We created a corrected VGV score (CorVGV) by multiplying the corrected dichotomous violent content variable by the original game time variable. The N for this variable was 750, which indicates that the original coding scheme was consistent with our corrected scheme, and that the errors fixed did not exclude or include participants in any concerning systematic way ¹⁰ . Table 3 shows the results of these tests (with gender included as a covariate) in the regression models with CP, Trait Physical, and the Buss Perry Aggression Questionnaire as dependent variables ¹¹ .

Table 3.

Effects of video game violence exposure on aggression, based on the corrected original VGV measure (CORvgv), Gender in all models.

Model	Non‐gender control variables	Dependent variable	CORvgv B	CORvgv β	t(740–747)	p =
Test VGV effects on DVs without BPAQ “controls”
1	None	SDQ‐CP	0.06	0.07	1.97*	0.05
2	None	BPAQ	0.14	0.10	2.55*	0.01
3	None	BPAQp	0.05	0.11	3.00*	0.00
4	None	BPAQp+v	0.08	0.10	2.80*	0.01
5	None	TPA	0.06	0.10	2.80*	0.01
Test VGV effects on DVs with “controls”
6	BPAQ	SDQ‐CP	0.00	0.00	0.07	0.95
7	Trait Anger, Trait Hostility	BPAQp	0.02	0.05	1.88	0.06
8	Trait Anger, Trait Hostility	BPAQp+v	0.02	0.03	1.64	0.10
9	Trait Anger, Trait Hostility	TPA	0.02	0.04	1.65	0.10

Note: Independent Variable: PWvgv = original Przybylski and Weinstein linear video game violence measure. BPAQ: Buss Perry Aggression Questionnaire subscales—Trait Anger, Trait Hostility, Trait Verbal Aggression (v), Trait Physical Aggression (p). SDQ‐CP: Strength and Difficulties Questionnaire conduct problems subscale. TPA = 3 Buss‐Perry physical aggressive behavior items and 1 CP aggressive behavior item.

^* p < 0.05. N = 748.

7.2.1. Effects Without BPAQ Overcontrol Problem

The first five models tested the associations between this second new VGV measure and key dependent measures with gender as a covariate. As shown in the first five models in Table 3, all five outcome measures yielded statistically significant effects, with the smallest effect size being the poorest of the aggression measures, the SDQ conduct problems subscale. Interestingly, the largest effect size was with the BPAQ physical aggressive behavior.

7.2.2. Effects with BPAQ Overcontrol Problems

The second set of models in Table 3 tested the VGV effects with various BPAQ measures as covariates. None of these inappropriately overcontrolled models yielded a significant unique VGV effect. Of course, the fact that adding BPAQ trait anger and trait hostility as covariates (Models 7, 8, & 9) reduced the aforementioned significant VGV effects aggressive behavior supports the theoretical model that views trait anger and trait hostility as mediating variables in the VGV‐aggressive behavior association.

7.2.3. Study 2 Overall Results

These results from the regression and the correlational analyses supported our hypotheses that the new CORvgv and AAvgv variables were significant predictors of aggressive behavior. They also showed that the SDQ conduct problems subscale is not a good measure of habitual aggressive behavior, and that inappropriate overcontrol was necessary to finding a nonsignificant association between video game violence exposure and trait aggressive behavior. However, the new AAvgv and CORvgv variables did not produce significant increases in the obtained effect sizes compared to the original PWvgv variable.

8. Discussion

It is clear from our analyses and results that two of our 3 concerns outlined earlier were indeed warranted. Figure 1 shows a comparison of the standardized betas for some of the key analyses, taken from Tables 1 and 3. It illustrates how our first and third concerns were justified. The first concern, about using the SDQ‐CP as a measure of aggressive behavior, is valid for both theoretical and empirical reasons (e.g., four of the five items do not assess aggressive behavior). In addition, that measure led to the smallest effect of VGV measures on the five “aggression” measures tested (see Table 1). The purest measures of the type of aggression that is most relevant to what is most modeled in violent games—physical aggression—yielded the largest effect sizes. This same pattern occurred when using our two new measures of VGV (Tables 2 and 3).

Figure 1.

Comparison of standardized beta size across key regressions. *p < 0.05. **p < 0.01. ***p < 0.001. N = 748. [Color figure can be viewed at wileyonlinelibrary.com]

The second area of concern, using a potentially weak measure of video game violence exposure, was both confirmed and disconfirmed. It was disconfirmed in that our new measure (AAvgv) did not yield consistently larger effect sizes on the aggressive behavior measures than did the original variable (PWvgv, compare Table 2 with Tables 1 and 3). However, our hypothesis was confirmed in that a corrected version of the original IV (CORvgv) yielded larger effect sizes and statistically significant effects.

The third concern, about overcontrolling for the dependent variable (i.e., BPAQ trait physical and verbal aggression) and for two mediating variables (BPAQ trait anger and hostility), was wholly validated. Regardless of VGV measure and of “aggression” measure, overcontrol led to decreases in the VGV effect, making it appear nonsignificant in all cases. Of course, either type of overcontrol (controlling for the dependent variable, controlling for mediating variables) should (theoretically and statistically) decrease the apparent VGV effect. If the basic effect is small, or if the mediators are strongly predictive, then the residual VGV effect on aggression should be reduced to nonsignificant levels. Figure 1 shows comparisons of some of the most relevant models.

One additional concern that arose during the course of our inquiries and investigation is the unavailability of details concerning recruitment materials, consent form description of the study, and specific instructions to participants. Given that many gamers ardently deny any potential harmful effects of violent games and that they bias their responses on obvious measures of harm (relative to nonobvious measures; see Bender et al. ²⁰¹³) we cannot help wondering if suspicions about or obviousness of the purpose of the key measures might have depressed the obtained effect sizes.

To reiterate, our goal for this article was not to cause controversy over any individual study. Rather, it was to test hypotheses (e.g., Bushman and Anderson ²⁰²³; Kim et al. ²⁰²¹) about potential methodological/statistical causes of null results in the media effects literature, causes that are prevalent in the literature. This is where open science practices become extremely important. We sincerely thank Dr. Przybylski for making these data available and for helping us understand and use the data set. He has been very generous with his time and thoughtful in his replies to our inquiries.

Finally, another consideration of this study is the potential implication of rigor that is commonly associated with open‐science practices. While open‐science is a helpful and necessary evolution of the scientific process, there cannot be a heuristic associated that simply being “open‐science” equates to “better” or “more valid” research results, methodology, and interpretations. This is something modern scientists should be incredibly critical of, especially during this transitory period where many publications, research groups, and disciplines are developing their own practices and standards in this evolving landscape.

While the push for “open science” has led to some incredible revolutions in transparency and access to scientific results and materials, other potentially predatory or exploitative practices, such as charging exorbitant fees to make data and publications publicly available, have begun to limit practicality of publishing research results for those who do not have access to substantial funding (see Poynder ²⁰¹⁴ for discussion on the costs of publication and open‐access). Thus, by conflating open‐science practices (such as being awarded an open science badge for choosing to pay an open science fee), with “better” research, we may effectively see a stratification and monetary‐based hierarchy of “valued” and visible research. We hope this study shows the immense promise of open‐science as means of creating a new level of verification and discussion amongst scholars in a specific field. However, we caution readers that a study simply being “open‐access” will not address core methodological, theoretical, and analytical concerns—or address deeper issues of equity within scientific publishing.

Ethics Statement

This study was based on data gathered from an IRB‐approved study; the original manuscript, materials, and ethical approvals can be acquired on Open Science Framework: https://osf.io/rkw6z/.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

This study was preregistered, and data are available on Open Science Framework: https://osf.io/ney9t/.