Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Jun 1.
Published in final edited form as: J Adolesc. 2018 Mar 21;65:101–110. doi: 10.1016/j.adolescence.2018.03.007

PROFILES OF INTERNALIZING AND EXTERNALIZING SYMPTOMS ASSOCIATED WITH BULLYING VICTIMIZATION

Meridith Eastman 1, Vangie Foshee 1, Susan Ennett 1, Daniela Sotres-Alvarez 2, H Luz McNaughton Reyes 1, Robert Faris 3, Kari North 4
PMCID: PMC5932115  NIHMSID: NIHMS953888  PMID: 29573643

Abstract

This study identified profiles of internalizing (anxiety and depression) and externalizing (delinquency and violence against peers) symptoms among bullying victims and examined associations between bullying victimization characteristics and profile membership. The sample consisted of 1196 bullying victims in grades 8–10 (Mage=14.4, SD=1.01) who participated in The Context Study in three North Carolina counties in Fall 2003. Five profiles were identified using latent profile analysis: an asymptomatic profile and four profiles capturing combinations of internalizing and externalizing symptoms. Associations between bullying characteristics and membership in symptom profiles were tested using multinomial logistic regression. More frequent victimization increased odds of membership in the two high internalizing profiles compared to the asymptomatic profile. Across all multinomial logistic regression models, when the high internalizing, high externalizing profile was the reference category, adolescents who received any type of bullying (direct, indirect, or dual) were more likely to be in this category than any others.

Keywords: bullying victimization, latent profile analysis, depression, anxiety, delinquency

Bullying is a pervasive public health problem. Between 20–40% of children experience bullying at least once during their school careers (Shetgiri, Lin, & Flores, 2013). Numerous studies have found that bullying victimization has negative effects in childhood and adolescence and, furthermore, evidence suggests that victims of bullying are at heightened risk for psychological maladjustment in adulthood (Arseneault, Bowes, & Shakoor, 2010; Copeland, Wolke, Angold, & Costello, 2013; Hawker & Boulton, 2000; Menard, 2002).

Not all victims respond to bullying the same way (Arseneault, Bowes, & Shakoor, 2010). For some, being bullied may result in internalizing problems (i.e., those harmful to self). These negative impacts include a range of deleterious mental health conditions including heightened social isolation, depression, and anxiety (Faris & Felmlee 2014; Kaltiala-Heino, Fröjd, & Marttunen, 2010; Nansel et al., 2001; Reijntjes, Kamphuis, Prinzie, & Telch, 2010; Zweirzynska, Wolke, & Lereya, 2013). Others may experience externalizing sequelae (i.e., those harmful to others). These include violent behavior towards others, carrying a weapon, and becoming a perpetrator of bullying behaviors (Arseneault et al., 2006; Barker, Arseneault, Brendgan, Fontain, & Maughan, 2008; Kim, Leventhal, Koh, Hubbard, & Boyce, 2006; Nansel, Overpeck, Haynie, Ruan, & Scheidt, 2003; Reijntjes et al., 2011; Ttofi, Farrington, & Lösel, 2012; Valdebenito, Ttofi, Eisner, & Gaffney, 2017). Another set of victims may experience both types of symptoms simultaneously (Arseneault, et al., 2010). For example, Hemphill and colleagues (2011) found that being victimized by bullying in grade 10 predicted a twofold increased likelihood of depressive symptoms as well as increased likelihood weapon-carrying, theft, and violent behavior in grade 11 (Hemphill et al., 2011). Lastly, some victims of bullying show very few signs of internalizing or externalizing and this has been attributed to the presence of other sources of support in (e.g., social connections, family relationships) in the victims’ lives (Sapouna & Wolke, 2013).

This variation in response to bullying victimization suggests that there may be different typologies, or profiles, of symptoms associated with victimization. Many studies examining psychopathology associated with bullying victimization focus on either internalizing or externalizing symptoms. Such variable-centered approaches focusing on a single outcome ignore underlying heterogeneity in symptoms among bullying victims and exclude more complex symptom constellations that incorporate internalizing and externalizing elements. By grouping individuals into categories based on similarity with one another and differences from those in other categories, person-centered approaches can unmask this underlying heterogeneity to reveal group differences in symptom patterns (Laursen & Hoff, 2006;Muthén & Muthén, 2000).

Latent profile analysis (LPA) is a person-centered statistical approach used to determine the number of subpopulations—or profiles--that exist for a given set of indicators, with what probability each individual is in each profile, and which variables are most strongly associated with profile membership (Collins & Lanza, 2010). Kretschmer and colleagues (2015) conducted a latent profile analysis of maladjustment in a sample of students in early-mid adolescence and assessed the influence of peer victimization on membership in the latent profiles, finding that peer victimization increased risk for internalizing problems. Their latent profile analysis, however, was conducted on a sample that included both victims and non-victims limiting the ability to describe victims’ maladjustment. For example, it is possible that the nature of the maladjustment profiles of victims could be distinct from the nature of the profiles of non-victims. The focus of the present study, therefore, is solely on the victims of bullying. Furthermore, we build upon the work of Kretschmer and colleagues by using more finely grained measures of bullying that allow for evaluation of associations between type and frequency of bullying and the identified profiles. The aims of this study were to 1) determine whether there are different profiles of internalizing (depression and anxiety) and externalizing (peer violence perpetration and delinquency) symptoms in a sample of 8th-10th grade self-identified victims of bullying and 2) examine whether characteristics of the bullying victimization (i.e., type of bullying and frequency) are associated with symptom profile membership. The goal of these aims was to identify symptom co-morbidity in victims of bullying and to understand whether bullying characteristics are associated with these symptoms.

Bullying types are defined based on the mode through which harm is perpetrated against the victim. Although numerous terms have been used in the literature to describe types of bullying (e.g. physical, verbal, overt, covert, relational), the terms “direct” and “indirect” bullying, the terms used in this study, capture this variation. Direct bullying includes anything requiring direct interaction between the bully and victim such as physical acts of aggression and name calling. Indirect bullying comprises spreading rumors, attempts at social exclusion, and talking behind the victim’s back (Arseneault et al., 2010; Dukes et al., 2009). In variable-centered studies, bullying victims’ symptoms have been found to differ depending on the type of bullying experienced. Specifically, a 2017 meta-analysis by Casper and Card found direct victimization to be more strongly associated with direct aggression, whereas relational victimization was more strongly related to internalizing problems, underscoring the importance of measuring these different types of victimization and understanding their differential associations with psychopathology (Casper & Card, 2017).

Victims’ symptoms have also been found to vary with the frequency of victimization. Frequency of bullying is defined here as the number of times a person has been bullied over a reference time period. Penning et al. (2010) found that frequency of being bullied (with no distinction made between bullying types) was associated with higher mean scores on five trauma subscales (anxiety, depression, posttraumatic stress, dissociation, and anger) of the Trauma Symptom Checklist for Children (TSCC). Similarly, Klomek and colleagues (2009) found that the more frequent involvement in bullying (either as a victim or a perpetrator; no distinction made between bullying types), the more likely an individual was to be depressed, to have serious suicide ideation, or to have attempted suicide (Klomek et al., 2009). Champion and Clay (2007) also found that more frequently victimized children responded to victimization with more intense feelings of anger, more motivation to retaliate, less motivation to improve the situation, and more frequent intentions to aggress (Champion & Clay, 2007). Taken together, these studies suggest that greater frequency of victimization is associated with more intense internalizing and externalizing symptoms among victims.

Hypotheses

Based on extant literature on internalizing and externalizing symptoms among bullying victims as well as Kretschmer and colleagues’ (2015) maladjustment profiles in early adolescence, we hypothesized (Hypothesis 1) that the following profiles would be identified through LPA: one profile low on both internalizing and externalizing (an asymptomatic profile, corresponding to Kretschmer et al.’s “Low” profile), one profile high on internalizing and low on externalizing (corresponding to Kretschmer et al.’s “Internalizing” profile), one profile low on internalizing and high on externalizing (corresponding to Kretschmer et al’s “Externalizing” profile), and one profile high on both internalizing and externalizing (corresponding to Kretschmer et al.’s “Comorbid” profile).

We hypothesized (Hypothesis 2) that adolescents who experienced any direct victimization would have a greater likelihood of membership in profiles characterized by high externalizing symptoms than in profiles not characterized by high externalizing symptoms. This hypothesis is supported by the empirical literature meta-analyzed by Casper and Card (2017), but also by Social Cognitive Theory (SCT) which posits that individuals model behavior they witness and experience in their social contexts (Bandura, 1986). If an adolescent experiences direct bullying, SCT suggests that he/she may copy this behavior and respond by victimizing peers or externalizing in some other way. Furthermore, Casper and Card (2017) suggest a unique association between risk-taking behavior, lack of impulse control, and direct victimization such that victims of direct aggression are likely to respond in kind to direct attacks.

In contrast, we hypothesized (Hypothesis 3) that adolescents who experienced any indirect victimization would have greater likelihood of membership in profiles characterized by high internalizing symptoms than in profiles characterized by low internalizing symptoms. This hypothesis is supported by the empirical literature (e.g., Casper & Card, 2017), but also reflects the subtle nature of indirect bullying. Because the harm is perpetrated not via direct attack, but rather through manipulation of social relationships, an effective external target for response may be difficult to identify. Direct confrontation with the perpetrator even if a perpetrator is identifiable, for example, would not necessarily be effective in extinguishing a socially harmful rumor. Without an effective external target, frustrated victims may internalize the experience, leading to depression and anxiety. Additionally, loneliness, which has been correlated with depression and lower self-worth, may result from the social exclusion concomitant with indirect aggression (Prinstein, Boergers, & Vernberg, 2001). Victims of indirect aggression may experience a vicious cycle wherein internalizing problems and fewer opportunities to develop appropriate social skills elicit further victimization and self-blame (Juvonen & Graham, 2014).

Reflecting Hypotheses 2 and 3 we further hypothesized (Hypothesis 4) that adolescents who experienced dual victimization (i.e., both direct and indirect victimization) would have greater likelihood of membership in the profile characterized by high internalizing and high externalizing than in the other profiles, thereby exhibiting victim characteristics of both direct and indirect bullying.

Whereas previous LPA analysis (i.e., Kretschmer et al., 2015) has used a dichotomous bullying measure to identify victimization over the past 2 years, the present study assessed the frequency with which victims experienced bullying over the measurement period, enabling us to measure the effects of frequency—regardless of bullying type—on symptom profile. Based on literature suggesting an intensifying effect of increased frequency of bullying victimization on internalizing and externalizing symptoms (or vice versa), we hypothesized (Hypothesis 5) that greater frequency of victimization would be associated with greater likelihood of membership in the symptomatic profiles than in the asymptomatic profile.

Method

Data were collected on paper questionnaires in the Fall 2003 from adolescents in the public school systems of three primarily rural counties in North Carolina as part of The Context Study on adolescent risk behaviors which has been described in detail elsewhere (Ennett et al. 2008; Foshee et al., 2011). All 6342 adolescents in grades 8 to 10 in these school districts (eight middle schools, two K-8 schools, three alternative schools and six high schools) were eligible to participate except for those who were unable to complete the questionnaire in English, in special education programs, or who were in long-term suspension or expulsion. A total of 5017 adolescents (79.1% of those eligible) completed the questionnaire. 6.3% of eligible students were absent the day of the questionnaire, 7.9% had parents who refused, 4.4% of eligible students refused prior to administration, and 2.3% of eligible students could not be contacted (i.e., no viable address was available to obtain parent consent). Parents had the opportunity to refuse consent for their child’s participation by returning a written form or by calling a telephone number. Assent was obtained from all individual adolescents included in the study immediately prior to the survey administration from students whose parents had consented. The Institutional Review Board for the University of North Carolina at Chapel Hill approved the study.

Analytic Sample

At the time of questionnaire administration, data collectors provided each student with a Student Directory that alphabetically listed students along with a unique four-digit peer identification number for each student. Bullying victimization was assessed by asking students to identify up to five peers who had been mean to them or who had picked on them in the past 3 months. This question deliberately excluded the term “bullying” because youth tend to hold stereotypes of bullying as direct physical violence (Naylor, Cowie, Cossin, Bettencourt & Lemme, 2006). Previous research using these data found that this measure of victimization is associated with subsequent increases in anxiety and depression, and decreases in school attachment and social integration (Faris & Felmlee, 2014).

The analytic sample was limited to the 1196 adolescents (23.8% of those who completed questionnaires) who indicated that any school peer had bullied them. The sample was 59.8% female. Fifty-six and nine tenths percent reported their race as White, 27.5% Black or African-American, 4.1% Hispanic or Latino, 2.4% American Indian or Native American, 1.3% Asian or Pacific Islander, 4.7% Multiracial (mixed race), and 1.7% Other (total 41.7% Non-white). Mean age was 14.4 years (SD=1.01).

Measures

Anxiety

Four items from the Revised Children’s Manifest Anxiety Scale (Reynolds & Richmond, 1979) assessed anxiety (e.g., “I worried about what was going to happen” and “I worried when I went to bed at night”) within the past three months. Responses ranged from 0 (“strongly disagree”) to 4 (“strongly agree”). Responses for the four anxiety items were summed to create the anxiety score (Cronbach’s α =.86, M=8.34, SD=4.95, range=0–16);

Depression

Four items from the Short Mood and Feelings Questionnaire (Angold, Costello, Messer, & Pickles, 1995) assessed feelings of depression (e.g., “I did everything wrong” and “I was tired a lot”). Responses for the four depression items were summed to create the depression score (Cronbach’s α= .86, M=6.47, SD=4.96, range=0–16).

Delinquency

Delinquency was measured with four items that asked the respondent to report the frequency with which he or she skipped school, damaged property, threatened a teacher, or threatened someone with a weapon (Farrell, Kung, White, & Valois, 2000). Response options were: 0=none; 1=1–2 times; 2=3–5 times; 3=6–9 times; and 4=10 times or more. Responses to these items were summed to create a composite delinquent behaviors score (Cronbach’s α =.80, M=1.16, SD=2.77, range=0–16).

Violence against Peers

Physical violence against peers was measured with six items that asked the respondent to report how often in the past 3 months he or she pushed, grabbed, shoved, or kicked a peer; slapped or scratched a peer; twisted a peer’s arm or bent back a peer’s fingers; hit a peer with a fist or with something else hard; beat up a peer; or assaulted a peer with a knife. Response options were: 0=none; 1=1–2 times; 2=3–5 times; 3=6–9 times; and 4=10 times or more. These six items were summed to create a composite physical violence against peers score (Cronbach’s α=0.88, M=1.93, SD=4.18, range=0–24).

The distributions of the delinquency and violence against peer scores were heavily right-skewed, violating the assumption of normality required for LPA. The indicators were trichotomized such that 0=none, 1=some, and 2=a lot of externalization. Cutoffs for the categories were based on univariate statistics so that the “a lot” category captured individuals at approximately the 90th percentile for the outcome and above, the “none” category consisted of individuals reporting no externalization, and individuals with scores between 0 and the approximate 90th percentile cutoff fell into the “some” category.

Direct bullying victimization

Adolescents were asked to indicate whether each student identified as someone who had been mean to them or picked on them in the past 3 months had “physically attacked you in any way (hitting, shoving, tripping)?”, “made fun of you or called you names to your face,” and or “talked badly about you behind your back or tried to get others not to be friends with you”. A dichotomous direct bullying victimization type variable was created such that 1 indicated that a peer had physically attacked them, made fun of them, or called them names to their face and 0 indicated that a peer had not done these things to them. Direct victimization was reported by 87.7% of bullying victims.

Indirect bullying victimization

A dichotomous indirect bullying victimization type variable was created such that 1 indicated that a peer had talked badly about them behind their back or tried to get others not to be friends with them and 0 indicated that a peer had not done these things to them. Indirect victimization was reported by 75.8% of bullying victims.

Dual bullying victimization

A dichotomous dual victimization type variable was created where 1 indicated that the adolescent was both directly and indirectly bullied and 0 indicated that the adolescent experienced only one type of victimization. Dual victimization was reported by 63.6% of bullying victims.

Bullying frequency

Adolescents were asked to indicate the frequency with which each identified peer was mean to or picked on them. Response categories included: 1=2 times in the past 3 months, 2=1 to 2 times per month, 3=1 to 2 times per week, 4=3 to 5 times per week, and 5=6 or more times per week. Frequency of bullying victimization was calculated by summing the frequency of victimization across all identified peers (up to five). The bullying victimization frequency ranged from 1 to 25; M=8.72, SD=6.75.

Analysis Strategy

The analytic approach consisted of two major steps. First, LPA was conducted in Mplus 7 using the expectation maximization algorithm with the robust maximum likelihood (MLR) estimator for the four indicators of anxiety, depression, delinquency, and physical violence against peers. Missing data were handled using full information maximum likelihood (FIML). A one-profile model was estimated first, followed by a two-profile model, and additional profiles were added sequentially until there was no improvement in model fit. Several criteria were used to evaluate the fit of latent profile models: Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), sample size adjusted BIC, bootstrapped likelihood ratio test (BLRT), Vuong-Lo-Mendell-Rubin (VLMR) likelihood ratio test, and entropy. After the number and nature of the profiles were identified, individuals were assigned to their most likely profile based on their posterior probabilities (that is, the set of values describing the likelihood of being assigned to that profile, given the data).

The second step was to conduct a series of multinomial logistic regressions using SAS v9.4 to test the hypotheses regarding associations between bullying type (direct, indirect, or dual) and frequency with symptom profile membership. All models controlled for gender, race, grade, and parent education. Because some studies have found that the consequences of bullying victimization vary by gender (e.g. Sullivan, Farrell, & Kliewer, 2006), we first tested for significance of interactive effects of gender and bullying characteristics on profile membership. In all cases, the interactions were non-significant; therefore, interactions with gender were dropped from subsequent models. Rather than selecting a single reference profile for all models, the reference profile for the logistic regression models necessarily varied according to the hypothesis being tested. In all cases, resulting odds ratios represent the likelihood of membership in each profile relative to the specified reference profile. Missing data for the covariates used in the multinomial logistic regressions were imputed using PROC MI, and PROC MIANALYZE was used to pool the results from the models fit on the imputed datasets.

Results

Latent Profile Analysis of Internalizing and Externalizing Symptoms

Table 1 shows parameters of model fit for the 1–6 profile models that were tested (Hypothesis 1). Because the AIC, BIC, LRT results for the two-profile model indicated improved fit over the one-profile model, a three-profile model was estimated, and so on, up to a six-profile model. Note, however, that the entropy value, an indicator of the amount of separation between profiles, was worse in the six-profile model as compared to the five-profile model despite improved AIC, BIC, and LRT results. This suggests that the five-profile model had better separation between profiles than the six-profile model. The five-profile model also demonstrated superior interpretability; the six-profile model did not provide conceptually meaningful distinctions because it produced multiple profiles with very similar moderate amounts of internalizing and externalizing symptoms. Taking together the parameters of fit and these conceptual considerations, the five-profile model was determined to best represent the data.

Table 1.

Parameters of fit for 1 −6 profile solutions for internalizing and externalizing symptoms

# of Profiles AIC BIC Sample-size
adjusted BIC		 LL p-value for
BLRT		 p-value for
VLMR
LRT		 Entropy
1 17204.014 17244.708 17219.297 −8594.007 -- -- --
2 16623.936 16700.237 16652.591 −8296.968 <0.0001 0.0001 .678
3 16266.324 16378.232 16308.352 −8111.162 <0.0001 0.0001 .741
4 16134.471 16281.986 16189.871 −8038.235 <0.0001 0.0008 .751
5 15925.759 16108.881 15994.532 −7926.879 <0.0001 0.0001 .843
6 15878.925 16097.655 15961.071 −7896.463 <0.0001 0.0060 .808
Open in a new tab

Note: Aikake Information Criterion (AIC; Akaike, 1974) and Bayesian Information Criterion (BIC; Schwarz, 1978) and sample-size adjusted BIC (Tofighi & Enders, 2007) are relative fit statistics where lower numbers indicate improved model fit as compared to higher numbers. These statistics are based upon the log likelihood (LL)—the logarithm of the likelihood ratio--which expresses how many times more likely the data fit under a k profile model than a k-1 profile model. The p value for the bootstrapped likelihood ratio test (BLRT; McLachlan & Peel, 2000) and for the Vuong-Lo-Mendell Rubin (VLMR) likelihood ratio test (LRT; Lo, Mendell & Rubin, 2001) represent the results of tests that assess whether a model of k profiles represents a better fit for the data than a model of k-1 profiles. P values of <.05 indicate that the k profile model better suits the data than a model of k-1 profiles. Lastly, entropy is a criterion that measures classification certainty and can range from 0–1 (Celeux & Soromenho, 1996). Higher values indicate that profiles have good separation; that is, that profiles are more distinct from one another. An entropy value of .80 represents good separation between the profiles (Ramaswamy et al, 1993).

Note: Best fitting model in bold.

The five profiles showed distinct characteristics (see Table 2) and we describe them according to level of symptoms (lowest to highest). Profile 1, named “asymptomatic,” consisting of 27% of the sample, had the lowest mean levels of anxiety and depression, and its members had the greatest probability of reporting no externalizing behaviors. Profile 2 was the second largest profile, consisting of 24.8% of the sample. This profile showed moderately high levels of anxiety, yet comparatively low levels of depression. These adolescents, in the “moderate anxiety, moderate externalizing” profile, had medium probabilities of reporting “some” delinquency and violence against peers and low probabilities of reporting “a lot” of externalizing behaviors. Profile 3 consisted of 22.4% of the sample and reflects internalizing scores that are in the middle when compared to other profiles and levels of externalizing behaviors similar to Profile 2. These “moderate internalizing, moderate externalizing” victims reported medium mean levels of depression and anxiety and medium probabilities of reporting “some” and “a lot” of delinquency and violence against peers. Profile 4, “high internalizing, moderate externalizing”, comprising 14% of the sample, consisted of adolescents reporting high levels of anxiety and depression with an appreciable probability of reporting “some” or “a lot” of externalizing behaviors. Lastly, Profile 5 was the smallest, consisting of 11.7% of the sample, and included adolescents with the highest mean levels of anxiety and depression and also the highest probabilities of reporting “a lot” of delinquency and violence against peers. Profile 5, therefore, was named “high internalizing, high externalizing”.

Table 2.

Profile prevalences, means (for internalizing symptoms), item response probabilities (for externalizing symptoms), and classification probabilities for the 5 symptom profiles

Profile 1:
Asymptomatic		 Profile 2:
Moderate
anxiety,
moderate
externalizing		 Profile 3:
Moderate
internalizing,
moderate
externalizing		 Profile 4:
High
internalizing,
moderate
externalizing		 Profile 5:
High
internalizing,
high
externalizing
N(%) 323 (27.0) 297(24.8) 269 (22.4) 167(14.0) 140(11.7)
Anxiety 2.75 8.11 9.16 11.69 13.76
Depression .38 3.76 7.80 11.26 15.40
Delinquency
   None .76 .66 .66 .57 .46
   Some .16 .26 .23 .34 .23
   A lot .08 .08 .11 .10 .31
Violence
   None .71 .62 .64 .56 .57
   Some .23 .30 .24 .31 .25
   A lot .07 .09 .12 .13 .19
Classification probability .95 .86 .86 .88 .92
Open in a new tab

Associations between Bullying Type and Frequency with Symptom Profiles

Table 3 summarizes the results from the five multinomial logistic regression models used to test Hypotheses 2–5, with each line representing results from a different model.

Table 3.

Adjusted associationsa between bullying victimization characteristics and symptom profile membership; results from five multinomial logistic regression models used to test Hypotheses 2-5, with each line representing results from a different model

Victimization
characteristic		 Profile 1 Profile 2 Profile 3 Profile 4 Profile 5
OR 95% CI OR 95% CI OR 95% CI OR 95% CI OR 95% CI
Directb .30 (.13,.68)** .28 (.13,.63)** .27 (.12,.60)** .41 (.17,.97)* Reference
Indirectc .80 (.50,1.28) .66 (.42,1.06) .81 (.50,1.32) Reference --
Indirectb .51 (.30,.87)* .42 (.25,.72)** .52 (.30,.90)* -- Reference
Dualb .43 (.27,.69)** .37 (.23,.60)*** .42 (.26,.68)** .56 (.33,.94)* Reference
Frequencyd Reference 1.01 (.99,1.04) 1.02 (.99, 1.05) 1.04 (1.01,1.07)** 1.09 (1.06,1.12)***
Open in a new tab

Note: Profile 1= Asymptomatic; Profile 2= Moderate anxiety, moderate externalizing; Profile 3= Moderate internalizing, moderate externalizing; Profile 4= High internalizing, moderate externalizing; Profile 5= High internalizing, high externalizing.

a

Adjusted odds ratios (OR) estimated from a multinomial logistic regression of symptom profile membership on specified bullying victimization characteristic adjusting for gender, race, parental education, and grade. The model testing association between direct victimization and symptom profile membership also controls for indirect victimization. The models testing associations between indirect victimization and symptom profile membership also control for direct victimization.

b

Reference is Profile 5, high internalizing, high externalizing.

c

Reference is Profile 4, high internalizing, moderate externalizing.

d

Reference is Profile 1, asymptomatic

*

p<05

**

p<01

***

p<.0001

Direct victimization

Our first multinomial logistic regression model tested Hypothesis 2 that adolescents who experienced any direct victimization would have a greater likelihood of membership in profiles characterized by high externalizing symptoms than in profiles not characterized by high externalizing symptoms. This hypothesis required comparing Profile 5 (high internalizing, high externalizing) against each of the other profiles. Profile 5, therefore, was used as the reference category in the regression model testing Hypothesis 2. As shown in the first line of Table 3, comparisons of each Profile with Profile 5 were significant. Adolescents who experienced direct victimization had 70% lower odds of membership in Profile 1, 72% lower odds of membership in Profile 2, 73% lower odds of membership in Profile 3, and 59% lower odds of membership in Profile 4 compared to Profile 5. Thus, Hypothesis 2 was supported.

Indirect victimization

The next set of regression models tested Hypothesis 3 that adolescents who experienced any indirect victimization would have greater likelihood of membership in profiles characterized by high internalizing symptoms than in profiles characterized by low internalizing symptoms. LPA identified two profiles with higher internalizing symptoms: Profile 4 (high internalizing, moderate externalizing) and Profile 5 (high internalizing, high externalizing). Three profiles have lower internalizing symptoms: Profile 1 (asymptomatic), Profile 2 (moderate anxiety, moderate externalizing), and Profile 3 (moderate internalizing, moderate externalizing). Our multinomial logistic regression, therefore, required two sets of contrasts: Profile 4 compared to Profile 1, 2, and 3 and Profile 5 compared to Profile 1, 2, and 3. As shown in line 2 of Table 3 and contrary to Hypothesis 3, indirect bullying victimization was not significantly associated with membership in any of the lower internalizing profiles when compared to Profile 4. However, as shown in line 3, there was a significant association between indirect victimization and membership in the lower internalizing profiles as compared to Profile 5. Adolescents who experienced indirect victimization had 49% lower odds of membership in Profile 1, 58% lower odds of membership in Profile 2, and 48% lower odds of membership in Profile 3 compared to Profile 5. Thus, Hypothesis 3 was partially supported.

Dual victimization

Our next multinomial logistic regression model tested Hypothesis 4, which was that adolescents who experienced both direct and indirect victimization would have greater likelihood of membership in the profile characterized by high internalizing and high externalizing (Profile 5) than in the other profiles. To do so, we fit a model with Profile 5 as the reference profile. Consistent with Hypothesis 4 and as shown in line four of Table 3, adolescents who experienced both types of victimization were indeed more likely to be in Profile 5 (high internalizing, high externalizing) than in any of the other symptom profiles. Compared to profile 5, adolescents who experienced both types of bullying had 57% lower odds of membership in Profile 1, 63% lower odds of membership in Profile 2, 58% lower odds of membership in Profile 3, and 44% lower odds of membership in Profile 4.

Frequency of victimization

Our last regression model tested Hypothesis 5 that greater frequency of victimization would be associated with greater likelihood of membership in all of the symptomatic profiles than in the asymptomatic profile. To test this hypothesis, Profile 1 (asymptomatic) was used as the reference category. There was only partial support for this hypothesis. As shown in line five of Table 3, frequency of victimization was significantly associated with membership in the high internalizing profiles (Profiles 4 and 5), with each one-unit increase in bullying frequency being equivalent to a 4% and 9% respective increase in odds of profile membership compared to Profile 1. Frequency of victimization did not increase odds of membership in Profiles 2 and 3, neither of which included high levels of internalizing.

Discussion

LPA results confirmed the presence of subgroups of symptoms within our sample of victims of bullying. In contrast to our hypothesized four profiles and the profiles found by Kretschmer and colleagues in a sample of victims and non-victims combined, the data supported a five-profile model consisting of a profile low on internalizing and externalizing (Profile 1), a profile of moderate anxiety, moderate externalizing (Profile 2), a profile of moderate internalizing and externalizing (Profile 3), a profile of high internalizing and moderate externalizing (Profile 4), and a profile characterized by high levels of internalizing and externalizing (Profile 5). The profiles that emerged reflect a more nuanced picture than the hypothesized four profiles, identifying more moderate levels of internalizing and externalizing symptoms and, in one case (Profile 2, moderate anxiety, moderate externalizing), distinguishing between the two types of internalizing symptoms measured in this study: anxiety and depression. Of note, internalizing and externalizing problems coexist in all profiles, underscoring the importance of secondary interventions that address both types of symptoms among victims of bullying. This is consistent with the only other study to use LPA to examine profiles of maladjustment to bullying, which found comorbidity of internalizing and externalizing symptoms in each profile (Kretschmer et al., 2015).

Overall, we found little support for the differentiating effects of indirect, direct, and dual victimization on symptom profile membership, despite support or partial support for the specific hypotheses tested. Thus, the findings contrast with metanalytic results by Casper and Card (2017) which found that indirect victimization is more strongly associated with internalizing whereas direct victimization is more strongly associated with externalizing maladjustment. Across all multinomial logistic regression models that specified Profile 5 (high internalizing, high externalizing) as the reference category, adolescents who received any type of bullying (direct, indirect, or dual) were more likely to be in this category than any others. Direct victimization over and above indirect victimization, indirect victimization over and above direct victimization, and both types of victimization in combination (i.e., dual victimization), were each significantly associated with membership in the highest symptom profile. There are a couple of potential interpretations of this finding. First, it is possible that Profile 5 represents a group particularly sensitive to bullying victimization of any type, perhaps due to biological liability or psychosocial vulnerability factors. Testing the influence of such factors as predictors of membership in the latent symptom profiles would illuminate this possibility. Second, it is possible that adolescents who exhibit high levels of internalizing and externalizing symptoms evoke bullying behavior of multiple types from their peers. A meta-analysis by Cook and colleagues (2010), suggests such a bidirectional relationship between displays of internalizing and externalizing behaviors and victimization (Cook, Williams, Guerra, Kim, & Sadek, 2010). The cross-sectional design of the present study precludes testing this bidirectionality.

We found that frequency of bullying victimization significantly increased the likelihood of membership in the two highest symptom profiles, Profile 4 (high internalizing, moderate externalizing) and Profile 5 (high internalizing, high externalizing), compared to the asymptomatic profile (Profile 1). This is consistent with variable-centered studies that have found bullying victimization frequency to be associated with internalizing symptoms (Penning et al., 2010) and externalizing (Champion and Clay, 2007). However, we note that since both Profile 4 and Profile 5 are high on internalizing symptoms, our results could suggest that internalizing symptoms are particularly sensitive to repetitive victimization.

This study has several strengths. First, our large sample size of victims (n=1196) enhances confidence in our ability to identify latent profiles of symptoms. Second, we used previously validated measures with high internal consistency reliability to measure anxiety, depression, delinquency, and violence against peers. Third, we used a person-centered approach, LPA, which enabled us to holistically and simultaneously examine internalizing and externalizing symptoms among bullying victims. In doing so, we identified four subtypes of symptoms (and one asymptomatic profile) that may be overlooked in variable-centered, single outcome studies. In all subtypes, internalizing and externalizing symptoms co-exist. It seems that adolescents are not either internalizers or externalizers. Victims of bullying should be screened for both types of symptoms to ensure appropriate treatment for their specific symptom constellation. Also, the measure of bullying victimization used was finely grained and allowed for analysis of effects of different types of bullying and their frequency, not performed in a previous LPA (Kretschmer et al., 2015). Lastly, our study is hypothesis driven; thus, the reference profiles in the multinomial regression analyses were set to match the theoretically- and empirically- based hypothesized comparisons. As a result, however, we did not test for differences between all combinations of profiles on each of the bullying characteristics. Future studies may consider testing other comparisons if justified.

Mixture models, including LPA, are not without methodological and substantive controversy. For example, Bauer and Curran (2003) note that overextraction of discrete classes or profiles is likely when data are non-normal. We have guarded against this possibility by using either normal or categorical indicators in the identification of our profiles. In addition to this methodological concern, critics of mixture modeling question whether true clustering of people along behavioral and psychological phenomena exists or whether the identification of such clusters is a statistical artifact using arbitrary cut points (see Eysenck, 1986, Meehl 1992, and Pickles & Angold, 2003 for more on this debate). We acknowledge this controversy and recognize that the profiles emerging from our data are based on probability and membership is subject to error; however, we believe that unmasking and describing subtypes of internalizing and externalizing symptoms among bullying victims has both theoretical and practical utility.

The primary limitation of the study is that we cannot attribute the internalizing and externalizing symptoms solely to the experience of bullying victimization. It is possible that the victims in our dataset were anxious, depressed, delinquent, or violent prior to their reported victimization. Longitudinal data are needed to confirm the temporality of the relationship. Although we used one wave of data from a multi-wave longitudinal study, we were limited in doing a longitudinal assessment of the hypotheses for several reasons. First, the bullying victimization measure was not added until this fourth wave of data collection. Also, the assessment of bullying was of the three months prior to the survey administration, rather than of a longer history of bullying victimization, which would have been needed to establish that the bullying happened prior to symptom development. Despite this limitation, the identification of profiles in this study can be used as the basis for future longitudinal research to test stability and transitions of symptom profiles among bullying victims over time.

Skewness of our externalizing measures required trichotomization, leading to loss of information which could have influenced our ability to identify the number and nature of latent profiles. Also, our definition of bullying deviates from one of the most common definitions of bullying used in the literature, that bullying 1) occurs between individuals of the same age group; 2) is characterized by an imbalance of power between the aggressor and the victim; and 3) occurs over a period of time (Olweus, 1978). However, some researchers have observed that even an action that is not intended by the perpetrator to cause harm may be interpreted by the victim as bullying (Guerin & Hennessy, 2002) and repetition of aggression may not need to be a criterion for bullying because one incident may cause the fear of repetition (Guerin & Hennessy, 2002). Additionally, Corvo and deLara (2010) suggest that measuring an imbalance of power between victim and perpetrator is unnecessary because children do not view a power differential as a dimension of bullying (Corvo & deLara, 2010). Finally, our profiles do not encapsulate all symptoms that bullying victims may present, such as substance use, eating disorders, and suicidality.

Conclusion

While the largest profile in our sample consisted of adolescent victims of bullying with few internalizing or externalizing symptoms (Profile 1, 27%), most of our sample fell into one of the four other profiles, demonstrating varying levels of internalizing and externalizing. This comorbidity of internalizing and externalizing symptoms must be considered in secondary interventions to reduce symptoms of psychopathology among victims of bullying. Future person-centered analyses of internalizing and externalizing symptoms of bullying victims should examine the stability and directionality of associations, as well as identify risk and protective factors that minimize psychopathology among victims of bullying.

Acknowledgments

This work was supported in part by a predoctoral fellowship provided by the National Institute of Child Health and Human Development (T32-HD07376) through the Center for Developmental Science, University of North Carolina at Chapel Hill, to Meridith Eastman. The parent study for this work was supported by the National Institutes of Health (R01 DA 13459 to Susan Ennett) and Centers for Disease Control and Prevention (CDC Grant No. R49 CCV423114 to Vangie Foshee).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control. 1974;19:716–723. https://doi.org/10.1109/TAC.1974.1100705. [Google Scholar]
  2. Allison PD. Quantitative applications in the social sciences. Thousand Oaks, CA: Sage Publications; 2002. Missing data. https://doi.org/10.4135/9781412985079. [Google Scholar]
  3. Angold A, Costello EJ, Messer SC, Pickles A. Development of a short questionnaire for use in epidemiological studies of depression in children and adolescents. International Journal of Methods in Psychiatric Research. 1995;5:237–249. [Google Scholar]
  4. Arseneault L, Bowes L, Shakoor S. Bullying victimization in youths and mental health problems:’Much ado about nothing’. Psychological Medicine. 2010;40:717–729. doi: 10.1017/S0033291709991383. https://doi.org/10.1017/S0033291709991383. [DOI] [PubMed] [Google Scholar]
  5. Arseneault L, Walsh E, Trzesniewski K, Newcombe R, Caspi A, Moffitt TE. Bullying victimization uniquely contributes to adjustment problems in young children: A nationally representative cohort study. Pediatrics. 2006;118:130–138. doi: 10.1542/peds.2005-2388. https://doi.org/10.1542/peds.2005-2388d. [DOI] [PubMed] [Google Scholar]
  6. Bandura A. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall, Inc; 1986. [Google Scholar]
  7. Barker ED, Arseneault L, Brendgen M, Fontaine N, Maughan B. Joint development of bullying and victimization in adolescence: Relations to delinquency and self-harm. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47:1030–1038. doi: 10.1097/CHI.ObO13e31817eec98. https://doi.org/10.1097/chi.0b013e31817eec98. [DOI] [PubMed] [Google Scholar]
  8. Bauer DJ, Curran PJ. Distributional assumptions of growth mixture models: implications for overextraction of latent trajectory classes. Psychological methods. 2003;8(3):338–363. doi: 10.1037/1082-989X.8.3.338. http://dx.doi.org/10.1037/1082-989X.8.3.338. [DOI] [PubMed] [Google Scholar]
  9. Casper DM, Card NA. Overt and relational victimization: A meta-analytic review of their overlap and associations with social-psychological adjustment. Child Development. 2017;88(2):466–483. doi: 10.1111/cdev.12621. https://doi.org/10.1111/cdev.12621. [DOI] [PubMed] [Google Scholar]
  10. Celeux G, Soromenho G. An entropy criterion for assessing the number of clusters in a mixture model. Journal of Profileification. 1996;13:195–212. https://doi.org/10.1007/bf01246098. [Google Scholar]
  11. Champion KM, Clay DL. Individual differences in responses to provocation and frequent victimization by peers. Child Psychiatry and Human Development. 2007;37:205–220. doi: 10.1007/s10578-006-0030-9. https://doi.org/10.1007/s10578-006-0030-9. [DOI] [PubMed] [Google Scholar]
  12. Collins LM, Lanza ST. Latent Profile and latent transition analysis: With applications in the social, behavioral, and health sciences. Hoboken, NJ: John Wiley & Sons; 2010. [Google Scholar]
  13. Cook CR, Williams KR, Guerra NG, Kim TE, Sadek S. Predictors of bullying and victimization in childhood and adolescence: A meta-analytic investigation. School Psychology Quarterly. 2010;35:65–83. https://doi.org/10.1037/a0020149. [Google Scholar]
  14. Copeland WE, Wolke D, Angold A, Costello EJ. Adult psychiatric outcomes of bullying and being bullied by peers in childhood and adolescence. JAMA Psychiatry. 2013;70:419–426. doi: 10.1001/jamapsychiatry.2013.504. https://doi.org/10.1001/jamapsychiatry.2013.504. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Corvo K, deLara E. Towards an integrated theory of relational violence: Is bullying a risk factor for domestic violence? Aggression and Violent Behavior. 2010;15:181–190. https://doi.org/10.1016/j.avb.2009.12.001. [Google Scholar]
  16. Dukes RL, Stein JA, Zane JI. Effect of relational bullying on attitudes, behavior and injury among adolescent bullies, victims and bully-victims. The Social Science Journal. 2009;46:671–688. https://doi.org/10.1016/j.soscij.2009.05.006. [Google Scholar]
  17. Elliott AC, Reisch JS. Proceedings of the SAS User’s Group International. San Francisco, CA: SAS Global User’s Group; 2006. Implementing a multiple comparison test for proportions in a 2xC crosstabulation in SAS; pp. 204–231. [Google Scholar]
  18. Ennett ST, Foshee VA, Bauman KE, Hussong A, Cai L, Reyes HLM, DuRant R. The social ecology of adolescent alcohol misuse. Child Development. 2008;79:1777–1791. doi: 10.1111/j.1467-8624.2008.01225.x. https://doi.org/10.1111/j.1467-8624.2008.01225.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Eysenck HJ. A critique of contemporary classification and diagnosis. In: Millon T, Klerman GL, editors. Contemporary directions in psychopathology. New York, NY: Guilford Press; 1986. pp. 73–78. [Google Scholar]
  20. Faris R, Felmlee D. Casualties of social combat school networks of peer victimization and their consequences. American Sociological Review. 2014;79:228–257. https://doi.org/10.1177/0003122414524573. [Google Scholar]
  21. Farrell AD, Kung EM, White KS, Valois RF. The structure of self-reported aggression, drug use, and delinquent behaviors during early adolescence. Journal of Clinical Child Psychology. 2000;29:282–292. doi: 10.1207/S15374424jccp2902_13. https://doi.org/10.1207/S15374424jccp2902_13. [DOI] [PubMed] [Google Scholar]
  22. Forero R, McLellan L, Rissel C, Bauman A. Bullying behaviour and psychosocial health among school students in New South Wales, Australia: Cross sectional survey. British Medical Journal. 1999;319:344–348. doi: 10.1136/bmj.319.7206.344. https://doi.org/10.1136/bmj.319.7206.344. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Foshee VA, Reyes HLM, Ennett ST, Suchindran C, Mathias JP, Karriker-Jaffe KJ, Benefield TS. Risk and protective factors distinguishing profiles of adolescent peer and dating violence perpetration. Journal of Adolescent Health. 2011;48:344–350. doi: 10.1016/j.jadohealth.2010.07.030. https://doi.org/10.1016/j.jadohealth.2010.07.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Graham JW, Olchowski AE, Gilreath TD. How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science. 2007;8:206–213. doi: 10.1007/s11121-007-0070-9. https://doi.org/10.1007/s11121-007-0070-9. [DOI] [PubMed] [Google Scholar]
  25. Guerin S, Hennessy E. Pupils’ definitions of bullying. European Journal of Psychology of Education. 2002;17:249–261. https://doi.org/10.1007/BF03173535. [Google Scholar]
  26. Hawker DS, Boulton MJ. Twenty years’ research on peer victimization and psychosocial maladjustment: A meta-analytic review of cross-sectional studies. Journal of Child Psychology and Psychiatry. 2000;41:441–455. https://doi.org/10.1111/1469-7610.00629. [PubMed] [Google Scholar]
  27. Hemphill SA, Kotevski A, Herrenkohl TI, Bond L, Kim MJ, Toumbourou JW, Catalano RF. Longitudinal consequences of adolescent bullying perpetration and victimisation: A study of students in Victoria, Australia. Criminal Behaviour and Mental Health. 2011;21:107–116. doi: 10.1002/cbm.802. https://doi.org/10.1002/cbm.802. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Juvonen J, Graham S. Bullying in schools: The power of bullies and the plight of victims. Annual Review of Psychology. 2014;65:159–185. doi: 10.1146/annurev-psych-010213-115030. https://doi.org/10.1146/annurev-psych-010213-115030. [DOI] [PubMed] [Google Scholar]
  29. Kaltiala-Heino R, Fröjd S, Marttunen M. Involvement in bullying and depression in a 2-year follow-up in middle adolescence. European child & adolescent psychiatry. 2010;19(1):45. doi: 10.1007/s00787-009-0039-2. https://doi.org/10.1007/s00787-009-0039-2. [DOI] [PubMed] [Google Scholar]
  30. Kim YS, Leventhal BL, Koh Y, Hubbard A, Boyce WT. School bullying and youth violence: Causes or consequences of psychopathologic behavior? Archives of General Psychiatry. 2006;63:1035–1041. doi: 10.1001/archpsyc.63.9.1035. https://doi.org/10.1001/archpsyc.63.9.1035. [DOI] [PubMed] [Google Scholar]
  31. Klomek AB, Sourander A, Niemelä S, Kumpulainen K, Piha J, Tamminen T, Gould MS. Childhood bullying behaviors as a risk for suicide attempts and completed suicides: A population-based birth cohort study. Journal of the American Academy of Child and Adolescent Psychiatry. 2009;48:254–261. doi: 10.1097/CHI.0b013e318196b91f. https://doi.org/10.1097/CHI.0b013e318196b91f. [DOI] [PubMed] [Google Scholar]
  32. Kretschmer T, Barker ED, Dijkstra JK, Oldehinkel AJ, Veenstra R. Multifinality of peer victimization: Maladjustment patterns and transitions from early to mid-adolescence. European Child and Adolescent Psychiatry. 2015;24:1169–1179. doi: 10.1007/s00787-014-0667-z. https://doi.org/10.1007/s00787-014-0667-z. [DOI] [PubMed] [Google Scholar]
  33. Laursen BP, Hoff E. Person-centered and variable-centered approaches to longitudinal data. Merrill-Palmer Quarterly. 2006;52:377–389. https://doi.org/10.1353/mpq.2006.0029. [Google Scholar]
  34. Lo Y, Mendell NR, Rubin DB. Testing the number of components in a normal mixture. Biometrika. 2001;88:767–778. https://doi.org/10.1093/biomet/88.3.767. [Google Scholar]
  35. McLachlan G, Peel D. Finite mixture models. Hoboken, NJ: John Wiley & Sons; 2000. https://doi.org/10.1002/0471721182. [Google Scholar]
  36. Meehl PE. Factors and taxa, traits and types, differences of degree and differences in kind. Journal of Personality. 1992;60:117–174. https://doi.org/10.1111/j.1467-6494.1992.tb00269.x. [Google Scholar]
  37. Menard SW. Youth Violence Research Bulletin. Washington, DC: Office of Juvenile Justice and Delinquency Prevention and the Centers for Disease Control and Prevention; 2002. Short and long-term consequences of adolescent victimization. [Google Scholar]
  38. Muthén B, Muthén LK. Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research. 2000;24:882–891. https://doi.org/10.1111/j.1530-0277.2000.tb02070.x. [PubMed] [Google Scholar]
  39. Nansel TR, Overpeck MD, Haynie DL, Ruan WJ, Scheidt PC. Relationships between bullying and violence among US youth. Archives of Pediatrics and Adolescent Medicine. 2003;157:348–353. doi: 10.1001/archpedi.157.4.348. [DOI] [PubMed] [Google Scholar]
  40. Nansel TR, Overpeck M, Pilla RS, Ruan WJ, Simons-Morton B, Scheidt P. Bullying behaviors among US youth: Prevalence and association with psychosocial adjustment. JAMA. 2001;285:2094–2100. doi: 10.1001/jama.285.16.2094. https://doi.org/10.1001/archpedi.157.4.348. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Naylor P, Cowie H, Cossin F, Bettencourt R, Lemme F. Teachers’ and pupils’ definitions of bullying. British Journal of Educational Psychology. 2006;76:553–576. doi: 10.1348/000709905X52229. https://doi.org/10.1348/000709905X52229. [DOI] [PubMed] [Google Scholar]
  42. Nylund KL, Asparouhov T, Muthén BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling. 2007;14:535–569. https://doi.org/10.1080/10705510701575396. [Google Scholar]
  43. Olweus D. Aggression in the schools: Bullies and whipping boys. Washington, DC: Hemisphere Publishing Corp; 1978. [Google Scholar]
  44. Penning SL, Bhagwanjee A, Govender K. Bullying boys: The traumatic effects of bullying in male adolescent learners. Journal of Child and Adolescent Mental Health. 2010;22:131–143. doi: 10.2989/17280583.2010.528580. https://doi.org/10.2989/17280583.2010.528580. [DOI] [PubMed] [Google Scholar]
  45. Pickles A, Angold A. Natural categories or fundamental dimensions: On carving nature at the joints and the rearticulation of psychopathology. Development and Psychopathology. 2003;15:529–551. doi: 10.1017/s0954579403000282. https://doi.org/10.1017/S0954579403000282. [DOI] [PubMed] [Google Scholar]
  46. Prinstein MJ, Boergers J, Vernberg EM. Overt and relational aggression in adolescents: Social-psychological adjustment of aggressors and victims. Journal of Clinical Child Psychology. 2001;30(4):479–491. doi: 10.1207/S15374424JCCP3004_05. https://doi.org/10.1207/S15374424JCCP3004_05. [DOI] [PubMed] [Google Scholar]
  47. Ramaswamy V, DeSarbo WS, Reibstein DJ, Robinson WT. An empirical pooling approach for estimating marketing mix elasticities with PIMS data. Marketing Science. 1993;12:103–124. https://doi.org/10.1287/mksc.12.1.103. [Google Scholar]
  48. Reijntjes A, Kamphuis JH, Prinzie P, Boelen PA, Van der Schoot M, Telch MJ. Prospective linkages between peer victimization and externalizing problems in children: A meta-analysis. Aggressive Behavior. 2011;37(3):215–222. doi: 10.1002/ab.20374. [DOI] [PubMed] [Google Scholar]
  49. Reijntjes A, Kamphuis JH, Prinzie P, Telch MJ. Peer victimization and internalizing problems in children: A meta-analysis of longitudinal studies. Child Abuse & Neglect. 2010;34(4):244–252. doi: 10.1016/j.chiabu.2009.07.009. https://doi.org/10.1016/j.chiabu.2009.07.009. [DOI] [PubMed] [Google Scholar]
  50. Reynolds CR, Richmond BO. Factor structure and construct validity of ‘what I think and feel’: The revised children’s manifest anxiety scale. Journal of Personality Assessment. 1979;43:281–283. doi: 10.1207/s15327752jpa4303_9. https://doi.org/10.1207/s15327752jpa4303_9. [DOI] [PubMed] [Google Scholar]
  51. Sapouna M, Wolke D. Resilience to bullying victimization: The role of individual, family and peer characteristics. Child abuse & neglect. 2013;37(11):997–1006. doi: 10.1016/j.chiabu.2013.05.009. https://doi.org/10.1016/j.chiabu.2013.05.009. [DOI] [PubMed] [Google Scholar]
  52. Shetgiri R, Lin H, Flores G. Trends in risk and protective factors for child bullying perpetration in the United States. Child Psychiatry and Human Development. 2013;44:89–104. doi: 10.1007/s10578-012-0312-3. https://doi.org/10.1007/s10578-012-0312-3. [DOI] [PubMed] [Google Scholar]
  53. Sullivan TN, Farrell AD, Kliewer W. Peer victimization in early adolescence: Association between physical and relational victimization and drug use, aggression, and delinquent behaviors among urban middle school students. Development and Psychopathology. 2006;18:119–137. doi: 10.1017/S095457940606007X. https://doi.org/10.1017/S095457940606007X. [DOI] [PubMed] [Google Scholar]
  54. Tofghi D, Enders CK. Identifying the correct number of classes in mixture models. In: Hancock GR, Samulelsen KM, editors. Advances in latent variable mixture models. Greenwich, CT: Information Age; 2007. pp. 317–341. [Google Scholar]
  55. Ttofi MM, Farrington DP, Lösel F. School bullying as a predictor of violence later in life: A systematic review and meta-analysis of prospective longitudinal studies. Aggression and Violent Behavior. 2012;17(5):405–418. https://doi.org/10.1016/j.avb.2012.05.002. [Google Scholar]
  56. Valdebenito S, Ttofi MM, Eisner M, Gaffney H. Weapon carrying in and out of school among pure bullies, pure victims and bully-victims: a systematic review and meta-analysis of cross-sectional and longitudinal studies. Aggression and Violent Behavior. 2017;33:62–77. https://doi.org/10.1016/j.avb.2017.01.004. [Google Scholar]
  57. Zar JH. Biostatistical analysis. Englewood Cliffs, New Jersey: Prentice Hall; 1999. [Google Scholar]
  58. Zwierzynska K, Wolke D, Lereya TS. Peer victimization in childhood and internalizing problems in adolescence: A prospective longitudinal study. Journal of Abnormal Child Psychology. 2013;41:309–323. doi: 10.1007/s10802-012-9678-8. https://doi.org/10.1007/s10802-012-9678-8. [DOI] [PubMed] [Google Scholar]

RESOURCES