Skip to main content
PMC home page
Springer logoLink to Springer
. 2021 Jan 22;49(6):727–736. doi: 10.1007/s10802-020-00760-2

Exposure to Bullying and General Psychopathology: A Prospective, Longitudinal Study

Jolien Rijlaarsdam 1,✉, Charlotte A M Cecil 1,2, J Marieke Buil 3, Pol A C van Lier 3, Edward D Barker 4
PMCID: PMC8096758  PMID: 33481128

Abstract

Although there is mounting evidence that the experience of being bullied associates with both internalizing and externalizing symptoms, it is not known yet whether the identified associations are specific to these symptoms, or shared between them. The primary focus of this study is to assess the prospective associations of bullying exposure with both general and specific (i.e., internalizing, externalizing) factors of psychopathology. This study included data from 6,210 children participating in the Avon Longitudinal Study of Parents and Children (ALSPAC). Child bullying was measured by self-report at ages 8 and 10 years. Child psychopathology symptoms were assessed by parent-interview, using the Development and Well-being Assessment (DAWBA) at ages 7 and 13 years. Bullying exposure significantly associated with the general psychopathology factor in early adolescence. In particular, chronically victimized youth exposed to multiple forms of bullying (i.e., both overt and relational) showed higher levels of general psychopathology. Bullying exposure also associated with both internalizing and externalizing factors from the correlated-factors model. However, the effect estimates for these factors decreased considerably in size and dropped to insignificant for the internalizing factor after extracting the shared variance that belongs to the general factor of psychopathology. Using an integrative longitudinal model, we found that higher levels of general psychopathology at age 7 also associated with bullying exposure at age 8 which, in turn, associated with general psychopathology at age 13 through its two-year continuity. Findings suggest that exposure to bullying is a risk factor for a more general vulnerability to psychopathology.

Keywords: Bullying exposure, General psychopathology, Longitudinal, ALSPAC

Introduction

Bullying is a form of aggressive behavior that is repeated over time against one or more individuals who are relatively powerless (Monks et al. 2009; Salmivalli 2010). The aggressive behavior can take many forms such as name calling, hitting, spreading rumors, and social exclusion. Bullying is considered a significant public health problem world-wide, with prevalence rates ranging from approximately 10 to 25% (Analitis et al. 2009; Nansel et al. 2001; Thomas et al. 2017). There is mounting evidence that the experience of being bullied associates with overall mental health problems, including both internalizing and externalizing symptoms (Klomek et al. 2015; Moore et al. 2017; Reijntjes et al. 2010; Schoeler et al. 2018; Singham et al. 2017; van Lier et al. 2012).

However, it is unclear at present whether these identified associations with exposure to bullying are specific to particular psychiatric outcomes, or shared between them. Psychiatric disorders and their symptoms co-occur substantially, even across the broadly defined internalizing and externalizing domains (Achenbach et al. 2016; Angold et al. 1999; Kessler et al. 2005; Krueger 1999). Additionally, psychiatric disorders often have a multifactorial etiology that includes (i) shared risk factors, such as bullying, childhood maltreatment, maternal depression, and stressful life events (Caron and Rutter 1991; Kessler et al. 1997; Kim et al. 2003; Vachon et al. 2015), and (ii) a shared genetic vulnerability associated with, for example, the risk of exposure to bullying (Schoeler et al. 2019). In recent years, studies have suggested that this general vulnerability to psychopathology may be usefully represented by a bifactor model that captures (i) shared variance among a broad range of mental health problems (i.e., general factor), as well as (ii) specific influences beyond those explained by shared variance (i.e., specific factors) (for studies in children and early adolescents, see Lahey et al. 2015; Neumann et al. 2016; Olino et al. 2014; Patalay et al. 2015). Although in children the focus has often been on internalizing and externalizing dimensions of psychopathology, thought disorders (e.g., obsessive–compulsive disorders, schizophrenia) have been identified as a distinct third dimension in adolescents and adults (Caspi et al. 2014).

The general factor of psychopathology in childhood has been found to predict various psychiatric outcomes in adolescence, such as anxiety disorders, mood disorders, and substance abuse (Pettersson et al. 2018). The specific factors, however, predicted only a subset of psychiatric outcomes. For example, the specific internalizing factor associated with anxiety and mood disorders but not with, for example, substance abuse. Furthermore, previous research found evidence for both genetic and environmental influences on the general factor of psychopathology (Brikell et al. 2020; Brodbeck et al. 2018; Caspi et al. 2014; Neumann et al. 2016; Riglin et al. 2019). Of particular interest is the finding that childhood maltreatment (e.g., harsh discipline, physical abuse, sexual abuse) associated most strongly to a general factor of psychopathology as opposed to symptom specific factors (e.g., internalizing, externalizing) (Brodbeck et al. 2018; Caspi et al. 2014). However, to the best of our knowledge, no study to date has examined the association of bullying exposure in childhood with the general factor of psychopathology in early adolescence.

The primary focus of the current study is to assess the prospective associations of bullying victimization (8, 10 years) with both general and specific psychopathology factors (7, 13 years) while accounting for other risk exposures known to associate with overall mental health problems (e.g., childhood maltreatment, maternal depression, stressful life events). To this end, the aim of the study was threefold; (i) testing general psychopathology bifactor models at different time points across development, specifying a general factor in addition to specific internalizing and externalizing factors, (ii) systematically characterizing the chronicity and type (overt/relational) of bullying victimization in relation to both general and specific psychopathology factors, and (iii) incorporating the repeated assessment of bullying exposure and general psychopathology into a path model, together with the other risk exposures, to test for prospective interrelations.

Methods

Participants

Participants were drawn from the Avon Longitudinal Study of Parents and Children (ALSPAC). ALSPAC is an ongoing epidemiological study of children born from 14,541 pregnant women residing in Avon, United Kingdom, with an expected delivery date between April 1991 and December 1992 (85% of eligible population; Fraser et al. 2013). When the oldest children were approximately 7 years of age, an attempt was made to bolster the initial sample with eligible cases who had not joined the study originally, resulting in an additional 713 children being enrolled. Ethical approval for the study was obtained from the ALSPAC Ethics and Law Committee as well as Local Research Ethics Committees. Informed consent was obtained from all ALSPAC participants. The original ALSPAC sample is representative of the general population (Boyd et al. 2013). Please note that the study website contains details of all the data that is available through a fully searchable data dictionary: http://www.bris.ac.uk/alspac/researchers/data-access/data-dictionary/.

Measures

Bullying Exposure

Exposure to bullying victimization was assessed at ages 8 and 10 years via child report, using the previously validated Bullying and Friendship Interview Schedule (Hamburger et al. 2011; Wolke et al. 2000). Trained psychology graduates asked children to rate victimization by peers during the past six months, using five stem/contingent question pairs on overt victimization (e.g., “having belongings stolen”) and four stem/contingent question pairs on relational victimization (e.g., “other children not wanting to play with them”). If children answered yes versus no (score 0) to any form of victimization, they were contingently asked how frequently it had occurred: 1 to 3 times in the past 6 months (infrequently, score 1), ≥ 4 times in the past 6 months but less than once per week (frequently, score 2), at least once per week (very frequently, score 3). The Bullying and Friendship Interview Schedule has demonstrated high inter-rater reliability and predictive validity (Zwierzynska et al. 2013). Using confirmatory factor analysis, the four relational victimization items showed acceptable internal reliability at age 8 (RMSEA = 0.020, CFI = 0.999, SRMR = 0.011) and age 10 (RMSEA = 0.010, CFI = 1.000, SRMR = 0.008). Similarly, the five overt victimization items showed acceptable internal reliability in a confirmatory factor analysis at age 8 (RMSEA = 0.026, CFI = 0.995, SRMR = 0.020) and age 10 (RMSEA = 0.025, CFI = 0.995, SRMR = 0.022). In line with previous research (Wolke et al. 2012), we computed continuous severity indices of exposure to bullying (overt and/or relational) by totaling the stem/contingent item pairs (each scaled 0–3) at 8 and 10 years, as well as across the two time points.

In follow-up analyses, we also examined categorical variables characterizing the chronicity and type (overt/relational) of bullying, based on previous research (Wolke et al. 2014; Wolke et al. 2012). Overt and relational bullying victimization was coded as present if children experienced victimization frequently or very frequently (Wolke et al. 2012). The following three categorical variables were constructed (Wolke et al. 2014, 2012): (1) Any bullying victimization (overt and/or relational) at 8 and/or 10 years of age; (2) Chronicity of victimization: never victimized (no report of victimization), unstable (reported at one time point), or stable (reported at both time points); (3) Type of victimization at 10 years: never victimized, victim of overt bullying only, victim of relational bullying only, victim of both relational and overt bullying. All categorical variables were dummy coded prior to analyses.

General Psychopathology

Psychopathology symptoms were repeatedly assessed with the Development and Well-being Assessment (DAWBA; R. Goodman et al. 2000), a validated semi structured interview. Parents completed open and closed questions about a range of symptoms relevant to youth psychiatric disorders, including both internalizing (generalized anxiety disorder [GAD], major depressive disorder [MDD], social phobia, separation anxiety, specific phobia) and externalizing (attention deficit hyperactivity disorder [ADHD], oppositional defiant disorder [ODD], conduct disorder [CD]) domains. For each disorder, an ordered categorical measure was generated using computer algorithms (A. Goodman et al. 2011), comprising six categories indicating the likelihood of each youth having the disorder from level 0 up to level 5. Using these DAWBA measures at ages 7, 10, and 13 years, we tested general psychopathology bifactor models specifying a general factor in addition to domain-specific internalizing and externalizing factors. See statistical analyses section for further details.

Covariates

All analyses adjusted for sex. We additionally accounted for a range of risk factors that may influence general psychopathology or have been routinely adjusted for in birth cohorts (Dunn et al. 2020). Specifically, we included previously established cumulative risk scores based on maternal reports (Cecil et al. 2014). For three developmental periods (i.e., pregnancy, early-childhood, late-childhood), risks were summed to create cumulative scores covering the following domains: (a) life events (e.g. death in family, accident, illness), (b) contextual risks (e.g. poor housing conditions, financial problems), (c) parental risks (e.g. parental psychopathology, criminal involvement and substance use), (d) interpersonal risks (e.g. intimate partner violence, family conflict), and (e) child maltreatment (e.g. child physically hurt or sexually abused, parent physically or emotionally cruel to child; available postnatally). As can be seen in Supplementary Tables S1.1-S1.3, the existing scores were slightly adapted to our study context, in such a way that (i) the bullying items were excluded, and (ii) the items were organized into one of three developmental periods so as to coincide with the general psychopathology data: pregnancy, birth–age 7 (early-childhood), and age 8–12 (late-childhood). The overall cumulative risk scores were estimated using confirmatory factor analysis, as described elsewhere (Cecil et al. 2014). In sensitivity analyses, we additionally corrected for child IQ as assessed at 8 years of age using the Wechsler Intelligence Scale for Children (3rd UK edition).

Attrition

For this study, we included youth from ALSPAC who had available data on general psychopathology at age 13 years (N = 6,210, 49% male; see Table S2 for numbers, descriptive statistics, and correlations). In a multivariate model, we examined the extent to which the study variables (i.e., severity index of bullying exposure at ages 8–10, general psychopathology at age 7) and covariates (i.e., early-childhood risk exposure, IQ) associated with exclusion from the current study. Children with a lower IQ were more likely to be excluded in the current analysis (OR = 0.990, 95% CI = 0.986–0.994). However, bullying exposure (OR = 0.994, 95% CI = 0.982–1,006), general psychopathology (OR = 1,110, 95% CI = 0.999–1.234) and risk exposure (OR = 0.937, 95% CI = 0.868–1.012) were unrelated to exclusion from the study.

Statistical Analysis

The analyses proceeded in three main steps. In the first step, we used confirmatory factor analysis (CFA) to fit a general psychopathology bifactor model (Gibbons and Hedeker 1992), with the 13-year time point as the main time point of interest, measured prospectively with bullying exposure at ages 8 and 10. All analyses were repeated for the 7- and 10-year time points. In the bifactor model, each psychopathology subdomain loaded on a single general factor and on one domain-specific internalizing (GAD, MDD, social phobia, separation anxiety, specific phobia) or externalizing (ADHD, ODD, CD) factor. Because these general and domain-specific factors are modelled to be orthogonal to each other (Gibbons and Hedeker 1992; Rodriguez et al. 2016b), we constrained all factor covariances to zero. A general psychopathology bifactor model was computed in R version 3.4.3, using the package Lavaan (Rosseel 2012) with robust weighted least square mean and variance adjusted (WLSMV) estimation for ordinal data. We tested the replicability of this bifactor model in two ways. First, bifactor-specific indices (Rodriguez et al. 2016a; Rodriguez et al. 2016b) were computed using Marley Watkins’s “Omega” software, including: reliability estimates for both the general and specific latent factors (omega values), the proportion of variability in the latent factors that is explained by its indicators (H), as well as explained common variance (ECV) and the percentage of uncontaminated correlations (PUC) that offer information about the extent to which the data are multidimensional. Second, we directly examined the replicability of the bifactor structure in our longitudinal data (7, 10, 13 years). To inform the role of general psychopathology in our data, we repeated our regression analyses for a correlated two-factors model in which internalizing and externalizing factors each are indicated by a subset of psychopathology domains and are assumed to be correlated, but no general factor is identified (see Figure S1).

In the second step, we output factor scores from the bifactor model at age 13, saved them, and examined their associations with the continuous severity index of exposure to bullying across time points (age 8,10). In follow-up analyses, we also individually assessed categorical variables characterizing the chronicity and type (overt, relational, combined) of being exposed to bullying in relation to general and specific factors of psychopathology at age 13. The maximum likelihood estimation with robust standard errors (MLR) was used to correct for possible non-normal distributions of study variables.

In the final step, the continuous severity indices of exposure to bullying (ages 8, 10) and the saved general psychopathology factor scores (ages 7, 13) were then incorporated into a path model, together with the cumulative risk scores (pregnancy, early-childhood, late-childhood) to test for prospective interrelations. We used bootstrapping with bias-corrected 95% confidence intervals (10,000 bootstraps) to derive variance from the empirical distribution of the observed data. Model fit was established using the root-mean-square error of approximation (RMSEA: acceptable fit ≤ 0.08), the comparative fit index (CFI; acceptable fit ≥ 0.90), and the standardized root mean square residual (SRMR; acceptable fit ≤ 0.08) (Hu and Bentler 1999; Perry et al. 2015).

All analyses in steps 2 and 3 were performed in Mplus version 7.11 (Muthén and Muthén 1998–2016) using maximum likelihood estimation. We addressed missing data in our sample through multiple imputation (step 2) and a full information maximum likelihood (FIML) approach (step 3). In the individual models regressing general psychopathology on bullying exposure (step 2), missing data on covariates were imputed 20 times using the ‘data imputation’ procedure implemented in Mplus. The samples after multiple imputation ranged from N = 4,613 to N = 5,370 (see Table 1), depending on the specific bullying exposure variables being assessed. In sensitivity analyses, we compared associations for the continuous severity index of bullying exposure in the imputed (N = 5,370) and non-imputed (N = 5,125) data sets.

Table 1.

Associations between exposure to bullying and general or specific factors of psychopathology

Model A Model B
GPF INT EXT GPF INT EXT
Bullying exposure β (SE) β (SE) β (SE) β (SE) β (SE) β (SE)
Severity index (n = 5,370)a, score 0.12 (0.01)*** 0.02 (0.01) 0.10 (0.01)*** 0.07 (0.01)*** 0.03 (0.01)* 0.04 (0.01)**
Any exposure (n = 4,613)
yes (53.4%) vs no 0.12 (0.01)*** 0.02 (0.01) 0.09 (0.01)*** 0.06 (0.01)*** 0.02 (0.01) 0.04 (0.01)**
Chronicity (n = 4,613)
none (reference)
unstable (41.0%) 0.10 (0.02)*** 0.03 (0.02) 0.07 (0.02)*** 0.05 (0.01)** 0.03 (0.01)* 0.03 (0.01)*
stable (12.4%) 0.12 (0.02)*** 0.00 (0.02) 0.10 (0.02)*** 0.07 (0.01)*** 0.00 (0.02) 0.06 (0.02)***
Type (n = 4,882)b
none (reference)
overt only (16.1%) 0.09 (0.01)*** 0.00 (0.01) 0.07 (0.02)*** 0.06 (0.01)*** 0.01 (0.01) 0.04 (0.01)**
relational only (2.8%) 0.04 (0.02)** 0.02 (0.02) 0.03 (0.01) 0.03 (0.01) 0.01 (0.01) 0.02 (0.01)
both (4.9%) 0.08 (0.01)*** 0.01 (0.01) 0.06 (0.02)*** 0.06 (0.01)*** 0.02 (0.01) 0.04 (0.01)*
Open in a new tab

Model A presents linear regression results controlling for sex and early-childhood cumulative risk exposure; Model B is similar to model A, additionally controlling for pre-existing psychopathology (GPF, INT, or EXT)

GPF = general psychopathology factor; INT = specific internalizing factor; EXT = specific externalizing factor

*p < 0.05; **p < 0.01; ***p < 0.001

aSummed across time points (ages 8 and 10 years)

bMeasured at age 10

Our integrative path model interrelating all study variables and covariates (step 3) offered maximum likelihood alternatives, such as the FIML approach. This FIML approach estimates a likelihood function for each individual based on the variables included in the model, and produces unbiased parameter estimates and standard errors. The sample in our integrative path model using FIML was 6,210, which represents the ALSPAC sample with data on general psychopathology at age 13.

Results

Step 1: General Psychopathology Factor

At age 13, all psychopathology sub-domains loaded significantly on the general psychopathology factor independent of the domain-specific internalizing and externalizing factors, with all but the specific phobia factor loading above 0.30 (see Table S3.1). Similarly, all internalizing sub-domains loaded significantly on the internalizing factor and all externalizing sub-domains loaded significantly on the externalizing factor independent of the general psychopathology factor, with all factor loadings above 0.30. The bifactor-specific model fit indices (see Table S3.4) supported a multidimensional conceptualization of general psychopathology, with the specific factors adding important variation to a unidimensional general factor. Specifically, the proportion of variance attributable to all sources of common variance was high (omega = 0.82), while the variance attributable to solely the single general factor was considerably lower (omegaH = 0.48). This is supported by the moderate explained common variance of the general factor (ECV = 0.44), in conjunction with a modest percentage of uncontaminated correlations (PUC = 0.54). Construct replicability was borderline acceptable for the general factor (H = 0.69) using the 0.70 benchmark (Rodriguez et al. 2016b) and was slightly lower for the specific factors (Hinternalizing = 0.65, Hexternalizing = 0.57). However, it should be noted that construct replicability is highly influenced by the number of variables used to define these general (8 indicators) and specific internalizing (5 indicators) and externalizing (3 indicators) factors (Rodriguez et al. 2016a).

As displayed in Tables S3.1-S3.4, the bifactor structure of psychopathology was largely consistent across the three time points (7, 10, 13 years) in that (i) all psychopathology sub-domains loaded significantly on the general psychopathology factor and on one specific factor (internalizing, externalizing), and (ii) construct replicability was borderline acceptable for the general factor (range H = 0.64—0.70) and slightly lower for the specific internalizing (range H = 0.51 – 0.65) and externalizing (range H = 0.55—0.66) factors. The most notable difference is that the specific phobia sub-domain, which also had the lowest general factor loadings at ages 10 and 13, was excluded at age 7 due to non-convergence. Nevertheless, the general factor (r = 0.50–0.53, all p < 0.001) as well as the specific internalizing (r = 0.32–0.40, all p < 0.001) and externalizing (r = 0.42–0.56, all p < 0.001) factors were at least moderately correlated with themselves over time (i.e., significant autocorrelations).

Step 2: The Association Between Exposure to Bullying and General Psychopathology

Correlations of bullying exposure with the general (rage 7 = 0.18, rage 10 = 0.15, rage 13 = 0.16, all p < 0.001) and the specific internalizing (rage 7 = 0.001, rage 10 = 0.02, rage 13 = 0.03, all p > 0.05) and externalizing (rage 7 = 0.17, rage 10 = 0.18, rage 13 = 0.12, all p < 0.001) factors defined in the bifactor model were similar across the three time points. Table 1 presents the linear regression results for bullying victimization in relation to general psychopathology and internalizing and externalizing factor scores from the bifactor model at age 13. We highlight here three findings. First, in the model correcting for sex and early-childhood environmental risk (model A), the continuous severity index of bullying exposure at age 8 and/or 10 significantly associated with general psychopathology (β = 0.12).

We then examined internalizing and externalizing factors, as defined in the bifactor model and the correlated two-factors model (CFI = 0.962, RMSEA = 0.055, SRMR = 0.047) in relation to bullying exposure. Although bullying exposure associated with both the internalizing (β = 0.08) and externalizing (β = 0.13) factors from the correlated two-factors model (see Table S4), only the externalizing (β = 0.10) and not the internalizing (β = 0.02) factor associated with bullying exposure in the bifactor model (see Table 1). Similarly, when partialling out the shared variance between the internalizing and externalizing factors from the correlated two-factors model, only the externalizing (β = 0.09) and not the internalizing (β = 0.01) factor remained associated with bullying exposure (see Table S4). As can be seen in Table S5, regression coefficients were similar in the imputed and non-imputed data sets (N = 5,125).

Second, as can be seen in Table 1, these associations of bullying exposure with the general (β = 0.07) and externalizing (β = 0.04) factors considerably attenuated but remained statistically significant after incorporating pre-existing psychopathology at age 7 in the regression models. However, regression coefficients remained largely unchanged when additionally correcting for IQ (see Table S6).

Finally, we highlight associations of the various characteristics of exposure to bullying measured in the study, including chronicity (unstable, stable) and type (overt, relational, combined). As shown in Table 1, the binary measure of any bullying experience at age 8 and/or 10 associated with both the general (β = 0.12) and the externalizing (β = 0.09) factors but not with the internalizing factor (β = 0.02). Regarding the chronicity of bullying victimization, both stable or unstable bullying victimization associated significantly with the general (βstable = 0.12; βunstable = 0.10) and externalizing (βstable = 0.10; βunstable = 0.07) factors. With regards to the different types of bullying exposures included in the study, particularly combined (both overt and relational) and overt only types associated with the general (βcombined = 0.08; βovert = 0.09; βrelational = 0.04) and externalizing (βcombined = 0.06; βovert = 0.07; βrelational = 0.03) factors.

Step 3: Integrative Longitudinal Model of General Psychopathology

As a final step, we made use of an integrative longitudinal model to examine the prospective interrelations between environmental risk exposure (prenatal, postnatal), the continuous severity index of exposure to bullying (age 8, 10) and general psychopathology (age 7, 13), while correcting for sex. Model fit was acceptable (RMSEA = 0.046, CFI = 0.988, SRMR = 0.027). Figure 1 shows that despite considerable continuity in the study variables (ranging from β = 0.36 to β = 0.50), higher levels of exposure to bullying at age 10 (β = 0.07) and both prenatal (β = 0.12) and postnatal (β = 0.05, β = 0.08) risk exposure independently associated with higher levels of general psychopathology at age 13. Higher general psychopathology levels at age 7 (β = 0.14) and prenatal risk exposure (β = 0.06) independently associated with higher levels of bullying victimization at age 8. Of note, associations were small in magnitude.

Fig. 1.

Fig. 1

Open in a new tab

Prospective interrelations between cumulative risk exposure, bullying exposure, and the general psychopathology factor (GPF), N = 6,210. Solid arrowed lines indicate standardized path coefficients that survived bootstrap-corrected confidence intervals (i.e., significant paths)

Discussion

The primary aim of this study was to investigate prospective associations of exposure to bullying with general psychopathology factors across childhood, accounting for a wide range of pre- and postnatal factors. Using prospective data from a large birth cohort study, we found that exposure to bullying in primary school associated with the general psychopathology factor in early adolescence. Our integrative longitudinal model showed that pre-existing psychopathology, together with cumulative risk exposure, is also a vulnerability factor for bullying exposure.

Although it is well established that those whose are bullied are at increased risk for a wide range of mental health disorders and symptoms, less is known about the extent to which these findings are specific to these outcomes or reflect a general vulnerability to psychopathology. We found that bullying exposure was associated with both internalizing and externalizing factors from the correlated-factors model that has been commonly used in research on the structure of psychopathology. However, after extracting the shared variance that belongs to the general factor of psychopathology, effect estimates for the internalizing factor decreased considerably in size and dropped to insignificant. In the correlated two-factors model, the observed link between bullying exposure and the internalizing factor was likely inflated because the internalizing factor contained variance shared with the externalizing factor, which associated with bullying exposure. The bifactor model, however, has the advantage of directly assessing the shared variance between internalizing and externalizing domains (general factor), while simultaneously modeling the unique variance in each domain (specific factors).

Concerns have been raised, however, about the stability of these general and specific factors across samples or indicators (Bornovalova et al. 2020). Although we found support for a general factor of psychopathology with borderline acceptable construct replicability across three different time points (ages 7, 10, 13), the pattern of the factor loadings for the specific factors did not replicate well. This finding supports previous research by Gluschkoff et al. (2019), showing that despite an equal degree of strong longitudinal invariance for the correlated factors model and the bifactor model, the specific internalizing factor demonstrated unacceptable construct replicability estimates. For example, some of the factor loadings dropped to insignificant once the general factor was included in the model. Although these irregular loading patterns can challenge the interpretation of the specific factors, our longitudinal data demonstrated consistent, moderate autocorrelations over time, as well as consistent correlations with bullying exposure.

In particular, youth chronically exposed to multiple forms of victimization (i.e., both overt and relational) displayed greater levels of general psychopathology in early adolescence. This finding is in line with previous research indicating a dose–response relationship between chronic (8 and 10 years) or combined (overt and relational) exposure to bullying and borderline personality symptoms at age 11 years (Wolke et al. 2012). All of these associations considerably attenuated but remained significant after correcting for pre-existing general psychopathology. This finding supports prior evidence that the association between exposure to bullying and mental health outcomes can partially be accounted for by pre-existing vulnerabilities of bullied individuals (Hodges and Perry 1999; Reijntjes et al. 2010; Singham et al. 2017). Children exposed to bullying may differ from children not exposed in individual characteristics, such as having fewer friends, withdrawal or aggressiveness, which in turn increase their risks for being bullied (Arseneault et al. 2010; Monks et al. 2009). It has been previously shown that children who displayed aggressive behaviors in early childhood were more likely than nonaggressive children to experience chronic or high levels of bullying in preschool (Barker et al. 2008).

Integrating repeated assessments of general psychopathology (7, 13 years), exposure to bullying (8, 10 years) and environmental risk exposure (prenatal, early-childhood, late-childhood) in a longitudinal risk model, we also found that general psychopathology prospectively associated with bullying exposure, with higher levels of general psychopathology at age 7 rendering youth more susceptible to exposure to bullying at age 8. Bullying exposure at age 8, in turn, associated with general psychopathology at age 13 through its two-year continuity. Although statistically significant, the observed association of exposure to bullying with general psychopathology was small in magnitude when controlling for the various variables included in the model. This is in line with multifactorial influences on overall mental health problems (Lereya et al. 2015). For example, we found that the effect size of bullying exposure was similar to that observed for the cumulative risk exposure score, including a wide range of adversities known to be associated with mental health.

Self-reports involve children’s subjective perception of being victimized and likely tap into children’s feelings and well-being. Indeed, it has been demonstrated that self-reports of bullying victimization are stronger predictors of internalizing problems than peer reports (Bouman et al. 2012). However, this also implies that parents may have under-reported their children’s internalizing symptoms. Nevertheless, we found that the internalizing and externalizing factors were largely equally indicative of the general factor and were similarly associated with bullying exposure in the correlated-factors model. By using different reporters for bullying exposure and child outcomes, we were able to control for common method variance attributable to the informants.

The current findings should be interpreted in the context of several limitations. First, as mentioned previously, the replicability of the specific factors was sub optimal, owing in part to their small number of indicators. Therefore, in the future, our findings need to be replicated in longitudinal data that allows a higher number of psychiatric domains and disorders. Second, as with most longitudinal studies, considerable attrition occurred. Attrition might result in a loss of power to detect effects and may also bias the findings to those individuals who continued participating in the study. However, bullied children with missing data on covariates were as likely as children with complete data to have higher levels of psychopathology. This supports previous simulations using ALSPAC data, demonstrating that associations between predictors and outcomes are unlikely to be substantially altered by selective dropout (Wolke et al. 2009). Of note, the full information likelihood (FIML) approach enabled our integrative path model to be conducted on the full sample of children with available data on general psychopathology at age 13. Third, the analyses are correlational in nature and, hence, causality cannot be inferred. Fourth, in line with previous child studies, the current study examined the structure of psychopathology using the broadly defined internalizing and externalizing domains. Other disorders not studied here (e.g., obsessive–compulsive disorders, schizophrenia) might also contribute to a general factor of psychopathology (Caspi et al. 2014). Finally, although we controlled for a wide range of covariates, spanning the prenatal period up to early adolescence, the possibility of residual confounding cannot be fully excluded.

These limitations notwithstanding, this is the first population-based longitudinal study to show that exposure to bullying in primary school is a risk factor for a more general vulnerability to psychopathology in early adolescence. Although small in magnitude, the findings of the current study highlight the potential value of a transdiagnostic approach to understanding psychopathology.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 249 KB) (249.7KB, pdf)
Supplementary file2 (XLSX 108 KB) (107.2KB, xlsx)

Acknowledgements

We are extremely grateful to all the families who took part in this study, the midwives for their help in recruiting them, and the whole ALSPAC team, which includes interviewers, computer and laboratory technicians, clerical workers, research scientists, volunteers, managers, receptionists and nurses. The UK Medical Research Council and the Wellcome Trust (Grant ref: 102215/2/13/2) and the University of Bristol provide core support for ALSPAC. A comprehensive list of grants funding is available on the ALSPAC website (http://www.bristol.ac.uk/alspac/external/documents/grant-acknowledgements.pdf). This publication is the work of the authors who will serve as guarantors for the contents of this article.

Funding

This research was specifically funded by research awards from the National Institute of Child and Human Development (R01HD068437) and the Economic and Social Research Council (ES/R005516/1) to E.D. Barker. The work of J. Rijlaarsdam was supported by the Netherlands Organization for Scientific Research (NWO ZonMw VENI, grant No 91618147). The work of C.A.M. Cecil has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement (grant No 707404).

Compliance with Ethical Standards

Conflict of Interest

For all authors no conflict of interest is declared.

Ethical Approval

Ethical approval for the study was obtained from the ALSPAC Ethics and Law Committee as well as Local Research Ethics Committees.

Informed Consent

Informed consent was obtained from all ALSPAC participants.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10802-020-00760-2.

References

  1. Achenbach TM, Ivanova MY, Rescorla LA, Turner LV, Althoff RR. Internalizing/Externalizing Problems: Review and Recommendations for Clinical and Research Applications. Journal of the American Academy of Child and Adolescent Psychiatry. 2016;55(8):647–656. doi: 10.1016/j.jaac.2016.05.012. [DOI] [PubMed] [Google Scholar]
  2. Analitis F, Velderman MK, Ravens-Sieberer U, Detmar S, Erhart M, Herdman M, Grp EK. Being Bullied: Associated Factors in Children and Adolescents 8 to 18 Years Old in 11 European Countries. Pediatrics. 2009;123(2):569–577. doi: 10.1542/peds.2008-0323. [DOI] [PubMed] [Google Scholar]
  3. Angold A, Costello EJ, Erkanli A. Comorbidity. Journal of Child Psychology and Psychiatry and Allied Disciplines. 1999;40(1):57–87. doi: 10.1111/1469-7610.00424. [DOI] [PubMed] [Google Scholar]
  4. Arseneault L, Bowes L, Shakoor S. Bullying victimization in youths and mental health problems: 'much ado about nothing'? Psychological Medicine. 2010;40(5):717–729. doi: 10.1017/S0033291709991383. [DOI] [PubMed] [Google Scholar]
  5. Barker ED, Boivin M, Brendgen M, Fontaine N, Arseneault L, Vitaro F, Tremblay RE. Predictive validity and early predictors of peer-victimization trajectories in preschool. Archives of General Psychiatry. 2008;65(10):1185–1192. doi: 10.1001/archpsyc.65.10.1185. [DOI] [PubMed] [Google Scholar]
  6. Bornovalova MA, Choate AM, Fatimah H, Petersen KJ, Wiernik BM. Appropriate Use of Bifactor Analysis in Psychopathology Research: Appreciating Benefits and Limitations. Biological Psychiatry. 2020;88(1):18–27. doi: 10.1016/j.biopsych.2020.01.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bouman T, van der Meulen M, Goossens FA, Olthof T, Vermande MM, Aleva EA. Peer and self-reports of victimization and bullying: their differential association with internalizing problems and social adjustment. Journal of School Psychology. 2012;50(6):759–774. doi: 10.1016/j.jsp.2012.08.004. [DOI] [PubMed] [Google Scholar]
  8. Boyd A, Golding J, Macleod J, Lawlor DA, Fraser A, Henderson J, Davey Smith G. Cohort Profile: the 'children of the 90s'–the index offspring of the Avon Longitudinal Study of Parents and Children. International Journal of Epidemiology. 2013;42(1):111–127. doi: 10.1093/ije/dys064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Brikell I, Larsson H, Lu Y, Pettersson E, Chen Q, Kuja-Halkola R, Martin J. The contribution of common genetic risk variants for ADHD to a general factor of childhood psychopathology. Molecular Psychiatry. 2020 doi: 10.1038/s41380-018-0109-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Brodbeck J, Fassbinder E, Schweiger U, Fehr A, Spath C, Klein JP. Differential associations between patterns of child maltreatment and comorbidity in adult depressed patients. Journal of Affective Disorders. 2018;230:34–41. doi: 10.1016/j.jad.2017.12.077. [DOI] [PubMed] [Google Scholar]
  11. Caron C, Rutter M. Comorbidity in child psychopathology: concepts, issues and research strategies. Journal of Child Psychology and Psychiatry and Allied Disciplines. 1991;32(7):1063–1080. doi: 10.1111/j.1469-7610.1991.tb00350.x. [DOI] [PubMed] [Google Scholar]
  12. Caspi A, Houts RM, Belsky DW, Goldman-Mellor SJ, Harrington H, Israel S, Moffitt TE. The p Factor: One General Psychopathology Factor in the Structure of Psychiatric Disorders? Clin Psychol Sci. 2014;2(2):119–137. doi: 10.1177/2167702613497473. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Cecil CA, Lysenko LJ, Jaffee SR, Pingault JB, Smith RG, Relton CL, Barker ED. Environmental risk, Oxytocin Receptor Gene (OXTR) methylation and youth callous-unemotional traits: a 13-year longitudinal study. Molecular Psychiatry. 2014;19(10):1071–1077. doi: 10.1038/mp.2014.95. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dunn, E. C., Nishimi, K., Neumann, A., Renaud, A., Cecil, C. A. M., Susser, E. S., & Tiemeier, H. (2020). Time-Dependent Effects of Exposure to Physical and Sexual Violence on Psychopathology Symptoms in Late Childhood: In Search of Sensitive Periods in Development. Journal of the American Academy of Child and Adolescent Psychiatry, 59(2), 283–295. 10.1016/j.jaac.2019.02.022. [DOI] [PMC free article] [PubMed]
  15. Fraser A, Macdonald-Wallis C, Tilling K, Boyd A, Golding J, Davey Smith G, Lawlor DA. Cohort Profile: the Avon Longitudinal Study of Parents and Children: ALSPAC mothers cohort. International Journal of Epidemiology. 2013;42(1):97–110. doi: 10.1093/ije/dys066. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gibbons RD, Hedeker DR. Full-information item bifactor analysis. Psychometrika. 1992;57:423–436. doi: 10.1007/BF02295430. [DOI] [Google Scholar]
  17. Gluschkoff, K., Jokela, M., & Rosenstrom, T. (2019). The General Psychopathology Factor: Structural Stability and Generalizability to Within-Individual Changes. Frontiers in Psychiatry, 10. 10.3389/fpsyt.2019.00594. [DOI] [PMC free article] [PubMed]
  18. Goodman A, Heiervang E, Collishaw S, Goodman R. The 'DAWBA bands' as an ordered-categorical measure of child mental health: description and validation in British and Norwegian samples. Social Psychiatry and Psychiatric Epidemiology. 2011;46(6):521–532. doi: 10.1007/s00127-010-0219-x. [DOI] [PubMed] [Google Scholar]
  19. Goodman R, Ford T, Richards H, Gatward R, Meltzer H. The Development and Well-Being Assessment: description and initial validation of an integrated assessment of child and adolescent psychopathology. Journal of Child Psychology and Psychiatry. 2000;41(5):645–655. doi: 10.1111/j.1469-7610.2000.tb02345.x. [DOI] [PubMed] [Google Scholar]
  20. Hamburger, M. E., Basile, K. C., & Vivola, A. M. (2011). Measuring bullying victimization, perpetration, and bystander experiences: A compendium of assessment tools. Atlanta, GA: Center for Disease Control and Prevention, National Center for Injury Prevention and Control. http://www.cdc.gov/violenceprevention/pdf/BullyCompendiumBk-a.pdf.
  21. Hodges EV, Perry DG. Personal and interpersonal antecedents and consequences of victimization by peers. Journal of Personality and Social Psychology. 1999;76(4):677–685. doi: 10.1037/0022-3514.76.4.677. [DOI] [PubMed] [Google Scholar]
  22. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equation Model. 1999;6:1–55. doi: 10.1080/10705519909540118. [DOI] [Google Scholar]
  23. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry. 2005;62(6):617–627. doi: 10.1001/archpsyc.62.6.617. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kessler RC, Davis CG, Kendler KS. Childhood adversity and adult psychiatric disorder in the US National Comorbidity Survey. Psychological Medicine. 1997;27(5):1101–1119. doi: 10.1017/s0033291797005588. [DOI] [PubMed] [Google Scholar]
  25. Kim KJ, Conger RD, Elder GH, Jr, Lorenz FO. Reciprocal influences between stressful life events and adolescent internalizing and externalizing problems. Child Development. 2003;74(1):127–143. doi: 10.1111/1467-8624.00525. [DOI] [PubMed] [Google Scholar]
  26. Klomek AB, Sourander A, Elonheimo H. Bullying by peers in childhood and effects on psychopathology, suicidality, and criminality in adulthood. Lancet Psychiatry. 2015;2(10):930–941. doi: 10.1016/S2215-0366(15)00223-0. [DOI] [PubMed] [Google Scholar]
  27. Krueger RF. The structure of common mental disorders. Archives of General Psychiatry. 1999;56(10):921–926. doi: 10.1001/archpsyc.56.10.921. [DOI] [PubMed] [Google Scholar]
  28. Lahey BB, Rathouz PJ, Keenan K, Stepp SD, Loeber R, Hipwell AE. Criterion validity of the general factor of psychopathology in a prospective study of girls. Journal of Child Psychology and Psychiatry and Allied Disciplines. 2015;56(4):415–422. doi: 10.1111/jcpp.12300. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Lereya ST, Copeland WE, Costello EJ, Wolke D. Adult mental health consequences of peer bullying and maltreatment in childhood: two cohorts in two countries. Lancet Psychiatry. 2015;2(6):524–531. doi: 10.1016/S2215-0366(15)00165-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Monks CP, Smith PK, Naylor P, Barter C, Ireland JL, Coyne I. Bullying in different contexts: Commonalities, differences and the role of theory. Aggression and Violent Behavior. 2009;14(2):146–156. doi: 10.1016/j.avb.2009.01.004. [DOI] [Google Scholar]
  31. Moore SE, Norman RE, Suetani S, Thomas HJ, Sly PD, Scott JG. Consequences of bullying victimization in childhood and adolescence: A systematic review and meta-analysis. World J Psychiatry. 2017;7(1):60–76. doi: 10.5498/wjp.v7.i1.60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Muthén, L. K., & Muthén, B. O. (1998–2016). Mplus. The comprehensive modeling program for applied researchers. User's guide. (version 7.22). Los Angeles, CA: Author.
  33. Nansel TR, Overpeck M, Pilla RS, Ruan WJ, Simons-Morton B, Scheidt P. Bullying behaviors among US youth - Prevalence and association with psychosocial adjustment. Jama-Journal of the American Medical Association. 2001;285(16):2094–2100. doi: 10.1001/jama.285.16.2094. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Neumann, A., Pappa, I., Lahey, B. B., Verhulst, F. C., Medina-Gomez, C., Jaddoe, V. W., Tiemeier, H. (2016). Single Nucleotide Polymorphism Heritability of a General Psychopathology Factor in Children. Journal of the American Academy of Child and Adolescent Psychiatry, 55(12), 1038–1045. 10.1016/j.jaac.2016.09.498. [DOI] [PubMed]
  35. Olino TM, Dougherty LR, Bufferd SJ, Carlson GA, Klein DN. Testing Models of Psychopathology in Preschool-aged Children Using a Structured Interview-based Assessment. Journal of Abnormal Child Psychology. 2014;42(7):1201–1211. doi: 10.1007/s10802-014-9865-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Patalay P, Fonagy P, Deighton J, Belsky J, Vostanis P, Wolpert M. A general psychopathology factor in early adolescence. British Journal of Psychiatry. 2015;207(1):15–22. doi: 10.1192/bjp.bp.114.149591. [DOI] [PubMed] [Google Scholar]
  37. Perry JL, Nicholls AR, Clough PJ, Crust L. Assessing Model Fit: Caveats and Recommendations for Confirmatory Factor Analysis and Exploratory Structural Equation Modeling. Measurement in Physical Education and Exercise Science. 2015;19(1):12–21. doi: 10.1080/1091367x.2014.952370. [DOI] [Google Scholar]
  38. Pettersson E, Lahey BB, Larsson H, Lichtenstein P. Criterion Validity and Utility of the General Factor of Psychopathology in Childhood: Predictive Associations With Independently Measured Severe Adverse Mental Health Outcomes in Adolescence. Journal of the American Academy of Child and Adolescent Psychiatry. 2018;57(6):372–383. doi: 10.1016/j.jaac.2017.12.016. [DOI] [PubMed] [Google Scholar]
  39. Reijntjes A, Kamphuis JH, Prinzie P, Telch MJ. Peer victimization and internalizing problems in children: a meta-analysis of longitudinal studies. Child Abuse and Neglect. 2010;34(4):244–252. doi: 10.1016/j.chiabu.2009.07.009. [DOI] [PubMed] [Google Scholar]
  40. Riglin L, Thapar AK, Leppert B, Martin J, Richards A, Anney R, Thapar A. Using Genetics to Examine a General Liability to Childhood Psychopathology. Behavior Genetics. 2019 doi: 10.1007/s10519-019-09985-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Rodriguez A, Reise SP, Haviland MG. Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures. Journal of Personality Assessment. 2016;98(3):223–237. doi: 10.1080/00223891.2015.1089249. [DOI] [PubMed] [Google Scholar]
  42. Rodriguez A, Reise SP, Haviland MG. Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods. 2016;21(2):137–150. doi: 10.1037/met0000045. [DOI] [PubMed] [Google Scholar]
  43. Rosseel Y. lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software. 2012;48(2):1–36. doi: 10.18637/jss.v048.i02. [DOI] [Google Scholar]
  44. Salmivalli C. Bullying and the peer group: A review. Aggression and Violent Behavior. 2010;15(2):112–120. doi: 10.1016/j.avb.2009.08.007. [DOI] [Google Scholar]
  45. Schoeler T, Choi SW, Dudbridge F, Baldwin J, Duncan L, Cecil CM, Pingault JB. Multi-Polygenic Score Approach to Identifying Individual Vulnerabilities Associated With the Risk of Exposure to Bullying. JAMA Psychiatry. 2019;76(7):730–738. doi: 10.1001/jamapsychiatry.2019.0310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Schoeler T, Duncan L, Cecil CM, Ploubidis GB, Pingault JB. Quasi-experimental evidence on short- and long-term consequences of bullying victimization: A meta-analysis. Psychological Bulletin. 2018;144(12):1229–1246. doi: 10.1037/bul0000171. [DOI] [PubMed] [Google Scholar]
  47. Singham T, Viding E, Schoeler T, Arseneault L, Ronald A, Cecil CM, Pingault JB. Concurrent and Longitudinal Contribution of Exposure to Bullying in Childhood to Mental Health: The Role of Vulnerability and Resilience. JAMA Psychiatry. 2017;74(11):1112–1119. doi: 10.1001/jamapsychiatry.2017.2678. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Thomas HJ, Connor JP, Lawrence DM, Hafekost JM, Zubrick SR, Scott JG. Prevalence and correlates of bullying victimisation and perpetration in a nationally representative sample of Australian youth. Australian and New Zealand Journal of Psychiatry. 2017;51(9):909–920. doi: 10.1177/0004867417707819. [DOI] [PubMed] [Google Scholar]
  49. Vachon DD, Krueger RF, Rogosch FA, Cicchetti D. Assessment of the Harmful Psychiatric and Behavioral Effects of Different Forms of Child Maltreatment. JAMA Psychiatry. 2015;72(11):1135–1142. doi: 10.1001/jamapsychiatry.2015.1792. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. van Lier PA, Vitaro F, Barker ED, Brendgen M, Tremblay RE, Boivin M. Peer victimization, poor academic achievement, and the link between childhood externalizing and internalizing problems. Child Development. 2012;83(5):1775–1788. doi: 10.1111/j.1467-8624.2012.01802.x. [DOI] [PubMed] [Google Scholar]
  51. Wolke D, Lereya ST, Fisher HL, Lewis G, Zammit S. Bullying in elementary school and psychotic experiences at 18 years: a longitudinal, population-based cohort study. Psychological Medicine. 2014;44(10):2199–2211. doi: 10.1017/S0033291713002912. [DOI] [PubMed] [Google Scholar]
  52. Wolke D, Schreier A, Zanarini MC, Winsper C. Bullied by peers in childhood and borderline personality symptoms at 11 years of age: a prospective study. Journal of Child Psychology and Psychiatry. 2012;53(8):846–855. doi: 10.1111/j.1469-7610.2012.02542.x. [DOI] [PubMed] [Google Scholar]
  53. Wolke D, Waylen A, Samara M, Steer C, Goodman R, Ford T, Lamberts K. Selective drop-out in longitudinal studies and non-biased prediction of behaviour disorders. British Journal of Psychiatry. 2009;195(3):249–256. doi: 10.1192/bjp.bp.108.053751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Wolke D, Woods S, Bloomfield L, Karstadt L. The association between direct and relational bullying and behaviour problems among primary school children. Journal of Child Psychology and Psychiatry. 2000;41(8):989–1002. doi: 10.1111/1469-7610.00687. [DOI] [PubMed] [Google Scholar]
  55. Zwierzynska K, Wolke D, Lereya TS. Peer victimization in childhood and internalizing problems in adolescence: a prospective longitudinal study. Journal of Abnormal Child Psychology. 2013;41(2):309–323. doi: 10.1007/s10802-012-9678-8. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary file1 (PDF 249 KB) (249.7KB, pdf)
Supplementary file2 (XLSX 108 KB) (107.2KB, xlsx)

Articles from Research on Child and Adolescent Psychopathology are provided here courtesy of Springer

RESOURCES