Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2010 Jun 1.
Published in final edited form as: Arch Gen Psychiatry. 2009 Jun;66(6):640–648. doi: 10.1001/archgenpsychiatry.2008.554

Environmental Adversity Increases Genetic Risk for Externalizing Disorders

Brian M Hicks 1, Susan C South 1, Ana C DiRago 1, William G Iacono 1, Matt McGue 1
PMCID: PMC2717707  NIHMSID: NIHMS72067  PMID: 19487629

Abstract

Background

Studies of gene-environment (G-E) interplay in the development of psychiatric and substance use disorders are rapidly accumulating. However, few attempts have been made to integrate findings and articulate general mechanisms of G-E influence in the emergence of psychopathology.

Objective

Identify patterns of G-E interplay between externalizing (EXT; antisocial behavior and substance use) disorders and several environmental risk factors.

Design

We used quantitative genetic models to examine how genetic and environmental risk for EXT disorders changes as a function of environmental context.

Setting

Participants were recruited from the community and took part in a day-long assessment at a university laboratory.

Participants

The sample consisted of 1315 male and female twin pairs participating in the age 17 assessment of the Minnesota Twin Family Study.

Main Outcome Measures

Multiple measures and informants were employed to construct a composite of EXT disorders and composite measures of 6 environmental risk factors including academic achievement and engagement, antisocial and prosocial peer affiliation, mother-child and father-child relationship problems, and stressful life events.

Results

A significant G × E interaction was detected between each environmental risk factor and EXT such that greater environmental adversity was associated with increased genetic risk in EXT.

Conclusion

Our findings demonstrate that in the context of environmental adversity, genetic factors become more important in the etiology of EXT disorders. The consistency of the results further suggests a general mechanism of environmental influence on EXT disorders regardless of the specific form of the environmental risk.

Various lines of evidence testify to the fact that both genetic and environmental factors contribute to psychiatric and substance use disorders1. Recently, psychiatric genetic research has evolved beyond simple estimates of heritable and non-heritable influences to investigations that begin to delineate the mechanisms of gene-environment (G-E) interplay. This includes studies utilizing the specific gene × measured environment design2, the most well known example being that variants of the 5-HTT gene increase risk for major depression in the context of stressful life events3,4. Additionally, there is an accumulating literature of quantitative genetic studies that utilize twin, adoption, and family designs to delineate processes of G-E interplay such as how the relative contribution of genetic and environmental risk factors changes as a function of the environmental context5-9.

While there has been a veritable explosion in studies of G-E interplay in psychopathology in recent years, there have been relatively few attempts to integrate findings in an effort to articulate more general principles of G-E influence across different environmental variables and psychiatric disorders10-12. Key questions that remain unanswered include: Is the mechanism of environmental influence the same regardless of the environmental variable such as parenting, peers, or stressful life events? Or, do different environmental variables exhibit varying mechanisms of influence? Also, are the mechanisms of G-E influence the same for all types of psychiatric disorders? For example, are processes of G-E interplay the same or different for internalizing (INT; major depression and anxiety disorders) versus externalizing (EXT; antisocial behavior and substance use) disorders?

We begin to answer these questions by examining G-E interplay processes between EXT disorders and 6 environmental variables in a large adolescent twin sample. By late adolescence, genetic risk for EXT disorders is largely non-specific, and primarily attributable to a highly heritable general vulnerability dimension (h2 = .80)14-16. However, EXT disorders also exhibit strong associations with environmental variables such as poor performance and lack of engagement in school17,18, deviant peer affiliation19, harsh discipline20,21 and poor parental monitoring22,23, and various stressful life events (e.g., poverty, parental discord, residential instability, familial psychopathology)24. The mechanisms underlying these associations, however, are less well delineated25. For example, do these environmental risk factors cause EXT symptoms, or does genetic risk for EXT instigate selection processes that result in greater exposure to these environmental risk factors?

Two G-E processes are essential to understanding the etiology of psychiatric disorders. G-E correlations are the first process, and refer to the fact that environmental risk is not distributed randomly but rather is, to some extent, a result of an individual's decisions and actions (specifically, active and evocative G-E correlations)25-28. For example, the behavior of pre-adolescents who exhibit childhood disruptive disorders or adolescents who precociously use substances can initiate a cascade of experiences including weakened attachment to and increased conflict with parents29,30, poor performance and lack of engagement in school17,18,31, and stronger affiliation with deviant peers32, which increases risk of developing psychiatric and substance use disorders in adulthood. G-E correlations also help account for the finding that nearly every putatively “environmental” measure exhibits heritable variance33,34. That is, an individual's genetically influenced characteristics such as personality and intelligence helps to shape their environment, including exposure to environmental risk factors which then increases risk for psychiatric disorders25,26.

G × E interactions are the second process, and refer to the finding that rather than being uniform across individuals, environmental effects seem to be most influential among a subset of individuals who are genetically at high-risk, or, alternatively, that the impact of genetic influences varies depending on the environmental context. For example, stressful life events are a risk factor for major depression, but this risk is greater if an individual carries multiple copies of the short allele of the 5-HTT gene3. Twin studies can also be utilized to identify G × E interactions. For example, twin studies have shown that parental monitoring22, rural residence35, marriage36, and religiousness37 attenuate genetic influences on substance use and abuse; the corollary being that weaker environmental constraints amplify genetic influences. This provides for the more general hypothesis that regardless of the specific mechanism, greater environmental stress will increase genetic risk for maladaptive behaviors38. Therefore, while quantitative genetic studies do not identify specific risk alleles, they can serve as a conceptual framework for measured gene × measured environment designs by identifying the environments in which genetic risk is suppressed or amplified38.

Recently, more sophisticated quantitative genetic models have been developed that incorporate both G-E correlation and G × E interaction39, a notable advance as the presence of a G-E correlation can confound the interpretation of a G × E interaction11,25. In this investigation, we utilized these quantitative genetic models to delineate mechanisms of G-E interplay in the developmental of EXT disorders during late adolescence, a critical development period when etiological processes are shifting from those of childhood to those of adulthood27,40.

Additionally, we examined the effects of multiple environmental risk factors that have robust and well-replicated associations with EXT. Finally, both the environmental risk factors and EXT were assessed using multiple methods and informants providing for excellent measurement of the constructs. With these various methodological strengths, we sought not just to detect a G × E interaction, but (1) to identify patterns of G-E interplay across the various environmental risk factors and EXT, and, (2) to begin to articulate a general model of G-E interplay in the development of EXT disorders.

Method

Sample

Participants were adolescent male and female twins taking part in the ongoing Minnesota Twin Family Study (MTFS). The MTFS is an epidemiological-longitudinal study of the families of same-sex twin pairs born in Minnesota that was designed to investigate the etiology of substance use disorders and related conditions. A comprehensive description of the design and methods of the MTFS has been provided elsewhere41,42. Briefly, the MTFS includes a younger cohort first assessed at age 11, and an older cohort who began the study at age 17. Twin participants are then offered the opportunity to return for follow-up assessments every 3-4 years. Families were initially located using publicly available birth records and databases, targeting the birth years from 1972 to 1984. Over 90% of eligible twin families were successfully located for each target birth year. Families are representative of the Minnesota population for the target birth years in terms of parental occupational status, educational attainment, and history of mental health treatment. Consistent with the demographics of those born in Minnesota during the target years, 96% of the participants were Caucasian. All twins and their parents provided informed consent or assent (with a parent providing informed consent) as appropriate prior to participation, and an internal review board approved all study protocols.

Data for the current investigation was collected as part of the age 17 assessment for both cohorts (i.e., intake assessment for the older cohort and the second follow-up assessment for the younger cohort). The total sample included 1315 twin pairs: 437 monozygotic (MZ) and 251 dizygotic (DZ) male twin pairs, and 418 MZ and 209 DZ female twin pairs. The mean age of the total sample at the time of assessment was 17.8 years (SD = .68 years). Zygosity was determined by the agreement of 3 separate reports: a standard zygosity questionnaire completed by parents, evaluation of the physical resemblance of the twin pairs including hair and eye color and face and ear shape completed by MTFS staff, and an algorithm assessing the similarity of twins on ponderal and cephalic indices and fingerprint ridge counts. A serological analysis was performed if these estimates did not agree.

Measures

Data was primarily collected during the day-long, in-person assessment at the Department of Psychology, University of Minnesota. The assessment included structured clinical interviews conducted by trained staff with either a bachelor's or master's degree in psychology; self-report questionnaires completed by twins and parents; and a rating form completed by teachers nominated by the family as being knowledgeable about the twin's behavior.

Our measures of externalizing disorders included DSM-III-R43 symptoms of adult antisocial behavior and alcohol, nicotine, and illicit drug dependence. Adult antisocial behavior (i.e., the adult criteria for antisocial personality disorder) was assessed using a modified version of the Structured Clinical Interview for Axis II Personality Disorders44. Substance use disorders were assessed using the Substance Abuse Module of the Composite International Diagnostic Interview45. The drug assessment included amphetamines, cannabis, cocaine, hallucinogens, inhalants, opioids, PCP, and sedatives. The drug class for which the participant endorsed the most symptoms was used as their number of drug dependence symptoms. Mothers also reported on the presence of symptoms of all substance use disorders in their twin offspring using the parent version of the Diagnostic Interview for Children and Adolescents-Revised46. A symptom was considered present if reported by either the twin or mother. Symptoms of abuse were included in the alcohol and drug symptom count variables. All interview data were reviewed in a case conference consisting of at least two advanced graduate students in clinical psychology. Consensus between the diagnosticians regarding the presence or absence of symptoms was reached prior to assigning symptoms, referring to audio tapes of the interview when necessary. The consensus process yielded uniformly high diagnostic kappa reliabilities of .95 for adult antisocial behavior and > .91 for each substance use disorder.

In addition to the structured interviews, ratings were obtained from up to 3 teachers on a 30-item scale of overall externalizing behavior (e.g., items similar to the criteria for oppositional defiant disorder and conduct disorder; 76.3% of the sample had ratings from at least one teacher and 61.3% had ratings from at least two teachers). The internal consistency reliability for the teacher rating was .92; the inter-rater reliability (intraclass correlation) for the mean of 2 raters was .71. Minnesota schools have a policy of placing twins in separate classrooms whenever possible, thereby minimizing the likelihood that members of a twin pair would be rated by the same teacher. All statistical analyses utilized an EXT composite variable that was calculated by taking the mean z-score of the symptom counts of adult antisocial behavior, nicotine dependence, alcohol abuse/dependence, drug abuse/dependence, and the teacher rating of externalizing behavior.

We also assessed multiple domains of each twin's environmental context including: academic achievement and engagement, peer affiliation, parent-child relationships, and stressful life events. Academic achievement and engagement was a composite of twin and mother reports regarding the twin's cumulative grade point average (GPA); self and maternal ratings regarding their expectation of the twin's ultimate educational attainment (e.g., complete high school, complete college, etc.); and a 7-item scale completed by the twin and mother assessing the twin's attitudes and engagement in school (e.g., good attitude about school, enjoys attending school; α = .83)47. The measure of academic achievement and engagement used in the analyses was the mean z-score for ratings of GPA, academic expectations, and academic attitudes averaged across the twin and mother reports (r = .77). While academic achievement and engagement can also be conceptualized as an outcome or individual differences variable, in the current investigation, the construct serves as an indicator of the individual's broader environmental context (in particular the context actively or evocatively shaped by the individual)48 that might amplify or suppress risk for EXT.

Peer affiliation was assessed by twin and teacher reports49. Twins completed a 19-item questionnaire assessing antisocial (e.g., my friends smoke, drink alcohol, steal, get in fights; α = .85) and prosocial (e.g., my friends work hard in school, popular with other kids, liked by teachers; α = .78) peer affiliation. Teachers completed similar ratings regarding the twin's antisocial (α = .85) and prosocial (α = .87) peer affiliation (average inter-rater reliability was .71). The mean z-score of the twin and teacher reports (r = .40) was used to calculate composite measures of antisocial and prosocial peer affiliation.

Quality of parent-child relationships was assessed using the Parental Environment Questionnaire (PEQ), a 50-item self-report questionnaire that assesses multiple dimensions of the parent-child relationship (e.g., conflict, involvement; scale α's range from .82 to .69)50,51. Twins completed separate PEQ ratings regarding their relationship with their mother and father. Parents also rated their relationship with each twin as well as the quality of the relationship between each twin and the other parent (e.g., mother rated the relationship between each twin and their father). Therefore, up to 3 ratings were available for the mother-child and father-child relationship. The measure of mother-child and father-child relationship problems used in the analyses was the mean of the 3 informant ratings on the first principal component among the PEQ scales (the scales exhibit a dominant 1st component; mean correlation across informants was .41).

Finally, stressful life events were assessed via a structured interview administered to each twin52. Our analyses are limited to what are referred to as “independent” life events53. That is, these events are largely independent of the individual's behavior (e.g., parent lost a job) as opposed to being in some way dependent on the respondent's behavior (e.g., failed a class). The stressful life events measure for this analysis was a tally of 18 life events covering the domains of parental divorce and discord as well as family money, legal, and mental health problems. As these events should be concordant for members of a twin pair, the correlation between twin reports provides an estimate of reliability (r = .81; inter-rater reliability was .89).

Analytic Approach

Structural equation modeling was used to examine G-E interplay between the EXT composite and 6 environmental risk factors. First, we used univariate biometric models to estimate the relative genetic and environmental effects on each variable54. These models assume variance in each measure is attributable to 3 independent sources: additive genetic (A), shared environment (C), and nonshared environment (E). Estimates of the ACE variance components are derived by comparing the similarity of members of MZ twin pairs relative to that of members of DZ twin pairs, given that MZ twins share all their genes and DZ twins share on average 50% of their segregating genes (A effects are present if rMZ > rDZ). Shared environmental effects refer to environmental effects that increase similarity among family members (C effects are present if rDZ > ½ rMZ) while nonshared environmental effects (including measurement error) contribute to differences between members of a twin pair (E effects are present if rMZ < 1.0).

Next, we examined whether the ACE effects on the EXT composite changed as a function of the environmental context, operationalized as different levels of the environmental risk factors. This G-E interplay model (depicted in Figure 1) can be conceptualized as a bivariate extension of the univariate mode that incorporates additional moderation terms on the ACE effects39. The first part of this G-E interplay model estimates the ACE contributions to the covariance between EXT and the environmental risk factor (e.g., the genetic covariance is a11× a21), as well as the ACE effects that are unique to EXT (e.g., a22). This allows for the estimation of the G-E correlation between EXT and the environmental variable, which accounts for any selection effects between EXT and exposure to the environmental risk factor.

Figure 1.

Figure 1

Open in a new tab

EXT = Externalizing disorders composite. Full model of moderation of the genetic and environmental influences on EXT as a function of different levels of Antisocial Peers. A refers to additive genetic effects, C to shared environmental effects, and E to nonshared environmental effects. The parameters a21, c21, and e21 include genetic and environmental influences that overlap between Antisocial Peers and EXT (i.e., can be used to derive the genetic and environmental covariance), while a22, c22, and e22 are genetic and environmental influences unique to EXT. Antisocial Peers can moderate either the common variance with EXT or the unique variance of EXT. The β's indicate the direction (+ or -) and magnitude of any moderation effects on the paths from the ACE effects to EXT, while M indicates the level of the moderator, that is, the number of antisocial peers.

If a G × E interaction is present, the initial ACE parameters derived from the bivariate model are then adjusted based on the direction (+ or -) and size of the moderation weight (βxij) and the level of the moderator (M; e.g., number of antisocial peers). Moderation can occur for either the ACE effects that overlap between EXT and the environmental risk factor (e.g., a21 + βa21*M) or those ACE effects that are unique to EXT (e.g., a22 + βa22*M). For example, a positive sign for βa22 would mean that the unique genetic variance in EXT would increase in the context of more antisocial peers.

Models were fit to the raw data using full information maximum likelihood estimation as implemented in the computer program Mx55 which allows for missing data and yields less biased parameter estimates than other procedures56. The EXT composite was log transformed to better approximate normality. We followed standard behavior genetic analytic procedures by regressing all variables on sex, age, age2, and the interactions of sex and the age variables prior to analyses57.

Model fit was evaluated using the –2 × loglikelihood (-2LL) and several information theoretic indices that balance overall fit with model parsimony including Akaike's Information Criterion (AIC)58, the Bayesian Information Criterion (BIC)59, the sample size adjusted Bayesian Information Criterion (BICadj)60,61, and the Deviance Information Criterion (DIC)62. The full G × E moderation model was compared to the no moderation model via the likelihood ratio test, which yields a χ2 statistic. If the full moderation model provided a better fit to the data, further model trimming analyses were conducted to identify the most parsimonious model, retaining only the significant moderation effects. The best fitting model was judged to be the model that yielded lower values for at least 3 of the 4 information theoretic indices relative to the full model.

Results

Phenotypic Correlations, Twin Correlations, and Univariate Biometric Analyses

Table 1 presents the phenotypic correlations among the study variables. Each environmental risk factor was significantly correlated with EXT (i.e., exhibited a main effect). Notably, all the environmental risk factors were significantly correlated with each other indicating adversity in one domain tended to be associated with problems in other domains. However, there was a wide range in the strength of the associations among the environmental risk factors (r = .18 to .57), and none was so high that any measure would be considered redundant.

Table 1.

Phenotypic Correlations Between Externalizing and Environmental Risk Factors.

Variable 1 2 3 4 5 6
1. Externalizing --
2. Academic Achievement & Engagement .52 --
3. Antisocial Peers .67 .49 --
4. Proscoial Peers .38 .48 .40 --
5. Mother-child Relationship Problems .27 .31 .24 .24 --
6. Father-child Relationship Problems .32 .29 .27 .24 .57 --
7. Stressful Life Events .31 .20 .25 .20 .18 .38
Open in a new tab

Note. All correlations are significant at p < .001. Academic Achievement & Engagement and Prosocial Peers have been reverse scored.

Table 2 presents the MZ and DZ twin correlations for each study variable as well as the estimates of the ACE variance components from the univariate biometric model. EXT was highly heritable with no shared environmental effects. Each environmental variable exhibited statistically significant heritable variance though there was a wide range (.08 to .72). With the exception of academic achievement and engagement, each environmental variable also exhibited significant shared and nonshared environmental variance.

Table 2.

Twin Correlations and Univariate ACE estimates (95% Confidence Intervals).

Variable MZ DZ A C E
Externalizing .74 .40 .76 (.65, .79) .00 (.00, .11) .24 (.21, .26)
Academic Achievement & Engagement .77 .42 .72 (.57, .79) .05 (.00, .19) .23 (.20, .26)
Antisocial Peers .70 .51 .42 (.28, .58) .28 (.13, .42) .30 (.26, .33)
Prosocial Peers .66 .44 .41 (.25, .59) .24 (.08, .39) .34 (.31, .38)
Mother-child Relationship Problems .79 .56 .49 (.38, .62) .30 (.17, .41) .21 (.18, .23)
Father-child Relationship Problems .83 .75 .23 (.15, .31) .62 (.54, .69) .15 (.14, .18)
Stressful Life Events .81 .80 .08 (.01, .15) .75 (.68, .80) .17 (.16, .20)
Open in a new tab

Note. MZ = monozygotic; DZ = dizygotic; A = additive genetic; C = shared environment; E = nonshared environment.

G-E Interplay of EXT and Environmental Risk Factors

Table 3 presents the fit statistics for the models of G-E interplay between EXT and each environmental variable. Each environmental variable exerted large moderation effects on EXT as evidenced by the highly significant likelihood ratio tests. Follow-up model trimming analyses showed that each environmental variable moderated the unique additive genetic and nonshared environment variance in EXT, with 3 variables also moderating the unique shared environmental variance. These moderating effects on the C component necessitated retaining it in the model despite the univariate estimate of C on EXT being zero. Academic achievement and engagement and both peer affiliation variables also moderated the common additive genetic variance with EXT (prosocial peers also moderated the common nonshared environmental variance). Because members of a twin pair should not differ on the life events that constitute our measure of independent stressful life events, no genetic effect is expected, so it is inappropriate to estimate the G-E correlation63. Therefore, for independent stressful life events, we only allowed for moderation on the unique variance in EXT. While not ideal, this approach allowed us to examine at least some of the effect of highly relevant environmental stressors (e.g., parental discord, poverty, family legal and mental health problems), and whether the mechanism of their effect was similar or different to that of other environmental risk factors.

Table 3.

Fit Statistics for G × E Interplay Models of EXT and Environmental Moderators

Model -2LL df AIC BICadj DIC BIC Δχ2 df p
Academic Achievement & Engagement
Full G × E 10,570.94 4702 1166.94 -3874.17 -7020.94 -11,341.79 -- --
No G × E 11,098.09 4708 1682.09 -3622.29 -6773.07 -11,099.44 527.15 6 < .001
Moderation effects for best fit: common A, unique ACE 10,573.13 4704 1165.13 -3876.98 -7025.09 -11,347.77 2.19 2 .33
Antisocial Peers
Full G × E 9111.65 4374 363.65 -3807.49 -6734.47 -10,753.91 -- --
No G × E 9990.19 4380 1230.19 -3379.69 -6310.69 -10,335.64 878.54 6 <.001
Moderation effects for best fit: common A, unique AE 9111.94 4377 357.94 -3813.08 -6742.07 -10,764.26 .29 3 .96
Prosocial Peers
Full G × E 10,742.34 4502 1738.34 -3301.49 -6314.25 -10,451.31 -- --
No G × E 11,108.35 4508 2092.35 -3130.05 -6146.82 -10,289.39 366.01 6 <.001
Moderation effects for best fit: common AE, unique AE 10,744.97 4504 1736.97 -3304.03 -6318.12 -10,457.02 2.63 2 .27
Mother-Child Relationship Problems
Full G × E 11,840.35 4914 2012.35 -3760.32 -7049.14 -11,564.81 -- --
No G × E 12,066.63 4920 2226.63 -3658.99 -6951.84 -11,473.01 226.29 6 < .001
Moderation effects for best fit: unique ACE 11,845.51 4917 2011.51 -3763.64 -7054.48 -11,572.90 5.16 3 .16
Father-Child Relationship Problems
Full G × E 11,240.88 4910 1420.88 -4050.18 -7336.32 -11,848.31 -- --
No G × E 11,641.04 4916 1809.04 -3861.91 -7152.08 -11,669.58 400.16 6 < .001
Moderation effects for best fit: unique AE 11,250.41 4914 1422.41 -4053.29 -7342.12 -11,857.78 9.53 4 .02
Stressful Life Events
Moderation effects for best fit: unique ACE 11,385.86 4893 1599.86 -3934.26 -7209.02 -11,705.38 -- --
No G × E 11,634.77 4896 1842.77 -3815.71 -7092.47 -11,591.60 248.91 3 < .001
Open in a new tab

Note. Full G × E refers to the gene-environment interplay model with all possible moderation parameters. No G × E refers to a bivariate biometric model with no moderation parameters. Lower values for all fit indices are indicative of better fit. The model for stressful life events only allowed for moderation of the unique variance EXT (see text for details), which was also the best fitting model. -2LL = -2 × loglikelihood; df = degrees of freedom; AIC= Akaike's Information Criterion; BICadj= sample-sized adjusted Bayesian Information Criterion; DIC = Deviance Information Criterion; BIC = Bayesian Information Criterion; χ2 = chi square; A = additive genetic; C = shared environment; E = nonshared environment.

Figure 2 illustrates the mechanisms of G × E interaction on EXT for each environmental moderator. All environmental variables have been coded such that higher values are indicative of greater adversity. For each environmental variable, greater environmental adversity was associated with substantially greater additive genetic variance in EXT, with modest to moderate increases in nonshared environmental variance. Note that the figure depicts the unstandardized variance estimates so that the ACE variance components do not necessarily sum to 1.0. For 3 environmental variables, there was also a modest moderation effect on the shared environmental variance with shared environmental variance tending to be greatest at low levels of environmental adversity. Given the consistency of these results, a plausible hypothesis is that moderation of EXT is due to the overlap among the different environmental variables. To test this hypothesis, we regressed each environmental risk factor on the 1st principal component among the 6 environmental variables, and examined whether the residual variance continued to moderate the ACE effects on EXT. While the common variance across the environmental variables accounted for a large portion of the moderation effects, the residual variance of each environmental variable continued to exert significant moderating effects on EXT, χ2(6) = 28.6 to 195.2, all p's < .001.

Figure 2.

Figure 2

Open in a new tab

Changes in the unstandardized variance of Externalizing (EXT) as a function of environmental risk factors for the best fitting model. All environmental risk factors have been coded so that higher levels are associated with greater environmental adversity.

Because the additive genetic and nonshared environmental variance of EXT was moderated, the size of the genetic (rA) and nonshared environmental (rE) correlations (i.e., the G-E correlations) also varied with levels of the environmental risk factor (this situation holds even if only the unique variance of EXT is moderated)38,39. As results were consistent across the environmental moderators, we discuss results for mother-child relationship problems as an illustrative example. At low levels of mother-child relationship problems (-2 SD), the rA with EXT was large (.78), but declined as levels of mother-child relationship problems increased (rA = .44 and .29 at 0 SD and +2 SD, respectively). Results for the rE were similar though of lesser magnitude (rE = .31, .16, and .10 at levels of mother-child relationship problems of -2 SD, 0 SD, and +2 SD, respectively). This indicates that as environmental adversity increases, EXT and mother-child relationship problems share less genetic and environmental variance. The uncoupling points to a true environmental effect or social causation process on the inherited vulnerability to EXT, an effect that is most pronounced in the context of extreme environmental adversity38.

Comment

Our analyses yielded surprisingly consistent findings regarding G-E interplay in the emergence of EXT disorders during late adolescence. Specifically, across 6 environmental risk factors, genetic variance in EXT increased in the context of greater environmental adversity. This indicates that as environmental stress increases genetic differences among people actually become more important in the etiology of EXT.

This finding is not necessarily intuitive. For example, each environmental risk factor was significantly correlated with EXT, evidence of a main effect such that mean-levels of EXT increase with greater levels of environmental risk. This has led some to conclude that environmental influence is causative, and that the greater environmental adversity would obscure genetic influences12. Clearly, however, the effect of environmental adversity differs for those who experience it, as many individuals under substantial environmental stress do not exhibit EXT disorders. Therefore, environmental risk factors must also exert differential or moderating effects on the variance of EXT, and it is these moderating effects on the genetic and environmental variance of EXT that provides a model for the mechanisms of environmental risk for EXT psychopathology38.

The finding of a consistent mechanism of G-E interplay across the different environmental risk factors suggests a general mechanism of environmental influence on EXT regardless of the particular manifestation of the environmental risk. The consistency is especially noteworthy given the differences in content of the environmental measures, the use of multiple informants and methods of assessment, and the differences in heritability. Also, the nature of the G × E interaction with EXT was the same regardless of whether the environmental variable conferred distal risk (independent life events), or risk that was proximal and malleable (peers, parent-child relationships, academic achievement and engagement), the latter being theorized to be most relevant to G-E processes2,5. Additionally, while all the environmental variables were correlated, the moderation effects were not solely due to the effect of a general environmental risk factor. Finally, our results are consistent with several previous studies that have examined single externalizing phenotypes (e.g., conduct problems, alcohol use, smoking) and environmental moderators (e.g., peers, social and demographic variables)5,6,22,35-37,64,65. Integrating our results with the broader literature then suggests a general principle of G-E interplay such that environments that are more constrained, structured, and less stressful suppress genetic risk while unconstrained and more stressful environments amplify genetic risk for externalizing behaviors (though some have suggested this general mechanism can be further delineated)66.

Another possibility is that our measures of environmental risk would have the same moderating effects on the genetic and environmental risk for any form of psychopathology. However, in a separate analysis with the same sample13, we examined G-E interplay between INT disorders and the same measures of environmental risk, and found that the nonshared environment variance of INT increased at higher levels of each environmental risk factor while the genetic and shared environmental variance remained stable. That is, the same environmental adversity was associated with a different mechanism of G-E interplay in the emergence of INT compared to EXT psychopathology. Taken together, the pattern of results across the two studies provides impressive evidence of convergent and discriminant validity. The intriguing hypothesis of our work then is that the mechanism underlying environmental influence is relatively general, but will differ depending on the nature of the psychopathological condition.

Some limitations of our study need to be considered. First, while the sample is representative of the Minnesota population from which it was drawn, it does not reflect the racial and ethnic diversity of the broader United States population. Also, our sample was limited to late adolescence and so it is unknown whether the same G-E interplay processes are present at other developmental stages. Another limitation is ambiguity regarding the direction of causation. That is, our decision to examine the moderating effects of environmental risk on EXT, although theoretically grounded, is methodologically arbitrary as an argument could be made that EXT moderates the genetic and environmental influences on the environmental risk factors. Supplemental analyses, however, showed that when such effects were present, they were much weaker than the moderation effects on EXT. Future analyses that utilize the prospective design of the MTFS will provide further insight into the causative associations between EXT and these environmental risk factors.

Our results have important implications for gene association and measured gene × measured environment studies. Specifically, studies attempting to detect associations between specific genes and EXT will be more likely to yield significant results if they also incorporate measures of environmental adversity. Our results can also inform developmental theories of psychopathology. For example, individuals who exhibit an early onset and persistent course of antisocial behavior and substance use disorders seem to experience both greater genetic and greater environmental risk33,67-71. A potential mechanism is that G-E correlation processes lead to greater exposure to environmental risk while the experience of greater environmental adversity then results in the expression of more genetic vulnerabilities (i.e., increased genetic variance via G × E interactions). These processes are then likely to magnify over time channeling individuals into relatively stable developmental trajectories69,70,72. At present, most gene association studies examine the link between a specific gene and a lifetime case of a disorder73. Given the interplay of these genetic, environmental, and developmental processes, a sensible prediction is that specific genes will be most reliably associated with the developmental characteristics of EXT disorders, especially an early onset and persistent course. As EXT disorders are of substantial public health importance, our hope is that continued attempts to integrate multiple processes and approaches will yield important insights into their etiology.

Acknowledgments

This research was supported in part by USPHS Grants AA09367, DA05147, and DA024417.

References

  • 1.Plomin R, DeFries JC, Craig I, McGuffin P. Behavior genetics in the post genomic era. Washington, D. C.: American Psychological Association; 2002. [Google Scholar]
  • 2.Moffitt TE, Caspi A, Rutter M. Strategy for investigating interactions between measured genes and measured environments. Arch Gen Psychiatry. 2005;62:473–481. doi: 10.1001/archpsyc.62.5.473. [DOI] [PubMed] [Google Scholar]
  • 3.Caspi A, Sugden K, Moffitt TE, Taylor A, Craig IW, Harrington H, McClay J, Mill J, Martin J, Braithwaite A, Poulton R. Influence of life stress on depression: Moderation by a polymorphism in the 5-HTT gene. Science. 2003;301:386–389. doi: 10.1126/science.1083968. [DOI] [PubMed] [Google Scholar]
  • 4.Uher R, McGuffin P. The moderation by the serotonin transporter gene of environmental adversity in the aetiology of mental illness: review and methodological analysis. Mol Psychiatry. 2007 doi: 10.1038/sj.mp.4002067. [DOI] [PubMed] [Google Scholar]
  • 5.Feinberg ME, Button TM, Neiderhiser JM, Reiss D, Hetherington EM. Parenting and adolescent antisocial behavior and depression: evidence of genotype × parenting environment interaction. Arch Gen Psychiatry. 2007;64:457–65. doi: 10.1001/archpsyc.64.4.457. [DOI] [PubMed] [Google Scholar]
  • 6.Button TM, Corley RP, Rhee SH, Hewitt JK, Young SE, Stallings MC. Delinquent peer affiliation and conduct problems: A twin study. J Abnorm Psychol. 2007;116:554–64. doi: 10.1037/0021-843X.116.3.554. [DOI] [PubMed] [Google Scholar]
  • 7.Lau JYF, Eley TC. Disentangling gene-environment correlations and interactions on adolescent depressive symptoms. J Child Psychol & Psychiatry. 2008;49:142–150. doi: 10.1111/j.1469-7610.2007.01803.x. [DOI] [PubMed] [Google Scholar]
  • 8.Cadoret RJ, Yates WR, Troughton E, Woodworth G, Stewart MA. Genetic-environmental interaction in the genesis of aggressivity and conduct disorders. Arch Gen Psychiatry. 1995;52:916–924. doi: 10.1001/archpsyc.1995.03950230030006. [DOI] [PubMed] [Google Scholar]
  • 9.Cloninger CR, Sigvardsson S, Bohman M, von Knorring AL. Predisposition to petty criminality in Swedish adoptees: II. Cross-fostering analysis of gene-environmental interactions. Arch Gen Psychiatry. 1982;39:1242–1247. doi: 10.1001/archpsyc.1982.04290110010002. [DOI] [PubMed] [Google Scholar]
  • 10.Thapar A, Harold G, Rice F, Langley K, O'Donovan M. The contribution of gene-environment interaction to psychopathology. Dev Psychopathol. 2007;19:989–1004. doi: 10.1017/S0954579407000491. [DOI] [PubMed] [Google Scholar]
  • 11.Reiss D, Leve LD. Genetic expression outside the skin: Clues to mechanisms of genotype × environment interaction. Dev & Psychopathol. 2007;19:1005–1027. doi: 10.1017/S0954579407000508. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Raine A. Biological studies of antisocial and violent behavior in children and adults: A review. J Abnorm Child Psychol. 2002;30:311–326. doi: 10.1023/a:1015754122318. [DOI] [PubMed] [Google Scholar]
  • 13.Hicks BM, DiRago AC, Iacono WG, McGue M. Gene-environment interplay in internalizing disorders: consistent findings across six environmental risk factors. doi: 10.1111/j.1469-7610.2009.02100.x. Submitted. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Young SE, Stallings MC, Corley RP, Krauter KS, Hewitt JK. Genetic and environmental influences on behavioral disinhibition. Amer J Medical Genetics (Neuropsychiatric Genetics) 2000;96:684–95. [PubMed] [Google Scholar]
  • 15.Krueger RF, Hicks BM, Patrick CJ, Carlson SR, Iacono WG, McGue M. Etiologic connections among substance dependence, antisocial behavior, and personality: modeling the externalizing spectrum. J Abnorm Psychol. 2002;111:411–24. [PubMed] [Google Scholar]
  • 16.Hicks BM, Krueger RF, Iacono WG, McGue M, Patrick CJ. Family transmission and heritability of externalizing disorders: A twin-family study. Arch of Gen Psychiatry. 2004;61:922–928. doi: 10.1001/archpsyc.61.9.922. [DOI] [PubMed] [Google Scholar]
  • 17.Barkley RA, Anastopoulos AD, Guevremont DC, Fletcher KE. Adolescents with ADHD: patterns of behavioral adjustment, academic functioning, and treatment utilization. J Am Acad Child Adolesc Psychiatry. 1991;30:752–61. doi: 10.1016/s0890-8567(10)80010-3. [DOI] [PubMed] [Google Scholar]
  • 18.Hinshaw SP. Is ADHD an impairing condition in childhood and adolescence? In: Jensen PS, Cooper JR, editors. Attention deficit hyperactivity disorder: State of the science-best practices. Kingston, New Jersey: Civic Research Institute; 2002. pp. 1–19. [Google Scholar]
  • 19.Dishion TJ, Owen LD. A longitudinal analysis of friendships and substance use: Bidirectional influence from adolescence to adulthood. Dev Psychol. 2002;38:480–491. doi: 10.1037//0012-1649.38.4.480. [DOI] [PubMed] [Google Scholar]
  • 20.Patterson GR, Stouthamer-Loeber M. The correlation of family management practices and delinquency. Child Dev. 1984;55:1299–1307. [PubMed] [Google Scholar]
  • 21.Jaffee SR, Caspi A, Moffitt TE, Polo-Tomas M, Price TS, Taylor A. The limits of child effects: evidence for genetically mediated child effects on corporal punishment but not on physical maltreatment. Dev Psychol. 2004;40:1047–58. doi: 10.1037/0012-1649.40.6.1047. [DOI] [PubMed] [Google Scholar]
  • 22.Dick DM, Viken R, Purcell S, Kaprio J, Pulkkinen L, Rose RJ. Parental Monitoring Moderates the Importance of Genetic and Environmental Influences on Adolescent Smoking. J Abnorm Psychol. 2007;116:213–218. doi: 10.1037/0021-843X.116.1.213. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Griffin KW, Botvin GJ, Scheier LM, Diaz T, Miller NL. Parenting practices as predictors of substance use, delinquency, and aggression among urban minority youth: Moderating effects of family structure and gender. Psychol Addict Behaviors. 2000;14:174–184. doi: 10.1037//0893-164x.14.2.174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Blazei RW, Iacono WG, Krueger RF. Intergenerational transmission of antisocial behavior: How do kids become antisocial adults? Appl & Prevent Psychol. 2006;11:230–253. [Google Scholar]
  • 25.Moffitt TE. The new look of behavioral genetics in developmental psychopathology: Gene-environment interplay in antisocial behaviors. Psychol Bulletin. 2005;131:533–554. doi: 10.1037/0033-2909.131.4.533. [DOI] [PubMed] [Google Scholar]
  • 26.Jaffee SR, Price TS. Gene-environment correlations: a review of the evidence and implications for prevention of mental illness. Mol Psychiatry. 2007;12:432–42. doi: 10.1038/sj.mp.4001950. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Scarr S, McCartney K. How people make their own environments: A theory of genotype => environment effects. Child Dev. 1983;54:424–435. doi: 10.1111/j.1467-8624.1983.tb03884.x. [DOI] [PubMed] [Google Scholar]
  • 28.Plomin R, DeFries JC, Loehlin JC. Genotype-environment interaction and correlation in the analysis of human behavior. Psychol Bulletin. 1977;84:309–322. [PubMed] [Google Scholar]
  • 29.Barkley RA, Fischer M, Edelbrock C, Smallish L. The adolescent outcome of hyperactive children diagnosed by research criteria: III. Mother-child interactions, family conflicts and material psychopathology. J Child Psychol & Psychiatry & Allied Disciplines. 1991;32:233–255. doi: 10.1111/j.1469-7610.1991.tb00304.x. [DOI] [PubMed] [Google Scholar]
  • 30.Danforth JS, Barkley RA, Stokes TF. Observations of parent-child interactions with hyperactive children: Research and clinical implications. Clin Psychol Rev. 1991;11:703–727. [Google Scholar]
  • 31.Lambert NM. Adolescent outcomes for hyperactive children: Perspectives on general and specific patterns of childhood risk for adolescent educational, social, and mental health problems. Amer Psychol. 1988;43:786–799. doi: 10.1037//0003-066x.43.10.786. [DOI] [PubMed] [Google Scholar]
  • 32.Keyes MA, Iacono WG, McGue M. Early onset problem behavior, young adult psychopathology, and contextual risk. Twin Res Hum Genet. 2007;10:45–53. doi: 10.1375/twin.10.1.45. [DOI] [PubMed] [Google Scholar]
  • 33.Plomin R, Bergeman CS. The nature of nurture: Genetic influences on “environmental” measures. Behav & Brain Sciences. 1991;14:373–427. [Google Scholar]
  • 34.Kendler KS, Baker JH. Genetic influences on measures of the environment: a systematic review. Psychol Med. 2007;37:615–26. doi: 10.1017/S0033291706009524. [DOI] [PubMed] [Google Scholar]
  • 35.Dick DM, Rose RJ, Viken RJ, Kaprio J, Koskenvuo M. Exploring gene-environment interactions: socioregional moderation of alcohol use. J Abnorm Psychol. 2001;110:625–32. doi: 10.1037//0021-843x.110.4.625. [DOI] [PubMed] [Google Scholar]
  • 36.Heath AC, Jardine R, Martin NG. Interactive effects of genotype and social environment on alcohol consumption in female twins. J Studies Alcohol. 1989;50:38–48. doi: 10.15288/jsa.1989.50.38. [DOI] [PubMed] [Google Scholar]
  • 37.Koopmans JR, Slutske WS, van Baal GC, Boomsma DI. The influence of religion on alcohol use initiation: Evidence for genotype × environment interaction. Behav Genet. 1999;29:445–453. doi: 10.1023/a:1021679005623. [DOI] [PubMed] [Google Scholar]
  • 38.Johnson W. Genetic and environmental influences on behavior: Capturing all the interplay. Psychol Review. 2007;114:423–440. doi: 10.1037/0033-295X.114.2.423. [DOI] [PubMed] [Google Scholar]
  • 39.Purcell S. Variance components models for gene-environment interaction in twin analysis. Twin Res. 2002;6:554–571. doi: 10.1375/136905202762342026. [DOI] [PubMed] [Google Scholar]
  • 40.Bergen SE, Gardner CO, Kendler KS. Age-related changes in heritability of behavioral phenotypes over adolescence and young adulthood: a meta-analysis. Twin Res Hum Genet. 2007;10:423–33. doi: 10.1375/twin.10.3.423. [DOI] [PubMed] [Google Scholar]
  • 41.Iacono WG, Carlson SR, Taylor J, Elkins IJ, McGue M. Behavioral disinhibition and the development of substance-use disorders: findings from the Minnesota Twin Family Study. Dev Psychopathol. 1999;11:869–900. doi: 10.1017/s0954579499002369. [DOI] [PubMed] [Google Scholar]
  • 42.Iacono WG, Malone SM, McGue M. Substance use disorders, externalizing psychopathology, and P300 event-related potential amplitude. Int J Psychophysiol. 2003;48:147–78. doi: 10.1016/s0167-8760(03)00052-7. [DOI] [PubMed] [Google Scholar]
  • 43.American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 3rd, rev. Washington, D.C.: Author; 1987. [Google Scholar]
  • 44.First MB, Gibbon M, Spitzer RL, Williams JBW. Structured Clinical Interviews for DSM–IV Axis II Personality Disorders(SCID-II) Arlington, VA: American Psychiatric Publishing; 1997. [Google Scholar]
  • 45.Robins LN, Babor TF, Cottler LB. Composite International Diagnostic Interview: Expanded Substance Abuse Module. St. Louis: Authors; 1987. [Google Scholar]
  • 46.Welner Z, Reich W, Herjanic B, Jung K, Amado H. Reliability, validity, and parent-child agreement studies of the Diagnostic Interview for Children and Adolescents (DICA) J Acad Child & Adoles Psychiatry. 1987;26:649–653. doi: 10.1097/00004583-198709000-00007. [DOI] [PubMed] [Google Scholar]
  • 47.Johnson W, McGue M, Iacono WG. Genetic and environmental influences on academic achievement trajectories during adolescence. Dev Psychol. 2006;42:514–32. doi: 10.1037/0012-1649.42.3.514. [DOI] [PubMed] [Google Scholar]
  • 48.Hicks BM, Johnson W, Iacono WG, McGue M. Moderating effects of personality on the genetic and environmental influences of school grades helps to explain sex differences in scholastic achievement. Eur J Pers. 2008;22:247–268. doi: 10.1002/per.671. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Walden B, McGue M, Iacono WG, Burt SA, Elkins I. Identifying shared environmental contributions to early substance use: the respective roles of peers and parents. J Abnorm Psychol. 2004;113:440–450. doi: 10.1037/0021-843X.113.3.440. [DOI] [PubMed] [Google Scholar]
  • 50.Elkins IJ, McGue M, Iacono WG. Genetic and environmental influences on parent-son relationships: evidence for increasing genetic influence during adolescence. Dev Psychol. 1997;33:351–63. doi: 10.1037//0012-1649.33.2.351. [DOI] [PubMed] [Google Scholar]
  • 51.McGue M, Elkins I, Walden B, Iacono WG. Perceptions of the parent-adolescent relationship: a longitudinal investigation. Dev Psychol. 2005;41:971–84. doi: 10.1037/0012-1649.41.6.971. [DOI] [PubMed] [Google Scholar]
  • 52.Billig JP, Hershberger SL, Iacono WG, McGue M. Life events and personality in late adolescence: genetic and environmental relations. Behav Genet. 1996;26:543–554. doi: 10.1007/BF02361227. [DOI] [PubMed] [Google Scholar]
  • 53.Masten AS, Neemann J, Andenas S. Life events and adjustment in adolescents. The significance of event independence, desirability, and chronicity. J Res Adolesc. 1994;4:71–97. [Google Scholar]
  • 54.Neale MC, Cardon LR. Methodology for genetic studies of twins and families. Dortrecht, The Netherlands: Kluwer Academic; 1992. [Google Scholar]
  • 55.Neale MC, Boker SM, Xie G, Maes HH. Mx: Statistical modeling (Version 6) [Computer software] Richmond, VA: Department of Psychiatry, Virginia Commonwealth University; 2002. [Google Scholar]
  • 56.Little RJA, Rubin DB. Statistical analysis with missing data. New York: Wiley; 1987. [Google Scholar]
  • 57.McGue M, Bouchard TJ., Jr Adjustment of twin data for the effects of age and sex. Behav Genet. 1984;14:325–43. doi: 10.1007/BF01080045. [DOI] [PubMed] [Google Scholar]
  • 58.Akaike H. Factor analysis and AIC. Psychometrica. 1987;52:317–332. [Google Scholar]
  • 59.Raftery AE. Bayesian model selection in social research. Sociological Methodology. 1995;25:111–163. [Google Scholar]
  • 60.Sclove LS. Application of model-selection criteria to some problems in multivariate analysis. Psychometrika. 1987;52:333–343. [Google Scholar]
  • 61.Schwartz G. Estimating the dimension of a model. Annals of Statistics. 1978;6:461–464. [Google Scholar]
  • 62.Spiegelhalter DJ, Best N, Carlin B, van der Linde A. Bayesian measures of model complexity and fit. (B).J Royal Statistical Society. 2002;64:583–640. [Google Scholar]
  • 63.Purcell S, Koenen KC. Environmental mediation and the twin design. Behav Genet. 2005;35:491–498. doi: 10.1007/s10519-004-1484-9. [DOI] [PubMed] [Google Scholar]
  • 64.Boardman JD, Saint Onge JM, Haberstick BC, Timberlake DS, Hewitt JK. Do schools moderate the genetic determinants of smoking? Behav Genet. 2008;38:234–246. doi: 10.1007/s10519-008-9197-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Harden KP, Hill JE, Turkheimer E, Emery RE. Gene-environment correlation and interaction in peer effects on adolescent alcohol and tobacco use. Behav Genet. 2008;38:339–347. doi: 10.1007/s10519-008-9202-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Shanahan MJ, Hofer SM. Social context in gene-environment interactions: retrospect and prospect. J Geront. 2005;60B(Special):65–76. doi: 10.1093/geronb/60.special_issue_1.65. [DOI] [PubMed] [Google Scholar]
  • 67.Moffitt TE, Caspi A, Dickson N, Silva P, Stanton W. Childhood-onset versus adolescent-onset antisocial conduct problems in males: Natural history from ages 3 to 18 years. Dev Psychopathol. 1996;8:399–424. [Google Scholar]
  • 68.Moffitt TE, Capsi A. Childhood predictors differentiate life-course persistent and adolescence-limited pathways among males and females. Dev Psychopathol. 2001;13:355–375. doi: 10.1017/s0954579401002097. [DOI] [PubMed] [Google Scholar]
  • 69.Sher KJ, Gotham HJ. Pathological alcohol involvement: a developmental disorder of young adulthood. Dev Psychopathol. 1999;11:933–956. doi: 10.1017/s0954579499002394. [DOI] [PubMed] [Google Scholar]
  • 70.Sher KJ, Gotham HJ, Watson AL. Trajectories of dynamic predictors of disorder: Their meanings and implications. Dev Psychopathol. 2004;16:825–856. doi: 10.1017/s0954579404040039. [DOI] [PubMed] [Google Scholar]
  • 71.Iacono WG, Malone SM, McGue M. Behavioral disinhibition and the development of early-onset addiction: common and specific influences. Ann Rev Clin Psychol. 2008;4 doi: 10.1146/annurev.clinpsy.4.022007.141157. in press. [DOI] [PubMed] [Google Scholar]
  • 72.Moffitt TE, Capsi A, Harrington H, Miline BJ. Males on the life-course-persistent and adolescence-limited antisocial pathways: Follow-up at age 26 years. Dev Psychopathol. 2002;14:179–207. doi: 10.1017/s0954579402001104. [DOI] [PubMed] [Google Scholar]
  • 73.Dick DM, Aliev F, Wang JC, Grucza RA, Schuckit M, Kuperman S, Kramer J, Hinrichs A, Bertelsen S, Budde JP, Hesselbrock V, Porjesz B, Edenberg HJ, Bierut LJ, Goate A. Using dimensional models of externalizing psychopathology to aid in gene identification. Arch Gen Psychiatry. 2008;65:310–318. doi: 10.1001/archpsyc.65.3.310. [DOI] [PubMed] [Google Scholar]

RESOURCES