Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 May 13.
Published in final edited form as: Dev Psychopathol. 2009 Spring;21(2):579–591. doi: 10.1017/S0954579409000315

Cortisol and Anti-social Behavior in Early Adolescence: The Role of Gender in an Economically Disadvantaged Sample

Roger Kobak 1, Kristyn Zajac 2, Seymour Levine 3
PMCID: PMC4019020  NIHMSID: NIHMS576438  PMID: 19338699

Abstract

This study examines the relation between adolescents’ anti-social behaviors and adrenocortical activity during a laboratory visit in a sample of economically disadvantaged families (N = 116, ages 12 – 14, 51% female). Pre-task cortisol levels indexed adolescents’ pre-challenge response to the lab visit, while adolescents’ response to a conflict discussion with their caregivers was indexed with residualized change in pre- to post- conflict cortisol levels. A trait measure of anti-social behavior (derived from parent, teacher, and self-reports) was associated with lower pre-task cortisol levels but greater cortisol response to the conflict discussion. Gender moderated anti-social adolescents’ cortisol response to the conflict discussion with girls who reported more covert risky problem behaviors showing an increased cortisol response. The findings suggest that, while anti-social adolescents had lower pre-task cortisol levels, conflict discussions with caregivers present a unique challenge to anti-social girls compared with anti-social boys.

Keywords: Parent-Adolescent Conflict, Anti-social Behavior, Cortisol

During the past two decades, a growing body of research has investigated the relation between regulation of the hypothalamic-pituitary-adrenocortical (HPA) axis and children and adolescents’ anti-social behavior. These studies have examined both patterns of diurnal regulation and response to a variety of social challenges and provide support a hypocortisolism hypothesis linking low basal cortisol levels to anti-social behavior problems (Shirtcliff, Granger, Booth, & Johnson, 2005). More generally, anti-social behavior has been associated with broader pattern of autonomic underarousal (Raine, 2002) and attenuated adrenocortical activity (van Goozen, Fairchild, Snoek, & Harold, 2007). In contrast, studies of children’s cortisol response to standardized and interpersonal challenge paradigms have produced less consistent findings (McBurnett et al., 2005). In this paper, we examined adolescents’ anti-social behavior and cortisol levels during a laboratory visit that included a parent-adolescent conflict discussion. The sample consisted of economically disadvantaged families with early adolescent children. Because, the early adolescent period is marked by increased rates of girls’ anti-social behavior (Moffitt, 2006) this sample facilitated tests of gender differences in the relation between anti-social behavior and the regulation of the HPA system.

Hypocortisolism and Anti-social Behavior

Under normal non-stressful conditions, human cortisol levels follow a diurnal pattern with higher levels after waking in the morning, followed by a decline in levels during the day and the lowest levels at bedtime. Basal cortisol level can be conceptualized as a measure of individual differences in non-stimulated HPA activity, preferably measured during morning hours. Using latent state-trait modeling of multiple morning samples, Shirtcliff and colleagues (2005) estimated that 28% of the variance in cortisol levels could be attributed to trait characteristics while 70% of the variance was attributable to state-like factors. Low basal cortisol levels have been associated with a range of problems including trauma, neglect, and anti-social behaviors (Gunnar & Vazquez, 2001). A broad hypocortisolism hypothesis posits that both low basal levels and blunted response to challenges are risk factors for externalizing and disruptive behavior problems (van Goozen et al., 2007).

Studies of HPA activity in non-stressful conditions have produced relatively consistent findings linking hypocortisolism to anti-social behaviors. McBurnett, Lahey, Rathouz, and Loeber (2000) found that clinic-referred school-aged boys with low cortisol concentrations exhibited triple the number of aggressive symptoms and were named as most aggressive by peers 3 times as often as boys who had high cortisol concentrations. In a large sample of both boys and girls, Shirtcliff and colleagues (2005) reported a link between externalizing behaviors and low morning cortisol only for boys. Cicchetti and Rogosch (2001) found a similar gender difference in a risk sample, with externalizing boys having lower morning and average daily levels than externalizing girls. Although these findings are largely limited to pre-adolescent samples, they highlight the need to examine gender as a moderator of the relation between adrenocortical activity and antisocial behavior.

While there is a relatively consistent pattern linking attenuated cortisol levels to anti-social behavior, investigations of children’s cortisol response to challenge have generally produced more contradictory findings. The nature of the challenge task presented to children may partially account for these inconsistent results. For instance, studies using standardized laboratory stressors often report an association between blunted cortisol and anti-social behavior (van Goozen et al., 2007). Snoek and colleagues (2004) found that blunted cortisol response to a frustration and provocation paradigm was associated with a diagnosis of oppositional defiant disorder (ODD) in a predominately male sample, particularly among the most disturbed boys who were psychiatric inpatients (Snoek, van Goozen, Matthys, Buitelaar, & van Engeland, 2004). Using a similar provocation paradigm, van Goozen and colleagues reported that boys with Disruptive Behavior Disorders had lower cortisol levels during stress than during rest (i.e., basal) compared with normal boys (van Goozen, Matthys, Cohen-Kettenis, Buitelaar, & van Engeland, 2000; van Goozen et al., 1998).

However, studies that use more naturalistic challenge paradigms, such as family conflict or talking about a traumatic event, have often reported associations between increased cortisol response and anti-social behavior. McBurnett and colleagues (2005) asked 335 male adolescents to think and then speak about the worst thing that ever happened to them. Adolescents with the most extreme conduct problems showed increased cortisol response to this worst event challenge. An extensive anthropological study of children’s daily cortisol in a Caribbean village (Flinn & England, 1995) found that stressors, such as punishments, quarreling, and changes in residence, substantially increased children’s cortisol levels while calm, affectionate contact with caregivers was associated with diminished cortisol levels. When highly stressed children were observed during non-stressful periods, the investigators noted that children with unusually low basal cortisol marked by occasional high spikes in response to conflict were more likely to engage in anti-social behavior that included delinquency and running away from home.

Two studies using experimenter-instigated conflict discussions have also reported an association between cortisol response to challenge and child behavior problems. Granger, Weisz, & Kauneckis (1994) reported that cortisol response to parent-child conflict was associated with parent and self reports of children’s social withdrawal and aggression. Klimes-Dougan and colleagues found an increased cortisol response to a conflict discussion among adolescents with externalizing symptoms (Klimes-Dougan, Hastings, Granger, Usher, & Zahn-Waxler, 2001). These studies suggest that greater cortisol response to parent-child conflict is an atypical pattern associated with increased risk for psychopathology. However, it is important to note that only 15% of high-risk children showed a cortisol increase during a conflict discussion videotaped in the home (Klimes-Dougan et al., 2001), while 33% of clinic-referred children showed an increase in response to a laboratory conflict task (Granger et al., 1994). The relatively low levels of cortisol response to parent-adolescent conflict discussion is consistent with an observational literature indicating that the majority of families maintain constructive problem-solving and emotion regulation (Kobak, Cole, Ferenz-Gillies, Fleming, & Gamble, 1993) or autonomy and relatedness (Allen, Hauser, Bell, & O’Connor, 1994) during discussions of disagreements.

Gender and Anti-social Behavior during Early Adolescence

Clinical investigations of pre-adolescents’ anti-social behavior and adrenocortical activity have relied almost exclusively on male samples (see van Goozen et al., 2007 for a review). In part, this results from clinical samples in which there is a substantially higher prevalence of disruptive behavior disorders among pre-adolescent males. In the Dunedin longitudinal study, childhood anti-social behavior occurred with a ten-fold prevalence in boys compared with girls (Moffitt, Caspi, Rutter, & Silva, 2001). However, the prevalence of anti-social behavior among females increases with the onset of puberty with an adolescent male/female ratio of anti-social behavior of only 1.5:1 (Moffitt, 2006). Moffitt (2006) describes the delinquent behavior that emerges following puberty as an “adolescent-limited” subtype of anti-social behavior that is generally fostered by involvement with deviant peers. Such involvement with deviant peers may be a necessary condition for the onset of delinquency among girls (Caspi & Moffitt, 1991; Moffitt, 2006). Early adolescence also marks a rapid growth in “covert” anti-social behaviors that are intended to avoid detection by adults (Dishion & Patterson, 2006). Whereas overt aggressive anti-social behavior typically develops in early childhood and remains relatively stable during middle childhood, covert anti-social behaviors emerge during early adolescence and increases rapidly, peaking in mid to late adolescence.

Although early adolescence is marked by increased prevalence of girls’ anti-social behavior and a rapid increase in covert forms of delinquent activity, relatively little is known about adolescent girls’ anti-social behavior and neuroendocrine activity. The only clinical study that investigated conduct disordered adolescent girls (Pajer, Gardner, Rubin, Perel, & Neal, 2001) found an association between low morning cortisol and antisocial behavior that was similar to the hypocortisolism reported in male samples. The two community studies that incorporated adolescent girls in their samples have produced somewhat different findings. Shirtcliff et al. (2005) reported no relation between girls’ morning cortisol levels and externalizing symptoms in a sample ranging from 6 to 16 years of age. Klimes-Dougan and colleagues (2001) reported gender differences in diurnal patterns with girls maintaining higher afternoon levels, but they did not find differences in how 11 to 17 year-old girls and boys responded to a conflict discussion. These studies point to the need for further investigation of cortisol-antisocial relations in samples that include adolescent females.

In summary, previous studies of children and adolescents provide mixed support for a broad hypothesis linking hypocortisolism to anti-social behavior. Studies using resting cortisol levels to identify a stable trait-like aspect of adrenocortical activity tend to support hypocortisolism in samples of preadolescent males with disruptive behavior problems. Studies of cortisol response to challenge have produced results that tend to vary with the nature of the challenge task. Blunted cortisol response is most evident in studies that use standardized laboratory challenges (Snoek et al., 2004; van Goozen et al., 1998, 2000). In contrast, studies that use tasks derived from participants’ experiences, such as talking about a worst experience or engaging in a parent-child conflict discussion, often report an association between increased cortisol response and anti-social behavior (Granger, Weisz, McCracken, Ikeda, & Douglas, 1996; Klimes-Dougan et al., 2001; McBurnett et al., 2005). The only naturalistic study of parent-child conflict, Flinn and England (1995) reported that anti-social children showed sharp spikes in cortisol response to naturally occurring family conflict. This suggests that hypocortisolism pertains to between-subject differences in trait-like aspects of adrenocortical activity but not to more state-like responses to challenge paradigms.

The Present Study

The goal of the present study is to examine gender differences in adrenocortical activity and antisocial behavior during early adolescence. Based on previous research, it is expected that anti-social adolescents will show decreased pre-task cortisol levels and increased cortisol response to a parent-adolescent conflict discussion. Our study is designed to address a gap in the literature on cortisol activity and girls’ anti-social behavior. By restricting our sample to economically disadvantaged families with early adolescents, we hope to obtain a sample with increased prevalence of girls’ anti-social behavior. By using multiple informants (teachers, self, and caregivers) as independent sources of information about adolescents’ anti-social behavior, we plan to examine both a common factor of anti-social behavior and a two-factor model that distinguishes between covert and overt forms of anti-social behavior. We expect that more covert anti-social behavior involving substance use and sexual risk-taking behavior may be particularly relevant for understanding the increased prevalence of girls’ anti-social behavior in early adolescence.

Method

Participants

Participants were 180 economically disadvantaged adolescents (92 female and 88 male) and their caregivers who were recruited to take part in a longitudinal investigation of risk for adolescent psychopathology. Due to typically high diurnal levels of cortisol during morning hours, response to the morning laboratory visits may be confounded with steeper declines in cortisol levels (Adam, Klimes-Dougan, & Gunnar, 2007). As a result, 58 families that attended morning lab sessions were excluded from the analyses, leaving a final sample of 116 families. Adolescents ranged in age from 12 to 14 (M = 13.2). Seventy-seven percent of participants were African-American, 21% were European-American, and 2% were Hispanic. Families had an average household income of $27,250 (SD = $22,829) and 27% of the families received welfare payments. A majority (57%) of the families had a single caregiver, 21% of the families included two biological parents, and 22% included a primary caregiver and a live-in boyfriend. In terms of their relationships with the adolescents, caregivers were 87% biological mothers, 7% grandmothers, 3% biological fathers, and 3% aunts or foster mothers.

Procedure

The families were recruited from two sources: 62 families were recruited from an earlier longitudinal study of children who had participated in Head Start (Ackerman, Kogos, Youngstrom, Schoff, & Izard, 1999) and 54 families were recruited from a list of 13-year old children whose families met income guidelines for free and reduced priced lunch. Initial interviews were conducted with families during a home visit in which demographic and personality measures were collected with both the adolescent and caregiver. Home visits lasted between 1 and 2 hours and were followed within a 2 to 4 week period by a University laboratory visit. Lab visits lasted between 2.5 and 3 hours and included the parent-adolescent conflict discussion. Within 6 months of the lab visit, adolescents were interviewed at their school and both self and teacher reports of adolescent symptoms were collected. The measures reported within the context of this study are a subset of the measures collected as part of the larger study.

Laboratory Procedure

Laboratory visits occurred during mid to late afternoon and began with a brief 5-minute introduction to the lab protocol. Caregivers and adolescents were then taken to separate rooms to complete interviews and questionnaires. At the end of the interview session, the caregivers and adolescents separately completed an Issues Checklist on which they rated topics that they viewed as sources of disagreement in their relationship with the other person. After both interviews were complete, the caregiver and adolescent were reunited. The 5-minute reunion was followed by two 5-minute conflict discussions (one discussion of the caregiver’s topic and a second discussion of the adolescent’s topic), a 10-minute co-construction of memory task, and a 10-minute joint puzzle solving exercise. All interactions were videotaped with brief interruptions during which research assistants introduced the tasks. When the interactions were complete, the families were debriefed and paid for their participation.

Parent-Adolescent Conflict Discussion

Parent-adolescent conflict discussions tested caregivers’ and adolescents’ ability to maintain constructive conversation about a disagreement. Conflict discussions were established by using the major areas of disagreement identified by the adolescent and caregiver on the Issues Checklist, which included 13 topics thought to be sources of parent-adolescent conflict (e.g., money, communication, and curfew). The order of the adolescent and caregiver topics was counterbalanced. When both caregiver and adolescent identified the same topic, a topic that was rated second was chosen. Dyads were instructed to discuss the topic and to try to reach an agreement. The interviewer then left the room and returned after 5 minutes to introduce the next topic.

Salivary Cortisol

Salivary samples were taken at three times during the laboratory visit using cotton rolls that participants chewed for a minimum of 45 seconds. Cotton rolls were then placed in salivettes (SARSTEDT). Participants were instructed not to eat during the half hour preceding the lab visit. The first salivary sample was taken during the first 5 minutes after the dyads arrived in the lab, immediately following the signing of consent forms, with an average time of 4:26 p.m. (SD = 1 hour 40 minutes, range = 12:05 p.m. to 7:15 p.m.). Caregivers and adolescents were then separated for approximately 75 minutes (range 58 to 92 minutes) to complete individually administered interviews and questionnaires. A second salivary sample was taken after the adolescent completed his or her interview. The average time for the second sample was 5:40 p.m. (SD = 1 hour 50 minutes, range = 1:05 p.m. to 8:30 p.m.). When both the caregiver and adolescent had completed the interviews, they were reunited in an observation room where they participated in a series of interactions. The final salivary sample was taken at the end of the interactions, 30 minutes after the beginning and 20 minutes after the end of the conflict discussion. The average time for the final post-conflict sample was 6:55 p.m. (SD = 1 hour 39 minutes, range = 1:55 p.m. to 10:10 p.m.). Neither food nor smoking occurred during the 30 minutes prior to each sample.

The samples were frozen immediately following the session at approximately −25° C. On the day of analysis, samples were thawed and spun in a centrifuge at 3000 rpm for 15 minutes. Samples were then pipetted into a high-sensitivity enzyme immunoassay plate from Salimetrics. A sample volume of 25 μl was used. All samples were analyzed in duplicate and any test values varying more than 8% were re-assayed. The assay has a lower limit sensitivity of 0.007 μg/dl and an upper sensitivity of 1.8 μg/dl. All controls came pre-packaged to maximize test accuracy. Controls that represented high and low cortisol were also analyzed with each plate. The average intra- and inter-assay coefficients of variation were 4.13% and 8.89%, respectively. The average reliability of duplicate standards across plates was r = .985. The average of the duplicate samples was used in all subsequent analyses.

Measures

Demographic interview

During the initial visit in the home, a demographic interview was administered to the caregivers to assess household income, number of children in the household, caregiver’s relation to child, child’s ethnicity, and level of education completed by the caregiver.

Assessment of anti-social behaviors

Ratings of the adolescents’ anti-social behaviors were collected from three different sources (adolescents, caregivers, and teachers) using the Youth Self Report (YSR), the Child Behavior Checklist (CBCL), and the Child Behavior Checklist - Teacher Report Form, (TRF; Achenbach & Rescorla, 2001), respectively. Caregivers completed the CBCL during the lab visit and adolescents completed the YSR during the school visit. Two teachers independently reported on each adolescent’s behaviors by completing the TRF within 6 months of the lab visit. Two teachers’ ratings were obtained for 73% of the sample and the correlation between teacher ratings was .45. Using a composite of two teachers ratings for a part of the sample would result in regression to the mean in a subset of our sample. As a result, we examined patterns of missing data for students with one versus two teacher ratings by coding “missing” (0) or “not missing” (1). T-tests with each predictor and outcome variable yielded no significant tests. As a result, data can be assumed to be missing completely at random (MCAR). Based on the MCAR assumption, we imputed the second teacher rating using Expectation Maximization (EM) from SPSS Missing Value Analysis 7.5 at the subscale level based on the first teacher’s scores. Teachers’ scores were then aggregated in order to increase reliability.

Problem behaviors, including alcohol and drug use and risky sexual behaviors, frequently co-occur with anti-social behavior and occur with increased frequency when adolescents affiliate with peers who engage in risky behaviors (Allen, Aber, & Leadbeater, 1990; Dishion & Andrews, 1995). Problem behaviors were assessed with three subscales from the Drug and Alcohol Use Questionnaire (Dishion & Loeber, 1985). The Own Use scale of this questionnaire reflects the frequency and severity of the adolescent’s own cigarette, drug, and alcohol use and has an internal consistency of .82. The Peer Use scale reflects the extent of drug and alcohol use by the adolescents’ peers and has an internal consistency of .78. The Substance Problems scale reflects the extent to which the adolescent has had interpersonal difficulties related to the use of drugs and alcohol and has an internal consistency of .82. In addition, sexual risk behaviors were assessed using the Scale of Sexual Risk-Taking (SSRT; Metzler, Noell, & Biglan, 1992), which is a 13-item measure of the frequency of engagement in unprotected sexual intercourse and sexual behaviors. Internal consistency for this scale was .56.

Data reduction for measures of anti-social behavior

Two factor analyses were conducted to reduce the multiple informant measures of aggression, delinquency, and problem behaviors to factor scores. See Table 1. The first principal components analysis was constrained to a single factor solution of the 10 measures that accounted for 37% of the variance (Eigenvalue = 3.69). Factor loadings for the individual measures ranged from .73 for adolescent reports of delinquent behavior on the CBCL to a low of .50 for caregiver reports of aggressive behavior. A single standardized factor score was derived from these loadings to index a general anti-social trait. The second principal components analysis was constrained to a two-factor solution (Eigenvalues = 3.69 and 1.69). The two factors accounted for 51% of the variance in the 10 measures of anti-social measures with no cross-loadings larger than .32. The first factor consisted of six measures reported by the adolescents, including problem behaviors and YSR reports of aggression and delinquency, with factor loadings ranging from .85 for reports of substance-related problems to .44 for YSR aggression. The second factor consisted of teacher and caregiver reports of aggression and delinquency with factor loading ranging from .83 for teachers’ ratings of aggression to .68 for teachers’ ratings of delinquency. The two scores derived from the factor loadings were significantly correlated (r = .35, p < .001) though the first score approximates more covert behavior while the second factor reflects adult reports (caregiver and teacher) of overt anti-social behavior.

Table 1.

One and Two-Factor Models of Multi-informant Measures of Anti-social Behavior

One Factor Solution Two-Factor Solution
Covert Overt
TRF Aggression .51 .82
TRF Delinquency .62 .68
CBCL Aggression .50 .78
CBCL Delinquency .59 .72
YSR Aggression .54 .44
YSR Delinquency .73 .72
Sexual Risk Taking .66 .64
Substance Use .50 .70
Substance Use Problems .75 .85
Peer Substance Use .60 .76
Open in a new tab

Results

Analytic Strategy

Examination of the raw data indicated that the distributions of the salivary cortisol were positively skewed. Log 10 transformations were used to establish normal distributions prior to analysis. After log transformations, the cortisol values had skewness statistics approaching zero and ranging from −0.72 for T1 to .23 for T3. Analyses were conducted in the following manner. First, we considered the relationships between cortisol levels on the three sampling occasions (T1, T2, and T3) for adolescents as a group. Next, we examined correlations between adrenocortical activity and the time of sampling, gender (coded 0 for males and 1 for females), and ethnicity. Regression models were then used to consider the main effects of anti-social behavior on T1 pre-task levels and cortisol response to the parent-adolescent conflict. Gender was examined as a moderator of the relationship between cortisol variables and anti-social behavior in all regression models.

Adrenocortical Activity: Effects of Gender, Sampling Occasion, and Time of Day

The average correlation between the cortisol samples was .61 with the samples in closest proximity (T1-T2, T2-T3) showing the largest correlations (r = .74 and .66, respectively). These correlations indicate substantial within-subject stability of cortisol levels across the three samples. Time of sampling was significantly correlated with all three samples (r = −.37, −.42, and −.49, respectively). Although all of the samples were during afternoon and early evening hours, diurnal declines in cortisol levels are evident with later times being associated with lower cortisol levels. As a result, start time of the visit is controlled in all subsequent analyses.

In the first set of analyses, T1, T2, and T3 cortisol samples were used as a within-subjects repeated measure, with gender as an independent variable and start time of the visit as a covariate. Time of visit produced a large between-subject effect on cortisol levels, with subjects who attended sessions earlier in the day having higher levels of cortisol, F(1, 119) = 30.53, p < .001. Within-subject contrasts in cortisol levels between T1 and T2 and between T2 and T3 were non-significant. Gender did not produce a between-subjects effect. For descriptive purposes, we computed delta change scores for the pre- and post-conflict interactions (T2 - T3). Positive change scores reflect cortisol increases from pre- to post-discussion. For the conflict interaction, delta scores ranged from −.26 μg/dl to .37 μg/dl (M = −.04 μg/dl, SD = .10); 14% of adolescents had increases in cortisol levels greater than .02 μg/dl, 33% of adolescents had decreases greater than .02 μg/dl, and 53% of adolescents showed little change, ranging from −.02 to 02 μg/dl.

Correlations were used to examine the association between the three cortisol samples and demographic variables including household income, caregiver education, and family structure. One of the nine correlations was significant, indicating a negative relationship between household income and adolescents’ cortisol response to the conflict discussion (r = −.21, p < .01).

Adolescents’ Anti-social Behavior and Adrenocortical Activity

Hierarchical regression models tested the main effect and gender interaction of adolescents’ anti-social behavior on pre-task cortisol levels and change in cortisol during the conflict discussion. In the first model shown in Table 2, pre-task cortisol levels (T1) were regressed on anti-social behaviors with gender as a moderator. Time of sampling and gender were entered in step 1 as control variables, followed by the anti-social factor score in step 2. In step 3, the interaction between gender and anti-social behavior was entered. Adolescents’ anti-social behavior was associated with decreased pre-task cortisol levels. However, this was not moderated by gender.

Table 2.

Pretask Levels and Cortisol Response to Conflict Regressed on Anti-Social Factor Score (N = 114)

Pre-Task (T1) Cortisol Post-Conflict Cortisol
Step 1 Step 2 Step 3 Step 1 Step 2 Step 3
Step 1
Pre-Conflict (T2) Cort ------ .53*** .58*** .59***
Gender .07 .06 .06 .02 .03 .02
Time of Lab Visit −.32*** −.34*** −.34*** −.18* −.15+ −.14+
Anti-social Behavior −.26** −.22+ .17* .00
Anti-social X Gender −.06 .23*
R2 Statistics for Step .11** .18*** .18*** .41*** .43*** .45***
Open in a new tab
*

p < .05,

**

p < .01,

***

p < .001.

The second regression model examined the effects of adolescents’ anti-social behavior on change in cortisol levels from pre- to post-conflict discussion. Post-conflict cortisol levels (T3) were regressed on the control variables and pre-conflict (T2) cortisol levels, the anti-social factor score, and the interaction between gender and anti-social behavior (see Table 2). Although there was a significant main effect with anti-social behavior being associated with increased cortisol response to the conflict discussion, this effect was moderated by gender. Figure 1 shows the mean cortisol levels for boys (n = 17) and girls (n = 11) who scored in the top 20% of the sample on the anti-social factor score. Adolescents in this group all had T scores greater than 65 on teachers’ ratings of externalizing behavior. The figure indicates that anti-social girls had an increased cortisol response (T2-T3) to the conflict discussion compared with anti-social boys.

Figure 1.

Figure 1

Open in a new tab

Levels of cortisol during the laboratory visit by gender and anti-social behavior.

Post Hoc Analyses of Anti-social Behavior

The two-factor model of anti-social behavior yielded a first factor of more covert externalizing and problem behaviors reported by adolescents and a second factor of more overt adult-reported aggressive and delinquent behaviors. Each factor score was entered into regression models to determine their relative effects on cortisol activity. In Table 3, pre-task (T1) cortisol was regressed on control variables and covert or overt anti-social factor scores. The covert anti-social factor accounted for a significant reduction in T1 cortisol levels while the overt factor score did not account for significant variance. In Table 4, change in cortisol levels during the conflict discussion was regressed on the covert and overt factor scores. There was a significant gender interaction for the covert anti-social score with girls’ covert behaviors associated with increased response to the conflict discussion while boys’ behaviors were not associated with increased response. In contrast to covert behaviors, overt anti-social behaviors produced a main effect on increased cortisol response to the conflict discussion, but this effect was not moderated by the gender. Because girls’ pubertal maturation has been associated with increased cortisol reactivity (Stroud, Papandonatos, Williamson, & Dahl, 2004), age at menarche was entered in the regression model with covert behaviors. The relation between girls’ covert anti-social behavior and reactivity the conflict discussion was not attenuated by pubertal status.

Table 3.

Pretask Cortisol Regressed on Covert and Overt Factor Scores (N = 114)

Pre-task Cortisol (T1) Pre-task Cortisol (T1)
Step 1 Step 2 Step 3 Step 1 Step 2 Step 3
Gender .07 .07 .06 .07 .06 .06
Time of Lab Visit −.32*** −.33*** −.34*** −.33*** −.33*** −.33***
Covert Anti-social −.28*** −.19 ------ ------ ------
Overt Anti-social ------ ------ ------ −.11 −.12
Covert A-S X Gender −.13 ------ ------ ------
Overt A-S X Gender ------ ------ ------ .01
R2 Statistics for Step .11** .19*** .20*** .11** .13** .13**
Open in a new tab
*

p < .05,

**

p < .01,

***

p < .001.

Table 4.

Cortisol Response to Conflict Discussion Regressed on Covert and Overt Factor Scores (N = 114)

Post-Conflict Cortisol (T3) Post-Conflict Cortisol (T3)
Step 1 Step 2 Step 3 Step 1 Step 2 Step 3
Pre-Conflict Cort (T2) .53*** .51*** .59*** .53*** .57*** .57***
Gender .02 .03 .02 .02 .03 .03
Time of Lab Visit −.18* −.18* −.18* −.18* −.15+ −.15+
Covert Anti-Social .13 −.07 ------ ------ ------
Overt Anti-social ------ ------ ------ .16* .12
Covert A-S X Gender .31** ------ ------ ------
Overt A-S X Gender ------ ------ ------ .06
R2 Statistics for Step .40*** .41*** .46*** .40*** .42** .42***
Open in a new tab
*

p < .05,

**

p < .01,

***

p < .001.

Discussion

There were two major objectives of this study. The first was to test whether a hypocortisolism hypothesis could account for adolescents’ pre-task cortisol levels and cortisol response to a caregiver-adolescent discussion. The second was to examine gender as a moderator of the relationship between adrenocorticol activity and anti-social behavior. Our low-income sample of early adolescents provided several advantages over previous studies. By restricting our sample to economically disadvantaged adolescents, we obtained a sample that had substantially elevated prevalence of anti-social behaviors. Further, by focusing on early adolescence, we were able to examine a developmental period in which girls’ anti-social behavior becomes nearly as prevalent as boys’ antisocial behavior.

The analyses of adolescents’ pre-task cortisol levels and anti-social behaviors lend support to the hypocortisolism hypothesis. Overall, both anti-social boys and girls had reduced cortisol levels at the beginning of the laboratory visit. Although laboratory pre-task cortisol levels may combine resting or basal levels with some anticipatory anxiety associated with the lab visit, the lower pre-task levels of anti-social adolescents are consistent with previous studies that report an association between attenuated adrenocorticol activity and anti-social behavior (Susman & Pajer, 2004). This pattern of low resting cortisol levels may reflect a lack of fear and a propensity to engage in risky problem behaviors as a form of stimulation or sensation-seeking (Raine, 2002). However, in contrast to previous studies that reported hypocortisolism among anti-social males, our findings indicated that this pattern was also evident for anti-social girls. Insofar as pre-task cortisol levels reflect a more trait-like component of children’s adrenocortical activity, the lack of gender moderation differs from studies that report hypocortisolism only for anti-social boys (Shirtcliff et al., 2005). Our finding of hypocortisolism among anti-social girls may be due to the increased prevalence of girls’ anti-social behavior in a sample that was restricted to low-income and early adolescents. We suspect that the link between girls’ hypocortisolism and anti-social behavior may be more robust in adolescence and in samples with symptom severity that approach or meet diagnostic thresholds (Pajer et al., 2001) than in pre-adolescent community samples.

Adolescents’ HPA reactivity to the conflict discussion with their caregivers indicated a more complex gender differentiated pattern. As in previous studies of parent-child conflict discussions, the majority of adolescents evidence little cortisol response to the conflict discussion while a minority of adolescent showed an increased or decreased adrenocorticol activity. These findings replicate prior studies and suggest that laboratory conflict discussions do not activate the HPA system in most parent-child relationships and may in fact reduce cortisol activity. More generally, parent-child relationships may play an important role in regulating children’s adrenocortical activity and so conflicts or distress only become problematic in relationships that are insecure or distressed (Adam et al., 2007). When viewed from this perspective, adolescents’ cortisol response to a laboratory interaction with their caregiver provides a measure of the social regulation of the HPA axis in a specific relationship.

However, adolescents’ anti-social behavior was associated with greater likelihood of a cortisol response to the conflict discussion. This finding replicates previous studies that report greater cortisol response to conflict among children with psychopathology. While other studies using parent-child conflict have found increased cortisol responses in symptomatic children, elevated response was more common in children with co-occurring internalizing and externalizing behaviors (Granger et al., 1996; Klimes-Dougan et al., 2001). In contrast to previous studies, the relation between cortisol reactivity and anti-social behavior was moderated by adolescents’ gender and to some extent, by the nature of symptom reports. Overt anti-social behavior reported by teachers and caregivers was associated with increased reactivity to the conflict discussion for both boys and girls. However, the strongest effects on cortisol reactivity were evident only for girls’ reports of more covert behaviors, including substance abuse and sexual risk-taking behaviors.

There are a number of possible explanations for anti-social girls’ reactivity to the conflict discussion. Girls’ covert anti-social behaviors may be indicative of “adolescent-limited” forms of anti-social activity that are fostered by association with deviant peer groups (Moffitt, 2006). Girls’ anti-social behaviors are often instigated by intimate relationships with anti-social males and may increase girls’ risk for early pregnancy and possible dating violence (Moffitt et al., 2001). As a result, the emergence of these problem behaviors around puberty may create strain in the caregiver-adolescent relationship and increase the degree to which girls perceived the laboratory conflict discussions as threatening. This suggests that the distinction between covert and overt forms of anti-social behavior may be particularly important in understanding adolescent girls’ externalizing problems. The need to distinguish between covert and overt forms of anti-social behaviors is supported by a growing literature suggesting different subtypes of externalizing problems (Dishion & Patterson, 2006; Moffitt, 2006).

Another possible explanation for anti-social girls’ reactivity to the conflict discussion with caregivers may reflect a gender difference in sensitivity to interpersonal contexts. Females’ heightened sensitivity to interpersonal threats has been observed in college students’ reactions to laboratory challenges (Stroud, Salovey, & Eppel, 2002). In this study, females showed greater cortisol response to a social rejection challenge whereas males showed greater reactivity to achievement challenges. While this explanation would account for an overall main effect of gender on response to the conflict discussion, we did not find such an effect. Rather, only girls who reported more problem behaviors, aggression and delinquency were more reactive to the discussion.

A third possibility is that gender differences in the timing of puberty may account for differential cortisol reactivity to the conflict discussion. A previous study found that as girls’ advanced through Tanner pubertal stages, their capacity to recover from a CRH challenge diminishes (Stroud et al., 2004). In addition, early pubertal maturation has been associated with increased risk for girls to develop behavior problems and affiliate with deviant peers (Ge, Brody, Conger, Simons, & Murray, 2002a, 2002b). As a result, early timing of puberty could account for the association between girls’ anti-social behavior and cortisol reactivity to the conflict discussion. Although this is a possibility that merits further study, the relation between girls’ cortisol reactivity and anti-social behavior was not attentuated when we controlled for their age at menarche.

More research is needed to determine the degree to which girls’ cortisol response to caregiver-adolescent conflict is specific to their relationships with their caregivers. Other types of stressor may yield results that are more consistent with the hypocortisolism-externalizing hypothesis (van Goozen et al., 2007). The specificity of cortisol response to parent-child conflict is also illustrated by Klimes-Dougan and colleagues’ (2001) report that children’s cortisol response to a public speaking task and cortisol response to parent-child conflict were independent of each other. In comparison with more standardized laboratory stressors, caregiver-adolescent conflict discussions are likely to differ based on adolescents’ prior history of resolving conflicts. Thus, adolescents with histories of harsh parenting may experience the discussion as more threatening than adolescents with histories of supportive parenting. This interpretation suggests that a history of harsh parenting would act as a third variable linking cortisol response to conflict and externalizing symptoms.

Much remains to be learned about the role that HPA reactivity plays in promoting or hindering adaptation (Boyce & Ellis, 2005). There may also be important individual differences in children’s sensitivity to environmental influences. These differences in susceptibility may produce differential outcomes depending on the overall level of environmental support or adversity (Ellis, Essex, & Boyce, 2005). As a result, stress reactivity may promote as well as hinder adaptation. For instance, HPA reactivity may play an important counter regulatory role in moderating sympathetic activation as is evident from recent studies that employ measures of the HPA, sympathetic, and parasympathetic systems (Bauer, Quas, & Boyce, 2002; Gordis, Granger, Susman, & Trickett, 2006). Further, there is some evidence that children who show greater stress reactivity to frustration/provocation may benefit more from therapeutic intervention (van de Wiel, van Goozen, Matthys, Snoek, & van Engeland, 2004). More attention to the person-situation transactions that influence stress reactivity and their relation to adaptive or maladaptive outcomes is an important direction for future studies of adrenocortical activity and psychopathology.

Limitations and Future Directions

In summary, our results suggest that adolescents’ cortisol activity is influenced by both trait and situational factors. A broad hypocortisolism hypothesis suggests that individuals with chronic under-arousal of the HPA axis will be at increased risk for externalizing problems. This hypothesis views under-arousal as a relatively stable aspect of the individual that will account for under-arousal in both basal or resting conditions and in situations that normally activate the HPA axis. This hypothesis is best tested with repeated measures of both basal cortisol activity and trait-like aspects of personality (Adam et al., 2007; Shirtcliff et al., 2005). Standardized stress paradigms may provide an optimal test of the hypocortisolism hypothesis in studies of cortisol response to challenge. When laboratory baseline cortisol measures are viewed as assessments of trait-like aspects of adrenocortical activity as well as possible anticipatory reactions to the laboratory visit, our findings support this perspective. In contrast, adolescents’ HPA response to conflict discussions may vary depending on their prior relationship with the caregiver and the adolescents’ behavior problems. Future studies should consider relationship processes that influence HPA regulation and consider stress responsiveness across different relationships and challenge situations. Greater attention to situational effects on cortisol response will provide stronger tests of the extent to which blunted cortisol response is a stable cross-situational trait associated with anti-social behavior.

Future investigations should employ more frequent sampling during laboratory visits and monitor recovery from peak cortisol response to stress paradigms more closely (see Klimes-Dougan et al., 2001). Designs should include counterbalancing of the presentation of stress paradigms to control for possible order effects and gathering home samples to differentiate between basal levels and anticipatory reactions to the laboratory visit. In addition, further assessment of factors that may influence cortisol levels would reduce measurement error in cortisol assays. Factors that were not controlled in the present study include smoking during the two days prior to the lab visit, medications, possible invisible blood contamination (Kivlighan et al., 2004), and phase of the menstrual cycle (Kirschbuam, Kudielka, Gaab, Schommer, & Helhammer, 1999). Finally, most studies have been limited to clinical or middle class samples. Our findings are limited to an economically disadvantaged population. To better understand the effects of demographic variables and ethnicity on children’s stress responses, larger and more representative samples will be required.

Acknowledgments

This research was supported by a grant from the National Institute of Mental Health (RO1MH59670) to Roger Kobak. We are grateful to the caregivers and adolescents who participated and to the many individuals who provided invaluable help with data collection, especially Laura Reilly, Alison Esposito, Natalie Rosenthal, and Nicole Bracy.

Contributor Information

Roger Kobak, Department of Psychology, University of Delaware.

Kristyn Zajac, Department of Psychology, University of Delaware.

Seymour Levine, University of California at Davis.

References

  1. Achenbach TM, Rescorla LA. Manual for the ASEBA school-age forms and profiles. Burlington, VT: ASEBA; 2001. [Google Scholar]
  2. Ackerman BP, Kogos J, Youngstrom E, Schoff K, Izard C. Family instability and the problem behaviors of children from economically disadvantaged families. Developmental Psychology. 1999;35:258–268. doi: 10.1037//0012-1649.35.1.258. [DOI] [PubMed] [Google Scholar]
  3. Adam EK, Klimes-Dougan B, Gunnar M. Social regulation of the adrenocortical response to stress in infants, children, and adolescents: Implications for psychopathology and education. In: Coch D, Dawson G, Fischer K, editors. Human behavior and the developing brain: Atypical development. New York: Guilford Press; 2007. pp. 264–304. [Google Scholar]
  4. Allen JP, Aber JL, Leadbeater BJ. Adolescent problem behaviors: The influence of attachment and autonomy. Psychiatric Clinics of North America. 1990;13:455–467. [PubMed] [Google Scholar]
  5. Allen JP, Hauser ST, Bell KL, O’Connor TG. Longitudinal assessment of autonomy and relatedness in adolescent-family interactions as predictors of adolescent ego development and self-esteem. Child Development. 1994;65:179–194. doi: 10.1111/j.1467-8624.1994.tb00743.x. [DOI] [PubMed] [Google Scholar]
  6. Bauer A, Quas JA, Boyce WT. Associations between physiological reactivity and children’s behavior: Advantages of a multisystem approach. Journal of Developmental and Behavioral Pediatrics. 2002;23:102–112. doi: 10.1097/00004703-200204000-00007. [DOI] [PubMed] [Google Scholar]
  7. Boyce WT, Ellis BJ. Biological sensitivity to context: I. An evolutionary-developmental theory of the origins and functions of stress reactivity. Development and Psychopathology. 2005;17:271–301. doi: 10.1017/s0954579405050145. [DOI] [PubMed] [Google Scholar]
  8. Caspi A, Moffit T. Individual differences are accentuated during periods of social change: The sample case of girls at puberty. Journal of Personality and Social Psychology. 1991;61:157–168. doi: 10.1037//0022-3514.61.1.157. [DOI] [PubMed] [Google Scholar]
  9. Cicchetti D, Rogosch FA. The impact of child maltreatment and psychopathology on neuroendocrine functioning. Development and Psychopathology. 2001;13:783–804. [PubMed] [Google Scholar]
  10. Dishion TJ, Patterson GR. The development and ecology of antisocial behavior in children and adolescents. In: Cicchetti D, Cohen DJ, editors. Developmental Psychopathology: Risk, Disorder and Adaptation. Vol. 3. Hoboken, NJ: Wiley Sons; 2006. pp. 503–541. [Google Scholar]
  11. Dishion TJ, Loeber R. Adolescent marijuana and alcohol use: The role of parents and peers revisited. American Journal of Drug and Alcohol Abuse. 1985;11:11–25. doi: 10.3109/00952998509016846. [DOI] [PubMed] [Google Scholar]
  12. Dishion TJ, Andrews DW. Preventing escalation in problem behaviors with high-risk young adolescents: Immediate and 1-year outcomes. Journal of Consulting and Clinical Psychology. 1995;63:538–548. doi: 10.1037//0022-006x.63.4.538. [DOI] [PubMed] [Google Scholar]
  13. Ellis BJ, Essex MJ, Boyce WT. Biological sensitivity to context: II. Empirical explorations of an evolutionary-developmental theory. Development and Psychopathology. 2005;17:303–328. doi: 10.1017/s0954579405050157. [DOI] [PubMed] [Google Scholar]
  14. Flinn MV, England BG. Childhood stress and family environment. Current Anthropology. 1995;36:854–866. [Google Scholar]
  15. Ge X, Brody G, Conger RD, Simons R, Murray V. Contextual amplification of pubertal transition effects on deviant peer affiliation and externalizing behavior among African American children. Developmental Psychology. 2002a;38(1):42–54. doi: 10.1037//0012-1649.38.1.42. [DOI] [PubMed] [Google Scholar]
  16. Ge X, Brody G, Conger RD, Simons R, Murray V. Pubertal transition, stressful life events, and the emergence of gender differences in adolescent depressive symptoms. Developmental Psychology. 2002b;37(3):404–17. doi: 10.1037//0012-1649.37.3.404. [DOI] [PubMed] [Google Scholar]
  17. Gordis E, Granger DA, Susman EJ, Trickett PK. Asymmetry between salivary cortisol and alpha-amylase reactivity to stress: Relation to aggressive behavior in adolescents. Psychoneuroendocrinology. 2006;31:976–987. doi: 10.1016/j.psyneuen.2006.05.010. [DOI] [PubMed] [Google Scholar]
  18. Granger DA, Weisz JR, Kauneckis D. Neuroendocrine reactivity, internalizing behavior problems, and control-related cognitions in clinic-referred children and adolescents. Journal of Abnormal Psychology. 1994;103:267–276. doi: 10.1037//0021-843x.103.2.267. [DOI] [PubMed] [Google Scholar]
  19. Granger DA, Weisz JR, McCracken JT, Ikeda SC, Douglas P. Reciprocal influences among adrenocortical activation, psychosocial processes, and the behavioral adjustment of clinic-referred children. Child Development. 1996;67:3250–3262. [PubMed] [Google Scholar]
  20. Gunnar MR, Vazquez DM. Low cortisol and a flattening of expected daytime rhythm: Potential indices of risk in human development. Development and Psychopathology. 2001;13:515–538. doi: 10.1017/s0954579401003066. [DOI] [PubMed] [Google Scholar]
  21. Kirschbaum C, Kudielka BM, Gaab J, Schommer NC, Hellhammer DH. Impact of gender, menstrual cycle phase, and oral contraceptives on the activity of the hypothalamus-pituitary-adrenal axis. Psychosomatic Medicine. 1999;61:154–162. doi: 10.1097/00006842-199903000-00006. [DOI] [PubMed] [Google Scholar]
  22. Kivlighan KT, Granger DA, Schwartz EB, Nelson V, Curran M, Shirtcliff EA. Quantifying blood leakage into the oral mucosa and its effects on the measurement of cortisol, dehydroepiandrosterone, and testosterone in saliva. Hormones & Behavior. 2004;46:39–46. doi: 10.1016/j.yhbeh.2004.01.006. [DOI] [PubMed] [Google Scholar]
  23. Klimes-Dougan B, Hastings PD, Granger DA, Usher BA, Zahn-Waxler C. Adrenocortical activity in at-risk and normally developing adolescents: Individual differences in salivary cortisol basal levels, diurnal variation, and responses to social challenges. Development and Psychopathology. 2001;13:695–719. doi: 10.1017/s0954579401003157. [DOI] [PubMed] [Google Scholar]
  24. Kobak RR, Cole HE, Ferenz-Gillies R, Fleming WS, Gamble W. Attachment and emotion regulation during mother-teen problem solving: A control theory analysis. Child Development. 1993;64:231–245. [PubMed] [Google Scholar]
  25. McBurnett K, Lahey BB, Rathouz PJ, Loeber R. Low salivary cortisol and persistent aggression in boys referred for disruptive behavior. Archives of General Psychiatry. 2000;57:38–43. doi: 10.1001/archpsyc.57.1.38. [DOI] [PubMed] [Google Scholar]
  26. McBurnett K, Raine A, Stouthamer-Loeber M, Loeber R, Kumar AM, Kumar M, et al. Mood and hormone responses to psychological challenge in adolescent males with conduct problems. Biological Psychiatry. 2005;57:1109–1116. doi: 10.1016/j.biopsych.2005.01.041. [DOI] [PubMed] [Google Scholar]
  27. Metzler CW, Noell J, Biglan A. The validation of a construct of high-risk sexual behavior in heterosexual adolescents. Journal of Research on Adolescence. 1992;7:233–249. [Google Scholar]
  28. Moffit T, Caspi A, Rutter M, Silva PA. Sex differences in anti-social behavior: Conduct disorder, delinquency, and violence in the Dunedin longitudinal study. Cambridge, England: Cambridge University Press; 2001. [Google Scholar]
  29. Moffitt T. Life-course persistent versus adolescent-limited antisocial behavior. In: Cicchetti D, Cohen DJ, editors. Developmental Psychopathology: Risk, Disorder and Adaptation. Hoboken, NJ: Wiley & Sons; 2006. pp. 570–598. [Google Scholar]
  30. Pajer K, Gardner W, Rubin RT, Perel J, Neal S. Decreased cortisol levels in adolescent girls with conduct disorder. Archives of General Psychiatry. 2001;58:297–302. doi: 10.1001/archpsyc.58.3.297. [DOI] [PubMed] [Google Scholar]
  31. Raine A. Biosocial studies of antisocial and violent behavior in children and adults: A review. Journal of Abnormal Child Psychology. 2002;30:311–326. doi: 10.1023/a:1015754122318. [DOI] [PubMed] [Google Scholar]
  32. Shirtcliff EA, Granger DA, Booth A, Johnson D. Low salivary cortisol levels and externalizing behavior problems in youth. Development and Psychopathology. 2005;17:167–184. doi: 10.1017/s0954579405050091. [DOI] [PubMed] [Google Scholar]
  33. Snoek H, van Goozen SH, Matthys W, Buitelaar JK, van Engeland H. Stress responsivity in children with externalizing behavior disorders. Development and Psychopathology. 2004;16:389–406. doi: 10.1017/s0954579404044578. [DOI] [PubMed] [Google Scholar]
  34. Stroud LR, Salovey P, Epel ES. Sex differences in stress responses: Social rejection versus achievement stress. Biological Psychiatry. 2002;52:318–327. doi: 10.1016/s0006-3223(02)01333-1. [DOI] [PubMed] [Google Scholar]
  35. Stroud LR, Papandonatos GD, Williamson DE, Dahl RE. Sex differences in the effects of pubertal development on responses to a corticotropin-releasing hormone challenge: The Pittsburgh psychobiologic studies. New York: New York Academy of Sciences; 2004. [DOI] [PubMed] [Google Scholar]
  36. Susman EJ, Pajer K. Biology-behavior integration and antisocial behavior in girls (pp. 23–47) New York, NY, US: Guilford Publications; 2004. [Google Scholar]
  37. van de Wiel NMH, van Goozen SH, Matthys W, Snoek H, van Engeland H. Cortisol and treatment effect in children with disruptive behavior disorders: A preliminary study. Journal of the American Academy of Child and Adolescent Psychiatry. 2004;43:1011–1018. doi: 10.1097/01.chi.0000126976.56955.43. [DOI] [PubMed] [Google Scholar]
  38. van Goozen SH, Fairchild G, Snoek H, Harold GT. The evidence for a neurobiological model of childhood antisocial behavior. Psychological Bulletin. 2007;133:149–182. doi: 10.1037/0033-2909.133.1.149. [DOI] [PubMed] [Google Scholar]
  39. van Goozen SH, Matthys W, Cohen-Kettenis PT, Buitelaar JK, van Engeland H. Hypothalamic-pituitary-adrenal axis and autonomic nervous system activity in disruptive children and matched controls. Journal of American Academy of Child and Adolescent Psychiatry. 2000;39:1438–1445. doi: 10.1097/00004583-200011000-00019. [DOI] [PubMed] [Google Scholar]
  40. van Goozen SH, Matthys W, Cohen-Kettenis PT, Gispen-de Wied C, Wiegant VM, van Engeland H. Salivary cortisol and cardiovascular activity during stress in oppositional-defiant disorder boys and normal controls. Biological Psychiatry. 1998;43:531–539. doi: 10.1016/S0006-3223(97)00253-9. [DOI] [PubMed] [Google Scholar]

RESOURCES