Trait and State Variance in Oppositional Defiant Disorder Symptoms: A Multi-Source Investigation with Spanish Children

Abstract

The objective was to determine and compare the trait and state components of oppositional defiant disorder (ODD) symptom reports across multiple informants. Mothers, fathers, primary teachers, and secondary teachers rated the occurrence of the ODD symptoms in 810 Spanish children (55% boys) on two occasions (end first and second grades). Single source latent state-trait (LST) analyses revealed that ODD symptom ratings from all four sources showed more trait (M = 63%) than state residual (M = 37%) variance. A multiple source LST analysis revealed substantial convergent validity of mothers’ and fathers’ trait variance components (M = 68%) and modest convergent validity of state residual variance components (M = 35%). In contrast, primary and secondary teachers showed low convergent validity relative to mothers for trait variance (Ms = 31%, 32%, respectively) and essentially zero convergent validity relative to mothers for state residual variance (Ms = 1%, 3%, respectively). Although ODD symptom ratings reflected slightly more trait- than state-like constructs within each of the four sources separately across occasions, strong convergent validity for the trait variance only occurred within settings (i.e., mothers with fathers; primary with secondary teachers) with the convergent validity of the trait and state residual variance components being low to non-existent across settings. These results suggest that ODD symptom reports are trait-like across time for individual sources with this trait variance, however, only having convergent validity within settings. Implications for assessment of ODD are discussed.

Oppositional Defiant Disorder (ODD) is considered a chronic condition with the primary characteristic being a “frequent and persistent pattern of angry/irritable mood, argumentative/defiant behavior, or vindictiveness” (American Psychiatric Association [APA], 2013, p. 463). The current conceptualization of ODD allows for the possibility that the occurrence of symptoms may be limited to home (i.e., “It is not uncommon for individuals with oppositional defiant disorder to show symptoms only at home and only with family members,” APA, 2013, p. 463). ODD can, however, also occur across multiple settings (e.g., home, school, and community) with such multiple setting occurrences being used to indicate the severity of the disorder (i.e., more settings indicates greater severity of ODD, APA, 2013, pp. 462–463).

The view of ODD has implications for research on the ODD symptom dimension. First, since ODD is considered a chronic condition, the ODD dimension should show meaningful levels of stability across time. Second, since ODD can occur across multiple settings, the ODD dimension should also show meaningful levels of consistency across settings. In other words, ODD symptom reports are assumed to reflect a more trait- than state-like construct. However, no study has explicitly determined the amount of trait (consistent) and state residual (variable) variance in ODD symptom reports. Determining the trait and state residual components in ODD symptom reports from different sources in different settings will help researchers better understand (1) how stable ODD symptoms are within different sources and (2) to what extent the stable and variable components of ODD symptom reports show convergent validity across sources within and across settings. For example, are ODD symptom reports similarly trait-like in the home (parent ratings) and school context (teacher ratings)? To which degree do parents’ and teachers’ reports converge on the trait-like aspects of ODD (i.e., trait-like across settings)? In the present study, latent state-trait (LST) analyses were used to answer these and other questions.

Research outside of the state-trait paradigm supports the trait-like view of the ODD symptom dimension. In terms of stability, Boylan, Vaillancourt, & Szatmari (2012) measured ODD symptoms in a sample of 932 four and five year olds five times over eight years and found that baseline ODD symptoms had significant correlations of .61, .51, .53, and .43 with ODD symptoms 2, 4, 6, and 8 years later, respectively (see also Lavigne et al., 2001; Whelan, Stringaris, Maughan, & Barker, 2013). In terms of consistency across settings, although correlations between different sources within the same settings (mothers with fathers; teachers with teachers) tend to be higher than between sources across different settings (parents with teachers), correlations between home and school are usually significant (e.g., Bernad, Servera, Grases, Collado, & Burns, 2014; Burns et al., 2008; Burns, Desmul, Walsh, Silpakit, & Ussahawanitchaki, 2009; Burns, Walsh, et al., 2013).

Although these findings suggest that the ODD dimension may be a trait-like construct, no study has determined which proportion of ODD symptom reports reflect (1) trait, (2) state residual, and (3) measurement error variance. This question can be answered by using LST measurement models that allow decomposing (1) observed ODD symptom variance into true score variance and error variance and (2) true score variance into trait (stable) and state residual (occasion-specific) variance (Steyer, Geiser, & Fiege, 2012).

Studies that have applied LST models to other clinical constructs such as anxiety (Olatunji & Cole, 2009), depression (Windle & Dumenci, 1998), ADHD-inattention (IN), and ADHD-hyperactivity/impulsivity (HI) symptom dimensions (Litson, Geiser, Burns, & Servera, in press) have shown substantial amounts of state residual variance (22% to 80%). These results indicate that it may also be important to distinguish between trait and state residual variance components in the ODD construct to determine the true level of stability and the true level of variability in this construct across time within different sources and settings.

The separation of trait and state residual variance in ODD symptoms also has implications for the analysis of the convergent validity of the ODD dimension within and across settings (Courvoisier, Nussbeck, Eid, Geiser, & Cole, 2008; Litson et al., in press). A multiple source LST model allows one to determine the proportion of trait and state residual variance that is either shared or not shared between a reference source (e.g., mothers) and non-reference sources (e.g., fathers, teachers). With two sources in the home (mothers and fathers) and two sources in the school (primary and secondary teachers), Litson and colleagues (in press) showed that trait variance in ADHD-IN and ADHD-HI constructs showed high levels of convergent validity within settings (78% to 82%) and moderate convergent validity across settings (i.e., school to home, 41% to 63%). State residual variance showed small to moderate convergent validity within settings (i.e., 29% to 53%) but essentially zero (0% to 2%) convergent validity across settings. Given the high comorbidity between ADHD and ODD—(Kimonis, Frick, & McMahon, 2014), we expected similar findings for ODD symptom ratings. However, because ADHD is considered a more pervasive disorder than ODD (APA, 2013), we expected more state residual variance for the ODD symptoms and especially less consistency across settings than for the ADHD symptoms.

We now describe in greater detail how single and multiple source LST models provide a better understanding of (1) the stability of the ODD construct across time within sources and (2) the convergent validity of trait and state residual components of the ODD construct within (i.e., mothers with fathers; primary teachers with secondary teachers) and across settings (parents with teachers).

Measurement of Trait and State Components of ODD Symptoms with Multiple Informants

With a minimum of two occasions of measurement, a single source LST model can separate the variance in ODD symptoms into trait variance (i.e., see trait consistency definition in Table 1), state residual variance (i.e., see occasion-specificity definition Table 1), and measurement error variance (unreliability; Steyer et al., 2012). Trait variance represents the amount of true score variance in ODD symptom reports that is purely person-specific and independent of the occasion and/or person-occasion interactions. State residual (occasion-specific) variance, in contrast, represents the amount of true score variance in ODD symptom ratings that reflects occasion-specific (situational) influences and/or person-occasion interactions. This LST model can determine if ODD symptoms are more trait- or more state-like for an individual source across multiple occasions of measurement. Figure 1 shows the single source LST model used in the present study. Table 1 shows the equations and coefficients for this model and the online supplemental materials the Mplus code.

Table 1.

Equations, Coefficients, and Definitions for Single and Multiple Source Latent State Trait Models

Equations/Coefficients	Definitions
Equations for Single Source Latent State Trait Model
Y­it = τit + εit	An observed variable (e.g., symptom ratings by mothers) is decomposed into a true score variable plus a measurement error variable.
τ­it = Ti + γiOt	A true score variable is decomposed into a variable-specific trait factor plus an occasion-specific (state) residual factor.
Y­it = Ti + γiOt + εit	Combing the two previous equations leads to the single-source latent state trait measurement model shown in Figure 1.
Equations for Multiple Source Latent State Trait Model
Y­ist = τist + εist	An observed variable is decomposed into a true score variable plus a measurement error variable.
τ­ist = Ti1 + Oi1t, for s = 1 τ­ist = λisTi1 + γis Oi1t,+ TSis + δisOst, for s ≠ 1	A latent true score variable is decomposed into a variable-specific trait factor pertaining to the reference source (i.e., mothers), a variable-specific state residual factor pertaining to the reference source (i.e., mothers), a source-specific trait residual factor (i.e., fathers, primary teachers, or secondary teachers), and a source- specific state residual factor (i.e., fathers, primary teachers, or secondary teachers).
Y­ist = Ti1 + Oi1t, + εist, for s = 1 Y­ist = λisTi1 + γis Oi1t, + TSis + δisOSst + εist, for s ≠ 1	Combing the two previous equations leads to the multiple-source latent state trait measurement model shown in Figure 2.
Coefficients for Singe Source Latent State Trait Model
$\mathit{Con}({\tau }_{it})=\frac{\mathit{Var}(T_i)}{\mathit{Var}({\tau }_{it})}=\frac{\mathit{Var}(T_i)}{\mathit{Var}(T_i)+{\gamma }_i^2\mathit{Var}(O_t)}$	Trait-consistency: The proportion of true score (error-free) variance that reflects stable trait variance. Consistency reflects true score variance that is purely person-specific and independent of the situation and/or person-situation interaction.
$\mathit{OSpe}({\tau }_{it})=\frac{{\gamma }_i^2\mathit{Var}(O_t)}{\mathit{Var}({\tau }_{it})}=\frac{{\gamma }_i^2\mathit{Var}(O_t)}{\mathit{Var}(T_i)+{\gamma }_i^2\mathit{Var}(O_t)}$	Occasion-specificity: The proportion of true score variance that reflects occasion-specific influences and/or person-situation interactions.
$\mathit{Rel}(Y_{it})=\frac{\mathit{Var}({\tau }_{it})}{\mathit{Var}(Y_{it})}=\frac{\mathit{Var}(T_i)+{\gamma }_i^2\mathit{Var}(O_t)}{\mathit{Var}(T_i)+{\gamma }_i^2\mathit{Var}(O_t)+\mathit{Var}({\epsilon }_{it})}$	Reliability: The proportion of observed score variance that is due to systematic sources of variance (Ti + γi Ot) rather than measurement error.
Coefficients for Multiple Source Latent State Trait Model
$S\_\mathit{Con}(T_{\mathit{ist}})=\frac{{\lambda }_{is}^2\mathit{Var}(T_{i1})}{{\lambda }_{is}^2\mathit{Var}(T_{i1})+\mathit{Var}(T{S}_{is})}$	Shared trait consistency: The proportion of trait variance in a non-reference source (i.e., fathers, primary teachers, or secondary teachers) that is shared with the reference source (i.e., mothers). Indicates convergent validity for trait components.
$U\_\mathit{Con}(T_{\mathit{ist}})=\frac{\mathit{Var}(T{S}_{is})}{{\lambda }_{is}^2\mathit{Var}(T_{i1})+\mathit{Var}(T{S}_{is})}$	Unique trait consistency: The proportion of trait variance in a non-reference source (i.e., fathers, primary teachers, or secondary teachers) that is not shared with the reference source (i.e., mothers). Indicates source-specificity of trait components.
$S\_\mathit{OSpe}(O_{\mathit{ist}})=\frac{{\gamma }_{is}^2\mathit{Var}(O_{i1t})}{{\gamma }_{is}^2\mathit{Var}(O_{i1t})+{\delta }_{is}^2\mathit{Var}(O{S}_{st})}$	Shared occasion-specificity: The proportion of occasion-specific variance in a non-reference source (i.e., fathers, primary teachers, or secondary teachers) that is shared with the reference source (i.e., mothers). Indicates convergent validity of state residual components.
$U\_\mathit{OSpe}(O_{\mathit{ist}})=\frac{{\delta }_{is}^2\mathit{Var}(O{S}_{st})}{{\gamma }_{is}^2\mathit{Var}(O_{i1t})+{\delta }_{is}^2\mathit{Var}(O{S}_{st})}$	Unique occasion-specificity: The proportion of occasion-specific variance in a non-reference source (i.e., fathers, primary teachers, or secondary teachers) that is not shared with the reference source (i.e., mothers). Indicates source-specificity of state residual components.

Note. Y = observed variable; τ = latent true score variable; ε = measurement error variable; T_i = indicator-specific trait factor; O_t = occasion-specific residual factor; γ_i = occasion-specific factor loading; T_i1 = indicator-specific trait factor specific to the reference method; O_i1t = occasion residual factor specific to the reference method; TS_is = source-specific residual trait factor; OS_st = source-specific occasion residual factor; λ_is = trait factor loadings; γ_is = occasion residual factor loadings; δ_is = source-specific occasion residual factor loadings. For all equations, i indicates the ith measured variable (indicator), s indicates the sth source (in the present study, the type of informant), and t indicates the tth time point.

Figure 1.

Single source latent state-trait model with parcel-specific trait factors measured at two time points. This model was applied to each of the four sources separately. The Mplus code for this model is shown in the supplemental materials T = trait factor; O = state (occasion-specific) residual factor.

A multiple source LST model (Courvoisier et al., 2008; Litson et al., in press) differs from a single source LST model by the simultaneous analysis of two or more source’s ratings. By including multiple sources in the same model, it is possible to study the convergent validity of trait and state residual components across sources. The multiple source LST model requires that one source be selected as reference source (e.g., mothers) with the other sources being the non-reference sources (e.g., fathers, teachers). Usually the most knowledgeable source about the behavior or the nature of the research question dictates the choice of the reference and non-reference sources (Geiser, Eid, & Nussbeck, 2008). The multiple source LST model can determine the amount of trait and state residual variance in ODD symptom reports that the non-reference sources either share or do not share with the reference source.

This multiple source LST model results in four variance components for the non-reference sources (see Table 1)—(1) shared trait consistency, or the proportion of trait variance in ODD symptom reports that a non-reference source shares with the reference source; shared trait consistency thus indicates the degree of convergent validity with regard to stable (trait) aspects of behavior; (2) unique trait consistency, or the proportion of trait variance in ODD symptom ratings that a non-reference source does not share with the reference source; unique trait consistency thus indicates the degree of source-specificity with regard to stable (trait) aspects of behavior; (3) shared occasion-specificity, or the proportion of occasion-specific variance in ODD symptom reports that a non-reference source shares with the reference source; shared occasion-specificity thus indicates the degree of convergent validity with regard to variable (state residual) aspects of behavior; and (4) unique occasion-specificity, or the proportion of occasion-specific variance in ODD symptoms that is unique to a non-reference source and not shared with the reference source; unique occasion-specificity thus indicates the degree of source-specificity with regard to variable (state residual) aspects of behavior.

The separation of true score variance into these four components for the non-reference sources allows for a sophisticated analysis of both temporal consistency and cross-informant convergent validity of trait and state residual components in ODD symptom reports. Figure 2 shows the multiple source LST model used in the present study with the equations/coefficients for this model shown in the Appendix and the Mplus code in the supplemental materials.

Figure 2.

Multiple source latent state-trait model with parcel specific trait factors. In order to reduce the complexity of the figure, only three sources (mothers, fathers, and primary teachers) are shown whereas the actual model involved four sources (mothers, fathers, primary teachers, and secondary teachers). A single residual trait factor was sufficient for the secondary teachers rather than three indicator-specific residual trait factors used for fathers and primary teachers (see Mplus code in the supplemental materials). T = reference trait factors that are defined by mother reports and are parcel-specific across sources; TS = residual trait factors that pertain to the non-reference sources and are source- and parcel-specific; O = reference state residual factors that are defined by mother reports and are parcel-specific across sources; OS = state residual factors that pertain to the non-reference sources and are source-specific across parcels.

Objectives of the Study

The present study used ratings of ODD symptoms by mothers, fathers, primary teachers, and secondary teachers of Spanish children at the completion of the first and second grades to pursue two objectives. The first objective was to determine the proportion of trait variance (i.e., temporal consistency, see Table 1) and state residual variance (i.e., occasion-specificity, see Table 1) in the ODD symptoms for each source separately. For this purpose, we fit a separate single source LST model to each source’s ratings (Figure 1). It was predicted that ODD symptoms would show a larger amount of trait than state residual variance within each source. Such results would indicate that ODD symptoms were more trait- than state-like within each source over a 12-month interval.

The second objective was to determine the proportion of trait and state residual variance in ODD symptoms that father, primary teacher, and secondary teacher reports either shared or did not share with mother reports. For this purpose, we fit a multiple source LST model to the four sources simultaneously (Figure 2). This multiple source LST analysis separated the trait variance into shared trait consistency (i.e., the trait variance that fathers, primary teachers, and secondary teachers shared with mothers, see Table 1) and unique trait consistency (i.e., the trait variance that fathers, primary teachers, and secondary teachers did not share with mothers, see Table 1). This analysis also separated the occasion-specific variance into shared occasion-specificity (i.e., occasion-specific variance that fathers, primary teachers, and secondary teachers share with mothers, see Table 1) and unique occasion-specificity (i.e., occasion-specific variance that fathers, primary teachers, and secondary teachers did not share with mothers, see Table 1). It was predicted that fathers would share a large amount of trait variance with mothers.

It was also predicted that fathers would share a smaller amount of state residual variance with mothers than was the case for the trait variance. In contrast, it was predicted that primary and secondary teachers would share little trait and state residual variance with mothers. Such results would indicate convergent validity for trait variance within the home setting but little convergent validity for trait or state variance across home and school settings. These outcomes in conjunction with the predicted outcome for the single source LST analyses would indicate that while ODD ratings are more trait- than state-like for individual sources across time this trait variance only has convergent validity for sources within the same setting.

So far no study has used LST models to investigate the trait and state components of ODD symptom reports across occasions of measurement, sources, and settings. Our findings should indicate if ODD symptoms reflect a more trait- or state-like construct for different sources as well as the degree of convergent validity of trait and state residual components within and across settings.

Method

Participants and Procedures

The 46 elementary schools on Majorca (Spain) were invited to participate with 43 expressing interest. Twenty-two of these 43 schools were then randomly selected with eight additional schools from Madrid also invited (eight were asked and eight agreed to participate). These eight schools were a sample of convenience (the schools had helped with other projects) and, since resources on Majorca only allowed data collection at 22 schools, the Madrid schools were recruited to increase the overall sample size. The potential participants in the study were the mothers, fathers, primary teachers (i.e., the children’s main classroom teacher) and secondary teachers (i.e., the children’s teachers of special subjects such as English, Catalan language, music, visual arts, and physical education) of the 1045 first grade children in these 30 schools.

There were 758 unique children at the first grade assessment (55% boys) and 718 (54% boys) at the second grade assessment with 810 unique children available for the analyses (it was possible for a child to have ratings at assessment 2 who did not have ratings at assessment 1). At the first assessment in the spring of the school year, the average age of children was approximately 7 years with little variation. Although ethnicity was not collected for the individual children, approximately 90% of the first grade children were Caucasian with 10% North African (the schools provided the ethnicity and age information at the grade level). It was not possible to collect social-economic-status information on the families. A cover letter that explained the purpose of the study was given to the parents. Parents also gave written consent for primary and secondary teachers to complete the ratings. Teachers also gave written consent. The protocol was approved by the University of the Balearic Islands’ IRB.

The ideal was for each child’s behavior to be rated by a mother, father, primary teacher, and a secondary teacher at assessments one and two (12-months later). For the assessment at the end of the first grade, 723 mothers and 603 fathers returned the measures with the numbers for the second assessment being 604 mothers and 540 fathers. At the first grade assessment, 61 primary teachers and 49 secondary teachers from 28 of 30 schools participated in the study (even when a school agreed to participate, primary and secondary teachers were still free to decline participation) with the primary teachers rating an average of 11.76 (SD = 5.09, n = 743) children and the secondary teachers an average of 9.02 (SD = 6.58, n = 574) children. For the second grade assessment, 62 primary teachers and 59 secondary teachers from 29 of 30 schools participated in the study with the primary teachers rating an average of 11.19 (SD = 4.43, n = 701) children and the secondary teachers an average of 10.45 (SD = 5.18, n = 664) children. Children had the same primary teacher for the two occasions of measurement with most of the children having same secondary teacher for the two occasions as well.

Measure

Child and Adolescent Disruptive Behavior Inventory (CADBI, Burns et al., 2014)

Mothers, fathers, primary teachers, and secondary teachers completed the parent and teacher CADBI. This study used the ODD toward adults (e.g., argues with adults; eight symptoms) and ODD toward peers (e.g., argues with siblings/peers [siblings/peers for parent scale and peers for teacher scale]; eight symptoms) subscales of the CADBI. Table 2 shows the ODD toward adults and ODD toward siblings/peers’ items on the parents’ scale. The 16 items are identical on the teachers’ scale except for the use of only the word “peers” rather than “siblings/peers”. Table 2 also shows how the 16 items were assigned to three parcels. This assignment process is explained later.

Table 2.

Sixteen Items on the ODD toward adults and ODD toward Siblings/Peers Subscales of the Parent Measure

Parcel 1

Spiteful or vindictive toward adults (e.g., says mean things to hurt adults’ feelings or does mean things to get back at adults) Spiteful or vindictive toward siblings/peers (e.g., says mean things to hurt siblings/peers’ feelings or does mean things to get back at siblings/peers Argues with adults Argues with siblings/peers Actively defies or refuses to obey adults’ requests or rules Refuses to cooperate with reasonable requests from siblings/peers

Parcel 2

Annoys adults on purpose Annoys siblings/peers on purpose Becomes annoyed or irritated by the behavior of adults Becomes annoyed or irritated by the behavior of siblings/peers Appears angry or resentful toward adults

Parcel 3

Loses temper with adults when doesn’t get own way Loses temper with siblings/peers when doesn’t get own way Blames adults for his or her own mistakes or misbehavior Blames peers for his or her own mistakes or misbehavior Appears angry or resentful toward siblings/peers

Note. The 16 items on the ODD toward adults and ODD toward peers subscales of the teacher scale were identical except the phrase “siblings/peers” was changed to just “peers.” The assignment of items to the three parcels was the same for items from the teacher scale. ODD = oppositional defiant disorder.

The 16 symptoms were rated on a 6-point scale (i.e., almost never [never or about once per month], seldom [about once per week], sometimes [several times per week], often [about once per day], very often [several times per day], and almost always [many times per day]). Mothers and fathers were asked to make their ratings on the basis of the children’s behavior in the home and community and to not consider reports of behavior at school. These instructions were designed so mothers and fathers were asked to rate the overall level of ODD behavior in the home and community rather than ODD behavior just toward the specific source completing the measure. Mothers and fathers were also asked to make their ratings independently. Primary teachers and secondary teachers were instructed to base their ratings on the children’s behavior at school and to make their ratings independently. Earlier studies support the reliability and validity of scores on the ODD scales (Burns et al., 2008, 2009; Burns, Walsh, et al., 2013).

Analyses

Analytic strategy

The item-level analyses treated the items as ordered categorical and used the robust weighted least squares estimator (WLSMV estimator, Mplus statistical software version 7.4). The parcel-level analyses used the robust maximum likelihood estimation (MLR estimator). All the analyses took into account that the children were nested within teachers (Type = complex Mplus option.). The fit of measurement models was evaluated with the comparative fit index (CFI, ideal study criterion ≥ .95), Tucker-Lewis Index (TLI, ideal study criterion ≥ .95), and the root mean square error of approximation (RMSEA, ideal study criterion ≤ .05). The evaluation of the invariance analyses used changes in the CFI value. If the decrease in the CFI value was less than .01 with the introduction of a set of constraints, then the constraints were assumed to be tenable (Little, 2013, chap. 5).

Preliminary analyses on the individual items to justify the creation of parcels

A series of preliminary confirmatory factor analyses were performed at the item level (i.e., 16 items for each source, that is, an ODD factor with the eight ODD symptoms where the target of the oppositional behavior was an adult, and an ODD factor with the eight ODD symptoms where the target of the oppositional behavior was a sibling/peer). The purpose of these analyses was to evaluate the invariance of like item loadings and like item thresholds across the two occasions of measurement within each source (i.e., a separate invariance analysis for mothers, fathers, primary teachers, and secondary teachers). Like item loadings and like item thresholds were also held equal across the two factors (ODD toward adults and ODD toward peers) at each assessment since the items were identical other than the target of the oppositional behavior. To summarize, like item loadings and like item thresholds were constrained equal across the two occasions of measurement as well as across the two ODD factors at each occasion for each of the four sources (a separate set of analyses for each source).

The second set of analyses tested whether the variables showed measurement invariance across occasions (assessments 1 and 2), constructs (ODD toward adults and ODD toward siblings/peers), and sources within each setting (mothers and fathers; primary and secondary teachers). That is, we tested whether the symptom ratings showed invariance of like symptom loadings and thresholds across occasions and constructs between (1) mother and father ratings and (2) between primary and secondary teacher ratings. If the invariance analysis for the ratings by mothers and fathers yielded invariant like item loadings and thresholds across factors, sources, and occasions, then such would indicate measurement invariance of the ODD factor across mothers and fathers for the two occasions. Such a result would also yield one set of eight factor loadings rather than eight separate sets of eight loadings (i.e., four sets at each occasions of measurement [two sets for mothers and two sets for fathers] times two occasions of measurement). This same invariance analysis was also applied to the primary and secondary teacher ratings for the two factors for the two occasions of measurement. It was not feasible to perform such an invariance analysis on the four sources simultaneously due to the large number of manifest variables for such an analysis (128 manifest variables).

The one set of eight loadings for mothers and fathers and the one set of eight loadings for teachers and aides were then used to create the parcels for the LST analyses. First, the two sets of eight loadings—one set for mothers and fathers and one set for primary teachers and secondary teachers—were ranked from highest to lowest. It was expected that the rank ordering of the loadings within each set would be similar with there also being little variability from lowest to highest. Given such results, it would be possible to establish the same rank ordering of the eight symptoms for mothers/fathers and for primary teachers/secondary teachers. The next step involved the use of a procedure recommended by Little (2013) to assign the items to parcels.

This procedure involved assigning the item with the highest and lowest loading to parcel one, the item with the second highest and second lowest to parcel two, and the item with the third highest and third lowest to parcel three with the repetition of this process until all items were assigned to the three parcels. This procedure increases the likelihood of homogenous parcels. This procedure results in a single ODD factor for each source (i.e., mothers, fathers, primary teachers, and secondary teachers) at assessments one and two with each ODD factor defined by three parcels.

To determine if the ODD factor defined by parcels was invariant over the two occasions of measurement, an invariance analysis was applied to each source (i.e., invariance of like parcel loadings and like parcel intercepts across the two occasions). A similar analysis was then applied to mothers and fathers simultaneously, primary teachers and secondary teachers simultaneously, and finally all four sources simultaneously. It was expected that the ODD factor defined by the parcels would show measurement invariance across the two occasions of measurement and four sources. These preliminary analyses were used to provide the foundation for the LST analyses.

Single source latent state trait analyses

A single source LST analysis with parcel-specific trait factors (Eid, 1996) was used to determine the proportion of trait (consistency) and state residual variance (occasion-specificity) for each source separately across the two assessments. Figure 1 shows this model with the equations and coefficients for this model shown in the Table 1. This model requires all the trait factor loading to be set to 1.0 and all occasion specific factor loadings γ_i to be held equal across occasions to establish measurement invariance (Geiser et al., 2015). The model also requires that the occasion-specific factors not be correlated with the trait factors and that error variables not be correlated with occasion-specific or trait factors to allow for additive variance decomposition (Steyer et al. 2012). The additive variance decomposition allows for the calculation of the proportion of variance accounted for by consistency (stable trait effects), occasion-specificity (state residual variability), and reliability (Table 1). The Mplus code for this model is shown in the online supplemental materials.

Multiple source latent state trait analyses

A multiple source LST analysis was applied to study the convergent validity of trait and state residual components of ODD symptom reports across mothers, fathers, primary teachers, and secondary teachers, using mothers as the reference source. Mothers were used as the reference source for two reasons—(1) mothers are the most common source used in ODD studies and (2) mothers probably spend the most amount of time with young children (see also Geiser et al., 2008 for more details on the selection of the reference source). Figure 2 shows this model with Table 1 showing the equations and coefficients for this model.

This model requires trait factor loadings λ_is to be set equal across time within each rater to establish measurement invariance. Occasion residual factor loadings γ_is were also held equal across time for each indicator to establish measurement invariance. Source-specific trait factor loadings were set to 1.0 and the source-specific occasion residual factor loadings δ_is were held equal across time (Geiser et al., 2015). This model does not allow trait factors or source-specific trait factors to correlate with occasion residual factors or source-specific occasion residual factors, or error variables to correlate with any of the latent variables, thereby allowing for a decomposition of observed and true score variance that is additive. This analysis allowed for the calculation of the proportion of shared trait consistency, unique trait consistency, shared occasion-specificity, and unique occasion-specificity for the three comparison sources (fathers, primary teachers, and secondary teachers) relative to mother reports (Table 1). The supplemental materials contain the Mplus code for this model.

Results

Measurement Models

Item Level CFA

Table 3 shows the invariance analyses at the item level for the four sources separately. Each of these four analyses indicated that the strong invariance model (i.e., like-item loadings and like-item thresholds constrained equal across occasions and across the same items on the two ODD constructs) resulted in an excellent fit with no meaningful decrement in fit from the configural model (i.e., none of the four CFI values showed a decrease greater than .007). The invariance analyses on mothers and fathers simultaneously resulted in an excellent fit with no meaningful decrement in fit from the configural model (i.e., decrease in CFI = .004). The same results occurred for the invariance analyses on primary teachers and secondary teachers simultaneously (i.e., decrease in CFI = .006)

Table 3.

Goodness of Fit Statistics for Item Level Invariance Analyses

Invariance level	df	χ2	CFI	TLI	RMSEA (90% CI)
Mothers
Configural	442	895	.982	.980	.036 (.033, .039)
Strong	550	1140	.977	.979	.037 (.033, .040)
Fathers
Configural	442	795	.980	.978	.034 (.030, .038)
Strong	550	988	.975	.978	.034 (.030, .038)
Primary Teachers
Configural	458	635	.996	.995	.022 (.018, .026)
Strong	551	1009	.989	.990	.033 (.029, .036)
Secondary Teachers
Configural	458	599	.995	.995	.020 (.015, .024)
Strong	551	936	.988	.989	.030 (.027, .034)
Mothers and Fathers
Configural	1892	2662	.979	.977	.023 (.021, .025)
Strong	2154	3026	.975	.977	.023 (.021, .024)
Primary and Secondary Teachers
Configural	1924	2206	.994	.994	.014 (.011. .016)
Strong	2157	2772	.988	.989	.019 (.017, .021)

Note. The ratings by mothers and fathers used correlated residuals between the same indicators across occasions while correlated residuals across occasions were not used for primary and secondary teachers because such resulted in a correlated residual greater than 1.00 for teachers. This is the reason for the difference in the degrees of freedom for parents and teachers. Configural invariance = no constraints; Strong invariance = constraints on like-item loadings and thresholds.

These last two invariance analyses yield a single set of eight symptom-factor loadings from the analysis on mothers and fathers as well as a single set of eight symptom-factor loadings from the analysis on primary and secondary teachers. The range of the eight standardized values for mothers/fathers was from .81 to .88 with the range for primary/secondary teachers being from .92 to .99). The rank order of the loadings for each set was the same within rounding error. The rank order of the symptoms was then used to assign the symptoms to three parcels in the manner described earlier (i.e., six items in parcel 1, five items in parcel 2, and five items in parcel 3 with each factor thus defined by three parcels, see Little, 2013). Table 2 shows the items within each parcel.

Parcel Level CFA

Table 4 shows the invariance analyses for the ODD factor defined by parcels. For the single source invariance analyses (e.g., the separate invariance analyses on mothers), there was a single ODD factor at assessment 1 (defined by three parcels) and a single ODD factor at assessment 2 (defined by three parcels). For the invariance analyses with two sources simultaneously, there were two ODD factors at assessment 1 and two ODD factors at assessment 2 (each defined by three parcels). Finally, for the invariance analyses with four sources simultaneously, there were four ODD factors at assessment 1 and four ODD factors at assessment 2 (each again defined by three parcels). The strong invariance model for each of these seven analyses yielded an excellent fit with no meaningful decrement in the CFI value (i.e., range of decrements from .000 to .007, see Table 4). The most important of these seven invariance analyses was for the four sources simultaneously. In this analysis, like-parcel loadings and like-parcel intercepts were invariant across the two occasions and the four sources.

Table 4.

Goodness of Fit Statistics for Parcel Level Invariance Analyses

Invariance level	df	χ2	CFI	TLI	RMSEA (90% CI)
Mothers
Configural	5	10.35ns	0.998	0.994	.037 (.000, .069)
Weak	7	9.31ns	0.999	0.998	.020 (.000, .051)
Strong	9	11.17ns	0.999	0.999	.017 (.000, .046)
Fathers
Configural	5	1.71ns	1.000	1.004	.000 (.000, .025)
Weak	7	4.22ns	1.000	1.002	.000 (.000, .033)
Strong	9	4.65ns	1.000	1.003	.000 (.000, .023)
Primary Teachers
Configural	5	10.91ns	0.998	0.993	.039 (.003, .070)
Weak	7	23.93*	0.993	0.985	.055 (.032, .081)
Strong	9	30.89*	0.991	0.985	.056 (.035, .078)
Secondary Teachers
Configural	5	6.24ns	0.999	0.998	.018 (.000, .056)
Weak	7	4.16ns	1.000	1.004	.000 (.000, .031)
Strong	9	9.64ns	1.000	0.999	.010 (.000, .043)
Mothers and Fathers
Configural	42	295*	0.957	0.933	.087 (.078, .096)
Weak	49	286*	0.960	0.946	.078 (.069, .087)
Strong	56	305*	0.958	0.951	.075 (.066, .083)
Primary Teachers and Secondary Teachers
Configural	42	98*	0.986	0.978	.041 (.031, .052)
Weak	49	109*	0.985	0.980	.040 (.030, .049)
Strong	56	126*	0.983	0.980	.040 (.031, .049)
Mothers, Fathers, Primary Teachers, and Secondary Teachers
Configural	212	571*	0.970	0.961	.046 (.041, .051)
Weak	229	571*	0.972	0.966	.043 (.039, .047)
Strong	246	657*	0.966	0.962	.045 (.041, .050)

Note. Correlated residuals were used for the same indicators across time for all sources. Configural invariance = no constraints; weak invariance = constraints on like-item loadings; Strong invariance = constraints on like-item loadings and intercepts;

^ns= non-significant;

^*p < .05.

Latent Factor Means

None of the four sources showed a significant change in the latent factor ODD means from assessment 1 to 2 (all ps > .29). The finding of strong measurement invariance across time in conjunction with stable mean levels (i.e., no overall trait change) indicated that an LST analysis was appropriate for the present data (see Geiser et al., 2015).

Factor Correlations

Table 5 shows the within-setting (within home and school), across-setting (between home and school), and across-occasion (assessments 1 to 2) factor correlations between ODD factors from the traditional confirmatory factor analysis (with no separation of trait and state residual variance). For two sources within the same setting and occasion, the factor correlations ranged from .70 to .75. For the same sources across the two occasions, the values ranged from .60 to .64. The values were only slightly lower—.51 to .58—for different sources within the same setting across occasions. However, across settings (home to school) within the same occasion (e.g., occasion 1) the values were much lower (i.e., .14 to .33) with the values being similarly low (i.e., .16 to .31) across occasions for different settings.

Table 5.

Factor Correlations for Oppositional Defiant Disorder Factor across Sources and Occasions

		First Grade				Second Grade
		M	F	PT	ST	M	F	PT	ST
First Grade	Mothers	---
	Fathers	.74	---
	Primary Teachers	.25	.24	---
	Secondary Teachers	.17	.14	.72	---
Second Grade	Mothers	.64	.55	.30	.31	---
	Fathers	.51	.62	.29	.29	.75	---
	Primary Teachers	.23	.22	.64	.57	.33	.30	---
	Secondary Teachers	.18	.16	.58	.60	.28	.24	.70

Note. Bold values are within occasion and across occasion correlations for sources within the same setting (home or school). Non-bold values are the across setting within occasion and the across setting across occasion correlations. All correlations significant at p < .05. M = mothers; F = fathers; PT = primary teachers; ST = secondary teachers.

Although these results indicated much higher levels of stability across time for sources within the same setting relative to the stability for sources in different settings across time, these analyses do not quantify how much of the true score variance within each source reflected trait (consistency) versus state residual (occasion-specific) variance and to what extent the trait and state residual components were shared across sources (i.e., convergent validity for the trait and state residuals components). LST models allow a much more sophisticated analysis of stability across time and convergent validity for sources within the same and different settings due to the separation of true score variance into the trait and state residual components. We now describe the results from the single and multiple source LST analyses.

Single Source Latent State-Trait Analyses

Table 6 shows the fit indices for the single source LST models for mothers, fathers, primary teachers, and secondary teachers. Each of these four analyses resulted in an excellent fit. Table 7 shows the estimated consistency, occasion-specificity, and reliability coefficients for each single source model. Results indicated that ODD symptom ratings for all four sources contained more trait variance than occasion-specific (state residual) variance. Specifically, mothers’ ratings contained the highest level of trait variance (M = 67% [proportion of true score variance that was trait variance]) relative to occasion-specific variance (M = 33%). Fathers’ ratings reflected similarly high levels of trait variance (M = 65%) relative to occasion-specific variance (M = 35%). The mean stable trait variance of primary and secondary teachers’ ratings was 62% and 59%, respectively with a mean occasion-specific variance of 38% and 41%, respectively. ODD thus reflected a slightly more trait- than state-like construct across the two occasions of measurement for each source, although slightly more than one third of the true score variance reflected fluctuating state residual variance for each source.

Table 6.

Goodness of Fit Statistics for Latent State Trait Analyses

Source	df	χ2	CFI	TLI	RMSEA (90% CI)
Single Source Latent State Trait Analyses
Mothers	8	6.43ns	1.000	1.001	.000 (.000, .036)
Fathers	8	4.79ns	1.000	1.002	.000 (.000, .030)
Primary Teachers	8	25.91*	.993	.986	.053 (.031, .077)
Secondary Teachers	8	8.99ns	.999	.999	.013 (.000, .046)
Multiple Source Latent State Trait Analysis
Four Sources	216	373.26*	.987	.983	.030 (.025, .035)

Note. Mothers were the reference source in the multiple source latent state trait analysis.

^ns= non-significant;

^*p < .05.

Table 7.

Average Consistency, Occasion-Specificity, and Reliability Estimates from Single Source Latent State-Trait Analyses

Source	Consistency	Occasion-Specificity	Reliability
Mothers	.67 (.59, .77)	.33 (.23, .41)	.91 (.86, .93)
Fathers	.65 (.60, .70)	.35 (.30, .40)	.89 (.87, .91)
Primary Teachers	.62 (.59, .66)	.38 (.34, .41)	.94 (.92, .97)
Secondary Teachers	.59 (.48, .70)	.41 (.30, .52)	.93 (.90, .97)

Note. Values indicate the average across parcels with the range in parentheses.

Multiple Source Latent State-Trait Analyses

Table 6 shows the fit values for the multiple source LST model. This model resulted in an excellent fit. Table 8 shows the shared trait consistency, unique trait consistency, shared occasion-specificity, and unique occasion-specificity of the four different sources. Consistent with the single-source analyses, mothers (which served as the reference source of the model), showed higher trait consistency (M = 61%) than occasion-specificity (M = 39%). In regards to the trait-like variance components of the non-reference sources (fathers, primary teachers, and secondary teachers, i.e., shared consistency and unique consistency), fathers showed high shared trait consistency with mother reports (M = 68%) and correspondingly low unique trait consistency (M = 32%). This indicated strong convergent validity of trait components within the home settings as rated by mothers and fathers. Conversely, primary and secondary teachers showed low shared trait consistency with mother reports (M = 31% and M = 32%, respectively) and correspondingly high unique trait consistency (M = 69% and M = 68%, respectively). This indicated rather modest (or low) convergent validity of trait components across the home and school settings as rated by mothers versus teachers.

Table 8.

Average Shared Consistency, Unique Consistency, Shared Occasion-Specificity, and Unique Occasion-Specificity for a Multisource Latent State Trait Analysis on the Oppositional Defiant Disorder Symptoms

Source	Shared Trait Consistency	Unique Trait Consistency	Shared Occasion- Specificity	Unique Occasion- Specificity
Mothers	.62 (.52, .74)	------	.38 (.26, .48)	------
Fathers	.68 (.65, .70)	.32 (.30, .35)	.35 (.32, .40)	.65 (.60, .68)
Primary Teachers	.31 (.29, .34)	.69 (.66, .71)	.01 (.00, .03)	.99 (.97, 1.00)
Secondary Teachers	.32 (.28, .37)	.68 (.63, .72)	.03 (.00, .09)	.97 (.91, 1.00)

Note. Values indicate the average across parcels with the range in parentheses. Mothers were used as the reference source. Unique source consistency and unique occasion-specificity values were not available for mothers’ ratings because mothers were the reference source.

Regarding the occasion-specific variance components (i.e., shared occasion-specificity and unique occasion-specificity), fathers, primary teachers, and secondary teachers all showed high average unique occasion-specificity (M = 65%, M = 99%, M = .97%, respectively) relative to the average shared occasion-specificity (M = 35%, M = 1%, M = 3%, respectively). Primary and secondary teachers thus shared essentially zero occasion-specific variance with mothers.

An inspection of the residual trait factor correlations among fathers, primary teachers, and secondary teachers provided additional information on whether the trait-like aspects of ODD symptom reports show convergent validity within and across settings. These correlations represent relationships among the residual trait scores pertaining to the non-reference sources after the trait variance that the non-reference sources share with mother reports (the reference source) has been partialled out. In other words, the residual trait correlations indicate whether there is residual stability in father and teacher reports that these reports do not share with mothers but do share with each other.

The residual trait factor correlations between primary and secondary teachers were large and ranged from .86 to .91 (ps < .001), indicating that teachers shared a large proportion of the residual trait variance that they did not share with mother reports. Fathers, however, shared essentially zero residual trait variance with primary and secondary teachers (rs from .00 to .12, ps > .25). These results further indicated that the trait components of ODD symptom reports showed strong convergent validity within settings (here: between teachers), but only low convergent validity across settings (here: between father and teachers).

The state residual factor correlations for fathers, primary teachers, and secondary teachers represent the relationships among occasion residual factor scores pertaining to the non-reference sources after the occasion-specific variance that the non-reference sources share with mother reports (the reference source) has been partialled out. The state residual factor correlations between teachers within occasions 1 and 2 were significant yet small (occasion 1: r = .30, p < .05; occasion 2: r = .43, p < .05) with the state residual factor correlations for fathers with primary and secondary teachers within occasions 1 and 2 being essentially zero and non-significant. Thus, while primary and secondary teachers shared a small amount of residual state variance with each other at each occasion, fathers shared no state residual variance with teachers.

Discussion

The first objective was to determine the amount trait and state residual variance in ODD symptom ratings for mothers, fathers, primary teachers, and secondary teachers separately. Each source showed more trait (Ms = 59% to 67%) than state residual (Ms = 33% to 41%) variance. Given this consistency across the four sources, these findings suggest that the ODD construct is more a stable trait than a fluctuating state within each source for at least the 12-month interval for these children.

Although ODD symptom reports were more trait- than state-like, there was still a substantial amount of fluctuating state residual variance (33% to 41%). The substantial amount of state residual variance for each of the four sources indicates that research on ODD should include at least two occasions of measurement to allow for the separation of true score variance into trait and state residual variance. The separation of trait and state residual variance is important because one can then determine if the trait and state residual components have different external correlates. Such research would further improve our understanding of ODD, especially if trait and state residual components have different predictors and outcomes.

The second objective was to determine the amount of trait and state residual variance in ODD symptoms that fathers, primary teachers, and secondary teachers either shared or did not share with mothers and/or with each other. This question deals with the convergent validity versus source-specificity of the trait and state residual components within and across settings. Fathers shared a large amount of trait variance (M = 68%) with mothers, whereas primary and secondary teachers shared a much smaller amount of trait variance (M = 32%) with mothers (i.e., the average unique trait variance for primary and secondary teachers was thus 68%). These results, in conjunction with fathers sharing zero residual trait variance with primary and secondary teachers while primary and secondary teachers shared 74% to 83% residual trait variance with each other, indicate that the trait components of ODD symptom reports showed good convergent validity within each setting, but only modest convergent validity across settings.

The findings from the single and multiple source LST analyses considered together support this conclusion. That is, while the ODD symptom ratings reflected slightly more trait- than state-like constructs within each of the four sources separately across occasions, strong convergent validity for the trait variance only occurred within settings (i.e., mothers with fathers; primary with secondary teachers) with the convergent validity of the trait variance across settings being modest (at best) and non-existent for the state residual variance.

Trait and State Variance in ADHD and ODD Symptom Ratings

The amount of trait variance in the ODD symptoms (Ms = 59% to 67%) in this study was almost as strong as the trait variance in ADHD-IN and ADHD-HI symptoms (Ms = 54% to 78%) in Litson et al.’s (in press) earlier study. However, convergent validity of fathers with mothers for ODD symptoms (M = 68%) was slightly smaller than convergent validity of fathers with mothers for ADHD-IN symptoms (M = 79%) and ADHD-HI (M = 80%). Finally, the convergent validity of primary and secondary teachers with mothers for ODD symptoms (M = 31% and M = 32%, respectively) was less than convergent validity of primary and secondary teachers with mothers for ADHD-IN (M = 46% and M = 55%, respectively) and ADHD-HI symptoms (M = 42% and M = 52%, respectively). The convergent validity of the trait variance for ADHD-IN and HI symptoms was thus stronger than for ODD symptoms, thus ADHD-IN and HI symptoms appear more consistent across sources and settings than ODD symptoms.

These results for ADHD and ODD symptoms support one of the diagnostic distinctions between ADHD and ODD that is, ADHD symptoms must occur in two or more settings while ODD symptoms need only occur in one setting for a diagnosis. This different diagnostic criterion is consistent with ODD showing less convergent validity across settings than ADHD. However, given that ODD still contained more trait than state residual variance for the four sources, over time the convergent validity for ODD might increase across settings without intervention.

Research and Clinical Implications

In terms of research implications, the single source LST model allows identification of external correlates of trait and state residual variance (Do the trait and state residual aspects of ODD have unique correlates?), while the multiple source LST model allows identification of external correlates of shared and non-shared trait and state residual variance components (Are the predictors of shared trait and state residual variance components different from the predictors of unique trait and state residual variance components?). The multiple source LST model provides a unique approach to determine the external correlates of agreements and discrepancies in symptom ratings (De Los Reyes et al., 2015).

In terms of clinical implications, our findings further confirm the importance of using multiple sources within multiple settings (Kimonis et al., 2014). Our findings also suggest the potential importance of multiple occasions of assessment for the evaluation of an individual child (e.g., the multiple sources completing the ODD measure twice over a two-week interval). Although such a multiple occasion assessment within a short interval would require special instructions (rate the symptoms for the past two weeks), such might provide useful information about symptom stability and variability (Were symptom reports mostly stable or variable over the interval for the child for the various sources?)

Limitations

It is impossible to know how much discrepancy between home and school is due to informant discrepancies (e.g., parents and teachers have different biases) or setting-specific effects (behavior specific to the settings). Future research should attempt to address this issue by using the same informants in different settings although we realize that the use of the same trained raters for the two settings has its own complexities. Future research should also attempt to replicate the findings across longer periods of time and more occasions with more diverse samples. Perhaps most importantly, our sample was a nonclinical sample, and similar findings in a clinical sample would strengthen the results.

Conclusion

Single source and multiple source LST models were used to determine the amount of trait and state variance in ODD symptom reports. Although the ODD construct was more trait-like than state-like for ratings by mothers, fathers, primary teachers, and secondary teachers considered separately (i.e., each individual source contained more trait than state variance), there was still a substantial amount of state residual variance in the ODD symptoms. In addition, the multiple source LST analysis indicated that the ODD construct was trait-like within settings and not strongly trait-like across settings. The assessment of ODD should therefore include multiple occasions and multiple sources within multiple settings.