Observational Assessment of Preschool Disruptive Behavior, Part II: Validity of the Disruptive Behavior Diagnostic Observation Schedule (DB-DOS)

Abstract

Objective:

To examine the validity of the Disruptive Behavior Diagnostic Observation Schedule (DB-DOS), a new observational method for assessing preschool disruptive behavior.

Method:

A total of 327 behaviorally heterogeneous preschoolers from low-income environments comprised the validation sample. Parent and teacher reports were used to identify children with clinically significant disruptive behavior. The DB-DOS assessed observed disruptive behavior in two domains, problems in Behavioral Regulation and Anger Modulation, across three interactional contexts: Examiner Engaged, Examiner Busy, and Parent. Convergent and divergent validity of the DB-DOS were tested in relation to parent and teacher reports and independently observed behavior. Clinical validity was tested in terms of criterion and incremental validity of the DB-DOS for discriminating disruptive behavior status and impairment, concurrently and longitudinally.

Results:

DB-DOS scores were significantly associated with reported and independently observed behavior in a theoretically meaningful fashion. Scores from both DB-DOS domains and each of the three DB-DOS contexts contributed uniquely to discrimination of disruptive behavior status, concurrently and predictively. Observed behavior on the DB-DOS also contributed incrementally to prediction of impairment over time, beyond variance explained by meeting DSM-IV disruptive behavior disorder symptom criteria based on parent/teacher report.

Conclusions:

The multidomain, multicontext approach of the DB-DOS is a valid method for direct assessment of preschool disruptive behavior. This approach shows promise for enhancing accurate identification of clinically significant disruptive behavior in young children and for characterizing subtypes in a manner that can directly inform etiological and intervention research.

This is the second of two articles introducing a new diagnostic observation method for assessing disruptive behavior in preschool children, the Disruptive Behavior Diagnostic Observation Schedule (DB-DOS). The goal of the DB-DOS is to provide a direct observation method for assessing preschool disruptive behavior that can serve as a companion tool to parent interviews. Drawing on traditions of both diagnostic observation¹ and developmental research,² critical features of the DB-DOS are behavioral presses that elicit a range of clinically salient behaviors, a coding system that rates these behaviors along a clinical continuum, and a structure that enables the assessment of these behaviors across interactional contexts involving both parent and examiner. In Part I, we demonstrated the reliability of the DB-DOS, including an excellent fit of our hypothesized model. In Part II, we examine the convergent and divergent validity of the DB-DOS. To test the clinical validity of the DB-DOS, we examine its criterion and incremental validity for discriminating disruptive behavior status and impairment, concurrently and predictively.

Numerous developmental studies have demonstrated that preschool behavior problems are measurable, moderately stable, and heritable.^3–6 More recently, clinically focused studies have shown that DSM-IV disruptive behavior disorder (DBD) diagnostic criteria (for oppositional defiant disorder [ODD] and conduct disorder [CD]) discriminate preschoolers with disruptive behavior in both clinic and community samples.^7–10

As consensus is gained that DBDs are detectable in early childhood, there is also increasing recognition that the current DBD nosology may be misspecified for preschoolers.^11,12 Careful analysis of the DSM-IV DBD nosology reveals three key areas of potential misspecification during the preschool period¹³:

1. Approximately 25% of the 15 CD symptoms are developmentally impossible (e.g., truancy), resulting in a restricted item pool.

2. Another 25% of CD symptoms are developmentally improbable (e.g., firesetting). Although preschoolers may be physically capable of these actions, they rarely perform these extreme behaviors. The developmental impossibility and improbability of a substantial proportion of the CD symptoms may reduce sensitivity by increasing the risk of false negatives; in other words, young children with milder but clinically significant disruptive behavior may go undetected.

3. The remaining symptoms for CD and many ODD symptoms are developmentally imprecise because they are worded in a way that entails substantial overlap with the normative misbehaviors of early childhood (e.g., often defies) and are without developmentally defined anchors. There is no specification of how frequently these common misbehaviors must occur to be considered symptoms nor do symptom definitions incorporate behavioral features beyond frequency that are likely to sharpen clinical distinctions.¹¹ The problem of developmental imprecision may reduce specificity because there are not empirically derived parameters to distinguish normative misbehaviors from clinically significant patterns of disruptive behavior during this developmental period.

Thus, although the evidence that DSM-IV DBD criteria can be applied to preschool children is consistent and represents important strides in demonstrating the validity of preschool psychopathology, further research is needed to more precisely characterize the nature of disruptive behavior in preschool children. In particular, little is known about the specific behaviors that best discriminate clinically significant disruptive behavior from normative misbehavior in the preschool years and about individual differences in patterns of young children’s disruptive behavior across contexts and the implications of these patterns for etiology and intervention.

In order to address these gaps, developmentally and theoretically derived assessment tools are needed.² To this end, the DB-DOS was designed to generate a more nuanced method of assessing a spectrum of disruptive behavior in preschoolers via incorporation of information from developmental research on typical and atypical manifestations of behavior during early childhood.^3,14 As detailed in Part I, behaviors were assessed on the DB-DOS within two broad domains conceptualized as the prototypical elements of disruptive behavior in young children: problems in Behavioral Regulation and problems in Anger Modulation.

In this article, we draw on multimethod, multi-informant longitudinal assessments of preschool disruptive behavior to do the following:

1. Examine the convergent and divergent validity of the DB-DOS by establishing associations with maternal and teacher reports of child behavior and independently observed behavior during developmental testing.

2. Test the clinical validity of the DB-DOS concurrently and at 1 year follow-up in terms of criterion validity for discriminating preschoolers with and without clinically significant disruptive behavior and incremental utility for prediction of impairment, beyond variance accounted for by meeting DSM-IV DBD symptom criteria based on parent/teacher report.

METHOD

Participants

Three hundred thirty-six mother-child dyads were recruited from clinics affiliated with two Midwestern universities serving urban, disadvantaged populations for a study of preschool disruptive behavior in children from low-income environments (for details, see Part I). Forty percent (n = 134) of the participants were referred to a preschool behavior problems clinic. The remaining preschoolers were recruited from affiliated pediatrics and family practice clinics: 30% (n = 102) were nonreferred children whose parents reported that they or other caregivers had expressed concerns about the child’s behavior and 30% (n = 100) were nonreferred children without behavioral concerns. Inclusion criteria were child age between 3 and 5 years, residence with biological mother, and attendance in preschool or out-of-home day care at least 3 hours per day three times per week. Children were excluded if they had a serious developmental disability. In addition to the one child excluded from the reliability analyses because he did not complete a DB-DOS (Part I), eight additional children were excluded from these validity analyses because they did not have full data across the three DB-DOS interactional contexts. The resultant analytic sample is 327. Eighty-four percent of the children were African American and 45% were female. Child age was evenly distributed across 3, 4, and 5 years (mean 4.48, SD 0.85).

Reliability of the DB-DOS was assessed at baseline and is briefly described below (for details, see Part I). For the present validity analyses, we also include data from the 1-year follow-up.

Measures

Disruptive behavior symptoms were assessed by parent report on a semistructured interview conducted by trained nonclinician research assistants (RAs) blind to child behavioral status and teacher report on a clinical symptom checklist.

Diagnostic Interview.

Mothers were administered the Kiddie Disruptive Behavior Disorders Schedule (K-DBDS), a semistructured clinical interview assessing DSM-IV DBD symptoms in preschool children,⁷ which includes probes to assess the frequency, severity, and pervasiveness of each symptom across school, home, and public settings. The K-DBDS has two scoring methods: a developmentally modified approach and a developmentally enhanced approach. The developmentally modified approach is designed to adhere as closely as possible to DSM-IV¹⁵; we have previously reported on its reliability and validity.⁷ In this article, we use the developmentally enhanced approach. This approach incorporates the more detailed information available on the KDBDS (e.g., severity, frequency) to determine clinical significance to enhance distinctions between symptoms and normative misbehaviors in preschool children. Clinical judgment was used to generate an algorithm incorporating this additional information for determining thresholds for symptom endorsement. A DBD diagnosis was made based on these symptom thresholds. The DBD group included children who met criteria for ODD (four or more symptoms), CD (three or more symptoms) or DBD-NOS (defined a priori as three or more disruptive symptoms). Using the developmentally enhanced approach, 1-week test-retest reliability was high for a DBD diagnosis (κ = .81, p < .001) and total number of DBD symptoms (intraclass correlation coefficient [ICC] = 0.82, p < .001).

Symptom Checklist.

Teachers reported on child disruptive behavior symptoms with the Early Child Symptom Inventory, a DSM-based checklist.¹⁶ The Early Child Symptom Inventory generates categorical and continuous symptom scores. Scores have demonstrated good test-retest reliability (r = 0.56 for ODD and 0.41 for CD) and differentiate clinic-referred from nonreferred children.¹⁷ Teacher data were obtained on 88% of the preschoolers at both baseline and 1-year follow-up.

Impairment.

To provide a global assessment of functioning, mothers and teachers independently completed the nonclinician version of the Children’s Global Assessment Scale (CGAS),¹⁸ which has been used in multiple studies of preschool DBDs.^19,20 The CGAS is scored from 0 to 100, with lower scores indicating greater impairment. Standard CGAS clinical cut points scores (≤60) were used to define impairment. Parent- and teacher-reported DBD symptoms and their CGAS rating correlated highly (r = −0.70 and −0.65 for parent and teacher, respectively, p < .001). For determining disruptive behavior status (below), parent and teacher CGAS scores were used. As an additional measure of impairment for the convergent validity analyses, mothers also completed the Impact on the Family Scale.²¹ The Impact on the Family Scale assesses social, financial, and personal burden resulting from child behavior problems. A single factor score is generated with high internal consistency and strong construct validity, including validity in ethnically diverse low-income samples.²²

Child Disruptive Behavior Status.

This was determined using combined parent and teacher reports of child disruptive behavior symptoms and CGAS impairment at baseline and at 1-year follow-up. (Parent report data were used to determine disruptive behavior status for the 12% of children missing teacher data; parental report on the K-DBDS also includes information on the child’s behavior at school.) Parent- and teacher-reported symptoms and CGAS scores were significantly correlated at both time points (symptom r ranging from 0.20 to 0.31; CGAS r ranging from 0.32 to 0.44). Preschoolers were classified as having clinically significant disruptive behavior if they met DBD symptom criteria (i.e., ODD, CD, or DBD NOS) and had a CGAS score ≤60 from the same informant. The disruptive group had significantly higher levels of disruptive symptoms (mean 5.6, SD 2.5) compared to the nondisruptive group (mean 1.0, SD 1.3) (t₈₅ = 14.8, p < .001). Twenty-three percent of preschoolers were classified as disruptive at baseline.

DB-DOS.

As described in detail in Part I, the DB-DOS is a 50-minute structured laboratory observation that is divided into three interactional contexts: one parent context and two examiner contexts. Behaviors on the DB-DOS are coded using an ordinal rating approach operationalized along a clinical continuum (0 = normative behavior,1 = normative misbehavior, 2 = behavior of concern, 3 = atypical behavior). Behaviors are coded within the domains of problems in Behavioral Regulation and problems in Anger Modulation. This yields six DB-DOS scores (two domains × three contexts). As detailed in Part I, the DB-DOS has demonstrated good reliability including inter- rater (mean weighted κ = .66 for Behavioral Regulation and .62 for Anger Modulation), and test-retest (mean ICC 0.70 for Behavioral Regulation and 0.77 for Anger Modulation) reliability. Internal consistency of the DB-DOS domains was also high (mean Cronbach α = .85 for Behavioral Regulation and .92 for Anger Modulation). An excellent model fit was demonstrated χ² = 2.5, CFI = 1.000, root mean square error approximation = 0.0000), including significant increase in overall model for the DB-DOS multidomain model of disruptive behavior compared to a unidimensional model (robust maximum likelihood χ² = 89.8, p < .001).

Additional Measures of Child Behavior and Functioning

Observed Behavior During Developmental Testing.

Child behavior during the developmental testing was also coded to provide parallel behavioral ratings during the testing situation as observed measures of convergent and divergent validity. Seven DB-DOS codes were slightly modified for use in the testing context (six codes from the Behavioral Regulation domain reflecting a range of noncompliant, resistant, and destructive behaviors and one code from the Anger Modulation domain to assess intensity of anger displays). These were combined to create an index of observed disruptive behavior during testing (α = .77). Observed energetic engagement during testing was also coded as a measure of divergent validity. This variable assessed the child’s eagerness, pleasure, and persistent engagement during testing activities. RAs coded the child’s behavior immediately following the testing:these RAs were different from those who administered the DB-DOS and were blind to child disruptive behavior status. All of the testing sessions were videotaped, and 23% of them were double coded to assess interrater reliability (mean ICC = 0.72).

Questionnaire Ratings of Child Functioning.

Two questionnaire measures were used as indicators of divergent validity. Teacher reports on the summary score of the Social Skills Rating Scale²³ were used to assess children’s social skills at school. This scale has well-established reliability (mean α .94) and validity, including negative associations with preschool disruptive behavior.^23,24 Maternal report on the Falling Reactivity/Soothability Scale of the Children’s Behavior Questionnaire²⁵ was used to assess children’s ability to modulate distress (higher scores indicate more rapid recovery). The Falling Reactivity/Soothability scale has demonstrated good psychometric properties (α = .66, stability = 0.61)²⁵ and is negatively associated with young children’s disruptive behaviors.²⁶

Covariates

Child Developmental Functioning.

The Differential Ability Scales-Preschool version, a well-validated developmental assessment measure for preschoolers, was administered.²⁷ The Differential Ability Scales General Conceptual Ability score (mean 100, SD 15) was used as an index of child cognitive functioning.

Family Risk.

Mothers provided all of the demographic information including child age (3, 4, or 5 years of age), sex (0 = male, 1 = female), ethnicity (African American, non-Hispanic white, or Hispanic), family income, and maternal education. Because of the diversity of family structures within low-income families of color,^28,29 we assessed risk in this domain with two indicators: maternal marital status regardless of whether she was married to the child’s biological father (0 = married, 1 = not married) and low father involvement, defined as the biological father’s lack of regular contact with the child, regardless of whether co-residing or married to the child’s mother (0 = regular father involvement, 1 = lacks regular father contact). High family stress was also assessed via the clinical cutoff (six or more stressors) on the Difficult Life Circumstances Scale,³⁰ (0 = not high stress, 1 = high stress). The Difficult Life Circumstances Scale has demonstrated acceptable reliability (1-year test-retest r = 0.70) and construct validity.^30,31

Parenting Style.

Mothers’ behavior during the DB-DOS Parent context was coded using the Parenting Clinical Observation Schedule (PCOS).^31a Like DB-DOS codes, PCOS codes are global, ordinal ratings assessing parenting behaviors along a continuum from competent to clinically concerning. Parenting behaviors on the PCOS are rated in three domains: Parental Responsive Involvement, Constructive Discipline, and Problematic Discipline. The PCOS has demonstrated good reliability, including interrater agreement (mean ICC 0.84) and internal consistency (mean α .71) as well as validity, with PCOS scores predicting change in child disruptive behavior status over time.^31a Because problematic parenting has been robustly associated with disruptive behavior problems,³² we created a composite problematic parenting style. Twenty-six percent of mothers who exhibited Problematic Discipline and did not demonstrate competence in either the Responsive Involvement or Constructive Discipline domain were characterized as problematic (0 = not problematic, 1 = problematic).

Procedures

Informed consent was obtained from the mother before the laboratory visit. All of the procedures were approved by the institutional review boards at both universities. Before the laboratory visit, mothers were mailed a packet of questionnaires regarding the child’s functioning and their parenting practices. One RA administered the K-DBDS interview to mothers and a second RA conducted the Differential Ability Scales with the child. Mother and child then participated in the DB-DOS. The mothers completed additional questionnaires while the children completed the DB-DOS with the examiner. Questionnaires were also mailed to teachers after maternal consent was obtained. Eighty-eight percent of teachers provided data at both time points. Assessment procedures at this follow-up visit were identical to those at baseline. Ninety-two percent (n = 301) of the analytic sample returned approximately 1 year later (mean 392 days, SD 53 days; range 294–697). Children who did not participate in the 1-year follow-up did not differ significantly from longitudinal participants on DBD status or any of the covariates. Families were paid $60 and $70, respectively, for their participation in the two study visits (with a $10 bonus provided if questionnaires were completed in advance).

Validity analyses were conducted to establish convergent, divergent, criterion, and incremental validity using SPSS 15.0. Convergent validity and divergent validity were assessed by examining the correlations of DB-DOS domain scores and multimethod assessments of child functioning as assessed by parent and teacher report and direct observation. To assess clinical validity, multivariate logistic regression analyses were used to assess the criterion and incremental validity of DB-DOS domain scores, with separate models to test the validity of each domain. Thus, each regression model used the domain within context scores as predictors (e.g., Behavioral Regulation score in Examiner-Engaged, Examiner-Busy, and Parent contexts). To assess criterion validity, logistic regression analyses were used to examine whether DB-DOS domain within context scores predicted child disruptive behavior status concurrently and at 1-year follow-up, and persistence in disruptive behavior across these two time points. Child (age, sex, and cognitive functioning) and family characteristics (observed problematic parenting and the family risk index) were controlled as covariates. Finally, to assess the incremental validity of the DB-DOS scores, we conducted a series of stepwise regressions in which the child’s DBD symptom status by parent/teacher report (i.e., meeting DBD symptom criteria), and these same child and family covariates were entered in the first block and DB-DOS domain by context scores were entered in the second block. The dependent variables for these analyses were impairment at baseline and 1-year follow-up and persistent impairment across these time points.

RESULTS

Disruptive and nondisruptive groups did not differ in terms of ethnicity, income, or maternal education. However, the disruptive group was significantly more likely to have low father involvement (41% versus 25%, χ² = 7.5, p < .006) and high stress (49% versus 30%, χ ² = 9.8, p < .002), with a trend for higher likelihood of mothers being unmarried (85% versus 76%, χ ² = 2.9, p < .10). In light of these group differences, low father involvement, high stress, and being a single mother were combined to form a family risk index (range 0–3), which was controlled in multivariate analyses.

Group Comparisons

Multivariate analyses of variance were used to examine group differences in DB-DOS scores. For these analyses, concurrent disruptive behavior status was the fixed factor and the six DB-DOS scores were the dependent variables (two domains × three contexts). The overall model was significant with a moderate effect size (Wilks λ = .88, F_6,320 = 7.50, partial η² = .12, p < .001). In addition, all of the individual DB-DOS scores significantly distinguished disruptive and nondisruptive groups (Fs ranging from 9.40 to 29.06, all p < .002, partial η² ranging from .03 to .08).

Convergent and Divergent Validity Analyses

A multimethod, multi-informant approach was used to assess convergent and divergent validity. As Table 1 illustrates, correlations were generally in the small to medium range.³³ Small but significant positive correlations were found between DB-DOS scores and parent and teacher reports of symptoms and impairment. Correlations between DB-DOS scores and observed disruptive behavior during developmental testing were moderate. A similar pattern was evident for the divergent analyses. Small but significant negative correlations were found between the majority of DB-DOS scores and parent reports of recovery from distress on the Children’s Behavior Questionnaire and teacher reports of social skills on the Social Skills Rating Scale. Significant negative associations were also evident between observed energetic engagement during testing and DB-DOS scores. These correlations were of small to medium effect size.

Table 1.

Association ^a of Disruptive Behavior Diagnostic Observation Schedule (DB-DOS) Scores by Context and Multi-informant Assessments of Child Disruptive Behavior and Functioning

	DB-DOS Scores
	Behavioral Regulation Domain			Anger Modulation Domain
	Examiner Engaged	Examiner Busy	Parent	Examiner Engaged	Examiner Busy	Parent
Convergent Validity Parent reported
K-DBDS symptoms	0.19†	0.11*	0.24†	0.15**	0.10+	0.21†
CGAS ratingsb	−0.18†	−0.17***	−0.20†	−0.19***	−0.12*	−0.21†
Impact on the family	0.22†	0.15**	0.16**	0.18***	0.14*	0.22†
Teacher reported
ECI symptoms	0.23†	0.20†	0.06	0.15**	0.20†	0.10+
CGAS ratingsb	−0.28†	−0.20†	−0.11+	−0.08	−0.17***	−0.16**
Behavior during testingc
Observed disruptive behavior	0.53†	0.44†	0.30†	0.34†	0.37†	0.22†
Divergent Validity
CBQ Falling Reactivity Scale (parent)	−0.10+	−0.01	−0.10+	−0.16***	−0.11*	−0.17***
SSRS score (teacher)	−0.25†	−0.16**	−0.14*	−0.12*	−0.23***	−0.15**
Observed energetic engagement during testingc	−0.34†	−0.25†	−0.28***	−0.26†	−0.30†	−0.18†

Note: K-DBDS = Kiddie Disruptive Behavior Disorders Schedule; CGAS = Children’s Global Assessment Scale; ECI = Early Child Symptom Inventory; CBQ= Children’s Behavior Questionnaire; SSRS = Social Skills Rating Scale.

^aCorrelations are Pearson r.

^bHigher scores indicate less impairment.

^cDB-DOS ratings and observed disruptive behavior during testing ratings were completed by independent coders.

⁺p < .10

^*p < .05

^**p < .01

^***p < .005

^†p < .001.

Multivariate Analyses

A series of logistic regression analysis was next conducted to test the criterion and incremental validity of the DB-DOS. For these models, DB-DOS problem scores were generated: each DB-DOS item was dichotomized so that scores of 2 or 3 (behaviors of clinical concern) were given a score of 1. Dichotomized items were then summed to generate a problem count for each DB-DOS domain-within-context score. These problem counts were subsequently dichotomized at the upper quartile to create problem scores (0 = not high observed problems, 1 = high observed problems). The first set of models tested the contribution of the three DB-DOS Behavioral Regulation problems scores to prediction of child disruptive behavior status. The second set of models tested the contributions of the DB-DOS Anger Modulation problems scores to disruptive behavior status. In addition to the DB-DOS domain level problem scores, all of the models controlled for child age, sex, and cognitive score, observed problematic parenting, and the family risk index. Tables 2 and 3 present regression statistics for the DB-DOS scores within these models. (Tables 2a and 3a present the full regression models from which these are derived and are available on the Journal’s Web site [ www.jaacap.com ] through the Article Plus feature.)

Table 2.

Logistic Regression Models Testing Criterion Validity of Disruptive Behavior Diagnostic Observation Schedule (DB-DOS) Problem Scores for Predicting Disruptive Behavior Status

DB-DOS Domain	Concurrently Disruptive			Disruptive at 1-Year Follow-Up			Persistently Disruptive
	Model χ2	Wald	OR (95% CI)	Model χ2	Wald	OR (95% CI)	Model χ2	Wald	OR (95% CI)
Problems in Behavioral Regulation	45.16***			46.43***			37.94***
Examiner Engaged		13.71	3.53*** (1.81–6.87)		9.25	3.09** (1.49–6.39)		14.51	5.97*** (2.38–14.96)
Examiner Busy		.20	1.18 (0.58–2.37)		0.19	1.19 (0.55–2.56)		0.00	1.00 (0.38–2.63)
Parent		3.93	1.86* (1.01–3.42)		0.02	1.05 (0.52–2.12)		0.42	1.34 (0.55–3.25)
Problems in Anger Modulation	52.40***			44.43***			29.44***
Examiner Engaged		2.03	1.65 (0.83–3.29)		1.76	1.67 (0.78–3.58)		2.04	1.93 (0.78–4.78)
Examiner Busy		4.40	2.01* (1.05–3.87)		5.06	2.24* (1.11–4.52)		2.11	1.94 (0.79–4.76)
Parent		15.27	3.73*** (1.93–7.22)		0.08	1.11 (0.52–2.36)		1.65	1.83 (0.73–4.58)

Note: The following factors were controlled in all of the models: child age, sex, cognitive functioning; family risk index; and observed problematic parenting. For presentation of full regression models, see Table 2a in the online-only ArticlePlus content for this article (www.jaacap.com ). OR = odds ratio; CI = confidence interval.

⁺p < .10

^*p < .05

^**p < .01

^***p < .001.

Table 3.

Stepwise Logistic Regression Models Testing Incremental Validity of Disruptive Behavior Diagnostic Observation Schedule (DB-DOS) Problem Scores for Predicting Impairment Status

DB-DOS Domain	Concurrent Impairment			Impairment at 1 Year Follow-Up			Persistent Impairment
	Model χ2	Wald	OR (95% CI)	Model χ2	Wald	OR (95% CI)	Model χ2	Wald	OR (95% CI)
Problems in	91.31***			72.58***			74.42***
Behavioral
Regulation
Examiner Engaged		7.54	2.43** (1.29–4.57)		8.07	2.62** (1.35–5.09)		14.71	4.53*** (2.09–9.80)
Examiner Busy			Not retained			Not retained			Not retained
Parent			Not retained			Not retained			Not retained
Problems in Anger   Modulation	94.25***			70.88***			64.08***
Examiner Engaged			Not retained		6.45	2.40* (1.22–4.71)		4.96	2.39* (1.11–5.13)
Examiner Busy			Not retained			Not retained			Not retained
Parent		10.31	2.84*** (1.50–5.37)			Not retained			Not retained

Note: Child disruptive behavior symptom status (three or more symptoms) and the covariates (child age, sex and cognitive functioning; family risk index; and observed problematic parenting) were entered in the first step, with DB-DOS domain within context scores entered in the second step. For presentation of full regression models, see Table 3a in the online-only Article Plus content for this article (www.jaacap.com). OR = odds ratio; CI = confidence interval.

Criterion Validity Analyses.

For the series of logistic regression analyses testing whether the DB-DOS discriminated children with clinically significant disruptive behavior, the dependent variables were a set of outcomes reflecting disruptive behavior status (disruptive = meets symptom criteria and is impaired) concurrently disruptive (23% of sample), disruptive at 1-year follow-up (17% of sample), and persistently disruptive across the two time points (10% of sample, 55% of those disruptive at baseline).

Problems in Behavioral Regulation in the Examiner-Engaged context strongly and consistently discriminated disruptive and nondisruptive groups (Table 2). Children with problems in Behavioral Regulation in the Examiner-Engaged context were approximately three times more likely to be classified as disruptive concurrently and longitudinally. Furthermore, the odds of being in the persistently disruptive group were almost six times greater for children with problems in Behavioral Regulation in the Examiner-Engaged context. Problems in Behavioral Regulation in the Parent context also contributed to concurrent discrimination of the disruptive group.

The pattern of discrimination for problems in Anger Modulation was more varied. Problems in Anger Modulation in the Examiner-Busy and Parent contexts significantly discriminated the disruptive group concurrently, whereas problems in Anger Modulation in the Examiner-Busy context more than doubled the risk of being in the disruptive group 1 year later. Problems in Anger Modulation did not discriminate the persistently disruptive group.

Incremental Validity.

The next set of analyses used stepwise logistic regression analyses (forward conditional) to test the incremental validity of DB-DOS problem scores for predicting impairment status, beyond variance explained by meeting DSM-IV DBD symptom criteria (Table 3). The dependent variables for these analyses were a set of outcomes reflecting impairment status: concurrently impaired (35% of sample), impaired at 1-year follow-up (31% of sample), and persistently impaired across the two time points (18% of sample, 53% of those impaired at baseline). Block 1 included the covariates described above plus a dichotomized variable that indicated whether the child met DBD symptom criteria based on parent/teacher report (0 = fewer than three disruptive symptoms, 1 = three or more disruptive symptoms). Because models were conducted separately for each of the two domains, block 2 was composed of the three DB-DOS problem scores within context for each domain (i.e., Behavioral Regulation and Anger Modulation, respectively).

As Table 3 illustrates, the pattern of incremental prediction for problems in Behavioral Regulation was similar to that in the discriminant analyses. Problems in Behavioral Regulation in the Examiner-Engaged context contributed incremental variance in predicting impairment in all of the models. Children with problems in Behavioral Regulation in this context were more than twice as likely to be impaired concurrently and longitudinally. Furthermore, problems in Behavioral Regulation in the Examiner-Engaged context more than quadrupled the odds of being persistently impaired, beyond the variance explained by meeting DSM-IV DBD symptom criteria based on parent/teacher report.

The more varied pattern for problems in Anger Modulation found in the discriminant analyses was also evident in the incremental validity analyses. Problems in Anger Modulation in the Parent context added incremental utility to concurrent discrimination of impairment. In contrast, problems in Anger Modulation in the Examiner-Engaged context more than doubled the risk of impairment 1 year later and of persistent impairment, beyond variance explained by meeting DSM-IV DBD symptom criteria.

DISCUSSION

In Part I, we demonstrated the reliability of the DB-DOS. In Part II, we demonstrate the validity of the DB-DOS in terms of its association to related and divergent constructs and its clinical utility for discriminating disruptive behavior and impairment in preschool children.

In demonstrating convergent/divergent via multi-informant, multimethod assessments, we provide evidence that the theoretically and clinically derived domains of the DB-DOS tap into hypothesized constructs of interest. The relatively modest size of the correlations between parent and teacher reports of child disruptive behavior and impairment and observed behavior on the DB-DOS is consistent with findings of relatively low cross-informant convergence across multiple samples and studies.³⁴ Of particular note, the relatively low correspondence of reports from different informants in these previous studies is found even when child behavior is assessed using the same measure, and associations are typically even smaller when both informants and methods vary. For example, one recent study of preschoolers that examined the association of parent, teacher, and examiner ratings of preschoolers’ disruptive behavior using the same measure also found modest correlations between examiner and teacher ratings but no association between examiner and parent ratings.³⁵ Furthermore, whereas parent and teacher ratings of DBD symptoms are global ratings of behavior and do not distinguish between behaviors that occur in different contexts, the DB-DOS scores are more specific in that they are based on behavior in a specific interactional context. In light of these methodological issues, the significant pattern of association between DB-DOS domain within context scores and both parent and teacher ratings is especially noteworthy.

The present findings demonstrate strong evidence of the criterion validity of the DB-DOS for distinguishing disruptive behavior status. This includes establishing the capacity of the DB-DOS to discriminate children with concurrently assessed disruptive behavior, as classified using multi-informant, developmentally sensitive methods and with co-occurring risks controlled. These findings suggest that observations on the DB-DOS capture meaningful distinctions between normative misbehavior and behaviors of clinical concern. The demonstration of the predictive validity of the DB-DOS for predicting child disruptive behavior over time importantly extends previous work on preschool DBDs, which has been predominantly cross-sectional (for exceptions, see Kim-Cohen et al.,⁸ Lavigne et al.,⁹ and Speltz et al.¹⁰). Finally, perhaps the most rigorous evidence of validity is the demonstration of the incremental utility of the DB-DOS for predicting impairment over time, above and beyond variance explained by meeting DSM-IV DBD symptom criteria based on parent/teacher report. For example, in the incremental validity analyses, observed problems on the DB-DOS increased the risk of impairment two- to fourfold, over and above the effects of disruptive behavior and key risk factors. This suggests that direct observation of young children’s problems in Anger Modulation and Behavioral Regulation across interactional contexts provides unique information for clinical assessment of preschool disruptive behavior.

How can these findings from the DB-DOS inform our understanding of the phenomenology of disruptive behavior in young children and its identification? First, they suggest that direct assessment of the quality of behaviors via diagnostic observation can enhance identification of impairing disruptive behavior in preschool children beyond that which can be achieved by simply using reported symptoms. Second, substantial variation was evident in the clinical utility of patterns across domains and contexts (with consistent prediction by problems in Behavioral Regulation in the Examiner-Engaged context and a more varied pattern of prediction from problems in Anger Modulation across the three DB-DOS contexts). These diverse patterns underscore the heterogeneity of disruptive behavior and highlight the potential utility of assessing young children’s disruptive behaviors across multiple domains and within interactional context. These individual differences in patterns of behavior also suggest the possibility of meaningful subtypes of preschool disruptive behavior. For example, failure to regulate behavior in keeping with social rules and norms may have different genetic and neural substrates than anger dyscontrol. In turn, problems in each of these domains of behavior may interact differentially with parenting processes to influence the likelihood of persistence or desistance over time.

Despite a long-standing tradition of using observation in the clinical assessment of young children, it has been challenging to disentangle the unique contributions of parent and child to disruptive behavior problems because existing observational paradigms have generally assessed child behavior only within the context of parent—child interaction. Because the DB-DOS directly assesses clinical patterns of child disruptive behavior with and without the parent and also directly assesses parenting behavior, it provides a standardized observational tool that may be useful in future investigations examining the prognostic significance of variations in child disruptive behavior across interactional contexts.

Generating a diagnostic algorithm for the DB-DOS is a critical next step for establishing its utility as a diagnostic tool for direct assessment of young children. In addition, establishing standardized procedures for combining clinical information across interview and diagnostic observation methods is important for generating systematic approaches for integrating multimethod data in the assessment of preschool psychopathology.³⁶ Diagnostic interviews, questionnaires, and diagnostic observations provide overlapping but unique information that is essential for capturing the heterogeneity of clinical manifestations of early-onset disorders and for distinguishing normative variation from clinical problems in this developmental period. For example, in identifying autism in young children, the combined use of both interview and diagnostic observation significantly reduces the number of false positives.³⁷ Even when using identical methods across informants, agreement between multiple informants is often low.³⁸ Future steps must also include demonstration of the validity of the DB-DOS with diverse samples. In Part I, we noted the need for replication with a more representative sample to ensure the DB-DOS’ validity across various ethnic and socioeconomic subgroups. This should also include establishing the validity of the DB-DOS with primary caregivers besides mothers (e.g., fathers). Explicating the clinical meaning and longitudinal predictive value of cross-domain, cross-context patterns on the DB-DOS will also require large samples that provide sufficient power to characterize this heterogeneity.

Developmentally informed specification of the major features of disruptive behavior in preschool children is a major step toward accurate identification, specifying meaningful disruptive behavior domains and subtypes, testing whether these subtypes have distinct etiologies, and determining the prognostic meaning of these patterns and their implications for therapeutic interventions. The DB-DOS shows substantial promise for contributing to this line of investigation by providing a nuanced characterization of the developmental phenomenology of disruptive behavior in young children. It also has the potential for providing a standardized method for clinicians to systematically integrate their own observations into clinical decision making. This may prove particularly useful for both diagnosis and treatment planning during a developmental period in which determining clinical significance is especially challenging.