Psychometric Properties of the Dominic Interactive Assessment

Abstract

The reliability and validity of the Dominic Interactive (DI) assessment were evaluated. The DI is a computerized, self-report measure for children, which assesses symptom presence for seven DSM-IV diagnoses. The participants were 322 children (169 cocaine exposed, 153 not cocaine exposed) who were recruited at birth for a prospective longitudinal study. At 6 years of age, measures of self-report, parent report, and observational data were collected. Moderate to excellent internal consistencies on the DI were found for the total sample as well as for cocaine status and gender. Concurrent validity correlations between DI scales and the Child Behavior Checklist, Affect in Play Scale, a modified Conners’s Teachers Rating Scale, and the Parenting Stress Index were examined. Significant relationships among scales were more apt to be among comparisons that assessed externalizing behaviors. Overall, low correlations were obtained, which are comparable to other studies that evaluate agreement between child and parent report of behaviors.

Reliable and valid assessment of mental health symptoms from a young child’s perspective is a topic of interest in clinical and research settings. Typically, assessments are conducted using standardized structured interviews, such as the Diagnostic Interview Schedule for Children (DISC) (Shaffer, Fisher, Lucas, Dulcan, & Schwab-Stone, 2000) or the Schedule for Affective Disorders and Schizophrenia for School-Age Children (K-SADS) (Kaufman et al., 1997). However, the development of children’s cognitive and emotional capabilities may affect the quality and accuracy of children’s self-report. Indeed, the reliability and validity of these diagnostic interviews with young children have been questioned (Breton et al., 1995; Edelbrock, Costello, Dulcan, Kalas, & Calabro Conover, 1985; Fallon & Schwab-Stone, 1994; Schwab-Stone, 1995; Schwab-Stone, Fallon, Briggs, & Crowther, 1994).

The criticisms include poor test-retest reliability (Edelbrock et al., 1985; Fallon & Schwab-Stone, 1994), poor comprehension (Breton et al., 1995), and lengthy administration time (Schwab-Stone et al., 1994). Thus, clinical assessments are often supplemented with children’s self-report measures that rely on a questionnaire format because these self-reports have been found to provide useful information. For example, one study found that school-age children’s report on the Children’s Depression Inventory, a questionnaire targeting depressive symptoms, was a better predictor of some later outcomes (e.g., diagnosis of major depression) than their parent’s report (Ialongo, Edelsohn, & Kellan, 2001). Efforts at improving reliability and validity of self-report measures for young children have resulted in format changes, with newly developed self-report questionnaires incorporating pictorial representations of questions and developmentally sensitive language into traditional self-report measures (e.g., Edelbrock, Crnic, & Bohnert, 1999; Ernst, Cookus, & Moravec, 2000; Martini, Strayhorn, & Puig-Antich, 1990; Valla, Bergeron, & Smolla, 2000).

One such measure, the Dominic Interactive (DI) assessment, is a computerized self-report of seven disorders from the DSM-IV that capitalizes on auditory and visual components, does not require the child to rate their behavior based on time components (e.g., past week, past month), has a running time of 15 min, and automatically records and calculates symptom scores (Valla et al., 2000). The DI was originally developed as a paper-and-pencil test titled the Dominic-R and was based on DSM-III-R criteria. Multiple psychometric studies have demonstrated good reliability and adequate validity of the Dominic-R (Bidaut-Russell, Valla, Thomas, Begeron, & Lawson, 1998; Murphy, Cantwell, et al., 2000; Murphy, Marelich, & Hoffman, 2000; Valla, Bergeron, Berube, Gaudet, & St-Georges, 1994; Valla, Bergeron, Bidaut-Russell, St-Georges, & Gaudet, 1997); however, the computerized adaptation—the DI assessment—has been studied less frequently and no study has compared the psychometric differences between the computer and noncomputer version.

A recent study using the DI in young French children reported significant differences between a school-based population and an outpatient clinical sample in terms of the proportion of children reaching two cutoff points designed to estimate clinical severity based on the number of symptoms the child endorses for a given disorder (Valla et al., 2002). In this study, 250 typical children and 150 inpatient and outpatient clinic children were assessed with the DI. The two groups were significantly different on most DI scales, with the treatment-based group reaching the clinical ranges more frequently compared to typical children (Valla et al., 2002). Thus, although preliminary evidence supports the discriminant validity of this scale, further research is needed about the scale’s reliability and validity.

The purpose of the present study was to examine the psychometric properties of the DI, including internal consistency reliability and concurrent validity. Given that the DI is not a well-validated measure, this information was important before pursuing additional analyses. Our sample of cocaine-exposed and non-cocaine-exposed children was not expected to differ on the basis of psychometric characteristics of the scale. Therefore, no hypotheses were made regarding differences between these two groups on psychometric properties. For the purposes of this article, the sample is presented as a whole; however, when differences were found between the two groups, those results are reported.

To examine reliability, the internal consistency of each DI scale was evaluated. To examine concurrent validity, scores from the DI were hypothesized to relate to scores on a variety of measures that contained conceptually similar scores, including the Child Behavior Checklist (Achenbach, 1991), Parenting Stress Index (Abidin, 1995), and a modified version of the Conners’s Teachers Rating Scale (Conners, 1990) completed by the child’s examiner. The DI also was hypothesized to relate to the Affect in Play Scale (APS), a structured play task that assessed the quality, frequency, and organization of affect observed in the child’s play. The frequency of negative affect score from the APS was hypothesized to relate to the DI based on the Tripartite Model and other theories of psychopathology that suggest a relationship between negative affect and maladaptive behaviors (Bradley, 2000; Clark, Watson, & Mineka, 1994). In summary, the purpose of the present investigation was to determine the reliability and validity of the DI in a sample of poly-drug-exposed children using child self-report, parent report, examiner report, and observational play data.

METHOD

Participants

Participants were 6-year-old children (169 cocaine-exposed and 153 non-cocaine-exposed children, n = 322) who had been enrolled in a longitudinal study since birth (Singer et al., 2002). Mothers and infants were recruited from a large, urban, county teaching hospital. Drug use was quantified through urine samples obtained before or after labor and delivery and/or meconium collected from infants’ diapers. Maternal self-report of cocaine use during pregnancy also was used to identify cocaine-exposed infants.

A nurse recruiter identified 647 mothers and their infants as eligible for inclusion in the study. Of the 647, 155 mothers refused to participate (49 cocaine exposed, 106 cocaine negative) and 54 (20 cocaine exposed, 34 cocaine negative) were excluded for various reasons, such as no meconium, teen mother, infant diagnosed with fetal alcohol syndrome or Down’s syndrome, and so forth. At birth, 415 women and their infants were enrolled in the study (218 cocaine exposed, 197 cocaine negative). Included in the present investigation are 322 children who were seen at the 6-year visit, were administered the DI assessment, and obtained a full-scale IQ (FSIQ) on the Wechsler Primary Preschool Scale of Intelligence–Revised (WPPSI-R) of 70 or greater. Children with less than 70 IQ were excluded due to cognitive limitations that were assumed to decrease their ability to self-reflect mental health symptoms. A total of 21 children were excluded based on the IQ criterion (n = 16, FSIQ 61–69; n = 4, FSIQ < 60). Of these 21 children, 8 were CE and 13 were NCE. No differences emerged between these two groups on demographic or drug-use variables.

Measures

The DI (Valla, 2000)

The DI assessment is a computerized self-report measure of common mental health disorders in childhood. The computer program presents each symptom in a written question format, accompanied by one or more colorful picture(s), music, and a voiceover that orally presents the question. The DI has an average running length of 15 min but can be shorter or longer depending on the speed of response by the child. The DI is specific to the child’s sex and ethnicity (Caucasian, African American, Latino, and Asian versions). The current study utilized the Caucasian version (character called Dominic) and the African American version (character called Terry). The children responded yes or no to 91 questions that asked whether they think, feel, or act like Dominic/Terry while portrayed in a variety of pictorial scenarios.

All questions correspond to symptom criteria set forth in the DSM-IV for six diagnostic categories. The seventh, specific phobias, does not map exactly to the diagnostic criteria of the DSM-IV as the other categories do (Valla, 2000) because of the assessment of multiple types of phobias (e.g., spiders, heights). The diagnostic categories assessed include specific phobias, separation anxiety disorder (SAD), generalized anxiety disorder (GAD), major depression/dysthymia (MDD), oppositional defiant disorder (ODD), conduct disorder (CD), and attention deficit hyperactivity disorder (ADHD). Also, summary scores are calculated for internalizing disorders (phobias, SAD, GAD, MDD), externalizing disorders (ODD, CD, ADHD), and a total score combining all of the symptoms endorsed.

Interpretation of the data obtained from the DI is based on cut points: a moderate cut point indicates there could be a problem and a severe cut point indicates that there is a problem. The cut points were based on statistical and clinical factors presented in the DI manual (Valla, 2000) and do not correspond to a diagnosis because frequency and duration of behavior is not assessed as is required for a DSM-IV diagnosis. Despite mapping onto DSM-IV symptoms, this assessment does not purport to diagnose; rather, it assesses the child’s self-report of symptoms in several commonly diagnosed childhood disorders.

The Child Behavior Checklist (CBCL) (Achenbach, 1991)

The 118-question CBCL is a commonly used parent report measure designed to assess problem areas in child behavior. A parent responds to each question using a 3-point response scale (e.g., not true, somewhat or sometimes true, and very true or often true). The CBCL provides eight narrowband scores (withdrawn, somatic complaints, anxious/depressed, social problems, thought problems, attention problems, delinquent behavior, and aggressive behavior) and three broadband scores (internalizing, externalizing, and total problems).

Conners’s Teacher Rating Scales-28 (CTRS-28) (Conners, 1990)

The CTRS-28 is designed to screen for behavioral difficulties in children and yields four scores: conduct problems (8 questions), hyperactivity (7 questions), inattentive-passive (8 questions), and a 10-item hyperactivity index. The questions are answered on a 4-point response scale (not at all, just a little, pretty much, very much). A modified version of the CTRS-28 was used in the present study. This original version was developed for classroom teachers who are familiar with the child’s behavior. Examiners completed the CTRS-28 in the present study to obtain a measure of the child’s behavior during the testing session. The testing session lasted approximately 5 hours, and a brief measure of the child’s behavior during this interaction was hypothesized to be informative. Examiners could not address questions specifically aimed at classroom activities; therefore, eight questions were omitted due to their irrelevance to the assessment setting (e.g., gets along well with others). An average score was computed to correct for the missing questions. Based on the Conners’s manual, four of the dropped questions did not load onto any of the calculated scores, three loaded onto the inattentive-passive score, and one loaded onto both the hyperactivity score and the hyperactivity index.

The Affect in Play Scale (APS) (Russ, 1993; Russ, Niec, & Kaugars, 2000)

The APS is designed to measure affect and cognitive processes in 6- to 10-year-old children’s play. The play task is videotaped and is administered with standardized instructions. The child is given two puppets, three blocks, and 5 min to play with the objects. If the child does not begin to play within 2 minutes of the initiation of the task, the examiner stops the testing. At the 4-min mark, the child is told that they have 1 min left to play. The APS is scored on affective (emotional expressions) and cognitive (quality and structure of play narrative) dimensions by trained raters. The current study had excellent interrater reliability (.90 and above). Two scores hypothesized to relate to the DI are included in the present analyses: the frequency of negative affect score, which is derived from expressions of anxiety, sadness, aggression, frustration, oral aggression, and anal content in the child’s play, and the total affect score, a combination of negative affect and positive affect (derived from expressions of happiness, nurturance, oral, sexual, and competition) in the child’s play. The psychometric properties of the APS have been demonstrated in a number of school-based studies (Christiano & Russ, 1996; D’Angelo, 1995; Niec & Russ, 1996, 2002; Russ & Grossman-McKee, 1990). Reliability estimates are good (.80 and above) and interrater reliability typically ranges from .80 to .90 (Russ et al., 2000).

Brief Symptom Inventory (BSI) (Derogatis, 1992)

The BSI is a self-report, 53-item questionnaire for adults that taps a range of psychiatric symptom patterns. The BSI measures somatic complaints, obsessive-compulsive behavior, depression, anxiety, phobic anxiety, paranoid ideation, hostility, and psychoticism and yields a summary score, the Global Severity Index (GSI), and a measure of overall psychological distress.

The Parenting Stress Index (PSI) (Abidin, 1995)

The PSI is a 120-item questionnaire designed to measure the parent-child relationship. The majority of the questions (101) are answered on a 5-point scale (e.g., strongly agree, agree, not sure, disagree, and strongly disagree). Nineteen of the questions follow a yes-or-no format and involve life events that have occurred in the family in the last 12 months, including divorce, death of a family member, and problems at school. Parents complete the PSI about their own adjustment and the target child’s adjustment and functioning. Two scores are included in this investigation: a total parenting stress score and a total child problem score.

WPPSI-R (Wechsler, 1989b)

The WPPSI-R is an individually administered, standardized, normative measure for assessing intelligence in young children, which yields an overall IQ as well as Verbal and Performance IQ scores.

Home Observation for Measurement of the Environment (HOME-Preschool) (Caldwell & Bradley, 1984)

The HOME-Preschool assessed the quality of the caregiving environment. This scale yields one global score, which is a summary score of the following: learning materials, language stimulation, physical environment, responsivity, academic stimulation, modeling, variety, and acceptance. The scale was administered in an interview format, which has been supported based on correlations between cognitive outcomes and the HOME score compared to samples using in-home observations (Barnard, Bee, & Hammond, 1984; Jacobson, Jacobson, Sokol, Martier, & Ager, 1993).

Maternal intelligence assessments

The Peabody Picture Vocabulary Test–Revised (PPVT-R) (Dunn & Dunn, 1981) and two subscales (Block Design [BD] and Picture Completion [PC]) of the Wechsler Adult Intelligence Scale–Revised (WAIS-R) (Wechsler, 1989a) were administered to the child’s caregiver to assess maternal intelligence.

Procedure

The participants were seen at birth, 6 months, 1 year, 2 years, 4 years, and 6 years of age. The primary caregiver, who accompanied the child to each assessment, was given an assessment battery that included a vocabulary test, drug interview, and questionnaires. An examiner, blind to drug status, administered the assessment battery to the child.

The testing session lasted approximately 5 hours, with a lunch break approximately halfway through. Other assessments in the child battery included standardized measures of language, motor skills, and neuropsychological tests. Only measures hypothesized to relate to the validity of the DI assessment are included in the present investigation.

The Institutional Review Boards of the participating hospitals approved the study and written informed consent was obtained from the caregiver. Confidentiality was assured to all participants, supplemented by a Writ of Confidentiality (DA-98-91) from the Department of Health and Human Services. All caregivers were given a monetary stipend for participation, with transportation costs to the lab and lunch provided.

RESULTS

Demographics

Characteristics of the 322 participants (169 cocaine-exposed [CE] and 153 non-cocaine-exposed [NCE] children] are reported in Table 1. Overall in this sample, the average age was 6 years and 1 month, predominantly African American (81%), and approximately evenly distributed by gender (45% boys, 55% girls). At birth, the CE children had smaller birth length and birth weight (ps < .01). Given that these differences may reflect gestational age alone, a MANCOVA was performed controlling for gestational age. Differences in weight, height, and head circumference persisted despite efforts at statistically controlling for gestational age. At 6 years of age, the CE children earned lower scores on the WPPSI-R Full Scale IQ (90.3 vs. 93.6, t = 2.4, p < .05) than did NCE children.

TABLE 1.

Demographic Characteristics of Cocaine-Exposed and Non-Cocaine-Exposed Children

	Cocaine (n = 169)		Noncocaine (n = 153)
	n	%	n	%	χ2/t	p
Gender (male)	77	46	69	45	.01	.93
Race (non-White)	139	82	122	80	.33	.57
Premature	49	29	30	20	3.8	.05
Use during pregnancy
Alcohol	140	86	95	65	18.2	.0001
Cigarettes	141	87	57	39	75.4	.0001
Marijuana	79	49	18	12	46.7	.0001
	M	SD	M	SD
Age	6.1	0.2	6.1	0.3	1.4	1.7
WPPSI-R full-scale IQ	90.3	11.8	93.6	13.1	2.4	.02
Gestational age (weeks)	37.7	2.9	38.5	2.9	2.6	.01
Birth weight (grams)	2710	678.1	3086	703.9	5.4	.0001
Birth length (cm)	47.2	3.9	49.0	4.1	4.5	.0001
Head circumference (cm)	32.2	2.3	33.4	2.5	4.8	.0001

NOTE: WPPSI-R = Wechsler Primary Preschool Scale of Intelligence–Revised.

The following demographics on the biological mothers are based on information provided at the child’s birth. The biological mothers were primarily low socioeconomic status (98%) based on the Hollingshead index, with the majority of the women unmarried (87%). A greater number of NCE biologic mothers were employed at the child’s birth (χ² = 22.2, p < .0001) and had completed high school as compared to CE children’s mothers (χ² = 10.2, p < .01). The cocaine-using women were older at the child’s birth (29.6 vs. 25.7), had fewer prenatal visits (5.1 vs. 8.7), and had more pregnancies (3.6 vs. 2.7) as compared to NCE children’s biologic mothers (ps < .0001). A greater percentage of the cocaine-using mothers reported using alcohol, cigarettes, and marijuana during pregnancy compared to the non-cocaine-using women (see Table 1). Although groups did not differ on the WAIS-R standard scores, CE women had lower PPVT-R scores (73.9 vs. 77.8, t = 2.5, p < .05) and higher self-reported psychological distress using the Brief Symptom Inventory than did NCE mothers (0.53 vs. 0.35, t = −5.3, p < .001). The measure of home environment (HOME-Preschool) did not differ significantly by group (41.7 vs. 41.3, t = −.48, p = .63).

Descriptive Data for the DI Assessment

The DI descriptive data for the total sample, including sample means, are presented in Table 2. The DI means were compared between CE and NCE children, with no differences emerging for the total (28.1 vs. 27.3, t = −.47, p = .14), externalizing (8.9 vs. 7.8, t = −1.7, p = .09), internalizing (19.2 vs. 19.6, t = .37, p = .71), specific phobias (2.2 vs. 2.2, t = −.04, p = .96), separation anxiety (4.0 vs. 4.2, t = .59, p = .56), generalized anxiety (6.5 vs. 6.8, t = .86, p = .39), major depression (6.4 vs. 6.4, t = −.1, p = .92), oppositional defiant disorder (2.4 vs. 2.0, t = −1.8, p = .07), conduct disorder (1.2 vs. .81, t = −1.9, p = .06), or attention deficit hyperactivity disorder scale (5.3 vs. 4.9, t = −1.0, p = .31). CE and NCE children were not different in mean scores on the CBCL, PSI, CTRS-28, and the APS, except for the total child domain score on the PSI. CE children received a higher problem score than did the NCE children (103 vs. 95.8, t = −2.8, p < .05). Gender differences also were examined. Girls reported significantly higher means on the DI phobias scale (2.5 vs. 1.8, t = 3.5, p < .001). Boys self-reported higher means on the DI ADHD scale (5.6 vs. 4.7, t = −2.2, p < .05) and the DI externalizing score (9.2 vs. 7.6, p < .05). No other gender differences were found.

TABLE 2.

The Dominic Interactive Means, Alphas, and Percentage in Clinical Ranges for the Total Sample

Dominic Interactive (# of items per scale)	M	SD	Cronbach’s α	Moderate Cut Point	Severe Cut Point
Total (94)	27.7	14.4	.92	—	—
Internalizing (52)	19.4	9.4	.88	—	—
Externalizing (42)	8.3	6.2	.86	—	—
Phobias (9)	2.2	1.9	.63	126 (39%)	40 (12%)
SAD (8)	4.1	2.3	.70	153 (48%)	98 (30%)
GAD (15)	6.7	3.2	.71	60 (19%)	27 (8%)
MDD/dysthymia (20)	6.4	4.1	.79	55 (17%)	15 (5%)
ODD (9)	2.2	1.8	.61	41 (13%)	5 (2%)
CD (14)	1.0	1.8	.79	39 (12%)	13 (4%)
ADHD (19)	5.1	3.7	.79	32 (10%)	10 (3%)

NOTE: SAD = separation anxiety disorder, GAD = generalized anxiety disorder, MDD=major depression/dysthymia, ODD = oppositional defiant disorder, CD = conduct disorder, ADHD = attention deficit hyperactivity disorder. Total n = 322.

Table 2 also presents internal consistency data for the broadband scales (internalizing, externalizing, and total score) on the DI, with alphas ranging from .86 (externalizing) to .92 (total score). Moderate to good internal consistency for the seven symptom scales was achieved, with alphas ranging from a low of .61 for ODD to a high of .79 for CD, ADHD, and MDD. Three analyses were conducted to determine if internal consistency varied as a function of group status. The sample was divided into three subgroups based on gender, test version (Dominic vs. Terry), and cocaine status. None of the between-group correlations was statistically significant based on t tests for independent correlations. In general, the reliability estimates for the subgroups and the total sample were comparable, with moderate to good internal consistency for all DI scores.

Frequencies for the total number of children reaching the two DI cut points, moderate and severe, are presented in Table 2. The total number of children reaching the moderate cut point ranged from 10% (ADHD) to 48% (SAD). For the severe cut point, the proportion of children in the clinical range varied from 2% (ODD) to 30% (SAD). Frequencies also were computed based on cocaine status and gender. No frequency differences emerged for CE and NCE children on any of the DI cut points. In contrast, more boys achieved the moderate cut point on the CD scale as compared to girls (16% vs. 9%, χ² = 4.7, p < .05).

Anecdotal evidence from examiners suggested that the children responded quite well to completing this self-report questionnaire. In addition, the final question on the DI evaluated the child’s opinion of the questionnaire by asking, “Did you like the Dominic Interactive?” On this question, 304 children (94%) answered in the affirmative. Of the 18 (6%) who said they did not like it, 5 were CE and 13 were NCE.

Concurrent Validity Correlations for the DI

Correlations for the total sample of the DI and the criterion measures are presented in Table 3. Significance level was set at p < .01. Although this is a liberal p value given the large number of comparisons, it was decided that the descriptive information provided by these correlations would outweigh a more conservative p value and that a cautious interpretation would be utilized.

TABLE 3.

Correlations Among the Dominic Interactive Assessment With Criterion Measures

	Total	INT	EXT	SPH	SAD	GAD	MDD	ODD	CD	ADHD
FSIQ	−.06	.02	−.20***	−.06	.07	.10	−.06	−.13	−.24***	−.17*
Home score	−.01	.01	−.05	−.02	.02	.02	−.02	−.04	−.01	−.05
PSI child total	.11	.09	.14	−.03	.08	.09	.11	.20*	.05	.09
PSI parent total	.05	.04	.06	.01	.08	.05	.04	.08	.01	.05
GSI parent	−.01	.01	−.03	.04	.02	.01	−.04	.04	−.01	−.07
APS total affect	.17*	.17*	.14	.12	.07	.14	.20***	.15*	.11	.08
APS negative affect	.19*	.16*	.20***	.09	.06	.13	.20***	.21***	.13	.13
CBCL total	.10	.07	.13	−.05	.01	.06	.11	.15*	.11	.10
CBCL internalizing	.08	.06	.08	−.03	.04	.06	.06	.13	.07	.05
CBCL externalizing	.11	.07	.14	−.04	.01	.06	.12	.16*	.13	.11
CBCL anxious	.07	.08	.04	.00	.07	.07	.06	.07	.07	.01
CBCL withdrawn	−.01	−.03	.03	−.04	−.02	−.03	−.02	.09	.07	−.02
CBCL aggression	.12	.08	.16*	−.02	−.01	.08	.14	.16*	.15*	.13
CBCL delinquency	.12	.11	.12	.01	.05	.10	.14	.16*	.09	.11
CBCL attention	.04	.02	.06	.00	−.03	.02	.03	.10	.05	.02
CTRS total	.15*	.12	.15*	.02	.08	.10	.14*	.09	.11	.13
CTRS hyperactivity	.14	.11	.16*	.01	.06	.08	.14	.12	.13	.13
CTRS inattention	.11	.09	.11	.03	.04	.05	.11	.07	.12	.08
CTRS conduct problems	.05	.05	.04	.01	.05	.03	.05	.01	.02	.03

NOTE: INT = internalizing, EXT = externalizing, SPH = specific phobias, SAD = separation anxiety disorder, GAD= generalized anxiety disorder, MDD = major depression disorder, ODD = oppositional defiant disorder, CD = conduct disorder, ADHD = attention deficit hyperactivity disorder, FSIQ = Full Scale IQ, PSI = Parenting Stress Index, GSI = Global Severity Index, APS = Affect in Play Scale, CBCL = Child Behavior Checklist, CTRS = Conners’s Teacher Rating Scale completed by examiner.

^*p < .013.

^***p <.001.

Overall, correlations between the DI scales and criterion measures yielded low correlations that did not reach statistical significance. Correlations between the DI and the WPPSI-R FSIQ produced three significant relationships, with the DI externalizing score (r = −.20, p < .001), DI Conduct Disorder scale (r = −.24, p < .001), and DI ADHD scale (r = −.17, p < .01). The DI was unrelated to the HOME-Preschool score and to parent psychopathology as measured by the GSI. The parenting stress index also produced low correlations, with the exception of the DI ODD scale and the PSI child total problem score (r = .20, p < .01).

Correlations among the APS and the DI yielded several significant findings. The total affect score was significantly correlated with the DI total (r = .17, p < .01), internalizing (r = .17, p < .01), MDD(r = .20, p < .001), and DI ODD scale (r = .15, p < .01). The negative affect score on the APS significantly correlated with the DI total (r = .19, p < .01), internalizing (r = .16, p < .01), externalizing (r = .20, p < .001), MDD (r = .20, p < .001), and ODD scale (r = .21, p < .001).

Similar findings were obtained on the evaluation of the DI and CBCL, with the majority of correlations low and nonstatistically significant. However, several of the comparisons measuring externalizing behaviors were statistically significant. The DI ODD scale was correlated with the CBCL total (r = .15, p < .01), externalizing (r = .16, p < .01), aggression (r = .16, p < .01), and delinquency scale (r = .16, p < .01). Also, the DI CD scale was correlated with the CBCL aggression subscale (r = .15, p < .01). Finally, the CBCL total score was correlated with the DI externalizing scale (r = .16, p < .01).

Correlations between the modified CTRS-28 and the DI yielded low correlations ranging from .01 (DI ODD and CTRS-28 conduct problems) to .13 (DI ADHD and the CTRS-28 hyperactivity index). The CTRS-28 total score was significantly correlated with the DI total score (r = .15, p < .01), DI externalizing scale (r = .15, p < .01), and the DI MDD scale (r = .14, p < .01). The DI externalizing scale also was correlated with the CTRS-28 hyperactivity score (r = .16, p < .01).

Cocaine Effects on the DI Validity Correlations

Correlations also were analyzed based on cocaine status (CE vs. NCE) and then compared to see if the two correlations were significantly different between the two groups using a t test for independent correlations. No differences emerged on the DI assessment correlations with the PSI, IQ, Home-Preschool, and the GSI. Several differences emerged on the APS. The negative affect score for CE children had a significant relationship with the DI Internalizing score (.28 vs. .04, z = 2.14, p = .03) and the DI GAD score (.27 vs. −.01, z = 2.48, p = .01) compared to NCE children. Also, the total affect score was related to the DI GAD score for CE children only (.28 vs. .01, z = 2.4, p = .01). Two of the comparisons of the DI and the CBCL produced correlations that were significantly different. The DI total score (.23 vs. −.02, p < .05) and DI Internalizing score (.22 vs. −.03, p < .05) for CE children were significantly related to the CBCL delinquency scale, whereas the correlation for NCE children was negatively related and nonstatistically significant. No significant differences were found on the CTRS-28 and the DI scales.

Concurrent Validity Correlations for the CBCL

To clarify our DI correlations and more precisely understand their magnitude, correlations between the CBCL and the criterion measures are presented in Table 4. The CBCL scales were uncorrelated with scores on the WPPSI-R FSIQ, the HOME-Preschool, and the APS. Comparison of the parent CBCL ratings and the PSI scores revealed significant correlations (see Table 4). Parental psychopathology from the GSI was significantly related to parent report on the CBCL total scale (r = .17, p < .01), internalizing (r=.17, p <.01), and attention score (r=.17, p< .01). Examiner ratings from the CTRS-28 were significantly correlated with the CBCL. The CBCL externalizing score was correlated with the CTRS-28 hyperactivity score (r = .15, p < .01) and the CTRS-28 conduct problems score (r = .16, p < .01). The CBCL aggression subscale correlated significantly with each CTRS-28 score (rs .20–.24, ps < .001). Finally, the CBCL attention problems subscale was significantly correlated with the CTRS-28 total score (r = .15, p < .01), the hyperactivity index (r = .16, p < .01), and the inattentive-passive score (r = .16, p < .01).

TABLE 4.

Correlations Among CBCL Scales and Criterion Measures

	Total	INT	EXT	Anxious	Withdrawn	Aggression	Delinquency	Attention
FSIQ	−.01	.04	.00	.13	−.02	−.04	.04	−.03
Home score	.05	.09	.06	.07	.07	.04	.12	.01
PSI child total	.60***	.51***	.57***	.46***	.36***	.57***	.42***	.50***
PSI parent total	.35***	.35***	.34***	.38***	.22***	.35***	.21***	.23***
GSI parent	.17*	.17*	.14	.09	.10	.14	.11	.17*
APS total affect	−.08	10	−.01	−.07	−.09	−.01	−.03	−.02
APS negative affect	−.09	11	−.03	−.11	−.09	.01	−.04	−.04
CTRS total	.11	.05	.13	.06	.07	.21***	.05	.15*
CTRS hyperactivity	.13	.07	.15*	.07	.06	.24***	.07	.16*
CTRS inattention	.10	.03	.13	.06	.06	.20***	.07	.16*
CTRS conduct problems	.12	.07	.16*	.07	.12	20***	.05	.08

NOTE: INT = internalizing, EXT = externalizing, FSIQ = Full-scale IQ, PSI = Parenting Stress Index, GSI = Global Severity Index, APS = Affect in Play Scale, CTRS = Conners’s Teacher Rating Scale completed by examiner.

^*p < .01.

^***p < .001.

Cocaine Effects on the CBCL Validity Correlations

These same comparisons of the CBCL scores and the criterion measures (PSI, GSI, APS, FSIQ, CTRS-28, Home-Preschool) also were conducted by cocaine status. Only four significant differences emerged between the two samples. The CBCL withdrawn subscale was significantly correlated with the PSI total parent stress score for NCE children only (.36 vs. .09, z = −2.35, p < .05). The GSI score was positively and significantly related to CBCL Internalizing scale (.21 vs. −.07, z = 2.42, p < .05), Anxious/Depressed (.21 vs. −.09, z = 2.59, p < .01), and the Withdrawn scale (.22 vs. −.09, z = 2.68, p < .01) for CE children’s caregivers only.

DISCUSSION

This study investigated the psychometric properties of the DI assessment. Internal consistency reliability estimates for each score in the total sample and subsamples were good. This study suggests that the internal consistency of the computer-based DI is similar to published norms for the paper version, the Dominic-R (Bidaut-Russell et al., 1998; Valla et al., 1994, 1997), despite changes in the measure that reflect the DSM-IV criteria and computerized method of administration. Because our study used a low SES, primarily African American polydrug exposed sample, this demonstration suggests that the DI may be appropriate for use with a variety of groups without changing the internal consistency reliability.

Overall, the DI scales had negative to low correlations with the majority of the criterion measures. The few significant correlations were on scales that measured externalizing behavior problems. The externalizing scales on the DI (CD and ODD) were related to the CBCL parent report of aggression and delinquency (only ODD), the parent report of child problems on the Parenting Stress Index, and observational play data from the Affect in Play Scale (APS). The consistency of these findings may reflect the fact that young children receive frequent feedback from their parents about overt behaviors, including fighting, hitting, and rule breaking, and thus may be more accurate reporters on these symptoms. In addition, the conduct disorder scale and the ADHD scale from the DI were significantly negatively related to the child’s full-scale IQ score. This relationship is consistent with prior research that has found that children with lower cognitive ability tend to have more attentional and behavioral problems (Evans & Short, 1991).

Correlations among internalizing scales on the DI and the criterion measures were generally low to negative. One possible explanation for this lack of relationship may lie in the covert nature of these symptoms. Anxiety and sadness are less visible and therefore lend themselves to less direct feedback by adults in the child’s environment (Ablow et al., 1999). These findings are consistent with research that often finds low agreement between child and parent report on structured interviews assessing anxiety disorders and depression (Grills & Ollendick, 2003; Hodges, Gordon, & Lennon, 1990; Jensen et al., 1999). However, one criterion measure, the Affect in Play Scale (APS), showed modest relationships with DI scales measuring internalizing behaviors. The frequency of negative affect score and the total affect score from the APS were positively and significantly related to the DI total score, internalizing score, and MDD scale. Consistent with the Tripartite Model of anxiety and depression (Clark & Watson, 1991), which suggests that negative affect is a key factor in psychopathology, the frequency of negative affect observed during a play task was significantly correlated with depressive symptoms as reported by the child.

Examiner ratings on the CTRS-28 revealed important rater differences. Comparisons between the examiner and the child showed minimal agreement, whereas correlations between examiner and parent measures of conduct problems, hyperactivity, and inattention showed modest levels of agreement. Even with limited contact and experience with the child, the examiner was somewhat consistent with parents on measures of conduct and attention problems. These findings may suggest that examiner ratings are a complementary source of information, as noted in several ADHD studies (Merrell & Wolfe, 1998; Nolan, Volpe, Gadow, & Sprafkin, 1999).

Some differences were observed between the CE and NCE samples in the pattern of correlations obtained. CE children’s self-report of internalizing symptoms (DI internalizing score) was more highly correlated with negative affect from the APS and the CBCL delinquency scale than was self-report from NCE children. One explanation for these correlations could be that more variability existed in the CE sample. Attempts to determine the validity of this hypothesis included examination of the range for each measure, with no significant differences emerging.

In an attempt to examine the magnitude of our DI assessment correlations, an examination of the criterion measures with the CBCL was conducted. In general, the CBCL did not appear to have higher correlations than were found for the DI assessment. In fact, the DI was related to child IQ on several of the scales, which was not the case for the CBCL scales. The highest correlations obtained in this study were found when comparing the CBCL and the PSI; however, these correlations are likely a result of informant effects, that is, the same informant completed both of these measures and inflated the resulting correlations obtained. Notably, the parent report of behavior problems on the CBCL did not relate significantly to the APS total affect or negative affect scores. This is important because the APS and the DI assessment are child expressions and thus may contribute to the greater agreement seen between those two as compared to the child expression on the APS and the parental report on the CBCL.

Several limitations of this study should be mentioned. Because this study was not designed to be a validation study, other measures of child self-report were not available with which to compare the DI assessment. In particular, the concurrent use of a standardized structured interview would have been useful in determining parent-child agreement on the diagnostic categories. However, several of the questionnaires available are considered gold standards (e.g., CBCL) in child behavior assessments. Also, sampling issues may limit the generalizations of this study. This sample is primarily an urban, high risk, low income, African American, polydrug-exposed sample and thus is not representative of the general population.

Despite these limitations, this study is the first to explore the internal consistency and concurrent criterion validity of the DI assessment. Children’s self-report on several DI externalizing scales correlated significantly with parent report of the same behaviors, as well as with observations of children’s behavior during a play task. Although these correlations were small in magnitude, they indicate a better than chance agreement among raters. In addition, the sizes of the correlations are comparable to prior research studies that analyze parent-child agreement. In particular, a large meta-analysis of child and parent agreement found an average correlation of .28 using the CBCL and Youth Self-Report (Achenbach, McConaughy, & Howell, 1987).

The DI contributes to the field of clinical assessment in several ways. First, the DI assessment may be useful during assessments of externalizing disorders. This finding supports prior research advocating for the importance of gathering child report of these symptoms (Loeber, Burke, Lahey, Winters, & Zera, 2000). Furthermore, despite low correlations for internalizing scales, our results are consistent with other reports of low agreement among children and their parents for internalizing disorders (Grills & Ollendick, 2003). In addition, agreement between the Achenbach’s CBCL and several DI scales is of clinical importance. The CBCL is a widely used parent checklist often employed in both research and clinical settings, and the convergent validity found with these externalizing scales may suggest that the DI may be a useful addition to clinical assessments. Finally, the DI assessment has a relatively short administration time (15 min), making it an efficient use of clinical resources, an issue of increasing importance in clinical and research settings.

Biographies

Teresa J. Linares Scott is pursuing a clinical psychology Ph.D. from Case Western Reserve University. She is currently completing her clinical internship at Children’s Hospital Boston/Harvard Medical School.

Elizabeth J. Short, Ph.D., is a full professor in the Department of Psychology at Case Western Reserve University. Her research has largely focused on the cognitive, neuropsychological, and emotional factors that influence self-regulated learning in both typical and atypical learners.

Lynn T. Singer serves as the deputy provost and vice president for academic programs at Case Western Reserve University. A professor of pediatrics, psychiatry, and general medical sciences, Dr. Singer also directs federally funded research projects that represent almost $25 million of funding to the university, including Project GLAD (Growth, Learning, and Development), a 15-year longitudinal study of infants with bronchopulmonary dysplasia and very low birth weight; Project Newborn-Next Steps, a longitudinal study of cocaine-exposed infants and their mothers, now in its 10th year; and Project Daisy, a study of infants exposed to MDMA (“ecstasy”) based in London. As of 2003, she is also project director of the National Science Foundation ADVANCE program to enhance the careers of women faculty in science and engineering.

SandraW. Russ, PhD, is professor of psychology at Case Western Reserve University. Her research is in children’s pretend play, affect, and creativity. She also has written in the area of child psychotherapy.

Sonia Minnes, Ph.D., is an assistant professor at Case Western Reserve University, School of Medicine, in Cleveland Ohio. Dr. Minnes has done research on the developmental effects of prenatal cocaine exposure for the past 10 years. Her area of special interest is in mental health and regulatory outcomes in prenatally drug-exposed children.