Assessing the Mental Health of Maltreated Youth with Child Welfare Involvement Using Multi-Informant Reports

Abstract

Researchers often question the validity of multi-informant assessments among adolescents with child welfare involvement. Yet, within other clinical populations, prior research finds that multi-informant reports have a discernable structure characterized by discrete patterns of agreement and disagreement. This structure “tracks” contextual displays of behavior and clinical severity. We examined the structure of multi-informant reports (i.e., adolescent, caregiver, teacher) of adolescent externalizing and internalizing problems in a sample of adolescents with a history of child welfare involvement. Across problem domains and informants, reporting patterns mirrored those observed in other clinical populations, and displayed characteristics robustly present in meta-analytic work on cross-informant correspondence. Specifically, informants agreed more on reports of externalizing problems than internalizing problems and caregiver-teacher dyads agreed more than adolescent-caregiver dyads. Overall, we found robust, replicable patterns of multi-informant reports among child welfare involved adolescents. These reporting patterns may facilitate use and interpretation of multi-informant evidence-based assessments among this population.

Youth with a history of foster care placement are at heightened risk for mental health problems including externalizing problems (e.g., conduct disorder, substance abuse) and internalizing problems (e.g., anxiety, depression).¹ This heightened risk stems, in part, from these youth experiencing maltreatment, frequent transitions among caregivers while in foster care, and multiple traumatic events.^1,2 Further, youth who have been in foster care are at heightened risk for poor long-term educational, employment, criminal, housing, and physical and mental health outcomes.³ The lived experience of an adolescent with a history of foster care placement necessitates evidence-based assessment of and early intervention for their mental health concerns. The implications of assessment for service delivery highlight the importance of conducting reliable and valid clinical assessments that inform sound clinical decision-making.

In routine mental health practice, informants such as youth, caregivers, and teachers often provide reports of youth mental health symptoms and functioning.⁴ Each of these informants is thought to provide unique and incrementally valid information. That is, the contexts in which these informants observe youth vary considerably (e.g., home vs. school).⁵ Further, youth may display considerable variation in the degree to which they display mental health concerns within and across these contexts.⁶ Thus, it is unsurprising that cross-informant reports display low-to-moderate levels of convergence with one another (i.e., rs in the .20s to .30s).^7,8 Importantly, these low levels of convergence present uncertainties for many clinical tasks, including identifying youth problems to target in treatment, determining prevalence rates of disorders, and evaluating the effectiveness of interventions.⁹ Unfortunately, when completing these important tasks, no consensus exists for how to interpret patterns of multi-informant reports to obtain clinically meaningful information.¹⁰

Issues surrounding the use of multi-informant reports are present across informants (e.g., youth, caregiver, teacher), problem types (e.g., internalizing, externalizing), and settings (school, clinic). Historically, informant discrepancies were theorized to result from measurement error or informant bias. However, meta-analytic work finds that multi-informant reports display predictable patterns. First, multi-informant reports of externalizing problems tend to yield higher levels of convergence compared to reports of internalizing problems.^7,8 This likely reflects that displays of externalizing problems (e.g., aggression, hyperactivity) are relatively more observable than internalizing problems (e.g., sadness, worry). Second, across problem types, informant dyads who observe youth in the same setting (e.g., mother-father) tend to yield higher levels of correspondence compared to informant dyads who observe youth in unique settings (e.g., caregiver-teacher). This finding supports that youth display mental health symptoms to differing degrees across contexts (e.g., problem is present at home but not school) and highlights the importance of considering the context in which informants observe youth behavior.¹¹

To identify patterns of multi-informant reports and determine if these patterns yield systematic data about the mental health domains assessed, researchers have leveraged person-centered analytic techniques (e.g., Latent Class Analysis [LCA]¹²).^13–21 This work consistently finds that the structure of multi-informant assessments is characterized by discrete patterns of reporting in which: (1) both informants agree on low levels of symptoms, (2) both informants agree on high levels of symptoms, and (3) informants disagree such that one informant endorses symptoms that the other informant does not endorse. Further, these patterns of multi-informant reports provide meaningful clinical information. Across assessment of conditions as diverse as anxiety, mood, autism, and disruptive behavior, these patterns predict such independent clinical outcomes as: (a) cross-contextual consistencies and inconsistencies in youth disruptive behavior,¹⁴ (b) observed parenting,²¹ (c) mental health service use,¹⁹ (d) mental health diagnoses,^15,16,20 (e) treatment response,²⁰ and (f) outcomes in adulthood.¹⁵ This work highlights the clinical utility of examining patterns of multi-informant reports and has important implications for clinical decision-making. However, it is unclear whether the robustness and clinical utility of multi-informant reporting patterns extend to assessments conducted among maltreated youth with child welfare involvement.

In particular, long-standing concerns about the validity of multi-informant reports collected in assessments of maltreated youth with child welfare involvement raise questions as to whether the structure of agreement and disagreement in reports of youth mental-health symptoms replicates in assessments of this population. Indeed, mental health assessments conducted with youth with a history of abuse and neglect in foster care settings pose additional and unique challenges for the complex task of integrating and interpreting multi-informant reports. Potential informants include not only teachers and biological parents, but also foster and adoptive parents. The evidence base on multi-informant reports of youth mental health in the child welfare population is limited. Mirroring patterns of multi-informant reports from the general population, youth and their foster parents show low-to-moderate levels of convergence in reports across problem types.^22,23 Further, teachers and foster parents demonstrate low levels of convergence in reports of internalizing problems and moderate-to-high levels of convergence in reports of externalizing problems.^24,25 In terms of the level of problem behaviors reported, foster parents, but not biological parents of maltreated youth, tend to report higher levels of internalizing and externalizing problems than the youth in their care.^23,26,27 This contrasts with reporting patterns in community samples, where adolescents consistently report higher rates of internalizing problems compared to their parents.²⁸

A handful of studies have sought to better understand the mental health of child welfare-involved youth by considering whether multi-informant reports vary according to youths’ living situation. That is, does placement type (e.g., kinship care vs. non-relative foster care) impact each informant’s reports and the level of convergence across informants? One study found that correlations between caregiver-teacher ratings were comparable for kinship caregivers and non-relative foster parents, and within the range reported for teacher-caregiver convergence in previous studies of youth in the general population.²⁹ Another study found that youth in foster care converged more in their reports of externalizing and internalizing problems with foster parents than with biological parents.²² In contrast, another study found that youth in kinship care had higher levels of convergence in reports with their caregivers compared to youth in non-relative foster care.²³ In terms of the level of problem behaviors reported, some evidence indicates that teacher-reported problem behaviors do not differ based on living situation²⁹, whereas non-relative foster parents tend to report higher levels of problem behaviors compared with kinship caregivers.³⁰ These results should be interpreted with caution, however, due to the relatively small number of studies conducted with this population. To our knowledge, only one study obtained reports from all three informants (youth, teacher, and caregiver), and only a small sub-sample of youth had caregiver assessments in this study.²⁹ Consequently, a key aim of the current study is to examine the structure of multi-informant reporting patterns among the most widely used informants involved in assessments of child welfare-involved youth.

Rater effects may play a role in multi-informant reporting patterns among youth with child welfare involvement and thus confound the interpretability of patterns of multi-informant reports about these youth. Non-relative foster parents may be more attuned to youth mental health challenges compared with kinship caregivers.^22,30 It may also be that non-relative foster parents are inadvertently affected by stereotyped beliefs about the poor mental health of youth in foster care, which may lead to overestimation of youth internalizing and externalizing symptoms.²³ On the other hand, youth in kinship care may be less inclined to report mental health problems because they fear disruption of placement stability and loss of custody if they disclose mental health challenges.^23,27,31 Finally, some question the validity of self-reports collected from youth in foster care settings due to low convergence with caregiver and teacher reports.²² All of these factors may impact divergence among reports, and thus create uncertainties in clinical decision-making when using and interpreting these reports in mental health assessments of youth with child welfare involvement.

Selection effects offer another explanation for patterns of multi-informant reports among child welfare involved youth. For example, kinship caregivers typically know youth entering their care and may be willing to take in youth with “manageable” behavior but not youth with particularly challenging behavior. In addition, the presence of relatives willing to provide a home may indicate a functional support system already in place for youth, reducing their risk for mental health problems.²⁹ Although living situation, rater effects, and selection effects have been theorized to impact patterns of multi-informant reports among child welfare involved youth, evidence supporting these explanations is limited due to the lack of studies examining multi-informant reports across living situations. In sum, a number of factors inherent in assessments of child welfare-involved youth create uncertainties as to whether patterns of multi-informant reports observed in prior work will generalize to reports collected in this setting.

Current Study

The current study provides a unique opportunity to examine patterns of caregiver, adolescent, and teacher reports of adolescent externalizing and internalizing problems in a high-risk sample of adolescents with a history of welfare involvement. Adolescents in the present study varied in their living arrangements, allowing us to examine hypotheses generated by prior research with this population. In particular, we addressed questions of specific relevance to the validity of multi-informant assessment for maltreated youth with child welfare involvement, and examined whether patterns of multi-informant reports exhibited patterns similar to those observed in other samples. Importantly, given the relatively large sample of youth assessed, the study permitted use of relatively sophisticated analytic strategies with fewer methodological limitations than prior multi-informant studies among youth with child welfare involvement.

We addressed four main aims. First, we examined correlations between adolescent, caregiver, and teacher reports of adolescent externalizing and internalizing problems. We hypothesized that, consistent with prior work,^7,8 informants’ reports would exhibit small-to-moderate levels of convergence.

Second, we took a person-centered approach to examining the structure of multi-informant reporting patterns. Specifically, using exploratory LCA we identified the number and structure of classes of multi-informant reports of youth mental health problems across problem domains (i.e., externalizing, internalizing) and informant dyads (i.e., adolescent-caregiver, adolescent-teacher, caregiver-teacher). Although we did not make a priori hypotheses about the number and structure of classes of multi-informant reports, our approach was informed by prior work finding that patterns of multi-informant reports of youth externalizing and internalizing problems vary, with some informant dyads characterized by convergence between reports (i.e., agree on high or low levels of problem behavior) and other informant dyads characterized by divergence between reports (i.e., one informant endorses problem behavior that the other does not endorse).^16,20

Third, we examined established moderators of multi-informant reports, including problem domain and informant dyad. Consistent with prior meta-analytic work,^7,8 we hypothesized that we would observe higher agreement for multi-informant ratings of externalizing problems compared to internalizing problems, and for pairs of observer informants (i.e., caregiver-teacher) relative to pairs of an observer and the adolescent (i.e., adolescent-caregiver, adolescent-teacher). We addressed our third aim using the best-fitting model identified in tests of our second aim.

Fourth, we explored whether type of living situation was associated with patterns of multi-informant reports. Based on limited prior research^22,23,30 we hypothesized that, relative to teacher reports, non-relative foster and adoptive parents would report more mental health problems, and kinship care providers and biological parents would report fewer problems. Given limited research with this population, we did not have any a priori hypotheses about adolescent-caregiver or adolescent-teacher convergence as a function of living situation.

Method

Participants

Participants included adolescents with a history of foster care placement, as well as their caregivers and teachers. Prior to the present study’s interview time point, participants were enrolled in the longitudinal Fostering Healthy Futures (FHF) study for an average of 32.6 months (Range = 25.0 to 46.0; SD = 7.7). Participants were recruited for the original study if the following inclusion criteria were met at baseline: (1) youth were 9–11 years old, (2) youth had been court-ordered into foster care within the preceding 12 months by participating county child welfare departments, and (3) youth were living in foster care at the baseline assessment. FHF enrolled 91% of all eligible children at this baseline interview. For the current study, 84.3% (359/426) of the youth participants from the original FHF study were recruited for follow-up interviews. Of the 359 adolescent participants interviewed at follow up, 294 (81.9%) had caregiver data and 265 (73.8%) had both caregiver and teacher data (teacher data were not collected for the last cohort). Attrition analyses compared those youth in the original FHF study to the 294 youth who had complete youth and caregiver data. There were no statistically significant differences between these two groups on any demographic variables (i.e., age, gender, race, ethnicity) or baseline behavior problem scores (i.e., internalizing and externalizing problems).

At the time of the follow-up interview, youth ranged in age from 11.0 to 15.7 years old (M = 13.0; SD = 1.1) and were in grades 5 through 9 (M = 7.0; SD = 1.0). Just over half of the sample was male (n=151; 51.4%) and 49.6% self-identified as Latino/Hispanic, 50.5% as White, and 30.0% as Black (non-exclusive categories). Although all children were in foster care at the baseline interview, children could be in one of five living situations at the follow-up interviews: reunified with a biological parent (n = 94), living with another family member (i.e., kinship care) (n = 104), living with an adoptive parent (n = 38), living with a foster parent (n = 56) or living in congregate care (n = 2). The length of time children had been living with their current caregiver (or were reunified with them) ranged from 1 to 174 months (M = 28.2; SD = 21.0). Teachers reported knowing their students for an average of 9.9 months (Range = 1–60, SD = 7.9).

Procedures

All procedures were approved by the Colorado Multiple Institutional Review Board. Consent for youth’s participation was obtained from youth’s legal guardians and caregivers and teachers consented for their own participation. Youth were assented prior to the interviews. Data were collected in person (unless the family was living out of town) for the youth and caregivers, and teachers completed on-line surveys. Interviews were conducted in convenient, private locations, typically at the family’s home. Adolescents and their caregivers were each interviewed separately and provided information verbally through interviews conducted by graduate student research assistants. The teacher interviews were typically conducted several months after the adolescent/caregiver interviews (M_months = 4.1, SD = 2.2). Youth and their caregivers were each paid $40 in cash and teachers were given $25 Amazon gift cards for participating in the study.

Measures

Multi-informant reports of adolescent internalizing and externalizing problems.

Adolescents, caregivers, and teachers completed measures of adolescent mental health from the widely used Achenbach System of Empirically Based Assessment (ASEBA) including the Youth Self-Report (YSR),³² Child Behavior Checklist (CBCL),³³ and Teacher’s Report Form (TRF).³⁴ These measures assess a wide range of emotional and behavioral problems over the prior 6 months and are rated on a 3-point scale ranging from 0 (not true) to 2 (very true or often true). Using informants’ ASEBA reports, we calculated narrowband scales for specific domains of externalizing (i.e., attention problems, rule-breaking behaviors, aggressive behaviors) and internalizing (i.e., anxious/depressed, withdrawn/depressed, somatic complaints) problems. For all narrowband scales, we used standardized age- and gender-normed T scores. Cronbach’s alphas for the YSR (α = .94), CBCL (α = .96), and TRF (α = .97) were all in the excellent range.

Data Analytic Plan

Correspondence between dyad reports.

To address our first aim, we computed bivariate correlations to examine relations among multi-informant reports on the ASEBA narrowband scales.

Classes of multi-informant reports.

We evaluated our second aim using exploratory LCA¹² on the multi-informant reports of externalizing (i.e., attention problems, rule-breaking behaviors, aggressive behaviors) and internalizing (i.e., anxious/depressed, withdrawn/depressed, somatic complaints) problem narrowband scales. This analysis can be used to determine whether qualitatively distinct subgroups of participants exist based on similar patterns of indicator variables. Consistent with prior use of LCA to address similar research questions,^13,14,35 we dichotomized each informant’s reports on the 6 externalizing and internalizing narrowband scales using a 25% cutoff (i.e., 0 = below top 25% of scores, 1 = at or above top 25% of scores). Dichotomized multi-informant reports on the externalizing and internalizing narrowband scales were entered into 6 separate LCA models using Mplus Version 7.1.³⁶ A separate model was run for each dyad and problem type.

We tested one- through four-class solutions. Model fit was evaluated in three ways. First, we examined the statistical significance of the Pearson and Likelihood Ratio Chi-Square tests of model fit with the addition of each class. The process began by assessing the fit of a one-class solution and continued sequentially with the addition of classes until the chi-square statistic was no longer significant, indicating that the number of classes providing the best fit to the data had been identified. In addition, we evaluated model fit indices for each class including the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC).³⁷ Third, absolute model fit for the selected number of classes was evaluated using estimated probabilities of class membership, which can be used to assess the confidence with which dyads have been assigned to classes.¹² For each LCA model, each dyad receives an estimated probability ranging from 0 to 1 for each class in the solution, with greater scores indicating greater probability of latent class membership.³⁸ Mean levels of estimated probabilities for each class needed to meet a 0.70 minimum threshold to ensure that participants had a high probability of being assigned to their particular class. Prior person-centered models examining multi-informant reporting patterns consistently find that three to four reporting classes emerge, with each class consisting of a significant portion of the sample.^13–21 Further, in these studies the probabilities of latent class assignment, in essence the precision of class assignment, often far exceeded recommended benchmarks (e.g., ≥ .70).³⁸ In light of our own sample characteristics and the “match” between our model fit estimates and those of prior work, the evidence indicates that our sample size was adequate for person-centered analyses.

Examining established moderators of multi-informant reports.

We addressed our third aim using Generalized Estimating Equations (GEE), an extension of the General Linear Model (GLM) that assumes correlated observations of dependent and independent variables.³⁹ We used a binomial distribution with a logit link function and unstructured correlation matrix. For GEE analyses, we combined informant agreement and disagreement groups within each LCA model, yielding a dichotomous dependent variable (i.e., 0 = disagreement, 1 = agreement). Second, we statistically modeled independent variables including problem domain (i.e., externalizing, internalizing) and dyad (i.e., caregiver-adolescent, caregiver-teacher, teacher-adolescent) as nested, repeated-measures (within subjects) variables. Problem domain and dyad were evaluated as predictors of multi-informant agreement in separate GEE models.

Exploring the association between multi-informant agreement and living situation.

To address our fourth aim, we first created two living situation groups: living with kinship caregivers (i.e., biological parents [n = 94] or other kin [n = 104]) and living with non-kinship caregivers (i.e., adoptive parents [n = 38] or foster parents [n = 56]). This was due to the relatively small sample size for each living situation and similar patterns of findings across these two caregiver types. We addressed our fourth exploratory aim using multinomial logistic regression analyses to examine the effect of living situation (i.e., biological parent/kin vs. adoptive/foster) on patterns of multi-informant reports (i.e., both informants’ reports low, informant 1 high/informant 2 low, informant 1 low/informant 2 high, both informants’ reports high). Across problem types and informant dyads, agreement in low levels of problems served as the reference group in comparisons across patterns of multi-informant reports.

Results

Preliminary Analyses

In Table 1, we report descriptive statistics for ASEBA narrowband scale reports, and separately for males and females. For reference, we also report normative data for narrowband scale reports across informants. Across informants and problem types, approximately 3 to 24 percent of reports were within the borderline or clinical range.

Table 1.

Means (M), Standard Deviations (SD), and Percentage Borderline and Clinical Range for Multi-Informant Reports of Adolescent Externalizing and Internalizing Problems, and Scale Scores for Nonreferred Normative Samples

	Study Boys	Study Girls	Nonreferred Norms Boys 11–18a	Nonreferred Norms Girls 11–18a	Total Sample	Total Sample
Narrowband Scale	M (SD)	M (SD)	M (SD)	M (SD)	% Borderline Range	% Clinical Range
Adolescent Report (n=294)
Attention Problems	54.3 (5.8)	56.6 (8.5)	54.4 (6.1)	54.5 (5.9)	12.9	4.1
Rule Breaking Behaviors	52.7 (4.7)	53.5 (5.1)	54.4 (5.8)	54.5 (5.9)	7.8	0.3
Aggressive Behaviors	54.2 (6.1)	55.0 (7.4)	54.2 (5.8)	54.4 (6.1)	7.5	5.8
Anxious Depressed	53.1 (5.3)	53.2 (5.6)	54.2 (5.6)	54.3 (6.1)	7.5	0.7
Withdrawn/Depressed	54.2 (5.2)	54.6 (5.5)	54.3 (6.0)	54.3 (5.9)	12.2	0.7
Somatic Complaints	52.6 (4.6)	52.3 (4.6)	54.2 (5.5)	54.4 (5.9)	2.7	1.7
Caregiver Report (n=294)
Attention Problems	57.3 (8.5)	61.0 (10.9)	54.6 (5.9)	54.5 (5.9)	14.3	11.6
Rule Breaking Behaviors	57.9 (7.2)	59.9 (8.1)	54.2 (5.7)	54.2 (5.7)	23.8	10.5
Aggressive Behaviors	59.2 (9.8)	59.4 (8.8)	54.3 (6.1)	54.2 (6.2)	19.4	15.3
Anxious Depressed	57.8 (8.4)	57.5 (8.7)	54.0 (5.7)	54.1 (5.7)	15.0	12.6
Withdrawn/Depressed	58.3 (8.5)	59.1 (8.5)	54.7 (5.7)	54.3 (5.7)	19.0	10.5
Somatic Complaints	55.9 (7.0)	56.2 (7.2)	54.1 (5.8)	54.2 (5.5)	11.2	6.1
Teacher Report (n=265)
Attention Problems	56.7 (7.1)	58.0 (8.6)	54.2 (6.2)	54.2 (6.4)	13.9	6.5
Rule Breaking Behaviors	58.6 (6.9)	58.6 (8.5)	54.3 (6.5)	54.1 (6.5)	12.6	9.5
Aggressive Behaviors	57.6 (8.2)	58.7 (8.4)	54.1 (6.1)	53.9 (6.6)	17.7	7.8
Anxious Depressed	55.3 (7.4)	56.6 (7.2)	54.0 (5.8)	54.2 (6.0)	9.2	4.8
Withdrawn/Depressed	55.4 (6.4)	56.9 (7.7)	54.4 (7.0)	54.1 (6.5)	11.2	4.8
Somatic Complaints	52.3 (5.1)	52.6 (5.9)	54.4 (7.0)	54.1 (6.5)	6.8	1.7

^aIn nonreferred normative samples, the 97^th percentile or higher (3%) fall in the clinical range, and the 93^rd to 97^th percent fall in the borderline clinical range (4%)

Correspondence between Dyad Reports

We computed bivariate correlations to examine associations between adolescent, caregiver, and teacher reports on the ASEBA narrowband scales (see Table 2). Consistent with study hypotheses, correlations between informant ratings were in the small-to-moderate range for externalizing and internalizing problems. However, we observed that the correlation between adolescent and teacher reports on the withdrawn/depressed narrowband scale was non-significant. When examining correlations across caregiver informants, we found small-to-moderate convergence among informants for youth living with biological parents or kin, and small-to-large convergence among informants for youth living with adoptive/foster parents (see Table 3). Overall, we found that patterns of correspondence between multi-informant reports in our sample were similar to previously reported meta-analytic effect size estimates (De Los Reyes et al., 2015; see Table 2 and 3).

Table 2.

Dyad Correspondence in Reports of Adolescent Externalizing and Internalizing Problems and Previously Reported Meta-Analytic Effects Sizes

Informant Dyad	Externalizing Problems		Internalizing Problems
Adolescent-Caregiver (n=294)	Narrowband Scale	r	Narrowband Scale	r
	Attention Problems	.34***	Anxious/Depressed	.12*
	Rule-Breaking	.37***	Withdrawn/Depressed	.16**
	Aggression	.38***	Somatic Complaints	.22***
Meta-analytic effect size		.32		.26
Adolescent-Teacher (n=265)	Narrowband Scale	r	Narrowband Scale	r
	Attention Problems	.26***	Anxious/Depressed	.19**
	Rule-Breaking	.17**	Withdrawn/Depressed	.09
	Aggression	.24***	Somatic Complaints	.23***
Meta-analytic effect size		.29		.20
Caregiver-Teacher (n=265)	Narrowband Scale	r	Narrowband Scale	r
	Attention Problems	.32***	Anxious/Depressed	.29***
	Rule-Breaking	.31***	Withdrawn/Depressed	.24***
	Aggression	.36***	Somatic Complaints	.25***
Meta-analytic effect size		.28		.21
Meta-analytic effect size a		.30		.25

^aMeta-analytic effect sizes derived from De Los Reyes et al. (2015)

^*p< .05;

^**p< .01;

^***p< .001

Table 3.

Correlations between Informant Reports of Adolescent Externalizing and Internalizing Problems and Previously Reported Meta-Analytic Effects Sizes

	Caregiver-Teacher		Adolescent-Teacher		Adolescent-Caregiver
Caregiver Informant	Externalizing r	Internalizing r	Externalizing r	Internalizing r	Externalizing r	Internalizing r
Biological Parent (n=94)	.38***	.30**	.37***	.26*	.41***	.28**
Kin (n=104)	.22*	.36***	.25*	.15	.44***	.11
Adoptive/Foster Parent (n=94)	.62***	.27*	.34**	.14	.33**	.11
Meta-analytic effect size a	.28	.21	.29	.20	.32	.26

^aMeta-analytic effect sizes derived from De Los Reyes et al. (2015)

^*p< .05;

^**p< .01;

^***p< .001

Classes of Multi-Informant Reports

We conducted exploratory LCA to identify classes of multi-informant reports of adolescent externalizing and internalizing problems. We conducted six separate LCA models that varied by problem type (i.e., externalizing, internalizing) and informant dyad (i.e., adolescent-caregiver, adolescent-teacher, caregiver-teacher). We began the process by specifying a one-class model and added additional classes until the model did not show further improvement in evaluation fit criteria.

Across all six LCA models, the Pearson and Likelihood Ratio Chi-Square tests of model fit were both significant for one-to-three class solutions and non-significant for a four-class solution. Across the LCA models, the AIC and/or BIC were lowest for a four-class solution, suggesting that a four-class solution provided maximal model parsimony. We observed one exception: The LCA model of caregiver-teacher reports of internalizing problems. Here, the BIC was lowest for a two-class solution and the AIC was lowest for the three-class solution. However, for the purposes of comparing solutions across informant dyads, we retained the four-class solution for caregiver-teacher reports of internalizing problems.

As a final test of model fit, we examined mean assignment probabilities using Nagin’s³⁸ minimum 0.70 criteria. For each four-class solution (including caregiver-teacher reports of internalizing problems), the mean assignment probabilities were well above the 0.70 cutoff: adolescent-caregiver internalizing (.84), adolescent-caregiver externalizing (.89), adolescent-teacher internalizing (.88), adolescent-teacher externalizing (.90), caregiver-teacher internalizing (.89), and caregiver-teacher externalizing (.94). Thus, across all six LCA models, we retained the four-class solution. Figure 1 shows a graphical depiction of the final latent class solutions and Table 4 reports the class frequencies and percentages.¹

Figure 1.

Graphical depiction of latent class solutions of multi-informant reports of adolescent internalizing and externalizing problems. For each panel, the x-axis denotes the specific domains of adolescent mental health being rated and the y-axis denotes the probability that informants endorsed elevated levels of symptoms in each domain. 1 = Adolescent-Caregiver Reports of Internalizing Problems; 2 = Adolescent-Teacher Reports of Internalizing Problems; 3 = Caregiver-Teacher Reports of Internalizing Problems; 4 = Adolescent-Caregiver Reports of Externalizing Problems; 5 = Adolescent-Teacher Reports of Externalizing Problems; 6 = Caregiver-Teacher Reports of Externalizing Problems; A/P = Attention Problems; RB = Rule-Breaking; AG= Aggression; A/D = Anxious/Depressed; W/D = Withdrawn/Depressed; SC = Somatic Complaints.

Table 4.

Latent Class Solutions of Multi-Informant Reports of Adolescent Mental Health Problems across Problem Domain and Informant Dyad

LCA Model	Both Informants Low N(%)	Informant 1 High, Informant 2 Low N(%)	Informant 1 Low, Informant 2 High N(%)	Both Informants High N(%)	Total N(%)
Adolescent (Informant 1), Caregiver (Informant 2)
Internalizing Problemsa	130 (44.2%)	73 (24.8%)	52 (17.7%)	39 (13.3%)	294 (100%)
Externalizing Problemsb	135 (45.9%)	41 (13.9%)	61 (20.7%)	57 (19.4%)	294 (100%)
Caregiver (Informant 1), Teacher (Informant 2)
Internalizing Problemsa	169 (57.5%)	56 (21.1%)	25 (8.5%)	15 (5.7%)	265 (100%)
Externalizing Problemsb	138 (52.1%)	39 (14.7%)	31 (11.7%)	57 (21.5%)	265 (100%)
Adolescent (Informant 1), Teacher (Informant 2)
Internalizing Problemsa	147 (55.5%)	72 (27.2%)	29 (10.9%)	17 (6.4%)	265 (100%)
Externalizing Problemsb	125 (47.2%)	56 (21.1%)	50 (18.9%)	34 (12.8%)	265 (100%)

^aInternalizing Problems: Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints

^bExternalizing Problems: Attention Problems, Rule-Breaking, Aggression

The four-class latent class solutions showed remarkably similar patterns of multi-informant reporting across problem domains and informant dyads. For all latent class solutions, we identified two classes characterized by convergence in informants’ reports of adolescent mental health problems. These two classes included one characterized by a low likelihood of both informants reporting elevated adolescent mental health problems, and another characterized by a high likelihood of both informants reporting elevated adolescent mental health problems. For all latent class solutions, we also identified two classes characterized by divergence in informants’ reports of adolescent mental health problems. Both of these classes could be characterized by one informant displaying a high probability of reporting elevated adolescent mental health problems, and the other informant displaying a relatively low probability of reporting elevated adolescent mental health problems.²

To understand the consistency of the latent class solutions across problem domain and informant dyad, we conducted a series of chi square tests and interpreted the Cramer’s V statistic to determine the effect size for each comparison. That is, we conducted a series of tests to determine whether individual adolescents were likely to be classified similarly (e.g., convergence in high levels of problems) across problem domains and informant dyads. With three degrees of freedom, we interpreted the Cramer’s V statistic using thresholds described by Gravetter and Wallnau⁴⁰: small (0 to .10), moderate (.10 to .30), and large (.30 or greater). As reported in Table 5, on average, latent class solutions demonstrated large associations across problem domains, mean Cramer’s V(3) = .33, ps < .001. In addition, on average, latent class solutions demonstrated large associations across informant dyads, mean Cramer’s V(3) = .51, ps < .001. Overall, participants had a high likelihood of being classified in the same latent class in all models (e.g., latent class in which both informants reported high levels of youth mental health symptoms), regardless of problem domain or informant dyad.

Table 5.

Cross-Tabulation of Latent Class Solutions of Multi-Informant Reports of Adolescent Mental Health Problems across Informant Dyad and Problem Domain

Across Problem Domain (Within Informant Dyad)
Comparison	χ2 (9)	Cramer’s V (3)
Adolescent-Caregiver	125.72*	.38*
Adolescent-Teacher	94.77*	.35*
Caregiver-Teacher	52.69*	.26*
Across Informant Dyad (Within Problem Domain)
Comparison	χ2 (9)	Cramer’s V (3)
Externalizing Problems a
Adolescent-Caregiver vs. Adolescent-Teacher	206.68*	.51*
Adolescent-Caregiver vs. Caregiver-Teacher	180.47*	.48*
Caregiver-Teacher vs. Adolescent-Teacher	268.09*	.58*
Internalizing Problems b
Adolescent-Caregiver vs. Adolescent-Teacher	196.80*	.50*
Adolescent-Caregiver vs. Caregiver-Teacher	195.54*	.49*
Caregiver-Teacher vs. Adolescent-Teacher	217.81*	.52*

^aExternalizing Problems: Attention Problems, Rule-Breaking, Aggression

^bInternalizing Problems: Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints

^*p < .001

Examining Established Moderators of Multi-Informant Reports

As described previously, we used GEE to address our aim of directly testing the effect of established moderators on multi-informant agreement. Our results revealed partial support for study hypotheses. First, we examined the effect of problem domain (i.e., externalizing, internalizing) on multi-informant agreement (i.e., agreement, disagreement). For adolescent-caregiver reports, GEE analyses revealed a significant effect of problem domain on multi-informant agreement (Wald X² = 4.98; p < .05). Post-hoc univariate analyses of this effect revealed that, compared to their reports of internalizing problems, adolescents and caregivers were more likely to agree in their reports of externalizing problems than to disagree (95% Wald Confidence Interval [CI]: [.54, .96]; p < .05). We did not find a significant effect of problem domain on multi-informant agreement for adolescent-teacher (Wald X² = 0.01; p = .92) or caregiver-teacher (Wald X² = 0.76; p =.38) reports.

Second, we examined the effect of informant dyad (i.e., adolescent-caregiver, adolescent-teacher, caregiver-teacher) on multi-informant agreement (i.e., agreement, disagreement). Our results were consistent with study hypotheses. For externalizing problems, GEE analyses revealed a significant effect of informant dyad on multi-informant agreement (Wald X² = 13.99; p < 0.01). Post-hoc univariate analyses of this effect revealed that, compared to adolescent-caregiver reports, caregiver-teacher dyads were more likely to agree in their reports of adolescent externalizing problems than to disagree in their reports (95% Wald CI: [−.76, −.06]; p < .05). However, adolescent-caregiver and adolescent-teacher dyads did not differ in their likelihood of agreeing in reports of adolescent externalizing problems (95% Wald CI: [−.16, .49]; p = .33). For internalizing problems, GEE analyses revealed a significant effect of informant dyad on multi-informant agreement (Wald X² = 12.26; p < 0.01). Post-hoc univariate analyses of this effect revealed that, compared to adolescent-caregiver reports, caregiver-teacher dyads were more likely to agree in their reports of adolescent internalizing problems than to disagree in their reports (95% Wald CI: [−.90, .25]; p < .01). However, adolescent-caregiver and adolescent-teacher dyads did not differ in their likelihood of agreeing in reports of adolescent internalizing problems (95% Wald CI: [−.48, .10]; p = .21).

Exploring the Association between Multi-Informant Agreement and Living Situation

Using multinomial logistic regression analyses, we examined the effect of living situation (i.e., biological parent/kin vs. adoptive/foster) on patterns of multi-informant reports (i.e., both informants’ reports low, informant 1 high/informant 2 low, informant 1 low/informant 2 high, both informants’ reports high). Across analyses, agreement in low levels of problems served as the reference group in comparisons across patterns of multi-informant reports.

Adolescent-caregiver reports.

Analyses revealed that living situation was associated with patterns of adolescent-caregiver reports of externalizing problems (χ²(3) = 23.9, p < .001). Compared with the reference group (i.e., informants agreeing on low levels of externalizing problems), adoptive/foster parents were more likely than biological parents/kin to agree with adolescents in high levels of problems (OR = 3.63, p < .001) and to report higher levels of problems than adolescents (OR = 4.06, p < .001). We observed no significant difference in odds of adolescents reporting higher levels of problems than caregivers between biological parents/kin or adoptive/foster living situations (OR = 1.80, p = .15), relative to the reference group.

Analyses revealed that living situation was also associated with patterns of adolescent-caregiver reports of internalizing problems (χ²(3) = 11.80, p < .01). Compared with the reference group (i.e., informants agreeing on low levels of internalizing problems), foster/adoptive parents were more likely than biological parents/kin to report higher levels of problems than adolescents (OR = 3.19, p < .01). We observed no significant difference in odds between biological parents/kin and adoptive/foster parents in agreement with adolescents in high levels of problems (OR = 1.86, p=.12) or adolescents reporting higher levels of problems than caregivers (OR = 1.50, p = .21), relative to agreement in low levels of problems.

Caregiver-teacher reports.

Analyses revealed that living situation was associated with patterns of caregiver-teacher reports of externalizing problems (χ²(3) = 22.7, p < .001). Compared with the reference group (i.e., informants agreeing on low levels of externalizing problems), adoptive/foster parents were more likely than biological parents/kin to report higher levels of problems than teachers (OR = 2.96, p < .01) and less likely to report lower levels of problems than teachers (OR = 0.18, p < .05). We observed no significant differences in odds between biological parents/kin and adoptive/foster parents in agreement with teachers in high levels of problems (OR = 1.72, p = .10), relative to the reference group.

Analyses revealed that living situation was also associated with patterns of caregiver-teacher reports of internalizing problems (χ²(3) = 9.20, p < .05). Compared with the reference group (i.e., informants agreeing on low levels of internalizing problems), adoptive/foster parents were more likely than biological parents/kin to report higher levels of problems than teachers (OR = 2.56, p < .01). We observed no significant differences in odds between biological parents/kin and adoptive/foster parents in agreement with teachers in high levels of problems (OR = 1.93, p = .21) or reporting lower levels of problems than teachers (OR = 1.07, p = .89), relative to the reference group.

Adolescent-teacher reports.

Analyses revealed that living situation was not associated with patterns of adolescent-teacher reports of externalizing problems (χ²(3) = 3.81, p = .28), or internalizing problems (χ²(3) = 0.25, p = .97).

Discussion

Main findings

The evidence base on multi-informant assessment among youth in the child welfare population is limited. We had five main findings that advance this literature. First, we found that patterns of correspondence between multi-informant reports for youth with a history of foster care were consistent with patterns reported across samples in prior meta-analytic work.^7,8 Specifically, correspondence between multi-informant reports of adolescent externalizing and internalizing problems were in the small-to-moderate range. Second, we took a person-centered approach to multi-informant assessment by using exploratory LCA to identify patterns of adolescent, caregiver, and teacher reports, and identified four classes with similar characteristics for all informant dyads (i.e., adolescent-caregiver, caregiver-teacher, adolescent-teacher) and problem types (i.e., externalizing problems, internalizing problems). Across these factors, informants’ reports could either be characterized by agreement between reports (e.g., elevated adolescent and elevated caregiver reports of internalizing problems) or disagreement between reports (e.g., elevated adolescent but not elevated caregiver reports of internalizing problems). The patterns of multi-informant reports observed in our sample were consistent with prior work examining multi-informant reporting patterns in non-clinical and clinical samples (i.e., outpatient mental health, inpatient psychiatric unit, school-based, longitudinal birth cohort).^13–21 Third, we found that for any individual adolescent, informants had a high likelihood of exhibiting the same patterns of reporting (e.g., latent class in which both informants reported high levels of youth mental health problems), independent of problem domain or informant dyad. Fourth, we examined two established moderators of multi-informant reports. For problem domain, we found that for adolescent-caregiver dyads only, externalizing problems exhibited higher levels of agreement than internalizing problems. For informant dyad, we found that for both externalizing and internalizing problems, caregiver-teacher dyads were more likely to agree in their reports than adolescent-caregiver dyads. Fifth, we explored whether adolescents’ living situation was associated with patterns of multi-informant reports. Across problem domains, we found that, compared to biological parents/kin, foster/adoptive parents were more likely to report higher levels of problems relative to adolescents and teachers. Overall, we observed robust patterns of multi-informant reports and replicated prior work in a population with long-known challenges in implementation of evidence-based assessment practices. These findings have important implications for both research and clinical practice.

Research and Theoretical Implications

To our knowledge, our study is the first to use person-centered models to examine patterns of multi-informant reports across informant dyads and problem types in the same sample, irrespective of population. As in prior work, we found that multi-informant reports of youth mental health problems could be reliably classified into patterns of reporting using LCA.^13–21 Across latent class solutions, approximately one-half to three-quarters of informants agreed in their reports whereas approximately one-fourth to one-half of informants disagreed in their reports. This finding suggests that despite the overall low-to-moderate levels of convergence between informants, dyads displayed individual differences in reporting patterns that were more often characterized by agreement than disagreement. We found that there was remarkable consistency in patterns of multi-informant reports across problem domains and informant dyads. This robustness in patterns of multi-informant agreement and disagreement in reporting suggests that these patterns can reliably be found when collecting multi-informant reports, including among youth with child welfare involvement. Rather than reflecting informant biases or measurement error, these robust patterns more likely reflect that informants vary in the contexts in which they observe youth.⁵

The present study extended prior research by examining established moderators of multi-informant reports within the same sample. First, consistent with prior work,^7,8 we found that adolescents and caregivers were more likely to agree in their reports of externalizing compared to internalizing problems. This finding suggests that adolescent-caregiver dyads are more likely to agree when rating relatively more overt and observable problems compared to relatively more covert and internal problems. However, we found no differences in the likelihood of agreement between externalizing and internalizing problems for adolescent-teacher or caregiver-teacher dyads. Consistent with prior work,^7,8 we also found that caregiver-teacher dyads were more likely to agree in their reports compared to adolescent-caregiver dyads. Further, significant differences were not observed between adolescent-caregiver and adolescent-teacher dyads. This finding is consistent with theoretical work suggesting that informants rate youth mental health problems from unique perspectives (i.e., self, other) that impact agreement levels.⁵ These unique perspectives each provide useful and valid information and may be best understood as providing information about intrapersonal and interpersonal manifestations of mental health problems.⁴¹

Given that collecting and interpreting multi-informant reports in foster care comes with unique challenges, the present study provided a conservative test of the interpretability of multi-informant reports. Prior work theorizes that patterns of informants’ reports stem from youths’ living situation, rater effects, and selection effects. This work includes a relatively small number of studies and is limited by small sample sizes and use of single informants or informant dyads. Given that no previous study to our knowledge has leveraged person-centered models to examine patterns of multi-informant reports among adolescents with child welfare involvement, we conducted a series of exploratory analyses. We found that foster/adoptive parents were more likely than biological parents/kin to endorse adolescent mental health problems at higher levels than adolescents and teachers. These patterns of findings are consistent with prior work finding that non-relative foster parents tend to report higher levels of problems compared to other kinship informants and youth.^23,26,30 Our findings suggest that foster and adoptive parents may also report higher levels of youth problems compared to teachers. It may be that non-relative foster parents are more attuned to youth mental health challenges, or are biased to overestimate youth with child welfare involvement’s mental health problems.^22,23,30 This may stem from kinship caregivers being more inclined underreport youth mental health problems due to concerns about placement disruption or the family’s loss of child custody.^23,27,31 Although our findings do not disentangle the exact cause of multi-informant divergence in foster care settings, our sample allowed us to examine reporting patterns across key informants and living situations. Given that we found that patterns of reports were robust across problem domains and informant dyads as well as consistent with patterns of reports in the general population, our findings suggest that it may be most prudent to consider the meaningful information that can be leveraged from multi-informant reports. That is, as opposed to interpreting divergence in reports as indicating measurement error or informant bias, it may be more clinically informative to ask: What do multi-informant reports signal about the manifestation of adolescent problems across contexts and informants’ perspectives? Drawing on prior work, future research should address this question using multi-informant reports and independent criterion variables (e.g., observed behavior, clinician diagnoses). This work will aid in determining how to leverage multi-informant assessments to provide individualized care for youth in foster care settings.

Clinical Implications

Our findings have broader implications for evidence-based assessment with adolescents generally and those in foster care in particular. Findings from the present study suggest that patterns of multi-informant reports can be used by clinicians to generate assessment hypotheses. Clinicians can consider the role of established moderators of reporting patterns, including problem domain and informant dyad. Clinicians should pay particular attention to adolescents’ reports of internalizing concerns when observing low informant correspondence and consider the covert nature of these problems, which may be more likely to be reported by adolescents than other observer informants. In addition, when finding low informant correspondence between informants observing an adolescent in unique contexts (i.e., home, school), clinicians can consider the implications of these patterns of reports (e.g., context-specific manifestations of mental health problems). When interpreting patterns of informants’ reports, clinicians can also consider contextual factors related to the setting and purpose of the assessment. For example, youth living with foster or adoptive caregivers may have greater behavioral concerns or caregivers who are more attuned to problems, which leads to heightened ratings of mental health problems in the home setting (and not necessarily the school setting). Overall, regardless of population, clinicians can leverage empirical work to make informed hypotheses about multi-informant reporting patterns that improve clinical decision-making.

Limitations and Future Directions

Study results should be interpreted in the context of study limitations. First, LCA models were based on discrete indicators using a top 25% cutoff point. We demonstrated that our LCA models accurately reflected the underlying continuous data from which we constructed these discrete indicators (see Footnote 1). Further, although we examined an at-risk youth sample, use of clinical cutoff scores to construct discrete indicators proved to lack feasibility, as they would have resulted in small cell sizes and thus a low-powered method for detecting hypothesized effects. Thus, future work should examine patterns of multi-informant reports using other methods of creating discrete scores (e.g., clinical cutoffs), to determine if the classes we observed are robust to methodological differences in constructing discrete indicators. Second, although established moderators and exploratory clinical variables were examined, our study lacked independent criterion variables that could be examined in relation to the latent class solutions. Future work should examine clinical variables such as these to determine whether multi-informant reports in foster care samples signal important information about independent criterion variables. Third, on average, teacher reports were collected four months after adolescent and caregiver reports. Although the length of time between collection of informants’ reports was not associated with reporting patterns, this aspect of data collection may have impacted study findings. Of note, the ASEBA scales assess the past six months of youth problems and show high test-retest reliability across time,^32–34 suggesting that patterns of findings would likely be relatively stable. Nonetheless, we encourage future research to address study aims in a sample that includes informants’ reports collected within a smaller time interval. Notwithstanding these limitations, the present study advances the literature on use of multi-informant reports and has important implications for assessment among child welfare-involved youth.

Summary

Maltreated youth in foster care are at heightened risk for externalizing and internalizing problems. Prior work raises questions as to the validity of the informants’ reports used to assess this population. Yet, recent research on assessing conditions as diverse as anxiety and mood, autism, and disruptive behavior reveal a discernable structure of cross-informant reports, characterized by discrete patterns of agreement and disagreement between informants. This structure “tracks” contextual changes in displays of behavior and clinical severity. As such, this structure may improve use and interpretation of multi-informant reports in foster care settings. We examined the presence of such a structure across multi-informant reports (i.e., adolescent, caregiver, teacher) of adolescent externalizing and internalizing problems in a sample of adolescents with a history of out-of-home placement. Across problem domains and informants, we observed reporting patterns identified in prior work, or groups of dyads in which: (a) both informants agreed on high/low youth problems, or (b) one informant endorsed problems that the other informant did not endorse. These patterns displayed characteristics robustly present in meta-analytic work on cross-informant correspondence, namely that informant dyads agreed more on reports of externalizing problems than internalizing problems, and caregiver-teacher dyads agreed more than adolescent-caregiver dyads. Robust, replicable patterns of multi-informant mental health reports occur in assessments of adolescents with a history of out-of-home placement, a setting in which the validity of informants’ reports is often questioned. These reporting patterns may facilitate using and interpreting multi-informant evidence-based assessments among youth with child welfare involvement.