Outcome Domains in Child Mental Health Research Since 1996: Have They Changed and Why Does it Matter?

Abstract

Objective

Child mental health treatment and services research yield more immediate public health benefit when they focus on outcomes of relevance to a broader group of stakeholders. We reviewed all experimental studies of child and adolescent treatment and service effectiveness published in the last 15 years (1996–2011) and compared the distribution and types of outcome domains to a prior review that focused on studies from the prior 15 years (1980–1995).

Method

Studies were included if they focused on children from birth to 18 years of age with specific or general psychiatric conditions, employed randomized designs, and examined intervention effects with a six-month or longer post-treatment assessment in treatment studies or a six-month or longer post-baseline assessment for services studies. Two hundred (n=200) studies met criteria. Reported outcome measures were coded into conceptual categories drawn from the 1980–1995 review.

Results

There was a five-fold increase in the total number of studies (38 versus 200) across the two 15-year time periods, with the largest increase in the number of studies that focused on consumer-oriented outcomes (from 8 to 47 studies, an almost 6-fold increase); two new domains, parent symptoms and health-related outcomes were identified. The majority of studies (95+%) continued to focus on symptoms and diagnoses as an outcome. Impact ratings were higher among studies examining four or more outcomes versus one to two outcomes in all categories with the exception of Posttraumatic Stress Disorder.

Conclusions

Given major shifts in healthcare policy affecting mental health services, the emergence of health and parent-related outcomes as well as greater attention to consumer perspectives parallels emerging priorities in healthcare and can enhance the relevance of child outcome studies for implementation in the real world.

The raison d’être for developing effective pediatric mental health treatments and services is to improve children’s mental health outcomes. Yet critical questions about outcomes still elude scrutiny: What constitutes an outcome? How are outcomes conceptualized and assessed? How consistent are outcome domains over time as the evidence base develops? What is the relevance or fit of outcomes to new policy and practice contexts? As the science base on pediatric clinical interventions expands, answers to these questions are needed to provide strategic direction to policy-makers, practitioners, and researchers. This becomes even more important as public demands to move effective interventions out of academia and into the real world increase.

In the first of two papers published in 1996, two authors suggested a conceptual typology, symptoms/diagnoses, functioning, consumer perspectives, environments, and systems (SFCES), as an organizing framework for understanding the emerging knowledge base about child and adolescent mental health outcomes.¹ SFCES included the outcome domains of symptoms and diagnoses; functioning and impairment across settings; consumer-oriented perspectives about their experience, (e.g., beliefs, expectations and attitudes of the youth or caregiver); interpersonal and environmental contextual factors (e.g. parenting practices, school climate); and service-related outcomes (e.g., costs, service use, placements, trajectories).

The second of the two original reports included a review of 15 years of treatment and service studies (1980–1995), and categorized the kinds of outcomes represented in those experimental studies according to the typology.² The impetus for those papers was concern about the narrow focus of pediatric intervention research on a small and limited set of outcomes—largely symptom reduction only—and the likelihood that such a narrow focus might thwart or at least constrain adoption of effective interventions in real world settings.

The present paper is an update of the original review. This review encompasses all experimental and rigorous pediatric treatment and service studies published between 1996 and 2011. Thus both reviews cover a 15-year span. Our primary goal with the current review is to examine changes in the distribution and types of outcomes reflected in these more recent studies and compare these to the original review. Specifically our purpose is to (a) assess whether the distribution of outcomes has changed since the 1990–1995 review; (b) determine whether new outcome types are reflected in the more recent literature; and (c) examine whether studies that include a broader range of outcomes (4 or more) have higher impact ratings than studies with a narrower range (1–2). This last goal is based on a premise that studies encompassing a broader range of outcomes may be more widely read and relevant to the concerns of multiple stakeholders (including consumers and policy makers), thus potentially addressing two impediments to implementation of new practices—lack of awareness and lack of practical applicability.

To provide a context, we first briefly summarize important policy and financing developments in child and adolescent treatment and services since 1996 that may have shaped the focus of the current review. We then explain why a typology of outcomes may be even more useful today than in the past given the current scientific and policy context, which is shaped increasingly by healthcare reform that requires attention to quality indicators and to implementation of effective interventions in healthcare systems.

Policy and Funding Initiatives

A series of Surgeon General’s Reports between 1999 and 2002 including the National Children’s Action Plan,³ followed by the President’s New Freedom Commission Report,⁴ all highlighted the gap between the growing knowledge base on efficacious and effective interventions and their absence in real world practices. Similar themes from the standpoint of healthcare were encapsulated in a series of Institute of Medicine (IOM) reports published between 1998–2011 on the quality chasm in healthcare. A second theme was the lack of integration of mental health and substance abuse services into health services.⁵ All of these documents were unanimous in their identification of these two issues—the science to practice gap and the asymmetrical and non-integrative structure of healthcare—as major barriers to improving the outcomes for adults and children with psychiatric conditions.

To support the need for expansion of the scientific knowledge base, the National Institute of Mental Health (NIMH) between 2001 and 2008 published a series of reports focused on bridging the science and service gap.⁶ These were followed by targeted program announcements and funding from the National Institutes of Health (NIH) and foundations to improve the installation of effective treatments and services in broader systems.⁷

The reports from NIMH, IOM, and the Surgeon General pointed to disparities in access to services for youth and families, particularly the gap between need for mental health services and receipt. Some positive trends suggest that an increase in access has occurred, moving from the mid-1980s (3%) to 1999 (10%) to 2008 (13%).^8–9 This increase in access is suggestive of greater availability of services and perhaps more acceptability among families to seek them—i.e., less stigma. The quality of the services received, however, is uneven at best.^10–11

Financing of mental health services has been largely fueled by federally-supported community mental health system of care grants,¹² which are now shifting away from communities and towards states.¹³ The State Children’s Insurance Program for uninsured youth and Medicaid waivers to states covers intensive community-based services, including family support.⁸ Managed care entities initially covered families with private insurance and now increasingly are managing Medicaid services in a growing number of states.¹² Twenty-five years in the making, the Wellstone and Domenici Mental Health Parity Act was passed in 2008. In 2010, the Affordable Health Care for America Act was signed into law. A major objective of it is to enforce the Mental Health Parity and Addiction Equity Act. Both of these pieces of legislation were enacted to offer affordable mental health care, ensure parity in coverage for behavioral health and physical illnesses, and facilitate the co-location of mental health services in health care settings. Another important development is reimbursement for evidence-based treatment by states, private insurers, and the Centers for Medicaid and Medicare Services.⁸

Undergirding the financing of services has been a solid set of values and principles derived from the system of care movement that began in the mid-1980s.¹⁴ An important offshoot of this movement has been a focus on family-directed (e.g., consumer-oriented) services.¹⁵ This focus is fortunately scaffolded by the existence of family advocacy organizations now operative in all 50 states.^16–18 The increased attention to consumer perspectives, along with mental health parity and more regionalized structures for financing and regulating healthcare¹⁹ have created a new context to support consumer activation in healthcare decisions. This is important for both ethical and fiscal reasons: consumer involvement may in some instances prevent the use of unnecessary and costly medical or psychiatric services.^5,20–21

Growth in Pediatric Intervention Research

Significant growth in the number of pediatric clinical trials on efficacious and effective treatments began around 1991 when NIMH issued a “National Plan for Research on Child and Adolescent Mental Disorders.” This plan helped to shape the clinical treatment and services research agenda with its emphasis on outcome studies. Accompanied by a general increase in NIH funding, this plan led to a tripling of NIMH funding for children’s treatment and services research between 1991 and 2001.³ Importantly this emphasis on strengthening the evidence base on clinical interventions was designed to address the significant gap between the knowledge base on effective interventions for adults with psychiatric issues vs. children.²² The resulting increase in funding for pediatric mental health treatment and services research led to several major multi-site clinical trials—the first of their kind in child mental health. These included the Multimodal Treatment of Attention-Deficit/Hyperactivity Disorder (ADHD),^23–24 the Preschool ADHD trial (PATS),²⁵ followed by the Treatment for Adolescents with Depression Study,²⁶ the Child/Adolescent Anxiety Multimodal Study,²⁷ and Treatment-Resistant Depression.²⁸ All of these studies were notable in advancing knowledge about the comparative effectiveness of medication, psychosocial, or other forms of treatments and services for children and adolescents.

Growth in the number of pediatric treatment and service studies continued until recently, when funding cuts at NIH across the board led to a leveling off of treatment studies.²⁹ Yet there has been a spurt of studies over the past decade fueled by political attention and funding support for science directed at children’s mental health. This body of scientific work has strengthened the evidence for the effectiveness and in some cases cost-effectiveness of clinical interventions for common childhood disorders (i.e., disruptive disorders including ADHD, depression, and anxiety)^30–31 and for intensive home- or community-based interventions which are not diagnosis-specific (e.g., Brief Strategic Family Therapy, Functional Family Therapy, Multisystemic Therapy, Multidimensional Treatment Foster Care).^32–34

Growth in Implementation of Evidence-based Treatments and Services

Since 2006, more than a dozen states,³⁵ healthcare entities (Kaiser, Value Options) and foundations (William T. Grant, Robert Wood Johnson, Annie E. Casey) are focusing policies and new funding initiatives towards improving implementation of evidence-based treatments and services in larger systems.^36–39 However, significant implementation challenges exist,^11,40–42 because of insufficient attention to the natural contexts within which the tested interventions will be embedded. Notably, researchers rely on sets of outcomes to establish intervention impact, but these may be irrelevant or tangential to the outcomes of interest to those responsible for implementing these interventions in the real world.^11,41–43 The lack of fit between the range and types of outcomes assessed by treatment researchers and the outcome expectancies and accountability standards of those responsible for installing the intervention (i.e., agency directors, state policy leaders) can lead to misunderstandings, wasted efforts, inefficiencies, and implementation failures. Thus, failure to specify and measure a range of outcomes that will be important for the intended end-users can impede implementation of effective interventions.

We undertook the present review with specific attention to the types of outcome domains and their distribution in experimental studies of child and adolescent clinical treatments and services. We were interested in whether the types of outcome domains have changed since 1996, when a spurt of funding for a national plan, and renewed policy emphasis sparked growth in treatment and services research. Our assumption is that a comprehensive typology of outcomes can support more successful implementation by providing “truth in advertising,” providing a blueprint by which researchers and adopter constituents can be fully informed about the range of outcomes associated with specific interventions and thereby attend to an intervention’s fit within different contexts.

Method

A comprehensive search of studies of outcomes associated with 1) medication, psychosocial, or combined treatments for specific disorders, and 2) services for general emotional or behavioral problems was conducted. The methodological approach we undertook was identical to our previous review. To locate studies, three of the authors (Hoagwood, Jensen, and Acri) developed a list of search terms within four categories: symptoms/diagnosis, which included both specific terms to capture treatment studies (e.g. anxiety, manic) and more general mental health keywords to locate service studies (e.g. emotion* or disturb*); treatment (e.g. psychotherapy), experimental design (e.g. randomized controlled trial), and follow-up period (e.g. follow-up). Search terms were then shortened to their base word (e.g. manic instead of manic depression), and an asterisk was added to each word to capture all derivatives of that particular term (e.g. treat* to capture both treatment and treated).

Next, search terms within each of the categories were linked with “or,” and larger categories were linked with “and” to capture studies that included at least one search term from all of the categories (e.g., studies of treatments for anxiety with a follow-up assessment point that used an experimental research design). The search was executed using the PsycINFO and Medline (OVID SP) databases through a university library system. Database search limits included year (1996–2011), children and adolescents 0–18 years of age, and type of article (e.g. randomized controlled trial, meta-analysis, controlled clinical trial).

Inclusion and Exclusion Criteria

Treatment and service studies were included if they met the following criteria: (a) published between 1996 and December 2011; (b) used a randomized experimental design; (c) analyses focused on main treatment or services effects; (d) targeted children between 0–18 years of age (studies that included both adolescents and young adults were included if the mean age of the sample was < 18 years of age); and (e) targeted specific psychiatric disorders as described by the Diagnostic and Statistical Manual of Mental Disorders⁴⁴ or general mental health conditions (e.g., severe emotional/behavioral problems). Finally, consistent with the original 1990–1995 review, included studies were required to have a six-month or longer post-treatment follow-up assessment for treatment studies, or a six-month or longer post-baseline data point for service studies.

Studies of primary or universal prevention interventions were excluded. Additionally, interventions for substance abuse were included only if both substance abuse and mental health functioning and symptoms were the focus of the intervention.

Based on the criteria outlined above, we reviewed approximately 1,200 treatment and service studies. Of these, 200 original studies and 224 total articles met criteria and are included in this review.^24,45–267 See Figure 1 for a depiction of our search history, based upon the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).²⁶⁸ The number of articles exceeds the number of studies because in some cases several articles were published from one study. For a PRISMA checklist, see Table S1, available online.

Figure 1.

Search history and results.

Coding and Analysis

Consistent with the original review, treatment and service studies were grouped separately, and treatment studies were further categorized by symptom or diagnosis, co-morbid conditions, or other. Outcome types were coded applying the original SFCES model: 1) symptoms and diagnoses, 2) functioning/impairment, 3) consumer-oriented perspectives, 4) interpersonal–environmental contexts, and 5) services/systems. Two additional constructs were identified in this review: 6) parent symptoms and 7) health. See Table 1 for a definition of each construct.

Table 1.

Description of Constructs

Construct	Definition
Symptoms and Diagnoses	Symptoms refer to emotions or behaviors a child may exhibit in different settings often leading to a formal psychiatric diagnosis according to the DSM-IV.
Parent Symptoms and Diagnoses	Similar to the above, but assigned to the parent or caregiver of the child.
Functioning/Impairment	Refers to the capacity to adapt to the demands of home, school, and community. It is a continuous variable that includes competencies, on one end, and impairment on the other. Functioning includes capacities listed in Axis V of DSM-IV.
Consumer-oriented Perspectives	Refers to perspectives of parent, caregiver, or youth about quality of life, satisfaction with care, family strain or burden, perceptions and beliefs about treatment, sense of competence, and empowerment, as well as knowledge about psychiatric disorders.
Interpersonal and Environmental Contexts	Refers to the broader context that affects children including parenting and the parent/child relationship, parenting styles, family functioning, marital issues, peer-related outcomes, and social-organizational contexts, such as culture and climate. This domain is a counterpoint to functioning as it refers to features of the children’s context that are modifiable.
Services/Systems	Refers to the level, type, duration, or change in use of services, restrictiveness, costs, and financing. It can include suspensions, arrests, changes in out of home placements, length of stay, or use of inpatient hospitalization or residential treatment.
Health	Refers to physical or biological changes in the child as an outcome of treatment or services, such as height, growth, weight loss, or cortisol levels. Does not include health-related factors that are symptoms of mental health disorders (e.g. weight loss relative to eating disorders).

Coding proceeded in multiple stages. First, one of the authors (Acri) reviewed each abstract to determine the study’s eligibility based upon the previously-described inclusion and exclusion criteria. Questionable studies (e.g. studies with an unclear follow-up period) were reviewed with two of the other authors (Hoagwood and Jensen) until consensus was met, and decision rules were established.

Then four authors (Acri, Hoagwood, Jensen, Lewandowski) check coded a random selection of 15 articles to categorize each main outcome. Any questionable outcomes were discussed until consensus was reached. One author (Acri) then coded the remaining articles. A kappa statistic was calculated to assess interrater reliability between all four coders. Agreement was 86%, yielding a free-marginal kappa statistic of .84.

We were also interested in whether studies that included a broader range of outcomes had higher impact ratings than studies with a narrower range. Therefore, we examined the number of times articles that examined one or two outcomes vs. four or five outcomes were cited, based upon the Web of Knowledge Platform. Of the 224 articles in our review, 142 examined one or two outcome domains, and 38 studied four or five, yielding a total of 180 articles. However, seven articles from the one to two outcomes group could not be located via Web of Knowledge, leaving a final subsample for the impact analysis of 173 articles (135 examining one to two outcomes, and 38 examining four or five outcomes).

Results

Changes in Number of Studies and Disorder Types

The original review, spanning 15 years from 1980 to 1995, identified 38 treatment or service studies for children and adolescents. By contrast, the present review, spanning the same number of years from 1996 to 2011, identified 200 studies. This represents a 426% or four-fold increase in the total number of studies. The original review identified six primary diagnostic areas as targets for the interventions: 1) behavior problems (conduct disorders, oppositional defiant disorders), 2) attention-deficit hyperactivity disorder, 3) anxiety, 4) depressive, 5) autistic, and 6) eating disorders. Among service studies the categories included therapeutic foster care,²⁶⁹ day treatment,²⁷⁰ and systems of care.²⁷¹ In the current update, additional diagnostic categories emerged: 1) comorbid conditions (n=12),^169,199 including mental health and substance abuse disorders⁷⁷ and mental health/intellectual functioning;^45,125 2) posttraumatic stress disorder and associated symptoms (n=11);^82,87,161 3)bipolar disorder (n=1); ¹⁷⁷ 4) personality disorder (n=1),¹⁶² and 5) an “Other” category (n=5) that included Tourette’s Syndrome, tics,²⁶⁴ and risk of self-harm.¹²³ Consistent with the prior review, service studies focused on systems of care^65–66 and residential treatment.²⁶¹ Additionally, a set of studies by Henggeler and colleagues concentrated on the effects of Multisystemic Therapy versus usual care delivered in hospital,^133,137 outpatient substance use,^130,132 and juvenile justice settings.¹³¹ Thus, in comparison to the 1980–1995 review, the current review reveals an expansion of both diagnostic targets and service interventions across settings.

Changes in Outcome Domains

The most common outcome domain assessed across all studies in the current review (as with the original review) was child symptoms and diagnoses accounting for 191 (95.5%) of the total number of studies. This was followed by functioning/impairment (n=95, 47.5%), services and systems (n=61, 30.5%), consumer-oriented perspectives (n=47, 23.5%) and interpersonal-environmental contexts (n=46, 23%). Two new domains emerged: Parent symptoms, specifically related to depression or anxiety (n=21, 10.5%); and health-related outcomes (n=6, 3%) (e.g., height, weight, cortisol levels).

Figure 2 shows the total number of studies in the two reviews (N=238) that targeted all seven domains. The percentage changes in absolute growth of studies across all domains in the two reviews are presented.

Figure 2.

Growth in studies of outcome domains in child psychiatric randomized controlled trials, 1980–2011 (N=238).

For comparative purposes, Figure 3 graphically presents the proportion of studies from the 1980–1995 and the 1996–2011 reviews that examined each outcome domain, using the total number of studies in each cohort as the denominator. Virtually all studies examined child symptoms/diagnosis outcomes across both reviews, while the relative proportion of studies examining functioning/impairment, interpersonal/environmental, and services/systems domains dropped substantially from the first to the second review. Chi square analysis indicated that these proportional differences in outcome domain distributions were non-random.

Figure 3.

Percent of studies examining 7 outcome domains from 1980–1995 vs. 1996–2011 (N=238).

Impact Factor Ratings

To investigate whether including a broader range of outcomes was associated with higher impact ratings, we examined the impact factors of articles that targeted one or two outcome categories vs. those that targeted four or more. Articles that included one or two outcomes were cited 23.4 times on average; in contrast, articles that studied four or more outcome categories were cited 44.5 times. Impact ratings were higher among studies examining 4 or more outcomes in all categories with the exception of PTSD (only 1 study with 4 or more outcomes) and services (56.45 vs. 44.33). Eating disorders, bipolar, and personality disorders had no studies with 4 or more outcomes. Among the group targeting 4 or more outcomes, ADHD had the highest mean impact rating (M=131.5) followed by autism (M=98.5) and anxiety (M=70.2). Table 2 provides a breakdown of the average number of citations overall, and by diagnosis or services.

Table 2.

Impact Ratings for One/Two vs. Four/Five Outcome Domains

Category	# of articles (1–2)	Mean times cited (1–2 outcomes)	# of articles (4+)	Mean times cited (4 + outcomes)
Attention Deficit Hyperactivity Disorder	7	37.71	2	131.50
Anxiety	35	30.17	5	70.20
Autism	3	17.67	2	98.50
Disruptive Behaviors	33	28.15	17	37.16
Bipolar	1	0	0	0
Co-morbid	6	18.50	4	42.75
Depression	20	41.45	2	51.50
Eating Disorder	9	16.63	0	0
Other	3	5.67	2	53.50
Personality Disorder	0	0	0	0
Post-Traumatic Stress Disorder	7	28.43	1	4.0
Service Effectiveness for Emotional/Behavioral Problems	11	56.45	3	44.33
Total	135	23.40	38	44.45

To look more closely at those articles that included a range of outcomes, we examined those that included four or more. This subgroup of 38 articles is summarized in Table 3. Types of outcomes in this subgroup included child symptoms/diagnoses (100%), followed by consumer-oriented (n=32, 84%) and interpersonal-environmental (n=31, 82%). Functioning/impairment (n=22, 58%), parent symptoms (n=19, 50%) and services/systems (n=16, 42%) were also included but less often. Four (10.5%) targeted health outcomes.

Table 3.

Articles With Four or Five Outcome Domains

Focus of Study	Symptoms/Diagnoses	Parent Symptoms	Functioning/Impairment	Consumer Perspectives	Interpersonal/Environmental	Services/Systems	Health
Attention-Deficit/Hyperactivity Disorder
1. MTA Cooperative Group (2004a)183	X		X		X	X
2. MTA Cooperative Group (2004b)24	X		X		X	X	X
Anxiety
3. Barrett et al. (2004)57	X	X		X	X
4. Barrett et al. (2005)56	X	X		X	X	X
5. Cobham et al. (1998)86	X	X	X	X	X
6. O’Leary et al. (2009)96	X	X		X	X
7. Silverman et al. (1999)225	X	X	X		X
Autism
8. Kasari et al. (2008)144	X		X		X	X
Comorbid
9. Azrin et al. (2001)50	X		X	X	X	X
10. Bor et al. (2002)69	X	X		X	X
11. Rickards et al. (2009)208	X		X	X	X
12. Roberts et al. (2006)210	X	X		X	X
Depression
13. Clarke et al. (2005)83	X		X	X		X	X
14. Kolko et al. (2000)151	X		X	X	X
Disruptive Behaviors
15. Berkovits et al. (2010)60	X			X	X	X
16. Bradley et al. (2003)71	X	X	X		X
17. Bywater et al. (2009)74	X	X		X	X	X
18. Patterson et al. (2002)202	X	X	X	X
19. Gardner et al. (2006)117	X	X	X	X	X
20. Hutchings et al. (2007)139	X	X		X	X
21. Larsson et al. (2009)156	X			X	X	X
22. Lavigne et al. (2008)158	X			X	X	X
23. Markie-Dadds et al. (2006)171	X	X		X	X
24. Martsch et al. (2005)172	X		X		X	X
25. Nickel et al. (2005)191	X		X	X			X
26. Nickel et al. (2006)192	X		X	X			X
27. Nixon et al. (2004)194	X	X		X	X
28. Sanders et al. (2000a)216	X	X		X	X
29. Sanders et al. (2000b)217	X	X		X	X
30. Sanders et al. (2007)215	X	X		X	X
31. Tucker et al. (1998)243	X		X	X	X
32. Webster-Stratton et al. (1997)254	X		X	X	X
Post-Traumatic Stress Disorder
33. Deblinger et al. (2006)91	X	X	X	X	X
Other
34. Weiss et al. (1999)258	X		X	X		X
35. Weiss et al. (2000)257	X		X	X		X
Service Effectiveness for Emotional/Behavioral Problems
36. Bickman et al. (1997)66	X		X	X		X
37. Bickman et al. (1999)65	X		X	X	X	X
38. Harrington et al. (2000)126	X	X		X	X	X

Discussion

Our review documents a huge increase in the number of child and adolescent controlled clinical trials since 1996: From 38 total experimental studies of treatments or services between 1980 and 1995 to 200 from 1996 to the present or a 5-fold increase. The growth in knowledge about effective treatments and services for children has been substantial and broad. Thus there now exists a wealth of rigorous research-based knowledge to guide clinicians, payors, policy-makers, and families as they select the type of treatment or service for the most common pediatric problems. Furthermore, through the lens of this classification system, a rational and considered judgment about which treatment or service to provide or to seek (as a family) can be made on the basis of the type of outcome that is sought.

There are several possible reasons for this rapid expansion in pediatric clinical trials. Some of it may have stemmed from growth in pharmaceutical industry trials, subsequent to the pediatric exclusivity option that the Food and Drug Administration (FDA) allowed in 2002. This enabled the industry to retain patent rights on pediatric drug agents for an additional 6 months after going to market, translating into huge potential profits. This rule may account for some of the increase in studies of ADHD and major depressive disorder.

However, most of the studies reflected in the increase are not of medication therapies but rather of psychotherapy or combined trials. Thus the expansion is better understood as a combination of federal leadership and funding initiatives, greater public awareness and demand for research on childhood psychiatric problems, increased availability of diagnostic tools, reduced stigma, support from family advocacy organizations for more research, and the fact that the field was very small to begin with.

The outcome domain showing the largest expansion was consumer-oriented perspectives, from 8 studies in the first review to 47 studies now—almost a 6-fold increase. This expansion also came at a time of increased federal emphasis on patient-centered health care (e.g., the consumer as “true north”²⁷²), growth in the mental health family advocacy movement,¹⁶ the systems of care movement^14,273 and a growing recognition among health care policymakers that health care reform depends in part on consumer ownership.^15,19

A surprising and disappointing finding in the review was the decrease in the proportion of studies of services and of studies targeting service-related outcomes. In contrast to the relative expansion of the consumer-oriented domain, studies targeting service system outcomes decreased from 19 of 38 (50%) in the original review to 61 of 200 (31%) in the present review. Furthermore, studies of service effectiveness decreased from 21 studies in the original review to only 12 in the current review, an absolute decrease of 43%.

We can only speculate on the reasons for this. It may be a result of a de-emphasis by granting agencies such as NIMH on service effectiveness studies in favor of basic science studies, and the migration of established services researchers to substance abuse, when the National Institute on Drug Abuse’s (NIDA’s) priorities shifted in the early to mid 2000. The introduction of the dissemination-implementation research agenda within NIH also enabled a number of services researchers to re-engineer their scientific work so that it would be relevant to the new agenda on implementation of effective treatments or service practices, including fidelity, supervision and training.^11,22,274

There was also a significant decrease in the proportion of studies targeting interpersonal-environmental contextual outcomes, from 23 out of 38 to 46 out of 200, or 61% to 23%. This too may be a function of the de-emphasis by funding agencies on contextual outcomes, which are more difficult to measure, more difficult to change, and that often require additional resources.

Expansion of Outcome Domains and Outcome Variables

It is noteworthy that new outcomes of interest have emerged in the past 15 years, specifically parent symptoms and health. Parental symptomatology, long-recognized as a crucial component of child well-being, has been explicitly identified and targeted in more recent studies of children’s mental health. Similarly, health-related medical or biological variables have been the focus of attention, along with mental health symptoms, in a few studies looking at overall health changes. The advent of health-related outcomes is especially propitious given the renewed attention to the interface between primary care and behavioral health.¹⁹

Consumer outcomes were assessed in a third of all studies, a huge expansion of interest in this domain. Within the consumer-oriented domain, many new variables and metrics have been identified or more finely partitioned, in part because of input from consumer advocacy groups²⁷⁵ and also because of new findings informed by basic cognitive and affective science about factors contributing to children’s overall well-being. These include (a) perceptions and beliefs about treatment, service engagement, satisfaction with services, parental sense of competence, empowerment (n=46, 33.6%); (b) parental stress or strain (n=20, 14.9%), (c) quality of life (n=4, 2.99%), and (d) knowledge about psychiatric disorders (n=4, 2.99%). The parental experience of raising a child with psychiatric issues raises a host of feelings including grief, loss, and changes in expectations about the child’s and parent’s future. The expansion of studies tapping into consumer experiences also reflects an acknowledgement that parent, youth, and family perspectives are critical to improving the context for service delivery—moving beyond mere symptom or functioning improvement to reflect outcomes of importance to consumers about their overall healthcare experiences. We expect this domain will show increasing research interest, as the knowledge base expands on the complex interplay between clinical symptom/functioning outcomes and parent and youth attitudes, beliefs and expectancies.

Our review tallied the number of studies focusing on specific disorder areas. Surprisingly, studies of ADHD, the most common child psychiatric disorder, had very few studies, possibly because of a perceived saturation of studies about ADHD subsequent to the NIMH multisite Multimodal Treatment of Attention-Deficit/Hyperactivity Disorder (MTA) study.²³ However, studies of depression, anxiety, and behavior problems showed large increases, perhaps reflecting increased NIH interest spurred by multi-site trials (Treatment for Adolescents with Depression Study [TADS], Child/Adolescent Anxiety Multimodal Study [CAMS], Treatment of SSRI-resistant Depression in Adolescents [TORDIA], etc.). There were 26 depression studies, up from 3 in 1996, likely reflecting improvements in screening tools, availability of feasible measures of adolescent depression for use in community settings, increased recognition of depression as a common reason for school health referrals,²⁷⁶ and acknowledgement that as with adult depression, it is treatable. Studies examining outcomes of anxiety disorders grew from 3 studies in the original review to 46 studies by 2011. Studies of disruptive behavior problems showed the largest absolute growth, from 6 studies between 1980–1995 to 57 studies between 1996 and 2011. Most of these studies used behavior checklists to identify non-diagnosis specific behavior problems.

Range of Outcome Domains

Even though a larger proportion of studies in the current review include outcome categories beyond symptoms and functioning, it is still the case that the majority of studies only examine these two domains. This suggests that our understanding of outcomes is still largely driven by mental health clinicians’ perspectives and needs. Our understanding of other perspectives—policymakers, parents, children (as patients), and others in the environment (teachers, peers) remain rudimentary. So if, as the IOM suggests, healthcare is to be “owned” by the patient/consumer, one would expect that the ultimate aim of healthcare outcomes should be shaped by attention to family and youth perspectives.

Although the addition of new outcome domains may increase the relevance of research findings to consumers and families, new domains raise new problems: how are the domains correlated (if at all), and which domains are most important, when, and why? How should clinicians, healthcare policy makers, and tax-payers grapple with the complex decisions about resources, particularly when treatments yield improvements in one key outcome domain but not in another? Thorough review of these challenges is beyond the scope of this paper, but this type of problem is familiar to clinicians and researchers accustomed to conducting multi-method/multi-trait/multi-informant assessments of children. When the child’s symptoms and impairment are the key outcome domains, different viewpoints about the child’s problems can usually be reconciled. But with the addition of whole new outcome domains (e.g., clinician-defined vs. consumer-defined vs. policy-maker/payer-defined) the stakes may become much higher, with potentially misaligned, contradictory, or even competing outcomes. Unless clinicians, families, policy-makers and researchers acknowledge, measure, study, and seek to understand these new domains together—and develop fully participatory processes for conducting these studies—the yield from the expansion of outcomes will devolve into ambiguity and lost opportunity to make children’s lives better.

New approaches to measurement development promote a more personalized, participatory, and consumer-driven approach. Weisz et al.⁴² developed and tested a “Top Problems” using a simple metric which taps into the top problems that parents and youth see as the most important to them. Following a diagnostic interview, both children and parents list the problems that are the most concerning to them, and then provide a severity rating (e.g. how much of a problem is it). Then they create a list of their top three problems based on the ranking. The majority of problems identified by caregivers and children matched items on the Child Behavior Checklist and Youth Self Report, suggesting that this client-directed approach is clinically sound.⁴²

Although we attempted to replicate our previous methodology by using the same databases, keywords, and coding, we acknowledge the possibility that the two approaches were not completely identical. It is possible that our current review was more comprehensive, as the three co-authors (Jensen, Hoagwood and Acri) created a thorough list of terms that were modified and rerun several times to ensure we were capturing relevant articles. Further, given that we wanted to replicate the original review, we excluded various additional databases, including the Education Resources Information Center and Web of Knowledge that may have yielded additional studies.

In addition, we recognize that the findings in Table 2 that describe linkages between research papers’ citation counts and the number of outcome domains may confound the relationship between range of outcomes and impact ratings with other factors. We cannot know whether the relationship is cause–effect (unlikely) or is simply a correlation due to some third variable (e.g., study size, funding). Nevertheless it is striking that only 38 of the 200 studies examined 4 or more outcomes, and yet these studies had almost twice the number of citations on average as studies reporting only 1 or 2 outcomes.

For future studies, we cautiously suggest that our field’s next 15 years of child mental health outcome research may benefit by more deliberate examination of the range of outcomes of interest to additional stakeholder groups—not just clinicians and researchers, but also healthcare planners and policy-makers, parent and family advocates, economists, community leaders, teachers, general healthcare providers, etc. To address this broader array of outcomes, researchers will need to obtain input from additional stakeholders to build expanded assessment domains into study design. Furthermore, given various stakeholder perspectives that need to be aligned for an evidence-based practice to be implemented (i.e., trained and willing clinicians, payors, and motivated consumers), inclusion of a broader array of outcome domains may become a sine qua non for evidence-based practice dissemination.

Over the last 15 years our understanding of treatments and services that improve children’s outcomes has become much more elaborated, sophisticated, and nuanced. Our review suggests that improving the quality of services and service systems to support and treat children and families requires a broader public health approach than can be provided by sole reliance on assessment of symptoms, diagnoses, and functioning. More comprehensive outcome assessments are especially critical given new healthcare accountability reforms and pressures on service providers and payors to focus on outcomes and to be held accountable for achieving them. These pressures in turn put additional demands on researchers to develop and test interventions that will not only improve symptoms and functioning but that are cost-effective (service/systems domain), palatable to families (consumer domain), affect the contexts for children so are likely to be sustained (interpersonal–environmental), and affect not only mental health but general health (health domain) as well.

Because some outcomes are likely to be more personally meaningful for parents and families, attention to these kinds of outcomes may help to construct a more ethical science. Parents’ hopes for their children generally extend well beyond mere symptom improvement. Similarly, from a public health and dissemination standpoint, if costs of services are not associated with other outcomes of relevance to consumers and to providers, the services will not be used. Consequently promoting inclusion of policy-relevant, parent and youth-focused, and contextual outcomes are likely to make research findings more responsive to healthcare system changes as well as reinforce an ethical foundation.

Unfortunately, our review indicates persistent unevenness in the types of outcomes targeted by clinical and services researchers, with a continuing skew towards symptoms and functioning only. In addition, little attention has been paid to disorders with potentially life-long disabilities, such as bipolar disorder, comorbidities, and PTSD. The 43% decrease in service effectiveness studies is especially problematic, given the increased need for practical findings that can improve the efficiency and quality of services and the unprecedented rapidity with which healthcare plans and states are revamping their systems to integrate behavioral health.¹⁹

While inclusion of consumer-relevant outcomes is growing in acceptance, most studies still tend to neglect outcome domains other than symptoms and functioning, even though inclusion of a wider range of outcomes incites more attention by more diverse groups of stakeholders, including payors, policy-makers, purchasers, and the consumer advocacy community. Until researchers more comprehensively examine the impact of interventions across a broader array of outcomes, the results of such studies will remain tangential.

We suggest that to craft an ethical, relevant, and usable science base, a broader group of stakeholders should work more assiduously together to ensure that the endpoints and outcomes of interest—implementing effective interventions in the real world to reach the greatest number of children in need— shape the selection of outcomes of key studies, especially those that are tax-payer funded. In this way the self-correcting lens of science can be focused on maximizing the utilitarian principle of the greatest good for the greatest number and ensure public health impact for the broadest population of children and families.

Clinical Guidance.

• Research on outcomes associated with child mental health treatments and services has identified 7 domains. These include symptoms and diagnoses; parent symptoms; functioning; consumer-oriented perspectives; interpersonal and environmental contexts; services and systems; and health. In some cases it may be valuable to select treatments and services that target a broad group of outcome domains to maximize impact.

• Consumer-oriented outcomes are a rapidly growing area of study. These types of outcomes are particularly relevant to healthcare policy changes.