Collaborative Care Outcomes for Pediatric Behavioral Health Problems: A Cluster Randomized Trial

Abstract

OBJECTIVE:

To assess the efficacy of collaborative care for behavior problems, attention-deficit/hyperactivity disorder (ADHD), and anxiety in pediatric primary care (Doctor Office Collaborative Care; DOCC).

METHODS:

Children and their caregivers participated from 8 pediatric practices that were cluster randomized to DOCC (n = 160) or enhanced usual care (EUC; n = 161). In DOCC, a care manager delivered a personalized, evidence-based intervention. EUC patients received psychoeducation and a facilitated specialty care referral. Care processes measures were collected after the 6-month intervention period. Family outcome measures included the Vanderbilt ADHD Diagnostic Parent Rating Scale, Parenting Stress Index-Short Form, Individualized Goal Attainment Ratings, and Clinical Global Impression-Improvement Scale. Most measures were collected at baseline, and 6-, 12-, and 18-month assessments. Provider outcome measures examined perceived treatment change, efficacy, and obstacles, and practice climate.

RESULTS:

DOCC (versus EUC) was associated with higher rates of treatment initiation (99.4% vs 54.2%; P < .001) and completion (76.6% vs 11.6%, P < .001), improvement in behavior problems, hyperactivity, and internalizing problems (P < .05 to .01), and parental stress (P < .05–.001), remission in behavior and internalizing problems (P < .01, .05), goal improvement (P < .05 to .001), treatment response (P < .05), and consumer satisfaction (P < .05). DOCC pediatricians reported greater perceived practice change, efficacy, and skill use to treat ADHD (P < .05 to .01).

CONCLUSIONS:

Implementing a collaborative care intervention for behavior problems in community pediatric practices is feasible and broadly effective, supporting the utility of integrated behavioral health care services.

What’s Known on This Subject:

Integrated or collaborative care intervention models have revealed gains in provider care processes and outcomes in adult, child, and adolescent populations with mental health disorders. However optimistic, conclusions are not definitive due to methodologic limitations and a dearth of studies.

What This Study Adds:

This randomized trial provides further evidence for the efficacy of an on-site intervention (Doctor Office Collaborative Care) coordinated by care managers for children's behavior problems. The findings provide support for integrated behavioral health care using novel provider and caregiver outcomes.

Gaps in the availability and impact of specialty mental health care and the increasing public health significance of untreated mental health problems have expanded the service delivery roles of pediatric primary care providers (PCPs). Recent models for enhancing mental health services in primary care^1–4 include outside psychiatric assessment and telephone consultation,⁵ collaborative peer consultation,^6–8 mental health assessment skills training,⁹ and collaborative care interventions.^10,11 As suggested in a recent review, these studies have revealed progress in improving provider care processes (eg, medication for attention-deficit/hyperactivity disorder [ADHD]; mental health assessment) and child symptoms (eg, ADHD, depression), but also call for larger and more rigorous trials.¹²

The Services for Kids in Primary-care (SKIP) treatment research program (www.skipprogram.org) integrates personalized behavioral health services in practice settings serving pediatric patients. An initial randomized trial evaluated a protocol for on-site nurse-administered intervention (PONI) relative to enhanced usual care (EUC) in children with behavior problems.¹³ PONI involved co-location of a nonmental health nurse trained as a care manager (CM) to implement a modular intervention (eg, parenting, child social skills, family problem solving, and communication) with minimal PCP involvement. PONI was superior to EUC in improving service use, child health and individualized behavioral targets, and satisfaction, but both groups showed significant gains on other clinical outcomes. Participating PCPs desired a broader, more interactive, and flexible delivery system.

A second SKIP study sought to enhance the clinical efficacy of PONI by adapting the chronic care model to develop a more collaborative approach (Doctor Office Collaborative Care; DOCC). Mental health clinicians were trained as CMs to administer an expanded set of content modules to manage child anxiety (eg, monitoring, relaxation) to support ADHD medication management in collaboration with the PCP. A pilot study documented the feasibility, fidelity, and acute impact of DOCC for behavior problems, as well as comorbid ADHD and anxiety, relative to EUC.¹⁴ However, the study’s scope, sample size, and methods (eg, PCPs were randomly assigned, not practices) were limited.

Using PCP and family feedback, the content and care processes in DOCC were expanded to better address the principles of the chronic care model in the current study (Table 1). DOCC incorporated participatory management for soliciting staff and family input, an expanded curriculum for the management of ADHD and anxiety, training for PCPs in the ADHD care management protocol, and technology-guided assessment and consultation procedures. This effectiveness trial evaluates the benefits of this expanded DOCC model in 8 pediatric practices that were cluster randomized to DOCC or EUC. We hypothesized that DOCC would be associated with gains in service use, child and parent mental health outcomes, and consumer satisfaction, and greater change in pediatrician’s treatment attitudes and practices.

TABLE 1.

Adaptations of the Chronic Care Model in DOCC

Goal	Function	Agent	Task/Activity
Leadership team (organizational partners)
Promote service mission	Develop practice-based research network	Team	Create administrative structure with all practices and consultants.
Share in governance	Establish executive committee	Team	Hold meetings to review agenda and make decisions.
Decision support (access to specialists, validated treatments)
Announce study and conduct screening	Identify and enroll eligible cases	All	Flyers and signs to publicize study. Case finding by using brief 7-item screen (PSC-17 externalizing scale score; ≥5).
Assess/diagnose key patient/family problems	Document clinical problems and service needs	CM	Brief diagnostic interview and rating scales to individualize care. Screen for parent distress/conflict (PHQ, DAS-4).
Apply evidence-based treatment of behavior problems and, as needed, anxiety	Teach skills in brief content modules to target child problems	CM, PCP	Administer intervention to parents (eg, parenting practices, anger control) and children (eg, social/academic skills,20 self-monitoring, relaxation training).
Administer ADHD care management regimen	Collaborate with PCP to assess, manage, and monitor ADHD	CM, PCP	CM reviews Vanderbilt ratings, coordinates PCP medication prescription for ADHD with input from PC, conducts follow-up visits, and reviews response with PCP.
Track outcomes	Monitor case progress and alter treatment	CM	Maintain contact to identify response and clinical concerns (IGAR, VADPRS).
Delivery system design (implement and coordinate care)
Establish patient registry (recruitment, monitoring, and personalization)	Obtain referrals from PCP; collect data from informants.	PCP, CM	Obtain individualized goal achievement ratings, document clinical progress and consider need for referral.
Facilitate service initiation and retention	Prepare CM to deliver and coordinate care	CM	Use of on-site visits, telephone, Internet, and manuals to apply materials. Coordinate with PCP, PC, and providers to address priorities (ADHD, then ODD).
Promote practice capacity to participate in treatment	Train all PCPs and staff	CM, PCP	Conduct in-service trainings, especially on ADHD care management.
Incorporate PCP feedback	Use participatory management to obtain feedback	PCP, Team	Conduct focus groups every 6 mo and make changes, as needed.
Clinical information systems (technology, communication)
Monitor treatment and response	Track progress with automated data collection tools (eg, tablet PC)	CM	Use PSC-17 for screening15 and other scales (IGAR, VADPRS) to document outcome over time and alter care plans.
Establish linkage with PCP	Review clinical status/outcome	CM	Give feedback to PCP (meeting, telephone), discuss problem cases; in 4–6 wk, and change treatment.
Conduct routine case reviews	Case supervision and quality monitoring	CM	Regularly review patient outcomes with supervisor and PCP (weekly case review; calls; notes/records).
Establish linkage with PC	Communicate with PC	CM, PC	Review services log; other monitoring materials; discuss challenges.
Establish linkage to specialty care	Communicate with provider	CM, PC	Review monitoring forms and therapy, consults, etc).
Self-management support (help family understand disorders/options)
Educate/destigmatize	Enhance self-management via psychoeducational materials	CM, PCP	Educate child and caregiver about diagnosis and treatment options (eg, AACAP Facts for Families, AAP ADHD background)
Monitor patient status	Communicate with patient via practice	CM, PCP	Arrange PCP visit for ADHD and monitor status.
Monitor patient status	Communicate with patient through MH specialty linkage	CM, PCP	Review referral for alternative treatment. Encourage/facilitate follow-up visits.
Provide assessment and discharge reports	Enhance family self-management	CM, PCP	Use brief motivational interview. Select individualized target and goals.
Community resources (outside services to assist patients)
Coordinate care with mental health specialist	Maintain working relationships with local providers	CM, MHS, PCP, PC	Update provider list by region, insurance, and specialty areas. Refer or review status of cases needing other services.

AACAP, American Academy of Child and Adolescent Psychiatry; AAP, American Academy of Pediatrics; DAS-7, Dyadic Adjustment Scale 7; MH, mental health; MHS, mental health services; ODD, oppositional defiant disorder; PHQ, Patient Health Questionnaire.

Methods

Settings and Participants

Practices

Study sites included 7 Children’s Community Pediatric practices and 1 general academic pediatric practice affiliated with Children’s Hospital of Pittsburgh. This study was approved by the University of Pittsburgh’s institutional review board. All PCPs and parents/legal guardians provided informed consent, and children provided assent.

Providers

A total of 74 of 75 available PCPs consisting of physicians (n = 67), certified nurse practitioners (n = 6), and physician assistants (n = 1) participated in the study. Most were women and white, with ages from 29 to 69. All but 2 were specialty-certified, and 29% had additional fellowship experience (Table 2), virtually all of which were in pediatrics or a pediatric subspecialty (eg, ambulatory pediatrics, pediatric environmental health).

TABLE 2.

Baseline Characteristics of the PCPs and Families in Both Conditions^a

	DOCC	EUC	Pb
PCP characteristics	n = 31	n = 43
Age in years, mean (SD)	47.4 (10.8)	46.0 (9.1)	.55c
Years worked in field, mean (SD)	12.8 (11.8)	12.0 (9.8)	.78c
Number of families enrolled, mean (SD)	6.6 (6.2)	4.7 (4.3)	.13c
Number of families referred, mean (SD)	29.9 (19.1)	23.6 (24.8)	.24c
Gender, woman	16 (51.6)	26 (60.5)	.45
Minority race or ethnic group	3 (10.3)	5 (12.5)	.99
Role			.22
Physician	29 (93.5)	38 (88.4)	—
Physician assistant	1 (3.2)	0 (0.0)
Nurse practitioner	1 (3.2)	5 (11.6)	—
Board certified	28 (96.6)	40 (97.6)	.99
Rotation completed during clinical training	24 (80.0)	34 (82.9)	.75
Fellowship/advanced certification completed	5 (17.9)	15 (37.5)	.08
Child/family characteristics	n = 160	n = 161	—
Age in years, mean (SD)	7.8 (1.9)	8.2 (2.0)	.07c
Gender, woman	59 (36.9)	55 (34.2)	.61
Race or ethnic group			.48
White	127 (79.4)	120 (74.5)	—
Black	26 (16.3)	30 (18.6)
Multiple	7 (4.4)	9 (5.6)	—
Unknown	0 (0.0)	2 (1.2)	—
Diagnosis (met research criteria)
ADHD	100 (62.5)	106 (65.8)	.53
ODD	62 (38.8)	68 (42.2)	.53
CD	2 (1.3)	6 (3.7)	.28
Anxiety disorder, any	25 (15.6)	26 (16.1)	.90
Affective disorder, any	1 (0.6)	0 (0.0)	.50
Elimination disorders, any	16 (10.0)	15 (9.3)	.84
VADPRS–ADHD symptoms, mean (SD)	29.2 (11.7)	29.1 (11.5)	.91c
VADPRS–ODD symptoms, mean (SD)	13.9 (5.3)	13.6 (5.3)	.66c
VADPRS–CD symptoms, mean (SD)	3.8 (3.5)	4.4 (3.7)	.13c
PedsQL total, mean (SD)	74.8 (12.5)	75.3 (11.9)	.69c
Married parents	104 (65.8)	99 (61.9)	.46
Number of children living in home, mean (SD)	1.5 (1.1)	1.7 (1.1)	.15c
Parent with some college education	134 (84.3)	115 (71.9)	.01
Social assistance, any kind	62 (39.0)	70 (21.9)	.39
Family’s practice, high diversity	83 (51.9)	70 (43.5)	.13
Family’s practice, previous collaborative care	72 (45.0)	104 (64.6)	.00

CD, conduct disorder; ODD, oppositional defiant disorder.

^aData are No. (%) unless otherwise indicated. Numbers do not always sum to group totals because of missing information.

^bAnalysis by χ² unless otherwise indicated.

^cAnalysis by t test for means.

CMs

Four Masters-level social workers with previous experience in outpatient or residential treatment were hired by the study to serve as CMs. They were trained over 4 months to deliver each treatment condition, and were supervised by a senior clinician with input from the study child and adolescent psychiatrist. Each CM was assigned to 2 practices (1 per condition each) and worked 2 days per week per practice.

Patients

Participating children (n = 321) were mostly boys and white. Ages averaged 8.0 years (Table 2). Most had a primary diagnosis of ADHD (64%) or disruptive behavior disorder (41%); 16% had comorbid anxiety disorder. Few (10%) participants received ADHD medication. Almost half received social assistance (eg, food stamps).

Screening and Recruitment

CMs conducted telephone screens by using the Pediatric Symptom Checklist 17 (PSC-17¹⁶) with caregivers of 5- to 12-year-old children referred by PCPs for behavior concerns, and invited those meeting the clinical cutoff (≥6th or 75th percentile) on the externalizing behavior subscale for an intake. Parents and children completed self-reports and clinical interviews identifying exclusions related to diagnosis (eg, bipolar disorder), emergent symptoms (eg, suicidal intent), or parallel treatments. Of 787 children referred for study consideration, 576 completed the screening procedures and met initial eligibility criteria, 353 completed a baseline assessment at intake, and 321 who met inclusion and no exclusion criteria agreed to participate and be randomly assigned (Fig 1). Randomization status was revealed after assessment.

FIGURE 1.

Flow of family participants in the intervention trial.

Intervention Conditions

Because 4 of 8 practices were involved in previous outcome studies, practices were stratified by previous participation (no versus yes) and level of patient diversity (low versus high) before cluster-based randomization by the statistician. In both conditions, CMs contacted parents after baseline to identify individualized targets, review findings and treatment recommendations, provide brief psychoeducation, and discuss questions. Both parents and PCPs received written evaluation summaries. The clinical supervisor monitored the integrity of the intake, case presentation, and treatment delivery procedures by reviewing all completed assessments and progress notes on a weekly basis and listening to periodic treatment session audio files. All treatment fidelity feedback was reviewed with the CM each week, and specific suggestions were made to address any questions or performance issues (eg, further role plays, review of materials). Supervisor records indicated that >90% of all CM-delivered sessions received the highest overall fidelity rating on a 4-point scale (1 = poor/incomplete; 4 = very good/complete).

DOCC

These practices offered on-site behavioral health services delivered and/or coordinated by CMs with PCP involvement by using content modules for behavior problems, ADHD, and anxiety¹³ (Table 2). Most of the content modules targeting behavior problems were adapted from an evidence-based treatment, Alternatives for Families: A Cognitive Behavioral Therapy (www.afcbt.org), designed for families presenting with child behavior problems^13,17,18 and/or exposure to physical abuse/discipline.^13,19–21 These primary topics were reviewed with all caregivers (eg, psychoeducation, managing stress, promoting positive behavior, home programs) and children (eg, anger control, social skills). As applicable, the ADHD care management module incorporated behavioral and medication guidelines from the American Academy of Pediatrics^1,2 (eg, rating scales, medication titration, monitoring of symptoms and side effects) that the CM reviewed with children and caregivers. The PCP was also directly involved with the CM and family in administering ADHD medication. For children with anxiety and fears, we incorporated cognitive behavioral therapy methods from a manual developed for primary care²² (eg, self monitoring, relaxation).

The intervention was designed to be delivered in a minimum of 6 and a maximum of 12 individual (child, caregiver) or joint/family sessions and within 6 months. Each session began with a review of the status of the child’s primary target behaviors (individualized goal attainment rating [IGAR]), which guided the duration and content of treatment. Based on this assessment, the CM’s activities generally included individual goal identification, patient self-management by using psychoeducational materials, delivery of content to children and caregivers, consultation with the PCP, collaboration with the office practice, and linkages with specialty services and the family (eg, calls to monitor treatment response). Services were considered completed if the family received at least 6 sessions and met its agreed upon goals. Families that needed continued care at the end of 12 sessions were referred for aftercare to a recommended provider. Disposition plans, including referral for continuity or aftercare, were reviewed with the child’s PCP and psychiatric consultant (PC). All recommendations were recorded in the medical record and study database.

EUC

After providing brief psychoeducation, the CM made a facilitated referral to a familiar local mental health provider who accepted the child’s insurance. CMs mailed assessment reports to providers and made follow-up calls to parents 2 weeks after referral. Children could also receive ADHD medication from their PCP.

Assessment Procedures

Two bachelors-level research associates unaware of treatment condition administered rating scales, interviews, and treatment response ratings (Table 3). Per intention to treat, all cases were followed. Assessment measures were collected at baseline, 6, 12, and 18 months. Different sources completed service use and treatment measures during and after treatment.

TABLE 3.

Summary of Assessment Measures, Timetables, and Variables

Measure/Variables	Timea	Units/Measure
Screening and diagnostic assessment
Master screening/demographic form	0	—
PSC-17 externalizing scale	All	—
K-SADS diagnostic interview (eg, ADHD, ODD, CD)	All	—
Provider care processes
Services provided log	6	Number of 15-min blocks in activity/client
Direct (eg, intake, psychoeducation)		Hours
Indirect (eg, case management calls)		Hours
Paperwork		Hours
Clinical supervision		Hours
Treatment summary report	6	Fill-in responses and checklist of parameters
Receipt of the assigned treatment		Yes or no
Dose of treatment		Number of hours of treatment
Duration of treatment		Number of weeks of treatment
Participants in treatment		Child, caregiver, family
On medication for a behavioral health problem		Yes or no; During treatment? At discharge?; 2 items
Case disposition		3 types (1 = dropped; 2 = still in treatment; 3 = completed)
Child and parent outcomes
VADPRS	All	Rating of symptom severity
Oppositional defiant/conduct disorder		Rating (0 = never; 3 = very often); 7 items
Hyperactivity/impulsivity subtype		Rating (0 = never; 3 = very often); 7 items
Inattention subtype		Rating (0 = never; 3 = very often); 7 items
Anxiety/depression		Rating (0 = never; 3 = very often); 7 items
PedsQL	All	—
Physical, emotional, school, and social scales		Rating (0 = never; 4 = almost always); overall mean
CGI	All	—
Level of dysfunction at intake		Rating (1 = normal; 7 = extremely ill)
Level of improvement		Rating (1 = very much improved; 7 = very much worse)
IGARs	All	—
Level of improvement for each treatment goal		Rating (1 = pretreatment severity, 3 = expected improvement; acceptable progress; 5 = exceeded/exceptional improvement); up to 4 goals
CSQ-8	6	Rating (1 = not at all; 4 = very much); 8 items
Parenting stress scale–short form	All	Rating of level of stress
Parent–child dysfunctional interaction		Rating (1 = strongly agree; 5 = strongly disagree); 12 items
Parental distress		Rating (1 = strongly agree; 5 = strongly disagree); 12 items
Difficult child		Rating (1 = strongly agree; 5 = strongly disagree); 12 items
Pediatrician attitudes and practice outcomes
PCP demographic form	0	Age, gender, ethnicity, fellowship, certification
Provider practices survey (adapted)	All	—
Change in management of behavior problems		Rating (1 = not at all; 4 = quite a lot); 1 item
Skill in providing services for behavior problems		Rating (1 = not at all; 5 = very); 6 items
Change in management of ADHD		Rating (1 = not at all; 5 = very); 1 item
Likelihood of medication use for ADHD comorbidities		Rating (1 = not at all; 5 = very); 5 conditions
Extent to which factors limit optimal ADHD treatment		Rating (1 = no limit; 3 = great limit); 8 limitations
MH-SKIP	All	—
Obstacles to making services available for behavior problems		Rating (1 = not at all; 5 = very much); 1 item
Frequency of outside referral for behavior problems		Rating (1 = not at all; 5 = very much); 1 item
Competency/effectiveness in addressing behavior problems		Rating (1 = not at all; 5 = very much); 7 items
PBS	All	Attitudes about delivering psychosocial treatment
Beliefs about treatment (eg, can’t help patient)		Rating (1 = disagree; 5 = agree); 8 items
Burdens to delivering treatment (eg, much effort)		Rating (1 = disagree; 5 = agree); 6 items
OSC	All	Scales to assess organizational climate
Cooperation/personal accomplishment		Rating (0 = not at all; 4 = to very great extent); 12 items
Role conflict/role overload		Rating (0 = not at all; 4 = to very great extent); 14 items

CD, conduct disorder; K-SADS, Kiddie-Schedule for Affective Disorders and Schizophrenia; ODD, oppositional defiant disorder.

^aAssessment times = 0, 6, 12, and 18 mo after baseline.

Processes of Care

CMs documented all activities performed for clients on a services provided log¹⁴ and completed a treatment summary report to document the parameters of treatment delivered by CMs (DOCC) or outside mental health providers (EUC).¹³ We computed an “any services” variable on the basis of responses to both measures.

Child and Parent Outcomes

Parents completed the Vanderbilt ADHD Diagnostic Parent Rating Scale (VADPRS)²³ to measure symptom severity and determine remission rates by using existing clinical cutoffs of 4 main symptom clusters (oppositional defiant/conduct disorder, hyperactivity/impulsivity; inattention; anxiety/depression). Health-related quality of life was assessed with the parent-completed Pediatric Quality of Life Inventory (PedsQL).^24,25 Parents also completed 36-item Parenting Stress Index-Short Forms (PSI-SFs) to document change in 3 primary subscales (ie, difficult child, parent–child dysfunctional interaction, parental distress).²⁶

Parents identified treatment goals for up to 4 child problems on an IGAR.¹⁴ At pretreatment, each problem and specific behavioral anchors of improvement were defined (eg, 1 = pretreatment severity, 3 = expected or acceptable improvement; 5 = exceeded expected improvement). Goals at pretreatment were rated a “1,” but any 1 to 5 rating could be used later.

The Clinical Global Impression-Severity (CGI-S) and Clinical Global Impression-Improvement (CGI-I) ratings were completed by a study CM who worked in a different practice and had no contact with the family to assess symptom severity at intake (CGI-S) and level of improvement at 6- and 12-month follow-ups (CGI-I)^27,28 on a 7-point scale. Treatment response was defined as a CGI-I rating of 1 (very much improved) or 2 (much improved), with high interrater agreement with the treating CM’s rating (r = 0.92, P < .001). Finally, parents completed the Client Satisfaction Questionnaire-8 (CSQ-8) at discharge.

Provider Outcomes

The Physician Belief Scale (PBS) documents provider attitudes about delivering psychosocial treatment in primary care (eg, beliefs and feeling about treatment, service burdens).²⁹ PCPs completed a Provider Practices Survey targeting changes in management and skill in addressing behavior problems and ADHD (α = .81 to 84) that was modified from a previous survey.³⁰ The Mental Health SKIP (MH-SKIP) assessment examines changes in treatment obstacles, use of outside referral, and competency and effectiveness in delivering psychosocial services (α = 0.77). Four subscales from the Organizational Social Context (OSC) scale evaluated changes in the climate of the practice.³¹ Two correlated positive subscales (cooperation, personal accomplishment; r = 0.55, P < .001) and 2 negative subscales were combined (role conflict, role overload; r = 0.60, P < .001).

Power Analysis

For hierarchical linear modeling (HLM) analyses, we used power calculation methods from Raudenbush.^32–34 A proposed sample size of 300 at baseline (with 20% attrition rate) with 10 clients per PCP, 30 PCPs, 4 time points, and a within-subject correlation of 0.10 to 0.06 (based on Kolko et al 2010¹³) would provide >80% power for finding an effect size (ES) of d = .33 for α = .05 (2-sided) for group differences on outcome measures. ESs of 0.3 to 0.5 were found on key outcomes in our previous studies.^13,14

Data Analysis

We first examined the equivalence of DOCC and EUC on demographic and baseline clinical characteristics by using t tests for dimensional variables and χ² tests for categorical variables (Table 2). Outcome analyses used SPSS (IBM SPSS Statistics, IBM Corporation; Predictive Analytics Software [PASW] 18) and HLM-6.³⁵ For child and parent outcomes, a piecewise growth curve modeling approach³⁶ with an intercept representing baseline levels of functioning and 2 linear slope factors representing change over time was estimated for each family at the model’s first level. Time (assessment) was nested within participants (practitioners). Full maximum likelihood estimate was used. Cases with data for baseline and ≥1 other time point were retained. The level 1 equations for the unconditional models were Y_ti = π_0i + π_1i(pre-later) + π_0i(follow-up) + e_ti, where Y_ti is the observed outcome at time t for participant i. The “pre-later” variable was coded 0, 1, 1, and 1 for the 4 time points. This pre-later slope is the change from baseline to postbaseline, and its coefficient reveals the change due to condition. The “follow-up” variable was coded as 0, 0, 1, and 2 for the 4 time points. This follow-up slope is the change during a 6-month period of the follow-up phase, and its coefficient reveals the change due to condition. We first ran piecewise models of our outcomes unrestricted at level 2 and then examined the effects of training by entering condition (DOCC = “1”; EUC = “0”) at level 2. Pre-later and follow-up are examples of cross-level interactions,³⁷ wherein the level 2 variable, condition, affects the slope of a level 1 predictor.

For PCP outcomes, a simpler growth curve model with a single linear slope representing change over time was estimated for each PCP at the first model level. The level 1 equations for the unconditional models were Y_ti = π_0i + π_1i(time) + e_ti. The time variable was coded 0, 1, 2, and 3 for the 4 time points. This time slope is the change during a 6-month study period. All other characteristics matched the family models.

ES Calculations

ES calculations for cross-sectional analyses used calculations for standardized mean differences (d) that were conducted with the Practical Meta-Analysis Effect Size Calculator.³⁸

Results

Group Equivalence

DOCC and EUC were comparable on all baseline PCP background and outcome variables, and family variables, with 2 exceptions (Table 2). DOCC had a higher proportion of parents who completed at least some college, but a lower proportion of families from practices with experience in a previous study. These findings highlight the initial equivalence of both conditions. Most PCPs in DOCC and EUC enrolled a family (87% vs 79%; P = .37), with a mean of 5.5 patients per PCP (SD = 5.2). Overall study retention was comparable (97% vs 93%; P = .48).

Processes of Care

The rate of any mental health service use was significantly higher for DOCC than EUC (Table 4). CMs in DOCC and EUC averaged 3.5 and 3.0 hours completing intakes, respectively, with more time spent in DOCC on psychoeducation, paperwork, and supervision (all Ps < .001). The mean session length for DOCC cases was 48 minutes (SD = 6.2). Among available reports, DOCC (versus EUC) providers reported more hours of service, longer duration of treatment, more outpatient work with the child and caregiver, and lower rates of referral. More DOCC cases completed treatment goals on time and were on medication at discharge, whereas more EUC cases left treatment early.

TABLE 4.

Processes and Description of Care in the Two Conditions^a

	N (DOCC)	N (EUC)	DOCC	EUC	Pb	ES
Service use
Any services for child behavior problems	158	142	99.4	54.2	<.001	1.25
Service parameters
Hours of service, mean (SD)	157	70	11.6 (4.9)	8.1 (6.3)	<.001c	0.65
Weeks of service, mean (SD)	157	70	18.1 (7.2)	12.6 (8.2)	<.001c	0.73
Use of outside referral	158	73	0.6	23.3	<.001	0.88
Outpatient work with child	157	71	91.1	76.1	<.001	0.64
Outpatient work with caregiver	157	71	91.7	26.8	<.001	0.74
Outpatient work with family	157	71	68.2	64.8	.65	0.08
On medication for a behavioral health problem	157	65	48.4	33.8	.05	0.34
On medication at discharge	76	22	52.6	9.1	<.001	1.33
On medication in treatment and at discharge	76	22	48.7	9.1	<.001	1.24
Case disposition goals status (at termination)	158	69
Completed	121	8	76.6	11.6	—	—
Dropped out	36	28	22.8	40.6	<.001	2.01
Treatment ongoing	0	29	0.0	42.0	—	—
Other	1	4	0.6	5.8	—	—
Services provided by CM (hours)
Screening/intake, mean (SD)	160	160	2.6 (0.5)	2.5 (0.5)	.18c	0.20
Psychoeducation, mean (SD)	160	160	0.5 (0.2)	0.3 (0.2)	<.001c	1.00
Paperwork, mean (SD)	160	160	4.4 (1.4)	2.8 (0.7)	<.001c	1.45
Clinical/supervision, mean (SD)	160	160	1.0 (0.5)	0.3 (0.1)	<.001c	1.94

^aData are No. (%) unless otherwise indicated.

^bAnalysis by χ² unless otherwise indicated.

^cAnalysis by t test for means.

Child and Parent Outcomes

Table 5 presents the descriptive statistics for the primary child and parent outcome measures at each time point. ES values are included for 2 of 4 time points: 6-month to illustrate the magnitude of acute differences immediately after intervention, and 18-month to show the magnitude of differences at the study’s conclusion.

TABLE 5.

Means, SDs, and ESs for Primary Child and Parent Outcomes

Outcomes	DOCC, N	EUC, N	DOCC, Mean (SD)	EUC, Mean (SD)	ES
VADPRS, symptom severity ratings
Oppositional defiant/conduct disorder
Baseline	160	161	17.7 (7.8)	18.0 (8.3)	—
6 mo	150	142	12.1 (8.1)	14.1 (8.4)	0.25
12 mo	147	141	11.6 (6.9)	13.9 (8.9)	—
18 mo	144	130	12.3 (7.6)	13.0 (8.1)	0.09
Inattention subtype
Baseline	160	161	14.4 (7.0)	15.0 (6.6)	—
6 mo	150	142	10.9 (6.0)	12.2 (6.1)	0.22
12 mo	147	141	10.7 (6.0)	11.7 (6.0)	—
18 mo	144	130	11.2 (6.3)	12.0 (6.0)	0.12
Hyperactivity/impulsivity subtype
Baseline	160	161	14.8 (6.6)	14.1 (6.7)	—
6 mo	150	142	10.2 (6.3)	11.3 (6.3)	0.17
12 mo	147	141	9.5 (5.8)	10.9 (6.7)	—
18 mo	144	130	10.1 (5.8)	10.5 (6.1)	0.07
Anxiety/depression
Baseline	160	161	6.5 (4.9)	6.2 (4.2)	—
6 mo	150	142	4.4 (3.6)	5.1 (3.7)	0.20
12 mo	147	141	4.8 (4.0)	4.9 (4.1)	—
18 mo	144	130	4.6 (3.9)	4.7 (4.0)	0.03
PSI-SF
Parental distress
Baseline	159	161	24.6 (7.7)	24.9 (8.1)	—
6 mo	150	138	23.0 (7.0)	25.8 (8.1)	0.36
12 mo	147	136	22.8 (7.9)	24.8 (8.7)	—
18 mo	141	129	23.5 (8.5)	24.2 (9.0)	0.08
Parent-child dysfunctional interaction
Baseline	159	160	23.7 (7.3)	24.2 (6.9)	—
6 mo	150	138	21.5 (7.1)	23.9 (7.8)	0.32
12 mo	147	136	21.8 (7.8)	24.0 (8.0)	—
18 mo	141	129	22.8 (7.7)	23.3 (7.8)	0.08
Difficult child
Baseline	159	160	36.0 (8.3)	36.2 (7.8)	—
6 mo	150	138	32.9 (9.6)	34.5 (9.3)	0.17
12 mo	147	136	31.8 (9.7)	33.4 (9.7)	—
18 mo	141	129	32.3 (9.9)	33.0 (9.9)	0.07

We first analyzed the severity of all problems and improvements in child health status and PedsQL. Using the pre-later model, both conditions revealed significant reductions for all 5 outcomes, but DOCC (versus EUC) revealed significantly greater reductions in behavior, hyperactivity, and internalizing problems (Table 6). In the follow-up model, significant changes over time were found only in severity of hyperactivity/impulsivity ratings. DOCC (versus EUC) did not reveal any significant changes over the follow-up phase on any of the 5 outcomes. The absence of significant follow-up differences does not mean differences in the pre-later model have disappeared. Rather, the earlier differences have not been altered during follow-up.

TABLE 6.

Hierarchical Linear Models for All Child and Parent Outcomes

Child and Parent Outcomes	N	Intercept	Pre-Later Time Main Effect	Condition × Time	Follow-Up Time Main Effect	Condition × Time
		β	β	β	β	β
VADPRS, symptom severity ratings	292
Oppositional defiant/conduct disorder		17.85a	−3.92a	−1.82c	−.52	.59
Hyperactivity/impulsivity subtype		14.47a	−3.12a	−1.42c	−.40c	.36
Inattention subtype		14.68a	−2.79a	−.92	−.18	.42
Anxiety/depression		6.36a	−1.24a	−.64c	−.16	.25
PedsQL total score	287	75.03a	3.28a	1.67	.56	−.54
PSI-SF	285
Parental distress		24.67a	.78	−2.64a	−.75b	.95c
Parent–child dysfunctional interaction		23.92a	−.21	−2.18b	−.14	.81c
Difficult child		36.10a	−1.80b	−1.71c	−.81b	.60
		OR	OR	OR	OR	OR
VADPRS, clinical cutoff rates	292
Oppositional defiant/conduct disorder		1.51a	0.34a	0.56b	0.82c	1.42b
Hyperactivity/impulsivity subtype		0.63a	0.47a	0.75	0.90	1.06
Inattention subtype		0.76b	0.37a	1.08	1.04	0.99
Anxiety/depression		0.33a	0.55a	0.71c	1.11	0.91

^aP < .001.

^bP between .001 and .01.

^cP between .01 and .05.

On the PSI-SF, DOCC (versus EUC) parents reported significantly greater reductions on all 3 subscales (parental distress, parent–child dysfunction, difficult child) using the pre-later model, and on the first 2 subscales on the basis of the follow-up model.

HLMs also documented higher VADPRS remission rates for both conditions in ADHD inattention and hyperactivity, but significantly greater remission for DOCC (versus EUC) in behavior problems and internalizing problems on the basis of the pre-later model. Remission rates for DOCC and EUC at posttreatment were as follows: behavior problems (71% vs 51%) and internalizing problems (76% vs 66%). Both conditions revealed greater remission in behavior problems on the basis of the follow-up model, but this was qualified by an interaction revealing higher remission since posttreatment of EUC than DOCC.

Using analysis of variance, the mean IGARs revealed significantly greater improvements for DOCC (versus EUC) at 6-, 12-, and 18-month follow-ups (Table 7). At baseline, DOCC and EUC had comparable proportions of children rated at each severity level on the CGI-S (P = .46), especially at the 2 lowest levels (2% vs 3%), but significantly more DOCC children were treatment responders (CGI-I) at the 6-month follow-up. Parents also reported greater service satisfaction with DOCC.

TABLE 7.

Cross-Sectional Analyses of Child and Parent Outcomes

Outcome	Time point, mo	N (DOCC)	N (EUC)	DOCC,N (%)	EUC,N (%)	P	ES
CGI improvement	6	150	139	58 (38.7)	38 (27.3)	.04a	0.28
	12	146	141	61 (41.8)	52 (36.9)	.40a	0.11
				Mean (SD)	Mean (SD)
IGAR average	6	150	142	3.3 (1.0)	2.7 (1.0)	<.001b	0.60
	12	147	141	3.2 (1.0)	2.9 (1.0)	.02b	0.30
	18	144	133	3.2 (1.0)	3.0 (1.0)	.03b	0.20
CSQ-8 total	6	148	89	28.9 (4.2)	25.5 (6.5)	<.001b	0.66

^aAnalysis by χ².

^bAnalysis by t test for means.

Provider Outcomes

Table 8 presents the descriptive statistics for the primary PCP outcome measures at each time point. As with the child and parent outcomes, ES values are included for 6-month and 18-month time points.

TABLE 8.

Means, SDs, and ESs for Provider Outcomes

Outcomes	DOCCN	EUCN	DOCC,Mean (SD)	EUC,Mean (SD)	ES
Provider practices survey (adapted)
Change in management of behavior problems
Baseline	31	42	2.8 (0.7)	2.8 (0.9)	—
6 mo	28	33	3.0 (0.6)	2.8 (0.7)	0.31
12 mo	28	32	3.0 (0.6)	2.9 (0.7)	—
18 mo	28	32	3.2 (0.6)	2.7 (0.7)	0.78
Skill in providing services for behavior problems
Baseline	31	42	2.2 (0.6)	2.1 (0.5)	—
6 mo	28	32	2.1 (0.6)	2.0 (0.6)	0.16
12 mo	28	32	2.2 (0.6)	2.0 (0.5)	—
18 mo	28	32	2.3 (0.5)	2.0 (0.5)	0.45
Change in management of ADHD
Baseline	31	42	3.3 (0.6)	3.4 (0.7)	—
6 mo	28	32	3.4 (0.6)	3.3 (0.8)	0.06
12 mo	28	31	3.4 (0.7)	3.1 (0.7)	—
18 mo	28	32	3.5 (0.6)	3.1 (0.7)	0.57
Likelihood of medication use for ADHD comorbidities
Baseline	31	41	2.9 (1.0)	2.9 (1.1)	—
6 mo	28	32	2.7 (1.1)	2.7 (1.1)	0.01
12 mo	28	31	3.1 (1.2)	2.6 (1.1)	—
18 mo	28	31	3.2 (1.0)	2.3 (1.0)	0.83
Extent to which factors limit optimal ADHD treatment
Baseline	31	42	2.1 (0.3)	2.0 (0.3)	—
6 mo	27	33	2.1 (0.4)	2.0 (0.4)	−0.12
12 mo	28	32	1.9 (0.3)	1.9 (0.3)	—
18 mo	28	32	1.8 (0.3)	1.9 (0.4)	0.18
MH-SKIP
Obstacles to making services available for behavior problems
Baseline	31	42	4.3 (0.7)	4.2 (0.8)	—
6 mo	28	31	4.1 (1.0)	3.9 (1.0)	−0.14
12 mo	26	32	4.1 (0.9)	4.2 (0.7)	—
18 mo	28	32	3.6 (1.0)	4.0 (0.9)	0.43
Frequency of outside referral for behavior problems
Baseline	31	42	4.3 (0.7)	4.2 (0.7)	—
6 mo	28	31	4.1 (0.6)	4.4 (0.9)	0.32
12 mo	26	32	4.3 (0.7)	4.6 (0.6)	—
18 mo	28	32	4.0 (0.7)	4.6 (0.6)	0.87
Competency/effectiveness in addressing behavior problems
Baseline	31	42	3.3 (0.5)	3.2 (0.7)	—
6 mo	28	31	3.3 (0.5)	2.9 (0.5)	0.76
12 mo	26	32	3.4 (0.7)	3.0 (0.6)	—
18 mo	28	32	3.5 (0.6)	2.9 (0.7)	0.77

The provider practices survey revealed more change by DOCC (versus EUC) PCPs in management practices and perceived skill in treating behavior problems and ADHD, and their comfort in addressing comorbidities (Table 9). Perceived obstacles to mental health service availability in the practice were similar in DOCC and EUC on the MH-SKIP. As expected, EUC (versus DOCC) clinicians were significantly more likely to make outside referrals, whereas DOCC (versus EUC) clinicians reported greater perceived competence and effectiveness in delivering on-site behavioral health services over time. There were no significant changes over time or any condition × time interactions on the PBS total score or the 2 derived subsets of OSC subscales (practice cooperation/personal accomplishment, role conflict/overload).

TABLE 9.

Hierarchical Linear Models for PCP Outcomes

Outcomes		Intercept	Time Main Effect	Condition × Time
	N	β	β	β
PBS (burdens and negative feelings)	67	11.71a	−.20	.25
Provider practices survey (adapted)
Change in management of behavior problems	67	3.18a	.02	.14b
Skill in providing services for behavior problems	67	2.08a	−.03	.08c
Change in management of ADHD	67	2.67a	−.09	.14c
Likelihood of medication use for ADHD comorbidities	66	3.15a	−.16b	.27a
Extent to which factors limit optimal ADHD treatment	67	2.02a	−.04c	−.03
MH-SKIP survey
Obstacles to making services available for behavior problems	65	4.20a	−.04	−.12
Frequency of outside referral for behavior problems	65	4.23a	.14a	−.19a
Competency/effectiveness in addressing behavior problems	65	3.23a	−.12b	.18a
OSC questionnaire
Cooperation and personal accomplishment total	65	29.26a	.16	−.46
Role conflict and role overload total	65	18.84a	.28	−.55

^aP < .001.

^bP between .001 and .01.

^cP between .01 and .05.

Discussion

This randomized trial provides further support for the feasibility, benefits, and acceptability of an expanded on-site intervention on the basis of the chronic care model (DOCC) for children referred by their PCPs for behavior problems. Like our pilot study,¹⁴ implementation of DOCC by trained CMs improved service access, child and caregiver participation, and treatment completion (versus EUC), highlighting the utility of delivering behavioral health services in pediatric offices.^10,11 DOCC improved mental health care by making counseling, medication management, and collaboration with PCPs and families more widely available.^39,40

Both DOCC and EUC showed improved outcomes,¹³ but DOCC showed significantly greater reductions in the severity of behavior problems, hyperactivity, and internalizing problems, greater remission of behavior and internalizing problems, and a higher proportion of overall treatment responders. Further, DOCC parents reported significant reductions over time in ratings of child difficulty, parent–child dysfunctional interactions, and parental distress related to child behavior. These findings demonstrating enhanced child and parent benefits associated with collaborative care extend those reported in quality improvement interventions for child behavior problems,^13,14 ADHD,^6–8 adolescent depression,^10,11 and other problems.⁵ In the follow-up period, EUC showed significantly greater remission since posttreatment in behavior problems than DOCC, which may reflect DOCC patients having achieved greater remission by the end of treatment.

As in our previous trials, individualized treatment goals (IGAR) showed greater improvement for DOCC at all 3 follow-ups. In contrast, fewer improvements were found on other measures, perhaps because the item content of these broad measures is less applicable to a given child.⁴¹ This pattern of findings highlights the potential of identifying individualized goals on methods that can compare outcomes across goals and guide selection of personalized intervention content.

PCPs reported no change in perceived burdens to treating mental health problems or adverse aspects of the organizational climate. As expected, DOCC PCPs reported greater treatment involvement in on-site service delivery than those in EUC, who were more likely to refer to outside providers. DOCC PCPs acknowledged greater treatment involvement, competency/effectiveness with behavior problem children, and ADHD medication management skills. Interestingly, these improvements were even more substantial during the follow-up period, suggesting that it may take time to achieve changes in attitudes and practices. The collaborative approach coordinated by CMs promoted PCP service involvement and continuity, especially around ADHD.

Among the study’s limitations, the broad array of clinical content modules (for behavior problems, ADHD, and anxiety) and care processes (eg, meetings with PCPs, weekly progress monitoring) in DOCC precludes evaluation of its components. Given group differences in content, duration, and other treatment parameters, future work could control for relationship or alliance effects. In addition, we had data missing from EUC providers, despite incentives and follow-up calls. The inclusion of more formal fidelity measures and teacher ratings would expand the objectivity of the assessment of provider practices and clinical outcomes, respectively.

We also recognize the need to explore the financing of collaborative care resources, as we chose to use grant funds to pay for the CMs to maximize fidelity to the program when implemented in a real-world clinical setting. Clearly, more research is needed to understand how practices adapt operational and financial strategies for sustaining key program resources, including focused training and technical assistance through the Replicating Effective Programs (REP) program,⁴² as well as discussions with state and local providers and stakeholders on a reimbursement model for care management activities so the clinics can absorb the costs.⁴³ It is important to point out that the participating pediatric practices in this clinical trial later hired their own clinicians for on-site services after the trial had ended.

Conclusions

A collaborative care management model in pediatric practice (DOCC) enhanced access to and completion of behavioral health services, child and parental outcomes, consumer satisfaction, and provider practices, relative to EUC. The inclusion of standardized assessments with all PCPs provided novel feedback on key implementation outcomes. In 3 clinical trials conducted by the SKIP program, on-site care has shown advantages over facilitated referral to a local mental health provider. Unlike our pilot study, this study included PCP training in an expanded ADHD care management protocol,¹ practice-based randomization to optimize PCP participation, technology to collect and share patient progress, and greater communication among CMs, PCPs, and families. Further efforts are needed to enhance primary care’s capacity to integrate and sustain collaborative care models for delivering high quality behavioral health services to children and adolescents.^44,45 The incorporation of compelling implementation and financial models may help ensure that these evidence-based practices are transported to scale.³⁹

Glossary

ADHD

attention-deficit/hyperactivity disorder

CGI

Clinical Global Impression Scale

CGI-I

Clinical Global Impression-Improvement Scale

CGI-S

Clinical Global Impression-Severity

CM

care manager

CSQ-8

Client Satisfaction Questionnaire-8

DOCC

Doctor Office Collaborative Care

ES

effect size

EUC

enhanced usual care

HLM

hierarchical linear modeling

IGAR

individualized goal attainment rating

MH-SKIP

Mental Health Services for Kids in Primary Care

OSC

Organizational Social Context

PBS

Physician Belief Scale

PC

psychiatric consultant

PCP

primary care provider

PedsQL

Pediatric Quality of Life Inventory

PONI

protocol for on-site nurse-administered intervention

PSC-17

Pediatric Symptom Checklist 17

PSI-SF

Parenting Stress Index-Short Form

SKIP

Services for Kids in Primary Care

VADPRS

Vanderbilt ADHD Diagnostic Parent Rating Scale