Improving Access to Care and Clinical Outcome for Pediatric Behavioral Problems: A Randomized Trial of a Nurse-Administered Intervention in Primary Care

Abstract

Objective

To determine the effectiveness of an on-site modular intervention in improving access to mental health services and outcomes for children with behavioral problems in primary care relative to enhanced usual care.

Setting

Boys and girls from six primary care offices in metropolitan Pittsburgh, PA.

Participants

One-hundred and sixty three clinically referred children who met a modest clinical cutoff (75^th percentile) on the externalizing behavior scale of the Pediatric Symptom Checklist-17 were randomized to a protocol for on-site, nurse-administered intervention (PONI) or to enhanced usual care (EUC). PONI applied treatment modules from an evidence based specialty mental health treatment for children with disruptive behavior disorders that were adapted for delivery in the primary care setting; EUC offered diagnostic assessment, recommendations, and facilitated referral to a specialty mental health provider in the community.

Main Outcome Measures

Standardized rating scales, including the PSC-17, individualized target behavior ratings, treatment termination reports, and diagnostic interviews were collected.

Results

PONI cases were significantly more likely to receive and complete mental health services, reported fewer service barriers and more consumer satisfaction, and showed greater, albeit modest, improvements on just a few clinical outcomes that included remission for categorical behavioral disorders at one-year follow-up. Both conditions also reported several significant improvements on several clinical outcomes over time.

Conclusions

A psychosocial intervention for behavior problems that was delivered by nurses in the primary care setting is feasible, improves access to mental health services, and has some clinical efficacy. Options for enhancing clinical outcome include the use of multifaceted collaborative care interventions in the pediatric practice.

The pediatric primary care setting is a critical venue for the identification and management of mental health problems,¹ including behavior problems (BP) such as oppositional and aggressive behaviors. BPs are relevant to primary care clinicians (PCCs)^{2, 3} given their prevalence in pediatric practice (10%–17%), associated physical⁴ and psychosocial impairments,⁵ and relationship to excessive health and emergency services use.^{6, 7} Access to specialty mental health services is limited for children with BP and other common mental disorders such as ADHD.^{8, 9} Consequently, PCCs are increasingly being called on to address such problems,¹⁰ but report unease, inadequate training, limited tools,¹¹ time pressures,^{5, 11} and minimal access to specialty mental health support ^{9, 12} as serious barriers to care delivery.¹³ It is, thus, important to determine if on-site mental health service delivery in primary care is feasible and improves access to care and clinical outcomes for children with BP.

Some efforts to deliver mental health services in pediatric primary care have shown promise in randomized trials.¹⁴ A study of internet-based psychoeducation and another of on-site family intervention for children with BP both documented success, ^{8, 15} whereas a trial involving preschoolers with Oppositional Defiant Disorder (ODD) found no significant advantages to an evidence-based treatment delivered either on-site (nurse) or in the specialty mental health setting (psychologist) relative to bibliotherapy.¹⁶ Two studies have shown modest responses by depressed adolescents to internet-based and on-site collaborative care interventions.^{17, 18} Work with depressed adults in primary care has also demonstrated that the provision of on-site services improves treatment engagement and outcomes relative to usual care.^{19, 20} Our own experience suggests heightened family compliance and satisfaction with services delivered by an on-site nurse clinician within a collaborative mental health team in a rural pediatric primary care practice.²¹

Given these experiences and research reports,^21–23 we sought to evaluate an intervention model for children with BPs that located a trained nurse in the primary care office who applied a modular intervention adapted for brevity and use in primary care from a treatment of pediatric BP with demonstrated efficacy.^24–27 The goal was to offer a feasible, efficient, and cost-effective alternative to the usual practice of off-site specialty mental health referral.²⁸ We conducted a randomized controlled trial that compared a protocol for an office-based nurse-administered intervention (PONI) to enhanced usual care (EUC) that consisted of a facilitated referral to an off-site mental health professional and the provision of information from the initial psychiatric assessment and brief treatment recommendations to the PCC. Relative to EUC, we hypothesized that PONI would offer superior: 1) access to and engagement with mental health services; 2) child clinical outcomes at post-treatment and long-term follow-up; and, 3) client satisfaction with services.

METHOD

Sample

The study enrolled 163 children ages 6 to 11 years old based on parent concerns about the child’s BPs and a parent rating at or above the clinical cutoff (6) for the 75^th percentile on the externalizing behavior problems scale of the Pediatric Symptom Checklist (PSC-17). The study was approved by the IRB of the Children’s Hospital of Pittsburgh and University of Pittsburgh Medical Center; participants completed informed consent/assent as appropriate. Children with emergent psychiatric or medical needs (e.g., psychosis, suicidality with a plan, unstable medical illness) or actively participating in specialty mental health treatment were excluded. Approximately two thirds of participants were male, 80% Caucasian, and 75% living at home with married adults (see Table 1 for sample characteristics). A DSM IV psychiatric disorder was identified in 94% of the sample, the most common being ODD and ADHD; 91% met diagnostic criteria for a disruptive behavior disorder. Baseline levels of externalizing problems on the PSC-17 were high (M = 8.6; SD = 2.2). Most of the sample showed evidence of functional impairment and scored above PSC-17 clinical cutoffs for attentional and internalizing problems. Participants were randomized to PONI or EUC after initial assessment, and were well balpopoanced on all demographic and diagnostic characteristics

Table 1.

Demographic, Clinical, and Diagnostic Characteristics of the Sample at Study Intake.

Variable	Overall (N=163)		PONI (n = 83)		EUC (n = 80)		Group Comparisons
	M	SD	M	SD	M	SD	F	df
Age in years	8.1	1.6	8	1.6	8.2	1.6	1.17	1, 163
Kaufmann Brief Intelligence Test	108.4	12.9	107.1	12.9	109.8	12.8	1.73	1, 162
	n	%	n	%	n	%	χ2	df
Gender – male	106	65.0	54	65.1	52	65	0.020	163
Race – Caucasian	130	79.8	65	78.3	65	81.2	0.220	156
Marital status of adults - married	123	75.4	63	75.9	60	75.0	0.080	155
Education – at least some college	132	83.0	66	79.5	66	82.5	0.260	151
<Median ($58,000) income	75	46.0	33	39.8	42	52.5	1.890	153
ADHD children on psychoactive medication at study intake	21	21.9	14	28.6	7	15.0	2.63	96
Children who received any mental health treatment in past 6 months	65	39.9	35	42.2	30	37.5	0.370	163
Cases above 90th percentile cutoff
PSC-17 Externalizing problems	133	81.6	12	14.5	18	22.5	1.75	163
PSC-17 Attentional problems	115	70.6	55	66.3	60	75.0	1.50	163
PSC-17 Internalizing problems	91	56.2	46	56.1	45	56.3	0.00	163
SDQ-Parent Total	126	77.3	66	79.5	60	75.0	0.47	162
Columbia Impairment Scale	117	71.8	55	66.3	62	77.5	2.54	163
Any Disruptive Behavior Disorder	148	90.8	75	90.4	73	91.3	0.04	163

PSC-17 = Pediatric Symptom Checklist-17; SDQ-P = Strengths and Difficulties Questionnaire-parent version

Note. F = ANOVA; χ² = Chi-Square test. No group comparison reached statistical significance.

Setting

Participants were recruited from six large urban and suburban pediatric primary care practices covering 44,642 pediatric lives (range: 3,438–16,233), of whom 22,282 were within the age range of the study. Approximately 13.5% of all patients received Medical Assistance. The practices employed 30 pediatricians, 19 nurse clinicians, five medical assistants, and 37 office/administrative staff, and were members of Children’s Community Pediatrics (CCP), a pediatric primary care practice network affiliated with the Children’s Hospital of Pittsburgh. All 30 pediatricians confirmed a willingness to participate and follow study protocol. The practices did not have prior access to on-site mental health services or formal arrangements with local mental health providers.

Study Personnel

Nurses

Because nurse clinicians with mental health training are in relatively short supply in most geographic regions, two registered nurses with at least one year of medical-surgical experience but without prior mental health training were hired as study clinicians to increase generalizability and potential for dissemination. Nurses received four months of ‘on-the-job’ training and ongoing supervision in diagnostic assessment and clinical intervention that included observation, role playing, in vivo practice, and feedback on videotaped interviews and clinical interactions specific to PONI and EUC. Nurses were also trained to complete a study database, interact with all office staff and PCCs, and participate in study case review and office practice meetings. Supervision by a Masters’-level clinician with experience administering the protocol was supplemented by access to a part-time child and adolescent psychiatrist for diagnostic/medical and medication questions. Each nurse was assigned to 3 practices, where they assisted with screening and initial assessments for all study participants prior to randomization and then completed treatment condition-specific tasks.

Research Assessors (RA)

Two bachelors-level research assessment staff, blind to child assignment, administered research interviews and assessments, with one RA administering rating scales and the other conducting the diagnostic interview. RAs received extensive training in using all measures and assessing safety risks and participated in weekly supervision and a diagnostic case review with the study psychiatrist. Recruitment and Assessment.

Potential subjects with BPs were identified by direct PCC referral, parental requests, or parental response to office screening or posted flyers in practice waiting areas. The nurse assigned to the practice conducted a screening interview (on-site or by phone) with the parent that established the child’s eligibility for age (5–12 yrs.), referral concern (behavior problems), and clinical severity (≥ 75^th percentile) on the PSC-17 externalizing behavior scale. A study RA was intermittently available in practice waiting areas to conduct the screening interview if the nurse was not available. PCC endorsement also was necessary for eligible children to participate. If the child was eligible and the family willing, parent and child were scheduled for a research assessment conducted by two RAs who administered standardized study assessments. Upon completion of the research assessment, enrolled and eligible subjects were randomized to PONI or EUC.

Figure 1 depicts the overall flow of subjects across all study phases. Parents of 2,301 children expressed a general interest in the study, with 578 children being eligible. A total of 236 cases were able to be contacted and agreed to be scheduled for a formal research assessment, with 178 completing the assessment and fully meeting entry criteria. Of these 178 cases, 163 were enrolled in the study and randomized to PONI or EUC, of whom 173 (88%) completed the 12-month follow-up. Enrollment rates were evenly distributed across offices (M = 14%; 10–18%). Sources of initial contact with enrolled subjects included parental inquiry generated by office-based screening (n=90, 66%), flyers/advertisements (17), unsolicited parent request (6), and direct PCC referral (50).

Figure 1.

Study Screening, Enrollment, and Participation Summary

Assessment Procedures

Research assessments were conducted with children and parents every 6 months (pre-treatment, post-treatment, and at 6- and 12-month follow-ups). Assessments lasted approximately 2.00 hours for children and 2.75 hours for caregivers. Each informant was paid in increasing amounts at each consecutive research assessment visit ($20, $30, $40, $50). All rating scales and diagnostic interviews were collected at the four research assessments. Pediatrician ratings were collected at the conclusion of the study.

Observations and Measurements

Clinical Problems, Impairment, and Adaptive Behaviors

Parent ratings on the 17-item PSC-17)^2,3 yielded scores on three factors (externalizing behavior, attentional, and internalizing problems), each with a clinical cutoff.²⁹ Parent and teacher reports on the Strength and Difficulties Questionnaire (SDQ)³⁰ gave a comprehensive rating of the child’s clinical and peer problems. Child-reported anxiety symptoms were surveyed using the psychometrically sound 5-item short-form of the Scale for Anxiety and Related Emotional Disorders (SCARED).³¹ the short-form has good psychometric properties and a cutoff (3) for young clinical samples. Child-reported depressive symptoms in the previous 2 weeks were assessed using the 32-item (0–2 pt. scale) Mood and Feeling Questionnaire(MFQ).^{27, 32} The 13-item parent completed Columbia Impairment Scale (CIS) provided a global measure of child impairment in 4 areas of functioning (i.e., family, peer, work, school);³³ the clinical cutoff is 15. Children completed the 45-item Child Health and Illness Profile CHIP³⁴ to provide an assessment of overall adjustment across five primary domains and health-related behaviors. The domains include satisfaction (with health and self), comfort (physical and emotional), and resilience (physical activity, social problem solving).

Psychiatric Disorder

The Schedule for Affective Disorders and Schizophrenia for School-Aged Children for DSM-IV--Present and Lifetime (Kiddie-SADS-PL)³⁵ was administered by trained diagnostic interviewers to detect current Axis I child and adolescent psychiatric disorders (e.g., ADHD, ODD, Major Depression). For each symptom which met the threshold criterion (rating of 3), several follow probes were asked (i.e., age of onset, age of offset, symptom frequency, problem settings, impairment level). In the case of ADHD, the diagnosis was rendered only if threshold symptoms were found at both home and school. Parents and children were interviewed separately and then jointly addressed any discrepancies. Inter-rater reliabilities for a subset of 36 cases were high (k’s = 0.75 – 1.00), which parallel our prior results with ODD and CD children.²⁴

Individualized Treatment Goals

Parents were interviewed to identify treatment goals for up to four individualized child problems. At pre-treatment, each problem on the Individualized Goal Achievement Rating (IGAR) form ²⁷ was defined. Specific behavioral anchors were obtained to define the level of improvement on a 5-pt scale (e.g., 1 = pre-treatment severity, 3 = expected level of improvement; 5 = exceeded expected level of improvement). All goals at pre-treatment were rated a “1”, but any 1–5 rating could be used subsequently to capture level of improvement per goal. Similar methods were used in our other behavioral outcome studies.²⁷

Consumer Satisfaction and Barriers to Treatment. Parents completed the 8-item Client Satisfaction Questionnaire (CSQ-8)³⁶ at post-treatment to capture perceptions of treatment acceptability and helpfulness, as well as the Barriers to Treatment Participation Scale (BTPS)³⁷ to evaluate the impact of treatment obstacles or concerns in four domains (e.g., therapist relationship, competing activities) on a 1–5 point Likert scale.

Treatment Termination Summary (TSR)

The TSR was designed to document parameters of the child’s treatment. It was completed in PONI by all study nurses (n = 80) and in EUC by either an outside specialist (n = 26) or the child’s PCC (n = 54). The parameters included the receipt of the assigned treatment, key participants, type, dose, and duration of treatment, completer status at termination (i.e., case terminated upon reaching treatment goals or at a time mutually agreed upon with provider), case disposition, and any discharge recommendations. There was good agreement (kappa = 0.82) between the TSR medication status item and parental reports on a medication use item that was administered to parents at posttreatment (n = 114).

Pediatrician Perspectives on Mental Health Service Delivery

Pediatricians rated a few questions about their beliefs and opinions regarding the availability and relevance of on-site mental health services on Likert scales (1 = not at all; 5 = very much).

Treatment Conditions

Due to the small number of practices (6), randomization was by subject. Enrolled children meeting inclusion criteria were randomly assigned to PONI or EUC on a 1:1 basis. To ensure group comparability, Efron’s biased coin toss³⁸ was used to balance the groups on gender, age (6–8 vs. 9–11), and a prior history of mental health treatment. Nurses reviewed assessment results and diagnostic impressions with the parents in both conditions, and gave a written summary of the initial assessment and recommendations to the PCC which was placed in the chart. Nurses attended practice meetings and adapted study procedures in accord with the patterns of each practice while preserving study integrity.

Protocol for On-Site, Nurse-Administered Behavioral Intervention (PONI)

The PONI condition was delivered in the practice by study nurses who coordinated the family’s care with the PCC. Families were informed that treatment would involve approximately 6 primary sessions (1.5 hrs each) for training in cognitive-behavioral treatment (CBT) skills and, as needed, 2–4 booster sessions to address emergent issues or promote maintenance on a key target. Thus, treatment was designed to be delivered in about 10 visits in 3–6 month. PONI cases were referred for outside mental health services only after study treatment had ended. The protocol incorporated “core” components in the treatment literature for BPs³⁹ and whose content was found to be efficacious in our prior outcome studies for BPs,^{24, 25} child firesetting,⁴⁰ and family aggression. ^{26, 42}

Protocol content was developed in seven intervention modules: ^{42, 24} 1) Child CBT. Children received training in the use of anger-control, self-instructions, or social skills supplemented with role-plays and home practice to promote self-management and prosocial behavior; 2) Medication for ADHD. Parents reviewed ADHD symptom ratings and the study psychiatrist’s medication recommendations with the nurse, and, if indicated, were encouraged to discuss medication treatment with their PCCs. PCCs could ask nurses to collect follow-up ratings and participate in a medication review session; 3) Enhanced Parent Management Training (PMT). Parents were trained to use positive reinforcement, non-corporal discipline involving contingency management (e.g., withdrawal of privileges), and anger management skills, to encourage appropriate behavior management; 4) Development: Expectations and Peer Enrichment. Parents received information on child development and age-appropriate behavioral expectations, and were taught to promote child involvement in peer and recreational activities; 5) Family Psychoeducation and Skills Training (PAST). Joint sessions discussed information about family rules, decision-making, and alternatives to the use of coercion or physical force, and then taught skills to enhance positive interactions (e.g., communication and problem-solving); 6) School/Teacher Consultation. Nurses contacted teachers to discuss relevant management or educational concerns, identify skills for remediation, and provide individualized behavioral suggestions based on teacher ratings; and 7) Case and Crisis Management. Nurses helped families respond to challenges and crises, and advocated with local service agencies and professionals, as needed.

The PMT, CBT, and PAST modules were administered first to teach basic skills, followed next by the remaining modules, as needed, to address problems related to ADHD, school misbehavior, problematic peer/community relations, and/or family crises/conflicts. The extent to which each module was administered reflected the severity of the child’s identified behavior problems and the family’s skill repertoire. Services were terminated when the family sought to end care or met treatment goals.

Enhanced Usual Care (EUC)

Children in the EUC condition were referred to an off-site specialty mental health provider. Study nurses identified local mental health providers based on a list generated by the practice, and facilitated referral by making an initial call to the provider that confirmed the family’s appropriateness (e.g., relevant clinical expertise) and acceptability for services (e.g., insurance). Referral information was conveyed to the PCC and documented in the medical record to facilitate PCP involvement in services, including medication prescription.

Study Attrition

Three PONI cases dropped out of the study before the first session, and six additional cases each in PONI and EUC dropped out after the first session. Overall study attrition was modest for PONI (11%) and EUC (8%), which compared to our prior clinic-based efforts (15%)²⁵ and other clinical trials (10–14%).^{26, 27, 42}

Treatment Integrity

We sought to minimize allegiance or contamination effects and enhance the integrity of both conditions using several steps: a) nurses were trained in both intervention procedures using specified manuals and clinical guidelines; b) nurses participated in weekly study team meetings which included discussions of overall study integrity but were not included in discussions of the study’s hypotheses or results; c) treatment adherence in both conditions was monitored routinely and weekly feedback given upon supervisor review of session notes and tapes; and d) we examined possible contamination effects at the level of the PCC.

In PONI, integrity was documented by a supervisor who rated a randomly selected set of treatment sessions (30%) using a Treatment Integrity Rating Form (TIRF).²⁶ TIRF ratings reflecting the percentage of correctly administered content in each module were collected for 23 of 24 skill tasks assessed and found to be high (Mean = 91%; range: .68 – 1.00). The supervisor also reviewed each nurse’s Services Provided Log (SPL), which was completed after each treatment contact to document services received,^{26, 43} including any adjunctive services (e.g., crisis and case management). In EUC, the supervisor reviewed audiotapes of selected (15%) feedback sessions and an abbreviated SPL to document the completion of key EUC tasks and the absence of PONI content using a similar task list, which also yielded high integrity (Mean = 94%; .87 – 1.00). EUC families were contacted by the supervisor two weeks before post-treatment assessment to get permission to request a TSR from outside providers.

Because of concerns about contamination, we computed Intra-class Correlation Coefficients (ICC) reflecting the correlation among cases in the two conditions from the same PCC on the six primary child behavioral outcomes. Very low ICCs (M = 0.06) suggest little contamination by condition.

Data Analysis

Enrolled cases were followed per the intent-to-treat model. Descriptive analyses conducted for primary outcomes at pre-treatment found them to be normally distributed. Alphas for study measures were typically acceptable (0.60 – 0.93); the alpha for the SDQ-P conduct problems scale was modest (0.51). Rates of completed assessments were high at post-assessment (93%), six-months follow-up (91%), and one-year follow-up (88%).

To determine group comparability, we compared the groups on pre-treatment background and clinical variables using χ² tests and ANOVAs.⁴⁴ As preliminary analyses found no significant differences between the outcomes of the two nurses in each condition, the acute treatment outcomes of PONI and EUC were compared using χ² tests and pre-post repeated measures ANOVAs. Chi-square tests examined the proportion of cases diagnosed with a psychiatric disorder at each assessment, and McNemar tests were used to examine changes over time in these proportions within conditions. For the subset of cases with ADHD at intake (n = 96), we also conducted 2-way repeated measures ANOVAs to evaluate the acute effects of medication (treatment condition × study medication) on all 11 child outcomes and Chi-square tests based on the rates of ADHD or ODD at posttreatment.

Repeated measures mixed effects models were conducted as primary analyses to evaluate the effectiveness of PONI (vs. EUC) on child outcomes through one-year follow-up. Descriptive analyses suggested that neither a linear nor quadratic curve model adequately described the change in outcomes over time. For most continuous outcomes, there was improvement from pre- to post-treatment, with change slowing during follow-up. Therefore, we adopted a repeated measures mixed-effect model with terms including treatment assignment, time as a factor, and the interactions of time and treatment assignment. We also included a random subject in the model allowing for different individual variability.

RESULTS

Treatment Involvement: Initiation, Participation, and Termination Status

On the TSR, 80 study nurses in PONI, and 24 specialists and eight PCPs in EUC, reported that they delivered services to the case. As shown in Table 2, children randomized to PONI were significantly more likely to initiate (access) their assigned treatment condition, receive more direct treatment and adjunctive services and a longer duration of treatment, and have sibling participation than those in EUC. PONI tended to have a greater number of sessions and a higher proportion of cases receiving psychosocial treatment only, although approximately three-quarters of all cases received psychosocial treatment only. Medication was prescribed for ADHD in 17 PONI children and 10 EUC children. PCPs prescribed the medication for all 17 PONI cases and 6 of the 10 EUC cases.

Table 2.

Characteristics of the Services Provided in the Two Treatment Conditions (N = 163)

Measure	PONI (n = 83)		EUC (n = 80)		Group Comparisons
	n	%	n	%	χ2	df
Treatment Summary Report
Cases who accessed their assigned treatment	80	96.4	32	40.0	59.59c	162
Child participants during treatment
Identified patient	75	93.8	30	93.8	0.00	112
Sibling(s)	32	40.0	6	19.4	4.41a	111
Type of services delivered by primary provider					5.54^	112
Psychosocial only	63	78.8	22	68.8
Medication only	0	0.0	2	6.3
Both	17	21.3	8	25.0
Cases that completed their treatment regimens as proposed or at a mutually agreed upon time	71	88.8	18	58.1	13.28c	111
Case disposition at discharge					58.26c	106
No additional services needed or requested	41	52.6	9	32.1
Continuation of current treatment regimen	0*	0.0	16	57.1
Referral for alternative or follow-up treatment	37	47.4	3	10.7
	M	SD	M	SD	ANOVA	df
No. of weeks in treatment	17.2	4.9	12.4	9.9	10.77c	103
No. of treatment visits attended	8.8	2.9	7.2	607	3.32^	109
No. of direct treatment hours provided	11.6	4.0	8.0	7.4	10.31b	109
No. of adjunctive treatment hours provided	4.9	2.0	1.6	4.8	22.30c	67
Consumer Satisfaction Questionnaire-8	28.2	8.2	21.7	6.6	33.71c	97
Barriers to Treatment Participation Scale
Relevance of treatment	4.7	4.5	6.6	4.5	4.59a	100
Relationship with therapist	0.9	2	3.8	2.8	25.33c	99
Treatment issues/demands	1.7	2.5	3.1	2.3	4.56a	100
Competing activities	7.2	6.0	6.9	6.3	0.04	145
Total (M)	14.5	11.2	20.1	12.7	5.61a	100

^{^}= .07;

^a=p < 0.05,

^b=p < 0.01,

^c= p < 0.001

^*= By study design, PONI was not continued past the post-treatment assessment.

At termination, PONI children were more likely to complete their agreed-upon treatment regimens and to receive a case disposition indicating no further need for services or outside referral than EUC children. Across both conditions, non-completers and completers were comparable on all intake demographic and clinical variables.

Consumer Satisfaction and Barriers to Treatment

Caregivers of PONI (vs. EUC) cases reported significantly greater satisfaction (CSQ-8) and fewer obstacles to treatment on three of the four BTPS factors. Many parents in PONI noted the advantage of on-site services (e.g., acceptable, close to home).

Outcomes for Clinical Rating Scales

Acute Effects

Table 3 presents the results of the analyses comparing PONI and EUC on ratings of the child outcomes collected at pre- and post-treatment. Relative to EUC, PONI was associated with significantly greater, albeit modest, improvement on the IGA R severity rating of the child’s primary behavior problem and the overall CHIP adjustment score. Analyses revealed small to moderate improvements across time on all other ratings of behavioral and emotional problems for both conditions. EUC was not superior to PONI on any outcome measure.

Table 3.

Child Clinical Outcomes at Pre-Assessment and Post-Assessment for the Two Treatment Conditions (N = 163)

	PONI (n=83)				EUC (n=80)				Group Comparisons
Variable	Pre		Post		Pre		Post		Time	Group × Time
	M	SD	M	SD	M	SD	M	SD	F	F	df
Primary Child Outcomes
IGAR average	1.0*	0.0*	2.8	0.8	1.0*	0.0*	2.6	0.8	71.51c	3.70a	1,147
PSC-17 - Externalizing	8.7	2.0	6.1	3.2	8.6	2.5	6.0	2.8	136.49c	0.01	1,148
SDQ-P total	21.2	5.9	19.1	6.3	21.1	5.9	19.3	5.1	25.83c	0.22	1,149
SDQ-T total	18.5	6.0	17.4	5.2	17.8	5.2	17.8	5.2	1.71	1.00	1,105
BERS –Interpersonal Strength	19.7	7.2	23.5	6.5	20.0	7.7	24.2	6.6	32.13 c	0.08	1,146
CHIP total	47.3	5.9	49.5	5.9	48.7	6.0	48.9	6.1	7.43b	4.07a	1,146
Secondary Child Outcomes
PSC-17 – Attention	5.7	3.0	4.4	2.7	6.2	2.4	4.6	2.6	46.57c	0.19	1,147
PSC-17 – Internalizing	4.2	2.8	2.5	2.5	4.1	2.5	2.6	1.9	48.74c	0.17	1,147
SCARED – Abbreviated	1.8	1.8	1.3	1.5	2.1	1.9	1.5	1.6	15.91c	0.17	1,146
SMFQ	15.4	10.5	11.1	10.2	17.0	10.3	12.8	9.8	11.03c	0.01	1,147
CIS	18.9	6.7	13.4	7.9	18.9	6.7	13.4	6.3	83.90c	0.01	1,149

IGAR = Individualized Goal Achievement Rating; PSC-17 = Pediatric Symptom Checklist-17; SDQ-P/T = Strengths and Difficulties Questionnaire total score-parent/teacher version; BERS = Behavior and Emotional Rating Scale; CHIP = Child Health and Illness Profile total score; SCARED = Screen for Child Anxiety and Related Emotional Disorders; SMFQ = Short Mood and Feelings Questionnaire; CIS = Columbia Impairment Scale

Pre = pre-assessment; Post = post-assessment

^*All baseline targets were assigned a rating of 1.0.

^a=p<0.05,

^c=p<0.001

We also examined the role of medication treatment for children with ADHD in the two conditions. In this subgroup, ANOVAs revealed no significant main effects of medication on the 6 primary behavioral and 5 secondary child outcomes (p’s < .78 - .08).

Follow-up Effects

Treatment effects across all time points were studied using repeated measures mixed effects models. Both treatments were associated with clinical improvements at follow-up on all 11 outcomes (p’s < .02 - .0001), with PONI showing superiority to EUC on the overall CHIP average score (F (4,552) = 2.61, p < .03) and tending to show greater reduction in teacher’s SDQ total scores (p < .06). To evaluate the maintenance of treatment gains, we compared the scores at post-treatment and one-year follow-up. Gains were generally sustained for most outcomes, with continued improvement documented on the IGAR average (t = 7.76, p < .0001), CIS (t = 3.33, p < .0001), MFQ (t = 5.97, p < 0.0001), SCARED-abbreviated (t = 2.7, p < .01), and teacher SDQ total scores (t = 2.29, p < .02).

Outcomes for Psychiatric Disorders

Acute Effects

As shown in Table 4, the proportion of PONI and EUC cases diagnosed with ADHD, ODD or CD, any mood/anxiety disorder, and any Axis I disorder were comparable at pre- and post-treatment. Figure 2 depicts the remission rates for ADHD and either ODD or CD at post-treatment. One-third of the cases in PONI and EUC were in remission for ADHD, and 53% and 41% had remitted from either ODD or CD, respectively. Based on McNemar tests, both conditions showed significant reductions in the rates of ADHD and ODD or CD at posttreatment (p’s < .001). ANOVAs comparing the mean number of diagnoses for PONI and EUC at pre-treatment (2.1, 2.0) and post-treatment (1.5, 1.4) revealed a significant effect over time (F = 48.36, p < .001), but no significant group x time interaction (F = 0.01, p < .89).

Table 4.

Psychiatric Diagnoses and Group Comparisons at each Timepoint (N = 163)

	PONI (n=83)						EUC (n=80)						Group Comparisons
Diagnosis	Pre		Post		1YFU		Pre		Post		1YFU		Pre	Post	1YFU
	n	%	n	%	n	%	n	%	n	%	n	%	χ2	χ2	χ2
ADHD	51	61.4	30	40.5	20	33.9	47	58.8	31	42.5	23	42.6	0.12	0.06	0.90
ODD or CD	53	63.9	27	36.0	25	37.3	50	62.5	27	36.5	18	30.0	0.03	0.01	0.76
Any Mood/Anxiety	16	19.3	17	22.7	13	19.4	15	18.8	13	17.6	10	16.7	0.01	0.60	0.16
Any Axis I	78	94.0	55	73.3	45	69.2	75	93.2	58	78.4	38	63.3	0.01	0.52	0.63

Pre = pre-assessment; Post = post-assessment; 1YFU = One-year follow-up

ADHD = Attention Deficit Hyperactivity Disorder; ODD = Oppositional Defiant Disorder;

CD = Conduct disorder

Note. χ² = Chi-Square test. No group comparisons reached statistical significance.

Figure 2.

Remission Rates for Primary Disorders at Each Timepoint for the PONI and EUC Conditions.

Follow-up Effects

As shown in Table 4, both conditions had comparable rates for all of the same disorders at one-year follow-up. As shown in Figure 2, the remission rates for ODD or CD in both conditions were even higher by follow-up (p’s < .001), but the remission rate for ADHD was significant for PONI (p < .001), and not EUC (p < .18). Overall, slightly less than one-half of all cases had met criteria for remission from ADHD and over one-half from ODD or CD.

Clinical Significance of Key Outcomes

As shown in Table 5, both conditions showed comparable proportions of cases whose scores on three key outcomes fell below the clinical cutoff at each assessment, with reductions found across timepoints. PONI demonstrated superiority to EUC in reducing total clinical problems on the parent SDQ at posttreatment and, to a lesser extent, one-year follow-up. The reductions in both conditions were maintained at one-year follow-up, with more than one-half of all cases falling within the normal range.

Table 5.

Cases Scoring Below the Clinical Cutoff on Key Child Outcomes in the Two Treatment Conditions

	PONI						McNemar (p<)		EUC						McNemar (p<)		Group Comparisons
Variable	Pre		Post		1YFU		Pre:Post	Pre:1YFU	Pre		Post		1YFU		Pre:Post	Pre:1YFU	Pre	Post	1YFU
	n	%	n	%	n	%	p	p	n	%	n	%	n	%	p	p	χ2	χ2	χ2
PSC-17 Ext.	12	14.5	43	57.3	41	61.2	.0001	.0001	18	22.5	49	62.7	40	66.7	.0001	.0001	1.75	0.44	0.41
SDQ-P Total	17	20.5	32	42.1	28	41.8	.0001	.0001	20	25.0	22	29.3	23	39.0	.66	.03	0.47	2.68	0.10
CIS	28	33.7	53	66.3	61	63.5	.0001	.0001	18	22.5	47	60.2	66	72.5	.0001	.0001	2.54	0.66	2.18

Pre = pre-assessment; Post = post-assessment, 1YFU = One-year follow-up

PSC-17 Ext. = Pediatric Symptom Checklist-17 Externalizing problems; SDQ-P = Strengths and Difficulties Questionnaire-parent version total scale; CIS = Columbia Impairment Scale

Note. χ² = Chi-Square test. No group comparisons reached statistical significance.

Pediatrician Perspectives on Delivery of Mental Health Services

Pediatricians believed highly in the importance of making mental health services available in the primary care office (4.7/5.0), found few obstacles to providing such services (4.0/5.0), and expressed an interest in collaborating with an on-site clinician (5.0/5.0). PCCs expressed considerable enthusiasm with the delivery and expansion of the PONI condition.

DISCUSSION

This study evaluates the initial findings from a treatment trial targeting BPs in 163 clinically referred children (6–11 years) randomized to a protocol for on-site, nurse-administered intervention in primary care (PONI) or enhanced usual care (EUC) characterized by on-site diagnostic assessment and facilitated referral to a local mental health provider. EUC augmented traditional primary care management by reviewing results of a comprehensive diagnostic assessment with the patient, family, and managing PCC, and then aiding in linkage efforts. PONI (vs. EUC) cases were nevertheless more likely to access, complete, and be satisfied with the mental health services received, and reported greater improvements on a few clinical measures. But, both conditions experienced clinical benefits on several measures that were maintained through final follow-up. The relevance and scientific integrity of the study was enhanced by including children with disruptive disorders of concern to parents and PCCs, standardized assessment measures across multiple informants, and an evaluation of treatment integrity.

This study’s most encouraging finding is that PONI was associated with higher rates of treatment initiation and completion, a larger dose and duration of intervention, greater consumer satisfaction, and fewer barriers to treatment involvement compared to EUC. The rates of service initiation and follow-up achieved contrast with the low rates of specialty care referral (between 20 and 30%),³ service use (about a third), and follow-up visits (median = 2)¹² reported in pediatric settings.¹¹ Though less dramatic, PONI produced significantly better clinical outcomes than EUC in a few domains which included modest improvements in individualized behavioral treatment targets and overall child health/adaptive behavior. Both conditions showed reductions in externalizing, attentional, and internalizing problems and gains in social behavior and peer relations; one-third of children with ADHD and one-half with ODD or CD were in remission at post-treatment.

Follow-up analyses documented the maintenance of improvements in both conditions,⁸ including reductions in conduct and externalizing problems, attention problems, depression symptoms, functional impairment, and child health/adaptive behaviors. Indeed, one-half of the children diagnosed originally with ADHD or either ODD or CD met criteria for diagnostic remission by one-year follow-up. However, only PONI showed a significant decrease in cases showing clinically significant levels of problems at school and increase in remission from ADHD at one-year follow-up. These diagnostic outcomes are important to evaluate, but few treatment trials report them,

The findings from this clinical trial provide further support for the delivery of mental health services in pediatric primary care. The feasibility of on-site care by a pediatric nurse (PONI) was supported by high rates of treatment initiation and completion, especially a nearly 2-½ fold increase in service use over EUC, and its acceptability was demonstrated by reports of fewer barriers to treatment participation and higher consumer satisfaction. Further, PONI was at least as effective as EUC and was more effective on a few key outcomes at post-treatment and one-year follow-up, even though their clinical significance was small to modest. Although a cost analysis is beyond the scope of this report, it is worth mentioning that on-site services are now being delivered by and reimbursed for licensed clinicians in the same practices. In highlighting some advantages to the provision of on-site mental health interventions in pediatric primary care, these findings also support efforts to train nurses who may not have prior mental health experience as clinicians to deliver mental health services in pediatric primary care.^{8, 18}

PONI’s modest outcomes are consistent with those from early trials of collaborative interventions for depressed adults in primary care,^{45, 46} and other pediatric trials,^{8, 18} highlighting the need for further evaluation and refinement of integrated mental health services for children with behavior problems. In particular, PONI might have been more effective had it included more active medication management, as only some children with ADHD received stimulant medication,⁴⁷ or if intervention had continued beyond 6 months, as was often the case for EUC. Contingent upon PCP willingness and preparation, application of the chronic care model involving active care management, more active collaboration with PCCs, and greater attention to the treatment of comorbid conditions, might have been helpful.⁴⁸ Accordingly, future efforts may benefit from a broader clinical agenda; formal PCC training in and decision support for medication use, especially for ADHD^{49, 50} or comorbid anxiety;^{51, 52,53, 54} and technological advances to enhance the efficiency with which on-site services are delivered and communicated.¹⁴

Study limitations deserve mention. The comparison EUC condition received care that was enriched by the availability of a comprehensive assessment, a diagnostic summary letter with recommendations, follow-up phone calls with referred families to confirm linkage, and extended services beyond the acute care period, making this a conservative test of PONI versus usual care. PONI provided only minimal attention to the clinical comorbidities of children with behavior problems, especially ADHD. We also recognize that the effects of treatment may vary by patient background, referral source, or other treatment variables.

In summary, a nurse-administered intervention applied in the pediatric primary care office was characterized by feasibility, integrity, acceptability, and modest clinical benefit, relative to a facilitated referral to an outside specialty provider. In extending findings from the few prior studies,^{8, 18} these promising results suggest the need for further study of innovative service models⁵⁵ to address children’s psychosocial needs in primary healthcare settings, as is being done with ADHD.⁵⁶ The improvements in access to care and clinical outcomes found for children in PONI provide a foundation upon which to develop and study more collaborative interventions whose quality may be improved by incorporating evidence-based programs that provide efficient screening, treatment, follow-up, and outside referral services. Although the application of an effective model of behavioral treatment in pediatric settings may not be a substitute for some specialty services, it may certainly help to define the foundation for effective mental health service provision in the primary care office.⁵⁸