Impact of a Web-Portal Intervention on Community ADHD Care and Outcomes

Abstract

BACKGROUND:

The quality of care for children with attention-deficit/hyperactivity disorder (ADHD) delivered in community-based pediatric settings is often poor. Interventions have been developed to improve community-based ADHD care but have not demonstrated that better care results in improved patient outcomes. The objective of this study was to determine whether an ADHD quality improvement (QI) intervention for community-based pediatric practices improves patient outcomes.

METHODS:

A cluster randomized controlled trial was conducted in which 50 community-based pediatric primary care practices (213 providers) were randomized either to receive a technology-assisted QI intervention or to a control condition. The intervention consisted of 4 training sessions, office flow modification, guided QI, and an ADHD Internet portal to assist with treatment monitoring. ADHD treatment processes and parent- and teacher-rated ADHD symptoms over the first year of treatment were collected for 577 patients.

RESULTS:

Intent-to-treat analyses examining outcomes of all children assessed for ADHD were not significant (b = –1.97, P = .08). However, among the 373 children prescribed ADHD medication, there was a significant intervention effect (b = –2.42, P = .04) indicating greater reductions in parent ratings of ADHD symptoms after treatment among patients treated by intervention physicians compared with patients treated at control practices. There were no group differences on teacher ratings of ADHD symptoms. ADHD treatment care around medication was significantly better at intervention practices compared with control practices.

CONCLUSIONS:

A technology-assisted QI intervention improved some ADHD care quality and resulted in additional reductions in parent-rated ADHD symptoms among patients prescribed ADHD medications.

What’s Known on This Subject:

Although successful interventions have been developed to address the quality of attention-deficit/hyperactivity disorder (ADHD) care in community pediatric settings, these efforts have not demonstrated that improved ADHD care translates into better outcomes for patients with ADHD.

What This Study Adds:

This cluster randomized controlled trial demonstrates that use of a technology-assisted quality improvement intervention not only improves the quality of some ADHD care delivered in pediatric settings but also results in greater medication-related ADHD symptom reductions according to parent ratings.

Approximately 7% to 9% of elementary school–age children in the United States are diagnosed with attention-deficit/hyperactivity disorder (ADHD).^1,2 Most of these children receive their ADHD care from primary care providers (eg, pediatricians^3,4). Unfortunately, ADHD care quality in these settings, especially medication monitoring and titration, is often poor.^5,6 For example, most children have no contact with their physician during the first month of medication treatment.⁵

Fortunately, there are several models for improving ADHD care delivered by pediatricians.^7–11 Epstein and colleagues have developed a technology-assisted quality improvement (QI) intervention to address the quality of community-based pediatricians’ ADHD care.¹² A pilot study of this intervention demonstrated improved ADHD care quality consistent with American Academy of Pediatrics ADHD guidelines across a diverse set of practices.¹³

To date, all community-based ADHD care interventions have focused on ADHD care quality (ie, physician behaviors) as the primary outcome. However, it has not been established that improved care quality leads to improved patient outcomes. In this large, cluster-randomized controlled trial, we examined the effectiveness of a technologically assisted QI intervention at improving patient outcomes for elementary school–age children with ADHD. We predicted that children cared for by pediatricians trained on this intervention would have significantly greater ADHD symptom reduction compared with those receiving typical care.

Methods

Participants and Settings

Mailings were sent to 128 practices in central and northern Ohio inviting their participation in a study of ADHD care QI. From August 2010 to December 2012, the first 50 pediatric practices that responded and met our inclusion and exclusion criteria (ie, has ≥2 physicians, uses an electronic billing system, office has Internet access, must not have colocated mental health care) were selected to participate (see Fig 1). These practices included 195 pediatricians, 4 nurse practitioners, and 14 pediatric resident physicians. Among nonresidents, mean provider age was 43.3 (SD 9.5) years, 15% were nonwhite, and 62% were female. The average time since training completion was 12.8 years (SD 9.1). There was diversity in terms of practice location (urban: 30.6%; suburban: 59.5%; rural: 11.1%) and population served (eg, proportions of Medicaid patients in their panels, range 0%–99%; mean 44.6%, SD 30.5%).

FIGURE 1.

CONSORT (Consolidated Standards of Reporting Trials) flow diagram. ITT, intention to treat.

Intervention

Practices randomly assigned to the intervention group received four 1-hour training sessions via Web conference. Two training sessions attended by all providers at the practice focused on evidence-based ADHD care per the American Academy of Pediatrics ADHD guidelines.¹⁴ The remaining 2 sessions were attended by all practice staff including providers and focused on how to complete systematic tests of change to improve the quality of ADHD care and use the ADHD Web portal (www.myADHDportal.com). Each practice selected an ADHD champion, typically an administrator or nurse, who coordinated activities at the practice.

At each intervention practice, ADHD patient flow was redesigned to center around a Web-based portal that facilitates the collection, scoring, and interpretation of parent- and teacher-ratings of ADHD symptoms and side effects. Providers were encouraged to register all of their patients with ADHD on the Web-based portal, irrespective of whether they were enrolled in the research study. Providers can customize a schedule of rating scale collection for each patient after which ratings are automatically solicited from parents and/or teachers. When ratings are completed, automated algorithms score and interpret the data, and the physician receives a report with text, graphs, and tables charting patient response to medication. Furthermore, an online report card provides physicians with feedback on several key indicators of ADHD care (eg, time lapse between prescribing ADHD medication and collection of first parent and teacher rating scale). At 3, 6, 9, and 12 months after training, study staff contacted the offices to prompt them to review their report cards. After identifying ADHD care behaviors in need of improvement, an ADHD care behavior was selected, and a test of change¹⁵ was implemented. By participating in this intervention, physicians earned maintenance of certification credit from the American Board of Pediatrics.

Study Design

The intervention requires modifying ADHD patient flow at the practice level, therefore a cluster randomized controlled trial was required. See Fig 1 for CONSORT (Consolidated Standards of Reporting Trials) diagram. Pairs of practices were matched according to number of providers in the practice and the percentage of Medicaid patients served. Within each pair, a practice was randomly assigned by a blinded research assistant using a random number generator to either immediate intervention or a wait-list control group. Practices were not blind to their assigned condition.

During the first year after randomization, pediatricians across both groups identified patients under their care who were in grades 1 through 5, presenting for ADHD evaluation, and were ADHD medication naive. If families agreed, research staff called families and obtained verbal consent to participate. Across intervention and control practices, 2.4 (SD 3.1) and 3.6 (SD 3.4) families per provider were enrolled respectively. After consenting and before treatment began, the Vanderbilt ADHD Parent Rating Scale¹⁶ was completed. Consent to contact the child’s teacher was also obtained, and teachers completed the Vanderbilt ADHD Teacher Rating Scale.¹⁷ Using the Vanderbilt Scales, which are based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, parents and teachers rate each of 18 ADHD symptoms as occurring “never” (0), “occasionally” (1), “often” (2), or “very often” (3). An ADHD total symptom score (TSS; range 0–54) is derived by adding the 18 symptom scores. Research staff members contacted the family at 3, 6, and 12 months after the initial contact to readminister parent ratings. Because of changes in teachers across grades, collection of teacher ratings was only attempted at 3 and 6 months after baseline. Attempts to contact parents and teachers continued until data collection was completed in December 2014.

At the conclusion of the study, the patient’s medical chart and ADHD portal record (at intervention practices) were reviewed to document 1 year of ADHD care dating back to the first ADHD-related contact. Chart data included the following: (1) dates of any ADHD-related contacts (ie, office visit, phone, or e-mail); (2) dates, medication, dosage, and number of pills dispensed for any ADHD medication prescribed; and (3) dates of any ADHD parent- and teacher-rating scales. Interrater reliability for chart reviews was calculated using a random sampling of 20% of patient charts, with intraclass correlations for continuous data and kappas for dichotomous data all exceeding 0.70.

To estimate medication continuity, the number of days covered by medication prescriptions (assuming a 7 days per week dosing schedule) was calculated and then converted to the proportion of days covered by dividing by the number of days from treatment start date until the end of the study. For children prescribed stimulant medication, the daily dosage for the final prescription written was calculated in methylphenidate-equivalent milligrams by converting nonmethylphenidate stimulant medications using the following conversions: mixed salt amphetamines dose or Focalin dose × 2; Vyvanse × 0.8.

The Cincinnati Children’s Medical Center and Nationwide Children’s Hospital Institutional Review Boards approved this study. Physicians gave written and parents gave verbal recorded, informed consent (over the phone) for participation.

Statistical Analyses

To compare rates of ADHD care across intervention and control practices, chart review data were analyzed by using 3-level mixed models (patients nested within providers, providers nested within practice, and variation across practices) using SAS Proc Mixed. Some ADHD care variables were measured by the number of days from when the patient was initially prescribed medication until a care event occurred (eg, time to first contact after being prescribed medication). When events did not occur, the observation was right-censored. For these variables, Cox proportional hazards regressions (including clustering of patients under providers and providers under practices and using robust standard errors) were conducted comparing time-to-events across the intervention and control groups (function coxme in R Version 3.01). Because the goal of the intervention was to improve ADHD care around medication treatment, all analyses comparing ADHD care across intervention and control groups were limited to children who were prescribed ADHD medication (n = 373).

Using the full sample (n = 577), an intent-to-treat analysis was conducted comparing patient outcomes across intervention and control groups using 4-level (repeated longitudinal assessments nested within patients, patients nested within care providers, care providers nested within practices, and variation across practices) mixed models (SAS Proc Mixed). Because the intervention was likely only to exert an effect on patients receiving treatment, a second set of analyses were conducted by using only children who were prescribed ADHD medication (n = 373). Separate models were conducted for parent and teacher ratings. Baseline ratings were entered as a covariate. Postbaseline ratings were entered as dependent variables. The research team attempted to collect postbaseline ratings at 3, 6, and 12 months postbaseline. The actual dates when ratings were collected varied considerably (see Supplemental Table 2). Hence, in our analyses of patient outcomes, a time variable indicating the number of days postbaseline was entered to account for the multiple postbaseline time points. Group (intervention vs control) was entered to signify group status. The statistical significance of the main effect of group indicated intervention effects. Group × time interactions were included to test for differential posttreatment trajectories across groups. However, most long-term ADHD medication studies show that the effects of medication on ADHD symptoms occur immediately (ie, within the first 3 months) and then are fairly stable.^18,19 Hence, an interaction was not expected. Because no models produced significant interactions, these interaction effects were dropped from the models. To calculate effect sizes for significant effects in these multilevel models, the proportion of reduction in variance (PRV²⁰) was calculated.

Missing Data

Though we collected baseline parent ratings of ADHD symptoms for the entire sample (n = 577), baseline teacher ratings were collected for only 64% (n = 367) of the sample. We collected at least 1 postbaseline parent rating and 1 postbaseline teacher rating for 90% and 74% of the sample, respectively. Rates of missing data across the intervention and control groups were not significantly different (all Ps > .5). There were a few demographic and clinical differences across those with and without missing data (see Supplemental Information). Multiple imputation was not used because, to our knowledge, there is no software that can impute data with 4 levels of nesting.

Power

Monte Carlo simulation power analyses with 3 levels of nesting indicated we had 80% power to detect a Cohen’s d effect size of 0.20 between intervention and control groups when examining ADHD care quality. For 4-level nesting, we had power of 0.80 to detect a between-group unstandardized difference of 2.0 and 2.6 on postbaseline parent and teacher ratings of ADHD symptoms in the intent-to-treat analyses. See Supplemental Information for power analysis details.

Results

ADHD Treatment Care

We enrolled 577 patients (258 from intervention practices; 319 from control practices). Children averaged 7.8 (SD 1.4) years of age, 70.5% were male, and 36.7% were nonwhite. Charts were reviewed for 574 patients. The 3 charts not reviewed were from a control practice. These reviews showed that 373 patients (165 at intervention practices; 208 at control practices) were prescribed ADHD medication in the year after initial presentation. Differences between intervention and control practices in patient age and sex were nonsignificant (see Supplemental Table 3). There was a race difference across groups in which the control group had a greater proportion of nonwhite children (42.3%) than the intervention group (29.8%; P < .01). Hence, all of the analyses examining intervention effects covary for race using a dichotomous white/nonwhite race indicator.

Among the 373 children who received ADHD medication, children at intervention practices had significantly more treatment contacts and more parent and teacher ratings to monitor medication treatment compared with children treated at control practices (see Table 1). Cox analyses indicated that after being prescribed medication, children at intervention practices had a shorter time to first contact (see Fig 2); however, this was due to use of phone contacts, not office visits, for the initial contact after prescribing medication (see Fig 3). Cox analyses also demonstrated the treatment outcomes of children at intervention practices were more quickly assessed with parent and teacher ratings during the first year of treatment (see Fig 2).

TABLE 1.

Differences in Rates of ADHD Treatment Care Across Intervention and Control Groups

	Control (n = 208)	Intervention (n = 165)	P Level for Group Difference	Cohen’s d
Number of contacts in first year	6.58 contactsa (0.31)	8.21 contactsb (0.32)	.0008	0.38
Number of parent scales to monitor treatment in first year	0.54 scales (0.31)	3.04 scales (0.33)	<.0001	0.57
Number of teacher scales to monitor treatment in first year	0.26 scales (0.34)	2.83 scales (0.36)	<.0001	0.54
Proportion of days covered by medication prescriptions	54.43% (1.98)	59.00% (2.15)	.09	0.16
Of children taking stimulants (n = 342), final daily dosec	23.40 mg (1.07)	25.17 mg (1.14)	.27	0.12

^aContacts included 75% office visits, 25% phone, and 0% e-mail.

^bContacts included 59% office visits, 39% phone, and 2% e-mail.

^cMethylphenidate or methylphenidate equivalent milligrams.

FIGURE 2.

Cox analyses examining time to key ADHD treatment care events across intervention and control groups. CI, confidence interval.

FIGURE 3.

Cox analyses examining time to first office visit and first phone contact after starting medication across intervention and control groups. CI, confidence interval.

There were no group differences in the proportion of days covered by prescriptions (P = .09), the ultimate medication dosage among those prescribed stimulants (n = 342; P = .27) or time to first medication change (P = .17). Proportions of children meeting ADHD Healthcare Effectiveness Data and Information Set (HEDIS) measures across groups are reported in supplemental materials.

ADHD Patient Outcomes

Intent-to-treat analyses including all enrolled patients (n = 577; practice-level interclass correlation [ICC] = 0.04; provider-level ICC = 0.01; patient-level ICC = 0.40) revealed a statistically significant effect of race (b = 1.67, P > .009) but nonsignificant effects of time (b = –0.003, P = .06) and group (b = –1.97, P = .08).

Among children who were prescribed ADHD medication (n = 373; practice-level ICC = 0.04; provider-level ICC = 0.01; patient-level ICC = 0.46), there was a significant effect of group (b = –2.42, P = .04) along with nonsignificant effects of race (b = 1.10, P = .14) and time (b = –0.001, P = .43) on parent ratings of ADHD symptoms. The estimate for the group effect indicates a 2.42-point incremental reduction in parent-rated ADHD total symptom scores for children treated at intervention practices compared with control practices (Fig 4). The PRV effect size estimate indicated that compared with a null model (level 4 TSS variance = 4.38), the inclusion of treatment group as a predictor variable (level 4 TSS variance = 3.12) reduced response variable variance by 29% (PRV = [4.38–3.12]/4.38 = 0.29).

FIGURE 4.

ADHD symptom reduction among patients treated by physicians randomly assigned to the intervention and control groups.

Analyses described using parent ratings were repeated using teacher ratings. All group effects were nonsignificant.

Discussion

The technology-assisted QI intervention had large effects on several indices of ADHD treatment care quality delivered by community-based providers. In particular, improvements were observed in physicians’ monitoring of their patients with ADHD during the initial year of treatment; gains consistent with previously reported intervention-related improvements.¹³ Compared with the usual care group, providers in the intervention group had 25% more patient contacts and collected 4.6 and 9.9 times more parent and teacher ratings, respectively. However, even the quality of care achieved at the intervention practices left much room for improvement. For example, providers did not collect parent or teacher ratings during the initial year of ADHD medication treatment of half of their patients. Also, medication dosages were lower than what is achieved with optimal titration.²¹

Consistent with an extensive literature on ADHD medication efficacy,^18,22 there were large reductions in parent-rated ADHD symptoms when newly diagnosed children with ADHD in both groups were prescribed ADHD medication (see Fig 4). The statistically significant incremental reduction (2.4 TSS points) in parent-rated ADHD symptoms over and above these medication effects reflects the incremental impact of improved quality of ADHD care on patient outcomes, achieved using our technology-assisted QI intervention. However, notably, intervention effects were not detected on teacher ratings of ADHD symptoms. It is possible that higher rates of missing data on teacher ratings led to reduced power to detect intervention effects.

Although we found that improved ADHD care improves parent-rated ADHD symptoms, the effects of our intervention were limited in that providers at the intervention practices did not consistently implement some standards of ADHD care. In fact, 7 of the 106 providers assigned to the intervention group did not register any children on the ADHD portal. Indeed, some of these care behaviors (eg, completion of rating scales) require parent and teacher compliance and access to the Internet in addition to provider action. For parents and teachers, education about the advantages of periodic visits and monitoring of their child’s medication treatment may help. Using technology (eg, online education materials) or peer support (eg, parent advocate) may spread this message more effectively to parents and teachers. For providers, while the ADHD portal intervention does alleviate some obstacles (eg, distribution and scoring of parent and teacher ratings), supplemental supports are likely necessary to facilitate consistent implementation. It may be that improving reimbursement for collection of these scales or according to compliance with specific standards of care may also reinforce reliable implementation of quality ADHD care. However, existing ADHD care quality standards such as the HEDIS measures may need to be reconsidered. In particular, the HEDIS measures rely heavily on office visits. Results of this intervention study suggest that increasing the rates and timing of any form of contact, either phone or visits, may improve child outcomes. Hence, less reliance on office visits and more emphasis on alternative forms of monitoring (eg, phone contacts, rating scale collection) for monitoring medication should likely be considered.

As is often the case with community-based research studies, collection of research measures was difficult, largely due to the research staff having minimal contact with research participants. Therefore, data collection rates were compromised. Moreover, recruitment of patients and continued participation in research assessments was lower at intervention practices than control practices. One possible contributor to this disparity was the increased demands for providers to adopt the ADHD portal into practice and the increased demands among families to complete both clinical and research assessments. Additionally, although the participating practices were diverse in terms of geography and populations served, they all volunteered to participate in an ADHD QI study, so our results may not generalize to all practices and providers. Likewise, patients included only those who volunteered to participate, which also may limit generalizability of our findings. Another limitation was that our primary outcome was ADHD symptoms rather than functional impairments (eg, school performance), which are generally the reason that families seek treatment.²³ The rationale for focusing on symptoms was that medication largely targets symptoms and typically does not produce change in function, particularly in community-based settings.^19,24 To improve functioning, behavioral treatments must be employed.^25,26 This leads to a final limitation of this study: our intervention focused on medication and did not assist providers in accessing or delivering behavioral treatments.

Conclusions

Using a technology-assisted QI intervention improves some indicators of ADHD care quality (eg, frequent contacts, more rating scales collected) in community-based pediatric settings, which produces incrementally greater reductions in parent-rated ADHD symptoms among patients treated with medication compared with typical care. This intervention is largely Web-based, facilitating dissemination and adoption. Future goals for this intervention include extending the software to facilitate behavioral treatment and exploring strategies to enhance consistent implementation across all patients with ADHD (eg, increased integration with electronic health record, pay for performance) and other pediatric mental disorders.

Glossary

ADHD

attention-deficit/hyperactivity disorder

HEDIS

Healthcare Effectiveness Data and Information Set

ICC

interclass correlation

PRV

proportion of reduction in variance

QI

quality improvement

TSS

total symptom score