Impact of a web-portal intervention on community ADHD care and outcomes

Abstract

Background

The quality of care for children with attention-deficit/hyperactivity disorder (ADHD) delivered in community-based pediatric settings is often poor. Interventions have been developed to improve community-based ADHD care but have not demonstrated that better care results in improved patient outcomes. The objective of this study was to determine whether an ADHD quality improvement (QI) intervention for community-based pediatric practices improves patient outcomes.

Methods

A cluster randomized controlled trial was conducted whereby 50 community-based pediatric primary care practices (213 providers) were randomized to receive either a technology-assisted QI intervention or to a control condition. The intervention consisted of four training sessions, office flow modification, guided QI, and an ADHD internet portal to assist with treatment monitoring. ADHD treatment processes and parent- and teacher-rated ADHD symptoms over the first year of treatment were collected for 577 patients.

Results

Intent-to-treat analyses examining outcomes of all children assessed for ADHD were not significant (b=−1.97, p=.08). However, among the 373 children prescribed ADHD medication, there was a significant intervention effect (b=−2.42, p=.04) indicating greater reductions in parent-ratings of ADHD symptoms after treatment among patients treated by intervention physicians compared to patients treated at control practices. There were no group differences on teacher ratings of ADHD symptoms. ADHD treatment care around medication was significantly better at intervention practices compared to control practices.

Conclusions

A technology-assisted QI intervention improved some ADHD care quality and resulted in additional reductions in parent-rated ADHD symptoms among patients prescribed ADHD medications.

Approximately 7–9% of elementary-school aged children in the US are diagnosed with attention-deficit/hyperactivity disorder (ADHD).^1,2 Most of these children receive their ADHD care from primary care providers (e.g., pediatricians^3,4). Unfortunately, ADHD care quality in these settings, especially medication monitoring and titration, is often poor.^5,6 For example, most children have no contact with their physician during the first month of medication treatment.⁵

Fortunately, there are several models for improving ADHD care delivered by pediatricians.^7–11 Epstein and colleagues have developed a technology-assisted quality improvement (QI) intervention to address the quality of community-based pediatricians’ ADHD care.¹² A pilot study of this intervention demonstrated improved ADHD care quality consistent with American Academy of Pediatrics (AAP) ADHD guidelines across a diverse set of practices.¹³

To date, all community-based ADHD care interventions have focused on ADHD care quality (i.e., physician behaviors) as the primary outcome. However, it has not been established that improved care quality leads to improved patient outcomes. In this large cluster-randomized controlled trial (RCT), we examined the effectiveness of a technologically-assisted QI intervention at improving patient outcomes for elementary-school aged children with ADHD. We predicted that children cared for by pediatricians trained on this intervention would have significantly greater ADHD symptom reduction compared to those receiving typical care.

Patients and Methods

Participants and Settings

Mailings were sent to 128 practices in central and northern Ohio inviting their participation in a study of ADHD care QI. From August 2010 to December 2012, the first 50 pediatric practices that responded and met our inclusion/exclusion criteria (i.e., has >= two physicians, uses an electronic billing system, office has internet access, must not have co-located mental health care) were selected to participate (see Figure 1). These practices included 195 pediatricians, 4 nurse practitioners, and 14 pediatric resident physicians. Among non-residents, mean provider age was 43.3 (SD=9.5) years, 15% were non-white, and 62% were female. The average time since training completion was 12.8 years (SD=9.1). There was diversity in terms of practice location (urban: 30.6%; suburban: 59.5%; rural: 11.1%) and population served (e.g., proportions of Medicaid patients in their panels (range=0–99%; mean=44.6%, SD=30.5%).

Figure 1.

CONSORT flow diagram

Intervention

Practices randomly assigned to the intervention group received four 1-hour training sessions via web conference. Two training sessions attended by all providers at the practice focused on evidence-based ADHD care per the AAP ADHD guidelines.¹⁴ The remaining two sessions were attended by all practice staff including providers and focused on how to complete systematic tests of change to improve the quality of ADHD care and use the ADHD web portal (www.myADHDportal.com). Each practice selected an ADHD champion, typically an administrator or nurse, who coordinated activities at the practice.

At each intervention practice, ADHD patient flow was re-designed to center around a web-based portal that facilitates the collection, scoring, and interpretation of parent- and teacher-ratings of ADHD symptoms and side effects. Providers were encouraged to register all of their patients with ADHD on the web-based portal, irrespective of whether they were enrolled in the research study. Providers can customize a schedule of rating scale collection for each patient after which ratings are automatically solicited from parents and/or teachers. When ratings are completed, automated algorithms score and interpret the data, and the physician receives a report with text, graphs, and tables charting patient response to medication. Further, an online report card provides physicians with feedback on several key indicators of ADHD care (e.g., time lapse between prescribing ADHD medication and collection of first parent- and teacher rating scale). At 3-, 6-, 9-, and 12-months after training, study staff contacted the offices to prompt them to review their report cards. After identifying ADHD care behaviors in need of improvement, an ADHD care behavior was selected and a test of change¹⁵ was implemented. By participating in this intervention, physicians earned maintenance of certification credit from the American Board of Pediatrics.

Study Design

The intervention requires modifying ADHD patient flow at the practice level, therefore a cluster randomized controlled trial was required. See Figure 1 for CONSORT diagram. Pairs of practices were matched according to number of providers in the practice and the percentage of Medicaid patients served. Within each pair, a practice was randomly assigned by a blinded research assistant using a random number generator to either immediate intervention or a wait-list control group. Practices were not blind to their assigned condition.

During the first year after randomization, pediatricians across both groups identified patients under their care who were in grades 1–5, presenting for ADHD evaluation, and who were ADHD medication naive. If families agreed, research staff called families and obtained verbal consent to participate. Across intervention and control practices, 2.4 (SD=3.1) and 3.6 (SD=3.4) families per provider were enrolled respectively. After consenting and before treatment began, the Vanderbilt ADHD Parent Rating Scale¹⁶ was completed. Consent to contact the child’s teacher was also obtained, and teachers completed the Vanderbilt ADHD Teacher Rating Scale.¹⁷ The Vanderbilt Scales are DSM-IV-based scales where parents and teachers rate each of the 18 ADHD symptoms as occurring “never” (0), “occasionally” (1), “often” (2), or “very often” (3). An ADHD total symptom score (TSS; range = 0–54) is derived by adding the 18 symptom scores. Research staff again contacted the family at 3-, 6-, and 12-months after the initial contact to re-administer parent-ratings. Due to changes in teachers across grades, collection of teacher-ratings was only attempted at 3- and 6-months post-baseline. Attempts to contact parents and teachers continued until data collection was completed in December, 2014.

At the conclusion of the study, the patient’s medical chart and ADHD portal record (at intervention practices) were reviewed to document one year of ADHD care dating back to the first ADHD-related contact. Chart data included: 1) dates of any ADHD-related contacts (i.e., office visit, phone, or email); 2) dates, medication, dosage, and number of pills dispensed for any ADHD medication prescribed; and 3) dates of any ADHD parent- and teacher-rating scales. Inter-rater reliability for chart reviews was calculated using a random sampling of 20% of patient charts, with intra-class correlations for continuous data and kappas for dichotomous data all exceeding .70.

To estimate medication continuity, the number of days covered by medication prescriptions (assuming a 7 day per week dosing schedule) was calculated and then converted to the proportion of days covered by dividing by the number of days from treatment start date until the end of the study. For children prescribed stimulant medication, the daily dosage for the final prescription written was calculated in methylphenidate-equivalent milligrams by converting non-methylphenidate stimulant medications using the following conversions (mixed salt amphetamines dose or Focalin® dose × 2; Vyvanse® × .8).

The Cincinnati Children’s Medical Center and Nationwide Children’s Hospital Institutional Review Boards approved this study. Physicians gave written and parents gave verbal recorded, informed consent (over the phone) for participation.

Statistical Analyses

To compare rates of ADHD care across intervention and control practices, chart review data were analyzed using three-level mixed models (patients nested within providers, providers nested within practice, and variation across practices) using SAS Proc Mixed. Some ADHD care variables were measured by the number of days from when the patient was initially prescribed medication until a care event occurred (e.g., time to first contact after being prescribed medication). When events did not occur, the observation was right-censored. For these variables, Cox proportional hazards regressions (including clustering of patients under providers and providers under practices and using robust standard errors) were conducted comparing time-to-events across the intervention and control groups (function coxme in R Version 3.01). Since the goal of the intervention was to improve ADHD care around medication treatment, all analyses comparing ADHD care across intervention and control groups were limited to children who were prescribed ADHD medication (n=373).

Utilizing the full sample (n=577), an intent-to-treat analyses was conducted comparing patient outcomes across intervention and control groups using four-level (repeated longitudinal assessments nested within patients, patients nested within care providers, care providers nested within practices, and variation across practices) mixed models (SAS Proc Mixed). Since the intervention was likely to only exert an effect on patients receiving treatment, a second set of analyses were conducted using only children who were prescribed ADHD medication (n=373). Separate models were conducted for parent- and teacher-ratings. Baseline ratings were entered as a covariate. Post-baseline ratings were entered as dependent variables. The research team attempted to collect post-baseline ratings at 3-, 6-, and 12-months post-baseline. The actual dates when ratings were collected varied considerably (see eTable 1). Hence, in our analyses of patient outcomes, a time variable indicating the number of days post-baseline was entered to account for the multiple post-baseline time points. Group (intervention vs. control) was entered to signify group status. The statistical significance of the main effect of group indicated intervention effects. Group × time interactions were included to test for differential post-treatment trajectories across groups. However, most long-term ADHD medication studies show that the effects of medication on ADHD symptoms occur immediately (i.e., within the first 3 months) and then are fairly stable thereon.^18,19 Hence, an interaction was not expected. Since no models produced significant interactions, these interaction effects were dropped from the models. To calculate effect sizes for significant effects in these multi-level models, the proportion of reduction in variance (PRV²⁰) was calculated.

Missing Data

Though we collected baseline parent-ratings of ADHD symptoms for the entire sample (n=577), baseline teacher ratings were collected for only 64% (n=367) of the sample. We collected at least one post-baseline parent-rating and one post-baseline teacher rating for 90% and 74% of the sample respectively. Rates of missing data across the intervention and control groups were not significantly different (all ps > .5). There were a few demographic and clinical differences across those with and without missing data (see supplemental text). Multiple imputation was not used because, to our knowledge, there is no software that can impute data with four levels of nesting.

Power

Monte Carlo simulation power analyses with three levels of nesting indicated we had 80% power to detect a Cohen’s d effect size of .20 between intervention and control groups when examining ADHD care quality. For four-level nesting, we had power of .80 to detect a between-group unstandardized difference of 2.0 and 2.6 on post-baseline parent- and teacher-ratings of ADHD symptoms in the intent-to-treat analyses. See supplemental text for power analysis details.

Results

ADHD treatment care

We enrolled 577 patients (258 from intervention practices; 319 from control practices). Children averaged 7.8 (SD=1.4) years of age, 70.5% were male, and 36.7% were non-white. Charts were reviewed for 574 patients. The three charts not reviewed were from a control practice. These reviews showed that 373 patients (165 at intervention practices; 208 at control practices) were prescribed ADHD medication in the year following initial presentation. Differences between intervention and control practices in patient age and gender were non-significant. See eTable 2. There was a race difference across groups whereby the control group had a greater proportion of non-white children (42.3%) than in the intervention group (29.8%; p < .01). Hence, all of the analyses examining intervention effects covary for race using a dichotomous white/non-white race indicator.

Among the 373 children who received ADHD medication, children at intervention practices had significantly more treatment contacts and more parent- and teacher-ratings to monitor medication treatment compared to children treated at control practices. See Table 1. Cox analyses indicated that after being prescribed medication, children at intervention practices had a shorter time to first contact (see Figure 2), however this was due to use of phone contacts, not office visits, for the initial contact after prescribing medication (see Figure 3). Cox analyses also demonstrated the treatment outcomes of children at intervention practices were more quickly assessed with parent- and teacher- ratings during the first year of treatment. See Figure 2.

Table 1.

Differences in rates of ADHD treatment care across intervention and control groups

	Control(n = 208)	Intervention(n = 165)	p-level for group difference	Cohen’s d
Number of contacts in first year	6.58 contactsa (.31)	8.21 contactsb (.32)	.0008	0.38
Number of parent scales to monitor treatment	.54 scales(.31)	3.04 scales(.33)	< .0001	0.57
Number of teacher scales to monitor treatment in first year	.26 scales(.34)	2.83 scales(.36)	< .0001	0.54
Proportion of days covered by medication prescriptions	54.43%(1.98)	59.00%(2.15)	.09	0.16
Of children taking stimulants (n = 342), final daily dosec	23.40 mg(1.07)	25.17 mg(1.14)	.27	0.12

^a75% visits; 25% phone; 0% email;

^b59% visits, 39% phone, 2% email;

^cmethylphenidate or methylphenidate equivalent milligrams

Figure 2.

Cox analyses examining time to key ADHD treatment care events across intervention and control groups

Figure 3.

Cox analyses examining time to first office visit and first phone contact after starting medication across intervention and control groups

There were no group differences in the proportion of days covered by prescriptions (p = .09), the ultimate medication dosage among those prescribed stimulants (n = 342; p = .27), or time to first medication change (p = .17). Proportions of children meeting ADHD Healthcare Effectiveness Data and Information Set (HEDIS) measures across groups are reported in supplemental materials.

ADHD patient outcomes

Intent-to-treat analyses including all enrolled patients (n = 577; practice-level ICC = .04; provider-level ICC = .01; patient-level ICC = .40) revealed a statistically significant effect of race (b = 1.67, p>.009) but non-significant effects of time (b = −.003, p = .06) and group (b = −1.97, p = .08).

Among children who were prescribed ADHD medication (n = 373; practice-level ICC = .04; provider-level ICC = .01; patient-level ICC = .46), there was a significant effect of group (b = −2.42, p = .04) along with non-significant effects of race (b = 1.10, p = .14) and time (b = −.001, p = .43) on parent-ratings of ADHD symptoms. The estimate for the group effect indicates a 2.42 point incremental reduction in parent-rated ADHD total symptom scores for children treated at intervention practices compared to control practices (Figure 4). The PRV effect size estimate indicated that compared to a null model (level 4 TSS variance = 4.38), the inclusion of treatment group as a predictor variable (level 4 TSS variance = 3.12) reduced response variable variance by 29% (PRV = [4.38–3.12]/4.38 = 0.29).

Figure 4.

ADHD symptom reduction among patients treated by physicians randomly assigned to the intervention and control groups

Analyses described above using parent-ratings were repeated using teacher-ratings. All group effects were non-significant.

Discussion

The technology-assisted QI intervention had large effects on several indices of ADHD treatment care quality delivered by community-based providers. In particular, improvements were observed in physicians’ monitoring of their patients with ADHD during the initial year of treatment; gains consistent with previously reported intervention-related improvements¹³. Compared to the usual care group, providers in the intervention group had 25% more patient contacts, and collected 4.6 and 9.9 times more parent and teacher ratings, respectively. However, even the quality of care achieved at the intervention practices left much room for improvement. For example, providers did not collect parent- or teacher-ratings during the initial year of ADHD medication treatment for half of their patients. Also, medication dosages were lower than what is achieved with optimal titration.²¹

Consistent with an extensive literature on ADHD medication efficacy,^18,22 there were large reductions in parent-rated ADHD symptoms when newly-diagnosed children with ADHD in both groups were prescribed ADHD medication (see Figure 4). The statistically significant incremental reduction (2.4 TSS points) in parent-rated ADHD symptoms over and above these medication-effects reflects the incremental impact of improved quality of ADHD care – achieved using our technology-assisted QI intervention - on patient outcomes. However, notably, intervention effects were not detected on teacher ratings of ADHD symptoms. It is possible that higher rates of missing data on teacher-ratings led to reduced power to detect intervention effects.

Though we found that improved ADHD care improves parent-rated ADHD symptoms, the effects of our intervention were limited in that providers at the intervention practices did not consistently implement some standards of ADHD care. In fact, 7 of the 106 providers assigned to the intervention group did not register any children on the ADHD portal. Indeed, some of these care behaviors (e.g., completion of rating scales) require parent and teacher compliance and access to the internet in addition to provider action. For parents and teachers, education about the advantages of periodic visits and monitoring of their child’s medication treatment may help. Using technology (e.g., online education materials) or peer support (e.g., parent advocate) may spread this message more effectively to parents and teachers. For providers, while the ADHD portal intervention does alleviate some obstacles (e.g., distribution and scoring of parent- and teacher-ratings), supplemental supports are likely necessary to facilitate consistent implementation. It may be that improving reimbursement for collection of these scales or according to compliance with specific standards of care may also reinforce reliable implementation of quality ADHD care. However, existing ADHD care quality standards such as the HEDIS measures may need to be re-considered. In particular, the HEDIS measures rely heavily on office visits. Results of this intervention study suggest that increasing the rates and timing of any form of contact, either phone or visits, may improve child outcomes. Hence, less reliance on office visits and more emphasis on alternative forms of monitoring (e.g., phone contacts, rating scale collection) for monitoring medication should likely be considered.

Limitations

As is often the case with community-based research studies, collection of research measures was difficult, largely due to the research staff having minimal contact with research participants. Therefore, data collection rates were compromised. Moreover, recruitment of patients and continued participation in research assessments was lower at intervention practices than control practices. One possible contributor to this disparity was the increased demands for providers to adopt the ADHD portal into practice and the increased demands among families to complete both clinical and research assessments. Additionally, though the participating practices were diverse in terms of geography and populations served, they all volunteered to participate in an ADHD QI study, so our results may not generalize to all practices/providers. Patients too included only those who volunteered to participate which also may limit generalizability of our findings. Another limitation was that our primary outcome was ADHD symptoms rather than functional impairments (e.g., school performance), which are generally the reason that families seek treatment.²³. The rationale for focusing on symptoms was that medication largely targets symptoms and typically does not produce change in function, particularly in community-based settings.^19,24 To improve functioning, behavioral treatments must be employed.^25,26 This leads to a final limitation of this study: our intervention focused on medication and did not assist providers in accessing or delivering behavioral treatments.

Conclusion

Using a technology-assisted QI intervention improves some indicators of ADHD care quality (e.g., frequent contacts, more rating scales collected) in community-based pediatric settings which produces incrementally greater reductions in parent-rated ADHD symptoms among patients treated with medication compared to typical care. This intervention is largely web-based, facilitating dissemination and adoption. Future goals for this intervention include extending the software to facilitate behavioral treatment and exploring strategies to enhance consistent implementation across all patients with ADHD (e.g., increased integration with electronic health record, pay for performance) and other pediatric mental disorders.

What’s Known on This Subject

Though successful interventions have been developed to address the quality of ADHD care in community pediatric settings, these efforts have not demonstrated that improved ADHD care translates into better outcomes for patients with ADHD.

What This Study Adds

This cluster randomized controlled trial demonstrates that use of a technology-assisted quality improvement intervention not only improves the quality of some ADHD care delivered in pediatric settings but also results in greater medication-related ADHD symptom reductions according to parent ratings.

Abbreviations

ADHD

Attention Deficit Hyperactivity Disorder

QI

Quality improvement

RCT

Randomized clinical trial

PRV

proportion of reduction in variance

TTS

Total symptom score