Use of an Internet Portal to Improve Community-Based Pediatric ADHD Care: A Cluster Randomized Trial

Abstract

OBJECTIVE:

To determine the effectiveness of a quality improvement program to improve pediatricians' adherence to existing, evidence-based, attention-deficit/hyperactivity disorder (ADHD) practice guidelines.

METHODS:

Forty-nine community-based pediatricians at 8 practices participated in a cluster-randomized trial. Practices were matched according to the numbers of pediatricians and the proportions of patients receiving Medicaid. The medical charts for a random sample of patients with ADHD for each of the participating pediatricians were examined at baseline and 6 months. All practices participated in 4 sessions of training, including didactic lectures and office flow modification workshops. Practices were then given access to an ADHD Internet portal that allowed parents, teachers, and pediatricians to input information (eg, rating scales) about patients, after which information was scored, interpreted, and formatted in a report style that was helpful for assessment and treatment of patients with ADHD. Physicians evaluated their practice behaviors quarterly and addressed underperforming areas.

RESULTS:

Pediatricians in the intervention group, compared with those in the control group, demonstrated significantly higher rates of many American Academy of Pediatrics–recommended ADHD care practices, including collection of parent (Cohen's d = 0.69) and teacher (d = 0.68) rating scales for assessment of children with ADHD, use of Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, criteria (d = 0.85), and use of teacher rating scales to monitor treatment responses (d = 1.01).

CONCLUSION:

A quality improvement intervention that can be widely disseminated by using Internet-based information technology significantly improved the quality of ADHD care in community-based pediatric settings.

WHAT'S KNOWN ON THIS SUBJECT:

Although evidence-based guidelines for pediatric attention-deficit/hyperactivity disorder (ADHD) care exist, adherence to the guidelines in community-based settings has been problematic. Effective interventions that improve the quality of pediatricians' ADHD care and are ready for widespread dissemination do not currently exist.

WHAT THIS STUDY ADDS:

By using a cluster-randomized trial design, results of this study provide evidence that a quality improvement intervention that is conducive to widespread dissemination improves the ADHD assessment and treatment behaviors of community-based pediatricians.

In 2000/2001, the American Academy of Pediatrics (AAP) issued consensus guidelines that provided evidence-based recommendations for the assessment and treatment of children with attention-deficit/hyperactivity disorder (ADHD).^1,2 Despite widespread efforts to promote physician awareness of these guidelines, AAP-recommended practice behaviors are not being implemented reliably.^3–6 Several attempts to develop interventions to improve physicians' use of evidence-based, ADHD-related practice behaviors have been reported.^5,7–10

In the most comprehensive of those interventions, Epstein et al⁵ conducted a community-wide intervention to train 202 physicians from 55 practices. The intervention focused on improving the assessment and treatment practices of community providers by using didactic and quality improvement methods. In an open-label study of the intervention's effectiveness, Epstein et al⁵ demonstrated significant improvements in a number of AAP-recommended practice behaviors at 1 year after the intervention. Intervention gains were maintained at 2 years after the intervention.¹¹ Furthermore, children treated by participating physicians demonstrated significant improvements in ADHD symptoms over time.¹²

That intervention model has been limited geographically, largely because the methods used for the model (ie, in-person didactic training and monitoring through chart review) are not amenable to widespread delivery. Dissemination efforts require that the intervention components be transportable. To this end, a model that uses videoconferencing for training and an ADHD Internet portal that facilitates evidence-based care and allows physicians to track their adherence to the guidelines and to evaluate patient outcomes was developed. The present study was an initial test of the effectiveness of this intervention model in improving ADHD-related practice behaviors.

METHODS

Participants and Settings

Eight practices were recruited in August and September 2009. Inclusion criteria were that the practice served primarily children and the practice was composed of ≥3 physicians who provided ADHD-related care. A mailing was sent to all members of the Kentucky AAP registry whose address was in Greater Lexington or Greater Louisville, Kentucky (153 physicians at 63 practices). The list included pediatricians, family physicians, and other specialists. Thirty-nine of the practices did not qualify for the study because the practice as a whole served primarily adults (n = 9) or the practice included <3 physicians (n = 30). Another 15 practices declined participation because they were busy with other initiatives (n = 3), had limited staff resources (n = 2), or thought that they did not have enough patients with ADHD to warrant participation (n = 10). Nine practices met the inclusion criteria and voluntarily agreed to participate. The first 8 of those 9 practices to provide informed consent were selected to participate. The 8 practices included 49 pediatricians, 26 to 68 years of age (mean: 46.8 years; SD: 10.7 years). Most (94%) of the pediatricians were white (n = 46), and 59% (n = 29) were female (Table 1).

TABLE 1.

Practice Characteristics

Practice	No. of Pediatricians	Proportion of Patients Who Receive Medicaid Assistance, %
Intervention practices
1: pediatric partnership	4	<1
2: pediatric partnership	12	8
3: pediatric partnership	7	30
4: community health center	5	90
Control practices
1: pediatric partnership	4	11
2: pediatric partnership	7	<1
3: pediatric partnership	6	20
4: community health center	5	85

Intervention Model

The intervention used in this study included four 1-hour training sessions conducted with remote, Internet-based, conferencing software. Two 60-minute didactic sessions were conducted by a practicing, community-based, primary care physician (Dr Lichtenstein), focusing on the evidence base for the AAP guideline recommendations. The first didactic focused on ADHD assessment, and the second focused on ADHD treatment. The didactic sessions were attended by all pediatricians in the practice and a practice-identified ADHD champion. The ADHD champion was an office manager or nurse who took primary responsibility for oversight and implementation of practice changes. Each didactic session was followed by a 60-minute workshop, led by a quality improvement consultant, that focused on 3 main goals, that is, (1) modifying office flow, (2) learning to perform tests of change, and (3) training on the ADHD Internet portal. Workshops were attended by all physicians and staff members at each practice. Next, practices were introduced to a performance improvement technique that focuses on performing small tests of change or plan-do-study-act cycles, as described below. Finally, study staff members provided a demonstration of the functionality of the ADHD Internet portal.

The ADHD Internet portal is an Internet-based platform through which parents, teachers, and pediatricians all input information (eg, rating scales) about the target child during initial ADHD assessment and treatment. After rating scales are input by parents and teachers, computerized algorithms score and interpret the data and then output a report that is helpful to pediatricians because it summarizes the results with text, graphs, and tables. All reports can be printed or exported electronically for inclusion in an electronic medical record.

Pediatricians were instructed to use the Internet portal for all new and existing patients, to assess ADHD, to titrate medications, to monitor responses to medications systematically, to communicate with parents and teachers through e-mail, and to monitor ADHD care quality by using an online report card. Physicians earned credit toward the American Board of Pediatrics Maintenance of Certification Performance in Practice requirement (part 4).

At 3, 6, 9, and 12 months after training, study staff members contacted the offices to prompt them to review their Internet portal practice report cards. After identifying underperforming practice behaviors, each practice identified an area to target and then created a plan-do-study-act cycle¹³ to address the target behavior.

Study Design

A cluster-randomized trial design was used. To create comparable groups of practices across conditions, matched practice pairs were created according to the size of the practice (ie, number of pediatricians) and the proportion of patients with Medicaid. Each pair of matched practices was assigned through simple random allocation to the intervention group that received the intervention immediately or to a control group that would receive the intervention after a 6-month period (Table 2). Randomization was performed by a researcher who was not familiar with the identity of the practices, by using a random number generator.

TABLE 2.

Proportions of Patients at Intervention and Control Practices Who Received Targeted, Evidence-Based, ADHD Care at Baseline and 6-Month Time Points

	Proportion, Mean ± SD, %							Parametric Tests Comparing Baseline-6 mo Changes Across Groups
	Intervention				Control
	Baseline	6 mo	Baseline-6 mo Changea	15 mo	Baseline	6 mo	Baseline-6 mo Changea	t	P	Cohen's db
Use of parent ratings of ADHD during assessment	20.0 ± 23.3	42.0 ± 25.9	23.8	70.2 ± 34.4	12.7 ± 23.8	18.1 ± 33.1	5.7	2.21	.03	0.69
Use of teacher ratings of ADHD during assessment	14.5 ± 16.1	36.3 ± 25.2	22.6	50.6 ± 37.3	4.8 ± 14.4	8.3 ± 24.7	6.0	2.17	.04	0.68
Use of DSM-IV ADHD criteria during assessment	0.0 ± 0.0	47.3 ± 30.7	47.3	55.7 ± 40.0	0.0 ± 0.0	16.7 ± 36.4	17.9	2.46	.03	0.85
Use of outside provider for ADHD diagnosis	100.0 ± 0.0	37.7 ± 33.4	−60.7	22.0 ± 32.5	100.0 ± 0.0	89.3 ± 27.2	−10.7	4.78	.0001	1.61
Use of parent ratings of ADHD to monitor treatment responses	0.0 ± 0.0	48.2 ± 35.8	48.2	32.1 ± 35.1	0.0 ± 0.0	25.0 ± 44.7	25.0	1.82	.07	0.59
Use of teacher ratings of ADHD to monitor treatment responses	0.0 ± 0.0	38.7 ± 36.1	38.7	26.6 ± 31.1	0.0 ± 0.0	6.3 ± 25.0	6.3	3.13	.003	1.01

^aMean changes are the average of individual practices' baseline-6 month changes.

^bCohen's d effect size was computed by comparing changes for intervention and control groups.

The study aimed to assess the effectiveness of the intervention in improving pediatricians' ADHD-related practice behaviors with their pediatric patients (of any age). For assessment of practice behaviors with patients, chart reviews were conducted at baseline and after the intervention. Charts were reviewed for evidence of documentation of the following specific guideline-related measures: (1) use of an ADHD parent rating scale during the initial ADHD assessment; (2) use of an ADHD teacher rating scale during the initial ADHD assessment; (3) for patients who were evaluated by the physician, evidence of use of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) to determine whether the child met diagnostic criteria for ADHD; (4) use of the ADHD parent rating scale to monitor responses to treatment for newly treated patients; and (5) use of the ADHD teacher rating scale to monitor responses to treatment for newly treated patients.

For baseline chart reviews, we performed a billing query for all cases coded with an ADHD diagnosis in the previous 2 years. Trained coders randomly selected 10 cases from this list for each physician. Because we wanted to sample physician ADHD assessment practices adequately, we continued to review patient charts beyond the required 10 patient charts per physician to obtain a minimum of 5 patients per physician for whom an assessment was conducted in the previous 2 years. For postbaseline chart review, coders randomly selected up to 10 charts per physician from a billing audit of all cases coded with an ADHD diagnosis during the 6 months after training. Figure 1 indicates the numbers of charts reviewed at each time point. After 6 months, the intervention group continued to use the intervention. A follow-up chart review was conducted for the intervention practices at 15 months. Because the control group received the intervention after 6 months, a follow-up chart review was not conducted for the control practices. Demographic or clinical information on the patient sample was not collected. The Cincinnati Children's Hospital Medical Center institutional review board approved this study, and all participants provided informed consent for participation.

FIGURE 1.

Study flow diagram.

Statistical Analyses

The proportions of patients for whom each pediatrician used each practice behavior at baseline and at 6 months were calculated. An intent-to-treat analysis was performed to analyze changes in practice behaviors from baseline to 6 months after the intervention across the intervention and control groups, by using t tests. For some outcomes that were not normally distributed, a Wilcoxon rank-sum analysis was used. The reported statistical analyses did not account for potential clustering because of the small number of practices in the study.

RESULTS

The study's primary findings are presented in Table 2. Patient chart reviews demonstrated low rates of evidence-based ADHD care before the intervention. For example, pediatricians in our sample were collecting parent and teacher ratings during assessment for <20% of their patients and never were collecting objective parent and teacher ratings to monitor response to treatment.

Pediatricians at practices assigned to the intervention group demonstrated significant increases for many ADHD care behaviors at the postintervention assessment (Table 2). There were significant intervention effects for most outcomes. Pediatricians in the intervention group demonstrated higher rates of improvement in collection of parent and teacher ratings during assessment, documentation of DSM-IV ADHD criteria in patient charts, and use of teacher ratings to monitor treatment outcomes, compared with pediatricians in the control group. Pediatricians in the intervention group also demonstrated decreased reliance on mental health referrals for diagnosis of ADHD among their patients, compared with pediatricians in the control group. Although pediatricians also increased collection of parent ratings to monitor treatment outcomes by 48.2%, group differences were not significant (P = .07), largely because of the 25.0% increase in this outcome variable in the control group.

At the 15-month follow-up assessment, pediatricians in the intervention group demonstrated continued improvement in their provision of ADHD care with respect to assessment behaviors, increasing their rates of use of parent rating scales (70.2%), teacher rating scales (50.6%), and DSM-IV criteria (55.7%). However, their performance rates regarding ADHD treatment behaviors (ie, using rating scales to monitor treatment responses) decreased at the 15-month follow-up assessment (Table 2).

Physicians' use of the Internet portal for provision of ADHD care was monitored during the 6 months of the randomized trial. The 27 physicians in the intervention group registered a total of 460 patients through the Internet portal over the 6-month intervention period. Of those 460 patients who were invited by their physicians to use the portal, 394 (86%) activated their accounts. Most of the patients who did not activate their accounts (n = 66) were registered for ADHD assessment (n = 54). Of the activated accounts, 80 accounts were used for assessment only, 159 were used for treatment monitoring only, and 155 were used for both assessment and treatment monitoring. Among the 219 accounts registered for ADHD assessment in the first 5 months of the intervention period (to provide ≥1 month for completion of the scales), 83.3% of parents returned rating scales (time to complete: range: 0–150 days; median: 2 days) and 83.4% of teachers returned rating scales (time to complete: range: 0–156 days; median: 3 days). Among the 250 accounts registered for treatment monitoring in the first 5 months of the intervention period, 48.4% of parents returned ≥1 set of parent ratings (median time: 67 days) and 55.0% of teachers completed ratings (median time: 64 days).

Consumer satisfaction ratings completed by pediatricians in the intervention group indicated high levels of satisfaction with the intervention. Eighty-six percent of pediatricians (n = 23) responded that the quality of care for children with ADHD had either slightly increased or greatly increased as a result of participation in the intervention. The other 4 pediatricians (15%) reported that the quality of ADHD care was unchanged. Except for 1 participating pediatrician, all reported that they would slightly recommend (37%) or strongly recommend (59%) the intervention to other offices that care for children with ADHD. Finally, most pediatricians rated their overall satisfaction with the intervention as satisfied (dissatisfied: n = 1 [4%]; neither satisfied nor dissatisfied: n = 1 [4%]; satisfied: n = 15 [56%]; very satisfied: n = 10 [37%]).

DISCUSSION

Consistent with previous research, this study documents that rates of adherence to AAP-recommended practice behaviors for ADHD care among community pediatricians are low and remain low without intervention. Although pediatricians in the control group did demonstrate improvement over time, likely because they knew that their performance was being monitored (ie, the Hawthorne effect¹⁴), results of the cluster-randomized trial demonstrated that pediatricians in the intervention group showed significantly improved rates of many AAP-recommended ADHD care practice behaviors, including collection of parent and teacher rating scales for assessment of children with ADHD, use of DSM-IV criteria, and use of teacher rating scales to monitor treatment responses, compared with rates observed among pediatricians assigned to the control group. In addition to being effective for most care outcomes, the intervention was well accepted by pediatricians, who largely expressed high levels of satisfaction with the intervention model and would recommend it to other pediatricians. The intervention model was designed to be able to be disseminated widely, by combining quality improvement methods,¹⁵ telemedicine,^16,17 and an innovative, Internet-based interface.

This is the first randomized trial to demonstrate the efficacy of a quality improvement intervention in improving community-based pediatricians' ADHD assessment and treatment practice behaviors. One of the primary focuses of the intervention is its ability to improve physicians' ability to collect teacher ratings during assessment and treatment. Teacher ratings typically are difficult to collect in pediatric settings, because of logistic problems with distributing forms and then obtaining completed ratings from teachers. The ADHD Internet portal seemed to alleviate typical barriers by allowing physicians to distribute and collect follow-up rating scales from teachers directly. When physicians registered patients on the Internet portal, there were high rates of return of teacher rating scales during assessment (83%) and treatment monitoring (55%). Moreover, ratings were returned quickly during assessment (median time to return: 3 days). Comparable rates of rating scale completion and turnaround times were observed for parent ratings.

An unexpected benefit of our intervention was the tendency for physicians in the intervention group to rely on themselves, rather than outside referrals, for documentation of DSM-IV ADHD criteria for their patients. Although all physicians in this study relied on outside referral sources to document ADHD criteria at baseline, physicians conducted the majority of ADHD evaluations for their patients after the intervention. One apparent effect of the intervention might have been to provide physicians with the confidence and tools to be able to conduct ADHD assessments without having to rely on other health care professionals. Given the high prevalence rates of ADHD and the shortage of pediatric mental health care professionals to conduct assessments,¹⁸ empowering pediatricians to conduct these initial assessments, particularly straightforward cases of ADHD, likely expedites the assessment and treatment process for families and reduces strain on the mental health system, thereby allowing children with more severe or complex presentations to obtain specialist care.

Another unexpected finding was the discrepancy between the baseline rates of ADHD care practice behaviors in the present study and existing literature findings.^3,4,6 For example, pediatricians reported collecting rating scales from parents and/or teachers as part of their ADHD evaluations 67% to 87% of the time.^3,19 In previous investigations of this intervention model using practice-selected charts, we reported higher baseline levels of practice behaviors.^5,11 The present study's use of randomly selected patient charts, as opposed to physician self-reports or practice-selected patient charts, produced dramatically lower rates of evidence-based practice behaviors (ie, 14%–20% used teacher rating scales during assessments). The discrepancies between these different data collection modalities are considerable and should be considered in interpretation of the results of studies that use self-reports or reviews of nonrandomly selected charts.

Although significant intervention-related improvements in ADHD care quality were observed, there continued to be significant room for improvement in the quality of care at the end of the 6-month randomized controlled trial. For example, pediatricians in the intervention group still used parent and teacher rating scales to monitor treatment responses <50% of the time. The low rates of evidence-based ADHD treatment practices have repeatedly been found to be resistant to reliable implementation.^5,7 Additional intervention components (eg, community collaborations²⁰ or assignment of case managers^21–23) might possibly help parents and teachers to improve adherence to prescribed practices.

The finding that the intervention was not able to promote higher levels of adherence to AAP practice guidelines represents a potential weakness of our intervention model. One explanation for the continuing low rates of adherence after intervention implementation is that the 6-month intervention period did not allow enough time for change. Indeed, data from the 15-month naturalistic follow-up assessment suggested that some practice behaviors, particularly those related to ADHD assessment, continued to improve beyond the 6-month time point. Changing practice behaviors is a process that occurs slowly.¹⁹ It may be that full adoption of the intervention components takes time or that many cycles of tests of change are required before adoption is fully solidified.¹⁵

Another explanation for physicians' nonuniform implementation of the intervention components may be that appropriate incentives were not in place to promote reliable usage. Although earning American Board of Pediatrics Maintenance of Certification Performance in Practice credits may incentivize pediatricians to enroll and to participate initially, because this requirement is time-limited and involves only a subset of patients, it does not necessarily incentivize uniform implementation. Physicians might have perceived that the time and effort toward implementation of the prescribed intervention methods were uncompensated or undercompensated in the current reimbursement system and therefore they decided not to implement the methods with all of their patients. Practices were encouraged to bill for collection of rating scales, and many did so successfully. Even with this additional billing, however, physicians likely remained undercompensated for their time and effort in administering high-quality care. Incentives such as pay for performance²⁴ and appropriate reimbursement for e-mail and telephone communications²⁵ likely are necessary to promote uniform adoption and widespread dissemination of quality improvement interventions such as that described in this study.

The current study has some limitations. First, intervention efficacy was established only to 6 months after baseline. Although chart reviews were collected to 15 months after baseline and suggested continued improvement for some outcomes, without comparable data from a control group we are not able to assess the longer-term sustainability of intervention effects. Second, it was impossible to keep chart reviewers blinded to treatment condition because of the need to query the Internet portal at intervention practices for patient care information (ie, rating scale completion). Therefore, data were susceptible to rater bias.

Although this study demonstrates intervention effectiveness with respect to the quality of ADHD care, additional studies are needed to elucidate the intervention adoption process and the full breadth of intervention effects. For example, it is likely that barriers at the patient level (eg, poverty and parents' drug use^21,26) and physician level (eg, computer access in patient rooms and attitudes regarding change²⁷) prevent uniform adoption of this study's intervention. Furthermore, the Internet-based portal presented additional technological challenges to parents in the form of the need for Internet access²¹ and to physicians in terms of adding another electronic patient medical record. Identifying and addressing these barriers may facilitate consistent implementation and widespread dissemination of this intervention. Although there is some evidence that high-quality ADHD care is associated with improved patient outcomes,^7,28,29 additional research is necessary to establish that adoption of this intervention would lead to better patient outcomes and patient satisfaction. Finally, future research must address the cost-effectiveness of this intervention by comparing the potential benefits of this intervention, with respect to clinical utility to pediatricians and potentially improved patient outcomes, with the cost of implementing this intervention (ie, physician/staff time, technical support, and server hosting).