Use of a Computerized Decision Aid for ADHD Diagnosis: A Randomized Controlled Trial

Abstract

OBJECTIVE:

To determine if implementing attention-deficit/hyperactivity disorder (ADHD) diagnosis and treatment guidelines in a clinical decision support system would result in better care, including higher rates of adherence to clinical care guidelines.

METHODS:

We conducted a cluster randomized controlled trial in which we compared diagnosis and management of ADHD in 6- to 12-year-olds after implementation of a computer decision support system in 4 practices.

RESULTS:

Eighty-four charts were reviewed. In the control group, the use of structured diagnostic assessments dropped from 50% in the baseline period to 38% in the intervention period. In the intervention group, however, it rose from 60% to 81%. This difference was statistically significant, even after controlling for age, gender, and race (odds ratio of structured diagnostic assessment in intervention group versus control group = 8.0, 95% confidence interval 1.6–40.6). Significant differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. Our study was not powered to detect changes in care and management, but the percent of patients who had documented medication adjustments, mental health referrals, and visits to mental health specialists were higher in the intervention group than the control.

CONCLUSIONS:

The introduction of a clinical decision support module resulted in higher quality of care with respect to ADHD diagnosis including a prospect for higher quality of ADHD management in children. Future work will examine how to further develop the ADHD module and add support for other chronic conditions.

What’s Known on This Subject:

Despite the existence of authoritative guidelines to assist primary care physicians in identifying and managing attention-deficit/hyperactivity disorder (ADHD), ample evidence demonstrates that they continue to diagnose and treat this disorder suboptimally.

What This Study Adds:

The introduction of a clinical decision support module resulted in higher quality of care with respect to ADHD diagnosis including a prospect for higher quality of ADHD management in children.

Attention-deficit/hyperactivity disorder (ADHD) is the most commonly diagnosed pediatric behavioral disorder.¹ The gold standard for diagnosis is standardized assessments of behavior based on specific diagnostic criteria defined in the Diagnostic and Statistical Manual of Mental Disorders.² Because there is no definitive diagnostic test for ADHD, diagnosis can be difficult. The core symptoms that characterize the disorder (inattention, hyperactivity, and impulsivity) lie on a spectrum; moreover, many other medical, psychiatric, and learning conditions can present with similar symptoms.^2,3

Primary care physicians, especially pediatricians, have historically played a large role in the diagnosis and treatment of ADHD.^4,5 Despite the existence of authoritative guidelines to assist primary care physicians,^6–10 ample evidence demonstrates that they continue to diagnose and treat this disorder suboptimally.^2,11–13 This is due, in part, to a lack of training and cumbersome delivery system designs.¹⁴ Modern computer decision support strategies offer the best hope of equipping general practitioners to deal with the mental health epidemic of ADHD.

We have developed a novel computer decision support system (CDSS) for implementing clinical guidelines in pediatric practice. The CHICA (Child Health Improvement through Computer Automation) system collects information directly from patients, delivers “just-in-time” patient-relevant guidelines to physicians during the clinical encounter, and accurately captures structured data from all who interact with it. Previous work with CHICA has demonstrated the feasibility of using the system to implement and evaluate clinical guidelines,^15,16 In 2010, we enhanced the CHICA system to include the American Academy of Pediatrics guidelines to assist physicians in the diagnosis⁶ and treatment¹⁷ of ADHD. We then conducted a study to see if the delivery of ADHD guidelines via a CDSS would result in better care, specifically higher rates of adherence to clinical care guidelines and improved patient functioning.

Methods

Setting

This study took place in 4 primary care practices in the Indiana University Medical Group-Primary Care practice network. In 2010, these 4 clinics hosted 37 425 patient visits. Of these, 77% to 88% of visits were for pediatric patients who were on Medicaid. Two of these clinics served as intervention sites, and the other 2 clinics functioned as control sites. This study took place between 2010 and 2012.

Participants

Patients between the ages of 5 and 12 years were automatically placed into either the control or intervention group based on which of the 4 pediatric clinics they attended. No potential subjects were contacted by researchers, their physician, or other staff personnel regarding the study.

Study Design

We conducted a cluster randomized controlled trial in which we compared diagnosis of ADHD in the intervention practices to the control practices. The 2 intervention clinics used the “enhanced” CHICA system, which included guidelines for ADHD diagnosis and management (ie, the ADHD module), and the 2 control clinics used the “traditional” CHICA system that did not include ADHD guidelines. To achieve randomization, the 4 clinics were ranked by their size, where size was defined by the number of physicians and residents in the clinic. At randomization, the first and last ranked clinics were assigned to one group (intervention) and the other 2 clinics were assigned to the other group (control). In total, there were 60 clinicians practicing at the 2 intervention sites, and 50 clinicians practicing at the 2 control sites.

Although randomization at the physician or patient level may have been sample-size efficient, we chose to randomize by clinic because contamination was a major concern. If we randomized at the physician level, physicians in the same clinic who were assigned to different treatment arms might communicate regarding the CHICA ADHD module in terms of its operation and consequences. Similarly, if we randomized at the patient level, the on-and-off use of the CHICA ADHD module might confuse or frustrate the physician and even lead some physicians to think that CHICA was malfunctioning. We have found previously that randomization at the patient level can lead to irritation and inconvenience for both physicians and clinic staff.¹⁸

CHICA System

This study was conducted by using the CHICA system, which has been described in significant detail in previous publications.^19,20 In short, CHICA is a CDSS coupled with an electronic medical record (EMR) that can be used for pediatric well care and chronic disease management.^18,19,21 CHICA produces 2 tailored and scannable paper forms. The prescreener form contains 20 questions that parents answer while in the waiting room. The physician worksheet contains up to 6 prompts to physicians that include check box responses to record the physician’s assessment and actions. Data from these forms are captured in the computer by scanning with optical mark recognition and optical character recognition. The questions and prompts are programmatically chosen according to a child’s age and data stored in the child’s EMR, the Regenstrief Medical Record System (RMRS).¹⁶ The RMRS contains additional information, such as diagnostic codes, physician orders, filled prescriptions, and laboratory data from the statewide health information exchange, the Indiana Network for Patient Care.²²

Intervention: The CHICA ADHD Module

The functioning of the CHICA ADHD module is described in Fig 1 and is based on the American Academy of Pediatrics guidelines to assist physicians in the diagnosis⁶ and treatment¹⁷ of ADHD. The first step concerned case finding. On the prescreening form, parents of children 5 to 12 years of age were asked the following 3 screening questions:

FIGURE 1.

Algorithm of diagnosis under the CHICA ADHD module.

• Does [CHILD’S NAME] OFTEN make careless mistakes or not pay close attention to details, causing problems at home or school?

• Does [CHILD’S NAME] OFTEN have difficulty remaining seated when asked to do so, causing problems at home or school?

• Does [CHILD’S NAME] OFTEN have a hard time paying attention to tasks or play, causing problems at home or school?

Positive answers to any of these 3 screening questions alerted CHICA to potential ADHD. Additionally, the CHICA system was also programed to determine whether the child had a previous diagnosis of ADHD by looking for the presence of ADHD diagnostic codes in the patient’s EMR. In the control group, screening was left to the discretion of the physician. No screening questions were included on the prescreening form for parents to answer in the control group.

The diagnosis of ADHD requires a combination of symptoms including inattention, distractibility, or hyperactivity that impairs the child’s function in >1 setting. One of the most practical ways to assess these criteria is to have the family complete a standardized instrument, for example, the National Initiative for Children’s Healthcare Quality ADHD Vanderbilt Assessment Scales.²³ To achieve this, the CHICA ADHD module automatically printed customized and scannable Vanderbilt assessment scales. The Vanderbilt assessment scales were printed if a child was suspected of having ADHD based on parent answers to the screening questions. Additionally, the Vanderbilt assessment scales were printed at appropriate intervals for those children already diagnosed with ADHD based on the child’s age and based on information contained in their EMR. For example, if a follow-up assessment was needed, CHICA would automatically print the parent and teacher follow-up Vanderbilt forms.

The CHICA ADHD module also instructed the physician in the proper distribution and completion of the Vanderbilt forms. For example, the parent assessment forms were completed by the family in the waiting room and scanned in during the office visit; in rare cases, they were faxed directly to the CHICA system. The teacher Vanderbilt assessment forms were given to the family and could be faxed by the teacher to a standard phone number in the CHICA system. Both the parent and teacher assessment forms were automatically scored by CHICA and recorded in the system. As a convenience for the clinician, CHICA generated a summary sheet with all subscores and interpretations from each assessment form. The interpretations were also made available as prompts on the physician worksheet at subsequent follow-up visits.

The CHICA ADHD module also made treatment recommendations based on established guidelines. This included appropriate starting doses for medications and suggested medication changes if treatment goals were not met. As an aid for the clinician, the patient and family, and teachers, CHICA also provided age-appropriate “just-in-time” handouts that included behavior management suggestions.

Main Outcomes

We used manual chart abstractions to assess each practice’s adherence to clinical care guidelines before and after implementing the CHICA ADHD module. Data collection began 6 months after the CHICA ADHD Module was turned on in the CHICA system, and data collection occurred until 6 months after the last patient included was diagnosed with ADHD. Data were collected on a revised version of a data abstraction form, which was developed for previous studies by members of our research group.¹² Research assistants were trained on the use of this form by the study investigators. Our primary outcome of interest was the percentage of patients who were diagnosed with ADHD who had a completed structured diagnostic assessment, defined as an initial Vanderbilt rating scale completed by both the parent and the teacher, before the diagnosis of ADHD being assigned. The number of core symptoms noted at the time of diagnosis was also recorded during the chart review process. Other data were collected regarding the care and management of ADHD such as reassessment of symptoms, medication adjustments, and mental health referrals. A random sample (20%) of the charts was abstracted twice to assess the reliability of chart abstraction. The agreement on overall chart abstraction in these categories was found to be 89%, with a κ of 0.76.

Data Analyses

Characteristics of the study participants at baseline were summarized by using descriptive statistics (ie, frequency counts and percentages by treatment groups). Baseline categorical variables were examined using χ² tests or Fisher’s exact test, and continuous measures were examined with Wilcoxon rank-sum test.

A logistic regression model including clinic as a cluster variable was used for comparison of dichotomous variables, for example, to examine whether a patient had a structured diagnostic assessment at postintervention. Poisson regression with clinic as a cluster variable was used to model the count variables (eg, the number of inattention behaviors recorded at home). Because we observed overdispersion in the distribution of count variables, we used a generalized Poisson distribution to model these outcomes. Group comparisons in all models were adjusted for age, gender, and race.

All calculations were performed by using the SAS Version 9.2 statistical package.

Sample Size Calculation

Because patients were nested within clinic, responses from the patients within clinic were likely to be correlated. This intracluster correlation was taken into account in the calculation of sample size and in the analyses. By using a χ² test and setting α at .05, the probability of detecting a statistically significant difference in the adherence rate (20% vs 60%) was 90% with an effective sample size of 35 chart reviews per group. Hence, the total effective sample size was 70. Using 2 clinics in the intervention group and 2 clinics in the control group, a random review of 21 charts per clinic would have 90% power to detect a difference of 40%.

Institutional Review Board

This study was reviewed and approved by the Institutional Review Board of the Indiana University School of Medicine.

Results

We queried the RMRS for patients who received a new diagnosis of ADHD during the study period. One hundred eighty-three patients were identified as being eligible for the study; 101 patients from the intervention clinics and 82 from the control clinics. By using a random number generator, we then selected 42 charts from the intervention and from the control clinics to review.

Characteristics of the patients included in this study can be seen in Table 1. There were no significant differences between the intervention and control groups with respect to gender for structured diagnostic assessment, although some differences were seen in insurance and race because the clinics serve different areas of Indianapolis, Indiana.

TABLE 1.

Demographic Characteristics of Study Population

	Control, n (%)	Intervention, n (%)	Total
Gender
Boys	31 (74%)	29 (69%)	60
Girls	11 (26%)	13 (31%)	24
Race
Black	29 (69%)	19 (45%)	48
Hispanic	1 (2%)	14 (33%)	15
White	8 (19%)	7 (17%)	15
Other	4 (10%)	2 (5%)	6
Insurance
Commercial	0 (0%)	3 (7%)	3
Medicaid	41 (98%)	30 (71%)	71
Self-Pay	1 (2%)	1 (2%)	2
Other	0 (0%)	8 (19%)	8

Because we wanted to determine a true change in the main outcome of interest (ie, change in the rate of structured diagnostic assessment), we conducted a baseline analysis that looked at the use of structured diagnostic assessments in both the intervention and control clinics before the initiation of our study. Eighty-four charts were pulled for the baseline analysis of children newly diagnosed with ADHD (42 charts from intervention group and 42 from control group). At baseline, the use of structured diagnostic assessment was 50% for the control group and 60% for the intervention group. During the course of our study, the use of structured diagnostic assessment dropped to 38% in the control group (see Fig 2). However, it rose to 81% in the intervention group. In a logistic regression model, this difference was statistically significant, even after controlling for age, gender, race/ethnicity, and insurance (odds ratio of structured diagnostic assessment in intervention group = 8.0, 95% confidence interval 1.6–40.6).

FIGURE 2.

Change in percent of patients diagnosed with ADHD who had a structured diagnostic assessment performed.

In a Poisson regression model, differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. Figure 3 shows the average number of inattentive or hyperactive/impulsive symptoms noted at home and school. Although the numbers of hyperactive symptoms at school did not achieve statistical significance (P = .075), increases in the 3 others measured were significant to P < .05 (Fig 3).

FIGURE 3.

Number of core symptoms noted at the time of diagnosis.

Although our study was not powered to detect changes in ADHD care and management, we also did preliminary analyses in these areas (Fig 4). There were no statistically significant differences in the percent of patients who had documented medication adjustments between the intervention and the control groups (45% vs 33%, P = .45). The percent of patients who had documented reassessment of symptoms at the 3-month follow-up visit was also higher in the intervention group, but it was not statistically significant (50% vs 33%, P = .36). More patients in the intervention group also had a mental health referral (74% vs 55%, P = .09) and a visit to a mental health specialist (67% vs 48%, P = .054); however these changes were also not statistically significant.

FIGURE 4.

Changes in care and management of ADHD.

Discussion

Evidence-based guidelines recommend that physicians conduct a structured diagnostic assessment and document ADHD symptoms resulting in impaired functioning in 2 settings. This study showed that the introduction of a clinical decision support module for ADHD could significantly improve the use of standardized rating scales from both a parent and teacher for the diagnosis of ADHD. Differences were also seen in the number of ADHD core symptoms noted at the time of diagnosis. This is an important finding because it demonstrates that our automated process can improve the specificity with which parents and teachers are accurately reporting symptoms. In addition, we saw trends toward improvements in areas of management, including timeliness of reassessment and medication titration. We also noted trends indicating that children in the intervention group were referred for comanagement with a mental health specialist more often than children in the control group. This is important because although treatment of ADHD symptoms with stimulant medication can reduce the number of core symptoms, children may continue to have impaired daily functioning, which requires comanagement with a mental health specialist.²⁴

Pediatricians are busy and often overwhelmed in clinical care due to time and resource constraints.²⁵ It is often difficult for them to properly screen for every condition, and things can get missed.²⁶ Even when they try to follow guidelines, care can be suboptimal. Some past CDSS approaches have been specific to mental health issues²⁷ and, as a result, may have been intrusive and disruptive to the usual processes of care.²⁸ The CHICA system developed by our group is unique in that respect. CHICA allows us to introduce guideline-based care into existing clinic practices and prioritize which care modules are most relevant for which children in a tailored fashion. Using a holistic CDSS like CHICA holds much promise for introducing better evidence-based care and ongoing chronic care management into busy pediatric practices. Another benefit to using a CDSS like CHICA is the ability to adjust clinical guidelines easily as the evidence supporting their use changes.

As with all such studies, there are limits to this study that warrant consideration. It is possible that existing clinical practice differences (eg, baseline rates of structured diagnostic assessments in the intervention and control groups) accounted for some of the differences that we found, although we did control for them in our analyses. Randomizing at the clinic level could also lead to biases due to knowledge of ADHD diagnosis and management by physicians, socioeconomic status of patients, and so on. To our knowledge, however, there is no reported significance and scale of the effect of these factors. After weighing the limitations and advantages of each option, we decided to allocate 2 clinics to each of the 2 arms of the study. Generalizability of this study may also be a limitation; this was 1 study of a specific CDSS. However, the CHICA system is built with open-source tools and can be adopted relatively easily and installed cheaply. Although most EMRs may lack the sophistication of CHICA, it is possible to implement the same algorithm with existing commercially available systems. Practices without CDSS would have difficulty implementing this algorithm, but the number of such practices should become fewer as more achieve meaningful use through the HITECH Act.²⁹ Moreover, through CHICA, we are also testing a long-term theory: that the best way to introduce new guideline-based interventions is to make them a part of existing operations in a transparent and unobtrusive manner. We believe that much of the success of the system is based on this. We are not studying the ADHD forms in isolation. Instead we are studying a combination of improved screening, improved document management, and improved CDSS. We were studying this intervention holistically, not pushing any 1 piece above another.

Changing physician behavior can be difficult. This study proved that a CDSS using direct patient input and data-driven algorithms to prioritize and deliver personalized, evidence-based recommendations to physicians at the point of care can accomplish higher quality of care with respect to ADHD diagnosis, as well as the prospect of higher quality for ADHD management. Future work will examine how to develop the CDSS further and support care for other chronic conditions.

Glossary

ADHD

attention-deficit/hyperactivity disorder

CDSS

clinical decision support system

CHICA

Child Health Improvement through Computer Automation

EMR

electronic medical record

RMRS

Regenstrief Medical Record System