Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Apr 1.
Published in final edited form as: J Atten Disord. 2016 Oct 1:1087054716672337. doi: 10.1177/1087054716672337

Validation of the Use of Electronic Health Records for Classification of ADHD Status

Siobhan M Gruschow 1, Benjamin E Yerys 2, Thomas J Power 3, Dennis R Durbin 4, Allison E Curry 5
PMCID: PMC5843549  NIHMSID: NIHMS934799  PMID: 28112025

Abstract

Objective

To validate an electronic health record (EHR)-based algorithm to classify ADHD status of pediatric patients.

Methods

As part of an applied study, we identified all primary care patients of The Children's Hospital of Philadelphia healthcare network who were born 1987–1995 and residents of New Jersey. Patients were classified with ADHD if their EHR indicated an ICD-9-CM diagnosis code of ‘314.x’ at a clinical visit or on a list of known conditions. We manually reviewed EHRs for ADHD patients (n=2,030) and a random weighted sample of non-ADHD patients (n=807 of 13,579) to confirm the presence or absence of ADHD.

Results

Depending on assumptions for inconclusive cases, sensitivity ranged from 0.96–0.97 (95% CI: 0.95, 0.97), specificity from 0.98–0.99 (0.97, 0.99), and positive predictive value from 0.83–0.98 (0.81, 0.99).

Conclusions

EHR-based diagnostic codes can accurately classify ADHD status among pediatric patients and can be employed by large-scale epidemiologic and clinical studies with high sensitivity and specificity.

Keywords: accuracy, adolescents, attention deficit disorder, medical records, sensitivity

Introduction

Attention-deficit/Hyperactivity Disorder (ADHD) is the most common childhood neurodevelopmental disorder, with a prevalence of 5–10% among 4- to 17-year-olds in the US and 5% worldwide (Polanczyk, Willcutt, Salum, Kieling, & Rohde, 2014; Thomas, Sanders, Doust, Beller, & Glasziou, 2015). ADHD is currently defined by persistent patterns of inattention and/or hyperactivity/impulsivity that are inconsistent with one’s developmental level, usually continue across the life span, and lead to impairments in social, educational, and work activities (American Psychiatric Association, 2013; Barkley, Fischer, Smallish, & Fletcher, 2002; Barkley, 1997; Levy, 2014).

The current best practice method used to diagnose ADHD in childhood is a comprehensive approach that combines information from multiple methods and informants, including parent and teacher ratings, and clinical information derived from parent interviews (Dupaul et al., 2015; Goldman, Genel, Bezman, & Slanetz, 1998; Willcutt, 2012). These methods, however, generally are not feasible to implement on a consistent basis in high volume primary care settings (Epstein et al., 2014), which may preclude their use in large-scale clinical or population-based studies of ADHD. Such studies—which play an important role in addressing key ADHD-related research questions—frequently utilize existing administrative databases such as medical or insurance claims to identify patients with ADHD or classify ADHD status among study patients (Chen et al., 2014; Christensen, Sasané, Hodgkins, Harley, & Tetali, 2010; Guevara, Lozano, Wickizer, Mell, & Gephart, 2002; Guevara, Lozono, Wickizer, Mell, & Gephart, 2001; Leibson, Katusic, Barbaresi, Ransom, & Brien, 2001; Mandell, Guevara, Rostain, & Hadley, 2003; Zima et al., 2010). These databases can be efficiently utilized in research as they contain diagnostic and health service information about large numbers of individuals, are time efficient, relatively inexpensive to access, and can be linked with other datasets to examine relationships between health services and outcomes (Burke et al., 2014; Leibson et al., 2001; Losina, Barrett, Baron, & Katz, 2003; Mandell et al., 2003; Roos, Walld, Wajda, Bond, & Hartford, 1996; Virnig & McBean, 2001). In particular, with the advent of the “meaningful use” criteria released by the Department of Health and Human Services and recent government investment to incentivize the use of medical electronic health records (EHR) (Jha, 2010), the amount of EHR data readily available for research purposes is rapidly growing and likely to increase substantially in the future.

One potential limitation of utilizing EHR data for research is the uncertain validity of these data given that they are not collected for research purposes, may not be collected systematically, and may be incomplete or contain coding errors (Van Walraven & Austin, 2012; Virnig & McBean, 2001; Walkup, J. and Yanos, 2005). To address this concern, several prior studies have examined different administrative databases—Medicaid and Medicare claims, pharmacy claims, or institution-level EHRs—to assess validity of diagnoses of psychiatric conditions (Walkup, J. and Yanos, 2005), including depression (Rawson, Malcolm, & D’Arcy, 1997; Spettell et al., 2003), schizophrenia (Lurie, Popkin, Dysken, Moscovice, & Finch, 1992; Rawson et al., 1997), and autism (Burke et al., 2014). Results have been mixed; some case definitions were found to have high specificity but low sensitivity and positive predictive value (Lurie et al., 1992; Spettell et al., 2003), while others have found high positive predictive values or a high proportion of agreement between hospital and medical chart data (Burke et al., 2014; Rawson et al., 1997). Similar to the manner in which ADHD is assessed, these mental health conditions are diagnosed in community settings using labor intensive best practice methods that are difficult to implement in high volume primary care settings. Validation studies of administrative data are vital to understanding the potential misclassification of ADHD status and allowing researchers to estimate the magnitude and direction of misclassification bias in applied studies.

Despite the potential utility of EHR data for use in epidemiologic studies related to ADHD, only two studies have assessed the utility of EHR in identifying children diagnosed with ADHD (Daley et al., 2014; Guevara et al., 2001). Daley et al. (2014) calculated confirmation rates, that is, the percentage of diagnoses of ADHD indicated in EHRs that were confirmed via medical record review. Within the context of a larger study determining the cost of health care for those with ADHD, Guevara et al. (2001) reviewed the medical charts of 70 randomly selected patients with an ADHD diagnosis to assess the diagnosis’ accuracy and to conduct sensitivity analyses. To our knowledge, no study has examined the sensitivity and specificity of an algorithm to identify ADHD cases using existing administrative EHR data. Thus, our objective was to validate an EHR-based algorithm to classify ADHD status among a cohort of primary care patients of The Children's Hospital of Philadelphia (CHOP) pediatric healthcare network. This internal validation study was part of a larger retrospective cohort study that aimed to examine the association between ADHD and driving outcomes among adolescents.

Methods

CHOP Electronic Health Record

The CHOP healthcare network encompasses over 50 locations throughout southeastern Pennsylvania and southern New Jersey (NJ), including 31 primary care centers, 14 specialty care centers, a 535 bed inpatient hospital, two emergency departments, and two urgent care centers supporting more than one million visits annually. The network serves a socioeconomically and racially diverse population and accepts most insurance plans, including Medicaid. CHOP first implemented its electronic health record (EHR) system (EpicCare®, Epic Systems, Inc, Madison, WI) in 2001 and it is currently used for all aspects of clinical care in all outpatient locations and within the hospital. For every hospital encounter or office visit, the CHOP EHR captures: (1) identifiable data (e.g., name, date of birth, address); (2) demographic data (e.g., sex, race/ethnicity); (3) International Classification of Diseases Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes that indicate the reason(s) for the visit (required for a provider to bill for services under that code); (4) ICD-9-CM diagnosis codes that document ongoing or historical problems (known as the “problem list”); (5) medications prescribed by CHOP or non-CHOP providers; and (6) provider notes, letters, communication with non-CHOP healthcare providers, and other qualitative descriptions of each visit that may provide important supplemental information on ADHD diagnosis (e.g., results of diagnostic testing, documentation of parent and teacher reports).

ADHD Classification: Algorithm Development

We first conducted an initial analysis of EHR data to develop an algorithm to identify whether each patient had been diagnosed with ADHD. To this end, we identified all patients of the CHOP healthcare network who were born 1987–1995 and NJ residents at their last CHOP visit (n=113,966). We then identified those who fit the following hierarchical criteria and, at each level, extracted a random sample of records for manual chart abstraction: (1) had an ICD-9-CM diagnosis code beginning with ‘314’ (i.e. indicative of ADHD) at a CHOP office or hospital visit (n=7,567; 172 sampled); (2) had an ICD-9-CM diagnosis code beginning with ‘314’ on their problem list (n=302; 30 sampled); (3) were prescribed a medication with primary indication for ADHD (n=880; 88 sampled) in the absence of an ADHD diagnostic code; and (4) had select ADHD-related keywords (e.g., “ADHD”, “hyperactivity”) in provider progress notes or qualitative descriptions of visits (n=3,055; 161 sampled) in the absence of an ADHD diagnostic code. This final step was a search for key terms within the provider notes and not a formal natural language processing evaluation. With oversight by two study authors (BY, TP), two trained abstractors performed a manual review of each sampled patient’s entire EHR to search for independent sources of information that confirmed ADHD case or noncase status. These sources included: letters, patient history, evaluations, appointment notes, telephone notes, problem list addendums, uploaded media (e.g., scanned prescription, external evaluations), laboratory results, emergency department notes, and office or hospital visits if they indicated a separate calendar date or different physician (i.e., separate visit) than the original entered ICD-9-CM code. Very few patients who were selected based on visit-or problem list-level ICD-9-CM codes were found to be noncases (<2% and 0%, respectively). Additionally, neither our keyword search in provider notes nor a medication prescription with primary indication for ADHD in the absence of ICD-9-CM diagnostic codes yielded additional confirmed cases of ADHD. Based on these results, we developed the following final algorithm to identify patients who had an ADHD diagnosis in our cohort study: patients were classified as having ADHD if their EHR indicated an ICD-9-CM diagnostic code beginning with ‘314’ either at a CHOP office or hospital visit or on their problem list.

Formal Internal Validation Study

The aim of this validation study was to develop an applied method to classify ADHD status for a retrospective cohort study examining the association between ADHD and motor vehicle crash outcomes. Eligibility for this formal validation study mirrored criteria for the larger applied study. Eligible patients included individuals who: were born 1987–1995; had two or more office visits at any of the six CHOP primary care practices located in New Jersey (NJ); were NJ residents at the time of their last CHOP network visit; were not identified as having an intellectual disability; and had their last primary care visit at the age of 12 years or older to ensure the individual was seen by a primary care provider at an old enough age to confirm ADHD status (DSM-5 criteria calls for ADHD symptom onset by age 12) (American Psychiatric Association, 2013). The final study cohort included 15,609 patients (Figure 1). Using the above-described ADHD algorithm, 2,030 (13.0%) of these patients were classified as having ADHD.

Figure 1.

Figure 1

Open in a new tab

Flowchart depicting selection of study cohort. Grey boxes indicate individuals excluded from the cohort.

We conducted an EHR review for all patients who were classified with ADHD and a weighted random sample of patients classified as non-ADHD with the goal of confirming the presence or absence of ADHD in this cohort. Similar to the algorithm development phase, two trained abstractors reviewed patient EHRs with oversight by two study authors (BY, TP). The review process for the 2,030 patients classified as having ADHD is shown in Figure 2. Using EHR data, the following were automatically confirmed as having ADHD: 940 patients who had three or more ADHD-related CHOP visits, and an additional 284 who were prescribed an ADHD medication by a CHOP provider (as medication was an independent source of confirmation). The remaining 806 patients included those who had either one or two ADHD-related CHOP visits or a problem list diagnosis and had no history of being prescribed an ADHD medication. For each of these patients, abstractors were instructed to manually review the entire EHR to locate any of the aforementioned independent sources confirming ADHD case status. When contradicting information was present in the EHR, abstractors weighed letters, evaluations, and appointment notes more in making a final diagnostic decision. A patient was determined to be a “confirmed ADHD case” when at least one independent source confirming ADHD diagnoses was recorded. Similarly, if an independent source was found that indicated the patient did not have ADHD, the patient was determined to be a “confirmed non-ADHD case”, weighing the most recent independent source more. Only one patient had information both confirming ADHD status and other information confirming non-ADHD status; this patient was confirmed as a non-ADHD case because one letter and two patient forms documented no history of ADHD. Finally, patients for whom neither a source confirming the presence or absence of ADHD was located were considered to have an unknown case status. Twenty-four percent of records were reviewed by both abstractors in order to assess inter-rater reliability. All disagreements were reviewed at consensus meetings and final decision was made by the two co-authors (BY, TP).

Figure 2.

Figure 2

Open in a new tab

Flowchart depicting manual review process of ADHD cohort (determined via algorithm) to confirm case status.

Notes: An ADHD-related visit is defined as OMITTED office or hospital visit that includes an ICD-9-CM diagnostic code beginning with “314”. “Problem list” is defined as the list of ICD-9-CM diagnosis codes that document ongoing or historical conditions. “Incomplete” refers to subjects who had their last OMITTED primary care visit in or before 2005.” Inconclusive” refers to subjects who had OMITTED primary care visit after 2005 but did not have information in their EHR confirming the absence or presence of ADHD. Grey boxes indicate ADHD subjects automatically confirmed as having ADHD.

We also reviewed a sample of 807 patients in the non-ADHD cohort to determine whether there were sources in their medical records indicating the presence of an ADHD diagnosis. To optimize the rigor of our assessment, we oversampled patients with visit- or problem list-level ICD-9-CM codes for disruptive behavior disorder (‘313.81’ or ‘312.X’) and learning disorder (‘315.X’)—conditions frequently comorbid among children with ADHD; approximately 70% of patients in each group were sampled. In addition, a 3.5% sample of the 13,005 patients who did not have a comorbid conditions were reviewed. Sizes of samples selected for review were based on power calculations; for each sampled group, we had at least 80% power to detect a 5% misclassification rate with a confidence interval width of 2%.

Statistical Analysis

We estimated the proportion of patients with ADHD that our classification algorithm correctly identified as having ADHD (i.e., sensitivity), the proportion of patients without ADHD that our classification correctly identified as not having ADHD (i.e., specificity), the proportion of patients classified as ADHD by our algorithm who had an confirmed ADHD diagnoses (i.e., positive predictive value), and the proportion of patients classified has non-ADHD by our algorithm who were confirmed non-ADHD patients (i.e., negative predictive value), along with exact 95% confidence intervals (CI) using standard measures for proportions. With respect to patients classified as “unknown ADHD case status” during the review of our ADHD cohort, we identified two distinct groups. The first was patients who had their last CHOP primary care visit in or before 2005, the year the EHR was fully implemented in NJ primary care practices; these patients were considered to have “incomplete records” as their EHR had very little information available beyond visit dates and diagnosis codes. The second group was patients who had visits after 2005 but did not have information in their EHR confirming the absence or presence of ADHD; we classified these patients as “inconclusive”. In analyses, patients with incomplete records were assumed to have the same distribution of ADHD, non-ADHD, and inconclusive status as patients for whom a case status classification was determined. Validation analyses were conducted under two different assumptions for those with inconclusive status, representing extreme conditions: (1) all were assumed to be true ADHD cases; and (2) all were assumed to be true non-ADHD cases. We then weighted the manual review results of the sample of non-ADHD patients to the entire non-ADHD cohort. In addition to estimating validity for the overall sample, we estimated the positive predictive value separately for each origin of diagnostic codes (visit-level vs. problem list). Analyses were conducted in SAS Version 9.3 (SAS Institute, Inc., Cary, NC). This study was approved by CHOP Institutional Review Board.

Results

Overall, patients were last seen at CHOP primary care at a median age of 17.9 years (interquartile range [IQR]: 15.9, 19.1 years) (Table 1). Patients in the ADHD cohort had a median of 27 (16, 44) office visits to the CHOP network compared with 18 (9, 31) for patients in the non-ADHD cohort. The majority of patients in both cohorts were Non-Hispanic white (ADHD: 69%; non-ADHD: 59%) and patients with ADHD were more likely to be male (72% vs. 47%, p<0.001).

Table 1.

Demographic and relevant characteristics for study subjects in ADHD and non-ADHD cohorts (N=15,609).a

Cohort
Characteristic All Subjects
(N=15,609)		 Non-ADHD
(N=13,579)		 ADHD
(N=2,030)		 P-Value
Age at last primary care visit, median (IQR) 17.9 (15.9–19.1) 17.9 (15.9–20.3) 18.1 (16.5–19.5) <.0001
Number of OMIT primary care visits, median (IQR) 17 (9–29) 16 (8–28) 23 (13–37) <.0001
Number of total OMIT visits, median (IQR) 19 (10–33) 18 (9–31) 27 (16–44) <.0001
Sex <.0001
    Female 7,004 (49.4) 7,138 (52.6) 566 (27.9)
    Male 7,905 (50.6) 6,441 (47.4) 1,464 (72.1)
Race/ethnicity <.0001
    Hispanic 421 (2.7) 373 (2.7) 48 (2.4)
    Non-Hispanic white 9,395 (60.2) 7,990 (58.8) 1,405 (69.2)
    Non-Hispanic black 2,762 (17.7) 2,458 (18.1) 304 (15.0)
    Non-Hispanic Asian/other/multiple 3,029 (19.4) 2,756 (20.3) 273 (13.4)
    Unknown/Refused 2 2 0
Open in a new tab

Abbreviations: IQR, interquartile range

a

Unless otherwise indicated, data are expressed as number (percentage) of participants. Denominators for percentages exclude missing data. Percentages are rounded and may not total 100.

A total of 806 EHRs of patients classified as having ADHD were manually reviewed (Figure 2). Of these, we were able to confirm the presence of ADHD in 376; when combined with those automatically confirmed based on 3+ visits or mediction use, a total of 1600 of the 2,030 (79%) were determined to be confirmed cases of ADHD. Only 1% of patients were confirmed as non-ADHD cases (n=25), while the remaining 405 (20%) patients were determined to have an unknown case status. Of these 405 patients, 26% had incomplete records while 74% were labeled inconclusive.

Of the 807 patients in the non-ADHD group who were manually reviewed, 9 (1%) were misclassified as non-ADHD patients when in fact they had been diagnosed with ADHD. After fully weighting these results up to the larger non-ADHD comparison group, we estimated that 67 (0.5%) patients in the non-ADHD cohort were in fact diagnosed with ADHD.

The overall inter-rater agreement for ADHD case classification was 96.4% with a kappa of 0.87 (95% CI 0.75, 0.99). We also estimated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the ADHD algorithm under the two assumptions (Table 2). When all inconclusive cases were assumed to be true ADHD cases, the overall sensitivity was 0.97 (95% CI: 0.96, 0.97), the specificity was 0.99 (95% CI: 0.99, 0.99), and the PPV was 0.98 (95% CI: 0.98, 0.99). Estimates among those who were classified via visit-level codes and those classified via problem list codes were similar. When all inconclusive cases were assumed to be true non-ADHD cases, the sensitivity was 0.96 (95% CI: 0.95, 0.97), specificity was 0.98 (95% CI: 0.97, 0.98), and PPV was 0.83 (95% CI: 0.81, 0.85). The estimated PPV among patients diagnosed using problem list codes was lower (0.67; 95% CI: 0.58, 0.75) than those diagnosed using visit-level codes (0.84; 95% CI: 0.83, 0.86). Notably, we found no evidence that validity varied appreciably by site; estimated sensitivity and specificity was greater than 92% for all sites.

Table 2.

ADHD status classified via algorithm and after manual review of electronic health records (i.e., true case status), along with sensitivity (SE), specificity (SP), positive predictive value (PPV), and negative predictive value (NPV) (95% confidence intervals) of algorithm (n=15,609) and stratified by origin of diagnostic code.

ADHD status classified via algorithm
ADHD (n=2,030) non-ADHD (n=13,579) SE (95% CI) SP (95% CI) PPV (95% CI) NPV (95% CI)
All inconclusive cases
assumed to be…		 True ADHD
cases		 True non-
ADHD cases		 True ADHD
casesa 		True non-ADHD
casesa
Overall sample
  True ADHD cases 2,004 26 67 13,512 0.97 (0.96, 0.97) 0.99 (0.99, 0.99) 0.98 (0.98, 0.99) 0.99 (0.99, 0.99)
  True non-ADHD cases 1,688 342 67 13,512 0.96 (0.95, 0.97) 0.98 (0.97, 0.98) 0.83 (0.81, 0.85) 0.99 (0.99, 0.99)
Classified based on visit-level codesb
True ADHD cases 1,878 26 0.98 (0.98, 0.99)
True non-ADHD cases 1,604 300 0.84 (0.83, 0.86)
Classified based on problem list codesc
True ADHD cases 126 0 1.00 (0.97, 1.00)
True non-ADHD cases 84 42 0.67 (0.58, 0.75)
Open in a new tab
a

True case status estimated based on manual review of sample of records.

b

Case status determined by ICD-9-CM diagnosis codes documented on an office or hospital visit that indicate the reason(s) for the visit

c

Case status determined by ICD-9-CM diagnosis codes documented on an list of ongoing or historical problems.

Discussion

This study developed and validated an EHR-based algorithm that classified ADHD status among a large cohort of primary care patients in a regional healthcare network. Overall, findings demonstrate that an algorithm that seeks to capture ADHD case staus among primary care patients can do so with a high degree of sensitivity using only ICD-9CM codes contained in visit-level diagnosis fields or in the problem list. Additionally, a very low proportion of patients without ADHD diagnosis codes were in fact diagnosed with ADHD, resulting in a very robust estimate of specificity for the algorithm. Notably, we were unable to use EHRs to independently confirm the presence or absence of an ADHD diagnosis for one in five patients with ADHD diagnosis codes; this likely reflects the lack of documentation of how ADHD diagnoses were made. Despite this limitation, we demonstrated that the selected ICD-based algorithm was able to identify the presence of ADHD with a high degree of accuracy.

Our findings establish the utility of EHR’s in classifying ADHD status in large population-based studies and quality improvement projects and uniquely extend results of two prior studies. Daley et al. (2014) used EHR data to identify potential cases of incident (i.e., newly diagnosed) ADHD among 3- to 9-year-old patients in 10 large health care organizations. To assess the accuracy of the EHR diagnosis data, the authors randomly selected 40 patients within each site for whom medical charts were reviewed to determine confirmation of incident ADHD. The investigators used five definitions of a confirmed case of ADHD and for each calculated the proportion of potential ADHD cases that were confirmed (Daley et al., 2014). When incident and prevalent cases were included, the proportion of confirmed ADHD cases was high (89.8%–94.2%); however the proportions were lower when a more conservative definition was used (48.9% – 59.1%). In order to assess the validity of ADHD diagnoses in claims data and evaluate the health care costs of ADHD, Guevera et al. (2001) conducted a sensitivity analyses within the context of the larger study. The investigators randomly selected 70 patients for whom medical charts were reviewed and categorized ADHD case status based on different ADHD definitions: probable, possible, and doubtful diagnoses. 72% of the 70 patients were classified as having a probable diagnosis, 21% had a possible diagnosis, and 7% had a doubtful diagnosis (Guevara et al., 2001). While both studies estimated the positive predictive value of an existing ADHD diagnosis in health records, neither included patients without ADHD; thus, they were not able to fully assess the quality of their classification by calculating sensitivity or specificity. While PPV is an important measure of classification quality and can be used to quantitatively estimate the extent of and correct for misclassification bias in applied studies, its dependence on the prevalence of the condition in the population makes portablility across studies with different populations problematic (Lash, Fox, & Fink, 2009). To our knowledge, this study is the first to estimate sensitivity and specificity of ADHD diagnosis codes, which can quantify a classification’s performance across populations. More generally, validation studies make it possible to conduct quantitative bias analyses, allowing researchers to account for potential bias within an applied study and ultimately enhance the robustness of study results. While epidemiology methodolgists strongly encourage the conduct of bias analysis (Lash et al. 2009), the process is not often employed in applied clinical research.

There are several limitations of this study. First, this method is limited by the quality of diagnostic information in medical records, which was highly variable; some records had parent and teacher rating scales and the results of evaluations conducted by external mental health experts, while others had no information beyond the visit date and diagnosis code. While ADHD diagnoses in the EHR accurately reflect real-world diagnostic decisions rendered by providers in primary care, these decisions may differ to some extent from those rendered by expert mental health providers (e.g., psychologists, psychiatrists, developmental pediatricians) applying best practice methods for assessing ADHD in a systematic manner. Further, inherent in the process of diagnostic decision making for mental health conditions is that, even when best practice methods for diagnosis are systematically applied, there is always an element of uncertainty in a small percentage of cases given that this diagnosis relies on clinical judgment. Accordingly, while this study is able to make conclusions about the extent of an EHR-based algorithm to distinguish patients who have and have not been diagnosed with ADHD in clinical settings, it was not designed to identify “true” ADHD status. Another potential limitation is that we only examined EHR data from a single healthcare network, raising questions about generalizability. However, the network includes a variety of providers across multiple settings, and serves a broad range of patients with regard to race/ethnicity and socioeconomic status. Further, the network utilizes a single EHR system, which promotes standardized practice, similar to other large healthcare systems. In addition, although there may have been some variability in how providers diagnosed ADHD across sites, we found no appreciable difference in accuracy across the six practices. As such, the practices, providers, and patients represented in this study likely reflect those in other large healthcare systems, although replications in other systems are needed to confirm generalizability. Study findings are likely more generalizable to studies that are conducted within large healthcare systems—in particular those that standardize practices—than to studies that combine data from multiple institutions. In addition, abstractors were not blinded to the ADHD status of the patients. It is possible that this knowledge might have biased the reviewers; however, we expect this was minimized given the we: developed strict criteria on information that was considered “confirmatory”; included a classification category of “inconclusive”; and ensured final results were reviewed by two licensed psychologists (BY, TP). Finally, this method does not capture patients who do have ADHD but either do not seek care or were unrecognized by CHOP providers.

Conclusion

Health service and applied clinical studies have utilized ICD-9-CM codes derived from EHRs to classify ADHD status; however, internal validity of study results are dependent on the extent to which diagnosis codes can accurately classify status. The current study describes the development an EHR-based approach to classify ADHD status among primary care patients and, for the first time, demonstrates that such an algorithm can be employed by large epidemiologic and clinical studies with high sensitivity and specificity. Estimates of sensitivity and specificity from the current study can be utilized by future applied studies to conduct analyses that correct estimates of measures of association (e.g., odds ratios, risk ratios) for misclassification of ADHD status.

Acknowledgments

Funding: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development at the National Institutes of Health (Grant 1R01HD079398, PI: Curry). The views presented are those of the authors and not necessarily the views of CHOP or NIH.

Contributor Information

Siobhan M. Gruschow, Center for Injury Research and Prevention, The Children’s Hospital of Philadelphia

Benjamin E. Yerys, Center for Autism Research, The Children’s Hospital of Philadelphia and Department of Child and Adolescent Psychiatry and Behavioral Science, Perelman School of Medicine at the University of Pennsylvania

Thomas J. Power, Department of Child and Adolescent Psychiatry and Behavioral Science, The Children’s Hospital of Philadelphia, Perelman School of Medicine at the University of Pennsylvania

Dennis R. Durbin, Division of Emergency Medicine, Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania

Allison E. Curry, Center for Injury Research and Prevention, The Children’s Hospital of Philadelphia

References

  1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®) 2013 doi: 10.1590/s2317-17822013000200017. [DOI] [PubMed] [Google Scholar]
  2. Barkley RA. Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin. 1997;121(1):65–94. doi: 10.1037/0033-2909.121.1.65. [DOI] [PubMed] [Google Scholar]
  3. Barkley RA, Fischer M, Smallish L, Fletcher K. The persistence of attention-deficit/hyperactivity disorder into young adulthood as a function of reporting source and definition of disorder. Journal of Abnormal Psychology. 2002;111(2):279–289. [PubMed] [Google Scholar]
  4. Burke JP, Jain A, Yang W, Kelly JP, Kaiser M, Becker L, Newschaffer CJ. Does a claims diagnosis of autism mean a true case? Autism : The International Journal of Research and Practice. 2014;18(3):321–30. doi: 10.1177/1362361312467709. [DOI] [PubMed] [Google Scholar]
  5. Chen Q, Sjölander A, Runeson B, D’Onofrio BM, Lichtenstein P, Larsson H. Drug treatment for attention-deficit/hyperactivity disorder and suicidal behaviour: register based study. BMJ. 2014;348:g3769. doi: 10.1136/bmj.g3769. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Christensen L, Sasané R, Hodgkins P, Harley C, Tetali S. Pharmacological treatment patterns among patients with attention-deficit/hyperactivity disorder: retrospective claims-based analysis of a managed care population. Current Medical Research and Opinion. 2010;26(4):977–89. doi: 10.1185/03007991003673617. [DOI] [PubMed] [Google Scholar]
  7. Curry A, Metzger KB, Pfeiffer MR, Elliot MR, Winston FK, Power TJ. Motor vehicle crash rates among adolescents and young adults with ADHD. Under Review. 2016 doi: 10.1001/jamapediatrics.2017.0910. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Daley MF, Newton DA, Debar L, Newcomer SR, Pieper L, Boscarino Ja, Bussing R. Accuracy of electronic health record-derived data for the identification of incident ADHD. Journal of Attention Disorders. 2014:1–10. doi: 10.1177/1087054713520616. [DOI] [PubMed] [Google Scholar]
  9. Dupaul GJ, Reid R, Anastopoulos AD, Lambert MC, Watkins MW, Power TJ. Parent and teacher ratings of attention-deficit / hyperactivity disorder symptoms: Factor structure and normative data. Psychological Assessment. 2015 doi: 10.1037/pas0000166. Advance online. [DOI] [PubMed] [Google Scholar]
  10. Epstein JN, Kelleher KJ, Baum R, Brinkman WB, Peugh J, Gardner W, Langberg J. Variability in ADHD care in community-based pediatrics. Pediatrics. 2014;134(6):1136–43. doi: 10.1542/peds.2014-1500. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Goldman LS, Genel M, Bezman RJ, Slanetz PJ. Diagnosis and treatment of attention-deficit / hyperactivity disorder in children and adolescents. Journal of the American Medical Association. 1998;279(14) doi: 10.1001/jama.279.14.1100. [DOI] [PubMed] [Google Scholar]
  12. Guevara J, Lozano P, Wickizer T, Mell L, Gephart H. Psychotropic medication use in a population of children who have attention-deficit/hyperactivity disorder. Pediatrics. 2002;109(5):733–739. doi: 10.1542/peds.109.5.733. [DOI] [PubMed] [Google Scholar]
  13. Guevara J, Lozono P, Wickizer T, Mell L, Gephart H. Utilization and cost of health care services for children with attention-deficit/hyperactivity disorder. Pediatrics. 2001 doi: 10.1542/peds.108.1.71. [DOI] [PubMed] [Google Scholar]
  14. Jha AK. Meaningful use of electronic health records. JAMA. 2010;304(15):1709–1710. doi: 10.1001/jama.2010.1497. [DOI] [PubMed] [Google Scholar]
  15. Lash T, Fox M, Fink A. Applying Quantitative Bias Analysis to Epidemiologic Data. New York: Springer US; 2009. [Google Scholar]
  16. Leibson C, Katusic SK, Barbaresi WJ, Ransom J, Brien PC. Use and costs of medical care for children attention-deficit / hyperactivity disorder. JAMA. 2001;285(1):60–66. doi: 10.1001/jama.285.1.60. [DOI] [PubMed] [Google Scholar]
  17. Levy F. [Retrieved March 5, 2015];DSM-5, ICD-11, RDOC and ADHD Diagnosis. 2014 doi: 10.1177/0004867414557527. from http://anp.sagepub.com/content/early/2014/11/01/0004867414557527.full. [DOI] [PubMed]
  18. Losina E, Barrett J, Baron Ja, Katz JN. Accuracy of Medicare claims data for rheumatologic diagnoses in total hip replacement recipients. Journal of Clinical Epidemiology. 2003;56:515–519. doi: 10.1016/s0895-4356(03)00056-8. [DOI] [PubMed] [Google Scholar]
  19. Lurie N, Popkin M, Dysken M, Moscovice I, Finch M. Accuracy of diagnoses of schizophrenia in Medicaid claims. Hospital & Community Psychiatry. 1992;43(1):69–71. doi: 10.1176/ps.43.1.69. [DOI] [PubMed] [Google Scholar]
  20. Mandell DS, Guevara JP, Rostain AL, Hadley TR. Medical expenditures among children with psychiatric disorders in a Medicaid population. Psychiatric Services. 2003;54(4):465–467. doi: 10.1176/appi.ps.54.4.465. [DOI] [PubMed] [Google Scholar]
  21. Polanczyk GV, Willcutt EG, Salum GA, Kieling C, Rohde LA. ADHD prevalence estimates across three decades: An updated systematic review and meta-regression analysis. International Journal of Epidemiology. 2014;43(2):434–442. doi: 10.1093/ije/dyt261. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Rawson NSB, Malcolm E, D’Arcy C. Reliability of the recording of schizophrenia and depressive disorder in the Saskatchewan health care datafiles. Social Psychiatry and Psychiatric Epidemiology. 1997;32(4):191–199. doi: 10.1007/BF00788238. [DOI] [PubMed] [Google Scholar]
  23. Roos LL, Walld R, Wajda A, Bond R, Hartford K. Record linkage strategies, outpatient procedures, and administrative data. Medical Care. 1996;34(6):570–582. doi: 10.1097/00005650-199606000-00007. [DOI] [PubMed] [Google Scholar]
  24. Spettell CM, Wall TC, Allison J, Calhoun J, Kobylinski R, Fargason R, Kiefe CI. Identifying physician-recognized depression from administrative data: Consequences for quality measurement. Health Services Research. 2003;38:1081–1102. doi: 10.1111/1475-6773.00164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Thomas R, Sanders S, Doust J, Beller E, Glasziou P. Prevalence of attention-deficit/ hyperactivity disorder: A systematic review and meta-analysis. Pediatrics. 2015;135(4) doi: 10.1542/peds.2014-3482. [DOI] [PubMed] [Google Scholar]
  26. Van Walraven C, Austin P. Administrative database research has unique characteristics that can risk biased results. Journal of Clinical Epidemiology. 2012;65(2):126–131. doi: 10.1016/j.jclinepi.2011.08.002. [DOI] [PubMed] [Google Scholar]
  27. Virnig BA, McBean M. Administrative data for public health surveillance and planning. Annual Review of Public Health. 2001;22:213–230. doi: 10.1146/annurev.publhealth.22.1.213. [DOI] [PubMed] [Google Scholar]
  28. Walkup J, Yanos P. Psychological research with administrative data sets: An underutilized strategy for mental health services research. Prof Psychol Res Pr. 2005;36(5):551–557. doi: 10.1037/0735-7028.36.5.551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Willcutt EG. The prevalence of DSM-IV attention-deficit/hyperactivity disorder: A meta-analytic review. Neurotherapeutics. 2012;9(3):490–499. doi: 10.1007/s13311-012-0135-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Zima BT, Bussing R, Tang L, Zhang L, Ettner S, Belin TR, Wells KB. Quality of care for childhood attention deficit/hyperactivity disorder in a managed care medicaid program. J Am Acad Child Adolesc Psychiatry. 2010;49(12):1–17. doi: 10.1016/j.jaac.2010.08.012. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES