Validity of Claims-based Algorithms to Identify Neurodevelopmental Disorders in Children

Abstract

Purpose:

To validate healthcare claim-based algorithms for neurodevelopmental disorders (NDD) in children using medical records as the reference.

Methods:

Using a clinical data warehouse of patients receiving outpatient or inpatient care at two hospitals in Boston, we identified children (≤14 years between 2010-2014) with at least one of the following NDDs according to claims-based algorithms: autism spectrum disorder/pervasive developmental disorder (ASD), attention deficit disorder/other hyperkinetic syndromes of childhood (ADHD), learning disability, speech/language disorder, developmental coordination disorder (DCD), intellectual disability and behavioral disorder. Fifty cases per outcome were randomly sampled and their medical records were independently reviewed by two physicians to adjudicate the outcome presence. Positive predictive values (PPVs) and 95% confidence intervals (CIs) were calculated.

Results:

PPVs were 94% (95% CI, 83%-99%) for ASD, 88% (76%-95%) for ADHD, 98% (89%-100%) for learning disability, 98% (89%-100%) for speech/language disorder, 82% (69%-91%) for intellectual disability, and 92% (81%-98%) for behavioral disorder. Nineteen of the 50 algorithm-based cases of DCD were confirmed as severe coordination disorders with functional impairment, with a PPV of 38% (25%-53%). Among the 31 false-positive cases of DCD were 7 children with coordination deficits that did not persist throughout childhood, 7 with visual-motor integration deficits, 12 with coordination issues due to an underlying medical condition and 5 with ADHD and at least one other severe NDD.

Conclusions:

PPVs were generally high (range: 82%-98%), suggesting that claims-based algorithms can be used to study NDDs. For DCD, additional criteria are needed to improve the classification of true cases.

INTRODUCTION:

Healthcare utilization databases are a valuable source for perinatal pharmacoepidemiologic research¹ as they reflect routine care, are typically large with comprehensive patient-level information, and allow for linkage between mothers and children and longitudinal follow-up. While evidence from studies using these databases on the risk of short-term complications associated with prenatal drug exposure has been accumulating,^2-8 reproductive safety data regarding longer-term outcomes such as neurodevelopmental disorders (NDD) remain scarce for most medications.

A challenge when using administrative data is that researchers must apply algorithms to identify the health conditions of interest. Such algorithms do not always adequately reflect the patient’s clinical conditions: diagnostic codes carried over from previous medical encounters, rule-out diagnoses or coding errors can be mistaken for evidence of the specific medical condition. Thus, to reduce outcome misclassification and invalid causal inference, the use of accurate and validated algorithms for outcome ascertainment is essential. While validated algorithms for several short-term pregnancy outcomes such as congenital malformations, small for gestational age, preterm birth and pre-eclampsia are available,^9-14 the ability to identify long-term developmental outcomes using administrative databases has been less well characterized.

The goal of this study was therefore to validate medical service claims-based algorithms for the identification of specific NDDs in children against medical records, which were considered the gold standard.

METHODS:

Study Population

The Research Patient Data Registry (RPDR) is a clinical data warehouse for patients receiving outpatient or inpatient care at Mass General Brigham (MGB) affiliated hospitals in the Boston area. Using the RPDR, we identified all children aged ≤14 years with a medical encounter at Massachusetts General Hospital or Brigham and Women’s Hospital between 2010-2014 who met the medical service claims-based definition for a specific NDD (see Outcome Definition). Because medical records from recent years are typically more easily accessible electronically and provide more complete information for adjudication, we selected 2010 as the start year. Since studies focusing on the impact of prenatal drug exposure on neurodevelopment generally require long follow-up periods, most data will come from a period when ICD-9 codes were in use (i.e., any time before October 2015 in the US). We therefore decided to develop the algorithms based on ICD-9 (using 2014 as the end year), and subsequently translated the ICD-9 definitions to ICD-10 (see below for details) to permit application to more recent years.

Outcome Definition

The claims-based algorithms used to identify the specific NDDs – (1) autism spectrum disorder/pervasive developmental disorder (hereafter referred to as ASD for brevity), (2) attention deficit disorder or other hyperkinetic syndromes of childhood (ADHD for brevity), (3) learning disability, (4) speech/language disorder, (5) developmental coordination disorder (DCD), (6) intellectual disability and (7) behavioral disorder – are described in Table 1. To maximize specificity (e.g., reduce the likelihood of coding errors and rule-out diagnoses), we required ≥2 medical encounters with a diagnostic code for the respective specific disorder. One exception to this approach was the ascertainment of learning disability which is not expected to require services reimbursable through health insurance, thus, making the presence of multiple codes unlikely. For ADHD, we additionally considered one relevant encounter and ≥1 prescription for an ADHD medication (atomoxetine, clonidine, guanfacine, [dextro/lisdex]amphetamine, [dex]methylphenidate), or no ADHD diagnosis but ≥2 relevant prescriptions. These additional criteria were chosen because ADHD medications are highly specific to ADHD and unlikely to be prescribed for other conditions in children.^{15, 16}

Table 1.

Claims-based Algorithms of Neurodevelopmental Disorders

Outcome	ICD-9 Dx	Description	Algorithm†
Autism Spectrum Disorder/Pervasive Developmental Disorder (ASD)	299.xx	Pervasive developmental disorders	At ≥ 1 year of age: ≥ 2 dates with ICD-9 Dx
	except 299.1x	Childhood disintegrative disorder
Attention Deficit Disorder/Hyperkinetic Syndrome of Childhood (ADHD)	314.xx	Hyperkinetic syndrome of childhood	At ≥ 2 years of age, any of the following: • ≥2 dates with ICD-9 Dx • ≥ 2 dispensings of atomoxetine, clonidine, guanfacine, (dextro/lisdex)amphetamine, (dex)methylphenidate • ≥ 1 ICD-9 Dx & ≥ 1 dispensing
Learning Difficulty	315.0x	Developmental reading disorder	At ≥ 2 years of age: ≥ 1 ICD-9 Dx
	315.1	Mathematics disorder
	315.2	Other specific developmental learning disorder
Developmental Speech or Language Disorder	315.3x	Developmental speech or language disorder	At ≥ 1.5 years of age: ≥ 2 dates with ICD-9 Dx
	except 315.34	Speech and language developmental delay due to hearing loss
Intellectual disability	317	Mild intellectual disabilities	At ≥ 2 years of age: ≥ 2 dates with ICD-9 Dx
	318.x	Other specified intellectual disabilities
	319	Unspecified intellectual disabilities
Developmental Coordination Disorder (DCD)	315.4	Developmental coordination disorder	At any age: ≥ 2 dates with ICD-9 Dx
Behavioral Disorder	312.xx	Disturbance of conduct not elsewhere classified	At ≥ 2 years of age: ≥ 2 dates with ICD-9 Dx
	313.xx	Disturbance of emotions specific to childhood and adolescence

Abbreviations: ADHD, attention deficit hyperactivity disorder and other hyperkinetic syndromes of childhood; ASD, autism spectrum disorder/pervasive developmental disorder; Dx, diagnosis; DCD, developmental coordination disorder; ICD, International Classification of Diseases.

^†While these are the minimum ages at which we start considering diagnostic codes based on the guidelines, the definitive diagnosis tends to be made at a later age.

For all outcomes, we required children to have the diagnosis recorded (or the medication prescribed) at an age when it is plausible that a correct diagnosis can be made. The minimum age for each outcome was selected after review of current diagnosis recommendations^17-21. The disorders were considered present from the day of the first outcome-related encounter or prescription that fell after the minimum age criterion, irrespective of whether the child also had a diagnosis recorded or a medication of interest prescribed prior to the selected minimum age.

Medical Record Retrieval

For each outcome, we randomly sampled 50 algorithm-identified cases (for a total of 350) using RPDR data. The Enterprise Master Patient Index – a unique patient identifier used across the entire MGB system – and date of birth were used to identify the patients’ medical records. Records of potential cases were independently reviewed by two physicians per case. Seven pairs of reviewers were formed among 14 physicians (LG, RH, CH, JH, DK, KL, ZL, LL, ML, NS, NT, FW, CW, SZ). Reviewers were asked to assess the presence of the outcome. Outcomes were considered present if they fulfilled the criteria listed in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)¹⁸, the most recent diagnostic tool on mental disorders published by the American Psychiatric Association. Whenever there was initial disagreement, reviewer pairs were asked to re-evaluate cases together and reach consensus. Reviewers were further asked to evaluate whether the child was diagnosed with additional developmental disorders.

Analysis

Using medical records as the gold standard, we calculated the positive predictive value (PPV) – which represents the proportion of algorithm-derived cases confirmed through medical record review – and corresponding 95% confidence interval (CI) separately for each disorder of interest (Table 2). For all false positive cases, we further explored why these children were wrongly classified as having the outcome and whether these children were diagnosed with any other NDDs. To translate our ICD-9 definitions to ICD-10, we applied the forward-backward mapping method created by the Centers for Medicare & Medicaid Services and the Centers for Disease Control and Prevention,^{22, 23} reviewed the identified ICD-10 codes, and explored the ICD-10 data dictionary to identify other codes of interest. The final list of selected ICD-10 codes is shown in Table 3.

Table 2.

Positive Predictive Values of Neurodevelopmental Disorders

Outcome	N Cases Identified	N Records Reviewed	N True Positives	N False Positives	PPV (95% CI)	Evaluation of False Positives
Autism Spectrum Disorder/Pervasive Developmental Disorder (ASD)	4093	50	47	3	0.94 (0.83 - 0.99)	All diagnosed with ADHD
Attention Deficit Disorder/Hyperkinetic Syndrome of Childhood (ADHD)	9709	50	44	6	0.88 (0.76 - 0.95)	• 3 identified based on prescriptions only • 1 did not meet all criteria of ADHD • 1 diagnosed with high-functioning autism
Learning Difficulty	2714	50	49	1	0.98 (0.89 - 1.00)	Diagnosed with ADHD
Developmental Speech or Language Disorder	2091	50	49	1	0.98 (0.89 - 1.00)	Limited information available suggesting learning disability
Intellectual disability	417	50	41	9	0.82 (0.69 - 0.91)	• All diagnosed with ≥2 other severe NDDs • 8 identified based on ICD-9 code 319 (unspecified intellectual disability) only
Developmental Coordination Disorder (DCD)	229	50	19 (45 if including coordination issues)	31 (5)	0.38 (0.25 - 0.53) (0.90 (0.82 - 0.98))	• 26 cases with "coordination issues": ○ 7 with visual-motor integration deficits ○ 7 with coordination deficits that did not persist throughout childhood ○ 12 with coordination issues due to an underlying medical condition (such as cerebral palsy, visual impairment) • 5 cases with no DCD but with ADHD and ≥1 other severe NDD
Behavioral Disorder	1844	50	46	4	0.92 (0.81 - 0.98)	All with minor behavioral issues but no formal diagnosis of behavioral disorders

Abbreviations: ADHD, attention deficit hyperactivity disorder and other hyperkinetic syndromes of childhood; ASD, autism spectrum disorder/pervasive developmental disorder; CI, confidence interval; DCD, developmental coordination disorder; N, number; NDD, neurodevelopmental disorder; PPV, positive predictive value.

Table 3.

Outcome Definition based on ICD-9 and ICD-10 Diagnostic Codes

Outcome	ICD-9	ICD-10
Autism Spectrum Disorder/Pervasive Developmental Disorder (ASD)	299.xx except 299.1x	F84.x (except F84.2, F84.3)
Attention Deficit Disorder/Hyperkinetic Syndrome of Childhood (ADHD)	314.xx	F90.x
Learning Difficulty	315.0x-315.2x	F81.0, F81.2, F81.8x, R48.0
Developmental Speech or Language Disorder	315.3x except 315.34	F80.xx (except F80.4), H93.25
Intellectual disability	317, 318.x, 319	F70-F79
Developmental Coordination Disorder (DCD)	315.4x	F82
Behavioral Disorder	312.xx, 313.xx	F63.xx, F91.x, F93.8, F93.9, F94.x, F98.8, F98.9

Abbreviations: ADHD, attention deficit hyperactivity disorder and other hyperkinetic syndromes of childhood; ASD, autism spectrum disorder/pervasive developmental disorder; DCD, developmental coordination disorder; ICD, International Classification of Diseases.

The research was supported by grant R01 MH116194 from the National Institute of Mental Health and approved by the Institutional Review Board at Brigham and Women’s Hospital.

RESULTS:

Key Findings

Based on the RPDR, we identified 4,093 algorithm-derived potential cases of ASD, 9,709 of ADHD, 2,714 of learning disability, 2,091 of speech/language disorder, 417 of intellectual disability, 229 of DCD and 1,844 of behavioral disorder. A total of 50 cases per outcome were randomly sampled and their records were reviewed. Because relevant sections in the medical records necessary to obtain information for validation could be retrieved for all 350 cases, no additional cases had to be resampled. PPVs were 94% (95% CI, 83%-99%) for ASD, 88% (76%-95%) for ADHD, 98% (89%-100%) for learning disability, 98% (89%-100%) for speech/language disorder, 82% (69%-91%) for intellectual disability, 38% (25%-53%) for DCD, and 92% (81%-98%) for behavioral disorder (Table 2).

Exploration of False Positive Cases

Four of the 6 false positive cases of ADHD were children identified based on prescriptions only (3 received clonidine and 1 received clonidine, guanfacine and methylphenidate). When excluding children who were prescribed ADHD-medications without having a diagnostic code (10 cases) from our ADHD definition, the PPV increased to 95% (88%-100%). Of the remaining 2 false positives, one child (identified based on diagnostic codes only) had symptoms of ADHD but was determined not to meet the criteria of ADHD, and one child (with diagnostic codes and methylphenidate dispensings) was diagnosed with high-functioning autism but did not meet the criteria of ADHD.

Eight out of 9 false positive cases of intellectual disability had only ICD-9 code 319 (unspecified intellectual disability) recorded. All false positives had multiple other NDDs, with learning disability, ASD and ADHD being the most common, co-occurring conditions. When excluding those identified solely through code 319 (N=22 cases), the PPV went up to 96% (89%-100%), at the cost of sensitivity.

Among the 31 false positive DCD cases were 7 children with coordination deficits that did not persist throughout childhood, 7 with visual-motor integration deficits, and 12 with coordination issues due to an underlying medical condition (such as cerebral palsy and visual impairment). Thus, if interested in studying “coordination issues” more generally, one could relax the strict definition of DCD to include these other deficits, which would increase the PPV to 90% (82%-98%). The remaining 5 false positive cases were in children with no DCD but with ADHD and ≥1 other NDD.

All 3 false positive ASD cases and the one false positive case of learning disability had ADHD. The limited information available for the 1 false positive case of speech/language disorder suggests the presence of a learning disability. All 4 false positive cases of behavioral disorder were in children with minor behavioral issues but no formal diagnosis of behavioral disorder.

The vast majority (95%) of all 350 algorithm-based cases were diagnosed at ≥3 years of age. Of the 19 cases diagnosed prior to the age of 3 (earlier than we would expect a definitive diagnosis for these disorders), only 2 were false positives – 1 case of ADHD identified based on prescriptions only and 1 case of DCD with coordination issues that resolved spontaneously at a later age.

CONCLUSIONS:

In this validation study of claims-based algorithms to identify specific NDDs, PPVs were generally high, ranging from 82% to 98%. While the PPV for DCD was low, the majority of false positives were other motor disorders that resolved spontaneously at a later age or were due to an underlying medical condition such as cerebral palsy.

There is limited data available on the validity of assessing NDDs using healthcare utilization databases. To the best of our knowledge, only claims-based algorithms for ASD and ADHD have previously been validated.^{16, 24-28}

Coleman et al. (2015) used data from four US healthcare sites obtained from the Mental Health Research Network. Of 1,272 algorithm-based cases, about one third did not have enough information on the medical charts to assess ASD diagnosis validity. Using information on the remaining 845 cases, they reported a PPV of 33% when only including confirmed ASD cases (those with complete documented assessment of ASD using the DSM-4 diagnostic criteria) and 81% when extending the definition to include probable/possible cases (which did not have all material necessary to complete a full ASD assessment based on DSM-4).²⁶ One difference between their definition and ours (which yielded a PPV of 94%) is that Coleman et al. required the presence of ≥1 ICD-9 code as compared to our more stringent requirement of ≥2 relevant encounters. Another study (Burke et al., 2013) using data from a national sample of privately insured children in the US required ≥2 ASD claims, which resulted in a PPV that fell within our 95% CI (Burke: 87.4%; 95% CI, 81.6%-91.8% vs. our study: 94%; 83%-99%).²⁴ These findings support the need of ≥2 claims to identify ASD when striving for a highly specific outcome definition.

Few studies have reported on the accuracy of ADHD diagnoses in healthcare utilization data, with PPVs generally consistent with our estimate, accounting for the width of the confidence interval.^{16, 25, 27, 28} One study reported that inclusion of patients with ≥1 ADHD-medication prescription without a documented diagnosis did not yield any additional confirmed cases.²⁵ We identified 10 cases with ≥2 ADHD-medication prescriptions and no diagnostic code, of which 4 were false positives. Excluding these 10 cases increased the PPV from 88% to 95%. Similar to these findings, a recent study by Morkem et al. (2020) using a sample from one local clinic within the Canadian Primary Care Sentinel Surveillance Network reported a PPV of 95.9% (92.6%-98.0%) when requiring either 1 diagnostic code and ≥1 ADHD-medication prescription or ≥2 encounters with a diagnostic code.²⁷ Thus, depending on the research question and the relative importance of high specificity versus sensitivity, using a more conservative definition (≥2 ADHD-related encounters or 1 encounter and ≥1 prescription) might be the preferred approach.

What is considered a valid outcome definition depends on the study objective. While there is no theoretically supported threshold for what constitutes a sufficiently high PPV for valid outcome identification in etiologic studies evaluating the relative risk of an outcome following an exposure, definitions with PPVs of >80% (and >90% for several outcomes) are generally considered valid, suggesting that our algorithms – with the exception of DCD – can accurately identify NDDs in pharmacoepidemiologic studies using claims data. Nevertheless, it has to be noted that there is uncertainty surrounding these PPVs as reflected in the 95% CI. Further, we opted for definitions expected to have high specificity so that relative risk estimates – assuming outcome classification with nondifferential sensitivity – will be unbiased.²⁹ We do not have data, however, on the number of children with the outcome who did not meet the criteria of our claims-based NDD algorithms and can therefore not report on our algorithms’ sensitivity. If sensitivity is low, using our NDD algorithms in the context of descriptive studies or drug safety studies in pregnancy will result in an underestimation of absolute risks and risk differences.

Our study has several limitations. The population is based on patients who received outpatient or inpatient services at two facilities in Boston known for their high quality of care, with diagnoses more likely to be rendered by mental health specialists rather than primary care providers. Results might therefore not be generalizable to other healthcare settings with different clinical and coding practices. However, the consistent PPVs observed for ASD and ADHD across studies using similar algorithms but applied in different healthcare settings suggest this is unlikely.

A general challenge when validating NDDs is that unlike other perinatal/childhood outcomes such as congenital malformations, NDDs are typically not informed by physical symptoms, biomarkers or imaging techniques, but are almost exclusively behaviorally based. Thus, even when NDDs are systematically assessed using best practice methods, the final diagnostic decision relies on clinical judgement.

The low PPV of only 38% for DCD shows that this outcome algorithm cannot accurately identify severe coordination disorders with functional impairment. However, depending on the context of the underlying study, the algorithm could be used to identify coordination issues in general.

Lastly, we did not validate ICD-10 codes to define NDDs. However, when converting our codes, we found good correspondence between ICD-9 and -10 codes of NDDs; for instance, ICD-9 code category 299 – pervasive developmental disorders – corresponds directly to ICD-10 code category F84 – pervasive developmental disorders. We therefore expect a very similar performance of NDD algorithms using the ICD-10 codes that we have identified. Nevertheless, in future studies it will be important to directly validate these ICD-10 based definitions using a similar approach.

Our study demonstrates that claims-based algorithms with stringent identification criteria can be used to study NDDs in children, allowing for a uniform assessment of risk across populations and over time. Further restriction criteria are needed to improve identification of true DCD cases. The respective PPVs can inform bias analyses that correct for outcome misclassification.^{30, 31}

Key points:

• Healthcare claims-based algorithms to identify children with specific neurodevelopmental disorders were validated through medical record review.

• Positive predictive values (PPVs) were high for most outcomes: 94% (95% CI, 83%-99%) for autism spectrum disorder/pervasive developmental disorder, 88% (76%-95%) for attention deficit disorder/other hyperkinetic syndromes of childhood, 98% (89%-100%) for learning disorder, 98% (89%-100%) for speech/language disorder, 82% (69%-91%) for intellectual disability, and 92% (81%-98%) for behavioral disorder; PPV was low for developmental coordination disorder (38%; 25%-53%).

• PPVs of neurodevelopmental outcomes can be used to inform bias analyses that correct for outcome misclassification. Further restriction criteria are needed to improve the classification of true developmental coordination disorder cases.