A Systematic Review of Algorithms for Identifying Pediatric Neurodevelopmental Outcomes

ABSTRACT

Purpose

Investigating pediatric neurodevelopmental outcomes (NDO) in studies using secondary data is often challenging due to heterogeneous clinical definitions and medical coding systems. This study aims to identify the algorithms used to define NDO in studies using electronic healthcare data through a systematic literature review.

Methods

A search strategy was developed to identify studies on NDO that describe phenotype algorithms from January 1, 2010, to March 10, 2025. The search strategy included terms to identify studies containing algorithms for NDO as an outcome, routinely collected healthcare data, epidemiologic designs likely to incorporate algorithms, and pregnant individuals and/or infants/children. Two independent reviewers assessed eligibility criteria and performed data extraction, with inconsistencies reviewed by a third reviewer. Descriptive statistics were used to summarize categorical and continuous variables appropriately.

Results

The review included 156 publications that implemented algorithms for NDO, with 18 of these studies validating the outcomes. Most publications studied autism spectrum disorder (ASD) (n = 103, 65.6%) and attention deficit hyperactivity disorder (ADHD) (n = 72, 45.9%) either as a single outcome or as a composite.

Conclusions

Instead of presenting NDO as a composite outcome, it is recommended to present multiple single outcomes. Validated outcomes in data from Nordic countries demonstrate a high positive predictive value when using one code for diagnoses, while more complex algorithms are required for US data. Clearly detailing and establishing the time of assessment for each NDO is critical to inform valid epidemiological estimates.

Summary.

• Healthcare data can be used to identify certain neurodevelopmental outcomes (NDO).

• Assessing multiple single NDO outcomes rather than a composite outcome is recommended.

• Algorithms using a single diagnosis code for defining NDOs demonstrate a high positive predictive value in Nordic countries databases, while complex algorithms are required in the USA.

• When comparing estimates between databases, age of patients included, length of follow‐up, outcome definitions, and estimates used (e.g., prevalence, incidence, rates, etc.) should be taken into account.

• Clearly detailing and establishing the time of assessment for each NDO is critical.

1. Introduction

Neurodevelopmental disorders (NDDs) usually refer to a group of clinical conditions that start during human development and result in significant impairments in functioning. The DSM‐5 classifies NDDs into six categories: intellectual, communication, autism, attention deficit hyperactivity, motor, and specific learning [1]. The International Statistical Classification of Diseases and Related Health Problems (ICD) in their 11th revision defines NDDs as behavioral and cognitive conditions that affect the acquisition and execution of specific intellectual, motor, and social domains [1, 2]. In clinical trials and epidemiological studies, the term “outcome” is used to define a disease or health‐related event that is monitored during a study [3]. Given the great variability in definitions, in this publication we will use the term “Neurodevelopmental Outcome” (NDO) as a broad term that encompasses NDD plus other single phenotypes from consensus guidelines such as memory, attention, intellectual functioning, academic achievement, motor skills, speech‐language skills, adaptive functioning, social–emotional and behavioral functioning, psychiatric outcomes, and health‐related quality of life [4, 5, 6].

Potential neurobehavioral effects of prenatal exposure during the early stages of fetal development can guide the need for medication safety studies [7, 8]. It becomes particularly important to study the safety of medications used during pregnancy when the fetus is exposed, and if their mechanism of action is hypothesized to impact the development of the fetal central nervous system. Fetal brain development initiates early in pregnancy and continues beyond the period of organogenesis, and any disruptions during this phase can have significant lifelong implications for the child. Medication exposure during pregnancy, especially substances capable of crossing the placenta and the fetal blood–brain barrier, may influence brain development [9].

Observational studies on medication safety typically focus on evaluating pregnancy outcomes and clinical outcomes at birth or shortly thereafter. However, this approach poses challenges in identifying difficulties that only become apparent later in childhood. To address this, long‐term follow‐up within data sources or registries is necessary, enabling population‐based studies to estimate the incidence, prevalence, and risk of NDOs. Moreover, these databases encompass various outcomes that can be studied in relation to neurodevelopment. This offers an opportunity to explore the long‐term effects of maternal exposure during pregnancy through pharmacoepidemiologic studies, utilizing routinely collected healthcare data. These data sources include electronic health records, health administrative data, and population‐wide health registers, and benefit from sufficient follow‐up when linkage of records is robust. Studies that utilize healthcare data have several advantages, including resource efficiency, large sample sizes, extended follow‐up times, and the potential to establish links between mothers and children. However, they also face certain limitations such as inaccuracies in determining outcome status, variations in exposure and outcome measurement across health systems, and the risk of misclassification due to heterogeneity in data recording practices.

For example, administrative healthcare data is primarily collected for documentation and reimbursement purposes rather than research. This can lead to the documentation of suspected diseases without confirmation and substantial variability in clinical procedures, even within the same country. In health insurance claims databases, diagnoses often require justification for insurance purposes, which can lead to potential overestimation or misclassification of disease prevalences [10]. In some healthcare or documentation systems, diagnoses may also be underreported due to factors such as unequal or limited resources, limited follow‐up in employer‐based claims, and delay in diagnosis resulting from longer waiting times [10]. However, to minimize misclassification of outcomes and enhance the reliability of NDO assessment studies, it is important to develop and employ validated algorithms. These validated algorithms can help in accurately identifying and classifying NDO based on the available healthcare data. By standardizing the way NDOs are identified and assessed, researchers can increase the consistency and reliability of their findings.

Phenotype algorithms consist of codes from biomedical vocabularies used in healthcare informatics systems. These algorithms typically involve a combination of diagnosis and medication codes and may incorporate complex logic considering time and clinical context. The validation of phenotype algorithms can be accomplished through traditional medical chart reviews, consultation with physicians, or innovative data‐driven approaches [11, 12, 13]. Once validated and confirmed to be reasonably accurate, these algorithms can be employed in studies conducted within the same data source.

The aim of this study was to conduct a comprehensive systematic literature review, summarizing the various approaches and algorithms utilized in previous research that focused on NDO using electronic healthcare databases in the USA, Canada, and Europe. Additionally, we examined the studies that included validations of these algorithms and provided a summary of their performance. Lastly, the estimates (e.g., prevalence, incidence) of NDO in the general population, as well as in unexposed groups, are presented based on the findings from the included studies.

2. Methods

This systematic review followed the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines of 2021 [14]. The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) under the registration number CRD42023489703.

2.1. Search Strategy

Table S1 shows the two searches conducted in PubMed to identify studies published between January 1, 2010 and March 10, 2025. The search strategy was developed based on previous work by Mini‐Sentinel and the Observational Medical Outcomes Partnership (OMOP) [15, 16]. The first search included terms to identify: (1) NDO outcomes, (2) routinely collected healthcare data, (3) epidemiologic studies, and (4) pregnant women and offspring. The second search was focused on identifying studies validating NDO algorithms. To ensure comprehensive coverage, the snowball technique and review of “cited references” in Web of Science were employed. To verify that all the validated studies were identified, each included study was reviewed to confirm if it utilized a validated algorithm.

2.2. Study Selection

The inclusion criteria comprised peer‐reviewed studies that utilized routinely collected healthcare data from the USA, Canada, or Europe, and were written in English. The articles were included in the review if they specified the code(s) or algorithm for identifying NDO. Two independent authors (G.M. and S.L‐.L.) reviewed all titles and abstracts of the complete list of search results. The eligible papers identified were selected for full‐text review. In the final step, each full text, as well as any supplements, were independently reviewed by two authors to ensure that the studies met the eligibility criteria. In case of discrepancies between the reviewers' assessments, a third author, A.S., was consulted.

2.3. Data Extraction and Synthesis

For each included study, one of the authors conducted a thorough full‐text review to extract relevant data. A standardized data collection form was used for this purpose. To ensure data accuracy, a second author performed quality assurance and accuracy checks on the extracted data. Table S1b presents the pre‐defined data elements in the extraction sheet used to collect the information from articles. All analyses conducted in the study were descriptive, and appropriate descriptive statistics were used to summarize categorical and continuous variables.

3. Results

3.1. General Characteristics of Studies

Figure 1 shows the PRISMA flow diagram, which explains how the studies were selected. A total of 156 studies were included in this review. Table 1 presents the characteristics of the studies. The country with the highest number of published studies was the USA (37.8%), followed by Denmark (17.9%). A substantial proportion (47.4%) required patients to have one or more diagnosis codes. For studies focusing on ADHD or major depression disorder, medication dispensation in addition to a diagnosis code was required. For studies assessing intellectual disabilities, in Europe, school records were considered. The start of follow‐up varied from 0 to 6 years of age. The majority of studies (42.3%) began identifying NDO in the newborn period, while 19.2% started investigating NDO at age 1. Approximately 56.4% of the studies only mentioned “end of follow‐up” without specifying a particular age. These studies defined the end of follow‐up similarly to traditional epidemiological studies, wherein patients are followed until the last available date of data, the patient transfers to another health plan, the patient moves to another country, or the patient passes away.

FIGURE 1.

PRISMA flow diagram.

TABLE 1.

General characteristics of studies.

Characteristic	% of total (N = 156)
Country
USA	37.8% (n = 59)
Denmark	17.9% (n = 28)
Sweden	16.0% (n = 25)
Finland	10.3% (n = 16)
Canada	9.6% (n = 15)
Norway	3.2% (n = 5)
> 1 Nordic Country	2.6% (n = 4)
France	1.3% (n = 2)
UK	1.3% (n = 2)
Data source
Nordic nationwide registers	48.7% (n = 76)
EHR	21.2% (n = 33)
Claims	16.6% (n = 26)
> 1 data source	13.5% (n = 21)
Definition of phenotype algorithm
At least one diagnosis code	47.4% (n = 74)
One diagnosis code and one medication code	4.5% (n = 7)
One diagnosis code or one medication code	9.6% (n = 15)
Two Diagnosis codes at different visits	12.2% (n = 19)
Other a	26.3% (n = 41)
Start of follow up
Born	42.3% (n = 66)
1 year	19.2% (n = 30)
2 years	13.5% (n = 21)
3 years	9.6% (n = 15)
4 to 6 years	7.7% (n = 12)
Not reported or not applicable	7.7 (n = 12)
End of follow up
“End of follow up of the study”	56.4% (n = 88)
4 to 6 years of age	8.3% (n = 13)
7 to 12 years of age	12.2% (n = 19)
13 to 18 years of age	12.2% (n = 19)
Other	10.9% (n = 17)

Note: N = total number, n = number.

^a Other = 6 studies age adults, 9 studies not reported or not applicable (refer Supporting Information: Tables).

Table 2 presents an overview of NDOs and the frequency of them across the studies. The most common outcomes studied either as single or composite were autism spectrum disorder (ASD) (n = 103, 65.6%) followed by attention deficit hyperactivity disorder/attention deficit disorder/hyperkinetic disorders (ADHD/ADD) (n = 72, 45.9%%) and intellectual disabilities (n = 37, 23.6%). Other outcomes included under “neurodevelopmental” were conduct disorders, tic disorders, behavior disorders, emotional disorders. Two studies included epilepsy and cerebral palsy under NDO [17, 18].

TABLE 2.

Outcomes in publications.

Neurodevelopmental composites	n = 25
ASD	n = 102
ADHD	n = 72
ID	n = 37
Behavioral disorder/disruptive	n = 14
Mood	n = 9
Anxiety	n = 5
Learning difficulty	n = 17
Developmental speech or language disorder	n = 15
Developmental coordination disorder/motor	n = 16
Hearing	N = 3

Note: Outcomes are counted if they were single outcomes or as composites. Number of papers overlaps given that studies usually included more than one outcome.

Abbreviations: AD, anxiety disorder; ADHD, attention‐deficit hyperactivity disorder; ASD, autism spectrum disorder; DBD, disruptive behavior disorders; DCD, developmental coordination disorder; MDD, major depressive disorder.

Refer Tables S2a and S2b for details on other NDOs, including behavioral disorder, intellectual or learning disability, and speech/language disorder.

3.2. NDO Validation Studies

Table 3 presents the 18 studies that validated NDO algorithms [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36]. There were two studies in the USA that used claims data to validate several outcomes [29, 30]. Twelve published studies validated algorithms for ASD. In the Canadian studies, the validated ASD algorithms included criteria such as age of assessment, number of claims, and setting [20, 22]. In the USA, five studies focused on ASD, including two that also investigated multiple outcomes [21, 28, 29, 30, 37]. The study with the highest PPV for ASD included diagnosis codes for pervasive developmental disorders (ICD‐9299.xx). Participants in the study by Straub et al. had to be at least 1 year of age, and the diagnoses had to be documented on two or more different dates [29]. Among the studies that utilized Nordic nationwide registers [24, 25, 26, 27], it was found that a single code for autism (ICD‐9299/ICD‐10 F84.0) had a PPV of 96% [25, 26]. One study conducted in Sweden stratified their analyses by ASD with and without ID [26].

TABLE 3.

Validation studies on neurodevelopmental outcome in healthcare databases.

Author, year	Data source name (type)	Country	Outcome	Validation technique	Algorithm codes used	Sample size of validation	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV	Goal
ASD
Bickford [19]	British Columbia Perinatal Database Registry	Canada	ASD	Clinical Autism Assessment Network and Ministry of Education	≥ 1 ICD‐9299.x or ICD‐10CA F84.X 11 different algorithms Best Sensitivity and NPV 1 physician or hospital Best Specificity and PPV by psychiatrist or neurologist	8670	14 to 75	67 to 97	83 to 90	35 to 56	Can algorithm discriminate between ASD with other developmental disorders
Brooks [20]	Administrative Ontario	Canada	ASD	153 different algorithms validated in 3 external cohorts: family practice, EMPRC, education settings, specialized clinical settings	The best one included one diagnosis in a hospital setting or three physician codes in 3 years. ICD‐9299.x, ICD10 F84.x Refer to table 1 of original study	ASD cases Family practice n = 1062 Education n = 1503 Specialized clinical settings n = 415	50.0 (40.7–88.7)	99.6 (99.4–99.7)	56.6 (46.8–66.3)	99.4 (99.3–99.6)	Algorithm to estimate prevalence of comorbid conditions
Burke [21]	Administrative private claims	USA	ASD	Chart review and abstraction based on CDC MADSP methods by experts	≥ 2 codes for ASD ICD‐9: 299.00–299.01, 299.80–299.81, 299.9 exclude 330.8, 299.1	432	—	—	87.4	—	Discriminate between ASD with other developmental disorders
Coo [22]	Administrative and education Manitoba	Canada	ASD	Four service providers identified children diagnosed with ASD	≥ 1 physician claims (ICD‐9‐CM 299) or ≥ 1 “ASD” special education record (2–5 years of age), and ≥ 2 physician claims or ≥ 1 “ASD” special education record (6–14 years of age)	1728	80 (77–83) to 88 (83–91)	—	70 (67–73) to 78 (75–81)	—	Surveillance of ASD
Coleman [23]	Administrative Mental Health Research Network	USA	ASD	Chart review and abstraction based on CDC MADSP methods by experts	≥ 2 codes at different visits ICD‐9: 299.0, 299.38, 299.9	1272	—	—	Broad case definition (confirmed, probable, or possible case): 33 Confirmed cases only: 81	—	Definition for population based studies
Hagberg [24]	CPRD Electronic health records	UK	ASD	Chart review, hospital, assessments	≥ 1 code Read codes: E140.00, E140000, E140100, E140.12, E140.13, E140z00, Eu84000, Eu84011, Eu84012, Eu84100, Eu84z11, Eu84500, Eu84.00, Eu84y00, Eu84z0	2154			91.9		Evaluate change of diagnostic criteria and terminology over time.
Idring [25]	NPR and regional registers	Sweden	ASD +/− ID	Case notes reviewed by physician reviewer. Cross‐validation was done with co‐existing cases in the National Twin Study	≥ 1 ICD‐9299, ICD‐10 F84 Stratify by having/not having an ID: ICD‐9317–319, ICD‐10 F70–79, DSM‐IV 317–319	177	—	—	96	—	Algorithm used to estimate prevalence
Lampi [26]	EHR	Finland	ASD	Autism Diagnostic Interview clinical tool	≥ 1 ICD‐10: F84.0	95	—	—	96	—	Algorithm to be used in epidemiological studies
Lauritsen [27]	Register	Denmark	Autism	Case records	≥ 1 ICD10 F84.0, F84.5, F84.1, F84.8, F84.9 Exclude 299.1295*, 330.8759.5759.83	499			94		Identify if patients with code have autism
Lingren [28]	EHR	USA	ASD	Chart review	≥ 1 ICD9 299.0, 299.80, 299.9 exclude	302	86				Algorithm that can be used in EHR in USA
Straub [29]	RPDR Claims inpatient and outpatient	USA	ASD	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates with ICD‐9 codes, takes into account age, for ADHD includes medication	350 (50 per diagnosis)	—	—	94	—	Identify algorithms for RR estimates
Shi [30]	KPNC	USA	ASD	Review of medical records	Refer to Table S1 ≥ 1 codes	180	100	100	99.2	100	Validity before, during and after pandemic
ADHD
Butt [31]	Ontario Health Admin. database	Canada	ADHD	Physician medical records	Two codes of ADHD or one medication	49 031	83.2	98.6	78.6	98.9	Incidence and prevalence
Morkem [32]	Primary Care Sentinel Surveillance Network	Canada	ADHD	Chart review by blinded abstractor	Age ≥ 4, ICD‐9314 code and/or prescription, exclusion of ICD codes for psychiatric and related codes Supplement of original study	492	—	—	98.0 (92.5, 99.5)	95. (92.9–98.6)	Algorithm used to estimate prevalence
Gruschow [33]	Children's Hospital of Philadelphia Health Care Network	USA	ADHD	Chart review	≥ 1 ICD‐9‐CM diagnosis code of 314.x	2030	96 (95–97)	98 (97–99)	83–93	99 (99–99)	Epidemiologic and clinical studies
Daley [34]	SPAN (10 large healthcare organizations)	USA	ADHD	Random sample of records manually reviewed.	≥ 1 ICD9: 314.0× excludes codes 299.×, 317.×, 318.×, 319.×	100 age 3 to 5 400 age 6 to 9	Age 3–5: 89.8 Age 6–9: 94.2				Algorithm used to estimate incidence
Peteri Joelsson [35]	Hospital discharge register	Finland	ADHD	Telephone call to parents	≥ 1 ICD‐9: 314.×, ICD10: F90.×	69	—	—	88	—	Algorithm for demographic characteristics and comorbidities
Mohr‐Jensen [36]	Danish Psychiatric Central Research Register	Denmark	ADHD	Patient records of psychiatric hospitals	≥ 1 ICD‐10: F90.0, F90.1, F90.8, F90.9	387	—	—	86.8	—	Algorithm used to estimate incidence
Straub [29]	RPDR Claims inpatient and outpatient	USA	ADHD	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates with ICD‐9 codes, takes into account age, medication	350 (50 per diagnosis)	—	—	88 (76–95)	—	Identify algorithms for RR estimates
Shi [30]	KPNC	USA	ADHD	Review of medical records	Refer to table 1 of supplemental ≥ 1 codes	180	100	99.9	99.2	100	Validity before, during and after pandemic
Other NDO Outcome
Shi [30]	KPNC	USA	DBD	Review of medical records	Refer to Table S1 ≥ 1 codes	180	100	100	100	100	Validity before, during and after pandemic
Shi [30]	KPNC	USA	MDD	Review of medical records	Refer to Table S1 ≥ 1 codes	180	100				Validity before, during and after pandemic
Shi [30]	KPNC	USA	Anxiety	Review of medical records	Refer to Table S1 ≥ 1 codes	180	87.7	100	100	99.2	Validity before, during and after pandemic
Straub [29]	RPDR Claims inpatient and outpatient	USA	LD	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates CD‐9 codes, takes into account age	50	—	—	98 (89–100)	—	Identify algorithms for RR estimates
Straub [29]	RPDR Claims inpatient and outpatient	USA	S/L	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates CD‐9 codes, takes into account age	50	—	—	98 (89–100)	—	Identify algorithms for RR estimates
Straub [29]	RPDR Claims inpatient and outpatient	USA	DCD	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates with ICD‐9 codes, takes into account age	50	—	—	38 (25–53) when including “coordination issues” then 90 (82–98)	—	Identify algorithms for RR estimates
Straub [29]	RPDR Claims inpatient and outpatient	USA	ID	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates CD‐9 codes, takes into account age	50	—	—	82 (69–91)	—	Identify algorithms for RR estimates
Straub [29]	RPDR Claims inpatient and outpatient	USA	BD	Review of medical records by two physicians using DSM‐5 criteria	Refer to table 1 of original study ≥ 2 dates CD‐9 codes, takes into account age	50			92 (81–98)		Identify algorithms for RR estimates

Note: There were 18 publications which validated different outcomes.

Abbreviations: ADHD = attention deficit disorder or other hyperkinetic syndromes, ASD = autism spectrum disorder/pervasive developmental disorder, BD = behavioral disorder, CDC MADSP = Centers for Disease Control and Prevention's Mental and Behavioral Health Data and Surveillance Program, DBD = disruptive behavior disorder, DCD = developmental coordination disorder, DSM = Diagnostic and Statistical Manual of Mental Disorders, EMPRC = Emergency Medical and Public Health Response Coordination, ICD = International Classification of Diseases, ID = intellectual disability, KPNC = Kaiser Permanente Northern California, LD = learning disability, NPR = National Patient Registry, RPDR = Research Patient Data Registry, RR = Relative Risk, S/L = speech/language disorder, SPAN = Scalable Partnering Network.

There were eight studies that identified validated algorithms for ADHD specifically [32, 33, 34, 35, 36]. The studies with the highest PPV included the presence of ADHD medication in the algorithm. In the study by Straub et al. (2021), they proposed criteria for validating ADHD, which included children being at least 2 years of age and having two or more ICD‐9 codes documented on different days, two or more prescribed medications for ADHD, or at least one code plus one medication [29]. This approach achieved a PPV of 88%.

There were two studies that validated several NDDs, which included ASD, ADHD, learning disability, speech/language disorder, developmental coordination disorder, intellectual disability, and behavioral disorder. The authors included children older than 2 years of age for ADHD, intellectual disability, developmental coordination disorder, and behavioral disorder; older than 1.5 years of age for developmental speech or language disorder; and older than 1 year for ASD. All the outcomes required at least 2 ICD diagnosis codes on different dates. For ADHD, the authors required at least two codes from either diagnosis or treatment domains. The study by Shi et al. 2024, in addition to validating ADHD and ASD, includes major depression disorder, anxiety, and disruptive behavior disorders for which the authors required only one code [30].

3.3. Non Validation Studies

The studies whose aim was not to validate an algorithm are presented in the Tables S2–S6. If the study used a validated algorithm from the same study or a previously published study, it was noted.

ASD included autism, Asperger's Disorder, or Pervasive Developmental Disorder. There were 71 studies that were non‐validation studies. The definitions of the age of assessment, and codes used to define the algorithms varied across studies. Some studies excluded patients with ID, Rett syndrome, disintegrative disorder, genetic syndromes, viral disease, postnatal injury, and fetal alcohol syndrome (Table 4).

TABLE 4.

Exclusion criteria included in some studies that should be considered depending on the database used.

Outcome	Exclusion	References
Autism	Rett's syndrome, disintegrative disorder	[21, 34, 38, 39]
	Rett's syndrome	[40]
	Disintegrative disorder	[29, 41]
	Genetic syndromes, viral diseases, postnatal injury, and Fetal Alcohol Syndrome	[42, 43]
	ID, MR	[35, 44, 45, 46, 47, 48]
ADHD	Autism	[49]
	Profound mental retardation	[48]
	Severe intellectual disability	[48]
Hyperkinetic disorders	Developmental and intellectual disorders	[50]
Mental retardation	Trisomy 13, 16–18, other chromosomal aberrations, Prader–Willi Syndrome, Rett's Syndrome, phenylketonuria, Fragile X Syndrome, postnatal injury, prenatal rubella, meningitis, encephalitis, and Fetal Alcohol Syndrome, developmental delay	[42, 51]
Intellectual disability	Autism spectrum disorder	[52]
Developmental speech or language disorder	Speech and language developmental delay due to hearing loss	[29]

Abbreviation: ADHD = attention deficit disorder or other hyperkinetic syndromes.

There were 47 non‐validated studies on ADHD. Both in the USA and Europe, some studies defined ADHD as having one or more diagnosis codes for ADHD, while others included ADHD medication. The start of the interval to identify the outcome varied among studies, with some starting at birth, some at age two or older, and others not specifying the age but following patients until the “end of follow‐up”. Some studies excluded patients with autism, profound mental retardation (MR) or severe MR (Table 4). Please refer to Tables S4a and S4b.

In relation to the studies assessing MR, there were studies that excluded known causes of MR: Trisomy 13, 16–18, other chromosomal aberrations; Prader–Willi Syndrome; Rett's Syndrome; phenylketonuria; Fragile X Syndrome; postnatal injury; prenatal rubella; meningitis; encephalitis; and Fetal Alcohol Syndrome; developmental delay (Table 4). The age ranges assessed differed across the studies, with one study focusing on ages 1 to 4 years, while the rest covered the period from birth to the end of follow‐up. Medications used for treating mood disorders were included in the algorithm, along with the diagnosis code to identify individuals with mood disorders (Table S6).

3.4. Epidemiological Estimates

With regards to the estimates presented, there was great heterogeneity in outcome definitions, age of patients, length of follow up, and estimates used (Tables S2–S6). The estimates, including how the authors referred to them, were included in the last column: incidence, cumulative incidence, incidence rate, incidence proportion, prevalence, prevalence rate, percentage.

4. Discussion

This systematic review identified 156 studies that used an algorithm to define NDO in analyses of routinely collected healthcare data. Eighteen studies focused on validating NDOs. Most of the studies only evaluated ASD and ADHD. Other NDO outcomes included ID/MR, learning disability, speech/language disorder, executive functioning, and mood/anxiety disorders.

The validation studies conducted in the Nordic countries showed that using one or more diagnosis codes demonstrated high accuracy. In the USA, the situation was different due to the utilization of diagnosis codes for ruling out diagnoses in claims data. It is possible that diagnosis codes in Nordic data are restricted to those recorded by specialists (e.g., psychiatrists) and thus observing one instance of a diagnosis code is sufficient to identify cases. Whereas in the US, rule‐out diagnosis codes exist in the data, and more complex algorithms are required to improve algorithm accuracy. Consequently, the authors recommend employing more complex algorithms including age at diagnosis, the presence of two or more codes, exclusion of codes, and inclusion of medications. One study was identified that validated several outcomes using ICD‐9 codes in a claims database in the USA [29]. Despite the high quality of this work, the validation process was performed for cases identified via ICD‐9 codes, which might not be applicable in other settings and would need to be replicated using other coding systems such as ICD‐10.

It should be noted that, if the final goal is to apply algorithms to estimate relative risk measures, phenotype algorithms may be created to maximize specificity as this approach could result in less bias. However, the same algorithm may have less sensitivity and might not be appropriate for estimating population incidence/prevalence. PPV and NPV depend on disease prevalence, while sensitivity and specificity do not. With higher disease prevalence, less positive results will be false positives, which will increase PPV. On the contrary, NPV will decrease as more negative results will be false negatives. Therefore, it is recommended to consider all performance metrics in validation studies (sensitivity, specificity, PPV, and NPV) [53], as well as comparing the prevalence in both the validation cohort and the main database. In addition, when using an algorithm, the objective of the research needs to be taken into account [54].

With regards to the assessment time of NDO, some researchers started evaluating the outcomes at birth, while others started at different ages. In approximately 56.4% of the studies, researchers did not define a specific follow‐up period. In studies utilizing European databases, individuals were typically followed until the end of the study period, while studies using claims databases often followed individuals until they left the insurance plan. This difference reflects the nature of these data sources and eligibility for healthcare benefits in different healthcare settings. It is uncommon to diagnose neurodevelopmental problems before the age of 1, and for many, the average age of diagnosis is around the time of school entry, so an imbalance in ages between comparison groups could lead to inconsistent results. In a validation study, it was proposed to use an age older than 2 for identifying ADHD, ID, BDs, and learning disabilities [29]. For language disorders, an age older than 1.5 is suggested, and for autism, an age older than 1 [29, 55]. The validation studies conducted in the Nordic countries did not give a recommendation on what age group to include in their algorithm.

The epidemiological estimates of NDO in unexposed populations varied across the studies, primarily due to differences in variables that were included among the data sources, criteria for diagnoses, age of diagnoses, exclusion criteria, and estimates used (Incidence, cumulative incidence, incidence rate, incidence proportion, prevalence, prevalence rate, percentage). Differences included the number of codes needed for diagnoses, age of diagnoses, and exclusion criteria (e.g., some studies evaluating autism had excluded patients with ID). Some studies reported estimates in the entire population, whereas others reported by exposure status. When comparing epidemiological estimates between populations, it is critical to compare studies that are not heterogeneous.

There are several high‐level learnings from this systematic review's findings. Most of the studies focused on assessing ASD and ADHD. Very few studies discussed the reason for the selection of these outcomes above others NDOs. It is likely that these outcomes were selected solely based on their availability in the database, ADHD being the most common type of NDO, or ASD being a more severe NDO with increasing prevalence over the past decades [6, 56]. The authors of this review emphasize the importance of having a rigorous approach for selecting and including NDOs, based on biological plausibility, signals from previous studies, or associations with similar medication effects.

As a future best practice, the authors of this review recommend that multiple individual outcomes be presented instead of presenting NDO as a composite measure. The brain is a set of complex networks that support different functions and whilst there is some overlap, there is also some degree of independence. Deficits in one network, leading to a specific NDO difficulty, do not provide information on the functioning of other neuronal networks and the skills reliant on them. Thus, investigating ASD will not necessarily inform on cognitive functioning. Focusing on a limited set of NDOs creates a high probability of missing deficits in other areas, and combining different NDOs into a single grouping may mask higher risks when only one or more NDOs are impacted.

In conclusion, there is a need for validation studies for different outcomes besides AD, ADHD, and ID. Electronic healthcare record databases with linked maternal medication exposure and child outcomes appear to be widely utilized for certain NDOs such as ASD and ADHD. However, their use has been more limited for other NDOs, restricting their ability to provide full phenotypic information on the outcomes following a medicine exposure in utero. Furthermore, standardized approaches to measuring NDO outcomes regarding the algorithm and follow‐up time are necessary, specifically for studies designed to fulfill regulatory commitments. In Europe, one or more diagnosis codes appear to be adequate for NDO measurement, while in US claims databases, more complex algorithms are needed. Instead of presenting NDO as a composite, the authors recommend presenting multiple single outcomes. When presenting the incidence/prevalence, stratifying by age is preferred to compare across studies. In all cases, studies are encouraged to use validated outcomes either from a reference using the same database or by including outcome validation within the study protocol. Five recommendations are made to improve future pregnancy pharmacovigilance studies (Table 5).

TABLE 5.

Take away messages.

Neurodevelopmental disorders go beyond autism spectrum disorders and attention deficit hyperactivity disorders. There are several other outcomes that fall under this category, such as intellectual disability, behavioral disorders, learning disabilities, memory dysfunction, attentional difficulties, speech/language disorders, executive functioning impairments, and coordination disorders, among others.

Instead of presenting neurodevelopmental disorders as a composite measure, it is recommended to present multiple individual outcomes separately stratified by age.

Validated outcomes in registries of Nordic countries demonstrate a high positive predictive value when using one code for diagnoses, while in US claims data, more complex algorithms are required.

Clearly detailing and establishing the time of assessment and age range for each neurodevelopmental outcome is critical to inform prevalence and for comparison with other studies.

Applying rigorous validated definitions for neurodevelopmental outcomes from a reference using the same database or by validating the outcomes is highly encouraged to assure evidence reliability.

4.1. Plain Language Summary

This review article summarizes measurement methods for assessing unfavorable brain development in children when using data in large healthcare data repositories. A systematic search identified 156 studies using electronic healthcare data from the USA, Canada, and Europe, with most focusing on ASD (65.6%) and ADHD (45.9%). The authors recommend analyzing individual conditions rather than grouping them into a single outcome, for example, ADHDs, ASDs, intellectual disabilities, learning disability, speech/language disorder, developmental coordination disorder, intellectual disability, behavioral disorder, and mood/anxiety disorders. Studies that validated measurement methods against expert reviews indicated that one medical diagnosis code sufficed in European databases, whereas the USA databases required a more complex combination for accuracy. Clearly detailing the timing of assessments for each condition is essential for informing epidemiological estimates.

Disclosure

An earlier version of this article was presented at the 2024 International Society and Pharmacoepidemiology Annual Meeting, ISPE 2024, Berlin.

Ethics Statement

S.L‐.L., A.S., R.L.D., J.S., A.S.A., M.L.N‐.L., M‐.L.K., M.S., S.C., and G.M. are employees of Pharmaceutical Companies. The statements presented do not necessarily represent the position of the company.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1: pds70196‐sup‐0001‐Supinfo.docx.

Lopez‐Leon S., Wen X., Gaitonde S., et al., “A Systematic Review of Algorithms for Identifying Pediatric Neurodevelopmental Outcomes,” Pharmacoepidemiology and Drug Safety 34, no. 9 (2025): e70196, 10.1002/pds.70196.