Skip to main content
NCBI home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2022 Nov 11;115(3):302–317. doi: 10.1002/bdr2.2112

Validation of case definition algorithms for the ascertainment of congenital anomalies

Yonabeth Nava de Escalante 1,✉, Aanu Abayomi 1, Sylvie Langlois 2,3, Xibiao Ye 1,4, Anders Erickson 1, Henry Ngo 1, Rosemary Armour 5, Reiko Okamoto 1,6, Laura Arbour 2,7, Tanya Bedard 8, Kenny Der 4, Margot Van Allen 2,9, Erik Skarsgard 10, Martin Lavoie 1, Bonnie Henry 1,11
PMCID: PMC10099451  PMID: 36369700

Abstract

Background

Congenital anomalies (CA) are one of the leading causes of infant mortality and long‐term disability. Many jurisdictions rely on health administrative data to monitor these conditions. Case definition algorithms can be used to monitor CA; however, validation of these algorithms is needed to understand the strengths and limitations of the data. This study aimed to validate case definition algorithms used in a CA surveillance system in British Columbia (BC), Canada.

Methods

A cohort of births between March 2000 and April 2002 in BC was linked to the Health Status Registry (HSR) and the BC Congenital Anomalies Surveillance System (BCCASS) to identify cases and non‐cases of specific anomalies within each surveillance system. Measures of algorithm performance were calculated for each CA using the HSR as the reference standard. Agreement between both databases was calculated using kappa coefficient. The modified Standards for Reporting Diagnostic Accuracy guidelines were used to enhance the quality of the study.

Results

Measures of algorithm performance varied by condition. Positive predictive value (PPV) ranged between approximately 73%–100%. Sensitivity was lower than PPV for most conditions. Internal congenital anomalies or conditions not easily identifiable at birth had the lowest sensitivity. Specificity and negative predictive value exceeded 99% for all algorithms.

Conclusion

Case definition algorithms may be used to monitor CA at the population level. Accuracy of algorithms is higher for conditions that are easily identified at birth. Jurisdictions with similar administrative data may benefit from using validated case definitions for CA surveillance as this facilitates cross‐jurisdictional comparison.

Keywords: algorithm validation, congenital anomalies, health administrative data, surveillance, validation study

1. INTRODUCTION

Congenital anomalies (CA) are one of the leading causes of neonatal and infant mortality, and long‐term disability in both developed and underdeveloped countries (Christianson, Howson, & Modell, 2006; Ely & Driscoll, 2020; Public Health Agency of Canada, 2013). Also known as birth defects, they refer to an abnormality of the body structure or function that occurred during the intrauterine period which can be diagnosed before or any time after birth (World Health Organization, 2020). Even though CA have a low prevalence, they can be a significant burden on families, society, and the healthcare system (Christianson et al., 2006; Glinianaia, Tennant, & Rankin, 2017).

Information pertaining to CA is used to conduct epidemiological research, monitor trends, guide resource allocation, and support program planning (Mburia‐Mwalili & Yang, 2014). This information can be collected via active surveillance systems—those that have dedicated staff for case identification and case confirmation from primary data sources; or passive surveillance systems—those that receive or extract information primarily from health care administrative databases (Luquetti & Koifman, 2011; Mburia‐Mwalili & Yang, 2014). Active surveillance systems are the ideal method for case ascertainment and provide more accurate and higher quality information (Mburia‐Mwalili & Yang, 2014; Reichard et al., 2016), the trade‐off being that they are more resource intensive, requiring more staff for data abstraction and having higher costs than passive surveillance systems (Mburia‐Mwalili & Yang, 2014; Miller, 2006; Jason L. Salemi et al., 2012). Under limited funding, passive surveillance systems have proven to be a cost‐effective alternative method for CA surveillance (Jason L. Salemi et al., 2016). A hybrid surveillance system uses a combination of activities from passive and active surveillance to address some of the limitations of passive surveillance (World Health Organization, 2020).

Health care administrative databases such as hospital discharge, medical billing claims data, and vital event registries are frequently used for health research and disease surveillance, including for CA (Andrade et al., 2017; Cronk et al., 2003; Lix et al., 2018; Jason L. Salemi et al., 2016). Information in these data sources is routinely collected during each patient encounter which makes it more readily available than primary data (Cadarette & Wong, 2015). Mother‐infant dyads are particularly important for CA surveillance because they are essential to understanding causal pathways, protective factors, and risk factors. The creation of these dyads is possible with health administrative data. Indeed, some administrative databases, such as the BC Perinatal Data Registry, already contain linked information for mothers and infants (Frosst et al., 2015). Administrative databases have the advantage of having one or multiple unique identifiers that can be used to link records across databases and track individuals across their lifespan interactions with the health system (Andrade et al., 2017; Cadarette & Wong, 2015). The availability, population coverage, and relatively low cost of these databases make them a feasible source for population health surveillance (Antoniou, Zagorski, Loutfy, Strike, & Glazier, 2011; Cadarette & Wong, 2015; Cronk et al., 2003), particularly for CA surveillance.

A key aspect related to health care administrative databases is the possibility to use case definition algorithms for the purpose of identifying cases with CA. A typical case definition algorithm might include/exclude specific diagnostic codes, health encounters, and/or include a specific number of data sources in which diagnoses appear (refer to Table 1). Although this approach has proven to be an effective way to monitor disease status (J. Salemi, Rutkowski, Tanner, Matas, & Kirby, 2018; Kharbanda et al., 2017; Shiff, Oen, Rabbani, & Lix, 2017; Nasr, Sullivan, Chan, Wong, & Benchimol, 2017), it is important to note that the development of standardized case definition algorithms can be challenging because of jurisdictional differences, differences in disease classification nomenclature, revisions in disease classification coding systems over time (Johnson & Nelson, 2013; Shiff et al., 2017) and concerns about the validity of diagnostic codes (Farr et al., 2021). Also, differences in presentation and severity of CA result in diagnostic variability, which influences ascertainment (Langlois, Sheu, & Scheuerle, 2010). Administrative databases are not developed specifically for disease surveillance or health research (Antoniou et al., 2011). Therefore, assessing the validity of administrative data sources is a necessity. This is relevant since changes in case definition algorithms not only influence prevalence estimates (Eltonsy, Forget, & Blais, 2017; J. Salemi et al., 2018) but also limit the ability to perform longitudinal surveillance.

TABLE 1.

Algorithms, diagnostic codes, and data sources used in the British Columbia Congenital Anomalies Surveillance System to identify specific congenital anomalies

Congenital anomalies Algorithm ICD‐10 code ICD‐9 code Exclusion Data source
Anencephaly and similar defects 1 VE record or 1 perinatal record or ≥1 hospitalization in the first month of life Q00.0 and Q00.1 740.0, 740.1 Amniotic band/limb body wall spectrum a and acalvaria b PDR, VSA, DAD
Encephalocele 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q01.0–Q01.9 742.0 Anencephaly (Q00.0) and amniotic band/limb body wall spectrum a PDR, VSA, DAD
Microcephaly 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q02 742.1 Anencephaly (Q00.0, encephalocele (Q01.0–Q01.9) PDR, VSA, DAD
Hydrocephalus 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q03.0–Q03.9 742.3 Acquired hydrocephalus (G91.x); spina bifida with hydrocephalus (Q05.0–Q05.4) PDR, VSA, DAD
Anomalies of the corpus callosum 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q04.0 742.21 None PDR, VSA, DAD
Holoprosencephaly and arhinencephaly 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q04.1–Q04.2 n/a None PDR, VSA, DAD
Spina bifida 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q05.0–Q05.9 741, 741.0, 741.9 Spina bifida occulta (Q76.0); tethered cord c PDR, VSA, DAD
Anophthalmos and microphthalmos 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q11.0–Q11.2 743.0–743.1 None PDR, VSA, DAD
Congenital cataract 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q12.0 743.3 None PDR, VSA, DAD
Common truncus 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q20.0 745.0 Pulmonary valve atresia (Q22.0) with VSD (Q21.0) PDR, VSA, DAD
Transposition of great arteries 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q20.3 745.1 None PDR, VSA, DAD
Endocardial cushion defect 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q21.2 745.6 None PDR, VSA, DAD
Tetralogy of fallot 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q21.3 745.2 None PDR, VSA, DAD
Tricuspid atresia/stenosis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q22.4 746.1 None PDR, VSA, DAD
Hypoplastic left heart syndrome 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q23.4 746.7 None PDR, VSA, DAD
Coarctation of the aorta 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q25.1 747.1 None PDR, VSA, DAD
Choanal atresia 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q30.0 748.0 None PDR, VSA, DAD
Cleft palate 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q35.1–Q35.9 749.0 Cleft uvula (Q35.7) PDR, VSA, DAD
Cleft lip 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q36 749.1 Median cleft lip with holoprosencephaly d PDR, VSA, DAD
Cleft lip with cleft palate 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q37 749.2

Amniotic band‐limb body wall spectrum a

PDR, VSA, DAD
Esophageal atresia 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q39.0–Q39.2 750.3 None PDR, VSA, DAD
Small intestine atresia/stenosis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q41.0–Q41.9 751.1 None PDR, VSA, DAD
Anorectal atresia/stenosis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q42.0–Q42.3 751.2 None PDR, VSA, DAD
Hirschsprung 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q43.1 751.3 None PDR, VSA, DAD
Atresia of bile ducts 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q44.2 751.6 None PDR, VSA, DAD
Cryptorchidism 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q53.1–Q53.9 752.5 Females, birthweight ≤2500 g, gestational age <37 weeks, stillbirths, ectopic testis (Q53.0) PDR, VSA, DAD
Hypospadias/epispadias 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life

Q54.0–Q54.9;

Q640

 752.6 Females, congenital cordee (Q54.4) e PDR, VSA, DAD
Indeterminate sex 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q56 752.7 None PDR, VSA, DAD
Renal agenesis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q60.0–Q60.2 753.0 None PDR, VSA, DAD
Cystic kidney 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life or > 2 physician encounter Q61.1–Q61.5; Q61.8; Q61.9 753.1 None PDR, VSA, DAD, MSP
Exstrophy of the urinary bladder 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q64.1 753.5 None PDR, VSA, DAD
Lower urinary tract atresia and stenosis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q64.2–Q64.3 753.8 and 753.6 None PDR, VSA, DAD
Upper limb reduction 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q71.0–Q71.9 755.2 None PDR, VSA, DAD
Lower limb reduction 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q72.0–Q72.9 755.3 None PDR, VSA, DAD
Congenital diaphragmatic hernia 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life or 1 physician encounter with procedure code CV70604 Q79.0 756.6 None PDR, VSA, DAD, MSP
Omphalocele 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q79.2 756.7 None PDR, VSA, DAD
Gastroschisis 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q79.3 756.7 None PDR, VSA, DAD
Trisomy 21 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life or ≥3 physician encounters Q90.0–Q90.9 758.0 None PDR, VSA, DAD, MSP
Trisomy 18 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q91.0–Q91.3 758.2 None PDR, VSA, DAD
Trisomy 13 1 VE record or 1 perinatal record or ≥1 hospitalization in the first year of life Q91.4–Q91.7 758.1 None PDR, VSA, DAD
Open in a new tab

Note: Composition of group conditions are: Selected chromosomal defects—trisomy 13, trisomy 18, down syndrome; Neural tube defects—anencephaly, encephalocele, spina bifida; oro‐facial clefts‐cleft lip only, cleft palate only, cleft lip and cleft palate; Selected genital anomalies—hypospadias, epispadias, indeterminate sex; Hypospadias—hypospadias/epispadias; Selected abdominal wall defects—omphalocele, gastroschisis, prune belly; Selected gastrointestinal defects—esophageal atresia/stenosis, small intestine absence/atresia/stenosis, ano‐rectal absence/atresia/stenosis, atresia of bile ducts, Hirschsprung disease; Selected congenital heart defects—common truncus, transposition of great vessels, endocardial cushion defects/AVSD, tetralogy of fallot, tricuspid atresia, hypoplastic left heart syndrome, coarctation of aorta; Limb deficiency defects—upper limb reduction, lower limb reduction; Selected sense organ—choanal atresia, anophthalmos/microphthalmos; Selected urinary tract defects—renal agenesia, cystic kidney, bladder and cloacal exstrophy, lower urinary tract obstruction.

Abbreviations: DAD, Discharge Abstract Database; EMR, Electronic medical record; MSP, physician billing claims database; PDR, Perinatal Data Registry; VE, vital event (births, deaths or stillbirths); VSA, Vital Statistics Agency Events Registry; VSD, ventricular septal defect.

a

Initially identified with the ICD code Q79.8 and confirmed by an autopsy report.

b

Initially identified with the ICD code Q75.8 and confirmed by an autopsy report.

c

Initially identified with the ICD code Q07 and confirmed by EMR or autopsy report.

d

Initially identified with the ICD code Q36 or Q37 and Q04.x and confirmed by EMR or autopsy report.

e

Only for hypospadias.

The accuracy of algorithms must be assessed to ensure the reliability and validity of the information collected (Shiff et al., 2017). Algorithm accuracy is determined in comparison to a data source that is considered to be the reference standard (Chubak, Pocobelli, & Weiss, 2012). Several studies have reported on measures of accuracy for specific CA case definition algorithms (Eltonsy et al., 2017; J. Salemi et al., 2018; Metcalfe, Sibbald, Lowry, Tough, & Bernier, 2014). These measures can be used to guide the selection of algorithms for surveillance systems; however, evidence suggests that the validity of diagnostic codes varies by data source, type of CA, and geographic region (Metcalfe et al., 2014; Riley, Phyland, & Halliday, 2004). For instance, Metcalfe et al. (2014) reported that in Alberta, Canada, the hospital discharge database is less sensitive for capturing cases of neurological and digestive anomalies. Therefore, high‐quality validation studies are fundamental to understand the strengths and limitations of data sources, including the risk of information bias (Fox, Lash, & Bodnar, 2020). Validated case definition algorithms are a necessity for jurisdictions performing passive or hybrid CA surveillance.

The purpose of this study is to validate the case definition algorithms used in the BC Congenital Anomalies Surveillance System (BCCASS), a hybrid surveillance system in British Columbia (BC), Canada, and to report on measures of algorithm performance while applying the modified Standards for Reporting Diagnostic Accuracy (STARD) criteria, a checklist developed to enhance the quality of health administrative validation studies (Benchimol et al., 2011).

2. METHODS

2.1. Study design

2.1.1. Data sources

Reference standard—The Health Status Registry (HSR)

The HSR, a population‐based surveillance system, was established in BC in 1952 to monitor CA and developmental disabilities (Mott, 1963). In 1992, amendments to the BC provincial Health Act established the legislative mandate and responsibilities for the registry which included the recording and classification of information related to CA, genetic and chronic disabling conditions. The new amendment gave legislative authority to the registry to collect information related to these conditions, so a person acting on behalf of the HSR could request any individual or organization to submit information pertaining to these conditions directly to the registry, provided the information would facilitate the registry's purpose or benefit the public (B. Lowry & Bedard, 2013). The system received information from more than 50 reporting sources and performed active follow up of cases with reporting sources to ensure the validity of the information collected. The HSR used probabilistic matching to link records pertaining to registrants. Matching depended on the degree of confidence of the match between the loading record and any existing HSR registrants. If the degree of confidence was high, the submitted record was used to update an existing matching record. If the degree of confidence was moderate, the submitted record went into a queue until clerical staff approved or rejected the match. If the degree of confidence was low, the record was uploaded as a new record.

While reporting was mandatory after 1992 (B. Lowry & Bedard, 2013) it should be noted that there was no active enforcement of that mandate. Sources that reported to the HSR included: the Vital Statistics Agency, provincial Public Health Units, special treatment centers located in outpatient clinics at BC Children's Hospital, Medical Genetics, voluntary health agencies such as the Canadian Arthritis Society, obstetrical discharge summaries, discharge abstracts from hospitals, and private physicians (R. Lowry, Miller, Scott, & Renwick, 1975). In the initial phases of the implementation, the registry established follow‐up cohorts at the ages of 7 and 14 years. These cohorts generated valuable information and resulted in a substantial increase in research projects between the 1970s and 2000s (R. Lowry et al., 1975). For instance, between 1961 and 2011 there were 127 peer‐reviewed articles published using HSR data with the majority of research projects occurring before 2001 (B. Lowry & Bedard, 2013).

The HSR also contributed to global CA surveillance efforts. In 2001, the registry became a member of the International Clearinghouse for Birth Defect Surveillance and Research (ICBDSR), an organization that conducts worldwide CA surveillance (B. Lowry & Bedard, 2013). During this period, the HSR was considered an exemplary model of CA surveillance in the Americas (B. Lowry & Bedard, 2013). However, due to resource constraints, support for the registry dwindled with resultant data quality decline. The last HSR report was published in 2005 and included CA data collected up to and including 2002 (B. Lowry & Bedard, 2013).

In 2015, a BC Ministry of Health environment scan concluded that “the HSR data were no longer actively monitored for completeness and accuracy, the number of data sources contributing to the registry had diminished, and the quality of data had declined such that it is no longer considered complete enough for surveillance purposes” (de Escalante et al., 2022). Given the existence of multiple administrative databases and the capacity to link them, a decision was made to implement a new system that would leverage these administrative databases while utilizing less resources than the HSR.

For the current validation study, all cases identified in the HSR with a birth registered in BC, who were born between April 1, 2000 and March 31, 2002, were included.

The BC Congenital Anomalies Surveillance System

This is a hybrid, population‐based surveillance system implemented in 2020 utilizing health care administrative data to capture cases with selected CA. The BCCASS ascertains selected CA in BC resident mothers with a delivery or termination of pregnancy in BC after March 31, 2000. Specifically, the BCCASS identifies CA among BC residents with (1) gestations over 19 weeks of gestational age in which the pregnancy was terminated due to a CA; (2) stillbirths and/or livebirths born in BC with one or more CA, and (3) infants up until the age of 1 year diagnosed with one or multiple CA born after March 31, 2000. The data sources include: the BC Perinatal Data Registry, the Discharge Abstract Database, the Vital Events Registry from the BC Vital Statistics Agency, and the BC medical billing claims data. Access to the data sources is possible through data sharing agreements with the relevant organizations. New data sources to be negotiated will be determined by the BCCASS Provincial Advisory Committee, a group of subject matter experts, government, and academia who provide strategic support to the surveillance system.

The Perinatal Data Registry (Perinatal Services British Columbia, 2002), a database implemented in 2000, collects information from obstetrical and neonatal charts for virtually all births in the province, regardless of whether the delivery occurred at home or in the hospital, as well as information on infant hospitalizations in the first 28 days of life. The Vital Events Registry includes reported births (British Columbia Ministry of Health, 2002b), deaths (British Columbia Ministry of Health, 2002c), and stillbirths (British Columbia Ministry of Health, 2002d). Diagnostic codes are applied using the World Health Organization (WHO) International Classification of Diseases, 10th revision (ICD‐10). Access to the documentation used to generate diagnostic codes is possible through a collaboration with the Medical Coding Unit at the Vital Statistics Agency. The Discharge Abstract Database (Canadian Institute for Health Information, 2002) contains detailed inpatient information pertaining to hospitalizations in BC as well as from other Canadian jurisdictions involving BC residents, including day surgery. Information is submitted by the hospitals to the Canadian Institute for Health Information (CIHI), which in turn provides the validated data to the BC Ministry of Health (Emam, Paton, Dankar, & Koru, 2011). Lastly, the BC medical billing claims database (British Columbia Ministry of Health, 2002a) records all medically required services from general practitioners, specialists, laboratory services, and diagnostic procedures. Under the Medicare Protection Act, enrollment with the Medical Service Plan (MSP) is mandatory for all residents in BC. The diagnoses are coded using WHO ICD‐9 and up to five diagnostic codes can be included per record. These codes are considered to be accurate at the three‐digit level and shown to be valid at the population level (Hu, 1996).

Data are linked using a stepwise, deterministic linkage strategy that uses unique identifiers for linkage. Case definition algorithms are then applied to identify specific CA diagnosed within the first year of life. An algorithm comprises a list of criteria that need to be met for a case to be classify as having a CA. Criteria include having specific diagnostic or procedure codes, appearing in specific databases, or having a specified number of health encounters with the same diagnostic code. Exclusion criteria are also used to enhance the specificity of algorithms. For instance, the algorithm for anencephaly includes one record in the birth/stillbirth/death registry or one hospitalization record in the first month of life with the ICD codes Q00.0 and Q00.1 and excludes fetuses diagnosed with limb body wall spectrum or acalvaria (see Table 1).

Cases of neural tube defects and common chromosomal abnormalities are validated with external data sources. Cases coded as gastroschisis and omphalocele, and those coded as cleft lip and holoprosencephaly are flagged and further explore with the Provincial Advisory Committee prior to their inclusion in the BCCASS. Further information related to the BCCASS program planning, methodology and case validation has been previously published (de Escalante et al., 2022). The final BCCASS cohort is a dataset of unduplicated birth and stillbirth records with documented baby or mother personal health number. For this study, all BCCASS identified cases born to a mother residing in BC between April 1, 2000 and March 31, 2002 were included.

2.1.2. Data linkage

The BCCASS and the HSR contain records of livebirths and stillbirths diagnosed with CA, but they do not have information for those not diagnosed with CA. Therefore, linkage to the population of births is necessary to ascertain non‐cases of CA. Deterministic stepwise linkage was used to link the BCCASS, the HSR and the cohort of births (livebirths and stillbirths) registered in BC during the study period. The first step involved dividing records into those with infant personal health number and those without. The initial linkage was between the HSR and the birth cohort and it used infant PHN as the linkage key. This HSR‐birth cohort was then linked to the BCCASS using the same identifier. Records with no infant PHN were then linked using maternal PHN, date of birth and birth order. Records that had no birth registration number and were missing infant and maternal personal health number were excluded since linkage across databases is not possible without these identifiers (refer to Figure 1).

FIGURE 1.

FIGURE 1

Open in a new tab

Flow diagram for subject selection, British Columbia, Canada, April 2000–March 2002

2.2. Statistical analysis

Once linkage was completed, cases and non‐cases of CA were identified within each surveillance system. A case is defined as a fetus or infant identified with a specific anomaly while a non‐case is defined as a fetus or infant that is not identified with the specific CA for which the measure of accuracy is being calculated (either because no anomaly is recorded or because the record did not link to the BCCASS/HSR). However, non‐cases for a particular CA could be classified as a case of CA when another CA is being reviewed.

Characterization of the HSR and the BCCASS are provided in the Supporting Information. Descriptive statistics were calculated for both cohorts. Continuous variables were presented as mean and standard deviation (SD), median was reported if appropriate. Categorical variables were presented as count and percentages. Birth prevalence and 95% confidence interval (CI) were calculated for the HSR and the BCCASS (Begaud et al., 2005). Prevalence difference and relative difference was reported. (Appendix A and B—Supporting Information).

Two‐by‐two tables were created for all anomalies and the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were computed for each CA and CA group with their correspondent exact binomial 95% CI using the epiR package (Stevenson et al., 2020; see Table 1). Given the comprehensive case ascertainment of the HSR and the active follow up of cases, the HSR was used as the reference standard. Sensitivity was defined as the proportion of cases identified by the HSR that were identified by the BCCASS with the same defect. Specificity represents the proportion of non‐cases identified by the HSR that were identified as non‐cases in the BCCASS for the same defect. PPV was the proportion of cases in the BCCASS that were confirmed to have the same defect in the HSR. NPV was the proportion of non‐cases in the BCCASS that were confirmed as non‐cases by the HSR. Since a fetus or infant can have one or multiple CA, a record can appear multiple times in the two‐by‐two tables. However, within a CA group, cases are deduplicated. For example, an infant with two congenital heart defects would only be counted once.

Kappa coefficient has been previously used to assess agreement between two different case ascertainment methods (Cronk et al., 2003; Metcalfe et al., 2014; Miao, Fell, Dunn, & Sprague, 2019). We calculated the level of agreement between the BCCASS and the HSR for conditions with higher than five cases using the kappa coefficient. Kappa statistics measures the level of agreement between two observers (or databases) beyond what is expected by chance (Cohen, 1960). We applied the following criteria to assess the strength of the agreement: Kappa coefficient < 0: less than chance agreement; 0.01–0.20: slight agreement; 0.21–0.40: fair agreement; 0.41–0.60: moderate agreement; 0.61–0.80: substantial agreement; 0.81–0.99: almost perfect agreement (Viera & Garrett, 2005). We used this framework to calculate the level of agreement between the BCCASS and the HSR. However, we exclude conditions with fewer than six cases because the 95% CI on these kappa estimates is large and may span the entire range, which limits the interpretation.

All analyses were performed in R version 3.6.1 (R Core Team, 2019). Research ethics board approval was granted by the University of British Columbia (Certificate # H21‐01903).

3. RESULTS

Between April 1, 2000 and March 31, 2002, there were 80,970 records which included livebirths, termination of pregnancies and stillbirths over 19 weeks of gestational age with and without CA. Among the list of CA under evaluation, there were 1,315 and 1,562 cases of specific CA ascertained through the BCCASS and the HSR, respectively. Twelve records were excluded because they had no identifiers to link them across databases (see Figure 1).

The distribution of maternal age and birth weight for cases and non‐cases is relatively similar for both systems. The proportion of males is much higher for infant/fetuses with anomalies (for both the HSR and the BCCASS) than those without anomalies while the mean gestational age at delivery or termination is about 2 weeks earlier. An extreme outlier in maternal age was further explored and confirmed valid (Appendix A—Supporting Information). The proportion of stillbirths is higher in the BCCASS compared to the HSR (9.6% vs. 6.5%). Prevalence at birth in the BCCASS was relatively lower for most CA but the prevalence difference, assessed by the CI not crossing zero, was only significantly different for microcephaly and cryptorchidism (Appendix B—Supporting Information).

Measures of algorithm performance varied by CA and group of CA (Table 2). Sensitivity was lower than PPV for most conditions. Overall, internal CA, such as congenital heart defects, or defects not as easily identifiable at birth or with longer ascertainment periods (such as atresia of bile ducts, cystic kidney, and cryptorchidism) had the lowest sensitivity. For group defects, the sensitivity ranged from 67.3% for selected urinary tract defects to 91.8% for common chromosomal abnormalities. For specific defects, the sensitivity ranged between 42.2% for microcephaly to 100.0% for trisomy 18 and holoprosencephaly. Five out of the seven congenital heart defects assessed during the study had a sensitivity below 80%. The specificity and NPV were close to 100% for all groups and defects.

TABLE 2.

Diagnostic accuracy measures by congenital anomalies groups and specific anomalies

Condition Sensitivity (95% CI) Specificity (95% CI) PPV (95% CI) NPV (95% CI) Kappa statistic (95% CI)
Selected chromosomal defects 91.8 (86.6–95.5) 100.0 (100.0–100.0) 88.7 (83.1–93.0) 100.0 (100.0–100.0) 0.9 (0.9–0.9)
Trisomy 18 100.0 (86.8–100.0) 100.0 (100.0–100.0) 83.9 (66.3–94.5) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Down syndrome 90.4 (84.2–94.8) 100.0 (100.0–100.0) 89.1 (82.7–93.8) 100.0 (100.0–100.0) 0.9 (0.9–0.9)
Trisomy 13 88.9 (51.8–99.7) 100.0 (100.0–100.0) 80.0 (44.4–97.5) 100.0 (100.0–100.0) 0.8 (0.6–1.0)
Neural tube defects 87.2 (74.3–95.2) 100.0(100.0–100.0) 83.7 (70.3–92.7) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Encephalocele 80.0 (28.4–99.5) 100.0 (100.0–100.0) 80.0 (28.4–99.5) 100.0 (100.0–100.0) n/a
Anencephaly and similar anomalies 86.7 (59.5–98.3) 100.0 (100.0–100.0) 81.2 (54.4–96.0) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Spina bifida 88.9 (70.8–97.6) 100.0 (100.0–100.0) 80.0 (61.4–92.3) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Oro‐facial clefs 86.1 (79.7–91.1) 100.0 (100.0–100.0) 90.7 (84.8–94.8) 100.0 (100.0–100.0) 0.9 (0.9–0.9)
Cleft lip with cleft palate 91.8 (81.9–97.3) 100.0 (100.0–100.0) 80.0 (68.7–88.6) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Cleft lip 73.0 (55.9–86.2) 100.0 (100.0–100.0) 81.8 (64.5–93.0) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Cleft palate 56.2 (44.1–67.8) 100.0 (100.0–100.0) 87.2 (74.3–95.2) 100.0 (99.9–100.0) 0.7 (0.6–0.8)
Selected genital anomalies 83.3 (78.3–87.5) 100.0 (100.0–100.0) 94.1 (90.3–96.7) 99.9 (99.9–100.0) 0.9 (0.9–0.9)
Hypospadias and epispadias 83.6 (78.5–87.9) 100.0 (100.0–100.0) 94.3 (90.4–96.9) 99.9 (99.9–100.0) 0.9 (0.9–0.9)
Indeterminate sex 75.0 (47.6–92.7) 100.0 (100.0–100.0) 92.3 (64.0–99.8) 100.0 (100.0–100.0) 0.8 (0.6–1.0)
Selected abdominal wall defects 82.7 (69.7–91.8) 100.0 (100.0–100.0) 86.0 (73.3–94.2) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Gastroschisis 85.7 (69.7–95.2) 100.0 (100.0–100.0) 93.8 (79.2–99.2) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Omphalocele 73.3 (44.9–92.2) 100.0 (100.0–100.0) 100.0 (71.5–100.0) 100.0 (100.0–100.0) 0.8 (0.6–1.0)
Selected gastrointestinal defects 72.3 (63.8–79.8) 100.0 (100.0–100.0) 89.5 (82.0–94.7) 100.0 (99.9–100.0) 0.8 (0.7–0.9)
Small intestine absence/atresia/stenosis 85.3 (68.9–95.0) 100.0 (100.0–100.0) 96.7 (82.8–99.9) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Esophageal atresia/stenosis 85.0 (62.1–96.8) 100.0 (100.0–100.0) 81.0 (58.1–94.6) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Hirschsprung disease 68.2 (45.1–86.1) 100.0 (100.0–100.0) 93.8 (69.8–99.8) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Ano‐rectal absence/atresia/stenosis 66.7 (49.8–80.9) 100.0 (100.0–100.0) 89.7 (72.6–97.8) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Atresia of bile ducts 57.7 (36.9–76.6) 100.0 (100.0–100.0) 83.3 (58.6–96.4) 100.0 (100.0–100.0) 0.7 (0.5–0.9)
Selected congenital heart defects 71.4 (63.8–78.3) 100.0 (100.0–100.0) 88.5 (81.7–93.4) 99.9 (99.9–100.0) 0.8 (0.7–0.9)
Hypoplastic left heart syndrome 86.4 (65.1–97.1) 100.0 (100.0–100.0) 95.0 (75.1–99.9) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Tricuspid atresia 80.0 (28.4–99.5) 100.0 (100.0–100.0) 80.0 (28.4–99.5) 100.0 (100.0–100.0) n/a
Tetralogy of Fallot 73.8 (58.0–86.1) 100.0 (100.0–100.0) 86.1 (70.5–95.3) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Coarctation of aorta 66.0 (51.7–78.5) 100.0 (100.0–100.0) 87.5 (73.2–95.8) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Common truncus arteriosus 60.0 (14.7–94.7) 100.0 (100.0–100.0) 75.0 (19.4–99.4) 100.0 (100.0–100.0) n/a
Transposition of great vessels 58.8 (40.7–75.4) 100.0 (100.0–100.0) 100.0 (83.2–100.0) 100.0 (100.0–100.0) 0.7 (0.6–0.8)
Endocardial cushion defects/AVSD 53.6 (33.9–72.5) 100.0 (100.0–100.0) 88.2 (63.6–98.5) 100.0 (100.0–100.0) 0.7 (0.5–0.9)
Limb deficiency defects 71.4 (53.7–85.4) 100.0 (100.0–100.0) 92.6 (75.7–99.1) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Lower limb reduction 73.7 (48.8–90.9) 100.0 (100.0–100.0) 100.0 (76.8–100.0) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Upper limb reduction 61.9 (38.4–81.9) 100.0 (100.0–100.0) 86.7 (59.5–98.3) 100.0 (100.0–100.0) 0.7 (0.5–0.9)
Selected sense organ defects 75.0 (56.6–88.5) 100.0 (100.0–100.0) 96.0 (79.6–99.9) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Choanal atresia 78.3 (56.3–92.5) 100.0 (100.0–100.0) 100.0 (81.5–100.0) 100.0 (100.0–100.0) 0.9 (0.8–1.0)
Anophthalmos /microphthalmos 66.7 (34.9–90.1) 100.0 (100.0–100.0) 88.9 (51.8–99.7) 100.0 (100.0–100.0) 0.8 (0.6–1.0)
Selected urinary tract defects 67.3 (57.4–76.2) 100.0 (100.0–100.0) 83.3 (73.6–90.6) 100.0 (99.9–100.0) 0.7 (0.6–0.8)
Bladder and cloacal exstrophy 83.3 (35.9–99.6) 100.0 (100.0–100.0) 100.0 (47.8–100.0) 100.0 (100.0–100.0) n/a
Renal agenesis 82.9 (66.4–93.4) 100.0 (100.0–100.0) 82.9 (66.4–93.4) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Cystic kidney 53.8 (39.5–67.8) 100.0 (100.0–100.0) 73.7 (56.9–86.6) 100.0 (100.0–100.0) 0.6 (0.5–0.7)
Lower urinary tract obstruction 52.9 (27.8–77.0) 100.0 (100.0–100.0) 100.0 (66.4–100.0) 100.0 (100.0–100.0) 0.7 (0.5–0.9)
Other conditions
Arhinencephaly /holoprosencephaly 100.0 (47.8–100.0) 100.0 (100.0–100.0) 100.0 (47.8–100.0) 100.0 (100.0–100.0) n/a
Hydrocephaly 80.0 (63.1–91.6) 100.0 (100.0–100.0) 77.8 (60.8–89.9) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Congenital diaphragmatic hernia 78.3 (56.3–92.5) 100.0 (100.0–100.0) 81.8 (59.7–94.8) 100.0 (100.0–100.0) 0.8 (0.7–0.9)
Cataract 66.7 (49.0–81.4) 100.0 (100.0–100.0) 85.7 (67.3–96.0) 100.0 (100.0–100.0) 0.7 (0.6–0.8)
Cryptorchidism 63.6 (58.7–68.2) 100.0 (100.0–100.0) 97.4 (94.7–98.9) 99.8 (99.8–99.8) 0.8 (0.8–0.8)
Anomalies of the corpus callosum 61.1 (35.7–82.7) 100.0 (100.0–100.0) 100.0 (71.5–100.0) 100.0 (100.0–100.0) 0.8 (0.6–1.0)
Microcephaly 42.2 (29.9–55.2) 100.0 (100.0–100.0) 87.1 (70.2–96.4) 100.0 (99.9–100.0) 0.6 (0.5–0.7)
Open in a new tab

Note: Group conditions are bolded and have alternative shadings. Composition of group conditions are: Selected chromosomal defects—trisomy 13, trisomy 18, down syndrome; Neural tube defects—anencephaly, encephalocele, spina bifida; Oro‐facial clefts‐cleft lip only, cleft palate only, cleft lip and cleft palate; Selected genital anomalies—hypospadias, epispadias, indeterminate sex; Hypospadias—hypospadias/epispadias; Selected abdominal wall defects—omphalocele, gastroschisis, prune belly; Selected gastrointestinal defects—esophageal atresia/stenosis, small intestine absence/atresia/stenosis, Ano‐rectal absence/atresia/stenosis, atresia of bile ducts, Hirschsprung disease; Selected congenital heart defects—common truncus, transposition of great vessels, endocardial cushion defects/AVSD, tetralogy of fallot, tricuspid atresia, hypoplastic left heart syndrome, coarctation of aorta; Limb deficiency defects—upper limb reduction, lower limb reduction; selected sense organ—choanal atresia, anophthalmos/microphthalmos; Selected urinary tract defects—renal agenesia, cystic kidney, bladder and cloacal exstrophy, lower urinary tract obstruction.

Abbreviations: BCCASS, British Columbia Congenital Anomalies Surveillance System. CI, confidence interval; HSR, Health Status Registry. PPV, positive predictive value. NPV, negative predictive value.

All group defects had a PPV that ranged between 83.3% for selected urinary tract defects to 96.0% for selected sense organ defects. For specific CA, three conditions had a PPV under 80.0% (cystic kidney, common truncus arteriosus, and hydrocephaly) and multiple conditions had a PPV over 95.0% including omphalocele, small intestine atresia/stenosis, transposition of great arteries, choanal atresia, bladder exstrophy, lower urinary tract obstruction, holoprosencephaly, anomalies of the corpus callosum, and lower limb reductions.

Kappa coefficient ranged from 0.6 for microcephaly to 0.9 for trisomy 18, down syndrome, and small intestine atresia. There was almost perfect agreement (between 0.8 to 1.0) for all chromosomal defects, neural tube defects, genital anomalies, abdominal wall defects, small intestinal atresia, hypoplastic left heart syndrome, lower limb reduction, choanal atresia, cleft lip with cleft palate, and renal agenesia. There was fair agreement (between 0.4 and 0.6) for microcephaly and cystic kidney. All remaining CA had moderate agreement.

4. DISCUSSION

This study assessed the performance of algorithms used in the BCCASS to ascertain CA. Although the validity of each case definition algorithm varied by defect, the BCCASS algorithms were able to achieve a PPV greater than 80% for many defects with substantial agreement between both surveillance system for most conditions that are easily identifiable at birth.

The proportion of stillbirths in the BCCASS was higher than in the HSR. The stillbirths in the BCCASS but not in the HSR may have come from the Perinatal Data Registry, which was not a reporting source for the HSR. The higher proportion of males in both the HSR and BCCASS is more likely related to the type of CA that were included in the study (e.g., hypospadias and cryptorchidism only occur in males). Therefore, it would be expected to see an overrepresentation of males among the study population.

The prevalence of microcephaly and cryptorchidism was much higher in the HSR. Prevalence of cryptorchidism in the HSR was 1.5 times that of the BCCASS. This might be related to differing case definitions. For instance, the BCCASS does not include a diagnosis of cryptorchidism in premature neonates or in those with a birth weight under 2,500 g, even if a case has an ICD code for cryptorchidism. However, there were about 65 records in the HSR coded as cryptorchidism that had a birth weight under 2,500 g. Another reason might be that the HSR had multiple reporting sources, which would have resulted in higher ascertainment for conditions that may not be diagnosed in the first year. The BCCASS has several data sources that capture CA diagnoses in the first month of life but data capture after that period is limited. Therefore, there is higher ascertainment for conditions that are more likely to be diagnosed at birth. The sensitivity was lower for limb reductions defects and internal CA such as cystic kidney and atresia of the bile ducts. The BCCASS team hypothesized that algorithms with lower sensitivity relate to defects that are more likely to be missed at birth or those that require a longer ascertainment period, for example, microphthalmos, congenital cataract, anorectal stenosis, cleft palate, atresia of bile ducts, Hirschsprung disease, lower urinary tract stenosis, and cystic kidney (Haargaard, Nyström, Rosensvärd, Tornqvist, & Magnusson, 2015; Hamrick et al., 2012; Lakshminarayanan & Davenport, 2016). The assessment and potential inclusion of new data sources is suggested as a future strategy to enhance case identification for conditions with low sensitivity. Differences in the prevalence of microcephaly remain mostly unexplained and this requires further exploration.

The BCCASS includes termination of pregnancies and stillbirths over 19 weeks of gestational age. As a result, there is some capacity to capture conditions in pregnancies that are terminated (early terminations are missed), such as chromosomal abnormalities and neural tube defects. For these two defects, the BCCASS was able to capture most cases identified by the reference standard (91.8% of cases for chromosomal abnormalities and 87.0% for neural tube defects). Therefore, the inclusion of stillbirths and termination of pregnancies are a vital source of ascertainment that resulted in the relatively high sensitivity for neural tube defects. Exclusion of these populations will result in a significant decrease in the sensitivity for these defects. For instance, the Florida Birth Defect Registry does not include stillbirths as a source of ascertainment and only achieved a sensitivity of 45.5% for anencephaly in a validation study against an enhanced surveillance system (Jason L. Salemi et al., 2012).

The results of this study are consistent with other similar analyses. For instance, PPV was relatively similar to the original case definition reported by J. Salemi et al (Salemi et al., 2018) for multiple defects, including anencephaly, spina bifida, cleft lip with cleft palate, gastroschisis, hypospadias, transposition of great arteries, hypoplastic left heart syndrome, and trisomy 21. Another study in Quebec assessed the validity of diagnostic codes for CA in two administrative databases when comparing asthmatic and non‐asthmatic women. Authors reported that diagnostic codes in the databases are valid, with a relatively high PPVs for all CA explored, except for other anomalies of the nervous system (PPV 75.0%, 95% CI 70.8%–79.2%; Blais, Bérard, Kettani, & Forget, 2013). Caution is warranted when interpreting measures of accuracy for CA with low birth prevalence. For these defects, the CI for sensitivity, PPV, and kappa can be large, which limits its interpretation.

The results, however, are not consistent with some other reports. For CA that are visible at birth such as anencephaly or limb reduction, Metcalfe et al. (2014) assessed the validity of diagnostic codes in three databases and reported lower measures of accuracy overall and for specific defects. For instance, sensitivity for neurological defects was 62.5% (95% CI 35.4%–84.4%), with a PPV 47.6% (95% CI, 25.7%–70.2%) and kappa 0.5 (95% CI, 0.35–0.7). The BCCASS did not report on overall neurological defects but for neural tube defects and holoprosencephaly specifically which are included within neurological defects, the BCCASS achieved higher sensitivity, PPV, and kappa. Metcalfe et al. also reported low agreement for respiratory, face/neck and digestive defects (Metcalfe et al., 2014). The BCCASS did not encounter the same results, with higher PPVs reported in our study. These differences could be due to that the CA groups used by Metcalfe et al were not comparable to ours. For instance, the group of digestive defects used by the BCCASS only includes five defects (anorectal or small intestine absence/atresia or stenosis, esophageal atresia, atresia of bile ducts and Hirsprung disease) while Metcalfe et al includes all defects coded with the ICD 10 codes Q38–Q45 and the ICD‐9 codes 750–751. Due to the range of conditions that are included in a group, it is possible that agreement varies significantly for conditions within a group. Therefore, assessing agreement at the group level might yield lower results than when specific defects are assessed. Other reasons that might contribute to these differences include the difference in case definition and the nature of our system. The BCCASS algorithms are more stringent than Metcalfe et al., which could help decrease the risk of misclassification resulting in a higher PPV. Further, due to the hybrid nature of the BCCASS, there is a limited capacity to validate certain defects prior to inclusion in the system. There is also a capacity to confirm prenatal diagnoses by reviewing autopsy reports. This case confirmation decreases the risk of misclassification, therefore enhancing the quality of the surveillance system.

PPV and NPV are dependent on prevalence. Given the low birth prevalence of CA and the large number of births during the time period, the number of true negatives is large. True negatives are used to calculate both specificity and NPV, which were both high for this study, most likely due to low number of cases among births. While Benchimol et al. (2011) has emphasized the importance for validation studies to report on all measures of algorithm performance, in the context of conditions with very low prevalence, high true negatives, and a large denominator, there is no added benefit in reporting specificity and NPV.

This study has several strengths. First, the study covered the entire population of births in the province during the study period rather than using a sample of the population. The study has also provided a detailed description of the methodology, description of the cohort and performance metrics including NPV and specificity (which require the identification of true negatives). This will be essential for other programs planning to implement these algorithms. The BCCASS is mainly based on passive case ascertainment. Therefore, engagement with subject matter expert members of the Advisory Committee to validate cases of chromosomal abnormalities and neural tube defects is essential for data quality enhancement as it decreases the risk of misclassification bias, enhances data quality, and ensures the validity of diagnostic codes for these two conditions. Further, the modified STARD criteria for validation studies reported by Benchimol et al. (2011) were followed to enhance the quality of the study.

This study is subject to several limitations. The most significant limitation is that not all HSR cases were confirmed according to medical charts. Although the HSR was considered to be an excellent source of case ascertainment, it was still a hybrid surveillance system. It was assumed that there was a low risk of misclassification due to the active follow up of cases included in the HSR; however, it is still possible that cases were missed. Further, the BCCASS was unable to use more recent case data from the HSR due to the decrease in data quality after 2003. As a result, this validation study was limited to a small temporal window of data collection between 2000 and 2002, given that the Perinatal Data Registry, the primary data source of the BCCASS, was only implemented on April 1, 2000. It is possible that case capture during the initial years of the PDR was limited, and this could negatively impact the study results. Due to the enhancement of data quality in administrative databases in the last decade, the results should be transferable to recent data. Another limitation is that the algorithms used for BCCASS might not be applicable in other jurisdictions as they are dependent on local databases and jurisdiction‐specific ICD coding practices. As previously suggested (Metcalfe et al., 2014; J. Salemi et al., 2018), programs should validate algorithms locally before implementing them for surveillance. Also, although the BCCASS and the HSR had different ascertainment methods, with the latter having more than 50 reporting sources, some reporting sources in the BCCASS were also shared by the HSR. Therefore, it is possible for the kappa metric to be slightly overestimated due to both surveillance systems sharing some reporting sources. Lastly, the BCCASS does not validate all conditions due to personnel and time constraints, nor does it ascertain CA in termination of pregnancies under 20 weeks.

Case definition algorithms are valuable because they use existing information, usually available at the population level, to support population‐level surveillance. Because algorithms are never 100% accurate, validation studies are a necessity to support decisions around algorithm selection and need for enhancement. As Chubak et al. (2012) suggested, lack of adequate detail in reports of validation studies make it challenging for programs to use algorithms developed in other settings. This article provides detailed methodology and performance metrics, so other programs can determine whether algorithms reported in this manuscript can be applied in their own settings.

In conclusion, case definition algorithms can be used in passive or hybrid systems to effectively provide estimates of case counts for select CA, particularly those that are easily identified at birth. Validated algorithms may be used in areas with similar data availability to facilitate comparison across jurisdictions. In circumstances of funding and other resource constraints, a combination of passive surveillance with limited (but targeted) case validation can be used to ensure the validity of diagnostic codes, decrease the risk of misclassification bias, and ultimately enhance data quality. This validation study also provided insight into which CA are most likely to be missed (or misclassified), thus assisting in clarifying the limitations of the current system and providing valuable guidance for fine‐tuning future surveillance activities.

CONFLICT OF INTEREST

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Supporting information

Appendix S1: Supporting Information

Click here for additional data file. (73.3KB, docx)

ACKNOWLEDGMENT

We would like to acknowledge all interested parties for their invaluable support. We appreciate the support and guidance from the Public Health Agency of Canada, the Provincial Health Services Authority, especially the Information Access and Privacy branch, Perinatal Services BC, the British Columbia Prenatal Genetic Screening program, the BC Vital Statistics Agency, in particular Bruce Klette for his support in facilitating the HSR data transfer and the Medical Coding Unit for their ongoing support in validating congenital anomaly cases. We would also like to acknowledge the Health Sector Information, Analysis and Reporting division at the BC Ministry of Health, in particular the Data Management and Stewardship team for their support in developing the data agreements required to access the databases necessary for BCCASS development and implementation.

Nava de Escalante, Y. , Abayomi, A. , Langlois, S. , Ye, X. , Erickson, A. , Ngo, H. , Armour, R. , Okamoto, R. , Arbour, L. , Bedard, T. , Der, K. , Van Allen, M. , Skarsgard, E. , Lavoie, M. , & Henry, B. (2023). Validation of case definition algorithms for the ascertainment of congenital anomalies. Birth Defects Research, 115(3), 302–317. 10.1002/bdr2.2112

Rosemary Armour is retired from British Columbia Vital Statistics Agency.

[Correction added on 15 November 2022, after first online publication: The affiliations 3 and 4 have been swapped in the affiliation list in this version.]

DATA AVAILABILITY STATEMENT

Access to data provided by the Data Steward(s) is subject to approval but can be requested for research projects through the Data Steward(s) or their designated service providers. All inferences, opinions, and conclusions drawn in this publication are those of the author(s), and do not reflect the opinions or policies of the Data Steward(s).

REFERENCES

  1. Andrade, S. E. , Bérard, A. , Nordeng, H. M. E. , Wood, M. E. , Van Gelder, M. M. H. J. , & Toh, S. (2017). Administrative claims data versus augmented pregnancy data for the study of pharmaceutical treatments in pregnancy. Current Epidemiology Reports, 4(2), 106–116. 10.1007/s40471-017-0104-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Antoniou, T. , Zagorski, B. , Loutfy, M. R. , Strike, C. , & Glazier, R. H. (2011). Validation of case‐finding algorithms derived from administrative data for identifying adults living with human immunodeficiency virus infection. PLoS One, 6(6), e21748. 10.1371/journal.pone.0021748 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Begaud, B. , Martin, K. , Abouelfath, A. , Tubert‐Bitter, P. , Moore, N. , & Moride, Y. (2005). An easy to use method to approximate Poisson confidence limits. European Journal of Epidemiology, 20(3), 213–216. 10.1007/s10654-004-6517-4 [DOI] [PubMed] [Google Scholar]
  4. Benchimol, E. I. , Manuel, D. G. , To, T. , Griffiths, A. M. , Rabeneck, L. , & Guttmann, A. (2011). Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. Journal of Clinical Epidemiology, 64(8), 821–829. 10.1016/j.jclinepi.2010.10.006 [DOI] [PubMed] [Google Scholar]
  5. Blais, L. , Bérard, A. , Kettani, F. Z. , & Forget, A. (2013). Validity of congenital malformation diagnostic codes recorded in Québec's administrative databases. Pharmacoepidemiology and Drug Safety, 22(8), 881–889. 10.1002/pds.3446 [DOI] [PubMed] [Google Scholar]
  6. British Columbia Ministry of Health . (2002a). Consolidation File (MSP Registration & Premium Billing). (2002 ed.): British Columbia Ministry of Health.
  7. British Columbia Ministry of Health . (2002b). Vital events births. In British Columbia Ministry of Health [producer] (Ed.). British Columbia.
  8. British Columbia Ministry of Health . (2002c). Vital events deaths. (2002 ed.): British Columbia Ministry of Health.
  9. British Columbia Ministry of Health . (2002d). Vital events stillbirths. (2002 ed.): British Columbia Ministry of Health.
  10. Cadarette, S. M. , & Wong, L. (2015). An introduction to health care administrative data. The Canadian Journal of Hospital Pharmacy, 68(3), 232–237. 10.4212/cjhp.v68i3.1457 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Canadian Institute for Health Information (2002). Discharge abstract database (hospital separations). In British Columbia Ministry of Health (Ed.), (2002 ed.): British Columbia Ministry of Health.
  12. Christianson, A. , Howson, C. , & Modell, B. (2006). Global report on birth defects. March of Dimes Birth Defects Foundation.
  13. Chubak, J. , Pocobelli, G. , & Weiss, N. S. (2012). Tradeoffs between accuracy measures for electronic health care data algorithms. Journal of Clinical Epidemiology, 65(3), 343–349.e342. 10.1016/j.jclinepi.2011.09.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. 10.1177/001316446002000104 [DOI] [Google Scholar]
  15. Cronk, C. E. , Malloy, M. E. , Pelech, A. N. , Miller, R. E. , Meyer, S. A. , Cowell, M. , & McCarver, D. G. (2003). Completeness of state administrative databases for surveillance of congenital heart disease. Birth Defects Research. Part A, Clinical and Molecular Teratology, 67(9), 597–603. 10.1002/bdra.10107 [DOI] [PubMed] [Google Scholar]
  16. de Escalante, Y. N. , Abayomi, A. , Erickson, A. , Ye, X. , Armour, R. , Arbour, L. , … Henry, B. (2022). Implementation of the BC congenital anomalies surveillance system (BCCASS). Canadian Journal of Public Health, 113, 465–473. 10.17269/s41997-021-00607-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Eltonsy, S. , Forget, A. , & Blais, L. (2017). The impact of different case ascertainment definitions on the prevalence of major congenital malformations and their association with asthma during pregnancy. Maternal and Child Health Journal, 21(3), 616–625. 10.1007/s10995-016-2147-1 [DOI] [PubMed] [Google Scholar]
  18. Ely, D. M. , & Driscoll, A. K. (2020). Infant mortality in the United States, 2018: Data from the period linked birth/infant death File. National Vital Statistics Reports, 69(7), 1–18. [PubMed] [Google Scholar]
  19. Emam, K. E. , Paton, D. , Dankar, F. , & Koru, G. (2011). De‐identifying a public use microdata file from the Canadian national discharge abstract database. BMC Medical Informatics and Decision Making, 11(1), 53. 10.1186/1472-6947-11-53 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Farr, S. L. , Riley, C. , Van Zutphen, A. R. , Brei, T. J. , Leedom, V. O. , Kirby, R. S. , & Pabst, L. J. (2021). Prevention and awareness of birth defects across the lifespan using examples from congenital heart defects and spina bifida. Birth Defects Research, 114, 35–44. 10.1002/bdr2.1972 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Fox, M. P. , Lash, T. L. , & Bodnar, L. M. (2020). Common misconceptions about validation studies. International Journal of Epidemiology, 49(4), 1392–1396. 10.1093/ije/dyaa090 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Frosst, G. , Hutcheon, J. , Joseph, K. , Kinniburgh, B. , Johnson, C. , & Lee, L. (2015). Validating the British Columbia Perinatal Data Registry: A chart re‐abstraction study. BMC Pregnancy and Childbirth, 15(1), 123. 10.1186/s12884-015-0563-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Glinianaia, S. V. , Tennant, P. W. G. , & Rankin, J. (2017). Risk estimates of recurrent congenital anomalies in the UK: A population‐based register study. BMC Medicine, 15(1), 20. 10.1186/s12916-017-0789-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Haargaard, B. , Nyström, A. , Rosensvärd, A. , Tornqvist, K. , & Magnusson, G. (2015). The Pediatric Cataract Register (PECARE): Analysis of age at detection of congenital cataract. Acta Ophthalmologica, 93(1), 24–26. 10.1111/aos.12445 [DOI] [PubMed] [Google Scholar]
  25. Hamrick, M. , Eradi, B. , Bischoff, A. , Louden, E. , Peña, A. , & Levitt, M. (2012). Rectal atresia and stenosis: Unique anorectal malformations. Journal of Pediatric Surgery, 47(6), 1280–1284. 10.1016/j.jpedsurg.2012.03.036 [DOI] [PubMed] [Google Scholar]
  26. Hu, W. (1996). Diagnostic codes in MSP claim data (program monitoring and Information management branch resource management division medical services plan, trans.). BC Ministry of Health.
  27. Johnson, E. K. , & Nelson, C. P. (2013). Values and pitfalls of the use of administrative databases for outcomes assessment. Journal of Urology, 190(1), 17–18. 10.1016/j.juro.2013.04.048 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kharbanda, E. O. , Vazquez‐Benitez, G. , Romitti, P. A. , Naleway, A. L. , Cheetham, T. C. , Lipkind, H. S. , … Nordin, J. D. (2017). Identifying birth defects in automated data sources in the vaccine safety datalink. Pharmacoepidemiology and Drug Safety, 26(4), 412–420. 10.1002/pds.4153 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Lakshminarayanan, B. , & Davenport, M. (2016). Biliary atresia: A comprehensive review. Journal of Autoimmunity, 73, 1–9. 10.1016/j.jaut.2016.06.005 [DOI] [PubMed] [Google Scholar]
  30. Langlois, P. H. , Sheu, S. U. , & Scheuerle, A. E. (2010). A physician survey regarding diagnostic variability among birth defects. American Journal of Medical Genetics Part A, 152, 1594‐1598. 10.1002/ajmg.a.33413 [DOI] [PubMed] [Google Scholar]
  31. Lix, L. , Ayles, J. , Bartholomew, S. , Cooke, C. , Ellison, J. , Emond, V. , … Pelletier, L. (2018). The Canadian Chronic Disease Surveillance System: A model for collaborative surveillance. International Journal of Population Data Science, 3(3), 1–11. 10.23889/ijpds.v3i3.433 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Lowry, B. , & Bedard, T. (2013). Birth defect registries: The vagaries of management—The British Columbia and Alberta case histories. Journal of Registry Management, 40(2), 98–103. [PubMed] [Google Scholar]
  33. Lowry, R. , Miller, J. , Scott, A. , & Renwick, D. (1975). The British Columbia registry for handicapped children and adults: Evolutionary changes over twenty years. Canadian Journal of Public Health/Revue Canadienne de Sante'e Publique, 66(4), 322–326. [PubMed] [Google Scholar]
  34. Luquetti, D. V. , & Koifman, R. J. (2011). Surveillance of birth defects: Brazil and the US. Ciência & Saúde Coletiva, 16(Suppl 1), 777–785. 10.1590/s1413-81232011000700008 [DOI] [PubMed] [Google Scholar]
  35. Mburia‐Mwalili, A. , & Yang, W. (2014). Birth defects surveillance in the United States: Challenges and implications of international classification of diseases, tenth revision, clinical modification implementation. International Scholarly Research Notices, 2014, 1–9. 10.1155/2014/212874 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Metcalfe, A. , Sibbald, B. , Lowry, R. , Tough, S. , & Bernier, F. (2014). Validation of congenital anomaly coding in Canada's administrative databases compared with a congenital anomaly registry. Birth Defects Research. Part A, Clinical and Molecular Teratology, 100(2), 59–66. 10.1002/bdra.23206 [DOI] [PubMed] [Google Scholar]
  37. Miao, Q. , Fell, D. B. , Dunn, S. , & Sprague, A. E. (2019). Agreement assessment of key maternal and newborn data elements between birth registry and Clinical Administrative Hospital Databases in Ontario, Canada. Archives of Gynecology and Obstetrics, 300(1), 135–143. 10.1007/s00404-019-05177-x [DOI] [PubMed] [Google Scholar]
  38. Miller, E. (2006). Evaluation of the Texas Birth Defects Registry: An active surveillance system. Birth Defects Research Part A: Clinical and Molecular Teratology, 76(11), 787–792. 10.1002/bdra.20331 [DOI] [PubMed] [Google Scholar]
  39. Mott, G. A. (1963). The registry of handieapped children and adults in British Columbia. Canadian Journal of Public Health, 54(6), 239–245. [PubMed] [Google Scholar]
  40. Nasr, A. , Sullivan, K. J. , Chan, E. , Wong, C. A. , & Benchimol, E. I. (2017). Validation of algorithms to determine incidence of Hirschsprung disease in Ontario, Canada: A population‐based study using health administrative data. Clinical Epidemiology, 9, 579–590. 10.2147/clep.s148890 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Perinatal Services British Columbia (2002). British Columbia Perinatal Data Registry. British Columbia Ministry of Health.
  42. Public Health Agency of Canada (2013). Congenital anomalies in Canada: A perinatal health surveillance report. Public Health Agency of Canada.
  43. R Core Team . (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. [Google Scholar]
  44. Reichard, A. , Mcdermott, S. , Ruttenber, M. , Mann, J. , Smith, M. G. , Royer, J. , & Valdez, R. (2016). Testing the feasibility of a passive and active case ascertainment system for multiple rare conditions simultaneously: The experience in three US states. JMIR Public Health and Surveillance, 2(2), e151. 10.2196/publichealth.5516 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Riley, M. , Phyland, S. , & Halliday, J. (2004). Validation study of the Victorian birth defects register. Journal of Paediatrics and Child Health, 40(9–10), 544–548. 10.1111/j.1440-1754.2004.00460.x [DOI] [PubMed] [Google Scholar]
  46. Salemi, J. , Rutkowski, R. , Tanner, J. P. , Matas, J. , & Kirby, R. (2018). Identifying algorithms to improve the accuracy of unverified diagnosis codes for birth defects. Public Health Reports, 133(3), 303–310. 10.1177/0033354918763168 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Salemi, J. L. , Tanner, J. , Sampat, D. , Anjohrin, S. , Correia, J. , Watkins, S. , & Kirby, R. (2016). The accuracy of hospital discharge diagnosis codes for major birth defects: Evaluation of a statewide registry with passive case ascertainment. Journal of Public Health Management and Practice, 22(3), E9–E19. 10.1097/PHH.0000000000000291 [DOI] [PubMed] [Google Scholar]
  48. Salemi, J. L. , Tanner, J. P. , Kennedy, S. , Block, S. , Bailey, M. , Correia, J. A. , … Kirby, R. S. (2012). A comparison of two surveillance strategies for selected birth defects in Florida. Public Health Reports, 127(4), 391–400. 10.1177/003335491212700407 [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Shiff, N. J. , Oen, K. , Rabbani, R. , & Lix, L. M. (2017). Validation of administrative case ascertainment algorithms for chronic childhood arthritis in Manitoba, Canada. Rheumatology International, 37(9), 1575–1584. 10.1007/s00296-017-3734-1 [DOI] [PubMed] [Google Scholar]
  50. Stevenson, M. , Nunes, T. , Heuer, C. , Marshall, J. , Sanchez, J. , Thornton, R. , … Rabiee, A. (2020). epiR: Tools for the analysis of epidemiological data .
  51. Viera, A. J. , & Garrett, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37(5), 360–363. [PubMed] [Google Scholar]
  52. World Health Organization . (2020). Birth defects surveillance: A manual for programme managers (2nd ed.). Geneva: WHO. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix S1: Supporting Information

Click here for additional data file. (73.3KB, docx)

Data Availability Statement

Access to data provided by the Data Steward(s) is subject to approval but can be requested for research projects through the Data Steward(s) or their designated service providers. All inferences, opinions, and conclusions drawn in this publication are those of the author(s), and do not reflect the opinions or policies of the Data Steward(s).

Articles from Birth Defects Research are provided here courtesy of Wiley

RESOURCES