Mixed-Methods Evaluation of Massachusetts’ Neonatal Abstinence Syndrome Surveillance System, 2020–2022

Highlights

• •Massachusetts’ neonatal abstinence syndrome surveillance produced high-quality data.
• •Case ascertainment using 6 diagnostic codes was estimated to be mostly complete.
• •Code P96.1 was the most accurate for identifying neonatal abstinence syndrome.

Abstract

Introduction

The Massachusetts Department of Public Health established neonatal abstinence syndrome surveillance as an active, state-wide, population-based system using the Council of State and Territorial Epidemiologists (Tier 1) neonatal abstinence syndrome standardized surveillance case definition. This evaluation assessed the Massachusetts neonatal abstinence syndrome surveillance system and estimated the completeness and positive predictive value of case ascertainment methods.

Methods

Neonatal abstinence syndrome surveillance system was evaluated through a system user survey (n=12) and neonatal abstinence syndrome surveillance system data reports to identify strengths and limitations. Completeness of case ascertainment using 6 neonatal abstinence syndrome–related ICD-10-CM codes was estimated using capture–recapture methodology for births between April 1, 2020, and September 30, 2020. The positive predictive value of the codes was estimated using data for births between April 1, 2020, and December 31, 2022 (n=2,455).

Results

Neonatal abstinence syndrome surveillance system was found to be simple to use, flexible, and representative, with opportunities to improve timeliness. Case ascertainment was estimated to be 85.9% complete. ICD-10-CM code P96.1 had the highest positive predictive value for confirmed cases (69.9%) and for all case types (confirmed, probable, and suspect) combined (95.1%). P04.40 had the lowest positive predictive value for confirmed cases, whereas P04.1A had the lowest positive predictive value for all case types (43.5% for both codes).

Conclusions

ICD-10-CM codes with higher positive predictive values can be utilized for neonatal abstinence syndrome case ascertainment, especially in jurisdictions with limited resources for case confirmation. Other sources of case ascertainment can be added to improve the completeness of neonatal abstinence syndrome surveillance. Findings from this evaluation can guide the evaluation and case ascertainment methods for jurisdictions already conducting or interested in establishing neonatal abstinence syndrome surveillance using the Council of State and Territorial Epidemiologists neonatal abstinence syndrome standardized surveillance case definition.

Graphical Abstract

INTRODUCTION

Neonatal abstinence syndrome (NAS) results from withdrawal after in utero exposure to substances such as opioids, benzodiazepines, or barbiturates.¹ NAS impacts the infant’s central and autonomic nervous systems, gastrointestinal tract, and respiratory system, and manifests as a constellation of signs, including hyperirritability, tremors, and hypertonia.¹ In Massachusetts, NAS incidence increased by 37% from 10.2 infants per 1,000 birth hospitalizations in 2010 to 13.9 in 2017, nearly twice the national incidence in that same year (7.3 per 1,000 birth hospitalizations).²

Major challenges to NAS surveillance exist. First, NAS is not a nationally notifiable condition nor a reportable condition in most states. Moreover, until recently, neither a standardized surveillance case definition nor a clinical case definition existed.^3,4 Standardized surveillance can improve the accurate identification of infants affected by NAS and, in turn, enable the quantification of NAS incidence, characterization of inequities, and the appropriate allocation of resources.³ In 2019, the Council of State and Territorial Epidemiologists (CSTE) ratified a standardized surveillance case definition for NAS and funded 4 jurisdictions to pilot the implementation.³ This report outlines how NAS surveillance based on the CSTE (Tier 1) NAS standardized surveillance case definition, was established at the Massachusetts Department of Public Health (MDPH) as an active, statewide, population-based system. It also includes an evaluation of MDPH’s NAS surveillance system (NSS) and an estimation of the completeness of case ascertainment using 6 NAS-related ICD-10-CM codes. Finally, the positive predictive value (PPV) of the 6 NAS-related ICD-10-CM codes was calculated based on the CSTE (Tier 1) NAS standardized surveillance case definition. Notably, this is the first estimation of the PPV of these ICD-10-CM codes based on the CSTE (Tier 1) NAS standardized surveillance case definition.

METHODS

The NSS was established using the CSTE (Tier 1) NAS standardized surveillance case definition starting with April 1, 2020 births. Tier 1 surveillance confirms potential cases through medical record review. Potential cases are ascertained by the required reporting of 6 NAS-related ICD-10-CM codes from hospital reports that include patient demographics, clinical information, admission and discharge information, and diagnostic and procedural coding recorded through 28 days after delivery from all Massachusetts birth and pediatric hospitals. These codes include P96.1 (neonatal withdrawal symptoms from maternal use of drugs of addiction), P04.49 (newborn affected by maternal use of other drugs of addiction), P04.17 (newborn affected by maternal use of sedative-hypnotics), P04.1A (newborn affected by maternal use of anxiolytics), P04.14 (newborn affected by maternal use of opiates), and P04.40 (newborn affected by maternal use of unspecified drugs of addiction). All infants in a multiple pregnancy are also ascertained if 1 of the infants is ascertained through reports of the 6 NAS-related ICD-10-CM codes.

After case ascertainment, medical record abstractors access and review hospital medical records and toxicology reports for the birthing person and their infant to collect the following information: the birthing person’s medication or substance use history during pregnancy (including medication for addiction treatment); toxicology test results for the birthing person and their infant; infant symptomatology and treatment; and social factors, including housing stability, discharge disposition of the infant, and whether a family care plan was documented in the medical record. These data are recorded in a REDCap database created specifically for NAS surveillance. Each potential case is then clinically reviewed by NSS’s clinical reviewer and classified according to the CSTE (Tier 1) NAS standardized surveillance case definition into 1 of 8 categories: confirmed, probable (Types 1–2), and suspect (Types 1–5).³ Between April 1, 2020 and December 31, 2022, there were 2,455 potential NAS cases ascertained by the NSS.

System User Survey and Neonatal Abstinence Syndrome Surveillance System Data Reports

NSS was evaluated using the Centers for Disease Control and Prevention’s (CDC) Updated Guidelines for Evaluating Public Health Surveillance Systems.⁵ A system user survey, created using Microsoft Forms, was conducted to evaluate the NSS during its first year of implementation (April 1, 2020–March 31, 2021). The survey instrument included 4 general open-ended questions and 7 Likert scale–type questions for all participants and 2–4 role-specific open-ended and yes/no questions tailored to each participant’s roles within MDPH’s NSS (e.g., abstractor, epidemiologist, clinician) (Appendix Table 1, available online). The survey instrument was designed to assess the following CDC’s key attributes of a surveillance system: simplicity, flexibility, acceptability, and stability.⁵ MDPH also assessed operational constructs as part of the survey. Twelve MDPH NSS users, including medical record abstractors, epidemiologists, surveillance and project coordinators, principal investigators, and the clinical reviewer, were invited to complete the survey, with 1 user invited to complete the survey under 2 roles. These 12 participants were chosen because they comprised the full set of MDPH staff actively involved in implementing and operating the NSS, thereby representing a census of system users. Survey open-text responses were analyzed using an inductive and descriptive framework to identify themes of strengths and limitations for each key attribute that emerged across all responses. Qualitative analysis software and a structured coding framework were not used because the volume of data was relatively small, and the goal was to identify broad patterns. Three reviewers manually reviewed the text responses independently to identify common themes. Categorical responses (e.g., yes/no or strongly agree/disagree) were analyzed as the percentage of total responses for each question (Table 1). CDC’s key attributes, such as data quality, timeliness, representativeness, and usefulness, were assessed through reports generated from NSS based on the data collected and Massachusetts birth certificate data (Table 1).

Table 1.

Findings of an Evaluation of Massachusetts Neonatal Abstinence Syndrome Surveillance System by Surveillance System Attribute: April 1, 2020 – March 31, 2021

Surveillance system attribute	Definition	Findings
Simplicity	Structure and ease of operation	NSS is simple to use and well streamlined. On average, medical record abstractions take approximately 1 hour, with a range of 30–180 minutes depending on the complexity of the case. Clinical case review and classification range from 10 to 25 minutes, and quarterly data submissions take about 2 and a half workdays
Flexibility	Ability to adapt to changing information or operating conditions with little additional time, personnel, or funds	The legislative framework facilitated the integration of NAS surveillance. REDCap database allowed for new variables to be added or existing variables to be modified, as additional data elements became of interest
Data quality	Completeness and validity of data recorded in the NAS surveillance system	Medical record abstraction yielded more detailed and accurate information than administrative reporting, such as infant discharge disposition and family care plan administration. Missingness ranged from 0.0% to 47.1% for all variables. A total of 33.0% of variables had no missing data, 20.0% were missing <1.0% of data, 34.0% of variables were missing between 1.0% and 10.0% of data, 4.0% of variables were missing between 10.0% and 20.0% of data, and the remaining 9.0% of variables were missing between 35.0% and 47.1% of data. High missingness was concentrated in data on toxicology screens, which are not always performed and are entered into REDCap as missing unless explicitly stated that no toxicology screen was performed
Acceptability	Willingness of persons or organizations to participate in the NAS surveillance system	Most users (88.9%) agreed that NSS is easy to use, 66.7% reported having enough time to complete assigned tasks, 80.0% agreed or strongly agreed that NAS surveillance was prioritized by their peers and supervisors, and 88.9% had a sense of accomplishment when completing NAS-related work. Over half (55.6%) were satisfied with the establishment of NAS surveillance
Representativeness	Ability to accurately identify the incidence of NAS over time and its distribution by place and person	NSS is an active, state-wide, population-based surveillance system, with the intent of capturing the incidence of NAS in the entire Massachusetts population. All birth and pediatric hospitals report 6 NAS-related ICD-10-CM codes to NSS for a total of 51 reporting facilities. When compared with another population-based source of NAS data in Massachusetts (birth certificates), NSS had similar distributions of characteristics (e.g., race/ethnicity, health insurance type) for birthing people and their infants
Timeliness	Speed between steps of surveillance system	ICD-10-CM codes were submitted to MDPH an average of 63.6 days after delivery (median=55.0 days, range=9–538). The average time from case ascertainment to medical record abstraction was 144.5 days (median=148, range=1–476). Timeliness could be improved
Stability	Reliability and availability of the NAS surveillance system	No stability issues were reported outside of limited data loss and individual connectivity issues
Usefulness	Meets intended objectives, contributes to the prevention and control of NAS, and improves the understanding of public health implications of NAS	NSS allows for the assessment of NAS incidence over time. The surveillance system also captures trends in care and treatment such as receipt of medication for addiction treatment during pregnancy, family care plan completion, infant discharge status, receipt of breastmilk at delivery, and type of treatment initiated (pharmacological versus Eat Sleep Console)
Operational constructs	Technology improvements and allocations/resources	Users suggested streamlining the REDCap database to improve data entry by creating dropdown lists for certain variables (e.g., substances and medication options) and aligning the REDCap forms with the medical records. Users also suggested having abstraction and clinical case review occur on a continuous basis throughout the year and that abstraction assignments be more evenly distributed among the abstractors

MDPH, Massachusetts Department of Public Health; NAS, neonatal abstinence syndrome; NSS, neonatal abstinence syndrome surveillance system.

Methods for Estimating Completeness

Completeness of NSS case ascertainment was estimated in 2 steps. First, a population estimate of the potential number of infants born in Massachusetts expected to be ascertained by at least 1 of the 6 NAS-related ICD-10-CM codes from hospital reports was calculated using capture–recapture methodology.5, 6, 7 This approach relied on 2 data sources: hospital reports of the ICD-10-CM codes and Massachusetts birth certificates, where NAS can be recorded as a reported condition of the infant by the birth registrar.⁵ Hospital birth registrars are instructed, according to the specifications of the Massachusetts Registry of Vital Records and Statistics, to record NAS as a condition of the infant if the infant was diagnosed with NAS on the basis of the hospital’s standard screening policy for maternal drugs of abuse and newborn NAS screening. Potential NAS cases were deduplicated and linked using the Massachusetts birth certificate identification number to assess those ascertained by both sources. A review of infants’ first and last name and date of birth was conducted to ensure that the linkage was correct. In the second step, the number of potential NAS cases ascertained by NSS through hospital reports of the 6 ICD-10-CM codes was divided by the population estimate calculated in Step 1 to estimate the completeness of case ascertainment. A 95% CI was also calculated for the population estimate.⁸ Completeness was estimated during the early phase of establishing the NSS to evaluate case ascertainment methods for births between April 1, 2020 and September 30, 2020. Capture–recapture methodology assumes that each case has a separate but equal likelihood of being ascertained by both data sources and that data sources are independent.^7,9

Methods for Estimating Positive Predictive Value

The PPV of each ICD-10-CM code was estimated by dividing the number of cases classified as having NAS per the CSTE (Tier 1) NAS standardized surveillance case definition by the number of potential NAS cases ascertained by each ICD-10-CM code. PPV and 95% CIs were estimated for each case type individually and combined, using NAS surveillance data for births between April 1, 2020, and December 31, 2022.

This public health surveillance activity was conducted consistent with applicable state law. Massachusetts holds legal authority to conduct NAS surveillance through Chapter 111, Section 67E of Massachusetts General Law, which defines a birth defect as any “structural, functional or biochemical abnormality, regardless of cause.”¹⁰ The MDPH’s IRB determined the quantitative analyses to be exempt from human-subject review. All analyses were performed using SAS.

RESULTS

System User Survey and Neonatal Abstinence Syndrome Surveillance System Data Reports Results

All the 12 MDPH NSS users invited to take the survey completed it; 13 responses were collected because 1 user fulfilled 2 roles. Most (88.9%) reported that NSS was simple to use and well streamlined. Almost all users (80.0%) agreed or strongly agreed that peers and supervisors prioritized NSS, and 55.6% were satisfied with its establishment, whereas all others felt neutral. The REDCap database provided flexibility because variables could easily be added or modified. On average, medical record abstractions took 1 hour per case, and clinical review and classification took 15 minutes (Table 1). Medical record abstraction enhanced the quality of the data by enabling the collection of more detailed and accurate medical information than administrative reporting. NSS also proved useful, allowing for more accurate assessments of NAS incidence in Massachusetts and capturing trends in care and treatment, such as receipt of a family care plan and infant discharge disposition from the delivery hospitalization. Reports created from NSS data showed that the median time from case ascertainment to medical record review was 148 days (range=1–476 days). This resulted from having to complete multiple medical record abstractions for the same case, potentially across different facilities; collecting missing data; and/or clarifying information from the medical record. To improve satisfaction, timeliness, and operational constructs, respondents recommended that the workload be more evenly distributed across the medical record abstractors and that abstraction and clinical review and case classification occur continuously throughout the year. Respondents also mentioned simplifying the REDCap database for data entry from the medical records (e.g., creating dropdown lists).

NSS data reports showed that missing data ranged from 0.0% to 47.1% for all variables. Variables with high missing data relate to toxicology screens, which are not always performed and are entered into REDCap as missing unless otherwise stated. Users recommended adding an option to indicate that no toxicology screens were performed, so that it is not considered missing in REDCap. NSS was also deemed representative, as it is a state-wide, population-based surveillance system, and all users reported no significant challenges to stability or access.

Completeness Estimate

Among April 1, 2020 – September 30, 2020 births, 482 potential NAS cases were ascertained through 2 sources; 456 were reported through at least 1 of the NAS-related ICD-10-CM codes, 185 had NAS indicated on their birth certificate, and 159 were ascertained through both sources (Figure 1). Of 185 potential NAS cases identified through the birth certificate, 26 (14.0%) were not documented in the hospital reports and therefore would not have been identified by the existing NSS processes of using hospital reports for case ascertainment alone.

Figure 1.

Completeness of the Massachusetts Department of Public Health’s NAS surveillance system using hospital reports of 6 NAS-related ICD-10-CM codes for case ascertainment—Lincoln-Peterson capture–recapture methodology: Massachusetts births, April 1, 2020 – September 30, 2020

Note: Presented is a visualization of the completeness of case ascertainment calculation using capture–recapture methodology and 2 data sources. M denotes the number of infants captured (ascertained) by hospital reports of ICD-10-CM code, R denotes the number of infants captured (ascertained) in both data systems, C denotes the number of infants captured (ascertained) through the birth certificate, and N denotes the number of potential infants with NAS in Massachusetts.

NAS, neonatal abstinence syndrome.

On the basis of Step 1, using capture–recapture methodology, the estimated number of potential NAS cases expected to have been ascertained over the 6 months through at least 1 of the 6 NAS-related ICD-10-CM codes was calculated to be 531 (95% CI=506, 556). The results of Step 2 estimated the completeness of NSS case ascertainment at 85.9% using ICD-10-CM codes from hospital reports alone.

Positive Predictive Value Estimates

The PPV for each ICD-10-CM code was estimated for April 1, 2020 – December 31, 2022 births classified per the CSTE (Tier 1) NAS standardized surveillance case definition (Table 2). P96.1 was the most reported ICD-10-CM code (documented among 1,529 of the 2,455 potential cases ascertained), followed by P04.14 (n=873). P04.17 (n=28) and P04.1A (n=23) were the least reported codes. P96.1 was estimated to have the highest PPV for confirmed cases (69.9%; 1,068 of 1,529; 95% CI=67.6%, 72.2%), followed by P04.14 (64.2%; 95% CI=61.0%, 67.3%). P04.17 (46.4%; 95% CI=28.0%, 64.9%) and P04.40 (43.5%; 95% CI=31.8%, 55.2%) were estimated to have the lowest PPV for confirmed cases. P96.1 (95.1%; 95% CI=94.0%, 96.2%) and P04.14 (92.2%; 95% CI=90.4%, 94.0%) were also estimated to have the highest PPV for all case types combined, whereas P04.1A (43.5%; 95% CI=23.2%, 63.7%) and P04.40 (66.7%; 95% CI=55.5%, 77.8%) had a lower PPV for all case types. Among the 2,455 potential cases ascertained through the ICD-10-CM codes delivered between April 1, 2020 and December 31, 2022, 2,054 infants were classified as having NAS per the CSTE (Tier 1) standardized surveillance case definition.

Table 2.

Positive Predictive Value of ICD-10-CM Codes Used in NAS Surveillance: Massachusetts Births, April 1, 2020 – December 31, 2022

Case type based on clinical review and classification per the CSTE (Tier 1) NAS standardized surveillance case definition (total potential cases=2,455)	PPV (%) and 95% CI
Potential cases identified by hospital reports (n)	P96.1 (n=1,529)	P04.49 (n=834)	P04.17 (n=28)	P04.1A (n=23)	P04.14 (n=873)	P04.40 (n=69)
Confirmed (n=1,380)	69.9% (67.6%, 72.2%) (n=1,068)	53.5% (50.1%, 56.9% (n=446)	46.4% (28.0%, 64.9%) (n=13)	—	64.2% (61.0%, 67.3%) (n=560)	43.5% (31.8%, 55.2%) (n=30)
Probable Type 1 (n=422)	20.0% (18.0%, 22.0%) (n=306)	13.2% (10.9%, 15.5%) (n=110)	—	—	19.4% (16.7%, 22.0%) (n=169)	7.3% (1.1%, 13.4%) (n=5)
Probable Type 2 (n=27)	1.0% (0.5%, 1.5%) (n=15)	1.0% (0.3%, 1.6%) (n=8)	—	—	0.8% (0.2%, 1.4%) (n=7)	—
Confirmed or Probable (Type 1 & 2) (n=1,829)	90.8% (89.4%, 92.3%) (n=1,389)	67.6% (64.5%, 70.8%) (n=564)	57.1% (38.8%, 75.5%) (n=16)	26.1% (8.1%, 44.0%) (n=6)	84.3% (81.9%, 86.7%) (n=736)	52.2% (40.4%, 64.0%) (n=36)
Suspect Type 1 (n=67)	1.8% (1.1%, 2.4%) (n=27)	2.9% (1.7%, 4.0%) (n=24)	—	—	0.8% (0.2%, 1.4%) (n=7)	—
Suspect Type 2 (n=0)	—	—	—	—	—	—
Suspect Type 3 (n=7)	—	—	—	—	—	—
Suspect Type 4 (n=137)	2.2% (1.4%, 2.9%) (n=33)	6.8% (5.1%, 8.6%) (n=57)	—	—	6.8% (5.1%, 8.4%) (n=59)	7.3% (1.1%, 13.4%) (n=5)
Suspect Type 5 (n=14)	—	1.0% (0.3%, 1.6%) (n=8)	—	—	—	—
All case types (n=2,054)	95.1% (94.0%, 96.2%) (n=1,454)	78.5% (75.8%, 81.3%) (n=655)	67.9% (50.6%, 85.2%) (n=19)	43.5% (23.2%, 63.7%) (n=10)	92.2% (90.4%, 94.0%) (n=805)	66.7% (55.5%, 77.8%) (n=46)
Not meeting case definition (n=401)	n=75	n=179	n=9	n=13	n=68	n=23

Note: Any n values <5 and their respective PPVs have been suppressed.

CSTE, Council of State and Territorial Epidemiologists; NAS, neonatal abstinence syndrome; PPV, positive predictive value.

DISCUSSION

NAS surveillance was successfully established at MDPH as an active, state-wide, population-based surveillance system. Survey respondents reported that NSS was simple to use, well streamlined from case ascertainment to data submission, and flexible owing to an easily modifiable REDCap database. Users also deemed NSS stable and mostly acceptable. The survey identified opportunities to improve workload distribution and the time from case ascertainment to completion of medical record review. As a result, MDPH worked to distribute abstraction assignments more evenly throughout the year and to further streamline the REDCap database. These changes occurred outside the scope and timeline of this study. In addition, NSS proved useful in characterizing the burden of NAS in Massachusetts and identifying trends in care management, such as documentation of a family care plan and infant discharge disposition. Finally, NSS was found to produce high-quality data, representative of the NAS population in Massachusetts.

NSS case ascertainment was estimated to be 85.9% complete using ICD-10-CM codes from hospital reports alone. To improve the completeness of the surveillance system, the Massachusetts birth certificate was added as an additional ascertainment method after the evaluation because some potential cases were not identified by the hospital reports of the 6 NAS-related ICD-10-CM codes. As with the hospital reports, all the infants in a multiple pregnancy are ascertained if 1 of the infants is ascertained through the birth certificate. This change also occurred outside the scope and timeline of this study.

PPV of ICD-10-CM codes ranged from 43.5% to 69.9% for confirmed cases and 43.5% to 95.1% for all case types combined. P96.1 and P04.14 had the highest PPV for confirmed cases and all case types combined, whereas P04.40 had the lowest PPV for confirmed cases, and P04.1A had the lowest PPV for all case types combined. Jurisdictions can prioritize case ascertainment methods based on their surveillance goals (e.g., identifying all potential cases versus minimizing false positives). Those unable to conduct medical record review could use ICD-10-CM codes with higher PPVs to improve their ability to identify cases using the CSTE NAS standardized surveillance case definition.

One study in Florida validating 2 ICD-10-CM codes individually against the CSTE (Tier 1) NAS standardized surveillance case definition found a lower PPV for confirmed cases (64.1% for P96.1, 11.8% for P04.49) than this report.¹¹ Another study calculating the PPV of P96.1 alone and combined with P04.49, before the implementation of the CSTE NAS standardized surveillance case definition, at a subset of Massachusetts birthing hospitals in 2017 (n=15), found that the PPV of P96.1 alone (92.3%) was greater than the PPV of both codes together (65.0%) when validated against medical record review.¹² These results demonstrate the varying levels of predictivity of ICD-10-CM codes for accurately identifying infants with NAS and emphasize the need for further investigation into validating ICD-10-CM codes for NAS case ascertainment.

This study builds on evidence emphasizing the importance of applying a standardized surveillance case definition for NAS. A retrospective cohort study applied 6 different clinical definitions of NAS to the same cohort in Tennessee and estimated NAS incidence to range from 17.4% to 53.9% using these different definitions.¹³ Differing case definitions yield varying estimates for the predictivity of ICD-10-CM codes and the incidence of NAS, highlighting how critical a standardized case definition for NAS is for accurate NAS surveillance and evaluation of NSSs.¹³ To the authors’ knowledge, this study is the first to estimate the completeness of potential NAS case ascertainment using 6 NAS-related ICD-10-CM codes and the PPV of those codes according to the CSTE NAS (Tier 1) standardized surveillance case definition.

Limitations

Findings in this report are subject to at least 4 limitations. First, the use of ICD-10-CM codes in hospital reports is not standardized across hospitals, and NSS does not require reports of all codes associated with substance use during pregnancy (e.g., P04.15, P04.16, P04.41, P04.42). ICD-10-CM codes may be applied differently across hospitals due to variability in how clinicians document a NAS diagnosis in medical records. Communicating these findings with Massachusetts birth hospitals could be a strategy to promote more standardization of ICD-10-CM coding guidelines across hospitals. Failure of the ICD-10-CM codes from hospital reports to capture all potential NAS cases in Massachusetts could lead to underreporting. Second, if any NAS cases were not ascertained through either method (i.e., hospital reports of ICD-10-CM codes or birth certificates), the population estimate would be biased and could impact the accuracy of the completeness estimate. The accuracy of the completeness estimate would be further impacted if cases had a different likelihood of being ascertained by ICD-10-CM codes than birth certificates and vice versa.^7,9 Capture–recapture methodology is still useful in estimating the completeness of surveillance systems and has been leveraged in other analyses.^7,14 Finally, specificity and negative predictive value could not be estimated because the NSS lacks information on infants not identified through these methods to determine whether they meet the CSTE (Tier 1) NAS standardized surveillance case definition (i.e., false negatives and true negatives).

CONCLUSIONS

An evaluation of the Massachusetts’ NSS identified ICD-10-CM codes with higher PPVs, which could be used in NAS surveillance, especially in jurisdictions with limited resources where medical record review is not feasible. The evaluation also highlighted that other sources of case ascertainment, such as birth certificates, could be added to improve the completeness of NAS case ascertainment. Lessons learned from this evaluation might be valuable to jurisdictions with existing NAS surveillance or those considering implementing NAS surveillance using the CSTE NAS standardized surveillance case definition. These results can help inform the appropriate allocation of resources to accurately identify infants with NAS. Accurate identification of infants with NAS is critical to ensure the appropriate and equitable allocation of resources, treatment, and service referrals for families affected by NAS.