Development of an Automated, Customized Data Report for Ongoing Aberration Detection in Syphilis Surveillance Data

Abstract

Introduction:

Identifying aberrations in live surveillance data representing prior to year-end data reconciliation can improve public health response and data quality. We developed an automated, customized data report to communicate potential aberrations to jurisdictions submitting syphilis data to the Centers for Disease Control and Prevention.

Development:

Our multiphase approach encompassed: requirements gathering, exploratory data analyses, rapid prototyping, evaluating relative performance of different aberration detection methodologies, and soliciting end-user feedback on the prototype.

Evaluation:

The final product encompasses statistical code that generates user-friendly quarterly data reports on live syphilis surveillance data, customized to each US jurisdiction. Data reports include an executive summary highlighting critical issues followed by detailed charts and tables showing anomalies in priority variables. The flexible code includes options for alternative methods, figures, and descriptions. End-users preferred a mix of tables and figures, with simple data representations.

Discussion:

Evolving epidemics, decentralized data collection, jurisdictions’ use of various data systems, and public health workforce shortages are critical challenges for national disease surveillance. Additional technical challenges include setting thresholds for aberrations and jurisdictions’ different information needs. End users expressed satisfaction with the prototype, identifying multiple use cases for how jurisdictions with variable morbidity and surveillance capacities could leverage the information for action.

Conclusion:

Our multiphase, user-centered design approach identified challenges in determining and communicating data quality aberrations in a complex data ecosystem. An important consideration for developing such products is balancing complex methodologies with visualizations that are easy to interpret by non-statistical audiences.

INTRODUCTION

Syphilis incidence has increased globally in recent years.¹ In the United States, primary and secondary (P&S) syphilis rates have increased by over 500% since 2003, leading to substantial increases in morbidity and healthcare costs.² Lifetime healthcare costs are estimated at $1,190 per case, resulting in substantial economic burden.³ Further, untreated P&S syphilis among pregnant women presents significant risk of vertical transmission, posing a high risk of stillbirth and neonatal death.⁴ Addressing the syphilis epidemic is a national public health priority.⁵

Accurate and timely surveillance is critical in addressing this epidemic as syphilis is treatable and nearly all syphilis related deaths are preventable with prompt treatment.^6,7 Data quality issues can create challenges for real-time surveillance, leading to potential delays in public health intervention. State, Tribal, local and territorial (STLT) and Centers for Disease Control and Prevention (CDC) sexually transmitted infection (STI) surveillance staff perform ongoing work to improve the completeness and accuracy of case data, with an intensive effort during the year-end reconciliation process. Automating these processes at a national level may expedite data reconciliation and improve surveillance activities throughout the year using the initially submitted, or ‘live,’ case data.

There have been prior CDC-led efforts to develop aberration detection methodologies for STI surveillance data; however, data aberration detection for syphilis is challenging due to its low morbidity in particular jurisdictions and rising rates nationally.⁸ A past national workgroup led by the Council for State and Territorial Epidemiologists developed guidance for monitoring changes in the syphilis epidemic; however, due to variable morbidity across jurisdictions, the guidance did not provide specific numerical thresholds for detecting outbreaks.⁹

Our objective was to develop an automated, customized data report that allowed CDC STI surveillance staff to identify aberrations in live syphilis surveillance data throughout the year and communicate information in a user-friendly manner to jurisdictions transmitting case data to CDC. This project was inspired by the New York State Department of Health’s AIDS Institute Excel-based heat map that visually displays changes in STI case notifications in New York counties. In New York’s tool, large deviations from expected counts are based on historical data.¹⁰ State health department staff use the heat map to prioritize counties to investigate changes in morbidity or data quality issues.

TOOL DEVELOPMENT

Data and Measures

National syphilis case data from 2008-2021 were used throughout the development process. STI case notification data reported to the CDC via the National Notifiable Diseases Surveillance System (NNDSS) were aggregated at the state level.¹¹ Analyses were limited to P&S syphilis because this is the most infectious stage and a primary focus of public health disease intervention.¹² However, consistent with public health disease intervention programs that prioritize all pregnant women, all stages of syphilis were included for cases among pregnant women.

For each year, five files were used for analysis, comprising four live weekly snapshot files representing quarterly data and the year-end reconciled file. Currently, jurisdictions submit live case notification data at least weekly to CDC, with a data reconciliation period after the Morbidity and Mortality Weekly Report (MMWR) year ends. Because it can take two or more months for a case notification to have complete information (leaving time for disease investigations to be completed), each live data file includes a lag. For example, the June 2019 live snapshot data file was used to examine data from the first MMWR quarter (cases reported from January through March). As a result, the annual dataset used in national reporting and evaluation is finalized within one or two years after the MMWR year.² It was envisioned that CDC would generate four quarterly customized data reports for jurisdictions using live weekly snapshot files in June, September, December, and February. Although the data report was developed for use on live data from weekly snapshot files, the final year-end data were used as a ground truth to develop baseline comparisons to assess various aberration detection algorithms. They were also historical references in the data reports; for example, the 2019 quarter 2 data report was developed based on reconciled year-end data from 2015-2018 (4 years) and the live snapshot file from September 2019 (limited to cases from January through June).

Three measures were included in the exploratory analysis and customized data report. The first was total case counts by quarter in the current year compared to expected cases based on historical data. For this measure, multiple methods were considered to determine the expected case count range including historical limits, variations of cumulative sums (CUSUMs), and iterated autoregression models.^13,14 These methods were selected following a literature review, exploratory analysis, and requirements gathering, described below.

The second measure was the percent of observations for four key variables (HIV status, race and Hispanic ethnicity, sex of sex partners, and sexual orientation) that were considered complete, which was operationalized as “valid and usable.” These measures were selected based on surveillance priorities, described below. Completeness was operationalized as “valid and usable” because in some instances, the value was not defined. For example, indeterminate character values were occasionally supplied for numeric variables and hence were not considered “valid and usable”. Further, values such as “missing,” “unknown,” or “refused to answer” were provided for some records and were considered not “valid and usable”. For this analysis, the time frame was all cases from the start of the MMWR year through the current quarter being reported; for example, the Quarter 1 data report (covering case data through March, based on the June data file) examined the percent of valid and usable observations for the first three MMWR months and the Quarter 3 data report (covering live case data reported through September, based on the December live data file) examined the percent of valid and usable observations for the first nine MMWR months. The cumulative time frame was used instead of the current quarter for consistency with other data quality reports that CDC routinely sends to health departments.

The third measure was the distribution of cases for four critical demographic variables (age group, pregnancy status, race and Hispanic ethnicity, and sex). These were prioritized based on expert guidance to identify and monitor high priority subpopulations.⁹ These variables were selected as congenital syphilis is of greatest concern to public health, and racial, ethnic, and age differences are known to exist in STIs.¹⁵ Completeness was not assessed for these variables because they typically have a high proportion complete ins case submissions to NNDSS. Instead, visualizing changes in their distribution could identify anomalies that could indicate data quality concerns such as systematic miscoding or shifts in the epidemic. Historical data from the previous five years of reconciled files and all cases through the end of the current quarter were included. Each variable was stratified by all possible values (including “missing” or “unknown”). For “age group,” “race and Hispanic ethnicity,” and “sex,” all P&S syphilis cases were included, whereas with “pregnancy status,” all stages of syphilis were included for pregnant women.

All analyses were conducted in SAS version 9.4 and R version 4.3.3. Protocols were reviewed by the University at Albany Office of Regulatory Research and Compliance and deemed as non-human subjects research.

Development Approach

A multiphase development process was used to simultaneously analyze results and prototype the end product (Figure 1). Regular meetings with the core team (three CDC STI surveillance staff with collective expertise in the NNDSS data system, syphilis surveillance, epidemiology and biostatistics, public health informatics, and public health surveillance program operations; and two university research partners with collective expertise in health policy, stakeholder-engaged tool development, and biostatistics) were held throughout the developmental process to obtain timely feedback. The core team’s complementary areas of expertise were necessary for ideation, understanding the context, gaining data access, methodology development, and ensuring the final data report would fit into CDC’s existing workflows and be implemented after the project’s completion.

Figure 1. Description of data tool development process.

Phase 1: Requirements Gathering

In the initial phase of development, a review of existing aberration detection methodologies, exploratory analyses, and requirements gathering were concurrently conducted. NNDSS collects approximately 100 variables for syphilis surveillance, some of which have been superseded as surveillance practices have evolved. Initial analyses aimed to identify specific variables and data aberration detection methodologies for further exploration and prioritization.

The review of existing methods covered broadly the public health literature and methods previously developed by CDC and other public health entities. Previous internal research by CDC identified several methodologies for case-based syphilis outbreak detection and highlighted potential benefits and challenges with each method.⁸ Although multiple aberration detection methodologies would be applicable to case count surveillance, our review identified five methods for further exploration: 1) CUSUMs, which compares current values to a rolling average; 2) historical limits, which compares current values to an average of the same time period across multiple years; 3) linear or log-linear regression models, which compare current values to a fitted regression line; 4) autoregressive integrated moving average (ARIMA) models, which are time series models regressing on lagged values from the same time series; and 5) time series generalized linear models which unify generalized linear model approaches and time series models.^13,14,16-18

During this phase, an initial round of focus groups was conducted. These focus groups consisted of three conversations with nine staff members from the CDC’s Division of STD Prevention with expertise in NNDSS data management and syphilis surveillance. They were supplemented by ad hoc conversations with subject matter experts about data tools developed in other CDC divisions. These conversations aimed to clarify data questions, understand existing aberration detection work, and identify priorities for the data report.

Phase 2: Paper Prototype Development

Internal team discussion about findings from the requirements gathering and exploratory analysis allowed the creation of a paper prototype of the data report.¹⁹ The paper prototype was a Microsoft Word document mock-up of the intended end product. Included were example figures and text that would be distributed to jurisdictions to assess their regular case report transmissions to CDC, based on synthetic data.

Core team conversations identified what form the tool would take and how it would fit into the existing surveillance data quality assurance workflow. CDC analysts conduct many of their internal analyses using SAS and R; as such, the data aberration detection tool would need to produce statistics in SAS with visualizations created in R. Generating data reports on a quarterly basis was determined to be most effective, as DSTDP staff currently have quarterly communications with jurisdictions regarding surveillance data quality.

Phase 3: Rapid Prototyping

Using the paper prototype as the baseline, a series of example documents were created in R with various figures focusing on data quality and syphilis morbidity. Multiple versions of figures were produced and discussed iteratively within the core team. As a result of these ongoing discussions, it was determined that the data report should include four sections. In addition to sections on case count aberration, completion of priority variables, and the distribution of selected variables, an introductory page was developed with information on the purpose of the data report and contact details for assistance with interpretation.

Phase 4: External Feedback on Prototype

After completing a prototype data report, we solicited feedback from a broader group of DSTDP subject matter experts. The second round of focus groups was larger and had a wider range of participants, including Surveillance and Data Science Branch leadership (n=6), the Case-based Surveillance Team (n=6), the Data Management Team (n=8), the Behavioral Science and Epidemiology Branch’s STD Outbreak Response Team (n=3), and the Program Development and Evaluation Branch’s Program staff (n=5). University-based authors EM and JA led these discussion groups as they were unaffiliated with CDC and less likely to introduce bias in the collection and interpretation of responses. This phase prioritized eliciting perspectives from CDC staff who had familiarity with disease surveillance and STI projects but were uninvolved with the development of this data report.

Themes from these focus groups were summarized by authors EM and JA and shared with the other core team members. Following team discussion of the synthesized findings, the prototype was updated, focusing on incorporating the lessons learned from focus group discussions. This resulted in the final prototype data report.

Phase 5: Prototype Handoff

Following the final prototype development, R code was automated to produce customized data reports for each jurisdiction. The vision is that these reports would be included in the existing packet of data reports that are currently sent by CDC to jurisdictions on a quarterly basis. The code was verified by a CDC coauthor on the core team (TP) for accuracy and ease of interpretation by future analysts. A user guide explaining all SAS and R scripts, data requirements, and user input requirements was also produced. CDC’s Division of STD Prevention is currently determining the best method to implement the data reports.

EVALUATION

The final product incorporates SAS and R code that generates a collection of quarterly data reports customized to each US jurisdiction. The code automates the process and can be run on live surveillance data. The user guide walks future analysts through all steps needed to produce customized data reports for all jurisdictions. The code is fully annotated and readily modifiable should user requirements (e.g., variables, visualization features, or, descriptive text ) change.

In team meetings and focus groups, aberrations in case counts and data quality were consistently identified as equally important; consequently, the data report was designed to address both concerns. Exhibits include a mix of tables and figures, meeting the needs of users with differing data representation preferences. Visualizations were developed to be accessible and present data in multiple ways for ease of interpretation. The figure for case count aberration detection (Figure 2) includes a historical timeline for a visual representation of recent trends in jurisdictional case counts, the current quarterly case counts, and a shaded region indicating the expected quarterly case count from a representative jurisdiction. The specific jurisdiction is not named to maintain confidentiality. This is supplemented by a table (Figure 3) displaying the same information and conditional coding that highlights case counts outside of the expected region.

Figure 2. Example case count aberration assessment figure for Quarter 1 and Quarter 4 of a high morbidity jurisdiction.

Notes: Line charts with expected regions provide a visual assessment of the jurisdiction’s case count. The shaded purple region indicates an expected range of quarterly P&S syphilis cases. A case count outside of the shaded region is indicative of a potential count aberration. The top panel is an example of the case count aberration figure for a first quarter data report, and the bottom panel is from an example fourth quarter data report. For both panels, the observed values from 2016 through 2020 Quarter 4 are identical because they are finalized, reconciled values. However, the bottom panel contains three additional values (for 2021 Quarter 2, 3, and 4) reflecting the increased amount of available data. Additionally, the 2021 Quarter 1 values differ because the data are not yet reconciled and may change throughout the year. In this example, no aberration is identified in the first quarter data report because the data point is within the predicted range, and two potential aberrations are identified in the fourth quarter data report because they fall outside the predicted range. As more complete data become available throughout the reporting year, the predictive model is updated, which may result in changes to the predictive window.

Figure 3. Example case count aberration table.

Notes: This figure displays an example of table representations of the data included throughout the data report for the same sample jurisdiction. This is an alternative to Figure 2. Observed and expected quarterly case counts are presented with conditional coding to highlight potential aberrations.

The section focusing on percent “valid and usable” of priority variables was divided into two parts. The first, included in the main section of the data report, displays the percent “valid and usable” of each priority variable as of the current quarter (Figure 4, left panel). Second, supplemental figures at the end of the data report provide the percent of “valid and usable” values from all reports during the year as a historical comparison (Figure 4, right panel). All “valid and usable” figures include a dashed line indicating the 70% threshold for inclusion in the national STI surveillance report.² These figures are supplemented by a table displaying the percentages and conditional formatting to highlight “valid and usable” percents under the 70% threshold for inclusion in national reporting. The final set of figures comprises stacked area charts displaying the stratified values of selected variables over the past five years (Figure 5). The purpose of these figures is qualitative insight regarding sudden shifts in the distributions that may indicate an outbreak in a specific demographic group or a potential data quality error.

Figure 4. Percent valid and usable figures from a high morbidity jurisdiction’s Quarter 4 data report.

Notes: The left panel shows the percent of cases with valid and usable data for four priority variables as of the date of the data report for the same sample jurisdiction. The right panel shows the percent of cases with valid and usable data for a single variable at the time of each of the quarterly data reports. In both panels, the green dashed line indicates a threshold for inclusion in national reporting, a baseline jurisdictions are encouraged to exceed in all priority variables. These figures provide a representation of the overall data quality of high priority variables both year-to-date and throughout the year. Although the percent of cases with valid and usable data is expected to increase over time as case investigations are completed (with increasingly higher bars in the right panel), this sample jurisdiction illustrates how the pattern can deviate from that expected for reasons such as increasing case counts throughout the year or changes in data collection practices. This pattern would vary dependent upon the jurisdiction. For example, a jurisdiction that had a reporting error early in the year may show an increase in the percent of observations that are valid and usable (right panel). A jurisdiction that had consistent data coding over time would have stable values for the bar chart.

Figure 5. Distribution of cases by sex and age group for a high morbidity jurisdiction’s Quarter 4 data report.

Notes: Stacked area charts display the distribution of required variables for the same sample jurisdiction. Miscoding or other data quality concerns would result in a sudden shift in the distribution of cases.

Most end users that we interviewed preferred simple data representations, easily interpretable exhibits, and plain language explanations. As a result, we intentionally selected figure types (line charts, bar charts, tables) that would be familiar to the audience, added explanations to each figure, and included longer descriptions in the user guide. Similarly, simpler aberration detection methodologies were preferred by the core team and focus group participants. Although more complex methodologies may give more accurate predictive windows, the interpretability of the applied method and resulting figure was a critical consideration.

DISCUSSION

Using a multi-phase user informed development process, our interdisciplinary team produced a customizable data report to identify potential data aberrations in STI surveillance data. The lengthy requirements gathering process resulted in multiple iterations of a data report providing jurisdictional STI surveillance staff with new tools to assess the quality of syphilis case data transmitted to CDC. While the data report primarily focuses on data quality, including a section on case count aberrations was critical as our requirements gathering indicated that data quality and morbidity are not mutually exclusive concerns.

Globally, different surveillance strategies have been developed to monitor syphilis incidence, with the US prioritizing a case-based surveillance system. NNDSS is a complex data ecosystem with thousands of reporting entities including STLT health departments, community partners, laboratories, and multidirectional data movement to and from federal agencies.²⁰ Multiple public health information systems are used by jurisdictional health departments for warehousing and transmitting STI case information, and data content does not always align between systems.²¹ Further, some jurisdictions have moved to updated Health Level 7 standard message mapping guides, which offer improvements over the older National Electronic Telecommunications System for Surveillance (NETSS) standard but require data transformation in order for national data to be comparable.²² The overall complexity of this system lends itself to an environment where data quality issues or disease outbreaks may not become immediately apparent to surveillance staff, particularly as data are transmitted (e.g., from jurisdictions to CDC). STLT health departments and CDC currently engage in multiple activities to identify and address data quality concerns as they arise; however, some of this work relies on manual analyses, which can result in delays.

The Data Modernization Initiative, and CDC’s Public Health Data Strategy envision public health data becoming timelier and more actionable.^21,22 Automated data reports are one way to improve the data surveillance workflow while meeting data modernization goals. By improving process efficiency, data quality can be assessed more quickly, ultimately allowing for more timely and accurate national reporting throughout the year.^23,24 Data aberrations detected by this data report can represent changes in morbidity or data quality issues. In both cases, the timely identification of an aberration could lead to a more rapid public health response, resulting in improved population health outcomes and a faster year-end data reconciliation process allowing for national statistics to be published earlier.

Our human-centered design approach and transdisciplinary development team bridging both academia and public health practice led to the production of an implementable and pragmatic data report to help identify and ultimately address syphilis surveillance data aberrations. The end-user feedback provided throughout the development process proved integral in shaping the final prototype. Multiple use cases for the resulting data reports were identified by end-users, highlighting the importance of the inclusion of a variety of figures, text, and tables. Automation of data report production in programming languages familiar to CDC analysts ensures that their generation will fit into existing workflows, without significantly impacting workloads. Our final design contained fewer variables and visualizations than initially conceived to allow for a phase-in period to socialize the new data report to jurisdictions and identify opportunities for improvement in future iterations.

This development process has several important limitations. The tool development process focused solely on NNDSS data and case-based surveillance; thus, findings may not be directly applicable to other types of surveillance systems. Although syphilis is a global issue, the variability in surveillance systems led to a focus on the US context with a case-based system. Syphilis is an evolving epidemic, and while the models and data representations presented were well received by subject matter experts consulted during this project, the content, and underlying methodologies may require modification as needs evolve. For example, the growing syphilis epidemic and long-term impacts of the COVID-19 epidemic on the spread of infection and delivery of healthcare and public health services may require use of a different methodology for generating predicted counts. Further, setting numeric thresholds for the number or percent increase in cases that constitutes an outbreak is challenging and thresholds presented in the data report may not be universally applicable beyond syphilis. Given the current geographic differences in the syphilis epidemic, thresholds for moderate to high morbidity regions may not be appropriate for those jurisdictions with lower rates of syphilis. NNDSS contains data for all nationally notifiable conditions, each with unique characteristics and distributions. The aberration detection methodology, thresholds, and key variables selected for syphilis may not be appropriate for a similar data report on a different condition.

Lessons learned from this tool development project can aid future development of data aberration detection methodologies and customized data reports in support of data modernization initiatives.²³ This report is an incremental step toward a more timely and efficient disease surveillance data ecosystem that can be adapted for use in other notifiable conditions and by STLT health departments. By engaging end users throughout the developmental process, a data report was developed that supplements and complements existing data quality and disease outbreak work undertaken by STLT health departments and CDC.

IMPLICATIONS FOR POLICY AND PRACTICE.

• Automated tools to identify aberrations in near real-time surveillance data can improve public health response and data quality.

• Utilizing a user-centered design approach in the development of aberration detection tools is critical to ensure the reports are useful, easy to interpret by the entire audience, and that they will fit within existing surveillance workflows.

• This approach can be adapted by other public health organizations interested in developing customized surveillance reports for their local jurisdictions across different disease areas.