Evaluating Data Quality of Newborn Hearing Screening

Abstract

Scope:

Jurisdictional-based Early Hearing Detection and Intervention Information Systems (EHDI-IS) collect data on the hearing screening and follow-up status of infants across the United States. These systems serve as tools that assist EHDI programs’ staff and partners in their tracking activities and provide a variety of data reports to help ensure that all children who are deaf/hard of hearing (DHH) are identified early and receive recommended intervention services. The quality and timeliness of the data collected with these systems are crucial to effectively meeting these goals.

Methodology:

Forty-eight EHDI programs, funded by the Centers for Disease Control and Prevention (CDC), successfully evaluated the accuracy, completeness, uniqueness, and timeliness of the hearing screening data as well as the acceptability (i.e., willingness to report) of the EHDI-IS among data reporters (2013–2016). This article describes the evaluations conducted and presents the findings from these evaluation activities.

Conclusions:

Most state EHDI programs are receiving newborn hearing screening results from hospitals and birthing facilities in a consistent way and data reporters are willing to report according to established protocols. However, additional efforts are needed to improve the accuracy and completeness of reported demographic data, results from infants transferred from other hospitals, and results from infants admitted to the Neonatal Intensive Care Unit.

Public health information systems play an essential role in measuring and monitoring health related events, as well as in identifying populations at high risk to guide immediate actions. State and territorial-based Early Hearing Detection and Intervention (EHDI) programs represent an evidence-based public health approach that connects public health and clinical preventive services to enable the early identification of infants who are deaf or hard of hearing (DHH; Brownson, Chriqui,& Stamatakis, 2009; U.S. Preventive Services Task Force, 2008). In the United States, congenital hearing loss affects 1.7 per 1,000 infants (Grosse, 2017), and can negatively impact children through delayed speech, language, social, and emotional development when undetected (Williams, Alam, & Gaffney, 2015). Each state and territory has an EHDI Information System (EHDI-IS) that not only captures data on the prevalence of the infants with hearing loss but more importantly serves as a tool to help programs ensure all infants receive follow-up services in accordance with the 1-3-6 national goals. The 1-3-6 goals include: (a) hearing screening at birth or no later than 1 month, (b) diagnosis of hearing loss no later than 3 months, and (c) intervention services beginning as early as possible but no later than 6 months of age. Significantly, better language scores for children who are DHH are associated with early enrollment in intervention (Moeller, 2000).

The use of EHDI-IS offers EHDI programs a way to consistently collect and document information in a standardized way about the population served. EHDI-IS also provides a variety of relevant data analysis and dissemination functions that aid in tracking, surveillance, and program performance assessments. CDC has actively supported the development and implementation of state and territorial-based EHDI-IS through funding and technical assistance. Although all EHDI-IS are intended to help programs ensure children who are DHH reach their full development potential, the infrastructure, operational protocols, and technical details of these systems often vary widely. A variety of salient questions should be asked when assessing these information systems: Are the systems capturing complete and accurate information? How timely is the data collected? How is the acceptability of the system among data reporters? Is the EHDI system flexible enough to accommodate changes in this environment of fast electronic and technology change? The Updated Guidelines for Evaluating Public Health Surveillance Systems (2001; background paragraph) states, “Surveillance systems should be evaluated periodically and the evaluation should include recommendations for improving quality, efficiency, and usefulness.”

During the years 2011 to 2016, CDC funded 52 EHDI awardees (states and territories) to develop and enhance their EHDI-IS. As a condition of funding, EHDI awardees were expected to conduct evaluations of the EHDI-IS to answer some of the above questions and to identify strengths and areas for improvements. In 2013, CDC began actively working with awardees to design and implement evaluation plans and enhance their program evaluation capacity, using a standard evaluation framework. Monitoring and evaluating such a large group of stakeholders with varied needs and requirements is challenging and requires a thoughtful process and advance preparation. Eight webinars on evaluation activities, including potential data collection, specifications, procedures, reporting templates, and methodologies were given to awardees, along with written guidance on evaluation (Planning an Evaluation, n.d.).

Methods

Evaluation Planning

EHDI Awardees programs began the planning process by identifying individuals who would serve as part of the evaluation team and engaging stakeholders (e.g., EHDI program coordinators, epidemiologists, informatics personnel, hospital staff, members of the EHDI Advisory committee, etc.). The involvement of stakeholders was important to the evaluation, as they ensured transparency and facilitated the evaluation process. Along with their evaluation team, CDC and states together developed a logic model (see Figure 1); this common tool used for planning, implementation, and evaluation is a simplified graphic representation of a program or system to gain clarity on the relationship between strategies/activities and their intended outcomes. During the evaluation planning process, each EHDI awardee described what their EHDI-IS entails, how the system works, and the system’s goals, objectives, and criteria for success. This step helped to get consensus among EHDI program staff and CDC over general goals and supporting activities. The development of a logic model also helped CDC to recognize lack of specific functional standards for the EHDI-IS. To address this need, a separate project was initiated by the CDC EHDI team and program managers/data system experts from nine jurisdictions. As a result, a set of eight standards were developed, which identified the suggested operational, programmatic, and technical criteria for EHDI-IS (EHDI-IS Functional Standards, n.d.)

Figure 1.

Final overall state Early Hearing Detection and Intervention Information Systems (EHDI-IS) logic model. CDC = Centers for Disease Control and Prevention.

Evaluation Design

The next step was to develop the evaluation design. A standard framework for evaluating state-based EHDI-IS was developed by the CDC EHDI staff. The framework combines and adapts guidelines from two published articles on information system evaluation to meet the specific needs of state EHDI programs:

1. Updated guidelines for Evaluating Public Health Surveillance Systems published in CDC’s Morbidity and Mortality Weekly Report (MMWR; German et al., 2001). This MMWR was developed to promote the best use of public health resources by developing efficient and effective public health surveillance systems.

2. A complete description of the Six Dimensions of EHDI Data Quality Assessment (n.d.).

CDC program staff combined both guidelines to determine seven system attributes that were the most relevant for evaluating state-based EHDI-IS during this project funding period. The seven attributes included (a) Acceptability, (b) Accuracy, (c) Completeness, (d) Uniqueness, (e) Timeliness, (f) Representativeness, and (g) Usefulness.

Data Collection

During 2015, awardees began collecting quantitative and qualitative data to evaluate the seven attributes of the EHDI-IS. The adapted definition of each attribute and its corresponding indicator and the data collection method is described below. From the three stages of the EHDI process (hearing screening, diagnostic assessment, and early intervention), most programs reported focusing their evaluation primarily on the hearing screening data; therefore, the information presented in this paper is limited to data submitted relating to this first stage. Due to limited evaluation resources, state EHDI programs prioritized the most important questions and attributes to be evaluated in their EHDI-IS, using a criteria-driven decision matrix (Planning an Evaluation, n.d.; see Table 1).

Table 1.

System Attributes and Indicators for Early Hearing Detection and Intervention Information Systems Evaluations

System Attributes	Indicators
Acceptability of the EHDI-IS: The willingness of persons and organizations to participate in the EHDI information system, e.g., hospitals, birthing facilities, Neonatal Intensive Care Units, midwives, military facilities, etc.	To assess acceptability, awardees reviewed hospitals’ participations rate; delinquent hospital reports; and percentage of key hearing screening fields completed.
Accuracy of the screening data: The extent to which data are correct, reliable, and certified free of error.	Awardees evaluated the percentage of records with incorrect values in data fields.
Completeness of the screening data: The proportion of stored data against the potential of “100% complete.”	Awardees identified the percentage of patients’ records that contained all minimum, required data elements.
Uniqueness of the screening data: Nothing will be recorded more than once based upon how that thing is identified.	Awardees identified the percentage of duplicated records.
Timeliness: The timeliness in the reporting or collection of data.	Awardees reviewed the average number of days between hearing screening date and information being recorded in EHDI-IS.
Representativeness: A public health surveillance system that is representative accurately describes the occurrence of a health-related event over time and its distribution in the population by place and person.	Because a low number of jurisdictions assessed representativeness and there were discrepancies in its definition, analysis and results were not included in this report.
Usefulness: Indicates the level of usefulness by describing the actions taken as a result of analysis and interpretation of the data from the public health surveillance system.	Because a low number of jurisdictions assessed usefulness and there were discrepancies in its definition, analysis and results were not included in this report.

Data Management and Analysis

Awardees shared evaluation results with stakeholders and sent their final evaluation report to CDC EHDI in July 2016. Because only programmatic information was collected from respondents reporting evaluation measures, Institutional Review Board approval was not required for data collection and analysis. Forty-eight EHDI programs out of fifty-two (92%) successfully completed their evaluations by the end of the funding cycle. Staff turnover and lack of key personnel in place were the primary reasons that four EHDI programs were unable to complete their planned evaluation. All evaluation reports were reviewed by CDC EHDI staff; codes and categories were developed to analyze quantitative and qualitative data. Descriptive statistics were calculated using Excel.

Results

The evaluation results from the 48 awardees that successfully completed an evaluation are presented below. Table 2 shows the number of evaluations conducted on each system attribute.

Table 2.

System Attributes and Indicators for Early Hearing

System Attribute	Number of Jurisdictional Evaluations
Acceptability of the EHDI-IS among hearing screening data reporters 30	30
Accuracy of the screening data 15	15
Completeness of the screening data 29	29
Uniqueness of the screening data	11
Representatives	3
Usefulness	5

Note. EHDI-IS = Early Hearing Detection and Intervention Information Systems.

Acceptability

Among the 30 evaluations conducted to assess the acceptability attribute, 26 awardees (86.2%) stated that data reporters demonstrated their commitment to report hearing screening results, and that in general, hospital compliance reporting was good. Nine evaluations showed that hospitals did not consistently submit screening results to EHDI programs from infants transferred from other hospitals. Birthing facilities were not clear about how to document hearing screening results for infants transferred from one birthing facility to another. Hearing screening results were also less likely to be reported for children born outside of a birthing facility.

In addition, reports showed that in some cases, birthing facilities were not clear about how to document hearing screening results for infants admitted to the Neonatal Intensive Care Unit (NICU), so hearing screening results were often missing for these infants (see Table 3).

Table 3.

Most Common Evaluation Findings

Acceptability of the EHDI-IS	Most EHDI programs, (26 out of 30) reported that hospital compliance reports were good. Hospital data reporters demonstrated their commitment to report screening results. Inconsistent reporting from: Infants transferred from other hospitals Infants admitted to the Neonatal Intensive Care Unit (NICU) Midwifery providers
Accuracy of the Screening Data	Errors in demographic data: Infant name Date of birth Maternal demographic information: Race, age, education Primary contact information: primary caregiver’s phone number and address
Completeness of Screening Data	Infants transferred from other hospitals Infants admitted to the Neonatal Intensive Care Unit (NICU) Demographic Data Reasons were not screened Primary contact information: primary caregiver’s phone number and address

Four awardees out of thirty (13.8%) reported issues with the acceptability attribute and the willingness of reporters to participate in the EHDI information system. Those programs described that not all hospitals’ staff were adequately trained to report, and lack of knowledge was the primary reason why they were not reporting. In addition, territorial EHDI programs reported that due to the shortage of nurses, some staff rotate to all territories without the needed training to conduct newborn hearing screening or complete and submit the reports. One respondent stated, “Many providers are not aware of the reporting requirements.”

There were a wide range of responses about the acceptability of the EHDI-IS among midwives and military hospitals. Four awardees that evaluated the willingness of midwife clinics and community birth centers to report data found high participation rates, but stated that additional training on timely reporting are needed. Statements from participants include:

“While challenges remain with ensuring that all infants born at home or a free-standing birth center receive a hearing screen, we have found that partnering with midwives by providing them with hearing screening equipment and training is an effective way to increase the number of out-of-hospital births screened for hearing loss.”

“Currently, there is no statute identifying the entity responsible for completing and reporting hearing screening for infants who are born at home or at midwife practices.”

Among the two state EHDI programs that evaluated the acceptability among military hospitals, one reported compliance of reporting protocols among military bases while the other described issues and barriers collecting data from these facilities.

Accuracy

Results showed that nearly all of the 15 EHDI programs that evaluated the accuracy of data reported issues and discrepancies. The most common issue reported in 14 evaluations was inaccuracy of demographic information: infant’s name was misspelled or not known

(e.g., “Baby Boy Doe”), or other information was missing or incorrect, such as date of birth or maternal demographic data: race, age, education, and phone number. Although states reported lower rates of errors for screening results, seven programs indicated that data entry errors are common and suggested that additional training for hospital staff would be required. For example: “Additional training of hospital staff regarding the importance of data accuracy for program follow-up may improve the quality of required fields in the system.”

“The online data reporting system needs additional validation rules in place to prevent users from making common errors.”

“…the EHDI program plans to provide key findings of the data quality evaluation with the … Medical Center regarding the importance of accurate demographics entered into the hearing screening equipment”.

Completeness

Among the 29 awardees that evaluated this attribute, no one reported 100% completeness of data. The most common issue was incomplete demographic data. Additional missing information included (a) reason infants were not screened, (b) primary contact information, (c) risk factor information, and (d) data for infants that were transferred to a hospital with a higher level of care or admitted to the NICU. As one respondent stated, “Hospital staff should be automatically alerted when a transfer exists in their queue instead of having to check to determine if a transfer exists or not. This will save time and prevent user frustration.” Evaluation results suggested that a strong linkage between the Vital Statistics system or program and EHDI-IS database plays a key role in gathering complete and accurate data.

Timeliness

Twenty-six EHDI programs evaluated the timeliness of their reporting or collection of data. Figure 2, shows the number of days between a screening conducted and screening data reported to the state EHDI-IS. Twelve EHDI programs could see records within seven days after completion of the screening. The average of data input among reporters was 13.8 days from the day of screening. For others, data is available only on a monthly basis when most hospitals and birthing centers report. Some EHDI programs indicated that the number of hospitals and clinics reporting their results within two weeks of the event has increased during the last several years, making the hearing records available to track in accordance with the 1-3-6 guidance. As one staff member stated, “We learned that the earlier we start on follow-up, the better the result.”

Figure 2.

Timeliness of hearing screening data

Uniqueness

Eleven of the 48 responding EHDI programs evaluated the uniqueness of the screening data. Among the 11 programs, nine reported issues with duplicity of records.

“When hospital users change the birth hospital or enter a baby admitted from home, a duplicate record is created in the system.”

“Hospitals don’t always take the time to find the right Primary Care provider and hearing coordinators spend a substantial amount of time reconciling duplicate entries in the library.”

Actions were taken by staff EHDI programs to correct issues with duplicate records as reports stated,

“Additional validation rules have been identified that will eliminate duplicative data entry and they will be put in place after the EHDI-IS upgrade.”

“The program determined that there continues to be variances between the EHDI database systems regarding annual live births and infants receiving newborn screening in the state. However, following enhanced quality assurance efforts to reduce duplication of patient charts, an improvement was noted.”

Two programs reported no duplicate records and indicated that their EHDI-IS systems have advanced features and processes in place to eliminate duplicate records. The likelihood of creating duplicate records when linking with other data systems, such as vital records, is highly dependent on the quality of the data produced by each system involved.

Representativeness and Usefulness

Only three awardees evaluated representativeness and five evaluated usefulness of their EHDI-IS. However, the definitions and indicators used by the programs were inconsistent and therefore are not included in this report. Usefulness appears to be a complex attribute to operationalize. So standardization of the definition for usefulness and specific guidelines may be helpful to states interested in evaluating this attribute in the future.

Discussion

The completed evaluation reports showed that during the years 2013–2016, EHDI awardees developed program evaluation capacity and were generally successful in conducting evaluation activities for their EHDI-IS. Staff turnover and a lack of key personnel were reported as the main barriers for EHDI programs to complete evaluation activities. Results from these evaluations indicate that reporting hearing screening data to state EHDI programs has become a standard practice, and overall, data reporters are willing to participate in the EHDI-IS.

When low acceptability of the EHDI-IS was reported, it was primarily because hospital staff were not aware of the reporting requirements or had not previously heard about the EHDI-IS. The data collection needed for an effective newborn hearing screening program requires extensive coordination with all key stakeholders. Even with reporting laws and/or procedures in place, continuing two-way communication with data reporting facilities can be essential for the success of EHDI programs. In addition, factors influencing the acceptability of a particular system include dissemination of aggregate data back to reporting sources and interested parties (German et al., 2001). Acceptability also depends on the data reporter’s ability to provide accurate, consistent, complete, and timely data.

Evaluation results showed that accuracy and completeness of demographic data should be improved among state EHDI programs due to errors in key data elements reported to the EHDI-IS. When complete and accurate demographic information is available to the EHDI program, tracking and surveillance for infants who need follow-up services can be improved and duplicate records reduced. Nationally, it is difficult to monitor children needing follow-up services and to accurately assess progress toward the 1-3-6 benchmarks when local data are incomplete and/or inconsistent (Alam, Satterfield, Mason, & Deng, 2016). EHDI programs can examine, through data analysis, if a certain factor (e.g., maternal education or age) is associated with infants and young children becoming lost to follow-up. Individual level data can also be combined with data from other public health databases, such as birth defects registries or education records, which makes it possible to conduct additional analyses to assess the delivery of services and outcomes among DHH children.

Transferred infants and those admitted to the NICU were among the most often-missing information according to the evaluation reports. Establishing protocols that specify how data is to be collected and reported are the best way to ensure all infants receive recommended hearing screening and rescreening services (EHDI Guidance Manual, n.d.).

The timeliness of the collection of the hearing screening results varied greatly among EHDI programs. States with the capacity to collect the timeliest screening data could see results nearly live. Conversely, there are a number of states that only collected data on a monthly basis. Timeliness in the reporting of data depends on the rules and agreements state EHDI programs have established with hospitals. States that collect the results within two weeks of the screen are likely to have more time to follow-up with newborns that did not pass screening compared to programs that only receive initial screen data once a month.

Limitations

This study involved a large group of EHDI programs with diverse EHDI-IS, staff and stakeholders, and the findings of this evaluation are subject to at least two limitations. First, each EHDI program developed their own instrument to collect information, instead of using a standardized set of evaluation instruments. This impacted the ability to make comparisons among states. Second, the design of this study was descriptive, and the responses and findings relied upon the accuracy of state reporting, which may be influenced by social desirability bias.

Conclusion/Next Steps

Evaluation is an important activity that can help EHDI program managers and staff identify obstacles to program effectiveness and provide guidance about where to adjust EHDI activities and strategies to optimize outcomes. Jurisdictions, with guidance from CDC, completed an evaluation of their

EHDI-IS, with a specific focus on screening data. Although each jurisdiction has their own EHDI-IS, there were similarities and trends in the evaluation findings. Most of the hospital and birth facilities’ data reporters across states have demonstrated their commitment to report screening results. However, additional efforts are needed among jurisdictions to ensure high quality data is consistently collected. These efforts can include

1. Maintaining communication with data reporters and disseminating aggregate data back to reporting sources and interested parties;

2. Keeping updated protocols in place on how to report and to establish specific protocols to deal with infants transferred from other hospitals and infants admitted to a NICU;

3. Emphasizing to data reporters the importance of the quality of demographic data;

4. Emphasizing to midwives the importance of timely reporting; and

5. Enhancing, when possible, the linkage between EHDI-IS and Vital Records to help ensure complete and accurate demographic data.

Lastly, state EHDI programs are encouraged to continue and expand their evaluation efforts by conducting formal evaluations related to the subsequent diagnostic and intervention phases of the EHDI process.

Acronyms:

CDC

Centers for Disease Control and Prevention

EHDI

Early Hearing Detection and Intervention

IS

Information Systems

DHH

deaf or hard of hearing

NICU

Neonatal Intensive Care Unit