Completeness and Accuracy of Local Clinical Registry Data for Children Undergoing Heart Surgery

Abstract

Background

Data routinely captured in clinical registries may be leveraged to enhance efficiency of prospective research. However, the quality of registry data for this purpose has not been studied. We evaluated the completeness and accuracy of perioperative data within congenital heart centers’ local surgical registries.

Methods

Within 12 Pediatric Heart Network (PHN) sites, we evaluated 31 perioperative variables (and their subcategories, totaling 113 unique fields) collected via sites’ local clinical registries for submission to the Society of Thoracic Surgeons Database, compared to chart review by PHN research coordinators. Both used standard STS definitions. Data were collected on 10 subjects for 2–5 procedures/site and adjudicated by the study team. Completeness and accuracy (agreement of registry data with medical record review by PHN coordinator, adjudicated by the study team) were evaluated.

Results

A total of 56,500 data elements were collected on 500 subjects. With regard to data completeness, 3.1% of data elements were missing from the registry, 0.6% from coordinator-collected data, and 0.4% from both. Overall registry data accuracy was 98%. In total, 94.7% of data elements were both complete/non-missing and accurate within the registry, although there was variation across data fields and sites. Mean total time for coordinator chart review per site was 49.1 hours vs. 7.0 hours for registry query.

Conclusions

This study suggests that existing surgical registry data constitute a complete, accurate, and efficient information source for prospective research. Variability across data fields and sites also suggest areas for improvement in some areas of data quality.

The availability of cardiovascular data contained in numerous sources, including clinical registries, has continued to increase over the past two decades. At the same time, federal funding to support research has become more scarce, leading to increasing interest in newer methods to support more efficient conduct of research. One proposed method is the use of existing clinical registry data to more efficiently power prospective investigation [1]. In adult cardiovascular disease, two groups have recently utilized these methods to successfully conduct prospective randomized trials, facilitating efficient enrollment, avoiding duplicate data collection, and lowering trial costs [2, 3].

However, this method has not been validated or widely adopted in other populations to date. Several questions remain concerning the rigor of this approach, particularly with regard to data quality. While clinical registries provide a readily available source of information, the data are primarily collected for benchmarking and quality improvement purposes. It is uncertain whether data quality standards are comparable to standards for data prospectively collected in the setting of a clinical trial. While many registries conduct audits to evaluate data quality, not all variables or sites may be audited [4–6]. More comprehensive evaluation of the quality of registry data across sites and networks actively engaged in multi-center prospective research is needed.

In the field of congenital heart surgery, nearly all US centers now collect standardized perioperative data for submission to the Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) [7]. Several centers also participate in the Pediatric Heart Network (PHN), funded by the National Heart, Lung, and Blood Institute, which conducts clinical trials and other prospective investigations related to pediatric cardiovascular disease [8]. While the PHN has traditionally relied upon prospective data collection by trained research coordinators, many of the surgical variables of interest are now routinely captured at PHN sites within their local registry for submission to the STS-CHSD. However, the quality of these registry data across PHN sites remains unclear as the STS currently audits only a subset of registry variables at ~10% of participating sites annually, such that not all PHN sites or variables of interest have been recently audited. Therefore, the purpose of the present study was to evaluate the completeness and accuracy of perioperative data available in PHN sites’ local surgical registries compared to data collected by trained PHN coordinators (current standard for PHN studies). Our overall goal was to assess whether these existing registry data may serve as a high quality data source for more efficient conduct of prospective trials and other studies in the congenital heart surgery population.

Patients and Methods

Overview

In this retrospective study (Figure 1), perioperative data collected through two different methods were evaluated: 1) existing data in sites’ local surgical registries collected for submission to the STS-CHSD and 2) chart review by PHN research coordinators Both used standard STS-CHSD definitions. Twelve US sites participating in both the PHN (9 core, 3 auxiliary sites) and STS-CHSD were included. Each site received institutional review board approval with waiver of informed consent.

Figure 1.

Schematic diagram of study

Sources of Data

Local surgical registry data

Registry data were obtained by querying sites’ local surgical registries containing data collected and submitted to the STS-CHSD. While practices vary across centers, these data are generally collected by a combination of clinicians and trained data managers. Resources for data managers include a manual of detailed data specifications including definitions, standardized training materials, regular phone conferences, updates on the STS website, site audits, and an annual symposium. STS variables are captured using standard definitions and nomenclature from the International Pediatric and Congenital Cardiac Code and Multisocietal Database Project [9]. Standard queries were developed to ensure uniform data extraction from the registry regardless of the software used by the center.

Coordinator-collected data

Data on the same data fields were also collected separately by trained PHN research coordinators at each site through retrospective chart review and entered into a REDCap (Research Electronic Data Capture) database [10]. The same standard STS-CHSD definitions for each variable were used [9, 11]. PHN coordinators primarily collect data related to prospective PHN studies and clinical trials. While the experience level among coordinators varies, all are required to complete standard training and demonstrate familiarity with methods for data collection, endpoint measurement, knowledge of definitions and source documents, and use of data management systems. There are monthly coordinator conference calls and two in-person meetings each year. The PHN coordinators who did the chart review had varying experience: 50% had > 5 years’ experience, 33% had 2–5 years’ experience, 17% < 2 years’ experience. Only 1 (8%) site’s coordinator was non clinical, 92% had clinical experience as nurse, nurse practitioner or respiratory therapist, with 67% having cardiac experience. Furthermore the PHN coordinator and registry data collectors were not the same individuals at any of the sites.

Study Population

Each site submitted data on the 10 most recent eligible subjects prior to January 1, 2014, who were ≤ 12 months of age and underwent the following procedures: 1) complete common atrioventricular septal defect (AVSD) repair; 2) repair of tetralogy of Fallot (TOF) with pulmonary stenosis (PS); 3) arterial switch operation (ASO) for d-transposition of the great arteries with intact ventricular septum (dTGA/IVS); 4) arch and ventricular septal defect (VSD) repair (including interrupted aortic arch (IAA)); and 5) Norwood procedure. We excluded subjects who had undergone prior open or closed heart surgery or prior cardiac catheterization intervention, with the exception of balloon atrial septostomy and/or atrial stenting prior to ASO or Norwood procedure. These procedures and inclusion/exclusion criteria were chosen to coincide with the planned PHN Residual Lesion Score (RLS) study which evaluates the impact of residual lesions following congenital heart surgery on outcomes.

Variable Selection

Study investigators (JWG, JJ, MJ, MN, JN, SP) selected common perioperative variables from the STS-CHSD for inclusion in the study [12], comprising of 31 main variables and their subcategories totaling 113 unique fields (Table 1).

Table 1.

Accuracy and Completeness of Registry Data across Study Data Fields

Data Fields	Definition of Acceptable Variance	Registry Data Accuracy (%)	Missing Registry Data (%)
Date of Birth	± 1 day	97.6	0.0
Birth weight	≤ 0.5kg	97.1	24.2
Gender	NA*	100	0.0
Gestational age (weeks)	± 1 week	96.4	21.9
Race	NA*
Asian		99.2	1.2
Black		96.8	1.0
Caucasian		93.9	1.0
Native American		99.8	1.2
Native Pacific		99.8	4.1
Other		90.9	3.0
Ethnicity	NA*	91.9	1.4
Mortality Date	± 1 day if death occurred close to midnight	99.4	0.6
Noncardiac Congenital Abnormalities
Anal Atresia		99.6	1.0
Congenital diaphragmatic hernia		100	1.0
Gastroschisis		100	1.0
Hirschsprung’s Disease		100	1.0
Intestinal Malrotation		99.4	1.0
Omphalocele		100	1.0
Tracheoesophageal fistula		100	1.0
Chromosomal abnormalities	either is acceptable for the following- 45X0(Turner), Tri 18(Edwards), Tri 21(Down), 22q11(Digeorge)
22q11 deletion		99.2	0.2
45XO		99.8	0.2
Trisomy 18		100	0.2
Trisomy 21		99.6	0.2
Syndromes	Same as above
CHARGE association		99.8	0.6
DiGeorge syndrome		99.2	0.6
Down syndrome		99.6	0.6
Edwards Syndrome		100	0.6
Heterotaxy syndrome		99.4	0.6
Heterotaxy, Polysplenia		99.8	0.6
Turner syndrome		100	0.6
VACTERL syndrome		100	0.6
VACTERL-H syndrome		99.8	0.6
Admission date	± 1 day	99.0	0.0
Surgery date	± 1 day	98.4	0.0
Height (cm)	≤ 5 cms	98.5	4.0
Weight (kg)	≤ 0.5 kg	100	0.2
Preoperative factors	NA*
Cardiopulmonary resuscitation		96.8	5.8
Mechanical circulatory support		97.9	5.8
Necrotizing enterocolitis, treated medically		99.1	5.8
Necrotizing enterocolitis, treated surgically		100	5.8
Sepsis		99.1	5.8
Sepsis with positive blood culture		99.1	7.8
Renal dysfunction		98.5	7.8
Renal failure requiring dialysis		100	7.8
Mechanical ventilation to treat cardiorespiratory failure		87.3	7.2
Antenatal Diagnosis	NA*	85.9	0.4
Diagnosis	Diagnosis codes for TOF and TOF/PS are interchangeable, Diagnosis codes for CoA, Arch hypoplasia, IAA with VSD are interchangeable
AVSD, complete AVSD		99	0.2
TOF		99.6	0.2
TOF, pulmonary stenosis		99.8	0.2
HLHS		98	0.2
Single ventricle, DILV		99.6	0.2
Single ventricle, DIRV		100	0.2
Single ventricle, mitral atresia		98.2	0.2
Single ventricle, tricuspid atresia		100	0.2
Single ventricle, unbalanced AV canal		99.4	0.2
Single ventricle, Heterotaxia		99.2	0.2
Single ventricle, other		99.0	0.2
Single ventricle+TAPVC		99.8	0.2
TGA,IVS		99.2	0.2
VSD + Aortic arch hypoplasia		98.8	0.2
VSD + Coarctation of aorta		99.0	0.2
Interrupted aortic arch + VSD		99.8	0.2
Status post	NA*
Hybrid Stage I, bilateral PA bands		100	0.2
Hybrid Stage I, PDA stent		100	0.2
Hybrid Stage I, PA bands and PDA stent		100	0.2
Therapeutic Balloon dilation		97.2	0.2
Therapeutic Balloon valvotomy		99.6	0.2
Therapeutic stent insertion		99.6	0.2
Therapeutic stent re-dilation		99.8	0.2
Modified Blalock Taussig Shunt		97.6	0.2
Central shunt		99.4	0.2
Other systemic to pulmonary shunt		99.4	0.2
Pulmonary artery band		99.8	0.2
Procedures	TOF repair and TOF/PS repair are inter changeable, CoA, Arch hypoplasia and IAA with VSD repair, are inter changeable
CAVSD repair		99.2	0.0
TOF repair, no ventriculotomy		98.4	0.0
TOF repair, ventriculotomy, nontransannular patch		98.2	0.0
TOF repair, ventriculotomy, transannular patch		98.4	0.0
Norwood procedure		98.8	0.0
Arterial switch operation		99.4	0.0
Coarctation repair + VSD repair		99.8	0.0
Aortic arch repair + VSD repair		99.6	8.0
Interrupted aortic arch repair		99.2	0.0
CPB time	±5 minutes	95.2	0.4
Cross clamp time	±5 minutes	98	0.4
Deep Hypothermic Circulatory arrest time	±5 minutes	99.3	3.0
Cerebral perfusion time	±5 minutes	98.3	3.6
Initial extubation date and time	± 1 day, ±10 minutes	76.6	4.8
Reintubated after initial postoperative extubation	NA*	93.7	4.6
Final extubation date	± 1 day, ±10 minutes	96.4	4.0
OR exit time	±10 minutes	91.5	24.2
Complications	NA*
MSOF, MSOD		99.3	8.0
Cardiac arrest		98	8.0
Cardiac dysfunction resulting in low cardiac output		91.7	8.0
Cardiac failure, severe cardiac dysfunction		91.5	8.0
Bleeding requiring reoperation		96.3	8.0
Sternum left open, planned		93	8.0
Sternum left open, unplanned		95.7	8.0
Unplanned noncardiac reoperation postop		93.7	8.0
Postop mechanical circulatory support		98	8.0
Postop mechanical ventilation >7 days		88.5	8.0
Postop respiratory insufficiency requiring reintubation		94.6	8.0
Respiratory failure requiring tracheostomy		99.3	8.0
Acute renal failure requiring dialysis at hospital discharge		99.1	8.0
Acute renal failure requiring temporary dialysis		96.3	8.0
Acute renal failure requiring temporary hemofiltration		99.6	8.0
Sepsis		98	8.0
Paralyzed diaphragm		98	8.0
Deep wound infection		99.3	8.0
Mediastinitis		97.2	8.0
Date of hospital discharge	± 1 day	97.8	0.0
Mortality status at hospital discharge	NA*	99.8	0.0
Status at 30 days after surgery	NA*	99.1	1.2

All data fields included in the study are listed, along with definitions of acceptable variance. For each data field, accuracy and % missing within the registry data are reported. Bold indicates values below predetermined thresholds of ≤ 97% agreement and ≥ 5% missing data

NA* indicates no acceptable variance defined (agreement requires perfect match).

Adjudication

The panel defined criteria for perfect matches, acceptable variance (small clinically insignificant differences), and unacceptable variance (mismatches) between the registry data and coordinator chart review (Table 1). All mismatches were adjudicated by the local site principal investigator (or their designee) who conducted an independent review of the medical record to classify mismatches as: registry data correct, coordinator data correct, or unable to adjudicate. Adjudication data were further reviewed by the panel to ensure consistency.

Analysis

Our primary goal was to evaluate the completeness and accuracy of the registry data. The following definitions were used for this study:

Completeness

Completeness was defined as the proportion of non-missing data. It was reported as missing in the registry, coordinator-collected data, or both.

Accuracy

Agreement of the registry data with medical record review by the PHN coordinator, adjudicated by the study team as described above, was used to describe accuracy. This included perfect matches, acceptable variances, and cases where data mismatches were adjudicated as the registry data being correct. Results are reported both overall and for non-missing data.

We also reported the proportion of registry data that were both complete (non-missing) and accurate. Results are reported overall, by site, and by data field. We considered inadequate accuracy to be <97% and inadequate completeness as >5% missing. Finally, to capture the time required for data collection for either method, each site provided the number of hours required to query the registry and for the PHN coordinator to collect and enter data, and summary data were reported. Given the descriptive nature of the study, standard descriptive statistics were used and no formal statistical comparisons were made.

Results

Study population

Eight of the nine core PHN sites enrolled 50 subjects each (10 subjects for each of 5 procedural categories). One core site was unable to provide data on the Arch/VSD procedural category, because their practice was to repair the arch and VSD in separate procedures. Thus, this site accrued 40 subjects. Three PHN auxiliary sites accrued 20 subjects each (10 subjects for each of 2 procedural categories - AVSD and Norwood). In total, 500 subjects and 56,500 data elements were included in the study.

Registry data completeness and accuracy

With regard to data completeness, 3.1% of data elements were missing from the registry only, 0.6% from coordinator-collected data only, and 0.4% from both (Table 2). In our evaluation of data accuracy, we found overall accuracy of the registry was 98.0% (for non-missing data elements; Table 2, Figure 2). Of note, during the adjudication process for the 521 data elements where there was a mismatch between the registry and coordinator chart review, the coordinator data was adjudicated as correct in 53.7% of instances, and the registry in 46.3% of instances (Table 2). In total, 94.7% of data elements were found to be both complete and accurate within the registry data (Figure 2).

Table 2.

Overall Accuracy and Completeness of Registry Data

Measurement	Percent (denominator includes all data elements)	Percent (denominator includes only non-missing pairs)
Data Accuracy
Overall agreement of registry data with adjudicated PHN coordinator medical record review*	93.9% (53054/56500)	98.0% (53054/54161)
Perfect matches	91.7%(51821/56500)	95.7% (51821/54161)
Acceptable variances	2.2%(1233/56500)	2.3% (1233/54161)
Mismatch of Registry and Coordinator Data and Results of Adjudication Process
Coordinator data correct	0.9% (521/56500)	1.0% (521/54161)
Registry data correct	0.8% (450/56500)	0.8% (450/54161)
Unable to adjudicate	0.2% (136/56500)	0.3% (136/54161)
Data Completeness
Missing in coordinator data only	0.6% (365/56500)	NA
Missing in registry data only	3.1% (1729/56500)	NA
Missing in both	0.4% (245/56500)	NA

NA: not applicable

^*As described in the methods, includes perfect matches, acceptable variances, and cases where data mismatches were adjudicated as the registry data being correct

Figure 2.

Overall accuracy and completeness of registry data

Variation across data fields and sites

Completeness and accuracy varied across data fields (Table 1). With regard to data completeness, 111/113 (98.2%) of data fields had <10% missing data in the registry, and 83/113 (73.4%) had <5% missing. Accuracy was >97% for 83.2% (94/113) of the data fields. Table 1 displays information on completeness and accuracy across all 113 data fields. The registry had a high degree of completeness and/or accuracy (0 to 4.9% missing, 97.2 to 100% accurate) for fields such as dates of birth, admission, and discharge, gender, height, weight, non-cardiac/genetic anomalies, most pre-operative factors, primary diagnosis and procedure, several post-operative complications, and mortality. The registry was less complete and/or accurate (5.8 to 24.2% missing, 76.6 to 96.8% accurate) for fields such as birth weight, gestational age, race/ethnicity, ventilation data, and certain complications. Missing data in particular was an issue for pre-operative factors and complications at some centers.

Site-specific completeness and accuracy are displayed in Table 3. There was minimal variation across sites in accuracy of registry data (ranging from 96.8%–99.1%), but considerably more variation for proportion of missing registry data (ranging from 0.1%–14.5%). For the two sites with the highest proportion of missing registry data, the main sources of missing data were gestational age (89% missing) and complications (24% missing) for site A; and complications (56% missing), pre-operative factors (46% missing), birth weight (29% missing), and gestational age (22% missing) for site F.

Table 3.

Site-specific Registry Data Accuracy and Completeness

Site (n)	Registry Data Accuracy (%)*	Missing Registry Data (%)
A (50)	98.2%	7.1%
B (50)	99.1%	0.8%
C (50)	98.7%	0.1%
D (50)	98.3%	0.5%
E (20)	98.1%	2.9%
F (50)	98.0%	14.5%
G (20)	97.9%	3.6%
H (50)	96.8%	1.1%
I (20)	99.1%	1.3%
J (40)	97.5%	2.1%
K (50)	97.2%	0.6%
L (50)	97.2%	1.3%

^*Reported for non-missing data elements

Time analysis

The time for PHN coordinator’s chart review, data collection, and entry averaged 0.9 (± 0.5) hours/subject. For the sites that enrolled 40 subjects or greater, this totaled a mean of 49.1 (± 26.9) hours/site. In contrast, the average time reported to query the registry was a mean of 7.0 (± 5.3) hours/site.

Comment

In this study, we evaluated the completeness and accuracy of perioperative data collected within sites’ surgical registry compared to chart review by PHN research coordinators. We found that the registry data were 98% accurate, and that 94.7% of data elements were both complete/non-missing and accurate in the registry data. Data extraction from the registry saved a considerable amount of time. Variation in completeness and accuracy across data fields and sites suggests areas for improvement.

In recent articles [1,13,14], leveraging existing registry infrastructure and data has been proposed as a platform to more efficiently conduct prospective research, including clinical trials. The potential advantages of this methodology were recently demonstrated by the Thrombus Aspiration during ST-Segment Elevation Myocardial Infarction trial, which used a Swedish catheterization registry as the trial backbone and enrolled >7,000 subjects in ~2 years with a total reported budget of only $300,000 [2]. These methods have also been successfully used in another study utilizing an American College of Cardiology catheterization registry as the trial platform [3].

In considering application of this methodology to the pediatric cardiovascular population, it is necessary to further understand data quality across available registries.. The most recent STS-CHSD audit results from 2013 across ~10% of participating sites demonstrated 100% completeness and 99% accuracy of data fields pertaining to both primary procedure and in hospital mortality. In addition, a previous study demonstrated 99.8% congruence of in-hospital mortality data between the STS-CHSD and a large administrative dataset [15]. Published audits of other registries in the field such as the Pediatric Cardiac Critical Care Consortium (PC⁴) Registry have also demonstrated high rates of overall accuracy (99.1%) [5].

Our study expands upon these findings by examining completeness and accuracy of a broad spectrum of registry data fields specifically across sites participating in the PHN who are actively engaged in conducting prospective research. Site-specific evaluation is important because, as demonstrated in our study, data quality can vary across sites. We found that data accuracy was generally high and consistent across sites; however there was some variability in completeness and accuracy across the different data fields, highlighting areas for improvement.

Based on these findings, we developed site-specific recommendations for data collection for the ongoing prospective multi-center RLS study being conducted within the PHN. For this study, PHN sites are able to utilize existing surgical registry data for data fields that have a high degree of completeness and accuracy at their site. Detailed reports and recommendations for data collection were provided to each site. For other fields not captured in the registry, or without adequate quality, data are collected by PHN coordinators.

The RLS study will be the first study to use surgical registry data for a multi-center prospective investigation in the congenital heart surgery population. Similar methods have also been used recently within the PC⁴ registry [16] in a study involving pediatric cardiac ICU patients.

Limitations

Our study was a retrospective study sampling a subgroups of patients across 12 sites. The similarity of our results to published registry audits supports the generalizability of our findings. Nonetheless, it is possible that results could differ across other sites, operations, or variables. Our study was retrospective and it is possible results may have differed if coordinator data were collected prospectively. Finally, we did not separately adjudicate all data elements and assumed that registry data in agreement with coordinator-collected data was accurate, as coordinator-collected data are the current standard for clinical trials. While it may be possible that the data may be in agreement but both sources incorrect, previous audit data suggest this is unlikely. Finally, we did not perform a detailed cost analysis, but reported data on the time required for the different data collection methods to aid in understanding potential differences in resource utilization.

Conclusions

This study suggests surgical registry data constitute a complete, accurate, and efficient information source for prospective research. The ongoing PHN RLS study will be the first example of the use of these surgical registry data in a prospective study. These methods may also be useful across other areas of pediatric cardiovascular medicine and other fields, allowing for greater research efficiency.

Abbreviations

ASO

Arterial switch operation

AVSD

Atrio ventricular septal defect

dTGA

d-transposition of great arteries

IAA

Interrupted aortic arch

IVS

Intact ventricular septum

PC⁴

Pediatric Cardiac Critical Care Consortium

PHN

Pediatric Heart Network

PI

Principle investigator

PS

Pulmonary stenosis

STS-CHSD

Society of Thoracic Surgeons Congenital Heart Surgery Database

TOF

Tetralogy of Fallot

VSD

Ventricular septal defect