Challenges and lessons learned from the Pediatric Heart Network Normal Echocardiogram Database study

Abstract

Background:

The Pediatric Heart Network Normal Echocardiogram Database Study had unanticipated challenges. We sought to describe these challenges and lessons learned to improve the design of future studies.

Methods:

Challenges were divided into three categories: enrolment, echocardiographic imaging, and protocol violations. Memoranda, Core Lab reports, and adjudication logs were reviewed. A centre-level questionnaire provided information regarding local processes for data collection. Descriptive statistics were used, and chi-square tests determined differences in imaging quality.

Results:

For the 19 participating centres, challenges with enrolment included variations in Institutional Review Board definitions of “retrospective” eligibility, overestimation of non-White participants, centre categorisation of Hispanic participants that differed from National Institutes of Health definitions, and exclusion of potential participants due to missing demographic data. Institutional Review Board amendments resolved many of these challenges. There was an unanticipated burden imposed on centres due to high numbers of echocardiograms that were reviewed but failed to meet submission criteria. Additionally, image transfer software malfunctions delayed Core Lab image review and feedback. Between the early and late study periods, the proportion of unacceptable echocardiograms submitted to the Core Lab decreased (14 versus 7%, p < 0.01). Most protocol violations were from eligibility violations and inadvertent protected health information disclosure (overall 2.5%). Adjudication committee reviews led to protocol changes.

Conclusions:

Numerous challenges encountered during the Normal Echocardiogram Database Study prolonged study enrolment. The retrospective design and flaws in image transfer software were key impediments to study completion and should be considered when designing future studies collecting echocardiographic images as a primary outcome.

The Pediatric Heart Network designed the Normal Echocardiogram Database Study to retro spectively collect demographic and echocardiographic data from healthy, non-obese children who had clinically indicated echocardiograms that were normal. The goal was to develop a robust z-score database that accounted for age, race, and gender.¹ The study aimed to overcome limitations of previous echocardiographic z-score databases, including small sample size from single institutions, non-standardised methods for performing and normalising measurements, and limited assessment of the potential effects of race and gender. Despite the detailed study design and considerable time spent planning the study, we encountered unexpected challenges in its implementation. Therefore, we sought to examine the lessons learned that may improve future investigations by identifying challenges encountered with the study design and implementation; describing solutions to these challenges; and describing the adjudication process used to resolve disputes concerning patient eligibility.

Materials and methods

The design and main results of the Pediatric Heart Network Normal Echocardiogram Database Study were previously published.¹ Briefly, the database was a retrospective collection of demographic and echocardiographic data obtained from healthy North American children ⩽18 years old from 19 congenital cardiac centres (10 Pediatric Heart Network core centres and 9 auxiliary centres) between April, 2013 and October, 2015. The targeted enrolment for the study included 3600 participants, divided into 36 groups, stratified by age (6 groups), race (White, African-American, Other), and gender, with ⩾80% of echocardiograms submitted expected to have all the necessary images to allow for a minimum of 80 echocardiograms per group to be analysed based on power calculations.¹ When each of the 36 groups reached the target number, it was closed for further enrolment. Medical records were first reviewed to ensure that all participants had a normal medical and family history. The echocardiograms of eligible participants were then reviewed locally to document that acquired images containing the required echocardiographic elements. For echocardiograms to be included into the database, the study had to have 19 imaging elements (left ventricular dimensions and volumes and all cardiac valvular dimensions) and at least 1 of the 3 additional anatomic structures (diameters of the pulmonary arteries, aortic arch, and/or coronary arteries, Table 1). Echocardiograms deemed acceptable locally were uploaded to the digital transmission software platform by the submitting centre, de-identified by the software, and transmitted digitally to the Core Pediatric Echocardiography Research Lab (Medical College of Wisconsin) for measurements; it was the responsibility of the centre’s principal investigator to verify a study was de-identified prior to forwarding an echocardiogram to the Core Lab. The Core Lab provided monthly feedback on imaging quality. If the Core Lab found the imaging was incomplete or inadequate for measurement, details justifying exclusion were recorded and sent to the centre. Results were reported for the 3215 echocardiograms analysed in the main study.¹

Table 1.

Mandatory and secondary imaging elements.

Mandatory (19)	Secondary*
Parasternal or subxiphoid short axis	Main and branch pulmonary arteries
LV wall thickness, systole and diastole	Main pulmonary artery diameter
LV cavity diameter, systole and diastole	Right pulmonary artery diameter
LV endocardial short-axis dimension, systole and diastole	Left pulmonary artery diameter
Parasternal long axis	Aortic arch and isthmus
Aortic valve annulus diameter	Proximal arch diameter
Aortic root diameter	Distal arch diameter
Aortic sinotubular junction diameter	Isthmus diameter
Ascending aorta diameter	Proximal coronary arteries
Mitral valve anteroposterior annulus diameter	Right coronary artery diameter
Tricuspid valve anteroposterior annulus diameter	Left main coronary artery diameter
Parasternal long or short axis	Left anterior descending diameter
Pulmonary valve annulus diameter
Apical four-chamber
LV endocardial long-axis dimension, systole and diastole
LV epicardial long-axis dimension, systole and diastole
Mitral valve lateral annular diameter
Tricuspid valve lateral annular diameter

LV = left ventricle.

^*For secondary imaging elements, each echocardiogram had to contain all three measurements from only one of the groups.

The Pediatric Heart Network added auxiliary centres to the core centres to decrease enrolment time and increase the generalisability of the results. The request for applications was sent to academic paediatric echocardiographic laboratories throughout North America. Digital Imaging and Communications in Medicine image capture and the routine use of a minimum of two-beat clips for image acquisition were required. Interested centres provided the following data as part of their application: number of echocardiograms performed annually; local technology available for the transfer of echocardiographic images; imaging elements captured in their standard complete echocardiogram; number of beats captured/clip; and proportion of complete studies with weight, height, gender, age, race, and ethnicity captured in either the medical record or the echocardiographic report (Data supplement, Online Appendix 2). In addition, each centre submitted six de-identified, retrospective, complete echocardiograms from six normal children (one from each of the targeted age categories). Of the 19 centres applying to be an auxiliary centre, 9 were selected and added to the 10 core centres, allowing recruitment from a total of 19 centres. Pediatric Heart Network core centres were not subject to the selection process.

The Institutional Review Board at each centre approved the original project. Informed consent was waived for this study.

Definitions of categories of challenges

We divided challenges encountered with design and implementation into three categories: enrolment; echocardiographic imaging; and protocol violations. Technical issues delayed image transfer to the Core Lab resulting in delays in the Core Lab review of echocardiograms. Therefore, we assigned different time periods to allow us to assess challenges with enrolment as well as challenges with echocardiographic imaging quality. When referencing challenges encountered with enrolment, year 1 was defined as April, 2013 (when the first patient was enrolled) through April, 2014; years 2 and 3 were defined as May, 2014 through October, 2015 (when the last patient was enrolled). When referencing challenges encountered with echocardiographic imaging quality, the early period was defined as July, 2013 (when the first echocardiogram was reviewed by the Core Lab) through July, 2014 and the late period is defined as August, 2014 through January, 2016 (when the last echocardiogram was reviewed by the Core Lab) (Fig 1).

Figure 1.

Schematic of how the time periods for challenges with enrolment and echocardiographic imaging quality were defined.

Memoranda and meeting minutes, Core Lab reports, and adjudication logs were reviewed. Screening logs were not required as part of the original study design and, thus, not available at all centres, so a centre-level questionnaire (Data supplement, Online Appendix 3) was developed to better understand local processes for identification of potential patient; centre-specific issues with enrolment; and local institutional review board processes, including amendments and approval dates. The questionnaire was developed after closure of study enrolment and was completed by all 19 centres.

Statistical analysis

Descriptive statistics were used to describe most of the challenges. A chi-square test was used to determine if there was a significant difference in the assessment of echocardiographic acceptability and grading of echocardiograms between the early and late periods.

Results

Challenges with enrolment

We identified four main challenges with enrolment. First, local institutional review boards varied in their interpretation of “retrospective” enrolment. The Normal Echocardiogram Database Study had a retrospective cohort design, with the intent that all healthy participants who had a normal echocardiogram performed solely for clinical reasons and met demographic and imaging inclusion criteria would be eligible for enrolment regardless of the date of the echocardiogram. However, 7/19 centres (37%) required an institutional review board amendment to enrol participants whose echocardiograms were performed after the study launch date, and one centre had their amendment denied and could not enrol participants who had an echocardiogram performed after the study launch date. The time between submission and approval of the amendments by the institutional review boards averaged 30 days (range from 1 to 103 days).

Enrolment was also impacted by overestimations of the number of non-White children available for study participation. The Normal Echocardiogram Database Study was designed to enrol equal numbers from the White, African-American, and Other race categories. However, the 2010 United States census reported African-American and Other race categories comprised only 13 and 15% of the US population, respectively. This discrepancy led to under-enrolment of non-White participants: by the end of the first year of enrolment (April, 2013–April, 2014), 1142/1277 (89%) of White participants were enrolled compared to only 550/1114 (49%) of African-Americans, and 201/1177 (17%) of participants from the Other race category (Fig 2). To increase enrolment of the non-White groups, the Pediatric Heart Network amended the protocol in August, 2015, allowing centres with locally available funds to focus on prospectively enrolling non-White participants. This amendment led to 67 participants (32 and 35 participants in the African-American and Other race categories, respectively) being added to the database.

Figure 2.

Enrolment by race in year 1 versus years 2 and 3.

The third challenge addressed discrepancies in the definition of children of Hispanic descent. The National Institutes of Health defines Hispanic as an ethnicity rather than a separate race category, and this definition was used in the development of the protocol. It quickly became problematic, however, as 10 centres (53%) defined Hispanic either as a race or as an ethnicity alone without designation of a separate race category. This discrepancy led to the unanticipated exclusion of >286 potential participants (some centres did not track these data, so the number is underestimated). The study committee overcame this challenge by amending the protocol to enrol Hispanic participants without a separate race designation into the Other race category.

Finally, we did not anticipate the large number of potential participants who were excluded based on clinical criteria, such as absent indication for the echocardiogram or body surface area out of the normal range, or because of incomplete demographic data from the medical record. Screening logs tracking reasons why participants were excluded were kept at 13/19 centres (68%). Of 13,612 potential participants screened at these 13 centres, 8991 (66%) were excluded based on clinical exclusion criteria or incomplete demographic data alone (Fig 3).

Figure 3.

Breakdown of potential participants. Data from the screening logs of 13/19 centres: Of 13,612 potential participants screened, 66% were excluded due to demographic inclusion/exclusion criteria (large, top circle). Of those participants whose echocardiograms were screened locally, 56% were excluded due to imaging quality, with 44% transmitted to the Core Lab (middle circle). With the addition of the echocardiograms sent to the Core Lab from 6 centres with no screening log data, the Core Lab reviewed 3566 echocardiograms, and 90% were included in the analysis (bottom circle).

Challenges with echocardiographic imaging quality

Another major obstacle to completion of the study was the large number of eligible participants whose echocardiograms did not have the required imaging elements for measurements in the Core Lab. All centres had designated paediatric cardiologists to review all echocardiograms for eligible participants, and 7/19 (37%) centres had the studies previewed by a paediatric cardiac sonographer or research coordinator. Of the 4621 echocardiographic studies screened for the protocol-mandated measurable images at the 13 centres that maintained screening logs (thus, an underestimate of the actual number of echocardiograms excluded locally based on imaging quality at all 19 centres), 2591 (56%) were excluded locally because of inadequate or incomplete imaging (Fig 3), leaving only 2030 (44%) to be forwarded to the Core Lab from 13 centres. Despite the strict local screening process documented at these 13 centres, a review of the Core Lab’s records of echocardiograms submitted from all 19 centres showed that 351/3566 (10%) of submitted echocardiograms were considered unacceptable and excluded (Fig 3). The selection of echocardiograms deemed acceptable by the Core Lab appeared to improve over time with 212/1524 (14%) rejected in the early period echocardiographic review by the Core Lab (July, 2013–July, 2014) versus 139/2042 (7%, p<0.01) in the late period (August, 2014–January, 2016). To provide feedback on image quality to the centre, the Core Lab graded acceptable echocardiograms as “excellent,” “good,” or “fair”. Between the early and late periods, the percentage of echocardiograms graded as “excellent” decreased (56 versus 28% respectively, p < 0.01), while the percentage graded “good” increased (43 versus 70%, respectively, p<0.01). The percentage of echocardiograms graded as “fair” imaging quality remained similar between the early and late periods.

Digitally transmitting echocardiograms from the centres to the Core Lab were initially challenging. Because of software malfunctions with the echocardiographic transmission platform, particularly in de-identifying protected health information as part of the transfer process, the Core Lab was unable to review echocardiograms for 3 months after the study launch. By the end of the first year of enrolment (April, 2013–April, 2014), only 953/1893 (50%) of echocardiograms submitted to the Core Lab could be reviewed based on Core Lab records, delaying feedback regarding image quality. Failures in the de-identification process of the system also contributed to protocol violations (Fig 4).

Figure 4.

Types of protocol violations occurred. PHI= Protected health information; IRB = Institutional review board.

Challenges with protocol violations

A total of 81 protocol violations occurred in 2.5% of participants in the Normal Echocardiogram Database Study (Fig 4). The majority (67/81, 83%) were classified as major protocol violations and were nearly evenly split between patient eligibility and loss of confidentiality. Most violations for eligibility involved participants who were outside the designated exclusion values for body size or age. Overall, protocol violations related to breaches in confidentiality were rare (<1% of all participants) and included protected health information present on source documents or E-mail (n = 17) and echocardiograms that were incompletely de-identified (n = 14). All minor protocol violations were due to enrolment of patients in age groups that had already been capped (n = 14).

Adjudication process

The Pediatric Heart Network developed an adjudication process for investigators who had questions regarding a potential patient’s eligibility prior to data submission and for those who challenged the Data Coordinating Center’s determination of patient eligibility based on demographic criteria or the Core Lab’s inability to make the required measurements. The five-member Adjudication Committee received 55 questions from the centres regarding potential patient eligibility, including 32 based on patient demographics and 23 based on echocardiographic imaging. There were 15 adjudication requests for re-review of echocardiographic studies, 6 originating from the Core Lab and 9 from the centres. The Core Lab requested determination of specific echocardiographic findings as normal variants versus abnormal. Of these six requests, four were included in the final analysis as normal variants and two excluded as abnormal. Of the nine requests for re-review from the centres, four studies were ultimately included as measurable and five remained excluded. Echocardiographic variants adjudicated as normal variants in contrast to abnormalities sometimes altered the inclusion/exclusion criteria. For example, anomalies of brachiocephalic vessel origins from the aortic arch were on the original list of echocardiographic exclusion criteria. However, given the number of otherwise eligible African-Americans being excluded based solely on this variant, the Adjudication Committee members agreed that anomalies of brachiocephalic origins such as common origin of the right innominate and left carotid arteries could be included. Other abnormalities such as a vertebral artery arising from the aortic arch continued to be excluded because of the potential to alter flow patterns and affect arch growth. The protocol and local worksheets were amended to reflect these changes.

Discussion

We summarised the unanticipated challenges encountered during the conduct of the Pediatric Heart Network Normal Echocardiogram Database Study and the strategies to address them to inform the design of future studies.

The retrospective study design failed to account for centre variations in practice in several areas. Local institutional review boards maintained autonomy and guided their centre’s conduct of the study allowing the investigators little influence over their decisions. As a result, retrospective enrolment had a variety of definitions and requirements for enrolment after study launch. Future studies may benefit from the use of a central institutional review board with consistent definitions and risk/benefit assessments across centres.^2,3 A prospective design would also overcome this challenge, but would require informed consent and may increase the accrual time and study costs.

The retrospective design also contributed to the large number of echocardiograms excluded because of incomplete acquisition of the required imaging elements. We do not know the exact number of echocardiograms reviewed, since screening logs were not required in the study design. However, centres with screening logs reported over half of the echocardiograms from eligible participants were rejected because the mandatory imaging elements had not been captured or were not clear enough to be measured. The American Society of Echocardiography published recommendations aimed at providing a standardised framework and platform of how measurements should be obtained,⁴ but stopped short of recommending a “standard measurement package” for paediatric echocardiograms. As a result, fewer than half of the normal studies had imaging elements that allowed basic measures to be performed, exposing the need to develop standard imaging recommendations for measurements in all age groups.

Other factors likely contributed to the high number of echocardiograms rejected from submission to the Core Lab. Experienced echocardiographers can often qualitatively assess chambers and valves as normal or abnormal, but this strategy is clearly different from image capture fidelity that allows precise placement of callipers for accurate measurements. This is particularly true for a child who has difficulty cooperating with the echocardiogram or the child with poor acoustic windows. In these cases, images may be adequate for qualitative assessment but inadequate for actually making objective measurements. Although auxiliary centres were selected largely on the completeness and quality of their submitted studies, the centres may have “cherry-picked” their best examples. Core centres were not vetted in a similar manner, and routine clinical omissions of study-mandated views were not discovered until the project was well underway. Given the retrospective nature of the study, centres did not have the opportunity to go back and obtain these views and some could not modify their lab standards to include them during the study period. We considered changing the design to decrease the required imaging elements in the study protocol. For example, rather than requiring 100 normal echocardiograms with all the mandatory imaging elements, we could have collected 100 measurable images for each element, regardless of the number of echocardiograms required. This was ultimately rejected as it required upload, transfer, download, and review of more echocardiograms, increasing total time and net study costs. Going forward, a prospective study with training for the imaging protocol would better ensure that all imaging elements were measurable.

The National Institutes of Health definition of Hispanic as an ethnicity to be accompanied by a racial designation⁵ was problematic, as the definition does not appear to be fully aligned with societal norms.⁶ Most centres listed Hispanic as a race or, if they listed it as an ethnicity, did not list an accompanying race. The discrepancy between the actual operation of health care systems and National Institute of Health definitions should be considered in future studies where assessment of race and ethnicity categories relies on the medical record without interaction with study participants. Similarly, specific National Institutes of Health-defined racial categories may be better guided by U.S. Census data rather than investigator estimates of the racial distribution of the local population. If equal enrolment of non-White and White participants is a targeted goal of a study, the study design should allow a longer enrolment period and recruit centres with higher minority populations.

Despite Core Lab assessment of echocardiographic imaging quality, we were unable to determine if it actually improved during the study. The significant reduction in the number of echocardiograms deemed unacceptable by the Core Lab over time may have resulted from an actual improvement in imaging quality, but better selection or discrimination of the echocardiograms being reviewed locally prior to transmission to the Core Lab could also be a factor. Because of study fatigue or even increased experience with obtaining measurements, the Core Lab may have measured less optimally imaged cardiac structures over the course of time. The reduction in the proportion of echocardiograms graded as “excellent” and increase in the proportion graded as “good” may reflect this drive to “acceptability” of an echocardiogram, without improvement in actual imaging quality. A prospective design would better address improvement in echocardiographic image quality over time.

Despite the numerous challenges that were encountered, the study completed enrolment in all 36 categories, with the data successfully analysed and disseminated.¹ Given the many unexpected challenges during study implementation, frequent feedback and adjudication were key to the success of the study. The study principal investigators and local study teams worked closely to troubleshoot local issues that arose and to disseminate the agreed upon amendments. Timely communication among the Core Lab, data coordinating centre, Adjudication Committee, and the centre Pis allowed quick resolution of conflicts and uniform application of inclusion/exclusion criteria.

In summary, numerous unexpected challenges occurred during the conduct of the Pediatric Heart Network Normal Echocardiogram Database Study. In particular, the use of a retrospective design and ineffective digital imaging transfer software were significant impediments to study completion. Although a prospective study design may have prevented some of the challenges, the effect on the duration of study enrolment, investigator time, and cost is unknown. We present the lessons learned from this study to inform the design and implementation of future studies.