The Congenital Heart Technical Skill Study: Rationale and Design

Abstract

Background:

We report the rationale and design for a peer-evaluation protocol of attending congenital heart surgeon technical skill using direct video observation.

Methods:

All surgeons contributing data to The Society of Thoracic Surgeons-Congenital Heart Surgery Database (STS-CHSD) are invited to submit videos of themselves operating, to rate peers, or both. Surgeons may submit Norwood procedures, complete atrioventricular canal repairs, and/or arterial switch operations. A HIPPA-compliant website allows secure transmission/evaluation. Videos are anonymously rated using a modified Objective Structured Assessment of Technical Skills (OSATS) score. Ratings are linked to 5-years of contemporaneous outcome data from the STS-CHSD and surgeon questionnaires. The primary outcome is a composite for major morbidity/mortality.

Results:

276 surgeons from 113 centers are eligible for participation; 77 surgeons (28%) from 48 centers (43%) have agreed to participate, with recruitment ongoing. These surgeons vary considerably in years of experience and outcomes. Participants, both early and late in their careers, describe the process as “very rewarding” and “less time consuming than anticipated”. An initial subset of 10 videos demonstrated excellent inter-rater reliability (ICC=0.85).

Conclusions:

This study proposes to evaluate the technical skills of attending pediatric cardiothoracic surgeons by video observation and peer-review. It is notable that over a quarter of congenital heart surgeons, across a range of experiences, from almost half of US centers have already agreed to participate. This study also creates a mechanism for peer feedback; we hypothesize that feedback could yield broad and meaningful quality improvement.

Mini Abstract:

We report the rationale and design for a peer-evaluation protocol of congenital heart surgeon technical skill using direct video observation. All attendings contributing data to The Society of Thoracic Surgeons-Congenital Heart Surgery Database (STS-CHSD) are invited to submit videos, rate peers, or both. Videos are anonymously rated using the Objective Structured Assessment of Technical Skills score. Ratings are linked to the STS-CHSD and surgeon questionnaires. Over a quarter of US congenital heart surgeons have agreed to participate.

Introduction:

For children undergoing congenital heart surgery, outcomes and resource utilization vary significantly across centers and among individual surgeons (1–3). Etiologies of this variation are incompletely understood and are likely multifactorial. Gaining insight into these differences requires measurement of the specific contributions of individual care elements (4).

Whereas the importance of surgeon technical skill might be assumed, the relationship between technical skill and outcomes after congenital heart surgery has not been well quantified. Previous studies have linked differences in residual lesions noted on postoperative echocardiograms to outcomes, but critics of this approach have voiced concerns about the use of such surrogates for technical skill, pointing to unmeasured differences in preoperative anatomic complexity as potential confounders (5–7). In 2013, Dr. Birkmeyer and colleagues demonstrated that outcomes after adult bariatric surgery were associated with peer ratings of video observations of operating surgeons (8, 9). Similar associations have not been tested for congenital heart surgeons.

We hypothesized that surgeon technical skill is one potential source of inter-surgeon variation in outcomes for children with congenital heart defects and that after accounting for differences in surgeon technical skill one can better understand the effects of other provider- or institution-level factors. We present here the methods, rationale, and preliminary experience for the Congenital Heart Technical Skill Study.

Subjects and Methods:

The Congenital Heart Technical Skill Study (CHTSS) was designed to assess associations between peer ratings of direct, video observation of operating surgeons with contemporaneous surgeon-specific outcomes from a national clinical registry. A secondary aim was to assess the associations between center and surgeon characteristics with patient outcomes, after adjusting for the effects of surgeon technical skill. The structured framework of an observational study allows the added benefit of providing operating surgeons with peer feedback on their technical skill.

Study Population

To determine study feasibility, beginning January 2015, all surgeons who were at least one year from terminal training and contributing data to The Society of Thoracic Surgeons-Congenital Heart Surgery Database (STS-CHSD) and The Pediatric Health Information System (PHIS) database were invited to participate, in anticipation of future investigations on resource utilization for which the PHIS database would be needed. In April 2018, to increase enrollment beyond tertiary care children’s hospitals, eligibility was expanded to include all US-based surgeons participating in the STS-CHSD, even if they do not participate in PHIS. Surgeons can choose to submit videos of themselves operating, to rate peer surgeons, or both. Providers with >15% missing data in the STS-CHSD for patient risk factors or outcomes and surgeons performing fewer than 5 of an index procedure over five years are excluded from analysis of that procedure. Surgeons are eligible even if they change institutions or performed some but not all of the studied procedures, and new surgeons recruited as they became eligible. Informational sessions were held in January 2015 and October 2016.

Video Acquisition

Video acquisition began in April 2016 and is ongoing. Each surgeon is asked to submit up to two representative videos of himself/herself or a designated trainee performing each of three operation types: 1) The Norwood/Damus-Kaye-Stansel [DKS] procedure for hypoplastic left heart syndrome, 2) atrioventricular canal repair for complete balanced atrioventricular canals [CAVCs], and 3) the arterial switch for dextrotransposition of the great arteries, with or without a ventricular septal defect. The three index procedures were selected as they allowed nearly complete visualization of the operative field via head or overhead camera in preliminary investigations and are of high technical complexity. Some surgeons allow trainees to perform all or substantial portions of the majority of their cases; these cases will be sensitivity analyses.

Cameras are turned on after patients are prepped and draped and turned off after decannulation. Videos are transmitted to the core facility via a secure, HIPPA compliant website and digitally edited by the investigators and The Comprehensive Online Archived Care Heuristic program at Columbia University (COACH; https://collectedmed.com/coachsurgery). All patient-, surgeon-, or center-identifying information is removed before any surgeon on the investigative team is allowed to see the submitted videos. Footage is then digitally edited to isolate pre-designated, key segments of the operations (Table 1).

Table 1:

Pre-designated Key Operation Segments for Evaluation in Videos by Procedure Type

NORWOOD	ARTERIAL SWICH	ATRIOVENTRICULAR CANAL
1. Cannulation/decannulation	1. Cannulation/decannulation	1. Cannulation/decannulation
2. Proximal arch reconstruction	2. Aortic transection	2. Atrial septal defect closure
3. Distal arch reconstruction	3. Pulmonary artery translocation / reconstruction	3. Ventricular septal defect closure
4. Atrial septectomy	4. Coronary artery translocation	4. Right-sided AV valve repair
5. Shunt placement	5. Neoaortic root reconstruction	5. Left-sided AV valve repair
6. Pulmonary artery reconstruction	6. Atrial septal defect closure	6. Overall operation
7. Overall operation	7. Ventricular septal defect closure
	8. Overall operation

AV=Atrioventricular

Assessment of Technical Skill

Each video is anonymously rated by peer surgeons using an instrument modified slightly from the Objective Structured Assessment of Technical Skills (OSATS) score (10) to reflect the skills of congenital heart surgeons (Table 2). The OSATS score is a structured measure of surgical skill, which has been used and validated extensively, including in adult cardiac surgery (8, 10–13). It is considered the gold standard for the assessment of surgical skill (12). In meta-analyses, inter-rater, intra-rater, inter-item, and inter-station reliability of this score have been shown to be consistently high to near-perfect across subspecialties (13), though it has never previously been tested for congenital heart surgeons. This scoring system was adopted in 2014 by the American Board of Colon and Rectal Surgery as part of board certification (14). Attending surgeon OSATS scores have been shown to correlate with patient outcomes in bariatric surgery (8). Operations are graded on five specific domains of technical proficiency: gentleness, tissue exposure, instrument handling, time/motion, and flow of operation (Table 2). Surgeons are also given a subjective summary score. Each domain is rated by a minimum of 5 surgeons using a Likert scale of 1 to 5, where 1 is supposed to be the expected skill of a starting congenital heart surgeon, 3 the expected skill of an average congenital heart surgeon, and 5 the expected skill of a master congenital heart surgeon. In addition to OSATS scores, surgeons reviewing videos are asked to submit subjective summaries of operators’ strengths and actionable areas for improvement. These summaries and scores are provided as feedback to participants. Surgeons are permitted to view, but not rate, the edited versions of their own videos.

Table 2:

Objective Structured Assessment of Technical Skill Score: 5 Specific Domains of Skill Assessment

1. Gentleness	Handling tissue appropriately with minimal damage
2. Tissue Exposure	Visualization of key aspects of operative field
3. Instrument Handling	Fluid movements of instruments, without awkwardness
4. Time/Motion	Clear economy of movement with maximal efficiency
5. Flow of Operation	Planfull and effortless flow from one movement to the next

Ratings are made on a scale of 1 to 5 (decimals are permitted):

1 = The expected skill of a starting/early career congenital heart surgeon

3 = The expected skill of an average/mid-career congenital heart surgeon

5 = The expected skill of a master congenital heart surgeon

To assess the effects of the individual components of the skill scores on outcomes, both simple and weighted averages will be tested. Correlation coefficients for the six core elements of the scores will be calculated and reported. Inter-rater reliability will be assessed, examining correlations between mean ratings of each surgeon’s videos. We will assess for rating bias by calculating z-scores for each score from each rater.

Data Sources and Data Linkage:

Patient characteristics and clinical outcome data are obtained via The Society of Thoracic Surgeons-Congenital Heart Surgery Database (STS-CHSD), the largest congenital heart surgery registry in the world. The database captures >90% of all congenital heart operations in North America, with high-level clinical granularity (15). Ten percent of contributing institutions are audited annually to ensure data accuracy and completeness. For the Norwood/DKS procedure, we include all children with hypoplastic left heart syndrome undergoing stage 1 palliation at ≤30 days of age. For the CAVC repair, all children with balanced CAVCs undergoing initial repair at ≤1 year of age are included. For the arterial switch operation, all children with dextro-transposition of the great arteries with or without a ventricular septal defect, undergoing repair at ≤30 days of age, are included; subjects with double-outlet right ventricle are excluded.

Data on provider characteristics are obtained via a surgeon questionnaire and The Pediatric Cardiothoracic Surgeon Masterfile, developed in previous investigations (16). These data include surgeons’ subjective impressions of the complexity of their cases, their seniority, technical skill, fatigue, speed, case mix, accessibility of mentors or coaches, and assistants. Surgeons also submit programmatic information, including the presence of dedicated cardiac intensive care units (ICUs) and cardiac nurses, rapid response ECMO, ICU and operating room assistants and staffing, 24-hour in-house intensivists and/or cardiologists, and details on operating room scheduling (17–19).

Technical skill scores and provider and center characteristics are transmitted to the Duke Clinical Research Institute (DCRI) via secure server and linked at DCRI by National Provider Index Number to the STS-CHSD. No raw data from the STS-CHSD is permitted to leave the DCRI.

Outcomes:

Outcomes are assessed at a patient level, using five years of contemporaneous data for each of the three designated operations, controlling for patient-, surgeon-, and center-level characteristics as well as surgeon- and center-level random effects. The primary outcome is major morbidity or mortality, using standard STS definitions (20–22). Major morbidity is defined as any of the following six major complications: 1) renal failure requiring temporary or permanent dialysis, 2) neurologic deficit persisting at discharge, 3) heart block or arrhythmia requiring a permanent pacemaker, 4) postoperative mechanical circulatory support, 5) phrenic nerve injury, and 6) unplanned re-intervention prior to discharge (20, 21). Mortality is defined as operative mortality, including all deaths, regardless of cause, occurring within 30 days of surgery or after 30 postoperative days, but before hospital discharge (23). Secondary outcomes are mortality alone and postoperative length-of-stay. In planned future investigations, we will examine resource utilization.

Protections for Human Subjects:

Surgeons are the subjects of this research. No patient identifiers are collected. As such, the Institutional Review Board (IRB) at Columbia University Irving Medical Center has assumed all responsibility for the study under Title 45CFR46, and surgeons do not need to submit applications to their individual institutions in order to participate. A handful of centers have elected to confirm this decision with their own IRBs; all IRBs have concurred.

Given the sensitive nature of the data in this study, several privacy protections have been established. Surgeons are the primary subjects of this research. All surgeons submitting videos are asked to sign written, informed consent. All videos are initially edited by non-surgeon members of the investigative team; only non-surgeons have access to unedited videos, ratings, or surgeon identifiers, and all feedback provided to participating surgeons is anonymized. Only the edited videos are stored, and all data are stored in a de-identified manner, on encrypted drives and servers. All videos will be destroyed 10 years after study completion and bivariate analyses that might in any way reveal the identity of individual surgeons will not be reported or shared. Finally, a Certificate of Confidentiality has been obtained from the National Institutes of Health. This prohibits the investigators from sharing data, even in the face of federal, state, or local subpoenas, or other forced disclosure (Public Health Service Act §301(d) and 42USC §241(d), https://www.hhs.gov/ohrp/regulations-and-policy/guidance/certificates-of-confidentiality/index.html).

Power Calculations:

Power calculations were performed based on the primary, composite outcome, major morbidity or mortality, modeled using logistic regression and a generalized estimating equations approach, accounting for clustering by institution with an exchangeable correlation structure. The primary model consisted of surgeon scores (centered and scaled). We assumed that this score is uncorrelated with other potential predictors. Thirty-three surgeons are needed to have at least 80% power to detect meaningful differences across all three operations (25 for the Norwood). For the Norwood, assuming 35 surgeons with 15 cases per surgeon over five years and an average morbidity or mortality rate of 35%, we simulated 1,000 datasets from generative models, in which a one standard deviation (SD) increase in surgeon score corresponded to a 10-percentage point decrease in major morbidity or mortality. We computed the empirical power for rejecting the null hypothesis and found: with low inter-class correlation between subjects (ICC) (0.05), power=0.978; medium ICC (0.10), power=0.925; higher ICC (0.20), power=0.779. For the CAVC repair, assuming 35 surgeons, with 15 cases per surgeon and an average morbidity or mortality rate of 10%, with a 4-percentage point decrease in the primary outcome corresponding to a one SD increase in score, we found: low ICC (0.05), power=0.880; medium ICC (0.10), power=0.834; higher ICC (0.20), power 0.677. For the arterial switch, assuming 35 surgeons with 12 cases per surgeon and an average morbidity or mortality rate of 14%, with a 5-percentage point decrease in the primary outcome corresponding to a one SD increase in score, we found: low ICC (0.05), power=0.893; medium ICC (0.10), power=0.818; higher ICC (0.20), power 0.684. With 40 surgeons submitting videos, assuming medium ICC, we would have nearly 90% power to detect meaningful differences for all three operations (96% for the Norwood).

Results:

Recruitment

One hundred and twenty-one surgeons from 43 centers met initial eligibility criteria (participating in both the STS-CHS and PHIS databases). Over the first year, an additional 15 surgeons became eligible. Of these, 54 surgeons (40%) from 28 centers (65%) agreed to participate. In preliminary analyses, using the STS-CHSD and the previously developed Pediatric Cardiothoracic Surgeon Masterfile (16), surgeons varied considerably in their years of experience (1–37 years) and outcomes (overall risk-adjusted mortality 0.0–6.3%; major morbidity or mortality 0–23.9%). Further, participants’ risk adjusted outcomes are similar to those of the average congenital heart surgeon contributing to the STS-CHSD (national average mortality 2.9%: major morbidity or mortality 11.1%, 2012–2017) (Fig. 1 and 2).

Figure 1.

Distribution of years of experience of enrolled surgeons

Figure 2.

Participating surgeons ranked by risk adjusted (A) mortality and (B) major morbidity or mortality rates

In April 2018, all congenital heart surgeons contributing data to the STS-CHSD became eligible whether or not their institution participates in PHIS. A series of individualized emails were sent to an additional 140 surgeons from an additional 70 centers. To date, 77 surgeons (28%) from 48 centers (43%) have agreed to participate, with recruitment ongoing.

Surgeon Response

Surgeons have generally expressed enthusiasm for the study. Participants across the experience spectrum have described the benefits of being able to see other surgeons’ techniques and of receiving peer feedback now that they are out of training. They have described the study as “a rare opportunity to learn new tricks and to really get better”. Frequently, participants both early and late in their careers have referred to a desire to keep learning and a dearth of opportunities to do so. As one surgeon explained, “This study provides the first opportunity for peer mentorship and coaching that many of us have not had since fellowship—early in our careers. It also provides a breadth of coaches that we would never have the chance to have in our busy lives.”

Early Surgeon Experience

Participating surgeons have generally reported video recording to be straightforward. Most had previously semi-routinely taped operations for teaching, quality improvement, and conference presentations. For some surgeons, there were initial obstacles, particularly in submitting their first videos. Several lessons were learned. There were early technical difficulties with the website. The most common issue resulted from larger than expected file sizes; some of the videos were recorded in high definition and ran for more than three or four hours, resulting in files >10GB. The platform was reconfigured to accommodate these files. Surgeons uploading videos later reported faster uploading times and fewer error messages. “Reconfiguring the platform was huge,” explained one surgeon. “There was a real learning curve there. It is still not perfect, but with the new processes in place, my job in this study is easy.”

There were surgeons who described initial nervousness at participating. For some this was because of privacy concerns. For others it was because they “overestimated the amount of time it would take” or the steps involved. Frequently, participants in the pilot phase said that they had assumed they would have to edit their own videos or “jump through IRB hurdles” (neither of which is necessary). Once surgeons submitted their first video, they typically described the process as less time consuming than anticipated. “I am super busy, as we all are,” said one surgeon. “I really wanted to participate, but I had just changed jobs and wasn’t in a position to take on anything more. I decided I would send just one video so that I could feel like I was doing my part, because I think the study is important. Once I sent the first case, I realized it only took about 10 minutes of my time and I decided I could send more.”

Some surgeons used research coordinators or assistants to remind them to turn on their cameras and to upload videos, thereby reducing the time burden. “I just had to remember to turn on my camera!” said one surgeon. “I kept forgetting at the beginning. The investigators offered to send reminders to my scheduler. My scheduler then started reminding me before I went into the operating room. From there it was easy. I just taped the case and handed it off to my research coordinator and there was nothing left for me to do.”

Video Editing and Rating

To date, 65 videos have been received from 16 surgeons at 14 different centers. On average, edited videos are 35 minutes (CAVCs) to 50 minutes (arterial switches and Norwoods) in length. Participating surgeons watching their own operations uniformly described the edited videos as accurate representation of their technical skill: “I quickly recognized my own operation and I was very impressed with the editing. I had been worried that some minor mistake would look like a major event. I was pleased that the surgery appeared to be an appropriate assessment of my performance.”

In preliminary work, an initial subset of ten videos was edited. Inter-rater reliability was very high (ICC=0.85). Most surgeons reported watching videos at 3–4 times speed, slowing down only for key segments and rating simultaneously. This reduced average rating times for most surgeons to approximately 10–15 minutes per video. Raters reported the scale and computer interface to be easy to use and the process as fun and educational. “I’ve personally enjoyed reviewing the videos I have seen (and perhaps need reminders to look at more)”, explained one surgeon. “I have seen several techniques that I don’t use. Now that I have seen them, I would feel comfortable using them if the case required.”

Comment:

In this national investigation, we use direct video observation and peer evaluation of congenital heart surgeon technical skill for the purposes of assessing associations between surgeon technical skill and outcomes for children undergoing congenital heart surgery and associations between other surgeon- and center-characteristics and outcomes after controlling for the technical skill of the surgeon. This study also has a secondary benefit of providing surgeons with an opportunity to receive feedback from a random selection of peer surgeons. We report here the methods and rationale and the preliminary experience with this ongoing investigation. The congenital heart surgical community has demonstrated enthusiasm for this effort, with nearly one quarter of operating congenital heart surgeons now participating, across a broad range of experience levels. As the result of participant feedback from the early portion of the study, we have been able to modify participant supports to make participation less time consuming. Participants have uniformly described their experience as positive and educational.

It has been long recognized that multiple factors, both patient-specific and system-related, impact the outcomes of surgical management of congenital heart disease (4). Studies from individual institutions, consortia, and databases have attempted to characterize patient factors that impact the outcome of surgical management. System-specific factors are more difficult to quantify and understand. For example, prior reports have demonstrated center-level volume-outcome relationships, at least for high complexity congenital heart surgery (3, 24). The exact reason for this variation in outcomes has not been elucidated. It has been speculated that this relationship may result from an institutional capability to handle the different aspects of the complex care these patients require. Implicit in this institutional capability argument is the skill of providers. Especially in a technically challenging field such as pediatric cardiac surgery, the technical ability of is likely important. Yet the current ability to quantify technical skills and provide feedback to surgeons is limited (4–7).

In 2008, the “Technical Performance Score” (TPS; now the Residual Lesion Score, RLS) was developed for congenital heart surgery (8). This score measures residual, anatomic lesions on postoperative echocardiograms and has been shown to correspond with clinical outcomes in multiple studies (16–18). A multi-center investigation is currently underway to further validate this score (7, 19). While this score is easily measurable, has been shown repeatedly to be associated with patient outcomes, and is highly useful in guiding postoperative counseling for families, it is an indirect measure of technique and does not account for differences in preoperative patient anatomic complexity.

The inability to evaluate technical skill is not new or exclusive to the field of pediatric cardiac surgery. The current paradigm of training surgeons and interventionalists throughout the world rarely includes objective assessments of trainees’ technical capabilities. Mechanisms for certification and credentialing typically test didactic and clinical knowledge of trainees, but leave technical training and evaluation to the discretion of the training programs. There is a nationwide effort to establish mechanisms for quantifying and grading the surgical ability of trainees and to establish benchmarks commensurate with their level of training. Some of the methods being evaluated include direct observation in the operating room, performance at simulated stations, and evaluation of recorded surgical procedures; in 2014, the rating methodology utilized in this study was adopted by the American Board of Colon and Rectal Surgery as part of their board certification (14). For congenital heart surgery, however, no formal system currently exists.

This study evaluates the technical skills of attending pediatric cardiothoracic surgeons by direct, video observation and a peer-review mechanism. It is exciting to note that over one quarter of congenital heart surgeons operating in the US from nearly half eligible centers have agreed to participate in this study, suggesting a robust interest and even enthusiasm in the congenital heart community for critical assessment of current performance, as well as an eagerness to improve. In spite of early technical difficulties, we were able to successfully leverage current technology to seamlessly receive and rate procedures across the continental United States. The process of editing the raw videos was undertaken by a centralized editorial office to easily and uniformly ensure that key points of the surgery are captured and the entire surgical procedure is abridged to a manageable time frame. We submit that these steps, in and of themselves, represent major milestones. These edited videos of surgical procedures will allow us to, first, evaluate associations between technical skill and patient outcomes, and then associations between other surgeon and center characteristics and outcomes, after controlling for the skill of the surgeon. This also affords the ability to provide direct feedback to operating surgeons and for surgeons to learn by watching their peers. We believe that receipt of peer feedback is an important and gratifying repayment to participating surgeons for their efforts in the study, and has the potential to become a method for broad, uniform, and sustained quality improvement efforts in technical aspects of congenital heart surgery.

Our study has some important limitations. First, this study was initially only open to surgeons operating at centers that participate in both the STS-CHSD and PHIS databases. This skewed enrollment to large centers. To address this bias, in April 2018, enrollment was opened to all surgeons participating in the STS-CHSD. Second, participation in the study is voluntary. While preliminary data suggests that participation has not been limited to surgeons with the lowest risk-adjusted outcomes, it is possible that surgeons who question their own technical skill might be less inclined to submit videos. This would decrease the inter class correlation in technical skill scores between surgeons. Third, we have chosen to record only three pediatric cardiac surgical procedures. These procedures are intended to be representative of the more technically challenging operations, but that premise is open to interpretation. Fourth, participating surgeons are allowed to choose which cases they wish to submit for evaluation. Surgeons are asked to select cases they view as “representative of their typical work” and are, in fact, instructed not to submit cases they feel go particularly poorly (or well). This allows for the possibility that some surgeons might attempt to “cherry pick” their best cases. And finally, the impact of assistants during surgery will need to be further explored, as some attending surgeons routinely allow their trainees to perform the majority or all of certain operations or performs complex cases with the assistance of another second attending. Surgeons are allowed to submit videos in which they assist trainees in performing the procedures or in which they are assisted by another congenital heart surgeon.

Despite these limitations, we believe that our study is an important first step in trying to objectively assess the impact of technical skills in pediatric cardiac surgery. Our preliminary progress shows that such an effort is feasible. Our ultimate goal is to show that such an effort is meaningful and reproducible and rewarding for both participants and their future patients.