Prerandomization Surgical Training for the National Surgical Adjuvant Breast and Bowel Project (NSABP) B-32 Trial

Abstract

Objective:

To train surgeons in a standardized technique of sentinel lymph node biopsy and to prepare them for the requirements of a prospective randomized surgical trial.

Summary Background Data:

The NSABP B32 trial opened to accrual in May 1999. A significant component of this trial was a prerandomization training phase of surgeons performed by a group of core surgical trainers. The goals of this training phase were to expeditiously instruct surgeons in a standardized technique of sentinel lymph node biopsy and to educate those same surgeons in complete and accurate data collection and source documentation for the trial.

Methods:

This study is a description of the training data collected in a prospective fashion for the training component for surgeon entry into the B32 trial, evaluating the effectiveness of the training program in regards to surgical outcomes and protocol compliance.

Results:

Two hundred twenty-six registered surgeons underwent site visit training by a core surgical trainer and 187 completed training and were approved to randomize patients on the trial. The results of 815 training (nontrial) cases demonstrated a technical success rate for identifying sentinel nodes at 96.2% with a false negative rate of 6.7%. A protocol compliance analysis, which included the evaluation of 94 separate fields, showed mean protocol compliance of 98.6% for procedural fields, 95.5% for source documentation fields and 95.0% for data entry fields.

Conclusions:

This training and quality control program has resulted in a large number of surgeons capable of performing sentinel lymph node biopsy in a standardized fashion with a high degree of protocol compliance and pathologic accuracy. This will ensure optimal results for procedures performed on the randomized phase of the trial.

A prerandomization training program was employed for surgeons participating in the National Surgical Adjuvant Breast and Bowel Project B-32 (NSABP B32) trial. This training program is reviewed and its effectiveness in regards to surgical results and protocol compliance on the training cases performed by these surgeons is reported.

Sentinel node biopsy (SNB) is a minimally invasive procedure that allows the surgeon to accurately remove the set of lymph nodes that first receive drainage from a primary tumor. This minimally invasive procedure is an attractive alternative to conventional axillary node resection (ANR) because it results in removal of less node-bearing tissue and because tumor-bearing lymph nodes can be identified if they are located outside the axilla. The reasons for performing regional node resection in breast cancer are to maximize survival, provide long-term regional disease control, and to obtain important staging information. Collectively, randomized trials comparing ANR to omission of ANR indicate a survival advantage associated with ANR of 5.4% (95% confidence interval = 2.7% to 8.0%).¹ Modern reviews of large sets of breast cancer patients also show survival reduction associated with omission of ANR.² The association of decreased survival with omission of ANR has been noted even with early (T1a-b) stage breast cancers.³ The survival outcome associated with SNB and omission of ANR in direct comparison to ANR is unknown.

The National Surgical Adjuvant Breast and Bowel Project clinical trials group (NSABP) in partnership with University of Vermont is conducting a randomized clinical trial, B-32, to determine whether SNB alone will provide the same survival and long-term regional disease control as ANR in clinically node-negative breast cancer patients.⁴ The primary endpoints of the B-32 trial are survival, regional control, and morbidity. To detect a possible survival difference as small as 2% between the 2 randomized groups, about 5600 patients will be enrolled.

Several different techniques are available for performing SNB in breast cancer patients and there is no consensus as to the optimal method. To minimize the potential negative impact that different SNB techniques could have on the interpretation of outcomes of the B-32 trial, a single standardized method of SNB was chosen. Our preliminary experience conducting a multicenter validation sentinel node study demonstrated that with direct intraoperative instruction, a group of surgeons could achieve overall SNB technical success rates and false-negative rates consistent with the overall international experience.⁵ We also learned that the process of generating source documentation and capturing real-time elements of the surgical procedure into a field-based data collection form were activities not familiar to all surgeons. Data from intraoperative procedures have unique issues compared with studies involving systemic or radiation therapy. This is because medical records related to drug administration or radiation therapy are well established and are often entered in a redundant manner, making this information readily accessible. Events occurring during surgery, however, are transient, and generally documentation is nonredundant. If the information is not in the dictated operative note, it simply becomes unavailable as source documentation. Completeness and accuracy of data entry in a clinical trial are just as important to outcome analysis as the actual performance of the surgical procedure.

A training and quality-control program was established for surgeon entry into the B-32 trial that focused both on performance of the SNB procedure as well as the methods of recording data and generating source documentation accurately and completely. The goals of the training process were to insure that surgeons performed sentinel node surgery according to specific guidelines used in the protocol, pathologists performed node evaluation according to guidelines listed in the protocol, source documents were appropriately generated, and information was accurately and completely recorded on the surgical pathology data forms. Source documentation was defined as trial-relevant information present in a permanent legal medical record. An important element of this program was a core group of surgeons from across the country that performed site visits to participating centers. This group of surgeons provided personal training and quality control in the performance of SNB, generation of source documentation, and data entry. To standardize training, each core trainer was accompanied on his or her first site visit with either the PI (Krag) or the protocol Training Chair (Harlow). This manuscript is an initial presentation and analysis of that training process.

METHODS

The overall training process, which was required before a surgeon could enter into the actual phase III NSABP B32 trial, included a training manual, a site visit, and follow-up of 5 prerandomization SNB procedures. Surgeons at NSABP affiliated institutions first registered for participation in the B-32 trial. The training manual was provided to the surgeon and a core trainer was assigned. The training manual included the trial protocol, educational material detailing the critical steps in performance of the SNB, a pathology section on processing the sentinel nodes, and a description of how to generate and complete study-specific source documentation.

During the site visit, the core trainer provided intraoperative instruction to the participating surgeon in a sentinel node surgery case. In addition, instruction was also provided to nonsurgical personnel including nursing, nuclear medicine, pathology, and data management. The training case allowed a hands-on opportunity for instruction to highlight the specific steps for SNB according to those outlined in the protocol. Each surgeon was instructed in the data collection that was required in the operating room and its appropriate documentation on the data collection sheets. During this phase, each surgeon performed a minimum of 5 training cases before being allowed to randomize patients on the B-32 trial. These training cases were expected to be performed according to protocol guidelines and were followed by a completion axillary node dissection. The data from each training case were reviewed at the University of Vermont. Each data set included operative reports, pathology reports, the Surgical-Pathology data form, and a procedural checkoff list. The procedural checkoff list included the important steps of the protocol that provided a step-by-step guide for the surgeon. These cases were evaluated for protocol compliance, source documentation, and accuracy of data entry on the forms. The cases were reviewed in detail by telephone conference between the core trainer and surgeon. After successful completion of this process, the surgeon would be registered to begin entry of randomized cases. Surgeons that did not successfully complete this process were asked to perform additional training cases.

The NSABP B-32 trial opened in May 1999. As of February 2003, 226 surgeons registered for participation and underwent site visits by a protocol core trainer. Of this group, 187 surgeons have completed their training and have been approved to randomize patients on the trial. Initially, the procedure for training case review was to have surgeons submit all 5 training cases, as a set, to the University of Vermont Training Center for review. This method of review was done for approximately 6 months and included 56 (56/187) of the approved surgeons. This process was altered after this period (post-November 1999), when we realized the process could be made more efficient by having surgeons submit their training cases for review as each case was completed. The individual cases were promptly reviewed, providing immediate feedback to the surgeon. This change was instituted to decrease persistent errors that might potentially occur in all 5 training cases.

After November 1999, we began to prospectively collect detailed information of the individual case reviews to allow a more careful evaluation of the training methods and the outcomes. During this post-November 1999 time period, 131 surgeons (131/187) completed all training and were approved to randomize patients. A detailed data set was obtained for 119 surgeons (complete data on 4 or more training cases) and these data are the basis for the majority of the compliance analysis in this manuscript.

The protocol compliance analysis involved evaluation of 94 specific fields for each case. The fields were grouped into the following categories: procedural (25 fields for the surgical procedure), source documentation (14 fields from the operative note and 11 from the pathology report), and data entry (44 fields from the Surgery-Pathology data forms). Any error, combination of errors, or omissions found in a screened field were recorded as an unweighted error occurring in that field. The fields were scored as having an error if the available source documentation did not include information that supported the information in the data-form field. In most cases, when performance errors were recorded, the actual procedure had been performed correctly based on the telephone review with the surgeon. For example, it was standard protocol practice to inject a defined volume of tracer; however, if this was not documented in the operative note, the tracer volume field was scored as incorrect.

Procedural success rates were available for the entire cohort of surgeons that completed training or were still in the training process. The results of 815 training-only cases were entered onto a computerized database and were available for determination of sentinel node identification rate and pathologic false negative sentinel node rates.

Statistics

In the protocol compliance analysis, the reported 95% confidence intervals used individual surgeons as the unit of analysis to describe compliance. The Wilcoxon rank sum test was used to compare surgeons needing 5 training cases versus >5 training cases to qualify for randomization, with a 5% significance level.

RESULTS

Based on the 815 training (nonprotocol) cases in our database, the overall success rate for removal of sentinel nodes was 96.2%, and the false-negative rate was 6.7%.

In the subgroup of surgeons who completed all training and were approved to randomize, 70.6% (132/187) were approved after the minimum number of 5 training cases, and 29.4% (55/187) were approved after completing more than 5 training cases.

The success of the type of review process (review after completion of 5 cases versus case-by-case review with feedback during training case accrual) was evaluated by determining the percentage of surgeons that were successfully approved to randomize after completing the minimum number of 5 training cases. For the 56 surgeons that had cases evaluated after completion of the 5-case training set, 48% were approved to randomize after completion of the minimum number of 5 training cases. For the 131 surgeons that had training cases evaluated case by case, 80% were approved to randomize after the minimum number of 5 training cases. This indicates a significant improvement in the efficiency of training as the result of the case-by-case screening process with immediate feedback.

The results of the detailed training case protocol compliance analysis for 119 surgeons are shown in Tables 1, 2, and 3. The overall compliance rate for all fields for all surgeons was quite high. The mean compliance for the 25 fields related to source documentation was 95.5%, with a range of 89.8% to 100% (Table 1). The mean compliance rate for the 25 procedural fields was 98.6% with a range of 94.6% to 100% (Table 2), and the mean compliance rate for the 44 data entry fields was 95.0%, with a range of 86.5% to 99.9% (Table 3).

TABLE 1. Compliance Analysis of Training Cases for Source Documentation Fields

TABLE 2. Compliance Analysis of Training Cases for Procedural Related Fields

TABLE 3. Compliance Analysis of Training Cases for Data Entry–Related Fields

The mean overall compliance values of surgeons who completed their training in 5 cases (5-case group) versus those who completed training in more than 5 cases (>5-case group) were quite similar. The results of this comparison showed that for the source documentation fields, the mean rates were 95.9% for the 5-case group versus 93.1% for the >5-case group. For procedural fields, the mean rates were 99.0% for the 5-case group versus 96.6% for the >5-case group. For data entry fields, the mean rates were 95.5% for the 5-case group versus 93.6% for the >5-case group. Statistically higher compliance rates were observed in 18 of the 94 fields screened in the 5-case group compared with the >5-case group. These 18 fields are highlighted with an asterisk in the appropriate field in each of Tables 1, 2, and 3.

DISCUSSION

The goals of the prerandomization training phase of the NSABPB-32 trial were to establish a large group of surgeons that would perform SNB according to a specific set of guidelines required in the protocol, generate the necessary source documentation, and enter complete and accurate pretrial specific data. Based on our review, it appears that surgeons were well prepared to proceed to the randomization phase of the protocol. Outcome measures that indicate compliance with the technical aspects of the surgical protocol were success rate of SNB and the false negative rate. Overall the SNB success rate was 96.2% and the FN rate was 6.7%. Both of these values are comparable to international standards and are consistent with our preliminary experience with this training method.^5,6 Other measures of surgical procedure protocol compliance are related to whether the specific steps of the surgical SNB protocol were followed. This addresses the consistency of the procedure performed among different surgeons. There was excellent consistency among surgeons for all the variables measured in this training phase evaluation. Almost all protocol procedures were followed more than 95% of the time. These results demonstrate a very high level of compliance with a complex protocol involving multiple hospital departments.

Our results compare favorably to other trials in which surgical procedure compliance has been evaluated. For example, the Dutch Gastric Cancer Trial⁷of D1 and D2 nodal dissection, which had a rigorous training and quality control program, showed protocol procedural noncompliance (inadequate nodal removal) of 15.3% in D1 and 25.9% of D2 patients. There was nodal contamination (excessive nodal removal) in 22.9% of D1 and 23.5% of D2 patients. The intergroup 0116 trial of postoperative chemoradiotherapy versus observation following surgery for gastric cancer⁸ further illustrates the extent to which surgical procedures can vary from protocol guidelines in the absence of careful training and monitoring. In this trial, the investigators made specific recommendations that surgeons perform a D2 nodal dissection on all study patients and included appropriate instructions describing this procedure in the written protocol. However, when the surgical procedures were reviewed, it was found that only 10% of patients had in fact received a D2 or greater lymphadenectomy, whereas 36% had a D1 and 54% a D0 resection.

Relatively few surgical trials have addressed and factored training and quality control of the surgical procedure into the protocol design. One of the most commonly employed methods has been the use of a “precertification” process. Recent surgical trials that incorporated a precertification requirement are the Asymptomatic Carotid Atherosclerosis trial⁹ and the North American Symptomatic Carotid Endarterectomy trial¹⁰ for the treatment of carotid artery atherosclerosis. The process of precertification of surgeons in these studies was believed to be responsible for the low levels of perioperative stroke and mortality observed on the surgical arms of those studies. This method has also been employed in the American College of Surgeons Oncology Group Z0010 and Z0011 sentinel node trials and the Axillary Lymphatic Mapping Against Nodal Axillary Clearance breast sentinel lymph node trial.¹¹

NSABP B-32 also uses an auditing process on the randomized protocol cases similar to that used in the training cases. This auditing process continues for at least the first 20 patients enrolled by each surgeon. The outcome of that process is not yet available for analysis since patients are still being enrolled in the trial. Maintaining quality control is important so as to maximize standardization of the trial-related surgical procedure. This will allow the best opportunity to insure that trial outcomes are valid. Elements of this auditing should allow evaluation of enrolled cases and provide rapid feedback to the individual surgeon. This will minimize perpetuation of errors. The use of such an auditing process, however, has not been common practice in surgical trials. A recent review by Hall et al¹² found that only 17% of surgical trials published in major journals had any description of methods used for assessing compliance with the surgical therapy dictated by the protocol. Additionally, only 35% of trials evaluated made any attempt to standardize the surgical procedure in any fashion, while 25% of trials failed to provide any account of the surgical procedure. The positive effect that a quality control process can have on the performance of a surgical trial was demonstrated in the EORTC Melanoma Cooperative Group regional limb perfusion trial.¹³ In this trial, the rate of protocol violations was found to be excessive, and a system of establishing quality control was instituted. This process initially involved group meetings of participating surgeons and was then followed by site visits to the participating institutions. These interventions lowered the rates of protocol violations from 28% to 17% and finally 11% when site visits were completed.

Examples of trials where a process of quality control was built into the initial protocol design include the previously described Dutch Gastric Cancer trial⁷ and the Italian Gastric Cancer trial.¹⁴ In these trials, an intraoperative evaluation of the procedure by a regional referent surgeon or study coordinator was performed. This level of oversight provided the most rapid evaluation and correction of errors that may occur in a surgical procedure. In other large trials, quality control has been accomplished with centralized reviews of data and results. This type of oversight was used by the Medical Research Council Gastric Cancer Surgical Trial,¹⁵ which included 32 surgeons and 400 randomized patients. The compliance results accomplished in this trial are quite similar to those obtained in the Dutch Gastric Cancer trial.⁷ A similar type of review system was performed in the Dutch Total Mesorectal Excision trial,¹⁶ which is still accruing patients.

The training methods employed for the prerandomization phase of NSABP B-32 are most consistent with the conventional methods used to train surgical residents. The key elements are didactic written material, direct instruction in the operating room, and feedback on subsequent cases. We found that immediate feedback on a case-by-case basis was important during training case accrual and significantly reduced the perpetuation of errors. This led to a rapid training phase, and the majority of surgeons required only 5 cases prior to initiation of randomization. Although this method required a significant commitment of time by the core trainers, it led to rapid and successful adoption of protocol techniques by participating surgeons. Although a cost analysis has not been performed of the training techniques used for this trial, in a trial as large as NSABP B-32, loss of even a modest percent of the randomized cases due to errors in technique or noncompliance with the protocol would likely exceed the costs of training and quality control.

In general, insufficient emphasis has been placed on the generation of appropriate source documentation in surgical trials. Source documentation is information present in a permanent medical record. This includes laboratory results, radiology results, operative notes, and office notes. For NSABP B-32, we chose to place important information related to the surgical procedure, such as radioactive counts of sentinel nodes and patient inclusion criteria, in the operative note. This was chosen because surgeons are accustomed to and skilled at dictating detailed operative notes. The training manual had a list of items to be included in the operative note. Our review indicated that the compliance rate for inclusion of study-specific information into the operative note exceeded 95% in the majority of categories.

The compliance rate for entry of study-specific information into the pathology report, although overall a few percentage points lower than for the operative note, was also high. Sentinel nodes were labeled with multiple descriptors (eg, 10-second radioactive count number and presence or absence of blue dye) so that even if 1 descriptor were absent, correct linkage could be made to the pathology result and the specific SN removed at surgery. Given the high rate of inclusion of correct source documentation by both surgeons and pathologists, the possibility of incorrect linkage of the pathology result and the specific sentinel node removed was very unlikely.

In conclusion, we present an analysis of the prerandomization training phase of NSABP B-32. The goals were to train a large group of surgeons in the surgical steps of SNB according to the B-32 protocol and to insure that source documentation and data-form entry were complete and accurate. The results of the analysis, based on SNB success rates, false negative rates, and detailed analysis of source documentation and data collection forms, indicate a high level of protocol compliance. The B-32 trial is designed to detect a narrow survival difference of less than 2% between the target groups. Minimizing variability in the surgical methods and maximizing the quality of source documentation and data entry should enhance the overall quality of the trial and the ability to draw meaningful conclusions from the observed results.

ACKNOWLEDGMENTS

The authors would like to acknowledge the extraordinary effort put into this study by Mary Krupski, Elaine Cahoon, Jenne Wax, Rachel Hechter, Abby Crocker, and Deborah Peterson in coordinating the B32 training center at the University of Vermont and Joan Skelly, MS, for her data management and data analysis efforts.