A Process for the Development of Multicenter Urodynamic Studies

Abstract

Objectives

In multicenter urodynamic studies (UDS) there can be considerable variations in testing procedures, training, equipment, and reviewer biases. The purpose of this paper is to describe our experiences with a continuous quality improvement process for the standardization of multicenter UDS in a multi-institutional network.

Methods

A quality control process was developed that included protocol development, certification of urodynamic testers, central review to assess compliance with protocol and quality, protocol modifications, standardization of equipment and signal configuration, development of an electronic signal repository and the development of UDS interpretation guidelines.

Results

We describe our experience and process in the development and implementation of a standardized UDS protocol in a multicenter surgical trial for stress urinary incontinence. The process describes our protocol development, quality control measures, standardization processes, electronic signal repository, and the need for UDS interpretation guidelines. A Urodynamic Testing Procedures protocol was implemented successfully by twenty urodynamic testers at nine continence treatment centers. The protocol provides explicit and detailed guidelines for equipment, calibration, patient position, specific annotations, lay language bladder sensation parameters, visual LPP techniques, modifications for prolapse, and data recording. A UDS Interpretation Guidelines document provides specific suggestions for validity and plausibility determination, expected ranges of urodynamic variables, and reasonable agreement of measuring systems. Both documents are available to urodynamic investigators on the Urinary Incontinence Treatment Network (UITN) website at http://www.uitn.net/resourcesforphysicians.htm.

Conclusions

Multi-center urodynamic studies require a continuous quality improvement process and the development of UDS testing procedures and interpretation guidelines.

Introduction

The reliability and reproducibility of urodynamic data depends on standardized technical performance, annotation, and interpretation ¹. International Continence Society (ICS) subcommittees have issued reports on the standardization of urodynamic terminology, terminology relating to lower urinary tract dysfunction, and Good Urodynamic Practice (GUP) guidelines ², however these reports do not focus on how to implement and assess compliance of both technique and interpretation in a multi-center environment. In large multicenter studies there can be considerable variations in testing procedures, training, equipment, and reviewer biases. In a multicenter study of benign prostatic hypertrophy, Schaefer et al. demonstrated that many urodynamic tracings could not be interpreted ³. Previous investigators involved in other multicenter studies have argued for a standard urodynamic protocol and central review of urodynamic data to provide a single standard of interpretation ⁴. Pressure flow studies (PFS) in multicenter investigations have been particularly difficult to interpret; up to 38% of the tracings in the multi-center ICS BPH study were rejected during a central review ⁵. One multicenter study on men with BPH eventually demonstrated adequate agreement between the investigator and a quality control center when a consensus protocol was developed, but this protocol included insertion of a suprapubic catheter under local anesthesia ⁶. There are no other standardization models or quality control publications on multicenter urodynamic studies (UDS) in women.

The Urinary Incontinence Treatment Network (UITN) is a consortium of investigators from 9 continence treatment centers (centers) with support from a biostatistical coordinating center (BCC). This network is supported by cooperative agreements from the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) and the National Institute of Child Health and Human Development (NICHD). In 2001, the network implemented the first protocol, the Stress Incontinence Surgical Treatment Efficacy Trial (SISTEr) – a randomized trial of 655 women to compare the modified Tanagho Burch procedure with the autologous rectus fascia sling procedure for women with predominant stress urinary incontinence. The primary aim of the SISTEr trial is to compare the efficacy of these two surgical procedures and 2 year outcomes are currently being assessed. One of the secondary aims of the SISTEr protocol is to determine the prognostic value of UDS and to identify the urodynamic parameters that predict treatment success in each surgical procedure.

The UITN Steering Committee -recognizing some of the potential problems that others had encountered with multicenter UDS - established a Urodynamic Studies Work Group (UDS WG) composed of two urologists and two urogynecologists to develop a standardized UDS protocol and maximize consistency and quality in the performance of urodynamic testing across study centers. The purpose of this paper is to describe the process involved with the development of a multi-center UDS protocol, the difficulties we encountered during that process, and the quality control efforts that were necessary in order to obtain reliable and interpretable UDS data from each of the centers.

Materials and Methods

The on-going quality control process occurred before the start of the surgical study and during early data acquisition and included:

• UDS protocol development, certification of testers and resolution of site-specific differences in UDS methods;

• Initial central review to assess compliance with protocol and quality of UDS.

• Protocol modifications, UDS conference calls, and regular assessment of protocol compliance;

• Standardization of UDS equipment, signal configuration, and the development of an electronic signal repository;

• Assessments of the reliability of UDS interpretation between local and central reviewers

• Development of UDS Interpretation Guidelines

Results

UDS Protocol development, certification of testers and resolution of site-specific differences in UDS methods

The UDS WG met at least monthly during the year prior to study implementation, developing and refining standardization for urodynamic testing procedures. The SISTEr UDS protocol complied with terminology from the Standardization Committee of the ICS ² and technical suggestions from the Good Urodynamic Practice guidelines⁷. Several study-specific urodynamic details that were not explicitly described in the ICS Standardization Committee documents were defined by Steering Committee consensus. These included:

1. Filling cystometry performed in the standing position. Voiding would be in the sitting position.

2. The filling cystometry infusion rate would be 50 ml per minute.

3. ICS definitions for bladder sensation parameters would be translated to lay language. A scripted prompt using a television watching scenario was developed and used by testers at all centers.

4. The minimum acceptable uroflowmetry volume would be 150 ml for both the free uroflowmetry and pressure flow studies.

5. Patients with anterior prolapse ≥ Stage III (based on POPQ examination) ⁸ would have leak point pressure (LPP) assessments with and without prolapse reduction. Prolapse reduction would be performed with a sponge stick or gauze pack.

6. Urethral relaxation during voiding pressure flow studies (PFS) would be assessed by perineal surface electromyography (EMG) patches.

All UDS testers were required to be certified on UDS protocol performance by the principal investigators at their center before beginning the study.

Initial central review to assess compliance with protocol and quality of UDS

The UDS WG reviewed the first 11 studies performed under the UDS protocol. This review identified several significant problems including poor protocol adherence, poor tracing quality and lack of annotations. Telephone conference calls with the UDS testers were initiated as a means to answer technical and procedural questions and emphasize protocol adherence and standard annotations. Investigators, rather than data entry personnel, were required to fill out the data forms to cut down on transcription errors. Follow-up assessment included a central review of the first three signals from each UDS tester with a written feedback program to each tester. Standing UDS WG reports, including a “Top Ten Reminders” list, were developed to review with investigators at each monthly Steering Committee meeting.

These initial reviews highlighted specific aspects of the protocol that proved difficult to standardize among the 20 testers. For instance, CLPP values had marked intra-individual variability and were non-reproducible in the first 11 cases. Cough Leak Point Pressure (CLLP) values require careful consistent patient instructions and patient cooperation to result in incrementally forceful coughs. Analysis revealed that Valsalva leak point pressures (VLPP) but not CLPP measurements could be reliably performed and interpreted, thus prompting UITN investigators to drop the measurement of CLPP from the protocol. Several measures were added to the protocol to enhance the reviewer's ability to interpret the plausibility of the study. These included annotation of baseline pressures for both the CMG and PFS studies. A cough was added at MCC to demonstrate urodynamic stress incontinence (USI) if the Valsalva maneuvers had not previously provoked USI. Pre-void pressure line concordance (agreement) was reassessed with a cough after the patients were seated and the transducers were adjusted, before the void and again at the end of the void, to establish proper functioning of the pressure measurement systems.

Consistency in procedure details, compliance, and annotation improved after implementation of the procedural modifications and the central review process. The Urodynamic Testing Procedures were implemented successfully by twenty urodynamic testers at nine continence treatment centers. The protocol provides explicit and detailed guidelines for equipment, calibration, patient position, specific annotations, lay language bladder sensation parameters, visual LPP techniques, modifications for prolapse, and data recording. Our specific UDS Protocol is available to urodynamic investigators on the UITN website at http://www.uitn.net/resourcesforphysicians.htm.

Standardization of UDS equipment, signal configuration and the development of an electronic signal repository

Subsequent central review for both quality assurance and to evaluate protocol compliance revealed several logistical problems including poorly legible copies of signals, varied configurations of signals, and the inability to manipulate the local urodynamic software for detailed analysis. Our central reviewers were scattered across the country which made it difficult to distribute signal copies for review. Since most of the centers were using similar urodynamics equipment and software (Laborie Medical Technologies Corp, Williston VT), the SC agreed to standardize equipment across all sites. This improved the central review process in a number of ways. First, a standardized UITN UDS signal configuration template was created from the UITN protocol and downloaded to each urodynamics machine study-wide. The configuration template standardized the format, the annotations, and the axis ranges for pressure and flow measurements for all measured variables. Once this standard urodynamic template was used at all centers, the technical quality of the UDS signals improved in consistency and legibility. Secondly, because the newer urodynamic technology is digital, signals could be transmitted electronically. A UDS repository of signals was established at the BCC and centers began transmitting signals to the repository electronically. This electronic signal repository allowed urodynamic signals to be forwarded electronically to central reviewers throughout the United States for frequent quality control assessments.

Preliminary evaluation of interpretation reliability assessments

The next stage of quality control was a systematic evaluation of the reliability of urodynamic interpretation across UITN physician reviewers. Interrater reliability between central reviewer and the local reviewers was assessed with a blinded re-interpretation of 24 signals originally reviewed by local UITN Physician Investigators. We defined “Agreement” if all 3 central reviewers recorded the same integer value or reached the same conclusion as the original local reviewer. In general, agreement was higher for quantitative than for qualitative measures. This early assessment of the UDS data indicated that there were significant issues that needed to be resolved. The inter-rater reliability for PFS was most problematic, as reviewers had widely differing views on when a particular voiding study could be considered plausible and interpretable. Investigators used different criteria for deciding how and when artifacts could be corrected. We concluded that inter-rater reliability and accuracy might be improved if standardized interpretation guidelines were written and used by all reviewers study-wide.

Development of Interpretation Guidelines

UDS Interpretation Guidelines were developed for use during local and central review of UDS tracings. ICS definitions and Good Urodynamic Practice Guidelines were referenced for all areas in which they were available, however the UDS WG created specific guidelines and definitions for controversial issues for which no guidance could be found in the current literature. Prior to implementation, the Interpretation Guidelines were reviewed and critiqued by an outside consultant and final refinements were approved by the SC. The UDS Interpretation Guidelines provide specific suggestions for validity and plausibility determination, expected ranges of urodynamic variables, reasonable agreement of measuring systems that were to be used by all UITN investigators in order to minimize the individual variation previously identified in our studies. Literature review and a review of the first 200 UITN baseline studies determined expected ranges for baseline pressure measures and queries were sent back to UDS reviewers whenever baseline values fell out of expected value ranges. At both local and central review, the UDS signal was not to be interpreted until the reviewer evaluated the UDS tracing to confirm that it met plausibility criteria. Plausibility rules addressed tracing legibility, protocol adherence and assessment of pressure measurements. The UDS Interpretation Guidelines can be found at http://www.uitn.net/resourcesforphysicians.htm. Once these guidelines were in place, we performed interrater reliability studies between central and local reviewers for both the CMG and PFS portions of the study. The results of these studies are reported separately.

Comment

Multicenter urodynamic studies require a continuous, committed quality improvement process to ensure that the studies are properly performed and consistently interpreted. Future multicenter urodynamic studies may benefit from the lessons learned by this group and it is for this reason that we have made specific recommendations, based on our experience during this process. In addition we have provided the documents developed for the UITN protocol (the Urodynamic Testing Procedures and the UDS Interpretation Guidelines) at http://www.uitn.net/resourcesforphysicians.htm in the event that other investigators might find them useful when establishing UDS protocols in future studies.

The International Continence Society and its subcommittees have worked diligently to improve the standards and comparability of worldwide UDS^2,7,9,10. Such work has been invaluable in providing a framework for our work. In spite of these efforts to standardize UDS, obtaining quality UDS data will not be possible in large multi-site studies by simply asking investigators to follow the Good Urodynamics Practice (GUP) guidelines. A recent survey in the U.K. revealed that clinicians were not convinced that they needed to change their methods to achieve uniformity¹¹. Even when the technical aspects of the studies are performed similarly, there remain differences of opinion on how to evaluate and interpret a study. For example, the urodynamic literature suggests that agreement of the abdominal and intravesical signals should be tested throughout the filling study with frequent ‘cough checks’, Theoretically, the cough spikes should be identical in each system, but a cough is a very rapid (millisecond) event and technical imperfections in the measurement process often lead to non-identical cough spikes. There is very limited data on what is acceptable and non-acceptable agreement with the measuring systems during cough. Sullivan et al ¹² considered cough signal quality to be “Grade A” when the measured cough spikes on the abdominal and intravesical signals showed the smaller spike to be 70-100% of the larger. The UDS WG adopted this 70% minimum agreement as our quality control measure for the cough signal. We understand that others will have different opinions about this issue and do not advocate 70% or 80% minimum signal agreement criteria as standards for all urodynamic investigators. In fact, prior to beginning a study, one should strive for near equivalence in the agreement of the pressure lines. The question is how much agreement is acceptable for a study to be included in a data set. Most UDS studies in the literature do not comment on whether the tracings were subjected to plausibility testing. We want to emphasize that some threshold of signal agreement needs to be determined for plausibility interpretation by urodynamic reviewers.

The goal of the UDS WG was to insure that UDS studies were being conducted as described in the protocol and that the data was being interpreted consistently across sites. If UDS are being interpreted by more than one reviewer, then a quality control process should demonstrate that the results are similar when they are reviewed by any competent and qualified interpreter. Interpretation Guidelines are necessary since the literature demonstrates that UDS interpretations are not reliable even with the simplest of urodynamic studies. For example, flow pattern interpretation during free uroflowmetry ^{13 14} has been shown to be unreliable. Without such guidelines each reviewer uses their own opinion to decide whether something is plausible or not. Even in studies that use one central reviewer, if the criteria are not established, the work is not reproducible. Using standardized Interpretation Guidelines, we have subsequently performed inter-rater reliability studies to determine the reliability among central reviewers and also between central and local reviewers for free uroflowmetry, filling cystometry, and PFS using the UITN Guidelines. The outcomes of these studies will be reported separately, but they demonstrated improvement over results in our preliminary review.

We were able to markedly improve the technical quality of the urodynamics and our ability to centrally review tracings by standardizing the equipment and creating an electronic central repository. We do not think this is a prerequisite for high quality UDS data, but certainly it is helpful for any central review quality control process.

We recognize the limitations of this report describing a quality control process for managing UDS data in a multicenter study on the surgical treatment of SUI. First of all, this is not an attempt to define how UDS should be performed, nor is it an attempt to define UDS parameters. Rather this is an assessment of the difficulties encountered in implementing the GUP in a multicenter study. UDS are performed nearly universally by clinicians, UDS parameters are frequently cited in the literature. Thus we feel our experience is generalizable to the majority of UDS currently being conducted. We acknowledge that many of the suggestions are not evidence based, but rather the result of consensus and compromise in a multicenter study. Other investigators have centralized the process of UDS interpretation with UDS experts reviewing all UDS tracings. We purposely chose not to have all UDS reviewed by a central expert reviewer because this would not be generalizable to how UDS are being conducted in the community. It is recognized that a different set of investigators with alternative clinical priorities might require a different protocol for the urodynamic procedures; however the urodynamic procedures that can be found on the website may serve as a starting point for urodynamic procedure design in multicenter studies of women with stress urinary incontinence.

Conclusions

This manuscript details our experience in the development and implementation of a standardized UDS protocol in a multi-center trial for surgical treatment of stress urinary incontinence. Based on this experience with a two year continuous quality control process, we suggest the following specific strategies for investigators embarking on multicenter urodynamic studies:

1. Establish a subcommittee to assume responsibility for UDS quality control.

2. Expect that every center will perform and interpret urodynamic studies differently.

3. Determine what outcome variables are most important to the study and how can they most reliably be obtained.

4. Develop a specific urodynamic procedure protocol and test this at several centers before the study begins.

5. Develop a certification process for UDS testers and interpreters that demonstrates ability and willingness to perform and interpret tests in a manner described by protocol.

6. Standardize the signal configuration template and axis ranges to allow other reviewers to review studies more easily and with more familiarity.

7. If feasible, standardize equipment, use digital signal acquisition, and develop a central repository of all electronic signals.

8. Develop a system to distribute these electronic signals to distant central reviewers for quality control purposes.

9. Review the signals regularly and provide feedback to testers on how to improve their performance.

10. Develop specific urodynamic interpretation guidelines to maximize interpretation reliability.

Abbreviations

UDS

urodynamic studies

UITN

Urinary Incontinence Treatment Network

BCC

Biostatistical Coordinating Center

SISTEr

Stress Incontinence Surgical Treatment Efficacy Trial

UDS WG

Urodynamic Studies Work Group