Low Interrater Reliability of Videourodynamic Diagnosis of Detrusor External Sphincter Dyssynergia

Abstract

Objectives:

Complex bladder dysfunction requires urodynamic testing, often under fluoroscopy termed videourodynamic study (VUDS), to assist the diagnosis and management. Videourodynamic study is an objective tool with high interrater reliability (IRR) for identifying detrusor overactivity. However, IRR has not been validated with disorders associated with neurogenic bladder (NGB). We aim to investigate the IRR of VUDS to diagnose detrusor external sphincter dyssynergia (DESD) among NGB patients and hypothesize a high IRR to DESD diagnosis on VUDS.

Methods:

Videourodynamics tracings with fluoroscopic images were rated either positive or negative for evidence of DESD by 4 raters (2 neurourologists, neurourology fellow, and urology postgraduate year 2 resident), in patients who underwent VUDS from 2013 to 2017. The study population had known NGB without previous bladder reconstruction. The IRR was determined using percent agreement and κ values.

Results:

The experts had a percent agreement of 82.1% (κ = 0.26). Expert 1 and expert 2 had a percent agreement of 63.6% (κ = 0.1497) and 68.9% (κ = 0.2967), respectively, when compared with the fellow. Sensitivity and specificity ranged from 28% to 75% and 64% to 93%, respectively. The negative predictive values ranged from 90% to 95%.

Conclusions:

The IRR to diagnose DESD on VUDS was much lower than expected, even among experts, and was likely multifactorial and partially owing to lack of clinical context and lack of standardized VUDS interpretation of the electromyogram. A high negative predictive value was found among all participants. Further research is needed to evaluate factors contributing to the low reproducibility of DESD diagnosis on VUDS.

Urodynamic study (UDS) is commonly used to study underlying lower urinary tract dysfunction when diagnosis is uncertain from clinical history and examination alone. In more complex conditions, videourodynamic study (VUDS), which adds real-time fluoroscopic data, can improve diagnostic value when applied appropriately. Urodynamic study serves as an objective tool with high interrater reliability (IRR) for identifying detrusor overactivity.¹ However, its validity has not yet been well demonstrated in other disorders associated with neurogenic bladder (NGB). Detrusor external sphincter dyssynergia (DESD) is the urodynamic description of one such condition in which there is a dysregulation of the micturition reflex, characterized by involuntary contractions of the external urethral sphincter during an involuntary detrusor contraction. Detrusor external sphincter dyssynergia is seen in neurologic conditions including spinal cord injury, multiple sclerosis, and spina bifida where lesions or spinal cord abnormalities affect the central nervous system.² The accurate diagnosis of DESD is of paramount importance for neurogenic patients to manage bladder emptying, prevent recurrent urinary tract infections, maintain bladder compliance, and limit renal failure through elevated storage pressures. The role of UDS in diagnosis of DESD is well described^2–4; however, the reliability of UDS has not been described in this setting.

We sought to investigate the IRR of VUDS to identify DESD among NGB patients. We hypothesized a high IRR for DESD diagnosis on VUDS.

MATERIALS AND METHODS

After obtaining institutional review board approval, all patients who underwent urodynamic testing at our tertiary referral center between 2013 and 2017 were retrospectively identified. International Classification of Diseases, Ninth Revision (ICD-9) and Tenth Revision (ICD-10), NGB-related codes were identified—N36.44, N31.0, N31.1, N31.2, N31.8, N31.9, Z87.448, 596.54, or 596.55. Inclusion criteria included a diagnosis of NGB using previously mentioned ICD-9 and ICD-10 codes and completion of a UDS with fluoroscopic images (VUDS). Patients with prior slings (midurethral or pubovaginal), bladder/bladder neck suspension operations, or previous bladder reconstruction procedures such as augmentation cystoplasty were excluded. Each UDS and corresponding fluoroscopic images was deidentified and assigned a random study number by a person not involved in the ratings. A printed document of each UDS tracing and fluoroscopic images was provided to the rater. Patient clinical information and access to the electronic UDS was withheld from the rater.

Four clinical raters, 2 neurourologists, expert 1 and expert 2; a neurourology fellow, fellow; and a urology resident, postgraduate year 2 (PGY2), were asked to evaluate each set of images and UDS tracings for each patient. Raters were instructed to use the following criteria for VUDS evidence of DESD: “increased electromyogaphy (EMG) activity above baseline with an involuntary detrusor contraction, and/or a dilated proximal urethra with closed external sphincter on fluoroscopic images with voiding.” Raters were asked to record “yes” for positive evidence of DESD, “no” if negative for evidence of DESD, or “unable to determine” if study was felt to be incomplete or rater was unable to clearly identify positive or negative evidence of DESD by these criteria.

Results were compared between raters as well as the recorded diagnosis on the patient’s chart. Statistical analysis, using the STATA 13 program (StataCorp [2013], Stata Statistical Software, Release 13; StataCorp LP, College Station, TX), includes sensitivity, specificity, negative predictive value, and positive predictive value. The percent agreements and associated Cohen κ values were compiled. A value of κ < 0 indicated no agreement, κ = 0.01 to 0.20 signified slight agreement, κ = 0.21 to 0.40 as fair agreement, κ = 0.41 to 0.60 as moderate agreement, and κ = 0.61 to 0.80 as substantial agreement.

RESULTS

A total of 2238 patients underwent UDS at our tertiary referral center from 2013 to 2017. A total of 446 patients were identified using the inclusion ICD-9 and ICD-10 codes and the criteria of additional fluoroscopic imaging (VUDS). Patients who did not meet the diagnosis criteria (appropriate diagnosis code, UDS with fluoroscopic images), duplicate studies, and those who had a previous bladder reconstruction procedure were removed, leaving a total of 302 patients meeting the inclusion criteria for this study. When reviewing the official chart diagnosis, a total of 124 of the 302 subjects had a recorded diagnosis of DESD.

Of the 302 subjects, 92 studies were deemed substandard by the raters owing to absence of a voiding phase, displacement of catheters, or poor image quality and were excluded from final analysis, leaving 210 subjects in our cohort. Expert 2 had the highest percent agreement of 85.43% when compared with the chart diagnosis with a κ = 0.4295 showing moderate agreement. Expert 1 also had a high percent agreement of 64.9% but with a much lower κ = 0.2183 correlating with fair agreement, and this was similar with the PGY2. The remaining κ values showed slight to fair agreement among the different raters (Table 1).

TABLE 1.

The Percent Agreements and Associated κ Values for Each of the Paired Evaluators

	Percent Agreements and κ Values
	Expert 1	Expert 2	Fellow	PGY2	Chart
Expert 1	—	82.1%, κ = 0.263	63.60%, κ = 0.1497	79.8%, κ = 0.1193	84.77%, κ = 0.2183
Expert 2	82.1%, κ = 0.263	—	68.87%, κ = 0.2967	78.48%, κ = 0.2396	85.43%, κ = 0.4295
Fellow	63.60%, κ = 0.1497	68.87%, κ = 0.2967	—	65.89%, κ = 0.2242	64.9%, κ = 0.1873
PGY2	79.8%, κ = 0.1193	78.48%, κ = 0.2396	65.89%, κ = 0.2242	—	82.5%, κ = 0.2771
Chart	84.77%, κ = 0.2183	85.43%, κ = 0.4295	64.9%, κ = 0.1873	82.5%, κ = 0.2771	—

Comparison of each rater’s interpretation of the blinded studies to the clinical record was evaluated. Sensitivity of prediction of diagnosis of DESD to the chart ranged from 27.8% to 75%. The specificity ranged from 63.5% to 92.5%. The positive predictive value ranged from 21.8 to 42.6. However, the negative predictive value was greater than 90 for all raters (Table 2).

TABLE 2.

The Raters Compared With the Listed Chart Diagnosis and the Associated Sensitivity, Specificity, and Positive and Negative Predictive Values

	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Valu
Fellow vs chart	75.0%	63.5%	21.8	94.9
PGY2 vs chart	44.4%	87.6%	32.7	92.1
Expert 2 vs chart	63.9%	88.4%	42.6	94.8
Expert 1 vs chart	27.8%	92.5%	33.3	90.4

DISCUSSION

Urodynamic testing is an objective tool previously shown to have high IRR when identifying detrusor overactivity.¹ The same has not been reported for the diagnosis of DESD. Urodynamic study had been considered the criterion standard in complex patients and is recommended in neurogenic patients to obtain baseline bladder function status⁵ and assess response to therapy. The urodynamic criteria for diagnosis of DSD have previously been published, but specific details regarding the EMG signal (method, amplitude, onset with contraction) remain to be determined. For neurogenic patients at our institution, we consistently use fluoroscopy during our urodynamic evaluations, but we do not routinely use urethral pressure measurements. Using the previously published fluoroscopy criteria by Blaivas et al,⁶ we defined DESD as a dilated posterior urethra with a closed external sphincter. Although needle electrodes in the anal sphincter have been considered criterion standard for EMG readings,² they are not commonly used for urethral sphincter evaluation owing to their pain and discomfort to the patients. Perineal surface electrodes are widely used during UDS based on International Continence Society UDS Good Practice Guidelines^7,8 where the criteria of increased EMG activity at time of attempted voiding above baseline are used for the diagnosis of DESD. Previously, there have been some reports correlating the different aspects of UDS in the diagnosis of DESD. Stoffel² reported a series of 67 patients with known DESD and found that 55% had findings on voiding cystourethrogram consistent with DESD, and 75% of their patients met EMG criteria.⁹ A more recent 2017 pilot study and 2018 multicenter study by Dudley et al^10,11 looked at the IRR in the interpretation of urodynamic tracings in the pediatric neuropathic population. They reported a strong correlation in the pilot study among raters in relation to the bladder neck status on fluoroscopy and a weak correlation for EMG synergy.¹⁰ When this was expanded in the multicenter study, EMG synergy and detrusor overactivity showed fair agreement.¹¹ Bladder neck status on fluoroscopy showed moderate agreement. Although not evaluating a specific diagnosis, this study does show that there appears to be a significant variability in the interpretation of aspects of UDS in the neuropathic population. It is also important to note that, in their study, the raters were given a clinical scenario about the patient in addition to the UDS tracing and corresponding fluoroscopic images.

We had hypothesized a high IRR for the diagnosis of DESD on VUDS. Although our study found a high percent agreement, this corresponded to a low κ value, between expert 1 and expert 2 when diagnosing DESD, indicating only fair agreement. For all comparisons between raters, most κ values indicated only fair agreement in the diagnosis of DESD. There are likely multiple factors that could contribute to this low κ value, including the blinded nature of the study (ie, no clinical vignette), the quality of the printed study, and the difference in experience with reading UDS between the raters. A PGY2 was included in this study to evaluate the impact of limited UDS experience on identification of DESD given clear diagnostic parameters. Our data did show that, among all raters, each had a negative predictive value of greater than 90, indicating that the ability to rule out the diagnosis of DESD on VUDS was much more reproducible and consistent regardless of training background.

Our study has its limitations, including the retrospective nature. Also, we believe that including a small clinical vignette with the VUDS information could improve the raters’ ability to accurately diagnose DESD. However, such a clinical vignette may bias the rater when seeking to diagnose DESD purely on VUDS findings. In practice, the value of pertinent clinical information intricacy of each study highlights the need for the evaluator to be present during the VUDS especially in challenging cases to provide the best real-time interpretation of the study. In addition, whether the rater’s evidence of DESD was by EMG, fluoroscopic images, or both, was not captured owing to the nature of the study design. This may have provided further valuable data on the impact of study parameters on diagnosis. Also, all of our VUDS were completed with patch electrodes to measure the EMG. Spettel et al⁴ showed a significant disagreement between voiding cystourethrogram and EMG diagnosis of DESD, although this was similar to the previous findings using the needle electrodes.

We believe that further research is needed to produce specific standards when diagnosing DESD that would allow for improved reproducibility among practitioners reviewing VUDS studies. We believe that further studies to better define the EMG requirements (ie, amplitude, onset, etc) would greatly help the reproducibility. These studies should then be included in the training of urology residents and fellows.

CONCLUSIONS

Detrusor external sphincter dyssynergia can be difficult to diagnose on VUDS alone (ie, no clinical information) and shows a low to moderate IRR in the reproducibility of the diagnosis. However, all raters had a high negative predictive value and showed a consistent ability to rule out the DESD diagnosis based on VUDS without a clinical history.