Use of anchor based receiver operating characteristics to evaluate clinical response of an adjunctive device used in successful surgical therapy

Abstract

Clinical trials designed to identify additional clinical benefit from an adjuvant added to typically successful surgical procedure, e.g., lumbar discectomy to treat sciatica, can be challenged by the size of the remaining primary outcome variable (e.g., reduction in leg pain score) available for further improvement. Patients who are high responders to treatment can be identified by receiver operating characteristic (ROC) analysis when patient satisfaction, obtained from a patient-reported outcome (PRO) questionnaire, is used to anchor the ROC analysis. ROC analysis provides a within-study method to assess clinical metrics. Utility of anchor-based ROC analysis is illustrated here using two published clinical trials that used similar protocols to measure ipsilateral leg and lower back pain (LBP) following lumbar decompression surgery to evaluate a commercially available adjuvant, dual-polymer gel (Oxiplex, Fziomed, Inc., San Luis Obispo, CA, USA). Both studies were FDA authorized, prospective, randomized, double-blind, multi-center trials reporting Level 1 evidence. Participants with unilateral herniation at L4-L5 or L5-S1 with severe leg pain [>68 mm on 100 mm visual analog scale (VAS); >62 on Lumbar Spine Outcome Questionnaire (LSOQ) and LBP >50 mm on VAS Scale; >63 on LSOQ scale] with radiological evidence of nerve root compression, underwent initial, single-level, partial, lumbar discectomy. Participants were randomized to surgery plus dual-polymer gel (Treatment) or surgery alone (Control). Ipsilateral leg pain scores (VAS and LSOQ) were obtained via PRO before and 6 months after surgery. Patient satisfaction at 6 months was determined from responses to the Satisfaction Global Assessment PRO (Fischgrund study) and Satisfaction (Rhyne study) domain of the LSOQ score. Anchor-based ROC analysis in both studies showed an increased number of patients in the Treatment groups compared to the Controls for both the Rhyne and Fischgrund studies who were high responders to surgery (12.8%, 12.5%, respectively). Anchor-based ROC analysis identified similar cohorts of patients in the two studies with meaningful clinical benefits following use of the adjuvant. ROC analysis is a robust and reliable statistical methodology that can identify high responders within heterogeneous populations and enable assessment of patient responses to adjuvant therapies associated with efficacious primary treatments in complex clinical trials.

Treatment of sciatica is commonly relieved by decompression of a herniated lumbar disc (1,2). However, patients may experience residual pain in their ipsilateral leg as well as lower back pain (LBP) following surgery due to fibrosis in the epidural space entrapping spinal nerves and influx of inflammatory mediators stimulating nociceptors of sensory nerves (3-9). As a result, there is need for an adjuvant applied at the end of the surgical procedure, especially in high-risk cases, e.g., patients with sciatica and LBP, undergoing lumbar discectomy (10,11). Additional patient benefits from use of an intraoperative adjuvant can be difficult to demonstrate since most patients experience ipsilateral leg pain reduction after lumbar decompression surgery (1,2,10,11). A commonly used measure to evaluate pain reduction in clinical trials is the visual analog scale (VAS) which records the patient’s perception of pain before and at various times following surgery. More recently, the Lumbar Spine Outcome Questionnaire (LSOQ) was developed to assess categorical changes in patient outcomes following decompression surgery (12). The patient’s recollection of pain levels evaluated by a single question (e.g., VAS), often many months later, adds confounders to evaluation of clinical benefit (13,14). Comparing the difference in change of VAS scores following surgery expressed as mean values between control and treatment groups often requires a large sample size for statistical significance due to patient heterogeneity, inherent differences in trial site management, and may obscure identification of patient cohorts who responded well to surgery (14-18). In addition, results from studies using other patient reported outcome (PRO) measures make it difficult to compare results in medical literature (18,19).

Receiver operating characteristic (ROC) analysis uses an anchor-based method that assesses the relationship between a numerical scale and a binary outcome (for example VAS or LSOQ leg pain score and patient satisfaction following surgery) to evaluate how well a metric compares clinical responses (20,21). By incorporating the anchor of patient satisfaction at the end of the study (22,23), ROC analysis can use patient’s perception of benefit to define patient responder thresholds based entirely on data obtained from within the study. Anchored-based ROC thresholds provide the ability to identify improvements in subjective measurements that discriminate patients satisfied vs. not satisfied with their treatment. Because ROC analyses use data generated within the study, anchor-based approaches are favored in this regard as they are thought to be independent from the underlying condition (19,20). These approaches are agnostic to the clinical context of the measures used. As a result, an ROC analysis anchored by the study participant’s satisfaction provides a valid, within-study method to assess clinical metrics and identify important clinical response benchmarks (e.g., substantial reduction in leg pain) in relation to the subject’s perspective on treatment effectiveness. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Written consent was obtained from all the patients who participated in the clinical trials discussed in this short report.

Here, we demonstrate the utility of anchor-based ROC analysis using two reports of published clinical trials that utilized similar protocols [Rhyne (24); Fischgrund (25)]. Rhyne used the leg pain LSOQ score and Fischgrund used the VAS to measure change in leg pain following lumbar decompression surgery to evaluate the patient’s response to a commercially available adjuvant, a dual-polymer gel (Oxiplex, Fziomed, Inc., San Luis Obispo, CA, USA). Both studies were FDA authorized, prospective, randomized, double-blind, multi-center trials reporting Level 1 evidence for safety and effectiveness of the adjuvant. Participants (22–70 years of age) with unilateral herniation of a lumbar disc at L4-L5 or L5-S1 presenting with severe leg pain (>68 mm VAS leg pain scale; >63 LSOQ leg pain score) and LBP (>50 mm VAS, >63 LSOQ back pain scale) with radiological evidence [magnetic resonance imaging (MRI) or computed tomography (CT) scan] of nerve root compression, underwent initial, single-level, partial, lumbar discectomy. Participants were randomized to surgery plus dual-polymer gel (Treatment) or surgery alone (Control). Leg pain scores were reported by patients in both studies before and at final study visit, 6 months following surgery. Overall patient satisfaction was measured at 6 months when the patient’s study participation ended. Individual patient satisfaction determinations were used to define responder thresholds using ROC analysis (Figures 1,2). The cut-off generated from an anchor-based ROC analysis of results from the Rhyne and Fischgrund studies showed increased number of patients in the Treatment groups who were high responders to surgery (12.8%, 12.5%, respectively, Figure 3).

Figure 1.

To evaluate whether pain relief (change in ipsilateral leg VAS from baseline to month 6 predicts patient-reported treatment satisfaction, an ROC curve was constructed comparing patients who were “Very Satisfied” against those who were not (somewhat satisfied, somewhat dissatisfied, or very dissatisfied) (25). The ROC curve plots sensitivity (the correct identification of “Very Satisfied” patients) against 1–specificity (the false-positive rate) across the full range of possible VAS-change cutoffs. The AUC summarizes overall discriminative accuracy, with 0.50 representing no better than chance and 1.00 representing perfect discrimination. In this analysis, VAS change showed good discriminative ability (AUC =0.79), with the curve rising steeply at low false-positive rates, indicating that patients with the greatest reductions in leg pain were consistently among the most satisfied with treatment outcome. The cut-off value for high responders is −68. AUC, area under the curve; ROC, receiver operating characteristic; VAS, visual analog scale.

Figure 2.

ROC curve comparing patients who were “Very Satisfied” against those who were not at month 6 (24). The ROC curve plots sensitivity (the correct identification of satisfied patients) against 1–specificity (the false-positive rate) across the full range of possible LSOQ leg painchange cutoffs. The AUC summarizes overall discriminative accuracy, with 0.50 representing no better than chance and 1.00 representing perfect discrimination. In this analysis, LSOQ leg pain change showed discriminative capacity (AUC =0.593, 95% CI: 0.506, 0.679) and statistically reliable evidence that the discriminative performance is greater than chance alone (AUC =0.50). The cutoff value for high responders is −3. AUC, area under the curve; CI, confidence interval; ROC, receiver operating characteristic.

Figure 3.

ROC-based threshold analysis of patient satisfaction PRO was applied to change in leg pain scores following lumbar discectomy in two different clinical trials (24,25) to define thresholds of high responders. There was a similar increase in the percentage of participants in both reports who were high responders to treatment in the Dual-Polymer Gel + Surgery group compared with the Surgery alone group. LSOQ, Lumbar Spine Outcome Questionnaire; PRO, patient-reported outcome; ROC, receiver operating characteristic.

ROC analysis of Fischgrund (25): to evaluate whether pain relief (change in ipsilateral leg VAS from baseline to Month 6) predicts patient-reported treatment satisfaction, an ROC curve was constructed comparing patients who were “Very Satisfied” against those who were not (somewhat satisfied, somewhat dissatisfied, or very dissatisfied, Figure 1). The ROC curve plots sensitivity (the correct identification of “Very Satisfied” patients) against 1–specificity (the false-positive rate) across the full range of possible VAS-change cutoffs. The area under the curve (AUC) summarizes overall discriminative accuracy, with 0.50 representing no better than chance and 1.00 representing perfect discrimination. In this analysis, VAS change showed good discriminative ability (AUC =0.79), with the curve rising steeply at low false-positive rates, indicating that patients with the greatest reductions in leg pain were consistently among the most satisfied with treatment outcome.

ROC analysis of Rhyne (24): to evaluate whether pain relief (change in leg pain LSOQ from baseline to month 6) predicts LSOQ patient-reported treatment satisfaction, an ROC curve was constructed comparing patients who were “Very Satisfied” against those who were not (Somewhat dissatisfied, neither satisfied or dissatisfied, somewhat dissatisfied, or very dissatisfied, Figure 2). The ROC curve plots sensitivity (the correct identification of satisfied patients) against 1–specificity (the false-positive rate) across the full range of possible LSOQ leg pain-change cutoffs. The AUC summarizes overall discriminative accuracy, with 0.50 representing no better than chance and 1.00 representing perfect discrimination. In this analysis, LSOQ leg pain change showed discriminative capacity (AUC =0.593, sensitivity 0.71, specificity 0.44). In contrast to the previous ROC analysis using VAS leg pain to patient satisfaction, the LSOQ leg pain is an ordinal variable with 5 categories rather than a continuous metric ranging from 0–100, like VAS. The use of a 5-level ordinal variable versus a 100-point continuous variable has a theoretical loss of information. In this application, a slight relative decrease was observed because of using a 5-level ordinal variable, especially when most subjects saw an improvement in leg pain.

High responders: ROC-based threshold analysis of patient satisfaction PROs at month 6 defined high-responder thresholds. In both studies, a greater percentage of participants in the dual-polymer gel plus surgery group were high responders compared with participants who underwent surgery alone (Figure 3). Rhyne study: using the cut off LSOQ score of 3 (change score of −3) categories from the ROC curve constructed from leg pain scores reported in the Rhyne study, 40 of 78 (51.3%) participants in the Treatment group were high responders. In contrast, 30 of 78 (38.5%) participants were high responders in the Control group, an increase of 12.8% in the Treatment group. Fischgrund study: Using the cut off VAS score of ≥68 (change score of − ≥68 change score) based on the ROC curve constructed for leg pain from the Fischgrund study, 51 of 88 (58%) participants in the Treatment group were high responders, an increase of 12.5% compared to Control patients (25).

A common method to report results from clinical trials is determination of the difference in the mean values of outcomes between the treatment and control groups. For example, following decompression surgery to treat sciatica, the difference in reduction of the mean VAS ipsilateral leg pain scores between the control and treatment groups is commonly used as an efficacy measure. Interpretation of that difference typically compares results from the trial in question to reports in the medical literature of minimal clinically important differences (MCIDs) in mean values (17,18,26-31). Use of MCIDs reported from other trials can be problematic in evaluating the results from a specific clinical trial, especially when different outcome measures are used.

Trials designed to evaluate the benefit of an adjuvant added to a typically successful surgical procedure, like lumbar discectomy to treat sciatica, using change from preoperative mean pain scores of the treated vs. control populations can be challenged by the size of the remaining primary outcome variable available for further improvement (i.e., reduction in mean VAS leg pain scores). If the window for additional reduction in mean values of leg pain reduction for the entire Treatment group is <15% of the baseline score, it may not be mathematically possible to identify additional patient benefit from use of the surgical adjuvant. Averaging metrics of clinical outcomes could obscure identification of true benefits that occurred in patient cohorts; for example, an increase in the percentage of patients who have clinically meaningful relief of pain when an adjuvant is added to a surgical procedure that typically reduces pain. In the examples of clinical trials cited above, anchor-based ROC analysis identified a cohort of patients who were high responders to surgery; an interpretation that is independent of MCIDs reported in other trials.

Measuring reduction in pain from baseline following lumbar discectomy with a quality-of-life PRO may be more problematic than measuring residual pain at the end of the study. Patients with high levels of pain prior to surgery are more likely to have a greater numerical reduction in pain at the end of the study, which is not necessarily related to the patient’s overall response (15,16,31). Averaging those reductions in pain together with the results from patients with lower pain levels prior to surgery may skew the overall cohort average and prevent identification of patients who were high responders. Evaluation of adjuvant therapies by PROs at the end of surgery is often obscured by the patient’s overall response to surgery (13-16). Patient recollection of comparative pain levels asked by a single question separated from baseline by many months adds additional challenges to the sample size determination required for statistical analysis and comparison of MCIDs. In addition, the site heterogeneity that is common in medical device trials, especially involving surgery and PROs, further challenges demonstration of statistical benefit from the adjuvant (17). Anchor-based ROC analysis, used to determine within study thresholds of benefit, provides an opportunity to identify clinically meaningful high responders to treatment (20,21).

Although this Brief Report demonstrates the use of anchor-based ROC analysis as an additional methodology to evaluate outcomes following decompression surgery and use of an adjunctive device to provide a “within study evaluation of potential clinical benefit”, thus avoiding cross-study comparisons of MCIDs, it contains the inherent bias that derives from use of previously published results. In addition, the anchor-based ROC analysis reported here is limited by the use of two different pain scoring methods; one using a Likert scale and the other using categorical variables to illustrate the utility of anchor-based ROC analysis.

In conclusion, the use of ROC analysis anchored by patient satisfaction allows for the determination of high responder thresholds which can provide complementary information to that obtained by traditional approaches focused on comparison of overall mean scores. Anchored ROC analysis is a robust and reliable statistical methodology that can identify high responder cohorts within heterogeneous populations and enable additional assessment of patient responses to adjuvant therapies associated with efficacious primary treatments in complex clinical trials.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Written consent was obtained from all the patients who participated in the clinical trials discussed in this short report.