Problems With O’Connell et al “Implanted Spinal Neuromodulation Interventions for Chronic Pain in Adults” (Cochrane Review)

The article by O’Connell et al, published in December 2021 in the Cochrane Database of Systematic Reviews, presents a systematic review and meta-analysis of implanted spinal neuromodulation therapies for the treatment of chronic pain.¹ The review is focused on spinal cord stimulation (SCS) and dorsal root ganglion stimulation (DRGS). We agree that given the relative costs of SCS and DRGS and their invasive nature, there is a need for rigorous review of their efficacy and safety, and we applaud the authors for the rigorous work they have performed.

In their review, the authors included randomized controlled trials (RCTs) that compared active stimulation with placebo/sham, no treatment, or usual care, and RCTs comparing stimulation plus another treatment with other treatment alone. Pain intensity, adverse events, disability, health-related quality of life (HR-QoL), and analgesic medication use were evaluated. Fifteen studies in SCS and none in DRGS were eligible for inclusion. For SCS vs sham, the authors found evidence of a small beneficial effect of SCS on pain intensity (six studies) and disability (three studies) for short-term follow-up only and concluded that “SCS may not provide clinically important benefits on pain intensity” with “very low certainty.” For SCS plus other treatment vs other treatment alone, they found evidence of a potentially important clinical difference in pain intensity for short-term (three studies) and medium-term (five studies) follow-up and a beneficial effect on HR-QoL for short-term (one study) and medium-term (five studies) follow-up with SCS. From this, they concluded that SCS “may provide clinically important benefits for pain intensity when added to conventional medical management (CMM) or physical therapy” with “low to very low certainty.” They question how much of the observed benefit of SCS may result from the stimulation itself. The authors also evaluated adverse events for medium-term (five studies) and long-term (three studies) follow-up only and state that SCS is associated with relatively common complications, such as infection, and more serious adverse events. They found no evidence at all to support or refute the use of DRGS. They found limited evidence on the economic implications of SCS and DRGS. The included evidence suggests that SCS is associated with substantial additional costs of the device/apparatus and the implantation processes and the costs of managing complications. This interpretation of published evidence by the authors is concerning given the widely accepted role of SCS in multimodal pain management.^2,3 A critical assessment of the Cochrane review does raise some questions about the ways the authors arrived at their conclusions.

The Cochrane Library (ISSN 1465–1858) is a collection of data bases that contain different types of high-quality, independent evidence to inform health care decision making. Evidence-based medicine (EBM) is “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” The aim of EBM is to integrate the experience of the clinician, the values of the patient, and the best available scientific information to guide decision making about clinical management. Cochrane reviews aim to deliver a synthesis of the highest quality of evidence and are often considered to be the gold standard in EBM. However, we wonder if this Cochrane review really did gather the best available evidence. Published Cochrane reviews have occasionally been found to fall short of Cochrane standards. Conway et al^4,5 have expressed their concerns with Cochrane reviews in anesthesia, critical care, and emergency medicine, stating that, “it was common for conclusive statements to be made about the effects of interventions despite evidence for the primary outcome being rated less than high quality.” Currow et al⁶ also discussed problems with Cochrane reviews related to subjectivity, authorship, timing, and updates. Unfortunately, we feel that the review by O’Connell et al falls short of Cochrane standards for several such reasons. We present our concerns below.

ISSUES WITH INCLUSION CRITERIA

The inclusion criteria of this review are problematic. First and foremost, the authors state that they searched for all the relevant studies in the medical literature yet limited their selection of available evidence to RCTs. They state that studies must have compared SCS or DRGS with either “placebo (sham) stimulation, usual care, no treatment, or other treatments”; however, they fail to include studies of SCS vs other treatments, such as the widely cited RCT by North et al⁷ comparing SCS with reoperation, which was included in the previous Cochrane review by Mailis-Gagnon et al.⁸ The authors also note that they found no studies evaluating DRGS and “no studies that compared spinal neuromodulation interventions with no treatment or usual care.” They also note that the included sham RCTs were “small, with short-term follow-up and all employed a crossover design.” Long-term data from two of the included sham-controlled crossover trials were available (PROCESS,⁹ PROMISE¹⁰) but were excluded because they did not meet the authors’ definition of “randomized.” Thus, the authors “found no studies to inform a comparison of SCS vs placebo in the medium or long-term” and provided no further discussion on this aspect. It seems that the authors have not considered the most important reason for this, which is that implanted neuromodulation is an advanced therapy indicated for patients with severe disabilities and who are refractory to treatment, so relying on these study designs to provide evidence of long-term outcomes presents an ethical dilemma. There are serious ethical concerns with long-term maintenance of a placebo/sham control arm and not implementing a crossover phase, thus withholding an efficacious treatment from patients with refractory chronic pain who would otherwise have been eligible to receive it for long periods. Crossover not only addresses ethical imperatives but also conforms with real-world clinical practice because patients are free to seek alternatives if their current treatment is inadequate. This suggests that the authors expect unethical research studies to qualify (ie, ≥ 2 years of placebo/sham or no treatment without crossover) when RCTs of long-term efficacy already exist but were excluded, either fully or in part, owing to the tight constraints of the review methods. Although we agree with the authors that there is greater risk of bias with quasi-randomized and nonrandomized studies, RCTs are not without bias, and in the situation of a lack of RCTs, large patient cohort studies also have value. Furthermore, most RCTs on pharmacologic analgesic therapies that have placebo comparators are limited to a duration of three months.

Secondly, the authors excluded studies on ischemia-related pain without providing any clear rationale for this. There are at least ten published RCTs on ischemic/anginal pain, including those by de Jongste et al,¹¹ Jivegård et al,¹² Hautvast et al,¹³ Mannheimer et al,¹⁴ Klomp et al,¹⁵ Lanza et al,¹⁶ McNab et al,¹⁷ Eddicks et al,¹⁸ Lanza et al,¹⁹ and Zipes et al.²⁰ Data from these RCTs would have made the review and analysis more robust and useful. We do not see any compelling justification for excluding ischemic pain while including a very heterogenous group of other clinical conditions: persistent spinal pain syndrome, diabetic neuropathy, complex regional pain syndrome (CRPS), and irritable bowel syndrome.

Thirdly, the authors did not include studies that compared different forms/regimens of stimulation. On the basis of this, it was not reasonable to combine studies involving different types of SCS (conventional, burst, high frequency) as they did. The authors should have included studies that compared existing stimulation technologies with new modes/techniques (eg, high frequency, DRGS). These kinds of trial designs are less likely to be affected by placebo effects, which are often strong with interventional treatments, or nocebo effects that occur with readministration of usual care because both/all groups receive an active treatment. It would be irrational and unethical to compare new modes of stimulation solely with placebo/sham or usual care or no treatment because patients who qualified for the new mode would have been offered SCS (the standard of care) in the absence of the new mode. Comparing newer modes of stimulation with existing technologies would also have ensured assay sensitivity, which could not be assessed owing to the authors’ inclusion criteria and approach to categorizing multiple types of SCS as one treatment.

We believe that more appropriate criteria would allow the inclusion of the following:

1. RCTs that compare existing SCS technologies with new modes/techniques, for example:

   1. The SENZA RCT, Kapural et al,^21,22 and Amirdelfan et al²³—High-frequency SCS vs low-frequency SCS
   2. The ACCURATE RCT, Deer et al²⁴—DRGS vs conventional SCS
   3. The Evoke RCT, Mekhail et al²⁵—Closed-loop SCS vs open-loop SCS
2. Long-term, postcrossover data from included RCTs (which were excluded because “the data were not reported for all participants as randomized”):

   1. The PROCESS RCT, Kumar et al 2008⁹
   2. The PROMISE RCT, Rigoard et al¹⁰

3. Quasirandomized studies and prospective open-label studies (to allow a more sufficient and ethical evaluation of clinically relevant long-term outcomes).^26,27

In fact, according to the Cochrane Handbook for Systematic Reviews of Interventions,^26,27 the inclusion of nonrandomized studies is encouraged:

1. when “…non-randomized evidence might address, for example, long-term or rare outcomes…”²⁶;

2. “To provide evidence of the effects (benefit or harm) of interventions that cannot be randomized, or that are extremely unlikely to be studied in randomized trials…”²⁶;

3. “To provide evidence of the effects (benefit or harm) of interventions that can feasibly be studied in randomized trials, but for which only a small number of randomized trials is available (or likely to be available)”²⁶;

4. “…when the evidence from randomized trials is rated as very low and non-randomized studies provide evidence of higher certainty.”²⁷

INSUFFICIENT LONG-TERM EVIDENCE

We also take issue with the statement that the long-term outcomes for SCS are supported by only one study: Kemler et al.^28–30 The study enrolled 24 patients who underwent implantation in the years 1997 to 1998 with a single Medtronic four-contact lead into an Itrel III pulse generator, and 18 control patients, followed up at six months, two years, and five years, respectively. These patients had an average of 3.3 years of CRPS before enrollment. It almost beggars belief that we would refer to the efficacy of existing medical technology by referring to limited early outcomes observed 24 to 25 years ago. That would be akin to saying that the current state of the art regarding aortic valve replacements is reflected in the clinical results of a cohort of patients implanted with an aortic valve in 1997. A subsequent study published in 2013 presented up to 11 years follow-up of 84 patients with CRPS implanted with legacy devices, noting that 63% of patients had effective long-term pain control.³¹ We regard 11 years of data as a more meaningful and comprehensive assessment of the technology that was available in 1997. In fact, when one looks comprehensively at long-term data for SCS, there is RCT evidence at 24-month outcomes from the included PROMISE¹⁰ and PROCESS⁹ RCTs (postcrossover data were excluded), the excluded SENZA RCT,²² and the recently published Evoke RCT.³² The results of these studies are presented in Table 1.

Table 1.

Twenty-Four–Month RCT Evidence for SCS.

Study Outcome	EVOKE RCT Closed-loop SCS	PROCESS RCT Open-loop SCS (MDT)	SENZA RCT Open-loop SCS (Test [NVR])	SENZA RCT Open-loop SCS (Control [BSC])	PROMISE RCT Open-loop SCS (MDT)
Pain (primary outcome)	Overall back and leg, VAS	Leg, VAS	Back, VAS	Back, VAS	Back, NRS
Percent change from baseline (mean)	72.60%	43.5%*	66.90%	41.10%	29.3%*
≥ 50% Reduction from baseline	84.00%	40.50%	76.50%	49.30%	20.60%
≥ 80% Reduction from baseline	50.00%	14.30%	43.5%*	19.7%*	NR
ODI
Change from baseline (mean)	26	151	NR**	NR***	9.4
Percent change from baseline (mean)	47.80%	20.3%*	NR†	NR††	NR
Percent minimal to moderate	78.00%	NR	64.70%	49.30%	NR
EQ-5D Index
Change from baseline (mean)	0.254	0.271	NC	NC	0.18
SF-12 PCS
Change from baseline (mean)	10.1	NR in Kumar et al9; 5*,††† in Eldabe 200933	NR ‡	NR ‡‡	6.56
SF-12 MCS
Change from baseline (mean)	6.7	NR in Kumar et al9; 5*,††† in Eldabe 200933	NR ‡‡‡	NR§	NR
POMS
Change from baseline (mean)	18.6	NC	NC	NC	NC
PSQI
Change from baseline (mean)	4.1	NC	NR§§	NR§§§	NR
PGIC
Percent very much or much improved	84.00%	NC	63.5%¶	36.60%	NR
Study Outcome	EVOKE RCT Closed-loop SCS	PROCESS RCT Open-loop SCS (MDT)	SENZA RCT Open-loop SCS (Test [NVR])	SENZA RCT Open-loop SCS (Control[BSC])	PROMISE RCT Open-loop SCS (MDT)
Opioid usage
Percent decrease MME (mean)	42.30%	−2.2% / −6.1%* (increases in MME for patients with low use and those with high use patients)	NRs	NR	NR
Percent reduced or eliminated	66.70%	NR	NR¶¶	NR¶¶¶	NR

Reproduced with permission from the authors.³²

BSC, Boston Scientific Corporation; EQ-5D, European Quality of Life Five-Dimensional (multiplied by a factor of 100 to represent visually); HR-QoL, Health-Related Quality of Life; MDT, Medtronic; NC, not collected; NR, not reported; NVR, Nevro; ODI, Oswestry Disability Index; PGIC, Patient Global Impression of Change; POMS, Profile of Mood States (TMD, Total Mood Disturbance); PSQI, Pittsburgh Sleep Quality Index; SF-12, Short-Form Health Survey (MCS, mental component score; PCS, physical component score); VAS, visual analog scale.

^*Estimated from data provided in the publication.

^**Senza RCT open-loop SCS test: 16.5 mean change from baseline in ODI at 12 months (Kapural 2015).²¹

^***Senza RCT open-loop SCS control: 13.0 mean change from baseline in ODI at 12 months (Kapural 2015).²¹

^†Senza RCT open-loop SCS test: 29.2% mean percentage change from baseline in ODI at 12 months (SSED P130022).³⁴

^††Senza RCT open-loop SCS control: 21.6% mean percentage change from baseline in ODI at 12 months (SSED P130022).³⁴

^†††SF-36.

^‡Senza RCT open-loop SCS test: 8.1 mean change from baseline in SF-12 PCS at 12 months (SSED P130022).³⁴

^‡‡Senza RCT open-loop SCS control: 6.0 mean change from baseline in SF-12 PCS at 12 months (SSED P130022).³⁴

^‡‡‡Senza RCT open-loop SCS test: 2.7 mean change from baseline in SF-12 MCS at 12 months (SSED P130022).³⁴

^§Senza RCT open-loop SCS control: 1.2 mean change from baseline in SF-12 MCS at 12 months (SSED P130022).³⁴

^§§Senza RCT open-loop SCS test: 2.6 mean change from baseline in PSQI at 12 months (SSED P130022).³⁴

^§§§Senza RCT open-loop SCS control: 1.8 mean change from baseline in PSQI at 12 months (SSED P130022).³⁴

^¶Percent a great deal better or better.

^¶¶Senza RCT open-loop SCS test: 35.5% patients reduced or eliminated opioids at 12 months (SSED P130022).³⁴

^¶¶¶Senza RCT open-loop SCS control: 26.4% patients reduced or eliminated opioids at 12 months (SSED P130022).³⁴

CONCERNS WITH METHODS

We were particularly concerned with the classification of follow-up periods and stimulation parameter subgroups that restricted the data used for analysis. Firstly, outcome measurement time points were restricted to short-term (≤ 1 month), medium-term (4–8 months) and long-term (≥ 12 months). When there were multiple follow-ups within these periods, the authors took the earliest measurement for short term, and the latest measurement for medium and long term. We question how these time points provide helpful outcomes, particularly for the short term, when the earliest measurement may be affected by the implant surgery and the programming may not yet be optimized. A primary outcome of at least three months would be much more reasonable. Furthermore, outcomes and adverse events from studies that performed follow-up in the intervening periods (ie, one to three and nine to 11 months) were hence missed.

Secondly, the subgroup classifications for different stimulation parameters seem somewhat arbitrary, and no rationale was provided, showing a lack of understanding of the clinical application of SCS. High frequency was defined as 1 kHz to 10 kHz, yet some would classify up to 1200 Hz as conventional, whereas others classify 500 Hz as high frequency; however, preclinical data suggest superiority of effects of high-frequency stimulation on wide dynamic range neurons to occur at stimulation frequencies > 5 kHz.³⁵ This meant that data from three studies were excluded because the stimulation parameters could not be fit into arbitrary subgroup classifications, yet two articles were included (with negative results) that were experimental investigations of waveforms that are not clinically applicable (3 kHz and 5 kHz).^36,37 This is a conflation of basic research work (involving human subjects) and clinical delivery of care, without regard to correct methods. This profoundly weakens the confidence in relying on the conclusions of the Cochrane study.

CONCERNS WITH PRESENTATION OF THE RESULTS

It is also unclear why the authors chose to acknowledge, within the context of adverse event reporting, that most sham-controlled studies recruited patients who had already been implanted with devices but failed to acknowledge the impact of the same fact on reported pain relief. It is logical that studies recruiting for a comparison between sham and an experimental SCS waveform are likely to attract a subpopulation of participants with less-than-perfect pain relief, thereby resulting in higher baseline pain scores and a lower likelihood of response to SCS, thus contributing to what the review authors choose to call no convincing evidence of SCS vs placebo/sham. It is interesting to note the authors review largely the same studies as Duarte et al³⁸ did in their systematic review. Contrary to Duarte et al, however, they concluded that there was no convincing evidence of SCS being effective vs placebo/sham; meanwhile, they concluded that SCS combined with CMM or physical therapy had large effects on pain intensity at both short- and medium-term follow-up compared with CMM or physical therapy alone. This finding defies biologic rationale and makes one wonder whether the included studies and/or the analytical approach were appropriate. Furthermore, the authors state, “at long-term follow-up we found no clear evidence for a benefit of SCS on average pain scores (very low certainty), and evidence of a large effect on the proportion of participants experiencing ≥ 50% pain relief (very low certainty).” This finding again raises questions about the approach to data synthesis and analysis.

In addition, there was inconsistent reporting in terms of the summary of findings tables, with probable outcomes for SCS reported as difference in mean pain intensity scores between SCS and control groups, whereas probable outcome for sham/control was reported as mean pain scores in sham/control groups.

We do agree with the authors’ judgement that all SCS vs sham results were at high risk of bias on more than one domain; however, there was no acknowledgment that adequate blinding is extremely difficult for these studies.

COMMENTS ON DISCUSSION AND CONCLUSIONS

One of the key shortcomings of the discussion and conclusions is that the authors failed to place SCS in the broader context of pain treatment. It is noted that many of the included patients had incredibly high pain levels to begin with (ie, they have failed most everything else), which can make it very challenging to discern whether the treatment is truly ineffective for pain or whether certain patients are more likely to benefit. Cost-effectiveness and adverse events were discussed without considering the alternative for many of these patients (ie, surgery and opioids, both of which are associated with increased health care costs and can have serious morbidity/mortality, and whose long-term efficacy is also debatable).

In terms of missing data, of course, as the authors note, it is highly likely that the data are not missing at random. This is not unique to SCS. It is an acknowledged shortcoming of most anal-gesic trials; very few pain studies report on long-term outcomes, and loss to follow-up, treatment nonadherence, and crossover to other arms are extremely high. There was also no consideration of the possibility that patients with SCS who are doing well may contribute to missing data by not presenting for follow-up. This attrition bias can result in data being influenced by patients who are not doing well and who continue to seek care at the SCS clinic in hope of pain relief.

The authors also make the comment that three SCS vs sham studies “might be considered to be of an enriched enrolment design as participants were pre-selected on the basis of their outcomes following SCS.” These were all studies that looked at high frequency (HF) or burst in patients who already responded to traditional SCS. Terming those studies “enriched enrolment” presupposes that the mechanism of action between traditional SCS and the study intervention is the same, and that the same patients benefit from both. However, if anything, preclinical data and clinical experience suggest just the opposite: that there are some patients who might respond better to traditional SCS, some who might respond better to HF stimulation, and some who might respond better to burst stimulation. Analyzing all these studies together fails to appreciate that these are not the same treatment. The authors state that another three studies might be considered to be of an enriched enrollment design because they randomized participants who showed a positive response to a trial of SCS, which is very acceptable in clinical practice (part of the diagnostic workup), particularly for those diagnoses that have different phenotypes, of which some will be responders and others not (eg, persistent spinal pain syndrome, CRPS).

Lastly, we agree with the authors regarding the IMMPACT recommendations. However, there needs to be a clearer statement from the authors to indicate that much of the existing data do not adhere to these recommendations, so we still cannot make any definitive conclusions regarding efficacy of SCS. We would argue that in this case, a meta-analysis is not the appropriate method for examining whether SCS works. As noted, many more recently published large-scale studies are comparative effectiveness/non-inferiority studies. Admittedly, those studies have their own challenges, but possibly a better approach to answer the question with all available data is to perform a network meta-analysis, rather than a traditional meta-analysis with curiously restrictive inclusion criteria and high clinical and methodologic heterogeneity.

CONCLUSIONS

Unfortunately (and with significant regret), we can only conclude that the Cochrane review as it stands is not fit for purpose and falls regrettably short of the Cochrane recommendations in several areas on how to conduct such a review.^26,27 The goal the authors set out to achieve, however, is a very worthy one and should continue to be explored. From the vantage point of our authors’ extensive experience in working in this field, we have seen a progression of clinical results over the last three decades that have improved on therapy that previously only sometimes worked over the long term, to now providing sustained analgesic benefit for a group of patients with truly refractory conditions for whom there are few realistic alternative options available.

This changing landscape of clinical results with improved device capabilities over time almost mandates a decade-by-decade analysis of results, which is likely to be more meaningful for current patients than is attempting to incorporate results from 30 years ago with results from 12 months ago as if they are the same thing. This should be obvious from the fact that the devices from 30 years ago are no longer on the market and can no longer be implanted into patients. Those results therefore inform current practice but only to a degree, as notes of historical interest in the development of the field.

Although having authors with no experience in practicing neuromodulation attempt a synthesis of the literature of the field may present the benefit of removing bias from the analysis, lack of appropriate clinical content experts is also likely to result in failure to appropriately interpret the field literature. It would seem intuitively obvious to us that the optimum is a combination of experts and nonpractitioners in the field to bring the best blend of experience, independent thought, and objective lack of bias into one holistic working group. We wonder whether a reconstituted Cochrane group along these lines would struggle less with a comprehensive synthesis of the data.

A final word about designs of studies and the use of a placebo as comparator: it is too easy to say that studies are only convincing if the comparator is a placebo or sham treatment. There are serious ethical barriers to doing this for a long time. Not all medical ethical committees accept long-term placebo or sham treatment studies. Very acceptable alternatives are large cohort studies, data gathered in real-world studies, or n-of-1 studies. There are claims that meta-analyses of n-of-1 studies may be even more convincing than are RCTs.^39–41 Another approach is, of course, that we try to better understand the underlying mechanisms. We must realize that RCTs are simply an artificial way of extrapolating certainty about effectiveness (real-world evidence). For example, it should be considered that there are zero meta-analyses of RCTs that show long-term efficacy of insulin in type 2 diabetes.⁴² Because we understand the underlying mechanism, RCTs are no longer necessary.

In summary, we do not think this Cochrane initiative should be abandoned but believe it should be renewed and reinvestigated to address some of the methodologic concerns that we have outlined in this article. It is through overt transparency and constructive dialogue that the best techniques can be applied to diverse data sets to inform clinical care.

COMMENTS.

In recent years, we have witnessed an attack on the validity and economic sustainability of SCS as a technique for pain control in patients who usually lack other therapeutic options to improve, at least in part, their quality of life. Review methods are often questionable regarding the selection of studies for inclusion and outcome measures. As for all interventional techniques, placebo or no-intervention control (who decides which patients should undergo surgery and which should not?) is not ethical, as stated in the paper, but although they are not claimed for other surgeries (for example, the same back surgeries that in some patients do not improve or even worsen pain and disability), for pain, management techniques seem essential. Outcome measures are really a problem in pain management: in patients with chronic pain, many causes and bio-psycho-social components contribute to the symptoms, probably in evolution with time, Visual analog scales and numeric ranking scales are poor indicators of improvement; functional and quality-of-life scales can enhance the perception of improvement but are not enough to describe the effect of SCS on some pain components and mechanisms. Outcome measures lose their meaning in the long term with the evolution of a clinical picture of patients. Possibly the best way to understand the usefulness of SCS is to evaluate, in real-life studies, the number (percentage) of patients still using their SCS system in the long term. There are studies evaluating the explant rates in the long term, but every Pain Center with experience can count on patients coming regularly for controls or generator changes for > 20 years.

Laura DeMartini, MD

Pavia, Italy

The authors conducted a critical appraisal on the published Cochrane review about implanted spinal neuromodulation for chronic pain with a specific focus of applying key aspects in clinical research on the field of neuromodulation. More specifically, the selection of timepoints to evaluate outcome measures, the ethical discussion of applying long-term placebo and control treatment arms in severely disabled patients and the value of longitudinal cohort data vs RCT data are extensively discussed. This clearly points out the difference between “theoretical” research and “clinical/applied” research and the complementary need for experts in both fields. As concluded by the authors, a network meta-analysis may provide better insight in this research question, which allows simultaneous comparisons of multiple treatment options and presumably will lead to comprehensive evidence in the field of managing chronic pain. Currently, initiatives are ongoing to conduct a network meta-analysis to evaluate treatment options for patients with persistent spinal pain syndrome post-surgery (PSPS Type 2), an idea that is perfectly in line with the suggestions for future research of the authors (PROSPERO CRD42022360160).

Lisa Goudman, PhD, MSc

Brussels, Belgium