Superiority Claims for Spinal Devices: A Systematic Review of Randomized Controlled Trials

Abstract

Study Design:

Systematic review.

Objectives:

Superiority claims for medical devices are commonly derived from noninferiority trials, but interpretation of such claims can be challenging. This study aimed to (a) establish the prevalence of noninferiority and superiority designs among spinal device trials, (b) assess the frequency of post hoc superiority claims from noninferiority studies, and (c) critically evaluate the risk of bias in claims that could translate to misleading conclusions.

Methods:

Study bias was assessed using the Cochrane Risk of Bias Tool. The risk of bias for the superiority claim was established based on post hoc hypothesis specification, analysis of the intention-to-treat population, post hoc modification of a priori primary outcomes, and sensitivity analyses.

Results:

Forty-one studies were identified from 1895 records. Nineteen (46%) were noninferiority trials. Fifteen more (37%) were noninferiority trials with a secondary superiority hypothesis specified a priori. Seven (17%) were superiority trials. Of the 34 noninferiority trials, 14 (41%) made superiority claims. A medium or high risk of bias was related to the superiority claim in 9 of those trials (64%), which was due to the analyzed population, lacking sensitivity analyses, claims not being robust during sensitivity analyses, post hoc hypotheses, or modified endpoints. Only 4 of the 14 (29%) noninferiority studies provided low bias in the superiority claim, compared with 3 of the 5 (60%) superiority trials.

Conclusions:

Health care decision makers should carefully evaluate the risk of bias in each superiority claim and weigh their conclusions appropriately.

Introduction

Randomized controlled trials (RCTs) are pivotal in establishing the safety and efficacy of novel spinal devices. Spinal device trials are designed either as noninferiority (NI) or superiority trials. In NI trials, the aim is to demonstrate that an investigational device is similar to an accepted surgical procedure or device by showing that the investigational device is not worse (by a small margin). In superiority trials, the goal is to show that the investigational device is superior to a control treatment, which may be nonsurgical care or a gold standard surgical procedure.¹

In the United States, most investigational device exemption (IDE) studies of novel spinal devices are designed as NI trials because of effect size, secondary benefits, and ethical considerations.^2,3 Many of these NI trials also test for superiority of the investigational device (NI + S), since sponsors are under pressure from physicians and payers to show improvements in safety, efficacy, and cost-effectiveness. The nuances associated with post hoc tests of superiority in this setting can make interpretation of such superiority claims challenging and potentially misleading.⁴ The aims of NI versus superiority trials differ substantially, so the methodology associated with design, analysis, and interpretation is also different. For example, it is conservative to analyze the intention-to-treat (ITT) population for superiority analyses, but it is not conservative for NI analyses since any confounding events will drive the result toward equivalence.^1,5-7 Additionally, post hoc specification of hypotheses must be avoided in confirmatory trials,⁴ which requires that superiority analyses are well-defined in the statistical plan a priori. Finally, it is critical to address not only the statistical superiority but also the clinical significance of the differences observed. This is particularly true when a NI margin is imposed for the primary analysis, so that interpretation can be symmetric with less potential for bias.⁸

The purpose of this study was to review the literature for reports of randomized controlled trials of spinal devices from the year 2000 to present. For each report, the primary study design was classified as NI, superiority, or NI with an additional predefined superiority analysis (NI + S). For each trial, superiority claims were identified and were assessed for potential sources of bias by multiple reviewers using a standardized tool. The hypothesis was that NI trials would predominate, and that superiority claims derived from NI trials would have a greater risk of potential bias and less reliability.

Methods

Study Selection

This systematic review was performed according to the guidelines provided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.⁹ Search criteria were developed to identify RCTs of medical devices or biologics for the spine through PubMed/MEDLINE, Embase, ClinicalTrials.gov, the World Health Organization’s International Clinical Trials Registry Platform (ICTRP), as well as the Food and Drug Administration’s (FDA) databases on premarket approvals (PMA), postapproval studies (PAS), and proceedings from FDA advisory committee meetings of the Orthopaedic and Rehabilitation Devices Panel. Search filters included information available in the English language and publication since the year 2000 to focus on more recent trends in trial design, analysis, and interpretation. Search terms and inclusion/exclusion criteria for record screening are summarized in Table 1 for the PubMed and Embase searches while further details for these and the other databases are provided in Appendix A (see Supplementary Material available in the online version of the article). Two independent researchers screened the identified records for inclusion and exclusion. The final search of each database was completed between May 15 and June 15, 2018. When relevant studies were identified through one database, the other databases were further queried to identify protocols or reports that may provide supplemental study information for data extraction. Only the primary endpoint and primary outcomes were evaluated in this review, considering those were the basis for trial design.

Table 1.

Search Terms and Screening Criteria Used for the PubMed and Embase Databases.

Anatomical Terms	Inclusion Criteria
1. Spine 2. Lumbar 3. Lumbosacral 4. Thoracolumbar 5. Cervical 6. Intervertebral disc	Spinal devices or biologics Randomized controlled trials (RCTs) with description of trial design (noninferiority or superiority) Reports on the primary study endpoint
Device Terms	Exclusion Criteria
7. Device 8. Instrumentation 9. Spacer 10. Cage 11. Interbody fusion 12. Annular (anular) closure 13. Fixation 14. Stimulator 15. Prosthesis 16. Replacement 17. Arthroplasty 18. Biomaterial 19. Biologic	Drugs, diagnostics, or nonsurgical management methods Surgical techniques or other non–device-based interventions (navigation and robot-assisted surgery were inclusive) Nonclinical studies (eg, preclinical research or economics analyses) Non-RCT studies (eg, single-arm or nonrandomized studies) Insufficient description of study design (eg, noninferiority or superiority) Information from multiple publications about a single clinical study were merged, but only for the primary endpoint
Study Design Terms	Search Filters
20. Investigational device exemption (IDE) 21. Non-inferior(-ity) 22. Noninferior(-ity) 23. Superiority 24. Randomized controlled trial (RCT) 25. Meta-analysis 26. Review	Language: English Years: 2000 to Present Study type: Randomized Controlled Trial
Search Term Combination Strategy
[1-6]/or AND {([7-19]/or AND [24]) or [20-24]/or} NOT [25-26]/or

Data Extraction

Relevant data was extracted from each included study by 2 independent researchers. Discrepancies in identifying the study design were resolved through discussion and identification of additional, clarifying documentation in five cases. The data of interest for this review included the study objective, hypotheses for primary endpoints (NI or superiority), margins or effect size used in trial design, primary outcomes and endpoints, sample sizes, conclusions or claims made in the report (NI or superiority), treatment effects of superior devices, and any statistical or clinical significance considerations relating to the superiority claims. When multiple articles reported on the same study (eg, outcomes at different time points), each article was screened for the data of interest related to the a priori study design and primary endpoint.

Risk of Bias Assessment

The general risk of bias was evaluated for each study using the Cochrane Risk of Bias Tool for Randomized Controlled Trials¹⁰ and interpreted according to the key domains described by Pavon et al¹¹ (Supplementary Table S1, Appendix B). The reporting of any financial disclosures, or lack thereof, was also noted but was not considered in the overall risk of bias evaluation. Additionally, the risk of bias specifically related to superiority claims was assessed. The criteria for this assessment included analyses that were not specified a priori, analysis of the ITT population, post hoc modification of primary outcomes, and any sensitivity analyses performed on the analysis population or missing value imputation (Supplementary Table S2, Appendix B). The reporting of confidence intervals for superiority claims was also noted but was not considered in the overall risk of bias for the superiority claim.

Results

Overview of Included Studies

Across all 7 databases, 1895 unique records were identified, and 41 unique studies met the inclusion/exclusion criteria (Figure 1). Among these 41 studies, the most common investigational spinal devices were cervical disc replacements (9/41; 22%), followed by interspinous/interlaminar spacers (7/41; 17%), biologics used to support spinal fusion (7/41; 17%), lumbar disc replacements (6/41; 15%), vertebroplasty materials (2/41; 5%), spinal cord stimulators (2/41; 5%), interbody fusion cages (2/41; 5%), and 1 each (2%) of a dural sealant, an adhesion barrier gel, an annular closure device, a sacroiliac joint fusion device, a dynamic posterolateral pedicle screw system, and a surgical robot used for pedicle screw placement (Table 2).

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram demonstrating flow of records from identification through inclusion.

Table 2.

Summary of Included Studies.

Device Type	Device	Study Identifier	Citation	Study Comparators	Sample Size	Risk of Study Bias
Cervical disc replacement	BRYAN	NCT00437190 PMA P060023	Heller et al, 200933	BRYAN vs ACDF at 2 years	Test: 242 Control: 221	L
	Prestige ST	NCT00642876 PMA P060018	Mummaneni et al, 200715	Prestige ST vs ACDF at 2 years	Test: 276 Control: 265	M
	Prestige LP	NCT00637156 PMA P090029	Gornet et al, 201737	2-level Prestige vs ACDF at 2 years	Test: 209 Control: 188	L
	ProDisc-C	NCT00291018 PMA P070001	Murrey et al, 200940	ProDisc-C vs ACDF at 2 years	Test: 103 Control: 106	M
	Kineflex|C	NCT00374413	Coric et al, 201142	Kineflex|C vs ACDF at 2 years	Test: 136 Control: 133	M
	Mobi-C	NCT00389597 PMA P110002 PMA P110009	Hisey et al, 201451	1-level Mobi-C vs ACDF at 2 years	Test: 164 Control: 81	L
			Davis et al, 201335	2-level Mobi-C vs ACDF at 2 years	Test: 225 Control: 105	L
	Secure-C	NCT00882661 PMA P100003	Vaccaro et al, 201316	SECURE-C vs ACDF at 2 years	Test: 240 Control: 140	M
	PCM Cervical Disc	NCT00578812 PMA P100012	Phillips et al, 201341	PCM vs ACDF at 2 years	Test: 189 Control: 153	M
Lumbar disc replacement	Charite	NCT00215306 PMA P040006	Blumenthal et al, 200552	Charite vs ALIF at 2 years	Test: 205 Control: 99	M
	ProDisc-L	IDE #G010133 PMA P050010	Zigler et al, 200743	1-level ProDisc-L vs fusion at 2 years	Test: 161 Control: 75	L
		NCT00295009	Delamarter et al, 201153	2-level ProDisc-L vs fusion at 2 years	Test: 165 Control: 72	M
	MAVERICK	NCT00635843	Gornet et al, 201117	MAVERICK vs ALIF at 2 years	Test: 405 Control: 172	L
	Kineflex	NCT00292292	Pettine et al, 201126	Kineflex vs Charite at 2 years	Test: 33 Control: 31	M
	activL	NCT00589797 PMA P120024	Garcia et al, 201518	activL vs ProDisc-L or Charite at 2 years	Test: 218 Control: 106	L
Biologic: Fusion	OP-1 Putty	NCT00677950	Vaccaro et al, 200854	OP-1 vs autograft (noninstrumented fusion) at 2 years	Test: 208 Control: 87	L
		ISRCTN43648350	Delawi et al, 201655	OP-1 vs autograft (instrumented fusion) at 1 year	Test: 60 Control: 59	L
	Novosis (rhBMP-2)	NCT01764906	Cho et al, 201756	Novosis vs autograft at 6 months	Test: 42 Control: 51	L
	Bonion	NCT01615328	Yi et al, 201557	Bonion vs. βTCP/HA allografts at 2 years	Test: 38 Control: 39	M
	i-Factor	NCT00310440 PMA P140019	Arnold et al, 201658	i-Factor vs autograft at 1 year	Test: 165 Control: 154	M
	INFUSE with LT-cage	PMA P000058	Burkus et al, 200259 and 200360	INFUSE vs autograft at 2 years	Test: 143 Control: 136	L
	AMPLIFY (rhBMP-2)	PMA P050036	FDA Executive Summary61	AMPLIFY vs autograft at 2 years	Test: 239 Control: 224	L
Adhesion barrier	Oxiplex/SP	PMA P070023	Rhyne et al, 201262 and FDA Executive Summary24	Surgery + Oxiplex vs surgery alone at 6 months	Test: 177 Control: 175	L
Vertebroplasty	Cortoss	NCT00290862	Bae et al, 201263	Cortoss vs PMMA at 2 years	Test: 162 Control: 94	L
	Kiva	NCT01123512	Tutton et al, 201564	Kiva vs. balloon kyphoplasty at 1 year	Test: 147 Control: 153	L
Dural Sealant	Adherus	NCT01158378 PMA P130014	Strong et al, 201727	Adherus vs DuraSeal at 4 months	Test: 124 Control: 126	L
Interlaminar/Spinous device	Superion	NCT00692276 PMA P140004	Patel et al, 201528	Superion vs X-Stop at 2 years	Test: 190 Control: 201	M
	Coflex	NCT00534235 PMA P110008	Davis et al, 201346	Coflex vs fusion at 1-2 levels at 2 years	Test: 230 Control: 114	L
		NCT01316211	Schmidt et al, 201825	Coflex vs decompressive surgery at 2 years	Test: 115 Control: 115	L
	Aperius	NCT00905359	Meyer et al, 201847	Aperius vs decompressive surgery at 2 years	Test: 82 Control: 81	M
	X-Stop	N/A	Strömqvist et al, 201348	X-Stop vs decompressive surgery at 2 years	Test: 50 Control: 50	M
		PMA P040001	FDA SSED44 and memo23	X-Stop vs nonsurgical management at 2 years	Test: 100 Control: 91	M
	DIAM	IDE G050025 PMA P140007	FDA Executive Summary21	DIAM vs nonsurgical management at 1 year	Test: 181 Control: 100	H
Annular closure device	Barricaid	NCT01283438	Thome et al, 201819	Barricaid vs discectomy only at 2 years	Test: 276 Control: 278	L
Sacroiliac joint fusion	iFuse	NCT01640353	Whang et al, 201520	iFuse vs nonsurgical management at 6 months	Test: 102 Control: 46	M
Interbody fusion cage	Novomax	N/A	Lee et al, 201629	NovoMax vs titanium cage at 1 year	Test: 41 Control: 39	L
	BAK/C	PMA P980048	Hacker et al, 200065 and FDA SSED66	BAK/C vs bone grafting at 2 years	Test: 164 Control: 134	M
Dynamic stabilization	Dynesys	NCT00759057 PMA P070031	FDA Executive Summary67	Dynesys vs posterolateral fusion at 2 years	Test: 253 Control: 114	M
Spinal cord stimulator	Senza	NCT01609972 PMA P130022	Kapural et al, 201630	Senza vs low-frequency stimulation at 2 years	Test: 101 Control: 97	L
	Axium	NCT01923285 PMA P150004	FDA SSED31	Axium vs marketed control device at 3 months	Test: 76 Control: 76	M
Surgical robot	Renaissance	NCT02121249	Kim et al, 201722	Robot-assisted pedicle screw accuracy vs freehand	Test: 37 Control: 41	L

Abbreviations: L, low; M, medium; H, high; N/A, not applicable; FDA, US Food and Drug Administration; SSED, Summary of Safety and Effectiveness Data.

There were 19 (46%) studies designed as NI trials, 15 (37%) studies designed as NI + S trials, and 7 (17%) studies designed as superiority trials. Five of the 7 superiority trials were reported within the past 3 years (Figure 2). A composite clinical success (CCS) criterion was the most common primary outcome measure and was typically defined as: an improvement in a patient reported outcome greater than a clinically relevant threshold; the absence of secondary surgical interventions or procedures; the absence of neurologic deterioration; the absence of device and/or procedure related serious adverse events; and possibly radiographic findings.¹²

Figure 2.

Trends in noninferiority versus superiority designs for randomized controlled trials of spinal devices that were included in this review since the year 2000.

Sample size calculations were often performed using the methods described by Blackwelder et al^13,14 with a NI margin of 10% (Table 3). A few studies assumed other margins for power calculations, but data was also analyzed with a 10% margin at the request of the FDA.^15-18 No study estimated the NI margin from a prior superiority study that measured the effect size compared with sham or placebo. Three of the superiority studies used Bayesian methods for sample size.^19-21 Two assumed superiority effect sizes of 9%²² and 23%.²³ One did not describe its power analysis²⁴ and one assumed a medium effect size (Cohen’s d=0.4) for differences in disability scores.²⁵ Three superiority studies compared with nonsurgical management,^20,21,23 while the rest of the studies used an active surgical control. The active surgical controls represented a standard treatment technique for the respective condition (eg, fusion as a control for disc replacements and autograft for biologics). Seven of the studies compared with devices of the same class that were already available on the market.^18,26-31

Table 3.

Summary of Study Design Characteristics and Conclusions.

Device Type	Device	Citation	Hypotheses	Primary Outcome	Margin	Report Claim	Risk of Superiority Claim Bias	Superiority Outcomes and Considerations
Cervical disc replacement	BRYAN	Heller et al, 200933	Primary: NI Secondary: S	CCS	NI: 10%	S	H	Test CCS: 83% (95% CI: 77%-87%) Control CCS: 73% (95% CI: 66%-79%) P = .01, one-sided FET; S claim not supported by PP analysis or sensitivity analyses of missing values
	Prestige ST	Mummaneni et al, 200715	Primary: NI Secondary: S	CCS	NI: 10%	S	M	Test CCS: 79% Control CCS: 68% P = .005, one-sided FET; Analysis population not described; FDA panel recommended limit of NI claim only
	Prestige LP	Gornet et al. 201737	Primary: NI Secondary: S	CCS	NI: 10%	S	H	Test CCS: 81% Control CCS: 69% BPP = 0.99; As-treated analysis, 8%-18% of patients received different treatment; no sensitivity analysis
	ProDisc-C	Murrey et al, 200940	NI	CCS	NI: 10%	CCS: NI mCCS: S	H	Test mCCS: 74% Control mCCS: 61% P = .047, one-sided FET; Original CCS confirmed NI, post hoc mCCS used for superiority test; analysis population not described
	Kineflex C	Coric et al. 201142	NI	CCS	NI: 10%	S	H	Test CCS: 85% Control CCS: 71% P = .05, 2-sided FET; Superiority testing not prespecified; analyzed population not described
	Mobi-C	Hisey et al, 201451	Primary: NI Secondary: S	CCS	NI: 10%	NI	N/A	N/A
		Davis et al, 2013a35	Primary: NI Secondary: S	CCS	NI: 10%	S	H	Test CCS: 70% Control CCS: 37% P < .0001, Farrington-Manning test; Analysis of as-treated population; Sensitivity analyses only described to support NI
	Secure-C	Vaccaro et al, 201316	Primary: NI Secondary: S	CCS	NI: 10%	S	L	Test CCS: 84% Control CCS: 73% BCI: 0.6%-20%, BPP = 0.98; Sensitivity analysis not performed for S claim
	PCM	Phillips et al, 201341	NI	CCS	NI: 12.5%	S	H	Test CCS: 75% (95% CI: 69%-81%) Control CCS: 65% (95% CI: 57%-73%) P = .02, one-sided Z-test; S analysis not defined a priori, no ITT analysis
Lumbar disc replacement	Charite	Blumenthal et al, 200552	NI	CCS	NI: 15%	NI	N/A	N/A
	ProDisc-L	Zigler et al, 200743	NI	CCS	NI: 12.5%	S	H	Test CCS: 53% Control CCS: 41% P = .044, Stat method not specified; S analysis not defined a priori, ITT population not analyzed, 50 nonrandomized patients included
		Delamarter et al, 201153	NI	CCS	NI: 12.5%	NI	N/A	N/A
	MAVERICK	Gornet et al, 201117	Primary: NI Secondary: S	CCS	NI: 10%	S	L	Test CCS: 74% Control CCS: 55% P < .001, one-sided FET
	Kineflex	Pettine et al, 201126	NI	ODI and VAS scores	NI: 10pt ODI and 18pt VAS	NI	N/A	N/A
	activL	Garcia et al, 201518	Primary: NI Secondary: S	CCS	NI: 10%	S	L	Treatment difference ∼ 14% P = .02, O’Brien-Fleming sequential spending function
Biologic: Fusion	OP-1 Putty	Vaccaro et al, 200854	NI	CCS	NR	NI	N/A	N/A
		Delawi et al, 201655	NI	CCS	NI: 15%	Inferior	N/A	N/A
	Novosis (rhBMP-2)	Cho et al, 201756	NI	Fusion rate	NI: 10%	NI	N/A	N/A
	Bonion	Yi et al, 201557	NI	Fusion rate	NI: 15%	NI	N/A	N/A
	i-Factor	Arnold et al, 201658	Efficacy: NI Safety: S	CCS	NI: 10-15%	S	H	Test CCS: 69% Control CCS: 57% P = .038, Wald asymptotic approach; Failed safety superiority endpoint; S claim based on CCS that was not predefined in statistical plan
	INFUSE with LT-cage	Burkus et al, 200259 and 200360	Primary: NI Secondary: S	Fusion rate	NR	NI	N/A	N/A
	AMPLIFY (rhBMP-2)	FDA Executive Summary61	Primary: NI Secondary: S	CCS	NI: 10%	NI	N/A	N/A
Adhesion barrier	Oxiplex/SP	FDA Executive Summary24	S	LSOQ	S: 0	Not S	N/A	N/A
Vertobroplasty	Cortoss	Bae et al, 201263	NI	VAS and ODI	\NI: 12.5%	NI	N/A	N/A
	Kiva	Tutton et al, 201564	Primary: NI Secondary: S	CCS	NI: 12.5%	NI	N/A	N/A
Dural sealant	Adherus	Strong et al, 201727	NI	CCS	NI: 10%	NI	N/A	N/A
Interlaminar/Spinous spacer	Superion	Patel et al, 201528	NI	CCS	NI: 10%	NI	N/A	N/A
	Coflex	Davis et al, 2013b46	Primary: NI Secondary: S	CCS	NI: 10%	NI	N/A	N/A
		Schmidt et al, 201825	S	CCS	S: 0	S	L	Test CCS: 58% Control CCS: 42% 95% CI of difference: 3%-30% P = .017; No sensitivity analysis
	Aperius	Meyer et al, 201847	NI	PF of ZCQ	NI: 10%	NI	N/A	N/A
	X-Stop	Strömqvist et al, 201348	NI	ZCQ	NR	NI	N/A	N/A
		FDA SSED44 and memo23	S	CCS	S: 0	S	L	Test CCS: 46% Control CCS: 5% P < .001, 2-sided FET Nonoperative care as control
	DIAM	FDA Executive Summary21	S	CCS	S: 0	S	H	Test CCS: 64% Control CCS: 15% BPP = 1.0 Nonoperative care as control High rate (>40%) of crossover in primary analysis dataset; No sensitivity analysis discussed; Not approved by FDA
Annular closure device	Barricaid	Thomé et al, 201819	S	CCS & recurrence	S: 0	S	L	Test mCCS: 76% Control mCCS: 66% 95% CI of difference: 2%-18%, P < .02 Test recurrence: 50% Control recurrence: 70% 95% CI of difference: −12% to −28%, P < .001
Sacroiliac joint fusion	iFuse	Whang et al, 201520	S	CCS	S: 0	S	M	Test CCS: 81% (72%-88%) Control CCS: 24% (13%-39%) BPP > 0.999; Analysis population and sensitivity analysis not reported; Nonoperative care as control
Interbody fusion spacer	Novomax	Lee et al, 201629	NI	Fusion rate	NI: 15%	NI	N/A	N/A
	BAK/C	Hacker et al, 200065 and FDA SSED66	NI	CCS	NR	NI	N/A	N/A
Dynamic stabilization	Dynesys	FDA Executive Summary67	NI	CCS	NI: 10%	NI	N/A	N/A
Spinal cord stimulator	Senza	Kapural et al, 201630	Primary: NI Secondary: S	≥50% pain reduction	NI: 10%	S	L	Test back pain response: 77% Control back pain response: 49% 95% CI of difference: 10%-42%, P < .001 Test leg pain response: 73% Control leg pain response: 49% 95% CI of difference: 6%-39%, P = .003
	Axium	PMA P15000431	Primary: NI Secondary: S	CCS	NI: 10%	S	L	Test CCS: 81% (95% CI: 70%-90%) Control CCS: 56% (95% CI: 43%-68%) P = .0004
Surgical robot	Renaissance	Kim et al, 201722	S	Screw accuracy	S: 9%	Not S	N/A	N/A

Abbreviations: NI, noninferiority; S, superiority; NR, not reported; N/A, not applicable; H, high; M, medium; L, low; PP, per-protocol; ITT, intention to treat; FDA, US Food and Drug Administration; BPP, Bayesian posterior probability; BCI, Bayesian credible interval; FET, Fisher’s exact test; CB, confidence bound; CCS, composite clinical success (reports often referred to this as “overall success”); mCCS, modified CCS; PF of ZCQ, physical function component of Zurich Claudication Questionnaire.

The overall risk of study bias was low in 22 of the 41 studies (54%), medium in 18 (44%), and high in 1 (2%) of the studies (Table 2; Supplementary Table S1). Medium risk ratings were attributable to potential attrition bias, an unclear blinding of outcome assessors, or potential limitations in randomization. The study with a high risk of bias suffered from a high rate of crossover subjects in the primary analysis dataset, a lack of sensitivity analyses, and uncertainty of concurrent interventions that could confound outcomes. The use of independent assessors, such as radiologists who were blinded to other outcomes, was considered an appropriate substitute for investigator blinding. Financial disclosure statements were only provided in 23 (56%) of the reports.

Evaluation of Superiority Claims

Among the 19 NI studies, 4 (21%) made post hoc superiority claims. Ten of the 15 (67%) NI + S studies and 5 of the 7 (71%) superiority trials satisfied their a priori superiority hypothesis (Table 3). All 19 superiority conclusions were based on statistical analyses with a superiority margin equal to zero. Although none of the studies discussed the superiority margin, the difference in proportions of treatment success (CCS) exceeded a +10% margin in 16 of the 19 studies (Table 3). However, the lower bound of the 95% confidence interval did not exceed +10% in most of the studies reporting that information.

The superiority claims in 4 of the 10 NI + S studies were found to be at a high risk of bias and 1 was at a medium risk of bias (Table 3; Supplementary Table S2). The NI + S study with a medium risk of bias for the superiority claim did not describe the analysis population and the FDA panel recommended only allowing NI claims.^15,32 In 1 NI + S study at a high risk of bias, the superiority claim was not robust to the sensitivity analyses of imputed values or the per-protocol analysis.^33,34 Two other NI+S studies at high risk of bias performed the superiority analysis on the as-treated population rather than the ITT population and did not describe any sensitivity analyses for the population or missing value imputations.^35-38 This is particularly important when up to 18% of patients did not receive the assigned treatment, which could compromise the efficacy of randomization.³⁷ The fourth NI + S study at high risk of bias only reported the safety analysis to be a predefined superiority analysis, which failed to meet statistical superiority; yet, overall success rates were claimed to be superior.³⁹

All 4 superiority claims from NI studies were rated to be at a high risk of bias due to the apparent post hoc specification of the superiority hypothesis and lack of multiplicity adjustment.^40-43 Furthermore, the analysis population was either not described^40,42 or the per-protocol population was used^41,43 in each of these studies. One NI study claimed superiority based solely on a post hoc modified CCS outcome since only NI could be claimed with the original primary endpoint.⁴⁰

Three of the 5 superiority trials were rated as a low risk of bias for the superiority claim,^19,23,25,44 1 was rated with a medium risk of bias due to the lack of reporting on the analysis population or sensitivity analyses,²⁰ and 1 was at a high risk of bias due to a high rate of crossover in the primary analysis dataset, no sensitivity analyses, and potentially confounding concurrent interventions.²¹ The study at high risk of bias did not lead to FDA approval of the investigational device. Among these 5 superiority trials, only 2 described sensitivity analyses (the conclusions were robust to the alternate analyses).^19,23 Although it was not considered in the risk of bias evaluation, 12 of the 19 (63%) studies claiming superiority reported the associated confidence intervals, which are useful for understanding the effect size. These 12 studies were comprised of 1 of the 4 NI studies, 7 of the 10 NI + S studies, and 4 of the 5 superiority studies.

Discussion

The majority of RCTs for spinal device trials are designed primarily as NI trials based on effect size, secondary benefits, or ethical considerations; however, sponsors frequently attempt to establish post hoc superiority claims. The present study demonstrates that post hoc superiority claims derived from NI trials often suffer from a high to medium risk of bias due to analyzing the per-protocol or as-treated populations without sensitivity analysis, the claims not being robust during sensitivity analysis, or the claim being based on post hoc modified endpoints. This is important, since sponsors are under pressure from physicians, payers, and health care systems to demonstrate improvements in safety, efficacy, and cost-effectiveness. By claiming superiority in some aspects of safety and effectiveness, the sponsor can argue an improved value proposition. The current study suggests that such post hoc claims may be valid in some instances but should be scrutinized closely by the intended audiences.

The strengths of the present study include the use of multiple databases, the inclusion of important governmental databases in addition to indices of journal articles, a rigorous query methodology, and the use of multiple reviewers to eliminate false positives and combine duplicates from the query results. However, there are several shortcomings to the results. No set of databases or queries can assure complete capture of relevant results. Also, many RCTs have multiple published reports at multiple follow-up timepoints. We focused on the timepoint for the trial’s primary endpoint; however, it is possible that additional superiority claims were made at later timepoints. Published protocols that provided adequate details of the a priori study plans were usually unavailable. A published protocol was only identified for 1 study.^19,45 Another limitation was that important details were sometimes not reported, which resulted in an “Unclear” rating for the bias assessments. Similarly, the disclosure of potential conflicts-of-interest was not consistent and could not be meaningfully collected and analyzed. While regulatory bodies and payers may receive additional, nonpublic details of the trials from the sponsor, other researchers must rely on publicly available data. Finally, only superiority claims related to primary outcomes at the primary endpoint were evaluated in this review; however, analyses of secondary outcomes specified a priori can be important for determining the utility of a new device, particularly for NI trials.

Only 17% of the reviewed RCTs of spinal devices since 2000 were designed as superiority trials. Major categories of NI trials included disc replacements (15 studies), biologics for fusion (7 studies), and interspinous/interlaminar spacers (7 studies). Most disc replacement studies compared with fusion, offering the secondary advantage of retaining range of motion. While some of these studies included radiographic measures of motion or fusion, they still used a NI design for the primary endpoint. Biologics studies had the secondary advantage of avoiding donor site morbidity compared with autologous iliac bone grafts, but this was not articulated as a superiority hypothesis and was only indirectly captured in patient reported outcomes described in the NI hypothesis. Interlaminar and interspinous process spacers are promoted as less invasive surgery, but only 1 trial compared an interlaminar device directly to fusion in order to justify the implication that reduced operating room time and blood loss resulted in a net benefit.⁴⁶ Other reports comparing interspinous process spacers to decompressive surgery alone referred to improving patient satisfaction, complication rates, and reducing subsequent surgical interventions for the potential advantages of the new devices.^47,48 Such comparisons would be most appropriate as a superiority trial with adverse events included in the CCS, as exemplified by Schmidt et al²⁵ for an interspinous process spacer versus decompression and analogously by Thomé et al¹⁹ for an annular closure device compared with discectomy alone. Updates to the Consolidated Standards of Reporting Trials (CONSORT) statement were proposed in 2006⁴⁹ and incorporated in 2012,⁵⁰ which suggest that studies should report the rationale for adopting a NI design and the associated NI margin. Most NI or NI + S studies published after these updates did not specifically discuss rationale for NI vs. superiority designs. Furthermore, only 4 reports provided any rationale for the NI margin, referring to requirements by the FDA.

Using well-rounded CCS measures as the primary endpoint may reduce the options for secondary benefits of the device beyond possible economic advantages. Among the reviewed RCTs on disc replacement, the primary endpoint CCS rates in the control group (fusion) ranged from 37% to 73%^{15,16,33,35,37,40-42,51} for cervical discs and from 41% to 55%^17,43,52,53 for lumbar discs, suggesting that a ceiling effect should not be a concern in those studies. Yet each disc replacement was evaluated with NI as the primary hypothesis and superiority as the secondary hypothesis. By focusing on appropriate endpoints, at-risk populations, and CCS criteria that demand well-rounded device success, the ceiling effect can be diminished and areas for improvement can be elucidated.

This review observed four superiority claims made through post hoc analyses of NI trials. These superiority claims were inherently at a high risk of bias due to post hoc hypothesis specification in a confirmatory trial.⁴ Furthermore, 50% of the superiority claims from NI+S studies were observed to be at a medium or high risk of bias due to inappropriate methodology for analysis or interpretation of the superiority hypothesis. This was usually attributable to analyzing the as-treated or per-protocol population without consideration of the ITT dataset. Relying solely on as-treated or per-protocol analyses could bias the conclusions, particularly if a significant number of patients did not receive the assigned treatment, there was missing follow-up data, or significant attrition.⁷ Overall, such deficiencies were apparent in 64% (9/14) of the NI or NI + S studies making superiority claims, which demonstrates the challenge of ensuring high fidelity conclusions when the superiority hypothesis is secondary to the NI design.

Based on the studies reviewed herein alongside the theoretical considerations of trial design and interpretation, superiority claims derived from NI trials may have a greater likelihood of confounding by methodological mistakes, ambiguities or sources of bias compared to claims derived from superiority trials. However, RCTs with an NI + S design can indeed be rigorous and present superiority claims with high levels of confidence. A few of the reviewed NI+S studies had a low risk of bias in the superiority conclusion because of the meticulous nature of the analysis and reporting, which included sensitivity analyses of both the population dataset and missing value imputations along with confidence intervals that demonstrated substantial margins.^17,18,30,31 The rationale for conducting these studies as NI + S trials rather than focusing on superiority was unclear. Regulatory or commercial considerations may provide a possible explanation.

Conclusions

Spine studies rarely employ superiority designs for confirmatory trials. NI studies can sometimes yield reliable superiority claims, but meticulous study conduct, analysis, reporting, and interpretation is paramount. Considering the singular goal of superiority trials and the standard methodology of such designs, greater confidence may be derived more readily from the resulting superiority claims. Investigators and sponsors are encouraged to consider superiority trial designs when evaluating novel technologies against a standard of care when feasible. Readers are encouraged to carefully evaluate the risk of bias behind each superiority claim by examining the methodology of the study and associated analyses.