Critical review of sham surgery clinical trials: Confounding factors analysis

Abstract

Objective

Sham surgery (placebo surgery) is an intervention that omits the step thought to be therapeutically necessary. In surgical clinical trials, sham surgery serves an analogous purpose to placebo drugs, neutralizing biases such as the placebo effect. A critical review was performed to study the statistical relevance of the clinical trials about sham surgery in the light of potential confounding factors.

Materials and methods

For the critical review 52 articles were included. The possible confounding factors have been studied using a structured interpretative research form designed by the authors. This form includes the following ten confounding factors: I), lack of homogeneity among inclusion/exclusion criteria. II), false double blind. III), lack of post-surgery double blind. IV), power of the study. V), sample characteristics. VI), lost patients to Follow-up. VII), gender distribution. VIII), age equilibrium. IX), lack of psychological patient evaluation. X), lack of psychiatric patient evaluation. In most of the studies, at least one confounding factor was present.

Results

The analysis of the confounding factors showed that they could influence the reliability of the surgical placebo effects.

Conclusions

The validity of sham surgery should be reconsidered.

Highlights

• •In sham surgery literature there's no assessment on confounding factors effect.
• •Even if sham surgery has been used as control for over 30 years it isn't a standard.
• •The validity of sham surgery is not completely supported by available literature.

1. Introduction

Sham surgery (placebo surgery) is a surgical intervention that omits the step thought to be therapeutically necessary. In clinical trials of surgical interventions, sham surgery serves an analogous purpose to placebo drugs, neutralizing biases such as the placebo effect [1].

New surgical procedures have been developed in the last years and sometimes the acceptance of new procedures is based on their perceived value relative to previously accepted treatments. This process can be biased because influenced by the enthusiasm, skill, and prominence of the surgeon reporting the results and by their selection of patients for treatment [2].

A double-blinded randomized placebo-controlled trial is recognized as the gold standard of clinical research [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19]. “Trial” is from Anglo-French “trier”, meaning “to try”, referring to the action or process of putting something to test or proof at the bedside of the patient. Broadly, clinical trial refers to any testing done on human beings for determining the value of a treatment or for preventing disease. The clinical trial, in simplest form, involves the application of the experimental variable (treatment to a person or group of persons) and follow-up observation of the treatment to measure its effect. That outcome measure may be death, occurrence or recurrence of some morbid condition, or a difference indicative of change. Trials are said to be controlled if the effect of the treatment is measured against a comparison treatment administered over the same time period and under similar conditions.

The comparison treatment may be another test treatment or a control treatment consisting of an accepted standard form of therapy, a placebo or sham treatment, or observation only (no treatment). Clinical trials evaluates the efficacy of an intervention towards a comparator of known efficacy or towards a placebo. The ideal of a clinical trial is that the researcher compares groups of patients who differ only with respect their treatment. If the groups differ by other characteristics then the comparison of treatments can be biased. If these can be identified, their effects on the cause-effect relation can be avoided, but unknown or unexpected biases cannot be dealt with. The method to design a clinical trial must be to avoid or to eliminate biases.

Some authors reported that placebo effect can be largely influenced by inadequate research methods producing artefacts [20].

Recently, a case-based ethical analysis on placebo in surgical research based on the recommendations about clinical research defined in the Helsinki Declaration was performed. Authors concluded focusing attention only on patient that should always come before any other implication [21].

Confounding is referred to as a ”mix of effects” [22]. This determines that the causal association mingles the effects of one or more factors that can change the intensity or even reverse this association in an unpredictable way the real association between exposure and outcome, in presence of confounding factors, can be mistakenly shown or failed to. In a clinical trial a known prognostic factor, that is not evenly distributed between the groups in the study, might underlie the presence of a confounding factor [23]. In a randomized controlled trial the treatment arms are compared to a control group that is treated with placebo (or no intervention) in most medical specialty studies and pharmacology studies. However, in surgical trials, on which we focused our interest, the use of a placebo, or a “sham” surgery, is controversial. In the European Union legislation (u) there is no specific or even general reference to sham surgery. No directives on this issue was never addressed to the Member States nor with the usual form of the ordinary legislative procedure (European Council and European Parliament together), nor with the least common form of the special legislative procedures (sole legislator the European Council). Also, there is no reference, even as recommendations, about acts of non-binding guidelines to member states and issued by the organs of the Union without a binding regulatory powers and/or when the use of the latter is not necessary. Because of the lack in literature of studies and analysis of those trials from the confounding point of view, we did this critical review using specific criteria of selection for the articles to analyze. In synthesis, the aim of this paper is to analyze all the trials regarding sham surgery in the light of confounder factors. This analysis will be useful to highlight the real statistical power of the best evidence available.

2. Materials and methods

2.1. Study design

A critical review was performed according to the Critical Review writing guidelines elaborated at the University of Technology of Sydney (http://www.uts.edu.au/current-students/support/helps/self-help-resources/academic-writing/critical-thinking-skills) adapted from the following sources: Royce, T 2009, The meaning of critical review, ELSSA Centre, UTS; Royce, T 2009, Skills to cultivate for research and critical review, ELSSA Centre, UTS. Royce, T 2009, Reading and writing critically, ELSSA Centre, UTS.

2.2. Inclusion and exclusion criteria

A total of 87 articles about sham surgery were found in PubMed database up to May 31, 2015. Studies were eligible if they were randomized clinical trials in which the efficacy of surgery was compared with placebo. Surgery is defined as any interventional procedure that changes the anatomy and requires a skin incision or the use of endoscopic techniques; dental studies were excluded. The term placebo refers to placebo surgery, or sham surgery, an imitation procedure intended to mimic the active intervention; including the scenario in which a scope was inserted and no procedure was performed but patients underwent sedation or general anesthesia and could not distinguish whether or not they had undergone the actual procedure. Exclusion criteria were: studies investigating anesthesia or other drugs; studies about major surgery procedures were excluded because the post-operative evidence of the treatment group is verifiable by the patient and health care staff (third operator) that follow the patient after surgery.

Fifty-three clinical trials resulted eligible. One study of 53 was excluded by the analysis because lack of informed consent and inside the medical record the type of intervention was not truly.

Finally, 52 randomized clinical trial were eligible for the critical review [10], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73].

2.3. Confounding factors

The possible confounding factors have been studied using a structured interpretative research form designed by the authors including the following ten confounding factors: I), lack of homogeneity among inclusion/exclusion criteria. II), false double blind. III), lack of post-surgery double blind. IV), power of the study. V), sample characteristics. VI), lost patients to follow-up. VII), gender distribution. VIII), age equilibrium. IX), lack of psychological patient evaluation. X), lack of psychiatric patient evaluation. The reasons for choosing these items were: Methodological (items 1–3); Statistical (item 4–8) and Clinical (item 9–10). To evaluate the presence of these confounding factors, the interpretive research form was applied to analyze the trials in order to record the ten items above reported and the type of pathology, the type of surgery (simulated or performed).

No evaluation regarding administrative treatment was performed because this information lack in the trials. Therefore, the economic influence and the difference between naive patients and not naïve patients on the confounding factors were not evaluated.

Two independent reviewers performed data extraction and analysis.

2.4. Statistical analysis

Data have been analyzed using MedCalc 13.2.2.0 (MedCalc Software, Mariakerke, Belgium). The normal distribution of each variable was tested by Kolmogorov-Smirnov test. Variables were compared using Mann-Whitney's test for independent samples. A p-value <0.05 was considered statistically significant.

3. Results

The characteristics of the 52 randomized clinical trials analyzed in the critical review are summarized in Table 1. The main results from the critical review of the 52 eligible studies are reported in Table 2. In 27 studies on 52 (52%) studies, a sort of “false randomization” has been used, resulting in not homogeneity in gender. In 7 on 52 studies (13%) authors did not report the age of the patient enrolled. One on 52 studies (2%) reported homogeneity between inclusion and exclusion criteria. Sixteen on 52 studies (31%) were single blind trials, 22 on 52 (42%) were post intervention studies and 14/52 (39%) classified as double blind trial. In 31 on 52 studies (60%) the power of the study was not mentioned, while in the remaining 21 only in one study the power was under the usual threshold of 80% (Table 2).

Table 1.

Characteristics of trial series.

Tabella	n° of Studies
Inclusion and exclusion criteria homogeneity	51/52 (98%)
Randomized	51/52 (98%)
Single blind	16/52 (31%)
Double blind	36/52 (69%)
Post-operative double blind	22/36 (61%)
Study power absence or <80%	32/52 (61%)
Sample size group presence	52/52 (100%)
Gender homogeneity	27/52 (51%)
Age homogeneity	45/52 (86%)
Psychological evaluation presence	16/52 (31%)
Psychiatric evaluation presence	10/52 (20%)

Table 2.

Main results from the critical review of the 52 eligible studies.

Critical review results	N° (%)	References
Not homogeneity in gender	27/52 (52)	[10], [24], [25], [27], [30], [32], [36], [38], [39], [41], [42], [47], [49], [54], [55], [57], [58], [59], [60], [61], [62], [63], [65], [66], [69], [72], [74].
Patients age not reported	7/52 (13)	[40], [44], [58], [59], [69], [72], [73].
Inclusion and exclusion criteria homogeneity	1/52 (2)	[65]
Single-blind trials	16/52 (31)	[26], [27], [31], [40], [41], [42], [44], [45], [51], [55], [56], [57], [62], [66], [67], [69].
Post-intervention trials	22/52 (42)	[25], [26], [28], [32], [33], [34], [35], [39], [43], [48], [49], [52], [58], [60], [63], [64], [65], [70], [71], [72], [73], [74].
Double-blind trials	14/52 (27)	[10], [29], [30], [36], [37], [38], [46], [47], [50], [53], [54], [59], [61], [68].
Power of the study not-mentioned	31/52 (60)	[10], [26], [28], [30], [31], [33], [34], [35], [38], [44], [45], [47], [48], [49], [50], [52], [54], [57], [58], [59], [63], [64], [65], [66], [68], [69], [70], [71], [72], [73], [74].
Power of the study under the threshold	1/52 (2)	[53]
Mean age value used for patients description	47/52 (90.4)	[10], [24], [25], [26], [27], [28], [29], [30], [31], [32], [34], [35], [36], [37], [38], [39], [41], [43], [45], [46], [48], [49], [50], [51], [52], [53], [54], [55], [56], [60], [61], [62], [63], [64], [65], [66], [67], [68], [70], [74].
Median age value used for patients description	5/52 (9.6)	[33], [42], [47], [57], [71]
Absence of psychological profile	36/52 (69)	[10], [24], [25], [27], [29], [30], [31], [32], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [50], [53], [54], [55], [56], [57], [59], [60], [61], [62], [65], [66], [67], [68], [69], [74].
Absence of psychiatric profile	43/52 (82)	[25], [26], [27], [29], [30], [31], [32], [33], [34], [35], [36], [37], [40], [41], [42], [43], [44], [45], [46], [47], [48], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [62], [63], [64], [65], [66], [67], [68], [69], [71], [72], [73].

The box plot analysis reported in Fig. 1, showed no difference between the median sample size comparing sham and not-sham surgery studies. The outliers, reported in Fig. 1, fall more frequently in the not-sham group and could represent a potential bias if a reliable meta-analysis of “confounding” has to be realized.

Fig. 1.

The distribution of sample sizes of trial series by study groups. The variable analysis was performed by Mann-Whitney's test for independent samples.

Regarding the median and mean values of the patient age, a great homogeneity has been observed between “sham” and “not sham” groups. As evidenced in Fig. 2, the age mean value has been used for population description in the majority of the studies; in 47/52 (90.4%) eligible studies the mean age value was used in both arms, sham and not-sham surgery groups, whereas the median age was used only in 5/52 (9.6%) studies (Table 2). In 35 on 47 studies (77%) on mean age (Fig. 2), no difference between sham and not sham arms was evident. Last, the absence of psychological and psychiatric profile has been noted in 36 on 52 (69%) and in 43 on 52 (82%) of the studies, respectively (Table 2).

Fig. 2.

Studies reporting mean age in each arm; Studies reporting median age in each arm.

Overall, 270 on 4697 (5%) participants were lost to the follow-up.

4. Conclusion

Sham surgery (placebo surgery) is a surgical intervention omitting the intended therapeutic procedure. The enrollment of participants in placebo controlled surgical trials to the arm of a sham surgical procedure has necessarily produced a lively debate about the ethical acceptability [8], [18], [19], [74], [75].

The role of the surgical placebo is fervently debated [76].

Some Authors raised doubts about the reliability of the placebo effects suggesting the possibility that artefacts from inadequate research methods could alter the results of the study [77].

Recently, Sihvonen et al. attempted to address these criticisms discussing some methodological issues essential for successful controlled surgical trial, minimizing bias and maximizing the validity of the study [20].

Confounding factors can change the intensity or even reverse the causal association between exposure and outcome [23]. The lack in literature of studies and analysis on sham surgical trials from the confounding point of view, this critical review should highlight the real statistical power of the best evidence available on this debated issue. The results of the critical review could suggest the following conclusions: 1) these studies involving sham surgery, which were published in medical literature, are exposed to a number of confounders, which should be considered in further analysis. 2) Given the number of confounding factors revealed by this short analysis (i.e. true double blind, study power, sex and age randomization, etc.) the validity of the use of sham surgery must be reconsidered. 3) Sex and age are considered the most frequently causes of confounding in the strength of the association measure in cause-effect model. The lack of these two important classic confounding factors is indicative of the low scientific reliability of data presented by these trials. 4) Because the different pathologies in trials did not reach a number sufficient for stratification, data in the whole was the unique type of analysis that could be performed. 5) The presence of inter-study variability between sham and not sham studies was not ascribable to mean or median age of patients but possibly to other factors. 6) The absence of psychological and psychiatric profile that is relevant for studies were the placebo effect have to be evaluated represents a very important confounding factor [10], [22], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [78], [79]. 7) A percentage of 5% of participants lost to the follow-up has been extracted from 30 on the 52 studies examined (ref.). This represents a limit and a further confounding factors influencing sham or not sham surgery effectiveness.

In conclusion, confounding factors could influence the reliability of the surgical placebo effects and have to be considered as potential bias in controlled surgical trial. Given this influence, the validity of sham surgery should be reconsidered.

Ethical approval

Ethical Approval and Informed Consent, due to the type of investigation, have not been necessary.

Funding

The research has been carried out with no funding by sponsors.

Author contribution

All authors have contributed as a group to all aspects of publications.

Conflicts of interest

All authors decline any financial and personal conflicts of interest.

Guarantor

Massimiliano A Vitali.