Inadequate description of placebo and sham controls in a review of recent trials

Abstract

Background:

Poorly described placebo/sham controls inhibit appraisal of active intervention benefits and harms. The 12-item Template for Intervention Description and Replication (TIDieR) checklist was developed to improve the reporting of active intervention components. The extent to which TIDieR is used to guide description of placebo or sham control is not known.

Materials and methods:

We examined all placebo/sham-controlled randomised trials published in 2018 in the top six general medical journals. We reported how many of the TIDieR checklist items they used to describe the placebo/sham control(s). We supplemented this with a sample of 100 placebo/sham-controlled trials from any journal, and searched Google Scholar to identify placebo/sham-controlled trials citing TIDieR.

Results:

We identified 94 placebo/sham-controlled trials published in the top journals in 2018; none reported using TIDieR. On average 8 items were addressed, with placebo/sham control name (100%) and when and how much was administered (97.9%) most commonly reported. Some items (rationale, 8.5%, whether there were modifications, 25.5%) were less often reported. In our sample of less well cited journals, reporting was poorer (average of 6 items) and followed a similar pattern. Since TIDieR’s first publication, six placebo-controlled trials have cited it according to Google Scholar; two of these used the checklist to describe placebo controls.

Conclusions:

Placebo and sham controls are poorly described within randomised trials, and TIDieR is rarely used to guide these descriptions. We recommend developing guidelines to promote better descriptions of placebo/sham control components within clinical trials.

INTRODUCTION

Placebo or sham controls come in many modalities, ranging from lactose pills and saline injections to sham acupuncture (of various types) and sham surgery.^1–5 These different placebos can have different effects.⁶ Even relatively simple drug placebos come in different formats (tablets, or capsules); they have different sizes, doses,¹ colors,⁷ packaging,⁸ and sizes.⁹ They have different ingredients,^10,11 and sometimes they contain ingredients to mimic the side effects of the ‘active’ drug.¹² All of these differences can influence how effective they are.

A core function of placebo or sham control interventions is to provide a comparative benchmark against which the active interventions’ benefits and harms can be measured.¹³ Such comparisons rely on the assumption that the placebo/sham intervention used is appropriate.¹³ This assumption is sometimes unjustified.¹⁴ For example, in trials of oseltamivir, the placebo contained dehydrocholic acid and dibasic calcium phosphate dehydrate. This was presumably to mimic the bitter taste of the active intervention (oseltamivir powder) and thus maintain blinding.¹⁵ However, dehydrocholic acid can cause gastrointestinal symptoms, as can oseltamivir.¹⁶ Hence, while placebo controlled trials of oseltamivir often found an increased risk of gastrointestinal symptoms in the oseltamivir group compared with the placebo group, this was probably an underestimate of the true incidence of the harms.¹⁴

Some placebo or sham interventions can also lead to exaggerated active intervention effects. For example, a 2016 review (including 1973 trials) found that up to 64% of placebo control interventions were not matched in terms of physical properties.¹⁷ Lack of matching makes placebos identifiable, thus unblinding the trial. Unblinded patients who know they are receiving a ‘mere’ placebo may have lower expectations about recovery. These lower expectations, in turn, can affect the trial outcome, especially when symptoms are subjective and susceptible to suggestion. As evidence for the influence of this ‘expectation bias’, a 2004 systematic review showed that intervention effects were smaller when expectation bias was reduced.¹⁸

The extent to which the assumption that placebo/sham controls are appropriate (leading to mistaken estimates of benefit or harm) is unknown because placebo/sham components are rarely reported. A systematic review found that disclosing placebo/sham ingredients is rare: 8.2% for pills, and 26.7% for injections.¹¹ Inadequate placebo/sham control description stands in the way of trial replication, appraising the validity of the apparent active intervention benefits and harms, and for evaluating whether the placebo/sham was well matched (to assess whether blinding was likely to have been achieved).¹⁷

The Template for Intervention Description and Replication (TIDieR) checklist was developed to improve the problem of poor reporting of interventions, and it is mainly used to guide the description of active interventions.¹⁹ The extent to which TIDieR is appropriate or being used to guide the reporting of placebo and sham interventions is unknown.

AIMS

Our main aim was to determine the extent to which placebo/sham-controlled trials report placebo/sham interventions using TIDieR reporting items. A secondary aim was to check whether placebo/sham-controlled trials that use TIDieR to report active interventions also use TIDieR to report the placebo/sham intervention.

MATERIALS AND METHODS

Selection of articles

We followed the methodology used to develop TIDieR,²⁰ and examined all relevant trials published in six general medical journals with the highest impact factors (New England Journal of Medicine, JAMA, Lancet, Annals of Internal Medicine, PLOS Medicine, and BMJ) in a single year (2018) (see Appendix 1). We supplemented this main sample with an additional search (see Appendix 2), and selected a pseudo-random sample of 100 placebo/sham-controlled trials published in any journal (including those with lower impact factors) in. This was achieved by identifying the first 100 alphabetically ordered records of placebo/sham-controlled trials published in 2018. This allowed us to see whether there is a difference in placebo control reporting between the higher cited journals (many of which claim to support TIDieR), and other journals. For our secondary aim of checking whether placebo/sham controlled trials that used TIDieR to describe the active intervention, also used TIDieR to describe placebo/sham interventions, we searched Google Scholar to identify any placebo/sham controlled randomised trial that cited TIDieR (no date/time limit) (see Appendix 3).

Data extraction

For all samples, we read through the main study manuscript and sought additional supplementary material (including protocols and trial registrations) that were available and extracted whether the placebo/sham control(s) had been described according to each of the 12 TIDieR checklist items. We did so whether or not the trial reported using TIDieR. In some studies, the placebo or sham control was reported to be the same as / equivalent to the active intervention other than certain (characteristic) features. We interpreted those reports charitably and appealed to the relevant descriptions of the active interventions as surrogate descriptions of the placebo/sham controls. We also extracted data about the type of intervention under investigation (drug pill, drug injection, physiotherapy, psychology, surgery, or complementary and alternative). JH piloted the extraction sheet, and three researchers (RW, KB, CM) subsequently extracted the data. Discrepancies were resolved by discussion between authors (JH, RW, KB, CM).

Data analysis

Descriptive statistics consisting of frequencies and percentages were used to describe whether studies adhered to individual items (n=12) in the TIDieR checklist. Excel was used to analyse the data descriptively.

RESULTS

Search results

Figure 1 shows the search results and reasons for exclusion for each of our three samples. The search for our main sample yielded 123 records. 21 were excluded for not being published in the top six journals, two for being published in 2019 rather than 2018, and four for not being placebo-controlled trials. A further two were subsequently excluded as they were secondary analyses of trials already published.

Figure 1.

Flow diagram of study selection for each of the three samples

For our second sample (of placebo-controlled trials published in any journal in 2018), our search yielded 3563 records in PubMed. To reach 100 trials eligible for inclusion, we screened the title and abstract of the first 126 alphabetically ordered reports. 12 of the 126 were not placebo-controlled trials, and a further 14 did not meet our inclusion criteria.

Of the 1654 studies identified on Google Scholar that cite TIDieR, six were placebo/sham-controlled trials, two of which mentioned using TIDieR to report the placebo/sham control.

Intervention type

The majority of placebo controls (90.4%) in our main sample related to trials investigating pharmacological interventions , followed by complementary and alternative medicine (CAM, 6.4%) such as acupuncture, then surgery (2.1%) and physiotherapy (1.1%). In our second sample of trials published in any journal, a smaller percentage of trials were pharmacological (61%), with a greater number of CAM (23%) and other intervention types including physiotherapy (4%), laser treatment (3%), and stimulation (3%). The second sample also included trials using’other’ intervention types including ultrasound, tobacco , and devices (see Table 1).

Table 1.

Categories of interventions of included trials

Intervention category	Sample 1 (% reporting among sample of placebo-controlled trials published in top journals (n=94))	Sample 2 (% reporting among sample of placebo-controlled trials (n=100))	Sample 3 (% reporting among placebo-controlled trials that reported using TIDieR (n=6))
Pharmacological (e.g. placebo drug pill/injection)	90.4	61	-
Sham exercise	-	-	50
CAM (e.g. sham acupuncture, dietary/herbal supplements)	6.4	23	16.7
Sham physiotherapy	1.1	4	16.7
Sham laser treatment	-	3	16.7
Stimulation (e.g. sham rTMS)	-	3	-
Sham surgery	2.1	-	-
Other	-	6	-

Note: CAM= complementary and alternative medicine, rTMS = repetitive transcranial magnetic stimulation

Among the placebo/sham controlled trials that cited using TIDieR, none were pharmacological interventions; they were exercise (n=3), CAM (n=1), physiotherapy (n=1) and laser treatment (n=1).

Completeness of TIDieR checklist

Table 2, and Figure 2 show the percentage of trials that addressed this item in their primary reports and/or any supplementary materials for each of the checklist items. None of our two samples of placebo-controlled trials published in 2018 mentioned using TIDieR. None of the trials from any of the samples fully adhered to all 12 TIDieR checklist items for reporting placebo/sham controls.

Table 2.

Adherence to individual TIDieR items

	Sample 1 % reporting among sample of placebo-controlled trials published in top journals (n=94)	Sample 2 % reporting among sample of placebo-controlled trials (n=100)	Sample 3 % reporting among placebo-controlled trials that reported using TIDieR (n=6)
Mentioned using TIDieR	0	0	100
TIDieR item
Brief name	100	98	100
Why	8.5	7	16.7
What (materials)	68.1	60	33.3
What (procedures)	92.6	65	100
Who provided	45.7	59	83.3
How	68.1	82	83.3
Where	68.1	63	100
When and how much	97.9	89	100
Tailoring	59.6	15	50
Modifications	25.5	4	33.3
How well (planned adherence/fidelity assessment)	55.3	20	33.3
How well (actual adherence/fidelity)	54.3	12	33.3
Addressed all TIDieR items	0	0	0
Mean number of items addressed	7.5 (SD 1.8)	5.7 (SD 1.7)	7.7 (SD 2.5)

SD= standard deviation

Figure 2.

Percentage of placebo-controlled trials that reported each TIDieR item

In our first sample of studies published in the top 6 medical journals, the included studies reported over half of the items (average 8, standard deviation =1.8). All trials included the brief name of the placebo, and most included details about the materials (68.1%), procedure (92.6%), how and where it was administered (68.1%) and, when and how much was administered (97.9%). However, reporting was poorer for: providing a rationale for the control (8.5%), who provided the sham intervention (45.7%) or whether it was modified (25.5%).

For our second sample (placebo-controlled trials published in 2018 which were not restricted to the top journals) reporting was poorer (average number of items included = 6, standard deviation =1.7), but followed a similar pattern. Most trials included the brief name of the placebo (98%), how it was administered (82%), and when and how much was administered (89%). However, fewer than half the sample reported planned (20%) or actual (12%) fidelity, whether it was tailored (15%), a rationale for the control (7%), or whether it was modified (4%)

For the six trials that mentioned using TIDieR to describe the active intervention, an average of 8 (standard deviation =2.5) items were reported, and again a similar pattern was observed. The name of the placebo and administration procedures were reported, whereas items concerning why that choice of placebo tailoring, modifications and adherence were not. We have provided abstractions of these articles for each of TIDieR items in Appendix 4 to show how these items were reported.

DISCUSSION

Summary of findings

TIDieR is rarely used to describe placebo/sham interventions, and reporting of placebo and sham interventions is poor. TIDieR items regarding WHAT, WHO, HOW, WHERE, WHEN and HOW MUCH are the most adequately addressed which are likely to be explained by default when authors provide information about the intervention and control conditions as required by CONSORT.²¹ However, more specific information as requested in TIDieR such as details of the components of the placebo/sham control, rationale for the placebo, whether it was modified, and whether it was adhered to is rarely reported for placebo controls even in trials that report its use.

Comparison with other related studies

In a 2010 study, Golomb et al.¹¹ found that disclosure of the composition of placebos varied from 8.2–26.7% (depending on placebo type). Hoffman et al. (2013) found that fewer than half of ‘active’ interventions were adequately described in 39% of trial reports.²⁰ Based on our limited sample, it seems that reporting of placebo components has improved slightly in the last decade. Our study also revealed that adequate reporting of placebo or sham control interventions is poorer than adequate reporting of active interventions.^22,23

Strengths and limitations

This is the first study to investigate the extent to which TIDieR is being used to describe placebo or sham control interventions. A limitation is that our samples did not contain any trials of psychological or behavioural interventions. This may be due to our use of PubMed a primarily clinical database, but it does not apply to our Google Scholar search for trials that cite TIDieR. With psychological or behavioural interventions, the control arm often consists of minimal intervention, treatment as usual or no intervention, which are not necessarily placebo/sham controls. Nonetheless, these types of studies have been shown to be no better at reporting intervention methods than pharmacological trials.²⁴ Moreover, the majority of our sample are placebo-controlled trials of pharmacological interventions which is a more regulated field. Hence, by not including trials with psychological/behavioural interventions, we may have underestimated the problem with failure to disclose placebo/sham components. Future research on how to improve reporting of placebo/sham interventions should ensure that any reporting guidance for placebo/sham controls applies to behavioural interventions.

Another limitation concerns the fact that our second sample of studies used alphabetical ordering to generate a proxy randomised sample of studies. As such this sample of studies may not be representative of all trials published that year. This limitation does not apply to our main sample. In addition, 11 of the studies included in our second sample of trials published in any journal, were in fact published in the top 6 general medical journals, hence there is a degree of overlap, and these studies may have inflated the TIDieR reporting standards in this sample. Our sample of 100 trials from any journal provides a more accurate description of reporting across journal types.

Recommendations for future research

In order to perform their function of providing an adequate benchmark against which the benefits and harms of active interventions can be measured, placebo or sham components should be described rigorously. Researchers should investigate why current guidelines for reporting ‘active’ interventions (TIDieR) are rarely used, even among journals such as the BMJ who allegedly require its use²⁵.

In addition, TIDieR may require adaptation for placebo or sham controls. Some items might not apply, others may require additional emphasis, and some additional items could be required. For example, it may be less important to include a rationale for the placebo, as it can be assumed to be to control for certain (‘characteristic’) features of the active intervention.¹³ To achieve this function, successful blinding may be required.^26,27 Relatedly, appraising the estimate of intervention harms requires that placebo controls designed to be ‘active’ (containing ingredients that mimic side-effects or taste of the active intervention)¹² should be reported as such. It may also be necessary to explicitly relate the reporting of the placebo/sham to the reporting of the active intervention, especially as the current prompts for the TIDieR items only refer to the ‘intervention’ which authors may interpret as only the ‘active’ intervention under investigation rather than the placebo/sham control as well. Finally, any adaptation or addition to current intervention reporting for placebo or sham controls should minimize additional burden to researchers,²⁸ in order to avoid barriers to implementation.

Conclusion

The extent to which placebo or sham interventions are reported within clinical trials—including trials reported in journals that require use of guidelines to describe active interventions—is poor. This inhibits assessing the benefits and harms of active interventions and trial replication. Designing and promoting reporting standards for placebo and sham controls is required.

Supplementary Material

supp info

Appendix 1. PubMed Search Strategy for Placebo Controlled Trials in Most Widely Cited Journals (search on 10th May 2019)

Appendix 2. PubMed Search Strategy (search on 6th February 2019)

Appendix 3. Google Scholar Search (search on 21st February 2019)

Appendix 4. Extractions for the 6 studies using TIDieR