Substituting a randomised placebo control group with a historical placebo control in an endometriosis pain trial: a case study re-evaluating trial data using historical control data from another trial

Abstract

Objective

The substitution of an in-study control population with a historical control (HC) population is considered a viable option for reducing the necessary recruitment of control patients. However, it is necessary to evaluate whether this method is applicable to studies on indications targeting endometriosis-associated pelvic pain (EAPP). This study aims to evaluate the potential bias in the results of an EAPP study with an HC arm.

Methods

For this case study, we re-evaluated data from a randomised, placebo-controlled trial using dienogest daily to treat EAPP with an HC arm based on data from a second randomised, placebo-controlled trial in the same indication. Propensity Score (PS) matching was used to match between the treatment and HC arm on all baseline variables. To evaluate the effect of matching on the introduced bias, we evaluated efficacy parameters with the full treatment and control group, as well as the matched group.

Results

The difference between means (placebo minus treatment) in change in pain, as measured on the Visual Analogue Scale from baseline to end of treatment, deviates in the comparison treatment/pool of HC (7.15 (0.22 to 14.08)) from the overall in-study group (reference: 11.89 (6.06 to 17.73)). After PS matching on the baseline variables, the difference between means (11.79 (4.09 to 19.5)) is close to the reference.

Conclusions

Using HC with PS matching has proven to be useful in the setting of treating EAPP, while emphasis must be given to the selection mechanism and the underlying assumptions. This case study has shown that even for studies which are very similar in design, heterogeneity and between-study variations are present. With the use of an HC arm, it was possible to reproduce similar results than in the original study, while the PS matching improved the comparability considerably. For the main endpoint, PS matching could reproduce the original study results.

Trial registration number

NCT00225199, NCT00185341

Strengths and limitations of this study.

• Our study demonstrates the potential applicability of using historical control data in pain therapy for endometriosis, which could benefit future patients.

• The use of secondary data analysis allowed for an evaluation of the method under ideal circumstances where the historical data was highly comparable to the primary study in terms of inclusion and exclusion criteria.

• One limitation of the study is the limited availability of data for the pool of historical controls, as only one study was available. Further analyses of the effects of selection mechanisms would benefit from a wider selection of studies.

• While the use of Propensity Score matching improved comparability between the treatment and control groups, it is still subject to inherent biases and assumptions in the data. Therefore, caution is warranted when interpreting the results.

Introduction

Using real-world data (RWD)/real-world evidence as a data source in trials for determining efficacy has seen more and more acceptance and application in recent years.^1–4 Past examples for these applications reach from determining historical response rates in a single arm trial,⁵ as well as changes of dose-label to an outpatient setting, effectively reducing the burden for patients,⁶ confirmation of clinical trial results in breast cancer in real-world setting where less strict eligibility criteria provide a more realistic population,⁷ or first-line approval of treatment of a rare oncological disease, where controls are selected from electronic health records.⁸ All have in common that by using available (or collected) external data, it avoided randomisation at least to a certain degree. Using randomisation in clinical trials is a highly relevant method in medical research. It is considered the reference standard of trial designs; the rigour and reliability of other designs and methodologies are often measured against existing or hypothetical randomised controlled trials (RCTs).⁹ Although undeniably an important method with unique features, there are some drawbacks to randomisation which the use of RWD aims to minimise.

The first advantage that is usually reported is the financial one, of limiting patient recruitment and instead using RWD, as recruitment for RCTs are known to be a cost factor. As has been laid out by Burger et al,¹⁰ the cost-effectiveness of reducing recruitment from, for example, 400 to 200 patients is usually overestimated and comes usually at a cost. This cost can be time, as planning and acceptance of innovative trials can be longer, but also monetary, as access to RWD can be associated with costs. Still, using RWD and historical controls as a subcategory of that, for amending or substituting treatment (or control) arms can provide added value. One of them being that larger sample sizes allow for a larger heterogeneity and can be more acceptable to regulators and health technology assessment agencies.¹⁰

Randomisation poses also two organisational drawbacks. Because of strict inclusion and exclusion criteria, as well as usually aiming for the smallest necessary sample size, the study population is very controlled. This can limit extrapolation and generalisability of the study results. In the case of rare diseases, it is challenging to recruit sufficient patients for the treatment and control arms. With the use of historical control data, the recruitment can be focused on the treatment arm, effectively shortening the time to get valuable medication to patients in need.

The last, but most relevant considerations of randomisation, are ethical ones. Patients take part in clinical studies in hopes of better treatment. Randomisation poses the possibility that they are allocated to no treatment. This uncertainty poses a burden onto the patients, which is often underestimated,¹⁰ and can be avoided if recruitment is only necessary for the treatment arm. In pain trails, for example, in the research for new treatments of endometriosis associated pelvic pain (EAPP), the control group plays a very special role, as the placebo effect can lead to a considerable reduction in pain perception.^{11 12} The patients receiving placebo endure the trial without an effective pain treatment (above rescue medication). Using a historical control population reduces this burden and could lead to better recruitment. Entering a trial without the imminent possibility of being assigned to placebo can be more attractive for patients.

Research related to endometriosis is of considerable importance and societal impact, but estimation of the prevalence still proves to be difficult, with a variety of methods used and numbers reported in the literature. The prevalence of endometriosis ranges between 1–5% in the overall population of women,¹³ 6–10% in women of reproductive age¹⁴ and over 25–100% in women with chronic pelvic pain.¹⁵ Reported symptoms are severe pelvic pain and infertility¹⁴ and significantly reduce the health-related quality of life.¹⁶ Although the cause of endometriosis is still unknown, pain is usually considered the most relevant symptom and the primary reason for treatment. Because of the mentioned powerful placebo effect, the control population is very important for studies of EAPP and is usually part of the trials.

For an effective application of RWD and historical controls in a trial setting, two pieces are important to understand: (a) the underlying assumptions which are accepted for the use of a certain method, and (b) the performance of the selected method under specific conditions. The existing body of literature includes studies using simulated data (targeting the former and adding to the generalised understanding of RWD) or studies in certain therapeutic areas (targeting the latter and improving understanding of important aspects of study design inherent to these areas). In obstetrics and gynaecology, especially when treating EAPP, these methods have not seen wide application, although the benefit for patients is clear. This study aims to fill this gap and is part of the second family of studies, which evaluates the performance of using a historical control arm under specific conditions.

Methods

To conduct the case study, we used study data from two randomised, double-blind, placebo-controlled, multicentre studies of women aged 18–45 years with symptomatic endometriosis. The first study (study by Strowitzki et al¹⁷; NCT00225199) examined the use of 2 mg dienogest daily to treat EAPP, while the second study (study by Trummer et al¹⁸; NCT00185341) used a C-C chemokine receptor type 1 (CCR1) antagonist to treat EAPP. The primary efficacy endpoints used in both studies were the absolute change in EAPP measured on a Visual Analogue Scale (VAS) from baseline to study end, and consumption of rescue analgesics via diary (ibuprofen).

The efficacy was re-evaluated and compared with the original results. A static approach was used, where the full control arm is substituted with historical control data.^{19 20} For the Propensity Score (PS) matching, a 1:1, greedy nearest neighbour matching with a calliper distance of 0.5 was used. Table 1 lists the variables used in the PS model.

Table 1.

Identified relevant baseline parameters for Propensity Score calculation

Variable	Unit	Data type
Age	(Years)	Numerical
Race		Categorical
Country		Categorical
Body mass index	(kg/cm²)	Numerical
Blood pressure systolic	(mm Hg)	Numerical
Blood pressure diastolic	(mm Hg)	Numerical
Pulse (bpm)		Numerical
Alcohol consumption	(Y/N)	Binary
Smoking status	(Y/N)	Binary
r-ASRM stage		Categorical
Endometriosis histology	(Y/N)	Binary
Tubal ligation performed	(Y/N)	Binary
Number of births		Count
Number of abortions		Count
Cycle, regularity		Categorical
Cycle average length	(Days)	Numerical
Average intensity of bleeding		Categorical
Menstruation average length	(Days)	Numerical
Menarche (years)		Numerical
Vaginal bleeding, intracyclic	(Y/N)	Binary

r-ASRM, revised American Society of Reproductive Medicine.

The endpoints considered in the case study are the absolute change in EAPP on VAS from baseline to end of treatment, the absolute change in consumption of analgesics (ibuprofen) 28 days prior to VAS measurement via diary from baseline to end of treatment, absolute changes in Biberoglu and Behrman score²¹ between baseline and end of treatment, and global assessment of efficacy by patients and investigators using the Clinical Global Impressions (CGI) scale.²²

For reference, the treatment difference from the original publication (∆_in-study) was used, calculated from the overall treatment group (x̅ _{Treatment, overall}) and the overall in-study control group (x̅ _{control; in-study, overall}). The in-study controls are then substituted for the overall pool of historical controls (x̅ _{control; historical, overall}) resulting in ∆_{historical, overall}. This treatment difference is a crude comparison without any adjustment of the study populations and can be used as a reference to evaluate the effect of the selection mechanism (PS). Lastly the treatment difference is calculated between the matched set of treatment (x̅ _{Treatment, matched}) and the historical control arm (x̅ _{control; historical, matched}) resulting in ∆_{historical, matched}.

• x̅_{Treatment, overall} – x̅ _{control; in-study, overall} = ∆_in-study

• x̅_{Treatment, overall} – x̅ _{control; historical, overall} = ∆_{historical, overall}

• x̅_{Treatment, matched} – x̅ _{control; historical, matched} = ∆_{historical, matched}

The different treatment differences are compared with each other to evaluate which method can reproduce the reference results, how much bias is introduced by switching to non-randomised controls and if PS matching can reduce this bias.

Patient and public involvement

No patient involved.

Results

The baseline characteristics for in-study treatment data¹⁷ as well as the pool of historical controls¹⁸ showed only minor differences. The PS matching between the in-study treatment group and the pool of historical controls resulted in a successful 1:1 greedy nearest neighbour matching with calliper distance of 0.5. Out of N=101 complete cases in-study treatment and N=109 complete cases from historical controls, N=73 pairs were matched. For the matched pairs, the standardised mean difference for all covariates was reduced to a negligible difference of ≤0.25, as recommended by Rubin²³ and Stuart²⁴.

The reference publication reported a mean treatment difference of 11.89 between the treatment and control arms for the main endpoint, pain measured on VAS. When a historical control arm was substituted for the control arm with no adjustments (crude comparison), the treatment difference was 7.15. After adjusting by PS matching, the groups resulted in a treatment difference of 11.79, which was close to the reference from the publication (11.89). For a detailed summary of results, please refer to table 2.

Table 2.

Summary of difference (placebo minus treatment) between mean difference from baseline to end-of-treatment (95% CI)

	DNG (N=102)	Placebo (in-study) (N=96)	Difference between means (placebo minus DNG) (95% CI)	Placebo (historical) (N=110)	Difference between means (placebo minus DNG) (95% CI)	DNG (matched) (N=73)	Placebo (historical, matched) (N=73)	Difference between means (placebo minus DNG) (95% CI)
	Mean (±SD)	Mean (±SD)		Mean (±SD)		Mean (±SD)	Mean (±SD)
VAS (mm)	−28.8 (24.54)	−16.9 (16.0)	11.89 (6.06 to 17.73)	−21.67 (24.94)	7.15 (0.22 to 14.08)	−31.19 (23.36)	−19.39 (22.79)	11.79 (4.09 to 19.5)
Rescue medication taken	−4.68 (6.47)	−3.99 (8.69)	0.69 (−1.64 to 3.02)	−0.77 (12.35)	3.91 (1.02 to 6.81)	−4.85 (6.51)	−1.19 (13.84)	3.66 (−0.02 to 7.35)
B&B Pelvic Pain Severity Profile Score	2.16 (1.94)	1.62 (1.66)	−0.54 (−1.05 to −0.03)	1.48 (1.77)	−0.68 (−1.21 to −0.15)	2.25 (1.83)	1.32 (1.80)	−0.93 (−1.55 to −0.32)
B&B physical signs severity profile	0.84 (1.04)	0.57 (1.21)	−0.27 (−0.58 to 0.04)	0.77 (1.18)	−0.06 (−0.37 to 0.25)	0.87 (0.94)	0.78 (1.17)	−0.1 (−0.44 to 0.25)

B&B, Biberoglu and Behrman score; DNG, dienogest; VAS, Visual Analogue Scale.

Regarding rescue medication taken 28 days pre-VAS measurements, the response to treatment in the reference publication was very similar between treated and placebo patients, showing only a very small treatment effect (difference between means of 0.69). The response to treatment in the pool of historical controls was much lower than the in-study controls, resulting in a difference between means of 3.91 when comparing historical controls to the in-study treatment (without adjustments). After matching the groups, the difference between means was 3.66, which was very similar to the previously mentioned crude comparison of treated and historical control patients.

The study power to show a treatment difference of the main endpoint (pain measured on VAS) of 11.79, with an average SD σ=23.08 and a significance level alpha=0.05, resulting in a power of 86.63%.

Discussion

Here we evaluated the substitution of a recruited control arm with a historical control arm in an RCT researching treatments for EAPP. PS matching between the in-study treatment and historical control group was used to reduce the bias introduced by a non-randomised control population. Using historical control arms based on data from previously conducted RCTs is a viable option in trials treating EAPP, when considering the respective type of data source and method for creating the control arm. The probability of introducing bias is still prevalent, even when the data sources are of the highest comparability. Therefore, controlling for unmeasured confounding, such as with PS matching, is crucial for unbiased effect estimates.

Using historical control data is a viable method in studies of pain treatment for endometriosis. We achieved a satisfactory precision (11.79) when reproducing the reference result (11.89) for the main endpoint pain measured on VAS. However, the results also show that even in near perfect conditions, it can introduce bias when the control population is changed. Although the inclusion/exclusion criteria, study population, time frame and larger geographical location of study centre were very similar or near identical between both data sources, the results were biassed. In particular, the change in pain measured on VAS (mm) differed after substituting the control population when no adjustments were made (from 11.89 to 7.15). This reinforces the importance of an appropriate selection mechanism for constructing the study population.

The selection method used in our example to construct the control population was PS matching on all baseline variables (demographic and endometriosis related). This is an effective and applicable method. Sufficient patients were matched from the treatment group with appropriate historical controls (N=73 matched pairs), while retaining the power to show the expected treatment effect (power=86.63%) and limiting the bias introduced from switching the control population (eg, VAS (mm) from reference 11.89 mm to PS matched 11.79 mm). The previous results were reproduced with satisfactory precision.

For two endpoints (ie, rescue medication taken, CGI efficacy index by patient) the precise reproduction of the reported effect reported by Strowitzki et al¹⁷ was not possible. In case of rescue medication taken, this could be attributed to some in-study variability of the reference study population, especially the control population. The response to treatment in the historical control group was more in line with what was expected, according to the literature on treatment of EAPP with dienogest.

In this study, the results reported by Strowitzki et al¹⁷ were considered the true underlying treatment effect, which we aim to replicate with our hypothetical trial. In real-world-applications however, the aim is to approximate the true effect of a drug in a target population, where the true effect is usually unknown. Ideally, the study population is a random sample from the target population, and the treatment and placebo response can differ from sample to sample. When a study population is sampled at different times and locations, the target population changes slightly, raising questions about whether the underlying true treatment effect remains the same. Combining data from different study populations (as done when using historical controls) can directly affect the estimated treatment effect. For example, in the case of the secondary endpoint rescue medication taken 28 days pre-VAS, the PS matched groups better represented the expected reduction in the use of rescue medication in the treatment group compared with the control group than the reference results. This could be due to random error or a different underlying treatment effect in both populations.

Strengths and weaknesses of the study

Compared with other studies on the subject of historical controls in clinical trials, this study was able to test the practical application of this method in a research field where it could be of great value. A strength of the study is the access to data sources that are highly comparable, allowing the effects of the method to be more easily singled out while minimising between-study variability.

Another strength is that the treatment under study (2 mg dienogest daily) has been proven to be effective in treating EAPP in previous studies. In outcomes where the reference study showed an unusually low response to the treatment, this could be put in context with other studies on the use of dienogest, and it could be attributed to expected in-study variability.

One weakness of this case study was that only one study was available for the pool of historical controls. To better test the effect of the selection mechanism, a pool of more studies with a wider selection of study time points, geographical origins and inclusion/exclusion criteria would have been valuable.

The meaning of the study: potential explanations and implications for clinicians and policymakers

Using historical controls for substituting in-study controls has proven to be a viable method, particularly when used with data of high comparability and PS matching as the selection method. Efforts should be made in future trials on EAPP to consider this option, thereby reducing the burden on patients in the control population.

Unanswered questions and future research

Further research should explore the impact of introducing more historical study populations into the pool of historical controls, particularly regarding the variability and bias introduced by such action, as well as selection mechanisms to reduce them. Dynamic methods that add historical control populations to smaller recruited control populations based on the difference between the two could be of interest and should be tested in studies on EAPP. The robustness of the method could be tested by adding historical controls to populations of less comparable RCTs, or even RWD, to estimate the introduced bias and measures to counteract.

Data availability statement

Data are available upon reasonable request. Bayer’s individual patient data sharing policy is found at https://vivli.org/ourmember/bayer/.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The original study protocols were approved by local independent ethics committees and all participants provided written informed consent to their respective trial.