A review of the statistical analysis of randomised controlled trials conducted within OCTRU

Abstract

Introduction

Despite a proliferation of statistical methodologies and developments within randomised controlled trials (RCTs) in recent decades, it is unclear which approaches are being implemented in practice. Oxford Clinical Trials Research Unit (OCTRU) is a UK Clinical Research Collaboration (UKCRC) registered Clinical Trials Unit (CTU) that has been operational since 2013 based in the Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences at the University of Oxford. We performed a review of all published RCTs conducted within OCTRU, with particular emphasis on trial methodology, statistical study design and statistical analysis.

Methods

Studies were considered eligible if they were: RCTs conducted by OCTRU, have been completed and disseminated their primary results. Studies were ineligible if they were: a pilot or feasibility trial, a simulation study, a secondary analysis of an existing RCT, or a phase I trial. Phase II trials were considered if they were randomised. We performed double data extraction of all fields for all eligible trials.

General trial information, such as primary disease area, main funding source, sample size, trial design and analysis information (e.g. number of study outcomes and analyses performed), were extracted and summarised. An analysis was defined as any time a statistical model was fit or a corresponding statistical test (e.g. χ² test) and/or estimation of a parameter was performed.

Results

Of the 142 OCTRU studies registered & funded (as of June 2023), 70 were completed and written up and 27 were eligible at the time of this review. The rest were ongoing or found to be ineligible. Included studies were published between 2014 and 2023, the majority in the last 5 years (20/27, 74% published between 2020 and 2023). All trials were multi-centre, prospectively designed and referred to both a study protocol and sample size justification (usually a power calculation) in their published results. Most included studies had elements of what could be referred to as a ‘standard’ RCT; used a parallel group design (93%), powered with superiority question (26/27, 96%), had two randomised groups (23/27, 85%) or used an equal allocation ratio (25/27, 93%).

The median sample size was 451 (interquartile range: 238–836). The median total number of analyses performed was 22 (Interquartile range: 14–30) with the most analyses performed within a single trial being 69. Eighty-one per cent (22/27) of trials had a primary outcome with either binary or continuous data. Linear mixed effects, linear regression or logistic regression was used as the primary analysis model in 74% of the 27 trials. All trials that included at least one analysis (26/27) featured at least one additional analysis on the primary outcome, the most popular additional analyses were on an alternative population (for example a per-protocol population), occurring in 20/27, 74% of all trials, or a subgroup (18/27, 67%)).

Conclusions

This review summarises RCTs conducted by one academic UKCRC-registered CTU with a focus on the trial design and statistical analysis. We found most RCTs adopted what could be considered a ‘standard’ design, using appropriate, but not complex, analysis methods. Consideration of variation in practice across other groups, both academic and commercial, through a larger review would allow systematic exploration of methodological differences, less common study design usage, and would enable a fuller understanding of practice, outcomes, and methods used in different clinical areas and contexts.

Introduction

The randomised controlled trial (RCT) is ubiquitous in medicine [1]. There have been a multitude of methodological developments over the last 100 years, particularly regarding trial design and statistical analysis. With this proliferation of statistical methodology within RCTs, it is pertinent to look at which techniques and designs are implemented in practice [2].

The Oxford Clinical Trials Research Unit (OCTRU) [3] is a UK Clinical Research Collaboration (UKCRC) registered clinical trials unit (CTU) based within the Centre for Statistics in Medicine (CSM), Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford. It has been operational since 2013. We conducted a review of all RCTs run within the clinical trials unit to ascertain which statistical techniques are most used in the CTU’s trials. We also extracted other, more general, information on the trials.

The overall aim of this project was to produce an overview of the statistical analysis of RCTs conducted by OCTRU. This was done by summarising the statistical aspects of the trial designs, use of different statistical methods, and quantifying the extent of the statistical analyses reported in randomised trial reports. This review provides an insight into RCT design and analysis methodology used in practice over the past decade.

Methods

The sample size for this descriptive review was opportunistic. All studies that were found to be eligible were included. The published RCTs conducted by OCTRU were extracted based on the following inclusion/exclusion criteria:

Inclusion criteria

• OCTRU conducted

• Randomised controlled trial

• Study analysis was completed and published in a peer-reviewed academic journal

Exclusion criteria

• Pilot or feasibility trial

• Non-randomised trial

• Simulation study

• Secondary analyses of an existing RCT

• Phase I trial

For this review, a randomised controlled trial was defined as a prospective study where human participants (or groups of humans) were randomly assigned to two or more study groups to receive some pre-determined intervention(s). All studies extracted were published in English.

Two reviewers independently performed data extractions for all variables extracted from all studies. Despite this being an internal review, with some reviewers having performed the actual analyses being extracted, reviewers only extracted information available in the public domain, i.e. the published results papers and associated supplemental materials, when available. Information was extracted onto a standardised form that was developed by the investigators prior to beginning the extractions.

Information was extracted on both general trial information and more specific methodology and analysis techniques. General trial information includes the year of publication, primary disease area, type of intervention, the type, and impact factor (as of July 2024) of the journal in which the results have been published. Some of the methodological and statistical features extracted included the type of primary outcome data, primary analysis method, total number of outcomes in the study and the number of total analyses performed for the study.

An analysis was defined as any instance where a statistical model was fit or a corresponding statistical test (e.g. χ² test) and/ or estimation of a parameter was performed. When a single model produced multiple estimates of interest, for example, a linear mixed model with a time by treatment interaction term for repeated measures data or a subgroup analysis with a treatment by baseline factor interaction term, these were counted as a single analysis. Between-group comparisons based on descriptive statistics only (i.e. had no statistical estimation or test-based comparison) were not counted as analyses.

Each study outcome may be associated with multiple analyses. For example, a common quality of life outcome used in the extracted studies is the EuroQoL EQ-5D [4]. This outcome consists of two components: a health index (range, − 0.594 to 1 [0 equivalent to death; a higher score relates to better quality of life]) and a 100-point visual analogue scale. This measure has been counted as one ‘outcome’ despite needing two tests to analyse all the score’s components.

Health economic outcomes, such as resource use and cost measures, and their associated analyses were included in the total count of study outcomes but not included in the count of total statistical analyses performed. The focus of this project is on statistical analyses so not including health economic outcomes ensures there is no artificial inflation of analysis counts due to large numbers of cost analyses that can be performed in RCTs. Translational and exploratory outcomes not covered within the statistical analysis were not included.

Data generated from the extraction were aggregated using descriptive statistics. Frequency and percentage were used for categorical data whilst medians and interquartile ranges (IQR) or means and standard deviations are presented for continuous data, as appropriate.

All extracted data are processed, and the manuscript produced using R version 4.3.1 (2023–06–16 ucrt).

Results

As of the 6th of June 2023, there were a total of 142 studies registered to OCTRU. Of these, 70 have been completed and 27 [5–29] were eligible and included in our final cohort (Fig. 1). Due to the low numbers of studies included, trial information is not presented by any stratifying variables but for the cohort overall only.

Fig. 1.

Study flow diagram

Table 1 provides information on general trial characteristics. Most trials had a design which might be considered a ‘standard’ RCT, i.e. parallel group (93%), powered with superiority question (96%), two randomised groups (85%), and an equal allocation ratio (93%). The two studies that did not use 1:1 allocation randomised used a 2:1 ratio. More than half (56%) used some form of blinding. Most studies were primarily funded from public research grants (70%) and there was an almost equal split between the results being published in General or Speciality medical journals, the median journal impact factor was 41.6 (range 2.4–98.4). All studies made reference to a sample size justification and their study protocol, the majority (81%) made reference to a statistical analysis plan. The most common primary disease area was Musculoskeletal (41%). Some of the musculoskeletal indications investigated were knee & hip replacements, wrist fractures, Achilles tendons ruptures and rotator cuff disorders. This is unsurprising given OCTRU is part of the Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences Department within the University of Oxford. All studies were multi-centre; the smallest number of recruiting centres was two, the largest was 184. The median sample size was 451. The smallest study recruited only eight participants to an osteosarcoma trial [30] before it was stopped early due to poor recruitment, no formal analyses were performed for this trial and all results presented in its results paper are descriptive. The largest trial had a sample size of 5247 and was a musculoskeletal trial with surgical intervention [31].

Table 1.

General trial information

	Total (n = 27)
Primary disease area
Dermatology	1 (4)
Gastroenterology	1 (4)
Musculoskeletal	11 (41)
Neurology	2 (7)
Oncology	4 (15)
Respiratory	3 (11)
Rheumatology	2 (7)
Surgery	3 (11)
Intervention type
Device	4 (15)
Drug	8 (30)
Non-invasive procedure	3 (11)
Physiotherapy/rehabilitation	6 (22)
Radiotherapy	1 (4)
Surgery	4 (15)
Other	1 (4)
Comparison type
Alternative active intervention	13 (48)
No intervention	1 (4)
Placebo	4 (15)
Standard of care	9 (33)
Main funding type
Charity	4 (15)
Industry	4 (15)
Public	19 (70)
Journal type
General	14 (52)
Speciality	13 (48)
Journal impact factora	n = 27, 42 (5, 97)
Year of publication
2014–2019	7 (26)
2020–2023	20 (74)
Trial design
Equivalence	1 (4)
Superiority	26 (96)
Trial design 2
Factorial	2 (7)
Parallel group	25 (93)
Blinding
Assessor blinded	6 (22)
Participant blinded	2 (7)
Participant and assessor blinded	7 (26)
No blinding	12 (44)
Number of randomised groups
Two	23 (85)
Three	2 (7)
Four	2 (7)
Randomisation method
Simple	0 (0)
Random permuted blocks	1 (4)
Random permuted block within strata	15 (56)
Minimisation	11 (41)
Adaptive	0 (0)
Allocation ratio
Equal	25 (93)
Unequal	2 (7)
Number of centres	n = 27, 19 (14, 26)
Sample size	n = 27, 451 (238, 836)
Sample size justification and/or power calculation present
Yes	27 (100)
No	0 (0)
Study protocol referenced
Yes	27 (100)
No	0 (0)
Statistical analysis plan referenced
Yes	22 (81)
No	5 (19)

Categorical data are presented as n (%), continuous data as median (IQR)

^aAs of July 2024

Table 2 gives an overview of the analyses performed in the extracted studies. The median total number of analyses performed (primary and secondary outcomes as well as any sensitivity/ supporting analyses) was 22, the mean was 24 per trial, standard deviation 15.1. The most analyses performed for any one trial was 69. There was one Bayesian trial design, however, this trial (MEMOS [30]) was the osteosarcoma trial previously mentioned with no analyses performed. To further support evidence that most OCTRU-conducted RCTs use a ‘standard’ design; 96% (26/27) of trials used a conventional, frequentist approach and the majority (89%) employed a single primary outcome, 3 (11%) used co-primary outcomes. Most trials collected a similar number of total outcomes; the range of outcomes collected was 3–15 with a median of 7.

Table 2.

Overall analysis methods

	Total (n = 27)
Overall analysis approach
Conventional (frequentist)	26 (96)
Bayesian	1 (4)
Other	0 (0)
Number of primary outcomes
One	24 (89)
Two	3 (11)
Number of secondary outcomes	n = 27, 6 (4, 10)
Total number of outcomes	n = 27, 7 (6, 10)
Total number of analyses performed	n = 27, 22 (14, 30)

Categorical data are presented as n (%), continuous data as median (IQR)

The results presented in Table 3 give insight into how the primary analysis for each trial was conducted, as well as details on what supporting analyses are performed. All trials with statistical analyses used an Intention-to-Treat population as their primary analysis population. All 26 analysed trials reported their treatment effect metric with appropriate confidence interval and most (24/26, 92%) also reported the associated p-value. No trials reported a p-value in isolation without an effect estimate for the primary analysis. Most analysed trials used a continuous (17/26, 65%) or binary (6/26, 23%) variable for their primary outcome. Of the two analysed trials with co-primary outcomes, one used two continuous measures and the other had a binary and a continuous outcome. The other trial with two primary outcomes in Table 2 was not analysed [30] and used two binary outcomes. Six (23%) analysed trials used a composite primary endpoint, for instance progression free survival where death also constitutes an event. Given the popularity of continuous and binary outcomes, it is perhaps predictable that the most common effect measures reported for a primary analysis, of the analysed trials, were a mean difference (14/26, 54%) or an odds ratio (5/26, 19%).

Table 3.

Main analysis of the primary outcome, excluding studies that had no analyses performed

	Total (n = 26)
Population target
Intention-to-treat	26 (100)
Per-protocol	0 (0)
Other	0 (0)
Data typea
Binary	6 (23)
Continuous	17 (65)
Ordinal	1 (4)
Time to event	4 (15)
Composite endpoint
Yes	6 (23)
No	5 (19)
Not applicable	15 (58)
Treatment effect metric reported
Yes	26 (100)
No	0 (0)
Treatment effect metric used
Difference in median	1 (4)
Geometric mean ratio	1 (4)
Hazard ratio	2 (8)
Mean difference	14 (54)
Odds ratio	5 (19)
Risk ratio	1 (4)
Time ratio	2 (8)
Statistical method used
Accelerated failure time	2 (8)
Cox regression	1 (4)
Linear mixed model	11 (42)
Linear regression	5 (19)
Log-rank test	1 (4)
Logistic regression	4 (15)
Mann–Whitney U test	1 (4)
Other	1 (4)
Covariate adjustment
Yes	24 (92)
No	2 (8)
All randomisation variables adjusted for
Yes	22 (85)
No	2 (8)
Not applicable	2 (8)
Number of variables adjusted for in primary analysis	n = 24, 4 (3, 5)
Multiple testing considered
Yes	3 (12)
No	1 (4)
Not applicable—single primary comparison	22 (85)
Confidence interval reported
Yes	26 (100)
No	0 (0)
P-value for treatment effect reported
Yes	24 (92)
No	2 (8)
P-value reported without estimated treatment effect
Yes	0 (0)
No	26 (100)
Further analyses (e.g. sensitivity, subgroup) performed
Yes	26 (100)
No	0 (0)
Number of further analyses performed	n = 26, 6 (4, 12)
Further analysis typea
Alternative analysis population (e.g. per-protocol)	20 (77)
Alternative modelb	1 (4)
Area under the curve	5 (19)
Complier average causal effect (CACE)	9 (35)
Different approach to missing datac	9 (35)
Extended model (additional covariates)	10 (38)
Reduced model (fewer covariates, including unadjusted)	9 (35)
Subgroup	18 (69)
Other	4 (15)
Missing data (%)	n = 26, 8 (2, 17)
Missing data handling method in primary analysis
Complete case analysis (CCA)	24 (92)
Multiple imputation (MI)	2 (8)

Categorical data are presented as n (%), continuous data as median (IQR)

^aData can have multiple options per trial, percentages are percentages of trials with this option and therefore sum to >100%

^bUse of a Cox model when the primary analysis used an accelerated failure time model

^cAlternative missing data approaches included: simple imputation, multiple imputation by chained equations, rctmiss Stata command, complete case

A linear mixed model, linear regression or logistic regression were used as the primary analysis model in 77% of all trials with at least one analysis. Although time-to-event data was the primary outcome data type in only four studies, there was a variety both of effect metrics used (hazard ratio and time ratio) and statistical methods used (accelerated failure time, Cox regression, log-rank test). Most, 92% (24/26), chose to use some form of covariate adjustment (i.e. including any variables other than the randomised group) in their primary analysis model. Three trials considered multiple testing in their primary analysis, both factorial trials and one of the two analysed trials with co-primary outcomes.

All studies that performed any analyses also performed further analyses on the primary outcome. The median number of additional analyses performed on the primary outcome was 6 (range 1–38). Most analysed studies included at least one analysis with an alternative analysis population (77%) and at least one subgroup analysis (69%). Just over one third of analysed trials (9/26, 35%) fitted a separate model with a different approach to missing data compared to what was used in the primary analysis model. Almost all analysed trials used a complete case analysis (24/26, 92%) for their primary analysis, the rest used multiple imputation.

Discussion

This is a review of the statistical design and analysis of RCTs conducted and published by a single UK trials group. To the investigators’ knowledge, no similar review, which provides a top-level summary of contemporary statistical analyses in primary RCT reports, has been reported.

Most of the trials conducted might be referred to as a ‘standard’ RCT design. They had components of a two-arm, superiority, multi-centre, parallel-group design with a single primary outcome. All trials were prospectively designed and referred to a pre-established study protocol and sample size justification (typically a power calculation). Half the analysed trials performed between 14 and 30 (IQR) separate analyses with between 4 and 12 (IQR) of those being additional analyses of the primary outcome. Our findings suggest the number of statistical analyses performed routinely in RCTs has grown over time. The 1948 trial of Streptomycin treatment in pulmonary tuberculosis by the Medical Research Council [32] contained only nine statistical significance tests, without quantification of treatment effect magnitude or related uncertainty measures. It would be interesting to know if this trend has been continuing in more recent years.

This work focuses exclusively on the main results publications for RCTs and therefore represents only a small portion of the work completed by a trial statistician and almost entirely ignores the vast work required from the wider trial team (trial managers, clinical staff, data teams, etc.) to facilitate the generation of study data to be analysed. It also did not quantify the amount of data processed for each analysis. Each analysis relies on a substantial amount of work from a large cross-functional group to process or derive the relevant data prior to analysis. As a case study, consider the DRAFFT2 trial published in 2022 [33]. This is a typical trial for this extraction; it is a two-arm, parallel-group, superiority trial, with a sample size of 500. It compared two different ways to fixate fractures of the distal radius (a broken wrist). The primary outcome measure was the Patient Reported Wrist Evaluation (PRWE) [34]; a validated score calculated from 10 questions about wrist pain and function and was collected at baseline (pre-injury (retrospectively) and post-injury), 3-month, 6-month, and 12-month post-randomisation. The primary analysis model was a linear mixed effects model adjusting for person within the recruitment centre as random effects and baseline (post-injury) PRWE values, type of fracture and age as fixed effects. Forty-four individual variables had to be collected, cleaned and in some cases manipulated further to fit this single model. A large amount of data is processed and presented for this relatively simple analysis. In the primary results table of the publication, there are 40 separate reported numbers to describe this analysis alone.

A wide range of treatment effects and analysis methods are used in this review. Naturally, a product of the type of primary outcome analysed, the treatment effects and analysis methods employed also reflect the variety of measures available to choose as a primary outcome when designing an RCT. No statistical method used may be considered ‘complex’, all use some form of regression or simple test. The strength of randomization is that when performed correctly, it produces valid comparisons between randomised groups and enables appropriate inferences through standard statistical techniques. The inclusion of health economic outcomes in a review such as this would add an additional level of understanding to the study design. This was not added to this manuscript for the reasons given but would be an opportunity for further work.

The trials reported frame their analysis populations in more traditional terms (intention-to-treat, per-protocol), rather than the modern estimands approach [35]. This is primarily a function of estimands being relatively new (2022), and trials published after 2022 being analysed as they were designed. All would have been initially worked up without the estimand framework. It will be interesting to see the development of RCT publications, particularly the description of populations and analysis approaches, in the coming years as trials designed post-estimand mature and are written up.

There is no consideration for the considerable number of descriptive statistics produced from trial data. A final results publication and its associated supplemental material is a truncated version of a final study report produced for each study. This study report includes large amounts of data on treatment compliance, withdrawals, protocol deviations/violations, data listings etc., all of which is not factored in this review. Considering only the number of analyses performed is a poor proxy for the full workload required to properly report a RCT.

Despite no analyses being performed, the MEMOS trial [30] fulfilled all the pre-defined inclusion and exclusion criteria and was therefore included in this review. This is considered a strength as it gives an accurate indication that not all trials run as planned and are successful.

An important limitation of this study is that it only included data from a single trial unit so it is likely there will be homogeneity in the methods, analyses performed and the types of trials. However, as OCTRU is a UKCRC-registered CTU, the sample may be representative of fellow UKCRC CTUs as all registered units must work under the regulatory and quality standards required by the Collaboration’s registration process [36]. OCTRU operates under a ‘Hub & Spoke’ system with each spoke having its own clinical areas of interest; these cover Oncology, Trauma and Emergency Care, Surgical Interventions, Plastics, Rehabilitation, Experimental Medicine and Rheumatology. Most of the trials run by the Unit are funded by a UK public body (e.g. through National Institute for Health and Care Research grants) or a charity (e.g. Cancer Research UK). Both the funding body and area of medicine shape the trial design [37] so our review may not be representative of all RCTs conducted globally.

As this review included RCTs exclusively, some information is representative of those trials only. For instance, if Phase I dose-escalation and/ or feasibility studies conducted by the CTU had been allowed into the review, it is likely that both the average sample size and the number of analyses performed would decrease. This is not necessarily a strength or limitation, merely the scope of this review.

To better understand the types of trials and thier analyses, a broader review of studies, including those from industry and academia, is needed. This will help examine the practical use of methodological advancements and potential differences in how RCTs are conducted between different sectors e.g. academic and industry.

The overview presented gives an insight into how RCTs are conducted from a statistical and trial design perspective, as well as what type of RCTs are performed within a single trial group (OCTRU).

Authors’ contributions

JAC conceived the idea of this review and was involved in all aspects of the study. CW, AO and EC extracted the data. AO processed the data and produced the first and subsequent drafts of the manuscript. All authors reviewed and contributed to subsequent versions. All authors read and approved the final manuscript.

Funding

No specific funding was received for this work. It has been supported by the Oxford Clinical Trials Research Unit.

Data availability

The extracted data and code used to process the data and produce this manuscript can be made available on request.

Declarations

Ethics approval and consent to participate

All RCTs included in this review obtained ethics approval and all participants were consented to each study prior to entering. No direct ethics approval or consent was required for this work.

Consent for publication

Not applicable.

Competing interests

All authors other than CW were involved in OCTRU trials assessed for inclusion in the review. No authors declared any other competing interests.