Improving clinical trial interpretation with ACCEPT analyses

Effective decision making from randomised controlled clinical trials relies on robust interpretation of the numerical results. However, the language we use to describe clinical trials can cause confusion both in trial design and in comparing results across trials. ACceptability Curve Estimation using Probability Above Threshold (ACCEPT) aids comparison between trials (even where of different designs) by harmonising reporting of results, acknowledging different interpretations of the results may be valid in different situations, and moving the focus from comparison to a pre-specified value to interpretation of the trial data. ACCEPT can be applied to historical trials or incorporated into statistical analysis plans for future analyses. An online tool enables ACCEPT on up to three trials simultaneously.

The classic superiority trial aims to generate robust evidence that a new treatment is better than placebo. Active controls are used to assess superiority of a new treatment when the use of placebo is unethical, such as when an effective treatment is available. Non-inferiority trials, aiming to show that the new treatment is not appreciably worse than the control, are often used to evaluate new drugs and interventions with similar expected efficacy to standard therapy but secondary advantages such as less toxicity, ease of implementation, benefits in particular subgroups only, or lower cost. Similarly, equivalence trials aim to show a new treatment is unlikely to differ appreciably from control in either direction and super-superiority trials aim to show evidence of a new treatment being better than control by at least a specified value.

The specification of trial type (e.g. as superiority or non-inferiority) is important to enable assessment of whether a trial has met its aims. However, the different terms are in themselves confusing. Additionally, seemingly paradoxical situations can arise when comparing across trials, such as trial type differing depending on which treatment is assigned as “control”, and trials with similar numerical results reaching different conclusions.

All clinical trial types can be linked by the pre-specified ‘unacceptable value’ to which the 95% confidence interval (CI) limits of the estimate of the difference between treatments are compared. Specification of trial type is equivalent to pre-specification of the unacceptable value: zero (or one for a relative effect measure) in superiority trials, the non-inferiority margin (less than zero) in non-inferiority trials and greater than zero in super-superiority trials.

Comparison to the pre-specified unacceptable value is an appropriate part of trial interpretation but leads to a binary conclusion of ‘trial aim met’ or ‘trial aim not met’. Outside drug regulation, binary conclusions are widely viewed as problematic¹, because evidence should not be reduced to a single threshold but should be considered in context with other factors such as the point estimate and CI². Interpretation of non-inferiority trial results also suffers from added complexity around whether the pre-set non-inferiority margin was justified or is relevant for settings outside the trial. Importantly, stakeholders such as clinicians, patients or policy makers may have differing but equally valid unacceptable difference values depending on the relative importance placed on secondary factors such as cost or toxicity.

We advocate the wider use of ACCEPT as secondary analyses in clinical trials. We illustrate this using two HIV trials, EARNEST³ and SECOND-LINE⁴, which had similar quantitative results but drew different conclusions⁵. We demonstrate how alternative presentation of results could aid better comparison and integration of their findings. We present the trials together, but imagine ACCEPT being presented in each trial results paper separately as secondary analyses. For ease throughout we measure differences as treatment minus control for a favourable outcome, so that positive values indicate higher efficacy in the tested treatment.

EARNEST and SECOND-LINE investigated raltegravir as a second-line therapy for HIV in comparison to standard therapy of nucleoside reverse transcriptase inhibitors (NRTI). EARNEST, was carried out in low and middle-income countries. It was pre-specified as a superiority trial because raltegravir was more expensive than NRTI; therefore, it was thought that clear benefit would have to be shown for implementation. The pre-specified unacceptable value was consequently zero.

SECOND-LINE, was carried out predominantly in high-income countries. It was pre-specified as a non-inferiority trial because raltegravir was considered to have a better toxicity profile than NRTI: implementation was therefore considered to be worthwhile with similar efficacy. The pre-specified unacceptable value was the non-inferiority margin of minus 12% on the risk difference scale.

The original analysis of EARNEST compared the lower limit of the 95% CI of the difference between treatments (-2.4%) to the unacceptable value of 0, drawing the conclusion of ‘superiority not shown’ and implementation was not recommended. Analysis of SECOND-LINE compared the lower limit of the 95% CI (-4.7%) to the unacceptable value of -12%, drawing the conclusion of ‘non-inferiority’ and implementation was recommended. The question then arises of how two trials with numerically similar results can reach opposing conclusions regarding implementation.

Differing, valid, opinions on the unacceptable differences values (0% in EARNEST and -12% in SECOND-LINE), driven in part by different emphasis on secondary benefits, led to the selection of different trial types and the resulting seemingly opposing recommendations. Interpretation through ACCEPT, including both graphs and tables, would have helped to clarify this paradox enabling more nuanced interpretation of the results.

ACCEPT uses the primary analysis from a trial to plot the probability of the true difference between treatments being above an ‘acceptability threshold’ for a range of possible threshold values (Figure 1). ACCEPT can be presented for all trial types and outcomes. ACCEPT has only been used sporadically in clinical trials ^6-12 with no common naming. ACCEPT is similar to cost-effectiveness acceptability curves widely used in health economics, where weight of evidence, rather than binary conclusions, is a more widely accepted paradigm.

Figure 1.

Acceptability Curve Estimation using Probability Above Threshold (ACCEPT) curves and tables for EARNEST and SECOND-LINE trials. An acceptability value is the probability that the true treatment difference is at least the acceptability threshold. Positive values indicate raltegravir is better than NRTI. For example, the probability that the true difference between treatments was at least 0 (i.e. that raltegravir is better than NRTI) was 89% in EARNEST and 70% in SECOND-LINE.

Median estimates, 95% credible intervals from models, and pre-specified unacceptable differences are marked.

ACCEPT output is best presented in a graph with associated tables. A graph shows a continuous range of acceptability thresholds where greater uncertainty around point estimates (with larger associated confidence intervals) is reflected in a shallower slope. Additional tables present selected acceptability thresholds or the probability that the true value is between selected thresholds. To enable comparison of ACCEPT between trials, tables should include acceptability values for the unacceptable difference (specified in the trial design), zero, a reasonable range of potential alternative unacceptable values, and acceptability thresholds for the 2.5^th, 50^th, and 97.5^th percentile acceptability values.

ACCEPT can be implemented using Bayesian analysis, which provides direct estimation of the probability one treatment is better or worse than another when the prior belief of the difference between treatments is added to the analysis. The degree of prior belief, termed priors, is based on existing data and/or expert opinion. Priors are specified as a distribution over the possible values that the difference can take, with non-informative priors essentially being a flat distribution, and strongly informative priors being very concentrated around the area of highest belief. ACCEPT within a Bayesian framework is more consistent with the overall philosophy than within a frequentist framework. However, frequentist analysis using confidence curves^{13 14} is expected to give very similar results to Bayesian analysis with uninformative priors. In the frequentist framework, the acceptability value is the one-sided p-value for the treatment effect exceeding the acceptability threshold. An online tool enables ACCEPT for up to three trials simultaneously using summary information from frequentist or Bayesian analysis https://egon.stats.ucl.ac.uk/projects/ACCEPT/. Further details of analyses, including statistical code, is available in the supplementary information.

Using ACCEPT for trial reporting, EARNEST results would still conclude that superiority was not shown but could also include a statement such as ‘ACCEPT suggested that there was a 89% probability that the true treatment difference was greater than zero (i.e. that raltegravir was better than NRTI) and 100% probability that the true treatment difference was above -5, equivalent to a 0% probability that raltegravir was worse than NRTI by at least five percentage points. There was a 39% probability raltegravir was better than NRTI by at least five percentage points.’

Similarly, reporting of SECOND-LINE results with ACCEPT would still conclude that non-inferiority was shown but could also add a statement stating ‘ACCEPT suggested that there was a 70% probability of the true treatment difference being greater than zero, a 97% probability of the true treatment difference being above -5 percentage points, and a 16% probability that raltegravir was better than NRTI by at least five percentage points’.

Using ACCEPT, stakeholders requiring clear benefit of raltegravir for implementation could use acceptability thresholds of zero and above, concluding that the probability of raltegravir being better than NRTI was 89% in EARNEST and 70% in SECOND-LINE, but the probability of being more than 5 percentage points better was much lower at 39% in EARNEST and 16% in SECOND-LINE. Other stakeholders more focused on other secondary benefits of raltegravir, such as lower toxicity, could use acceptability thresholds of zero and below, concluding almost certainty of the true treatment difference being more than -5 percentage points in either trial. This allows better comparison across trials than the primary analysis alone.

Interpretation through ACCEPT has three main strengths. Firstly, it enables comparison between trials and trial types by harmonising reporting of results; the use of probabilities is straightforward, widely understood and reflects the uncertainty around the point estimate. Secondly, presentation of ACCEPT acknowledges different acceptability threshold may exist in different situations. ACCEPT allows clinicians, policymakers, and patients to make informed decisions based on their setting and individual circumstances if they feel the original choice of unacceptable difference is not appropriate for their context. Thirdly, ACCEPT moves the focus from comparison with the pre-specified unacceptable value to interpretation of the trial data. This may be especially useful for trials pre-specified as non-inferiority to reduce focus on the selected unacceptable value and in situations where restricted sample size reduces power, such as subgroup analysis and uncommon conditions. For subgroup analysis, ACCEPT can be presented separately for each subgroup using output from either models run separately for each subgroup or a single model where an interaction between subgroup and trial arm is fitted.

Use of ACCEPT does not remove all of the concerns that can arise with non-inferiority trials, which are caused by the unacceptable difference being less than zero. Non-inferiority trials cannot always provide assurance that the new treatment has a clinically relevant effect (greater than zero) relative to placebo and so it is important to carefully assess evidence about how much better the control treatment is than placebo when selecting the pre-specified unacceptable difference/non-inferiority margin to prevent biocreep. Non-adherence in clinical trials may bias the estimate of treatment differences towards zero, especially if treatment cross-over occurs, meaning that conclusions of non-inferiority may be more likely with substantial non-adherence. Analysis of different trial populations (per protocol and intention to treat) or statistical adjustment must still be used to allow for this, but ACCEPT can help improve interpretation when comparing across different populations within a trial.

ACCEPT can be applied to historical trials or incorporated into statistical analysis plans for future analyses. ACCEPT has been previously advocated for use in clinical trials reporting, but its use has not become widespread, perhaps due to lack of common language to discuss the analyses. Increased use of a variety of different trial designs means the time is right for unified design and interpretation though ACCEPT.