Abstract
Background
Clinical trials are of central importance for the evaluation and comparison of treatments. The transparency and intelligibility of the treatment effect under investigation is an essential matter for physicians, patients, and health-care authorities. The estimand framework has been introduced because many trials are deficient in this respect.
Methods
Introduction, definition, and application of the estimand framework on the basis of an example and a selective review of the literature.
Results
The estimand framework provides a systematic approach to the definition of the treatment effect under investigation in a clinical trial. An estimand consists of five attributes: treatment, population, variable, population-level summary, and handling of intercurrent events. Each of these attributes is defined in an interdisciplinary discussion during the trial planning phase, based on the clinical question being asked. Special attention is given to the handling of intercurrent events (ICEs): these are events—e.g., discontinuation or modification of treatment or the use of emergency medication—that can occur once the treatment has begun and might affect the possibility of observing the endpoints or their interpretability. There are various strategies for the handling of ICEs; these can, for example, also reflect the existing intention-to-treat (ITT) principle. Per-protocol analyses, in contrast, are prone to bias and cannot be represented in a sensible manner by an estimand, although they may be performed as a supplementary analysis. The discussion of potential intercurrent events and how they should appropriately be handled in view of the aim of the trial must already take place in the planning phase.
Conclusion
Use of the estimand framework should make it easier for both physicians and patients to understand what trials reveal about the efficacy of treatment, and to compare the results of different trials.
cme plus
This article has been certified by the North Rhine Academy for Continuing Medical Education. Participation in the CME certification program is possible only over the internet: cme.aerzteblatt.de. The deadline for submission is 26 December 2022.
Clinical trials represent one of the cornerstones of evidence-based medicine and are intended to investigate a medical question of interest, the answer to which is formulated as a trial objective. For a trial in which a new drug to treat type-2 diabetes is under investigation, one could formulate the following trial objective: “The objective of the trial is to investigate whether drug x reduces the mean change in HbA1c from treatment initiation to 24 weeks compared to placebo.” The formulation of the trial objective makes it clear that the treatment effect of the experimental treatment on the change in HbA1c is to be determined. However, many aspects remain implicit, such as dealing with patients that do not take the drug as intended. In reality, deviations from the intended course of treatment will be observed. These may include events such as treatment discontinuation/modification or use of emergency medication, thereby affecting the observed values. If one is interested in the treatment effect under ideal conditions without events of this kind, then the assessment based on such observed patient data is no longer straightforward. Specifying how such events should be dealt with in the actual trial is important. In the trial used here as an example, one needs to consider, among other things, how the changes in HbA1c in patients that received emergency medication during the course of the trial will be taken into account. One possible option is to consider all observations irrespective of the use of emergency medication. However, if one wishes to investigate the isolated effect of the trial treatment without the emergency medication, observed values following the use of the emergency medication would not be used as such. Although both options are feasible, each addresses a different question of interest.
The estimand framework offers a systematic approach to avoiding the abovementioned areas of uncertainty and resulting problems by means of clear and pre-specified definitions. The following article will define the estimand framework and will illustrate its application using the example trial. It charts the shift from current approaches to the estimand framework and outlines the latter’s practical application as well as its scientific relevance.
General definition of the estimand framework
The estimand is a systematic description of the treatment effect to be quantified in order to answer the trial’s research objective. The estimand consists of the following five attributes: Treatment, Population, Variable, Population-Level Summary, and Handling of Intercurrent Events (ICEs). These terms are deliberately left in English since this is the language of the primary literature on the estimand framework and enables uniform and unambiguous communication across language borders. For the definition of the estimand, the individual attributes are specified based on the clinical question being asked. ‘Treatment’ describes the treatments to be investigated, while ‘Population’ defines which group of patients is addressed. The ‘Variable’ describes the measure based on which the treatment will be assessed (endpoint), and the ‘Population-Level Summary’ specifies how the patient-specific endpoints for the group comparison are summarized, for example, as the difference between the two group means. Examples of this can be found in the Table, where the attributes for the example trial are given. It is important to note here that there is an interplay between the attributes, hence they cannot be considered separately.
Table. Example formulation of the attributes of the described estimand strategies for the example trial on diabetes in relation to the intercurrent event (ICE) ‘use of emergency medication.’ The trial objective is the starting point for the definition of the attributes.*.
| Strategy Attribute | Ideal | Treatment Policy | Hypothetical | Composite Variable | While on Treatment |
| Handling of Intercurrent Events | Use of emergency medication not necessary | Use of emergency medication is ignored and shown in the definition of the treatment | As if no emergency medication had been used (hypothetical case without ICE) | Use of emergency medication is shown in the definition of the variable | Use of emergency medication is shown in the definition of the variable |
| Treatment | Experimental versus control treatment | Experimental & emergency medication vs. control treatment & emergency medication | Experimental versus control treatment | Experimental versus control treatment | Experimental versus control treatment |
| Population | Defined by inclusion and exclusion criteria | Defined by inclusion and exclusion criteria | Defined by inclusion and exclusion criteria | Defined by inclusion and exclusion criteria | Defined by inclusion and exclusion criteria |
| Variable | Change in HbA1c from baseline to 24 weeks | Change in HbA1c from baseline to 24 weeks | Change in HbA1c from baseline to 24 weeks (Note: values post ICE not to be used as such → suitable statistical estimation methods) | Treatment success = change in HbA1c from baseline to 24 weeks ≥ threshold & no emergency medication | Change in HbA1c from baseline to 24 weeks or to use of emergency medication |
| Population-Level-Summary | Mean difference | Mean difference | Mean difference | Rate difference | Mean difference |
| Trial Objective | Investigation of the mean difference in the change in HbA1c from baseline to 24 weeks between experimental and control treatment | Investigation of the mean difference in the change in HbA1c from baseline to 24 weeks between experimental & emergency medication and control treatment & emergency medication | Investigation of the mean difference in the change in HbA1c from baseline to 24 weeks between experimental and control treatment without use of emergency medication | Investigation of the difference in rates regarding treatment success, defined as change in HbA1c from baseline to 24 weeks greater or equal to threshold and no emergency medication, between experimental and control treatment | Investigation of the mean difference in the change in HbA1c from baseline to 24 weeks or to the use of emergency medication between experimental and control treatment |
*The table shows how the individual attributes of the estimand change when the various strategies are used for the ICE ‘use of emergency medication.’
Each strategy addresses a different trial objective. Differences in the trial objective and in the attributes compared to the ideal in which no ICEs occur are written in bold.
Since the ideal is not a strategy, a different color background is used.
ICEs are events that can occur once the treatment has started, affect the presence and/or interpretability of the observed variable, and are not bound to the trial setting (for example, treatment discontinuation due to toxicities, use of emergency medication). ICEs are not the same as missing values. Missing values are values that theoretically exist and are useful but which have not been observed (for example, due to loss to follow up, lost laboratory sample). An ICE, on the other hand, can result in a value being observed but not usable in any helpful way in the analysis, or in a value not being observed at all, as for example in the case of the ICE “death.” The Figure shows a schematic representation of possible patient-specific courses over the trial, including ICEs. There is no comprehensive list of relevant ICEs that is valid for all trials, since the specific trial situation (indication, treatments, follow-up period, among others) determines which ICEs may occur. Different ICEs may be seen in trials of long-term medication (for example, discontinuation of the medication) compared to surgical trials (for example, re-operation).
Figure.
Schematic representation of ICEs over the patient-specific course in an example trial.
A central aspect of the estimand framework is the targeted focus on possible ICEs and how they are dealt with. The framework offers various strategies to do this. The four most important of these are described and illustrated below using the ICE ‘Use of emergency medication’ in the example trial. Further strategies are proposed in the literature (1– 4). It is important to note that the selected strategy impacts the other attributes, as shown and marked in bold in the Table.
The ‘treatment policy’ strategy ignores the ICE. This means that the observed values of the endpoint (variable) are included in the analysis, irrespective of whether or not the ICE previously occurred. In the example trial, this means that the change in HbA1c at 24 weeks is used, even if the patient previously received emergency medication.
The ‘hypothetical’ strategy evaluates the treatment effect in a hypothetical scenario in which the observed ICE would not have occurred. This means that values observed after the occurrence of an ICE are not included as such in the analysis. Instead, suitable statistical methods are used in order to estimate the treatment effect that would have been seen had the ICE not occurred. Thus, in the example trial, the actual, observed HbA1c value at 24 weeks would not be included in the evaluation in the case of patients that received emergency medication.
The ‘composite variable’ strategy views the ICE as a source of information and integrates it in the definition of the endpoint. This results in a composite endpoint. If one has, as in the example trial, a continuous endpoint (change in HbA1c) and an ICE, one can, for instance, convert the continuous endpoint into a binary endpoint. To this end, a threshold value that reflects a clinically relevant change is set. The combined endpoint then describes for each patient whether there has been success or failure, with success defined as exceeding the threshold value while at the same time not having used any emergency medication.
In the ‘while on treatment’ strategy, the focus shifts in terms of the temporal classification of the treatment effect—this is now only considered up until immediately before the time of onset of the ICE. In the example trial, this means that the change in HbA1c is investigated from the time of treatment initiation to 24 weeks or the time at which emergency medication was used.
The ‘treatment policy’ and ‘hypothetical’ strategies were already touched upon in the Introduction in the context of the example trial. The example relates to a trial comparing dapagliflozin to placebo (5) in which the sponsor chose a hypothetical strategy to take emergency medication into account in the analysis, whereas in talks on approval, the regulatory authorities requested a ‘treatment policy’ strategy (6). The trial in question was conducted before the estimand framework was introduced, and the application of the strategies was implicit. However, this shows that such differing expectations can be avoided with the estimands framework by specifying early on the strategies to be used for handling of ICEs.
Much of the content of the described attributes has already been specified in clinical trials, for example, which treatment will be investigated in which patient population, which endpoint is to be recorded, and how this is to be summarized. What is new is that, by using the attributes, this is now presented more accurately and more systematically. This should also be reflected in the relevant trial documents (including protocol and publication of results). The handling of events following treatment initiation, which now come under the uniform name intercurrent events, was also mostly thought of in terms of an intention-to-treat (ITT) or a per-protocol (PP) analysis. In the latter, only those patients that completed the trial according to protocol were taken into consideration. However, if the protocol violations that could also result from an ICE are related to treatment, this can lead to unequal groups and thus to biased results (7, 8). In the example trial, it could be that the trial medication causes more side effects in older patients, thereby leading to treatment discontinuation. If this is deemed to be a protocol violation, these patients would be excluded from a PP analysis, and unequal groups would be compared with each other. PP analyses cannot be represented in a sensible manner by an estimand and will play a minor role in future as supplementary analyses (1). In an ITT analysis, all randomized patients are considered in the group to which they were randomized, irrespective of what happens thereafter (randomization means random assignment to a group). Although this approach often most closely reflects clinical routine, an ITT analysis does not necessarily evaluate in all situations the effect that is of interest to the regulators, patients, and/or clinicians (9– 11). In the estimand framework, the ‘treatment policy’ strategy equates to the ITT principle since, here too, what happens following randomization is ignored. Having said that, ‘hypothetical,’ ‘while on treatment,’ and ‘composite variable’ strategies also include all randomized patients in the analysis (12). Here, however, the handling of each individual ICE is explicitly specified, whereas in an ITT analysis, there is a tendency to give no explicit thought to ICEs, although the act of ignoring them results in answering a different research question. Therefore, the novel content-related features of the estimand framework come into play particularly as a result of the precisely defined handling of ICEs, which is then also reflected in the other attributes.
Practical application and relevance
In practical application, the first step involves a medical question, on the basis of which the estimand is defined. It is important here that the clinician and statistician jointly address possible ICEs and their handling, whereby the expertise and clinical experience of physicians is elementary. In a joint exchange, these experts should discuss possible ICEs and specify how they are to be handled, in order that the design, implementation, and analysis strategies of the trial can be set out in alignment with these specifications (13, 14). In this way, the treatment effect that is of interest can be evaluated at the end of the trial. The treatment effect of primary interest can differ depending on the individual perspective. A physician and their patient would probably like to know which treatment effect they can expect if the treatment is carried out as intended. For regulatory authorities, the relevant effect may be the one that can be expected in clinical routine. This generally equates to an average effect for the patient population that incorporates influences from events such as premature treatment discontinuation (9).
A variety of ICEs can occur in a trial, and a strategy for handling each of these ICEs needs to be defined. Also, different strategies can be deployed for the various ICEs. For example, emergency medication use could be handled with a ‘composite’ strategy and treatment discontinuation with a ‘treatment policy’ strategy. Both approaches influence the attributes and, in an interplay, yield the estimand that reflects the trial objective. A new estimand emerges as soon as an attribute and/or the strategy for an ICE is altered. In a trial, it is important to define in the planning phase the primary estimand that belongs to the investigation of the primary trial objective. Additional estimands that differ from the primary estimand, and thus address different questions, can also be defined. It should be pointed out that the primary estimand is not the same as the primary endpoint. The endpoint is part of the estimand and is represented by the ‘Variable’ attribute. The endpoint alone is not sufficiently informative in relation to the question at hand. This also means that separate estimands are defined for secondary objectives. These include the secondary endpoint as Variable and, together with the other attributes, the estimand produces a more precise specification compared to using the endpoint alone.
Systematic reviews will also benefit from the more precise definition of the trial objective and the transparent description of the quantified treatment effect. These reviews aim to identify and summarize trials that have investigated the same question of interest and often include meta-analyses in which individual trial results are combined into one total estimator across all trials. Trials that use the estimand framework simplify the selection of suitable and, above all, comparable trials for systematic reviews.
The estimand framework came into effect in 2020 as an addendum to the ICH E9 guideline “Statistical principles for clinical trials” (1) and has received great attention among statisticians since the publication of the draft version in 2017. For clinicians, the topic is of considerable relevance despite the fact that it can be found in a statistically oriented guideline, since it relates to the planning and interpretation of clinical trials in general. By embedding the estimand framework in the ICH guidelines, it will become a basic element of clinical trials in the near future. In the short term, the framework will come into play mainly in the planning of new trials and, in the mid-term, become manifest in the scientific publications of trial results. In their author guidelines, scientific journals often refer to guidelines such as the CONSORT statement, which already addresses the estimand framework in its “Adaptive Designs Extension Statement” (15). One can assume that this will also be adopted in other guidelines and statements (for example, SPIRIT), thereby representing a necessary prerequisite for the publication of trial results in line with good scientific practice.
The ICH offers extensive online training material on the topic of estimands (16). The material also includes suggestions as to where and/or how estimands can be described in the trial protocol.
In addition to numerous publications in statistics journals, the first articles addressing the topic of estimands in general have appeared in clinical journals (for example, [17–21]). The estimand framework has also already found its way into indication-specific guidelines (for example, [22, 23]). Furthermore, there are publications on trial protocols and the corresponding trial results that have already implemented the estimand framework (24, 25).
Conclusion and outlook
The estimand framework offers a systematic approach to defining the treatment effect investigated in a trial in such a way that the initial clinical question being asked can be answered. This puts greater focus on the questions: “What could/will happen during the trial?” and “How will this impact the treatment effect?” Addressing these questions should already form a basic element of the planning phase. With the estimand framework, a clearer and, above all, uniform framework has now been created to this end. Before the estimand framework can become a basic element of clinical trials, efforts are needed to raise awareness of the framework. This could be achieved, for example, through targeted further training programs for trial teams.
The implementation of the framework in the planning phase can initially lead to somewhat greater time requirements. However, this should not be regarded as a disadvantage, since the greater clarity and transparency represent a sign of quality, as well as preventing misunderstandings and problems in the further course and in the final analysis. This transparency increases our understanding of the efficacy of treatments. The framework has the potential to improve the reproducibility of trial results—an important aspect in view of the prevailing reproducibility crisis in medical research (26– 28). Furthermore, the framework is intended to simplify the comparison of trial results in the future and support practicing physicians as well as affected patients in an informed decision-making process according to the principles of evidence-based medicine.
BOX. Glossary.
Estimand
Detailed description of the treatment effect of interest to be quantified, consisting of the five attributes: Treatment, Population, Variable, Handling of Intercurrent Events (ICEs), and Population-Level Summary.
Intercurrent Event (ICE)
Events that occur after the start of the trial and affect the presence and/or interpretability of observed values.
Treatment Policy
With the ‘treatment policy’ strategy, the ICE is ignored. Observed values of the endpoint (variable) are included as such in the evaluation, irrespective of whether or not the ICE previously occurred.
Hypothetical
Here, the treatment effect is evaluated for a hypothetical scenario in which the observed ICE did not occur. Values that are observed after the occurrence of an ICE are not included as such in the evaluation; instead, suitable statistical methods (for example, imputation, censoring) are chosen to estimate the treatment effect that would have been observed if the ICE had not occurred.
Composite Variable
With this strategy, the ICE is regarded as informative and included in the definition of the endpoint, for example, in the form of a combined endpoint.
While on Treatment
With the ‘while on treatment’ strategy, the treatment effect up to the time of onset of the ICE is of interest.
Intention to Treat (ITT)
In an ‘intention-to-treat’ analysis, all randomized patients are considered in the group to which they were randomized, irrespective of what happens thereafter.
Per Protocol (PP)
In a ‘per-protocol’ analysis, all patients that completed the trial according to protocol are taken into consideration.
Questions on the article in issue 51–52/2022:
Estimands—A Basic Element for Clinical Trials
The submission deadline is 26 December 2022. Only one answer is possible per question. Please select the answer that is most appropriate.
Question 1
What does the abbreviation ICE stand for in conjunction with the estimand framework?
Intercurrent Effect;
Interrupting Event;
Improbable Event
Intercurrent Events;
Interrupting Effect
Question 2
Which patients are taken into consideration according to the ITT principle?
All randomized patients
Those patients that strictly adhered to the treatment plan
All patients that at least started the treatment
Only those patients in which no deviations from the trial protocol occurred
All patients that discontinued the treatment during the course of the trial
Question 3
How many attributes does an estimand consist of?
Three;
four;
five;
six;
seven
Question 4
Which estimand strategy ignores the ICE and includes all observed values of the endpoint in the analysis?
In the ‘composite variable’ strategy
In the ‘treatment policy’ strategy
In all estimand strategies
In the ‘hypothetical’ strategy
In the ‘while on treatment’ strategy
Question 5
At what point in time must the primary estimand be defined?
At onset of the first ICE in the course of the trial
Once all patients have been included in the trial
During the planning of the trial
Once data collection has been completed
Once the first patients have completed the trial treatment
Question 6
Which characteristic does an ICE have?
ICEs occur at least once per patient.
Each missing value is an ICE.
As part of the estimand framework, all patients with an ICE are excluded from the trial.
ICEs affect the presence or interpretability of the target variable.
It is documented in a list that applies to all clinical trials.
Question 7
In relation to the estimand framework, what is the procedure for secondary questions of interest in a trial?
Separate estimands are defined for secondary questions.
The same estimand is used for secondary and primary questions.
Secondary questions are evaluated according to the principle of per-protocol analysis.
Secondary questions cannot be represented with the estimand framework.
When using the estimand framework, no provision is made for secondary questions in the trial design.
Question 8
Which estimand strategy does not include observed data as such in the evaluation?
The ‘hypothetical’ strategy
The ‘while on treatment’ strategy
The ‘treatment policy’ strategy
The ‘composite variable’ strategy
None of the estimand strategies
Question 9
How is the primary estimand in a trial defined?
By the clinician alone
By the statistician alone
The primary estimand is directly derived from the primary endpoint
By random selection of the strategy for handling ICEs
In an interdisciplinary exchange
Question 10
What applies if several different ICEs are expected in a trial?
A common handling strategy should be applied for all ICEs.
The selected strategy is applied for the most frequent ICE, while all other ICEs are ignored.
Different handling strategies can be applied for the various ICEs.
If different ICEs are expected, the ‘composite variable’ strategy must be applied.
If different ICEs occur, a ‘per-protocol’ analysis must be carried out.
Acknowledgments
Translated from the original German by Christine Rye.
Footnotes
Conflict of interest statement The authors declare that no conflict of interests exists.
References
- 1.ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials - Step 5, EMA/CHMP/ICH/436221/2017. www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials (last accessed on 25 October 2021) 2020 [Google Scholar]
- 2.Darken P, Nyberg J, Ballal S, Wright D. The attributable estimand: a new approach to account for intercurrent events. Pharm Stat. 2020;19:626–635. doi: 10.1002/pst.2019. [DOI] [PubMed] [Google Scholar]
- 3.Michiels H, Sotto C, Vandebosch A, Vansteelandt S. A novel estimand to adjust for rescue treatment in randomized clinical trials. Stat Med. 2021;40:2257–2271. doi: 10.1002/sim.8901. [DOI] [PubMed] [Google Scholar]
- 4.Bornkamp B, Rufibach K, Lin J, et al. Principal stratum strategy: Potential role in drug development. Pharm Stat. 2021;20:737–751. doi: 10.1002/pst.2104. [DOI] [PubMed] [Google Scholar]
- 5.Ferrannini E, Ramos SJ, Salsali A, Tang W, List JF. Dapagliflozin monotherapy in type 2 diabetic patients with inadequate glycemic control by diet and exercise: a randomized, double-blind, placebo-controlled, phase 3 trial. Diabetes care. 2010;33:2217–2224. doi: 10.2337/dc10-0612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Holzhauer B, Akacha M, Bermann G. Choice of estimand and analysis methods in diabetes trials with rescue medication. Pharm Stat. 2015;14:433–447. doi: 10.1002/pst.1705. [DOI] [PubMed] [Google Scholar]
- 7.Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: intention-to-treat versus per-protocol analysis. Perspect Clin Res. 2016;7:144–146. doi: 10.4103/2229-3485.184823. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kabisch M, Ruckes C, Seibert-Grafe M, Blettner M. Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2011;108:663–668. doi: 10.3238/arztebl.2011.0663. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Keene ON, Ruberg S, Schacht A, et al. What matters most? Different stakeholder perspectives on estimands for an invented case study in COPD. Pharm Stat. 2020;19:370–387. doi: 10.1002/pst.1986. [DOI] [PubMed] [Google Scholar]
- 10.Keene ON, Wright D, Phillips A, Wright M. Why ITT analysis is not always the answer for estimating treatment effects in clinical trials. Contemp Clin Trials. 2021;108 doi: 10.1016/j.cct.2021.106494. 106494. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Akacha M, Bretz F, Ruberg S. Estimands in clinical trials—broadening the perspective. Stat Med. 2017;36:5–19. doi: 10.1002/sim.7033. [DOI] [PubMed] [Google Scholar]
- 12.Leuchs AK, Brandt A, Zinserling J, Benda N. Disentangling estimands and the intention-to-treat principle. Pharm Stat. 2017;16:12–19. doi: 10.1002/pst.1791. [DOI] [PubMed] [Google Scholar]
- 13.Mallinckrodt C, Molenberghs G, Lipkovich I, Ratitch B. Estimands, estimators and sensitivity analysis in clinical trials 1st ed. Boca Raton. Chapman and Hall/CRC. 2020 [Google Scholar]
- 14.Fang Y, Jin M. Sample size calculation when planning clinical trials with intercurrent events. Ther Innov Regul Sci. 2021;55:779–785. doi: 10.1007/s43441-021-00284-x. [DOI] [PubMed] [Google Scholar]
- 15.Dimairo M, Pallmann P, Wason J, et al. The adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design. BMJ. 2020;369 doi: 10.1136/bmj.m115. m115. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.ICH E9(R1) Expert Working Group Draft (Step 2) guideline ICH E9(R1) - estimands and sensitivity analysis in clinical trials - training module 1-3, www.database.ich.org/sites/default/files/E9-R1_EWG_Step2_TrainingMaterial.pdf(last accessed on 25 October 2021) [Google Scholar]
- 17.Fiero MH, Pe M, Weinstock C, et al. Demystifying the estimand framework: a case study using patient-reported outcomes in oncology. Lancet Oncol. 2020;21:e488–e494. doi: 10.1016/S1470-2045(20)30319-3. [DOI] [PubMed] [Google Scholar]
- 18.Wharton S, Astrup A, Endahl L, et al. Estimating and reporting treatment effects in clinical trials for weight management: using estimands to interpret effects of intercurrent events and missing data. Int J Obes. 2021;45:923–933. doi: 10.1038/s41366-020-00733-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Donohue MC, Model F, Delmar P, et al. Initiation of symptomatic medication in Alzheimer‘s disease clinical trials: hypothetical versus treatment policy approach. Alzheimers Dement. 2020;16:797–803. doi: 10.1002/alz.12058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Kahan BC, Morris TP, White IR, Carpenter J, Cro S. Estimands in published protocols of randomised trials: urgent improvement needed. Trials. 2021;22 doi: 10.1186/s13063-021-05644-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Aroda VR, Saugstrup T, Buse JB, Donsmark M, Zacho J, Davies MJ. Incorporating and interpreting regulatory guidance on estimands in diabetes clinical trials: the PIONEER 1 randomized clinical trial as an example. Diabetes Obes Metab. 2019;21:2203–2210. doi: 10.1111/dom.13804. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.European Medicines Agency. Guideline on the clinical investigation of medicines for the treatment of Alzheimer’s disease, CPMP/EWP/553/95 Rev . 2020;2 [Google Scholar]
- 23.European Medicines Agency. Guideline on clinical investigation of medicinal products in the treatment or prevention of diabetes mellitus - Draft, CPMP/EWP/1080/00 Rev. 2018;2 [Google Scholar]
- 24.Wilding JPH, Batterham RL, Calanna S, et al. Once-weekly semaglutide in adults with overweight or obesity. N Engl J Med. 2021;384:989–1002. doi: 10.1056/NEJMoa2032183. [DOI] [PubMed] [Google Scholar]
- 25.Aroda VR, Rosenstock J, Terauchi Y, et al. PIONEER 1: Randomized clinical trial of the efficacy and safety of oral semaglutide monotherapy in comparison with placebo in patients with type 2 diabetes. Diabetes care. 2019;42:1724–1732. doi: 10.2337/dc19-0749. [DOI] [PubMed] [Google Scholar]
- 26.Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2 doi: 10.1371/journal.pmed.0020124. e124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Niven DJ, McCormick TJ, Straus SE, et al. Reproducibility of clinical research in critical care: a scoping review. BMC Med. 2018;16 doi: 10.1186/s12916-018-1018-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Stupple A, Singerman D, Celi LA. The reproducibility crisis in the age of digital medicine. npj Digital Medicine. 2019;2 doi: 10.1038/s41746-019-0079-z. [DOI] [PMC free article] [PubMed] [Google Scholar]