R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 11

Abstract

In this latest update we highlight a study from the REPEAT initiative that evaluates the reproducibility of real-world data studies, the publication of the HARPER Protocol Template developed by a joint ISPE/ISPOR taskforce, and discuss recent US FDA guidance on external control arms.

In order for studies using real-world data (RWD) to be more widely accepted for health technology assessment (HTA) decision making, reproducibility of findings is important. Reproducibility essentially means getting the same results when analyzing the same dataset to address the same research question. There are many choices that researchers can make when designing and carrying out a RWD study, including everything from nuances in how outcomes and exposures are defined to details in how statistical methods are applied. Demonstrating that the findings of a published study can be replicated by an independent researcher shows that these key details have been reported thoroughly and accurately. Perhaps, the most comprehensive analysis of the ability to reproduce RWD studies to date is contained in a recent publication from the REPEAT initiative [1]. The authors found 150 RWD studies for replication using systematic searches of publications utilizing one of four commonly used RWD sources from the USA and UK, for either descriptive or comparative effectiveness/safety studies. Sample sizes, baseline characteristics and effect estimates between the original and reproduced studies were found to be close on average, indicating strong overall reproducibility, although there were some outliers that were less reproducible. There was little evidence to suggest that reproducibility of associations was related to key study characteristics such as effect size, statistical significance, study funding source, data source or year of publication. Three reasons were identified for studies not being reproducible, including: incomplete information on design and analysis parameters (e.g., details regarding when study parameters were measured relative to the study entry date, or absence of detailed algorithms for study parameters); within-study inconsistency in the reporting of parameters (e.g., differences in what is reported in the text and figures); and incomplete information about the version of the data source used (important as despite knowing study periods, some data sources retroactively update historical years of data, resulting in shifts in the underlying population as well as how certain variables are populated). From the HTA perspective, this study is helpful in demonstrating that the quality of RWD study reporting is generally fairly good – having enough information to replicate a study means there is likely also enough detail to critically appraise the quality of the evidence generated by the study. Nonetheless improvements can be made, particularly in the gaps in design and data source reporting which the study identified. Greater transparency when communicating critical details of RWD studies will not just enable more reproducibility but also help make the evidence generated by RWD studies to be more useful for healthcare decision making.

A joint taskforce between the International Society for Pharmacoepidemiology (ISPE) and The Professional Society for Health Economics and Outcomes Research (ISPOR), set out to agree on a set of core principles for transparent, comprehensive and rigorous study protocols for RWD studies to improve reproducibility. Differences in the expectations from key stakeholders about the content, structure and format of RWD study protocols is not ideal, potentially causing confusion, omissions or lack of clarity when investigators are communicating key study details. In order to address this issue, the taskforce recently published the HARmonized Protocol Template to Enhance Reproducibility (HARPER) for studies that make secondary use of RWD, evaluate a hypothesis and are intended to inform healthcare decision making [2]. The taskforce found that RWD studies that deal with questions of causal inference are likely the most important for decision making and as such focused their efforts on templates for these types of studies. The taskforce undertook a systematic search of protocol templates, and identified ones from the EMA, ISPE, STaRT-RWE [3] and NEST to act as a starting point. After confirming that the main components of study design and analysis were largely agreed upon across all four templates, the taskforce started with the core sections of the EMA’s PASS template and evaluated guidance and/or structure of the more recently developed protocol templates within each section, incorporating pertinent updates. The draft template was then piloted with five sample case studies with various designs and data sources by committee sub teams in order to refine it. The template layout mainly mirrors the EMA PASS template’s headers. Free text and prompts are aligned with the NESTcc protocol guidance and the ISPE’s GPP section on protocol development, which encourage users to provide context and justification for scientific decisions. Free-text is combined with structured tables from the STaRT-RWE template in the research methodology sections, where users are encouraged to add information about how the study was operationalized. Detailed clinical code lists, algorithms and explanations of data linking or data transformation should be included in appendices.

It requires considerable effort to design and analyze a comparative RWD study well. Because of the number of choices these investigations require, crucial methodological information may be left out of study protocols and reports, and due to their complexity, researchers may make different but equally valid choices regarding study design and analysis. Of course, if HARPER is to be useful it needs to be used, and its impact will be strongly influenced by how much it is adopted by RWD researchers – the planned approach to make the HARPER template freely available should encourage uptake. Although not all studies can easily be described in this type of one-fits-all template, this type of standardization in the ‘toolkit’ of RWD studies is welcome and wide adoption of the template should help to drive better reporting, transparency and reproducibility of RWD studies.

The US FDA has recently released specific guidance on the design and conduct of trials incorporating an external comparator group [4]. This guidance represents a follow-up to other guidances around the use of RWD released last year, and is caveated with the fact that the ‘likelihood of credibly demonstrating the effectiveness of a drug of interest with an external control is low’. The guidance is split into three main sections, covering the design, data and analysis considerations for an externally controlled trial. The FDA have a number of design considerations including: a prespecified protocol before the initiation of the externally controlled trial; following the estimand framework to precisely define the treatment effect; identifying potential confounding factors and making sure they are available in the data source chosen; evaluating the trial eligibility criteria and assessing whether they can be applied in the data source chosen; and appropriate selection of the study index date. Importantly all data sources assessed for use as an external control should be described in detail in the accompanying study protocol, along with a justification for sources that were not used. The guidance includes a key table to evaluate any potential data source which encompasses elements to consider such as missing data, geography and time period. The FDA does not specify specific analytical strategies, but states that identifying and managing sources of confounding is important. Potential statistical approaches can be discussed with the FDA, and sensitivity analyses will be needed to show the robustness of results. Quantitative bias analysis is suggested to assess the impact of confounding.

While perhaps not as detailed as the RWE Framework issued by NICE [5], the guidance issued by the FDA is nevertheless to be welcomed in assisting the generation, usefulness and (where appropriate) acceptance of external controls to enable patient access to new medicines. The pragmatic and non-prescriptive nature of the FDA guidance is constructive and will help the guidance to stay relevant given the rapid pace of statistical and epidemiological methods development in this area. However, the trade-off with this flexibility is that it does leave significant room open for researcher interpretation, which then leaves the door open for debate as to whether investigators followed the best approach, and ultimately to the evidence generated by the study not being accepted by key stakeholders. Given the existence now of the HARPER template it will be interesting to assess how this is used for the preregistration of external controls, and ultimately what data sources and analytical methods lead to acceptance of external comparators by regulatory and HTA stakeholders.