Protocol for data extraction: how real-world data have been used in the National Institute for Health and Care Excellence appraisals of cancer therapy

Abstract

Introduction

Due to the limitations of relying on randomised controlled trials, the potential benefits of real-world data (RWD) in enriching evidence for health technology assessment (HTA) are highlighted. Despite increased interest in RWD, there is limited systematic research investigating how RWD have been used in HTA. The main purpose of this protocol is to extract relevant data from National Institute for Health and Care Excellence (NICE) appraisals in a transparent and reproducible manner in order to determine how NICE has incorporated a broader range of evidence in the appraisal of oncology medicines.

Methods and analysis

The appraisals issued between January 2011 and May 2021 are included following inclusion criteria. The data extraction tool newly developed for this research includes the critical components of economic evaluation. The information is extracted from identified appraisals in accordance with extraction rules. The data extraction tool will be validated by a second researcher independently. The extracted data will be analysed quantitatively to investigate to what extent RWD have been used in appraisals. This is the first protocol to enable data to be extracted comprehensively and systematically in order to review the use of RWD.

Ethics and dissemination

This study is approved by the Ethics Committee of the London School of Hygiene and Tropical Medicine on 14 November 2019 (17315). Results will be published in peer-reviewed journals.

Strengths and limitations of this study.

• This protocol enables data to be extracted in a transparent and systematic manner for the study of how real-world data (RWD) have been used in National Institute for Health and Care Excellence (NICE) appraisals including all the different ways an economic evaluation might use RWD.

• This study facilitates systematic understanding of the use of RWD in NICE appraisals over the last 10 years.

• Since it is focused on cancer, the methods and eventually the findings are to some extent cancer-specific.

• The protocol could be modified to reflect the health technology assessment (HTA) context in different countries although the extraction protocol is not fully applicable to the practice of other HTA bodies as much of the protocol reflects the NICE appraisal process.

• Since data extraction is based on the four main types of appraisal document it is possible, but not likely that some relevant information concerning RWD is missed.

Introduction

In the last few years, interest in real-world data (RWD) has grown in healthcare decision-making.¹ Health Technology Assessment (HTA) refers to the systematic evaluation of clinical-effectiveness and cost-effectiveness of health technology.^{2 3} Health technologies include drugs, medical devices, diagnostics, surgical procedures to mitigate health issues and improve the quality of life.⁴ HTA requires valid and reliable information to evaluate such technoglogies. Randomised controlled trials (RCTs) have mainly provided the information.⁵ However, it is challenging to meet all information needs from RCTs since the new generation of therapies poses several assessment challenges. For example, when treatment options are expanding rapidly, it is increasingly unlikely that there are RCTs featuring all of the relevant comparators. Furthermore, the traditional design of RCTs is possibly less appropriate for new technologies such as those targeting rare genetic mutations where it is harder to recruit patients from the clinically relevant populations.⁶ Moreover, RCTs often have strict inclusion criteria reducing generalisability.⁷ Another barrier to obtaining the information required for HTA from RCTs relates to the extrapolation of survival. Extrapolation is required in order to incorporate the survival data from RCTs in the health economic model.⁸ It is more challenging to identify the most appropriate extrapolation the shorter the duration of the trial. If survival data from RCTs are based on a very limited observation period, the extrapolation of the survival curve is likely to fail to predict the long-term effect.⁹

The potential benefits of RWD in enriching evidence for HTA are highlighted by the limitations of relying on RCTs.¹⁰ This research focuses on the use of RWD in HTA by the National Institute for Health and Care Excellence (NICE). NICE has achieved an international reputation for rigorous development and application of scientific methods to appraise new health technologies to provide its decisions with robust and fair justification.¹¹ More importantly, NICE is noted for the transparency of its processes, responsiveness to change and commitment to using the best available evidence.¹² The structure of the relevant documents facilitates identification of the key information and the documents are available on the NICE website. Therefore, review of these appraisals can provide comprehensive information on the evidence used for decision-making. In April 2020, NICE signalled its intention to integrate broader types of data in developing NICE guidance.¹³ Although it is primarily a statement of intent, it is not a new development in NICE practice since NICE already incorporates a diverse range of published scientific evidence when developing its guidance on health technologies. For example, UK audit data (TA255, 2012), Hospital Episode Statistics (TA559, 2018) and registry data such as the Edinburgh Ovarian Cancer Database (TA598, 2019), Surveillance, Epidemiology and End Result programme (TA562, 2019) have been used in the development of NICE technology appraisal (TA) guidance. While a wide range of data are already used in NICE guidance, there is limited understanding regarding how and where RWD have been used, and in which circumstances RWD are accepted as relevant. Research is required to investigate systematically patterns in the use of RWD and to understand the driving forces behind its use in NICE appraisals.

Several researchers have reviewed practice across HTA bodies^{14 15} or reported the use of RWD in HTA.¹⁶ However, little systematic research has been conducted. Important information is missing such as how they included literatures without selection bias, which parts of the evidence were reviewed, whether they have clearly defined RWD and justified or explained why this definition is relevant and how different HTA systems were compared given their different practices. Roberts et al addressed the potential role of RWD in bridging the evidence gaps.¹⁷ However, they illustrate the use of RWD with a few examples, rather than providing a fuller picture of current practice when using RWD. Bullement et al recently reviewed how RWD informed single TAs of cancer drugs in NICE.¹⁸ Although this study follows a more systematic approach to the review of the use of RWD, a data extraction table was not provided and the authors focused only on how real-world evidence (RWE) influenced the cost-effectiveness analysis, and not on how RWE was used to support or establish the appraisal. Due to limited information presented concerning the review process in this study, it is unclear whether the information presented provides a full picture of the use of RWD. Bullement et al included 113 single-technology appraisals (STAs) issued between April 2011 and October 2018. As interest in RWD is increasing over time, it may miss relevant information from recent years. This extraction protocol is required to help extract the data systematically from appraisals, to increase the reliability of the results of the analysis and to permit a more detailed description of the use of RWD and analysis of factors influencing its use.

A protocol is required to ensure the consistency of data extraction so that the risk of unsystematic data collection is reduced. The main purpose of this protocol is to extract data from NICE appraisals in a transparent and reproducible manner to answer, ‘how has NICE incorporated a broad range of evidence in the appraisal of oncology medicines.’ Without proper justification and operational rules, the data may not be extracted consistently, with a risk of biasing the analysis. The extracted data are expected to be objective and less biased. By consolidating these data, subsequent analysis can provide more robust answers to questions regarding how RWD have been used in NICE TAs. Furthermore, this protocol facilitates the development of a rich dataset which can highlight not just where RWD have been used but also what types of evidence have been used in the HTA process in line with NICE’s interest in incorporating a broad range of evidence. The data can be analysed to answer several research questions including ‘how has RWD been used in NICE appraisals’ and ‘which factors are associated with increased likelihood of the use of RWD’ in depth.

Methods and analysis

NICE appraisal documents are identified following inclusion criteria (figure 1). The information is extracted from identified appraisals in accordance with extraction rules. The detailed extraction rules can be found in online supplemental file 1. The extraction tool includes evidence-related information such as characteristics of the main clinical evidence and the economic evaluation model and other information. Using this tool, information will be collected about which parts of the cost-effectiveness analyses used RWD. Analyses of the intensity of use of RWD and regression analyses are planned. The data analysis is expected to start from January 2022 and be completed by December 2022.

Figure 1.

Inclusion/exclusion criteria. STA, single-technology appraisal.

Definition of RWD

A definition of RWD is clearly required before extracting information about the use of RWD in NICE. RWD are umbrella terms which cover broad categories of data. Although RWD are increasingly addressed in the literature, there is no consensus over the definition. One of the commonly used definitions of RWD is that of the US Food and Drug Administration (FDA).¹⁹ Another widely cited study regarding the definition of RWD is Makady et al.¹ Each definition has relatively large operational flexibility to be used for data extraction. For example, companies sometimes present phase 1 clinical trial as RWD. However, these data hardly provide insights in the discussion of the use of RWD in HTA. Requiring data to meet both definitions can help to reduce the discretionary interpretation of RWD. Hence, this study uses a definition combining a category of the study designs of collecting RWD explored by Makady et al’s study and the FDA’s definition of RWD focusing on routinely collected data. In this research, RWD are defined as the data relating to patient health status and/or the delivery of healthcare routinely collected from non-experimental settings.

Step 1: appraisal selection

The first step of the research identifies the NICE TA guidance which meets the eligibility criteria. TA guidance is publicly available on the NICE website (www.nice.org.uk). This study focuses on four types of appraisal documents, the final scope, the manufacturer’s submission, the evidence review group report and the final appraisal determination. These documents are reviewed to establish whether RWD are used to determine any components of the economic evaluation.

Data sources

This research exclusively includes STAs of oncology medicines. Figure 1 shows the inclusion and exclusion criteria. One aim is to understand how and where RWD have been used in the appraisal process. Therefore, it is necessary that the appraisal process should be identical. However, the STA and multiple technology appraisal (MTA) processes differ substantially. The MTA has different format of appraisal documents to assess several drugs or treatments used for one or more condition. It is challenging to gather the same information in the MTA process as different actors are responsible for producing and reviewing the main pieces of evidence.²⁰ Besides, STAs are the predominant form in practice, 93% of appraisals of oncology. The small number of the MTAs, only 18 oncology appraisals, limits the scope for a comparison of MTAs and STAs in terms of the use of RWD. Therefore, this study focuses on STAs, which assess a single treatment. It also limits analysis to appraisals published between January 2011 and May 2021 in order to have a long enough time period to capture potential changes over time in how RWD have been used but also recognising that STAs from earlier years might be of less interest because enthusiasm for RWD was largely absent. Here, the date when guidance was published refers to the date of issuing the final appraisal determination document (FAD) which can be regarded as an end point of the evidence synthesis process (in the absence of a successful appeal).

Operational separation

Following the inclusion and exclusion criteria, appraisals are identified. Among these appraisals, some TAs have more than one clinical indication or involve combination therapy. It is possible that different evidence was used for the different patient populations in the appraisal. Hence, these appraisals are separated by clinical conditions or treatment lines and reviewed in order to avoid losing information. For example, olaparib for maintenance treatment of relapsed platinum-sensitive ovarian, fallopian tube or peritoneal cancer (NICE TA620) has two separate recommendations for different indications. While a patient who has a BRCA1 or BRCA2 mutation and has had three or more courses of platinum-based chemotherapy is eligible for the treatment, a patient who has a BRCA1 or BRCA2 mutation and has had two courses of platinum-based chemotherapy is able to use the treatment within Cancer Drug Fund. Consequently, these indications are included separately in the analysis.

Step 2: data extraction

A detailed protocol is developed to guide the extraction of essential data for each appraisal in order to investigate the use of RWD in NICE TAs in a systematic and reproducible manner. The protocol is designed to extract information from both the manufacturer’s submission (manufacturer’s cost-effectiveness analysis) and the final appraisal document (the model preferred by the committee) regarding where RWD were used, and to determine the extent to which the committee supported the use of RWD in these appraisals and understand what factors are associated with supporting or not supporting their use. Figure 2 shows the structure of the data extraction template. In summary, the extraction tool consists of three parts—general information, explanatory variables and outcome variables. The outcome of interest being the use of RWD. The outcome variables record use or non-use of RWD for different elements of the economic evaluation. The information in the base-case analysis and sensitivity analyses will be separately extracted. The tool includes all important elements of an economic evaluation. The study will analyse the data to investigate patterns in the use of RWD in NICE appraisals, and the association between several factors and the use of RWD. Explanatory variables are suggested based on the hypotheses presented under step 4: data analysis. All items in the extraction template and how to code them are described in the glossary (online supplemental file 1). To convey the type of information to be extracted, some examples from a preparatory review are presented in the glossary.

Figure 2.

The framework for data extraction. *Published date of MS: the date when it was submitted by the manufacturer, which is stated on manufacturer submission document. ERG, evidence review group; FAD, final appraisal determination document; RCTs, randomised controlled trials.

Parametric and non-parametric use

This protocol distinguishes two categories of outcome variable, parametric and non-parametric use of RWD. Parametric use of RWD is the use of such data to define the numerical value of a specific variable in the economic evaluation, whereas non-parametric use is where data are used to develop the model structure or to determine the scope of the evaluation. For example, when RWD are used to estimate survival, this will be counted as parametric use with respect to clinical outcomes (overall survival/progression free survival). Parametric use is reviewed and recorded for the intervention and comparators separately as different data could be used in the cost-effectiveness analysis. An example of non-parametric use of RWD can be found in the appraisal of palbociclib for previously untreated, hormone receptor-positive, HER2-negative, locally advanced or metastatic breast cancer (NICE TA495). In this appraisal, the company used information from a study of medical records to determine the subsequent treatments to be assumed in the economic model. This case is regarded as non-parametric use since RWD were used to specify the treatment sequence but not the quantity and cost of subsequent treatment.

Parametric and non-parametric use of RWD and the different categories shown in figure 2 facilitate more consistent data extraction by highlighting the different ways RWD might be used, and provide greater flexibility when testing hypotheses regarding the use of RWD, and the exploration of ways to measure the intensity of use of RWD.

Coding

A key issue with respect to improving the reliability of data extraction is how many distinct variables to identify and how finely to divide the potential responses to these variables. One option, in order not to lose information, is to have many distinct variables with binary responses. Another option is to merge many variables but have multilevel responses. This coding system has advantages which include avoiding information loss, and also grouping together ‘similar’ information used during appraisals to establish patterns of the use of RWD. This is closely linked to the reason for not using multiple responses in the coding. The template takes an ‘including all and combining trivia’ approach. It helps to include all relevant variables where RWD data can potentially be used, but also to list variables more concisely by merging unnecessarily trivial variables so that the outcome of the extraction can be concretely analysed. Based on two categories, the parametric and non-parametric use of RWD, the areas where data are likely to be used are carefully searched. As a backbone of the extraction structure, distinguishing two categories helped to search each component systematically. Under parametric use, clinical effectiveness, health utility and cost and healthcare resource use were thoroughly reviewed. After sorting variables, they were aggregated if the information is minor and can be categorised into one variable. The area where aggregation is mostly required is resource use. In order to reflect routine clinical practice, especially the cost part has naturally incorporated RWD into the analysis. Estimates of unit costs are usually informed by the National Health Service (NHS) reference costs (a form of RWD) and thus in order to provide a more sensitive measure of the use of RWD the extraction template focuses on resource use (with respect to cost). However, the measures of resource use are not fully differentiated. Different health technologies include different elements of resource use reflecting their characteristics. Distinguishing all resource use is not an accurate way to understand why and how RWD were used. Although all individual resource uses are not identified, some resource uses, which can be critical in appraisals are differentiated. Variables such as volume of treatment or dose adjustment have potentially critical impacts on the result of economic evaluation. Therefore, these variables are separated from overall resource use.

Step 3: validation of data extraction tool

The data extraction tool will be validated by a second researcher independently repeating the data extraction for a random sample of appraisals (20% of all appraisals). This validation is required to check the replicability of the data extraction and the clarity of the extraction tool. Any disagreements between the researchers will be resolved by discussion. Peer discussion following the validation process is important not only to check the clarity of this protocol but also to investigate any deviations caused by unclear information. It will help pinpoint where a higher degree of subjectivity may arise in the data extraction.

Step 4: data analysis

The extracted data will be analysed quantitatively in two different ways. First, counts and proportions will summarise where and how RWD have been used in appraisals. This will be supplemented by an analysis of the intensity of use of RWD in order to explore changes in the pattern of use of RWD over time and differences with respect to cancer type. In addition to descriptive statistics, the association between years and the intensity of use of RWD will be examined. Second, a regression analysis will be performed to investigate which factors are associated with the greater use of RWD in a company’s submission. As part of the protocol development, some appraisal documents were reviewed to identify factors potentially associated with the use of RWD. Five factors were identified and formulated into hypotheses about increased use of RWD (figure 3).

Figure 3.

Hypotheses about increased use of real-world data (RWD). NICE, National Institute for Health and Care Excellence; TA, technology appraisal.

Methodological issues

The design of this data extraction protocol, in which information is reliably and repeatedly extracted across appraisals, will allow us to review evidence for the use of RWD more systematically than could be obtained from conducting several case studies. However, several methodological challenges can be anticipated. This section addresses these challenges and how they might be mitigated.

Issue 1: unclearly stated information

Overall, NICE appraisals clearly describe the data used in the evidence synthesis. However, sometimes the search process may not be well-documented and the precise source of information may not be clear. Systematic literature reviews are carried out to identify all relevant evidence in appraisals. Clinical effectiveness evidence is carefully examined and described in detail, with clear reasons for the inclusion and exclusion of studies. On the other hand, the systematic search for resource use and cost information usually enumerates miscellaneous studies with bibliographic information and a summary, but the critical review of minor components of health cost is sometimes missing. While manufacturers provide the result of their assessments, some manufacturers’ submissions do not clearly state whether a particular study was used to determine an element of resource use making up the health state costs. However, it appears to be rare for there not be an explicit statement regarding the evidence used, mostly with respect to resource use.

Issue 2: level of aggregation

An important question is the most appropriate level of aggregation. This is best illustrated with respect to healthcare costs. It would be possible to have a variable indicating use or non-use of RWD for every single element of cost (distinguishing general practice (GP) visits, frequency of hospitalisation, etc). At the opposite extreme, there could be a single cost variable which indicated whether RWD were used for any element of cost. The more aggregated the measure the greater the loss of information, but some elements of cost are much more important than others and the potential analyses of the use of RWD will multiply greatly if there is no attempt at aggregation. The current protocol tries to balance the advantages and disadvantages of different levels of aggregation by combining several elements into a health state cost variable but distinguishing other important components of cost, such as volume of treatment, dose adjustment and resource use for adverse events.

Issue 3: no consensus on the definition of RWD

This research uses a definition of RWD merging definitions from the FDA and Makady et al. The distinctive part of the definition used in this research is ‘routinely collected’ data from a ‘non-experimental study’. Although this definition provides a specific and clear definition for this research, there is no consensus on the best definition of RWD. Even the same definition can be interpreted in different ways. For example, some researchers interpret that routinely collected in the FDA definition is ‘collected in routine care’ whereas other interpret it as ‘how frequently data are collected.’ It is likely that other definitions of RWD are preferred by other researchers and the data extracted will be influenced by the definition of RWD chosen. While the use of multiple definitions of RWD was considered, it would create practical problems such as multiplying the number of potential analyses and making data extraction take longer. Although the chosen definition can be questioned by other researchers who have different views, the various definitions overlap considerably. It is thus unlikely there will be a marked divergence in the data extracted when using the different definitions.

Design to mitigate methodological issues

Several operational rules have been designed to minimise bias likely to come from the methodological issues encountered in the data extraction. First, ‘not clear’ is recorded separately in order to provide a more accurate description of the use of RWD. However, for purposes of data analysis, we anticipate treating these instance as ‘no RWD’ since the code not clear cannot be independently analysed. In addition, having a not clear category in analysis is unlikely to improve data quality since we anticipate that this problem will arise in very few appraisals. Also, information which is not clearly recorded in the appraisal documents is usually not important information with respect to the evidence synthesis. The approach (extracting all relevant information which can provide meaningful data for analyses) is also closely linked to the reason for using binary code for analysis in this research. Decomposing levels of codes into several small parts can facilitate data extraction. However, it is more likely to increase the complexity since trivial information is individually recorded. The extracted trivial data should be interpreted based on another operational rule. It is subject to increased error, particularly when testing hypotheses. For these reasons, the benefit of using multilevel codes does not outweigh the benefit of binary codes while separation is much more time consuming. Instead of adapting multilevel codes, this study will adopt an alternative approach, an intensity analysis which helps to identify important differences within the diverse patterns of use of RWD. When looking at the pattern of use of RWD, the intensity of use will be analysed. Simply counting the number of times RWD are used is not an accurate way to understand why and how RWD were used. Alternatively, this study focuses on variables which are potentially important determinants of cost-effectiveness in appraisal. Variables such as survival outcome, volume of treatment and choice of comparators are more likely to influence estimated cost-effectiveness. Especially, the survival outcome is the most important information in both clinical and cost-effectiveness as well as one of the controversial areas where to use RWD. The intensity analysis is a framework to show whether RWD are used in these components alongside the quantity of the use of RWD. It can offer more benefits in deeper understanding of the use of RWD than counting all miscellaneous uses of RWD.

Strengths and limitations

To the best of the authors’ knowledge, this is the first study protocol to investigate to what extent RWD have been used in NICE appraisals. It allows the practice of extracting information to be reproducible, systematic and transparent. Strengthening the reproducibility and transparency of data extraction can maximise understanding of the use of RWD by allowing more accurate interpretation and use of findings. This protocol could be relevant to researchers or HTA agencies who aim to understand how various data resources are used in HTA in England. Analysis of data generated using this protocol can provide a detailed picture of the use of RWD in NICE appraisals over 10 years. Moreover, the study findings could add value to NICE’s ongoing work to broaden the evidence used in appraisals.

The protocol has the limitation that it has been developed to study the use of RWD in NICE appraisals of oncology drugs. Consequently, the data extraction protocol may not be fully applicable to appraisals in other disease areas or to the different practice of other HTA bodies. Since the documentation is significantly different depending on each country’s context, it may not be feasible to extract the same information as in the English context. However, many of the distinctions are of wider application, for example, parametric versus non-parametric use of RWD, and the taxonomy of where in an economic evaluation it might be relevant to look for use of RWD. Also, the hypotheses are potentially of wider application. The results are going to be specific to NICE but otherwise the structure of this research has wider application. Although not fully transferrable, this protocol can be modified for use in other HTA contexts. Lastly, this protocol focuses on four main documents. Relevant RWD may arise at the clarification or technical engagement stage. It is possible there is some information regarding use of RWD that is not reported in any of the four main documents. However, only a small number of such cases are anticipated. If RWD are critically used in a revised model and the committee thinks it is an important change, this evidence is likely to be addressed in the FAD.

Ethics and dissemination

This study was approved by the Ethics Committee of the London School of Hygiene and Tropical Medicine on 14 November 2019 (17315). Results will be published in peer-reviewed journals.

Ethics statements

Patient consent for publication

Not applicable.