Use of real-world evidence in economic assessments of pharmaceuticals in the United States

Abstract

BACKGROUND:

Despite the increasing interest in expanding the use of real-world evidence (RWE) in economic assessments of pharmaceuticals, decision makers face uncertainty about how RWE should be used.

OBJECTIVE:

To assess the use of RWE in economic assessments of drugs by the Institute for Clinical and Economic Review (ICER).

METHODS:

We reviewed cost-effectiveness and budget impact analyses in final evidence reports of pharmaceuticals published by ICER. We calculated the total number of RWE uses and the proportion of model inputs informed by RWE per report. We classified model inputs into 15 categories based on their attributes and then examined what category each RWE informed to classify the reason for RWE use. Finally, we characterized RWE by study design, data source, and sponsor type.

RESULTS:

We identified 33 reports, all of which used RWE; the mean RWE use per report was 12 (range = 4-26). The average proportion of model inputs informed by RWE per report was 32.7%, but this proportion had a wide range (range = 4.1%-76.9%). RWE was most commonly used for disease progression inputs (28.7%) and health care resource utilization and costs (21.1%), but was rarely used for drug-specific clinical outcomes such as effectiveness (1.5%), adverse drug event rates (0.5%), and discontinuation rates (1.2%). The most frequently used study design was a retrospective cohort (56.6%), and the most frequently used data source was registry data (41.4%). About a third (30.2%) of RWE was industry-sponsored.

CONCLUSIONS:

RWE has been commonly used to inform pharmaceutical value assessments conducted by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness.

What is already known about this subject

• A recent FDA guidance enables the use of off-label evidence in communicating health economic information to payers, which is likely to increase the use of real-world evidence (RWE) in economic analyses.

• A majority of U.S. payers recognize the value of RWE in informing health care decision making.

• Recent studies about the use of RWE in health technology assessment in other countries found that RWE is being used in most of the economic evaluations of drugs, but there is a significant variation across countries.

What this study adds

• About a third of model inputs from Institute for Clinical and Economic Review economic assessment models were informed by RWE on average, and this proportion showed an increasing trend over time.

• RWE is commonly used to inform mortality, disease progression rates, and healthcare costs but has a relatively minor role in informing drug effectiveness and safety.

• The most common study design was a retrospective cohort study, and the most common data source was registry data.

Real-world evidence (RWE) is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of data routinely collected from sources regarding patient health status and/or the delivery of health care.¹ There has been a growing interest in incorporating RWE to assess the economic value of drugs because of its ability to reflect real-world outcomes and limited availability of evidence from randomized controlled trials (RCTs) (e.g., long-term effects and economic outcomes).² Furthermore, the U.S. Food and Drug Administration (FDA) recently released guidance on how drug companies may communicate health care economic information to payers and formulary committees that enable the use of off-label evidence related to an approved indication, which is likely to increase the use of RWE in economic analyses.^3-6

A recent study suggested that a majority of U.S. payers recognize the value of RWE in informing health care decision making. They believe that health care decisions should be informed by RWE in addition to RCTs and intend to use more evidence derived from real-world settings once they have a better understanding of the use of RWE.⁷ Furthermore, it has been recognized in guidelines for health technology assessment (HTA) that RWE can be used to improve evidence synthesis and integration by supplementing the evidence from RCTs.^8-10 Therefore, RWE, while not a replacement for RCTs, can provide the best available evidence for an HTA that fits for purpose when it is difficult to implement a well-controlled clinical trial.¹¹

Although there were a few studies that examined how RWE has been incorporated in economic evaluation of drugs, primarily in Europe, it has not been systematically assessed in the United States.^12,13 Considering the variations in policies for RWE use across countries, quantitative examination of RWE use in the United States may give payers an objective view of the possible roles of RWE in economic assessments of drugs.¹⁴ This is especially important considering that the use of RWE varies from payer to payer and is generally limited because of the lack of understanding of how to incorporate RWE into the value assessments of drugs.⁷ Also, the previous study in European countries was focused on a specific type of disease and thus there is not enough information about the use of RWE across a broad range of diseases.¹³ Furthermore, the understanding of the current roles of RWE in the United States can help identify the gap between what has been called for and what has been done in terms of RWE use.

The Institute for Clinical and Economic Review (ICER) is a nonprofit independent research organization that evaluates the clinical and economic values of medical interventions. ICER reports have been increasingly used by U.S. payers in their decision-making processes, such as pharmacy and therapeutics review and pricing negotiations.^15-18 The goal of this study was to assess the use of RWE in economic assessment of drugs by ICER to understand the recent volumes, trends, reasons, and characteristics of RWE use in the United States.

Methods

STUDY DESIGN

We conducted a retrospective analysis of the use of RWE in the long-term cost-effectiveness analysis (CEA) and potential budget impact analysis (BIA) sections of final evidence reports published by ICER. We used the ICER database because ICER is one of the most influential HTA agencies in payer decision making in the United States and makes their evidence reports publicly available.^15-18 ICER has defined a clear process for performing an HTA that includes topic selection, scoping, clinical evidence review and economic modeling, report generation, public meeting, and finalization of materials.¹⁹ In general, all final evidence reports include CEA and BIA sections, and details about CEA and BIA models, including structure, assumptions, and model inputs, are provided in their methods section. ICER contracts with academic units to conduct the majority, if not all, of the economic models.

We used the FDA definition of RWE as the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from the analysis of real-world data (RWD), and RWD is defined as data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.¹ This definition is consistent with ICER’s definition: RWE is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD, and RWD are data relating to patient health status and/or the delivery of health care collected either prospectively or retrospectively from observations of routine clinical practice.¹⁶

INCLUSION AND EXCLUSION CRITERIA

Final evidence reports that assessed at least 1 pharmaceutical intervention and published between January 2014 and June 2019 were included. We decided to focus on this period because a 5-year period is enough to see the time trend of the use of RWE, the vast majority of ICER reports were published in this period, and 2014 is approximately when ICER started to conduct economic analyses consistently. In case an evidence report has been updated, we only included the most recent version. Reports were identified using the Assessments page on the ICER website (https://icer.org/explore-our-research/assessments/).

DATA EXTRACTION

We developed a data extraction form (DEF) based on questions adapted from the study by Makady et al. (2018) to abstract information about each report and RWE use in a standardized way (Supplement 1, available in online article).¹³ The DEF consisted of 4 parts, each with different types of information abstracted.

• In part 1, we collected general information on each ICER report, such as indication and drugs assessed.

• In part 2, for each use of RWE identified in the CEA and BIA models, we examined for which model inputs each RWE was used for. To do this, we first identified model inputs and variables on outcomes of a drug or patient characteristics (i.e., clinical, utility, economic inputs). The description and categorization of model inputs along with sources of their values are available in the model inputs section of the methods section. We searched for, identified, and read the sources of each model input to determine whether it meets the definition of RWE used in our study. Based on the FDA’s definition, we considered evidence related to the usage (e.g., health care utilization and costs) and benefits or risks of a medical product (e.g., effectiveness and utilities) as RWE if it was derived from data that meet the FDA’s definition of RWD.

• In part 3, we abstracted characteristics of each RWE, such as study design, data source, and sponsorships.

• In part 4, we determined whether the value of each model input was informed by RWE only, non-RWE only, or both RWE and non-RWE. The included ICER reports were divided among 2 independent reviewers, and the abstraction results were cross-checked (see Supplement 2, available in online article, for the consensus on the data extraction).

OUTCOMES

The extracted data were used to examine (a) the volume of RWE uses and its time trend; (b) reasons for RWE use; and (c) characteristics of RWE including study design, data source, whether or not it was preregistered, funding support for the study, and the number of RWE uses by calendar year of FDA approval of drugs.

ANALYSIS

The volume of RWE use was measured by the absolute number of RWE uses per report. If RWE was used multiple times (e.g., for different model inputs), it was counted as many times as it was used for different inputs. To account for potential bias in the absolute number of RWE uses due to model-specific characteristics, such as the number of drugs and health states included, we measured the relative number of RWE uses per report by calculating the proportion of all model inputs informed by RWE per report. The trend of this measure was examined over time.

Reasons for RWE use were described using the frequency distribution of model input types for which RWE was used. Model inputs were classified into 15 types based on their attributes (e.g., mortality or disease progression rate, health care costs, and adverse drug events) for ease of categorization.

Study design and data source of RWE were characterized using frequency distribution. We further calculated the frequency distribution for each model input to identify the pattern of study designs and data sources by inputs. In terms of preregistration and study funding, the proportion of RWE preregistered in any study registry (e.g., clinicaltrials.gov) and the proportion of industry-sponsored RWE was calculated. Lastly, we calculated the number of RWE uses by the calendar year of FDA approval of drugs to assess the trend of RWE uses by the post-market periods of drugs.

FDA approval year of each of the drugs assessed in all reports was obtained from Orange Book,²¹ Purple Book²² or package inserts, then the number of RWE uses was counted for each group of drugs approved in the same year. Only RWE that studied specific drugs was included in this analysis. In other words, population-level statistics (e.g., U.S. vital statistics and death certificate data) or nondrug-specific RWE (e.g., transition probabilities, utility, and health care resource utilization among general patients) were not included in this analysis. RWE that studied multiple drugs was counted multiple times for each of the drugs. In addition, we normalized the absolute number of RWE by FDA approval year by accounting for the total number of FDA-approved drugs included in ICER reports in a given year.

Results

ICER REPORTS

Thirty-three reports met the inclusion criteria and were included in the review (Figure 1 and Supplementary Table 1, available in online article). The reports included a total of 123 pharmaceutical interventions and comparators indicated for 29 diseases, including 5 cancers (e.g., ovarian cancer and prostate cancer), 9 rare diseases (e.g., cystic fibrosis and tardive dyskinesia), and 15 non-rare and non-cancer diseases (e.g., atopic dermatitis and osteoporosis). All reports included a CEA, but 2 reports did not include a BIA (Supplementary Table 2, available in online article).

FIGURE 1.

Flow Diagram

VOLUME OF RWE USE

We identified 407 RWE uses in 33 ICER reports, all of which used at least 1 RWE source. The mean number of RWE uses per report overall was 12.3 (range = 4-26), 10.7 (range = 2-26) for CEAs, and 1.6 (range = 0-7) unique to BIAs. Of note, BIA models in ICER reports are built upon CEA models and thus some of the RWE included in a CEA might have been shared by the BIA.

On average, 32.7% of all model inputs were derived from RWE (Figure 2 and Supplementary Table 3, available in online article). However, the proportion ranged from 4.1%-76.9%, showing a large variance. The mean proportion of inputs informed by RWE per report had an overall increasing trend over the past 5 years, although it was high in 2015 (Supplementary Figure 1, available in online article).

FIGURE 2.

The Number of Model Inputs and Proportion Informed by RWE

REASONS FOR RWE USE

The most common reasons for RWE use were to inform mortality or disease progression rate (117/407, 28.7%) and health care costs (86/407, 21.1%; Supplementary Figure 2, available in online article). Drug-specific clinical inputs were rarely informed by RWE. Real-world evidence used to inform drug-specific effectiveness, discontinuation rates, and adverse event rates accounted for 1.5% (6/407), 1.2% (5/407), and 0.5% (2/407), respectively, of all uses of RWE.

STUDY DESIGN, DATA SOURCES, AND OTHER CHARACTERISTICS OF RWE

The most frequently used study design was a retrospective cohort, which accounted for more than 50% (207/407) of all uses of RWE, followed by a prospective cohort (69/407, 17.0%; Supplementary Figure 3a, available in online article).

Registry data were the most frequent source of data (163/407), accounting for 40% of all uses of RWE, followed by administrative claims data (74/407, 18.2%) and patient survey/patient diary data (74/407, 18.2%; Supplementary Figure 3b, available in online article).

Retrospective studies accounted for more than half of RWE uses for each nondrug-specific clinical input, such as disease progression and mortality rates (64/117, 54.7%), patient characteristics (16/23, 69.6%), and incidence (5/10, 50%; Table 1). Drug-specific clinical inputs, which included effectiveness, drug-specific discontinuation rates, and adverse drug event rates, were also informed both by prospective (4/13, 30.8%) and retrospective studies (6/13, 46.2%). Retrospective studies were generally used for health care costs (68/86, 79.1%) and treatment pattern or market share (9/13, 69.2%).

TABLE 1.

Study Designs and Data Sources by Model Inputs

Model Inputs	Total Number of RWE Uses	Study Designa	Data Sourcea
Nondrug-specific clinical inputs
Disease progression/mortality rates	117	Retrospective (64)b	Registry data (60)
Prevalence	44	Retrospective (17) Prospective (15)	Registry data (17) Patient survey data (14)
Patient characteristics	23	Retrospective (16)	Registry data (14)
Other clinical inputs	14	Prospective (5) Retrospective (5)	Registry data (7) Patient survey data (4)
Incidence	10	Retrospective (5) Prospective (4)	Registry data (6)
Nondrug-specific discontinuation rate	2	Prospective (1) Retrospective (1)	Registry data (1) Other observational data (1)
Drug-specific clinical inputs
Effectiveness	6	Prospective (3) Retrospective (3)	Electronic health records (2) Registry data (2)
Drug-specific discontinuation rates	5	Retrospective (2) Unidentifiable (2)	Administrative claims data (3) Registry data (2)
Adverse drug event rates	2	Prospective (1) Retrospective (1)	Electronic health records (1) Registry data (1)
Utility inputs
Nondrug-specific utility	38	Utility study (33)	Patient survey data (27)
Drug-specific utility	1	Utility study (1)	Registry data (1)
Economic inputs
Health care costs	86	Retrospective (68)	Administrative claims data (49)
Non-health care costs	27	Cross-sectional for survey (16)	Patient survey data (15)
Treatment patterns/market share	13	Retrospective (9)	Administrative claims data (4)
Assumptions
RWE used to support assumptions	19	Retrospective (8) Prospective (6)	Registry data (9)

^aFor each of the model inputs, study designs or data sources that make up at least 30% are included.

^bNumbers in parentheses indicate the number of RWE uses with the corresponding study design or data source.

RWE = real-world evidence.

Registry data were the main sources of nondrug-specific clinical inputs, accounting for 50% (105/210). Drug-specific clinical inputs, on the other hand, consisted of electronic health records (3/13, 23.1%) and administrative claims data (3/13, 23.1%) as well as registry data (5/13, 38.5%). Administrative claims data were mainly used for health care costs (49/86, 60.0%) and treatment pattern/market share inputs (4/13, 30.8%), whereas patient survey data were the main sources of non-health care costs (15/27, 55.6%) and utility inputs (27/38, 81.8%).

Among 407 uses of RWE, only 5 (1.2%) studies were preregistered. One was a meta-analysis that was registered in the Prospective Register of Systematic Review system. The other 4 studies were prospective observational studies that were preregistered in ClinicalTrials.gov. Approximately a third of RWE studies (123/407, 30.3%) were sponsored by industry and another 60.7% (247/407) by nonindustry.

A total of 86 RWE uses were included when examining the number of RWE uses by calendar year of FDA approval of drugs. More than half (47/86, 54.7%) of RWE studied drugs that were approved between 1996 and 2005. RWE that studied drugs approved between 2006 and 2015 and after 2016 made up 20.9% (18/86) and 0% (0/86), respectively (Supplementary Table 4, available in online article). When the number of RWE uses was normalized by the total number of drugs approved in a given year, drugs that were approved by the FDA earlier in the calendar year had a greater number of RWE uses. The numbers of RWE uses per drug were around 1.8, 1.6, and less than 1 for drugs approved before 1990, between 1996 and 2005, and after 2006, respectively.

Discussion

We evaluated the use of RWE in the United States in economic analyses of pharmaceuticals by reviewing ICER Final Evidence Reports between 2014 and 2019. Although all economic models had at least 1 instance of RWE use, there was a wide variation in the extent to which RWE was used to inform model inputs. RWE was commonly used to inform disease progression and health care costs, but it was rarely used for drug effectiveness. The most common study design was a retrospective cohort, and the most common data source was registry data, respectively.

We found RWE has a minor role in informing drug-specific clinical outcomes such as effectiveness, safety, and drug discontinuation in ICER’s CEAs; these estimates were primarily derived from RCTs only. A study showed that payers are hesitant to use RWE because of reservations regarding quality of RWE and lack of ability to evaluate RWE.⁷ This indicates the lack of awareness and underutilization of quality assessment tools for RWE.^22-26 Another reason for the underutilization is due to the time required to generate RWE.⁷ ICER reports are usually published near the time drugs are approved, which does not provide much time for RWE to be generated. Therefore, there are limited resources available for economic assessments that inform initial health care decision making. The lack of RWE for newer drugs often leads to a situation where the RWE for older drugs is extrapolated to a newer drug. ICER often updates the assessments and incorporates recent RWE that was not available originally, which sometimes results in a significant difference in conclusions. In the psoriasis report, for example, discontinuation estimates that were originally informed by extrapolating RWE of older drugs to newer drugs have been updated as RWE for newer drugs became available later.²⁷ Our finding may suggest the need of reassessment of a drug and adjustment of health care decisions based on updated evidence after the launch.²⁸

The characteristics of RWE used for drug-specific clinical outcomes did not appear to be different than RWE used for nondrug-specific inputs. To be more specific, there was not a noticeable difference in sample sizes, data sources, study designs, or whether or not it was preregistered. These findings may be a result of the lack of guidance on types of RWE used for regulatory decisions (i.e., the FDA does not endorse any one type of data or study design).²⁹ Our findings suggest the need for approaches to assessing the appropriateness of RWE use that differ based on circumstances (e.g., data availability) or purposes (e.g., outcomes).³⁰

We found few preregistered RWE studies. Our finding is in line with previous findings that, although much of the ethical and scientific rationale for registering clinical trials also applies to observational studies, only about 17% of registered studies at ClinicalTrials.gov are observational studies.³¹ Preregistration can benefit the quality of RWE by improving reliability and transparency. As the need for credibility in RWE increases, there have been several efforts to facilitate the registration of observational studies, such as a certification process for payers and the RWE Transparency Initiative to establish a culture of RWE transparency that was launched by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR).^32,33 The initiative’s steering committee, in its recent white paper, outlined an approach designed to facilitate the registration of observational studies, particularly those evaluating effectiveness or safety. Such efforts to enhance the trustworthiness of RWE may further facilitate using drug-specific clinical outcomes estimated based on RWE.

The increasing proportion of model inputs was informed by RWE over time, which is in part explained by the evolution of ICER’s value framework. While the essential valuation criteria (i.e., long-term value for money and short-term affordability) did not change, there have been updates in the economic model framework that could have possibly affected the use of RWE.³⁴ The most notable change was the use of a societal perspective for CEAs. In 2017, ICER announced that the base-case incremental cost-effectiveness ratio will continue to be calculated from the health system perspective, but a scenario analysis from a societal perspective will be performed. Since this requires models to have non-health care sector benefits and costs (e.g., productivity costs) that are hard to be informed by RCTs, this change might have increased the number of RWE uses.

Our findings are for the most part consistent with previous studies. Makady et al. reviewed the use of RWE in European countries and found that RWE is used in the majority of economic assessments, and many of them were used to inform costs or resource use.¹³ They also found that registry data and claims data were most frequently used, which was in line with findings from another study by Parody-Rúa et al. (2020), which was a systematic review of economic evaluation studies that used RWD for both cost and clinical inputs.³⁵ Parody-Rúa et al. also found that only about 5% of the studies that used RWE were published before 2011, suggesting the recent increase in RWE use.

Furthermore, our finding that RWE is rarely used for treatment effectiveness is consistent with existing policies on RWD use. A study that reviewed policies for RWD use in HTAs stated that it is recommended by most agencies using RWE under specific circumstances—for instance, in the absence of RCT evidence.¹⁴ ICER, however, recently announced that it will start to generate decision-grade RWE with partners to improve understanding of the real-world performance of treatments.³⁶ This is in line with the most recent ICER value assessment framework where it was stated that ICER seeks to use RWE to inform long-term safety or durability of a medication’s effect.¹⁰ This movement will be facilitated by assessing the internal and external validity based on the outline presented in the ISPOR RWD Task Force report.³⁷ ICER also announced that de novo RWE generation will be facilitated to inform clinical and economic outcomes that are not included in RCTs. As such, the increasing use of RWE for drug effectiveness is expected in the United States, along with RWE use for other purposes.

Although most of our findings coincide well with existing studies and policies, there were a few differences from findings in Europe. First, Makady et al. found that 12% of 25 cost-effectiveness models they reviewed did not use any RWE.¹³ Our evaluation did not find a comparable frequency of RWE use, likely because Makady et al. only reviewed HTA reports on melanoma drugs while we reviewed all types of drugs. Since melanoma trials generally follow patients for sufficient time to estimate survival probabilities, less RWE is likely needed for disease progression rates compared to other types of diseases. Moreover, Makady et al. found that RWE is most frequently used for extrapolating long-term effectiveness and costs, while we found that RWE is frequently used for mortality and disease progression rates but rarely used for informing drug effectiveness. There are 2 potential reasons for this difference: (1) many of the European melanoma reports used the same registry data to estimate long-term overall survival beyond the trial, and (2) the definition of effectiveness in Makady et al. appears to be broader than ours. We did not classify RWE on mortality or disease progression rates that are not on specific drugs into the effectiveness category to make a distinction between evidence on natural disease progression and drug-specific effectiveness.

LIMITATIONS

This study has some limitations to consider. Our study may not be representative of the RWE use in all U.S. HTA agencies, since we only focused on ICER reports. There is considerable variation in the absolute number of RWE uses per report, which is due to various factors such as the number of drugs and health states assessed in the economic models and how much RWE is available for the conditions or drugs. Therefore, the numbers should be interpreted with caution. To avoid this bias, the relative number of RWE uses that is less likely to be affected by the aforementioned factors was measured. It is important to note, however, that the absolute and relative numbers of RWE uses alone are not likely to inform whether or not RWE is used sufficiently, since the optimal use of RWE is not simply determined by the number of uses.

Although there was an increasing trend in the relative number of RWE uses over time, the number was exceptionally high in 2015 due to a higher frequency of RWE use in 2 of the 4 ICER publications in 2015: the reports for hepatitis C and high cholesterol. Both reports were class reviews and thus there might have been more RWE available for older drugs. Also, the report for hepatitis C was an updated version, which is likely to include more RWE than the original version. Considering that the number of RWE uses can vary by the types of reports (e.g., class reviews and updated report), future work is needed to adjust for this factor.

Also, some people may find it unclear whether evidence on health care costs should be considered as RWE or the examples of RWD use, because the FDA definition states that RWE is “the clinical evidence.” We counted it as RWE if the evidence was derived from the analysis of data of usage or delivery of health care (e.g., claims and billing activities), assuming evidence derived from any RWD is RWE. Our interpretation was also based on the value framework by ICER where RWE is considered as a source to inform nonclinical outcomes. By using a broad definition of RWE, however, we might have overemphasized RWE use.

Finally, we did not take into account the publication year of ICER reports when calculating the number of RWE uses by the FDA approval year of drugs. Since the FDA approval year does not necessarily correspond to how long a drug had been in the market before its ICER report was published, the results should be interpreted with caution. That said, since all ICER reports included in this study were published within the 5-year time window, we expect that there would be no considerable difference in the trend.

Conclusions

RWE has been used for various purposes in the economic assessment of pharmaceutical interventions in the United States by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness. These findings can help identify the gap between what has been called for and what has been done in the use of RWE, which can give directions to the better use of RWE in HTAs.