The real-world impact of National Institute for Health and Care Excellence's real-world evidence framework

Background

The international health and care landscape is undergoing sweeping changes, involving: the rapid pace of innovation in technologies [1,2]; improved quality and access to routine data sources [3,4]; and strong economic pressures driving the need to optimize resources [5]. These trends were accelerated by the global COVID-19 pandemic.

Post pandemic, real-world data continues to deliver evidence with richer outcomes, more timely conclusions and populations more representative of those to which care will be delivered, including patients traditionally underrepresented in randomized controlled trial environments, or with unequal access to care. Recognizing the still substantial untapped potential, the National Institute for Health and Care Excellence (NICE) is looking to real-world data to support: quicker, more proportionate appraisals [6]; dynamic ‘living’ guidance with responsive updates [7]; to monitor and influence the uptake of its guidance [8]; and to manage uncertainty post evaluation for earlier access to technologies [9,10].

Broadly, the use of real-world evidence is hampered for two main reasons: the trustworthiness of real-world data and the trustworthiness of methods used to analyze that data. Non-systematic identification of data sources, unclear curation processes, and complex and opaque study designs lead to general concerns over data suitability for the research question of interest and risk of bias, particularly for studies of comparative intervention effects. Meanwhile, for those developing evidence, it may also be unclear what NICE considers to be a reasonable threshold of quality to support decision making [11].

NICE's Real-world evidence framework was therefore developed to provide a steer for ‘what good looks like’ in the design and reporting of real-world evidence and its use, encouraging both a more rigorous standard of evidence and more consistent appraisal of that evidence [11]. A separate document to NICE's methods and guidelines manuals, it provides more detailed advice on the identification of suitable data, and the conduct and reporting of real-world studies, without being overly prescriptive.

The framework has a strong emphasis on supporting implementability with links to numerous case studies displaying the range of use cases in supporting NICE guidance. Also included are longer form vignettes exploring relevant academic work. Tools and resources are referenced or have been developed if not available externally. For example, the Data Suitability Assessment Tool (DataSAT) within the framework provides a reporting template to help developers report all the key elements of data suitability to sufficiently inform NICE committees.

The RWE framework was welcomed by stakeholders, and now sits with several other guidance publications from international regulators and HTA bodies including the FDA [12], EMA [13], HAS [14], and CADTH [15] guiding the use of real-world data to support policy decisions.

RWE framework, 1 year on

The framework has a considerable focus on the use of real-world evidence studies to demonstrate comparative treatment effects which require greater scrutiny to enable trust for decision makers. However, beyond comparative effectiveness, RWE remains the preferred source of evidence for numerous other use cases, which have been quietly influential in NICE since the publication of the framework (Table 1). Updates to the technical manuals across NICE signal an intent to use this evidence more often [16,17].

Table 1. . Influential uses of real-world evidence discussed in appraisals since the publication of NICE's RWE framework in June 2022 (non-exhaustive).

RWE use discussed by NICE committee	Appraisal	Ref.
To demonstrate the generalisability of trail evidence to the UK population for patient characteristics	TA904, TA883	[18,19]
To estimate cost-per-use for diagnostic technology	DG48	[20]
To estimate baseline event rates, in modelling, to which relative effects from trial data are applied	TA897	[21]
To demonstrate an early signal of value for conditional recommendation of a digital therapy	HTE9	[22]
To scrutinize or support extrapolated outcomes in economic modelling	TA883, TA870, TA864, TA801	[19,23,24,25]
To enable effectiveness, or cost-effectiveness estimation for an important subpopulation	TA880, HST23	[26,27]
To provide reassurance that outcomes observed in key trial data are reflected in routine practice	TA872	[28]
As the main source of comparative effectiveness evidence	HST22, TA855, TA850	[29,30,31]
To estimate dose in clinical practice, and therefore, costs	TA866, TA808	[32,33]
To estimate rates of complications beyond the duration of available trial data and health state transition probabilities in economic modelling	TA860, TA804	[34,35]
To provide supportive evidence for an uncertain indirect treatment comparison	TA816	[36]

NICE: National Institute for Health and Care Excellence; RWE: Real-world evidence.

While the direct analysis of real-world data for comparative effects estimation remains in its infancy for appraisals of medicines, it's playing an increasingly influential role. Collaborative demonstration projects are underway exploring the use of RWE of comparative effects, for exploring health inequalities and the use of robust research methods in bespoke analysis to support clinical guidelines [37] and the use of federated data networks and international data [38]. We examine the following case study in greater detail, which shows the use of the framework in improving evidence submissions to NICE for a rare treatment indication.

A ‘living’ framework

In a rapidly developing environment, NICE is taking a living approach to the framework to ensure that it remains useful, usable, and up to date. For example, recent small updates to the framework have included a link to the HARmonized Protocol Template to Enhance Reproducibility (HARPER) tool for supporting protocol design for real-world evidence studies [39], and additional details on conducting systematic searches for real-world data.

During the development and launch of the RWE framework, stakeholder feedback was sought on areas of the document that could be developed or improved for future updates. For example, stakeholders were concerned that while best practice principles outlined in the framework broadly support robust study design and analysis, this can be difficult to achieve in the very scenarios where real-world evidence is most likely to be relied upon, i.e., rare diseases. Here, the few choices available for sourcing real-world data can make cutting edge methods for comparative effects estimation difficult to apply. For this reason, a trade-off is generally necessary between representativeness of data and better quality data which may be available internationally.

In the age of digital health transformation, federated data networks (FDNs) and registry data have emerged as separate potential solutions to the rare disease problem [40]. FDNs facilitate the integration of patient data across various sources without moving the data from its origin, maintaining privacy and security and allowing decentralized data analysis. Registries are specifically designed for the collection of data related to particular diseases or conditions. They provide a comprehensive, longitudinal view of patient care and outcomes allowing for a broader understanding of disease progression, treatment adherence, and long-term safety and efficacy of interventions. Registry data can also contribute to FDNs, resulting in a potent repository of data that may be harnessed to support robust study design and analysis in rare diseases, where traditional data sources may be limited.

Stakeholders also recommended the need for more guidance specifically for those developing evidence for medical technologies who face several unique challenges [41]. Foremost, small and medium sized organizations have less well resourced research departments, and struggle to demonstrate the efficacy and safety of a new technology through time-consuming and expensive randomized controlled trial design. Moreover, the technology under assessment may become obsolete or superseded by the time evidence generation is complete. Increasingly the technology can also undergo iterative changes and improvements, meaning it is altered by the end of the observation period – leading to further challenges in understanding long term effects.

Guidance is needed to overcome these evidence challenges as technology continues to innovate at an unprecedented pace, and is increasingly adopted in the NHS, in which technology is seen a key solution to current patient waitlists and the workforce crisis [5]. Real-world evidence can offer a solution to safe adoption of technologies that show early value in priority areas, while raising the standard of evidence on which such adoption tends to be based.

Real world evidence framework supports evidence generation plans in early value assessment of medical technologies

NICE has piloted a new approach to assess early value of technologies that are most needed and in demand [9]. This approach allows rapid assessment of digital products, devices and diagnostics for clinical effectiveness and value for money. So, the NHS and patients can benefit from these promising technologies sooner, while incentivizing the generation of robust evidence while these are in use.

Using this approach, promising technologies can be recommended for use in the health service on the condition that real-world evidence is generated to address existing evidence gaps. To support this, NICE develops an evidence generation plan prioritising the areas of uncertainty, the real-world evidence that needs to be gathered while it's in use, and any forecasted implementation challenges. This provides further opportunity for the RWE framework to directly impact the quality of generated evidence upstream of its reaching NICE decision-making committees.

Conclusion

The shifting landscape of health technology assessment, bolstered by the unprecedented pace of technological innovation, demands adaptive, real-world evidence frameworks. NICE's strategy to harness real-world data and evidence across different product-types and their lifecycles is positioned to provide greater evidential support to decision making at NICE, while making this evidence base more patient-centric and capable of addressing the pressing challenges of the 21st century.