Navigating the Real World: A Scoping Review of Structured Frameworks to Effectively Identify, Evaluate, and Select Real‐World Data Sources for Fit‐for‐Purpose Studies

Abstract

The potential of real‐world data (RWD), particularly from patient registries, has been increasingly recognized over the last decade by academia, regulators, and health technology assessment (HTA) bodies for its role in assessing a product's effectiveness and supporting regulatory submissions. The selection of an appropriate RWD source is of primary concern, since the success of regulatory processes depends on the quality and relevance of the data. In more recent years, EMA and FDA have released extensive guidance on the use of RWD to produce evidence. Simultaneously, public and private research institutions have proposed structured frameworks developed to guide stakeholders in evaluating and selecting “fit‐for‐purpose” RWD sources. This scoping review provides an overview of these structured frameworks, identifying nine key tools, including the Registry Evaluation and Quality Standards Tool (REQueST) and the Structured Process to Identify Fit‐For‐Purpose Data (SPIFD2). Each framework is briefly described, followed by a comparative analysis of the criteria they assess. These criteria relate to dimensions such as study design, data reliability, data relevance, ethical considerations, and practical factors such as cost and feasibility. Our findings indicate that while these frameworks offer robust tools for ensuring the suitability of RWD sources, each has unique strengths and limitations depending on the specific context of use. By providing a comprehensive understanding of these frameworks, this review aims to assist stakeholders in identifying and/or evaluating and/or selecting the most appropriate RWD sources for generating high‐quality evidence for regulatory and HTA purposes.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Currently, RWD including registries are more widely used in clinical research and especially when submitting studies for regulatory purposes (e.g., drug acceptance for a particular disease). Given the stakes involved, the choice of the RWD source is of crucial importance and multiple guidance documents were released by global health authorities such as the EMA and FDA to help several stakeholders (e.g., regulators, researchers, HTA agencies) identify, evaluate, and select a “fit‐for‐purpose” RWD source. Several structured frameworks as tools or methods were developed to guide users to appraise the relevance and reliability of a given RWD source, but there is no review discussing in a single paper the existing frameworks.

• WHAT QUESTION DID THIS STUDY ADDRESS?

This scoping review allows capturing several frameworks developed to help in identifying and/or evaluating and/or selecting a RWD source for regulatory purposes. A succinct description of each identified framework is provided, and all criteria considered in each of those are captured. Finally, this review provides a depth comparison of all the identified frameworks in terms of criteria considered and the capabilities of each given framework in a practical and concise way.

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

This study allows one to pinpoint the strengths and limitations of each of the identified frameworks as well as their intended use.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

Based on frameworks capabilities and aspects, the provided results could help researchers and other stakeholders in all steps when selecting a RWD source and, consequently by ensuring the suitability of this given RWD source, increase the chances of acceptance of regulatory submissions.

Medicinal products' marketing authorizations are usually based on data obtained after years of clinical research, in which randomized controlled trials (RCTs) are the gold standard to assess a product's efficacy and safety. RCTs allow for a high control over several study aspects, such as inclusion/exclusion criteria, the study duration, and the treatment intake, resulting in an accurate treatment effect estimation in a relatively homogeneous study population. Real‐world data (RWD) complement the findings of RCTs and mitigate some of the limitations, such as enhancing the generalizability to real‐world target populations by including a more heterogeneous population. ¹ , ² Moreover, the use of RWD allows for larger sample sizes and longer follow‐up periods, enabling subgroup analyses, pharmacoeconomic analyses, the detection of rare adverse events, and studying long‐term effectiveness, and special populations such as children, elderly people, and pregnant women. There exist several examples of studies where RWD supported or even served as the main source of evidence generation to prove efficacy and allow for regulatory approval. ³ According to the European Medicines Agency (EMA), RWD are defined as “routinely collected data relating to patient health status or the delivery of health care from a variety of sources other than traditional clinical trials.” This variety of sources includes registries, health insurance data, pharmacy claims data, and electronic health records (EHR). ⁴

RWD, however, has limitations, including that it is often of unstructured nature, it may be incomplete with regard to some outcomes/measurements, ² and its management is still challenging in terms of collection, access, linkage, storage, quality, and analysis. In addition, the lack of use of standard terminologies and data formats across RWD challenges the interpretability of the available data, and the comparability across populations. From the various types of RWD sources, regulators consider in particular patient registries, defined by the EMA as “an organised system that collects uniform data (clinical and other) to identify specific outcomes for a population defined by a particular disease, condition, or exposure,” ⁵ as a potentially useful source of RWD. Today, registry‐based RWD are widely used in epidemiology, ⁶ post‐marketing safety monitoring, and pharmacoeconomic analysis. Its application to support the efficacy and effectiveness of medicinal products is recognized by regulators ⁷ and health technology asessment (HTA) bodies ⁸ , ⁹ and is progressively gaining importance in their decision‐making. ¹⁰ Moreover, the potential of linking registry data to other types of sources such as claims or EHR data constitutes a great opportunity to further improve the observation of the longitudinal patient journey. Nevertheless, development is still ongoing, and challenges persist, such as the need for harmonization of the terminology, accessibility, improved convergence in guidance and methods, and enhanced transparency.

Earlier efforts in the United States and Europe, including the Registry of Patient Registries (2011–2014) and the Patient Registries Initiative (2012–2014), aimed to encourage the use of patient registries in research and regulatory decision‐making. ¹¹ , ¹² This scoping review also aimed to capture frameworks or methods proposed specifically on patient registries assessment.

It is essential to ensure that RWD, including registries, can best meet the research needs, whether in a regulatory, scientific research, or health technology policy‐making context. Some submitted studies to the EMA have been rejected by regulators because the required level of evidence was not successfully achieved. ¹⁰ The rejection of these studies may be due to substantial bias, ² caused by methodological issues (e.g., selection bias or failure to account for confounders) or caused by factors intrinsic to the RWD source itself (e.g., missing endpoints, inappropriate follow‐up time, or a high proportion of missing data). Some of these issues could have possibly been avoided if a tool was used to identify data sources and assess them regarding fit‐for‐purpose.

Recently, the EMA has published guidelines on registry‐based studies and the US Food and Drug Administration (FDA) has released a guidance for the industry, providing a framework for the design, conduct, and analysis of registry‐based studies, including recommendations on data collection and quality, and statistical analysis to facilitate regulatory decision‐making. ¹³ , ¹⁴ Prior to these guidelines, the US Agency for Healthcare Research and Quality (AHRQ) ¹⁵ , ¹⁶ endorsed the use of the PICOTS framework as a valuable tool for considering RWD fitness to address a specific clinical research question. This model is an extension of the PICO (Populations, Interventions, Comparators, and Outcome) framework, disseminated by Richardson in 1995 ¹⁷ and Counsell in 1997, ¹⁸ with the additions of “T” for timing or timeframe and “S” for setting. More recent similar initiatives to evaluate the potential of registries as data sources to support HTA have been proposed, ¹⁹ including the Registry Evaluation and Quality Standards Tool (REQueST) by the European Network for Health Technology Assessment (EunetHTA) to evaluate the relevance of a registry to conduct registry‐based HTA studies. ²⁰

As the term “framework” can be used in a variety of settings, in contrast to descriptive frameworks generally providing larger, more detailed information, we focused on structured ones corresponding to tools or methods that enable users to quickly evaluate distinct dimensions or the suitability of a particular RWD source to potentially deliver evidence including for instance tools that help for the selection of the appropriate study design or the RWD source. Thus, the aim of this scoping review was (i) to identify structured frameworks within the published literature, and (ii) to perform a comparative analysis of the identified structured frameworks based on their cumulative criteria.

The use of reliable, structured frameworks for identifying and/or evaluating and/or selecting RWD sources can be a game changer for stakeholders, helping them save time and mitigate risks associated with data sources, like registries or EHR, when conducting studies and generating evidence for regulatory or HTA purposes. This scoping review aimed to provide both a quick‐reference document and a comprehensive overview of existing structured frameworks.

METHODS

Study design and data sources

A scoping review was conducted. Articles published between January 1, 2013, and December 5, 2023, were identified from Medline (Pubmed), Web of Science and Scopus based on their titles and abstracts using a combination of search terms related to RWD, real‐world evidence (RWE), registries, fit‐for‐purpose or fit‐for‐use, and the regulatory field. In this paper, “fit‐for‐purpose” expresses whether a given RWD source seems adequate to address a specific research question, and consequently, whether data seem of sufficient quality to generate relevant and reliable evidence to inform regulatory decision‐making. The search terms and strings on the selected sources are depicted in Table S1 .

Studies selection

Three reviewers (JM, JT, SZ) independently screened the titles and abstracts of the identified articles. Included were the articles proposing or mentioning a structured framework to help users identify and/or evaluate and/or select RWD sources on a “fit‐for‐purpose” basis. Publications not in English, case reports, opinion letters, and conference abstracts were excluded. Results from reviewers were compared, and in cases of divergence, the full article was read to determine whether to include or exclude it.

Data extraction and reporting

Two reviewers (SZ, JM) fully read and reviewed the selected articles to extract the information about the frameworks. Data extraction was performed by one reviewer using a predefined spreadsheet format to collect the name of the identified framework, the cumulative criteria mentioned, the frameworks' intended application field being regulatory purposes, and the appropriateness of the frameworks' application field, that is, to identify and/or evaluate and/or select RWD sources on a fit‐for‐purpose basis. The second reviewer checked the extractions. Disagreements were handled by consulting a third reviewer (JT).

We defined identification as the ability to propose a standard method or interface to efficiently and reproducibly search for RWD sources to be considered in the evidence generation process. Evaluation was defined as a tool's capability to offer a standardized approach for evaluating to what extent a given RWD source is fit‐for‐purpose, and selection was defined as providing structured elements or rankings for selecting the most pertinent source. Cumulative criteria described in each identified framework were listed and grouped into newly identified overarching covered dimensions and descriptively compared among the selected frameworks by one reviewer (SZ) and checked by another (JM). The aim of this step was to provide a more visual and systematic comparison of the frameworks, even though their applicability was not the same. Subsequently, we established whether the frameworks integrated a qualitative, numerical, or color‐coded scoring system as part of the evaluation approach. This comparative analysis aimed to discern what aspects are covered or omitted by each framework, providing a comprehensive understanding of their potential. Finally, a non‐exhaustive search was conducted to determine which of the identified frameworks were cited in guidance white papers published by the most well‐known medicines and HTA agencies. The results of this search are presented in Table S2 .

RESULTS

The search revealed 977 articles, of which 45 were duplicates (Figure 1 ). This led to the screening of the titles and/or abstracts of 932 articles, of which 877 were excluded, as they did not discuss frameworks for RWD source identification, evaluation, or selection. The 55 remaining articles were fully reviewed (Data S1 ). From this selection, 45 articles comprised studies related to the topic, reviewing the state of the art in terms of guidance or perspectives but not discussing nor presenting structured frameworks or methods. Finally, nine references presented and discussed structured frameworks facilitating the evaluation of RWD or registry sources for their fitness‐for‐purpose. Of note, five non‐structured frameworks and one semi‐structured were identified through this review but were not further described in detail in this paper. Their descriptions are available in Table S3 .

Figure 1.

PRISMA flow diagram from identification, screening, to those included in the final review.

Overview of the identified structured frameworks

The scoping review revealed, by chronological order, nine structured frameworks: (i) Structured Pre‐approval and Post‐approval Comparative study design (SPACE); (ii) Patient populations, Interventions, Comparators, Outcome measures of interest, Timing, and Settings (PICOTS) for claims and EHR data sources; (iii) Coordinated registry networks (CRNs) maturity framework (MATURITY); (iv) Structured Process to Identify Fit‐For‐Purpose Data (SPIFD); (v) PICOTS—three‐step conceptual model for oncology; (vi) the RWD‐Cockpit; (vii) REQueST; (viii) SPIFD2; (ix) and Authentic Transparent Relevant Accurate Track‐Record (ATRaCTR). Of these identified frameworks, SPACE was the first, published in 2019 (Table 1 ).

Table 1.

List and general characteristics of the identified structured frameworks

Framework		SPACE	PICOTS* (feasibility of claims/EHR)	Maturity	SPIFD	PICOTS (3‐step conceptual mode)	RWD‐Cockpit	REQueST†	SPIFD2	ATRAcTR
Year of publication		2019	2020	2022	2022	2022	2022	2022†	2023	2023
Main objective described by authors		To propose a structured design framework to generate RWE, identifying elements and criteria to assess feasibility and validity, as well as tracking decisions	To provide a structured framework that uses PICOTS framework to assess the feasibility of using claims/EHR data source for specific research question	To build a robust framework to assess the maturity of registries and CRNs for medical device research and surveillance, to address increasing evidentiary needs of regulators	To improve previous structured framework for conducting feasibility assessments, bases on step‐by‐step process and aligned with FDA RWE framework	Building on previous work, introduce a conceptual mode to assist researchers in assessing the suitability of an RWD source for answering a specific cancer‐related research question	To develop a web application serving as a tool for translating the characterization of specific quality parameters of RWD into a metric. Additionally, the goal was to put forth a standardized framework for appraising the quality of RWD	To provide a tool enabling organizations in guiding and evaluating registries for effective deployment on HTA studies or purposes	Building upon SPACE and SPIFD, authors proposed an improved version of the tool to provide a comprehensive transparent documentation for protocol decisions, to guide on studies design and to help users assessing the feasibility of candidate sources	To help researchers assessing whether existing reusable RWD sources may be fit‐for‐purpose when their objective is to answer questions from regulatory agencies or to support claims regarding benefits and risks of therapies
Application field according to guidance(s) cited		Regulatory, researcha,b	Regulatory, researcha	Regulatory, HTA, researcha	Regulatory, HTA, researcha,b,c,d	Regulatory, research in oncologya	Regulatory, researcha,b	HTA, regulatory, researchb,d	Regulatory, researcha,b,c	Regulatory, researcha,b,c,d
Capability covered on a fit‐for‐purpose	Identification	No	No	No	No	No	Partially‡	No	No	No
	Evaluation	Yes	Partially	Yes	Yes	Yes	Yes	Yes	Yes	Yes
	Selection	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

ATRaCTR, Authentic Transparent Relevant Accurate Track‐Record; EMA, European Medicines Agency; FDA, Food and Drug Administration; HTA, Health Technology Assessment; MATURITY, maturity of coordinated registry networks (CRNs) and registries; NICE, National Institute for Healthcare Excellence; PICOTS, Patient populations, Interventions, Comparators, Outcome measures of interest, Timing, and Settings; REQueST, Registry Evaluation and Quality Standards Tool; RWD, Real‐World Data, SPACE, Structured Pre‐approval and Post‐approval Comparative study design; SPIFD, Structured Process to Identify Fit‐For‐Purpose Data.

^*PICOTS framework is presented here as proposed in the publication of Richtey et al. 2020, but its initial blocks were proposed by Counsell et al. 1997 and Whitock 2010.

^† The initial white paper on REQueST tool was published by EUnetHTA in September 2019; the year 2022 referenced here is linked to the publication date of the study by Allen et al. (2022) on which the tool was analyzed and tested on 2 distinct registries.

^‡ RWD‐Cockpit tool partially covers the identification step, since authors proposed associated with the tool a compiled and structured repository of 106 real‐world data sources. Reference/guidance cited: a‐FDA, b‐EMA, c‐NICE, d‐EUnetHTA.

Blue‐coloured boxes indicate “No,” green‐coloured boxes indicate “Yes,” and purple‐coloured boxes indicate “Partially.”

Nine frameworks were designed for research and regulatory purposes, three of which additionally based their design on HTA guidance documents (Table 1 ). All the identified frameworks cover, at least to some extent, the application of evaluation and selection. Only one of the identified frameworks, the RWD‐Cockpit, partially addresses identification by providing a selection of sources from an in‐house catalogue of RWD. ²¹ For the other frameworks, the process of identifying RWD sources is not among their respective main goals/applications. Below, a concise description of each of the frameworks is presented following the chronological order of their publication, except for SPIFD and SPIFD2, as SPIFD was intended to be used with SPACE, and SPIFD2 represents an evolution of these frameworks.

SPACE (Structured Pre‐approval and Post‐approval Comparative study design), SPIFD (Structured Process to Identify Fit‐For‐Purpose Data) and SPIFD2

SPACE corresponds to an iterative structured framework proposed by Gatto et al. (2019). ²² Their aim was to guide users on evaluating, independently of RWD type, the global design of a study to produce valid and transparent RWE. SPACE provides tabular templates to follow a four main steps method: (1) establishing a clear research question and structuring the corresponding study design components in a flexible and pragmatic manner that aligns with real‐world considerations, unlike the more rigid approach typically employed in RCTs; (2) identification of the potential confounding variables through a causal diagram; (3) listing the desirable minimal criteria to obtain a valid capture of design elements as a baseline feasibility evaluation; and (4) providing a flowchart as a decision aid to determine whether candidate data sources are “fit‐for‐purpose” by identifying validity concerns and potential solutions. All details gathering motivations, assumptions, and evidence are reported to justify the choice of the final candidate data sources. As the framework was developed considering multiple guidance documents on RWE and pharmacoepidemiology studies, some of which have been published by the Duke Margolis Health Policy Center and the FDA at the time of its development, ²³ , ²⁴ , ²⁵ the deployment of this four‐step approach and tracking of elements is meant to enable users to argue for and reach agreement that a specific real‐world study will provide evidence suitable for regulatory purposes. While the authors only provide examples of RWD post‐approval safety study designs, the framework is also intended for use in pre‐approval study settings. Although, the intent of SPACE was to guide a researcher through designing a fit‐for‐purpose study to address a particular research question, the completed tables can be provided to a stakeholder to review the rationale for the design decisions.

Based on the recommendations from the FDA ²³ , ²⁵ , ²⁶ and from the Duke Margolis Center for Health Policy, ²⁷ , ²⁸ Gatto et al. (2022) ²⁹ incorporated three new steps between the steps 3 and 4 of the SPACE ²² framework resulting in the SPIFD framework. Of the three added steps, the first one consists of ranking the minimal criteria to identify the “must‐have” variables. The second step provides a flowchart to identify the best candidate data source having the highest ranked criteria. The third step allows to evaluate each component of the minimal criteria and about logistical information (e.g., time to data availability, time to contract execution) for each data source candidate by ranking each component from 0 (=does not meet study requirement) to 5 (=many/nearly all data requirements met). Various templates are offered including one utilizing an automated heatmap to visually determine the most suitable data source.

In light of evolving guidance and publications from regulatory and HTA stakeholders concerning RWE since 2021, ²⁶ , ³⁰ , ³¹ Gatto et al. (2023) ³² later updated and merged their previous tools, SPACE ²² and SPIFD, ²⁹ resulting in an improved framework named SPIFD2. SPIFD2 incorporates enhancements in evaluation aspects related to study design and real‐world bias evaluation. Additionally, a reporting layer aligned with the Structured Template and Reporting Tool for Real‐World Evidence (STaRT‐RWE) ³³ developed to enhance transparent communication of RWE generation has been integrated into the tool. Thus, compared to SPIFD, three main improvements according to Gatto et al. ³² can be observed. Firstly, modifications in the main table design include the addition of rows to comprehensively describe study objectives, design elements, and practical considerations such as budget and primary/secondary data usage. Secondly, a new table is incorporated to document the rationale behind selecting potential confounding variables, aiming to capture bias sources more exhaustively. Thirdly, a template is added to document the process of narrowing down candidate data sources for feasibility evaluation. Like SPIFD, data fitness criteria are prioritized based on uniqueness and importance, with a limit of five candidate data sources to be effectively evaluated simultaneously. In addition, authors declared that this new version of the framework seamlessly connects with the HARmonized Protocol template to Enhance Reproducibility (HARPER), ³⁴ which guides researchers in writing a protocol. The SPACE, SPIFD, and SPIFD2 frameworks allow for both evaluation and selection of data sources.

PICOTS (Patient populations, Interventions, Comparators, Outcome measures of interest, Timing, and Settings) based algorithm for claims and EHR data sources

Predicated on the PICOTS ¹⁷ , ¹⁸ framework, Ritchey et al. (2020) proposed an algorithm ³⁵ to evaluate the feasibility of using a claims/EHR data source focusing on data relevance, the adequacy of data capture, and data quality and reliability. The algorithm follows a decision path, where researchers need to address each of the PICOTS elements and key confounders sequentially. This demonstration suggests that the framework can be readily adapted to evolving guidelines and practices, thereby aiding in the evaluation of the feasibility of using RWD for a specific research purpose. Ritchey et al., ³⁵ present it as an option to more complex frameworks such as SPACE to allow users in evaluating and selecting EHR or claims data sources.

PICOTS—Three‐step conceptual mode for oncology

Penberthy et al. (2022) ³⁶ proposed a 3‐step conceptual model, partially based on the initial PICOTS framework, to evaluate the suitability of a RWD source for addressing oncology‐specific research questions. In step 1, the research question is formulated by incorporating the PICOTS framework to emulate a target trial with non‐randomized data, ensuring clear objectives and interpretation. In step 2, the user needs to understand the content of the data source (variables) to ascertain the population covered and confirm the beneficial applicability of the results to the intended population. Step 3, the final step, involves evaluating the relevance of the data source to the research question through an evaluation of the data provenance (purpose of data generation), data quality (completeness, validity), and data limitations (e.g., lack of specific data). Despite the authors' emphasis on RWD source exploration, the proposed model meticulously evaluates the suitability of the given RWD in steps 2 and 3, ensuring that studies conducted using this data yield valid evidence. This framework covers the domains of evaluation and selection of RWD specifically for oncology studies.

MATURITY—a framework of maturity of coordinated registry networks (CRNs) and registries

The “MATURITY” framework, developed by Sedrakyan et al. (2022), ³⁷ evaluates the fitness of registries and CRNs as RWD sources for medical devices research and surveillance. This framework evaluates registries based on their capacity to deliver accessible, relevant, and reliable information for evidence generation. The Medical Device Epidemiology Network (MDEpiNet) Coordinating Centre, the FDA and other regulators, patient advocacy groups, academics, clinicians, and industry stakeholders collaborated to create the seven evaluated domains of the framework. These domains are promoting unique device identification, improving data collection efficiency, advancing data quality for regulatory decision‐making, considering the total medical device life cycle research, establishing governance, and ensuring sustainability, leveraging registries as quality systems, and incorporation of patient generated data and patient‐reported outcomes in CRNs. The publication provides a tabular view of these seven domains, where each domain is categorized into five levels: early learner, making progress, defined path to success, well managed, and optimized. A registry or CRN does not need to achieve the highest score in all seven domains to be considered mature. MATURITY covers the domains of evaluation and selection of RWD for oncology studies.

RWD‐cockpit

Considering the challenges linked with the use of RWD on studies to support regulatory decision‐making, Brabak et al. (2022), ²¹ have proposed a web‐based tool to be used as a framework for preliminary evaluating the quality of RWD sources. As a proof of concept, authors have first created a repository of 106 RWD sources eligible for tool deployment. The tool utilizes seven variables in its scoring system: manageability (4 sub domains) corresponding to access rights or previous studies using the given RWD source; complexity (3 sub domains) evaluating whether the RWD source contains single, multiple, or longitudinal measurements; sample size (4 sub domains); privacy and liability (5 sub domains); data accessibility (4 sub domains), periodicity of RWD sources update (4 sub domains), and standardization (4 sub domains) indicating if RWD adhere to technical or metadata international standards. Each metric variable is associated with several descriptors that define specific characteristics of the dataset and their level. The adopted rating procedure consists of four stages. Initially, a score ranging from 0 to 100 is assigned to every descriptor of each variable. It is important to note that each variable in a particular dataset can be characterized by one or more descriptors (e.g., data can be multivariable and longitudinal). Subsequently, an aggregate score for each variable is derived by computing the mean of all its descriptor scores. Next, the mean score of all the variables is computed. Finally, this cumulative score undergoes normalization to yield a score within a range of 1 to 5, with 5 representing the highest quality of a dataset. The normalization procedure involves dividing the cumulative score by the total number of variables (7 in this case), followed by division by 100, and multiplication by 5. The resultant score for a particular dataset is termed the “quality identifier.” As of late 2023, the platform is no longer accessible to external user testing. ³⁸ RWD‐Cockpit partially addresses the capability of identifying RWD sources and can be used for RWD evaluation and selection.

REQueST (registry evaluation and quality standards tool)

In the light of previous findings from initiatives such as PARENT ³⁹ and EUnetHTA, ⁴⁰ Allen et al. (2022) ⁴¹ explored the development and applicability of the REQueST tool ²⁰ for patient registries evaluation in the intent to use such sources to produce RWE and guide decision‐making on HTA applications. The REQueST tool comprises three steps, each with specific tables detailing content or “area” for which the responsibility to demonstrate registry or content quality rely on registry holders, while HTA agencies, regulators, or researchers will evaluate the information completed by data holders. Step 1 of the tool collects methodological information aspects from the registry. Subsequent step 2 compiles prerequisites for good practices and quality of evidence generated from the source. Finally, step 3 includes the evaluation of other aspects corroborating to the quality of RWE. Each step's output column indicates if the minimal standard is met, according to 3 categories—“Satisfactory,” “Needs development/clarification,” or “Not suitable.” Upon completion, REQueST generates a final comprehensive table, color‐coded to reflect the evaluation of the categories. The tool is available in a tabular format to be shared and completed by registry owners and later evaluated by requesting stakeholders (e.g., HTA agencies, researchers) to identify if the data source is fit‐for‐purpose for a specific scope. REQueST is tailored for RWD evaluation and selection but not to discover existing databases.

ATRaCTR (Authentic Transparent Relevant Accurate Track‐Record)

Deploying a literature review of frameworks and guidance focused on quality aspects of data and generated evidence, Berger et al (2023) ⁴² proposed a framework of screening criteria to evaluate five dimensions for a given RWD source: authenticity, transparency, relevance, accuracy, and track‐record. ATRAcTR is displayed as a questionnaire of 20 closed and 24 open questions that are intended to guide users on the validity of RWD in terms of mostly data quality aspects. First, questions related to “Authenticity” allow for providing information about data type, provenance, collection purpose and processes, as well as data access. Then, the “Transparency” section communicates about processes used in data acquisition, curation, editing, and linkage. The “Relevance” section informs whether the sample size or length of follow‐up are sufficient and whether the data comprises necessary elements to implement a RWD study. Questions related to “Accuracy” pertain to data quality checks employed to prove to what extent data reflects the concept(s) they are intended to represent. Finally, questions dedicated to the “Track‐record” dimension documents existing cases where the given RWD source was used to generate RWE. RWD evaluation and selection can be performed through ATRAcTR.

Comparative analysis of the identified structured frameworks according to the captured criteria

As explained previously, a comparative analysis was proposed listing all the cumulative criteria covered by the selected frameworks and analyzed together independently of framework application field to better visualize the dimensions assessed by the tools. The identified frameworks often covered common dimensions related to study type, data relevance and reliability, ethics, and practical aspects, and provided an assessment approach.

Regarding study type, the captured criteria encompassed research question, population, outcome(s), exposure (e.g., treatment, test or procedure), comparator (e.g., placebo, any blinding), length of follow‐up, settings (e.g., primary or secondary care), sample size, in−/exclusion criteria, presence of confounding variables, estimator(s), and study design (e.g., RCT, observational). Most of these aspects were covered by SPIFD2 ³² and REQueST ²⁰ , ⁴¹ and the least by ATRAcTR ⁴² and MATURITY ³⁷ (Table 2 ). Length of follow‐up was covered by all frameworks, except for the RWD‐Cockpit. ²¹

Table 2.

Comparative analysis of study type aspects covered by the identified structured frameworks

Study type aspects	SPACE	SPIFD	SPIFD2	PICOTS (feasibility of claims/HER)	MATURITY	PICOTS (3‐step conceptual mode)	RWD‐Cockpit	REQueST	ATRAcTR
Research Question	+	+	+	+/−	−	+/−	−	+/−	−
Population	+	+	+	+	−	+	−	+	+/−
Outcome(S)	+	+	+	+	−	+	−	+	−
Exposure (e.g., Treatment, test or Procedure…)	+	+	+	+	−	+	−	+	−
Comparator (e.g., Placebo, any blinding…)	+	+	+	+	−	+	−	+	+/−
Length of Follow‐up	+	+	+	+	+	+	−	+	−
Settings (e.g., Primary Care, Secondary…)	−	−	−	+	−	+	−	+	−
Sample Size	+	+	+	−	−	−	+/−	+	−
Inclusion/Exclusion Criteria	+	+	+	+	−	+	−	+	−
Control of confounding variables	+	+	+	−	−	−	−	+	−
Estimator(s)	−	−	+	−	−	+/−	−	−	−
Study type (e.g., RCT, observational…)	+/−	+/−	+/−	+/−	−	+/−	−	+	−

Based on the more recent Heads of Medicines Agencies (HMA)‐EMA Data Quality Framework, ⁴³ the following aspects with regard to data relevance and reliability comprised data content, periodicity, coverage, completeness, accuracy, validity, consistency, plausibility, and conformance. Most of these aspects were covered by ATRAcTR, ⁴² followed by MATURITY, ³⁷ and REQueST ²⁰ , ⁴¹ (Table 3 ). PICOTS ³⁵ did not cover any of the data relevance and reliability aspects. Data content and validity were addressed in most of the frameworks, whereas consistency was covered in one framework only, that is, ATRAcTR. ⁴²

Table 3.

Comparative analysis of data relevance and reliability aspects covered by the identified structured frameworks

HMA/EMA data quality framework	Data relevance & reliability aspect	SPACE	SPIFD	SPIFD2	PICOTS (feasibility of claims/HER)	MATURITY	PICOTS (3‐step conceptual mode)	RWD‐Cockpit	REQueST	ATRAcTR
Data type	Data content (Demographics, Clinical, PROs)	+	+	+	−	+	+	+/−	−	+
Timelines > Currency	Periodicity	+	+	+	−	−	−	+	−	+/−
Extensiveness > Coverage	Coverage	−	−	−	−	+	+	−	+	+
Extensiveness > Completeness	Completeness	−	−	−	−	+	+	−	+	+
Reliability > Accuracy	Accuracy	−	−	−	−	+	−	−	+	+
Reliability > Precision	Validity	+	+	+	−	+	+	−	+	+/−
Extensiveness > Coherency	Consistency	−	−	−	−	−	−	−	−	+
Reliability > Plausibility	Plausibility	−	−	−	−	+	−	−	−	+
Extensiveness > Coherency > Conformance	Conformance	−	−	−	−	+	−	+	+	+

The captured criteria related to the ethical and practical aspects were primary or secondary use of data, data linkage, transparency, data protection policies, governance, timelines (contract/data access/analysis), budget, privacy rules, access (rights/methods), and confidentiality. All of these were covered by REQueST, ²⁰ , ⁴¹ whereas none were covered by SPACE ²² and PICOTS ³⁵ , ³⁶ (Table 4 ). Transparency was covered by the largest number of frameworks, namely five, that is, SPIFD, ²⁹ PICOTS, ³⁵ , ³⁶ REQueST, ²⁰ , ⁴¹ SPIFD2, ³² and ATRAcTR. ⁴² The definition of each of the above‐listed criteria is provided in Table S4 .

Table 4.

Comparative analysis of ethical and practical aspects covered by the identified structured frameworks

Ethics & practical aspects	SPACE	SPIFD	SPIFD2	PICOTS (feasibility of claims/HER)	MATURITY	PICOTS (3‐step conceptual mode)	RWD‐Cockpit	REQueST	ATRAcTR
Primary or secondary use of data	−	−	+	−	−	−	+	+	+
Data linkage	−	−	+/−	−	−	−	−	+	+
Transparency	−	+	+	−	−	+	−	+	+
Data protection policies	−	−	−	−	−	−	+	+	+/−
Governance	−	−	−	−	+	−	−	+	+/−
Timelines (contract/data access/analysis)	−	+	+	−	−	−	−	+	−
Budget	−	−	+	−	−	−	−	+	+/−
Privacy rules	−	−	+/−	−	−	−	+	+	+/−
Access (rights/methods)	−	+/−	+/−	−	−	−	+	+	+/−

Regarding the assessment approach, in most frameworks, that is, SPACE, ²² PICOTS, ³⁵ and PICOTS‐3, ³⁶ and ATRAcTR, ⁴² a qualitative scoring system is used to review the criteria or items (Table 5 ). Four other frameworks, that is, MATURITY, ³⁷ SPIFD, ²⁹ RWD‐Cockpit, ²¹ and SPIFD2, ³² have opted to provide users with a numerical scoring system. A color‐coded scoring system was used by only one framework, that is, REQueST. ²⁰ , ⁴¹

Table 5.

Assessment approach used by the identified structured frameworks

Assessment approach	SPACE	SPIFD	SPIFD2	PICOTS (feasibility of claims/HER)	MATURITY	PICOTS (3‐step conceptual mode)	RWD‐Cockpit	REQueST	ATRAcTR
Qualitative	•			•		•			•
Numerical scoring system		•	•		•		•
Color‐coded scoring system								•

Green‐coloured boxes indicate the adopted type of assessment approach.

DISCUSSION

In this scoping review, we identified, summarized and compared nine structured frameworks to identify and/or evaluate and/or select RWD or registries, on a “fit‐for‐purpose” approach to inform regulatory decision‐making, that is, SPACE, ²² SPIFD, ²⁹ SPIFD2, ³² PICOTS, ³⁵ MATURITY, ³⁷ PICOTS‐3, ³⁶ RWD‐Cockpit, ²¹ REQueST, ²⁰ , ⁴¹ and ATRAcTR. ⁴²

Although several frameworks have been identified, currently, there is no established consensus on which framework provides the best solution during the deployment of RWE studies, which highlights the need for a more complete tool to guide users through all the steps of evidence generation in real‐world settings. Whereas all these frameworks can be applied for the evaluation and selection of RWD, only one framework (i.e., the RWD‐cockpit) partially facilitates the identification of RWD. The identification of RWD sources constitutes a critical step, as the users need to know the existence of potential candidate RWD sources before assessing their suitability to the research objective. All nine frameworks include several aspects related to the dimensions that include study type, data relevance and reliability, and ethics and practicalities. Regarding these dimensions, the frameworks most often covered the aspects of length of follow‐up, data content and validity, and transparency. REQueST covered most of the aspects across all dimensions, whereas the least number of aspects were covered by RWD‐cockpit.

Besides similarities among the identified frameworks, their application is tailored to meet the diverse needs of users (e.g., researchers, HTA bodies, regulators) according to the guidance cited for their design. In contrast, when analyzing potential guidance from leading medicines and HTA agencies, as well as quality and harmonization bodies such as ENCePP and the International Council for Harmonization (ICH), that could cite one of the nine identified frameworks, only the REQueST and SPIFD tools were referenced in an EMA/ENCePP guidance, while PICOTS and SPIFD were cited in a NICE guidance. SPIFD has been mentioned in ICH documents, and none of the frameworks were identified as being mentioned or qualified by the FDA (see Table S2 ). Some frameworks, that is, PICOTS, offer the ability to filter data sources, allowing for a quick feasibility evaluation of their suitability for a planned study. Others, that is, REQueST, offer more comprehensive and structured methods for assessing data sources to determine their relevance and reliability, thereby ensuring the delivery of evidence that aligns with the users' real‐world study plan. All identified frameworks have their advantages and disadvantages. The RWD‐Cockpit, for instance, appears to be very intuitive but the authors do not explicitly discuss the criteria applied for selecting the RWD sources included in the repository, nor do they address the tool's reliability in relation to published guidance from public authorities in the field of RWE. The color‐coded system proposed on the final sheet of the tool in REQueST is also intuitive, but analysis of the other criteria may generate bias as it does not explicitly state the minimum number of “satisfactory” criteria (green‐coded criteria) that a registry must achieve to be considered fit‐for‐purpose for a specific scope. The tool seems to contribute mostly to performing a feasibility assessment through the appraisal of the final comprehensive table. ATRaCTR is an easy‐to‐use tool that is likely most suitable for use in the early stages of feasibility evaluation. Also, the definition of its fit‐for‐purpose aspects is not clearly outlined, and the methods underlying the development of the framework are only briefly described. The templates of SPIFD could be relevant, but consolidating all templates into a single interface could potentially facilitate wider adoption of the tool.

The identified frameworks are published in recent years (2019–2023). This aligns with the recent release or update of main guidance from regulatory agencies, HTA bodies, and research groups in the field of RWD and RWE. ⁹ , ¹³ , ¹⁴ , ²³ , ²⁴ , ⁴⁴ There is increased attention for the use of RWE by regulatory agencies and HTA bodies, but there are concerns about the quality of RWD due to the lack of control over data collection and quality checks. ⁴⁵ , ⁴⁶ Various efforts have been made to improve the quality and utility of RWD by various stakeholders. In 2016, stakeholders were encouraged to comply with the Findability, Accessibility, Interoperability and Reusability (FAIR) principles that have been formulated to ease the reusability of existing datasets/analyses eliminating the need of primary data collection and supporting reproducibility and replicability thanks to studies' transparency. ⁴⁷ Following this initiative, the development of the Meaningful, Valid, Expedited and Transparent (MVET) principles supported the strong need to ensure stakeholders generate transparently robust, valid and meaningful causal inferences considering an acceptable time period from a regulatory perspective. ⁴⁸ , ⁴⁹ Since the quality of evidence is strongly related to the RWD source to be analyzed, introducing these principles led to the development of numerous frameworks built upon the MVET principles to determine the suitability of RWD sources for a targeted research question. This includes SPACE, SPIFD and SPIFD2 among the frameworks identified in this study.

Understanding data quality of conducted studies is crucial for regulatory and HTA decision‐making. In this line, extensive guidance has been published, and some initiatives were deployed to promote transparency and reproducibility in healthcare decision‐making. Among the latter, HARPER ³⁴ constitutes a template for protocol writing to better document the study plan and facilitate the assessment of potential sources of bias, and STaRT‐RWE ³³ provides a template for transparent communication of RWE generation. In the context of data quality verification, tools and initiatives have also been published which may be relevant to other aspects of fit‐for‐purpose evaluations. An example is the Algorithm CertaInty Tool (ACE‐IT) which has recently been developed to assess EHR and claims‐based algorithms' fit‐for‐purpose for safety outcomes. ⁵⁰ ACE‐IT consists of 34 items, of which 24 items are related to internal validity; seven items are related to external validity; and three items are related to ethical conduct and reporting of the validation study.

Further developments

It must be highlighted that none of the identified frameworks offer automated methods to discover optimal RWD sources or registries that match the clinical research domain, for which the identified frameworks can only be applied when the user has previously pinpointed potential candidate data sources. Currently, clinical studies and database repositories are the main source for database identification or discoverability. Utilizing the identified frameworks presumes, therefore, that users have already conducted repository searches and have access to minimal data information, allowing them to evaluate the fitness‐for‐purpose of various real‐world databases, including registries. Even when ranking or classification methods are proposed in the presented frameworks, thresholds or standardization of scoring methods are often not transparent nor systematic. Accordingly, there exists no unified tool or framework that encompasses both the identification and assessment of RWD or registries. Ideally, a single framework or tool should integrate all necessary steps for guiding users toward effective RWE generation. The goal of such a single tool or framework should be to incorporate existing tools, databases, and guidance, building upon established and effective methods, without duplicating existing solutions. While existing frameworks offer individually structured methods for specific data assessments, it is essential to promptly align and adopt globally standardized terminologies and criteria for RWD, highlighting the need for further developments.

Strengths and limitations

This study provides a comprehensive overview of existing structured frameworks used to identify and/or evaluate or select RWD to generate evidence fit for regulatory purposes and highlights their strengths and limitations. In addition, some of these frameworks were found to be fit‐for‐use for HTA purposes. Nevertheless, the authors encourage readers to contact the lead researchers involved in the framework design to further explore elements not highlighted here or to verify information that was not clear on tools structure. The search strategy may not have been fully exhaustive; indeed, some existing frameworks may have been missed through our search strategy such as the Data Utility framework. This resulted from the absence of the terms “fit‐for‐purpose” and “real‐world data” in the title and/or abstract of their respective articles. In addition, the term “HTA” was currently not included in the search terms, yet specific HTA structured frameworks were found. Although the inclusion of articles and extraction of data can be subjective in part, subjectivity was mitigated by having three reviewers independently perform the screening and selection, after which two reviewers independently extracted the data.

CONCLUSIONS

Nine structured frameworks to identify and/or evaluate and/or select fit‐for‐purpose RWD sources were found in this review study (i.e., SPACE, SPIFD, SPIFD2, PICOTS, MATURITY, PICOTS—three‐step conceptual mode for oncology, RWD‐Cockpit, REQueST, and ATRaCTR). These tools primarily focus on assessing data sources after their identification and when research questions are already established or in progress. As a result, there is currently no tool that enables stakeholders to seamlessly combine both the identification and assessment processes on a fit‐for‐purpose basis. Elements from the identified tools can be incorporated into the development of a new tool designed to automatically identify and assess registries and RWD sources. Such a tool would be useful for facilitating the conduct of research studies based on RWD.

FUNDING

This study was conducted in the context of the More‐EUROPA project. This project has received funding from the European Union's Horizon Europe Research and Innovation Actions under grant no. 101095479. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union nor the granting authority. Neither the European Union nor the granting authority can be held responsible for them.

CONFLICT OF INTEREST

Quinten Health is a French private company applied to healthcare for the pharmaceutical industry, hospitals, and regulatory agencies. GS reports grants and personal fees outside of this study from CSL Vifor, Boehringer Ingelheim, AstraZeneca, Servier, Novartis, Cytokinetics, Pharmacosmos, and personal fees from Roche, Abbott, Edwards Lifescience, Medtronic, TEVA, Menarini, INTAS, GETZ, Hikma, and grants from Boston Scientific, Merck, Bayer, all outside the submitted work. JH declares research grants outside of this study from Biogen, Bristol‐Myers‐Squibb, Janssen, Merck KGaA, Novartis, Roche, and Sanofi‐Genzyme, and speaker's fees or fees for serving on advisory boards for Biogen, Bristol‐Myers‐Squibb, Janssen, Merck KGaA, Novartis, Sandoz and Sanofi‐Genzyme and Teva. All other authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

S.Z., J.T., E.B., S.T.V., R.D.B., E.X., A.G., G.S., J.H., P.G.M.M., K.P., B.A., G.Y.H., and J.M. wrote the manuscript; S.Z., J.M., and J.T. designed the research; S.Z., J.M., and J.T. performed the research and analyzed the data.

DISCLAIMER

The authors' perspective is solely conveyed in any dissemination of results and should not be interpreted or cited as representing their respective affiliations, including Karolinska Institute, University Medical Center Groningen, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Università Cattolica del Sacro Cuore, European Medicines Agency, or any of their committees or working parties. The views expressed in this article are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the regulatory agency/agencies or organizations with which the author(s) is/are employed/affiliated.

Supporting information

Data S1.

Table S1.