Expert survey on real‐world data utilization and real‐world evidence generation for regulatory decision‐making in drug lifecycle in Korea

Abstract

As the importance of utilizing real‐world data (RWD)/real‐world evidence (RWE) for supporting regulatory scientific decision‐making continues to grow, experiences and inputs from experts become crucial for developing a systematic and practice‐oriented plan for the use of fit‐for‐purpose RWD/RWE. This study aimed to survey relevant experts from government agencies, industries, and academia to identify prerequisites for the drug life cycle in Korea. The questionnaire comprised the following: (A) the definition and categories of RWD/RWE, (B) the suitability and feasibility of using RWD/RWE at each authorization stage by the types of RWD, and (C) the challenges and solutions for the use of RWD/RWE. A total of 46 respondents completed the online survey, with 89.1% of them having prior experience with RWD/RWE usage. A majority of respondents agreed that RWD can be obtained from various sources. Among these sources, the registry was the most suitable source. It is suitable to compensate for the limitations of randomized control trials and ensure quality in data collection. Though there was consensus among the respondents for the use of RWD/RWE in post‐marketing surveillance, the use of such data in new drug application (NDA) was disagreeable. Respondents considered it necessary to write a protocol in advance for RWD collection and RWE generation, for all RWD types. In conclusion, this study examined the perceptions of experts for RWD/RWE use at each approval stage of drugs. The results suggest that guidelines for the fit‐for‐purpose use of RWD/RWE should be developed via careful deliberation among experts in the future.

Abbreviations

CDM

common data model

EMA

European Medicines Agency

FDA

Food and Drug Administration

MFDS

Ministry of Food and Drug Safety

NDA

new drug application

PMS

post‐market surveillance

PRO

patient‐reported outcome

RCT

randomized controlled trial

RWD

real‐world data

RWE

real‐world evidence

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Currently, there is a significant increase in the importance of utilizing real‐world data (RWD) and real‐world evidence (RWE) to support regulatory scientific decision‐making across the entire pharmaceutical life cycle. However, there remains a gap in understanding among stakeholders, not only regarding the definition and types of RWD but also the feasibility of RWE at each stage of the licensing process.

• WHAT QUESTION DID THIS STUDY ADDRESS?

We aimed to collect extensive opinions from Korean experts in the field of RWD/RWE, targeting stakeholders through the online survey, regarding the suitability of research designs for RWD/RWE use, necessity, feasibility, possibility of complementing limitations of randomized controlled trials (RCTs), and possible challenges, spanning from drug approval to post‐marketing management.

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Most experts concurred with the definition of RWD as data routinely collected from various sources and its relevance in post‐marketing surveillance. Nevertheless, discrepancies arose in responses concerning the types of RWD/RWE applicable and the appropriateness of study designs depending on the licensing stage. Registry data emerged as the preferred RWD source for facilitating regulatory scientific decision‐making, and a consensus was reached regarding the necessity for pre‐protocol development before RWD/RWE utilization.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

This study has the potential to influence clinical pharmacology and translational science by shedding light on the need for policy refinements. It assesses the situational awareness of experts across various stakeholder groups in the context of the growing importance of drug lifecycle safety management and the utilization of RWD/RWE. Given the multifaceted nature of regulatory scientific environments, the study highlights the necessity for developing guidelines to assist in the selection of RWD/RWE that aligns with the fit‐for‐purpose approach.

INTRODUCTION

Real‐world data (RWD) and real‐world evidence (RWE) are significant owing to their capacity to depict the attributes of actual patients with diverse underlying medical conditions. Hence, there is a concerted effort to incorporate them as supplementary components to randomized controlled trials (RCT) in regulatory decision‐making. ¹ The US Food and Drug Administration (FDA) has established an approval‐review system to use RWD/RWE for adding new indications to drugs that have already been approved or for post‐marketing surveillance (PMS) through the “21st Century Cures Act” passed in 2016. ² The European Medicines Agency (EMA) also recommends using RWD/RWE in regulatory decision‐making to increase the efficiency of reviews for new drug new drug application (NDA). ³

Despite global attempts to leverage RWD/RWE for regulatory decision‐making throughout the drug life cycle, the application of RWD/RWE in the pre‐approval phase has been comparatively limited compared with that in the post‐approval phase. Among the cases using RWD/RWE approved by the FDA or EMA during 2016–2020, 142 cases were for PMS, whereas only 14 cases were for new drug applications or line extensions. ⁴ , ⁵ In Korea, the use of RWD/RWE has been proposed for post‐marketing safety management through the “Standard for Re‐examination of New Drugs,” etc., announced in 2020. ⁶ However, discussions on acquiring or expanding new indications coupled with the formulation of specific plans for the utilization of RWD/RWE are infrequent.

It is important to discuss and develop strategies to utilize RWD/RWE throughout the life cycle of drugs, given that the RWD/RWE utilization in the pre‐approval phase is insufficient despite its escalating necessity. Experiences and opinions of experts related to each stage of regulatory decision are crucial to establishing a systematic, specific, and feasible plan for the integration of RWD/RWE. The present study surveyed prerequisites (such as systems and infrastructure) that national institutions or industries should prepare for utilizing RWD/RWE during the life cycle of drugs, targeting experts in the relevant fields.

MATERIALS AND METHODS

Selection of survey participants

We divided the process of using RWD/RWE during the drug approval cycle into four stages, namely RWD generation, RWD provision, RWE generation, and RWE utilization, and Korean experts involved in each stage were designated as survey participants (Figure S1).

RWD generation

In this study, RWD encompassed data from the registry, electronic medical records, common data model (CDM), national health insurance claims, drug adverse event reports, patient‐reported outcomes (PRO), health surveys, social media, and pharmacy. Hence, we included clinicians, pharmacists, pharmaceutical/healthcare companies, and two national health insurance institutions (Health Insurance Review and Assessment Service and National Health Insurance) in the survey as RWD creators.

RWD provision

RWD providers are mainly RWD generators. However, additionally, since lawyers, civic groups, and patient groups may be involved in legal and ethical issues that may arise in the process of obtaining RWD, such groups were also invited for this survey.

RWE generation

Academic societies, government agencies, and pharmaceutical companies related to pharmaceutical statistics were involved in this process since RWE is generated by processing RWD with statistical techniques.

RWE utilization

Pharmaceutical companies require the use of RWE for processes, such as NDA and PMS. The Ministry of Food and Drug Safety (MFDS) reviews the dossier submitted by the companies and compares it with the RWE available at their disposal.

We have reclassified the stakeholder groups involved in each stage into the categories of Industry, Government, Academia, Legal, and Others. The ‘Industry’ category included experts responsible for regulatory affairs, market access, data management, clinical trials, and outcomes research. The ‘Government’ category included respondents from agencies such as the Ministry of Food and Drug Safety, the Korea Institute of Drug Safety and Risk Management, health technology assessment agencies, and pharmaceutical research institutions (the National Evidence‐based Healthcare Collaborating Agency and the Korea Institute for Health and Social Affairs). The ‘Academia’ category predominantly comprised professors from medical and pharmaceutical faculties who are involved in either generating RWD or conducting RWE research. The ‘Others’ category encompassed patient advocacy groups like the Korea Health Justice Union and professional organizations like the Korean Pharmaceutical Association.

Development of questionnaire

A draft survey questionnaire was prepared based on the previous study, including a literature review and stakeholder interviews. ⁷ The draft version of the survey was modified after a pilot test with five experts, and the final survey questionnaire was developed. The final version of the questionnaire, reflecting the results of the pilot survey, comprised three sections: (A) the definition and categories of RWD/RWE, (B) stages at which RWD/RWE can be introduced into the drug lifecycle, and (C) possible challenges in using RWD/RWE and their solutions (File S1).

Section A includes three questions. The first question enquired about the preference for RWD definitions among the three most common definitions of RWD, based on the guidelines issued by various national organizations. ⁸ , ⁹ , ¹⁰ The three definitions are outlined as follows: (1) decision‐support data not collected by conventional RCTs; (2) various types of medical data not collected from clinical trials by conventional interventional methods; (3) various evidence related to patient/health condition/health care delivery system collected via diverse data sources. For the second question, we enquired about the suitability of the use of RWD in drug safety management (on a one to six scale) for six typical RWD types, including registry, electronic medical data, CDM, health insurance claims data, adverse events reporting data, and patient self‐reported data. We consolidated these types of RWD via a pilot study conducted with the aim of deriving feasible RWD and reviewing the overall questionnaire structure. This pilot study involved five experts, including an epidemiologist, a medical doctor, and health economists. With the third question, we assessed the suitability of the research design for generating RWE for each utilization plan, which included the following processes: new approval, conditional approval, expansion/reduction in indications for previously approved products, and PMS management. We divided the study design into practical RCT and observational studies. Practical RCTs included large simple clinical trials (LSTs) and pragmatic clinical trials (PCTs), and observational studies included external comparator studies (i.e., historical control studies), extension studies, prospective cohort studies, retrospective cohort studies, case–control studies, and cross‐sectional studies.

Section B comprised three questions for each RWD type: (1) necessity and feasibility of RWD quality assurance and RWE generation method; (2) possibility of complementing RCT limitations; (3) suitability for use. All questions were answered using a 5‐point Likert scale, with higher scores indicating more positive responses. The RWE generation methods were analyzed with respect to the following aspects: protocols (preparation of a protocol and subsequent approval by the Minister of Food and Drug Safety prior to RWE generation), manpower (collection and management of RWD, RWE generation, RWE archiving, education, and training), institutions (designation), research methodology (validity), and utilization of personal information (non‐identification processing and protection of same patients). Limitations of RCT included a short observation period, limited study subjects (i.e., excluded patients with comorbidities, as well as the elderly and children), differences with real‐world practice, difficulty in obtaining clinical evidence (i.e., in the case of medications where the target patient population was small, or comparable treatment alternatives were absent), and ethical issues.

Section C was structured to obtain responses against potential challenges that may arise when using RWD/RWE on a Likert 5‐point scale, where 1 point represented ‘not likely to occur at all’ and a 5‐point response denoted ‘very likely to occur’. Possible challenges were presented in terms of quality and reproducibility, accessibility, use of personal information, and legal aspects. Potential threats to quality and reproducibility included difficulties in controlling confounding factors, handling of missing and omitted data, and constraints in disclosing analysis protocols and establishing standardized protocols. Accessibility‐related challenges comprised difficulties in accessing data from other institutions and linking data across institutions. Concerns regarding the use of personal information included information infringement and deidentification issues. Additionally, the lack of relevant policies and guidelines was also included as a challenge.

Survey procedure

We reached out to relevant academic societies, associations, and organizations to solicit recommendations for experts who could participate in our survey. Subsequently, we distributed the questionnaire to these recommended experts. Additionally, we distributed the survey to experts who had prior experience in participating in government‐organized RWD/RWE working groups or had conducted previous research in the field. The survey was conducted by sending emails in October 2021. After receiving the initial responses, two reminder emails were sent to secure additional responses. This study was approved by the Yonsei University Institutional Review Board (IRB No: 7001988‐202109‐HR‐1225‐03).

Analysis

The characteristics of the survey respondents were presented using descriptive statistics and frequency analysis. For questions answered on a Likert scale, we computed the mean scores and standard deviations, and ranks were provided to identify responses with high frequency. The responses of the necessity of RWD quality assurance and RWE generation method were further analyzed by differentiating according to the respondents' affiliated institutions.

RESULTS

Characteristics of survey respondents

Among the total of 46 respondents out of 96 experts (response rate: 48%), the largest number belonged to the industry (13; 28.3%), followed by 12 (26.1%) in government agencies, 11 (23.9%) in academia, 8 (17.4%) in legal circles, 4 (8.7%) in medical institutions, and 2 (4.3%) inpatient groups (Table 1). Among the 12 government respondents, five were from drug approval agencies (the Ministry of Food and Drug Safety and the Korea Institute of Drug Safety and Risk Management), three were from health technology assessment agencies (the Health Insurance Review & Assessment Service and the National Health Insurance Service), and four were from pharmaceutical research institutions (the National Evidence‐based Healthcare Collaborating Agency and the Korea Institute for Health and Social Affairs). The majority of respondents (89.1%) had experience using RWD. Among the 10 types of RWD, health insurance claims data was the most frequently used (87.0%), followed by medical records (39.1%), registry data (21.7%), and adverse drug reporting data (19.6%). The study topics using RWD with the highest frequency included ‘safety/side effect monitoring and research’ (54.3%), followed by comparative‐effect study (30.4%), economic evaluation (15.2%), epidemiological study (13.0%), and drug usage pattern analysis (13.0%).

TABLE 1.

Characteristics of the survey respondents.

Characteristics	Overall a (n = 46, 100%)	Industry (n = 13, 28.3%)	Government agencies (n = 12, 26.1%)	Academia (n = 11, 23.9%)	Legal circles (n = 8, 17.4%)	Others b (n = 7, 15.2%)
Experience of using RWD/RWE
Yes	41 (89.1%)	13 (100.0%)	12 (100.0%)	11 (100.0%)	4 (50.0%)	6 (85.7%)
No	5 (10.9%)	–	–	–	4 (50.0%)	1 (14.3%)
Work experience
Below 5 years	4 (8.7%)	1 (7.7%)	2 (16.7%)	1 (9.1%)	1 (12.5%)	1 (14.3%)
5–9 years	12 (26.1%)	1 (7.7%)	2 (16.7%)	6 (54.5%)	3 (37.5%)	1 (14.3%)
10–14 years	16 (34.8%)	5 (38.5%)	4 (33.3%)	4 (36.4%)	2 (25.0%)	2 (28.6%)
15–19 years	6 (13.0%)	3 (23.1%)	1 (8.3%)	–	1 (12.5%)	2 (28.6%)
Above 20 years	8 (17.4%)	3 (23.1%)	3 (25.0%)	–	1 (12.5%)	1 (14.3%)
RWD which participants have used
Claims data	40 (87.0%)	11 (84.6%)	9 (75.0%)	11 (100.0%)	4 (50.0%)	5 (71.4%)
Medical record	18 (39.1%)	5 (38.5%)	3 (25.0%)	5 (45.5%)	2 (25.0%)	3 (42.9%)
Registry data	10 (21.7%)	2 (15.4%)	1 (8.3%)	3 (27.3%)	1 (12.5%)	3 (42.9%)
Adverse event report	9 (19.6%)	1 (7.7%)	1 (8.3%)	4 (36.4%)	–	3 (42.9%)
CDM	4 (8.7%)	1 (7.7%)	1 (8.3%)	1 (9.1%)	–	1 (14.3%)
PRO	2 (4.3%)	1 (7.7%)	–	–	1 (12.5%)	–
Wearables data	1 (2.2%)	1 (7.7%)	–	–	–	–
PMS	2 (4.3%)	–	–	–	–	2 (28.6%)
Pharmacy data	1 (2.2%)	–	–	–	–	1 (14.3%)
DUR	1 (2.2%)	–	1 (8.3%)	–	–	–
Purpose of RWD use
Safety/side effect monitoring and research	25 (54.3%)	7 (53.8%)	4 (33.3%)	9 (81.8%)	1 (12.5%)	4 (57.1%)
Comparative‐effect study	14 (30.4%)	4 (30.8%)	3 (25.0%)	4 (36.4%)	–	3 (42.9%)
Economic evaluation	7 (15.2%)	3 (23.1%)	1 (8.3%)	1 (9.1%)	1 (12.5%)	1 (14.3%)
Epidemiological study	6 (13.0%)	2 (15.4%)	2 (16.6%)	–	1 (12.5%)	1 (14.3%)
Drug usage pattern analysis	6 (13.0%)	1 (7.7%)	1 (8.3%)	2 (18.2%)	2 (25.0%)	–
Cost analysis	3 (6.5%)	–	–	1 (9.1%)	1 (12.5%)	1 (14.3%)
Risk factor analysis	3 (6.5%)	1 (7.7%)	–	1 (9.1%)	–	1 (14.3%)
Others c	11 (23.9%)	3 (23.1%)	–	2 (18.2%)	3 (37.5%)	3 (42.9%)

Abbreviations: CDM, common data model; DUR, drug utilization review; PMS, post‐marketing surveillance; PRO, patient‐reported outcome; RWD, real‐world data; RWE, real‐world evidence.

^a Sum of the number of institution categories exceeds the total respondents because they can be classified into multiple institutions.

^b Others include patient group and civic group.

^c Others include willingness to pay research, quality of life analysis, treatment pattern analysis, validation of survey result, drug repositioning, making control group for one‐arm clinical trials, pragmatic clinical trials, claims data analysis for litigation, anonymization research methodology, and drug policy evaluation.

Agreement with the definition of RWD

More than half (54.3%) of the respondents agreed with the definition of RWD as ‘various data related to patients/health conditions/health care delivery systems collected using diverse data sources’, and approximately one‐third (34.8%) agreed with ‘a comprehensive term for various types of medical data not collected from clinical trials by interventional methods’ (Figure 1). The results were similar when respondents were classified by the type of institution.

FIGURE 1.

Agreement with the definition of real‐world data (RWD).

Types of suitable study designs for RWD and RWE across the drug cycle

Although suitability was calculated by assigning scores in the reverse order (e.g., six points for the first place), the registry was selected with a score of 4.93 as the most suitable source of RWD. It was followed by CDM (4.40), insurance claims data (3.98), and medical records data (3.34). Registries were considered highly suitable by those in government agencies (5.58) and medical institutions (5.75), and CDM received relatively high scores from the respondents in the industry (4.54). The suitability of PRO data was evaluated as low, especially by the respondents in medical institutions (1.25) (Table 2).

TABLE 2.

Types of RWD suitable for use in drug safety regulation.

Types of RWD	Overall (rank)	Score according to affiliation
		Industry	Government agencies	Academia	Medical institutions	Others a
Registry data	4.93 (1)	4.15	5.58	4.50	5.75	5.33
Medical record	3.34 (4)	3.92	3.00	3.20	3.25	3.00
CDM	4.40 (2)	4.54	4.50	4.20	3.75	4.67
Claims data	3.98 (3)	4.08	3.75	4.70	3.75	3.36
Adverse event report	2.87 (5)	2.62	2.83	3.00	3.25	3.18
PRO	1.60 (6)	1.69	1.64	1.56	1.25	1.45

Abbreviations: CDM, common data model; PRO, patient‐reported outcome.

^a Others include patient group and civic group.

Overall, several respondents indicated that RWE is suitable for PMS management and for expanding and reducing the indications of licensed products, regardless of the study design. However, with the exception of external comparator studies (50%), respondents predominantly regarded the use of all clinical or observational studies for obtaining new drug applications as unfavorable. For conditional approval, more than half of respondents indicated suitability of external comparator studies (84%), PCTs (73%), LSTs (63%), and extension studies (52%), whereas retrospective cohort studies (28%), case–control studies (12%), and cross‐sectional studies (3%) were considered unsuitable. For indication expansion or reduction, PCTs (84%), extension studies (77%), external comparator studies (75%), and both prospective (75%) and retrospective (63%) cohort studies were agreed upon as appropriate strategies. Cross‐sectional studies were the exception, which were associated with low agreement. For PMS management, all methods, except for the external comparator study, were considered highly suitable for generating RWE (Table 3).

TABLE 3.

RWE generation suitability of each research design by RWE utilization plan.

Study design		RWE generation suitability
		New drug application	Conditional approval	Expansion or reduction in indication	Post‐marketing surveillance
Clinical trials	Large simple trials	34%	63%	70%	71%
	Pragmatic clinical trials	30%	73%	84%	80%
Observational studies	External comparator studies	50%	84%	75%	50%
	Extension studies	11%	52%	77%	84%
	Prospective cohort studies	11%	43%	75%	91%
	Retrospective cohort studies	7%	28%	63%	95%
	Case–control studies	12%	24%	48%	83%
	Cross‐sectional studies	3%	12%	27%	91%

Necessity and feasibility of the processes required to assure the quality of RWD and generate RWE

Participants responded that it is both necessary and feasible to develop protocols prior to collecting RWD and generating RWE, regardless of the type of RWD. The registry was regarded as highly necessary (4.75) and feasible (4.71) for ex‐ante protocol development. However, the perceived necessity was relatively low (3.05–3.40) for designating specific institutions in response to legal or regulatory requirements for RWD collection and RWE generation, particularly for adverse event reporting data and patient‐reported outcomes. A high necessity (3.90–4.71) was recorded for privacy protection, such as patient anonymization and protection of identical patient information among multiple RWD sources. However, respondents perceived that the feasibility of implementing privacy protection was low (1.38–2.14) (Table 4).

TABLE 4.

Necessity and feasibility of the way to assure the quality of RWD and generate RWE.

The way to assure the quality of RWD and generate RWE		Necessity (1–5 points)						Feasibility (1–5 points)
		Registry data	Medical record	CDM	Claims data	Adverse event report	PRO	Registry data	Medical record	CDM	Claims data	Adverse event report	PRO
Protocol	Protocols must be created before collecting RWD and generating RWE	4.75	4.52	4.57	4.40	4.07	4.26	4.71	4.07	4.21	4.38	3.93	3.67
	Protocols must be approved by the minister of MFDS	3.98	3.62	3.57	3.38	3.26	3.60	3.93	3.29	3.48	3.48	3.36	3.02
Human resources	Assigning human resources to the collection and management of RWD	4.45	4.24	4.45	3.98	3.95	3.95	4.12	3.76	3.90	3.90	3.74	3.36
	Principal investigators and researchers should be selected from those who have sufficient experience to create RWE	4.67	4.38	4.48	4.33	4.05	4.10	4.29	3.95	3.90	4.24	3.64	3.45
	Assigning human resources for RWE‐related records and data storage	4.31	3.95	4.00	3.81	3.83	3.86	4.14	3.76	3.86	3.88	3.60	3.33
	Training RWD/RWE‐related human resources	4.67	4.33	4.45	4.32	4.10	4.17	4.45	3.98	4.12	4.19	3.83	3.69
Institution	Creating RWE in institutions with sufficient facilities designated by laws and regulations	3.40	3.21	3.31	3.31	3.05	3.07	3.31	3.19	3.19	3.29	3.12	2.76
Research methods	When creating an RWE, it is necessary to define the research method to obtain internal validity that is as valid as randomization in RCT	3.81	3.69	3.79	3.83	3.12	3.17	3.64	3.24	3.43	3.64	3.00	2.79
Personal information utilization	Protecting personal information by de‐identifying data	4.71	4.69	4.48	4.67	4.25	4.62	1.45	1.83	1.45	1.38	1.57	1.86
	Protecting data in identical patients between RWD	4.31	4.24	4.17	4.21	3.90	4.14	1.81	2.12	2.00	1.79	2.05	2.14

According to the results of a subgroup analysis that categorized respondents based on their affiliated institutions into government agencies (n = 12), industry (n = 13), and academia (n = 11), the development of a pre‐protocol was found to vary according to the type of RWD among the response groups (File S2). Respondents working in government agencies indicated a high necessity for the development of pre‐protocols when analyzing registry data (mean = 4.91), spontaneous adverse event reports (4.73), and patient‐reported outcomes (5.0). In contrast, respondents from the industry sector perceived a lower necessity for this (4.69, 4.08, 3.85, respectively). Regarding the need for pre‐protocol approval by the director of the regulatory agency, irrespective of the RWD type, respondents from government agencies reported a higher necessity compared with those affiliated with the industry. Regarding the necessity of the “Personnel,” “Institution,” “Research Methods” sections, respondents from government agencies generally indicated a high level of necessity, while those affiliated with the industry perceived it as relatively less necessary. Academia respondents, in particular, provided lower scores on the necessity of legally designating institutions that generate RWE, regardless of the type of RWD, with scores ranging from 2.36 to 2.91, which were comparatively lower than those from government agencies (3.36–3.82) and industry (2.36–3.62).

Possibility of complementing limitations of RCT by RWD type

Respondents indicated that the registry (4.08) is likely to circumvent the limitations of a short observation period of RCT. CDM (4.23), medical records (4.00), and claims data (4.00) were considered highly likely to supplement the limitations of RCT with a limited study population. CDM (4.23) was considered capable of overcoming the limitation posed by the RCT not being based on a real‐world medical environment (Table 5).

TABLE 5.

Possibility of complementing limitations of RCT by RWD type.

Limitations of RCT	Possibility of complement limitations of RCT (1–5 points)
	Registry data	Medical record	CDM	Claims data	Adverse event report	PRO
Short observation period: hard to get long‐term effects and safety	4.08	3.65	3.70	3.83	2.88	2.58
Restricted study population: The elderly, children, or patients who have comorbidity are excluded from study population	3.93	4.00	4.23	4.00	3.15	2.78
Difference from the actual clinical environment: It is carried out in a controlled environment different from the actual clinical field	3.98	4.08	4.23	3.63	3.50	3.15
Difficulty in generating clinical evidence: If there are few patients or there is no treatment alternative, it is difficult to generate clinical evidence through RCT	3.93	3.65	3.78	3.55	2.63	2.40
Ethical issue: Administration of medicines that are expected to have better therapeutic effects only for some patients can cause ethical problems	3.55	3.75	3.95	3.65	3.33	2.95

Suitability of RWD/RWE utilization plans in the pharmaceutical life cycle by RWD type

When the RWD/RWE utilization plans were divided into new approval, conditional approval, changing approval (including expansion or reduction indications), and PMS in the drug approval process, the suitability of the RWD/RWE use in PMS was considered high regardless of the RWD type (3.38–4.10). The suitability score ranges for the use of RWD/RWE for new and conditional approval were low at 1.59–3.28 and 1.71–3.72, respectively, with the highest score reported for the registry and the lowest for PRO. Registry source exhibited high suitability scores for all approval processes (Table 6).

TABLE 6.

Suitability for RWD/RWE Utilization plans in the pharmaceutical life cycle by RWD type.

RWD/RWE utilization plans		Suitability for utilization (1–5 points)
		Registry data	Medical record	CDM	Claims data	Adverse event report	PRO
Approval	New drug approval	3.28	2.74	2.77	2.18	1.67	1.59
	Satisfying country‐specific requirements when applying for approval of approved drugs in another country	3.49	3.00	3.13	2.51	1.97	1.74
Conditional approval	Conditional approval	3.72	3.26	3.36	2.90	2.00	1.71
Changing approval	Expansion or reduction in application scope/target and indication of approved products	3.92	3.56	3.72	3.18	2.39	2.05
	Addition of efficacy information to approved drug labels	3.95	3.64	3.79	3.13	2.38	2.23
Post‐marketing surveillance	Comparing therapeutic benefits and risk when serious adverse events are reported	4.10	3.95	4.08	3.51	3.97	3.38
	Monitoring adverse events	4.08	3.92	4.03	3.46	4.56	3.67

Possible challenges when using RWD/RWE

Respondents expressed concerns about the quality, reproducibility, and accessibility of RWD/RWE. Particularly, they responded that PRO was more likely to be biased compared with RCTs due to unadjusted confounding variables, and they anticipated a higher prevalence of missing data. The registry was perceived as the most reliable RWD type in terms of data collection and analysis protocols (2.52, with lower scores indicating fewer perceived problems) and was credited for its reproducibility (2.59). The CDM was valued for its minimal privacy issues (2.45) and standardized nature (2.66). Claims data were lauded for privacy protection (2.66), data standardization (2.73), and better accessibility (2.80) compared with other RWD sources (Table 7).

TABLE 7.

Possible problems when using RWD/RWE.

Possible problems when using RWD/RWE		Possibility of problems (1–5 points)
		Registry data	Medical record	CDM	Claims data	Adverse event report	PRO
Quality and reproducibility	Confounding factors are difficult to measure and control, resulting in greater bias than RCTs	3.41	3.68	3.57	4.05	4.50	4.80
	Frequent missing and dropout data	3.18	3.66	3.50	3.36	4.34	4.59
	Difficult to trust as data collection and analysis protocols are not transparently disclosed	2.52	3.34	2.93	3.00	3.34	3.98
	Poor reproducibility and validity due to no standardized study protocol established	2.59	3.32	3.00	3.23	3.39	4.05
Accessibility	Difficult to access RWD/RWE created by other organizations	3.98	4.77	3.20	2.80	3.25	4.18
	It is difficult to link RWD/RWE from other institutions because RWD collection method, study design, statistical analysis method, etc. by institution are not standardized	3.30	4.48	2.66	2.73	3.09	4.05
Personal information utilization	Privacy issues may arise	3.20	4.00	2.45	2.66	2.80	3.55
	It is difficult to link the data of the same patients between RWD due to the deidentification of personal information	3.02	3.57	3.39	3.05	3.59	3.55
Legal basis	No policies have been established to support the generation and utilization of RWD/RWE	3.56	3.88	3.51	3.14	3.09	3.72
	No guidelines have been developed for the generation and utilization of RWD/RWE	3.72	4.00	3.81	3.37	3.42	4.05

DISCUSSION

This study is significant as it was the first to explore the definition of RWD and the challenges associated with RWD/RWE, targeting stakeholders across the entire cycle of drug approval and regulatory decision‐making. Given that approximately 90% of the respondents had extensive experience with RWD/RWE, this survey aptly surmised the perception of Korean experts on RWD/RWE and identified preparatory needs for the utilization of RWD/RWE. A few studies reported the current state of using RWD/RWE in the regulatory decision‐making processes for pharmaceuticals in East Asian countries such as Japan and China. ¹¹ , ¹² , ¹³ Nevertheless, they do not delve into surveys or assessments of the perspectives and opinions of experts specifically regarding data governance for RWD and RWE. This study addresses a gap in the literature by conducting an expert survey, particularly targeting experts in Korea. We explored the definition of RWD, the essential elements for deriving RWE, and considerations related to data integrity. By focusing on experts in one country, the survey aimed to provide substantial and in‐depth information on the practical utilization of RWD/RWE in regulatory decision‐making processes.

Though the common definition of RWD encompasses the concepts of “various data” and “healthcare‐related data,” the majority of respondents acknowledged and agreed that RWD includes a diverse range of healthcare‐related data. However, unlike the definition (various types of medical information data other than data collected from clinical trials by conventionally interventional methods) provided by the MFDS, ¹⁴ the FDA and EMA definition does not incorporate the concept of “not collected from clinical trials,” ⁸ necessitating a consensus on whether data obtained outside clinical trials should be defined as RWD. Furthermore, discussions are needed on how to define clinical trials, including classifying data from phase IV clinical trials after new approval or various types of clinical trials (i.e. LSTs, PCTs).

Registry data were identified as the most suitable type of RWD followed by CDM, claims data, and medical record data, respectively. However, the suitability score of drug adverse event reporting data was 2.87 points (out of 6 points) and that of the PRO data was 1.60 points, indicating the lack of reliability of data directly reported by patients. Over 60% of respondents (including duplicate responses) had experience using claims data, whereas approximately 10% and <5% of respondents had experience using a registry and CDM, respectively. Thus, the results may also reflect expectations rather than experience‐based assessments for registries and CDM. Given that several regulatory agencies have already recognized the high suitability of registries and have issued guidelines for their use in regulatory science decision‐making, ¹⁵ , ¹⁶ , ¹⁷ it can be inferred that the respondents likely held high expectations for registries in a similar context. ¹⁵ For instance, the Agency for Healthcare Research and Quality in the U.S. unveiled the third edition of the “Registries for Evaluating Patient Outcomes: A User's Guide” in 2020, ¹⁶ and the EMA also released the “Guideline on Registry‐Based Studies” in 2021. ¹⁷

Respondents, regardless of the type of RWD, felt that the suitability of using RWD/RWE for new approval was limited. They asserted that only external comparator studies for single‐arm clinical trials that could only be conducted in restricted conditions could use RWD for new drug approval. ¹⁸ , ¹⁹ Conversely, most respondents stated that the use of RWD/RWE is appropriate at the PMS stage, particularly, in the cohort study designs (prospective: 90.9%; retrospective: 95.3%). This indicates that different levels of RWD and study designs are appropriate for different licensing stages, and different research designs should be used even within the same licensing stage.

In the future, when utilizing RWD/RWE for regulatory decision‐making, such as during drug licensing and safety management, it is implied that materials and study methods should be prepared that are suitable for the purpose. ²⁰ Furthermore, according to the recent report on RWD/RWE usage experience from 2021 to 2023 published by the EMA, ²¹ it is possible to use extended RWD not limited to a single source by connecting multiple RWD sources or linking RWD across countries. The report also stated that the use of RWD/RWE extends beyond the realm of drug approval management, offering significant insights into the natural history of rare diseases like spinal muscular atrophy, as well as pandemic diseases, such as COVID‐19. ²¹ Meanwhile, the increase in studies replicating RCTs with RWD (RCT emulation and replication study) suggests that the application of RWD/RWE may extend beyond the categories defined in this study. ²² , ²³ Additionally, there will be a need for a detailed discussion on whether RWD/RWE can complement and even potentially replace RCTs in which cases. ²⁴

To ensure the quality of RWD and the generation of valid RWE, the development of a preliminary protocol was highlighted as a necessary and feasible solution. However, the adoption of measures resembling those of Good Clinical Practice guidelines, ²⁵ such as protocol approval by regulatory authorities or the assignment of specific institutions and individuals, were not deemed as highly necessary or feasible. This underscores the imperative for a consensus on the necessities and feasible preparations for employing RWD/RWE in multifaceted processes, ²⁶ especially for NDA. By prioritizing items with high agreement among respondents, it is required to lay a foundation for crafting strategies to utilize RWD/RWE in actual regulatory decision‐making concerning NDA. For instance, many respondents concurred on assigning personnel to manage registries, electronic medical records, and CDM to ensure the quality of RWD, reflecting a notable consensus on the matter. While there is significant demand for personal information across all RWD types, its practical application appears limited. In Korea, efforts, like the introduction of the Health and Medical Data Utilization Promotion Act, ²⁷ are in motion to exploit various consolidated RWD sources. Nonetheless, there is also a voiced need for amplified personal data protection. Upon conducting additional analyses by differentiating the stakeholder groups, we found discrepancies in opinions regarding the necessity of measures to ensure the quality of RWD and to generate RWE (File S2). Respondents affiliated with government agencies generally indicated a high necessity for regulatory items, while those from the industry responded with a relatively lower necessity. Therefore, to effectively utilize RWD/RWE in regulatory decision‐making in the future, active discussions for consensus among stakeholders are necessary.

To the best of our knowledge, expert surveys or studies on the use of RWD and the derivation of RWE are not widely conducted. Solà‐Morales et al. (2023) conducted a Delphi survey with 24 European experts, including health policymakers, Health Technology Assessment (HTA) experts, and hospital managers for a data governance checklist for RWD and RWE. ²⁸ The aim of the survey was to gather consensus on best practices and considerations related to the governance of RWD and the generation of RWE. Another relevant study is by Facile et al. (2022), which conducted a Delphi survey to explore the applicability of the Clinical Data Interchange Standards Consortium as a data quality management tool for submitting RWD to regulatory decision‐making processes. ²⁹ The survey involved a total of 49–66 participants who were recruited globally from diverse regions, including the United States, Asia, Europe, Africa, and others, and from various sectors such as academia, government, research institutions, non‐governmental organizations, international organizations, and others. This study has several limitations. First, as the use of RWD/RWE is rapidly increasing, the respondents in this study may not represent all stakeholders. Hence, caution is advised during the interpretation of the results. However, the study carries significance as an initial RWD study in Korea, given the high number of respondents, the wide experience with RWD/RWE among the respondents, and the inclusion of respondents from diverse groups. Second, it is plausible that responses stemming from experiential insights and those based on expectations may have been mixed in the results, depending on the types and uses of RWD by the respondents. For instance, few respondents had experience with registries or CDM, and some respondents had a limited understanding of the use RWD/RWE for new approvals, such as an external comparator. Third, this study aimed to investigate the overall suitability of RWD in the decision‐making process of pharmaceutical regulations. Therefore, there is a limitation in applying the survey results to specific situations for each product, type of RWD, or specific regulatory decision types. To gain a more specific understanding of stakeholders' opinions in these nuanced situations, additional research is necessary. This may involve developing scenario‐specific analyses and conducting surveys or in‐depth interviews accordingly. Fourth, in conditional approval, the use of RWD/RWE can take various forms, such as replacing a comparator, supplementing a comparator (e.g., using a hybrid design), validating an end point, or serving as an internal control in an observational study. However, this survey did not delve into specific details regarding these applications. Finally, further delineation of RWD types including CDM may enhance our understanding of the usefulness of RWD for regulatory decision‐making. CDMs, such as OMOP‐CDM, Sentinel, and PCORnet, offer various frameworks for standardizing and harmonizing data from electric medical records.

In conclusion, it is essential to elucidate the definitions of clinical trials or interventional trials for a more precise definition of RWD. Moreover, it is necessary to consider whether to include data derived from these trials within the concept of RWD. Our findings confirmed that a fit‐for‐purpose design is important because the suitability of RWD and study design varied across different types of studies. Particularly, thoughtful deliberation with stakeholders is required to apply RWD/RWE for new drug approvals. The registry was evaluated as the most suitable RWD source and presented relatively few issues, indicating a need for future discussions on how to implement quality management of the registry in regulatory decision‐making.

AUTHOR CONTRIBUTIONS

H.L., H.A., and S. K. wrote the manuscript. H.L., and E. H. designed the research. H.L., H.A., S.K., H.Y.K., and E.H. performed the research. H.L., H.A., and S.K. analyzed the data.

FUNDING INFORMATION

This study was funded by the Korea Ministry of Food and Drug Safety (MFDS, grant number: 21183SilImsang54, RS‐2023‐00214944).

CONFLICT OF INTEREST STATEMENT

The authors declared no competing interests for this work.

Supporting information

File S1.

File S2.

Figure S1.

Lee H, Ahn H‐S, Kwon S, Kang H‐Y, Han E. Expert survey on real‐world data utilization and real‐world evidence generation for regulatory decision‐making in drug lifecycle in Korea. Clin Transl Sci. 2024;17:e13801. doi: 10.1111/cts.13801