Evaluating effectiveness of risk minimization through healthcare professional surveys: A review of post‐authorization studies, 2022–2024

Abstract

Aims

Risk minimization measures (RMMs) are key regulatory tools to ensure safe medicinal product use. Regulatory guidelines recommend pre‐specifying the targets, or ‘thresholds’, for assessing RMM effectiveness. We aimed to review recent industry‐sponsored post‐authorization safety studies (PASS) that evaluated the effectiveness of RMMs, focusing on the use and justification of predefined thresholds.

Methods

We searched the Heads of Medicines Agencies‐European Medicines Agency Catalogue to identify PASS evaluating the effectiveness of RMM through knowledge‐based surveys of healthcare professionals. A total of 17 eligible studies in 2022–2024 were included in the descriptive analysis.

Results

All studies used cross‐sectional designs, and 70.6% were Category 3 PASS. Only 29.4% evaluated RMM effectiveness within the guideline‐recommended time frame (12–24 months) following the launch and regulatory implementation of RMMs. Nearly all studies (16 of 17, 94.1%) predefined effectiveness thresholds. Overall, thresholds ranged from 60% to 80%, with the majority (88.2%) being set at 80%. Thresholds for assessing awareness, knowledge, utilization and behaviours were set at 80%, 80%, 60%–80% and 75%–80%, respectively. However, only eight studies (47.1%) provided rationales for threshold determination. Among them, six studies justified thresholds based on success rates applied in comparable studies or regulatory guidance, while two relied on subjective criteria without specifying prior knowledge, experience or established scientific evidence.

Conclusion

Our findings suggest that although most industry‐sponsored PASS evaluating RMM effectiveness included predefined thresholds, these were often applied to the knowledge indicator rather than awareness, utilization and behaviour indicators. Notably, only about half of the studies provided clear justifications for threshold selection. Clear success threshold justifications are essential for interpreting RMM effectiveness.

What is already known about this subject

• Good pharmacovigilance practices recommend a priori specification of the ‘thresholds’ for evaluating the effectiveness of risk minimization measures (RMMs).

• A previous analysis of post‐authorization safety studies (PASS) conducted in 2016–2021 found that only half of the studies used predefined thresholds.

What this study adds

• Of the included PASS, 94% evaluated RMM effectiveness through knowledge‐based survey predefined thresholds, with 80% being the most commonly selected threshold.

• Only half of the PASS provided rationales for selecting thresholds.

• Robust justification and indicator‐specific thresholds are needed when designing survey‐based PASS to improve interpretability and regulatory utility of RMM effectiveness evaluations.

1. INTRODUCTION

Risk management is a continuous iterative process that marketing authorization holders undertake to ensure the safe and effective use of medicinal products throughout their life cycle. ¹ , ² This management includes the use of risk minimization measures (RMMs), such as summary of product characteristics, package leaflet, educational materials or safety advice tools, that are designed to prevent or reduce the occurrence of adverse reactions or to mitigate their severity or impact on patients. ³

A central component of risk minimization is the evaluation of the effectiveness of RMMs. For example, monitoring the effectiveness of RMMs is an explicit requirement under the European Union (EU) pharmacovigilance legislation. ⁴ Similarly, the US Food and Drug Administration (FDA) considers the evaluation of risk minimization action plan (RiskMAP) effectiveness to be important and recommends periodical evaluation of its effectiveness after implementation. ¹ Both EU and US regulations stipulate that success criteria or ‘thresholds’ need to be specified as a priori and clearly defined in the study protocol, with justifications for the choice of indicators for success. ¹ , ³ These thresholds represent quantifiable targets (e.g., percentages) that determine whether an RMM is considered effective in achieving its objectives. Ideally, these thresholds should be grounded in empirical evidence, such as historical benchmarks from similar populations or interventions, therapeutic contexts or societal acceptability. Yet despite regulatory emphasis on threshold‐based evaluation, ⁵ the transparency and consistency in how thresholds are selected and justified remain unclear in PASS.

We reviewed industry‐sponsored PASS that evaluated the effectiveness of RMMs through knowledge‐based surveys among healthcare professionals between 2022 and 2024. We focused on knowledge‐based surveys because they are the primary tool for assessing awareness, understanding or intended actions related to risk communications. ⁶ , ⁷ , ⁸ We summarized general principles and methodologies related to RMM assessment, emphasizing the thresholds and rationales applied to determine the effectiveness of such measures as well as other important study characteristics, such as recruitment methods and time points for evaluation after implementation.

2. METHODS

2.1. Database

We performed a descriptive, cross‐sectional study based on industry‐sponsored PASS registered in the Heads of Medicines Agencies‐European Medicines Agency (HMA‐EMA) Catalogue of Real‐World Data studies. ⁹ This repository captures important study features and publishes protocols and results of non‐interventional PASS, aiming to promote information exchange and facilitate collaboration among regulators, industry companies and researchers and to ensure compliance with EU pharmacovigilance legislation requirements. ⁹ So far, it has been extensively used in prior reviews. ⁶ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵

2.2. Inclusion and exclusion criteria

We focused on studies that evaluated the effectiveness of RMMs among healthcare professionals using knowledge‐based survey studies, with quantitative measurements of indicators, such as dissemination, knowledge and behaviour outcomes. Thus, we included PASS that were (1) registered between 01 January 2022 and 31 December 2024; (2) evaluated RMM effectiveness; (3) included healthcare professionals in the study population; and (4) used surveys as the primary data source. We excluded studies if they (1) did not evaluate the effectiveness of risk minimization (e.g., drug utilization study or effectiveness study of a product); (2) lacked information on threshold selection due to unavailability of study protocols and reports; (3) included only patients; and (4) exclusively used qualitative methods.

2.3. Study selection

All studies that were first registered between 2022 and 2024 in the HMA‐EMA Catalogue were accessed on 06 January 2025. The screening process was conducted by two independent reviewers (K.Z. and J.Y.E.), and discrepancies were discussed and resolved. Study titles were first screened using keywords including “RMM”, “risk minimisation”, “risk minimization”, “survey”, “effectiveness”, “utilization”, “utilisation”, “behaviour”, “behavior”, “knowledge”, “awareness”, “education”, “physician”, “healthcare professional” and “health care professional”. If the study titles contained any such keywords, we then reviewed study details and documents such as protocols or study reports.

2.4. Study outcomes

Our primary outcome was the quantitative threshold and scientific rationale used to evaluate the effectiveness for a given PASS. The thresholds were summarized by measures of effectiveness: awareness, utilization, knowledge and behaviour. Awareness indicators assess if healthcare providers are aware of having received educational materials. Utilization indicators assess if participants have used RMMs, such as reading educational materials and using healthcare professional checklists. Knowledge indicators refer to healthcare providers' knowledge of risks intended by RMMs. Behaviour indicators measure healthcare professionals' self‐reported behaviours related to risk minimization intended by RMMs.

We also characterized, for a given study, whether or not the findings evaluating the effectiveness of the RMMs were considered conclusive or inconclusive. To do so, we used the abstract of the study report contained within the HMA‐EMA Catalogue. An example of conclusive study results is as follows: ‘Risk minimization measures are effective with respect to physicians' awareness of risks and less effective with respect to physicians' knowledge’.

2.5. Data extraction and analysis

Two independent reviewers (K.Z. and J.Y.E.) extracted data, and discrepancies were discussed and resolved. We extracted data on the magnitude and rationale of predefined thresholds determining success. We also extracted study characteristics, including study design, study region, PASS category, indication for the product, indicators for effectiveness measurements, sample size, source of recruitment list and time points for evaluation (implementation of RMMs and start and end of the effectiveness study) from study protocols and abstracts of the study reports. We used descriptive statistics to summarize the extracted data.

3. RESULTS

3.1. Study selection

We identified 720 records of planned, ongoing or completed PASS between 2022 and 2024. After title screening using keywords, we included 180 studies for further screening. We excluded studies that were unrelated to RMMs (n = 156), qualitative only (n = 1), focused exclusively on patients (n = 2) and lacking information on threshold selection due to unavailability of protocols and reports (n = 4), leaving 17 studies for the final review (Figure 1).

FIGURE 1.

Flow chart of selection of included studies. Abbreviations: DUS, drug utilization study; HMA‐EMA, Heads of Medicines Agencies‐European Medicines Agency; RMM, risk minimization measure; RWD, real‐world data.

3.2. Study characteristics

Table 1 summarizes the study characteristics and details are presented in Table S1. All studies employed a cross‐sectional study design, with 29.4% being multi‐wave surveys and 52.9% being conducted in multiple countries. Most studies (70.6%) were Category 3 PASS. About half (52.9%) of the studies evaluated the effectiveness for healthcare professional guide/brochure, 35.3% for patient card, 23.5% for healthcare professional checklist and 11.8% for routine measures (e.g., summary of product characteristics). The most common effectiveness indicator was knowledge (100%), followed by awareness (64.7%), utilization (58.8%) and self‐reported behaviours (23.5%).

TABLE 1.

Characteristics of included studies (N = 17).

	N	%
Study design
Single‐wave, cross‐sectional survey	12	70.6
Multi‐wave, cross‐sectional survey	5	29.4
Study region
Single country	8	47.1
Multi‐country	9	52.9
Post‐authorization safety study category
Categories 1 and 2	0	0
Category 3	12	70.6
Non‐European Union risk management plan only a	5	29.4
Indication for the product
Oncology	6	35.3
Bone disease	2	11.8
Infection	1	5.9
Haematologic disease	1	5.9
Gastrointestinal disease	1	5.9
Endocrine disease	1	5.9
Neurologic disease	1	5.9
Autoimmune disease	1	5.9
Other b	3	17.6
Type of risk minimization measure
Routine measure (summary of product characteristics)	2	11.8
Healthcare professional guide/brochure	9	52.9
Healthcare professional checklist	4	23.5
Patient card	6	35.3
Direct healthcare provider communication	1	5.9
Risk evaluation and mitigation strategy	3	17.6
Other (including unspecified educational materials)	3	17.6
Type of indicators for risk minimization measure effectiveness
Awareness	11	64.7
Utilization	10	58.8
Knowledge	17	100
Behaviours	4	23.5
Sample size
10–100	5	29.4
101–200	5	29.4
201–400	6	35.3
401–600	1	5.9
Source of recruitment list
Sponsors' distribution list of risk minimization measure	5	29.4
Healthcare provider panel and professional healthcare databases	6	35.3
Both	3	17.6
Not available	3	17.6
Year of start of data collection
Before 2022	2	11.8
2022	4	23.5
2023	5	29.4
2024 and after	6	35.3
Duration of study (start to end of data collection)
Less than 6 months	7	41.2
6–12 months	2	11.8
Over 12 months	5	29.4
Unknown	3	17.6
Time interval from risk minimization measure implementation to end of data collection
12 to <18 months	3	17.6
18–24 months	2	11.8
Over 24 months	2	11.8
Not available	10	58.8
Pilot study
Yes	9	52.9
No/unknown	8	47.1

^a If the PASS has been requested only by a regulator from outside the European Union.

^b Including yellow fever vaccine (n = 1), vaginal atrophy (n = 1) and chronic iron overload due to frequent blood transfusions (n = 1).

About a third (35.3%) of the studies recruited participants from professional society lists and congresses, and 29.4% targeted those on the distribution list of RMMs, while 17.6% used both methods. Approximately half of the studies (52.9%) had study duration of 12 months or less. Only 29.4% of the studies reported the completion of data collection within 24 months following the implementation of RMMs. About half (52.9%) of the studies reported survey pretesting, such as the use of a pilot study validating survey questions among targeted participants.

3.3. Thresholds for indicators evaluating RMM effectiveness

Sixteen of 17 studies (94.1%) predefined thresholds of RMM effectiveness, and a threshold of 80% was most commonly used for all indications (Table S2). Nearly all studies (16/17, 94.1%) predefined an effectiveness threshold for the knowledge indicator, followed by indicators for behaviours (3/4, 75.0%), utilization (6/10, 60.0%) and awareness (5/11, 45.5%) (Figure 2). Thresholds for assessing knowledge, awareness, utilization and behaviours were set at 60%–80%, 80%, 80% and 75%–80%, respectively; 80% was the most common effectiveness threshold set for all the indicators (Figure 2).

FIGURE 2.

Thresholds for defining the effectiveness of risk minimization measures for each indicator. Column percentages were calculated with the number of studies using each threshold (e.g., 80%, 70%, 60% or not defined) for an indicator as the numerator and the total number of studies measuring that indicator as the denominator (i.e., n = 11 for awareness, n = 10 for utilization, n = 17 for knowledge and n = 4 for behaviour indicators).

3.4. Rationales for thresholds of RMM effectiveness

The distribution of rationales for effectiveness thresholds is presented in Table 2. Half of studies (9/17, 52.9%) did not specify the rationales for threshold selection, while two studies (11.8%) acknowledged that their thresholds were subjective and not based on prior knowledge, experience or established scientific criteria. The remaining six studies (6/17, 35.3%) set thresholds in accordance with success rates that were applied in comparable studies or regulatory guidance. Among these six studies, three selected thresholds based on previous studies approved by regulatory authorities, one based on success rates applied in similar surveys approved for other treatments, one based on previous surveys of RMM effectiveness conducted by the sponsor and one aligned with regulatory guidance.

TABLE 2.

Thresholds for the knowledge indicator defining the effectiveness of risk minimization measures and rationales for the threshold selection.

Rationales	Overall	Thresholds
		80%	70%	60%	None
N	17 (100)	14 (100)	1 (100)	1 (100)	1 (100)
Objective thresholds, based on	6 (35.3)	5 (35.7)	0 (0)	1 (100)	0 (0)
Previous studies approved by regulatory authorities	3 (17.6)	2 (14.3)	0 (0)	1 (100)	0 (0)
Similar risk minimization measure effectiveness surveys approved for other treatments	1 (5.9)	1 (7.1)	0 (0)	0 (0)	0 (0)
Previous sponsor‐conducted risk minimization measure effectiveness surveys	1 (5.9)	1 (7.1)	0 (0)	0 (0)	0 (0)
Regulatory guidance	1 (5.9)	1 (7.1)	0 (0)	0 (0)	0 (0)
Subjective thresholds a	2 (11.8)	2 (14.3)	0 (0)	0 (0)	0 (0)
Unspecified	9 (52.9)	7 (50.0)	1 (100)	0 (0)	1 (100)

^a Subjective thresholds not based on prior knowledge, experience or established scientific criteria.

3.5. Outcomes of RMM effectiveness in completed studies

Nine of 17 studies had final reports at the time of the present study, which allowed final interpretation of RMM effectiveness (Table 3). Among the nine finalized studies, 66.7% made conclusions based on the predefined thresholds. Among seven studies that concluded the effectiveness of RMMs with respect to risk knowledge, six (85.7%) were based on predefined thresholds (e.g., one study found that results for the knowledge indicator reached the predefined threshold and concluded RMMs being determined to be effective). Due to the lack of information in the abstracts of the study report, we were unable to determine whether the conclusion was based on predefined thresholds or not for one conclusive study and for two studies that were inconclusive (i.e., they did not reach a clear conclusion whether the RMMs were considered effective).

TABLE 3.

Summary of results of finished studies (N = 9).

Conclusions per predefined thresholds	N	Conclusiveness in terms of effectiveness of risk minimization measures
		Conclusive	Inconclusive
Total	9 (100)	7 (100)	2 (100)
Yes	6 (66.7)	6 (85.7)	0 (0)
Unknown a	3 (33.3)	1 (14.3)	2 (100)

^a Information on whether the conclusion was based on a predefined threshold or not was unavailable in the abstract of the study report.

4. DISCUSSION

Despite the critical role that RMMs play in optimizing drug safety, little is known regarding the thresholds that are set to define whether or not a given risk minimization approach is deemed effective by pharmaceutical manufacturers and regulators. We reviewed the use and justifications of predefined thresholds in industry‐sponsored PASS evaluating the effectiveness of RMMs through knowledge‐based surveys in healthcare professionals between 2022 and 2024. While nearly all of the studies we examined included predefined effectiveness thresholds, in most cases the threshold was 80%, and of the studies examined, more than half failed to specify the rationale for the selected threshold. We also found that the predefined thresholds were more frequently applied to the knowledge indicator than other indicators such as awareness, utilization or behaviour.

Assessment of RMM effectiveness is an essential element of risk management, ³ , ¹⁶ and clearly predefined success thresholds are important for effectiveness interpretation and regulatory decision‐making. ¹¹ , ¹² , ¹⁴ Most of studies we examined reflected a priori effectiveness thresholds consistent with regulatory guidelines ³ , ⁵ and represented increases in use of such thresholds compared with historic levels, as a prior review of PASS evaluating the effectiveness of RMMs between 2016 and 2021 found fewer than half of studies incorporated predefined effectiveness criteria. ¹² Notably, predefined success criteria were related to the conclusiveness of the study; 85.7% of conclusive studies used predefined thresholds, whereas neither inconclusive study incorporated such thresholds. Despite the small sample size of finished studies (N = 9), one can speculate that clearly defined thresholds may facilitate interpretation and conclusions regarding RMM effectiveness. ¹² Our findings highlight the need for predefined, evidence‐based justification of threshold values in designing future PASS to enhance interpretability and transparency. Clear justification of certain success criteria in knowledge‐based surveys is important for judging fairly whether an educational effort has been effective. A variety of approaches that can be used to determine such thresholds, including the use of baseline or historical data or expected frequency in similar populations. ³ Current guidance also provides factors for consideration when determining effectiveness of RMM, such as therapeutic context (e.g., seriousness of the indication and risk), adherence to local clinical guidelines and other dimensions (e.g., ethical or societal acceptability). ³

We also found the absolute thresholds used varied across different indicators, which is consistent with prior work indicating that a single threshold may not adequately capture varying levels of effectiveness. ⁸ , ¹² In addition to the knowledge indicator, the implementation and evaluation pathway of RMM should include dissemination and receipt of risk minimization materials, changes in risk awareness and self‐reported behavioural changes in target population. ³ , ¹⁷ Predefined, indicator‐specific thresholds along such pathway are important for assessing effectiveness of RMM from implementation to health outcomes, as they better capture the multifaceted nature of RMM effectiveness than a single, uniform threshold.

Although surveys are well‐established for measuring tool uptake and risk awareness to identify dissemination barriers and knowledge gaps, they are prone to multiple methodological challenges, such as lack of representativeness of target population and various biases. ⁷ Selection bias, information bias and non‐response are frequently reported limitations that hinder robust conclusions. ⁷ In our review, four of nine finished studies acknowledged limitations such as self‐selection, non‐response bias, recall bias and inadequate sample size (Table S2). Although these survey‐related limitations can be hard to completely overcome, multiple strategies have been provided to minimize their impact in the guidelines and literature. ⁷ , ¹⁶ , ¹⁸ Pilot testing of surveys may mitigate survey limitations ⁷ , ¹⁸ ; however, only half of the studies reported survey pretesting. Guidelines for appropriate conduct of feasibility assessments of surveys may be developed to aid minimizing limitations and achieving the ultimate study objectives of PASS evaluating RMM effectiveness.

Appropriate evaluation time points for RMM effectiveness depend on several factors, including time since RMM implementation in clinical practice, estimated magnitude of exposure, seriousness and severity of the risks and study design. ³ The EU guidelines recommend initial evaluation within 12–24 months of RMM regulatory implementation to allow necessary healthcare changes, with overall assessment within 4 years to inform the evaluation of marketing authorization renewal. ³ However, only 20% of RMM effectiveness PASS are estimated to complete within 5 years after initial marketing authorization, ¹⁹ with a median duration of 52 months from study request to completion. ¹³ In our review, only half of the studies finished data collection within 24 months of RMM implementation, indicating that many may not meet recommended time points for evaluating RMM effectiveness.

Our analysis has limitations. First, we limited our investigation to survey‐based studies among healthcare professionals within a single PASS repository. While we believe the repository is unique in capturing important study features, full protocols and study reports, future work may explore other platforms (e.g., FDA's Searchable Database for Postmarketing Requirements and Commitments or the safety reporting platform by Japan's Pharmaceuticals and Medical Devices Agency). Second, we focused on effectiveness thresholds applied to healthcare provider surveys, and future research may be extended to other target populations such as patients. Finally, we focused on post‐approval studies of proximal assessments of providers' knowledge, attitudes and belief in surveys, rather than quantitative analyses of downstream impact of RMMs on clinical behaviour or health outcomes using healthcare databases. Future research may be designed to summarize studies evaluating the ultimate effectiveness of RMMs in improving drug safety.

5. CONCLUSIONS

In this analysis of industry‐sponsored post‐authorization studies evaluating the effectiveness of RMMs in healthcare professionals between 2022 and 2024, we found that 94% of studies included predefined thresholds, most commonly set at 80% and more often applied to the knowledge indicator. Notably, only about half of the studies provided clear justifications for threshold selection. These findings highlight the increasing adoption of predefined success criteria in recent studies, while also underscoring the need for more detailed justification and indicator‐specific thresholds to enhance the interpretability and regulatory utility of RMM effectiveness evaluation during the design of survey‐based post‐authorization studies.

AUTHOR CONTRIBUTIONS

K.Z. and J.Y.E. designed and performed the research; K.Z. analyzed the data; all authors interpreted results; K.Z. wrote the manuscript; W.A.B., F.G.C., G.C.A., L.Z., J.J., C.J.R., and J.Y.E. revised the manuscript. All authors reviewed and approved the final manuscript.

CONFLICT OF INTEREST STATEMENT

K.Z., W.A.B., F.G.C., L.Z., C.J.R. and J.Y.E. are Sanofi employees and may hold shares and/or stock options in the company. J.J. was an employee of Sanofi at the time of conducting this study. G.C.A. is past Chair of FDA's Peripheral and Central Nervous System Advisory Committee and a co‐founding Principal and equity holder in Stage Analytics. These arrangements have been reviewed and approved by Johns Hopkins University in accordance with its conflicts‐of‐interest policies.

Supporting information

TABLE S1. DETAILED CHARACTERISTICS OF INCLUDED STUDIES (N = 17).

Table S2. THRESHOLDS FOR INDICATORS, RATIONALES FOR THRESHOLD SELECTION, RESULTS, AND LIMITATIONS OF INCLUDED STUDIES (N = 17).

Zhu K, Blumentals WA, Celzo FG, et al. Evaluating effectiveness of risk minimization through healthcare professional surveys: A review of post‐authorization studies, 2022–2024. Br J Clin Pharmacol. 2026;92(10):3495‐3502. doi: 10.1002/bcp.70647

DATA AVAILABILITY STATEMENT

All data used in the present study have been included in the manuscript.