Methodological guidelines and publications of benefit–risk assessment for health technology assessment: a scoping review protocol

Abstract

Background

Benefit–risk assessment (BRA) is used in multiple phases along the health technology’s life-cycle to evaluate the balance between the benefits and risks, as it is fundamental to all stakeholders. BRA and its methodological approaches have been applied primarily in the context of regulatory agencies. However, BRA’s application and extent in the context of health technology assessment (HTA) bodies remain less clear. Our goal is to perform a scoping review to identify and map methodological guidelines and publications on methods of BRA. This will be done considering the different phases of the life-cycle of health technologies to underline both the depth and extent of research concerning BRA, especially in the context of HTA.

Methods and analysis

This scoping review protocol was developed following the framework proposed by Arksey and O’Malley, and the updated guidelines by the Joanna Briggs Institute. We will include methodological publications that provide recommendations or guidelines on methods for BRA. We will conduct electronic searches on Medline (PubMed) and EMBASE (Ovid) databases; manual searches on the main websites of HTA bodies and drug regulatory organisations; and contact experts in the field. Systematic extraction forms will be used to screen and assess the identified publications by independent assessors. We will provide a qualitative synthesis using descriptive statistics and visual tools. Results will be summarised in systematic evidence tables and comparative evidence scoping charts.

Ethics and dissemination

This review will use data publicly available and does not require ethics approval. The results of this scoping review will contribute to scientific knowledge and act as a basis for methodologists, guideline developers and researchers for the development of BRA to inform regulatory decisions, reimbursement and coverage decision making. The results will be disseminated through peer-reviewed articles, conferences, policy briefs and workshops.

Trial registration number

Open Science Framework (https://doi.org/10.17605/OSF.IO/69T3V).

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• First scoping review aiming at mapping methodological guidelines and publications on methods of benefit–risk assessment, especially in the health technology assessment context.

• We will use the framework proposed by Arksey and O’Malley and the refinements made by the Joanna Briggs Institute.

• We will perform an electronic search on the main databases as well as manual searches on a vast source of grey literature, including health technology assessment bodies and networks, and regulatory agencies' websites.

• Despite our attempt to conduct a comprehensive search, we may miss documents reporting methodological recommendations or guidelines for benefit–risk assessment methods used in practice and not available in the publicly available online literature.

• It is expected that the lack of consistent classification and high variability of benefit–risk assessment frameworks and methods used in different contexts and technologies will increase the complexity of extraction and synthesis.

Introduction

Benefit–risk assessment (BRA), also called benefit–harm assessment, is used in multiple phases along a health technology’s life-cycle in a dynamic way to evaluate the balance between its expected benefits and risks (or harms) and it is fundamental to all stakeholders, including patients, healthcare providers, regulators, payers and pharmaceutical companies.^1–4 Over the past two decades, the BRA has been used predominantly in regulatory agencies to estimate the benefit–risk or benefit–harm balance of health technology for marketing authorisation and surveillance; to estimate the benefit–risk balance of the new technology as compared with existing ones, as part of the health technology assessment (HTA) process to support decision making on pricing and reimbursement; or to support clinical decisions at patient-level.^{1 2} Although different stakeholders are interested in determining the benefit–risk balance, they may have different perspectives, values and priorities; subsequently, the assessment methodologies may differ depending on the context.⁵

Structured BRA processes have been developed to support decision making where multiple outcomes need to be considered. Usually, BRA processes employ framing to clearly define the scope and the decision problem, in a transparent, rational and consistent way.^{3 6} Such approaches can help establish internal standards for decision making, enhancing the clarity of the decision-making process, encouraging the use of appropriate documentation, and providing consistency of communication and visualisation of benefits and risks to various stakeholders.³

Methodological guidelines for BRA in the context of marketing authorisation have been launched to provide all stakeholders with more accurate and useful information supporting continuous BRA.^{1 7 8} Several initiatives and methodological publications addressed different frameworks and methodological approaches for BRA, provided recommendations for the conduction of BRA, and incorporated patients’ perspectives into such assessments and tools to present the results of BRA.3–5 7–10 However, most of them were discussed and applied in the context of regulatory decisions and may fail to incorporate the particularities of BRA for the HTA decision-making process.

In a review of key publications and initiatives on frameworks and methodologies for BRA, Mt-Isa et al¹¹ highlighted that by 2013 there were few initiatives within the HTA context. One of these is the initiative of the European Network for Health Technology Assessment (EUnetHTA) and the European Medicines Agency (EMA), which aimed to place the decision-analytic models within the context of BRA to provide more integrated tools.¹² The collaboration between EUnetHTA and EMA can foster exchanges to reduce duplication of efforts between regulators and HTA bodies, enhance mutual understanding of evidence needs, facilitate more efficient allocation of resources, improve the efficiency of decision-making processes and enhance access to health technologies.¹³

However, there are differences in the types of approaches and principles between methods of BRA used for regulatory and HTA decisions,^{1 14} and the use of BRA and the implementation of such collaboration in the context of HTA is still further to go.¹ While BRA done by regulatory agencies tends to address the clinical efficacy and safety perspectives, its application in HTA reports may be broader and is still a work in progress. The systematisation and increased transparency associated with the use of structured BRA frameworks could benefit HTA by not only assessing the balance between benefits and risks (which might include aspects beyond efficacy and safety) but also addressing uncertainties and introducing quantitative parameters for assessment, particularly in cases of data-scarce scenarios. Also, BRA can play a central role in postcommercialisation monitoring studies and adaptive HTA.¹⁵ In summary, expanding the scope of BRA and incorporating these elements can improve and fortify the collaboration between regulatory agencies and HTA authorities, resulting in more comprehensive, transparent and informed evaluations of health technologies.¹⁶

There is still a gap in the literature regarding the availability of methodological guidelines and publications on methods for BRA and their characteristics in the context of HTA and at population decision-making levels. Therefore, this scoping review aims to map the available methodological guidelines and publications of methods for BRA in the different phases of the life-cycle of health technologies to underline both the depth and extent of research concerning BRA, especially in the context of HTA. Our goal is to identify which definitions have been used, describe BRA approaches (frameworks, metrics, estimation techniques and utility survey techniques) and discuss methodological characteristics of existing quantitative and descriptive approaches for assessing benefits and risks, focusing on quantitative methods. With our findings, we will support researchers, methodologists and health organisations in developing, conducting and reporting BRA.

This review represents the first phase in a larger project to improve BRA use in the context of the Brazilian HTA bodies. Through a partnership with Rede Brasileira de Avaliação de Tecnologias em Saúde—REBRATS, a strategic network built to facilitate the elaboration and dissemination of priority HTA studies for the Brazilian health system,¹⁷ this partnership will provide methodological and training support to researchers and technical professionals to increase the use of BRA methods in the reports used during deliberative decision-making processes. Furthermore, findings from this scoping review will inform the development of a methodological guideline on BRA for the Brazilian National HTA body, the Comissão Nacional de Incorporação de Tecnologias no Sistema Único de Saúde (CONITEC), using lessons learnt and best methodological practices from similar organisations.

Methods and analysis

Study design

While scoping reviews share rigorous and transparent methods in their conduct with systematic reviews, scoping reviews are more appropriate than systematic reviews to identify and map evidence relevant to a particular topic in broad sources of evidence.^{18 19} Specifically, scoping reviews can be useful tools to investigate the design and conduct of research on a particular topic including methodological approaches used, concepts and characteristics.²⁰ This scoping review methodology was based on the methodological framework proposed by Arksey and O’Malley²¹ and the updated guidelines by the Joanna Briggs Institute.^{19 22} For this scoping review protocol, we used Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)²³ (online supplemental file 1). The scoping review protocol was registered in the Open Science Framework (https://doi.org/10.17605/OSF.IO/69T3V).

Research question and eligibility criteria

Figure 1 summarises the relationship between our objectives, research questions and eligibility criteria. Our driven research question is ‘What are the methodological guidelines for methods of BRA in the context of health technology’s life-cycle?’

Figure 1.

Relationship between the objectives, research question and eligibility criteria for the scoping review. BRA: benefit–risk assessment; HTA: health technology assessment; PCC, population, concept and context.

We developed a research question structured with inclusion and exclusion criteria following the PCC mnemonic (population, concept and context) as described in figure 1 and above in detail.

Population

We will consider methodological documents of methods of BRA conducted for a specific health technology including pharmaceutical drugs, procedures, medical devices, equipment and diagnostic tests. As we will consider methodological documents that may not refer to a specific intervention or focus on a specific population, there will be no restriction regarding any characteristics of populations.

Concept

The overarching concept of interest for this scoping review is to identify and map methodological guidelines and studies of quantitative and descriptive frameworks that address quantitative methods of BRA of health technologies, to find out which definitions have been used to define BRA and to describe methodological characteristics of BRA approaches. This includes the type of framework and the application of formal quantitative methods for measuring/expressing outcomes (metrics), extracting data (estimation techniques) and eliciting preferences (utility survey techniques).^{4 5} This review defines BRA as the balancing of the expected favourable effects (benefits) and expected unfavourable effects (risks or harms) of health technology. When balancing benefits against risks, the impact of uncertainty on these outcomes and trade-offs between these outcomes must be considered.²⁴

We will exclude studies that reported methods exclusively for assessing harm or benefit outcomes, not the overall BRA balance, that is, for example, a review of only unintended adverse events or other harmful outcomes. Also, we will exclude publications that have focused on methodological approaches that could be used for BRA but did not present their application for BRA. For example, this exclusion criteria would apply to publications discussing the development or use of multicriteria decision analysis for decision making without its direct application in the context of BRA.

Context

This scoping review will consider methodological research studies or documents that provided recommendations or guidance on conducting or reporting a BRA in the context of health technology’s life-cycle such as decision making at the population level including the regulatory decision and HTA decision-making processes, clinical trials and evidence synthesis. These documents can be endorsed or produced by governments or non-governmental organisations or institutions, professional bodies, working groups or expert panels. They may be used to guide methodologists, guideline developers and researchers as the basis for the development of benefit–risk studies to inform regulatory decisions, reimbursement and coverage decision making as part of HTA reports. Also, it can be used as a methodological reference for the development of primary or secondary studies.

Types of publications

We will consider full-text documents that provide recommendations on how to conduct or report BRA. We will include two types of evidence sources: (1) methodological guidelines (ie, guidance, standards, frameworks, models, appraisals, recommendations or consensus) as any related tools or checklists and (2) methodological reviews, methodological studies or research reports addressing specific methods for BRA. We will exclude editorials, comments, opinion papers, conference abstracts, and primary studies of BRA.

Information sources and literature search

We will perform a comprehensive search for both published and unpublished evidence. The identification of evidence will be divided into three information sources: (1) electronic searches on the main life sciences and biomedical databases (electronic databases), (2) manual search for guidelines on the main websites of HTA and drug regulatory organisations, and other related sources (grey literature) and (3) contacting experts in the field. No restrictions regarding language or publication date will be made.

Electronic databases

A three-step search strategy approach was used.^{19 22} The first step includes an initial limited search on EMBASE (Ovid) and Medline (PubMed). This search was followed by an analysis of the text words contained in the title and abstract of retrieved papers, and of index terms used to describe the articles. A second search using all identified keywords and index terms was undertaken across all included databases. As a result of these two steps, the search strategy was built using free-text terms, indexing terms and validated filters^{25 26} to balance the sensibility and specificity of the searches to capture the CC’s key concepts: benefit–risk assessment, methods and guidelines or reviews or HTA reports (see online supplemental file 2). The search strategy has been developed and tested by three members of the research team (BDOA, FHdAM and FCG) with experience in systematic and scoping reviews and BRA studies, validated by an experienced research librarian (RC), and peer-reviewed by another reviewer using the Peer Review of Electronic Search Strategies checklist.²⁷ The search strategy was adapted for each database and/or information source (online supplemental files 2 and 3).

With no language limitations, a final search using the developed search strategy will be conducted in the electronic databases EMBASE (Ovid) and Medline (PubMed).

As the third step of search strategy, the reference list of included sources of evidence will also be screened for additional studies. Specifically, the references that will be examined will include those reported in reviews and relevant studies.

Grey literature

An online search on grey literature will be based on an elaborated list of the most relevant HTA bodies, regulatory agencies and health-related sources from the Grey Literature tool of the Canadian Agency for Drugs and Technologies in Health .²⁸ The search for HTA organisations will be supplemented by including other relevant HTA bodies and the leading global HTA networks (online supplemental file 4).

Regarding drug regulatory agencies, we have chosen 12 well-established drug regulatory authorities from four continents, including the USA, Canada, Europe, the UK, Australia, Japan and Brazil^28–31 (online supplemental file 5). Additionally, a hand search will be conducted in the references of all included studies and Google Scholar.

Contacting experts in the field

To supplement the systematic online searches, a member of our research team (AFRB) has contacted renowned researchers via email, requesting any available potential guidelines or publications related to methods of BRA. This outreach to experts in the field took place during the period spanning February to March 2023.

Selection process

Electronic databases

A team of four reviewers (BDOA, FCG, FHdAM and AFRB) will perform the selection process according to the predefined inclusion and exclusion criteria. Since a high number of documents are expected to be retrieved from the literature, we will divide the total number of records to be screened and selected by two pairs of reviewers. First, each pair of reviewers will screen independently the title and abstract. The full-text assessment of potentially eligible publications will also be done by two pairs of independent reviewers. Studies that do not meet the inclusion criteria defined previously will be excluded. The reasons for excluding any documents will be recorded. Any disagreements will be solved through a consensus between the team of reviewers, and if there is any disagreement, a third reviewer will be involved (PCS).

Pilot testing will be performed with a random sample of 100 titles/abstracts.²² The entire team will screen the random sample using the prespecified eligibility criteria. A consensus meeting to discuss the agreement will be settled. In case of discrepancies or adjustment needed of the eligibility criteria, another random sample will be screened. The team will start screening when 85% (or greater) agreement is achieved.

Grey literature

The search in grey literature sources will be developed in a stepwise approach. (1) All HTA, drug regulatory, and health-related grey sources will be identified with name, abbreviation, country, language and website link. (2) Two reviewers (BDOA and FCG) will search for eligible studies, technical papers and reports on each webpage of the listed grey sources. (3) An online search will be done on these web pages to identify potential documents with the key terms: ‘benefits and (risks or harms) and methods’. In the case of many search records, we will add other relevant terms, apply search filters or enter specific website sections. This search will be made according to the language of origin of the evidence source, or if more than one language is available, preference will be given to the English, Spanish and Portuguese languages. When documents written in other languages are identified, a certified translator will be hired to translate them. The reviewers (BOA and FCG) will read the title and abstract, and any potentially eligible documents will be saved for eligibility assessment. (4) All full-text documents will be downloaded and saved with the website’s ID, name, language, and URL. The total number of documents found on each website will be recorded. (5) All documents identified in the grey literature will be screened for eligibility based on a full-text examination. The conflicts of eligibility will be solved by consensus, or another reviewer (PCDS) will be available in case of disagreements. Reasons for excluding will be reported.

Contacting experts in the field

In case the contacted experts send potential documents, two reviewers will apply the inclusion and exclusion criteria to select eligible documents.

Software used in the selection process

After running the search strategy in the electronic databases, all identified citations will be uploaded into Rayyan,³² an online platform that facilitates the article screening and selection process and removes duplicates. The selection process of grey literature and documents that will be sent from experts will be performed in Excel. The full text of selected citations will be stored on EndNote with a file in PDF attached.

Charting the data, summarising and reporting the results

A draft data charting form to guide the data extraction from the publications was developed by BDOA and PCDS, and validated by the other authors (FCG, EAS and FHdAM) (table 1). Previous studies focusing on BRA methodologies were used to develop the draft data charting form, we will consider the BRA methodologies classification proposed by Mt-Isa et al⁵ and Hallgreen et al³³ and exemplified by Hughes et al⁴ and Juhaeri et al.³⁴ This may be further improved at the review stage and the charting table updated accordingly. The vocabulary used for this protocol, including those on the charting data form and that will be used in the scoping review study, is given in online supplemental file 6.

Table 1.

Data charting form for the scoping review on methods of benefit–risk assessment

	Questions and information	Description/options (examples)
General information	First author and publication year	Silva, 2012
	Institutions involved in the document development	Industry, academic institution, health institution, regulatory agency, HTA body, consulting firm, others (specify)
	Main institution(s)/ organisation(s)/body(ies)/agency(ies) who produced the document	Description
	Main institution (s)/organisation (s)/body (ies)/agency (ies) who sponsored/supported/funded/ the development of the document	Description
	A geographical area that the institutions who developed the methodological document	Description. Examples: international (global) or continent (Europe) or country (Germany) or not reported
	A geographical area of the context of where the methodological document is destinated	Description. Examples: international (Global) or continent (Europe) or country (Germany) or not reported
	Composition body	Description of individuals involved in the development process (methodology) such as methodologists, health professionals and patients. If not explicitly described, it will be informed as not reported
	Target audience	Description of the target audience (individual to whom the document is intended). Examples: researchers, methodologists, payers, the public and others
	The conflict of interest was reported?	Yes, no
	Is there a conflict of interest from individuals involved in research support or employment, including salaries, equipment, supplies, reimbursement for attending symposia and other expenses; any stocks or shares and any consultation fees or other forms of remuneration from the manufacturer?	Yes, no, not possible to identify/evaluate
General methodological information	Publication type	Methodological guidelines, report of HTA body, report of a regulatory agency, methodological report of health and/or academic and/or research institution, methodological systematic review, narrative review with methodological recommendations and others (specify)
	Types of technology addressed	Pharmaceutical drugs, vaccines, medical devices, procedures, diagnostic tests, equipment, general (it can be addressed or be applied to any health technology) and others (specify)
	Context of benefit–risk assessment was made for	Regulatory decision, POS marketing guidance, coverage, or reimbursement in HTA decisions, evidence synthesis, clinical trials, clinical decision and others (specify)
	General methodology	Report with systematic and transparent methods, systematic review, systematic literature review as rapid literature review, literature review, interviews, review of reports, methodological guidelines/manuals, multiple methods that were not available as an option, relevant papers in the field, real-world case studies and others (specify)
	Was the definition of benefits given?	Yes, no Description from the authors
	Was the definition of risks or harms given?	Yes, no Description from the authors
	Was the definition of benefit–risk assessment given?	Yes, no Description from the authors
	Information regarding evidence to be used in the benefit–risk assessment	Real-world evidence, systematic reviews and meta-analysis, phase III clinical trials: non-randomised clinical trial, phase III clinical trials: randomised clinical trial, phase II clinical trials: non-randomised clinical trial, phase II clinical trials: randomised clinical trial, phase I clinical trials, observational studies, others (specify) or not reported
	Types of frameworks	Type of framework (quantitative and/or descriptive framework, not reported) Type of framework (designation name of the framework) Framework(s) recommended by the authors
Specific methodological information (methods for balancing benefit–risk)	Types of metrics (systems of measurement, eg, health indices and trade-off indices)	Type of metrics (designation name of the metric) Metric(s) recommended by the authors
	Types of estimation techniques (generic statistical techniques)	Type of estimation techniques (designation name of the estimation techniques) Estimation techniques recommended by the authors
	Types of utility survey techniques (methods to elicit and collect utilities and value preferences of various outcomes)—only for quantitative framework	Type of utility survey techniques (designation name of utility survey techniques) Utility survey techniques recommended by the authors
	Does the methodological document recommend a checklist for benefit–risk assessment?	Yes, no
	Is there any description of visual tools for communicating evidence and translating knowledge of the benefit–risk assessment?	Yes, no
	Is there any description of general recommendations for communicating evidence and translating knowledge of the benefit–risk assessment?	Yes, no. If yes, provide the general recommendations
	Types of identified visual tools (visual representation of results)	Type of visual tools (designation name of the visual tools)

HTA, health technology assessment.

Data will be extracted independently by three pairs of experienced reviewers (BDOA, FCG, EAS, FHdAM, AFRB and NSF) with an appropriate background in guidelines and BRA, using an online semistructured Google form based on the validated data charting form. Any disagreements will be solved through a consensus between the team of reviewers, and if there is any disagreement, a third reviewer will be involved (PCDS).

We will provide a qualitative synthesis reported in systematic evidence tables along with descriptive statistics to identify common characteristics and map the evidence. Also, we will use appropriate visual sources to provide an overall summary of the results such as charts, figures and maps. Furthermore, we will interpret and discuss the meaning of the findings and implications for future research and practice, and we will report the gaps that should be addressed in future research. The PRISMA Extension for Scoping Reviews will be used to guarantee the quality of study reporting and replicability of the scoping review study.³⁵

Critical appraisal of individual sources of evidence

Since we will include a wide variety of methodological guidelines and publications and there is no standard quality tool to perform the critical appraisal assessment of included studies, we will not perform a critical appraisal of included studies or work.

Patient and public involvement

Patients and/or the public were not involved. This is a scoping review study with no primary data collection. We intend to collaborate with a patient advisory group in the development of future BRA, based on the results of this study.

Ethics and dissemination

With the findings of this scoping review, we will contribute with a better understanding of which BRA methods are available; what were definitions that have been used; and what are the BRA approaches and the main methodological characteristics of existing quantitative approaches for assessing benefits and risks. These contributions are not in the context of regulatory approval or clinical trials, but mainly in the context of HTA bodies. This knowledge will help to provide a scientific basis for researchers, guideline developers, and all stakeholders interested in BRA.

In addition, the insights of this scoping review could be used to enhance the use of BRA in the context of HTA worldwide, enhancing the collaboration of different stakeholders and improving the efficiency of decision-making processes. This protocol and the full publication of this protocol will be disseminated through peer-reviewed publications, conferences, policy briefs and workshops.

Ethics statements

Patient consent for publication

Not applicable.