Guidelines for evidence synthesis reporting: medications

Abstract

In the process of determining if a drug is valuable enough to be included in a hospital’s pharmacotherapeutic repertoire many factors should be taken into account. In order to develop a guide, the methodology of different appraisal working groups and similar methodological documents published by Health Technology Assessment agencies have been taken into account. We recommend that reports are structured with the following headings: Medication/Description/Authorised indication; Description of the disease; Pathology reference treatment; Evaluation of efficacy and safety (Bibliographic search, Quality assessment, Efficacy and safety results); Assessment of ethical, organisational, social and legal aspects; Strengths and limitations of available evidence; Pharmacoeconomic evaluation; and Key points. This guide to evaluate technologies may be used as a tool in decision-making scenarios related to health innovation. It could be used by hospital pharmacists and by clinicians, health system professionals and public services advisors.

Introduction

As Professor Honoré has pointed out previously in this journal, drug evaluation involves many actors.¹ In the process of determining if a drug is valuable enough to be included in a hospital’s pharmacotherapeutic repertoire many opinions should be taken into account: the pharmaceutical company developing the drug, the national drug agency, international authorities such as the European Medicines Agency (EMA) and the US Food and Drug Administration (FDA), committees for establishing national drug treatment guidelines, drug selection committees and the prescribing physician. In addition to these professional groups, concerns have also been raised about including patients in the evaluation process. However, an appropriate procedure to incorporate the patients has not been thoroughly developed.

We present here a paper which may act as a guide to evaluate technologies. These kinds of guides can be used as tools in decision-making scenarios related to health innovation. Although these guides will most likely be used by hospital pharmacists, they could also be used by other kinds of researchers, clinicians, health system professionals and advisors of other public services involved in developing and reviewing technologies.

Objectives

Develop and describe the methodology used to produce reports of ‘Synthesis of Evidence: Health technology, new drugs or new indications of previously authorized drugs’.

Methodology for developing the guide

In order to develop this guide, the methodology of different appraisal working groups such as the Group for Innovation, Assessment, Standardisation and Research in the Selection of Drugs of the Spanish Society of Hospital Pharmacy (GENESIS-SEFH) (http://gruposdetrabajo.sefh.es/genesis) and the European Network for Health Technology Assessment (EUnetHTA) has been taken into account.^2–13 In addition to this methodology, other similar methodological documents published by Health Technology Assessment (HTA) Agencies have also been referenced.^14–18

The document should be reviewed and updated on a regular basis to incorporate the improvements identified in the established process.

Evidence synthesis reports

Structure and contents

It is recommended that reports be divided into these sections:

• Medication/Description/Authorised indication.

• Description of the disease.

• Reference pathology treatment.

• Evaluation of efficacy and safety.

• Key points.

The first three are intended to define the first three components of the research question: Population, Intervention, Comparator, Outcomes and Study Design (PICO(s)).

The efficacy and safety assessment section aims to present the synthesis of data on efficacy and safety, compared with the reference treatments, clearly defining information about the new drug or the new indication available at the time the report is compiled. It will also present the main strengths and limitations of the available evidence, including the assessment of the internal validity of the evidence evaluated as well as the applicability of the evidence. The detected gaps of evidence (if any) will also be collected. In addition, a pharmacoeconomic evaluation, including budgetary impact, would be necessary to complete the evaluation. In the last section of the report, the key points of the evaluation of effectiveness and safety will be summarised together with the level of evidence.

Report title

The title should be brief and concise, explicitly naming the drug to be evaluated and the pathology or disease for which it is applied.

Medication/Description/Authorised indication

This section will describe the main characteristics of the medicine to be evaluated. The following sources of information will be consulted: the drug’s technical file (European public assessment reports, EPAR) and the EMA evaluation report, indicating the date of publication of the report. If not, if the EPAR has not yet been published, the report of the FDA will be consulted. In all cases, it will be clarified if it is a new drug or if it is a new indication for a drug already authorised.

Description of the disease

This section will briefly describe the health problem. It should be written to facilitate the understanding and critical reading of the report by a professional who may not necessarily be an expert in the related clinical area. Epidemiology, clinical presentation, risk factors and prognostic factors will be described briefly.

It should be guided by the research questions included in the domain ‘health problem and current use of technology’ of the HTA Core Model for Rapid Relative Effectiveness Assessments.²

Pathology reference treatment

This section should be considered very important as it explains the selection of the comparators for the evaluation process, which is a critical step during the evaluation. The recommendations set out in the main guidelines of clinical practice will be summarised. These guidelines are compiled mainly in the National Guideline Clearinghouse and Guidelines International Network databases and the websites of the main scientific societies. Information about the comparator can also be obtained in Rapid Assessment Reports (RAR), regulatory agency reports and consensus documents. As a general rule, guidelines published in the last 5 years should be considered. In the case of cancer medicines, only guidelines published in the last 3 years should be taken into account.

A description of the usual practice would be necessary if it differs from that which is described in the clinical guidelines. In this case, systematic reviews and published original randomised controlled trials (RCT) should also be consulted.

A number of research questions in the domain ‘health problem and current use of technology’ of the HTA Core Model for Rapid Relative Effectiveness Assessments² will therefore be answered in a synthetic way.

Evaluation of efficacy and safety

Methodology

The first step should be the establishment of the PICO(s) question. To do this, a table will be constructed, which will specify the population, the medicine to be evaluated, the optimal comparator(s), the outcome variables and the study designs to be considered. It may be helpful consulting different articles published, like the one about dealing with clinical queries.¹⁹

For the selection of comparators and outcome variables, the recommendations established in the guidelines methodologies of EUnetHTA3 5–9 20 should be considered.

In general, the designs of the studies to be selected will be in the following order: RAR, network meta-analysis, traditional meta-analysis, systematic reviews and pivotal clinical trials (RCTs) and studies of non-randomised controlled study (NRS), in the case of non-RCTs.

If HTA reports are located, it will be assessed whether the document can be used, that is, if it can be adopted (use the report without making any changes, simply translate it) or adapt.

To evaluate the possible adaptation of a report, the tool developed by EUnetHTA HTA Adaptation Toolkit and Glossary¹² will be used to assess the relevance, reliability and transferability of the results.

Bibliographic search

To identify available evidence on efficacy and safety, a manual search will be conducted on the web pages of regulatory agencies such as the EMA (for EPAR location, if already published) and the FDA.

A manual search will also be conducted on the websites of local, national and international HTA agencies and bodies, such as the Agency for Health Quality and Assessment of Catalonia, Catalan Hospital Outpatients Drugs Assessment Committee, National Institute for Health and Care Excellence, French National Authority for Health, Ludwig Boltzmann Institute, Institute for Quality and Efficiency in Health Care, National Center for Pharmacoeconomics, Swedish Council on Technology Assessment in Health Care, Norwegian Knowledge Center for Health Services, EUnetHTA, Canadian Agency for Drugs and Technologies in Health, and Agency for Healthcare Research and Quality. It will also explore the search engines of the International Network of Agencies for HTA. Documents published in the last year will be reviewed.

Subsequently, if no report has been found for its adoption, a bibliographic search will be carried out in the following databases: the Cochrane Library, Center for Reviews and Dissemination, International Information Network on New and Emerging Health Technologies (EuroScan International Network) and MEDLINE (through PubMed, with Medical Subject Headings). Documents published in the last 2 years will be reviewed.

A search will also be made in the registries of studies under development in the databases of EudraCT, EU Clinical Trials Register (available at: http://clinicaltrialsregister.eu/) and the US National Institutes of Health (available at: http://clinicaltrial.gov/).

In addition, the database will be searched in the Planned and Ongoing Projects database, which allows European agencies to share information on projects planned, under development and published in the last 3 months by each agency.

Quality assessment

The tool proposed by the Cochrane Collaboration²¹ will be used to assess the risk of bias of the included RCTs. In the case of systematic reviews, its methodological quality will be evaluated by means of the checklist AMSTAR (A MeaSurement Tool to Assess systematic Reviews).²² The use of both tools is recommended in the EUnetHTA methodology guide for the evaluation of internal validity.¹⁰ For the critical evaluation of the NRS, the Transparent Reporting of Evaluations with Non-Randomized Designs checklist, which sets out the essential points to be described in the publication of these studies, will be used.

Whenever possible, the selection of articles, critical reading, extraction and analysis of the results will be done by pairs.

Efficacy and safety results

In the event that RARs have not been located or are not suitable for our evaluation:

• The main features of the included meta-analysis/systematic reviews/RCT/NRS will be presented in tables.

• The main results of efficacy and safety will be indicated in tables, whenever possible.

For each of the selected studies, three tables will be elaborated: the first one will summarise the characteristics of the study, the second, the results for the considered efficacy variables, and the third, the safety results.

It is recommended to specify clinically significant endpoints used to evaluate new medicine versus comparator; to quantify in terms of frequency, severity and safety results versus comparator and where data come from; what are the known and estimated benefits and harms for patients when implementing or not the new medication.

For the exposure of the efficacy and safety results, the main variable will be collected first and then the secondary variables. These can initially be presented for the global population and, subsequently, according to subgroups. The frequency and severity of adverse events shall be reported for safety results.

Assessment of ethical, organisational, social and legal aspects

Due to the brief nature of the evidence synthesis report, the drug’s focus is on the study of efficacy and safety. In addition, a questionnaire may be completed to assess whether there are specific aspects at the ethical, organisational, social and legal levels that should be evaluated. In online supplementary appendix, we report a checklist template for potential ethical, organisational, patient, and social and legal aspects.² If the answer to any of the questions is ‘yes’, further analysis will be required on that question.

Only issues where there is a difference between the evaluated drug and the main comparators will be considered. Therefore, pre-established issues that are common to the new drug and its comparators will not be described because the incorporation of a new drug does not imply changes in them.

Strengths and limitations of available evidence

In this section, the risk of bias and/or the methodological quality of the included studies (internal validity) will be made explicit. To guide the evidence evaluation, the group Critical Appraisal Skills Programme offers in their web page interesting tools.²³ This helpful resource was mentioned among others by Wiffen et al in a publication that promotes mentoring in evidence-based pharmacy.²⁴

A brief analysis will be made of the comparators used, variables evaluated and analysis of subgroups performed, where appropriate. If the comparator used in pivotal RCTs is not the most appropriate, the need to carry out an additional report should be indicated, where possible, network meta-analysis or indirect comparisons of the drug evaluated against the comparators appropriate.

The applicability of the evidence (external validity) will be assessed. This can be defined as the degree to which the effects observed in clinical studies may reflect the expected results when a specific intervention is applied to the population of interest. A table will be drawn up based on the recommendations set out in the EUnetHTA methodological guide on the applicability of evidence.¹¹

Results with a high level of uncertainty for relevant variables, absence of results for clinically relevant variables or absence of studies indicate, therefore, the existence of uncertainty, and therefore of evidence gaps. We need to be careful in extrapolating results from small trials. It will be necessary to compare if these uncertainties can be solved through the results that will be obtained from RCTs that are in development. It is compulsory to evaluate if results translate into clinical benefit, though it is not an easy task.²⁵

It is important to make clear which advantage would represent the drug that is being evaluated. What would be the clinical benefit obtained from the use of this drug? The European Society of Clinical Oncology proposed a clinical benefit scale as an objective measure to determine if the results of oncology trials are relevant.²⁶ Setting the standards of gain relevance is important to guarantee equity. It would be very interesting to have these kinds of scales available to apply to other clinical conditions.

Pharmacoeconomic evaluation

As previously mentioned, a pharmacoeconomic evaluation, as complete as possible, would be necessary to complete the evaluation. First, prices indicating local discounts of the evaluated drug and the existent alternative must be indicated. Second, the price according to the gain related to the main objective of the RCT (overall survival or progression-free survival, per month, for example). Last, the cost of the estimated length of treatment (calculated from medium treatment administration time in RCTs). Measurement of health-related quality of life is important when assessing the impact of long-term illness of chronic disease where the goal of medication existing is to improve compared with alternative interventions. GENESIS-SEFH has recently edited a guide to help in this purpose.²⁷ In online supplementary tables 1–3, we reproduce some examples that can be done to perform the economic analysis. Additional materials (online supplementary table 1—Costs of drug therapy and the different alternatives; online supplementary table 2—Cost-efficacy analysis; online supplementary table 3—Budgetary impact analysis) are published online only.

Key points

The key points will be summarised in a clear and concise manner. Both the strengths and limitations of the evidence and the key points will focus on the evaluation of effectiveness and safety.

To sum up, EUnetHTA Rapid Relative Effectiveness Assessments (2016–2020) could be found on https://www.eunethta.eu/rapid-reas/, and EUnetHTA Joint Action 2, Work Package 8. HTA Core Model version 3.0 (Pdf); 2016. Available from www.htacoremodel.info/BrowseModel.aspx.