Skip to main content
GMS Health Technology Assessment logoLink to GMS Health Technology Assessment
. 2005 Nov 15;1:Doc09.

Methods for the comparative evaluation of pharmaceuticals

Methoden zur vergleichenden Bewertung pharmazeutischer Produkte

Annette Zentner 1,*, Marcial Velasco-Garrido 1, Reinhard Busse 1
PMCID: PMC3011319  PMID: 21289930

Abstract

Political background

As a German novelty, the Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; IGWiG) was established in 2004 to, among other tasks, evaluate the benefit of pharmaceuticals. In this context it is of importance that patented pharmaceuticals are only excluded from the reference pricing system if they offer a therapeutic improvement.

The institute is commissioned by the Federal Joint Committee (Gemeinsamer Bundesausschuss, G-BA) or by the Ministry of Health and Social Security. The German policy objective expressed by the latest health care reform (Gesetz zur Modernisierung der Gesetzlichen Krankenversicherung, GMG) is to base decisions on a scientific assessment of pharmaceuticals in comparison to already available treatments. However, procedures and methods are still to be established.

Research questions and methods

This health technology assessment (HTA) report was commissioned by the German Agency for HTA at the Institute for Medical Documentation and Information (DAHTA@DIMDI). It analysed criteria, procedures, and methods of comparative drug assessment in other EU-/OECD-countries. The research question was the following: How do national public institutions compare medicines in connection with pharmaceutical regulation, i.e. licensing, reimbursement and pricing of drugs?

Institutions as well as documents concerning comparative drug evaluation (e.g. regulations, guidelines) were identified through internet, systematic literature, and hand searches. Publications were selected according to pre-defined inclusion and exclusion criteria. Documents were analysed in a qualitative matter following an analytic framework that had been developed in advance. Results were summarised narratively and presented in evidence tables.

Results and discussion

Currently licensing agencies do not systematically assess a new drug's added value for patients and society. This is why many countries made post-licensing evaluation of pharmaceuticals a requirement for reimbursement or pricing decisions. Typically an explicitly designated drug review body is involved.

In all eleven countries included (Austria, Australia, Canada, Switzerland, Finland, France, the Netherlands, Norway, New Zealand, Sweden, and the United Kingdom) a drug's therapeutic benefit in comparison to treatment alternatives is leading the evaluation. A medicine is classified as a therapeutic improvement if it demonstrates an improved benefit-/risk-profile compared to treatment alternatives. However, evidence of superiority to a relevant degree is requested.

Health related quality of life is considered as the most appropriate criterion for a drug's added value from patients' perspective. Review bodies in Australia, New Zealand, and the United Kingdom have committed themselves to include this outcome measure whenever possible.

Pharmacological or innovative characteristics (e.g. administration route, dosage regime, new acting principle) and other advantages (e.g. taste, appearance) are considered in about half of the countries. However, in most cases these aspects rank as second line criteria for a drug's added value.

All countries except France and Switzerland perform a comparative pharmacoeconomic evaluation to analyse costs caused by a drug intervention in relation to its benefit (preferably by cost utility analysis). However, the question if a medicine is cost effective in relation to treatment alternatives is answered in a political and social context. A range of remarkably varying criteria are considered.

Countries agree that randomised controlled head-to-head trials (head-to-head RCT) with a high degree of internal and external validity provide the most reliable and least biased evidence of a drug's relative treatment effects (as do systematic reviews and meta-analyses of these RCT). Final outcome parameters reflecting long-term treatment objectives (mortality, morbidity, quality of life) are preferred to surrogate parameters. Following the concept of community effectiveness, drug review institutions also explicitly favour RCT in a "natural" design, i.e. in daily routine and country specific care settings.

The countries' requirements for pharmacoeconomic studies are similar despite some methodological inconsistencies, e.g. concerning cost calculation.

Outcomes of clinical and pharmacoeconomic analyses are largely determined by the choice of comparator. Selecting an appropriate comparative treatment is therefore crucial. In theory, the best or most cost effective therapy is regarded as appropriate comparator for clinical and economic studies. Pragmatically however, institutions accept that the drug is compared to the treatment of daily routine or to the least expensive therapy.

If a pharmaceutical offers several approved indications, in some countries all of them are assessed. Others only evaluate a drug's main indication. Canada is the only country which also considers a medicine's off-label use.

It is well known that clinical trials and pharmacoeconomic studies directly comparing a drug with adequate competitors are lacking - in quantitative as well as in qualitative terms. This is specifically the case before or shortly after marketing authorisation. Yet there is the need to support reimbursement or pricing decisions by scientific evidence. In this situation review bodies are often forced to rely on observational studies or on other internally less valid data (including expert and consensus opinions). As a second option they use statistical approaches like indirect adjusted comparisons (in Australia and the United Kingdom) and, commonly, economic modelling. However, there is consensus that results provided by these techniques need to be verified by valid head-to-head comparisons as soon as possible.

Conclusions

In the majority of countries reimbursement and pricing decisions are based on systematic and evidence-based evaluation comparing a drug's clinical and economic characteristics to daily treatment routine. However, further evaluation criteria, requirements and specific methodological issues still lack internationally consented standards.

Executive Summary

1. Political background

According to the Act for Modernisation of the Statutory Health Insurance (Gesetz zur Modernisierung der Gesetzlichen Krankenversicherung; GMG) the German Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; IQWiG) was established in 2004 to, among other tasks, evaluate the benefit of pharmaceuticals. The institute is commissioned by the Federal Joint Committee (Gemeinsamer Bundesausschuss; G-BA) or by the Ministry of Health and Social Security. In this context it is of importance that patented (and often expensive) pharmaceuticals are only excluded from the reference pricing system if they offer a therapeutic improvement.

2. Research questions

The German health policy objective is to perform a differentiated assessment of pharmaceuticals in comparison to already available treatments but procedures and methods are still to be established. This health technology assessment (HTA) report was commissioned by the German Agency for HTA at the Institute for Medical Documentation and Information (DAHTA@DIMDI). DAHTA@DIMDI asked the question: Are there methods for a comparative evaluation of pharmaceutical products? Which are these? The research question was specified as follows: How do other EU-/OECD-countries compare medicines in connection with pharmaceutical regulation, i.e. licensing, reimbursement and pricing? This study thus analysed criteria, procedures, and methods of comparative drug assessment used by public institutions in other countries.

3. Methods

As a first step national public institutions in other EU-/OECD-countries as well as documents concerning comparative drug evaluation (e.g. regulations, guidelines) were identified through internet searches.

Additionally a systematic literature search was performed: 27 databases for medical, economic, social, and natural sciences were searched for articles published by February 2005. Synonyms for the term "drug evaluation" in combination with names of the respective country were used as search terms. After duplicates had been removed, 1619 publications were analysed for their relevance in a two step selection process, in accordance with pre-defined inclusion and exclusion criteria. Finally 85 articles were included in the study.

The search was complemented by hand search in bibliographies and, in case of doubt, by email communication with the institutions. All documents were analysed in a qualitative matter and according to an analytic framework that had been developed in advance. Results were summarised narratively and presented in evidence tables.

4. Results and discussion

Currently licensing agencies (i.e national institutions and the European Agency for the Evaluation of Medicinal Products, EMEA) do not systematically assess a new drug's added value for patients and society. This is why many countries made post-licensing evaluation of pharmaceuticals a requirement for reimbursement or pricing decisions. Typically an explicitly designated drug review body is involved.

In this study twelve national public institutions in eleven EU-/OECD-countries performing comparative drug evaluation were included: Commission for Evaluation of Pharmaceuticals (Heilmittelevaluierungskommission; HEK) in Austria, Pharmaceutical Benefits Advisory Committee (PBAC) in Australia, Patented Medicine Prices Review Board (PMPRB) and Canadian Expert Drug Advisory Committee (CEDAC) in Canada, Confederate Pharmaceutical Commission (Eidgenössische Arzneimittelkommission, EAK) in Switzerland, Pharmaceuticals Pricing Board in Finland, Transparency Commission (Commission de Transparence) in France, Committee for Pharmaceutical Aid (Commissie Farmaceutische Hulp, CFH) in the Netherlands, Norwegian Medicines Agency (NoMA) in Norway, Pharmaceutical Management Agency (PHARMAC) in New Zealand, Pharmaceutical Benefits Board (PBB) in Sweden, and National Institute for Clinical Excellence (NICE) in the United Kingdom.

In about half of the countries national drug review bodies act as advisors. They make reimbursement or pricing recommendations to a national or regional government, a ministerial department or to a self-governance body (in Austria, in Australia, in Canada (reimbursement), in France (reimbursement), in the Netherlands, in Switzerland, and in Norway).

The remaining countries have established regulatory-type drug review bodies. Though accountable to health ministries, they function as the decision-maker responsible for listing or pricing of drugs (in Canada (pricing), in Finland, in France (pricing), in New Zealand, in Sweden, and in the United Kingdom).

Information provided by the documents analysed on drug evaluation procedures and methods varied to a great extent among countries. Australia and Canada belong to the countries which published the most comprehensive and detailed guidelines. Procedures in these countries seem to be characterised by a high degree of transparency - at least in theory.

It can be stated that pharmacoeconomic methoologies are described in more detail than procedures for the clinical review or for assessment of other drug evaluation aspects.

In all countries a drug's therapeutic benefit in comparison to treatment alternatives is leading the evaluation. A medicine is classified as a therapeutic improvement if it demonstrates an improved benefit-/risk-profile compared to treatent alternatives. However, evidence of superiority to a relevant degree is requested.

Health related quality of life is considered as the most appropriate criterion for a drug's added value from patients' perspective. Review bodies in Australia, New Zealand, and the United Kingdom have committed themselves to include this outcome measure whenever possible.

Pharmacological or innovative characteristics (e.g. administration route, dosage regime, new acting principle) and other advantages (e.g. taste, appearance) are considered in about half of countries. However, in most cases they ranked as second line criteria for a drug's added value.

The majority of countries perform a comparative pharmacoeconomic evaluation to analyse costs caused by a drug intervention in relation to its benefit (preferably by cost utility analysis). However, the question if a medicine is cost effective in comparison to treatment alternatives is answered in a political and societal context. A range of criteria are considered. They vary remarkably from country to country and include: availability of alternative therapies, social and ethic aspects like equity and solidarity, a drug's potential impact on public health, on organisations, and professionals, the community need and compliance with government-defined priorities, a medicine's estimated budget impact, and manufacturing research and development expenses.

In the case of therapeutic equivalence the drug is either not accepted for public reimbursement or it is subject to a reference pricing system (e.g. in New Zealand and in the Netherlands). In Australia and Canada equivalence is tolerated if a cost minimisation analysis demonstrates fewer costs.

However, there is international consensus that a drug therapy with a benefit-/risk-profile worse than other therapies (inferiority) should not be reimbursed, even in the case of fewer costs.

In Austria, in Australia, in Norway, and in the Netherlands drug evaluation is based on submissions by manufacturers. These comprise a systematic literature search and analysis of clinical and economic studies, in parts including modelling. Institutions have published guidelines to outline the methodological requirements for manufacturers and reviewers which, however, vary in the level of detail.

France, Switzerland, Canada (PMPRB), and Finland require that manufacturers submit a limited number of pivotal clinical trials. Since some details of the evaluation procedure are not made transparent it could not be clarified if these institutions additionally carry out a (systematic) research and analysis on their own.

Finally, in a third group of countries review bodies perform a systematic and comprehensive literature synthesis themselves and independently from manufacturers, including modelling. This is the case in Canada, in New Zealand, in the United Kingdom, and in Sweden.

Assessment of unpublished evidence is explicitly included in Austria, in Australia, in Canada, in the Netherlands, in Sweden, and in the United Kingdom. However, commercial in confidence data is excluded in parts (in Austria and in the United Kingdom).

Corresponding to the procedures' diversity, drug assessments last between a couple of weeks (e.g. in Australia and in France) and a year (e.g. in Sweden and in the United Kingdom). However, for European member states EC transparency directive 89/105/EEC determines that decisions on reimbursement and pricing of new pharmaceuticals are to be made within 180 days after marketing authorisation.

Countries agree that randomised controlled head-to-head trials (head-to-head RCT) with a high degree of internal and external validity provide the most reliable and least biased evidence for a drug's relative treatment effects (as do systematic reviews and meta-analyses of these RCT). Final outcome parameters reflecting the long-term treatment objective, i.e. final changes in health status caused by the therapy, are preferred (mortality, morbidity, quality of life). For study designs relying on surrogate parameters a strong and scientifically accepted association between intermediate effect and final outcome is required.

Following the concept of community effectiveness, nine out of twelve drug review institutions explicitly favour RCT in a "natural" design, i.e. in daily routine and country specific care settings. Concerning literature searches, selection of studies, assessment of internal and external validity of clinical trials and systematic reviews/meta-analyses, the review bodies apply internationally established standards (e.g. guidelines from Cochrane Collaboration, the CONSORT (Consoldation of Standards for Reporting Trials)- and the QUOROM (Quality of Reports of Meta-analyses of Randomised Controlled Trials)-Statement) or use comparable own standards.

The countries' requirements for pharmacoeconomic evaluations are similar. However they include some methodological problems. Cost utility analysis (CUA) measuring health outcomes in terms of Quality Adjusted Life Years (QALY) is most often recommended as appropriate analytic design. The rationale is that CUA can be applied for comparison of different health interventions and thus can be used for priority setting. It is considered to be associated with fewer methodological problems as cost-benefit analysis. There is also consensus that pharmacoeconomic studies should be, at least additionally, carried out from a societal perspective. According to the theory of social welfare, costs and benefits outside the health sector should be taken into account (direct and indirect non-medical costs).

However, there is no consensus how loss of productivity should be determined (human capital approach or friction cost method). There is also inconsistency if losses in spare time and in housework time are adequately included in changes of quality of life or if these should be additionally measured. Addressing the health economic controversy if costs of other diseases should be included if an intervention results in a prolonged life, guidelines recommend to include only costs that are directly associated with the intervention.

Utility measures should be based on preferences of a country's population. Since data is not always transferable from one country to another institutions also request that type and quantity of resource consumption and prices should be based on national data. A high degree of transparency in cost calculation is required, i.e. identifying costs accurately, presenting quantity of resources consumed separately from the respective price and accomplished by the sources of data.

Discounting is recommended throughout the guidelines with an annual rate between 2.5% and 10%. From a societal perspective a discounting rate of 5% is typically recommended. Discounting of costs as well as of benefits at identical rates is suggested. As a rule, institutions require sensitivity analyses to determine effects of discounting on the study results by testing different rates and inclusion of benefit discounting.

Outcomes of clinical and pharmacoeconomic analyses are largely determined by the choice of comparator. Selecting an appropriate comparative treatment is therefore crucial. Concerning the institutions analysed in this study two procedures can be distinguished:

A. Some institutions require that a pharmaceutical should be compared with up to three explicitly defined comparators (in Australia, in Canada (CEDAC), in Finland, in New Zealand, in Sweden, and in the United Kingdom). In theory, the best or the most cost effective therapy is regarded as appropriate comparator. Pragmatically however, institutions accept if the drug is compared to daily treatment routine or the least expensive therapy. This includes other pharmaceuticals as well as non-pharmaceutical interventions and doing nothing. Concerning drugs the daily treatment routine should be identified by prescription or sales volumes. Dosage and treatment regime of medicines need to be therapeutically equivalent. Few institutions however, provide details how to determine the common practice of a non-pharmaceutical treatment: NICE in England/Wales and CEDAC in Canada specify that the choice of comparator should be based on clinical practice guidelines or on sales volumes. In Australia expert opinions are considered to select the appropriate comparator. Institutions demand that manufactures and reviewers follow the guidelines as closely as possible when choosing a comparator. It is therefore recommended to contact the institutions early, i.e. in the phase of study design.

B. Other institutions (in Austria, in Canada (PMPRB), and in Switzerland) require that the pharmaceutical should be compared to all medicines of the same therapeutic group - based on the ATC (Anatomical-Therapeutic-Chemical)-classification of the World Health Organization. Thus, only currently reimbursed or marketed pharmaceuticals are considered as comparators. For each reference medicine an equivalent dosage form should be defined.

France combines both procedures: From all medicines in the same therapeutic group the most often prescribed, the least expensive and the most recently listed (positive list) are selected for comparison.

If a pharmaceutical provides several approved indications, review bodies in Canada (CEDAC), in Sweden, in the Netherlands, in Norway, and in the United Kingdom commonly evaluate the drug for all of them. CEDAC in Canada is the only institution which also considers a drug's off-label use. In Austria, in Australia, and Canada (PMPRB) a medicine is only evaluated for its main or primary indication, i.e. either the indication (presumably) prescribed for the largest proportion of the population (in Austria and in Australia) or the indication offering the most therapeutic benefit compared to treatment alternatives.

Transferability of results derived from RCT to a country's daily care setting (external validity) is often limited. This can be due to several reasons: an experimental study design, i.e. artificial treatment setting; a time horizon (i.e. follow up) that is too short; a study population that is not representative for the general population; and inadequate choice of outcome parameter and comparator. Additionally, it is well known that clinical trials and pharmacoeconomic studies that directly compare a drug with adequate competitors are lacking, in quantitative as well as in qualitative terms. This is specifically the case before or shortly after marketing authorisation. Yet there is the need to support reimbursement or pricing decisions by scientific evidence. In this situation review bodies are often forced to rely on observational studies or other internally less valid data (including expert and consensus opinions). As a second option they use statistical approaches like indirect adjusted comparisons (in Australia and the United Kingdom) and, more commonly, economic modelling. Institutions have defined strict requirements for modelling and indirect comparison techniques. However, there is consensus that results provided by these methods need to be verified by valid head-to-head comparisons as soon as possible.

Thus many review bodies perform a re-evaluation after a certain period of time. Re-assessment is either done according to a fixed time schedule (e.g. in Finland and in France) or if new evidence has become available (e.g. in Austria and in the United Kingdom).

5. Conclusions

Most evidence concerning post-licensing evaluation of pharmaceuticals is derived from non-European countries like Australia and Canada. However, an increasing number of EU-countries also look back on several years' experience in comparative drug evaluation in connection with reimbursement or pricing decisions. There is consensus internationally that a new drug which is the first product offering an effective therapy for a particular indication should be classified as a major improvement. If a treatment alternative already exists, the majority of countries perform a systematic and evidence-based evaluation of a drug's clinical and economic characteristics in comparison to daily treatment routine. In the case of lacking or unreliable evidence results of a comparative drug assessment are considered as of preliminary nature. However, further evaluation criteria, requirements and specific methodological issues still lack internationally consented standards.


Articles from GMS Health Technology Assessment are provided here courtesy of German Medical Science

RESOURCES