Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community

Abstract

Background

Incremental cost-effectiveness and cost-utility analyses [health economic evaluations (HEEs)] of vaccines are routinely considered in decision making on immunization in various industrialized countries. While guidelines advocating more standardization of such HEEs (mainly for curative drugs) exist, several immunization-specific aspects (e.g. indirect effects or discounting approach) are still a subject of debate within the scientific community.

Objective

The objective of this study was to develop a consensus framework for HEEs of vaccines to support the development of national guidelines in Europe.

Methods

A systematic literature review was conducted to identify prevailing issues related to HEEs of vaccines. Furthermore, European experts in the field of health economics and immunization decision making were nominated and asked to select relevant aspects for discussion. Based on this, a workshop was held with these experts. Aspects on ‘mathematical modelling’, ‘health economics’ and ‘decision making’ were debated in group-work sessions (GWS) to formulate recommendations and/or—if applicable—to state ‘pros’ and ‘contras’.

Results

A total of 13 different aspects were identified for modelling and HEE: model selection, time horizon of models, natural disease history, measures of vaccine-induced protection, duration of vaccine-induced protection, indirect effects apart from herd protection, target population, model calibration and validation, handling uncertainty, discounting, health-related quality of life, cost components, and perspectives. For decision making, there were four aspects regarding the purpose and the integration of HEEs of vaccines in decision making as well as the variation of parameters within uncertainty analyses and the reporting of results from HEEs. For each aspect, background information and an expert consensus were formulated.

Conclusions

There was consensus that when HEEs are used to prioritize healthcare funding, this should be done in a consistent way across all interventions, including vaccines. However, proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs. Due to the complexity of and uncertainties around vaccination, transparency in the documentation of HEEs and during subsequent decision making is essential.

Key Points for Decision Makers

Health economic evaluations (HEEs) on vaccines and vaccination programmes should always be considered by decision-making bodies when considering inclusion of a new vaccine into the national programme to avoid suboptimal allocation of resources.

Proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs in HEEs. However, vaccines should only be treated differently where they really are different (e.g. indirect effects).

Funders and decision-makers should recognize that proper and valid HEEs (of vaccines) demand time and resources.

Introduction

Health economic evaluations (HEEs) are widely used to evaluate new health technologies, and several guidelines of the World Health Organization (WHO) and other authorities exist and provide guidance on how to properly conduct such analyses [1–8]. Furthermore, many national immunization technical advisory groups (NITAGs) and/or respective national institutions in high-income countries routinely consider results from HEEs for the evaluation of vaccines (Fig. 1 [9–14]). However, comprehensive guidelines or frameworks on (1) how to best conduct HEEs of vaccines and (2) how to implement their findings into immunization decision making are scarce. Members of a European vaccine economics community developed a framework on how to deal with several relevant aspects in the field of HEEs of vaccines. The aim of this paper was to provide a consensus framework on how to apply HEEs to vaccines and to identify areas where further work is needed to reach harmonization. In this paper, we focused on vaccines for the prevention of infectious diseases, on aspects relevant to high-income countries, and on incremental cost-effectiveness and cost-utility analyses that serve as reference case in most European healthcare reimbursement systems.

Fig. 1.

Map of Europe indicating whether results from economic evaluations of vaccines are routinely considered in recommendations, based on [9, 11–14] (asterisk no response from country, country not considered, Malta and Cypress not shown)

Methods

Experts (authors of this manuscript) with expertise in mathematical modelling, health economics and/or immunization decision making from Europe were invited by the Robert Koch Institute (RKI, Germany) to develop a framework on methods for HEE of vaccines and immunization decision making. European experts from academia, national public health or health technology assessment bodies as well as from relevant national and international health authorities were selected by following criteria: (1) authorship of key publications and/or (2) (former) membership of a NITAG and/or (3) (former) employee of a (inter-) national decision-making body and/or (4) (former) employee of a health technology assessment body. To minimize potential and perceived conflicts of interest, experts employed by a pharmaceutical company were not invited to this meeting. However, since several of the invited experts had experience in conducting HEEs commissioned by industry or interacting with industry, e.g. during scientific debates, a comprehensive view on relevant aspects was ensured.

Systematic Literature Review and Preparation of a Workshop

After obtaining feedback from the expert group, a systematic literature review on methodologies and guidelines related to HEEs of vaccines was conducted. The databases MEDLINE and EMBASE were searched with a time horizon from 1 January 1990 to 21 February 2014 targeting five topics: ‘Vaccines and infectious diseases’, ‘economic evaluation’, ‘guidelines’, ‘methods’ and ‘decision making’. Search terms identified in study title and/or abstract within each topic were connected with an ‘OR’ (see Table 1). Three search branches were developed. In each search, branch topics, ‘Vaccines and infectious diseases’ and ‘economic evaluation’ were connected with an ‘AND’ and further connected (‘AND’) with the following:

• ‘Guidelines’ (search branch 1) OR

• ‘Methods’ (search branch 2) OR

• ‘Decision making’ (search branch 3).

Table 1.

Systematic literature research

Topics	Vaccines and infectious diseases	Economic evaluation	Guidelines	Methods	Decision making
Search terms (in Title OR Abstract)	Vaccine OR Vaccination OR Vaccinate OR Vaccinating OR Vaccinated OR Immunization OR Immunisation OR Immunize OR Immunise OR Infectious disease OR Communicable disease OR Preventable disease	Cost OR Cost effectiveness OR Cost utility OR Cost benefit OR Benefit cost OR Cost saving OR Economic OR Pharmacoeconomic OR Pharmacoeconomics OR Budget impact OR Efficiency OR Efficient OR Monetary OR Financial OR ICER OR QALY	Guideline OR Guide OR Good-practice OR Good OR practice OR Good research practice OR Standards OR Standard OR Recommendation OR Recommendations OR Framework OR Frameworks OR Primer OR Consensus	Methoda OR Methodsa OR Methodological OR Decision analytic OR Decision analysis OR Decision analyses OR Modela OR Modelsa OR Modellinga OR Modelinga OR Model based OR Simulation OR Simulation OR Mathematical OR Transmission OR Dynamic OR Discounting OR Interaction OR Herd immunity OR Herd protection OR Herd effects OR Indirect effects OR Population-wide benefits OR Waning	Decision making OR Reimbursement OR Fourth hurdle OR Payer OR Pricing OR Funding OR Willingness to pay OR Threshold OR Value for money OR Social value OR Social preferences OR Public health
Search branch 1	•	AND •	AND •
Search branch 2	•	AND •		AND •
Search branch ‘methods’ (1 + 2)	•	AND •	AND •	AND •
Search branch ‘decision making’ (3)	•	AND •			AND •

^aSearched in title only

Studies were excluded on the following criteria:

• they did not have a methodological purpose, or

• they did not have a vaccine context, or

• they considered exclusively non-industrialized countries.

Titles and abstracts of studies were screened independently by two reviewers. Potentially relevant studies were retrieved and assessed according to the three exclusion criteria by reviewers. Disagreements between both reviewers on exclusion of particular studies were resolved by consensus. Search branches 1 and 2 were further combined with an AND. Additionally, studies found by hand search (e.g. by the ‘snowballing technique’) were included. Finally, the identified literature was analysed for prevailing opinions and/or remaining questions.

Based on the included studies, pertinent aspects were identified and experts were asked to select the most relevant aspects in which currently required forms of HEE [mainly incremental cost-effectiveness ratio (ICER) calculations] of vaccines preventing infectious diseases differ from HEE of curative drugs. According to the experts’ feedback, aspects were selected for discussion in a 2-day workshop. With this approach of combining systematic reviews and subsequent expert consultation, a coverage of most relevant aspects was achieved, including differences between HEE of curative drugs and HEE of preventive vaccines.

Conduct of a Workshop

After an introduction and key note lectures, aspects in the area of (1) modelling methods, (2) health economics, and (3) decision making were discussed in group-work sessions (GWS). Findings from the literature review were used to guide discussions during GWS. Based on a workshop reader (available online [15]) the groups on ‘modelling methods’ and ‘health economics’ were asked to provide—where possible—concrete ‘recommendations/suggestions’ for use of a specific item per aspect and/or were asked—if applicable—to list ‘pros and contras’ as well as ‘future challenges’ of certain items within an aspect. For the ‘decision making’ GWS, a more explorative approach was chosen. Findings from GWS were then presented and discussed in plenary. Besides the identification of relevant aspects and remaining issues, the experts were asked to formulate consensus on how to handle certain aspects. Consensuses and compromises were achieved through expert discussions and voting. Where no consensus could be achieved, experts were asked to present different options or points of view.

Results

Identified Studies

The flow chart in Fig. 2 describes the identification process of the relevant studies. Search branch ‘methods’ (branch 1 and 2) resulted in the identification of 42 [16–57] studies. Search branch ‘decision making’ (branch 3) identified 26 studies [9, 21, 44, 58–80].

Fig. 2.

Flow chart of the systematic literature review

Identified Aspects for Discussion

Based on the identified studies and experts’ feedback, 17 aspects were selected for discussion in the workshop. The results of expert discussion on each aspect are presented in Sects. 3.3.1–3.3.13 and 3.4.1–3.4.4.

Modelling Methods and Health Economics

Model Choice

Background [17, 18, 20, 22–25, 27, 28, 30, 32, 34, 37, 42, 44, 46, 47, 54]:

There are several types of models:

• i.Static cohort models;
  1. Decision tree,
  2. Markov model.
• ii.Population models;
  1. Static model,
  2. Dynamic model:
     1. Compartmental transmission dynamic models,
     2. Agent-based models,
     3. Discrete-event models.

Models from the category (ii-b) can intrinsically account for transmission of pathogens between individuals or population fractions that influence diseases transmission by indirect effects—dynamic models. Models from categories (i) and most of (ii) are often of a deterministic nature. Input parameters are set deterministically, and base-case results and uncertainty analyses are fully replicable. Models from category (ii-b-2 and ii-b-3) usually use a stochastic approach. For example, a study from Spain analysed the health economic effects of a seasonal influenza vaccination by using both a static and a dynamic model approach [46]. In contrast to the dynamic model’s results, the static model, neglecting indirect effects (such as herd protection), could not show that the influenza vaccination is cost saving. Hence, using a static model might underestimate a vaccine’s value [46].

Expert consensus:

• For infectious disease modelling, the sole use of static models should always be justified. Static models can be used as a conservative estimate when there is no evidence for harm (e.g. age shifts with adverse effects) if indirect effects are ignored [4, 30, 34, 81, 82]. WHO has developed a flow chart that provides assistance when choosing an adequate type of model (Fig. 3) [4].

• A challenge is to handle realistic demographic predictions in models (with a long time horizon) because of migration, demographic changes and scarcity of contact studies in (special) populations.

• Stochastic models;

Fig. 3.

Flow chart for model choice adopted from the World Health Organization [4]

Advantages:

• Simulate a more realistic world.

• Can follow an individual’s life course, which is easier for decision-makers to understand.

• In stochastic models, the randomness is of first-order uncertainty; therefore, they provide an alternative to account for heterogeneity (if events are not rare) in subgroups as it is done in a deterministic model [81].

Disadvantages:

• Model calibration and the probabilistic sensitivity analysis (PSA) become more challenging and computationally intensive, hence, transparency might suffer.

• Data sources may not be accessible, as more fine-grained (non-aggregated) data are needed.

• A remaining challenge is to find adequate ways to conduct efficient uncertainty analyses on stochastic models [83].

Time Horizon of Models

Background [18, 27, 34, 37]:

Outcomes of static cohort models are typically estimated over a single cohort’s lifetime or a specific time horizon in which the disease typically occurs that is set as the time horizon of the model. In dynamic population-based models, the entire population is modelled, and the time horizon of those models has three phases:

1. Run-in phase Dynamic models need a run-in phase to model the epidemiological conditions in the pre-vaccination era. This is important for a realistic implementation of the respective vaccination. The duration of the run-in phase impacts the model’s results.

2. Evaluation phase This phase starts with the implementation of the vaccination. The duration of the phase should be set in the model as long as necessary to cover indirect (positive or negative) effects caused by the vaccination in the population.

3. Steady-state After a certain duration of the evaluation, an epidemiological equilibrium is reached—the steady state—where the epidemiological variation terminates.

Combining model choice with time horizon, Mauskopf et al. [37] identified four categories of evaluation strategies (see Table 2). For example, a cost-effectiveness analysis of human papillomavirus (HPV) vaccination for Austria showed in sensitivity analyses that an extension of the model’s time horizon by almost three decades decreased the ICER from €50,000/life-years gained to <€0/life-years gained [84].

Table 2.

Evaluation strategies of health economic models [37]

Evaluation strategy	Target population	Time horizon	Start of evaluation
Category 1	Entire population	Fixed TH (several years or decades)	From implementation of vaccination
Category 2	Entire population	1 year	From steady state
Category 3	Cohorta	Cohort’s lifetime or fixed TH	From implementation of vaccination
Category 4	Cohorta	Cohort’s lifetime or fixed TH	From steady state

TH time horizon

^aOne or more cohorts

Expert consensus:

• The time horizon of a dynamic model should last until the steady state is achieved in order to deliver valuable results. Hence, the model’s time horizon should not be defined prior to the analysis.

• Dynamic population models should be evaluated by category 1 (Table 2). ICERs should be projected over time until their cumulative value stabilizes. However, if this takes more than two decades, cumulative ICERs should be presented for different time horizons (e.g. Kim and Goldie [34]).

• Future research must analyse the impact of these different strategies.

• A model usually should be run to epidemiological equilibrium, but it ideally should reproduce historical epidemiological (and demographic) values that may not be in equilibrium. There is a need for validation of pre-vaccination as well as post-vaccination epidemiology (whatever is available/applicable). However, this is often technically challenging and may suffer from lack of data.

• A surveillance system should be implemented to monitor the impact of a vaccination programme. The results can be used to compare the real impact with the impact predicted in the model that was developed before the vaccine was implemented and to evaluate the model.

Natural History of Disease

Background [30, 52]:

Pathogen-specific naturally acquired immunity (including its waning) is usually an important feature in dynamic models. The way this is modelled has an influence on the results [exemplary model structures: ‘susceptible-infectious-susceptible’ (SIS), ‘susceptible-infectious-recovered’ (SIR), ‘susceptible-infectious-recovered-susceptible’ (SIRS), ‘susceptible-exposed-infectious-recovered-susceptible’ (SEIRS)]. In principle, the structure of a model should be developed on the base of the characteristics of the specific target disease, the vaccine of interest and the respective research question. However, for the same vaccine-preventable disease different compartment structures have been used in published studies. For example, a study analysing the health economic effects of seasonal influenza vaccination in Spain used an SIR approach, whereas a study considering the same disease in England used an SEIR structure [46, 85]. Particularly in health economic models for HPV vaccination, the use of an SIR and the use of an SIS structure are common, and the use of both structures can be justified [52, 86].

Expert consensus:

• The model’s structure should represent the state of knowledge on the specific disease.

• If uncertainty exists, structural uncertainty analysis should be performed [40, 87].

• Better (sero-) epidemiological and immunological data are needed to inform such models.

Measures of Vaccine-Induced Protection

Background [16, 29, 42, 45, 54]:

Different vaccine efficacy (VE) measures can be considered in HEEs to account for protection against different outcomes such as infection, symptomatic illness/complication, and/or infectiousness.

If an infection does not cause any symptoms at all at any stage, and thereby does not require treatment and/or prophylaxis costs, there is no need to consider this in an economic assessment. For example, based on data from influenza challenge studies, Basta et al. [16] estimated absolute VEs for different endpoints. For example, vaccine efficacy for seasonal live-attenuated influenza vaccines in homologous seasons ranged between 40 and 90 % (VE for susceptibility 40 %; VE for infectiousness 50 %; VE for illness given infection 83 %; VE for infection-confirmed influenza illness 90 %).

When developing a model, the hierarchy of disease states/endpoints must be incorporated to consider the relevant type of VE.

• There is a ‘sequential’ [targeting the first endpoint only, e.g. VE protecting against herpes zoster (HZ) only]

• A ‘non-sequential’ [targeting all VE-relevant endpoints independently, e.g. VE protecting against HZ and also against HZ complication post-herpetic neuralgia (PHN)] approach

A study evaluating the cost effectiveness of a vaccine preventing HZ in the USA used, for instance, a sequential approach [88]. Hence, only VE against HZ was considered, and VE against the complication PHN was neglected. However, this approach was criticized by Brisson and colleagues [89, 90], who used a non-sequential approach when evaluating the cost effectiveness of an HZ vaccine in Canada. In this study, the model was informed with both VE against HZ and VE against PHN, and delivered results more in favour for the vaccine [90].

In clinical trials, two different approaches of analysis are often used to measure VE:

• per protocol (PP) and

• intention to treat (ITT).

PP usually produces more favourable VE results for the intervention/vaccine than ITT.

Furthermore, in models, the degree of protection and take can be distinguished.

• The degree of protection (or leaky protection) is the percentage of (partial) protection in successfully vaccinated individuals (e.g. 100 % of vaccinated individuals have a protection of 50 %).

• Take (or ‘all or nothing’) describes the percentage of successfully vaccinated individuals with full protection (e.g. 50 % of vaccinated individuals have a protection of 100 %).

A mathematical model, for instance, calculated the impacts of fictional HIV vaccines on seroprevalence, depending on whether the vaccine leads to a take or a degree protection. The seroprevalence proportion was, after several decades, lower when using a take protection than a degree protection [91]. This difference can impact the overall results of a health economic model [92].

Finally, a major challenge is a lack of clinical endpoints in clinical trials, especially when immunogenicity (or other surrogate of protection) is the outcome considered for licensure. The preferences, whether to use efficacy or effectiveness data in HEEs, are rather diverse in European guidelines [14].

Expert consensus:

• The model structure should account for the type of VE measure incorporated. VE in terms of reducing susceptibility to infection is fundamentally different to VE reducing infectiousness. These different aspects of VE have a differential impact on the results. Modelling can be used to estimate unknown parameters including VE estimates by using, for example, a Bayesian framework utilizing Markov Chain Monte Carlo inference [93]. More studies assessing VE against infectiousness are warranted (e.g. challenge studies).

• Model structure and the decision-maker’s research question determine the use of a sequential or non-sequential approach.

• ITT data, when available, should be taken for base-case analyses and PP data for uncertainty analyses. However, the use of PP data for the base-case is sufficient when the difference between ITT and PP data is completely explained by the different proportions of susceptible individuals in the study population, since this is ideally incorporated in a model. PP data should be preferably chosen if a specific result on vaccine-dose compliance and/or completion of a vaccine course is of relevance.

• The choice of representing VE with degree of protection versus take depends on the type of protection conferred by the vaccine of interest. When there is no evidence on whether the vaccine confers a leaky or an all-or-nothing protection, different approaches to account for vaccine efficacy (leaky or all-or-nothing or combination) should be used.

• The quantitative relationship between immune response and the degree of vaccine-induced protection against clinical disease is often unclear. Validated surrogates can be considered if no clinical endpoints are available.

• The impact of negative vaccine effects, both at an individual level (i.e. adverse events) and at a population level (i.e. replacement or age shift) needs to be considered. Cases of vaccine-preventable diseases and cases of adverse events are equally relevant outcomes.

• Vaccine manufacturers currently have little incentive to collect some specific clinical data (e.g. head-to-head comparisons of different vaccine products); however, these are relevant for modelling and public health. For comparison of different studies, standardized case definitions for clinical outcomes are needed, and methods for the implementation of such indirect comparisons into models should be developed/standardized.

Duration of Vaccine-Induced Protection

Background [30, 52]:

The waning of vaccine-induced protection plays a major role when modelling vaccine effects. Clinical trials are often too short to generate robust data on the duration of vaccine protection. In a model, either a lifetime duration (neglecting waning) is assumed or waning is incorporated. Waning can be designed to start right after vaccination or after a delay period, and to decay in different ways (e.g. exponential, stepwise). With the example of HZ vaccination, it can be shown that different assumptions on the VE waning rate can impact the health economic results: in a study from Germany, vaccination at the age of 60 years resulted in an ICER of €21,565/quality-adjusted life-year (QALY) gained or €34,606/QALY gained assuming an annual waning rate of 1 or 20 %, respectively [94]. In this model, when vaccinating individuals at the age of 50 years, the same waning rates led to €18,486 and €43,701/QALY gained, respectively. Especially in health economic models on HPV vaccination, the waning of vaccine-induced protection is an influential determinant of overall results [52, 86, 95].

Expert consensus:

• If vaccine waning is not well understood, then different waning scenarios should be considered in an uncertainty analysis and their impact compared. Immunological memory can be integrated.

• The availability of detailed trial data on vaccine-induced protection and waning at the patient level would enable more rapid and less uncertain economic evaluations following the marketing of a new vaccine.

Indirect Effects Apart from Herd Protection

Background [18, 20, 27, 30, 37]:

Vaccination-specific negative externalities such as age shift of peak incidence, serotype replacement or impact on antibiotic resistance might be relevant. The cost effectiveness of pneumococcal vaccination, for example, is strongly influenced by the degree of serotype replacement that is expected after vaccine introduction. This influence was shown by van Hoek et al. [96] in a study on the cost effectiveness of a 13-valent pneumococcal conjugate vaccination for infants in England. Brisson and Edmunds [97, 98] illustrated that the routine infant vaccination against varicella is expected to increase the average age at infection. This age shift is caused by two effects: the cohort effect and the herd protection effect. If the disease severity increases with age at infection, this age shift negatively impacts the overall health-economic results.

Expert consensus:

• Ecological effects such as intra-population immune boosting following exposure to a pathogen (cf. varicella-zoster virus), replacement of pathogen strains covered by a vaccine by strains not covered (e.g. pneumococcal serotypes) and antibiotic resistance should be part of uncertainty analyses whenever they are possibilities.

• Ideally, pathogen replacement, eradication, genetic selection in host, changes in behaviour (e.g. screening uptake, risk behaviour such as unprotected sex or social mixing), weakening of maternal immunity, and using vaccination as a platform for adding other interventions should also be considered wherever relevant.

Target Population

Background [20, 23, 29, 30, 42, 43]:

Besides the question of whom to vaccinate (e.g. total population at a certain age versus risk groups) it is necessary to consider how populations/groups mix with each other. In dynamic models, contact patterns and the mixing matrix influence the model outcome. The literature offers data gathered by different approaches, e.g. survey-based (e.g. POLYMOD) and synthetic (social demographic data) contact matrices. For example, to understand contact patterns of infants, a study group sent contact diaries to a representative sample of mothers in the UK. Thereby, the average number, the type, and the duration of daily contacts of infants were documented. These data can serve as important input parameters for dynamic models evaluating the impact of vaccines [99]. Since observational data on contact patterns in the form of age-specific contact matrices are difficult to gather and are currently available only for few countries, Fumanelli et al. [100] presented a computational approach. Based on the simulation of a virtual society of agents based on data collected in eight European countries, contact patterns by age can be estimated for 26 European countries. This approach might serve as a valuable alternative to observational data [100]. Such synthetic contact matrices have been used by Poletti et al. [101] in a varicella-zoster virus study.

Expert consensus:

• Survey-based mixing data are considered most adequate and should be used wherever possible. Synthetic methods need to be validated against survey data where available.

• Future research should better measure contact patterns for children and parents, and evaluate what kind of contact is relevant. Knowledge is still lacking about how well contact patterns represent occasions for transmission for specific infections. It is therefore important to assess which subset of mixing data (e.g. contacts involving touching) provides the best fit to relevant observational data for the disease in question (e.g. seroprevalence data). Research funders need to understand that contact matrices are important for dynamic model projections, and that these differ between countries or regions.

Model Calibration and Validation

Background [18, 23, 34, 40, 42, 50, 51, 53, 55]:

Calibration means estimating and adjusting a model’s parameters to generate the expected outcomes observed in real life. The literature reports several approaches such as manual, random (e.g. Monte Carlo) or optimized (e.g. Nelder-Mead [102]) approaches. External validation compares the results of the model with the best available evidence. For example, Kim et al. [103] present calibration methods on how to use real-world data to develop a comprehensive natural history model of HPV. Hence, a calibrated model that fits to real-world data tends to generate more reliable results [103]. However, Basu and Galvani [104] claim that a Bayesian approach provides several advantages compared with the approaches presented by Kim et al. [103].

Expert consensus:

• The manual calibration approach should be based on a structured process, and the algorithm should be reported. However, the random or optimized approach is considered to be more adequate [53, 105–107]. A random calibration approach may have an identifiability issue. Hence, researchers should make sure that the shape and range of the posterior distribution is plausible.

• A plain visual validation is not considered sufficient. Instead, the use of goodness-of-fit criteria is recommended.

• The dataset used for validation should be independent from that used for calibration (ideally even with different endpoints). An alternative is to hold back a portion of the calibration data (e.g. test/training datasets). An alternative option is a cross-model validation approach in which the same data are used on different models. Lack of data points might be a limitation.

Handling Uncertainty

Background [18, 22, 26, 30, 34, 37, 42, 45, 48, 54]:

Structural (or model) uncertainty can be handled by scenario analysis (i.e. presenting results for different models), model averaging or parameterizing the structural uncertainty [108–110]. Parameter uncertainty is quantified with PSA. However, PSAs are often not performed in dynamic models because they are computationally difficult (especially to include the uncertainty of parameters affecting transmission). Alternatively, parameters affecting transmission are excluded from PSA, or two models are developed: an economic sub-model including PSA and a dynamic sub-model focusing on transmission-specific issues. Furthermore, there are many vaccination-specific key aspects for uncertainty analysis, such as duration of vaccine protection, vaccination coverage in dynamic models, time horizon, boosting, contact patterns and targeted age/risk groups. For example, to consider both indirect effects caused by vaccination and PSA, Christensen et al. [111] developed two models (static and dynamic) to evaluate the potential impact of introducing vaccination against serogroup B meningococcal disease in the UK. The static model without parameters affecting the transmission was used to perform PSAs. The dynamic model including parameters affecting the transmission was used to account for indirect effects [111].

Expert consensus:

• All identifiable sources of uncertainty should be accounted for, if not by PSA then by other analyses. The parameter distributions used in PSA need to be justified. Transparency is important because dynamic models can have many ‘deep parameters’ (i.e. parameters that are not directly observable, such as the probability of infection transmission per contact event). Calibrated parameters also need to capture information about uncertainty [87, 112].

• Decision-makers need to understand the relevance of uncertainty analyses. PSA can help to identify future research priorities. However, uncertainty measures in calibrated parameters remain challenging.

• Structural uncertainty should be parameterized where possible, but uncertainty in normative aspects such as perspective, vaccine price and discounting should not be analysed in PSA [87, 112]. Vaccine coverage should be varied between desirable and undesirable levels. Uncertainty in contact patterns should be parameterized wherever possible (see Table 3).

Table 3.

Definition of uncertainty analyses based on expert consensus and literature [22, 30, 42, 55, 87, 112]

Type of uncertainty	Sensitivity analysis			Scenario analysis
	Parameter uncertainty	Methodological/normative uncertainty	Structural/model uncertainty
Deterministic sensitivity analysis	Yes	Yes	Yes	Yes
PSA	Yes	NA	NA	NA
Examples	• Efficacy • Costs	• Transmission dynamic vs. discrete-event simulations • Discount rate	• Presence of a immune state (SIS vs. SIR) • Pathogen strain competition and replacement	• Coverage • Target age/risk group • Vaccine price

NA not applicable, PSA probabilistic sensitivity analysis, SIR susceptible-infectious-recovered, SIS susceptible-infectious-susceptible

Discounting

Background [1, 18, 19, 25, 35, 38, 39, 41, 42, 44, 45, 49, 54, 56, 57, 113]:

Uniform discounting (using the same rates for costs and health effects) is most commonly applied in HEEs. However, this approach might considerably influence the long-term ICERs of vaccination in particular, since costs and benefits usually occur at different points of time. Differential discounting (with lower rates for health effects) is used, for example, in Belgium, the Netherlands, Poland and, until 2004, the UK, for the assessment of all health technologies, including vaccines. Differential discounting seems to be technically feasible and fairer from an ‘inter-generational’ perspective [114]. Different recommendations on discounting approaches, for example, drugs and vaccines within one country, do not exist. If lower discount rates for health effects are considered, the level of discount rate for health effects needs to be set separately. Another issue in this context is whether a constant or a changing discount rate over time is used. Constant rates are more widely used, possibly because of pragmatism and ease. In France, the first 30 years are discounted with a uniform rate of 4 %, and years thereafter at 2 %. This ‘slow’ discounting procedure is also recommended by WHO if the effects of vaccination begin only long after the intervention (e.g. vaccination against HPV) [4]. The type of decline can vary (e.g. stepwise, linear, or exponential). For example, Westra et al. [57] used nine different discounting approaches and rates in a model evaluating the cost effectiveness of the HPV vaccination in the Netherlands. Ceteris paribus, the ICERs ranged between €7,600/QALY gained and €165,400/QALY gained.

Expert opinion1:

• The majority of, but not all, experts recommended differential discount rates for costs and effects in HHEs (exclusively in cost-utility and cost-effectiveness analysis) if the model’s time horizon is long (e.g. >20 years).

• The discount rate of health effects could be around 50 % of the discount rate for costs. However, for a more evidence-based recommendation, empirical research has to be conducted [39].

• Constant discount rates over time should not be applied in models with a long time horizon (e.g. >20 years) according to a majority of experts.

• Further research on the, to date rarely applied, approach called time-shifted discounting approach is needed [39, 57, 115].

• Since discount rates and discount approaches usually have a major impact on results of HEE of vaccines, the variation of these aspects need to be analysed (see Sect. 3.3.9) and explained to decision-makers.

Health-Related Quality of Life and Quality-Adjusted Life-Years as Outcome Measures

Background [20, 25, 32]:

QALYs are a common outcome measure used in cost-utility analyses and are accepted in several countries in Europe [14]. Since vaccine-preventable diseases often affect children, the impact on health-related quality of life (HR-QOL) of carers measured as QALYs is important in HEEs of vaccines. Another prevention-specific issue is utility in anticipation, measured as QALYs. Vaccinated individuals might experience a higher HR-QOL, because they feel ‘protected’ after vaccination. As counterpart, fear of adverse events measured as QALYs also needs to be mentioned. That might lead to an HR-QOL decrease, but on a rather short time horizon. For example, Weinke et al. [116] used a survey to assess the impact of HZ and PHN on the patients’ life but also on life of the family members who cared for them. Most family members (69 % children; 80 % life partners) of patients with HZ or PHN reported that caring for the patient caused a moderate to severe impact on their life. Hence, these impacts might have an effect on overall health economics results [116].

Expert consensus:

• HR-QOL of carers should routinely be considered in uncertainty analysis in both payers’ and societal perspectives. However, input data might be scarce.

• Utility in anticipation and fear of adverse events are also difficult to consider due to limited data. When appropriate data are available, they might be considered in uncertainty analyses.

Cost Components

Background [18, 32, 36]:

Even though the literature and national guidelines on (direct or indirect) cost components seem very comprehensive, some factors are vaccine specific and require attention. Traditionally, vaccines mainly protect against diseases occurring in childhood. Since sick children need care, indirect costs of carers (e.g. productivity loss) occur. Set-up costs (e.g. of vaccine campaigns) might also play an important role in immunization programmes. Depending on the disease and the target group, campaigns are needed to reach as many individuals as possible. Respective national guidelines often neglect such cost components. For example, in the model by Hornberger and Robertus [117], that evaluated the cost effectiveness of HZ vaccination in the USA, the price for the vaccine included the unit vaccine cost, a public awareness campaign, administration costs, patient travel time and time receiving vaccine as well as the cost of treating adverse events. Implementing $US500 per vaccine dose caused high ICERs of between $US280,000 and $US560,000/QALY gained [117].

Expert consensus:

• Indirect costs of carers should be considered for both perspectives. From a payers’ perspective, they are considered as sick pay (if the payer has to cover these costs) and from a societal perspective as productivity loss.

• If set-up costs (e.g. for campaigns) are not included in the vaccine price (i.e. promotion and distribution of a vaccine is not done by its manufacturer), they should be considered in the perspective that covers these costs.

Perspectives

Background [18–20, 32, 54]:

Some prevailing methodological primers recommend using the societal perspective when assessing vaccination programmes designed to improve public health. A recent systematic literature review on varicella and HZ vaccination concluded that the varicella vaccination (neglecting the impact on HZ) becomes cost saving when switching from a payer to a societal perspective [118]. Hence, indirect costs especially of carers when considering infant vaccination, tend to have a major impact on overall health economic results [118]. However, for example, if costs for vaccination are higher from a societal perspective due to co-payments, the overall results from the societal perspective can become less cost effective when compared with those from the payer perspective [119, 120].

Expert consensus:

• A societal perspective should ideally be taken for the base-case analysis when considering infectious diseases (i.e. not for vaccines exclusively), unless this contradicts national guidelines.

Decision Making

Background [9, 24, 41, 58–70, 72–79]:

Purposes of Health Economic Evaluations in Decision Making

From the decision-makers’ perspective, results from HEE of vaccines can be utilized to identify the most efficient vaccination strategies (e.g. targeting the total population or specific age or risk groups—technical efficiency), to support yes/no decisions for vaccine introduction, and/or to support price negotiations with manufacturers (allocative efficiency). Furthermore, the introduction of a new vaccine usually substitutes for curative treatment and/or screening measures in a healthcare system. Those substitutions are also of relevance for decision-makers and are usually considered in HEEs. Furthermore, the experts clearly stated that additional budget impact analyses (BIA) can be useful, especially when a costs of a prevention/intervention measure is extremely high and affordability is unclear.

Integration of Health Economic Results in Decision-Making Processes

The introduction of a new vaccine into a healthcare system affects several aspects of the system. Hence, many questions, such as acceptance, practicability or equity in access, must be addressed. Erickson et al. [62] provide a comprehensive view on all factors that could be considered in immunization decision making. A broad micro- and macro-economic view might be useful. Many but not all questions can be addressed within a health economic model, but should also be considered within a broader appraisal. However, details of a broader appraisal were not subject to this framework. Furthermore, other health economic approaches, such as cost-benefit analyses or BIAs, may also serve as a basis for decision making. However, which or how many approach(es) are considered for decision making is a rather normative question that needs to be address by relevant stakeholders at the national level. Therefore, experts concentrated in this framework on incremental cost-effectiveness and cost-utility analyses, that are the most commonly used forms of HEEs for national decision making in Europe [14]. Yet, the experts acknowledged the usefulness of broader and alternate approaches in the economic analysis of vaccines.

In general, the experts cautioned that lack of transparency and high complexity of evaluations leads to results from HEEs appearing like a black box to decision-makers. Based on a decision making continuum, experts stated pros and contras of three decision making approaches: pure threshold, multi-criteria decision analysis (MCDA), and informal judgment (Fig. 4). Decision making based on a pure threshold is transparent. Especially if the budget is very constrained, a fair use of public resources is possible and it is a useful tool for price negotiations with manufacturers. However, pure threshold does not consider clinical/epidemiological severity or budget impact. Furthermore, the choice of the level or range of the threshold sometimes seems rather arbitrary, and in some countries such willingness-to-pay thresholds are not accepted. MCDA is also transparent and applicable without a threshold. However, MCDA is very complex, needs a well-developed design and requires proportional weights of each criterion. Clearly, in terms of MCDA, more research is needed. In terms of informal judgment, experts recognized advantages such as applicability without a threshold and comprehensive summary of many parameters related to the respective vaccine and disease, including issues that are difficult to quantify (e.g. implementation issues or acceptability of a vaccine).

Fig. 4.

Decision making continuum, adapted from expert-group discussion. HEE health economic evaluation, MCDA multi-criteria decision analysis

Key Parameters that Should be Varied in Uncertainty Analyses

The experts made clear that parameter uncertainty should be considered in PSA (see Table 3; Sect. 3.3.9). Additionally, the experts recommended including the variation of age and risk group, vaccination schedule, herd protection, booster or catch-up vaccination, delivery strategy and vaccination coverage in scenario/uncertainty analyses.

Vaccination-Specific Aspects of Reporting Results

Experts listed aspects that are essential to be reported in HEEs:

• Discounted and also undiscounted results should be presented.

• Cumulative results should be reported at various time points over a model’s construed decision horizon, including a longitudinal view up to the end of the defined time horizon of the model.

• Results from various relevant perspectives.

• Cost-effectiveness acceptability curves (CEACs).

• Best- and worst-case scenarios.

• Absolute values and ICERs for all disease-specific outcomes.

• A report of an HEE should describe the validation/calibration process, the strength of evidence behind the input data, and should discuss the potential variation of results in uncertainty analyses.

• The most recent questionnaire [71] assessing the credibility of a modelling study needs at least one infectious disease-specific addition: “If applicable, why was a dynamic model not used?”

Summary

As described in the Introduction, the aim of this paper was to provide a consensus framework on how to apply HEEs to vaccines and to identify areas where further work is needed to reach harmonization. The manuscript provides a comprehensive overview on many important aspects related to the economic evaluation of vaccines.

Experts from the field of modelling, health economics, and immunization decision making discussed 17 vaccine-specific aspects and reached consensus between modellers and decision-makers. Steered by a systematic literature review and expert opinion, this framework suggests the following:

• In general, international standards as laid down in established guidelines should be applied and adopted to specific problems where necessary.

• HEEs on vaccines and vaccination programmes should be considered by decision-making bodies such as NITAGs when considering inclusion of a new vaccine into the national programme to avoid inferior allocation of resources.

• A mechanical use of a threshold without considering other criteria may not be necessary. However, information about incremental costs and incremental outcomes of relevant vaccination strategies and ICERs (with adequate comparator(s)) should be delivered to decision-making bodies.

• Other interventions (e.g. drugs in preventive medicine) often share similar characteristics as vaccines. Vaccines should only be treated differently where they really are different (e.g. indirect effects).

• HEEs must be objective, systematic and transparent.

• HEEs should be as complex as necessary but as simple as possible.

• Ideally, infectious disease models should be dynamic.

• Models should focus on patient-relevant clinical endpoints wherever possible. But surrogates may be used if no clinical endpoints are available, preferably if they are validated. Both relying on surrogates and disregarding them is risky. The uncertainty concerning surrogates should be made clear to decision-makers.

• Future costs and outcomes should be discounted. There are arguments for differential and over time decreasing discounting (not only for vaccines).

• Information about costs and outcomes from the societal perspective is relevant. This perspective should ideally be reported in addition to the payer perspective.

• A broad set of utility-generating characteristics (such as carer quality of life, utility in anticipation) may be adequate. However, more research is needed.

• All uncertainties should be accounted for. Uncertainty analysis plays an important role.

• Methodological problems need to be solved. However, it is not necessary to reject HEE per se because of the methodological challenges.

• Funders and decision-makers should recognize that HEEs (of vaccines) demand time and resources.

Compliance with Ethical Standards

Funding

This study is part of a research project that is fully funded by the Federal Ministry of Health Germany.

Disclosure

All authors, Bernhard Ultsch, Oliver Damm, Philippe Beutels, Joke Bilcke, Bernd Brüggenjürgen, Andreas Gerber-Grote, Wolfgang Greiner, Germaine Hanquet, Raymond Hutubessy, Mark Jit, Mirjam Knol, Rüdiger von Kries, Alexander Kuhlmann, Daniel Levy-Bruhl, Matthias Perleth, Maarten Postma, Heini Salo, Uwe Siebert, Jürgen Wasem, and Ole Wichmann confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.