Skip to main content
PMC home page
Springer logoLink to Springer
. 2017 Sep 13;36(1):79–90. doi: 10.1007/s40273-017-0569-2

Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework

Natalie Carvalho 1, Mark Jit 2,3, Sarah Cox 4, Joanne Yoong 5,6, Raymond C W Hutubessy 7,✉
PMCID: PMC5775390  PMID: 28905279

Abstract

Background

In low- and middle-income countries, budget impact is an important criterion for funding new interventions, particularly for large public health investments such as new vaccines. However, budget impact analyses remain less frequently conducted and less well researched than cost-effectiveness analyses.

Objective

The objective of this study was to fill the gap in research on budget impact analyses by assessing (1) the quality of stand-alone budget impact analyses, and (2) the feasibility of extending cost-effectiveness analyses to capture budget impact.

Methods

We developed a budget impact analysis checklist and scoring system for budget impact analyses, which we then adapted for cost-effectiveness analyses, based on current International Society for Pharmacoeconomics and Outcomes Research Task Force recommendations. We applied both budget impact analysis and cost-effectiveness analysis checklists and scoring systems to examine the extent to which existing economic evaluations provide sufficient evidence about budget impact to enable decision making. We used rotavirus vaccination as an illustrative case in which low- and middle-income countries uptake has been limited despite demonstrated cost effectiveness. A systematic literature review was conducted to identify economic evaluations of rotavirus vaccine in low- and middle-income countries published between January 2000 and February 2017. We critically appraised the quality of budget impact analyses, and assessed the extension of cost-effectiveness analyses to provide useful budget impact information.

Results

Six budget impact analyses and 60 cost-effectiveness analyses were identified. Budget impact analyses adhered to most International Society for Pharmacoeconomics and Outcomes Research recommendations, with key exceptions being provision of undiscounted financial streams for each budget period and model validation. Most cost-effectiveness analyses could not be extended to provide useful budget impact information; cost-effectiveness analyses also rarely presented undiscounted annual costs, or estimated financial streams during the first years of programme scale-up.

Conclusions

Cost-effectiveness analyses vastly outnumber budget impact analyses of rotavirus vaccination, despite both being critical for policy decision making. Straightforward changes to the presentation of cost-effectiveness analyses results could facilitate their adaptation into budget impact analyses.

Electronic supplementary material

The online version of this article (doi:10.1007/s40273-017-0569-2) contains supplementary material, which is available to authorized users.

Key Points

Despite their equal importance in aiding decision makers to allocate limited resources, fewer budget impact analyses are published in the literature compared with cost-effectiveness analyses.
Furthermore, published budget impact analyses do not meet current best-practice recommendations such as not discounting future costs, providing annual or budget relevant financial streams of costs, model validation, and sensitivity and scenario analyses among others.
The proposed framework through which cost-effectiveness analyses could be used as a tool to provide useful budget impact analysis information could facilitate the uptake and improvement of good-quality budget impact analyses that would be useful for decision makers, in particular in low- and middle-income countries.
Open in a new tab

Introduction

Cost effectiveness is one of several key gateway criteria that inform new technology adoption decisions. However, interventions found to be cost effective or very cost effective are often not implemented [1]. While current uncertainties surrounding appropriate cost-effectiveness thresholds may explain some of this [1–3], overall healthcare budget implications and issues of sustainability also play an important role [4]. Decision makers require estimates of the real financial consequences of introducing a new intervention within a defined budget and budgetary period rather than relying on anticipated savings in economic costs alone [1, 5, 6]. The importance of high-quality budget impact analyses (BIAs) for accurate budgeting and resource allocation is increasingly recognised, especially for severely resource-constrained environments [7, 8]. Many high-income countries require budget impact information alongside cost-effectiveness estimates when making decisions to adopt new interventions [7, 9, 10]. In low- and middle-income countries (LMICs), understanding the short- and long-term impact of intervention adoption on national budgets is critical for ensuring programme sustainability [11, 12], especially when international funding is uncertain or of a temporary nature [13].

Unfortunately, the majority of published economic evaluations are cost-effectiveness analyses (CEAs), with very few BIAs, especially focused on LMICs [14]. Budget impact analyses may be conducted as often but less commonly published compared with CEAs for a variety of reasons, including secrecy on behalf of industry not wanting to disclose tender prices, or journal editors believing BIAs to be more of an administrative or financial task rather than research. Another possible explanation is that appropriate methodology for a BIA remains less well researched and less widely understood than for a CEA [9]. For instance, a previous review appears to classify nearly all (57/68) economic evaluations of rotavirus vaccination as BIAs if they provide an estimate of the net costs of vaccination as part of a CEA, despite these estimated economic (and often discounted) costs being very different from the true financial stream of costs that would be estimated within a BIA [15].

The first BIA analytic framework was published in 1998 [16]. Since then, additional publications have offered methodological guidance for BIAs [7, 10, 17–19], most recently, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force’s best-practice recommendations on conducting BIAs [18]. The recent ISPOR guidelines, however, do not include a checklist that summarises the methodological recommendations and supports a critical appraisal of the quality of BIAs. Such checklists are useful in ensuring the quality of research outputs. For example, CEAs are often appraised using Drummond’s checklist for assessing economic evaluations [20] or the Consolidated Health Economic Evaluation Reporting Standards checklist [21].

In this study, we develop a framework that incorporates the latest ISPOR recommendations into a BIA quality assessment checklist. This framework can be turned into a scoring system to critically assess the quality of a BIA. While there have been previous quality appraisal checklists in this area [14, 22], none have been explicit about how to use them to grade individual studies in particular intervention areas.

As an illustrative example, we develop a scoring system for vaccines. Newer vaccines are more costly than traditional childhood vaccines, representing increasingly large investments [23]. Delivery costs can be as great as vaccine costs, creating sustainability challenges over the long term [24]. Although development assistance for vaccination has increased over the last two decades, this is still a small proportion of total health expenditures for vaccination in LMICs [25]. In particular, for countries ineligible for assistance through Gavi, the Vaccine Alliance or those graduating out of Gavi support, the long-term budget implications from new vaccine adoptions are very important to consider. In fact, government expenditure on vaccines in countries that have adopted new vaccines was recently found to be on average double that of countries that have not [3].

As CEAs are more common than BIAs, particularly in LMICs [14], it would be helpful if they could be adapted to provide budget impact information without requiring much additional data or analysis. Such pragmatic adaptations could be immediately useful to policy makers seeking to fill the gap in stand-alone BIAs. This idea has been proposed by others in relation to vaccine CEAs, based on how population estimates are taken into account [26]. However, to the best of our knowledge, the feasibility of such adaptation has never been systematically assessed, and there is no existing guidance on preparing CEAs with such an objective in mind. We therefore also modify the BIA checklist and scoring system for CEAs, to evaluate whether, and if so, the extent to which, a CEA can be used to provide sufficient information for a BIA.

We first systematically review the literature to identify existing economic evaluations in LMICs. To keep the review manageable, we focus on rotavirus vaccination, which is recommended by the World Health Organization for all countries, particularly those with high rotavirus gastroenteritis case-fatality risks [27]. Most of the estimated 215,000 deaths in children under 5 years of age from rotavirus gastroenteritis occur in LMICs [28], and while there exist many studies demonstrating the cost effectiveness of rotavirus vaccination [15, 29], it has seen only partial uptake in LMICs. We then apply the checklists and scoring systems to examine the extent to which existing economic evaluations provide sufficient evidence about budget impact to enable decision making, and identify key areas where BIA and CEA reporting could be improved for this purpose.

Methods

Search Strategy

We conducted a systematic literature review to identify economic evaluations of rotavirus vaccine in LMICs. Our search was conducted in November 2015 and updated in February 2017. We included articles published between January 2000 and February 2017. Search terms were broken down into four components, which included terms relating to (1) costs/BIA/CEA/economic evaluation, (2) vaccines, (3) rotavirus and (4) LMICs [see Online Resource 1 of the Electronic Supplementary Material (ESM) for full details on search terms used].

Relevant Medical Subject Headings terms were included where appropriate. We developed and used an LMIC filter based on the most recent World Bank country income classification [30], expanded to include 25 countries that transitioned from low or middle income to high income from 2000 to 2017. We supplemented our LMIC filter with the Cochrane 2012 LMIC Filters [31] (Online Resource 2 of the ESM).

We searched the following electronic databases: EconLit, EMBASE (Ovid), MEDLINE (Ovid) and the National Health Service Economic Evaluation Database (through the Cochrane Library). Additionally, we ran basic searches in Research Papers in Economics and the Tufts CEA Registry [32]. We also carried out searches of the first 150 hits in Google and Google Scholar using BIA-specific search terms to identify any unpublished BIAs (Online Resource 1 of the ESM). We supplemented the database searches by conducting manual bibliographic searches from recent and relevant rotavirus vaccine review papers [14, 15, 29, 33–35].

Duplicate citations were removed and all remaining papers were screened based on title and abstract. Two reviewers (NC and SC) screened titles and abstracts during the original search conducted in November 2015. One reviewer (NC) screened titles and abstracts identified in the updated search conducted in February 2017. We verified inclusion as a LMIC based on the World Bank’s fiscal year at the time of publication. Two reviewers (NC and SC) read the full text of papers identified as relevant. Non-English language papers were translated to English. Studies included consisted of those self-defined as a BIA or other economic evaluation including a CEA and cost-utility analysis (we refer to both types as CEA from now on). Studies classified as a cost-benefit, fiscal or revenue analysis, or costing studies that did not capture both the costs of rotavirus vaccination and the cost savings owing to reduced disease, were excluded, as were papers tagged as Review, Editorial, Perspective or Discussion pieces. Full inclusion and exclusion criteria are summarised in Table 1. We followed the Preferred Reporting Items for Systematic Reviews and Meta-analysis guidelines and a checklist for the review [36].

Table 1.

Study inclusion and exclusion criteria

Include Exclude
Publisheda from January 2000 to February 2017 Publisheda before 2000
Focused on one or more LMICb Focused on high-income country(ies)
Target population children under 5 years of age Target population children over 5 years of age or unspecified
Intervention is any rotavirus vaccine delivered in any manner N/A
Full-text paper Abstracts, posters or presentations
1. Study self-described as a BIA
2. Other full economic evaluation (CEA or CUA)		 1. Fiscal, revenue or cost-benefit analyses, or costing studies that do not capture both:
(a) Costs of rotavirus vaccination; and
(b) Cost savings
2. Studies that do not report either:
(a) Novel analysis from primary data collection; or
(b) Secondary analysis or modelling of primary data
3. Review, editorial, perspective or discussion pieces
Open in a new tab

BIA budget impact analysis, CEA cost-effectiveness analysis, CUA cost-utility analysis, LMIC low- and middle-income country, N/A not applicable

a If unpublished, refers to date reported

b Based on World Bank Analytical Classification using gross national income per capita (Atlas methodology), based on the World Bank’s fiscal year at the time of publication

Development of a Budget Impact Analysis (BIA) Checklist

Based on the ISPOR best-practice recommendations [18], we produced a framework to provide guidance on assessing the quality of BIAs. The BIA Checklist consisted of 15 items divided into four categories: Background, Interventions, Analytic Framework and Results (Table 2). From this framework, we developed a vaccine-specific scoring system to critically appraise papers. Each item was assigned a full (1), partial (0.5) or null (0) score, based on how closely the article met the relevant recommendation. Strict scoring rules were followed for each item, as specified in Online Resource 3 of the ESM, and described in the following sections.

Table 2.

Items included in the budget impact analysis (BIA) checklist and modified BIA checklist for cost-effectiveness analysis

Item no. Category/itema Recommendation summaryb Classification for modified BIA checklist for CEA
Background
 1. Healthcare system/setting Consider relevant features of the healthcare system that may influence budget and possible access restrictions Desirable
 2. Study perspective Perspective should be that of the decision maker/budget holder Essential
 3. Eligible population Estimate size of eligible population, and distribution of any characteristics that may influence budget impact Essential: specify size of eligible population
Desirable: describe characteristics/justify population size estimates
Interventions
 4. Current interventions Lay out the current mix of interventions (use and effects) and the expected mix after the introduction of the new intervention Essential: health systems costs included
Desirable: microcosting of healthcare systems costs
 5. Uptake of new intervention Take into account the anticipated uptake of the new intervention Essential: report coverage level of new intervention
Desirable: discussion of where coverage estimates come from
 6. Costs of introducing new intervention Identify all cost categories included. Describe approaches used to estimate costs of new intervention Essential: all intervention cost categories included
Desirable: microcosting of intervention costs
 7. Impact on healthcare systems costs A description of how the intervention’s impact on healthcare costs was modelled should be included, including estimation of indirect effects where relevant Essential: model impact of intervention
Desirable: describe how intervention impact was modelled
Analytic framework
 8. Time horizon State and justify the time horizon(s) over which costs and consequences are being evaluated. Time horizon should be appropriate to the budget holder Essential: time horizon stated
Desirable: time horizon justified
 9. Discounting and time dependencies Financial streams at each budget period should be undiscounted. Other aspects that vary over time (inflation/deflation, changes in price) should be included Essential
 10. Model type Describe and justify the specific type of model used Essential: report model used
Desirable: describe model used
 11. Data sources Specify data sources and, if possible, obtain estimates directly from budget holders Desirable
Results
 12. Cost estimates/budget impact Present results (both resource use and costs) for each budget period after the new intervention is adopted Essential
 13. Validity Determine face validity through: (1) agreement with relevant decision makers on the computing framework, aspects included, and how they are addressed; and (2) verification of cost calculator or model implementation, including all formulas Desirable
 14. Uncertainty and scenario analyses Present alternative scenarios (e.g. allow users to view results with and without condition-related costs, to include or exclude different categories of costs) Desirable
 15. Conclusions and limitations State main conclusions on the basis of the results of the BIA. Report the main limitations regarding key issues including assumptions and completeness and quality of data inputs and sources Desirable
Open in a new tab

CEA cost-effectiveness analysis

a Each item gets a maximum score of 1 and a minimum score of 0. Items subdivided into ‘Essential’ and ‘Desirable’ components are scored 0/0.5 for each sub-component such that the maximum still sums to 1 for each item. Maximum score for the full checklist is 15 points. Maximum score for Essential vs. Desirable items in the modified BIA checklist for CEA are six and nine points, respectively

b Adapted from Sullivan et al. (2014) (International Society for Pharmacoeconomics and Outcomes Research Task Force on Good Research Practices—Budget Impact Analysis)

Background

For this category, relevant features of the healthcare system that may influence the budget must be considered. We identified five such features: financing available, budget for vaccines, the country’s decision to introduce the new vaccine, rotavirus disease burden and other relevant healthcare system factors such as availability of infrastructure. The recommended perspective is that of the decision maker or budget holder. Finally, the size of the eligible population must be described and data sources or approaches used to estimate population size explained.

Interventions

Articles should describe the current mix of interventions and the expected mix after the introduction of the new vaccine. This includes identifying all cost categories relevant to the current mix of interventions, including outpatient cases and hospitalisation. To receive a full score, we specified that costs must be estimated using microcosting. For pragmatic reasons, a study that uses local reimbursement rates (or reference costs) from the country (for example, based on a basic benefits package) was considered equivalent to microcosting. Second, the anticipated uptake and coverage of the new vaccine must be considered. For a full score, the article must discuss where coverage estimates come from, why they are reasonable, and their reason for modelling or not modelling scale-up. Third, all cost categories included in estimating the cost of vaccine introduction should be identified. This includes microcosting of operational delivery and administration costs of the vaccination programme in addition to specifying the vaccine procurement cost. Finally, the impact on healthcare costs should be modelled, including a description of how this was done.

Analytic Framework

Within this category, we assessed aspects related to modelling choices and data inputs, including stating and justifying a time horizon appropriate to the budget holder, not discounting costs, and providing full details of the model used and input parameters. If an article makes no mention of discounting costs, we assumed no discounting was used. We generated a list of the six most relevant data inputs: demography; estimated vaccine coverage; burden of disease; vaccine efficacy; vaccine-related costs; and other health systems-related costs. Articles were scored based on how much local level data were used to inform these inputs.

Results

Results must be presented as estimates of financial costs at each budget period after the new vaccine was introduced. This final category also assessed the determination of face validity, the presence of uncertainty and scenario analyses, and the article’s main conclusions from a budget impact perspective, including discussion of the main limitations. For these last three sub-sections within the results (validity, uncertainty and scenario analyses, and conclusions), scores do not take into account how well each of these items was done, but rather provides guidance on whether they were done at all.

Development of a Modified BIA Checklist for Cost-Effectiveness Analyses (CEAs)

We modified the BIA Checklist for CEAs by classifying each item as ‘Essential’ or ‘Desirable’ for estimating budget impact. Some items were subdivided into both an ‘Essential’ and ‘Desirable’ component. As a result, the Modified BIA Checklist for CEAs consisted of ten Essential components, and a further 12 Desirable components (Table 2). The scoring system was similarly modified to provide a ‘feasibility score’ to reflect a CEA’s suitability for adaptation for BIA purposes (Online Resource 4 of the ESM). Cost-Effectiveness Analyses were first scored on the Essential criteria, and only articles receiving a full score (6.5 points total) were assessed on the Desirable components (8.5 additional points).

All articles included in the final review were scored based on the BIA Checklist or the Modified BIA Checklist for CEA scoring systems. The maximum score for both checklists was 15 points. Two reviewers (NC and SC) scored the articles independently. Scoring disagreements across any of the items within the checklists were identified and discussed by the two reviewers prior to jointly agreeing on a score. If an agreement was not reached, the conflicting scores were discussed with the broader team (all authors) until a consensus was reached.

Results

Articles Included in the Review

Our search yielded 834 articles, of which 305 were duplicates. The 529 original articles were screened for inclusion and exclusion criteria based on title and abstract. Two reviewers read the full-text version of 103 studies, of which an additional 37 articles were excluded. Sixty-six articles were included in the final review, of which six were BIAs and 60 were CEAs (Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Literature search flow diagram. BIAs budget impact analyses, CEAs cost-effectiveness analyses, NHS EED RePEc. aIncluding one article in Russian for which full text was sought but could not be obtained and one PhD dissertation thesis under embargo until 2018

The six BIAs were published between 2010 and 2013, and focused on countries across different regions of the world (Indonesia, Thailand, Armenia, Brazil), with one that evaluated rotavirus vaccine across all Gavi-eligible countries (Online Resource 5 of the ESM). Five BIAs were conducted alongside a CEA.

The CEAs (Online Resource 5 of the ESM) were published between 2005 and 2016, with the largest numbers of studies published in 2009 and 2015 (12 in each year). The CEAs focused on individual LMICs around the world as well as groups of countries.

Assessment of Papers Reviewed

Application of the BIA Checklist

Background

All BIAs described the burden of rotavirus disease and whether a decision had been made to introduce the vaccine in the country. Most (4/6) [37–40] adequately described with justification the estimated target population. All analyses were conducted from the decision maker or budget holder’s perspective. Most BIAs fully considered the relevant features of the healthcare system that could influence the budget, and reported on the current budget for existing vaccines as well as financing available for the introduction of the new vaccine, such as whether the country is eligible for Gavi support and, if so, when this support would end.

Interventions

All but one [40] BIA laid out the current mix of interventions and expected mix after the introduction of the new vaccine. Half [38–40] took into account the anticipated uptake of the new vaccine, including describing the estimated scale-up of the vaccine over time with justification based on prior vaccine scale-up trends or other sources. Most (4/6) BIAs [37–40] identified all relevant cost categories and described the approaches used to estimate the costs of introducing the new vaccine. Some articles were penalised for not microcosting to estimate costs associated with the current interventions (1/6) [40] or the new vaccine (2/6) [41, 42], or not justifying the coverage level at which the vaccine would be rolled out or discussing scaling up (3/6) [37, 41, 42]. All BIAs modelled the impact of the vaccine on health care costs and described how this was done.

Analytic Framework

All BIAs specified either the programmatic time horizon or stated that they were modelling one birth cohort; however, only two justified this time horizon from the perspective of the budget holder [38, 39]. All BIAs accounted for time dependencies of costs; however, three discounted future costs at 3% [37, 41, 42]. All articles described the model and input parameters, and used budget holder or local country-specific data sources to inform model input parameters.

Results

Only two BIAs presented financial cost estimates and resource use at each budget period after the new vaccine is introduced [39, 40]. One BIA presented financial cost estimates only [38]; the remaining appeared to present annual total costs based on a mature programme operating at full scale [37, 41, 42]. Determination of face validity was present in some capacity in four of six articles: one conducted a partial verification of model projections by comparing two different models [40]; three compared the financial cost estimates for one budget period to the current immunisation budget in the country [37, 41, 42]. With one exception [39], all BIAs carried out uncertainty and scenario analyses. All but one stated the main conclusions from a budget impact perspective [38], including discussion of the main limitations.

Overall BIA Quality Assessment Scores

Among all BIAs, validity, time dependencies and discounting, and programmatic time horizon received the lowest average scores across all checklist items. Average scores for the BIA papers across each of the 15 checklist items are shown in Fig. 2. Overall, the total score received by BIAs ranged from 10.5 to 13.5 (out of a maximum of 15 points). Article-specific scores for each checklist item are shown in Online Resource 5 of the ESM.

Fig. 2.

Fig. 2

Open in a new tab

Average budget impact analysis checklist scoring for budget impact analysis papers (n = 6). Average score (range in brackets) shown for each category. Average overall quality score for all budget impact analysis papers reviewed: 11.9 (10.5, 13.5) out of 15 possible points. Each category is worth 0, 0.5 or 1 point (maximum = 1)

Application of the Modified BIA Checklist for CEA

Background

The majority of CEAs reviewed (53/60) were conducted from the perspective of the decision maker or budget holder. Many (38/54) also included some form of a societal perspective. Of the seven CEAs not conducted from the healthcare payer or provider perspective, most included household out-of-pocket costs that could not be separated from the healthcare systems costs. The size of the eligible population was specified or data sources provided in most (53/60) CEAs.

Interventions

All CEAs performed well in this category. All articles received a full score for laying out the current mix of interventions and expected mix after the introduction of the new vaccine, taking into account the anticipated uptake of the new vaccine, and modelling the impact on healthcare costs. Most (56/60) CEAs adequately identified all major cost categories relevant to the new vaccine introduction.

Analytic Framework

The programmatic time horizon was clear in all but one CEA. All reported on the model used for the analysis. In many cases (14/60), it was unclear whether future costs were discounted or not. Either nothing was stated or discounting was only mentioned in the context of disease burden. The majority of articles that discounted costs used a 3% discount rate. Less than half of all studies (18/60) presented costs undiscounted or presented both discounted and undiscounted costs. While some studies also specified varying the discount rate in sensitivity analyses (usually assuming no discounting as a lower bound), most did not adequately show their results (total costs, or programme costs plus cost savings) undiscounted.

Results

The cost estimates/budget impact category scored lowest of all items included in the modified checklist. Only four of 60 CEAs received a full score, while five articles received a partial score. Most articles presented the total costs and/or resource use for the entire period modelled, rather than disaggregating it year by year. Furthermore, most articles did not present the true time-varying financial costs of rolling out the vaccine and subsequently scaling up incrementally, instead choosing to report annualised costs based on a mature programme operating at full scale.

Overall BIA Feasibility Scores for CEAs

Average BIA feasibility scores across all articles for the nine essential items are shown in Fig. 3. The two worst performing categories were ‘Discounting and time dependencies’, and ‘Cost estimates/budget impact’. Overall feasibility scores across all essential items ranged from 3 to 6.5, with an average of 4.7 out of 6.5 possible points. Three papers scored full points (6.5/6.5) on the essential items and were subsequently scored on the desirable items. Final scores for the complete checklist ranged from 12 to 13.5 out of 15 possible points for these three CEAs (Online Resource 5 of the ESM).

Fig. 3.

Fig. 3

Open in a new tab

Average modified checklist scoring for cost-effectiveness analysis papers (n = 60). Average score (range in brackets) shown for each ‘Essential’ category. Average overall feasibility score for all cost-effectiveness analysis papers reviewed: 4.7 (3, 6.5) out of 6.5 possible points. Each category with a solid line is worth 0, 0.5 or 1 point (maximum = 1). Each starred category with a dashed line is worth 0 or 0.5 points (maximum = 0.5)

Discussion

The majority of economic evaluations of rotavirus vaccine in LMICs identified in this review were CEAs. Only six of 66 articles were classified as a BIA, and all but one were conducted alongside a CEA. The BIAs adhered well to the majority of ISPOR’s best-practice recommendations. However, our review identified three areas where BIAs fell short from current best-practice recommendations: not discounting future costs, providing annual or budget relevant financial streams of costs and model validation. Most BIAs were conducted alongside a CEA, and intervention costs were often handled in a manner more appropriate to CEA rather than BIA in terms of discounting and reporting. Previous reviews of BIAs have determined that good methods are not used consistently, with discounting, reporting and sensitivity analyses also reported as key issues in need of improvement, among others [14, 19, 43, 44].

The majority of existing rotavirus CEAs did not provide adequate detail to be useful for budget impact assessment. Only three of 60 CEAs passed the ‘minimum requirements’ to be assessed using the full checklist. Similar to the BIAs, CEAs rarely present undiscounted annual costs, or estimate financial streams during the first few years of programme scale-up. One pragmatic recommendation for CEAs (and those who commission them) could therefore be to explicitly require the reporting of yearly undiscounted costs in addition to the standard reporting of costs within a CEA (possibly in an appendix if not directly relevant for the cost-effectiveness results), and estimates of real financial streams during the first few years of programme scale-up.

There are important limitations of this study that merit mentioning. First, in constructing a checklist and scoring criteria, we have simplified the recommendations laid out by the ISPOR Task Force. While we attempted to capture the most important considerations addressed in the best-practice recommendations, the categories included in the checklist are not an exhaustive list. For example, our checklist does not capture whether the model allows users to input or vary parameter values, an important consideration for BIAs according to the guidelines. Additionally, some categories included in the checklist can be subjective, for example, stating appropriate conclusions and limitations from a budgetary perspective. As such, in developing the scoring system, we do not consider the quality and completeness of each of the items in the checklist. Instead, we have developed the scoring system to be as explicit as possible for scoring vaccine-specific articles.

We focused only on full economic evaluations in this review. Thus, we do not include vaccine costing studies that may have tried to provide an estimate of budget impact but did not take into account cost savings from reduced burden on the healthcare system. Only a small number of BIAs was included in our review, limiting the discriminatory power of our checklist and the generalisability of our findings to BIAs conducted in areas outside of rotavirus vaccine. Finally, it is important to reiterate that the majority of papers reviewed were not intended to be formal BIAs. As such, the ‘feasibility’ scores for CEAs do not reflect an assessment of their quality as an economic evaluation per se, but rather the extent to which they provide useful budget impact information. This is especially the case for articles scored only on Essential items. For these items in particular, scores do not reflect how well, or how appropriate for the budget holder this item was handled in the analysis. Rather, they reflect whether or not the item was taken into account or specified at all. For example, CEAs specifying the size of the eligible population score full marks for the essential component of this item, regardless of whether the population size was appropriate or justified.

Despite these limitations, our study provides a simple and easily implementable quality assessment checklist and illustrates how it can be converted into a scoring system, in this case for vaccine-specific BIAs. It is to the best of our knowledge, the first framework for systematically evaluating the feasibility of adapting a CEA to a BIA. We identified several areas where existing BIAs fell short from best-practice recommendations. We also identified relatively straightforward changes to the presentation of CEA results that could facilitate the adaptation of most cost-effectiveness studies into BIAs, potentially increasing their utility and application. Although we have used produced a vaccine-specific scoring system, the checklist could similarly be adapted to other types of interventions.

To assist countries in making informed decisions about adding a vaccine to a national immunisation programme, the World Health Organization emphasises that programmatic, economic and financial feasibility are important criteria among others to consider [45]. This is especially true in the case of LMICs not eligible for financial support or price negotiations through Gavi and those graduating out of Gavi support [4]. For these countries, the real fiscal costs of introducing and sustaining new vaccines within immunisation schedules will have enormous implications on a country’s immunisation budget and overall healthcare budget. The World Health Organization’s global vaccine and immunisation policy recommendations also acknowledge budget impact considerations as being highly relevant alongside cost effectiveness [46].

There is potential for improved decision making in the area of vaccines through programmes such as the ProVac Initiative, which provides economic evaluation tools including a CEA model (TRIVAC) and BIA model (COSTVAC) along with country-level support. [47]. Beyond vaccines, the ‘Gates Reference Case, recently developed through funding by the Bill and Melinda Gates Foundation with the aim to improve the quality of economic evaluations conducted in LMICs, recommends reporting on budget impact as one of 11 key principles [48].

The limited number of published rotavirus vaccine BIAs in LMIC settings compared with CEAs may reflect the need for more guidance on best-practice methodologies for BIAs of preventive interventions in these settings. In the context of the current difficulties in determining appropriate threshold values for cost-effectiveness ratios in LMICs, there is potentially greater value for budget impact estimates for reflecting opportunity costs for these countries. Additional use and validation of the BIA quality assessment framework proposed in this review is necessary and would be useful for further work. For example, these checklists and a vaccine-specific scoring system could be applied to economic evaluations of other vaccines, and results compared with the findings in this review. Further research is needed to investigate how this framework could be adapted to other public health or medical interventions.

Conclusion

Both budget impact and cost effectiveness are key criteria, among others, for policy makers deciding how to allocate limited resources. In our study, we propose and demonstrate the use of a generalised BIA checklist and vaccine-specific scoring system for guiding and assessing the quality of BIA as a stand-alone analysis or alongside CEA, and highlight current gaps in standard practices. The proposed framework could facilitate the uptake and improvement of the quality of BIAs useful for decision making, in particular in LMICs.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 112 kb) (112.9KB, docx)
Supplementary material 2 (DOCX 145 kb) (145.4KB, docx)
Supplementary material 3 (DOCX 96 kb) (96.8KB, docx)
Supplementary material 4 (DOCX 120 kb) (120.6KB, docx)
Supplementary material 5 (DOCX 198 kb) (198.5KB, docx)

Acknowledgements

We thank Mabel Fong and Sucitro Sidharta (National University of Singapore) for their valuable contributions as research assistants. We also thank Tanji Hwang and Arun Bharatula (University of Melbourne) for their research assistance. Raymond C.W. Hutubessy is a staff member of the World Health Organization. The views expressed here are his and not necessarily those of the World Health Organization.

Author Contributions

MJ and RH conceived the study. NC developed the search strategy, inclusion and exclusion criteria, as well as conducted the searches with input from SC, RH, MJ and JY. NC and SC performed the initial screening of titles and abstracts. NC developed the quality assessment checklist and scoring with input from SC, RH, MJ and JY. NC and SC assessed full-text articles for inclusion in review, with any discrepancies discussed by NC, SC, RH, MJ and JY. NC and SC scored and abstracted data from all articles reviewed, with any discrepancies discussed by NC, SC, RH, MJ and JY. NC and SC wrote the report with input from all authors. All authors approved the final version of the report.

Compliance with Ethical Standards

Funding

Natalie Carvalho was funded by the University of Melbourne McKenzie Post Doctoral Fellowship Scheme. Mark Jit and Joanne Yoong were funded by the World Health Organization and Gavi, the Vaccine Alliance. Sarah Cox was funded by the Bill and Melinda Gates Foundation, the Johns Hopkins Vaccine Initiative and the Bloomberg School of Public Health through the Program in Applied Vaccine Experiences. No other funding was received for this work.

Conflict of interest

Natalie Carvalho, Mark Jit, Sarah Cox, Joanne Yoong and Raymond C.W. Hutubessy have no conflicts of interest directly relevant to the content of this article.

Data availability statement

This article is a systematic review of the literature, and thus there are no underlying data used for this research apart from the data extracted from the articles included in this review. A list of all articles included in the review is available in the Electronic Supplementary Material (Online Resource 5). Data extracted from all budget impact analyses used to inform the budget impact analysis checklist scores are available in the Electronic Supplementary Material (Online Resource 5). Data extracted from cost-effectiveness analyses used to inform the budget impact analysis checklist for cost-effectiveness analysis scores are available from the corresponding author upon reasonable request.

Footnotes

Electronic supplementary material

The online version of this article (doi:10.1007/s40273-017-0569-2) contains supplementary material, which is available to authorized users.

References

  • 1.Newall AT, Jit M, Hutubessy R. Are current cost-effectiveness thresholds for low- and middle-income countries useful? Examples from the world of vaccines. Pharmacoeconomics. 2014;32(6):525–531. doi: 10.1007/s40273-014-0162-x. [DOI] [PubMed] [Google Scholar]
  • 2.Shillcutt SD, Walker DG, Goodman CA, Mills AJ. Cost effectiveness in low- and middle-income countries: a review of the debates surrounding decision rules. Pharmacoeconomics. 2009;27(11):903–917. doi: 10.2165/10899580-000000000-00000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Glassman A, Canon O, Silverman R. How to get cost-effectiveness analysis right? The case of vaccine economics in Latin America. Value Health. 2016;19(8):913–920. doi: 10.1016/j.jval.2016.04.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Makinen M, Kaddar M, Molldrem V, Wilson L. New vaccine adoption in lower-middle-income countries. Health Policy Plan. 2012;27(Suppl. 2):ii39–49. [DOI] [PubMed]
  • 5.Hoffmann C, Graf von der Schulenburg JM. The influence of economic evaluation studies on decision making: a European survey. The EUROMET Group. Health Policy. 2000;52(3):179–192. doi: 10.1016/S0168-8510(00)00076-2. [DOI] [PubMed] [Google Scholar]
  • 6.Jit M, Hutubessy R. Methodological challenges to economic evaluations of vaccines: is a common approach still possible? Appl Health Econ Health Policy. 2016;14(3):245–252. doi: 10.1007/s40258-016-0224-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Trueman P, Drummond M, Hutton J. Developing guidance for budget impact analysis. Pharmacoeconomics. 2001;19(6):609–621. doi: 10.2165/00019053-200119060-00001. [DOI] [PubMed] [Google Scholar]
  • 8.Carroll S, García Rojas AJ, Glenngård AH, Marin C. Vaccination: short- to long-term benefits from investment. J Mark Access Health Policy. 2015;12:3. doi: 10.3402/jmahp.v3.27279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Garattini L, van de Vooren K. Budget impact analysis in economic evaluation: a proposal for a clearer definition. Eur J Health Econ. 2011;12(6):499–502. doi: 10.1007/s10198-011-0348-5. [DOI] [PubMed] [Google Scholar]
  • 10.Nuijten MJ, Mittendorf T, Persson U. Practical issues in handling data input and uncertainty in a budget impact analysis. Eur J Health Econ. 2011;12(3):231–241. doi: 10.1007/s10198-010-0236-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Ngcobo NJ, Cameron NA. The decision making process on new vaccines introduction in South Africa. Vaccine. 2012;30(Suppl. 3):C9–C13. doi: 10.1016/j.vaccine.2012.04.027. [DOI] [PubMed] [Google Scholar]
  • 12.Mori AT, Norheim OF, Robberstad B. Budget impact analysis of using dihydroartemisinin-piperaquine to treat uncomplicated malaria in children in Tanzania. Pharmacoeconomics. 2016;34(3):303–314. doi: 10.1007/s40273-015-0344-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Marseille E, Larson B, Kazi DS, Kahn JG, Rosen S. Thresholds for the cost-effectiveness of interventions: alternative approaches. Bull World Health Organ. 2015;93(2):118–124. doi: 10.2471/BLT.14.138206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Orlewska E, Gulacsi L. Budget-impact analyses: a critical review of published studies. Pharmacoeconomics. 2009;27(10):807–827. doi: 10.2165/11313770-000000000-00000. [DOI] [PubMed] [Google Scholar]
  • 15.Aballea S, Millier A, Quilici S, et al. A critical literature review of health economic evaluations of rotavirus vaccination. Hum Vaccin Immunother. 2013;9(6):1272–1288. doi: 10.4161/hv.24253. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Mauskopf J. Prevalence-based economic evaluation. Value Health. 1998;1(4):251–259. doi: 10.1046/j.1524-4733.1998.140251.x. [DOI] [PubMed] [Google Scholar]
  • 17.Mauskopf JA, Sullivan SD, Annemans L, et al. Principles of good practice for budget impact analysis: report of the ISPOR Task Force on good research practices: budget impact analysis. Value Health. 2007;10(5):336–347. doi: 10.1111/j.1524-4733.2007.00187.x. [DOI] [PubMed] [Google Scholar]
  • 18.Sullivan SD, Mauskopf JA, Augustovski F, et al. Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value Health. 2014;17(1):5–14. doi: 10.1016/j.jval.2013.08.2291. [DOI] [PubMed] [Google Scholar]
  • 19.Mauskopf J, Earnshaw S. A methodological review of US budget-impact models for new drugs. Pharmacoeconomics. 2016;34(11):1111–1131. doi: 10.1007/s40273-016-0426-8. [DOI] [PubMed] [Google Scholar]
  • 20.Drummond M, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 4th ed. Oxford: Oxford University Press; 2015. p. 42–44
  • 21.Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Value Health. 2013;16:e1–e5. doi: 10.1016/j.jval.2013.02.010. [DOI] [PubMed] [Google Scholar]
  • 22.Patented Medicine Prices Review Board. Budget impact analysis guidelines: guidelines for conducting pharmaceutical budget impact analyses for submission to public drug plans in Canada. 2007. http://www.pmprb-cepmb.gc.ca/cmfiles/bia-may0738lvv-5282007-5906.pdf. Accessed 9 Sept 2017.
  • 23.Médecins Sans Frontières. The right shot: bringing down barriers to affordable and adapted vaccines. 2nd ed. 2015. http://cdn.doctorswithoutborders.org/sites/usa/files/attachments/the_right_shot_2nd_edition.pdf. Accessed 9 Sept 2017.
  • 24.Brenzel L. What have we learned on costs and financing of routine immunization from the comprehensive multi-year plans in Gavi eligible countries? Vaccine. 2015;33(Suppl. 1):A93–A98. doi: 10.1016/j.vaccine.2014.12.076. [DOI] [PubMed] [Google Scholar]
  • 25.Haakenstad A, Birger M, Singh L, et al. Vaccine assistance to low- and middle-income countries increased to $3.6 billion In 2014. Health Aff (Millwood) 2016;35(2):242–249. doi: 10.1377/hlthaff.2015.1080. [DOI] [PubMed] [Google Scholar]
  • 26.Mauskopf J, Talbird S, Standaert B. Categorization of methods used in cost-effectiveness analyses of vaccination programs based on outcomes from dynamic transmission models. Expert Rev Pharmacoecon Outcomes Res. 2012;12(3):357–371. doi: 10.1586/erp.12.11. [DOI] [PubMed] [Google Scholar]
  • 27.World Health Organization WHO position paper, January 2013. Wkly Epidemiol Rec. 2013;5:49–64. [Google Scholar]
  • 28.Tate JE, Burton AH, Boschi-Pinto C, Parashar UD, World Health Organization-Coordinated Global Rotavirus Surveillance Network Global, regional, and national estimates of rotavirus mortality in children <5 years of age, 2000–2013. Clin Infect Dis. 2016;62(Suppl. 2):S96–S105. doi: 10.1093/cid/civ1013. [DOI] [PubMed] [Google Scholar]
  • 29.Tu HA, Woerdenbag HJ, Kane S, et al. Economic evaluations of rotavirus immunization for developing countries: a review of the literature. Expert Rev Vaccines. 2011;10(7):1037–1051. doi: 10.1586/erv.11.65. [DOI] [PubMed] [Google Scholar]
  • 30.The World Bank. World Bank analytical classifications, country analytical history (Atlas methodology). http://siteresources.worldbank.org/DATASTATISTICS/Resources/OGHIST.xls. Accessed 9 Sept 2017.
  • 31.The Cochrane Collaboration. Cochrane 2012 LMIC filters. 6 November 2015. Available from: http://epoc.cochrane.org/lmic-filters. Accessed 5 Sept 2017.
  • 32.Cost-Effectiveness Analysis Registry. Available from: http://healtheconomics.tuftsmedicalcenter.org/cear4/Home.aspx. Accessed 5 Sept 2017.
  • 33.Rheingans R, Amaya M, Anderson JD, Chakraborty P, Atem J. Systematic review of the economic value of diarrheal vaccines. Hum Vaccin Immunother. 2014;10(6):1582–1594. doi: 10.4161/hv.29352. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Ozawa S, Mirelman A, Stack ML, Walker DG, Levine OS. Cost-effectiveness and economic benefits of vaccines in low- and middle-income countries: a systematic review. Vaccine. 2012;31(1):96–108. doi: 10.1016/j.vaccine.2012.10.103. [DOI] [PubMed] [Google Scholar]
  • 35.Thiboonboon K, Santatiwongchai B, Chantarastapornchit V, Rattanavipapong W, Teerawattananon Y. A systematic review of economic evaluation methodologies between resource-limited and resource-rich countries: a case of rotavirus vaccines. Appl Health Econ Health Policy. 2016;14(6):659–672. doi: 10.1007/s40258-016-0265-y. [DOI] [PubMed] [Google Scholar]
  • 36.Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi: 10.1136/bmj.b2535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Muangchana C, Riewpaiboon A, Jiamsiri S, Thamapornpilas P, Warinsatian P. Economic analysis for evidence-based policy-making on a national immunization program: a case of rotavirus vaccine in Thailand. Vaccine. 2012;30(18):2839–2847. doi: 10.1016/j.vaccine.2012.02.047. [DOI] [PubMed] [Google Scholar]
  • 38.Jit M, Yuzbashyan R, Sahakyan G, Avagyan T, Mosina L. The cost-effectiveness of rotavirus vaccination in Armenia. Vaccine. 2011;29(48):9104–9111. doi: 10.1016/j.vaccine.2011.08.127. [DOI] [PubMed] [Google Scholar]
  • 39.Centenari C, Gurgel RQ, Bohland AK, Oliveira DM, Faragher B, Cuevas LE. Rotavirus vaccination in northeast Brazil: a laudable intervention, but can it lead to cost-savings? Vaccine. 2010;28:4162–4168. doi: 10.1016/j.vaccine.2010.04.015. [DOI] [PubMed] [Google Scholar]
  • 40.Kim SY, Sweet S, Slichter D, Goldie SJ. Health and economic impact of rotavirus vaccination in Gavi-eligible countries. BMC Public Health. 2010;10:253. doi: 10.1186/1471-2458-10-253. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Suwantika AA, Postma MJ. Effect of breastfeeding promotion interventions on cost-effectiveness of rotavirus immunization in Indonesia. BMC Public Health. 2013;13:1106. doi: 10.1186/1471-2458-13-1106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Suwantika AA, Tu HA, Postma MJ. Cost-effectiveness of rotavirus immunization in Indonesia: taking breastfeeding patterns into account. Vaccine. 2013;31(32):3300–3307. doi: 10.1016/j.vaccine.2013.04.055. [DOI] [PubMed] [Google Scholar]
  • 43.Mauskopf JA, Earnshaw S, Mullins CD. Budget impact analysis: review of the state of the art. Expert Rev Pharmacoecon Outcomes Res. 2005;5(1):65–79. doi: 10.1586/14737167.5.1.65. [DOI] [PubMed] [Google Scholar]
  • 44.van de Vooren K, Duranti S, Curto A, Garattini L. A critical systematic review of budget impact analyses on drugs in the EU countries. Appl Health Econ Health Policy. 2014;12(1):33–40. doi: 10.1007/s40258-013-0064-7. [DOI] [PubMed] [Google Scholar]
  • 45.World Health Organization . Principles and considerations for adding a vaccine to a national immunization programme: from decision to implementation and monitoring. Geneva: World Health Organization; 2014. [Google Scholar]
  • 46.World Health Organization . Guidance for the development of evidence-based vaccination-related recommendations. Strategic Advisory Group of Experts on Immunization. Geneva: World Health Organization; 2017. [Google Scholar]
  • 47.Jauregui B, Janusz CB, Clark AD, et al. ProVac Global Initiative: a vision shaped by ten years of supporting evidence-based policy decisions. Vaccine. 2015;33(Suppl. 1):A21–A27. doi: 10.1016/j.vaccine.2014.12.080. [DOI] [PubMed] [Google Scholar]
  • 48.NICE International. The Gates reference case: what it is, why it’s important, and how to use it. Bill and Melinda Gates Foundation Methods for Economic Evaluation Project. 2014. Bill and Melinda Gates Foundation, NICE International, the Health Intervention and Technology Assessment Program (Thailand), and The University of York, Centre for Health Economics. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-International/projects/Gates-Referencecase-what-it-is-how-to-use-it.pdf. Accessed 9 Sept 2017.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material 1 (DOCX 112 kb) (112.9KB, docx)
Supplementary material 2 (DOCX 145 kb) (145.4KB, docx)
Supplementary material 3 (DOCX 96 kb) (96.8KB, docx)
Supplementary material 4 (DOCX 120 kb) (120.6KB, docx)
Supplementary material 5 (DOCX 198 kb) (198.5KB, docx)

Data Availability Statement

This article is a systematic review of the literature, and thus there are no underlying data used for this research apart from the data extracted from the articles included in this review. A list of all articles included in the review is available in the Electronic Supplementary Material (Online Resource 5). Data extracted from all budget impact analyses used to inform the budget impact analysis checklist scores are available in the Electronic Supplementary Material (Online Resource 5). Data extracted from cost-effectiveness analyses used to inform the budget impact analysis checklist for cost-effectiveness analysis scores are available from the corresponding author upon reasonable request.

Articles from Pharmacoeconomics are provided here courtesy of Springer

RESOURCES