Inclusion of Additional Unintended Consequences in Economic Evaluation: A Systematic Review of Immunization and Tuberculosis Cost-Effectiveness Analyses

Abstract

Objective

Our objective was to review economic evaluations of immunization and tuberculosis to determine the extent to which additional unintended consequences were taken into account in the analysis and to describe the methodological approaches used to estimate these, where possible.

Methods

We sourced the vaccine economic evaluations from a previous systematic review by Nymark et al. (2009–2015) and searched PubMed/MEDLINE and Embase from 2015 to 2019 using the same search strategy. For tuberculosis economic evaluations, we extracted studies from 2009 to 2019 that were published in a previous review by Siapka et al. We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidance. Studies were classified according to the categories and subcategories (e.g., herd immunity, non-specific effects, and labor productivity) defined in a framework identifying additional unintended consequences by Nymark and Vassall. Where possible, methods for estimating the additional unintended consequences categories and subcategories were described. We evaluated the reporting quality of included studies according to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) extraction guideline.

Results

We identified 177 vaccine cost-effectiveness analyses (CEAs) between 2009 and 2019 that met the inclusion criteria. Of these, 98 included unintended consequences. Of the total 98 CEAs, overall health consequence categories were included 73 times; biological categories: herd immunity 43 times; pathogen response: resistance 15 times; and cross-protection 15 times. For health consequences pertaining to the supply-side (health systems) categories, side effects were included five times. On the nonhealth demand side (intrahousehold), labor productivity was included 60 times. We identified 29 tuberculosis CEAs from 2009 to 2019 that met the inclusion criteria. Of these, six articles included labor productivity, four included indirect transmission effects, and one included resistance. Between 2009 and 2019, only 34% of tuberculosis CEAs included additional unintended consequences, compared with 55% of vaccine CEAs.

Conclusions

The inclusion of additional unintended consequences in economic evaluations of immunization and tuberculosis continues to be limited. Additional unintended consequences of economic benefits, such as those examined in this review and especially those that occur outside the health system, offer valuable information to analysts. Further work on appropriate ways to value these additional unintended consequences is still warranted.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41669-021-00269-4.

Key Points for Decision Makers

Between 2009 and 2019, only 34% of tuberculosis cost-effectiveness analyses (CEAs) included additional unintended consequences, compared with 55% of vaccine CEAs.

There is a clear absence of evidence of additional unintended consequences outside the health system for both tuberculosis and vaccine CEAs.

Further work on appropriate ways to value additional unintended consequences in CEAs is needed.

Introduction

Guidelines for performing economic evaluations of healthcare interventions recommend that all relevant direct and indirect health effects are considered, whereas other guidelines highlight the importance of mapping indirect nonhealth effects into economics frameworks for value assessment [1–5]. Infectious disease-specific guidelines have also made attempts at scoping the inclusion of nondirect effects, particularly for economic evaluation of immunizations [6–8].

Given these developments in guidelines, it seems reasonable to expect that the inclusion of indirect health and nonhealth effects would be standard practice. However, the literature indicates that it is still challenging to comprehensively identify which indirect health and nonhealth effects to include in cost-effectiveness analyses (CEAs) [9–11]. Several previous reviews of CEAs suggest that indirect health and nonhealth effects are often excluded, even when they may be relevant and significant [12–15]. Different practices across economic evaluations can mean that cost effectiveness can be difficult to compare across interventions. Therefore, continued attention is required to define, examine, and map out the extent to which all consequences are considered in economic evaluation.

We presented a comprehensive framework elsewhere [16] to assist analysts in identifying and characterizing the additional costs and effects beyond that of the direct health impact that was primarily intended to be influenced by the intervention/technology. We refer to these additional costs and effects hereafter as “additional unintended consequences” [16]. On the whole, the inclusion of additional unintended consequences is relevant in studies using the societal perspective. However, some of the additional unintended consequences can also be relevant to the healthcare system perspective. In this study, we build on the existing literature by assessing the extent to which economic evaluations of vaccines and tuberculosis treatments consider the different types of additional unintended consequences based on our framework. These were chosen to provide a comprehensive summary of the evidence and to explore the consistency of findings (and therefore generalizability) across different disease areas. We also highlight the different methods that were used to measure these additional unintended consequences.

Methods

Search strategy and Data Extraction

We used a combination of previous searches from two separate vaccine and tuberculosis reviews that both followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

The vaccine review by Nymark et al. [15] searched PubMed/MEDLINE and Embase. It applied a search strategy by Kim and Goldie [17] using free text and medical subject heading (MeSH) terms such as vaccin∗, economic evaluat∗, and humans. It limited the search to the English language and covered articles published between 2009 and 2015. The details of the data included, covering searches, article selection, and data extraction, are presented elsewhere [15]. We identified CEAs for low- and middle-income countries from this review. We merge two reviews in this paper. The review by Siapka et al. [18] only included CEAs for low- and middle-income countries. To account for these differences, we restricted the review by Nymark et al. [15]. The search was updated to identify CEAs of vaccines from 1 August 2015 to 31 December 2019 and included the dengue vaccine, which was not included in the original review. CEAs were eligible for inclusion if the analysis included both costs and health effects and presented a decision-analytic model. Given the importance of model choice to accurately predict several additional unintended consequences, studies needed to present a model. Two reviewers independently screened titles and abstracts and reviewed the full texts to determine inclusion.

To identify tuberculosis CEAs, we included all of the CEAs of tuberculosis treatment that were included in the review by Siapka et al. [18]. The authors mainly searched the UCSR, PubMed, EMBASE, EconLit, Cochrane, NHS EED, and CEA Registry databases using broad searches including economic terms (e.g., cost, economic, or financial), disease-related terms (e.g., TB, tuberculosis, MDR [multi-drug resistant], XDR [extensively drug resistant]) and intervention-specific keywords (e.g., treatment, DOTS [directly observed treatment – short-course], isoniazid preventive therapy, patient cost). The full details of the data included, covering searches, article selection, and data extraction, are presented elsewhere [18]. We retrieved all the studies from 1 January 2009 to 31 December 2019, excluded cost studies, and selected a subset of studies if the analysis included both costs and health effects and presented a decision-analytic model in line with the criteria used in the review by Nymark et al. [15]. However, it should be noted that the review by Siapka et al. [18] only included studies that had primary data collection on costs. This means it missed other tuberculosis CEAs that, despite not having primary cost data collection, may have included transmission effects. Two reviewers independently screened the titles and abstracts and reviewed the full texts to determine inclusion. We also restricted this review to studies presented in English.

Data Analysis

We presented a comprehensive framework elsewhere that identifies and characterizes additional unintended costs and effects beyond that of the direct health impact primarily intended by the intervention/technology [16]. We briefly present and describe the framework (Fig. 1).

Fig. 1.

Conceptual framework ‘internal’ and ’external’ consequences

Interventions/technologies may have an impact beyond the intended direct health consequences. These are defined as “internal” consequences that occur within the individual (“internality”) or as “external” consequences that occur outside the individual (“externality”). There are several types of internalities and externalities within health impact. These types can be divided into biological effects, demand-side behavioral consequences, and supply-side behavioral consequences.

The “biological” types describe how cells and molecules within organisms interact and carry out their chemical and physical functions. It describes how these interactions are regulated, for example by control mechanisms and communication between cells. Within the type “biology,” there are three categories of potential additional internalities and externalities: (1) non-specific effects (NSE; impact on other diseases), (2) transmission (infection to others) or herd immunity effects (indirect protection), and (3) pathogen response (the pathogen is resistant). These effects are only relevant for vaccines and infectious diseases. The NSE of an immunization refers to the beneficial impacts of the immunization beyond protection against the pathogen it is directly intended for. Transmission effects are effects that reduce the transmission of an infectious agent from an infected individual to another individual. Herd immunity is one example: a form of indirect protection that occurs when a large enough percentage of the population is vaccinated (and therefore immune to infection) so that the unvaccinated individuals avoid infection. In the case of tuberculosis, interventions reduce the risk of infection to others as treatment reduces transmission; however, most tuberculosis interventions do not confer herd immunity. Pathogen response refers to the immune system’s response when an infectious agent causes disease or illness in its host. Vaccination can lead to serotype replacement, whereby the infectious agent with the serotype targeted by the vaccine is reduced or eliminated, allowing other serotypes the vaccine does not target to replace it. It can also induce cross-protection, which occurs when protection resulting from infection with one strain of a virus prevents infection by another related strain of that virus. Though the framework is specific to infectious diseases/vaccines for the biological type effects, it can apply in a broader sense when applied to other nonbiological impacts.

Demand-side impacts include individual, household, and population health-related consumption. There are two categories related to the health demand-side type: changes in health influencing behavior and changes in health services consumption (internalities). Changes in health influencing behavior refers to changes in a person’s actions, as a result of the intervention, that impact on their health. Changes in health services consumption refers to the changes, as a result of the intervention, in utilization of health services by a person for the purpose of promoting their health and well-being. On the supply-side impacts, the interventions can change the behavior of healthcare providers and impact on other health services or the provision of nonhealth services. Here, we identify one category that falls under the health sector perspective (health systems [external]) and two subcategories (side effects and scientific spill-over effects). Although the term side effect is predominantly used to describe adverse effects, it can also refer to unintended consequences of the use of the intervention. Scientific spill-over effects refer to the knowledge gained from development of a new drug or vaccine that might offer value beyond the drug or vaccine itself. For example, it could lead to further innovations in drug or vaccine development, the development of drugs or vaccines for other diseases, or the development of other health-related technologies.

Finally, there may be internal and external nonhealth consequences affecting the demand and supply side. For the demand side, we present two broad categories: behavior/education/knowledge (internal person) and consumption of nonhealth goods. Categories under this are intrahousehold (subcategories: informal care and change in behavior) and education and labor productivity. For the supply side, we identify two categories under the nonhealth consequences perspective: outside health systems (subcategory: public services) and provision of nonhealth services (subcategory: change in behavior).

We assessed and recorded which of the categories and associated subcategories for both nonhealth and health consequences were included in each of the immunization and tuberculosis CEAs. For each type, we also extracted the methods used to estimate the additional unintended consequences.

Reporting Quality Appraisal of Included Cost-Effectiveness Analyses (CEAs)

We used the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, consisting of 24 requirements, to appraise the quality of the included CEAs [2]. Data on the 24 items in the CHEERS statement were extracted, such as whether the article included a clear description and justification of the model used, a reference to the choice of health outcomes, mention of the measurement of effectiveness, and estimation of the resources and costs. For each of the 24 items, we assigned a yes/no judgment and then calculated the total number of confirming items (“yes”) to assess the overall reporting quality of each study.

Results

Paper Selection

The PubMed/MEDLINE and Embase database searches for articles published between 2015 and 2019 for vaccines returned 1074 papers after duplicates were removed (Fig. 2a). We screened 239 full-text articles, 163 of which did not meet the inclusion criteria: 32 did not present a clear model based on the model type classification used in the review by Nymark et al. [15], 128 focused on high-income countries, two were in Spanish, and one was a review. We screened 76 CEAs for inclusion of any of the categories included in the framework for additional unintended consequences, and 43 of these were included. Of the 101 articles focusing on low- and middle-income countries between 2009 and 2015 from the review by Nymark et al. [15], 55 articles included one or more categories.

Fig. 2.

a PRISMA flowchart article selection—vaccines 2009–2019. b PRISMA flowchart article selection—tuberculosis 2009–2019

To identify tuberculosis CEAs, 205 studies from the review by Siapka et al. [18] presenting primary collected tuberculosis cost data in low- and middle-income countries were screened. Of these, 65 articles reporting only costs were excluded (Fig. 2b). We assessed the remaining 140 articles for eligibility based on the criteria used by Nymark et al. [15] and found that 111 did not meet the inclusion criteria: one was a duplicate, three had no clear model classification, and the remaining 107 did not combine costs and effects. We screened 29 articles and found that ten included any of the categories included in the framework for additional unintended consequences.

Number of CEAs Including Categories of Additional Unintended Consequences

A total of 177 vaccine CEAs from 2009 to 2019 were eligible for inclusion in the review. In total, 98 vaccine CEAs included one or more categories or subcategories of additional unintended consequences (55%) (see Table 1). For health consequences, the category “biology” was included 73 times (41%). This was split between the subcategories cross-protection and resistance, which were included 15 times each (8%), and herd immunity, which was included the highest number of times (43 [24%]). We identified no CEAs with the category “non-specific effects.” Additionally, under health consequences on the supply side, the subcategory side effects was included five times (3%). We identified no CEAs that included health consequences on the demand side. For nonhealth consequences, labor productivity was the only category included on the demand-side (60 times [34%]). We identified no CEAs that included nonhealth consequences on the supply side.

Table 1.

Summary of cost-effectiveness analyses including additional unintended consequences (2009–2019)

Vaccine, study, year, country	Health				Nonhealth
	Biology			Supply	Demand
	Transmission	Pathogen response		Health systems	Intra household
	Indirect protection	Resistance	Cross-protection	Side effects	Productivity
Dengue
Shim [57], 2017, Brazil	Yes
Fitzpatrick et al. [58], 2017, Six LMICa	Yes
Shafie et al. [72], 2017, Malaysia			Yes		Yes
Zeng et al. [73], 2018, Ten LMICb			Yes		Yes
Perera et al. [111], 2019, Sri Lanka					Yes
Rotavirus
Atherly et al. [98], 2009, Gavi-eligible countries					Yes
Clark et al. [94], 2009, Peru					Yes
Flem et al. [86], 2009, Kyrgyzstan					Yes
Kim et al. [89], 2009, Vietnam					Yes
Rheingans et al. [92], 2009, Gavi-eligible countries					Yes
Rose et al. [99], 2009, India					Yes
Wang et al. [77], 2009, China				Yes	Yes
Wilopo et al. [91], 2009, Indonesia					Yes
Chotivitayatarakorn et al. [87], 2010, Thailand					Yes
Kim et al. [90], 2010, Gavi-eligible countries					Yes
Jit et al. [84], 2011, Armenia					Yes
Atherly et al. [45], 2012, Gavi-eligible countries	Yes
Liu et al. [93], 2012, China					Yes
Tu et al. [96], 2012, Vietnam					Yes
Alkoshi et al. [97], 2014, Libya					Yes
de Blasio et al. [85], 2014, Kazakhstan					Yes
Ahmeti et al. [88], 2015, Albania					Yes
Diop et al. [42], 2015, Senegal	Yes
Javanbakht et al. [43], 2015, Iran	Yes
Sigei et al. [44], 2015, Kenya & Uganda	Yes
Rose et al. [46], 2017, India	Yes				Yes
Loganathan et al. [95], 2018, Malaysia					Yes
Measles
Bishai et al. [115], 2011, Uganda					Yes
Levin et al. [60], 2011, Six LMICc	Yes
Varicella
Esmaeeli et al. [119], 2017, Iran					Yes
You et al. [78], 2019, China				Yes
Influenza
Meeyai et al. [75], 2015, Thailand				Yes
Jamotte et al. [74], 2017, Latin America			Yes		Yes
Vo et al. [76], 2018, Vietnam				Yes	Yes
Sribhutorn et al. [110], 2018, Thailand					Yes
Polio
Duintjer Tebbens et al. [59], 2011, 104 countriesd	Yes				Yes
HAV
Carlos et al. [120], 2016, Mexico					Yes
HBV
Tu et al. [104], 2012, Vietnam					Yes
Lu et al. [105], 2013, China					Yes
Jia et al. [103], 2014, China					Yes
Zheng et al. [102], 2015, China					Yes
Lee et al. [100], 2018, South Korea					Yes
Wang et al. [101], 2019, China					Yes
HPV
Aguilar et al. [114], 2015, Honduras					Yes
Novaes et al. [112], 2015, Brazil					Yes
Setiawan et al. [68], 2016, Indonesia			Yes
Bardach et al. [69], 2017, Venezuela			Yes
Van Minh et al. [113], 2017, Vietnam					Yes
Germar et al. [70], 2017, Philippines			Yes
Van Kriekinge et al. [71], 2018, Malaysia			Yes
Burger et al. [54], 2018, Gavi-eligible countries	Yes
Portnoy et al. [53], 2019, Uganda	Yes
Pneumococcal
Vespa et al. [19], 2009, Brazil	Yes				Yes
Kim et al. [28], 2010, The Gambia	Yes	Yes	Yes
Aljunid et al. [20], 2011, Malaysia	Yes			Yes
Nakamura et al. [21], 2011, 77 LMIC	Yes	Yes	Yes
Sartori et al. [79], 2012, Brazil					Yes
Ayieko et al. [22], 2013, Kenya	Yes	Yes			Yes
Gomez et al. [23], 2013, Peru	Yes	Yes	Yes		Yes
Kulpeng et al. [24], 2013, Thailand	Yes	Yes			Yes
Martí et al. [29], 2013, Six Latin American countriese	Yes	Yes	Yes		Yes
Sibak et al. [38], 2015, Egypt	Yes	Yes
Che et al. [25], 2014, China	Yes				Yes
Hu et al. [30], 2014, China	Yes
Caldwell et al. [26], 2015, China	Yes
De Soarez et al. [80], 2015, Brazil					Yes
Kieninger et al. [35], 2015, Paraguay	Yes	Yes
Komakhidze et al. [36], 2015, Georgia	Yes	Yes
Mezones-Holguin et al. [37], 2015, Peru	Yes	Yes	Yes
Ordonez et al. [27], 2015, Colombia	Yes
Constenla [83], 2015, Three LMICf					Yes
Mo et al. [39], 2016, China	Yes
Zhao et al. [82], 2016, China					Yes
Sundaram et al. [33], 2017, Mongolia	Yes	Yes			Yes
Wang et al. [31], 2017, Malaysia	Yes	Yes	Yes
Castaneda-Orjuela et al. [66], 2018, Colombia		Yes	Yes
Zhou et al. [32], 2018, China	Yes	Yes			Yes
Shen et al. [41], 2018, China	Yes
Lara et al. [40], 2018, Colombia	Yes	Yes			Yes
Dorji et al. [34], 2018, Bhutan	Yes
Kebede et al. [81], 2019, Ethiopia					Yes
Marijam et al. [67], 2019, Turkey			Yes
Meningococcal
De Soarez et al. [116], 2011, Brazil					Yes
Hib
Griffiths et al. [52], 2011, Belarus and Uzbekistan	Yes
Muangchana et al. [108], 2011, Thailand					Yes
Moradi-Lakeh et al. [106], 2012, Iran					Yes
Clark et al. [51], 2013, India	Yes
Griffiths et al. [107], 2013, Gavi-eligible countries					Yes
Le et al. [109], 2015, Vietnam					Yes
Ning et al. [50], 2018, China	Yes
Typhoid fever
Antillon et al. [55], 2017, LMICg	Yes
Lo et al. [118], 2018, LMICh					Yes
Bilcke et al. [56], 2019, Gavi-eligible countries	Yes
Cholera
Jeuland et al. [48], 2009, LMICi	Yes				Yes
Schaetti et al. [47], 2012, Zanzibar (Tanzania)	Yes				Yes
Khan et al. [49], 2018, Bangladesh	Yes
Diphtheria tetanus
Sartori et al. [117], 2016, Brazil					Yes
Fernandes et al. [61], 2019, Brazil	Yes
Tuberculosis
Datiko et al. [122], 2010, Ethiopia					Yes
Steffen et al. [123], 2010, Brazil					Yes
Prado et al. [124], 2011, Brazil					Yes
Winetsky et al. [62], 2012, Russia and Latvia	Yes	Yes
Wang et al. [126], 2014, China					Yes
Sekandi et al. [121], 2015, Uganda					Yes
Vassall et al. [125], 2017, South Africa					Yes
Wikman-Jorgensen et al. [63], 2017, Mozambique	Yes
Mandalakas et al. [64], 2013, South Africa	Yes
Fitzpatrick et al. [65], 2015, China	Yes

HAV hepatitis A virus, HBV hepatitis B virus, Hib haemophilus influenzae type b, HPV human papillomavirus, LMIC low- to middle-income countries

^aBrazil, Columbia, Malaysia, Mexico, the Philippines, and Thailand

^bIndonesia, Malaysia, Philippines, Thailand, Vietnam, Brazil, Colombia, Honduras, Mexico, and Puerto Rico

^cBangladesh, Brazil, Colombia, Ethiopia, Tajikistan, and Uganda

^d64 low-, 35 lower middle-, and five upper middle-income countries

^eArgentina, Brazil, Chile, Colombia, Mexico and Peru

^fEcuador, Honduras, and Paraguay

^gIndia, Kenya, and Vietnam

^hSouth Asia

ⁱIndia, Indonesia, and Mozambique

A total of 29 tuberculosis CEAs from between 2009 and 2019 were eligible for inclusion. Of these, ten (34%) included one or more categories or subcategories of additional unintended consequences. Under the health type “biology,” transmission effects were identified in four CEAs (14%), and resistance was identified in one CEA (3%). We did not identify any demand or supply categories under health consequences. For nonhealth consequences, on the demand side, we identified labor productivity in six CEAs (21%). We did not identify any supply categories under nonhealth consequences.

Methods Used to Measure Additional Unintended Consequences

Health Consequences: Biology

Transmission: Indirect protection

We identified 43 vaccine CEAs that included herd immunity.

Pneumococcal: We identified 23 pneumococcal vaccine CEAs that considered herd immunity effects. Vespa et al. [19], Aljunid et al. [20], Nakamura et al. [21], Ayieko et al. [22], Gomez et al. [23], Kulpeng et al. [24], Che et al. [25], Caldwell et al. [26], and Ordonez et al. [27] all estimated herd immunity effects from US surveillance data in unvaccinated people for different age groups. Assumptions about the reduction of incidence of invasive pneumococcal diseases were used to account for herd immunity in the CEAs by Kim et al. [28], Martí et al. [29], Hu et al. [30], Wang et al. [31], Zhou et al. [32], Sundaram et al. [33], and Dorji et al. [34]. Kieninger et al. [35], Komakhidze et al. [36], Mezones-Hilguin et al. [37], Sibak et al. [38], and Mo et al. [39] used a simple multiplier to calculate a percentage increase in health benefits due to herd immunity effects. In sensitivity analyses, Lara et al. [40] included herd immunity but did not state which method was used to estimate these effects. In the CEA by Shen et al. [41], estimates of herd immunity effects were generated by the calculation of the reduction in carriage of vaccine serotypes after vaccine introduction.

Rotavirus: We identified five rotavirus vaccine CEAs that included the category herd immunity. Diop et al. [42], Javanbakht et al. [43], and Sigei et al. [44] estimated herd immunity effects using a multiplier and inflating health benefits to 120% of direct effects in children aged <5 years. In a sensitivity analysis, Atherly et al. [45] assumed that unvaccinated children would receive 15% protection at 50% vaccination coverage These indirect effect scenarios assumed that nonvaccinated children would receive a level of protection proportional to the efficacy in vaccinated children and the level of coverage. Rose et al. [46] used a dynamic model of rotavirus transmission to account for herd immunity effects.

Cholera: We identified three cholera vaccine CEAs that considered herd immunity.

Schaetti et al. [47] estimated herd immunity effects by multiplying the annual incidence of cases without vaccination by the protective efficacy among unvaccinated people. Jeuland et al. [48] used a mathematical equation linking oral cholera vaccine effectiveness to varying coverage rates in the study population. Khan et al. [49] used a dynamic model of cholera transmission to include herd immunity effects.

Haemophilus influenzae type b (Hib): We identified three Hib vaccine CEAs that included herd immunity. The Hib vaccine CEAs by Ning et al. [50], Clark et al. [51], and Griffiths et al. [52] assumed that herd immunity effects would increase the vaccine’s impact by 20%.

Other vaccines: The human papillomavirus (HPV) CEAs by Portnoy et al. [53], and Burger et al. [54] used a disease transmission dynamic approach to account for herd immunity. The typhoid fever CEAs by Antillon et al. [55] and Bilcke et al. [56] used a dynamic model of typhoid transmission to capture herd immunity effects. The two dengue CEAs by Shim [57] and Fitzpatrick et al. [58] used age-dependent dynamic transmission models to account for herd immunity effects. The polio vaccine CEA by Duintjer Tebbens et al. [59] used a dynamic transmission model. The measles vaccine CEA by Levin et al. [60] used a dynamic transmission model to include herd immunity effects. The diphtheria tetanus CEA by Fernandes et al. [61] used a disease dynamic transmission approach to account for herd immunity.

We identified four tuberculosis CEAs that included transmission effects.

Tuberculosis: Winetsky et al. [62] used a dynamic transmission model to account for indirect effects (reduced risk of the infection to others when treating tuberculosis). Wikman-Jorgensen et al. [63] and Mandalakas et al. [64] used Markov models to capture transmission effects. Fitzpatrick et al. [65] considered transmission effects using a mathematical model, but how the secondary cases were included was unclear.

Pathogen Response: Resistance

We identified 16 CEAs that included resistance.

Pneumococcal: We identified 15 pneumococcal vaccine CEAs that included resistance (serotype replacement). Nakamura et al. [21], Ayieko et al. [22], Gomez et al. [23], and Kulpeng et al. [24] assumed a US serotype replacement effect and modeled an increase in nonvaccine-type invasive pneumococcal disease following the introduction of 7-valent pneumococcal vaccine (PCV-7). Serotype replacement was assumed as the cause of an increase in acute otitis media disease due to noncovered pneumococcal serotypes for PCV-7 (33% increase) in the CEAs by Kim et al. [28], Martí et al. [29], Wang et al. [31], Zhou et al. [32], and Sundaram et al. [33]. In the CEAs by Kieninger et al. [35], Komakhidze et al. [36], Mezones-Hilguin et al. [37], Sibak et al. [38], and Castaneda-Orjuela et al. [66], a simple multiplier was used to calculate the percentage at which the serotype replacement could impair the indirect protection. Lara et al. [40] did not state which methods were used.

Tuberculosis: We identified one tuberculosis CEA that included resistance (pathogen resistance). Winetsky et al. [62] estimated the impact on resistance itself (i.e., acquired resistance or onwards transmission) using a dynamic transmission model.

Pathogen Response: Cross-Protection

We identified 15 vaccine CEAs that included cross-protection.

Pneumococcal: We identified eight pneumococcal vaccine CEAs that included cross-protection. Nakamura et al. [21], Kim et al. [28], Martí et al. [29], Wang et al. [31], Castaneda-Orjuela et al. [66], Mezones-Hilguin et al. [37], Gomez et al. [23], and Marijam et al. [67] assumed cross-protection for pneumococcal polysaccharide protein D-conjugate vaccine against serotype 6A to be equal to that of PCV-7 (76%) on the basis of noninferiority immunogenicity data.

HPV: We identified four HPV vaccine CEAs that included cross-protection. Setiawan et al. [68], Bardach et al. [69], Germar et al. [70], and Van Kriekinge et al. [71] assumed an effect of cross-protection against HPV types at various percentages of vaccine efficacy.

Dengue: We identified two dengue vaccine CEAs that included cross-protection. Shafie et al. [72] and Zeng et al. [73] used a dynamic transmission approach to account for temporary or permanent cross-protection, cross-enhancement, or their combination.

Influenza: We identified one influenza vaccine CEA that included cross-protection. Jamotte et al. [74] assumed an effect of cross-protection against influenza types at various percentages of vaccine efficacy.

Health Consequences: Supply

Health Systems: Side effects

We identified five vaccine CEAs that included side effects.

Influenza: We identified two influenza vaccine CEAs that included side effects. In the study by Meeyai et al. [75], side effects were incorporated as disutility values in the disability-adjusted life-year measure. Vo et al. [76] did not state the methods used.

Rotavirus: We identified one rotavirus vaccine CEA, by Wang et al. [77], that included side effects but did not state the methods used.

Varicella: We identified one varicella vaccine CEA, by You et al. [78], that included side effects. In the study, side effects were incorporated as disutility values in the quality-adjusted life-year (QALY) measure.

Pneumococcal: We identified one pneumococcal vaccine CEA, by Aljunid et al. [20], that included side effects but not the methods used.

Nonhealth Consequences: Demand

Intrahousehold: Labor Productivity

We identified 66 CEAs that included labor productivity.

Pneumococcal: We identified 14 pneumococcal vaccine CEAs that included labor productivity. Gomez et al. [23], Martí et al. [29], Sartori et al. [79], Che et al. [25], Lara et al. [40], Vespa et al. [19], and De Soarez et al. [80], Zhou et al. [32], Kebede et al. [81], and Sundaram et al. [33] all used the human capital approach to estimate labor productivity loss. Kulpeng et al. [24], Zhao et al. [82], Ayieko et al. [22], and Constenla et al. [83] did not state the methods used.

Rotavirus: We identified 18 rotavirus vaccine CEAs that included labor productivity. Wang et al. [77], Jit et al. [84], and de Blasio et al. [85], Flem et al. [86], Chotivitayatarakorn et al. [87], Ahmeti et al. [88], Kim et al. [89, 90], Wilopo et al. [91], Rheingans et al. [92], Liu et al. [93], and Clark et al. [94] used the human capital approach to estimate labor productivity loss. In the studies by Loganathan et al. [95] and Tu et al. [96], labor productivity losses were embodied in the QALY measure. Alkoshi et al. [97], Atherly et al. [98], and Rose et al. [46, 99] did not state the methods used.

Hepatitis B: We identified six hepatitis B vaccine CEAs that included labor productivity. Lee et al. [100], Wang et al. [101], and Zheng et al. [102] all used the human capital approach to estimate labor productivity loss. In the study by Jia et al. [103], losses in labor productivity were incorporated in health-related quality of life (HRQoL). Tu et al. [104] and Lu et al. [105] did not state the methods used.

Hib: We identified four Hib vaccine CEAs that included labor productivity. Moradi-Lakeh et al. [106], Griffiths et al. [107], and Muangchana et al. [108] used the human capital approach to estimate labor productivity loss. Le et al. [109] used the friction cost method to estimate the indirect cost due to productivity loss.

Influenza: We identified three influenza vaccine CEAs that included labor productivity. Jamotte et al. [74], Vo et al. [76], and Sribhutorn et al. [110] used the human capital approach to estimate labor productivity loss.

Dengue: We identified three dengue vaccine CEAs that included labor productivity. Zeng et al. [73], Shafie et al. [72], and Perera et al. [111] used the human capital approach to estimate labor productivity loss.

HPV: We identified three HPV vaccine CEAs that included labor productivity. Novaes et al. [112], Van Minh et al. [113], and Aguilar et al. [114] used the human capital approach to estimate labor productivity loss.

Other vaccines: The two cholera vaccine CEAs, by Jeuland et al. [48] and Schaetti et al. [47], used the human capital approach to estimate labor productivity loss. The measles vaccine CEA by Bishai et al. [115] used the human capital approach. The polio CEA by Duintjer Tebbens et al. [59] used the human capital approach. The meningococcal vaccine CEA by De Soarez et al. [116] used the human capital approach. The tetanus-diphtheria-acellular pertussis vaccine CEA by Sartori et al. [117] used the human capital approach to estimate labor productivity loss. The typhoid fever vaccine CEA by Lo et al. [118] used the human capital approach to estimate labor productivity loss. The varicella vaccine CEA by Esmaeeli et al. [119] used the human capital approach to estimate labor productivity loss. The hepatitis A vaccine CEA by Carlos et al. [120] used the human capital approach to estimate labor productivity loss.

Tuberculosis: We identified six tuberculosis CEAs that included labor productivity. Sekandi et al. [121], Datiko et al. [122], Steffen et al. [123], Prado et al. [124], Vassall et al. [125], and Wang et al. [126] all used the human capital approach to estimate labor productivity loss.

Reporting Quality Assessment

In total, 85% of the included studies met 20 or more of the 24 CHEERS checklist criteria. The mean number across the 108 CEAs was 21, with a range between 13 and 24 checklist criteria. Table S1 in the electronic supplementary material provides a complete overview of conforming items for individual CEAs.

Discussion

Even though tuberculosis treatments and the vaccines covered in this review have clear indirect effects, only 34% of tuberculosis CEAs included additional unintended consequences, compared with 55% of vaccine CEAs. One factor that may account for the low proportion of studies for tuberculosis is that the review by Siapka et al. [18], which we used to identify tuberculosis CEAs, only included studies that had some sort of primary data collection on costs. This means it missed out other tuberculosis CEAs that, despite not having primary cost data collection, may have included transmission and therefore may have increased the proportion of tuberculosis studies. Tuberculosis has an evidenced impact on labor productivity, but these costs were only included in 21% of the tuberculosis CEAs [127]. Productivity savings are also evident for some vaccines (e.g., Hib and pneumococcal), particularly work days lost by a parent caring for a sick child. In the case of influenza, there is well-established evidence of both household and macro-economic impact, yet these effects were only included in 34% of all vaccine CEAs [128].

Aside from this, other aspects stand out as areas of potential under-inclusion in CEAs. Resistance is a critical issue in both immunization and tuberculosis. We have used the word resistance in a general sense as describing a situation where a pathogen shifts so it no longer reacts to medicines or immunization. For example, in tuberculosis, resistance to several medicines is widespread and described by the World Health Organization as drug-resistant tuberculosis. In the case of vaccines, resistance is an internal biological consequence as serotype replacement is a phenomenon that induces resistance to subtypes of serotypes if the frequency of a subtype of serotype declines because of high levels of immunity, allowing other serotypes to replace it. However, resistance was considered in only in 8% of all vaccine CEAs compared with 3% of all tuberculosis CEAs. Likewise, cross-protection and side effects were also likely to be substantially under-included. Cross-protection is the protection conferred on a host (“internal”) by infection with one strain of a virus that prevents infection by a closely related substrain of that virus. This biological effect is particularly relevant for pneumococcal, HPV, dengue, and influenza vaccinations. However, cross-protection was only included in 8% of all vaccine CEAs. Side effects were only included in 3% of all vaccine CEAs. The biological category “non-specific effects” of vaccines was not included in any CEAs. These are beneficial effects that offer protection beyond specific pathogens and are particularly important for live vaccines. A clear distinction must be made between non-specific effects of vaccines that are due to hypothesized vaccine-induced improvements to vaccinees’ immunity against nontargeted antigens, and non-specific disease outcomes. The latter category is often used in vaccine impact evaluations and CEAs because pathogen-specific disease outcomes (e.g., gastroenteritis due to rotavirus) are difficult to measure and because the etiology of diseases (e.g., diarrheal diseases occurring after measles or circulatory diseases occurring soon after an influenza infection) is often unclear. For instance, nonlive vaccines (e.g., inactivated flu vaccines) can have an impact on non-specific outcomes (e.g., circulatory diseases).

As mentioned previously, the inclusion of additional unintended consequences in economic evaluation is mainly relevant in studies using a social perspective, though some additional unintended consequences can also be relevant to the healthcare system perspective. Whether to include or exclude additional unintended consequences ideally is based on weighting the likely importance, the extent of evidence, and the analytical complexity of doing so. In part, the lack of data, and appropriate methods to include several of the additional unintended consequences covered in this review, may be because of the current novelty of some of these measures. However, this is unlikely to apply to transmission or labor productivity effects. Although dynamic transmission models are often recommended and are important for capturing herd immunity effects, we found a lack of use of this type of model in the CEAs reviewed. Methodologically, we found a strong reliance on the human capital approach to measure the loss of productivity across the disease areas covered in the review. For more minor consequences, comprehensive guidelines for economic evaluations about which of these additional unintended consequences should be reported and how are lacking. Analytically, the complexity of the relationships between internal and external nondirect health and nonhealth impact and how to quantify this requires a range of types of evidence and techniques, but guidance on how to address feedbacks between different consequences is sparse. Furthermore, quantifying and linking changes in nondirect health effects to nonhealth impact (e.g., behavioral outcomes) is complex. Given the difficulty with measuring, it may be challenging to develop measurements capturing all relevant health and economic consequences in immunization, and tuberculosis, respectively.

This is the first review to evaluate the inclusion of additional unintended consequences in economic evaluation studies as presented in the framework by Nymark and Vassall [16]. However, we only included vaccine and tuberculosis CEAs in our search. We acknowledge this as a limitation as it is therefore not possible to generalize the results obtained across several disease areas. We acknowledge that additional unintended consequences may bias results either way. For example, although some unintended consequences, such as reducing susceptibility to other diseases, are likely to be positive, others are likely to be negative. We highlight the latter in the case of tuberculosis as failing to include transmission and productivity may bias results to underestimate cost effectiveness. However, without conducting a formal analysis, we cannot assess this, and the review aims more to identify the extent and nature of the inclusion of these consequences rather than to summarize the extent of their impact on results and conclusions in each study. In addition, the analysis carried out included only papers for low- and middle-income countries. We acknowledge this as a limitation, but to account for the differences between the two reviews used in the analysis, we restricted the review by Nymark et al. [15].

The results of this review should not be taken to suggest that all additional consequences should be included in every economic evaluation but instead should encourage analysts to provide transparency where unintended additional consequences are excluded and to provide the reasons for this. We have previously provided a clear framework that can be used [16].

At the very least, we recommend moving towards reporting against a comprehensive framework of types and categories of additional effects. We hope that further transparency in this aspect of CEA is a feasible first step toward accounting for additional unintended consequences in economic evaluations.

Conclusion

The inclusion of additional unintended consequences in economic evaluations of immunization and tuberculosis continues to be limited, even though they offer valuable information to analysts. Only 34% of tuberculosis CEAs included additional unintended consequences, compared with 55% of vaccine CEAs. Further work on appropriate ways to value additional unintended consequences is still warranted, especially for those that occur outside the health system. In particular, work is still needed on how to link changes in internal consequences to external consequences and on combining several additional unintended consequence categories or subcategories in economic evaluations.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

Funding was received from the Dutch Ministry of Foreign Affairs (grant number DSO/GA-264/15).

Conflict of interest

Liv Nymark, Alex Miller, and Anna Vassall have no conflicts of interest that are directly relevant to the content of this article.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material

The datasets generated and/or analyzed in the current study are available from the corresponding author on reasonable request.

Code availability

Not applicable.

Author contributions

The study was designed by LN and AV. The search and extraction were carried out by AM. AM and LN conducted second reviews of the data extraction results. Analysis and interpretation of the data was carried out by LN. LN prepared the manuscript. All authors read and approved the final manuscript. LN is the guarantor of this work.