Assessing key model parameters for economic evaluation of pandemic influenza interventions: the data source matters

Abstract

Background

In our previous systematic review of economic evaluations of pandemic influenza interventions, five model parameters, namely probability of pandemic, duration of pandemic, severity, attack rate, and intervention efficacy, were not only consistently used in all studies but also considered important by authors.

Objectives

Because these parameters originated from sources of varying quality ranging from experimental studies to expert opinion, this study aims to analyze the variation in values used according to sources of information across studies.

Methods

An analysis of estimated values of key parameters for economic modeling was performed against their different data sources, following the standard hierarchy of evidence.

Results

A lack of good‐quality evidence to estimate pandemic duration, pandemic probability, and mortality reduction from antiviral treatment results in a large variation of values used in economic evaluations. Although there are variations in quality of evidence used for attack rate, basic reproduction number, and reduction in hospitalizations from antiviral treatment, the estimated values do not vary significantly. The use of higher‐quality evidence results in better precision of estimated values compared to lower‐quality sources.

Conclusion

Hierarchies of evidence are a necessary tool to identify appropriate model parameters to populate economic evaluations and should be included in methodological guidelines. Knowledge gaps in some key parameters should be addressed, because if good‐quality evidence is available, future economic evaluations will be more reliable. Some gaps may not be fulfilled by research but consensus among experts to ensure consistency in the use of these assumptions.

Background

The H5N1 and pH1N1 outbreaks that occurred in recent years directed international attention toward the cost‐effectiveness of interventions aiming to prevent and control pandemic influenza. As a result, and part of its Public Health Research Agenda for Influenza, the World Health Organization (WHO) commissioned the Health Intervention and Technology Assessment Program (HITAP) of Thailand to conduct a systematic review of preparedness strategies and interventions against pandemic influenza, published in early 2012.1 By offering a comprehensive framework for efficient allocation of resources – a tool that was lacking until then – the study intended to serve as a reference for the future revision of the WHO pandemic preparedness and response guidance.2

In our study, we searched relevant databases as well as screened references and contacted authors up to September 2011. Eligible papers were full and partial economic evaluations including both costs and outcomes, while editorials, reviews, and papers on economic impact or complications were excluded. We selected a total of 44 evaluations for the review. Although in general the methods applied were appropriate, we detected important shortcomings in the quality of evidence used. There were also considerable variations in drug regimens and vaccination protocols. In summary, pharmaceutical interventions ranged from cost‐saving to high cost‐effectiveness ratios. Combinations of pharmaceutical and non‐pharmaceutical interventions were cost‐effective compared with vaccines and/or antivirals alone. Reduction in contacts, prevention with antivirals together with school closure demonstrated to be especially cost‐effective for all countries. In contrast, quarantine for household contacts was cost‐ineffective in all settings. Finally, we provided recommendations on practical issues necessary to improve the quality and generalizability of economic evaluation studies in the future. In particular, we underlined the importance of five model parameters (i.e., probability of pandemic, duration of pandemic, severity, attack rate, and intervention efficacy/effectiveness) that were not only consistently employed in all evaluations but also considered important by study authors.

In this study, our purpose is to describe and analyze the variation in key parameter values employed according to sources of information across studies. These key parameters originate from different sources of varying quality ranging from experimental studies to expert opinion. An investigation into this variation is warranted, with the aim to promote the reaching of consensus on certain important parameters used for future economic evaluations and identify future priority research areas.

Methods

We conducted a descriptive analysis of five key parameters for economic modeling of pandemic influenza interventions. The identified parameters were cross‐tabulated against the different data sources, following the hierarchy developed by Cooper et al.3 In this hierarchy, different data sources are assessed according to their level of quality: a) clinical effect sizes, b) adverse events and complications, c) baseline clinical data, d) resource use, e) costs, and f) utilities (only in cost utility analyses). Parameter sources are given a rank from 1–6 and 9 in descending order, with rank 1 applied to parameters derived from the best quality sources. In summary, in the case of clinical effect sizes/adverse events and complications, rank 1 is given to meta‐analyses of randomized controlled trials (RCTs) or RCTs that directly evaluate comparator interventions and quantify final outcomes; rank 2 is given to similar designs but measuring surrogate outcomes or using placebo as a comparator while measuring final outcomes for each intervention; rank 3 is applied to meta‐analyses or RCTs that use placebo as comparator and measure surrogate outcomes; rank 4 is given to observational studies; rank 5 to non‐analytic studies; and, finally, rank 6 and 9 are given to expert opinion and cases where the source is not clear, respectively. In the case of baseline clinical data, rank 1 is given to purposely conducted case series/analyses of dependable databases including patients from the study setting; rank 2 is given to similar studies that were conducted recently; rank 3 is given to similar studies conducted recently in a different setting; rank 4 is given when these studies are old or the estimates are derived from RCTs; rank 5 is given to estimates retrieved from other economic evaluations; and ranks 6 and 9 are given in the same fashion as for clinical effect sizes/adverse events and complications.

Results

Key parameters related to baseline clinical data

Figure 1 illustrates the variation of means of the parameters for attack rate, basic reproduction number (R₀), pandemic probability, and pandemic duration. The graph clearly demonstrates a lack of high‐quality evidence for pandemic probability and pandemic duration, because authors of all reviewed papers derive these parameters from previous economic evaluations, expert opinion, or unclear sources.

Figure 1.

Hierarchy of quality of evidence for pandemic duration, probability of pandemic, basic reproduction number, and attack rate parameters. Vertical lines represent the range of values. Square boxes represent average values. The numbers above each line are the number of studies. The hierarchy of evidence is based on Cooper et al.3

For pandemic duration, there is only one study using data from a previous economic evaluation of antiviral stockpiling in Singapore, which estimates pandemic duration at 12 weeks.4 In six studies that estimate pandemic duration from expert opinion, the value varies from 15 to 43 weeks. Regarding pandemic probability, although most studies apply the common belief that a pandemic is expected to occur every 30 years, there is one study where the estimate is unexpectedly five years.5

In the cases of attack rate and R₀, authors of the reviewed studies select evidence from a wide array of data sources ranging from recent cases series/analyses of reliable administrative databases to expert opinion. Although higher‐quality sources tend to provide less variation in estimates, there are not considerable differences in absolute values used.

Key parameters related to clinical effect sizes

As for pharmaceutical interventions, there is a knowledge gap in the value of antiviral efficacy measured as mortality reduction, with all six studies using parameters derived from expert opinion or unclear sources (Figure 2). This is not the case for the estimates of antiviral efficacy measured as reduction in hospitalizations, where authors employ a broader range of data sources. In the studies, the absolute values are also similar. For vaccine efficacy, the variations in estimates observed are large, especially evident in seven studies where authors use expert opinion or estimated from other sources, for example one study based on previous pandemics6, one study based on seasonal influenza7, and five additional studies where the source of the estimates is unclear.5, 8, 9, 10, 11

Figure 2.

Hierarchy of quality of evidence for intervention efficacy parameters. Vertical lines represent range of values. Square boxes represent average values. The numbers above each line are the number of studies. The hierarchy of evidence is based on Cooper et al.3

To assess the effect of the 2009 pandemic on the quality of evidence used in economic evaluation studies, Figure 3 compares the ranking of each parameter in studies conducted before and after the 2009 pandemic. Even though the number of studies is small, the 2009 pandemic seems to provide a positive benefit to the quality of evidence used for two of the parameters, namely attack rate and R₀, but not for the other parameters.

Figure 3.

Comparison of number of studies according to quality of evidence between ex‐ante and ex‐post studies. Ex‐ante = before the 2009 H1N1 pandemic; Ex‐post = after the 2009 H1N1 pandemic. The hierarchy of evidence is based on Cooper et al.3

Discussion

From our analysis, we identified an important knowledge gap in three key parameters necessary for economic evaluation studies of pandemic influenza preparedness strategies and interventions. This includes pandemic duration, pandemic probability, and mortality reduction from antiviral treatment. Because there is no high‐quality evidence for these parameters, resulting both in a large variation of estimated values used and a high impact on economic evaluation results, this will ultimately hinder cross‐study comparisons of economic information to guide policy decisions.

In the cases of attack rate, R₀, and reduction of hospitalizations from antiviral treatment, there are not large variations in values used. This may be because there is already high‐quality evidence available and that most experts are aware of the existing evidence. For example, because there were two studies conducted by Khazeni et al.12 and Tuite et al.13 that analyzed information from U.S. and Canadian administrative databases14 to estimate R₀, and economic evaluations conducted after that year employed an estimate that did not differ significantly from the results of those two studies. Even in three other studies published more recently in 200915, 201016, and 201117, which derived the value of R₀ from the authors’ own assumptions, we found these estimates comparable to the results from the two studies mentioned above.13, 14

There are some limitations in this study. As this study was derived from a systematic review completed in September 2011, studies published later are not included. Nonetheless, an update of our search strategy in MEDLINE/PubMed covering from September 2011 to September 2012 indicates that, although the scope of settings and interventions has broadened, there are few eligible additional economic evaluations18, 19, 20, 21 and most of the studies are cost analyses.22, 23, 24, 25 We believe that our results are still valid even though these new studies are not included. Moreover, we only reviewed economic evaluation studies and, therefore, can only capture the data sources selected by these studies. If there is better‐quality evidence available but not used in these reviewed studies, it is not included in our analysis. Lastly, we employed a hierarchy of evidence developed for health economic evaluations of general diseases. However, pandemic influenza is not a disease where the past is a reliable guidance for the future. Hence, the hierarchy may not always be relevant for all parameters. For example, results from genetic studies that are not population‐based may be more reliable in giving an indication of how a pandemic flu strain may drift and attenuate over time than observational studies of the past pandemic.

Recommendations

Firstly, we found that hierarchies of evidence are a necessary tool to help identify appropriate model parameter estimates to populate economic evaluations. Research funders and health economic evaluation methodological guideline developers should request that researchers select the highest quality data sources as possible according to standard hierarchies of evidence.

Secondly, we also identified a knowledge gap in some key parameters that should be addressed by funders of responsible agencies, who should include them in future research programs. This is because future economic evaluations will tend to have less variation in values of parameters used (e.g., the case of R₀), if good‐quality evidence is available and utilized.

Finally, it may not be possible to fill some evidence gaps (i.e., pandemic probability and duration) by research and these will need to be addressed by reaching consensus among experts to ensure a better consistency in the use of these assumptions, so that future economic evaluations can be comparable and meaningful for guiding resource allocation decisions.

Funding

The Health Intervention and Technology Assessment Program is funded by the Thailand Research Fund under the Senior Research Scholar on Health Technology Assessment (RTA5580010), the Thai Health Promotion Foundation, the Health System Research Institute and the Bureau of Health Policy and Strategy, Ministry of Public Health. This study did not receive any specific funding.

Conflict of interest

The authors have declared that no competing interests exist.

Supporting information

Table S1. Hierarchies of data sources according to quality of evidence (adapted from Cooper et al.).

Table S2. Model parameters of infectivity and disease severity values with quality of evidence (QoE) ranking of reviewed studies.

Table S3 Model parameters of intervention effectiveness values with quality of evidence (QoE) ranking of reviewed studies.

Table S4 Model parameters of pandemic duration and probability values with quality of evidence (QoE) ranking of reviewed studies.

Praditsitthikorn et al (2013) Assessing key model parameters for economic evaluation of pandemic influenza interventions: the data source matters. Influenza and Other Respiratory Viruses 7(Suppl. 2), 59–63.