Economic Evaluation of Health Interventions: A Critical Review

Abstract

Background:

Economic evaluation is used for the optimal allocation of resources in the health sector. While a large number of economic evaluation studies have been conducted, there is less critical review of these studies. We critically examined the economic evaluation studies of preventive health interventions.

Methods:

The study was carried out using critical review method. Seven databases (i.e., PubMed, Cochrane Database of Systematic Reviews, Web of Science, Science Direct, Scopus, Springer Link, and Elsevier) were searched to find articles on economic evaluation of health interventions published from 1985 to 2018. In addition, the references of retrieved studies were hand screened for articles that were not indexed in these databases. Finally, 206 articles, including 33 cost- benefit analysis, 146 cost- effectiveness analysis, and 27 cost-utility analysis were included in this study. These studies were critically evaluated using a checklist of 11 criteria.

Results:

Only 20% of the studies met all the methodological criteria of health economic evaluation. The cost perspective, costs type, cost data source, and cost measurement were not explained and discussed in 17%, 20%, 5%, and 33% of studies respectively. Outcome data sources and outcome valuation method were only mentioned in 53% and 69% of studies. The sensitivity analysis and results’ generalizability were not reported in 16% and 46% of studies.

Conclusion:

The quality of economic evaluation studies is low, and it can be misleading if resource allocation decisions are made using this evidence. Authors should use valid protocols to conduct and report economic evaluation studies, and journals’ editors should use valid checklists to evaluate these articles.

Introduction

Health intervention is defined as the use of a health program, action, or technology to reduce or eliminate a health problem (1). The ultimate goal of health interventions is to improve the health and quality of life of people and patients (2). However, the environment in which healthcare organizations operate is dynamic, complex, complicated, and challenging. As a result, health care organizations face many challenges in terms of effectiveness, efficiency, quality, and equity (3). Increasing demand for effective and quality health services, rising costs, and lack of resources have posed many challenges for health policymakers and managers (4). Hence, healthcare services should be delivered efficiently to make sure that all people have access to these services. However, the available resources are insufficient to satisfy all human needs.

The economic evaluation aims to identify, measure, value, and compare the costs and consequences of several alternative programs or interventions. Economic evaluation is remarkably used in priority setting and allocating scarce healthcare resources, where, policymakers are obliged to choose from alternative programs with different effects due to limited resources (5). Economic evaluation helps health policymakers implement more effective health interventions.

Economic evaluation studies are divided into two general categories: Cost-Benefit Analysis (CAB) and Cost-Effectiveness Analysis (CEA). In economic evaluation, it is better to do a cost-benefit analysis of health interventions first. Cost-benefit analysis compares the benefits of an intervention with the costs incurred. The monetary value of the costs and consequences of the health intervention is calculated, and as a result, a more valuable intervention will be chosen. A healthcare intervention with a benefit-to-cost ratio of more than one is recommended.

Cost-effectiveness analysis can be used for those low-cost-benefit interventions, if they promote people’s health and their quality of life. More effective interventions with low cost are suggested. The CEA can be used if it is not possible to monetize the results of the intervention. It shows the ratio of the cost of a health intervention to the health outcomes and determines the value of the intervention. The benefits of healthcare interventions in CEA are measured on a clinical basis. If the main goal of a health intervention is to increase people's life expectancy, it is better to use CEA.

In contrast, if the health intervention affects a patient’s quality of life, it is better to use cost-utility analysis (CUA), which takes into account the quality and quantity of health outcomes. The Quality Adjusted Life (QALYs) index is used in CUA to measure health outcomes, and the parameter obtained is called the Incremental Cost-Utility Ratio (ICUR). Finally, the ICUR is compared with the consumers’ willingness to pay service (threshold) to make the final decision on whether or not to approve the cost-effectiveness of a health intervention. In other words, by setting the threshold of cost-effectiveness, the monetary value of a QALY is calculated. Interventions whose cost per QALY (ICER) is below this threshold are cost-effective and should be funded (6).

Economic evaluations help to enhance the health system efficiency by prioritizing, rationing, and allocating resources optimally, as well as increasing its effectiveness by improving access and equity. Many economic evaluation studies have been conducted in recent years. However, some studies suffer from limitations. Some of these studies did not use appropriate analytical methods (7, 8). Using an inappropriate perspective and evaluation technique, exclusion of some important costs, lack of transparency of data sources; and ignoring sensitivity analysis of results are some of the limitations of economic evaluation studies (8–11). As a result, the findings of these studies are biased and misleading. Therefore, we aimed to evaluate critically the economic evaluation studies of health interventions.

Methods

This research was conducted using the critical review method. The purpose of a critical review study is not just to describe and summarize previous studies. A good critical review study should be analytical, critical, and prescriptive. Critical review, unlike narrative review, expresses the strengths and weaknesses of studies conducted in a specific field of knowledge. As a result, the areas of need for further research are identified. A critical review research does not only provide a summary of studies conducted in a particular field of knowledge, but also aims to review the most important and relevant studies conducted in that field of knowledge and critically discuss them. Critical review by critically evaluating the methodology and findings of research conducted in a particular field of knowledge, provides valuable information to knowledge users, and provides the necessary research evidence for decision making (12).

A critical review provides an opportunity to evaluate and describe differences in studies’ methodology, assumptions, and data collection and analysis methods, and to identify key factors that make a difference in conclusions. In fact, critical review studies are an integral part of distinguishing poor and low-quality studies from robust and high quality studies. Therefore, researchers can integrate the results of valid economic evaluations of health interventions to inform evidence-based policymaking.

In this study, the methods, techniques, and content of cost-effectiveness, cost-benefit, and cost-utility analyses of published health interventions were critically evaluated. Items such as study details (authors name, article title, journal name, year of publication, country of study), economic evaluation perspective, methods for calculating costs and results of health interventions, tools used to measure the effectiveness of health intervention, models, and methods used for cost-effectiveness analysis, cost discounting and intervention results, epidemiological data, study sensitivity assessment, and findings validation were reviewed in this critical review.

Seven databases including PubMed, Cochrane Database of Systematic Reviews, Web of Science, Science Direct, Scopus, Springer Link, and Elsevier were searched to find articles on economic evaluation of health interventions published between 1985 and 2018. In addition, the references of retrieved studies were hand screened for articles that were not indexed in databases. The keywords used for the literature search were MeSH terms and other words such as economic evaluation, cost-benefit analysis, cost-utility analysis, cost-effectiveness analysis, and health.

Searching in PubMed and Cochrane Database of Systematic Review found 3365 and 5324 articles respectively. In addition, 1325 articles were identified in Web of Science, Elsevier, and Science Direct databases. Moreover, 81 and 1500 articles were found by searching the database of Science direct and Scopus respectively. Finally, 250 articles were identified from the references of the retrieved articles.

All retrieved articles were screened by title (first step), abstract (second step), and the whole-text (third step). First, from 11845 retrieved articles, 452 articles were selected after removing treatment interventions’ economic evaluations, unrelated and repetitive articles. Then, two researchers assessed the quality of retrieved articles using a valid checklist and examining research sample, research method, data collection, and analysis methods and tools (13). Disagreements between researchers were resolved through mutual discussion, and the third researcher was asked to intervene if it was necessary. Finally, 206 articles selected for analysis. The summary of the search strategy is provided in Fig. 1.

Fig. 1:

Flowchart of screening included articles

The critical assessment of retrieved articles was performed using a checklist developed by the literature review (8, 14,15). The checklist had general and technical questions. General questions included the publication year, geographic region (the country), the disease type, and the type of study’s analysis. Technical questions included 5 questions about cost (i.e., cost perspective, cost type, cost data source, cost measurement and cost valuation method), 3 questions about outcomes (outcome type, outcomes data source, and outcome valuation methods), and three more items including summary measures, sensitivity analysis and generalizability of results.

Results

Overall, 206 articles, including 33 cost-benefit analysis, 146 cost-effectiveness analysis, and 27 cost-utility analysis were included in this study. The first article was published in 1985. Over the past decades, a large number of economic evaluation articles on health interventions have been published. The general characteristics of the articles are shown in Fig. 2.

Fig. 2:

Number of economic evaluation studies of preventive health interventions

The studies were categorized into six groups of health interventions, which include disease vaccination, educational interventions, nutritional interventions, sexually transmitted disease prevention interventions, obstetric and gynecological disease interventions, and insect-borne diseases interventions. Overall 59, 50, 62, 22, and 13 studies were performed in Asia, America, Europe, Africa, and Oceania, respectively.

Cost perspective

The study perspective should be clearly specified in the economic evaluation of health care interventions. Overall, 172 authors had clearly expressed the perspective of their studies. Most studies were conducted from a societal perspective. Other perspectives included health care provider, patients, government, employers, and payers.

Costs type

Costs include direct, indirect, and intangible costs. Direct medical costs incurred by the health care provider and patient for delivery or receiving healthcare services. Direct non-medical costs include travel, accommodation, and other expenses related to receiving healthcare services. Indirect costs include lost productivity while receiving health care, disability, or premature death. Finally, intangible costs include pain and suffering. Overall, 88 articles identified direct and indirect costs. Seventy-seven studies calculated only direct costs. However, sometimes certain aspects, such as the existence of donated resources such as medications, and vaccines, supplies and equipment, and volunteer time was not thoroughly considered. Few studies have considered intangible costs. In some studies, costs have been evaluated based on local currency such as Canadian dollar, Korean won, Finnish markka, Italian Euro, German Euro, and New Zealand dollar.

Cost data source

There are several ways to collect cost data. The majority of studies used multiple sources. The sources of cost data was not explained in 10 studies. Cost data were collected through published sources in 65 studies. The remaining articles had collected data from primary sources. They used a variety of techniques for collecting data such as interviews with staff and patients and using questionnaires (n=21), reviewing documents and reports (n=71), and epidemiological studies (n=46). In eight studies, the authors have carried out an intervention and used outcome data generated in comparison with the control group. In fact, these studies have used actual cost data.

Cost measurement

Very few studies have divided costs into operational costs and overhead costs. In many cases, the components of cost such as capital costs (e. g. building and equipment) and operating costs (food, transportation, medical supplies, and so on) were clearly described. Overall, 138 studies provided details of the resources used in health interventions in physical units. About 68 papers did not present cost details (Fig. 3).

Fig. 3:

Measurement of costs

Cost valuation

The US dollar (n= 99) and the international dollar (n=6) were the most common currencies used for cost valuation. Other studies used local currencies. Exchange rates in developing countries often do not reveal the real economic costs and benefits of importing and exporting, so in the presence of floating exchange rates, a shadow exchange rate is needed. Only a few studies evidently illustrated this issue. Ignoring the role of these factors does not consider the opportunity cost of goods or services. Finally, a large number of studies had converted the currencies by using purchasing power parity (PPP).

Outcome type

Many studies had used natural outcomes such as number of prevented diseases (n=94), number of prevented deaths (n=42), and saved monetary units (n=82), and QALY (n=55). The prevented DALYs, increase in quality of healthy life, and other intermediate outcomes were also more common. In most studies, a combination of outcomes was used.

Outcomes data source

The sources of outcome data in 52 studies were based on intervention and observational studies, using modeling methods and randomized clinical trials. Forty-four studies used secondary data sources including data from other published studies that have been modeled for generating outcome data. Sources of outcome data were unclear in 110 studies (Fig. 4).

Fig. 4:

Sources of outcome data in economic evaluation of health interventions studies

Outcome valuation

The most common values assigned to outcomes were monetary values of the US dollar (n=22) and the international dollar (n=13) and QALY (n=58), and prevented DALYs (n=13). Some studies used a combination of monetary and non-monetary values for the consequences. Sixty-four papers did not present details for outcomevaluation.

Summary measures

The most frequent summary measure used for CEA and CUA was the Incremental cost-effectiveness Ratio (n= 47). Other summery measures were Cost-Effectiveness Ratio, Cost-Utility Ratio, Cost-Benefit Ratio, Life Years gained, Cost Per Day and Net Benefit (Fig. 5).

Fig. 5:

Summary measures

Sensitivity analysis

About 173 studies performed sensitivity analysis. Univariate sensitivity analysis (one-way) was the most commonly used technique (n=89). Three papers used threshold analysis. Thirty-three studies had mentioned doing sensitivity analysis without giving explanations about its type.

Generalizability

Ninety-four studies did not report generalizability at all. Overll, 112 studies have cited the possibility of generalizing their own results to other environments or countries (Fig. 6).

Fig. 6:

The generalizability of the results of health interventions studies

Adherence to technical criteria

Adherence to technical criteria is a measure of overall quality and as a necessity for providing high internal validity of an article. Forty-one studies had entirely adhered to technical criteria, and 5 studies adhered to 7 criteria out of 11 (Fig. 7).

Fig. 7:

Adherence of articles to technical criteria of economic evaluation

Discussion

The health system policymakers and managers have paid attention to efficiency and cost management due to the economic recession and lack of financial resources (16). As a result, economic evaluation of health care interventions is of great interest to them. However, not all economic evaluation studies are of high quality and validity. The aim of this study was to critically evaluate the economic evaluation studies of health interventions. The number of economic evaluation studies of health interventions has increased significantly in the last two decades. The CEA was the most frequently published economic evaluation studies.

When conducting economic evaluation studies of health interventions, the transparency and critical evaluation of findings should be increased by special attention to important issues such as the cost perspective, costs type, cost measurement and valuation, source of cost data, outcome type, outcome data sources, sensitivity analysis, and generalizability of the results.

The perspective of the economic evaluation studies should be clearly identified. In this study, perspectives of about 83% of studies were clearly expressed. About 44% of studies were conducted using a societal perspective. In these studies, incured costs ranged from healthcare staff salaries, capital and operating costs, and patient out of pocket payment. Eighteen percent of studies were conducted using health providers’ perspective. The study perspective was not clear in a few articles. However, it can be understood reading the whole text. Some of the authors(8,14) reported the same findings.

The economic evaluation of an intervention and its alternative(s) should always determine the costs associated with the program. The range of these costs depends on the perspective chosen and can include direct medical costs incurred by the health care provider or patient, direct non-medical costs which include travel, accommodation, and other expenses. Indirect costs mainly involved loss of productivity of patients and intangible costs due to patient suffering. Furthermore, distinguishing between the types of costs is important. For example, it is important to distinguish between the costs of vaccination (vaccine accompanied by injection) from the cost of the vaccine (the vaccine only). Moreover, it is necessary to distinction between the main cost components such as transport, labor and consumables, cost of the vaccine, and administrative costs, etc. Administrative costs estimated in studies consist of the cost of nursing, healthcare professionals, warehousing, and general education. However, administrative costs were different between countries.

Several methods were used for measuring indirect costs such as loss in salaries and wages (human capital method) or productivity (friction cost method). In this review, almost 80% of the studies identified the type of cost. In some cases, all cost items were not identified, which underestimates the actual cost of the intervention. Typically, some studies did not estimate costs such as volunteer activities that are often used in developing countries. Ignoring calculation of some types of costs can be due to the researcher's weakness in measurement methods, which highlights the need for costing guidelines developed by experts (14).

Measuring and valuing of costs, which are more important and are technical issues in economic evaluation, were done in most studies. In fact, most researchers are familiar with costing methods. Ideally, the quantity of resources used along with price data should be provided to estimate the total costs. The more directly the cost calculation, the more transparent the results. In this study, only 67% of reviewed studies provided details of the resources used in physical units. Other studies did not provide details of the resources used in the health intervention. Therefore, this shortcoming reduces the quality and validity of the study. Developing and using cost measurement guidelines for health economic evaluations makes the research more standardized and the results comparable, and provides benefits to health policy makers and managers (17).

Some economic evaluation studies are suffering from a lack of transparency, which negatively affects the validity and quality of studies. The source of cost data has not been mentioned in 5% of studies. It was not clear whether the identified costs were also measured and valued, and the quantity and price data were combined. Additionally, comparators (alternative interventions) had not been explained clearly.

Most studies measured a combination of the outcomes of an intervention. Natural units such as the number of prevented diseases and deaths were the most common outcome units (66%). It can be because of the popularity of cost-effectiveness analysis studies, the availability of natural units of the effectiveness of health programs and their simple calculation. Other outcome measures were saved money (40%), QALY (27%), prevented DALYs, and increased quality of life. This finding is consistent with the results of other studies (8, 14).

Experimental and quasi-experimental studies were mainly used for the economic evaluation of health interventions. Primary data are available from epidemiologic studies, including experimental studies (randomized clinical trials), quasi-experimental (an interventional group against the control group or in research that does not allow the investigator to control or manipulate the variable or the effective variables), and observational studies. However, in about 47% of the articles, the data sources of the outcomes are not specified. Therefore, the sources of outcome data are not mentioned in many articles, and often a detailed description of the data collection process is not provided, which indicates a lack of transparency in many studies.

A sensitivity analysis is central to assess the validity and reliability of the assumptions and values of the input variables. Sensitivity analysis usually tests the highest, average, and lowest estimation in relation to affecting factors (such as cost, effectiveness, and discount rate) to highlight the robustness of the results. If these changes are little, the results may be valid and stable. If many changes to be made in the results, more caution is needed when interpreting the results. Therefore, it is necessary to perform sensitivity analysis in economic evaluation studies. In this review, about 84% of the studies performed different types of sensitivity analysis.

Generalizability of the results to other settings and countries is an important factor in increasing the study validity. Totally, 54% of studies have pointed out the generalizability of the results to other settings or countries and had discussed why their results can be generalized or not. A small number of articles have systematically addressed the generalizability of results. Mostly, studies tried to provide their assumption, data, and conclusion. They have not made effort to examine generalizing the obtained results to other contexts. If studies provide different assumptions and rely on them, then uncertainty about the results will be greater and there is no possibility of conducting sensitivity analysis for variations.

Overall, the quality of economic evaluation studies is low. Previous studies have also shown the low quality of published health economic evaluation studies (18, 19). Using guidelines may improve the quality of studies (20). Authors should use valid protocols to conduct and report economic evaluation studies, and journals’ editors should use valid checklists to evaluate these articles.

We studied all three types of economic evaluations including cost-benefit analysis, cost-effectiveness analysis, and cost-utility analysis. Therefore, due to the large scope of the study and time constraints, we examined only studies on economic evaluation of preventive health interventions. We reviewed only peer-reviewed articles that had been published in valid journals and gray literature were excluded (12). Although we tried to search all valid databases, it is possible that some journals or articles are not searched and excluded.

Conclusion

This study critically evaluated the economic evaluation studies of health interventions using 11 criteria. Adherence to technical criteria in health economic evaluation studies is weak and approximately 20% of the studies met all criteria of economic evaluation. Therefore, the quality of most studies is low. Health economic evaluation researchers should adhere to the economic evaluation guidelines to conduct and report economic evaluation studies, and journals’ editors should use valid checklists to evaluate these articles.

Journalism Ethics considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.