Evaluation of Reporting Quality of Systematic Reviews on Health Economic Evaluation Studies Based on the CHEERS Statement: An Overview of Reviews

Abstract

Background:

Economic evaluations in healthcare are designed to inform decisions by the estimation of cost and effect trade-off of two or more interventions. We aimed to evaluate the standards of systematic reviews on health economic evaluation studies using the CHEERS (Consolidated Health Economic Evaluation Report Standards) tool.

Methods:

We searched the PubMed database with keywords CHEERS and its complete form in combination with keywords related to cost or economic evaluation without language and time limits until November 17, 2021. The CHEERS tool was then used to include systematic reviews.

Results:

Overall, 32 systematic reviews, included 610 primary studies were included. Of the 32 included studies, only 1 study (3.1%) had poor quality, 5 studies (15.6%) had good quality, remaining studies had very good and excellent quality.

Conclusion:

Some studies still have problems in expressing the standards. The necessity of standards for reporting economic evaluation studies in the field of health is very serious, and Cheers is one of the most important tools.

Introduction

Due to the lack of required healthcare resources, cost-effective choices are necessary. Decision makers use economic evaluations to effectively allocate resources so that they choose the best service or intervention by identifying, measuring, valuing, and comparing the cost and results of various services (1). Therefore, access to sufficient, accurate, and reliable information is essential for decision-making that can lead to economic and health effectiveness (2). The use of economic evaluation research is increasing, but methodological errors and their usefulness for healthcare decisions affect their validity (1). Therefore, qualitative evaluation of economic evaluation studies is very necessary before applying the results.

Systematic reviews of health economic evaluation studies can provide policymakers, physicians, patients, and other decision-makers with useful information for counseling and decision-making. They can identify the scope and quality of existing studies, conditions promoting the effectiveness and efficiency of the intervention under evaluation, and understand the impact of key parameters on the general outcome (3).

One of the tools used for qualitative review of economic evaluation studies is Consolidated Health Economic Evaluation Reporting Standards (CHEERS). In 2013, CHEERS statement (which was recommended as minimum information that is needed for reporting economic evaluations) was released and in this way, the previous guidelines for health economic evaluation were made available to researchers as a single reporting guideline (4).

Therefore, we aimed to summarize the standards of the conducted systematic review studies on economic evaluation of various diseases.

Methods

This study was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guideline.

Criteria for considering studies for this review

This systematic literature review was conducted to identify other systematic review studies that have been critically appraised by researchers using CHEERS tools in the PubMed database. We included all studies that were free to access and the quality of the study was assessed with the mentioned tool without any restriction in language or time of publication. Studies in which 23 instrument items were not reviewed for each study, and studies in which wrong references were given to some of the reviewed studies, which caused the year of the study to be unclear, were excluded from the study.

Search strategy

Based on the inclusion criteria, we searched the PubMed database with keywords CHEERS and its open form i.e., Consolidated health economic evaluation reporting standard in combination with keywords related to cost or economic evaluation with this search strategy: ((Consolidated Health Economic Evaluation Reporting Standards) OR (CHEERS) AND (ffrft[Filter])) AND ((Costs and Cost Analysis[Mesh Terms]) OR (cost) AND (ffrft[Filter])) Filters: Free full text, without language and time limits until November 17, 2021. All studies imported into EndNote ×20.

Selection of studies

As shown in Fig. 1 (the PRISMA Flow diagram) which indicates the process of identifying, reviewing, and selecting articles, at first, 186 studies were obtained. All duplicate records (n= 82) and 102 records remained for further review. We reviewed the titles and abstracts of remaining review articles. Finally, 32 studies met the inclusion criteria for this review study.

Fig. 1:

The PRISMA flow diagram illustrating article selection and elimination

Quality assessment

Methodological quality of reviews was assessed using quality criteria adapted from De Vet, De Ridder and De Wit (5) and based on the Quality Assessment Tool for Reviews (6). This criterion is an 8-point criterion (0–8). If every criterion was met by the review, it received a score of one and otherwise a score of zero. According to the agreement between the research team, if the total score of the criteria for each review was less than 6, it was excluded from the study. Therefore, 2 other studies were excluded from the reviews and 32 studies leaved. These 32 review studies included 636 primary studies, 26 duplicated primary studies were excluded, and finally 610 studies were included.

Data extraction and analysis strategy

Data extraction included specific details about the number of included studies, the average quality score of the included studies, the quality of the included studies (based on the average quality score), the period of the included studies, the country/region where the primary studies were conducted, and the scope of the study (Table 1).

Table 1:

Number and details of the included studies

Ref.	Original studies (N)	Average cheers score of included studies (%)	Quality status of the included studies	The time frame of the included studies	Country/region	Scope of study	Quality score
Ma, et al (7) 2016	32	77.3	Very good	2003–2014	China	Pharmacoeconomic	8
Zakiyah, et al (8) 2016	9	73.5	Very good	2006–2013	Low- and Middle-Income Countries	Family Planning Interventions	8
Banke-Thomas, et al (9) 2017	5	90.5	Excellent	2002–2011	Nigeria, Indonesia, Kenya and Tanzania, Mozambique and Zambia	emergency obstetric care training	8
Gillespie, et al (10) 2017	5	68.3	Good	2000–2016	Great Britain, Spain, Japan and two studies from Australia	intraoperative interventions to prevent surgical-site infection	7
Hope, et al (11) 2017	14	86.7	Excellent	2005–2014	All of the world	population-based sodium reduction interventions	8
Ibrahim, et al (12) 2017	5	91.7	Excellent	2009–2017	Western countries	Antimicrobial Stewardship Programs	8
Iribarren, et al (13) 2017	30	78.6	Very good	2005–2016	19 countries, most of which were conducted in upper and upper-middle income countries	MHealth	8
Melendez-Torres, et al (14) 2017	19	81.6	Very good	2005–2016	a lot of countries	Drugs	8
Velentzis, et al (15) 2017	5	58.3	Good	2005–2009	2 USA, UK, Finland, Sweden	Menopausal hormone therapy	7
Wong, et al (16) 2017	9	89.8	Excellent	2001–2014	Hong Kong	vaccination programs	8
Dritsaki, et al (17) 2018	4	98.2	Excellent	2011–2015	2USA, UK, CANADA	managing Dupuytren’s disease	8
Grochtdreis, et al (18) 2018	15	78.1	Very good	2004–2018	UK, US, Austria, Australia, Belgium, Canada, Switzerland, Germany, France, Italy, the Netherlands, New Zealand, Portugal, Sweden	castration-resistant prostate cancer	7
Jiang, et al (19) 2019	14	79.2	Very good	2011–2018	Australia, UK, Spain, USA,	Digital Health Interventions on the Management of Cardiovascular Diseases	8
Ling, et al (20) 2019	15	58.1	Good	2007–2017	Uganda, Kenya, Afghanistan, Ethiopia, Tanzania, Senegal, Congo, Brazil, Ghana, Sub-Saharan, Nigeria	malaria rapid diagnostic tests	6
Mendivil, et al (21) 2019	33	76.5	Very good	2012–2018	Most studies were conducted in Europe (36,4%), followed by United States (24,2%) and Asia (24,2%)	Screening strategies for early detection of colorectal cancer	7
Sultana, et al (22) 2019	19	61.3	Good	1998–2017	Germany, USA, France, Malaysia, Scotland, Multicountry (mainly North America), Multicountry (17 countries in Europe, Latin America and South Africa), Belgium, Hong Kong, UK, Spain, Malawi, Netherlands	community acquired pneumonia management strategies	6
Anopa, et al (23) 2020	16	79.5	Very good	1986–2017	United States, United Kingdom, Canada, Sweden, Australia, USSR, Chile, Finland, Taiwan, Uzbekistan	Primary prevention of tooth decay in preschool childrenaged 2 to 5 years	8
dela Perrelle, et al (24) 2020	8	74.3	Very good	1999–2017	3The Netherlands,4 United States of America, Niger	Quality Improvement Collaboratives in healthcare	8
Hao, et al (25) 2020	21	80.6	Very good	2011–2018	India, South Africa, Uganda, Russia, Eastern Europe, Uganda, USA, Tanzania, Hongkong, Malawi, Brail, Mozambique, Ethiopia	Xpert in detecting Mycobacterium tuberculosis	7
Niyomsri, et al (26) 2020	14	90.2	Excellent	2002–2017	Sweden, UK, US, The Netherlands, Canada,	Spinal Cord Stimulation	8
Ten Ham, et al (27) 2020	12	82.7	Very good	2013–2019	US, France,	Gene Therapies and Their Application	8
Woods, et al (28) 2020	19	78.5	Very good	1997–2019	Peru, Australia, Canada, US, UK, Germany, Netherlands, Sweden, Colombia, Slovakia, China	diabetic foot ulcer infections	8
Qiu, et al (29) 2021	13	85.7	Excellent	2007–2019	Belgium, USA, UK, Hong Kong, Southeast Asia, Singapore, Germany, Japan, Italy	Aprepitant in Preventing Chemotherapy-Induced Nausea and Vomiting	8
Rezapour, et al (30) 2021	26	76	Very good	2020	A lot of countries (such as؛ China, Israel, us, Uganda …)	programs against COVID-19	8
Schwander, et al (31) 2021	4	81.2	Very good	2007–2014	UK	Health Economic Obesity Models	8
Stawowczyk, et al (32) 2018	15	88.1	Excellent	2008–2017	Canada, the UK and Poland were mostly performed	Biological drugs compared to (conventional, surgery, drugs, etc.) in Ulcerative Colitis	8
sanyal, et al (33) 2019	20	84.1	Very good	2007–2018	UK, Spain, France, Netherlands, Belgium, Italy, Canada, USA, Brazil	Community-based services by pharmacists	7
Avanceña, et al (34) 2021	54	90.8	Excellent	2010–2019	From all continent	Health Interventions	8
El Alili, et al (35) 2017	45	43	low	2000–2017	All Countries	Obstetrics and Gynecology	6
Ding, et al (36) 2020	22	96.2	Excellent	2010–2019	countries all over the world, with ten from the USA, seven studies from China (one from Hong Kong), two studies from Canada, one each from Australia, France, Switzerland	immune checkpoint inhibitors for treatment of non-small cell lung cancer	8
Werner, et al (37) 2020	39	63.2	Good	1983–2019	low and middle-income countries	emergency care interventions	7
Galekop, et al (38) 2021	49	76.3	Very good	1987–2017	Sweden, Switzerland, Australia, UK, USA, Canada	Interventions with a personalized nutrition item in adults	7

Given that each of the 32 studies encompassed multiple primary investigations, a total of 610 studies were meticulously examined by two independent researchers, relying on the evaluations assigned by the original study authors. The objective was to extract the conclusions and synthesize the overarching findings of the studies into a set of 24 items, which were subsequently entered into Excel-2013 software. A score of 1 was allocated to studies that fully adhered to the established criteria, while a score of 0.5 was assigned to studies that partially complied with the standards, and no score was designated for studies that failed to meet the criteria. In instances where the application or non-application of the standard was ambiguous, such studies were categorized as non-applicable, thus excluding the respective item from scoring, with the score for the corresponding item redistributed among the remaining items.

The included primary studies were divided in to 4 categories based on the quality score. So that the quality score above 85 was classified as excellent, 70–85 as very good, 55–70 as good, and below 55 as poor (11, 22). Therefore, more than 70% of the studies were in very good or excellent quality category, and only 15% of the primary studies were in low-quality category.

Results

Description of the included studies

As shown in Table 1, of the 32 included studies, only 1 study (3.1%) was poor quality (35), 5 studies (15.6%) were good quality (15) (20) (22) (10, 37), and remaining studies had very good and excellent quality.

All 32 secondary studies were published after 2013, include 610 primary economic evaluation studies that were conducted during 1986–2020. 33.2% of the studies had an excellent quality score (score from 85–100), 35.4% of studies had a very good quality (score from 70–85), 17.5% of studies have a good quality (score from 55–70) and only 14% of the studies had obtained a score below 55, that is, weak (the score obtained by adjusting the non-applicable items).

As shown in Fig. 2, the item 4 has the highest score in the times of 1983–2010. After that, in the period of 2015–2011, the item 3 with 92% and in the period of 2016–2020, the item 22 with 92.8% had the highest percentage among other items. In the period of 2000–1883, item 12 ‘Measurement and valuation of preference-based outcomes’ (0%), in the period of 2001–2005, item 24 ‘Conflicts of interest’ (37.2%), and from 2006 to 2020, item 21 ‘Characterizing heterogeneity’ had the lowest percentage. In general, percentages have improved in all items despite some fluctuations over time.

Fig 2:

Average score of 24 items and their trend

According to Fig. 3, a comprehensive examination reveals that items 21 ‘Characterizing heterogeneity’, 9 ‘Discount rate’, 15 ‘Choice of model’, and 18 ‘Study parameters’ exhibited the lowest quality evaluations, with mean scores of 43.6%, 62%, 63.2%, and 64.5%, respectively. In contrast, items 3 ‘Background and objectives’, 7 ‘Comparators’, 22 ‘Study findings, limitations, generalizability, and current knowledge’, 4 ‘Target population and subgroups’, and 13 ‘Estimating resources and costs’ achieved the highest quality evaluations, with average scores of 83.91%, 85.2%, 83.9%, 82.6%, and 80.8%, respectively. The overall mean score for the 24 items is calculated to be 72.7%, whereas the average score for the CHEERS framework across all studies is determined to be 76.6%. This observed discrepancy could be attributed to the exclusion of non-applicable items. In addition, items 2 ‘Abstract’ (79.3%), 7 ‘Comparators’ (77.8%), 9 ‘Discount rate’ (77.7%), 16 ‘Assumptions’ (78.2%), 17 ‘Analytical methods’ (73.3%), and 18 ‘Study parameters’ (76.7%) were in range 70–80 %, mean while other items were more than 80%.

Fig. 3:

Quality score of all items

Furthermore, each item was assigned a score on a 5-point scale, where complete adherence to the item is denoted as YES with a corresponding score of 1(1), partial adherence as PA with a score of 0.5, non- adherence as NO with a score of 0, non-applicable instances are marked as NA, and items that were not evaluated or were overlooked by the secondary researchers are represented as M (missing) (4, 11, 22, 39). In instances where items are deemed non-applicable and remain unexamined by secondary researchers, the score attributed to such items was proportionately allocated among the remaining items.

As shown in Fig. 4, the most frequent items that were not reported in the articles implied item 21 ‘Characterizing heterogeneity’ (39% no compliant), item 9 ‘Discount rate’ (26% no compliant), and item 24 ‘Conflicts of interest’ (31% no compliant) and item 20 ‘Characterizing uncertainty’. The most common missing items were item 15, ‘Choice of model’, item 16 ’Assumptions’, and item 18, ‘Study parameters’ with almost 0/06 %.

Fig. 4:

CHEERS Items met by the included studies

According to the findings, adherence with all cheers items by researchers has increased in studies since 1983. In addition, this improvement is more visible in items such as 6 ‘Study perspective’, 12 ‘Measurement and valuation of preference-based outcomes’, 18 ‘Study parameters’, and 24 ‘Conflicts of interest’.

Discussion

The aim of this study was to review systematically standards used in health economic evaluation studies with the CHEERS tool. Overall, 32 review studies met the inclusion criteria which included 610 primary studies. More than 70% of the studies had very good and excellent quality (CHEERS score above 70) and only 14.6% of the studies had low quality (CHEERS score less than 55). In Miroshnychenko et al., (40) which reviewed studies conducted in 2012–2019 the average CHEERS checklist adherence score was 63%. In addition, in Rezapour et al., which reviewed studies in 2019–2020, the CHEERS scores for more than 65% of studies most studies were good and excellent quality (30). In Nguyen et al., study, of all the included articles that were conducted in the period of 2016–1996, the CHEERS score for more than 58.8% of studies was good and excellent (2). Meanwhile, the score of item 21’Characterizing heterogeneity’ (50%) was the most poorly reported items on the CHEERS checklist and has been in the low range compared to other items in all time periods, the causality can be expressed as the assessment of sources of heterogeneity was not conducted in 40% of included economic evaluations. About item 24 ‘Conflicts of interest’, because in 24% of studies this item was not reported, quality score of that was in low range. Item 12 ‘Measurement and valuation of preference-based outcomes’ in 28% was NA (non- applicable) so its score was low than other items.

To our knowledge, this study is the first systematic review study that all the studies that were reviewed were secondary studies and no other study that reviewed secondary studies in this way was found in other databases. Although one of the constraints of this research may be attributed to the multitude of scholars who have examined the CHEERS framework across various publications, resulting in potentially divergent interpretations among different evaluators, it appears that this limitation is somewhat mitigated by the substantial sample size and the consistency observed in the item scores across the studies conducted by distinct researchers. Furthermore, it is noteworthy that the literature search was exclusively performed within the PubMed database, leading to a considerable number of articles being scrutinized; however, it is conceivable that certain relevant studies may have been overlooked, and additional researchers could expand their inquiry by accessing other databases such as Web of Science and Scopus.

It seems imperative that, similar to the investigations conducted on other methodological frameworks such as STROBE, which have been undertaken by domestic researchers, a collaborative effort involving multiple scholars should be pursued to independently examine the preliminary economic evaluation studies conducted over various years employing this particular tool; nonetheless, given the intricate nature of economic evaluations and the nascent introduction of this tool in Iran, the process of identifying several researchers proficient in its application may require an extended duration.

It is recommended that future studies involve at least two independent researchers who would delineate economic evaluation studies within a specified timeframe, geographic region (for instance, Iran), a particular category of technology (such as pharmaceutical economic evaluations), or a distinct type of economic evaluation (such as solely cost-utility analyses), and subsequently assess these investigations. The findings generated by a third researcher should then be reviewed, particularly in instances where discrepancies in evaluative scores arise between the two evaluators, necessitating a comprehensive discussion among all three researchers to reach a consensus, which would ultimately be documented as the definitive evaluation outcome.

Conclusion

Most of the reviewed studies have obtained very good grades from the reviewed standards, but some studies still have problems in expressing the standards. It seems that the necessity of standards for reporting economic evaluation studies in the field of health is very serious, and considering that CHEERS is one of the most important tools, it is necessary that the reports submitted to the Health Technology Assessment Office and other centers such as National Research Institute and Food and Drug Organization should be considered. In addition to the mentioned cases, the use of this tool is very useful for researchers who work in the field of economic evaluation to provide a standard report. Therefore, it seems necessary to hold workshops and journal clubs to introduce this tool to internal journal reviewers and researchers in this field.

Journalism Ethics considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.