Quality of pharmacoeconomic research in China

Abstract

Background:

The number of pharmacoeconomic publications in the literature from China has risen rapidly, but the quality of pharmacoeconomic publications from China has not been analyzed.

Objectives:

This study aims to identify all recent pharmacoeconomic publications from China, to critically appraise the reporting quality, and to summarize the results.

Methods:

Four databases (PubMed, Web of Science, Medline, and EmBase) were searched for original articles published up to December 31, 2014. The Consolidated Health Economic Evaluation Reporting Standards statement including 24 items was used to assess the quality of reporting of these articles.

Results:

Of 1046 articles identified, 32 studies fulfilled the inclusion criteria. They were published in 23 different journals. Quality of reporting varied between studies, with an average score of 18.7 (SD = 4.33) out of 24 (range 9–23.5). There was an increasing trend of pharmacoeconomic publications and reporting quality over years from 2003 to 2014. According to the Consolidated Health Economic Evaluation Reporting Standards, the reporting quality for the items including “title,” “comparators of method,” and “measurement of effectiveness” are quite low, with less than 50% of studies fully satisfying these reporting standards. In contrast, reporting was good for the items including “introduction,” “study perspective,” “choice of health outcomes,” “study parameters,” “characterizing heterogeneity,” and “discussion,” with more than 75% of the articles satisfying these reporting criteria. The remaining items fell in between these 2 extremes, with 50% to 75% of studies satisfying these criteria.

Conclusion:

Our study suggests the need for improvement in a number of reporting criteria. But the criteria for which reporting quality was low seem to be limitations that would be straightforward to correct in future studies.

1. Introduction

Pharmacoeconomics refers to the scientific discipline that compares the value of 1 pharmaceutical drug or drug therapy to another.^[1,2] The number of pharmacoeconomic publications in the literature from China has risen rapidly.^[3] Ideally, these studies should follow evaluation criteria commonly accepted by researchers in this field.^[4,5] This is critical for helping to ensure the quality and reliability of the research. Several studies have examined trends in pharmacoeconomic publications from China in terms of publication numbers, journal placement, and the authors and universities involved in the research.^[6] To our knowledge, however, no study has yet analyzed the “quality” of pharmacoeconomic publications from China.

Evaluating the quality of pharmacoeconomic research is challenging. These studies are complex, often including several treatments or interventions, and may involve detailed clinical pathways. Various cost and effectiveness measures must be obtained as well. A variety of issues, including the perspective of the study, economic modeling and assumptions, addressing uncertainty, and so on, present significant challenges to editors and reviewers in judging the quality of a pharmacoeconomic study. As a result, it is perhaps not surprising that the quality of pharmacoeconomic publications in the literature varies substantially.^[7]

Despite its complexity, pharmacoeconomic research offers important information for health care and health policy decision makers to help allocate healthcare resources more efficiently. Until relatively recently, there was a paucity of pharmacoeconomic research to guide policy making in China, but the number of such studies has risen rapidly. China has implemented major healthcare system reforms since 2009, and pharmacoeconomic studies have become increasingly critical in light of these changes to inform policies aimed at controlling costs while enhancing quality of care. Thus, it is important to assess the quality of this growing literature in pharmacoeconomic research. The present study seeks to perform such an evaluation, to identify any weaknesses in existing research and to provide recommendations for ensuring the quality and integrity of future pharmacoeconomic research in China.

The present study will evaluate the quality of pharmacoeconomic papers published in English, but conducted in China. Whereas there are more pharmacoeconomic publications in Chinese than in English, English journals are on average more impactful. Moreover, English publications are becoming an increasingly important outlet for pharmacoeconomic research in China. Vaccines are excluded from the present study, because they have not been regarded as pharmaceutical products in China.

Guidelines for performing appropriate pharmacoeconomic studies have been well delineated. The present study employs a recently developed health economic quality criterion: Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement to evaluate the quality of pharmacoeconomic studies examined in this review.^[8]

2. Methods

2.1. Literature search

A systematic search of the literature was conducted in March 2015 using PubMed, Web of Science, Medline, and EmBase to identify pharmacoeconomic studies pertaining to China. We found a total of 1046 potential articles by this initial search. Search terms in all 4 databases included “Drug economic,” “pharmacoeconomic,” “Economics, Pharmaceutical,” “economics,” “pharmacy,” “pharmaceuticals,” “health economic,” “Medical Economics,” “cost,” “Cost Measures,” “cost-effectiveness analysis,” “cost-minimization analysis,” “cost-utility analysis,” “cost-benefit analysis,” “Benefits and Costs,”“Data,” “Cost-Benefit,” and “China.” These key words were used alone and indifferent combinations. The exclusion criteria were as follows: duplicated articles; not pharmacoeconomic studies; studies comparing multiple countries, not only about China; not full-journal articles, such as meeting abstracts, letters to the editor, treatment guidelines or recommendations, expert opinion, and narrative reviews. Studies comparing multiple countries were also excluded.

Two researchers carried out the literature search using the English-based search engines and identified articles independently. They assessed the abstracts of the identified studies, and all abstracts that met the inclusion criteria were confirmed by a third researcher. Full articles were then obtained for further evaluation.

We used NoteExpress V3.0 to review and evaluate the included studies. First, all the searched articles were compiled into NoteExpress. Second, 2 reviewers simultaneously screened the articles by titles and abstracts according to the exclusion criteria. Third, the full texts of the included articles were downloaded into NoteExpress. Finally, the full texts of the included articles were reviewed by 3 researchers.

Ethical approval was not necessary, because the present systematic review did not involve patients. The present article reviewed the previous publications about quality of pharmacoeconomic research in China, and all the materials were from the previous publications.

2.2. Evaluation of studies

There are a number of published criteria for evaluating pharmacoeconomic research. In the 1970s, Alan Williams at the University of York developed the first widely adopted health economic evaluation guidelines.^[9] In 1987, Michael Drummond suggested 10 health economic standards, which have been widely accepted by researchers in the field of health economics. In 1995, the British Medical Journal developed guidelines for pharmacoeconomic studies.^[10] Each of these evaluation standards or guidelines are extremely helpful for health economic research and their quality improvement. In 2003, a Quality of Health Economic Studies (QHES) instrument was designed to evaluate all 3 common types of health economic analyses: cost-minimization, cost-effectiveness, and cost-utility.^[11,12] The instrument emphasizes appropriate methods, valid and transparent results, and comprehensive reporting of results in each study. The present study uses “The CHEERS.

The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) introduced CHEERS statement, and it has been endorsed and published by the 10 publications.^[13] The ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force was approved by the ISPOR Board of Directors in 2009, and it aimed to develop guidance to improve the reporting of health economic evaluations. Task force membership was comprised of health economic journal editors and content experts from around the world. Forty-seven participants representing academic, biomedical journal editors, the pharmaceutical industry, government decision makers, and those in clinical practice were invited to the 2-round Delphi Panel. The task force submitted their first draft to the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force Review Group, and 24 reviewers submitted written comments. The report was revised and re-titled CHEERS in May 2012, and the revised CHEERS report was presented at the ISPOR 17th Annual International Meeting in Washington, DC.^[14]

The CHEERS statement attempts to consolidate and update previous efforts into a single useful reporting standard. The CHEERS statement is not intended to prescribe how economic evaluations should be conducted; rather, analysts should have the freedom to innovate or make their own methodological choices. Its objective is to ensure that these choices are clear to reviewers and readers. The present study uses CHEERS to evaluate the quality of pharmacoeconomic publications from China as it is the most recent and comprehensive guideline for this purpose.

The CHEERS is a 24-item scale covering 6 main categories: title and abstract, introduction, methods, results, discussion, and others. To estimate a summary reporting score, it is suggested to assign a value of 1 if the study fulfilled the requirement of reporting for that item completely, 0.5 for partially completing the requirement, and otherwise 0. Therefore, the maximum score for a publication that reports completely according to these standards is 24. At least 2 reviewers independently appraised the studies considered in this review. When results differed, they were discussed by all 4 researchers until any discrepancies were resolved.

2.3. Statistical analysis

In presenting our findings, we report the number of publications in each year, the country of the first author, and the publication journals. As noted above, the CHEERS is used to assess the quality of pharmacoeconomic publications from China up to December 2014.

3. Results

3.1. Study selection process

Figure 1 shows the flowchart for searching pharmacoeconomic publications from China. The initial database search identified 1046 articles. After screening by title and abstract, 178 full economic evaluations were identified. Of those, only 32 satisfied study inclusion criteria. Studies were mainly excluded because they were duplicated results (the same articles searched from different databases, n = 386), not pharmacoeconomic studies (n = 532), not about China (n = 39), or were themselves review articles (n = 57). It is quite common to make the initial search very broad to avoid the possibility of omitting any relevant studies, but the final number of studies satisfying inclusion criteria is typically much smaller than the initial search identified.^[6,15–16]

Figure 1.

Flowchart of search results and selection process.

3.2. Overview of included studies

A description of the 32 articles is presented in Table 1. These studies were published between 2003 and 2014. There have been more publications in recent years. In 2013, there were 8 publications, and in 2012 and 2014, there were 5 publications in each year. The first authors are mainly from China (28 of 32 publications, 87.5%). These studies appeared in a variety of journals. Nineteen (59.3%) papers were published in journals from the United States. The majority of publications were in medical journals (25 publications or 78.1%). The rest appeared in health economic journals or health services research journals (7 publications or 21.9%).

Table 1.

Description of articles (n = 32).

3.3. Results of the quality of reporting assessment

The results of the assessment of reporting quality per study are summarized in Table 2 and Table 3. The complete references of these 32 articles were reported in the reference list of 26 to 57. Substantial differences in the quality of reporting were observed among articles with an average score of 18.7 (SD = 4.33) out of 24 (range 9–23.5). Figure 2 shows the average score for pharmacoeconomic publications by year. Figure 3 shows each item of 24 reporting assessments by 3 categories: completely adequate, partially adequate, or inadequate.

Table 2.

Quality of pharmacoeconomic publications (articles 1–16).

Table 3.

Quality of pharmacoeconomic publications (articles 17–32).

Figure 2.

Average score of the publications by year.

Figure 3.

Quality of publications per items of the CHEERS checklist. CHEERS = Consolidated Health Economic Evaluation Reporting Standards.

Several observations are worth noting in above results. The reporting quality for item 1, and item 7 and item 11 are quite low—less than 50% of studies fully satisfied these criteria. We discuss each of these items in turn.

Item 1 is to identify the study as an economic evaluation, or use more specific terms such as “cost-effectiveness analysis” and describe the interventions compared. Only 15 publications specifically identify their study as economic evaluations and the drugs compared in their studies. Another 15 publications are partially adequate, and 2 publications do not report the type of economic evaluation. In addition, the titles of some articles may not correctly describe the types of studies.

Item 7 is to describe the interventions or strategies being compared and state why they were chosen. Thirteen publications fully satisfied this item, 18 publications partially satisfied it, and 1 study failed to satisfy it. Most publications that did not fully satisfy this item did not specify why the alternative drugs should be compared in the studies.

Item 11a is to describe fully the design features of the single effectiveness study and why the single study was a sufficient source of clinical effectiveness data. There are only 5 publications relevant to this item: 2 publications fully satisfied this criterion and 3 partially satisfied it.

Item 11b is to describe fully the methods used for identification of included studies and syntheses of clinical effectiveness data. There are 27 publications relevant to this item. Twelve publications fully met this criterion, 13 publications partially satisfied it, and 2 publications failed to satisfy it.

In contrast, reporting was relatively complete for items 3and item 22, with at least 28 articles satisfying these criteria. More than 75% publications fully satisfied the following items: item 3, item 6, item 10, item 18, item 21, and item 22.

Item 3 is to provide explicit statements of the broader context for the study. Thirty publications not only satisfied this criterion but also presented the study question and its relevance for health policy or practice decisions. Only 2 publications were scored as partially satisfying this criterion because they did not present the relevance of the study for health policy or practice decisions.

In item 22, 28 out of 32 publications summarized key study findings and described how they support the conclusions reached. They discussed limitations and the generalizability of the findings and how the findings fit with current knowledge. Four publications did not discuss generalizability of the findings or how the findings fit with current knowledge.

In item 6: 26 out of 32 publications described the perspective of the study and relate this to the costs being evaluated.

In item 10, 25 publications described what outcomes were used as the measures of benefit in the economic evaluation and their relevance for the type of analysis performed. Six were scored as partially satisfying this criterion because they did not present their relevance for the type of analysis performed. Only 1 publication completely failed to satisfy this criterion.

In item 18, the study parameters were clearly made by 26 out of 32 publications.

In item 21, 26 publications described characterizing heterogeneity and 1 publication did not describe it. Five publications did not report fully about the differences in costs, outcomes, or cost-effectiveness that can be explained by variations between subgroups of patients with different baseline characteristics.

The remaining items fell in between these 2 extremes, with 50% to 75% of studies satisfying these criteria.

4. Discussion

The assessment criteria used in the present study—CHEERS—have been applied in a number of systematic reviews reporting the quality of pharmacoeconomic publications.^[49–51] Our findings confirm that there have been an increasing number of pharmacoeconomic publications from China during the period of 2003 to 2014, and the reporting quality has also been improving significantly over time. Pharmacoeconomics has received greater attention in China, which is particularly important for resource allocations and health policy decision making. This study reviews and summarizes the current state of pharmacoeconomic research and provides insight into how to improve the quality of such research in China. The results of our review may provide health policymakers with a sense of the reliability of existing research to help inform their decisions. Our study suggests that the potential for improving the quality of pharmacoeconomic analyses to inform health policy is substantial.

Some systematic reviews have studied the quality of pharmacoeconomic publications from other countries.^[52–54] Desai et al^[20] examined the quality of pharmacoeconomic studies in India with 29 articles, and found that the lack of sensitivity analyses and discounting was the most important problem. Woersching et al^[21] assessed the quality of economic evaluations of US Food and Drug Evaluation (FDA) novel drug approvals as a systematic review including 36 articles and found that a minority of the 2008 to 2009 novel drugs had mixed study quality. Gavaza et al^[54] studied the state of health economic evaluation (including pharmacoeconomic) research with 44 articles in Nigeria and concluded that the conduct of health economic (including pharmacoeconomic) research in Nigeria was limited, and about two-thirds of published articles were of suboptimal quality. The number of articles included in the above systematic reviews is in the range of 20 to 40, and the present study reviews 32 articles.

4.1. Quality of the studies

A lack of clarity in some basic items makes it difficult for readers to understand study objectives and to judge the quality and appropriateness of results. Although most published studies from China in our review have adhered to CHEERS, there are still a significant number of publications with low reporting quality. Significant items that affected the quality of economic studies included “Title,” “Comparators,” and “Measurement of effectiveness.” Our study suggests the need for improvement in these areas. Fortunately, these would seem to be reporting limitations that would be straightforward to correct in future studies.

Chinese authors should try to identify the study as an economic evaluation or use more specific terms such as “cost-effectiveness analysis,” and describe why they chose the compared interventions. In addition, some Chinese authors did not provide adequate information for the measurement of effectiveness.

There are approximately 1 million publications each year in the world.^[55] If the article titles are clear and indicate the nature of health economic studies, it will be much easier for researchers to search for them electronically and to compare results across similar studies. Previous studies have shown that a structured abstract is preferable to a descriptive structure for researchers to understand the main contents in a publication.^[56,57] In comparative studies, the choices of alternative interventions or drugs are particularly important to the results and conclusions of a study. The authors also need to explain the criteria for choosing the alternatives drugs that are being compared. Pharmacoeconomic publications from China have often omitted this important item, which is a serious shortcoming at present.

4.2. Limitations

The present study has several limitations that must be noted. First, the researchers are not blind to the authors and journals of pharmacoeconomic publications, so the assessment of reporting quality may be related to the personal opinions of reviewers. Second, all 24 items are treated equally, but some items may be more important in evaluating quality. Finally, the present study does not assess the scientific quality of pharmacoeconomic publications, as it only studies the reporting quality instead. Although reporting quality and scientific quality may be related, they are not the same thing.

5. Conclusions

Our systematic review identified 32 economic evaluations of drugs published between 2003 and 2014. Clarity of reporting should be an important part of every pharmacoeconomic study. Medical journals, particularly those with limited experience publishing pharmacoeconomic analyses, should adopt and enforce standard protocols for conducting and reporting this research.