Check Your Checklist: The Danger of Over- and Underestimating the Quality of Economic Evaluations

Introduction

‘Check your checklist’, appears at first glance to be a catchy marketing slogan developed for the mass media. Its imperative nature suggests that readers need to be cautious and to act upon this caution. This is indeed necessary in the field of economic evaluation; we should think through whether, which, and how we will use or demand the use of a checklist. In this editorial, the dangers of incorrect use will be outlined and possible solutions towards improved future use of checklists are suggested.

The history of quality appraisal and reporting checklists goes back to the introduction of economic evaluation as a tool to inform decision makers. After its introduction as a mandatory tool in various countries, several checklists have been developed over the years in an attempt to ensure the quality of the economic evaluations conducted. Many of us, if not all, will be aware of the Drummond checklist [1], the Philips checklist for modelling [2], and the Consensus on Health Economic Criteria (CHEC) checklist [3] for trial-based evaluations. The Assessment of the Validation Status of Health-Economic Decision Models (AdViSHE) checklist for appraising validation steps taken by authors will now also be familiar to many [4]. In addition to the above-outlined checklists, the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist for reporting was created to update previous guidelines and was published in various journals at the same time [5].

These checklists have helped to form and improve the quality of economic evaluations. Every student and professional in the field of health technology assessment (HTA) will be aware of the Drummond textbook, explicitly outlining the reporting checklist published in the British Medical Journal in the 1990s [1]. Checklists have helped to determine a minimum quality of the performed and (non) published economic evaluations. Furthermore, it can be argued that checklists have raised the quality bar.

Although checklists have helped to improve quality, the current problem we observe is a proliferation of available checklists, often with very narrow focuses, and a lack of guidance on which checklist(s) to use and when [6, 7]. In a recent review, Wijnen et al. outlined the presence of a total of 13 different quality appraisal checklists, referred to by them as risk-of-bias assessments [7]. This number is already overwhelming for professionals in the field of HTA, let alone for clinical professionals who are not familiar with the field but who may wish to check the quality of an article or articles they are interested in.

This editorial does not aim to give a complete overview of all checklists and/or guidance in choice of checklist. It is not ‘a checklist for choosing the right checklist’, but instead is a follow-up to previous articles, and aims to be a call for action [6, 7]. We need to think carefully about when checklists are applied or demanded.

A lack of guidance and the associated freedom of choice for analysis when selecting checklists can result in three important issues: (1) choice of the wrong checklist; (2) wrong conclusions; and (3) incorrect use of the checklists. With regard to these three issues, we should act differently according to the different hat we are wearing when setting up or reviewing a new manuscript, i.e. that of a reader, author, reviewer, or editor of a journal.

Choice of the Wrong Checklist

A commonly observed mistake is the choice of the wrong checklist. In a previous review, it was outlined that the Drummond (general reporting), CHEC (quality of trial-based evaluations), and Philips (quality of model-based evaluations) checklists were regularly used in an incorrect manner [6]. For example, using the Drummond checklist for appraising modelling studies will lead to an incomplete quality assessment as the focus of this checklist is on reporting of the general methods and results, not on the inclusion of characteristics necessary for a good modelling study. We should at least be consistent in differentiating between checklists for complete and accurate reporting, such as the Drummond checklist, and tools for assessing risk of bias in methods applied, such as the CHEC and Philips checklists. Checklists should be used keeping in mind the objective for which they have been developed.

In addition, we cannot conclude from the quality of reporting checklists, such as the Drummond checklist, that the methodological quality of the study is good or bad; we only assess whether reporting is complete. Of course, incomplete reporting will hamper the assessment of risk of bias as there will be uncertainty whether something was correctly done but not reported, or incorrectly done.

Wrong Conclusions

In addition to the wrong choice of checklist, too much emphasis is often placed on the outcomes of checklists; in other words, blind trust is often placed based on the adherence to checklist characteristics (percentage) coming out of the assessment. We, not only as authors but also reviewers, should be aware of the fact that even 100% adherence to checklists does not make the study perfect. Previous research has already indicated that current checklists are still not exhaustive [8–11]. This is inherent in attempts to make checklists generic, the focus being on establishing a minimal set of important criteria relevant for a broad range of studies. In response, typical advice is to add further relevant items that are specific but important for your study or clinical field, but to always keep the original set of items. An example is the inclusion of disease-specific parameters in checklists, characteristics that are often not picked up by current checklists but make the outcomes of economic evaluations less or more valid.

Moreover, it is often overlooked that not every characteristic of a checklist is as relevant or important, which becomes a serious issue when percentages of adherence to checklists are used as a guage of quality. It is generally accepted that the percentage of ticks on a checklist is not in itself a good measure of quality, although it is often done that way. We should therefore not add up the number of boxes checked in a checklist, but transparently communicate specific outcomes in a table for instance.

Incorrect Use of Checklists

Guidance for the correct use of checklists is missing and therefore incorrect use of checklists is observed when reviewing or reading manuscripts. Adapting checklists, selectively choosing certain questions and removing others, are steps taken by authors of economic evaluations. We as reviewers, readers, and editors should be aware of this as both quality and reporting checklists become invalid when they are adapted. Such guidance should be given in guidelines for authors at manuscript submission. Apart from choosing the correct checklist and outlining why a checklist is chosen, it is essential to stress upfront that adaptations of quality or risk-of-bias checklists are unwanted.

Action

Performing critical appraisals of economic evaluations is a time-consuming task, especially when performing a systematic review and numerous economic evaluations are included. Checklists can consist of a large number of questions, and good practice demands that reviews be performed independently by two researchers, including discussion to reach consensus. In contrast to authors, editors should therefore provide upfront guidance to authors regarding the correct choice of checklists to complete before manuscript submission takes place. Due to the time-consuming task, it is of utmost importance to help and guide authors in the most efficient use of their limited time and resources to ultimately prevent research waste.

A checklist for selecting the right checklist could be seen as a solution to the problem. If this is the way forward, more research is needed to efficiently incorporate and combine all available checklists and to categorise and set out types of studies for which different checklists would be suitable. More guidance is needed not only from the scientific community but also from editors and journals themselves. What can be seen as both the correct and incorrect use of a checklist? Which checklists are preferred by journals and how do they handle adapted checklists by authors?

We should be aware that such efforts will not be cost-free. In addition to guidance, it is, for instance, important to adapt existing checklists over time in order to reflect methodological advances and to improve the language to make them more easily interpretable; something that takes resources, leading to the question who is responsible for these adaptations. Continuous attention of each different stakeholder involved in the cascade from research conceptualization to manuscript writing is therefore needed. We should also act when new checklists become available, i.e. the additive value of these checklists should be explained and guidance should be given on how they should be used, and when, in relation to existing checklists.

In addition to the above-outlined solutions, both the availability of open source and transparent models and prospective registration of protocols for economic evaluation could also play a role in quality improvement [12]. Having such models and protocols would increase quality over time and upfront publication of results, increase validity of future decisions, and decrease the amount of research waste, a cost-efficient approach for performing cost-effectiveness analyses. Complete quality checklists could tie everything together by, for instance, including requirements for registration and open-source models. Relevant, and of utmost importance in such complete assessments, remains to be the validity of the checklist itself. Therefore, check your checklist.

Compliance with Ethical Standards

Conflicts of interest

Geert W.J. Frederix has no conflicts of interest.

Funding

No funding was received to assist in the preparation of this editorial.