Call for discussion about the framework for categorizing economic evaluations of health information systems and assessing their quality

In the recent JAMIA article ‘Measuring value for money: a scoping review on economic evaluation of health information systems (HIS),’ Bassi and Lau¹ identified 42 studies conducted from different perspectives within public, private and mixed healthcare systems and assessed 33 of them as good quality. The importance of establishing value for money of large HIS investments is indisputable. Numerous assessments of economic aspects of different HIS implementations have been conducted; however, there is a paucity of reviews classifying and summarizing their results. Bassi and Lau's review makes a useful contribution by introducing a new methodological framework, which classifies HIS studies with an ‘economic component’ and assesses their quality. In our view, the framework has some limitations that may confuse readers and result in misguided conclusions. Here we briefly discuss these limitations and we hope this will stimulate further discussion.

One of the central challenges is that many HIS studies, including those identified by Bassi and Lau, contain some information on benefits and/or costs, but such studies were often not designed to conduct an economic evaluation or assess economic efficiency as defined by the economics discipline. The stated objectives and the correspondent methods (eg, econometric modelling) may be sufficient to quantify ‘cost savings’ associated with investing in HIS, but cannot show that the investment is economically efficient (ie, provides better value than the alternative use of resources). Readers may incorrectly interpret results of such studies as evidence of economic efficiency. It would be helpful to establish what is required to demonstrate ‘value for money’ of HIS, and how to differentiate between studies that set out, and achieve, this objective from those studies that serve other, also useful, purposes. For example, in an earlier systematic review Shekelle and colleagues² identified 256 studies on costs and benefits of health information technology and used a multidimensional classification which, among other criteria, included study objectives and analytic techniques employed by the reviewed studies—namely, ‘predictive analysis’, ‘hypothesis-testing’, and ‘meta-analysis’.

In the absence of a conventional framework for categorizing HIS economic studies and assessing their quality, Bassi and Lau used the health economics framework of Drummond and colleagues³ and supplemented it with the usual criteria for assessing academic papers. Drummond and colleagues developed their framework over many years by numerous discussions, consultations, and meetings with academic experts, clinicians, and editors of leading medical journals.⁴ This widely accepted and extensively used framework includes a classification of health economic evaluation studies and a checklist for assessing their quality.

The classification is based on mutually exclusive categories and distinguishes full economic evaluations (cost-effectiveness analysis (CEA), cost-utility (CUA), and cost-benefit (CBA) analyses) from partial economic evaluations (eg, cost analyses or cost-outcome descriptions). Full economic evaluation is a comparative analysis of alternative options/programmes that involves identification, measurement, and valuation of both costs and outcomes, and establishes the difference in costs in relation to difference in outcomes in an incremental fashion. Results of economic analyses are expressed as an incremental cost-effectiveness ratio, net monetary benefit, or net health benefit.³ Conducted from a societal perspective (ie, including all costs and consequences, regardless of who bears them), health economic evaluations are consistent with the economic theory of maximizing social welfare, because by allocating limited resources to interventions with the lowest cost per unit of health gain, decision-makers can increase the total health and well-being of the society.

Drummond's economic evaluation study checklist includes quality assessment criteria (eg, whether the costs and outcomes for each alternative option were well defined, whether the perspective of the analysis was stated, whether discounting of costs and outcomes for a specified time frame was conducted). Bassi and Lau selected five of these criteria (perspective of the analysis, a specified time frame, presence of an alternative option for comparison, assessment of costs and outcomes) to identify some of the reviewed studies as ‘economic analyses’. Their sixth criterion stated that costs and outcomes should be compared for each option. Thus this sixth criterion called for an average cost-effectiveness ratio rather than an incremental cost-effectiveness ratio. However, an average cost-effectiveness ratio is likely to underestimate the cost for each additional unit of outcome compared with an incremental cost-effectiveness ratio.³

Using their definition of economic analysis, the authors identified only one CUA, two CEAs and six CBAs, while most studies (21 studies) were classified as either ‘input cost analysis’ or ‘cost-related outcome analysis’. Although the latter term resembles Drummond's ‘cost-outcome description’, it is rather ambiguous as some papers could be classified as both an ‘input cost analysis’ and ‘cost-related outcome analysis’. Some of the high-quality ‘hypothesis-testing’ papers that investigated associations between adoption of HIS and potential cost savings fell into this category. However, as argued above, no amount of cost savings provides evidence of economic efficiency unless the alternative use of resources and associated outcomes is considered.

The difficulties experienced by Bassi and Lau in adequately identifying and classifying HIS studies with an economic component, suggest that Drummond's framework may apply only to studies that were conducted to demonstrate economic efficiency by explicitly comparing two options in an incremental fashion. This is not to say that other HIS economic studies are unimportant. Full economic evaluation is not the only method of estimating costs and benefits. Society may be concerned with improving technical and productive efficiency from the resources that have already been allocated; however, to avoid confusion such studies should be differentiated accordingly.

This letter is written with the intention of stimulating further discussion about frameworks for categorizing HIS economic studies and assessing their quality. The framework needs to be sensitive to the stated objectives of HIS economic papers and specific enough to minimize ambiguity in allocating studies into categories. Reaching greater agreement about a framework is important for both assessing current evidence and guiding future research.