Systematic review of cost effectiveness studies of telemedicine interventions

Abstract

Objectives

To systematically review cost benefit studies of telemedicine.

Design

Systematic review of English language, peer reviewed journal articles.

Data sources

Searches of Medline, Embase, ISI citation indexes, and database of Telemedicine Information Exchange.

Studies selected

55 of 612 identified articles that presented actual cost benefit data.

Main outcome measures

Scientific quality of reports assessed by use of an established instrument for adjudicating on the quality of economic analyses.

Results

557 articles without cost data categorised by topic. 55 articles with data initially categorised by cost variables employed in the study and conclusions. Only 24/55 (44%) studies met quality criteria justifying inclusion in a quality review. 20/24 (83%) restricted to simple cost comparisons. No study used cost utility analysis, the conventional means of establishing the “value for money” that a therapeutic intervention represents. Only 7/24 (29%) studies attempted to explore the level of utilisation that would be needed for telemedicine services to compare favourably with traditionally organised health care. None addressed this question in sufficient detail to adequately answer it. 15/24 (62.5%) of articles reviewed here provided no details of sensitivity analysis, a method all economic analyses should incorporate.

Conclusion

There is no good evidence that telemedicine is a cost effective means of delivering health care.

What is already known on this topic

The use of telemedicine has garnered much attention in the past decade

Hundreds of articles have been published claiming that telemedicine is cost effective

However, missing from the literature is a synthesis or meta-analysis of these publications

What this study adds

A comprehensive literature search of cost related articles on telemedicine identified more than 600 articles, but only 9% contained any cost benefit data

Only 4% of these articles met quality criteria justifying inclusion in a formalised quality review, and most of these were small scale, short term, pragmatic evaluations with few generalisable conclusions

There is little published evidence to confirm whether or not telemedicine is a cost effective alternative to standard healthcare delivery

Introduction

Proponents of telemedicine have championed its potential economic benefits in recent years. In Britain telemedicine systems have been proposed as a cost effective means of responding to structural problems in the organisation of the NHS.¹,² In the United States telemedicine systems have been used, especially in rural areas, as a means of easing the problem of obtaining specialist advice and making referrals over wide distances.³ Even so, major questions about the cost effectiveness of telemedicine systems remain because the data used to support such claims are dispersed across studies of widely differing systems and diverse organisational contexts. Reported studies are often small in scale, methodologically flawed, and reflect pragmatic evaluations rather than controlled trials, making them unsuitable for formal meta-analysis.⁴ Given these problems, our objective was to address the question of whether the existing literature allows any conclusions to be drawn about the cost effectiveness of telemedicine interventions.

Telemedicine is defined here as clinical practice for diagnosis, review, or management undertaken synchronously or asynchronously through the medium of information and telecommunications technologies (excluding telephone and fax).

Methods

Two independent systematic reviews were started, one in the United States (Whitten and colleagues) and the other in Britain (Mair and colleagues). When each group became aware of the other's work it was decided to adopt a common strategy and share results, while still reviewing articles independently. As a result, both groups performed analyses on articles obtained through a common search strategy. Both groups categorised articles into those with or without data. The first group analysed articles without data by themes and articles with data by results. The second group determined which articles with data met quality criteria justifying inclusion in a quality review and performed that review.

Search strategy

We identified empirical studies of the costs associated with telemedicine interventions by searching the following electronic bibliographic databases: Medline (1966-June 2000), Embase (1988-June 2000), ISI Science Citation Index (1981-June 2000), ISI Social Science Citation Index (1981-June 2000), ISI Arts and Humanities Citation Index (1981-June 2000), and the database of the Telemedicine Information Exchange (TIE). Searching was restricted to articles in English language journals, with the keywords “telemedicine or telehealth or telemonitoring or telecommunications and cost or cost-effectiveness or economic or cost analysis or budget or financial or health care costs.” We used multiple keyword sets to maximise results from the searches.

Data analysis

We divided the articles for analysis into two sets: those with and those without data. We categorised articles without data by theme, and achieved intercoder reliability of 90% for this thematic coding. For articles with data, we devised a data extraction instrument for coding aspects of each study. These included: location; type of service (such as specialty); delivery (synchronous or asynchronous); type of facility; duration and dates of study; number of patients, consultations, and images transmitted; cost revenue variables; and study conclusions.

The next stage required us to select articles with data for inclusion in a full review to assess their scientific quality. Variations in study design, patient groups, study context, and types of analysis meant that formal meta-analysis was inappropriate. For this reason, we present a qualitative analysis of the studies obtained. Although randomised controlled trials are the preferred source of evidence for inclusion in systematic reviews,⁵ hardly any such studies are to be found in the field of telemedicine. In view of this, we decided to include all published studies of telemedicine systems. When one project led to multiple reports (redundant publications), we reviewed the principal paper (with the greatest number of subjects) focusing on cost effectiveness. Our inclusion criterion for formal review was

• Original research on telemedicine examining cost effectiveness of healthcare delivery.

Our exclusion criteria were

• Papers reporting cost benefits of telemedicine used mainly for educational or administrative purposes

• Papers reporting hypothetical cost analyses or modelling exercises without any associated formal clinical trial

• Papers reporting economic analyses without any means of substantiating claimed resource use.

Two groups of reviewers examined the papers. Reviewers had expertise in telecommunications (PSW), family practice (FSM), health economics (AH) and sociology (CRM); all had experience with telemedicine services and research. The almost complete absence of randomised trials in the dataset meant that it was not possible to use traditional measures for assessing trial quality. However, two reviewers (FSM and AH) independently rated reports using an established checklist of criteria for assessing the quality of economic evaluations in health care (see box).⁶

Criteria used for assessing cost effectiveness studies⁶

• Presence of a clear hypothesis

• Presence or absence of a clear statement regarding the perspective from which cost effectiveness assessed

• Methods

• Presence or absence of a comparator and whether appropriate

• Is the quality of medical evidence adequate?

• Appropriate costs considered

• Appropriate benefits considered

• Assessment of whether a marginal analysis has been undertaken

• Has a sensitivity analyses been undertaken?

• Is the analysis appropriate to the local environment?

Results

Our literature search identified 612 articles, which we categorised as without cost data (n=557) or with cost data (n=55). We then thematically categorised the reports without cost data to determine the main premise for each article, and table 1 shows the primary categories.

Table 1.

 Primary categories of 557 articles examining cost effectiveness of telemedicine for healthcare delivery that lacked cost data

Category	No (%) of articles
Project overview, or telemedicine history	194 (34.8)
Technology	95 (17.1)
Positive economic outcomes	80 (14.4)
Legal, regulatory, or policy	52 (9.3)
Miscellaneous	49 (8.8)
Editorial or opinion	27 (4.8)
Positive medical outcomes	20 (3.6)
Negative economic outcomes	13 (2.3)
Review analysis	12 (2.2)
Continuing medical education or continuing education	11 (2.0)
Negative medical outcomes	4 (0.7)

Quantitative analysis of reports with cost data

The 55 articles with data on costs and benefits reflected the diversity of the clinical and institutional contexts in which telemedicine is undertaken.^w1–w55 Most of the reports (33, 60%) were from the United States; seven (13%) were from Norway, and the rest were from Australia, Canada, the rest of Europe, and Japan. Table 2 shows the specific variables used in the 55 studies, and table 3 lists the conclusions that were drawn, almost universally positive.

Table 2.

 Variables used in the cost analysis of the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data

Variable	No (%) of studies
Line charges	36 (65)
Equipment cost	33 (60)
Other project specific cost	25 (45)
Consultation fees or wages	25 (45)
Support staff fees or wages	23 (42)
Patient non-emergent transportation expenses	21 (38)
Standard ancillary cost (such as labs, ambulance)	19 (35)
Consultation transportation cost	13 (24)
Standard hospital cost (inpatient hospital costs)	12 (22)

Table 3.

 Conclusions drawn in the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data

Conclusion	No (%) of studies
Telemedicine saves money	20 (36)
Telemedicine saves time and money	11 (20)
Telemedicine cost effective only if a certain threshold is achieved	9 (16)
More work needed to accurately determine cost effectiveness	7 (13)
Other (such as enthusiasm, staff involvement)	4 (7)
Telemedicine does not save money	4 (7)

Qualitative analysis of reports with cost data

Of the 55 studies, we subjected only 24 to full review with our checklist.^{w4–w6 w8 w13–w16 w19 w24 w26 w31 w33–w35 w39 w42 w48 w50–w55} (See extra table on bmj.com.) We excluded the other 31 studies because of duplication in publication of final results or their being limited to hypothetical or modelling exercises without formal trial data, being published only in abstract form, or providing insufficient information to assess the quality and reliability of the underlying study.

Although most of the 24 reports showed clear aims, only three constructed a formal hypothesis. Twenty were restricted to simple cost comparisons, rather than exploring wider implications for patient care. Only 11 studies specified the perspectives from which economic analysis was undertaken and so delineated the boundaries of analysis in a way that made their proper interpretation possible. Only four studies used a societal perspective, whereas the other 20 examined costs and benefits only from the narrow perspective of the healthcare provider (such as the hospital or prison). This partial approach impedes generalisability and neglects the comprehensive analysis of overall long term implications of a system—especially the analysis of opportunity costs, where health resources are valued in terms of the best alternative use for such resources.

Failure to make clear the perspective and boundaries of analysis was paralleled by use of inappropriate economic analytical techniques. Five of the studies used cost analysis, 15 used cost minimisation analysis, and four used cost effectiveness analysis. None of the studies used cost utility analysis, the conventional means of establishing the “value for money” that an intervention represents. Failure to calculate costs and benefits using an appropriate technique did not prevent considerable attention being paid to the potential economies of scale that would come about once telemedicine systems were more widely used. However, only seven studies attempted to explore the level of use that would be needed for telemedicine services to compare favourably with traditionally organised health care, and none addressed this question in sufficient detail to answer it adequately. Comparisons of costs with traditional forms of health care were absent in two studies and hypothetical in nine. (See extra table on bmj.com for details of the comparators and methods used in each study.)

Most of the studies entirely equated benefits with cost savings, with no analysis of changes in benefit to patients. Seventeen avoided the problem of benefits measurement by assuming equivalence with conventional medical practice on the basis of little or no efficacy data. This fundamental assumption of equivalence with conventional medical practice remains unproved.⁷

The short duration of projects (only 20% of interventions lasted longer than 12 months) introduced additional problems in interpreting the results. In any service evaluation, early point estimates of costs and benefits may not reflect costs and benefits that arise once the service has become established and is operating at a steady state. Sensitivity analysis is therefore essential to incorporate anticipated future changes in costs and benefits arising either through technological development, enhanced scale, or simply as a consequence of increasing familiarity with telemedicine. However, 15 of the studies simply relied on single point estimates obtained in a highly variable and dynamic environment.

Discussion

Our database search of English language articles identified only 55 that provided actual cost data on telemedicine interventions, and of these only 24 stood up to a full review using an established instrument for assessing the quality of economic evaluations. Most of the studies analysed were small scale, short term, pragmatic evaluations that added little to our knowledge of the costs and benefits that would be expected to result from the introduction of telemedicine services into routine clinical practice.

With such services it is difficult to generalise the results of individual cost effectiveness studies. In large part, comparative cost effectiveness of telemedicine systems depends on the unique local aspects of the individual service being evaluated. A telemedicine service that is cost effective in the remote highlands of Scotland is unlikely to generate the same cost effectiveness in the middle of Manchester. The difficulty of generating generalisable messages from evaluations undertaken in specific contexts emphasises the importance of assessing the local applicability of individual examples of the use of telemedicine. It is important to recognise that a service may be highly clinically and cost effective in one context but highly ineffective when transferred to another context in which accessibility and quality of local services are far higher.

Use of an established instrument to assess quality in economic evaluations serves to highlight the serious deficiencies in this body of peer reviewed and published literature. There was no uniformity of analysis, and the studies we reviewed were marked by poor design and inadequate technical quality. None of the studies used cost utility analysis to compare simultaneously variations in cost and outcomes. Cost utility analyses use outcome measures that are comparable and generalisable across different treatments as a means of comparing their “value for money.” The lack of such studies means it is impossible to assess the extent to which telemedicine represents a sensible priority for healthcare investment. Until research in this area is conducted in accordance with generally accepted standards for economic analysis of health services, claims made on the basis of this body of literature should be regarded with considerable caution.

Conclusion

Reviews leading to negative conclusions are rarely welcomed. In the case of telemedicine, although claims about the utility and efficacy of new telecommunications systems in practice have been widely made,⁴ these are not founded on strong evidence. Given the paucity of methodologically sound studies producing robust and generalisable conclusions, there is presently no persuasive evidence about whether telemedicine represents a cost effective means of delivering health care.

The conventional response to such news is to emphasise the need for better designed and conducted trials, and these are certainly necessary. However, focusing on deficiencies in the design and conduct of research should not divert our attention from the other deficiencies revealed by this review. Peer review also seems to have failed to address adequately the range of basic errors in design and analysis that littered the studies we reviewed. Interested clinicians, policy makers, and healthcare providers should not assume that peer reviewed publication is necessarily an adequate guarantee of quality for economic evaluations of telemedicine.