Assessing telemedicine: a systematic review of the literature

Abstract

Background

To clarify the current status of telemedicine, we carried out a systematic review of the literature. We identified controlled assessment studies of telemedicine that reported patient outcomes, administrative changes or economic assessments and assessed the quality of that literature.

Methods

We carried out a systematic electronic search for articles published from 1966 to early 2000 using the MEDLINE (1966–April 2000), HEALTHSTAR (1975–January 2000), EMBASE (1988–February 2000) and CINALH (1982–January 2000) databases. In addition, the HSTAT database (Health Services/Technology Assessment Text, US National Library of Medicine), the Database of Abstracts of Reviews of Effectiveness (DARE, NHS Centre for Reviews and Dissemination, United Kingdom), the NHS Economic Evaluation Database and the Cochrane Controlled Trials Register were searched. We consulted experts in the field and did a manual search of the reference lists of review articles.

Results

A total of 1124 studies were identified. Based on a review of the abstracts, 133 full-text articles were obtained for closer inspection. Of these, 50 were deemed to represent assessment studies fulfilling the inclusion criteria of the review. Thirty-four of the articles assessed at least some clinical outcomes; the remaining 16 were mainly economic analyses. Most of the available literature referred only to pilot projects and short-term outcomes, and most of the studies were of low quality. Relatively convincing evidence of effectiveness was found only for teleradiology, teleneurosurgery, telepsychiatry, transmission of echocardiographic images, and the use of electronic referrals enabling email consultations and video conferencing between primary and secondary health care providers. Economic analyses suggested that teleradiology, especially transmission of CT images, can be cost-saving.

Interpretation

Evidence regarding the effectiveness or cost-effectiveness of telemedicine is still limited. Based on current scientific evidence, only a few telemedicine applications can be recommended for broader use.

Telemedicine is the use of information and communications technology to provide health care services to individuals who are some distance from the health care provider. Rather than being a single technology, telemedicine is part of a wider process or chain of care. It has been assumed that telemedicine can improve this chain and thus enhance the quality and efficiency of health care. Telemedicine is also expected to increase the fairness and equality of the distribution of services, because the accessibility of health services, especially in remote areas, can be improved. Although the use of older approaches (telephone, fax) is commonplace, telemedicine applications increasingly use the latest innovations in computer and network technologies and other equipment.

Before adoption into routine use, any new technology has to be proved to be superior to the approach that it is intended to replace, that is, it has to be more effective or more cost-effective than the alternative(s). Telemedicine is no exception to this rule. There is growing acceptance that telemedicine systems require assessment and the ongoing collection of relevant data for administrative purposes before they can be considered for routine use on a large scale. Furthermore, assessment of telemedicine applications is needed to assist purchasing and planning decisions and also to monitor and modify the use of the technology when it is in place.¹

Ideally, assessment should provide a broad description of telemedicine that covers technical, clinical, economic, ethical, legal and organizational issues. In practice, assessments have been constrained by the availability of data, the timing of policy and administrative decisions, a shortage of evaluators and inertia within health care systems. Earlier reviews^2,3,4,5,6,7 have indicated that assessment studies dealing with telemedicine are scarce. In a 1997 review of telemedicine applications,⁸ only one cost-effectiveness study could be identified. Since then several other economic studies have been published, but there is still a great need for high-quality evaluation.

In this paper, we examine the evidence for the effectiveness and economic efficiency of telemedicine in order to clarify the current status of the technology. The review is intended to help decision-makers who are under commercial or public pressure to establish telemedicine services, by providing an objective view of what is known at present about the effectiveness and cost-effectiveness of telemedicine.

Methods

Computerized literature searches were performed using the MEDLINE (1966–April 2000), HEALTHSTAR (1975–January 2000), EMBASE (1988–February 2000) and CINALH (1982–January 2000) databases and the search strategy described in Table 1. In addition, the HSTAT database (Health Services/Technology Assessment Text, US National Library of Medicine), the Database of Abstracts of Reviews of Effectiveness (DARE, NHS Centre for Reviews and Dissemination, United Kingdom), the NHS Economic Evaluation Database and the Cochrane Controlled Trials Register were searched using the search term “telemedicine.” Some articles were also identified by reading reference lists of published review articles and by consulting experts in the field of telemedicine.

Table 1

Inclusion criteria were the following: articles had to consider, in a scientifically valid manner, the outcomes of a form of telemedicine in terms of administrative changes, patient outcomes or economic assessment. In addition, studies were required to include a comparison between a telemedicine application and a conventional alternative. Criteria for scientific validity were that the nontelemedicine alternative was related to the same application and health system as the telemedicine project and that sufficient data were included to permit comparison of the outcomes of the telemedicine and nontelemedicine alternatives.

The exclusion criteria required the rejection of articles that were limited to describing the feasibility or technical evaluation of a certain system, and to the rejection of noncontrolled studies. Articles that were duplicates of the same authors' other published studies were also excluded; the most representative of the studies was included for further consideration.

Initial screening of the articles that were identified was based on their abstracts. All abstracts were read independently by at least 2 of the authors, who recorded their opinions. The selection of the relevant articles was based on the information obtained from those abstracts, which gave some expectation that inclusion criteria would be met, and was agreed upon in consensus meetings among the reviewers. Full-text articles that were obtained for closer inspection were again evaluated independently by 2 of the authors who, in a consensus meeting, made the final decision on whether or not an article should be included in the final review. Abstracts and full-length papers were examined by the same criteria.

The strength of the evidence in each of the studies included, other than those concerned only with economic analysis, was judged according to the classification system drawn up by Jovell and Navarro-Rubio,⁹ in which study design is specified as one of 9 levels in descending order of strength (Table 2). Each level is further qualified by conditions of scientific rigour for the study.

Table 2

Each selected article was described using the strength of evidence according to the 9-level classification, the objectives of the study, approach, setting and subjects, type of economic analysis, and the results and conclusions of the authors. Original descriptions by the authors were mainly used, although in some cases they appeared to be misleading.

Results

The use of this approach enabled us to identify 1124 articles dealing with telemedicine. One hundred and thirty-three full-text articles were obtained for closer inspection. Of these, 50 were deemed to fulfill the inclusion criteria of the review and are listed and briefly described in Tables 3–6.^{10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59} Fourteen of the studies considered the application of telemedicine to medical consultation of various types, and 7 dealt with patient monitoring or counselling. A further 13 were concerned with teleradiology, and the remainder with emergency department care, psychiatry, dermatology, cardiology, ophthalmology and pathology.

Table 3

Table 4

Table 5

Table 6

Thirty-four of the articles assessed at least some clinical outcomes; the remaining 16 were mainly economic analyses. Some kind of economic analysis was included in 30 (60%) of the studies. In terms of study design, the quality of the clinical studies ranged in most cases from fair to poor. According to the Jovell and Navarro-Rubio classification,⁹ 6 were randomized controlled trials (RCTs), corresponding to levels 2 or 3, 4 were level 4 or 5, 13 level 6, 6 level 7 and 5 level 8. Conditions of scientific rigour varied considerably. In many papers, procedures for the selection of patients and for the reading and interpretation of clinical findings were not adequately described. The outcome measures used were sometimes vaguely defined or clinically not very relevant.

Although RCTs provide the strongest study design, the strength of the evidence obtained will also be dependent on the quality of the study. The 6 papers that were located that described RCTs provide an illustration of the variation in study quality and reported outcomes. Two of the larger RCTs^29,52 were well described in terms of the randomization and subsequent procedures. The first of these, which considered automated telephone calls and management of diabetes, showed improvement in glycemic control and other benefits through the use of a telemedicine approach. However, no effect on health-related quality of life (HRQOL) was demonstrated. The second, which considered real-time teledermatology, indicated that there was no significant clinical difference from conventional consultations. A linked economic analysis indicated that teledermatology was not cost-effective under the conditions of the trial.

A further report of a large RCT²⁸ had a more limited description of randomization but showed that telelemedicine using a telephone-based system improved compliance with medication and led to a significant decrease in blood pressure.

Of the papers about smaller RCTs, that by Brennan and colleagues⁴⁵ appeared to have been well performed; the authors found that clinical outcomes were similar for telemedicine and for the alternative approach in an emergency department setting. Another paper, which described a pilot project for a larger RCT, indicated time savings for patients as a result of video consultations, but no significant difference in HRQOL between groups.¹⁷ This study appeared to be more limited in quality, with substantial dropping out of patients and possibly insensitive outcome measures. The third small RCT found no significant difference between tele–exercise monitoring and a hospital-based program.²⁴ The power of the study was low, and further investigations would be needed to assess this application.

The nonrandomized clinical studies also varied in their quality, as judged by the descriptions in the articles, and in their outcomes. Some would have provided useful indications to decision-makers in the health systems concerned. For example, the study by Trippi and colleagues⁴⁴ showed that 72% of patients scheduled for hospital admission had normal results in dobutamine stress tele-echocardiography and could, therefore, be discharged instead of being admitted to hospital. Giovas and colleagues⁴⁶ reported that pre-hospital diagnosis by electrocardiography, using a telemedicine link to the ambulance, took place 25 minutes before hospital diagnosis for a control group. Other studies indicated important clinical benefits through avoiding the unnecessary transfer of patients. For example, Goh and colleagues³⁷ reported that the use of teleradiology in the management of neurosurgical patients reduced both numbers of transfers and adverse events during transfer and also increased the number of therapeutic measures before transfer was undertaken.

In most of the studies, effectiveness was defined in clinical terms. Only 2 studies^17,29 included standardized HRQOL measures. No studies employed quality-adjusted life-year (QALY) calculations. Given the diversity of the studies in terms of design, topics covered, populations and health care settings, calculation of a notional average for effect size was not feasible. Indications of effect sizes for some of the studies are given in Tables 3–6.

Few comprehensive economic analyses were included in the articles. The analyses mainly measured direct medical costs, although some kind of estimation of transportation costs was included in 25 studies. Indirect costs were assessed in 4 studies,^22,34,53,59 incremental cost analysis was performed in one¹⁸ and cost-effectiveness ratios were also calculated in one.²⁸ Discounting of costs was included in 7 studies,^{18,20,22,34,39,52,58} and 14 included some kind of sensitivity analysis or break-even analysis of the study results.

Most of the economic analyses were variants of cost analysis. Cost–benefit analysis was said to have been carried out in 3 studies.^13,41,52 However, these were methodologically more like cost-analysis studies, because the benefits were estimated as savings (mainly the cost of travel) compared with the conventional alternative.

Demonstrated savings in costs of transportation varied considerably among the different health care situations described in the papers, from a 40% reduction to no savings as a result of telemedicine. Three of 4 studies of the transmission of diagnostic images indicated that telemedicine was more costly than the cheapest alternative.^42,54,59

Economic analyses have mostly shown that teleradiology, especially transmission of CT images, can be cost-saving, although one of the studies, which was of good quality, did not find this to be the case.³⁴ An important contribution to the discussion about the cost-effectiveness of teleradiology is the study by Bergmo,³⁹ which explicitly provides a measure of the workload that has to be exceeded in order to achieve cost savings by using teleradiology (break-even analysis). A similar study, also undertaken by Bergmo, has shown that specialist consultations in the field of otorhinolaryngology can be performed in a cost-saving way when the workload exceeds a certain number of patients.¹⁸

Pilot projects in telepsychiatry, the provision of orthopedic and dermatology services via telemedicine and the evaluation of the costs and benefits of a prison telemedicine program used a similar approach.^22,23,47 Such studies that give a clear number needed to treat by the telemedicine option are helpful for decision-makers when faced with the question of whether or not to start a new telemedicine service. Teledermatology, with short distances (26 km) between sites, appeared not to be cost-saving in one study.⁵²

The quality of the economic analysis in the papers was relatively low, with a few exceptions. The papers by Bergmo,^18,39 Agha and colleagues,⁵⁹ Stensland and colleagues,²² Halvorsen and colleagues³⁴ and Wootton and colleagues⁵² provide examples of better-quality economic studies. The costs included varied significantly among studies, so that comparison of the cost estimates may not be feasible in many cases. There were also several economic studies that did not give detailed information about the empirical background of the costs or benefits, or both, included in the calculations. These studies were excluded from the review. For example, we excluded a teleradiology cost–benefit analysis,⁶⁰ because the theoretically good economic model did not make use of the empirical cost and benefit estimations made at specific sites by the study group.

Interpretation

The review shows that there are still few data on the effectiveness and cost-effectiveness of telemedicine. Of the more than 1000 articles surveyed, most were reports about the feasibility of various applications, and only a few of the studies reported a controlled comparison of a telemedicine application with conventional means of providing services.

The review indicates that, at the moment, the most convincing published evidence regarding the effectiveness of telemedicine deals with teleradiology, teleneurosurgery (transmission of CT images before patient transfer), telepsychiatry, transmission of echocardiographic images, and the use of electronic referrals enabling email consultations and video conferencing between primary and secondary health care providers. However, even for these applications, most of the available literature refers only to pilot projects and short-term outcomes, and in many cases the efficacy of the application was being considered, rather than its effectiveness. Promising results have been obtained for the transmission of electrocardiograms and teledermatology. For other applications, scientific data concerning the effectiveness of telemedicine remain limited.

There are still few cost-effectiveness studies of telemedicine. A systematic comparison of the costs and more work on the effects of the alternatives should be done in the future. Although the term “cost-effectiveness” was frequently used in the studies, the effectiveness (and sometimes costs) were assumed to be established for telemedicine without any scientific evidence. As a result, decision-makers must be cautious regarding the degree to which they can apply the results of such assessments to their own circumstances. Assessments of telemedicine have so far been on stronger ground when considering the effects of the technology on the time-related consequences of health care services and on organizational issues.

Five years ago, an editorial in the Lancet stated that “although much is claimed, the economic benefits of telemedicine have yet to be proved.”⁶¹ Although a limited number of telemedicine applications have up to now been shown to be effective and cost-effective in specific settings, that original conclusion still remains valid for most of the suggested ways in which to use telemedicine. Although a number of detailed studies are in progress in several countries, the assessment literature has yet to address aspects of telemedicine applications as they move into routine use, or their longer-term impact on health status, costs and organization. Other dimensions will also require consideration when formulating approaches to further economic analysis. These will include the sustainability of a telemedicine service, decisions about equipment and telecommunications, impact on the overall use of health program resources and measurement of outcomes.⁶²

We conclude that further assessment studies in the field of telemedicine are still clearly needed. Decision-makers who are under public and commercial pressure to start new telemedicine services should link the implementation of new and, in many instances, costly technology to realistic development of a business case and subsequent data collection and analysis. Guidance for performing such an assessment can be found easily in a number of frameworks formulated by various authors.^{1,8,63,64,65,66,67}