Experience with low-cost telemedicine in three different settings. Recommendations based on a proposed framework for network performance evaluation

Abstract

Background

Telemedicine has been used for many years to support doctors in the developing world. Several networks provide services in different settings and in different ways. However, to draw conclusions about which telemedicine networks are successful requires a method of evaluating them. No general consensus or validated framework exists for this purpose.

Objective

To define a basic method of performance measurement that can be used to improve and compare teleconsultation networks; to employ the proposed framework in an evaluation of three existing networks; to make recommendations about the future implementation and follow-up of such networks.

Methods

Analysis based on the experience of three telemedicine networks (in operation for 7–10 years) that provide services to doctors in low-resource settings and which employ the same basic design.

Findings

Although there are many possible indicators and metrics that might be relevant, five measures for each of the three user groups appear to be sufficient for the proposed framework. In addition, from the societal perspective, information about clinical- and cost-effectiveness is also required. The proposed performance measurement framework was applied to three mature telemedicine networks. Despite their differences in terms of activity, size and objectives, their performance in certain respects is very similar. For example, the time to first reply from an expert is about 24 hours for each network. Although all three networks had systems in place to collect data from the user perspective, none of them collected information about the coordinator's time required or about ease of system usage. They had only limited information about quality and cost.

Conclusion

Measuring the performance of a telemedicine network is essential in understanding whether the network is working as intended and what effect it is having. Based on long-term field experience, the suggested framework is a practical tool that will permit organisations to assess the performance of their own networks and to improve them by comparison with others. All telemedicine systems should provide information about setup and running costs because cost-effectiveness is crucial for sustainability.

All doctors have their own personal network of contacts that they use when the diagnosis and management of cases exceeds the breadth of their own knowledge and experience. However, personal networks are not always sufficient and have shown many limitations in practice (1). Thus, doctors often make use of telemedicine networks that give access to a wider range of contacts than individual doctors commonly have available themselves.

There is a long history of telemedicine networks being used to support doctors, particularly in low-resource settings, such as developing countries (2). Such telemedicine networks are used mainly for educational or clinical purposes. Networks that are used clinically generally provide diagnostic and management advice about specific patients and/or care problems (3). In the present article, we describe the characteristics of a form of telemedicine called teleconsultation (sometimes referred to as the provision of tele-expertise or second opinions).

Among telemedicine networks with a clinical purpose, two fundamentally different models of operation exist. These correspond to the different ways in which an expert is chosen to reply to a query. In the first, a query is directed to a specific expert for reply; this might be called the targeted model. In the second, a query is directed to a group of experts, of whom all, some or none may reply; this might be called the open or distributed model. Examples of such networks are listed in Table 1.

Table 1.

Telemedicine networks for low- and middle-income countries (second opinion teleconsultation networks providing services in the developing world with more than 7 years of experience)

Network	Countries	Model
Swinfen Charitable Trust	Various	Targeted
Ukraine	Mainly Ukraine	Targeted
Institute of Tropical Medicine	Mainly Africa	Targeted
iPath	Various	Open
Partners Healthcare, Boston, USA	Cambodia	Targeted
Tripler Army Medical Center, Honolulu, USA	Mainly Pacific region	Targeted

Telemedicine appears to be useful in the context of the developing world (2) and store-and-forward teleconsultation is one of the main applications used in this context (3). But which is the best model for teleconsultation? Which networks are performing well and which are not? To answer these questions, we need a framework for evaluating network performance. Frameworks exist in other domains (e.g. business, health improvement), but these are not really appropriate for our purposes and there does not appear to be a validated framework for evaluation and assessment of telemedicine systems. Thus, the objectives of the present study were:

1. to define a basic method of performance measurement that can be used to improve and compare teleconsultation networks;

2. to employ the proposed framework in an evaluation of three existing networks; and

3. to make recommendations about the future implementation and follow-up of such networks.

Methods and findings

In accordance with the three objectives, the work was conducted in three stages, as follows.

Development of a framework

To develop a suitable framework for measuring network performance, we first defined the concept. Then, we defined the actors. Finally, we defined the measurement perspective. This allowed the framework itself to be defined.

Concept of performance

Measuring the performance of a system is one aspect of its overall evaluation. A telemedicine network does not exist in isolation. It is a component or sub-system within an organisation. The performance of the organisation as a whole can be represented by its capacity to fulfil four functions:

1. to realise its mission, for example to achieve its goals;

2. to acquire resources, such as money and prestige, and adapt them to its needs;

3. to produce services of appropriate quality and volume; and

4. to maintain and develop its internal values, that is its organisational culture.

In practice, there will be a dynamic tension between the achievement of these four functions4.

A telemedicine network can, therefore, be viewed as a tool for achieving the third of the four organisational functions. How then can the performance of such a system be measured? To evaluate any system requires:

1. selecting the characteristics to be measured;

2. choosing a suitable method to measure these characteristics;

3. collecting the data;

4. analysing the collected data;

5. making decisions on the basis of the results; and

6. implementing those decisions.

Thus, performance measurement encompasses steps (2–4). It may involve measuring how the system works (i.e. process measures) or what it achieves (i.e. outcome measures).

Process measures assess the general operation of the system. Here, the focus is on the ability of the system to conduct its work. This may entail examining whether the necessary resources exist (e.g. funding, staff and equipment) to achieve the planned activities. A qualitative method is likely to be the most appropriate technique.

Outcome measures assess the changes or benefits that result from the activities of the system. Typically, quantifiable measures are used to determine whether the system has achieved a certain level of performance.

In the present context, the aim is to understand both whether the telemedicine network is working as intended and what effect it is having. Therefore, both process and outcome measures are required (even though outcome measures may be difficult to obtain in practice and need time before the effects can be measured). The choice of performance measures may be facilitated by considering a telemedicine network as a ‘black box’.

A black box is a device or system that can be viewed solely in terms of its input, output and transfer characteristics without any knowledge of its internal workings. Almost anything might be referred to as a black box, e.g. a transistor, an algorithm or a hospital. In the context of telemedicine network performance, the network (black box) has an input (the queries that are made) and an output (the answers provided), see Fig. 1. So that the black box can perform its function, resources have to be supplied. In the present context, these resources (principally the experts) are controlled by one or more telemedicine coordinators.

Fig. 1.

A telemedicine network viewed as a black box.

The actors

For the purpose of the present article, we describe the characteristics of a particular form of telemedicine called teleconsultation. The service is provided via a network operated by volunteers, although the principles are also relevant in networks operated by staff who are paid to do the work. Thus, we define a teleconsultation as taking place when:

1. a requester, such as a referring doctor, requests information from an expert. In practice, the requester makes the request to the telemedicine network, rather than to a specific expert;

2. a telemedicine coordinator receives the request, evaluates it and decides which would be the appropriate expert to answer it. (In the open model, the coordinator may allocate the request to a group of experts.) The coordinator then notifies the chosen expert that a query is waiting to be dealt with. Each request may generate multiple queries, for example if the first specialist to whom the query is sent is not available, then the query will need to be sent to another. In practice, because the coordinator is likely to require both medical and IT skills, the coordinator is often supported by one or more assistants; and

3. an expert, such as a medical specialist, receives a request for information and (hopefully) responds to the requester.

Thus, the present article is concerned with three types of persons (requesters, coordinators and experts) and how to measure their performance, although not in a punitive way.

Measurement perspective

In attempting to measure performance, it is helpful to consider whose perspective should be used. This is because the desirable performance from the point of view of the requester may be very different from that of the expert, or the coordinator. For example, the requester might hope for an immediate reply, whereas the specialist – who is likely to have many other demands on his or her time – may wish to deal with requests at leisure.

User perspective. The first perspective is that of the network users, who all have the same desire: a reliable and easy to use system, for which the minimum of training is required. Thus, the requester would like:

1. a rapid answer;

2. a definitive response to the questions posed (‘definitive’ meaning comprehensive, appropriate and adapted to the local context);

3. the possibility of discussing the case with the specialist, if the answer received is not definitive; and

4. additional information, if relevant, such as references from the literature.

The coordinator would like:

1. a ‘clean’ referral, that is one without poor quality pictures or information contained in files of proprietary format. There may also be a requirement to receive carefully anonymised referral data if confidentiality is critical, as in humanitarian operations in war zones;

2. access to a panel of experienced and available specialists, covering all sub-specialities; and

3. a rapid response from the specialist, even if the response is only that a different specialist will be required.

The expert would like:

1. only appropriate referrals, that is requests that fall within his or her areas of expertise;

2. sufficient information about the patient to permit an appropriate response to be provided;

3. information about the local context to provide a reply that is well adapted to circumstances; and

4. feedback from the referrer about the reply and/or subsequent follow-up data about the patient.

Societal perspective. The other perspective that is relevant is that of society as a whole. Telemedicine is not simply a tool but a procedure allowing interactions between a network and the local health care systems. So, to measure the impact of the network, we must take into consideration the context, Fig. 2. Telemedicine may create a new way of accessing health care, for example obtaining a second opinion in remote areas or in a humanitarian setting, where the population often has no access to specialist physicians. This is a completely different matter from telemedicine that improves or optimises an existing health care system, for example telemedicine projects set up in industrialised countries. The distinction is important because it has direct consequences in terms of evaluation: in the latter case, a comparison between face-to-face and telemedicine consultation can be done, for example, whereas in the former case, this is not possible because there is no previously existing reference system.

Fig. 2.

Overview of a telemedicine system. Telemedicine networks are not isolated but located within larger health organisations that in turn are part of the general environment.

The societal aims will, therefore, be a telemedicine network that provides clinically useful information at a ‘reasonable’ cost. That is, the use of telemedicine will contribute to strengthening the local health care system.

Patient's perspective. Finally, the patient's perspective should not be forgotten, as patient care is the ultimate goal of any telemedicine service. Even if the patient and requester both aspire to obtain the best possible health care management, their perception of what is best can differ. The doctor can take advantage of expert advice (e.g. from an educational point of view), whereas the patient would find no benefit at all. Patient satisfaction must, therefore, be assessed independently, and this requires a specific qualitative approach. Despite these potential differences between the views of requester and patient, we consider in the present context that the patient's perspective and the requester's perspective are tightly linked.

The framework

There is a wide range of indicators and metrics that might be relevant to the measurement of performance. Indicators ‘indicate’ impact but they do not attempt to quantify that impact, whereas metrics are ‘numerical indicators’ that allow the impact to be quantified. A combination of indicators and metrics is, therefore, needed. Because of the complexity of the environment in which telemedicine networks operate, both quantitative and qualitative methods are needed, and a multi-disciplinary approach is required for analysing results. In particular, a qualitative approach allows a deeper understanding of system use and, therefore, a better measurement of its true impact and so an improved assessment of quality.

As far as the performance of the network as a whole is concerned, important metrics include:

1. the average speed of response to requests, that is, the average delay before a requester receives a definitive reply; and

2. the cost of providing the service, for example, the coordinator's time, the infrastructure required.

As far as performance of the network for the individual users is concerned, important metrics include:

1. the ‘value’ of the answer provided, for example, was it relevant (in accordance with the most up-to-date medicine adjusted by the expert's experience)? Did it help the patient? How long did it take? Did significant knowledge transfer occur between requester and expert (but also for the coordinators)?

Although a large number of possible indicators and metrics might be employed to answer these questions, in our experience five measures in each of the three user groups will suffice, that is, after discussion, it appears that these five measures capture the majority of the information required.

Suggested performance measures

There are basically five characteristics of the system that a requester needs to know about, in advance of submitting a new referral:

1. the rate at which queries are being submitted to the network (which tells the referrer how busy the network is and how well it is working);

2. the proportion of failed queries (which tells the referrer what the chance of getting a reply is);

3. the time to reply (which tells the referrer how long they will have to wait);

4. the quality of the reply (which tells the referrer how good the replies are). The definition of ‘quality’ will include how useful the information is to the requester, how well adapted it is to the requester's local situation and also whether it follows the most recent guidelines as well as the educational value of the answer (knowledge transfer); and

5. how easy the system is to use.

The things that a coordinator needs to know are:

1. How many questions will need to be handled? (Strictly speaking, what is the rate of arrival of requests for me to deal with?)

2. How much time and effort will this entail?

3. What resources are available to me? For example, how many experts are there, and what subject areas do they cover? What will I do if I do not have an appropriate expert for a particular query? Note that this leads to the question whether a more effective form of global telemedicine would be a network of networks. In other words, should there be a single, high-level clearing house for receiving telemedicine queries, from which queries could be directed to the participating networks as appropriate?

4. Will I be able to obtain feedback from the users, for example, from the experts? Will I be able to obtain feedback on patient outcomes?

5. How easy is the system to use? Are there likely to be IT problems and if so, how will these be solved?

The things that an expert needs to know are:

1. How many queries will I receive? That is, what is the rate of work requests that I can expect?

2. How much time and effort will be required to answer them? (Note that responding to a single query about a clinical case may be much less demanding than answering a generic query that may require providing the referrer with an informal summary of the literature.) How much time do I have to respond to the query?

3. Will the queries be relevant to my area of expertise?

4. Will I receive feedback about the patient outcomes?

5. How easy is the system to use? For example, can I notify periods when I am not available to answer requests because of absence due to holidays or conferences?

Some of the above items are aspects of quality, including the technical quality of attached images.

Finally, from the societal perspective, the aspects of system performance that are important are:

1. Is the network clinically effective? That is, does it improve patient outcomes?

2. Is the network cost-effective? That is, is it an appropriate use of scarce resources?

3. Are the network and its participants properly integrated into the relevant health care organisations?

Use of the framework

To illustrate the use of the framework for measuring network performance, we applied it to the networks of our own experience. The authors are responsible for the operation of three telemedicine networks in different settings (low- and middle-income countries). The three networks all provide teleconsultation services, that is, they have a clinical purpose and have been in operation for periods of 7–10 years, see Table 2.

Table 2.

Three second opinion networks that have operated for periods of 7–10 years

1. Swinfen Charitable Trust network

This service is provided by a UK charity and started operations in 1999. It provides advice to doctors in developing countries using web-based store-and-forward messaging. In 2009, there were approximately 300 referrers from 45 developing countries. The advice is provided by a panel of about 400 consultants who donate their expertise to the charity. In the first 10 years of operation, the network handled approximately 2,000 telemedicine cases.

2. Ukrainian Teleconsultation network

This service started in 2000. This network is based in an expert trauma centre in Donetsk and provides treatment advice to seven community hospitals in the Donetsk region and four regional centres in the Ukraine using web-based store-and-forward and low-cost videoconferencing (Skype) methods. In the first 10 years, more than 700 teleconsultations were conducted in trauma and orthopaedics.

3. Institute of Tropical Medicine HIV/AIDS Telemedicine network

This service is based at the Institute of Tropical Medicine in Antwerp and started in 2003. It provides second opinion advice about the use of antiretroviral therapy and AIDS care delivery using email and web-based store-and-forward methods. Expert advice from a network of 20 specialists in HIV/AIDS is offered to health care workers in resource-limited settings. In the first 7 years, the telemedicine service provided over 1,000 teleconsultations to health workers in more than 40 countries, most located in sub-Saharan Africa.

Previous evaluations of the three networks

Each of the three networks has been evaluated at various times during its operation:

Swinfen charitable trust network. Type of evaluation conducted: Regular audits of system activity have been performed. Other forms of evaluation have included surveys of user satisfaction, analysis of submitted image quality and follow-up of patients.

Main results. Audits of system activity show that the time required to provide a specialist's response has remained at 2 days or less over 10 years of operations (5–7); surveys of user satisfaction show that the referring doctors who responded made positive comments about the service and half said that it improved their management of the case (8); analysis of image quality showed that although average image size increased over a 4-year period, the quality of the submitted images did not (9); follow-up of patients showed that telemedicine had assisted with the diagnosis in all cases and the clinical outcome for the patient was considered to be good in 15 of the 44 cases10.

Ukrainian teleconsultation network. Type of evaluation conducted: A retrospective cohort study (n=210) was carried out in 2007–2009. The aim was to define the quality of the teleconsultations and their influence on clinical outcomes. Patients were divided into three groups (telemedicine and two control groups) (11).

Main results. The majority of teleconsultations (92%) had high relevance. The experts’ recommendations had a real influence on the clinical strategy in 81% of the cases. There were more good clinical outcomes in the telemedicine patients studied than in a group of control patients (66% vs. 31%). An evaluation showed that telemedicine was associated with a sixfold increase in the odds of good clinical outcomes for trauma patients.

Institute of tropical medicine HIV/AIDS telemedicine network. Type of evaluation conducted: A web-based survey was sent to network users in 2006 and in 2008 to assess their perception of the system and its value in establishing the diagnosis, changing the management of care, providing reassurance to the referring doctor and any educational benefit (12).

Main results. The referring clinicians reported that the service was beneficial in several ways: Establishing the diagnosis (42%), clinician's education (27%), literature provision (14%) and for reassurance about decisions made prior to the consultation (12%). The service was judged to have been useful in influencing the management of the patients in 100% of cases. Most users (88%) reported that the service had significantly influenced the way they managed their cases and also other similar cases subsequently (13).

Use of the performance framework

The performance framework described above was applied to the three networks, using data available to the coordinators. The results are summarised in Table 3.

Table 3.

Performance of three telemedicine networks

	SCT		Ukraine		ITM
Performance measure	Comment	Value	Comment	Value	Comment	Value
	Statistics for the year 2009		Statistics for the year 2007–2009		Statistics from April 2003–March 2009
Requester's perspective
1. Rate of query arrival (new cases)		19.4/month		5.8/month		15.5/month
2. Proportion of failed queries		0		0		0
3. Time to first reply from an expert	Median	17 hours		30 hours		24 hours
4. Quality of replies	Some evidence from a research project (10). Referring clinicians reported that in 100% of cases followed up (n=22), the advice was useful	Not measured routinely	Some evidence from a research project (11). The quality of replies was measured according to a specially developed scale (20). Ninety-two per cent of replies (n=132) were judged to be highly relevant	Not measured routinely	Some evidence from a research project (12)
					User satisfaction (38 respondents) measured on a scale 1–10. Utility of advice was rated 8–9 by 70% of the respondents	Not measured routinely
5. Ease of system usage		Not measured		Not measured		Not measured
Coordinator's perspective
1. Rate of query arrival	New queries, plus those requiring additional responses	37.7/month	New queries	5.8/month	New queries.	15.5/month
2. Time required		Not measured		15–30 min		Not measured
3. Resources available		406 experts available		Five experts available		16 experts available
4. Feedback from experts/feedback on patient outcomes		Not measured	Some evidence from a research project (11). Feedback was available on the relevance of the teleconsultation for 63% of the cases (n=132)
			Feedback was available on patient outcomes for 33% of the patients (n=132)	Not measured routinely		Not measured
5. Ease of system usage		Not measured		Not measured		Not measured
Expert's perspective
1. Rate of requests received (for those experts who received queries)		2.8/year per expert		5.4/year per expert		9.9/year per expert
2. Time required to answer	Median time from allocation to response	11.0 hours		30 min		Not measured
3. Relevance to own expertise		Not measured		Not measured		Not measured
4. Feedback on patient outcomes	Some evidence from a research project (10)	Not measured routinely	Some evidence from a research project (11)	Not measured routinely		Not measured
5. Ease of system usage		Not measured		Not measured		Not measured
Societal perspective
1. Clinical-effectiveness	Some evidence from a research project (10). Referring clinicians reported that the telemedicine advice was useful in 21 of 22 cases followed up, and the outcome for the patient was good in 15 cases	Not measured routinely	Some evidence from a research project (11). Referring clinicians reported that telemedicine produced satisfactory outcomes in 22% of cases (n=132) and good or excellent outcomes in a further 76%	Not measured routinely	Some evidence from a research project (13). Referring clinicians reported that telemedicine was useful in influencing the management of their patients in 100% of cases (n=37)	Not measured routinely
2. Cost-effectiveness		Not measured		Not measured		Not measured
3. Integration into the health care system, for example, involvement of local people		Not measured		Not measured		Not measured

Recommendations

The following analysis is based on the experience of running three second opinion telemedicine networks for periods of 7–10 years and on an evaluation of the performance of the three networks using the proposed framework, that is, the aim is to offer evidence-based advice about practical implementation.

Common features of the three networks

Although the three networks are different in terms of their activity, their size and their objectives, their performance in certain respects is very similar. For example, the time to first reply from an expert is about 24 hours for each network, see Table 3. However, in illustrating the use of the framework to assess the performance of the three telemedicine networks, it became clear that only some of the information required was in fact available. All networks had systems in place to collect certain data from the user perspective. None of them collected information about the coordinator's time required or about ease of system usage (for any of the three user types). They had only limited information about cost and quality. We recommend that networks collect such information routinely.

System operation and usability

Information about the coordinators’ time was not collected by any of the networks. Nor was information collected about usability. Measuring ease of system usage requires data collection directly from the field to see how the system works in practice from the perspective of its users. This requires a qualitative approach, based on well-defined tools such as observation (participative or not), interviews (in depth, semi structured or guided discussion) or scale/questionnaires (14). Considering the major impact of the environment on system use, only such a pragmatic approach can provide insights into potential causal mechanisms of dysfunction and, therefore, generate hypotheses to improve the system. Mixed methods have been used successfully in outcomes research in other disciplines (15) and, therefore, seem appropriate in telemedicine as well. We recommend that data on system operation and usability be collected routinely.

Measurement of cost

The issue of cost is less important in networks run by volunteers. However, estimation of costs is likely to be required in future operations where staff are reimbursed. Clearly, rigorous cost-effectiveness studies are best performed by an expert, for example, a health economist16, 17), but some estimation of cost seems essential in the context of a telemedicine operation that relies on technology. A recent review of telemedicine in developing countries3 showed that only a few articles provided such data although in the context of scarce resources, the appropriate allocation of resources is crucial (i.e. the consequences of wastage that will have little effect on health care in industrialised countries may have a profound impact in low-resource settings). Cost details also provide global information on the system, which can be useful for comparisons with other systems. We recommend that cost data be collected routinely.

Assessment of quality of replies

The quality of the replies was not systematically assessed by any of the three example networks – nor has it been reported more generally in other telemedicine networks – despite being fundamental to the main value of a telemedicine system. Content analysis of replies needs to be performed by independent experts, and ideally within the framework of a multi-disciplinary evaluation (medical, sociopsychological and anthropological) to take fully into account the environment and its impact on patient management. This kind of data collection has to be planned beforehand, which in turn emphasises the need for a standardised framework when setting up a new project.

User satisfaction, including patient feedback is another aspect of the quality of the replies. Simple and reliable tools are available, although they may need to be adapted to suit the specific system being studied. The use of a numerical scale by Zolfo et al. (13). – that was based on the visual-analogue scale used for assessing pain – represents a useful example of the adaptation of validated tools for assessing user satisfaction. We recommend that quality of service data be collected routinely.

Societal perspective

As well as an absence of data about the user perspective, the three networks collected very little information about network performance from the societal perspective. In a study of telehealth adoption in Quebec hospitals, Gagnon et al. (18) found that the local context was a crucial factor in long-term success. Understanding the local context requires a field approach and feedback from local users. Close collaboration with local national health care workers is necessary to make sure that any proposed telemedicine system will address real needs and meet local priorities.

Only limited data were available on clinical effectiveness and none at all on cost-effectiveness. Measuring health outcomes (e.g. mortality and morbidity, diagnostic accuracy) is difficult in resource-limited settings, especially in situations where health is influenced by a range of factors that are difficult to control for. Measuring cost-effectiveness is, therefore, likely to be extremely difficult, although some limited information about costs might be obtainable. We recommend that data be collected routinely to enable measurement of clinical- and cost-effectiveness.

Implementation recommendations

Based on our experience, the targeted model of telemedicine network operation appears to be successful and sustainable. The three networks we have used to illustrate the proposed framework are different in terms of their activity, their size and their objectives, but their performance in certain respects is very similar. On the other hand, we cannot conclude that the targeted model is superior to the open model without further evidence such as performance measurements from a wider range of networks.

Discussion

A telemedicine network can be regarded as an example of a ‘diffuse technology’, rather like a digital picture-archiving system (PACS). As was pointed out during the evaluation of early PACS networks, there are ways of combining individual studies to arrive at strong evidence about the impact of a diffuse technology (19). For example, information from individual case studies can be combined, so that even if they represent relatively weak evidence individually, the aggregated value is strong. Thus, two or three separate case studies based on different data sources that show similar effects will help to confirm the value of a diffuse technology. Our hypothesis is that stronger evidence about the value of telemedicine networks in the developing world can be obtained from studies of multiple networks than from a study of any network alone.

We have proposed a performance measurement framework for assessing telemedicine networks that provide teleconsultation services to doctors in resource-constrained settings, although the principles are more general and could be applied to other telemedicine networks as well. The suggested performance measures for the three types of users show certain common, cross-cutting themes. All users are concerned with:

1. Access (how busy is the network? what will my workload be?).

2. Quality (how good are the responses? can feedback be obtained about patient outcomes?).

3. Usability (how easy is the system to use? what will I do if there are technical problems?).

Similar themes can be seen in the societal perspective.

The framework is based on a core set of indicators and metrics. We have illustrated its use to assess the performance of three mature telemedicine networks. The results help us to understand whether these telemedicine networks are working as intended and the effect that they are having.

Telemedicine is a well-known minefield. It is a recipe for disaster to begin a pilot project without having a satisfactory method for evaluating network performance. We recommend that everyone uses a common framework. This will facilitate comparisons between networks. More important, it allows those responsible to make longitudinal comparisons within their own networks, for example, to assess the effects of improvements. We also recommend that all networks collect basic information about their costs.

Conclusion

Measurement of performance is a fundamental aspect of network evaluation, yet there is no existing framework in the literature that is relevant in the present context. Conceptualising a telemedicine network as a ‘black box’ allows the identification of important indicators and metrics that are common to the different perspectives. While examining these different perspectives (user and societal), we propose a framework for performance evaluation based on five main items from each perspective. This framework was applied to three different telemedicine networks, and certain common performance characteristics emerged. It also became clear that some important data were not measured at all, possibly because acquiring this information would require a specific field approach.

For a comprehensive evaluation that takes into account the environment, mixed methods are required, which combine both quantitative and qualitative methods. Cost-effectiveness analysis must be performed by an expert, but all telemedicine systems should at least provide information about setup and running costs.

The present article describes the performance of three different telemedicine networks, operating in low- and middle-income countries. The performance framework is based on our experience. No telemedicine network should be started without having evaluation built in. Our suggested framework will permit an organisation to understand both whether the telemedicine network is working as intended and what effect it is having.

Conflicts of interest

The authors have no conflicts of interest, such as financial or personal relationships, that could bias the work reported here.