Abstract
Evaluating the success of e-health initiatives and programmes means looking beyond the number of users to their impact on the people and this, Claire Keeton reports, is difficult to measure.
If you live in South Africa and need some advice on the human immunodeficiency virus (HIV), all you have to do is pick up a mobile phone and connect with a counsellor through “Mxit”, an instant message chat-line. This service, run by the Cape Town-based organization Cell-Life in partnership with counselling service LifeLine, reaches about 10 million people.
“But it is hard to measure impact,” says Katherine de Tolly, m-health senior researcher at Cell-Life, a not-for-profit organization that makes technology and services accessible to try and improve health, “mobile usage is different to impact”.
“We’ve looked at the stats, analysed exchanges, done research reports and we know the users like it and use it. But to measure the impact on health outcomes, such as preventing HIV infection or alleviating depression, takes a level of research that we’ve not been able to afford.”
Mobile technology is expanding rapidly in low- and middle-income countries, particularly in Africa and in the less affluent and rural communities mobile access is far greater than access to computers.
Mobile phones can help people in remote communities access health-care information
Cell-Life
In South Africa for instance, de Tolly says, 50 million people have mobile phone access and everyday experience confirms near-universal mobile ownership, with everyone from street vendors to top-level executives carrying one.
Yet many countries in Africa still struggle with access to computers, the Internet, a reliable power supply and other infrastructure challenges.
Vincent Shaw, executive director of the South-Africa Health Information Systems Programme (HISP), a nongovernmental organization that specializes in the development and maintenance of health information systems, recalls how the HISP lost data in a project in Nigeria when the power failed and all the stored mobile phone text messages were deleted after 24 hours.
In developing countries, health technology via mobile phones helps to overcome the lack of universal and reliable access to information and communication technology, just as the use of desktop computers will persist until bandwidth and internet access improve, says Shaw.
“We have seen that Internet access is not that great, including in South Africa, and have stood firm on the need for stand-alone computers rather than pursuing web-based data entry systems.
HISP – which has university partners in Norway and Sweden – is active in 15 countries and its free web-based software has been translated into nine languages including Chinese, Russian and Swahili.
Shaw says: “Measuring the success of e-health is not just about value for money but also about the development of individuals and communities.”
“Measuring the success of e-health is not just about value for money but also about the development of individuals and communities.”
Vincent Shaw
An example of this, says Shaw, was when the HISP team was giving computer training in South Sudan to people who would be working with the new health information system.
Implementing the HISP software has already made a difference, says Shaw. “South Sudan has been ravaged by war – and had little or no health information in its health-care system – but it is reporting health data on a regular basis now.”
He says: “We estimate about 1.5 billion people are accessing e-health services across Africa and Asia, according to our research. They may be paper-based at the start but at some point the information is stored in computers.”
“We can see the numbers – who is reporting and who is not – but at what percentage do you peg success?” Shaw asks.
He says that one measure of the success of such systems is that they make data available, so that they can be used to allocate resources more rapidly than before. This means that data are available to managers in health-care systems within a short space of time, between one and three months, so that they are able to assess services using more recent data, rather than older data from six to 12 months ago.
Data, when interpreted by health-care workers, can alert health departments to risk and priority areas, where more resources may be needed. However, data need to be reliable to have a concrete impact on health planning.
Lyn Hanmer, of South Africa’s Medical Research Council, explains: “Health workers must trust the data or they will use another information system.”
Hanmer also cautions that e-health systems must meet the needs of the health workers and not add to their workload.
She did a study to find out whether e-health systems in hospitals in two provinces in South Africa were working and found that two systems were reasonably successful, while a third was not.
The study included data from 30 hospitals and there were 70 individual respondents.
In the study, a model of factors associated with the success of these systems was developed. These included system performance, perception of usefulness, management commitment to system success, and user knowledge and understanding of the system.
A survey of users was conducted and respondents were asked to rate the system which they were using in terms of these factors. In addition, users were asked whether they thought that the system was successful in supporting their work.
The study showed that the perception of success among respondents to the three systems stood at 97% for one, 100% for another and 37% for another.
The results were presented in a paper to the Medinfo 2010 conference held in Cape Town.
Marc Mitchell, president of Africa-based nongovernmental organization D-tree International, identifies a challenge that is specific to e-health systems: their design can be unduly influenced by technological wizardry.
He says: “The critical question is: ‘Does this improve health?’ A lot of people in the field flock to e-health because of the technology, hype, bells and whistles and money. It is essential that health people keep the focus on health priorities.”
“A lot of people in the field flock to e-health because of the technology, hype, bells and whistles and money. It is essential that health people keep the focus on health priorities.”
Marc Mitchell
Another focus for e-health projects needs to be how to serve individuals, for instance, by connecting patients to health workers through mobile phones.
Text messaging on mobile phones has played a major role in promoting adherence to treatment for HIV/AIDS and tuberculosis treatment and in reducing patient loss–to-follow-up in countries across southern Africa.
In a study on HIV testing, Cell-Life found that text messaging can make recipients change their behaviour. Designed by de Tolly and the health and research society unit at the University of Stellenbosch, the study tested whether sending text messages that encourage people to go for HIV testing had any impact.
“We found that 10 motivational SMSs did have a statistically significant impact on getting people to test for HIV,” says de Tolly, adding that researchers also found that recipients were nearly twice as likely to get tested for HIV following the study, which was published in the journal Telemedicine and e-Health in February of this year.
Another study tried to evaluate whether e-health, in this case mobile phone (m-health) technology can contribute to better quality care.
In a D-tree study, which is under review, a trial was conducted in the United Republic of Tanzania to find whether using an electronic version of the chartbook for the Integrated Management of Childhood Illness (IMCI) resulted in improved adherence to the protocols and better treatment. Health workers access the electronic version by cell phone or other mobile devices and it guides them through each step of this approach to assessing a child.
Increasingly health officials and political leaders are recognizing the benefits of e-health technologies.
South Africa’s deputy minister of health, Gwen Ramokgopa, recently acknowledged that information and communication technologies “have a huge role to play in extending the reach, improving the quality and reducing the cost of health care in South Africa”.
The South African Department of Health is currently reviewing its e-health strategy, which has been developed over the past six years but is yet to be implemented.
Shaw observes: “More and more national governments are using e-health systems and starting to get the numbers together to measure the work they are doing.
“Obviously the question is whether the data are accurate but that’s not the most important issue: we need to start by collecting the information and have databases so that we can ask further questions and make sure e-health solutions are appropriate to the context.”
The speed at which technology evolves, health workers’ lack of the necessary skills to use e-health technologies, the infrastructure and cost are among the challenges to measuring e-health initiatives. “Technology changes quickly and we can’t go that quickly in our studies,” says de Tolly.
Vincent Shaw says measuring the impact of e-health is not just about value for money
Claire Keeton
Furthermore, overextended health workers, who are trained to provide health care rather than analyse numbers, can be slow to embrace technology. The pace at which technology evolves is another reason why pilot projects are essential. De Tolly says: “A lot of this is really new. At the beginning of a study you learn what you must do to adjust the project for it to work.”
While evaluations of the role of e-health – including mobile – technologies in health care are becoming more common, given the explosion in mobile phone usage alone, its impact remains difficult to measure.