Evaluation of an Android-based mHealth system for population surveillance in developing countries

Abstract

Objective

In parts of the developing world traditionally modeled healthcare systems do not adequately meet the needs of the populace. This can be due to imbalances in both supply and demand—there may be a lack of sufficient healthcare and the population most at need may be unable or unwilling to take advantage of it. Home-based care has emerged as a possible mechanism to bring healthcare to the populace in a cost-effective, useful manner. This study describes the development, implementation, and evaluation of a mobile device-based system to support such services.

Materials and Methods

Mobile phones were utilized and a structured survey was implemented to be administered by community health workers using Open Data Kit. This system was used to support screening efforts for a population of two million persons in western Kenya.

Results

Users of the system felt it was easy to use and facilitated their work. The system was also more cost effective than pen and paper alternatives.

Discussion

This implementation is one of the largest applications of a system utilizing handheld devices for performing clinical care during home visits in a resource-constrained environment. Because the data were immediately available electronically, initial reports could be performed and important trends in data could thus be detected. This allowed adjustments to the programme to be made sooner than might have otherwise been possible.

Conclusion

A viable, cost-effective solution at scale has been developed and implemented for collecting electronic data during household visits in a resource-constrained setting.

Traditional healthcare delivery paradigms in which patients present to healthcare providers for care have proved inadequate in many developing country settings. In one observational study from Nigeria, over half of pregnant women delivered outside of hospital facilities and 82% of these women had no skilled attendants at delivery.

In Ethiopia, over half of patients diagnosed with pulmonary tuberculosis waited an average of 30 days before presenting to a public health facility.^{1 2} Even worse, many patients who have particular medical conditions never get diagnosed (or receive late diagnosis) simply because they rarely interact with the healthcare system. As an example, it is estimated that nearly 80% of HIV-infected adults in sub-Saharan Africa are unaware of their status.³ The problem of accessing healthcare in these settings is twofold: ‘On the supply side, good quality, effective healthcare may not be offered. On the demand side, individuals may not utilize services from which they could benefit.’⁴

As an alternative to waiting for patients to present to health facilities for screening and care, many healthcare delivery systems in developing countries have initiated efforts to get care out of the physician's office and into communities. One such care system is the USAID–Academic Model Providing Access to Healthcare (USAID–AMPATH) Partnership⁵ in western Kenya. This programme has embarked on an effort to conduct home visits to all two million individuals in its catchment area. The goal of the home-based counseling and testing (HCT) programme is to collect basic health information, offer rapid HIV testing, collect sputum for individuals at risk of tuberculosis, and offer focused care services when needed. Most HCT tasks are performed by several hundred government-trained community-based health workers. Data from the HCT encounters were initially collected using personal data assistants (PDA) physically tethered to separate global positioning system (GPS) units. The community-based health workers who used this initial data collection system felt that this solution was faster, easier to use, and produced higher quality data than using paper-based data collection forms.⁶ These findings are comparable to those from previous studies comparing PDA and paper-based systems.⁷

Despite its advantages over a paper-based system, this initial data collection approach had several significant shortcomings. First, although costs were significantly lower than paper-based data collection methods the costs were still substantial.⁶ Second, the data collected could not be directly integrated into the electronic medical record system, which was already in use at the USAID–AMPATH clinics—integration required dedicated time by several experienced data managers. Third, the cable connection between the PDA and GPS devices was not always reliable, and GPS information occasionally had to be entered manually into the PDA devices. Fourth, the use of the proprietary Pendragon forms software (Pendragon Software Corporation, Illinois, USA) on the PDA devices limited flexibility to incorporate some functionality into the data collection software—some of this functionality included advanced barcode scanning and check digit algorithms.

Objectives

In response to these shortcomings, we developed a second-generation tool with the following design goals:

• The system must be reliable in resource-constrained settings.

• The system must be scalable to support the desired scope of the HCT programme.

• The system must be developed in an open source methodology to lower the cost of future implementations.

• The system must work on a variety of handheld devices to facilitate the selection of hardware with the least cost for future implementations.

• The system must be implemented on a device that has inbuilt GPS capability to improve reliability under difficult operating conditions.

• The system must seamlessly integrate with an open source medical record system so that the tool can be extended to other settings in the future.^{8 9}

We hypothesized that revising our implementation with these goals would yield reduced costs at comparable effectiveness and end-user satisfaction to our previous efforts. In this paper, we present the development, implementation, and evaluation of this system.

Our goals in creating this system remained the same as our initial work—to develop a mobile device-based tool that would guide community health workers through the process of screening the population for the presence of HIV and other important health risks in their own homes. The tool works by providing structure to the interaction between the community health worker and the persons being screened. By using mobile devices, we improve on this process by facilitating the direct collection of several key pieces of information, such as GPS location of the screening. We also facilitate direct, digital capture of the information collected, which greatly facilitates analysis and review.

Materials and methods

Background: setting

The HCT programme is being conducted in western Kenya, within the catchment area served by the USAID–AMPATH Partnership. This partnership is composed of Moi University School of Medicine and a consortium of North American universities led by Indiana University School of Medicine. The partnership has provided comprehensive HIV care to individuals in western Kenya since 2001 and is now transitioning to providing comprehensive primary care.⁵ The HCT programme, including the research reported in this brief, were approved by both the Institutional Review Board at Indiana University as well as the Independent Review Commission of Moi University in Kenya.

Development: the HCT system—technical details

We selected the Android operating system (Google, Inc, Mountain View, California, USA) for mobile devices because of several factors. Android was chosen because it is open source. Android is now the most popular smartphone operating system and runs on the widest selection of smartphones whose costs are rapidly declining.¹⁰ To implement the HCT mobile data collection functionality, we took advantage of an existing Android-based tool called ‘Open Data Kit (ODK) Collect’¹¹ developed by members of this project's team. ODK Collect is freely available on the Android Market, uses forms written in the XForms standard, and displays these to users on mobile devices running Android. ODK Collect can support multiple types of questions including text entry, multiple choice, and check boxes. In addition, it can easily handle multiple data types such as text, GPS location information, photos, videos, audio, and barcodes. Functionality within ODK Collect can also be extended to take advantage of thousands of other available Android applications.

The Xforms created for the HCT programme allowed the collection of household information (including GPS), individual demographics, HIV and tuberculosis histories, testing result information, and follow-up appointments. Contents of these forms were based on direct input by the end-users, with particular attention paid to the workflow for the health workers. The HCT application used almost all ODK Collect's functionalities, and had user authentication, branching logic, and basic decision support. We used ODK Collect's extensibility by incorporating a customized check digit algorithm for patient IDs. In addition, we used a freely available barcode scanning application from the Android Market to scan individual ID card barcode numbers with the smartphone's camera, and incorporated these into the collected HCT data. The HTC Dream (HTC America, Inc, Bellevue, Washington, USA) smartphone was used for the initial implementation as it had a GPS system and a camera. We are currently in the process of implementing the Huawei IDEOS (Huawei North American Headquarters, Plano, Texas, USA).

Data collected on the smartphones were exported into an instance of OpenMRS. OpenMRS is a freely available, Java-based, open source electronic health record system (EHR) that can be installed on many different operating systems including Linux, Windows, or Mac OS X.⁸ OpenMRS has been in use at USAID–AMPATH clinics since 2006, and is currently also deployed in over 15 other African countries.¹² HCT data collected on smartphones were transferred as XML files into OpenMRS using an existing XForms module within the EHR. Synchronization of the mobile HCT programme and the EHR was done by the community health workers, who used direct USB or wireless network connections—functionality for wireless network transmission was available in our application but was not used because of cost and security considerations. The downloaded data were stored in a specific directory in the OpenMRS server's file system. We built a custom OpenMRS module that scanned this directory to determine if any new XML data files had been downloaded. Data contained in the identified files were then parsed and stored in relevant tables within the OpenMRS database—either as person or location attributes, or as observations that corresponded to particular concepts in USAID–AMPATH's concept dictionary.

Development: usability testing

We conducted two rounds of formal field-based usability testing of the application. In the first round, we involved five pairs of two community counselor end-users—Nielsen has shown that approximately 80% of usability issues can be uncovered with approximately five subjects, with diminishing returns using more subjects.¹³ One member of this pair was given an Android-based device with the HCT software installed, while the other member used the existing PDA/GPS solution. These pairs were joined by two members of the investigation team—a member with clinical background and a member from the application development team. The pairs of end-users conducted HCT home visits for 1 day, at which time the investigators took notes on their actions and verbalizations (‘think aloud’ technique). The goal of these notes was to assess efficiency and satisfaction with the new tool. These notes were analyzed for critical incidents (ie, usability issues) and difficulty completing tasks. Conclusions regarding efficiency and learnability were drawn from erroneous assumptions and actions, and from tasks taking longer than anticipated. At the end of the session, a brief semistructured interview and a group discussion were conducted. These discussions focused on the usefulness and effectiveness of the new tool as well as user satisfaction. The content of these interviews covered the strengths and weaknesses of the Android-based system when compared with the existing PDA/GPS solution. End-users were also invited to contribute corrections and comments on the interface design.

Both investigators reviewed and agreed upon their assessments before their delivery to the application development team. Major usability issues uncovered during this assessment were addressed before deployment of the application. After we incorporated feedback from the first round into the application, we conducted a second round of usability testing and debriefing interviews in a similar fashion to the first round, and with the same number of providers. The second round of testing focused on making the interface more intelligent and user friendly.

Implementation: data collection

Before deployment of the Android-based application, formal training of end-users and data managers for the HCT programme was conducted over 2 days. Each end-user was assigned responsibility for a specific device. These end-users carried the devices with them during home visits and recorded all data into our customized HCT forms after seeking consent from each individual. Collected data were synchronized to the OpenMRS server as described above.

Implementation: data security

To maintain the highest levels of security possible during this project, several measures were undertaken. Despite the capability for data transmission over wireless networks, we elected only to allow direct connections to our data repositories. Collected data were stored on the mobile devices in field, and after transmission to our servers these files were then deleted from the device's memory. Counselors were made personally reponsible for their devices and took great care to protect them from theft and loss. As of the time of writing, no devices have been lost or stolen.

Evaluation: user survey

To assess end-user attitudes towards the new Android-based data collection tool, we conducted an anonymous, self-administered survey of end-users who had used both the previous PDA/GPS solution as well as the current Android-based solution. Respondents were asked to evaluate the reliability of the solution as well as their satisfaction compared with the previously available tool.

Evaluation: cost calculations

We calculated costs of using the Android-based solution to allow for direct comparison with costs of using PDA/GPS devices (US$0.15 per individual) or a pen and paper-based system (US$0.21 per individual), which had been determined in a previous study.⁶ Our model also included estimates for device breakdown, development of the system, maintenance costs, and training.

Results

Usability testing

Several issues were uncovered during the first round of usability testing. End-users found the font too small and difficult to read in either high light (under direct sunlight) or low light conditions. They also felt that the navigation was not intuitive. Android devices acquired GPS coordinates more slowly than the previous solution, but the fact that end-users did not have to use a separate GPS device was a positive with the Android setup. Furthermore, providers felt that navigation between screens was not very intuitive. To address these problems, we increased the font size and taught the end-users how to adjust the contrast of the display with considerations of the battery life. We improved our programming to make GPS acquisition happen in the background, and to allow the ‘Enter’ key to make a selection and move to the next screen. The ‘Shift–Enter’ key combination returned to the previous screen. The changes were implemented before the second round of testing.

Feedback received from the second round of usability testing largely centered on features to make the interface more intelligent and user friendly. For example, end-users suggested that information about the administrative locations should be displayed in an order based on the location hierarchy (ie, sublocation, location, division, and district) and not alphabetically as we had done. The end-users also suggested adding several other functionality features, namely to: allow for direct navigation to the end of the form; to allow the user to save forms as in progress or incomplete; and to review saved forms before transmission. Based on the day's experiences, we also added options to keep track of households in which no one was found at home during the visit, and functionality to allow for return dates to be entered if the household requested a later date for interview. Finally, we improved the logic in many fields on the form such as disallowing non-sensible values for age, sex, or identification number. These changes were implemented for the final version of the Android application.

Data collection and analysis

Between March and August 2010, end-users visited 18 850 households in the Ainabkoi and Kesses Division in western Kenya to conduct HCT, with data from these encounters recorded using the Android-based application. End-users were allowed entry into 18 108 (96%) households, and they visited on average two to eight households per day. Counseling and testing was completed for 63 470 persons. All data were collected electronically by the developed system and were directly inputted into OpenMRS via a USB cable and its XForms module. The data were immediately accessible for query and reporting.

User survey

At the time of the survey, 70 end-users had used both the Android-based HCT programme as well as our previous PDA/GPS solution. The survey was administered to a convenience sample of 58 of these end-users during a weekly team meeting.

Compared to our previous PDA/GPS solution and on a five-point scale, end-users felt the Android system was faster (4.26±0.83), easier to use (4.43±0.81), and resulted in higher quality data (4.18±0.80). End-users felt using the smartphone sytem facilitated their interactions during home visits (3.98±0.88). Users felt that the training they received was adequate (4.21±0.87), and wished to continue using the Android-based system (4.47±0.83) compared with the earlier PDA/GPS system (table 1).

Table 1.

Responses to an 11-question survey posed to 57 community health workers of the HCT programme in western Kenya

Question (compared with using PDA/GPS system)	No of respondents for each option of a 5-point Likert scale (total no surveyed n=57)
	1	2	3	4	5
How reliable is the Android device in collecting data?*	0	2	5	14	30
How reliable is the Android device in collecting GPS coordinates?*	0	2	3	8	37
How do you rate the quality of data collected on the Android device?†	0	0	12	17	21
How easy is it to use the Android device to collect data?‡	0	2	4	15	30
How fast can you collect data on the Android device?§	1	1	5	20	23
Does the Android device make it easier or harder to interact with individuals?‡	0	2	4	18	17
Should end-users continue to use Android devices?¶	1	1	2	16	31
How easy is it to use the Android device to collect GPS coordinates?‡	1	1	2	9	38
Was your training on using the Android device good or poor?**	1	2	2	24	19
How much experience do you have using computers?††	10	0	30	3	7
How much experience do you have with Android devices?††	40	0	6	1	2

The survey asked them to compare two tools used for electronic data capture during their home visits. The two tools were an Android device using ODK Collect and a Palm device using Pendragon forms.

^*Scale: 1, very unreliable to 5, very reliable.

^†Scale: 1, much worse to 5, much better.

^‡Scale: 1, much harder to 5, much easier.

^§Scale: 1, much slower to 5, much faster.

^¶Scale: 1, definitely discontinue to 5, definitely continue.

^**Scale: 1, very poor to 5, very good.

^††Scale: 1, never used to 5, regular expert user.

GPS, global positioning service; HCT, home-based counseling and testing; PDA, personal data assistant.

Cost calculations

The HCT programme is tasked to visit two million people over the course of 3 years (760 workdays). To meet this goal, the programme must visit an average of 2565 persons per day. During the observation period, counselors were able to visit 10–16 persons per day using the Android-based system. For this calculation, we will take the conservative estimate of 10 persons per day. To meet our need, we must sustain 257 counselors with Android-based devices operating daily for the 3-year period. We will assume a 25% device breakdown rate as in our previous study. We thus require 322 Android devices over the 3-year period.

At the time of preparing this manuscript, the HTC Dream can be purchased for anywhere from US$250 to US$550. This was determined by using the Google search engine (http://www.google.com/products) with the keywords ‘HTC Dream’. The median cost determined from this search is approximately US$400. As all software used is open source and freely available, this figure represents the total cost per counselor. Based on the projection of the number of devices we require above, this results in a total cost of US$128 800. Our previous projected cost for the PDA/GPS solution was US$183 360. It should be noted that Android-based mobile devices are now available to us in Kenya for approximately US$150.¹⁴

The data collected on the Android-based devices will be synchronized with a desktop computer at each of the USAID–AMPATH clinics; based on our previous work we will assume 18 clinics and US$1500 for a desktop computer at each clinic (US$27 000 total). In implementing the Android-based solution, we required 9 months of support by a mid-level programmer as opposed to 1 month in our previous PDA/GPS solution. Personnel costs include 9 months programming time by a mid-level programmer in Kenya (US$1600 per month, US$14 400 total), 50% of an IT person's time (US$22 000 for 3 years), 50% of a data manager's time (US$22 000 for 3 years), and two dedicated data assistants (US$45 000 for 3 years). Training costs will come to US$1000 per day, assuming 2-day training sessions for counselors in groups of 70–80. The total for all fixed costs is US$120 400.

The total projected costs for this project are thus US$303 760. For each of the two million persons surveyed, the cost of using the Android-based solution is approximately US$0.15 per person. As in our previous work, we assume that the Android devices will not be resold or used for other purposes at the conclusion of this work. This cost can be compared with our previous estimates of US$0.15 per individual for a PDA/GPS solution and US$0.21 per individual for the data entry of a pen and paper system alone.

Discussion

Our implementation in western Kenya remains one of the largest applications of a handheld device-based system for performing clinical care during home visits in a resource-constrained environment. In conducting a redesign of our solution, we have achieved cost savings over both pen and paper collection solutions and remained comparable with our PDA/GPS solution. Our solution will yield significantly increased cost savings in the near future as the price of Android mobile phones declines. Also, we and others are able to reuse a significant amount of the code generated to support this project, and 9 months of a dedicated programmer's time should not be necessary.

A limitation of our cost analysis is that many of the figures we used to estimate cost can and will change over time. For example, although the actual salaries of our project's employees had not changed from our initial study to this one, the value of the US dollar did change in comparison with the Kenyan shilling. Several other costs also changed, such as the cost of shipping these devices to Kenya and the actual cost of each handset. We present our findings on cost as an estimate only and consider a full economic analysis of work such as this as a highly valuable potential future study.

At the same time of realizing these cost savings, we preserved the high level of user satisfaction we enjoyed with our previous PDA/GPS solution. We have again demonstrated that users with little to no previous computer or PDA experience can be rapidly trained and used to provide clinical care at the household level while documenting the interaction electronically. Our experience reiterates previous findings that, compared with pen and paper-based systems, information collected by means of handheld devices contains fewer errors, is more complete, requires less cleaning, and is not more expensive.⁷

In performing this study, we also learned valuable lessons regarding the creation and deployment of mHealth applications in developing settings. By taking our solution to the field and conducting two rounds of usability testing, we were able to discern factors that had previously not been an issue. Some of these include lighting, ease of navigation, difficulty with acquiring GPS coordinates, and battery life in areas of weak cellular signal. We were able to address many of these issues after detection. We feel that this process contributed to our high satisfaction scores with our end-users and was ultimately a key determinant in the success of this project to date.

The direct capture of electronic records greatly facilitated the expeditious performance of initial analyses and reports before the conclusion of the 3-year HCT programme. This matches previous research showing the benefit of electronic information by hand-held devices for use in queries, report generation, case management, and the establishment of follow-up.¹⁵ Our work has highlighted some of these findings, most notably that only 28% of persons we are identifying as infected with HIV are presenting for follow-up care. We feel compelled to address this finding immediately, as 3 years is a dangerous period of time to leave HIV untreated. We are currently considering directions in which we can effect this—some of our initial approaches may be similar to text message-based reminders that have been shown to be effective in previous research.¹⁶

The Ministry of Health in Kenya has been highly supportive of the USAID–AMPATH partnership throughout its history. We are hopeful that the outcomes of this project, when completed, will merit consideration by the ministry for inclusion in its HIV programmes.

Conclusion

In this brief study we have described the development and implementation of an open source, freely available solution for the provision of clinical care at the household level. We have expanded on our previous work by demonstrating increased efficiencies in cost while preserving a high level of user satisfaction and data quality. We also re-demonstrate the usefulness of electronic records in such applications, especially in the area of rapid analysis.

We have demonstrated three key findings of this work:

1. It is possible to collect data electronically in the field in resource-constrained environments at scale. We demonstrated this by deploying over 200 health workers and surveying 18 108 households composed of 63 470 persons during this implementation. We remain on target to complete our goal of surveying our entire two million person catchment area within our allotted timeframe.

2. End-users in resource-constrained environments prefer to use these systems. We demonstrated this first by showing satisfaction with our PDA/GPS solution and re-demonstrated it in the surveys during this implementation.

3. These systems can be cost effective. During this implementation we demonstrated a cost reduction of US$0.06 per person surveyed. Furthermore, the cost of these systems will probably decrease with the reuse of open source tools and the reduction in the costs of handsets capable of running the ODK Collect system.