Digital health interventions have the potential to improve the health workforce by supporting training, supervision, and communication. More evidence is needed on the effectiveness of interventions implemented at scale, including the return on investment, the effect of government and donor policies on scale up, and the role of the private sector.
Digital health interventions have the potential to improve the health workforce by supporting training, supervision, and communication. More evidence is needed on the effectiveness of interventions implemented at scale, including the return on investment, the effect of government and donor policies on scale up, and the role of the private sector.
ABSTRACT
The collection of journal articles, systematic reviews, and reports published over the last decade that attest to the potential of digital technologies to achieve health workforce improvements across all aspects of the health system is vast. As a capacity-building mechanism, digital technology has potential for low- and middle-income countries (LMICs) to support development of the health workforce, including those health workers based in remote or rural areas, to train, motivate, support, monitor, and pay them. The purpose of this scoping review to present, at a high level, the state of the evidence and best practices in digital strategies for human resources for health and to propose a roadmap for a research agenda to fill identified gaps in the evidence. A variety of peer-reviewed and gray literature sources were searched using selected key terms related to digital health and health workforce, limited to materials published from 2010 to 2018. More than 70 articles, reports, and blog posts were reviewed, with in-depth analysis of 29 articles. Findings show that a range of digital health solutions for health workforce development have been tested and used, such as for health worker training, provider-to-provider communication and professional networking, and supervision of and performance feedback to health workers. There is some evidence of improved efficiency and effectiveness, at least at the level of pilots or small-scale projects. There is, however, a growing urgency in global health to move beyond small-scale demonstration projects and to define the capital and recurring costs of implementation and scale up of digital health interventions, including the return on investment. The next frontier is to select, adapt, and implement at scale those digital health interventions for health worker development and management found to be most promising.
INTRODUCTION
It is estimated that 2 billion more people will be born around the world over the next 35 years.1 The challenges facing fast-growing developing countries are serious and real, and the threat of global pandemics as well as increasing morbidity linked to non-communicable diseases (NCDs), such as cardiovascular conditions, cancers, and diabetes, will have dramatic and negative impacts on the health of populations globally.2,3
Evidence indicates that where frontline health workers are effectively trained and deployed, there is a reduction in maternal and child mortality, a reduction in the spread of HIV, TB, and malaria, and better management of chronic diseases.4 Coupled with effective support and supervision, relevant and regular training has the potential to build a health workforce that can extend the reach of quality health care, in particular to poor and marginalized populations.5 The Sustainable Development Goals (SDGs), especially SDG 3 (“ensure healthy lives and promote wellbeing for all at all ages”), recognize this and emphasize the need for a substantial increase in the recruitment, development, training, and retention of the health workforce, especially in least-developed countries.6,7 Yet gains in survival are being lost because of weak and fragmented health systems with overburdened and overstressed health workers—too few in number and without the necessary support, training, and supervision that they need.8
There is an emerging consensus among researchers, as well as among global leaders and donors, that digital health represents an opportunity for developing stronger health systems to ensure adequate service delivery by frontline health workers.9 Mobile connections globally now stand at 7.6 billion (with 4.7 billion unique users) and mobile broadband penetration has risen sharply in the last 10 years.10 Smartphone penetration is already 48%, and predictions are that there will be 5.6 billion smartphones by 2020 with 90% of users in low- and middle-income countries (LMICs).10
Digital health represents an opportunity for developing stronger health systems.
The collection of journal articles, systematic reviews, and reports published over the last decade that attest to the potential of digital technologies to achieve health workforce improvements across all aspects of the health system is impressive.11,12 As a capacity-building mechanism, digital technology—comprising the use of computers, tablets, and mobile phones—represents an opportunity for LMICs to train, motivate, support, monitor, and pay health workers. For example, digital health (a term used synonymously with e-health, mHealth, and mobile health in this article) offers a good opportunity to reach cadres such as community health workers (CHWs)13,14 or health workers in remote and hard-to-reach areas that are routinely underrepresented in human resources for health (HRH) information systems. A comprehensive World Health Organization (WHO) classification of digital health interventions15 highlights how technology potentially offers opportunities for intelligent devices to automate data collection, disease surveillance, and rapid testing as well as to improve accountability and strengthen the interconnections between frontline health workers, health facilities, and ministries of heath.14 However, despite this general understanding of its potential benefits, digital health at a national scale is still slow to becoming a reality in many LMICs and there remains limited use of technology for training, communication, and data sharing between health professionals.10
The purpose of this article is to provide an overview of the state of the evidence at a high level and best practices in digital strategies for HRH/health workforce development and to propose a research agenda where gaps in the evidence are identified. We used a scoping review approach, a form of knowledge synthesis, which aims to summarize and synthesize evidence to inform practice, programs, and policy and to provide direction to future research priorities.16,17
METHODS
A scoping review was conducted using the framework recommended by Arksey and O'Malley16 and further developed by Colquhoun et al.17 The framework includes agreeing on clearly articulated questions, listing detailed inclusion criteria, designing a structured search process to locate and select relevant documentation, and agreeing on methods for critically appraising, extracting, and presenting the data.
This review was guided by the following 2 questions: “How are digital technologies potentially usable to improve health workforce capacity, confidence, and motivation?” “How can digital solutions improve monitoring of health staff performance across health system levels?” Documents included in the review were systematic reviews, thematic reviews, and other scoping reviews, as well as articles, reports, and blogs including published and unpublished (gray) literature. We excluded articles or reports that did not focus on LMICs, that considered digital health broadly (across several sectors), or that focused on digital financing or areas covered by the other articles included in this special supplement. We conducted searches on Google, Google Scholar, PubMed, the World Health Organization website, the Broadband Commission Working Group on Digital Health website, and the United States Agency for International Development website, limiting our search to materials published from 2010 to 2018. Search terms included “mobile health”, mHealth, “digital health”, eHealth, “ICT for health”, “frontline health worker”, “human resources for health”, “health workforce”, and “community health worker”.
Study selection involved applying the post-hoc inclusion and exclusion criteria based on the specifics of the research questions and on new familiarity with the subject matter after reading the studies. The most promising articles and reports were subjected to an in-depth analysis by the first author, who extracted relevant information using a descriptive analytical method for the contextual or process-oriented information.17 Examples of literature that were scrutinized included literature that featured successful implementation of digital health programs that provided effective health worker training using mobile devices, improved staff performance, and supported enhanced health service delivery. The descriptive data extracted were organized in conceptual categories to specifically shed light on the strength of current evidence for scaling programs using digital health technologies, as well as the documented weaknesses of that research.
RESULTS
More than 70 reviews, articles, reports, and blog posts were analyzed. A more in-depth review of 29 articles (of which 8 were systematic, thematic, or scoping reviews) and reports informed the views and conclusions presented in this article.
Why Digital Health?
The evidence suggests digital health has the potential to improve the efficiency of the health workforce, advance quality health services coverage, and enable better health outcomes.18 Digital health solutions that fit into existing health systems functions can support health systems strengthening objectives such as a well-performing health workforce; a functioning health information system; cost-effective use of medical products, vaccines, and technologies; and accountability and governance.5 Digital technologies can be used to train health workers, facilitate communication between them, provide job aids, and assist with supervision. In a review of the existing literature on use of digital health strategies by frontline health workers, DeRenzi et al.19 outlined 6 health systems functions that are typically addressed using digital technologies, including access to training and information for health workers, facilitation of communication between health workers, provision of job aids and decision support tools, and assistance with health worker supervision. The evidence to support these strategies is rapidly growing, though yet remains in early stages.9,19
Digital technologies can be used to train health workers, facilitate communication between them, provide job aids, and assist with supervision.
In relation to health workforce development specifically, there is a range of digital health interventions with some evidence of effectiveness, at least at the level of pilots or small-scale projects. Huang and colleagues state20:
There is a voluminous literature on a multiplicity of digital health innovations in middle and low income countries and in recent years a growing number of evaluation studies that have demonstrated beneficial impacts from these interventions.
Some of the benefits of introducing digital tools into existing systems for health workforce development include access to digital health training content, better support with professional networks, and an expanded role for health workers in active case detection using disease surveillance systems. For example, Cesario et al.21 find that:
Cellular phone use may contribute to the diagnosis and epidemiological surveillance of diseases in areas where physical access is difficult and/or which have limited human resources, medical, and technical equipment.
Furthermore, Nicole Ippoliti et al.22 indicate that:
Better data for decision making with virtually real-time reporting of services and commodities through a health management information system (HMIS) and logistics management information system (LMIS) improved provider capacity through continuous learning, digital provider tools, and mobile supervision [and] [i]ncreased transparency, efficiency, and accountability through digital financial services.
Digital tools can also promote continuing professional development to increase knowledge and skills as well as improve health workers' confidence in their day-to-day practice.23
Some of the recent research on digital health indicates promising impact and potential to scale, including:
Health worker training: In a correspondence to the Lancet Global Health, O'Donovan and Bersin report on how mHealth strategies can be a low-cost, high-impact solution to mapping outbreaks and providing education for both health workers and the public.24 For example, mobile phones were used by the local community to send free text messages about Ebola to the Government in Sierra Leone, who then created heat maps that health workers could use to track the epidemic. In addition, digital strategies were successfully deployed for health care workers in Liberia where they downloaded onto their phones reference and training materials demonstrating the correct and proper use of personal protective equipment and safe injection and burial practices.
Provider-to-provider communication: A number of digital health applications have been designed as tools to increase communication and professional networking to support health workers, with many concluding that basic mobile phone use along with SMS capabilities can improve CHW efficiency while potentially reducing overall program costs.25 In Malawi, CHWs were provided with SMS capabilities to report medical supply shortages and communicate both general information and specific information about patients with emergencies.26 The average cost per communication was about 5 times less expensive using SMS than in areas without SMS service, and reporting was significantly faster. Similarly, in another study using SMS for CHWs providing tuberculosis (TB) and HIV care in Malawi,27 communication between CHWs and the hospital staff freed up over 2,000 hours of hospital staff time, which resulted in expanded health care delivery capacity and a greater number of TB patients starting treatment.
Human resource management: Supervision of and performance feedback to health workers is critical to maintaining their motivation, improving their performance, and retaining them in their posts.28 Digital tools can provide a mechanism for delivering supervision to remote health workers who often work in isolation with little and infrequent support.29 In sub-Saharan Africa and in rural areas more generally, where health worker coverage is the lowest, digital health offers the promise of maximizing health care worker impact and efficiency.25,30 In India, for example, CHWs working to reduce malnutrition in their communities who received performance feedback and supportive supervision via mobile phone calls were shown to have improved motivation and performance compared with their motivation and performance before the program.31 The study suggests that regardless of the performance information disclosed, calls can improve performance due to elements of supportive supervision encouraging CHW motivation. There is also evidence that establishing a high-quality human resources information system can support health workforce management and efficient targeting of limited domestic resources.12
Other literature pointed to gaps where additional research is needed, including:
Total cost of ownership and implementation: Although digital health interventions offer the prospect of long-term cost savings, they usually require significant investments upfront as well as regular expenditure on training and maintenance. There has been some research on total cost of ownership32,33 but mostly with the purpose to develop an advocacy tool to highlight how a certain device or particular software provides the most cost-effective way to deliver digital health programs. There is less evidence on the capital and recurring costs needed to implement and maintain, let alone scale up, a digital health program.
Linked to this is the need to demonstrate the return on investment (ROI) in digital health. Government (largely via donor funding) is likely to be the largest funder of digital health initiatives in developing countries. Digital health stakeholders need to stimulate government investment by demonstrating how digital health solutions help address national health care issues. This is a critical knowledge gap; without longitudinal evidence and in the face of limited examples of digital enterprise architectures delivering better health outcomes, governments and global health donors remain unlikely to prioritize investments into large-scale HRH digital health programs.
Strength of the Evidence
As of 2015, there were more than 6,000 peer-reviewed or gray literature articles or documents on digital health.10 Evidence on the actual impact of digital health on population health outcomes is still largely lacking in the peer-reviewed literature.34,35 Bergmo explains35:
Despite promising results, more evidence is needed on the cost implications of digital health and the degree to which it can improve health outcomes over the short and long term. While there is anecdotal evidence that digital health can bring health benefits, the lack of sufficient rigorous clinical evidence and large-scale studies to confirm this claim is a barrier to investment.
It is this lack of evidence that presents one of the key challenges to moving digital health approaches from pilot projects to national programs while properly engaging health workers and communities in the process. A common feature emerging from literature published in the last 5 years (i.e., once the initial excitement of digital health innovation began to settle into a valley of ‘reality-on-the-ground’) is that a major weakness of digital health interventions is that the claimed benefits are unclear and long-term results remain uncertain.36 David Novillo-Ortiz, coordinator of the Regional eHealth Program of the Pan American Health Organization/WHO, is quoted in the Broadband Commission for Sustainable Development10 as saying:
The big challenge is to ensure the sustainability and continuity of digital health initiatives, whose benefits can sometimes only become apparent after ten to fifteen years. And to accomplish this, it is fundamental to promote the production of scientific evidence to raise awareness among decision-makers about the importance of investing in eHealth.
While studies may be successful at a small or moderate scale, large-scale or nationwide coverage of digital health interventions to support health workforce development is still rarely reported in the literature. Though rife with a descriptive analysis of the potential of such interventions on health care outcomes, few have empirically assessed the incremental effectiveness of such interventions on health care coverage, utilization, efficiency, quality, or outcomes.37 The speed of innovation, including the constant evolution of platforms, the broad range of initiatives and tools, and the heterogeneity of reporting have also made it difficult to uncover and synthesize how digital health tools might be effective.38 Until the WHO published the classification of digital health interventions in 2018, an absence of shared language and approach to describe digital health interventions made it challenging to compare results across programs.5 Furthermore, economic evidence for investing in digital health is limited. While often reputed to be cost-effective or cost-saving, the strength of the evidence supporting this assertion is typically considered insufficient to attract significant investment. In a systematic review of the economic evaluations of digital health, unknown cost-effectiveness was listed as one of the top 6 barriers to digital health implementation and a key factor in limited digital health policy investment.39
DISCUSSION
Digital technologies, when focused on delivery of care, training, staff performance, and monitoring, have the potential to improve health workforce capacity, confidence, and motivation. With partnerships forming between governments, technologists, NGOs, academia, and industry, the literature suggests there is increasing potential to improve health services delivery and support health workforce development by using digital health in LMICs. By harnessing the increasing presence of mobile phones among health workers, digital health can be used to deliver increased and enhanced health care services to individuals and communities while helping to strengthen health systems.14
Digital health use at national scale is slow to becoming a reality in many LMICs despite the general understanding of its potential benefits,6 and there remains limited use of technology for communication and data sharing between health professionals.40 Although digital technology can be a highly effective delivery system for national health systems, it, so far, has not been deployed as widely as technologists have anticipated. The research community has struggled to create an evidence base for decision makers to confidently expand digital health pilots to a provincial or national level.34,35
Digital health use at national scale is slow to becoming a reality in many LMICs.
There are several well-catalogued reasons for this general failure of pilots being taken to scale. The research points to the lack of coordinated approaches to digital health investments, leading to a fragmented digital health landscape, with multiple pilot interventions or short-term projects that do not include planning for scale or sustainability. Furthermore, traditional development funding flows are often disease- or country-specific, limiting adaptation, inhibiting data sharing, and resulting in inefficiencies.
In addition, until recently, there have been few agreed digital health commons, hubs of best practice, or standardized guidelines to guide implementers and governments in how to develop scalable, interoperable, and sustainable digital health systems for health workforce development. There is some progress in this respect, however: broad consensus around the shared values of digital development are enshrined in the Principles for Digital Development,41 and guidance has been developed by WHO, the International Telecommunication Union, and others on how to develop and maintain digital health systems at scale.42–44 Nonetheless, it remains challenging for donors to align funding and activities to support governments to create effective digital systems that are sustainable beyond the term of any one project or program.45 This despite recommendations urging greater collaboration around investment in digital health systems at national level46 and commitment from over 20 donor organizations to align around investments in digital health.47 In fact, it is still not unusual for digital health interventions to be developed and operated without any involvement of national governments. In addition, digital health solutions may be introduced by organizations without clear knowledge of the state of affected countries' information technology infrastructure at national and decentralized levels, making implementation of solutions inappropriate and, in the worst cases, resource-draining.45
A Roadmap for Research
Key areas of digital health research needed, as highlighted by the literature, include the following:
Value of scaling up digital health approaches to HRH management and support. Given the clear message from the literature that the case for investing in, and scaling up, digital health remains to be made, areas for additional inquiry include how digital technologies could be leveraged to expand the reach of performance management and timeliness of feedback for CHWs and other frontline health workers not currently being reached, timely updates of data for workforce planning and projections, cost-effectiveness of continuing education using digital health versus traditional approaches, and updating training needs assessments by regularly reaching out to health workers to ensure training content is well tuned to actual needs on the ground, among others.
Better evidence on the ROI of digital health. This could be addressed in the short term by economic modeling and research into the ROI of digital approaches in other sectors (e.g., the financial and agriculture sectors). In the longer term, the need for better ROI evidence could be addressed by working with governments implementing regional or national digital health strategies, such as the East African Community Digital Health Roadmap (Digital REACH) and the Tanzania Roadmap for Digital Health.
Effect of current donor and government procurement policies on scale up of digital health technologies for health workforce development. Although technology used in health worker training, supervision, and data collection programs is often designed to handle a large volume of information, it is often procured on the basis of single health issues and/or limited to specific geographic regions within countries. Procurement on this basis does not take advantage of the inherent scalability of the technology.
Role of the private sector and philanthropists in digital health. Despite calls for greater engagement of and investment by the private sector and philanthropic investors in taking digital health technologies to scale in a sustainable way, there are few examples of this kind of partnership happening at scale (or at all). This links back to ROI—without a strong business case for investment in national-level digital health infrastructure, the private sector will not move beyond the transactional corporate social responsibility approach to one based on long-term relationship building and investment. More research is needed into the role of the private sector, taking into account the diverse nature of that sector, and philanthropists in developing and implementing digital health strategies and how to engage them more effectively in this space.
The authors propose that an implementation research perspective, involving an iteration of learning by doing, analysis, and selection of the most promising approaches, will help document additional lessons learned and answer some of the remaining critical questions.48 An approach of ‘more of the same’ will result in the continued lack of evidence and represent a lost opportunity to leverage the potential capabilities that improved digital technologies can contribute to the health sector.
Strengths and Limitations
The scoping review approach we used has some strengths and limitations. On the one hand, it allows for presentation, in one publication, of the state of evidence on a particular issue that is useful for policy makers wishing to obtain a high-level view of the topic. But in doing so, it misses out on the type of details that some may want to see. As stated earlier, our main purpose was to present a high-level view of the state of evidence for using digital health technologies for health workforce development and to propose areas for further inquiry in the form of a research agenda that donors and researchers may wish to consider.
CONCLUSION
There is an urgent need to provide the evidence that will support a move beyond small-scale, demonstration projects to selecting, adapting, and implementing at scale those digital health interventions found to be most promising. More research is needed to respond to questions about how governments can develop their digital health leadership and harness the dramatic penetration of mobile and web technology to help improve the lives of their citizens. There needs to be a better understanding of how governments can motivate private- and public-sector staff to deliver tangible growth and security through digital health solutions that harness the advances in technology domains such as network speed and efficiency, cloud computing, data connectivity, and data analytics to improve health workforce capacity and improve access to and quality of health care services. An implementation research perspective will help uncover answers to these questions and document additional lessons learned. An ongoing reluctance to examine these bigger issues will lead to lost opportunities to leverage the remarkable capabilities that improved digital technologies have brought to the health sector.
Peer Reviewed
Competing Interests: None declared.
Funding: This work was made possible by a grant from the Aetna Foundation, with support from the World Health Organization and The Johns Hopkins University Global mHealth Initiative.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of The Johns Hopkins University or the Karolinska Institutet.
Cite this article as: Long L-A, Pariyo G, Kallander K. Digital technologies for health workforce development in low- and middle-income countries: a scoping review. Glob Health Sci Pract. 2018;6(suppl1):S41-S48. https://doi.org/10.9745/GHSP-D-18-00167
REFERENCES
- 1. World population projected to reach 9.7 billion by 2050. United Nations Department of Economic and Social Affairs website. http://www.un.org/en/development/desa/news/population/2015-report.html. Published July 29, 2015 Accessed September 16, 2018.
- 2. European Environment Agency. Global megatrends: changing disease burdens and risks of pandemics (GMT 3). Copenhagen, Denmark: European Environment Agency; 2015. https://www.eea.europa.eu/soer-2015/global/health Accessed September 4, 2017 (Archived by WebCite at http://www.webcitation.org/6tDuRsqQf.) [Google Scholar]
- 3. World Economic Forum. The Global Risks Report 2016. 11th ed Geneva: World Economic Forum; 2016. http://www3.weforum.org/docs/GRR/WEF_GRR16.pdf Accessed September 4, 2017 (Archived by WebCite at http://www.webcitation.org/6tDujkw2U.) [Google Scholar]
- 4. Jones G, Steketee RW, Black RE, Bhutta ZA, Morris SS; Bellagio Child Survival Study Group. How many child deaths can we prevent this year? Lancet. 2003;362(9377):65–71. 10.1016/S0140-6736(03)13811-1. . [DOI] [PubMed] [Google Scholar]
- 5. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Health Sci Pract. 2013;1(2):160–171. 10.9745/GHSP-D-13-00031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. de Francisco Shapovalova N, Meguid T, Campbell J. Health-care workers as agents of sustainable development. Lancet Glob Health. 2015;3(5):e249–e250. 10.1016/S2214-109X(15)70104-X. [DOI] [PubMed] [Google Scholar]
- 7. SDG 3: ensure healthy lives and promote wellbeing for all at all ages. World Health Organization website; http://www.who.int/sdg/targets/en/ Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J8FUz8B.) [Google Scholar]
- 8. The Joint Learning Initiative. Human Resources for Health: Overcoming the Crisis. Boston: The President and Fellows of Harvard College; 2004. http://www.who.int/hrh/documents/JLi_hrh_report.pdf Accessed August 31, 2017 (Archived by WebCite at http://www.webcitation.org/6t8jITuYC.) [Google Scholar]
- 9. Agarwal S, Perry HB, Long LA, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health. 2015;20(8):1003–1014. 10.1111/tmi.12525. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Broadband Commission for Sustainable Development. Digital Health: A Call for Government Leadership and Cooperation between ICT and Health. Broadband Commission for Sustainable Development; 2017. http://vitalwave.com/wp-content/uploads/2017/02/Broadband-Commission-Digital-Health-Final-Report-20170222.pdf Accessed September 9, 2018. [Google Scholar]
- 11. Levine R, Corbacio A, Konopka S, et al. mHealth Compendium: Volume 5. Arlington, VA: African Strategies for Health, Management Sciences for Health; 2015. http://www.africanstrategies4health.org/uploads/1/3/5/3/13538666/mhealthvol5_final_15jun15_webv.pdf Accessed September 12, 2018. [Google Scholar]
- 12. Likofata Esanga JR, Viadro C, McManus L, et al. How the introduction of a human resources information system helped the Democratic Republic of Congo to mobilise domestic resources for an improved health workforce. Health Policy Plan. 2017;32(suppl 3):iii25–iii31. 10.1093/heapol/czx113. [DOI] [PubMed] [Google Scholar]
- 13. Schuttner L, Sindano N, Theis M, et al. A mobile phone-based, community health worker program for referral, follow-up, and service outreach in rural Zambia: outcomes and overview. Telemed J E Health. 2014;20(8):721–728. 10.1089/tmj.2013.0240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Källander K, Tibenderana JK, Akpogheneta OJ, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low- and middle-income countries: a review. J Med Internet Res. 2013;15(1):e17. 10.2196/jmir.2130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. World Health Organisation (WHO). Classification of Digital Health Interventions. Geneva: WHO; 2018. http://www.who.int/reproductivehealth/publications/mhealth/classification-digital-health-interventions/en/ Accessed September 12, 2018. [Google Scholar]
- 16. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:1, 19–32. 10.1080/1364557032000119616 [DOI] [Google Scholar]
- 17. Colquhoun HL, Levac D, O'Brien KK, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67(12):1291–1294. 10.1016/j.jclinepi.2014.03.013. [DOI] [PubMed] [Google Scholar]
- 18. Smith V, Long L, Moore C. Unlocking the community health workforce potential, post-Ebola: what models and strategies work? West Sussex, UK: Wilton Park; 2014. https://www.wiltonpark.org.uk/wp-content/uploads/WP1447-Report.pdf Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J90oKAe.) [Google Scholar]
- 19. Derenzi B, Borriello G, Jackson J, et al. Mobile phone tools for field-based health care workers in low-income countries. Mt Sinai J Med. 2011;78(3):406–418. 10.1002/msj.20256. [DOI] [PubMed] [Google Scholar]
- 20. Huang F, Blaschke S, Lucas H. Beyond pilotitis: taking digital health interventions to the national level in China and Uganda. Global Health. 2017;13(1):49. 10.1186/s12992-017-0275-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Cesario M, Lundon M, Luz S, Masoodian M, Rogers B. Mobile support for diagnosis of communicable diseases in remote locations. In: Proceedings of the 13th International Conference of the NZ Chapter of the ACM's Special Interest Group on Human-Computer Interaction New York: Association for Computing Machinery; 2012: 25–28. https://www.scss.tcd.ie/∼luzs/publications/CesarioLundonEtAl12m.pdf Accessed August 31, 2017 (Archived by WebCite at http://www.webcitation.org/6t8ZhKYrx.) [Google Scholar]
- 22. High Impact Practices in Family Planning (HIPs). Digital Health: Strengthening Family Planning Systems. Washington, DC: US Agency for International Development; 2017. https://www.fphighimpactpractices.org/briefs/digital-health-systems/ Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J98lesF.) [Google Scholar]
- 23. Continuing Professional Development in Ghana. Knowledge for Health website. https://www.k4health.org/projects/ghana Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J6OTpxg.)
- 24. O'Donovan J, Bersin A. Controlling Ebola through mHealth strategies. Lancet Glob Health. 2015;3(1):e22. 10.1016/S2214-109X(14)70357-2. [DOI] [PubMed] [Google Scholar]
- 25. Betjeman TJ, Soghoian SE, Foran MP. mHealth in Sub-Saharan Africa. Int J Telemed Appl. 2013;2013:482324. 10.1155/2013/482324. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Lemay NV, Sullivan T, Jumbe B, Perry CP. Reaching remote health workers in Malawi: baseline assessment of a pilot mHealth intervention. J Health Commun. 2012;17(suppl 1):105–117. 10.1080/10810730.2011.649106. [DOI] [PubMed] [Google Scholar]
- 27. Mahmud N, Rodriguez J, Nesbit J. A text message-based intervention to bridge the healthcare communication gap in the rural developing world. Technol Health Care. 2010;18(2):137–144. 10.3233/THC-2010-0576. [DOI] [PubMed] [Google Scholar]
- 28. Strachan DL, Källander K, Nakirunda M, Ndima S, Muiambo A, Hill Z; inSCALE study group. Using theory and formative research to design interventions to improve community health worker motivation, retention and performance in Mozambique and Uganda. Hum Resour Health. 2015;13:25. 10.1186/s12960-015-0020-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Strachan DL, Källander K, ten Asbroek AH, et al. Interventions to improve motivation and retention of community health workers delivering integrated community case management (iCCM): stakeholder perceptions and priorities. Am J Trop Med Hyg. 2012;87(5 suppl):111–119. 10.4269/ajtmh.2012.12-0030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Thondoo M, Strachan DL, Nakirunda M, et al.; InSCALE Study Group. Potential roles of mhealth for community health workers: formative research with end users in Uganda and Mozambique. JMIR Mhealth Uhealth. 2015;3(3):e76. 10.2196/mhealth.4208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Kaphle S, Matheke-Fischer M, Lesh N. Effect of performance feedback on community health workers' motivation and performance in Madhya Pradesh, India: a randomized controlled trial. JMIR Public Health Surveill. 2016;2(2):e169. 10.2196/publichealth.3381. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Dimagi. Total Cost of Ownership Model – CommCare. HingX website. https://hingx.org/Share/Details/1039 Accessed November 20, 2017.
- 33. Jefferson B, Taylor M, Borrelli A, Segan R, Chelvayohan M. Community health worker (CHW) training–total Cost of ownership (TCO). Futures Group presentation. https://solutionscenter.nethope.org/assets/collaterals/Final_TCO_CHW__02_13_14.pdf Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J9w2013.)
- 34. Agarwal S, LeFevre AE, Lee J, et al. ; WHO mHealth Technical Evidence Review Group. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ. 2016;352:i1174. 10.1136/bmj.i1174. [DOI] [PubMed] [Google Scholar]
- 35. Bergmo TS. How to measure costs and benefits of eHealth interventions: an overview of methods and frameworks. J Med Internet Res. 2015;17(11):e254. 10.2196/jmir.4521. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health. 2014;14:188. 10.1186/1471-2458-14-188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Bailey M. Scale the Technology Now: Applying Engineering Principles to Promote Rapid Deployment of Mobile Digital Content Delivery Systems. Washington, DC: mPowering Frontline Health Workers; 2016. https://mpoweringhealth.org/wp-content/uploads/2016/12/mPowerering-Paper-Jan-2017-v1r7.pdf Accessed August 31, 2017 (Archived by WebCite at http://www.webcitation.org/6t8fKgE1Q.) [Google Scholar]
- 38. Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV treatment & prevention: a systematic review of the literature. Open AIDS J. 2013;7:17–41. 10.2174/1874613620130812003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One. 2017;12(2):e0170581. 10.1371/journal.pone.0170581. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. GSMA Intelligence. Scaling Digital Health in Developing Markets: Opportunities and Recommendations for Mobile Operators and Other Stakeholders. London: GSMA Intelligence; 2017. https://www.gsmaintelligence.com/research/?file=c581aa43bdb7b7d236bb937698c2d6fd&download Accessed August 31, 2017. [Google Scholar]
- 41. Principles for Digital Development Working Group. Principles for Digital Development website. https://digitalprinciples.org/ Accessed September 9, 2018 (Archived by WebCite at http://www.webcitation.org/72J5l3lKw.)
- 42. World Health Organization (WHO); International Telecommunication Union (ITU). National eHealth Strategy Toolkit. Geneva: WHO and ITU; 2012. https://www.itu.int/dms_pub/itu-d/opb/str/D-STR-E_HEALTH.05-2012-PDF-E.pdf Accessed August 31, 2017 (Archived by WebCite at http://www.webcitation.org/72J61xECh.) [Google Scholar]
- 43. World Health Organization (WHO). The MAPS Toolkit: mHealth Assessment and Planning for Scale. Geneva: WHO; 2015. http://www.who.int/reproductivehealth/publications/mhealth/maps/en/ Accessed August 31, 2017 (Archived by WebCite at http://www.webcitation.org/72J6kd8nn.) [Google Scholar]
- 44. Digital Impact Alliance (DIAL). Beyond Scale: How to Make Your Digital Development Program Sustainable. Washington, DC: DIAL, United Nations Foundation; 2017. https://digitalimpactalliance.org/wp-content/uploads/2017/12/DIAL_BeyondScale_eBook-v11.pdf Accessed September 7, 2018 (Archived by WebCite at http://www.webcitation.org/72J6wjqfm.) [Google Scholar]
- 45. Fast L, Waugaman A. Fighting Ebola with Information: Learning From Data and Information Flows in the West Africa Ebola Response. Washington, DC: US Agency for International Development; 2016. https://www.usaid.gov/sites/default/files/documents/15396/FightingEbolaWithInformation.pdf Accessed September 4, 2017 (Archived by WebCite at http://www.webcitation.org/6tEdgKBsO.) [Google Scholar]
- 46. National Academies of Sciences, Engineering, and Medicine. Global Health and the Future Role of the United States. Washington, DC: The National Academies Press; 2017. 10.17226/24737 [DOI] [PubMed] [Google Scholar]
- 47. The Principles of Donor Alignment for Digital Health. Digital Investment Principles website. http://digitalinvestmentprinciples.org/ Accessed September 7, 2018 (Archived by WebCite at http://www.webcitation.org/72J7iANQB.)
- 48. Peters D, Tran N, Adam T. Implementation Research in Health: A Practical Guide. Geneva: Alliance for Health Policy and Systems Research, World Health Organization; 2013. http://www.who.int/alliance-hpsr/resources/implementationresearchguide/en/ Accessed September 12, 2018. [Google Scholar]