Assessing costs of developing a digital program for training community health workers to deliver treatment for depression: A case study in rural India

Abstract

Digital technology has emerged as a promising approach for training and building capacity of community health workers in low-income and middle-income countries (LMICs). Little is known about the cost of developing digital training programs in LMICs, which hinders the adoption, implementation, and scaling up of the programs in routine primary care settings. This study assessed the costs of developing a digital program for training community health workers to deliver a psychological treatment for depression in a rural district of Madhya Pradesh, India. We developed survey instruments to document required resources in development, including involved personnel (their roles, responsibilities, time spent, and salaries or payments), information technologies (e.g., smartphones, software programs), and infrastructure-related costs (e.g., vehicle, office space, utilities). Costs were estimated from an accounting perspective. Over a 10-month developmental period, the total costs were 208,814 USD, with the largest portion on human resources (61%, with 14% on management and supervision), followed by information technologies (33%), and infrastructure-related costs (6%). These findings could inform policymakers in LMICs on costs of developing online-training programs, which will be especially useful during the COVID-19 pandemic.

1. Introduction

Community health workers (CHWs) have become an important component of the healthcare workforce in many low-income and middle-income countries (LMICs) since the 1978 Declaration of Alma-Ata (Gopalakrishnan et al., 2020; Lehmann and Sanders, 2007). Decades of research demonstrate compelling evidence of their important role in increasing community access to necessary care (Lewin et al., 2010; Perry et al., 2014). While the role of CHWs in many LMICs was predominantly on improving child and maternal health, over time, CHWs have been involved in addressing the increased burden of communicable and recently non-communicable diseases, including mental disorders (Saprii et al., 2015; Tsolekile et al., 2014). Scaling up evidence-based mental health treatments has been identified as a priority by the World Health Organization (WHO) (Chisholm et al., 2007; Collins et al., 2011). There is considerable evidence demonstrating that first-level health care providers, such as CHWs and other health workers without specialized training in mental health care, could effectively deliver brief psychological treatments for mental disorders (such as depression) across diverse community and primary care settings (Organization, 2015; Singla et al., 2017; Van Ginneken, N., 2016). As a promising method for bridging the mental health care gap in LMICs, success of CHW initiatives relies on effectively training CHWs and developing their skills and knowledge for supporting mental care delivery (Balaji et al., 2012; Pereira et al., 2011; Van Ginneken, N., 2016).

Implementing conventional in-person training for CHWs could be difficult in resource-limited setting due to administrative, economic, and logistic challenges such as travelling long distances to attend in-person trainings (often requiring overnight stays), requirements for classroom space and facilities to host the trainings, and limited number of available qualified trainers (Ministry of Health and Family Welfare, 2013; Winters et al., 2019). In the era of smartphones and mobile internet, digital technology has emerged as a potential approach to overcome challenges in conducting in-person trainings (Chibanda et al., 2016; Diez-Canseco et al., 2018; Maulik et al., 2017; Naslund et al., 2019; Rahman et al., 2019; Xu et al., 2019). In 2019, WHO published guidelines on digital health interventions, and specifically recommending digital training as a complement to conventional training for health workers (World Health Organization, 2019). With the ongoing COVID-19 global pandemic, many LMICs have experienced lockdowns at either the national- or regional-level, and remote learning has become an important option for CHWs to receive necessary training (Kola et al., 2021).

Despite mounting research reporting on the benefits of digital mental health interventions (Carter, H., R. Araya, 2020; Kaonga and Morgan, 2019; Naslund et al., 2019, 2017), there remains limited uptake and adoption of these technologies in primary care settings, particularly in LMICs (Torous et al., 2018). One of the key barriers is the lack of evidence on costs of developing digital technologies in LMICs to inform interested policymakers and other stakeholders (Mitchell, L.M., 2021). This study intends to contribute to the knowledge by assessing the costs of developing a digital program for training CHWs, as part of a broader effort to scale up task sharing of a brief psychological treatment for depression in primary care settings in rural India. The study constructed survey instruments to document required resources for the development and accounted for their associated costs. The reported cost data could inform health system administrators and other key stakeholders regarding the resources needed for developing or adopting similar digital training programs in rural settings in LMICs.

2. Methods

2.1. Setting

This study was part of a larger research project “Enabling translation of Science to Service to ENhance Depression CarE” (ESSENCE)”, carried out in Madhya Pradesh, one of the largest states of India with a population over 72 million, of which nearly 73% live in rural areas (Chandramouli, C. and R. General, 2011; Menon et al., 2008; Suryanarayana et al., 2016). Madhya Pradesh faces severe gaps in available mental health services, with over 91% of individuals in need of mental health services not having access to adequate care (Kokane et al., 2019). To address this care gap through public sector facilities, Sangath, a non-governmental, not-for-profit research organization committed to improving public health in India, has been working with the Department of Public Health and Family Welfare, Government of Madhya Pradesh to support the training of female CHWs, referred to as Accredited Social Health Activists (ASHAs), in delivering a psychological treatment for depression (called the Healthy Activity Program [HAP]). Between September 2018 and July 2019, we developed a digital program for training ASHAs to deliver HAP in Sehore, a rural district of Madhya Pradesh. Formative research conducted by our team has demonstrated that use of digital technology is feasible and acceptable among the target group of CHWs (Muke et al., 2019), and as reflected in a recent pilot study, also appears to achieve comparable training outcomes as conventional in-person training methods (Muke et al., 2020). All procedures described in this study were approved by ethical review boards at Sangath, India, and Harvard Medical School, USA.

2.2. Healthy Activity Program

The Healthy Activity Program (HAP) is an evidence-based brief six to eight sessions psychological intervention based on behavioral activation offered to adult primary care attenders who have moderately severe to severe depressive symptoms, assessed by a Patient Health Questionnaire-9 score >14. Each session lasts 30–40 minutes, with the initial sessions being at weekly intervals. The program is recommended by the WHO, and consists of core strategies including psychoeducation, behavioral assessment, activity monitoring, activity structuring and scheduling, and activation of social networks (Patel et al., 2017; WHO, 2017). HAP was originally developed and tested in Goa, India (Chowdhary et al., 2016), with a large-scale trial demonstrating the effectiveness of the program for treating depression (Patel et al., 2017). HAP has since been evaluated in primary care settings in Madhya Pradesh, India (Shidhaye et al., 2019), as well as in Nepal (Jordans et al., 2019). As part of this project, the program content was digitized for making the training accessible from a smartphone application and adapted to the local context in Madhya Pradesh, India, as outlined in the following section.

2.3. Developing a digital training program

The steps for developing the digital program for HAP training are outlined in Supplemental Figure 1. The content of the HAP program is presented as part of 2 instructional manuals, one focused on general counseling skills, and one focused on specific skills for treating depression. Both manuals are available open source from Sangath (https://www.sangath.in/). Firstly, we reviewed these 2 HAP manuals, as this is the core content used for guiding clinicians in delivering the brief psychological treatment (Patel et al., 2017). Secondly, we adapted the contents from the HAP manuals to create the 16-module training program. The content was initially developed in English to allow for review from subject matter experts and stakeholders, and further contextualized to account for local language and culture. Thirdly, the content was digitized (by creating video lectures and role plays, power-point videos, and additional digital graphics and supplemental materials). The digitized contents were uploaded and hosted on the Moodle Learning Management System (LMS) accessible from a smartphone app. Fourthly, we tested the platform and digital content with ASHAs over 4 days using smartphones for identifying any potential technical challenges while assessing the usability, program content on the smartphone app. Lastly, we revised the training program by incorporating the suggestions and feedback received from the user testing ahead of our pilot trial. These steps are described in greater detail elsewhere (Khan, A., 2020).

The digital training program consists of 16 modules contain a total of 85 short videos with 39 video lectures, 13 role-play videos, 20 PowerPoint lectures with voiceover, and 13 summary videos providing a recap of the content displayed in a given module. The videos ranged in duration from about 1 to 13 min, though the average duration was approximately 3–5 min. In addition to the video content, we included a total of 92 engagement quizzes and 192 knowledge assessment questions embedded throughout the modules. The total duration of the training was about 8 hours of content, but estimated to require about 48 hours to complete accounting for the time to fully read the material and respond to the questions. This training duration was specifically matched to the duration of the face-to-face residential training program as this is the typical format for training ASHAs within government health facilities completed over 6 full days of classroom-based instruction (Khan, A., 2020).

2.4. Constructing survey instruments

To estimate the costs associated with the resources used during development of the training program, we designed survey instruments for cost data collection based on a WHO framework which describes a health system in terms of six building blocks (service delivery, health workforce, health information, access to essential medicines, financing, leadership and governance) (WHO, 2010), an approach that has been applied previously for estimating costs of health services in low-resource settings (Lu et al., 2014). Adapting the WHO building blocks helped policy makers understanding resources required for program development by six building blocks from health system perspective. A breakdown of reported costs into these building blocks makes it possible to assess whether or not the resources are available in the related building blocks and how much more budget is needed for developing similar programs. This exercise also allows for avoiding potential duplication, overlap and confusion.

The survey instruments we constructed captured cost-related information in three categories: (1) human resources required for developing the digital training program, including an involved individual’s role, responsibility, time spent in development, as well as his/her salary or received payments; (2) information technologies used in the development, including unit price and quantity of the used items (e.g. smartphones); and (3) infrastructure-related support, such as vehicle/transportation, office space and supplies, utilities. During the development process, we invited external experts and advisors to attend seven consultation meetings. To capture costs of their consultations, we included one section in the survey to track time effort of these voluntary services. The survey instrument is presented in the appendix (see supplemental Table 1).

2.5. Cost Data Collection

We constructed a local data collection team which includes three data collectors who received training on data collection, data entry, and data storage. During the data collection process, the team held regular meetings with senior collaborators to discuss challenges and potential solutions.

Data on participants’ roles/titles, their responsibilities and time spent, and monthly salary (including benefits) were collected from the Administration and Finance Department at the Sangath office in Bhopal. We also gathered the payment data from the Department on all used materials in development, as well as contracting external agencies for supporting the development of the digital training content and an online learning platform (i.e., the LMS). The data were captured via time logs, payroll data, vendor receipts, invoices, and project records on a monthly basis. The data were entered and double-checked for quality control by one member of the data team and one member of the Department at the Sangath office. Any discrepancy in the data was followed-up by a third member of the team to reach consensus. Data were collected from September 2018 to mid of July 2019.

2.6. Cost Estimation:

Costs of development were measured from health system and societal perspectives and included opportunity costs estimated in monetary terms. We obtained costs in human resources by multiplying the participants’ monthly salary by proportion of their time spent on the development. For ASHAs participating in development, we estimated the costs of their time using corresponding average daily wage in their group. We used the same method to measure the costs of external experts and advisors who attended the consultation meetings without being paid: their costs of time were estimated with a wage rate of any person who would normally be employed to do the same tasks. For information technologies used in the development, we calculated the costs of the items purchased for the development using financial records obtained from the Department. The costs of using existing items (e.g., laptops, office supplies) were derived by using straight line depreciation with zero salvage value assumed. For costs of office space, we estimated it with the market rent for the same size. Monthly costs of utilities (e.g., electricity, fuel charges) for the office designated to the development activities were estimated by multiplying the proportion of monthly time spent on development by monthly bills; and if the utility costs were missing in a month, we adopted the value documented in the previous month. To inform interested policymakers and implementers on resource allocation within and across the three categories, we estimated the itemized costs for each category (human resources, information technologies, and infrastructure-related support) as well as the total costs of development. To allow for international comparison, all costs were converted to the 2017 Purchasing Power Parity (PPP) (39) and 2018 US dollars (USD) (40). We report costs in Indian Rupees (INR), PPP, and USD in the tables.

3. Results

The one-time development costs of the digital training program took approximately 10 months with a total of $208,814 USD. As presented in Figure 1, the largest proportion of the costs was attributed to human resources (61%; $127,628 USD), followed by information technologies (33%, $69,070 USD), and infrastructure-related support (6%, $12,116 USD). Table 1 describes the role and responsibility of involved personnel, their devoted time efforts, and the related costs over the 10-month period. Several types of personnel were recruited for development, including (a) experts for developing training contents (e.g. psychologists), (b) script writer and translators for developing training materials, (c) technology personnel (e.g., LMS developer, graphic and video designers, actors and producers), (d) ASHA workers and their supervisors for testing the training materials, (e) researchers and experts for developing assessment tools, finalizing training materials, and uploading the training content to the LMS, and (f) supervisors or managers who oversaw the development activities. In addition, three international experts (one psychologist and two psychiatrists) were invited to attend consultation meetings for planning and reviewing progress with the project.

Figure 1.

Total costs and proportion of cost components for developing the digital training program over the 10-month period, in 2018 USD

Table 1.

Itemized costs for human resources used in digital training development over the 10-month period

Role/Title	Responsibility	Proportion of FTE1 or unit cost	Total Costs (INR)	Total Costs (USD)2	Total Costs US PPP3
1a. Personnel directed involved in developing digital training program
Sangath office
Project manager (1)	Supporting module structuring; reviewing Hindi translations;	0.5	430,526	6,295	24,169.2
Psychologist (1)	Developing blueprint; designing and developing the content (i.e. adaptation of content; module structuring); creating power points; reviewing Hindi translations	1	744,744	10,890	41,809
Coordinator (1)	Developing assessment questions and power points; supporting review of Hindi translations	0.5	367,598	5,375	20,636.5
Graphic designer (1)	Digitizing and uploading the content on the LMS4; creating line art & icons for the intervention	1	214,900	3,142	12,064.2
Researcher (1)	Developing assessment questions; adapting and structuring digital content on LMS	1	488,876	7,148	27,444.9
Research assistants (3)	Assisting the team in developing digital training;	1	998,753	14,604	56,068.8
External consultancy
Post-doctoral research fellow (1, US-based)	Reviewing the English translations and scripts; attending every meeting related to development of digital training	0.2	658,244	9,625	36,953
Subject matter expert (English, 1, US-based)	Reviewing and advising on fidelity of the English content; reviewing assessment questions	0.05	532,887	7,792	29,915.6
HAP5 counsellors to review the scripts (2)	Reviewing the English scripts, assessment questions and structuring of training with the digital intervention;	15000 INR per month for 3 months (219 USD)	90,000	1,316	5,052.5
Subject matter expert (Hindi, 1)	Reviewing and advising on fidelity of the Hindi content; deliver training lectures in front of the camera for video productions; reviewing assessing questions	300 INR (4.4 USD) per page	78,600	1,149	4,412.5
ASHA6 Supervisors and workers (4)	Assisting in designing the content of the intervention		8,100	118	454.7
Advisory board meetings	Expert meetings with collaborators from oversee and incountry for planning, reviewing and feedback on the training intervention development		1,668,776	24,401	93,683
Outsource contract in local area
Scriptwriter (1)	English script writing (1–13)	17,647 INR per module script (258 USD)	229,411	3,355	12,878.9
Translator (1)	Translating English script to Hindi; Translating HAP training manual, participant workbook	300 & 400 INR per page (4.4 & 5.8 USD)	90,900	1,329	5,103
Actors hired for video production	Playing the role of counsellors and patients for scripted role-plays in the videos		112,900	1,651	6,338.1
1b. Personnel who provided management and supervision for developing digital training program
Psychiatrist (1, Principal Investigator, US-based)	Overall guidance to the team and management	0.05	687,857	10,058	38,615.4
Site Principal Investigator (1)	Overall guidance of the local team	0.05	79,661	1,165	4,472.1
Post-doctoral research fellow (1, US-based)	Overseeing and guiding the team on ground; maintaining regular communication between offsite and onsite team;	0.3	987,400	14,438	55,431.4
Project manager (1)	Managing all the components of the project at local level	0.3	258,316	3,777	14,501.5
Total			8,728,449	127,628	490,004.4

^1:Full time effort

^2:1 USD = 68.389 INR in 2018

^3:1 USD PPP = 17.813 INR in 2017

^4:LMS: Learning Management System

^5:HAP: Health Activity Program

^6:ASHA: Accredited Social Health Activist

Among all items in the “Human resources” category, spending on salaries and fringe benefits was the largest ($47,454 USD; 23% of total development costs), followed by external consultancy and meetings ($44,401 USD; 21% of total development costs), management and supervision ($29,438 USD; 14% of total developmental costs), and outsourced contracts ($6,335 USD; 3% of total development costs) (Table 1).

Table 2 presents the costs of information technologies used in the development. Developing high-quality videos and establishing the LMS platform were crucial in creating digital training materials. Spending in this category covered activities such as filming, recording, voiceover, digital platform customization for smartphone app, and server installation, modifications, and configuration. Over the 10-month period, approximately 30% of total development costs was on video production ($43,110 USD; 21% of total development costs) and establishing LMS ($19,364 USD; 9% of total development costs).

Table 2.

Itemized costs for information technology used in digital training development over the 10-month period

Items	Function	Total Costs (INR)	Total Costs (USD)1	Total Costs US PPP2
E-abhyas (LMS3 development agency)	Development and modification of LMS2 on Sangath platform	1,324,258	19,364	74,342.22
Video production (53, by two production agencies)	Developing videos, including cinematography, filming, editing, sound recording, voiceover	2,948,200	43,110	165,508.34
Smartphones (12)	User testing	96,000	1,404	5,389.32
Equipment and software	Creating graphic designs, reviewing the content etc.; antivirus for laptops; speakers for meetings	228,466	3,341	12,825.8
Communication fees, printing and stationery		126,575	1,851	7,105.77
Total		4,723,499	69,070	265,171.45

^1:1 USD = 68.389 INR in 2018

^2:1 USD PPP = 17.813 INR in 2017

^3:LMS: Learning Management System

Table 3 presents the itemized costs for infrastructure-related support over the 10-month period, with a total of $12,116 USD. More than half of spending in this category was on paying for utilities (rent, electricity, water, etc.).

Table 3.

Itemized costs for infrastructure-related support for digital training development over the 10-month period

Items	Total Costs (INR)	Total Costs (USD)1	Total Cost US PPP 2
Vehicle	22,726	332	1,275.81
Furniture and office supply	59,043	863	3,314.6
Utilities (rent, electricity, fuel, water etc.)	475,144	6,948	26,674
Other (housekeeping, office maintenance)	271,695	3,973	15,252.62
Total	828,608	12,116	46,517.04

^1:1 USD = 68.389 INR in 2018

^2:1 USD PPP = 17.813 INR in 2017

4. Discussion

This study provides a comprehensive assessment of the resources required and associated costs of developing a digital program for training CHWs to deliver a brief psychological treatment for depression in a rural district of India. The total development costs over approximately 10 months were $208,814 USD, with the largest portion devoted to human resources (61%), suggesting that developing digital training is labour intensive. This finding is consistent with prior studies on costs of developing digital mental health interventions. For example, one study conducted in the United States reported that the personnel costs represented 54% of total development costs ($138,683 USD) for an online depression intervention for adolescents (Ruby et al., 2013). In our study, over the 10-month period, a total of 23 individuals with various expertise contributed to the development, including psychologists, psychiatrists, IT experts, video producers, script writers and translators, and ASHAs workers and their supervisors. Developing a digital training program requires expertise and time commitment in both psychological treatments and technologies.

As digital technology has shown promise for training and supporting community health workers in delivering mental health care across a variety of settings (Naslund et al., 2019), understanding the costs of developing a digital mental health training program is crucial for policy makers and other stakeholders in guiding their decision regarding use of different training methods. The importance of reporting development costs of digital health interventions has been previously recognized (McNamee et al., 2016; Naslund, J.A., 2020). However, the related literature is scarce in LMICs (Winters et al., 2019, 2018). To the best of our knowledge, no existing studies reporting economic evaluations or development of digital mental health interventions in LMICs have documented the specific costs for intervention development. Even in high-income settings, development costs are rarely reported in the digital mental health literature. For instance, in a recent systematic review of 24 studies on costs and cost-effectiveness of digital interventions for depression or anxiety, only four studies reported development costs (Mitchell, L.M., 2021). Lack of cost evidence represents a gap in the field of digital mental health, and potentially impedes development of digital health technologies in LMICs (LeFevre et al., 2017).

When reviewing the existing literature on developmental costs of digital mental health interventions in high-income countries, we observed a wide range of cost estimates, from $24,891 USD (£19,340 in the paper) for an online intervention on bipolar disorders (Lobban et al., 2017) to $138,683 USD for an online intervention for adolescent depression (Ruby et al., 2013). Developmental costs for digital behavior change interventions ranged from as low as $30,912 USD (£20,000 in the paper) to approximately $789,890 USD (£500,000 in the paper) (McNamee et al., 2016). The large variation in development costs could be partly due to the variations in type of content of the programs, level of technological complexity of the programs, price variation between years or across locations, or length of development duration (McNamee et al., 2016; Mitchell, L.M., 2021). This wide variation could also partly be due to non-standardized practices in defining developmental costs, collecting cost data, and reporting cost components (Mitchell, L.M., 2021). Among these four studies that reported development costs of digital mental health interventions, one study only presented a single number without offering any details about breakdown of costs (Nordgren et al., 2014); two studies had few descriptions on included cost components (e.g. writing the treatment program, computer programming) (Hedman et al., 2013, 2011); while only one study provided detailed summary of the cost components (e.g. salary and time effort, intervention infrastructure, travel and consultation, etc.) (Ruby et al., 2013). The lack of standardized reporting makes it difficult to compare development costs across digital interventions, and to understand specifically what types of resources are required to ensure successful development and replication across other settings.

To improve data quality, transparency, and comparability of developmental costs of digital training programs, we recommend a reporting checklist as described in Table 4. Building on our digital program development experience, literature review, and drawing from meetings with international and local experts, we propose that at least the following information be reported: (1) length of time for development; (2) human resources involved, including number of personnel, their titles/roles and responsibilities, time efforts, and related costs; (3) information technologies used: including name and quantity, their functions, and related costs; and (4) infrastructure-related costs: including logistic support, office space and supplies, and utilities. Our recommendation is based on the fact that while the salary amounts for a position or unit price of an item could vary widely across different settings, the overall resources required should remain consistent. To interested policy makers and implementers, information on what types of experts and how much of their time are needed in development will be more useful than simply presenting dollar values of development activities. Reporting information as recommended would enable policymakers and other stakeholders to obtain necessary information for cost projection if they intend to develop similar interventions, or leverage existing technical and content development, or comparing the costs of a digital training technology to alternative approaches. If these digital training programs become commercial products, having itemized developmental costs can help ensure fair and transparent pricing. In addition, the data from this study will demonstrate the extent of costs for developing a digital training program, which will inform the formal cost analysis of our randomized controlled trial evaluating this program (Naslund et al., 2021). These findings can also offer guidance to health systems in rural and underserved communities about what is necessary to build capacity of frontline health workers to deliver a brief psychological treatment for depression through a low cost digital approach.

Table 4.

Recommended checklist for reporting developmental cost of a digital training interventions

Categories	Components and examples
Duration of the development	10 months (January 2020 – October 2020)
Costs of human resources directly involved in development (including volunteers or free experts)	Role and responsibility of each personnel (e.g. medical doctors, training experts, IT/computer staff, video producers, etc.) Time spent by each personnel Cost of time spent by each personnel (e.g. salary and fringe benefits of an expert’s time for developing training contents, costs of a technician’s time for establishing an online platform, costs of a volunteer for translation, etc.) Costs of consultation meetings if any (e.g. travel, room, lodging)
Costs of information technology	Name and quantity of an item (e.g. 12 smart phones) Function of an item (e.g. user testing) Costs of buying new or using existing technologies
Costs of management and supervision	Role and responsibility of each personnel Time spent by each personnel Cost of time spent by each personnel Other related costs not included in above items (e.g. meetings, communications)
Infrastructure-related costs	Office space, supplies, and utilities Vehicle use Other related costs not included in above items

Cost estimates reported in this study have some limitations. Firstly, some infrastructure-related costs (e.g. costs of office maintenance) were difficult to track and could be subject to inherent bias of under- or over-reporting. Secondly, when costing usage of existing items, such as office supplies or laptops, we used straight line depreciation method and assumed zero salvage value. Costs could vary if alternative methods (e.g. accelerated depreciation method) were adopted. Thirdly, over time it may be necessary to make updates or revisions to the digital training program may also need to be updated or revised (McNamee et al., 2016), yet with advancements in information technology and the volatility of pricing and changing costs of items, it may be difficult to predict any additional costs in advance. Furthermore, while the costs reported here are specific to India, it is important to highlight that regardless of setting, the human resources required to replicate the development of this kind of digital training would be comparable, yet the actual costs in dollars might fluctuate across other LMICs based on various factors depending on location, inflation, and other global trends in pricing. Lastly, it is important to recognize that the costs described in this study represent the one-time development costs; as such, there will be recurring costs required for implementation of the digital training program such as server costs, tech support, web hosting, managing health workers, health workers time for completing the training, providing access to smartphones for completing the training, and other data costs while accessing the training program. However, these costs will be modest relative to the development costs reported here, and as the economies of scale rises the operating cost will reduce, thereby offering opportunities to scale up the training to a large number of health workers across a wide range of geographic regions in ways that would not previously have been feasible with conventional in-person training methods.

Despite these limitations, our study contributes to limited existing literature on developmental costs of digital training interventions for CHWs in LMICs. An important strength with our study is that the digital training content was specifically developed for frontline health workers with min 8^th standard education (to ensure minimum literacy level for reading the content), and that no prior experience or exposure to digital technology was necessary among participating CHWs. This supported our goal to make the digital training program as broadly accessible as possible to avoid subsequent modifications or simplifications to the content for future cohorts of frontline health workers. Given that ASHAs are the most widely deployed type of frontline health worker within India, representing approximately 1 million frontline health workers covering nearly every state in the country (Scott et al., 2019; Ved et al., 2019), it is anticipated that if found to be effective, this digital training will have important national policy implications as it can be widely disseminated without requiring significant adaptation of the content or digital platform for accessing the training program. The current digital training program was adapted for use in the Hindi language, and therefore, translation into local languages would be necessary for use in other regions of the country. By providing itemized costs of development, our study offers insights about the resources, including both personnel and materials, and their monetary values that are required for developing a digital training program. Grouping the developmental costs according to the WHO’s health system framework allows interested policy makers and health professionals to understand in which building block the costs might occur, which block will be the main drivers of costs, and when adopting an existing digital training program, how many additional costs might be incurred if revisions are needed. The proposed survey instruments and checklist contribute to the discussions on standardizing tools for collecting and reporting development costs – a necessary step to conduct cost comparison across different interventions and settings. This study also contributes to the WHO’s guidelines on monitoring and evaluating digital health interventions by sharing our experience in rural India.

5. Conclusion

This study advances research on costs of developing digital programs for training CHWs in LMICs. It demonstrates that development costs are substantial, mainly driven by labour costs and expenditure on technologies. Given the progress in digital literacy and increasing affordability and growing access to digital devices in most LMICs, digital technology has become an important means of training the health workforce. Digital training is especially relevant during the coronavirus disease (COVID-19) pandemic, where in many LMICs, CHWs are playing an important role in responding to the pandemic (Denise O. Smith Ashley Wennerstrom, 2020). Due to restrictions on in-person contact, providing remote training to CHWs on service delivery has become essential, which further highlights the importance of continued economic evaluation on developing new or adopting existing digital training programs in LMICs.

Highlights:

1. Total costs for developing the digital training program was 208,814 USD, with 61% for salary and benefits of the personnel involved in development.

2. This paper contributes towards India’s experience to the WHO’s guideline on monitoring and evaluating digital health interventions.

3. Policymakers or implementers considering adoption of digital training program within their respective settings can use this information as a reference point.

4. We propose survey instruments and checklist for collecting and reporting development costs of the digital training program.

5. We also recommend standardizing cost reporting practice for developing digital health interventions.