Assessment of monitoring and online payment system (Asha Soft) in Rajasthan using benefit evaluation (BE) framework

Abstract

Context:

e-Health programs are implemented assuming that e-health/digital health can prove beneficial but pieces of evidence for assessing the actual benefits of e-health programs are lacking.

Aims:

To utilize the benefit evaluation (BE) framework to assess Asha Soft, which is an online payment and performance monitoring system initiative taken by Rajasthan.

Settings and Design:

BE of Asha Soft in Rajasthan.

Methods and Materials:

BE of ASHA Soft was done using scoping review with consultation exercise. The rationale behind using this methodological framework is to contextualize knowledge of the current state of understanding within BE framework practice contexts. The themes used for data compilation and analysis were based on three broad dimensions of BE framework namely, health information technology quality, use, and net benefits.

Results:

The state of Rajasthan has been the first in the country to start an online system of payment and monitoring of ASHA workers, through Asha Soft. It has administrative and supportive functions. Its simple and easy to use graphical user interference helps users to make accurate data entries and obtain desired monitoring and analytical reports. It has attributed to the availability of data on various parameters which help decision-maker to decide about the performance of ASHA worker and has brought a positive impact on the work performance of ASHAs. This online payment and monitoring mechanism has argumented motivational level and intention of use. The program has optimally utilized available human resources and no apparent monetary cost was involved in developing this software.

Conclusions:

This scoping study using the BE framework has provided evidence on the potential benefits of Asha Soft adoption in Rajasthan. It is recommended that future in-depth assessment of other e-health initiatives could be undertaken to guide the decision making.

Introduction

Rajasthan is one of the states of India in which numerous e-health initiatives are ongoing under the National Health Mission, Govt. of Rajasthan. Asha Soft, that is, online payment and monitoring system for accredited social health activists (ASHAs) in the Rajasthan state is one such initiative.[1] ASHAs are community health workers appointed by the Ministry of Health and Family Welfare (MoHFW), Government of India under National Rural Health Mission (NRHM)[2] with the sole purpose of working in close contact with the community as a primary care provider.[3] ASHA is a grass-root-level community health worker which functions as a primary health care facilitator and service provider. Asha Soft was launched in December 2015 with an objective to ensure timely and transparent online payment to ASHAs and for monitoring their performance on various parameters.[1] Reduced cost of computer technology, telecommunications, and internet offers an opportunity to employ online e-health systems which were previously only possible in higher-income countries.[4] Most of the e-health program is implemented assuming that e-health/digital health can improve beneficially but most of the time it lack pieces of evidence. The pieces of evidence for assessing the actual benefits of e-health programs are lacking and, moreover, for which there is no standard framework for assessing the actual benefit of these digital health ventures.[5] A large number of e-health system which is being deployed give rise to the importance of the new pragmatic framework to demonstrate the benefits of these program and guide adaptation decision. There are plethora of different kinds of literature that have been proposed to assess the e-health system but pieces of evidence about their actual implementation in a real-world scenario are lacking.[6] The handbook of e-health evaluation proposed six empirical frameworks, which can be used as a roadmap for e-health practitioners to describe the factors that influence the design, implementation, usage, and effect of the e-health system in a health care setting. Out of these frameworks, we choose to use the first proposed benefit evaluation (BE) framework, which has been applied in more than 50 evaluation studies throughout Canada, New Zealand, USA, Australia, etc., This framework has also widely cited in Canadian evaluation studies and reports, but the pieces of evidence about the use of this framework are lacking in India. Since the BE is considered as a proven framework for evaluating the success of the e-health system across different health care settings.[6] This study was planned to utilize this framework to assess Asha Soft, which is an online payment and performance monitoring system initiative taken by the Rajasthan state of India.

Subjects and Methods

BE of ASHA Soft was done using a scoping review of available literature with consultation exercise.[7] Primary and secondary data were collected to achieve the study objectives. Scoping review design for collecting secondary data was used to assimilate all the relevant and unstandardized information from all sources. An internet-based search of peer-reviewed and non-peer-reviewed literature and was extended to web-based portals, newspaper articles, profit, and not-for-profit organizations’ websites, relevant documents, databases, and reports to ensure that all relevant information was obtained. The design of this scoping was developed utilizing a methodological framework provided by Arksey and O'Malley's and was supplemented with required modifications.[8] Other than the paucity of literature available, one of the intentions of including the literature other than commercially published literature was also to facilitate a more balanced view of the evidence reducing publication bias. Feedback interviews of key informants at managerial level and the end-user level were conducted to seek additional information needed for accomplishing comprehensive BE assessment. The rationale behind using this methodological framework is to contextualize knowledge of the current state of understanding and then describing it within the BE framework practice context.[6] Approval was obtained from Institution Ethical Committee of All India Institute of Medical Sciences, Jodhpur, Rajasthan on September 17, 2018.

The methods adopted for identifying literature in this study were focused to achieve in-depth and broad results corresponding to the BE framework. After identification of objectives, a web search of relevant documents on Asha Soft initiatives was performed using keywords like “Asha Soft” and “monitoring and online payment system in Rajasthan.” Documents dating from 2014 to the present were reviewed. Stages of the framework followed while conducting the scoping study are mentioned in Table 1. This process was iterative rather being linear, engaging the authors within each stage in a reflexive way. Steps were repeated as required to ascertain that the literature is covered in a comprehensive way. The themes used for data compilation and analysis were based on three broad dimensions of BE framework namely, health information technology (HIT) quality, use, and net benefits.[6]

Table 1.

Framework used for conducting scoping study

Stages		Strategy
Stage-1	Identifying the research question	Broad search strategy with clearly defined concepts and continuous refinement based on
		Knowledge obtained from the existing literature about monitoring and online payment system (Asha Soft) in Rajasthan
Stage-2	Identifying relevant studies/documents	Search for research evidence via different sources (peer-reviewed and non-peer-reviewed literature)  electronic databases  references in the literature  hand-searching key journals reports from different organizations, workshop, and conferences
Stage-3	Study selection	Inclusion and exclusion criteria developed post hoc, after gaining familiarity with the literature  Iterative team approach multidisciplinary expertise group consultation/consensus
Stage-4	Data charting	Sifting, charting, and sorting material as per key issues and themes for interpreting data.
		dimensions of benefit evaluation framework narrative/descriptive approach
Stage-5	Summarizing and presenting results	An analytic or thematic framework to guide the narrative account of existing literature based on the benefit evaluation framework.
Stage-6	Consultation exercise	To provide valuable insights and “added value” to the literature review. Key informants feedback interviews

Results

The findings of this review are presented as a descriptive summary using the thematic framework. Themes-based BE Framework was used as a conceptual scheme for understanding Asha Soft adoption outcomes. BE evaluates the adoption and success of e-health systems on the basis of three conceptual dimensions: HIT quality, use, and net benefits.[6] Table 2 shows a summary of BE measures.

Table 2.

Summary of benefit evaluation (BE) measures adopted from the handbook of e-health evaluation (University of Columbia, 2016).[6]

Dimension	Category	Details
System	Functionality, Performance, Security	Type of features available
		Level of features available
		Accessibility
		Reliability
		System response time
Information	Content, Availability	Accuracy and relevance
		Completeness and comprehensiveness
		Timeliness, reliability, and consistency of data
Service	Responsiveness	Extent and adequacy of implementation
		Training and ongoing support availability
Use	User behavior and pattern, Self-reported use, Intention to use	Type, frequency, duration, location, and flexibility of actual and perceived usage
Satisfaction	Competency, User satisfaction, Ease of use	User knowledge, skills, and experience
		Extent of user gratification
		User-friendliness and learnability
Net benefits	Care quality, Access, Productivity	Risk management, monitoring, and overall safety
		Appropriateness and effectiveness
		Health outcomes, ability of patients, and providers to access services
		Efficiency, care coordination, and net cost

Overview of Asha Soft

Asha Soft was launched by the Government of Rajasthan on 25^th December 2014. This e-health program was developed for the seamless financial inclusion of frontline health workers, ASHAs. Rajasthan was the first state of India to start an online system of payments to ASHA workers.[1] This online system aims to capture beneficiary-wise details of services delivered by an ASHA to the community, provide online payment of ASHA in their bank accounts based on the details, and generate various kinds of reports to aid in monitoring the progress of various programs.[9] This system was developed by National Informatics Centre (NIC), Rajasthan and initiated by the National Health Mission (NHM), Rajasthan.[1]

Health information technology quality (HIT)

Based on BE framework the first conceptual dimension which is considered for assessing Asha Soft adoption in this study is HIT quality. HITs have three quality dimensions: system, information, and service.[6]

System quality of Asha Soft refers to the technical characteristics of HIT, that is, functionality, performance, and security of the system. ASHA Soft is an operational and relational E-HRM which is concerned with administrative and supportive functions.[10] Administrative functions include incentives management, Asha sahyogini personal data, and supportive functions include a record of training, performance management.

Functionality is assessed by type and level of features available. The development of software was done using ASP. Net (Front End) and MS SQL 2008 R2 (Back End/DB).[11] Features available in Asha Soft includes master entry, line list, verification, sanction, report, and release of payment. Master entry includes verification of account details and defines user permission for administrator; data entry; and data verification of records, generates sanction, and releases the amount to the ASHAs. Line list covers the major service categories, namely, maternal and child health, immunization, family planning, national health programs, and meetings of ASHA workers.[9] Data entry operator performs the online entry of certified Asha claim form of mention services provided by ASHAs. The verification feature enables the verification of information entered in the line-list menu. Sanction feature releases the sanction, or generates fund transfer order for mention services to the verification. Release payment is done accordingly by chief medical and health officer (CMHO) using Digital Signature Certificate (DSC). Payment information to ASHA is sent by SMS to the ASHA's registered mobile number.[12]

Performance is assessed by technical behavior through accessibility, reliability, and system response time of the system. The project is successfully implemented across 33 districts of Rajasthan.[1] Other states like Maharashtra, Punjab, and Karnataka are keen to replicate the same in their respective states. Direct beneficiaries of the project are more than 47,000 ASHA workers.[1] This e-health program has initiated the financial strengthening of the health system in Rajasthan. It also has provided a transparent benefit transfer that motivates frontline health workers to work with more dedication and enthusiasm. Timely payments to ASHAs are in place which has curbed the menace of nontransparency and delay in payments.[11] Few technical glitches were also identified while interviewing key informants. It seems that when ASHA shifts to another AGW center, updating the new details take time from central servers, as a result, ASHA's work/data are not uploaded on ASHA Soft. This gives error while making payment. Also, there is popping up of data verification reminders even after comments have been placed in context to payment verification comments.

Asha Soft system is a secure system and has the ability to protect the integrity as well as misuse of the captured data. It also ensures authorized access only. Users in this system are categorized as an administrator, data-entry, and report-viewer at state, district, block, and health institution level.[9] Users can access the various features of the software based on the permission and rights provided. The release of the payment to the beneficiaries is provided through DSCs which ensures payment in a timely and secure manner.[13]

The information quality of Asha Soft refers to the data characteristics of the system, its content and availability.[6]

Content refers to the accuracy, reliability, completeness, and comprehension of the data in Asha Soft. The beneficiary line list capturing RCH activities (ANC, Immunization, FW) is being used for performance monitoring and calculation of the amount to be paid to ASHA. Asha Soft is also closely integrated with the PCTS (pregnancy, child tracking, and health services management system) which has improved the credibility of the data and reduction in data redundancy.[14] Asha Soft provides a profile of registered ASHA such as demographic details, qualifications, services, and activity provided Incentives details, training, and skills details and account verification status.[14]

Availability refers to the prompt and timely access to data when and where needed through Asha Soft. Asha Soft is available at all public health care facilities and enables online download and submission of the claim form.[9] Desired analytical reports can be accessed easily through the online platform. These reports include details of ASHAs’ work performance, ASHAs which are not functioning or getting exceptional high payments, PHCs, and CHCs with low-performing ASHAs, blocks which are not performing as per basic requisites of RCH, districts having low-ASHA performance indicators and programs that are usually not taken up by the ASHA workers, etc., It was also reported during consultation exercise that all reports corresponding to ASHAs’ work are not available in one single tab but they could be obtained from different tabs.[9] Claim form submitted is verified by ANM and online data entry of ASHA claim form with verification on ASHA Soft by ASHA supervisor/data entry operator is done between 26^th and 30^th of every month. This is followed by the release of sanction by MOIC with the help of LHV/accountant by the 4^th of the next month and finally, release of payment (using DSC) by CMHO between 5^th to 7^th of the same month. Additionally SMS sent to ASHA's mobile number as soon as the payment is transferred.[11,13] It was apparent from the views of informants that in case of payment rejection of ASHAs it is difficult to track the cumulative rejected transactions and a separate listing for the same could be added where all rejected payments can be seen.

Service quality pertains to the responsiveness of Asha Soft in terms of HIT implementation, training, and ongoing support by staff. Training and workshops for the software are organized at the state, district, and block-level to make the staff familiar and comfortable with the use of the software. ASHA also participates every month at sector/PHC level in a meeting for their work review and claim forms submission.[11]

Use and user satisfaction

Based on BE framework the second conceptual dimension which is considered for assessing Asha Soft adoption in this study is use and user satisfaction.

Use dimension is categorized into three categories, namely, usage behavior and pattern, self-reported use and intention to use.[6] A claim form that has monthly details of ASHA's activities and is submitted at the end of the month for verification by ANM. These forms are forwarded to the respective PHCs, CHCs and block for carried out data entry.[10,13] Details in claim forms are entered into Asha Soft by data entry operator for verification. Sanctions made for every service category are generated by in-Charge Medical Officer for all ASHA in their jurisdiction which is maintained online record and reference.[11] Finally, Fund Transfer Orders are generated by concerned CMHO using DSC at the district level and online payments are done by the 7^th of the upcoming month. This robust mechanism has helped in reducing the delay in payment as compared to a conventional system. Transparency of the process has also augmented the motivation level and intention to use.[10,11,12,13]

The user satisfaction dimension is categorized into three parts i.e. competency, user satisfaction and ease of use.[6] Before implementations of Asha Soft, there was no time-bound well-structured procedures for appraising the performance and payment monitoring for ASHAs due to which the process was prone to many errors.[13] Orientation and training are imparted to all involved manpower and time-bound paperless work in Asha Soft has minimized the errors in the process of payment and performance monitoring. Work-related satisfaction level of front line workers has also substantially improved and the error-free unbiased system has developed the sense of job security in them.[11,13] It has a simple and easy to use graphical user interface (GUI). This simple user-friendly interface of this online system helps the data entry operators to make accurate data entries from the ASHA claim form. Even this system has empowered grass-root workers ASHA's to access the online system. Desired monitoring and analytical reports can be easily and seamlessly generated through the system.[11,12,13,15,16] During consultation exercise, most of the ASHA workers were seem to be satisfied with the implementation of Asha Soft compared to previous traditional paper-based multi-level processes as there is a timely payment with transparency. Apart from all the benefits, they were bit concern about reporting grievances as they were not aware of any online grievance reporting in context to Asha Soft.

Net benefits

The third conceptual dimension for assessing Asha Soft adoption in this study is “net benefits.” This dimension has three types of measures that are based on care quality, access and productivity.[6]

Care quality is further subcategories into safety, appropriateness and effectiveness, and health outcomes.[6] To reduce the risk of faulty overpayments and related discrepancies, physical verification of claim forms is done thoroughly by the state and district team (in case of higher claims). Various reports like head wise payment, district wise payment etc., which are generated through Asha Soft act as checkpoints prior to the release of payments and ensure seamless payment and performance monitoring.[10,11] For timely and transparent online payment to ASHAs service level agreements have been signed with a Nationalize bank for secured integration of bank server with department server for ensuring the real-time realization of payments to ASHAs. Payment by Asha Soft has enhanced the motivational level of ASHA shayogini and performance monitoring as per line listing needs of software has resulted in better performance of ASHA's, thus contributing to improving health indicators of the state.[13] With the adoption of ASHA Soft, in merely one year, the availability of data pertaining to maternal and child health was drastically increased. Data availability of pregnant females’ newborn babies increased by 31% and 19%, respectively. There was also a significant increase in home-based neonatal care (HBNC) which resulted in better care of infants and timely referral of ill neonates.[10,11,13,16]

Access and productivity are assessed by participation efficiency and net cost. Asha Soft has brought ASHA into an active state of participation and has helped in streamlining various health service delivery activities. After the implementation of Asha Soft significant positive work perception is seen in Asha shayogini, as this system has improved their quality of work. Confidence and motivation level of the ASHA, ANM, computer operator/PHC ASHA supervisor/information assistant, medical officer in-charge (MOIC), and CMHO has improved due to the implementation of Asha Soft.[11] In Rajasthan since the introduction of Asha Soft, the delays in payment have reduced considerably. Different level monitoring (like verification of records based on PCTS) also helps in verification of the work of ANM along-with that of ASHAs. Asha Soft has attributed to the availability of data on various parameters which help decision-makers to decide where ASHAs are performing better or need improvement.[15,16] This provides a requirement of training to be provided to ASHAs in low performing areas. This e-Health venture has brought a considerable positive impact on the work performance of ASHAs. It has also helped to access and monitor the work performance of ASHAs and current health status in a matter of few clicks. This e-Health program was developed by utilizing the resources of NIC (National Informatics Center) and the department used its existing server for the deployment of Asha Soft software.[12] The nationalized bank has provided free support in online payment to ASHA and its integration with Asha Soft. This program has optimally utilized available human resources at different levels (state to the village) and except training cost, no apparent monetary cost was involved in developing this software.[10,11,16,17,18]

Discussion

This study is an attempt to describe the benefits evaluation framework for the Asha Soft e-Health system which is currently ongoing across the state of Rajasthan, India. This framework is used with the perception of illustrating the relationship between the implementation of Asha Soft as an effective solution, its adoption, and the resulting impacts.

Findings from various studies, reports, presentations, and consultation exercises (key informant interviews) were triangulated to provide a comprehensive evaluation report. Similar work was done by PricewaterhouseCoopers (PwC) study which evaluated the benefits of electronic health records use in community-based healthcare and RNAO nursing peer leader network to appraises the NPLs’ effectiveness at the project and national/program levels.[6,19]

Benefits Evaluation framework in this study showed Asha Soft effectiveness in influencing: the quality of the decision support in terms of the system, information and quality of service; the end-user experience quality and the net benefits derived from the program which relate to safety, adoption, health outcomes, efficiency, and cost.[18] Many studies which were conducted prior to the launch of Asha Soft in context to satisfaction levels of the ASHA Sahyogini revealed that delays and complexities of the payment processes were hampering the motivation level which in turn were impeding the health outcomes but the drastic difference is seen in satisfaction and motivational level of Asha workers in the studies done after the launch of Asha Soft.[10,13,16] Asha Soft e-health system has been successful in ensuring prompt and transparent online payment to ASHAs, effective monitoring of work performance and reduced paperwork and workload at multiple levels.[10,12,13,16]

There are some conceptual limitations related to the BE framework. This framework does not include contextual and organizational components that could have a significant impact on the success of e-Health systems. Dimensions, categories and measures are concepts that have been established from pragmatic evidence over time that can be interpreted differently based on one's experience and perception of the meaning of these terms. This study tried to minimize these limitations by using a scoping review of available literature including consultation exercise.[5,20]

Conclusion

This scoping study using BE framework has provided evidence on potential benefits of Asha Soft adoption in Rajasthan. It is recommended that future in-depth assessment of other e-health initiatives could be undertaken to guide the decision making.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.