Evaluation of a village-based digital health kiosks program: A protocol for a cluster randomized clinical trial

Abstract

Background

To address disparities in healthcare quality and access between rural and urban areas in China, reforms emphasize strengthening primary care and digital health utilization. Yet, evidence on digital health approaches in rural areas is lacking.

Objective

This study will evaluate the effectiveness of Guangdong Second Provincial General Hospital's Digital Health Kiosk program, which uses the Dingbei telemedicine platform to connect rural clinicians to physicians in upper-level health facilities and provide access to artificial intelligence-enabled diagnostic support. We hypothesize that our interventions will increase healthcare utilization and patient satisfaction, decrease out-of-pocket costs, and improve health outcomes.

Methods

This cluster randomized control trial will enroll clinics according to a partial factorial design. Clinics will be randomized to either a control arm with clinician medical training, a second arm additionally receiving Dingbei telemedicine training, or a third arm with monetary incentives for patient visits conducted through Dingbei plus all prior interventions. Clinics in the second and third arm will then be orthogonally randomized to a social marketing arm that targets villager awareness of the kiosk program. We will use surveys and Dingbei administrative data to evaluate clinic utilization, revenue, and clinician competency, as well as patient satisfaction and expenses.

Results

We have received ethical approval from Guangdong Second Provincial General Hospital (IRB approval number: GD2H-KY IRB-AF-SC.07-01.1), Peking University (IRB00001052-21007), and the University of North Carolina at Chapel Hill (323385). Study enrollment began April 2022.

Conclusions

This study has the potential to inform future telemedicine approaches and assess telemedicine as a method to address disparities in healthcare access.

Trial registration number: ChiCTR2100053872

Introduction

Digital health technology, which includes various categories such as mobile health (mHealth), health information technology (IT), wearable devices, telehealth and telemedicine, and personalized medicine ¹ has been actively supported by the Chinese government as a way to address critical deficits in China's healthcare delivery system. One such deficit is a scarcity of qualified primary care providers in rural areas. Primary care strengthening is a critical component of China's continuing health reforms (referred to as Healthy China 2030). The overarching goal is to eventually transition the health system to a gatekeeping paradigm. However, the disparity in medical resources between urban and rural populations is a primary barrier to this vision. ² Access to quality care in rural areas is projected to deteriorate as present practitioners retire and outside options for healthcare workers improve in urban areas. ³ Digital health has been proposed as a partial solution to address this growing gap in access due to its potential to improve diagnosis and treatment accuracy to enhance the delivery of healthcare individuals. ¹ However, rigorous evidence is currently lacking on the effectiveness and cost-effectiveness of alternative strategies to deploy these tools in rural areas, as well as the potential behavioral and institutional barriers that may drive a wedge between the potential of these tools and realized outcomes. ⁴

The objective of the Digital Health Kiosks program is to improve care by linking local village doctors with providers located at higher level facilities in the provincial capital of Guangzhou via a telemedicine platform called “Dingbei Doctor.” Additionally, village clinicians will be provided with access to a diagnostic support tool (referred to as “AI doctor”) that was developed by Dingbei and is integrated into the platform. The AI doctor utilizes clinical guidelines to assist clinicians via a chatbot user interface. The hospital has provided equipment—including telemedicine equipment (including devices, software, and internet access) and ancillary medical devices to conduct examinations—to village clinics in 2277 villages officially designated as “low-income” across Guangdong. Over the next 2 years, these systems will be put online as village clinicians are trained on the use of telemedicine software and auxiliary equipment. Once deployed, this will allow village doctors to connect patients to providers at Guangdong Second Provincial General Hospital (GD2H) for primary care visits, specialist consultations, ongoing disease management, as well as referrals to the upper level of health facilities for further diagnosis and treatment.

Despite the need and promise of leveraging digital health to deliver high-quality care in rural areas, there remain important questions regarding the intended and unintended impacts of such a program (e.g. overutilization and increased costs). Moreover, the program could potentially face a number of implementation challenges. A similar intervention by Mohanan et al., which utilizes social franchising and connects rural health providers in Bihar, India, to highly trained physicians via telemedicine, found low use among providers and patients. ⁵ Though the intervention did not improve appropriate treatment rates for diarrhea or pneumonia or related healthcare outcomes, the researchers argue that the implementation warrants further research. ⁵ We acknowledge that, as seen in the study from Mohanan et al., a primary challenge is how best to incentivize providers to use the telemedicine system, given their existing/competing incentives. This evaluation has been designed to identify these issues and strategies to improve the program's effectiveness and cost-effectiveness.

Objectives

The overall goal of this project is to generate evidence on effectiveness and costs to guide implementation of future village-based digital health kiosk systems. To meet this objective, we have the following specific objectives in this cluster-randomized trial:

• Objective 1: Document the causal impacts of the program on the quality of care that patients receive (including appropriate referrals/detection for infectious diseases and index NCDs), patient utilization of primary, tertiary and specialized healthcare, village clinician revenues, public hospital system, and patient health expenditures, and ultimately on patient health outcomes.

• Objective 2: Estimate the causal impact of monetary incentives to clinicians tied to use of kiosks on kiosk utilization and downstream outcomes.

• Objective 3: Estimate the causal impact of demand-side (community-based) social marketing on platform utilization and downstream outcomes.

• Objective 4: Assess the interaction between clinician incentives and demand-side marketing on kiosk utilization (across incentive levels—that is, on the slope of the supply curve).

• Objective 5: Evaluate comparative cost-effectiveness of alternative program design options.

• Objective 6: Evaluate impacts of the program among different key population groups identified by public health policy (elderly, pregnant women, and children).

Methods

Study design

We have designed the evaluation as a cluster randomized controlled trial across 144 townships (the administrative level below prefecture) in two cities in Guangdong province. Quantitative data will be collected from public health insurance administrative datasets as well as house-hold, facility surveys, and case vignettes in a subset of villages.

Study setting and eligibility

The Digital Health Kiosk program is an initiative of the Guangdong Second Provincial Hospital. The objective of the program is to implement the intervention across all 2277 officially designated “low-income” villages in Guangdong Province. For the current study, two adjacent prefecture-level cities were selected for their predominantly rural population and in consultation with local authorities. As of the 2020 census, these two cities had a combined rural population of approximately 4.2 million. Villages within these two cities will be eligible for inclusion in the study if (a) they are on the list of 2277 officially designated “low-income” villages slated to receive the Digital Health Kiosk program, (b) have at least 15 households, and (c) the village doctors consent to participation in the program.

Randomization

The 144 townships will be randomized across the experimental groups as shown in Figure 1, stratified by the 11 total counties across the two prefectures. Within each township, one village will be randomly selected to receive the intervention. The clinical trial design is shown in Figure 1. The 144 eligible villages will be randomized into five study groups in a 2:1:1:1:1 ratio, with the control group (48 villages) given twice as many villages as the other four study groups (24 villages each). We will use Stata16.0(SE) for randomization.

Figure 1.

Randomized cluster trial design.

Interventions

Each township will be randomized into one of the five study groups below:

Intervention group 1: Control (general clinical training only)

48 villages/townships

General clinical training: Training will be provided to one clinician from each village clinic. Over a 3-day course in Guangzhou and supplemental online training, the village doctors will be trained in the following areas:

• First aid knowledge training (cardiopulmonary resuscitation, poisoning, snake-bite management, anaphylactic shock, etc.).

• Treatment of common chronic diseases (breathing, digestion, cardiology, endocrine, etc.).

• Promotion and application of Chinese traditional medicine.

The course will be didactic, and village clinicians will be provided with a handbook/training materials provided by the hospital. Doctors in this arm will not have free access to the AI doctor or telemedicine through Dingbei telemedicine platform (叮呗医生).

Intervention group 2: General clinical training  +  telemedicine platform access and system support

24 villages/townships

Village clinicians in villages allocated to this group will receive the same General clinical training as Intervention Group 1 and will additionally receive Telemedicine platform access and system support.

Telemedicine platform access and system support: The telemedicine intervention will be implemented through a platform called “Dingbei Doctor.” This platform enables village doctors to connect via phone or video conference with physicians, based at the Guangdong Second Provincial Hospital in Guangzhou, for disease diagnosis and treatment assistance. In addition, the platform includes an “AI Doctor” diagnostic support application implemented through a chatbot user interface. The platform will be accessible either in the clinic or on a mobile device. Clinicians will be given (a) a computer and mobile device loaded with the Dingbei Doctor application, (b) training on the use of the platform, and (c) ongoing technical support and service.

Training on the platform will include platform introduction and field practice. All village doctors who attend this training session will install the platform application in their smart phone and register in the platform with the help of technicians. An assessment will be conducted at the end of each training to make sure all the trainees are capable of using the platform.

Intervention group 3: General clinical training + telemedicine platform access and system support + monetary incentives

24 villages/townships

In addition to receiving the same General clinical training and Telemedicine platform access and system support as the above group, clinicians in Group 3 will additionally be offered Monetary incentives tied to their use of the Dingbei platform.

Monetary incentives: In addition, this group has the opportunity for monetary incentives. Providers can receive a set capitation fee of 5 yuan for each verified patient visit that utilizes the telemedicine platform. To limit gaming, each village clinic randomized to an incentive group (Groups 3 and 5) will receive incentives for the first 30 valid patient visits each month. Details of the incentives will be specified in a contract between village clinicians and GD2H. It will be specified that capitation fees may only last for 3 months, after which they will be discontinued.

Intervention group 4: General clinical training  +  telemedicine platform access and system support  +  demand-side marketing

24 villages/townships

In addition to receiving the same General clinical training and Telemedicine platform access and system support as the above groups, a Demand-side marketing campaign will be conducted in villages allocated to Group 4. Clinicians in this group will not receive a monetary incentive.

Demand-side marketing: Townships randomized into this group will also experience a social marketing campaign through multiple channels (banners, posters/flyers, and text messages). The objective is to test (a) whether such a marketing campaign could increase use of village kiosks independent of incentives to clinicians and (b) whether demand-side marketing and supply-side incentives are complements or substitutes. Households in this group will receive information on the availability of village kiosks with the village doctor. This will consist of three components:

1. Village doctors will receive a banner (hengfu) that they can post at the clinic. These materials will be distributed at the General clinical training.

2. Posters and flyers will also be distributed at the General clinical training. The doctors will place posters outside/inside the clinic, and flyers should be placed in the village committee (cunweihui), the community dining halls (shitang) and local grocery stores (chaoshi) with the approval of local authorities. The recruited enumerators can assist with the placement of the posters in the proposed community locations.

3. Text messages will be sent to village residents on a weekly basis for the first month, biweekly for the second month, and then once a month thereafter. Text messages will be sent via GD2H and are limited to 200 words per message. The hospital will alternate between five prepared text messages.

Intervention group 5: General clinical training  +  telemedicine platform access and system support  +  monetary incentives  +  demand-side marketing

24 villages/townships village clinicians in Group 5 will receive General clinical training, Telemedicine platform access and system support, and be offered Monetary incentives tied to use of the platform as above. The Demand-side marketing campaign will also be conducted in these villages.

Intervention groups 2–5

Doctors randomized to groups 2–5 can receive 4 yuan for each verified patient registration to the Dingbei Doctor platform as a part of the training initiative. Each village clinic in an incentive group can receive incentives for the first valid 50 patients.

Theory of change/conceptual framework

Our evaluation of the program will be guided by the Theory of Change (ToC) presented below (Figure 2). This framework maps intervention activities and inputs to intermediate and final outcomes. ⁶ We use this ToC to guide our measurement of key indicators for steps and associated assumptions along the hypothesized causal pathway to impact. ⁷

Figure 2.

Theory of change for telemedicine platform and system.

In addition to the effectiveness of intervention activities, we will also evaluate possible constraints to program impact. These may include the availability of equipment and drugs at the village clinic, the usability of the platform, the availability of referred doctors at higher level hospitals, costs facing villagers from seeking care at higher levels (either too low or too high), or trust in care available through the platform.

Outcomes

Knowledge of telemedicine and increased motivation via incentives to use the kiosks for doctors is hypothesized to affect intermediate outcomes such as improved diagnostic knowledge and performance. Better quality diagnosis and management, especially for referral, may then induce changes in patients’ healthcare-seeking behavior. The ultimate objective is to improve health outcomes, increase patients’ satisfaction with primary care services and decrease their out-of-pocket expenditure.

Outcomes will be assessed through a household electronic questionnaire at baseline and endline (6 months after the start of interventions) and will be continuously assessed via the administrative system over the intervention period.

Primary outcomes

Primary care utilization: Primary care utilization will be assessed via the household survey. At baseline and endline, sample households will be asked to recall how and whether each household member sought care for their most recent illness. For each illness episode, respondents will be asked to recall their care seeking pathway including whether and where they initially sought care (did not seek care/self-treatment, pharmacy, village clinic, township health center, county or city hospital, or online platform), whether they were referred to another provider or chose to subsequently visit another provider on their own, and treatments provided subsequent to each visit until the resolution of the illness episode. In addition to the most recent illness episode, respondents will be asked if they experienced certain combinations of symptoms (corresponding to vignette cases presented to providers, discussed below), and if so, to recall the same information regarding their care-seeking pathway. From responses to these questions, we will calculate the following specific outcomes:

• Proportion of illness episodes where individuals sought initial care in the public health system (from village clinics, township health centers, or hospitals).

• Proportion of illness episodes where individuals sought initial care in village clinics, conditional on seeking formal care in the health system (i.e. whether they chose to bypass village clinics for upper-level facilities).

• Proportion of illness episodes where individuals did not seek any care or self-treated.

• Proportion of illness episodes where individuals sought initial care at a pharmacy.

• The amount of time elapsed for each illness episode from onset of symptoms to initial treatment and illness resolution.

• Total costs incurred by patients for each illness episode.

In addition to care-seeking for specific illness episodes, respondents will be asked how often they sought care in the past month, where this care was sought, and total health expenditure in the preceding month.

Telemedicine platform, and AI doctor utilization: In addition to the above, respondents will be asked to recall if, conditional on visiting village clinics for each illness episode, village clinicians used the telemedicine kiosk technology during their visit. In addition, the kiosk utilization over time will be observed from anonymized platform administrative data. The total utilization per population over the study period and for each month separately will be calculated by dividing the number of patient visits utilizing the telemedicine platform and/or the AI doctor by the total village population.

Secondary outcomes

Village receipt of healthcare services: A comprehensive survey module will assess multiple dimensions of the nature of healthcare services received by patients, following the indicators included in the World Bank's Service Delivery Indicators Surveys. ⁸ These will focus on care received in the village clinic and include: user focus (including wait time), out-of-pocket expenditures, and expectations from and satisfaction with care. These questions will be combined into an index for each of these domains for analysis.

Village doctor revenue/sources: The village doctors’ revenue and sources will be obtained from the village facility and clinician questionnaire at baseline and endline.

Diagnostics and treatment of key health conditions: Diagnostic and treatment knowledge will be assessed for TB, HBV, depression, and angina using structured clinical vignettes presented to healthcare providers. In addition, the quality of care provided for each of these conditions will be assessed through unannounced standardized patient visits.

Data collection

Sample selection

Clinic and household surveys will be conducted in a randomly selected sub-sample. Within each of the 144 project townships, we will randomly select one village per township. Within each village we will conduct a facility survey of the village clinic and administer surveys to all village clinicians. Using a list of all households in the village, 15 will be randomly selected for a household survey. The target size of this sub-sample is therefore 144 village clinics and 2145 households in total.

Inclusion criteria for villages in the survey sub-sample are the same as for the intervention. The inclusion criteria for households are (a) members are current residents in the selected village, (b) at least one household member has lived in the village for more than three months in the past 12 months, and (c) at least one respondent can answer an introductory survey that is initiated by the local, trained students.

For intervention groups (Groups 2–5), we will also collect administrative data from the GD2H platform throughout the course of the intervention.

Instruments

We will draw on three sources of data for the study. Administrative data will be pulled from the GD2H hospital kiosk program records in treatment groups continuously throughout the intervention. At baseline (before the launch of the kiosks) and endline (six months later), we will conduct household, village clinic, and clinician surveys.

Platform administrative data: Data obtained from the Dingbei platform will include information on platform utilization and anonymized visit records (including chief complaints, recorded diagnoses, treatments and recommendations, referral and billing).

Household survey: Baseline and endline household surveys (15 households per selected village) will be used to collect basic household information, health status of family members, health service utilization, telemedicine platform and system (knowledge, awareness, attitude and comments, utilization), behavior of visiting clinicians, perceptions of quality of village and higher-level providers, household income and assets, and drugs and medication list.

Village facility and provider surveys: Baseline and endline surveys will be conducted in each of the selected villages. These surveys will collect the following information:

1. Clinic facility: numbers of village clinicians and other staff, curative care and public health activities, clinic revenue and sources, service volume (numbers of patients per month, cost of medical consultation, number of villages receive service from this clinic facility), medical record keeping, availability of medical equipment and drugs.

2. Village clinicians: basic information (name, gender, birth date/year, household, location, education, start time, and job description), training program status (attendance of general clinical training), utilization of the telemedicine platforms (awareness, capability, attitude, frequency, interaction with township or municipal-level hospitals), income/wage and work time (wage level, compensation and welfare, any other jobs, average work hours per day). At the endline survey, additional information will be collected on activities and revenue over the past year, as well as, for the treatment groups, attendance of training on telemedicine platform and telemedicine system support, satisfaction with the program. In addition, clinical vignettes depicting cases of TB, HBV, angina, and depression will be administered to all clinicians.

Power calculations

The first primary outcome of the trial will be the utilization rate of village clinics (a measure of bypassing behavior). Past surveys have indicated a village clinic utilization rate of 50% for common cold symptoms. We calculated that to detect a 10-percentage point increase in household-level utilization at 80% power, we require 24 townships in the treatment group and 47 townships in the control group with 15 households per township cluster per group for each pairwise comparison assuming an adjusted ICC of 0.015.

The second primary outcome of the trial will be the correct case management rate of interactions between vignettes administered and village clinicians. Past surveys have indicated a correct case management rate of 22%. We calculated that to detect a 30 percent point increase in village level at 80% power, we require 29 townships in the treatment group and 58 townships in the control group per group for each pairwise comparison assuming an adjusted ICC of 0.025.

Data analysis

Impacts on intermediate and final outcomes will be assessed by comparing differences in means of these variables across the study arms using standard t-tests. To increase power, we will also estimate impacts using linear ordinary least squares (OLS) regressions. The dependent variable in these regressions will be outcome at endline, and this will be regressed on dummy variables indicating the treatment assignment of each village, randomization strata (county) fixed effects, and the baseline value of the dependent variable (when available). Standard errors will be adjusted for clustering at the village level. We will compare the effects of each treatment cell to the control group and between each for the four treatment groups. To account for multiple hypotheses, we will compute q-values that control the false discovery rate.^9,10 Those in the data management team will be blinded. Data will be unblinded after data analysis is complete.

Ethics and dissemination

We have received multi-center ethical approval from Guangdong Second Provincial General Hospital (IRB approval number: GD2H-KY IRB-AF-SC.07-01.1), Peking University (IRB approval number: IRB00001052-21007-免), and the University of North Carolina at Chapel Hill (IRB approval number: 21-0549). This study is also registered under the clinical trial registry in China (ChiCTR2100053872). The study personnel and management group will ensure that the study is conducted within professional ethical guidelines. Informed consent will be obtained from all village clinicians prior to enrollment. Consent will be obtained from households for survey participation at the time of the baseline survey. Confidentiality was assured for every survey respondent. All stored patient data will be encrypted on Dingbei, and any information used for research purposes is de-identified. Patient information, test results, and other important medical information stored on Dingbei are only accessible to those with proper clearance. Only study members can access identifiable data. All identifiable data will be deleted once the study is complete. The results of this trial will be published once the study is concluded.

Results

We started study enrollment next month, April 2022.

Discussion

Telemedicine and digital health are poised to bridge the growing gap in quality primary healthcare in rural China. However, the effectiveness of these methods is under researched and potential barriers are still unknown. The Digital Health Kiosks program presents a solution to this gap in quality care by connecting local village doctors to higher level facilities, while our evaluation addresses the gap in literature on effectiveness and implementation. However, there are a few factors that could undermine the benefit of kiosks, including low participation from the doctors, low demand from patients, and spillover.

Low physician participation could be due to several different perceptions. If the village doctors believe that the care they provide is sufficient and that the platform provides little additional benefit to patients, they could be reluctant to use it. Additionally, though the platform helps provide superior care, village doctors may decline its services because they believe there might be a revenue reduction in the sale of drugs and services due to the platform. Village clinicians may also believe that the use of the platform would adversely affect their standing in the community. If village doctors receive pressure from township hospitals not to use the platform (township health centers may believe this platform would reduce the number of patients or their own revenue), this could also cause low participation.

Villagers may also be disinclined to use the platform if they believe that the kiosks do not enable doctors to provide care that is of higher quality than what they receive by bypassing to the township or county (or the benefit from perceived quality and the costs of seeking care at higher levels). Alternatively, they may choose to seek care through an online hospital or high-level/specialized hospital directly.

Financial incentives and demand-side marketing campaigns in Groups 2–5 may be able to address both challenges. Village doctors with incentives may be more likely to use the platform with existing patients and increase local demand for their services. The marketing campaign will increase villagers’ awareness of the kiosks’ potential health benefits and ease of access, positively affecting participation. Incentives and marketing could either be complements or substitutes depending on the nature of incentives and perceptions on the quality of care, costs, and revenue with and without the kiosk. However, one limitation of the marketing campaign is the possibility of spillover. The study team has attempted to limit spillover through employing small-scale approaches that are easier to control by village: posters, flyers, banners, and text messages to villagers in intervention groups 4 and 5.

Despite limitations, this study has potential to provide rigorous evidence to inform similar telehealth approaches in China and elsewhere. Our research will also provide implications and recommendations for health policymakers to design innovative, centralized telehealth service delivery models for rural areas. In addition, our multi-level interventions target providers through incentives and communities through the social marketing campaign. It is crucial to incentivize health service providers and change their behaviors to adopt new technology from the supply side. Our research on the financial incentive intervention will provide new evidence on the utilization of the telemedicine system and its impact on the behavior change of doctors and further contribute to designing village doctors’ salary system and telemedicine payment mechanism. Moreover, telemedicine has been proposed as a crucial method to address the inequality of medical resources between urban and rural areas. So empowering existing village doctors and enhancing the awareness of villagers to utilize the kiosks will potentially improve healthcare quality and health equity in rural areas.