How much did it cost to develop and implement an eHealth intervention for a minority children population that overlapped with the COVID-19 pandemic?

Abstract

Background

eHealth interventions using active video games (AVGs) offer an alternative method to help children exercise, especially during a pandemic where options are limited. There is limited data on costs associated with developing and implementing such interventions.

Objectives

We quantified the costs of delivering an eHealth RCT intervention among minority children during COVID-19.

Methods

We categorized the total trial cost into five subcategories: intervention material development, advertising and recruitment, intervention delivery, personnel salaries, and COVID-19-related equipment costs.

Results

The total RCT cost was $1,927,807 (Direct: $1,227,903; Indirect: $699,904) with three visits required for each participant. The average cost per participant completing the RCT (79 participants/237 visits) was $24,403 (Direct: $15,543; Indirect: $8860). Due to no-shows and cancellations (198 visits) and dropouts before study completion (61 visits; 56 participants), 496 visits had to be scheduled to ensure complete data collection on 79 participants. If all 496 visits were from participants completing the three-visit protocol, that would correspond to 165 participants, bringing the average cost per participant down to $11,684 (Direct: $7442; Indirect: $4242). Of the subcategories, intervention material development accounted for the largest portion, followed by personnel salaries. While the direct COVID-19-specific cost constituted <1% of the entire budget, the indirect effects were much larger and significantly impacted the trial.

Conclusion

RCTs typically involve significant resources, even more so during a pandemic. Future eHealth intervention investigators should budget and plan accordingly to prepare for unexpected costs such as recruitment challenges to increase flexibility while maximizing the intervention efficacy.

1. Introduction

Physical activity (PA) plays a critical role in the battle against the worldwide epidemic of childhood obesity, which disproportionately impacts Black and Hispanic populations [16]. Successful activity-related interventions have resulted in a reduction in sedentary behavior and an increase in energy expenditure to combat obesity. [4]. An interesting alternative to traditional physical activity interventions such as sports or physical education classes is eHealth interventions using active video games (AVGs). AVGs have been increasingly employed in research as they can encourage PA behavior in a format that is accessible, engaging, and exciting [8]. Previous studies have found that with the appropriate narrative immersion into a storyline, AVGs can elicit increased minutes of moderate-to-vigorous physical activity (MVPA) in children [17]. AVGs have a promising combination of enjoyment, appropriate exercise intensity, and potential for sustainable involvement that may offer an exercise remedy for the obesity epidemic; however, more work is needed to understand their utility [2].

The aim of this paper is to calculate and categorize into distinct subcategories the direct costs of carrying out an eHealth intervention trial. Our goal is to provide insight to future investigators as they plan and budget studies that will further aid in our knowledge of AVGs or other eHealth interventions. We also share the obstacles encountered in pursuit of an eHealth intervention trial so that future research proposals are more informed by our experience. We hope to encourage research toward a better understanding of what it costs to run an eHealth study and the most cost-effective way to approach it.

To study the ability of AVGs to elicit PA, our team conducted an eHealth randomized controlled trial (RCT) from late 2019 through May 2022. At this time, the study data collection has been completed and data analysis for assessing primary outcomes is ongoing. Our protocol has been registered at ClinicalTrials.gov (NCT04116515) and fully detailed in Alon et al. [1], with the updated protocol regarding COVID-19 pandemic-related changes in Monashefsky et al. [13]. In summary, we planned to test the long-term effect of narratives on players' MVPA levels as well as multiple other health outcomes, with the hypothesis that adding narratives to the AVGs would increase time spent playing AVGs and thus increase PA levels.

Our study was a six-month randomized controlled single-blind trial with participants randomized into three groups attempting to specify the added contribution of a narrative: condition A [Narrative + AVG] received an Xbox with an intervention narrative animation, Ataraxia, and six AVGs during the six-month participation; condition B [AVG only] received access to the same Xbox model and games without Ataraxia; and condition C [Control] that did not receive an Xbox or games until the conclusion of their participation. The narrative animation of Ataraxia was created for this trial. Ataraxia contains elements of stories that previous studies in our lab identified as appealing to children [11] and combines those elements into a storyline that pairs with six commercial AVGs available on Xbox. The story takes place over 72 cartoon episodes in six seasons, depicting a post-apocalyptic science fiction plot further detailed in Alon et al. [1].

Each participant, regardless of condition, was required to undergo three in-person data collection visits over the course of six months. Each visit included measurements for height and weight, a fasting blood draw, a Dual-energy X-ray absorptiometry (DEXA) test for body mass and composition, a cognitive test, multiple questionnaires, and an accelerometer set up to track activity for the next seven days. The visits were spaced two-to-three months apart to track the longitudinal effects of active video gameplay. Due to the pandemic, we allowed flexibility in times of assessment. The main trial was initiated on January 11, 2020, and was suspended on March 17, 2020, due to the COVID-19 pandemic. It resumed on September 12, 2020 and continued until May 1, 2022.

In the setting of severe funding constraints on public health research [15], the execution of this study was relatively expensive, especially when it overlapped with the pandemic. Thus, it was important to explore what strategies could be implemented in an RCT budget to increase spending efficiency and adjust enrollment expectations while still retaining scientific rigor, as the most successful RCTs are the ones that are robust and flexible enough to adapt to these unexpected issues, including monetary ones [3]. This is especially relevant to our RCT, as we implemented a longitudinal study consisting of three visits over six months, requiring a relatively large sample size to accurately analyze as well as account for an increasing number of dropouts as the study continued.

The recent and ongoing COVID-19 pandemic has also further complicated our research progression as it not only disproportionately impacted the Black and Hispanic populations [6], who were our primary study groups, but also significantly increased their financial stress, which might have prevented them from traveling to visits [19]. Therefore, it was highly likely that these unprecedented pandemic-related factors might have led to substantially more no-shows or dropouts.

RCTs are expensive. Further research on how to increase the monetary value and reduce the costs would be beneficial [9]. However, there is limited existing research on the costs of active video game RCT interventions, especially during the COVID-19 pandemic. There is not enough available data and transparency regarding the costs of different components of an RCT, such as advertising and recruitment versus intervention delivery versus personnel. There is also a lack of data on the cost of intervention material development, which we defined as the cost it takes to design and create an eHealth intervention to promote physical activity.

This paper aims to address these issues by categorizing the direct cost it took to carry out an AVG intervention into multiple subcategories. We aim to provide insight for future investigators as they estimate the various financial costs of their proposed RCTs. Within our sample population, almost half of families had annual incomes lower than $40 K and all children identified as persons of color, with around 80% identifying as Black or Hispanic. In addition to costs, this paper also shares unexpected obstacles, such as the disproportional impact of COVID-19 on our sample population of primarily underprivileged children, to provide examples of the many ways in which the pursuit of a research protocol may differ from original expectations. A better understanding of what eHealth interventions actually cost may create more informed research proposals and expectations, leading to increasingly efficient research environments with potentially increased intervention efficacy.

2. Methods

We have been actively monitoring costs using a master Excel workbook with over 20 sheets to track the cost of each spending category, which has been validated against the PI's internal financial report each month. The total cost can be retrieved from NIH RePORTER with the last author's name over the years.

We organized the cost related to the RCT intervention into five primary subcategories: (1) intervention material development, (2) advertising and recruitment, (3) intervention delivery, (4) personnel salaries, and (5) COVID-19-related costs. Next, we reported the total amount of money allocated to each area, which was summed to calculate the overall study cost.

To identify per-participant costs, the total cost of the trial was divided by 79, the number of participants who successfully completed all three required visits. While additional 56 participants completed either just one (51 participants) or two visits (5 participants), which is important for intention-to-treat analyses [7], we did not include them in the denominator since we wanted to estimate the cost per completed participant. To identify the cost per scheduled visit, we took the total study cost and divided it by the number of total visits scheduled (496 visits), including those in which participants did not show up or canceled at the last minute. We included the no-show visits in our calculations because staff time was already spent regardless of the attendance of the participant as long as they confirmed they were coming one week before and then again 48 h before their visit; thus the appropriate resources and time were blocked out for them and could not be easily recovered. We separately report the potential costs per participant should all of the study visits have been part of the three-visit intervention and generated valid data for the goals of the RCT.

3. Results

3.1. Total trial costs

The terms “direct” and “indirect” refer to the NIH definitions of costs directly allocatable to the study (direct) with additional funds allocatable to institutional overhead (indirect), respectively. The total cost of the RCT was $1,927,807 (Direct: $1,227,903; Indirect: $699,904), which included the creation of an animated show that cost $544,000 (Direct only; Indirect: $310,080) to create before the start of the trial. The animated show was deployed in the intervention. Aside from the intervention material development, the total cost solely dedicated to the RCT was $683,903 (Direct only; Indirect: $389,825).

3.2. Cost per subcategory

The total costs dedicated to material development totaled $546,288 (Direct only; Indirect: $311,384). This accounted for funds dedicated to scripting and creating the animated show Ataraxia (72 episodes in six seasons; around three minutes per episode) with a professional media production company ($544,000) [12] as well as additional research dedicated to exploring and qualifying Xbox active video games on the market and available for use within the trial. The intervention material development was planned at the onset of the R01 budget planning with $500,000 dedicated to the material development. This was later increased to $544 k to finetune the material for the actual intervention deployment, with $50 K being dedicated to initial story selection, $294 K for investigating the effective plot design and character presentation as well as initial animation production, and $200 K for continued work for completing the production of the six seasons. Intervention material development was completed before the pandemic and therefore its costs were not affected by the pandemic.

Costs for advertising and recruitment totaled $70,937 (Direct only; Indirect: $40,434), which included funds dedicated to recruitment personnel (wages of research assistants who were exclusively working on participant recruitment), printer hardware and supplies, print order costs for information flyers and letters, and incentives such as wristbands and erasers given to potential participants. Research assistants were paid on an hourly basis to print and prepare mailings to a pre-made list of eligible participants provided by Boston Children's Hospital.

Costs for intervention delivery totaled $158,436 (Direct only; Indirect: $90,309), which included funds dedicated to assessment equipment costs (DEXA machine computer, printer and maintenance, wet lab materials, privacy screen, iPad, laptops for assessment, accelerometers and accessories, cognitive testing button board, and weight scale), research supplies (snacks for participants and office supplies), technology intervention delivery (Xbox consoles, Kinect sensors, Kinect adapters, and AVGs), and participant rewards (up to $100 Amazon gift cards per participant). The DEXA and blood centrifuge machines were in the research space prior to the RCT and were not separately purchased. An annual maintenance fee was charged to the grant and factored into this category. Research personnel were trained by the DEXA's manufacturer (General Electric, Boston, MA) to be certified to operate the machine and their training costs were factored into this category.

Costs for personnel totaled $449,012 (Direct only; Indirect: $255,937), which included funds to support research staff salaries over the course of the trial, excluding the principal investigator and other co-investigator salaries as they were not directly involved in data collection. Research staff included all personnel involved in the coordination, data collection, and management of the study. This total also excludes research assistants solely hired for in-person recruitment, as their wages were already included under the advertising and recruitment subcategory.

COVID-19-related specific equipment costs totaled $3229 (Direct only; Indirect: $1841), which included funds dedicated to cleaning products, COVID-19 rapid tests, contactless thermometers, and Personal Protective Equipment (PPE) (disposable face masks and face shields) for both research staff and participants.

Table 1 summarizes the direct cost of each of the five subcategories along with their percentages.

Table 1.

Break down of total direct costs into each subcategory.

Category	Total Cost ($)	Percentage of Total (%)
Intervention Material Development	546,288	44.5
Advertising and Recruitment	70,937	5.8
Intervention Delivery	158,436	12.9
Personnel Salaries	449,012	36.6
COVID-19 Related Equipment Costs	3229	0.3
Total:	1,227,902	100

3.3. Per participant costs

Our study had 79 participants (out of 154 total recruited sample, 135 showed up for their first visit and completed the consent and assent forms) complete all three visits required per person in the trial. Therefore we had 237 fully completed visits. To achieve this, a total of 154 participants were scheduled to start the intervention, with 19 not showing up for their first visit, and 135, 84, and 79 completing one, two and all three study visits, respectively. The average total cost per completed participant (that completed three visits) was $24,403 (Direct: $15,543; Indirect: $8860), or $8134 per visit (Direct: $5181; Indirect: $2953).

3.4. Per scheduled visit costs

Our study had 496 scheduled visits, which include the 237 fully compliant visits (those completed by the 79 participants who ended up finishing the study) in addition to completed visits of participants who dropped out of the study (56 participants, 61 visits in total), as well as visits that were scheduled and confirmed but not attended (no show or rescheduled at the last minute, 198 in total). Therefore, the cost per scheduled visit was $3887 (Direct: $2476; Indirect: $1411).

However, since only 237 of 496 visits were useful for the primary outcome assessment of fully completed participants, the cost per visit from the 79 participants who concluded the entire study protocol was more than twice as high (Total: $8134). Ideally, the number of scheduled visits would equal the number of completed visits. For example, in a best-case scenario with no participant dropouts and no visit cancellations, we could have had 496 completed visits, or approximately 165 participants completing the trial (495 visits, $11,684 per participant, Direct: $7442; Indirect: $4242). Therefore, we would have collected more data with reduced per-visit costs, although we acknowledge that this is not likely for real-world interventions.

To sum up, the total cost of our eHealth intervention was $1,927,807 (Direct: $1,227,903; Indirect: $699,904), with intervention material development making up the largest proportion of the subcategories, accounting for 44% of the total cost, followed by personnel (37%), intervention delivery (13%), advertising and recruitment (6%), and finally COVID-19-related equipment costs (<1%). The average cost for one participant to complete the trial was estimated to be $24,403 (Direct: $15,543; Indirect: $8860), or $8134 per visit (Direct: $5181; Indirect: $2953).

4. Discussion

Previous research has outlined costs associated with similar trials. In a study reporting the financial costs of a phase III multi-site exercise intervention trial using fMRI, the cost per participant was reported to be $16,494 with a total trial cost estimated to be around $21 million [5]. Both our study and the phase III intervention study are within the range of the observed cost. For example, a meta-analysis by Speich et al. [18] systematically searched three databases for the publication of empirical data on resource use and costs of RCTs and found overall costs could run between $43 and $103,254 per participant. However, it was worth mentioning that the meta-analysis included a variety of RCTs (e.g., pharmaceutical trials) and did not specify exercise intervention trials similar to the current study.

Although seemingly expensive to develop, eHealth interventions are promising methods for increasing PA among children. This is important during public health emergencies that restrict normal activities, such as the COVID-19 pandemic. The importance of identifying effective interventions is rising as the rates of PA in children are decreasing while childhood obesity is increasing, especially among Black and Hispanic children [16]. Instead of being a hindrance to physical activity, the long-lasting changes in our everyday routines, including our increasing reliance and usage of technology, can be utilized by eHealth interventions using AVGs. It is important to study the efficacy of eHealth interventions to influence the health of children at risk of childhood obesity, as changing behavior is difficult, especially in our fast-changing world.

The trial cost $1,927,807. We were able to collect data from 79 children and their families. A large contributor to this steep price despite a medium-sized sample was the discrepancy between scheduled and completed visits. On average, each participant attended 1.7 out of 3 visits (Range: 0–3, SD = 1.3) and needed to reschedule 1.2 visits (Range: 0–14, SD = 1.6) for each visit over the course of this study. There were 496 scheduled visits, however, only 298 were attended (135 first visits, 84 s visits, and 79 third visits) due to participants not showing up or rescheduling at the last minute. Further, of these 298 visits, only 237 visits counted toward the overall RCT completed participant data set, i.e., when all three visits were completed by the participant. Consequently, the resources attributed to scheduled visits that ended up being no-shows or rescheduled were significant. These costs consisted of wages for research assistants and the phlebotomists scheduled for that day, along with the resources and supplies set aside for the child and their families. It is also worth mentioning that due to the pandemic-related physical distancing requirement, we were unable to double-book participants during a scheduled time slot or allow any overlap of two families in the lab. Therefore, if one family did not show up, the team would end up having an hour of unproductive time. These difficulties in recruitment and scheduling, just two of the many challenges of running an RCT, could result in clinical trials with high monetary expenses and little to no scientific benefit. When enrollment suffers, the trials may have a lesser likelihood of being accepted for publication [10].

These research assistant wages paid for non-productive hours take grant money that would preferably be spent on the trial. In addition, due to the pandemic research hiatus, our trial (and the length of time salaried workers were paid) had to be extended. We extended the trial via a no-cost extension. This was helpful for us as we had the funds remaining in the budget, but no-cost extensions do not provide additional funds to cover added months of data collection.

Additionally, we also had a high dropout rate. Approximately 61.5% of pre-pandemic participants were lost to follow-up after the resumption of research post-COVID-19 suspension [13]. The pilot test of this trial saw a dropout rate of only 20–33%, much less than the dropout rate of the actual post-pandemic trial of 54%. We attributed attendance issues largely to challenges brought on by the pandemic, such as economic instability or health issues, especially among low socioeconomic status populations included in our sample [6,19]. The indirect effects of COVID-19 can be seen in each subcategory of costs in our breakdown showing the pervasive effects the pandemic had on our trial. The pandemic led to substantial changes in our protocol and a more participant-centric experience. We began to focus increasingly on how to incentivize participants to show up as well as participate fully in-between visits. We also increased recruitment efforts to combat dropouts, doubling our output of recruitment mail since October of 2021 compared to pre-pandemic levels, which led to increased resource spending. Doubling the number of recruitment mailings per week seemed to be the most effective method to combat the low participation. Another strategy that was effective in recruiting participants was doing two rounds of mailings separated one week apart to each participant (first a full mailing packet with flyers and information sheets, and second just an informative postcard).

Before March 2020, our team did not anticipate a scenario in which a pandemic erupted across the world and completely disrupted the daily life of billions of people. Therefore, many issues and obstacles were not anticipated in our study. For example, our study costs were further exacerbated by having a hiatus during the lockdown, extending the study duration, and implementation of post-resumption precautionary measures directly impacting resource costs. We would like to take this opportunity to share our experience dealing with such unanticipated events. We hope to demonstrate the reality of real-world recruitment and participant retention especially with minority populations during the pandemic and the potential extra cost due to visit no-shows or incomplete visits. Because climate change can potentially exacerbate 58% of infectious diseases confronted by humanity [14] and may affect many weather and climate extremes around the world, we hope to use this work as a springboard for future researchers' pandemic and disaster outbreak preparedness.

Better ideas and strategies for recruitment techniques and participant engagement should be shared and lead to more efficient budgets and thus more efficient intervention knowledge and practice. An advantage of our analysis is that we were able to obtain fully calculated costs after data collection when the trial was completed. Therefore, we have a concrete number instead of an estimated number for the entire cost. It is helpful to share the information with agencies funding eHealth clinical trials to inform their costs to successfully and efficiently execute a study.

Several considerations are needed in applying our costs to other studies. Since inflation has been rapidly rising, our costs cannot be directly projected into the future. For example, in addition to direct financial impacts, the pandemic indirectly affected many aspects related to running the RCT as well, such as the rising costs of research materials due to inflation and supply/demand challenges of equipment needed. Small changes in our protocol, such as the type of video game console used, or the number of visits per participant could drastically alter the costs. Being in a university environment, some resources were available for free that other studies may not have access to, such as free laboratory space, the DEXA and centrifuge machines.

On the other hand, setting aside the relatively high cost of preparing the intervention, if we imagine stripping away the research aspect and considering just the deployment of the intervention outside of a research study, the cost for ongoing maintenance and delivery would be considerably lower. Assuming (1) the narrative animation has been developed and can be deployed freely to the participants; (2) the participants do not have the game system or the games; and (3) the story delivery status does not depend on research data collection but will be done by an automated server, we would need to include the cost of the game system and the games plus the salary of one staff member whose duty is to ensure that the stories are delivered correctly to participants through remote monitoring. Given the vast number of Xbox game players in the world, we anticipate the material expense to be around $700–1000 per participant to implement this intervention out of a research context. The material expense will only include the Xbox game console and Kinect sensor, along with the AVGs. We provide a range rather than a specific price for these items because the cost may surge due to the lack of manufacturer supply.

Our systematic efforts to organize and quantify the costs of an RCT into separate and meaningful subcategories revealed crucial insights. In our efforts to run an eHealth intervention during the pandemic era, one may assume the primary expenditure of money would go into PPE and precautionary measures. Interestingly, direct COVID-19-related equipment costs were not the major source of cost within our trial. Instead, what was most damaging was the impact of the pandemic on participant attendance rates. We hope our experience will help further the pursuit of efficient science.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Data will be made available on request.