Recruitment and Retention Issues in a Technology-Based Intervention Among Korean American Midlife Women with Depressive Symptoms’

Abstract

The number of health technology-based intervention studies has grown significantly. However, issues in the recruitment and retention for such studies, especially of Asian Americans, have rarely been discussed. The purpose of this paper is to discuss issues in the recruitment and retention of a specific group of Asian Americans—Korean American midlife women with depressive symptoms—into a technology-based intervention study using computers and mobile devices with a measurement device and to provide directions for future participant recruitment and retention in technology-based intervention studies. The written memos of research team members and the written records of research team meetings were analyzed using a content analysis. The issues in the recruitment and retention process included: (a) low recruitment and retention rates; (b) the perceived long intervention period, (c) strict inclusion/exclusion criteria, (d) concerns related to the use of a measurement device, and (e) the perceived adequacy of monetary incentives. Based on the issues identified in the study, several suggestions are made for future recruitment and retention of racial/ethnic minorities in technology-based intervention studies (e.g., appropriate intervention period, innovative and creative motivation strategies, acceptable measurement scales and devices, and adequate monetary reimbursement).

In recent years, technology-based interventions are widely welcomed and used by diverse groups of users including lay people, researchers, and health care providers^1,2 mainly due to its easy accessibility without physical attendance. An increasing number of technology-based intervention programs for behavioral changes have been developed and tested,^3,4 and their acceptance and effectiveness have begun to be supported among diverse groups including those with low resources, across gender and age, and among racial/ethnic minorities.^1,2 With the COVID19 pandemic, technology-based interventions became more popular, and their high potential in minimizing racial/ethnic health disparities has been highlighted.⁵

Despite the high potential and acceptability of technology-based interventions,^1,2 they have rarely been adopted and tested among Asian American populations including Korean American midlife women. Especially, they have rarely been used to promote physical activity among Asian American midlife women who tend to have high risk for chronic diseases (e.g., depression, hypertension, cardiovascular diseases, and Type 2 diabetes).^6,7 In addition, despite an increasing number of studies on various strategies to recruit the participants into technology-based intervention studies, little is still known about the issues in recruiting and retaining racial/ethnic minorities, specifically a group of Asian American women with a specific health condition, into technology-based interventions. In a PUBMED search using a combination of keywords including computers, mobile devices, interventions, participant recruitment, retention, and issues (without a time limitation), only 35 articles were retrieved. Yet, none of them were about recruitment and retention issues except this research team’s own papers among different populations from previous studies (references blinded for the review).

In heath-related research, potential research participants are usually accessed through direct recruitment strategies such as newspapers, recruitment posters/flyers, and radio advertising.^8–10 Also, indirect recruitment strategies through contacting gatekeepers in clinics and community settings are often used. ^8–10 Despite continuous and significant efforts for the recruitment and retention of research participants, recruitment and retention issues have been major barriers that researchers need to tackle in most studies, especially among racial/ethnic minorities.^8–10 Indeed, technology-based interventions using computers and mobile devices are situating researchers in a special context of participant recruitment.¹¹ Moreover, computer and mobile technologies are advancing on a daily basis.

The purpose of this discussion paper is to share issues in the recruitment and retention of a specific group of Asian Americans—Korean American midlife women with depressive symptoms—into a technology-based intervention study using computers and mobile devices with a measurement device and to provide directions for future participant recruitment and retention in technology-based intervention studies among racial/ethnic minorities. The parent study is summarized first. Then, the approach used to provide the evidence for the recruitment and retention issues is described. The recruitment and retention issues follow the approach section. Finally, future directions for participant recruitment and retention in technology-based intervention studies are suggested based on the issues.

The Study

The ongoing parent study aims to increase lifestyle physical activity and subsequently reduce depressive symptoms during the menopausal transition among Korean American midlife women using a technology-based physical activity (PA) promotion program though computers and mobile devices with a PA measurement device. The study is conducted under the approval of the Institutional Review Board of the institutes where the study is being conducted. The participants are being recruited through online and offline communities/groups for Korean Americans, and the target number of the participants is 330. Currently, the recruitment is ongoing; so far, the recruitment has been conducted for 6 months. In this study, online communities/groups are defined as groups of people with shared interests on the Internet to communicate their interests and work together over time. Offline communities/groups are groups of people with shared interests in physical/real settings to communicate their interests and work together over time. The participants are randomized into: (a) the intervention group; and (b) the control group. Then, the intervention group uses the technology-based physical activity promotion program and the Center for Disease Control and Prevention (CDC) website on physical activity and depression while the control group uses only the CDC website. The 12-month intervention includes three major components: (a) educational modules related to physical activity and depression; (b) online resources related to physical activity and depression; and (c) a social media site with a chat function. Group coaching/support is asynchronously provided through the social media site with educational modules and online resources; the participants could get group coaching/support at any time through the social media site. Individual coaching/support is weekly provided through the chat function. Group and individual coaching/support is provided only for 6 months (the adoption period), and the participants could use all other components for the remaining 6 months (the maintenance period). During the time, they complete the questionnaire at three time points (pretest, post-one-month and post-three-months test). Only those who meet the following criteria are included: self-reported Korean American women who are 40 to 60 years old, whose parents and grandparents are of Korean descent, who can read or write in English, or Korean and living in the United States (US), have access to the Internet using computers or mobile phones, who are sedentary (without any disabilities preventing physical activity), and who have experienced depressive symptoms during the past two weeks (1 to 10 on the Patient Health Questionnaire [PHQ-9],¹² which is equivalent to the cut-point of minimal to moderate depression¹²). The cut-points of mild, moderate, moderately severe, and severe are 5, 10, 15, 20, respectively. In order to block any risky effects of increasing physical activity, multiple factors (e.g., medical history and family history of hypertension, diabetes, cardiovascular disease, etc.), were checked against the screening questions by Wilbur.¹³ The study is being announced in online and offline communities for Korean Americans.

Once the visitors at the project website review and sign the electronic informed consent, they are screened using the inclusion/exclusion criteria. After screening them, those who are eligible for the study are automatically connected to online survey questionnaires. Once they complete the questionnaire, they are automatically randomized into the intervention and control groups using a server-side programming. Then, both groups are asked to visit the project website and use the intervention and/or the Center for Disease Control and Prevention (CDC) website through the project website.

The instruments include the questions on background characteristics and health/disease status, the Center for Epidemiologic Studies Depression Scale-Korean (CESD-K),¹⁴ the Acculturation Stress Scale,¹⁵ the Social Readjustment Rating Scale,¹⁶ the Personal Resource Questionnaire,¹⁷ the Kaiser Physical Activity Survey (KPAS),¹⁸ and a measurement device (Fitbit Charge 5 accelerometer). The reliability and validity of all instruments are supported in Korean populations with Cronbach’s alphas ranging from 0.80 to 0.98. Both groups are reimbursed with $50 electronic gift cards at each time point. Data collection for the parent study is ongoing. Since the data collection is ongoing, the results of the study cannot be shared yet.

The Approach

During the recruitment and data collection process, research team members have collected their research memos on any issues as they have experienced related to the participant recruitment and retention and possible reasons for the issues. Weekly group discussions have been conducted, and written records of these discussions have been kept as research memos. Each research team member wrote 1–3 pages of research memos each week (8 members x 1–3 pages per week x 4 weeks x 6 months = 192–576 pages). The written memos and records by June 30, 2023 were analyzed using a content analysis. Individual words were the unit of analysis, and the memos and written records were firstly coded using line-by-line coding. Then, subsequent categorization was performed on the content, and idea categories were identified from the process. Finally, themes reflecting the recruitment and retention issues were extracted to support the discussion points on the issues. Those issues will be described in the next section.

Recruitment and Retention Issues

The issues related to the recruitment and retention that were identified through the analysis process included: (a) low recruitment and retention rates; (b) the perceived long intervention period, (c) strict inclusion/exclusion criteria, (d) concerns related to the use of a measurement device, and (e) the perceived inadequacy of monetary incentives. Each issue is discussed as follows.

Low Recruitment and Retention Rates

Because the participants were Korean American midlife women in menopausal transition who experienced depressive symptoms, the research team expected a high response rate from potential participants. According to the U.S. Census, in 2021, about 2 million Korean Americans reside in the U.S., about 38.5% were middle-aged (from 35 to 65 years old), and about 51.8% were women.¹⁹ Thus, the statistical facts supported that there would exist a large number of Korean American midlife women who were eligible for the study nationally. However, the actual response rate was not high enough to recruit the target number of participants for the given timeline so far. Again, the study is ongoing; so far, the recruitment has been conducted for 6 months, and 106 potential participants went through the screening process. However, only 22 potential participants met the inclusion criteria, 15 completed the pre-test questionnaire, and 8 dropped out right after the intervention started.

The Perceived Long Intervention Period

One of the major reasons for the dropouts that was reported by the participants was: the perceived long intervention period. The intervention period was set as 12 months (6 months of adoption and 6 months of maintenance) based on the literature on physical activity promotion.^20–22 In most physical activity studies, it took about 6 months to have significant improvements in physical activity and 12 months to maintain the achieved level of physical activity.^20–22 However, in this study, the participants were clearly not happy about their 12 month involvement in the study. While claiming that they did not know that they should stay at the study for 12 months (although it was clearly mentioned in the informed consent), over 50% of the participants who completed the pre-test and had an information session with the interventionists (to initiate the weekly coaching/support sessions) dropped out of the study in the first month of the intervention. Their main reason was they were too busy to have weekly coaching/support sessions.

The research team speculated two plausible reasons for the perceived long intervention period. First, because this intervention aimed at midlife women with depressive symptoms, it would be naturally difficult to adequately motivate them to participate in the planned intervention period. The literature reported that depressive women were difficult to get engaged even in daily activities including leisure activities.^23,24 In a pilot study of this study,²⁵ similar responses were noted. The participants did not respond to the interventionists promptly and often forgot about the appointments for weekly coaching/support. Frequent reminders were essential to make the participants engaged in the intervention. Although the literature on physical activity promotion suggested 6 months to have significant improvements in physical activity and 12 months to maintain the achieved level of physical activity,^20–22 the intervention period should have been shortened while considering the health condition of the participants.

Strict Inclusion/Exclusion Criteria

This study tended to have strict inclusion and exclusion criteria. In addition to the inclusion and exclusion criteria that are described above, to block potentially dangerous effects of increasing physical activity, the screening questions by Wilbur¹³ were included. Because of the low percent of eligible participant among those agreed to participate in the study (22 among 106 screened participants), the research team needed to make changes in the inclusion and exclusion criteria. The first modification was made on age range (from 40~60 years to 40~65 years) because midlife could usually mean up to 65 years old. Also, the definition of “being sedentary” was changed from “not doing >3 times a week” to “not doing 30 minutes x 5 days per week planned physical activity (moderate and vigorous)” because the target of the intervention also included those with low to moderate physical activity. The criteria related to a family history of chronic diseases (e.g., heart diseases, diabetes) were also excluded because the family history would not be a factor directly influencing participants’ conditions during their participation in the study. The criteria related to a medical history of chronic diseases were also excluded because of redundancies with specific criteria related to BP, diabetes, and smoking, and a medical history of chronic diseases would not matter if the participants’ conditions were currently normal. Those who did not know their BP and cholesterol level were also included because they could identify their BP and cholesterol level through their healthcare providers; they were requested to get the information. After the first revision of inclusion/exclusion criteria, the research team expected that a larger percent of the potential participants who agreed to participate would be eligible to participate.

The percent of the eligible participants, however, was still low even after changing the criteria; only two additional participants (8%) met the new inclusion criteria. Thus, the research team carefully re-reviewed the number of potential participants who were screened out by individual criteria. The major reason for those who were screened out was: they did not meet the criteria related to the PHQ-9 question (the level of depression). The cut point was set as 10, which indicated minimal to moderate depression.¹² Among 100 women who went through the Korean version of screening questions, 19 women met the inclusion criteria. When the cut point was moved up to 14, 28 women could meet the inclusion criteria. Thus, in consultation with a psychiatrist, we increased the cut point from 10 to 14.

An interesting finding during the process of changing the inclusion criteria related to the level of depression was: some women over-reported their depressive symptoms while others under-reported their depressive symptoms. It is well-known that technology-based approaches using non-face-to-face interactions could reduce the bias and/or unfairness from different contextual factors (e.g., gender, race/ethnicity). Thus, it was expected that potential participants would respond to the questions on depressive symptoms honestly. Yet, those with severe depressive symptoms tended to under-report their symptoms while minimizing their symptoms. For instance, in a consultation with a psychiatrist, one participant was dropped by the research team because she actually had severe depressive symptoms that would need immediate treatment. In contrast, many women with mild depressive symptoms seemed to over-report their symptoms to be eligible for the study.

Concerns related to the Use of a Measurement Device for Physical Activity

In this study, a measurement device for physical activity was mainly used to obtain objective data on physical activity experience. The measurement device was also expected to provide an incentive for the participants to participate in the study based on previous studies.^26,27 Indeed, in a systematic literature review and meta-analysis study,²⁶ the use of the measurement device was reportedly helpful in recruiting the participants and had the potential to increase physical activity. However, in this study, the measurement device was a barrier for some participants in their participation in the study; 15 complaints were received. There were four major issues related to the measurement device that the research team found.

First, the use of the measurement device (Fitbits in this study) made one participant withdrew from the study because she felt the use of the device would disclose her personal biological data. The participants were actually required to wear the measurement device, monitor their physical activity through the device, and synchronize the device with the database. Thus, the interventionists gave an instruction session with the participants to explain the usages of the measurement devices and the study process at the completion of the pre-test questionnaire. The research team expected that there would be some technology literacy issues; indeed, two participants had difficulties in synchronizing their devices with the database. One participant thought she did synchronize, but she actually did not (she turned off the synchronizing process early, so all the data were not uploaded). After several trials, she withdrew from the study. Four participants claimed that they would have not agreed to participate in the study if they had known that they would be required to wear the measurement devices. The research team showed the informed consent sheet that they clicked “yes” to participate in the study, but the participants still insisted that they did not know the use of the measurement devices, and three of them dropped out of the study. Thus, the research team added one more question for potential participants to check if they would be okay with wearing a measurement device before allowing them to join the study.

Second, once the participants began to use the measurement devices, four participants began to complain about wearing the devices while sleeping. To measure resting heart rates, they needed to wear the devices during the night-time, but they reported discomfort of using the devices (e.g., feeling some electricity through the sensor on the device). Four participants said that they had suffered from insomnia, and they felt uncomfortable about wearing the device because it made their tactile senses more sensitive and made their insomnia worse. The research team needed to persuade the participants that the measurement devices were safe and explained the necessity of wearing the device during the nighttime for the study. However, two declined to wear the devices while they were sleeping, and the research team allowed them not to wear the devices during the nighttime. Indeed, the discomfort of wearing the device while sleeping has been reported.²⁸

Third, three participants reported their skin allergies to the silicon band of the measurement device. One of the participants reported her atopy problem with the device. Even research team members of this study who were wearing the devices to have actual experience of wearing the devices (to understand the participants’ actual experience with the devices) experienced skin allergy problems. Two of them said that they wiped their devices everyday so that the devices would not have any remaining sweats in order to prevent any skin problems. One of them reported that he was wiping the devices with alcohol every day to prevent any skin problems from wearing the devices. Another research team member purchased a hypoallergenic band and replaced the original band with the purchased band in order to prevent further skin problems. A research team member was alternating her wrist to avoid further skin problems. Even at the device company website, there was a warning on possible skin problems, and gave advice on how to minimize skin irritation.²⁹ The research team provided the participants with the strategies that they had been using, all three participants were retained in the study.

Finally, one participant withdrew from the study because she did not want to wear the device instead of another device that she was wearing at that time. The research team asked her to wear the measurement device on the other wrist, but the participant declined and dropped out. The research team needed to let the participant withdraw because she did not agree to provide her data through her own device as well. There exist many other devices that have similar functionalities of the measurement devices that are used in this study (e.g., Apple watch, Google Pixel watch, Samsung Galaxy Watch, Garmin, Xiaomi, HUAWEI). Thus, many potential participants who were using this kind of devices before participating in the study did not perceive the use of the devices as an incentive to participate in the study. Rather, they perceived it as an unnecessary barrier to participate in the study.

Adequacy of Monetary Incentives

To increase the recruitment and retention rates in this study, the research team used monetary incentives. Monetary reimbursement has been reported to be more effective than gifts in retaining research participants when it is provided in an ethical way.^30,31 Indeed, providing monetary incentives to research participants in clinical research is widespread and longstanding.³¹ Yet, monetary incentives still could be a topic of substantial debate because they could coerce or induce potential participants to participate.^30,31 In this study, each participant received $50 electronic gift certificates upon the completion of pre-test, post 6-months, and post 12 months. The amount was decided based on the pilot study²⁵ of the research team while considering the internet and smartphone fees that might occur due to the participation.

An interesting phenomenon was: seven participants dropped out after getting the first incentives of $50 electronic gift certificates upon the completion of the pre-test questionnaire. A major reason for their dropouts was low monetary incentive for post 6-months and post-12 months. The participants perceived that monetary incentives were too low for the long period of the intervention (12 months) that they would need to participate. They perceived that $50 would be adequate for their completion of the one-time questionnaire (pre-test), but no payment about their participation in the intervention itself was not attractive to them at all. Actually, the measurement devices that were used in this study would be theirs at the completion of participation (after 12 months), but the participants did not perceive the devices as an adequate incentive for 12 months of their research participation.

Implications for Future Recruitment and Retention

Based on the discussion on the issues, several implications are proposed as follows for future participant recruitment and retention in technology-based intervention studies using computers and mobile devices with a measurement device. Table 1 summarizes the implications according to the issues. First of all, future researchers need to consider adopting the right period of intervention for their target populations. Considering that the study population had depressive symptoms and the intervention was through computers and mobile devices, the research team originally proposed 3 months as the intervention period based on a literature review on technology-based interventions and the findings from a previous study of the research team. However, the intervention period needed to be extended to 12 months through a series of grant reviews. The research team needed to follow the suggestions from experts in physical activity (e.g., Sports Medicine) who were the members of the grant review process, but this issue certainly supported the original stance of the research team. The intervention period should be chosen based on the population’s specific condition/status and the specific contexts of technology-based interventions.

Table 1.

A Summary of Recruitment and Retention Issues and Suggestions for Future Research

Recruitment and Retention Issues	Suggestions
Low recruitment and retention rates	Adopt innovative and creative motivation strategies to have an adequate number of participants in technology-based intervention studies (e.g., establishing relationships with online and offline communities/groups before actually starting a research study, adopting incentives for community gatekeepers).
The perceived long intervention period	Consider adopting the right period of intervention based on the population’s specific condition/status and the specific contexts of technology-based interventions.
Strict inclusion/exclusion criteria	Consider the practicality of inclusion and exclusion criteria in actual applications to the participants.
Concerns related to the use of the measurement devices	Consider the necessity of the devices carefully. Consider the consumer reviews on the devices if they are available. Carefully review possible side effects of using devices and make a plan on how to deal with the side effects. Consider the perception and responses of potential participants about the devices and measurement scales that will be used in their studies.
The perceived adequacy of monetary incentives	Carefully review the literature related to target population. Select the right amount of a monetary incentive for their potential participants while considering their current economic contexts (e.g., through a small focus group right before the study).

Second, future researchers need to consider the practicality of inclusion and exclusion criteria in actual applications to the participants. All the original inclusion and exclusion criteria were set based on a literature review and recommendations by consultants who were experts in physical activity and depressive symptoms. In order to prevent potential harmful effects, it was necessary to set all these inclusion and exclusion criteria. However, in reality, these criteria drastically restricted the pool of eligible participants. The research team eventually needed to relax the criteria as discussed above, which was essential to increase the number of eligible participants. Future researchers need to carefully review and set the inclusion/exclusion criteria considering the situation of their specific populations.

Third, future researchers need to consider the perception of potential participants about the measurement devices and scales that will be used in their studies. They need to firstly consider the necessity of the devices carefully and need to consider the consumer reviews on the devices if they are available. All the participants’ complaints on the use of the measurement devices were originally unexpected because the measurement devices were highly welcomed and accepted in the previous studies of the research team. The worth of the device during the time when the previous studies were conducted might be higher than its present value; a variety of devices are currently available in the market, and some of them have higher monetary values than the devices that are used in this study. Thus, future researchers need to carefully review possible side effects of using devices in their studies before adopting them and make a plan on how to deal with the side effects of using devices. As this research team found several strategies to minimize the side effects of wearing the devices, these strategies need to be set in advance and provided to the participants. Also, researchers need to be aware on differences in participants’ perception and responses to measurement scales. There is no control on how individual participants respond to specific items of questions, but the researchers need to consider the potential influences of contextual factors that might influence the participants’ responses to specific items of measurement scales.

Fourth, future researchers need to carefully review the literature related to their target population and select the right amount of monetary incentive for their potential participants while considering their current economic contexts. In the Korean traditional culture, mentioning money in a direct way was considered as being rude. Researchers working with Korean populations have emphasized the importance of explaining the study significance to research participants in order to motivate them to participate in the studies rather than paying a high amount of money. Also, in a previous study, $50 of electronic gift certificates per time point successfully worked for the recruitment and retention of the same population. Thus, the research team thought $50 of electronic gift certificates per time point ($150 in total) would be adequate to motivate the potential participants while emphasizing the significance of the study. However, in reality, an electronic gift certificate of $50 per time point was perceived to be too small for the intervention period (maybe due to the recent inflation). Also, the research team thought the measurement devices would be a very good additional incentive, but it was not. Maybe, a small focus group with potential participants right before starting a study would help decide the right amount of a monetary incentive for the specific population, especially during a high inflation period.

Finally, future researchers need to be prepared to deal with possible low response rates and adopt necessary innovative and creative motivational strategies. In these days, almost everybody is using computers and mobile devices in his/her daily lives around the world. Thus, researchers could reasonably expect that response rates to technology-based interventions would be similar to those of traditional interventions using pamphlets, in-person sessions, etc. However, this study supported that low response rates would still be one of the major issues that researchers using computers and mobile devices could have.⁸ Thus, it would be essential for researchers to adopt innovative and creative motivation strategies to have an adequate number of participants in technology-based intervention studies as well.¹¹ For instance, establishing relationships with online and offline communities/groups before actually starting a research study would be helpful in motivating the communities/groups to be cooperative with researchers. Also, adopting incentives for community gatekeepers (e.g., providing honorarium, continuing education credits for healthcare providers) would help improve the recruitment and retention of research participants into technology-based intervention studies as well.

Conclusions

In this paper, the recruitment and retention issues in a technology-based intervention study using computers and mobile devices with a measurement device were identified and discussed. The issues included: the low recruitment and retention rates, the perceived long intervention period, strict inclusion/exclusion criteria, concerns related to the use of the measurement devices, and the perceived adequacy of monetary incentives. Based on the issues identified in the study, several suggestions were made for future recruitment and retention of racial/ethnic minorities in technology-based intervention studies (e.g., appropriate intervention period, innovative and creative motivation strategies, acceptable measurement scales and devices, and adequate monetary reimbursement).