Developing Message Strategies to Engage Racial and Ethnic Minority Groups in Digital Oral Self-Care Interventions: Participatory Co-Design Approach

Stephanie M Carpenter; Zara M Greer; Rebecca Newman; Susan A Murphy; Vivek Shetty; Inbal Nahum-Shani

doi:10.2196/49179

. 2023 Dec 11;7:e49179. doi: 10.2196/49179

Developing Message Strategies to Engage Racial and Ethnic Minority Groups in Digital Oral Self-Care Interventions: Participatory Co-Design Approach

Stephanie M Carpenter ^1,^2,^✉, Zara M Greer ³, Rebecca Newman ³, Susan A Murphy ^4,⁵, Vivek Shetty ³, Inbal Nahum-Shani ⁶

Editor: Amaryllis Mavragani

Reviewed by: Xinghui Yan, Jamie Yap

PMCID: PMC10750234 PMID: 38079204

Abstract

Background

The prevention of oral health diseases is a key public health issue and a major challenge for racial and ethnic minority groups, who often face barriers in accessing dental care. Daily toothbrushing is an important self-care behavior necessary for sustaining good oral health, yet engagement in regular brushing remains a challenge. Identifying strategies to promote engagement in regular oral self-care behaviors among populations at risk of poor oral health is critical.

Objective

The formative research described here focused on creating messages for a digital oral self-care intervention targeting a racially and ethnically diverse population. Theoretically grounded strategies (reciprocity, reciprocity-by-proxy, and curiosity) were used to promote engagement in 3 aspects: oral self-care behaviors, an oral care smartphone app, and digital messages. A web-based participatory co-design approach was used to develop messages that are resource efficient, appealing, and novel; this approach involved dental experts, individuals from the general population, and individuals from the target population—dental patients from predominantly low-income racial and ethnic minority groups. Given that many individuals from racially and ethnically diverse populations face anonymity and confidentiality concerns when participating in research, we used an approach to message development that aimed to mitigate these concerns.

Methods

Messages were initially developed with feedback from dental experts and Amazon Mechanical Turk workers. Dental patients were then recruited for 2 facilitator-mediated group webinar sessions held over Zoom (Zoom Video Communications; session 1: n=13; session 2: n=7), in which they provided both quantitative ratings and qualitative feedback on the messages. Participants interacted with the facilitator through Zoom polls and a chat window that was anonymous to other participants. Participants did not directly interact with each other, and the facilitator mediated sessions by verbally asking for message feedback and sharing key suggestions with the group for additional feedback. This approach plausibly enhanced participant anonymity and confidentiality during the sessions.

Results

Participants rated messages highly in terms of liking (overall rating: mean 2.63, SD 0.58; reciprocity: mean 2.65, SD 0.52; reciprocity-by-proxy: mean 2.58, SD 0.53; curiosity involving interactive oral health questions and answers: mean 2.45, SD 0.69; curiosity involving tailored brushing feedback: mean 2.77, SD 0.48) on a scale ranging from 1 (do not like it) to 3 (like it). Qualitative feedback indicated that the participants preferred messages that were straightforward, enthusiastic, conversational, relatable, and authentic.

Conclusions

This formative research has the potential to guide the design of messages for future digital health behavioral interventions targeting individuals from diverse racial and ethnic populations. Insights emphasize the importance of identifying key stimuli and tasks that require engagement, gathering multiple perspectives during message development, and using new approaches for collecting both quantitative and qualitative data while mitigating anonymity and confidentiality concerns.

Keywords: engagement, oral health, mobile health intervention, mHealth intervention, formative, racial and ethnic minority group, digital health, mobile health, mHealth, message development, health equity, racial minority, ethnic minority, digital intervention, dental care, barrier, oral self-care, mobile phone

Introduction

Background

Oral diseases constitute a significant public health issue and reflect social inequalities in health care [1]. Indeed, racial and ethnic minority populations are especially susceptible to poor oral health, largely due to barriers to accessing dental care and insurance coverage [2]. Although regular toothbrushing is considered a key self-care behavior necessary for sustaining good oral health, adherence to recommended brushing practices remains a challenge, particularly among individuals from low socioeconomic backgrounds [3]. Therefore, developing accessible and unobtrusive strategies to encourage regular oral self-care behaviors in at-risk groups could significantly lower the rates of dental disease and enhance health outcomes [4].

The ubiquity of digital technology, including mobile devices and wearable sensors, offers unique opportunities to expand access and enhance health outcomes, particularly among underserved populations [5]. With the ownership of smartphones approaching 85% within racial and ethnic minority populations [6], leveraging these cutting-edge devices can foster and encourage beneficial health behavior transformations in communities that have historically encountered health care inequities. Furthermore, digital tools facilitate the continuous tracking of a person’s condition and surroundings, thereby providing a means to offer timely and cost-effective digital prompts for health-related behaviors [7-9]. Despite the transformative promise of digital interventions, their true effectiveness is often undermined by lackluster engagement and the lack of concerted efforts toward their use [10-12]. Overcoming this significant hurdle is not just necessary but imperative to fully harness the power of digital health technologies and foster a more balanced and equitable health care environment.

In the realm of digital health interventions, the significance of formative work in crafting tailored messages cannot be overstated, particularly when addressing the unique needs and preferences of racial and ethnic minority groups. Well-designed mobile health tools that deliver culturally sensitive and personally relevant content can promote sustained engagement with the target behavior, thereby fostering positive health outcomes. By incorporating insights gleaned from formative research, digital interventions can have a greater impact than generic health promotion strategies by resonating with the target audience.

In this manuscript, we describe formative research undertaken to facilitate the creation of content for a digital oral health intervention specifically tailored to cater to an ethnically and racially diverse subpopulation. The messages, aimed at stimulating habitual oral hygiene practices in the morning and evening, will be delivered through a corresponding smartphone app, Oralytics. The app provides personalized messages and assists users in monitoring their brushing habits while presenting them with visual feedback on the duration and regularity of their brushing sessions.

Drawing on existing perspectives on engagement in digital interventions [9], message content was designed to foster engagement in 3 critical aspects: the targeted behavior (morning and evening brushing), the mobile app, and the messages themselves. We begin by discussing the strategies used to promote engagement in each dimension. Next, we describe the videoconference-based participatory co-design approach used to develop and refine the messages; this approach involves dental experts, individuals from the general population, and individuals from the target population—dental patients, predominantly from low-income racial and ethnic minority groups. Finally, we delve into a methodological approach used to address ethical considerations associated with the use of digital platforms (ie, Zoom [Zoom Video Communications]) for conducting internet-based discussion groups during the COVID-19 pandemic. Insights gleaned from this formative research have the potential to inform and shape the design of messages for future digital health behavior interventions.

Creating Template Messages

A recent framework for engagement in digital interventions [9] encourages careful consideration of the question, “engagement with what?” when developing strategies for promoting engagement. Specifically, digital interventions often use multiple digital stimuli and tasks as a vehicle for engaging individuals with other tasks, nondigital or digital. As different strategies may be needed to increase engagement in different stimuli or tasks, an important first step is to clearly specify the key aspects that require engagement. Here, the focus is on engagement with the target behavior, the mobile app, and the mobile-delivered messages.

Our primary focus was on promoting engagement with the target behavior—oral self-care practices. Clinical guidelines typically endorse toothbrushing twice daily, in the morning and evening, as the most effective preventive routine [13]. Thus, the goal was to design messages that serve as a subtle cue for brushing by making oral health salient in the morning and evening. Second, given that the Oralytics app provides feedback on brushing practices (eg, time spent brushing and how well each quadrant of the teeth was brushed) essential for improving oral self-care, our goal was to design the messages in a way that also promotes engagement with the Oralytics app. Finally, our goal was to ensure that the participants engaged with the messages themselves. Accordingly, the message content was designed to be efficient for the participant’s attention and cognitive resources while also being appealing and original.

In subsequent sections, we present each engagement strategy and detail the message characteristics used to operationalize each strategy into a digital prompt that promotes oral self-care practices. Table 1 provides examples of the messages.

Table 1.

Sample engagement messages across 2 formative webinars.

Message type	Message	Follow-up if message clicked
Reciprocity	A gift awaits you! Redeem $0.5 from Oralytics towards your Prize Account!	N/A^a
Reciprocity-by-proxy	You make the world a better place: Redeem $0.5 from Oralytics for your <<charity>> Account!	N/A
Question and answer	Do you ever have mouth sores?	Educational statements in Oralytics: If response=yes: Mouth sores can be really unpleasant. Brush twice a day, floss once a day, and use mouth wash to reduce the bacteria that can cause sores. If response=no: Great! Did you know that mouth sores can be caused by trauma, bacteria, viruses, or even ingredients in your food or toothpaste (for example, Sodium Lauryl Sulfate)?
Feedback	Want to know how well you’ve been brushing lately? Tap to view	Graphical feedback in Oralytics

Open in a new tab

^aN/A: not applicable.

Engagement With the Target Behavior

Focusing on prompting engagement with morning and evening oral health practices, we selected 3 theoretically grounded strategies that hold great promise for encouraging engagement in health behaviors and can be translated into a mobile health (mHealth) intervention setting: reciprocity, reciprocity-by-proxy, and curiosity.

Reciprocity

The concept of reciprocity suggests that providing an individual with a small, no-strings-attached reward has the potential to capitalize on an innate human tendency to return favors and acts of kindness [14-16]. This strategy can be translated into a digital intervention setting by delivering a prompt containing a small reward that is not contingent on the participant’s behavior [17]. This is expected to increase the likelihood that the participant will reciprocate by engaging with the behaviors encouraged by the digital intervention.

To operationalize this strategy, we developed messages that notify participants about a small US $0.5 gift that they can redeem for their “prize account.” These messages have the following structure: (1) be framed as a US $0.50 gift from Oralytics to ensure participants understand that Oralytics is the source of the gift and the target for reciprocation and (2) include the word “Redeem” to signal the need for individuals to click on the message for the US $0.50 to be added to their prize account. At the end of the intervention, individuals will earn a gift card for the amount of money accrued in their prize account.

Reciprocity-by-Proxy

The related concept of reciprocity-by-proxy [18] centers on the notion of providing an unsolicited, no-strings-attached reward or benefit to a third party that an individual values (ie, considers important and desirable) to amplify the person’s sense of indebtedness and obligation to reciprocate. The strategy can be adapted for use in digital interventions by prompting participants with information about an unconditional donation that was made on their behalf to a charity they care about. This is expected to increase the likelihood that participants will reciprocate by engaging in the behaviors encouraged by the intervention.

To operationalize this strategy, we developed messages that notify participants about a small, no-strings-attached US $0.50 gift that they can redeem for their “charity account.” These messages have the following characteristics: (1) they are framed as a US $0.50 gift from Oralytics to ensure participants understand that Oralytics is the source of the gift and the target for reciprocation; (2) the charity that each participant selected (during an onboarding session) is explicitly mentioned to highlight that the donation was made to the person’s valued charity, and (3) the word “Redeem” is included so that individuals know that they must click directly on the message for the US $0.50 to be added to their charity account. At the end of the study, Oralytics will donate to the charity of the participant’s choice, in an amount equal to the funds accumulated in their charity account.

Curiosity

The concept of curiosity is defined as “a desire to acquire new knowledge and new sensory experience that motivates exploratory behavior” [19]. This desire can be leveraged in digital intervention settings by delivering prompts that contain questions, generate inquiry, and encourage inquisitive thinking related to the target behavior [20-22]. Such prompts are expected to increase engagement in the target behavior.

To operationalize curiosity, we developed 2 types of messages. The first type was designed to provide information to improve individuals’ oral health knowledge. The goal was to create messages that contain a question relevant to the participant’s current oral health. Answering the question with either “yes” or “no” will trigger the Oralytics app to display relevant, tailored educational information and advice. These question and answer (Q&A) messages have the following characteristics: (1) they contain an oral health–related question that can be answered with either yes or no, and (2) the answer triggers a follow-up statement with educational information or advice tailored to the participant’s answer. The second type of curiosity message was designed to encourage participants to access graphically displayed feedback in the Oralytics app (see the Engagement With the Mobile App section below for further description).

Engagement With the Mobile App

To increase engagement with the Oralytics app, we designed a second type of curiosity message that encouraged participants to access feedback about their brushing behaviors within the app. The goal was to create messages that are nonevaluative, nondirective, and generally supportive of an individual’s autonomy [23-26]. Specifically, these feedback messages were designed to adhere to the following characteristics: (1) encourage the participant to seek out and access feedback on their brushing behaviors and (2) communicate autonomy support by emphasizing that the individual has a choice of whether to view their feedback. This autonomy support is accomplished by using suggestive (eg, consider viewing your brushing statistics) rather than directive (eg, you must check your brushing statistics now) framing.

Beyond the design of the curiosity messages, all messages were designed to facilitate participant interactions with the Oralytics app. Specifically, the reciprocity and reciprocity-by-proxy messages were designed such that participants were guided to click on the message to redeem the gift in the app. The Q&A curiosity messages were designed such that clicking either yes or no in response to the question would direct the participant to the appropriate answer within the app. Finally, the feedback curiosity messages were designed to direct the participant to the location of their brushing feedback in the app (ie, the participant can view the feedback in the app by clicking on the message).

Engagement With the Messages

To promote engagement with the messages, their content was designed in a way that is resource efficient (in terms of the participant’s attention and cognitive processing), appealing, and novel.

Resource Efficiency

Individuals must navigate a large amount of information in their daily lives, which can often tax attentional and cognitive resources. Given this high volume of information, individuals are more likely to read and click on brief messages than messages that require a great deal of attention and concentration to read and understand [27]. Thus, all messages were designed to be brief in length (ie, no longer than 140 characters) and to use relatively simple language to improve comprehension and reduce cognitive burden.

Message Appeal

Messages that are framed in positive and encouraging ways are likely to enhance positive mood and approach motivations [28], where approach motivations reflect a drive or an “impulse to go toward positive stimuli” [29]. Thus, a message that is framed in a positive and generally upbeat manner is likely to encourage people to actively engage with (eg, fully read or click on) the message. Furthermore, by increasing engagement with the messages, a positive tone may also enhance the salience of and promote more positive attitudes toward brushing. Such positive attitudes may further encourage engagement with brushing and the Oralytics app. Given these numerous plausible benefits, all messages within the 4 categories were designed to be positive, encouraging, and enthusiastic in tone.

Message Novelty

A collection of messages were generated for each message category to increase message novelty and reduce habituation over time. Specifically, for each category, we generated messages that adhered to the characteristics described earlier but still varied in other respects. For example, the feedback messages were framed in one of three ways: (1) as a question (to communicate autonomy support), (2) as a statement highlighting the benefits of viewing feedback (to communicate that the feedback is useful and relevant to the participant), or (3) as a general statement (to provide a simple reminder that feedback is available in the app). In the following section, we describe the process of further developing and refining these messages.

Methods

Message Development and Refinement

After generating an initial collection of messages for each category (see examples in Table 1), we used a videoconference-based participatory co-design approach to further develop and refine the messages. This approach involved gathering and integrating insights from (1) dental experts via small groups of 4 to 6 dental residents and dental students; (2) individuals from the general population via Amazon Mechanical Turk (MTurk); and (3) dental patients drawn from community clinics who are representative of the target population via 2 facilitator-led videoconference sessions.

Feedback From Dental Experts

Three generated message categories (reciprocity, reciprocity-by-proxy, and Q&A curiosity) were developed and refined based on feedback from small groups of 4 to 6 dental experts, including dental residents and dental students. Through meetings with experts, we obtained initial ratings of message liking and suggestions on how to improve the messages to make them more relevant and appealing to dental patients. The meetings provided invaluable feedback for the initial development and refinement of the messages. However, these meetings did not focus on the second category of curiosity messages (ie, feedback messages), as they were relatively simple and straightforward and did not require dental expertise. To ensure that the messages were understandable and appealing to the general population, it was critical to obtain additional feedback from nonexperts who were more similar to individuals who would be targets of future mHealth digital oral self-care interventions.

Feedback From MTurk Employees

The use of MTurk workers provided an affordable and efficient way to collect initial message ratings that were balanced by key demographic characteristics (eg, gender, racial, or ethnic minority status). The MTurk phase also allowed for the generation of additional messages by people in the general population, which provided key insights beyond those of the authors and dental experts.

All 4 message categories were pretested with MTurk workers via the CloudResearch Platform. The CloudResearch Platform helps to exclude scam workers and ensure higher quality worker responses. Two web-based pretest sessions were conducted with MTurk workers located in the United States. In the initial session, 100 MTurk workers (n=50, 50% female; n=33, 33% minority group) assessed the reciprocity, reciprocity-by-proxy, and Q&A curiosity message types. In a second session, 50 MTurk workers (n=25, 50% female; n=17, 34% minority group) assessed the second type of curiosity messages (ie, feedback curiosity messages). MTurk workers in both sessions rated each message using the following criteria: (1) how much they liked or disliked the message, (2) how likely they thought another person would be to tap on the message, (3) whether the length of the message was acceptable, and (4) the message grammar.

To assist MTurk workers with assessing the probability of an individual tapping on a given message, we initiated each session with a context-setting scenario. The scenario featured an individual named “Alex,” whose gender was not specified. Alex had difficulties maintaining a consistent toothbrushing routine and was participating in an intervention program aimed at improving this habit. Messages would be sent once or twice daily to motivate Alex and other participants. These messages encouraged them to engage with “Oralytics,” a dedicated oral health smartphone app. This scenario was framed to provide the context for message delivery so that workers could evaluate them even if they do not personally struggle with oral self-care. Furthermore, we asked workers to evaluate messages in terms of their length and grammar, in addition to liking, because these aspects play an important role in message processing and appeal [30-32]. Although we could have incorporated other criteria, such as evaluating the novelty of the educational content for the worker or the potential of the message framing to incite preferable actions, our objective was to keep the message-rating task concise. This was done to prevent overburdening the MTurk workers, thereby reducing potential fatigue and facilitating more meaningful ratings. If any worker indicated that the message required grammatical cleanup, they were prompted to provide suggestions for correcting the grammar. The MTurk workers were also given the option of generating additional messages. On the basis of these insights, messages were identified for inclusion in the facilitator-mediated group webinars 1 and 2. The message inclusion criteria and webinar methods are described in detail in subsequent sections.

Feedback From Dental Patients

Focus groups are typically defined as a group interview involving explicit interaction among group members that is intended to help explore people’s experiences, attitudes, and knowledge of some topic of interest [33]. Although traditional focus groups allow group members to communicate with each other, they may also involve ethical challenges given privacy concerns that arise when participants directly interact and share their opinions with each other [34]. The COVID-19 pandemic has increased such concerns, as many focus groups conducted during the time of quarantine moved to videoconference formats, where the participants’ names and locations (eg, the inside of their homes) are often visible [35-37]. Dos Santos Marques et al [37] took steps to mitigate some of these concerns by providing participants with anonymous names in Zoom focus group sessions. However, the participants were still asked to turn their cameras on during the session. Although participants can use web-based background features to control what their camera reveals [38], they may still have limited control over the behavior of other people in their physical environment (eg, family, roommates, and a partner), which may lead to a privacy breach. These considerations are particularly important when conducting web-based research with economically and socially vulnerable populations because of the fear of social stigma and discrimination [39,40]. To mitigate these concerns, we adopted a facilitator-mediated approach to conduct 2 videoconference sessions with dental patients. This approach, which we describe in detail in the Procedure section, enabled participants to provide feedback and generate suggestions during the sessions, while shielding their identities.

The participants were recruited from a pool of patients obtaining care at a large community dental clinic. A large proportion of the dental community clinic patients are from low-income racial and ethnic minority groups and are a key target demographic for the digital oral health intervention. Table 2 summarizes the sociodemographic characteristics of the participants. Although 15 participants were scheduled for the second session, only 7 attended.

Table 2.

Participant characteristics for videoconference session 1 and 2.

Category and characteristics		Webinar session 1 (n=13)	Webinar session 2 (n=7)
Session date		March 2022	July 2022
Age (y), median (range)		30-49 (18-50+)	30-49 (30-50+)
Sex, n (%)
	Female	5 (38)	4 (57)
	Male	8 (62)	3 (43)
Race and ethnicity, n (%)
	Asian	2 (15)	—^a
	Black or African American	3 (23)	7 (100)
	White (non-Hispanic)	1 (8)	—
	Hispanic	7 (54)	—

Open in a new tab

^aNo data were collected from individuals who identify with these racial and ethnic groups.

Ethical Considerations

The procedures involving the participants were performed in accordance with the 1964 Helsinki Declaration and its later amendments. The research protocol was reviewed and approved by the institutional review board (IRB) at UCLA (IRB#20-000106).

Informed consent was obtained from all webinar participants. The participants were adequately informed about the study, and all their questions were answered before their involvement. The consent procedure included information about the purpose, procedures, potential risks and benefits of participation, and participants’ rights, including their right to withdraw at any time without any penalties. The original informed consent documents explicitly allowed for the potential use of the collected data for future research without requiring additional consent, in line with IRB guidelines.

To mitigate privacy concerns and comply with the IRB policies on protecting the identity of participants during web-based research sessions, we developed and refined messages through 2 facilitator-mediated formative sessions using a videoconferencing platform (Zoom). Each participant was only able to see and hear the facilitator with whom they interacted through Zoom polls and a chat window anonymous to other participants. Participants were not able to interact with each other directly and thus could not see any identifying information from fellow participants. This strategy effectively shielded the participants’ identities during the sessions.

Participants in both webinars were compensated for their time and efforts with a US $25 Amazon gift card upon completion of the session. This form of compensation was guided by the need to respect participants’ time and efforts while avoiding undue inducement.

All study data were anonymized and deidentified to protect the privacy and confidentiality of the participants. When necessary, additional protective measures were applied, such as data encryption and restricted access.

Message Inclusion

The first webinar session focused on the reciprocity, reciprocity-by-proxy, and Q&A curiosity message types. It included 20 messages (5, 5, and 10, for each message type, respectively) that were rated below average during the first MTurk session and were thus revised based on MTurk worker feedback. Focusing on the revised messages minimized participant fatigue and kept the session to 1 hour in length. The follow-up webinar session (session 2) focused only on the feedback curiosity messages. As these messages were not evaluated by dental experts, we generated 52 feedback curiosity messages for the second MTurk session with the intention of dropping the lowest rated 22 messages from further consideration. The remaining 30 messages were selected and revised based on MTurk worker feedback and included in webinar session 2. The session duration was similar to session 1.

Procedure

Each session used a combination of methods to collect quantitative and qualitative feedback about the messages. First, the webinar facilitator explained that the purpose of the session was to refine messages in preparation for an intervention that sought to encourage people to use (engage with) an oral health smartphone app. Messages were presented to participants one at a time, and participants were asked to rate each message on the following scale: 1=do not like it; 2=like it somewhat; and 3=like it. Messages were rated using the Zoom webinar polling feature. For any message that received a rating of “1=Do not like it,” participants were given the opportunity to send the facilitator (through the Zoom webinar chat feature) suggestions for improving the message. Participants could also provide reactions or further feedback in response to key suggestions read aloud to the group by the facilitator. Participants volunteered their suggestions and were not cold-called or otherwise required to provide feedback. Therefore, not all participants responded with suggestions for every message. Screenshots of all polls were taken for later data entry, and all participant chat messages were saved.

Results

Quantitative

The mean ratings of reciprocity (mean 2.65, SD 0.52), reciprocity-by-proxy (mean 2.58, SD 0.53), Q&A curiosity (mean 2.45, SD 0.69), and feedback curiosity (mean 2.77, SD 0.48) message types indicated that all were rated highly, with a high overall message rating collapsing across types (mean 2.63, SD 0.58; Multimedia Appendix 1).

The means of the 4 message types ranged between mean 2.08 (SD 0.76; corresponding closely to “Like it somewhat”) and mean 3.00 (SD 0.00; corresponding to “Like it”). On the basis of feedback given by participants during the 2 webinars, a threshold of 2.50 was identified, whereby any messages that received a mean rating of mean 2.50 to 3.00 (60% reciprocity; 80% reciprocity-by-proxy; 40% Q&A curiosity; 90% feedback curiosity; 76% overall) were not further modified. This threshold was identified because it fell between the “Like it somewhat” and “Like it” ratings and >75% of messages, collapsing across the 4 categories, fell at or above this rating. Messages that received an average rating of <2.50 (40% reciprocity, 20% reciprocity-by-proxy, 60% Q&A curiosity, 10% feedback curiosity, and 24% overall) were revised using the webinar participant suggestions as a guide. One exception was a feedback curiosity message that despite receiving a high average rating (mean 2.71) also had a relatively high SD of 0.76 and received critical participant feedback during the webinar. Thus, this message was also revised for clarity.

Qualitative

Participant feedback varied according to each specific message type. For the reciprocity and reciprocity-by-proxy messages, quantitative ratings showed that participants generally did not dislike these message types. This meant that there were fewer suggestions for improvement as participants were only prompted for feedback when they rated a message as “1=Do not like it” during the webinar polling. In the first facilitator-led webinar, only 1 reciprocity message (and none of the reciprocity-by-proxy messages) received a participant rating of “1=Do not like it.” One participant suggested making the app’s intentions clearer in the message. This could be interpreted as a need for clarity that the app is intended for oral self-care. However, as reciprocity messages focused on the no-strings-attached reward provided by the app, this feedback was used to emphasize that the reward was a gift from Oralytics without any conditions rather than a reward for a specific behavior. Reciprocity and reciprocity-by-proxy messages with ratings below the identified mean 2.50 threshold were also modified to be less complex, even if they did not receive a rating of “1=Do not like it” during the session. For example, the lowest rated (mean 2.31) reciprocity-by-proxy message: “Think about the impact you can have, <<Name>>. Redeem a $0.5 gift from Oralytics to your <<Charity>> Account!” did not receive any ratings of “1=Do not like it” and therefore participants were not prompted to provide feedback on this message. However, given the low average rating, this message was revisited and appraised by the researchers as being relatively complex because it asked the individual to think about their possible impact. To simplify, the message was updated as follows: “You are making an impact, <<Name>>. Redeem a $0.5 gift from Oralytics to your <<Charity>> Account!”

The participants provided more suggestions for the curiosity messages. We also obtained feedback on the answers (ie, educational statements) provided in Oralytics after participants responded yes or no to a Q&A curiosity message. Participant feedback was greater for the Q&A curiosity messages than for the feedback curiosity messages. Table 3 provides examples of qualitative participant suggestions for 3 Q&A messages and 2 feedback curiosity messages.

Table 3.

Example message updates based on participant suggestions.

Type and message		Answers^a	Examples of participant suggestions	Suggestions interpretation	Revised message	Revised answers^a,b
Question and answer curiosity
	“Do you have any tooth pain?”	If response=yes: “We are sorry to hear this. If it persists for more than a day, make an appointment with your dentist as soon as you can. In the meantime, try to rinse with salt water or apply a cold compress.” If response=no: “This is good news! Often if you wait until your teeth hurt to see a dentist it’s more difficult to treat a cavity or gum disease.”	“never like to hear sorry to hear this.” “if no starting with ‘that’s great’ sounds weird” “You can say ‘Ouch! Tooth pain is never fun’”	Language that seems odd or disingenuous, like “sorry to hear this” should be avoided.	N/A^c	If response=yes: “Tooth pain is never fun^b. If it persists for more than a day, make an appointment with your dentist as soon as you can. In the meantime, try to rinse with salt water or apply a cold compress.” If response=no: “Glad to hear it! Often if you wait until your teeth hurt to see a dentist it’s more difficult to treat a cavity or gum disease.”
	“Do you sleep well at night?”	If response=yes: “That’s great! Sleeping well at night is important to your health, including your dental health. Make sure to keep your stress levels low and get enough sleep each night.” If response=no: “Not sleeping well at night may be a sign of sleep apnea. Sleep apnea can cause breathing from one’s mouth at night, which can lead to more plaque retention on your teeth and make you susceptible to cavities. Your dentist can prescribe sleeping devices for sleep apnea.”	“Better to say ‘Do you get enough sleep?’ like it’s a number rather than a feeling” “yes it should be worded as how much sleep over how well” “If no shouldn’t say you can get the prescription from the dentist. It almost sounds like you’re suggesting that person to get the prescription. like Q2 you should just say contact your dentist.”	Frame questions about sleep (or similar behaviors) in terms of quantity over quality. Avoid recommending courses of action that may not be generally relatable.	Do you get enough sleep at night?	If response=yes: “That’s great! Getting enough sleep at night is important to your health, including your dental health. Make sure to keep your stress levels low to get enough sleep each night.” If response=no: “Not getting enough sleep at night may be a sign of sleep apnea. Sleep apnea can cause breathing from one’s mouth at night, which can lead to more plaque retention on your teeth and make you susceptible to cavities. Check with your doctor or dentist if you continue to have problems sleeping.”
	“Have you had something sugar sweetened to eat or drink today?”	If response=yes: “Occasional added sugar is ok when part of a balanced diet. Brush tonight to reduce the bacteria that feeds on sugar in your mouth.” If response=no: “Making healthy food choices is important. A balanced diet that limits added sugar will help you achieve a healthy smile!”	“Maybe change the question to ‘Have you had something with added sugars to eat or drink today?’” “I like ‘added sugar’ better” “either sounds fine.” “really liked both answers” “Yeah, the suggestion is a more common way of saying it”	Questions should be worded in a common or conversational style (how people speak).	Have you had something with added sugar to eat or drink today?	N/A
Feedback curiosity
	“Brush up on your Oralytics feedback”	Graphical feedback provided in the app	“Here how you did on your oralytics”	The message should be straightforward, without using phrases such as “brush up” that not everyone may understand.	Click here to view your Oralytics feedback.	N/A
	“Learn how you brush to improve your health. Click to view.”	Graphical feedback provided in the app	[Add] “Improve oral health”	Clarify that the feedback is specifically about oral health.	Learn how you brush to improve your oral health. Click to view.	N/A

Open in a new tab

^aThese are educational statements sent by Oralytics tailored to the participant’s response to a Q&A message.

^bUpdates are italicized in the “Revised message” and “Revised answers” columns.

^cN/A: not applicable.

Across both webinar sessions, participant suggestions for modifying the message language fell into the following categories: (1) more straightforward (less confusing or complex); (2) more conversational (less formal); (3) enthusiastic (eg, with an exclamation point); (4) avoid mention of specific actions that may not be relatable to everyone (eg, how to seek a prescription to treat sleep apnea); (5) avoid phrases that could be interpreted as disingenuous (eg, “sorry to hear that”); and (6) for some message types (ie, feedback curiosity) specific to the domain of oral health.

Discussion

Overview of Findings

The formative studies described here sought to develop and refine messages for a digital oral health intervention. These messages were designed to promote engagement in (1) morning and evening oral health practices, by highlighting reciprocity, reciprocity-by-proxy, and curiosity; (2) the mobile app, by creating messages that encourage the participant to access and view feedback on their brushing behaviors in the Oralytics app; and (3) the messages themselves, by creating content that is resource efficient, appealing, and novel.

To develop the messages, we first identified the specific (digital and nondigital) stimuli and tasks that require engagement to achieve the ultimate goal of the digital oral health intervention. Next, we identified theoretically grounded strategies that have the potential to enhance engagement in each identified stimulus or task. Third, guided by the selected engagement strategies, we created message templates, namely protocols that outline the specific characteristics of each category of messages. These templates were used to generate an initial set of messages for each category. To promote greater message engagement, their content was designed to be resource efficient (low burden in terms of attention and cognitive processing), appealing, and novel. Finally, to further develop and refine the collection of messages, we used a web-based participatory co-design approach involving several methods, including gathering and integrating feedback from small groups of dental experts, using MTurk to gather insights from the general population, and further evaluation and refinement by dental patients through facilitator-mediated group webinar sessions. We found this multistage approach to provide highly valuable feedback that highlighted aspects of the messages that needed improvement. This feedback was then used to facilitate the development of messages that have the potential to be highly engaging.

Participants recruited for the group webinar sessions were primarily from underserved racial and ethnic minority groups, which reflects the general sociodemographic makeup of the Los Angeles metropolitan area and is generally representative of the target population for the planned intervention study. Over the course of these group webinar sessions, we found that all 4 categories of messages were highly rated, with the lowest average rating for any given message corresponding near the scale label of “like it somewhat.” Participants offered more suggestions for enhancing the Q&A curiosity messages compared with other types of messages. This could be attributed to several factors. First, Q&A curiosity messages are relatively complex, as they require individuals to respond to a personal oral health–related question. Second, these messages often elicit longer answers. Third, the evaluation process included 2 components: the message itself and the corresponding answers. Finally, this type of message generally received lower numerical ratings (as shown in Multimedia Appendix 1), which led the facilitator to seek additional feedback.

Overall, the qualitative feedback suggests that participants preferred messages that were more straightforward, enthusiastic, conversational, relatable, authentic, and, for some message types (ie, feedback curiosity), specific to oral health. Through these group webinar sessions, we identified which messages needed final refinement and made relatively minor modifications to this subset of messages using suggested feedback from participants.

The facilitator-mediated group webinars used in this research combined both qualitative and quantitative methods that allowed participants to provide message ratings and feedback without access to each other’s names, faces, or other immediately identifying information. We found these procedures useful in developing and refining digital intervention messages with racial and ethnic minority groups. This approach holds promise for increasing population reach and providing enhanced anonymity and confidentiality to underserved groups.

Limitations

Although not knowing the names and identities of other participants in the sessions may enhance anonymity and confidentiality, it is currently unclear to what extent participants in the study perceived this to be the case. For instance, it is still possible for participants to infer general sociodemographic information from the type and content of peer feedback provided, even if read by the facilitator. Furthermore, although participants did not have access to peer names and other identifying information, the facilitators, who were university-affiliated White women, viewed participants’ names during the sessions. Future research should thus seek to determine whether this facilitator-mediated group webinar approach indeed leads participants from racial and ethnic minority groups to perceive greater anonymity and confidentiality and in turn to feel more comfortable providing honest feedback relative to traditional focus groups. Future research should also determine whether this kind of facilitator-mediated group webinar has an advantage relative to other qualitative methods, including individual interviews. For instance, it could be that participants in facilitator-mediated group webinars feel less pressure to provide socially desirable responses than participants in either individual interviews or focus group settings, which could improve the accuracy and generalizability of their responses.

An important limitation relates to the approach used to elicit feedback from webinar participants. Specifically, the facilitator asked for suggestions only if a message was rated “1=Do not like it” by the participant. More feedback could have been obtained by asking participants for comments regardless of the quantitative ratings they provided. However, doing so would have substantially increased the length of the webinar and facilitated a discussion that was less focused on the messages that were most critical to refine. Future research should develop new strategies for eliciting rich feedback in a manner that is also focused and resource efficient.

Another limitation of this research is the uneven sample size across the 2 facilitator-mediated group webinar sessions, with the first session including nearly double the sample size of the second session. This discrepancy was due to scheduled participants not attending the second webinar session, resulting in the sample being comprised entirely of African American participants. It would have been ideal to have a larger sample size that represented a greater diversity of racial and ethnic minority identities. However, there is no reason to suspect that the appeal of feedback messages would be substantially different among African Americans compared with other underserved minority groups. Future research should seek to identify the best way to recruit a broad and diverse range of racial and ethnic minority participants. Future research should also involve the target population in the very initial process of identifying and translating engagement strategies so that their perspectives, skills, knowledge, and expertise can contribute to the entire process of message development [41,42].

Next Steps

The messages described here will be used in a microrandomized trial [43-45] study with 70 dental patients to empirically optimize the delivery of just-in-time support [21,46] for oral self-care. The goal of this microrandomized trial is to investigate whether delivering a message, what type of message, and under what conditions delivering a message increases engagement in ideal oral self-care. The results will build the empirical foundation necessary to develop an optimized digital intervention that delivers the right type of message at the right time to promote oral self-care in the home setting.

Conclusions

This formative work has the potential to guide the development of messages for digital interventions that target individuals from racial and ethnic minority groups. It emphasizes the importance of identifying the key stimuli and tasks that require engagement, involving the perspectives of multiple people (eg, participants, dental experts, and researchers) in the process of message development, and using new approaches for collecting both quantitative and qualitative data while plausibly enhancing participant anonymity and confidentiality. Future research should continue to elucidate the best approaches for refining mHealth messaging designed to support racial and ethnic minority populations.

Acknowledgments

This work was funded by the National Institutes of Health (grants UG3DE028723, P50DA054039, P41EB028242, and R01 DA039901).

Abbreviations

IRB: institutional review board
MTurk: Mechanical Turk
Q&A: question and answer

Multimedia Appendix 1

Mean ratings of the 4 message types. Q&A: Question and Answer.

formative_v7i1e49179_app1.png^{(117KB, png)}

Data Availability

Deidentified data sets generated and analyzed during this research are available in the Open Science Framework repository [47].

Footnotes

Conflicts of Interest: None declared.

References

1.Peres MA, Macpherson LM, Weyant RJ, Daly B, Venturelli R, Mathur MR, Listl S, Celeste RK, Guarnizo-Herreño CC, Kearns C, Benzian H, Allison P, Watt RG. Oral diseases: a global public health challenge. Lancet. 2019 Jul 20;394(10194):249–60. doi: 10.1016/S0140-6736(19)31146-8.S0140-6736(19)31146-8 [DOI] [PubMed] [Google Scholar]
2.Flores G, Lin H. Trends in racial/ethnic disparities in medical and oral health, access to care, and use of services in US children: has anything changed over the years? Int J Equity Health. 2013 Jan 22;12(1):10. doi: 10.1186/1475-9276-12-10. https://equityhealthj.biomedcentral.com/articles/10.1186/1475-9276-12-10 .1475-9276-12-10 [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Raison H, Corcoran R, Harris RV. Is toothbrushing behaviour habitual? Cues, context, motivators and patient narratives. Community Dent Oral Epidemiol. 2021 Oct;49(5):478–86. doi: 10.1111/cdoe.12624. [DOI] [PubMed] [Google Scholar]
4.Elyousfi S, Innes N, Leggett H, Ainsworth H, Chestnutt IG, Day P, Robertson M, Pavitt S, Kellar I, Dey D, Marshman Z. Acceptability of the Brushing RemInder 4 Good oral HealTh (BRIGHT) trial intervention: a qualitative study of perspectives of young people and school staff. BMC Oral Health. 2022 Feb 23;22:44. doi: 10.1186/s12903-022-02073-w. https://bmcoralhealth.biomedcentral.com/articles/10.1186/s12903-022-02073-w .10.1186/s12903-022-02073-w [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med. 2015 Apr 15;7(283):283rv3. doi: 10.1126/scitranslmed.aaa3487. https://europepmc.org/abstract/MED/25877894 .7/283/283rv3 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Atske S, Perrin A. Home broadband adoption, computer ownership vary by race, ethnicity in the U.S. Pew Research Center. 2021. [2023-11-02]. https://www.pewresearch.org/fact-tank/2021/07/16/home-broadband-adoption-computer-ownership-vary-by-race-ethnicity-in-the-u-s/
7.Carpenter SM, Menictas M, Nahum-Shani I, Wetter DW, Murphy SA. Developments in mobile health just-in-time adaptive interventions for addiction science. Curr Addict Rep. 2020 Sep;7(3):280–90. doi: 10.1007/s40429-020-00322-y. https://europepmc.org/abstract/MED/33747711 . [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Carpenter SM, Yap J, Patrick ME, Morrell N, Dziak JJ, Almirall D, Yoon C, Nahum-Shani I. Self-relevant appeals to engage in self-monitoring of alcohol use: a microrandomized trial. Psychol Addict Behav. 2023 May;37(3):434–46. doi: 10.1037/adb0000855.2022-81168-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. 2022 Oct;77(7):836–52. doi: 10.1037/amp0000983. https://europepmc.org/abstract/MED/35298199 .2022-41985-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Boucher EM, Ward HE, Mounts AC, Parks AC. Engagement in digital mental health interventions: can monetary incentives help? Front Psychol. 2021 Nov 18;12:746324. doi: 10.3389/fpsyg.2021.746324. https://europepmc.org/abstract/MED/34867629 . [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Gan DZ, McGillivray L, Larsen ME, Torok M. Promoting engagement with self-guided digital therapeutics for mental health: Insights from a cross-sectional survey of end-users. J Clin Psychol. 2023 May 24;79(5):1386–97. doi: 10.1002/jclp.23486. [DOI] [PubMed] [Google Scholar]
12.Szinay D, Forbes CC, Busse H, DeSmet A, Smit ES, König LM. Is the uptake, engagement, and effectiveness of exclusively mobile interventions for the promotion of weight-related behaviors equal for all? A systematic review. Obes Rev. 2023 Mar;24(3):e13542. doi: 10.1111/obr.13542. [DOI] [PubMed] [Google Scholar]
13.Muscente F, De Caterina R. Challenges in ischaemic heart disease: not sleeping enough, not brushing your teeth, and skipping breakfast-three ways of increasing your risk of myocardial infarction? Eur Heart J Suppl. 2020 Nov;22(Suppl L):L57–60. doi: 10.1093/eurheartj/suaa136. https://europepmc.org/abstract/MED/33654468 .suaa136 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Burger JM, Sanchez J, Imberi JE, Grande LR. The norm of reciprocity as an internalized social norm: returning favors even when no one finds out. Soc Influ. 2009 Jan;4(1):11–7. doi: 10.1080/15534510802131004. https://www.tandfonline.com/doi/full/10.1080/15534510802131004 . [DOI] [Google Scholar]
15.Cialdini RB, Goldstein NJ. Social influence: compliance and conformity. Annu Rev Psychol. 2004;55:591–621. doi: 10.1146/annurev.psych.55.090902.142015. [DOI] [PubMed] [Google Scholar]
16.Gouldner AW. The norm of reciprocity: a preliminary statement. Am Sociol Rev. 1960 Apr;25(2):161. doi: 10.2307/2092623. https://www.jstor.org/stable/2092623 . [DOI] [Google Scholar]
17.Nahum-Shani I, Rabbi M, Yap J, Philyaw-Kotov ML, Klasnja P, Bonar EE, Cunningham RM, Murphy SA, Walton MA. Translating strategies for promoting engagement in mobile health: a proof-of-concept microrandomized trial. Health Psychol. 2021 Dec;40(12):974–87. doi: 10.1037/hea0001101. https://europepmc.org/abstract/MED/34735165 .2022-00033-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Goldstein NJ, Griskevicius V, Cialdini RB. Reciprocity by proxy: a novel influence strategy for stimulating cooperation. Adm Sci Q. 2011;56(3):441–73. doi: 10.1177/0001839211435904. https://journals.sagepub.com/doi/10.1177/0001839211435904 . [DOI] [Google Scholar]
19.Litman JA, Spielberger CD. Measuring epistemic curiosity and its diversive and specific components. J Pers Assess. 2003 Feb;80(1):75–86. doi: 10.1207/S15327752JPA8001_16. [DOI] [PubMed] [Google Scholar]
20.Liu F, Ngai E, Ju X. Understanding mobile health service use: an investigation of routine and emergency use intentions. Int J Inf Manage. 2019 Apr;45:107–17. doi: 10.1016/j.ijinfomgt.2018.09.004. https://www.sciencedirect.com/science/article/abs/pii/S026840121730943X . [DOI] [Google Scholar]
21.Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, Murphy SA. Just-in-Time Adaptive Interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann Behav Med. 2018 May 18;52(6):446–62. doi: 10.1007/s12160-016-9830-8. https://europepmc.org/abstract/MED/27663578 .10.1007/s12160-016-9830-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Reeve J. The interest-enjoyment distinction in intrinsic motivation. Motiv Emot. 1989;13(2):83–103. doi: 10.1007/bf00992956. https://link.springer.com/article/10.1007/BF00992956 . [DOI] [Google Scholar]
23.Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010 Jan;44(1):101–8. doi: 10.1111/j.1365-2923.2009.03546.x.MED3546 [DOI] [PubMed] [Google Scholar]
24.Buder J, Schwind C. Learning with personalized recommender systems: a psychological view. Comput Human Behav. 2012 Jan;28(1):207–16. doi: 10.1016/j.chb.2011.09.002. https://www.sciencedirect.com/science/article/abs/pii/S0747563211001956 . [DOI] [Google Scholar]
25.Cheung KL, Durusu D, Sui X, de Vries H. How recommender systems could support and enhance computer-tailored digital health programs: a scoping review. Digit Health. 2019 Jan 24;5:2055207618824727. doi: 10.1177/2055207618824727. https://journals.sagepub.com/doi/10.1177/2055207618824727?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_2055207618824727 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Hors-Fraile S, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit A, Spachos D, Bamidis P, de Vries H. Tailoring motivational health messages for smoking cessation using an mHealth recommender system integrated with an electronic health record: a study protocol. BMC Public Health. 2018 Jun 05;18(1):698. doi: 10.1186/s12889-018-5612-5. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-018-5612-5 .10.1186/s12889-018-5612-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
27.Muench F, Baumel A. More than a text message: dismantling digital triggers to curate behavior change in patient-centered health interventions. J Med Internet Res. 2017 May 26;19(5):e147. doi: 10.2196/jmir.7463. https://www.jmir.org/2017/5/e147/ v19i5e147 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Fredrickson BL. The broaden-and-build theory of positive emotions. Philos Trans R Soc Lond B Biol Sci. 2004 Sep 29;359(1449):1367–77. doi: 10.1098/rstb.2004.1512. https://europepmc.org/abstract/MED/15347528 .DH5NJJTNQCAMP8UT [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Harmon-Jones E, Harmon-Jones C, Price TF. What is approach motivation? Emotion Review. 2013 Jun 11;5(3):291–5. doi: 10.1177/1754073913477509. https://journals.sagepub.com/doi/10.1177/1754073913477509 . [DOI] [Google Scholar]
30.Steffens PR, Weeks CS, Davidsson P, Isaak L. Shouting from the ivory tower: a marketing approach to improve communication of academic research to entrepreneurs. Entrep Theory Pract. 2014 Mar 01;38(2):399–426. doi: 10.1111/etap.12079. https://journals.sagepub.com/doi/abs/10.1111/etap.12079 . [DOI] [Google Scholar]
31.Reese JM, Joseph RP, Cherrington A, Allison J, Kim YI, Spear B, Childs G, Simpson T, Durant NH. Development of participant-informed text messages to promote physical activity among African American women attending college: a qualitative mixed-methods inquiry. J Transcult Nurs. 2017 May;28(3):236–42. doi: 10.1177/1043659616644959. https://europepmc.org/abstract/MED/27093904 .1043659616644959 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Ownby RL, Acevedo A, Waldrop-Valverde D. Enhancing the impact of mobile health literacy interventions to reduce health disparities. Q Rev Distance Educ. 2019;20(1):15–34. https://europepmc.org/abstract/MED/31537979 . [PMC free article] [PubMed] [Google Scholar]
33.Cerutti P, Marconi A, Pozzi S, Gragnano G, Pain D, Solara V, Morales JR, Paone E, Emolumento AM, Rossi G. Psychological intervention in amyotrophic lateral sclerosis: suggestions for clinical practice. J Psychol Clin Psychiatry. 2017 Oct 11;8(2):00480. doi: 10.15406/jpcpy.2017.08.00480. https://medcraveonline.com/JPCPY/psychological-intervention-in-amyotrophic-lateral-sclerosis-suggestions-for-clinical-practice.html . [DOI] [Google Scholar]
34.Sim J, Waterfield J. Focus group methodology: some ethical challenges. Qual Quant. 2019 Jul 16;53(6):3003–22. doi: 10.1007/s11135-019-00914-5. https://link.springer.com/article/10.1007/s11135-019-00914-5 . [DOI] [Google Scholar]
35.Cornejo M, Bustamante J, Del Río M, De Toro X, Latorre MS. Researching with qualitative methodologies in the time of coronavirus: clues and challenges. Int J Qual Methods. 2023 Jan 03;22:16094069221150110. doi: 10.1177/16094069221150110. https://europepmc.org/abstract/MED/36628132 .10.1177_16094069221150110 [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Johnson B, Odhner K. Focus groups from home: conducting virtual focus groups during the COVID-19 pandemic and beyond. Coll Res Libr News. 2021;82(6):258–61. doi: 10.5860/crln.82.6.258. https://crln.acrl.org/index.php/crlnews/article/view/24971/32810 . [DOI] [Google Scholar]
37.Dos Santos Marques IC, Theiss LM, Johnson CY, McLin E, Ruf BA, Vickers SM, Fouad MN, Scarinci IC, Chu DI. Implementation of virtual focus groups for qualitative data collection in a global pandemic. Am J Surg. 2021 May;221(5):918–22. doi: 10.1016/j.amjsurg.2020.10.009. https://europepmc.org/abstract/MED/33070983 .S0002-9610(20)30620-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Hwang AH, Wang CY, Yang YY, Won AS. Hide and seek: choices of virtual backgrounds in video chats and their effects on perception. Proc ACM Hum Comput Interact. 2021 Oct 18;5(CSCW2):1–30. doi: 10.1145/3476044. [DOI] [Google Scholar]
39.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health. 2014 Feb;104(2):e16–31. doi: 10.2105/AJPH.2013.301706. [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Lathen L, Laestadius L. Reflections on online focus group research with low socio-economic status African American adults during COVID-19. Int J Qual Methods. 2021 Jun 03;20:160940692110217. doi: 10.1177/16094069211021713. https://journals.sagepub.com/doi/full/10.1177/16094069211021713 . [DOI] [Google Scholar]
41.Collins SE, Clifasefi SL, Stanton J, The Leap Advisory Board. Straits KJ, Gil-Kashiwabara E, Rodriguez Espinosa P, Nicasio AV, Andrasik MP, Hawes SM, Miller KA, Nelson LA, Orfaly VE, Duran BM, Wallerstein N. Community-based participatory research (CBPR): towards equitable involvement of community in psychology research. Am Psychol. 2018 Oct;73(7):884–98. doi: 10.1037/amp0000167. https://europepmc.org/abstract/MED/29355352 .2018-02701-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
42.Torres VN, Williams EC, Ceballos RM, Donovan DM, Duran B, Ornelas IJ. Participant engagement in a community based participatory research study to reduce alcohol use among Latino immigrant men. Health Educ Res. 2020 Dec 23;35(6):627–36. doi: 10.1093/her/cyaa039. https://europepmc.org/abstract/MED/33025021 .5918718 [DOI] [PMC free article] [PubMed] [Google Scholar]
43.Qian T, Walton AE, Collins LM, Klasnja P, Lanza ST, Nahum-Shani I, Rabbi M, Russell MA, Walton MA, Yoo H, Murphy SA. The microrandomized trial for developing digital interventions: experimental design and data analysis considerations. Psychol Methods. 2022 Oct;27(5):874–94. doi: 10.1037/met0000283. https://europepmc.org/abstract/MED/35025583 .2022-21177-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
44.Liao P, Klasnja P, Tewari A, Murphy SA. Sample size calculations for micro-randomized trials in mHealth. Stat Med. 2016 May 30;35(12):1944–71. doi: 10.1002/sim.6847. https://europepmc.org/abstract/MED/26707831 . [DOI] [PMC free article] [PubMed] [Google Scholar]
45.Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, Murphy SA. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2015 Dec;34S:1220–8. doi: 10.1037/hea0000305. https://europepmc.org/abstract/MED/26651463 .2015-56045-003 [DOI] [PMC free article] [PubMed] [Google Scholar]
46.Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol. 2015 Dec;34(Suppl):1209–19. doi: 10.1037/hea0000306. https://europepmc.org/abstract/MED/26651462 .2015-56045-002 [DOI] [PMC free article] [PubMed] [Google Scholar]
47.Supplemental materials: developing message strategies to engage racial and ethnic miniority groups in digital oral self-care interventions: participatory co-design approach. Open Science Framework. 2023. Aug 02, [2023-11-14]. https://osf.io/e9rbn/?view_only=7e043ebcf5774dbf81dffffe8f5db9a3 . [DOI] [PMC free article] [PubMed]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

Mean ratings of the 4 message types. Q&A: Question and Answer.

formative_v7i1e49179_app1.png^{(117KB, png)}

Data Availability Statement

Deidentified data sets generated and analyzed during this research are available in the Open Science Framework repository [47].

[ref1] 1.Peres MA, Macpherson LM, Weyant RJ, Daly B, Venturelli R, Mathur MR, Listl S, Celeste RK, Guarnizo-Herreño CC, Kearns C, Benzian H, Allison P, Watt RG. Oral diseases: a global public health challenge. Lancet. 2019 Jul 20;394(10194):249–60. doi: 10.1016/S0140-6736(19)31146-8.S0140-6736(19)31146-8 [DOI] [PubMed] [Google Scholar]

[ref2] 2.Flores G, Lin H. Trends in racial/ethnic disparities in medical and oral health, access to care, and use of services in US children: has anything changed over the years? Int J Equity Health. 2013 Jan 22;12(1):10. doi: 10.1186/1475-9276-12-10. https://equityhealthj.biomedcentral.com/articles/10.1186/1475-9276-12-10 .1475-9276-12-10 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref3] 3.Raison H, Corcoran R, Harris RV. Is toothbrushing behaviour habitual? Cues, context, motivators and patient narratives. Community Dent Oral Epidemiol. 2021 Oct;49(5):478–86. doi: 10.1111/cdoe.12624. [DOI] [PubMed] [Google Scholar]

[ref4] 4.Elyousfi S, Innes N, Leggett H, Ainsworth H, Chestnutt IG, Day P, Robertson M, Pavitt S, Kellar I, Dey D, Marshman Z. Acceptability of the Brushing RemInder 4 Good oral HealTh (BRIGHT) trial intervention: a qualitative study of perspectives of young people and school staff. BMC Oral Health. 2022 Feb 23;22:44. doi: 10.1186/s12903-022-02073-w. https://bmcoralhealth.biomedcentral.com/articles/10.1186/s12903-022-02073-w .10.1186/s12903-022-02073-w [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref5] 5.Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med. 2015 Apr 15;7(283):283rv3. doi: 10.1126/scitranslmed.aaa3487. https://europepmc.org/abstract/MED/25877894 .7/283/283rv3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref6] 6.Atske S, Perrin A. Home broadband adoption, computer ownership vary by race, ethnicity in the U.S. Pew Research Center. 2021. [2023-11-02]. https://www.pewresearch.org/fact-tank/2021/07/16/home-broadband-adoption-computer-ownership-vary-by-race-ethnicity-in-the-u-s/

[ref7] 7.Carpenter SM, Menictas M, Nahum-Shani I, Wetter DW, Murphy SA. Developments in mobile health just-in-time adaptive interventions for addiction science. Curr Addict Rep. 2020 Sep;7(3):280–90. doi: 10.1007/s40429-020-00322-y. https://europepmc.org/abstract/MED/33747711 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref8] 8.Carpenter SM, Yap J, Patrick ME, Morrell N, Dziak JJ, Almirall D, Yoon C, Nahum-Shani I. Self-relevant appeals to engage in self-monitoring of alcohol use: a microrandomized trial. Psychol Addict Behav. 2023 May;37(3):434–46. doi: 10.1037/adb0000855.2022-81168-001 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref9] 9.Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. 2022 Oct;77(7):836–52. doi: 10.1037/amp0000983. https://europepmc.org/abstract/MED/35298199 .2022-41985-001 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref10] 10.Boucher EM, Ward HE, Mounts AC, Parks AC. Engagement in digital mental health interventions: can monetary incentives help? Front Psychol. 2021 Nov 18;12:746324. doi: 10.3389/fpsyg.2021.746324. https://europepmc.org/abstract/MED/34867629 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref11] 11.Gan DZ, McGillivray L, Larsen ME, Torok M. Promoting engagement with self-guided digital therapeutics for mental health: Insights from a cross-sectional survey of end-users. J Clin Psychol. 2023 May 24;79(5):1386–97. doi: 10.1002/jclp.23486. [DOI] [PubMed] [Google Scholar]

[ref12] 12.Szinay D, Forbes CC, Busse H, DeSmet A, Smit ES, König LM. Is the uptake, engagement, and effectiveness of exclusively mobile interventions for the promotion of weight-related behaviors equal for all? A systematic review. Obes Rev. 2023 Mar;24(3):e13542. doi: 10.1111/obr.13542. [DOI] [PubMed] [Google Scholar]

[ref13] 13.Muscente F, De Caterina R. Challenges in ischaemic heart disease: not sleeping enough, not brushing your teeth, and skipping breakfast-three ways of increasing your risk of myocardial infarction? Eur Heart J Suppl. 2020 Nov;22(Suppl L):L57–60. doi: 10.1093/eurheartj/suaa136. https://europepmc.org/abstract/MED/33654468 .suaa136 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref14] 14.Burger JM, Sanchez J, Imberi JE, Grande LR. The norm of reciprocity as an internalized social norm: returning favors even when no one finds out. Soc Influ. 2009 Jan;4(1):11–7. doi: 10.1080/15534510802131004. https://www.tandfonline.com/doi/full/10.1080/15534510802131004 . [DOI] [Google Scholar]

[ref15] 15.Cialdini RB, Goldstein NJ. Social influence: compliance and conformity. Annu Rev Psychol. 2004;55:591–621. doi: 10.1146/annurev.psych.55.090902.142015. [DOI] [PubMed] [Google Scholar]

[ref16] 16.Gouldner AW. The norm of reciprocity: a preliminary statement. Am Sociol Rev. 1960 Apr;25(2):161. doi: 10.2307/2092623. https://www.jstor.org/stable/2092623 . [DOI] [Google Scholar]

[ref17] 17.Nahum-Shani I, Rabbi M, Yap J, Philyaw-Kotov ML, Klasnja P, Bonar EE, Cunningham RM, Murphy SA, Walton MA. Translating strategies for promoting engagement in mobile health: a proof-of-concept microrandomized trial. Health Psychol. 2021 Dec;40(12):974–87. doi: 10.1037/hea0001101. https://europepmc.org/abstract/MED/34735165 .2022-00033-001 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref18] 18.Goldstein NJ, Griskevicius V, Cialdini RB. Reciprocity by proxy: a novel influence strategy for stimulating cooperation. Adm Sci Q. 2011;56(3):441–73. doi: 10.1177/0001839211435904. https://journals.sagepub.com/doi/10.1177/0001839211435904 . [DOI] [Google Scholar]

[ref19] 19.Litman JA, Spielberger CD. Measuring epistemic curiosity and its diversive and specific components. J Pers Assess. 2003 Feb;80(1):75–86. doi: 10.1207/S15327752JPA8001_16. [DOI] [PubMed] [Google Scholar]

[ref20] 20.Liu F, Ngai E, Ju X. Understanding mobile health service use: an investigation of routine and emergency use intentions. Int J Inf Manage. 2019 Apr;45:107–17. doi: 10.1016/j.ijinfomgt.2018.09.004. https://www.sciencedirect.com/science/article/abs/pii/S026840121730943X . [DOI] [Google Scholar]

[ref21] 21.Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, Murphy SA. Just-in-Time Adaptive Interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann Behav Med. 2018 May 18;52(6):446–62. doi: 10.1007/s12160-016-9830-8. https://europepmc.org/abstract/MED/27663578 .10.1007/s12160-016-9830-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref22] 22.Reeve J. The interest-enjoyment distinction in intrinsic motivation. Motiv Emot. 1989;13(2):83–103. doi: 10.1007/bf00992956. https://link.springer.com/article/10.1007/BF00992956 . [DOI] [Google Scholar]

[ref23] 23.Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010 Jan;44(1):101–8. doi: 10.1111/j.1365-2923.2009.03546.x.MED3546 [DOI] [PubMed] [Google Scholar]

[ref24] 24.Buder J, Schwind C. Learning with personalized recommender systems: a psychological view. Comput Human Behav. 2012 Jan;28(1):207–16. doi: 10.1016/j.chb.2011.09.002. https://www.sciencedirect.com/science/article/abs/pii/S0747563211001956 . [DOI] [Google Scholar]

[ref25] 25.Cheung KL, Durusu D, Sui X, de Vries H. How recommender systems could support and enhance computer-tailored digital health programs: a scoping review. Digit Health. 2019 Jan 24;5:2055207618824727. doi: 10.1177/2055207618824727. https://journals.sagepub.com/doi/10.1177/2055207618824727?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_2055207618824727 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref26] 26.Hors-Fraile S, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit A, Spachos D, Bamidis P, de Vries H. Tailoring motivational health messages for smoking cessation using an mHealth recommender system integrated with an electronic health record: a study protocol. BMC Public Health. 2018 Jun 05;18(1):698. doi: 10.1186/s12889-018-5612-5. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-018-5612-5 .10.1186/s12889-018-5612-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref27] 27.Muench F, Baumel A. More than a text message: dismantling digital triggers to curate behavior change in patient-centered health interventions. J Med Internet Res. 2017 May 26;19(5):e147. doi: 10.2196/jmir.7463. https://www.jmir.org/2017/5/e147/ v19i5e147 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref28] 28.Fredrickson BL. The broaden-and-build theory of positive emotions. Philos Trans R Soc Lond B Biol Sci. 2004 Sep 29;359(1449):1367–77. doi: 10.1098/rstb.2004.1512. https://europepmc.org/abstract/MED/15347528 .DH5NJJTNQCAMP8UT [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref29] 29.Harmon-Jones E, Harmon-Jones C, Price TF. What is approach motivation? Emotion Review. 2013 Jun 11;5(3):291–5. doi: 10.1177/1754073913477509. https://journals.sagepub.com/doi/10.1177/1754073913477509 . [DOI] [Google Scholar]

[ref30] 30.Steffens PR, Weeks CS, Davidsson P, Isaak L. Shouting from the ivory tower: a marketing approach to improve communication of academic research to entrepreneurs. Entrep Theory Pract. 2014 Mar 01;38(2):399–426. doi: 10.1111/etap.12079. https://journals.sagepub.com/doi/abs/10.1111/etap.12079 . [DOI] [Google Scholar]

[ref31] 31.Reese JM, Joseph RP, Cherrington A, Allison J, Kim YI, Spear B, Childs G, Simpson T, Durant NH. Development of participant-informed text messages to promote physical activity among African American women attending college: a qualitative mixed-methods inquiry. J Transcult Nurs. 2017 May;28(3):236–42. doi: 10.1177/1043659616644959. https://europepmc.org/abstract/MED/27093904 .1043659616644959 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref32] 32.Ownby RL, Acevedo A, Waldrop-Valverde D. Enhancing the impact of mobile health literacy interventions to reduce health disparities. Q Rev Distance Educ. 2019;20(1):15–34. https://europepmc.org/abstract/MED/31537979 . [PMC free article] [PubMed] [Google Scholar]

[ref33] 33.Cerutti P, Marconi A, Pozzi S, Gragnano G, Pain D, Solara V, Morales JR, Paone E, Emolumento AM, Rossi G. Psychological intervention in amyotrophic lateral sclerosis: suggestions for clinical practice. J Psychol Clin Psychiatry. 2017 Oct 11;8(2):00480. doi: 10.15406/jpcpy.2017.08.00480. https://medcraveonline.com/JPCPY/psychological-intervention-in-amyotrophic-lateral-sclerosis-suggestions-for-clinical-practice.html . [DOI] [Google Scholar]

[ref34] 34.Sim J, Waterfield J. Focus group methodology: some ethical challenges. Qual Quant. 2019 Jul 16;53(6):3003–22. doi: 10.1007/s11135-019-00914-5. https://link.springer.com/article/10.1007/s11135-019-00914-5 . [DOI] [Google Scholar]

[ref35] 35.Cornejo M, Bustamante J, Del Río M, De Toro X, Latorre MS. Researching with qualitative methodologies in the time of coronavirus: clues and challenges. Int J Qual Methods. 2023 Jan 03;22:16094069221150110. doi: 10.1177/16094069221150110. https://europepmc.org/abstract/MED/36628132 .10.1177_16094069221150110 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref36] 36.Johnson B, Odhner K. Focus groups from home: conducting virtual focus groups during the COVID-19 pandemic and beyond. Coll Res Libr News. 2021;82(6):258–61. doi: 10.5860/crln.82.6.258. https://crln.acrl.org/index.php/crlnews/article/view/24971/32810 . [DOI] [Google Scholar]

[ref37] 37.Dos Santos Marques IC, Theiss LM, Johnson CY, McLin E, Ruf BA, Vickers SM, Fouad MN, Scarinci IC, Chu DI. Implementation of virtual focus groups for qualitative data collection in a global pandemic. Am J Surg. 2021 May;221(5):918–22. doi: 10.1016/j.amjsurg.2020.10.009. https://europepmc.org/abstract/MED/33070983 .S0002-9610(20)30620-6 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref38] 38.Hwang AH, Wang CY, Yang YY, Won AS. Hide and seek: choices of virtual backgrounds in video chats and their effects on perception. Proc ACM Hum Comput Interact. 2021 Oct 18;5(CSCW2):1–30. doi: 10.1145/3476044. [DOI] [Google Scholar]

[ref39] 39.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health. 2014 Feb;104(2):e16–31. doi: 10.2105/AJPH.2013.301706. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref40] 40.Lathen L, Laestadius L. Reflections on online focus group research with low socio-economic status African American adults during COVID-19. Int J Qual Methods. 2021 Jun 03;20:160940692110217. doi: 10.1177/16094069211021713. https://journals.sagepub.com/doi/full/10.1177/16094069211021713 . [DOI] [Google Scholar]

[ref41] 41.Collins SE, Clifasefi SL, Stanton J, The Leap Advisory Board. Straits KJ, Gil-Kashiwabara E, Rodriguez Espinosa P, Nicasio AV, Andrasik MP, Hawes SM, Miller KA, Nelson LA, Orfaly VE, Duran BM, Wallerstein N. Community-based participatory research (CBPR): towards equitable involvement of community in psychology research. Am Psychol. 2018 Oct;73(7):884–98. doi: 10.1037/amp0000167. https://europepmc.org/abstract/MED/29355352 .2018-02701-001 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref42] 42.Torres VN, Williams EC, Ceballos RM, Donovan DM, Duran B, Ornelas IJ. Participant engagement in a community based participatory research study to reduce alcohol use among Latino immigrant men. Health Educ Res. 2020 Dec 23;35(6):627–36. doi: 10.1093/her/cyaa039. https://europepmc.org/abstract/MED/33025021 .5918718 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref43] 43.Qian T, Walton AE, Collins LM, Klasnja P, Lanza ST, Nahum-Shani I, Rabbi M, Russell MA, Walton MA, Yoo H, Murphy SA. The microrandomized trial for developing digital interventions: experimental design and data analysis considerations. Psychol Methods. 2022 Oct;27(5):874–94. doi: 10.1037/met0000283. https://europepmc.org/abstract/MED/35025583 .2022-21177-001 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref44] 44.Liao P, Klasnja P, Tewari A, Murphy SA. Sample size calculations for micro-randomized trials in mHealth. Stat Med. 2016 May 30;35(12):1944–71. doi: 10.1002/sim.6847. https://europepmc.org/abstract/MED/26707831 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref45] 45.Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, Murphy SA. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2015 Dec;34S:1220–8. doi: 10.1037/hea0000305. https://europepmc.org/abstract/MED/26651463 .2015-56045-003 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref46] 46.Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol. 2015 Dec;34(Suppl):1209–19. doi: 10.1037/hea0000306. https://europepmc.org/abstract/MED/26651462 .2015-56045-002 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref47] 47.Supplemental materials: developing message strategies to engage racial and ethnic miniority groups in digital oral self-care interventions: participatory co-design approach. Open Science Framework. 2023. Aug 02, [2023-11-14]. https://osf.io/e9rbn/?view_only=7e043ebcf5774dbf81dffffe8f5db9a3 . [DOI] [PMC free article] [PubMed]

PERMALINK

Developing Message Strategies to Engage Racial and Ethnic Minority Groups in Digital Oral Self-Care Interventions: Participatory Co-Design Approach

Stephanie M Carpenter, PhD

Zara M Greer, BSc

Rebecca Newman, BSc

Susan A Murphy, PhD

Vivek Shetty, DDS, DrMedDent

Inbal Nahum-Shani, PhD

Abstract

Background

Objective

Methods

Results

Conclusions

Introduction

Background

Creating Template Messages

Table 1.

Engagement With the Target Behavior

Reciprocity

Reciprocity-by-Proxy

Curiosity

Engagement With the Mobile App

Engagement With the Messages

Resource Efficiency

Message Appeal

Message Novelty

Methods

Message Development and Refinement

Feedback From Dental Experts

Feedback From MTurk Employees

Feedback From Dental Patients

Table 2.

Ethical Considerations

Message Inclusion

Procedure

Results

Quantitative

Qualitative

Table 3.

Discussion

Overview of Findings

Limitations

Next Steps

Conclusions

Acknowledgments

Abbreviations

Data Availability

Footnotes

References

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases