Skip to main content
PMC home page
Springer Nature - PMC COVID-19 Collection logoLink to Springer Nature - PMC COVID-19 Collection
. 2020 Oct 28;1(4):235–246. doi: 10.1007/s42844-020-00019-1

Best Practices for Engaging Pregnant and Postpartum Women at Risk of Substance Use in Longitudinal Research Studies: a Qualitative Examination of Participant Preferences

Lana O Beasley 1,✉, Lucia Ciciolla 2, Jens E Jespersen 1, Ashleigh L Chiaf 3, Mallory Schmidt 1, Karina M Shreffler 1, Florence J Breslin 4, Ludmila N Bakhireva 5,6, Pilar M Sanjuan 7, Julia M Stephen 8, Claire D Coles 9, Christina D Chambers 10, Julie A Kable 9, Lawrence Leeman 11, Lynn T Singer 12, Jennifer Zellner 10, Amanda S Morris 1, Julie M Croff 3,13
PMCID: PMC7592139  PMID: 33134976

Abstract

There are significant barriers in engaging pregnant and postpartum women that are considered high-risk (e.g., those experiencing substance use and/or substance use disorders (SUD)) into longitudinal research studies. To improve recruitment and retention of this population in studies spanning from the prenatal period to middle childhood, it is imperative to determine ways to improve key research engagement factors. The current manuscript uses a qualitative approach to determine important factors related to recruiting, enrolling, and retaining high-risk pregnant and postpartum women. The current sample included 41 high-risk women who participated in focus groups or individual interviews. All interviews were analyzed to identify broad themes related to engaging high-risk pregnant and parenting women in a 10-year longitudinal research project. Themes were organized into key engagement factors related to the following: (1) recruitment strategies, (2) enrollment, and (3) retention of high-risk pregnant and parenting women in longitudinal research studies. Results indicated recruitment strategies related to ideal recruitment locations, material, and who should share research study information with high-risk participants. Related to enrollment, key areas disclosed focused on enrollment decision-making, factors that create interest in joining a research project, and barriers to joining a longitudinal research study. With regard to retention, themes focused on supports needed to stay in research, barriers to staying in research, and best ways to stay in contact with high-risk participants. Overall, the current qualitative data provide preliminary data that enhance the understanding of a continuum of factors that impact engagement of high-risk pregnant and postpartum women in longitudinal research with current results indicating the need to prioritize recruitment, enrollment, and retention strategies in order to effectively engage vulnerable populations in research.

Keywords: Research engagement, Substance use disorders, Substance use, Recruitment, Retention

Introduction

There are significant barriers to recruiting and retaining individuals with overlapping vulnerabilities (i.e., substance use disorders (SUD)) in the pregnancy or postpartum period (Davis, Yao, & Bierer, 2019; McHugh Votaw, Sugarman, & Greenfield (2018) Wetherington & Roman, 1998). This may result in challenges for generalizability and therein create a relatively sparse knowledge base about the long-term outcomes for these women and their children including the environmental, mental health, physiological, and neurological factors. Filling these knowledge deficits and gaps requires ongoing assessment because research tools including those for recruitment and retention change; in addition, substance exposures in pregnancy change (e.g., prescription opioid exposure, cannabis use), thereby shifting methods to reach target populations of interest and methods to measure outcomes of interest. It is imperative for the field to identify and address engagement in research, to ensure representation of pregnant and postpartum women that use substances. Engagement in longitudinal studies will allow a more complete understanding of maternal and child health outcomes as a result of new and emerging trends in prenatal substance exposure. Enhanced understanding of participants’ perspectives on engagement and study participation will allow researchers to more fully address this pressing research and public health need.

Prenatal exposure studies began in earnest in the 1970s, after the identification and diagnosis of fetal alcohol syndrome (Jones & Smith, 1973). Careful participant selection and comparison selection were and are necessary to classify effects of prenatal exposures. Protectionist and paternalistic regulations (e.g., the National Research Act of 1974, and federal regulations designating pregnant women as a vulnerable population) excluded women from health research and limited the field’s understanding about how sex and gender shape substance use and SUD (Davis, et al. 2019; Wetherington & Roman, 1998; Institute of Medicine, 1994; Institute of Medicine, 1999a). Research studies on substance exposures during pregnancy expanded rapidly in the past 30 years, in recognition of the cocaine epidemics of the 90s, and the current increases in prenatal opioid and methamphetamine exposures (Gabrhelík, et al. 2020). Indeed, research that focused specifically on prenatal exposures and other women’s health issues has been encouraged by journal editors, policymakers, and funding agencies including the NIH Helping End Addiction Long-term (HEAL) initiative.

Despite bioethical, legal, and social concerns regarding the risks and benefits of research participation for pregnant and postpartum women who use alcohol and drugs (Davis, et al. 2019), the inclusion of vulnerable populations who are marginalized or stigmatized in research on sensitive topics has not demonstrated undue harm or exposure to unacceptable risk, and in fact, has been associated with potential benefits, such as altruism, catharsis, and gained knowledge (Alexander, Pillay, & Smith, 2018). Of course, it is important for researchers to adopt careful experimental design and safeguards that will uphold the principal of non-maleficence and protect vulnerable participants from harm (Alexander et al., 2018; Sikeweyiya & Jewkes, 2013). Exclusion of substance using populations may violate important bioethical principles of human subjects research, particularly the principles of autonomy, beneficence, and justice (Alexander et al., 2018). Exclusion from research not only strips individuals from making decisions about their own autonomy and denies them potential benefits of participating, but also exposes them to greater societal marginalization and may ultimately place them at increased risk of harm due to deficits in critical health knowledge and exposure to inappropriate or ineffective treatments (Alexander et al., 2018; Johnson et al., 2014; Institute of Medicine, 1999b).

Unfortunately, prenatal exposures to alcohol, tobacco, and other drugs are rising (Substance Abuse and Mental Health Services Administration, 2018), with 1 in 4 pregnancies exposed to tobacco (18.9%), alcohol consumption (10.0%), or illicit drug use (4.7%) (Ebrahim & Gfroerer, 2003; Havens, Simmons, Shannon, & Hansen, 2009). Specifically, opioid exposed pregnancies have increased from 1.5 to 6.5 per 1000 pregnancies (Haight, Ko, Tong, Bohm, & Callaghan, 2018). Yet, cannabis exposures are the most prevalent drug exposure, with nearly 7–8% reported exposure in the first trimester (Alshaarawy & Anthony, 2019). Rising rates of substance exposure correspond to increasing health risks and adverse outcomes at great societal cost and burden to systems of health care and social services, as well as criminal justice.

Notably, researchers involved in the NIDA-funded Perinatal-20 Treatment Research Demonstration Program that focused on SUD treatment for pregnant and postpartum women identified seven clinical factors that contributed to significant difficulty and complexity in the recruitment and retention of women in substance use treatment research, including as follows: (1) severity of SUD, (2) legal system involvement, (3) housing instability, (4) interpersonal relationship challenges, (5) parenting responsibilities, (6) employment challenges, and (7) need for more intensive services. These difficulties with recruitment and retention contribute to additional complications for research, including biased samples of convenience recruited through referrals from social and health agencies, limited sample diversity, deviations from the research design, and ethical issues associated with risk and benefits of participation and involvement with the criminal justice or child welfare system. In particular, when research designs do not involve the possibility of direct benefit due to participation (i.e., observational versus intervention studies), it is important to understand the unique reasons and motivations that drive decision-making about research participation (Hallowell et al., 2010).

Due to all of the aforementioned factors that potentially inhibit the inclusion and engagement of high-risk participants (i.e., participants previously or currently using substances), it is imperative to understand the motivations for engagement in research among high-risk participants, focusing specifically on understanding motivation for research participation, factors that influence decision-making about participation, and barriers to participation.

HEALthy Brain and Child Development (HBCD) Study

The current study reports results from a qualitative research study conducted as part of an 18-month, multi-site pilot study aimed to develop and demonstrate feasibility of an experimental design for a 10-year, prospective, longitudinal investigation of normative childhood brain development, beginning in pregnancy. A major aim of the 10-year study will be to determine factors that alter brain development including prenatal exposure to opioids and other psychoactive substances, as well as other prenatal and childhood environmental exposures. This goal necessitates recruiting pregnant women previously or currently using substances, as well as a large group of pregnant women who are at low risk of prenatal substance use. Two of the primary aims of the pilot are (1) developing and testing recruitment and retention strategies and (2) addressing ethical and legal challenges of conducting research with a stigmatized and vulnerable population.

Current Research Study

The current qualitative study is one arm of the 5-site consortium to improve understanding, from a qualitative perspective, the continuum of engagement of low- and high-risk participants in research. This manuscript focuses on the results of the distinct needs and responses of high-risk participants. Specifically, the objectives in this analysis were to address important factors that impact best practices in (1) promoting longitudinal research to high-risk participants, (2) enrolling high-risk participants in research, and (3) retaining high-risk participants in research studies.

Methods

Participants

Individual interviews and one focus group were conducted with a total of 41 women (five participated in a focus group). Women were at high-risk of prenatal or postnatal substance use and were identified through medical clinics, other research study involvement, or SUD treatment programs. Recruitment took place across five sites in the USA located in California, Georgia, New Mexico, Ohio, and Oklahoma (see Table 1 for demographic information). High-risk pregnant and postpartum women were defined in the current study as a parenting or pregnant woman who had used alcohol and tobacco and/or had a current or past history of SUD. Some participants were currently receiving SUD treatment. Contact was made through trained research personnel located at each specific site with 41 total participants taking part in the current study (California, N = 3; Georgia, N = 4; New Mexico, N = 9, Ohio, N = 8; Oklahoma, N = 17). Only one focus group that included five women was combined with the individual interviews. The one focus group was conducted in New Mexico prior to group restrictions imposed due to COVID-19.

Table 1.

Key socioeconomic and demographic variables

Demographics Outcome
Marital status, n (%)
  Never married 51%
  Married 27%
  Living with significant other 8%
  Divorced 10%
  Separated 2%
  Widowed 2%
Educational background, n (%)
  Completed less than 9th grade 2%
  Completed grades 9–12 (did not complete high school) 10%
  Completed high school 22%
  Obtained GED 27%
  Completed some college (no degree) 20%
  Attended vocational technology program 5%
  Associate’s degree 10%
  Bachelor’s degree 2%
  Master’s degree 2%
Received government assistance, n (%) 90%
  Household income (last month) $0–$8500 (Mean = $1470.73; SD = $1978.95)
Race/ethnicity
  White 56%
  Hispanic/Latino 15%
  Black/African American 10%
  Multi-racial 10%
  American Indian 7%
  Asian 2%
Open in a new tab

Procedures

Qualitative methods for the research team, study design, and analysis followed the guidelines recommended by Tong, Sainsbury, and Craig (2007). Qualitative study recruitment began with sites contacting participants in person or by phone and describing the current study and qualitative interview process. All women who expressed interest in participating were scheduled for either a focus group or individual interview depending on whether the interview took place prior to or following COVID-19 restrictions regarding in-person gatherings. Interviews conducted during the COVID-19 restrictions were conducted individually by phone. All participants gave oral informed consent. During the consent process, a brief overview of the qualitative study and all safety measures taken to ensure confidentiality were discussed.

Trained qualitative research assistants collected all qualitative data from March 2020 through June 2020. Before engaging in focus groups/individual phone interviews, all participants completed an in-person or online survey that included a demographic questionnaire and watched a short video describing the protocols planned for the larger, longitudinal study including neuroimaging (e.g., MRI), neurodevelopmental, and biospecimen collection. For the focus group, snacks were provided. Participants received a $50–75 incentive for their participation, and this varied based on site. All focus groups and individual interviews were audio-recorded and lasted approximately 45–60 min. Transcription work was conducted by qualitative team members or a transcription company, with team members crosschecking all transcripts to verify accuracy. During the transcription process, all identifying information was removed to ensure privacy. All procedures were approved by the sIRB for the 5-site consortium.

Individual Interview Guide Development and Data Merging

Focus group and individual interview guides for the current project were developed by the first author, in conjunction with the evaluation team and other sites within the research consortium reviewing and revising the guide as needed. Focus group and individual interviews were coded individually and combined for data analysis. All coding and data analysis was conducted at one site. Recordings were transferred securely according to IRB-approved methods. It is important to note that focus group and individual data themes were examined a priori and themes were congruent and therefore data were merged.

Data Analysis

Qualitative data was analyzed using the NVivo© 11 software. Five qualitative researchers worked together to develop a codebook focused on broad themes influenced by the semi-structured interview guide. Thematic analysis was used to define specific themes within the broader categories (Braun & Clarke, 2014). The codebook was developed using an agreed upon coding scheme with themes not being predetermined but rather emerging from the data. Upon completion of the codebook, two teams consisting of two qualitative researchers coded all transcriptions using developed coding templates. Cleaning of data took place as needed (broader codes enveloping smaller codes). Once coded, inter-coder reliability was established using simple percent agreement, which is a commonly used method for assessing reliability in qualitative studies (Lombard, Snyder-Duch, and Bracken, 2006; Stemler, 2004). Average inter-coder reliability was over 85%. In the “Results” section, themes are described in more detail.

The validity of the current research findings are enhanced by several design factors such as the calculation of salient factors using percentage of comments and the team-based approach used for coding. Specifically, calculating the percentage of comments from participants related to specific themes allowed the research team to ensure that themes discussed in the current paper were saturated or were discussed frequently in focus group/individual interviews. Therefore, relying on percent of comments strengthens demonstration of saturation in the current study. Further, the majority of qualitative data were collected from individual interviews (N = 36) rather than a focus group (N = 5), which allows for a more in-depth conversation. Specifically, during individual interviews, comments were able to be probed deeply with rich content emerging throughout the qualitative data, rather than simple agreement or disagreement that often emerges from focus group data collection. Additionally, the fact that both primary researchers as well as consortium partners were involved in developing the focus group/individual interview guide is a strength, increasing the likelihood that the items on the interview guide validly and comprehensively captured the intent of the research aims.

Results

Qualitative results for the current study focused on a continuum of engagement of high-risk participants in a multi-year prospective longitudinal, cohort design research study. Specifically, themes were organized into key engagement factors related to the following: (1) promoting research to high-risk participants, (2) enrollment of high-risk participants in research, and (3) retention of high-risk participants in a multi-year prospective longitudinal, cohort design research study.

Key socioeconomic and demographic variables for the study participants are presented in Table 1. The current sample was more racially/ethnically diverse than the general US population, with most participants identifying as White (56%), followed by Hispanic/Latino (15%), Black/African American (10%), Multi-racial (10%), American Indian (7%), and Asian (2%). Many participants reported they had never married (51%) with other participants reporting they were currently married (27%), living with a significant other (8%), or divorced (10%). Many participants reported completing high school (22%) with an additional 27% reporting obtaining a GED. The average monthly income was $1470.73 per month (SD = $1978.95) and most participants reported receiving some form of government assistance (90%).

Marketing Research to High-risk Participants

Best Ways to Promote Research Studies to High-risk Participants

Participants were asked about the best ways to promote a longitudinal research study to recruit participants and their children into studies with participants explaining the best recruitment locations, ideal individuals to share research study information, and the best type of recruitment material.

Participants reported that social media is an ideal platform to reach participants (31% of comments). Specific social media locations include Facebook, Instagram, and online mother groups. One participant explained, “A lot of people use social media. So, if it got posted to somebody’s account, then you could share it with all of your friends. So, I think that would be great.” Participants also reported that medical offices (15% of comments) and community/state agencies (10% of comments) were good locations to share information about research studies. An example quote includes:

And maybe … when I go to (local agency) for counseling and you know, my mental health needs for me and my kids, therapy anything like that. They always have all kinds of [information] … are you struggling with this, are you interested in this, here’s a study to help you earn extra money, are you a single mom, do you have this or that, well then you may qualify for this study that we’re doing.

Other suggested locations included bus stops (8% of comments), billboards (7% of comments), and grocery/convenience stores (7% of comments). Participants also reported that child-friendly locations (i.e., library, parks; 6% of comments) and educational settings (4% of comments) were good locations to share study information. One participant explained, “Maybe flyers at childcare centers and stuff like that, where they have the younger school-aged kids from infant to whatever. I know a lot of moms frequent those places.” When participants were asked about locations they would trust the most to receive information, it was disclosed that medical offices such as doctor offices/clinics, state agencies (e.g., women and children food programs, other human services), and educational settings were considered most trustworthy.

Participants also discussed the type of material they would recommend using to promote research studies with participants explaining they would use commercials (28% of comments), brochures (23% of comments), and radio (15% of comments) to share information about research studies. It was also reported that using news/newspapers (6% of comments), online marketing strategies (6% of comments), and sharing information through word of mouth (4% of comments) would be most effective.

In terms of the individuals that would be best to share research study information, participants stated that study information would best come from medical personnel (42% of comments), friends (15% of comments), family members (13% of comments), other participants (9% of comments), and professionals (9% of comments). Professionals included counselors/therapists, daycare personnel, clergy, and staff at resources centers. Participants were also asked who they would find the most trustworthy in sharing information with them and disclosed doctors (57% of comments), family/other mothers (14% of comments), and friends (12% of comments) would be most trusted. An example quote regarding who participants would trust the most was:

…when you think of doctors you think you can trust them more, because they’re there to help you take care of your kids or take care of yourself … the doctor isn’t going to try to scam you out of something so when you get information from them about a study … you tend to want to read about what they are giving you.

Enrollment of High-risk Participants in Research

Research Enrollment Decision-making

Participants were asked if any information could be provided during recruitment to help them make the decision of whether or not to enroll in a 10-year research project that includes data collection from them and their children. The most frequent request for more information involved study procedures (45% of comments). Specifically, participants wanted to know more information about data collection, including procedures related to neuroimaging such as magnetic resonance imaging (MRIs), biosampling, and frequency of procedures. An example quote of wanting more information on study procedures was:

…looking at everything that might be required of participants, it seems kind of … a bit invasive and like kind of a large commitment in regards to [sic] coming for MRIs and sending blood work, the diapers, giving hair, things like that.

Participants also explained that they wanted to know additional information about the purpose of the study (22% of comments) with participants explaining that they wanted an explanation of the research study in its entirety and how it related to child development. Another area that participants explained they needed more information about was study logistics (17% of comments) with participants explaining they would like to know the location of the study and commitment involved with being in the study. In terms of research commitment, one participant shared needing to know “… times and dates to make sure everything is scheduled right … trying to get a couple of days in at the job and … just timing stuff.”

Participants were also asked if there was anyone they would need to speak with to make a decision about enrolling in research involving themselves and their child(ren). Participants indicated they would speak to a variety of key individuals, including their significant other or the biological father of the child (52% of comments), family members (23% of comments), and medical personnel (16% of comments).

What Would Create Interest in Joining a Research Study

Participants reported that several things would make them interested in joining a research study with the most frequent being understanding the research benefits (48% of comments) to others (66% of comments) and themselves (34% of comments). An example quote regarding research benefits to others was:

I think if participating contributes to information that can help other moms in the future, if there’s more information that can be gained or developed out of this study that can be provided to other moms, before they get pregnant, as they’re pregnant, to help in their baby’s development. I feel like that’s, in some part a small contribution I can make.

Participants also reported that for research studies that involved their children, there was a strong interest in being provided study results about their children (28% of comments). This included information on brain development and overall child development. One participant explained, “it seemed like something that would be really interesting is to find out more information about your own child …not? just brain development. That’s really something that interests me.” Participants also reported that compensation would impact their interest in joining a research study (10% of comments) with several participants indicating that a potential increase in compensation over time would be attractive. Specifically, one participant explained, “I would say maybe increase the compensation as the years go on because … it’s hard to stick with a program.”

Barriers to Joining a Research Study

Participants also discussed barriers to joining a research study. Participants reported that a busy schedule (35% of comments) could make it difficult to join a research study. This includes challenges regarding the time commitment for the study and also balancing their work schedule around research study demands. Other barriers included transportation difficulties (32% of comments), travel time and location of the study (11% of comments), and having childcare while participating in research activities (11% of comments). Many comments from participants regarding barriers were concerns about taking part in a research study. Primary concerns included potential risks to the fetus/child (31% of comments) both during pregnancy and after the child is born. One participant explained, “Just making sure no harm to the baby … I mean I understand the blood samples but just making sure it’s 100% safe.” Other concerns included ensuring that participant personal and research information was kept confidential (31% of comments) as well as understanding the invasiveness of biospecimen collection (20% of comments). In terms of confidentiality, an example quote from a participant was:

…a reassurance that the information would stay private and … the only thing that made me a little uncomfortable was when you were like “we will have to take your identifying information but then it will be destroyed.” I feel like there might be something to substantiate that it’s going to be destroyed, and not just a word of mouth thing.

Retention of High-risk Participants in Research

Supports Needed to Stay in a Research Study

Additional advice was collected from participants regarding what would help them stay in a research study after enrollment. Participants reported a variety of research supports that would be helpful to stay in a research project (31% of comments). Specific research supports included providing childcare during data collection, compensation for time spent in research, and provision of transportation as needed for families. Related to this theme, one participant explained that “… childcare on site is probably a huge one” and another participant reported that “transportation would be very helpful.”

It was also explained that specific research study logistics are important to consider in supporting participants to stay in a research study (21% of comments). These included receiving regular research updates including the benefits of the study, having well-trained research staff, research staff engaging in regular contact with participants, families being close to the research/data collection site, and families being provided advanced notice of data collection. With regard to receiving research updates, a participant explained:

…getting kind of feedback about … here’s what you contributed and here’s what … we’re gonna use … information on what we found, about your baby’s development ... for me that would be important.

Flexibility was another key area of importance for participants (16% of comments). Specifically, it was reported that flexibility of the participant schedule, when research appointments are scheduled, and who brings the child to research appointments are important to support ongoing research engagement. An example quote regarding flexibility was:

So, I don’t know how long an appointment would take. But if appointments could be, quick, able to work around my schedule, flexible, and maybe offer something like what we’re doing now (phone interview). If we’re not able to meet, like some kind of tele-health option. That would make it so much easier.

Participants also reported that home visits could potentially increase their ability to stay in a research study (18% of comments) as it would make visits more convenient. It is important to note that some families reported concerns with home visits indicating that not all families feel comfortable with research personnel coming into their home. Another theme that emerged included the importance of becoming familiar with research staff (8% of comments) as it creates comfort in staff working with children involved in the research study.

Barriers to Staying in a Research Study

Participants explained barriers to staying in a research study after enrollment with the most frequent barrier being family-related barriers (61% of comments) including work schedule, family moves, generally busy schedule, lack of childcare, and child(ren) not being interested in the study (when older). With regard to family moves, a participant shared “… it would depend on where I moved and if you guys had the same research study where I moved to.” Specific research study logistics (30% of comments) were also reported as a potential barrier to staying in a research project with participants explaining barriers related to transportation, time commitment, and frequency of data collection. Specific to time commitment/frequency of data collection, one participant explained wanting more information on “How often do you want me to come in? How long are we gonna be there?”

Best Ways to Stay in Contact with High-risk Participants

Participants were also asked about the best way to stay in contact with participants during a longitudinal research study with participants reporting that collecting contact information from family members is recommended (26% of comments). Specific to information important to collect from participants, personal information such as phone number (23% of comments), email (19% of comments), social media information (11% of comments), and home address (11% of comments) were reported as ideal.

Discussion

The current qualitative data describes a continuum of factors that impact engagement of high-risk participants (i.e., those currently or in the past using substances) in a 10-year complex longitudinal research study. This continuum included factors that impact research promotion strategies, enrollment, and retention of high-risk participants in research studies (see Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Recruitment, enrollment and retention strategies of high-risk participants in research

Recruitment Strategies

Recruiting a representative sample of pregnant high-risk participants for a longitudinal study is challenging. However, considering promotion strategies, it is evident that there are a number of avenues whereby researchers may have success both finding and creating contact points with high-risk participants. Our findings illustrate how the construct of research promotion is multifaceted. Responses from interviews included three major factors: (1) the location where research material is shared, (2) the type of material used, and (3) the person sharing the research material all potentially impact marketing success.

Considering recruitment locations, participants frequently recommended social media, followed by medical offices, and community/state agencies. In regard to social media, studies have increasingly indicated social media as a key location to gain access to individuals who may be harder to reach, including high-risk individuals (Betsch, 2014; Frandsen, Walters, & Ferguson, 2014). Additionally, marketing through social media can contribute to reduced recruiting costs, shorter recruiting periods, and better population representation (Maloni, Przeworski, & Damato, 2013; Whitaker, Stevelink, & Fear, 2017). Social media has become increasingly common as a major component of research recruitment strategies in recent years (Fusar-Poli et al., 2016; Whitaker, et al. 2017).

Television commercials, brochures, and radio ads were the three most often recommended mediums. These more traditional marketing methods have been used in research for decades; however, barriers in the use of television and radio advertisements have become more relevant in recent years. For example, many of those who watch television have begun resorting to streaming services. These streaming services often allow users to skip ads, or even remove ads all together. Additionally, online radio services with ad-free options have become increasingly common (Wlömert & Papies, 2016). However, although these trends are occurring in the general population, less is known regarding how changes in television and music consumption have changed specifically for pregnant and parenting women with SUD.

Medical personnel, family members, and friends comprised the limited sources for trusted information on research studies for participants. This suggests close interpersonal relationships are important to consider during recruitment. Moreover, doctors and nurses may be successfully engaged in the recruitment process where feasible. This is consistent with research Newington and Metcalfe (2014) showing that forming collaborations with trusted medical professionals aided in both identifying and gaining access to eligible, hard-to-reach participants. Adding to support this, when asked who they would trust the most to receive research recruitment information materials from, the most common answer among participants was doctors.

Enrollment Strategies

The decision to enroll in a long-term research study with few or no direct benefits for participants is complex and multifaceted (Hallowell et al., 2010). For pregnant and parenting women who use drugs and/or alcohol, the decision can be further complicated by concerns about privacy and safety, logistics related to participation, and details about study procedures and how they relate to child development. The interview and focus group responses from our participants indicated that comprehensive information about specific study procedures and the purpose of the study was most important for informing research participation, with particular interest in understanding the commitment and burden (i.e., time, effort) associated with participation. Additionally, almost a third of participants reported concerns about safety and invasiveness of study procedures, particularly the risks to their child, as well as concerns about confidentiality and maintaining the privacy of their personal information. In addition to the concerns about privacy, participants expressed a desire for reassurance that they could trust the research team, and to that end, it was important that study procedures were clearly explained so that they could weigh the risks of participating with potential benefits. This speaks to the importance of autonomy in decision-making and informed consent practices, whereby participants’ ability to weigh the costs and (potential) benefits of participation and to make the decision for themselves should be valued and respected (Alexander et al., 2018).

Interestingly, when participants were asked about factors that would influence motivation to participate in non-therapeutic research, a number of participants spoke about understanding the perceived benefits for others as well as for themselves. Consistent with previous research showing that altruism was a key motivator for research participation (Alexander et al., 2018; Hallowell et al., 2010), current participants indicated that “helping other moms in the future” was an important factor in driving motivation. Specifically, participants explained that related to others, they wanted to know more about how research knowledge could support other participants and children as well as how physicians could support young children and families.

Although altruism has been associated with positive health benefits (Post, 2005), the indirect benefits of charitable helping, such as positive mood and enhanced meaning, can be difficult to articulate and capture, especially when the help provided does not have a clearly defined beneficiary or observable impact (i.e., helping an unknown other, Sikeweyiya & Jewkes, 2013). Notably, altruistic motivation was more likely to occur among participants who were better informed about research generally and what might be gained through research participation at the level of the community or society more broadly (Sikeweyiya & Jewkes, 2013). These findings suggest that altruism is a motivating factor that could be more directly addressed in marketing and informed consent processes, with additional information provided to potential participants about the value of research participation more generally, as well as how participants’ specific data will help others, as much as it can be known or anticipated.

Direct compensation for participating was also identified as an important factor in deciding whether to participate in research. The decision to participate in research is shaped by personal situations and life factors, in particular, money and time. Even among participants who reported altruism as a motivating factor, many expected a mutually beneficial interaction that both contributed to the betterment of society and compensated them directly (Owen-Smith et al., 2016; Sikeweyiya & Jewkes, 2013). Previous research has shown that participants who are older or have financial hardship were more likely to expect a direct material compensation for participation (Sikeweyiya & Jewkes, 2013). If the real costs of study participation are not covered, then study participation can add to an already financially burdened household. In attempting to recruit high-risk or vulnerable populations into research, it is important to consider material compensation as both a motivating factor and a factor that reduces barriers to participation in the context of socioeconomic hardship without introducing undue coercion. A number of participants spoke of concerns about balancing research participation around their work schedules, as well as the costs associated with participation in terms of time, transportation, and childcare. Thus, consistent with previous research with vulnerable populations (Owen-Smith et al., 2016), compensation that adequately compensates time (especially if any work needs to be missed or childcare must be obtained), effort, and inconvenience is an important factor for motivating enrollment, with special attention given to unique participant needs and preferences regarding type of incentive (George, Duran, & Norris, 2014; Owen-Smith et al., 2016). Transportation has also been identified as a barrier in multiple studies involving high-risk families (e.g., Mendez, 2010), and providing transportation is necessary to ensure a diverse sample.

Retention Strategies

One of the most common threats to internal validity to any longitudinal research is attrition and loss to follow-up bias. Therefore, the thoughtful implementation of retention strategies can prove critical for conducting research among high-risk populations (Dumka, Garza, Roosa, & Stoerzinger, 1997). In regard to such strategies, three major themes emerged when considering how to enhance study retention: (1) specific supports that can help participants remain in the research study, (2) barriers to be aware of that can potentially make it difficult to continue study participation, and (3) the best channels for staying in contact with participants over the course of the study.

Participants most often reported needs of support in areas of childcare, transportation, and being compensated for their time. These findings align well with previous research, as childcare and transportation needs tend to be more common among vulnerable populations (Dilworth-Anderson, 2011; Haley et al., 2014a, b). As such, offering support in terms of transportation assistance and compensation could prove beneficial in terms of retention. Additionally, many participants recommended home visits by research staff as a potential solution to transportation and childcare barriers. However, it should be noted that while home visits may aid in reducing potential barriers, a number of participants in the current study voiced that they would not feel comfortable with individuals coming in their home. Therefore, prior to the use of home visits, researchers must ensure that families feel comfortable with visits taking place within the home or consider giving participants an option of laboratory-only visits. These options support participant decision-making, a recurring theme among participants.

Logistical factors mentioned to increase retention included providing participants with regular research updates, advanced notice for study appointments, and a sense of familiarity with research staff. In line with these recommendations, studies that consistently engage with participants via appointment reminders and research updates can foster a sense of anticipation and progress in participants (Kim, Hickman, Gali, Orozco, & Prochaska, 2014). Moreover, a number of researcher characteristics can contribute to participant engagement and retention including being well experienced with the services provided/research protocols, flexibility, being non-judgmental, and being culturally competent (Beasley et al., 2018; O’Brien et al., 2012).

High-risk participants often face a disproportionate number of barriers to remaining engaged in research studies (Kim et al., 2014). Indeed, it has been documented that retention rates for vulnerable families are often mitigated by higher instances of unpredictable negative life events (e.g., car problems, unreliable phone access, eviction) while possessing fewer resources to compensate for them (Heinrichs, Bertram, Kuschel, & Hahlweg, 2005; Nicholson et al., 2011). Additionally, it has been found that low-income, high-risk families tend to move more often than those in elevated SES categories, while also being at a higher risk for experiencing evictions and homelessness (Phinney, 2013).

Participants in the current study mentioned barriers related to work schedules, inconsistent daily routines, a lack of childcare, and the potential lack of child interest when children are older. Despite these barriers, a number of studies have made attempts to work around some of these issues. For example, studies have been successful in addressing schedule barriers by allowing participants to designate appointment times that would be most convenient for their family (Dumka et al., 1997), by meeting participants in-person at a hospital or clinic appointment (Kim et al., 2014) or by offering services within the home to reduce transportation and child care needs (Fifolt, Lanzi, Johns, Strichik, & Preskitt, 2017). Moreover, it has been found in vulnerable populations that providing childcare and an environment that is child-friendly can ease parents’ burden as well and boost young children’s motivation and interest in participating (Chaffin et al., 2009; Dumka et al., 1997).

The most common methods for maintaining contact with participants throughout the 10-year study period that were recommended by participants were personal phone number, email, social media, and home address, in that order. While personal phone numbers can provide the most immediate access to an individual, mobile phone numbers have been found to change more often than other modes of contact, such as email or even social media accounts (Haley et al., 2014a, b). Therefore, gathering as many contact modalities as possible, as well as contact information from two close friends or family members, can contribute to better participant tracing and retention rates (Haley et al., 2014a, b; Nicholson et al., 2011).

This study may be limited in generalizability by sample demographics. Caution should be used in applying the findings of these high-risk participants to all women at-risk of substance use in pregnancy. This study was also limited because the research method changed during the course of the study due to COVID restrictions, resulting in the combination of a focus group with individual interviews. In addition, almost half of the women interviewed were from the Oklahoma site. Findings should be interpreted in light of these limitations.

Conclusions

Overall, the current study adds vital information to understanding the complexities of marketing, enrollment, and retention strategies when conducting research with participants that are at high risk for substance exposed pregnancies. Several key factors proved to be important across a variety of areas related to enrollment and retention. Specifically, transportation was found to influence enrollment decision-making, as a barrier to joining a research project and as a support that was needed to stay in a research project. Childcare is another area that was reported to impact enrollment and retention. These results indicate the importance of understanding transportation and childcare availability during data collection and to consider ways to support participants in accessing the study location with child supports in place. Other key areas that were discussed within enrollment and retention were benefits of the study and compensation. Specifically, participants reported that understanding the benefits to self and others was important. These findings indicate the importance of reporting potential benefits and compensation not only when recruiting participants to enroll in a research project, but also to continue this conversation to retain research participants. Lastly, across enrollment and retention, busy schedule of participants is an important consideration. Leaning on another theme within the retention strategies, it is important to remain flexible with scheduling data collections so that participants are able to work appointments around a potentially chaotic schedule.

A potential research barrier in the current study is the possibility that participants might have difficulties in conceptualizing the continuum of engagement in a longitudinal research study without actually being enrolled and experiencing the study. To mediate this barrier, many of the participants in the current study had been involved in other research in an effort to include participants that had some research experience. Additionally, answers to qualitative questions were varied and robust which indicates that participants had a wealth of ideas regarding engagement in research. Another potential limitation of the current study was that all participants were considered “high-risk.” To remediate this limitation, researchers from the current study are currently analyzing qualitative data to determine potential key differences in engaging low-risk versus high-risk participants in longitudinal research.

It is important to note that although it was not specifically examined in this study, it is crucial for researchers and staff to be trained in, and understand, culturally competent methods for recruitment, data collection, and retention. This is particularly the case for building trust among researchers and participants from different cultural and high-risk backgrounds. For example, bicultural research staff and bilingual team members are needed, as well as specific training regarding cultural norms and sensitivity (see Mendez, 2010; Beasley et al., 2020).

Overall, researchers need to be aware of barriers to enrollment and study engagement strategies for recruiting and retaining high-risk participants in research. Future research should focus on understanding further behaviors and techniques in supporting high-risk participants, as engaging this population is essential for understanding developmental trajectories of risk and resilience among children starting already at risk for mental and physical health difficulties.

Funding

This work was supported by the National Institute on Drug Abuse (NIDA) research grant numbers R34DA050343 (LOB, JMC, and ASM) and R34 DA050237 (LNB, JS, and LL).

Data Availability

Not applicable

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Code Availability

Not applicable

Disclaimer

The views and opinions expressed in this manuscript are those of the authors only and do not necessarily represent the views, official policy, or position of the U.S. Department of Health and Human Services or any of its affiliated institutions or agencies.

Footnotes

Amanda Sheffield Morris and Julie M. Croff are co-senior authors.

References

  1. Alexander S, Pillay R, Smith B. A systematic review of the experiences of vulnerable people participating in research on sensitive topics. International Journal of Nursing Studies. 2018;88:85–96. doi: 10.1016/j.ijnurstu.2018.08.013. [DOI] [PubMed] [Google Scholar]
  2. Alshaarawy O, Anthony JC. Cannabis use among women of reproductive age in the United States: 2002-2017. Addictive Behaviors. 2019;99:106082. doi: 10.1016/j.addbeh.2019.106082. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bateman BT, Hernandez-Diaz S, Rathmell JP, Seeger JD, Doherty M, Fischer MA, Huybrechts KF. Patterns of opioid utilization in pregnancy in a large cohort of commercial insurance beneficiaries in the United States. Anesthesiology. 2014;120(5):1216–1224. doi: 10.1097/aln.0000000000000172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Beasley LO, King C, Esparza I, Harnden A, Robinson LR, So M, Morris A, Silovsky JF. Understanding initial and sustained engagement of Spanish-speaking Latina mothers in the Legacy for Children program™: a qualitative examination of a group-based parenting program. Early Childhood Research Quarterly. 2020;54:99–109. doi: 10.1016/j.ecresq.2020.08.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beasley LO, Ridings LE, Smith TJ, Shields JD, Silovsky JF, Beasley W, Bard D. A qualitative evaluation of engagement and attrition in a nurse home visiting program: from the participant and provider perspective. Prevention Science. 2018;19(4):528–537. doi: 10.1007/s11121-017-0846-5. [DOI] [PubMed] [Google Scholar]
  6. Betsch C. Social media targeting of health messages: a promising approach for research and practice. Human Vaccines & Immunotherapeutics. 2014;10(9):2636–2637. doi: 10.4161/hv.32234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Braun V, Clarke V. What can “thematic analysis” offer health and wellbeing researchers? International Journal of Qualitative Studies on Health and Well-Being. 2014;9:26152. doi: 10.3402/qhw.v9.26152. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Chaffin M, Valle LA, Funderburk B, Gurwitch R, Silovsky J, Bard D, McCoy C, Kees M. A motivational intervention can improve retention in PCIT for low-motivation child welfare clients. Child Maltreatment. 2009;14(4):356–368. doi: 10.1177/1077559509332263. [DOI] [PubMed] [Google Scholar]
  9. Davis JM, Yao L, Bierer BE. Protecting pregnant women with substance use disorders and their neonates participating in research. The Journal of the American Medical Association. 2019;322(7):609–610. doi: 10.1001/jama.2019.9002. [DOI] [PubMed] [Google Scholar]
  10. Dilworth-Anderson P. Introduction to the science of recruitment and retention among ethnically diverse populations. The Gerontologist. 2011;51(suppl_1):S1–S4. doi: 10.1093/geront/gnr043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dumka LE, Garza CA, Roosa MW, Stoerzinger HD. Recruitment and retention of high-risk families into a preventive parent training intervention. Journal of Primary Prevention. 1997;18(1):25–39. doi: 10.1023/A:1024626105091. [DOI] [Google Scholar]
  12. Ebrahim SH, Gfroerer J. Pregnancy-related substance use in the United States during 1996–1998. Obstetrics & Gynecology. 2003;101(2):374–379. doi: 10.1016/S0029-7844(02)02588-7. [DOI] [PubMed] [Google Scholar]
  13. Fifolt M, Lanzi RG, Johns E, Strichik T, Preskitt J. Retention and attrition in a home visiting programme: looking back and moving forward. Early Child Development and Care. 2017;187(11):1782–1794. doi: 10.1080/03004430.2016.1189420. [DOI] [Google Scholar]
  14. Frandsen M, Walters J, Ferguson SG. Exploring the viability of using online social media advertising as a recruitment method for smoking cessation clinical trials. Nicotine & Tobacco Research. 2014;16(2):247–251. doi: 10.1093/ntr/ntt157. [DOI] [PubMed] [Google Scholar]
  15. Fusar-Poli P, Schultze-Lutter F, Cappucciati M, Rutigliano G, Bonoldi I, Stahl D, et al. The dark side of the moon: meta-analytical impact of recruitment strategies on risk enrichment in the clinical high risk state for psychosis. Schizophrenia Bulletin. 2016;42(3):732–743. doi: 10.1093/schbul/sbv162. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gabrhelík, R., Skurtveit, S., Nechanská, B., Handal, M., Mahic, M., & Mravčík, V. (2020). Prenatal methamphetamine exposure and adverse neonatal outcomes: a nationwide cohort study. European Addiction Research, 1–10. 10.1159/000509048. [DOI] [PubMed]
  17. George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. American Journal of Public Health. 2014;104:e16–e31. doi: 10.2105/AJPH.2013.301706. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Haley DF, Golin C, El-Sadr W, Hughes JP, Wang J, Roman Isler M, et al. Venue-based recruitment of women at elevated risk for HIV: an HIV Prevention Trials Network study. Journal of Women's Health. 2014;23(6):541–551. doi: 10.1089/jwh.2013.4654. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Haley DF, Lucas J, Golin CE, Wang J, Hughes JP, Emel L, et al. Retention strategies and factors associated with missed visits among low income women at increased risk of HIV acquisition in the US (HPTN 064) AIDS Patient Care and STDs. 2014;28(4):206–217. doi: 10.1089/apc.2013.0366. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hallowell N, Cooke S, Crawford G, Lucassen A, Parker M, Snowdon C. An investigation of patients’ motivations for their participation in genetics-related research. Journal of Medical Ethics. 2010;36:37–45. doi: 10.1136/jme.2009.029264. [DOI] [PubMed] [Google Scholar]
  21. Haight SC, Ko JY, Tong VT, Bohm MK, Callaghan WM. Opioid use disorder documented at delivery hospitalization—United States, 1999–2014. Morbidity and Mortality Weekly Report. 2018;67(31):845. doi: 10.15585/mmwr.mm6731a1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Havens JR, Simmons LA, Shannon LM, Hansen WF. Factors associated with substance use during pregnancy: results from a national sample. Drug and Alcohol Dependence. 2009;99(1–3):89–95. doi: 10.1016/j.drugalcdep.2008.07.010. [DOI] [PubMed] [Google Scholar]
  23. Heinrichs N, Bertram H, Kuschel A, Hahlweg K. Parent recruitment and retention in a universal prevention program for child behavior and emotional problems: barriers to research and program participation. Prevention Science. 2005;6(4):275–286. doi: 10.1007/s11121-005-0006-1. [DOI] [PubMed] [Google Scholar]
  24. Institute of Medicine (US). (1994). Women and health research: ethical and legal issues of including women in clinical studies: Volume 1. From https://www.ncbi.nlm.nih.gov/books/NBK236535/. Accessed 12 Aug 2020.
  25. Institute of Medicine (US). (1999a). Women and health research: ethical and legal issues of including women in clinical studies: Volume 2. From https://www.ncbi.nlm.nih.gov/books/NBK236577/. Accessed 12 Aug 2020.
  26. Institute of Medicine (US). (1999b). Women and health research: ethical and legal issues of including women in clinical studies: Volume 2. From https://www.ncbi.nlm.nih.gov/books/NBK236564/. Accessed 12 Aug 2020.
  27. Johnson, P.A., Fitzgerald, T., Salganicoff, A., Wood, S.F., Goldstein, J.M., Colson, Y.L., Cohen, L., Ranji, U., Camp, A., Luk, C., Mauery, D.R., & Rich-Edwards, J. (2014). Sex specific medical research: why women’s health can’t wait. A Report of the Mary Horrigan Connors Center for Women’s Health & Gender Biology at Brigham and Women’s Hospital. From https://www.brighamandwomens.org/assets/bwh/womens-health/pdfs/connorsreportfinal.pdf. Accessed 12 Aug 2020.
  28. Jones KL, Smith DW. Recognition of the fetal alcohol syndrome in early infancy. Lancet. 1973;302:999–1001. doi: 10.1016/S0140-6736(73)91092-1. [DOI] [PubMed] [Google Scholar]
  29. Kim R, Hickman N, Gali K, Orozco N, Prochaska JJ. Maximizing retention with high risk participants in a clinical trial. American Journal of Health Promotion. 2014;28(4):268–274. doi: 10.4278/ajhp/120720-QUAN-355. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lombard M, Snyder-Duch J, Bracken CC. Content analysis in mass communication: assessment and reporting of intercoder reliability. Human Communication Research. 2006;28(4):587–604. doi: 10.1111/j.1468-2958.2002.tb00826.x. [DOI] [Google Scholar]
  31. Maloni JA, Przeworski A, Damato EG. Web recruitment and internet use and preferences reported by women with postpartum depression after pregnancy complications. Archives of Psychiatric Nursing. 2013;27(2):90–95. doi: 10.1016/j.apnu.2012.12.001. [DOI] [PubMed] [Google Scholar]
  32. Mendez JL. How can parents get involved in preschool? Barriers and engagement in education by ethnic minority parents of children attending Head Start. Cultural Diversity and Ethnic Minority Psychology. 2010;16(1):26–36. doi: 10.1037/a0016258. [DOI] [PubMed] [Google Scholar]
  33. McHugh RK, Votaw VR, Sugarman DE, Greenfield SF. Sex and gender differences in substance use disorders. Clinical Psychology Review. 2018;66:12–23. doi: 10.1016/j.cpr.2017.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Newington L, Metcalfe A. Factors influencing recruitment to research: qualitative study of the experiences and perceptions of research teams. BMC Medical Research Methodology. 2014;14(1):10. doi: 10.1186/1471-2288-14-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Nicholson LM, Schwirian PM, Klein EG, Skybo T, Murray-Johnson L, Eneli I, Boettner B, French GM, Groner JA. Recruitment and retention strategies in longitudinal clinical studies with low-income populations. Contemporary Clinical Trials. 2011;32(3):353–362. doi: 10.1016/j.cct.2011.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. O’Brien RA, Moritz P, Luckey DW, McClatchey MW, Ingoldsby EM, Olds DL. Mixed methods analysis of participant attrition in the nurse-family partnership. Prevention Science. 2012;13(3):219–228. doi: 10.1007/s11121-012-0287-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Owen-Smith AA, Woodyatt C, Sineath C, Hunkeler EM, Barnwell L, Graham A, Stephenson R, Goodman M. Perceptions of barriers to and facilitators of participation in health research among transgender people. Transgender Health. 2016;1(1):187–196. doi: 10.1089/trgh.2016.0023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Phinney R. Exploring residential mobility among low-income families. Social Service Review. 2013;87(4):780–815. doi: 10.1086/673963. [DOI] [Google Scholar]
  39. Post, S.D. (2005). Altruism, happiness, and health: it’s good to be good. International Journal of Behavioral Medicine, 12, 66–77. 10.1207/s15327558ijbm1202_4. [DOI] [PubMed]
  40. Sikeweyiya Y, Jewkes R. Potential motivations for and perceived risks in research participation: ethics in health research. Qualitative Health Research. 2013;23(7):999–1009. doi: 10.1177/1049732313490076. [DOI] [PubMed] [Google Scholar]
  41. Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research and Evaluation. 10.7275/96jp-xz07.
  42. Substance Abuse and Mental Health Services Administration, 2018. Key substance use and mental health indicators in the United States: results from the 2017 National Survey on Drug Use and Health (HHS Publication No. SMA 18-5068, NSDUH Series H-53). Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration. From https://www.samhsa.gov/data/. Accessed 12 Aug 2020.
  43. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care. 2007;19(6):349–357. doi: 10.1093/intghc/mzm042. [DOI] [PubMed] [Google Scholar]
  44. Wetherington CL, Roman AB. Drug addiction research and the health of women. DHHS publication no. 98-4290. Rockville: US Department of Health and Human Services, National Institutes of Health, National Institute on Drug Abuse; 1998. [Google Scholar]
  45. Whitaker C, Stevelink S, Fear N. The use of Facebook in recruiting participants for health research purposes: a systematic review. Journal of Medical Internet Research. 2017;19(8):e290. doi: 10.2196/jmir.7071. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Wlömert N, Papies D. On-demand streaming services and music industry revenues—insights from Spotify’s market entry. International Journal of Research in Marketing. 2016;33(2):314–327. doi: 10.1016/j.ijresmar.2015.11.002. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Not applicable

Articles from Adversity and Resilience Science are provided here courtesy of Nature Publishing Group

RESOURCES