Maximizing Retention with High Risk Participants in a Clinical Trial

Abstract

Purpose

To describe effective retention strategies in a clinical trial with a high risk, low income, and vulnerable patient population with serious mental illness.

Design

Follow-up assessments were conducted for a randomized clinical tobacco treatment trial at 3-, 6-, and 12-months post-baseline. Initial follow-up rates of <40% at 3-months led to implementation of proactive retention strategies including obtaining extensive contact information; building relationships with case managers and social workers; contacting jails and prisons; text messaging, e-mailing, and messaging via social networking sites; identifying appointments via electronic medical record; and field outreach to treatment facilities, residences, and parks.

Setting

Large urban public hospital

Subjects

Participants were current smokers recruited from 100% smoke-free locked psychiatry units.

Measures

Assessments covered demographics, substance use, and mental health functioning.

Analysis

Retention rates were plotted over time in relation to key retention strategies. Chi-square and t-tests were used to examine participant predictors of retention at each follow-up. At the 12-month follow-up, the retention strategies that most frequently led to assessment completion were identified.

Results

The sample (N=100) was 65% male; age M=39.5 years (SD=11.3); 44% non-Hispanic White; 46% on Medicaid and 34% uninsured; 79% unemployed; and 48% unstably housed. Proactive retention strategies dramatically increased follow-up rates, concluding at 3-months=82.65%, 6-months=89.69%, and 12-months=92.78%. Married and divorced/separated/widowed participants, those with higher income, and participants with alcohol or illicit drug problems had increased retention from 3 to 12-months follow-up.

Conclusion

Follow-up rates improved as proactive methods to contact participants were implemented. Dedicated research staff, multiple methods, community networking, and outreach within drug treatment settings improved retention.

Purpose

Successfully engaging and retaining participants in research is essential to internal and external validity. The Centers for Disease Control and Prevention’s Efficacy Criteria for Best-Evidence in Prevention Research Synthesis sets a minimum of 70% retention at a single follow-up for each treatment arm.¹ The inclusion of diverse populations in research is important, both to ensure an equitable share of the benefits of research and to reduce disparities in the burden of disease.² As the U.S. population becomes increasingly diverse, the need to represent various racial/ethnic and socioeconomic backgrounds and other underrepresented groups in health-related research is imperative to informing national policies.^3–4 Yet, there is a general lack of research on effective retention strategies with low-income and ethnically diverse populations, especially those with mental illness.

Conducting research with low income and ethnic minority participants presents a unique set of challenges including “fear and distrust of the research enterprise, lack of knowledge, lack of transportation, interference with work and/or family responsibilities, subject burden as a result of participation in clinical study, and financial costs.”⁵ Working with study participants with mental illness adds even more challenges including lack of primary or secondary forms of contact, unstable housing, high mobility, unreliable phone access, custody/legal issues, and history of treatment and medication non-compliance.^6–7 To successfully recruit and retain diverse and vulnerable populations, Dilworth-Anderson emphasized providing resources such as transportation, well-trained staff, convenient meeting places for interviews, community outreach to enable effective communication, and continuous contact with participants.⁸

Studies examining retention methods have yielded positive results; however, many of the strategies were specific to a particular racial group and may not generalize to others. El-Khorazaty et al., for example, achieved a 79% retention rate in a study with pregnant African American and Latina women by using financial incentives and continuously updating contact information. Incentives were $5 for completing the screening, a 30-minute pre-paid phone card for consenting to participate, $15 for each phone interview, $10 for each intervention session, and additional $15 and $25 gift certificates for 2 post-partum intervention sessions.¹ Retention strategies in the Familias Fuertes-Georgia program, which aimed to reduce high-risk behaviors among Latino youth, included access to a community liaison, bilingual and biculturally trained facilitators, financial and meal incentives ($20 per family, per session), and free childcare for participating families.⁹ DeCoux Hampton et al. focused on the recruitment and retention of African Americans with severe mental illness and achieved a 71% retention rate (defined as completing at least 1 follow-up) using contact information provided at baseline and through site visits by research assistants to treatment programs and homeless shelters.³

The current study aimed to contribute to the literature on clinical research with underrepresented, high-risk, and vulnerable populations with severe mental illness. Herein detailed are initial struggles with attrition in a randomized clinical tobacco treatment trial that led to the adoption of proactive and multi-method outreach strategies to maximize retention over 12-months of follow-up.

Methods

Design

The study was conducted from April 2009–April 2011 and received Institutional Review Board approval for the research procedures. The study, a randomized controlled tobacco treatment trial with a 2-group design, recruited participants in-hospital and followed them over 12-months post-hospitalization (Figure 1). The intervention aimed to address significant disparities in tobacco use among smokers with mental illness.¹⁰ The intervention was tailored to readiness to quit smoking and included a stage-specific computer program, counseling, and nicotine replacement therapy (NRT). Intention to quit smoking was not required to participate in the study. The control condition was usual care characterized by NRT provided during hospitalization only and brief advice to quit. Follow-up time points included assessments at months 3, 6, and 12 post-baseline, ideally in-person but possible by phone, mail, or email. Collateral reports of smoking status (i.e., by participant-identified sources of contact) were also permitted if study participants were unreachable.

Figure 1.

Consort Diagram

Demand characteristics of the trial were low, as the intervention was tailored to readiness to quit (i.e., quitting was not a set expectation) and there exists greater acceptance of tobacco use among psychiatric populations. Hence, there was little incentive for participants and collaterals to falsify reporting of smoking status. At the baseline assessment, collaterals were selected based on familiarity with the participant and frequency of contact. A randomized tobacco treatment trial by Patten and colleagues with 256 smokers with alcohol use disorders found that collateral reports had high concordance (92%) with measured breath carbon monoxide levels.¹¹

Sample Recruitment

Adult smokers were recruited during a 100% smoke-free acute psychiatric hospitalization from three locked units at San Francisco General Hospital (SFGH). A large urban public hospital, SFGH provided 60 of the 81 adult inpatient psychiatric beds in San Francisco, CA with over 2200 admissions per year. Based on 2010–2011 data, hospitalized psychiatric patients at SFGH were 51% men; 68% aged 25–64; and 30% Hispanic, 25% non-Hispanic Caucasian, 23% Asian American/Pacific Islander, and 17% African American.¹²

To participate, patients had to be current smokers of ≥ 5 cigarettes per day, 18 years or older, fluent in English, and have demonstrated capacity to consent to research (i.e., able to adequately identify the study purpose, risks, and benefits).¹³ Study staff performed chart reviews to check for eligibility; requested clinical staff to introduce them to eligible patients; and met with patients to describe the study, review the consent form, and screen patients for capacity to consent. If all approved, study staff obtained informed participant consent. Individuals were excluded if they had a contraindication for NRT use (e.g., pregnancy, recent myocardial infarction, liver/kidney disease) or planned to move out of the area in the next 12 months. Patients who were acutely psychotic, hostile, aggressive, assaultive or sedated were reassessed for eligibility at a later time. The recruitment rate was 71%. The informed consent process included information on participants’ rights in research and their ability to drop out of the study if so desired at any time.

Retention Strategies

Initial retention strategies were collection of detailed contact information and study incentives for participant’s time: $20 per in-person interview ($10 if completed by phone/mail/email) plus a $20 bonus for completing all assessments. The 3-page contact information sheet included the participant’s mailing address, phone number, e-mail address; 3 secondary contacts; specific information for locating homeless participants (e.g. sleep location when it rains, place where letters can be sent, place to leave phone messages); and contact information for clinicians, payees, and parole officers. Initial follow-up rates of <40% at 3 months spurred the need for more proactive efforts for retaining participants.

More rigorous, IRB-approved, multi-method retention strategies were implemented at different time points of the study and used throughout the study once initiated. In June 2009, the team obtained approval to access the county’s general electronic medical record (EMR) and in October 2009 access to the EMR for the hospital’s methadone clinic. Research staff used the EMRs to identify participants’ ambulatory care appointments to meet them to complete study follow-ups. The study contact form was expanded to 6-pages (available upon request from the corresponding author) to collect additional information on participants’ use of shelters, needle exchange programs, and free meal programs in the Bay Area; additional information on current living situation (e.g. residential treatment programs or sleeping and hangout locations for the homeless); and the location of frequented places. With participants’ consent, their pictures were taken for identification purposes, and starting in late October 2009, two staff conducted field visits together at local parks, coffee shops, treatment centers, the library, and single residency occupancy hotels (SROs) to locate participants lost to follow-up. A study cell phone was acquired to contact participants while out of the office and to send text messages. In December 2009, study staff began contacting local jails and state prisons to find incarcerated participants and confirm smoking policies in those facilities. Towards the end of the study, social networking websites (Facebook and MySpace) were utilized to contact participants via private messages. Participant contact information and all outreach activities were recorded in the study’s ACT! database, a client relationship management software program useful for alerting staff to pending and overdue appointments and organizing contact information.

In terms of committed personnel time, the clinical trial staffed two part-time research assistants contributing a total of 60 hours a week over a 104 week study period, who were supervised by a post-doctoral fellow and faculty member. Figure 2 summarizes the relative resource demand for each type of retention strategy used. Resource demand ranged from a few minutes (e.g, leaving a voicemail or sending an email), to less than an hour (e.g., communicating with several people to locate a participant), to several hours (e.g., going out into the field to meet a participant at a clinic).

Figure 2.

Staff time and effort demand by outreach strategy

Low Demand (few minutes): Direct outreach to participant via phone or internet

Moderate Demand (less than an hour): Involvement of one or multiple intermediaries

High Demand (several hours): Field work and research

*EMR: Electronic Medical Record

Study Measures

Study measures to describe the sample included demographic characteristics (ethnicity/race, age, gender, insurance status, living situation, employment, and marital status), smoking behaviors (cigarettes/day, years of smoking, Fagerstrom Test of Nicotine Dependence¹⁴ stage of change for quitting¹⁵), alcohol and illicit drug use (past 30 day use items from the Addiction Severity Index (ASI)¹⁶, Alcohol Use Disorders Identification Test (AUDIT)¹⁷, Drug Abuse Screening Test (DAST)¹⁸). Depressive symptoms were assessed with the Center for Epidemiologic Studies Depression Scale (CESD-10)¹⁹. Overall mental and physical health functioning were assessed by recording reason for hospitalization, length of stay, and using the Health Status Survey Short Form (SF-12), which is computed into Physical and Mental Health Composite Scores (PCS & MCS)²⁰. SF-12 scores range from 0–100, lower scores are an indicator of poorer health. Study retention was recorded at each follow-up time point as completion or not of study measures (at least smoking status) reported by the study participant or identified collateral.

Statistical Analysis

Analyses were conducted using SPSS version 20. Descriptive statistics (means, frequencies) were run to describe the sample with regard to demographic variables, tobacco and other substance use, and psychiatric characteristics. Retention rates were plotted over time in relation to key retention strategies. Next, chi-square and t-tests were run to examine participant predictors of completed follow-ups at each time point. Lastly, retention strategies that led to completed follow-ups at the 12-month assessment were identified and frequency counts were entered into Wordle.net, a website that generates a customized word cloud using font size as a demonstration of frequency. Visual displays of verbal information are useful for demonstrating prominent terms and for understanding associations and hierarchical relationships.²¹

Results

Descriptive Statistics

The sample (N=100) was 65% male; 20% identified as LGBT; mean age was 39.5 (SD=11.3); 15% were foreign born; race/ethnic identity was 44% non-Hispanic White, 27% African American, 11% Asian, 9% Latino, and 9% multi-racial or other ethnic group; 46% had Medi-Cal and 34% were uninsured; 48% reported an unstable living situation (i.e., living in their current situation for < 6 months). Of the full sample, 19% were homeless, 28% lived in an SRO, 14% were in a residential treatment facility or jail, 25% lived in the residence of a friend or family, and 10% rented or owned their own apartment or house. Completed education averaged 13 years (SD=2.5). Employment status was 79% unemployed, 12% employed, and 9% retired/student. Annual income levels were 67% < $10,000, 15% $10,000–$19,999, and 18% >$20,000. Marital status was 62% never married, 28% divorced/separated/widowed, and 10% married. Over the course of the study, 16 participants were incarcerated. Figure 1 summarizes follow-up rates, which did not differ by condition at any of the time points, p-values > 0.160.

The sample smoked an average of 19.3 cigarettes per day (SD=12.2), was moderately dependent on nicotine based on an average FTND score of 5.3 (SD=2.3), and had smoked cigarettes for an average of 23 years (SD=13). In terms of stage of change for quitting smoking, 33% were in precontemplation (not intending to quit), 42% were in contemplation (intended to quit within the next six months), and 25% were in preparation (intended to quit within 30 days).

In the 30 days prior to hospitalization, 77% of the sample used illicit drugs including marijuana (49%), crack (42%), methamphetamines (32%) and heroin (8%); 67% consumed alcohol; and 55% used both illicit drugs and alcohol. A majority (60%) engaged in problematic illicit drug use (i.e., DAST score >=3), while 25% were at risk for alcohol dependence (i.e., Audit score >=15 for men and >=13 for women).

Reason for psychiatric hospitalization was danger to self (73%), danger to others (18%) and grave disability (9%). Hospitalization duration was a median of 8 days (IQR= 5–13 days). Diagnoses in patients’ medical records were unipolar depression (54%), bipolar disorder (14%), and psychotic disorder (46%). The average CESD-10 total score was 16.0 (SD=8.1) with 51% scoring >= 16 indicating depression.¹⁸ The average MCS and PCS scores were 33.5 (SD=14.7) and 46.4 (SD=11.9), respectively, indicating poor mental health and below average physical health.

Change in Follow-up Rates over Time

Using all retention strategies listed, follow-up rates increased over the course of the trial from the initial rate of < 40% in May–July 2009 and concluded in April 2011 as follows: 3 Month = 82.65%, 6 Month = 89.69%, and 12 Month = 92.78% (Figure 3). A dip in follow-up rates occurred during December 2009–January 2010 as a result of low staffing with the winter holidays. In December 2010–January 2011, adequate staffing was scheduled to avert a similar drop in retention. By December 2010, it also helped that the study was not behind with assessments and that the bulk of 3 and 6 month follow-ups were completed so the primary focus was on a single epoch, the 12-month follow-up. The number of completed assessments that were collateral reports were 6, 6, and 9 at the 3-, 6-, and 12-month follow-up time points, respectively (8% of completed follow-ups).

Figure 3.

Follow-up rates at 3-, 6-, and 12-months and implementation of retention strategies throughout study period. Vertical lines represent implementation of new strategies: 1. Methadone Clinic EMR, 2. County EMR, 3. Field Outreach, 4. Updated Contact Form, 5. Calling Jails/Prisons, 6. Facebook, 7. MySpace

For the full sample, 73 participants completed all three follow-up assessments; 15 completed only two, 9 completed only one, and 3 did not complete a single follow-up. Two participants withdrew from the study stating no longer wanting to participate and believing she/he did not smoke enough to need to participate any longer; both completed at least one follow-up. Three participants died after enrollment in the study; one completed one follow-up, the other two died prior to the 3-month follow-up. The deaths were deemed unrelated to the study.

Participant Predictors of Follow-up

Study retention varied by several participant characteristics. Figure 4 shows the improvement in follow-up rates from 3- to 12-months by participants’ report of alcohol or illicit drug problems, marital status, and income level. At the 3-month follow-up, retention was lower for participants with alcohol problems on the AUDIT (X2=3.66, df=1, p=.056 relative to participants without alcohol problems); for divorced/separated/widowed or married participants (X2=9.49, df=2, p=.009 relative to never married participants); and for participants with incomes ≥ $10,000, (X2(1)=4.09, p=.043 relative to incomes < $10,000). By the 12-month follow-up, retention rates improved for participants with alcohol problems, those who were separated/divorced/widowed or married, and those of higher income such that group comparisons on these factors were no longer significant. No other demographic, psychiatric, illicit drug, or tobacco use or motivation measure predicted retention at 3-months follow-up and no participant factors predicted retention at the 6-month follow-up. The only measured predictor of retention at the 12-month follow-up was illicit drug problems as identified on the DAST-10 at baseline. Improvements in follow-up over time resulted in better retention at 12-months for participants with drug problems compared to those without, X2=4.17, df=1, p=.041.

Figure 4.

Participant characteristics predictive of study retention. The darker bar shows the improvement in retention rate from 3- to 12-months follow-up.

Effective Follow-up Strategies

Retention strategies that resulted in completed assessments at the 12-month follow-up are displayed in Figure 5, wherein larger font size indicates greater frequency of a strategy resulting in follow-up completion. Participants were most likely to be reached by phone, at outpatient medical/drug treatment facilities, during a hospital readmission, or through their case managers.

Figure 5.

Frequency of strategies for maximizing retention at 12-month follow-up.

NOTE: EMR: Electronic Medical Record; Residence: Board & Care and Hotels; Electronic Message includes contact via social networking sites and e-mail; and Field Visit includes visits to SROs, other residences, clinic appointments, and treatment facilities

Discussion

Participant attrition is a challenge in longitudinal trials. The current investigation, with smokers with serious mental illness and substance use disorders over a 12-month follow-up period, illustrated the need for research retention efforts that were pro-active, multi-pronged and flexible to ensure adequate representation.

Mental illness, substance use, and poverty each bring unique complexities that can adversely impact study participation over time. The study participants were challenged by lack of insurance and changes in clinical care, instability in housing, legal problems, limited computer access or contact with family and friends, and frequent changes in phone numbers. Early in the trial, faced with a > 60% attrition rate, it was readily apparent that greater efforts would need to be dedicated to study retention for the research findings to be meaningful.

Various retention strategies were implemented to meet the needs of the diverse patient population, and with dedicated staff, the study ultimately achieved 93% follow-up at12-months. By regularly updating the study contact sheet, utilizing the county’s EMR to identify participants’ ambulatory care appointments and methadone clinic dosage times, and going out into the field, the ability to track and effectively reach participants significantly improved over time. Participants’ cellular phones and their increasing use of social media also supported study retention efforts. Despite being unstably housed, many participants had free computer access at public libraries. Importantly, the study’s level of financial incentive for study participation was not coercive; rather it was consistent with participants’ time investment and travel costs. Collateral reports by family, friends, and clinicians provided a back up when participants were unreachable, yet overall represented < 10% of completed study follow-up contacts. In the literature, studies with other high-risk populations have reported success with retaining participants via personal contacts with family and friends; outreach to shelters, drug treatment programs, and jails; and visits to community gathering places.²² Recent reports also suggest social media networks, such as Facebook, have good potential for reaching participants over time.²³ More broadly, a continuously dynamic process of monitoring trial progress and tailoring strategies to particular circumstances, while not compromising trial protocols, has been recommended for maximizing retention.²⁴

Attrition at 3-months follow-up was found to vary by participant factors including marital status, income level, and alcohol problems. By the 6- and 12-month follow-ups, attrition was reduced to ≤ 10%. Improvements in retention were particularly notable among married participants, those with incomes ≥$10,000, and those with illicit drug problems at baseline. While the study financial incentive may have been sufficient motivation for low-income individuals to complete assessments, those with higher incomes may have had more competing demands on their time due to work or other responsibilities. Participants who were never married may have responded well to initial follow-ups because of a possible need for a support system. Greater outreach efforts, however, were required to retain participants who were married and those with higher income. For participants with drug problems, the 95% follow-up rate at 12-months was largely achieved through staff visits to the methadone clinic and other treatment facilities.

Only two individuals withdrew from the study, and none of the participants voiced concerns with the study’s retention efforts. Though not formally recorded, participants reached out in the community or visited at ambulatory care appointments expressed appreciation, familiarity, and kindness, not irritation, confusion, or suspicion. The participants were pleased that study staff had taken the extra efforts to locate them, meeting them in their environment, rather than requiring them to travel to complete study activities.

Limitations

The trial’s primary aim was to evaluate treatment of tobacco dependence among patients with mental illness. The amount of time spent on outreach was not recorded, and calculation of a cost analysis is not possible. The study sample was 100 participants recruited from a single urban site. It may be more difficult to implement these retention strategies with a larger sample and in less densely populated areas. Nevertheless, the study methods utilized and results achieved are relevant to informing research with other vulnerable populations.

SO WHAT? Implications for Health Promotion Practitioners and Researchers.

What is already known on this topic?

Successfully conducting research studies with high-risk, vulnerable populations is challenging. Creative and effective retention strategies are needed to meet the needs of individuals in underserved communities and to successfully complete research studies.

What does this article add?

The article contributes to the literature by detailing effective strategies that maximized retention over a 12-month study period with low-income, ethnically diverse participants with severe mental illness.

What are the implications for health promotion practice and research?

The results of this study show that, with pro-active, multi-pronged and flexible research retention protocols and staff, it is possible to successfully recruit and retain underserved, high-risk, and vulnerable participants with mental illness in research studies.