Recruitment Outcomes, Challenges and Lessons Learned: The Healthy Communities Study

Abstract

Background:

The Healthy Communities Study (HCS) was a national study of community programs and policies (CPPs) that aimed to address childhood obesity; it necessitated recruitment of a large sample of children from communities throughout the United States.

Objective:

The HCS aimed to complete visits with an average of 45 children and 12 Key Informants (KIs) from at least 120 communities, diverse with respect to region of the country, urbanicity, socioeconomic status, race, ethnicity, and intensity of CPPs that aim to address childhood obesity.

Methods:

Purchased address lists were utilized to select households for recruitment during Wave 1 of the study and recruitment of families through schools was employed for Wave 2.

Results:

The HCS successfully obtained approval from 149 school districts and 478 schools in 130 communities, recruited 5,138 families, and interviewed 1,421 KIs to allow for characterization of overall intensity of obesity prevention/treatment efforts in each community.

Conclusions:

Lessons learned are presented. Future studies should plan for inclusion of the following in development of recruitment strategies: literature review, formative research, pilot testing, and ongoing monitoring and adjustment.

SUMMARY

What is already known about this subject:

Few publications have reported recruitment strategies, challenges, and solutions for national studies of school-aged children in the United States.

Conducting nationwide field data collection involving schools, families, and community stakeholders is challenging and requires a creative and flexible approach.

What this study adds:

This study highlights recruitment methods that were both unsuccessful (e.g., utilization of purchased address lists, solicitation of school representatives to lead recruitment efforts) and successful (e.g., engagement of staff in communities).

This study demonstrates the need for a combined approach to development of recruitment strategies including: review of the literature, formative research and pilot testing specific to the included populations/communities, and ongoing monitoring and adjustment of recruitment approaches during data collection.

Introduction

Childhood obesity is a widespread public health problem in the United States. Between 2011 and 2014, over 17% of children ages six through 19 years had obesity ¹. Children who are overweight in childhood are more likely to be overweight as adults and thus more likely to develop diabetes, heart disease, and certain cancers ².

While several studies have reviewed the impact of programs and policies targeting obesity in children ^3–5, the impact of combinations of programs and policies within communities is not well understood. The Healthy Communities Study (HCS) was designed to assess the relationship between characteristics of community programs and policies (CPPs) and child physical activity, dietary behaviors, and body mass index (BMI) ⁶.

Few publications have reported recruitment strategies, challenges, and solutions for national studies of school-aged children in the United States. Lessons learned to date include: formative research to tailor recruitment strategies; multiple recruitment strategies; fostering relationships and trust; pilot testing; individuals dedicated to championing the project; ideal construction of consent and incentive procedures; imposing as little burden as possible; utilization of staff with similar demographic characteristics to the community members; and cultural competency ^7,8. The HEALTHY Study (a seven-site school-based study of type 2 diabetes risk factors) emphasized the importance of getting support from the district and school administration, identification of motivating factors, extensive outreach, and tailoring recruitment efforts ⁹. The current paper provides information on a much larger recruitment process. The recruitment protocol for HCS was developed based on experiences of the investigators. The results are being presented to contribute to development of evidence-based recruitment strategies for future studies aiming to enroll school-aged children from communities across the United States into population-based obesity research.

To assess the relationships between CPPs, child physical activity, child dietary behavior, and BMI, the HCS collected data from community stakeholders (key informants, KIs), schools, households, and children’s medical records. Recruitment required creative strategies to maximize response and participation. The goal of this study was to recruit 45 children and 12 KIs from at least 120 communities, diverse with respect to region of the country, urbanicity, socioeconomic status, race, ethnicity, and intensity CPPs. The purpose of this paper is to discuss the outcomes, challenges, and lessons learned related to recruitment of school districts, schools, households, and KIs.

Methods

The HCS involved two waves of data collection. Wave 1 aimed to recruit 70 families with children in each of four selected communities using purchased address lists. Advance letters were followed by a telephone call to determine eligibility; 7,987 households were contacted and 263 agreed to participate. Seventy percent of the households contacted were deemed ineligible. Challenges with the address list approach included: (1) no children in the target age group in the home; (2) home phone numbers only; (3) outdated information; (4) invalid addresses/outside of catchment area; and (5) individuals not recalling receiving a mailing when the call center reached them. Having only home phone numbers is problematic due to increasing numbers of families with mobile phones only and associated bias^10,11,12. Because of these challenges, the study shifted to school recruitment for Wave 2. This rest of this paper focuses on Wave 2 of the HCS.

Recruitment Protocol

The Battelle Memorial Institute’s Institutional Review Board provided oversight for the study. IRB approval was received in 2011 with annual reviews. All data collection forms received approval from the U.S. Office of Management and Budget (#0925–0649).

The community selection process for the study has been published in detail elsewhere ¹³. Communities were selected using a combination of a national probability-based sample and communities identified as having promising programs and policies to address childhood obesity. The probability-based sample selection was structured to promote diversity across specific community characteristics including race/ethnicity, urbanicity, region of the country, income, and a rating to indicate the hypothesized intensity of programs and policies in the community. Some communities were selected specifically because they had a 30% or greater African American or Hispanic population ¹³. A description of the community categorizations can be found in an Appendix table published by Strauss, et. al., 2015 ¹³.

School districts in each of the selected communities were recruited first. For a description of the operational aspects of the study, please see John et al 2015 ¹⁴. The district recruitment approach included an advance letter followed by phone and e-mail contacts. District clearance packages were prepared for the school district IRB or research approval office.

After obtaining district approval, school recruitment began. The study aimed to recruit four schools in each community with elementary and middle school-level representation. Schools were sampled with probability according to size based on the total number of students that represented the race/ethnicity of the community stratum (e.g., number of African American, Hispanic or Latino, or total number of students). The sampling of schools was done separately for elementary (K-5) and middle school (7–8) grades. Sixth graders were not counted in the measure of size as their placement in elementary or middle schools is not consistent across school districts. The recruitment team began contacting the top two elementary and middle schools on the list by sending the principal an advance letter, followed by phone and e-mail contacts. Schools that agreed to participate designated a staff member within the school as the liaison (School Liaison - SL) for the study.

SLs were provided with toolkits including language to be included in school newsletters and publications, social media, twitter messages, posters, scripts for automated voicemails, announcements, etc. to advertise the study. The SL was also responsible for distributing Parent Interest Forms (PIFs) to the students (to collect household contact information and identify interested families), collecting completed PIFs, and returning PIFs to the study team.

By agreeing to participate in the study, schools were also consenting to allow the study staff to visit for additional data collection activities. Incentives were offered to the schools ($150) and SLs ($50) as a token of appreciation.

Household recruitment began when the minimum number of schools were successfully recruited in a community. Recruitment goals were stratified by grade and gender and, in African American and Hispanic communities, by race/ethnicity. Household eligibility criteria included: a non-institutionalized, ambulatory child in kindergarten through eighth grade attending a participating school, ability to complete the visit in English or Spanish, child residency in the community for ≥ one year, and only one child per household.

Once PIF data entry and cleaning were completed and Field Data Collectors (FDCs) were hired and trained in the community, the call center began calling families to screen, recruit, and schedule a study visit. The call center software probabilistically selected the order of households to call based on the age and gender of children in the household, the age and gender recruitment goals not yet met in the community, and the pool of potential participants remaining. Families were eligible for incentives worth up to $30 or $80 depending on their participation.

The objective of the KI interviews was to catalogue community nutrition and physical activity policies and programs (CPPs) available in the community for children. A structure of priority and alternate targets by sector was developed and is described in detail elsewhere ¹⁵. The study aimed to interview 10 to 14 stakeholders across at least three sectors, identify at least 40 CPPs, and have a minimum of 80% of CPPs with data sufficient to characterize per community.

All recruitment staff were trained on study recruitment protocols. School recruitment was performed by 10 field staff with past school-based experience in recruiting, prompting, and data collection. SLs – staff members appointed by the principal – led initial recruitment efforts. FDCs hired from the communities and overseen by Field Supervisors were sent into schools that requested assistance with recruitment. Telephone screening calls were conducted by experienced telephone center interviewers supervised by telephone center supervisors. In some cases, FDCs assisted with recruitment with phone calls/home visits. Certified CLs (study staff) recruited Key Informants.

Results

The following sections describe the results of recruitment of districts, schools, and households (also displayed visually in the flow diagram in Figure 1). Community KI Recruitment results are described in Collie-Akers, et. al. ¹⁶.

Figure 1.

Flow Diagram of HCS District, School, and Household Recruitment

District Recruitment

Final school district recruitment results are presented in Table S1. The number of districts did not match the number of communities as some communities spanned multiple districts and some large school districts contained more than one community. The study team received approval from 22.64% of the districts approached across 146 communities, while 55.78% refused, 4.40% were successfully recruited but subsequently eliminated, 16.72% were pending a research clearance decision upon conclusion of district recruitment efforts or were not needed (as an alternate district/community was recruited), and 0.46% were deemed out of scope.

The average number of days to a decision was 77 (72 days for refusals and 102 for approvals) with an average of 15 contact attempts required to obtain approval. These averages are affected by several factors including: pre-determined schedules for research review that could last several months; refusal soon after initial contact; and, eventual placement of limits on how long districts were given to respond.

The demographic distribution of district approvals is presented in Table S2. The highest number of approvals was in the South (37.3%), in urban (41.4%), and in moderate-high income (62.1%) communities and the fewest approvals were in the Northeast (20.1%), in rural (21.3%), and in low income (37.9%) areas. District approvals were evenly distributed across race/ethnicity with 29.6%, 32.0%, and 38.5% from African American, Hispanic, and “Other” communities, respectively.

School Recruitment

Final school recruitment results are presented in Table S3. Out of 855 schools contacted, 55.9% agreed to participate, 26.0% refused, 37.1% closed or did not have appropriate grades for the study, and 11.5% were determined not to be needed by the study when higher-ranked schools within the community agreed to participate. The average number of days to a decision was 62 days for refusals and 52 for approvals, with an average of 13 contact attempts required to obtain approval.

Household Recruitment

Table 1 presents the household recruitment results by community characteristics including race/ethnicity, urbanicity, region of the country, pre-selection intensity rating, and income. Of the 15,047 unique households in the sample generated from the PIFs collected across the 130 communities, 50.14% agreed to participate, 4.41% refused during screening, 2.69% were ineligible, 12.49% reached maximum call attempts, 7.20% had a bad phone number, 16.30% were not available during the study period, 1.83% were not needed due to cells being filled, and 4.94% did not finalize screening. A total of 5,138 households completed visits, representing 34.04% of the total PIFs and 67.88% of those that agreed to participate.

Table 1.

Household Recruitment by Community Characteristics

		Agreed to participate			Completed Household Visits					Refused at screening
	# HH	#	% (of total HH)	p- value*	#	% (of total HH)	p- value*	% (of agreed to participate)	p- value*	#	% (of total HH)	p- value*
Total**	15047	7563	50.26		5138	34.15		67.94		666	4.43
Race/ethnicity
Hispanic	5714	2877	50.35	0.06	2045	35.79	<0.01	71.08	<0.01	244	4.27	0.52
African American	3222	1564	48.54		1059	32.87		67.71		154	4.78
“Older”	6111	3122	51.09		2034	33.28		65.15		268	4.39
Urbanicity
Urban	5920	2962	50.03	0.72	1942	32.8	0.02	65.56	<0.01	239	4.04	<0.01
Suburban	5829	2954	50.68		2034	34.89		68.86		246	4.22
Rural	3298	1647	49.94		1162	35.23		70.55		181	5.49
Region of Country
South	6108	3018	49.41	0.30	2135	34.95	0.08	70.74	<0.01	279	4.57	0.86
West	3653	1844	50.48		1221	33.42		66.21		160	4.38
Midwest	3037	1541	50.74		991	32.63		64.31		134	4.41
Northeast	2249	1160	51.58		791	35.17		68.19		93	4.14
Pre-selection intensity rating
High	4740	2287	48.25	<0.01	1549	32.68	<0.01	67.73	0.03	216	4.56	0.82
Moderate	4528	2347	51.83		1641	36.24		69.92		194	4.28
Low/ None	5779	2929	50.68		1948	33.71		66.51		256	4.43
Income
Moderate/ High	9760	4897	50.17	0.77	3349	34.31	0.56	68.39	0.26	442	4.53	0.41
Low	5287	2666	50.43		1789	33.84		67.1		224	4.24

^*P-values are for Chi-square tests for differences between groups.

^**Table S4 provides final recruitment results for all households.

Chi-square tests were performed to detect demographic differences in recruitment, refusals, and visit completion (Table 1). The proportion of total households that agreed to participate was statistically significantly different by pre-selection intensity rating, with the highest proportion of households agreeing to participate in communities with a moderate rating. The proportion of households that completed visits was highest amongst Hispanic, rural, and moderate pre-selection intensity rated communities. Although rural communities had the highest proportion of completed visits, they also had the highest proportion of households refuse to participate during screening and this difference was statistically significant. The proportion of those who agreed to participate that completed a visit was statistically significantly different by region of the country with the highest proportion of completed visits in the South. The demographic distribution of completed visits is displayed in Strauss et. al. ¹⁷.

Lessons Learned

The HCS was implemented in a standardized manner in 130 diverse communities across the United States in 438 schools and 5,138 households. The study team devised and implemented strategies to attempt to overcome challenges. Table 2 summarizes the major challenges and obstacles to recruitment and the key solutions implemented that were most successful.

Table 2.

Recruitment Challenges Faced and Key Strategies Implemented

Topic area	Challenges faced	Key Strategies Implemented
District Recruitment	Time required to recruit school districts was longer than expected	Contacted large batches of original districts simultaneously in order to recruit multiple districts at the same time Reduced the timeframe for districts to respond to approximately 4 weeks, allowing staff to reach out to replacement districts more quickly
	Districts would not agree to participate due to demands on school staff	Provided local field staff to support PIF distribution and collection
	District refusal rate was higher than expected rate	Identified (via probabilistic sampling without replacement) and contacted up to three replacement communities for each originally selected community whose district refused When both a priority and a replacement community agreed to participate, rather than selecting only the priority community, both communities were included
School Recruitment	Time required to recruit enough schools in a community was longer than expected	Reduced time for schools to respond to two weeks, allowing staff to reach out to additional schools on the list more quickly Using probabilistic sampling, tripled the contact from two to six elementary schools and six middle schools simultaneously in each community, and increased the number of outreach staff
	Principal refusal rate was higher than expected	Offered to provide local field staff to support PIF distribution and collection
PIF Recruitment	Schools not responding	Sent HCS staff into schools to follow up in-person
	Low numbers of interested families	Redistributed PIFs within schools Contacted households with incomplete PIFs to get complete information and enroll them
Household Recruitment	Slow return of PIFs causing a delay in launching communities	Released PIFs in batches instead of waiting for all PIFs to be collected before beginning data collection in a community
	PIFs not evenly distributed by gender/grade	Collapsed recruitment strata into groups (i.e., combined kindergarten through 2nd grade) Increased the goals within each recruitment strata to allow recruitment of a greater proportion of the families that completed PIFs
	Reaching families	Enlisted the help of local FDCs to make additional attempts to reach households for recruitment by sending letters, making phone calls, or making in-person visits Provided contact information for recruited families without a scheduled visit to local FDCs so that they could attempt to schedule a household visit
	Staffing challenges	Arranged for staff with established success with recruitment and data collection to travel to communities with staffing issues to allow data collection to continue where insufficient local staff were available
KI Recruitment	KI nonresponse	Hired and trained “scheduler” staff to identify and contact KIs and schedule interviews. Utilized field data collectors to attempt recruitment through face-to-face visits for hard to recruit KIs in communities with few interviews.
	Not all priority KIs exist in every community	Thoroughly researched communities to identify KIs outside the priority roles to ensure all relevant CPPs were identified.
	Length of time needed to identify, recruit, and schedule interviews	Trained additional CL staff to conduct interviews by phone.

School District Recruitment took Longer Than Expected

Due to a higher than expected district refusal rate, longer time to obtain district approval, and contractual requirements, the study revised the district recruitment strategy to include provision of two replacement communities to be contacted simultaneously with the originally sampled community. If a replacement community approved first, the other communities were informed that if they did not agree within three days, they were unable to participate. Out of the 130 communities included in the study, 67 (51.54%) were communities/districts that were originally selected and 63 (48.46%) were “replacement” communities.

Limited School Staff

In some schools, staff were unable to collect PIFs due to competing priorities, so the study supplied HCS staff to work with the schools to increase household recruitment. The study offered assistance to 95 schools and collected 2,592 additional PIFs.

Additional Household Outreach

Household recruitment was designed to be completed through the study call center; however, FDCs located in or near the community made additional attempts to reach households by sending letters, making phone calls, or making in-person visits. Of the 3,309 households contacted, 13.90% refused to participate and 6.20% completed a household visit. The remaining households were finalized as non-complete when the field period ended.

The study team also assigned reliable, high-performing FDCs to communities with staffing issues (i.e., resignations, poor performance). These FDCs made 893 contacts (letters, phone calls, or in-person visits) to 420 households and completed 130 of 177 scheduled visits while travelling to other communities.

Community Context

The key players in program and policy implementation varied widely by community. In many communities, individuals initially identified as potentially knowledgeable about a range of CPPs had limited information about specific CPPs but connected study staff with KIs who were able to provide complete information. Due to the wide variation across communities, it was challenging to assess whether KI recruitment efforts had fully identified and recruited the most knowledgeable informants. In communities with fewer completed KI interviews, the study team conducted additional review to determine whether low numbers reflected inadequate KI identification or lower actual CPP activity in the community.

Initially, Community Liaisons (CLs) were to conduct KI interviews in-person when traveling to the community and by phone when in-person interviews were not possible. Additional interviewers were later hired and trained to conduct the interview by phone.

Discussion

HCS was a national study that aimed to capture a representative sample with respect to multiple demographic characteristics. Initially, the study was designed to follow a specific plan rather than obtain a convenience sample, which made recruitment challenging. Recruitment, however, did not have the challenges some other childhood obesity studies face because data collection was cross-sectional.

While not specifically guided by principles set forth by Schoeppe and colleagues, recruitment efforts in HCS did address several of the recommendations: the first Wave of HCS served as a feasibility study; CLs and FDCs from the community were engaged to facilitate trust, attempt to match demographic characteristics of the community, and to establish a study champion; the HCS did obtain approvals to participate from district and school levels ^7–9. In practice, the SLs were school staff and did not have adequate time to dedicate to the study, reinforcing the concept of imposing as little burden as possible ⁷.

Due to the national scale of the study, other suggestions made in the literature (e.g., identification of motivating factors, tailoring efforts to the school) were not feasible/cost-prohibitive ⁹.

Based on lessons learned during the implementation of the HCS protocol, we have several recommendations that may help future, similar studies in their recruitment efforts. When recruiting school districts, it is important to plan for sufficient time to recruit prior to the beginning of the school year as some school districts require that any research being conducted in the schools be approved prior to the beginning of the year. It is also prudent to ensure that the study plans a sufficient pool of replacement school districts to be recruited and to contact them simultaneously, when possible. To minimize impact on overall project timeline, studies involving research with school districts should limit the amount of time provided for school districts to make a final decision regarding participation. For individual school recruitment, simultaneous contact of multiple schools, limitations on response time, and on-site recruitment support are recommended.

To minimize delays due to staff loss, it is important to hire and train a larger number of FDCs than needed to account for attrition. Utilizing a traveling team of data collectors, if funds are available, would cut down on the number of staff that need to be trained and could reduce attrition due to more consistent hours. In studies where recruitment is primarily conducted by telephone, in-person household visits may be necessary to boost recruitment. Similarly, while conducting KI data collection in-person offers face-to-face opportunities to build rapport and an increased understanding of community context, supplementing this team with additional staff to recruit and interview KIs by telephone allows an increased number of interviews to be completed in a shorter time period.

Switching from mailing list-based recruitment to school-based recruitment had both advantages and disadvantages. Using school-based recruitment increased the likelihood that the children in the sample would be exposed to school-based programs and policies identified through KI interviews. However, this recruitment approach caused a shift from a probability-based sample of child participants to a largely volunteer sample.

When planning this type of study, it is important to be realistic about response rates, timeframes, and degree of effort required to secure participation at each level. Working with schools brings unique challenges that should be considered such as multiple levels of administration requiring approval, scheduling restraints with testing and school-wide activities, competing priorities, and privacy concerns. Future studies should begin recruitment strategy development via literature review and formative research, and should plan to pilot test strategies with their population of interest and to continue monitoring and adjusting recruitment methodology during data collection. It is critical that future studies continue to document challenges and lessons learned to disseminate this information to help inform future work.

Abbreviations:

HCS

Healthy Communities Study

CPP

Community Programs and Policies

CL

Community Liaison

KI

Key Informant

BMI

Body Mass Index

FDC

Field Data Collector

SL

School Liaison

PIFs

Parent Interest Forms