Analysis of four recruitment methods for obtaining normative data through a Web-based questionnaire: a pilot study

Abstract

Background

Quality normative data requires a diverse sample of participants and plays an important role in the appropriate use of health outcomes. Using social media and other online resources for survey recruitment is a tempting prospect, but the effectiveness of these methods in collecting a diverse sample is unknown. The purpose of this study is to pilot test four methods of recruitment to determine their ability to produce a sample representative of the general US population.

Methods

This project is part of a larger study to gather normative data for the Michigan Hand Outcomes Questionnaire (MHQ). We used flyers, e-mail, Facebook, and an institution-specific clinical research recruitment Web site to direct participants to complete an online version of the MHQ. Participants also provided comorbidity and demographic information.

Results

The institution-specific recruitment Web site yielded the greatest number of respondents in an age distribution that mirrored the US population. Facebook was effective for recruiting young adults, and e-mail was successful for recruiting the older adults. None of the methods was successful in reaching an ethnically diverse sample.

Conclusions

Obtaining normative data that is truly representative of the US population is a difficult task. The use of any one recruitment method is unlikely to result in a representative sample, but a greater understanding of these methods will empower researchers to use them to target specific populations. This pilot analysis provides support for the use of Facebook and clinical research sites in addition to traditional methods of e-mail and paper flyers.

Introduction

The collection of normative data is an important step in the continued assessment of any outcome instrument [21]. This data help providers to compare a patient’s current state to that of the general population, not simply before and after intervention or over time [34]. The Michigan Hand Outcomes Questionnaire (MHQ) is widely used to assess conditions and injuries involving the upper extremity [6]. Although much data exist regarding MHQ scores for individuals with specific ailments, there are no normative MHQ data [23, 24, 42]. In June 2013, we began a project to provide age- and gender-adjusted normative values for each domain of the MHQ and to compare MHQ scores for different demographic groups.

Obtaining normative data requires a sample that is an accurate representation of the general population. This will encourage broad comparisons and more specific ones based on groups defined by age, sex, and/or race. Therefore, we must survey a wide range of individuals of varying age, sex, and race, as well as individuals representing a diverse range of socioeconomic status [18]. Prior to data collection, we created a sample distribution representative of the total US population in sex, race, and age using 2010–2013 Census data estimates [3]. Given this diversity and the myriad of recruitment methods available, we wished to determine which approach would be most successful before launching a large-scale enrollment effort over a wide geographic area.

We chose four recruitment strategies to pilot test. These include two traditional methods, paper flyers and e-mail, which have both been widely used in research recruitment for over 20 years [33, 28, 27]. We also used two contemporary methods that have been in existence for less than 10 years to recruit participants: Facebook, launched in its current form in 2005, and an institution-specific study registry [11, 45]. In 2013, over 85 % of the US population regularly accessed the internet and over 80 % of households subscribed to an Internet service at home [4, 20]. Social networking sites such as Facebook are extremely popular. In fact, only Google is accessed more than Facebook [2]. Approximately 93 % of adult Internet users have a Facebook account, and each user spends an average of 11 h per month on the Web site [1]. Smartphone users are even more active, accessing their Facebook accounts an average of 14 times per day [17]. We also used our institution’s clinical research recruitment Web site [38]. The site was launched in 2005 to connect researchers with individuals interested in participating in research studies, including clinical trials, surveys, and qualitative research [11]. Members of the general public have the ability to confidentially provide health information and participation preferences (i.e., study duration, location, and compensation) and be matched with appropriate studies. Users may opt to be alerted when studies they may be eligible for are posted to the site. The site also allows researchers to search registered users to suggest studies to those meeting inclusion/exclusion criteria. As of June 2014, over 17,000 volunteers have created profiles [38]. Similar clinical research portals are available through the US government and other academic institutions including National Institutes of Health’s ClinicalTrials.gov, the World Health Organization’s International Clinical Trials Registry Platform, Duke’s DCRU.org, and Mayo Clinic’s ClinicalTrials.Mayo.edu [8, 19, 10, 7].

Each of these four methods has been successfully used in the past to publicize studies to specific subsets of the population. For example, printed flyers have been used to recruit African-American women ages 45–65 with cardiovascular risk factors [44], e-mail for US physicians using recently installed electronic medical records systems [13], Facebook for families with boys ages 8–18 with Klinefelter syndrome [9], and institutional research portals for patients with depression and chronic pain [26]. Despite these successes, however, the use of these methods to recruit a nationally representative sample is untested. The purpose of this project is to determine the effectiveness of four recruitment methods to provide an age-, gender-, and ethnically-representative sample.

Materials and Methods

Michigan Hand Outcomes Questionnaire

The MHQ is a hand- and wrist-specific outcome instrument that has been in use since 1998 [35]. The survey has 37 distinct questions divided into six domains: overall hand function, activities of daily living, work, pain, aesthetics, and satisfaction. For this study, the MHQ was administered using our institution’s Qualtrics portal, a Web-based software for survey research [40]. In addition to the MHQ, we asked participants to provide demographic information (sex, age, race, employment status, educational attainment, and household income) in a multiple choice format and to provide any relevant information about past and current hand injuries and conditions. Participants were also asked to complete the Self-Administered Comorbidity Questionnaire [32]. Based on past experience with the MHQ, we expected the study to take approximately 10 min to complete [35]. Participants were not compensated. As with any online survey, the MHQ had potential for respondent dishonesty. However, in the absence of an incentive for falsification of answers, research suggests that the majority of individuals participating in electronic medical surveys are truthful in their responses [22, 29]. We initiated participant recruitment using all four methods over the same period of time. Descriptive statistics were calculated using two-tailed chi-squared tests. Statistical significance was set at P = 0.05.

Paper Flyers

Brightly colored printed notifications with information regarding the study and tear-off tabs with the study URL were posted in areas highly trafficked by faculty, staff, students, visitors, and patients throughout our institution’s hospitals and at three outpatient clinics home to primary care and medical and surgical specialty clinics. Flyers were also posted in the medical school complex in instructional, research, and administrative areas. Flyers were checked weekly and replaced as needed.

E-mail

The research team e-mailed information about the study and a link to the online survey to all staff members in the authors’ section. The e-mail invited participants to forward or otherwise share the study with family and friends.

Facebook

Members of the research team used personal accounts to publicly post an IRB-approved Facebook status message with study information and a link to the survey. The status could be viewed not only by friends but also by individuals who were connected to the researchers via common friends, interest groups, and geography. As with the e-mail, users were invited to share the study information.

Institution-specific Clinical Research Recruitment Web Site

An IRB-approved listing was created for this study on our institution’s clinical research recruitment Web site. The only inclusion criteria listed were age of 18 years or older and the ability to read and write in English. Because of these broad criteria, nearly every adult registered user was notified that they may be eligible for this study. The study URL was provided to interested users who met the inclusion criteria.

Results

Pilot recruitment took place from June 2013 through October 2013. During this time, 374 individuals completed the survey. Demographic information for participants is listed in Table 1. Our sample was overwhelmingly female (80 %), white (90 %), and well-educated (97 % reported at least some college experience). Our institution’s clinical research recruitment Web site was the most successful recruitment method (60 %), followed by e-mail, Facebook, and flyers (Fig. 1). Facebook recruited significantly more young participants (18–37), whereas e-mail recruited significantly more participants age 68–77. Flyers recruited significantly fewer participants age 18–27 (Fig. 2). Our institution’s clinical research recruitment Web site yielded respondents in a distribution that mirrored the US population, including a large portion of young adults (48 % under age 37) and gradually smaller proportions with increasing age (28 % above the age of 58) [37]. There were no statistically significant differences between other respondent characteristics and recruitment method. The results reveal marked deficiencies in representation of racial minorities, individuals of lower socioeconomic status, and men.

Table 1.

Respondent demographic information, June–October 2013

	n (%)
Gender
Male	70 (20 %)
Female	304 (80 %)
Age (years)
18–27	86 (23 %)
28–37	85 (23 %)
38–47	48 (13 %)
48–57	80 (21 %)
58–67	58 (15 %)
68–77	12 (3 %)
78–87	5 (1 %)
88+	0 (0 %)
Race
White	338 (90 %)
Black	5 (1 %)
American Indian/Alaskan Native	1 (<1 %)
Asian	20 (5 %)
Hawaiian/ Pacific Islander	0 (0 %)
Two or more races	10 (3 %)
Employment status
Employed full time	221 (59 %)
Employed part time	65 (17 %)
Retired	29 (8 %)
Receiving disability payments	8 (2 %)
Full-time student	39 (10 %)
Part-time student	2 (<1 %)
Unemployed	10 (3 %)
Educational attainment
High school graduate	10 (3 %)
Some college	84 (22 %)
College graduate	147 (39 %)
Post-graduate degree	133 (36 %)
Annual household income
<$10,000	19 (5 %)
$10,000-$19,999	13 (3 %)
$20,000-$29,999	40 (11 %)
$30,000-$39,999	41 (11 %)
$40,000-$49,999	35 (9 %)
$50,000-$59,999	39 (10 %)
$60,000-$69,999	21 (6 %)
$70,000+	161 (43 %)

Fig. 1.

“How did you hear about the questionnaire?”

Fig. 2.

MHQ respondent age distributions for each recruitment method. An asterisk indicates statistical significance of P = 0.05 by two-tailed chi-squared test

Discussion

The results of our initial data collection efforts support the use of institution-specific clinical research recruitment Web sites and Facebook. The two sources were highly efficient, accounting for 71 % of our total sample and requiring relatively little time and effort and no direct cost to the research team. Facebook was especially successful in its ability to attract young adults. E-mail, on the other hand, proved to be effective in recruiting the retirement-age and elderly population, accounting for the greatest proportion of individuals 58 years old and older. This first round of data collection has helped to identify a number of populations who will be more difficult for us to reach. These include racial minorities (10 % of our total), unemployed and student respondents (14 %), individuals with lower household income (19 % reported an annual household income of less than $30,000), and men (20 %).

This initial data have provided new and interesting insights regarding specific recruitment methods. For instance, given that 90 % of the US population age 18–29 regularly uses Facebook whereas only 34 % of the US population over the age of 65 actively uses the Web site, it was no surprise that Facebook recruited significantly more individuals from the young adult population compared to the retirement-age population [30]. However, with 73 % of the African-American population and 72 % of the white population actively using Facebook, it was surprising that our sample had only 1 % of respondents self-identify as African-American [30]. This discrepancy is at least partly due to the demographics of the research team, creating a community bias in the sample [15]. In other words, Facebook status messages publicizing the study likely reached a sample of primarily well-educated, Asian and white individuals, reflecting the families and friends of the research team, in addition to those sharing common interests or living in the same geographic area. Moving forward, we plan on posting information regarding the study on our research section’s and university’s public Facebook pages, two sites accessed frequently by a more diverse population. Nevertheless, while the use of Facebook status messages is a valuable approach to participant recruitment, it is not a true random sample, and community bias is likely inevitable. An awareness of this bias allows researchers to overcome overrepresentation by targeting populations that are not being reached through the current strategy.

Similar insights can be gained through an analysis of the e-mail results. As communication technology has diversified in recent years, both teens and young adults alike have largely abandoned e-mail in favor of messaging via cell phones and social networks [36]. However, older adults have actually increased their use of e-mail. For example, compared to 5 years ago, time spent on e-mail has dropped by at least 12 % for every age group under the age of 55 years, but individuals over the age of 55 have increased time spent on email by over 15 % [12]. This helps explain why e-mail was a weak recruitment method for young adults but a substantial source of retirement-age and elderly respondents. E-mail remains a central aspect of internet usage of this older population and lends support to our success using e-mail as a recruitment method for this demographic.

Of the four methods employed, the posting of paper flyers is undoubtedly the oldest, having been widely used for over 30 years [27]. The flyers were posted in clear sight and in high-traffic areas for approximately 5 months. Given that younger people are less likely to need medical attention and visit the hospital system, it is not surprising that this method recruited significantly fewer young adults than the others [5]. However, of all four methods, the flyers likely reached the largest audience, with an estimated 800,000 patients visiting the hospital system while the flyers were posted [39]. Therefore, it is surprising that only 12 % of respondents reported being recruited via this method. This result is a further testament to the usefulness of contemporary methods of recruitment that offer more direct and convenient access to the survey.

The county in which this study was performed boasts a population that is approximately 75 % white and 90 % of the population is above the poverty line [41]. Therefore, we expected our data to be biased toward these demographics, but we were surprised by the extent to which this was true. In response, we intend to employ the same methods in our next round of recruitment but will alter and add methods to increase our most needed participant groups. One strategy that may be of use is snowball sampling. This method of sampling recruits one participant with the desired characteristics, known as the “seed,” who then can recruit more participants with similar characteristics. This recruitment method has helped researchers overcome many of the challenges associated with enrolling participants from hard-to-reach populations, such as general distrust of health care personnel, lack of community support, and unavailability of study information [16, 31]. Snowball sampling has been shown to be an effective strategy in which the seed’s response rates are often higher than those of the initial study administration [14]. For our study, we will identify appropriate seeds in our community who have not previously publicized the study, based on characteristics such as age, sex, race, and level of education. These individuals will be asked to post the IRB-approved Facebook status message and to encourage their friends to share the study information, hopefully resulting in a snowball sample of the desired populations. Although this sample will not be random, it will ultimately allow the researchers to collect data from a sample that more appropriately represents the USA as a whole.

Another planned recruitment method is to publicize the study at locations frequented specifically by elderly and racially diverse populations, such as senior centers and supportive housing communities. These centers of activity have been proven resourceful in previous research efforts [25, 43]. Additionally, our institution is affiliated with a number of free care clinics and community resource centers that treat diverse populations throughout the larger metro. By actively publicizing and offering the survey at these locations using tablets and laptops, we hope to increase the representation of these hard-to-reach groups.

Normative data for health outcome measures such as the Michigan Hand Outcomes Questionnaire are valuable, but collection poses a number of unique challenges. Though clinical research portals serve as effective means to recruit participants and social networks like Facebook are successful in attracting young adults, it is evident that hard-to-reach populations still exist. Our first wave of data collection has shed light on biases and allowed us to install efforts to reduce their impact. The data has indicated to us which populations will require additional focus to be sure our sample is truly representative. Contemporary recruitment methods such as social networks and clinical research portals can save researchers time, money, and effort as many fields seek to obtain normative data and increase their clinical research portfolios.