Development of an Informational Web Site for Recruiting Research Participants: Process, Implementation, and Evaluation

Abstract

Internet-based research is increasing, yet there is little known about recruitment approaches that target the Internet. Investigators have been slow to discuss how to plan, develop, and enhance recruitment using the Internet when well-concealed or disparate populations, sensitive topics, or qualitative methods are interspersed into the aims of the study. The twofold purpose of this paper is to 1.) highlight the major steps and strategies undertaken to develop and implement an innovative web site for recruiting high-genetic-risk couples who were considering preimplantation genetic diagnosis use, and 2.) present the recruitment results and lessons learned based on enrollment, self-evaluation, and descriptive data. The web site was developed using a five-step process designed by the investigators. A significant step in the process was determining the web site objectives, which were enacted through contextual and design decisions, and also by incorporating a brief video and study logo into the web site. The recruitment results indicate that of the 22 participant couples, ~82% were recruited via the Internet versus traditional recruitment approaches (i.e., clinics, newsletters) and that the majority of couples viewed the web site prior to enrolling in the study. In conclusion, developing a web site using the five-step process can facilitate recruitment.

Obtaining an adequate sample for research studies is challenging for many investigators. This challenge can be insurmountable for nurse researchers and social scientists who are interested in examining concealed, disparate, or vulnerable populations^{1, 2} or who have implemented an aggressive sampling plan^{3, 4} into their research. With use of the Internet rising,^5–7 nurse investigators are slowly, yet increasingly, turning to the Internet to obtain their samples exclusively or are incorporating Internet strategies into their recruitment plans.^8–10

The purpose of this paper is to report on the innovative Internet recruitment strategies that were undertaken to overcome the challenge of recruiting well-concealed and disparate participant couples for a qualitative study involving sensitive research topics and questions. Specifically, the purpose is twofold, and we begin by presenting the major steps that occurred in the process of developing and implementing an informational web site. Then we report on the recruitment results and the lessons we learned based on enrollment, our own self-evaluation, and the descriptive data collected from the 22 participant couples. We hope this information is useful for other investigators who are interested in developing and using an Internet web site for recruiting study participants.

Background

The target sample was 20 high-genetic-risk couples who were actively contemplating whether or not to use Preimplantation Genetic Diagnosis (PGD) or who had recently (i.e., within a three-month window) made a decision to accept or decline PGD use to prevent the transmission of a known genetic disorder to their future child(ren). PGD is a contemporary reproductive option that allows skilled reproductive clinicians and experts in genetics to evaluate oocyte or embryos in vitro for known genetic disorders, prior to uterine implantation and pregnancy.¹¹ We targeted couples who were actively or recently engaged in the decision-making process surrounding PGD use, because a primary aim of the study was to explore the couples’ decision-making processes in-depth using qualitative methods.

The recruitment challenge for our project was that according to available data the population of high-genetic-risk couples, particularly for couples opting to use PGD, is quite low. Determining the actual number of couples using PGD in the United States is difficult, as these data were just beginning to be compiled¹² when our recruitment plan was launched. Even with the efforts of the Centers for Disease Control and Prevention,¹² little national data existed on the number of couples who undergo PGD to prevent a known genetic disorder,¹³ which added to the concealment of the target population. However, there were several large private PGD centers located across the United States that compiled and published data on the number of PGD cycles undertaken,^13–15 which provided an indication of low but growing use of PGD in the United States. For example, Gutiérrez-Mateo and colleagues¹⁴ reported that they performed 224 cycles of PGD for 162 high-genetic-risk couples in the United States, Canada, and Japan over a recent 51-month period.¹⁴ Given these data, the sensitive nature of the study (i.e., personal accounts of reproductive and genetic decisions), and the short internal timeframe for eligibility (i.e., three-month timeframe from decision) we initiated a multifaceted recruitment plan using traditional and innovative Internet approaches to determine feasibility and evaluate appropriateness. A predominant innovation was the development and implementation of an informational web site to augment Internet recruitment. This article discusses the five major steps undertaken in this endeavor and then presents our recruitment results. We conclude this article with a discussion of the results and the lessons we learned through this process.

Major Steps in Developing the Web Site

The five major steps that we formulated and carried out in developing the web site are presented in logical order; however, the process was iterative and insight gained in one area often resulted in the revision of another area. Furthermore, extensive detail is provided in Step 4, Conceptualize the Purpose and Objectives of the Web Site, as we incorporated information about contextual and design decisions in this step.

Step 1: Review the Literature

The literature was reviewed to gain insight and understanding of web site development and Internet recruitment. Although there is an abundance of information on web site development in general, few reports addressed systematic information about how to develop a web site for recruiting eligible participants.¹⁶ To obtain beneficial information, we sought out reports from investigators who had used the Internet successfully for recruiting hidden, concealed, or disparate populations and/or when the study focused on sensitive topics. ^17–19 We were also interested in gleaning knowledge from nurse investigators who used the Internet successfully for accessing hidden populations using qualitative methods on sensitive topics.^{9, 10}

We also sought out and paid thorough attention to reports where Internet recruitment was slower and more challenging than expected²⁰ or resulted in disappointingly low response rates.²¹ These investigators often provided insightful information and strategies for facilitating future Internet recruitment success. In particular, we noted that the majority of participants who enrolled in several research projects did so after visiting a project web site.²⁰ Furthermore, if a project web site was developed, care should be taken to make the project web site attractive with pictures and special fonts,²⁰ and text should be carefully selected to convey a trustworthy and legitimate message.²¹

Other investigators made suggestions about web site design that were helpful. Zhang and von Dran²² encouraged web site designers to carefully consider the overall purpose of the web site (e.g., business, education, recruitment) and to pay particular attention to details important to the future users.^{22, 23} Gehrke and Turban²⁴ determined that web site users appear to be most concerned about page-loading speed, quality and usefulness of the content, and navigation efficiency, whereas others suggested a contextual reading level of less than tenth grade or the most appropriate level to the target population.^{25, 26}

Finally, we sought out literature that provided valuable information on broader issues surrounding Internet recruitment. Predominant issues reported were establishing participant authenticity;²⁷ identifying and working with moderators or gatekeepers of Internet communities and web sites to obtain access; ²⁷ and addressing security, confidentiality, and ethical issues^27–30 Another important issue was negotiating use of and navigation with stakeholders for the Internet server.²⁸

Step 2: Incorporate and Expand Previously Successful Recruitment Strategies

Prior experiences of the research team in recruiting similar concealed, disparate, or vulnerable populations for sensitive research guided our willingness to develop a project web site even though it extended our prior recruitment strategies.^31–34 For example, knowledge gained from the first author’s qualitative study examining the lived experience among donor egg recipient women in their second and third trimesters of pregnancy was evaluated.³¹ In addition, the second author’s expertise in theoretical recruitment approaches was incorporated into all aspects of the web site.³⁵

Step 3: Seek Expert Advice

Early in the process, experts were sought out to assist with the development of the web site and also to explore in-depth the issues surrounding web site development and Internet recruitment. Consultants who were sought out were an individual with expertise in Internet recruitment (Dr. Rebekah Hamilton), a psychologist with clinical experience and expertise with the population (Dr. Susan Klock), a well-established nurse researcher with expertise in women’s reproductive health (Dr. Ellen Olshansky), and an expert in decision science (Dr. Penny Pierce). A graduate student with expertise in web design and healthcare graphics (Ms. Lisa Merry) was also added to the team to assist in the technological development of the web site.

In addition to seeking out experts in substantive areas related to web site design and recruitment, representatives of the University of Illinois at Chicago Institutional Review Board (IRB) were contacted early in the process to obtain information about procedures to ensure human subject protections. Suggestions made by the IRB representatives were then incorporated into the development of the web site. These included having text within the web site that contained elements similar to that of traditional research recruitment flyers or brochures and clearly stating the IRB approval notice and protocol number on the web site. Noteworthy was the suggestion to include off-line hardcopies of the web pages and a transcript of any audio materials used in the web site included in the IRB application. Once IRB approval was obtained, the web site was uploaded to a web server, making it available online and accessible via the Internet.

During formative work, experts in budgetary considerations were consulted for cost estimates of implementing a research web site. The range of cost was variable from no fees for an investigator-designed and -developed web site to moderate estimates of 2,000 U.S. dollars for contractual work. After much consideration, a hybrid model was adopted where the first author, the principal investigator (PI), contracted and worked closely with a graduate student (i.e., fifth author) who contributed to the technological design and development of the web site.

Step 4: Conceptualize the Purpose and Objectives of the Web Site

After reviewing the literature, evaluating prior recruitment successes and challenges, and consulting with experts, the purpose and objectives of the web site were finalized. This was an important step and sufficient time and effort for reflective work was necessary. For our study, the main purpose of the web site was to inform potential participant couples about the study to ultimately foster participation. The objectives we set for the web site were to (a) communicate information, (b) establish credibility, (c) facilitate contact, and (d) demonstrate caring.

A. Communicate Information

Communicating and transmitting information effectively about the purpose of the study was paramount. Providing the eligibility criteria with the purpose allowed for potential participants to self-evaluate their own eligibility and optimize participant involvement from the onset of the study. It also served to minimize investigator workload regarding eligibility screening. Additionally, we also informed potential participants about the protocol and expectations of participating in the research.

In addition to deciding what information to communicate, we deliberated about how to communicate the information. Initially, we determined the number and style of visuals including an attractive color scheme and web site navigation aids. The color scheme was considered with respect to the target population of men and women of reproductive age, thus, we opted for colors that would appeal to both genders. Because the main aim of the study was to explore decision-making across a wide range of cultures, we developed and incorporated an ethnically neutral silhouette of a couple in a contemplative pose for the study logo, which served both as a visual reminder of the overall aim of the study and the target population. The logo, designed to color coordinate with the web site color scheme, can be seen in the top left corner of Figure 1. We also wrote the text for the web site at a tenth grade reading level (Flesch-Kincaid Grade Level score = 10.2). Although this is relatively high, our eligibility criteria required that couples have an awareness of PGD, which we found difficult to discuss at lower reading levels.

Figure 1.

Next, we contemplated how best to display the information on the web site. The two options we considered were to display the information on one large web page where potential participants could scroll down the page for additional information, or to display the information on several web pages contained on the web site and controlled by navigation tabs. We opted for the latter and developed a Home page that displayed vital study information to maximize interested participant preferences with brief information, and developed four additional web pages for detailed information: Study Details, Research Team, Contact, and PGD Resources. Figure 1 is a screenshot of the web site Home Page with the navigation tabs visible.

Another strategy to communicate information was to develop and include a brief (~4 minute) video on the Home page. In the video, the PI offers a brief welcome to the web site and provides key information about the study. The video was inserted for potential participants who might prefer receiving information through an audiovisual media versus a text media and was programmed to begin automatically when viewing the Home page.

B. Establish Credibility

The second objective was to inform potential participant couples about the high quality of the study and the expertise of the research team. This was accomplished through several strategies. First, the official University of Illinois at Chicago College of Nursing logo was displayed on the top right corner of each of the web pages. Additionally, information about funding sources, including grant numbers, and information about institutional IRB approvals, including protocol numbers, was delineated at the top of the Study Details page.

Another strategy to demonstrate the credibility of the study was to host the web site on the university’s web server that was linked to the college affiliated with the PI for the study, rather than have the web site as a stand-alone site. This was also financially feasible as the university-linked site was available at no cost. The study web site URL, linking to both the university and the college of nursing, became http://www.uic.edu/nursing/fertilitydecisions/PGD.

To demonstrate the high quality of the research team members, the Research Team web page was developed. Each of the key research team members (i.e., PI and Co-Investigators) had his or her name, credentials, title, a photograph, and a short biography that delineated his or her role and expertise for the study displayed on the Research Team page. The biographies were structured so that the expertise and experience of each key researcher was highlighted to reflect his or her particular contribution to one or more substantive areas of the study. An electronic hyperlink was inserted into each research team member’s name and linked to his or her individual faculty page at his or her respective university. The three consultants for the study were also listed on the Research Team page with their names, credentials, titles, and university affiliations.

C. Facilitate Contact

Facilitating contact between potential participants and the PI was an important objective because the PI determined eligibility and obtained follow-up information, if appropriate, for entry of the couple into the study. Because enrolling couples was paramount to the success of the study, information detailing how potential participant couples could establish contact with the PI was displayed on two web site pages. The Home and Contact pages contained the PI’s name, credentials, office phone number, and e-mail address. The e-mail address had an embedded hyperlink to minimize participant burden should a potential participant choose to establish contact with the PI. Information about how to contact the PI was also discussed verbally in the video and displayed on text superimposed during the video.

D. Demonstrate Caring

The objective to demonstrate caring was implemented to provide a virtual milieu through which the web site would be philosophically consistent with the nursing discipline and the underlying philosophical stance of the PI and the research team members. Specifically, the objective was to demonstrate care and compassion for potential and actual participants in the study and high-genetic-risk couples in general.

This objective was implemented through careful and considerate use of tone embodied in the text and audiovisuals. Another mechanism to demonstrate caring was the creation of the PGD Resource web page. This page was designed to include information and resources, available on the Internet and hyperlinked to corresponding web pages that might be helpful to high-genetic-risk couples regardless of whether they choose to participate in the study. For example, one of the resources hyperlinked is the National Institutes of Health’s Genetic Testing web page.

Step 5: Determine Conduits, Links, and Analytics

Brainstorming sessions were held among research team members to identify and determine appropriate Internet organizations and mechanisms that could serve as conduits for the web site and announce the study to potential participants. Prior to obtaining funding for the study, two Internet community groups, one professional group, and one professional group’s electronic mailing list were identified as key conduits for announcing the study and linking to the web site. Because we were also using traditional recruitment approaches involving brochures that announced the study at clinics and through advertisements in printed newsletters, we also included the web site address in these traditional approaches.

After the web site was developed and online, a gatekeeper at one of the previously supportive Internet community groups failed to respond to the PI’s numerous phone calls and e-mail messages to announce the study and post the web site link. This experience led to the PI developing a recruitment journal where names, dates, and records detailing discussion and talking points were recorded to facilitate communication and follow-up. Additionally, other Internet organizations where potential participant couples could be made aware of the study and a link to the web site posted were identified for potential use, if our initial strategies faltered.

We also formulated strategies to optimize web site searchability and visibility and decided on analytic procedures that could provide information about web site use. Regarding web site searchability, we opted to include meta tags in the html code of the web site to maximize search engine optimization. Meta tags enable search engines, such as Google and Yahoo!, to better understand how to categorize the web site and improve the web site’s search rankings. Thus, meta tags can increase and ease searchability and visibility by facilitating awareness of the web site among Internet users. The meta tags we used were PGD, Preimplantation Genetic Diagnosis, study, research, information, resources, high risk, genetic disorder, genetic testing, reproductive testing, and prevention. To further enhance searchability and visibility, we accelerated the process of having Internet search engines locate, retrieve, and display the study web site following queries by submitting the web site URL to the indexes of Google and Yahoo!.

Another strategy we used was to insert a two-sentence description of the study web site in the html code, enabling search engines to list a description of the site. In this instance, when the web site was retrieved after an Internet search, a description of the web site appeared underneath the URL link. Figure 2 provides the text used for this strategy. Lastly, we set up a free analytic system using Google Analytics that would record the number of visitors to our web site and collect useful data that can enhance our understanding of web site usage.

Figure 2.

Description of Web Site for Search Engines.

Recruitment Results

The web site went live in April 2009 and was subsequently linked to community and professional web sites and the electronic mailing list, which took several months to accomplish. During this time, gatekeepers and web site managers who had agreed to announce the study were contacted about posting an announcement or e-message that would provide the PI’s contact information and a link to the study web site. We also implemented our analytic strategies after the web site was live on the Internet. The results from Google Analytics for April 2009 through May 2010 demonstrated that there were 853 web site visits, of which 634 were first-time visitors. The average number of web pages viewed during visits was 2.59 pages. As a result of all of our efforts, the first couple entered the study in early June 2009 and recruitment was closed after the 22^nd couple entered into the study in May 2010. Results for recruitment indicate that of the 22 participating couples, ~82% (n = 18) were recruited via the Internet. Table 1 delineates how the couples found out about the study, as reported by the couples themselves.

Table 1.

Recruitment Details.

Recruitment Sites	Couples Reporting Awareness of Study from Site	Percent of Total Enrollment*
Internet
American Society for Reproductive Medicine E-mail List and Web site	5	22.7%
Charcot-Marie-Tooth Association Web site	1	04.5%
Muscular Dystrophy Association Web site	8	36.4%
Patient Support Web sites	4	18.2%
Total Internet Recruitment	18	81.8%
Traditional
Muscular Dystrophy Newsletter	1	04.5%
Private PGD Clinic	3	13.6%
Total Traditional Recruitment	4	18.1%
Total Recruitment – All Sites	22	99.9%

^*Numbers are rounded to nearest tenth.

We questioned each partner within the dyad as to whether they had viewed the study web site. Consistent with our success of Internet recruitment strategies, a large majority of the couples (73%) acknowledged that they viewed the web site prior to entering the study. Among the couples that viewed the web site were two couples that were made aware of the study by traditional recruitment strategies. Additionally, four couples that were recruited via the Internet did not view the web site, but opted to contact the PI directly after learning about the study from Internet announcements or e-messages from electronic mailing lists. Furthermore, we found that 59% of the female partners and only 14% of the male partners viewed the web site prior to participating in the study.

To obtain more in-depth understanding about participants’ perceptions and evaluations regarding the web site, using a semi-structured interview guide we asked each individual partner who viewed the web site to provide feedback on 1.) what was beneficial about the web site for recruiting participants and 2.) how the web site could be improved. Overall, the participants reported that the web site was helpful and informative. For example, participants said, “I … found it very helpful. It was very clear and easy to read and understand. I like that you kept it fairly simple and brief, yet explained all of the study details,” and “[I found] out, you know, what was going to be involved.” Another participant stated, “ [I used the web site to] just to get a sense of, you know, who you were, and, you know, all you researchers, and, you know, where you guys were coming from, and, you know, just to kind of inform myself more about it [the study].” Although few participants offered suggestions for improving the web site, we did ascertain that for several participants, the amount of time spent viewing the web site was limited. Comments such as, “Well, honestly, I didn’t spend a lot of time on it,” and “I actually did not look at the web site. I just jumped straight in [to enrolling in the study],” helped us gain insight about participant use.

Several participants also offered their perceptions about the video, which appeared to be an effective strategy based on this typical comment: “I thought that it was brief and to the point, but it gave enough information to pique an interest in participating in the study. I really liked that it seemed very personal and that you actually really care about helping others who are in the process of making a decision to use or not use PGD.” Another participant summed up the thought of others by saying succinctly, “I saw what you looked like. I heard you talk.” Yet, there was discrepancy about the content and length of the video. As one participant remarked, “I thought it was very thorough. You spoke very carefully and used simple language, which made the video understandable. For me, it was a little too slow, but I think others may need the more thorough video.”

Lessons Learned

We learned foremost what a tremendous amount of effort, planning, and coordination is needed to develop a web site. We underestimated the amount of time needed for thoughtful consideration of how to convey content, communicate information, navigate human subject protections, and work with Internet gatekeepers and managers. We did find our extensive review of the literature and consultation with multiple media and content experts valuable. However, there is a dearth of literature on web site design for recruiting concealed and disparate populations for sensitive research, and we often found ourselves traversing undocumented territory even with the advice and direction of experts. It should also be noted that if our budget was larger, we may have been able to minimize our effort and time. Because this was not the case, the PI undertook a systematic process where she was involved with all aspects and coordination of the web site, along with the assistance of a technologically experienced and knowledgeable graduate student (fifth author).

Nevertheless, about 82% of our sample was recruited via the Internet, and most couples had at least one partner view the web site prior to entering the study. We conclude that our multifaceted recruitment plan, aided by an informative web site, was highly successful for this study. We are keenly aware, however, that there may have been potential participant couples that found the web site not helpful or difficult to navigate or understand. Given the high number of individuals who viewed our web site (634 were first-time visitors) relative to our sample size, it is likely that there were individuals who opted not to participate in the study because of web site issues and thus their perceptions about the web site were not captured. What was demonstrated is that for the majority of participating couples, one partner viewed the web site and found the information helpful, and the couple enrolled in the study. It is noteworthy, too, that for couples entering the study by traditional recruitment approaches, 50% viewed the web site. Our findings also indicate that women were more likely to view the web site than men, and that our efforts to interject a caring philosophical stance throughout all aspects of the web site were positively perceived.

It is unknown whether or not the sensitive nature of the research, the focus on reproductive decision-making, or other unknown factors compelled the female partners to view the web site more than their male counterparts. Future research that explores these gender issues within dyadic couples’ research would be beneficial. In addition, we did learn that despite efforts to communicate effectively, some potential participants prefer brief and focused information. Furthermore, our interview data demonstrated the effectiveness of and preference for a short video in reaching some participants, while our analytic data suggested that web site visitors viewed about two to three web pages. In-depth research into the contextual factors that influence and enhance recruitment would aid understanding about web site development and design.

Regarding the broader issues surrounding Internet recruitment and use of a study web site to augment recruitment, we did not find authenticity of the couples to be problematic. Whether the study procedures (e.g., mailing informed consent documents to eligible couples’ private residences) and/or the qualitative and sensitive nature of the study deferred spurious participants remains unknown. Whatever the underlying instances were, there were no participants whose authenticity we questioned. Yet, because of the effort required by the couple and the PI to send and receive official study documents by mail and given our lack of bogus participants, we are considering using an online informed consent strategy in future studies as suggested by other investigators.²⁸

Another issue that warrants discussion is the effort expended to establish and achieve support from gatekeepers and web site managers. For this study, the PI developed and kept a recruitment journal to facilitate and ease follow-up communication. When recruitment closed, a total of 36 individuals had been contacted, representing 22 Internet organizations, electronic mailing lists, or private community web sites, and the journal contained 104 pages and consisted of 27,803 words. Although our efforts in recruiting the target sample were successful, obtaining access from Internet gatekeepers and managers took a significant amount of time and was challenging, as other investigators have also discovered.^{20, 27, 36} In contrast, in several instances, we experienced an effortless and supportive milieu among gatekeepers and managers. We are in the process of evaluating our strategies and data in this area to understand the phenomena and how best to navigate the relationship with the wider Internet community and professional groups.

In summary, developing an informational web site to aid in recruiting high-genetic-risk couples on the Internet was challenging, yet beneficial. The five-step process we undertook to develop and implement the web site resulted in the successful recruitment of a well-concealed and disparate sample to participate in sensitive research. We also found that half of the traditionally recruited couples viewed the web site, while not all participants recruited through the Internet viewed the web site, rather several “jumped straight in.” Research in this area is needed to understand how investigators can maximize use of a study web site as the Internet evolves and to enhance the research experience for the individuals, couples, and communities that we serve.