Recommendations for Internet-Based Qualitative Health Research With Hard-to-Reach Populations

Abstract

Researchers new to online qualitative health research frequently have questions about how to transfer knowledge of offline data collection to an online environment. In this article, we present best-practice guidelines derived from the literature and our experience to help researchers determine if an online qualitative study design is appropriate for their research project and, if so, when to begin data collection with a hard-to-reach population. Researchers should reflect on administrative, population, and data collection considerations when deciding between online and offline data collection. Decisions must be made regarding whether to conduct interviews or focus groups, to collect data using asynchronous or synchronous methods, and to use only text or incorporate visual media. Researchers should also reflect on human subjects, recruitment, research instrumentation, additional data collection, and public relations considerations when writing protocols to guide the research team’s response to various situations. Our recommendations direct researchers’ reflection on these considerations.

Online qualitative data collection has existed since the 1990s when email, message boards, instant messaging, and chat software became accessible and affordable. Today, it is increasingly being used to generate research data across disciplines, including health. One reason that online qualitative data collection is preferable to offline qualitative data collection is that transcripts are instantly available and free of transcription errors, assuming text-based data collection methods are employed (Adler & Zarchin, 2002). Another reason is that, depending on the data collection software used, all comments can be easily associated with a participant and date and time stamped (Gaiser, 2008). However, for many researchers, especially those new to online qualitative health research, questions remain about how to conduct online qualitative data collection. Researchers thus need guidelines for making key decisions related to designing and conducting online qualitative research studies.

In this article, we present best-practice guidelines derived from the literature and our experience to help researchers determine if an online qualitative study design is appropriate for their research project and, if so, how to conduct online qualitative data collection. Specifically, these guidelines will direct researchers’ reflection on the various considerations associated with conducting an online qualitative study. We also highlight the utility of online qualitative research methods for collecting sensitive health data from a range of hard-to-reach populations. We define hard-to-reach populations as those often absent from mainstream research because of an ambiguous sampling frame or systemic exclusion from data collection instruments (Benoit, Jannsson, Millar, & Phillips, 2005; Brackertz, 2007).

In the United States, 82% of all adults use the Internet and 88% use mobile phones (46% use smartphones; Pew Research Center’s Internet and American Life Project, 2012). The widespread adoption of mobile phones and the increasing affordability of smartphones have allowed groups that were initially on the other side of the digital divide to gain access to the Internet. Among smartphone owners, young adults, racial and sexual minorities, those with no college experience, and those with lower household income levels are more likely than other groups to report their phone as their main source of Internet access (Zickuhr & Smith, 2012). Although current limitations in mobile technology and research design will often require these groups to access a desktop or laptop computer to participate fully in an online qualitative health study, advertisements and screeners on mobile devices can be used to recruit such groups.

Our Online Qualitative Experience with Hard-to-Reach Populations

In this article, we specifically focus on three examples from our experience working with Internet-based methods, which spans the last 15 years. In this section, we provide a brief overview of the study design and online qualitative methods used for each of the three studies. In subsequent sections, we discuss the choices that were required during the study development phase and the lessons learned from each situation. For additional information about these projects, including their methodology and available results, please refer to the citations provided within the project descriptions.

Sexually Explicit Media (SEM) Study

In this study, we sought to understand if and how use of sexually explicit media influences sexual risk behavior in men who have sex with men (MSM). For the qualitative component of this study, participants were recruited online over a 2-week period in January 2010 using banner advertisements, which directed interested persons to a Web page hosted on a dedicated university server with appropriate encryption to ensure data security. The Web page included information about the study procedures and a link to the eligibility screener. A total of 79 MSM participated in 1 of 13 online focus groups conducted between January and March 2010. Each focus group lasted approximately 90 min. After each focus group, participants were invited to participate in a follow-up discussion on a message board; 66 (84%) participants posted additional responses in this format. Transcripts were analyzed using NVivo 8. Results from this qualitative study were published by Wilkerson et al. (2012).

All Gender Health Online

In this study, we addressed the deficit of knowledge about the characteristics and sexual behavior of non-transgender men who have sex with transgender persons. We examined issues of identity, relationships, sexual health, risk factors for human immunodeficiency virus (HIV) transmission, and preventive education needs to minimize HIV transmission. One of the components of the study included online focus groups with non-transgender men who have sex with transgender people and transgender people who have sex with non-transgender men. Participants were recruited using banner advertisements that directed interested persons to the study’s Web site, which included a description of the project and a link to the eligibility screener.

Between September 2008 and February 2009, a total of 34 transgender individuals participated in one of seven focus groups, and 23 non-transgender men participated in one of six focus groups. Each focus group lasted approximately 120 min. Transcripts were analyzed using NVivo 8. Results from All Gender Health Online are currently being analyzed for publication. To read more about related research, please see Feldman, Swinburne Romine, and Bockting (in press); Iantaffi and Bockting (2011); and Miner, Bockting, Swinburne Romine, and Raman (2012).

Polyamorous Parenting

In this study, we explored the experiences of family among polyamorous parents and their children using personal accounts collected via email-based interviews. This qualitative, narrative study also included an autoethnographic component (Chang, 2008; Maréchal, 2010). Iantaffi recruited English-speaking individuals from online forums dedicated to polyamorous parents (LiveJournal and listservs) via an email appeal in June 2007. When necessary, Iantaffi was introduced by other trusted members of the networks approached because many of those groups had closed memberships. Iantaffi received permission to use entries from blogs kept by some of the participants to record their own experiences and reflections, including those on parenting. A total of 11 parents, one adult child, and three young children from the United Kingdom and the United States participated in the study. For more information about this study, please see Iantaffi (2009).

Designing an Online Qualitative Study

The following sections of text parallel the sections of the checklist shown in Appendix A. Each section provides a narrative of considerations to reflect on when designing an online qualitative study. Researchers will need to decide between employing an online or offline study design, conducting interviews or focus groups, collecting data with asynchronous or synchronous methods, and capturing text-only or text and visual media data. As noted above, the considerations described below and the checklist items in Appendix A are grounded in both the literature and our experience.

Deciding Between an Online and Offline Qualitative Study Design

When considering an online qualitative study design, researchers must first decide if online data collection methods are appropriate. The literature indicates that online qualitative data collection is equivalent, and in some cases superior, to face-to-face interactions (Campbell et al., 2001; Hinchcliffe & Gavin, 2009; Kenny, 2005; Reid & Reid, 2005), especially when trying to access hard-to-reach populations (Adler & Zarchin, 2002; Fox, Rumsey, & Morris, 2007; Nicholas et al., 2010; Sweet, 2001; Tates et al., 2009; Thorsten, Isabelle, Alexander, & Roediger, 2008). However, equivalency at the meta-level does not always translate to reality. Each time researchers begin a project, they should reflect on several considerations to determine which data collection method will best produce the desired outcome. In general, there are administrative, population, and data collection considerations that researchers must reflect on to make an informed decision. Section 1 of the checklist in Appendix A includes specific questions related to each of these areas of consideration. For researchers responding “yes” to most of these checklist items, online methods are a viable option for qualitative data collection.

Administrative considerations

Administrative considerations include the experience of the research staff in employing online and offline qualitative data collection methods and the budgetary constraints of the project. When conducting online qualitative data collection, research staff must have both facilitation skills and technological expertise. The literature suggests that it is more challenging to facilitate online qualitative data collection than offline qualitative data collection (Frisoli, 2010; Henrietta & Clare, 2003; James & Busher, 2006). Online researchers must be fluent in the use of online data collection tools and in the deployment of qualitative data collection techniques, as well as be able to transfer offline data collection skills to the online environment (Gaiser, 2008; Im & Chee, 2006). Our experience is consistent with what is reported in the literature.

Preferred skills include familiarity with database management, instant messaging, and chat and video conferencing platforms. Research staff should be comfortable manipulating graphic, sound, and video files to extract a clip for use during a focus group, and browsing the Web to quickly access and link to information. In addition, staff should be trained in qualitative research methods, including how to structure questions during an interview or a focus group to elicit detailed narratives from participants. Because the pace of online data collection is often fast, we recommend that staff practice facilitating in-house, critiquing both the quality of questions asked and the responses generated, before engaging participants online. After data collection begins, we recommend that staff continue the practice of critiquing the quality of questions and responses and that they modify their interview guide as appropriate. The result of this iterative process should contribute to the constant improvement of the research team’s data collection skills and, thus, to increased data quality.

Budgetary constraints also influence the decision to engage in online or offline qualitative data collection. There is literature supporting online data collection as an affordable alternative to offline data collection. For example, in an online environment, costs associated with space rental, refreshments, and transcription can be avoided (O’Connor, Madge, Shaw, & Wellens, 2008). However, the costs associated with some online data collection software and online recruitment advertisements can negate any potential savings. Although some software packages are free, others cost up to a few hundred dollars per user license.

Recruiting a sufficient number of participants might also cost more for online studies than for offline studies. Many Web site developers charge for each person who clicks on a banner advertisement, regardless of whether the person decides to complete an eligibility screener or consent form. If researchers want to recruit from a very specific population (e.g., substance-using MSM), they should expect to pay for a number of clicks associated with ineligible persons. Our experience suggests that researchers should expect a click-through rate of 0.1–0.3%. In addition, many Web site owners or online advertisers require researchers to sign a contract for a minimum dollar amount and number of clicks. Unless there is a staff graphic designer, researchers should include money in their budget to pay someone to design study advertisements.

Population considerations

Population considerations include the extent to which members of the target population will be comfortable using technology, the level of literacy of the target population, and the extent to which the target population needs to be geographically diverse. There are many situations in which using online methods increases access to members of hard-to-reach populations. Online methods are particularly useful when researchers want to recruit from various regions (O’Connor et al., 2008; Schneider, Kerwin, Frechtling, & Vivari, 2002) or eliminate participants’ travel burden (Campbell et al., 2001; Kenny, 2005). Although technology is becoming increasingly integrated into U.S. society, older adults and those with lower household income levels are less likely than other groups to have access to or be comfortable using technology (Pew Research Center’s Internet and American Life Project, 2012).

Projects relying heavily on text-based data collection methods such as online interviews or focus groups might inhibit participation by persons with limited typing skills, or with dyslexia, visual impairments, or low literacy levels (Clark, 2007). This can be especially problematic when numerous persons are involved in a conversation simultaneously. The speed with which the conversation occurs and the cognitive load placed on participants is substantial. Participants must quickly read, process, and type a response. This can limit participation for persons less familiar with technology or result in more parsimonious responses (Campbell et al., 2001). Similarly, projects in which participants are asked to view audio or video files can inhibit participation by persons with certain disabilities if the files were not developed using guidelines for universal design (e.g., video captioning and reader compatibility; Rose & Meyer, 2002).

Data collection considerations

Data collection considerations include researchers’ ability to secure data, the extent to which they want to guarantee participants’ confidentiality or anonymity, and their capacity to create researcher–participant rapport. Researchers conducting online qualitative data collection should work closely with a member of their institution’s information technology staff to identify a dedicated, secure server for their project. The use of a secure sockets layer (SSL; a protocol for encrypting information over the Internet), the utilization of hypertext transfer protocol secure (HTTPS; a protocol for secure communication over the Internet), and the implementation of data safety protocols can help to ensure confidentiality or anonymity. Despite these data security measures, total anonymity or confidentiality cannot be guaranteed.

Researchers should recognize that persons with technical expertise interested in accessing study data (commonly known as hackers) cannot be stopped completely (Gaiser, 2008; Im & Chee, 2006). In addition, because participants will be taking part in the study from their own computer, researchers have less control over who has access to the data. Participants can save the data file from the discussion in which they participated and share it with others without the knowledge of the research team (Kazmer & Xie, 2008). Therefore, when obtaining consent, researchers should clearly state that data are the property of the research team’s institution and that saving a copy of the data file or sharing it with others is prohibited.

Technical problems might also interfere with data collection. For example, slow connection speeds can inhibit participation in a discussion, and loss of Internet connectivity can result in a loss of unsaved data (Moloney, Dietrich, Strickland, & Myerburg, 2003). To minimize the risk of losing data, we recommend saving a backup data file every 15 min to the study server, even if using software that saves automatically. Doing so will help to protect study data in the event of technical problems.

The literature and our experience suggest that online methods allow for greater disclosure than offline methods, possibly because of perceived anonymity (Ayling & Mewse, 2009; Campbell et al., 2001; Mann & Stewart, 2000). This is especially important when collecting information about sensitive topics or from hard-to-reach populations. In addition to perceived anonymity, participants are also more likely to perceive a higher level of group homogeneity, increasing their willingness to disclose personal experiences (Joinson, 2005). Because of the increased perceived anonymity, researchers have an obligation to protect the identity of study participants. In all our projects, we ask participants to create unique user identifications, and we provide links to free email service providers for those who wish to create a separate email address for project-related communications. By taking these steps, we try to increase anonymity among participants and confidentiality between participants and the research team.

Increased perceived anonymity can decrease social pressure to continue participation in an interview or focus group. Consequently, participants in an online study might be more likely than those in an offline study to become disengaged and drop out (Johns, Hall, & Crowell, 2004; Tates et al., 2009). Thus, researchers need a strategy to maintain participant engagement, and they need to consider the pace of the discussion, as well as the wording of questions, to minimize opportunities for participants to become distracted. There is a greater risk that participants in an online study will engage in multiple activities while participating than participants in an offline study (Frisoli, 2010; Gaiser, 2008).

Posting profiles of research team members and allowing time at the beginning of an online data collection session for introductions and informal communication can begin creating rapport between researchers and participants (Henrietta & Clare, 2003; O’Connor et al., 2008). Because the use of technology reduces researchers’ and participants’ access to visual cues, it is often difficult to know how someone is feeling. Encouraging the use of emoticons and typical text or chat abbreviations, such as “LOL” for “laughing out loud,” can provide insight into feelings. However, researchers should be careful not to assume the meaning of emoticons and abbreviations: they should clarify their meaning when conducting online data collection, just as they would when conducting offline data collection (Fox, Morris, & Rumsey, 2007; Im & Chee, 2006). Although the use of audio or video data collection could build rapport, researchers should weigh the use of these technologies against the cost of transcription, assuming a written transcript is desired. Many qualitative data analysis software packages allow researchers to import and code audio and video files.

After data have been collected, many researchers need a method for compensating study participants. In an offline study, the accounting policies of many institutions require the collection of name, address, phone numbers, and occasionally, other personally identifiable information before issuing a check or distributing cash to study participants. Disclosure of such information can be a major concern for members of hard-to-reach populations who have experienced discrimination for one or more aspects of their identities. In an online study, however, participants’ confidentiality is protected during the compensation process. By issuing online gift certificates to a particular email address via an online retailer, researchers are able to minimize the collection of participants’ personal information and satisfy accounting requirements. Researchers are encouraged to check with their institution’s accounting department to develop an online payment procedure that allows for minimal collection of participants’ personally identifiable information.

Deciding Between Online Interviews and Focus Groups

Interviews and focus groups indicate whether one person or, collectively, a group of persons is being interviewed at a point in time. The reasons for choosing to do individual interviews versus focus groups are the same for both online and offline environments. Depending on their familiarity with individual interviews and focus groups, researchers should consult the following publications: Denzin and Lincoln (2011), Krueger and Casey (2009), Patton (2002), and Rubin and Rubin (2011). These publications include detailed information about these two data collection techniques, including the context in which each is superior and the important differences between them. Section 2 of the checklist in Appendix A includes questions designed to assist researchers in deciding whether to conduct interviews or focus groups.

There are several contexts in which individual interviews are typically superior to focus groups. First, when researchers want to gain in-depth information, rather than a breadth of perspectives, on a topic—frequently in the form of detailed narratives or stories. Second, when researchers are collecting data that are highly sensitive or could result in harm to an individual if disclosed to a group. Third, when researchers want to compare opposing opinions on a topic, unless participants are being stratified into homogenous groups based on a priori information. In contrast, focus groups are usually superior to individual interviews when researchers want to quickly capture a breadth of perspectives on a topic that is perceived by the group as not being too sensitive to discuss in the presence of other people, or when the group’s perceived homogeneity allows views to be expressed more freely.

When conducting online focus groups, we recommend following best practices for conducting offline focus groups (Krueger & Casey, 2009), including a minimum of three to four focus groups per strata, and five to eight participants per group. With fewer than five participants, it is difficult to maintain a lively discussion, and with more than eight, it is difficult to facilitate a discussion. We recommend recruiting at least twice as many persons as needed for each group, and to expect up to half of participants to not log in at the scheduled time. If a participant attempts to log into the system more than 10 min after a discussion has begun, we recommend denying this participant entry. The introduction of a new person could change the group dynamic in an unpredictable way, and valuable time could be lost trying to integrate the new person into the conversation. Instead, the participant should be invited to participate in a future focus group.

When facilitating an online focus group, we recommend scheduling three staff members. The first staff member can serve as moderator, typing text and facilitating the discussion. The second staff member can serve as co-moderator, scanning the entered text, making notes, and assisting the moderator with the formation of probing questions. The third staff member can serve as technical support assistant, verifying the participants’ user identification at log-in, rescheduling persons who try to log in late, helping participants resolve technical problems, and saving backup copies of the discussion. To assist with pacing of the conversation, facilitators might find it helpful to have a copy of their interview guide in an open word processing file, so that commonly asked questions can be copied and pasted into the data collection software. At the beginning of each focus group, we recommend reviewing important elements of the consent and establishing ground rules for the discussion. At the end of each focus group, we recommend reminding participants how they will be compensated and when to expect compensation, because this is a frequently asked question.

Our experience suggests that participants who frequently meet other persons online quickly become comfortable sharing personal information and interacting with other participants. When a conversation begins to lag or when the discussion is ending, participants are more likely to begin soliciting personal information from others. To minimize opportunities for participation in unrelated conversations, we recommend using software that prevents participants from privately instant messaging each other. Facilitators can quickly redirect a conversation if participants begin to ask personal information, and if a participant is persistent, the technical support assistant can privately contact the offending individual via email or an instant message and ask them to curb conversations unrelated to the study. Although we have never had to remove someone during a focus group discussion, we were prepared to do so if a participant continued to solicit personal information from others.

Deciding Between Online Asynchronous and Synchronous Data Collection

After determining whether interviews or focus groups are most appropriate for their project, researchers need to determine whether to collect data in real-time (synchronous) or not (asynchronous). An instant message conversation between researchers and a participant is an example of a synchronous interview, whereas an email conversation is an example of an asynchronous interview. Group chats and message boards are examples of synchronous and asynchronous focus groups, respectively. Section 3 of the checklist in Appendix A includes questions designed to assist researchers in deciding whether to collect data asynchronously, synchronously, or both.

Researchers must decide how employing asynchronous or synchronous data collection will impact the quality of collected data. Asynchronous data collection is useful when researchers expect participants to have difficulty scheduling a real-time conversation; participants can respond at their own leisure (James & Busher, 2006; Moloney et al., 2003; O’Connor et al., 2008; Sweet, 2001). This might be particularly useful if collecting data during holidays or weekends when participation in a synchronous discussion might decline (Moloney et al., 2003). In addition, asynchronous data collection allows researchers more time to develop thoughtful probing questions based on participants’ responses to previous questions (Im & Chee, 2006; Moloney et al., 2003), which is potentially useful when the study design includes grounded theory (Glaser, 1992; Strauss & Corbin, 1998). Participants also have time to reflect and to formulate a more thoughtful response (James & Busher, 2006; O’Connor et al., 2008).

Asynchronous data collection might be better than synchronous data collection for certain populations. For example, asynchronous data collection might be better for older adults, who might be more comfortable with email and less comfortable with instant messaging and chat rooms (O’Connor et al., 2008). Asynchronous data collection might also be better for participants who do not have access to high-speed Internet connections. However, researchers wanting to engage participants over time using an asynchronous method will need a strong retention plan. The longer researchers attempt to retain participants, the greater the risk participants will quit responding to queries (Im & Chee, 2006).

In contrast to asynchronous data collection, synchronous data collection has the benefit of mimicking a real-time face-to-face conversation, to the extent possible with today’s technology (O’Connor et al., 2008). Although synchronous focus groups allow researchers to collect data quickly from numerous participants, they can be challenging to facilitate well. Facilitators might find it difficult to know when one comment ends and another comment begins because of the lag between the time when researchers pose a question and the time when participants type their response. This lag time will likely result in a disjointed transcript in which a comment from one or more participants can refer to a question or comment from earlier in the conversation. This problem is usually exacerbated as the number of focus group participants increases (Gaiser, 2008; O’Connor et al., 2008). This challenge is manageable by ensuring research staff is well trained in online data collection and data analysis.

In our work, we have begun to favor a two-staged study design in which we begin with synchronous data collection and follow up asynchronously with clarifying questions. Specifically, at the end of a synchronous data collection session, we review the transcript to identify clarifying questions. We then email the participants the questions and ask them to respond within a defined amount of time (when conducting focus groups, we post questions to a secure message board and email participants instructions for logging into the message board). To encourage follow-up participation, we offer an additional incentive to participants who respond within the requested timeframe. By including asynchronous follow-up, we find participants share longer narratives and elaborate on comments that arose during the synchronous conversation.

Deciding Between Text and Visual and Text-Only Data Collection

Internet-based methods of data collection need not be limited to text only. Several software platforms now offer the possibility to include audiovisual media or to let the participants create visual representations of their experiences through drawing and other graphic tools (e.g., photo galleries). Because visual methods of data collection are still relatively new in many disciplines (Reavey, 2011), especially in Internet-based qualitative research, we are currently exploring our own best-practice guidelines in this area. For example, we have started using a software platform that includes a white board, which allows for the application of an offline visual data collection technique, Rivers of Experience (Iantaffi, 2011), to an online environment. Other offline visual tools, such as Photovoice, can also be adapted for online use. Section 4 of the checklist in Appendix A includes questions designed to assist researchers in deciding whether to collect text and visual or text-only data.

When considering whether to collect visual data, researchers first need to evaluate the technical capabilities of the software platforms available to them, the speed of Internet connections that participants might have access to, and the acceptability and usability of the selected visual tools. Researchers need to be able to teach participants how to use those tools. They also need to allow extra time either before or during the interview for participants to become familiar with both the software platform and the selected visual tool. Furthermore, researchers need to consider the higher cognitive load for participants when visual methods of data collection are used. Finally, researchers need to consider access to resources, such as devices to take, scan, and upload photos, if they choose to use these data collection methods.

Visual methods of data collection can be particularly helpful when researchers want to go beyond purely verbal constructs. As participants use visual images and metaphors to help describe their identities, experiences, and practices, researchers are able to obtain more detailed narratives. When collecting chronological data in particular, adding a visual element might enhance the breadth and depth of the collected data by engaging participants in a joint reflexive journey alongside the researchers (Iantaffi, 2011). Collecting visual data could also be useful if researchers are asking participants to compare experiences, identities, or practices across time and space. In those cases, asking participants to create visual artifacts can provide anchors during the conversations and reduce cognitive load for the participants.

Preparing for Online Data Collection

Having a good study design is only one aspect of conducting online qualitative research. Before beginning online data collection, researchers should engage in an extensive planning process, especially when collecting data on a sensitive topic or from a hard-to-reach population. In Appendix B, we provide a checklist for researchers to ensure all necessary steps have been undertaken before launching recruitment. Researchers should be able to respond yes to all these checklist items before beginning data collection. In this section, we offer general recommendations of what to include in each of the corresponding protocols. The protocol discussion is positioned within a larger discussion about human subjects, recruitment, research instrumentation, data collection, and public relations.

Human Subjects Considerations

Before beginning data collection, we recommend that researchers ensure that their Institutional Review Board (IRB), or equivalent ethical review boards for researchers outside of the United States, has approved final versions of advertisements, consent forms, and research questions. In addition to ensuring that research team members have completed human subjects training, researchers should establish a written protocol to follow when interacting with study participants. In this protocol, we recommend including sections on how to respond to inquiries from participants via phone, email, and social media, both during and after office hours. The protocol should include standardized responses to common questions, as well as an algorithm for determining which questions can be answered by research staff and which questions should be answered by a co-investigator, the principal investigator, or a member of the university community authorized to speak on behalf of the institution.

The protocol should also include a process for identifying and documenting a reportable event as defined by the researchers’ institution. If participants will be able to reach a project voicemail or auto-reply, we recommend including a statement referring participants to their local 911 services (or equivalent if outside of the United States) in the event of an emergency. This is especially important if participants might experience psychological or emotional distress as an artifact of their participation in the research study. We also recommend compiling a referral list for the geographical areas in which researchers will be recruiting participants and to log all communications with participants in a secure database. When collecting data from hard-to-reach populations or about sensitive topics, we recommend obtaining a Certificate of Confidentiality if the study is funded by the U.S. National Institutes of Health.

Recruitment Considerations

Before recruiting participants, we recommend that researchers create a detailed, written recruitment protocol, which describes the study’s recruitment and retention plan, as well as the participant compensation plan, and includes drafts of emails, Web site postings, and online advertisements. When recruiting online, researchers must choose between respondent-driven sampling, snowball sampling, Web site banner advertisements, online advertisements, or video campaigns. In our work, we rarely use respondent-driven sampling because we find it difficult to select appropriate seeds when so little is known about the demographic profile of our target populations. We find snowball sampling to be more cost-effective than banner advertisements, online advertisements, or video campaigns. For our larger studies, we do frequently use banner advertisements to recruit participants.

Snowball recruitment relies on email distribution lists, as well as text and, potentially, video messages about the study on social media (e.g., Facebook, Twitter, YouTube, Tumblr). As previously discussed, banner advertisements can be expensive and usually involve contracting with one or more Web sites, or with an advertising agency that has a relationship with numerous Web sites frequented by the target population. Emails, text and video messages, and banner advertisements should link to a brief description of the study and an eligibility screener. Ineligible persons should be thanked for their participation and redirected to an exit page for the project. Eligible participants should be directed to the online consent form and then to the data collection instrument. For information about how to obtain consent online, please see Rosser et al. (2009).

Research Instrumentation Considerations

There are more instrument considerations when collecting data online than when collecting data offline. After obtaining sign-off from the IRB, principal investigators, co-investigators, and the community advisory board, researchers need to ensure that the online data collection software is free of “bugs” (i.e., programming errors). To the extent possible, we recommend setting up a codebook with expected codes based on the research questions. This should expedite data cleaning and analysis. We also recommend checking that the data collection software correctly stores data files on a secure study server (e.g., checking that input and output match) during internal testing, and then verifying that the process works smoothly during pilot testing. This will ensure that study data are correctly stored during actual online date collection.

There is a range of data collection software choices. Some software is free and easy to use, whereas other software requires purchasing a site license and is more challenging to use. Researchers wanting to conduct asynchronous individual interviews might be able to use a free email service by creating a project-only email account and encouraging study participants concerned about confidentiality to do the same. However, relying on this method creates challenges for data security because data are stored on the server of the email provider rather than a dedicated study server, and messages could be mistakenly sent to someone else. Instant messaging allows for synchronous interviews. Chat rooms and audio/video conferencing also allow for synchronous interviews and focus groups, but they have data security concerns similar to those of using an email service provider.

Researchers should consider differences in Internet connection speeds and variations in ability to export data easily into qualitative data analysis software. In our experience, it is best to purchase software that allows data to be stored on a dedicated project server to avoid data storage concerns and to protect participants’ confidentiality. Currently, we use Adobe Connect 9. In addition to resolving data security concerns and having the potential to facilitate visual data collection, this Web conferencing software allows us to customize the user interface and to incorporate polls and various media into our discussions.

Additional Data Collection Considerations

Online data collection is vulnerable to participant fraud, especially when the incentive for participating has monetary value. Currently, there is no way to verify the identities of online research participants, so researchers relying on self-reported data must be aware that participants could misrepresent themselves. This is particularly a concern in online qualitative research because many of the data collection methods rely on textual rather than visual modes of communication, additionally limiting the researchers’ ability to detect such misrepresentations. To minimize the risk of fraud, we recommend having a written data collection and participant verification protocol that includes a comparison of Internet protocol and email addresses, as well as relevant demographic information. Although the implementation of this protocol is more critical for online studies with larger sample sizes, we believe it is good practice to develop and implement the protocol for all online studies. All the research team should be trained on how to follow the data collection protocol, and all staff interactions with data should be recorded in a secure document.

Public Relations Considerations

Researchers collecting data on sensitive topics or from hard-to-reach populations have a responsibility to protect both the data and participants from harm. Often, this harm can come from persons with different ideological perspectives who have the ability to use media to misrepresent data or to stigmatize and exploit members of a hard-to-reach population. Because media can be used to pressure politicians or funders to enact legislation or policies that are hurtful to research agendas and members of hard-to-reach populations, we recommend having a comprehensive public relations (PR) protocol in place before data collection begins.

A PR protocol differs from the previously discussed communication protocol. Whereas the communication protocol is concerned with communication between researchers and participants, the PR protocol is concerned with communication between researchers and persons external to the project. The purpose of the PR protocol is to plan for releases of information and for external inquiries. A PR protocol should include a list of key personnel allowed to respond to external requests for information, key messages, and guidelines for using social networking media—both study-related pages and personal pages of research team members. Within the protocol, it is also important to identify likely questions and the institution’s response to each possible inquiry. Members of an institution’s PR department should be included in the development of this protocol to ensure compliance with institutional policies. For more information on how we use our PR protocol, please see Rosser, Kilian, and West (2013).

Concluding Remarks

In this article, we discuss many of the considerations associated with online qualitative data collection, and offer recommendations based on the literature and our lengthy experience conducting studies in an online environment. We also highlight the potential of using Internet-based research methods to access hard-to-reach populations. The Internet has provided opportunities for members of hard-to-reach and geographically dispersed populations to find each other and to organize on the basis of common identities, experiences, or practices. Web sites, blogs, and message boards catering to small populations, such as MSM, transgender persons, and polyamorous parents, are common.

Researchers working with hard-to-reach populations must be mindful of the impact their presence as researchers has on the community’s group dynamics. They should also keep in mind their impact on the personal safety of persons using technology to connect with others like themselves. In this article, we did not explicitly discuss study ethics, but they are critical for conducting online qualitative research with hard-to-reach populations. Researchers using technology to access hard-to-reach populations have a responsibility to respect people’s identities, protect study participants and, when possible, use research to reduce stigma and promote the health and well-being of persons in the populations of interest.

Researchers new to online qualitative data collection can use this article as a starting point in their learning process, and to help them establish their own best practices for conducting online qualitative research. As new technology becomes available, there will be opportunities to experiment with new qualitative data collection techniques, such as the visual methods briefly mentioned in this article. When learning to apply new techniques, we encourage researchers to ask themselves how the application should occur in light of current knowledge about qualitative data collection and analysis. They should also ask themselves what protocols are necessary to protect the collected data, the study participants, and the population from which participants were recruited.

We encourage researchers to view online qualitative data collection as an extension of current best practice in the field of qualitative research, while considering the unique challenges and opportunities offered by online environments. In this article, we describe many of the characteristics that distinguish online qualitative research from offline qualitative research. However, regardless of whether it is conducted online or offline, outstanding qualitative research is based on the same principles that have guided the field for decades. The application of online techniques is simply another way to understand the world in which we live, a world in which technology is increasingly becoming an integral part of everyday life. With this article, researchers have access to best-practice guidelines for integrating technology into their research practices.

Biographies

J. Michael Wilkerson, PhD, MPH, is an assistant professor in the Division of Health Promotion and Behavioral Sciences at The University of Texas Health Science Center at Houston (UTHealth) School of Public Health, Houston, Texas, USA.

Alex Iantaffi, PhD, LMFT, is an assistant professor in the Department of Family Medicine and Community Health at the University of Minnesota Medical School, Minneapolis, Minnesota, USA.

Jeremy A. Grey, PhD, is a postdoctoral fellow in the Department of Epidemiology at Emory University Rollins School of Public Health, Atlanta, Georgia, USA.

Walter O. Bockting, PhD, LP, is a clinical psychologist and co-director of the Initiative for LGBT Health in the Division of Gender, Sexuality, and Health at the New York State Psychiatric Institute and the Columbia University School of Nursing, New York, New York, USA.

B. R. Simon Rosser, PhD, MPH, LP, is a professor and co-director of HIV/STI Intervention and Prevention Studies in the Division of Epidemiology and Community Health at the University of Minnesota School of Public Health, Minneapolis, Minnesota, USA.

Appendix A. Decision-making Checklist for Online Qualitative Data Collection

Section 1 Directions: Answer the following questions to decide between on online or offline study design. If you respond “Yes” to most items, consider online data collection.
	Online	Offline
Administrative Considerations
Can staff transfer offline qualitative data collection skills to an online environment with minimal training?	Yes	No
Is there money in the budget to cover the costs of online qualitative data collection software?	Yes	No
Is there money in the budget to cover the costs of online recruitment?	Yes	No
Does the budget limit the ability to pay a transcriptionist to produce written transcripts of audio or video recordings?	Yes	No
Are members of the research team trained to code disjointed text transcripts, audio, video, or other visual files collected online from study participants?	Yes	No
Population Considerations
Are members of your population able to use the technology required to participate in your online study with minimal training?	Yes	No
Do members of your population have access to the technology required to participate in an online study, including high-speed Internet access?	Yes	No
If members of your population will be dependent on community spaces, e.g., libraries or Internet cafés, to participate in an online study, are you confident the location will not limit the time they can participate or bias their responses?	Yes	No
Do members of your population have the literacy to participate in online text-based data collection, e.g., message boards or chat rooms?	Yes	No
Is it important to have participants from geographically diverse locations?	Yes	No
Data Collection Considerations
Compared with offline data collection, will online data collection increase the confidentiality of participants?	Yes	No
Is it desirable for your study participants to have greater perceived anonymity?	Yes	No
Is it desirable for your study participants to have greater perceived homogeneity?	Yes	No
If collecting data online, do you have access to strong data security systems, e.g., a dedicated study server and data encryption?	Yes	No
If collecting data online, is there technical support for members of the research team?	Yes	No
If collecting data online, is there technical support for participants?	Yes	No
Are there significant barriers to securing participant transportation or a physical space if data were collected offline?	Yes	No
If collecting data online, are you confident that members of the research team will be able to build rapport with study participants?	Yes	No
If collecting data online, does your qualitative data collection software allow participants to record emotional reactions, e.g., emoticons?	Yes	No
If the viewing of media is critical to your study, will the qualitative data collection software you are using support the media, e.g., hyperlinks, photographs, videos?	Yes	No
Does the study timeline prohibit delayed access to data while it is being transcribed?	Yes	No
Is it important to have every comment date and time stamped and linked to a participant identification number?	Yes	No
Are there significant barriers to securing participant transportation or a physical space if data were collected offline?	Yes	No

Section 2 Directions: If you decide to collect data online, answer the following questions to help you determine whether you should use individual in-depth interviews, focus groups, or both. If you respond “Yes” to all items, consider collecting data using individual in-depth interviews. Conversely, if you respond “No” to all items, consider collecting data using focus groups. If your responses are in both columns, consider whether it would be beneficial to use both data collection methods.
	Interviews	Focus Groups
Are you more interested in gaining an in-depth understanding rather than a breadth of information about your topic?	Yes	No
Do you want personal narratives (stories) from participants about their experiences with your topic?	Yes	No
Are you asking questions that could harm an individual if their responses were disclosed to a group of people?	Yes	No
Are you asking questions that are likely to be perceived as highly personal or sensitive, raising questions about trustworthiness of data if collected in a group situation?	Yes	No
Are participants being recruited into the study to capture opposing opinions or experiences?	Yes	No

Section 3 Directions: If you decide to collect your online data using focus groups, answer the following questions to help you determine whether you should use synchronous or asynchronous data collection, or both. If you respond “Yes” all items, consider using synchronous data collection. Conversely, if you respond “No” to all items, consider using asynchronous data collection. If your responses are in both columns, consider whether it would be beneficial to use both data collection methods.
	Asynchronous	Synchronous
Are you expecting to have participants who lack experience communicating with other people in an online chat room environment?	Yes	No
Is participants’ reflexivity important to you?	Yes	No
Are participants more likely to be engaged if they are allowed to participate on their own time?	Yes	No
Are you willing to sacrifice the spontaneity of an interactive conversation to allow for reflexivity or to accommodate participants’ schedules?	Yes	No
Does your retention plan allow you to contact participants who do not log into an asynchronous data collection tool within an agreed on period of time?	Yes	No
Are you confident that if your retention plan is implemented appropriately, participant dropout will be minimal?	Yes	No
Is the use of the constant-comparative method critical to your study design?	Yes	No

Section 4 Directions: After determining whether to use synchronous or asynchronous data collection, or both, answer the following questions to determine if you should use text-only or incorporate visual data collection techniques. If you respond “Yes” to some items, consider using text and visual data collection techniques. If you respond “No” to all items, consider using text-only data collection techniques.
	Text & Visual	Text Only
Does the software you plan to use allow for a virtual whiteboard?	Yes	No
Are you collecting chronological data, e.g., a life history or event diary that could be improved with the use of Rivers of Experience, Timeline Follow Back, or a similar technique?	Yes	No
Are you asking participants to engage in a comparative analysis or other highly cognitive activity that could be improved by using a comparative line or similar visual interviewing technique?	Yes	No
Are you asking participants to upload and discuss archival items, documents, or photos, e.g., Photovoice?	Yes	No

Appendix B. Decision-making Checklist for Deciding When to Begin Online Data Collection

Directions: Answer the following questions to decide if you are ready to begin data collection. If you respond “Yes” to all items, you are ready to begin collecting data. However, if you respond “No” to any item, you are not ready to begin collecting data.
	Begin	Wait
Human Subjects Considerations
Has the final versions of your advertising, consent forms, and research questions been approved by your institution’s Institutional Review Board?	Yes	No
Do you have a written protocol for addressing risks to study participants that has been approved by your institution’s Institutional Review Board?	Yes	No
Does your protocol include guidelines for logging and addressing participants’ concerns, including how this information will be communicated to your institution’s Institutional Review Board and to your funder?	Yes	No
Does your protocol identify conditions and procedures for stopping data collection if there appears to be harm to participants?	Yes	No
Does your protocol have guidelines for restarting a study if the issue(s) that causes data collection to stop are resolved?	Yes	No
Are all databases and logs ready to be activated?	Yes	No
Do you have a current referral list applicable to participants in all geographical areas from which your sample is drawn?	Yes	No
Have all members of your research team completed human subjects training, per requirements of your institution’s Institutional Review Board?	Yes	No
If funded by NIH, did you obtain a Certificate of Confidentiality?	Yes	No
Recruitment Considerations
Do you have a signed contract with the vendor(s) you will rely on to publish your recruitment banner advertisements to send out emails?	Yes	No
Do you have written documentation from the vendor(s) that your recruitment plan, advertisements, and email content developed by your research team conforms to the vendor(s) marketing and advertising specifications?	Yes	No
Do you and your vendor(s) have a written plan to monitor participant recruitment, as well as a written contingency recruitment plan if recruitment is slower than desired?	Yes	No
Do you have adequate money in your budget to cover the costs associated with recruitment, including an estimated 50% cost overrun contingency budget if recruitment takes longer than expected?	Yes	No
Do you have a written protocol for ending recruitment, including how to remove advertising, respond to inquiries from persons interested in participating in the study, block access to online instruments, and lock the database?	Yes	No
Do you have a written protocol for processing participant payments?	Yes	No
Research Instrumentation Considerations
Have you obtained written sign-off from the Principal Investigator, Co-Investigators, and Methodologist for all instruments?	Yes	No
Have you obtained written sign-off from members of your Community Advisory Board (if applicable)?	Yes	No
Have online data collection instruments been tested in-house and all identified “bugs” resolved?	Yes	No
Are all variables names and codes set up to facilitate easy data cleaning and analysis?	Yes	No
Have you verified that online data are being correctly stored in the study database and on the designated server?	Yes	No
Have you completed pilot testing?	Yes	No
Data Collection and Participant Safety Considerations
Do you have a written data collection protocol, including procedures for identifying and removing duplicate participants from the study?	Yes	No
Have all members of the research team been trained on the data collection protocol?	Yes	No
Public Relations Considerations
Do you have a written public relations protocol that includes a list of key personnel allowed to interact with media, key messages, and guidelines for using online social networking media (both study-related pages and personal pages of members of the research team)?	Yes	No
Have all key personnel received a copy of the emergency communication protocol?	Yes	No