Conducting Internet Research With the Transgender Population: Reaching Broad Samples and Collecting Valid Data

Abstract

Health research on transgender people has been hampered by the challenges inherent in studying a hard-to-reach, relatively small, and geographically dispersed population. The Internet has the potential to facilitate access to transgender samples large enough to permit examination of the diversity and syndemic health disparities found among this population. In this article, we describe the experiences of a team of investigators using the Internet to study HIV risk behaviors of transgender people in the United States. We developed an online instrument, recruited participants exclusively via websites frequented by members of the target population, and collected data using online quantitative survey and qualitative synchronous and asynchronous interview methods. Our experiences indicate that the Internet environment presents the investigator with some unique challenges and that commonly expressed criticisms about Internet research (e.g., lack of generalizable samples, invalid study participants, and multiple participation by the same subject) can be overcome with careful method design, usability testing, and pilot testing. The importance of both usability and pilot testing are described with respect to participant engagement and retention and the quality of data obtained online.

Introduction

Since its inception, the Internet has had the potential for changing the way survey research is conducted. Especially when studying small, stigmatized populations, such as transgender people (individuals whose gender identity and/or expression varies from their birth sex), the Internet provides the opportunity for researchers to reach large, diverse, nonclinical samples (Mathy, Schillace, Coleman, & Barquist, 2002; Pequegnat et al., 2007). Using Internet technology to conduct psychological studies has made research easier by decreasing the cost of recruiting large, diverse, or specialized samples, giving researchers access to a wide range of social behaviors, and allowing for automation and experimental control. In particular, this enhances the ability of researchers to develop questionnaires that are more flexible while avoiding the traditional errors of data transcription and entry (Kraut et al., 2004). Along with these advantages, there are also significant challenges to Internet research. These include sampling bias (e.g., in spite of commercially available sampling frames, for many populations of interest, we are unable to draw a random sample of Internet users), having a lack of control over the data collection setting (website), and a number of challenges specific to the protection of human subjects (e.g., the inability to enforce exclusion/inclusion criteria and having cumbersome procedures for answering respondents’ questions; Kraut et al., 2004).

Sexuality researchers, especially those studying small, stigmatized communities like the transgender population, can benefit from the use of the Internet for recruitment and data collection. Studies using the Internet for data collection have been able to recruit samples that are four times as large as those obtained through in-person recruitment (Buchanan & Smith, 1999) and have found that data collection via the Internet, per unit time, is much more efficient and cost effective than mailed, telephone, or in-person surveys (Mehta & Sivadas, 1995; Pequegnat et al., 2007).

A major problem with Internet-based research, however, especially when all recruitment is done online, is that the samples generated are likely to be biased in a number of ways. First, one can only sample those who have access to the Internet.While Internet users are a growing segment of the population (Mathy, Kerr, & Haydin, 2003), they are still not representative of the population as a whole (USC Center for Digital Future, 2007; U.S. Department of Commerce, 2004). Also, while there are commercially available sampling frames that may be useful for certain types of marketing research and other general population studies, for studies of sexual minorities one cannot draw a random sample. There is no census from which to draw, no available sampling frames, and it is difficult to calculate the number of real, unique people active on an Internet site during a specific period of time (Pequegnat et al., 2007). So, by definition, any sample recruited via the Internet is a convenience sample made up of those who chose to link to the study website and complete study instruments.

While this sampling issue is problematic for large-scale population-based studies that can be conducted as effectively off-line, this may be less of an issue for studies of small, hard-to-reach, and geographically dispersed populations. For example, when studying transgender people, a large online convenience sample is likely a step forward from what has been feasible thus far. Traditionally, most studies of transgender people have used clinical samples (e.g., Pfäfflin & Junge, 1998 or samples recruited from social support organizations (e.g., Docter, 1988). More recently, particularly in HIV prevention research, studies have focused on inner-city transgender sex workers or clients from HIV/AIDS or social service agencies (Bockting & Avery, 2005; Herbst et al., 2008; Nemoto, Operario, Keatley, Han, & Soma, 2004). These samples may provide a biased picture of the sexual behavior of this population, since they comprise those most vulnerable to HIV infection and transmission (Operario, Soma, & Underhill, 2008). In contrast, the Internet provides the opportunity to conduct research on a broader segment of transgender people, particularly given the popularity of the Internet among members of this stigmatized, diverse, and geographically dispersed minority population (Horvath, Iantaffi, Grey, & Bockting, 2011).

In this article, we will describe some of the issues encountered when using web-based survey and chat technology to conduct an Internet-based study of the U.S. transgender population. The results of this study are being published elsewhere. Our experience indicates that research on the Internet is not as simple as it initially appears. In fact, Internet research involves the consideration of a number of issues both common to other forms of sampling and unique to the Internet environment. That is, in order to gain many of the advantages of using Internet-based data collection, one must anticipate issues with the performance of data collection instruments and procedures. Thus, the team must think about the process as much from a web design as an experimental design perspective. The purpose of this article is to discuss the process of study implementation and the associated methodological challenges and solutions.

Implementation of a Valid, Reliable Internet Survey

Researchers have described the cost advantages of conducting questionnaire and survey data collection on the Internet and many have rushed to post their data collection instruments on websites. We, however, found that using the Internet to collect valid data is not simply a process of posting an .html version of a paper-and-pencil survey on a website. In fact, in order to maximize respondent recruitment and retention, careful usability and pilot testing is essential. In some ways, conducting a survey on the Internet is analogous to conducting a survey on a street corner on Fifth Avenue in New York City during the morning rush. You are attempting to attract people who are rushing to get somewhere and do something important to them and get them to stop, postpone their activity, and complete your survey in a thoughtful, careful manner.

Usability Testing

Usability testing is considered a very important part of the development of web applications and other software that presents graphical user interfaces. It was particularly important for this study because of the sensitivity of certain segments of the transgender community to nuances of language and the complexity involved in measuring sexual behavior in a population at various stages of sex reassignment. We carried out extensive usability testing of the study website as part of the developmental process. The main purpose of this testing was to get an idea about the time required to complete the survey, locate confusing questions and website bugs, and to identify hard-to-navigate points and ambiguous instructions in the survey. The usability-testing team included several of the investigators, the project coordinator, process coordinators, and the software development team.

The testing effort included direct observation by a member of the software development team who accompanied the user in the testing room; cameras were set up in the testing room to record the user’s reactions as he or she went through the survey while thinking aloud. In the adjoining room, other members of the software development and research team viewed the user’s navigation of the interactive website and their verbal and nonverbal reactions on a video capture (picture in picture). Notes were taken in both rooms of the users’ reactions and sticking points in the survey. Finally, an exit interview was conducted in which users were asked about their experience with the survey, including any difficulties encountered, level of comfort with providing the requested information, and suggestions for improvement.

The users who answered the web survey as part of this testing effort were members of a national transgender community advisory board (N = 10) established by our project to ensure community involvement in all aspects of our study. We found through usability testing that people were quite unforgiving of the computer and seemed more frustrated and annoyed by issues they experienced online compared to paper-and-pencil surveys or face-to-face interview settings. Findings indicated the need to reduce complexity of enrollment (screening, consent, and registration), improve clarity of directions (item and scale instructions, user support), accommodate greater gender diversity, and reduce survey length. Detailed analysis of the usability testing videos indicated a need to substantially redesign sections of the web survey. Bugs were found that made navigation difficult, there were errors related to conditional branching which annoyed participants because they were asked to respond to questions they felt did not pertain to them, and results indicated that some of the instructions were confusing or ambiguous. A data integrity check that ensured that data were downloaded from the web survey so that survey responses were recorded in the correct data fields was also done at this stage. This data integrity check was repeated during the pilot testing.

Pilot Testing

Following the extensive modifications to the survey as a result of what was learned during usability testing, the next step in study development was to conduct a pilot test of all study procedures. The goal of the main study was to enroll 600 transgender adults across the United States recruited through banners and links embedded in messages on transgender community websites. In addition, a randomly selected subsample of 60 (10%) participants were to be interviewed using a combination of asynchronous (private bulletin board) and synchronous (instant messaging/real-time chat) computer-mediated communication.

We conducted an online pilot to test all aspects of the study, including recruitment, data collection, validation of data, and instrumentation (psychometrics of the measures). For this pilot test, 102 participants answered the web survey and time-based batch sampling was used to randomly select online interview participants. Except for the smaller number of participants, the online pilot testing process was designed to be an exact copy of the process followed during the final survey and interviews. The pilot test also involved asking 20 participants to return 1 week later to complete the survey again so that test–retest reliability could be calculated. The pilot data included hits received on the survey website from various sites used for advertising, the average time taken by a participant to complete the survey, the survey completion rate, and the response rate when participants were contacted to participate in the online interviews.

Selected results from pilot testing

Like the usability test, the online pilot test helped us identify problems and characteristics of the survey and interview procedures that we would have missed without this step. Of particular interest was the time frame for asking critical sexual behavior questions. That is, the major criterion variable for our study was the number of unprotected sexual acts during a specified time frame. We wanted to know how many times respondents had unprotected anal/vaginal sex with a primary partner and the number of times they had unprotected anal/vaginal sex with other partners. Given our population—transgender individuals at various stages of gender role transition and body modification—this was a complex issue, requiring a considerable expansion of the Risk Behavior Assessment (National Institute on Drug Abuse [NIDA], 1993). While the usability testing identified some of branching problems, we found further branching problems in the pilot test (i.e., individuals who were asked to respond to questions about sexual behaviors they could not perform because of the genital status of themselves and their partners). Also, we found through test-retest analyses that the number of sexual partners over the last 3 months (r_xx =.91) showed better temporal stability than the number of sexual partners over the last 30 days (r_xx = .59). However, the test-retest reliability for the frequency of condom use was essentially the same, whether measured over the last 3 months (r_xx = .81) or last 30 days (r_xx = .86). The questions about condom use with other partners for penis/anus or penis/vagina sex showed that respondents were more consistent over 30 days (r_xx = .99) than over the last 3 months (r_xx = .23). This complex pattern of responses resulted in a decision to retain both the 30-day and the 3-month time frame for questions about sexual behavior.

Additionally, we found that the asynchronous method of qualitative data collection resulted in a higher response rate (62%) and more detailed narratives compared to the synchronous interview (52%); the synchronous method was helpful, however, to probe asynchronous responses. Both of these response rates were lower than those of the survey, and we encountered a number of problems with software incompatibility and connectivity. These problems would have been identified earlier if we had conducted a usability test for the qualitative instruments and site, as was done for the survey.

Success of Survey in Reaching Desired Population

The aim of our study was to investigate the attitudes, behaviors, and HIV risks of a cross-section of the transgender community, a community that is ill defined and hard-to-reach. We used a targeted recruitment strategy, where we first identified websites that catered to specific transgender subgroups (i.e., male-to-female and female-to-male transsexuals, cross-dressers, drag queens/kings and female/male impersonators, and others, such as bigender and gender queer persons) as well as websites that are frequented by transgender persons across these subgroups. Our goal was not a representative sample of the transgender population but equal representation across subgroups. We wanted to reach a much broader cross-section of the transgender community than has previously been studied and to quantitatively explore differences across the various subgroups that make up the transgender community. Hence the relative sample sizes of the various subgroups in our study cannot be considered representative of the population distribution.

In general, we were successful in recruiting large numbers of individuals who identified themselves as male-to-female and female-to-male transsexuals (23%and 21%of our total sample, respectively), male-to-female cross-dressers (15%), as well as others, such as bigender or gender queer persons (30%). However, our recruitment strategy was not particularly effective in identifying and recruiting drag queens/female impersonators (4%), drag kings/male impersonators (4%) or femaleto- male cross-dressers or transvestites (2%). This may be because these three subgroups are not always included in the definition of transgender; hence, they may not consider themselves to be part of the transgender community. Moreover, female-to-male cross-dressers and drag kings/male impersonators are thought to be a very small proportion of the transgender population, so it may be that our recruitment was in proportion to their representation in the community. The lack of any previous population-based research with the transgender community makes it impossible to determine which of the above interpretations are valid.

Indication of Sample Representativeness

Generalizability from Internet samples may be less of an issue for those who are dealing with small, select groups, such as the transgender population. In general, research with transgender individuals has lacked representativeness because of the limitations on subject availability (Rosser, Oakes, Bockting, Gabes, & Miner, 2007). Samples are relatively small, even in large population centers, thus generalizable samples can only be obtained through multisite sampling. This has been problematic not only because of the cost of multisite recruitment but also because many individuals are closeted and hence not available for recruitment. This has forced most researchers to recruit the visible population, not the population as a whole. Another reason that Internet recruitment of stigmatized, hidden populations like transgender people may result in sufficiently representative samples is that the Internet is thought to provide an environment that is sufficiently anonymous and available to allow the more closeted and geographically dispersed members of the population to interact with each other (Shapiro, 2004), making them available for recruitment. In fact, numerous studies have been conducted with sexual minorities using Internet recruitment and data collection (see Pequegnat et al., 2007).

Since this is the first study of transgender people to sample on a national level, there are no ‘‘off-line’’ data to help us understand whether our sample is representative of the transgender population as a whole. However,Rosser et al. (2007) provide some indication of the representativeness of this Internet sample through their social demographic analysis of our study sample. They found that participants represented 48 of the 50 United States, along with the District of Columbia, Puerto Rico, the Virgin Islands, and military zip code for Europe. The proportion of study participants from each jurisdiction approximated the relative population size of those jurisdictions, and only Montana and South Dakota were not represented. Our sample was younger and more likely to identify as White or multiracial than was the U.S. census population, which is consistent with other studies of Internet samples (Ross, Mansson, Daneback, Cooper, & Tikkanen, 2005). They were also more educated, had lower household income, and were less likely to currently be married (p. 55). It is interesting to note that the sample reached in this study was less likely than the U.S. population to live in major metropolitan areas (Rosser et al., 2007).

The results ofRosser et al. (2007) indicate that our sample approximates the population of the United States in terms of geographic distribution. There are, however, differences in important demographic characteristics that have been identified in other Internet studies, including Internet users being generally younger,more educated, and less likely to be persons of color (Ross et al., 2005). Contrary to other studies of Internet samples (USC Center for Digital Future, 2007; U.S. Department of Commerce, 2004), our sample had lower income than the U.S. Census sample. It is likely that this difference reflects a characteristic of the transgender population, who could be expected to be underemployed due to the stigma related to being transgender. In spite of the possible biases, using the Internet we were able to reach a more heterogeneous sample of transgender people than have been reached in previous studies (Bockting & Avery, 2005; Herbst et al., 2008; Nemoto et al., 2004; Pfäfflin & Jung, 1998). Our study appears to have oversampled individuals in nonurban settings, thus reflecting a population not currently available in the literature on transgender people. We have also reached a sample that is less likely to be engaged in sex work compared to other HIV prevention studies targeting the transgender population, with only 4% of our sample reporting having engaged in sex for payment in the last year. This is significant because a review of the available literature gives the impression that sex work is extremely prevalent among the transgender population, with prevalence rates ranging from 24% to 75% (Herbst et al., 2008). While transgender individuals are indeed vulnerable to sex work due to a number of reasons (e.g., survival or validation sex, see Bockting, Robinson,& Rosser, 1998 or Nuttbrock et al., 2009) sex work appears to be overrepresented in HIV prevention research using convenience sampling with visible subgroups of this population in inner cities. Thus, previous studies may have overestimated the HIV risk behavior in the transgender population and obscured subgroup differences in risk behavior. Our Internet-based study may provide a more accurate picture of HIV risk among the transgender population in general, allowing us to identify within the broader population-specific factors or subgroups at highest risk.

Dropouts and Frauds

Two issues that have been identified as possible problems with Internet studies are that the dropout rate may be much higher than for other survey methods (Birnbaum, 2004) and that, especially when incentives are offered, individuals may enter the study fraudulently (Smith & Leigh 1997; Kraut et al., 2004). That is, they may not really meet study selection criteria, or they may complete the survey more than once (i.e., how do you know whether you have 1,000 subjects or 1 subject 1,000 times?).

Of the 1,373 individuals who answered the first question on our survey, 1,273 (93%) completed enough of the survey to provide useful data, including our main dependent variable of sexual risk behavior, and thus were considered ‘‘completers.’’ We then eliminated another 44 respondents because of irregularities in their surveys (criteria discussed below), leaving the final valid sample of 1,229, or 89.5% of those who began the survey. This completion rate appears to compare favorably with the completion rates of other survey methodologies (Denscombe, 2006; Hox and De Leeuw, 1994). However, this completion rate does not reflect the proportion of the available population that we were able to attract to the study. This could be estimated using the ‘‘click-through rate,’’ defined as the number of individuals who clicked on an advertisement and accessed the web-page for the study divided by the total number of times advertisements could have been viewed (Pequegnat et al., 2007, p. 512). However, this also has limitations, in that the click-through rate does not reflect the number of potential subjects who viewed the advertisements, only the number of times the advertisement was available. To follow-up on our previous analogy, a true response rate cannot be calculated because we do not know how many potential subjects rushed past us on 5th Avenue, failing to stop and engage our interview.

Researchers have been justifiably concerned about the rate of fraudulent respondents to online surveys (Kraut et al., 2004; Murray & Fisher, 2002; Smith & Leigh, 1997). In response, we developed a detailed, automated system for detecting suspicious surveys, which were then reviewed manually. This computerized de-duplication, cross-validation protocol checked each participant’s e-mail and IP address, user name, password, date of birth and age, zip code, and completion time with other participants’ responses, to identify participants who may have participated more than once or who provided false or unreliable data. The computerized protocol grouped individuals into four categories: certain fraud, highly suspicious, suspicious, and valid based on the number of failed checks of the above factors. The computer-generated highly suspicious and suspicious lists were then checked manually for further indications that they were either duplicates or invalid. This check prioritized duplicate IP addresses, zip codes, and passwords, as well as two-time variables: taking less than 30 min for the entire survey, or completing the three most important subsections of the survey in less than 19 min. Finally, the database was checked for response sets, such as sticky fingers (always the same response) or patterns of responding (e.g., 1,2,3,4, 1,2,3,4). Using this protocol, 44 surveys were identified as suspicious and excluded. This is actually only a very small proportion of the completed surveys (3.5%) and probably would not have changed our results had they not been detected.

Collecting Qualitative Data

Interactive technology provides the capacity for adding a qualitative component to a survey such as the one implemented here. Online interviews have the advantage of resulting in an immediate transcript saving time, effort, and expense associated with audio recorded transcription standard in face-to-face or phone-based interviews. Our pilot data indicated it would be difficult to recruit participants, in that although 8 of 10 of those invited to participate in an asynchronous private bulletin board did so, only 40% of those invited to participate in a synchronous chat were successfully interviewed. Thus, when the survey went live, we used a batched, random sampling procedure with replacement, so as to maximize our qualitative sample size. Our response rates were substantially the same as for the pilot study, with slightly fewer, about two thirds of those invited, participating in the asynchronous bulletin board and half of those invited to participate in synchronous chat successfully interviewed. In general, we found that the responses to online qualitative procedures were shorter and more parsimonious than that usually gained in qualitative face-to-face or phone interviews. The asynchronous method resulted in greater narratives and richer information than the synchronous real-time private chat, as the former allowed participants to reflect and type at their own pace. Thus, while the information was easier to process and we were able to reach a more heterogeneous sample than previous qualitative researchers, the qualitative data was not as rich has we had hoped.

Conclusions

The widespread use of the World Wide Web and other Internet-related technology has provided the opportunity for researchers to conduct more sophisticated research with small, geographically dispersed and hard-to-reach populations. In general, we found that implementation of a well-designed web-based study, which included both quantitative survey and qualitative interview procedures, is not an easy task. In addition to the usual research design and psychometric concerns, the unique environment of the Internet and the requirements of technology require the investigator to confront a variety of other methodological issues. Our experience indicates that problems in study validity and integrity are not inherent in the Internet medium but are more likely the result of failure to conduct the appropriate preimplementation testing. Internet methods provide multiple advantages, including the ability to reach large numbers of study subjects, to individually tailor survey instruments to the respondent, and the possibility of conducting interactive discussions with individuals in remote locations. However, this is only the case if the procedures attract participants, retain them in the study, provide valid data, and identify respondents participating multiple times (especially when incentives are offered) or respondents who do not meet inclusion criteria.

Our experience with this project indicates that usability testing and pilot testing are critical phases of Internet study design. During our usability testing, we identified a number of aspects of our website that frustrated and confused participants. These were corrected prior to pilot testing, where additional problems were identified. Had our survey gone live before these two phases, we would have had major problems retaining participants, because they would have gotten frustrated with the performance of the study site, would have been asked to answer questions that did not pertain to them, and would have been confused by many of the questions asked in the survey. Additionally, through pilot testing, we were able to determine the time frames we needed to use for our most important variables—those involving sexual activity and condom use—and we identified incompatibility problems with our online interviewing technology. Had we not done the pilot phase, we would have collected data of questionable reliability and would not have realized the need for sampling with replacement in the qualitative aspect of our study until well into data collection, which would have severely compromised our study.

We did not encounter problems in interacting with our Institutional Review Board (IRB). Part of the reason is that our study was not the first at our university to use an Internet-based survey to conduct a study of HIV risk behavior. Thus, our IRB had wrestled with many of the issues that have been raised by others (Kraut et al., 2004) as barriers to use of the Internet. Additionally, we maintained ongoing communication with our IRB, providing them with materials as they were developed and informing them of even small changes in procedures. We also took great care in ensuring that each step of data acquisition, data transfer, and data storage was secure and we described our data security measures in detail.

In summary, Internet technology allows researchers to obtain data on small, hard-to-reach populations in large enough numbers to make reliable estimates of behavior and explore important relationships. Although the information obtained from online chat interviews is not as rich as that obtained face-to-face, via the phone, or other conferencing technology, it allows researchers to access populations that are unavailable to them without such technology. Our data indicate that we were able to reach a broad sample of transgender individuals at various stages of their transgender experience, with different identities, and who were not necessarily involved in the mental health system or sex work (Rosser et al., 2007). These factors likely contribute to the very different risk profile that we found as compared to previous studies of this population and have likely allowed for a better estimate of the HIV risk and the risk factors for such behavior present in the transgender population as a whole. Our experience indicates that researchers attempting Internet-based studies must consider website development issues, as well as research methodology, since both are intertwined.

Biographies

Michael H. Miner, PhD, is a professor of family medicine and community health in the Program in Human Sexuality at the University of Minnesota,, Minneapolis, Minnesota (miner001@umn.edu).

Walter O. Bockting, PhD, is an associate professor of family medicine and community health in the Program in Human Sexuality at the University of Minnesota. Dr. Bockting is also a coordinator of transgender services at the Program in Human Sexuality (bockt001@umn.edu).

Rebecca Swinburne Romine, PhD, is a research associate at the University of Minnesota Medical School, Program in Human Sexuality. Her research interests focus on the areas of juvenile delinquency, public health, and criminal justice, with a particular interest in adapting a public health approach to the prevention of adolescent criminal behavior, particularly sexual offending behavior (rebswinburne@gmail.com).

Sivakumaran Raman is a physician informaticist who received his degree in medicine from the University of Delhi, India, and his master’s of science in health informatics from the University of Minnesota. He is currently with the Enterprise Clinical Alignment Group in UnitedHealth Group corporate and is responsible for Clinical System Integration and Analytics. He is also an adjunct assistant professor in the Institute for Health Informatics at the University of Minnesota (rama0083@gmail.com).