The Importance of Non-Probability Samples in Minority Health Research: Lessons Learned from Studies of Transgender and Gender Diverse Mental Health

Abstract

Non-probability sampling methods utilize nonrandom research participant selection, which may generate study samples that are not representative of the general population. Non-probability sample studies are often regarded as inferior due to uncertainty about their generalizability and external validity. In reality, non-probability sampling offers advantages that make this method particularly valuable for minority health research. In this perspective article, we review the strengths and limitations of probability and non-probability samples, examining three landmark survey studies used to study transgender and gender diverse mental health. We conclude that both types of studies provide important and actionable data about mental health inequities experienced by minority populations.

Introduction

In an ideal world, we would sample every member of a population of interest when conducting epidemiological research. Practically, of course, this is rarely feasible. We instead define a population of interest, then design studies that sample a subset of this population to make inferences about the broader population of interest. For this article, we will focus on transgender and gender diverse (TGD) people in the Unites States.

In probability sampling methods, each member of the population has a known and random chance of selection into the sample for research participant recruitment, with the goal of obtaining a study sample that closely approximates representation of the entire population.¹ An illustrative example of probability sampling is random digit dialing, in which investigators enumerate a list of telephone numbers, select telephone numbers at random, and call these numbers to recruit study participants.

Such sampling methods are ideal when, for example, researchers are attempting to estimate the population-wide prevalence of a measure of interest. Of note, probability samples are lacking in the field of TGD health research, with the Behavioral Risk Factor Surveillance System (BRFSS),² Youth Risk Behavior Surveillance System (YRBSS),³ and TransPop⁴ studies being among the very few probability samples to ask about gender identity.

By contrast, non-probability sampling involves recruitment that is not achieved through random selection.¹ When using non-probability sampling techniques, it is less likely the sample will be representative of the general population of interest. One example of non-probability sampling is convenience sampling, in which investigators recruit participants based on relative ease of access—for example, from a clinic that serves the population of interest—or conduct focused recruitment efforts to purposively sample minority communities.

Due to the potential for biased and less generalizable estimates from non-probability samples, it is generally presumed that studies based on such samples are inferior in quality to studies that employ probability sampling, as the latter are more likely to yield findings that are representative of the general population of interest. This assumption regarding research quality, however, is an oversimplification that does not hold true for all research objectives and questions, given that a range of other study design-related and methodological factors may also be important when assessing the quality of a research study, such as sample size or relevance to the population of interest.

Cross-sectional probability studies are particularly useful when determining the prevalence of a variable of interest within a priority population; this study design, however, is not well suited for facilitating causal inference. Randomized controlled trials, for instance, rarely comprise samples recruited via probability sampling, however these are generally considered the gold standard design for causal inference. Recently, some authors have inappropriately suggested that non-probability samples are not of value in the study of TGD health.⁵ We strongly disagree with this characterization and highlight that nearly all data about TGD people come from convenience and clinical samples, which have been key in advancing our understanding of this population.

In this perspective article, we characterize the value of non-probability samples for minority health studies, with TGD mental health research as an illustrative example. We discuss how these samples allow for the recruitment of large sample sizes of TGD people and the study of experiences and exposures that are of particular relevance to this population. We highlight three datasets as case examples of strengths and limitations of various sampling techniques (Table 1). We discuss two probability studies: The 2017 YRBSS³ and The TransPop Study,⁴ and one large non-probability study: the 2015 United States Transgender Survey (USTS).⁶ We selected these examples to underscore key points and they are not an exhaustive list of valuable datasets for studying TGD mental health.

Table 1.

Advantages and Disadvantages of Surveys Examining the Mental Health of Transgender People in the United States

Study name	No. of transgender participants	Description	Advantages	Disadvantages
2017 Youth Risk Behavior Survey	2,845	Probability sample of U.S. high school students in grades 9–12, with a focus on variables relevant to the general population	Large federally funded population survey yielded a relatively large sample size Representative sample of transgender youth within regions that opted into the gender identity module	Survey questions focused on the general population, with no questions specific to the experiences of transgender people
TransPop Study	274	Probability sample of transgender Americans, with a focus on variables relevant to the transgender community	Random digit dialing allowed for close approximation of a sample representative of the U.S. transgender population Survey questions focused on experiences relevant to the transgender community	Random digit dialing resulted in a relatively small sample size
U.S. Transgender Survey	27,716	Non-probability convenience sample of TGD adults in the United States, with a focus on variables relevant to TGD communities	Convenience sampling allowed for recruitment of a large sample size Survey questions focused on experiences and exposures relevant to TGD communities	Nonrepresentative sample, limited generalizability

TGD, transgender and gender diverse.

The 2017 YRBSS

The YRBSS is a probability survey conducted every 2 years by the U.S. Centers for Disease Control and Prevention.³ It is a national school-based survey intended to build a representative nationwide sample of high school students. The survey assesses a range of physical and mental health variables, as well as risk behaviors for adverse health outcomes. In addition to the national school-based survey, some U.S. states include additional modules to assess specific areas of interest.

In 2017, for the first time, the YRBSS provided an optional module to collect data on respondent gender identity. In the first year of this module's administration, 10 states and 9 urban school districts opted to use it. The 2017 YRBSS showed that of 118,803 high school students surveyed with this module, 2845 (1.8%) identified as transgender. Because the dataset includes various metrics regarding mental health, including suicidality, these data have been used to study mental health disparities between transgender and cisgender youth.⁷

Unfortunately, these results must be taken with caution, as the YRBSS did not use the two-step method of asking about gender identity, in which one asks first about sex assigned at birth and second about gender identity.⁸ In particular, the phrasing used in the YRBSS limits our ability to identify nonbinary youth.

Although probability samples such as the YRBSS provide important national health surveillance data, they present certain limitations for minority health research. TGD youth have higher rates of school dropout than cisgender youth, often attributed to bullying victimization, which could impact generalizability and introduce bias. Further, small sample sizes of TGD people that result from using general population studies to understand minority populations may limit statistical power to test empirical questions that are relevant for public health intervention.

The YRBSS was also not designed to study TGD health per se, thus it lacks questions specific to lived TGD experience (e.g., gender affirmation or gender identity conversion efforts). This limitation precludes studying certain nuanced and TGD-specific exposures, or drivers of adverse mental health outcomes among TGD people. Other surveys that were specifically designed for research with TGD populations, such as The TransPop Study and the 2015 U.S. Transgender Survey, examine a more comprehensive set of experiences that are specific to TGD communities.

The TransPop Study

The TransPop Study was the first national probability sample for which U.S. transgender adults were the population of interest.⁴ It is the second probability sample of U.S. transgender adults, after the BRFSS.² Investigators employed random digit dialing over a 1-year period to recruit a probability sample of transgender people in the United States. Each respondent was screened to assess transgender status, using an established two-step process that asks participants about their sex assigned at birth and their current gender identity.⁸

Participants whose self-reported gender identity differed from the sex they were assigned at birth, or who identified with the term transgender, were classified as transgender. Of note, participants were also screened for additional eligibility requirements: being 18 years of age or older, having an education above sixth grade, and being able to complete an interview in English—which limits the representativeness of the sample for the general U.S. population.

The research team screened 581,844 potential participants over 1 year. Due to the relatively low probability of each phone number belonging to a transgender person, only 1,114 (0.2%) screened eligible. Of these, 274 participants agreed to participate and completed the survey questionnaire. This participant yield illustrates a major limitation of probability sampling for minority populations: due to the relatively low probability of recruited participants belonging to the minority population of interest, the feasibility of achieving a sufficient sample size to power relevant analyses is sometimes limited.

Nevertheless, TransPop is an invaluable resource for understanding the prevalence and demographic distribution of transgender identification in the United States, as well as prevalent health conditions. Unlike probability-based studies of the general population, such as the 2017 YRBSS, TransPop was able to assess a range of exposure variables of particular relevance for TGD populations, including gender identity conversion efforts, gender affirmation, discrimination, and internalized transphobia, because it was specifically designed to investigate the health of TGD populations.

The 2015 USTS

The 2015 USTS was a non-probability, convenience sample survey that remains the largest survey ever conducted of TGD adults.⁶ The survey was developed and conducted over 1 year by the National Center for Transgender Equality through collaboration with researchers, advocates, members of TGD communities, and subject matter experts. The final questionnaire featured 324 possible questions across 32 domains, including health and health care access, employment, law enforcement, public accommodations, and discrimination. Recruitment was performed by community-based outreach. The final sample included responses from 27,715 TGD adults from all 50 states, the District of Columbia, several U.S. territories, and overseas U.S. military bases.

The USTS' non-probability sample offers several advantages over probability-based samples. The use of a convenience sample facilitated generation of the largest existing survey dataset regarding experiences of TGD people. Due to its large sample size, the USTS is uniquely well powered to identify mental health determinants among TGD adults. For this reason, the USTS was the basis for seminal studies on the mental health benefits of gender-affirming medical and surgical care.^9,10

Its large sample size has also allowed investigators to examine specific subpopulations of TGD people and intersectional influences on mental health, thereby advancing our understanding of racial disparities among TGD Black, Indigenous, and People of Color.¹¹ Though its non-probability design limits generalizability, particularly as the study participants were disproportionately white and highly educated, it has contributed key insights to the field of TGD mental health. Importantly, correction procedures exist to compensate for some biases related to non-probability sampling. For example, weighting procedures, applied to account for underrepresentation or overrepresentation of specific demographic groups, can facilitate analysis that more accurately reflects the general population's sociodemographic distribution.¹²

Conclusion

In this article, we reviewed some strengths and limitations of probability and non-probability samples through an examination of three landmark surveys used to study TGD mental health. Although non-probability samples are sometimes viewed as methodologically inferior to probability samples, particularly due to concerns regarding external validity and generalizability, these have great value in minority health research and several important advantages over probability sampling—including the ability to answer questions of particular importance to the minority population of interest and the acquisition of larger sample sizes. Although probability samples are useful in understanding, for example, the prevalence of TGD identities among the general population, they are limited in their ability to achieve large sample sizes.

Probability-based sampling of the general population to study minority health may also be limited by the lack of culturally tailored questions pertaining to the experiences of minority subpopulations. Non-probability surveys of minority subpopulations are particularly effective at generating these culturally specific data. This is especially salient in minority health research, where populations of interest may be smaller and more difficult to recruit in the absence of culturally responsive study outreach and community engagement strategies.

Probability-based sampling of minority communities ought to be a priority for all public health researchers and U.S. federal health surveillance systems, to identify, document, and monitor health disparities. Non-probability samples, however, must also be employed to fill a crucial gap in studying the health of specific minority populations, a critical step toward addressing minority health inequities.

In conclusion, both probability and non-probability sampling studies are needed—each captures valuable, complementary, and actionable scientific data to understand and improve the health and well-being of minority populations. Broad statements characterizing non-probability samples as lacking utility represent a misunderstanding of epidemiological methods for minority health research.⁵ Rather than focusing on the relative inferiority or superiority of either sampling methodology, research and public health communities stand to benefit from recognizing the merits and limitations of both.

Abbreviations Used

BRFSS

Behavioral Risk Factor Surveillance System

TGD

transgender and gender diverse

USTS

United States Transgender Survey

YRBSS

Youth Risk Behavior Surveillance System

Author Disclosure Statement

J.L.T. reports receiving textbook royalties from Springer Nature and expert witness payments from The American Civil Liberties Union and Lambda Legal. A.S.K. and S.L.R. report receiving textbook royalties from McGraw Hill.

Funding Information

Funders played no role in the writing of this article. Dr. Turban has received funding from The National Institute of Mental Health (MH094612), a fellowship from The Sorensen Foundation, and a pilot research award from The American Academy of Child & Adolescent Psychiatry and their Industry Sponsors (Arbor & Pfizer). Dr. Keuroghlian has received funding from The Health Resources and Services Administration Bureau of Primary Health Care (U30CS22742).

Cite this article as: Turban JL, Almazan AN, Reisner SL, Keuroghlian AS (2023) The importance of non-probability samples in minority health research: lessons learned from studies of transgender and gender diverse mental health, Transgender Health 8:4, 302–306, DOI: 10.1089/trgh.2021.0132.