Socially assigned gender nonconformity: A brief measure for use in surveillance and investigation of health disparities

Abstract

Discrimination and violence targeting people perceived as gender nonconforming have been linked to a range of negative health outcomes, and large-scale representative data are needed to begin population surveillance of associated health disparities. A brief self-report measure of gender expression as perceived by others was tested using cognitive interviewing methods in a diverse sample of 82 young adults aged 18–30 years, recruited from the New England region in the U.S. Results identified themes related to item clarity, gender expression variation, undesirability of highest or lowest ends of item range, and tension between self and others’ perceptions. The item performed as expected and is recommended for use on studies of health disparities, including statewide and national public health surveillance tools.

Introduction

Despite the worldwide shift towards greater acceptance of nontraditional gender roles (Seguino, 2007), individuals perceived to have nonconforming gender expression continue to be victimized and subjected to discrimination. Nonconforming gender expression, or conveying feelings of masculinity or femininity through one’s appearance and behavior in a way nonconcordant with the ways society has assigned to one’s biological sex (Grossman, D'Augelli, Salter, & Hubbard, 2005), has been linked to myriad negative outcomes, including verbal and physical victimization (Pilkington & D'Augelli, 1995), parental and peer rejection (Landolt, Bartholomew, Saffrey, Oram, & Perlman, 2004), childhood bullying (Friedman, Koeske, Silvestre, Korr, & Sites, 2006; Ploderl & Fartacek, 2009), suicidality (Fitzpatrick, Euton, Jones, & Schmidt, 2005), and substance abuse (Rosario, Schrimshaw, & Hunter, 2008). In spite of the demonstrated health sequelae associated with targeted abuse, no surveillance system exists to systematically investigate the relationship between gender expression and health. Representative data are necessary to monitor the effects of discrimination and reduce its prevalence, yet research in this area is hampered by the lack of validated brief measures of gender expression suitable for large-scale public health surveillance. The current study uses cognitive interviewing methods and thematic analysis to evaluate a brief self-report measure of gender expression in young adults representing a range of genders, sexual orientations, and race/ethnicities from the New England region in the U.S. Our goal was to develop a measure that is appropriate for use on instruments surveying a diverse population to inform how gender expression is related to health.

The notion that how one’s identity is perceived – or socially assigned – by others is an important health determinant has gained recognition in recent years. In 2001, the U.S. Centers for Disease Control and Prevention (CDC) Measures of Racism Working Group developed a six-item measure to determine socially assigned race for the Behavioral Risk Factor Surveillance System (BRFSS), an ongoing state-based health survey administered in the U.S. to individuals age 18 and older (Jones, Truman, Elam-Evans, Jones, Jones, Jiles et al., 2008). Development of this measure was motivated by the recognition that systemic discrimination is often rooted in unfair treatment of an individual based on instantaneous visual cues without any prior knowledge of that individual’s background or self-identity. Jones and colleagues examined socially assigned race and health status in the BRFSS and found that being perceived as White was linked with better health outcomes in individuals who self-identified as a racial/ethnic minority (Jones et al., 2008). Given that perceived gender nonconformity, like socially assigned race, is an external cue used to judge an individual, it follows that socially assigned gender expression could be similarly linked to health disparities.

The effects of discrimination and victimization based on socially assigned gender nonconformity can be severe and long-lasting. Verbal abuse towards children perceived as gender nonconforming can begin by age six years, if not earlier, while studies from the U.S. and the United Kingdom have found that victims report enduring psychological distress, suicidality, posttraumatic stress disorder, depression, anxiety, and physical health symptoms such as dizziness, headaches, or vomiting (Carbone, 2008; D'Augelli, Grossman, & Starks, 2006; Fitzpatrick et al., 2005; Gruber & Fineran, 2008; Hughes, Johnson, Wilsnack, & Szalacha, 2007; Rivers, 2004; Rosario, Schrimshaw, Hunter, & Levy-Warren, 2009). While studies with adults (Lippa, 2002; Skidmore, Linsenmeier, & Bailey, 2006) and youth (Blashill & Powlishta, 2009; D'Augelli, Grossman, & Starks, 2005) have found lesbian, gay, and bisexual (LGB) populations to be more gender nonconforming than heterosexuals, harassment and discrimination targeting nonconforming gender expression is not restricted to people with a minority sexual orientation. Heterosexuals may also be targets for bullying and verbal or physical abuse based on their gender expression (Horn 2007). Gender expression has been associated with acceptance by parents, peers, and society independent of sexual orientation, especially after middle school when standards of gender conforming behavior decrease in flexibility (Alfieri & Ruble, 1996; Ma’Ayan 2003). In one study in which U.S. high school students rated the acceptability of hypothetical peers displaying a range of sexual orientations and gendered behaviors, gender nonconforming students were ranked as less acceptable than conforming individuals regardless of sexual orientation (Horn, 2007). Elevated rates of victimization in LGB populations may be attributable in part to higher levels of gender nonconforming behavior compared to heterosexuals (Corliss, Cochran, & Mays, 2002; Rivers & Cowie, 2006; Saewyc, Skay, Richens, Reis, Poon, & Murphy, 2006), although there may be important differences in the ways that sexual minority men and women are affected by societal restrictions on and reactions to gender nonconformity. In one study of men and women with a minority sexual orientation, 42% of life events attributable to nonconforming gender expression involved either assault or threatened violence, (Gordon, 2007) and women were significantly more likely than men to report discrimination due to nonconforming gender expression (OR 4.14, 95% CI 2.34–7.35). In the family context, children perceived as gender nonconforming are targeted with verbal and physical abuse perpetrated by both parents and siblings (D'Augelli et al., 2005, 2006; Grossman et al., 2005; Pilkington & D'Augelli, 1995; Rosario et al., 2009).

One group at particular risk for discrimination and violence victimization is the transgender population, whose inner perception of being male or female is different from their birth sex. Transgender individuals may be targets of violent attacks and suffer acute psychological distress as a result of pervasive hostility toward appearance and behavior perceived as gender nonconforming (Clements-Nolle, Marx, & Katz, 2006; Corliss et al., 2007; Grossman et al., 2005; Stotzer, 2008). A recent review of violence against transgender individuals in the US found that rates of lifetime physical assault due to gender nonconformity ranged from 33–53%, with one study of transgender residents of Virginia found that 82% of respondents had been victimized more than once (Stotzer, 2009). The review also found that rates of sexual assault ranged from 10–69%; one study of transgender high school-aged youth reported that 86% of respondents experienced some type of sexual violence related to their gender identity (Stotzer, 2009). By comparison, rates of sexual assault victimization in general population samples of young adult women and men are estimated to be around 15–20% and 2–10%, respectively (Kessler, Sonnega, Bromet, Hughes, & Nelson, 1995). It has been proposed that male-to-female (MTF) transgender youth may also be victimized starting at a younger age than female-to-male (FTM) transgender youth, though systematic study is still needed (Grossman et al., 2005). Even with accumulating evidence of widespread victimization and its negative health effects, nonconforming gender expression and transgender identity have only begun to be incorporated into state antidiscrimination laws in the last 10 years in the United States (National Gay and Lesbian Task Force, 2008), and significant Federal legislation barring hate crimes because of the victim’s gender identity and sexual orientation has just recently been signed into law (Ward, 2009).

Systematic monitoring of health disparities, discrimination, violence victimization, and health in the U.S. is not currently possible because no large-scale statewide or nationally representative surveillance systems include measures of socially assigned gender nonconformity. A major barrier to this type of systematic surveillance has been the lack of brief validated measures of socially assigned gender nonconformity. In previous literature, gender expression has often been defined as whether an individual exhibits personality traits considered stereotypically masculine or feminine (Bem, 1974; Udry & Chantala, 2006). One well-known example of this approach is the Bem Sex Role Inventory (BSRI), where individuals mark on a scale of 1 to 7 how well certain traits describe them. These traits are either “masculine,” “feminine,” or “neutral,” for example, “aggressive,” “childlike,” or “helpful,” respectively. An individual’s total score indicates the degree to which she or he is masculine, feminine, androgynous (score high on masculine and feminine traits), or undifferentiated (score low on masculine and feminine traits). Bem’s original work in the early 1970s showed the 60-item BSRI had high test-retest reliability and seemed to measure an aspect of gender roles that was not measured by other scales available at the time, such as the masculinity-femininity scale of the California Psychological Inventory (Bem, 1974). The Personal Attitudes Questionnaire (PAQ) uses an approach similar to the BSRI with 55 “male valued,” “female valued,” and “sex specific” characteristics, which subjects use to first evaluate themselves and then evaluate the typical male or female (Spence, 1975). The Occupations, Activities, and Traits-Attitudes Measure (OAT-AM) advances this framework by including job types and hobbies that are considered stereotypically masculine or feminine (Liben & Bigler, 2002). This measure, which includes 75 items evenly divided between occupations, activities, and traits, displays good internal consistency and test-retest reliability, and has been used both for self-assessment and for rating the masculinity or femininity of others (Blashill & Powlishta, 2009). These measures have several limits. Personality traits are not usually immediately visible to others; thus, their connection to victimization may be distal. Cultural shifts in which specific traits and behaviors are considered “masculine” or “feminine” could also decrease the validity of measures based on historically dated norms. For the purposes of the current study, all three of these scales are too lengthy to be included in large-scale surveillance instruments where survey space is at a premium.

Another approach to measuring gender expression has been to use parental reports (Blakemore & Hill, 2008) or retrospective self-report measures using gender-typed childhood activities to define and measure nonconforming gender expression, such as a young boy playing with dolls or a girl playing football. One example of this type of measure is the 23-item Recalled Childhood Gender Identity/Gender Role Questionnaire (Zucker, Mitchell, Bradley, Tkachuk, Cantor, & Allin, 2006). For this type of measure, adults are asked to answer questions such as, “As a child, my favorite playmates were…” with response options ranging from “always boys” to “always girls.” Retrospective measures have been adapted for use with diverse samples, suggesting that they may be more flexible and culturally relevant than those relying on personality traits (Friedman et al., 2006). Although these measures do display expected differences between the sexes and across different sexual orientation groups, retrospective measures can also be subject to recall bias and may or may not be linked to gender expression in adolescence or adulthood. As with the BSRI and PAQ, these measures may also be too long for use on large-scale public health surveys that cover a wide variety of topics.

An approach used in LGB populations to measure gender expression is to ask whether respondents describe themselves as “butch,” “femme,” or “androgynous,” terms that have been used for decades to describe gender self-presentation, that is, masculine, feminine, or ambiguous, respectively (Hiestand et al., 2007; Levitt & Horne, 2002; Rosario et al., 2009). Though these terms are familiar to many individuals who identify with LGB communities, this terminology may be difficult to interpret for people with limited exposure to these groups. Even within the LGB community, definitions of these terms vary and individuals may ascribe to multiple identities or none at all (Levitt & Hiestand, 2004; Rosario et al., 2009). In sum, a brief, validated self-report measure of socially assigned gender expression is needed for use in public health surveillance systems with general population-based samples, which include people of all sexual orientations and gender identities, to assess how others regard a respondent’s current gender expression.

To develop survey items, cognitive interviewing techniques are often used to gather insight into participants’ understanding and interpretation of questions as well as the thought process behind each participant’s response (Presser, Couper, Lessler, Martin, Martin, Rothgeb et al., 2004; Willis, Royston, & Bercini, 1991). Cognitive interviews, where participants are asked to answer survey questions and then verbally describe how they decided to respond, have an important role in clarifying whether survey items capture the desired information (Beatty & Willis, 2007). Interviewers use probes to elucidate how participants interpreted each item and came up with their answers, any problems or difficulties participants may have had with item comprehension, and additional information that might clarify the background of a participant’s response, such as life experience. Cognitive interviewing methods employ a variety of scripted, semi-scripted, and improvisational methods to accomplish these goals (Beatty & Willis, 2007) and have been used to study survey items assessing sexual orientation (Austin et al., 2007), gender identity (Conron et al., 2009), and gender roles (Braun et al., 2008).

Clark and colleagues used the cognitive interview method to evaluate a series of brief self-report questions assessing different aspects of gender expression (Clark, Armstrong, & Bonacore, 2005). These researchers recruited a community sample of women between the ages of 40–75 years in order to document the experience of older, unmarried women getting screened for cancer. One item related to gender expression asked respondents how they would describe their own appearance on a seven-point Likert scale ranging from “Very masculine” to “Very feminine,” with the additional options “Not sure” or “None of the statements describe how you think of yourself.” Importantly, the research team found that when assessing factors associated with negative treatment in healthcare settings, participants ascribed more importance to how others perceived their appearance than to their own perceptions. Based on these results, the item was revised to ask specifically about how others perceived the respondent’s gender expression through her appearance.

The Current Study

In the current study, we used cognitive interview methods to build on the work of Clark et al. (2005) by adapting their measure of socially assigned gender expression through appearance and evaluating it in an adolescent and young adult population. While Clark et al.’s measure was developed for use with middle-aged and older women, we aimed to identify a measure of socially assigned gender expression that could also be used with youth and people of all genders and sexual orientations. There is some evidence that gender-nonconforming mannerisms, another aspect of gender expression that can be readily observed by others, are associated with lower peer acceptability independent of appearance in adolescents (Horn, 2007). Therefore, we also evaluated a new item we developed asking participants how others might describe their mannerisms. Using cognitive interview techniques allowed us to understand how participants interpreted the new items as well as the thought processes behind each response (Bradburn & Sudman, 1980; Groves, Fultz, & Martin, 1992). We used a semi-structured interview format to standardize interviews, but probes were scripted in such a way to encourage participants to freely describe any thoughts or experiences that might have influenced their responses (Willis, 1994). Interviewers also asked follow-up questions to clarify specific points or allow participants to provide more detail.

Our study aims were to: (1) use cognitive interviewing methods to refine a brief self-report measure of current socially assigned gender expression appropriate for use in large-scale population-based health surveillance and (2) examine discriminant validity of two brief self-report items on socially assigned gender expression. We had no specific hypotheses for our first study aim since qualitative analysis uses emergent themes to identify common associations or factors influencing item performance as opposed to using a preconceived thematic framework (Miles & Huberman, 1994). For our second study aim, we used quantitative methods to assess the discriminant validity of the new items. We hypothesized that completely heterosexual females and males would score significantly more feminine and masculine, respectively, and that individuals with a minority sexual orientation would score significantly more gender nonconforming than completely heterosexuals, as has been shown in other studies of gender expression (Lippa, 2002). We also hypothesized that individuals not currently identifying with their birth sex would score significantly more gender nonconforming than individuals whose birth sex and current gender identification were concordant.

Method

Study Sample and Research Design

Participants aged 18–30 years were recruited using a variety of methods designed to enroll the full range of sexual orientations and gender expressions. Recruitment occurred through an adolescent and young adult clinic at a pediatric hospital and at a private university in the Boston area as part of a larger cognitive interviewing study on survey items assessing social demographics and different types of health issues. Ads were placed in a free daily Boston newspaper and a weekly LGB and transgender (LGBT) newspaper. Information about study recruitment was also distributed at a local LGBT community parade. In order to increase enrollment of transgender participants, study information was distributed via four New England transgender community listservs, the Boston craigslist.org website, two Boston health clinics serving transgender patients, and LGBT student groups at seven Boston area universities. Enrollment and interviews were conducted from March 2007 through December 2008. Eighty-three adolescents and young adults were enrolled. One interview could not be transcribed due to a technical problem with the recording and was excluded, leaving us with survey and interview data from 82 participants. The mean age of the sample was 23.8 years (standard deviation 3.2); 22% were transgender, 48% female, and 30% male; 62% of participants were White; and 70% identified their sexual orientation as other than completely heterosexual (Table I).

Table I.

Demographics of participants in a cognitive interviewing study with adolescents and young adults to assess measures of socially assigned gender expression (N=82)

	N	Mean (SD)
Age (years)	82	23.8 (3.2)
	N	Percent
Highest Degree or Diploma Obtained
High school/GED equivalent or less	10	12.2
Some college/trade vocational school	24	29.3
Bachelor’s degree or higher	48	58.5
Race/Ethnicity
Hispanic/Latino	8	9.8
American Indian/Alaskan Native	1	1.2
Asian	9	11.0
Black or African American	5	6.1
White	51	62.2
Multiracial	7	8.5
Missing	1	1.2
Sexual Orientation
Completely Heterosexual	24	29.3
Mostly Heterosexual	14	17.1
Bisexual	9	11.0
Mostly Homosexual	9	11.0
Completely Homosexual	18	21.9
Not Sure	5	6.1
Other/Missing a	3	3.7
Gender Typology
Born female; identify as female	39	47.6
Born male; identify as male	25	30.5
Born female; do not identify as female	12	14.6
Born male; do not identify as male	6	7.3

^aIncludes two participants who did not respond to the question and one participant who chose more than one option.

Two data collection protocols were used. Participants recruited for the larger health measures study completed a self-report questionnaire composed of the items to be tested and then were interviewed immediately after by study staff in a private room at a health clinic; these participants also received a slightly longer survey as part of the larger health study. Participants not part of the larger study completed the self-report questionnaire of items to be tested via email and were then interviewed via telephone immediately after. Both questionnaires took 5–10 minutes to complete, and cognitive interviews lasted approximately 30–60 minutes. Participants provided consent before beginning the study. On completing the interview, participants received a $10 gift certificate to an area store. This study was approved by a pediatric hospital institutional review board.

Questionnaires used in both protocols included two items on socially assigned gender expression adapted from a single item assessing appearance conformity used in the Cancer Screening Project for Women (Clark et al., 2005). We revised the original item so that it was no longer female-specific and added a second question inquiring about mannerisms. The appearance item read, “A person’s appearance, style, or dress may affect the way people think of her or him. How do you think people describe your appearance, style, or dress?” The mannerisms item read, “A person’s mannerisms (such as the way a person walks or talks) may affect the way people think of her or him. How do you think people describe your mannerisms?” Response options for both items ranged on a seven-point scale from “very feminine” to “very masculine.”

Two items adapted from a prior cognitive interviewing study were used to ask about birth sex and gender identity separately (Conron, Scout, & Austin, 2009). The first item read, “What sex were you born?” with the options “female” and “male.” The second item read, “How do you describe yourself now?” with the options, “female,” “male,” “transgender female-to-male,” “transgender male-to-female,” and “do not identify as female or male.” An item on sexual orientation asked, “Which of the following best describes your feelings?” with five response options ranging from “completely heterosexual” to “completely homosexual” and an additional option “unsure.” This question has been previously cognitively tested (Austin, Conron, Patel, & Freedner, 2007). We also collected information on age, race/ethnicity, and level of education.

After participants completed the self-report questionnaire, research staff conducted semi-structured, individual, in-depth interviews with participants. Researchers developed an interview guide using the cognitive interviewing method, which is designed to assess participant understanding of individual items (Bradburn & Sudman, 1980; Groves, Fultz, & Martin, 1992). Details provided by the participant on question interpretation and information retrieval patterns enabled assessment of variation in item clarity, valence of emotional reaction to items, respondent burden, and perceived threat associated with the measures. The interviewer asked questions such as:

What went through your mind when you read this question?

How did you choose your answer?

What kinds of things did you think about when you saw the word “mannerisms” [in Item #2]?

The scripted probes were designed to standardize interviews across interviewers (SAW, HLC, LAP) and reduce bias, but probes were broad enough to encourage participants to relate any experience they thought was relevant to the questionnaire (Willis, 1994). Interviewers asked additional follow-up questions to clarify participant responses or encourage greater detail. All interviews were conducted in English and digitally recorded for verbatim transcription.

Data Coding and Analysis

Mixed qualitative and quantitative methods were used to analyze the data from all 82 participants. For qualitative analyses, transcripts of interviews were examined for participants’ responses relating to the two gender expression items using methods described by Willis (1994). To develop a coding scheme, two coders (SAW, VB) read all the transcripts independently and identified common themes relating to item comprehension, interpretation of survey items, and emotional reactions relevant to study aims (Miles & Huberman, 1994). Themes were discussed in the research group (SAW, VB, HLC, SBA) to formulate a common coding scheme. The final scheme included the following themes: item interpretation, participant anticipated variations in response related to social context, emotional reactions to the items, personal reflections, societal perceptions or assumptions about the participant’s identity, the role of gender identity or sexual orientation in choosing a response option, and suggested revisions to the item. To assess agreement between coders, each coder independently analyzed the same four randomly chosen transcripts. Coding of these four transcripts was then discussed in the research group to identify any interpretation differences between coders and why those differences may have occurred (Willis, 1994). The coding template was revised based on these discussions, and coders independently analyzed all transcripts. To assess interrater agreement, an independent research staffer not involved in the study calculated Cohen’s K coefficient (Cohen, 1960) and simple percent agreement based on the coding templates completed by the two coders. Estimated with a randomly selected subsample of 30 participants for this assessment, K was 0.51 (95% CI 0.45, 0.56) and percent agreement was 88.4%. Once coding was complete, the research team discussed emergent subgroup differences. Based on these discussions, coders independently summarized findings relevant to item performance in subgroupings defined by participant gender and sexual orientation. Participants who identified as transgender or as neither female nor male or whose birth sex and gender identity were not concordant were considered transgender and were analyzed separately from nontransgender participants. Summaries were synthesized, compared, and contrasted across subgroups. Summaries were discussed by the entire research group to understand how interpretation of the gender expression items varied by gender identity and sexual orientation.

To test the hypotheses relating to our second study aim, we used multivariate analysis of variance (MANOVA) to assess the association between identifying as transgender or having a minority sexual orientation and gender expression. We evaluated discriminant validity of the gender expression items specifically in terms of how well our new measures of gender expression corresponded with findings from prior research (Lippa, 2002). Responses to the two gender expression items were recoded so that a value of “1” corresponded to gender expression that was very conforming to the participants’ birth sex; and a value of “7” meant that the participants’ gender expression was very nonconforming relative to her/his birth sex.

To explore further how the gender expression items performed within gender and sexual orientation groups, we calculated means, standard deviations, and Spearman correlations for the appearance and mannerisms items across gender identity and sexual orientation groups. For these exploratory analyses, we did not have specific hypotheses about how correlations between the two measures might operate as a function of gender or sexual orientation. For these analyses, transgender participants were excluded from examination of sexual orientation group differences because many reported difficulty answering the sexual orientation item as worded, as has been reported previously (Austin et al., 2007; Conron et al., 2009). We also conducted cross-tabulations and Chi-square or Fisher exact tests to evaluate if nonconcordance differed by sexual orientation and transgender status. All statistical analyses were conducted using SAS 9.1.

Results

Qualitative Results

Four salient themes directly related to item performance emerged from the cognitive interview transcript analysis: overall clarity of item wording, within-person variation in gender expression, desirability of highest or lowest ends of item range, and tension between own versus others’ perspectives.

Overall clarity of item wording

Participants generally found both gender expression items clear and easy to understand. Although three participants (3.6%) were not familiar with the word “mannerisms,” they were able to deduce its definition from the examples “walks” and “talks” that were included in the question. When asked how he would define mannerisms, a 21-year-old completely heterosexual male responded, “Mannerisms? Um, I don’t really know the word, but I kind of thought it was like manners, something like that.” (ID73, lines 79–80) While other participants provided a variety of definitions for mannerisms, from “vocabulary” to “relating to others,” most agreed that “walks” and “talks” were good examples. Five participants (6.1%) from a variety of sexual orientation groups and genders suggested revising both items to remove gender-specific pronouns. There was one instance (1.2%) for the mannerisms item where a male participant chose “Equally feminine and masculine” but realized while reading his answers to the interviewer that he meant to choose “Somewhat feminine” instead.

Most participants were comfortable with the feminine-masculine Likert scale, but 11 participants (13.4%) thought it was limiting. A 27-year-old mostly homosexual male complained, “…it’s just frustrating, too, because I don’t necessarily think of myself just in terms of mannerisms or style or whatever in that range…I don’t really feel accurate to try to describe myself in that way.” (ID47, lines 136–145) The remainder of participants, however, either mentioned no difficulties or liked the scale, as in the case of one 23-year-old mostly homosexual female who said, “…the continuum was kind of wide and I liked that. So, I felt comfortable picking a choice where – kind of exactly what I felt like.” (ID77, lines 61–67) A 30-year-old transgender female-to-male participant pointed out that the feminine-masculine scale was useful as it would be impossible to stay current with an ever-changing popular vocabulary: “Youth have a tendency to perpetually be four steps ahead of us…they’re coming out with new terms every day to define how they feel.” (ID20, lines 374–386)

Within-person variation in gender expression

A challenge mentioned by 42 participants (51.2%) was difficulty checking a single response option to describe how others perceive their gender expression when their appearance or mannerisms might vary from day to day or setting to setting, but only two participants (2.4%) chose multiple responses for the appearance or mannerisms item. A 22-year-old, completely heterosexual female explained her confusion by saying, “I don’t know what that really means, like, ‘cause sometimes I dress really girly in, like pink and everything, and sometimes I dress kind of sporty and I guess that could be masculine…I don’t really understand.” (ID39, lines 15–24) An 18-year-old, completely heterosexual male explained that in his case, appearance from day to day was a choice based on his interest in fashion: “I have, like, lots of looks…I’m just, like, really mixed into, like, all the clothing cultures.” (ID27, lines 133–136)

Sexual and gender minority participants (61.0%, 55.6%, respectively) considered variation in their appearance or mannerisms more frequently than nonminorities (30.4%). Some described modifying their gender expression depending on the degree of safety or threat they felt in different settings. A 20-year-old female who identified as mostly homosexual said, “If I’m going somewhere where I don’t think it’s appropriate to dress in guys’ clothes, I obviously don’t. My school for one. It’s pretty homophobic so I don’t dress gay there.” (ID34, lines 82–89) An 18-year-old mostly homosexual male had difficulty describing himself because he felt pulled between the culture of his home neighborhood and prep school:

It’s difficult for me to answer, just because part of the culture that I was born into without a choice, so meaning, like, my African-American slash Black culture, um, but also having gone to a prep school for four years in suburbs in the middle of Massachusetts, which is a very, very liberal state…There are certain expressions, a way of expressing yourself that wouldn’t fly too well [at home.]

(ID43, lines 68–104)

While some transgender participants reported that their appearance ranged from very masculine to very feminine depending on the day or setting, 10 transgender participants (55.6%) felt that their mannerisms were more consistent. Three transgender participants (13.7%) reflected that they made less of a conscious effort to change their mannerisms than their appearance post-transition from their birth sex to their current gender. A 25-year-old transgender female-to-male participant said that he was more comfortable with his mannerisms in his current gender:

When I was pre-transition…I was just trying to be as hetero-normative and as masculine as possible, because I just wanted to present that so that people would stop seeing me as a female….now, I have no problem…talking about, you know, homosexuality or…whether I find a man attractive or not…I just feel more comfortable in—in, uh, my sexuality and my gender and my gender presentation.

(ID71, lines 143–154)

Participants resolved the dilemma of variation in their gender expression by reporting how they believed they were perceived by others on average. A 23-year-old mostly homosexual female explained, “I sort of feel free to be very different from one day to the next…I guess I picked [somewhat masculine] based on my, like, average, like, going to school kind of day to day.” (ID77, lines 82–83) Fifty-nine participants (69.5%) echoed this sentiment, averaging how they believed they were perceived by others across different days and social contexts to come up with an appropriate response.

Desirability of highest and lowest ends of item range

Nineteen participants (23.2%) remarked that choosing a response option was more than simply describing how others perceived their appearance or mannerisms; they felt it also implied a judgment of their character or values. Explaining why she could not describe her mannerisms as “very feminine,” a completely heterosexual 18-year-old female said, “You know, not everyone’s perfect, everyone swears that once…everyone has their days.” (ID22, lines 614–618) A 28-year-old mostly heterosexual male felt that “very masculine” represented an unrealistic ideal:

It’s almost impossible to be…the ideal Marlboro man riding a horse, smoking a cigarette, you know, with four girls on my horse with me and the gun and all…I think everyone who’s actually honest with themselves [has to say], “Well, you know I don’t—I don’t—I'm not living up to every ideal of what the world’s idea of masculine is.”

(ID76, lines 169–173)

For others, being perceived by others as “very feminine” or “very masculine” had a negative connotation. A 30-year-old transgender female-to-male participant described wanting to distance himself from negative characteristics he associated with “very masculine”:

I’d rather not think of myself on that extreme, even though it’s probably truthful to the people I know…I thought of the visual image of, um, kind of the very large, muscular man who takes up a lot of space…is loud in presence and, uh, is just kind of rude. And I didn’t want to identify as that.

(ID20, lines 139–150)

Tension between own versus others’ perspectives

One source of confusion was whether participants were describing their own perception of their appearance and mannerisms or how they thought other people viewed them. Seven participants (8.5%) had not realized that the appearance question asked about an outside perspective and had assumed it asked about self-perception; one participant (1.2%) misinterpreted the mannerisms question in this way. Participants who expressed difficulty answering from another person’s perspective, either because they disagreed with others’ perceptions or because they were unsure of how others saw them, resolved this dilemma by recalling reactions of friends or family to their gender expression or thinking about what a stranger might think upon meeting them for the first time. The feeling of being misperceived was a distressing subject, especially for some transgender participants. A 19-year-old who was born female but no longer identified as male or female expressed frustration, saying, “I do everything I can to avoid doing a lot of girly stuff because people get the totally wrong impression of me…it’s like, no I hate them all.” (ID59, lines 125–127)

References to stereotypes, labels, and judgment were also common when thinking about how others viewed them. While these sentiments were present in all sexual orientation and gender groups, it was especially prevalent in sexual and gender minority participants. 30.4% of nonminority participants discussed stereotypes in comparison with 48.8% of participants who identified as a sexual minority and 50.0% of transgender participants. When speaking about how she felt the world saw her, a 25-year-old completely homosexual female explained, “Again, it’s the whole society viewing you. And in general they’re passing judgment, but I guess it’s just how people have reacted in the past.” (ID64, lines 147–149) For several participants, the items brought up negative feelings about gender stereotypes. A 24-year-old completely homosexual female resented that confidence was often perceived as being masculine: “I think that me walking with my head up and straight and walking with confidence and speaking with confidence, you know, I don’t think that should be a masculine trait, however, a lot of people ascribe that to the masculine.” (ID04, lines 143–147)

Quantitative Results

Preliminary analyses showed that age, race/ethnicity, and level of education were not significantly associated with nonconforming gender expression and thus were excluded from statistical models. As predicted, comparing within completely heterosexuals, the mean scores for females on the appearance and mannerisms items were 1.9 and 1.8, respectively, while males scored 5.1 and 5.9. Quantitative results confirmed that the appearance and mannerisms items performed as expected in discriminating between the gender and sexual orientation groups. In MANOVA analyses stratified on birth sex that included sexual orientation and transgender status, transgender status was significantly associated with nonconforming gender expression on both items (Table II). Minority sexual orientation status was also associated with nonconforming appearance and mannerisms, though in males this association was weaker than in females and did not reach statistical significance. In these analyses, transgender status and sexual orientation explained more variation in appearance for both females and males (R²=0.63, 0.72, respectively) than in mannerisms (R²=0.56, 0.57, respectively).

Table II.

Results from multivariate analysis of variance (MANOVA) models stratified on birth sex estimating group differences in appearance and mannerisms gender nonconformity in a cognitive interviewing study with adolescents and young adults (N=82)

Independent variable	Dependent variable	Wilks’ λ	F (df)	Univariate F	β (SE β)	P-value
Birth Sex Female (n=51)
Transgender vs. nontransgender		.55	19.55 (2,47)			<0.0001
	Appearancea			35.25	2.32 (0.39)	<0.0001
	Mannerisma			21.33	1.66 (0.36)	<0.0001
Minority sexual orientation vs. heterosexual		.68	11.15 (2,47)			0.0001
	Appearancea			16.61	1.45 (0.36)	0.0002
	Mannerisma			16.50	1.34 (0.33)	0.0002
Birth Sex Male (n=31)
Transgender vs. nontransgender		.29	33.52 (2,27)			<0.0001
	Appearancea			69.20	3.44 (0.41)	<0.0001
	Mannerisma			31.10	2.80 (0.50)	<0.0001
Minority sexual orientation vs. heterosexual		.90	1.48 (2,27)			0.25
	Appearancea			1.53	0.46 (0.37)	0.23
	Mannerisma			3.03	0.79 (0.45)	0.09

F=F Value, df=Degrees of Freedom (numerator, denominator) β=Parameter estimate, SE=Standard Error.

^aValue of 1 indicates that participant’s gender expression was very conforming to birth sex while 7 corresponds with very nonconforming to birth sex.

Among females, total R² for appearance is 0.63 and for mannerisms is 0.56. Among males, total R² for appearance is 0.72 and for mannerisms is 0.57.

Among nontransgender participants, females reported an average value of 2.8 on the appearance item and 2.5 on the mannerisms item while males reported 5.7 and 5.2, respectively, with 1 being “very feminine” and 7 being “very masculine” (Table III). Among participants who no longer identified with their birth sex, participants who were born female reported average values of 5.7 and 4.8, and participants who were born male reported average values of 2.2 and 2.3. Additional quantitative analyses illustrated in more detail how the two items performed in each gender and sexual orientation group. The appearance and mannerisms items were significantly correlated within nontransgender females (Spearman r₀ = 0.74, p<0.0001) and males (Spearman r₀ = 0.68, p=0.0002), but the correlation was substantially lower and not significant in transgender participants (Spearman r₀ = 0.39, p=0.11) (Table III). The correlation between appearance and mannerisms also varied in nontransgender participants based on majority or minority sexual orientation status. Although the correlations between the two items were significant in both completely heterosexual and sexual minority participants, correlations were lower in individuals with a minority sexual orientation (Table III).

Table III.

Correlation between gender expression appearance and mannerisms survey items by gender and sexual minority status in a cognitive interviewing study with adolescents and young adults (N=82)

	N	Appearancea	Mannerismsa	Correlation (Spearman Rank)
Gender		Mean (SD)	Mean (SD)	r0	p
Female	39	2.8 (1.3)	2.5 (1.1)	0.74	<0.0001
Male	25	5.7 (0.9)	5.2 (1.2)	0.68	0.0002
Born female; do not identify as female	12	5.7 (1.1)	4.8 (1.3)	0.39	0.20
Born male; do not identify as male	6	2.2 (0.8)	2.3 (1.0)	0.39	0.44
Sexual Orientationb
Females
Completely heterosexual	16	1.9 (0.8)	1.8 (0.7)	0.80	0.0002
Minority sexual orientation	23	3.4 (1.2)	3.1 (1.0)	0.44	0.04
Males
Completely heterosexual	7	6.1 (1.1)	5.9 (1.0)	0.90	0.005
Minority sexual orientation	18	5.5 (0.9)	4.9 (1.1)	0.47	0.05

^aValue of 1 corresponds with a gender expression of “Very feminine” while 7 corresponds with “Very masculine.”

^bTransgender participants excluded from these analyses because many expressed difficulty endorsing a sexual orientation.

Inspecting cross-tabulations of responses to the appearance and mannerisms items, we found substantial nonconcordance. For the sample as a whole, 47.5% were perfectly concordant in their responses to the two items, while 20.7% were one unit more nonconforming (on the scale ranging from 1 to 7) for appearance than for mannerisms and an additional 21.9% were one unit more nonconforming for mannerisms than for appearance. An additional 9.8% were discordant by two or more units. Among nontransgender participants, concordance occurred less often in those with a minority sexual orientation (43.9%) compared to completely heterosexuals (73.9%) (χ² = 5.4, df =1, p =0.02). Concordance in transgender participants was 22.2% compared to 54.7% in nontransgendered individuals (p=0.02, Fisher’s exact test).

Discussion

Using cognitive interviewing methods, we evaluated two survey items measuring socially assigned gender expression in a diverse adolescent and young adult population. Our aims were to refine two brief self-report measures appropriate for use in large-scale population-based health surveillance and to validate those measures using prior knowledge of gender expression, minority sexual orientation, and gender identity. Based on results from semi-structured interviews, we found that participants largely understood both questions and correctly interpreted both the question stem and the response options. In the few instances where a participant did not know the word “mannerisms,” the examples of “walks” and “talks” provided in parentheses were sufficient clarification. The most important source of confusion was related to within-person variation from day to day and setting to setting, making it challenging to choose only one response. Participants who raised this issue explained that they handled this variation by answering the items based on how they felt they were perceived on average.

Another important finding was that some individuals perceived the extreme options of “very feminine” and “very masculine” as having a positive connotation; whereas, others felt they had a negative connotation. In a variation on end-aversion bias, where survey respondents avoid extreme options because these circumstances may not always be true, our study showed that some respondents avoided the extremes because they did not want these circumstances to always be true (Streiner & Norman, 1995). Given that end-aversion bias was not the case for all respondents and that the extremes were selected, it seems unnecessary to provide additional extreme response options as anchors or “throw away” categories designed to be excluded from analyses, as is sometimes recommended (Streiner & Norman, 1995). It is also important to note that for some participants, responses differed based on whether they thought the question asked about self-perception of gender expression or others’ perceptions. Approximately 11% of participants commented that they misinterpreted items as asking about their own perception as opposed to how others perceive them.

Our study has several limitations to be considered. Our measures ask specifically about current socially assigned gender expression and might not be indicative of past childhood experiences affecting adult health status. Also, subgroups based on race/ethnicity and levels of education were too small to allow for more in-depth analyses of item performance by these demographic characteristics. We also grouped transgender participants into two groups based on birth sex (born female, no longer identify as female; born male, no longer identify as male) due to sample size constraints; however, there was heterogeneity within these groups with respect to current gender identification. In addition, our item asking about birth sex did not give an option for intersex respondents.

Based on our findings, we revised both items to include the qualifier “on average” in the item stem to mitigate uncertainty caused by within-person variation in gender expression and removed gender-specific pronouns (Table IV). We recommend both items for inclusion on public health surveillance system surveys since each item may capture individuals who are perceived as gender nonconforming in one respect but not the other. Using both measures to assess gender expression may provide important detail that would not be captured if only one item were used. Our finding that the correlation between appearance and mannerisms was lower in participants with a minority sexual orientation or transgender identity than in heterosexual and nontransgender participants indicates that subpopulations who are most vulnerable to victimization targeting gender nonconformity could be misclassified if only one item were included on survey instruments. Future studies with larger samples should further explore differences between the two items, specifically if one item is more strongly associated with experiences of discrimination, violence, and poor health outcomes. Another area for investigation is whether the relationship between socially assigned gender nonconformity and health sequealae varies based on gender identity or other factors such as race/ethnicity, immigrant status, socioeconomic status, or region of residence. For example, it is conceivable that an individual who was born male but now identifies as female would be at lower risk if she were perceived by others as female instead of transgender. Thus, the association between degree of gender nonconformity and discrimination could be nonlinear (e.g. an inverted U) in certain situations. It would also be worthwhile to compare sensitivity to gender nonconformity with our measures versus the BSRI or other measures of gender expression. It is possible that our wording included vocabulary, such as “mannerisms” or “style,” that is understood differently in the nonheterosexual community and perhaps inflating the association between gender nonconformity and sexual orientation. However, in our cognitive interviews we did not observe qualitative differences in item interpretation between these groups, suggesting that both items performed similarly across genders and sexual orientations.

Table IV.

Recommended wording of items to assess socially assigned gender nonconformity

Gender Expression
1. A person’s appearance, style, or dress may affect the way people think of them. On average, how do you think people would describe your appearance, style, or dress? □Very feminine □Mostly feminine □Somewhat feminine □Equally feminine and masculine □Somewhat masculine □Mostly masculine □Very masculine
2. A person’s mannerisms (such as the way they walk or talk) may affect the way people think of them. On average, how do you think people would describe your mannerisms? □Very feminine □Mostly feminine □Somewhat feminine □Equally feminine and masculine □Somewhat masculine □Mostly masculine □Very masculine

A growing body of research has exposed discrimination and violence victimization targeted toward perceived gender nonconformity. While previous work has in large part focused on the role that socially assigned gender nonconformity plays in victimization in the LGBT community (Carbone, 2008; D'Augelli et al., 2006; Skidmore et al., 2006), it is becoming apparent that individuals are targeted if perceived as gender nonconforming regardless of sexual orientation (Hiestand et al., 2007; Horn, 2007). Although these disparities are documented, pervasive, and pernicious, they have received almost no recognition within the nation’s public health surveillance system. The CDC recently included a measure assessing perceived race on the BRFSS in order to assess the health impact of socially assigned race. A similar need exists for a measure of socially assigned gender expression. Our recommended measure is appropriate for use on statewide and national public health surveillance tools such as the BRFSS, the National Survey of Family Growth (NSFG), the National Survey on Drug Use and Health (NSDUH), the Youth Risk Behavior Surveillance System (YRBSS), and others. Systematic monitoring will allow us to assess the extent to which socially assigned gender nonconformity is a determinant of health disparities and identify opportunities for legislation and expanded services to redress the harm caused by gender-based discrimination.