Lack of Representation in Psychiatric Research: A Data-Driven Example From Scientific Articles Published in 2019 and 2020 in the American Journal of Psychiatry

Abstract

Objective:

The authors examined representation and accuracy of descriptions of sociodemographic identities in psychiatric research through quantifying data contained in recently published articles from a high-impact psychiatry journal.

Methods:

Sociodemographic data were aggregated from articles (i.e., studies that provide information on individual samples) published in the American Journal of Psychiatry in 2019 and 2020 (N=125). Articles were coded by two raters for sociodemographic data, acknowledgment of lack of representation as a limitation, and focus on health disparities or inequities.

Results:

While 90% of studies provided the age of participants and 84% provided information about the sex/gender of participants, only 43% presented information about the racial or ethnicity identities of participants. One study reported the sexual identity of participants. Lack of representation relative to 2019 U.S. Census data was found for multiple racial groups, Latino/Hispanic individuals, and women (genetic studies only). Only 25% of studies acknowledged lack of representation as a limitation, and two studies focused on health disparities or inequities.

Conclusions:

These findings highlight a need to increase representation in psychiatric research and improve accuracy of language when describing the sociodemographic characteristics of participants.

Across assigned sex at birth and racial, ethnic, sexual (e.g., bisexual, gay, lesbian, queer), and gender (i.e., one’s internal sense of their gender) identities, there are important differences in prevalence and determinants of psychiatric illness, access to psychiatric care, and outcomes (e.g., ^1–4). However, a growing body of research has shown a lack of representation and missing sociodemographic information regarding racial and ethnic identities in psychiatric research (5–9). While less research has examined representation of people with minoritized (i.e., experiencing discrimination, prejudice, and reduced power because of historical and current social hierarchies) sexual or gender identities, one review showed a near-absolute lack of description of these identities (10).

While previous reviews convey a clear message that lack of representation has been a persistent issue in psychiatric research, they have focused on specific disorders (e.g., alcohol use disorder) or on specific subpopulations (e.g., women), or have spanned decades. Research spanning psychiatric illnesses, examining multiple sociodemographic characteristics, and focused on recently published science will substantively add information about the state of the field today and provide a benchmark to track progress.

In this study, we analyzed published scientific articles in the American Journal of Psychiatry in 2019 and 2020 (capturing articles accepted for publication prior to the murder of George Floyd and subsequent calls for increased equity in science) to characterize representation across racial, ethnic, gender, and sexual identities. Inaccurate labels and categories, acknowledgment of representation as a limitation, and health inequities or disparities as a focus of the article were also assessed.

METHODS

Given the lack of description of sociodemographic data of individual (human) samples, meta-analyses, animal studies, commentaries, editorials, reviews, and letters were excluded, resulting in the identification of 125 scientific articles. Two independent raters (S.L.P., P.P., C.E.V., and K.L.) reviewed all components of each article and supplementary material for information on sociodemographic variables (age, assigned sex at birth, and racial, ethnic, sexual, and gender identities) and description of these questions. Raters also searched the entire article for words (e.g., discrimination, disparities, inequities, race, sexual orientation, generalizability, representation) and topics (e.g., depression in Black adults) related to examination of health inequities and inclusion of lack of generalizability or representation as a limitation. The infrequent discrepancies (e.g., raters disagreed 2.4% [3/125] of the time about whether the article examined health inequity, and 19% [24/125] of the time about representation/generalizability being mentioned) were resolved by a third rater.

Raters coded the type (e.g., genetic, neuroimaging) and the location (e.g., United States, United Kingdom) of the study. Data are described separately for genetic and nongenetic studies, given previous research showing that genetic studies almost exclusively include participants of European ancestry (>78%) (11) and separately for studies conducted within and outside of the United States to allow for comparison with 2019 U.S. Census data population estimates (12). Studies conducted within the United States were assessed for over- or underrepresentation of participants with specific sociodemographic characteristics by comparing to local U.S. Census estimates for articles where a specific location or region was provided (N=19 for race/ethnicity; N=25 for sex/gender). Articles for which a location could not be determined were tested against national U.S. Census estimates. Z-tests of the difference of proportions using the prop.test function in R were run to assess whether the probability of the numbers of female individuals/women and people from specific racial and ethnic groups were higher or lower in the published articles compared to the U.S. Census data (13, 14). The z-tests provide a 95% confidence interval and chi-square with a p value.

RESULTS

Representation Across Racial Groups

While 90% (113/125) of studies reported the age of their participants, only 43% (54/125) provided information about race or ethnicity. Across these 54 studies (Table 1), 94.6% of participants were described as White or Caucasian. When only nongenetic studies were examined (N=39), 71.4% of participants were described as White or Caucasian; the second most prevalent group was described as “other” or “nonwhite” (10.6%), followed by African American, Black, or African (7.6%), and Hispanic or Latino (5.3%). The remaining racial groups comprised <1% of the participants. Table 2 shows the summary results of over- and underrepresentation across race and ethnicity when each study is compared to local or national U.S. Census estimates. Asian/Asian American and Hispanic/Latino participants were underrepresented in 78% of articles. Biracial/multiracial participants were underrepresented in 49% of articles. Black and White participants were underrepresented in 29% and 37% of the articles, respectively.

TABLE 1.

Sample sizes in all studies

Measure or Group		All Studies	U.S. Only	All Studies	U.S. Only	All Studies	U.S. Only	All Studies	U.S. Only	All Studies	U.S. Only	All Studies	U.S. Only
		Full sample N across all articles		Number of articles		Nongenetic N across all articles		Number of articles		Genetic N across all articles		Number of articles
Total sample size		18,939,676	474,978	125	62	15,099,944	162,739	92	49	3,839,731	312,239	33	13
Total sample size of articles that provided information on sex/gender		15,711,826	358,718	105	57	14,953,766	46,479	80	44	758,060	312,239	22	13
Total sample size of articles that provided information on race or ethnicity		1,172,244	250,144	54	41	48,834	45,005	39	32	1,123,410	205,139	15	9
	U.S. Census dataa (%)	Full sample N across all articles		% (out of applicable total sample size)		Nongenetic N across all articles		% (out of applicable total sample size)		Genetic N across all articles		% (out of applicable total sample size)
Female	50.8	7,976,830	99,484	50.8	27.7	7,662,372	22,482	51.2	48.4	314,458	77,002	41.5	24.7
Male	49.2	7,712,711	259,231	49.1	72.3	7,269,109	23,994	48.6	51.6	443,602	235,237	58.5	75.4
Total	100.0	15,689,541	358,715	99.9	100.0	14,931,481	46,476	99.8	99.9	758,060	312,239	100.0	100.0
American Indian or Alaska Native/Native American	0.7	6	6	0.0	0.0	6	6	0.0	0.0	0	0	0.0	0.0
Asian/Asian American	5.6	2,554	206	0.2	0.1	547	202	1.1	0.4	2,007	4	0.2	0.0
Biracial/multiracial/more than one race/multiple	2.5	52	50	0.0	0.0	39	37	0.1	0.1	13	13	0.0	0.0
Black/African American/African	12.4	29,788	29,306	2.5	11.7	3,718	3,236	7.6	7.2	26,070	26,070	2.3	12.7
Hispanic/Latino	18.4	2,602	2,602	0.2	1.0	2,602	2,602	5.3	5.8	0	0	0.0	0.0
Native Hawaiian or other Pacific Islander	0.2	1	1	0.0	0.0	1	1	0.0	0.0	0	0	0.0	0.0
Other/nonwhite	0.3	5,427	4,913	0.5	1.9	5,171	4,657	10.6	10.4	256	256	0.0	0.1
White/Caucasian	60.0	1,109,474	211,281	94.6	84.5	347,869	32,499	71.4	72.2	1,074,605	178,782	95.7	87.2
Total	100.0	1,149,904	248,365	98.1	99.3	46,953	24,353	96.1	96.1	1,102,951	205,125	98.2	99.9

^aU.S. Census data are from the 2019 American Community Survey 1-year estimates.

TABLE 2.

Summary of significant overrepresentation and underrepresentation in articles^a

	Female		American Indian or Alaska Native Alone		Asian Alone		Black or African Hispanic or American Latino (Of Alone Any Race)		Hispanic or Latino (Of Any Race)		Native Hawaiian or Other Pacific Islander Alone		Some Other Race Alone		Two or More Races		White Alone
All studies	N	%b	N	%c	N	%c	N	%c	N	%c	N	%c	N	%c	N	%c	N	%c
Articles overrepresenting	14	25	0	0	0	0	19	46	2	5	0	0	20	49	2	5	15	37
Articles underrepresenting	27	47	10	24	32	78	12	29	32	78	4	10	2	5	20	49	15	37
Articles with NS difference	16	28	31	76	9	22	10	24	7	17	37	90	19	46	19	46	11	27
Nongenetic studies only	N	%d	N	%e	N	%e	N	%e	N	%e	N	%e	N	%e	N	%e	N	%e
Articles overrepresenting	13	30	0	0	0	0	14	44	2	6	0	0	16	50	2	6	10	31
Articles underrepresenting	21	48	7	22	23	72	10	31	23	72	3	9	0	0	14	44	11	34
Articles with NS difference	10	23	25	78	9	28	8	25	7	22	29	91	16	50	16	50	11	34
Genetic studies only	N	%f	N	%g	N	%g	N	%g	N	%g	N	%g	N	%g	N	%g	N	%g
Articles overrepresenting	1	8	0	0	0	0	5	56	0	0	0	0	4	44	0	0	5	56
Articles underrepresenting	6	46	3	33	9	100	2	22	9	100	1	11	2	22	6	67	4	44
Articles with NS difference	6	46	6	67	0	0	2	22	0	0	8	89	3	33	3	33	0	0

^aArticles that exhibited significant overrepresentation or significant underrepresentation were significant at p<0.05.

^bPercentage out of 57 articles.

^cPercentage out of 41 articles.

^dPercentage out of 44 articles.

^ePercentage out of 32 articles

^fPercentage out of 13 articles.

^gPercentage out of nine articles.

Representation Across Sex/Gender

One hundred and five studies (84%) reported on the sex/gender of their participants. Sixty-four studies only presented the sample size for one sex/gender. In these instances, we imputed the other number by assuming the authors restricted the options to binary (male/female, man/woman) responses. Following imputation, 50.8% of the total sample of participants was described as female/women/girls. While this distribution was similar for nongenetic studies, in U.S. genetic samples, only 24.7% are described as female/women/girls (Table 1). Table 2 shows that female participants/women/girls were underrepresented in 47% of articles and overrepresented in 25% of articles.

Inaccurate or Missing Information

One study described the question used to ask participants about their racial or ethnic identity and 43% (26/60) of articles used nonexistent racial categories such as “nonwhite,” “other,” and/or “mixed.” Additionally, no study provided a description of the question asked to establish sex or gender identity, precluding our ability to know if the articles were referring to participants’ assigned sex at birth or gender identity (given this lack of clarity, we used “sex/gender” in the present study). One study presented the sexual identity of participants. Only 25% (31/125) of articles identified lack of representation or generalizability concerns based on sample composition as a study limitation.

Prevalence of Health Equity Research

Two of the 125 studies examined inequities/disparities or psychiatric illness within minoritized populations.

DISCUSSION

We examined representation across a recent 2-year period (2019–2020) in a high-impact psychiatry journal. We found that minoritized racial and ethnic groups, especially Asian American and Latino/Hispanic participants, and female participants/women were underrepresented in these articles. Our results also showed that descriptions of sociodemographic characteristics were frequently missing, use of inaccurate labels was prevalent, and acknowledgment of lack of representation as a limitation was uncommon.

To increase diversity within research samples, investigators can connect with community-engaged researchers at their universities and read the existing literature (e.g., ¹⁵) to learn about best practices for partnering with minoritized populations. Researchers should also review the sociodemographic characteristics of participants in an ongoing fashion, like other enrollment criteria, and adjust as needed to ensure adequate representation based on local population estimates. Additionally, it is imperative to collect genetic data from ancestrally diverse populations and interpret these data in culturally informed ways. There are published recommendations to guide researchers (16).

Our results also found missing or inaccurate language about race and ethnicity in the articles. First, less than half of the articles provided the racial composition of the sample, limiting our ability to even evaluate representation. Further, within the studies that provided descriptors for racial identities, minoritized racial and ethnic groups maybe being homogenized into an “other/nonwhite” category, as this group was overrepresented compared to U.S. Census data in 49% of the articles (and underrepresented in 5%). This is an example of “white centering” in research (17, 18), which assumes similarity of all minoritized racial and ethnic groups into a single group in relation to White participants. Additionally, “Caucasian,” a label with a racist history (19), was used as a racial category in 45% of articles.

Despite the Institute of Medicine stating that assessing and reporting sexual and gender identities is important (20), only one article reported the sexual identity of participants, no studies clarified if they were reporting on assigned sex or gender identity, and descriptions of samples defaulted to an assumed gender binary. Researchers should review how they are assessing sociodemographic characteristics to ensure that accurate and affirming questions are being used to collect information on sexual and gender identities (as well as racial and ethnic identities). For the 1.2 million people in the United States who have a minoritized gender identity (21), binary response options (e.g., only man or woman) are often inaccurate and hurtful. The PhenXToolkit (22) provides examples of how to assess sociodemographic characteristics.

Despite lack of representation or missing data related to sociodemographic characteristics, there is significant value in utilizing existing data for secondary data analyses. However, when interpreting results and writing manuscripts, directly acknowledging these limitations and framing conclusions accordingly is crucial. This will spur further innovative research and increase rigor and reproducibility. Authors are also called upon to be thoughtful about conducting and interpreting subgroup analyses. It is imperative to place results within a sociohistorical context that recognizes structural inequities experienced by minoritized individuals and communities and to define race, ethnicity, and gender as social constructs. Additionally, if scientists more clearly and accurately describe the sample characteristics and questions used, meta-analyses can be utilized to examine determinants of health inequities by pooling across samples that were not initially powered to examine processes within and across subpopulations.

In addition to individual investigators, scientific journals and funding agencies (e.g., the National Institutes of Health [NIH]) can foster investment in recruiting representative samples and describing sociodemographic characteristics. For example, the Journal states on its website, “The methods section should provide a comprehensive description of the nature of the study group.” However, in the articles reviewed here, this guideline was largely not followed, indicating that expanded guidelines and provision of concrete requirements for characterizing sample sociodemographic characteristics (e.g., sample description must include racial and ethnic identities, gender identities, and age) may be beneficial. Resources can also be added to journal and funding agency websites. For example, the American Psychological Association has published guidelines that are available (23, 24) for reducing bias in scientific language, and there are published recommendations on how to report racial and ethnic identities in medical and scientific journals (25). Journals and funding agencies could also include specialty reviewers to provide critiques focused on inaccurate language, interpretation of results, and representation, ensuring that conclusions are framed within this context. This will also educate scientists and improve future research.

The lack of research (<2% of articles) on health inequities/disparities in 2019 and 2020 in a top-tier psychiatry journal is another urgent issue that can be remedied by journals and funding agencies. Our findings dovetail with a recent review of over 26,000 articles published in top-tier psychology journals that found that around 5% of the articles reviewed focused on outcomes for minoritized racial groups (26). Individuals from minoritized identities are more likely to conduct health equity research, which is subject to publication bias (27), subsequently contributing to NIH funding inequities (28), further hampering the careers of scientists already facing discrimination (28). As NIH pushes to fund more health equity science and scientists call for NIH to address long-standing funding inequities (29), journals could set benchmarks (and track progress over time) for inclusion of health equity research.

While the present study utilized a large sample of 125 recently published articles, several limitations are worth noting. First, we focused on articles only from 2019 and 2020 and one journal. We chose these years because they include recent research that was likely accepted for publication prior to the murder of George Floyd, which occurred in May 2020. We examined one high-impact journal to allow for in-depth review of each article for multiple sociodemographic characteristics. This approach also allows our findings to serve as a “baseline” for the Journal to track progress over time and provides an example that other journals can follow, as these issues are not specific to this journal. We also did not know the specific location of all U.S. studies, which diminished the specificity of the representation results for these studies.

Despite these limitations, the present study significantly advances understanding about representation and scientific language in psychiatric research. Further research that integrates the identities of authors, reviewers, and editorial board members and examines change over time in comparison to this “baseline” is needed. We hope that these results and conclusions encourage action and further discussion across levels of the scientific community (e.g., researchers, community members, institutions, journals, funding agencies).