Inadequate reporting of COVID-19 clinical studies: a renewed rationale for the Sex and Gender Equity in Research (SAGER) guidelines

Summary box.

• Women are under-represented as research participants in most interventional and observational studies on COVID-19.

• Main outcome data from interventional and observational studies are rarely reported disaggregated by sex.

• Sex and gender differences are inadequately examined in the analysis of the data.

• Lack of sex and gender analysis are seldom justified, and gender implications rarely discussed.

• There is a renewed rationale for strengthening the reporting of sex and gender dimensions in clinical research using the Sex and Gender Equity in Research guidelines.

Introduction

The COVID-19 pandemic has brought to the fore the implications of sex and gender differences to human health and well-being.1 Evidence to date indicates that men are at higher risk of severe disease and death than women, while women seem more likely to suffer from long-term COVID-19, and experience negative social and economic impacts.2 3 Biological sex as well as gender roles and behaviours can affect exposure to the virus and influence the prevalence of pre-existing conditions or harmful habits.4 Incorporating sex and gender considerations into clinical research on COVID-19 from the outset would help to identify differential patterns of risk, underlying reasons for disparate outcomes and to develop interventions to effectively respond to the various needs of women and men.5 Given the growing recognition of the relevance of sex and gender in the COVID-19 pandemic, the rationale for this analysis was to examine whether these dimensions are taken into account in clinical research on COVID-19.

We carried out a systematic review of COVID-19 research to examine how sex and gender are integrated and reported using the Sex and Gender Equity in Research (SAGER) guidelines recommendations as analytical categories.6 A PubMed search for interventional and observational trials on COVID-19, published from 1 December 2019 to 12 September 2020 resulted in 2066 studies. PubMed was chosen for offering the most comprehensive and up-to-date database of clinical studies on COVID-19.7

Our inclusion criteria were interventional or observational clinical studies the primary objective of which was related to COVID-19. Studies that did not focus on COVID-19 as an endpoint, commentaries, reviews, preprints, case studies and studies that had less than 10 participants were excluded. After screening the titles and abstracts, 1178 articles were excluded for not meeting the inclusion criteria. Of the 888 remaining articles, 816 were observational studies and 72 were interventional trials. Among the 816 observational studies, a 10% sample, that is, 82 articles, was randomly selected. The random selection ensured the representativeness of the sample of the retrieved observational studies, while ensuring a feasible number of papers to review. The full text of the 82 observational studies and the 72 interventional trials was reviewed, and 11 and 17 articles, respectively, were excluded for not meeting the inclusion criteria, for which the justifications are found in online supplemental material 1.

Overall, data were extracted from a total of 126 articles: 55 interventional (five on vaccines, 50 on COVID-19 treatments) and 71 observational studies. We extracted data on the number of participants and the percentage of female participants, and the adherence to the SAGER guidelines categories of data disaggregation, and sex-gender based analysis. See our data extraction table in online supplemental materials 2. One-sided Fisher’s Exact Test was used for statistical analysis of female participation, as it is more accurate than other tests for assessing the relationship between two variables in small samples. The level of significance was set at p<0.05.

Recognising the importance of sex and gender

In line with the SAGER guidelines, we first examined title, abstract and introduction. Among five single sex studies, four observational studies were on pregnancy as indicated in the title and abstract. One article focused on young males and highlighted the single-sex nature of the study in the title but did not justify the exclusion of women. Although sex and gender differences of COVID-19 have been reported, none of the clinical trials and only four observational studies (6%) reflected this in their introductions (table 1).

Table 1.

Adherence to SAGER guidelines by study type

SAGER guidelines recommendations	Type of study
	Interventional trials	%	Observational trials	%
Introduction: sex and gender differences in the infection, manifestation, or outcomes of COVID-19 should be acknowledged in the introduction.	0/55	0	4/71	6
Methodology1: papers should report how sex and gender were taken into account in the design of the study.	0/55	0	0/71	0
Methodology2: papers should justify reasons for the exclusion, or differing numbers, of males or females.	0/55	0	0/71	0
Results: ‘Sex- and gender-based analyses should be reported regardless of positive or negative outcome’.6 Articles should note if there is a difference between sexes or genders, or if there is no difference, in their results.	2/55	4	12/71	17
Discussion: what the results of the study mean for women and men should be analysed in the discussion section	1/55	2	12/71	17
Generalisability: if a sex and gender analysis is not done, then this should be justified or addressed in relation to the generalisability of the results.	0/55	0	2/71	8

SAGER, Sex and Gender Equity in Research.

Under-representation of women in clinical research

The SAGER guidelines recommend reporting how sex and gender were considered in the study design and encourages gender balance, or a justification for exclusion of one gender. None of the interventional or observational studies mentioned how sex or gender were considered in recruitment strategies or justified differing rates of women and men.

Overall, the mean proportion of women in our studies was 43% (figure 1). Only 31% of the studies were gender balanced, defined as 45%–55% female participation. A majority of studies (57%) reported lower than 45% women. A small proportion of studies (12%) had an over representation of women (>55%). There was no significant difference observed between interventional and observational studies (p=0.693), nor between large (>median; n=102 and n=154 for observational and interventional studies, respectively) and small studies (p=0.436). Vaccine studies were more likely than treatment trials (p=0.038) to have better gender balance.

Figure 1.

Female participation by study type and intervention type.

None of the studies justified under-representation or over-representation of women. In observational studies, most of which recruited participants in hospital settings, the over-representation of men may reflect the corresponding higher rate of hospitalisation among men. In clinical trials, where recruitment and sampling strategies are designed by researchers, women’s under-representation suggests gender-related barriers to women’s participation, or a lack of attention by the researchers to equal recruitment into the study.

We noted that pregnant women were ineligible for all the interventional trials in our sample and only included in four observational studies. This represents a persistent trend where pregnant women are excluded from clinical studies. However, such exclusion puts women and the fetus at greater risk as they may not benefit from treatment and vaccination due to insufficient data on safety and efficacy during pregnancy.8 9

Data disaggregation by sex and gender analysis

The SAGER guidelines advise that outcome data and data on participant withdrawals should be routinely disaggregated by sex, and that sex-based and gender-based analyses should be reported regardless of positive or negative outcome.

Out of 121 studies that were eligible to both women and men, only eight studies (7%) (seven observational and one interventional) disaggregated their main outcome data by sex. Only four observational studies disaggregated their main outcome data by age and sex concurrently.

Of 14 interventional trials that reported participant discontinuations, none provided the data by sex. While this may be due to small number of dropouts, this trend renders it difficult to investigate possible sex-related or gender-related reasons for discontinuation, such as greater or more serious adverse effects or the inconvenience of participating in the study.

A majority (89%) of studies did not mention whether or not sex or gender differences were analysed or observed. Only 14 studies (11%) (12 observational and two interventional trials) included an analysis of sex differences in their results. All of these studies noted a difference between women and men, either in the differing numbers of women and men in the hospital settings, or whether being female or male were associated with differing disease outcomes. One clinical trial noted different responses between women and men to a vaccination. While the numbers were low, observational studies were more likely than interventional trials to include sex or gender in their results sections (p=0.022). In studies that identified differences between women and men, there was an inadequate analysis of the underlying mechanism, and generally, there was a lack of analysis of gender-related factors in the studies, despite gender being an influential factor on risk, exposure and outcome.

While studies may not be sufficiently powered to allow meaningful subgroup analysis, the SAGER guidelines recommend integration of these dimensions at the onset and in the design of the studies to enable meaningful analysis. Regardless, making outcome data available disaggregated by sex is important to allow systematic reviews and meta-analyses, or reveal trends that merit further investigation.

Discussion of gender implications

The SAGER guidelines further recommend authors to discuss potential implications of sex and gender on the study results and ‘discuss the implications of the lack of such analysis on the interpretation of the results’.6 Thirteen studies (10%) (one interventional and 12 observational) mentioned sex or gender in their ‘Discussion’ sections, with observational studies more likely than interventional trials to do so (p=0.006). The implications were discussed in terms of suggested future research (three studies), or comparing results to previous findings, either confirming (seven studies) or refuting noted differences (three studies).

A renewed rationale for the SAGER guidelines

Our analysis confirms previous reports that have revealed inadequate reporting of sex disaggregated data and gender analysis in academic publications. In 2016, Avery and Clark examined 54 clinical trials published in prestigious medical journals and noted a similar level (41%) of female participation. While they noted a higher share (48%) of trials reporting sex-related analyses than our analysis, they noted that only 5% of the trials discussed any sex differences in the discussion section. A more recent article on cardiovascular interventional trials report that among articles published in 2017, only 23.5% included sex-specific efficacy endpoints and 8.6% sex-specific safety outcomes. This represents a decline since 2010 when the corresponding figures were 34.4% and 11.1%, respectively.10 In a recent systematic review by Brady et al of COVID-19 registered studies, only 4.1% mention sex and/or gender in the analysis, with only ‘20 of the 1381 interventional trials planning to consider sex as a variable upon analysis’.11 This analysis confirms the persistent lack of consideration for sex and gender at the planning stages of clinical trials on COVID-19 and in reporting. A recent report of the adverse effects following the administration of the Pfizer-BioNTech COVID-19 Vaccine in the USA reveals that more than 90% of anaphylaxis and non-anaphylaxis allergic reactions occurred in women, illustrating the importance of capturing these trends during clinical trials.12 Similar trends have been reported following the administration of the Moderna vaccine.

The low levels of adherence to the SAGER guidelines are in stark contrast to commitments to the SAGER guidelines and gender sensitive reporting by some leading journals, such as The British Medical Journal Global Health, The Lancet and The Journal of the American Medical Association. The gender blind reporting is particularly worrisome in the context of COVID-19, when scientific articles are being produced in an unprecedented volume and speed and risk generating biased and incomplete evidence that can hamper more tailored and effective response.11 13–15

These results are disappointing given the growing number of recent efforts to address the gender data gap in research and reporting practices. The SAGER guidelines were developed in 2016 as a tool for researchers, reviewers and editors to address the gender bias in research and reporting.6 Editors can play an important role encouraging researchers to conduct exploratory subgroup analysis ‘in response to reviewer and editorial comments’, to make sex disaggregated data available16 and to integrate sex and gender dimentions in to future reserach design.16

The SAGER guidelines have been widely cited, endorsed and adopted by publishers, such as Springer and Elsevier and an increasing number of journals, including those that have published some of the COVID-19 clinical trials in this systematic review.17 18 However, these commitments have not yet translated into systematic implementation or routine requirements by journals. By implementing the SAGER guidelines, editors could be instrumental in improving the reporting of sex and gender dimensions.

Despite the growing evidence illustrating important differences in health outcomes for women and men, sex and gender differences in COVID-19 interventional or observational studies are inadequately examined and reported. This highlights the necessity of renewing our commitment to the SAGER guidelines and working towards more gender sensitive research and reporting practices. This will ensure that research-based health interventions are equally safe and effective for women and men.

Data availability statement

All data relevnat to the study are included in the article or uploaded as supplementary information.