The More Things Change, the More They Stay the Same: A Study to Evaluate Compliance With Inclusion and Assessment of Women and Minorities in Randomized Controlled Trials

Abstract

Purpose

The National Institutes of Health (NIH) Revitalization Act of 1993 requires NIH-funded clinical trials to include women and minorities as participants and assess outcomes by sex and race or ethnicity. The objective of this study was to investigate current levels of compliance with these guidelines for inclusion, analysis, and reporting in NIH-funded randomized controlled trials (RCTs) and compare the results with those from 2009 and 2004, which the authors reported previously.

Method

The authors identified 782 RCTs published in 14 leading U.S. medical journals in 2015 with a PubMed search. Of those, 142 were the primary report of a NIH-funded RCT, conducted in the United States, and eligible for analysis. The authors reviewed abstract, text, and tables of each eligible study as well as any follow-up published commentary to determine compliance with NIH guidelines.

Results

Thirty-five studies limited enrollment to one sex. The median enrollment of women in the remaining 107 studies was 46%, but 16 (15.0%) enrolled less than 30% women. Twenty-eight of the 107 (26%) reported at least one outcome by sex or explicitly included sex as a covariate in statistical analysis. Of the 142 studies, 19 (13.4%) analyzed or reported outcomes by race or ethnicity. There were no statistically significant changes in inclusion, analysis, or reporting by sex, race, or ethnicity compared with the previous studies.

Conclusions

NIH policies have not resulted in significant increases in reporting results by sex, race, or ethnicity. The authors recommend strong journal policies to increase compliance with NIH policies.

Plus ça change, plus c’est la même chose. [The more things change, the more they stay the same.]

—Jean-Baptiste Alphonse Karr (Les Guêpes, January 1849)

Thirty-one years ago, the U.S. Public Health Service Task Force on Women’s Health Issues released a report outlining why the low representation of women in clinical trials had led to suboptimal women’s health care.¹ In response, the National Institutes of Health (NIH) released a 1986 policy advising scientists to examine sex/gender differences in all clinical trial outcomes.² With the NIH Revitalization Act of 1993, this policy became law.³ The act requires NIH-funded researchers to enroll women and ethnic/racial minorities, including women of childbearing age, into clinical research trials, and forbids the use of cost as a reason for their exclusion.³ Guidelines (amended in 2001) require that investigators discuss the inclusion of women and minorities in their applications for clinical research funding, report to the NIH the female and minority enrollment in funded studies, and conduct preliminary trials that provide enough information to inform the development of phase III trials, which are also required to test for sex differences as appropriate based on prior research.⁴ The ultimate goal of these recommendations is to ensure that all people, regardless of sex/gender or race/ethnicity, will fully benefit from advances in biomedical science.^5,6

In 2009, the NIH convened an internal task force to determine best practices for monitoring and tracking the inclusion of women and minorities, which led to the appointment of an Inclusion Policy Officer to oversee enrollment and ensure that inclusion guidelines are adequately followed.² The latest biannual inclusion report indicated that NIH clinical research enrollees were 57.1% female, 51.8% in non-sex-specific studies; 26.2% were from racial/ethnic minority groups; and that the NIH has succeeded in increasing representation of women and minorities in clinical research.⁷

Notwithstanding this achievement, a 2015 Government Accountability Office (GAO) report highlighted drawbacks to current tracking mechanisms that make it difficult to assess adherence to NIH recommendations on inclusion.⁵ Two critical shortcomings were the aggregation of biannual report numbers across NIH Institutes and Centers, which makes it difficult to discern if specific areas are failing to meet inclusion requirements, and the lack of adequate reporting mechanisms at the NIH to evaluate whether or not researchers are analyzing and reporting outcomes by sex.⁵

The GAO report recommends evaluating the peer-reviewed publications of NIH-funded research to monitor compliance with policies on inclusion, analysis, and reporting.⁵ Our group has conducted two prior studies examining the inclusion of women and minorities in NIH-funded randomized controlled trials (RCTs).^8,9 According to our findings, in 2004, 33% of sampled publications analyzed data by sex or explained why they did not, and in 2009, only 36% reported this information.^8,9 This was the case for race/ethnicity in 18% of 2004 publications and 21% of 2009 publications.^8,9 Further, as compared to 2004, our 2009 analysis found no significant change in enrollment of women or individuals from underrepresented racial/ethnic groups into clinical trials.^8,9

Since the publication of our 2009 study, other researchers have examined the inclusion of women and minorities as well as reporting by sex or race/ethnicity in specific fields (e.g., HIV, cardiology)^10,11 and in particular clinical trial types (e.g., phase III clinical trials),^12,13 finding similarly inadequate adherence to NIH policies. The purpose of this study was to repeat our evaluation of NIH-funded RCTs in a broad sample of high impact journals published in 2015 to determine whether rates of inclusion and reporting have changed since 2009, more than two decades since the passage of the NIH Revitalization Act.

Method

We identified RCTs published in 2015 through a PubMed search of 14 of the leading U.S. medical journals in the fields of general internal medicine and a wide range of medical specialty and subspecialty areas. Journals were selected if they published at least 25 RCTs in 2015 and had a 2014 impact factor in the top five for that medical specialty or subspecialty. Impact factor is a rating determined by Journal Citation Reports using the frequency with which articles are cited in a given year (retrieved from ISI Web of Science).¹⁴ Our journal selection criteria were intended to identify the leading journals that regularly publish RCTs. Though not necessarily representative of the entire spectrum of RCTs, studies published in these journals may be more accessible to clinicians and the general public compared with less well known or less widely read journals. Our PubMed search parameters included “randomized controlled trial,” English language, human subjects, and publication in calendar year 2015. In cases where an article was published online and also in print, we used the date of print publication for eligibility (see Table 1 for the journal names, impact factors, number of RCTs published in 2015, and number of eligible studies included in this analysis).

Table 1.

Characteristics of Journals Selected for Inclusion in the Analysis of NIH-Funded Randomized Controlled Trials Published in 2015, From a Study Evaluating Compliance With NIH Guidelines for Inclusion and Assessment of Women and Minorities in RCTs

Journal	2014 impact factor	RCTs published in 2015	Eligible for analysis	% of total RCTs eligible for analysis
American Journal of Psychiatry	12.3	25	15	60.0
Journal of Infectious Disease	8.9	44	13	29.5
Journal of the American Medical Association	35.3	72	20	27.8
Obstetrics & Gynecology	5.2	29	7	24.1
Journal of Clinical Oncology	18.4	114	22	19.3
Journal of the American Geriatric Society	4.6	32	6	18.8
American Journal of Obstetrics and Gynecology	4.7	34	6	17.6
New England Journal of Medicine	55.9	149	24	16.1
Diabetes Care	8.4	91	14	15.4
American Journal of Kidney Disease	5.9	24	3	12.5
Gastroenterology	16.7	28	3	10.7
Chest	7.5	37	3	8.1
Neurology	8.2	44	3	6.8
Journal of the American College of Cardiology	80.9	59	3	5.1
Total	19.5 (22.8)a	782	142	19.4 (13.8)a

Abbreviations: NIH indicates National Institutes of Health; RCT, randomized controlled trial.

^aReported as mean (standard deviation).

Several of the authors (A.R.K., P.R., A.F., E.H.) reviewed all articles meeting these criteria, identifying funding sources and dates of study recruitment. Letters, Brief Communications, and clinical trials that started recruitment before 1994, as well as studies with no NIH support, were excluded. We also excluded studies that were secondary reviews of previously completed RCTs, combined data from various RCTs, analyzed a subset of study subjects, included only subjects from outside the U.S., or analyzed or randomized at a unit other than the individual.

For analysis of sex-based reporting, we excluded studies from obstetrics and gynecology journals as well as those that enrolled only males or females. We did not exclude studies on conditions that may disproportionately affect members of one sex (e.g., autoimmune diseases). Similarly, studies focused on veteran populations were only excluded if they addressed a condition found only in men (e.g., prostate cancer). We analyzed articles for the inclusion of women and minorities and for reporting of sex-specific and race/ethnicity-specific results. We further recorded whether sex and/or race or ethnicity were taken into consideration during outcomes analysis and whether the authors mentioned the potential impact of sex, race, or ethnicity on the findings and/or their generalizability. Obstetrics and gynecology articles were evaluated for race/ethnicity-specific results only.

The abstract, text, and tables of each eligible manuscript were reviewed. In addition, we examined any follow-up articles or commentaries published by the author(s) or other researcher(s) for information related to sex, race, and ethnicity. When an article’s reporting of sex-specific or race/ethnicity-specific analyses or results was ambiguous, multiple reviewers on our research team evaluated and discussed the article until we reached consensus. The absolute numbers and percent distribution of samples across sex and race/ethnicity were recorded for all articles. We included absolute numbers because sample size drives the ability to find statistical significance. We compared our findings to our 2004 and 2009 findings using chi-square and Fisher exact tests as appropriate. The race/ethnicity portion of the analysis is limited to black and Hispanic subgroup reporting, as other racial/ethnic minorities were rarely and inconsistently reported. All analyses were conducted using SAS statistical software, version 9.3 (SAS Institute, Cary, NC).

To determine whether there was any relationship between the likelihood of adhering to NIH guidelines for inclusion of women in clinical research and the gender of the person leading the research, we assessed the gender of the first and senior (last) authors of studies in our final dataset. We assessed gender using the author’s name and completing an internet search to find photographs and text containing gendered pronouns (e.g., “She did her undergraduate work at…”) to verify the gender of the first and senior authors.

Results

We identified 782 studies in the 14 journals that met our search criteria. Of these, 142 studies were eligible for this analysis (Figure 1). The most common reasons for exclusion were absence of NIH funding for the study, participants residing solely outside the U.S., and reported results limited to a secondary or subgroup analysis of a parent study (see Supplemental Digital Appendix 1 [LWW INSERT LINK]).

Figure 1.

Flow chart of study eligibility for analysis, from a study evaluating compliance with NIH guidelines for inclusion and assessment of women and minorities in RCTs 2004, 2009, and 2015. Abbreviations: NIH, National Institutes of Health; RCT, randomized controlled trial.

Sex

Of the 142 studies, 13 were published in obstetrics and gynecology journals and were excluded from the analysis for reporting of sex differences. Of the remaining 129 studies, 22 enrolled only one sex, either because the study addressed a condition or disease that affects only one sex (e.g., pregnancy, prostate cancer) or tested an intervention targeted to one sex (e.g., estrogen modulation in Alzheimer’s disease, depression screening in obstetrics/gynecology clinics). Four (3.1%) of these sex-specific studies were restricted to male subjects and 18 (14.0%) were restricted to female subjects.

Among the 107 studies that were not sex-specific, the majority of studies (87, 81.3%) enrolled ≥ 30% women, and mean and median proportion of female enrollment was 47.3% and 46.0%, respectively. However, 16 (15.0%) of these studies enrolled less than 30% women and, of those, 7 (6.5%) enrolled less than 15% women without explanations. Four studies (3.7%) that were not explicitly sex-specific did not report the number of male or female subjects nor offer reasons for not reporting the sex of their participants.

There was no statistical improvement since 2009 in the proportion of studies that addressed and reported sex in their analyses. Twenty-eight of the 107 studies (26.2%) reported at least one outcome by sex or explicitly included sex as a covariate in statistical analysis (Table 2). Two additional studies (1.9%) did not include sex in their analyses, but explained why they did not, a statistically significant decrease compared with 2009 (P = .01). The remaining 77 studies (72.0%) did not mention whether sex was included in their analysis, did not report any sex-specific outcomes, and did not provide explanation. In only 4 of the 16 studies (25%) with female enrollment below 30% did the authors note that their findings may not be generalizable to women. We found follow-up publications (e.g., letters, editorials) for 9 of the 16 studies with <30% females enrolled; the only follow-up publication to comment on the low female enrollment was an editorial about a study of active-duty U.S. Army soldiers.

Table 2.

Inclusion and Analysis by Sex in NIH-Funded Clinical Trials Published in 2004, 2009, and 2015, From a Study Evaluating Compliance With NIH Guidelines for Inclusion and Assessment of Women and Minorities in RCTs^a

Studies	2004	2009	2015
Median % of women enrolled (interquartile range)	43 (25–61)	38 (28–54)	46 (34–56)
Analysis by sex provided or sex included in statistical analysis, no. (%)	6 (13.0)	14 (25.0)	28 (26.2)
Did not analyze by sex but provided explanation, no. (%)	9 (19.6)	6 (10.7)	2 (1.9)
Did not include sex in analysis or provide an explanation, no. (%)	31 (67.4)	36 (64.3)	77 (72.0)
Total, no. (%)	46 (100.0)	56 (100.0)	107 (100.0)

Abbreviations: NIH indicates National Institutes of Health; RCT, randomized controlled trial.

^aAmong studies including both male and female subjects.

Among the 107 studies that included both sexes, 38 studies (35.5%) had a female lead author, and 68 (63.6%) had a male lead author. The gender of the lead author could not be determined for one study. There was no difference by author gender in the proportion of studies that analyzed or reported their outcomes by sex: 10 of the 38 studies (26.3%) with female lead authors compared with 20 of the 68 studies (29.4%) with male lead authors. Likewise, 6 of the 27 studies (22.2%) with female senior authors reported sex-specific outcomes compared with 24 of 80 studies (30.0%) with male senior authors.

Race/ethnicity

There were no studies of a disease or condition that affected only individuals from a specific racial or ethnic background. The majority of studies reported the racial and/or ethnic composition of their participants (120, 84.5%) as well as the number of White or Black participants (115, 81.0% and 98, 69.0%, respectively), but Hispanic enrollment was not reported in over half of the studies (Table 3). Only 4 studies (2.8%) were conducted within a single racial or ethnic group. Among the 31 studies with <10% Black participants, 9 (29.0%) had follow-up published commentary, but none of the follow-up commentary mentioned limitations by race or ethnicity.

Table 3.

Reporting of Subjects by Racial/Ethnic Groups in NIH-Funded Randomized Controlled Trials Published in 2004, 2009, and 2015, From a Study Evaluating Compliance With NIH Guidelines for Inclusion and Assessment of Women and Minorities in RCTs

	White			Black			Hispanic
Studies	2004a	2009b	2015c	2004a	2009b	2015c	2004a	2009b	2015c
Studies with unknown number of subjects in given racial/ethnic group, no. (%)	17 (24.6)	23 (26.7)	27 (19.0)	23 (33.3)	37 (43.0)	46 (32.4)	36 (52.2)	52 (60.5)	76 (53.5)
Studies reporting <10% of subjects in given racial/ethnic group, no. (%)	0	0	1 (0.7)	9 (13.0)	14 (16.3)	31 (21.8)	19 (27.5)	19 (22.1)	30 (21.1)

Abbreviations: NIH indicates National Institutes of Health; RCT, randomized controlled trial.

^a2004 n = 69.

^b2009 n = 86.

^c2015 n = 142.

Analysis and reporting of study outcomes by race/ethnicity continued to be uncommon in 2015, with less than 5% (7) of studies reporting primary outcomes by race/ethnicity. In total, 19 studies (13.4%) either reported outcomes by race and/or ethnicity or included it in their analyses, a proportion statistically indistinguishable from that of earlier years. Similarly, the percentages of studies that did not report race- and/or ethnicity-specific outcomes or include race/ethnicity in their analyses in 2015 did not change as compared with 2004 and 2009.

Discussion

In this study of peer-reviewed articles published within 2015 in high impact journals and describing NIH-funded RCTs, we found that 15% of studies continued to enroll fewer than 30% women. Further, we found an overall underperformance in adherence to NIH guidelines in the analysis and reporting of women and individuals from racial and ethnic minority groups. There has been a slight increase, from 43% to 46%, in the median number of women enrolled in RCTs since 2004.

Our findings suggest that merely enrolling women and non-white participants in clinical studies does not translate to analysis and reporting by sex or race/ethnicity. More than two decades since passage of the NIH Revitalization Act, 72% (77) of studies in our review did not include sex in their analyses or provide explanations as to why not. As stated in the 2015 GAO report, the lack of adequate NIH reporting mechanisms to evaluate whether or not researchers analyze outcomes by sex remains a critical issue.⁵ In addition, reporting of the racial/ethnic composition of study participants did not improve since 2004.

Research on institutional equity directives and actual practice shows that formal policies often paradoxically result in greater bias because they diminish the onus on individuals to promote equity.^15,16 Thus, it is possible that having an overt NIH policy requiring the inclusion and reporting of data from women and minorities may inadvertently relieve investigators of the responsibility of actually conducting equitable research. It is also not surprising that we found that men and women researchers performed equally poorly in inclusion, analysis, and reporting of women in the studies they conducted. In nearly all controlled experiments, men and women participants are found to demonstrate the same degree of gender bias in judgment and decision-making.^17,18

There is a growing consensus that the results of clinical trials should be analyzed and reported for male/men and female/women subjects separately.^19,20,21 A Lancet panel¹⁹ proposes, and we support, that publishers work with funders to offer clear guidance to editors and authors on reporting results by sex and/or gender in clinical trials. The guidelines include correct use of the terms sex and gender; reporting sex and gender, where appropriate, for study participants, animals, and cells; analyzing data by sex and gender or making the raw data accessible for analysis; and studying the influence of sex and gender on results. Further, if sex and/or gender are not reported or analyzed, authors should justify this within the article. Regarding the correct usage of sex and gender, sex refers to the genetic differences between males and females and gender is socially constructed. However, the bidirectional interactions between biological and sociobehavioral processes are complex. Therefore, for convenience, we have used sex/gender throughout this manuscript.

As the Lancet panel¹⁹ notes, failure to account for sex and/or gender may mask differences in outcomes for men and women, limit researchers’ ability to reproduce study findings, and contribute to less than adequate patient care. For example, an RCT of naltrexone for drug and alcohol dependence found that it decreased use and severity in men but increased use and severity in women.²² Had these results not been analyzed and reported by sex, the apparent effect of the drug would have been attenuated and neither men nor women would have received appropriate care as a result.

Of the 14 journals studied, only the Journal of the American College of Cardiology has specific guidelines for authors to provide gender-specific data in reporting outcomes of epidemiologic analyses or clinical trials. Diabetes Care advises authors to report whether sex was included in statistical analysis, but does not require results to be reported by sex. None of the journals had a guideline for authors to report clinical trial results by race or ethnicity. The 2010 CONSORT Statement for RCTs²³ does not include reporting results by sex/gender, race, or ethnicity.

The primary limitation of this study is that by selecting studies published in 14 journals we do not examine reports from the full spectrum of NIH-funded RCTS. Because of this, our findings regarding enrollment, analysis, and reporting of women and minorities in RCTs may not be representative of the complete scope of NIH-funded RCTs and may differ from the rates reported in the NIH biannual inclusion report. However, we did select the leading 14 journals that publish RCTs from the fields of general internal medicine and a wide range of medical specialty and subspecialty areas. Our analysis included one large HIV-related international study, which may not reflect the pattern of recruitment of domestic studies.

We recommend the following remedial actions: open discussion among all members of research teams regarding implicit sex/gender and race bias that may affect participant recruitment and data analysis; creation of a sense of urgency by principal investigators leading clinical trials about the importance of broad representation among participants from the standpoints of equity and good science; strong author guidelines for reporting clinical trial results by sex, race, and ethnicity within peer-reviewed journals; and revision of NIH grant policy to score grant applications on their plans for representation, recruitment strategies, and whether they will be powered to detect differences between groups.

Our findings indicate that federal law alone is insufficient to change practice in the conduct of NIH-funded RCTs regarding the inclusion, analysis, and reporting of women and ethnic/racial minorities. The lack of progress towards the goal of better understanding sex-specific and race/ethnicity-specific differences in biomedical research limits our ability to provide the best clinical care to all people.

Table 4.

Analysis by Racial/Ethnic Groups in NIH-Funded Clinical Trials Published in 2004, 2009, and 2015, From a Study Evaluating Compliance With NIH Guidelines for Inclusion and Assessment of Women and Minorities in RCTs

Studies	2004, no. (%)	2009, no. (%)	2015, no. (%)
Analysis by race/ethnicity provided or variable included in statistical analysis	6 (8.7)	12 (14.0)	19 (13.4)
Did not analyze by race/ethnicity but provided explanation	6 (8.7)	6 (7.0)	3 (2.1)
Did not include race/ethnicity in analysis or provide an explanation	57 (82.6)	68 (79.0)	120 (84.5)
Total	69 (100.0)	86 (100.0)	142 (100.0)

Abbreviations: NIH indicates National Institutes of Health, RCT, randomized controlled trial.