Diversity in Medical Device Clinical Trials: Do We Know What Works for Which Patients?

Abstract

Policy Points:

• A 1993 law required the National Institutes of Health to include women and racial and ethnic minorities in relevant research studies. Most federal health agencies adopted the same policy, but the US Food and Drug Administration (FDA) did not.

• A 2012 law encouraged the FDA to ensure that new medical products be analyzed for safety and effectiveness for key demographic patient groups.

• Our study of high‐risk medical devices reviewed by the FDA in 2014‐2017 found that due to lack of patient diversity and publicly available data, clinicians and patients often cannot determine which devices are safe and effective for specific demographic groups.

Context

Demographic differences can influence the safety and effectiveness of medical devices; however, clinical trials of devices for adults have historically underrepresented women, people of color, and patients over age 65. The US Food and Drug Administration (FDA) Safety and Innovation Act became law in 2012, encouraging greater diversity and subgroup analyses. In 2013, the FDA reported that there was diversity in clinical trials considered “pivotal” for approval decisions and that subgroup analyses were conducted for most applications for the highest‐risk medical devices. However, the FDA's report did not specify whether analyses included sufficient numbers to be meaningful, whether analyses were conducted for most major subgroups, or whether analyses included safety, effectiveness, or accuracy.

Methods

We examined publicly available documents for all 22 medical devices that the FDA designated “highest risk” or “novel,” were reviewed through the premarket approval pathway, and were scrutinized at FDA public meetings from 2014 to 2017. We evaluated patient demographics and subgroup analyses for all pivotal trials.

Findings

Only 3 (14%) of the devices provided subgroup analyses for both effectiveness and safety or both sensitivity and selectivity for gender, race, and age. However, 55% of the devices reported both of those subgroup analyses for at least 1 of the 3 subgroups. Whether analyses were reported or not, the number of patients in most subgroups was too small to draw meaningful conclusions. Subgroup analyses were more likely to be reported to the FDA's Advisory Committees than in the FDA's public reviews or labeling.

Conclusions

Despite a law encouraging more diversity and subgroup analyses in pivotal trials used as the basis for FDA approval, the results of our study indicate relatively few subgroup analyses are publicly available for the highest‐risk and novel medical devices. The lack of subgroup analyses makes it impossible to inform patients or physicians as to whether many newly approved medical devices are safe and effective for specific demographic subgroups defined by gender, race, and age.

Demographic differences among individuals, such as gender, race/ethnicity, and age, have been shown to influence the safety and effectiveness of certain drugs and medical devices. The extensive research literature indicates that differences observed between these subgroups may be attributed to numerous factors, including metabolism, tissue shape or size, or genetic differences such as the likelihood of carrying a specific allele.1, 2, 3 Analyses within relevant subgroups and sufficient participation for meaningful analysis are necessary for the US Food and Drug Administration (FDA) to make decisions about indications for approval, since indications include types of patients as well as specific diseases or conditions. Subgroup analyses are also needed for patients and health professionals to make informed decisions about treatment options.

In response to the lack of diversity in clinical trial populations more than 2 decades ago, President Clinton signed a law in 1993 that requires the National Institutes of Health (NIH) to include women and racial and ethnic minorities in research studies when the research is relevant to those populations.4 The US Department of Health and Human Services subsequently adopted the same policy.5 However, the studies used by the FDA for determining which medical products can be sold in the United States were exempted from that requirement. The explanation for the exemption from the diversity requirement was that studies conducted for FDA product review are designed and funded by industry or other private entities, rather than government employees or US taxpayers.

The FDA has been attempting to address the lack of diversity in clinical trials used for medical product approvals for 3 decades. In 1988, the FDA issued guidance recommending the analysis of effectiveness and safety data by gender, race, and age for new drug applications.6 In 1994, the Office of Device Evaluations at the Center for Devices and Radiological Health (CDRH) indicated that device applications and summaries of safety and effectiveness data (SSEDs) should address the potential for differences in the selection of participants between the genders and differences in safety and effectiveness;7 however, the impact of that recommendation has not been reported. Also in 1994, the FDA issued a guideline to encourage representation of elderly patients in clinical trials for drug and biologic applications.8 It has only been in the past few years that the CDRH released guidance on evaluation and reporting related to age, race, and ethnicity for medical devices and guidance on evaluating sex for devices.9, 10

Although the FDA bases its approval decisions for drugs and the highest‐risk medical devices on data designed to determine whether each medical product is safe and effective for specific types of patients for specific indications, the agency can recommend but does not require companies that are responsible for the clinical trials to include specific demographic groups in those trials or subgroup analyses.

Research conducted in the past decade indicated that studies submitted to the FDA did not include adequate demographic diversity or subgroup analyses; for example, studies of high‐risk cardiovascular devices tended to include few women or people of color.11, 12 In response, Congress addressed the problem in the FDA Safety and Innovation Act (FDASIA), which became law in 2012.13 Rather than requiring diversity in clinical trials, however, FDASIA directed the FDA to investigate to what extent demographic subgroups were included in clinical trials and whether or not subgroup‐specific safety and effectiveness data were available. The FDA's report, submitted in 2013, concluded that “the majority of applications submitted to FDA include demographic subset analyses.”14 Unfortunately, the FDA's report did not specify whether analyses were conducted for most major demographic subgroups (eg, age and race as well as gender; all 5 major racial groups or only black or nonwhite) or whether they were statistically significant or clinically meaningful.

Clinical trials demonstrating safety and effectiveness for devices are only required for devices that the FDA considers to be high risk, defined by the agency as life‐sustaining, life‐saving, or potentially posing an unreasonable risk of harm. Thus, while clinical trial data are typically available for devices that go through the premarket approval (PMA) and humanitarian device exemption pathways, they are rarely available for low‐ or moderate‐risk devices that go through 510(k) (premarket notification) and de novo pathways.15, 16, 17, 18

The purpose of this study was to evaluate whether the information provided to the public is sufficient to determine whether the highest‐risk medical products are safe and effective for all patients who fit the indication. We examined all of the pivotal clinical trials used as the basis for FDA PMAs for medical devices that were of sufficient importance or controversy to be publicly reviewed by an FDA Advisory Committee in the period 2014‐2017. We looked at products discussed at Advisory Committee meetings because these provide more publicly available clinical trial data. In addition to the FDA's SSEDs and labeling, we examined briefing materials containing reviews of these devices that were provided to the committees and the public by the FDA and the manufacturer. The briefing materials are commonly much longer than the SSED and labeling, so they may contain additional analysis.

This study assesses the extent to which demographic subgroups were included in the trials, as well as the presence of analyses by gender, race/ethnicity, and age for effectiveness and safety endpoints in public documents. We illustrate the need for sufficient numbers of patients from each subgroup for analyses to be able to conclude whether the device should be used for those subgroups. We further compare the data available in briefing documents to the FDA Advisory Committee to the review and labeling materials, which are available for all devices, to estimate the extent to which these data are publicly available.

Methods

We examined the clinical trials that the FDA considered “pivotal” to determining safety and effectiveness for all 22 medical devices used for diagnosis or treatment that were publicly reviewed at an FDA Advisory Committee meeting between 2014 and 2017. Medical devices reviewed at public FDA Advisory Committee meetings comprise the most controversial of the highest‐risk devices that the FDA considers for approval. For these devices, the FDA provides the public with detailed clinical trial data analyzed by the device manufacturer as well as by FDA scientific reviewers; these documents are commonly longer and contain additional information relevant for approval decisions that may not be publicly available for other high‐risk devices. Pivotal trials were selected because they are the only trials for which detailed information about demographics and subgroup analysis are consistently available, either in FDA or company documents provided to the FDA Advisory Committee, through SSEDs, or on labeling. Device name, indication, demographics of trial participants, and the presence of subgroup analysis for gender, race/ethnicity, and age were extracted from these company and FDA documents. By law, FDA and manufacturer Advisory Committee summary documents are made publicly available at https://www.fda.gov/advisorycommittees/default.htm. Product labeling and the FDA's public reviews for each approved device are available through the PMA device database at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/pma.cfm. The product labeling is written by the manufacturer and must be approved by the FDA, to provide physicians and patients with information about the risks and benefits of the product. The SSED is drafted by the product's manufacturer and edited by the FDA to present and critique the scientific evidence used to make the approval decision.19 Although the SSED and labeling provide clinical trial data similar to data provided to the Advisory Committee, they are not identical.

The documents were examined for each device to determine whether subgroup analysis was performed by either the FDA or the manufacturer. We categorized a subgroup analysis as “provided” if the data analysis was presented as a graph or table or in the narrative. We categorized a subgroup analysis as “mentioned” if the documents included a statement that the analysis occurred but did not include the data itself. For the 19 devices used for treatment, reviews were examined for subgroup analyses for safety and effectiveness. For the 3 in vitro diagnostic (IVD) devices, which are used to diagnose cancer, reviews were examined for subgroup analyses related to sensitivity and selectivity. Although subgroup analysis best practice typically calls for a test of interaction of the treatment effects across subgroups,20 that is not the best strategy for identifying safe and effective treatments for patients. For example, a drug can be more effective for younger patients than older ones, but very beneficial for both. Using that patient‐centered approach, we identified examples of potential differences between subgroups as defined by (1) statistically significant findings, based on the FDA/manufacturer‐specified p‐value (typically 0.05); (2) lack of statistically significant improvement for all subgroups or a statistically significant safety issue; or (3) an instance where the manufacturer states that a difference between subgroups was observed. In studies where the FDA or manufacturer used a p‐value of 0.1 or 0.15, we specified that less stringent statistical standard.

Results

Twenty‐two medical devices in the PMA pathway were evaluated at FDA Advisory Committee meetings from 2014 to 2017. Of these, 18 were subsequently approved by the FDA as of June 2018 (Table 1); given the potential time lag between public meetings and FDA approval decisions, it is possible that 1 or more of the other 4 will be approved in the future. These 18 represent less than 15% of all devices approved through the PMA pathway during those years.21

Table 1.

Devices Considered for FDA Approval With Advisory Committee Meetings, 2014‐2017 (n = 22)

					Subgroup Analysis
	Product Name	Purpose	Meeting Date	Approval Date	Gender	Race	Age
Treatment Devices	AcrySof® IQ ReSTOR® Toric Multifocal Intraocular Lens	Implanted lens for correction of aphakia	11/14/2014	12/22/2016	―	Safe Eff	Eff
	Inspire® II Upper Airway Stimulator	For moderate to severe obstructive sleep apnea	2/20/2014	4/30/2014	Effb	―	Effa, b
	KAMRA® Inlay	Corneal implant to improve presbyopia	6/6/2014	4/17/2015	Safeb Effb	Safeb Effa, b	Safea, b Effa, b
	LUTONIX® 035 Drug Coated Balloon PTA Catheter	Open blocked arteries in thigh and knee	6/12/2014	10/9/2014	Safeb Effa, b	―	Safeb Effb
	MAESTRO® Rechargeable System	Nerve modulator for morbid obesity	6/17/2014	1/14/2015	Effb	Effb	Effb
	ResQCPRTM System	CPR assist device	5/6/2014	3/6/2015	Safeb Effb	―	Safeb Effb
	TissuGlu®	Tissue adhesive for large‐flap surgeries	8/1/2014	2/3/2015	―	Safe Eff	Safe Eff
	Visian Toric Implantable Collamer® Lens (TICL)	Implanted lens for myopic astigmatism	3/14/2014	Pending	―	―	―
	WATCHMANTM Left Atrial Appendage Closure Technology	Prevent thromboembolism	10/8/2014	3/13/2015	Effb	―	Effb
	Radiesse® Dermal Filler	Dermal filler to add volume to hands	2/28/2015	6/4/2015	―	―	Safeb Effb
	VertiFlex® Superion® InterSpinous Spacer	Spacer to treat lumbar stenosis	2/20/2015	5/20/2015	Safeb Eff	―	Safeb Effb
	Absorb GT1™ Bioresorbable Vascular Scaffold (BVS) System	Absorbable drug‐eluting coronary stent	3/15/2016	7/5/2016	Safeb Effb	Effb	Effb
	AMPLATZERTM Patent Foramen Ovale (PFO) Occluder	Implant to block hole in heart septum	5/24/2016	10/28/2016	Effb	―	Effb
	AngelMed Guardian® System	Implanted cardiac monitor	3/16/2016	4/9/2018	Effb	―	Effb
	Cartiva® Synthetic Cartilage Implant (SCI)	Treat arthritis in big toe	4/20/2016	7/1/2016	Effb	―	Effb
	DIAM® Spinal Stabilization System	Spacer to treat lumbar stenosis	2/19/2016	Pending	―	―	―
	TOPAS™ Treatment for Fecal Incontinence	Mesh anal sling for fecal incontinence	2/25/2016	Pending	NA	―	Effb
	Barricaid® Anular Closure Device	Prevention of lumbar disc reherniation	12/12/2017	Pending	Safe Eff	Safe Eff	Safe Eff
	Organ Care SystemTM (OCSTM) Lung System	Preservation/transport of donor lung	5/17/2017	3/22/2018	Effa, b	―	―
IVDs	Cobas® HPV Testc	Primary screening for cervical cancer	3/12/2014	4/24/2015	NA	―	Sena, b Sela, b
	CologuardTM	Screen for colorectal cancer	3/27/2014	8/11/2014	Sena, b Sela, b	Sena, b Sela, b	Senb Selb
	Epi proColon®	Screen for colorectal cancer	3/26/2014	4/12/2016	―	Senb Sela, b	Senb Sela, b

Abbreviations: Eff, effectiveness; Safe, safety; IVD, in vitro diagnostic device; NA, not applicable; Sel, selectivity; Sen, sensitivity.

― Did not conduct subgroup analysis.

^aDifference observed.

^bShowed data for subgroup analysis.

^cApproval for new indication.

Of these 22 devices, 4 were based on 2 pivotal trials and the rest were based on a single pivotal trial. All pivotal trials for treatment devices were prospective, as were all except 1 IVD trial. For the treatment devices, the clinical trial was randomized for 18 trials, and the device was compared to a comparator, standard care, or sham. The nonrandomized trials included comparisons to baseline, to a delayed treatment group, or to a predetermined effect level. The 3 IVDs were compared to control methods for all patients.

The number of patients in each clinical trial ranged from 114 to 2,008 for treatment devices and from 301 to 40,944 for IVDs. Most clinical trials were powered to support primary and, in many cases, secondary endpoints for the whole trial population. None were reported to be powered to support statistical evaluation of subgroups.

When subgroup analysis was presented, it was typically for the primary endpoint(s) and sometimes also for secondary endpoints. Most analyses were considered exploratory. Only rarely were the analyses described as adjusted for multiple tests or prespecified.

Gender

Eighteen of the 19 treatment devices and 2 of the 3 IVDs were intended for both men and women. Information about patients’ gender was available for all pivotal trials. The number of patients in the minority gender in a clinical trial ranged from 1 (1%) to 593 (30%) for treatment devices and 146 (49%) to 4,645 (46%) for IVDs (Figure 1A). The pivotal trials for 11 devices (55%) included at least 35% of the minority gender. Women made up less than 35% of participants in pivotal trials for 5 devices, primarily cardiovascular devices, and made up more than 65% of participants for 4 devices, relating to plastic surgery, obesity, and orthopedic surgery.

Figure 1.

Pivotal Trials That Included Patient Subgroups and Subgroup Analyses

Trial counts are based on either the number of patient participants who were of the minority gender (A) or who were nonwhite (B) or the mean age of patients (C). Dark bars indicate trials for which subgroup analysis was presented in public documents, and light bars indicate that the analysis was not publicly available. Trials that did not provide the necessary demographic information were not included in the graph.

While documents for 72% (13/18) of treatment devices included subgroup analysis by gender for effectiveness in at least 1 pivotal trial, only 33% (6/18) included it for safety as well (Table 1). Documents provided subgroup analysis data in the form of a table, graph, or listing for effectiveness for 87% (13/15) of the devices and for safety for 86% (6/7) (Table 1).

Treatment devices that included subgroup analysis included 21 patients (17%) to 593 patients (30%) of the minority gender. One of the 2 IVDs that included analysis by gender included it for both selectivity and sensitivity and included 4,645 (46%) patients of the minority gender. The other included analysis for sensitivity only and had 146 (49%) patients of the minority gender. Trials for devices that did not include subgroup analysis in materials included between 1 (1%) and 725 (47%) patients of the minority gender (Figure 1A).

Race and Ethnicity

Information about patients’ race was available for the pivotal trial(s) for 20 (91%) devices. For each pivotal trial, white patients made up 69% to 99% of patients. The number of nonwhite patients in pivotal trials for treatment devices and IVDs ranged from 4 (3%) to 280 (17%) and from 91 (30%) to 6,788 (17%), respectively (Figure 1B). Most pivotal trials included categories for white, black, and Asian, with fewer including other categories.

All of the devices that included information about race also included information about Hispanic background. Approximately half of the pivotal trials included Hispanics as a race, rather than an ethnicity. Pivotal trials listed 1 (1%) to 73 (4%) Hispanic patients for treatment devices and 41 (14%) to 7,370 (18%) for IVDs. Only 7 (37%) of the treatment device trials included subgroup analysis by race or ethnicity for effectiveness, and only 3 (21%) included analysis for safety (Table 1). Two of the 3 IVD tests included subgroup analyses for sensitivity and selectivity. Of these 9 devices, 3 looked at white and nonwhite patients, 4 looked at 3 or more races, and 2 did not specify which groups were included. Six (67%) showed data for the analyses presented, while the others stated that analyses were conducted without presenting the data (Table 1).

Pivotal trials for the treatment devices included 8 (2%) to 250 (12%) patients per minority racial category used in the subgroup analysis. Pivotal trials for IVDs, all tests for cancer, included 12 (4%) to 1,071 (11%) patients per minority racial category; however, only a small number of patients had cancer and thus contributed to the subgroup analysis on sensitivity. The treatment devices for which subgroup analyses for race or ethnicity were not available included 4 (3%) to 6,788 (17%) nonwhite patients in each clinical trial.

Age

All of the devices were tested primarily on adults. Age ranges and means were available for 19 (86%) devices for at least 1 pivotal trial. The mean age of clinical trial participants ranged from 35 to 74 years (Figure 1C). The minimum age of patients ranged from 18 to 50 years old, and the oldest patients ranged from 45 to 95 years old. Three devices included a reference to age in their indication, which corresponded to approximately the age range evaluated in the pivotal trial.22, 23, 24 Additional information about the number of patients in subsets of age ranges was provided for pivotal trials for 9 (41%) devices.

Most pivotal trials for devices indicated for all adults included at least 1 patient who was at least 65 years old. While most trials did not list the number of patients who were over 65, the patients in pivotal trial(s) for 7 (32%) devices had a mean or median age ranging from 64 to 74. However, 2 other devices were approved for all adults even though the oldest patients in their studies were 65 (mean was 47) or 67 (means were 42 and 43).

Of the 19 treatment devices, 15 (79%) conducted subgroup analysis by age for effectiveness, and 7 (37%) provided subgroup analysis of safety as well (Table 1). All 3 IVDs included subgroup analyses for sensitivity and selectivity. Thirteen (72%) devices presented data for all the analyses described, while the other 5 simply mentioned that the analysis was conducted or stated that age did not significantly affect the results. Age was analyzed either as a continuum or as discrete age categories.

Subgroup analyses for older populations were conducted for 7 devices. These included at least 1 age group that was 65 or older based on the mean or median age, quartiles, or preset age groups, such as 5‐ or 10‐year age categories or 75 years and over. These trials included between 37 (24%) and 961 (48%) patients in their older age group.

Location of Subgroup Data

We next evaluated where subgroup analyses were available. Fifteen treatment devices and the 3 IVDs were approved and therefore had SSEDs and labeling.

Subgroup analysis was primarily available only in the public documents provided to the FDA Advisory Committee. For the 10 (71%) of 14 instances for which subgroup analyses were conducted for safety on treatment devices, the results were included only in the summaries provided to the Advisory Committees. For 3 devices, information was provided in all 3 documents. Of the 31 instances when subgroup analyses were performed for effectiveness, 19 (61%) were available only in the Advisory Committee summary. For 7 devices, analyses were available in the Advisory Committee summary as well as the review and labeling. For 1 device, analyses were included in the labeling and review documents that were not included in the Advisory Committee summaries.

Subgroup information for the IVDs was more consistent. For 2 IVDs, all subgroup analyses were presented in the Advisory Committee summary, SSED, and labeling, comprising 4 tests for sensitivity and for specificity. For only 1 device, a subset (3 of 5) of the analyses that were available in the Advisory Committee summary were available in the SSED, and only 1 analysis was available in the labeling. Differences between subgroups were more common with IVDs, which might explain why the analyses were included in the labeling and reviews more often (Table 1).

Subgroup analyses found 14 potential differences in effectiveness and safety or sensitivity and specificity for at least 1 demographic group for 6 approved devices (Table 1). In 8 (57%) of these cases, the analyses were presented or discussed in all 3 documents. In 2 additional cases, the analyses were discussed in both the Advisory Committee summary and either the labeling or the SSED. The 4 subgroup analyses that were not available in the labeling or SSED were for 2 devices.

These are examples where differences within demographic subgroups were discussed in the labeling and SSED and led to a warning or notice in the device labeling.

• Gender was a statistically significant covariant for effectiveness for the LUTONIX drug‐coated balloon catheter, which is used to reduce blockage in the leg.25, 26 The device was found to be effective in the total patient sample, but that was due to its effectiveness in men. When data were stratified by gender, the results did not demonstrate effectiveness for women. In fact, women assigned to the LUTONIX study arm had slightly worse outcomes than women in the control arm of the study. One year after the procedure, the artery remained sufficiently dilated in 56% (57/101) of women in the LUTONIX study arm, compared to 61% (27/44) of women who were given an uncoated balloon catheter. The outcome with LUTONIX was significantly worse for US women, where only 51% (36/71) showed effectiveness with the LUTONIX coated balloon catheter compared to 70% (19/27) with the control balloon catheter.

• The selectivity of the Epi proColon test for colorectal cancer statistically significantly decreased with increasing age. Overall, the percentage of patients that incorrectly tested positive was 21% (218/1,500). The likelihood of a false positive increased from 16% (100/611) for patients 50‐59 years to 26% (88/337) for patients older than 69 years.

In other cases, the analysis was included in labeling or SSEDs, but the number of patients was so small that differences were assumed by the FDA and the manufacturer to be an artifact.

• Sensitivity of Cologuard test for colorectal cancer was found to be significantly affected by gender and race. It was tested on more than 4,500 patients of each gender and more than 900 white, black, and Hispanic patients. However, only a small number of patients had cancer. Sensitivity was significantly higher for males (100%: 34/34) than for females (84%: 26/31).27, 28 The sensitivity was also significantly higher for white patients (96%: 53/55) compared to black patients (63%: 5/8).

Other differences led to a general statement in labeling or SSED or were not included.

• The specificity of the Epi proColon test was affected by race. The false‐positive rate was significantly higher for black patients (27%: 70/258) than for other races (white, 20%: 221/1,093; “other,” 18%: 27/149).29, 30 This difference in specificity interacted with age. For white patients the false‐positive rate was significantly lower for patients 50‐59 years old (13%: 56/419) than for patients over 69 years old (26%: 68/263). However, for black patients there was no significant difference between the oldest and youngest age groups (26%: 33/127 vs 29%: 12/42).

• The effectiveness of KAMRA corneal implants may have been influenced by race or age. The company evaluated subgroup interactions using the less stringent p‐value of 0.15. After a year, all 8 patients who were categorized as black or “other” were more likely to have improved near vision of at least 20/40, as were 23 of the 24 Asian patients.31, 32 However, white patients (81%: 340/421) and Hispanic patients (72%: 18/25) were statistically less likely to have this level of improvement. It is plausible that individuals with specific racial backgrounds would have different outcomes because the shape of the cornea differs among racial groups.33, 34 However, the numbers of patients in the nonwhite groups were small. In addition, patients 45‐49 years old were significantly more likely to have improved near vision of 20/40 after 1 year than individuals 55‐60 years old (88%: 133/152 vs 78%: 87/111, respectively).31, 32 Paradoxically, there was also a small but statistically significant increase in the number of patients who had worsening distance vision in the younger age group compared to the older group (0%: 0/111 for 55‐ to 60‐year‐olds vs 2%: 3/153 for 45‐ to 49‐year‐olds).

Discussion

Despite a 2012 law urging the FDA to improve information about demographic subgroups, and an FDA report concluding that the analyses are being conducted, device review and labeling documents for only 3 of 22 devices included information about both safety and effectiveness or sensitivity and specificity for all applicable demographic subgroups. Of those that discussed subgroup analyses for both safety and effectiveness or for sensitivity and specificity, 7 did so for gender, 6 for at least 2 racial or ethnic groups, and 10 for age. Although these documents included either safety or effectiveness analyses by age for 18 devices, only 7 included analyses of age groups that would be relevant to coverage decisions for Medicare, which primarily serves patients 65 and over. Even when analyses were included, they rarely reflected best practices. Information was rarely included about whether analyses were prespecified, how many subpopulations were tested, and if there were multiplicity adjustments. None of the analyses stated that the trials were powered to detect treatment effects within a demographic subgroup.

The FDA encourages but does not require subgroup analyses as part of the application process, and there were often too few patients in major demographic subgroups to have confidence in an analysis of safety or effectiveness. As a result, even when analyses were conducted and differences were noted, FDA reviewers often stated that the difference was “probably” due to chance given the small number of patients. However, when demographic differences were not statistically significant, FDA reviewers did not point out that it could be due to the study being underpowered for subsets of patients. As noted previously, statistically significant differences between demographic groups provide less useful information to patients and providers than within group analyses that determine whether a product is safe and effective for each major group that is likely to use it. For example, if 100% of men benefit but only 90% of women benefit, a device would still be very beneficial for both. However, when women in a control group have better outcomes than the women using a device (as was the case with LUTONIX), then that is essential information for physicians and patients, even when the sample size is relatively small.

Historically, clinical trial participants have been relatively young, white males.35, 36 Although our study found similar numbers of women and men were included for some conditions, for other conditions one gender was dramatically underrepresented. In addition, our study indicated that people of color were still dramatically underrepresented in clinical trials submitted to the FDA as the basis for approval, similar to previous studies examining participation in pivotal trials for novel therapeutics.37, 38 This underrepresentation should be considered unacceptable, since more than 50% of the US population will be nonwhite by 2044.39 There is limited documentation of the number of patients over 65 in pivotal trials and even less information about safety or effectiveness for those patients. Patients 65 and older are a crucial subgroup to analyze, since most Medicare patients are in that age group and such information could be useful as Medicare considers coverage policies for specific devices. This is of particular concern because the use of medical products, including devices, increases with age.

The FDA's 2013 congressionally mandated report on diversity found that a large percentage of device applications included subgroup analyses for gender and age while fewer included race or ethnicity.14 Our study found that review and labeling documents for few devices included analyses for all 3 demographic categories, the analyses frequently did not include both safety and effectiveness, and the analyses often included too few patients in a subgroup to provide meaningful information about those patients. If additional subgroup analyses were conducted by the company or by the FDA, the results were not made publicly available. Despite the FDA's statements that such subgroup analyses are very important, only a small percentage of public reviews or official device labeling included information about these analyses, even when they were available in documents provided to the Advisory Committees. Similarly, an analysis of subgroup analysis of all PMA devices approved in 2015 found that less than 10% of clinical trials listed in the SSED or labeling included analysis of age or race and/or ethnicity, while less than 20% included analysis by gender.40

In rules and official guidance documents that the FDA has released since at least 1985, the agency has acknowledged that different demographic subgroups may respond differently to a treatment.41 In addition to the 2013 report cited earlier, the 2012 FDASIA law required the FDA to create an action plan aimed at enrolling more diverse, representative populations and to have sufficient data to determine safety and efficacy for the intended population.41 The FDA asked stakeholders for feedback at a public meeting in April 2014 and released its action plan in August. Numerous stakeholders, such as our center's Cancer Prevention and Treatment Fund, the National Women's Health Network, WomenHeart, and the Society for Women's Health Research, recommended that the FDA use enforcement mechanisms, such as requiring companies to include necessary populations and subgroup analyses if they want to have their products approved for all adults.42 However, the FDA did not incorporate those recommendations in the subsequent action plan. Instead, the FDA emphasized the need for the agency to “work in concert with the National Institutes of Health (NIH), advocacy groups, and industry to raise awareness and identify best practices for how to improve the inclusion of demographic subgroups in biomedical research.”41 The FDA's action plan focused on 3 goals, none of which included enforcement: (1) improve the completeness and quality of demographic subgroup data collection, reporting, and analysis; (2) identify barriers to subgroup enrollment in clinical trials and employ strategies to encourage greater participation; and (3) make demographic subgroup data more available and transparent. Although the FDA does not have the authority to require companies to conduct meaningful subgroup analyses, it can indirectly enforce its recommendations through its approval decisions. For example, the FDA has stated that clinical trial “enrollment should reflect the patients most likely to use a medical product” and that “we do require that each manufacturer provide sufficient data to evaluate the safety and efficacy for the intended population.”43 That suggests that the lack of data on patients of color and those over the age of 65 that we documented in our study would justify the FDA approving a device only for the demographic groups for which the device was demonstrated to have benefits that outweigh the risks.

The FDA creates a loophole in enforcement when it states, “If a subgroup is known to have a significantly different response than the rest of the population, or if a specific claim is sought in a certain subgroup, additional analyses may be needed.”43 This is a loophole because it is often difficult if not impossible to “know” of a significantly different response if no subgroup analyses have been conducted. Moreover, the above statement implies that subgroup analyses are only needed to make claims about specific subgroups, not to make claims of safety and effectiveness for “all adults,” despite an absence of evidence for specific subgroups.

The lack of robust subgroup data for many medical products, as shown in this study, is of concern given the known demographic differences that affect specific medical conditions and treatment outcomes. For example, 1 of the devices in our study, Inspire, is a treatment for sleep apnea. Risk factors for sleep apnea include being male, over age 60, postmenopausal, or of black or Asian descent. Some of these factors are thought to have a biological basis.44, 45 The angle at which the head sits on the neck and the structure of the neck and mouth differ between races, which may reduce the size of the airway during sleep. The distribution of fat and hormone levels may account for the difference in incidence between premenopausal women and men. Further evidence for this is that while men have almost twice the risk of sleep apnea as premenopausal females, after menopause women have the same or greater risk as men regardless of age or body mass index. Other comorbidities and environmental factors increase the incidence of sleep apnea and its severity. These incidence factors also vary according to health issues such as obesity, smoking, and asthma that may be associated with specific demographic groups. Yet, the Inspire device was only tested on a small number of nonwhites and an unknown number of patients over 60.46, 47 The labeling included analyses showing that age influenced the likelihood that the device would be effective, but gender did not.48

In our study, the Advisory Committees discussed the relevance of subgroup analysis in at least 2 cases, the apparent ineffectiveness of LUTONIX in women and the lower specificity of Epi proColon in black patients and patients over the age of 75. Despite these findings, both of these devices were approved for adults in all demographic groups. For LUTONIX, the committee suggested that the FDA require a postapproval study to clarify effectiveness for women and men,49 and that concern was stated in the labeling.50 In the case of Epi proColon, the panel suggested that a warning be included in the device labeling indicating that there is an increased false‐positive rate for older patients; the FDA complied with this recommendation.51, 52 However, the panel discouraged including a similar warning for black patients due to the concern that black patients would opt out of the test, which was seen as problematic since blacks are already at risk for underscreening. The panel suggested the FDA update the labeling when postmarket data were collected on black patients, but the postmarket study that is under way does not specifically mention race as a variable being examined.53

Based on our results, it is clear that subgroup analyses are not always reported publicly, even when adequate analyses are conducted. It was sometimes impossible to know whether the information was not provided because there were no significant differences, because subgroup analyses were not conducted, or because results were not considered meaningful due to underpowered analyses. Since Congress and the FDA have publicly stated that subgroup analyses are important, the FDA should require such analyses be conducted and the results made public, especially on labeling, medical guidelines, and patient materials.

Limitations of the Study

The main limitation to this study is the small number of devices that was examined. We analyzed only the subset of highest‐risk devices that the FDA considered important or controversial enough to warrant a public Advisory Committee meeting between 2014 and 2017. Focusing on the highest‐risk devices was necessary because only devices that the FDA categorizes as the highest risk (Class III) are required by law to provide clinical trial data to the FDA. Since the FDA very rarely requires clinical trial data for any but the 1% of medical devices it considers highest risk,54 subgroup analyses are not likely to be conducted for any but the highest‐risk devices, such as those in this study.

To maximize the likelihood that subgroup analyses that were conducted would be publicly available, we limited the study to devices reviewed at public FDA Advisory Committee meetings; as we expected, those devices have the most comprehensive publicly available data on clinical trials. We limited the analysis to devices publicly reviewed during 2014‐2017 for 2 reasons: (1) it enabled us to evaluate devices reviewed in the years when the FDA was most focused on improving diversity and subgroup analyses, and (2) we were able to analyze SSEDs and labeling for devices that were subsequently approved by the FDA.

As our data sources are restricted to publicly available information, we do not know whether there were additional subgroup analyses provided by companies or conducted by the FDA that the agency decided not to include in public documents. However, given the 2012 law urging the FDA to conduct subgroup analyses, the FDA had a strong incentive to provide subgroup data at public meetings when it was available.

In our experience attending dozens of FDA Advisory Committee meetings for more than a decade, subgroup analyses are rarely presented or discussed, and that has not changed in recent years. When they are, we have never observed a panel that insisted that greater diversity in clinical trials be required prior to FDA approval, even when panel members express concern about the lack of people of color or other demographic homogeneity. FDA panels sometimes recommend that better diversity and appropriate subgroup analysis be required in postapproval trials. Unfortunately, even when the FDA includes that recommendation, completion of postapproval studies tends to be delayed for years.55 Moreover, postmarket studies are less likely to include US patients, are less likely to be multicenter, and are less likely to include clinical endpoints.56 Few postapproval device studies are large enough to be powered to appropriately analyze demographic subgroups.57

Conclusions

Despite a 2012 law encouraging the FDA to ensure that demographic subgroup analyses are conducted, and despite an FDA report in 2013 and related FDA statements14, 58 concluding that such analyses are conducted and publicly available, most of the data for devices that were provided to FDA Advisory Committees from 2014 to 2017 did not include meaningful subgroup analyses. Nor were the data included in FDA reviews or labeling for those devices. While the FDA correctly states that “the majority” of applications include subgroup analyses, that might be misinterpreted to mean that most applications include most types of subgroup analyses. Our study found that while a few subgroup analyses are often included, many relevant analyses are not. Moreover, simply complying with the law by increasing analyses of demographic subgroups is not sufficient. The FDA should require that analyses be meaningful and useful to patients and providers, including incorporating enough participants of specific subgroups, conducting all appropriate analyses, and making the data easily available.

Our study shows that as a result of the lack of diversity and the lack of appropriately conducted subgroup analyses, it is not possible for patients or physicians to know whether many newly approved highest‐risk medical devices are safe and effective or whether diagnostic tests are accurate for women and men, most minority racial or ethnic groups, or patients over age 65. Moreover, even when analyses were conducted, differences that were found were often assumed to be due to chance because of small subsamples, and those differences were not included in the labeling. As a result, many of the highest‐risk medical devices, which are typically the only devices tested in clinical trials, are rarely proven in statistical analyses to be safe or effective for many of the patients who are most likely to rely on them.