FRAMM: Fair ranking with missing modalities for clinical trial site selection

Summary

The underrepresentation of gender, racial, and ethnic minorities in clinical trials is a problem undermining the efficacy of treatments on minorities and preventing precise estimates of the effects within these subgroups. We propose FRAMM, a deep reinforcement learning framework for fair trial site selection to help address this problem. We focus on two real-world challenges: the data modalities used to guide selection are often incomplete for many potential trial sites, and the site selection needs to simultaneously optimize for both enrollment and diversity. To address the missing data challenge, FRAMM has a modality encoder with a masked cross-attention mechanism for bypassing missing data. To make efficient trade-offs, FRAMM uses deep reinforcement learning with a reward function designed to simultaneously optimize for both enrollment and fairness. We evaluate FRAMM using real-world historical clinical trials and show that it outperforms the leading baseline in enrollment-only settings while also greatly improving diversity.

Highlights

• •The underrepresentation of minorities in clinical trials is a big problem
• •We propose FRAMM to guide fair trial site selection to address this problem
• •FRAMM uses reinforcement learning to optimize for both diversity and enrollment
• •On historical trials, FRAMM boosts minority enrollment by up to 20% for some groups

The bigger picture

The issue of underrepresentation of gender, racial, and ethnic minorities in clinical trials is a long-standing and severe problem. This gap in enrollment has historically led to treatments that are less effective for these groups, highlighting a significant challenge in achieving equitable healthcare outcomes. By proposing FRAMM, a deep reinforcement learning framework for fair trial site selection, we take a significant step toward more inclusive and effective medical research. FRAMM’s ability to simultaneously optimize for both enrollment and diversity helps to ensure that clinical trials more accurately reflect the diverse populations that they aim to serve, without sacrificing the overall enrollment necessary to make them work. While this paper serves mainly as a case study and proof of concept, the ambitions are to transform both the ideology and methodology of trial enrollment and research to make it more equitable and fair at every step of the way.

FRAMM, a cutting-edge machine learning framework, addresses the critical issue of minority underrepresentation in clinical trials by innovating fair trial site selection. Using deep reinforcement learning, FRAMM adeptly navigates incomplete data and simultaneously optimizes enrollment and diversity. Tested on real, historical trials, it not only surpasses existing methods in enrollment efficiency but also significantly enhances participant diversity, paving the way for more inclusive and effective healthcare research.

Introduction

Clinical trials are the currently widely accepted process for evaluating the efficacy and safety of proposed new treatments for diseases. Enrolling sufficient patients from all gender, racial, and ethnic groups is essential for ensuring the treatment’s efficacy on all groups. Despite many efforts to address the disparities,^1,2 the underrepresentation of minorities in clinical trials remains a problem.^3,4 This consequently undermines the fairness for minorities in obtaining effective treatments. For example, reports show that African Americans make up $13.4\%$ of the US population, but only $5\%$ of trial participants. Hispanics represent $18.1\%$ of the US population, but less than $1\%$ of trial participants.⁵ These gaps can then have drastic and problematic downstream effects due to well-documented racial and ethnic differences in response to medicine.⁶ Without representation of these groups, we are unable to obtain precise estimates of treatment effects within different subgroups and thus unable to guide both the development and use of medicine with these groups in mind.

To address the enrollment disparity, existing efforts have included government policy,⁷ softening eligibility criteria to make trials more accessible,⁸ and a community engagement-based approach.⁹ Recently, deep learning has been introduced to site selection. For example, Doctor2Vec¹⁰ proposed to select sites based on predicted patient enrollment. This development brings promise to scalable site selection, but it does not yet have any consideration for diversity.

This is especially important as historical realities make the problem of site selection necessarily a trade-off between enrollment and diversity. While there certainly exist diverse and high-enrolling sites, the inequities in past enrollment and data mean that most previous sites that would typically be selected again are predominantly White. So, we must by definition go away from these predictably high-enrolling sites to introduce greater diversity. Furthermore, we must do so within the constraints of a fixed or at least limited number of sites due to the unpredictability of enrollment and the huge cost of approaching new sites to enroll further patients.¹¹ Such realities make approaches such as simply adding additional, diverse sites or imposing enrollment caps to limit majority group patients undesirable as they are costly and limit overall enrollment, respectively. Thus, there are still the following challenges to be solved.

• (1)Missing data modalities across sites. Different trial sites can have different modalities of features that can be predictive of patient enrollment when seeking to pick sites for future trials. Some of these feature modalities such as claims or clinician specialty data can also be missing at different sites. Trial sites with a greater minority population are more likely to have missing data due to insufficient data collection and reporting, so failure to handle this problem only exacerbates the underlying unfairness.
• (2)The enrollment-diversity trade-off. While selecting sites only to maximize enrollment can be treated as a prediction task, the addition of diversity adds a unique challenge to the problem. We can not simply constrain fairness by setting minimum percentage thresholds for each group because they would effectively set enrollments caps due to the low minority population selected by enrollment-only models, and the administrative constraints limiting the overall number of sites we can choose. So, the problem is necessarily a trade-off between enrollment and fairness, and we thus need to optimize simultaneously for both objectives.

To address these challenges, we propose a deep reinforcement learning framework named FRAMM, shown in Figure 1, which is enabled by the following technical contributions.

• (1)Modality encoder for missing data handling.FRAMM handles missing data by first mapping all the different feature modalities in their diverse data formats into a shared, uniformly formatted representation space. It then combines the present modality representations into a single site representation via a masked cross-attention mechanism. This missing data module uses the trial as a query to build a site representation without needing complete site features.
• (2)Deep reinforcement learning for efficient trade-offs.FRAMM is equipped with a deep reinforcement learning setup with a specifically built reward function that simultaneously optimizes for both enrollment and fairness metrics. It also has a deep Q-Value network that approximates the contribution of each individual site to the corresponding reward given their site features.

Figure 1.

A visualization of the FRAMM framework

FRAMM uses multi-modal site features and the trial representation to generate scores for, rank, and select a subset of prospective trial sites. The pipeline used to do so consists of modality encoders, a missing data handling mechanism, a scoring network, and a reinforcement learning-based ranking approach.

We evaluate FRAMM using 4,392 real-world clinical trials ranging from 2016 to 2021 from a large clinical trial company. We show that FRAMM outperforms the leading baseline in enrollment-only settings while also achieving large gains in diversity. Specifically, it is able to produce a 9% improvement in diversity with similar enrollment levels over the leading baselines. That improved diversity is further manifested in achieving up to a 14% increase in Hispanic enrollment, 27% increase in Black enrollment, and 60% increase in Asian enrollment compared with selecting sites with an enrollment-only model.

Background and related work

Machine learning for clinical trials

There have been a number of recent applications that look at using machine learning to optimize clinical trial operations. These include matching patients to trials that they are eligible for,^12,13,14 searching for similar trials,¹⁵ and predicting trial outcomes.¹⁶ There have even been some works seeking to predict site enrollments to help select trial sites.^10,17 However, none of the existing works were designed to optimize enrollment diversity.

Fairness in machine learning

Nevertheless, fairness more generally has been a well-explored topic in the wider machine learning domain. Many applications within the area (see surveys^18,19,20) seek to limit or eliminate biases within the model inputs via preprocessing or model outputs via post-processing. However, our application belongs to the third class of fairness-based models, in-processing, which reduces biases and improves fairness during training via approaches such as adversarial learning,²¹ bandits,²² regularization,²³ and re-weighting.²⁴ Specifically, we seek to utilize a constraint optimization approach to in-processing as in Zemel et al.,²⁵ Nabi et al.,²⁶ Celis et al.,²⁷ and Narasimhan et al.²⁸by tapping directly into the learning framework and adding a fairness component to our loss function. However, our framework FRAMM is unique even in comparison to these similar approaches as we avoid directly learning within some fairness constraint but explicitly optimize and maximize fairness metrics.

Ranking methods

The standard ranking problem is a well-study topic.^29,30,31 Despite a good amount of literature dealing with fair ranking, most approaches constrain fairness to an acceptable level,^32,33,34 ensure certain protected features do not have an impact on any model outputs,^35,36 or add regularization to the loss function^36,37 rather than seeking to maximize fairness metrics as we do.

In particular, our approach uses reinforcement learning (RL) to rank with an abstract reward function, which can be made to incorporate fairness metrics. This is an especially exciting subdomain of the learning to rank problem with numerous promising approaches^38,39. Liu et al.⁴⁰ use a fairness-dependent loss function in a different setting involving recommendations over time. Feng et al.⁴¹ introduce a multi-agent setup designed to optimize multiple objectives. Clark and Manning⁴² and Wei et al.⁴³ present new policy gradient algorithms used to train their ranking models. However, while RL-based approaches are flexible enough to support our goal, our problem setting of explicitly optimizing for fairness with missing data is still a unique foray into the field.

Missing data handling

Classical missing data handling mainly relies on data imputation. Most existing methods typically reconstruct embeddings of missing modalities based on the embeddings of present ones as in Ma et al.,⁴⁴ Tran et al.,⁴⁵ and Lau et al.⁴⁶This requires some complete data points and pre-training of the imputation model. Other approaches include modality dropout during training,⁴⁷ or learning multiple conditional distributions for different combinations of present modalities.⁴⁸ However, alongside the growth in the popularity of attention mechanisms within architectures such as Transformers, there has also come an exploration of how attention, with built-in support for masking certain inputs can be used to handle missing data.^49,50,51 Our framework leverages a similar approach via a cross-attention mechanism that allows us to entirely bypass data imputation and the need for complete data in model training.

Results

Problem formulation

Clinical trial site features

For each clinical trial, the trial is represented by a vector $\mathbf{t}\in {\mathbb{R}}^{n_t}$ containing features for things such as disease information, trial logistics, and eligibility information. Its sites may then contain any number of features in a variety of different modalities depending on the available data. In our dataset, our sites correspond to individual clinicians or investigators, and they are represented by the following data modalities:

• (1)Static information${\mathrm{s}}_i\in {\mathbb{R}}^{n_s}$ is a vector representing a site’s primary clinician’s gender, profession type, primary specialty, patient demographic distributions, and the geo coordinates of the site.
• (2)Diagnosis history${\mathrm{D}}_i\in {\mathbb{R}}^{n_c\times n_d}$ is a sequence of $n_c$ one-hot vectors representing the ICD-10 codes (out of $n_d$ options) of the diagnoses of the most recent patients for a given site.
• (3)Medication history${\mathrm{P}}_i\in {\mathbb{R}}^{n_c\times n_p}$ is a sequence of $n_c$ one-hot vectors representing for the Uniform System of Classification ontology level 2⁵² codes (out of $n_p$ options) of the most recent prescriptions at a given site.
• (4)Enrollment history${\mathrm{E}}_i\in {\mathbb{R}}^{n_h\times \left(n_t^{\prime }\right)}$ is a sequence of $n_h$ trial representations (omitting the inclusion/exclusion criteria here due to dimensionality/memory concerns) and enrollment numbers for the most recent trials of a site.

In addition, trial sites also have an input feature mask ${\mathbf{m}}_i\in {\mathbb{R}}^4$ represented as a 4-dimensional binary vector signifying whether each feature modality is present or missing. Finally, trial sites are labeled using their enrollment value for the trial $e_i\mathbb{\in }\mathbb{R}$ and their 6-dimensional vector ${\mathbf{r}}_i\in {\mathbb{R}}^6$ representing their racial distribution. For example, if site 4 enrolled 95 participants and has a racial makeup of 47% White, 23% Hispanic, 15% Black, 10% Asian, 4% Mixed, and 1% Others, we have $e_4=95$ and ${\mathbf{r}}_4=\left[47,23,15,10,4,1\right]$.

So, each trial $\mathbf{t}$ is scored against M trial sites, which are each represented as $S_i=\left(\left({\mathbf{s}}_i,{\mathbf{D}}_i,{\mathbf{P}}_i,{\mathbf{E}}_i\right),\left({\mathbf{m}}_i\right),\left(e_i,{\mathbf{r}}_i\right)\right)$, where $\left({\mathbf{s}}_i,{\mathbf{D}}_i,{\mathbf{P}}_i,{\mathbf{E}}_i\right)$ are feature modalities, ${\mathbf{m}}_i$ is the feature mask, and $\left(e_i,{\mathbf{r}}_i\right)$ the labels of enrollment value $e_i$ and racial distribution ${\mathbf{r}}_i$.

Clinical trial site selection

For an input trial, the task is to select K sites from its M choices to maximize overall diversity and enrollment. Mathematically, given a trial $\mathbf{t}$ and its M prospective sites $\left[S_1,S_2,\cdots ,S_M\right]$, the task becomes to select K of those sites based on their present input features to maximize the received reward $R\left(\mathcal{R}\right)$. In this paper, this is achieved by ranking the M sites as $\mathcal{R}$, where ${\mathcal{R}}_j$ is the j-th site in the ranking, and selecting the K highest ranked sites. The ranking $\mathcal{R}$ then produces two additional outputs:

• (1)$\tilde{\mathrm{e}}\in {\mathbb{R}}^M$, a vector of the enrollment values ($e_i$’s) of each site in the ranked order.
• (2)$\tilde{\mathrm{R}}\in {\mathbb{R}}^{M\times 6}$, which is the 6 racial distribution values (${\mathrm{r}}_i$’s) for each of the sites in the ranked order.

The notations are summarized in Table 1 for reference.

Table 1.

Table of notations

Notation	Description
$M\mathbb{\in }\mathbb{N}$	The number of site options for a trial
$K\mathbb{\in }\mathbb{N}$	The number of sites to select from the M
$\lambda \in {\mathbb{R}}^{+}$	The relative weighting of utility and fairness
$\mathbf{t}\in {\mathbb{R}}^{n_t}$	A vector representation of a clinical trial
$S_i$	The i-th site option (of M) for the trial
${\mathbf{s}}_i\in {\mathbb{R}}^{n_s}$	The i-th site’s static features modality
${\mathbf{D}}_i\in {\mathbb{R}}^{n_c\times n_d}$	The i-th site’s diagnosis history modality
${\mathbf{P}}_i\in {\mathbb{R}}^{n_c\times n_p}$	The i-th site’s prescription history modality
${\mathbf{E}}_i\in {\mathbb{R}}^{n_h\times \left(n_t^{\prime }+1\right)}$	The i-th site’s enrollment history modality
${\mathbf{m}}_i\in {\left\{0,1\right\}}^4$	The i-th site’s modality presences
$e_i\mathbb{\in }\mathbb{N}$	The i-th site’s enrollment for the trial
${\mathbf{r}}_i\in {\left[0,1\right]}^6$	The i-th site’s racial distribution vector
R	The reward function for a selection of sites
$\mathcal{R}$	An ordered ranking of the M sites
$\mathbf{e}\in {\mathbb{N}}^M$	The site enrollment numbers ordered by $\mathcal{R}$
$\mathbf{R}\in {\left[0,1\right]}^{Mx6}$	The racial distributions ordered by $\mathcal{R}$

Experimental design

We design experiments to evaluate our proposed FRAMM model and answer the following questions:

• (1)Is FRAMM effective at enrolling large patient populations in enrollment-only settings?
• (2)Can FRAMM make efficient trade-offs between enrollment and diversity to achieve high levels of both?
• (3)Does FRAMM improve diversity compared to enrollment-only and post hoc constrained models?

Experimental setup

Datasets

We use IQVIA’s historical real-world clinical trials and claims data in evaluation. The clinical trial database contains 33,323 sites matched with 4,392 trials. We first build the site pool by constructing input features. We then create a separate dataset for each value of M that we use. We match each trial to M sites, using the top M sites (determined by enrollment) in the database if there are enough and otherwise completing the set of M by randomly selecting sites from the overall pool and assigning an enrollment of 0 for the trial. Additional details regarding the collection and preprocessing of these data are provided in the supplemental information.

We randomly split these datasets by trial into training and test datasets with an 80-20 ratio so that we may test on true historical data and enrollment counts that are unseen by the compared models. We then further split off 10% of the training set into a validation set. We additionally create a missing data version of the test set in the same way as outlined within our method section by randomly masking each modality with a 20% chance. Using these datasets, we train our models within the PyTorch framework⁵³ for 35 epochs at a 0.00001 learning rate and using the Adam optimizer. We save the model that best performs on the validation set as determined by our reward function and evaluate it using the test set.

Baselines

We consider the following baseline models.

• (1)Doctor2Vec¹⁰ is the current state of the art in enrollment-only trial site selection. It constructs a memory network doctor representation based on static features and patient visits. That representation is then queried by a specific trial representation and fed into a downstream network to predict the doctor’s enrollment count for the trial. Note that Doctor2Vec does not handle missing data and so cannot be trained nor tested on any missing data.
• (2)Random selects K sites at random from the available M.
• (3)One-Sides Policy Gradient (PGOS)³⁷ is a fairness baseline that replaces the fairness function F with a one-sided loss function that ensures through regularization that groups are not underrepresented within rankings. This baseline represents the typical approach of constraining or regularizing fairness rather than explicitly optimizing diversity. Note that when $\lambda =0$, F does not contribute to the overall reward, and this is identical to our standard framework. So, it is omitted from any enrollment-only results.

Ablation models

To demonstrate FRAMM’s effectiveness at handling missing data and its ability to use missingness as a data augmentation technique to combat low data settings such as ours (the dataset has fewer than 5,000 trials), we add two ablation models without FRAMM’s missing data augmentation and instead trained on the same dataset without any missingness as Doctor2Vec. The first is the full FRAMM model trained on this smaller dataset, and the second removes the missing data mechanism altogether and replaces it with a fully connected layer. We call these models “FRAMM No Missing” and “FC No Missing,” respectively.

Metrics

We consider both enrollment and diversity metrics.

For enrollment, we compared the size of each model’s enrolled cohort with the ground truth maximal enrollment via a pair of metrics. First, we use Relative Enrollment Gap calculated by

$$\text{Relative}\text{Enrollment}\text{Gap}=\frac{\mathrm{Max}\text{Enrollment}-\text{Model}\text{Enrollment}}{\mathrm{Max}\text{Enrollment}}$$ (Equation 1)

where max enrollment is the total enrollment from the top K sites after the trial completion (a theoretical ceiling), and model enrollment is the total enrollment from the K sites selected by the model. We also report the standard ranking metric normalized Discounted Cumulative Gain (nDCG), defined as

$$\text{nDCG}=\sum _{j=1}^K\frac{2^{m_j}-1}{{\log }_2\left(j+1\right)}/\sum _{j=1}^K\frac{2^{o_j}-1}{{\log }_2\left(j+1\right)}$$ (Equation 2)

where m is the model ranking enrollment list, and o is the optimal ranking enrollment list. For example, if we have four sites A, B, C, and D with enrollment values of 5, 10, 8, and 7, respectively, and our model ranked them B $\to$ C $\to$ A $\to$ D, then we would have $m=\left[10,8,5,7\right]$ and $o=\left[10,8,7,5\right]$.

To measure diversity, we use the entropy of the overall racial distribution of the final enrolled population. This is defined in the same way as above within our reward function, F, by

$$H\left(\mathbf{p}\right)=-\sum _{k=1}^6{p}_k\log p_k$$ (Equation 3)

where $\mathbf{p}$ is the vector of the proportions of each racial group within the overall enrolled population and so $p_k$ is the percentage of a given group.

Q1. Enrolling large patient populations

We first evaluate each model in enrollment-only settings (with $\lambda =0$ for FRAMM variants) to examine their ability to select sites with only enrollment in mind. We display both of our enrollment metrics for the $M=20$, $K=5$, $\lambda =0$ setting for each compared model in Table 2.

Table 2.

Enrollment-only performance

	Relative enrollment gap ($\downarrow$)	nDCG ($\uparrow$)
Random	0.621 $\pm$ 0.019	0.320 $\pm$ 0.017
Doctor2Vec	0.525 $\pm$ 0.021	0.402 $\pm$ 0.018
FRAMM No Missing	0.572 $\pm$ 0.020	0.359 $\pm$ 0.018
FC No Missing	0.566 $\pm$ 0.020	0.363 $\pm$ 0.017
FRAMM	0.512 $\pm$ 0.020	0.409 $\pm$ 0.018

We do see that Doctor2Vec is able to outperform our two ablation, FRAMM-style models trained on the same smaller, full data training dataset of fewer than 5,000 trials without missing data augmentation. However, FRAMM’s missing data mechanism unlocks an effective form of data augmentation that allows it to train on a larger, augmented missing data training dataset. Accordingly, it is able to achieve the best enrollment performance in both metrics on this full data test set, even though it was trained on a different type of data.

Q2. Making efficient trade-offs

We then showcase our framework’s ability to make effective trade-offs between enrollment and diversity by showing the trajectories of Relative Enrollment Gap vs. Entropy for varying λ values.

We compare FRAMM with our PGOS and Random baselines in the M = 10, K = 5 setting on the missing data test set in Figure 2. Here we see that while both FRAMM and the PGOS model greatly outperform the Random baseline and are able to increase diversity at the expense of enrollment, FRAMM makes much more efficient and tunable trade-offs than the PGOS baseline. As such, it maintains much higher enrollment rates for a given level of diversity, achieving a roughly 5% higher peak diversity value and providing up to 9% higher levels of diversity for the same enrollment value. Furthermore, it offers the ability for much more granular tuning through different λ values, whereas the PGOS model is largely constrained to the same region once λ is increased from 0. A version of these results on synthetic data for reproducibility can also be found in Figure S1A in the supplemental information with similar findings.

Figure 2.

Relative enrollment gap vs. entropy trade-off curves for λ equaling 0.5, 1, 2, 4, and 8 for $M=10$, $K=5$

Both FRAMM and the PGOS model are able to increase diversity at the expense of enrollment, but FRAMM makes much more efficient and tunable trade-offs than the PGOS baseline. It maintains much higher enrollment rates for a given level of diversity, achieving a roughly 5% higher peak diversity value and providing up to 9% higher levels of diversity for the same enrollment value.

We also compare FRAMM with our two ablation models and the two enrollment baselines on the core, full data (no missingness) test set in Figure 3. There we see again that FRAMM is effectively leveraging training with missing data in making more efficient trade-offs than either ablation baseline. Finally, we see that it is able to achieve more optimal combinations of enrollment and diversity than the Doctor2Vec and Random baselines, which are constrained to a single point without any ability to increase diversity. A version of these results on synthetic data for reproducibility can also be found in Figure S1B in the supplemental information with similar findings.

Figure 3.

Relative enrollment gap vs. entropy trade-off curves comparing FRAMM with our two ablation models and the two enrollment baselines on the core, no missingness test set

FRAMM leverages training with missing data to make more efficient trade-offs than either ablation baseline and achieves more optimal combinations of enrollment and diversity than the Doctor2Vec and Random baselines that are constrained to a single point without any ability to increase diversity.

Q3. Improving diversity

Finally, we examine the effect of our model on the enrolled populations of the studies themselves as compared with those selected by the enrollment-only Doctor2Vec model, presenting both the aggregate effect and the effect on a single randomly selected study about multiple sclerosis.

Aggregate effect

The aggregate effects are striking in terms of improving diversity. We see a big reduction of the enrollment of the White population with a corresponding increase in each minority group, with the most significant increase for Black and Asian groups. The comparison of the mean percentages of each racial group across the trials in the test set for Doctor2Vec’s chosen cohorts, the population enrolled by FRAMM with $\lambda =1$, and the population enrolled by FRAMM with $\lambda =4$ can be seen in Figure 4.

Figure 4.

The aggregate effect of different λ values as compared with enrollment of our Doctor2Vec baseline without consideration of fairness on the racial makeup of the average enrolled population

We see a dramatic decrease of White enrollment and a clear increase in underrepresented groups.

Individual effect

This effect is mirrored in the case of a single, randomly selected trial for relapsing multiple sclerosis (RMS). We present comprehensive results including all possible site options in Table 3. In comparison with the sites chosen by Doctor2Vec, we see a concerted shift from sites with overwhelmingly White populations to those that enroll just as many but possess much more diversity. Specifically, FRAMM can be seen as extracting the optimal cohort in selecting each of the top five racially diverse sites. As a result, the racial distribution of sites changes from $\left[56.1,15.8,18.1,6.5,2.8,0.7\right]$ to $\left[45.9,15.5,26.2,7.0,4.2,0.9\right]$, increasing the entropy of the enrolled population from 1.240 to 1.362 while simultaneously enrolling more patients.

Table 3.

Case study trial site selection

Site location	White	Hispanic	Black	Asian	Mixed	Others	Enrolled
Birmingham, AL	67.7	3.8	25.5	1.7	1.2	0.1	23.0
Wellesley Hills, MA	80.4	4.2	2.0	9.6	3.2	0.5	17.0
Tacoma, WA	60.0	10.9	14.5	5.2	7.8	1.6	16.0
Ann Arbor, MI	66.9	3.7	7.5	17.9	3.9	0.1	16.0
Fort Lauderdale, FL	29.5	25.2	39.4	4.0	1.8	0.1	14.0
San Antonio, TX	53.4	30.8	5.2	8.3	1.6	0.6	12.0
Raleigh, NC	73.3	6.2	10.6	6.5	2.8	0.5	12.0
Kirkland, WA	69.9	8.1	1.4	15.8	4.2	0.7	12.0
Oklahoma City, OK	42.0	3.0	36.1	10.6	6.4	1.9	11.0
Tucson, AZ	77.7	12.4	1.4	5.7	1.8	1.1	11.0
Knoxville, TN	88.2	2.4	2.3	4.1	2.6	0.4	10.0
Charlotte, NC	70.2	2.9	19.5	3.8	2.7	0.9	9.0
Asheville, NC	76.6	9.6	8.0	1.8	3.2	0.9	9.0
Greer, SC	76.1	9.3	7.8	4.9	1.2	0.7	9.0
Cleveland, OH	40.9	2.5	44.7	9.5	2.3	0.2	8.0
Owosso, MI	93.9	3.4	0.5	0.6	1.4	0.3	8.0
Toledo, OH	85.3	3.6	6.0	2.5	1.9	0.6	8.0
Louisville, CO	80.0	6.5	0.4	10.7	2.2	0.2	8.0
Flossmoor, IL	42.5	3.0	50.3	1.3	1.5	1.4	7.0
Cullman, AL	95.2	4.0	0.4	0.2	0.2	0.0	7.0

This performance appears even stronger when put in the context of other options for improving diversity. We already saw in the previous sections that our PGOS fairness baseline made less efficient trade-offs than FRAMM. However, if a trial decided to forgo an RL setup altogether and not optimize for diversity but instead constrain fairness within the selections of an enrollment-only model, the enrollment to achieve that level of diversity would be incredibly low. Specifically in the case of this study, forcing an acceptable level of diversity (for example matching FRAMM’s cohort distribution) on Doctor2Vec’s 5 selected sites through enrollment caps would further reduce its enrollment by at least 18% just to reduce the White population to the same proportion.

Discussion

In this paper, we proposed a deep RL framework, named FRAMM, for fair trial site selection. Our method uses a missing data mechanism to account for the fact that different modalities of input features can be missing at different sites. It also uses RL with a specifically designed reward function that simultaneously optimizes for both enrollment and fairness to account for the need to make efficient trade-offs between the two objectives. We demonstrate strong performance in achieving state of the-art enrollment levels and also the ability to make efficient and tunable trade-offs between enrollment and diversity. Finally, we show that when diversity is increased, we achieve much more fair site selection in enrolling far more underrepresented populations than other enrollment-only models. While the problem of a lack of diversity in clinical trials is undoubtedly multifaceted and goes far beyond inequitable site selection, we hope our work can bring further attention to this issue, and we believe it is an important step in addressing it head on.

We now discuss a number of limitations regarding our work and our task here. There are a number of realities inherent to our dataset and the site selection task as a whole that may limit the feasibility and effectiveness of our approach.

First, while the dataset we used offers a comprehensive mix of trials across every size and phase as well as the vast majority of medical indications, there may remain some biases based on both the dataset composition and IQVIA’s specific site selection and trial execution process. As such, a model trained on these data should be taken as suited specifically for this dataset and company. Any extrapolation to another setting would require retraining or at least awareness of that fact. However, much like many machine learning tasks and models, we would generally expect FRAMM to be able to learn different patterns for different types of trials at once, despite any compositional disparities, though this should be explored further.

Furthermore, there are some additional nuances on a trial-by-trial basis, such as a small trial running at a limited number of sites, varying eligibility criteria, and geographic constraints, which all can make choosing a more diverse selection more difficult. However, we believe that those challenges require awareness during deployment or use rather than representing an impediment to the feasibility of the task and method in general. Some trials may be unable to successfully use this type of approach (though even geographically or numerically limited sites should still be able to become more diverse even if they struggle to be fully diverse, a reality that makes our approach more attractive than constraint-based methods), but we can still measure the success and ability of a model in the aggregate over a test set of trials as we do here. So, individual trials may have differing levels of both success and ability in terms of population diversification, and we should always be aware of the context when analyzing individual results. However, that reality should not discredit or dishearten our ability to make improvements, even if they are only at the margins or in the aggregate. This work is too important not to.

We strongly believe that it is highly beneficial to create large, publicly available datasets that are de-identified and cross-setting. These datasets can be used to mitigate and analyze the limitations that we have mentioned earlier. Such work can aid in further research and exploration and can help to combat biases. We hope that our work can encourage the creation of such datasets.

Experimental procedures

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Jimeng Sun (jimeng@illinois.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

The clinical trial data are publicly available from clinicaltrials.gov, and we share them in processed form. The medical claims (for diagnosis and medication histories) as well as site features and enrollment histories cannot be deposited in a public repository due to privacy and compliance issues. To request access to a processed version of such datasets, please reach out to the lead contact. In addition, we release a synthetic version of the dataset that is generated using the statistical distribution of the real features. All outlined resources along with the code used to run the experiments have been deposited at https://doi.org/10.5281/zenodo.10499129⁵⁴ (representing the repository originally published at https://github.com/btheodorou99/FRAMM) and are publicly available as of the date of publication.

Fair ranking with missing modalities

Our proposed fair ranking with missing modalities (FRAMM) framework ranks and selects potential trial sites with the following two modules.

• (1)The missing data modality encoders that generate trial site representations while handling the fact that some of the input site modality embeddings may be missing.
• (2)The scoring and ranking network that maps each trial site representation to a single score approximating the value of a site to a particular clinical trial before converting the scores into a probability distribution over different rankings according to a learned policy.

Modality encoding with missing data handling

For each of the M sites (where subscript i throughout refers to the i-th site of the M), the modality encoders first embed each available modality and trial representation into a shared representation space, ${\mathbb{R}}^{n_{\text{emb}}}$ (where we use $n_{\text{emb}}=128$).

For the three sequential modalities, diagnosis ${\mathbf{D}}_i$, prescription ${\mathbf{P}}_i$, and enrollment history ${\mathbf{E}}_i$, we embed the inputs using f, parameterized for the different settings by d, p, and e, respectively, such that they map ${\mathbf{d}}_i^{\mathbf{e}}=f_d\left({\mathbf{D}}_i\right)$, ${\mathbf{p}}_i^{\mathbf{e}}=f_p\left({\mathbf{P}}_i\right)$, and ${\mathbf{e}}_i^{\mathbf{e}}=f_e\left({\mathbf{E}}_i\right)$. In our experiments, we implement f with a bidirectional LSTM followed by a fully connected neural network with ReLU activation function after the first linear layer as in Equation 4.

$$f\left(x\right)=\max \left(0,\text{biLSTM}\left(x\right)\mathbf{W}+\mathbf{b}\right)\mathbf{V}+\mathbf{c}$$ (Equation 4)

where the LSTM has hidden dimension 128, $\mathbf{W},\mathbf{V}\in {\mathbb{R}}^{128\times 128}$, and $\mathbf{b},\mathbf{c}\in {\mathbb{R}}^{128}$.

We then embed the static site modality and trial representation using $g_s$ and $g_t$, respectively, such that they map ${\mathbf{s}}_i^{\mathbf{e}}=g_s\left({\mathbf{s}}_i\right)$ and ${\mathbf{t}}_i^{\mathbf{e}}=g_t\left(\mathbf{t}\right)$. We implement g as a fully connected neural network with ReLU activation function between layers as in Equation 5,

$$g\left(x\right)=\max \left(0,x\mathbf{W}+\mathbf{b}\right)\mathbf{V}+\mathbf{c}$$ (Equation 5)

where $\mathbf{W}\in {\mathbb{R}}^{\dim \left(x\right)\times 128}$, $\mathbf{V}\in {\mathbb{R}}^{128\times 128}$, and $\mathbf{b},\mathbf{c}\in {\mathbb{R}}^{128}$. So, for each site we have access to four possibly missing modality embeddings ${\mathbf{s}}^{\mathbf{e}}$, ${\mathbf{d}}^{\mathbf{e}}$, ${\mathbf{p}}^{\mathbf{e}}$, and ${\mathbf{e}}^{\mathbf{e}}$ and a trial embedding ${\mathbf{t}}^{\mathbf{e}}$ all within the shared representation space ${\mathbb{R}}^{n_{\text{emb}}}$. These five embeddings are then fed into the missing data mechanism within the module.

Missing data handling

To handle missing data modalities, most existing strategies, including modality dropout (MD),⁴⁷ unified representation network (URN),⁴⁶ and cascaded residual autoencoder (CRA)⁴⁵ either do not directly model missing data or require pre-training. To avoid these issues, we use an attention-based approach similar to that in Liu et al.,⁴⁹ which uses the trial embedding as the query for a masked multi-head cross-attention mechanism where the site modality embeddings serve as both keys and values, and the site feature masks dictate whether a given modality can be attended to. The output of this mechanism is the intermediate site embedding, ${\mathbf{h}}_i^{\prime }$, which is then concatenated back with the trial embedding to become the trial site representation ${\mathbf{h}}_i$. More formally, let the missing data handling procedure be $miss$ such that ${\mathbf{h}}_i=miss\left(\left[{\mathbf{s}}_i^{\mathbf{e}},{\mathbf{d}}_i^{\mathbf{e}},{\mathbf{p}}_i^{\mathbf{e}},{\mathbf{e}}_i^{\mathbf{e}},{\mathbf{t}}_i^{\mathbf{e}}\right]\right)$. For a given site, this masked cross-attention approach arrives at ${\mathbf{h}}_i$ by Equation 6,

$$\begin{array}{l}{\mathbf{h}}_i^{\prime }=\text{att}\left({\mathbf{t}}_i^{\mathbf{e}},\left[{\mathbf{s}}_i^{\mathbf{e}},{\mathbf{d}}_i^{\mathbf{e}},{\mathbf{p}}_i^{\mathbf{e}},{\mathbf{e}}_i^{\mathbf{e}}\right],\left[{\mathbf{s}}_i^{\mathbf{e}},{\mathbf{d}}_i^{\mathbf{e}},{\mathbf{p}}_i^{\mathbf{e}},{\mathbf{e}}_i^{\mathbf{e}}\right]\right)\\ \text{att}\left(Q,K,V\right)=\text{concat}\left({\text{head}}_1,\cdots ,{\text{head}}_4\right){\mathbf{W}}^{\mathbf{O}}\\ {\text{head}}_j=\sigma \left(\frac{\left(Q{\mathbf{W}}_j^{\mathbf{Q}}\right){\left(K{\mathbf{W}}_j^{\mathbf{K}}\right)}^T}{\sqrt{n_{emb}/n_{\text{head}}}}+{\mathbf{m}}_i^{\prime }\right)\left(V{\mathbf{W}}_j^{\mathbf{V}}\right)\\ m_{ik}^{\prime }=\{\begin{array}{cc}0& m_{ik}=1\\ -\infty & m_{ik}=0\end{array}\\ {\mathbf{h}}_i=\text{concat}\left({\mathbf{h}}_i^{\prime },{\mathbf{t}}_i^{\mathbf{e}}\right)\end{array}$$ (Equation 6)

where σ is the softmax function, ${\mathbf{W}}_j^{\mathbf{Q}},{\mathbf{W}}_j^{\mathbf{K}},{\mathbf{W}}_j^{\mathbf{V}},{\mathbf{W}}^{\mathbf{O}}\in {\mathbb{R}}^{128\times 128}$, and ${\mathbf{m}}_i^{\prime }\in {\mathbb{R}}^4$ is a conversion of the i-th site’s binary mask vector into the form required for masking in attention. That conversion converts a mask value of 0 (signifying a missing modality) to $-\infty$ to prevent any remaining attention weight after a softmax function over the sum of the values and masks, and it converts a value of 1 (signifying a present modality) to 0 to not interfere with the softmax calculation.

Missing data augmentation

This missing data handling also unlocks a new form of data augmentation that we incorporate directly into the FRAMM framework. For any data we aim to train a FRAMM model on, we may augment the dataset by repeating each trial data point while randomly discarding or masking each site modality. Specifically, we create 10 versions of each trial, and for each site in each version of each trial we randomly create a mask that prevents the architecture from using certain modalities. To do this, we set a 20% chance of each modality being missing, and we stipulate that if a modality is truly missing it must be masked but also necessitate that at least one modality must remain unmasked. In this way, FRAMM’s missing data handling allows it to not only adapt to different settings and varying input features but also augment any training data to achieve better performance regardless of the underlying missingness.

Deep Q-network and ranking policy

The second module takes the M trial site representations, now without any missingness, and uses them to rank the sites. To do so, it first maps the representations $\mathbf{H}=\left[{\mathbf{h}}_1\cdots {\mathbf{h}}_M\right]$ to M real-valued scores $\mathbf{q}\in {\mathbb{R}}^M$, which approximate the value each site will provide toward the final reward obtained by selecting them for the current trial. The $\mathbf{q}$ here serves an analogous (although not identical) role to a Q-Value in RL as in Equation 7,

$$Q\left(s,a\right)=R\left(s,a\right)+\gamma \underset{a^{\prime }}{\max }Q\left(s^{\prime },a^{\prime }\right)$$ (Equation 7)

This mapping is denoted generally by the $score$ function in Equation 8,

$$\mathbf{q}=score\left(\mathbf{H}\right)$$ (Equation 8)

Reward function

The reward function used in this paper and which the Q-Network learns to approximate consists of two components for enrollment utility and fairness objectives and is defined by Equation 9,

$$R\left(\mathcal{R}\right)=U\left(\mathcal{R}\right)+\lambda F\left(\mathcal{R}\right)$$ (Equation 9)

where λ is a hyperparameter determining the relative weighting of the two reward components. This relative weighting forms a trade-off such that multiple λ values can be utilized and the desired point along the trade-off curve can be selected. We chose λ as 0, 0.5, 1, 2, 4, or 8 in our experiments.

The utility component is defined by Equation 10,

$$U\left(\mathcal{R}\right)=\frac{{\sum }_{j=1}^K{\tilde{e}}_j-{\sum }_{j=K+1}^M{\tilde{e}}_j}{{\sum }_{j=1}^M{\tilde{e}}_j}$$ (Equation 10)

which is the enrollment difference between the chosen sites (top-K sites) and those not (the remaining $M-K$ sites), normalized for the total enrollment numbers.

The fairness component is the entropy of the racial distribution of the total patient population enrolled by the K chosen sites, formally defined by Equation 11,

$$F\left(\mathcal{R}\right)=H\left(\frac{{\sum }_{j=1}^K{\tilde{e}}_j{\tilde{\mathbf{r}}}_j}{{\sum }_{j=1}^K{\tilde{e}}_j}\right)$$ (Equation 11)

where both ${\tilde{e}}_j$ (the enrollment values in the ranked order) and the bottom sum are scalars applied to the entire 6-dimensional vector ${\tilde{\mathbf{r}}}_j$, the top sum operates element-wise over those vectors, and H is the standard definition of entropy where for a probability distribution $\mathbf{n}$ (such as the one outputted by our sum), $H\left(\mathbf{n}\right)=-{\sum }_k{n}_k\log n_k$.

Q-network architecture

We implement this mapping using a transformer encoder layer with a fully connected head. We postulate that transformer layers and their ability for each site’s score to be affected by the other sites available can be especially valuable given the dynamic nature of F in which the fairness component of our reward evaluates all of the selected sites together rather than acting as a sum of functions of each individual site as with the utility component. The generation of the scores is formally given by Equation 12,

$$\begin{array}{l}{\mathbf{H}}^{\left(0\right)}=\mathbf{H}\\ {\mathbf{H}}^{\prime \left(i\right)}=\mathrm{LN}\left({\mathbf{H}}^{\left(i-1\right)}+\text{att}\left({\mathbf{H}}^{\left(i-1\right)},{\mathbf{H}}^{\left(i-1\right)},{\mathbf{H}}^{\left(i-1\right)}\right)\right)\\ {\mathbf{H}}^{\left(i\right)}=\mathrm{LN}\left({\mathbf{H}}^{\prime \left(i\right)}+\left(\max \left(0,{\mathbf{H}}^{\prime \left(i\right)}{\mathbf{W}}^{\left(i\right)}+{\mathbf{b}}^{\left(i\right)}\right){\mathbf{V}}^{\left(i\right)}+{\mathbf{c}}^{\left(i\right)}\right)\right)\\ \mathbf{q}=\max \left(0,{\mathbf{H}}^{\left(n_l\right)}{\mathbf{W}}^f+{\mathbf{b}}^f\right){\mathbf{V}}^f+{\mathbf{c}}^f\end{array}$$ (Equation 12)

where $i\in \left\{1,\cdots ,n_l\right\}$, LN denotes layer normalization, ${\mathbf{W}}^{\left(i\right)},{\mathbf{V}}^{\left(i\right)}\in {\mathbb{R}}^{128\times 128}$, ${\mathbf{b}}^{\left(i\right)},{\mathbf{c}}^{\left(i\right)}\in {\mathbb{R}}^{128}$, ${\mathbf{W}}^f\in {\mathbb{R}}^{128\times 64}$, ${\mathbf{b}}^f\in {\mathbb{R}}^{64}$, ${\mathbf{V}}^f\in {\mathbb{R}}^{64\times 1}$, and ${\mathbf{c}}^f\mathbb{\in }\mathbb{R}$.

Ranking and policy learning

These “Q-Value” scores are then fed into the final portion of our framework where they are used to generate rankings and their probabilities using a non-deterministic policy π. While simply selecting the top K scores represents the best site selection as currently approximated by the network (and is used for testing), it prevents exploration of the ranking space during training. Instead, we define our stochastic policy by Equation 13,

$$\pi \left(\mathcal{R}\right)=\sum _{{\mathcal{R}}^{\prime }\in \phi \left(\mathcal{R}\right)}\prod _{j=1}^K\frac{\exp \left(q\left({\mathcal{R}}_j^{\prime }\right)\right)}{{\sum }_{k=j}^M\exp \left(q\left({\mathcal{R}}_k^{\prime }\right)\right)}$$ (Equation 13)

where φ maps to the set of other rankings that are the same as $\mathcal{R}$ except for permuting the first K elements, and the q function returns the score given by the Q-Value network for the given site. Our policy’s probability represents the odds of a given top-K combination (where order does not matter) achieved by sampling K sites without replacement according to the softmax probabilities defined by their scores. However, in practice, this quickly becomes impractical for any sizable K as the permutation space grows factorially. Instead, we use an unbiased estimate of the probability of a top-K combination calculated for a given ranking by randomly permuting the order of the first K elements (to remove the bias of higher probability rankings in the permutation space being drawn more often), calculating the product in the policy’s equation, and scaling by $K!$. In this way, we are able to estimate according to the expectation in Equation 14,

$$\pi \left(\mathcal{R}\right)=\sum _{{\mathcal{R}}^{\prime }\in \phi \left(\mathcal{R}\right)}\prod _{j=1}^K\frac{\exp \left(q\left({\mathcal{R}}_j^{\prime }\right)\right)}{{\sum }_{k=j}^M\exp \left(q\left({\mathcal{R}}_k^{\prime }\right)\right)}=K!\underset{{\mathcal{R}}^{\prime }\in \phi \left(\mathcal{R}\right)}{\mathbb{E}}\prod _{j=1}^K\frac{\exp \left(q\left({\mathcal{R}}_j^{\prime }\right)\right)}{{\sum }_{k=j}^M\exp \left(q\left({\mathcal{R}}_k^{\prime }\right)\right)}$$ (Equation 14)

Given our architecture for obtaining scores for each site, and our method of sampling and obtaining probability estimates of rankings given those scores, all that is left is to handle optimization and policy learning. The overall goal is to maximize the expected reward in Equation 15,

$${\mathbb{E}}_{\mathcal{R}\sim \pi }\left[R\left(\mathcal{R}\right)\right]$$ (Equation 15)

We implement this using REINFORCE.⁵⁵ It is a common policy gradient algorithm that directly optimizes the expected reward as calculated through Monte Carlo sampling weighted by the log-likelihoods of rankings. Using this or any other method policy gradient algorithm, we are able to use backpropagation back through our policy, Q-Network, and modality encoders to train our framework via real-world enrollment and diversity data.

Assumptions regarding randomness

Finally, we conclude our presentation of FRAMM by discussing some minutiae regarding the assumptions and flexibility of FRAMM’s augmentation approach as well as providing some brief experimental results demonstrating its value even if those assumptions are not upheld.

Different patterns of randomness

FRAMM’s augmentation is currently constructed to follow the pattern of modalities missing completely at random (MCAR). However, there are many different patterns of missingness that may exist in real-world sites, and this augmentation design could be adjusted if the pattern of missingness is known or at least suspected. Furthermore, if true missing modality data are available, FRAMM may be trained on that as well. Regardless, even if the true pattern of missingness is unknown, and we use this MCAR approach, which does not match the true pattern, we believe the augmentation helps.

Handling serially missing modalities

To that end, we show the ability of our framework to be robust in adapting to new settings with different data availability such as would be found in a new healthcare system with access to different types of data. Specifically, we explore the performance of FRAMM in the face of an unexpected serially missing modality, which appears in training but then is completely missing during testing. We do this by testing FRAMM and our two ablation baselines not on the full data test set but instead the same test set with the enrollment history modality missing for each investigator in each trial. We show the trade-off results of this experiment in the $M=20$, $K=5$ setting with varying λ in Figure 5. While each suffers a drop-off in comparison with the best evaluation on the full data test set, we see that our model clearly outperforms both of the models that were trained solely on the full data, reducing the enrollment gap from that full data benchmark nearly in half for similar levels of diversity and approaching the full data performance for large λ.

Figure 5.

Visualizations of relative enrollment gap vs. entropy trade-off curves for λ equaling 0.5, 1, 2, 4, and 8 for the missing modality experiment (with the core FRAMM model evaluated on the full data test set as a gold standard benchmark) for $M=20$, $K=5$

While each of the three compared models suffers a drop-off compared with the full data results, the FRAMM framework trained on a larger missing data dataset outperforms the other two by a sizable margin and approaches the full data performance for larger λ.

Published: March 1, 2024