Optimizing the synthesis of clinical trial data using sequential trees

Abstract

Objective

With the growing demand for sharing clinical trial data, scalable methods to enable privacy protective access to high-utility data are needed. Data synthesis is one such method. Sequential trees are commonly used to synthesize health data. It is hypothesized that the utility of the generated data is dependent on the variable order. No assessments of the impact of variable order on synthesized clinical trial data have been performed thus far. Through simulation, we aim to evaluate the variability in the utility of synthetic clinical trial data as variable order is randomly shuffled and implement an optimization algorithm to find a good order if variability is too high.

Materials and Methods

Six oncology clinical trial datasets were evaluated in a simulation. Three utility metrics were computed comparing real and synthetic data: univariate similarity, similarity in multivariate prediction accuracy, and a distinguishability metric. Particle swarm was implemented to optimize variable order, and was compared with a curriculum learning approach to ordering variables.

Results

As the number of variables in a clinical trial dataset increases, there is a pattern of a marked increase in variability of data utility with order. Particle swarm with a distinguishability hinge loss ensured adequate utility across all 6 datasets. The hinge threshold was selected to avoid overfitting which can create a privacy problem. This was superior to curriculum learning in terms of utility.

Conclusions

The optimization approach presented in this study gives a reliable way to synthesize high-utility clinical trial datasets.

INTRODUCTION

It is important for analysts and researchers to get access to high-quality individual-level data for secondary purposes (such as for building statistical and machine learning models). In the case of clinical trials, the reanalysis of data from previous studies can provide new insights compared with the original publications.¹ Secondary analysis has produced informative research results including on drug safety, evaluating bias, and replication of studies, and for meta-analysis,² with the most common purposes being new analyses of the treatment effect and the disease state.³ Also, there has been strong interest in making more clinical trial data available for secondary analysis by academia, the pharmaceutical industry, and regulators.^4–9

However, data access remains a challenge. For example, an examination of trials registered on ClinicalTrials.gov found that only 15% of trials launched in 2019 plan to share data.¹⁰ An analysis of the success rates of getting individual-level data for research projects from authors found that the percentage of the time these efforts were successful varied significantly and was generally low at 58%,¹¹ 46%,¹² 14%,¹³ and 0%.¹⁴ Some researchers note that getting access to datasets from authors can take from 4 months to 4 years.¹⁴

One reason for this challenging environment is increasingly strict data protection regulations: a recent National Academy of Medicine and Government Accountability Office report highlights privacy as presenting a data access barrier for the application of AI and machine learning in healthcare.¹⁵ While patient (re)consent is one legal basis for making data available for secondary purposes, it is often impractical to get retroactive consent under many circumstances and there is significant evidence of consent bias.¹⁶

Anonymization is another approach for addressing privacy concerns when making clinical trial data available for secondary analysis. However, there have been repeated claims of successful reidentification attacks on anonymized data,^17–23 eroding public and regulator trust in this approach.^23–32

To solve this problem, there is growing interest in using and disclosing synthetic data instead. There are many use cases where synthetic data can provide a practical solution to the data access problem.^33,34 In fact, it was recognized some time ago that synthetic data are a key approach for data dissemination compared with more traditional disclosure control methods.³⁵ Furthermore, data synthesis has been highlighted as a key privacy enhancing technology to enable data access for the coming decade.³⁶

Multiple researchers have noted that synthetic data does not have an elevated identity disclosure (privacy) risk because there is no unique or one-to-one mapping between the records in the synthetic data with the records in the original data.^35,37–43 Therefore, our focus in this article will be on ensuring that the synthesized data has sufficient utility, which is generally defined as the ability to replicate patterns and conclusions that were in the original data from the synthetic data.

Classification and regression trees⁴⁴ have been proposed for data synthesis when implemented in a sequential manner.⁴⁵ Using a scheme similar to sequential imputation,^46,47 trees are used quite extensively for the synthesis of health and social sciences data.^48–56 With these types of models, a variable is synthesized by using the values earlier in the sequence as predictors. Conceptually, sequential synthesis is similar to modeling multiple outcome variables using classifier chains⁵⁷ and regressor chains.⁵⁸ Compared with deep learning synthesis methods that require large datasets,^37,59–61 sequential decision trees work well for small datasets, such as clinical trial data, and work well with heterogenous variable types (such heterogeneity remains an area of research using other synthesis methods).⁶²

It is known that the order of the variables can influence the accuracy of model chains.⁶³ The dependence of the synthetic data utility on the order that the variables are synthesized in is also a recognized issue⁶⁴ but has not been investigated in depth. For sequential data synthesis, variable order is important because each variable’s generative model is fitted using only the variables before it in the order. When the preceding variables are weak predictors of subsequent variables, the synthesized values will have low utility, and synthesis errors will propagate, and potentially be amplified, through the chain. If the utility is dependent on variable order, then an arbitrary factor would effectively be driving nontrivial variation in the quality of synthesized data.

One approach to address this problem is to select the highest utility dataset among those generated through multiple random variable orders. However, this would not ensure that the utility meets acceptable threshold levels, and it is an inefficient way to search for a good utility variable order. One can instead model the dependence among the variables and select the variable order accordingly. However, dependence does not imply directionality, which is important for selecting an order.

The purpose of the current study is to assess the variation in the utility of synthetic clinical trial data generated using sequential decision trees, and if the variation is high, to optimize the order to meet data utility thresholds. In such a case, the optimal selection of variable order will ensure more consistent results and better data utility.

MATERIALS AND METHODS

Our methods had 2 main components. The first was to simulate and assess the variation in the utility of synthetic data due to variable order when using sequential techniques. This analysis would allow us to draw empirically supported conclusions about the extent to which variable order is indeed a problem. The second component was to evaluate an optimization algorithm that would target a threshold level of utility in the generated data. In that way we would have stronger assurances that variable order would not degrade utility in an arbitrary manner.

In the following sections we describe the clinical trial datasets and their sources, the variable order simulation process, the data utility metrics that we used, and the optimization method we used and its evaluation.

Sources of clinical trial data

To perform our simulations, we need access to clinical trials data. The following discussion presents the criteria and considerations for identifying usable datasets for our analysis. However, it also highlights the challenges in getting access to clinical trial datasets for this type of research project.

We defined the following criteria when selecting a data source for the clinical trial datasets:

• Access should be provided to individual-level patient data (instead of documents or summary statistics). The individual data is needed to perform the simulations.

• The datasets should be downloaded. We have access to significant high-performance computing CPU and GPU capacity to enable us to perform the simulations in a reasonable amount of time and with negligible incremental costs. Alternatively, any data source that only allows access to data in a virtual secure enclave would need to provide comparable cost-effective high-performance computing capacity for us to perform the study.

• The datasets should be readily available to other researchers to enable replication and extension of the current analysis.

• Relative time from request to data access, and incremental costs would also be relevant factors in selecting data sources.

Three data sources for clinical trial datasets were examined: (1) regulatory authorities (ie, the European Medicines Agency, Health Canada, and the Food and Drug Administration); (2) individual investigators; and (3) data sharing platforms, namely the Yale University Open Data Access project,⁶⁵ Project Data Sphere,⁶⁶ clinicalstudydatarequest.com,⁶⁷ and Vivli.⁶⁸ In Supplementary Appendix A, we provide a review of potential data sources and assess them with respect to our previous criteria.

We concluded that, relative to other options, the Project Data Sphere (PDS) data sharing platform met our criteria, while other options would not have met all of our criteria or would not have been practical for our purposes: we were able to download individual-level patient datasets within days of making the request to run the simulations on our own high-performance computing infrastructure, other researchers can readily request the same data under the same conditions, and there would be no incremental costs. PDS has holdings of oncology clinical trial datasets sponsored by industry and public funders.

We selected 6 studies that could be downloaded from PDS, ensuring that we had variability in the number of patients, the number of variables within the range typical of clinical trials, disease areas, and therapeutic interventions. This would allow for a broader generalization of the results.

Datasets

We performed our simulations on the 6 oncology clinical trial datasets from PDS as summarized in Table 1. We considered the screening criteria, demographic variables, medical history, baseline characteristics, and the endpoints in this analysis.

Table 1.

Summary of the 6 oncology trials used in the analysis with the National Clinical Trial number and the primary sponsor indicated, as well as the number of patients and variables used in the synthesis

Dataset	Individuals	Variables
		Total	Binary/Categorical		Discrete/Continuous
Trial 1 (NCT00041197): National Cancer Institute
Tests if postsurgery receipt of imatinib could reduce the recurrence of GISTs. Imatinib is an Food and Drug Administration approved protein-tyrosine kinase inhibitor for treating certain cancers of the blood cells. This drug is hypothesized to be effective against GIST as imatinib inhibits the kinase which experiences gain of function mutations in up to 90% of GIST patients.69 At the time of this trial the efficacy of imatinib for GISTs as well as the optimal dosage for treatment of GISTs was unknown.	773	129	71 (55)		58 (45)
Trial 2 (NCT01124786): Clovis Oncology
Most pancreatic cancer patients have advanced inoperable disease and potentially metastases. At the time of this trial the first line therapy for patients with inoperable disease was gemcitabine monotherapy. One transporter (hENT1: human equilibrative nucleoside transporter-1) has been identified as a potential predictor of successful treatment via gemcitabine. This trial compares standard gemcitabine therapy to a novel fatty acid derivative of gemcitabine. This is hypothesized to be superior to gemcitabine in metastatic pancreatic adenocarcinoma patients with low hENT1 activity as it exhibits anticancer activity independent of nucleoside transporters like hENT1, while gemcitabine seems to require nucleoside transporters for anticancer activity.	367	88	24 (27.2)		64 (72.3)
Trial 3 (NCT00688740): Sanofi
This phase 3 trial compares adjuvant anthracycline chemotherapy (fluorouracil, doxorubicin, and cyclophosphamide) with anthracycline taxane chemotherapy (docetaxel, doxorubicin, and cyclophosphamide) in women with lymph node positive early breast cancer.	746	239		148 (61.9)	91 (38.1)
Trial 4 (NCT00113763): Amgen
This was a randomized phase 3 trial examining whether panitumumab, when combined with best supportive care, improves progression-free survival among patients with metastatic colorectal cancer, compared with those receiving best supportive care alone.70,71 Patients included in the study had failed other chemotherapy options available at the time of the study. Participants were enrolled between 2004 and 2005.	463 (sponsor only provided 370 in the dataset)	59		22 (37.2)	37 (62.8)
Trial 5 (NCT00460265): Amgen
This was also a randomized phase 3 trial on panitumumab but among patients with metastatic and/or recurrent squamous cell carcinoma of the head and neck. The treatment group received panitumumab in addition to other chemotherapy (cisplatin and fluorouracil), while the control group received cisplatin and fluorouracil as first-line therapy.72 Participants were enrolled between 2007 and 2009.	657 (sponsor only provided 520 in the dataset)	401		162 (40.3)	239 (59.6)
Trial 6 (NCT00119613): Amgen
This was a randomized and blinded Phase 3 trial aimed at evaluating whether “increasing or maintaining hemoglobin concentrations with darbepoetin alfa” improves survival among patients with previously untreated extensive-stage small cell lung cancer. The treatment group received darbepoetin alfa with platinum-containing chemotherapy, whereas the control group received placebo instead of darbepoetin alfa.	600 (sponsor only provided 479 in the dataset)	382		82 (21.4)	300 (78.6)

Values are n (%). Variables are classified as either binary/categorical or ordered discrete/continuous. Dates are converted to relative days and therefore are considered continuous.

GIST: gastrointestinal stromal tumor.

Simulation and synthesis processes

For the simulations, we repeated the synthesis 1000 times for each dataset, and each time randomly shuffling the variable order that was used in the sequential tree generation process. The method we used to generate synthetic data is called conditional trees,⁷³ although other tree algorithms could also be used. A summary of the algorithm is provided in Figure 1, and additional implementation and data preparation details are included in Supplementary Appendix B. When a fitted model is used to generate data, we sample from the predicted terminal node in the tree to get the synthetic values.

Figure 1.

A description of the sequential data synthesis process using classification and regression trees. Although any set of classification and regression methods can be used in principle.

Evaluating synthetic data utility

Data utility is broadly defined as the ability to replicate patterns and conclusions that were in the original data from the synthetic data. The type of results that should be examined will depend on the data uses. For example, a statistical analysis of the data will require different utility evaluations and benchmarks than using the data to test statistical software.

A recent review identified 7 types of utility assessments for synthetic data.⁷⁴ Our focus here is on analytic uses of the data and therefore we examine general metrics comparing real and synthetic data analysis results. These metrics are not workload aware, but they reflect common data analysis tasks and are used often within the data synthesis and machine learning community.

For each simulation iteration, the synthetic data utility was estimated using 3 metrics, as summarized in El Emam et al³⁴: comparisons of univariate distributions, “all-models” comparisons of multivariate prediction accuracy, and distinguishability. The former 2 reflect a broad spectrum of common statistical analysis on clinical datasets. The latter is an omnibus comparison of multivariate distributions using a binary classifier,^75,76 and is equivalent to the discriminator function used to evaluate (and train) performance in generative adversarial networks (an architecture of deep artificial neural networks).⁷⁷

Univariate comparisons

We first compared the univariate distributions between the real and synthetic datasets on all variables. The comparison of univariate distributions as a utility metric is common in the synthesis literature.^37,78 For that purpose we utilized the Hellinger distance.⁷⁹ This has been shown to behave in a consistent manner as other distribution comparison metrics in the context of evaluating disclosure control methods when comparing original and transformed data.⁸⁰ It also has the advantage of being bounded between 0 and 1, which makes it easier to interpret. We computed the median Hellinger distance across all variables for each iteration during the simulation.

Comparing multivariate predictions

The second metric was a measure of multivariate prediction accuracy. It tells us the extent to which the prediction accuracy of synthetic data models is the same as the models from the real data.^81,82 The comparison of prediction model accuracy has been used, for example, to compare the prediction of hospital readmissions between real and synthetic data⁵⁹ and to predict treatment arms from a large synthetic clinical study dataset.⁸³

We built general boosted regression models,⁸⁴ taking each variable as an outcome to be predicted by all of the other variables. Hence, we built “all multivariate models” for the synthetic and real datasets. For each model 10-fold cross validation was used to compute the area under the receiver-operating characteristic curve (AUROC)⁸⁵ as a measure of model accuracy. We then compared the synthetic data and the real data accuracy by computing the relative absolute difference in the median AUROC measures for each dataset. Because AUROC requires a discrete variable, we discretized all continuous outcome variables using univariate k-means.⁸⁶

Distinguishability

The third utility metric is based on propensity scores.^87,88 The real and synthetic datasets are pooled, and a binary indicator is assigned to each record depending on whether it is a real data record or a synthetic data record. A binary classification model is then constructed to distinguish between the real and synthetic records where the original variables are predictors and the binary indicator variable is the outcome. A 10-fold cross-validation is used to compute the propensity score (the predicted probability). The specific classification technique we use is generalized boosted models.⁸⁹

The distinguishability score is computed as the mean square difference of the predicted probability from 0.5, which is the value where it is not possible to distinguish between the 2 datasets:⁸⁷

$$d=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N$}\right.\sum _i{\left(p_i-0.5\right)}^2$$ (1)

where N is the size of the synthetic dataset and $p_i$ is the propensity score for observation i.

If the 2 datasets are the same, then there will be no distinguishability between them. One reason for such a result would be if the synthetic data generator was overfit and effectively recreated the original data. In such a case the propensity score of every record will be close to or at 0.5, in that the classifier is not able to distinguish between real and synthetic data, and $d$ approaches 0. If the 2 datasets are completely different, then the classifier will be able to distinguish between them. In such a case the propensity score will be either 0 or 1, with $d$ approaching 0.25.

Across all 1000 simulation runs, we examined the median and 95% confidence interval on each dataset for the 3 utility metrics (the 2.5th percentile and the 97.5th percentile). This will indicate how stable the utility of the datasets are as the variable order is shuffled.

Because the generation of synthetic data is stochastic, there will be confounding variability in the utility metrics due to the synthesis process itself. Therefore, we average this out by generating 50 synthetic datasets for each of the 1000 variable orders, compute the utility metrics, and take the average of these 50 values to represent the value for that variable order. That way, we can factor out the impact of the stochastic synthesis process from the variability that we are interested in measuring. We did not observe meaningful fluctuations in that average when we used 100 or 150 generated datasets, and therefore the 50 iterations were deemed to be representative.

Curriculum learning

As a baseline method to defining an appropriate order for variables in sequential synthesis, we examined a curriculum learning approach.⁹⁰ For machine learning tasks, it is hypothesized that starting the training with easier or more general examples, and then continuing on to the more complex ones accelerates convergence and improves model accuracy. In the context of sequential data synthesis, it was argued that curriculum learning would better capture the dependence among the attributes.⁶¹

To operationalize this we discretized all continuous variables using univariate k-means.⁸⁶ Variables with fewer categories were put first in the order because they were deemed to be simpler. That curriculum learning order was evaluated as the baseline order, and all utility metrics were computed for that (note that the synthesis was performed on the original data and not the discretized values).

Particle swarm optimization

To optimize on the variable order we used a particle swarm algorithm.^91,92 This uses a search heuristic to find the global optimum without requiring the objective function to be continuous. For the objective function we computed the utility metrics and used a hinge loss function that was being minimized.⁹³ The hinge loss considers the utility metric to be 0 if it is below a threshold. For example, the distinguishability loss is 0 if distinguishability is below 0.05. The threshold ensures that we do not overfit the generated trees to the data. The overall loss for each of the utility metrics is therefore:

$$\begin{array}{c}los{s}_1=\max \left(0,d-0.05\right)\\ los{s}_2=\max \left(0,h-0.1\right)\\ los{s}_3=\max \left(0,a-0.1\right)\end{array}$$ (2)

where $h$ is the Hellinger distance and $a$ is the median relative absolute AUROC difference. A compound loss was also computed as the unweighted sum of all 3 losses as an optimization criterion:

$$\mathit{los}{s}_4=\max \left(0,d-0.05\right)+\max \left(0,h-0.1\right)+\max \left(0,a-0.1\right)$$ (3)

Therefore, in total the optimization was evaluated with 4 different loss functions.

RESULTS

We present 3 sets of graphs showing the different utility scores across all 6 trials.

Figure 2 shows the results across the 6 trials for the Hellinger distance. While there is a little bit of variation, in general the distance was relatively low and the variation within a narrow range.

Figure 2.

The Hellinger distance median value and 95% confidence intervals for the 6 clinical trial datasets. This is a value between 0 and 1, with lower values indicating that the univariate distributions of the real and synthetic variables are similar. In general, values in the lowest decile (≤0.1) would be indicative of reasonable similarity.

Figure 3 has the results for the multivariate prediction models with the AUROC accuracy results. Although trial 4 has the most variation. This trial had the fewest variables and this may have created more instability in prediction performance relative to the other trials, hence the wider variation (although the amount of variation in this case is not that large).

Figure 3.

The relative absolute difference in area under the receiver-operating characteristic curve (AUROC) median value and 95% confidence intervals for the 6 clinical trial datasets. This is a value between 0 and 1, with lower values indicating that the multivariate models built using the real and synthetic datasets are similar. In general, values in the lowest decile (≤0.1) would be indicative of reasonable similarity.

In Figure 4, we see nontrivial variation in the distinguishability score. Specifically, trials 3, 5, and 6 show a large amount of variation due to variable order. The unoptimized (default) variable order does not necessarily capture the dependencies among the variables, and therefore in a sequential synthesis process, a particular variable may have been poorly modeled by the variables before it in the sequence. These poorly synthesized variables are then easier for a discriminator to use to distinguish between the real and synthetic datasets. This inability to capture dependencies is more pronounced the more variables there are in a dataset, as is the case with trials 3, 5, and 6.

Figure 4.

The distinguishability score median value and 95% confidence intervals for the 6 clinical trial datasets. This is a value between 0 and 0.25, with lower values indicating that the overall real and synthetic datasets are not distinguishable from each other by a discriminative model. In general, values in the lowest quintile (≤0.05) would be indicative of reasonable nondistinguishability.

The differences among the 3 utility metrics are not surprising because they are measuring different things, and they are also influenced by outliers differently. However, it is clear that the larger the number of variables there are in the dataset, the greater the variability in the distinguishability score is. There was no discernable relationship between the proportion of categorical vs continuous variables and the utility outcomes.

The curriculum learning baseline order results are shown in Table 2. Compared with the median values that were observed during the simulation, curriculum learning was not consistently better than the random median result, and was not consistently lower than our utility thresholds (used in the loss metrics).

Table 2.

Utility results for the curriculum learning variable order

Trial	Distinguishability	Hellinger	AUROC
1	0.114	0.0147	0.002
2	0.064	0.026	0.001
3	0.2	0.011	0.003
4	0.034	0.03	0.059
5	0.101	0.019	0.012
6	0.232	0.023	0.008

AUROC: area under the receiver-operating characteristic curve.

After optimization, we show the results in Table 3 for optimizing on distinguishability ($\mathit{los}{s}_1$). An example illustrating how the optimization has affected the ordering of variables is provided in Supplementary Appendix C, in which we show the original ordering and the ordering after optimization that produced the result in Table 3. The results for the Hellinger distance loss, relative absolute AUROC difference loss, and the compound loss are included in Supplementary Appendix D.

Table 3.

Utility results after the optimal variable order was selected with optimization on distinguishability

Trial	Distinguishability	Hellinger	AUROC
1	0.0113	0.0118	0.0019
2	0.033	0.027	0.001
3	0.049	0.017	0.0026
4	0.02	0.0204	0.0584
5	0.044	0.0135	0.0118
6	0.0388	0.0277	0.009

AUROC: area under the receiver-operating characteristic curve.

We can make 3 observations. The first is that optimization on the Hellinger distance and the relative absolute AUROC difference do not ensure that the utility meets the threshold for the other utility metrics. The second is that with distinguishability as the main loss ($\mathit{los}{s}_1$) we meet the threshold values for the other utility metrics. The third observation is that the results for optimizing on the distinguishability loss ($\mathit{los}{s}_1$) are the same as for the compound loss ($\mathit{los}{s}_4$). When we checked the optimal orders, they were also the same when using distinguishability loss and compound loss. The optimization on the compound loss is more computationally intensive than just for distinguishability. Given that the results are the same across all 6 trials, then a strong case can be made for only optimizing on the distinguishability metric.

Additional details on the performance of the optimization approach compared with random search is included in Supplementary Appendix E.

DISCUSSION

Summary

Our results indicate that the variation in the data utility of synthesized clinical trial data using (unoptimized) sequential trees was impacted significantly by the variable order, after accounting for natural variation due to the stochastic nature of data synthesis. The variability in utility was more pronounced as the number of variables increased, meaning that some orders will result in quite poor utility results on some of the key utility metrics. The number of patients did not seem to play a prominent role in affecting the variation in data utility, although our datasets did err on the small side.

Optimization using the particle swarm algorithm combined with a distinguishability hinge loss reliably found the variable orders that ensure that the utility metrics are below an acceptable threshold level across multiple utility metrics. This optimization strategy also had better utility than a curriculum learning approach. Particle swarm optimization was faster than a random search for an acceptable variable order. Therefore, we recommend implementing this optimization strategy when using sequential synthesis techniques. Furthermore, using a curriculum learning strategy to set the order of the variables did not ensure that the utility was consistently acceptable across datasets and utility metrics, and therefore that approach is not recommended.

Practical implications

Sequential trees as a method for data synthesis have the advantage in that they can work with datasets with few patients as well as for very large datasets (this is a feature of decision trees in general), datasets that are heterogeneous in the variable types (eg, combining continuous and high-cardinality categorical variables), and for datasets with significant missingness. These are common characteristics of many real health datasets. While the impact on datasets with few variables will be smaller, there would be minimal downside to the optimization of variable order every time a clinical trial dataset is synthesized using sequential trees. And our results suggest that this optimization should be performed on a hinge loss function based on the distinguishability metric.

Order-optimized sequential trees can therefore be good methods for the synthesis of clinical trial data. Other methods, such as deep learning models that are also being applied to the synthesis of health data, do not scale down well to the small datasets typically encountered with clinical trials.

For data custodians sharing synthetic data, providing data utility metrics as part of the documentation accompanying the synthetic data would be desirable. This allows the data users to validate that the overall utility is acceptable. The key utility metric is distinguishability, but other univariate and multivariate utility metrics, as described in this article, would also be informative.

Research contributions

The contributions of this work are the following: (1) we empirically demonstrate that variable order has an impact on synthetic clinical trial data utility for a sequential synthesis method commonly used on health data, (2) we propose and evaluate a method for optimally selecting the variable order, and (3) this is the first study to examine optimal synthesis for clinical trial datasets. Given the growing demand for access to clinical trial data, this can be another technique to make such data broadly available to researchers. Further research can expand the loss functions to include privacy metrics, and optimize other hyperparameters (eg, tree depth).

Limitations

Our empirical analysis was performed on oncology clinical trial datasets. There is no intrinsic reason to believe that the therapeutic area covered by a dataset would impact the results. The main driver of utility variation was the number of variables being synthesized. There will be nononcology clinical trial datasets with many variables, and nonclinical trial datasets which have many variables. Consequently, these findings should generalize outside oncology and to nonclinical trial data.

Three clinical trials (numbers 4, 5, and 6) had some records missing. They were all from the same sponsor, in which they consistently provided only 80% of the data. It is not clear whether this was a measure to protect patient privacy, or if there was another reason for these omissions (eg, these were screening failure patients). Given that we did not find the number of patients or observations to be a driver of the outcomes we were interested in, the impact on our results is expected to be limited.

Our utility metrics were not workload aware. Future studies could further evaluate the optimization strategy presented here on replications of published research analyses.

We did not examine the privacy risks from optimizing sequential synthesis. These risks can be controlled by adjusting the threshold in the loss function, and therefore privacy considerations would not take away from the importance of dealing with the arbitrary utility impacts of variable order.

Conclusions

There is growing demand to access clinical trial data. Data synthesis is one way to address this demand and simultaneously satisfy privacy concerns. The objective of this study was to evaluate and optimize the variable order for sequential synthesis of clinical trial data, with the loss being a measure of synthetic data utility. Sequential synthesis is commonly used for health and social science datasets, and it is suited to synthesizing small datasets such as clinical trial data. Furthermore, it can handle heterogeneous inputs and missingness well.

Through a simulation on oncology trial data we found that variable order affected utility, and more so with a higher number of variables in a dataset. Particle swarm optimization was able to identify variable permutations that ensure consistent data utility. Optimized sequential synthesis can provide a reliable way to synthesize clinical trial data.

FUNDING

This work was partially funded by a Discovery Grant RGPIN-2016-06781 (KEE) from the Natural Sciences and Engineering Research Council of Canada, and by Replica Analytics Ltd.

AUTHOR CONTRIBUTIONS

KEE contributed to designing the study, performed the analysis, and contributed to writing the article. LM contributed to designing the study, prepared the datasets for the study, implemented some of the code used to perform the study, and contributed to writing the article. CZ contributed to designing the study, prepared the datasets for the study, implemented some of the code used to perform the study, and contributed to writing the article.

ETHICS APPROVAL

This project was approved by the Children’s Hospital of Eastern Ontario Research Institute Research Ethics Board, protocol number CHEOREB# 20/39X.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online