Synthetic data can aid the analysis of clinical outcomes: How much can it be trusted?

In 1997, then-governor of Massachusetts Bill Weld received a nasty surprise in the mail: his medical records, sent by computer scientist Latanya Sweeney. It was a stunt to demonstrate how supposedly anonymous data released by the state on citizens could be cross-referenced to re-identify people and so reveal their secrets (1).

Machine learning offers a way to share patient data with a much-reduced risk of re-identification. Image credit: Shutterstock/Chaikom.

The incident was an opening salvo in a struggle between personal privacy and researchers’ hunger for data. In the decades since, a mosaic of overlapping policies and restrictions have been put in place by countries and organizations that rightly preserve privacy. These policies also hinder the distribution and sharing of potentially useful datasets among scientists. And the more data-dependent clinical science becomes, the more researchers feel the constraints.

Machine learning now offers what seems at first blush a bizarre solution to allow data sharing with a much-reduced risk of re-identification. Rather than studying the details of real citizens, researchers are using statistics and AI to generate large cohorts of fake patients to work with instead.

Starting with a dataset from real volunteers and patients, the researchers first identify and map its key characteristics. They then generate a new set of artificial data with the same characteristics. That could be everything from the mean age of a cohort to the more complex mathematical relationships between several related physiological variables. So, while the fake datasets do not correspond to private information from real individuals, they should still contain the useful patterns and information within a population that researchers want to analyze.

By replicating the key properties of real people in the data of the fakes—patterns of genetic variation, perhaps, or sexual orientation—these synthesized datasets provide a clinical population that scientists can study freely. And because the synthetic profiles do not match real individual records, it’s much harder to trace and identify those who volunteered their data.

Called synthetic data, this approach emerged to protect the privacy of census and economic data. But researchers are realizing its power to do that and much more in clinical research. Synthetic patients could be used to generate larger sample sizes and so increase the statistical power of analyses (2). The technique can address biases in historical information and fill holes where real-world data are hard to find, such as on the impact of medicines in pregnant women and children.

Promising Tool

Synthetic data have already been used to generate biobank-style population-scale datasets that allow researchers to find links between diseases and genotype (3), search for risk factors of suicide among military veterans (4), and imitate the behavior of construction workers to train computer vision systems that monitor safety on building sites (5).

The restrictions on sharing real data mean there is “an urgent need for new technologies and new opportunities to increase the capability to provide access to healthcare data for the scientific community,” says Matteo Della Porta, a hematologist at Humanitas University in Milan, Italy, who developed an AI-based tool to generate synthetic populations. “And synthetic data is one of the most promising approaches.”

They’re promising, but not yet foolproof. While synthetic populations based on demographics are commonly used by economists and others interested in modeling human activities, experts warn that synthetic datasets are not yet ready to replace real-world information in clinical science without a lot more validation. Confirming that the synthetic dataset yields the same results as the original from which it’s derived requires access to the original patient data, which restricts the effort to a small number of groups. And this is why some in the field argue that, in the short term, the greatest benefits of synthetic data may come from its use to improve the realism of preclinical simulations and models in drug and medical device development.

“If you talk to most pharma companies who are doing drug applications, there will be some element of simulation,” says Puja Myles, a data expert with the UK’s Medicines and Healthcare Products Regulatory Agency in London. “It may not normally make big headlines, but that’s happened already in the background, even before they’ve gone to the first clinical trial.”

Keeping Confidential

The use of synthetic data in research dates to the early 1990s, when Donald Rubin, a statistician at Harvard University, was working on ways to preserve the confidentiality of data collected by the US Census before it was released to academics and others.

At the time, such datasets were often, prior to release, altered to make records hard to re-identify—whether by masking some details or by pooling others. For example, research might present data by region rather by than individual zip code. But Rubin, now retired, thought this was too complicated and risked legal challenge as data analysis techniques improved.

“It seemed to me so cumbersome,” he says. “The only way of really releasing data that satisfied all the confidentiality constraints that may come up in the future was not to release anything that was confidential.” Which is to say, any of the original identifiable data.

“Ultimately, you still have to have some sort of human testing; we can’t work entirely in a model. But this helps you fine-tune and reduce any safety concerns even further.”

—Puja Myles

He came up with an alternative solution. By using statistical models, he was able to capture the distribution of important features, including racial group and location, from the real census data and replicate them in a synthetic dataset. Effectively, Rubin created records for millions of US citizens who never existed, but whose personal details mirrored those of the real population. Colleagues told him the idea was “absurd,” he says, given the limits then of computational power. “But by building models based on real data, I could generate useful fake data.”

He converted the data by using a statistical technique called a posterior predictive distribution, which generates possible unobserved values based on the spread of observed data. In all, he produced a handful of synthetic databases, each of which contained the details of a million different fictional citizens, which could be passed to sociologists and others with no risk to privacy (6).

Using this technique, the US Census Bureau began, about 20 years ago, to make available two standard sets of synthetic data based on the information it collects. The first, the Synthetic Longitudinal Business Database, holds records of 21 million firms for the years 1976 to 2000. The second, the Survey of Income and Program Participation (SIPP) Synthetic Beta database, combines person-level microdata from household surveys from 1984 to 2008 with administrative tax and benefit data. The fake citizen information has been used to study, for example, why certain people earn more over their lifetimes.

The apparent usefulness of synthetic data, Rubin says, convinced him that the idea would take off. In the years since, statisticians have developed other ways to map the key patterns and relationships in datasets and to re-create those in synthetic data. These can be complex and time-consuming. Machine learning can do the same job, so AI-based methods have emerged in recent years. The result is a range of available AI and statistical approaches that can generate synthetic data in different contexts, including for clinical research studies.

One research study compared analyses of patient survival time for myelodysplastic syndromes, a group of cancers that affect bone marrow, using synthetic versus real patient data (A versus B). They found no difference in survival probabilities between the two datasets. Image credit: Reprinted with permission from ref. 2.

Synthesizing Data

In Milan, Della Porta employs a conditional generative adversarial network (GAN)—a type of AI in which two systems compete to generate fake data that cannot be distinguished from real data. Like all GANs, the system uses two models: one that learns the features of a real dataset to create new data with similar characteristics and a second, a discriminator, that tries to classify examples as either real or fake. The two models essentially compete, with the first trying to fool the second and the second trying to spot the fake data. As they compete, the generated data become more similar to the real data, and so more useful.

Focusing on people with the rare blood disease myelodysplastic syndromes (MDS), a cancer in which bone marrow does not work properly, Della Porta’s team trained a GAN model by using data from 2,000 real MDS patients (2). Relevant information presented to the AI included age at diagnosis and survival, as well as clinical and genomic observations such as blood platelet count, hemoglobin level, and mutations in specific genes.

After studying real patient data, the computer could generate a mirror cohort of 2,000 fake patient records, which captured key features of the real patients, such as clinical measurements, likely treatment, and outcomes. Importantly for privacy preservation, none of the 2,000 synthetic patient records re-created the complete details of a real person from the training set. This makes it much simpler for the Italian group to share its data on MDS cases with other research groups and companies, Della Porta says.

Each of the 2,000 real patients needed to give their consent for their data to be studied by the AI. “But when the transformation into synthetic data is completed, no further authorization is needed in order to share data, even for commercial use,” he adds.

Last year, his team used the AI findings to make available an online tool that allows those working on MDS to generate cohorts of up to 10,000 synthetic patients, complete with clinical, genomic, and follow-up data. That’s a potential game-changer for research on disorders for which “very few data are publicly available,” Della Porta says.

His group does not currently track who downloads the synthetic data or how they use it, but he says it could help drug companies investigate treatments for MDS and help researchers generate large sample sizes to test hypotheses on mechanisms and possible targets to tackle the disease.

But even advocates for synthetic data agree that clinical conclusions drawn from it need to be validated by confirming the findings using the original patient information.

Indeed, as with any data manipulation technique, synthetic data are only as good as the real-world data on which they’re based. The quality of synthetic data generation will be closely tied to the accuracy and completeness of the underlying real-world data. Imperfections, biases, or gaps in the real-world data will infect the synthetic data as well. And synthetic data could, in principle, fail to capture the complexities found in real-world datasets, potentially leaving out crucial details or relationships.

Seeking Validation

Thus far, however, early tests of synthetic data show that they can be a reliable substitute for real data. Work by Randi Foraker, a bioinformatics researcher at the Washington University School of Medicine in St. Louis, and others suggests that the results from studies with synthetic data are consistent with those from studies that use real data (7). Foraker’s team has tested, for example, applications such as predicting sepsis among hospital patients from medical records and calculating annual rates of chlamydia by zip code (8). “So far, my research team and I have not gotten a different answer with the original data as compared to the synthetic data,” she says.

Validating the accuracy and, hence, the utility of the synthetic populations has been slow-going. It takes both access to original patient data and the expertise to convert those to synthetic data, as well as the skills to compare the two. That’s a relatively rare combination in what is still a young field, Foraker says. “There’s only a few of us. So, the data that we have aren’t necessarily representative of the entire patient population,” she says. That puts synthetic data researchers in a bind. “If we don’t have more people doing this work with synthetic data, then we have just a limited number of patients that we’re basing these conclusions on.”

As in any research based on statistical methods, sample size is crucial. In 2020, Foraker’s team published a study that showed analysis of AI-generated synthetic cardiac patients could accurately predict the risk of heart failure (9). That study trained the AI on data from 26,000 real patients over a decade of admissions. “When you have a sample size of 26,000 patients, you’re not worried at all,” she says.

More problematic, she suggests, could be research on a rarer disease. Synthesizing a dataset from, say, just 10 cases of monkeypox carries a greater trade-off between protecting privacy and being useful. In that case, some information from the original data—a date of birth, perhaps, or gender or race—is masked before the AI sees it. And that means the synthetic patients it produces are less realistic. “With that resulting dataset, I would’ve been better off using the original data because I wouldn’t have had those holes,” Foraker says.

Trial and Error

Use of synthetic patients in primary clinical research might have to wait, but synthetic data advocates say that the field has the potential to streamline and improve the work in other ways.

The Milan scientists are currently talking to European regulators about how their synthetic patients could play a role in trials. One idea is to use the condition and progress of these fake patients as a control group to help assess the efficacy of possible new treatments. Right off the bat, that would halve the number of real patients needed for a randomized controlled trial, Della Porta points out, making such studies quicker and cheaper. It would also avert the ethical conundrum of recruiting ill people for research and then giving them a placebo.

Some trial organizers have experimented with other ways to generate alternative control groups, such as using historical or unrelated patients, or so-called digital twin control arms. None of these are synthetic populations, although these kinds of data are commonly referred to as “synthetic.” But as far as the US Food and Drug Administration and European Medicines Agency know, no one has used a truly synthetic cohort as a control group in an approved trial.

Myles, with the UK’s regulatory agency, says regulators are still working out their approach to synthetic data. “So far, at least for us in the UK, we have not accepted a pure synthetic data control arm. But I do not see why that would be an issue,” she says.

A more interesting question for synthetic data, she says, is whether it can be used to augment the often-sparse information available to simulate how new therapies might help understudied populations, such as pregnant women and children. Synthetic cohorts of such patients could significantly improve so-called in silico trials, she says, which use computer models to assess, say, the likely physiological effects of inserting a heart valve.

Results from such trials alone are unlikely to suffice for regulatory approval, Myles notes, but they could offer a vital way to generate useful data at an early preclinical stage. “Ultimately, you still have to have some sort of human testing; we can’t work entirely in a model. But this helps you fine-tune and reduce any safety concerns even further,” she says.

With applications like these, Myles argues that synthetic data have now moved beyond Rubin’s initial emphasis on protecting privacy to become a powerful tool in their own right—and one with potential that many researchers have not yet fully realized. “There is so much more that you can do with synthetic data,” she says.