An ELIXIR scoping review on domain-specific evaluation metrics for synthetic data in life sciences

Abstract

Synthetic data (SD) has become an increasingly important asset in the life sciences, helping address data scarcity, privacy concerns, and barriers to data access. Creating artificial datasets that mirror the characteristics of real data allows researchers to develop and validate computational methods in controlled environments. Despite its promise, the adoption of SD in life sciences hinges on rigorous evaluation metrics designed to assess their fidelity and reliability. To explore the current landscape of SD evaluation metrics in distinct life sciences domains, the ELIXIR Machine Learning Focus Group performed a systematic review of the scientific literature following the PRISMA guidelines. Six critical domains were examined to identify current practices for assessing SD. Findings reveal that, while generation methods are rapidly evolving, systematic evaluation is often overlooked, limiting researchers’ ability to compare, validate, and trust synthetic datasets across different domains. This systematic review underscores the urgent need for robust, standardized evaluation approaches that not only bolster confidence in SD but also guide its effective and responsible implementation. By laying the groundwork for establishing domain-specific yet interoperable standards, this scoping review paves the way for future initiatives aimed at enhancing the role of SD in scientific discovery, clinical practice and beyond.

Introduction

Synthetic data (SD) is gaining traction across fields to address the scarcity of real data (RD), privacy concerns, and access limitations, while enabling robust machine learning (ML) and artificial intelligence (AI) applications and benchmarking studies [1]. By generating artificial datasets, SD allows researchers to develop and validate computational methods in controlled environments by facilitating the evaluation of statistical models and predictive algorithms. However, as technologies continue to advance, the volume and complexity of biological and clinical data have grown significantly. While this expansion presents new opportunities for discovery, it also introduces challenges in ensuring data availability, quality and privacy protection. A promising solution lies in SD, which enables the creation of high-fidelity datasets that bypass privacy restrictions while preserving essential statistical properties [2].

Beyond practical considerations, SD is particularly valuable in contexts where strict data governance regulations limit access to RD. Legal frameworks like the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA) impose stringent requirements on sharing and using sensitive data in healthcare research [3]. SD offers a potentially viable solution by providing synthetic versions of patient records that maintain statistical properties of RD while reducing privacy risk. This facilitates inter-institutional collaborations by loweing bureaucratic barriers and ensuring compliance with ethical and legal standards [4].

The effective integration of SD into scientific and clinical workflows depends on its ability to accurately mirror the structure of RD. Evaluating SD quality remains a major challenge, as it requires assessing both structural and inferential fidelity in addition to its practical utility for downstream analyses. Evaluation needs vary by application: some focus on improving ML/AI model performance, while others aim to uncover novel patterns for hypothesis generation. Consequently, there is no one-size-fits-all evaluation method. Assessment strategies must be tailored to the goals and context of each use case [5].

To address these challenges, robust evaluation recommendations and best practices are essential for assessing SD across diverse applications. Much like ongoing initiatives in areas such as cross-domain FAIR data assessment and the Critical Assessment of Structure Prediction (CASP), the development of guidelines for SD could progressively lead to a more structured and coherent framework for evaluating prediction methods across domains. Such guidelines will enhance the trustworthiness of SD, ensuring its reliability in biomedical research and clinical decision-making [6]. Furthermore, as SD continues to mature, its successful integration into healthcare will require more rigorous evaluation recommendations and clear regulatory guidelines as well as ongoing adaptation to uphold its reliability and ethical implementation in medical research and practice.

The ELIXIR ML Focus Group is part of ELIXIR, the European infrastructure for life sciences, which brings together over 250 research organizations across 24 countries. In this work, a collaborative effort was led by the ELIXIR ML focus group, specifically the Task Force on SD, in order to establish an evaluation set of guidelines for SD in life sciences within Europe. A scoping review based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) standards was carried out by experts in the Task Force across different domains adjacent to life sciences in order to identify the current evaluation metrics that are utilized to assess SD (individual flow diagrams for each domain are available in Supplementary Figs S1–S6). The key domains selected for the review are genomics, transcriptomics, proteomics, phenomics, imaging, and electronic health records (EHRs), ensuring a thorough and comprehensive investigation of the topic. Figure 1 displays a word cloud visualization of the metrics identified across the six domains, illustrating the diversity of the metrics in a visual format (with all metrics visualized collectively in Supplementary Fig. S7).

Figure 1.

Word cloud visualization of the identified evaluation metrics in six application domains in the life sciences. Legend: Word cloud visualization of the metrics identified across genomics, transcriptomics, proteomics, phenomics, imaging, and EHRs. The figure illustrates the diversity of terms used as evaluation metrics in different domains of SD research. Alt text: A word cloud showing various metric names of different sizes, representing how frequently each metric appears across six SD domains including genomics, transcriptomics, proteomics, phenomics, imaging, and EHRs.

This scoping review maps current methodologies and highlights gaps in evaluation strategies to support the systematic assessment of SD quality, reliability, and utility in life sciences research. While existing studies provide in-depth surveys of SD quality metrics in biomedicine, they tend to focus on specific aspects, like privacy and utility [7], benchmarking general-purpose implementations [8], or particular healthcare domains collectively [9]. To our knowledge, this is the first comprehensive review that systematically examines evaluation metrics for SD across multiple application domains in the life lciences, addressing each domain in detail.

Materials and methods

Inclusion criteria

Following the PRISMA [10] guidelines, we devise a systematic approach to identifying relevant literature on SD across various domains, ensuring the inclusion of high-quality, peer-reviewed, and openly accessible research. To identify relevant studies, we conducted a comprehensive literature search using inclusion criteria as shown in the PRISMA diagram for each domain (Supplementary Figs S1–S6 and Supplementary Material).

The selection of studies for this scoping review was guided by predefined inclusion criteria to ensure a comprehensive yet focused assessment of evaluation metrics for SD. Literature searches were conducted across multiple databases, including PubMed, SCOPUS, and Google Scholar, while Web of Science (WoS) and DOAJ.org as optional sources as their specialized focus might not align with all research requirements. To maintain relevance to current research trends only studies published within the last ten years were included.

All selected publications were in English and accessible as Open Access articles to facilitate transparency and reproducibility. Furthermore, only peer-reviewed journal publications were considered to ensure the scientific rigor and credibility of the included studies. Given the large volume of unspecific results typically obtained by Google Scholar using queries, those search results were ranked by relevance [11], with the top 20 articles reviewed for potential inclusion. While this approach efficiently captures commonly referenced literature, we acknowledge that relevance rankings are algorithmically determined and not publicly documented. To mitigate this limitation, all retrieved records were manually inspected to ensure representativeness and to confirm that key, highly cited papers in each domain were included. This approach ensures that the review captures high-quality, methodologically sound literature that contributes meaningfully to the evaluation of SD (see Table 1).

Table 1.

The inclusion criteria used for our comprehensive literature search in order to identify relevant studies

Criteria	Details
Databases	PubMed, SCOPUS, Google Scholar, Web of Science (WoS) (optional), DOAJ.org (optional)
Timeframe	Studies published in the last 10 years
Language	English
Accessibility	Open Access
Peer-reviewed	Articles from peer-reviewed journals
Ranking	In the case of Google Scholar, top 20 results ranked by relevance

Search queries

A diverse set of queries were formulated, tailored to two key aspects: first, to accurately reflect the specific domain of the evaluation, and second, to align with the search engine used in each case. All queries incorporated the terms “synthetic” and “evaluation”, followed by domain-specific keywords to ensure relevance for each domain. The primary domains deliberately chosen for exploration include genomics, transcriptomics, proteomics, phenomics, imaging and EHRs in order to ensure a comprehensive investigation of the topic. For further details please see the Supplementary Material file.

Identification of relevant publications

The PRISMA 2020 flow diagram, depicted for each domain search in Supplementary Figs S1–S6, provides a structured representation of the study selection process in systematic reviews that involve searches from databases and registers. It consists of three main stages: identification, screening and inclusion. During the identification stage, records are collected from multiple sources including databases and registers. Before the screening process begins, duplicate records are removed and additional exclusions may occur (as described later in the limitations section) due to automation tools or other filtering criteria.

During the screening stage, the remaining records undergo an initial review to determine their relevance. At this step, reports are sought for retrieval and any that cannot be accessed are documented. The retrieved reports are then assessed for eligibility and exclusions are made based on specific reasons, which are clearly recorded (i.e. not relevant for evaluating SD).

Finally, in the inclusion stage, the studies that meet all criteria are incorporated into the systematic review and the total number of included reports is documented. This method ensures transparency and reproducibility by visually summarizing the number of records processed at each stage and detailing exclusions, thus enhancing clarity. All domain-related PRISMA flowcharts can be found in the Supplementary Material (Supplementary Figs S1–S6).

The complete list of all extracted evaluation metrics, their assigned domains, and the corresponding source publications are provided in the supplementary dataset (Supplementary Files 1 and 2), enabling full transparency and reproducibility of the results.

Limitations

During the search process, several observations and challenges were identified. The term “synthetic” often refers to synthetic biology rather than SD generation, leading to unrelated findings. Additionally, many studies mention “synthetic data” in abstracts without detailing evaluation metrics or generation tools. The number of relevant papers varies significantly by domain, with fewer studies found in highly specialized fields such as synthetic phenomics. Furthermore, additional care was necessary to distinguish studies focused on synthetic peptides and data derived from them, as opposed to computationally generated SD in proteomics. Although the majority of the literature search and curation were conducted in the first half of 2024, we continuously reviewed additional papers in each domain and found no additional relevant metrics beyond those that we had already surveyed. This strengthens the authors' confidence that this review accurately reflects the current state of the art.

Results

Synthetic genomics

Domain overview

Synthetic genomics involves using computational tools to design, analyze and simulate synthetic genomes. It aims to create accurate representations of genetic sequences by leveraging bioinformatics and computational biology along with wet-lab expertise. Some key concepts include designing genomes computationally, modelling the functionality of the synthesized genomes, utilizing this type of data for the development and the benchmarking of bioinformatics tools. Designing optimized genomic sequences for specific functions is driving advancements in vaccine development [12] as well as synthetic and evolutionary biology [13]. However, challenges such as data limitations, biosecurity risks, and model interpretability persist, highlighting the need for solutions to ensure safety and ethical compliance.

Review outcomes

After the examination of 769 articles, 7 publications were identified appropriate for our effort. These selected articles fall under two main categories, reviewing genomics simulating tools and suggesting guidelines for the evaluation [14]. Due to the broad use of the term ‘synthetic,’ many irrelevant publications (either outside of the inclusion criteria or due to technical aspects that were unsuitable for this review) were initially identified, requiring us to refine the queries with more appropriate search terms. Furthermore, the term “synthetic DNA” typically refers to data that is generated in wet labs rather than in-silico.

Evaluation metrics

The reported metrics can be divided into two groups: those that can be found and utilized in a standard bioinformatics pipeline and analysis, and those tailored to specific use cases.

• Bioinformatics related metrics [15–17]: Proportion of reads on each chromosome, coverage, type of reads, paired-end fragment length, alignment quality, rates of sequencing error, substitution, insertion, and deletion rates, quality scores for reads, GC-coverage bias, computational cost, mapping sensitivity of bases in reads, read mapping precision, Phred score.

• Use case related metrics [18]: Principal component analysis (PCA) to evaluate qualitatively by visualization and supervised discriminators for quantitative evaluation.

• Statistics based metrics [19]: Mean of significant variables in case/control, dispersion parameter of the negative binomial distribution, and signal to noise ratio, sample size, number of variables, mean of insignificant variables.

Synthetic transcriptomics

Domain overview

Synthetic transcriptomics is a key research area within the field of SD generation for biological studies, enabling the creation and analysis of artificial RNA-Seq data that mirrors the characteristics of RD [20]. Such SD can aid in standard downstream transcriptomics analyses like the identification of differentially expressed genes and isoforms and pathway enrichment analysis [6, 21], as well as more advanced applications, such as the identification of clinically relevant biomarkers. Synthetic transcriptomic data is often used in the context of oversampling approaches to correct imbalance between different sample groups within a dataset. This can help to compensate for bias due to underrepresentation of smaller sample groups, lead to increased statistical power, and reduce the effect of outliers [22]. A key factor contributing to the increasing use of synthetic transcriptomic data relates to the fact that the process of RNA isolation, library preparation, and sequencing to obtain RD is relatively time, labour, and cost intensive, making the alternative approach of generating realistic SD attractive in terms of both efficiency and conservation of resources [23]. This is particularly true for the adoption of synthetic transcriptomic data for the training of ML algorithms where adequate sample size, and restricted access to RD can be a concern.

Review outcomes

The search criteria for ‘synthetic transcriptomic’ or ‘synthetic RNA’ resulted in 424 articles (139 in Scopus, 23 in Pubmed, 16 in Web of Science, 246 in Google Scholar). After removing duplicate records (n = 14), additional filtering was carried out to exclude articles not meeting other criteria (articles in other languages, preprints or conference proceedings (n = 120), articles found to be >10 years old (n = 3), articles belonging to a different domain (n = 33). Ultimately, 84 articles remained for curation and the resulting evaluation metrics from those that report them are summarized below.

A key aspect of SD evaluation in this domain is the distinction between intrinsic and extrinsic metrics. Extrinsic evaluation refers to performance-based metrics that assess how well a model behaves when trained on SD compared to RD. Common extrinsic metrics include precision, recall, F1-score, area under the receiver operating characteristic curve (AUC-ROC), and other standard measures used in ML/AI. These metrics are particularly relevant when assessing the usability of SD in real-world applications. On the other hand, intrinsic evaluation focuses on the inherent quality of SD itself, independent of any downstream task. These metrics assess characteristics such as distribution similarity, perplexity (e.g. in the case of SD for language models), and various statistical measures that capture how closely SD resembles RD. One approach to intrinsic evaluation involves confusion matrices, which can provide insight into how different SD generation methods relate to one another, helping quantify similarities or systematic biases. By combining both intrinsic and extrinsic evaluation methods, a more comprehensive understanding of SD quality can be achieved.

Evaluation metrics

Many different evaluation parameters were identified during the review. Here, we have attempted to segregate them into different categories according to their underlying properties.

• Quantitative metrics: These metrics are used for comparison of quantitative parameters between the SD and RD. These can be subclassified as follows:

  • Performance metrics: These are the standard statistical metrics that are generally used to test the performance of the data generation models. Examples include Sensitivity, F1-score, Precision, AUC-ROC score, AUC-PRC score, Confusion matrix.

  • Classification metrics: These metrics are used to benchmark the classification accuracy of the model. They consist of parameters like Matthews Correlation Coefficient and Cohen’s Kappa.

  • Clustering metrics: The clustering metrics assess the quality of clusters formed from SD. Examples are Adjusted Rand Index and Normalized Mutual Information.

  • Ranking metrics: They are used to compare the rank of the data generated by the model in order of relevance. Examples are Spearman’s rho and Kendall’s tau.

  • Regression metrics: These metrics evaluate the predictive ability of a given model. Examples are Huber’s Loss and Error rate.

• Biologically informed/use case metrics: These metrics assess the difference between the biological properties of the SD and RD. Examples include: read coverage, read count, read depth, and related read metrics like reads per kilobase million (RPKM) and fragments per kilobase million (FPKM), overlap in annotation/classification of single cells, fold change in gene/isoform expression, and clustering accuracy.

• Qualitative metrics: These metrics focus on the comparison of the aspects of data generating algorithms. Examples include: Speed, Complexity of datasets, Privacy, Diversity.

Synthetic proteomics

Domain overview

Proteomics falls under the biological study of proteins and is embedded in the wider Omics field. The breakdown of proteomics can be split into: quantitative proteomics, functional proteomics, structural proteomics, protein–protein interactions, qualitative methods with mass spectrometry (MS) imaging, proximity extension assays, and aptamer based platforms amongst others [24]. The instrumentation and technical methodologies also span multiple formats, with emerging single cell proteomics making a foray [25]. Comprehensive overview of the listed methodologies and their nuanced analytical approaches are described in [26, 27] for further reading. Data generated from Quantitative Mass Spectrometry (qMS) proteomics, includes spectral peaks, abundance values, intensity signals in MS, amounts of protein in samples, and many other measurement outputs depending on sub-field and instrumentation, which are of relevance when discussing SD in this domain.

Review outcomes

We identified 129 initial articles and 68 after duplicate removals, which went through an abstract screening phase, before the full text review (n = 57), and then data extraction (n = 15). The initial searches used combined queries per database (see Supplementary Material), Pubmed n = 52, Scopus n = 35, Google Scholar n = 24 (top 20 by relevance plus second query), Web of Science (n = 16), DOAJ.org (n = 6), total 129 papers. The reviewed papers did not have SD generation as the core focus, but rather SD was used as part of wider research questions, and evaluation metrics were extracted from multiple sections to infer any measures taken to evaluate the data.

Evaluation metrics

The reported metrics span the broad themes of quantitative, biological and qualitative sub categories. An example of sub category types would be statistical measures for quantitative metrics, such as false discovery rate (FDR) during identification and quantification stages of MS spectra. For qualitative metrics, this would include expert inspections, overlays, benchmarking, whilst biological modetrics would include similarity measures against biological benchmarks, closeness to true outputs from protein/peptide measures, and/or synthetic peptide standards.

The papers included reported evaluations of the SD produced, and further details around how they assessed these data in relation to overall study goals.

Types of SD generated or used in the included papers:

1. Synthetic mass spectra, spectral data, spectra, isobaric peptides [28–34]

2. Synthetic 2-dimensional gel electrophoresis (2DGE) spots, images [35, 36]

3. Synthetic expression data, synthetic features of proteomic data for designated models (high dimensional formats, signal intensities, perturbations, cut-offs) [37–40]

4. RPPA, P100 assays, and global chromatin profiling (GCP) [38]

5. Human Protein Atlas (HPA), large-scale immunohistochemistry (IHC) [41]

6. Networks (biological with proteins) [41, 42]

The main metrics and most represented area of proteomics from the papers was from MS based proteomics research. A detailed list of quality metrics identified for proteomics can be found in the Supplementary Material.

• Quantitative metrics (statistical and performance): Signal-to-noise ratio (SNR), spot detection efficiency, sensitivity and FDR, low-abundance protein detection, peak separation (e.g. normalized separation parameter), perturbation modelling (Gaussian, Poisson), threshold-based matching (cosine similarity, m/z tolerance), AUC and ROC curves, F1 score, MSPE and mean absolute error (MAE), feature matrix noise modelling, binomial and Markov chain Monte Carlo (MCMC) - based distributions, dataset balancing, and replicate consistency.

• Qualitative metrics: Perceptual image evaluation, visual distribution comparisons (e.g. RD versus SD), manual peptide inspection, replicate similarity, network connectivity patterns, and use of labeled ground truth for validation.

• Biological and domain-specific metrics: Protein abundance ranges, peptide length distributions, isotope pattern fidelity, expression heterogeneity (SNR), reduction of isobaric interference, representation of disease-associated proteins (e.g. known subsets, module overlaps), and molecular ion similarity.

• Comparison metrics: Prediction overlap percentage, number of recovered oscillating proteins, performance comparisons between RD and SD models, accuracy of identifications (e.g. peptide-spectrum match (PSM) counts), and graphical analysis under varying noise and perturbation scenarios.

Metrics like ROC, AUC, MSPE, and F1 were the most suitable metrics for assessing SD for model based validations, whilst noise and perturbations were used to mimic real time variations and heterogeneity, the LogFC thresholds thereby maintained relevancy to downstream analytics and classification. Ground truth recovery was identified as a way to ensure robustness in noisy network based paradigms, with grid searches allowing for many simulations improving reproducibility, whereas those employing manual inspection or subjective evaluation provide expert interaction with quantitative metrics. Metrics like pooling and identifying isobaric peptide signals as well as peptide length are considered contextually relevant items related to MS instrumentation.

Of the metric types recorded, in the context of proteomic data the comparison metrics would be deemed bespoke in that the model applications, which were designed to be validated themselves were used to show comparative metrics between RD and SD, with the main goal of highlighting the efficacy of the model as opposed to the SD generation. However, those which carefully considered instrumentation parameters and key features from RD could still provide a richer view of the SD quality metrics. The metrics used for algorithmic performance should ideally be comparable between SD and RD.

Some papers focused on SD tailored to specific model evaluations such as linear or correlation models without aiming to replicate RD. These SD were often designed for targeted validation tasks rather than broad mimicry of RD. However, reporting on how well the SD captured the intended characteristics was often insufficient. In cases where SD generation methods were adopted from prior work, descriptions of the process and evaluation were frequently minimal. Although the use of existing methods is common, clearly reporting the conditions and assumptions remains important. Furthermore, the review did not apply snowballing techniques, making it difficult to retrieve details when SD generation was only referenced indirectly.

Research papers using synthetic peptides for quantification required nuanced reviewer skills, the generation of synthetic peptides has existing and established quality control metrics, however papers which used synthetic spectra from synthetic peptide quantification for model evaluation purposes and provided evaluations either on how they then used MS to quantify alongside the RD or how they assessed these data in their own right were included for those metrics.

Many of the excluded papers had limited or unclear reporting of evaluation metrics which required further inference. However there was a critical evaluation paper on the use of artificial spectra data which is already successfully used in MS peptide identifications and considering these SD for training algorithms in de novo peptide identification [43]. Further expert commentary on the ML development in proteomics with relevance to SD and SD generation in the field can be found elsewhere [44, 45].

The range of papers included in the review represent a wide scope of the types of data and contexts available to proteomics and the extracted metrics for SD quality can provide an insightful index for future standardization.

Synthetic phenomics

Domain overview

Phenomics is a relatively new discipline of Biology that has been applied in several fields, mainly in crop sciences [46, 47]. Phenomics is the systematic study of the complete set of expressed phenotypes and the structure of such a set in relation to the genetic factors and environmental perturbations that caused their expression [48]. It combines concepts and methods from various disciplines to understand the complex interactions between genotype, phenotype, and environmental factors that influence organismal development, function, and evolution [49, 50]. While traditional approaches to phenomics rely heavily on experimental measurements [51], synthetic phenomics offers a complementary strategy that harnesses the power of computation to create simulated phenotypes.

Synthetic phenomics is a subfield within the broader scope of phenomics, focusing on the creation and utilization of SD to investigate complex traits. By leveraging state-of-the-art computational models and simulation techniques, we can now generate vast quantities of SD that closely resemble phenotypes from RD. This innovative approach has the potential to significantly enhance our understanding of complex traits, such as those involved in cancer, neurological disorders, and metabolic diseases [52]. Overall, phenomics has the potential to transform our understanding of complex diseases and traits, paving the way for the development of personalized therapies and targeted interventions.

Review outcomes

From the initial large number of publications [i.e. a total of 6920 identified from the queries on PubMed (n = 2), Web of Science (n = 21), Google Scholar (n = 6890), and ELIXIR curation registry (n = 7)] only 22 were finally selected for the review after the screening process. Before screening, we applied an extra filter to reduce the untreatable number of records obtained from Google Scholar to the top 20 ranked per relevance. This resulted in 50 records to be screened. We excluded five synthetic phenomics datasets for not providing a peer-reviewed supporting publication. From the 45 retrieved reports, we excluded 3 publications for not being open access, and 20 for being false positives, i.e. they did neither generate nor use SD. We observed that 8 out of these 20 false positives were due to the ‘synthetic biology’ topic of the publication. Noteworthy, of the final 22 publications we reviewed, a significant number, the 91% (20/22), fall into the Plant Sciences domain, and the majority of publications, the 32% (7/22), were issued in 2021.

Evaluation metrics

Only three publications from the total reviewed publications, i.e. 14%, mentioned the use of evaluation metrics to assess the quality of the SD used or/and generated. The metrics identified were both quantitative and qualitative and overall dependent on the modality of the generated dataset:

1. Quantitative metrics [53]: widths and heights in pixels of real versus synthetic maize tassels images; structural similarity index computation (SSIM) to indicate the similarity between two images, real and synthetic; usability of SD by accuracy score of a classifier model.

2. Perceptual qualitative metrics [53]: human perception annotation about real or synthetic image; human rating on the similarity of a pair real-synthetic image.

3. Count distributions similarity [54]: distributions of the count table elements.

4. Distributions of information measures [54]: joint entropies and any information theoretic measure which is a function of the entropies such as multi-information Ω.

5. Statistical testing of distribution equivalence [54]: quantile-quantile plot of P-values from Epps–Singleton tests comparing synthetic versus expected distributions.

6. Realism of the simulated dataset [55]: Euclidean distance.

7. Image spatial resolution [55]: spatial resolution per pixel.

Synthetic imaging

Domain overview

Synthetic imaging is a rapidly evolving field in medical imaging, leveraging advanced computational techniques to generate realistic medical images. These synthetic images are pivotal in various applications, including training ML/AI models, enhancing diagnostic accuracy, and facilitating the development of new imaging technologies. Synthetic imaging presents numerous advantages and diverse applications in the medical field [56]. One key benefit is data augmentation, where synthetic images enrich existing datasets by providing a wider variety of samples, effectively addressing the limitations of RD. This approach also enhances privacy protection, as SD eliminates the need to use sensitive patient information, while simultaneously reducing the costs and effort associated with annotating real medical images. Furthermore, synthetic imaging serves as a valuable tool for training and educating healthcare professionals, offering a risk-free environment that safeguards patient confidentiality. Beyond education, it opens up new avenues for research, including modality translation and the development of AI-driven diagnostic tools.

Review outcomes

A comprehensive review of synthetic imaging literature from the past decade was conducted, which identified and screened a large number of publications. The initial search outcomes were: PubMed identified 13 articles, with 10 fitting the criteria after applying the last 10 years filter; SCOPUS identified 21 articles, with 18 fitting the criteria after filtering; and Google Scholar identified 307 articles, with 296 fitting the criteria after applying the last 10 years filter.

After examining 341 articles, 45 articles were finally identified as an appropriate fit for our effort. These selected articles fall under two main categories: those reviewing tools for simulating synthetic images and those suggesting guidelines for evaluation. The initial search yielded a large number of publications with only a few that were relevant, mostly due to the broad usage of the term “synthetic”, prompting the need for more precise search terms.

Evaluation metrics

Evaluating the quality and performance of synthetic images involves both quantitative and qualitative metrics.

Quantitative metrics as identified in the curated publications [57–93] include:

• Peak signal-to-noise ratio (PSNR): Measures the ratio between the maximum possible power of a signal and the power of corrupting noise, indicating image quality.

• Structural similarity index (SSIM): Assesses the similarity between two images, focusing on luminance, contrast, and structure.

• Mean squared error (MSE): Calculates the average squared differences between the generated and reference images.

• Mean absolute error (MAE): Measures the average absolute differences between the pixel values of the synthetic and real images.

• Fréchet inception distance (FID): Evaluates the distance between the distributions of real and synthetic images in a feature space, providing an overall measure of image quality.

Use case related metrics: These metrics are tailored for specific applications:

• Visual Inspection: Expert radiologists visually assess image quality, anatomical detail, and clinical utility.

• Radiologist evaluation: Radiologists use rating scales to assess image sharpness, contrast, and artifact presence.

Statistics based metrics: These metrics are derived from statistical analyses:

• Wilcoxon signed-rank test: A nonparametric test to compare paired samples, like synthetic and real images.

• Friedman test: A nonparametric test to detect differences in treatments across multiple test attempts.

• Principal Component Analysis (PCA): Used for qualitative data visualization.

• Supervised discriminators: Employed for quantitative evaluation, measuring how well synthetic images can be distinguished from real ones.

Synthetic EHRs

Domain overview

EHRs serve as an extensive, lifelong repository of patient health information, encompassing diverse data types like medical history, medications, allergies, lab results and more. These records, often dispersed across multiple platforms and locations, provide a detailed, chronological overview of a patient’s medical journey and facilitate data sharing across healthcare settings. However, the fragmented nature of EHR systems complicates access to patient records, creating challenges for clinical practitioners, researchers, and developers [94].

Review outcomes

After curating 83 articles, 18 of them were finally selected. Regarding the generation methods the main sources mainly involve types of GANs, probabilistic models, classification models, weighted bayesian association rule mining, Gaussian multivariate, bayesian networks and sequential tree synthesis. As indicated by [94] and further supported by other works [3, 95–97], different evaluation metrics are often utilized, usually based on the type of generation method that was initially implemented to produce SD, which makes the direct comparison of the results quite challenging. On the other hand, the work of [98] distinguishes the metrics used based on data type focusing on tabular data and time series.

Evaluation metrics

A plethora of metrics are utilized for the evaluation of EHR SD. It is important to clarify that this literature is primarily targeting the structured part of the EHRs, and therefore the textual form is not taken into consideration (nor any AI-drive aspects). As such, the SD in this case are tabular data and sometimes longitudinal data. The majority of the publications that were selected indicated three main categories of evaluation metrics regarding the fidelity of the SD, the utility and finally the preservation of privacy. Furthermore, many studies note the distinction of the latter category into further information disclosure related subcategories. The first subcategory includes those regarding the identity disclosure which occurs when an attacker can accurately determine that a specific individual is part of the training dataset. This risk arises when one or more SD points closely resemble a RD sample used in generating the synthetic dataset, potentially revealing its inclusion in the original confidential data. The second subcategory includes those that deal with the attribute disclosure which refers to the risk that an attacker can accurately predict the original values of sensitive attributes for an individual in the confidential dataset. This occurs when the attacker uses known variables from the RD, combined with patterns in the SD, to infer the sensitive values. The likelihood of such disclosure depends on factors like the number of known variables, the size of the synthetic dataset, and the configuration of the attack model.

The metrics that were identified after carefully curating the papers that were included in the manuscript are categorized and presented below:

• Fidelity evaluation: variable distributions, frequency of data features, dimensional probability or probability distributions, Consultation with clinical experts (qualitatively), Data similarity, Synthetic to Synthetic (STS), Real to Synthetic (RTS) and Real to Real (RTR), Pairwise Pearson correlation coefficient, most common values, Statistical similarity (mean, std, miss rate), Visualization techniques (PCA, histograms, and correlation matrices), Nearest neighbor adversarial accuracy (AA) and fidelity loss, average trends, Train an ML classifier to label data as real or synthetic, distribution of attributes, Kolmogorov–Smirnov (K–S), Welsch t-tests, Student t-test, Chi Square tests, Dimension-wise distribution similarity [Dimension-wise Probability, Dimension-wise Average, K–S test, Support Coverage, Kullback–Leibler divergence (KLD)] [99–102].

• Utility evaluation: Augment data for ML model training, Use SD in ML models, KLD, pairwise correlation difference, log-cluster, support coverage, cross-classification, Multivariate Hellinger distance, Wasserstein Distance, Cluster analysis measure, Propensity and prediction MSE, Dimension-wise distribution, Column-wise correlation, Latent cluster analysis, Clinical knowledge violation, Medical concept abundance, Feature selection [103, 104].

• Privacy evaluation: Identity disclosure and attribute disclosure, Distance to the closest record, Membership attack, Max-RTS similarity, Formulation of DP, Privacy loss, Maximum mean discrepancy, JS divergence and Wasserstein distance, Differential privacy cost, Distance to the optimal point, Attribute inference risk, Membership inference risk, Meaningful identity disclosure risk, Nearest neighbor adversarial accuracy risk [94, 101, 103–105].

• Extra metrics: ML performance metrics (Sensitivity, Specificity, Positive Predictive Value/Precision, Negative Predictive Value, AUC score, Precision–Recall AUC score), Rule similarity (for specific data type generation) [105–107].

Discussion

Our review of evaluation metrics for SD across six distinct life sciences domains (genomics, transcriptomics, proteomics, phenomics, imaging, and EHRs) highlights both the diversity and domain-specific nature of the metrics commonly used in the field. While SD is playing an increasingly important role in scientific research, evaluating its quality and applicability remains a significant challenge, particularly when comparing across these diverse domains. Although SD is tailored to specific applications within each domain, the lack of standards makes it difficult to assess and compare SD quality even within the same application domain. This challenge becomes even more pronounced with the growing interest in multimodal applications of SD, where multiple data types from different domains are artificially generated and merged. This trend highlights the opportunity to develop new evaluation metrics tailored to these complex use cases, which will require a separate review as applications begin to develop further.

In total, after the systematic screening of over 8000 initial records retrieved from multiple databases 188 publications were finally curated across all SD domains. The refinement process involved multiple layers of filtering, including duplicate removal, exclusion of preprints and nonpeer-reviewed material, non-English texts, articles older than 10 years and records that misused or ambiguously applied the term “synthetic”. A total of 156 metrics were identified (Fig. 2), of which 142 are unique to a single domain and 14 are shared across multiple domains. Notably, domains with a larger number of reported metrics (such as EHRs) do not necessarily reflect greater methodological maturity; instead, they may indicate fragmentation and the absence of widely adopted evaluation standards. This is an interesting observation that emerged from our curation effort that needs further investigation in future works. These shared metrics are illustrated in Fig. 3 and explored in greater detail in Supplementary Fig. S8.

Figure 2.

Barplot of number of SD evaluation metrics per domain. Legend: Barplot showing the number of SD evaluation metrics that were retrieved for each domain in the systematic review. The figure highlights the differences in how many metrics have been reported across genomics, transcriptomics, proteomics, phenomics, imaging, and EHRs. Alt text: A bar chart with six bars, each representing a life sciences domain, showing how many SD evaluation metrics were found in each domain.

Figure 3.

Bipartite graph of shared evaluation metrics across domains. Legend: Bipartite graph displaying evaluation metrics (nodes on the left) that are shared across multiple domains (nodes on the right). This visualization shows the connections between metrics and the domains where they are applied, revealing overlap among life sciences areas in SD evaluation practices. Note: Recall and sensitivity are equivalent metrics and are both mentioned due to their common usage in the literature. AUC denotes the area under the curve, SNR the signal-to-noise ratio, EHRs electronic health records and ROC the receiver operating characteristic curve. Alt text: A bipartite network diagram showing which evaluation metrics are shared between different life sciences domains.

In the domain of genomics the review found that the term “synthetic” has often a different meaning, with many studies referring to wet-lab synthesized DNA rather than computationally generated sequences. This issue underscores the need for clearer terminology and domain-specific criteria for evaluating SD in genomics. Regarding the curation effort, the outlined metrics can be put into three categories: bioinformatics-related metrics, use case-related metrics and finally statistics-based metrics. Similarly, in transcriptomics, SD plays a crucial role in gene expression analysis and ML/AI applications. The primary focus for the application of SD in this domain include data augmentation and balancing underrepresented classes to increase statistical power for the downstream analyses. Extrinsic metrics assess model performance when trained on SD versus RD, using measures like precision, recall, F1-score and AUC-ROC, which are crucial for real-world applicability. Intrinsic metrics, on the other hand, evaluate the inherent quality of SD, focusing on distribution similarity, perplexity and statistical comparisons. While extrinsic metrics ensure that SD is suitable for practical applications, intrinsic metrics provide fundamental insights into data fidelity, consistency and variability. By combining both approaches, a more comprehensive and robust SD assessment can be achieved, contributing to more reliable benchmarking frameworks.

In proteomics research, SD is frequently used, yet its evaluation often lacks standardized metrics. While many reviewed studies describe SD generation methods, few detail how the data is assessed. However, emerging metrics, such as ROC, AUC, peak accuracy, sensitivity, specificity, and image fidelity, suggest the potential for developing best practices across subfields like 2DGE and MS. These metrics are highly context-dependent, varying with the methodology employed. Clear ontologies and evaluation frameworks are needed to guide SD use across proteomics workflows, which involve multiple stages from sample preparation to data interpretation. For quantitative MS, this includes considering different acquisition methods (e.g. DDA, DIA), labelling strategies, and spectral features like noise and fragmentation. Notably, this review excluded synthetic peptide production, which follows separate standards. Ultimately, SD in proteomics aims to support, not replace, RD, enabling more efficient hypothesis testing and model development ahead of experimental validation.

In phenomics, the reviewing process found that while SD offers a promising approach for studying genotype-phenotype interactions, the primary focus remains on data generation rather than systematic assessment, indicating a gap that needs to be addressed to ensure the reliability of SD for this domain. Furthermore, the dominance of plant-based studies suggests a potential bias in SD research, requiring further exploration in human-related phenomics applications. The metrics here can be divided into various categories, including qualitative, perceptual qualitative, distribution similarity, information measure, statistical testing of distribution equivalence, realism of SD and image spatial resolution.

Medical Imaging is one of the more mature fields in terms of SD applications, as synthetic medical images are widely used for training AI models and augmenting RD. However, despite the significant number of studies reviewed, most publications focused on the development of synthetic imaging tools rather than comprehensive evaluations of their reliability. Metrics like structural similarity and image fidelity were commonly reported, but there was a lack of consistency in benchmarking methodologies. The metrics that were identified as relevant in this domain can be split into quantitative, use case-related and statistics-based metrics.

Lastly, in the domain of EHRs, SD is used to generate privacy-preserving patient records for research and AI model training. The review found a wide variety of generation methods, including GANs, probabilistic models and Bayesian networks, each employing different evaluation metrics. The lack of uniformity in assessment strategies makes it difficult to establish clear guidelines for evaluating synthetic EHR data, further emphasizing the need for a standardized approach. At this point it should be noted that both Medical Imaging and EHR SD applications are particularly relevant in a clinical context, and the respective need for more rigorous and standardized evaluation ways to assess the clinical utility of the SD in these fields. Importantly, clinical utility is intrinsically use-case dependent. What constitutes “useful” SD varies according to whether the downstream task involves risk prediction, association testing, subgroup comparison or model development. Evaluating utility therefore requires examining whether analyses performed on SD lead to the same clinical conclusions as those produced by RD. Regarding synthetic EHRs, the work of [108] provides a clear example of this by comparing confidence-interval overlap in estimates across real and synthetic datasets, demonstrating how closely clinical inference aligns when predicting diabetes onset. Similarly, the work of [109] extends utility assessment beyond statistics by incorporating clinical judgment. Clinicians evaluated whether learned relationships were medically plausible and a blinded experiment asked them to distinguish real from synthetic patient records. Their performance offers a practical benchmark of clinical credibility and consistency of analytical outcomes, highlighting the importance of evolving domain experts in these evaluations.

A key challenge across domains is the inconsistency in evaluation criteria, which often depends on the SD generation method. Some studies emphasize statistical similarity to RD, while others focus on downstream utility, like improving ML/AI performance. Additional approaches assess data integrity, reproducibility, or real-world applicability. This diversity highlights the need for domain-specific yet practical evaluation guidelines that ensure consistent, rigorous, and meaningful assessments tailored to each domain’s unique needs.

Furthermore, as members of ELIXIR Europe [110], we advocate for interoperability in SD evaluation metrics that align with existing standards and related initiatives. This approach is consistent with the software best practices defined under the ELIXIR-STEERS project [111] and the software quality indicators developed within the European Open Science Cloud (EOSC) ecosystem [112]. For example, the EVERSE project [113] aims to establish a framework for research software excellence by integrating community curation, quality assessment, and best practices, which will be encapsulated in the Research Software Quality toolkit (RSQkit) as a comprehensive knowledge base for high-quality software development across disciplines. The interoperability of SD evaluation metrics could be achieved by developing dedicated ontologies and semantic strategies to represent the evaluation metrics to allow their integration in SD generation benchmarking platforms, such those currently developed in European projects like SYNTHIA [114] and SYNTHEMA [115].

Despite advancements in medical AI research, the clinical adoption of emerging AI solutions remains challenging due to limited trust and ethical concerns. Several recent initiatives are contributing to the development of robust best practices and guidelines for the evaluation of AI, such as FUTURE-AI consortium [116] and the AHEAD project [117], addressing ethical concerns, privacy implications, fairness, transparency, and the broader societal impact.

We believe that by integrating both technical and societal evaluation approaches, more robust and comprehensive recommendations for assessing SD can be established. Such an integrated approach would not only promote responsible AI innovation but also ensure that the deployment of SD across various domains is both reliable and aligned with broader societal values.

Finally, this foundational mapping also lays the groundwork for implementing the metrics collected in this work in a FAIR and actionable evaluation framework for the ELIXIR community and beyond, accompanied by practical guidelines for assessing SD across biomedical domains.

Conclusions

This ELIXIR scoping review highlights the urgent need for standardized evaluation metrics to assess the quality, reliability, and applicability of SD across scientific domains. The review found that most current research emphasizes SD generation rather than its evaluation, creating an imbalance that may hinder the credibility and adoption of SD in scientific and clinical settings. To address this, future work should focus on developing robust, shared evaluation frameworks to ensure consistency, comparability, and trust in SD use. Establishing clear assessment guidelines will support better validation of SD methods and promote responsible integration into research and healthcare. The review’s findings lay the groundwork for initiatives aimed at improving SD evaluation practices, ensuring synthetic datasets are effectively and ethically applied across disciplines.

Supplementary data

Supplementary data is available at NAR Genomics & Bioinformatics online.

Conflict of interest

None declared.

Funding

This work was supported by ELIXIR, the research infrastructure for life-science data. The Centre for Research & Technology Hellas (CERTH), Barcelona Supercomputing Center (BSC), and Leiden University Medical Center (LUMC) receive support from SYNTHIA. SYNTHIA (Synthetic Data Generation framework for integrated validation of use cases and Al healthcare applications) is supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under grant agreement no. 101172872. The JU receives support from the European Union’s Horizon Europe research and innovation programme, COCIR, EFPIA, Europa Bío, MedTech Europe, Vaccines Europe, and DNV. The UK consortium partner, The National Institute for Health and Care Excellence (NICE) is supported by UKRI Grant 10132181. Funded by the European Union, the private members, and those contributing partners of the IHI JU. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the aforementioned parties. Neither of the aforementioned parties can be held responsible for them.

Data availability

No new data were generated or analysed in support of this research.