A benchmark of RNA-seq data normalization methods for transcriptome mapping on human genome-scale metabolic networks

Abstract

Genome-scale metabolic models (GEMs) cover the entire list of metabolic genes in an organism and associated reactions, in a tissue/condition non-specific manner. RNA-seq provides crucial information to make the GEMs condition-specific. Integrative Metabolic Analysis Tool (iMAT) and Integrative Network Inference for Tissues (INIT) are the two most popular algorithms to create condition-specific GEMs from human transcriptome data. The normalization method of choice for raw RNA-seq count data affects the model content produced by these algorithms and their predictive accuracy. However, a benchmark of the RNA-seq normalization methods on the performance of iMAT and INIT algorithms is missing in the literature. Another important phenomenon is covariates such as age and gender in a dataset, and they can affect the predictivity of analysis. In this study, we aimed to compare five different RNA-seq data normalization methods (TPM, FPKM, TMM, GeTMM, and RLE) and covariate adjusted versions of the normalized data by mapping them on a human GEM using the iMAT and INIT algorithms to generate personalized metabolic models. We used RNA-seq data for Alzheimer’s disease (AD) and lung adenocarcinoma (LUAD) patients. The results demonstrated that RNA-seq data normalized by the RLE, TMM, or GeTMM methods enabled the production of condition-specific metabolic models with considerably low variability in terms of the number of active reactions compared to the within-sample normalization methods (FPKM, TPM). Using these models, we could more accurately capture the disease-associated genes (average accuracy of ~0.80 for AD and ~0.67 for LUAD) for the RLE, TMM, and GeTMM normalization methods. An increase in the accuracies was observed for all the methods when covariate adjustment was applied. We found a similar accuracy trend when we compared the metabolites of perturbed reactions to metabolome data for AD. Together, our benchmark study shows that the between-sample RNA-seq normalization methods reduce false positive predictions at the expense of missing some true positive genes when mapped on GEMs.

Introduction

The development and widespread use of next-generation sequencing (NGS) technology induced a shift in transcriptome profiling from microarray technology to RNA-seq in the last decade. Count data generated by NGS has technical biases like gene length, library size, and sequencing run differences, which must be corrected by a normalization method before performing further analysis^1,2. Many normalization methods are available in the literature such as transcripts per million (TPM), fragments per kilobase of transcript per million fragments mapped (FPKM), trimmed mean of M values (TMM) introduced through the edgeR package in R, gene length corrected trimmed mean of M-values (GeTMM), and the Relative Log Expression (RLE) provided by the DESeq2 package in R. TMM is based on the hypothesis that most genes are not differentially expressed. It is the sum of rescaled gene counts³. DESeq2, another Bioconductor package, uses RLE as a normalization method^4–6 with a similar assumption to TMM³. A correction factor is calculated as the median of the ratios of all genes in a sample. TMM uses a correction factor applied to the library size while the correction factor is applied to the read counts of the individual genes for RLE^2,3,7. GeTMM is the newer version of TMM combining gene-length correction with the normalization procedure². In FPKM, the sum of all mapped reads in a sample is scaled to a million followed by normalizing each gene by its gene length. FPKM and TPM differ only in the order of normalization operations⁸. These normalization methods are divided into two major groups. FPKM and TPM are the most popular methods among the within-sample normalization methods, while TMM and RLE are the most popular methods among the between-sample normalization methods. Additionally, GeTMM has been recently suggested to reconcile within-sample and between-sample normalization approaches². The normalization method of choice can have a significant effect on the results in downstream analyses. Many benchmarking studies are available in the literature concerning the effects of RNA-seq normalization methods on differential gene expression analysis^1,9,10.

Establishing the known biochemical reactions in a cell or organism with their corresponding genes and enzymes as a mathematical platform is called genome-scale metabolic model (GEM) reconstruction^11,12. GEMs gained increased popularity in the last decade since they enable omics data integration for metabolic phenotype simulation^13,14. Different algorithms are available to reconstruct context-specific GEMs based on the expression levels of genes¹⁵. Benchmarking studies have been carried out to compare the power of these algorithms to capture metabolic changes^15–19. iMAT (Integrative Metabolic Analysis Tool) is one of the most popular context-specific GEM reconstruction algorithms, especially to map human transcriptome on GEMs. The algorithm aims to remove reactions controlled by lowly expressed genes based on an optimization framework, assuming that they will be inactive²⁰. Since iMAT has significant advantages such as not requiring the mathematical definition of a biological objective and precise physiological constraints as inputs, it is widely used for human diseases^21–24 to create context-specific GEMs for mammalian cells¹⁵. Since there is no consensus in the literature on defining a biological objective function for human metabolism²⁵, mapping algorithms that do not require an objective function are commonly preferred for the reconstruction of metabolic models for human diseases. The Integrative Network Inference for Tissues algorithm (INIT)²⁶ is another mapping method for building condition-specific metabolic models. It is similar to iMAT in that it does not require an objective function.

System-level analysis of complex metabolic dysfunctions in human diseases, especially cancer and neurodegenerative diseases, by integrating genome-scale metabolic models and omics data is a current and common research field^27–29. In such studies, condition-specific GEMs are reconstructed for both control and disease states to compare metabolic differences between the two states. Deciding the normalization method of RNA-seq data before applying a context-specific GEM reconstruction algorithm is a challenging and undervalued issue. Although there are many benchmarking studies comparing normalization methods and context-specific GEM reconstruction algorithms separately, a study investigating the effects of normalization methods on context-specific GEM reconstruction algorithms is not available yet.

Through improvements in healthcare services and quality of life, the world population is aging and there is an increase in related diseases. Alzheimer’s disease (AD), the most common neurodegenerative disorder in the world, is also becoming more common with the aging population³⁰. Lung cancer is another disease having an increased risk in older populations³¹, and it has the highest mortality rate of all major cancers³². Lung adenocarcinoma (LUAD) is the most common type of lung cancer. The effect of age and gender can be prominent in various diseases, especially neurodegenerative disorders and cancer. For instance, the symptoms for AD develop after 60 years of age, and the disease is more prevalent in women³³. Lung cancer progression is also associated with age³⁴ and gender³⁵. The effect of these covariates (e.g., age and gender) can be estimated and removed from the data. In recent years, covariate adjustment has commonly been applied before the statistical analysis of AD^36–38 and lung cancer³⁹ RNA-seq datasets.

In this study, the effects of five RNA-seq normalization methods (RLE, TMM, GeTMM, TPM and FPKM) and covariate adjustment on reconstructing context-specific personalized GEMs were systematically investigated using two most popular mapping algorithms for human metabolism, iMAT and INIT (Supplementary Table 1). The algorithms were applied to popular gene expression datasets coming from big cohorts of prevalent diseases, which are AD and LUAD.

Results

Between-sample and within-sample normalization methods are clustered together based on the similarity of the personalized metabolic models

Five different RNA-seq normalization methods and their covariate-adjusted versions were compared by generating personalized iMAT models (Fig. 1) for two different diseases: AD and LUAD. Statistical tests were applied on the generated personalized metabolic models and significantly affected reactions/pathways, and genes associated with the affected reactions were compared and investigated to evaluate the normalization methods.

Fig. 1. Workflow for the study.

a We used FPKM, TPM, TMM, RLE, and GeTMM methods for the normalization of two datasets: ROSMAP for AD, and TCGA for LUAD. We also used covariate adjustment for both datasets. b iMAT and INIT were used for the integration of transcriptome data into the metabolic model. Generated personalized metabolic models were binarized and Fisher’s exact test was performed to determine significantly affected reactions/pathways. c The similarity of normalization methods was evaluated using the Jaccard similarity index.

It is known that datasets obtained from complex organisms such as humans may have different disease-related covariates (e.g., age and gender) affecting the structure of the data. Thus, age and gender are considered as covariates for both diseases. Since brain tissue samples are post-mortem, Post-mortem interval (PMI) affecting RNA degradation was also considered as a covariate for the AD dataset. As a result, ten different inputs (datasets normalized with five different methods and their covariate adjusted versions) were used to generate personalized iMAT models for both datasets. Figure 1 shows the workflow of the study.

iMAT models were generated with both covariate-adjusted and non-adjusted normalized expression values in a personalized way, i.e., a separate model for each sample in the ROSMAP and TCGA data. Then, we compared (i) the number of reactions in the generated personalized models (ii) the number of significantly affected reactions, and (iii) their pathway associations across all the normalization methods and their covariate versions (Fig. 2).

Fig. 2. Comparison of the metabolic content of the reconstructed GEMs by iMAT in the two datasets.

a Number of active reactions in the personalized control and AD models reconstructed from the data normalized with RLE, TMM, FPKM, GeTMM and TPM. b Number of significantly affected reactions based on Fisher’s exact test (p-value < 0.05) and significantly enriched metabolic pathways based on the hypergeometric test (p-value < 0.05). Number of pathways and reactions are reported in log2 scale. c Heatmap plots of the Jaccard index similarity scores of active reactions, d significantly affected reactions and, e enriched metabolic pathways respectively in GEMs reconstructed using the AD dataset generated by each transcriptome data normalization strategy. f–j All steps were repeated for the LUAD dataset. Brain-2 icon and lung-cancer icon by Servier https://smart.servier.com/ are licensed under CC-BY 3.0 Unported https://creativecommons.org/licenses/by/3.0/.

We evaluated whether there were differences in the metabolic content of the reconstructed personalized GEMs in terms of the normalization methods. We first plotted the range of number of reactions in each sample determined to be active in the control and disease cases (Fig. 2a). In the ROSMAP data, both control and AD models have a high variability across the samples in terms of the active reactions for TPM and FPKM, the within-sample normalization methods. Specifically, covariate-adjustment decreased the variability in TPM and FPKM to a certain extent, making the size of personalized models more similar to each other. On the other hand, the number of active reactions in GEMs reconstructed with the TMM, RLE, and GeTMM normalized data show low variability. Both covariate-non-adjusted and covariate-adjusted versions showed the same trend of high-variability in the within-sample normalization methods and low-variability in the between-sample normalization methods (Fig. 2a, f). We further tested whether this observation was reflected in the set of significantly affected reactions identified from the personalized GEMs generated by each normalization method (Fig. 2b, g). RLE, TMM, and GeTMM have comparatively similar number of significantly affected metabolic reactions whereas TPM and FPKM are close to each other. Both TPM and FPKM identified the highest number of affected reactions and associated pathways in both ROSMAP and LUAD datasets. On the other hand, TMM, GeTMM and RLE approaches identified similar number of affected reactions. When compared to TPM and FPKM, the between-sample normalization methods are more consistent with one another based on the analyses with two independent datasets. Moreover, covariate adjustment can affect the size of personalized binary models, and the set of significantly affected reactions and related pathways (Fig. 2a, b, f–g). Significantly affected reactions and overrepresented pathways are listed in Supplementary Data 1 for ROSMAP, and Supplementary Data 2 for LUAD for each normalization method.

Two normalization methods can lead to similar-size metabolic models or similar number of differential reactions/pathways, but these models/reactions/pathways can be very different from each other. Therefore, we further investigated the similarity of the normalization methods using Jaccard index and visualized the results as heatmaps in Fig. 2c–e, h–j. Similarity among the normalization methods was independently evaluated based on (i) the generated personalized metabolic models (Fig. 2c), (ii) significantly affected reactions between control and disease groups (Fig. 2d), and (iii) overrepresented pathways (Fig. 2e) for AD and LUAD (Fig. 2h–j). RLE, TMM, and GeTMM and their covariate versions have a high similarity in terms of the generated binary metabolic models and are clustered together. We observed the same behavior for FPKM and TPM, the within-sample normalization methods, but with a lower similarity than the other methods (Fig. 2c, h). Although the three between-sample methods are clustered together, the similarity index between RLE and TMM is higher compared to GeTMM. Similar groupings were observed in the significantly affected reactions-based (Fig. 2d, i) and overrepresented pathways-based (Fig. 2e, j) similarity heatmaps for both datasets. Interestingly, the covariate-adjusted version of a method did not directly cluster with the non-adjusted version in the similarity maps (Fig. 2c–e, h–j). Reactions and associated pathways in personalized models were demonstrated to be clearly affected from the choice of normalization methods. To support our results with another transcriptome mapping algorithm, INIT was applied to the ROSMAP dataset for five normalization methods. Although the number of active reactions in the personalized metabolic models reconstructed by INIT is low compared to the iMAT-based models, the clustering behavior of between-sample and within-sample normalization methods was found to be similar (Supplementary Fig. 1).

Metabolites included in the significantly affected reactions were also investigated as a supportive analysis to evaluate the effect of normalization methods. Metabolome data is available for 236 of 566 individuals (70 control, 166 AD) in the ROSMAP cohort. Differential metabolites obtained from the analyses of the metabolome dataset were compared to the metabolites in the significantly affected reactions for each normalization method and their covariate versions. Within-sample normalization methods have low accuracy based on the metabolite-based predictions for both mapping algorithms (Supplementary Fig. 3).

Genes associated with differential reactions show a variable degree of overlap with disease-associated genes across normalization methods

We further assessed the five normalization methods and their covariate-adjusted versions by comparing the list of genes in the significantly affected reactions and the list of genes known to be associated with AD (Supplementary Table 2) or LUAD phenotypes (Supplementary Table 3). Accordingly, the genes of differential reactions were first determined, and normalization methods were evaluated based on these gene sets. Similar to the reaction-based analysis, the number of genes in the differential reactions was found to vary significantly according to the normalization method of choice (Fig. 3b). The numbers of these reactions ranged from 393 to 3083 for the iMAT model-based analysis of the normalized AD data while the number of the associated genes varied between 256 and 1327. For the LUAD data, the number of significantly affected reactions was found to be between 1326 and 3115 while the number of their genes ranged from 780 to 1245. For the AD case, covariate adjustment was shown to reduce the difference between the within-sample and between-sample normalization methods to a certain level in terms of the number of differential reactions and their genes. In this regard, removing the effects of covariates led to a significant reduction in the number of model-predicted differential reactions (ranged from 262 to 703 reactions) and the associated genes (ranged from 186 to 458 genes) for AD, as illustrated in Fig. 3b, whereas such a high effect was not determined with covariate adjustment for the LUAD case (ranged from 2216 to 2831 reactions and 801 to 1101 genes, not shown). Collectively, the use of within-sample normalization methods resulted in a higher number of differential reactions and genes for both AD (Fig. 3b) and LUAD (Fig. 3c) datasets.

Fig. 3. Gene-based analyses for different normalization methods.

a These analyses include the identification of genes in the significantly affected reactions using gene-protein-reaction (GPR) associations in the Human-GEM model, the elucidation of enriched disease terms, similarity analysis, and the evaluation of the normalization methods based on the accuracy of model-predicted disease genes. Radar plots demonstrate the numbers of differential reactions and their associated genes for b AD and c LUAD. Reaction and gene numbers are displayed by blue and orange lines on the charts. d The enriched key disease terms are shown in the y-axis of the bubble plot for each normalization method, and the dot sizes represent -log₁₀(FDR) values. The heatmap plots were generated based on the Jaccard index similarity of disease-associated genes captured by each normalization method for AD (e) and LUAD (f), respectively. The legends on the right show the similarity scores and the increasing darkness of the colors on the heatmap represents a higher similarity of the model-predicted disease genes between the normalization methods.

The predicted genes in the differential reactions of iMAT models were first characterized through disease enrichment analysis (Fig. 3a; also see Fig. 3d). For the AD case, “Alzheimer’s Disease” term was significantly enriched (FDR at 0.05 level) for all methods apart from the TMM normalization and the covariate-adjusted version of GeTMM normalization. Moreover, the use of GeTMM and TPM normalizations or the covariate-adjusted versions of RLE, FPKM, and TPM enabled the elucidation of several additional AD-related terms including “Alzheimer’s disease, early onset”, “Alzheimer’s disease, focal onset”, “Alzheimer’s disease, late onset”, and “familial Alzheimer’s disease”. We uncovered a considerably higher number of AD-related terms with the covariate-adjusted version of TMM through the analysis of the INIT models even if the use of this dataset with the iMAT algorithm only allowed the significant enrichment of “Alzheimer’s Disease” (Supplementary Fig. 4a). Consistently, we found over twice as many significantly affected reactions and genes in the models reconstructed by the INIT algorithm in comparison with the iMAT algorithm. The analysis of regulated genes in the LUAD models revealed the enrichment of “carcinoma of lung” for all normalization methods. More specifically, the over-representation of “non-small cell lung carcinoma” was identified for RLE, TMM, and GeTMM in addition to the covariate-adjusted version of the GeTMM normalization. Importantly, “adenocarcinoma of lung” was only over-represented for the GeTMM normalization. “Small cell carcinoma of lung”, “lung neoplasms”, “primary malignant neoplasm of lung”, and “malignant neoplasm of lung” were shown among the other lung cancer-associated terms that were significantly enriched for at least one normalization method (Fig. 3d).

In the second step, binary matrices were created for each normalization method according to the regulation (presence or absence) of the experimentally confirmed AD- or LUAD-associated metabolic genes in the reconstructed models. We compared these matrices using the Jaccard index in Fig. 3e, f by estimating the similarity between the normalization methods based on the predicted gene profiles. In this way, the pairwise comparisons of the normalization methods through the regulated genes of the AD- and LUAD-associated iMAT models revealed the clustering of within-sample normalization methods (FPKM and TPM) in agreement with the previous similarity heatmaps (Fig. 2c–f) obtained for the significantly affected reactions and over-represented pathways. For the AD case, RLE and GeTMM were grouped together, whereas TMM exhibited a diverse profile from these methods (Fig. 3e) by differing from the reaction-based results in Fig. 2c where all three between-sample normalization methods grouped together. On the other hand, the analysis of the covariate-adjusted datasets by reconstructing the AD-specific INIT models demonstrated the separate clustering of between- and within-sample normalization methods (Supplementary Fig. 4b). Similarly, the gene-based similarity analysis with the LUAD dataset provided consistent results with the aforementioned similarity profiles of significantly affected reactions and the corresponding pathways. For this dataset, RLE and TMM were found to be closely related, and GeTMM was also clustered with these between-sample normalization methods. In contrast, the within-sample normalization methods were grouped together by exhibiting diverse model-predicted disease gene profiles from the other three methods (Fig. 3f).

For the AD dataset, FPKM and TPM normalization methods allowed the prediction of more AD-related experimentally confirmed genes (Supplementary Tables 4, 5). Especially the TPM normalization predicted most of the genes obtained via the RLE and GeTMM normalizations regardless of the covariate adjustment (Fig. 4a). Even if the number of AD-related genes (true positives) was found to be higher for the within-sample normalization methods, incorrect predictions also increased with these approaches. Given that accuracy is among the most widely adopted metrics relying on the correct classification of both positive and negative cases, we assessed the predictive performances of the normalization methods based on the accuracy scores of the iMAT model-based predictions. For the AD dataset, RLE, GeTMM, and TMM normalizations had an average of almost 0.80 accuracy, and they exhibited superior predictive performance than the remaining methods, displaying the best compromise between true positives and true negatives. It should be noted that covariate adjustment considerably improved the accuracy scores for all normalization methods, particularly FPKM (almost 0.55 to 0.77) and TPM (almost 0.43 to 0.75) (Fig. 4b). Consistent results were obtained through the INIT model-based predictions (Supplementary Fig. 4c, d). On the contrary, the number of true positives was found to be close to each other regardless of the selected normalization methods for the LUAD dataset due to the presence of fewer disease-associated metabolic genes (Supplementary Table 6) in the validation set (Fig. 4c). Nevertheless, RLE (~0.68), GeTMM (~0.65), and TMM (~0.67) were shown to outperform in terms of accuracy values, compared to FPKM (~0.54) and TPM (~0.50) (Fig. 4d). Applying covariate adjustment did not considerably influence the accuracy values computed for this dataset. Together, the between-sample normalization methods were found to be among the top three methods for all data groups based on the accuracy metric, suggesting their robustness across different datasets.

Fig. 4. Evaluating the accuracy and similarity of model-predicted disease genes for the AD and LUAD datasets and their covariate-adjusted versions (AD (cov) or LUAD (cov)).

a The pairwise comparisons of true positive predictions obtained by the overlap between the model-predicted genes in the differential reactions and the experimentally confirmed AD genes. The increasing darkness of the blue color on the plots represents a higher number of the model-predicted AD genes while the dark orange colors show the lower number of the true positives. b The performances of the normalization methods were compared by computing accuracy values representing the ratio of all correct predictions (true positives and true negatives). An increasing gradient toward dark blue on the plot shows higher accuracy scores. c, d The same comparisons are also illustrated for the LUAD dataset.

Discussion

Standardizing the scale and range of RNA-seq data across different samples or conditions by normalization methods is crucial¹³. Although a considerable number of disease metabolism-related studies mapping RNA-seq data on GEMs are available in the literature, none of them emphasizes the importance of normalization methods for RNA-seq data and each of them used a different normalization method^40–42. To investigate the effects of the methods used for RNA-seq data normalization on the generation of condition-specific metabolic models, the iMAT and INIT algorithms, the two most popular mapping algorithms for human metabolism^21–24, was applied on the transcriptome data normalized by five different widely used normalization methods (TPM, FPKM, RLE, TMM, and GeTMM). AD and LUAD, prevalent in the aged populations and commonly studied disease types with GEMs, were selected as case studies. All methods were applied separately for the ROSMAP dataset containing AD samples and the LUAD dataset from TCGA (Fig. 1). The differences/similarities between the normalization methods were benchmarked based on the (i) personalized GEMs reconstructed by iMAT and INIT, (ii) significantly affected reactions between those GEMs for control and disease states and overrepresented pathways related to these reactions, (iii) the genes related to affected reactions and their disease enrichment results. The similarity heatmaps and number-based results for five normalization methods showed a highly similar pattern for AD and LUAD.

Metabolic models obtained by RNA-seq data normalized with between-sample methods are more comparable in terms of the number of active reactions. Samples were taken from the same tissue of different individuals in the transcriptome datasets used in our study. GEMs specific to a certain tissue should be similar although they are reconstructed from individual transcriptome data^28,43. The number of significantly affected reactions and overrepresented pathways also show a similar phenomenon with the number of active reactions in the normalization methods comparison.

The similarity between GEMs reconstructed with different methods or different data is a commonly used criterion in benchmarking studies related to GEMs^16,17,44. Although FPKM and TPM are clustered together in terms of the generated personalized metabolic models, the similarity index between FPKM and TPM is lower than the similarity indices among the between-sample normalization methods. After covariate-adjustment, the reconstructed models produced by the between-sample normalization methods were more consistent. GeTMM shows slightly different clustering behavior compared to between-sample methods. This is probably because GeTMM uses within-sample normalization as well as between-sample normalization unlike the other two between-sample normalization methods (Fig. 2c, f). The analogue similarity patterns were also observed in the significantly affected reaction-based (Fig. 2d) and overrepresented pathway-based (Fig. 2e) similarity heatmaps. The similarities based on overrepresented pathways are stronger than those based on significantly affected reactions, which shows that different reactions involved in the same pathway were identified to be affected with the normalization methods.

The clustering behavior of the regulated disease-associated gene profiles is consistent with the similarity analysis results of the metabolic content of the reconstructed GEMs for ROSMAP. However, covariate-non-adjusted version of TMM contradictorily expressed a separated clustering profile while RLE and GeTMM were clustered together in the AD-related genes similarity heatmap. TMM also gave inconsistent results in disease enrichment analysis for AD by being the only method that could not capture the “Alzheimer’s Disease” term as significantly enriched. In addition, the gene-based analysis demonstrated that GeTMM could predict more specific disease terms (adenocarcinoma of lung) for LUAD even if it predicted a lower number of regulated genes than within-sample normalization methods. Even though the number of significantly affected reactions, pathways, and regulated genes obtained by covariate-adjusted versions of the normalized data is lower compared to their covariate-non-adjusted versions, the accuracy scores based on AD-related genes and AD metabolome data analysis are higher. Furthermore, Choi et al. ⁴⁵ examined the effect of covariate adjustment on gene expression levels of RNA-seq data and concluded that removing the confounder effects reduces the false positive rate⁴⁵. However, applying covariate adjustment to the LUAD dataset does not have a noticeable effect on the accuracy scores based on the LUAD-related genes.

Reconstructed GEMs created with the iMAT algorithm were compared using number-based and similarity-based analyses depending on affected reactions, pathways, and disease-associated genes. Whether the number of significantly affected reactions/pathways is high or low is not a criterion for interpreting GEMs reconstructed from transcriptome data normalized with different methods. The most important hallmark for the evaluation of the number-based results is the number of true positives and/or false positives. A higher number of predicted reactions or pathways has also a higher risk of having a high number of false positives. This phenomenon was also confirmed by the gene-based analyses. Even if FPKM and TPM methods allowed the prediction of a larger number of disease-related genes, their accuracy values were found to be lower in comparison with RLE, TMM, and GeTMM. This highlights the significance of the compromise between true positive and true negative predictions, and it is crucial in the medical field for the elucidation of drug targets or biomarkers. On the other hand, it should be noted that we had a limited validation gene set, especially for LUAD. Similarly, a lack of knowledge about truly affected reactions and pathways in the disease processes may affect the proper assessment of the model-predicted reactions and pathways. Thus, it is challenging to comment on which methods should be preferred in the model-based analyses. Therefore, further studies about diverse disease phenotypes may provide more insight into the predictive performances of the normalization methods and confirm our results. On the other hand, when the number of disease-associated genes controlling the differential reactions were used as a criterion, TMM showed consistently high performance for AD and LUAD datasets, and FPKM and TPM showed consistently low performance. Metabolite-based predictions (Supplementary Fig. 3) also showed that FPKM and TPM had low accuracy compared to the between-sample normalization methods. Moreover, GEMs reconstructed by data normalized with between-sample normalization methods were found to be more consistent and comparable in terms of active reaction number, affected reactions, and pathways. Zhao et al. ⁸ compared within-sample (TPM, FPKM), and between-sample (RLE and TMM) normalization methods based on clustering analyses between replicate samples of patient-derived tumor models. They similarly concluded that RLE and TMM performed in general better than TPM and FPKM⁸. In another study, it was shown that TMM gave the best result for the subnetwork discovery algorithms⁴⁶. Corchete et al. ⁴⁷ and Maza et al. ⁴⁸ also found that TMM and RLE led to high performance compared to FPKM or/and TPM based on the analysis of differentially expressed genes (DEGs).

The INIT-based results were compared with the iMAT-based personalized models. While the numbers of active reactions in the reconstructed personalized metabolic models by INIT are low, the numbers of significantly affected reactions are high compared to the iMAT-based models. Both iMAT and INIT-based models have a high variability in terms of the number of active reactions in personalized models for the within-sample normalization methods while the variability is considerably small in the between-sample normalization methods. Moreover, the between-sample normalization methods were clustered together in the INIT results similar to the iMAT results. While the GEMs reconstructed by iMAT and INIT are completely different in terms of binary active reaction matrix similarity, general inference about the effect of RNA-seq normalization methods on metabolic models is the same (Supplementary Fig. 2). This was also supported by the gene-based analyses. Similarity and accuracy calculations predominantly confirmed the results obtained by the analyses of iMAT models.

Transcriptome data of diseases are commonly used to compare control and disease states using genome-scale metabolic modeling approach, to decipher molecular and metabolic changes associated with the disease. In this perspective, GEMs reconstructed for control and disease states should be consistent and comparable. In conclusion, between-sample normalization methods are more consistent than within-sample normalization methods in condition-specific personalized GEM reconstruction by iMAT and INIT in terms of the range of the number of reactions and genes, and clustering behaviors of normalization methods. Moreover, disease-related gene-based analyses demonstrated in Figs. 3–4, and a supportive analysis based on the comparison of ROSMAP metabolome data analysis and metabolites included in the significantly affected reactions (Supplementary Fig. 3) show that between-sample normalization methods are preferable to within-sample normalization methods. Although FPKM and TPM are within-sample normalization methods and theoretically not suitable for sample-based transcriptome mapping to GEMs if GEMs of different samples will be compared, our results show that they can still detect several disease-associated reactions/pathways/genes. Consequently, the usage of these methods for GEM based analyses cannot be completely precluded, but it should be approached with caution. Within-sample RNA-seq data normalization methods can be preferred to capture a broader list of biologically relevant perturbed pathways at the expense of more false positives while between-sample normalization should be the method of choice to obtain a more reliable list of perturbed pathways at the expense of fewer true positives, especially when the research focus is to test these pathways as drug targets or biomarkers.

Our study has the following limitations. Our benchmarking focused on the generated personalized models by iMAT or INIT algorithms and does not include any comparison that involves flux prediction. Since flux prediction requires other challenges such as definition of a biological objective function, we preferred to not to add another parameter into our study. Benchmarking of RNA-seq normalization methods based on flux distributions can be the topic of a future study. Moreover, performing analyses applied in this study with alternative datasets and different mapping algorithms is the topic of future research to improve the accuracy and generalizability of the findings. Only mapping truly affected gene expression levels onto the GEMs by using a simulated dataset can be another future aspect to test the power of mapping algorithms to reveal truly affected reactions/pathways. Such benchmarking studies will give insights for researchers to select RNA-seq normalization methods in studies that integrate transcriptome data with GEMs.

Methods

RNA-seq datasets and pre-processing

To investigate the effects of different normalization methods on the iMAT and INIT based metabolic networks, two different transcriptome datasets were used in this study: (1) the ROSMAP AD dataset consisting of a large sample size, arguably the largest bulk RNA-seq dataset in the literature for AD brain, and (2) the LUAD dataset from TCGA, another common dataset in the literature.

ROSMAP⁴⁹ is a large cohort that includes post-mortem brain samples from the dorsolateral-prefrontal cortex of AD patients and controls. The raw RNA-seq reads (i.e., fastq files with accession number: syn17024112) and clinical meta-data file were retrieved from Synapse (https://www.synapse.org/). First, the quality control of the fastq data was performed using the FastQC tool (version 0.11.9)⁵⁰. Then, the low-quality reads were trimmed with the Trimmomatic tool (version 0.39)⁵¹. The clean reads were aligned to the reference genome using the STAR algorithm (version 2.7.8a)⁵². The reference human genome (GRCh38) from GENCODE⁵³ was used for the annotation of the data. Finally, the read counts were calculated through the featureCounts tool (version 2.0.2)⁵⁴. Using clinical metadata, the samples with a CERAD score of 1 or 2 were selected as AD and the samples with a score of 4 were selected as control. The final pre-processed ROSMAP data contains 402 AD and 164 control samples.

RNA-seq data of 58 control and 524 LUAD samples was accessed from The Cancer Genome Atlas (TCGA)⁵⁵ database using the R package TCGAbiolinks⁵⁶. Metadata for the STAR-aligned RNA-seq data was first downloaded using the GDCquery function, and raw count data was subsequently retrieved using the GDCdownload function.

Normalization of the pre-processed datasets

Five normalization methods with different mathematical and biological approaches were used among the most frequently used normalization methods in the literature⁵⁷ for the normalization of RNA-seq data. On the other hand, there are other normalization algorithms available in the literature^3,10. These methods differ based on whether they are suitable for intra-sample or/and inter-sample comparisons. Two of them (RLE and TMM) perform between-sample normalization while TPM and FPKM perform within-sample normalization, and GeTMM applies both normalization steps (within-sample and between-sample normalization). Only protein-coding genes in the RNA-seq datasets were retained before the normalization steps. In addition, a count per million (CPM) filter was applied on the protein-coding genes to ensure that the gene matrices used in all analyses were the same. For each dataset, we computed the CPM value for each gene. We removed the genes from the dataset if its CPM value was less than 0.1 in n samples, n being the number of samples in the larger group among the two groups to be compared. All the normalization methods were applied to the CPM filtered data.

TPM

TPM, a commonly used within-sample normalization method, normalizes gene expression by considering both the gene length and the sequencing depth. In this approach, the count detected for each gene is first scaled by the corresponding gene length (in kilobase, kB) to obtain reads per kilobase. Subsequently, dividing the reads per kilobase of each gene by the sum of all reads per kilobase in the sample—normalized by a common denominator (a million reads)—gives the TPM normalized gene expression. This allows for the direct comparison of the expression levels of different genes within a sample⁸.

FPKM

Similar to the TPM approach, FPKM normalizes gene expression levels by factoring in gene length and the total number of mapped reads per sample. However, the sum of all mapped reads in a sample are first divided by a million to obtain a per million scaling factor for the sample. This per sample scaling factor is then used to scale the mapped reads of every gene. The scaled gene data is eventually normalized by dividing each gene by its corresponding gene length (in kB), thus, obtaining FPKM. Thus, FPKM and TPM differ only in the order of normalization operations^8,58.

RLE

The R package DESeq2⁶ uses RLE as a normalization method, and it is one of the most commonly used tools to determine DEGs from raw RNA-seq count data. It uses the median of the ratios relative to the geometric mean for a gene as a size factor for normalization. The size factors are calculated using the estimateSizeFactors function in the package.

TMM

edgeR⁵⁹ is another commonly used R package for the DEGs analysis of RNA-seq raw count data. It uses the TMM method for normalization, which minimizes the log-fold changes between the samples for the majority of genes. For the application of the TMM normalization, scaling factors are calculated with the calcNormFactors function in the package. The normalized counts are calculated by providing the scaling factors into the cpm function as an input.

GeTMM

GeTMM² is an RNA-seq normalization technique that was designed to correct between-sample comparisons for gene length, a common feature in within-sample normalization. It performs between-sample normalization based on TMM in addition to the within-sample normalization accounting for only gene lengths. For each gene, the corresponding raw count is divided by gene length, and, as in the TMM method, normalization factors are calculated and normalized counts are obtained.

Covariate adjustment

To remove the covariate effects for AD and LUAD RNA-seq data, covariate adjustment was applied to the normalized data described in the previous section. In this step, age, and gender were considered as covariates for both datasets. PMI was also considered as a covariate for the AD dataset. Using the lm function in R, the normalized RNA-seq datasets were fitted to a linear model by considering the covariates. The effects of covariates were subsequently calculated and removed from each normalized data separately.

Genome-scale metabolic modeling

A human genome-scale metabolic model includes all known metabolic reactions in human with no tissue specificity. It also provides all known gene associations for the included reactions. Human-GEM (version 1.12.0), which is the most recent genome-scale metabolic network reconstruction of human⁶⁰, was retrieved from the GitHub repository (https://github.com/SysBioChalmers/Human-GEM). This comprehensive metabolic network includes 13,070 reactions associated with 3,067 genes, 8,369 metabolites and 143 pathways. It was used to reconstruct sample-based iMAT- and INIT-derived metabolic network models from the ROSMAP and TCGA datasets normalized by five different methods (i.e., FPKM, TPM, RLE, TMM, and GeTMM) and their covariate adjusted versions.

Generating iMAT-based personalized metabolic models

Each sample in the normalized and/or covariate adjusted expression datasets was separately mapped to Human-GEM using iMAT²⁰ in order to generate personalized metabolic network models for all control and disease (AD or LUAD) cases, respectively. The iMAT algorithm requires two thresholds to represent lowly and highly expressed genes. Here, the threshold values were defined as the 25th and 75th percentile of the average gene expression values of protein-coding genes across all samples. These thresholds were used to determine whether a specific gene within any given sample was lowly expressed (i.e., below the 25th percentile) or highly expressed (i.e., above the 75th percentile)^22,61. In the iMAT algorithm, reactions associated with highly expressed enzyme-coding genes are aimed to be kept in the model while the reactions associated with lowly expressed enzyme-coding genes are forced to be removed from the model based on an optimization framework, subject to mass-balance constraints around intracellular metabolites. Reactions associated with moderately expressed enzyme-coding genes are classified as active or inactive based on their mass-balance based interconnection with the other reactions^61,62. Rates of glucose uptake, oxygen uptake and macromolecule (protein, RNA, lipid, carbohydrate) synthesis were constrained to a minimum value in all iMAT simulations to prevent the removal of the associated reactions from the model by iMAT, similar to our previous studies^22,61. Simulations were performed using MATLAB R2018a with the Gurobi solver, and the iMAT algorithm is available under the COBRA Toolbox⁶³.

Generating INIT-based personalized metabolic models

To generate INIT-based personalized metabolic models, first, covariate-adjusted data from each normalization method was mapped to Human-GEM, and reactions were scored using the scoreComplexModel function from the RAVEN Toolbox⁶⁴ for each sample. Then, personalized metabolic models were generated using the INIT function from the COBRA Toolbox⁶³. This function uses a metabolic model and reaction scores to generate condition-specific models. Rates of glucose uptake, oxygen uptake and macromolecule synthesis were constrained similar to iMAT, as mentioned above. Simulations were performed for the ROSMAP data, and MATLAB R2023b with the Gurobi solver was used.

Identification of affected metabolic reactions and overrepresented pathways

The iMAT and INIT-based personalized models were represented as binary vectors where each reaction in the Human-GEM model was labeled as 1 or 0 based on whether they were kept in the model by the iMAT algorithm or removed. After this binarization step, each reaction was compared between control and AD samples for each normalization method and its covariate-adjusted version using Fisher’s exact test⁶⁵ to identify the reactions that were significantly associated with AD (p-value < 0.05). A similar statistical analysis was also applied for the normalized LUAD dataset to identify differential reactions between control and cancer samples. Significantly affected reactions determined by the Fisher’s exact test were subsequently characterized in terms of their pathway associations in the Human-GEM model. We performed hypergeometric test-based enrichment analysis to determine the overrepresented metabolic pathways in AD and LUAD (p-value < 0.05).

Identification and characterization of the genes in the affected metabolic reactions

The significantly affected reactions identified in the previous section were further examined by analyzing the corresponding genes. To determine the genes involved in these reactions, gene-protein-reaction (GPR) associations in the Human-GEM model were used. After creating the regulated gene lists for each normalization method, these genes were characterized in terms of the enriched disease terms using the Enrichr tool⁶⁶. To determine the over-represented terms, this tool compared the regulated gene lists across the gene set library derived from the DisGeNET database⁶⁷. Then, we filtered the significantly enriched disease terms (FDR < 0.05) according to the keywords associated with AD (“Alzheimer’s disease”, “Alzheimer’s disease, early onset”, “Alzheimer’s disease, focal onset”, “Alzheimer’s disease, late onset”, “familial Alzheimer’s disease) and lung carcinoma (“carcinoma of lung”, “adenocarcinoma of lung (disorder)”, “non-small cell lung carcinoma”, “small cell carcinoma of lung”, “lung neoplasms”, “primary malignant neoplasm of lung”, and “malignant neoplasm of lung”). Thus, we evaluated whether the model-predicted regulated genes are associated with AD or LUAD for each normalization method.

In addition to the disease enrichment analysis, pairwise gene set comparisons were performed to assess the similarity of the normalization methods. To this end, binary matrices were created in this section. The experimentally confirmed metabolic gene lists were first collected from the literature^68–73 and databases including KEGG⁷⁴, DisGeNET⁶⁷. Only curated genes in the databases were considered while creating the lists of experimentally validated AD and lung cancer genes. Furthermore, the genes covered by the NGS-based FDA-approved diagnostic gene panel by Foundation Medicine (Foundation Medicine, 2019), called FoundationOne CDx, were also included in the cancer gene list. In the next step, binary vectors were created for each sample by intersecting the experimentally validated disease genes with the model-predicted genes in the significantly affected reactions. Then, the binary vectors were combined to create matrices for each group. For both AD and LUAD, the binary matrices consisted of “1 or “0” to represent the presence or absence of each model-predicted gene in the associated experimentally validated gene list (see the following section for the similarity calculations).

Lastly, the genes of the differential reactions also shared by the experimentally confirmed disease gene lists (i.e., true positive predictions) were compared in a pairwise manner between the normalization methods. Thus, the similarities of the model-based predictions in capturing known metabolic disease genes were further analyzed. The performance of each normalization method was evaluated by computing an accuracy value that represents the ratio of the number of all correct predictions (true positive and true negative predictions) to the total number of all predictions in four confusion matrix categories (true positives, false positives, true negatives, and false negatives). Note that the inactive reactions in all control and disease models for all normalization methods were excluded from the accuracy calculations.

Similarity analysis among normalization methods

Similarity of the normalization methods and their covariate versions was assessed using Jaccard index⁷⁵. Similarity was calculated separately for the list of significantly affected reactions, the list of overrepresented pathways, binary matrices representing active reactions in the reconstructed personalized models and binary matrices representing disease-related genes in the corresponding models. Here, given two normalization methods, P_method1 and P_method2, we computed the Jaccard index (J) using the formula below:

$$J\left(\mathit{method}1,\mathit{method}2\right)=\frac{\left|P_{\mathit{method}1}\cap P_{\mathit{method}2}\right|}{\left|P_{\mathit{method}1}\cup P_{\mathit{method}2}\right|}$$ 1

Since pathways and reactions were specified as vectors, the Jaccard index was calculated with the classical Jaccard index formula (Eq. 1). For binary matrix similarity of models or disease-related genes, to compute the intersection between two matrices, we summed the number of reactions/genes that were jointly identified as active/disease related (’1’) in the two matrices. Similarly, for the union of the matrices, we summed the reactions that were identified as active/disease-related (‘1’) in either matrix. These were subsequently used as the numerator (intersection) and denominator (union) to calculate the Jaccard index.

Author contributions

H.B.L. and T.Ç. designed the study. H.B.L., D.U., R.O., M.F.C., E.D., and E.A. contributed to bioinformatics data analyses, and co-wrote the paper. H.B.L., R.O. and M.F.C. prepared the figures. A.İ. preprocessed the ROSMAP data, R.O. processed the LUAD data. T.Ç. supervised the study and co-wrote the paper. H.B.L. coordinated the analyses and manuscript writing. All authors discussed the results and approved the final manuscript.

Data availability

All data generated or analysed during this study are included in this published article and its supplementary information files.

Code availability

The underlying code for this study is available in GitHub and can be accessed via this link: https://github.com/SysBioGTU/Tx2MetModelBenchmarkingStudy.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41540-024-00448-z.