On Selecting Robust Approaches for Learning Predictive Biomarkers in Metabolomics Data Sets

Abstract

Metabolomics, the study of small molecules within biological systems, offers insights into metabolic processes and, consequently, holds great promise for advancing health outcomes. Biomarker discovery in metabolomics represents a significant challenge, notably due to the high dimensionality of the data. Recent work has addressed this problem by analyzing the most important variables in machine learning models. Unfortunately, this approach relies on prior hypotheses about the structure of the data and may overlook simple patterns. To assess the true usefulness of machine learning methods, we evaluate them on a collection of 835 metabolomics data sets. This effort provides valuable insights for metabolomics researchers regarding where and when to use machine learning. It also establishes a benchmark for the evaluation of future methods. Nonetheless, the results emphasize the high diversity of data sets in metabolomics and the complexity of finding biologically relevant biomarkers. As a result, we propose a novel approach applicable across all data sets, offering guidance for future analyses. This method involves directly comparing univariate and multivariate models. We demonstrate through selected examples how this approach can guide data analysis across diverse data set structures, representative of the observed variability. Code and data are available for research purposes.

Introduction

Metabolites, the small molecules involved in a biological system, play key roles in medical applications such as diagnosis, predicting treatment responses, and assessing mortality risk. Changes and patterns in their abundance provide valuable insights into a patient’s health condition and have been used to diagnose diseases such as lung cancer, breast cancer, and H1N1 influenza pneumonia. Moreover, these small molecules offer a unique perspective in multiomics studies, where they are integrated with other types of biological datasuch as genomics, transcriptomics, and proteomicsto uncover holistic patterns at the systems biology level. ,

Recent advances in metabolomics have been driven by rapid improvements in data acquisition technologies. Measurement devices, such as mass spectrometers, are increasingly precise in measuring metabolite abundance, with expanding dynamic ranges. Data quality has also improved through enhanced methodologies for mapping measured signals to known molecules. These advances, coupled with the ability to generate large volumes of data, position the field as being ripe to benefit from the application of machine learning techniques. This is apparent in the recent literature, where machine learning has been used at all steps of the data acquisition and analysis pipeline. Applications range from improving peak alignment in mass spectrometry data to the crucial problem of biomarker discovery, i.e., metabolic characteristics that are predictive of a phenotype of interest. − The latter is crucial to build accurate diagnostic tests and to improve our understanding of the mechanistic underpinnings of such phenotypes.

Machine learning methods have proven effective in identifying metabolic biomarker signatures, using algorithms such as Partial Least Square Discriminant Analysis (PLS-DA), Orthogonal PLS-DA (OPLS-DA), ,, elastic net regression, random forests, and boosting of decision trees. Among these, PLS-DA and OPLS-DA stand out as the most prevalent methods due to their ability to handle highly collinear and noisy datacharacteristics common to metabolomics experiments. The reviews by Gromski et al. and Ruiz-Perez et al. provide key considerations for applying PLS-DA to metabolomics data. Other works, such as that of Ferrario et al., propose univariate approaches that seek associations between single metabolites and the phenotype of interest. However, as emphasized by Wolpert et al.’s no free lunch theorem, no single machine learning algorithm works universally well across all prediction tasks.

In fact, metabolomics data sets make for a considerable challenge for machine learning algorithms. Modern measurement devices generate so many measurements that there is often a high imbalance between the number of metabolites measured in an experiment and the number of available samples, which is often small in comparison. This problem setting, termed fat data by Romero et al., creates a needle-in-a-haystack situation which introduces several challenges, including difficulties in parameters estimation, increased computational complexity, and susceptibility to noise. , For example, spurious patterns resulting from noise may appear as predictive and lead to fitted models that have poor out-of-sample generalization.

These challenges have led the community to conduct exploratory studies to understand which algorithms are useful in which contexts in metabolomics. Some studied the performance of various algorithms on specific metabolomics data sets. ,, Others, such as Mendez et al., considered panels of a few data sets, concluding that linear-based and nonlinear machine learning models have similar performances and that algorithm performance often depends on the data set characteristics such as the dimensionality. Nevertheless, their analysis was limited to data sets with tens to hundreds of features, which do not come close to the dimensionality of data produced by modern instruments. Importantly, as opposed to related fields, such as bacterial genomics, no studies have been conducted at the biobank scale.

In this sense, the present study proposes the first biobank-scale analysis of machine learning algorithms for predictive biomarker identification in metabolomics. Our work disentangles the settings in which simple univariate patterns, such as those identified by Ferrario et al., outperform complex multivariate nonlinear patterns. , To this end, we perform a comparative analysis over 835 data sets extracted from the MetaboLights database, comparing state-of-the-art machine learning methods with simple univariate approaches. Our results reveal a strong association between the properties of the data at hand and the performance of certain approaches. Moreover, they provide methodological insights that can help practitioners understand when simple versus complex approaches may be warranted.

Methods

Data Sets

We collected the data sets from the MetaboLights database. As of May 20, 2024, the database contained 7889 published studies. A semiautomated process was used to generate the data sets for our experiments, following several steps. First, we excluded studies that are inaccessible through either the API or the website, which accounted for 6453 studies. These studies are likely unpublished. We further exclude studies where no data files are available, such as those containing only metadata and no metabolites data; this accounted for 68 studies. Another category of nonusable studies includes those with data matrices providing only metabolite descriptions or summary statistics (e.g., mean and standard deviation) without data for individual samples. This accounted for 472 studies. Some studies included both metadata and data matrices, but the identifiers of the samples did not match between the two files, making them unusable. This accounted for 95 studies. In some cases, the metadata matrix contained no phenotype relevant for prediction, for instance, when the metadata variables described only data acquisition methods. This accounted for 80 studies. As we collected data sets with at least 10 samples in each class, some studies had data matrices with too few data points to be usable for predictive modeling. This accounted for 395 studies. Finally, we were able to generate data sets from 326 studies.

For these 326 studies, we generated a data matrix X. We then explored the metadata to create a class vector Y composed of elements from the binary set {y ₀, y ₁}, identifying variables that describe the samples and could be grouped into two or more classes. We denote D as the set of candidate class variables. Throughout this process, we aimed to cover a broad range of tasks, refraining from excluding variables based on subjective judgments of interest or relevance. For each variable d in D, binary class labels were generated based on the values of the variable. If d takes only two values, then this process is straightforward. For unordered variables with three values, such as d ₁, d ₂, and d ₃, we generated three binary vectors: “y ₀ is d = d ₁ and y ₁ is d = d ₂”, “y ₀ is d = d ₂ and y ₁ is d = d ₃” and “y ₀ is d = d ₃ and y ₁ is d = d = d = d ₁”. If d contained four or more unordered values, we generated k vectors, where for each value d _i , “y ₁ is d = d _i and y ₀ is d ≠ d _i ”. In this case, the number k of generated data sets from variable d is limited to the four most frequent classes.

When a study’s metadata includes multiple variables of potential interest, separate binary data sets are generated for each variable, unless the variables are clearly redundant. In some cases, the study contains several metabolomics data matrices as a consequence of the acquisition technique, for example, data from positive and negative ionization. In this case, a separate data set is generated for each data matrix. Consequently, multiple data sets can originate from the same study. The maximal number of data sets generated from a single study is 16, while 143 of the studies generated only one data set. Details about the number of data sets from each study are shown in Figure S4. The published data matrices contain both annotated and unannotated metabolites from targeted and untargeted approaches; all categories of variables are retained while building a data set. The missing values are handled variable-by-variable; if a variable column contains more than 20% of missing values, this column is removed; otherwise, the missing values are replaced by zeros. A data set is only created if there are at least 10 samples in each class. However, no minimum threshold is imposed on the number of features. This results in the pattern shown in Figure a, where no data sets contain fewer than 20 samples, yet some data sets have as few as 6 features. This approach generated 835 data sets. Detailed information on the characteristics of the data sets is available in Table S2, joined as a separate file.

3.

Dimensions of the data sets and the best model category. Figure a: Each point represents a data set. The x-axis is the number of features, and the y-axis is the number of samples. The points are colored according to the category of the outcome on the size of the best model of the data set. The dashed line represents the case in which the number of samples is equal to the number of features. The dashed-dotted line represents the case where the number of samples is 10 times greater than the number of features. The dotted line represents the case where the number of features is 10 times greater than the number of samples. Figure b: Number of data sets for each category of outcome given the data set is “fat” or not (corresponding to the lower-left part and the upper-right part of Figure a).

Predictive Tasks and Machine Learning Problem Setting

The data sets are built to offer a binary predictive task. A single observation (data point) is referred to as an example x from a domain $\mathcal{X}$ . These examples are described by p variables (features) {f ₁... f _p }, which are the components of vector x. Hence, $\mathcal{X}\subseteq {\mathbb{R}}^p$ . Machine learning refers to algorithmic methods that learn a task from observing data. A machine learning algorithm seeks to build a predictor $h:\mathcal{X}\to \mathcal{Y}$ , that assigns a label $y\in \mathcal{Y}$ to each example x. Here, we focus on the case of supervised learning, where the learning algorithm is given a training set of n examples $\{(x_1,y_1),...,(x_n,y_n)\}\in {(\mathcal{X}\times \mathcal{Y})}^n$ . The examples are assumed to be drawn independently from an unknown, but fixed, data generating distribution.

Univariate Method

The predictive performance of each feature is computed individually. Each feature’s score is determined by the performance of the best decision stump derived from that feature. This score is averaged over an 8-fold cross-validation, ensuring decision stumps are evaluated on a data fold not used for threshold determination. Consequently, the univariate score of each feature can be directly compared to the scores of machine learning methods, as shown in Figure .

4.

Focus on a selection of three data sets, showing performances of all multivariate and univariate models. Each row corresponds to one of the three data sets. The left graph shows the scores of multivariate machine learning models, and the right graph shows the scores associated with each individual feature seen as an univariate predictive model for the same data set. The y-axis of both graphs displays the test-set score of the models averaged over 8 splits. The horizontal lines are shared by the two graphs: the blue long-dashed line marks the score of the best multivariate method, while the red short-dashed line marks the score of the best variable as a univariate predictor. Univariate models are decision rules that rely on a single feature. Therefore, the graphs on the right contain one point for each data set feature, and the features are sorted by decreasing average score.

The univariate score of a feature f _j is intended to quantify the amount of information that f _j provides about the target variable y. In this context, the univariate score reflects the potential unconditional association between f _j and y. It is important to note that this association is not necessarily causal as it could be mediated by other variables, either measured or unmeasured. Alternative methods for estimating feature importance might capture different types of associations, as discussed in the work of Ewald et al. However, since our primary objective is to compare individual features directly to machine learning models, we chose to use the univariate score.

Multivariate Methods

In most cases, we used the Python implementations available in the scikit-learn library. For PLS-DA, we used the PLSRegression implementation from scikit-learn, which includes the necessary steps of scaling and centering. Class predictions for PLS-DA are made by binarizing the regression outputs. For the Support Vector Machine with a Linear Kernel (SVMlinear), we employed the LinearSVC implementation from scikit-learn, with data scaling performed using the StandardScaler function. Similarly, for the Support Vector Machine with a Radial Basis Function Kernel (SVMrbf), we used the SVC implementation from scikit-learn, with data scaling also performed using the StandardScaler function. The Decision Tree (DT) model is based on the DecisionTreeClassifier from scikit-learn, while the Random Forest (RF) is based on the RandomForestClassifier from scikit-learn.

We included the Set Covering Machine (SCM) method due to its success in biomarker discovery in genomic data, which produces extremely sparse decision rule lists. We used the implementation available at https://github.com/aldro61/pyscm. The Random Set Covering Machine (rSCM) is specifically implemented for this work, designed to capture information from metabolic pathways while maintaining sparsity. The rSCM constructs an ensemble model using decision lists from the SCM. Our implementation is available at https://github.com/thibgo/randomscm.

For AdaBoost, we used the AdaBoostClassifier implementation from scikit-learn. The Gradient Boosted Trees (GBtree) model is implemented using the GradientBoostingClassifier from scikit-learn. SCMBoost is developed by combining the SCM implementation with the AdaBoostClassifier from scikit-learn, using SCM as the base estimator for AdaBoost. Finally, for XGBoost, we used the XGBClassifier implementation from the xgboost library, available at https://github.com/dmlc/xgboost.

Note that no method based on neural networks is included in the benchmark. This is because high dimensionality prevents learning such a model adequately from the limited sample size. In a standard neural network, the size of the input layer is the number of features in the data. This implies the need for more data samples to converge to a well-fitted network. This limitation significantly hinders the ability of neural networks to learn from data when the number of variables is considerably larger than the number of samples.

Evaluation of Predictive Performances

Ideally, the models would have been evaluated on an independent data set from a separate study: one that examined the same Omics data and focused on the same phenotype but used a different set of samples. Unfortunately, no such complementary studies were available for the data at our disposal. As a result, we estimated the predictive performance through multiple empirical evaluations using cross-validation techniques.

For assessing the models, we selected the balanced accuracy score as the primary metric for both training and evaluating the predictive performance. While no single metric is perfect for all scenarios, the balanced accuracy is chosen because it considers equally the false positives and false negatives and is not affected by imbalanced ratios of positive and negative samples.

Hyperparameters Choice

The choice of the hyperparameter space for each algorithm in the benchmark is of high importance, as it may greatly affect the model performance. Our work aims to compare the best predictive performance of multiple machine learning algorithms across a variety data sets. As we do not want to include prior knowledge about hyperparameter values to prefer, we favor broad hyperparameters search space for each machine learning algorithm. However, some algorithms have too many hyperparameters to be exhaustively included in the search space. In particular, the inclusion of multiple hyperparameters that have similar effects on the obtained model can lead to an inefficient searching procedure. For instance, the random forest implementation in the Python library scikit-learn includes 14 hyperparameters: one controls the number of trees, four govern the diversity among trees, seven determine the size of the trees, and two influence how decision rules are selected at tree nodes. To ensure that each of these four key characteristics is optimally tuned, we selected one or two hyperparameters to optimize per group; the other hyperparameters are fixed to the default values of the used implementation. This same methodology is applied when the hyperparameters are selected for the other machine learning algorithms. The list of optimized hyperparameters and their range is provided in Table S1 (Supporting Information).

Hyperparameters Tuning

The hyperparameter values are selected from a Bayesian optimization search. Compared to other standard hyperparameter search methods, such as grid search and random search, the Bayesian optimization approach is an iterative procedure that aims to learn from previous model-building instances to get closer to the optimal hyperparameter values for the hyperparameters. The first Bayesian optimization steps broadly explore the hyperparameter space, and the subsequent steps focus on regions that show greater promise to improve the final model score. For each multivariate machine learning algorithm, we perform a fixed number of 50 Bayesian optimization iterations. A consequence of using Bayesian optimization is that all algorithms undergo the same number of optimization steps. In contrast, grid search may disproportionately favor algorithms with more hyperparameters by allocating more computational effort. Of note, we compared Bayesian optimization with grid search on a subsample of the data sets in our preliminary experiments. The obtained results showed no significant difference in the predictive performance of the models produced by the two hyperparameter search methods. However, the grid search required substantially more computational resources, so we retained the Bayesian optimization method to produce the reported experimental results.

Embeddings of Data Sets Descriptions

Figure provides an overview of the data set collection using a two-dimensional projection derived from the textual descriptions of the data sets. These descriptions include the target phenotype, the two classes to be distinguished, and the organism name. Text embeddings were generated for each description using “text-embedding-3-small” API from OpenAI, producing 835 embedding vectors. We applied dimensionality reduction using Principal Component Analysis (PCA) to produce a two-dimensional representation. The resulting projection is used to create Figure , where data sets are plotted in the reduced space and color-coded to reflect their characteristics.

1.

Overview of the data sets in the collection. Each point represents a data set. For visualization purposes, the data sets are projected onto the first two dimensions of a principal component analysis computed on embedding vector representations of the literal descriptions of the data sets. See Figure S1 for details. This projection provides an overview of the characteristics of the data sets. Figure a: data sets are colored given keywords present in the description of the classes to predict. Figure b: data sets are colored given the organism studied. Figure c: data sets are colored given the family of metabolomics acquisition techniques used.

Results and Discussion

Characteristics of the Metabolomics Data Sets Benchmark

Our experiments are conducted on a collection of 835 metabolomics data sets generated from the public database MetaboLights, following the curation process described in detail in the Methods section. The current Results and Discussion section first presents an overview of the data sets’ main characteristics. We then compare the performances of machine learning model categories (univariate and multivariate), which study highlights the correlation between the data set dimensions and observed outcomes. Lastly, we demonstrate using several specific data sets that comparing models following our suggested approach gives valuable insights into the data set structure, orienting further metabolic analyses.

The distinctive characteristics of the collected data sets are summarized in Figure , which in turn highlights trends within the metabolomics research community. Notably, 165 of the data sets are labeled with keywords “disease” or “cancer”, underscoring a strong focus on studying disease-associated metabolomics. Nearly all of these disease-related data sets are derived from human samples. Human (Homo sapiens) samples represent 305 of the data sets, more than a third of the total. The second most frequent organism is mice (Mus musculus), the preferred animal for biomedical research with a perspective on human health applications. Mice samples are present in data sets with the keywords “mutation” and “treatment”. We also observe a significant interest in plant metabolomics. The category “Plants” mainly encompasses important agricultural species like rice and wheat. For plant metabolomics data set, the most frequent descriptive variables are about the variety (cultivar) and sampling location. The most utilized data acquisition technique is liquid chromatography-mass spectrometry (LCMS), representing more than half of the data sets. Other common techniques are gas chromatography-mass spectrometry (GCMS) and nuclear magnetic resonance (NMR) spectroscopy. There is no clear association between phenotype keywords and technique categories. There is also no clear association between the organism studied and the technique category.

Detailed data on the characteristics of the data sets and studied organisms are listed in Table S2 and Figure S3, provided as Supporting Information. Note that among the 835 collected metabolomics data sets, there is an average of 164.6 samples per data set, with a median of 62. The average number of features per data set is 760.7, with a median of 135. The average feature-to-sample ratio is 13.9, while the median ratio is 2.2.

The data set collection method is influenced by implicit choices, the most significant of which is the definition of binary classes derived from the study metadata of the studies, as described in the Data Sets section. Such researcher degrees of freedom can have a big effect on the final results. Here, we present some reflections on the choices that we made. Some studies were not primarily designed for binary classification or even not designed for classification at all. We choose to keep them when it is possible to define binary classes from metadata variables. For example, several plant metabolomics studies use “cultivar” or “variety” as the only descriptive variable in the metadata. In those cases, we defined binary data sets using the values of this variable, even though the study may have not been primarily designed for binary classification and biomarkers discovery. At worst, including these data sets in our collection is inconsequential and does not affect the validity of our results.

Our experiments are restricted to binary phenotypes. But some phenotypes of interest are multiclasses or continuous. We chose to restrict the data set definition to binary phenotypes to ensure consistency across the collection. However, it is possible to build both multivariate and univariate predictive models in contexts in which the phenotype is not binary. Therefore, our suggestions for data exploration also apply to multiclass and continuous phenotypes of interest. However, by focusing on binary classification, we may have oversimplified the problem space, and some methods may perform differently in multiclass problems.

Metabolomics research is rapidly evolving, and the development of comprehensive benchmarks and robust analytical methods is crucial for advancing the field. We collected data sets from the MetaboLights database, but there are other online metabolomics databases that could be included to create a larger and more diverse benchmark. It remains to be determined whether incorporating data sets from these additional sources would result in a benchmark that more accurately represents the diversity of data generated by the metabolomics research community.

Comparison of Performances between Univariate Models and Multivariate Machine Learning Models

Figure is a pie chart that summarizes the comparative performance between our two main categories of machine learning models across the collection of data sets: univariate models (relying on a single data set feature) and multivariate models (relying on two features or more). The machine learning algorithms tested in both categories and the learning methodology for each algorithm are presented in the Methods section. Multivariate models outperform univariate models in 44.9% data sets analyzed while univariate models perform better in 21.2% of the data sets. In 32.3% of the cases, both types of models perform equally well and, in the remaining 1.6%, no method surpasses the baseline. Main observations from Figure are summarized in Table .

2.

Performance summary of univariate versus multivariate models across the 835 data sets extracted from the MetaboLights database, measured in terms of balanced accuracy score. Each data set is represented by two connected points: the bullet (•) pinpoints the score of the best multivariate model and the cross (×) pinpoints the best univariate model score. The length of the line segment linking both points reflects the performance gap between model categories. The chart colored areas highlight how the best univariate model compares to the best univariate model on each data set; In the blue area, the bullet point dominates the cross, indicating that multivariate models achieve higher scores, while in the yellow area, the converse is true.

1. High-Level Summary of the Results is Presented in Figure .

Among the data sets for which multivariate models perform better (blue area of the chart), the best overall balanced accuracy scores range from 0.5 to perfect prediction (score of 1). Most data sets in this category achieve a score above 0.8. The performance gain obtained by using multiple variables is shown by the length of the line segment connecting the bullet point (multivariate model) to the cross (univariate model). However, there is no apparent relationship between the best model score and the size of the performance gain. The distribution of the differences is shown in Figure S5. Among the data sets for which univariate and multivariate models perform similarly (green area of the chart), a perfect score of 1 is achieved for over half of the data sets. It shows that this category regroups mostly data sets where both types of models are able to predict perfectly (or almost perfectly) the phenotype. This may be explained by the presence of at least one feature that contains enough information to separate all of the samples in the two phenotype classes. Among the data sets for which the univariate models perform better (yellow area of the chart), the score distribution is similar to the one observed in the blue area of the chart. However, the performance gap between univariate and multivariate models tends to be larger when the best overall score is below 0.8, and smaller when the best score falls between 0.8 and 1.

A surprising observation from Figure is that in 21% of the data sets, a single feature predicts the classes more effectively than any machine learning model. This raises the following question: If the optimal prediction is achieved using a single decision rule, then why does hyperparameter tuning not cause the decision tree to simplify to a single node, thereby replicating the optimal decision rule. The answer lies in overfitting. Even though model selection and training follow established best practices, we observe that in these data sets, the models tend to use more than one variable, leading to overfitting.

We evaluated the univariate association between variables and phenotypes through the lens of the predictive performance of a model based on the variable. This choice is made to facilitate a direct comparison to multivariate machine learning models. However, from a data exploration perspective, alternative measures of variable-phenotype association can be used. Consequently, our conclusions are limited to the specific univariate approach we selected and do not extend to all possible univariate methods.

Univariate Models Often Outperform Multivariate Models in Fat Data Sets

Figure illustrates the influence of the data set dimensions, in terms of the number of samples (y-axis) and number of features, on the best-performing machine model category (based on the color code established by Figure ). A data set containing more features than samples is qualified as “fat data”. This opposes the case of “big data”, which is more commonly encountered in machine learning applications. In the literature, the “fat data” situation is also referred to as the “p > n problem”, with p being the number of variables and n being the number of samples. As depicted by the figure, metabolomics data sets are more commonly “fat” than “big”. Out of the 835 data sets, 546 (66%) have more features than samples. Since we only collected data sets with at least 10 samples in each class, there are no data sets with less than 20 samples (see the Data Setssection).

The main observation stemming from Figure is the association between the data set dimension and the univariate being better than multivariate models. For the 286 data sets that are not fat, multivariate models do better on 180 (63%) of the data sets and univariate models do better on 28 (10%) of the data sets, the remaining 78 (27%) data sets being undecided. For the 549 data sets that are fat, multivariate models do better on 195 (36%) of the data sets and univariate models do better on 149 (27%) of the data sets, the remaining 205 (37%) data sets being equivalent.

The diversity of metabolomics data sets is illustrated by the various characteristics describing the collection of data sets in Figure and the large range of dimensionality visible in Figure . Among the data sets characteristics, the dimensionality is the characteristic that most strongly influences the nature of the best predictive models. This aligns with the conclusions of Mendez et al., who highlighted that the size of the data set has more influence on predictive performance than the structural complexity of the models.

Assessing Both Multivariate and Univariate Models’ Performances Gives Significant Information on the Data Set Structure

We have observed that there exists significant variety in metabolomics data sets. This prevents the formation of a priori expectations about the models. In this section, we describe in detail the observations made on three specific data sets of the collection. This selection aims to represent the diversity of data sets. The three selected data sets include different organisms, phenotype keywords, and metabolomics acquisition techniques. They also fall into each of the three main categories represented in Figure .

The plants metabolomics data set A comes from study MTBLS2289 of the MetaboLights database. The 259 biological samples are from Arabidopsis thaliana, a common model plant organism in genetics studies. Each sample is described by 62 features measured by GCMS. The predictive task is to distinguish plants expressing the “BRL3” gene from wild-type plants. Among multivariate models, the best balanced accuracy score is 0.88, outperforming the best univariate score of 0.66. Score discrepancies indicate a significant gain of information when combining multiple variables. Analyzing the univariate models first, we observe that the best two variables, sharing scores of 0.66 each, are “4-hydroxyproline” and “Proline”. The third and fourth best scores are obtained by “Spermidine” (0.59) and “Threonate” (0.58). As for multivariate models, we note that 10 machine learning algorithms out of 12 outperform the univariate models. The two SVM variants and the four boosting methods obtain similar scores. The Random Forest is close behind with a score of 0.84. The latter algorithm was chosen because it represents nonlinear models and its structure is well known and interpretable. Here we provide details about the corresponding learned models. To do so, we search for co-occurring feature pairs among the 8 models trained across the 8 data set splits; the models contain an average of 74 decision trees. The most frequently co-occurring feature pairs are “4-hydroxyproline” + “Proline” (appearing together in an average of 40 decision trees per model), “Proline” + “Serine” (34 times per model), and “Proline” + “Threonate” (33 times per model). Furthermore, the triplet “4-hydroxyproline” + “Proline” + “Serine” is the most frequently associated combination, appearing on average 23 times in each model. These findings suggest that these combinations of variables are strong candidates for establishing a robust set of biomarkers for the phenotype.

The human metabolomics data set B comes from study MTBLS4206 of the MetaboLights database. The 56 biological samples are described by 95 features measured with LCMS. The predictive task is to distinguish patients with Psoriatic Arthritis from patients with Rheumatoid Arthritis. The best multivariate score is 1.0, and the best univariate score is 0.98. All but one rule-based multivariate model achieves almost perfect predictive score, while linear-based methods have lower scores. The variable achieving the highest univariate score is “Epsilon caprolactam” (0.98), followed by a group of variables with scores between 0.89 and 0.91: “Dihydrosphingosine 6”, “Dl-Dihydrosphingosine”, “Hexadecasphinganine 8”, and “2-(5,8-Tetradecadienyl)­cyclobutanone 2”.

The human metabolomics data set C comes from study MTBLS5297 of the MetaboLights database. The 33 biological samples are described by 137 features measured with LCMS. The predictive task requires distinguishing two stages of the disease Rheumatoid arthritis: patients on the first of four stages from patients on the third of the four stages. The best univariate model has a score of 0.91, outperforming the best multivariate score of 0.70. The variable achieving the highest univariate score is “Uridine” (0.91), followed by “Thymidine” (0.81), “Uracil” (0.80), “Acamprosate” (0.79), and “Uric acid” (0.77). While almost every multivariate model performs better than the baseline of 0.5, no algorithm is able to capture as much information as the single variables.

In this study, we evaluated several biomarker discovery techniques on 835 metabolomics data sets. For researchers using machine learning models for biomarker discovery, we recommend including a “variable-by-variable” univariate analysis as part of the experimental framework for prediction. This approach provides valuable insights into the relationships between individual features and the phenotype and helps address key questions such as: Is there an advantage in using multivariate models over single variables? If the answer is yes, the focus shifts to exploring the combinations of features that provide the most information about the phenotype, leading to the question: What is the structure of the best-performing multivariate models? This requires analyzing the machine learning models to identify the most influential variables. If the answer is no, attention should instead be directed to univariate scores, asking: Is there a metabolite significantly more predictive than any other? Such a metabolite could be either a highly effective biomarker or a bias induced by the study design. In other words, this approach highlights cases where the Rashomon set of models that perform equally contains a model of size 1, which is key information in a biomarker discovery process. In either case, identifying the most relevant variables is essential, whether they perform best alone or in combination within a multivariate model. To further evaluate the benefits of multivariate models, one could implement a feature-deletion process, systematically removing featuresstarting with the most important, and assessing the predictive performance of the model based on the remaining features. All the methods mentioned above contribute to find the variables most associated with the phenotype of interest y, but these associations may not be causal. There are theoretical studies that propose methods for identifying causal variables in large data sets; however, it remains to be determined whether these methods are applicable to metabolomics.

The results of Figures and show the diversity of methods that can achieve an optimal prediction score. Inside the category of multivariate methods, the diversity of the best-performing algorithm also exists, as shown by Figure S7. This aligns with the conclusions of the works of Le et al. and Shen et al. which illustrate the advantages of combining multiple selection techniques to avoid being influenced by the biases or limitations of individual feature selectors.

Conclusions

Our work proposes heuristic principles for researchers exploring metabolomics data sets in conjunction with a prediction task. First, we demonstrated significant diversity in metabolomics data sets, highlighting the need for an initial approach without strong prior assumptions that guide the following analysis. Second, evaluating the predictive performance of individual features (treated as predictive models) provides valuable insights into the data set’s structure and the relationships between variables and the target phenotype. Finally, when multivariate machine learning is applied to better understand metabolomics data, it is crucial to explore a variety of methods, including both linear and rule-based models, since the commonly used PLS-DA method is not always the most effective.

The proposed “multivariate versus univariate” data set analysis offers valuable insights by identifying when a single-feature model falls within the Rashomon setthe collection of models that perform equally well on a given predictive task. We advocate that approaching biomarker discovery from this perspective is particularly advantageous as it highlights the potential simplicity and interpretability of effective models. However, a method to determine the minimum number of features required to construct a predictive model within the Rashomon set is still lacking. We hope that this work serves as a foundation for developing such a method, paving the way for more efficient and interpretable biomarker discovery.

All the source data files are publicly available via the MetaboLights API, without any access request needed. The code to generate the collection of 835 data sets from this database is available at https://github.com/thibgo/metabolightsbinarydatasets

The code used to compute the experiments presented in this work is published at https://github.com/thibgo/onselectingrobustapproachesinmetabolomics

The python implementation of the Random Set Covering Machine (rSCM) algorithm, developed specifically for this work, is published in https://github.com/thibgo/randomscm

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.analchem.5c01049.

• Descriptive characteristics of the data sets in two different projections; characteristics of the data sets grouped by category of observed outcome; frequency species of living organisms in the data sets collection; number of data sets generated per study; distribution of differences in predictive scores between univariate and multivariate methods; positions of the 3 selected data sets of Figure in the projections of Figures and ; frequency of top-prediction achievement by each multivariate methods; geographical provenance of the data sets on a world map; hyperparameters tuned for each multivariate algorithm; and detailed characteristics and main experiment results for each data set of the collection (PDF)

T.G. conceived and conducted the experiments. All authors analyzed the results and contributed to the redaction of the manuscript.

The authors declare no competing financial interest.