Slimformer: An NLP-based web server for semantic categorization of gene sets

Abstract

Omics data analysis often yields extensive lists of genes or enriched gene sets, making it difficult to interpret the underlying cellular mechanisms. Existing gene set categorization methods typically rely on the Gene Ontology hierarchy, neglecting semantic similarity encoded in textual descriptions. We developed Slimformer, an embedding-based Natural Language Processing model that learns contextual relationships between gene sets based on their names, descriptions, and associated genes. A supervised classifier then assigns these embeddings to process categories, trained on a manually curated gold standard. Applied to 2856 annotated gene sets, Slimformer achieved 82.4 % balanced accuracy and an F1-score of 0.867. Applied to gene expression data from human cells infected with Respiratory Syncytial Virus, Slimformer revealed strong downregulation of major cell cycle processes which is highly relevant for the viral pathomechanism, which was overlooked by other tools we tested. By integrating linguistic and functional information, Slimformer enhances the interpretability of omics data and provides a flexible framework for systematic gene set categorization.

Graphical Abstract

1. Introduction

Genome–wide omics methods, such as transcriptional profiling, proteomics, screening of genome–wide functional knockouts, or protein–protein interaction screens are commonly used to investigate the cellular mechanisms from a systems perspective. Statistical enrichment analysis of functionally annotated gene categories is often performed to obtain an overview of large gene or protein lists [26]. Most commonly, gene set definitions from Gene Ontology (GO) are used, providing a controlled vocabulary for annotating and grouping genes [1]. Numerous web-based tools have been developed for gene set enrichment analyses [12]. The lists of GO terms can then be grouped into broader categories, denoted as GO slims, to gain a quick overview of the main commonalities in these lists. GO slims typically focus on terms relevant to a specific conditional context [5]. In GO, gene sets are in a parent–child relationship, which leads to a hierarchical structure within a directed acyclic graph (DAG). This structure has been used to group GO terms into GO slims, categorizing a GO term by following the graph upward from the term along all possible paths [11], [13]. This approach assumes that all terms in a parent-child relationships are equally distant semantically, which is not functionally accurate. Tools such as Categorizer [21] or REVIGO [31] use a semantic similarity measure based on an information content score, which is high for GO terms that are less frequently used (being rather unique) and low for more regularly used GO terms. However, these approaches do not consider semantic similarity derived from the descriptions of the gene sets.

While existing tools rely primarily on the structure of the GO graph, advances in natural language processing (NLP) enable the capture of richer contextual relationships. Embedding-based NLP approaches have become increasingly important for addressing complex challenges in bioinformatics, such as the prediction of promoter regions, transcription factor binding sites, or splice sites [14], as well as protein structure prediction [18]. We used Doc2Vec [17], which learns fixed-length vector representations of documents by predicting a target word from its context within a document. For each GO term, a document was compiled using the GO term’s name, description (definition), its associated genes (represented by gene symbols), and the names of its parents, ancestors and children. We fine-tuned the model by predicting predefined categories of manually annotated gene sets.

Here, we present Slimformer, an embedding-based NLP model that integrates textual and functional annotations to create context-aware categorizations of Gene Ontology terms. As a case study, we applied Slimformer to gene expression data from cells infected with Respiratory Syncytial Virus (RSV), one of the major causes of hospitalisation with acute respiratory infections and a leading cause of death in infants under one year of age, elderly individuals, and immunocompromised patients.

2. Methods

2.1. Databases and classes

Gene set definitions were taken from the database of g:Profiler2 (https://biit.cs.ut.ee/gprofiler_archive3/e111_eg58_p18_f463989d/, updated January 2024) [16], the definitions of the aspect biological process, and the relationships defined by the Alliance of Genome Resources, the Gene Ontology (version from https://release.geneontology.org/2024–11-03/ontology/index.html) [1], [32]. We focused on the aspect of biological process containing 15,234 gene sets. We removed gene sets with more than 5000 associated genes to exclude overly broad terms, resulting in 15,200 remaining gene sets. As the gold standard for machine learning, we manually annotated 2856 gene sets. Manual annotation was done by three different scientists, one researcher trained in biochemistry and molecular medicine, one researcher trained in bioinformatics, and one senior researcher trained in bioinformatics. After initial annotation by the researchers, the annotation was discussed among the authors of this work. Seventeen categories customized for viral research were defined. These categories became apparent through manual annotation of gene sets derived from enrichment analyses of many different gene expression and gene knockout screening data sets across a large range of different virus studies. Additionally, the category Other was defined for gene sets that did not fit into any of these categories. We prioritized immune response during viral infection. For example, GO terms describing the maturation of immune cells were assigned to Immune system process rather than Morphogenesis/development. If a gene set could fit into more than one category, the more concise category, typically the smaller category, was selected. A list of all manually annotated gene sets is provided in Supplementary Table S1.

2.2. Preprocessing

To assemble comprehensive information for each gene set, we compiled a text-based document comprising (1) the words of the name of the GO term, (2) gene symbols of the genes associated with each GO term, (3) each word of the description (definition) and (4) the words of the names of the GO terms related to the GO term in an “is_a” relation to make use of the parent and child relationships provided by the GO hierarchy. To further incorporate the GO hierarchy, (5) all the intermediate GO terms along the path from the term to the root of the DAG were additionally included as ancestors. These lists of words were joined together for each GO term into a continuous text, which formed a single document. The “is_a” relationships, definitions, and genes of the GO terms were extracted from the basic version of GO via ontologyIndex [10] and GSA [30] in R (version 4.4.1) (R [24]). Next, we applied standard text pre-processing steps for data cleansing. Using the tm package [8], the documents were converted to lowercase, and punctuation and English stop words were removed. Supplementary Figure S1 shows an example of a GO term document before and after pre-processing.

2.3. Pre-training

For the embedding-based model, a numerical vector representation for each of the documents was computed. This was accomplished via the Paragraph Vector (PV) model of Le and Mikolov [17]. The distributed bag of words (PV-DBOW) version of PV was chosen because word order was considered to be not relevant for our classification task. PV-DBOW uses the complete document of a GO-term as input to predict $\mathrm{k}$ masked words sampled from that document. We used $\mathrm{k}=1$ as this led to the best performance after initial trials (data not shown). The minimal word frequency was set to 1 to include gene symbols and trained for 25 iterations. Other parameters were set by their default values. The embedding dimension $\mathrm{d}$ was determined using multiple different sizes (10–500 in steps of 10) and evaluated by computing the Euclidean distance between each pair of document vectors, followed by calculating the Dunn index [6] for each of the categories. A higher Dunn index corresponds to more compact clusters and a more distinct separation of clusters. The Dunn indices for the different dimensions were smoothed by a local regression (LOESS, default parameters). This led to the embedding dimension $\mathrm{d}$ = 200, as described in the Results section.

2.4. Classifying gene sets into categories and visualization

To classify the embedded document vectors and the corresponding gene sets into the 18 categories, three classifier methods implemented in scikit-learn [22] were trained and tested on 2856 manually annotated GO terms. We used Random Forests (RF), a Logistic Regression (LR) classifier, and Support Vector Machines (SVM), using default hyperparameters. Multiclass classification was performed for LR via multinomial loss for multiclass classification and for SVM via one-versus-rest classification. Training and test data were split in a stratified manner, and 10-fold cross-validation (CV) was applied. For each of the cross-validation runs, the models were fine-tuned via parameter optimization by a grid search within an internal five-fold cross-validation loop. Supplementary Table S2 lists the tested parameter settings, Supplementary Table S3 shows the results of a sensitivity test for the optimized parameters. The performance of the models was measured by their balanced accuracy and F1-score. The best-performing model was used to categorize the remaining 12,431 non-categorized GO terms. In the prediction output, for each GO term probabilities were computed indicating how likely they were to be categorized into a specific category. The probability computation for the SVM in scikit-learn is implemented according to Platt [23] and Wu, Lin and Weng [36].

Fig. 1 shows an exemplary run for the GO term “nuclear chromosome segregation”. A flowchart of the algorithm and the pre-processing of the term “nuclear chromosome

Fig. 1.

Example run for the gene set “nuclear chromosome segregation”: During pre-training, a document vector comprising all the words of a GO term is fed into a shallow neural network, which predicts the masked words of the document. Once the neural network is trained, the output of the nodes in the hidden layer is used as the representation of the respective gene set, known as the embedding vector. These vectors reflect semantic similarity between gene sets and passed to a second machine learning model (e.g. SVM), which is trained and tested on manually categorized gene sets.

segregation” is shown in Supplementary Figure S2 and S1, respectively.

For visualization, the dimension of the document vectors was reduced from 200 to 2 dimensions using the t-distributed stochastic neighbour embedding (t-SNE) algorithm with default values [33] with a maximum of 5000 iterations. The resulting figures and all other figures were plotted with the R package ggplot2 [35], except for pie charts, confusion matrix and scatter plots, which were generated with plotly [29].

Two visualization modes were implemented for the pie charts. In the first mode, the number of gene sets are proportional to the size of the pie slice for their category. Thus, a category $i$ has a size of $s_i=\frac{n_i}{K}$, with $n_i$ being the number of gene sets in category $i$ and $K$ being the total number of gene sets in the list of enriched gene sets. The second mode is the “Gene Weight” mode. In this mode, the intersecting genes of the input query and the enriched gene sets were considered. Each intersecting gene of an enriched gene set was assigned to the category of the gene set. If a gene was assigned to several gene sets, weights for each category were calculated according to the number of gene sets with this category (details are provided in Supplementary Text S1). The corresponding mode is available depending on the uploaded data. If the uploaded dataset contains a column with genes associated with each of the gene sets (denoted as intersection), the Gene Weight mode can be activated.

Within the GUI, Slimformer also allows enriched gene sets to be displayed as a t-SNE dimensionality-reduced scatter plot. This plot is based on the manually categorized gene sets. Gene sets categorized by Slimformer are placed in the scatter plot by using a weighted mean of the five most similar, manually categorized gene sets. Similarity between these gene sets is determined by the cosine similarity of their embedded vectors. The weights for the weighted mean are equivalent to the computed cosine similarity. A finer-grained clustering of the embedded gene sets was implemented using Self-Organizing Maps (SOMs) with the kohonen R package [34]. The SOM consisted of 37 × 34 hexagons, which minimised the average distance between best matching units (BMUs). Neighbour joining of the SOM nodes yielded 53 clusters, with the optimal number of clusters determined by a silhouette analysis. The web interface of Slimformer was programmed with shiny [3] for R and exported with shinylive [28].

2.5. Analysis of transcription profiles of RSV-infected cells

As a case study, we investigated publicly available, normalized RNA-sequencing data (read counts) from Xu et al. Xu [37]. Xu et al. investigated immortalised human small airway epithelial cells (hSAECs) 24 h post-infection with RSV. Differential gene expression was determined between RSV-infected and mock-infected control samples using edgeR [27], followed by multiple testing correction using the Benjamini-Hochberg method. A gene was considered up- or downregulated if the log2-fold change (log2FC) was greater than or lower than 1 or −1, respectively, and if P < 0.05. g:Profiler2 was used to identify gene sets enriched in up- or downregulated genes. To avoid excessively broad or specific gene sets, enriched gene sets were retained only if they contained between 10 and 500 genes.

3. Results

3.1. Embedded documents cluster into categories

For each gene set, a document was assembled containing the name, description and list of the genes and names of the children and parents. Unnecessary words were removed, reducing the vocabulary of all documents from 44,950 to 32,629 words. To use these documents for the embedding, a numerical representation of the text was computed. The simplest form would be a document–term matrix (DTM) in which the value of a cell $i,j$ represents the count of word $j$ in document $i$. This method is fast to implement and easy to understand. However, it cannot convey semantic information, as two different words always have the same semantic distance, whereas they could have very similar or dissimilar meanings. A solution to this problem is the use of word or document embeddings. These embeddings attempt to capture the semantic similarities of words and documents. To compute these embeddings, an embedding-based NLP model was employed. To avoid overfitting while retaining sufficient information for class (category) separation, an embedding dimension of 200 was selected (an elbow plot is provided in Supplementary Figure S3). Fig. 2a shows the clustering results for the embedding vectors of the manually annotated gene sets. The gene sets of most of the categories grouped together. Notably, the larger categories clustered well, for example Morphogenesis/development, Cell cycle, Gene regulation, Metabolic process, and Immune system process. In contrast, some categories were more dispersed, such as Protein modification/signaling. This category was more challenging to separate, as its content overlapped with several other categories. For example, the gene set “protein phosphorylation” is a descendant of “metabolic process”, which may have pushed it towards the category Metabolic process. However, “protein phosphorylation” is a protein modification and was therefore manually categorized as Protein modification/signaling.

Fig. 2.

(a) T-SNE based scatter plot visualisation of the manually categorized gene sets. Distinct clusters for most categories are visible; (b) enlarged area from the left of the plot, as indicated in (a) showing gene sets related to interleukin production; (c) enlarged area from the right of the plot, as indicated in (a) showing gene sets related to nucleotide metabolism; (d) enlarged area from the upper part of the plot, as indicated in (a) in which gene sets categorized with Cell cycle and Morphogenesis/development overlap.

To investigate more fine-grained structures, several regions of the t-SNE plot were examined in greater detail. The cluster located to the very left contained mostly gene sets related to interleukins (Fig. 2b; a larger area is given in Supplementary Figure S4b). Additional discussion on these fine-grained structures is provided in Supplementary Text S2. Fig. 2c shows an enlarged area containing gene sets related to nucleotide synthesis (a larger area is given in Supplementary Figure S4c). A cluster representing a fraction of the cell cycle category formed close to Morphogenesis/development in the upper part of Fig. 2a. Detailed inspection of this region (Fig. 2d and Supplementary Figure S4j) revealed, for example “Cell proliferation involved in kidney development” and “Positive regulation of mesenchymal cell proliferation”. These gene sets are related to proliferating processes (cell cycle related) and developmental processes (morphogenesis and development related), making it difficult for the unsupervised learning algorithm to distinguish them as clearly as the manual curation suggested. Further enlarged areas are provided in Supplementary Figure S4d–k.

3.2. Learning the categorizations using classical machine learning

As shown in the last section, gene sets of most of the categories grouped together, encouraging the use of a machine learning pipeline to systematically learn the category assignments. The document embeddings were used to train and test Random Forests (RF), Logistic Regression (LR) classifiers, and Support Vector Machines (SVM). We next evaluated the performance of these models, particularly focusing on their behaviour with smaller annotated datasets to estimate the minimal amount of manual categorization required for accurate learning. For this, all different classifier methods were trained on the same training sets, following a cross-validation scheme with nine different training set sizes. Supplementary Table S3 shows the sensitivity of the different optimized hyperparameters for the classification task. While for RF none of their hyperparameters had a high impact on the performance, for LR models the regularization parameter and the type of penalty was highly relevant (0.55 and 0.41, respectively), and for SVM the choice of the kernel (0.57), regularization value (0.44) and kernel parameter gamma (0.42) had a substantial impact on model performance. LR and SVM achieved the best performance, with 84.2 % and 83.8 % mean balanced accuracy, respectively, when 90 % of the data were used for training (Fig. 3A). LR and SVM significantly outperformed RF (P = 2.981*10⁻⁵ for both models when all training set sizes were considered using a pairwise t-test), while no significant difference was observed between the LR and SVM. Using a reduced list of 1407 gene sets for training and validation (testing was then performed with the remaining 1407 gene sets), the SVM performed consistently above 82.2 % balanced accuracy. The null accuracy (random guessing) was 16.2 %, corresponding to the percentage of the largest category (Immune system process). The same performance pattern was observed when regarding F1-scores. LR and SVM again showed significantly better performance compared to RF (P = 0.00245 and P = 0.00048, respectively, when all training set sizes were considered using a pairwise t-test, Fig. 3B). Employing SVM, F1-scores were also calculated for each category separately. We observed a high performance for most categories (83.3 % of all classes were above 0.8). The lowest F1-score, besides Other, was 0.73 for “Receptor related process”. For the category Other we observed a weak performance (mean F1-score of 0.54) reflecting the large heterogeneity of this category (Supplementary Figure S5). An ablation test was performed for all three machine learning models by systematically removing information from the GO term documents such as information about the ancestors, the children, gene symbols, or the gene set description. After each removal, new embedding vectors were generated, and the models were optimized and fitted within the same procedure as for the models with full information. In general, we observed no major declines in performances across all models with these ablations. The SVM demonstrated the highest robustness to ablations. A significant drop in performance for the SVM, as well as for the LR and RF models, occurred when only GO term names remained in the documents. In addition, RF and LR showed already markedly reduced performances when only ancestor terms and gene symbols were removed (Supplementary Figure S6). For subsequent analyses, the best-performing SVM model was selected, as it showed slightly higher accuracy and lower variance, a better F1-score and greater robustness compared to LR and RF. The selected SVM model is denoted as “classifier” in the following.

Fig. 3.

Results of the different classification models for different training set sizes. (a) shows the mean balanced accuracy with standard deviation, (b) shows the mean F1-score with standard deviation. Linear regression classifiers and Support Vector Machines outperformed Random Forests.

Overall, the classification for all the different categories performed very well (Fig. 4, more details are provided in Supplementary Table S4). The category Other showed the worst performance (shown in Fig. 4 and Supplementary Figure S5), which may be due to low consistency in the gene sets categorized as Other. The confusion matrix reflects, to some extent, the clustering-based separation performance. For categories with minimal cluster overlaps (Fig. 2), the models performed better than for those with stronger overlaps. To assess model performance on an independent dataset, 300 additional gene sets (not used in the gold standard) were randomly sampled. Comparing the manual annotation with the machine learning predictions, we observed a balanced accuracy of 86.7 %, which was similar to the performance results obtained for the gold standard (83.8 %). Summarized results are shown in Supplementary Table S5, a complete list of the performance values for each category is provided in Supplementary Table S6. The F1-score was 87.8 %, also comparable to the F1-score from the gold standard (86.7 %). The category fractions were similar to those of the gold standard. Finally, all remaining gene sets of the aspect biological process were categorized. The complete list of categorized gene sets is given in Supplementary Table S7, which contains the id of each GO term (column Term_id), the name of each GO term (Term_name), the assigned category (Group), and a confidence score from the model (QC), or “+ ” indicating manual annotation. The confidence score corresponds to the probability assigned by the SVM that a GO term belongs to a specific category determined after Platt [23] and Wu, Lin and Weng [36]. To further resolve substructures within the embedding space, a clustering of the identified gene sets was performed based on Self Organizing Maps (SOM) leading to 53 finer clusters consistent with the local semantic relationships among gene sets.

Fig. 4.

Confusion matrix of the selected classifier showing the percentages of correctly categorized gene sets on the diagonal and the falsely categorized terms at all other entries. Supplementary Fig. S7 shows the exact count numbers.

3.3. Slimformer highlights differential gene regulation of immune response, metabolism and cell cycle in RSV-infected cells

As a case study, we applied our method to analyse published gene expression data of RSV-infected cells [37]. In total, 2275 genes were upregulated, and 2578 genes were downregulated following infection. Gene set enrichment analysis resulted in 503 and 267 gene sets enriched in up- and downregulated genes, respectively. The results from Slimformer (using the Gene Weight mode) are shown in Fig. 5. The categorized gene sets are listed in Supplementary Table S8. The largest fraction of upregulated genes (22 %) belonged to the category Immune system process, reflecting the strong innate immune response of the host cell. In contrast, Metabolic process represented the category with the largest fraction of downregulated genes (32.1 %), while also having a considerable fraction of 7.99 % of upregulated genes, followed by Cell cycle (27.9 %). Strong up- and downregulation was also detected in Cellular component organization (15.3 % and 17.0 %, respectively). Fig. 5c shows several smaller clusters related to Immune system process, and Fig. 5d shows clusters related to Metabolic process and Cell cycle.

Fig. 5.

Categories of gene sets enriched with upregulated (a) and downregulated (b) genes in cells 24 h post infection with RSV. Due to infection, the major upregulated genes were associated with Immune system process, followed by upregulation in Cellular component organization and Morphogensis/development. Most of the downregulated genes were associated with Metabolic process and Cell cycle. Categories Other and Protein modification/signaling contained 0.654 % and 0.0824 % of the downregulated genes, respectively. (c) t-SNE plots of upregulated and (d) downregulated gene sets. For upregulated genes, the two major categories, Immune system process and Morphogenesis/development formed broad clusters. Similarly, Cell cycle and Metabolic process formed larger downregulated clusters. Hovering over the dots representing the gene sets in the GUI reveals the names of the corresponding gene sets, as exemplified by Interferon-beta production and Centromere complex assembly in (c) and (d).

3.4. Qualitative comparison with other established tools

To evaluate the performance and categorization consistency of Slimformer relative to existing GO-slimming tools, we compared its results with three well established tools: CateGOrizer [11], Categorizer [21], and REVIGO [31]. In GO, gene sets exist in parent–child relationships, forming a hierarchical structure within a directed acyclic graph (DAG). This hierarchy is based on relationships such as is_a, part_of, has_part, and regulates, which define the connectivity and directionality between terms. Tools such as CateGOrizer, Categorizer, and REVIGO integrate gene sets into higher-order categories using this DAG structure to simplify the interpretation of long gene set lists. CateGOrizer relies primarily on structural relationships within the DAG, whereas Categorizer and REVIGO incorporate semantic similarity through an information content (IC)-based measure in addition to the DAG structure.

For comparison with CateGOrizer, a custom GO-slim definition was assembled by selecting GO terms most similar to our categories (listed in Supplementary Table S9). Of the 503 and 267 gene sets enriched with up- and downregulated genes, respectively, CateGOrizer associated 764 gene sets with 14 different categories (Supplementary Figure S8). The higher number of categorizations compared with the input query resulted from multiple assignments of single gene sets to several GO-slim terms. GO terms that could not be mapped by CateGOrizer were manually categorized as Other.

For the upregulated genes, the largest category in CateGOrizer was Response to stimulus (23.56 %), followed by Immune system process (16.75 %) (Fig. 6a), whereas Slimformer identified Immune system process as the most abundant category (22 %) and response to stimulus as considerably smaller (7.78 %). Notably, CateGOrizer did not associate any enriched gene set with Cell cycle, Neural related process, and Receptor related process. Terms such as “response to virus” and “defense response to virus” were categorized as Viral related process by Slimformer but distributed across Response to stimulus, Immune system process, and Stress response by CateGOrizer, highlighting the improved specificity of Slimformer for virological contexts.

Fig. 6.

Visual comparison of the results from Slimformer (a, b), CateGOrizer (c, d), Categorizer (e, f), and REVIGO (g, h). Upregulated (a, c, e, g) and downregulated (b, d, f, h) categories of the studied RSV-infected cells are shown for all tools. Higher-resolution versions are provided in Fig. 5 and in Supplementary Figs. S8-S10.

For the downregulated genes (Fig. 6b), CateGOrizer assigned the 267 gene sets to eight categories, while Slimformer identified seven. Cell cycle was notably underrepresented in CateGOrizer (11.08 %) compared to Slimformer (27.9 %), largely because several gene sets related to Cell cycle, such as “nuclear chromosome segregation”, “mitotic nuclear division”, “regulation of cell cycle phase transition”, “cell cycle checkpoint organization” and “negative regulation of cell cycle”, were not categorized into any GO-slim category by CateGOrizer, therefore categorized as Other. Similarly, CateGOrizer was unable to map several gene sets associated with amino acid metabolism into the category Metabolic process, such as “proteinogenic amino acid metabolic process” and “L-amino acid metabolic process” (see Discussion). In contrast, Slimformer categorized these appropriately. CateGOrizer categorized gene sets into ER/endosome/lysosome related process and Response to stimulus. Both categories were absent in the results of Slimformer as the according gene sets were categorized into the more general category Cellular component organization (see Discussion). Detailed numbers for CateGOrizer are provided in Supplementary Table S10.

Next, Slimformer was compared with Categorizer [21], which employs an IC-based semantic similarity measure. Similar to CateGOrizer, Categorizer identified a smaller fraction of upregulated gene sets for Immune system process (5.96 %), but larger fractions for Response to stimulus (13 %) and Stress response (9.23 %). Downregulation of Cell cycle was considerably underrepresented compared to Slimformer (3.57 % versus 27.9 %). The corresponding pie charts are provided in Fig. 6d/e and, in more detail in Supplementary Figure S9. Finally, Slimformer was compared with REVIGO [31], which also uses IC-based semantic similarity but groups gene sets into clusters and superclusters via hierarchical clustering. REVIGO identified 15 superclusters for upregulated and 12 for downregulated genes (Supplementary Figure S10). For upregulated genes, the largest supercluster was Response to virus. However, inspecting the gene sets within this category, many gene sets were related to Immune system process (“canonical NFκB signaling”, “wound healing”), or Response to stimulus (“response to biotic stimulus”, “response to tumor necrosis factor”), rather than specifically to viral infection. The second largest supercluster, regulation of inflammatory response, also included immune-related and viral processes (“positive regulation of cytokine production”, “negative regulation of viral processes”). As the authors of REVIGO note, these superclusters are loosely defined, making direct quantitative comparison with Slimformer challenging. The supercluster results of REVIGO are listed in Supplementary Table S11.

3.5. Access and usability of Slimformer

Slimformer can be accessed via a web interface at https://konig-lab.github.io/Slimformer. The user can upload the results of any gene set enrichment analysis as a CSV or TSV file, as long as a column with GO IDs is provided and optionally a column with genes for the Gene Weight mode. The web interface provides the categories for the uploaded GO-terms within a downloadable CSV file and downloadable plots of the pie charts and t-SNE plots.

4. Discussion

As shown in the data of RSV-infected cells, a single gene set enrichment analysis can easily yield hundreds of significantly enriched gene sets, making it difficult to get an overview of the expression changes. Several excellent tools have been developed to summarise results into higher-level categories. However, they do not take advantage of semantic similarity derived from textual descriptions. With Slimformer, we trained an embedding-based NLP model to obtain feature vectors that capture intrinsic similarities between terms based on their descriptions, names, associated genes, and the names of their children and parent terms.

Visualization of the model embeddings (Fig. 2) revealed that the categories Immune system process, Morphogenesis/development, Metabolic process, and Cell cycle formed the most distinct clusters, highlighting a pronounced polarisation between these four categories. This is in line with observations in cancer research, where highly proliferating cancer cells with strong cell cycle activity often exhibit reduced morphological differentiation. This polarity is often used in tumour grading systems, for instance in the Gleason score for prostate cancer [7]. Furthermore, a clear opposition between Immune system process and Metabolic process was observed. This is biologically reasonable, as we, the curators of the categorization for the gold standard, annotated gene sets as Metabolic process if their definitions and names described clearly cellular biochemical reactions, whereas gene sets categorized as Immune system process were more related to signaling, or receptor or regulatory processes associated with immune responses. In contrast, the categories Immune system process, Response to stimulus, and Stress response were much more difficult to separate as they are often functionally related. We addressed this overlap through a prioritisation scheme that assigned ambiguous gene sets primarily to Immune system process.

Even within these broader clusters, Slimformer captured fine-grained biological similarity. Gene sets related to interferon responses clustered close to those describing interleukin responses, while in Metabolic process, gene sets associated with purine and pyrimidine biosynthesis and processes formed distinct subclusters. In summary, learning gene set descriptions and annotations through embedding let to a semantically coherent vector space, reflecting underlying biological relationships. Nevertheless, it was not sufficient for the categorization of all terms accurately. To address this, we applied supervised machine learning, based on the embedding vectors as features and the categories from manual annotation of the gene sets as class labels for benchmarking. Support Vector Machines (SVM) achieved the best performance, reaching on average 82.4 % balanced accuracy and an F1-score of 0.867 with the largest training set. With about half the training data the performance remained high, at 82.3 % and an F1-score of 0.835. Logistic Regression and SVM models outperformed Random Forests, consistent with the predominantly linear relationships captured by the embedding. Decision trees are the basic components in Random Forests, which separate the search space only paraxially, making it infeasible to solve a linear problem in general. Random Forests, which aggregate decision trees, solve this problem. However, their stepwise approximations of continuous functions can lead to overfitting. Viewing at the F1-score, the SVM performed better than the Logistic Regression, and in the ablation test the SVM showed higher robustness compared to the other two models. Therefore, SVM were selected for all subsequent analyses. The SVM could effectively resolve ambiguities between Cell cycle and Morphogenesis/development as visualised in Fig. 4.

When applying Slimformer to gene expression data of RSV-infected cells, Immune system process was the most upregulated category, followed by Cellular component organization and Morphogenesis/development, indicating major cellular restructuring during infection. This observation is consistent with Rajan et al. Rajan [25], who reported extensive morphological changes in RSV-infected human cells due to actin filament reorganization. Similarly, eight of the 46 gene sets categorized as Cellular component organization were related to actin, and 14 were associated with cell shape and protein localization (Supplementary Table S8). Among the gene sets categorized as Cellular component organization, two were related to the formation of autophagosomes, which have been shown to contribute to RSV budding [20], [4]. In contrast, metabolism was strongly downregulated, particularly in purine and pyrimidine biosynthesis, but also mitochondrial processes. The downregulation of nucleotide biosynthesis is a key restriction for the cell cycle, which was the second largest category of downregulated genes categorized by Slimformer. Gibbs et al. reported that RSV infection induced TGF-β signaling in epithelial cells, leading to cell cycle arrest, and enhancing RSV replication [9]. Similarly, Bian et al. reported that RSV-induced cell cycle arrest and enhanced virus replication [2]. In summary, downregulation of the cell cycle suggests a strong means for RSV to reprogram the host cells for its needs.

We further compared Slimformer with CateGOrizer, Categorizer and REVIGO. First, we did not recognize any runtime issues for all tools. Once learned, Slimformer bases on a lookup table, which access is fast. The other tools assign gene set categories by climbing up the Gene Ontology graph, which is also not computationally expensive. Regarding the results, for upregulated genes, CateGOrizer identified Response to stimulus (23.6 %) and Immune system process (16,8 %) as major categories, whereas Slimformer prioritised Immune system process (22 %) and reduced Response to stimulus (7.8 %). This difference likely reflects the training of Slimformer, which emphasised processes related to immune responses during annotation. Slimformer also produced a higher representation for Cellular component organization (15.3 % versus 6.68 % in CateGOrizer). Gene sets, such as “positive regulation of cellular component biogenesis“, “lipid localization“ and “regulation of actin filament-based process“, were categorized as Cellular component organization by Slimformer but not by CateGOrizer. This is due to CateGOrizer relying solely on hierarchical traversal within the GO graph. Therefore, such terms lack direct ancestral links to the selected GO-slims (Supplementary Figure S11–13). Similar discrepancies were observed for Cell cycle. Slimformer categorized this as the second largest downregulated category with 27.9 %, while CateGOrizer reported only 11.08 %. Several gene sets which were categorized by Slimformer didn’t receive any category from CateGOrizer (thus assigned to Other). These included “chromosome segregation”, “nuclear chromosome segregation”, “regulation of cell cycle phase transition”, “negative regulation of cell cycle”, and “positive regulation of cell cycle”, because CateGOrizer does not consider the relations “Part_of”, “Regulates, “Positively_regulates” and “Negatively_regulates”. We therefore recommend that such relationships be integrated into future versions of CateGOrizer. Slimformer did not assign any enriched gene sets to the category ER/endosome/lysosome related process, whereas CateGOrizer included terms like “actin filament organization”, “positive regulation of organelle organization”, “regulation of actin cytoskeleton organization”, and “vacuole organization”. Slimformer included these gene sets in Cellular component organization. CateGOrizer also included them into Cellular component organization, providing a double categorization. During manual curation, we annotated such terms into the more general category Cellular component organization, making the more specific category ER/endosome/lysosome related process redundant for gene sets with multiple associated categories. Categorizer showed a similar underrepresentation of Immune system process for the upregulated genes and of Cell cycle in the downregulated genes, despite incorporating semantic similarity, indicating that its information-content-based approach does not fully overcome the limitations of the directed acyclic graph of GO and its subsequent hierarchical traversal. REVIGO turned out not to address the same purpose as Slimformer, CateGOrizer, and Categorizer. While valuable for forming smaller clusters following a greedy procedure, REVIGO generated a large number of clusters. When forming superclusters, the distinct subgroups lacked functional resolution. Besides the qualitative comparison, we also performed a quantitative comparison between the three different tools, and compared the fraction of gene sets which could not be assigned to any category (for Slimformer represented by the category Other). Again, tested on the RSV data, Slimformer assigned only 2.31 % of upregulated and 0.654 % of downregulated gene sets to Other. In contrast, CateGOrizer didn’t categorize 15.7 % of the upregulated and 24.3 % of the downregulated gene sets. Categorizer, which operates at the gene rather than gene-set level, didn’t categorize 11.2 % and 28.0 % of up- and downregulated genes, respectively. REVIGO does not provide such non-categorized gene sets, but rather assigns subclusters consisting of only one gene set. We found 22.5 % (upregulated) and 24.7 % (downregulated) of such single-member subclusters. In summary, Slimformer allowed a much higher coverage of category annotation, compared to CateGOrizer, Categorizer and REVIGO. The latter methods rely only on the structure of the directed acyclic graph. By contrast, Slimformer leverages semantic similarity and contextual information, substantially reducing the number of gene sets that remain uncategorized.

The current version of Slimformer was trained with categories for viral research. It is ready to use for this field and provides categorized gene sets, pie charts, and t-SNE plots for these categories. The set categories may require adaptation for other research areas, such as aging or cancer, where more granular subdivision of cell cycle or morphogenesis/development may be relevant, while the category of viral related processes may be of less interest. Once having set up tailored categories and a benchmarking annotation set for them, Slimformer can be straight away applied and trained using these user-defined annotated gene sets. Similarly, it can be extended to other GO aspects and databases such as REACTOME [19] or KEGG [15]. Due to Slimformer relying on textual and relational consistency within the GO database, retraining will be necessary after major GO updates to maintain accuracy. Slimformer currently only accepts gene sets of GO Biological Process as input. Further developments will focus on the addition of GO Cellular Component and GO Molecular Function with tailored categories, and gene set definitions from other resources as mentioned above, retraining across multiple species, and using other methods to embed the GO-terms, such as transformer or large language models.

Declarations

none

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Supplementary information

Supplementary data are available at Computational and Structural Biotechnology Journal online.

Funding

This work was supported by the Bundesministerium für Forschung, Technologie und Raumfahrt, Germany [Fkz: 13N15711, Fkz: 13N15745], and the European Union [GA No 101137311]. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the EU or the European Commission. Neither the EU nor the granting authority can be held responsible for them.

CRediT authorship contribution statement

Rainer Koenig: Writing – original draft, Supervision, Conceptualization. Jonas Marx: Writing – original draft, Visualization, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Alicia Hiemisch: Visualization, Methodology. Fionn Daire Keogh: Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis, Conceptualization.

Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work the author(s) used ChatGPT 5.1 in order to assist with language editing. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article.

Declaration of Competing Interest

The authors declare that they have no competing interests.

Appendix A. Supplementary material

Supplementary material

Supplementary material

Supplementary material

Supplementary material

Supplementary material

Supplementary material

Availability and implementation

Slimformer can be accessed via a web interface at https://konig-lab.github.io/Slimformer