Skip to main content
NCBI home page
Glycobiology logoLink to Glycobiology
. 2025 May 22;35(7):cwaf030. doi: 10.1093/glycob/cwaf030

GlycoSiteMiner: an ML/AI-assisted literature mining-based pipeline for extracting glycosylation sites from PubMed abstracts

Robel Kahsay 1,✉, Urnisha Bhuiyan 2, Cyrus Chun Hong Au 3, Nathan Edwards 4, Luke Johnson 5, Sujeet Kulkarni 6, Karina Martinez 7, Rene Ranzinger 8, K Vijay-Shanker 9, Jeet Vora 10, Kate Warner 11, Michael Tiemeyer 12, Raja Mazumder 13
PMCID: PMC12130968  PMID: 40401984

Abstract

Over 50% of human proteins are estimated to be glycosylated, making glycosylation one of the most common post-translational modifications (PTMs) of proteins. A glycoinformatics resource such as the GlyGen knowledgebase, consisting of experimentally verified sequence-specific glycosylation sites, is critical for advancing research in glycobiology. Unfortunately, most experimental studies report glycosylation sites in free text format in scientific literature, mentioning gene names and amino acid positions without providing protein sequence identifiers, making it difficult to mine reported sites that can be mapped onto specific protein sequences. We have developed GlycoSiteMiner, which is an automated literature mining-based pipeline that extracts experimentally verified protein sequence-specific glycosylation sites from PubMed abstracts. The pipeline employs ML/AI algorithms to filter out incorrectly identified sites and has been applied to 33 million PubMed abstracts, identifying 1118 new sequence-specific glycosylation sites that were not previously present in the GlyGen resource.

Keywords: glycosylation, GlyGen, MLP, PubTator, SVM

Introduction

Glycosylation plays a variety of important roles in structural, signaling, recognition, and many other cellular mechanisms (Dennis et al. 1999; Varki 2017; Varki et al. 2022). One of the major types of glycosylation is N-linked where an N-acetylglucosamine (GlcNAc) residue is attached by an amide bond to an asparagine residue at the beginning of a protein sequence motif NX(C/S/T), where X can be any amino acid except proline. The presence of this sequence motif is required for N-glycosylation, however not all NX(S/T) motifs present in a protein sequence are occupied. Another major type of glycosylation is O-linked glycosylation which occurs at any serine or threonine residue. A single protein sequence can have several glycosylation sites, and each of these sites can be occupied by many distinct glycan structures resulting in diverse glycoforms of the protein sequence. For example, a single protein sequence with 4 glycosylation sites can have a total of 256 glycoforms if each of these sites can be occupied by 4 glycans.

Identification of protein glycosylation sites that are actually occupied by glycans is performed using experimental methods such as Mass Spectrometry (MS) (Park and Lebrilla 2005) using matrix-assisted laser desorption/ionization (MALDI) and electrospray ionization (ESI) (Harvey 2009; Delafield and Li 2021). The glycosylated protein sites discovered by these experimental glycoprotein analysis studies are usually reported in the abstract and full body text in a free text form mentioning gene names and amino acid positions without providing protein sequence identifiers needed for mapping these sites onto specific protein sequences. Integrating reported glycosylation sites that are mapped to protein sequences into glycoinformatics databases, such as the GlyGen knowledgebase (Kahsay et al. 2020; York et al. 2020), is essential for linking glycosylation sites with protein functions and enriching other types of biological annotations. If done manually by biocurators, this extraction task can be time-consuming and daunting, and an automated text-mining approach is needed.

There have been many reported text mining approaches used to extract post-translational modification (PTM) sites. The protocol published recently (Arumugam et al. 2022) reviews many text mining techniques used to extract PTM information from scientific literature such as eFIP (Tudor et al. 2012), RLIMS-P (Torii et al. 2015), MPTM (Sun et al. 2017) and SVM-based phosphorylation site extractor (Xu et al. 2012; Raja and Natarajan 2018). The iPTMnet (Huang et al. 2018) PTM integration resources use text mining tools RLIMS-P and eFIP which extract phosphorylation sites from PubMed articles. The PPTM web server (Wang et al. 2014) uses dependency parse trees for mining protein phosphorylation. All these tools with the exception of MPTM extract protein phosphorylation information. The MPTM text mining tool extracts information about 11 PTM types from PubMed abstracts using parse trees and a heuristic algorithm. The work reported on detecting mention of protein glycosylations (Shukla and Jayaraman 2012) using dictionary and rule-based approaches to classify glycosylation-related abstracts from others, and is not related to extraction of sequence-specific glycosylated protein sites.

There are also several BioBERT (Lee et al. 2020) based pipelines that were successfully employed to extract PTM information on a large scale, leveraging confidence calibration to improve precision and recall. For example, distantly supervised and confidence-calibrated BioBERT has been used to extract protein–protein PTM extraction (Elangovan et al. 2022). The BioBERT model has been fine-tuned for biomedical relation extraction (Su and Vijay-Shanker 2022) and used to review abstracts in medical research (Masoumi et al. 2024). Text mining tools have also been used for curating PTM annotations (Veuthey et al. 2013; Ross et al. 2016). Most of these published tools focus on extracting phosphorylation information, and those tools that extract glycosylation site information identify only high-level glycosylation site information and not protein sequence-specific glycosylation sites. In other words, these tools provide gene or protein name entities and glycosylation site entities but do not attempt to map extracted glycosylation sites to positions on specific protein sequences.

In this study, we developed GlycoSiteMiner which is a novel literature mining-based pipeline for extracting glycosylation sites from PubMed abstracts. Unlike existing tools that focus on extracting PTM types or protein-level annotations, GlycoSiteMiner’s emphasis is on the identification of protein sequence-specific glycosylation sites by resolving ambiguities in gene-to-protein mappings using AI based filtering layer. This targeted and integrated approach complements existing efforts and demonstrates how leveraging supervised machine learning in tandem with text-mining pipelines can enhance the precision of PTM-specific information from the literature. The pipeline consists of seven stages: entity extraction using a Named Entity Recognition (NER) model, entity extraction using a Large Language Model (LLM), entity mapping, match scoring, feature generation, supervised machine learning-based filtering, and LLM-based verification. The entity extraction process employs a spaCy-based NER model (Honnibal and Montani 2017) and the GPT-4o Mini LLM (openai.com) to identify glycosylation site entities. To reduce errors, only sites extracted by both models are considered. In the entity mapping stage, species and gene entities are linked to protein sequence identifiers, ensuring sequence-specific glycosylation site annotation. The final two stages filter out incorrect sequence-specific matches that may have been introduced earlier in the process. Applied to 33 million PubMed abstracts, the pipeline successfully identified 1118 novel sequence-specific glycosylation sites that were not previously included in the GlyGen resource.

Results

Site entities extracted using NER and LLM models

In the first stage of the pipeline (Fig. 1), the NER model processed 33 million PubMed abstracts and identified site entities in 9311 abstracts. These abstracts were then used in the second stage, where site entities were extracted using the GPT-4o Mini LLM. The NER model was designed to recognize site entities without species restrictions, while the GPT-4o Mini LLM can recognize both site and species entities and was restricted to 12 GlyGen reference species: Human, Mouse, Rat, Pig, Chicken, Fruit fly, Yeast, Cellular slime mold, SARS-CoV, SARS-CoV-2, HCV-Japanese, and HCV-H77. To minimize errors introduced during this stage, only the 6650 site entities identified by both models were considered.

Fig. 1.

Fig. 1

Open in a new tab

Pipeline stages. Site entity extraction using NER model: Glycosylation site entities are found in 9311 out of 33 million abstracts using named entity recognition (NER) model. Site entity extraction using LLM model: The 9311 abstracts found to contain site entities are given to the GPT-4o mini model to extract site entities. Entity mapping: Extracted site entities are mapped to UniProtKB sequences to generate sequence-specific sites. Match scoring: Sequence-specific sites are scored to generate match scoring table showing match sites and mismatch sites depending on whether the amino acid on the UniProtKB sequence matches or mismatches the amino acid on the extracted site. Feature generation: Features generated for each match site. ML-based classifiers: Trained classifiers are applied to filter out incorrect match sites. LLM-based verification: The GPT-4o mini model is used again for automated verification of match sites that passed the ML-based filtering.

Sequence-specific match sites

In the entity mapping stage of the pipeline, the 6650 site entities extracted by the NER and LLM models are mapped to 7147 UniProtKB sequences, yielding a total of 36,975 sequence-specific sites. The match scoring stage then evaluates these sites by assigning a score of 1 or − 1, depending on whether the amino acid in the UniProtKB sequence matches or mismatches the amino acid in the extracted site. Since this stage introduces potential errors in mapping gene entities to protein sequence accessions, some of the 3676 match sites are expected to be incorrect. In other words, even if the NER and LLM models correctly extract a site entity, it may still be mapped to an incorrect protein sequence, and any apparent amino acid match may occur by chance. Fortunately, these errors can be systematically mitigated using a supervised machine learning approach.

Sequence-specific match sites passing filtering and verification

To eliminate incorrect match sites introduced during the entity mapping stage, we trained and validated supervised machine learning classifiers using two different algorithms. Our model evaluation, based on a subset of 783 known correct match sites and 363 known incorrect match sites, demonstrates that both classifiers perform effectively in identifying correct matches. Results from our 10-fold cross-validation experiment, which generated ROC curves, Precision-Recall curves, and confusion matrices (Fig. 2A–C), further support this. As shown in the confusion matrix (Fig. 2C), the recall rates are 88% for the SVM and 87% for the MLP, with both classifiers achieving a precision of 91%. Please note that precision and recall are calculated as Tp/(Tp + Fp) and Tp/(Tp + Fn) respectively where Tp is true positives, Fp is false positives and Fn is false negatives given in the confusion matrix. Overall, the accuracy performance metrics for both models are highly comparable.

Fig. 2.

Fig. 2

Open in a new tab

Model validation. A) ROC curves, B) precision-recall curves, C) confusion matrices, and D) tuned decision thresholds for the two supervised machine learning algorithms used. Overall, the plots show the MLP classifier having slightly better performance compared to the SVM classifier.

The decision threshold for both classifiers was fine-tuned to maximize balanced accuracy. As shown in Fig. 2(D), this optimization resulted in threshold values of 0.7260 for the SVM classifier and 0.8702 for the MLP classifier. Applying these thresholds, 2219 match sites successfully passed the SVM or MLP classifier filters. The number of match sites that passed the classifier filters for 8 GlyGen reference species are given in Table 1. Beyond supervised ML-based filtering, we employed the GPT-4o Mini LLM to further verify the accuracy of these 2219 sequence-specific match sites. By prompting the model with a specific yes-or-no question, the LLM confirmed 1745 (79%) of them as accurate. To assess the reliability of the LLM-based verification, we conducted a manual review of 103 randomly selected match sites from 20 abstracts and found that 98 (95%) were accurate. With this level of confidence, we integrated 1118 of the 1745 sequence-specific match sites that were not previously included in the GlyGen resource. The match sites that did not pass the supervised ML filtering and LLM verification will serve as a priority list for manual curation efforts planned by GlyGen in collaboration with the GlySpace Alliance (Aoki-Kinoshita et al. 2020; Lisacek et al. 2023).

Table 1.

Number of sequence-specific match sites for 8 out of 12 GlyGen reference species passing the ML based filters.

Species SVM MLP SVM or MLP LLM verified
Human 1626 1398 1626 1294
Mouse 277 191 277 196
Rat 142 104 142 108
SARS-CoV-2 69 59 69 63
Yeast 27 27 27 23
Pig 35 16 35 31
Fruit fly 30 30 30 20
Chicken 13 7 13 10
Total 2219 1832 2219 1745
Open in a new tab

Discussion

In this study, we developed a pipeline to mine protein sequence-specific glycosylation sites mentioned in PubMed abstracts. The process begins by identifying abstracts that reference protein sites, such as “Asn-247” or “N247,” in various formats, along with glycosylation-related terms like “glycosylated” or “glycosylation.” These abstracts are then processed by NER and LLM models to extract site entities, which are subsequently mapped to UniProtKB protein accessions corresponding to specific protein sequences. The extracted sequence-specific sites are classified as either match or mismatch sites based on whether they align with the corresponding protein sequence. A sequence-specific match site is represented by a combination of key values, including the PMID, UniProtKB accession, amino acid, and position number (e.g. “19369259|P16671|Asn|247”). This representation serves as a claim generated by the pipeline, indicating that the protein sequence identified by “P16671” has been studied and that the asparagine (“Asn”) residue at position “247” has been glycosylated.

We recognize that the sequence-specific match sites identified by the pipeline may contain errors for several reasons. First, during the entity extraction stage, the NER and LLM models may incorrectly identify “Asn-247” as a glycosylation site for the gene “CD36.” One common source of error occurs when an abstract refers to the site as putative or potential, meaning it is not a confirmed glycosylation site reported in the study. Second, errors can also arise in the entity mapping stage. For example, the gene term “CD36” may have been incorrectly mapped to the protein sequence identifier “P16671,” or it may be associated with multiple protein sequences, with “P16671” not being the one studied. To minimize these errors, we implemented two filtering stages designed to refine the extracted data. Our results demonstrate that the sequence-specific match sites that pass these filters exhibit a high level of accuracy.

The GlyGen resource contains 47,640 protein sites glycosylated with known glycan structures, and 52,371 sites without known glycan structures. Even though the volume of new sequence-specific sites identified by this pipeline is relatively small (1118 compared to 52,371), we consider the addition to be very valuable for the growth of the resource. Most of the sites without glycan structures (50,450 out of 52,371) in GlyGen come from UniProtKB, and 46,563 of them are N-glycosylation sites. We analyzed the site sequence features of these N-glycosylation sites from UniProtKB and found out that only 105 occur at non-consensus motif (not NXS/T). On the other hand, 785 of the 1118 sites identified by GlycoSiteMiner are N-glycosylation sites, and 137 of them occur at non-consensus motif (not NXS/T). As indicated in the UniProtKB annotation website (https://www.uniprot.org/help/carbohyd), the UniProtKB annotation includes predicted sites of N-glycosylation identified by the “NetNGlyc” predictor (Gupta and Brunak 2002) or the PROSITE pattern PS00001 (Sigrist et al. 2013) which are always on consensus motif, and this is one of the reasons for having small proportion of N-glycosylation sites on non-consensus motif. Moreover, consensus motifs are widely recognized and have historically been the focus of both experimental studies and biocuration efforts. As a result, databases such as UniProtKB can be biased toward annotating canonical motifs, which can lead to an underrepresentation of non-consensus variants. In contrast, our approach does not rely on predefined consensus patterns. Consequently, it is not unexpected that our method identifies a higher proportion of non-consensus motifs, many of which may represent biologically relevant but previously unannotated sites in databases.

Due to the stringent filtering stages, we recognize that the pipeline may be discarding a significant number of accurate sequence-specific match sites. Developing a more refined filtering mechanism remains a key focus for future work. Additionally, there is room for improvement in the entity extraction stage, particularly in the performance of the LLM model. Exploring more advanced (though potentially more expensive) LLMs or developing a customized local model tailored for higher extraction accuracy could further enhance the pipeline’s effectiveness.

Materials and methods

Site entity extraction stages

We downloaded XML files containing around 33 million PubMed abstracts from the NCBI ftp site (https://ftp.ncbi.nlm.nih.gov/pubmed/). In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is one of the subtasks involving the identification of specific entities like people, places, organizations, dates, and quantities within a text. The spaCy open-source library, which is a Python-based library featuring fast statistical NER as well as an open-source named-entity visualizer was used to build a NER model that will recognize glycosylation site entities in sentences from PubMed abstracts. The NER model was built starting with an empty spaCy model and adding compiled patterns for protein site and glycosylation terms shown in Table 2. The protein site patterns are given to spaCy in the form of a regular expression with one letter, three letters, and full name amino acid representations followed by a hyphen and a digit. For glycosylation term patterns, we collected an exhaustive list of glycosylation phrases also shown in Table 2. The NER model is then used to recognize glycosylation site entities in each sentence from the 33 million PubMed abstracts resulting in 9311 PubMed abstracts containing both protein site and glycosylation term entities.

Table 2.

Compiled patterns used for the extraction of protein sites and glycosylation terms from around 33 million PubMed abstracts.

Pattern type Pattern list
Protein site [aa_full]-[digit], [aa_three]-[digit], [aa_one]-[digit]
Glycosylation term glycosylation, glycosylated, glycan, N-glycosylation, N-glycosylated, N-glycan, N-glycans, N-linked, N-Linked, N-glycopeptides, N-glycoproteins, N-glycosidic, n-oligosaccharide, n-oligosaccharides, O-glycosylation, O-glycosylated, O-glycan, O-glycans, O-linked, O-Linked, o-fuco-oligomers, O-fucosidically, O-fucosylated, O-fucosylation, O-Fucosylation, O-fucosyltransferase, o-fuco-xylo-glucurono-mannan, o-galloyl-beta-d-Glucose, O-galloyl-beta-D-glucose, O-GlcNAc, O-GlcNAc-Akt1, O-GlcNAcase, O-GlcNAc-deficient, O-GlcNAc-dependent, O-GlcNAc-modification, O-GlcNAc-modified, O-GlcNAc-modifies, O-GlcNAc-occupied, O-GlcNAc/phosphate, O-GlcNAcylated, O-GlcNAcylation, O-GlcNAcylation-defective, O-GlcNAcylation-dependent, O-GlcNAcylationmediated, O-GlcNAcylation-mediated, O-glycosidic, O-glycosidically, O-β-linked
Open in a new tab

The abstracts identified by the NER model were also given to the GPT-4o Mini Large Language Model (LLM) to extract site entities with the prompt – “Extract gene entity, protein entity, glycosylation sites and organism from the following text: <abstract_text>”. The GPT-4o Mini LLM is a compact and cost-effective version of the GPT-4o model designed to balance performance with affordability offering advanced language processing capabilities suitable for various applications.

Entity mapping stage

When protein glycosylation sites are reported on PubMed abstracts, gene or protein names are mentioned, but protein sequence identifiers are rarely given. However, it is a requirement for our pipeline to identify the sequences used in the studies and verify that the reported sites exist on the sequences identified. To achieve this, we used gene and species annotations from the PubTator resource (Wei et al. 2019) and then mapped gene entities to UniProtKB (UniProt 2023) protein accessions associated with sequences. The PubTator resource provides automated or text-mined bioconcept annotations from state-of-the-art text mining systems for genes/proteins, genetic variants, diseases, chemicals, species, and cell lines. The annotations for gene entities include Entrez gene identifiers as part of the annotation, and species annotations contain NCBI taxonomy identifiers (Sayers et al. 2022). It is possible that this mapping of gene entities to Entrez gene identifiers can be one to many. For example, the gene entity “VEGFRs” in the abstract for PMID “23,878,260” is mapped to Entrez gene identifiers “2324” and “3791.”

The Entrez gene identifiers are mapped to UniProtKB protein accessions using mapping datasets retrieved from the GlyGen resource, and we find that “2324” and “3791” map to “P35916” and “P35968” respectively. Here again, the mapping between Entrez gene identifiers to UniProtKB protein accessions can be one to many. Out of the 9311 abstracts containing both protein site and glycosylation term entities, 5917 (64%) of them also contain PubTator gene entities. This means PubTator fails to identify gene entities in 36% of the abstracts that contain both protein site and glycosylation term entities. For example, PubTator fails to recognize “CD36” as a gene entity in the abstract for PMID “19,369,259” which reports three glycosylation sites “Asn-247,” “Asn-321” and “Asn-417.”

To reduce this gap, we implemented a simple dictionary-based gene entity recognizer that takes gene names and synonyms from GlyGen gene name datasets and tries to find them in the 9311 PubMed abstracts. The gene entities “CD36” mentioned in PMID “19,369,259” which were missed by PubTator is identified by this mapping since the entities are mapped to UniProtKB accessions “P16671,” “Q3HUX1” and “Q27367” in the GlyGen gene name datasets. This process of recognizing additional gene entities increased the number of abstracts containing gene entities to 7048 (76%). Since the GlyGen resource is restricted to a set of 12 species including Human, Mouse, Rat, Pig, Chicken, Fruit fly, Yeast, Cellular slime mold, SARS-CoV, SARS-CoV-2, HCV-Japanese and HCV-H77, papers describing data from other organisms cannot be processed using approach based on GlyGen data.

Match scoring stage

In general, for a given abstract, we have a total of N x M sequence-specific sites where N is the number of UniProtKB protein sequence accessions and M the number of extracted sites. We denote S = [s1, s2, …, sN] to be the set of UniProtKB sequences and P = [p1, p2, …, pM] to be the set of extracted sites representing in a total of N x M possible sipj sequence-specific sites. In the example abstract for PMID “19,369,259,” these two sets are S = [“P16671,” “Q3HUX1,” “Q27367”] and P = [“Asn-247,” “Asn-321,” “Asn-417”]. The sipj sequence-specific sites can be match sites or mismatch sites depending on whether the amino acid on the UniProtKB sequence si matches or mismatches the amino acid on the extracted site pj. An example of a match scoring table for abstract with PMID “19,369,259” is given in Fig. 1 showing 1 and − 1 values for match and mismatch sites respectively.

If sipj is a mismatch site, it means either the recognized protein site pj is incorrect or si is not the sequence used in the study. On the other hand, if sipj is a match site, even if not guaranteed, it is possible that sipj is a correct match site. In general, we can collect a set of K match sites with the possibility that some of them can be incorrect.

Feature generation stage

As described earlier, sequence-specific match site sipj represents a protein site pj on sequence si. Building ML model that can filter out incorrect sequence-specific match sites using supervised machine learning framework requires design and generation of features associated with each match site. To generate features associated match site sipj, we define a set of sibling sites [sipk] where k ∈ [1, L] with k ≠ j and L is the largest possible value of M (10 in this study). We consider the match scores for sipj and [sipk] given in match scoring table of to be the first set of features as shown in the features table (columns F1 to FL) of Fig. 1. The logic behind this design is the fact that if most of the sibling sites are also match sites, the sipj match site is most likely to be a correct sequence-specific protein site. The next two features associated with match site sipj, columns FO and Fs, are related to organisms mentioned in the abstract, and the number of sequences found to be matching the site. The feature FO takes a value of 1 if the organism or species corresponding to sequence si has been mentioned in the abstract. If the abstract does not mention any organism, the value for this feature is set to 0. Finally, the value is set to −1 if one or more organisms have been mentioned in the abstract and none of them matches the species corresponding to sequence si. The feature related to the number of sequences, Fs, is set to 1 or − 1 depending on whether the number of sequences matching the site is one or many.

The best way to have a clear understanding of this process is to give an example that shows how the first row in features table corresponding to match site “P16671|Asn-247” is populated. The match site “P16671|Asn-247” has a match score of 1 (column F1) and each of its siblings “P16671|Asn-321” and “P16671|Asn-417” also has a match score of 1 (columns F2 to F3). Since match site “P16671|Asn-247” has only two sibling sites, values for columns F4 to FL are set to 0. Next, the value for column FO is set to 1 since the organism corresponding to sequence “P16671” is mentioned in the abstract. Finally, the value for Fs is set to −1 since there are more than one sequences matching the site “Asn-247.”

Supervised ML based filtering stage

Unsupervised and supervised learning are the two major types of machine learning tasks. The task of unsupervised learning is to discover patterns and relationships in the data without any explicit guidance or training data. On the other hand, in supervised learning, the task is to learn a function or model that maps an input to an output based on training sample input–output pairs. The training data used in supervised learning is also referred to as labeled data because the output or label is known for every input data point. One of the most common supervised machine learning tasks is classification where a model is trained to map input data into output classes or categories. In supervised learning, we need to evaluate the prediction accuracy of the trained model before using it for prediction.

Typically, this is done by performing a 10-fold cross validation experiment where available labeled or input–output pairs are divided into 10 partitions. During the ith validation experiment, the ith partition is set aside for testing and the model is trained on the remaining 9 partitions. The trained model is then used to predict the class for each input in the test set. Since the actual class labels are also known for each input in the test set, we can find out the percent of input data points in the test set that were predicted correctly. The accuracy found during the 10 experiments can be summarized and the model accuracy can be reported with that value. If the 10-fold cross-validation experiment yields satisfactory prediction accuracy (small enough false positive and false negative rates), the model can be trained using all labeled or training data and is ready to make predictions on new input data that is not labeled, and the resulting predictions are assumed to have false positives and false negatives as reported during model validation. We can also go one more step to verify the predictions using the same approaches that were used to generate the labeled training dataset. In this study, the aim is to train supervised classifiers that can map match site features (input) into match site class labels which can be “correct” or “incorrect” (output). The resulting classification model is validated using a 10-fold cross validation experiment before using it for prediction.

Correct and incorrect sequence-specific match site training samples

To train a classifier using a supervised learning framework, we need labeled correct and incorrect examples of sequence-specific match sites. Out of the 3676 match sites found in this study, 783 of them are already integrated into the GlyGen resource and can be considered as correct examples. If a match site sipj extracted from a given abstract is a correct example (already integrated into GlyGen), we consider the set of match sites [skpj] where k ∈ [1, N] with k ≠ i to be incorrect examples if skpj is not known to be a correct example. If we take PMID “19,369,259” as an example, since match site “P16671|Asn-321” is a correct example, match sites “Q3HUX1|Asn-321” and “Q27367|Asn-321” are considered as incorrect examples since they are not correct examples (not in GlyGen). Using this framework, we were able to identify a set of 363 incorrect examples.

Algorithms for training supervised classifier

The task of classification or pattern recognition is formulated based on the theory of inductive principle of Structural Risk Minimization (SRM). According to this theory, to guarantee a high level of generalization ability, one must minimize the risk by controlling both the value of the empirical risk and the value of the confidence interval. As a result, there are two constructive approaches to the learning process. The first approach, implemented in Neural Network (NN) learning machines, minimizes the empirical risk while keeping the confidence interval fixed by choosing an appropriate construction of the machine. In the second approach, implemented in Support Vector Machines (SVMs), the confidence interval is minimized while keeping the empirical risk fixed. In this study, the classifier model was built using both Multi-Layer Perceptron (MLP) and SVM classifiers from the scikit-learn (https://scikit-learn.org/) open-source machine learning library for the Python programming language. The MLP classifier is a feedforward artificial neural network, consisting of fully connected neurons with a nonlinear activation function, which can distinguish data that is not linearly separable.

Tuning the decision threshold for class prediction

Like any binary classification task, once a model has been learned to predict class probabilities, we must investigate the decision problem and find an optimal threshold on the class probabilities that is suitable for our application. When the model is validated using a 10-fold cross-validation scheme, class predictions are done using a hard-coded cut-off value of 0.5 on the prediction probability, and a positive class label is predicted when then the prediction probability is greater than this cut-off value. While this hard-coded rule might seem reasonable as a default behavior, it is not ideal for most use cases. As a result, the decision threshold of the classifier should be tuned to find an optimal threshold that maximizes a given performance metric such as precision, recall, accuracy, or balanced accuracy. We chose the balanced accuracy metric which can deal with imbalance in the number of correct and incorrect match sites in our training dataset and is defined as the average of recall obtained on each class.

L‌LM based site verification stage

The GPT-4o Mini LLM to verify the accuracy of sequence-specific match sites that passed the supervised ML based filtering by prompting it with a specific yes or no question. For example, if we take the sequence-specific match site represented by “19369259|P16671|Asn-247,” since the gene entity “CD36” and species entity “human” were identified and mapped to UniProtKB accession “P16671” during the entity mapping stage, the corresponding prompt for the LLM is —.

“Is human CD36 glycosylated at position 247 based on the information provided in the text: <abstract_text>”.

Acknowledgments

We thank all current and past GlyGen developers and collaborators for the continuous development of GlyGen knowledgebase and software tools, which have greatly facilitated this work.

Contributor Information

Robel Kahsay, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Urnisha Bhuiyan, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Cyrus Chun Hong Au, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Nathan Edwards, Department of Biochemistry and Molecular & Cellular Biology, 37th and O Street NW. Georgetown University, Washington, DC 20007, United States.

Luke Johnson, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Sujeet Kulkarni, Complex Carbohydrate Research Center, The University of Georgia, 314 Riverbend Rd, Athens, GA 30602, United States.

Karina Martinez, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Rene Ranzinger, Complex Carbohydrate Research Center, The University of Georgia, 314 Riverbend Rd, Athens, GA 30602, United States.

K Vijay-Shanker, Department of Computer and Information Sciences, University of Delaware, 18 Amstel Ave, Newark, DE 19716, United States.

Jeet Vora, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Kate Warner, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Michael Tiemeyer, Complex Carbohydrate Research Center, The University of Georgia, 314 Riverbend Rd, Athens, GA 30602, United States.

Raja Mazumder, Department of Biochemistry & Molecular Medicine, The George Washington School of Medicine and Health Sciences, 2300 Eye Street NW, Washington, DC 20052, United States.

Funding

This work was supported and funded by the National Institute of General Medical Sciences under grant # 1R24GM146616–01 and the National Institutes of Health Office of Strategic Coordination — The Common Fund under grant # 1OT2OD032092.

Conflict of interest statement

None declared.

Data and software availability

Data and software used in this study are available at https://github.com/glygener/glycositeminer

References

  1. Aoki-Kinoshita  KF, Lisacek  F, Mazumder  R, York  WS, Packer  NH. 2020. The GlySpace alliance: toward a collaborative global glycoinformatics community. Glycobiology. 30:70–71. 10.1093/glycob/cwz078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Arumugam  K, Sellappan  M, Anand  D, Anand  S, Radhakrishnan  SV. 2022. A text mining and machine learning protocol for extracting posttranslational modifications of proteins from PubMed: a special focus on glycosylation, acetylation, methylation, hydroxylation, and ubiquitination. Methods Mol Biol. 2496:179–202. 10.1007/978-1-0716-2305-3_10. [DOI] [PubMed] [Google Scholar]
  3. Delafield  DG, Li  L. 2021. Recent advances in analytical approaches for glycan and glycopeptide quantitation. Mol Cell Proteomics. 20:100054. 10.1074/mcp.R120.002095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Dennis  JW, Granovsky  M, Warren  CE. 1999. Protein glycosylation in development and disease. Bioessays. 21:412–421. 10.1002/(SICI)1521-1878(199905)21:5<412::AID-BIES8>3.0.CO;2-5. [DOI] [PubMed] [Google Scholar]
  5. Elangovan  A, Li  Y, Pires  DEV, Davis  MJ, Verspoor  K. 2022. Large-scale protein-protein post-translational modification extraction with distant supervision and confidence calibrated BioBERT. BMC Bioinf. 23:4. 10.1186/s12859-021-04504-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Gupta  R, Brunak  S. 2002. Prediction of glycosylation across the human proteome and the correlation to protein function. Pac Symp Biocomput. 2002:310–322. [PubMed] [Google Scholar]
  7. Harvey  DJ. 2009. Analysis of carbohydrates and glycoconjugates by matrix-assisted laser desorption/ionization mass spectrometry: an update for 2003-2004. Mass Spectrom Rev. 28:273–361. 10.1002/mas.20192. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Honnibal  M, Montani  I. 2017. spaCy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. https://spacy.io/.
  9. Huang  H  et al.  2018. iPTMnet: an integrated resource for protein post-translational modification network discovery. Nucleic Acids Res. 46:D542–D550. 10.1093/nar/gkx1104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Kahsay  R  et al.  2020. GlyGen data model and processing workflow. Bioinformatics. 36:3941–3943. 10.1093/bioinformatics/btaa238. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Lee  J  et al.  2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 36:1234–1240. 10.1093/bioinformatics/btz682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Lisacek  F, Tiemeyer  M, Mazumder  R, Aoki-Kinoshita  KF. 2023. Worldwide Glycoscience informatics infrastructure: the GlySpace alliance. JACS Au. 3:4–12. 10.1021/jacsau.2c00477. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Masoumi  S, Amirkhani  H, Sadeghian  N, Shahraz  S. 2024. Natural language processing (NLP) to facilitate abstract review in medical research: the application of BioBERT to exploring the 20-year use of NLP in medical research. Syst Rev. 13:107. 10.1186/s13643-024-02470-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Park  Y, Lebrilla  CB. 2005. Application of Fourier transform ion cyclotron resonance mass spectrometry to oligosaccharides. Mass Spectrom Rev. 24:232–264. 10.1002/mas.20010. [DOI] [PubMed] [Google Scholar]
  15. Raja  K, Natarajan  J. 2018. Mining protein phosphorylation information from biomedical literature using NLP parsing and support vector machines. Comput Methods Prog Biomed. 160:57–64. 10.1016/j.cmpb.2018.03.022. [DOI] [PubMed] [Google Scholar]
  16. Ross  KE  et al.  2016. Scalable text mining assisted curation of post-translationally modified Proteoforms in the protein ontology. CEUR Workshop Proc. 2016:1747. http://ceur-ws.org/Vol-1747/BIT103_ICBO2016.pdf. [PMC free article] [PubMed] [Google Scholar]
  17. Sayers  EW  et al.  2022. Database resources of the national center for biotechnology information. Nucleic Acids Res. 50:D20–D26. 10.1093/nar/gkab1112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Shukla  D, Jayaraman  VK. 2012. A text mining approach to detect mentions of protein glycosylation in biomedical text. Bioinformation. 8:758–762. 10.6026/97320630008758. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Sigrist  CJ  et al.  2013. New and continuing developments at PROSITE. Nucleic Acids Res. 41:D344–D347. 10.1093/nar/gks1067. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Su  P, Vijay-Shanker  K. 2022. Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction. BMC Bioinf. 23:120. 10.1186/s12859-022-04642-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Sun  D, Wang  M, Li  A. 2017. MPTM: a tool for mining protein post-translational modifications from literature. J Bioinforma Comput Biol. 15:1740005. 10.1142/S0219720017400054. [DOI] [PubMed] [Google Scholar]
  22. Torii  M  et al.  2015. RLIMS-P 2.0: a generalizable rule-based information extraction system for literature Mining of Protein Phosphorylation Information. IEEE/ACM Trans Comput Biol Bioinform. 12:17–29. 10.1109/TCBB.2014.2372765. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Tudor  CO, Arighi  CN, Wang  Q, Wu  CH, Vijay-Shanker  K. 2012. The eFIP system for text mining of protein interaction networks of phosphorylated proteins. Database (Oxford). 2012:bas044. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. UniProt  C. 2023. UniProt: the universal protein knowledgebase in 2023. Nucleic Acids Res. 51:D523–D531. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Varki  A. 2017. Biological roles of glycans. Glycobiology. 27:3–49. 10.1093/glycob/cww086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Varki  A  et al., editors. 2022. Essentials of glycobiology [internet]. 4th ed. Cold Spring Harbor (NY): Cold Spring Harbor Laboratory Press;  https://www.ncbi.nlm.nih.gov/books/NBK579918/ [PubMed] [Google Scholar]
  27. Veuthey  AL  et al.  2013. Application of text-mining for updating protein post-translational modification annotation in UniProtKB. BMC Bioinf. 14:104. 10.1186/1471-2105-14-104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Wang  M  et al.  2014. Literature mining of protein phosphorylation using dependency parse trees. Methods. 67:386–393. 10.1016/j.ymeth.2014.01.008. [DOI] [PubMed] [Google Scholar]
  29. Wei  CH, Allot  A, Leaman  R, Lu  Z. 2019. PubTator central: automated concept annotation for biomedical full text articles. Nucleic Acids Res. 47:W587–W593. 10.1093/nar/gkz389. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Xu  Y, Teng  D, Lei  Y. 2012. MinePhos: a literature mining system for protein phoshphorylation information extraction. IEEE/ACM Trans Comput Biol Bioinform. 9:311–315. 10.1109/TCBB.2011.85. [DOI] [PubMed] [Google Scholar]
  31. York  WS  et al.  2020. GlyGen: computational and informatics resources for Glycoscience. Glycobiology. 30:72–73. 10.1093/glycob/cwz080. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data and software used in this study are available at https://github.com/glygener/glycositeminer

Articles from Glycobiology are provided here courtesy of Oxford University Press

RESOURCES