GPT2-ICC: A data-driven approach for accurate ion channel identification using pre-trained large language models

Abstract

Current experimental and computational methods have limitations in accurately and efficiently classifying ion channels within vast protein spaces. Here we have developed a deep learning algorithm, GPT2 Ion Channel Classifier (GPT2-ICC), which effectively distinguishing ion channels from a test set containing approximately 239 times more non-ion-channel proteins. GPT2-ICC integrates representation learning with a large language model (LLM)-based classifier, enabling highly accurate identification of potential ion channels. Several potential ion channels were predicated from the unannotated human proteome, further demonstrating GPT2-ICC’s generalization ability. This study marks a significant advancement in artificial-intelligence-driven ion channel research, highlighting the adaptability and effectiveness of combining representation learning with LLMs to address the challenges of imbalanced protein sequence data. Moreover, it provides a valuable computational tool for uncovering previously uncharacterized ion channels.

Graphical abstract

Highlights

• •GPT2-ICC was constructed to accurately distinguish ion channels from huge protein spaces.
• •GPT2-ICC combines representation learning and LLM-based classifier to enable the accurate detection of ion channels.
• •Potential ion channels in human proteome were investigated, demonstrating GPT2-ICC’s high enrichment ability and diversity.

1. Introduction

Advances in genomics and proteomics have facilitated the discovery of diverse functional proteins. Among these, ion channels are pivotal in regulating ion passage across cell membranes at diffusion-controlled rates [1]. These channels are integral to numerous physiological and pathological processes, including neural excitability, signal transduction, muscle contraction, pain sensation and cardiopathies [2]. Given their considerable therapeutic potential, a large number of drugs have been designed to target ion channels [3]. Consequently, ion channels studies, particularly focus on their role in diseases and drug design targeting them, has been a prominent area for decades [4]. Characterizing the typical properties of ion channels has provided valuable insights into their diverse functions and links to diseases. Classical ion channels were well characterized by central pores facilitating ion flow, selectivity filters allowing specific ions to pass, and gating sensors that control channel opening in response to various stimuli [5]. Proteins possessing these features often share similar functional profiles. For instance, the selectivity filter of potassium channels is primarily formed by carbonyl oxygen groups from the signature “TVGYG” motif (consisting of Thr-Val-Gly-Tyr-Gly residues) [6]. And these common characteristics provide a systematic approach for identifying new ion channels.

The discovery of new ion channels is critical for understanding the fundamental biological processes they regulate. Currently, a substantial portion of protein sequences in databases such as UniProt remain experimentally unverified [7], suggesting the potential existence of undiscovered ion channels within these sequences. However, traditional methods for identifying ion channels are labor-intensive, complex, and often rely on serendipitous discoveries from large candidate pools [8]. Moreover, ion channels often exhibit multifunctionality beyond ion transport [9]. For instance, the stimulator of interferon genes (STING) has been implicated in interferon induction, light chain 3B (LC3B) lipidation, and nucleotide-binding domain (NBD), leucine-rich repeat (LRR), and pyrin domain (PYD)-containing protein 3 (NLRP3) inflammasome activation, in addition to its recently identified role in proton transport, highlighting that known non-ion-channel proteins may also harbor undiscovered ion channel functions [[10], [11], [12], [13]]. Experimental validation of the full spectrum of unknown proteins functions presents a formidable challenge. In contrast, computational methods offer a faster and more efficient alternative. Several computational approaches have been proposed to classify ion channels from other proteins [[14], [15], [16], [17]]. However, these methods are often evaluated using datasets with limited negative samples, raising concerns about their generalizability and enrichment capacity when applied to the broader proteome. Ion channel classification remains a fundamental scientific challenge that requires further methodological innovation and validation.

The rapid evolution of artificial intelligence (AI) models offers a promising solution to address the challenges in ion channel classification. High-quality protein sequence representations are fundamental to statistic learning in biological data [18]. Early studies in ion channel classification relied on feature engineering techniques to manually or automatically select physicochemical properties of amino acids based on their performance in downstream tasks [14,16,17,[19], [20], [21]]. However, given the structural and functional complexity of proteins, these engineered features often fail to capture their full diversity. Recently, representation learning methods have gained traction [22], leveraging embedding vectors extracted from pretrained, task-specific models to improve performance in downstream applications [[23], [24], [25], [26], [27], [28], [29], [30]]. Once robust protein representations are established, the choice of AI model architecture significantly influences performance, particularly when working with datasets characterized by imbalanced distributions. Traditional machine learning algorithms like support vector machine (SVM), logistic regression, random forest and k-nearest neighbors (kNN) often struggle with overfitting to dominant negative samples when trained on whole-proteome datasets with a high proportion of non-ion-channel proteins [31]. To deal with imbalanced datasets, data sampling strategies have been employed, typically reducing the proportion of negative samples to improve classification accuracy [15]. Nevertheless, this method frequently leads to a higher false positive (FP) rate, reducing the number of confirmed hits during experimental validation. Compared to these traditional machine learning algorithms, fine-tuning pretrained large language models (LLMs) offers distinct advantages. LLMs effectively handle extensive datasets, extract features using learned attention weights [32] and exhibit strong generalization capabilities [33]. Models such as BERT have shown exceptional accuracy in predicting the functions of millions of genes, underscoring the unique potential of LLMs [28]. By combining embeddings from models like Evolutionary Scale Modeling (ESM) with LLM-based classifiers, researchers can achieve a viable solution for large-scale classification tasks. This approach holds the promise for improving hit rates while maintaining high accuracy, thereby facilitating the discovery of novel ion channels.

In this study, we developed GPT2 Ion Channel Classifier (GPT2-ICC), a data-driven AI method designed to effectively and accurately identify ion channels from large proteomic datasets. We constructed a comprehensive dataset comprising 337,101 sequences from the UniProtKB/Swiss-Prot database, ensuring sequence similarity was less than 90%. The dataset maintained a realistic distribution, with a ratio of ion channels to non-ion-channel proteins of 1:243. Sequence embeddings were generated using the Evolutionary Scale Modeling 2 (ESM2) protein structure prediction model, which significantly enhanced performance in downstream tasks. The dataset was randomly split, with 80% allocated for training and validation and 20% reserved for testing. By fine-tuning a locally pretrained GPT2-base model, we developed GPT2-ICC, which accurately distinguishes ion channels from non-ion-channel proteins in the test set. The model exhibited a low FP rate and a high Matthews correlation coefficient (MCC), outperforming previous methods and a benchmark across all metrics. These results highlight the model’s reliability, particularly when handling datasets heavily skewed toward negative samples. Furthermore, we applied GPT2-ICC to predict ion channels within 1.8 million human proteome sequences from the unreviewed UniProt database, completing the task in under 2 min. Among the 849 sequences predicted as ion channels, only 19 did not match the automatically annotated functional labels from UniProt. In summary, GPT2-ICC is a powerful AI-driven tool for large-scale ion channel classification. By combining high efficiency, precision, and robust generalization, it is ideally suited for tackling the challenges of proteomic analysis in complex and imbalanced datasets.

2. Experimental

2.1. Datasets

We collected ion channel protein sequences from the UniProtKB/Swiss-Prot database using the protocols [14] to construct datasets (as of September 2024). First, each sequence was manually checked to ensure it functions as an ion channel rather than a regulatory subunit, transporter or others. We then collected 571,600 protein sequences which contain 2,516 ion channels. All sequences were clustered using MMseqs2 [34], with coverage and sequence identity thresholds set at 90%, resulting in a total of 337,101 sequences with 1,381 ion channels, as shown in Table 1. The label of each sequence is represented as a 2-dimensional (2D) one-hot vector, with the first element being 1 if the sequence functions as an ion channel otherwise 0. This encoding ensures that each class is uniquely identified, allowing the classifier model to easily distinguish between different categories.

Table 1.

Data statistics of original and balanced training sets.

Data processing	Class	Train	Test	Total
Original	Ion channel	1,100	281	1,381
	Non-ion-channel	268,589	67,131	335,720
Balanced	Ion channel	1,100	281	1,381
	Non-ion-channel	11,000	67,131	78,131

2.2. Learning information using the ESM2

ESM2 [35] is a protein language model which is capable of predicting protein structures at the atomic level. Learning from more than 617 million proteins from MGnify90 [36], ESM2 can generate protein sequence representations with abundant information. These representations are widely used for sequence-based protein function prediction to enhance classification performance. For our dataset, each sequence was generated by mean-pooling of the last hidden layer of the ESM2 ‘esm2_t33_650 M_UR50D’ and represented as a 1,280D protein feature embedding. Sequences longer than 12,290 amino acids were truncated to 12,290 amino acids for the limitation of GPU memory.

2.3. GPT2-ICC training and fine-tuning settings

After randomly shuffling the order of the non-redundant set composed of 337,101 UniProtKB/Swiss-Prot sequences, we partition the dataset into training, validation, and test sets, with approximate ratios of 70%/10%/20%.

We implemented an mean squared error (MSE) loss function to quantify the difference between predicted and actual values during the training process. The loss function is defined as follows:

$$\mathcal{L}=\frac{1}{n}\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2$$ (1)

where $n$ is the number of data samples. There are two elements in the 2D tensor $y_i$ that represent the true labels. It is set to [1,0] if x corresponds to an ion channel, and [0,1] otherwise. This encoding allows for clear differentiation between the two classes in the classification task. Similarly, ${\hat{y}}_i$ is a predicted softmaxed label.

GPT2-ICC was trained in single-precision with the batch size of 1,280 and a maximum of 1,000 epochs on one NVIDIA RTX6000 Ada GPU, which is equipped with 48 GB of GPU memory. Fine-tuning algorithms are illustrated in Algorithm 1. Self-attention blocks and feed-forward networks (FFNs) of GPT2-base were frozen by setting the “requires_grad” value to “False” to preserve the powerful feature extraction ability. Early stopping was implemented with a patience of 10 epochs to mitigate overfitting. To further enhance generalization, a dropout rate of 0.3 was applied. The Adam [37] optimizer was employed with a learning rate of 0.0001, and the parameters were set to beta1 = 0.9, beta2 = 0.999, along with a weight decay of 0.0005 to regulate model complexity. This configuration ensured robust training and effective optimization of the model parameters.

where $FFN(\ast )$ is the feedforward layer which represents a linear transformation, $x_{pn}$ is the normalized protein sequence sample (see sections below), $CL{S}_{pred}$ is the predicted label of input protein sequence sample, and $CL{S}_{true}$ is the ground truth label of input protein sequence sample.

2.4. GPT2-ICC architecture

Given our selection of GPT2-base [38] as the backbone model, the overall model architecture we employ is shown in Fig. S1. In addition to the original blocks of the GPT2-base backbone, we also introduce several additional blocks. The main components of GPT2-ICC are described as follows.

2.4.1. Instance normalization (IN)

Data normalization is crucial for pre-trained LLMs applied to protein classification tasks, as the data distribution of protein sequences varies greatly across different categories. Drawing inspiration from RevIN [39] method for time series analysis tasks, in addition to the layer normalization employed in the pre-trained LLMs, we first introduce the IN block to further enhance the stability of fine-tuning. This IN block simply normalizes the given protein sequence $x_p$ using its mean and variance, and then adds them back to the output. It can be formulized as follows:

$$x_{pn}=\gamma \left(\frac{x_p-\mathbb{E}\left(x_p\right)}{\sqrt{Var\left(x_p\right)+ϵ}}\right)+\beta$$ (2)

where $x_{pn}$ is the normalized protein sequence sample, $\mathbb{E}$ is the average operation, $Var$ is the variance calculation operation, $ϵ$ is a small constant and $\gamma$, $\beta$ are learnable affine parameter vectors.

2.4.2. Input embedding

To enable the pre-trained LLM backbone to process special protein sequences data, we employ linear probing to project the protein sequences data into the input dimension required by the backbone model. Additionally, we fine-tune the input embedding layer, along with the existing positional embeddings and layer normalization layers, to learn a reasonable representation of the protein sequences data in this dataset. The mapped token embeddings $H_{token}$ will be combined with positional and temporal embeddings to yield the final embedding. The mapping process can be formulized as follows:

$$H_{token}=FFN\left(x_{pn}\right)\in {\mathbb{R}}^{Hidden}\text{,}$$ (3)

where $FFN(\ast )$ is the feedforward layer which represents a linear transformation applied to the input $x_{pn}$ projecting it into a latent space of dimension $Hidden$.

2.4.3. GPT2 blocks

As shown in Fig. S1, the core of our architecture is the pre-trained GPT2-base model, which consists of 6 Transformer decoder blocks. The hidden size of each decoder block is 768. Each block includes the following components:

2.4.3.1. Masked multi-heads self attention (MMSA)

This module allows the model to focus on different parts of the input sequence while maintaining the autoregressive property. It uses a mask to prevent the model from attending to future tokens, ensuring that predictions at each step are based only on the information available up to that point. The MMSA layer takes the input embeddings $H_{\text{token}}$ and performs self-attention over the sequence, generating a new set of hidden states $H_{mmsa}$. The number of the attention heads is 12. The attention mechanism can be described as follows:

$$H_{mmsa}=Attention\left(Q,K,V\right)=Softmax\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$$ (4)

where $Q=W_Q{H}_{token}$, $K=W_K{H}_{token}$, and $V=W_V{H}_{token}$ are the query, key, and value matrices, respectively, derived from the input embeddings $H_{\text{token}}$ through linear transformations. $W_Q$, $W_K$, and $W_V$ are learnable projection matrices, and $d_k$ is the dimension of the key vectors.

2.4.3.2. Layer normalization

Layer normalization is applied after the MMSA and FFNs to stabilize the hidden state activations. It normalizes the activations across the features within the same sample, which helps to improve the training dynamics. The normalized output $H_{\mathit{ln}}$ from the MMSA layer can be expressed as:

$$H_{\mathit{ln}}=LayerNorm\left(H_{mmsa}\right)$$ (5)

2.4.3.3. FFN

The FFN consists of two linear transformations with a rectified linear unit activation (ReLU) function in between. It provides the model with the ability to learn more complex functions and capture non-linear relationships within the data. The FFN takes the normalized output $H_{\mathit{ln}}$ and transforms it into a new set of hidden states $H_{ffn}$:

$$H_{ffn}=ReLU\left(W_1{H}_{\mathit{ln}1}+b_1\right)W_2+b_2$$ (6)

where $W_1$, $b_1$, $W_2$, and $b_2$ are the weights and biases of the two linear transformations.

Each block processes the input data, transforming it through these layers, and passes the resulting hidden states to the next block. Then, the final hidden states are input into the output linear probing to obtain the classification results. The entire stack of transformer blocks forms the decoder, which captures the intricate dependencies and features in the ion channel data, enabling accurate and robust classification.

2.4.3.4. Output embedding

We also employ linear probing to project the latent embeddings $H_{latent}$ into the output dimension required by the classification tasks. The dimension of the output embedding layer is determined by the number of ion channel categories. The process can be described as:

$$C_{out}=FFN\left(H_{latent}\right)\in {\mathrm{ℝ}}^{T\times 1}$$ (7)

where $FFN(\ast )$ is the feedforward layer which represents a linear transformation applied to the latent variants $H_{latent}$ projecting it into the required dimension [$T\times 1$] of output.

2.5. Principal component analysis (PCA) of GPT2-ICC embeddings

To visualize the separability of ion channels and non-ion-channel proteins, we followed the process described in Algorithm 2 to project high-dimensional embedding vectors into a 2D space using PCA, as illustrated below:

2.6. Performance metric

Consistent with prior ion channel classification research, the following metrics are utilized to objectively evaluate the predictive performance: MCC, precision, recall, and F1 score (F1),

$$\mathrm{MCC}=\frac{\mathrm{TP}\times \mathrm{TN}-\mathrm{FN}\times \mathrm{TP}}{\sqrt{\left(\mathrm{TP}+\mathrm{TN}\right)\times \left(\mathrm{TP}+\mathrm{FP}\right)\times \left(\mathrm{TN}+\mathrm{FN}\right)\times \left(\mathrm{TN}+\mathrm{TP}\right)}}$$ (8)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$ (9)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$ (10)

$$\mathrm{F}1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$ (11)

where $\mathrm{TP}$, $\mathrm{TN}$, and $\mathrm{FN}$, represent the numbers of true positives, true negatives, and false negatives, respectively, which are used to evaluate the model’s performance metrics.

2.7. Comparison with other methods

2.7.1. Transformer

The Transformer [40] model is a state-of-the-art neural network architecture primarily used in natural language processing tasks and has recently gained traction in various domains, including the biomedical field. It relies solely on attention mechanisms, discarding the traditional recurrence and convolution, which makes it particularly effective for processing sequential data. The model consists of two main components: an encoder and a decoder, both of which are composed of multiple layers with self-attention layers and FFNs. The hidden state dimension of Transformer is 1,280. We use the 1 layer of Transformer as our experimental model backbone.

2.7.2. LLaMA2

LLaMA2 [41] is a large-scale language model developed by Meta, based on the Transformer architecture. It is designed to generate and understand natural language and is capable of performing various tasks such as text generation, translation, and question answering. LLaMA2 employs a Transformer encoder-decoder architecture. The hidden state dimension of LLaMA2 is 4,096. We use the first 6 layers of LLaMA2 as our experimental model backbone.

2.7.3. BERT

The BERT [42] model is a language representation model that has revolutionized natural language processing. It uses the Transformer architecture, which allows for self-attention mechanisms to capture contextual relationships between words in a sentence. In BERT, only the encoder is used. The hidden state dimension of BERT is 768. We use the first 6 layers of BERT as the backbone of our experimental model.

2.7.4. Convolutional neural network (CNN)

The CNN [43] model is a deep learning architecture that has revolutionized image and video processing tasks. CNNs automatically learn and extract relevant features from images, reducing the need for manual feature engineering. The CNN model has found numerous applications in the biomedical field, addressing complex challenges related to medical imaging, genomics, and more. We implemented the CNN network using Pytorch library. We built a CNN model using 4 layers of building blocks, each consisting of sequential convolution, ReLU and maxpool layers, and the CNN architectures are shown in Fig. S2.

2.7.5. SVM

SVM [44] is a powerful supervised learning model and algorithm used for classification and regression analysis. SVM aims to find the optimal separating hyperplane that maximizes the margin between two classes of data points. In a binary classification problem, SVM seeks to construct a decision boundary that separates the instances of one class from those of the other class, with the largest possible distance between the nearest points of the two classes. It is particularly effective in high-dimensional spaces and can handle both linear and nonlinear classification problems through the use of kernel functions. We implemented SVM using scikit-learn library [45].

2.7.6. kNN

The kNN [46] is a non-parametric, supervised learning method that classifies a new data point based on the class of its k nearest neighbors in the training dataset. It operates under the assumption that similar points are likely to be close to each other in the feature space. The kNN algorithm calculates the distance between the new data point and each data point in the training set. Common distance metrics include Euclidean distance and Manhattan distance. In a classification problem, the kNN algorithm assigns the label of the majority class among the k nearest neighbors to the new data point. We implemented kNN using scikit-learn library.

2.7.7. PSIONplus

PSIONplus [21] is an accurate sequenced-based predictor of ion channel. It uses BLAST [47] to perform sequence similarity search, combining evolutionary profiles, predicted secondary structure, solvent accessibility and intrinsic disorder to generate input embeddings. Then, it utilizes an SVM model to predict if a protein sequence functions as an ion channel. We compare our model to PSIONplus by evaluating the performance in our test sets.

3. Results

3.1. GPT2-ICC: an LLM classifier for identifying ion channel sequences

We developed the GPT2-ICC method by leveraging ESM2 for protein sequence representation and fine-tuning the GPT2-base model to classify protein sequences as either ion channels or non-ion-channels. As illustrated in Fig. 1A, the dataset was constructed by first curating ion channel sequences from the UniProtKB/Swiss-Prot database using a keyword-based search protocol [14], with auxiliary subunits manually filtered out. To reduce redundancy, sequences exhibiting more than 90% similarity were removed using MMseqs2 [34], resulting in a dataset of 1,381 ion channel proteins and 335,720 non-ion-channel proteins. Despite the significant class imbalance (negative samples outnumbering positives by a ratio of 243:1), we chose not to artificially balance the dataset. This decision reflects the real-world distribution of ion channels, which are a small fraction of the genome, and artificially balancing the dataset could compromise the model’s ability to identify novel ion channels in unannotated proteomes.

Fig. 1.

The overview of GPT2 Ion Channel Classifier (GPT2-ICC). (A) Data acquisition of ion channel sequences. Auxiliary subunits are discarded along with non-ion-channel proteins. Each sequence is labeled with a 2-dimensional (2D) one-hot vector. (B) Data processing of protein sequences. 1,280-dimensional protein representations are generated by mean-pooling of the last hidden layer of the Evolutionary Scale Modeling 2 (ESM2). (C) Pipeline of GPT2-ICC fine-tuning process. After data processing, representations are projected by an adaptive layer to fit pretrained GPT2-base model for fine-tuning. By classification layer, hidden vectors from the last layer of GPT2-base are projected to a 2D probability vector, whose first element stands for probability of ion channel. LLM: large language model.

Comprehensive representation of protein sequences can enhance downstream tasks. Therefore, we used ESM2, an LLM designed for versatile protein structure prediction, to convert each sequence into a 1,280D embedding vector (Fig. 1B). Rather than relying on manual feature selection or feature engineering, we applied linear probing using a linear adaptive layer to project each embedding into the input space of the GPT2-base model (Fig. 1C). This enabled the pre-trained LLM backbone to interpret and process the representation learned from ESM2. During fine-tuning, we froze the self-attention blocks and FFNs of GPT2-base, as these components are highly generalized and perform feature extraction in a manner similar to PCA [32]. Freezing these layers preserved the model’s feature extraction capabilities while reducing the risk of overfitting. After a linear classification layer, the model outputs a 2D softmaxed tensor, where the first dimension represents the probability that a given protein sequence functions as an ion channel.

3.2. GPT2-ICC can accurately discriminate ion channels from huge protein spaces

To evaluate the classification ability of GPT2-ICC, we visualized the embedding vectors of sequences from the training set. These embedding, extracted before the classification layer, were reduced to two dimensions using PCA (see Section 2.5). As shown in Fig. 2A, GPT2-ICC successfully learned a classification boundary where a simple linear separation effectively distinguishes ion channels from non-ion-channel proteins. We further evaluated GPT2-ICC model using an unseen test set comprising 281 ion channels and 67,131 non-ion-channel proteins. As illustrated in Fig. 2B, the accuracy calculated from this test set provides an estimation of the model’s performance across the entire proteome. Notably, only 5 non-ion-channel proteins were misclassified as ion channel proteins. Typically, low FP counts could indicate overfitting, especially if recall is also low. However, GPT2-ICC achieved a high recall of 0.90, demonstrating that the model effectively captured ion-channel-specific features and mapped them into distinguishable subspaces. Additionally, the model exhibited robust performance with an AUC of 0.97 (Fig. S3), further confirming that GPT2-ICC’s classification ability was not compromised by the imbalanced nature of the dataset. Taken together, these results show that GPT2-ICC is a highly robust classifier, unaffected by the long-tail distribution of the dataset, and it effectively distinguishes ion channel proteins from the vast majority of non-ion-channel proteins.

Fig. 2.

GPT2 Ion Channel Classifier (GPT2-ICC) accurately discriminate ion channels from huge protein spaces. (A) Principal component analysis (PCA) visualization using last embedding vectors from GPT2-ICC of all training data. (B) Matthews correlation coefficient (MCC), recall, precision rates, and F1 score (F1) of GPT2-ICC evaluated on unseen test sets.

3.3. GPT2-ICC outperforms other competing methods

We then benchmarked GPT2-ICC against other LLMs (BERT and LLaMA2), traditional methods (CNN, SVM, kNN, and Transformer), and benchmark method (PSIONplus) (see Section 2 Experimental). This comparison was designed to evaluate GPT2-ICC’s ability to accurately classify ion channels while preventing overfitting. All models were evaluated using independent test sets. As shown in Figs. 3A−D, GPT2-ICC consistently outperformed the other methods across multiple evaluation metrics, including MCC, recall, precision, and F1. It is worth noting that LLaMA2 completely overfit on the negative non-ion-channel protein samples (Table 2). This could be attributed to the insufficient size of our dataset for fine-tuning LLaMA2, given its large number of parameters (Table 3), or potentially due to the redundancy introduced during the projection of the 1,280D embedding to a 4,096D vector. These findings suggest that LLaMA2 is not a suitable LLM classifier for our data. BERT exhibited slightly lower performance, with FP counts approximately four times higher (Fig. 3E). SVM, Transformer and kNN models also exhibit varying degrees of overfitting. We also compared GPT2-ICC to the previous benchmark, PSIONplus [21], an SVM-based model that generates sequence features using BLAST. GPT2-ICC outperformed PSIONplus across all metrics (Figs. 3A−D), demonstrating its superiority in handling imbalanced datasets and its ability to capture nuanced sequence features beyond traditional similarity-based approaches.

Fig. 3.

Benchmarking of GPT2 Ion Channel Classifier (GPT2-ICC) with other competing methods. (A–E) Performance metrics of GPT2, BERT, convolutional neural network (CNN), support vector machine (SVM), k-nearest neighbors (kNN), Transformer, and PSIONplus were compared using the unseen test sets in this study: Matthews correlation coefficient (MCC) (A), recall (B), precision (C), F1 score (F1) (D), and false positive (FP) counts (E).

Table 2.

Prediction results of models on the test set.

Model	TP	FP	TN	FN
GPT2	254	5	67,126	27
BERT	245	27	67,104	36
CNN	189	8	67,123	92
SVM	189	7	67,124	92
kNN	128	20	67,111	123
Transformer	72	0	67,131	209
LLaMA2	0	0	67,131	281
PSIONplusa	173	14,944	51,265	107

CNN: convolutional neural network; SVM: support vector machine; kNN: k-nearest neighbors; TP: true positive; FP: false positive; TN: true negative; FN: false negative.

^a923 sequences in the test set were not successfully processed due to unnatural amino acids or other errors, including 1 ion channel and 922 non-ion-channel sequences.

Table 3.

Statistics of parameters of large language models (LLMs).

Model	Parameters	Trainable parameters
GPT2	82,899,458	44,302,082
BERT	110,469,122	986,882
LLaMA2	1,350,639,618	1,219,514,307

To examine whether data balancing could improve performance, we adjusted the training dataset to a 10:1 ratio of negative to positive samples (Table 1). Then we evaluated these balanced models using the same independent test set, and the prediction results and evaluation metrics are illustrated in Table 4 and Figs. 4A−E, respectively. As expected, this adjustment led to improved recall in the test dataset (Fig. 4B). However, the FP counts for SVM, Transformer, and kNN were approximately 21, 40 and 181 times higher respectively, compared to GPT2-ICC (Fig. 4E). These results indicate that, despite data imbalance normalization, models such as SVM, Transformer, and kNN are not well-suited for large-scale ion channel classification tasks. By outperforming traditional models, other LLMs, and benchmark methods like PSIONplus, GPT2-ICC underscores its robustness and suitability for large-scale ion channel identification tasks.

Table 4.

Prediction results of balanced models on the test set.

Model	TP	FP	TN	FN
CNN-b	218	458	66,673	63
SVM-b	244	111	67,020	37
kNN-b	235	910	66,221	46
Transformer-b	276	207	66,924	5
LLaMA2-b	69	52	67,079	212

CNN-b: convolutional neural network balanced; SVM-b: support vector machine balanced; kNN-b: k-nearest neighbors balanced; TP: true positive; FP: false positive; TN: true negative; FN: false negative; -b: models trained using balanced training data sets.

Fig. 4.

Comparison of GPT2 Ion Channel Classifier (GPT2-ICC) with other methods after balancing the training sets. (A–E) Performance metrics of GPT2, support vector machine balanced (SVM-b), Transformer-b, convolutional neural network balanced (CNN-b), k-nearest neighbors balanced (kNN)-b, and LLaMA2-b were compared using the unseen test sets in this study: Matthews correlation coefficient (MCC) (A), recall (B), precision (C), F1 score (F1) (D), and false positive (FP) counts (E).

3.4. Predicting functions of unannotated proteins

We applied GPT2-ICC to the human proteome within the UniProt-unreviewed database, which contains approximately 1.8 million automatically translated protein sequences without manual annotation or experimental validation. Fig. 5 illustrates the ion channel classification workflow, including the GPT2-ICC prediction process and subsequent manual validation steps. Based on their gene names and protein sequences, the predicted functions and interactions among proteins provided can provide valuable insights and serve as a reference for assessing the accuracy of function prediction methods. After converting these sequences into embedding vectors using our established data processing pipeline, we input these embeddings into GPT2-ICC and used the argmax function to classify each sequence as either an ion channel or a non-ion-channel protein. From this analysis, a total of 849 sequences (Table S1) were predicted to be ion channels. Manual validation against UniProt annotations revealed that only 19 sequences were not associated with ion channel genes (Table S2). Notably, two of these were GPCR sequences predicted to have ion-transporting capabilities, consistent with their automatically annotated ion-transporting function. This suggests that GPT2-ICC effectively captures the general features of ion transport rather than relying on sequence memorization. Given GPT2-ICC’s high accuracy and robustness on the test dataset, we hypothesize that these 19 sequences may indeed include ion channels. This finding warrants further experimental validation to confirm their functions.

Fig. 5.

Large-scale ion channel classification on unannotated proteins from UniProt. (A) Pipeline for ion channel classification on unannotated proteins using GPT2 Ion Channel Classifier (GPT2-ICC) model. (B) Examples of UniProt codes of potential ion channels including both known ion channel genes and unknown genes.

4. Discussions

The accurate identification of ion channels remains a challenging task in molecular biology. Traditionally, ion channels have been identified through experimental methods like patch-clamp techniques, which are labor-intensive and time-consuming, and often yield a low hit rate among numerous candidates [8]. Computational methods like BLAST [47] have been widely used for protein classification. However, BLAST relies heavily on sequence similarity. This reliance can lead to the omission of ion channels with minimal sequence similarity, particularly non-classical channels, where functional similarities may be more evident in 3D or 4D structures rather than in linear sequences. To address these challenges, we developed a data-driven LLM called GPT2-ICC, which outperforms existing methods in terms of accuracy, efficiency, and FP rates. Trained on a dataset of 337,101 sequences with an ion channel to non-ion-channel proteins ratio of 1:243, GPT2-ICC successfully established an accurate classification boundary, and overcame the overfitting issues observed in other methods. The model demonstrated generalization capabilities in experiments, accurately predicting ion transport functions within the human proteome and identifying a diverse array of proteins with these functions. Collectively, we have established an extensible AI framework for large-scale ion channel identification that can be readily adapted for accurate function classification of other proteins once the training datasets are prepared.

In this study, representation learning techniques were employed to enhance classification performance, with ESM2 chosen as the representation model. Representation learning has gained increasing importance in biological research due to its ability to capture comprehensive features, encompassing both sequence and structural information. By leveraging ESM2, which incorporates structure predictions, we improved GPT2-ICC’s performance compared to traditional feature-engineering methods that primarily rely on the physicochemical properties of amino acids. Previous approaches, such as those employing one-hot encoding, have demonstrated limitations in capturing meaningful patterns for gene function prediction [28]. These methods often fail to represent the intricate relationships within protein sequences and their structural contexts. In summary, the adoption of ESM2 representations established a robust foundation for GPT2-ICC’s reliable ion channel classification, showcasing the critical role of representation learning in overcoming the limitations of traditional methods.

We selected GPT2-base, a widely recognized pretrained LLM [38], as the backbone for our model due to its proven adaptability for domain-specific fine-tuning. Recent studies, such as TooT-PLM-ionCT [15], have similarly leveraged task-specific protein language models, including ESM2, for ion channel classification. However, these approaches primarily relied on traditional machine learning models like SVM, kNN, and CNN, which exhibited overfitting when tested on our test sets. To evaluate the effectiveness of GPT2-base, we compared it with BERT, another prominent LLM previously employed in the gLM model, which has demonstrated high accuracy in gene function prediction across millions of genes. While both GPT2-base and BERT achieved strong performance in ion channel classification, GPT2-base exhibited a slight yet consistent advantage. This performance edge could be attributed to our strategy of freezing the self-attention blocks and FFNs during fine-tuning, which enabled the model to effectively leverage the ESM2 representations for feature extraction while minimizing the risk of overfitting.

Despite these advances, our study has several limitations. Ion channels often function as multimeric complexes, with specific functional domains formed by interactions between different subunits to mediate gating mechanisms (e.g., voltage-gated, ligand-gated) and ion selectivity (e.g., potassium, sodium, or calcium). In this study, we primarily utilized 2D information and partial 3D structural features, which may not fully capture the complexity of ion channel assemblies. While our approach is effective for broad ion channel classification, it lacks the granularity required to differentiate between closely related subclasses, especially when functional similarities arise from dynamic structural changes rather than static sequences or partial structures. Additionally, the limited incorporation of complete 3D or 4D structural data, which could provide insights into the conformational states of ion channels, poses a significant challenge for achieving high specificity and accuracy in sub-class tasks. Previous work, such as LucaProt [48], has shown that integrating comprehensive structural information can significantly enhance specificity and accuracy in tasks like viral species classification. However, for ion channels, the substantial size and complexity of these systems, coupled with the exponential increase in computational memory required by transformer-based models, make the incorporation of complete structural data a challenge. Advances in AI algorithms and hardware capabilities may eventually make it feasible to integrate full structural representations, potentially improving classification accuracy. Moreover, while explainable AI techniques, such as weight analysis, have been successfully applied to enzyme function classification by linking function directly to catalytic residues, this study does not provide an explainable framework for understanding how a protein functions as an ion channel. This limitation arises from the scale and structure of our dataset, which may not support such granular analyses. Developing explainable frameworks tailored to ion channel classification remains an important direction for future research, as it could offer deeper insights into the relationship between sequence, structure, and function. Finally, the 19 potential ion channel sequences identified by GPT2-ICC have yet to be experimentally validated. Given the model’s high accuracy and robustness, it is likely that some of these sequences represent novel ion channels. However, experimental validation will be crucial to confirm these predictions and assess the broader applicability of GPT2-ICC for ion channel discovery. Future studies integrating structural data, explainable AI methodologies, and experimental validation are expected to enhance the utility and reliability of our approach.

5. Conclusions

Overall, this study employed a data-driven AI framework, demonstrating the accuracy, robustness, and enrichment capabilities of GPT2-ICC for large-scale ion channel classification. By integrating representation learning techniques with domain adaptation and fine-tuning in LLMs, GPT2-ICC consistently outperformed other competing methods and identified numerous potential ion channel sequences within the unannotated human proteome. While experimental validation remains a necessary step to confirm these findings, the performance of GPT2-ICC underscores its potential as an efficient tool for protein classification and discovery.

CRediT authorship contribution statement

Zihan Zhou: Writing – original draft, Software, Methodology, Investigation, Formal analysis, Data curation. Yang Yu: Writing – original draft, Software, Methodology, Investigation. Chengji Yang: Visualization, Investigation, Formal analysis, Data curation. Leyan Cao: Visualization, Investigation, Formal analysis, Data curation. Shaoying Zhang: Writing – original draft, Visualization. Junnan Li: Writing – original draft, Validation. Yingnan Zhang: Visualization, Validation, Data curation. Huayun Han: Validation, Visualization. Guoliang Shi: Writing – original draft, Methodology. Qiansen Zhang: Writing – review & editing, Supervision, Resources, Funding acquisition. Juwen Shen: Writing – review & editing, Supervision, Resources, Funding acquisition. Huaiyu Yang: Writing – review & editing, Supervision, Resources, Funding acquisition, Conceptualization.

Data and code availability

We provide our GPT2-ICC model and all data for training and evaluation at https://github.com/InvincibleZZH/GPTICC. The pretrained GPT2-base model can be downloaded at https://huggingface.co/openai-community/gpt2?text=My+name+is+Thomas+and+my+main for user-specific fine-tuning following our protocols.

Declaration of competing interest

The authors declare that there are no conflicts of interest.

Appendix A. Supplementary data

The following is the Supplementary data to this article: