GINGER: an integrated method for high-accuracy prediction of gene structure in higher eukaryotes at the gene and exon level

Abstract

The prediction of gene structure within the genome sequence is the starting point of genome analysis, and its accuracy has a significant impact on the quality of subsequent analyses. Gene structure prediction is roughly divided into RNA-Seq-based methods, ab initio-based methods, homology-based methods, and the integration of individual prediction methods. Integrated methods are mainstream in recent genome projects because they improve prediction accuracy by combining or taking the best individual prediction findings; however, adequate prediction accuracy for eukaryotic species has not yet been achieved. Therefore, we developed an integrated tool, GINGER, that solves various issues related to gene structure prediction in higher eukaryotes. By handling artefacts in alignments of RNA and protein sequences, reconstructing gene structures via dynamic programming with appropriately weighted and scored exon/intron/intergenic regions, and applying different prediction processes and filtering criteria to multi-exon and single-exon genes, we achieved a significant improvement in accuracy compared to the existing integration methods. The feature of GINGER is its high prediction accuracy at the gene and exon levels, which is pronounced for species with more complex gene architectures. GINGER is implemented using Nextflow, which allows for the efficient and effective use of computing resources.

1. Introduction

Advancements in DNA sequencing technology and genome assembly methods have greatly facilitated the acquisition of comprehensive and nearly comprehensive genome sequence data. These data can be utilized to investigate the functions and evolutionary positions of genomes as well as to gain insights into higher-level biological phenomena by integrating omics data. Now that determining the draft genome sequence has been becoming routine, the most critical step involved in the prediction of gene structures that encode proteins that play crucial roles in these phenomena. In the past, large-scale genome projects, such as those of Caenorhabditis elegans,¹Drosophila melanogaster,^2,3 and Homo sapiens,^4,5 based on first-generation sequencing technology, required significant numbers of personnel and extensive time for genome assembly and annotation, including gene structure determination. However, in recent years, genome projects have continued to be carried out on a small scale, and consequently, gene structure predictions are often performed with limited resources that lack sufficient data, making the task of prediction a formidable bottleneck in the research process. Typically, these predicted gene structures are deposited in public databases alongside functional information derived from additional investigation. However, many of these predictions are insufficient.^6,7 To accurately examine the biological characteristics of various species through secondary analysis, more precise and efficient annotation techniques are necessary.

To date, various gene structure prediction methods have been developed and can be broadly classified into four categories: ab initio-based, RNA-Seq-based, homology-based, and integrated methods. Ab initio-based methods, such as AUGUSTUS,⁸ SNAP,⁹ and GeneMark,¹⁰ predict gene structures using statistical models trained on highly reliable gene structure annotations from past research. RNA-Seq-based methods, however, utilize sequenced fragment data of mRNA to make predictions. These methods can be further divided into reference-based assembly methods, such as StringTie¹¹ based on mapping results from HISAT2¹² or STAR,¹³ and de novo-based assembly methods, including Trans-ABySS,¹⁴ SOAPdenovo-Trans,¹⁵ Trinity,¹⁶ and Oases.¹⁷ The former maps fragment sequences to genome sequences and constructs gene structures using position information, whereas the latter assembles fragments and makes spliced alignments against genome sequences. RNA-Seq-based methods are highly reliable because they utilize the mRNA information of the corresponding species, but they are ineffective for genes that are not expressed in the target sites/tissues. Homology-based methods make spliced alignments between the protein sequences of related species and their genome sequences. Examples of such tools include AAT,¹⁸ GeneWise,¹⁹ Exonerate,²⁰ genBlastG,²¹ and Spaln.²² These methods are more sensitive than RNA-Seq or ab initio-based methods when protein sequences of closely related species are abundant. However, in the opposite case, many correct predictions may be missed, and predictions cannot be made for genes that are evolutionarily distant and contain unique sequences. Integrated methods, such as MAKER,^23,24 EvidenceModeler (EVM),²⁵ Finder,²⁶ and EvidentialGene²⁷ integrate the outputs of other prediction methods to reconstruct gene structures as new prediction results. Previous research has demonstrated that this method can produce more comprehensive and accurate gene structures, including those that cannot be predicted by a single method.²⁵

Although this approach has gained widespread use in genome studies in recent years, it still suffers from the problem of capturing a significant amount of noise because of the large volume of input data and the potential for erroneous predictions. Commonly employed tools, such as MAKER and EVM, have been found to possess high prediction accuracy at the base and exon levels, but not at the gene level. One contributing factor is the high dependency on ab initio-based methods. Prior research has shown that these methods tend to ligate multiple genes (misligation) or the misclassify a single gene as multiple genes (missegmentation).²⁸ This can be attributed to the fact that at the time of MAKER and EVM development, RNA-Seq and protein sequence data were not as abundant as they are today. Recent technological advances and research advancements have resulted in the availability of large amounts of RNA-Seq data at low cost, as well as a significant increase in the amount of data registered in public databases, including proteins.²⁹ Therefore, it is worth considering the integration of outputs of ab initio, RNA-Seq, and homology-based methods in a unified manner for more accurate predictions.

In this paper, we present GINGER, a novel integrated method that aims to address the abovementioned issues and align them with the current research trends.

2. Materials and methods

2.1. Input data

In addition to target genome sequence data, GINGER input data consists of RNA-seq data and protein sequence data of closely related species as materials for gene structure prediction. Both genome sequence and protein sequence data were in Multi FASTA format, while RNA-Seq data were in FASTQ format.

2.2. Overview of pipeline

GINGER consists of a preparation phase, in which individual methods are employed to predict gene structures as inputs for the subsequent phase, and a merge phase, in which they are integrated to reconstruct the final gene structures (Fig. 1). In the latter phase, gene structure predictions for multi-exon and single-exon genes were performed separately. For multi-exon genes, calculations of exon/intron/intergenic scores, grouping of predicted exons from the preparation phase, splitting of groups at unreliable positions, and reconstruction of gene structure for the groups were sequentially performed. At this stage, exons of single-exon gene candidates predicted in the preparation phase were excluded and used for single-exon gene predictions. Conservative selection criteria were applied for these predictions, as it is challenging to increase the sensitivity while avoiding false positives.

Figure 1.

Overview of the GINGER pipeline. Material gene structures are obtained at the preparation phase. The gene structure prediction results output by this tool are reconstructed at the merge phase.

2.3. Preparation phase

In the preparation phase, gene structure prediction using RNA-Seq-based, ab initio-based, and homology-based methods was performed independently. For RNA-Seq-based methods, a genome-guided assembly-based method and a de novo assembly-based method were employed. The genome-guided assembly-based method first uses HISAT2 to map RNA-Seq reads to genome sequences and then StringTie to predict gene structures from the mapping results. For single-exon genes without strand information, open reading frames (ORFs) were searched for in each frame and the longest ORF was adopted. For multi-exon genes, strand information obtained by StringTie was adopted, and ORFs were extracted using TransDecoder.³⁰ The de novo assembly-based method used Trinity and Oases separately to assemble RNA-Seq data, and contigs of 300 bp or more were extracted from their results. Identical sequences were removed using CD-HIT.³¹ The obtained sequences were aligned to the target genome sequences using GMAP,³² and gene structures with at least 95% identity were extracted. The results from genome-guided assembly-based method and de novo assembly-based methods were applied to TransDecoder, and the output ORFs of 90 AA or more were the results of the preparation phase.

In the ab initio-based method, a training dataset was constructed from a random sample of 1,000 gene structures obtained using the genome-guided assembly-based method. Gene structures whose exons overlapped with repetitive sequences searched by the RepeatMasker³³ program in the genome sequences were removed from the dataset. Note that in GINGER, repetitive sequences do not include low-complexity repeat sequences such as microsatellites or simple repeat sequences. Using this dataset, statistical models of gene structures were trained by Augustus and SNAP, and gene structure prediction was performed using the trained models.

In the homology-based method, Spaln is used to align the protein sequences of multiple closely related species to their genome sequences. The alignment results in which stop codons exist were removed from the final output.

2.4. Merge phase

Step 0: Exon scoring

To reconstruct the gene structures, scores for exons obtained in the preparation phase were calculated. The exon score S_exon is given by:

$${\displaystyle \begin{array}{l}{\displaystyle S_{exon}=p_{exon}\times w_{exon},}\end{array}}$$ (1)

where p_exon represents the exon potential and w_exon represents the weight for each method. p_exon is a value in the range (0,1] and is calculated for each method (as detailed in Supplementary Table S1). These exon potential decision schemes were designed to reflect the correlation between the parameter value selected for each prediction method and the accuracy of exon candidates measured from actual nematode, rice, and zebrafish data (Supplementary Fig. S1). Broadly speaking, RNA-seq-based methods do not show a clear correlation between coverage or identity and exon precision, whereas homology-based methods and ab initio-based methods show a high correlation between identity or score and exon precision, and thus, exon potential decision functions reflecting these trends are established for each method.

Step 1: Multi-exon gene prediction

In this step, the gene structures from various prediction methods obtained in the preparation phase are grouped and then merged and reconstructed into a single predicted gene structure within those groups. Overlapping gene structures are grouped based on their genomic positions; however, if there are significantly longer introns, they are subdivided. Every gene structure prediction method used in the preparation phase may produce output results with extremely long introns as artefacts specific to the methods. In GINGER, it is possible to improve the prediction performance and reduce the calculation time by splitting them into subgroups in advance (Fig. 2A). To achieve this idea, base-by-base scores within the target genomic region are calculated. If the score falls below a threshold value, it is reasonable to assume that those low-score regions are not likely to be included as true genes, and the target region is split at that position. The base-by-base scores s_base are given by the equation:

Figure 2.

Processing procedure in multi-exon gene prediction. (A) Calculation of gene and base scores for the split groups. Base scores are calculated by summing gene scores overlapping on their genomic regions, and groups are split at intron regions under the threshold t_gene. (B) Calculation of intron and intergenic scores with exon scores. Exon scores were calculated using Step 0. Exons that overlapped completely in their genomic regions were merged. (C) Reconstruction of gene structures using dynamic programming.

$${\displaystyle \begin{array}{l}{\displaystyle s_{base}=\sum s_{gene}\mathrm{a}\mathrm{n}\mathrm{d}{s}_{gene}=\sum (s_{exon}\times l_{exon})/\sum l_{exon},}\end{array}}$$ (2)

where s_gene, s_exon, and l_exon represent the gene score, exon score (defined in Step 0), and length of the exon, respectively. The threshold t_group is given by:

$${\displaystyle \begin{array}{l}{\displaystyle t_{group}=s_{base\mathrm{\_}max}\times 0.25,}\end{array}}$$ (3)

where s_{base_max} is the maximum value of the base score in the target genomic region.

In addition, the target groups are split according to unusually long introns that cannot be addressed by the abovementioned conditions. A specific cause of artefacts is that an extremely long intron is sandwiched to align the terminal exons of several bases. Another study reported that intron length could be fitted with a convolution of one to three extreme values or lognormal distributions.³⁴ In GINGER, the upper limit of intron length is determined from the intron length distribution of the gene structure prediction results obtained in the preparation phase, and the grouped gene structures are split into subgroups using this information. Supplementary Figures S2 and S3 show the distribution of intron lengths obtained from the curated gene structures and the detailed method for determining the critical intron length.

After grouping, identical exons output by each prediction method were merged into a single exon (Fig. 2B). Their scores, denoted as s_{merged_exon}, were calculated by:

$${\displaystyle \begin{array}{l}{\displaystyle s_{merged\mathrm{\_}exon}=\sum (s_{exon\mathrm{\_}max}\times l_{merged\mathrm{\_}exon}),}\end{array}}$$ (4)

where s_{exon_max} is the largest score outputted by each prediction method and l_{merged_exon} is the length of the merged exon. s_{exon_max} was normalized to the range (0, 1). Intron and intergenic scores were then calculated with reference to adjacent exon scores for both ends, as it is assumed that the reliabilities of the intron and intergenic regions are dependent on the reliabilities of the exons. Specifically, the intron score s_intron is given by the minimum score of the neighboring exons, s_{meged_exon}. The intergenic score s_intergenic is given by $s_{intergenic}=s_{merged\mathrm{\_}exon}$ if an adjacent exon starts with a start codon or ends with a stop codon, while $s_{intergenic}=1-s_{merged\mathrm{\_}exon}$ if an adjacent exon is partial. The overlapping intron and intergenic regions outputted by each method are merged, and their scores s_{merged_intron} and s_{merged_intergenic} are given by:

$${\displaystyle \begin{array}{l}{\displaystyle s_{merged\mathrm{\_}intron}=\sum ({\displaystyle \underset{bothends}{\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}}}\left(s_{intron}\right)\times l_{intron})}\end{array}}$$ (5)

and

$${\displaystyle \begin{array}{l}{\displaystyle s_{merged\mathrm{\_}intergenic}=\sum ({\displaystyle \underset{bothends}{\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}}}(s_{intergenic})\times l_{intergenic}),}\end{array}}$$ (6)

where l_intron and l_intergenic are their lengths (see Supplementary Fig. S4 for examples of the calculation).

Gene structure reconstructions in target genomic regions are performed as optimization problems that seek combinations of merged exons, introns, and intergenic regions that yield the highest score while adhering to constraints that the positional relationship on the genome sequence is not disturbed and that the reading frame consisting of the merged exons is consistent. Dynamic programming was used to solve this problem (Fig. 2C). The path score s_path was calculated as follows: If the pth exon was a terminal exon,

$${\displaystyle \begin{array}{l}{\displaystyle s_{path,f-c}=max(\mathrm{m}\mathrm{a}\mathrm{x}(s_{path,f-p})+s_{merged\mathrm{\_}exon,c}+s_{merged\mathrm{\_}intergenic,p-c})}\end{array}}$$ (7)

If the pth exon is not the terminal exon,

$${\displaystyle \begin{array}{l}{\displaystyle s_{path,f-c}=max(\mathrm{m}\mathrm{a}\mathrm{x}(s_{path,f-p})+s_{merged\mathrm{\_}exon,c}+s_{merged\mathrm{\_}intron,p-c})}\end{array}}$$ (8)

where f, p, and c are indices for the first, previous, and current exons, respectively (see Supplementary Fig. S5 for details). If there is a stop codon in the middle of a gene followed by a start codon, the gene is split into a separate gene structure at that point. Additionally, partial genes were also predicted, as it is not essential to have a start codon or stop codon at the beginning and end of the gene structure.

Gene score s_gene are given to the gene structures obtained through dynamic programming, and those above a threshold were taken as the final result. s_gene is calculated as:

$${\displaystyle \begin{array}{l}{\displaystyle s_{gene}=s_{path}/l_{group},}\end{array}}$$ (9)

where l_group is the length of the group region to which the reconstructed gene structure belongs.

Step 2: Single-exon gene prediction

Single-exon gene prediction often generates false positives represented by pseudogenes, making it difficult to improve the accuracy compared to multi-exon gene prediction. For single-exon genes, their s_gene was obtained by summing the s_exon for exon regions supported by at least two prediction methods. The results were extracted by applying the threshold determined in the multi-exon gene section.

2.5. Implementation

GINGER is implemented as executable code in Nextflow, a workflow management system.³⁵ This enables efficient utilization of computational resources. In addition to the gene structure prediction method implemented in the Preparation phase of GINGER, there may be methods worth using for reading the results into the Merge phase. Accordingly, a function to import the gene structure prediction results created by the user into GFF and to set their weights is prepared.

2.6. User-configurable parameters

The GINGER system has two user-configurable parameter sets. One set includes the weight coefficients w_exon for each prediction method in determining the exon score (see Equation (1)), and the other is the threshold for outputting final gene candidates that have s_gene scores greater than the threshold value. These parameters can be set by the user at the time of GINGER execution, but the default values and automatic determination are set as the default ones, respectively.

For defaults weight coefficients w_exon, they were set as follows: 1.6 for the genome-guided assembly-based method, 1.4 for the de novo assembly-based method, 1.8 for the ab initio-based method (Augustus), 1.2 for the ab initio-based method (SNAP), and 2.0 for the homology-based method. These default values were determined as intermediate values between the three sets of parameters (Supplementary Table S2) optimized to achieve the highest sensitivities using a genetic algorithm for the three model species nematode, rice, and zebrafish (as detailed in Supplementary Figs. S6 and S7).

The default threshold value for s_gene was determined automatically from the distribution of _Sgene for all the reconstructed gene structures. Analysis of the score distribution using curated genes revealed that gene candidates that formed peaks on the low-score side were mostly false positives. Therefore, a threshold can be set to exclude the peak from the score distribution. Supplementary Figure S8 shows the observation and threshold setting methods.

2.7. Benchmark test

To benchmark GINGER against other tools, we utilized genome sequence, RNA-Seq, and protein sequence data from representative model species, such as nematodes, fruit flies, rice, zebrafish, and humans. Specifically, for nematodes, in addition to the Caenorhabditis elegans genome sequence data and RNA-seq data, protein sequence data of Caenorhabditis briggsae, Pristionchus pacificus, Drosophila melanogaster, and Homo sapiens were used. For fruit flies, in addition to the D. melanogaster genome sequence data and RNA-Seq data, protein sequence data of Drosophila mojavensis, Bombyx mori, C. briggsae, and H. sapiens were used. For rice, in addition to the Oryza sativa genome sequence data and RNA-Seq data, protein sequence data of Oryza brachyantha, Panicum hallii, Zea mays, and H. sapiens were used. For zebrafish, in addition to the Danio rerio genome sequence data and RNA-Seq data, the protein sequence data of Cyprinus carpio, Oryzias latipes, Takifugu rubripes, and H. sapiens were used. For humans, in addition to genome sequence data and RNA-seq data in H. sapiens, protein sequences of Mus musculus, Equus caballus, Canis lupus, and Sus scrofa were used. All data were downloaded from the NCBI FTP site (see Supplementary Table S3 for more details).

MAKER and EVM were selected as tools for comparison with GINGER because they have been widely used in recent genomic studies. To ensure fair evaluation as an integrated tool, the same input data produced by the various tools from the preparation phase to GINGER, MAKER, and EVM were utilized. Specifically, the input data for MAKER comprised RNA-Seq assembly results from Trinity, ab initio statistical models learned by Augustus and SNAP, and the same protein sequences used for homology-based predictions in GINGER. For EVM, gene structures obtained through genome-guided assembly-based methods, de novo assembly-based methods, ab initio-based methods (Augustus and SNAP), and homology-based methods were used. (Supplementary Table S4). In the execution, each parameter was set as follows. For GINGER, all parameters were set to default (default weight coefficients w_exon and automatically determined thresholds for _Sgene were used). For EVM, the parameters recommended in the original paper (genome-guided: 1, de novo assembly: 10, Augustus: 0.3, SNAP: 0.3, homology: 1) and those based on GINGER weights (genome-guided: 10, de novo assembly: 4, Augustus: 9, SNAP: 4, homology: 8) were preliminarily tested for several cases. In addition, we also tested two patterns of giving prediction results based on RNA-seq: one is to give the results as they are labelled as ‘TRANSCIPT’ while the other involved applying TransDecoder and presenting the results as ‘OTHER_PREDICTION’. In both cases, the latter approach produced superior results in terms of parameter selection. Furthermore, when considering the manner in which the RNA-seq results were presented, the results obtained with TransDecoder yielded better outcomes (Supplementary Table S5). Therefore, we opted to utilize the TransDecoder results with GINGER-based parameters for the EVM benchmarks. MAKER has no particular parameters to be set and it was executed by default.

In this benchmark, curated gene structures maintained by each genome consortium were regarded as the ‘correct answers’, and the accuracy of the predictions at the base, exon, and gene levels were evaluated. At the base level, accuracy was evaluated based on the degree to which the correct and predicted exon regions match base-by-base, and at the exon level, it was evaluated to determine whether the ends of the exon region are completely matched; at the gene (transcript) level, whether the exon/intron structures were completely consistent. If isoforms were present, the longest isoforms were used as the correct answers. Sensitivity (Sn), precision (Pr), and F-score were used as evaluation indices. Both sensitivity and precision were obtained using GffCompare³⁶ with following options: ‘--strict-match -e 0’.

3. Results

3.1. Benchmark results of individual methods in the preparation phase

First, the results of individual gene prediction tools in the preparation phase are shown in Supplementary Table S6, Figure 3, and Supplementary Figure S9. Basically, we can confirm that both Sn and Pr decrease at the nucleotide level, exon level, and gene level, in that order, in the results of all tools for all species. Among the tools, ab initio-based (Augustus) and genome-guided assembly-based method, recorded the highest F-score for all species. However, the F-score decreased with increasing genome size, and this trend was significantly observed in the prediction performance of Augustus at the gene level. At the exon level, Augustus recorded the highest prediction performance for all species, with only a slight degradation (about 10 points) with increasing genome size, whereas at the gene level, the F-score marked about a 40 point decrease as in zebrafish (1 Gbp) and human (3 Gbp). Among all species cases, ab initio methods yielded average Sn and Pr while the other methods, especially the de novo assembly-based and homology-based methods have higher Sn than Pr. The reason for this is assumed to be that the former does not obtain RNA-seq assembly well in loci where coverage is low. The homology-based approach generally yielded lower Prs, significantly biased toward Sn, and in some cases marked the highest Sn between all prediction methods. This is likely a reflection of the attempt to increase coverage, even if false positives are higher, by including much more distantly related species data as input to the homology-based methods.

Figure 3.

Prediction performance of the benchmark test (fruit flies). The results are shown at the nucleotide, exon, and gene levels for individual prediction methods and integrated tools calculated from benchmark test results using a fruit flies data set.

The details of the relationship between exon score and exon prediction performance are shown in Supplementary Figure S10, where A and B show the relationship between exon potential and Sn and Pv (the higher the exon potential, the higher the Pv), and C and D show the relationship between exon score calculated by applying weight coefficients to the exon potential and Sn and Pv. Since only weights are multiplied, the basic trend does not change, but it can be observed that low coefficients are applied to SNAP and RNA-seq-based methods, for which it is difficult to find a clear correlation between exon prevalence and exon potential. This might be originally ascribed to the low correlation between exon Pv and parameters to determine exon potential (Supplementary Fig. S1). That means the discrimination performance between correct and incorrect answers based on exon potential is not high in these tools. These results confirm that the determination concept of exon potential in GINGER and genetic algorithm-based weight coefficients are working correctly.

3.2. Benchmark results of integrated tools

Benchmark results for the integrated tool are shown in Table 1, Figure 3, and Supplementary Figure S9. As detailed in the Methods section, we compared the performance of the three integration tools, GINGER, MAKER, and EVM, using the same input data as the results obtained from each prediction tool shown in Section 3.1. GINGER parameters are default values, and EVM parameters are the set of parameters that performed well in the preliminary analysis.

Table 1.

Benchmark test results of the integrated tools

	Correct gene (#)	Predicted gene (#)	Base level			Exon level			Gene (transcript) level
			Sn	Pr	F-score	Sn	Pr	F-score	Sn	Pr	F-score
Nematode (Caenorhabditis elegans) 100.3Mba
GINGER	19,984	19,675	92.2	93.2	92.7	85.8	86.9	86.3	58.5	59.4	58.9
EVM		17,694	88.8	92.2	90.5	81.6	84.2	82.9	49.6	56.1	52.7
MAKER		18,056	80.9	89.6	85.0	62.1	54.4	58.0	24.3	26.9	25.5
Rice (Oryza sativa) 374.4 Mba
GINGER	28,721	28,021	87.8	93.2	90.4	86.9	89.0	87.9	66.2	67.8	67.0
EVM		38,168	92.5	83.1	87.5	87.2	77.6	82.1	64.7	48.7	55.6
MAKER		33,487	81.2	81.9	81.5	69.0	59.4	63.8	21.8	18.7	20.1
Zebrafish (Danio rerio) 1,373.5 Mb *
GINGER	25,536	25,579	90.4	93.0	91.7	90.3	90.3	90.3	58.4	58.3	58.3
EVM		33,239	89.4	84.6	86.9	88.6	79.1	83.6	51.0	39.2	44.3
MAKER		30,967	72.5	84.2	77.9	54.2	58.8	56.4	11.7	9.6	10.5
Fruit Fly (Drosophila melanogaster) 143.7 Mb
GINGER	13,963	12,347	90.2	98.1	94.0	84.5	91.4	87.8	64.6	73.1	68.6
EVM		13,042	88.1	94.5	91.2	81.1	83.3	82.2	58.9	63.1	60.9
MAKER		12,640	84.8	95.9	90.0	69.9	72.3	71.1	40.0	44.2	42.0
Human (Homo sapiens) 3,099.4 Mb
GINGER	21,658	20,811	84.8	94.5	89.4	83.2	91.1	87.0	45.3	47.2	46.2
EVM		36,275	88.7	77.7	82.8	85.4	77.1	81.0	39.6	23.6	29.6
MAKER

Sn is sensitivity: $\mathrm{S}\mathrm{n}=\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{s}/\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{s}$. Pr is precision: $\mathrm{P}\mathrm{r}=\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{s}/\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{s}$. F-score is a harmonic mean of precision and recall: $\mathrm{F}\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2(\mathrm{P}\mathrm{r}\times \mathrm{S}\mathrm{n})/(\mathrm{P}\mathrm{r}+\mathrm{S}\mathrm{n})$. The calculation for humans with MAKER takes time (more than two months), and its result has not been obtained.

*Species used to determine the default weighting coefficients between prediction tools in GINGER.

As with the results of the individual gene prediction tools, it can be confirmed that both Sn and Pr decrease in the order of base level, exon level, and gene level in the results of all types of integrated tools, and that both Sn and Pr decrease with increasing genome size. Moreover, EVM and GINGER achieved higher F-scores than either of the stand-alone predictions in all species, indicating the effectiveness of the integration. In terms of comparisons between integrated tools, GINGER outperformed the other tools in all species and at all levels of accuracy. In particular, differences in Pr and F-score were large at the exon and gene level, and it was clear that the ratio of correct predictions with GINGER was relatively higher than that of others. This difference widened as genome size increased, from about 7 points of F-score difference at the gene level with EVM, the second-best performer in nematode, to 14–16 points in zebrafish and human. GINGER was notable for its excellent performance in predicting entire gene structures and complete exons. This trend is not a phenomenon that occurs only for the species referenced in the determination of weight coefficients between tools in GINGER but is seen in the pure benchmark fruit flies and human data as exactly the same trend.

3.3. Comparison of individual predictions with each tool

GINGER was developed with the aim of improving accuracy at the gene level, and the benchmark results show that it was achieved. To analyze the prediction accuracy of each tool in further detail and capture trends, the overlap of correct answers among the three tools was examined at the base, exon, and gene levels (Table 2). As can be seen from the results, at the base and exon levels, a high percentage of correct bases (67.0%) and exons (50.3%) were predicted by any of the tools, while the accuracy at the gene level was very low (8.8%) for all tools. Similarly, at the base and exon levels, the number of cases where predictions were not made by any of the tools was low (5.1% and 6.3%, respectively), while it reached as high as 36.4% at the gene level. Furthermore, when we examine the prediction results of GINGER and EVM, we find that the percentage of correct predictions shared by both tools is notably high, reaching 86.3% at the base level and 86.2% at the exon level. However, at the gene level, this percentage decreases to 46.9%. These results highlight inconsistencies in gene-level predictions, with GINGER exclusively predicting 10.1% of genes, in contrast to EVM’s solitary prediction of only 3.7% of genes. A detailed analysis of the genes that could be correctly predicted only by GINGER revealed that one of its characteristics to be that in many cases, the first exon was not correctly predicted by the other integration tools. Of the 2,590 genes that only GINGER was able to predict, 1,682 genes were incorrectly predicted by EVM, and 1,871 genes were incorrectly predicted by MAKER for the first exon. Figure 4 provides examples of successful predictions made solely by GINGER, where EVM and MAKER failed because of connections between multiple correct gene structures and the inclusion of extra exons. These cases also highlight the significance of proper grouping of gene structures.

Table 2.

Correspondence of correct answers for each tool

Prediction tools			Base level			Exon level			Gene (transcript) level
GINGER	EVM	MAKER	Number or base (#)	(%)		Number of exons (#)	(%)		Number of genes (#)	(%)
			29,803,416	67.0	*	123,516	50.3	*	2,235	8.8
			8,869,898	19.9		88,082	35.9	*	9,732	38.1	*
			1,037,428	2.3		4,444	1.8		374	1.5
			748,740	1.7		2,244	0.9		115	0.5
			720,635	1.6		5,916	2.4		2,590	10.1	*
			358,221	0.8		3,232	1.3		948	3.7
			661,240	1.5		2,507	1.0		256	1.0
			2,274,382	5.1		15,502	6.3		9,286	36.4	*
Total			44,473,960	100.0		245,443	100.0		25,526	100.0

A black cell in each column for each tool means it is correct. * means its values are notable.

Figure 4.

Examples of successful predictions against zebrafish genome with GINGER. In each row, from the top, the correct gene structures, ones by GINGER, ones by EVM, ones by MAKER are arranged. These are visualized with igv.³⁹ (A) Predicted gene structures with EVM are concatenated. (B) Predicted gene structures with MAKER are concatenated. (C) Extra exons are added outside the correct gene structure.

3.4. Benchmark results for GINGER in the case of giving equal exon potential

For exploring the cause of GINGER’s high predictive performance, we ran benchmark tests that cut off the function to calculate exon potential depending on various statistical parameters. Specifically, we set the exon potential of all inputs to 1.0. The results are shown in Supplementary Table S7. Compared to GINGER using the default values, the prediction performance is reduced for all species, with 2.2–3.6 and 3.2–8.5 point reductions in F-score for exon level and gene level, respectively. This result suggests that the exon potential based on the statistical parameters set for each prediction tool is responsible for GINGER’s high performance.

4. Discussion

We successfully developed a novel integrated gene structure prediction tool, GINGER, for higher eukaryotic species. As shown above, the benchmark test utilizing a wide range of species indicated that GINGER outperformed other integration-based gene prediction tools, particularly at the gene and exon levels. We were also able to show from the analysis of the predicted results comparison and some additional benchmark tests why GINGER is able to achieve high performance. Because of the complex interplay of various functions and parameters, it is difficult to clearly show which functions are directly related to the high performance, but we were able to identify that some GINGER-specific functions are likely responsible for the overall effect.

The first one is the subgrouping function for determining gene regions from predicted exons. From the analysis of genes that could be predicted only with GINGER, we found that the ratio of correctly predicted first exons was higher than with other integration tools, as well as gene predictions that resulted in an appropriate number of exons per gene. In fact, in the overall zebrafish benchmark results, the number of cases where multiple genes were fused was 627 for GINGER, compared to 1,232 for EVM and 1,254 for MAKER. This data also confirms the contribution of the subgroup function in GINGER to the high prediction performance.

The next feature that might be contributing GINGER’s high prediction performance is the exon potential given for each exon. In GINGER, for each prediction tool, the exon potential is based on statistical parameters such as the identity between the protein sequence used for prediction and the target genome. In EVM, a score is given for the category of each method, but not for each predicted exon. It is directly shown that GINGER’s exon potential is correlated with the exon prevalence, but it is also shown to contribute to the final predicted results, as the benchmark test with the same exon potentials worsened the results. It is assumed that GINGER’s high performance is achieved by the combination of these functions.

On the other hand, the original EVM paper obtained better results in benchmarking with the same species (rice, human) as this paper. However, this might be largely due to the difference in the genomic data used. For example, for rice, EVM was applied to extracted genes and the flanking 30 kb region from the genome. In contrast, when the whole genome is used, as is in this paper, the performance is naturally inferior to that of the benchmark applied only to the genomic region around the gene, which is also a more realistic benchmark and closer to the actual problem.

However, there is still potential for development. GINGER was designed to achieve high prediction accuracy for draft genomes; however, it is challenging to obtain accurate alignments in the presence of gaps in the gene regions. Although the number of draft genome sequences consisting of numerous small fragments is declining as genome assembly using long-read technologies represented by Pacific Biosciences and Oxford Nanopore Technologies sequencers has become more mainstream, completely eliminating assembly errors remains a challenge. To resolve the problem of missing regions of a gene in the genome due to gaps or misassemblies, GINGER currently handles it by detecting extremely long introns in the merge phase, step 1. However, in terms of fundamental resolution and portability, it is preferable to address this issue during the preparation phase by improving alignment tools or improving pre- and/or post-processing. Slightly different from the issue of the integration tool itself, achieving even more accurate alignment of mRNA/protein sequences with genome sequences will also be an issue to be addressed in the future. For example, it may be possible to obtain a more accurate alignment by paying close attention to the maximum intron length. By using appropriate settings for each species, it is possible to avoid long introns due to artefacts and avoid predicted gene fusions. Furthermore, in addition to the tools mentioned in this paper, new alignment tools have appeared even in recent years, for example, miniprot.³⁷ The use of these tools may help solve the problem. We recognize that there are many tools worth considering for incorporation in GINGER, including alignment tools and ab initio methodology tools such as GeneMark. Incorporating new tools into the preparation phase of GINGER requires pre- and post-processing and performance tuning, warranting caution. Therefore, here, we provide a function to import structure prediction results by GFF and a function to set their weights. In future versions, we would like to start by considering support for other tools. Finally, it is considered useful to predict the gene structures of higher eukaryotes given the presence of various isoforms. RNA-Seq-based methods typically predict the gene structures of isoforms, and an integrated method such as FINDER enables this prediction. In GINGER, suboptimal solutions of gene structures from the merge phase, step 1, can serve as candidates, and the results from RNA-Seq-based methods can support these candidates as output. Alternatively, it may be possible to address the isoform issue by using the prediction output from a tool such as GINGER as input for a tool such as PASA.³⁸

Although there are still many issues that need to be addressed, there are many genes that have been correctly predicted only by GINGER. Another feature of GINGER is the efficient use of computer resources in Nextflow. MAKER, which is widely used, requires an extremely large amount of computing time, and unfortunately, benchmarking against the human genome cannot be completed in two months. In contrast, GINGER requires less than one day to perform the merge phase. From these points of view, GINGER is expected to be very useful as a stand-alone tool or at least as a complementary tool to other gene prediction methods.

Funding

This work was supported by JSPS KAKENHI grants 22H04925 (PAGS), 16H06279 (PAGS), and 22H02598 from the Ministry of Education, Culture, Sports, Science and Technology of Japan.

Data Availability

GINGER is freely available from GitHub (https://github.com/i10labtitech/GINGER) or Anaconda package (https://anaconda.org/i10labtitech/ginger) under the GNU General Public License (GNU GPL) ver. 3. Prediction results of gene structure obtained by each tool used to calculate the accuracy index for the benchmark, as well as setting information of commands and options are available from Zenodo (https://zenodo.org/record/8126530).