Comparative study of bioinformatic tools for the identification of chimeric RNAs from RNA Sequencing

ABSTRACT

Chimeric RNAs are gaining more and more attention as they have broad implications in both cancer and normal physiology. To date, over 40 chimeric RNA prediction methods have been developed to facilitate their identification from RNA sequencing data. However, a limited number of studies have been conducted to compare the performance of these tools; additionally, previous studies have become outdated as more software tools have been developed within the last three years. In this study, we benchmarked 16 chimeric RNA prediction software, including seven top performers in previous benchmarking studies, and nine that were recently developed. We used two simulated and two real RNA-Seq datasets, compared the 16 tools for their sensitivity, positive prediction value (PPV), F-measure, and also documented the computational requirements (time and memory). We noticed that none of the tools are inclusive, and their performance varies depending on the dataset and objects. To increase the detection of true positive events, we also evaluated the pair-wise combination of these methods to suggest the best combination for sensitivity and F-measure. In addition, we compared the performance of the tools for the identification of three classes (read-through, inter-chromosomal and intra-others) of chimeric RNAs. Finally, we performed TOPSIS analyses and ranked the weighted performance of the 16 tools.

Introduction

Chimeric RNAs, or fusion transcripts, are RNA transcripts composed of RNA fragments encoded by two separate genes. Chimeric RNAs have been known to be the products of gene fusions created via chromosomal rearrangement, and are often considered ideal biomarkers/drug targets for cancer [1–7]. Recently, more and more chimeric RNAs are being discovered in non-cancerous cells and tissues, and may be produced due to intergenic splicing [8–15], establishing a new paradigm that chimeric RNAs could represent a new means to expand the functional genome. In the past, chimeric RNAs were discovered as products of gene fusions at the DNA level by traditional techniques including Southern blot, and fluorescence in situ hybridization, which are tedious and slow. Modern techniques, especially RNA-Seq allows for quick, easy, and high throughput discoveries of chimeric RNAs. Over 40 software tools have been developed to facilitate the process [16]. As a result, thousands, if not more, are being reported, and several databases have been built to incorporate these chimeras [17–19]. Most of the methods use an initial alignment step to identify discordant reads mapping to two different genes, followed by series of filtering and/or realignment steps to identify fusion events. The alignment can be done using either genome or transcriptome or both as reference. The filtering steps may include criteria like the distance between fusion partners, the minimum number of supporting reads, homology-based filters, etc. [16]. Some methods utilize k-mer-based alignment to decrease the time required to run the software [20].

However, with a large number of tools being developed comes the confusion as to which tool is the best to serve the distinct purpose of each end user. In addition, the developer of an individual tool tends to compare the performance of their tool with others on a selected dataset, making some comparisons biased. A few efforts have been made to benchmark chimeric RNA prediction tools. The first of these efforts was done by Carrara et al. [21] in 2013, wherein they benchmarked six methods (ChimeraScan and ShortFuse were not included, due to technical problems in running the software). They used a simulated positive dataset of 50 true events and a negative dataset having no fusion events, and observed that FusionMap had best trade-off between true and false-positive predictions. They further examined the performance of only FusionMap on a real dataset. Liu et al. [22] in 2015 benchmarked 15 methods on a synthetic positive dataset having 150 true events, and also used real datasets from cancer cell/tissue RNA sequencing. The authors identified SOAPfuse as the best performing method on the synthetic positive dataset. Furthermore, in 2016, Kumar et al. [23] benchmarked 12 methods on the same positive and negative datasets as used by Carrara et al., and also used a mixed simulated dataset as well as a real dataset from the study done by Qin et al. [24]. These previous benchmark studies were important as they did provide some insights into the performance of different tools. However, the conclusions of these studies are not always consistent. Additionally, they are outdated, since the list of chimeric RNA prediction methods has significantly expanded in the past three years. In addition, the number of true fusion events used to compare the performance of different methods has changed as more fusions have been discovered in real datasets. For instance, the list of 27 true fusions (Edgren dataset) [25] used by Liu et al., has now been expanded to 99 in subsequent studies. This difference in the number of true fusions may dramatically affect the performance evaluation of the tools.

This benchmarking study is thus designed to compare the performance of both new and old top performing software tools using two stimulated and two real datasets. We included ten chimeric fusion RNA prediction software packages that have been developed since the latest benchmarking study. We also included the top four best performing methods from each of the previous three benchmarking studies. This resulted in a total of 19 software tools. Out of the 19, we failed to run GFusion, FusionMap and FusionFinder, either due to the software being unavailable or errors in installation. In the end, we were able to benchmark 16 fusion RNA prediction methods. The list includes seven top performers from previous benchmarking studies: SOAPfuse [26], MapSplice [27], EricScript [28], ChimerScan [29], FusionCatcher [30], JAFFA [31], and TopHat-Fusion [32]; and nine new methods since 2016: pizzly [33], FuSeq [34], InFusion [35], Arriba [36], INTEGRATE [37], STAR-Fusion [38], STARChip [39], ChimPipe [40], and ChimeRScope [20].

Materials and Methods

Fusion RNA prediction software(s)

A list of ten fusion RNA prediction software packages were collected, which were published/developed since year 2016. These software packages are GFusion [41], pizzly [33], STARChip [39], ChimeRScope [20], ChimPipe [40], STAR-Fusion [38], InFusion [35], Arriba [36], FuSeq [34] and INTEGRATE [37]. We also collected top four best performing methods based on each of the previous three benchmarking studies. Carrara et al. [21] benchmarked six fusion RNA prediction methods, and identified FusionFinder [42], TopHat-Fusion [32], FusionMap [43] and MapSplice [27] as the top best methods based on their sensitivity to identify 50 true positive events using simulated dataset. Liu et al. [22] benchmarked 15 fusion RNA prediction methods, and identified SOAPfuse [26], EricScript [28], FusionCatcher [30], and ChimeraScan [29] as the top best methods based on their sensitivity to identify 150 true positive events using simulated dataset. In a previous study (Kumar et al.) [23], our group benchmarked 12 fusion RNA prediction methods, and identified JAFFA [31], MapSplice, SOAPfuse and EricScript as the top best methods based on their sensitivity to identify 50 true positive events using the same simulated dataset as used by Carrara et al. The combination of top performing methods from three benchmark studies results in nine unique fusion RNA prediction methods. The final list therefore includes these nine methods, as well as ten methods developed recently, making a total of 19 software. Out of 19 software, we failed to successfully run GFusion, FusionMap and FusionFinder either due to unavailability, or errors in installation. In this study, we benchmarked a total of 16 fusion prediction software packages.

We used genome and annotation files based on hg19 for running all of the software, except FusionCatcher and ChimeRScope where we used hg38 because we were unable to get, or generate database files for hg19 for these two methods. For all of the software, we used default parameters (or default configuration file) to run their pipeline in order to make the comparisons fair. We identified six software tools (Arriba, FuSeq, JAFFA, MapSplice, STARChip and TopHat-Fusion) which used distance between parental genes as a criteria to discard read-through class of chimeras. To include read-through chimeras in their prediction, we set the distance parameter to 0 for these six tools. Since Tophat-Fusion failed to predict any chimeric RNAs with distance parameter set to 0 (possibly due to technical issues), we set this parameter at 1000 so that we can get predicted chimeric RNAs from Tophat-Fusion. For ChimeRScope and pizzly, which use k-mer(s) for prediction, we used k = 17 and k = 31 as suggested in their manual/instructions. For FuSeq, which also uses k-mer, we used k = 21 for reads shorter than 100 bp while we used k = 31 for reads longer than 100 bp. The list of 16 fusion RNA prediction software packages benchmarked in this study, and the detailed information including versions of the software used, and commands to run all of the software is given in supplementary file 1. The output of all the fusion prediction methods on all of the datasets is provided in the following link (https://drive.google.com/file/d/1ole6oOh2OvfEkmGPYRB6lLUZ8xJ1ueeI/view?usp=sharing).

Datasets

We used four datasets to benchmark all of the 16 software packages. Two of these datasets are simulated and the other two are real datasets. Description of the datasets are given below.

InFusion simulated dataset: This dataset [35] contains a total of 100 true positive fusions taken equally from five different classes of fusions which are: (i) both parental genes having breakpoint at exon boundary, (ii) one or both parental genes having breakpoint inside an exon, (iii) one or both parental genes having break point inside an intron, (iv) one parental gene originating from an intergenic region and (v) different isoforms of chimeric RNAs having breakpoint at exon boundary. We discarded the fusions from one class (i.e. 20 fusions) wherein either of the parental genes were originating from an intergenic region because the gene name of either partner genes of the fusion was not available. The rest 80 fusions represent 68 gene pairs, which were used as true positive events to benchmark the methods on this dataset. The read length of this paired-end dataset is 75 bp with a total number of 307,920 reads. This dataset was downloaded from https://bitbucket.org/kokonech/infusion/downloads/InFusion_test_dataset_02.tar.gz

ChimPipe simulated dataset: This dataset [40] contains a total of 250 true positive fusion RNA events (250 pairs). The dataset was simulated using the ChimSim module (part of ChimPipe package) wherein hg19 genome and Gencode v19 protein-coding genes were used as input to generate the fusion transcripts, as well as 102,149 normal transcripts, including transcripts from the parental genes of the 250 events. From the same set of fusion and non-fusion transcripts, they generated three paired-end libraries with read lengths of 50, 76 and 101 bp, having total number of reads as 64,640,140, 42,184,900 and 31,460,720, respectively. In addition, these 250 fusions have been equally selected from five different fusion categories, which are ‘read-through’, ‘intra-chromosomal’, ‘inverted’, ‘interstrand’, and ‘inter-chromosomal’. This dataset was made in an attempt to mimic the real dataset.

Qin et al., real dataset: This dataset [24] contains a total of 62 true positive fusion events (62 gene pairs), identified from human prostate cancer cells (LNCaP). Qin et al. first used the SOAPfuse software to predict fusions on this dataset, and experimentally validated 62 fusions containing a mixture of read-throughs, inter-chromosomal, and intra-others. There are six total runs in this paired-end dataset with two runs having read lengths of 101 bp, while four runs having read lengths of 50 bp. For the purpose of our benchmark, we used only the two runs (SRR1657556 and SRR1657557) having read lengths of 101 bp with a total of 67,070,290 and 62,840,006 reads. Prior to running software on this dataset, we used the NGSQC Toolkit [44] with default parameters, to retain only high-quality reads, in the same manner as our previous benchmark paper [23]. Finally, after filtering, the two runs had total numbers of reads being 62,117,396 and 58,070,054, respectively.

Edgren real dataset: This dataset [25] is one of the standard datasets used by different methods to show their performance. The dataset is from the breast cancer cell lines (SRR064286 from MCF-7, SRR064287 from KPL-4, SRR064438 and SRR064439 from BT-474, SRR064440 and SRR064441 from SKBR3 cell lines). It originally had 27 true positive fusion events (27 gene pairs), which was later expanded to 99 true positive fusion events in subsequent studies [45–48]. We evaluated the performance on: (i) using only 27 true positive fusion events (Edgren 27 dataset) and (ii) using all 99 true positive fusion events (Edgren99 dataset). This dataset is also unique in the sense that none of the software benchmarked in this study were used to predict any fusions; instead, they performed individual alignment steps, and in house scripts to predict fusions and further experimentally validated them. Therefore, this dataset is totally independent. We downloaded the list of 99 gene pairs present in this dataset from the JAFFA study, where they provided the dataset in their additional file#3 [31]. Some fusions in this list have two gene names in the 5ʹ or 3ʹ parental gene instead of one, e.g. SKA2|TRIM37 + MYO19. We considered the predictions to be true if the software gave either SKA2-MYO19 or TRIM37-MYO19. The read length of this paired-end dataset is 50 bp for all of the runs and the total number of reads for runs SRR064286, SRR064287, SRR064438, SRR064439, SRR064440, SRR064441 are 16,824,862; 13,600,332; 27,030,264; 15,830,764; 18,096,704 and 18,194,304, respectively.

Subset of datasets with different classes of chimeric RNAs: For each dataset, we also created subset datasets based on three classes of chimeric RNAs which are read-through (parental genes are neighbouring genes transcribing in the same direction), inter-chromosomal (parental genes on different chromosomes) and intra-others (parental genes on the same chromosome excluding read-through chimeras). To distinguish between read-through and intra-others whose parental genes are on the same chromosome and same strand, we calculated distance between their parental genes and assigned class as read-through if the distance was ≤70,000. If any subset dataset had less than 10 chimeric RNAs, it was discarded. From InFusion simulated dataset having 68 chimeras, 64 chimeras belong to inter-chromosomal class. From ChimPipe simulated dataset having 250 chimeras, we got three subset datasets having 50 read-throughs, 50 inter-chromosomal and 150 intra-others classes. In Qin et al. dataset having 62 chimeras, 46 chimeras were experimentally validated as read-throughs; for the rest 16 chimeras, neither inter-chromosomal nor intra-others class reached cut off of ten chimeras. From Edgren 27 dataset having 27 chimeras, 18 chimeras belong to intra-others class. From Edgren 99 dataset having 99 chimeras, we got two subset datasets having 30 and 61 chimeras belonging to inter-chromosomal and intra-others classes, respectively.

Performance parameters

Sensitivity: It is defined as the percentage of total number of fusions which are correctly predicted out of the total number of true positive fusions present in the dataset. It is also known as recall. It is represented by the following formula:

Sensitivity = TP ÷ TFD

% Sensitivity = Sensitivity x 100

where TP is True Positive predictions made by the software and TFD means Total (true positive) Fusions present in the Dataset.

Positive Predictive Value (PPV): It is defined as the total number of fusions, which are correctly predicted out of the total number of predictions made by the software. It is represented by the following formula:

PPV = TP ÷ TFS

where TP is True Positive predictions made by the software and TFS means Total Fusions predicted by the Software.

F-measure: This score balances the sensitivity as well as PPV value. It is also known as F₁ score, or simply as F-score, and is represented by the following formula:

F-measure = (2 x PPV x Sensitivity) ÷ (PPV + Sensitivity)

Computational time and memory calculation

We used the ‘time’ command on the linux operating system to calculate the total time (in minutes), and maximum memory (RAM in GB) used by the software. If a software has multiple steps, then the ‘time’ command was used for each of the step, and the total time was calculated by adding time requirement of each step, while maximum memory (RAM) was calculated by taking the maximum value from all of the steps. We used only a single processor/core for calculating the time requirement, such that the results can be compared in a fair manner. For both time and memory, the values are rounded to the next whole number. Among simulated datasets, the ChimPipe PE50 sample was used because it has the greatest number of reads (~64.6 million total reads) while among real datasets, the SRR1657556 sample from the Qin et al. dataset was used which has the maximal number of reads (~62.1 million).

TOPSIS analysis

We performed TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) analyses on the ChimPipe PE50 simulated dataset and the Edgren99 real dataset to rank the software methods by giving different weights to four criteria: 1) Sensitivity, 2) PPV, 3) Computational Time (in minutes) and 4) Computational memory (in GB). Using TOPSIS, we can adjust the importance of multiple performance criteria by giving different weights and can get TOPSIS scores, which help in ranking the methods. We used the ‘topsis’ R package to perform the analyses [49]. First, we used equal weights (i.e. weight = 0.25) for all of the methods and ranked the methods based on the TOPSIS score. Next, we gave more weight (weight = 0.40) to sensitivity and PPV, and less weight to time (weight = 0.15) and memory (weight = 0.05) requirements. Between time and memory, we gave more weight to time, keeping in mind that in the current scenario with better availability of computational machines, high memory requirements are not a large constraint. Time and memory values of methods on PE50 dataset and SRR1657556 sample from Qin et al. dataset was used for TOPSIS analysis as they had maximum number of reads among simulated and real datasets, respectively.

Results

Comparison of the performance using the InFusion simulated dataset

There are 80 fusions representing 68 gene pairs that are true positive events [35]. The read length of this paired-end dataset is 75 bp with 307,920 total reads. ChimeraScan (88.2%), Arriba (83.8%), and STAR-Fusion (79.4%) had the highest sensitivity (the percentage of correctly predicted fusions out of the total number of true positive fusions), followed by INTEGRATE (76.5%) and SOAPfuse (73.5%) (Fig. 1A, and Table S1). FuSeq failed to predict any fusions on this dataset, possibly due to the minimum expression level required by the software. The performance of STARChip (26.5%) was also poor (Fig. 1A, and Table S1).

Figure 1.

Performance of software methods on Infusion dataset. (A) Sensitivity, (B) PPV and F-measure, (C) Overlap of correctly predicted chimeric RNAs and (D) Sensitivity and F-measure of pair-wise combination of methods. (Abbreviations: AR = Arriba; CS = ChimeraScan; CR = ChimeRScope; CP = ChimPipe; ES = EricScript; FC = FusionCatcher; IF = InFusion; IN = INTEGRATE; JF = JAFFA; MS = MapSplice; PZ = pizzly; SO = SOAPfuse; SF = STAR-Fusion; SC = STARChip; TF = TopHat-Fusion). FuSeq was excluded in this figure because it did not predict any fusions on this dataset

We then compared positive predictive value (PPV), which is defined as the total number of correctly predicted fusions out of the total number of predictions made by the software. InFusion and MapSplice (0.9) has the highest PPV followed by pizzly (0.88), while STAR-Fusion, ChimeraScan, ChimPipe and Ericscript were collectively ranked the 3^rd (0.86). In contrast, STARChip had a low PPV (0.26) (Fig. 1B, and Table S2). In terms of F-measure which balances sensitivity and PPV, ChimeraScan performed best with a score of 0.87 followed by STAR-Fusion (0.82) and INTEGRATE (0.81), while STARChip (0.26) and ChimeRScope (0.33) performed poorly (Fig. 1B, and Table S3). Since FuSeq did not predict any fusions on this dataset, its PPV or F-measure cannot be calculated.

We then calculated the overlap of correctly predicted chimeric RNAs among different software. We found the maximum overlap occurred between Arriba and ChimeraScan with 54 common chimeras. The minimum overlap occurred between STAR-Chip and ChimeRScope with four common chimeras. ChimeRScope had minimal overlap with all the other software methods (Fig. 1C).

The above finding suggests that none of the software is inclusive, consistent with our previous observation [23]. To gain an idea about the maximum sensitivity achievable, we combined all of the methods. The maximum sensitivity achieved was 94.1%, which is a 5.9% increase, when compared to the single best method (Table S1). However, a higher sensitivity is a tradeoff for lower specificity. What is more, combining all 16 software is not feasible due to computational time, and memory constrains. We decided to investigate the best option of combining two software tools (FuSeq was not included in the pair-wise combination because it did not predict any chimeric RNAs on this dataset). In terms of sensitivity, the best combination of methods was ChimeraScan and INTEGRATE, achieving 94.1%, which is also the maximum sensitivity achieved by combining all of the methods (Fig. 1D and Table S4). Impressively, the F-measure achieved by combining ChimeraScan and INTEGRATE was also the highest (0.86) (Fig. 1D and Table S5).

Performance on ChimPipe simulated dataset

The caveat of the above study is that the total number of reads are around 0.3 million, much lower than the total reads of a current typical RNA-Seq. We therefore downloaded the ChimPipe simulated dataset, which contains 250 true fusions, in the background of 64 million, 42 million, and 31 million, with read length at 50, 76, and 101 bp, respectively [40]. Based on sensitivity, Arriba performed the best (90.4–93.2%) on all of the PE50, PE76, and PE101 datasets (Fig. 2A, and Table S1). FuSeq, ChimeraScan and SOAPfuse were ranked the 2^nd (82.8–84.4%), 3^rd (80–82%), and 4^th (79.6–81.6%) on all of the PE50, PE76 and PE101 datasets. Thereafter, the performance decreased with pizzly (78.4–79.2%) and ChimPipe (72.8–76%) occupying the 5^th and 6^th rankings. The software STARChip (0.4–17.2%) and ChimeRScore (22.4–34.4%) were poor performers. For most of the software, there was little difference in the performance on different read lengths; however, the performance of JAFFA, ChimeRScope and STARChip improved on the dataset with longer read length (Fig. 2A, and Table S1).

Figure 2.

Performance of software methods on ChimPipe datasets (PE50, PE76 and PE101). (A) Sensitivity, (B) PPV, (C) F-measure, (D-F) Overlap of correctly predicted chimeric RNAs on PE50, PE76 and PE101 datasets and (G) Sensitivity and F-measure of pair-wise combination of methods. (Abbreviations: AR = Arriba; CS = ChimeraScan; CR = ChimeRScope; CP = ChimPipe; ES = EricScript; FS = FuSeq; FC = FusionCatcher; IF = InFusion; IN = INTEGRATE; JF = JAFFA; MS = MapSplice; PZ = pizzly; SO = SOAPfuse; SF = STAR-Fusion; SC = STARChip; TF = TopHat-Fusion)

Based on PPV, MapSplice performed the best (0.99) on all of the PE50, PE76, and PE101 datasets. It was followed by ChimPipe (0.92–0.93) and SOAPfuse (0.81–0.82). ChimeRScope (0.07–0.08) was among the poor performers (Fig. 2B, Table S2). While Arriba was the best performer based on sensitivity, it was ranked the 6^th (on PE50 and PE76) and 5^th (on PE101) based on PPV (Fig. 2B). This means that while Arriba predicts most of the true positive events, it also predicted many false positives. In terms of F-measure, Arriba (0.81–0.83) is the best performing method on all of the PE50, PE76 and PE101 datasets. ChimPipe (0.84, 0.83) and SOAPfuse (0.81, 0.81) occupied 2^nd and 3^rd positions, respectively, on PE50 and PE76, while on PE101 they switched positions. ChimeRScope (0.11–0.14) and STARChip (0.01–0.23) were among the poor performers (Fig. 2C, and Table S3).

In terms of the overlap between different methods, the Arriba + FuSeq combination had the maximum overlap on all of the PE50, PE76 and PE101 datasets. Due to its low sensitivity, STARChip method had minimal overlap with all of the other software methods (Fig. 2D-2F).

The maximum sensitivity achieved by combining results from all of the software was 96.8% on PE50, PE76 and 96% on PE101 datasets. The increase was 3.6%, 6% and 5.6% on PE50, PE76 and PE101 datasets, respectively, when compared to the single best method (Table S1). On the PE50 dataset, the best combination of software in terms of sensitivity was ChimeraScan + Arriba, achieving 96% which is very close to the maximal sensitivity (96.8%) achieved by combining all 16 methods (Fig. 2G, and Table S6). However, it also resulted in a decrease in the F-measure (0.54) due to more false positives from both software tools. In terms of F-measure, the ChimPipe + STARChip combination was the best one (0.88), with lower sensitivity (80.4%) (Fig. 2G and Table S7). We then considered the balanced performance of both sensitivity and F-measure by searching for combination having sensitivity above 93.2%, which is the highest sensitivity achieved by single best method. Arriba + STAR-Fusion was observed as a good combination having 94.8% sensitivity and 0.82 F-measure. On PE76 and PE101 datasets, the best pair-wise combination of software in terms of sensitivity was Arriba + pizzly (95.2% and 94.4%), with F-measure (0.81 and 0.83) (Table S8-S11). In terms of F-measure, ChimPipe + MapSplice (0.86 and 0.85) was the best, while the sensitivity was low (84% and 81.2%) (Table S8-S11). Arriba + pizzly combination was observed as a good combination while considering the balanced performance of both sensitivity (95.2%, 94.4%) and F-measure (0.81, 0.83).

Performance on Qin et al. real dataset

We then decided to evaluate the performance of the software tools using real datasets. However, there are two issues with real datasets. One, for a given real dataset, the total number of true fusions is unknown. Second, in the past, software developers have assumed that chimeric RNAs do not exist in normal cells, and have used normal tissues/cell RNA-Seq as negative datasets. Such an assumption would punish software tools that identify real chimeras in normal tissues/cells, obviously resulting in faulty conclusions. Qin et al., analysed RNA-Seq from prostate cancer cells, and experimentally validated 62 fusions containing a mixture of read-throughs, inter-chromosomal and intra-others [24]. Based on sensitivity, JAFFA (72.6%) performed the best, followed by SOAPfuse (67.7%) and ChimeraScan (54.8%) (Fig. 3A, and Table S1). STARChip (0%), TopHat-Fusion (1.6%), ChimeRScope (4.8%), INTEGRATE (4.8%), MapSplice (4.8%), STAR-Fusion (8.1%), and Arriba (9.7%) all performed poorly.

Figure 3.

Performance of software methods on Qin et al dataset. (A) Sensitivity, (B) PPV and F-measure, (C) Overlap of correctly predicted chimeric RNAs and (D) Sensitivity and F-measure of pair-wise combination of methods. (Abbreviations: AR = Arriba; CS = ChimeraScan; CR = ChimeRScope; CP = ChimPipe; ES = EricScript; FS = FuSeq; FC = FusionCatcher; IF = InFusion; IN = INTEGRATE; JF = JAFFA; MS = MapSplice; PZ = pizzly; SO = SOAPfuse; SF = STAR-Fusion; TF = TopHat-Fusion). STARChip was excluded in this figure because it had sensitivity of 0 and therefore its inclusion did not help in pair-wise combination

Based on PPV, SOAPfuse (0.58) performed the best followed by InFusion (0.29) and FuSeq (0.28) (Table S2). Based on F-measure, SOAPfuse (0.62) performed the best followed by FuSeq (0.35). JAFFA which was best performing method based on sensitivity, had poor PPV (0.02) and F-measure (0.05). The F-measure of STARChip cannot be calculated because both its sensitivity and PPV was 0 (Fig. 3B, and Table S3).

Overall, most of the software performed poorly on this dataset. The better performance of SOAPfuse on this dataset was expected because the study itself involved using SOAPfuse to predict 95 fusions, out of which 62 were experimentally validated [24]. This highlights the danger of performing comparison studies on sensitivity or specificity, while relying on true fusion events discovered by a given software tool. Although SOAPfuse should have 100% sensitivity on this dataset, the difference in sensitivities is due to the fact that Qin et al. combined the short read (length 50 bp) and long read (length 101 bp) data for each group to run SOAPfuse, while in this study we excluded the short reads.

In terms of overlap of correctly predicted chimeras between different software methods, JAFFA and SOAPfuse had the greatest overlap with 33 common chimeras (Fig. 3C). STARChip and TopHat-Fusion had minimal overlap with all the other software methods (Fig. 3C). Overall, the overlap between different methods is small. This is in contrast to the simulated datasets in that there were many more overlaps in the predicted chimeras between different methods.

The maximal sensitivity achieved on this dataset by combining the results of all of the methods was 93.5%, a 20.9% increase when compared to the single best method (Table S1). In terms of the best pair-wise combination of software methods, SOAPfuse + JAFFA achieved a maximum sensitivity of 87.1% (Fig. 3D and Table S12); however, the F-measure dropped to 0.06 (Fig. 3D and Table S13). Based on F-measure, SOAPfuse + TopHat-Fusion achieved the highest value of 0.61 (STARChip was not considered, because it had sensitivity of 0) (Fig. 3D and Table S13). We then considered the balanced performance of both sensitivity and F-measure, by requiring sensitivity above 67.7%. SOAPfuse + InFusion turned out to be a good combination, achieving 71% sensitivity and an F-measure of 0.56 (Fig. 3D and Table S12-S13).

Performance on the Edgren real dataset

To overcome the bias we observed in the above real dataset, we then used the Edgren dataset, which is one of the standard datasets used by different developers [26,28,31,40]. The authors of the Edgren dataset used none of the software in this study. Instead, they performed individual alignment steps and used in house scripts to predict fusions, and further experimentally validated them. Therefore, in this sense, this dataset is unbiased to the software tools benchmarked here. Initially, only 27 true fusions were identified, and in subsequent studies the list expanded to 99.

Based on sensitivity on the Edgren 27 dataset, ChimeraScan (85.2%) was found to be the best software (Fig. 4A, and Table S1). Arriba (81.5%) and STAR-Fusion (81.5%) collectively occupied the 2^nd position followed by FuSeq (77.8%) at 3^rd position. STARChip and ChimeRScope performed poorly on this dataset (Fig. 4A, and Table S1). Based on PPV, FusionCatcher (0.45) was the best software followed by MapSplice (0.44) and INTEGRATE (0.32). ChimeraScan (0.01) and pizzly (0.02) were the poor performers (Fig. 4B, and Table S2). Interestingly, the ranking of F-measure is almost similar to that of PPV (Fig. 4C, and Table S3), suggesting that PPV is the determining factor for F-measure in this dataset.

Figure 4.

Performance of software methods on Edgren dataset. (A) Sensitivity, (B) PPV, (C) F-measure, (D) Overlap of correctly predicted chimeric RNAs on Edgren99 dataset and (E) Sensitivity and F-measure of pair-wise combination of methods. (Abbreviations: AR = Arriba; CS = ChimeraScan; CR = ChimeRScope; CP = ChimPipe; ES = EricScript; FS = FuSeq; FC = FusionCatcher; IF = InFusion; IN = INTEGRATE; JF = JAFFA; MS = MapSplice; PZ = pizzly; SO = SOAPfuse; SF = STAR-Fusion; SC = STARChip; TF = TopHat-Fusion)

On Edgren99 dataset, sensitivity dropped for all tools. Among them, Arriba (45.5%) performed the best followed by ChimeraScan (41.4%), STAR-Fusion (41.4%) and pizzly (41.4%) collectively taking the 2^nd position. The poor performing methods were STARChip (8.1%), TopHat-Fusion (15.2%) and ChimeRScope (17.2%) (Fig. 4A, and Table S1). Based on PPV, FusionCatcher (0.61) performed the best followed by STAR-Fusion (0.55) and MapSplice (0.54) (Fig. 4B, and Table S2). Based on F-measure, STAR-Fusion (0.47) performed the best, followed by FuSeq (0.38) and FusionCatcher (0.38) (Fig. 4C, and Table S3). As the number of true chimeric events on the Edgren dataset increased from 27 to 99, we observed that in terms of F-measure, the ranking of the methods changed (Fig. 4C). This is also the limitation of any benchmarking study using real datasets in that we do not have a full list of true chimeric RNAs, and certain software may be penalized if they predict unknown true fusions.

We examined overlaps between different methods on the Edgren99 dataset, Arriba and STAR-Fusion had the largest overlap with 42 common chimeras, while ChimeRScope and STARChip and ChimeRScope and TopHat-Fusion had the smallest overlap with only four common chimera. STARChip had minimal overlap with all of the other software methods, consistent with its lowest ranking in sensitivity (Fig. 4D).

The maximum achieved sensitivity by combining results from all the methods was 88.9% on Edgren 27 (7.4% increase from the single best method), and 61.6% on Edgren99 (16.1% increase), respectively (Table S1). The maximum sensitivity on the real datasets especially Edgren99 is much lower than that on simulated datasets. This finding suggests that there is still much to be desired for sensitivity on real datasets.

On the Edgren99 dataset, the maximum sensitivity achieved by combining two software tools was 52.5%, and it was achieved by Arriba + ChimeraScan and Arriba + pizzly (Fig. 4E and Table S14). However, their F-measures were poor (0.06, 0.09) (Fig. 4E and Table S15). Based on F-measure, FusionCatcher + STAR-Fusion (0.47) achieved the best performance with 43.4% sensitivity (Fig. 4E and Table S15) and is also the best combination when considering the balance between both sensitivity and F-measure.

Identification of different classes of chimeras

Based on the location of parental genes, chimeric RNAs can be classified into read-through, inter-chromosomal and intra-others. We created sub-datasets from each dataset based on the above classes of chimeras and accessed the performance of all the tools on these three classes to identify tools performing best on specific classes of chimeric RNAs.

For the read-through class, in terms of sensitivity, Arriba performed the best on PE50, PE76 and PE101 simulated sub-datasets (Table 1). Thereafter, SOAPfuse on PE50 and JAFFA on PE76 occupied 2^nd spot while on PE101 SOAPfuse and JAFFA had the same sensitivity and collectively occupied 2^nd spot. In terms of F-measure, ChimPipe performed the best on PE50, PE76 and PE101 followed by SOAPfuse (Table S16). On Qin et al. real sub-dataset, JAFFA performed the best followed by SOAPfuse and ChimeraScan in terms of sensitivity, while SOAPfuse performed the best followed by FuSeq and FusionCatcher in terms of F-measure (Table 1 and Table S16).

Table 1.

Sensitivity of software methods on read-through, inter-chromosomal and intra-others classes of chimeric RNAs

	read-through				inter-chromosomal					intra-others
	ChimPipe					ChimPipe				ChimPipe
Software	PE50 (50*)	PE76 (50*)	PE101 (50*)	Qin et al (46*)	Infusion (64*)	PE50 (50*)	PE76 (50*)	PE101 (50*)	Edgren99 (30*)	PE50 (150*)	PE76 (150*)	PE101 (150*)	Edgren27 (18*)	Edgren99 (61*)
AR	86	84	82	2.2	82.8	98	94	94	40	94	92	92	72.2	54.1
CS	78	78	78	56.5	89.1	82	82	82	36.7	82.7	83.3	79.3	77.8	47.5
CR	18	32	32	2.2	29.7	30	40	40	6.7	21.3	33.3	32.7	38.9	21.3
CP	80	74	76	21.7	35.9	76	74	68	26.7	74.7	76	70	55.6	31.1
ES	64	64	58	43.5	60.9	64	64	66	23.3	62.7	64	58.7	44.4	24.6
FS	78	78	78	47.8	NA	94	84	88	36.7	83.3	84.7	82.7	66.7	37.7
FC	50	56	56	30.4	64.1	58	62	66	30	63.3	68	69.3	66.7	29.5
IF	64	60	58	13	68.8	62	62	68	30	68.7	67.3	70.7	55.6	24.6
IN	0	0	0	0	76.6	78	78	74	23.3	69.3	72	68	50	31.1
JA	64	82	80	73.9	54.7	56	72	72	26.7	48.7	66.7	70.7	44.4	21.3
MS	0	0	0	0	39.1	74	78	72	26.7	80.7	81.3	77.3	55.6	21.3
PZ	42	42	40	26.1	54.7	90	88	90	43.3	86.7	88.7	88	61.1	45.9
SO	80	80	80	67.4	75	82	80	82	33.3	82	81.3	78.7	61.1	41
SC	0	0	0	0	26.6	0	24	22	6.7	0.7	20.7	21.3	22.2	9.8
SF	4	4	4	0	79.7	82	86	84	36.7	86	86.7	83.3	72.2	49.2
TF	4	2	2	0	42.2	56	46	32	20	48	43.3	34	38.9	14.8

(Abbreviations: AR=Arriba; CS=ChimeraScan; CR=ChimeRScope; CP=ChimPipe; ES=EricScript; FS=FuSeq; FC=FusionCatcher; IF=InFusion; IN=INTEGRATE; JF=JAFFA; MS=MapSplice; PZ=pizzly; SO=SOAPfuse; SF=STAR-Fusion; SC=STARChip; TF=TopHat-Fusion)

* represents number of class specific true chimeric RNAs present in these datasets.

Highest scores are marked bold.

For the inter-chromosomal class, in terms of sensitivity, Arriba performed the best on PE50, PE76 and PE101 while on InFusion and Edgren99 sub-datasets, ChimeraScan and pizzly, respectively, performed the best followed by Arriba occupying 2^nd spot. In terms of F-measure, MapSplice followed by INTEGRATE were the best performers on PE50, PE76 and PE101, while on InFusion and Edgren99 sub-datasets, ChimeraScan and FusionCatcher, respectively, performed the best (Table 1 and Table S16).

For the intra-others class, in terms of sensitivity, Arriba performed the best on PE50, PE76, PE101 and Edgren99 sub-datasets while ChimeraScan performed the best on Edgren 27 followed by Arriba and STAR-Fusion occupying 2^nd spot. In terms of F-measure, MapSplice performed the best on PE50, PE76 and PE101 while FusionCatcher and STAR-Fusion performed the best on Edgren 27 and Edgren99 sub-datasets (Table 1 and Table S16). Overall, the sensitivity of many software tools on read-through class of chimeras is lower than inter-chromosomal and intra-others.

Computational requirements

We tested the computational time and memory/RAM requirements of each software on one simulated dataset, PE50 from ChimPipe, as well as a real dataset, SRR1657556, from Qin et al. On the PE50 dataset, pizzly took the least amount of time (5 min), while ChimeraScan took the most time (1715 min) (Fig. 5A). In terms of memory requirements, ChimeraScan took minimal memory (4 GB), followed by pizzly, InFusion and EricScript (5 GB), while ChimeRScope required the most memory (56 GB) requirement (Fig. 5A). Considering both parameters, pizzly was the least computationally demanding, followed by FuSeq. Both software tools can even be run on ordinary laptops, taking only minutes to half an hour, and using only 5–8 GB of RAM (Fig. 5A).

Figure 5.

Computational time and memory requirement. The computational time and memory requirement of different software methods on (A) simulated (ChimPipe PE50) dataset and (B) real (Sample SRR1657556 from Qin et al.) dataset

On the SRR1657556 dataset from Qin et al., pizzly took the least amount of time (11 min), while MapSplice (2585 min), and ChimPipe (1679 min) took significantly more time. In terms of memory requirements, ChimeraScan took minimal memory (4 GB), followed by pizzly, InFusion, and JAFFA (5 GB) (Fig. 5B).

TOPSIS analyses

To gain some insights on balanced performance, we performed TOPSIS analyses to rank the methods based on a combined TOPSIS score of sensitivity, PPV, computational time, and memory. Using equal weights for all four performance criteria, the top performing methods on the ChimPipe PE50 dataset were pizzly > SOAPfuse > FuSeq and InFusion (Fig. 6A).

Figure 6.

TOPSIS analyses. TOPSIS analyses of different methods on ChimPipe PE50 dataset with (A) equal weights (Sensitivity = PPV = Time = Memory = 0.25) and (B) different weights (Sensitivity = PPV = 0.40; Time = 0.15; Memory = 0.05) and Edgren99 dataset with (C) equal weights and (D) different weights. The nine newly developed software are in green, and the seven older ones in red

However, we realized that sensitivity and PPV may be more important parameters to consider than time and memory for researchers aiming to identify ideal biomarkers and therapeutic targets. In between time and memory, we chose to give more weightage to time because the memory requirement is readily achievable currently with the availability of better computational machines. Based on these considerations, we performed another TOPSIS analysis using following weights: sensitivity = 0.40, PPV = 0.40, time = 0.15, and memory = 0.05. In this situation, the top performing methods were Arriba > SOAPfuse > pizzly > MapSplice (Fig. 6B). Similarly, we performed TOPSIS analysis on the Edgren99 dataset with equal weights and observed FuSeq > FusionCatcher > InFusion > SOAPfuse were the top performing methods (Fig. 6C). TOPSIS analysis with different weights yielded the following top performing methods STAR-Fusion > FusionCatcher > FuSeq > INTEGRATE (Fig. 6D).

Pizzly, FuSeq, and SOAPfuse gave balanced performance on the simulated dataset, while FuSeq is the only one that is in the top five under all considerations on both datasets.

Discussion

With many chimeric RNA detection software tools available, and a large portion being recently developed, it becomes difficult to choose the right one for individual use. In this study, we benchmarked newly developed chimeric/fusion RNA prediction software along with the state-of-art software which were already accepted, and used by the scientific community. Newly developed software like FuSeq and pizzly compete well with older top performing methods (Fig. 6). We found that most of the recently developed software, especially pizzly, STARChip, FuSeq, and Arriba are computationally less demanding in terms of time, which is one important factor to consider when choosing a software tool for large-scale projects. For instance, pizzly and FuSeq only require 5 GB and 8 GB computation memory, respectively, and can be run on desktop computers. However, we should keep in mind that time and memory are not the most discriminant parameters when the end users want to search for important biomarkers or therapeutic targets and desktop is not a solution to analyse large-scale projects.

As we were preparing our manuscript, Haas et al. [38], who developed STAR-Fusion software, also benchmarked various fusion transcript detection methods and concluded that their method along with Arriba and STAR-SEQR are the most accurate methods. Even though their study evaluated the performance of many tools, and yielded meaningful conclusions, it has the following limitations: (i) For real RNA-seq datasets, they randomly sampled 20 million paired-end reads for each dataset and therefore there is a possibility that some of the reads which may have supported true fusions were discarded which may have impacted the accuracy of some of the fusion prediction methods. (ii) The true positive fusions in the real dataset are not necessarily experimentally validated fusions, but rather those which are predicted by at least n different methods (wisdom of crowds approach). Thus, all the fusions, which are uniquely predicted by any software were considered as false positive. This approach can be heavily influenced by the portfolio of the tools, potentially bogged down by a larger number of bad tools, and punish good tools for their unique features. (iii) They did not include FuSeq, which is one of our best performers here.

We conducted this study using both simulated data and real data, and summarized our findings separately. We observed that in general, the performance of individual software methods on simulated datasets is much better than that on real datasets. This is in part due to the fact that the exact number of true positive events is not known in any real dataset. Therefore, the evaluation of specificity or PPV performance of certain software is compromised. The performance of a certain software may be punished if it correctly predicts some unknown true fusions. For instance, in Edgren datasets, only 27 fusions were identified and validated initially. Later studies extended the true fusion number to 99. When we compared the F-measure, STAR-Fusion had the highest score on Edgren99, but was ranked #3 using the Edgren 27 data. In the future, a real dataset from which all fusions have been exhaustedly discovered will be extremely useful for benchmarking studies.

On all four datasets, we found that none of the tools are inclusive. This became more obvious on real data, where the overlaps between the tools are even less. To improve the sensitivity, we first calculated the maximal sensitivity achievable with the combination of all 16 tools. We then evaluate the two software tool combinations. There are often tradeoffs between sensitivity and F-measure. When both aspects are considered, we found that Arriba + pizzly was the best combination on the ChimPipe PE101 simulated data (94.4% for sensitivity and 0.83 for F-measure). None of the combinations had satisfying their performances on the Edgren99 real dataset, with maybe FusionCatcher + STAR-Fusion possibly being a balanced combination (43.4% on sensitivity and 0.47 F-measure).

Before a win-it-all software tool is developed, the best strategy may be to develop methods which can rank or score large numbers of collective fusions predicted from combinations of software. Some methods for fusion ranking/scoring already exist, for example, FuGePrior [50] uses the results from three fusion prediction methods, two of which are deFuse and ChimeraScan, while a third software can be selected by the user. Using a series of filtering, it prioritizes the fusions to enrich the true positive events. Another method, confFuse [51], assigns a confidence score to each fusion, and suggest to select high confidence fusions with a score ≥8. In the future, the benchmarking of these prioritization/scoring algorithms would be required to effectively combine the best fusion prediction methods, and the best scoring methods, so as to get a large number of high confidence fusion events.

We also showed the performance of software tools on different classes of chimeric RNAs. We observed that the performance of most of the methods on read-through class of chimeric RNAs was lower than inter-chromosomal and intra-others. In the chimeric RNA field, read-through events were considered as false-positive candidates, or were not considered to be present in cancer physiology. However, recent studies have shown that read-through chimeras are widely present in both cancer and normal physiology [8,15,52], and have identified some cancer-specific read-throughs in various cancer types [53–55]. For the end users working on read-through chimeras, software tools with high sensitivity such as JAFFA, SOAPfuse, ChimeraScan and FuSeq would be appropriate. Among these four tools, based on F-measure, SOAPfuse or FuSeq will be a better choice.

Finally, we performed TOPSIS analysis on the ChimPipe and Edgren99 datasets. FuSeq is among the top five performing software considering four criteria on both datasets. Even though we considered two weighting schemes, there are situations when end users may value each criterion differently. Individual end users should choose a tool based on their needs.

Funding Statement

This work is supported by NIGMS grant GM132138 (HL);National Institute of General Medical Sciences [GM132138];

Supplemental material

Supplemental data for this article can be accessed here.

Declarations

Ethics approval and consent to participate: Not applicable.

Consent for publication: Not applicable

Availability of data and material:

All data generated or analysed during this study are included in this published article [and its supplementary information files]

Disclosure statement

The authors declare that they have no competing interests.

Authors’ contributions:

S.S. and H.L. designed experiments. S.S. performed the experiments. Both authors interpreted the data. S.S. and H.L. wrote the manuscript.

Supplementary material

Supplemental data for this article can be accessed here.