Clinical validation of RNA sequencing for Mendelian disorder diagnostics

Summary

Despite rapid advancements in clinical sequencing, over half of diagnostic evaluations still lack definitive results. RNA sequencing (RNA-seq) has shown promise in research settings for bridging this gap by providing essential functional data for accurate interpretation of diagnostic sequencing results. However, despite advanced research pipelines, clinical translation of diagnostic RNA-seq has not yet been realized. We have developed and validated a clinical diagnostic RNA-seq test for individuals with suspected genetic disorders who have existing or concurrent comprehensive DNA diagnostic testing. This diagnostic RNA-seq test processes RNA samples from fibroblasts or blood and derives clinical interpretations based on the analytical detection of outliers in gene expressions and splicing patterns. The clinical validation involves 130 samples, including 90 negative and 40 positive samples. We developed provisional expression and splicing benchmarks using short-read and long-read RNA-seq data from the GM24385 lymphoblastoid sample produced by the Genome in a Bottle Consortium. For clinical validation, we first established reference ranges for each gene and junction based on expression distributions from our control data. We then evaluated the clinical performance of our outlier-based pipeline using 40 positive samples with previously identified diagnostic findings from the Undiagnosed Diseases Network project. Our study provides a paradigm and necessary resources for independent laboratories to validate a clinical RNA-seq test.

Graphical abstract

RNA sequencing has emerged as a powerful adjunct to DNA sequencing for Mendelian disease diagnosis. Our study establishes a diagnostic RNA-sequencing pipeline and provides a paradigm and necessary resources for independent laboratories to validate a clinical RNA-sequencing test.

Introduction

The rapid advancements of sequencing technologies, such as clinical exome sequencing (ES) and whole-genome sequencing (GS), have revolutionized the diagnosis of Mendelian disorders in the past decade.^1,2 Meanwhile, the implementation of genome-wide next-generation sequencing (NGS) has led to the identification of numerous variants with unknown impacts on RNA and protein, which brings challenges to clinical interpretation.³ Recently, transcriptome RNA sequencing (RNA-seq) has emerged as a powerful adjunct to ES and GS in Mendelian disorder diagnostics.⁴ Owing to its ability to detect abnormal expression and splicing patterns, transcriptome sequencing can improve molecular diagnostic rates by 7.5%–36% compared with DNA testing alone.^5,6,7,8 Despite the development of advanced research pipelines by various groups, the clinical translation of diagnostic RNA-seq has not yet been realized. The field lacks the comprehensive implementation knowledge required for such a transition.

Several guidelines have been established for the clinical validation of NGS-based DNA sequencing.^{9,10,11,12,13} While the principles and experiences from DNA NGS tests can guide RNA-seq validation, key differences add to the challenges of validating RNA-seq tests. Despite originating from the same type of sequencing read raw data, DNA-based NGS and RNA-seq diverge in their analytical endpoints. DNA sequencing typically focuses on detecting SNVs/indels or copy-number variants (CNVs), whereas RNA-seq aims to measure gene-expression levels and splicing-junction status. Therefore, efforts are needed to adapt the DNA-based validation framework to RNA-seq analytical endpoints. Benchmark data should be generated, preferably using publicly accessible resources, such as those from the Genome in a Bottle (GIAB) Consortium from the National Institute of Standards and Technology (www.nist.gov/programs-projects/genome-bottle), to facilitate seamless adoption for diagnostic labs in implementing this validation.

Both DNA- and RNA-based diagnostic tests are designed to identify rare genetic findings that explain the rare disease phenotype. DNA variants are characterized by discrete genotypes (reference, heterozygous, or homozygous), with a substantial portion of the population exhibiting reference genotypes. This characteristic facilitates effective cross-platform data comparison, enabling the use of external large control databases, such as the gnomAD,¹⁴ thus reducing the burden on clinical labs to produce control data. In contrast, gene-expression data from RNA-seq can exhibit a wide distribution in the “normal” population, with the degree of dispersion varying significantly across genes.¹⁵ Furthermore, naturally occurring alternative splicing further challenges the identification of diagnostic outliers from background noise.¹⁶ Therefore, it is crucial for RNA-seq tests, especially in the clinical context, to generate control data using the same experimental and bioinformatics pipeline. Additionally, there is a strong need to establish reference ranges for all targets in the RNA-seq test, a component that is usually less extensively considered when validating DNA-based NGS tests.

Tissue-specific expression of genes and transcripts presents another challenge for the design and clinical validation of the RNA-seq tests. Data from the Genotype-Tissue Expression (GTEx) Portal indicate that 37.4% of all coding genes in blood and 48.3% in fibroblasts, the most commonly used clinically accessible tissues, exhibit low average expression (transcript per million, [TPM] lower than 1).^17,18 Due to tissue-specific splicing patterns, around 34% and 12% of canonical transcripts are not adequately represented in blood and fibroblast samples, respectively.¹⁹ Therefore, diagnostic RNA-seq validations should be designed in a tissue-dependent manner to address test performance and clinical limitations.

Here, we report the clinical validation processes of an RNA-seq test for the diagnosis of Mendelian disorders. We included publicly available benchmark samples, clinically positive samples, and negative control samples. We conducted optimization and familiarization (O&F) and performed quality control (QC). We set up a provisional RNA standard reference using data from the GIAB Consortium and evaluated the analytical performance of our test. Additionally, we established transcriptome-wide reference ranges for all reportable targets. Finally, we assessed the clinical performance using positive samples with previously identified diagnostic findings from the Undiagnosed Diseases Network (UDN) project.

Material and methods

Ethics approval

This study has been approved by the Institutional Review Board (IRB) at Baylor College of Medicine (H-42680). The study subjects were originally recruited through informed consent approved by the IRB at the National Human Genome Research Institute (15HG0130) or Baylor College of Medicine (H-34433 and H-44172) and de-identified for the current study.

Collection of validation samples

A total of 130 samples were collected from 110 individuals, including 73 fibroblast samples and 55 blood samples (Table 1). None of the individuals are related to each other.

Table 1.

Sample collection and design of clinical validation

Sample type	Count
Blood	55
O&F	10
Negative sample	24
Positive sample	20
Reproducibility	1
Fibroblast	73
O&F	10
Negative sample	43
Positive sample	20
LCL	2
Benchmark and reproducibility (GM24385)	1
Reproducibility (K562)	1
Total	130

O&F, optimization and familiarization; LCL, lymphoblastoid cell line.

All 73 fibroblast samples were obtained from the UDN, comprising 20 positive samples with a molecular diagnosis and 53 “negative” samples from apparently healthy siblings or parents of unrelated participant families. Ten of the negative samples were utilized for O&F. Among the 55 blood samples, 54 were sourced from the UDN and one from a healthy volunteer (BG1477). The UDN blood samples included 20 positive samples and 34 “negative” samples from apparently healthy siblings or parents of unrelated participant families, with 10 negative samples designated for O&F. Among the negative samples, the blood sample from BG1477 was used for reproducibility testing.

The two lymphoblastoid/lymphocyte samples, GM24385 and K562, were procured from the Coriell Institute. GM24385 (from HG002) is a standard reference sample from the GIAB project and was used for both reproducibility testing and analytical validation. K562, derived from the bone marrow of an individual with chronic myeloid leukemia, was used for reproducibility testing.

Positive samples from UDN were selected under the following criteria.

• (1)A molecular diagnosis of a Mendelian disorder supported by DNA variant(s)
• (2)DNA variant(s) predicted to result in RNA-level changes (altered expression or alternative splicing)
• (3)Prior RNA-seq studies identified the predicted RNA-level changes
• (4)At least two independent experimental data sources support the result
• (5)No recorded kinship to other negative or positive samples in the cohort

Negative samples from the UDN were selected based on the following criteria.

• (1)No known molecular diagnosis of a Mendelian disorder
• (2)No recorded kinship to other negative or positive samples in the cohort

Sample processing and RNA extraction

Fibroblast samples were cultured in high-glucose DMEM, supplemented with 10% fetal bovine serum, 1% non-essential amino acid, and 1% penicillin-streptomycin. Blood samples were stored in PAXgene tubes at −80°C before RNA extraction. RNA was extracted from around 10⁷ cells using the RNeasy mini kit (Qiagen) following the manufacturer’s instructions with the inclusion of an on-column genomic DNA-removal step. The integrity and quality of the RNA were assessed using the Qubit 4 fluorometer and the Qubit RNA HS assay kit (Thermo Fisher).

Library preparation and NGS

RNA from fibroblast and lymphoblastoid/lymphocyte samples was processed using the Illumina Stranded mRNA prep kit. RNA from whole-blood samples was processed using the Illumina Stranded Total RNA Prep with Ribo-Zero Plus kit to remove human globin RNA and rRNA. Sequencing was conducted on the Illumina NovaSeqX platform, which produces paired-end short-read data at 150 bp. Each validation sample was sequenced on average to a target depth of 150 million reads.

Raw data processing and expression quantification of RNA-seq

The raw data processing pipeline was adapted from the GTEx version 10 pipeline (https://github.com/broadinstitute/gtex-pipeline/blob/master/rnaseq/README.md). Sequencing FASTQ data were aligned to the reference genome GRCh38 with STAR v.2.7.8a_sentieon and SAMtools 1.15.1/HTSlib v.1.10.2. Picard v.2.23.3 was used to mark duplicates. Gene expressions were quantified using RNA-SeQC v.2.4.2.²⁰ Isoform-level quantification was performed with RSEM v.1.3.3.²¹ Transcripts were annotated with GENCODE v.39. FastQC v.0.11.9 provided QC measurements. For identity verification, SNPs/indels were called from the RNA-seq data using the haplotyper from Sentieon DNAseq. The resultant variants were compared to those obtained from DNA-sequencing data of the same individual to ensure identity matching.

Reproducibility test

Using a 3-1-1 validation framework, a reproducibility test was performed on GM24385 (from HG002), K562, and BG1477. We conducted an intra-run with triplicate preparations of the same sample, followed by two inter-runs of the same sample, resulting in a total of five tests across three different batches prepared on three different days. The intra-run and inter-run batches were prepared by two different technicians using different Bio-Rad thermocyclers, pipettes, vortexers, centrifuges, and reagents. This comprehensive approach enabled us to measure resulting variations, affirming the assay’s consistent and reliable performance.

Gene-expression reproducibility was calculated using gene read counts. Splice-junction detection reproducibility was calculated using junction read counts. Only canonical junctions annotated by GENCODE v.39²² were included. Reproducibility was assessed by two statistical methods: Pearson correlation on log-scaled counts and cosine similarity on raw counts.

RNA-seq benchmark data

GM24385 is a human lymphoblastoid cell line derived from a female donor HG002, widely used for benchmarking and validating DNA-based genomic analyses due to its easy accessibility and extensive characterization.^{23,24,25,26,27} However, a “gold standard” benchmark has not yet been established at the transcriptome level. Therefore, we sought to create a provisional benchmark by aggregating sequencing data generated independently from renowned groups from the GIAB team. The analysis incorporated data from different institutions, produced in multiple runs using both short- and long-read sequencing technologies. We obtained sequencing data that are released by the GIAB team.²⁸

For short-read data, four sequence datasets from two groups were used.

• (1)UNC data: a triplicate set of Illumina short-read RNA-seq data from The University of North Carolina (UNC). These data were generated on three GM24385 cell lines as part of the NIST-GIAB RNA-seq pilot sequencing project (https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data_RNAseq/AshkenazimTrio/HG002_NA24385_son/UNC_Illumina/).
• (2)Google data: another set of Illumina short-read RNA-seq data provided by Google, with sequencing contracted out to Novogene as part of the NIST-GIAB RNA-seq pilot sequencing project (https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data_RNAseq/AshkenazimTrio/HG002_NA24385_son/Google_Illumina/).

Short-read benchmark data were processed using the same pipeline mentioned above. We first averaged the TPM of protein-coding genes within the three UNC datasets and further averaged the result with the Google sample to create a standard mean TPM matrix. We applied three different TPM thresholds (TPM ≥ 1, TPM ≥ 3, TPM ≥ 5) to define the positive expression set. Genes with mean TPM = 0 were categorized as the negative expression set. To validate our in-house data, a gene was defined as detected if the read count was ≥50 and undetected if the read count was <50. Sensitivity (positive rate of genes in the positive expression set) and specificity (negative rate of genes in the negative expression set) were used to evaluate the performance of gene-expression quantification.

For long-read data, two sequence datasets from two groups were used.

• (1)PacBio data: PacBio long-read RNA-seq data produced from the Sequel Revio system by PacBio, using the Kinnex full-length RNA protocol (https://downloads.pacbcloud.com/public/dataset/Kinnex-full-length-RNA/DATA-Revio-HG002-1/).
• (2)Baylor data: PacBio long-read RNA-seq data produced by Baylor College of Medicine, with sequencing contracted out to Novogene as part of the NIST-GIAB RNA-seq pilot sequencing project (https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data_RNAseq/AshkenazimTrio/HG002_NA24385_son/). Iso-Seq SMRT libraries were constructed and sequenced using the PacBio Sequel Systems.

The Iso-Seq workflow (v.4.0.0, https://isoseq.how/) was used for data processing. Reads were first mapped to hg38 using pbmm2 v.1.13.1. Mapped reads were then collapsed into unique isoforms using Iso-Seq and classified/filtered using Pigeon (v.1.1.0, https://github.com/PacificBiosciences/pigeon).

A provisional benchmark for splicing junctions was constructed by intersecting junctions passing QC from each source. The consensus list was filtered for canonical junctions represented in GENCODE v.39.²² The same TPM thresholds used for the expression benchmark (TPM ≥ 1, TPM ≥ 3, TPM ≥ 5) were applied to define the genes included in the positive splicing set. Canonical junctions in genes from the negative expression set were used as the negative splicing set. To validate our in-house data, a junction was defined as detected if the junction read count was ≥5 and undetected if the junction read count was <5. Sensitivity (positive rate of junctions in the positive set) and specificity (negative rate of junctions in the negative set) were used to evaluate the performance of splicing-junction detection.

Reference ranges, outlier analysis, and clinical performance evaluation

We established a reference panel consisting of 73 fibroblast samples and 55 blood samples, which includes both positive and negative cases. Outlier analyses were performed independently for fibroblast samples and blood samples.

The expression outlier pipeline was adapted from OUTRIDER v.1.17.2.²⁹ A negative binomial distribution was modeled based on the raw counts of the gene across all samples using the R package MASS v.7.3-60.0.1. Expression outliers were identified by comparing the expression value of each gene in each sample against this distribution. The range of expression folds corresponding to a p value of <0.05 was defined as the reference range of a gene. The Benjamini-Yekutieli false discovery rate (FDR) method was used to adjust for multiple tests.

The splicing outlier pipeline was adapted from FRASER v.1.99.1.³⁰ We modeled the intron Jaccard index of each junction across the reference panel into a beta-binomial distribution using the R package VGAM v.1.1-9. Splicing outliers were identified by comparing the Jaccard index of each junction in each sample against this distribution. The range of Jaccard index corresponding to a p value of <0.05 was defined as the reference range of a junction. The Benjamini-Yekutieli FDR method was used to adjust for multiple tests.

For clinical performance, we evaluated the concordance of the outlier results with the known expression or splicing abnormalities in the positive samples. For expression abnormalities, the direction of change (up- or downregulation) and p value from OUTRIDER were used to determine whether the expected expression abnormality was detected. For splicing abnormalities, a splice-altering variant may result in changes in splicing conditions at multiple loci within a gene. For example, a DNA variant causing exon skipping leads to the alteration of multiple splice conditions: a reduction of Jaccard index for the intron proceeding and the intron following the skipped exon and an increase of Jaccard index for the junction spanning the skipped exon. In the clinical validation, the detection of splicing abnormality was defined as having at least one of the expected Jaccard index changes captured by FRASER. Sensitivity analysis was performed by using different p-value cutoffs to filter the outlier results. At each p-value cutoff, the number of outliers per sample and the number of positive cases were calculated.

Results

Design of a clinical transcriptome sequencing test and scope of the clinical validation

We developed a clinical RNA-seq test for the diagnosis of Mendelian disorders. This test is recommended for individuals with suspected genetic disorders who are undergoing or have completed a comprehensive DNA analysis, such as large-panel testing, ES, or GS. Acceptable sample types include blood or cultured fibroblast samples or total RNA extracted from these two sample types.

The test workflow is depicted in Figure 1. For fibroblast samples a poly(A) enrichment library is prepared, while for blood samples a ribosomal RNA-depletion library is used. RNA-seq is performed on the Illumina platform, targeting a depth of 150 million reads. The raw data are processed using an in-house analytical pipeline adapted from GTEx version 10.

Figure 1.

The workflow of clinical RNA-seq

The procedures of clinical RNA-seq are shown along with the passing criteria set for validation of the test.

An outlier-based pipeline is used to identify abnormal expression and splicing events, using fixed reference data for blood (n = 55) and fibroblast (n = 73) samples. The abnormal expression and splicing events are interpreted in combination with DNA variants and patient phenotypes by the American Board of Medical Genetics and Genomics (ABMGG)-certified professionals (Figure S1). The result interpretation and clinical recommendations are reported within a turnaround time of 8 weeks.

The validation workflow involves selecting validation samples, O&F, formulating QC metrics, testing reproducibility, evaluating analytical performance, establishing reference ranges, and assessing clinical performance (Figure 1).

Quality control

The clinical validation included 130 samples from 110 individuals, including 73 fibroblast samples, 55 blood samples, and 2 lymphoblastoid/lymphocyte samples (Table 1). The RNA-seq yield was targeted at 150 million reads. Before sequencing all samples, an O&F procedure was conducted on ten blood and ten fibroblast samples. This involved calibrating equipment, testing reagents, establishing protocols, and inspecting the QC metrics (Figure 2A). Both RNA/library quality and data quality were evaluated (Figure 2A). Notably, blood and fibroblast showed large discrepancies in exonic rate and split rate, likely due to differences in library design (ribosome depletion vs. poly(A)). This observation was further supported by comparing the same QC metrics between blood and fibroblast samples in the GTEx database, where poly(A) library preparation was used for all samples (Figure S2). After the O&F procedure, the remaining samples were sequenced and demonstrated a consistent distribution of QC metrics with the O&F samples (Figure 2A). No sample swaps were detected through SNP-based identity matching.

Figure 2.

Quality control and reproducibility tests for pre-analytical sample, post-analytical library, and RNA-seq data

(A) RNA-seq quality control metrics for obtained from optimization and familiarization (O&F) and other validation runs.

(B) Reproducibility tests for gene expression and splicing metrics for GM24385. Intra-batch comparisons are boxed in blue dotted lines, whereas inter-batch comparisons are boxed in red.

Reproducibility test

The reproducibility analysis was conducted to evaluate the consistency of results across repeat runs. The setup included five replicates: one triplicate in a batch and two individual runs in two other batches, enabling intra- and inter-batch reproducibility evaluation. We decided to calculate reproducibility using two fundamental measurements from RNA-seq results: gene-expression levels and splicing status at exon-intron junction sites. Three distinct samples were included in this analysis: GM24385 (a lymphoblastoid cell line from HG002), K562 (a lymphoblast cell line), and BG1477 (a blood sample).

The reproducibility of expression levels between each pair of replicates was calculated using read count for coding genes. The splicing-level reproducibility between each pair of replicates was calculated using read counts at canonical junctions annotated by GENCODE. Pearson correlation (on log-scaled counts) and cosine similarity (on raw counts) were employed as statistical methods to quantify reproducibility. We confirmed the sensitivity of these statistical methods by calculating reproducibility within and across samples (Table S1). Expression reproducibility for each of the three samples was above 0.99 using both statistical methods. Splicing reproducibility was above 0.99 for GM24385 and K562 but was around 0.98 for BG1477, likely due to differences in library design (Figures 2B, S3, and S4).

Validation of analytical performance

The validation of a diagnostic test’s analytical performance typically involves assessing the detection of fundamental analytical elements. As an analogy, DNA-based NGS often measures the accuracy of base calling using gold-standard samples. In the context of RNA-seq, we posit that the detection of gene-expression levels and splicing status at exon-intron junctions should be extensively characterized.

We used GM24385, the lymphoblastoid cell line from the GIAB reference individual HG002, for our benchmarking. We established expression and splice-junction benchmarks for GM24385 using publicly available RNA-seq data. The expression benchmark was constructed using four short-read RNA-seq datasets from two independent laboratories. We calculated the mean TPM for each protein-coding gene between two laboratories and selected genes for benchmarking based on different TPM thresholds (TPM ≥ 1, TPM ≥ 3, or TPM ≥ 5 for positive genes and TPM = 0 for negative genes), acknowledging that low-expression genes are more prone to inaccurate TPM estimates under the current provisional benchmark. The splicing benchmark was constructed using two long-read RNA-seq datasets from two independent laboratories. Positive junctions were defined as canonical junctions in the aforementioned positive genes that are detected in both long-read datasets. Negative junctions were defined as canonical junctions in the negative gene set that were not detected in either long-read dataset. The positive and negative genes/junctions were then used as a reference to calculate the sensitivity and specificity of our test. In our test data, a gene was considered detected if its read count was ≥50 and undetected if the read count was <50. Similarly, a junction was classified as detected if its junction read count was ≥5 and undetected if the count was <5.

When using the most stringent criterion (TPM ≥ 5) for positive gene definition, the five GM24385 replicates yielded a sensitivity ranging from 99.9% to 99.93% and a specificity of 100% (Table 2). Five of eight false-negative (FN) genes exhibited discrepancies (TPM fold change >2) in expression levels between the two benchmark resources (Table S2), suggesting an impact of laboratory context on RNA-seq results. This finding underscores the importance of including data from multiple sources and warrants further investigation. At the splicing level, the sensitivity ranged from 99.64% to 99.78%, and the specificity ranged from 99.79% to 99.9% (Table 2). Notably, none of the FN junctions overlapped with the expression-level FN genes, suggesting that FN junctions were driven by alternative splicing patterns rather than inadequate gene expression. This observation was confirmed by manual inspection (Table S3 and Figure S5). In contrast, false-positive (FP) junctions were mostly attributed to mapping issues (Table S3 and Figure S6). Including more low-expression genes in the benchmark (TPM ≥ 3 or TPM ≥ 1) decreased the concordance of gene and junction detection between the validation runs and the provisional benchmark (Figure S7 and Table S4). This observation suggests that low-expression genes have lower detection sensitivity in the clinical test or that the benchmark’s validity is reduced for these genes.

Table 2.

Analytical performance evaluation on GM24385 (HG002)

Repeat	Expression sensitivity	Expression specificity	Splicing sensitivity	Splicing specificity
R1	0.9993 (0.9985–0.9997)	1 (0.997–1)	0.9976 (0.997–0.998)	0.9983 (0.9966–0.9992)
R2	0.9993 (0.9985–0.9997)	1 (0.997–1)	0.9978 (0.9972–0.9982)	0.999 (0.9976–0.9996)
R3	0.9993 (0.9985–0.9997)	1 (0.997–1)	0.9967 (0.9961–0.9972)	0.999 (0.9976–0.9996)
R4	0.999 (0.9981–0.9995)	1 (0.997–1)	0.9976 (0.9971–0.9981)	0.9979 (0.9959–0.9989)
R5	0.9991 (0.9982–0.9995)	1 (0.997–1)	0.9964 (0.9958–0.997)	0.9986 (0.9969–0.9993)

We established provisional expression and splicing benchmarks using short-read and long-read RNA-seq data of the GM24385 (HG002) lymphoblastoid sample from the Genome in a Bottle Consortium. The benchmarks are based on short-read and long-read sequencing data from independent laboratories. Using a TPM = 5 for positive gene selection, the benchmark comprises 8,991 positive genes and 1,296 negative genes. At the splicing level, the benchmark includes 38,110 positive junctions and 4,195 negative junctions. Sensitivity is measured by the detection rate of positively identified genes or junctions, while specificity is measured by the proportion of negatively identified genes or junctions. Both sensitivity and specificity for expression and splicing were calculated across five experimental repeats, R1 through R5. The Wilson score method was used to calculate 95% confidence intervals for performance metrics.

Establishing transcriptome-wide reference ranges

The detection of outliers relies on the definition of a reference range, which is the range of values for given targets expected in a reference population. For this transcriptome test, we first aimed to define the scope of genes sufficiently interrogated by the assay for outlier analysis and then establish reference ranges for every gene within this scope. The 73 fibroblast samples and 55 blood samples from the validation cohort were used to establish the reportable targets and the reference ranges.

Among all 19,216 coding genes, 11,962 (62.2%) from blood and 12,614 (65.6%) from the fibroblast are deemed reportable (Tables S5 and S6), with a minimum of 50 sequencing reads from the average of all samples. When breaking down the gene list into nine disease-specific subsets, blood RNA-seq demonstrated gene coverage ranging from 56.8% to 85.4%, whereas fibroblast RNA-seq ranged from 67.3% to 86.7% (Figure 3A). Consistent with previous reports,^6,19 fibroblast demonstrated higher gene coverage compared to blood for all nine disease-specific panels tested except for immunodeficiency.

Figure 3.

Characterization of transcriptome-wide expression profiles and reference range

The gene-expression profiles (A) were based on the mean expression value of our fibroblast and blood reference cohorts. The proportions of genes with read counts ≥50 among all coding genes (n = 19,216) are shown across various disease-specific gene panels. The lower and upper boundaries of reference ranges for gene expressions are plotted in (B), and those for splicing are plotted in (C). The fold change/Jaccard index change (deltaJ) values corresponding to the lower and the higher boundaries for each gene/junction are depicted on the graph. The range between the lower and higher boundaries represents the reference range, while the ranges outside show the extent of abnormal changes that can be detected. Genes and junctions with low expressions are excluded from this analysis, resulting in the following number of targets in each assay: expression in blood, n = 11,962; expression in fibroblast, n = 12,614; splicing in blood, n = 105,114; splicing in fibroblast, n = 119,403. The splicing reference ranges of PRUNE1 is illustrated as an example based on our fibroblast (D) and blood (E) tests.

To establish expression reference ranges, we modeled the read count of each gene across the reference panel into a negative binomial distribution. The upper and lower boundaries of the range of expression fold changes corresponding to a p value of <0.05 were defined as the reference range of a gene (Tables S5 and S6; Figure 3B). The distribution of expression fold changes for all reportable genes showed mode values of 0.83–1.17 for blood and 0.85–1.16 for fibroblasts. This analysis suggests that the levels of dispersion of gene expression in blood and fibroblast are similar despite their different expression profiles.

To establish splicing reference ranges, we modeled the inclusion ratio (Jaccard index) of each junction across the reference data using a beta-binomial distribution. The range of delta Jaccard index (deltaJ) corresponding to a p value of <0.05 was defined as the reference range of a junction (Tables S7 and S8; Figure 3C). As an example, the splicing reference ranges for all splice junctions in PRUNE1 are shown in Figures 3D and 3E.

After excluding canonical introns with fewer than three junctions in more than 20% of samples, 119,403 and 105,114 canonical introns remain for splicing reference range analysis in fibroblast and blood, respectively. The splicing reference range pattern differs dramatically between the two sample types. The distribution of fibroblast junction Jaccard index shows a narrow reference range (equivalent to a wide outlier detection range) with mode values at −0.003 to 0.0002, compared to the mode reference range of −0.13 to 0.1 for blood. This difference is likely attributed to the distinct library preparation methods used. The poly(A) enrichment protocol for fibroblast boosts the proportion of canonical splicing in mature mRNA, leading to lower variation in Jaccard index. In contrast, the ribosome-depletion protocol for blood preserves unspliced pre-mRNA, resulting in higher Jaccard index variation.

Validation of clinical performance

We propose implementing a dual approach for the clinical review of transcriptome RNA-seq data. Thus, the clinical validation should be designed to align with the proposed clinical workflow. This dual approach integrates parallel DNA-driven and RNA-driven review procedures.

In the DNA-driven procedure, DNA data, such as whole-genome sequencing, is reviewed first to identify candidate diagnostic variants. These variants guide targeted interpretation of the RNA-seq data, focusing on specific genes. For this review mode, validating the sensitivity of gene-level outlier detection is critical.

The RNA-driven review procedure, in contrast, involves transcriptome-wide clinical interpretation without prior DNA data review. Because this review is performed on a transcriptome-wide level agnostic to DNA data, statistical detection of outlier events requires multiple testing correction. Key metrics for this mode include detection sensitivity and the number of positive outliers labeled by the pipeline, reflecting the practical workload for clinical interpretation.

To calculate the clinical detection sensitivity of RNA-seq, we considered two primary outlier types: gene-expression outliers and splice-junction outliers. We calculated stand-alone detection rates for each of the two outlier types. Additionally, we determined individual-level detection rates by requiring the detection of only one outlier type when both expression and splicing outliers are expected in a single person. This approach is justified because, when two outlier types are predicted from one event, it typically involves cryptic splicing that leads to nonsense-mediated decay, resulting in reduced expression levels. The intensity of residual cryptic splicing and the degree of gene-expression reduction tend to counterbalance each other.

Positive clinical samples from the UDN, comprising 20 blood samples and 20 fibroblast samples from 23 individuals, were selected for the evaluation. These individuals harbor diagnostic variants encompassing a wide range of variant types, including putative loss-of-function variants, splicing variants, and deletions. The resultant RNA changes span a full spectrum including aberrant expression at the gene level and aberrant splicing in the form of cryptic splice sites, exon skipping, cryptic exons, and fusion genes. To ensure the authenticity of these positive findings, all samples required at least two independent sets of experimental data affirming the diagnostic finding. The DNA variants, expected RNA findings, and descriptions of the supportive evidence are listed in Table S9. In some individuals both expression and splicing abnormalities were expected, while in others only one type of abnormality was anticipated. For our clinical validation, the detection of at least one type of abnormality (expression or splicing) was required in order to make a molecular diagnosis (Figure 4A).

Figure 4.

Clinical performance validation

(A) Clinical performance in fibroblast samples and blood samples. We evaluated whether each expression/splicing change was detected as an outlier by our clinical pipeline and the significance level of the outlier. The “overall” column represents the most significant outlier event of the gene. “Not applicable” indicates that the expression/splicing change in this gene is not expected. FDR, false discovery rate.

(B) The relationship between the number of outliers and the number of positive cases successfully detected by the outliers at different p-value cutoffs. Horizontal dashed lines indicate the total number of positive cases. Vertical dashed lines indicate the number of outliers at p = 0.05, and FDR = 0.05.

Using a p-value cutoff of 0.05, simulating the DNA-first targeted RNA analysis approach, we detected an average of 1,060 expression outliers and 3,910 splicing outliers per sample in fibroblast samples, and an average of 1,182 expression outliers and 6,920 splicing outliers in blood samples. For detection sensitivity, we identified 12 out of 13 expected expression abnormalities and 16 out of 17 splicing abnormalities in fibroblast samples (Tables S10 and S11; Figure 4A). For combined sample-level findings, 19 out of 20 fibroblast samples showed at least one outlier abnormality. In blood samples, we identified 10 of 12 expected expression abnormalities and 15 of 17 splicing abnormalities, leading to a combined detection rate of 19 out of 20 samples (Tables S10 and S11; Figure 4A). The missing molecular diagnoses were attributed to two main factors: high background noise obscuring the low-impact positive findings (low expression of AP4M1 in blood, Figure S6; low reduction of PPP3CA expression in fibroblast; Table S10) and the apparently variable efficiencies of nonsense-mediated mRNA decay (NMD) (low NMD for AP4M1 expression in blood, Figure S6).

We then investigated how adjusting p-value cutoffs affects the number of outliers to review (reflecting positive predictive value) and the number of positive molecular diagnoses identified (reflecting sensitivity). Our findings indicate that fibroblast data are more robust against stringent p-value cutoffs. In fibroblast samples, most positive findings remain detectable even when cutoff stringencies are raised to retain approximately 100 reviewable expressions or splicing outliers (Figure 4B). In contrast, applying the same filtering criteria to blood samples results in the loss of more than half of the positive findings (Figure 4B).

When multiple testing correction is applied at FDR < 0.05, simulating the RNA-first analysis approach, the clinical detection sensitivities are reduced to 14 out of 20 for fibroblast samples and 6 out of 20 for blood samples (Figure 4A). This stringent threshold also significantly lowers the number of positive outliers identified, with an average of 0.8 expression outliers and 8 splicing outliers in fibroblast samples and an average of 0.6 expression outliers and 3 splicing outliers in blood samples.

Therefore, we predict that the actual detection sensitivity in clinical practice will fall between the levels observed in DNA-first and RNA-first approaches, influenced by the availability of DNA data and the success rate of identifying diagnostic candidates from it. Clinical laboratories must carefully balance clinical sensitivity with interpretation workload and make informed decisions regarding filtration cutoffs for reviewable data. Under the current interpretation framework, incorporating DNA variants into the RNA data review ecosystem is an effective strategy for enhancing the overall diagnostic workflow (Figure S1).

Discussion

Here, we report the development and clinical validation of an RNA-seq test for the diagnosis of Mendelian disorders. We provide essential resources and considerations for conducting clinical validation along with metrics necessary to define test limitations in the validation and to monitor quality performance during production post validation.

Previous RNA-seq benchmarking studies have primarily focused on evaluating performance metrics that reflects junction discovery as well as differential expression analysis.^31,32,33 For instance, the MicroArray and Sequencing Quality Control (MAQC and SEQC) initiatives utilized RNA samples from multiple cell lines and donor brain tissues, spiked with synthetic RNA controls, to systematically assess cross-platform and cross-laboratory consistency.³⁴ The Quartet project leveraged immortalized B-lymphoblastoid cell lines from a quartet family to evaluate genes with a wide range of differential expression levels.³² In these studies, gold-standard references were typically derived from synthetic RNA spike-ins, predefined sample mixing ratios, and complementary experimental methods such as RT-PCR and microarray measurements.

In contrast, due to different application scenarios and analytical objectives, our study tailored its benchmarking strategy to the clinical diagnostic context of Mendelian disorders, with a focus on expression and splicing characterization and outlier detection. Specifically, we developed a provisional benchmark using the GM24385 lymphoblastoid cell line from the GIAB Consortium, integrating both short-read and long-read RNA-seq data. This approach prioritizes clinically relevant endpoints, such as the detection of expression outliers and splicing abnormalities, rather than emphasizing general performance metrics. Furthermore, our study underscores the need to continuously improve the quality of gold-standard data, particularly to enhance the assessment of genes with low expression or challenging mapping metrics. Analytical methods with high sensitivity, such as ultra-deep RNA-seq, hold promise for addressing these limitations.

Detection of abnormal expression and splicing depends on the reference range of the target in the tested tissue, which is influenced by both the abundance and variability of the target gene expression. The commonly used metric for expression level, TPM, falls short in precisely estimating the clinical reference range. For example, compared to TRIP11, PPP3CA has a higher TPM in fibroblasts (31.78 vs. 20.02) but a less sensitive assay reference range (0.72–1.26 vs. 0.81–1.21). The wide reference range of 0.72–1.26 predicts that the expected positive finding with an expected fold change at 0.81 falls within the interval of background noise, resulting in FN detection. Interestingly, PPP3CA has a more sensitive reference range in our blood assay (0.85–1.16), enabling the detection of this challenging expression reduction in blood. A similar trend of a more sensitive reference range in blood compared to fibroblasts is observed for many other genes (Tables S5 and S6; Figure 3B). In the case of the DNM1 variant, although gene-level expression is adequate, reference-range analysis reveals insufficient sequencing coverage at the junction region for the expected abnormal splicing, which explains the FN result for the splice junction.

The rate of NMD is a factor that can also influence diagnostic performance. When it occurs at high efficiency, NMD degrades abnormal junctions, making it easier to detect expression outliers but harder to detect splicing outliers. This explains the FN aberrant splicing detection of the MIPEP junction in fibroblast and the DNM1 junction in blood. When NMD occurs at low efficiency, expression outliers may become too modest to be detected. Our RNA-seq results revealed different fold changes for expression reduction of AP4M1 between fibroblasts and blood, suggesting that NMD is incomplete in the blood, contributing to FN expression results in blood (Figure S8). The low expression level at the junction site in blood contributed to the FN detection of aberrant splicing. Further understanding of tissue-specific NMD rates will guide future design and interpretation of the RNA-seq tests.

The diagnostic capability of RNA-seq tests is influenced by several technical factors, including, but not limited to, sample collection methods, cell-culture techniques, cDNA synthesis, library preparation, sequencing read length, sequencing depth, bioinformatics pipelines, and the configuration of control datasets (Table 3). Although many of these variables were not specifically examined in the current study, they warrant careful consideration and standardization during the clinical implementation of RNA-seq.

Table 3.

Factors that can impact the sensitivity and specificity of RNA-seq test

Factor	Description	Impact on result	Recommendation
Sample collection	for blood sample, the type of tube used; for fibroblast samples, the site of skin biopsy	different sample collection methods may introduce methodological variability, impacting statistical results	standardize the sample collection method
Cell culture	for fibroblast samples: culture conditions, passage number, and circadian rhythm before RNA extraction	variations in cell conditions may result in technical variation, influencing the statistical outcomes	culture fibroblast cells under consistent conditions before RNA extraction
cDNA synthesis method	the protocol and kit used for cDNA synthesis before sequencing	different cDNA synthesis methods can introduce biases (e.g., base composition bias, strand-specificity issues) that impact gene-expression quantification and splicing detection	choose a cDNA synthesis method that minimizes bias and aligns with the specific goals of the RNA-seq experiment
Library preparation	the use of poly(A) or ribosome-depletion kits during library preparation	ribosome-depletion kits may capture more pre-mRNA, resulting in lower sensitivity in junction analysis	use poly(A) kit for coding RNA and ribosome-depletion kit for non-coding RNA
Read length	insert size and read length of RNA-seq	longer read lengths may improve sensitivity and specificity in junction analysis	opt for longer read lengths when possible
Sequencing depth	sequencing throughput of RNA-seq	higher sequencing depth may increase sensitivity but reduce specificity	balance sequencing depth with cost considerations and sequencing noise
Bioinformatics pipeline	the software and algorithms used for aligning, normalizing, analyzing RNA-seq data, and the choice of genome build	the bioinformatics pipeline, including genome build selection, can significantly impact gene-expression estimates, detection of splicing events, and overall reproducibility and interpretation of results	select bioinformatics tools and pipelines based on their performance characteristics, suitability for the dataset, and robustness across different genome builds
Control dataset	the control RNA-seq dataset used for outlier analysis	larger control sample sizes may enhance sensitivity. Heterogeneity in the reference dataset can affect outlier analysis results. Unknown or undetected shared ancestry between case and controls	use a larger control sample size and improve homogeneity between the test and control datasets

Pre-analytical variables can be introduced at various stages, such as during sample collection, handling, and cell culture. The use of blood-collection tubes, for example, can affect RNA integrity, yield, and gene-expression profiles, potentially reducing the consistency of gene-expression data across different runs.^35,36 Similarly, variations in skin biopsy collection, such as differences in the collection site, can lead to discrepancies in gene expression.³⁷ Additionally, different culturing techniques and the intrinsic characteristics of the cultured cells, such as passage number and metabolomic status, can also influence gene expression.

While it is essential for clinical labs to standardize protocols to control these variables, these challenges suggest that developing a gold standard for gene-expression benchmarks based on precise read-count values may not be feasible due to the high level of noise introduced by lab-specific practices in cell-line strains, RNA extraction, and library preparation. To address this issue, we tentatively defined the gene-expression benchmark using a binary classification of expressed or unexpressed genes. This approach allowed us to derive positive and negative gene sets from the short-read data of the reference sample GM24385 from GIAB for benchmarking. Although this method provides a low-resolution characterization of expression levels, it enables inter-laboratory performance comparisons. As a complementary assessment, precise read-count values were utilized in the reproducibility evaluation, allowing clinical labs to ensure consistent experimental and analytical procedures across replicates without relying solely on benchmark data.

Post-analytical variables can arise from sequencing experiments and data-analysis procedures. Bias related to base composition, transcript strandedness, and coding versus non-coding characteristics can be introduced during cDNA synthesis and library preparation.^38,39 Additionally, sequencing factors such as read length and sequencing depth play crucial roles in influencing test performance. Although we did not validate long-read sequencing, we utilized long-read RNA-seq data from GM24385 to construct the junction-level benchmark due to its ability to capture full-length transcripts.⁴⁰ The extended read length facilitates accurate identification and quantification of splice junctions, enabling the detection of complex alternative splicing events that resulted in FN junctions during validation (Table S4 and Figure S4).

Regarding sequencing depth, we opted for a higher overall throughput (∼150 million reads per sample) compared to previous RNA-seq diagnostic studies.^5,6,7 This decision was prompted by our separate investigation, which highlighted the benefits of conducting RNA-seq at higher depth.⁴¹ We found that more genes and isoforms can be detected at higher depths, and the number keeps increasing even at a depth of 1 billion reads. The rate of positive findings increases with sequencing depth, suggesting a higher clinical sensitivity at higher depth. The number of total outliers detected also increases with sequencing depth, suggesting a decreasing specificity.

Decisions related to the bioinformatics pipeline, such as the choice of reference genome build version, impact the clinical performance of RNA-seq.⁴² Importantly, the robustness of outlier analysis depends on the sample size and homogeneity of the control cohort. Further research is needed to determine an optimum control sample size and to prioritize critical factors for homogeneity, such as gender and age. In clinical practice, increasing the control sample size with the accumulation of RNA-seq data will increase the power of outlier detection and enable reanalysis. In conclusion, our study provides a paradigm and necessary resources for independent laboratories to validate a clinical RNA-seq test.

Data and code availability

Raw RNA-seq data from the validation cohort have been deposited in the NCBI Sequence Read Archive under accession number NCBI: PRJNA1124992.

Acknowledgments

The research was supported by the National Institutes of Health Common Fund (U01HG007709 and U01HG007942) and a grant from the National Human Genome Research Institute (R35HG011311).

Declaration of interests

Baylor College of Medicine (BCM) and Miraca Holdings Inc. have formed a joint venture with shared ownership and governance of Baylor Genetics (BG), which performs genetic testing and derives revenue. P.L., H.D., and C.E. are employees of BCM and derive support through a professional services agreement with BG.

Published: March 4, 2025

Web resources

• GTEx Portal, https://www.gtexportal.org/home/

• gnomAD, https://gnomad.broadinstitute.org/

• Genome in a Bottle, www.nist.gov/programs-projects/genome-bottle

• Undiagnosed Diseases Network, https://undiagnosed.hms.harvard.edu/