Approaching neural net feature interpretation using stacked autoencoders: gene expression profiling of systemic lupus erythematosus patients

Abstract

Systemic lupus erythematosus (SLE) is a rare, autoimmune disorder known to affect most organ sites. Complicating clinical management is a poorly differentiated, heterogenous SLE disease state. While some small molecule drugs and biologics are available for treatment, additional therapeutic options are needed. Parsing complex biological signatures using powerful, yet human interpretable approaches is critical to advancing our understanding of SLE etiology and identifying therapeutic repositioning opportunities. To approach this goal, we developed a semi-supervised deep neural network pipeline for gene expression profiling of SLE patients and subsequent characterization of individual gene features. Our pipeline performed exemplar multinomial classification of SLE patients in independent balanced validation (F₁=0.956) and unbalanced, under-powered testing (F₁=0.944) cohorts. A stacked autoencoder disambiguated individual feature representativeness by regenerating an input-like(A ‘) feature matrix. A to A’ comparisons suggest the top associated features to be key features in gene expression profiling using neural nets.

Introduction and Background

Introduction to scientific question: Systemic lupus erythematosus (SLE) is a rare, heterogeneous autoimmune disorder known to affect most organ sites¹. The heterogenous disease state complicates clinical management and identification of actionable clinical subtypes¹. SLE is B-cell based disorder, that uniquely shares etiologic characteristics with B-cell maliginancies². While small molecule drugs and biologics are available to treat SLE, including those more commonly utilized for therapeutic management of B-cell malignancies, they are often blunt in efficacy. While characterizing complex biological signals is critical to understanding etiology and clinical subtypes, SLE is ripe for therapeutic repositioning due to the availability of numerous biologics with known cell surface targets that are currently classified as orphan drugs³. Treatment personalization also stands to gain from unpacking the complex biological signals contained within gene expression profiles.

Neural nets background: While traditional deep learning approaches are known to produce high-performing classification algorithms, their broad applicability have often been hampered by their lack of interpretability due to the use of complex, high-dimensional networks⁴. Such complexity can limit description of model components (features) and interpretation of their importance⁵. In translational bioinformatics-led inquiry, model performance and biological interpretability are of simultaneous, critical importance⁶ – approaches with limited ability to describe their feature space (i.e. biological interpretability) are particularly problematic in guiding biological inference in gene expression profiling endeavors. New approaches are needed to access the features contained within the ‘memory’ of neural network hidden layers that are needed to make strong predictions in neural network classification.

In this study, we introduce an approach for reversal of neural networks using a stacked autoencoder. Our corresponding pipeline allows for generation of high-performing gene expression profiles and characterization of individual biological features well-represented across the deep neural network. We provide a first ever deployment of this pipeline with an application of gene expression profiling of SLE patients from bulk RNA-seq data. Our current study is a stand-alone innovative methodological application and constitutes a preliminary study within our larger project focused on feature interpretation from deep neural networks. We aim for this current study to clarify the potential utility of this approach in classifying and regenerating the input feature matrix overall and amongst unbalanced classes with limited statistical power individually. The development of this pipeline represents an informatics innovation. The successful deployment performance of this pipeline suggests potential applicability in therapeutic repositioning and precision medicine discovery, were subgroups are commonly small and underpowered with limited pathophysiologic differences (i.e. weak signals) between treatment options in a heterogenous disease state.

Methods

Data sources: A compendium of gene expression observations (n=l,576) characterizing disease state of SLE patients⁷ was used in this study. The dataset contained 15,838 features – encompassing SLE disease state classes, cluster of differentiation biomarker expression features⁸, and other gene expression features. These data were used for training, validation, and testing of pipeline modules, authentication of gene expression profiles, and statistical evaluation of feature uniqueness and individual feature importance. The SLE Compendium is deidentified and readily downloadable by external research teams for benchmarking purposes (In the event the link is found broken, please email a member of the research team to obtain access to the dataset): https://upenn.box.eom/s/nuiv0292vbwkolmrqtzqcerpcvue4mwu

Pipeline overview: We have assembled a deep learning pipeline that generates biologically-interpretable gene expression profiles from highly-accurate neural network classifications. Outputs from intermediate layers of our neural nets are taken out and fed through an “reversal” pipeline, consisting of a stacked autoencoder⁹. This autoencoder regenerates the input matrix with an ‘input-like’ matrix, encompassing estimates of the original gene expression features and reflective of features that were well-represented (‘representativeness’) by our neural network classifier. Our pipeline currently allows for characterization of individual feature ‘representativeness’ within a gene expression profile ascertained deep neural network via 1:1 input matrix [A] to input-like matrix [A’] feature comparisons). Future work will build on this architecture, with plans that encompass development of individual feature uniqueness, feature importance, and profile stability evaluation approaches. An overview of our study and pipeline, termed as the ‘Neural Network Reversal Pipeline’, are provided below (Figure 1). The pipeline code has been publicly released on the Breitenstein Lab GitHub page: https://github.com/BreitensteinLab/NeuralNets Featurelnterpretation.

Figure 1:

Overview of pipeline.

Figure 1 Legend: Our pipeline consists of 3 distinct modules (i-iii) – The pipeline initiates with multinomial classification of SLE patients (module i) using multilayer perceptron deep neural networks. This architecture consists of several hidden layers (i.e. transformation of matrices [A B C]) of far lower dimension than the inputted gene expression data (input matrix). Prediction (i.e. classification) of observations was performed by passing the output of the hidden layers through an output layer resulting in a 3 nodes output. These are then passed through a softmax activation layer to obtain predictions of labels with associated probabilities. Module iii) The intermediate output of a hidden layer (in this case, the penultimate layer l₂ from module i) was introduced as a latent data input into a decoding “generatof’ neural network, for input feature matrix regeneration. Module ii) The ADAM optimizer was applied iteratively to both the classifier and generator neural networks. From there optimal hyperparameters were obtained using Bayesopt, a Bayesian optimization commonly used in hyperparameter tuning. Module iv) Statistical characterization of ‘representativeness’ was performed on the regenerated ‘input-like’ matrix (e.g. [A’ B’ C’]), which maps 1:1 (e.g. A:A’, B:B’, C:C’) with the original input matrix.

Note: A key assumption of our pipeline is that provided a low-enough intermediate output dimension, only features deemed to be important by the classifier (module i) would be “remembered” and reasonably regenerated (i.e. reversed) by the generator (module iii). In effect, reversal of the network amplifies error accumulated across pipeline training, which manifests as generating less-representative estimates of the original feature expression values. The research team discloses a current mathematical limitation in disambiguating the source of error as attributable to a feature being placed into the null space (non-random error – our desired characteristic) or to standard random error accumulated in network training. Regardless of the source of error, the regenerated input-like feature matrix holds critical clues to understanding the relevance of individual features within the gene expression profile.

Pre-processing: Observations were randomly subset into training (60%), validation (20%), and test (20%) cohorts. To overcome known class imbalance within our input dataset, Synthetic Minority Over-sampling Technique (SMOTE)¹⁰, was performed on the training as well as validation dataset using Python’s imbalanced-learn library. The dataset has been previously studied by the research team and has been previously studied via Relief-based machine learning¹¹ and extensively characterized⁷.

Multilayer perceptron classifier (Module i): Our classifier module used a multilayer perceptron (MLP) approach^18,19, whose structure allows for non-linear activation. Each linear threshold unit (LTU) in the hidden layers consists of Leaky ReLU activation with slope= 0.2. This activation function introduces non-linearity in the network. Leaky ReLU, being invertible, were used for the sake of a linear algebra approach for network reversal. The classifier network was trained using one input layer and 2 hidden layers, containing 256 (l₁) and 128 nodes (l₂), respectively. (Note: any number of divisible nodes can be used to a neural network). The number of nodes in the output layer (l₃) is equal to the number of classes i.e. 3, and thus returns a 3-dimensional “logit” output.

MLP utilizes a supervised backpropagation technique for classifier model training. To obtain predictions, the logit output obtained from our output layer (l₃), is passed through a softmax activation, which results in prediction of class label probabilities²⁰. In training the model, the cross entropy loss between the predicted predictions and true classes was used. This scheme generated traditional statistical predictions. In order to avoid our model from overfitting the training data and improve its performance, L₂ regularization was used to limit weight values.

Stacked autoencoder (Module iii): The ill-posed problem of network reversal²¹ (mapping from intermediate output space to input space) was addressed using a decoding “generator”, a stacked autoencoder⁹. It was composed of 2 hidden layers l₁’ and l₂’ consisting of 256 and 512 nodes respectively. Leaky ReLU activation was used to introduce non-linearity in the generator network. The output of these hidden layers was then passed through an output layer to obtain a generated gene expression data. The process from input data to the generator output is structurally identical to an autoencoder; however, each section of the autoencoder is trained separately, with the encoder primarily being trained and used for the classification task, and the decoder only on the reversal task. Furthermore, the decoder is penalized on both the fidelity of reconstruction, as well as the ability for the classifier to correctly identify the correct class from generated output. The ratio of these two losses added together can be adjusted by an additional alpha hyperparameter to bias this towards generating outputs that ensure classification and outputs that have high fidelity.

Pipeline optimization (Module iii): Bayesopt, a Bayesian hyperparameter tuning package, was used for hyperparameter tuning. The optimal hyperparameters of the learning algorithm (learning rate, beta1 and beta2, epsilon, regularization scale, and number of epochs) were first determined using the Bayesian optimization technique. Following identification of optimal hyperparameters by Bayseopt, The ADAM (Adaptive Moment Estimation) Optimizer was used to update the parameters (weights) in our model with 31 iterations per epoch. An ADAM Optimizer is gradient-based optimization algorithm that has seen much success in training neural nets.

Optimization was performed sequentially for both our classifier and generator networks. The performance of the generative network reversal module is optimized by minimizing the generator loss composed of two components: comparison loss between our input and input-like matrix and the classifier loss. A hyperparameter ‘a’ is introduced to represent the importance of the classifier loss in the overall loss. The optimum values for hyperparameters (learning rate, beta1, beta2, epsilon, alpha and maximum number of epochs) were obtained using Bayesian Optimization on the generator – following Bayesopt optimization, identified hyperparameters became fixed, whereas the ADAM Optimizer performed iteratively on neural networks as new data perspectives are passed through the pipeline – ADAM optimization was used to update the weights in the hidden layers of the generator.

Network authentication (module iv) – subject of future directions: We suggest using the following measures to authenticate inclusion of individual features, and overall components of the network^22,23: Feature Reproducibility associations (e.g. A’ to A) assesses the ability for additional (assumed to be latent) features to explain network variation as captured by an original feature (A) or reverse generated feature (A’). Feature independence via canonical correlations (e.g. A’ to B C ) to asses potential feature independence and uniqueness of a network features. Feature reliability via boot strap replication to generate confidence intervals for individual feature expression estimates.

Note: Despite knowing SLE to have heterogenous gene expression profiles, the gene expression features are assumed to overlap due to bona fide interdependence of features within biological networks. As such, while independence might indicate the statistical uniqueness of an individual feature, this does not necessarily equate to importance of a feature within a network. In particular, interdependence of features might potentially point to a stronger meta-signal of biological features, a subject of our recent¹¹ and ongoing research.

Results

A matrix of 1,576 observations and 15,838 features was available for analysis. Labelled patient classes consisted of 160 healthy control observations (Class 1 – controls), 1,290 observations from SLC patients who received only standard of care (Class 2 – standard treatment), and 126 observations from SLE patients who were exposed to various experimental treatments (Class 3 – experimental treatment). Due to known class imbalance within the original dataset, upsampling was utilized to randomly select a balanced population (n=762) with 254 observations within each class for the training and validation cohorts. For parsimonious explanation of pipeline performance, we highlight accuracy and error measures for classification of the SLE standard of care group (class 2) from the other two observations classes. In interpretation of classification performance, it is important to note that pathophysiology of SLE standard of care patients and SLE experimental treatment patients (class 3) are posited to have only nuanced pathophysiological differences, whereas the healthy controls (class 1) will be very distinct from either class 2 or class 3 observations. Ability to classify observations as belonging to class 3 with high accuracy and low error is a noteworthy achievement of this pipeline.

Part 1: Pipeline Training-balanced data

Bayesopt was used for hyperparameter tuning and included parameters needed for the ADAM optimizer to further tune (learning rate, beta1, beta2, epsilon), the number of epochs to train, and the scale of regularization (L2 penalty). Bayesopt was run for 250 iterations for the classifier and 75 iterations for the generator to achieve high-performing and robust training parameters across multiple seed values. The variation in validation loss was estimated by training the model with tensorflow’s default ADAM optimizer parameters for 50 epochs and no regularization a total of 30 seed values, and obtaining the standard deviation of the validation loss. Classifier training: Optimal classifier performance was achieved with 3,100 iterations, where Learning Rate = 1.000e^-2, β₁ = 1.550e^-1, β₂ = 9.816e^-5, ε = 8.355e^-1, Epoch=100, L2 penalty(regularization scale) = 1.057e^-5 parameters were identified. These optimal parameters were applied for evaluation of classifier performance on the balanced validation cohort and underpowered, unbalanced testing cohort. Generator training: Optimal generator performance was achieved with 1,799 iterations, where (LR = 2.783e^-3, β₁ = 2.124e^-1, β₂ = 3.404e^-1, ε = 1.424e^-1, Epoch=58 parameters were identified. These optimal parameters were also applied for evaluation of generator performance on both the validation cohort and testing cohorts.

Part 2: Pipeline Validation – balanced data

Classifier (module i): Overall, within our balanced validation cohort, our pipeline achieved strong performance (P>0.95, R>0.94, and F1>0.95) and produced minimal error (<0.02). Within our target class of standard of care SLE patients, the classifier model performed with high precision (P=0.972) and recall (R=0.941), and minimal error. Class 3 observations achieved top balanced accuracy (F1>0.97). Corresponding confusion matrix and model performance characteristics are detailed below (Table 2a).

Table 2a.

Classifier network performance: Validation cohort (balanced data)

	Prediction			Accuracy			Error
True class (label)*	Class 1	Class 2	Class 3	P	R	F1	Type I	Type II
Class 1 (control)	247	7	0	0.9574	0.9724	0.9648	0.0144	0.0092
Class 2 (SLE standard)	7	239	8	0.9715	0.9409	0.9560	0.0092	0.0197
Class 3 (SLE experimental)	4	0	250	0.9690	0.9843	0.9766	0.0105	0.0052

^*Due to known class imbalance, up-sampling was used to introduce a balanced population (n=762) for classifier training. A confusion matrix noted predicted classes in comparison to true class labels. Precision (P), Recall (R) and F1 - the weighted harmonic mean of precision and recall – measures were used to describe training model accuracy. Estimates of model error were ascertained from false positive (Type I) and false negatives (Type II) rates.

Generator (module iii): For all classes of observations our pipeline performed exemplar, suggesting that we can reliably achieve feature matrix regeneration for most observations on balanced data. Within our target class of standard of care SLE patients, the generator model performed with high precision (P=0.964) and recall (R=0.950), and minimal type I and type II error. The corresponding confusion matrix and model performance characteristics are listed below (Table 2b).

Table 2b.

Generator network performance: Validation cohort (balanced data)

	Prediction			Accuracy			Error
True class (label)	Class 1	Class 2	Class 3	P	R	F1	Type I	Type II
Class 1 (control)	245	9	0	0.9800	0.9646	0.9722	0.0066	0.0118
Class 2 (SLE standard)	5	241	8	0.9640	0.9488	0.9563	0.0118	0.0171
Class 3 (SLE experimental)	0	0	254	0.9695	1.0000	0.9845	0.0105	0.0000

Part 3: Pipeline Testing – unbalanced data

Overall, our pipeline characterized standard of care SLE observations with acceptable accuracy in both the classifier (F1=0.946) and generator (F1=0.944) Potential physiologic-driven errors were noted when classifying the nuanced differences between standard of care and experimental treatment SLE observations, with substantially attenuated performance observed for both classifier (F1 = 0.761) and generator (F1=0.757). Further, it is noteworthy that while both class 1 and class 3 are statistically underpowered, only Class 3 observations suffered from significant (T1=0.054) error within the generator. Evaluation of feature representativeness is recommended to occur amongst only Class 2 observations on unbalanced data.

A confusion matrix noted predicted classes in comparison to true class labels. Precision (P), Recall (R) and F1 – the weighted harmonic mean of precision and recall – measures were used to describe training model accuracy. Estimates of model error were ascertained from false positive (Type I) and false negatives (Type II) rates.

Discussion

Overview of findings: We developed a novel neural network reversal pipeline that is capable of recommending classical predictions while suggesting relative importance of specific features (in our study individual genes) needed to train a high-performing neural network. (1) Such an approach helps bring us closer to solving the vexing problem in deep learning of obfuscated networks, with little room for biological interpretability. This pipeline consisted of paired classifier and generator (network reversal) models that were (2) high-performing on balanced training data, and (3) high-performing on unbalanced testing data for large classes. (4) Our pipeline successfully reversed the classifier network to ascertain a gene expression profile of standard of care SLE patients. (5) Our pipeline successfully classified and regenerated an input feature matrix with minimal pre-processing and no prior dimensionality reduction. (6) Our pipeline is readily portable to other disease applications and biomarker expression data sources. Improvements to the generator model training are planned to allow for scaling to classification using substantially larger data matrices, such as choosing static optimization thresholds, with potential relevance to single cell RNA-seq applications. While results are stand-alone, this study is our initial application and serves as preliminary research. Fine-tuning of the model to optimize computational performance and validating measures for network authentication is a subject to our ongoing research.

Potential Limitations: (1) Primary study findings were generated from secondary data, which were pooled and normalized to form a compendium of SLE patients. Despite our conspicuous labelling of study biomarkers and cohort characteristics, care should be taken not to over-interpret ascertained biological insights from this data source. Independent in vitro and vivo replication of findings is planned for future directions. (2) The study evaluated bulk RNA-seq to generate gene expression profiles of SLE. While we anticipate the pipeline can be readily adapted to single cell RNA-seq data, such a benchmark needed to make this claim with certainty is not contained within this study. (3) Hyperparameter tuning is a potential limiting (computationally intensive) step. (4) Mathematical advancements are needed to characterize model error contained within the regenerated feature matrix as belonging to random noise accumulated during model training or error attributable to a feature being pushed into the null space during network training – the latter of which is our desired source of error

We demonstrated our pipeline to perform classification and regeneration with exemplar accuracy and minimal error on independent balanced validation data. Within an unbalanced testing dataset, also constrained by limited statistical power, the pipeline was successful in characterizing the largest class. However, performance was substantially attenuated for classification of smallest classes, suggesting these small classes are poor candidates for regeneration of the feature matrix via the stacked autoencoder. Together these findings suggest that gene expression profiling using deep neural nets might successfully be performed on both balanced and highly unbalanced data, for large classes. Limitations for classification of small classes, might be overcome by standard balancing approaches. Finally, the underlying strength of deep neural nets allows with easily implementable tuning and balancing approaches allows for classification and regeneration of feature matrix between treatment classes with only nuanced pathophysiological differences. While additional mathematical and statistical enhancements will strengthen confidence in the regenerating matrix estimates of feature expression, being apparently robust to sample size and balance limitations highlight this approach as being of great value for future precision medicine applications and therapeutic repositioning endeavors. The potential generalizability of this pipeline for therapeutic repositioning via interpretation of the ascertained inputlike feature matrix is subject to our ongoing research.

Conclusion

We have developed a neural network pipeline with a stacked autoencoder that performs highly accurate multinomial classification and regeneration, both of which are needed for human interpretable gene expression profiles ascertained from deep neural networks. Our pipeline was robust to unbalanced data that also had limited statistical power and only nuanced differences in biological signals between therapeutic classes. Our findings suggest opportunity for application of similar deep neural network pipelines in evaluation of therapeutic response using real-world gene expression profiling data.

Table 1.

SLE Compendium Characteristics

	Cohort 112	Cohort 213	Cohort 314	Cohort 415	Cohort 516	Cohort 617	Overall
Study PMID	18631455	23203821	24644022	25736140	27040498	26138472	---
Study GEO identifier	GSE11907	GSE39088	GSE49454	GSE61635	GSE65391	GSE78193	---
Healthy control*	0	46	0	30	72	12	160
median age (range)	---	34.5 (19-50)	---	---	12 (6-21)	---	16 (6-50)
gender - female/male	---	34	---	---	57	---	91
SLE standard of care	37	21	177	99	924	32	1290
median age (range)	14 (8-17)	43 (20-50)	40 (18-71)	---	15 (6-19)	---	16 (6-71)
gender - female	35	21	148	---	817	---	1021
SLE experimental treatment	0	57	0	0	0	69	126
median age (range)	---	36 (19-50)	---	---	---	---	36 (19-50)
gender - female/male	---	57	---	---	---	---	57

Table 3a.

Classifier network performance: Testing cohort – unbalanced data

	Prediction			Performance			Error
True class (label)	Class 1	Class 2	Class 3	P	R	F1	Type I	Type II
Class 1 (control)	32	1	0	0.7273	0.9697	0.8312	0.0380	0.0032
Class 2 (SLE standard)	10	229	15	0.9957	0.9016	0.9463	0.0032	0.0791
Class 3 (SLE experimental)	2	0	27	0.6429	0.9310	0.7606	0.0475	0.0063

Table 3b.

Generator network performance: Testing cohort – unbalanced data

	Prediction			Performance			Error
True class (label)	Class 1	Class 2	Class 3	P	R	F1	Type I	Type II
Class 1 (control)	32	1	0	0.7619	0.9697	0.8533	0.0316	0.0032
Class 2 (SLE standard)	9	228	17	0.9956	0.8976	0.9441	0.0032	0.0823
Class 3 (SLE experimental)	1	0	28	0.6222	0.9655	0.7568	0.0538	0.0032