A deep-learning-based unsupervised model on esophageal manometry using variational autoencoder

Abstract

High-resolution manometry (HRM) is the primary method for diagnosing esophageal motility disorders and its interpretation and classification are based on variables (features) from data of each swallow. Modeling and learning the semantics directly from raw swallow data could not only help automate the feature extraction, but also alleviate the bias from pre-defined features. With more than 32-thousand raw swallow data, a generative model using the approach of variational auto-encoder (VAE) was developed, which, to our knowledge, is the first deep-learning-based unsupervised model on raw esophageal manometry data. The VAE model was reformulated to include different types of loss motivated by domain knowledge and tuned with different hyper-parameters. Training of the VAE model was found sensitive on the learning rate and hence the evidence lower bound objective (ELBO) was further scaled by the data dimension. Case studies showed that the dimensionality of latent space have a big impact on the learned semantics. In particular, cases with 4-dimensional latent variables were found to encode various physiologically meaningful contraction patterns, including strength, propagation pattern as well as sphincter relaxation. Cases with so-called hybrid L2 loss seemed to better capture the coherence of contraction/relaxation transition. Discriminating capability was further evaluated using simple linear discriminative analysis (LDA) on predicting swallow type and swallow pressurization, which yields clustering patterns consistent with clinical impression. The current work on modeling and understanding swallow-level data will guide the development of study-level models for automatic diagnosis as the next stage.

Graphical Abstract

1. Introduction

High-resolution Manometry (HRM) is the primary method for the clinical evaluation of esophageal motility and the diagnosis of esophageal motility disorders [1, 2, 3]. The approach for clinical interpretation of HRM studies is guided by the Chicago Classification (CC), which categorizes each HRM study into one of 10 classes (i.e. esophageal motility diagnoses) based on a decision-tree-like algorithm [1]. A typical HRM study is generated from a clinical procedure that involves a patient completing multiple swallows. From data of individual swallow, referred to here as swallow data, around three to four outcomes are computed based on pre-defined algorithms and manual landmark identification. Those swallow-level outcomes could be considered as swallow-level features and firstly applied to a labeling scheme to derive a classification of each individual swallow (i.e. swallow type). Then, the composite (median) values among the swallows that constitute the HRM study and proportions of swallow types are applied to the hierarchical decision tree of the CC to generate an esophageal motility diagnosis on the study.

Two issues may arise from this diagnosis platform. First, due to reliance on pre-defined outcomes, the CC algorithm is sensitive to noise/outliers that may lead to misclassification. For example, integrated relaxation pressure (IRP), as one key outcome/feature to quantify the behavior of lower esophageal sphincter (LES), computes the sum of relaxed pressure along user-picked LES channels. The CC applies the IRP such that a low IRP is associated with normal swallow, whereas an elevated IRP leads to abnormal classifications of disorders, such as so-called esophageal-gastric junction outflow obstruction (EGJOO). However the parameter of the IRP could be sensitive to wall-catheter contact or catheter movement (i.e. pressure artifacts) or esophageal shortening. Second, the feature extraction/calculation relies on expert experience, which could lead to inter-rater disagreement. For example, the manual identification of LES channel in computing IRP not only requires the knowledge of esophageal sphincter function, but also could become complicated/problematic in cases where hiatal hernia and/or catheter movement occurs. Even metric-related parameters generated by the analytic software are reliant on manual review and positioning of anatomic and physiologic landmarks. Thus interpretation of HRM is subject to inter-rater variation and rater associated inaccuracy, which is impacted by rater experience and expertise [4, 5, 6]. Reliance on subjective experience also poses a challenge in medical training. A recent post-training evaluation survey from GI fellows showed only 84 percent were confident in the interpretation of esophageal manometry [7].

However, machine learning could provide a viable solution to alleviate issues associated with HRM interpretation. In particular, deep learning models developed with large set of raw manometry data could learn the distinctive patterns that separate various clinical phenotypes of esophageal motility disorders. Those learned patterns could further be synthesized or encoded as new outcomes/features that potentially generalize better than pre-defined features based on limited datasets. Moreover, the trained model could also automate the pipeline from raw manometry data to final diagnosis, and therefore acts as an artificial intelligence (AI) assistant to facilitate clinical practice and medical training.

AI (or machine learning) has already found wide applications in various fields of medicine [8, 9, 10], including gastroenterology [11]. Based on several review reports [11, 12, 13], the primary focus of AI in gastroenterology is generally related to interpretation of endoscopic images, such as detection of intestinal malignancies or premalignant lesion or evaluation of inflammatory bowel disease and gastrointestinal bleeding (see [11]). Regarding esophageal manometry, however, there seems to be no machine learning model to predict type of swallow and esophageal motility diagnosis directly from raw manometry data. These deep learning model (as to differentiate them from feature-based machine learning models) may be lacking from esophageal manometry interpreter, due to lack of large cleaned manometry dataset to counter-against the curse of dimensionality [14]. Thus, based on a newly created dataset with 32-thousand manometry swallows, this study aimed to develop an unsupervised deep learning model using variational auto-encoder (VAE). VAE, as a generative approach to approximate high-dimensional data distribution, has been successfully used in many applications including image analysis [15, 16], abnormality detection, and semi-supervised learning [17]. The VAE model developed in this work is conjectured to not only represent the distinctive patterns of various types of swallows, but also could act as features extractors for building study-level models for automatic diagnosis.

The presentation of this work is organized as follow. After the brief introduction of HRM procedure and dataset, the VAE formulation is revisited and modified to include additional type of loss motivated by the esophageal functioning. Then the training of the VAE model are discussed. Cases with different dimensionality of latent space and specific type of loss are compared. The model’s generative and discriminative capabilities are evaluated and discussed.

2. Materials and Methods

2.1. Overview of High-resolution Manometry data

After a minimum 6-hour fast, HRM studies were completed using a 4.2-mm outer diameter solid-state assembly with 36 circumferential pressure sensors at 1-cm intervals that samples at 100-Hz frequency (Medtronic Inc, Shoreview, MN). The HRM assembly was placed transnasally and positioned to record from the hypopharynx to the stomach with approximately three intragastric pressure sensors. After a 2-minute baseline recording, the HRM protocol was performed with ten, 5-ml liquid swallows in a supine position. Additionally, five, 5-ml liquid swallows were performed in the upright, seated position. During each swallow, pressure variations reflecting physiology of esophageal function were recorded, which is referred here as swallow data. HRM swallows and studies were interpreted according to CC v3.0 [1], in which swallow-level features were applied to derive swallow types, which were applied at the study-level to assign the esophageal motility classification (study label). Study and swallow-level labels were assigned prospectively into patients’ medical records during the course of standard clinical interpretation by a single interpreter with expertise in HRM interpretation (JEP). The classifications among the study cohort were retrospectively reviewed by a group of experienced HRM interpreters (DAC, ED, AJB) to verify and confirm classifications; questions were settled via group review and consensus assignment (JEP, DAC, ED, AJB).

An illustration of information flow is shown in Figure. 1, which indicates the swallow-level modeling and learning is a building block for study-level prediction. Based on the approach outlined by the CC, the pressure data within a 12 to 20-second window starting from swallow onset, was characterized independently to obtain various swallow-level features. The swallow types were assigned as Normal, Weak, Failed, Fragmented, Premature or Hypercontractile, and the pressurization types included Normal (NP), Compartmental pressurization (CP) and Panesophageal pressurization (PEP). Finally, 2,161 HRM studies, consisting of 32,415 labeled swallows, were included. This labeled 32,415 swallow data was used to develop the VAE model described in this work. Next, the whole dataset was split into three datasets: training, validation and test dataset. The train-validation-test split, were stratified to 70-15-15 splits at the study level, so that the individual swallows within a single study fell into the same dataset, avoiding information leaking. Table. 1 lists the data distribution with respect to both study-level category and swallow-level categories. It can be seen that the data distribution of swallow type and swallow pressurization was similar across training, validation and test datasets.

Figure 1:

Illustration of high-resolution manometry procedure and associated information flow from raw data to final diagnosis in current clinical practices

Table 1:

Sample distribution with respect to study-level categories (the Chicago Classification) and two swallow-level categories (swallow type and swallow pressurizaton).Note for category, Chicago Classification, listed are the number of studies in each dataset for the corresponding label. Also, Label IEM includes the studies labeled as Fragmented Peristalsis (FRP) in original Chicago Classification, and Label T3A includes those labeled as Distal Esophageal spasm (DES) originally [1]. For Categories, swallow type and swallow pressurizaton, listed are the number of swallows in each dataset.

Category	label	label details	train #	valid. #	test #
Chicago	ABC	Absent Contractility	65	14	15
classification	T1A	Type1 Achalasia	48	10	11
	T2A	Type2 Achalasia	93	19	20
	T3A	Type3 Achalasia	70	14	16
	EGJOO	EGJ outflow obstruction	238	51	52
	JES	Jackhammer esophagus	33	7	8
	NEM	Normal motility	722	154	155
	IEM	Ineffective esophageal motility	242	51	53
swallow type	N	Normal	12636	2719	2869
	W	Weak	2752	575	521
	F	Failed	5340	1074	1194
	FR	Fragmented	694	188	122
	P	Premature	641	121	119
	H	Hypercontraction	602	123	125
swallow	NP	Normal pressurization	19544	4119	4302
pressurization	CP	Compartmental pressurization	1892	404	306
	PEP	Panesophageal pressurization	1229	277	342

2.2. data augmentation and rebalance

Table. 1 also highlights the issue of data imbalance. Imbalanced data distribution could bias the generator to focus more on the majority class. Therefore, data rebalance was conducted as part of data augmentation, which will be discussed later. Another part of data augmentation is about down-sampling of raw data to alleviate the issue from the curse of dimensionality. In particular, we set the time duration for each swallow as 24 second and down-sample the data from 100Hz to 10Hz. This choice of time window is based on the following reasons. First, a baseline data occurred before swallow onset is needed to inform the resting state of esophageal body and EGJ, for which 4 seconds of data seems sufficient. Second, duration of a typical swallow lasts about 12–16 seconds from our experience [6]. Finally, an additional 4–8 second post-swallow data with redundancy permits time rolling as one way of data augmentation. The temporal down-sampling from 100Hz to 10Hz is motivated by two reasons. First, to our knowledge, all of the relevant physiological signals measurable by manometry have a frequency below 10Hz. These signals include respiration, pulse due to heartbeat, muscle contraction-relaxation wave, and gastric slow wave. Hence down-sampling to 10Hz should lead to minimal loss of information associated with patterns of esophageal swallow. Second, 10Hz is the sampling frequency of another data source from a comparable procedure, Functional lumen imaging probe (FLIP) [18]. FLIP, similar to manometry, manifests the dynamics of esophageal functions based on readings of pressure and impedance sensors, but with a sampling frequency as 10Hz. Adopting 10Hz in our current model may help the future development of multi-module models that incorporate those two data sources. After the down-sampling, the dimensionality of swallow data shrinks from 2400 × 36 to 240 × 36. To boost the counts of minority classes, we designed data augmentation by artificially rolling data along the channel-wise axis and time-wise axis, respectively. This is illustrated in Figure. 2. Note that the whole procedure of data augmentation is done on the fly by a user-defined data generator built on top of the keras library [19].

Figure 2:

Exploration of swallow data showing the variation of pressure pattern by swallow type (Upper) and by swallow pressurization (Lower left). Lower right shows the data augmentation by temporal rolling (i.e. along x-direction) and channel-wise rolling (i.e. along y-direction)

2.3. generative model: variational auto-encoder

Generative modeling is a broad area of machine learning that aims to model the distribution P(X), defined over datapoint X = X_n in an n-dimensional space $\mathcal{X}$. For the current HRM swallow data, datapoint X is the raw pressure recordings from n_y = 36 channels, each of which records n_x samples in time. The dimensionality, n = n_y × n_x = 36 × 240, as we adopted 240 samples in time. A straightforward kind of generative models that directly compute P(X) could suffer from the curse of dimensionality. Techniques related to dimension reduction and/or compressive sensing, such as principle component analysis, independent component analysis, auto-encoder, and variational auto-encoders (VAE) are often used [14]. In particular, VAE proposes to approximate an optimal P(X) through maximizing a tractable lower bound, and offers structured latent space for data generation. Due to these features, VAE is adopted in this work. Details of VAE methodology have been well studied and reported in literature [14, 20]. Hence we only discuss several key ingredients in this section for completeness. We begin by introducing the latent space as below.

$$P(X)=\int P(X|Z)P(Z)dZ$$ (1)

$$P(X|Z)=R\left(X|d\left(Z;{\theta }_d\right)\right);$$ (2)

where, Z = Z_r is the latent variables in a r-dimensional space $\mathcal{Z}$, which is a typically low-dimension space. A priori density function of latent space, P(Z) is typically assumed as r independent Gaussian with zero mean and unit variance. d(Z; θ_d) denotes a decoder model (or generator) with parameters θ_d to be learned. The decoder model, depending on the needed capacity, could be any directed graphical model, including the neural network that is adopted here. It signifies how the information flows in an expanded fashion from low-dimensional latent space to high-dimensional data space. The R(·) is a density function to facilitate the quantification of reconstruction loss between X and outputs of decoder, $\overline{X}=d\left(Z;{\theta }_d\right)$. For example, for n-dimensional continuous data X, one could choose R(·) = N(·,σ × I), n independent Gaussian distribution with constant variance σ, yielding logR(·) as L2 loss (plus a constant). For n-dimension binary data X, one could choose $R\left(X|\overline{X}=d\left(Z;{\theta }_d\right)\right)=e^{{\sum }_{i=1}^n(X_i\mathit{\text{log}}\overline{X_i}+\left(1-X_i\right)\mathit{\text{log}}\left(1-\overline{X_i}\right)}$, with logR(·) as the cross-entropy loss.

Another key ingredient for VAE is the evidence lower bound objective (ELBO), denoted as $\mathcal{L}$

$$\begin{gathered} \mathcal{L}=E_{Q(Z|X)}[\mathit{\text{log}}P(X,Z)-\mathit{\text{log}}Q(Z|X)] \\ =E_{Q(Z|X)}[\mathit{\text{log}}P(X)+\mathit{\text{log}}P(X|Z)-\mathit{\text{log}}Q(Z|X)] \\ =\mathit{\text{log}}P(X)-D_{KL}(Q(Z|X)\Vert P(Z,X)) \\ ⩽\mathit{\text{log}}P(X), \end{gathered}$$ (3)

where, Q(Z|X) is a density function to model the posterior of latent variables Z, discussed later. D_KL(Q(Z|X)∥P(Z, X)) is non-negative Kullback–Leibler (KL) divergence. Evidently, Eq. 3 shows one can maximize logP(X) by maximizing ELBO, $\mathcal{L}$, which can be further written as a sum of expected log likelihood and the KL divergence between the prior P(Z) and Q(Z|X),

$$\begin{gathered} \mathcal{L}=E_{Q(Z|X)}[\mathit{\text{log}}P(X|Z)+\mathit{\text{log}}P(Z)-\mathit{\text{log}}Q(Z|X)] \\ =E_{Q(Z|X)}[\mathit{\text{log}}P(X|Z)]-D_{KL}(Q(Z|X)\Vert P(Z)). \end{gathered}$$ (4)

Examining Eq. 4 gives the intuition on how Q(Z|X) is encouraged during the optimization of ELBO. The first term is an expected likelihood that moves the density mass on configurations of the latent variables that explain the observed data, whereas the second term encourages the density function Q(Z|X) to be close to the prior P(Z). The balance between the likelihood and prior in the ELBO allows us to bias one against the other by incorporating a weight scalar, λ. This was referred to as the modified ELBO, $\overline{\mathcal{L}}$

$$\overline{\mathcal{L}}=E_{Q(Z|X)}[\mathit{\text{log}}P(X|Z)]-\lambda D_{KL}(Q(Z|X)\Vert P(Z)$$ (5)

Now we come to discuss the posterior Q(Z|X). Q(Z|X) is another directed probabilistic graph, pointing from the high-dimensional data space, $X\in \mathcal{X}$ toward a low-dimensional latent space $Z\in \mathcal{Z}$. Similar to P(X|Z). Q(Z|X) typically incorporates a neural network model, referred to as the encoder, to capture the information flow. Thus we have

$$Q(Z|X)=S\left(Z|e\left(X;{\theta }_e\right)\right)$$ (6)

$$\begin{gathered} \overline{\mathcal{L}}=\overline{\mathcal{L}}\left(X;{\theta }_e,{\theta }_d\right) \\ =E_{S\left(Z|e\left(X;{\theta }_e\right)\right)}[\mathit{\text{log}}R\left(X|d\left(Z;{\theta }_d\right)\right]-\lambda D_{KL}\left(S\left(Z|e\left(X;{\theta }_e\right)\right)\Vert P(Z)\right]. \end{gathered}$$ (7)

In Eq. 6, e(X; θ_e) is a deterministic encoder mode (a neural net here), with θ_e as parameters to be learned. S(·) is a density function that depends on the latent space $\mathcal{Z}$. For r-dimensional continuous latent variables used in this work, S(·) is taken as r independent Gaussian, whose mean and variance are outputs of the encoder neural network e(X; θ_e). Hence, the objective function $\overline{\mathcal{L}}\left(X;{\theta }_e,{\theta }_d\right)$ identifies two sets of parameters, θ_e in the encoder and θ_d in the decoder, to be optimized. The optimization with respect to θ_d appears to be straight forward, with the gradient computed as below,

$${\nabla }_{{\theta }_d}\overline{\mathcal{L}}\left(X;{\theta }_e,{\theta }_d\right)=E_{S\left(Z|e\left(X;{\theta }_e\right)\right)}\left[{\nabla }_{{\theta }_d}\mathit{\text{log}}R\left(X|d\left(Z;{\theta }_d\right)\right]\right].$$ (8)

However, the optimization with respect to θ_e is not obvious, as apparently ${\nabla }_{{\theta }_e}{E}_{S\left(Z|e\left(X;{\theta }_e\right)\right)}[\mathit{\text{log}}R\left(X|d\left(Z;{\theta }_d\right)\right]\ne E_{S\left(Z|e\left(X;{\theta }_e\right)\right)}[{\nabla }_{{\theta }_e}\mathit{\text{log}}R\left(X|d\left(Z;{\theta }_d\right)\right]$. This motivates the introduction of the re-parameterization trick that moves the θ_e out of the expectation evaluation, shown as below.

$$ϵ~p(ϵ)$$ (9)

$$Z=g\left(ϵ,e\left(X;{\theta }_e\right)\right)=g(ϵ,e)$$ (10)

$$E_{Z~S(Z|e)}[\mathit{\text{log}}R\left(X|d\left(Z;{\theta }_d\right)\right]=E_{ϵ~p(ϵ)}[\mathit{\text{log}}R\left(X|d\left(g(ϵ,e);{\theta }_d\right)\right].$$ (11)

As illustrated above, we introduce a new random variable, denoted as ϵ with density function as p(ϵ), to generate random variable Z through a deterministic mapping g(ϵ, e). Then the expectation is transferred from Z to ϵ, as in Eq. (11). Hence, the gradient with respect to θ_e in the objective could be conveniently evaluated as below,

$${\nabla }_{{\theta }_e}\overline{\mathcal{L}}\left(X;{\theta }_e,{\theta }_d\right)=E_{ϵ~p(ϵ)}[{\nabla }_{{\theta }_e}\mathit{\text{log}}R\left(X|d\left(g\left(ϵ,e;{\theta }_e\right);{\theta }_d\right)\right]-\lambda {\nabla }_{{\theta }_e}{D}_{KL}\left(S\left(Z|e;{\theta }_e\right)\Vert P(Z)\right)].$$ (12)

With both P(Z) and p(ϵ) specified as r independent Gaussian with zero mean and unit variance, and $g(ϵ,e)={\left({\mu }_i+{\sqrt{\sigma }}_i\times {ϵ}_i\right)}_{i=1}^r$, we obtain Z ~ S(Z|e; θ_e) as joint distribution of r independent Gaussian with mean μ_i and variance σ_i for i ∈ 2 (1, 2, …, r). So the KL divergence term in Eq. 12 can be evaluated analytically as,

$$D_{KL}\left(S\left(Z|e;{\theta }_e\right)\Vert P(Z)\right]=\frac{1}{2}\sum _i^r\left({\sigma }_i+{\mu }_i^2-\mathit{\text{log}}\left({\sigma }_i\right)-1\right).$$ (13)

By contrast, the expectation term in Eqs. 5 and 12, is evaluated by single-point Monte-Carlo sampling, denoted as ϵ^I, when we feed into the model a data point X^I from a dataset. Thus, we have

$$Z^I=g\left({ϵ}^I,e;{\theta }_e\right)={\left({\mu }_i+{\sqrt{\sigma }}_i\times {ϵ}_i^I\right)}_{i=1}^r$$ (14)

$$\begin{gathered} E_{ϵ~p(ϵ)}\left[\mathit{\text{log}}R\left(X^I|d\left(g\left(ϵ,e;{\theta }_e\right);{\theta }_d\right)\right)\right]=\mathit{\text{log}}R\left(X^I|d\left(g\left({ϵ}^I,e;{\theta }_e\right);{\theta }_d\right)\right) \\ =\mathit{\text{log}}R\left(X^I|d\left(Z^I;{\theta }_d\right)\right). \end{gathered}$$ (15)

Note that, both mean μ_i and variance σ_i, as output of our encoder model, will be functions of model parameters θ_e.

logR(·), as discussed before, defines the reconstruction loss. Here, we introduce a hybrid reconstruction loss to facilitate comparison among different types of loss through hyper-parameter tuning. The hybrid reconstruction loss associated with one single data point X^I reads,

$$\mathit{\text{log}}R\left(X^I|\overline{X^I}=d\left(Z^I;{\theta }_d\right)\right)={\gamma }_1\left(\sum _{i=1}^n\left(X_i^I\mathit{\text{log}}\overline{X_i^I}+\left(1-X_i^I\right)\mathit{\text{log}}\left(1-\overline{X_i^I}\right)\right)+{\gamma }_2(\sum _i^n{\left(X_i^I-{\overline{X}}_i^I\right)}^2+{\gamma }_2^t\sum _i^n{\left(d_t{X}_i^I-d_t{\overline{X}}_i^I\right)}^2\right),$$ (16)

where, the first term represents the cross-entropy loss, the second term represents the L2 loss. The third term, referred to as hybrid L2 loss, is motivated by the temporal contraction-relaxation pattern in manometry, where d_tX_i = X_t+1,y − X_t,y is the temporal difference of pressure values along each channel y. This term acts to be a regularizer that encourages the generator to capture the temporal discontinuity of esophageal pressure.

Now, all the terms of the ELBO associated with one single data point X^I can be summarized as below.

$${\overline{\mathcal{L}}}^I\left(X^I;{\theta }_e,{\theta }_d\right)=\mathit{\text{log}}R\left(X^I|{\overline{X}}^I\right)]-\frac{1}{2}\lambda \left(\sum _i^r\left({\sigma }_i+{\mu }_i^2-\mathit{\text{log}}\left({\sigma }_i\right)-1\right)\right)$$ (17)

$$\mathit{\text{log}}R\left(X^I|\overline{X^I}\right)={\gamma }_1\left(\sum _{i=1}^n\left(X_i^I\mathit{\text{log}}\overline{X_i^I}+\left(1-X_i^I\right)\mathit{\text{log}}\left(1-\overline{X_i^I}\right)\right)+{\gamma }_2(\sum _i^n{\left(X_i^I-{\overline{X}}_i^I\right)}^2+{\gamma }_2^t\sum _i^n{\left(d_t{X}_i^I-d_t{\overline{X}}_i^I\right)}^2\right)$$ (18)

$$\overline{X^I}=d\left(Z^I;{\theta }_d\right)$$ (19)

$$Z^I={\left({\mu }_i+{\sqrt{\sigma }}_i\times {ϵ}_i^I\right)}_{i=1}^r$$ (20)

$${\left({\mu }_i,\mathit{\text{log}}\left({\sigma }_i\right)\right)}_{i=1}^r=e\left(X^I;{\theta }_e\right)$$ (21)

$${ϵ}_i^I~N(0,1),$$ (22)

where, λ is a hyper parameter to balance the reconstruction loss and KL divergence. γ₁, γ₂, ${\gamma }_2^t$ are hyper-parameters to tune the specific form of reconstruction loss. d(Z^I; θ_d) and e(X^I, θ_e) are decoder and encoder, respectively. They are modeled as convolutional neural networks. The total ELBO associated with a minibatch of size m is just a simple summation, $\overline{\mathcal{L}}={\sum }_{I=1}^m{\overline{\mathcal{L}}}^I$. Optimization on the stochastic gradient descent could be appliedLby computing gradients with respect to θ_d and θ_e, as shown in Eqs. 8 respectively.

2.4. Other details on model setup

As mentioned above, the encoder and decoder models in this work were implemented as a convolutional neural network (CNN), respectively. The encoder acts like information compressor composed by multiple convolution, pooling filters and fully-connected layers. The decoder, also referred to as generator, attempts to reconstruct the source data fed to the encode. Note that, the reconstructed data X needs to satisfy, X^I ∈ [0, 1], in order to be compatible with the cross-entropy loss. Therefore, before feeding it to the encoder, we scale pressure data as, (X + 10)/160 → X and then clip it at (0, 1). We found the scaled data were also beneficial even for cases with L2 loss and helped to mitigate the issue of gradient exploding. With the scaled data, the last layer of decoder adopted sigmoid activation for output, where a low value indicates a low pressure, a high value indicates a high pressure. In the context of binary distribution implied in the cross-entropy loss, the output of sigmoid activation layer could be interpreted as the probability of high pressure. Both the cross-entropy loss and L2 loss attains minimum when each dimension of data is exactly reconstructed, but the slope of former becomes more gradual when the minimum is away from both 0 and 1.

Regarding the implementation, the model was developed based on several libraries including tensorflow [21] and keras [19]. The training was performed on Northwestern Quest, a cluster to provide super computing services.

3. Results and Discussions

3.1. Overview: encoder and decoder models

Through extensive experimentation, we obtained the architecture of the encoder and decoder neural networks that seemed to perform well. Specific layers of each neural network are listed in Table. 2 and Table. 3, respectively. The same architectures were used for all case studies, except the adjustment of the shape of encoder output and decoder input for different dimensionality of Z_r. The training process was found very sensitive to the step size. However, scaling of the ELBO as $\mathcal{L}/n\to \mathcal{L}$, n = 36 × 240, seemed to alleviate the issue exploding gradient. During training, we also adopted early stopping as one regularizer and reduced learning rate once validation loss was plateaued. The initial learning rate was set as, lr = 1.0e⁻³. Unlike the learning rate, the weight parameter λ was found to have little impact, and thus was simply chosen to be 1.0e⁻⁵ × n, n = 36 × 240. Each case study included an initial run, followed by 5 restart runs using previously trained weights with the lowest validation loss. Each run lasted four hours unless early stopping was triggered. An example of training history is illustrated in Fig. 3. Evidently, the validation loss and training loss was plateaued before the onset of second restart run (i.e. the 2nd red triangle in the figure). For the last three restarts, the initial phase was accompanied with a short spike of loss increase, due to reset of learning rate (i.e. lr = 1.0e⁻³). This also seems to indicate that the training is sensitive to the learning rate.

Table 2:

Layers in the encoder neural net for cases with a 4-dimensional latent space. The first dimension of Output shape, labeled as None, corresponds to the size of mini-batch. Details of layer type could be found from Keras documentation [19]

Layer type	Output Shape
InputLayer	(None, 36, 240, 1)
Conv2D	(None, 34, 79, 8)
MaxPooling	(None, 16, 39, 8)
Conv2D	(None, 14, 12, 16)
MaxPooling	(None, 13, 10, 16)
Conv2D	(None, 11, 8, 32)
MaxPooling	(None, 5, 4, 32)
Conv2D	(None, 3, 2, 64)
MaxPooling	(None, 1, 1, 64)
global average pooling2d	(None, 64)
Dense	(None, 128)
Dense	(None, 8)

Table 3:

Layers in the decoder neural net for cases with a 4-dimensional latent space. The first dimension of Output shape, labeled as None, corresponds to the size of mini-batch. Details of layer type could be found from Keras documentation [19]

Layer type	Output Shape
InputLayer	(None, 4)
Dense	(None, 128)
Reshape	(None, 1, 1, 128)
UpSampling2D	(None, 1, 1, 128)
UpSampling2D	(None, 2, 2, 128)
Conv2DTranspose	(None, 4, 4, 64)
UpSampling2D	(None, 8, 8, 64)
Conv2DTranspose	(None, 10, 10, 32)
UpSampling2D	(None, 10, 20, 32)
Conv2DTranspose	(None, 12, 41, 16)
UpSampling2D	(None, 12, 82, 16)
Conv2DTranspose	(None, 37, 247, 8)
Conv2D	(None, 36, 240, 1)

Figure 3:

Training history of Case 2 listed in Table. 4. Line curves: loss, val_loss, lr represent the history of training loss, validation loss, and learning rate, respectively. Same to all the other cases, training of Case 2 includes a fresh run followed by 5 restart runs, where the onset of the nth restart run is indicated by #n red triangle.

3.2. Impact of dimensionality of the latent space: r

The performance of VAE model would likely depend on the dimensionality of the latent space, which determines the narrowness of the information bottleneck arising from the latent space Z that bridges encoder and decoder models. Cases with r = (2, 4, 6, 8) were studied, as listed in Table. 4. For performance evaluation, the trained VAE model was used to first encode and then decode the training/test/validate dataset to compute the reconstruction loss, also listed in Table. 4. For each case, the loss of 3 datasets are roughly the same, indicating that no apparent overfitting occurred. By contrast, the loss decreases with the dimensionality, since larger dimension provides a larger modeling space to explore a better loss-less reconstruction.

Table 4:

Cases with di erent dimensionality, r, of the latent space. For all cases, the reconstruction loss was chosen as L2 loss (see Eq. (16) for details). The train/validation/test loss was computed using trained encoder and decoder network on corresponding dataset, and then averaged over both sample and dimension of data space X.

Case #	r	train loss	validation loss	test loss
Case 1	2	1.186e-3	1.181e-3	1.208e-3
Case 2	4	1.017e-3	1.016e-3	1.045e-3
Case 3	6	9.245e-4	9.275e-4	9.515e-4
Case 4	8	8.514e-4	8.612e-4	8.784e-4

The learned semantics could be illustrated by the generated images from sampling in the latent space, as shown in Figure. 4. Case 1 with even 2 latent variables, (z1, z2), clearly shows disentanglement. The high-pressure band, indicating the muscle contraction, diminished diagonally from corner: (left,upper)=(−2,2) to corner: (right,lower)=(2,−2). z1 = x seemingly encoded the spasm pattern, whereas z2 = y the contraction strength. Case 2 with 4 latent variables, (z1, z2, z3, z4), needs projections onto 2D images for illustration. For each projection, 2 latent variables were chosen as (x, y) and varied within (−2, 2), with other latent variables fixed at the origin. Compared with Case 1, Case 2 showed a richer disentanglement. The upper-middle panel of Figure. 4 suggested z1 may encode the transition from normal contraction pattern to concurrent contraction observed in PEP (see Figure. 2). By contrast, z4 focused more on the transition from normal contraction to non-relaxing body contraction in cases of Hyper-contraction (Figure. 2). z2, z3 encoded the relaxation pattern of lower esophageal sphincter (LES), and contraction strength, respectively. Case 3 with a 6-dimensional latent space adopted similar projections, as shown in the lower panel of Figure. 4. Evidently, with more latent variables, Case 3 is also capable to disentangle weak vs. strong contraction, normal vs. concurrent contractions. However, Case 3 also showed patterns that are physiologically implausible. For example, the lower-middle panel of Figure. 4 shows higher contraction in LES region during swallow and/or post-swallow period, encoded by z2, z5. But the esophageal physiology suggests LES should typically show a higher tone during pre-swallow period, and a lower or diminished tone during swallow period to allow the passage of food bolus. Even associated with disorders, the during-swallow and post-swallow LES tone is rarely higher than pre-swallow tone. This might suggest that Case 3 could be over-encoding, i.e. over-exaggerating/extrapolating information from the dataset. Similar over-encoding phenomenon were also observed in Case 4 with 8-dimensional latent space. Hence, Case 2 with 4-dimensional latent space appeared ideal for further study in the following sections.

Figure 4:

Illustration of generated swallow data for Cases 1, 2, and 3, listed in Table. 4. For high dimensional latent space, various projection onto 2D (x-y)-plane is applied with un-projected dimensions set at the origin.

3.3. Impact of reconstruction loss type

Besides the dimensionality of the latent space, we conjecture the specific form of loss (i.e. reconstruction loss, in particular) could also impact the VAE model. This is because the loss type influences how the optimization procedure pushes the model parameters to represent (i.e. reconstruct) the raw data X by $\overline{X}$. Based on studies on dimensionality of latent space from the last section, we choose dimensionality of latent space, r = 4, and varies the specific reconstruction form as shown in Eq. (16). Cases are listed in Table. 5, which also includes computed reconstruction loss on train/validation/test dataset using the trained encoder and decoder models. Across the train/validation/test dataset, the reconstruction loss are almost the same, indicating no apparent overfitting. Among different cases, however, the magnitude varies greatly. Compared with Cases with L2 loss, cases with hybrid loss showed slight increase of reconstruction loss due to an additional term in Eq. (16). Case 7 with cross-entropy loss showed the highest level of reconstruction loss, probably because the optimization tries to encode/decode only the overall level of high and low pressure (or contraction). Interestingly, if we adopt L2 loss for comparison, all cases seemed to fall into the same level, as shown in the last column of Table. 5. This might imply that the magnitude of original loss may not be always a reliable metric in comparing model performance without accounting for the specific loss type. A better comparison could be made by investigating the generated images from sampling in the latent space. Generated images on Case 5, 6, and 7 are shown in Figure. 5, which utilized the same projection approach as in Figure. 4 for illustration. Compared with cases with L2 and hybrid-L2 loss, Case 7 with the cross-entropy loss seems to over-emphasize on (repeatedly) encoding spasm contraction, evidenced by two regions (z1 ~ −2, 0, 0, z4 ~ 2) and (0, z2 ~ 2, z3 ~ 2, 0) (see the two left panels). By contrast, Case 5 and Case 6 with hybrid-loss shows a larger spectrum that includes both contraction’s strength and propagation pattern. Compared with Case 2 with L2 loss, Cases 5 and 6 seem to capture a better coherence of transition between contraction/relaxation(i.e. transition between high and low pressure bands). Cases 5 and 6, however, does not reveal significant visual difference, after we compare the projected images from all the 6 pairs of latent variables (not shown here). Therefore, we identify Case 5 as the optimal case for next section of study.

Table 5:

Cases with different type of reconstruction loss. The reconstruction loss is dictated by γ₁, γ₂, ${\gamma }_2^t$, weights of the cross-entropy loss, L2 loss, and hybrid L2 loss, respectively, as in Eq. (16). The dimensionality of latent space, r, for all cases was chosen as 4. The train/validation/test loss was computed using trained encoder and decoder network on corresponding dataset, and then averaged over both sample number and dimension in the data space X.

Case #	(γ1, γ2, ${\gamma }_2^t$)	train loss	validation loss	test loss	test L2-loss
Case 2	(0,1,0)	1.017e-3	1.016e-3	1.045e-3	1.045e-3
Case 5	(0,1,0.5)	1.558e-3	1.544e-3	1.598e-3	1.088e-3
Case 6	(0,1,5)	5.833e-3	5.833e-3	6.001e-3	1.054e-3
Case 7	(1,0,0)	3.772e-1	3.787e-1	3.766e-1	1.059e-3

Figure 5:

Illustration of generated swallow data for Cases 5, 6, and 7, listed in Table. 5. For high dimensional latent space, various projection onto 2D (x-y)-plane is applied with un-projected dimensions set at the origin.

3.4. Evaluation of discriminative capability : linear discriminant analysis

The above analysis evaluated the capability of generator on pattern generation and disentanglement. Now we move to the evaluation on the model’s discriminative capability. In particular, we choose the VAE model of Case 5 for this study and use the linear discriminant analysis (LDA) for evaluation. The LDA serves as both a simple classification model for prediction and a supervised dimension-reduction model for 2D illustration. The dataset for developing LDA models are (Z, y), where Z = (z1, z2, z3, z4) are 4-dimensional latent variables and y are labels from either 6-class swallow type or 3-class swallow pressurization (see Table. 1). For each category, The LDA model was trained using the same training dataset as our VAE model and then evaluated on the same validation and test dataset as VAE. The specific implementation is based on machine learning library, scikit-learn, in which two LDA components were adopted for dimensionality reduction. Therefore, the trained LDA model also served as a transformer to map Z = (z1, z2, z3, z4) onto 2-dimensional LDA space, L = (L1, L2) for illustration. Figure. 6 shows the mapped LDA space of test dataset for predicting the 6-class swallow type. The overall accuracy for train/validation/test dataset was 0.64/0.63/0.64. For each class, the class mean, and the class standard deviation of L1(L2) components are denoted as the center, and twice of x(y) radius of the corresponding ellipse (See the upper panel). Three clusters seem to emerge. The first cluster could be the Hypercontraction located on the far right. The second cluster likely includes Premature, Fragmented and widely spread Normal groups, located near upper right. The third cluster consists of Failed and Weak groups, which are both associated with diminished peristalsis. These three emerged clusters are actually consistent with clinical impression. Figure. 7 shows the mapped LDA space of test dataset for predicting the 3-class swallow pressurization. The overall accuracy for train/validation/test dataset was 0.87/0.86/0.87. Note that metrics based on accuracy may be biased because of sample imbalance. But the separation, illustrated by the three corresponding ellipses on class mean and class standard deviation, seemed to be reasonable good.

Figure 6:

Mapped LDA space of test dataset for predicting the 6-class swallow type using the 4-dimensional latent variables from Case 5. The ellipse associated with each class is centered at the class mean, with x(y) radius equal to half of the class standard deviation of L1(L2) component.

Figure 7:

Mapped LDA space of test dataset for predicting the 3-class swallow pressurization using the 4-dimensional latent variables from Case 5. The ellipse associated with each class is centered at the class mean, with x(y)-radius equal to half of the class standard deviation of L1(L2) components.

To better appreciate the class separation with the LDA model, we also provided the mapped LDA space of the training dataset as interactive plots. See supplemental data for details.

Unsupervised dimension-reduction technique was also adopted to examine the clustering. Due to simplicity, the principle-component analysis (PCA) with n-components=2 was conducted using 4-dimensional latent variables. Results on both training dataset and testing dataset showed an impressive pattern separation on swallow types (see supplemental interactive plots). In particular, a clear separation was found between Hyper-contractile (H) swallows with Failed (F)/Weak (W) swallows, with widely spread Normal (N) swallows.

4. Conclusion

Current clinical diagnosis algorithms, like Chicago Classification(CC) for Manometry, are largely built on a group of experts’ experience. The generated algorithm, however, could suffer from bias due to subjective experiences and lack of generalized capability for new data, which may disagree with previous impression/knowledge built on limited dataset. Those two issues, from the machine learning’s perspective, could be alleviated by increasing the model’s complexity (to deal with bias) and size of training data (for model generalization). For example, the current CC algorithm essentially assumes as a classification tree based on manually extracted features. Both the assumption on the existence of an optimal classification tree and a handful pre-defined features could be relaxed, for diagnosis algorithm to be determined by a more capable machine learning model, such as multi-layer neural networks. Our current work on learning and understanding swallow-level raw data is one step forward along this route. Moreover, as the first deep-learning-based unsupervised model on raw manometry data, the current VAE model could serves as a benchmark for future related tasks.

The current VAE model also offered several interesting findings. In terms of methodology, the current VAE model identifies the sensitivity of learning rate, which could be tuned with a modified ELBO scaled by the dimension of data space. Parametric studies illustrate the impact of dimensionality of the latent space and the specific form of reconstruction loss. In particular, a hybrid L2 loss with a physiology-motivated penalty term is found to capture a better coherence during contraction/relaxation transition. This could be one example that model tuning could be benefited from the guidance of the domain knowledge. In terms of application, the learned latent variables are able to disentangle various contraction patterns and encode physiologically meaningful scores on contraction strength and motility patterns. The LDA model derived from latent variables, though simplistic, shows clustering patterns that are consistent with clinical impressions. For downstream modeling with the latent variables, however, more capable model could be adopted to overcome the bias from LDA model and likely yield a better classification schemes. This could be pursued in a future work.

Although modeling and learning of the study-level data is the ultimate goal for deriving automatic diagnosis, modeling and learning its basic building block, swallow-level data, is useful for two reasons. First, a well-learned swallow-level model could be a high-level feature extractor for developing study-level models. This is especially the case for VAE model, which generates latent variables to automatically encode scores of various patterns of esophageal contractility and motility. Second, the extracted information and patterns from each swallow could serve as reasoning elements for checking/validating study-level predictions. This is particular useful when the study-level model itself acts to be a black-box model due to its complexity.

One limitation of the current work is that the developed VAE model is purely based on machine learning (ML) approaches, lacking the interpretation from bio-physical principles. Incorporation of bio-physical models into evidence-based ML models could offer more insights in interpretation and understanding of biological phenomena. This methodology has recently been adopted in study of esophageal physiology. For example, Carniel et al. [22] proposed a physiological model to investigate esophageal motility, which was later incorporated into an automatical procedure to identify esophageal diseases based on manometry data [23]. Developing integrated models with both bio-physical principles and evidenced-based ML models is an interesting direction to pursue. In conjunction with our previously developed bio-mechanical esophageal models [24, 25], the current VAE model could potentially be an important component during the future integration.

In conclusion, the current VAE model on swallow data will guide ongoing efforts on developing study-level classification models. It could also serve as an important component in developing an integrated model that incorporates both bio-physical principles and evidence-based principles.

Highlights.

• With 32-thousand swallows of esophageal manometry data, a first-of-its-kind deep-learning-based unsupervised model was developed using variational auto-encoder (VAE).

• The generative capability on disentangling semantics was evaluated and compared among models trained with different latent space and reconstruction loss.

• Various physiologically-meaningful motility patterns including contraction strength and propagation trend as well as lower sphincter relaxation was found to be well-encoded by 4-dimensional latent variables.

• The discriminating capability was also examined based on linear discriminative analysis (LDA) trained with latent variables, which revealed clustering patterns that are consistent with clinical impression.

• Modeling and understanding of swallow-level data will facilitate the development of study-level models for artificial-intelligence-based automatic diagnosis of esophageal manometry.