IGAMT: Privacy-Preserving Electronic Health Record Synthesization with Heterogeneity and Irregularity

Abstract

Utilizing electronic health records (EHR) for machine learning-driven clinical research has great potential to enhance outcome predictions and treatment personalization. Nonetheless, due to privacy and security concerns, the secondary use of EHR data is regulated, constraining researchers’ access to EHR data. Generating synthetic EHR data with deep learning methods is a viable and promising approach to mitigate privacy concerns, offering not only a supplementary resource for downstream applications but also sidestepping the privacy risks associated with real patient data. While prior efforts have concentrated on EHR data synthesis, significant challenges persist: addressing the heterogeneity of features including temporal and non-temporal features, structurally missing values, and irregularity of the temporal measures, and ensuring rigorous privacy of the real data used for model training. Existing works in this domain only focused on solving one or two aforementioned challenges. In this work, we propose IGAMT, an innovative framework to generate privacy-preserved synthetic EHR data that not only maintains high quality with heterogeneous features, missing values, and irregular measures but also achieves differential privacy with enhanced privacy-utility trade-off. Extensive experiments prove that IGAMT significantly outperforms baseline and state-of-the-art models in terms of resemblance to real data and performance of downstream applications. Ablation studies also prove the effectiveness of the techniques applied in IGAMT.

1. Introduction

The availability of electronic health records (EHR) not only improves patient care but also boosts the advancement of medical research. However, due to privacy and security concerns, secondary use of EHR data for research purposes is always regulated, thus constraining researchers’ access to EHR data (Choi et al. 2017).

A practical and promising solution to mitigate the privacy concern is to generate synthetic EHR data that are realistic for machine learning tasks, offering not only a supplementary resource for downstream applications but also avoiding the privacy risks associated with real patient data. To achieve this goal, synthetic EHR data need to retain the sophisticated characteristics of the real data because these attributes can substantially impact the usage of the synthetic data in the downstream tasks. The specific characteristics of EHR data are shown in Figure 1 and summarized below:

Figure 1:

Illustration of EHR raw data

1. Heterogeneity of features: Each record has both temporal and non-temporal features. Some features are time-related (blue blocks) such as heart rate, which will be recorded at each visit. For temporal features, each record can be viewed as a matrix consisting of multiple visits (time steps) and each visit contains multi-dimensional features (Shickel et al. 2017). Some features are non-temporal (not related to time) such as demographic features including gender and race (green blocks).

2. Missing values: EHRs may contain structurally missing data that correspond to specific clinical scenarios. That is, certain events or measurements are intentionally omitted or not recorded by clinicians. For example, additional tests (e.g., glucose levels) will not be measured (i.e., become missing values) if the patient’s vital signs (e.g., blood pressure) are normal during a clinical visit. The illustration of such missing value is represented as “/” in Figure 1) (Bang, Wang, and Yang 2020).

3. Irregularity of features: The temporal features may be measured in different frequencies, for instance, some features are measured on an hourly time scale while others are on a monthly time scale (Shickel et al. 2017; Bang, Wang, and Yang 2020).

Capturing these sophisticated characteristics of heterogeneous features, missing values, and irregular measures poses challenges to deep learning models. Besides the challenge in representation learning of these characteristics, another challenge lies in crafting synthetic EHR data that retains these characteristics. Most of the existing works focused on isolated aspects of these characteristics, resulting in synthesized EHR data that cannot fulfill the downstream requirements. For instance, some works (Neil, Pfeiffer, and Liu 2016) only concentrated on the representation learning of irregular measures while disregarding the impact of missing values, which can lead to completely opposite diagnosis. The detailed related works about EHR representation learning and synthesizing are discussed in Section 2.

Privacy leakage is another major challenge for models built on sensitive data like EHR. Synthetic data is typically generated by a deep generative model trained on real data, therefore when the model and synthetic data are published, the original data can be still inferred and incurs privacy leakage (Rahman et al. 2018). To prevent this issue, differential privacy (DP), a formal mathematical privacy-preserving framework, is widely applied in the model training stage (Beaulieu-Jones et al. 2019; Lee et al. 2020). One limitation of the state-of-the-art DP techniques, like gradient perturbation (Abadi et al. 2016), is that they can undermine the utility of the model because of the randomization introduced in the model. Therefore, how to mitigate utility degradation and balance the trade-off between utility and privacy is a major challenge. Existing works on EHR data synthesization can neither maintain all the special characteristics nor provide a formal privacy guarantee to the training data (Choi et al. 2017; Beaulieu-Jones et al. 2019; Lee et al. 2020; Baowaly et al. 2019; Chin-Cheong, Sutter, and Vogt 2020; ?).

Contributions.

In this work, we propose the Imitative Generative Adversarial Mixed-embedding Transformer (IGAMT) to generate differentially private EHR with sophisticated characteristics. As shown in Figure 3a, the architecture of IGAMT contains three generative adversarial networks (GANs) (Goodfellow et al. 2014) and an autoencoder (Hinton and Salakhutdinov 2006). IGAMT leverages transformer (Vaswani et al. 2017) to capture both temporal and non-temporal features. In addition, we utilize masks and time embedding to capture missing values and irregular measures and combine sequence-to-sequence autoencoder with transformer and GAN to better maintain the sophisticated characteristics. We further adopt a new structure, Imitator, to reduce the randomization required by the DP technique while keeping the complex architecture for enhanced privacy and utility trade-off.

Figure 3:

Model architecture of IGAMT training and synthesization.

IGAMT is the first framework to generate differentially private EHR data of high quality with heterogeneous features, missing values, and irregular measures. Our key contributions are listed as follows:

1. We propose an EHR data generative model that not only maintains the specific characteristics of EHR but also provides a differential privacy (DP) guarantee.

2. We leverage sequence-to-sequence transformer with missing value masks, time embedding, and non-temporal embedding in our generative model to learn the sophisticated characteristics of EHR and generate synthetic data.

3. We incorporate a novel Imitator in our architecture to imitate the behaviors of the decoder. Applying gradient perturbation to the Imitator rather than the decoder itself improves the model utility (quality of the synthetic EHR) while preserving the same level of DP.

4. Extensive experiments on real-world EHR data demonstrate that IGAMT significantly outperforms baseline and state-of-the-art models in terms of resemblance of the synthetic data to real data and performance of downstream applications and achieves enhanced privacy utility trade-off.

2. Related Work

In this section, we briefly introduce the existing work on EHR data representation learning and synthesization, and the differential privacy techniques, especially the applications in generative models.

EHR representation learning.

Several works focused on representation learning of EHR data by building specific neural networks to capture these characteristics. Neil, Pfeiffer, and Liu (2016) proposed a novel recurrent network, Phased-LSTM, to capture irregular measures of temporal data, and Bang, Wang, and Yang (2020) further improved Phased-LSTM to fit missing values and irregular measures.

EHR data synthesization.

For EHR synthesization, Choi et al. (2017) proposed medGAN to generate multi-label discrete records. However, medGAN only works on discrete features and does not address the potential privacy leakage. Hyland, Esteban, and Rätsch (2018) proposed recurrent conditional GAN (RCGAN), which can generate temporal medical features. However, RCGAN does not take non-temporal features, missing values and privacy protection into consideration. Xu et al. (2019) built CTGAN for tabular medical data, but cannot be directly applied to EHR data. Baowaly et al. (2019) introduced medWGAN and medBGAN on top of medGAN by replacing GAN with more powerful variants, WGAN (Arjovsky, Chintala, and Bottou 2017; Gulrajani et al. 2017) and boundary-seeking GAN (BGAN) (Hjelm et al. 2017). However, they did not take temporal features and privacy preservation into consideration.

Differential privacy.

Differential Privacy (DP) (Dwork 2011; Dwork et al. 2006; Dwork, Roth et al. 2014) is a theoretical privacy framework for aggregate data analysis, which ensures the output of a randomized algorithm is indistinguishable between two neighboring datasets that differ in one record (or bounded by a distance metric) with a certain probability. Gradient perturbation is a common practice to achieve DP for deep learning models by injecting perturbation into the gradient of each parameter (Song, Chaudhuri, and Sarwate 2013; Bassily, Smith, and Thakurta 2014; Abadi et al. 2016; Wang, Ye, and Xu 2017; Lee and Kifer 2018; Yu et al. 2019; Wang et al. 2021).

Privacy-preserving generative model for EHR.

To obtain a privacy-preserving generative model for EHR data, Beaulieu (Beaulieu-Jones et al. 2019) applied DP into the training process of the discriminator of AC-GAN (Odena, Olah, and Shlens 2017). However, this work does not take the temporal features and missing values into consideration. Chin-Cheong, Sutter, and Vogt (2020) proposed a DP-GAN to generate heterogeneous EHRs with non-temporal features and missing values. However, temporal features are still missed in this work. Lee et al. (2020) proposed a dual adversarial autoencoder (DAAE) to generate temporal EHR and employ DP during training to prevent privacy leakage. DAAE is the existing state-of-the-art generative model with DP for EHR, but is incapable of capturing non-temporal features, missing values, and irregular measures. We will use DAAE as the baseline comparison to demonstrate the effectiveness of IGAMT.

3. Preliminaries

Differential Privacy

Differential Privacy (DP) ensures that the output of a randomized algorithm is indistinguishable between two neighboring datasets that differ in one record (or bounded by a distance metric) with a certain probability.

Definition 1.

$(ϵ,\delta )$-Differential Privacy) A randomized mechanism $ℳ:\mathbf{D}\to \mathbf{R}$ with domain $\mathbf{D}$ and range $\mathbf{R}$ satisfies $(ϵ,\delta )$-differential privacy if for any two adjacent input datasets $𝒟,{𝒟}^{\mathrm{\text{'}}}\in \mathbf{D}$ and for any subset of outputs $\mathbf{S}\subseteq \mathbf{R}$ it holds that

$$\mathit{\text{Pr}}\left(ℳ\left(𝒟\right)\in \mathbf{S}\right)\le e^{ϵ}\mathit{\text{Pr}}\left(ℳ\left({𝒟}^{\prime }\right)\in \mathbf{S}\right)+\delta ,$$

where $ϵ$ denotes the privacy level (or privacy budget) and $\delta$ denotes the probability that the inequality breaks.

The lower the $ϵ$, the stronger the privacy. The common approach to achieving $(ϵ,\delta )$-DP is the Gaussian mechanism that adds calibrated Gaussian noise to the output.

Gradient perturbation.

The most commonly used approach to achieve differential privacy in deep learning systems is gradient perturbation. It injects calibrated noise into the gradient during training with the following objective function and gradient update.

$$𝒥\left({\theta }_t\right)=\frac{1}{N}\sum _{i=1}^{N}l\left(x_i,y_i,{\theta }_t\right),{\theta }_{t+1}={\theta }_t-\eta \left(\nabla 𝒥\left({\theta }_t\right)+p\right)$$

where ${\theta }_t$ denotes the parameter at training step $t,\nabla 𝒥\left({\theta }_t\right)$ denotes the gradient which is bounded by a clipping norm or constrained by Lipschitz continuity of loss function $l$, and $\mathbf{p}$ denotes the gradient perturbation typically as a Gaussian noise $𝒩\left(0,{\sigma }^2\right)$.

Moment accountant is commonly used to quantify the overall privacy cost from multiple iterations of the entire training.

Theorem 1.

Moment Accountant (Abadi et al. 2016). Let $ℱ:{\mathbb{R}}^v\to {\mathbb{R}}^w$ be an $w$-dimensional model, and its sensitivity ${\mathrm{\Delta }}_{ℱ}={\text{max}}_{𝒟,{𝒟}^{\mathrm{\text{'}}}}{‖ℱ\left(𝒟\right)-ℱ\left({𝒟}^{\mathrm{\text{'}}}\right)‖}_2$. Given training batch size $B$, the total training size $N$, the number of training steps $T$, gradient perturbation with Gaussian noise $𝒩\left(0,{\sigma }^2\right)$, there exists constants $c_1$ and $c_2$ for any $ϵ<c_1{\left(\frac{B}{N}\right)}^{2}T,ℱ$ is $(ϵ,\delta )$-DP for any $\delta >0$, if we choose

$$\sigma \ge c_2\frac{B/N\sqrt{\mathit{\text{Tlog}}(1/\delta )}}{ϵ}.$$

Dual Adversarial Autoencoder (DAAE)

The architecture Dual Adversarial Autoencoder (DAAE) (Lee et al. 2020) combines a recurrent autoencoder with two generative adversarial networks (GANs) (Goodfellow et al. 2014). Two discriminators in GAN can not only distinguish the central hidden state in the autoencoder but also distinguish real data from reconstructed data and synthetic data.

4. IGAMT

In this section, we will first present the architecture of IGAMT and demonstrate how IGAMT solves three challenges: feature representation learning, synthetic EHR generation, and privacy preservation. Then we will introduce the training process of IGAMT.

Representation Learning

IGAMT incorporate sequence-to-sequence autoencoder (seq2seq AE) to capture the sophisticated characteristics from data. Transformer (Vaswani et al. 2017) is used to implement both encoder and decoder, which uses self-attention to capture the correlation among features at different time steps. We improve Transformer by incorporating several well-designed techniques to learn sophisticated feature representations of EHR.

Non-temporal features.

To simultaneously learn temporal and non-temporal feature representation and capture the connection between these features, non-temporal features are transformed to a vector of the same size as temporal features at each time point which are denoted as the start feature, as shown on the left of Figure 2a. In addition, to better learn the non-temporal feature representations, we also transform gender and race into embedding vectors respectively, and broadcast them to all time steps before applying them to hidden states, as shown on the right of Figure 2a.

Figure 2:

Model representation learning.

Missing values.

The missing data in EHR have structural patterns and correspond to specific clinical scenarios instead of missing at random. Synthetic EHR data directly generated from deep learning models cannot learn these structural missing values. The models will generate continuous values for each feature and timestep, thus the characteristics of missing values in the real EHR data will be lost in the synthetic EHR. To overcome this challenge and better capture missing values, we create a mask consisting of 1s and 0s to mark the element-wise missing value positions as shown in Figure 2b. Seq2seq AE of IGAMT will not only generate synthetic data but also its corresponding synthetic mask. In this way, element-wise multiplication of data and mask will generate the final synthetic EHR data which maintains the missing value characteristics of the real EHR.

Irregular measures.

To better capture irregular measures, time steps are extracted from EHRs by calculating the increment of two neighboring time steps and adding 0 as the initial increment. Then we transform the time features into embedding vectors as shown in Figure 2c. These embeddings are then applied to the hidden states during training.

Loss function for representation learning.

Seq2seq Transformer AE with specific embeddings in IGAMT is leveraged to capture the characteristics related to heterogeneous features, missing values, and irregular measures. This autoencoder takes $x$ and its mask $m$ as the input and generates synthetic data $\hat{x}$ and mask $\hat{m}$, the reconstruction loss is the cross-entropy loss ($𝒞ℰ$):

$${ℒ}_{\mathit{\text{rec}}}=𝒞ℰ\left(x,\widehat{x}\right)+𝒞ℰ\left(m,\widehat{m}\right),$$ (1)

where $\mathbf{x}$ denotes element-wise multiplication of $x$ and $m$, and parameters will be optimized accordingly.

Architectures

Training framework.

As shown in Figure 3a, the architecture of IGAMT has three modules. First, as explained in the previous section, a seq2seq AE with transformer blocks ($ℰ\mathit{\text{nc}}$ and $𝒟\mathit{\text{ec}}$) is implemented to learn the sophisticated feature representation. This module serves as the generator and, together with the discriminator ${𝒟}_x$, constitutes the first GAN $\left(\mathit{\text{GA}}{N}_1\right)$. The goal of the discriminator ${𝒟}_x$ is to discriminate between the real data and missing value mask ($x$ and $m$) and the fake ones generated by the seq2seq AE. This GAN is the main part of the IGAMT architecture to generate synthetic EHR.

Second, to improve the generative ability of IGAMT, we incorporate another GAN $\left(\mathit{\text{GA}}{N}_2\right)$ formed by generator $𝒢$ with $ℰ\mathit{\text{nc}}$ and discriminator ${𝒟}_z$ as discussed in DAAE. The goal of the discriminator ${𝒟}_z$ is to discriminate between “real” hidden states $z$ from encoder $ℰ\mathit{\text{nc}}$ and “fake” states $\tilde{z}$ from the generator $𝒢$. This module is to improve the model coverage rate and quality of generated sequences by adversarially learning both the continuous latent distribution ($z$ from encoder $ℰ\mathit{\text{nc}}$ and $\tilde{z}$ from generator $𝒢$) and the data distribution.

Third, the Imitator $𝒟e{c}_i$ together with the generator $𝒢$ and the discriminator ${𝒟}_x$ constitutes $\mathit{\text{GA}}{N}_3$. The Imitator is incorporated to support differential privacy (DP). Directly applying the DP technique to IGAMT without the imitator will bring overwhelmingly large noise to the training process. This occurs because perturbations are required for both generator $𝒟\mathit{\text{ec}}$ and discriminator ${𝒟}_x$ as both parts access the real data (${𝒟}_x$ access real data from the forward pass while $𝒟\mathit{\text{ec}}$ from back-propagation). This process will ultimately compromise the model utility and the quality of the synthetic EHRs. We explain below how the imitator is utilized to support DP and analyze the DP in more detail later.

DP guarantee.

To reduce the DP randomization and maintain the model utility, we introduce a novel module Imitator with the same structure as $𝒟\mathit{\text{ec}}$ to mimic the behavior of the decoder $𝒟\mathit{\text{ec}}$. Compared with $𝒟ec$, the Imitator $𝒟e{c}_i$ does not access real data (because it uses $\tilde{z}$ from the generator $𝒢$), thus only adding gradient perturbation to the discriminator ${𝒟}_x$ can ensure DP for ${𝒟\mathit{\text{ec}}}_i$ (post-processing theorem of DP). Similarly, $𝒢$ does not access real data, thus only adding gradient perturbation to the discriminator ${𝒟}_z$ can ensure DP for $𝒢.𝒢$ and $𝒟e{c}_i$ can be then used to generate synthetic data with DP. Note that the architecture of generators in GAN is always much more complicated than the discriminators which require more gradient perturbation to achieve the same level of DP guarantee. Therefore, incorporating Imitator can significantly reduce the DP randomization and improve the model utility. In practice, to better guide the Imitator to mimic $𝒟\mathit{\text{ec}}$, we let these two structures share the same last layer during training ($𝒟\mathit{\text{ec}}$ and $𝒟{\mathit{\text{ec}}}_i$ have the same architecture) and also utilize an imitation loss for the imitator. We will analyze the DP in detail in the following sections.

Algorithm 1: IGAMT algorithm
Input: preprocessed training EHRs $x$ and masks $m$, total training epoch $T$, gradient perturbation scale $\sigma$, learning rate $\eta$, batch size $B$, discriminators update frequency base $f_b$ and frequency hit $f_h$, gradient clipping norm $C$
1 $t=0$;
2 initialize parameters of $\mathit{\text{IGAMT}}$;
3 while $t<T$ do
4 get mini-batch EHRs $x_{(t)}$ and masks $m_{(t)}$;
5 $z_{(t)}=ℰnc\left(x_{(t)},m_{(t)}\right)$;
6 ${\widehat{x}}_{(t)},{\widehat{m}}_{(t)}=Dec\left(x_{(t)},m_{(t)},z_{(t)}\right)$;
7 ${\tilde{z}}_{(t)\mid }=𝒢(B)$ (generate synthetic hidden states);
8 sample start features $s_f$ and craft start masks $s_m$;
9 ${\tilde{x}}_{(t)},{\tilde{m}}_{(t)}=Dec\left(s_f,s_m,{\tilde{z}}_{(t)}\right)$;
10 ${\overline{x}}_{(t)},{\overline{m}}_{(t)}=𝒟e{c}_i\left(s_f,s_m,{\tilde{z}}_{(t)}\right)$;
11 if $t\text{\%}{f}_b<f_h$ then
// Update ${𝒟}_x$ with DP perturbation
12  ${ℒ}_{{𝒟}_x}=d_{{𝒟}_x}(x,\widehat{x})+d_{{𝒟}_x}(x,\tilde{x})+d_{{𝒟}_x}(x,\overline{x})$;
13  $\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_x}=\frac{1}{B}{\nabla }_{{\theta }_{(t)}^{{𝒟}_x}}{ℒ}_{{𝒟}_x}$;
14  $\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_x}=\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_x}/\mathit{\text{max}}\left(1,\Vert \mathit{\text{gra}}{d}_{(t)}^{{𝒟}_x}\Vert /C\right)$;
15  ${\theta }_{(t+1)}^{{𝒟}_x}={\theta }_{(t)}^{{𝒟}_x}-\eta \left(\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_x}+𝒩\left(0,{\sigma }^2\right)\right)$;
// Update ${𝒟}_z$ with DP perturbation
16  ${ℒ}_{{𝒟}_z}=d_{{𝒟}_z}(\tilde{z},z)$;
17  $\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_z}=\frac{1}{B}{\nabla }_{{\theta }_{(t)}^{{𝒟}_z}}{ℒ}_{{𝒟}_z}$;
18  $\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_z}=\mathit{\text{gra}}{d}_{(t)}^{{𝒟}_z}/max\left(1,\Vert \mathit{\text{gra}}{d}_{(t)}^{{𝒟}_z}\Vert /C\right)$;
19  ${\theta }_{(t+1)}^{𝒟z}={\theta }_{(t)}^{𝒟z}-\eta \left(\mathit{\text{gra}}{d}_{(t)}^{𝒟z}+𝒩\left(0,{\sigma }^2\right)\right)$;
20 end
21 ${\theta }^{\mathit{\text{last}}}$: parameters of the shared last layer between $𝒟ec$ and $𝒟e{c}_i$; // Update $ℰnc$
22 ${ℒ}_{ℰnc}={𝒟}_z(z)+{ℒ}_{\mathit{\text{rec}}}$;
23 ${\theta }_{(t+1)}^{\epsilon nc}={\theta }_{(t)}^{ℰnc}-\eta \frac{1}{B}{\nabla }_{{\theta }_{(t)}^{ℰnc}}{ℒ}_{ℰnc}$;
// Update $𝒟ec$ excluding the last layer
24 ${ℒ}_{𝒟ec}=-{𝒟}_x(\widehat{x})-{𝒟}_x(\tilde{x})+{ℒ}_{\mathit{\text{rec}}}$;
25 ${\theta }_{(t+1)}^{Dec}={\theta }_{(t)}^{Dec}-\eta \frac{1}{B}{\nabla }_{{\theta }_{(t)}^{Dec}}{ℒ}_{𝒟ec}$;
// Update $𝒟e{c}_i$′s last layer with gradient perturbation
26 ${ℒ}_{\mathit{\text{im}}}=\text{MSE}(\widehat{\text{x}},\overline{\text{x}})+\text{MSE}(\tilde{\text{x}},\overline{\text{x}})+\text{MSE}(\text{x},\overline{\text{x}})$;
27 ${ℒ}_{𝒟e{c}_i}=-{𝒟}_x(\overline{\text{x}})+{ℒ}_{\mathit{\text{im}}}$;
28 ${\mathit{\text{grad}}}_{(t)}^{\mathit{\text{last}}}=\frac{1}{B}{\nabla }_{{\theta }_{(t)}^{\mathit{\text{last}}}}{ℒ}_{𝒟e{c}_i}$;
29 ${\widehat{\mathit{\text{gra}}d}}_{(t)}^{\mathit{\text{last}}}=\mathit{\text{gra}}{d}_{(t)}^{\mathit{\text{last}}}/\mathit{\text{max}}\left(1,\Vert \mathit{\text{gra}}{d}_{(t)}^{\mathit{\text{last}}}\Vert /C\right)+𝒩\left(0,{\sigma }^2\right)$;
30 ${\theta }_{(t)}^{\mathit{\text{last}}}={\theta }_{(t)}^{\mathit{\text{last}}}-\eta \left({\widehat{\mathit{\text{gra}}d}}_{(t)}^{\mathit{\text{last}}}\right)$;
// Update $De{c}_i$ excluding the last layer with chain rule
31 $\mathit{\text{gra}}{d}_{(t)}^{𝒟e{c}_i}={\widehat{\mathit{\text{gra}}d}}_{(t)}^{\mathit{\text{last}}}*{\nabla }_{{\theta }_{(t)}^{{𝒟ec}_i}}^{\mathit{\text{Output}}({\theta }_{(t)}^{𝒟e{c}_i})}$;
32 ${\theta }_{(t+1)}^{𝒟e{c}_i}={\theta }_{(t)}^{𝒟e{c}_i}-\eta \frac{1}{B}\mathit{\text{gra}}{d}_{(t)}^{𝒟e{c}_i}$;
// Update $𝒢$
33 ${ℒ}_{𝒢}=-{𝒟}_z(\tilde{z})$;
34 ${\theta }_{(t+1)}^{𝒢}={\theta }_{(t)}^{𝒢}-\eta \frac{1}{B}{\nabla }_{{\theta }_{(t)}^{𝒢}}{ℒ}_{𝒢}$;
35 end
Output: $𝒟e{c}_i$ and $𝒢$

Synthesization framework.

We explained the training architecture of the IGAMT in the above section. After training, we use the DP components of IGAMT to generate synthetic EHR, as shown in Figure 3b, which contains $G$ and ${𝒟\mathit{\text{ec}}}_i$ including the shared last layer of $𝒟\mathit{\text{ec}}$ and $𝒟e{c}_i$. The synthesization process can be divided into the following steps: 1) sampling random states from a Gaussian distribution, 2) $𝒢$ takes random states as the input and generates central hidden states $\tilde{z},3$) the Imitator ${𝒟\mathit{\text{ec}}}_i$ takes $\tilde{z}$ as input and generates data and masks, and 4) assemble the generated data and masks to form the synthetic EHR.

The synthetic EHRs generated from IGAMT retain the heterogeneous features, missing values, and irregular measures. Moreover, because the generative model is differentially private, these synthetic EHRs are correspondingly privacy-preserved.

Loss Functions and Optimization

In this section, we will first elaborate on each loss function designed to solve each challenge. Then we will present our optimization process and training algorithm (Algorithm 1).

Discriminator

${𝒟}_x.𝒟\mathit{\text{ec}}$ and $𝒟e{c}_i$ are generators in two GANs respectively, sharing the same discriminator ${𝒟}_x$. The loss for ${𝒟}_x$ consists of the discrimination loss between each synthetic data generated from $𝒟\mathit{\text{ec}}$ and ${𝒟\mathit{\text{ec}}}_i$ and the real data, which can be stated as:

$${ℒ}_{{𝒟}_x}=d_{{𝒟}_x}\left(x,\widehat{x}\right)+d_{{𝒟}_x}\left(x,\tilde{x}\right)+d_{{𝒟}_x}\left(x,\overline{x}\right)$$ (2)

where $\mathbf{x}$ denotes element-wise multiplication of $x$ and $m,\stackrel{\mathbf{ˆ}}{\mathbf{x}}$, $\tilde{\mathbf{x}}$ and $\overline{\mathbf{x}}$ denotes the generator outputs of the three GANs respectively (illustrated in Figure 3a), and $d_{{𝒟}_x}(u,v)={𝒟}_x\left(v\right)-{𝒟}_x\left(u\right)$. The updates of ${𝒟}_x$ are using gradient perturbation (Algorithm 1 lines 12–15) to ensure ${𝒟}_x$ is DP.

Generator $𝒟ec$ and Imitator $𝒟e{c}_{\mathit{i}}$.

The Imitator ${𝒟\mathit{\text{ec}}}_i$ and $𝒟\mathit{\text{ec}}$ share the same last layer during training. $Dec$ excluding the last layer is optimized through the back-propagation of associated discrimination loss and the reconstruction loss. The discriminator tries to minimize the discrimination loss while the generator tries to maximize it:

$${ℒ}_{𝒟ec}=-{𝒟}_x\left(\widehat{x}\right)-{𝒟}_x\left(\widehat{x}\right)+{ℒ}_{\mathit{\text{rec}}}$$ (3)

where ${ℒ}_{\mathit{\text{rec}}}$ refers to Equation1. $Dec$ excluding the last layer is updated without gradient perturbation (line 24–25). We note that $𝒟\mathit{\text{ec}}$ except the shared last layer is not DP since the back propagation uses the real data to compute the gradient for updating those layers.

The loss for $𝒟e{c}_i$ consists of two parts, the imitation loss and the associated discrimination loss. The goal is to generate $\bar{x}$ and $\bar{m}$ that is close to both real data and the other two sources of synthetic data generated by $𝒟\mathit{\text{ec}}$. The loss in optimizing $𝒟e{c}_i$ can be stated as:

$${ℒ}_{\mathit{\text{im}}}=\text{MSE}\left(\widehat{x},\overline{x}\right)+\text{MSE}\left(\tilde{x},\overline{x}\right)+\text{MSE}\left(x,\overline{x}\right)$$ (4)

$${ℒ}_{𝒟e{c}_i}=-{𝒟}_x\left(\overline{x}\right)+{ℒ}_{im}$$ (5)

To update the parameters of $𝒟e{c}_i$, we first update the last layer with gradient perturbation (line 28–30). This ensures the last layer is DP. Then, we update the remaining layers of ${𝒟\mathit{\text{ec}}}_i$ with the chain rule (line 31–32). As the gradient of the last layer is DP, and the gradient computation for the remaining layers does not use real data ($𝒟e{c}_i$ is based on $\tilde{z}$ from generator $𝒢$), so the remaining layers also ensure DP. Hence, the entire Imitator $𝒟e{c}_i$ including the shared last layer is DP.

Intuitively, while the Imitator ${𝒟\mathit{\text{ec}}}_i$ mimics $𝒟\mathit{\text{ec}}$, this mimicry is limited to a controlled extent, by making both discriminator ${𝒟}_x$ and the shared last layer DP. Consequently, the Imitator will not memorize the training data the same way as Dec.

Encoder $ℰ\mathit{n}\mathit{c}$, generator $𝒢$, and discriminator ${𝒟}_{\mathit{z}}$.

The GAN to improve the generative ability of IGAMT consists of the encoder $ℰ\mathit{\text{nc}}$, the generator $𝒢$ and discriminator ${𝒟}_z$, where $ℰ\mathit{\text{nc}}$ provides “real” hidden states $z,𝒢$ synthesizes “fake” states $\tilde{z}$, and ${𝒟}_z$ aims to distinguish $z$ from $\tilde{z}$. The loss for the encoder consists of two parts, the reconstruction loss ${ℒ}_{\mathit{\text{rec}}}$ and the loss from discriminator ${𝒟}_z$, which can be stated as:

$${ℒ}_{ℰnc}={𝒟}_z\left(z\right)+{ℒ}_{\mathit{\text{rec}}}$$ (6)

The loss for discriminator ${𝒟}_z$ and generator $𝒢$ are:

$${ℒ}_{{𝒟}_z}=d_{{𝒟}_z}\left(\tilde{z},z\right)$$ (7)

$${ℒ}_{𝒢}=-{𝒟}_z\left(\tilde{z}\right)$$ (8)

${𝒟}_z$ is updated with gradient perturbation (line 16–19) to ensure DP. The updates of $ℰ\mathit{\text{nc}}$ and $𝒢$ are denoted in line 22–23 and 33–34 respectively. Since $𝒢$ is updated with the discrimination loss of ${𝒟}_z$ which is DP, $𝒢$ also ensures DP.

Complete algorithm.

Algorithm 1 shows the complete training process of IGAMT. At the start of training, $ℰ\mathit{\text{nc}}$ and $𝒟\mathit{\text{ec}}$ together with $𝒢$ and ${𝒟\mathit{\text{ec}}}_i$ reconstructs and synthesizes ${\bar{x}}_{\left(t\right)},{\bar{m}}_{\left(t\right)},{\hat{x}}_{\left(t\right)},{\hat{m}}_{\left(t\right)}$ and ${\tilde{x}}_{\left(t\right)},{\tilde{m}}_{\left(t\right)}$ (lines 4 – 10). Lines 11 to 34 show the optimization of each component in IGAMT, which can be divided into two stages: updating discriminators (line 11 – 20) and updating generators (line 21 – 34). The discriminators are updated less frequently than the generators at the ratio of $f_h/f_b$ (line 11). To guarantee DP, gradient perturbations are applied when updating discriminators ${𝒟}_x$ (line 13–15), ${𝒟}_z$ (line 17–19) and the last layer of $𝒟\mathit{\text{ec}}$ (line 28–30).

DP Analysis

As mentioned before, once the model is trained, we are only releasing $𝒢$ and ${𝒟\mathit{\text{ec}}}_i$ for the synthesization. To guarantee DP of these two components, gradient perturbation is applied to the discriminators ${𝒟}_x,{𝒟}_z$, and the shared last layer ${𝒟\mathit{\text{ec}}}_i^{\mathit{\text{last}}}$ of $𝒟\mathit{\text{ec}}$ and $𝒟e{c}_i$. As ${𝒟}_x,{𝒟}_z$ and $𝒟e{c}_i^{\mathit{\text{last}}}$ are trained using the same dataset, the overall privacy can be analyzed under simple composition, and DP guarantee for each part is analyzed under moment accountant (Algorithm 1). Therefore, the total privacy of the final generative model ($𝒢$ and $\left.{𝒟\mathit{\text{ec}}}_i\right)$ is $\left({ϵ}_1+{ϵ}_2+{ϵ}_3,{\delta }_1+{\delta }_2+{\delta }_3\right)$-DP if ${𝒟}_x,{𝒟}_z$ and ${𝒟\mathit{\text{ec}}}^{\mathit{\text{last}}}$ are $\left({ϵ}_1,{\delta }_1\right)-\text{DP}$, $\left({ϵ}_2,{\delta }_2\right)-\text{DP}$ and $\left({ϵ}_3,{\delta }_3\right)-\text{DP}$ respectively.

5. Experiments

In this section, we demonstrate the effectiveness of IGAMT^* using synthetic EHRs from two aspects: visual similarity to real data and downstream applications with comparable performance to real data.

Experimental Setup

Baselines.

We compare IGAMT with DAAE, the existing state-of-the-art generative model for EHR data with DP. Since it is incapable of capturing non-temporal features, missing values, and irregular measures, we slightly adapt the it to conduct a fair comparison. We also build four more baselines: VAE (Variational Autoencoder), GAN, VAE-GAN (Larsen et al. 2016), and AAE (Makhzani et al. 2015) to have a more comprehensive comparison.

EHRs and data preprocessing.

We use two EHR datasets in this work. One is Physionet MIMIC-IV-ED (Goldberger et al. 2000), which is an open-sourced EHR dataset that encompasses over 425,000 ED stays collected from emergency department (ED) admissions from 2011 to 2019. In the paper, we utilize a subset that covers all vital sign data, which comprises 14,024 training, 1,753 validation, and 1,754 testing records. The other one is from the Emory Synergy project, which contains 5,747 training, 718 validation, and 719 testing records.

EHR data is preprocessed before feeding into the model. For temporal feature preprocessing, we first normalize them to the range of [0, 1]. Then for the irregular measures in the time-space (Section 4), we extract the time features and follow a similar process to scale them to [0, 1]. We also pad the time feature of all the examples to 50. For non-temporal feature preprocessing (Section 4), we similarly normalize them to [0, 1]. Then, we transform the discrete features into one-hot vectors to form the start features, which have the same size as the temporal features of each timestep.

The preprocessing of missing values (Section 4) is to generate a mask consisting of 1 and 0s where 0 represents the missing values. After preprocessing, each record has 50 time steps with each timestep having 10 and 9 features for MIMIC-IV-ED and Emory Synergy respectively.

Privacy budget.

The privacy budgets used in the experiments are $(ϵ,\delta )=(\mathrm{1.5,1}e-5)$. Our experiments currently use equal budget allocation among the three components.

Experimental Results

Evaluation 1. PCA visulization.

We use PCA to reduce the real and synthetic data to two-dimensional space and visually show the difference between real and synthetic EHR. PCA results aim to validate IGAMT’s ability to capture the feature distributions of real EHR by measuring subspace similarity. It reflects whether synthetic data maintains the underlying structure and correlations present in the real data. Figure 4 and Figure 5 demonstrate the results on MIMIC-IV-ED and Emory Synergy datasets. In both figures, the blue dots represent the real EHR, and the green dots represent the synthetic EHR from different models. The first row shows the non-DP results of the baseline model and IGAMT and the second row shows the results from the DP version of models corresponding to the first row. From the result, we can note that after dimension reduction, the synthetic data generated by IGAMT can fit the real data the best. The similarity in the principal components suggests that the subspace of the synthetic data closely aligns with that of the real data in terms of the inherent similarity and underlying structure, which indicates that IGAMT is well-designed for the synthesization of temporal EHR compared with the baseline architectures.

Figure 4:

PCA visualization of real and synthetic EHRs on MIMIC-IV-ED

Figure 5:

PCA visualization for real and synthetic on Emory Synergy

In addition, the DP technique applied during training can degrade the performance of baseline models. This trend is more notable when the architecture is more complex. However, for IGAMT, incorporating gradient perturbation does not compromise the model utility which verifies the effectiveness of the Imitator module. It overcomes the large randomization typically required for the generator and significantly enhances privacy utility trade-off.

Evaluation 2. Closer look at the feature similarity.

To provide a more detailed comparison of temporal features between real and synthetic EHRs, we pick three vital temporal features (“time in year”, “heart rate”, “SBP”), and randomly sample 100 EHRs from real test data and synthetic data, and plot the average value of the three selected features over 50 time steps. As shown in Figure 8, the blue curve represents the real EHRs, and black represents EHRs from IGAMT. For all three feature plots, black curves partially match the patterns of real features and outperform $\mathit{\text{DAAE}}$, which indicates that the synthetic temporal features generated from IGAMT better maintain the characteristics of real temporal features.

Figure 8:

Visualization of three vital features

We also compare the KL divergence of feature distributions between real and synthetic data generated by IGAMT and DAAE. As shown in Table 1, IGAMT dominates on almost all features, especially on features #2, #3, #5 #6, #8.

Table 1:

Feature similarity: KL divergence of feature distribution

Model	#0	#1	#2	#3	#4	#5	#6	#7	#8	#9
${\mathit{\text{DAAE}}}_{\mathit{\text{DP}}}$	0.2	25	24.9	33	11.01	15.18	46.8	48.33	32.7	45.02
${\mathit{\text{IGAMT}}}_{\mathit{\text{DP}}}$	0.2	20.78	7.88	3.37	11.09	0.75	0.04	34.06	2.98	33.87

To illustrate the statistics of missing values and irregular measures in EHRs, we count the mark-off positions per feature in masks and plot the histogram of counts averaged over features among 1000 samples, and calculate the elapsed time between two neighboring time steps and plot the histogram of the elapsed time averaged over time steps among 1000 samples. The results are shown in Figure 6 and Figure 7 respectively. As can be seen, while all baseline models fail, IGAMT is able to capture the distributions of missing values and elapsed time between visits that resemble the real data, thanks to its time embedding and the missing values masks.

Figure 6:

Feature similarity: missing values histogram of real and synthetic EHRs

Figure 7:

Feature similarity: elapsed time histograms of real and synthetic EHRs

Evaluation 3. Unsupervised downstream application: clustering.

To demonstrate that synthetic data generated by IGAMT are not only visually similar to the real data but also maintain the same characteristics of the real data for downstream tasks, we conduct unsupervised and supervised downstream applications. First, we use clustering as an unsupervised downstream application to show synthetic data and real data have similar clustering results. We use Minkowski distance and cosine similarity to measure the distance of clustering centers from synthetic data and real data. The lower Minkowski distance and higher cosine similarity represent better performance.

As shown in Table 2, IGAMT outperforms all other baseline models by achieving the smallest Minkowski distance and highest cosine similarity overall models. This indicates that the synthetic data generated by IGAMT can best maintain the clustering performance of the real data, which reflects the higher synthetic ability of IGAMT.

Table 2:

Downstream clustering performance

Model	Minkowski Distance	Cosine Similarity
${\mathit{\text{VAE}}}_{\mathit{\text{DP}}}$	5.36	−0.07
${\mathit{\text{GAN}}}_{\mathit{\text{DP}}}$	5.51	−0.19
${\mathit{\text{VAE\_GAN}}}_{\mathit{\text{DP}}}$	5.47	−0.16
${\mathit{\text{AAE}}}_{\mathit{\text{DP}}}$	5.22	0.20
${\mathit{\text{DAAE}}}_{\mathit{\text{DP}}}$	5.97	0.47
${\mathit{\text{IGAMT}}}_{\mathit{\text{DP}}}$	3.97	0.82

Evaluation 4. Supervised downstream application: classification.

Another downstream application involved is the classification task. We label the data by separating the “DBP” feature values into 4 categories. DBP stands for diastolic blood pressure, which measures the pressure in the arteries. The high DBP is an indicator of hypertension, thus we use different ranges of DBP values as labels to train a DBP classification model.

We train classifiers for real and synthetic data respectively using the same size of training data with the same training epochs under the same architecture. A CNN-LSTM (Sainath et al. 2015) model is adopted as shown in Figure 9. Initially, the data undergoes permutation and is fed into three layers of CNN. Then the data is reshaped and fed into a bidirectional LSTM with a hidden size of 128. As shown in Figure 10, after 1000 training epochs, the training loss and test accuracy for the real data converge to 0.866 and 80.50% respectively, while the synthetic data demonstrates a loss of 0.871 and an accuracy of 82.00%. These results indicate that the synthetic EHR by IGAMT can maintain the characteristics of the real EHR and achieve comparable downstream performance.

Figure 9:

Downstream Classifier Architecture

Figure 10:

Downstream classification performance

Evaluation 5. Privacy analysis.

The challenge of applying DP in deep learning systems is to balance the utility-privacy trade-off. Figure 11a and 11b demonstrate the overall Minkowski distance and cosine similarity of the downstream clustering results using models with different DP budget $ϵ$ and perturbation magnitude $\sigma$, where the black curve represents the IGAMT while other curves represent the baseline models. We can note that under the same privacy budget (the same $ϵ$ and $\sigma$), IGAMT can achieve the lowest Minkowski distance and highest cosine similarity, achieving the best synthetic performance.

Figure 11:

Minkowski distance and cosine similarity on different models.

6. Conclusions and Future Works

In this paper, we proposed a novel framework IGAMT to generate differentially private EHRs with heterogeneous features, missing values, and irregular measures. IGAMT leverages missing value masks and sequence-to-sequence transformers with well-designed embeddings to learn the underlying characteristics of EHRs and generate synthetic data of high quality. By leveraging the elaborate architecture and objective functions, the Imitator of IGAMT is capable of imitating the behaviors of the decoder while reducing the randomization required to achieve DP for the generator. After training with gradient perturbation, IGAMT will release $𝒢$ and ${𝒟\mathit{\text{ec}}}_i$ including the last shared layer with $𝒟\mathit{\text{ec}}$ as a DP generative model. We demonstrate that IGAMT achieves state-of-the-art performance in synthesizing DP EHRs.

Our experiments currently use equal privacy budget allocation among the three DP components. It can be further optimized for future work. For example, the gradient perturbation of the discriminator ${𝒟}_x$ will not be needed if we are not using its loss for updating generator $𝒢$. We also plan to utilize the more advanced DP analysis approach such as (Balle and Wang 2018; Wang et al. 2023) for tighter privacy analysis and further improve privacy and utility trade-off.