A novel method leveraging time series data to improve subphenotyping and application in critically ill patients with COVID-19

Abstract

Computational subphenotyping, a data-driven approach to understanding disease subtypes, is a prominent topic in medical research. Numerous ongoing studies are dedicated to developing advanced computational subphenotyping methods for cross-sectional data. However, the potential of time-series data has been underexplored until now. Here, we propose a Multivariate Levenshtein Distance (MLD) that can account for address correlation in multiple discrete features over time-series data. Our algorithm has two distinct components: it integrates an optimal threshold score to enhance the sensitivity in discriminating between pairs of instances, and the MLD itself. We have applied the proposed distance metrics on the k-means clustering algorithm to derive temporal subphenotypes from time-series data of biomarkers and treatment administrations from 1039 critically ill patients with COVID-19 and compare its effectiveness to standard methods. In conclusion, the Multivariate Levenshtein Distance metric is a novel method to quantify the distance from multiple discrete features over time-series data and demonstrates superior clustering performance among competing time-series distance metrics.

1. Introduction

Computational subphenotyping [1-3], a data-driven approach to identifying hidden patterns in complex diseases, is an important research topic in the medical domain. Multiple ongoing investigations have been conducted on developing cutting-edge computational methods for discovering subphenotypes in complex diseases [4-8]. These methods can transform data into novel subphenotypes, facilitating the development of high-performing diagnostic/prognostic models [9-11] and optimized treatment decisions [12,13]. However, these developments have yet been translated into clinical practice.

The reasons for this are multifactorial, but a major reason is the discordance between current subphenotyping methods and actual clinical practice. Healthcare providers make clinical decisions based on temporal observations available on electronic health records (EHR) by determining in at which step patients are in the course of disease progression and, more importantly, which trajectories within complex diseases patients are following [12,14-17]. Thus, clinically actionable subphenotypes need to represent time-varying characteristics, which is not accounted for by most methods that use only cross-sectional data. In turn, there exists a great need for computational subphenotyping methods to identify latent clinical information and subphenotypes from temporal observations.

Computational subphenotyping methods require addressing four challenges arising from the unique characteristics of medical data and domain-specific requirements. The first challenge is correlation [18], where multiple features in medical data are correlated, such as blood glucose level and triglycerides [19]. Failure to account for correlation can lead to biased subphenotyping results. The second challenge is quantifying temporal similarity [20-23], as medical data observations occur irregularly and in varying contexts based on patient characteristics and clinical guidelines [24,25]. An example that demonstrates this point is the variation observed in serum creatinine levels in critically ill patients between day 1 and day 7, as well as over a period of seven consecutive days. While the former may indicate the recurrence of acute kidney injury, the latter might suggest acute kidney disease [26]. Accurately quantifying temporal similarity requires an appropriate research design and proper computational subphenotyping methods. The third challenge is concisely deriving subphenotypes, [27,28] as some subphenotyping methods may generate excessive subphenotypes. While these may be useful for demonstrating improved numerical performance, such as in risk models, they can pose obstacles to clinical implementation. The final challenge is data types. It is widely recognized that various methodologies are best suited for specific data types [29]. For example, the Euclidian distance method is effective for continuous features but not suitable for discrete features. Although several accomplishments have been achieved in identifying subphenotypes from multiple continuous features in time-series data [30-33], the application of subphenotyping to multiple discrete features in clinical settings remains restricted.

In this study, we focus on computational subphenotyping methods that involve analyzing multiple discrete features over time-series data. While association rule mining (ARM) techniques such as sequential pattern mining (SPM) [34-38] and temporal abstraction (TA) [37-39] are commonly used to extract frequently occurring subsequences in time-series data, they have limitations for clinical applications. SPM and TA may produce subphenotypes that are clinically identical and lack interpretability due to the exponentially growing number of sequential patterns. Tensor factorization (TF) [6-8], another approach that decomposes three-dimensional tensors into smaller tensors, has received attention in machine learning communities, but its mathematical complexity and lack of replicability in clinical practice limit its applicability. Finally, clustering with time-series distance metrics [16,17] is a promising approach to address the limitations of current subphenotyping methods. Time-series distance metrics are not a new concept, as they are derived from edit distance [21,22], which has been used for DNA/RNA/protein sequence alignment [22] and natural language processing [40]. Recent efforts have been made to expand the use of time-series distance metrics to incorporate multiple discrete data and temporal information. Although these efforts have shown promise for improving computational subphenotyping, challenges still remain, including substantial information loss [16] and correlation issues [17].

To address these limitations, we propose a multivariate Levenshtein distance (MLD), a generalization of Levenshtein distance (LD) [21], that can quantify the distance between multiple discrete features over time-series data. The unique problem that MLD solves is the classification of binary labels as equal or not-equal across multiple discrete features without ground truth labels. The true binary labels are generally unknown for unsupervised learning settings. We solve this problem through multivariate distance metrics with a user-defined threshold. Specifically, we adopt a unified distance metric (UMD) [41] to measure the distance between two sets of multiple discrete features, where UMD can address the correlation between features and relax the existing entropy-based discrete distance assumption without losing information. In addition, we also propose an optimal substitution threshold discovery to identify a threshold for classifying equal or not-equal binary latent labels between two sets of multiple discrete features over time-series data.

We then evaluate this new distance metric on the clinically relevant problem of identifying dynamic subphenotypes in critically ill patients with COVID-19 using the same dataset from our published work [16]. We evaluate MLD with two competing distance metrics, Levenshtein distance with dimensionality reduction (LDDR) [16] and medication alignment (MedAl) [17], on the derivation of subphenotypes from biomarkers and treatments during the first 24 h of the critical care stay. We successfully demonstrated that the MLD identifies the optimal substitution threshold and shows superior performance compared to time-series distance metrics. Fig. 1 illustrates the workflow of this study.

Fig. 1.

We identified 1306 critically ill COVID-19 patients admitted to the Mount Sinai Health System in New York City. We quantified time-series distances of biomarkers and treatment administration from the first 24 h of intensive care unit (ICU) stay using the proposed multivariate Levenshtein distance (MLD) and two standard methods, Levenshtein distance with dimensionality reduction (LDDR) and medication alignment (MedAl). For evaluation purposes, we derived subphenotypes using distances with two clustering methods: k-means and hierarchical clusterings. We first evaluated the association between threshold scores for MLD and those clustering performances. Second, we compared clustering performances among MLD, LDDR, and MedAl.

2. Related works

The field of computational subphenotyping has expanded significantly, with numerous methods developed to analyze numerical features in time-series data. Our focus, however, is specifically on computational subphenotyping methods that analyze multiple discrete features within such data. In this section, we critically examine these methods, considering how their unique requirements impact their suitability for computational subphenotyping.

2.1. Association rule mining

Association rule mining (ARM)-based approaches, particularly its application to time-series data through sequential pattern mining (SPM) [34-38] and temporal abstraction (TA) [37-39], are prevalent in computational subphenotyping for time-series data.

Sequential pattern mining (SPM) is a machine learning method that identifies frequently occurring sequences of items or events within a dataset. The process involves discovering subsequences that appear repeatedly across a data sequence within a specified time period or across different sequences. In the context of disease progression, the items or events represent various diseases, and the subsequences represent segments of disease progression. SPM is particularly useful for analyzing time-series data, such as consumer purchasing patterns [42,43] and genomic sequences [44,45].

Temporal Abstraction (TA), in contrast, is a technique that refines raw, time-stamped data into a more comprehensible, interval-based format. It breaks down intricate time-series data into segments associated with qualitative descriptors such as “high,” “low,” or “increasing.” This simplification aids in the interpretation of complex data by emphasizing critical events and their temporal evolution.

While SPM and TA are adept at uncovering patterns of short event sequences in clinical contexts, such as readmissions [46] and medication administration [47], they are not without limitations. These methods are prone to a lack of a mechanism for distinguishing independent subphenotypes, leading to potential overlaps in identified subphenotypes and a loss of interpretability due to the vast number of patterns generated.

2.2. Tensor factorization

Tensor Factorization (TF) [6-8] is a mathematical operation that simplifies higher-order tensors into their constituent tensors, with the term “tensor” denoting a multi-dimensional generalization of vectors. Within the realm of computational subphenotyping applied to time-series data, TF leverages a tri-dimensional model that includes dimensions for patients, features, and temporal sequences. The decomposed tensors obtained through this process are indicative of subphenotypes, each comprising a cohort of patients, associated features, and pertinent time frames.

The machine learning community has shown considerable interest in TF due to its applicability in various domains such as signal processing [48] and neuroscience [49,50], where the data is inherently multidimensional. Tensor factorization’s utility extends to functions including data compression, noise reduction, and the extraction of significant features from complex datasets. Despite its utility, TF’s complex mathematical structure and the challenges associated with replicating its outcomes in clinical practice have impeded its widespread application in the healthcare domain. Another limitation of TF is its inherent inability to accommodate varying lengths of disease progression timelines among patients who otherwise share similar disease characteristics. This inflexibility can be a critical shortfall when modeling the dynamic nature of patient health trajectories in medical settings.

2.3. Clustering with time-series distance metrics

Clustering with time-series distance metrics [16,17], particularly for handling multiple discrete features in time-series data, is a technique used to group similar time-series data based on specific distance measures. The distance between time-series data is calculated using specialized metrics that consider the sequence and timing of the data points, such as Levenshtein distance with dimensionality reduction (LDDR) [16] and medication alignment (MedAl) [17]. These metrics are designed to handle the variability in time-series data, such as differing lengths of sequences or time shifts within the data. Clustering time-series data involves using these distance metrics within clustering algorithms like k-means or hierarchical clustering to form groups of time-series that behave similarly over time.

Distance metrics for time-series data involving discrete features are well-established in analytical domains. Traditionally, these metrics have been optimized for single-feature analyses rather than for multifaceted features. Their applications have been prolific in fields such as DNA/RNA/protein sequence alignment [22] and natural language processing [40]. Recent initiatives have aimed to extend the scope of these time-series distance metrics to simultaneously address multiple discrete data points and integrate temporal aspects. While this expansion has potential for refining the process of computational subphenotyping, it is not without hurdles, such as the potential for substantial information loss and complications due to correlational factors.

3. Materials and methods

3.1. Mathematical representation of time-series observations

Let $x_k$ be a binary matrix of a patient $k$’s string set, i.e., time-series data, with a size of $T\times F$ where $T$ is the number of times, and $F$ is the number of features. Specifically, $x_k(t,f)=1$ indicates that a patient $k$ presents $f$-th clinical features at $t$ time. Likewise, $x_k(t,f)=0$ indicates that a patient $k$ does not present $f$-th clinical features at $t$ time, $x_k(t,)$ and $x_k(i:j,)$ represent a patient $k$’s observations at $t$ time and between $i$ and $j$ time, which we call instance and substring set, respectively.

3.2. Multivariate Levenshtein distance

Here, we generalize Levenshtein distance (LD) [21], which can take multiple discrete features over the time-series data. Let $x_a$ and $y_b$ as binary matrices representing the string sets of patients $a$ and $b$, as delineated in Section 3.1. The multivariate Levenshtein distance (MLD) between $x_a$ and $x_b$ is given by ${MLD}_{X_a,X_b}(\mid x_a\mid ,\mid x_b\mid )$

$${MLD}_{X_a,X_b}(i,j)=min\{\begin{array}{cc}0\hfill & ,\text{if}i=j=0\hfill \\ {MLD}_{X_a,X_b}(i-1,j)+1\hfill & ,\text{if}i>0\hfill \\ {MLD}_{X_a,X_b}(i,j-1)+1\hfill & ,\text{if}j>0\hfill \\ {{MLD}_{X_a,X_b}(i-1,j-1)+1}_{\left({\widetilde{ND}}_{X_a,X_b}(i,j)>\theta \right)}\hfill & ,\text{if}i,j>0\hfill \end{array}\phantom{\}}$$ (1)

where ${\widetilde{ND}}_{x_a,x_b}(i,j)$ be a normalized discrete distance between two instances, i.e., $x_a(i,)$ – a patient $a$’s observation at the $i$ time and $x_b(j,)$ – a patient $b$’s observation at the $j$ time, $\theta$ is a threshold to classify either match or mismatch of binary labels, and $1_{(•)}$ is the indicator function to map either 1 or 0. ${MLD}_{X_a,X_b}(i,j-1)+1$, ${MLD}_{X_a,X_b}(i-1,j)+1$, and ${MLD}_{X_a,X_b}(i-1,j-1)+1_{\left({\widetilde{ND}}_{X_a,X_b}(i,j)>\theta \right)}$ represent costs for insertion, deletion, and match/substitution operations, respectively. MLD recursively finds the minimum cost for insertion, deletion, or match/substitution operations over two time-series observations. This process is visually depicted with example in Supplementary Fig. 1.

The unique conundrum of quantifying the similarity over the multiple discrete time-series observations, as opposed to LD, can be found in the match/substitution operation. The match/substitution operation relies on either an equal or not-equal binary label between two instances. It is legitimate that the result of the identical comparison between single discrete features is the same as the value of binary label. However, the true binary labels are generally not accessible in comparing multiple discrete features. Here, binary label classification is made through the discrete distance metrics with a user-defined threshold where unified distance metric (UMD) [41] was applied to quantify the distance between discrete variables while addressing correlation.

3.3. Optimal substitution threshold discovery

The remaining conundrum of MLD is the discovery of the optimal substitution threshold for determining either an equal or not-equal binary label between two instances. Substitution thresholds larger than optimal values can cause the loss of discrimination ability among patients with distinct time series observations. In comparison, thresholds smaller than optimal values can overreact even with negligible differences. Here, we propose a threshold discovery score based on categorizing pairs of substring sets.

3.3.1. Background

Let $x_a^{\prime }$ and $x_b^{\prime }$ be $2L+1$ length substring sets from string set $x_a$ and $x_b$, where $L$ is the number of observations to determine the length substring sets. A pair of substring sets can be aligned optimally with minimum edit operations, where the alignment can be altered with different substitution threshold $\theta$. Many pairs of substring sets can be aligned optimally with various edit operations, but not all. Some pairs of substring sets can be aligned with a single insertion, deletion, match, or substitution, undoubtedly if we constrain up to 1 edit operation in the middle of substring sets. Here, we categorize those pairs of substring sets which can be aligned with up to 1 edit operation in the middle of substring sets as insertion, deletion, match, or substitution. Furthermore, we categorize the rest of the pairs of substring sets which cannot be aligned with up to 1 edit operation in the middle of substring sets as others. Fig. 2 (a) illustrates the 5 categories, i.e., insertion, deletion, match, substitution, and others. Supplementary Fig. 2 shows examples of pairs of substring sets that can fall into the category others.

Fig. 2.

(a) Pair of substring sets that can classify insertion, deletion, match, substitution, undoubtedly. (b) Identification of pairs of substring sets that represent insertion, deletion, match, substitution undoubtedly.

The 5 categories introduced above, i.e., insertion, deletion, match, substitution, and others, are the crucial components for the optimal substitution threshold discovery. The optimal substitution threshold discovery consists of three parts. First, we will introduce a method to classify categories given a pair of substring sets, $x_a^{\prime }$ and $x_b^{\prime }$, and a threshold $\theta$. Second, we will introduce the likelihood of observing a particular category given a pair of substring sets, $x_a^{\prime }$ and $x_b^{\prime }$. Third, we will introduce a threshold discovery score. Specifically, the score accounts for a) the mean likelihood of observing each category given patients with the corresponding category and b) differences between distributions where one is the likelihood of observing a particular category given patients with the same category and the other is a different category. The rest of this section will be devoted to explaining three parts.

3.3.2. Mathematical representation for a pair of substring sets

A pair of substring sets can be divided into 5 subsets, infix, prefix, suffix, right shift, or left shift. infix refers to a pair of instances in the middle of each substring set, $x_a^{\prime }(L+1,)$ and $x_b^{\prime }(L+1,)$. prefix denotes a pair of first $L$ instances on each substring set, $x_a^{\prime }(1:L,)$ and $x_b^{\prime }(1:L,)$, while suffix denotes a pair of last $L$ instances on each substring set, $x_a^{\prime }(L+2:2L+1,)$ and $x_b^{\prime }(L+2:2L+1,)$. right shift refers to a pair of last $L$ instances on each substring set where the right-hand side substring set is lagged, $x_a^{\prime }(L+2:2L+1,)$ and $x_b^{\prime }(L+1:2L,)$. Likewise, left shift refers to a pair of last $L$ instances on each substring set where the left-hand side substring set is lagged, $x_a^{\prime }(L+1:2L,)$ and $x_b^{\prime }(L+2:2L+1,)$. Fig. 2 (b) shows dividing a pair of substring sets with length 5 into 5 subsets, infix, prefix, suffix, right shift, or left shift. For notational convenience, we use the first letter of each subset.

3.3.3. Classification of 5 categories given a pair of substring sets and a threshold

Let $AE(•,\theta )$ be a logical product of equal or not-equal binary labels from a set of comparisons across a subset from a pair of substring sets $•$ given a threshold $\theta$. For instance, $AE(p,\theta )={\bigwedge }_{l\in \{1,\cdots ,L\}}{\widetilde{ND}}_{x_a^{\prime },x_b^{\prime }}(l,l)\le \theta$ refers to a logical product of binary labels of the instances on the fragment prefix given a threshold $\theta$, where $AE(p,\theta )$ will be TRUE only if all the binary labels of the instances on the component prefix are equal. Let $AN(•)$ be a logical product operation of the negation of equal or not-equal binary labels of the instances on the fragment $•$ given a threshold $\theta$.

We classify a pair of substring sets $x_a^{\prime }$ and $x_b^{\prime }$ into insertion, deletion, match, substitution, or other categories using $AE(•,\theta )$ and $AN(•)$ with 5 subsets as follows:

$$CS(x_a^{\prime },x_b^{\prime },\theta )=\{\begin{array}{cc}insertion,\hfill & \text{if}AE(p,\theta )\wedge AN(i,\theta )\wedge AN(s,\theta )\wedge AE(r,\theta )\wedge AN(l,\theta )\hfill \\ deletion,\hfill & \text{if}AE(p,\theta )\wedge AN(i,\theta )\wedge AN(s,\theta )\wedge AN(r,\theta )\wedge AE(l,\theta )\hfill \\ match,\hfill & \text{if}AE(p,\theta )\wedge A\mathrm{E}(i,\theta )\wedge AE(s,\theta )\wedge AN(r,\theta )\wedge AN(l,\theta )\hfill \\ substitution,\hfill & \text{if}AE(p,\theta )\wedge AN(i,\theta )\wedge AE(s,\theta )\wedge AN(r,\theta )\wedge AN(l,\theta )\hfill \\ others,\hfill & \text{others}\hfill \end{array}\phantom{\}}.$$ (2)

Remember that the classification needs to be mutually exclusive so that a pair of substring sets with one category cannot be another. Tues, for instance, category insertion can be classified if the following 5 conditions are satisfied: a) $AE(p,\theta )$ – all pairs of instances along the prefix are all equal, b) $AN(i,\theta )$ – a pair of instances on infix is not-equal, c) $SN(s,\theta )$ – all pairs of instances along the postfix are all not-equal, d) $SE(r,\theta )$ – all pairs of instances along the right shift is all equal, and e) $SN(l,\theta )$ – all pairs of instances along the left shift are all not-equal. Fig. 2 (b) illustrates the classification based on the 5 subsets, i.e., infix, prefix, suffix, right shift, and left shift.

3.3.4. Likelihood of observing a particular category given a pair of substring sets

Let $SE(•)$ be a product of the similarity of instances across a fragment $•$. Like the classification of 5 categories given a pair of substring sets and a threshold, $SE(p)={\prod }_{l\in \{1,\cdots ,L\}}{S}_{x_a,x_b}(l,l)$ denotes a product of the similarity of the instances on a subset prefix, where $S_{x_a,x_b}(i,j)=1-{\widetilde{ND}}_{x_a,x_b}(i,j)$ is a similarity of two instances. Likewise, let $SN(•)$ be a product of the dissimilarity of instances on a subset $•$. We assume the similarity and dissimilarity follow Gaussian distribution.

We introduce 5 likelihoods of observing each category given a pair of substring sets, $x_a^{\prime }$ and $x_b^{\prime }$, which follows the same manner of the classification of 5 categories as follows:

$$\begin{gathered} \text{Insertion}:\ell (insertion\mid x_a^{\prime },x_b^{\prime }) \\ ={(SE(p)\times SN(i)\times SN(s)\times SE(r)\times SN(l))}^{1∕(4L+1)} \end{gathered}$$ (3)

$$\begin{gathered} \text{Deletion}:\ell (deletion\mid x_a^{\prime },x_b^{\prime }) \\ ={(SE(p)\times SN(i)\times SN(s)\times SN(r)\times SE(l))}^{1∕(4L+1)} \end{gathered}$$ (4)

$$\text{Match}:\ell (match\mid x_a^{\prime },x_b^{\prime })={(SE(p)\times SE(i)\times SE(s)\times SN(r)\times SN(l))}^{1∕(4L+1)}$$ (5)

$$\begin{gathered} \text{Substitution}:\ell (substitution\mid x_a^{\prime },x_b^{\prime }) \\ ={(SE(p)\times SN(i)\times SE(s)\times SN(r)\times SN(l))}^{1∕(4L+1)}. \end{gathered}$$ (6)

For instance, the likelihood of observing category insertion is a multiplication of a) $SE(p)$ – a product of the similarity of the instances on prefix, b) $SN(i)$ – a product of the dissimilarity of the instances on infix, c) $SN(s)$ – a product of the dissimilarity of the instances on postfix, d) $SE(r)$ – a product of the similarity of the instances on right shift, e) $SN(l)$ – a product of the dissimilarity of the instances on left shift, where for conveniences, we take $1∕(4L+1)$ root on the multiplication.

3.3.5. Threshold discovery score

Multiple criteria can exist to determine optimal thresholds for MLD. For instance, the likelihood of observing each category given patients with the same category needs to be maximal. In addition, the likelihood of observing each category given patients with different categories needs to be minimal. Here, we formally define the above two criteria as follows.

Let $X^{\prime }$ be a set of all possible pairs of substring sets from time-series data, and let $C=\{noedit,substitution,insertion,deletion\}$ be 4 categories for a pair of substring sets except for others. Let threshold discovery score for having the same categories be

$${TDS}_{same}(\mathbb{X},\theta )=\frac{1}{4}{\sum }_{c\in C}E[\ell (c\mid x^{\prime },x^{″}{x}^{\prime },x^{″}\in X^{\prime }\&x^{\prime }\ne x^{″}\&CS(x^{\prime },x^{″},\theta )=c)],$$ (7)

where we initially take the mean of the likelihood of observing each category given patients with the same category, then take the mean of the mean likelihood of each category across the categories. Let threshold discovery score for having different categories be

$${TDS}_{different}(\mathbb{X},\theta )=\frac{1}{6}{\sum }_{c\in C}E[\ell (c\mid x^{\prime },x^{″}{x}^{\prime },x^{″}\in X^{\prime }\&x^{\prime }\ne x^{″}\&CS(x^{\prime },x^{″},\theta )\ne c)],$$ (8)

where, similar to ${TDS}_{same}(\mathbb{X},\theta )$, we take the mean of the mean likelihood of observing each category given patients with different categories. Finally, let the threshold discovery score be

$$TDS(\mathbb{X},\theta )={TDS}_{same}(\mathbb{X},\theta )+\left(1-{TDS}_{different}(\mathbb{X},\theta )\right),$$ (9)

where $TDS(\mathbb{X},\theta )$ takes the sum of both the likelihood of observing each category given patients with the same category, i.e., ${TDS}_{same}(\mathbb{X},\theta )$, and the unlikelihood of observing each category given patients with different categories, i.e., $1-{TDS}_{different}(\mathbb{X},\theta )$.

3.4. Experimental setup

We conducted a retrospective observational study using 9524 patients with coronavirus disease 2019 (COVID-19) from the Mount Sinai Health System (MSHS). We defined a confirmed COVID-19 case [51] with a positive reverse transcription-polymerase chain reaction (RT-PCR) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) from a nasopharyngeal swab sample. We excluded patients under the age of 18 (n = 14), without ICU admission (n = 7738), with an unknown discharge date (n = 23), patients admitted to the intensive care unit (ICU) outside of the study period (n = 368), with <24 h of stay (n = 122), with no vital signs (n = 174), and with no death information (n = 2). 1036 patients left in our final cohort.

We extracted sociodemographic information (e.g., age, sex, race, and ethnicity). We also extracted discretized clinical/biochemical measurements and treatments, which are widely assessed and used in the ICU setting. This data includes indicators such as low systolic blood pressure (below 90 mm Hg), elevated heart rate (over 90 beats per minute), increased respiratory rate (above 90 breaths per minute), reduced partial pressure of carbon dioxide (PaCO2 below 32 mm Hg), high white blood cell count (above 12,000 cells/μL), a low P/F ratio (PaCO2 to FiO2 ratio below 150), high random glucose levels (above 250 mg/dL), low arterial pH (below 7.25), elevated serum bicarbonate (above 12 mmol/L), a doubling of serum creatinine from the baseline, along with interventions such as red blood cell transfusion, administration of vasoactive agents, loop diuretics, mechanical ventilation, neuromuscular blocking agents, insulin, and hemodialysis. More detailed information regarding the patient characteristics in our cohort can be obtained from our previous publication [16].

The evaluation of the proposed MLD was carried out In two distinct phases. First, we assessed the optimal threshold score and establish its correlation with clustering efficacy, with a detailed discussion on clustering performance metrics to follow in this section. Second, we applied the MLD using the established optimal thresholds (0.35 for 1-hour, 0.30 for 1.5-hour, 0.15 for 2-hour, and 0.15 for 3-hour intervals) to measure its effectiveness against alternative methodologies. Consensus clustering was utilized to assess the performance of various time-series distance metrics. We derived 2 to 7 clusters on 1-hour, 1.5-hour, 2-hours, and 3-hour time slots using the k-means and agglomerative hierarchical clustering with Ward’s minimum variance linkage.

We applied and compared three distance metrics: multivariate Levenshtein distance (MLD), Levenshtein distance with dimensionality reduction (LDDR) [16], and medication alignment (MedAl) [17]. LDDR quantifies time-series multivariate discrete distance using Levenshtein distance, where multiple discrete features are encoded into a single discrete variable. MedAl quantifies time-series multivariate discrete distance through the sum of individual edit distance from each feature, where edit distance is calculated using scores on the customized pattern table.

We employed two independent clustering performance metrics [52]: normalized mutual information (NMI) and adjusted Rand index (ARI). Both metrics are used to evaluate the quality of clustering results against a ground truth set of labels, with the consensus of clusters serving as our ground truth.

An ensemble of clusters is denoted by $\Pi =\{{\pi }_1,\dots ,{\pi }_C\}$, where ${\pi }^{\ast }$ represents the consensus cluster within $\Pi$. ARI is a function of the number of agreements and disagreements between two data clustering assignments, such that

$$ARI(\pi ,{\pi }^{\ast })=\frac{n_{10}+n_{01}}{n_{11}+n_{10}+n_{01}+n_{00}}$$ (10)

where $n_{00}$ and $n_{11}$ denote the numbers of pairs in the same clusters, whereas $N_{10}$ and $N_{10}$ represent the number of pairs in the different clusters. The ARI adjusts the Rand Index for the chance grouping of elements, providing a value between −1 and 1. An ARI of 0 indicates random labeling, independent of the number of clusters and sample size, while an ARI of 1 indicates that the clusters from the clustering solution and the ground truth match perfectly.

Conversely, NMI is based on information theory and compares how much information is shared between the clustering assignments and the true labels. It measures the agreement between two different clusters by quantifying the distance between two probabilities, such that

$$NMI({\pi }_c,{\pi }^{\ast })=\frac{MI({\pi }_c,{\pi }^{\ast })}{\sqrt{H({\pi }_c)H({\pi }^{\ast })}}$$ (11)

where $Pr(k)=\frac{n_k}{N}$ is a probability of each cluster, $H(\pi )=-{\sum }_{k\in K}p(k)logp(k)$ is an entropy associated with the cluster, and $MI(\pi ,{\pi }^{\ast })={\sum }_{k\in K}{\sum }_{k^{\ast }\in K^{\ast }}Pr(k,k^{\ast })log\frac{Pr(k,k^{\ast })}{Pr(k)Pr(k^{\ast })}$ is a mutual information of two clusters $\pi$ and ${\pi }^{\ast }$. NMI is normalized to a scale from 0 to 1, where 0 means no mutual information and 1 indicates a perfect correlation between the two label sets. NMI is particularly robust to the number of clusters, which is advantageous when the number of clusters is not known a priori.

Both NMI and ARI are widely used in statistical data analysis to compare different clustering algorithms or to tune the parameters of a single algorithm. However, they may sometimes give conflicting results, as they emphasize different aspects of the clustering structure. NMI is generally more sensitive to the amount of shared information between the clusters, while ARI focuses on the exact pair-wise relationships between samples.

3.5. Statistical analysis

First, we evaluated the association between threshold discovery scores (TDS) and clustering performance. For each threshold between 0.05 and 0.75, we calculated TDS as well as measured ARI and NMI of k-means and agglomerative hierarchical clustering using MLD with 2–7 different numbers of clusters and 4 different time resolutions on 100 random samples (sampling without replacement). We drew scatter plots to show the relationship between TDS and ARI/NMI, and included those trend lines. Trend lines with an upwards slope indicate that higher TDS is associated with higher clustering performance and vice versa.

Second, we compared three time-series distance metrics, i.e., MLD, LDDR, and MedAl. Here, we used thresholds with the highest TDS in the previous section. We measured ARI/NMI of k-means and agglomerative hierarchical clustering with 2–7 clusters and 4 time resolutions on 100 random samples (sampling without replacement). We first evaluated at which number of clusters the elbow of ARI/NMI forms (the point where ARI/NMI drops substantially), and second, we evaluated the overall ARI/NMI before the elbow formed. The elbow represents that a clustering method can discover stability up to the number of clusters where the elbow forms.

We performed all statistical analyses using R software version 4.1.1 (R Foundation for Statistical Computing, Vienna, Austria) [53]. The source code is available at https://github.com/Nadkarni-Lab/ohw_aim_2022.

4. Results

4.1. Evaluation of optimal threshold discovery method for multivariate Levenshtein distance

Figs. 3 and 4 show scatter plots and trend lines of threshold discovery scores (TDS) against two clustering performance evaluation metrics, i.e., normalized mutual information (NMI) and adjusted Rand index (ARI), over 2–7 clusters, 1-, 1.5-, 2-, and 3-hour time resolutions, and k-means clustering. In addition, Supplementary Figs. 3 and 4 show the results from hierarchical clustering. Overall, positive correlations are found between TDS and clustering performance.

Fig. 3.

Threshold diseovery scores against normalized mutual information (NMI) of k-means clustering from 100 subsampling without replacement.

Fig. 4.

Threshold discovery scores against adjusted Rand indexes (ARI) of k-means clustering from 100 subsampling without replacement.

First, the evaluation of TDS and NMI forms consistent uptrend lines regardless of the different numbers of clusters, time resolutions, and clustering algorithms. 23 out of 24 trend lines from k-means clustering, and all trend lines from hierarchical clustering point toward the positive. One exception is from 5 clusters derived using k-means with 3 h as a time resolution.

These patterns are consistently present in the trend lines from TDS and ARI of fewer than 6 clusters from k-means clustering. Interestingly, ARI for cluster sizes of 6 or more from k-means clustering shows a negative correlation with TDS. We will discuss details in Section 3.2, but we expect this is due to the number of clusters that can be derived using k-means clustering with multivariate Levenshtein distance (MLD) on our dataset.

Lastly, TDS and ARI from hierarchical clustering show inconsistent results. Negative correlations are generally presented, yet this can be considered due to the low clustering performance on hierarchical clustering. In detail, k-means clustering shows, on average, 0.86 of NMI and 0.88 of ARI, while hierarchical clustering shows 0.57 of NMI and 0.57 of ARI, generally considered a lack of clustering performance.

After all, we identify 0.35, 0.30, 0.15, and 0.15 as the optimal thresholds for 1-, 1.5-, 2-, and 3-hour time resolutions.

4.2. Evaluation of multivariate Levenshtein distance against competing methods

Now, we move on to evaluating multivariate Levenshtein distance (MLD) against competing sequence distances. Figs. 5 and 6 show line plots of normalized mutual information (NMI) and adjusted Rand index (ARI) from k-means clustering with MLD, Levenshtein distance with dimensionality reduction (LDDR), and medication alignment (MedAl) over various numbers of clusters and time resolutions. We present consensus heatmaps of MLD on k-means clustering in Fig. 7. In addition, we demonstrate the results from hierarchical clustering, presented in Supplementary Figs. 5, 6, and 7.

Fig. 5.

The number of clusters against normalized mutual information (NMI) of k-means clustering from 100 subsampling without replacement.

Fig. 6.

The number of clusters against adjusted Rand indexes (ARI) of k-means clustering from 100 subsampling without replacement.

Fig. 7.

Consensus heatmaps of k-means clustering from 100 subsampling without replacement.

First, we evaluate clusters derived from k-means clustering with MLD. The NMI and ARI from k-means clustering with MLD form the elbows of the curves at 5 clusters on 1-, 1.5-, and 2-hour window resolutions and 4 clusters on 3-hour window resolutions. These elbows indicate that k-means clustering with MLD can discover up to 5 clusters on 1-, 1.5-, and 2-hour window resolutions and 4 clusters on 3-hour window resolutions. On average, NMI and ARI show 0.96 and 0.97 before the elbows of the curves, while these decrease to 0.81 and 0.76 after the elbows. These patterns are presented consistently on the NMI and ARI from hierarchical clustering with MLD. The consensus heatmaps in Fig. 7 confirm our findings. The x- and y-axes represent individual patients, and each point represents the frequency at which the same cluster occurs simultaneously between the two patients in 100 bootstrap resampling. Dark spots mean that two patients are likely to belong to the same cluster, and white indicates two patients who are less likely to belong to the same cluster. 5 clusters on 1-, 1.5-, and 2-hour window resolutions show that the squares along the diagonal are generally dark, which indicates a high consensus of the clustering results from the 100 resamples. In contrast, after 6 clusters, the squares tend to be lighter and darker regions appear outside the squares.

We compare the k-means clustering with MLD against clusters from competing distance metrics. The results show that the clusters from k-means clustering with MLD exhibit overall highest or at least comparably high NMI and ARI than competing distances. MLD placed the highest on 12 out of 15 from NMI and 13 out of 15 from ARI on the clusters, which k-means clustering with MLD can stably derive. In addition, clusters that k-means clustering with MLD cannot derive stably still place at least second in all comparisons. Among the competing distance metrics, MedAl tends to derive fewer clusters stably, and LDDR drops its performance rapidly as we use wider time resolutions.

Clusters derived from hierarchical clustering with MLD show lower NMI and ARI than clusters from k-means clustering with MLD. Clusters from k-means clustering with MLD show 0.91 NMI and 0.90 ARI on average across the different number of clusters and time resolutions, while clusters from hierarchical clustering with MLD show 0.58 NMI and ARI. Despite this unfavorable application of MLD on hierarchical clustering, MLD still shows at least the second-highest performance in general among all the competing distance metrics. These comparative results reassure us that using MLD allows us to deliver outstanding sequence distance measurements for various applications.

4.3. Evaluation of subphenotype consistency across competing methods

We conducted a comparison of the subphenotypes generated by 3 distinct distance metrics. To establish a reference point, we used the subphenotypes produced by LDDR as they have already been published, and then measured the degree of similarity between these subphenotypes and those generated by the other metrics.

We reused four subphenotypes that were previously identified based on COVID-19 patient characteristics at ICU admission and within the first 24 h of their stay, using the LDDR distance metric. In summary, subphenotype I (SP-I) was characterized by tachypnea and low mechanical ventilation at ICU admission, with a high prevalence of unfavorable biomarkers and treatments at 24 h. SP-I had the highest probability of survival and the second-highest probability of discharge from the hospital at 30 days. Subphenotype II (SP-II) had the lowest prevalence of unfavorable biomarkers and was less likely to require vasoactive agents or mechanical ventilation. SP-II had the highest probability of hospital discharge at 30 days. Subphenotype III (SP-III) had the highest incidence of shock at ICU admission, with the need for support with vasoactive agents and mechanical ventilation increasing over the first 24 h. SP-III showed the second-lowest probability of survival, although the proportion discharged from the hospital was comparable to SP-I and SP-IV. Subphenotype IV (SP-IV) had a high prevalence of respiratory failure, and all patients required mechanical ventilation at 24 h. SP-IV had a similar hospital discharge rate to SP-I and SP-III. Further details can be found in our previous publication [16].

Figs. 8 and 9 illustrate the Sankey diagrams representing changes in subphenotypes resulting from comparisons between LDDR and MLD, and LDDR and MedAl, respectively. Each node on the diagrams corresponds to subphenotypes identified by the respective distance metrics. The node and line width are scaled to indicate the proportionate quantity. Thus, when two nodes are joined by a line of equal width, it signifies a higher consistency in the clustering outcomes. First, Fig. 8 indicates that there is a good degree of consistency in the membership of the subphenotypes derived from LDDR and MLD. Specifically, the subphenotype SP1 derived from LDDR changes to SP3 in MLD for 19 % (N = 66) of cases, while for 2 % (N = 6) of cases, the subphenotype SP2 from LDDR changes to SP1 in MLD. The subphenotype SP3 derived from LDDR changes to SP4 in MLD for 26 % (N = 57) of cases. The multiclass AUROC value for subphenotypes between LDDR and MLD is 0.93, indicating a high level of agreement. Fig. 9, on the other hand, shows a low degree of consistency in subphenotypes derived from LDDR and MedAl. Only 26 % (N = 90), 45 % (N = 159), 46 % (N = 101), and 77 % (N = 89) of cases in LDDR remain in the same subphenotypes when using MedAl. The multiclass AUROC value for subphenotypes between LDDR and MedAl is 0.79, indicating a relatively low level of consistency.

Fig. 8.

Sankey diagrams representing changes in subphenotypes resulting from comparisons between LDDR and MLD.

Fig. 9.

Sankey diagrams representing changes in subphenotypes resulting from comparisons between LDDR and MedAl.

5. Discussion

In this study, we propose a novel metric for multiple discrete features over time-series data, entitled multivariate Levenshtein distance (MLD). We evaluate the proposed distance metric with three competing distance metrics, Levenshtein distance with dimensionality reduction (LDDR) [16] and medication alignment (MedAl) [17], on cluster analysis for the first 24 h of the episode of biomarkers and treatments after the COVID-19 associated intensive care unit (ICU) admissions in New York City between March and December 2020. We show that MLD exhibits the highest clustering performance among competing time-series distance metrics.

Nevertheless, the more important question might be understanding how MLD outperformed two competing distance metrics. To understand this, we first need to understand the bottleneck of the expanding (univariate) edit distance on multiple discrete features over time-series data. Edit distance relies on classifying equal or not-equal binary labels between two observations without accessing ground truth labels. While the binary classification of single discrete features is a self-evident proposition, multiple discrete features are an example of an ill-posed problem. Two standard methods, MedAl and LDDR, adopt different approaches to address this issue.

MedAl does not address this issue; instead, it takes the Manhattan distance (simply sum) of all (univariate) edit distances. This approach does not share the same limitation as LDDR, which we will discuss later in this section, but Manhattan distance cannot address correlation on multiple discrete features. Our proposed MLD addresses this by integrating a unified distance metric (UMD) – a discrete metric designed to quantify the distance between correlated discrete variables. It also introduces a thresholding technique to determine binary labels, enhancing discrimination sensitivity between pairs of instances. This novel approach allows for the measurement of feature correlations and overcomes the limitations of previous entropy-based discrete distance metrics without compromising information integrity. The performance of MLD is validated by the clustering outcomes it produces. Notably, the “elbows” – the points indicating the most appropriate number of clusters – of both Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI) for subphenotypes identified by MLD consistently suggest a greater number of clusters in comparison to those determined by MedAl. In contrast, the NMI and ARI for subphenotypes generated through MedAl tend to present at a reduced number of clusters, or “elbows” emerge at fewer subphenotypes. This pattern implies that subphenotypes inferred via MedAl are significantly influenced by the correlation of features, highlighting a potential limitation in its ability to discern distinct subphenotypic patterns within the data.

On the other hand, LDDR encodes multivariate discrete features into a univariate discrete feature, enabling binary classification as a self-evident proposition. However, this results in substantial information loss with increased variability. MLD classifies binary labels using a user-defined threshold on the discrete distance, where UMD is not affected by the substantial information loss and increased variability issue. Notably, this approach introduces a in new problem related to the optimal threshold for MLD, where thresholds higher or lower than optimal can lead to loss of discrimination and increase the volatility of minor differences between two sequences. Indeed, discovering the optimal threshold is crucial to the success of MLD. Our threshold discovery scores are calculated by the probability of observing pairs of substrings along 4 categories of no edit, substitution, insertion, and deletion, undoubtedly. As a result, thresholds with a higher threshold discovery score positively correlate with two clustering performance metrics in our use case. Furthermore, MLD with optimal threshold shows the highest overall performance compared to the competing distance metrics.

Determining the optimal time resolution depends on the clinical application since applications require different time resolutions due to different progression patterns, follow-up times and interventions. For instance, type 2 diabetes (T2D) takes multiple years to progress [14]. Thus, for assessing T2D trajectories in time-series analysis, <1 year of time resolution is not optimal since excessive granularity will cause substantial noise. On the other hand, most critically ill patients have minute-to-minute changes in status, and time-series analysis with a time resolution of >24 h may not capture some clinically meaningful patterns, such as trajectories with rapidly deteriorating patient conditions or recurrent symptoms over a short period.

Thus, a trade-off between desired time resolution and robustness exists. Rationally, a shorter time slot interval may allow the extraction of temporal patterns with high granularity, but it faces overfitting issues. On the other hand, longer time slot intervals may allow us to access robust temporal patterns, but the patterns can easily lose granularity. This can also be found in our evaluation that changes in time resolution can affect the distance’s variance and the degree of information retrieval. Our experiment demonstrates that 1.5 h of resolution time slots are sufficient to explore the temporal patterns for this use case, since the numerical performance does not show a statistical difference between 1 h and 1.5 h of time resolution. In addition, MLD decreases clustering performance as we reduce the time resolution, but the amount of performance decrease is subtle compared to competing methods. Thus, we anticipate this metric could be used for various use cases with different time resolutions, however, this will need to be evaluated in further studies.

This study is not without limitations. First, our evaluation of clustering performance relies on a small subset of clustering performance metrics, namely adjusted Rand index (ARI) and normalized mutual information (NMI), as most evaluation metrics, e.g., average Silhouette width and gap statistics, are designed for comparing different clustering algorithms, not distance metrics. Our next study aims to develop predictive models based on temporal subphenotypes, which can be used to evaluate distance metrics more comprehensively. Second, our evaluation was limited to the first 24 h of ICU stay in a single healthcare system. This is because, as we tried to extend the observation period beyond 24 h, the cohort size decreased substantially. Thus, we have not formally evaluated the ability of MLD to capture patterns that appear later in the ICU process or the robustness of our approach to other healthcare systems, although we believe that our method is likely to generalize to long-term or other health systems. To address this limitation, we plan to conduct a follow-up study using National COVID Cohort Collaborative (N3C) [54] data to investigate the robustness of subphenotypes across patients from various healthcare systems. Third, the proposed MLD relies on discrete features, such as normal-to-abnormal laboratory measurements and without-to-under treatments. While some features can discretize additional micro-levels to reduce the information lost, such as normal, borderline, and high abnormal laboratory measurements, and the indicator function we adopt, i.e., UMD, can quantify the distance over ordinal features effectively, we do not encode so to maintain consistent evaluation with our previous studies. This evaluation will need to be evaluated in further studies.

6. Conclusion

In our research, we introduced a new metric known as the Multivariate Levenshtein Distance (MLD), designed to measure the divergence among multiple discrete attributes within time-series data. Our algorithm is twofold: it establishes an optimal threshold score to determine the discrimination sensitivity between pairs of instances, and it defines the MLD itself. We assessed our metric using real-world data from critically ill COVID-19 patients. We have shown that the optimal threshold score effectively identifies the threshold without the need for exhaustive searching. Furthermore, our results illustrate that MLD is capable of quantifying distances across various discrete features in time-series data and outperforms other competing time-series distance metrics in terms of clustering efficacy. The primary limitation of this study lies in its focus on short-term follow-up of acute diseases, based solely on data from a single hospital setting. In future work, we aim to broaden our validation efforts to encompass more extensive follow-up periods and a wider range of chronic diseases.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author. Our institution has a data use committee and due processes requiring the transfer of data external to our institution.