Toward Identifying New Risk Aversions and Subsequent Limitations and Biases When Making De-identified Structured Data Sets Openly Available in a Post-LLM world

Abstract

Making clinical datasets openly available is critical to promote reproducibility and transparency of scientific research. Currently, few datasets are accessible to the public. To support the open science initiative, we plan to release the structured clinical datasets from the CONCERN study. In this paper, we are presenting our de-identification approaches for structured data, considering the future inclusion of de-identified narrative notes and re-identification risks in the LLM era. Through literature review and collaborative consensus sessions, our team made informed decisions regarding dataset release, weighing the pros and cons of each choice, outlining limitation and bias introduced by the de-identification algorithm. To our best knowledge, this is the first study describing the rationales of de-identification decisions in the LLMs era, delineating the consequent problems that should be considered when using our data set. We advocate for transparent disclosure of de-identification decisions and associated limitations and biases with all openly available datasets.

Introduction

Open science is a priority of the National Institutes of Health and other federally funded agencies, with the goal to make research inputs, activities and outputs accessible to all to enable principles such as reproducibility and fairness while maintaining security and privacy¹. There are some published accessible clinical datasets, particular in the setting of Intensive Care Units (ICUs) and critical care units, such as MIMIC IV (Medical Information Mart for Intensive Care)², HiRID (a high time-solution ICU dataset in Switzerland)³, PIC (Pediatric Intensive Care database in China)⁴, eICU-CRD (eICU Collaborative Research multi-center database in United States)⁵, and AmsterdamUMCdb (Amsterdam university medical centers database)⁶. MIMIC IV is particularly notable for the extensive value provided to the data science community in creating and maintaining a large openly available clinical data set. However, the availability and diversity of open datasets remain limited in scale. External validation of data-driven clinical models, such as those used for prediction, is necessary to showcase performance in real-world settings^7,8, given that models built on one medical center’s data can be highly biased and do not necessary capture the diverse characteristics of the population well. Moreover, the limited availability of data representing multiple and diverse patient populations for use by data scientists has demonstrated clinical implications. A widely circulated study⁹ performed an external validation of the Epic Sepsis Model (ESM) which had already been widely adopted by many hospitals, and found that the model did not effectively detect the early onset of sepsis in their cohort, highlighting the importance of external model validation across diverse clinical sites prior to local adoption. It is not uncommon for predictive models, such as those predicting the risk of adverse events or short-term mortality for patients in ICU^10–13, to have been developed based on a single medical site; while these practices raise questions about robustness and generalizability they are understandable given the limited openly available clinical data sets that exist and resource challenges of conducting multi-site studies. Making clinical data derived from electronic health records (EHRs) publicly available empowers the reproducibility, transparency of scientific research, and advances knowledge discovery^14,15.

In the light of the open science initiative and advancement of nursing research activities, we plan to release a subset of patient data from acute care and intensive care units (ICU) as part of the Communicating Narrative Concerns Entered by RNs (CONCERN: https://www.dbmi.columbia.edu/concern-study/) study¹⁶ by the end of 2024. The CONCERN data set is comprised of structured and unstructured (narrative nursing notes) clinical data that serves as the input for the CONCERN’s model features and additionally includes care provider information (e.g., specialty and type) and linkage between the actions taken and respective providers. Our first release will include only structured data. To our knowledge, this is first dataset for open public sharing that captures both the care providers’ information and patient data simultaneously, offering insightful knowledge in understanding the behaviors of care providers in response to patients’ health status.

As mandated by Health Insurance Portability and Accountability Act (HIPAA)¹⁷, all clinical data is required to undergo de-identification procedure for privacy protection prior to public release or any sharing across healthcare institutions. De-identification refers to the removal of protected health information (PHI) that could potentially reveal individual’s or a small subgroup’s identity. Safe Harbor is one of the de-dentification approaches recommended by HIPAA, where clinical data (both structured and unstructured) are deemed free of PHI and can be shared after the removal of all pre-defined 18 PHI categories (e.g., Names, Dates, SSN, locations). However, there remains a risk of re-dentification under such approach^18,19, and the emergence of large language models (LLMs) raises further concern associated with data privacy and security²⁰. LLMs have been showcased to achieve the state-of-the-art performances on various natural language processing tasks^21,22, which can easily integrate diverse sources to expand knowledge domain and optimize decision-making²³. The advancement of LLMs has unexplored implications on current best practices for de-identification. While these implications are unexplored, it is believed that LLMs will likely have capabilities that did not exist previously for re-identifying data sets²², substantially and necessarily changing decision-making considerations today for what and how to make data sets openly available. In this study, we considered the potential re-identification risks imposed by LLMs when designing our de-identification algorithm. Importantly, we also consider our future plans to include de-identified narrative notes²⁴ in subsequent versions of our openly available data set.

The aim of this paper is to describe the considerations for releasing a clinical data set in the era of LLMs, the specific decisions our team arrived at through consensus sessions, the pros and cons of each decision, and the known and unknown implications of each decision on secondary analyses by other researchers once the data set is made public. We present a preliminary design of the databases and provide the description of de-identification procedure, identifying potential bias induced by the design of de-identification algorithm. We end with a robust discussion of our resultant decisions and their implications. To our best knowledge, this is the first study describing the rationales of de-identification decisions in the era of LLMs and explicitly delineate the consequent limitations and bias imposed on future users of that data set. To ensure transparency and validity of subsequent scientific research using any openly available data set, it is necessary to acknowledge and disclose any limitations of the dataset resulting from the de-identification algorithms used, and it is also imperative for researchers to understand these issues when using and analyzing openly available datasets.

Methods

The dataset was collected from acute care and ICU settings at 2 large health systems in the Northeastern United States, spanning the years 2020 to 2022.

We followed a 5-step approach to reach consensus on our de-identification methods and their implications:

1. Review our team’s prior findings regarding de-identification of narrative notes to inform decisions for de-identification of structured data, with the future premise to release narrative notes at later stage.

2. Identify de-identification methods from: a) safe harbor, b) other studies using similar EHR data.

3. Identify novel sources of potential risk for re-identification in the era of LLMs.

4. Iteratively review each de-identification approach and risks during team consensus sessions for pros and cons along the following factors: a) balance between data utility and privacy protection, b) risk of re-identification

5. Finalize de-identification decisions.

Results

Step 1: Review previous outcomes regarding de-identification of narrative notes.

In our previous study²⁴ examining the generalizability of pre-trained de-identification transformers on narrative nursing notes, we found that the F1-measure in detecting PHI tokens was 0.932, and models (RoBERTa, ClinicalBERT) did not perfectly capture all PHI instances from notes. Information such as phone number, patients’ affiliated organization names or location can still remain present in the notes even after de-identification algorithms are applied. We do not plan to release the notes in our initial release and we will conduct thorough manual validation to ensure all PHI is removed before proceeding with the release of notes. We recognize, and it is important to convey, that one can still identify an individual by deliberately combining residual information from unstructured and structured data. Therefore, a cautious and conservative approach related to de-identification decisions for our structured data is preferred given our eventual plan to release both structured data and narrative notes.

Step 2 Identify de-identification methods from: a) safe harbor, b) other studies using similar EHR data (summarized in Table 1)

Table 1.

Summary of de-identification employed by other studies.

Given the primary targeted population of the CONCERN study is focused on adults, patients age below 18 at the time of visit were excluded from the data. The de-identification followed the Safe Harbor method by removing all 18 PHI categories, as applicable. In selecting the specific strategies for our de-identification, we reviewed the main de-identification strategies implemented by other studies, which are summarized in Table 1.

Step 3) Identify novel sources of potential risk for re-identification in the era of LLMs

De-identifying structured data can be regarded as relatively straightforward²⁵, whereas de-identifying narrative notes is perceived as a challenging task. In anticipation of eventually incorporating the unstructured component to offer auxiliary enrichment clinical details, we must also consider additional risks of privacy, and re-identification particular in the era dominated by LLMs. While LLMs demonstrate remarkable ability in understanding, reasoning and in-context learning, there has been increasing concerns towards data privacy and safety. Firstly, LLMs can combine multiple available sources online to potentially infer individual’s identity. As demonstrated by one study²⁶, pre-trained language models can leverage different kinds of knowledge to generate accurate prediction of the target, and with strong and larger scale of the models, more personal information can be extracted and recovered. With exponential growth in the size and complexity of the models, they become increasingly susceptible to incidental leakage of private information²⁷.

Step 4) Iteratively review each de-identification approach and risk during team consensus sessions for pros and cons along the following factors: a) balance between data utility (secondary purposes) and privacy protection, b) risk of re-identification.

We identified three main de-identification needs: 1) dealing with date/times; 2) dealing with ages; and 3) dealing with rare data which entails all conditions with low occurrence in our dataset, including but not limited to rare diseases.

Step 5) Final De-identification Decisions

We transformed the dates and specific timestamps by using relative time in minutes since the admission. Some studies^31–33 revealed the effect of night-time and weekend admission on in-hospital mortality. Therefore, we included the binary day night and weekend indicators to capture these important factors which may contribute to ultimate patients’ outcomes. To comply with HIPAA regulations which classify ages above 89 as PHI, we obscured ages above 89 by replacing the specific ages to “age above 89” for privacy protection. Furthermore, we removed all conditions with low occurrence in the cohort based on certain threshold determined by analyzing the distribution of condition occurrence.

The elements of the databases were shown in Figure 1. Tables can be linked together using identifiers such as dummy MRN (patient ID) and encountered_id.

Figure 1:

An overview of the data elements. The final version is subject to change and might differ from the preliminary version depicted above.

Discussion

We identified several major issues carried by the design choice of our proposed de-identification algorithm. Below, we described several implications of each de-identification decision.

Dates / Time: Relative to the reference point (admission date)

We obscured the dates and specific timestamps by using relative time (in minutes) from the individual’s corresponding admission date. Admittedly, keeping only the time interval will inevitably result in the loss of temporal information, restricting the ability to examine seasonal effects. It is important to acknowledge that our dataset is not ideal for supporting temporal analysis, examining the effect of seasonality on patients ‘outcome as well as how it may alter providers ‘behaviors and decision-making processes. Some datasets like MIMIC-IV², PIC⁴, AmsterdamUMCdb⁶ retained the seasonality and fine temporal details such as time of the day and day the week; nevertheless they require large amount of datasets spanning over wide calendar years. The size of the dataset we intend to release is relatively small, covering a short timeframe with approximate one to two years. The risk of re-identifying the actual dates is much higher when maintaining such fine level of temporal descriptions. As indicated by Sarpatwari et al.²⁹, simply leveraging time interval is still possible to make some inferences about true temporal details by analyzing interaction with clinical study periods. Essentially, both the time shifting approach without truncation of data points and the relative time approach poses a certain level risk of re-identification, which are less robust for continuously updated datasets. One study suggested the usage of Shift and Truncate (SANT) method²⁸ that performs random shifting of patients’ records and data points within the truncate period are removed to help preserve temporal relations yet ensuring the protection of privacy. Several drawbacks of such approach were mentioned. First, the date shifting method fails to maintain seasonality, and truncation of the data points may result in the loss of potentially clinically useful information. For instance, in the cases of diseases such as new infectious diseases will be largely underrepresented in the dataset, particular when it closes to the time boundary of the dataset²⁸. Therefore, researchers should select the ideal date/time obscuring approach by weighing the limitations it may impose on the potential research questions and balancing between the clinical utility and risk of exposure. In our case, a combination of relative time in minutes along with binary day/night and weekend indicators is more suitable for the initial version of our dataset release, whereby maximizing protection of privacy and preserving the completeness of the patients ‘records. However, people who intend to use the datasets should be mindful of acknowledged limitations beforehand.

Age-based Exclusion

The CONCERN study targets adult patients in the acute care and intensive care settings, thus in the original CONCERN database, we removed patients whose ages at the admission time were under 18. MIMIC-IV had the similar exclusion criteria for the protection of children and ethical considerations. One distinction was that MIMIC removed patients whose age was below 18 at the first visit, while we excluded patients with ages below 18 at the admission; however, they could be added into the cohort once they turned 18. Both MIMIC-IV and CONCERN databases were inapplicable for studying pediatric populations, and findings derived from these data cannot be readily generalized to minors. Beyond that, MIMIC-IV specified the removal of patients under the enhanced protection list to maintain privacy and confidentiality; nevertheless, we perceived some ambiguation in the definition of enhanced protection which is assumed to encompasses a broad spectrum of criteria. To our best knowledge, a standardized enhanced protection list is not available, which usually is curated locally and may vary between different institutions. We note the need to have explicit definitions to guide data scientists and other researchers in their use of these data sets for transparency of scientific research and to drive reproducibility.

In our study, the ages for patients with age above 89 were changed as “age above 89”. Alternatively, the most conservative approach is to remove patients above 89. This is most appropriate especially for a small number of individuals in this cohort meeting such criteria, removing these records can reduce the risk of identity exposure. In our cohort, the number of patients with age above 89 is quite large (above 100,000), thus less likely to identify at the individual level. Hence, we retain these patients’ records at this stage. However, for the future of adding narrative notes on top of the structured information, with the advancement of large language models (LLMs), the joining of all available information sources (e.g., open internet, social media, released medical dataset and clinical documentations) can potentially pinpoint to a subgroup of population. The level of risk should be re-evaluated carefully, considering the evolving capabilities of LLMs and inclusion of new medical data such as images and clinical notes.

Low Occurrence Concepts Removal

We have taken additional steps to enhance the privacy protection by removing concept codes (ICD codes, procedure codes, etc.) with low occurrence within our cohort, at which we denoted these concepts as rare conditions in this dataset. Note that our definition of rare conditions is not limited to the list of rare diseases published by National Institute of Health³⁴. The removed concept terms were determined using a data-driven approach by analyzing their respective the frequency distribution. This procedure was conducted to protect patients with rare diseases and the initial occurrences of unknown diseases, for instance, the first COVID-19 case which was well-documented in the media. It is considered as a more dynamic and flexible strategy, compared to a static list of uncommon conditions. The compromise of such approach entails the loss of the information in the patients’ record, depending on the number of excluded conditions and number of patients affected. Importantly, given this approach, the dataset may not be appropriate for studies focusing on clinical edge cases. Additionally, it is also well-known that data collected from the EHR entails various issues entailing systematic errors, incompleteness and inaccuracy, presenting with significant bias when using for clinical research purposes³⁵. Deleting certain conditions from patients’ records can further augment bias issues when conducting secondary analyses or building predictive models. In the era of LLMs with growing dependence on their powerful capabilities for automating large-scale computational analysis processes which leads to significant reduction in manual efforts, more investigation and regulatory process is necessitated to oversee any unintended consequences and algorithm bias. There has been some ongoing investigation^36,37 to mitigate bias in downstream applications, and the first important step is to identify and understand the kinds of biases that exist in the open data sources commonly used by data scientists as we have aimed to in this paper.

Limitations

This analysis is based on our best current knowledge of potential reidentification risks with the use of LLMs. We expect this knowledge will rapidly evolve and the actual risks may differ than those we describe. Nonetheless, our stance is that a conservative approach is required for data set that are currently being released as those data sets will be publicly available to LLMs in the future.

Conclusion

In summary, we presented our de-identification decisions regarding structured data by comprehensively considering data utility, privacy protection, and risk of re-identification. Although the extent to which LLMs can re-identify or recover individual information based on structured data remains uncertain, we are inclined toward a more conservative stance when making de-identification decisions. The limitations and bias induced by our de-identification approach have been explicitly discussed and acknowledged, and such transparency will be beneficial for future utilization of the datasets.

Figures & Tables

Table 2.

Summary of several de-identification methods, evaluated under three aspects data utility, privacy protection and risk of re-identification.

De-identification methods	Data Utility	Privacy protection	Risk of re-identification
Dealing with Date/times
Remove all times	Low. Temporality is critical in understanding studying patients ‘health status.	High. Removing all timestamps can prevent the inadvertent disclosure of behavior patterns and clinical routines.	Low. Much difficult to link the conditions/procedures to individual.
Random shifting of the dates, and times.	High. Most parts of the temporality are preserved.	Low. One can narrow down the date ranges by observing the patients with data at the end of the shifted28, allowing to make inferences about individuals.	High. Preserving temporal details at fine level can impose high re-identification risk for dataset that has small time span.
Change to relative time	Moderate. Some temporalities can be retained, knowing order of conditions and procedure relative to the admission time.	Moderate. Duration although less likely but can still reflect temporal information via interaction with clinical study periods29.	Moderate. It is still possible to simply leverage the relative times to make some inferences, for instance, combining the relative times and hospital units.
Dealing with Ages
Remove patients age above 89	Moderate. Less representative of the population.	High. Completely exclude information from protected population.	Low. No information can be used to re-identify such group of individuals.
Obscure ages above 89	Moderate. Enable secondary analysis of older population.	Moderate. The cohort population age above 89 is relatively large (above 100,000).	Moderate. Can further identify their identity by combining narrative notes released later.
Dealing with Rare Data
Remove only rare disease diagnostic codes	High. In general, we anticipated prevalence of each rare disease should be less than 1%30.	Low. The list of rare disease codes is incomplete and requires regular, dynamic update.	High. The risk of re-identification is high, when certain information is retained in the chart that can be linked with rare diseases.
Remove all data for patients with any condition that has a low occurrence in the data set	Moderate. Truncation of the dataset may lead to loss of information.	High. All relevant information was removed, thus high protection of privacy.	Low. The complete removal of patients’ record makes it difficult to re-identify them.
Remove only the specific conditions that have low occurrence in the data set, preserving the rest of the patient’s data	Moderate. Less information loss.	High. Rare concept codes such as diagnosis or procedure codes were removed.	Low. Excluding the low occurrence concept codes minimizes the risk of re-identifying individuals.