Privacy and Security Throughout the Health Data Life Cycle: From Primary Care to Research Networks

Abstract

Health data are increasingly generated, shared, and analyzed across an ever-growing collection of settings. While these developments enable new forms of biomedical discovery and clinical decision support, they also introduce evolving privacy, security, and trust challenges that extend beyond traditional regulatory and technical frameworks. In this review, we characterize the various risks and protections throughout the health data life cycle, from data generation and primary use in healthcare to secondary use in research and artificial intelligence (AI) model development. We discuss how regulation, organizational practices, and technological choices shape data protection requirements, and we discuss and contextualize emerging threats, such as incidental disclosures through AI tools. We further review technical approaches for mitigating these risks, including access control and auditing, re-identification risk assessment and statistical mechanisms for risk mitigation (e.g., differential privacy), and synthetic data generation. We also consider how collaboration across disparate organizations may be achieved through federated learning mechanisms and cryptographic technologies, such as secure multiparty computation. Throughout, we highlight trade-offs between privacy protection and data utility, and we articulate practical challenges in deploying these methods at scale. We conclude by identifying open issues for the field, including the need for standardized metrics and greater transparency to support trust in data-driven healthcare and research.

1. INTRODUCTION

The biomedical community continues to innovate and implement technologies that enable the creation, collection, and redistribution of ever-increasing amounts of data related to a patient’s health and their care. At the same time, computational approaches for the analysis of such data have evolved and, as in so many domains, the rapid advancement and proliferation of artificial intelligence (AI), and particularly machine learning, over the past several years has substantially influenced the way health information is processed and reasoned over (1). Still, as AI-based technologies have grown in their popularity, so too have the privacy and security threats to the data they are based on and applied to (2). This article reviews the various ways in which health data are collected, shared, and used and examines how the threat landscape is affected by recently adopted and emerging information technologies. Figure 1 provides a high-level summary of the health data workflow, along with the various risks and protection techniques that are touched on in this review.

Figure 1.

Simplified representation of different stages in the health data flow with potential risks and mitigation strategies. In the data-generation stage, health data are generated by individuals (e.g., patients). In the primary use stage (pink), data are collected for care by healthcare organizations or third-party systems. In the secondary use stage (yellow), data are shared and analyzed to accelerate knowledge discovery.

2. HOW CONTEXT INFLUENCES SECURITY REQUIREMENTS FOR HEALTH DATA

There are numerous ways in which health data can be generated, and this is important to recognize, as context influences which legal and/or policy requirements for data protection are relevant. Here, we outline several for illustrative purposes. First, data about one’s health can be created and documented in the context of routine clinical care, that is, when a patient receives support from licensed medical professionals. In this situation, data are typically stored in one of the many information systems that are managed by a healthcare organization. Examples of such systems include, but are not limited to, electronic health record (EHR) systems, laboratory information management systems, and picture archiving and communication systems. Historically, these systems were managed on-premises, with the computational infrastructure under the control of the healthcare organization, though backups for disaster recovery were stored off-site. However, along with the proliferation of cloud computing environments, as well as partnerships between health information system manufacturers and cloud service providers, health information has increasingly moved into environments managed off-premises, which bring their own set of privacy and security challenges, but ones that affect all domains and are not specific to biomedicine (3). Regardless of where the data are stored, in the United States, these environments are subject to oversight by various laws and regulations, most notably the Health Insurance Portability and Accountability Act of 1996 (HIPAA) (Publ. Law No. 104–191), for which there are two notable supplemental rules. The HIPAA Security Rule of 2003 (45 C.F.R. Part 160 and Subparts A and C of 164) specifies a collection of administrative, physical, and technical safeguards that must be set in place in order for the organization to manage and process such data. By contrast, the HIPAA Privacy Rule of 2000 (45 C.F.R. Part 160 and Subparts A and E of 164) specifies not only the conditions under which covered information can be utilized but also the conditions under which it is no longer considered to be subject to the regulation, such as when data are considered to be de-identified.

Second, health data can be generated outside of a healthcare setting, such as when an individual agrees to be part of a research study. In these situations, data may be collected through various alternative mechanisms, such as a REDCap environment (4) managed by the research team. In such a situation, data that are not collected for the purposes of care are not governed by HIPAA, but other regulations, such as the Common Rule (2018 Revised Common Rule, 45 C.F.R. Part 46, Subpart A), which specifies protections for human subjects in research, are applicable.

Third, it is possible that an individual may generate information on their own, such as through direct-to-consumer genetic testing companies, through wearables (e.g., a smartwatch with an electrocardiogram lead), or by documenting and stating their health status (e.g., via a post on social media). In this case, regulation and thus privacy and security requirements are typically governed by the contracts or terms of service that an individual agrees to with third-party service providers (e.g., the company hosting the social media platform). Yet, as several recent investigations have demonstrated, due to the lack of regulation around them, it is not uncommon for applications in this space, particularly mobile health apps, to include code that collects user data, transmits such data in their traffic, sends traffic to external service providers, and fails to encrypt data in transit (5, 6). In recognition of the limits of regulation for personally generated data about one’s health, various states are beginning to enact laws (7) to specifically cover health-adjacent data that are not collected by organizations subject to HIPAA or the Common Rule, such as the My Health My Data Act in Washington State (House Bill 1155, Chapter 191, Laws of 2023; codified at R.C.W. 19.373) and the Health Data Privacy Act (Nevada Health Data Privacy Act, S.B. 370, 2023) in Nevada. These laws would apply, for instance, to an app dedicated to health information that is personally generated, such as period tracking or measuring an individual’s cognitive reasoning capability. While this review focuses more on the data collected in the context of healthcare and biomedical research, the growing utilization of personal technologies will become an increasingly critical area in need of privacy- and security-enhancing technologies in the near future.

3. SECURITY POLICY AND MODERNIZATION PLANS

In primary care environments (e.g., EHRs), a patient’s information is typically stored in a centralized repository (or database) and is then made accessible to credentialed employees of a healthcare organization. In such a situation, laws like the HIPAA Security Rule define a set of physical, administrative, and technical protections that are either (a) required or (b) addressable. Protections that are considered addressable are expected to be implemented, unless an organization can sufficiently justify why such an implementation is not needed. This has created some degree of contention (8). For instance, the encryption of data in transit is a requirement, whereas the encryption of data at rest is only addressable. The HIPAA Security Rule is now more than 20 years old, and this lack of requirement in the healthcare domain seems to be incongruent with modern threats to our cyber-infrastructure. As has become evident, the past several decades have witnessed substantial growth in the number of data breaches that have occurred (9). Moreover, there is a growing sophistication in ransomware attacks that have been committed against healthcare organizations (10). As an example of the magnitude of such problems, consider Change Healthcare, one of the largest healthcare clearinghouses in the United States, which processes more than 15 billion transactions annually, or approximately one-third of all healthcare insurance claims. When Change Healthcare was subject to a ransomware attack in 2024, the effects were nearly catastrophic for the healthcare industry because the lack of payments to healthcare organizations for their claims significantly hindered their financial viability—so much so that UnitedHealthcare (the parent company of Change Healthcare) and the US Centers for Medicare and Medicaid Services had to extend loans to healthcare organizations at financial risk (11). In recognition of these issues, the Office for Civil Rights at the US Department of Health and Human Services recently issued a Notice of Proposed Rulemaking that aims to strengthen the security requirements for the management of patient data (12). Notably, one of the proposed changes is the elimination of the addressable, such that all of the aspects of the Security Rule would become a requirement.

4. INSIDER THREAT DETECTION

While legal and policy changes are being contemplated, it is important to recognize that protections articulated in the aforementioned regulations typically address routine standard security principles, such as the encryption of data, application of unique logins and passwords for users, and limitations on the accessibility of the physical machines where data are stored or accessed. Still, there are limits to the protections that such principles can support. For instance, one of the foundational principles of computer security is that of least privilege, which indicates that data, or the functions one can execute over the data, should be limited only to those who need access to such information to perform the responsibilities of their job. Over the past several years, concerns over conflicting perspectives in what constitutes healthcare, particularly over the Supreme Court’s ruling in Dobbs v. Jackson’s Women’s Health Organization, have led to the development of frameworks to segment medical records into various sections, such that health information considered to be sensitive (e.g., reproductive care, transgender care, or substance abuse) is more easily controlled and provisioned on a need-to-know basis (13–15).

While segmentation can be formally designed and engineered, there are challenges to applying segmentation in EHRs in practice. One issue arises from the fact that EHR systems are increasingly interoperable across healthcare organizations. As such, it is possible that a patient’s information could be transferred from an EHR system under the management of one organization to that of another organization with different perspectives on what is considered sensitive (16). Another challenge is that healthcare operations are fluid and composed of many different workflows, which makes it difficult to define who specifically, or even what type of care provider, will need access to which information and when. As such, over time, healthcare organizations have increasingly adopted technologies that attempt to detect and deter insider threats. In this scenario, authenticated users of the EHR attempt to access and potentially misuse patient records, though they lack a legitimate reason to do so.

Approaches to detect such behavior have been driven by rules defined by managers, and machine learning classifiers have been developed to build on such rules to predict when an access request is seemingly justified or suspicious (17–19). In recognition of the fact that healthcare is a dynamic environment, where each patient has a unique set of circumstances, it has been suggested that the detection of suspicious behavior should be supplemented through unsupervised learning strategies that attempt to characterize whether a user’s actions are anomalous and thus of interest to security staff (20), along with automatically generated explanations to assist in investigations (21). Still, it has recently been shown that when such detection mechanisms are rooted in machine learning, EHR users can, under the right circumstances, take evasive maneuvers to mask their behavior and fool the auditing tool (22). In anticipation of such a behavior, it has been suggested that healthcare organizations could formally model the insider threat mitigation problem as a security game, where one set of players are the security staff members of the organization while a second set of players are the end users of EHR system (23). Given a limited budget, the problem can then be represented as a Stackelberg game, also known as a leader–follower game, where the limited security controls can be optimally allocated to mitigate the likelihood of a successful misuse (24, 25).

5. INCIDENTAL DISCLOSURES

It is important to recognize that the insider threat problem is one in which end users are expected to be oriented toward a misuse of a patient’s data. Yet it is increasingly the case that the users of a patient’s data are not intentionally jeopardizing the privacy or security of the data. As suggested earlier, AI technologies in the context of primary care pose a new type of threat. Large language models (LLMs) have had a major influence over the way patients, healthcare professionals, and biomedical scientists interact with health data. However, not all LLMs are built or managed in the same manner, an aspect that nontrivially influences the way in which they are utilized and the threats that influence them (26).

Like EHR systems, LLMs, along with their applications in chatbots, can be deployed in secured healthcare environments. If the language model is relatively small, it may even be managed on computational infrastructure controlled by the healthcare organization. However, many commercial-grade LLMs have grown to sizes that are substantially larger than what a typical healthcare organization has the ability to manage. Additionally, the creators of such LLMs may have an interest in keeping the technology in a proprietary manner, such that the healthcare organization is not able to access the parameters of the system directly but rather can query a server hosted by the owner of the technology. In this situation, organizations need to take extra care because if there is no contractual agreement of confidentiality, such as what might be specified in a business associate agreement, then the data sent to the LLM service may be reused by the service provider in a manner that is incongruent to the use of the data permitted under HIPAA (27). Thus, various service providers have established secured HIPAA-compliant versions of their cloud environments available for clinicians and researchers to process health data in a manner such that the data are not reused by the service provider; these are increasingly utilized by medical centers in the United States (28).

Still, secured cloud-based computing environments are not free, and many organizations lack the budget necessary to utilize them. This is a concern for several reasons. First, if LLMs are not made accessible within a secured space, then it should be anticipated that some people will move health data into unsecured spaces, potentially unknowingly. While employees of healthcare organizations are continually educated on the importance of not divulging patient information in unsecured settings, the same cannot be said about patients and their informal caregivers. And, as LLMs become commonplace to everyday use, it should not be surprising when patients submit their health information into unsecured environments. This may also happen through encouragement of the makers of such technologies, such as when the team behind the popular LLM Grok encouraged people to submit medical images to the system (29, 30). Even without such encouragement, patients may choose to use LLMs, and particularly chatbots, out of convenience. For instance, patients who are not native speakers of the language of their care providers may take communications (e.g., patient portal messages) out of secured EHR environments and send them to an LLM to have the message translated into their own language and/or summarized in a form that they can more easily understand. Yet, in doing so, the patient would be waiving their rights to HIPAA protection since the LLM service provider is not covered by the regulation.

Moreover, this could happen when individuals think that they are interacting with a trusted service provider, but instead data are shared with an untrusted LLM service provider. An illustration of such a problem was when the users of the kokobot service from the mental health support organization Koko were not informed that the organization was passing the queries on to ChatGPT (31). In addition to concerns over privacy, this led to concerns about trust in technology and healthcare.

This leads to the question of whether an organization could minimize the risks of exposing patient identifiers to the service providers of such systems. One straightforward approach to addressing the risk would be to de-identify the data before sending them to the LLM for processing. If the data are well-structured, then it is straightforward to detect and amend patient identifiers to mitigate the risk of re-identification to an acceptable level. This becomes more challenging when the data are in the form of natural language, such as what one would come across in clinical notes or patient–provider messaging platforms. Still, there are many natural language de-identification methodologies that have been developed, many of which take advantage of state-of-the-art AI methodologies (32, 33). While they may not be 100% capable of detecting and removing all direct patient identifiers and quasi-identifying demographics, they may provide sufficient mitigation of re-identification. At the same time, one must consider how such amendment to the data influences the capabilities of an AI technology. As one recent study demonstrated, the more heavy-handed the amendment made to the demographics of a patient’s record, the less likely an LLM will be able to correctly diagnose a patient or provide an appropriate treatment recommendation (34).

6. MEMORIZATION THREATS TO BIOMEDICAL ARTIFICIAL INTELLIGENCE MODELS

The previous section considered situations for how data may be misused or leaked in the context of primary care, yet there are many situations when a healthcare organization or biomedical researcher may be asked, or required, to make health data available beyond their initial point of collection. For instance, when a scientist publishes a manuscript, a journal may request them to make data available in a manner that substantiates the claims made within to ensure reproducibility (35). In other situations, it may be that an organization or researcher aims to be altruistic and support unknown future investigations and/or development endeavors by making a new dataset available to the world.

Earlier, we posed the situation in which an LLM has already been developed and made available to end users. However, new LLMs are constantly under development. Some are based on new computational architectures, such that they can be trained on the same data as their predecessor LLMs. Yet many new LLMs are built entirely from, or fine-tuned with, new data. To date, much of the data used to develop LLMs, including those that have been specialized for various biomedical applications, have come from the public domain, but these data are limited in what they can communicate about humans and the healthcare they receive. As such, there is growing interest in using data from nonpublic environments, such as those stored in the information systems of healthcare organizations. An illustration of such a language model is GatorTron, a language model built on more than 90 billion words from University of Florida EHR data (36). However, there are concerns with supplying private data to LLMs, as it has been shown that they have the potential to memorize the information that they are trained on, initially with LLMs like GPT-2 (37) and then with LLMs like ChatGPT (38). This is not guaranteed, but it has been demonstrated that well-constructed (yet esoteric) queries can enable the user of an LLM to have an LLM regurgitate partial, and at times entire, training instances that it was based upon. This is likely due to the fact that LLMs are designed to incorporate millions to billions of parameters, such that the models can overfit the training data, unlike more traditional machine learning frameworks. Techniques are emerging on how to mitigate memorization (39), but there is no perfect solution yet.

It should also be recognized that the threats to such AI systems are not limited to LLMs per se. Let us consider several other machine learning architectures. The first type of architecture is YOLO (you only look once), an approach for object detection that was introduced by Redmon and colleagues (40). This model is based on a convolutional neural network (CNN), which takes an input image such as a chest X-ray and then continually compresses the image into smaller and smaller layers, allowing for the discovery of relationships between different parts of the image. A second type of architecture is ResNet, a version of a residual network, an approach for object recognition that was introduced by He and colleagues (41). This model, like the CNN used by YOLO, passes data through a series of convolutional layers and allows for jumps between layers at times to account for nonlinearities in the data [though rectified linear units (ReLUs)]. One of the more popular of these systems is ResNet-50, which consists of 50 layers in the architecture.

Notably, both YOLO and ResNet-50 apply compression techniques, so that when training the models, the input images are converted into convolutional, or compressed, representations. However, this convolution does not mean that the models are impregnable to attack. Notably, there are several different types of attacks that have been developed and applied to these types of architectures. The first is a membership inference attack (42), where the recipient of the model constructs multiple shadow models to mimic the behavior of the model and infers whether the data of a target individual were included among the training samples. This type of attack has been adapted to CNN-based object recognition technologies, including YOLO and ResNet (43). However, this type of intrusion requires that the adversary already has access to partial information about the original training data to effectively construct the shadow models. A second type of attack is an attribute inference attack (44), where the recipient knows certain factors about a targeted individual or image and attempts to learn new knowledge based on the model parameters. Again, this type of attack presupposes that the adversary is in possession of some subset of the training data.

A third attack that has been committed against these architectures is the model inversion attack, where the machine learning model parameters are exploited to attempt to determine the original values of the training data. Like membership intrusion attacks, the inversion attack has been adapted to object recognition frameworks. An illustration of model inversion against ResNet-34 with images is depicted clearly in Reference 45. State-of-the-art techniques for inversion have been shown to achieve relatively high accuracy in such situations—on the order of 70% or better.

At the same time, it needs to be recognized that these attacks are limited in several ways. First, the majority of existing attacks have been demonstrated with handwriting of letters (or numbers) and facial images, where the similarity between the original and reconstructed images is readily apparent to a human reviewer. There have been several evaluations of model inversion against medical images, such as brain magnetic resonance imaging scans (46) and mammograms (47), demonstrating that reconstruction can be accomplished. However, these investigations are not generalizable to more generic attacks, as all of the images were highly similar in their pose, contrast, and clinical phenomena. When the images become more diverse, the findings are less reliable and, while the type of image (e.g., a brain scan) can be detected, the specific details of the image are less reliable, as shown in a brain reconstruction attack from Wu and colleagues (47), where there are different slices of the brain incorporated into the training data. There is further evidence that as other types of images are incorporated, the reconstruction reliability is severely diminished (48).

7. PROTECTIONS FOR PUBLISHING DATA FROM A SINGLE SITE

The previous section illustrates how data can be subject to risk, even if they are integrated into a large AI framework. So, if that is the case, then how do we mitigate the risk?

7.1. Re-identification Risk Assessment and De-identification

One of the most popular approaches to privacy protection when sharing data beyond the initial point of collection, which is supported through HIPAA, the Common Rule, and the European Union’s General Data Protection Regulation, is through de-identification strategies. While it is obvious that direct identifiers, like the name of a patient, need to be obscured or removed before sharing health data, it is also the case that indirect identifiers can be used to re-identify an individual as well. As an example, the basic demographics of a patient, such as date of birth, gender, and ZIP code of residence, are expected to be unique for over half of the US population (49, 50), and as the number of factors in the set of indirect identifiers grows, so does the chance that an individual is unique (51, 52). Moreover, such information can be found in various resources that are publicly accessible, such as voter registration lists (53), which are required to be made accessible to the public in various capacities depending on the US state in which they are based. Indirect identifiers are not limited to a patient’s demographics, as it has been shown that under the right circumstances, various aspects of a patient’s health status or experience can be leveraged for linkage purposes. Such indirect identifiers include survey responses (54), standardized diagnosis codes (55), and particularly details corresponding to publicly reportable events, such as those reported in court records or motor vehicle accidents (56). More recently, it has also been shown that people who choose to participate in research studies may self-disclose their information, as well as their participation status in such a study, thus creating another quasi-identifier that might be leveraged by a data recipient for re-identification purposes (57).

Based on such threats, various approaches have been developed to assess re-identification risk and sufficiently mitigate it by building an adversarial model that ensure the data are shared in a manner that accounts for such an adversary. In this sense, one would model the ways in which a plausible adversary (recipient of the data) would attempt to perform a re-identification. In such a situation, one would categorize the data under consideration into (a) types of data that directly identify an individual (e.g., personal name or Social Security number), (b) types of data that indirectly identify an individual (e.g., date of birth, gender, and geographic area of residence), and (c) data that do not have the potential to directly or indirectly identify an individual (58–60). There are debates over how to determine which types of data fall into which camp (61), but assuming such a categorization can be accomplished, next the data can be amended to satisfy a formal protection model. This may be accomplished through approaches that generalize and/or randomize the data.

For instance, one line of strategies relies on the notion of k-based approaches. In this situation, data are shared when it can be shown that a record corresponds to no fewer than k people. Typically, this situation is framed from the perspective of a sample of data to be shared and a population from which it is derived, which includes the sample, the relationship of which can greatly influences risk measurements (62). In this case, if the recipient is expected to know that a patient is in the sample, then the goal is to ensure that, for every record in the same shared dataset, there are k – 1 other records that appear indistinguishable based on the set of indirect identifiers. This is what is often referred to as k-anonymity (63, 64). Recently, this strategy was adopted by the National Center for Health Statistics at the US Centers for Disease Control and Prevention to share COVID-19 statistics during the pandemic (65).

It has also been suggested that the number of people who will present with symptoms can be forecasted to allow for generalization or suppression policies to be adapted before the actual events transpire (66). By contrast, if the recipient holds weaker knowledge, such that they do not know who from the population is in the sample, then one only needs to show that there are at least k people in the population that are indistinguishable from the sample. This is a premise that is known as k-map (67). At the same time, it should be recognized that such a data protection framework assumes that the data will be made available to anyone without any additional controls. However, there are many situations in which other controls can be put in place, which is useful for when there are a nontrivial number of quasi-identifying factors. For instance, the National Institutes of Health’s All of Us Research Program (68) requires that all data be maintained in a sandboxed environment. All users must undergo training and enter into a contractual use agreement with the management of the program, and their affiliated institutions must also enter into an agreement with the program (69). Furthermore, given these controls, the program generalized and suppressed various demographic factors and showed that the chance that a recipient could use a reasonable number of quasi-identifying factors to identify a participant was below an acceptable threshold (70).

7.2. Provably Private Randomized Methods

Potential vulnerabilities in the presence of an informed adversary and practical challenges (e.g., computational cost) are among the main limiting factors of traditional k-based (or generalization) approaches. As an alternative to them, strategies based on randomization that are built on provable mathematical principles have gained in popularity. Differential privacy (DP) is a rigorous mathematical model that provides provable privacy protection to individuals within a dataset when aggregate information is shared or analyzed (71). That is, the degree of privacy can be controlled from low (i.e., no privacy) to extremely high (i.e., perfect privacy), with a corresponding utility price on the released data by tuning the values of the privacy parameter (ϵ). Compared to k-based anonymization methods, DP does not make assumptions about the background information or attack models a malicious user may have, providing improved privacy resilience. The core principle of DP is based on the premise that the output derived from a DP method should be indistinguishable whether any single individual’s data are included in the dataset or not. This privacy definition is independent of the actual value of individual data, and it only depends on the computation (e.g., finding the mean, training a deep learning model). Over the last decade, DP approaches have been developed to address privacy risks in a variety of tasks in health applications.

Among the main applications of DP is the sharing of aggregate statistics (e.g., the number of patients who exhibit specific conditions of interest, reporting summary statistics, and performing statistical tests) collected by a trusted aggregator (e.g., healthcare provider). While sharing these aggregate-level results may seem harmless to privacy, researchers have shown that information about the contributing individuals may be disclosed (e.g., re-identification, membership inference, and feature reconstruction) in some circumstances (e.g., the presence of an informed adversary). DP techniques can alleviate these privacy risks. A first implementation of DP for aggregate data sharing has been proposed by McSherry (72) with privacy integrated queries (PINQ). PINQ provides DP implementations of common data transformations and aggregations to perform a variety of analytics including clustering, classification, and contingency tables. The use of DP for sharing aggregate statistics is central in genomic applications, where privacy risks are heightened by the sensitive nature of genomic data (73). This is particularly acute in the context of genome-wide association studies (GWASs), where the release of aggregate statistics can enable membership inference attacks, revealing the participation of a target in the data (74, 75). DP methods for GWASs provide provable privacy protection against these attacks. These approaches analyze the contribution (sensitivity) of each data record on the reported GWAS statistics and carefully perturb the exact output results with calibrated random noise to provide protection in the sharing of minor allele frequencies, chi-squared test, and p-value statistics (76–78). Beyond GWASs, DP solutions have also been integrated into survival analysis tasks to enable the sharing of survival results (e.g., survival curves) obtained with estimation methods (e.g., Kaplan–Meier). DP approaches inject noise to the aggregated survival curve before sharing, protecting the counts of events and individuals at risk for each time point (79, 80). A challenge is to ensure that the shared survival curves are clinically useful (e.g., monotonically decreasing), which requires a careful analysis of the utility–privacy trade-off. In a broader context, the approaches have been designed to support the sharing of aggregate population statistics. For example, the US Census Bureau has carefully incorporated DP techniques into their disclosure avoidance system for the release of the 2020 Census Data (81). While the use of DP methods enhances privacy protection by design, the usability of the protected data has been under scrutiny (82, 83).

Extensions of the DP model (e.g., local DP) have been proposed to alleviate some of the privacy risks in sharing data with untrusted service providers. The key idea is to generate a perturbed version of the original individual-level data using randomized response techniques prior to sharing (84). As a simple example, consider the collection of individually generated responses (e.g., Do you have diabetes?). Local DP randomizes the answer yes/no based on a probability, ensuring that the true values are protected while retaining utility of the aggregate statistics. Under the local privacy model, stronger privacy control is provided to individual data contributors, enabling them to protect local data without relying on a trusted data curator. Applications of local DP can be found in real-world settings, including in systems developed by major companies (e.g., Apple iOS, Google Chrome) (85). At the same time, balancing strong privacy protection and utility at the local level is a challenging problem, especially for nonbinary data types. Recent studies have proposed several extensions to improve utility by restricting the privacy only on sensitive private data and exploring the use relaxations of the privacy constraints. Promising results of local DP methods have been shown in several health applications, including genomic and EHR data sharing (86, 87).

More recently, DP has been at the forefront of privacy-protecting technologies for machine learning tasks to mitigate privacy inference risks that may arise in the sharing of AI models trained on sensitive data. Abadi et al. (88) proposed a DP stochastic gradient descent approach (DP-SGD), which protects individuals during training of deep learning models by scaling and perturbing the gradient of the loss function at each epoch. DP-SGD has created new avenues for achieving rigorous privacy protection in numerous AI-driven health tasks. As one example, Beaulieu-Jones et al. (89) proposed a new data synthesis approach that combines generative adversarial networks (GANs) with DP to generate synthetic longitudinal clinical data. A recent study investigated the applicability of DP for supporting deep learning tasks in medical imaging, demonstrating the ability of DP to defend against reconstruction attacks even with large privacy budgets, while noticing only negligible drops in performance (90). Additionally, Ziller et al. (91) provided an open-source framework for supporting DP deep learning tasks in medical imaging, including disease classification and semantic segmentation. Researchers have also explored the applicability of DP-SGD in enhancing privacy protections for LLM applications. For example, Li et al. (92) investigated new optimization techniques to improve the balance between model performance and privacy protections for LLM models (GPT-2). Recently, Google introduced VaultGemma, a model with one billion (1B) parameters trained using DP-SGD, which provides high-utility capabilities while protecting the privacy of training samples (93). Despite these promising results, training a large-scale LLM (e.g., ≥20B parameters) with DP would require collecting significantly larger datasets than are currently used in its nonprivate counterparts (94).

Over the years, DP has evolved from a theoretical privacy principle to a set of more user-friendly techniques, with numerous tools that can be seamlessly integrated into state-of-the-art AI analysis pipelines (e.g., TensorFlow Privacy and PyTorch Opacus). The rigorous privacy guarantees provided by DP make it one of the most powerful privacy frameworks today capable of achieving core principles in current data privacy policies (e.g., the European Union’s General Data Protection Regulation). At the same time, there are some important limitations. First, data correlation may weaken the privacy protection provided by DP, resulting in privacy violations. This problem is particularly critical in health applications, where blood relatives may share common health conditions. Recent studies have investigated extensions of DP to account for correlated data (95). Second, DP protects against a strong informed adversary who may have access to all records but one, requiring a large perturbation noise to satisfy privacy, which may diminish utility. This can be overly strong in practical settings, as protected data in healthcare organizations are unlikely to be exposed. Recent efforts have been made both toward proposing extensions of DP that can provide a more practical formulation of privacy requirements (96, 97) and understanding the balance between privacy protection versus practical privacy risk (98).

7.3. Synthetic Data

Despite the merits of de-identification, anonymization, and DP methods in mitigating privacy risks for data sharing, these approaches are not without limitations. First, while de-identification and anonymization, when performed appropriately, can largely reduce the likelihood of re-identification, the link between a record and a real person may be still present. This implies that an adversary with strong background knowledge about a person or a cohort can still succeed in re-identifying their records. Second, all these privacy techniques face inherent trade-offs between privacy and data utility: As the privacy protection strengthens, the fidelity and statistical usefulness of the shared data often decline. Importantly, this can lead to the situation in which rare subpopulations can be disproportionately distorted (99), thereby reducing the visibility of minoritized groups in analyses. In particular, in high-dimensional healthcare datasets, such as EHR data, medical imaging, or clinical audio, the complex relationships across features can easily be weakened or destroyed. As a consequence, the processed data can no longer reliably support key downstream tasks, including predictive modeling, risk stratification, and subgroup discovery. Third, there are practical and methodological challenges that can invalidate protections in practice. These include, but are not limited to, selecting and justifying DP budgets, degradation of privacy guarantees under composition when there are multiple releases, and increasing linkage risk over time as additional external data sources become available. These limitations motivate alternative strategies to balance privacy protection with analytical utility.

The notion of synthetic data has been promoted as an important health data sharing strategy, easing the tension between preserving privacy and maintaining utility (100, 101). With a successful history of more than three decades in aiding federal statistical agencies with individual-level data publication (102, 103), this research area has achieved a revival in the age of generative AI, where advanced AI methods, such as GAN-based (104–107), language model–based (108), and diffusion-based models (109), are broadly employed to produce cross-sectional, longitudinal, and medical imaging data for various tasks (110–114). These methods generally involve first training a model on real data to learn complex dependencies and then generating synthetic data from the trained model that closely resemble the nature of real data while upholding privacy (115, 116). In the scenario of fully synthetic data, where all features are synthetically generated, the link between records and real persons is broken, such that re-identification is no longer a valid concern. A series of recent studies has demonstrated that synthetic data, when generated appropriately, can replicate the results of using real data in diverse applications, such as geotemporal epidemic forecasting (112), clinical trial analysis (117), outcome prediction (118), and DNA–disease association research (119). Also, multiple pieces of evidence have shown that synthetic data can assist medical AI in achieving higher performance by augmenting imbalanced datasets for medical image understanding (120) and disease diagnosis (121).

The potential for synthetic data extends beyond the support of reproducibility of investigations and data augmentation for novel studies. Notably, synthetic data have demonstrated promise for the support of de novo discovery by enabling counterfactual and large-scale hypothesis generation that is infeasible under conventional cohort sizes and data-sharing constraints. For example, GAN-based clinical trial digital twins, conditioned on external trial populations, have been used to probe treatment effect modification and to explain discordant findings across clinical trials via transportability analyses (122). Moreover, virtual patient trial simulations have been leveraged to assess broad biomarker panels and prioritize early on-treatment and less-invasive biomarkers for immunotherapy response (123). Synthetic data can also facilitate data sharing by allowing prospective users to rapidly understand what a health dataset contains and whether it is suitable for their needs, generate preliminary hypotheses, and determine whether to pursue access to the underlying real-world data that may require substantial administrative lead time. Nevertheless, discoveries derived from synthetic data should be treated as exploratory and must undergo rigorous validation in real-world settings before drawing substantive clinical or causal conclusions.

By contrast, partially synthetic data can be generated by conditioning on selected features, such as demographics and/or key clinical conditions. In this setting, while the remaining features are newly generated, the dataset retains some degree of risk, since the preserved features maintain a relatively stronger link to real individuals compared to fully synthetic data. A related emerging practice in this setting is the use of personalized treatment assessment and optimization prior to clinical execution, a core concept in medical digital twins, which hinge on generating synthetic health trajectories based on patients’ medical history to explore counterfactual or what-if questions for optimized care (124, 125). All of these benefits of synthetic health data position them as a compelling resource for the research community by supporting hypothesis generation, enabling the preliminary testing of novel ideas, and facilitating broader knowledge dissemination with substantially reduced concerns about compromising patient privacy. Due to the privacy-preserving nature of synthetic health data, a growing number of medical research centers have chosen to waive the requirement for normal ethical review when these data are used in research (126).

Despite the privacy-preserving nature of synthetic data (particularly against re-identification), this approach does not imply zero privacy risks in data sharing. Privacy vulnerabilities may still arise if the synthetic data generation model overfits the real training data, such that the produced synthetic data replicate or even memorize some of real patients’ features. In such cases, synthetic data recipients could potentially infer sensitive information about real-world identities when combined with external data sources. Both membership inference attacks and attribute inference attacks described previously are also valid in the setting of sharing synthetic data. A membership inference attack occurs when an adversary determines whether a complete record in their possession was part of the training data of a synthetic data generation model (127), whereas an attribute inference attack happens when an adversary leverages the released synthetic data to predict unknown sensitive features of an individual based on partial knowledge of that individual’s record (104). In addition, poorly generated synthetic data can also allow identity inference attack (or identity disclosure), where an individual’s identity can be linked to a specific record, thereby revealing that person’s association with the data. Although membership and attribute inference risks have been extensively explored, effective measures for identity inference attacks are still lacking (128), particularly considering that similarity-based measures alone fail to provide reliable privacy risks (129).

8. PROTECTIONS FOR LEARNING FROM MULTIPLE SITES

The previous sections assumed that a single organization was sharing data. Yet it is increasingly the case that multiple organizations aim at collecting large amounts of data on different populations or modalities to enable powerful AI-driven analysis. At times, these organizations can exchange such data and collaboratively study them, but this may not be possible, particularly at the level of detail needed to run a novel investigation. As such, a number of methods have been developed to support the exchange of biomedical information in a way that enables the composition of analytical models without disclosing specific records.

One approach to doing so is through federated learning (130, 131). In a federated learning framework, a set of organizations initially agree upon the type of model they wish to learn. For instance, the model could be as simple as a linear or logistic regression or as complex as a deep neural network. Based on the model architecture, each site would then estimate the coefficients for each of the variables in the model or the edge weights of the network. These coefficients would then be exchanged, typically with a single orchestrating organization to minimize bandwidth. The centralizing organization would then compose a single model (e.g., for each parameter, they can take the average of each organization’s submission) from the various submissions and send the resulting model back to each organization. The organizations would then apply the centralized model to their data and update their local model’s parameters, which again are sent to the centralizing organization. This process will iterate until convergence of the centralized model. There are numerous ways in which federated models can be composed, each with benefits and drawbacks. We refer the reader to the survey by Wen and colleagues (132) for further details on federated learning and the articles by Guan and colleagues (133) and Li and colleagues (134) for specific illustrations with medical images and clinical data, respectively.

It should be recognized, however, that federated learning is not without limitations. First, while exchanging aggregated information, it has been shown that, at times, the parameters associated with the gradients of machine learning models can be inverted to disclose the original information they correspond to (i.e., gradient leakage) (135). This problem is accentuated when the centralizing organization is able to monitor how the gradient changes from one iteration to the next. Notably, though, it has been shown that the iterative process is unnecessary for certain models, such that only a single submission from each site is needed to build the model (136). This line of research holds promise, as it can both reduce the computational time required to achieve the final model and mitigate risk of the aforementioned inversion attack.

Still, a second limitation of federated learning is that, by design, it exchanges only aggregated information. This prevents comparisons and contrasts between records and can limit what can be learned. To overcome this limitation, techniques derived from the cryptography community offer a notable opportunity. One broad class of such techniques, referred to as secure multiparty computation (SMC), enables organizations to encrypt each of their records, such that they can exchange such encryptions and collaboratively learn relationships that exist within the data. A proper treatment of this topic is beyond the scope of this article, so we refer the reader to Lindell (137) for an overview of methods and to illustrative examples of SMC for secure GWASs (138) and discovery of genetic relatives (139).

9. OPEN ISSUES AND THE FUTURE

This article highlights the various ways in which health data are collected, shared, and utilized, along with how emerging technologies can both intrude and protect them from intrusions. Still, there are numerous open issues in the space of health data privacy and security. Here, we take a moment to highlight several.

9.1. The Protection–Usability Trade-Off

The development of data protection techniques should be oriented toward preserving data utility, especially in clinical care and research practices, where poor data quality may diminish trust and introduce patient safety risks (e.g., medical errors). However, it is often the case that privacyenhancing techniques introduce a trade-off between data utility and privacy protection. For instance, access control frameworks for limiting access to the minimal amount of information needed to support healthcare operations are challenging to implement, as it is unclear what information is not needed. Moreover, as AI becomes increasing ingrained in healthcare operations, it is possible that such systems will rely upon as much data as is possible to maximize the likelihood of detecting a pattern and providing an appropriate diagnosis or treatment recommendation. As another example, DP may require a large amount of noise for stronger privacy, diminishing the utility of data and models. Retaining utility is particularly challenging in the presence of high-dimensional data. Generalization techniques to achieve k-anonymity have been shown to result in data that are practically useless, while data synthesis methods may struggle to preserve the original relationships between data variables.

9.2. Implementation Challenges

Broadening the adoption of data protection techniques in healthcare settings requires addressing practical challenges in their integration with existing workflows, data processing techniques, and analysis pipelines. For instance, in the context of game-theoretic strategies for auditing, it is unclear how to translate a healthcare organization’s budget in a quantified amount that can be relied upon to optimally allocate resources for protection. As another example, security-based approaches for privacy (e.g., SMC) tend to be computationally intense, requiring organizations to have advanced computational infrastructures. By contrast, methods like federated learning have shown better scalability, but they often rely on an implicit assumption of synchronization, in which all participating sites are expected to coordinate their training cycle in a timely manner. In reality, it is challenging to ensure synchronization, as computational devices across sites and communication bandwidth are highly variable.

9.3. Lack of Standards for Measuring Privacy Protection and Utility

The lack of privacy and utility standards makes it almost impossible to navigate the current privacy landscape for several reasons. First, different techniques provide different definitions of privacy, which makes it difficult to find consensus and benchmarks across these solutions. General rules of thumb have been proposed for guiding the selection of privacy budgets for DP, but further studies are needed to understand whether the provided privacy protection is adequate with respect to application-specific settings. Second, standardized metrics for measuring the impact of privacy techniques are central in assessing whether the protected data and models are still useful. While certain studies have examined the utility implications of privacy techniques, current investigations are often focused on simple settings where evaluation metrics are narrowly defined. Additional evaluations and the availability of public datasets resembling practical health application settings could help facilitate the benchmarking of current privacy methods.

9.4. Trust and Transparency Challenges

Building trust and promoting transparency are key in the adoption of privacy methods, but there are various challenges. First, privacy techniques (e.g., DP, federated learning) are built on complex concepts that are often difficult to explain to data contributors and organizations. Research aimed at improving privacy explainability and supporting verifiability is central in improving transparency and building trust. Second, data contributors may be unaware of possible privacy risks. For example, attacks on AI applications are continuously evolving, making currently AI training practices potentially vulnerable to future risks. Additionally, data, once shared, could be reused in unforeseen ways, for example, training an AI model that is then shared for commercial use. Addressing these challenges requires solutions that consider a host of intersecting factors, including technical methods for data provenance and better informed consent mechanisms.