Privacy preservation for federated learning in health care

Summary

Artificial intelligence (AI) shows potential to improve health care by leveraging data to build models that can inform clinical workflows. However, access to large quantities of diverse data is needed to develop robust generalizable models. Data sharing across institutions is not always feasible due to legal, security, and privacy concerns. Federated learning (FL) allows for multi-institutional training of AI models, obviating data sharing, albeit with different security and privacy concerns. Specifically, insights exchanged during FL can leak information about institutional data. In addition, FL can introduce issues when there is limited trust among the entities performing the compute. With the growing adoption of FL in health care, it is imperative to elucidate the potential risks. We thus summarize privacy-preserving FL literature in this work with special regard to health care. We draw attention to threats and review mitigation approaches. We anticipate this review to become a health-care researcher’s guide to security and privacy in FL.

Graphical abstract

The bigger picture

Significant improvements can be made to clinical AI applications when multiple health-care institutions collaborate to build models that leverage large and diverse datasets. Federated learning (FL) provides a method for such AI model training, where each institution shares only model updates derived from their private training data, rather than the explicit patient data. This has been demonstrated to advance the state of the art for many clinical AI applications. However, open and persistent federations bring up the question of trust, and model updates have raised considerations of possible information leakage. Prior work has gone into understanding the inherent privacy risks and into developing mitigation techniques. Focusing on FL in health care, we review the privacy risks and the costs and limitations associated with state-of-the-art mitigations. We hope to provide a guide to health-care researchers seeking to engage in FL as a new paradigm of secure and private collaborative AI.

AI can enhance health care by using data to create useful models. However, sharing data between institutions is challenging due to legal and privacy issues. Federated learning (FL) allows institutions to train AI models without sharing data, but it also has its own security concerns. As FL becomes more commonplace in health care, it is crucial to understand its risks. This work reviews the literature on privacy-preserving FL, highlighting threats and solutions, aiming to guide health-care researchers on FL’s security and privacy aspects.

Introduction

The health-care domain has always dealt with privacy concerns and threats due to the sensitive nature of the information in the domain. For example, there have always been unauthorized access to medical records (which can be mitigated by requiring string access controls, user authentication, and audit logs),¹ insider threats that lead to misuse or inappropriate disclosure of health-care data (which can be mitigated by specific training, implementing policies that allow data access to employees where it is needed, and monitoring access to data),² and data breaches and/or cyber attacks (which can be mitigated by encrypting data, implementing robust network security policies, and regular security monitoring).³ Although these challenges are critical, they have been documented and identified and are well studied. For the purposes of this review, we will be focusing on the use of advanced computational techniques in health care, where the privacy issues are more nuanced and their associated mitigation strategies are not that well studied compared to other fields.⁴ The use of such tools (such as artificial intelligence [AI]) can make addressing these concerns more complicated due to the possibility that interactions with the application may leak information about the data used to train it.³ Since health-care data are almost always tied with specific regulatory provisions,^5,6,7 privacy concerns of AI applications in this domain need a deeper understanding of the technical issues at hand, especially to provide guidelines for computational researchers developing algorithms and solutions in this field. This article aims to provide privacy and security guidelines for both computational researchers developing AI solutions in health care and regulatory authorities, so that they are mindful of both traditional information security concerns.

AI approaches have shown great promise for augmenting clinical workflows.⁸ However, large and diverse datasets are required to train robust and generalizable AI models for clinical applications.^{9,10,11,12,13,14,15} One method to acquire sufficient data is through multi-institutional collaborations, currently following a paradigm of centrally sharing data, also known as “data pooling”^{16,17,18,19,20} (Figure 1A). However, such centralized data collection is not always possible due to various factors, such as patient privacy concerns, prohibitive costs of central data management and storage, and institutional or even regional data-sharing policies.^21,22 Federated learning (FL) is an alternative approach to the data pooling paradigm for multi-institutional collaborations that begins to address some privacy concerns, since model learning is performed locally at each institution and only the resulting local model parameters are shared (Figure 1B).

Figure 1.

Illustration of different collaborative learning approaches

(A) Using a data-sharing paradigm, where data from the three individual data custodians are shared to a central data aggregation site for training and (B) using federated learning, where the training happens at each individual data custodian and only the model updates are shared to a secure aggregation server for combination.

Although FL allows for training an AI model across private data without requiring that data to be shared, there are still questions that remain regarding the need and the way one needs to protect against leakage of information about these private data via the model updates shared throughout the FL workflow. There is hope that through the incorporation of additional security and privacy technologies into FL, a level of security and privacy can be achieved that will enable a greater degree of trust in the resulting federation. Many factors in addition to trust can prevent institutions from participating in FL training, such as coordination and overhead of data preparation, institutional information security, and compute hardware requirements. However, the use of more secure and private FL frameworks toward increasing trust in the system is expected to enable a more diverse collection of clinical institutions willing participate in FL projects. The models that result from such collaborations can potentially benefit from the data diversity, resulting in better model generalizability. In particular, secure and private FL has hope to greatly benefit collaborations in domains with stringent policies around data sharing, such as is the case for health care.^{14,21,22,23,24,25,26,27,28} Some literature exists reporting on general vulnerabilities of FL^29,30 and even exploring privacy and security concepts related to FL,^31,32,33 albeit without providing a focus on their implications in the health-care domain. We further note a survey of open-source frameworks enabling FL, although without adequately exploring concepts of privacy.³⁴ Thus far, works that survey privacy issues related to FL in medicine^{26,35,36,37,38,39} have provided details about FL in health care while not adequately expanding on the specific privacy threat associated with each attack.

In this work, we provide an overview of the current privacy threats and associated threat mitigations for FL workflows⁴⁰ while keeping the health-care context in mind. We summarize the key factors involved in determining the nature of privacy violation that can be related to each threat. Finally, we categorize the privacy threat mitigation technologies into distinct categories, and for each group we discuss what threats they address, as well as the costs associated with each mitigation technique. We hope that model developers having domain expertise can use this work to make more informed decisions with regard to patient privacy when using FL to train their models, and we hope to provide the context necessary for researchers to design experiments that most effectively advance our understanding of the problems and potential solutions for their domain.

In order to facilitate the focus on privacy concerns to FL deployments specifically in the health-care setting, we start by briefly introducing the concept of FL in health care. We then list the primary assets in AI that need to be protected in such an FL schema and then proceed with introducing a taxonomy of the potential threats to these assets, using the well-known confidentiality, integrity, and availability (“CIA”) triad.⁴¹ Notably, for each compromise, we consider the way in which it could adversely affect the collaborators. Finally, we discuss the mitigation methods that exist for these threats by considering how they reduce the impact of the threat, as well as the costs that are associated with the usage of the mitigation.

Federated learning

FL describes a collaborative approach to train AI models across decentralized collaborators (e.g., client servers on health-care sites) without directly sharing any training data between them.^22,23,24,25 This approach differs from the standard/traditional approach followed during training of AI models, which typically assumes that the data and model reside on a single, centralized location (see Figure 1 for an illustration). FL allows multiple institutions to train a single aggregate model without explicitly sharing any individual institution’s patient data outside of that institution.

For example, to train a neural network (NN) with FL, an NN architecture needs to be chosen and code to implement training on this architecture incorporated into the system code to be distributed to all collaborators. We consider the use case of tumor boundary detection^14,22,24 for our explanation, where the trained AI model has an image as input and an image as output. The input image represents a clinically acquired scan, and the output is meant to identify regions of pathology associated with the presence of a cancerous tumor. The typical FL system consists of multiple participant sites, all independently connected via the network to a (central) “aggregator” node as depicted in Figure 1. To start the process, model initial weights are chosen at the aggregator as the initial global model and distributed via the network to all participants.

A typical “round” of federated training proceeds as follows. All participants perform training and validation on the aggregate model using their local data and compute resources and send their local model update as well as local validation scores to the aggregator node. The aggregator then aggregates all received local model updates and local validation scores, to form the updated aggregate model and aggregate validation scores for that round. This updated aggregate model is then sent back to all participants to initialize the next round of training. The complete course of FL training consists of multiple FL rounds, with a stopping/convergence condition and model selection criterion enforced by the aggregator using the model validation scores.

We take the opportunity to point out that model updates and validation scores should be viewed as a potential way to obtain information about the training input data. The local training described above consists of iteratively (1) passing batches of input images into the model, (2) measuring how well the model did at predicting the correct pathology locations, and (3) adjusting the model weights using small corrections obtained through the NN backpropagation process. Each batch of input data shifts the model weights using the information in that batch for performance to increase. This influence can potentially lead to information about the batches being detectable in the resulting model weights, as we will see in subsequent sections. The same is true for the validation metrics, although to a lesser degree. The intuition here is that patterns may be unique enough for one particular data sample that the results of training or validating on it could indicate its presence.

By being an approach in which data from one institute are never seen by another, it helps alleviate some privacy and security concerns, especially when dealing with sensitive medical data.^25,42 This potentially enables collaborations to be formed with larger and more diverse training data, which can result in trained models that generalize better.^22,43 In most FL algorithms, each institution independently performs training on their respective (local) data and provides the results of their computation (usually weights of a model) to be incorporated by the FL workflow (such as peer-to-peer aggregation or based on an aggregation server²⁵; see Figure 1 for an illustration of FL using an aggregation server). Details of the typical central aggregator-based version that we provide as an example above can also differ depending on the implementation, for example, how model aggregation is performed or how the clients are selected in each round.^44,45,46,47

FL has the potential to play a critical role in the next generation of privacy preservation strategies for health care, as evidenced by several recent high-impact studies.^{14,22,24,25,27,48,49} The degree to which an FL system can be thought to be “secure and private” largely depends on the additional security and privacy technologies that it incorporates into the system. Concerns need to be addressed to mitigate malicious code execution, such as processes running at the collaborating institution’s infrastructure that ex-filtrate their raw data or execute malicious forms of training. However, there are also threats from collaborators during FL that may attempt to extract information about the training data via the model updates that are shared between collaborators in the workflow. These updates may indeed leak information if in the hands of an adversary, as was suggested above.

Security and privacy for FL in health care

Importance of privacy in health care

Privacy attacks during the course of an FL collaboration result in exposure of data, model, or code (see Figure 2). Data privacy in health-care scenarios is crucial toward ensuring confidentiality and ethical handling of protected health information (PHI). Identifying robust mitigation techniques corresponding to specific threats, as well as implementing security measures, adopting encryption technologies, and adhering to privacy regulations (such as Health Insurance Portability and Accountability Act [HIPAA] from the United States⁵ and General Data Protection Regulation [GDPR] from the European Union⁶ or Digital Information Security in Healthcare Act [DISHA] from India⁷), is essential for safeguarding sensitive patient information and maintaining trust in health systems. Key health-care scenarios relevant to privacy preservation related to FL include (1) electronic health records (EHRs), which contain verbose textual data about a patient’s medical history, procedures, diagnoses, medical scans, treatments, and medications, exposure of which would be a subject to patient confidentiality loss; (2) wearable devices, which collect health-related, fitness, and nutrition patient data directly associated with PHI; (3) local/institutional data repositories such as Biobanks and picture archiving and communication systems (PACSs), including both data from routine clinical practice, clinical trials, and emergency services and unpublished research outcomes; (4) health insurance/billing; and (5) health-care policy.

Figure 2.

Illustration of overall privacy threat categories and their associated risks for stakeholders in a federated learning system

The “system-level threats” (in the green box) target the data and model; the “poisoning” attacks (in the orange box) attempt to expose the data, model, and code; and “information extraction” (in the blue box) targets model inversion, membership inference, data attribute inference, and model extraction.

Assets

There are three AI-specific assets to consider with regard to security and privacy around FL deployments. These are (1) the entire data cohort to be used for training; (2) the quantitative performance evaluation metrics that signify model performance, generated against both the local and the aggregated model updates; and (3) the model parameters themselves, for both the local and the aggregated model updates. In addition, the system on which the AI system is being deployed includes three additional assets that should be considered: (1) the hardware on which all the computations (i.e., training and FL aggregation) occur, (2) the actual source code for model training and FL execution, and (3) any additional metadata or configuration information that can be defined either in memory or as files on disk. Table 1 offers a summary of each asset and the properties we propose to address in this work. In the following sections, we will proceed through each of the CIA properties and elaborate on their meaning. On a high level, hardware protection is expected to already be in place, participants are expected to report correct validation metrics with what we see as minimal privacy consequences otherwise, and we see minimal privacy impact to participants dropping out and/or network connections being lost and so do not address issues of unavailable FL system resources.

Table 1.

Security and privacy assets in an FL system, including the CIA properties we propose to address in this work

Asset	Confidentiality	Integrity	Availability
Training data	✓	✓	✗
Quantitative metrics	✓	✗	✗
Model parameters	✓	✓	✗
Hardware	✗	✗	✗
Source code	✓	✓	✗
Additional files or information	✓	✓	✗

Confidentiality

By “confidentiality” we refer to the degree to which the asset is hidden from others. As an example, if collaborator A sends their model update to the aggregator using transport layer security (TLS), then A has some assurances as to the confidentiality of the update while it is being transmitted to the aggregator. Once the aggregator receives the update and the decryption in TLS is performed, the degree to which the update remains confidential depends on what the aggregator code logic does with the update (for example, it could simply broadcast it to others), as well as how protected the aggregator processes are (e.g., code, memory, hardware instructions) from inspection by other processes and users on the aggregator infrastructure. These issues are exacerbated in the case of peer-to-peer aggregation methods,²⁵ where each collaborator performs weight aggregation on the model updates received by a peer-collaborator.

The confidentiality of any part of the data cohort and model parameters is considered here, with a break in confidentiality of either representing either a privacy violation or a leak of intellectual property (IP). Exposure of the complete data cohort in general can be a privacy violation, and model parameters can be used in an effort to reverse engineer training data.^50,51 Both model parameters and data can be considered IP, as both have value to organizations. We also consider exposure of approximations to these assets (i.e., data and model parameters) as a break in confidentiality. In the health-care setting, an approximation of a medical image may violate privacy just as much as the originally acquired image, and an approximation to model parameters may preserve enough model utility as to continue to hold a great deal of value as IP. The confidentiality of quantitative evaluation measures of model performance will also be considered, because such scores can be used to approximately recover the parameters of the model being evaluated, which, as discussed earlier, can further lead to approximate recovery of the underlying data.

Although physical isolation of hardware as well as the confidentiality of system code and additional files may be a general concern, they are considered out of the scope of this review, which instead focuses on the privacy and security aspects of medical data to be used in FL.

Integrity

By “integrity” we mean the degree to which the asset is precisely what it was expected to be. As an example, collaborator A may want to establish the integrity of the code running on the compute infrastructure of collaborator B. In some rare cases, though, A may trust B and their infrastructure to the extent that A is confident of such integrity.

The integrity of system hardware, i.e., being able to rely on the proper execution of hardware for a given hardware state, will be considered as out of scope of this work. This work shall instead focus on the integrity of training data and model parameters, as well as the integrity of system code and additional files and metadata. Although the integrity of model validation scores could be a concern in the general setting, we will not address this issue here, as the corruption of such scores would primarily be involved in an attack that was attempting to alter model selection (as such scores inform that process) and, as such, is not a significant concern in the health-care domain.

Availability

By “availability” we mean the degree to which an asset is available to use. As an example, a local model update may not be available at the aggregator if the network infrastructure at that collaborator is down, or the entire federation might get hampered if the network access at the aggregator level is lost.

Addressing availability issues for all listed assets in the previous section is considered out of the scope of this work, as we start with the assumption that the networking infrastructure for all collaborators in an FL system focusing on health care data is controlled by the respective clinical entities and, as such, is reliable and stable throughout the computation process.

Threats to privacy during FL

In this section, we will be discussing the various threats to privacy in an FL system, an illustration of which can be found in Figure 2.

System-level threats

The threats in this category involve an adversary gaining access to either the raw data or the model weights. The adversary can acquire direct access through different means, such as privilege escalation and/or physical access, but always ends up with the requisite assets in their raw exact form. In contrast, all other threat categories in this section involve an adversary deriving approximations to these raw assets, such as extraction of training data information via the model weights⁵² or manipulating their local data and/or training algorithm to exacerbate such an attack.

• (1)Data ex-filtration: this involves an adversary obtaining access to the raw data (such as health-care scans or medical records) of one or multiple collaborators. The nature of privacy violation in this case is given by a patient’s right to not have their data given to anyone whom they did not explicitly authorize to have them.^5,6
• (2)Model ex-filtration: this involves an adversary obtaining access to the weights and biases of a local model update or global model aggregate(s). The primary concern would be either that the model represents IP or that the model could be used to extract information about the raw data used to train it.⁵² Therefore, the nature of the privacy violation in this case can be the peculation of IP or could be any of the privacy violations^5,6 that can come from a successful attack to extract information about the training data from the model.

Information extraction

There are different types of privacy attack objectives related to the extraction of information about the training data from the model weights during, or at the conclusion of, the model training process. Multiple studies show that rare or unique parts of the training data are unintentionally retained by NNs.^52,53,54 The trained model weights transferred from a local institution (as well as the aggregate models they become a part of) can therefore be potentially exploited by any user with access to the model, taking advantage of this unintended memorization to gain sensitive information about the dataset being used by other collaborators in a federated setting. Some of the examples of such threats are illustrated below, and clinical researchers should consider the privacy impact of each of these threats independently.

After the attacks that use model access to approximate information about the data used to train it, we include one more attack that instead uses access to the model validation scores to approximate the model itself. This is an attack that is of concern in a federation that was otherwise acting to control access to the model (as IP) and, in addition, is a concern because such an approximate model can also be further used to approximate the training data themselves.

• (1)Model inversion: such attacks involve an adversary with the ability to query the model or observe a model update, constructing a data sample meant to approximate an actual sample in a collaborator’s dataset.^55,56,57 Although the accuracy of these reconstructions can vary, the exposure of such a reconstruction may violate a patient’s privacy if features in the reconstruction are highly correlated to the original samples (for example, chest radiographs⁴⁸). One form of this attack is carried out by an adversary with access to any version of the model and has been demonstrated outside of the FL context. Fredrikson et al.⁵⁶ showed how an adversary may use the prediction confidence values to approximate associated faces in a facial recognition system. Zhang et al.⁵⁸ demonstrated that an adversary with access to the model weights can approximate training examples for various classes using some auxiliary knowledge, such as blurred images from each of those classes. Other forms of this attack utilize single FL model updates from a particular collaborator and may make certain assumptions about the setting and attempt to approximate aspects of local training, such as batch normalization statistics or what labels were used for the batches.⁵⁹ State-of-the-art versions of these stronger attacks have demonstrated pixel-perfect reconstruction of images⁶⁰ when the attacker has access to local updates created with few samples, so that each sample has more relative influence. However, most FL round model updates are processed using many local data samples so that individual sample influence is reduced. In these cases, it is more difficult to reconstruct an exact training data sample from a local model update and even harder to reproduce one from the global aggregate model(s) it is included in. An overview of model inversion attack implementations and defense approaches is already described in prior work.⁶¹ Advancements in these attacks continue to be put forward, and works that demonstrate such attacks in the setting of FL for medical models that demonstrate successful approximation of hidden batch normalization statistics, for example,⁶² acknowledge the importance of understanding such threats in these settings.
• (2)Membership inference: membership inference is the process by which an adversary has possession of a particular data sample and is attempting to infer whether it was included in the training set of the model.^50,51,63 Exposure of whether a specific patient’s data sample was used in training may be sensitive information, for example, if the presence of that sample implied something about the sample custodian (i.e., the dataset consisted of information about known felons or the dataset consisted of patients with a certain type of cancer). Success in accurately predicting which samples were involved in training a model is correlated with the degree to which a model encodes sample-specific information during training.^64,65 Due to this fact, membership inference attack success measurements are thought of as building blocks for state-of-the-art tools for generically determining the amount to which a model leaks information about its training data.⁶⁶ Clinicians and researchers should therefore consider successful membership inference attacks as a privacy concern, regardless of how compelling the concern is regarding the leakage of membership information alone. Such success may indicate that other more concerning attacks may be successful as well. For more details regarding the various membership inference attacks and defenses, see Hu et al.⁶⁷.
• (3)Data attribute inference: instead of attempting to recover an entire data sample, this type of attack is characterized when an adversary attempts to recover only a subset of the data attributes from particular samples in the training set or, alternatively, the adversary attempts to learn attributes of the training set as a whole (such as aggregate statistical information). Such attacks have been demonstrated outside of the FL setting,⁶⁸ as well as within the FL setting with the attacker being an FL participant that only has access to the aggregated model,⁵⁰ although the use of alternative aggregation functions other than a simple weighted average could make this more difficult. Here, the attacker may estimate the accuracy of the reconstruction over other possible alternate values for the unknown data fields. Measures of confidence in the reconstruction may play a role in the impact of its exposure. Take the case of an attack to disclose the value (“true” or “false”) of the attribute indicating a positive diagnosis of a particular medical condition. A confidence measure of $80\%$ for such an attack, conditioned on a specific gender, location, and age of patient, may be used to assert that $80\%$ of the samples in the training set corresponding to that specific gender, location, and age had a positive test result, which (depending on the characteristics) could be considered as exposure of PHI (defined as any information about health status, provision of health care, or payment for health care that is created or collected by a covered entity [or a business associate of a covered entity] and can be linked to a specific individual⁵) and, hence, identifying specific subjects from the training data. Such confidence measures have indeed been considered in previous studies and are easier to interpret when the feature space is relatively limited, such as when using categorical data with numerical digits (demonstrated by the attack example seeking to recover social security numbers⁵²), and alternatives to standard model averaging, such as having collaborators withhold a subset of their local update, have been demonstrated to influence the effectiveness of such attacks.⁵⁰ Such confidence measurements would be more challenging to obtain for training sets containing only high-resolution images, for example.
• (4)Model extraction: the traditional forms of this attack use the ability to obtain model outputs associated with arbitrary inputs chosen by the adversary^69,70 in order to estimate the model weights after a number of such queries. This is especially related to “black box”-style attacks, where a black box system⁷¹ (basically, any system that does not expose any aspect of the inference process, starting from the data processing to the model weights, and only provides inference results) is used to generate outputs for multiple inputs in the form of predictions or validation scores. Depending on the amount of information provided by validation scores during FL training, such attacks based on validation scores alone may be more limited in their ability to approximate the model parameters, and with increasing numbers of input samples, the adversary can obtain a closer approximation to the model under attack. Such efforts to attain model weights, despite an appearance that access to such weights is restricted, could result in loss of valuable IP to a participant during a medical model federation.

Poisoning

This type of threat is specifically relevant for an adversary that is part of an FL system, such as a collaborator. A collaborator during the FL training process can maliciously alter their computations (by virtue of changing the local code, or data, or model) in order to magnify the effect of one of the attacks listed above on the private assets of others in the system, for example, by increasing data memorization. As such behavior serves to modify the model behavior in a malicious way, this falls under the category of model poisoning. Such advanced attacks are demonstrated for standard FL (where aggregation is simple model averaging) in Nasr et al.⁵¹ and Hitaj et al.⁷² and provide for a significant increase in the severity of the baseline attack. Alternative mechanisms for model averaging, such as limiting the portion of the local updates shared to the aggregator or employing a more robust aggregation, such as median rather than mean, can have an effect on the adversary’s success but also affect the convergence properties of the global model. As such, these alternatives need to be considered together with the potential drawbacks to final model utility. Therefore, model poisoning can be considered fundamentally as a security attack by nature, since the primary attack vector is system asset corruption.

Approaches to mitigate privacy threats during FL

The amount of information that is exposed by the attacks described in the previous section can depend on the adversary’s level of access to the various assets in an FL system. As a basic requirement for considerations in health-care information technology (IT) infrastructures, we will assume that all FL systems will incorporate basic security measures, such as authentication to verify identities, support for encrypted communication, and other access control mechanisms meant to prevent exposure of assets to those who do not need to use them. However, since FL is a collaborative procedure, involving many parties who do need to handle assets in the system, the focus of the threat mitigations in this paper is to share assets and computational duties on those assets while mitigating the threats that exist when doing so with potentially untrusted parties. As an example, this includes the use of the final model, hosted by or with an untrusted user. As mentioned in the last section, such access can be sufficient for extracting model IP and/or carrying out membership or attribute inference attacks.

Some mitigations provide confidentiality of data while being used for computation, and others provide assurances that system code preapproved by participants matches identically to that being used at execution time. In addition, when an asset will be exposed due to requirements for its use, there are technologies to employ before the release of that asset to help mitigate a potential information extraction attack. We therefore consider two broad non-overlapping categories for the technologies that can be used during FL to mitigate the threats discussed in the previous section, which can be combined as needed (for an illustration, please refer to Figure 3): (1) controlled-use protection and (2) free-use protection technologies.

Figure 3.

Various mitigation strategies and the respective privacy threats they aim to address

The “controlled-use protection” mechanisms of secure multi-party computation and homomorphic encryption are used to mitigate the system-level threats. The “free-use protection” mechanisms of differential privacy and privacy-aware model objective can be used to mitigate against information extraction. Confidential computing, on the other hand, can be used to counteract poisoning attacks in addition to the system-level threats and information extraction. The computational performance decreases as we move from secure multi-party computation to homomorphic encryption, confidential computing, differential privacy, and, finally, the privacy-aware model objective, being the most computationally inefficient.

Controlled-use protection

These methods perform little to no alteration of the asset, but instead provide a way for potentially untrusted entities to perform computations on those assets without accessing the assets themselves, potentially controlling what computations are performed. None of the solutions in this category provide any protection against attempts to reverse engineer information from the outputs of the computations. For example, cryptography-based algorithms can be used to carry out remote algorithm execution with limitations on exposure of information to the remote parties during their calculation tasks (software-based confidentiality, such as homomorphic encryption [HE]).^73,74,75 In addition, specialized hardware solutions can provide computational resources while limiting the exposure of data used in the computation, by incorporating integrity checks during code execution to ensure that the appropriate code is being executed (hardware-based confidential computing [CC] with trusted execution).^76,77,78

The first two solutions belonging to this category, i.e., secure multi-party computation (SMPC) and HE, allow for outsourcing computation with confidentiality of the inputs, intermediate results, and outputs and are implemented via software. These solutions may or may not provide assurances as to the integrity (correctness) of the computation. The other solution (i.e., confidential computing) is a hardware-based approach providing confidentiality of inputs, intermediate results, and outputs, while also providing assurances as to the integrity of the computation,¹⁴ though usually with different assumptions as to in which circumstances protection is provided. CC solutions can generally help mitigate all threats listed in the previous section. Due to this broad threat coverage, combining CC with mitigation strategies that tackle broad information extraction threats (i.e., free-use protection category) can ensure the most robust security design. None of the solutions in this category provide any protection against reverse engineering of inputs to the computation itself. In general, there is a cost associated with these solutions that comes in the form of either increased computation and communication or special hardware requirements, in the case of CC.

Secure multi-party computation

SMPC^73,74,75 is an umbrella term that refers to a set of algorithms used to allow multiple entities to collectively calculate some function with controls as to what is exposed to the individual entities regarding both the inputs and the outputs of the function. As an example, suppose we wanted the aggregator in an aggregator-based FL^22,24,25 to know the output of a function of two inputs, and we want this to be computed using an input from each collaborator of a two-collaborator federation. The most basic example of an SMPC protocol here would be to use the trusted third party (TTP) protocol⁷⁹ to allow a third party (trusted by each collaborator and the aggregator) to take the inputs from each collaborator and send the output of the function applied to these inputs to the aggregator, without sharing either collaborator’s inputs nor sharing the output to anyone except the aggregator. Using TTP, each collaborator would not learn anything new by participating in the protocol, and the aggregator only learns whatever can be deduced from the output of the function that was provided to it. Due to its simplicity, as well as the minimal information exposure involved, TTP is the benchmark protocol used to evaluate the properties of all other SMPC protocols.⁷⁹ Ongoing research is exploring the use of SMPC as a privacy enhancement to FL on medical data either by helping to prevent malicious models or by improving the confidentiality of model aggregation^80,81 or by making progress on the overhead that is incurred by its use.^82,83 These protocols can incur high computational and network communication overhead costs, as significant computation can be required to obfuscate information by encrypting/encoding and splitting into parts to avoid recovery, and significant communication protocols can be involved in order to coordinate the compute on the information pieces as well as combining the results to recover the function output without revealing information to unwanted parties in the process.

Homomorphic encryption

Although SMPC allows multiple institutions to jointly evaluate a function without needing to share their respective private inputs, the design of an SMPC protocol needs to take into account the specific function whose output is desired from the protocol, and a good deal of the protocol itself is dedicated to obfuscating the inputs. In contrast, HE^84,85,86,87 is a type of data encryption (and therefore provides cryptographic guarantees of confidentiality) that allows for generic computation on the data when in its encrypted form. The result of a computation on the encrypted data, when decrypted, is identical (or very close) to the result of the same computation performed on the unencrypted data. One benefit over SMPC is that multiple adversaries cannot collude to significantly increase the threat. However, the encryption for HE requires keys, and so key management is a necessary consideration here. HE by design provides a robust privacy solution for the application of FL.⁸⁸ However, almost all efforts in this regard suffer from a huge computational cost, and even an incremental increase in the data size (or in the NN layers) leads to an exponential increase in runtime. As such, more work needs to be done on improving the computational efficiency to render this approach practical for modern NN architectures. Although we list this as a software-based approach, efforts are ongoing to provide hardware acceleration for HE^89,90 (https://www.darpa.mil/news-events/2021-03-08), which has the potential to significantly reduce the overhead of this solution. It can be said that HE in its present technological development is most suited to those applications that are not time sensitive, where it can offer an extremely secure form of privacy preservation solution. A recent work by Froelicher et al.⁹¹ shows the success of HE in providing truly private federated evaluation for applications within oncology and medical genetics, and work by Chen et al.⁹² demonstrates success using HE for model aggregation during FL when transfer learning is used to reduce the size of the model weights that are processed using HE.

Confidential computing

In the previous points, we described methods that use encryption, encoding, or secret sharing to increase data confidentiality during computation. Alternatively, such confidentiality can also be obtained by means that are hardware enabled with so-called hardware-based CC with trusted execution,⁷⁸ though usually there are different assumptions here as to in which circumstances protection is provided. In CC, processes can be run inside so-called “enclaves” that essentially serve as a trusted third party in the sense that we used in the points related to SMPC and HE (not even privileged users on the system can access the memory or alter the execution). Code to run an algorithm can be put into the enclave, encrypted data can be passed in and unencrypted inside the enclave, then the algorithm can execute on the inputs and the result can be encrypted and passed out. In contrast to the previous solutions, these enclaves generally have the ability to attest to the fact that the code run inside the enclave was precisely what was expected, providing assurance as to the correctness of the result. Trust in the CC itself depends on trust in the hardware vendor that designs and distributes it, and trust in attestation will depend on trust in those who implement the service that carries it out. This feature allows for trust in all components of the FL system provided they are run with CC.²⁶ For example, during FL, running local training at a particular collaborator with CC can help prevent an insider adversary on that compute infrastructure from modifying their training code to compute updates using only a few data samples, in order to increase the ease with which another adversary could extract information about those few samples from the local updates sent from this collaborator. In addition, CC is more scalable and provides faster computation time compared to SMPC and HE.

Free-use protection

These approaches do nothing to protect data confidentiality while the computations are conducted and do nothing to ensure that the computations proceed as intended. However, they have the advantage of limiting how much of the result of the computation can be used to infer information about the original inputs. Therefore, solutions in this category are ideal for mitigating the threats of the previous section in the information extraction category. The costs associated with these solutions are increased computation and a reduction in the asset’s utility due to its modification. For example, we will see that differentially private model training is a modification of a standard training algorithm that reduces the ability for someone with free access to the resulting model to carry out an information extraction attack that exposes information about the training data used to create it.

Differential privacy

When using the mitigation strategies described in the previous section for model training during FL, we are able to prevent data exposure during training. However, these techniques do nothing to prevent an adversary from using the trained model to reverse engineer information about (memorized/learned) training data, as is possible in a membership inference attack.⁹³ Differentially private model training, however, is a common approach toward mitigating the degree to which a model memorizes individual contributions to the data during training. It does so by introducing randomization during training to obfuscate the influence of these individual contributions while being able to learn over the data as a whole. DP algorithms come with privacy guarantees that relate to the likelihood that any single data point can be detected.⁹³

An algorithm can be loosely defined as “differentially private” if the output of the algorithm cannot be used to distinguish whether a particular contribution to the data is present in the dataset used as input for the algorithm training.⁹³ Common examples of what type of contribution DP considers are those of a single data record or contributions of whole collaborator datasets. While the concepts surrounding DP were generally developed for use in data analytics, DP training algorithms have become a popular method for addressing user privacy concerns in AI.^94,95

In the federated setting, DP algorithms can be used independently to train local model updates (local DP FL) or instead for the global consensus model aggregation (global DP FL). In local DP FL, each participating institution applies a DP training algorithm to perform their local training.⁹⁶ Here, the local model updates sent to the aggregator are produced with a DP privacy guarantee. This may be desired when the entity administering or running the aggregator is not trusted to prevent privacy attacks on its infrastructure. For global DP FL, the aggregation of model updates is made DP but there may be no guarantee with respect to the privacy of the local model updates handled by the aggregator.⁹⁷ If trust in the aggregator infrastructure is not already established, this may be done through the use of privacy solutions discussed in the previous section. Global DP FL is preferable to local, as it allows more data (all collaborator updates) to be combined before noising, which in principle improves the utility that is obtained for a given privacy level.⁹⁸

Although DP has started gaining traction for deep-learning applications in medicine, it comes at the cost of a reduction in “model utility,” which defines how well the model generalizes to new data when deployed,⁹⁹ as well as increased computation.⁹⁴ The model utility reduction comes from the noise addition during the training process, and the increased computation relates to potential changes in the way the training utilizes the underlying computational framework,¹⁰⁰ in addition to the potential need for more rounds of training. Importantly, DP training in FL could inhibit the use of data quality checks from specific collaborators, as privatized local model updates at the aggregator may mask signals that would otherwise indicate issues.

Survey articles for DP^{101,102,103,104} can help to summarize the various approaches, best practices, and future research needed. However, more work is needed to understand the trade-offs associated with specific use cases, such as how much utility loss will be incurred at a given privacy level. We find in recent work^{105,106,107,108,109} on DP FL training in medicine that federated training using DP (at $ϵ=4$, for example) can reach within 5% of the scores that would be achievable if DP were not used. As more research is done across different datasets and model architectures, a better understanding will form around how well these initial results will generalize. The privacy achieved by a DP algorithm should also be carefully considered. Most papers explore only ϵ values (lower indicates more privacy) greater than 1, and many explore ϵ values that are much greater. Since the worst-case odds of privacy exposure for an ϵ-DP algorithm is $e^{ϵ}:1$, the value $ϵ=4$, for example, is associated with worse than $50:1$ odds of privacy exposure.

Another complicating factor is the difficulty for data custodians to understand the privacy guarantee associated with the use of DP training, as it is very technical. In addition, the likelihood of specific privacy threats is even less likely to be understood until more research is done. This makes the proper balance of utility loss against true privacy concerns difficult to reason about, which is a very important aspect for the design of a practical solution.

Privacy-aware model objective

An alternative to making model training DP during FL is to incorporate the incentive against susceptibility to privacy attack into the training objective. Here an “attack model” is used during training in order to simulate a privacy attack on the primary model. Adversarial training is then performed at each collaborator during each round, alternating between improving a locally held attack model and improving the primary model by attempting to minimize a privacy-aware model objective (PAMO) (for example, the sum of the primary model loss and a measure of success for the attack model).^110,111 Such approaches may utilize mutual information estimations in order to establish a measure of success in minimizing information leakage or could use other measures to determine this success. The privacy protections afforded by this approach are similar to that of DP; however, the measure of protection is usually empirically based, in contrast to the theoretical privacy guarantee provided by DP. The costs associated with this approach are that of potential loss in model utility as well as potentially increased computation and overall local training required, as the local training is more complex.

An information conservation approach

In this section we describe a general approach that is meant to address the concern of information leakage from model updates concerning the individual data samples used during training. The mitigation measures discussed here are ones that restrict or obscure information during training, but without formal analysis on how such measures affect information flow. As such, an empirical assessment of the utility loss in the model as a result of each technique must be weighed against the ease of use, as well as any loss of privacy (calculated with privacy risk scores coming from empirical privacy vulnerability tools such as those discussed in Murakonda and Shokri⁶⁶ and Jayaraman and Evans¹¹²).

Some examples are as follows:

• (1)Privacy-goal-oriented training methods: perform local training in a way that has been demonstrated as resistant to subsequent privacy attacks against the data used in training. In this approach, the training is tailored (e.g., by loss function or data pipeline specifications) to reduce memorization of the training data during training. Such efforts are described in Liu et al.¹¹³ The PAMO mitigations of the previous section that are not accompanied by a measure of privacy afforded by the training fall into this category.
• (2)Partial weight sharing: instead of sharing all the weights of the model to the aggregator during FL, only a predefined percentage (largest components) of the model is shared.^43,49

Discussion and conclusions

In this work, we provide a taxonomy and a deeper understanding of current privacy threats with their associated mitigation approaches, by keeping the focus on the context of FL in a health-care setting. We have provided common definitions that could be used in this field, while giving the reader detailed summaries of possible violations of privacy and their strategies to mitigate them, along with a meaningful categorization for both. We have begun to explore the veracity of these techniques in the context of FL for health care, following the mounting evidence that FL represents a potential paradigm shift on how multiple health-care institutions can collaborate to develop AI models without sharing any of their local data.^{14,21,22,24,49,97,114} We hope that, by building upon previous works in the field,^{26,29,30,31,32,34} we have provided an opportunity to current and future researchers in the field of health-care informatics to make better informed decisions during model training to appreciate potential security issues.

All the privacy threats and threat mitigation technique categories discussed in this review have been encapsulated in Table 2. The appropriate techniques to employ for a specific case differ due to the variety of protections afforded by each.

Table 2.

Comparison of privacy-enhancing techniques in terms of properties that would need to be considered for deployment

Technique	SMPC	HE	CC	DP	PAMO
Exposure of data in use	no	no	no	yes	yes
Integrity of data in use	no	no	no	yes	no
Result unprotected from information extraction	yes	yes	yes	no	no
Execution integrity	depends on protocol	no	yes	no	no
Performance overhead (implementation dependent)	high	high	medium	medium	medium
Mathematical parity to the original results	yes	yes	yes	no	no
Threats mitigated	system threats	system threats	system threats, information extraction, poisoning	information extraction	information extraction

Each row is the property and the head of each column is the name of a privacy-enhancing technique.

Although SMPC, HE, and CC (which protect the assets during controlled use) provide confidentiality of input data, as well as the intermediate results (“exposure of data in use”), they do not provide protection against an adversary reverse engineering this information from the final results (“results unprotected from information extraction”). In contrast, the DP and PAMO mitigation strategies (which provide free-use protection) alone do not address the confidentiality of input or intermediate results, but instead have the advantage of limiting the amount with which the final result of the computation can be used to infer information about the original inputs. Mitigations in the CC category may provide assurances as to the correctness of the computations being performed (“execution integrity”), whereas those in the other categories generally do not.

Although the costs associated with mitigation solutions can vary significantly, in general, the categories SMPC, HE, PAMO, and DP incur the cost of increased computation. In addition, SMPC can incur the cost of increased communication, and CC implementations in general have specific hardware requirements associated with enablement of hardware-supported trusted execution environments.¹¹⁵ Finally, mitigations in the categories DP and PAMO generally incur the cost of a reduction in model utility (e.g., classification accuracy).

Protecting patient privacy must always be one of the primary considerations of health-care institutions, and it becomes more important as more clinical sites are initiating or joining collaborations that leverage health-care data to train AI models for further precision medicine.^{14,16,17,18,19,20,116,117} As outlined in Kairouz et al.,³¹ privacy attacks within the FL setting are a cause for concern. A significant amount of experimentation on the associated threats and mitigations, crucially in realistic scenarios on real-world data, is required to understand how they play out in different settings where health-care models are trained using FL. Understanding of the costs and benefits of these additional privacy protections during FL for health care is also critical, as there is mounting evidence^{14,21,22,24,49,97,114} that FL with additional privacy protections may represent a potential paradigm shift on how multiple health-care institutions can collaborate to develop AI models without sharing any of their local data.

Application of the threat mitigation techniques outlined in this paper poses particular challenges in health care. There are no turnkey implementations, as solutions require careful configuration (and in some cases iterative tuning) to be effective. It is difficult to ensure that solutions will be accepted by the stakeholder involved, as policies on data security vary widely by institution. Even performing FL with no additional privacy threat mitigation can be difficult in the health-care domain, since most data that could be used for these purposes reside in private institutions that lack incentives to participate in FL activities, and often institutions use operational systems for data handling that make connecting the data to FL platforms a difficult task. This is in addition to the requirement that institutions export the data cohort and de-identify them prior to starting any research. These issues, specific to the health-care setting, will need to be addressed as research activities in privacy-threat-mitigating machine learning (ML) model training for medicine are undertaken. Solutions for these issues require community-driven standards to be developed in concert with relevant stakeholders.¹¹⁸ Otherwise, the legal risks of loss of patient privacy due to an improperly designed solution may outweigh the benefits of integrating AI models in clinical workflows.

We are still in the early days of considering these privacy concerns and using the security technologies highlighted in this review in large-scale FL deployments. There is a significant amount of research to be done, especially into how the incorporation of select privacy mitigations into a federated study will impact the stakeholders involved. As we discussed above, the costs and benefits of individual solutions are being explored and improved in the literature against standard measures. The results are frequently dependent on specifics such as the trained algorithm, the data distributions involved, and the compute and physical network infrastructure to be used. More studies are needed to get a better understanding of how the current results will generalize in large-scale federations. In addition, it can be difficult for prospective FL participants to make decisions based only on the measures of privacy benefit currently reported for a solution in the literature. These measures may not map well to either regulation or common patient privacy concerns. Although some work exists, more research to help bridge this gap would be valuable for those trying to balance the concerns of stakeholders to a federation.^{39,45,61,67,82,104,119}

In conclusion, this review encapsulates and illustrates some of the major research directions pertaining to privacy in FL, and we hope it can be used as a primer and reference for future research studies as security becomes a growing concern in the health-care informatics community. Although a lot of work has been done in this area, more detailed experimentation of these methods in realistic scenarios with ample, diverse, and clinically relevant use cases will be essential for their proper quantification and subsequent evaluation for clinical deployment.