Efficient Privacy-preserving Logistic Model With Malicious Security

Abstract

Conducting secure computations to protect against malicious adversaries is an emerging field of research. Current models designed for malicious security typically necessitate the involvement of two or more servers in an honest-majority setting. Among privacy-preserving data mining techniques, significant attention has been focused on the classification problem. Logistic regression emerges as a well-established classification model, renowned for its impressive performance. We introduce a novel matrix encryption method to build a maliciously secure logistic model. Our scheme involves only a single semi-honest server and is resilient to malicious data providers that may deviate arbitrarily from the scheme. The $d$-transformation ensures that our scheme achieves indistinguishability (i.e., no adversary can determine, in polynomial time, which of the plaintexts corresponds to a given ciphertext in a chosen-plaintext attack). Malicious activities of data providers can be detected in the verification stage. A lossy compression method is implemented to minimize communication costs while preserving negligible degradation in accuracy. Experiments illustrate that our scheme is highly efficient to analyze large-scale datasets and achieves accuracy similar to non-private models. The proposed scheme outperforms other maliciously secure frameworks in terms of computation and communication costs.

I. Introduction

Internet of Things (IoT) approaches our lives gradually with the wireless communication systems increasingly employed as technology driver for smart monitoring and applications. An IoT system can be depicted as smart devices that interact on a collaborative basis for a common goal. Smart cities are incorporating a wide range of advanced IoT infrastructures, resulting in a large amount data gathered from different devices deployed in many domains, such as health care, energy transmission, and transportation [1]. Smart things provide efficient tools for ubiquitous data collection or tracking, but also faces privacy threats.

In order to solve the challenges arising from IoT data processing and analysis, an increasing amount of innovations have been emerged recently. For instance, collaborative learning is a desirable and empowering paradigm for smart IoT systems. Collaborative learning enables multiple data providers to learn models utilizing all their data jointly [2], [3]. Typical collaborative systems are distributed computing systems such as secure multi-party computation (SMC) frameworks [2], [4]. SMC enables parties to jointly compute on private inputs without revealing anything but the result.

Collaborative learning has benefited the society including medical research [5]. Data containing healthcare informatics are usually collected in medical centers such as hospitals. Generally, the study center does not share data with other institutes considering the confidentiality of participants. To learn disease mechanisms especially rare diseases for which each center has limited cases, it is of importance to perform data analysis combining data from multiple institutes. Collaborative learning provides great promise to connect healthcare data sources. Since data sharing of individual levels is not permitted by law or regulation in many domains, various privacy-preserving techniques have been developed to perform collaborative learning.

Many privacy-preserving techniques assume semi-honest models, in which the server and clients follow the protocol specification. Because clients could be any arbitrary entity, it is less likely that all the clients (i.e., data providers) would be semi-honest. Recently, maliciously secure models [6], [7] have been proposed to achieve privacy in the presence of malicious adversaries that could deviate arbitrarily from the protocol specification. Based on the assumption of the number of servers that can be malicious in the protocol, maliciously secure frameworks operate in either an honest-majority setting [6]–[14] or a malicious-majority setting [15]–[18]. These frameworks typically rely on multiple servers (e.g., threeserver model [6], [7], [9], [10], four-server model [8], [12]–[14]), with the most common assumption being an honest-majority setting (i.e., a majority of servers are semi-honest). In contrast, the malicious-majority setting anticipates a scenario where a majority of servers may behave maliciously. This setting enhances security in environments where a significant portion of servers may be untrustworthy, providing a more realistic and robust solution in adversarial conditions. Since the efficiency of SMC protocols is highly dependent on the number of honest servers [13], maliciously secure frameworks with the malicious-majority setting is less efficient than those with the honest-majority setting.

In this paper, we propose a privacy-preserving logistic model scheme, assuming a dishonest majority in a maliciously secure setting. We assume that data are horizontally distributed among data providers (i.e., each data provider is a client that collects information of the same features for different samples). Our contributions are summarized as follows:

1. We propose a novel matrix encryption technique to build a maliciously secure logistic model. Unlike state-of-the-art frameworks that necessitate the involvement of two or more servers in an honest-majority setting, our scheme involves only a single semi-honest server and is resilient to malicious attacks conducted by data providers. Malicious behaviors conducted by data providers are detectable during the verification stage.

2. The proposed matrix encryption method combines Gaussian matrix encryption with $d$-transformation and commutative matrix encryption. The implementation of $d$-transformation ensures that random records within any energy range are indistinguishable. The commutative matrix encryption is applied to preserve data utility.

3. We utilize a lossy compression method to reduce communication costs while ensuring negligible degradation in accuracy. Compared with other maliciously secure frameworks, our scheme is more efficient to analyze large-scale datasets in terms of computation and communication costs.

II. Related work

Secure multi-party computation (SMC)

SMC frameworks with a small number of parties have shown to be particularly attractive recently. Among these frameworks, homomorphic encryption (HE) [22], [23] has been widely used to protect data privacy [10], [11], [18]-[20]. Recent advances on garbled circuit [24], [25] have led to a set of privacy-preserving protocols [15]-[17] for SMC tolerating an arbitrary number of malicious corruptions. Garbled circuits and HE techniques require large volumes of ciphertexts to be transferred or have high computation complexity. In terms of efficient constructions, various secure frameworks in an honest-majority setting [6]–[14] have drawn phenomenal attention. The details of these frameworks are summarized in Table I.

Table I:

Recent secure multi-party computation (SMC) frameworks

Framework	No. of parties/servers	Encryption method	Threat model	Collusion assumption
[15]	2	Garbled circuit	Malicious	Malicious-majority
[16]	≥2	Garbled circuit	Malicious	Malicious-majority
[17]	≥2	Garbled circuit	Malicious	Malicious-majority
[8]	4	Garbled circuit	Malicious	Honest-majority
[9]	3	Garbled circuit	Malicious	Honest-majority
[18]	–	Homomorphic encryption	Malicious	Malicious-majority
[19]	1	Homomorphic encryption	Semi-honest	Passive adversary
[10]	3	Mixed	Malicious	Honest-majority
[11]	2	Mixed	Malicious	Honest-majority
[20]	2	Mixed	Semi-honest	Passive adversary
[12]	3, 4	Joint message passing	Malicious	Honest-majority
[13]	3, 4	SPDZ	Malicious	Honest-majority
[21]	2	Secret sharing	Semi-honest	Passive adversary
[14]	4	Secret sharing	Malicious	Honest-majority
[6]	3	Secret sharing	Malicious	Honest-majority
[7]	3	Secret sharing	Malicious	Honest-majority
Our	1	Matrix encryption	Malicious	Malicious-majority

Malicious model: the entity deviates arbitrarily from the protocol specification. Semi-honest model: the entity follows the prescribed protocol but attempts to gain unauthorized information by covertly observing the communication or computations of other entities involved. Mixed encryption method: the framework applies both garbled circuit and homomorphic encryption.

Differential privacy

Differential privacy (DP) [26] has been widely incorporated into distributed deep learning [27]–[29] by adding noise to input data, loss functions, gradients, weights, or output classes. Moreover, DP has been applied to enable secure exchanges of intermediate data and obtain models resilient to adversarial inferences in the federated learning [30]–[32]. There are still some challenging issues to implement DP in practice since it requires high privacy budgets to train robust and accurate models and the level of privacy achieved in practice remains unclear [33].

Matrix encryption

Matrix encryption has been extensively utilized in the development of compressed sensing (CS)-based cryptosystems [34]–[37]. This approach is well-suited for ensuring the security of practical applications, such as the Internet of Things and multimedia. The Gaussian one-time sensing CS-based cryptosystem, which employs a random Gaussian matrix and renews the matrix at each encryption, has been proven to be asymptotically secure for the plaintext with constant energy [34]–[36]. It is challenging to practically implement these CS-based cryptosystems because the indistinguishability of Gaussian matrix encryption is highly sensitive to variations in the energy of plaintexts.

To summarize, a majority of maliciously secure models necessitate the involvement of two or more servers in an honest-majority setting. Moreover, existing secure models have relatively low efficiency for large-scale data analysis. Previous studies of Gaussian matrix encryption ensures indistinguishability among records with constant energy (i.e., Euclidean norm) and poses a practical challenge when implementing it with data having arbitrary energy ranges. This paper introduces a maliciously secure logistic model which ensures indistinguishability among random records within any energy range. Our model assumes the malicious-majority setting and is highly efficient in analyzing datasets of substantial size.

III. Preliminaries

A. Logistic model

Consider a set of data $D=\left\{\left(x_1,y_1\right),\cdots ,\left(x_n,y_n\right)\right\}$, where $x_i\in {ℛ}^q$ and $y_i\in \left\{\mathrm{0,1}\right\}$ denotes the binary outcome such as case/control status of $x_i(i=1,\cdots ,n)$. Without loss of generality, a constant 1 is typically added as the first element to the record $x_i(i=1,\cdots ,n)$ to account for the intercept. The logistic model [38], [39] has the form

$$\text{log}\frac{\text{Pr}\left(y_i=1\mid x_i\right)}{\text{Pr}\left(y_i=0\mid x_i\right)}=x_i^T\beta .$$ (1)

where $\beta ={\left({\beta }_1,\cdots ,{\beta }_q\right)}^T$ is a $q$-dimensional coefficient vector and Pr(⋅) is the probability function. Model estimate of the logistic model is typically fitted through maximum likelihood, using the conditional likelihood. The log-likelihood is

$$\ell \left(\beta \right)=\sum _{i=1}^n\left\{y_i\text{log}\left[p\left(x_i;\beta \right)\right]+\left(1-y_i\right)\text{log}\left(1-p\left(x_i;\beta \right)\right)\right\}$$ (2)

where $p\left(x_i;\beta \right)=\text{Pr}\left(y_i=1\mid x_i;\beta \right)=\frac{\text{exp}\left(x_i^T\beta \right)}{1+\text{exp}\left(x_i^T\beta \right)}$.

For ridge regularized logistic model, we maximize the log-likelihood subject to a size constraint on $L_2$-norm (i.e., Euclidean norm) of the coefficients. The ridge estimate is

$${\beta }_{\text{ridge}}=\underset{\beta }{\text{argmin}}\left\{-\ell \left(\beta \right)+\frac{\lambda }{2}\parallel \beta {\parallel }_2^2\right\}$$ (3)

where $\lambda \ge 0$ is the ridge parameter.

Let $Y={\left(y_1,\cdots ,y_n\right)}^T$ denote the outcome, $X={\left(x_1,\cdots ,x_n\right)}^T$ denote the $n\times q$ feature matrix, and let $W$ be the $n\times n$ diagonal matrix of weights (Equation 4).

$$W\triangleq \left(\begin{array}{ccc}p\left(x_1;\beta \right)\left(1-p\left(x_1;\beta \right)\right)& & \\ & \ddots & \\ & & p\left(x_n;\beta \right)\left(1-p\left(x_n;\beta \right)\right)\end{array}\right)$$ (4)

We use Newton’s method to fit the logistic model. Given ${\beta }^{\text{old}}$, a single Newton update is

$${\beta }^{\text{new}}={\beta }^{\text{old}}-{\left(\frac{{\partial }^2\ell \left(\beta \right)}{\partial \beta \partial {\beta }^T}\right)}^{-1}\frac{\partial \ell \left(\beta \right)}{\partial \beta }$$ (5)

where the derivatives are evaluated at ${\beta }^{\text{old}}$. The equation can be expressed using matrix notations as follows.

$${\beta }^{\text{new}}={\left(X^T{W}^{\text{old}}X+\Lambda \right)}^{-1}\left[X^T{W}^{\text{old}}X{\beta }^{\text{old}}+X^T\left(Y-p^{\text{old}}\right)\right]$$ (6)

where $p^{\text{old}}={\left(p\left(x_1;{\beta }^{\text{old}}\right),\cdots ,p\left(x_n;{\beta }^{\text{old}}\right)\right)}^T$ and $p\left(x_i;{\beta }^{\text{old}}\right)\left(1-p\left(x_i;{\beta }^{\text{old}}\right)\right)$ is the $i$-th diagonal element in the diagonal matrix $W^{\text{old}}.\Lambda$ is a matrix of zeros for non-regularized logistic model. For ridge regularized logistic model, $\Lambda$ is a diagonal matrix with the diagonal elements being $\{0,\lambda ,\cdots ,\lambda \}$.

B. Indistinguishability

Indistinguishability has been widely used as the security measure in recent cryptosystems. Using different notations (e.g., Definitions 3.9 and 3.10 in [40], Definition 2.1 in [41], Definition 2 in [42], Definition “PrvInd” in [43], Definition 2 in [44], Definition 1 in [45], Definition 1 in [35], Section III in [36]), all these indistinguishability definitions express the same security level: a cryptosystem has the indistinguishability if no adversary can determine in polynomial time which of the two plaintexts corresponds to the ciphertext, with probability significantly better than that of a random guess. In other words, within a cryptosystem with indistinguishability, an adversary cannot learn any partial information of the plaintext in polynomial time given a ciphertext. Comprehensive comparisons of indistinguishability with other security measures (e.g., differential privacy) are given in [41], [42]. In line with other cryptosystems utilizing matrix encryption methods [35], [36], [44], we provide the formal definition of indistinguishability, denoted as Definition 1, following Definition 1 in [35], Section III in [36], and Definition 2 in [44].

Definition 1. Let $p_d$ be the probability that an adversary can successfully discern which of the two plaintexts corresponds to the ciphertext using any algorithm that operates within polynomial time. Then a cryptosystem is indistinguishable if there is a negligible function $ϵ\left(q\right)$ such that for all plaintext length $q$,

$$p_d\le \frac{1}{2}+ϵ\left(q\right).$$ (7)

$ϵ\left(q\right)$ is negligible if there exists an integer $q_c$ for every positive constant $c$ such that $ϵ\left(q\right)<q^{-c}$ for all $q>q_c$.

Let $d_{\text{TV}}\left(p_1,p_2\right)$ be the total variation (TV) distance [46] between $p_1=\text{Pr}\left(y\mid t_1\right)$ and $p_2=\text{Pr}\left(y\mid t_2\right)$ where $p_i$ is the probability distribution of $y$ conditioned on $t_i(i=\mathrm{1,2})$. Based on [47], the probability to successfully distinguish the plaintexts is bounded by

$$p_d\le \frac{1}{2}+\frac{d_{\text{TV}}\left(p_1,p_2\right)}{2}.$$ (8)

where $d_{\text{TV}}\left(p_1,p_2\right)\in \left[\mathrm{0,1}\right]$. If $d_{\text{TV}}\left(p_1,p_2\right)=0$, the probability of success is at most equivalent to a random guess, leading to indistinguishability [40].

Computing $d_{\text{TV}}\left(p_1,p_2\right)$ directly is difficult [48] and we employ the Hellinger distance [46] to bound the TV distance. Let $d_{\text{H}}\left(p_1,p_2\right)$ be the Hellinger distance [46] and it can give both lower and upper bounds on the TV distance [49], i.e.,

$$d_{\text{H}}^2\left(p_1,p_2\right)\le d_{\text{TV}}\left(p_1,p_2\right)\le d_{\text{H}}\left(p_1,p_2\right)\sqrt{2-d_{\text{H}}^2\left(p_1,p_2\right)}$$ (9)

where $d_{\text{H}}\left(p_1,p_2\right)\in [0,1]$. Moreover, if $p_1$ and $p_2$ are multivariate Gaussian distributions (i.e., the ciphertext $y$ conditioned on $t_h$ follows Gaussian distribution with zero mean and covariance matrix $C_h,h\in \left\{\mathrm{1,2}\right\}$), the Hellinger distance between $p_1$ and $p_2$ is given by [50] and [51]

$$d_{\text{H}}\left(p_1,p_2\right)=\sqrt{1-\frac{{\left|C_1\right|}^{\frac{1}{4}}{\left|C_2\right|}^{\frac{1}{4}}}{{\left|C_3\right|}^{\frac{1}{2}}}}$$ (10)

where $C_3$ is defined as the average of $C_1$ and $C_2$ (i.e., $C_3\triangleq \frac{C_1+C_2}{2}$). Formal definitions and properties of total variation and Hellinger distances are given in [46]–[48].

C. Adversarial attacks on matrix encryption methods

In previous privacy-preserving frameworks using matrix encryption techniques [37], [52]–[54], adversarial attack models are classified into four levels: ciphertext-only attack (COA), known-plaintext attack (KPA), chosen-plaintext attack (CPA), and chosen-ciphertext attack (CCA).

Ciphertext-only attack (level 1):

The adversary is assumed to have access to ciphertexts and no other information. Within the COA, the adversary attempts to retrieve sensitive information using the ciphertexts.

Known-plaintext attack (level 2):

The adversary has access to the ciphertexts and corresponding plaintexts. Within the KPA, the attacker attempts to recover sensitive information by analyzing ciphertexts and their corresponding plaintexts.

Chosen-plaintext attack (level 3):

Given any plaintext, the adversary can get its corresponding ciphertext within the CPA. The adversary attempts to recover the encryption key or algorithm by examining associations between plaintexts and ciphertexts.

Chosen-ciphertext attack (level 4):

Within the CCA, the adversary has the capability to obtain the decryption of any ciphertexts of its choice. The adversary attempts to determine the plaintext that was encrypted to give some other ciphertexts.

In our scheme, no ciphertext is decrypted and thus it is impossible for adversaries to obtain the decryption of any ciphertext. Therefore, adversaries could not perform CCA and we only consider the first three attacks.

D. Matrix encryption

Assume $x$ is a random row in the dataset $X$ containing $n$ rows and $q$ columns. To encrypt $x$, a random Gaussian matrix, denoted as $B$ with dimensions $q\times q$, is generated, where each element follows a Gaussian distribution $N\left(\mu ,{\sigma }^2\right)$, and $\mu$ and $\sigma$ are parameters of the distribution. The encryption function for row $x$ and data $X$ can be summarized as $f_{B,r}\left(x\right)\triangleq xB$ and $f_{B,r}\left(X\right)\triangleq XB$. Each column in the dataset $X$ can be encrypted similarly. Specifically, let $x_c$ be a random column in $X$. A $n\times n$ random Gaussian matrix $A$ is generated for $x_c$ encryption. The encryption function for column $x_c$ and data $X$ can be summarized as $f_{A,l}\left(x_c\right)\triangleq A{x}_c$ and $f_{A,l}\left(X\right)\triangleq AX$.

IV. System Overview

A. System model

We investigate collaborative learning in which data are collected and owned by different data providers, referred to as clients. The goal is to build an efficient logistic model using data from all the clients while ensuring privacy protection. We consider that data are horizontally distributed, i.e., clients have different sets of samples and the same set of features (Figure 1A.

Fig. 1:

A: an example showing the horizontal partitioning scenario with three data providers (referred to as clients); B: workflow of the proposed privacy-preserving logistic model.

The proposed scheme involves multiple clients and one server who is responsible for the secure computation. The privacy-preserving scheme contains four stages: encryption, modeling, decryption and verification (Figure 1B). Clients perform data encryption as the initial step. After the encryption process, the data are sent to the server for the secure computation. The server then sends encrypted model results back to clients. Subsequently, clients decrypt the model results. Finally, the server and clients initiate the verification stage to identify any malicious activity conducted by the clients.

B. Threat model

Maliciously secure frameworks with multiple servers have been particularly rich [6], [7], [12], [13]. These frameworks assume at least one server is semi-honest and does not collude with malicious adversaries. Following these frameworks, the sole server in our scheme is assumed to be semi-honest, while clients are allowed to be malicious. Precisely, we assume that the server faithfully executes the delegated computations but may be curious about the intermediate data and try to learn or infer any sensitive information. In contrast, clients are likely to act maliciously (i.e., arbitrarily deviate from the predefined scheme to cheat others). Clients may collude with each other while the server is not allowed to collude with malicious clients. The possible adversary behaviors of malicious clients and the semi-honest server are summarized as follows.

1. To perform the ciphertext-only attack (CPA), clients insert fake plaintexts into the privacy-preserving scheme and collude with each other to share both plaintexts and their corresponding ciphertexts. Detailed description of the CPA is given in Section VI.

2. Malicious clients do not follow the proposed encryption method to encrypt data (i.e., each client’s data should be encrypted sequentially by all the clients using commutative matrices). After getting sufficient data for CPA, malicious clients may choose to skip subsequent computations in order to reduce the computation cost.

3. Malicious clients do not follow the decryption procedure (i.e., the encrypted model result derived by the server should be decrypted sequentially by all the clients using commutative matrices which have been used for data encryption). Once malicious clients have gathered enough information for CPA, they may choose to skip the computations and send fake data to other clients in the decryption procedure.

4. The semi-honest server attempts to retrieve sensitive information from received ciphertexts.

C. Design goals

Our design goals contain four aspects. Privacy: The private data have to remain confidentiality at any time. Learning verifiability: There should be a verification stage to check whether all the clients behave honestly. Correctness: The scheme is able to derive correct model result if all the clients and the server behave honestly. Efficiency: The scheme is computationally efficient and achieves high accuracy.

V. Proposed scheme

A. Data encryption, modeling and decryption

Suppose there are $K$ clients and client $i$ owns the data $X_i$ (i.e., plaintext) with $n_i$ samples and $q$ features $(i=\mathrm{1,2},\cdots ,K)$. These $K$ clients collect the same $q$ features for different samples. The analytical model will be built using the aggregated data, i.e., $X=\left(\begin{array}{c}{X}_1\\ ⋮\\ X_K\end{array}\right)$. To ensure data confidentiality, we introduce a novel privacy-preserving logistic model that employs random matrices for encryption. Specifically, client $i$ encrypts $X_i$ using random encryption matrices $A_i$ and $B_i$. Aggregating these encrypted datasets, we get $X^{\text{enc}}=\left(\begin{array}{c}{A}_1{X}_1{B}_1\\ ⋮\\ A_K{X}_K{B}_K\end{array}\right)$. Since $A_i$ and $B_i(i=1,\cdots ,K)$ are generated randomly by client $i$, the aggregated dataset does not preserve data utility. In order to maintain the utility of the data, we require that 1) $B_i(i=1,\cdots ,K)$ is designed to be commutative (i.e., $B_i{B}_j=B_j{B}_i$ for $i\ne j$, Appendix A), 2) $X_i$ is subsequently encrypted by client $j$ using $B_j$ $(j\ne i)$, and 3) the encryption matrix $A_i$ is decrypted by the server prior to the secure computation. The commutative nature of the random encryption matrix $B_i$ guarantees that the resulting encrypted dataset remains independent of the order in which clients perform encryption. Clients send the encrypted data (i.e., ciphertexts) to the server after the encryption. The server then decrypts $A_i$ and obtains the aggregated data $\left(\begin{array}{c}{X}_{1}B\\ ⋮\\ X_{K}B\end{array}\right)=XB$ where $B=\prod _i^K{B}_i=B_1{B}_2\cdots B_K$, as $B_i$ are $B_j$ designed to be commutative (i.e., $B_i{B}_j=B_j{B}_i$ for $i\ne j$). Table II summarizes symbols in the proposed scheme.

Table II:

Notations

Notation	Description
$K$	Number of clients (data providers)
$X_i,Y_i$	Data (plaintext) collected by client $i(i=1,\cdots ,K)$
$Z_i$	Transformed outcome $\left(Z_i=Y_i^T{X}_i\right)$
$n_i$	Number of samples in $X_i$ and $Y_i$
$q$	Number of features in $X_i$
$X,Y$	Aggregated data (plaintext)
$n$	Number of samples in $X$ and $Y$
$x_i$	The $i$-th record (row) in $X$
$x_{c,i}$	The $i$-th column in $X$
$W$	A diagonal matrix (Equation 4)
$A_i$	Random Gaussian matrix generated by client $i$
$A_i^{-1}$	Inverse of matrix $A_i$
$B_0^2$	Random Gaussian matrix shared among $K$ clients
$b_{ij}$	Random coefficient generated by client $i(j=1,\cdots ,q)$
$B_i$	Commutative matrix generated by client $i\left(B_i={\sum }_{j=1}^q{b}_{ij}{B}_0^j\right)$
$B$	Encryption matrix $\left(B={\prod }^{\Lambda }{B}_i\right)$
$B^{-1}$	Inverse of matrix $B^i$
$X^{\text{enc}}$	Encrypted $X$ (ciphertext)
$Z^{\text{enc}}$	Encrypted outcome (ciphertext)
$X^T$	Transpose of $X$
$d$	A constant to ensure indistinguishability (Algorithm 1)
$\beta$	Model estimate (a $q$-dimensional vector) of non-secure model
${\beta }^{\text{enc}}$	Model estimate (a $q$-dimensional vector) derived by the server
$∥x_i∥$	Euclidean norm of $x_i$
$d_{\text{H}}$	Hellinger distance
$d_{\text{TV}}$	Total variation (TV) distance
$d_{\text{TV,low}}$	Lower bound of the TV distance
$d_{\text{TV},\text{up}}$	Upper bound of the TV distance
$p_d$	Success probability in the indistinguishability experiment
$ϵ\left(q\right)$	A negligible function
$Y_s$	Pseudo outcome $\left(Y_s=X^{T}X1\right)$
$Y_s^{\text{enc}}$	Encrypted pseudo outcome
$f_{A_i,l}\left(X\right)$	Encryption function $\left(f_{A_i,l}\left(X\right)=A_{i}X\right)$
$f_{B_0,r}\left(X\right)$	Encryption function $\left(f_{B_0,r}\left(X\right)=X{B}_0\right)$
$f_r\left(X\right)$	Encryption function $\left(f_r\left(X\right)=X\left({\sum }_{j=1}^q{b}_j{B}_0^{(j-1)}\right)\right)$

$1=(\mathrm{1,1},\cdots ,1{)}^T$.

Pre-processing

Before data encryption, a pre-processing procedure is conducted by each client. Specifically, client $i$ generates a pseudo record with all the values being 1 (i.e., $(\mathrm{1,1},\cdots ,1)$) and adds it to $X_i$ as the first row. The added row is used for malicious behavior detection. To encrypt the outcome information (i.e., $Y_i$ collected by client $i$), client $i$ computes $Z_i=Y_i^T{X}_i$ and concatenates it into $X_i$ as the last row. Additionally, client $i$ calculates the Euclidean norm of each column in $X_i$. Let $c_j$ be the Euclidean norm of the $j$-th column $(j=1,\cdots ,q)$ and $c_m=\underset{j}{\text{max}}{c}_j$. Client $i$ generates a vector, $c_v$, with the $j$-th element of $c_v$ being $\sqrt{c_m^2-c_j^2}.c_v$ is added to $X_i$ as the last row. This procedure guarantees that the Euclidean norm of each column equals $c_m$ and is essential to ensure the indistinguishability of our encryption approach.

Without loss of generality, we include an intercept to the logistic model. Specifically, a vector of ones is added to $X_i(i=1,\cdots ,K)$ as the first column. To achieve indistinguishability, each client multiplies the elements in the first column by a constant $d$, where $d$ is selected by Algorithm 1 (i.e., $d$-transformation). The $d$-transformation is performed before data encryption.

The proposed encryption procedures can be categorized into two layers: internal and external. The data are first encrypted by its owner internally and then subsequently encrypted by other clients, referred to as the external encryption.

Internal encryption

Client $i$ first generates a random Gaussian matrix $A_i$ to encrypt $X_i$. To improve the computation efficiency of the encryption for clients with large sample size, we partition the encryption matrix $A_i$ into a diagonal matrix. The detailed description is given in Appendix B. Client $i$ shares $A_i$ with the server and encrypts the data as $A_i{X}_i$. Client $i$ further encrypts $A_i{X}_i$ using a specifically designed matrix $B_i$. To generate the specific $B_i$, a random $q\times q$ Gaussian matrix $B_0$ is generated and shared among the $K$ clients. Client $i$ subsequently generates a random coefficient vector $\left(b_{i1},\cdots ,b_{iq}\right)$ and a client-specific matrix $B_i=\sum _{j=1}^q{b}_{ij}{B}_0^j\left(i=1,2,\cdots ,K\right)$. This ensures that $B_i$ (generated by client $i$) and $B_j$ (generated by client $j\left)\right(i\ne j)$ are commutative, i.e., $B_i{B}_j=B_j{B}_i$ (Appendix A). Client $i$ computes $X_i^{\text{enc}}=A_i{X}_i{B}_i$ and sends $X_i^{\text{enc}}$ to other clients for external encryption.

External encryption

Upon receiving $X_i^{\text{enc}}=A_i{X}_i{B}_i$ from client $i$, client $i+1$ further encrypts it using $B_{i+1}$ (i.e., $X_i^{\text{enc}}=A_i{X}_i{B}_i{B}_{i+1}$) and sends the updated ciphertext $X_i^{\text{enc}}$ to client $i+2$. Client $i+2$ then encrypts the ciphertext using $B_{i+2}$ and sends the ciphertext to client $i+3$. After all the $K-1$ clients complete the external encryption, the ciphertext is in the form of $X_i^{\text{enc}}=A_i{X}_{i}B$ where $B=\prod _i^k{B}_i=B_1{B}_2\cdots B_K.X_i^{\text{enc}}$ is then sent to the server.

To build the ridge regression model, client $i$ computes and sends $B_i^T{B}_i$ to client $i+1$. Client $i+1$ calculates $B_j^T{B}_i^T{B}_i{B}_j$ and sends it to client $i+2$. Each of the $K$ clients conducts the encryption sequentially. Once all clients complete encryption, $\prod _{j=1}^K{B}_i^T{B}_i$is sent to the server for ridge model computation.

Algorithm 2 describes the detailed encryption procedures and Figure 2 gives an example of the encryption procedures with three clients. Table III summarizes the internal and external encryption procedures. The primary goal of the internal encryption layer is to protect against malicious adversaries. To preserve data utility, the data are further encrypted by the other clients using commutative matrices in the external encryption layer.

Fig. 2:

An example showing the proposed encryption procedures including data transformation details with three clients. $X_i$ denotes the data collected by client $i(i=1,\cdots ,K=3).Y_i^T{X}_i$ is integrated into $X_i$ prior to the data encryption. Data are encrypted by all the clients sequentially. The arrows connect the origin and endpoint of the data transmission.

Table III:

Encryption details for the plaintext $X_i$ (owned by client $i$)

Encryption layer	Client	Rationale	Affected by malicious adversaries?	Encryption matrix
Internal	$i$	Withstand malicious adversaries	No	$A_i,B_i$
External	$\mathrm{1,2},\cdots ,i-1,i+1,\cdots ,K$	Preserve data utility	Yes	${\prod }_{j\ne i}{B}_j$

Modeling

Upon receiving $X^{\text{enc}}=\left(\begin{array}{c}{A}_1{X}_{1}B\\ ⋮\\ A_K{X}_{K}B\end{array}\right)(i=1,\cdots ,K)$ (and $\prod _{j=1}^K{B}_i^T{B}_i$ for ridge regression), the server decrypts $A_i$ to get $\left(\begin{array}{c}{X}_{1}B\\ ⋮\\ X_{K}B\end{array}\right)=XB$. For the subsequent analysis, the server defines $XB$ as $X^{\text{enc}}$ (i.e., $X^{\text{enc}}\triangleq XB$) and further eliminates pseudo records in $X^{\text{enc}}$. The detailed procedures are described in Algorithm 3. The server retrieves the encrypted outcome from $XB$ and denotes it as $Z_i^{\text{enc}}$. According to the encryption procedure, $Z_i^{\text{enc}}=Y_i^T{X}_{i}B$. Then the server derives a $q$-dimensional vector $Z^{\text{enc}}\triangleq {\sum }_{i=1}^K{Z}_i^{\text{enc}}$. Given the encrypted data, the Newton update becomes

$${\left({\beta }^{\text{new}}\right)}^{\text{enc}}={\left[{\left(X^{\text{enc}}\right)}^T{W}^{\text{enc}}{X}^{\text{enc}}+\Lambda B^{T}B\right]}^{-1}\times \left[{\left(X^{\text{enc}}\right)}^T{W}^{\text{enc}}{X}^{\text{enc}}{\left({\beta }^{\text{old}}\right)}^{\text{enc}}+{\left(Z^{\text{enc}}\right)}^T-{\left(X^{\text{enc}}\right)}^T{p}^{\text{enc}}\right]$$ (11)

where $p^{\text{enc}}={\left({\left[p\left(x_1;{\beta }^{\text{old}}\right)\right]}^{\text{enc}},\cdots ,{\left[p\left(x_n;{\beta }^{\text{old}}\right)\right]}^{\text{enc}}\right)}^T$,

$${\left[p\left(x_i;{\beta }^{\text{old}}\right)\right]}^{\text{enc}}=p\left(x_i^{\text{enc}};{\left({\beta }^{\text{old}}\right)}^{\text{enc}}\right)=\frac{\text{exp}\left[{\left(x_i^{\text{enc}}\right)}^T{\left({\beta }^{\text{old}}\right)}^{\text{enc}}\right]}{1+\text{exp}\left[{\left(x_i^{\text{enc}}\right)}^T{\left({\beta }^{\text{old}}\right)}^{\text{enc}}\right]}$$ (12)

for $i=1,\cdots ,n,W^{\text{enc}}$ is a $n\times n$ diagonal matrix with the $i$-th diagonal element being ${\left[p\left(x_i;{\beta }^{\text{old}}\right)\right]}^{\text{enc}}\left\{1-{\left[p\left(x_i;{\beta }^{\text{old}}\right)\right]}^{\text{enc}}\right\}$, Λ is a matrix of zeros for the non-regularized logistic model while Λ is a diagonal matrix with the diagonal elements being $\{0,\lambda ,\cdots ,\lambda \}$ for ridge regression. The server computes model estimates using Equation 11 until converged (e.g., the minimum of the absolute difference between (${\left.{\beta }^{\text{new}}\right)}^{\text{enc}}$ and ${\left({\beta }^{\text{old}}\right)}^{\text{enc}}$ is smaller than 10⁻⁶).

Theorem 1. The privacy-preserving logistic model converges as long as the non-secure logistic model converges.

Proof. Let ${\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}$ denote the initial point in Newton’s method within the privacy-preserving logistic model. According to Theorem 9 (Appendix C), it is equivalent to setting $B\left({\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)$ as the initial point within the non-secure model. Given the initial point $B\left({\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)$, suppose the non-secure model converges after $s$ iterations with the model estimate being $\beta$. Based on Theorem 9 (Appendix C), our privacy-preserving model also converges after $s$ iterations with the model estimate being ${\beta }^{\text{enc}}=B^{-1}\beta$. □

Decryption

The server sends the converged model estimate ${\beta }^{\text{enc}}$ to the clients. As shown in Theorem 9 (Appendix C), we have ${\beta }^{\text{enc}}=\prod _{i=1}^K{\left(B_i\right)}^{-1}\beta$ where $\beta$ is the estimate of the non-secure model. To get the true model estimate $\beta$, client $i(i=1,\cdots ,K$) uses the encryption matrix $B_i$ to decrypt ${\beta }^{\text{enc}}$. Figure 3 shows the detailed decryption procedure. $\beta =(\prod _{i=1}^K{B}_i){\beta }^{\text{enc}}$is the result once all clients complete decryption.

Fig. 3:

The decryption procedure. Client $i$ decrypts ${\beta }^{\text{enc}}$ sequentially $(i=1,\cdots ,K)$. The data above the arrow are those transferred among clients.

B. Multiclass classification

Our scheme can be modified to solve the multiclass classification problem. Suppose the outcome contains a total of $e_y$ classes. Client $i$ defines $e_y$ sub-outcomes $Y_{i\left(1\right)},Y_{i\left(2\right)},\cdots$,$Y_{i\left(e_y\right)}$ using the indicator function $\left({\mathbf{1}}_j\left(Y_i\right)\right)$ as follows.

$$Y_{i\left(j\right)}={\mathbf{1}}_j\left(Y_i\right)=\left\{\begin{array}{ll}1& Y_i=j,\\ 0& Y_i\ne j,\end{array}\text{for}j=1,\cdots ,e_y\right..$$ (13)

Following the above described scheme, client $i$ calculates $Z_{i\left(j\right)}=Y_{i\left(j\right)}^T{X}_i\left(j=1,\cdots ,e_y\right)$. Before data encryption, client $i$ adds $Z_{i\left(1\right)},\cdots ,Z_{i\left(e_y\right)}$ to the feature matrix $X_i$ as the last $e_y$ rows. Define the sub-outcome for the $j$-th classes as $Z_{\left(j\right)}=\sum _{i=1}^K{Y}_{i\left(j\right)}^T{X}_i$ for $j=1,\cdots ,e_y$. After data encryption, the server performs the secure logistic model computation (Algorithm 3) for each of the $e_y$ sub-outcomes using each pair of feature matrix and outcome $\left\{X^{\text{enc}},Z_{\left(j\right)}^{\text{enc}}\right\}$ for $j=1,\cdots ,e_y$.

C. Lossy compression with SZ

To reduce the communication cost, we employ the lossy compression technique with SZ [55] for data compression. Lossy compression with $\text{SZ}$ is an error-bounded lossy compression scheme [55]–[57]. In our scheme, the data are compressed before being transferred between clients and the server.

D. Verification: malicious behavior detection

To identify if any client has conducted malicious behavior, a designated pseudo outcome $Y_s$ is subjected to the same procedures as the original outcome $Y$. Assuming all clients adhere to our scheme for both $Y$ and $Y_s$, predetermined outputs are anticipated after the verification stage.

Specifically, client $i$ defines a constant ${\tau }_i\triangleq Y_i^T{X}_i\mathbf{1}$ where $\mathbf{1}=(\mathrm{1,1},\cdots ,1{)}^T$ and shares ${\tau }_i$ with the server. For the verification, client $i$ generates a pseudo outcome $Y_{si}\triangleq X_i^T{X}_i\mathbf{1}.Y_s$ is the sum of $Y_{si}$, denoted as $Y_s\triangleq \sum _{i=\mathbf{1}}^K{Y}_{si}$. $Y_s$ can be expressed as $Y_s=X^{T}X1$. Client $i$ further encrypts $Y_{si}$ using $B_i$ (i.e., $Y_{si}^{\text{enc}}=B_i^T{Y}_{si}$) and sends the ciphertext $Y_{si}^{\text{enc}}$ to the other clients. $Y_{si}^{\text{enc}}$ is subsequently encrypted by client $j$ using $B_j(j=1,\cdots ,i-1,i+1,\cdots ,K)$. After all the clients have completed the encryption process, $Y_{si}^{\text{enc}}$ is shared with the server. Upon receiving $Y_{si}^{\text{enc}}$, the server verifies if $Z_i^{\text{enc}}{\left[{\left(X_i^{\text{enc}}\right)}^T{X}_i^{\text{enc}}\right]}^{-1}{Y}_{si}^{\text{enc}}={\tau }_i$ where $Z_i^{\text{enc}}=Y_i^T{X}_{i}B$. If any client exhibits malicious behavior during the encryption process, the equation will not hold (Theorem 2). Since ${\left(X^{T}X\right)}^{-1}{X}^{T}X\mathbf{1}=\mathbf{1}$, we design the following process to verify if the clients follow the proposed decryption procedure. The server first calculates the sum of encrypted pseudo outcomes (i.e., $Y_s^{\text{enc}}\triangleq \sum _{i=1}^K{Y}_{si}^{\text{enc}}$) and the estimate ${\beta }_s^{\text{enc}}\triangleq {\left[{\left(X^{\text{enc}}\right)}^T{X}^{\text{enc}}\right]}^{-1}{Y}_s^{\text{enc}}.{\beta }_s^{\text{enc}}$ can be simplified as follows.

$$\begin{gathered} {\beta }_s^{\text{enc}}={\left[{\left(X^{\text{enc}}\right)}^T{X}^{\text{enc}}\right]}^{-1}{Y}_s^{\text{enc}}={\left[{\left(X^{\text{enc}}\right)}^T{X}^{\text{enc}}\right]}^{-\mathbf{1}}\left[\sum _{i=\mathbf{1}}^K{Y}_{si}^{\text{enc}}\right] \\ ={\left[{\left(X^{\text{enc}}\right)}^T{X}^{\text{enc}}\right]}^{-1}\left(\sum _{i=1}^K{B}^T{X}_i^T{X}_i\mathbf{1}\right) \\ ={\left[(XB{)}^{T}XB\right]}^{-1}{B}^T{X}^{T}X1=B^{-1}\mathbf{1}. \end{gathered}$$ (14)

The server shares ${\beta }_s^{\text{enc}}$ with the client who performs the verification (e.g., client 1). To confirm that no malicious behavior was conducted within the decryption process, client 1 combines ${\beta }_s^{\text{enc}}$ with ${\beta }^{\text{enc}}$. Specifically, client 1 generates two random constants, ${\alpha }_1$ and ${\alpha }_2$, and defines a new estimate as $(\tilde{\beta }{)}^{\text{enc}}\triangleq {\alpha }_1{\beta }^{\text{enc}}+{\alpha }_2{\beta }_s^{\text{enc}}$. $(\tilde{\beta }{)}^{\text{enc}}$ is decrypted by all the clients following the procedure in Figure 3. Let $\tilde{\beta }$ be the decrypted estimate. Upon obtaining $\tilde{\beta }$, client 1 calculates ${\beta }_s\triangleq \frac{1}{{\alpha }_2}\left[\tilde{\beta }-{\alpha }_{1}B{\beta }^{\text{enc}}\right]$ where $B{\beta }^{\text{enc}}$ is the decrypted model estimate. ${\beta }_s$ is expected to be a vector of ones if all the clients correctly decrypt both $(\tilde{\beta }{)}^{\text{enc}}$ and ${\beta }^{\text{enc}}$ following the proposed decryption procedure (Theorem 3). Algorithm 4 summarizes the verification process.

In order to preserve data utility, all the clients need to follow three encryption criteria. We utilize a case study involving two clients to better illustrate these criteria. The encryption procedures are shown in Figure 4. Initially, it is necessary for clients to employ uniform encryption matrices to encrypt datasets owned by other clients. This condition implies that $B_3=B_2$ and $B_4=B_1$ in the example. Additionally, these encryption matrices must be commutative with each other, which implies that the multiplication of $B_1$ and $B_2$ equals the multiplication of $B_2$ and $B_1$, denoted as $B_1{B}_2=B_2{B}_1$. Thirdly, clients use uniform inputs across the entire encryption process. More precisely, each client does not alter the data owned by other clients during data encryption. To violate the third criterion, client 2 may selectively encrypt specific rows within the dataset $A_1{X}_1{B}_1$ or substitute $A_1{X}_1{B}_1$ with fake data prior to transmitting it to the server.

Fig. 4:

An example of the proposed encryption procedures with two clients. $X_i$ denotes the data collected by client $i(i=\mathrm{1,2}).Y_i^T{X}_i$ is integrated into $X_i$ prior to the data encryption. The internal encryption is performed by each client, encrypting its own data internally before transmitting it to other clients. The external encryption is further conducted by the other clients. Finally, the server decrypts $A_i$ from the data it has received and retrieves the encrypted outcome information.

Let ${\left({\tilde{X}}_1\right)}^{\text{enc}}$ and ${\left({\tilde{X}}_2\right)}^{\text{enc}}$ denote the ciphertexts transmitted to the server from client 2 and client 1, respectively. The server then decrypts $A_i$ and extracts the encrypted outcome ${\left({\tilde{Z}}_i\right)}^{\text{enc}}(i=\mathrm{1,2})$. If all the clients adhere to these three criteria, the server should get the ciphertexts in Equations 15 and 16 (C-1, C-2, and C-3 refer to criteria 1 through 3).

$$X^{\text{enc}}=\left[\begin{array}{c}{\left({\tilde{X}}_1\right)}^{\text{enc}}\\ {\left({\tilde{X}}_2\right)}^{\text{enc}}\end{array}\right]\stackrel{\text{C}-3}{=}\left[\begin{array}{c}{A}_1{X}_1{B}_1{B}_3\\ A_2{X}_2{B}_2{B}_4\end{array}\right]\stackrel{\text{C}-1}{=}\left[\begin{array}{c}{A}_1{X}_1{B}_1{B}_2\\ A_2{X}_2{B}_2{B}_1\end{array}\right]\stackrel{\text{C}-2}{=}\left[\begin{array}{c}{A}_1{X}_1{B}_1{B}_2\\ A_2{X}_2{B}_1{B}_2\end{array}\right]=\left[\begin{array}{c}{A}_1{X}_1\\ A_2{X}_2\end{array}\right]B_1{B}_2\stackrel{\text{Decryption}}{\Rightarrow }\left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]B_1{B}_2.$$ (15)

$$Z^{\text{enc}}=\left[\begin{array}{c}{\left({\tilde{Z}}_1\right)}^{\text{enc}}\\ {\left({\tilde{Z}}_2\right)}^{\text{enc}}\end{array}\right]\stackrel{\text{C}-3}{=}\left[\begin{array}{c}{Y}_1^T{X}_1{B}_1{B}_3\\ Y_2^T{X}_2{B}_2{B}_4\end{array}\right]\stackrel{\text{C}-1}{=}\left[\begin{array}{c}{Y}_1^T{X}_1{B}_1{B}_2\\ Y_2^T{X}_2{B}_2{B}_1\end{array}\right]\stackrel{\text{C}-2}{=}\left[\begin{array}{c}{Y}_1^T{X}_1{B}_1{B}_2\\ Y_2^T{X}_2{B}_1{B}_2\end{array}\right]=\left[\begin{array}{c}{Y}_1^T{X}_1\\ Y_2^T{X}_2\end{array}\right]B_1{B}_2.$$ (16)

To preserve data utility, 1) $X_i$ and $X_j(i\ne j)$ need to be encrypted by the same encryption matrix $\prod _{K=1}^K{B}_k$, and 2) $X_i$ and $Y_i^T{X}_i(i=1,\cdots ,K)$ need to be encrypted by the same encryption matrix $\prod _{K=1}^K{B}_k$. The malicious behavior of any client during data encryption results in a breach of one or both of these two requirements, thereby affecting the utility of the data.

Theorem 2. The proposed verification algorithm can identify the malicious behavior during the encryption process.

Proof. Firstly, the server verifies if $X_i$ and $X_j(i\ne j)$ are encrypted by the same encryption matrix by checking whether the first rows in $X_i^{\text{enc}}$ and $X_j^{\text{enc}}$ are identical. To meet this requirement, all clients must adhere to the three encryption criteria indicated in Equations 15 and 16. Secondly, the server checks whether $Z_i^{\text{enc}}{\left[{\left(X_i^{\text{enc}}\right)}^T{X}_i^{\text{enc}}\right]}^{-1}{Y}_{si}^{\text{enc}}$ equals ${\tau }_i$ to ascertain if $X_i$ and $Y_i^T{X}_i$ have been encrypted using the same encryption matrix. Upon receiving $Z_i^{\text{enc}}=Y_i^T{X}_{i}B$ and $X_i^{\text{enc}}=X_{i}B$, the server verifies if $B=B$ by calculating

$$\begin{gathered} Z_i^{\text{enc}}{\left[{\left(X_i^{\text{enc}}\right)}^T{X}_i^{\text{enc}}\right]}^{-1}{Y}_{si}^{\text{enc}} \\ =Y_i^T{X}_{i}B{\left({\left[X_{i}B\right]}^T{X}_{i}B\right)}^{-1}{B}^T{Y}_{si} \\ =Y_i^T{X}_{i}B{B}^{-1}{\left(X_i^T{X}_i\right)}^{-1}{\left(B^T\right)}^{-1}{B}^T{X}_i^T{X}_i\mathbf{1} \\ \stackrel{B=B}{=}{Y}_i^T{X}_i\mathbf{1}={\tau }_i \end{gathered}$$ (17)

for $i=1,\cdots ,K$. The server affirms the absence of any malicious activities by validating the fulfillment of the two conditions mentioned above. □

In the decryption procedures, client $i$ should use the encryption matrix $B_i$ to decrypt ${\beta }^{\text{enc}}$ (output in Algorithm 3) (decryption criterion).

Theorem 3. The proposed verification algorithm can identify if the client violates the decryption criterion.

Proof. In our scheme, both ${\beta }^{\text{enc}}$ and $(\tilde{\beta }{)}^{\text{enc}}$ need to be decrypted following the decryption procedures. Let $B{\beta }^{\text{enc}}$ and $B(\tilde{\beta }{)}^{\text{enc}}$ be the decrypted data of ${\beta }^{\text{enc}}$ and $(\tilde{\beta }{)}^{\text{enc}}$, respectively. Since

$$(\tilde{\beta }{)}^{\text{enc}}\triangleq {\alpha }_1{\beta }^{\text{enc}}+{\alpha }_2{\beta }_s^{\text{enc}},$$ (18)

we have

$$\tilde{\beta }\triangleq B(\tilde{\beta }{)}^{\text{enc}}={\alpha }_{1}B{\beta }^{\text{enc}}+{\alpha }_{2}B{\beta }_s^{\text{enc}}.$$ (19)

Suppose client $i(i=1,\cdots ,K)$ follows the proposed decryption procedures to decrypt both ${\beta }^{\text{enc}}$ and $(\tilde{\beta }{)}^{\text{enc}},B$ and $B$ should be identical (i.e., $B=B=B$ where $B=\prod _{i=1}^K{B}_i$). So

$$\begin{gathered} {\beta }_s\triangleq \frac{1}{{\alpha }_2}\left[\tilde{\beta }-{\alpha }_{1}B{\beta }^{\text{enc}}\right] \\ =\frac{1}{{\alpha }_2}\left[{\alpha }_{1}B{\beta }^{\text{enc}}+{\alpha }_{2}B{\beta }_s^{\text{enc}}-{\alpha }_{1}B{\beta }^{\text{enc}}\right] \\ \stackrel{B=B=B}{=}{\beta }_s^{\text{enc}}. \end{gathered}$$ (20)

According to Equation 14,

$$B{\beta }_s^{\text{enc}}=B{\left[{\left(X^{\text{enc}}\right)}^T{X}^{\text{enc}}\right]}^{-1}{Y}_s^{\text{enc}}=\mathbf{1}.$$ (21)

Based on Equations 20 and 21, ${\beta }_s$ is expected to be a vector of ones if no malicious activities are involved in the decryption procedures. Therefore our verification stage can identify whether the client breaks the decryption criterion. □

VI. Security analysis

The encryption matrices $A_i$ and $B_i$ may be recovered if ciphertexts of different plaintexts are distinguishable. Once the encryption matrix is recovered, the client can recover other clients’ data. Potential attacks to achieve such goals include CPA, KPA, COA and CCA [34]–[36], [40] as described in Section III-C. It is impossible to perform CCA for our scheme because adversaries cannot obtain the decryption of any ciphertext. Since CPA is more threatening than KPA and COA, a secure scheme is resilient to KPA and COA if it protects against CPA.

CPA [37], [52]–[54] is reasonable in our scheme. Consider a robust threat model in which all clients, except one, can be compromised in a collusion attack (Figure 5). Suppose client 1 is the only honest client. In the external encryption layer, client $i(i\ne 1)$ generates fake data $X_i$ and sends it to client 1 for encryption. Client 1 uses $B_1$ to encrypt $X_i$ and returns the ciphertext to the other clients for further encryption. During the process, colluded clients share received ciphertexts from agency 1 and are able to match each plaintext $X_i$ with its ciphertext $X_i{B}_1$. In a collusion attack, colluded clients cooperate as a group to share plaintexts and ciphertexts with each other. So the colluded group is able to insert arbitrary plaintexts and get the corresponding ciphertexts for the purpose of CPA. The colluded group will try to first recover the encryption matrix $B_1$ and then retrieve the plaintext $X_1$ owned by client 1.

Fig. 5:

An example of the strong threat model with all the clients except one can be compromised. Suppose client 1 is the only honest client and $B_1$ denotes the commutative matrix for data encryption. $X_i$ denotes the plaintext from client $i(i=2,\cdots ,K)$.

To be resilient to CPA, the encrypted data in our privacy-preserving model should have indistinguishability for any random plaintexts. In this section, we demonstrate that the ciphertexts of two arbitrary plaintexts are indistinguishable in our privacy-preserving scheme.

Define the encryption functions $f_{M,l}\left(X\right)\triangleq MX$ and $f_{M,r}\left(X\right)\triangleq XM$ where $M$ is a random Gaussian matrix, i.e, each element of $M$ follows Gaussian distribution. Since $B_i=B_0\left(\sum _{j=1}^q{b}_j{B}_0^{(j-1)}\right)$, the internal encryption function $f\left(X_i\right)=A_i{X}_i{B}_i$ can be split into 3 sub-functions. Specifically, $f\left(X_i\right)=A_i{X}_i{B}_i=A_i{X}_i{B}_0\left(\sum _{j=1}^q{b}_j{B}_0^{(j-1)}\right)=f_r\left(f_{B_0,r}\left(f_{A_i,l}\left(X_i\right)\right)\right)$ where $f_{A_i,l}\left(X\right)=A_{i}X,f_{B_0,r}\left(X\right)=X{B}_0$, and $f_r\left(X\right)=X\left(\sum _{j=1}^q{b}_j{B}_0^{(j-1)}\right)$.

The clients have access to the ciphertext $A_i{X}_{i}B$. In contrast, the server receives the encryption matrices $A_i$ from client $i(i=1,\cdots ,K)$ and derives the ciphertext $X_{i}B=A_i^{-1}{A}_i{X}_{i}B.A_i$ is only used in the internal encryption layer and the function $f_{A_i,l}\left(X\right)=A_{i}X$ is employed to ensure that clients cannot conduct effective CPA. We first prove that records in $A_i{X}_{i}B$ are indistinguishable to the clients (Section VI-A) and then demonstrate that records in $X_{i}B$ are indistinguishable to the server (Section VI-B).

A. Indistinguishability of $A_i{X}_{i}B$: security against clients

Theorem 4. Given the Gaussian matrix encryption function $f_{M,l}\left(X\right)=MX$, where $M$ is a random Gaussian matrix, the worst-case lower and upper bounds on $d_{TV}\left(p_1,p_2\right)$ are

$$d_{TV,\text{low}}=1-{\left(\frac{2∥x_{c,1}∥∥x_{c,2}∥}{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}\right)}^{q/2},$$ (22)

$$d_{TV,up}=\sqrt{1-{\left(\frac{2∥x_{c,1}\mid ∥∥x_{c,2}∥}{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}\right)}^q}.$$ (23)

where $p_1=P\left(z\mid x_{c,1}\right),p_2=P\left(z\mid x_{c,2}\right)$ and $x_{c,h}(h=\mathrm{1,2})$ are two arbitrary columns in $X$.

Proof. Based on the proof of [[35], Lemma 1], the covariance matrix of $z_h=M{x}_{c,h}$ conditioned on the plaintext $x_{c,h}$ is $C_h={∥x_{c,h}∥}^{2}I$ where $∥x_{c,h}∥$ denotes the Euclidean norm of $x_{c,h}$ and $I$ is the identity matrix. Therefore, $C_3\triangleq \frac{C_1+C_2}{2}=\left(\frac{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}{2}\right)I$. Because $C_1,C_2$ and $C_3$ are diagonal matrices, we can get their determinants as

$$\left|C_h\right|={\left({∥x_{c,h}∥}^2\right)}^q,h\in \left\{\mathrm{1,2}\right\}$$ (24)

and

$$\left|C_3\right|={\left(\frac{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}{2}\right)}^q.$$ (25)

So

$$d_H\left(p_1,p_2\right)=\sqrt{1-\frac{{\left|C_1\right|}^{\frac{1}{4}}{\left|C_2\right|}^{\frac{1}{4}}}{{\left|C_3\right|}^{\frac{1}{2}}}}=\sqrt{{\left.1-\frac{2∥x_{c,1}∥∥x_{c,2}∥}{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}\right)}^{q/2}}.$$ (26)

According to the inequality relation between the Hellinger distance and the TV distance (Equation 9), we can get the lower and upper bounds of the TV distance as follows.

$$\begin{gathered} d_{\text{TV},\text{low}}=d_{\text{H}}^2\left(p_1,p_2\right)=1-{\left(\frac{2∥x_{c,1}∥∥x_{c,2}∥}{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}\right)}^{q/2}, \\ {d}_{\text{TV},\text{up}}=d_{\text{H}}\left(p_1,p_2\right)\sqrt{2-d_{\text{H}}^2\left(p_1,p_2\right)} \\ =\sqrt{1-{\left(\frac{2∥x_{c,1}∥∥x_{c,2}∥}{{∥x_{c,1}∥}^2+{∥x_{c,2}∥}^2}\right)}^q}. \end{gathered}$$ (27)

□

Theorem 5. Given the Gaussian matrix encryption function $f_{M,r}\left(X\right)=XM$, where $M$ is a random Gaussian matrix, the worst-case lower and upper bounds on $d_{TV}\left(p_1,p_2\right)$ are

$$d_{TV,low}=1-{\left(\frac{2∥x_1∥∥\mid x_2∥}{{∥x_1∥}^2+{∥x_2∥}^2}\right)}^{q/2},$$ (28)

$$d_{TV,up}=\sqrt{1-{\left(\frac{2∥x_1∥∥\mid x_2∥}{{∥x_1∥}^2+{∥x_2∥}^2}\right)}^q}$$ (29)

where $p_1=P\left(z\mid x_1\right),p_2=P\left(z\mid x_2\right)$ and $x_h(h=\mathrm{1,2})$ are two arbitrary rows in $X$.

Proof. The proof is identical to that provided in Theorem 4. □

Corollary 1. The success probability of an adversary in the indistinguishability experiment is bounded by

$$p_d\le \frac{1}{2}+\frac{1}{2}\sqrt{1-{\left(\frac{2∥x_1∥∥x_2∥}{{∥x_1∥}^2+{∥x_2∥}^2}\right)}^q.}$$ (30)

If each plaintext has constant Euclidean norm (i.e., $∥x_1∥=∥x_2∥$ for two random records $x_1$ and $x_2$), the cryptosystem has indistinguishability since $p_d\le 0.5$.

Corollary 1 ensures that no adversary can learn any partial information about the plaintext from a given ciphertext, as long as each plaintext has constant Euclidean norm. Because the Euclidean norms of all the columns in the plaintext $(\left.X_i\right)$ are designed to be the same (Section V-A), any two arbitrary columns in $A_i{X}_i$ are indistinguishable.

Theorem 6. The TV distance does not increase by encryption functions $f_{B_0,r}$ and $f_r$. In other words, $d_{TV,up}\left({\tilde{p}}_1,{\tilde{p}}_2\right)\le d_{TV,up}\left(p_1,p_2\right)$ where ${\tilde{p}}_h$ and $p_h$ denote the probability distribution of a random row in $A_i{X}_h{B}_i$ and $A_i{X}_h$, respectively $(h\in \{\mathrm{1,2}\left\}\right)$.

Proof. Hellinger distance can be expressed as a function of Rényi divergence [58], i.e.,

$$d_{\text{H}}\left(p_1,p_2\right)=\sqrt{2\left(1-e^{-\frac{1}{2}{D}_{\frac{1}{2}}\left(p_1\mid p_2\right)}\right)}.$$ (31)

where $D_{\frac{1}{2}}\left(p_1\mid p_2\right)$ denotes the Rényi divergence of $p_1$ from $p_2$. Based on the data processing inequality [[58], Theorem 1], $D_{\frac{1}{2}}\left({\tilde{p}}_1\mid {\tilde{p}}_2\right)\le D_{\frac{1}{2}}\left(p_1\mid p_2\right)$. So $d_{\text{H}}\left({\tilde{p}}_1,{\tilde{p}}_2\right)\le d_{\text{H}}\left(p_1,p_2\right)$. Because $0\le d_{\text{H}}\left(p_1,p_2\right)\le 1$ and function $u\left(\sqrt{2-u^2}\right)$ is monotonically increasing on $0\le u\le 1$,

$$\begin{gathered} d_{\text{TV}}\left({\tilde{p}}_1,{\tilde{p}}_2\right)\le d_{\text{H}}\left({\tilde{p}}_1,{\tilde{p}}_2\right)\sqrt{2-d_{\text{H}}^2\left({\tilde{p}}_1,{\tilde{p}}_2\right)} \\ \le d_{\text{H}}\left(p_1,p_2\right)\sqrt{2-d_{\text{H}}^2\left(p_1,p_2\right).} \end{gathered}$$ (32)

In other words, $d_{\text{TV},\text{up}}\left({\tilde{p}}_1,{\tilde{p}}_2\right)\le d_{\text{TV},\text{up}}\left(p_1,p_2\right)$. □

Corollary 2. Clients are unable to learn any partial information of $X_i$ in polynomial time within our privacy-preserving scheme, thereby ensuring that our scheme is resilient to the CPA conducted by the colluded clients.

Proof. According to Theorems 4, 6 and Corollary 1, the internal encryption function $f\left(X_i\right)=A_i{X}_i{B}_i$ is indistinguishable. Theorem 6 indicates that the TV distance does not increase by the external encryption layer and thus $A_i{X}_{i}B$ is indistinguishable where $B=\prod _i^K{B}_i.$ Therefore, clients cannot perform effective CPA to learn the sensitive information of the other clients. □

B. Indistinguishability of $X_{i}B$ : security against the server

We further illustrate that any two arbitrary rows in $X_{i}B$ are indistinguishable. In the initial model training process (Algorithm 3), the server decrypts $A_i$ from $A_i{X}_{i}B$ and thus has access to $X_{i}B$. With the indistinguishability of the encryption function $f\left(X_i\right)=X_{i}B$, the server cannot learn any partial information about $X_i$ in polynomial time (Corollary 3).

Theorem 7. Given $\gamma =\frac{∥x_1∥}{∥x_2∥}(x_1$ and $x_2$ are two arbitrary rows in $X_i$) and a negligible function $ϵ\left(q\right),x_{1}B$ and $x_{2}B$ are indistinguishable if $\gamma$ satisfies $\gamma +\frac{1}{\gamma }\le 2{\left[1-4{ϵ}^2\left(q\right)\right]}^{-1/q}$ where $q$ is the number of features.

Proof. The encryption function $f\left(X_i\right)=X_{i}B=X_i{B}_0\left(\sum _{j=1}^q{b}_j{B}_0^{(j-1)}\right)$ can be split into 2 sub-functions, i.e., $f_{B_0,r}\left(X\right)=X{B}_0$ and $f_r\left(X\right)=X\left(\sum _{j=1}^q{b}_j{B}_0^{(j-1)}\right)$. According to Theorem 5 and the data processing inequality [[58], Theorem 1], the upper bound of the TV distance between $P\left(x_{1}B\mid x_1\right)$ and $P\left(x_{2}B\mid x_2\right)$ is $d_{\text{TV,up}}=\sqrt{1-{\left(\frac{2\gamma }{{\gamma }^2+1}\right)}^q}$. Indistinguishability requires that $p_d\le \frac{1}{2}+ϵ\left(q\right)$. To achieve this, we require that $d_{\text{TV},\text{up}}\le 2ϵ\left(q\right)$. So ${\left[1-4{ϵ}^2\left(q\right)\right]}^{1/q}\le \frac{2\gamma }{{\gamma }^2+1}$ for a given $q$. This leads to $\gamma +\frac{1}{\gamma }\le 2{\left[1-4{ϵ}^2\left(q\right)\right]}^{-1/q}$. □

We propose a data transformation method, i.e., “ $d$-transformation”, to ensure the indistinguishability for arbitrary records, irrespective of whether they possess a consistent Euclidean norm or not. Assume $∥x_1∥<∥x_2∥,\frac{\gamma }{{\gamma }^2+1}$ is monotonically increasing on $0<\gamma <1$. The decrease in $\gamma$ leads to the increase in $d_{\text{TV,up}}=\sqrt{1-{\left(\frac{2\gamma }{{\gamma }^2+1}\right)}^q}$. To maintain $d_{\text{TV},\text{up}}$ within an acceptable range, the Euclidean norms of any two arbitrary records need to be close to each other (i.e., $\gamma \to 1$). To achieve this, we perform the $\mathit{d}$-transformation for each record. More precisely, we use $\mathit{d}$ (i.e., a vector of constant $d$) instead of a vector of ones as the intercept in $X_i$. The first element of $x_1$ and $x_2$ becomes $d$ instead of 1. As presented in Figure 6A, a large $d$ ensures that $d_{\text{TV},\text{up}}$ is close to 0 for a fixed $q$. For fixed $d,d_{\text{TV},\text{up}}$ increases as $q$ rises (Figure 6B).

Fig. 6:

Fig. A: $d_{\text{TV},\text{up}}$ given $q=100$ and different values of $d$. Fig. B: $d_{\text{TV,up}}$ given $d=\mathrm{2,000}$ and different values of $q.\gamma =\frac{∥x_1∥}{∥x_2∥}$ and $0<\gamma <1$.

Theorem 8. The $d$-transformation ensures indistinguishability for any $q$ and $\gamma$.

Proof. As described in Theorem 7, our scheme achieves indistinguishability depending on $\gamma$. Since $\gamma +\frac{1}{\gamma }$ is monotonically decreasing on $0<\gamma <1$, there exists a minimum threshold ${\gamma }_{-}$ such that $\gamma +\frac{1}{\gamma }\le 2{\left[1-4{ϵ}^2\left(q\right)\right]}^{-1/q}$ for any $\gamma >{\gamma }_{-}$. Let ${\gamma }_{\text{original}}=\frac{∥x_1∥}{∥x_2∥}$ where $x_1$ and $x_2$ are two random records in the plaintext. Define ${\gamma }_{\text{new}}=\frac{∥{\tilde{x}}_1∥}{∥{\tilde{x}}_2∥}$ where ${\tilde{x}}_1$ and ${\tilde{x}}_2$ are the $d$-transformed $x_1$ and $x_2$, respectively. So we have ${\gamma }_{\text{new}}\triangleq \frac{\sqrt{{\gamma }_{\text{original}}^2+d^2}}{\sqrt{1+d^2}}$. For $0<\gamma <1,{\gamma }_{\text{new}}$ increases as $d$ goes up. So there exists a constant $d$ such that ${\gamma }_{\text{new}}\to 1$, implying that ${\gamma }_{\text{new}}+\frac{1}{{\gamma }_{\text{new}}}\le 2{\left[1-4{ϵ}^2\left(q\right)\right]}^{-1/q}$ given any $q$ and $\gamma$. To conclude, the $d$-transformation guarantees indistinguishability for any $q$ and $\gamma$. □

Considering that $ϵ\left(p\right)$ (e.g., $ϵ\left(p\right)\triangleq 2^{-q}$) can be large given small $q$, we set an upper bound for $ϵ\left(q\right)$ (e.g., $ϵ\left(p\right)=\text{min}\left\{{10}^{-5},2^{-q}\right\}$). As shown in Figure 6B, $d_{\text{TV},\text{up}}$ increases when $q$ goes up. Given $ϵ\left(p\right)=\text{min}\left\{{10}^{-5},2^{-q}\right\},d=\mathrm{2,000}$ is sufficient to ensure indistinguishability when $q\le 20$ (Figure 6B). For a large $q$, clients follow Algorithm 1 to select a constant $d$ such that the encryption function in our scheme has indistinguishability.

Corollary 3. With the $d$-transformation (Algorithm 1), the encryption function $f\left(X_i\right)=X_{i}B$ ensures the indistinguishability among any arbitrary records in $X_i$. This demonstrates that the server cannot learn any partial information about $X_i$ in polynomial time.

Proof. Given any negligible function $ϵ\left(q\right)$ and data $X_i$, Algorithm 1 selects a constant $d$ such that $\gamma +\frac{1}{\gamma }\le 2{\left[1-4{ϵ}^2\left(q\right)\right]}^{-1/q}$ where $\gamma =\sqrt{\underset{j,k}{\text{min}}\frac{{‖x_j‖}^2+d^2}{{‖x_k‖}^2+d^2}}$ ($x_j$ and $x_k$ are two arbitrary records in $X_i$). With the $d$-transformation, any two arbitrary records in $X_i$ are indistinguishable (Theorem 8). □

The $d$-transformation (i.e., multiplication of the intercept with a constant $d$) does not alter logistic model estimates, except for the intercept estimate. After multiplying the intercept by $d$, the dataset becomes $X_{\text{new}}=X{I}_D$, where $I_D$ represents a diagonal matrix with its diagonal elements being $d,\mathrm{1,1},\cdots ,1$. Based on Equation 6, the model estimate for $X_{\text{new}}$ becomes ${\beta }_{\text{new}}=I_D^{-1}\beta$ where $I_D^{-1}$ refers to a diagonal matrix with the diagonal elements being $\{1/d,\mathrm{1,1},\cdots ,1\}$. So only the intercept estimate is altered by the multiplication of constant $d$, while the estimates for the $q$ features remain unchanged.

VII. Performance evaluation

We perform experiments using the MNIST dataset from the UCI repository [59]. The MNIST dataset consists of hand-written digit images, each comprising a 28 by 28 pixel grid. Each image is associated with an integer label ranging from 0 to 9. The dataset consists of 60,000 images for training and 10,000 images for testing. In our privacy-preserving learning, we assume samples in each dataset are evenly distributed among $K$ clients, with each subset encompassing all the features. All the experiments are performed in Matlab on the University of Florida HiPerGator 3.0 (i.e., high computing performance) with 1 CPU and 40 Gb RAMs.

Our privacy-preserving logistic model is applied to differentiate label 9 from the remaining labels (0 to 8). To evaluate model performance on the large-scale dataset, we apply the bootstrap method [60] to create three datasets with sample sizes of $n=\mathrm{100,000},$ $n=\mathrm{500,000}$, and $n=\mathrm{1,000,000}$, respectively. To minimize communication cost, we employ SZ for lossy compression, ensuring a relative error threshold of less than 0.01 (i.e., the difference between raw data and compressed data $<0.01*$(maximum value-minimum value)). As shown in Table IV, our scheme with SZ compression after 20 iterations achieves an accuracy level equivalent to that of the non-secure model. The non-secure model is constructed using aggregated data from all clients without incorporating privacy protection considerations. The utilization of SZ compression leads to a notable decrease in communication costs. In scenarios where the number of clients involved in the privacy-preserving learning framework is less than 10, our scheme incorporating SZ compression incurs lower communication costs compared to the non-secure model (Figure 7A). Figure 7B shows that our scheme has high computation efficiency in analyzing large-scale datasets.

Table IV:

Model accuracy of the proposed privacy-preserving logistic model (relative error bound < 0.01 in the SZ compression)

Dataset	Non-secure model	Our scheme w/o SZ	Our scheme with SZ
			$\mathit{\text{Iter}}=5$	$\mathit{\text{Iter}}=10$	$\mathit{\text{Iter}}=20$	$\mathit{\text{Iter}}=50$
$n=60,000$	97.27%	97.27%	96.73%	97.22%	97.27%	97.27%
$n=100,000$	97.0%	97.0%	96.46%	96.97%	97.0%	97.0%
$n=500,000$	97.25%	97.25%	96.69%	97.18%	97.24%	97.25%
$n=1,000,000$	97.25%	97.25%	96.70%	97.20%	97.23%	97.25%

Non-secure model: the model built on the aggregated data from all the clients without considering privacy protection. w/o SZ: without SZ compression. Iter: iteration times. $n$: No. of samples in the aggregated data.

Fig. 7:

A: communication cost of the proposed scheme. w/o SZ: without SZ compression. $n$: No. of samples in the aggregated data. $K$: No. of the clients. B: computation time of 20 iterations in the proposed scheme. w/o SZ: without SZ compression. $n$: No. of samples in the aggregated data. $K$: No. of the clients.

We further compare the performance of our scheme with four state-of-the-art frameworks that provide malicious security. First, we evaluate the performance of our scheme for the binary classification problem and compare with two maliciously secure frameworks, SWIFT [12] and Fantastic4 [13]. Specifically, we construct the secure logistic model to distinguish between the digits 4 and 9 in the MNIST dataset (binary classification). A total of 11,791 samples are included in the training set, while the testing set comprises 1,991 samples. Moreover, we apply our scheme to solve the multiclass classification problem (Section V-B) and compare our model with state-of-the-art privacy-preserving neural networks, SecureNN [6] and Falcon [7]. The multiclass classification utilizes 60,000 images from the MNIST dataset for training and the remaining 10,000 images for testing.

Table V summarizes results of our scheme and other privacy-preserving frameworks. The computation and communication cost of our scheme goes up with an increasing number of clients. For the comparison, we consider three scenarios, with the number of clients being 10, 20, or 50. Compared with SWIFT [12], our scheme is computationally faster for clients up to 50 and has competitive communication cost for clients up to 20. In contrast, our scheme has improved communication efficiency but higher computation cost compared with Fantastic4 [13]. Our model has higher model accuracy compared with these two maliciously secure frameworks. Compared with their respective frameworks with 3 servers, the 4-server frameworks in [12], [13] have better performance in both computation and communication aspects. Given that the secure frameworks in [12], [13] rely on the honest-majority setting (where a malicious adversary can corrupt at most one server), the inclusion of an extra server imposes a more stringent security prerequisite for the successful execution of the secure frameworks. In contrast, our scheme conducts a secure logistic model that is resilient to malicious clients, utilizing only a single semi-honest server in the process. For multiclass classification, our scheme has better computation and communication performance when compared to maliciously secure neural networks under the semi-honest assumption [6], [7]. The accuracy of our secure scheme is also comparable to these two neural network models with malicious security.

Table V:

Comparison between our model and other maliciously secure frameworks using MNIST dataset

Data	Framework	Comp.	Comm.	Accuracy
MNIST (4 vs. 9)	SWIFT [12] (3PC)	12 mins	96.5 Mb	–
	SWIFT [12] (4PC)	8.6 mins	44 Mb	–
	Fantastic4 [13] (3PC)	8.5 s	2.8 Gb	96.5%
	Fantastic4 [13] (4PC)	3 s	167 Mb	96.5%
	Our (w/o SZ, $K=10$)	17.5 s	204 Mb	98.9%
	Our (SZ, $K=10$)	17.7 s	29.2 Mb	98.9%
	Our (SZ, $K=20$)	29.6 s	58.4 Mb	98.9%
	Our (SZ, $K=50$)	65 s	146 Mb	98.9%
MNIST	SecureNN [6] (3PC)	1.03 hrs	110 Gb	93.4%
	Falcon [7] (3PC)	33.6 mins	88 Gb	97.4%
	Our (w/o SZ, $K=10$)	17.6 mins	1.1 Gb	98.0%
	Our (SZ, $K=10$)	17.8 mins	164 Mb	98.0%
	Our (SZ, $K=20$)	18.5 mins	328 Mb	98.0%
	Our (SZ, $K=50$)	20.7 mins	821 Mb	98.0%

The performance of binary classification is compared with that of SWIFT [12] and Fantastic4 [13], while the performance of multiclass classification is compared with that of SecureNN [6] and Falcon [7].

The performance statistics of SWIFT [12] are sourced from [13], whereas the statistics for the other three SMC frameworks are obtained from their respective publications.

Comp.: computation cost; Comm.: communication cost.

3PC: 3-party computation (i.e., 3 servers); 4PC: 4-party computation (i.e., 4 servers).

$K$: No. of clients participated in our privacy-preserving scheme.

w/o SZ: without SZ compression.

VIII. Conclusion

In this paper, we introduce a maliciously secure logistic model for horizontally distributed data, utilizing a novel matrix encryption technique. Unlike state-of-the-art secure frameworks that require the participation of two or more servers in an honest-majority setting, our scheme utilizes only a single semi-honest server. Our scheme ensures that any two arbitrary records are indistinguishable through the $d$-transformation. A verification stage can detect any deviations from the proposed scheme among malicious data providers. Lossy compression is employed to minimize the communication cost while ensuring negligible degradation in accuracy. Compared with other maliciously secure models, our scheme has higher computational and communication efficiency. One prospective avenue for future research involves expanding our secure scheme to other nonlinear models, such as support vector machine and neural network.

Appendix A. Commutative matrix

Matrix $B_1$ and $B_2$ are commutative if $B_1{B}_2=B_2{B}_1$. To ensure negligible degradation in accuracy, the proposed privacy-preserving scheme generates commutative matrices to encrypt the plaintexts. The commutative encryption matrix is constructed based on matrix polynomial (i.e., a polynomial with matrices as variables) [61]. For instance, assume there are 2 clients in the collaborative learning. Each client first generates a common encryption key $B_0$ (a random nonsingular matrix). Client 1 then generates a vector of random coefficients $\left(b_{11},b_{12},\cdots ,b_{1q}\right)$ and an encryption matrix $B_1=\sum _{j=1}^q{b}_{1j}{B}_0^j=b_{11}{B}_0+b_{12}{B}_0{B}_0+b_{13}{B}_0{B}_0{B}_0+\cdots +b_{1q}{B}_0^q$. Similarly, client 2 generates a vector of random coefficients $\left(b_{21},b_{22},\cdots ,b_{2q}\right)$ and an encryption matrix $B_2=\sum _{j=1}^q{b}_{2j}{B}_0^j=b_{21}{B}_0+b_{22}{B}_0{B}_0+b_{23}{B}_0{B}_0{B}_0+\cdots +b_{2q}{B}_0^q.B_1$ and $B_2$ are commutative (i.e., $B_1{B}_2=B_2{B}_1$) because $B_1$ and $B_2$ are both matrix polynomials of the common matrix $B_0$.

Appendix B. Pre-processing and internal encryption for data with large sample size

To enhance the computational efficiency of encryption for clients with large sample sizes, we partition the encryption matrix $A_i$ into a diagonal matrix. Specifically, client $i$ generates $A_i$ as

$$A_i=\left(\begin{array}{cccc}{A}_{0i}& & & \\ & A_{0i}& & \\ & & \ddots & \\ & & & A_{0i}\end{array}\right)$$ (33)

where $A_{0i}$ is a $100\times 100$ random Gaussian matrix. $X_i$ is also partitioned into sub-matrices. Following the pre-processing procedure, a pseudo record is added to each sub-matrix to ensure indistinguishability. Specifically, $X_i$ (assuming that a vector of 1 is already included as the first row) is partitioned into sub-matrices $(\left.X_{il},l=\mathrm{1,2},\cdots \right)$, with each sub-matrix containing 99 samples and all the features. Before sub-matrices being encrypted by $A_0$, client $i$ computes the Euclidean norm of each column in $X_{il}$. Let $c_{lj}$ be the Euclidean norm of the $j$-th column $(j=1,\cdots ,q)$ in $X_{il}$ and $c_{lm}={\text{max}}_j{c}_{lj}$. Client $i$ generates a vector, $c_{lv}$, with the $j$-th element of $c_{lv}$ being $\sqrt{c_{lm}^2-c_{lj}^2}$. $c_{lv}$ is added to $X_{il}$ as the last row. Subsequently, the $l$-th sub-matrix consists of 100 samples, and the Euclidean norm of every column equals $c_{lm}$. As the total number of samples in $X_i$ may not be divisible by 99, client $i$ generates a random set of pseudo records to be vertically integrated into the original matrix such that each sub-matrix contains 99 samples. For example, consider the dataset $X_i$ containing 1,070 samples. Client $i$ generates 19 pseudo records, which are then vertically concatenated with $X_i$. Following this concatenation, $X_i$ can be split into 11 submatrices, i.e., $X_{il}(l=1,\cdots ,11)$. Client $i$ first concatenates $X_{il}$ with the pseudo record $c_{lv}$ and then encrypts each $X_{il}$ with $A_{0i}$. After data encryption, client $i$ sends $A_i$ and row indices of the pseudo records to the server.

Appendix C. Logistic model estimate using encrypted and original data

Theorem 9. Data matrices and model estimates of the secure and non-secure computation have the following properties.

1. $p\left(x_i^{enc};{\beta }^{enc}\right)=p\left(x_i;\beta \right)$;

2. $W^{enc}=W$;

3. $\begin{array}{c}{\beta }^{enc}=B^{-1}\beta \end{array}$ where $B=\prod _{i=1}^K{B}_i$.

Proof. Let

$$P_1\triangleq \left(\begin{array}{cccc}p\left(x_1;\beta \right)& & & \\ & p\left(x_2;\beta \right)& & \\ & & \ddots & \\ & & & p\left(x_n;\beta \right)\end{array}\right)$$ (34)

and

$$P_2\triangleq \left(\begin{array}{cccc}1-p\left(x_1;\beta \right)& & & \\ & 1-p\left(x_2;\beta \right)& & \\ & & \ddots & \\ & & & 1-p\left(x_n;\beta \right)\end{array}\right).$$ (35)

$P_1$ and $P_2$ are diagonal matrices. Let

$$P_1^{\text{enc}}\triangleq P_1\left(X^{\text{enc}},{\beta }^{\text{enc}}\right)\text{and}{P}_2^{\text{enc}}\triangleq P_2\left(X^{\text{enc}},{\beta }^{\text{enc}}\right)$$ (36)

be the corresponding matrices in the privacy-preserving model.

In iteration $m=0$ (initial setup), set the starting point as ${\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}$ for the privacy-preserving model and ${\beta }^{\left(0\right)}\triangleq B{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}$ for the non-secure model. Because $x_i^{\text{enc}}=B^T{x}_i$, we have

$$\begin{gathered} p\left(x_i^{\text{enc}};{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)=\text{Pr}\left(y_i=1\mid x_i^{\text{enc}};{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right) \\ =\frac{\text{exp}\left({\left(x_i^{\text{enc}}\right)}^T{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)}{1+\text{exp}\left({\left(x_i^{\text{enc}}\right)}^T{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)} \\ =\frac{\text{exp}\left(x_i^{T}B{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)}{1+\text{exp}\left(x_i^{T}B{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)} \\ =p\left(x_i;B{\left[{\beta }^{\left(0\right)}\right]}^{\text{enc}}\right)=p\left(x_i;{\beta }^{\left(0\right)}\right). \end{gathered}$$ (37)

According to Equations 34, 35 and 37, we have $P_1^{\text{enc}}=P_1$ and $P_2^{\text{enc}}=P_2.W$ (Equation 4) can be expressed as $W=P_1{P}_2$. Thus we have

$$W^{\text{enc}}={\left(P_1{P}_2\right)}^{\text{enc}}=P_1^{\text{enc}}{P}_2^{\text{enc}}=P_1{P}_2=W.$$ (38)

Therefore properties $I-III$ hold in the initial step.

Next we prove that properties $I-III$ hold in the $\left(m+1\right)$-th iteration assuming that properties $I-III$ hold in the $m$-th iteration. Let notations with ${\text{superscript}}^{\left(m\right)}$ or ${\text{subscript}}_{\left(m\right)}$ denote parameters derived during the $m$-th iteration. Given ${\beta }^{\left(m\right)},p\left(x_i;\beta \right)$ is updated as

$$p{\left(x_i;{\beta }^{\left(m\right)}\right)}_{(m+1)}=\frac{\text{exp}\left(x_i^T{\beta }^{\left(m\right)}\right)}{1+\text{exp}\left(x_i^T{\beta }^{\left(m\right)}\right)}.$$ (39)

Assuming that properties $I-III$ hold in the $m$-th iteration, we have ${\left[{\beta }^{\left(m\right)}\right]}^{\text{enc}}=B^{-1}{\beta }^{\left(m\right)}$. Since $x_i^{\text{enc}}=B^T{x}_i$, we have

$$\begin{gathered} p\left(x_i^{\text{enc}},{\left[{\beta }^{\left(m\right)}\right]}_{(m+1)}^{\text{enc}}\right)=\frac{\text{exp}\left({\left(x_i^{\text{enc}}\right)}^T{\left[{\beta }^{\left(m\right)}\right]}^{\text{enc}}\right)}{1+\text{exp}\left({\left(x_i^{\text{enc}}\right)}^T{\left[{\beta }^{\left(m\right)}\right]}^{\text{enc}}\right)} \\ =\frac{\text{exp}\left(x_i^{T}B{B}^{-1}{\beta }^{\left(m\right)}\right)}{1+\text{exp}\left(x_i^{T}B{B}^{-1}{\beta }^{\left(m\right)}\right)} \\ =\frac{\text{exp}\left(x_i^T{\beta }^{\left(m\right)}\right)}{1+\text{exp}\left(x_i^T{\beta }^{\left(m\right)}\right)} \\ =p{\left(x_i;{\beta }^{\left(m\right)}\right)}_{(m+1)}. \end{gathered}$$ (40)

Similar to the proof for iteration $m=0$, we have Enc $\left(W\right)=W$. So properties $I-II$ hold in the $(m+1)$-th iteration. Moreover, based on Equation 11, we have

$$\begin{gathered} {\left[{\beta }^{(m+1)}\right]}^{\text{enc}}={\left[{\left(X^{\text{enc}}\right)}^T{\left[W^{\left(m\right)}\right]}^{\text{enc}}{X}^{\text{enc}}+\Lambda B^{T}B\right]}^{-1}\times \left\{{\left(X^{\text{enc}}\right)}^T{\left[W^{\left(m\right)}\right]}^{\text{enc}}{X}^{\text{enc}}{\left[{\beta }^{\left(m\right)}\right]}^{\text{enc}}+{\left(Z^{\text{enc}}\right)}^T-{\left(X^{\text{enc}}\right)}^T{\left[p^{\left(m\right)}\right]}^{\text{enc}}\right\} \\ ={\left(B^T{X}^T{W}^{\left(m\right)}XB+\Lambda B^{T}B\right)}^{-1}\times \left[B^T{X}^T{W}^{\left(m\right)}XB{B}^{-1}{\beta }^{\left(m\right)}\right.\left.+B^T{Z}^T-B^T{X}^T{p}^{\left(m\right)}\right] \\ =B^{-1}{\beta }^{(m+1)}. \end{gathered}$$ (41)

To conclude, properties $I-III$ hold for all iterations. □