Robust Transparency Against Model Inversion Attacks

Abstract

Transparency has become a critical need in machine learning (ML) applications. Designing transparent ML models helps increase trust, ensure accountability, and scrutinize fairness. Some organizations may opt-out of transparency to protect individuals’ privacy. Therefore, there is a great demand for transparency models that consider both privacy and security risks. Such transparency models can motivate organizations to improve their credibility by making the ML-based decision-making process comprehensible to end-users. Differential privacy (DP) provides an important technique to disclose information while protecting individual privacy.

However, it has been shown that DP alone cannot prevent certain types of privacy attacks against disclosed ML models. DP with low ϵ values can provide high privacy guarantees, but may result in significantly weaker ML models in terms of accuracy. On the other hand, setting ϵ value too high may lead to successful privacy attacks. This raises the question whether we can disclose accurate transparent ML models while preserving privacy. In this paper we introduce a novel technique that complements DP to ensure model transparency and accuracy while being robust against model inversion attacks. We show that combining the proposed technique with DP provide highly transparent and accurate ML models while preserving privacy against model inversion attacks.

1. Introduction

MACHINE learning techniques enriched our lives in many different aspects. Some of these aspects are as trivial as personalized web search, while others are as critical as credit\loan approval and justice decisions. These data mining\machine learning tools have been applied to assist industries and organizations into making better decisions beyond the scope of traditional human expertise. However, the great opportunities machine learning models contribute to the society come with challenges; they are often opaque and therefore their trustworthiness is questionable. Given their employment in critical decision making, we need to be able to understand the reasoning behind these machine learning models to ensure fairness and hold entities accountable.

Transparent machine learning is an emerging area that has been called for by organizations to increase their credibility and to ensure fairness and trust. Given data and machine learning models, companies and organizations can selectively share more in-depth machine learning results with their clients for greater transparency. Unfortunately, for many institutions transparency often comes with a price. On the one hand, the greater their transparency, the greater their credibility. On the other hand, greater transparency may involve disclosing privacy sensitive information, and as a result many of them may choose to hide as much detail as possible about their machine learning process. For this purpose, a transparent model that does not elevate the user’s privilege to a level that threatens individual privacy and trade secrets is highly desired.

Traditionally, a machine learning model can be either a white-box or a black-box model. A white-box model makes its learned model parameters publicly available, but the sensitive information related to individuals used to build the model may be disclosed to the general public using various attacks and therefore has high privacy risk. On the contrary, a black-box machine learning model can only be probed for predictions, and therefore is non-transparent and may provide a certain degree of privacy. Given these two models, it seems impossible to preserve privacy without sacrificing transparency. We tackle this challenge by replacing a white-box model with a simpler transparency model that provides better user understandability while minimizing privacy loss, and making it robust to certain sensitive information prediction attacks.

In this paper, we investigate methods and metrics for evaluating security and privacy as we increase the transparency of machine learning models. Especially, we focus on transparency models where complex decisions are explained to individuals using a set of simple if-then-else rules; and explore how to prevent model inversion attacks [14] where an attacker may try to use such transparency models (e.g., rule sets used for medical diagnosis) combined with background information (e.g., some demographic information about the patient) to infer sensitive information (e.g., predicting sensitive lifestyle information). To prevent such attacks, we provide different strategies to produce a privacy-preserving transparency rule set. Our first strategy is “post-transparency privacy risk measure”, where we prune the rules from the transparency model based on their perceived privacy risk¹. For this strategy, we introduce a new privacy risk assessment protocol referred to as α-violation. The second strategy is “pre-transparency privacy” where privacy techniques are applied at the machine learning model level. We use differential privacy as the “pre-transparency privacy” technique with which we produce differentially private white-box learning models. However, it has been shown that differential privacy alone can significantly weaken the performance of machine learning models when protecting against inference attacks [14]. As a remedy, we combine α-violation based post-rule pruning with differential privacy and show that this combined method can produce a highly accurate machine learning model with very low privacy loss, and robust to model inversion attacks.

We choose rule-based transparency models due to their proven high precision when compared to other forms of transparency models [5], [26]. Rule-based models have also grown in popularity as they have become part of the new Microsoft and IBM toolkits: InterpretML [23] and AI Explainability 360 [17]. These toolkits contain a mix of transparency algorithms, including rule-based transparency models.

Our contributions can be summarized as follows:

• We introduce a novel model inversion attack that is applicable to rule-based transparency models.

• We introduce a novel privacy risk estimator named α-violation that can be successfully used to enhance the robustness of any rule-based transparency model against model inversion attacks.

• We show that by combining differential privacy with the α-violation privacy risk estimator, we can protect against both inference attacks and sensitive feature reconstruction attacks.

• Our work is the first comprehensive study on trade-offs between transparency and privacy in the context of rule based transparency models.

The remainder of this paper is organized as follows. In Section 2, we discuss the research related to our work. Section 3 describes the background information our research is based on. We then introduce our privacy model in Section 4. In Section 5, we introduce our model inversion attack. Section 6 presents our experimental results. Finally, we discuss our work and results in Section 7 and conclude in Section 8.

2. Related Work

Transparency has received a wide attention in the past few years. Still, it is a new and emerging area of research. To the best of our knowledge, there are three main transparency models: Local Interpretable Model-agnostic Explanations (LIME) [25], Quantitative Input Influence (QII) [9], and Instance-based Transparency (IT) [5]. All these models are used to explain the predictions of a white-box classifier. We can divide these transparency models into two different categories: features-based transparency and rule-based transparency, according to the strategies they follow. LIME and QII belong to the feature-based transparency while IT falls into the rule-based transparency.

LIME is a linear model that provides instance-based explanations that are locally faithful to the model. An explanation for an instance x is generated by sampling a set of instances in the neighborhood of x. Using the distance between neighborhood samples and x and the labels of the neighborhood samples, LIME generates the top features that can explain the reasoning behind the prediction for x. SP-LIME is an extension of LIME which is also provided in [25]. Using SP-LIME, the complete white-box model is explained by selecting a set of the explanations generated for the instances. QII is a set of measures designed to measure the influence of the input of a white-box classifier on its outputs\predictions. QII measures the influence of an input i by observing the output of the classifier when i is replaced by an independent random sample. QII can be used to generate transparency reports for instances to explain their predictions. Moreover, QII can also be used to measure the influence of the input on a group of instances and to detect discrimination. These features-based transparency models can preserve privacy by adding noise to them based on their sensitivity so that they become differentially private as proposed in [9].

IT is a rule-based transparency model [5]. Using IT one can generate a small set of rules that can explain the prediction for any instance. IT searches for a set of rules that has high consistency with the model it explains and maintain good coverage for the instances used to build the model. For any predictive model that consists of a set/list of rules, one can convert it to the rule-based transparency models by mimicking the predictions made by the white-box classifier. An example of such predictive model is the Bayesian Rule Lists (BRL) proposed by Letham et al [21]. BRL is a predictive model that is both accurate and interpretable. It consists of a series of if… else … rules. More details about IT and BRL will be provided in Sections 3.2 and 3.3 respectively.

Differential privacy was first introduced by Dwork in [10]. Since then, differential privacy became the golden standard privacy definition. However, it has been shown in the previous research that differential privacy is not designed to protect against inference attacks on sensitive features [11], [14]. Kifer and Machanavajjhala discussed in [20] several misconceptions about differential privacy. They argued that without making any assumption about the data, it is impossible to achieve both privacy and utility. They showed that differential privacy can not hide evidence of participation especially in the case where correlation exists in the data.

The trade-off between privacy and utility has been heavily studied in the literature. Brickell et al. studied the privacy vs. utility trade-off when k-anonymity, ℓ-diversity, and t-closeness are applied to the data [6]. They showed that it is difficult to maintain both utility and privacy using these privacy definitions. Haney et al. studied the cost of privacy on employer-employee (ER-EE) data [16]. Their work demonstrates that differential privacy fails to satisfy both privacy and utility. They further used the Pufferfish framework to develop a new privacy definition that can grantee privacy with little utility loss given their privacy requirements. Fredrikson et al. studied the effect of differential privacy on pharmacogenetics models [14]. They conducted a case study of privacy in pharmacogenetics where machine learning models are used to predict patients’ dosage of warfarin based on their genotype, demographic information, and background information. Warfarin is an anticoagulant medication that requires careful dosing. They introduced a generic model inversion attack where an adversary would infer patients’ genotype (i.e. sensitive information) given the model and some information about the patients. They showed that when differential privacy is applied to protect against model inversion attack, it would lead to severe impact on utility. In the case of warfarin dosing, the impact would be an increased risk of bleeding or even mortality. A newer version of the model inversion attack was introduced by Fredrikson et al. in [13]. Their model inversion was specifically designed to infer sensitive information from decision trees. They demonstrate the success of their attack by recovering the sensitive feature values for survey respondents with no false positives. They were also successful in recovering images from facial recognition services.

Our work differs from all these previous research in that it is the first to study the trade-off between differential privacy and transparency. We also introduce a privacy protocol that allows for the disclosure of a rule-based transparency model with minimum risk of leaking individuals’ sensitive features. To the best of our knowledge, our work is the first attempt to tackle the privacy issue in the context of rule-based transparency models.

3. Preliminaries

In this section, we formally define transparency and robustness, and discuss existing transparency models and model inversion attacks.

3.1. Transparency and Robustness

Transparency is a property of a machine learning model that produces a decision function f if f supplies chains of reasoning for any given input, and the justification of the decisions made by f that are internally accessible to a human is externally gauged in terms of accuracy, consistency, reliability, and robustness. In general, there is no specific formality required for the creation of “chains of reasoning”. Linear functions, graphs, lists of rules, if simple enough for human to comprehend, all can serve as a reasonable manifestation of transparency. In our work, we require that the chosen “transparent” counterpart f_t of a given model f to be accurate and consistent with f. More precise definitions are given as follows.

Definition 1.

Given a black box decision function f and the data distribution (x, y) ~ D used to build f, a human understandable transparency model f_t is defined to be (η, γ) accurate and consistent, if f_t satisfies

$$P_{(x,y)~D}\left[f_t(x)=y\right]>\eta$$

$$P_{(x,y)~D}\left[f(x)=f_t(x)\right]>\gamma$$

Here Accuracy is used as the standard metric to evaluate a machine learning model so that we can ensure the transparent counterpart of a given model is accurate as well. Consistency requires that f_t and f produce the same output for the same input with a high probability. Robustness to privacy attacks, strictly speaking, is concerned with information safety—whether data used to train the learning model can be hacked, for example, through inversion attacks.

Definition 2.

Let S be the sensitive component of a data distribution D. Given (η, γ)-transparent model f_t, it is ξ-robust to inference attack A if

$$P\left(A\left(f_t\right)\in S\right)<\xi$$

for a small ξ.

Here robustness is defined with respect to a specific attack A such as the model inversion attack discussed in [14]. Just as we define an encryption function with respect to certain attacks, such as chosen plain text attacks, it is only feasible to define robustness with specific attacks in mind. Note that f_t can satisfy any other privacy definition such as differential privacy (see Section 3.4).

In this work, we make f_t differentially private and robust to known model inversion attacks in advance to addressing major privacy attacks against transparent classifiers.

3.2. Instance-based Transparency (IT) Model

The Instance-based Transparency (IT) model is a rule-based model that offers simple explanations for decisions made by machine learning models [5]. IT reduces algorithmic complexity while maintaining high agreement with the machine learning model it explains. IT takes as an input a machine learning model f, its training samples D, and a complete set of rules $\mathcal{R}$, and generates a small set of rules $R\in \mathcal{R}$ that explains the model. IT guarantees that: 1) R is maximally consistent with f where consistency between IT model and the original white-box model is defined as the probability that they agree on the predictions for a set of data samples; 2) R satisfies a minimum coverage λ for each instance x_i ∈ D. An instance x_i is said to be covered by IT if there is at least one rule R_j ∈ R that is applicable to x_i. A rule R_j is applicable to x_i if the antecedents of the rule matches x_i’s features. For a class c ∈ C, IT assigns weights to rules as follows:

$$R_{wj}(c)=\sum _{i=1}^n{I}_{\mathit{\text{match}}}\left(f,R_j,x_i,c\right)A(j)\text{ where},$$ (1)

$$I_{\mathit{\text{match}}\left(f,R_j,x_i,c\right)}=\{\begin{array}{l}1,\text{ if }{f}_R\left(x_i\right)(j)=1\text{ and }\hfill \\ R_j\left(x_i\right)=f\left(x_i\right)=c\hfill \\ 0,\text{ otherwise }\hfill \end{array}$$

A is a boolean vector indicating rules selected by IT, f_R(x) is a function that marks the rules covering x, f_R(x)(j) is the value of the j-th element in f_R(x), R_j(x_i) is R_j’s prediction for x_i. The final weight R_wj for rule R_j is: argmax_c∈C R_wj(c).

3.3. Bayesian Rule Lists (BRL)

A Bayesian Rule List consists of a series of if… else rules. It is a predictive model that is both accurate and interpretable. Each rule consists of two parts: the conditions and the outcome. The conditions are a set of features and the outcome is the predicted class for instances that match the conditions. BRL contains a default rule that has only a class value which is used to predict for instances that do not match any of the rules. The goal of BRL is to provide an explanation for each decision made by the predictive model. More details about BRL are available in [21].

BRL is originally used as an explainable predictive model that can be easily understandable by human when it is trained on the original labels. By using the original labels as an input for training, BRL learns a pattern from the data and therefore provides predictions. We can also use BRL as a transparency model that explains predictions produced by any classifier by training BRL on the classifier’s predictions instead of the original labels. By following this step, BRL will learn a pattern from the classifier, instead of the data, and produce a list of rules that explains the classifiers’ predictions.

3.4. Differential Privacy

Differential privacy became the golden standard privacy preserving-model. An algorithm that satisfies differential privacy guarantees that the output of the algorithm changes very little with the addition, removing, or altering of any single entry by adding noise to the output. It has been proven that differential privacy can produce privacy-preserving output while maintaining accuracy [8], [15], [18]. The classic definition of differential privacy is:

Definition 3.

A randomized algorithm A satisfies ϵ-differential privacy [10] for a non-negative number ϵ if for two neighboring datasets D₁ and D₂ differing in a single element and all S ∈ Range(A):

$$\text{Pr}\left[A\left(D_1\right)\in S\right]\le e^{ϵ}\text{ Pr}\left[A\left(D_2\right)\in S\right]$$ (2)

The ultimate goal of differential privacy is protecting individuals from inference attack: a technique designed to infer whether an individual is included in the data. However, there is no guarantee that differential privacy can preserve individuals’ privacy while maintaining transparency. Transparency is measured by the accuracy of the differentially private model and its agreement with the original non-private model.

There has been a massive amount of literature on applying differential privacy to machine learning models [7], [8], [18], [19], [27], [28]. To make our white-box models differentially private, we will use the objective perturbation mechanism introduced by Chaudhuri et al. [8]. This mechanism can be applied to any regularized empirical risk minimization (EMR) classifier. These privacy-preserving EMR models have shown exceptional performance as reported by the authors. In the objective perturbation mechanism, a random noise is added to the objective function of the classifier and the empirical risk is minimized according to the perturbed objective function.

Similar to the work in [8], we will use linear SVM and logistic regression as our empirical risk minimization classifiers. The interested reader can refer to [8] for detailed explanations of objective perturbation.

3.5. Generic Model Inversion Attack

Fredrikson et al. [14] introduced a generic model inversion attack to reconstruct individuals’ sensitive features. Their attack was specifically designed to reconstruct generic markers for patients given a linear regression model f that is used to predict warfarin dosages. The model f is built on patients demographic data, medical history and generic markers.

Given an instance x, its marginal priors p = {p₁, …, p_n} of the features, and a linear regression model f with err(y, f(x)) where err is the probability that the original label is y given that f predicts f(x), the generic model inversion attack reconstruct the generic marker for x by computing Pr(w_i|x, f) for each w_i value of the sensitive feature as: Pr(w_i|x, f) ← err(y, f(x′)) · ∏_i p_i(x) where x′ is identical to x except that the sensitive feature has the value w_i. The attack is marked as successful for x if x^s = argmaxPr(w_i|x, f) where x^s is the sensitive value for x.

We will introduce in Section 5 a new model inversion attack inspired by the attack proposed by Fredrikson et al. Our attack is different in that it can be applied to any rule-based transparency model.

4. Privacy-Preserving Transparency

Our model focuses on the scenarios where institutions could benefit from greater transparency in decision making. However, transparency can be at once beneficial and pernicious. While greater transparency in decision processes can help organizations increase their trustworthiness, it may also empower their competitors, and sometimes adversaries, with more access to sensitive information that is supposedly kept inside the organization. In this section, we introduce our privacy-preserving transparency model that offers higher availability of information on organizational decisions made using machine learning, without threatening the protection of the organizations’ secrets and security. The privacy is controlled using a new privacy risk estimator that we combine with differential privacy to keep individuals’ sensitive features hidden without sacrificing transparency.

4.1. Problem Definition

Given a set of samples x = {x₁, …., x_n} in a data collection that consists of n-dimensional feature space X = X₁ × … × X_n, a white-box machine learning model f is built to assist decision making. Each instance x_i is mapped to one class value y_i in the class domain C = {c₁, …, c_i, …}. We will assume that we have access to the training data D used to build the machine learning model f and a test data T to evaluate the performance of f.

A privacy-preserving transparency model provides sufficient information to help users understand decisions made using f, while keeping them from obtaining sensitive information. The privacy-preserving transparency model is composed of two components: a transparency model and a privacy risk model, where the transparency model defines trustability while the privacy risk model determines protection of sensitive information assets.

The privacy-preserving transparency model assumes that the transparency model consists of a set of decision rules R and that the data (and therefore the decision rules) consists of Private\Sensitive (S) data, Public (P) data, and a class\prediction value. The ultimate goal of our model is to prevent an adversary from gaining access to the Sensitive data when having access to the Public data and the class value. The Sensitive data can consist of one or more attributes and each sensitive attribute S can take multiple values $S=\left\{w_1^s,\dots ,w_v^s\right\}$. The Public data and the class value can be revealed by the transparency model even to the adversary without any privacy risk. Similar to the Sensitive data, the Public data can have multiple attributes with multiple values for each attribute $P=\left\{w_1^p,\dots ,w_u^p\right\}$.

4.2. The α-violating Privacy Risk Estimator

While it is nice to show that an adversary gains no privacy-violating information through a released transparency model, in many cases we will not be able to claim this. Privacy is not absolute; most privacy laws provide for cost\benefit trade-offs when using private information. For example, many privacy laws include provisions for use of private information “in the public interest” [3]. To trade off the benefit vs. the cost of privacy loss, we need a metric for privacy loss that could allow us to quantify the privacy risks.

In this paper, we will consider the expected increase in predicting Sensitive data S due to the disclosure of rules as the main risk metric. In other words, due to the disclosed classifier (e.g., the set of decision rules), how much better the adversary can predict the Sensitive data S for the individuals that have participated in the machine learning process? Of course, computing such risk metric will depend on the background information of the adversary (e.g., which attributes are already known to the adversary?) and the precise definition of S. As a part of this task, we will explore how knowing such metrics can help protect individuals privacy when facing model inversion attacks designed to infer the sensitive feature S.

We devise an α-violation privacy risk estimator for preserving the privacy of transparency models when combined with differential privacy. With α-violation, we assume that the transparency model generates a set of rules R that explains the predictions made by the white-box classifier. We define α-violating as follows:

Definition 4.

A rule R_j is considered an α-violating rule if ∃S ∈ R_j such that:

$$\begin{array}{r}\hfill \frac{\text{Pr}\left[S=w_i^s\mid P=w_k^p,R_j(x)=f(x)\right]}{\text{Pr}\left[S=w_i^s\right]}\le e^{-\alpha }\text{}\\ \hfill \text{or}\\ \hfill \frac{\text{Pr}\left[S=w_i^s\mid P=w_k^p,R_j(x)=f(x)\right]}{\text{Pr}\left[S=w_i^s\right]}\ge e^{\alpha }\end{array}$$ (3)

where:

$$\begin{gathered} \text{Pr}\left[S=w_i^s\mid P=w_k^p,R_j(x)=f(x)\right]= \\ \frac{\text{Pr}\left(S=w_i^s,P=w_k^p,R_j(x)=f(x)\right)}{\text{Pr}\left(P=w_k^p,R_j(x)=f(x)\right)} \end{gathered}$$ (4)

In other words, a rule R_j is an α-violating rule if it contains Sensitive data, Public data, and class information that can be used to recover the Sensitive data for instances used to build the white-box model, with a probability larger than e^α or smaller than e^−α.

Each rule R_j will be assigned an α-violation score independently from any other rule. The α-violation probabilities are estimated from the training data. After assigning these scores, we prune the rules generated from the transparency model based on the privacy rate. The privacy rate is the ratio of the total α-violation probability that we can disclose. For example, when the privacy rate is 10%, we are allowed to disclose 10% of the α-violation rules. We define the total privacy risk as T_p = r_p.P_max where r_p ∈ [0%, 100%] is the privacy rate and $P_{\mathit{\text{max}}}={\sum }_{j=1}^{m}v\left(R_j\right)$. The privacy rate is our method of controlling the percentage of pruned α-violating rules. Without the privacy rate, we would eliminate all the α-violating rules which could be an extreme measure that can affect the utility\accuracy. We let v(R_j) = 1 if R_j is an α-violating rule and v(R_j) = 0 otherwise. In Section 5, we introduce a model inversion attack to evaluate the effectiveness of our α-violation privacy risk estimator in preserving privacy.

4.3. Combining Differential Privacy with α-violating Privacy Risk Estimator

We propose a novel privacy-preserving model that combines differential privacy with our α-violating privacy risk estimator. Differential privacy has been the golden standard for defining privacy. However, Fredrikson et al. show that differential privacy with small ϵ results in so much noise and therefore renders the machine learning model useless [14]. Our model is not intended to replace differential privacy; it is an answer to the challenge proposed by Fredrikson et al. [14]. In this model, we propose using differential privacy with larger ϵ combined with our α-pruning technique to preserve both utility and privacy. Our strategy proceeds as follows: 1) We build a differentialy private machine learning model; 2) The predictions produced by the machine learning model are used as an input to a rule-based transparency model (e.g. IT or BRL). Therefore, the rules are generated from the predictions produced by differentially private learning models such as DP-SVM and DP-LR; 3) The α-violation privacy risk estimator is applied to prevent model inversion attacks. The complete strategy for combining differential privacy with α-violating privacy risk estimator is given in Figure 1.

Fig. 1:

The steps for combining differential privacy with α-violating privacy risk estimator.

5. Model Inversion Attack (MI)

Model Inversion (MI) attacks are designed to exploit the privacy risk associated with disclosing a model. In this section, we introduce our MI attack that is constructed to evaluate the effectiveness of our α-violation privacy risk estimator in preserving privacy when the adversary’s goal is to reconstruct individuals’ sensitive features with access to a transparency model. We use the success of the MI attack to calculate the disclosure risk.

Our attack is inspired by the attack proposed by Fredrikson et al. [14] with the same goal to reconstruct the sensitive features but with different assumptions. While the attack in [14] assumes access to the machine learning model, our new attack assumes access to the rules generated by the transparency model. In our settings, we assume that the machine learning model will be kept hidden from the end user for privacy concerns and only the transparency model that mirrors the original machine learning model is published. By replacing the original model with a simpler transparency model, we can maintain better user understandability while minimizing privacy loss.

The MI attack computes the disclosure risk for each instance x individually. The disclosure risk for an instance x is the success of re-identifying x’s private feature value x^s using the decision rules revealed by the transparency model. Formally, the disclosure risk for x is defined as:

$$\text{disclosure risk }(x)=\{\begin{array}{ll}1,\hfill & \text{ if }{x}^s=\mathit{\text{argmax}}\text{ Pr}\left(w^s\mid x,R\right)\hfill \\ 0,\hfill & \text{ otherwise }\hfill \end{array}$$ (5)

Where

$$\text{Pr}\left(w^s\mid x,R\right)=\frac{{\sum }_{j=1}^m{I}_{\mathit{\text{match}}\left(x,R\right)}*R_{wj}}{{\sum }_{j=1}^m{I}_{\mathit{\text{match}}\left(x_{-S},R\right)}*R_{wj}}$$ (6)

where x^s is the value of the private feature S for the instance x, R_wj is the weight of rule R_j, I_{match(x, R)} = 1 if R_j matches x’s Public data, Sensitive data and class label, and $I_{\mathit{\text{match}}\left(x_{-S},R\right)}=1$ if R_j matches x’s Public data and class label only. The goal of Equation (6) is to measure the increased probability of knowing the value of the Sensitive data of instance x when disclosing rules that match the class value, Public data, and Sensitive data of x, compared to disclosing rules that match everything except for the Sensitive data. In this setting, we assume that the adversary has access to the publicly published transparency model and the Public data of instance x and aims to disclose the exact value of the Sensitive data of x.

The final attack success is the average of disclosure risk:

$$\text{ attack success }=\frac{{\sum }_{i=1}^n\text{disclosure risk}\left(x_i\right)}{n}$$ (7)

The MI attack presented in this section can be applied to any rule-based transparency model.

6. Experiments

In this section, we demonstrate the success of our privacy-preserving transparency technique. We evaluate our privacy model when applied to two different transparency models: IT and BRL. We begin by describing the datasets, the white-box models, and the parameters used in our evaluation in Sections 6.1, 6.2, and 6.3, respectively. In Section 6.4, we evaluate the baseline transparency models to ensure that these models closely resemble the original machine learning models in terms of agreement and attack success. The agreement represents the percentage of matching labels between the white-box classifier and the transparency model. The attack success is measured as in Equation (7). The main results for evaluating the effectiveness of our α-violation privacy risk estimator when combined with differential privacy are described in Section 6.5. We further discuss the impact of using differential privacy by itself on transparency and model inversion attacks.

The results of applying α-violation privacy risk estimator alone to non-differential private models are presented in the supplemental file due to page limitations.

Since BRL provides a rule list, there is dependency between the rules, where a rule j can only be checked for a given instance x when all the previous rules i < j do not cover x. For this reason, an extra step needs to be taken when applying α-violation because α-violation assumes that the rules are independent from each other. When estimating α-violation probabilities for a rule j we only consider the subset of the data not covered by any of the rules i < j.

All the results reported in this section are the average of 10-fold cross-validation.

6.1. Datasets

We evaluate our techniques on four real datasets. For simplicity, we assume that each dataset has only one sensitive feature. The following is the description of these datasets.

• Mammographic Mass dataset contains information about mammographic masses screening for breast cancer [22]. It consists of 916 instances and 5 features. These features are: assessment score, age, mass shape, mass margin, and mass density. The class value for the data is the severity of the (benign or malignant) of the mass with 54% benign cases and 46% malignant cases. We consider the score as the sensitive feature in this data.

• Arrest dataset is generated from the Longitudinal Surveys (NLS) which are a set of surveys of young men and women born in the years 1980 to 1984 [1]. The data contains information such as gender, birth year, race, location, smoking history, drug use history and arrest history. The arrest history is used as our class value and it consists of two values: “Yes” if the instance has arrest history and “No” otherwise. The drug history will be our sensitive feature and it has two values: “Yes” if the instance has any drug history and “No” otherwise. The data contains 7,063 instances and 90% of the instances have positive arrest history which creates imbalanced data issue. We balance the data by randomly selecting the same number of instances from the positive arrest history class and the negative arrest history class. The final size of the data becomes 1,144 instances.

• Lending Club dataset contains loan information along with the current loan status [4]. The loan status can be: “fully paid”, “charged off”, “late (31–120 days)”, “does not meet the credit policy. Status:Fully Paid”, and “does not meet the credit policy. Status:Fully Paid”. We remove from the data unverified instances and instances with empty values. We let the class value be the “loan status” and only consider the values “fully paid”, “charged off” because the remaining values are under-represented in the data. We further select a subset of the features using Sequential Forward Selection. The class distribution for this data is 85% (fully paid) and 15% (charged off). To prevent class imbalance, we randomly select an equal number of “fully paid” and “charged off” instances. The original data contains 115 features and 42,540 instances. After the pre-processing steps, the resulting data contains 11 features and 6,432 instances. The sensitive feature in this data is the FICO score. We discretize FICO score based on Experian credit score range which are: exceptional (800+), very good (740 to 799), good (670 to 739), fair (580 to 669), and poor (579 and lower) [12].

• COMPAS data was collected by ProPublica and it contains information about criminal defendants to determine their likelihood of becoming a recidivist [24]. COMPAS stands for Correctional Offender Management Profiling for Alternative Sanctions. The data has 6,479 instances and 7 features. These features are: gender, age, race, priors count, and charge degree risk score, and whether the defendants re-offended in two years or not. We let the binary re-offending feature be our class and let the risk score be our sensitive feature. The risk score can take the following values: high, medium, or low. The class distribution is 56% for re-offenders and 44% for none re-offenders.

The sensitive features selected for each dataset represent the features that users want to keep hidden and protected. For example, FICO score in lending club is considered sensitive because it affects credit\loan approval and the interest rate a person is qualified for [2]. In all the datsets, the number of samples is independent from the dimension.

6.2. White-box Models

To make our white-box models differentially private, we use the objective perturbation mechanisms introduced by Chaudhuri et al. [8]. We use regularized Support Vector Machine (SVM) and Logistic regression (LR) as our white-box models in order to enable fair comparison between α-violation and the differentially private white-box models that require regularized empirical risk minimization as discussed in Section 3.4.

6.3. Parameter Selection

We now discuss our parameter selection methodology. The most important parameters that must be carefully selected are the ones that determine the accuracy of the white-box models. The accuracy of the regularized empirical risk minimization models varies based on the value of the regularizer Λ. Figure 2 presents the accuracy of the non-private SVM (SVM), differentially private SVM (P-SVM), non-private LR (LR), and differentially private LR (P-LR) with all the datasets. For each one of these models, we only present the accuracy of the model with the Λ value that maximizes the accuracy. The results for Λ = {10⁻², 10⁻³, 10⁻⁵, 10⁻⁷} for all the models are presented in Tables 1 and 2. For all the remaining experiments, we continue using these algorithms with the Λ value that maximizes the accuracy.

Fig. 2:

Accuracy of non-private and private SVM and LR on the four datasets.

TABLE 1:

SVM accuracy with different regularization parameters

Mammographic Mass
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	78.28%	79.49%	80.57%	82.62%	82.75%	82.14%
Λ = 10−3	78.16%	79.49%	80.33%	81.30%	80.57%	82.50%
Λ = 10−5	78.28%	79.49%	80.57%	80.82%	82.62%	82.50%
Λ = 10−7	78.28%	79.49%	80.57%	80.70%	82.75%	82.50%
Arrest
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	39.77%	39.68%	47.90%	63.90%	68.10%	69.50%
Λ = 10−3	39.95%	41.08%	46.33%	54.46%	58.91%	69.41%
Λ = 10−5	39.95%	40.91%	45.11%	55.86%	64.69%	69.41%
Λ = 10−7	39.95%	40.82%	45.20%	55.95%	64.51%	69.41%
Lending
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	59.94%	73.43%	82.09%	82.82%	82.82%	82.82%
Λ = 10−3	56.14%	62.45%	64.99%	75.34%	82.39%	82.82%
Λ = 10−5	56.11%	62.16%	65.66%	70.82%	75.11%	82.76%
Λ = 10−7	56.11%	62.02%	66.32%	70.49%	74.86%	82.77%
COMPAS
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	56.91%	65.37%	65.40%	65.49%	65.58%	65.58%
Λ = 10−3	52.63%	63.19%	63.39%	65.40%	65.43%	65.55%
Λ = 10−5	52.63%	63.62%	63.65%	65.70%	65.94%	66.92%
Λ = 10−7	52.68%	63.58%	63.54%	65.70%	65.77%	67.06%

TABLE 2:

LR accuracy with different regularization parameters

Mammographic Mass
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	78.16%	80.09%	81.42%	82.62%	83.59%	82.99%
Λ = 10−3	78.28%	79.49%	80.33%	80.70%	82.02%	84.19%
Λ = 10−5	78.28%	79.49%	80.33%	80.94%	81.18%	85.04%
Λ = 10−7	78.28%	79.49%	80.33%	80.82%	81.42%	85.04%
Arrest
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	39.77%	43.10%	47.82%	59.71%	66.88%	69.15%
Λ = 10−3	39.86%	40.91%	39.95%	51.14%	63.03%	69.58%
Λ = 10−5	39.86%	40.82%	44.93%	49.13%	58.57%	69.85%
Λ = 10−7	39.86%	40.82%	44.58%	49.04%	59.01%	69.85%
Lending
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	60.87%	70.54%	78.61%	82.11%	82.77%	82.82%
Λ = 10−3	51.06%	63.79%	69.47%	76.01%	81.14%	82.82%
Λ = 10−5	56.03%	62.02%	65.44%	68.92%	72.14%	82.54%
Λ = 10−7	56.03%	62.16%	64.13%	69.23%	72.03%	82.56%
COMPAS
∈	0.1	0.25	0.5	1	2	None
Λ = 10−2	55.69%	63.37%	65.57%	65.77%	65.83%	65.92%
Λ = 10−3	49.42%	61.88%	64.07%	66.00%	66.18%	66.49%
Λ = 10−5	52.12%	60.09%	64.47%	65.44%	67.80%	68.22%
Λ = 10−7	52.18%	60.10%	64.58%	65.67%	67.80%	68.38%

After choosing the best white-box model for the private and the non-private algorithms, the next step is selecting the privacy parameters for the α-violation privacy risk estimator and differential privacy. As discussed in Section 4.2, α-violation uses two parameters: the privacy parameter α and the privacy rate r_p which represents the percentage of the allowed α-violating rules. We experiment with α = 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1 and 2 when the privacy rate r_p = 50%. Differential privacy has only one parameter ϵ. In all the experiments, we use ϵ = 0.1, 0.25, 0.5, 1, 2, and 5.

For the Instance-based Transparency model (IT), we set the coverage λ = 2, nTrees = 10, and β = 1 − P(c_h) where Pr(c_h) is the probability of the class with the highest prior. We use a uniform value for the weight R_wj of IT rules to allow fair comparison between IT and BRL since BRL does not define weights for its rules.

6.4. Baseline Model Results

Figure 3 presents the agreement and the attack success results for all the datasets before applying any privacy technique.

Fig. 3:

Agreement and attack success for all the datasets before applying any privacy technique.

The attack success results before applying any privacy technique are presented in Figures 3(c) and 3(d) for the SVM and LR classifiers accordingly. We can observe that in most cases, attack success against both transparency models is almost always lower or approximately equal to the original model without any transparency. The only exception where a transparency model has an attack success that is higher than the original model by over 2% is with LR and Mass data. In this case, IT has an attack success approximately 10% higher than the original model. On average, IT reduces the attack success by 2% and BRL has almost the same attack success as the original model. These results demonstrate that releasing the transparency model can either reduce or maintain the probability of reconstructing individuals’ sensitive features when compared to releasing the original classifier, and at the same time can preserve high agreement with the model they explain.

Now we discuss the results of all the datasets when applying differential privacy alone and when combining differential privacy with α-violation.

6.5. Transparency Model Evaluation Under Attack

For all the results on privacy presented in this section, we display two figures for each privacy technique: agreement and attack success. For the agreement figures, we present IT and BRL agreements with the different classifiers they explain as solid lines. We also include in these figures the classifier accuracy (WB accuracy), IT accuracy and BRL accuracy represented by the dashed lines. For the attack success figures, we include the probability of the marginal represented by the dashed line and the attack success against IT, and BRL. For differential privacy results, we also include the attack success of the differentially private model with no transparency which we refer to as “None”. We cannot display the attack success of the model with no transparency with α-violation results since this privacy definition can only be applied to rule-based models.

Mammographic Mass

Figure 4 contains all the results for the Mass dataset. We notice in Figure 4(a) that differential private SVM has the highest accuracy and agreement when considering all the values for ϵ. However, differential privacy raises serious privacy concerns as the attack success remains high with all the ϵ values as shown in Figure 4(c). The lowest attack success is about 31% above the marginal with BRL when ϵ = 0.1.

Fig. 4:

Results for Mammographic Mass (Sensitive Feature is Assessment Score)

We can achieve high accuracy with differential privacy when ϵ ≥ 1 and reduce the attack success by applying α-violation for both IT and BRL based on the differentially private model. For example, when applying IT to differentially private SVM with ϵ = 5 and α = 0.6, we can achieve an accuracy of 77.32%, an agreement of 91.44%, and an attack success of 78.78% which is 11% less than the attack success without applying α-violation as shown in Figures 4(b) and 4(d).

Similar to SVM, differential private LR has the highest accuracy, agreement and attack success. With both differential privacy and α-violation, by using IT we can reduce the attack success by 23% with a high agreement rate of 82% when ϵ = 5 and α = 0.5. With BRL the best results for ϵ = 5 is with α = 0.1 where the attack success becomes 8% less than the baseline and the agreement is about 92%.

Arrest

The results on the Arrest data are presented in Figure 5. We notice that we can maintain high agreement with differential private SVM. Even though IT with differential privacy can achieve attack success lower than the marginal when ϵ ≤ 1, the accuracy is 40%, 40%, 48%, and 64% when ϵ = 0.1, 0, 25, 0.5, and 1 respectively. When using ϵ = 5 with α = 0.3 with IT, we can have a classifier accuracy of 69%, agreement of 94%, and attack success equal to the marginal as shown in Figures 5(b), and 5(d). With BRL when applying differential privacy with ϵ = 5 and α = 0.1, we can lower the attack success to 4% above the marginal with an agreement of 92%.

Fig. 5:

Results for Arrest (Sensitive Feature is Drug History)

We also notice that with differential privacy the attack success is not directly related to the ϵ value—decreasing ϵ may not necessarily make the model more robust against model inversion attack, i.e., it may not reduce the attack success. The reason is that differential privacy is designed to achieve individual privacy protection by making sure that with or without any individual’s data used for model learning, the learned models look similar. Differential privacy, by definition, provides no guarantees against the sensitive feature inference attacks used in our evaluation. Therefore, adding more noise using differential privacy with smaller ϵ can result in higher protection against instance inference attacks but does not guarantee higher protection against sensitive feature inference attacks, as demonstrated in our results.

The results for LR with differential privacy are very similar to SVM. When applying α-violation to differential private LR with ϵ = 5, both IT and BRL are able to reduce the attack success from the baseline (when no privacy model is applied) where it becomes equal to the marginal with IT and only 5% above the marginal with BRL.

We further demonstrate that our privacy model works even if we choose a sensitive feature with a less informative power for the classification. In Figure 6, we allow the gender to be the sensitive feature instead of the drug history for the Arrest data, with differential private SVM. The results on differential private SVM alone remain the same as in Figure 5(a) since we are using the same classifier and generating the differential private model is independent from the choice of the sensitive feature. However, the agreement of α-violation combined with differential privacy and the attack success for all the privacy models are affected by the choice of the sensitive feature as shown in Equation 3 and Equation 7 respectively. Similar to the previous results, α-violation combined with differential privacy can maintain high agreement while preserving the original classifier accuracy when ϵ = 5 as shown in Figure 6(a). When using differential privacy alone, we observe the same inconsistency in the increase of the attack success. In Figure 6(b) we show that the attack success is the highest when using differential private SVM with ϵ = 0.1. When combining differential privacy with α-violation, the attack success is the same as the marginal with IT when α = 0.1.

Fig. 6:

Results for Arrest-SVM (Sensitive Feature is Gender)

Hence, we show in these results that our method does not require the most important feature to be changed.

Lending Club

Figure 7 presents the results for lending data with SVM and LR. With differential privacy, the only case where we can achieve an attack success as low as the marginal is when ϵ = 0.1. The classifier accuracy for this case is about 60% with both SVM and LR.

Fig. 7:

Results for Lending Club (Sensitive Feature is FICO Score)

With ϵ = 5, the differentially private model has the same accuracy as the non-private classifier, 83% accuracy, and therefore applying α-violation will give us the same result as when applying α-violation to the non-private classifiers. In this case, we can have an attack success equal to the marginal with α ≤ 0.5 with IT for both SVM and LR. IT accuracy and agreement in this instance are 68.71% and 82.78% respectively. With BRL, the attack success can reach 5% above the marginal with an accuracy of 66% and agreement of 70%.

COMPAS

The results for the COMPAS data are presented in Figure 8. For differential private SVM, the lowest attack success is obtained with ϵ = 0.1 as shown in Figure 8(a). For this value, the classifier accuracy is only 57%, Figure 8(b). When combining differential private SVM with α-violation, BRL can have attack success equal to the marginal for α ≤ 0.5. However, for this case BRL agreement is about 52%.

Fig. 8:

Results for COMPAS (Sensitive Feature is Risk Score)

The best results for differential private LR is when ϵ = 0.25 where the classifier accuracy is 63.37%, IT accuracy is 63% and its agreement is 99%. The attack success in this case is 62.25% which is about 7% more than the marginal. However, with BRL the lowest attack success is 14% above the marginal when ϵ = 0.1. In this case the classifier accuracy is 56%. With both α-violation and differential privacy when ϵ = 5, we can lower the attack success even further so that it is very close to the marginal and in the mean time maintain good accuracy and good agreement. For example, with ϵ = 5 and α = 0.4 combined, we can have an attack success that is only 2% above the marginal with classifier accuracy of 66%, IT accuracy of 65% and agreement of 89%. The BRL accuracy is very close to the IT accuracy, while BRL always has higher attack success compared to IT.

7. Discussion

In this paper, we combined two different privacy techniques that can be applied to rule-based transparency models. We started by learning a transparency model using a differentially private classifier. We showed that differential privacy by itself fails to maintain high accuracy when it preserves privacy against model inversion attacks. We also proposed α-violation as a new privacy risk estimator to enhance privacy for rule-based transparency models. We showed that α-violation can enhance privacy protections against model inversion attacks while achieving higher utility. For example, in Table 3, we compare differential privacy results to α-violation when both models achieve similar attack success. Even though differential privacy can maintain higher agreement than α-violation, this agreement is not very useful since the white-box classifier always has much lower accuracy when it is differentially private. Combining differentially private classifier learning using large ϵ values with α-violation leads to high accuracy, high agreement, and low privacy leakage. The results for differentially private SVM and LR with ϵ = 1, 2, and 5 with all the datasets are shown in the supplement materials file. These results show that combining both techniques could be a reasonable approach for protecting the privacy of the transparency models while providing higher utility. Also, when comparing two different transparency models, we notice that there is a trade-off between model agreement and attack success. When a transparency model achieves higher agreement, it tends to leak more private information about individuals.

TABLE 3:

IT Attack Success-SVM

Data	Transparency Model	Privacy Model	Attack Success	WB Accuracy	Transparency Model Accuracy	Agreement	Rules
Arrest	BRL	α-violation, α=0.5	72.21%	69.50%	67.84%	93.62%	12.6
		Objective DP, ∈=0.1	72.38%	39.77%	40.47%	95.81%	18.6
Lending	IT	α-violation, α=0.5	36.29%	82.82%	68.71%	82.78%	2
		Objective DP, ∈=0.1	37.61%	59.94%	59.83%	96.69%	74.1
COMPAS	IT	α-violation, α=0.5	68.22%	67.06%	65.13%	89.55%	35.4
		Objective DP, ∈=0.1	64.25%	56.91%	56.60%	98.58%	47.6

The limitations of our technique are as follows: 1) The α-violation probabilities are estimated from the training data; 2) the technique is applicable to the global rule-based transparency models only; 3) the technique focuses on MI attacks as we are not aware of other attacks that consider conditional probabilities. Accessing the complete training data for α-violation probability estimation may be not be a feasible option due to privacy concerns. Therefore, we can overcome this limitation by assuming access to a sample of high quality data that has similar distribution to the training data. We also plan to expand our work in the future so that it can be applied to other types of transparency models such as feature-based models [9], [25].

8. Conclusion

In this paper, we investigated the privacy loss that can result from disclosing a rule-based transparency model. We show that without any privacy definition, disclosing a transparency model results in a high privacy risk. We examine the effect of learning a transparency model using a differential private classifier. Our results show that differential privacy can preserve privacy when ϵ value is small. However, for these cases, most of the times, differential privacy accuracy becomes comparable to random guessing. We propose α-violation privacy risk estimator that can result in private and accurate transparency model, especially when combined with differential privacy with large ϵ values. Our results demonstrate the success of α-violation in combination with differential privacy in preserving privacy while maintaining high accuracy.

Biography

Yasmeen Alufaisan is an IT System Analyst in the EXPEC Computer Operations Department at Saudi Aramco. She received her PhD from the University of Texas at Dallas. Her research focuses on different aspects of accountability in machine learning and data mining techniques.

Murat Kantarcioglu is a Professor in the Computer Science Department and Director of the UTD Data Security and Privacy Lab at the University of Texas at Dallas and a visiting scholar at Harvard University Data Privacy Lab. He is a recipient of NSF CAREER award, and Purdue CERIAS Diamond Award for Academic excellence. His research focuses on creating technologies that can efficiently extract useful information from any data without sacrificing privacy or security. His research has been supported by grants from NSF, AFOSR, ONR, NSA, and NIH. He has published over 160 peer reviewed papers related to data security, privacy and privacy-preserving data mining. His research work has been covered by the media outlets, such as Boston Globe, ABC News, and has received three best paper awards.

Yan Zhou Dr. Yan Zhou is a research scientist of the Data Security and Privacy Lab at the University of Texas at Dallas. She received her D.Sc. in Computer Science from Washington University in St. Louis. Dr. Zhou’s research focuses on developing robust machine learning techniques for problems where there are adversaries. She has also worked in the areas of semi-supervised learning, active learning, multiple instance learning, and compression-based classification.