Towards formalizing the GDPR’s notion of singling out

Significance

This article addresses a gap between legal and technical conceptions of data privacy and demonstrates how it can be minimized. The article focuses on “singling out,” which is a concept appearing in the GDPR. Our analysis draws on the legislation, regulatory guidance, and mathematical reasoning to derive a technical concept—“predicate singling out”—aimed at capturing a core part of GDPR’s intent. Examination of predicate singling out sheds light on the concept of singling out and the question of whether existing technologies protect against such a threat. Conceptually, this work demonstrates the role that principled analysis supported by mathematical argument can and should play in articulating and informing public policy at the interface between law and technology.

Abstract

There is a significant conceptual gap between legal and mathematical thinking around data privacy. The effect is uncertainty as to which technical offerings meet legal standards. This uncertainty is exacerbated by a litany of successful privacy attacks demonstrating that traditional statistical disclosure limitation techniques often fall short of the privacy envisioned by regulators. We define “predicate singling out,” a type of privacy attack intended to capture the concept of singling out appearing in the General Data Protection Regulation (GDPR). An adversary predicate singles out a dataset $\mathbf{x}$ using the output of a data-release mechanism $M\left(\mathbf{x}\right)$ if it finds a predicate $p$ matching exactly one row in $\mathbf{x}$ with probability much better than a statistical baseline. A data-release mechanism that precludes such attacks is “secure against predicate singling out” (PSO secure). We argue that PSO security is a mathematical concept with legal consequences. Any data-release mechanism that purports to “render anonymous” personal data under the GDPR must prevent singling out and, hence, must be PSO secure. We analyze the properties of PSO security, showing that it fails to compose. Namely, a combination of more than logarithmically many exact counts, each individually PSO secure, facilitates predicate singling out. Finally, we ask whether differential privacy and $k$-anonymity are PSO secure. Leveraging a connection to statistical generalization, we show that differential privacy implies PSO security. However, and in contrast with current legal guidance, $k$-anonymity does not: There exists a simple predicate singling out attack under mild assumptions on the $k$-anonymizer and the data distribution.

Data-privacy laws—like the Health Insurance Portability and Accountability Act, the Family Educational Rights and Privacy Act (FERPA), and Title 13 in the United States; and the European Union’s (EU’s) General Data Protection Regulation (GDPR)—govern the use of sensitive personal information.* These laws delineate the boundaries of appropriate use of personal information and impose steep penalties upon rule breakers. To adhere to these laws, practitioners need to apply suitable controls and statistical disclosure-limitation techniques. Many commonly used techniques, including pseudonymization, $k$-anonymity, bucketing, rounding, and swapping, offer privacy protections that are seemingly intuitive, but only poorly understood. And while there is a vast literature of best practices, a litany of successful privacy attacks demonstrates that these techniques often fall short of the sort of privacy envisioned by legal standards (e.g., ref. 1).

A more disciplined approach is needed. However, the significant conceptual gap between legal and mathematical thinking around data privacy poses a challenge. Privacy regulations are grounded in legal concepts, such as personally identifiable information (PII), linkage, distinguishability, anonymization, risk, and inference. In contrast, much of the recent progress in data-privacy technology is rooted in formal mathematical privacy models, such as differential privacy (2), that offer a foundational treatment of privacy, with provable privacy guarantees, meaningful semantics, and composability. And while such models are being actively developed and implemented in the academy, industry, and government, there is a lack of discourse between the legal and mathematical conceptions. The effect is uncertainty as to which technical offerings adequately match expectations expressed in legal standards (3).

Bridging between Legal and Technical Concepts of Privacy

We aim to address this uncertainty by translating between the legal and the technical. To do so, we begin with a concept appearing in the law, then model some aspect of it mathematically. With the mathematical formalism in hand, we can better understand the requirements of the law, their implications, and the techniques that might satisfy them.

In particular, we study the concept of “singling out” from the GDPR (4). We examine what it means for a data-anonymization mechanism to ensure “security against singling out” in a data release. Preventing singling out attacks in a dataset is a necessary (but maybe not sufficient) precondition for a dataset to be considered effectively anonymized and thereby free from regulatory restrictions under the GDPR. Ultimately, our goal is to better understand a concept foundational to the GDPR, by enabling a rigorous mathematical examination of whether certain classes of techniques (i.e., $k$-anonymity and differential privacy) provide security against singling out.

We are not the first to study this issue. The Article 29 Data Protection Working Party, an EU advisory body, provided guidance about the use of various privacy technologies—including $k$-anonymity and differential privacy—as anonymization techniques (5). Its analysis is centered on asking whether each technology effectively mitigates three risks: “singling out, linkability, and inference.” For instance, their “Opinion on Anonymisation Techniques” concludes that with $k$-anonymity, singling out is no longer a risk, whereas with differential privacy, it “may not” be a risk (5). Though similar in purpose to our work, its technical analyses are informal and coarse. Revisiting these questions with mathematical rigor, we recommend reconsidering the Working Party’s conclusions.

This work is part of a larger effort to bridge between legal and technical conceptions of privacy. An earlier work analyzed the privacy requirements of FERPA and modeled them in a game-based definition, as is common in cryptography (6). The definition was used to argue that the use of differentially private analyses suffices for satisfying a wide range of interpretations of FERPA. An important feature of FERPA that enabled that analysis is that FERPA and its accompanying documents contain a rather detailed description of a privacy attacker and the attacker’s goals.

1. Singling Out in the GDPR

We begin with the text of the GDPR (4). It consists of articles detailing the obligations placed on processors of personal data, as well as recitals containing explanatory remarks. Article 1 of the regulation delineates its scope:

This regulation lays down rules relating to the protection of natural persons with regard to the processing of personal data and rules relating to the free movement of personal data.

On the other hand, the GDPR places no restrictions on the processing of nonpersonal data, which includes personal data that have been “rendered anonymous.”^†

The term “personal data” is defined in Article 4 of the GDPR to mean “any information relating to an identified or identifiable natural person; an identifiable natural person is one who can be identified, directly or indirectly.” What it means for a person to be “identified, directly or indirectly” is further elaborated in Recital 26:

To determine whether a natural person is identifiable, account should be taken of all of the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly.

Thus, singling out is one way to identify a person in data, and only data that do not allow singling out may be excepted from the regulation.

Singling out is the only criterion for identifiability explicitly mentioned in the GDPR, the only occurrence of the term being the passage quoted above. For insight as to the meaning of singling out, we refer to two documents prepared by the Article 29 Data Protection Working Party.^‡“Opinion on the Concept of Personal Data” (7) elaborates on the meaning of “identifiable, directly or indirectly.” A person is identified “within a group of persons [when] he or she is distinguished from all other members of the group.” One way of distinguishing a person from a group is by specifying “criteria which allows him to be recognized by narrowing down the group to which he belongs.” If the group is narrowed down to an individual, that individual has been singled out.^§ Similarly, the Working Party’s “Opinion on Anonymisation Techniques” describes singling out as “isolat[ing] some or all records which identify an individual in [a] dataset.” Looking ahead, we will call this “isolating” an individual in the dataset and argue that not every instance of isolation should be considered a singling-out attack.

We highlight three additional insights from ref. 7 that inform our work. First, identification does not require a name or any other traditional identifier. For instance, singling out can be done with a “small or large” collection of seemingly innocuous traits (e.g., “the man wearing a black suit”) (7). Indeed, this is what is meant by indirectly identifiable. An example of singling out in practice cited by the Working Party “Opinion on Anonymisation Techniques” (5) showed that four locations sufficed to uniquely identify 95% of people in a pseudonymized dataset of time-stamped locations. This is considered singling out, even without a method of linking the location traces to individuals’ names.

Second, identifiable data may come in many forms, including microdata, aggregate statistics, news articles, encrypted data, video footage, and server logs. What’s important is not the form of the data—it is whether the data permit an individual to be singled out. We apply this same principle to the manner in which an individual is singled out within a dataset. Most examples focus on specifying a collection of attributes (e.g., four time-stamped locations) that match a single person in the data. A collection of attributes can be viewed as corresponding to a “predicate”: a function that assigns to each person in the dataset a value 0 or 1 (interpreted as $\mathsf{f}\mathsf{a}\mathsf{l}\mathsf{s}\mathsf{e}$ or $\mathsf{t}\mathsf{r}\mathsf{u}\mathsf{e}$, respectively). We interpret the regulation as considering data to be personal data if an individual can be isolated within a dataset using any predicate, not only those that correspond to collections of attributes.

Third, whether or not a collection of attributes identifies a person is context-dependent. “A very common family name will not be sufficient to identify someone—i.e., to single someone out—from the whole of a country’s population, while it is likely to achieve identification of a pupil in a classroom” (7). Both the prevalence of the name and the size of the group are important in the example and will be important in our formalization.

2. Main Findings

A. Defining Security Against Predicate Singling Out.

We formalize and analyze “predicate singling out,” a notion which is intended to model the GDPR’s notion of singling out. Following the discussion above, we begin with the idea that singling out an individual from a group involves specifying a predicate that uniquely distinguishes the individual, which we call “isolation.” Using this terminology, an intuitive interpretation of the GDPR is that rendering data anonymous requires preventing isolation. Trying to make this idea formal, we will see that it requires some refinement.

We restrict our attention to datasets $\mathbf{x}=\left(x_1,\dots ,x_n\right)$ of size $n$, where each row $x_i$ is sampled independently from some underlying probability distribution $D$ over a universe $X$. The dataset $\mathbf{x}$ is assumed to contain personal data corresponding to individuals, with at most one row per individual. For example, $\mathbf{x}$ might consist of home listings, hospital records, internet browsing history, or any other personal information. A mechanism $M$ takes $\mathbf{x}$ as input and outputs some data release $\mathbf{y}=M\left(\mathbf{x}\right)$, be it a map of approximate addresses, aggregate statistics about disease, the result of applying machine learning techniques, or pseudonymized internet histories. We call $M$ an “anonymization mechanism” because it purportedly anonymizes the personal data $\mathbf{x}$.

An adversary $\mathsf{A}$ attempts to output a predicate $p:X\to \left\{\mathrm{0,1}\right\}$ that isolates a row in $\mathbf{x}$, i.e., there exists $i\in \left[n\right]$ such that $p\left(x_i\right)=1$ and $p\left(x_j\right)=0$ for all $j\hspace{0.17em}\ne \hspace{0.17em}i$. We emphasize that it is rows in the original dataset $\mathbf{x}$ on which the predicate acts, not the output $\mathbf{y}$. In part, this is a by-product of our desire to make no assumptions on the form of $M$’s output. While it might make sense to apply a predicate to isolate a row in pseudonymized microdata, it is far from clear what it would mean for a synthetic dataset or for aggregate statistics. Observe that this choice also rules out predicates $p$ that “isolate” rows by referring to their position in $\mathbf{x}$ (i.e., “the seventh row”).

$M$ prevents isolation if there doesn’t exist an adversary $\mathsf{A}$ that isolates a row in $\mathbf{x}$, except with very small probability over the randomness of sampling $\mathbf{x}\leftarrow D^n$, the randomness of the mechanism $\mathbf{y}\leftarrow M\left(\mathbf{x}\right)$, and the randomness of adversary $\mathsf{A}\left(\mathbf{y}\right)$. Unfortunately, this is impossible to achieve by any mechanism $M$. To wit, there are trivial adversaries—those that do not look at the outcome $\mathbf{y}$—that isolate a row with probability approximately $37\%$. The trivial adversaries simply output any predicate $p$ that matches a $1/n$ fraction of the distribution $D$. Isolation is, hence, not necessarily indicative of a failure to protect against singling out, as a trivial adversary would succeed with $\approx 37\%$ probability (for any $n$), even if $M$ does not output anything at all.

Example: Consider a dataset of size $n=365$, including information about random people selected at random from the US population. To isolate a person, a trivial adversary may output $p=\left(\text{born on March 15th}\right)$. This predicate will isolate a row with probability

$$\left(\genfrac{}{}{0ex}{}{365}{1}\right)\cdot \frac{1}{365}\cdot {\left(1-\frac{1}{365}\right)}^{\mathrm{364}}\approx 37\%.$$

Furthermore, a trivial adversary need not know the distribution $D$ to isolate with probability $\approx 37\%$, as long as $D$ has sufficient min-entropy (Fact 3.1).

A trivial adversary can give us a baseline against which to measure isolation success. But the baseline should not simply be a 37% chance of success. Consider the earlier example of a dataset of 365 random Americans. What if an adversary output predicates like $p=\left(\text{born on March}15\text{th}\wedge \text{vegan}\wedge \text{speaks Dutch}\wedge \text{concert pianist}\right)$, and managed to isolate 10% of the time? Though 10% is much less than 37%, the predicate is extremely specific and unlikely to isolate a person by chance.

We formalize this intuition by considering the baseline risk of isolation as a function of the weight of the predicate $p$: the chance that $p$ matches a random row sampled from the distribution $D$. The baseline for predicates of weight $1/n$ is 37%, but the baseline for an extremely specific predicate may be much lower. The more specific the predicate, the closer the baseline gets to zero. Our primary focus in this paper is on the regime of predicate weights where the baseline is negligible, corresponding to predicates with negligible weight.^¶

Definition 4.5 (PSO Security, Informal): An adversary predicate singles out a row in $\mathbf{x}$ if it outputs a predicate that isolates a row with probability significantly higher than the baseline risk. A mechanism $M$ is “secure against predicate singling out” (PSO secure) if no adversary can use its output $\mathbf{y}=M\left(\mathbf{x}\right)$ to predicate single out.

B. Analyzing Security Against Predicate Singling Out.

Having formulated PSO security, our next goal is to understand the guarantee it offers, what mechanisms satisfy it, and how this concept relates to existing privacy concepts, including differential privacy and $k$-anonymity.

Two desirable properties of a privacy concept are robustness to “postprocessing” and to “composition.” The former requires that if a mechanism $M$ is deemed secure, then anything that can be computed using the outcome of $M$ should also be deemed secure. Hence, the outcome may be reused without creating additional privacy risk. For instance, if a PSO-secure mechanism $M$ outputs microdata, then any statistics that can be computed from those microdata should also be PSO secure. PSO security is robust to postprocessing. The analysis is simple and shown in ref. 9.

Incomposability of PSO Security.

We would like that the privacy risk of multiple data releases is not significantly greater than the accumulated risks of the individual releases. In this case, we say that the privacy concept composes. We prove that PSO security does not compose and give two examples of this failure. First, we show that releasing a single aggregate statistic is PSO secure, but superlogarithmically many statistics may allow an adversary to predicate single out. Concretely, a mechanism that outputs a single count is PSO secure. Yet a mechanism that outputs $\omega \left(\log \left(n\right)\right)$ counts may allow an adversary to isolate a row with probability arbitrarily close to one by using a predicate with negligible weight (and negligible baseline). Second, we construct a less natural pair of mechanisms that individually are PSO secure but together allow an adversary to recover a row in the dataset. Consequently, the adversary can predicate single out by isolating this row using a predicate with negligible weight.

Do Differential Privacy and $k$-Anonymity Guarantee PSO Security?

Differential privacy is a requirement of data analyses mechanisms that limits the dependency of a mechanism’s output distribution on any single individual’s contribution (10). $k$-anonymity is a requirement from data releases that the (quasi) identifying data of every person in the release should be identical to that of at least $k-1$ other individuals in the release (11, 12) (see Definitions 6.1 and 7.1 for formal definitions).

Differential privacy is not necessary for PSO security, as an exact count is PSO secure, but is not differentially private. However, differential privacy does provide PSO security. The proof relies on the connection between differential privacy and statistical generalization guarantees (13, 14). We show that predicate singling out implies a form of overfitting to the underlying dataset. If a mechanism is differentially private, it prevents this form of overfitting and, hence, protects against predicate singling out.

On the other hand, we show that $k$-anonymity does not prevent predicate singling out attacks. Instead, it enables an adversary to predicate single out with probability approximately 37%, even using extremely low-weight predicates for which the baseline risk is negligible. Briefly, the attack begins by observing that typical $k$-anonymous algorithms “almost” predicate single out. They reveal simple predicates—usually, conjunctions of attributes—that are satisfied by groups of $k$ rows in the dataset. In an effort to make the $k$-anonymized data as useful as possible, these predicates are as descriptive and specific as possible. To predicate single out a row from the $k$-anonymous dataset, it roughly suffices to isolate a row from within one of these groups.

C. Implication for the GDPR.

Precisely formalizing predicate singling out attacks allows us to examine with mathematical rigor the extent to which specific algorithms and paradigms protect against them. In particular, we show that k-anonymity fails to prevent predicate singling out, but that differential privacy prevents predicate singling out. Our conclusions contrast with those of the Article 29 Working Party: They conclude that $k$-anonymity eliminates the risk of singling out, while differential privacy “may not” (5). This disagreement may raise doubts about whether our modeling indeed matches the regulators’ intent, which we now address.

Our goal in interpreting the text of the GDPR and related documents, and in defining predicate singling out, is to provide a precise mathematical formalism to capture some aspect of the concept of personal data (as elucidated in the regulation and in ref. 7) and the associated concept of anonymization. We want to render mathematically falsifiable a claim that a given algorithmic technique anonymizes personal data under the GDPR by providing a necessary condition for such anonymizers.

We argue that predicate singling out succeeds. A number of modeling choices limit the scope of our definition, but limiting the scope poses no issue. Specifically, 1) we only consider randomly sampled datasets; 2) we only consider an attacker who has no additional knowledge of the dataset besides the output of a mechanism; and 3) we do not require that isolation be impossible, instead comparing against a baseline risk of isolation. A technique that purports to anonymize all personal data against all attackers must at least do so against randomly sampled data and against limited attackers. And unless the idea of anonymization mechanisms is completely vacuous, one must compare against a baseline risk.

We don’t mean to claim that our modeling is the only one possible. The starting point for the analysis is a description which does not use mathematical formalism, but is, rather, a (somewhat incomplete) description using natural language. Alternative mathematical formalizations of singling out could probably be extracted from the very same text, and we look forward to seeing them emerge.

Policy Implications.

Assuming our formalization is in agreement with the GDPR’s notion of singling out, the most significant consequence of our analysis is that $k$-anonymity does not prevent singling out (and likewise $\ell$-diversity and $t$-closeness). Thus, it is insufficient for rendering personal data anonymous and excepting them of GDPR regulation. If PSO security is a necessary condition for GDPR anonymization, then something more is required. On the other hand, differential privacy might provide the necessary level of protection. At least it is not ruled out by our analysis.

More abstractly, we believe that self-composition is an essential property of any reasonable privacy concept. Our finding that PSO security does not self-compose is evidence that self-composition should not be taken for granted, but be a criterion considered when developing privacy concepts.

One may still claim that the assessments made in ref. 5 should be taken as ground truth and that the Article 29 Working Party meant for any interpretation of singling out to be consistent with these assessments. According to this view, the protection provided by $k$-anonymity implicitly defines the meaning of singling out (partially or in full). Such a position would be hard to justify. To the best of our knowledge, the assessments made by the Article 29 WP were not substantiated by a mathematical analysis. Defining privacy implicitly as the guarantee provided by particular techniques is an approach proven to fail (1).

Is PSO Security a Good Privacy Concept?

A predicate singling out attack can be a stepping stone toward a greater harm, even in settings where isolation alone may not. It may enable linking a person’s record in the dataset to some external source of information (15), or targeting individuals for differential treatment. As such, it is meaningful as a mode of privacy failure, both in the GDPR context and otherwise.

While we believe that PSO security is relevant for the GDPR as a necessary property of techniques that anonymize personal data, we do not consider it sufficiently protective by itself. First, singling out is only one mode of privacy failure. Many other failure modes have been considered, including reconstruction attacks, membership attacks, inference, and linkage. Second, our definition considers a setting where the underlying data are drawn from some (unknown) underlying distribution, an assumption that is not true in many real-life contexts. In such contexts, PSO security may not prevent singling out under the GDPR. Lastly, the incomposability of PSO security renders it inadequate in isolation and suggests that it should be complemented by other privacy requirements.

3. Notation

A dataset $\mathbf{x}=\left(x_1,\dots ,x_n\right)$ consists of $n$ elements taken from the data universe $X={\left\{\mathrm{0,1}\right\}}^d$. We consider datasets where each entry $x_i$ is independently sampled from a fixed probability distribution $D$ over $X$. We denote by $U_d$ a random variable sampled uniformly at random from ${\left\{\mathrm{0,1}\right\}}^d$.

A mechanism $\mathsf{M}$ is a Turing machine that takes as input a dataset $\mathbf{x}\in X^n$. Mechanisms may be randomized and interactive.

A predicate is a binary-valued function $p:X\to \left\{\mathrm{0,1}\right\}$. We define the weight of a predicate $p$ to be ${\mathsf{w}\mathsf{t}}_D\left(p\right)\triangleq \underset{x\sim D}{Pr}\left[p\left(x\right)=1\right]$. For a dataset $\mathbf{x}\in X^n$, let $p\left(\mathbf{x}\right)\triangleq \frac{1}{n}{\sum }_{i=1}^{n}p\left(x_i\right)$. We occasionally use indicator notation $\mathbb{I}\left(\right)$ to define a predicate; for example, $p\left(x\right)=\mathbb{I}\left(x\in A\right)$ equals 1 if $x\in A$ and 0 otherwise.

For the purposes of asymptotic analyses, we use the number of rows $n$ in a dataset as the complexity parameter. We omit the dependence of $d=d\left(n\right)$ on n.^∥A function $f\left(n\right)$ is negligible, denoted $f\left(n\right)=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$, if for all $c>0$, there exists $n_c>0$ such that $f\left(n\right)\le n^{-c}$ for all $n>n_c$. Informally, this means that $f\left(n\right)$ approaches 0 faster than any inverse polynomial function for large enough $n$.

Many of our results apply to all data distributions with sufficient “min-entropy,” a measure of randomness useful for cryptography. The min-entropy of a probability distribution $D$ over a universe $X$ is $H_{\infty }\left(D\right)=-\log \left(\underset{y\in X}{max}\underset{x\sim D}{Pr}\left[x=y\right]\right)$. The relevant results apply to all distributions with even a moderate amount of min-entropy: $H_{\infty }\left(D\right)>\omega \left(\log \left(n\right)\right)+\log \left(1/\alpha \right)$ for some $\alpha =\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ (e.g., $H_{\infty }\left(D\right)>{\log }^{1+c}\left(n\right)$ for $c>0$). We will call such distributions “min-entropic.”

Fact 3.1. For any min-entropic $D$ and any weight $w\in \left[\mathrm{0,1}\right]$, there exist predicates $p_{-}$, $p_{+}$ and a negligible function $\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ such that ${\mathsf{w}\mathsf{t}}_D\left(p_{-}\right)\in \left[w-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right),w\right]$ and ${\mathsf{w}\mathsf{t}}_D\left(p_{+}\right)\in \left[w+\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)\right]$ [using the Leftover Hash Lemma (16)].

4. Security Against Predicate Singling Out

Consider a data controller who has in its possession a dataset $\mathbf{x}=\left(x_1,\dots ,x_n\right)$ consisting of $n$ rows sampled independently from a probability distribution $D$. We think of the dataset as containing the personal data of $n$ individuals, one per row. The data controller publishes the output of an anonymization mechanism $\mathsf{M}$ applied to the dataset $\mathbf{x}$. A predicate singling out adversary $A$ is a nonuniform probabilistic Turing machine with access to the published output $M\left(\mathbf{x}\right)$ and produces a predicate $p:X\to \left\{\mathrm{0,1}\right\}$. We abuse notation and write $\mathsf{A}\left(M\left(\mathbf{x}\right)\right)$, regardless of whether $M$ is an interactive or noninteractive mechanism. For now, we assume that all adversaries have complete knowledge of $D$ and are computationally unbounded**

The Article 29 Working Party describes singling out as “isolat[ing] some or all records which identify an individual in [a] dataset” (5). This is done by “narrowing down [to a singleton] the group to which [the individual] belongs” by specifying “criteria which allows him to be recognized” (7). We call this row isolation.

Definition 4.1 (Row Isolation): A predicate $p$ isolates a row in a dataset $\mathbf{x}$ if there exists a unique $x\in \mathbf{x}$ such that $p\left(x\right)=1$. That is, if $p\left(\mathbf{x}\right)=1/n$. We denote this event $\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)$.

It is tempting to require that an anonymization mechanism $M$ only allows an adversary to isolate a row with negligible probability, but this intuition is problematic. An adversary that does not have access to $M$—a trivial adversary—can output a predicate $p$ with ${\mathsf{w}\mathsf{t}}_D\left(p\right)\approx 1/n$ (by Fact 3.1) and thereby isolate a row in $\mathbf{x}$ with probability

$$\left(\genfrac{}{}{0ex}{}{n}{1}\right)\cdot {\mathsf{w}\mathsf{t}}_D\left(p\right)\cdot {\left(1-{\mathsf{w}\mathsf{t}}_D\left(p\right)\right)}^{n-1}\approx {\left(1-\frac{1}{n}\right)}^{n-1}\approx \frac{1}{e}\approx 37\%.$$

In Bounding the Baseline, we will see that, in many cases, the trivial adversary need not know the distribution to produce such a predicate.

A. Security Against Predicate Singling Out.

We first define a measure of an adversary $\mathsf{A}$’s success probability in isolating a row given the output of a mechanism $M$ while being restricted to output a predicate $p$ of at most a given weight $w$.

Definition 4.2 (Adversarial Success Probability): Let $D$ be a distribution over $X$. For mechanism $M$, an adversary $\mathsf{A}$, a dataset size $n\in \mathbb{N}$, and a weight $w\le 1/n$ let

$${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M}\left(n,D\right)\triangleq \underset{\begin{array}{c}\mathbf{x}\leftarrow D^n\\ p\leftarrow \mathsf{A}\left(M\left(\mathbf{x}\right)\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\right].$$

Instead of considering the absolute probability that an adversary isolates a row, we consider the increase in probability relative to a baseline risk: the probability of isolation by a trivial adversary, $\mathsf{T}$.

Definition 4.3 (Trivial Adversary): A predicate singling out adversary $\mathsf{T}$ is trivial if the distribution over outputs of $\mathsf{T}$ is independent of $M\left(\mathbf{x}\right)$. That is, $\mathsf{T}\left(M\left(\mathbf{x}\right)\right)=\mathsf{T}\left(\perp \right)$.

Definition 4.4 (Baseline): For $n\in \mathbb{N}$ and weight $w\le 1/n$,

$${\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}}_D\left(n,w\right)\triangleq \underset{\text{Trivial}\mathsf{T}}{max}\hspace{0.17em}{\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{T},\perp }\left(n,D\right).$$

The baseline lets us refine the Working Party’s conception of singling out as row isolation. We require that no adversary should have significantly higher probability of isolating a row than a trivial adversary, when both output predicates of weight less than $w$.

Definition 4.5 (Security Against Predicate Singling Out): For $\epsilon \left(n\right)\ge 0$, $\delta \left(n\right)\ge 0$, $w_{\mathsf{m}\mathsf{a}\mathsf{x}}\left(n\right)\le 1/n$, we say a mechanism $M$ is $\left(\epsilon ,\delta ,w_{\mathsf{m}\mathsf{a}\mathsf{x}}\right)$-secure against predicate singling out ($\left(\epsilon ,\delta ,w_{\mathsf{m}\mathsf{a}\mathsf{x}}\right)$-PSO secure) if for all $\mathsf{A}$, $D$, $n$, and $w\le w_{\mathsf{m}\mathsf{a}\mathsf{x}}$:

$${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M}\left(n,D\right)\le e^{\epsilon \left(n\right)}\cdot {\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}}_D\left(n,w\right)+\delta \left(n\right).$$

We often omit explicit reference to the parameter $n$ for $\epsilon$, $\delta$, $w_{\mathsf{m}\mathsf{a}\mathsf{x}}$, and to the distribution $D$ when it is clear from context.

The definition is strengthened as $\epsilon$ and $\delta$ get smaller. As shown next, $\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ when $w=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. This is the most important regime of Definition 4.5, as such predicates not only isolate a row in the dataset, but likely also isolate an individual in the entire population.

We say a mechanism is PSO secure if for all $w_{\mathsf{m}\mathsf{a}\mathsf{x}}=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ there exists ${\delta }^{\prime }=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ such that $M$ is $\left(0,{\delta }^{\prime },w\left(n\right)w_{\mathsf{m}\mathsf{a}\mathsf{x}}\right)$-PSO secure. Observe that for all $\epsilon =O\left(\log n\right)$, $\delta =\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$, and $w_{\mathsf{m}\mathsf{a}\mathsf{x}}\le 1/n$, $\left(\epsilon ,\delta ,w_{\mathsf{m}\mathsf{a}\mathsf{x}}\right)$-PSO security implies PSO security.^††

B. Bounding the Baseline.

In this section, we characterize the baseline over intervals in terms of a simple function $B\left(n,w\right)$. For $n\ge 2$ and a predicate $p$ of weight $w$, the probability over $\mathbf{x}\sim D^n$ that $p$ isolates a row in $\mathbf{x}$ is

$$B\left(n,w\right)\triangleq n\cdot w\cdot {\left(1-w\right)}^{n-1}.$$

$B\left(n,w\right)$ is maximized at $w=1/n$ and strictly decreases moving away from the maximum. ${\left(1-1/n\right)}^{n-1}$ approaches $e^{-1}$ as $n\to \infty$ and does so from above (recall that ${\left(1-1/n\right)}^n\approx e^{-1}$ even for relatively small values of $n$).

Claim 4.1. For all $n$, $w\le 1/n$, and $D$, ${\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}}_D\left(n,w\right)\le B\left(n,w\right)$. For min-entropic $D$, ${\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}}_D\left(n,w\right)\ge B\left(n,w\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$.

Proof: Observe that for any randomized trivial adversary $\mathsf{T}$, there exists a deterministic trivial adversary ${\mathsf{T}}^{\prime }$ such that ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{{\mathsf{T}}^{\prime },\perp }\left(n\right)\ge {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{T},\perp }\left(n\right)$. Therefore, without loss of generality, one can assume that the trivial adversary that achieves the baseline success probability is deterministic.

A deterministic ${\mathsf{T}}^{\prime }$ always outputs a predicate $p^{\prime }$ of weight $\mathsf{w}\mathsf{t}\left(p^{\prime }\right)=w^{\prime }$.

$$\underset{\mathbf{x}\sim D^n}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p^{\prime },\mathbf{x}\right)\right]=\left(\genfrac{}{}{0ex}{}{n}{1}\right)\cdot w^{\prime }\cdot {\left(1-w^{\prime }\right)}^{n-1}.$$

Therefore,

$$\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)=\underset{\begin{array}{c}{w}^{\prime }\le w\\ w^{\prime }=\mathsf{w}\mathsf{t}\left(p^{\prime }\right)\end{array}}{sup}n{w}^{\prime }{\left(1-w^{\prime }\right)}^{n-1}\le B\left(n,w\right),$$

where the last inequality follows from the monotonicity of $B\left(n,w^{\prime }\right)$ in the range $w^{\prime }\in \left[0,w\right]$. By Fact 3.1, there exists $p^{\prime }$ such that $\mathsf{w}\mathsf{t}\left(p^{\prime }\right)\ge w-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. Hence, $\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)\ge B\left(n,\mathsf{w}\mathsf{t}\left(p^{\prime }\right)\right)\ge B\left(n,w\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. $\square$

Examples: For a (possibly randomized) function $f$, consider the mechanism $M_f$ that on input $\mathbf{x}=\left(x_1,\dots ,x_n\right)$ outputs $M_f\left(\mathbf{x}\right)=\left(f\left(x_1\right),\dots ,f\left(x_n\right)\right)$. Whether $M_f$ is PSO secure depends on the choice of $f$:

• •Identity function: If $f\left(x\right)=x$, then $M_f\left(\mathbf{x}\right)=\left(x_1,\dots ,x_n\right)$ and every distinct $x_i\in \mathbf{x}$ can be isolated by a predicate $p_i:x\mapsto \mathbb{I}\left(x=x_i\right)$. If $\underset{x\sim D}{Pr}\left[x=x_i\right]=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ (e.g., when $D$ is uniform over ${\left\{\mathrm{0,1}\right\}}^{\omega \left(\log \left(n\right)\right)}$ or $D$ is min-entropic), then ${\mathsf{w}\mathsf{t}}_D\left(p_i\right)=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. Hence, $M_f$ is not PSO secure.
• •Pseudonymization: If $f$ is one-to-one and public, it offers no more protection than the identity function. For unique $x_i\in \mathbf{x}$ and $y_i=f\left(x_i\right)$, the predicate $p_i:x\mapsto \mathbb{I}\left(f\left(x\right)=y_i\right)$ isolates $x_i$. If $D$ is min-entropic, ${\mathsf{w}\mathsf{t}}_D\left(p_i\right)=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. Observe that $M_f$ is not PSO secure, even if $f^{-1}$ is not efficiently computable. Furthermore, $f$ being many-to-one does not guarantee PSO security. For instance, suppose the data are uniform over ${\left\{\mathrm{0,1}\right\}}^n$ and $f:{\left\{\mathrm{0,1}\right\}}^n\to {\left\{\mathrm{0,1}\right\}}^{n/2}$ outputs the last $n/2$ bits of an input $x$. $M_f$ is not PSO secure. In fact, it is possible to single out every row $x\in \mathbf{x}$ using the same predicates $p_i$ as above. Together, these observations challenge the use of some forms of pseudonymization.
• •Random function: If $f\left(x\right)$ is a secret, random function, then $M_f\left(\mathbf{x}\right)$ carries no information about $\mathbf{x}$ and provides no benefit to the adversary over a trivial adversary. For every $\mathbf{x}$, a trivial adversary $\mathsf{T}$ can perfectly simulate any adversary $\mathsf{A}\left(M_f\left(\mathbf{x}\right)\right)$ by executing $\mathsf{A}$ on a random input. Hence, $M_f$ is PSO secure.

C. Reflections on Modeling.

In many ways, Definition 4.5 demands the sort of high level of protection typical in the foundations of cryptography. It requires a mechanism to provide security for all distributions $D$ and against nonuniform, computationally unbounded adversaries.^‡‡ The main weakness in the required protection is that it considers only data that are independent and identically distributed (i.i.d.), whereas real-life data cannot generally be modeled as i.i.d.

Any mechanism that purports to be a universal anonymizer of data under the GDPR—by transforming personal data into nonpersonal data—must prevent singling out. Our definition is intended to capture a necessary condition for a mechanism to be considered as rendering data sufficiently anonymized under the GDPR. Any mechanism that prevents singling out in all cases must prevent it in the special case that the data are i.i.d. from a distribution $D$ and for $w_{\mathsf{m}\mathsf{a}\mathsf{x}}=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. We view a failure to provide security against predicate singling out (Definition 4.5) as strong evidence that a mechanism does not prevent singling out as conceived of by the GDPR.

On the other hand, satisfying Definition 4.5 is not sufficient for arguing that a mechanism renders data anonymized under the GDPR. Singling out is only one of the many “means reasonably likely to be used” (4) to identify a person in a data release. Furthermore, the definition considers only i.i.d. data; it may not even imply that a mechanism prevents singling out in other relevant circumstances.

It is important that our definitions are parameterized by the weight $w$. An unrestricted trivial adversary can isolate a row with probability of about $1/e\approx 37\%$ by outputting a predicate of weight about $1/n$. If 37% was used as the general baseline (without consideration of the weight of the predicate), then the definition would permit an attacker to learn very specific information about individuals in a dataset, as long as it does so with probability less than $37\%$. For instance, such a definition would permit a mechanism that published a row from the dataset with a one in three chance.

Remark 4.2. The baseline is also negligible when the trivial adversary is required to output predicates with weight at least $\omega \left(\log n/n\right)$. It is not clear to the authors how beneficial finding a predicate in this regime may be to an attacker. This high-weight regime is analyzed analogously in ref. 9.

5. Properties of PSO Security

Two desirable properties of privacy concepts are 1) immunity to postprocessing (i.e., further processing of the outcome of a mechanism, without access to the data, should not increase privacy risks), and 2) closure under composition (i.e., a combination of two or more mechanisms which satisfy the requirements of the privacy concept is a mechanism that also satisfies the requirements, potentially with worse parameters). It is easy to see that PSO security withstands postprocessing. However, it does not withstand composition, and we give two demonstrations for this fact.

First, we consider mechanisms which count the number of dataset rows satisfying a property and show that every such mechanism is PSO secure. However, there exists a collection of $\omega \left(\log \left(n\right)\right)$ counts which allows an adversary to isolate a row with probability arbitrarily close to one using a predicate with negligible weight. Second, we construct a (less natural) pair of mechanisms that are individually PSO secure, but together allow the recovery of a row in the dataset. This construction borrows ideas from ref. 17.

Not being closed under composition is a significant weakness of the notion of PSO security. Our constructions rely on very simple mechanisms that we expect would be considered as preventing singling-out attacks (as a legal matter), even under other formulations of the concept. As such, it may well be that nonclosure under composition is an inherent property of the concept of singling out.

As a policy matter, we believe that closure under composition (as well as immunity to postprocessing) should be considered prerequisites for any privacy concept deemed sufficient to protect individuals’ sensitive data. Pragmatically, the fact that PSO security is not closed under composition suggests that this concept is best used for disqualifying privacy technology (i.e., if it is not PSO secure). This concept should not be used alone to certify or approve the use of any technology.

A. A PSO-Secure Counting Mechanism.

For any predicate $q:X\to \left\{\mathrm{0,1}\right\}$, we define the corresponding Counting Mechanism:

Algorithm 1: Counting Mechanism $M_{\#q}$:
input: $\mathbf{x}=\left(x_1,\dots ,x_n\right)$
return $\left|\left\{1\le i\le n:q\left(x_i\right)=1\right\}\right|$

For example, consider the least-significant bit predicate $\mathsf{l}\mathsf{s}\mathsf{b}$, that takes as input a string $x\in {\left\{\mathrm{0,1}\right\}}^d$ and outputs the first bit $x\left[1\right]$. The corresponding Counting Mechanism $M_{\#\mathsf{l}\mathsf{s}\mathsf{b}}$ returns the sum of the first column of $\mathbf{x}$.

$M_{\#q}$ is PSO secure for any predicate $q$. This is a corollary of the following proposition.

Proposition 5.1. For all $\mathsf{A}$, $n>0$, $w\le 1/n$, and $M:X^n\to Y$, ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M}\left(n\right)\le \left|Y\right|\cdot \mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)$.

Proof: We define a trivial adversary $\mathsf{T}$ such that for all $\mathsf{A}$, ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{T},\perp }\left(n\right)\ge \frac{1}{\left|Y\right|}\cdot {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{A,M}\left(n\right)$. The proposition follows by the definition of $\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)$. $\mathsf{T}$ samples a random $y{\in }_{R}Y$ and returns $p\leftarrow \mathsf{A}\left(y\right)$. For all datasets $\mathbf{x}$, there exists $y^{*}=y^{*}\left(\mathbf{x}\right)\in Y$ such that

$$\begin{array}{ll}\hfill & \underset{p\leftarrow \mathsf{A}\left(y^{*}\right)}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\wedge \mathsf{w}\mathsf{t}\left(p\right)\le w\right]\hfill \\ \hfill & \hspace{1em}\ge \underset{p\leftarrow \mathsf{A}\left(M\left(\mathbf{x}\right)\right)}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\wedge \mathsf{w}\mathsf{t}\left(p\right)\hspace{0.17em}\le \hspace{0.17em}w\right].\hfill \end{array}$$

For all $\mathbf{x}$, ${Pr}_{y{\in }_{R}Y}\left[y=y^{*}\right]=\frac{1}{\left|Y\right|}.$ Therefore,

$${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{T},\perp }\left(n\right)=\underset{\begin{array}{c}\mathbf{x}\leftarrow D^n\\ y{\in }_{R}Y\\ p\leftarrow \mathsf{A}\left(y\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\wedge \mathsf{w}\mathsf{t}\left(p\right)\le w\right]\ge \frac{{\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{A,M}\left(n\right)}{\left|Y\right|}.$$

$\square$As exact counts are not differentially private, the PSO security of $M_{\#q}$ demonstrates that differential privacy (Definition 6.1) is not necessary for PSO security. The security of a single exact count easily extends to $O\left(1\right)$-many counts (even adaptively chosen), as the size of the codomain grows polynomially.

B. Failure to Compose.

Our next theorem states that a fixed set of $\omega \left(\log \left(n\right)\right)$ counts suffices to predicate single out with probability close to $e^{-1}$. For a collection of predicates $Q=\left(q_0,\dots ,q_m\right)$, let $M_{\#Q}\left(\mathbf{x}\right)\triangleq \left(M_{\#q_0}\left(\mathbf{x}\right),\dots ,M_{\#q_m}\left(\mathbf{x}\right)\right)$. Let $D=U_d$ be the uniform distribution over $X={\left\{\mathrm{0,1}\right\}}^d$ for some $d=\omega \left(\log \left(n\right)\right)$.

Theorem 5.2. For any $m\le d$, there exists a $Q$ and $\mathsf{A}$ such that

$${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_{2^{-m}}^{\mathsf{A},M_{\#Q}}\left(n\right)\ge B\left(n,1/n\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right).$$

Choosing $m=\omega \left(\log \left(n\right)\right)$ yields $2^{-m}=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$.

Proof: Treating $x\in {\left\{\mathrm{0,1}\right\}}^d$ as a binary number in $\left[0,2^d-1\right]$, let $q_0\left(x\right)=\mathbb{I}\left(x<2^d/n\right)$. Observe that $\mathsf{w}\mathsf{t}\left(q_0\right)=1/n-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$.

For $i\in \left\{1,\dots ,m\right\}$, define the predicate $q_i\left(x\right)\triangleq \left(q_0\left(x\right)\wedge x\left[i\right]\right)$, and let $y_i=M_{\#q_i}\left(\mathbf{x}\right)$. Observe that if it happens that $q_0$ isolates row $j^{*}$ of $\mathbf{x}$, then $y_i=x_{j^{*}}\left[i\right]$. Consider the deterministic adversary $\mathsf{A}$ that on input $M_{\#Q}\left(\mathbf{x}\right)=\left(y_0,\dots ,y_m\right)$ outputs the predicate

$$p\left(x\right)=q_0\left(x\right)\wedge \left(\underset{i=1}{\overset{m}{\bigwedge }}\left(x\left[i\right]=y_i\right)\right).$$

Observe that $\mathsf{i}\mathsf{s}\mathsf{o}\left(q_0,\mathbf{x}\right)\hspace{0.17em}\Rightarrow \hspace{0.17em}\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)$ and that by construction $\mathsf{w}\mathsf{t}\left(p\right)\le 2^{-m}$. Thus,

$$\begin{array}{ll}\hfill {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_{2^{-m}}^{\mathsf{A},M_{\#Q}}\left(n\right)& =\underset{\begin{array}{c}\mathbf{x}\leftarrow U_d^n\\ p\leftarrow \mathsf{A}\left(M_{\#Q}\left(\mathbf{x}\right)\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\right]\hfill \\ \hfill & \ge \underset{\begin{array}{c}\mathbf{x}\leftarrow U_d^n\\ p\leftarrow \mathsf{A}\left(M_{\#Q}\left(\mathbf{x}\right)\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(q_0,\mathbf{x}\right)\right]\hfill \\ \hfill & \ge B\left(n,1/n\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right).\hfill \end{array}$$

$\square$Observe that $\mathsf{i}\mathsf{s}\mathsf{o}\left(q_0,\mathbf{x}\right)\hspace{0.17em}\Rightarrow \hspace{0.17em}\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)$ and that by construction $\mathsf{w}\mathsf{t}\left(p\right)\le 2^{-m}$. Thus,

$$\begin{array}{ll}\hfill {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_{2^{-m}}^{\mathsf{A},M_{\#Q}}\left(n\right)& =\underset{\begin{array}{c}\mathbf{x}\leftarrow U_d^n\\ p\leftarrow \mathsf{A}\left(M_{\#Q}\left(\mathbf{x}\right)\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\right]\hfill \\ \hfill & \ge \underset{\begin{array}{c}\mathbf{x}\leftarrow U_d^n\\ p\leftarrow \mathsf{A}\left(M_{\#Q}\left(\mathbf{x}\right)\right)\end{array}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(q_0,\mathbf{x}\right)\right]\hfill \\ \hfill & \ge B\left(n,1/n\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right).\hfill \end{array}$$

$\square$

Remark 5.3. When the attack succeeds, all of the predicates $q_i$ match 0 or 1 rows in $\mathbf{x}$. It may seem that an easy way to counter the attack is by masking low counts, a common measure taken, e.g., in contingency tables. However, it is easy to modify the attack to only use predicates $Q$ matching $\mathrm{\Theta }\left(n\right)$ rows using one extra query. This means that restricting the mechanism to suppress low counts cannot prevent this type of attack. Let $q^{*}$ be a predicate with ${\mathsf{w}\mathsf{t}}_{U_m}\left(q^{*}\right)=1/2$ (e.g., parity of the bits), and let $q_i^{*}=q_i\vee q^{*}$. The attack succeeds whenever $q^{*}\left(\mathbf{x}\right)=q_0^{*}\left(\mathbf{x}\right)+1$. If $q^{*}\left(x\right)$ and $q_0\left(x\right)$ are independent, then this occurs with probability at least $\frac{1}{2}\cdot B\left(n,1/n\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. As before, the probability can be amplified to $1-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$.

While a single count is PSO secure for any data distribution, the above attack against $\omega \left(\log \left(n\right)\right)$ counts applies only to the uniform distribution $U_d$. We can extend the attack to general min-entropic distributions $D$ at the cost of randomizing the attacked mechanism $M$ (i.e., the set of predicates $Q$). Furthermore, for min-entropic $D$, this probability can be amplified to $1-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ by repetition (9).

C. Failure to Compose Twice.

Borrowing ideas from refs. 17 and 18, we construct two mechanisms $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ and $M_{\mathsf{e}\mathsf{n}\mathsf{c}}$ which are individually PSO secure (for arbitrary distributions), but which together allow an adversary to single out with high probability when the data are uniformly distributed over the universe $X={\left\{\mathrm{0,1}\right\}}^d$. With more work, this composition attack can be extended to more general universes and to min-entropic distributions.

Theorem 5.4. $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ and $M_{\mathsf{e}\mathsf{n}\mathsf{c}}$ described below are PSO secure. For $m=\omega \left(\log \left(n\right)\right)$ and $m\le min\left(d,\left(n-1\right)/4\right)$, $X={\left\{\mathrm{0,1}\right\}}^d$, and $D=U_d$ the uniform distribution over $X$, there exists an adversary $\mathsf{A}$ such that

$${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_{2^{-m}}^{\mathsf{A},M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}}\left(n\right)\ge 1-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right),$$

where $M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}=\left(M_{\mathsf{e}\mathsf{x}\mathsf{t}},M_{\mathsf{e}\mathsf{n}\mathsf{c}}\right)$.

We divide the input dataset into two parts. We treat ${\mathbf{x}}_{\mathsf{e}\mathsf{x}\mathsf{t}}=\left(x_1,\dots ,x_{n-1}\right)$ as a “source of randomness” and $x_n$ as a “messsage.” $M_{\mathsf{e}\mathsf{x}\mathsf{t}}\left(\mathbf{x}\right)$ outputs an encryption secret key $\mathsf{s}$ based on the rows in ${\mathbf{x}}_{\mathsf{e}\mathsf{x}\mathsf{t}}$, using the von Neumann extractor as described in Algorithm 2.

$M_{\mathsf{e}\mathsf{n}\mathsf{c}}\left(\mathbf{x}\right)$ runs $\mathsf{s}\leftarrow M_{\mathsf{e}\mathsf{x}\mathsf{t}}$. If $\mathsf{s}\hspace{0.17em}\ne \hspace{0.17em}\perp$, it outputs $\mathsf{c}=\mathsf{s}\oplus x_n\left[1:m\right]$ (using $\mathsf{s}$ as a one-time pad to encrypt the first $m$ bits of $x_n$); otherwise, it outputs $\perp$. Alone, neither $\mathsf{s}$ nor $\mathsf{c}$ allows the adversary to single out, but using both, an adversary can recover $x_n\left[1:m\right]$ and thereby predicate single out the last row.

Proof Outline: We must prove that $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ and $M_{\mathsf{e}\mathsf{n}\mathsf{c}}$ are PSO secure and that $M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}$ is not. Note that the security of $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ and $M_{\mathsf{e}\mathsf{n}\mathsf{c}}$ do not follow merely from the fact that their outputs are nearly uniform.^§§

$M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ is $\left(\ln \left(2\right),\mathrm{0,1}/n\right)$-PSO secure.

Consider the mechanism $M_{\mathsf{e}\mathsf{x}\mathsf{t}}^{\sigma }\left(\mathbf{x}\right)$ that samples a random permutation $\sigma :\left[n\right]\to \left[n\right]$ and outputs $M_{\mathsf{e}\mathsf{x}\mathsf{t}}\left(\sigma \left(\mathbf{x}\right)\right)$. For any $\mathbf{x}$, $M_{\mathsf{e}\mathsf{x}\mathsf{t}}^{\sigma }\left(\mathbf{x}\right)$ is uniform conditioned on not outputting $\perp$. Its security is equivalent to a mechanism outputting a single bit of information, which itself is $\left(\ln \left(2\right),\mathrm{0,1}/n\right)$-PSO secure by Proposition 5.1 (and therefore also PSO-secure by the observation at the end of Security Against PSO). To complete the proof, one can show that ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M_{\mathsf{e}\mathsf{x}\mathsf{t}}}\left(n\right)={\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M_{\mathsf{e}\mathsf{x}\mathsf{t}}^{\sigma }}\left(n\right)$. $\square$

$M_{\mathsf{e}\mathsf{n}\mathsf{c}}$ is PSO-secure.

We separately consider the two possible values of $p\left(x_n\right)$, where $p$ is the predicate returned by $\mathsf{A}$. Let ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M_{\mathsf{e}\mathsf{n}\mathsf{c}}}\left(n\right)={\gamma }_0+{\gamma }_1$, where

$${\gamma }_b\triangleq Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\wedge p\left(x_n\right)=b\right].$$

The output of $M_{\mathsf{e}\mathsf{n}\mathsf{c}}\left(\mathbf{x}\right)$ is deterministic and information-theoretically independent of $x_n$. Thus, for any $w=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$,

$${\gamma }_1\le \underset{\mathbf{x},\mathsf{A}}{Pr}\left[p\left(x_n\right)=1\mid {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\right]\le w=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right).$$

If $\mathsf{A}$ singles out and $p\left(x_n\right)=0$, then it is effectively singling out against the subdataset ${\mathbf{x}}_{-n}=\left(x_1,\dots ,x_{n-1}\right)$. That is,

$${\gamma }_0=\underset{\mathbf{x},\mathsf{A}}{Pr}\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,{\mathbf{x}}_{-n}\right)\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\wedge p\left(x_n\right)=0\right].$$

We construct $\mathsf{B}$ that tries to single out against mechanism $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$ using $\mathsf{A}$. On input $\mathsf{s}$, $\mathsf{B}$ samples $x_n^{\prime }\sim D$ and runs $p\leftarrow \mathsf{A}\left(\mathsf{s}\oplus x_n^{\prime }\left[1:m\right]\right)$.

$$\begin{array}{ll}\hfill {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{B},M_{\mathsf{e}\mathsf{x}\mathsf{t}}}\left(n\right)& \ge Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,{\mathbf{x}}_{-n}\right)\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\wedge p\left(x_n^{\prime }\right)=0\right]\hfill \\ \hfill & \hspace{1em}\cdot Pr\left[p\left(x_n\right)=0\mid {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\right]\hfill \\ \hfill & \ge {\gamma }_0\cdot \left(1-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)\right).\hfill \end{array}$$

By the PSO security of $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$, ${\gamma }_0$ is negligible. $\square$

Insecurity of $M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}$ for $D=U_d$.

The output of $M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}\left(\mathbf{x}\right)$ is a pair $\left(\mathsf{s},\mathsf{c}\right)$. If $\left(\mathsf{s},\mathsf{c}\right)=\left(\perp ,\perp \right)$, $\mathsf{A}$ aborts. By a Chernoff Bound, for $m\le \left(n-1\right)/4$, $\underset{\mathbf{x}}{Pr}\left[\mathsf{s}=\perp \right]\le e^{-\left(n-1\right)/16}=\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. If $\left(\mathsf{s},\mathsf{c}\right)\hspace{0.17em}\ne \hspace{0.17em}\left(\perp ,\perp \right)$, $\mathsf{A}$ recovers $x_n=\mathsf{c}\oplus \mathsf{s}$ and outputs the predicate

$$p\left(x\right)=\left(x\left[1:m\right]=x_n\left[1:m\right]\right).$$

By the choice of $m=\omega \left(\log \left(n\right)\right)$, ${\mathsf{w}\mathsf{t}}_{U_d}\left(p\right)=2^{-m}<\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. $Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\mid \mathsf{s}\hspace{0.17em}\ne \hspace{0.17em}\perp \right]=1-Pr\left[\exists j\hspace{0.17em}\ne \hspace{0.17em}n:x_j\left[1:m\right]=x_n\left[1:m\right]\right]=1-n\cdot 2^{-m}>1-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. The bound on ${\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_{2^{-m}}^{\mathsf{A},M_{\mathsf{E}\mathsf{x}\mathsf{t}\mathsf{E}\mathsf{n}\mathsf{c}}}$ follows, completing the proof of the claim and the theorem. $\square$

D. Singling Out and Failure to Compose.

The failure to compose demonstrated in Theorem 5.2 capitalizes on the use of multiple counting queries. Such queries underlie a large variety of statistical analyses and machine learning algorithms. We expect that other attempts to formalize security against singling out would also allow counting queries. If so, our negative composition results may extend to other formalizations.

The failure to compose demonstrated in Theorem 5.4 is more contrived. We expect that other attempts to formalize security against singling out would allow mechanisms like $M_{\mathsf{e}\mathsf{x}\mathsf{t}}$—ones where for every input $\mathbf{x}$ randomly permuting the input and applying the mechanism results in the uniform distribution over outputs (as in the proof of Theorem 5.4). It is less clear to us whether other possible formalizations of security against singling out would allow a mechanism like $M_{\mathsf{e}\mathsf{n}\mathsf{c}}$. If it is to compose, it likely must not.

6. Differential Privacy Provides PSO Security

In this section, we demonstrate that differential privacy implies PSO security. Because exact counts are not differentially private but are PSO secure (Proposition 5.1), we have already shown that PSO security does not imply differential privacy.

Recall that we model $\mathbf{x}\in X^n$ as containing personal data of $n$ distinct individuals. For $\mathbf{x},{\mathbf{x}}^{\prime }\in X^n$, we write $\mathbf{x}\sim {\mathbf{x}}^{\prime }$ if $\mathbf{x}$ and ${\mathbf{x}}^{\prime }$ differ on the data of exactly one individual $x_i$.

Definition 6.1 [Differential Privacy (10, 19)] A randomized mechanism $M:X^n\to T$ is $\left(\epsilon ,\delta \right)$-differentially private if for all $\mathbf{x},{\mathbf{x}}^{\prime }\in X^n,\mathbf{x}\sim {\mathbf{x}}^{\prime }$ and for all events $S\subseteq T$,

$$Pr\left[M\left(\mathbf{x}\right)\in S\right]\le e^{\epsilon }Pr\left[M\left({\mathbf{x}}^{\prime }\right)\in S\right]+\delta ,$$

where the probability is taken over the randomness of the mechanism $M$.

Our analysis relating PSO security to differential privacy is through a connection of both concepts to statistical generalization. For differential privacy, this connection was established in refs. 13 and 14. We use a variant of the latter from ref. 20:

Lemma 6.1 (Generalization Lemma). For all distributions $D$ and for all $\left(\epsilon ,\delta \right)$-differentially private algorithms $\mathsf{A}:\mathbf{x}\mapsto p$ operating on a dataset $\mathbf{x}$ and outputting a predicate $p:X\to \left\{\mathrm{0,1}\right\}$

$${\mathbb{E}}_{\mathbf{x}\sim D^n}\left[{\mathbb{E}}_{p\leftarrow \mathsf{A}\left(\mathbf{x}\right)}\left[p\left(\mathbf{x}\right)\right]\right]\le e^{\epsilon }\cdot {\mathbb{E}}_{\mathbf{x}\sim D^n}\left[{\mathbb{E}}_{p\leftarrow \mathsf{A}\left(\mathbf{x}\right)}\left[{\mathsf{w}\mathsf{t}}_D\left(p\right)\right]\right]+\delta .$$

Theorem 6.2. For all $\epsilon =O\left(1\right)$, $\delta =\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$, and $w_{\mathsf{m}\mathsf{a}\mathsf{x}}\le 1/n$, if $M$ is $\left(\epsilon ,\delta \right)$-differentially private, then $M$ is $\left({\epsilon }^{\prime },{\delta }^{\prime },w_{\mathsf{m}\mathsf{a}\mathsf{x}}\right)$-PSO secure for

$${\epsilon }^{\prime }=\epsilon +\left(n-1\right)\ln \left(\frac{1}{1-w_{\mathsf{m}\mathsf{a}\mathsf{x}}}\right)\le \epsilon +1\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}{\delta }^{\prime }=n\delta +\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right).$$

In particular, for $w_{\mathsf{m}\mathsf{a}\mathsf{x}}=o\left(1/n\right)$, ${\epsilon }^{\prime }=\epsilon +o\left(1\right)$.

Proof: For simplicity of exposition, we present the proof for min-entropic distributions $D$. The proof for general distributions follows from a similar argument.

Given $p\leftarrow \mathsf{A}\left(M\left(\mathbf{x}\right)\right)$, $w\le w_{\mathsf{m}\mathsf{a}\mathsf{x}}$, and $D$, define the predicate $p^{*}$:

$$p^{*}\left(x\right)\equiv \left\{\begin{array}{cc}p\left(x\right)\hspace{1em}\hfill & \text{if}{\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\hfill \\ 0\hspace{1em}\hfill & \text{if}{\mathsf{w}\mathsf{t}}_D\left(p\right)>w\hfill \end{array}\right..$$

Observe that ${\mathsf{w}\mathsf{t}}_D\left(p^{*}\right)\le w$. The predicate $p^{*}$ can be computed from $p$, $D$, and $w$ without further access to $\mathbf{x}$. Because differential privacy is closed under postprocessing, if $M$ is $\left(\epsilon ,\delta \right)$-differentially private, then the computation that produces $p^{*}$ is $\left(\epsilon ,\delta \right)$-differentially private as well. Recall $\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)\hspace{0.17em}\iff \hspace{0.17em}p\left(\mathbf{x}\right)=1/n$.

$$\begin{array}{ll}\hfill {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},M}\left(n\right)& =\underset{\mathbf{x},p}{Pr}\left[p\left(\mathbf{x}\right)=1/n\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\right]\hfill \\ \hfill & \le \underset{\mathbf{x},p}{Pr}\left[p\left(\mathbf{x}\right)\ge 1/n\wedge {\mathsf{w}\mathsf{t}}_D\left(p\right)\le w\right]\hfill \\ \hfill & =\underset{\mathbf{x},p}{Pr}\left[p^{*}\left(\mathbf{x}\right)\ge 1/n\right]\hfill \\ \hfill & \le n\cdot {\mathbb{E}}_{\mathbf{x},p}\left[p^{*}\left(\mathbf{x}\right)\right]\hfill \\ \hfill & \le n\cdot \left(e^{\epsilon }w+\delta \right)\left(\text{Lemma 6.1}\right)\hfill \\ \hfill & =e^{\epsilon }\frac{B\left(n,w\right)}{{\left(1-w\right)}^{n-1}}+n\delta \hfill \\ \hfill & \le e^{\epsilon }\frac{\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)}{{\left(1-w\right)}^{n-1}}+n\delta +\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)\left(\text{Claim 4.1}\right)\hfill \\ \hfill & \le e^{{\epsilon }^{\prime }}\mathsf{b}\mathsf{a}\mathsf{s}\mathsf{e}\left(n,w\right)+{\delta }^{\prime }.\hfill \end{array}$$

$\square$

7. Does $k$-Anonymity Provide PSO Security?

$k$-anonymity (11, 12) is a strategy intended to help a data holder “release a version of its private data with scientific guarantees that the individuals who are the subjects of the data cannot be re-identified while the data remain practically useful” (12). It is achieved by making each individual in a data release indistinguishable from at least $k-1$ individuals along certain attributes. Typically, a $k$-anonymized dataset is produced by subjecting it to a sequence of generalization and suppression operations.

In this section, we analyze the extent to which $k$-anonymity provides PSO security. We show that a $k$-anonymized dataset typically provides an attacker information sufficient to PSO with constant probability. This result challenges the determination of the Article 29 Working Party.^¶¶

A. Preliminaries.

Let $\left(A_1,\dots ,A_m\right)$ be attribute domains. A dataset $\mathbf{x}=\left(x_1,\dots ,x_n\right)$ is a collection of rows $x_i=\left(a_{i,1},\dots ,a_{i,m}\right)$, where $a_{i,j}\in A_j$. For subsets ${\hat{a}}_{i,j}\subseteq A_j$, we view $y_i=\left({\hat{a}}_{i,1},\dots ,{\hat{a}}_{i,m}\right)$ as a set in the natural way, writing $x_i\in y_i$ if $\forall j\in \left[m\right]$, $a_{i,j}\in {\hat{a}}_{i,j}$. We say that a dataset $\mathbf{y}=\left(y_1,\dots ,y_n\right)$ is derived from $\mathbf{x}$ by generalization and suppression if $\forall i\in \left[n\right]$, $x_i\in y_i$. For example, if $\left(A_1,A_2,A_3\right)$ correspond to “5-Digit ZIP Code,” “Gender,” and “Year of Birth,” then it may be that $x_i=\left(91015,F,1972\right)$ and $y_i=\left(91010\text{–}91019,\star ,1970\text{–}1975\right)$, where $\star$ denotes a suppressed value.

$k$-anonymity aims to capture a sort of anonymity in a (very small) crowd: A data release $\mathbf{y}$ is $k$-anonymous if any individual row in the release is indistinguishable from $k-1$ other individual rows. Let $\mathsf{c}\mathsf{o}\mathsf{u}\mathsf{n}\mathsf{t}\left(\mathbf{y},y\right)\triangleq \left|\left\{i\in \left[n\right]:y_i=y\right\}\right|$ be the number of rows in $\mathbf{y}$ which agree with $y$.***

Definition 7.1 (k-Anonymity [Rephrased from Ref. 12]): For $k\ge 2$, a dataset $\mathbf{y}$ is $k$-anonymous if $\mathsf{c}\mathsf{o}\mathsf{u}\mathsf{n}\mathsf{t}\left(\mathbf{y},y_i\right)\ge k$ for all $i\in \left[n\right]$. An algorithm is called a $k$-anonymizer if on an input dataset $\mathbf{x}$ its output is a $k$-anonymous $\mathbf{y}$ which is derived from $\mathbf{x}$ by generalization and suppression.

For $k$-anonymous dataset $\mathbf{y}=\left(y_1,\dots ,y_n\right)$, let ${\varphi }_1\left(x\right)=\mathbb{I}\left(x\in y_1\right)$ be the predicate that returns 1 if $y_1$ could have been derived from $x$ by generalization and suppression. Let ${\mathbf{x}}_{{\varphi }_1}=\left\{x\in \mathbf{x}:{\varphi }_1\left(x\right)=1\right\}$. We assume for simplicity that $k$-anonymizers always output $\mathbf{y}$ such that $\left|{\mathbf{x}}_{{\varphi }_1}\right|=k$, but the theorem generalizes to other settings.

B. k-Anonymity Enables Predicate Singling Out.

Before presenting the main theorem of this section, we provide an example of a very simple $k$-anonymizer that fails to provide security against predicate singling out. Let $D=U_n$ be the uniform distribution over $U={\left\{\mathrm{0,1}\right\}}^n$.

Consider the $k$-anonymizer that processes groups of $k$ rows in index order and suppresses all bit locations where any of the $k$ rows disagree. Namely, for each group $g=1,\dots ,n/k$ of $k$ rows $\left(x_{gk+1},\dots ,x_{\left.gk+k\right)}\right)$, it outputs $k$ copies of the string $y_g\in {\left\{\mathrm{0,1},\star \right\}}^n$, where $y_g\left[j\right]=b\in \left\{\mathrm{0,1}\right\}$ if $x_{gk+1}\left[j\right]=\cdots =x_{gk+k}\left[j\right]=b$ (i.e., all of the $k$ rows in the group have $b$ as their $j$th bit) and $y_g\left[j\right]=\star$ otherwise.

The predicate ${\varphi }_1\left(x\right)$ evaluates to 1 if $y_1\left[j\right]\in \left\{x\left[j\right],\star \right\}$ for all $j\in \left[n\right]$ and evaluates to 0 otherwise. Namely, ${\varphi }_1\left(x\right)$ checks whether $x$ agrees with $y_1$ (and, hence, with all of $x_1,\dots ,x_{\left.k\right)}$) on all nonsuppressed bits.

In expectation, $n/2^{k-1}$ positions of $y_1$ are not suppressed. For large enough $n$, with high probability over the choice of $\mathbf{x}$, at least $\frac{n}{2\cdot 2^{k-1}}$ positions in $y_1$ are not suppressed. In this case, ${\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)\le 2^{-\frac{n}{2^k}}$ which is $\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$ for any constant $k$.

We now show how ${\varphi }_1$ can be used adversarially. In expectation, $n\left(1-2^{-\left(k-1\right)}\right)\ge n/2$ positions of $y_1$ are suppressed. For large enough $n$, with high probability over the choice of $\mathbf{x}$ at least $n/4$ of the positions in $y_1$ are suppressed. Denote these positions $i_1,\dots ,i_{n/4}$. Define the predicate $p_k\left(x\right)$ that evaluates to 1 if the binary number resulting from concatenating $x\left[i_1\right],x\left[i_2\right],\dots ,x\left[i_{n/4}\right]$ is greater than $2^{n/4}/k$ and 0 otherwise. Note that ${\mathsf{w}\mathsf{t}}_D\left(p_k\right)\approx 1/k$ and, hence, $p_k$ isolates within group 1 with probability $\approx 1/e\approx 0.37$, as was the case with the trivial adversary described after Definition 4.1.

An attacker observing ${\varphi }_1$ can now define a predicate $p\left(x\right)={\varphi }_1\left(x\right)\wedge p_k\left(x\right)$. By the analysis above, $\mathsf{w}\mathsf{t}\left(p\right)$ is negligible (as it is bounded by $\mathsf{w}\mathsf{t}\left({\varphi }_1\right)$) and $p\left(x\right)$ isolates a row in $\mathbf{x}$ with probability $\approx 0.37$. Hence, the $k$-anonymizer of this example fails to protect against singling out.

Theorem 7.1 captures the intuition from this example and generalizes it, demonstrating that $k$-anonymity does not typically protect against predicate singling out.

Theorem 7.1. For any $k\ge 2$, there exists an algorithm $\mathsf{A}$ such that for all min-entropic $D$, all $k$-anonymizers $\mathsf{A}\mathsf{n}\mathsf{o}\mathsf{n}$, and all $w\le 1/n$:

$$\begin{array}{ll}\hfill & {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},\mathsf{A}\mathsf{n}\mathsf{o}\mathsf{n}}\left(n\right)\ge \eta \cdot B\left(k,1/k\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)\approx \frac{\eta }{e},\hfill \\ \hfill & \text{where}\hspace{1em}\eta \triangleq \underset{\begin{array}{c}\mathbf{x}\leftarrow D^n\\ \mathbf{y}\leftarrow \mathsf{A}\mathsf{n}\mathsf{o}\mathsf{n}\left(\mathbf{x}\right)\end{array}}{Pr}\left[{\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)\le w\right].\hfill \end{array}$$

To predicate single out, the $\mathsf{A}$ must output a predicate that both isolates $\mathbf{x}$ and has low weight. The theorem states that these two requirements essentially decompose: $\eta$ is the probability that the predicate $k$-anonymizer induces a low-weight predicate ${\varphi }_1$, and $B\left(k,1/k\right)$ is the probability that a trivial adversary predicate singles out the subdataset ${\mathbf{x}}_{{\varphi }_1}$ of size $k$. Algorithms for $k$-anonymity generally try to preserve as much information in the dataset as possible. Thus, we expect such algorithms to typically yield low-weight predicates ${\varphi }_1$ and correspondingly high values of $\eta$.

Proof Outline: $\mathsf{A}$ will construct some predicate $q$ and output the conjunction $p\triangleq {\varphi }_1\wedge q$. Noting that ${\mathsf{w}\mathsf{t}}_D\left(p\right)\le {\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)$, and that $\mathsf{i}\mathsf{s}\mathsf{o}\left(q,{\mathbf{x}}_{{\varphi }_1}\right)\hspace{0.17em}\Rightarrow \hspace{0.17em}\mathsf{i}\mathsf{s}\mathsf{o}\left(p,\mathbf{x}\right)$,

$$\begin{array}{ll}\hfill {\mathsf{S}\mathsf{u}\mathsf{c}\mathsf{c}}_w^{\mathsf{A},\mathsf{A}\mathsf{n}\mathsf{o}\mathsf{n}}\left(n\right)& \ge Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(q,{\mathbf{x}}_{{\varphi }_1}\right)\wedge {\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)\le w\right]\hfill \\ \hfill & =\eta \cdot Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(q,{\mathbf{x}}_{{\varphi }_1}\right)\hspace{0.17em}\hspace{0.17em}{\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)\le w\right].\hfill \end{array}$$ [1]

It remains only to show that for min-entropic distributions $D$, $Pr\left[\mathsf{i}\mathsf{s}\mathsf{o}\left(q,{\mathbf{x}}_{{\varphi }_1}\right)\hspace{0.17em}\hspace{0.17em}{\mathsf{w}\mathsf{t}}_D\left({\varphi }_1\right)\le w\right]\ge B\left(k,1/k\right)-\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{l}\left(n\right)$. This claim is reminiscent of Fact 3.1, but with an additional challenge. The rows in ${\mathbf{x}}_{{\varphi }_1}$ are not distributed according to $D$; instead, they are a function of $\mathsf{A}\mathsf{n}\mathsf{o}\mathsf{n}$ and the whole dataset $\mathbf{x}$. They are not independently distributed, and even their marginal distributions may be different from $D$. Nevertheless, for the purposes of the baseline, the rows in ${\mathbf{x}}_{{\varphi }_1}$ have enough conditional min-entropy to behave like random rows. $\square$

Data Availability

This paper does not include original data.