ON THE PRIVACY AND UTILITY PROPERTIES OF TRIPLE MATRIX-MASKING

Abstract

Privacy protection is an important requirement in many statistical studies. A recently proposed data collection method, triple matrix-masking, retains exact summary statistics without exposing the raw data at any point in the process. In this paper, we provide theoretical formulation and proofs showing that a modified version of the procedure is strong collection obfuscating: no party in the data collection process is able to gain knowledge of the individual level data, even with some partially masked data information in addition to the publicly published data. This provides a theoretical foundation for the usage of such a procedure to collect masked data that allows exact statistical inference for linear models, while preserving a well-defined notion of privacy protection for each individual participant in the study. This paper fits into a line of work tackling the problem of how to create useful synthetic data without having a trustworthy data aggregator. We achieve this by splitting the trust between two parties, the “masking service provider” and the “data collector.”

1. Introduction

In the digital age, vast amount of data becomes available for research. At the same time, there is increasing pressure to protect the privacy of study subjects when their data is used. For medical research, the Health Insurance Portability and Accountability Act of 1996 and subsequent rulings have imposed legal requirements for privacy protection on the collection and handling of health data. Among other things, basic privacy protection measures include the removal of all personal identifiers when releasing data for use. However, simply removing the personal identifier variables does not prevent possible identification of the individual from other variables. To prevent the identification of an individual record, researchers have shown that released data should be aggregated to satisfy privacy conditions such as k-anonymity [Sweeney, 2002], l-diversity [Machanavajjhala et al., 2007] and t-closeness [Li et al., 2007].

However, releasing data only at aggregated levels severely restricts its usefulness in many research studies. Alternatively, methods are designed to release obfuscated micro-data that allows for the usual statistical analysis while preserving the privacy at individual levels. Some examples of such obfuscated micro-data publishing are: noise addition [Brand, 2002], multiple imputation[Rubin, 1993, Drechsler and Reiter, 2010], information preserving statistical obfuscation [Burridge, 2003], random projection based perturbation [Liu et al., 2006], random orthogonal matrix masking [Ting et al., 2008]. Particularly, in the random orthogonal matrix masking scheme, a masked data set AX is published, where X denotes the data matrix of real responses and A is a random orthogonal matrix. The published data AX keeps the exact values for sufficient statistics of linear models, thus allowing exact statistical inference for many standard data analysis methods [Ting et al., 2008, Wu et al., 2017b] while protecting privacy by denying the user’s direct access to the raw data X. While the above methods all protect the privacy of individual entries through publishing only the random perturbed micro-data, the privacy protection can be lost when multiple micro-data sets are combined from multiple inquires to the same database. Differential privacy is proposed to quantify the effectiveness of privacy protection of the random noise addition/perturbation schemes [Dwork et al., 2006, Dwork, 2006, Dwork and Naor, 2008] against multiple inquires to the database. Then the noise level can be adjusted to achieve a quantified tradeoff between inference efficacy and privacy preservation (measured by the differential privacy metric).

Traditionally, there is a trustworthy data collector/manager that collects raw data and ensure privacy protection by releasing the data sets with random perturbations. Such procedures however do not protect against attacks where an unscrupulous party has unauthorized access to the raw data set X kept by these centers. Such security breaks are becoming more common as shown by the recent well-publicized incidences involving hacking against databases at major retailers, banks and credit bureaus [Huffington Post, 2011, Reuters., 2015, 2017].

This paper fits into a line of work tackling the problem of how to create useful synthetic data without having a trustworthy data aggregator, and provides a theoretical study of the triple matrix-masking (TM²) procedure [Wu et al., 2017b] that does not assume such a trustworthy data collector/manager. The TM² procedure is a multi-party collection and masking system that aims to collect and publish the random orthogonal masked data set AX. We prove that, assuming no collusion between parties, no party learns more than the orbit of the data matrix under the action of the orthogonal group. More specifically, given the view of a particular party, let $\mathcal{S}$ be the set of data matrices that could possibly have resulted in that view. We show that $\mathcal{S}$ contains the full orbit of the data matrix and that given any prior on the data matrix, the party’s posterior is simply their prior restricted to $\mathcal{S}$. We call data collection procedures with such properties as strong obfuscating since any extra information beyond AX available to a party does not help in further identifying the individual level data.

In the differential privacy literature, the issue of untrustworthy data collector can be dealt with using local differential privacy procedures [Kasiviswanathan et al., 2011], where noises are added to the individual data before passing to the data collector. The resulting synthetic data from differential privacy procedures, however, does not preserve exact statistics hence require special inference procedures designed to achieve optimal statistical inference Duchi et al. [2017]. Our TM² procedure provides an alternative where the published masked data exactly preserve any statistics of the data that are preserved under the action of the orthogonal group. This provides an useful utility that exact statistical inference for linear models are preserved, thus standard linear statistical inference procedures can be applied directly on the resulting synthetic data from the TM² procedure. On the other hand, the TM² procedure is only for a one-shot collection of each individual’s data. When the individual data providers are sampled in multiple independent collections by different data collectors, differential privacy procedures can measure and limit the privacy leakage for the composition of the multiple collections. The TM² procedure does not consider the privacy leakage for the composition of the multiple collections.

Section 2 describe the TM² procedure and two new modifications to make it strong obfuscating. The theoretical analysis is provided in Section 3. Section 4 provides a summary and more detailed discussions for the relationship of the TM² procedure to the differential privacy and multi party computation methods.

2. The Masked Data Collection Procedure TM²nd Its Modification

The privacy-preserving data collection scheme TM² was proposed first in Wu et al. [2017b] and later expanded by Wu et al. [2017a]. We describe our modified basic version of the TM² method here:

Step 1. The data collectors plan the data collection, create the database structure, program the data collection system. They randomly generate a p × p random orthogonal matrix B, which is distributed to the participants’ data collection devices.

Step 2. Each participant’s data x₁ (a vector of dimension p₁) is collected and merged with Gaussian noise x₂ (of dimension p₂) into a vector x = (x₁, x₂) of dimension p = p₁ + p₂. Then x is right multiplied by B on the participant’s device, and only the resulting masked data xB leaves the device and is sent to the masking service provider.

Step 3. The masking service provider generates another n × n random orthogonal matrix A₂. After receiving data from all participants, it combines the individual data xB into a n × p matrix XB, left multiplies by A₂ and sends the doubly masked data A₂XB to the data collectors.

Step 4. The data collectors multiply A₂XB by B⁻¹ to get back A₂X and take the first p₁ columns to get A₂X₁. Then the data collectors generate another n × n random orthogonal matrix A₁, left multiply it to A₂X₁, and publish AX₁ (where A = A₁A₂) which is accessible by all data users.

Detailed theoretical analysis of the privacy guarantee on the TM² method has been missing. This paper fills that gap by proving theoretically that this modified version of the TM² method is strong obfuscating. We prove the strong obfuscating guarantee by showing that: (A) the extra information that any party in the process owns will not allow the party to reduce the data domain (possible values of data) small enough to identify individual level data; and (B) there is no statistical information leakage beyond the domain restriction considered in (A).

Compared to the original TM² scheme in Wu et al. [2017b], we make two modifications on the TM² procedure. For the first modification, we add random Gaussian noise in Step 2. The data collector wants to collect p₁ variables on n individuals so that the real response matrix becomes X₁ of dimensions n × p₁. We ask each participant to generate p₂ pure Gaussian noise variables, on his/her device according to a fixed variance parameter σ². Hence, the full data matrix would be X = (X₁, X₂). For privacy protections proved in later sections, we require that p₁ < n ≤ p = p₁ + p₂. In Step 2 of this modified procedure, Gaussian noise x₂ is mingled with real response x₁ to provide protection in addition to the random mask B. In Step 4, after the collectors get back A₂X = (A₂X₁, A₂X₂), they separate the matrix and discard those noises. Therefore the published data set AX₁ with A = A₁A₂ still gives the exact summary statistic, as it is only masked by A without containing the added noise.

For the second modification, instead of using a random invertible matrix for the right mask B as originally proposed by Wu et al. [2017b], we use a random orthogonal matrix for the right mask B. As we will see in the privacy analysis in the next section, using an invertible matrix does make one part of the privacy proof easier. However, the other part of privacy proof depends on using a uniformly distributed random matrix to avoid information leakage that can lead to probabilistic attacks. While there is a natural uniform distribution on all orthogonal matrices that is well studied in literature, there is no natural uniform distribution on the set of all invertible matrices. The uniformly distributed orthogonal matrix B does provide sufficient privacy protection when combined with the addition of noise X₂.

2.1. Privacy Analysis of TM²etup.

To rigorously study the privacy protection issues in this data collection process, we analyze the information that can be accessed by each party and analyze whether such information allows inference of the individual level data.

First, we illustrate how to analyze the privacy protection assuming that the adversary only has access to the publicly published left-masked data set M_L = AX₁ where A is a random n × n orthogonal matrix. The issue becomes whether an adversary can identify individual level data knowing only M_L = m_L.

We consider the analysis in two stages. When given M_L = m_L, this restricts the possible values of X₁ and can thus reveal information. In the first stage, we consider whether this support restriction on X₁ (due to M_L = m_L) enables the identification of individual data. Let ${\mathcal{S}}_{{\mathit{X}}_1}$ denote the support of X₁, and let ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L\right)$ denote the restricted support of X₁ given that M_L = m_L. The privacy preservation depends on the size of ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L\right)$. For example, in an extreme case, if the ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L\right)$ contains only one matrix, then X₁ is known to everyone and data privacy cannot be protected. Generally we can show, in next section, that this restricted support ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L\right)$ is big enough so that identification of individual data is impossible.

In the second stage, we consider whether the adversary can learn any information beyond the restriction on support which was analyzed in the first stage. Such information can enable adversaries to launch probabilistic attacks [Machanavajjhala et al., 2007, Fung et al., 2010]. Fortunately, due to the independence between the mask A and the raw data X₁, we can show that the posterior density of X₁ given M_L = m_L is the same as the prior density of X₁ restricted to the support ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L\right)$. Thus any loss of privacy is through the support restriction already studied in stage one. Therefore, knowing M_L = m_L does not identify individual level data.

Next, we consider the privacy protection for all parties involved in the whole TM² data collection process. That is, we conduct the above two-stage privacy protection analysis given all information available to one party in the process. The data collector and the masking service provider each have access to some intermediate masked data in addition to the public data. Hence, we need to analyze privacy protection for an adversary knowing this intermediate masked data together with the public data set M_L = AX₁.

The data collector knows, in addition to M_L = AX₁, the double masked data A₂XB. Since the data collector knows the masks A₁ and B, knowing these data A₁A₂X₁ and A₂XB are simply equivalent to knowing A₂X. Due to the fact that X₂ is purely noise independent of raw data X₁, the theoretical privacy analysis for the data collector knowing A₂X = (A₂X₁, A₂X₂) will have basically the same results as the analysis for the user with access only to M_L = AX₁.

The masking service provider has access to the right-masked data M_R = XB in addition to the public left-masked data M_L = AX₁. This information results in the most severe restriction on the support when compared to what resulted from knowledge by other parties. Thus, this is the weakest link for privacy preservation in the whole TM² data collection scheme. In Section 3, we present details of the two-stage privacy protection analysis when both M_L and M_R are known.

3. Theoretical Analysis of Privacy Preservation of TM²

3.1. Notations, Formalizations and Technical Preliminaries.

We denote the probability densities of random matrices X₁, X₂, A and B as ${\pi }_{{\mathit{X}}_1}\left(x_1\right)$, ${\pi }_{{\mathit{X}}_2}\left(x_2\right)$, π_A(a) and π_B(b) respectively. The supports of these distributions are denoted respectively as ${\mathcal{S}}_{{\mathit{X}}_1}$, ${\mathcal{S}}_{{\mathit{X}}_2}$, ${\mathcal{S}}_{\mathit{A}}$ and ${\mathcal{S}}_{\mathit{B}}$.

We want to study, based on information INFO available to one party, what this party can infer about the individual level data. Here this INFO includes the publicly available final left-masked data M_L = AX₁ and some extra information available to the particular party. The restricted support of X₁ given INFO is denoted as ${\mathcal{S}}_{{\mathit{X}}_1}\left(INFO\right)$ which consists of all n × p matrices that can be the value of X₁ which is compatible with INFO.

For example, given only the public masked data INFO = M_L, the restricted support is

$${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L\right)=\left\{U:\exists \tilde{\mathit{A}}\in {\mathcal{S}}_{\mathit{A}}\text{such}\text{that}\tilde{\mathit{A}}U={\mathit{M}}_L.\right\}$$

Let ${\mathcal{O}}_n$ denote the set consisting of all orthogonal matrices. In the case of left masking with a random orthogonal matrix A, for any orthogonal $\overline{\mathit{A}}\in {\mathcal{O}}_n$, $U=\overline{\mathit{A}}{\mathit{X}}_1$ is compatible with INFO = M_L. That is, ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L\right)={\mathcal{O}}_n{\mathit{X}}_1$. To see this, let $\tilde{\mathit{A}}=\mathit{A}{\overline{\mathit{A}}}^T$, then $\tilde{A}\in {\mathcal{O}}_n$ and $\tilde{\mathit{A}}\left(\overline{\mathit{A}}{\mathit{X}}_1\right)={\mathit{M}}_L$. Here and throughout this paper, we use ^T to denote the transpose of a matrix.

For the strong obfuscating guarantee, we wish to show that the extra information available to the parties in the process does not cause any privacy loss more than the publicly released final left-masked data M_L = AX₁. We want to show that: stage one (i) the restricted support ${\mathcal{S}}_{{\mathit{X}}_1}\left(INFO\right)$ is the same as ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L\right)={\mathcal{O}}_n{\mathit{X}}_1$; stage two (ii) the conditional probability distribution of X₁ given INFO is similar to the probability distribution of X₁ given support ${\mathcal{S}}_{{\mathit{X}}_1}\left(INFO\right)$, thus there is no privacy loss through probability attacks beyond the loss from the support restriction considered in stage one.

We now formalize the precise mathematical statements to prove in stages one and two. More precisely for stage one, we hope that the restricted support is the same as if only the public left-masked data is available.

$${\mathcal{S}}_{{\mathit{X}}_1}(INFO)={\mathcal{O}}_n{\mathit{X}}_1,$$ (i)

for INFO available to any one party in the process. For the second stage, we denote ${\pi }_{{\mathit{X}}_1\mid INFO}\left(x_1\mid INFO\right)$ as the posterior distribution of X₁ given INFO. The prior density ${\pi }_{X_1}$ restricted on the support ${\mathcal{S}}_{X_1}\left(INFO\right)$ is

$${\pi }_{{\mathit{X}}_1\mid \mathcal{S}}{{}_{{}_{{\mathit{X}}_1}}}_{(INFO)}\left({\mathit{x}}_1\right)=\frac{{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)}{\underset{{\mathcal{S}}_{{\mathit{X}}_1}(INFO)}{\int }{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1^{*}\right)d{\mathit{x}}_1^{*}}.$$

To show that there is no extra privacy loss beyond the support restriction considered in stage one, we prove that these two probability densities agree with each other. That is, we wish to prove

$${\pi }_{{\mathit{X}}_1\mid INFO}\left({\mathit{x}}_1\mid INFO\right)={\pi }_{{\mathit{X}}_1\mid \mathcal{S}}{{}_{{}_{{\mathit{X}}_1}}}_{(INFO)}\left({\mathit{x}}_1\right).$$ (ii)

Definition 3.1.

A data collection process is strong collection obfuscating if conditions (i) and (ii) hold for the information INFO available to any party in this process.

A slightly weaker version is that the above property holds with a high probability. Notice that the INFO available to any party in this process can be determined from the values of X₁, X₂, A and B which are generated respectively from distributions with densities ${\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)$, ${\pi }_{{\mathit{X}}_2}\left({\mathit{x}}_2\right)$, π_A(a) and π_B(b). Thus such INFO is generated from a probability distribution defined by ${\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)$, ${\pi }_{{\mathit{X}}_2}\left({\mathit{x}}_2\right)$, π_A(a) and π_B(b). We want that, with high probability from this probability distribution, the generated values of INFO satisfy conditions (i) and (ii).

Definition 3.2.

A data collection process is ϵ-strong collection obfuscating if, with probability at least 1 − ϵ, conditions (i) and (ii) hold for the information INFO available to any party in this process.

Our definition of the strong collection obfuscating procedure ensures that there is no privacy loss due to observations by any party in the process beyond those contained in the publicly released final data. This definition delineates the privacy protection in the collection process from the privacy protection in the publicly releasing of final data M_L = AX₁. The theoretical analysis concentrates on the soundness of the collection process.

Given the public left-masked data M_L = AX₁, the statistics ${\mathit{X}}_1^T{\mathit{X}}_1$ are released to the user. The user has the first two exact statistical moments and statistical models, such as linear regression, can be fitted exactly as if the user has the raw data set X₁. And the residuals are known up to an orthogonal matrix multiplication, therefore the usual statistical model diagnostics methods can also be carried out as if done on the raw data set.

For continuous data, the user cannot recover the individual level data since the user only sees a linear combination of all individuals’ data, and there is no utilizable statistical distributional information other than the prior (population) density ${\pi }_{{\mathit{X}}_1}$. This ensures the privacy of individual data.

In practice, the types of elements in X₁ may also be known to the user. This can further restrict the support. We assume that the elements in data matrix X₁ are all encoded as numerical values (e.g., “yes/no” answer to a question may be encoded as 1 and 0). We consider that the type of data in each column, either as continuous/discrete/binary, is public knowledge. Let ${\mathcal{S}}_j$ denote the support of the type of data in the j-th column of X₁. For example, if data are continuous, then ${\mathcal{S}}_j=\mathcal{R}$; if the data are binary, then ${\mathcal{S}}_j=\{0,1\}$; if the data are positive integers, then ${\mathcal{S}}_j=\{1,2,\dots \}={\mathcal{N}}^{+}$. Knowing the type of data in each column would restrict the support of X₁ to

$${\tilde{\mathcal{S}}}_{{\mathit{X}}_1}=\{\mathit{U}:\text{(all}\text{entries}\text{of}j\text{-th}\text{column}\text{in}\mathit{U})\in {\mathcal{S}}_{j}j=1,\dots ,p_1.\}.$$

Then with knowledge of both INFO and types of data, the restricted support becomes the intersection of ${\tilde{\mathcal{S}}}_{{\mathit{X}}_1}$ and ${\mathcal{S}}_{{\mathit{X}}_1}\left(INFO\right)$,

$${\mathcal{S}}_{{\mathit{X}}_1}(INFO;TYPE)={\tilde{\mathcal{S}}}_{{\mathit{X}}_1}\cap {\mathcal{S}}_{{\mathit{X}}_1}(INFO).$$

Let ${\mathcal{P}}_n$ denote the set of all permutation matrix P. Since all permutation matrices are orthogonal and permutation does not change the type of elements, we have the following Lemma:

Lemma 3.3.

For any strong obfuscating data collection process,

$${\mathcal{P}}_n{\mathit{X}}_1\subseteq \left[{\tilde{\mathcal{S}}}_{{\mathit{X}}_1}\cap {\mathcal{O}}_n{\mathit{X}}_1\right]={\mathcal{S}}_{{\mathit{X}}_1}(INFO;TYPE).$$

Lemma 3.3 indicates that a strong collection obfuscating data collection process offers some privacy protection even when the data types are known. Since all permutations are in the ${\mathcal{S}}_{{\mathit{X}}_1}(INFO;TYPE)$, any individual cannot be identified here without extra side information. It is not clear whether the type can be combined with some side information (such as data that a particular individual is a smoker) to reveal other individual level data. However, notice that any weakness in this aspect is inherently due to releasing the public data AX₁. Our strong collection obfuscating procedure ensures that no extra privacy loss is added during the process beyond the privacy loss in releasing AX₁.

As we discussed in the previous section, the party with the most information during the TM² process is the masking service provider who knows INFO = (M_L, M_R). Therefore, in the next section, we study when (i) and (ii) hold for INFO = (M_L, M_R). Here we first state some technical preliminary results on the characterization of the uniform distribution on orthogonal matrices. These preliminary results are used in studying the second stage condition (ii) later.

Under the matrix multiplication, the orthogonal matrices form a compact Hausdorff topological group ${\mathcal{O}}_n$. Therefore, there is a unique Haar measure μ(·) on ${\mathcal{O}}_n$ such that the measure of the whole sample space ${\mathcal{O}}_n$ equals one. Then this Haar measure induces a natural uniform distribution on ${\mathcal{O}}_n$. See Chapter 2 of Zhang [2014] for a detailed technical equivalent characterization of the uniform distribution on ${\mathcal{O}}_n$. Since a Haar measure μ(·) is invariant under the matrix multiplication, the uniform distribution is also invariant under the matrix multiplication.

Lemma 3.4.

Let π₀(·) denote the probability density function of the uniform distribution on ${\mathcal{O}}_n$. Then for any orthogonal matrix $A_0\in {\mathcal{O}}_n$,

$${\pi }_0(\mathit{a})={\pi }_0\left({\mathit{A}}_0\mathit{a}\right)={\pi }_0\left(\mathit{a}{\mathit{A}}_0\right),for\mathit{a}ll\mathit{a}\in {\mathcal{O}}_n.$$ (3.1)

Also, the product of two uniformly distributed orthogonal matrices is also uniformly distributed.

Lemma 3.5.

If A₁ ~ π₀ and A₂ ~ π₀ are independent of each other, then their product A = A₁A₂ also follows the uniform distribution π₀ on ${\mathcal{O}}_n$.

The proof is straightforward and can be found in Chapter 2 of Zhang [2014].

In the TM² scheme, when the masking service provider and the data collector each generate a random orthogonal matrix A₁ and A₂ respectively according to π₀, then the mask A = A₁A₂ for the publicly released data set is also uniformly distributed. In practice, the uniformly distributed random orthogonal matrices can be generated using algorithms described in Heiberger [1978], Anderson et al. [1987], Wu et al. [2017b].

3.2. Restricted Support Given Knowledge of Masked Data Sets.

We first prove that condition (i) holds for INFO = (M_L, M_R) when invertible matrices are used for right mask B as originally proposed by Wu et al. [2017b]. That is, $\mathit{B}\in {\mathcal{I}}_n$, where ${\mathcal{I}}_n$ denote the set of all n × n invertible matrices. Condition (i) then becomes that all orthogonal transformations of X₁ are contained in the restricted support

$$\begin{gathered} {\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)=\{\mathit{U}:\exists \tilde{\mathit{A}}\in {\mathcal{S}}_{\mathit{A}},\tilde{\mathit{B}}\in {\mathcal{S}}_{\mathit{B}}\text{and}\tilde{\mathit{U}}\in {\mathcal{S}}_{{\mathit{X}}_2}\text{such}\text{that} \\ \tilde{\mathit{A}}\mathit{U}={\mathit{M}}_L\text{and}(\mathit{U},\tilde{U})\tilde{\mathit{B}}={\mathit{M}}_R\}. \end{gathered}$$ (3.2)

Theorem 3.6.

Suppose ${\mathcal{S}}_{\mathit{A}}={\mathcal{O}}_n$ and ${\mathcal{S}}_{\mathit{B}}={\mathcal{I}}_p$, p₁ ≤ n ≤ p and X is full rank (i.e., rank(X) = n), then for any $\mathit{P}\in {\mathcal{O}}_n$, $P{\mathit{X}}_1\in {\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)$. In other words, ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)={\mathcal{O}}_n{\mathit{X}}_1={\mathcal{O}}_n{\mathit{M}}_L$.

Proof.

We need to show that, for any $\mathit{P}\in {\mathcal{O}}_n$, $\mathit{U}=\mathit{P}{\mathit{X}}_1\in {\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)$.

Since $\mathit{P}\in {\mathcal{O}}_n$ and $\mathit{A}\in {\mathcal{O}}_n$, then $\tilde{\mathit{A}}=\mathit{A}{\mathit{P}}^T\in {\mathcal{O}}_n$. Then we have

$$\tilde{\mathit{A}}\mathit{U}=\mathit{A}{\mathit{P}}^T\mathit{P}{\mathit{X}}_1=\mathit{A}{\mathit{X}}_1={\mathit{M}}_L.$$ (3.3)

Since X is full-rank and n ≤ p, there exists a (p − n) × p matrix X∗ such that $\left(\begin{array}{c}\mathit{X}\\ \mathit{X}\ast \end{array}\right)$ is full-rank and thus invertible. Since $\mathit{P}\in {\mathcal{O}}_n$, $\left(\begin{array}{c}{\mathit{P}}^T\mathit{X}\\ \mathit{X}\ast \end{array}\right)$ is also full-rank and invertible. Hence we can define an invertible matrix

$$\tilde{\mathit{B}}={\left(\begin{array}{c}\mathit{X}\\ \mathit{X}\ast \end{array}\right)}^{-1}\left(\begin{array}{c}{\mathit{P}}^T\mathit{X}\\ \mathit{X}\ast \end{array}\right)\mathit{B}.$$

Also let $\tilde{\mathit{U}}=\mathit{P}{\mathit{X}}_2$ thus $(\mathit{U},\tilde{\mathit{U}})=\mathit{P}\mathit{X}$, we have

$$\left(\begin{array}{c}(\mathit{U},\tilde{\mathit{U}})\\ \mathit{X}\ast \end{array}\right)\tilde{\mathit{B}}=\left(\begin{array}{cc}\mathit{P}& 0\\ 0& \mathit{I}\end{array}\right)\left(\begin{array}{c}\mathit{X}\\ \mathit{X}\ast \end{array}\right)\tilde{\mathit{B}}=\left(\begin{array}{cc}\mathit{P}& 0\\ 0& \mathit{I}\end{array}\right)\left(\begin{array}{c}{\mathit{P}}^T\mathit{X}\\ \mathit{X}\ast \end{array}\right)\mathit{B}=\left(\begin{array}{c}\mathit{X}\\ \mathit{X}\ast \end{array}\right)\mathit{B},$$

where I is the identity matrix of size (p − n) × (p − n). The first p rows in the last equation are

$$(\mathit{U},\tilde{\mathit{U}})\tilde{\mathit{B}}=\mathit{X}\mathit{B}={\mathit{M}}_R.$$ (3.4)

(3.3) and (3.4) together means that U satisfies (3.2), thus U belongs to ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)$. □

Theorem 3.6 states that condition (i) is satisfied when the X is full rank. In the original TM² scheme proposal, the full rank condition may or may not be satisfied because it is determined by the underlying probability distribution of X₁ which is outside the control of the designer of this procedure. With the modification of extra noise matrix X₂, we can ensure the full rank condition by specifying the noise generation mechanism. Particularly, we specify that each individual data provider generates a p₂-dimension noise vector with i.i.d. elements from a Gaussian distribution with p₂ ≥ n. This will ensure with probability one that X is indeed full rank.

Remark 1. (Size of the right mask)

For privacy preservation, the size of right mask p has to be bigger than the data size n as assumed in Theorem 3.6. When p < n, some rows of M_R are linear dependent which provides further restriction on the support. We provide such a counter example in Appendix A to illustrate that such a restriction together with knowledge of data type can reveal individual level data.

Above we considered the support restriction under the original TM² scheme proposal [Wu et al., 2017b] of invertible right mask, ${\mathcal{S}}_{\mathit{B}}={\mathcal{I}}_p$. However, unlike ${\mathcal{O}}_p$, ${\mathcal{I}}_p$ does not form a compact Hausdorff topological group. Therefore, there exists no uniform distribution on ${\mathcal{I}}_p$. Due to the non-uniformity of B, the posterior distribution of X₁ given (M_L, M_R) leaks information beyond the support restriction, thus the second stage condition (ii) no longer holds. This makes the usage of random invertible right masks in the TM² scheme very tricky. It is unclear what distribution on ${\mathcal{I}}_p$ should be used to generate the random invertible B.

Here, we consider the modification of the TM² scheme where the right mask B is a random orthogonal matrix generated from the uniform distribution π₀ on ${\mathcal{O}}_p$. We show that if the random noise X₂ is large enough, then condition (i) still holds when the orthogonal right mask B is used.

Let λ_min(M) and λ_max(M) denotes the minimum and the maximum eigenvalues of a semi-positive definite matrix M. The restricted support will remain big if the noise is large enough:

$${\lambda }_{\mathit{m}in}\left({\mathit{M}}_R{\mathit{M}}_R^T-{\mathit{X}}_1{\mathit{X}}_1^T\right)={\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)>{\lambda }_{\mathit{m}ax}\left({\mathit{X}}_1{\mathit{X}}_1^T\right).$$ (3.5)

Now we have a result similar to Theorem 3.6.

Theorem 3.7.

Suppose ${\mathcal{S}}_{\mathit{A}}={\mathcal{O}}_n$ and ${\mathcal{S}}_{\mathit{B}}={\mathcal{O}}_p$, p₁ ≤ n ≤ p. If condition (3.5) holds, then ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)={\mathcal{O}}_n{\mathit{X}}_1$.

The proof is provided in Appendix B.

Next, we show that condition (ii) also holds when under condition (3.5). Then we discuss how achievable the technical condition (3.5) is in practice.

3.3. Information Leakage Beyond the Support Restriction.

We now study the second stage condition (ii) by checking the amount of information an adversary can get from the posterior distribution of X₁ given (M_L, M_R) beyond their restriction on the support of X₁. Given INFO = (M_L, M_R), the posterior density is denoted as ${\pi }_{{\mathit{X}}_1\mid \left({\mathit{M}}_L,{\mathit{M}}_R\right)}\left(x_1\mid {\mathit{m}}_L,{\mathit{m}}_R\right)$. The prior density ${\pi }_{{\mathit{X}}_1}$ restricted on the support ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{m}}_R\right)$ is denoted as ${\pi }_{{\mathit{X}}_1\mid \mathcal{S}}{}_{{}_{{\mathit{X}}_1}}\left({\mathit{m}}_L,{\mathit{m}}_R\right)$.

Theorem 3.8.

Let X₁ be a random matrix with probability density ${\pi }_{{\mathit{X}}_1}$. We assume that the elements in X₂ are generated i.i.d. from a Gaussian distribution with mean zero. When condition (3.5) holds, given M_L and M_R, the posterior density of X₁ is the same as the prior density restricted on ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{M}}_L,{\mathit{M}}_R\right)$. That is,

$${\pi }_{{\mathit{X}}_1\mid \left({\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1\mid {\mathit{m}}_L,{\mathit{m}}_R\right)={\pi }_{{\mathit{X}}_1\mid \mathcal{S}}{}_{{}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{m}}_R\right)}\left({\mathit{x}}_1\right).$$ (3.6)

The proof of Theorem 3.8 is provided in Appendix D.

3.4. ϵ-strong obfuscating TM².

Theorem 3.7 and Theorem 3.8 states, respectively, that conditions (i) and (ii) hold under condition (3.5). Combining them, we have the following Theorem.

Theorem 3.9.

If

$$Pr\left[{\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)>{\lambda }_{\mathit{m}ax}\left({\mathit{X}}_1{\mathit{X}}_1^T\right)\right]\ge 1-ϵ,$$ (3.7)

then the proposed TM² procedure is ϵ-strong collection obfuscating by Definition 3.2.

The ϵ-strong collection privacy property ensures that there is at most ϵ probability for the process to leak any privacy information beyond the public released data AX₁. TM² achieves this property when the technical condition (3.7) holds. To achieve the technical condition (3.7), we generate the p₂-dimensional noise vector x₂ with i.i.d. Gaussian elements of mean zero and a sufficiently large variance σ². We present a technical probability bound in Appendix C, where the probability of violating condition (3.5) is decreasing exponentially and specifics a σ² value which ensures condition (3.7). Larger variance σ² always increases the probability that condition (3.5) holds. In practice, the variance σ² is only limited due to the computation accuracy. That is, σ should not exceed raw data values by the orders of magnitude allowed by the machine precision.

3.5. Extension to Alleviate Collusion Risks.

We have shown that the privacy of individual data can be protected when no party in the TM² scheme knows all the masks. However, there are also risks of collusion among different parties in the procedure. Since the right mask B is known to the data collector and all individual data providers, if one of them share this info with the masking services provider, then the privacy protection can be broken.

Wu et al. [2017a] proposed ideas to protect against this collusion risk using the ideas of multiparty computation. For each individual, the data vector x can be broken up as K₁ random components $x^1,\dots ,x^{K_1}$ where $x=x^1+\dots +x^{K_1}$. Then such components are sent to K₁ right masking service providers, one to each. The resulting masked data xⁱB_i, i = 1, ..., K₁, are then sent to the left masking service provider to be merged and created the double masked data AXⁱB_i, i = 1, ..., K₁. For further protection, they can be passed through K₂ left masking service providers to generate ${\mathit{A}}_{K_2}\dots {\mathit{A}}_1{\mathit{X}}^i{\mathit{B}}_i$. Let $\mathit{A}={\mathit{A}}_{K_2}{\mathit{A}}_{K_2-1}\dots {\mathit{A}}_1$. Then the masked data AXⁱB_i, i = 1, ..., K₁, are sent to the corresponding right masking service providers to remove the right masking. Then the resulting AXⁱ, i = 1, ..., K₁, are sent to the data collector to generate $\mathit{A}\mathit{X}=\mathit{A}{\mathit{X}}^1+\dots +\mathit{A}{\mathit{X}}^{K_1}$. Unless all K₁ right (or all K₂ left) masking service providers collude, they cannot find values of all components ${\mathit{X}}^1,\dots ,{\mathit{X}}^{K_1}$.

The stage one theoretical analysis on this extended TM² scheme can be analyzed similarly as before, where the restricted support condition (i*) holds given condition (3.5). The stage two analysis is more involved, as the posterior distribution of X₁ given some shares, depends on the distribution of the shares. Which random distributions should the shares be generated from to effect no additional privacy loss remains an open question, and will be investigated in future work.

4. Discussions and Conclusions

This paper conducts a theoretical analysis of privacy preservation in a modified TM² scheme. Random noises were used with uniformly distributed orthogonal matrix masks to hide individual data during the data collection process. The noise addition in the first step of the TM² scheme is similar to the idea of noise perturbed response schemes. However, the critical difference is that our noise addition is only intended to help mask data during the transition, and is in fact removed after the right mask removal. The resulting published data set is a left masked data set with exact summary statistics, unlike many other noise addition schemes where the summary statistics are randomly approximated.

This work is aimed to protect against unscrupulous access to the raw data X₁ traditionally hold by a trusted operator. We would like to further clarify the relationship to differential privacy methods [Dwork et al., 2006] which aims to provide a strong privacy protection and closure under composition of multiple accesses to the database. There are two types of differential privacy models. In the central model, a trusted database operator holds the raw data, and releases noise perturbed summary statistics for inquires. In the local model [Evfimievski et al., 2003, Dwork et al., 2006, Kasiviswanathan et al., 2011, Cormode et al., 2018], noise is added at the individual level based on the idea of randomized response methods [Warner, 1965, Blair et al., 2015]. The local differential privacy procedures similarly addresses the issue of untrustworthy central database operator. In recent years, Goolge [Erlingsson et al., 2014], Apple [Thakurta et al., 2017] and Microsoft [Ding et al., 2017] have all developed and deployed local differential privacy procedures in data collection.

There are two type of possible unscrupulous access to the raw data X₁ to be addressed. The first is that the data collector is untrustworthy. The second is that an unscrupulous party might break in to the server containing data collected by an honest data collector. In the differential privacy literature, the first type is handled by using local differential privacy procedures, while the second type is addressed via pan-private data analysis [Dwork et al., 2010]. Our TM² scheme protects against both type of unscrupulous accesses, but only allow for a one-shot collection for each individual’s data.

While both the local differential privacy procedures and the TM² scheme can provide protection against unscrupulous accesses, the goals are somewhat different. The TM² scheme aims to collect a masked data set that preserves the first two statistical moments of the variables (note that ${\mathit{X}}_1^T{\mathit{X}}_1$ is knowable from the publicly available AX₁). This allows exact statistical inferences on quantities depending on these statistical moments. The local differential privacy methods, on the other hand, aims to provide a stronger privacy protection under composition of multiple data collections/accesses.

The idea of the TM² scheme is similar to secure multi-party computation (SMC) procedures, in that this scheme tries to distribute information among parties so that each party does not get access to individual level data other than its own. There are also important differences between TM² and SMC. They differ in their designed purposes even though both want parties to cooperate in a joint task while keeping privacy. SMC is designed to conduct joint statistical analysis without the parties revealing their data to each other. TM² wants to collect the masked data set, which enables statistical analysis, without parties revealing the actual data to the data collector. Operationally, SMC requires distributed storage of data as well as distributed computation. Specifically, if we require that the private data of parties never leave their devices, then SMC needs the parties to stand by ready for any statistical analysis that may occur much later in the future. In contrast, the TM² method is only distributive in the data collection stage. The private data leaves the parties’ devices in a masked form, and later is centrally stored in masked form AX₁. Since all future statistical analysis is conducted on the publicly released AX₁, there is no need for the parties to be available for future analysis.

In this paper, we presented a privacy analysis clearly separating the risks coming from support restriction and the risks of probabilistic attacks beyond the support restriction. With the analysis, we show that the TM² scheme is safe to collect a synthetic data set AX₁ which is a random orthogonal transformation of the raw data set X₁. All information during the data collection procedure is masked, and no one during the procedure can access the raw data set. This removes the issue of trusting a data record keeper and provides a new tool for researchers to collect data allowing exact statistical inference for linear models while provide a privacy protection: no hacking attack against a party in the data collection procedure can access real individual level data since all parties do not have enough information to infer the private individual data.

Figure 1:

This diagram shows each party’s knowledge about the data and the masking matrices in the modified TM² method. Each party knows some masked version of the data: XB for the masking service provider, A₂X for the data collector, and A₁A₂X₁ for everybody including the public. Nobody knows the original data X₁, with each data provider (participant) knowing only his/her row x₁

Appendix A. A Counter Example

We illustrate that p ≥ n and the full rank condition on X are needed for the privacy preservation in the TM² scheme through a simple counter example here. We consider a 3 × 2 matrix X, where the first column X₁ contains binary sensitive information and the second column X₂ contains continuous random noise. Suppose that only one of the three individuals answered “1” on the sensitive question, so that the data matrix is

$$\mathit{X}=\left(\begin{array}{ll}{x}_{11}\hfill & x_{12}\hfill \\ x_{21}\hfill & x_{22}\hfill \\ x_{31}\hfill & x_{32}\hfill \end{array}\right)=\left(\begin{array}{ll}1\hfill & a_1\hfill \\ 0\hfill & a_2\hfill \\ 0\hfill & a_3\hfill \end{array}\right).$$ (A.1)

We decompose $\mathit{X}=\left(\begin{array}{l}{\mathit{X}}_a\hfill \\ {\mathit{X}}_b\hfill \end{array}\right)$ with the first two rows as X_a and the last row as X_b. Without loss of generality, we assume that a₂ ≠ 0 so that $\left(\begin{array}{ll}1\hfill & a_1\hfill \\ 0\hfill & a_2\hfill \end{array}\right)$ is non-singular, and we assume that a₃/a₂ is not an integer. Then the first column X₁ = (1, 0, 0)^T can be uniquely determined from the masked data M_R.

To see this, we decompose ${\mathit{M}}_R=\left(\begin{array}{l}{\mathit{M}}_a\hfill \\ {\mathit{M}}_b\hfill \end{array}\right)$ similarly as in the decomposition of $\mathit{X}=\left(\begin{array}{l}{\mathit{X}}_a\hfill \\ {\mathit{X}}_b\hfill \end{array}\right)$. Then ${\mathit{M}}_b{\mathit{M}}_a^{-1}=\left({\mathit{X}}_b\mathit{B}\right){\left({\mathit{X}}_a\mathit{B}\right)}^{-1}={\mathit{X}}_b\mathit{B}{\mathit{B}}^{-1}{\mathit{X}}_a^{-1}={\mathit{X}}_b{\mathit{X}}_a^{-1}$ is known to anyone with access to M_R. Using (A.1), this means ${\mathit{X}}_b{\mathit{X}}_a^{-1}=\left(0,a_3/a_2\right)$ is determined from the masked data M_R. Then the first element in $\left(X_b{\mathit{X}}_a^{-1}\right){\mathit{X}}_a={\mathit{X}}_b$ indicates that (0, a₃/a₂)(x₁₁, x₂₁)^T = x₃₁. That is, (a₃/a₂)x₂₁ = x₃₁.

Since x₂₁ and x₃₁ are binary entries in X₁ and a₃/a₂ is not an integer, the attacker can infer from (a₃/a₂)x₂₁ = x₃₁ that x₂₁ = x₃₁ = 0. Then we must have x₁₁ = 1 due to M_R (and thus X) being full-rank. That is, we now know every entry in X₁ = (1, 0, 0)^T from the masked data M_R.

Notice that according to Lemma 3.3, a strong collection obfuscating procedure would not have allowed this identification of individual data from the random permutation. There is indeed additional privacy loss without assuming p < n. In general, when p < n and M_R is full rank, $\left({\mathit{M}}_b{\mathit{M}}_a^{-1}\right){\mathit{X}}_a={\mathit{X}}_b$ along with knowledge of the data type may leak sensitive information about original data X.

Appendix B. Proof of Theorem 3.7.

Proof.

Same arguments in proof of Theorem 3.6 shows that (3.3) holds. Therefore we only need to show that there exist ${\tilde{\mathit{X}}}_2$ and $\tilde{\mathit{B}}$ satisfying (3.4): $\left(\mathit{P}{\mathit{X}}_1,\tilde{\mathit{U}}\right)\tilde{\mathit{B}}=\mathit{X}\mathit{B}={\mathit{M}}_R$.

Using condition (3.5), we have

$$\begin{gathered} {\lambda }_{\mathit{m}in}\left({\mathit{X}}_1{\mathit{X}}_1^T+{\mathit{X}}_2{\mathit{X}}_2^T-\mathit{P}{\mathit{X}}_1{\mathit{X}}_1^T{\mathit{P}}^T\right)\ge {\lambda }_{\mathit{m}in}\left({\mathit{X}}_1{\mathit{X}}_1^T\right)+{\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)-{\lambda }_{\mathit{m}ax}\left(\mathit{P}{\mathit{X}}_1{\mathit{X}}_1^T{\mathit{P}}^T\right) \\ \ge {\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)-{\lambda }_{\mathit{m}ax}\left({\mathit{X}}_1{\mathit{X}}_1^T\right) \\ \text{>0}. \end{gathered}$$

Hence ${\mathit{X}}_1{\mathit{X}}_1^T+{\mathit{X}}_2{\mathit{X}}_2^T-\mathit{P}{\mathit{X}}_1{\mathit{X}}_1^T{\mathit{P}}^T$ is a positive definite matrix. Therefore, there exists a matrix $\tilde{\mathit{U}}$ such that

$$\tilde{\mathit{U}}{\tilde{\mathit{U}}}^T={\mathit{X}}_1{\mathit{X}}_1^T+{\mathit{X}}_2{\mathit{X}}_2^T-\mathit{P}{\mathit{X}}_1{\mathit{X}}_1^T{\mathit{P}}^T.$$ (B.1)

This is equivalent to

$$\left(\mathit{P}{\mathit{X}}_1,\tilde{\mathit{U}}\right){\left(\mathit{P}{\mathit{X}}_1,\tilde{\mathit{U}}\right)}^T=\tilde{\mathit{U}}{\tilde{\mathit{U}}}^T+\mathit{P}{\mathit{X}}_1{\mathit{X}}_1^T{\mathit{P}}^T={\mathit{X}}_1{\mathit{X}}_1^T+{\mathit{X}}_2{\mathit{X}}_2^T=\mathit{X}{\mathit{X}}^T={\mathit{M}}_R{\mathit{M}}_R^T.$$ (B.2)

Now we apply a singular value decomposition on M_R = SDV where $\mathit{S}\in {\mathcal{O}}_n$, $\mathit{V}\in {\mathcal{O}}_p$ and D is a diagonal matrix with nonincreasing nonnegative diagonal elements. Then, due to (B.2), the singular decomposition of $\left(\mathit{P}{\mathit{X}}_1,\tilde{\mathit{U}}\right)$ is $\mathit{S}\mathit{D}\tilde{\mathit{V}}$ for a $\tilde{\mathit{V}}\in {\mathcal{O}}_p$. Therefore $\tilde{\mathit{B}}={\tilde{\mathit{V}}}^T\mathit{V}$ is the orthogonal matrix satisfies (3.4). □

Appendix C. Bound on condition (3.7).

To achieve the condition (3.7) we can specify the noise distribution to have large noise values. Let x_max denote the largest possible absolute value of entries in X₁. Then

$${\lambda }_{\mathit{m}ax}\left({\mathit{X}}_1{\mathit{X}}_1^T\right)={\Vert {\mathit{X}}_1\Vert }_2^2\le {\Vert {\mathit{X}}_1\Vert }_F^2=\sum _{i=1}^n\sum _{j=1}^{p_1}{x}_{1,ij}^2\le n{p}_1{x}_{\mathit{m}ax}^2=C_n,$$

where ‖ · ‖₂ and ‖ · ‖_F are the operator norm and Frobenius norm respectively. Note that $C_n=n{p}_1{x}_{\mathit{m}ax}^2$ is a known constant to the designer of the TM² scheme. Condition (3.5) holds when ${\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)$ exceeds this constant.

We require each data provider to generate a p₂-dimensional random noise vector with i.i.d. Gaussian elements of mean zero and variance σ². Assume that γ = p₂/n > 1, when n → ∞, Corollary 13 in Ledoux et al. [2010] provides a probability bound on the ${\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)$ for any δ > 0:

$$Pr\left[{\lambda }_{\mathit{m}in}\left({\mathit{X}}_2{\mathit{X}}_2^T\right)\le {(\sqrt{\gamma }-1)}^{2}n{\sigma }^2(1-\delta )\right]\le C_0{e}^{-n{\delta }^{3/2}/C_0},$$

for some constant C₀. The bound on the right side decrease exponentially in n so that, for large n, it can be made smaller than ϵ for a δ < 1. Choosing ${\sigma }^2>C_n/\left[{(\sqrt{\gamma }-1)}^{2}n(1-\delta )\right]$ will ensure that (3.7) holds.

Appendix D. Proof of Theorem 3.8.

Proof.

We study the posterior density

$${\pi }_{{\mathit{X}}_1\mid \left({\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1\mid {\mathit{m}}_L,{\mathit{m}}_R\right)=\frac{{\pi }_{\left({\mathit{X}}_1,{\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1,{\mathit{m}}_L,{\mathit{m}}_R\right)}{\underset{{{}_{\mathcal{S}}}_{{}_{{\mathit{X}}_1}}\left({\mathit{m}}_L,{\mathit{m}}_R\right)}{\int }{\pi }_{\left({\mathit{X}}_1,{\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1^{*},{\mathit{m}}_L,{\mathit{m}}_R\right)d{\mathit{x}}_1^{*}},$$ (D.1)

and compare it with the prior density ${\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)$ restricted on the support ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{M}}_R\right)$.

Recall that the probability densities for X₁, X₂, A and B at values X₁ = x₁, X₂ = x₂, A = a and B = b are denoted respectively as ${\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)$, ${\pi }_{{\mathit{X}}_2}\left({\mathit{x}}_2\right)$, π_A(a) and π_B(b). Due to the independence of the generation mechanism of these quantities, their joint density is

$${\pi }_{\left({\mathit{X}}_1,{\mathit{X}}_2,\mathit{A},\mathit{B}\right)}\left({\mathit{x}}_1,{\mathit{x}}_2,\mathit{a},\mathit{b}\right)={\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right){\pi }_{{\mathit{X}}_2}\left({\mathit{x}}_2\right){\pi }_{\mathit{A}}(\mathit{a}){\pi }_{\mathit{B}}(\mathit{b}),$$ (D.2)

for $\left({\mathit{X}}_1,{\mathit{X}}_2,\mathit{A},\mathit{B}\right)\in {\mathcal{S}}_{{\mathit{X}}_1}\times {\mathcal{S}}_{{\mathit{X}}_2}\times {\mathcal{S}}_{\mathit{A}}\times {\mathcal{S}}_{\mathit{B}}$.

Since the elements of X₂ are i.i.d. from the Gaussian distribution N(0, σ²),

$${\pi }_{{\mathit{X}}_2}\left({\mathit{x}}_2\right)=\frac{1}{{(\sqrt{2\pi }\sigma )}^{n{p}_2}}{e}^{-\frac{{\sum }_{1\le i\le n,1\le j\le p_2}{x}_{2,ij}^2}{2{\sigma }^2}}=f\left({\Vert {\mathit{x}}_2\Vert }_F^2\right),$$

where $f(x)=\frac{1}{{(\sqrt{2\pi }\sigma )}^{n{p}_2}}{e}^{-\frac{x}{2{\sigma }^2}}$ and ${\Vert {\mathit{x}}_2\Vert }_F^2={\sum }_{1\le i\le n,1\le j\le p_2}{x}_{2,ij}^2$ with ‖ · ‖_F denote the Frobenius norm. Thus the joint density becomes

$${\pi }_{\left({\mathit{X}}_1,{\mathit{X}}_2,\mathit{A},\mathit{B}\right)}\left({\mathit{x}}_1,{\mathit{x}}_2,\mathit{a},\mathit{b}\right)={\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)f\left({\Vert {\mathit{x}}_2\Vert }_F^2\right){\pi }_{\mathit{A}}(\mathit{a}){\pi }_{\mathit{B}}(\mathit{b}).$$ (D.3)

We now plug (D.3) into (D.1) for calculation.

First, we calculate the numerator ${\pi }_{\left({\mathit{X}}_1,{\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1,{\mathit{m}}_L,{\mathit{m}}_R\right)$ in (D.1). We denote the restricted sample spaces of random variables A and B respectively given knowledge of some other quantities as:

$$\begin{array}{l}{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)=\left\{\mathit{a}:\mathit{a}{\mathit{x}}_1={\mathit{m}}_L\right\},\\ {\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)=\left\{\mathit{b}:\exists {\mathit{x}}_2\text{such}\text{that}\left({\mathit{x}}_1,{\mathit{x}}_2\right)\mathit{b}={\mathit{m}}_R\right\}.\end{array}$$ (D.4)

Notice that given A = a and M_L = m_L, then X₁ = a^Tm_L. Also, given B = b and M_R = m_R, then (X₁, X₂) = m_Rb^T so that

$${\Vert {\mathit{X}}_2\Vert }_F^2=tr\mathit{a}ce\left({\mathit{X}}_2{\mathit{X}}_2^T\right)=tr\mathit{a}ce\left[{\mathit{m}}_R{\mathit{b}}^T\mathit{b}{\mathit{m}}_R^T-{\mathit{X}}_1{\mathit{X}}_1^T\right]=tr\mathit{a}ce\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-tr\mathit{a}ce\left({\mathit{X}}_1{\mathit{X}}_1^T\right).$$

Hence given A = a, M_L = m_L and B = b, we have

$${\Vert {\mathit{X}}_2\Vert }_F^2=tr\mathit{a}ce\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-tr\mathit{a}ce\left({\mathit{a}}^T{\mathit{m}}_L{\mathit{m}}_L^{T}a\right)=trace\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-trace\left({\mathit{m}}_L{\mathit{m}}_L^T\right).$$

Then using this and equation (D.3), we have

$$\begin{gathered} {\pi }_{\left({\mathit{X}}_1,{\mathit{M}}_L,{\mathit{M}}_B\right)}\left({\mathit{x}}_1,{\mathit{m}}_L,{\mathit{m}}_R\right) \\ =\underset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)}{\int }\{\underset{{\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)f\left[trace\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-trace\left({\mathit{m}}_L{\mathit{m}}_L^T\right)\right]{\pi }_{\mathit{A}}(\mathit{a}){\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}\}d\mathit{a} \\ ={\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)f\left[trace\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-trace\left({\mathit{m}}_L{\mathit{m}}_L^T\right)\right]\underset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)}{\int }\{\underset{{\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{A}}(\mathit{a}){\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}\}d\mathit{a} \\ ={\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)f\left[trace\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-trace\left({\mathit{m}}_L{\mathit{m}}_L^T\right)\right]\underset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)}{\int }[\underset{{\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}]{\pi }_{\mathit{A}}(\mathit{a})d\mathit{a} \\ ={\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)f\left[trace\left({\mathit{m}}_R{\mathit{m}}_R^T\right)-trace\left({\mathit{m}}_L{\mathit{m}}_L^T\right)\right][\underset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)}{\overset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right){\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }}{\pi }_{\mathit{A}}(\mathit{a})d\mathit{a}][\underset{{\mathcal{S}}_{\mathit{B}}\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}]. \end{gathered}$$ (D.5)

Now for any pair of x₁ and ${\mathit{x}}_1^{*}$ that both belongs to ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{m}}_R\right)$, there exist (a, a^∗) such that $\mathit{a}{\mathit{x}}_1={\mathit{m}}_L=\mathit{a}\ast {\mathit{x}}_1^{*}$. Denote A₀ = (a)⁻¹a∗. Then $A_0{\mathit{x}}_1^{*}={(a)}^{-1}{\mathit{m}}_L={\mathit{x}}_1$ and $A_0^{-1}{\mathit{x}}_1={\mathit{x}}_1^{*}$. Hence for any $\tilde{a}\in {\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)$ we have

$$\tilde{\mathit{a}}{A}_0{\mathit{x}}_1^{*}=\tilde{\mathit{a}}{\mathit{x}}_1={\mathit{m}}_L,$$

i.e., $\tilde{\mathit{a}}{\mathit{A}}_0\in {\mathcal{S}}_{\mathit{A}}\left({\mathit{x}}_1^{*},{\mathit{m}}_L\right)$. On the other hand, for any $\overline{\mathit{a}}\in {\mathcal{S}}_{\mathit{A}}\left({\mathit{x}}_1^{*},{\mathit{m}}_L\right)$, $\overline{\mathit{a}}{\mathit{A}}_0^{-1}{\mathit{x}}_1=\overline{\mathit{a}}{\mathit{x}}_1^{*}={\mathit{m}}_L$, i.e., $\overline{\mathit{a}}{\mathit{A}}_0^{-1}\in {\mathcal{S}}_{\mathit{A}}\left({\mathit{x}}_1,{\mathit{m}}_L\right)$. Taken together, we have a one-to-one mapping between the two sets ${\mathcal{S}}_{\mathit{A}}\left({\mathit{x}}_1,{\mathit{m}}_L\right)$ and ${\mathcal{S}}_{\mathit{A}}\left({\mathit{x}}_1^{*},{\mathit{m}}_L\right)$. Particularly,

$${\mathcal{S}}_A\left({\mathit{x}}_1^{*},{\mathit{m}}_L\right)={\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right){\mathit{A}}_0.$$ (D.6)

Hence for uniform density π_xA = π₀, (D.6) and Lemma 3.4 implies that

$$\underset{{\mathcal{S}}_A\left({\mathit{x}}_1,{\mathit{m}}_L\right)}{\int }{\pi }_{\mathit{A}}(\mathit{a})d\mathit{a}=\underset{{\mathcal{S}}_A\left({\mathit{x}}_1^{*},{\mathit{m}}_L\right)}{\int }{\pi }_{\mathit{A}}(\mathit{a})d\mathit{a}.$$ (D.7)

Plug (D.5) and (D.7) into (D.1) and cancel the common factors, we get

$$\begin{array}{l}{\pi }_{{\mathit{X}}_1\mid \left({\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1\mid {\mathit{m}}_L,{\mathit{m}}_R\right)\\ =\frac{{\pi }_{{\mathit{X}}_1}({\mathit{x}}_1)[\underset{{\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}]}{\underset{{\mathcal{S}}_{{\mathit{X}}_1}({\mathit{m}}_L,{\mathit{m}}_R)}{\int }{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1^{*}\right)[\underset{{\mathcal{S}}_B\left({\mathit{x}}_1^{*},{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}]d{\mathit{x}}_1^{*}},\end{array}$$ (D.8)

Next, for any pair of x₁ and ${\mathit{x}}_1^{*}$ that both belongs to ${\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{m}}_R\right)$, there exist (x₂, b) and $\left({\mathit{x}}_2^{*},\mathit{b}*\right)$ such that $\left({\mathit{x}}_1,{\mathit{x}}_2\right)\mathit{b}={\mathit{m}}_R=\left({\mathit{x}}_1^{*},{\mathit{x}}_2^{*}\right)\mathit{b}*$. Let B₀ = b(b∗)⁻¹. Then $\left({\mathit{x}}_1,{\mathit{x}}_2\right)B_0=\left({\mathit{x}}_1^{*},{\mathit{x}}_2^{*}\right)$. Similar to (D.6), we have

$${\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)={\mathit{B}}_0{\mathcal{S}}_B\left({\mathit{x}}_1^{*},{\mathit{m}}_R\right).$$ (D.9)

For uniform density π_B = π₀, using (D.9), Lemma 3.4 implies that

$$\underset{{\mathcal{S}}_B\left({\mathit{x}}_1,{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}=\underset{{\mathcal{S}}_B\left({\mathit{x}}_1^{*},{\mathit{m}}_R\right)}{\int }{\pi }_{\mathit{B}}(\mathit{b})d\mathit{b}$$ (D.10)

Plug-in (D.10) into equation (D.8), we have

$${\pi }_{{\mathit{x}}_1\mid \left({\mathit{M}}_L,{\mathit{M}}_R\right)}\left({\mathit{x}}_1\mid {\mathit{m}}_L,{\mathit{m}}_R\right)=\frac{{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1\right)}{\underset{{\mathcal{S}}_{{\mathit{X}}_1}\left({\mathit{m}}_L,{\mathit{m}}_R\right)}{\int }{\pi }_{{\mathit{X}}_1}\left({\mathit{x}}_1^{*}\right)d{\mathit{x}}_1^{*}}.$$

□