On the Downstream Performance of Compressed Word Embeddings

Abstract

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging—existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to 2× lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.

1. Introduction

In recent years, word embeddings [23, 29, 24, 30, 10] have brought large improvements to a wide range of applications in natural language processing (NLP) [1, 5, 38]. However, these word embeddings can occupy a large amount of memory, making it expensive to deploy them in data centers, and impractical to use them in memory-constrained environments like smartphones. To reduce and amortize these costs, embeddings can be compressed [e.g., 34] and shared across many downstream tasks [7]. Recently, there have been numerous successful methods proposed for compressing embeddings; these methods take a variety of approaches, ranging from dictionary learning using neural networks [34, 6] to simpler compression using k-means clustering [2].

The goal of this work is to gain a deeper understanding of what makes compressed embeddings perform well on downstream tasks. Practically, this understanding could allow for evaluating the quality of a compressed embedding without having to train a model for each task of interest. Our work is motivated by two surprising empirical observations: First, we find that existing ways [41, 3, 42] of measuring the quality of compressed embeddings do not effectively explain the relative downstream performance of different compressed embeddings—for example, failing to discriminate between embeddings that perform well and those that do not. Second, we observe that a simple uniform quantization method can match or outperform the state-of-the-art deep compositional code learning method [34] and the k-means compression method [2] in terms of downstream performance. These observations suggest that there is currently an incomplete understanding of what makes a compressed embedding perform well on downstream tasks. One way to narrow this gap in our understanding is to find a measure of compression quality that (i) is directly related to generalization performance, and (ii) can be used to analyze the performance of uniformly quantized embeddings.

Here we introduce the eigenspace overlap score as a new measure of compression quality, and show that it satisfies the above two desired properties. This score measures the degree of overlap between the subspaces spanned by the eigenvectors of the Gram matrices of the compressed and uncompressed embedding matrices. Our theoretical contributions are two-fold, addressing the surprising observations and desired properties discussed above: First, we prove a generalization bound for the compressed embeddings in terms of the eigenspace overlap score in the context of linear and logistic regression, revealing a direct connection between this score and downstream performance. Second, we prove that in expectation uniformly quantized embeddings attain a high eigenspace overlap score with the uncompressed embeddings at relatively high compression rates, helping to explain their strong performance. Inspired by these theoretical connections between the eigenspace overlap score and generalization performance, we propose using this score as a selection criterion for efficiently picking among a set of compressed embeddings, without having to train a model for each task of interest using each embedding.

We empirically validate our theoretical contributions and the efficacy of our proposed selection criterion by showing three main experimental results: First, we show the eigenspace overlap score is more predictive of downstream performance than existing measures of compression quality [41, 3, 42]. Second, we show uniform quantization consistently matches or outperforms all the compression methods to which we compare [2, 34, 16], in terms of both the eigenspace overlap score and downstream performance. Third, we show the eigenspace overlap score is a more accurate criterion for choosing between compressed embeddings than existing measures; specifically, we show that when choosing between embeddings drawn from a representative set we compressed [2, 34, 12, 16], the eigenspace overlap score is able to identify the one that attains better downstream performance with up to 2× lower selection error rates than the next best measure of compression quality. We consider several baseline measures of compression quality: the Pairwise Inner Product (PIP) loss [41], and two spectral measures of approximation error between the embedding Gram matrices [3, 42]. Our results are consistent across a range of NLP tasks [33, 19, 38], embedding types [29, 24, 10], and compression methods [2, 34, 12].

The rest of this paper is organized as follows. In Section 2 we review background on word embedding compression methods and existing measures of compression quality, and present the two surprising empirical observations that motivate our work. In Section 3 we present the eigenspace overlap score along with our corresponding theoretical contributions, and propose to use the eigenspace overlap score as a selection criterion. In Section 4, we show the results from our extensive experiments validating the practical significance of our theoretical contributions, and the efficacy of our proposed selection criterion. We present related work in Section 5, and conclude in Section 6.

2. Background and Motivation

We first review different compression methods in Section 2.1 and existing ways to measure the quality of a compressed embedding relative to the uncompressed embedding in Section 2.2. We then show in Section 2.3 that existing measures of compression quality do not satisfactorily explain the relative downstream performance of existing compression methods; this motivates our work to better understand the downstream performance of compressed embeddings.

2.1. Embedding Compression Methods

We now discuss a number of compression methods for word embeddings. For the purposes of this paper, the goal of an embedding compression method C(·) is to take as input an uncompressed embedding $X\in {ℝ}^{n\times d},$ and produce as output a compressed embedding $\tilde{X}:=C(X)\in {ℝ}^{n\times k}$ which uses less memory than X, but attains similar performance to X when used in downstream models. Here, n denotes the vocabulary size, d and k the uncompressed and compressed dimensions.

Deep Compositional Code Learning (DCCL)

The DCCL method [34] uses a dictionary learning approach to represent a large number of word vectors using a much smaller number of basis vectors. These basis vectors are organized into multiple dictionaries, and each word is represented as a sum which includes one basis vector from each dictionary. The dictionaries are trained using an autoencoder-style architecture to minimize the embedding matrix reconstruction error. A similar approach was independently proposed by Chen et al. [6].

K-means Compression

The k-means algorithm can be used to compress word embeddings by first clustering all the scalar entries in the word embedding matrix, and then replacing each scalar with the closest centroid [2]. Using 2^b centroids allows for storing each matrix entry using only b bits.

Uniform Quantization

To compress real numbers, uniform quantization divides an interval into sub-intervals of equal size, and then deterministically or stochastically rounds the numbers in each sub-interval to one of the boundaries [12, 14]. To apply uniform quantization to embedding compression, we propose to first determine the optimal threshold at which to clip the extreme values in the word embedding matrix, and then uniformly quantize the clipped embeddings within the clipped interval. For more details about uniform quantization and how we use it to compress embeddings, see Appendices A.1 and D.3 respectively.

Dimensionality Reduction

Another simple baseline for compressing embeddings is dimensionality reduction. Specifically, one can train an embedding with a lower dimension, or use a method like principal component analysis (PCA) to reduce the dimension of an existing embedding.

2.2. Measures of Compression Quality

We review ways of measuring the compression quality of a compressed embedding relative to the uncompressed embedding. For our purposes, an ideal measure of compression quality would consider the compressed and uncompressed embeddings to be similar when they are likely to perform similarly on downstream tasks, and different when this is unlikely. Such a measure would shed light on what determines the downstream performance of a compressed embedding, and give us a way of measuring the quality of a compressed embedding without having to train a downstream model for each task.

Several of the measures discussed below are based on comparing the pairwise inner product (Gram) matrices of the compressed and uncompressed embeddings. The Gram matrices of embeddings are natural to consider for two reasons: First, the loss function for training word embeddings typically only considers dot-products between embedding vectors [23, 29]. Second, one can view word embedding training as implicit matrix factorization [21], and thus comparing the Gram matrices of two embedding matrices is similar to comparing the matrices these embeddings are implicitly factoring. We now review the existing ways of measuring compression quality.

Word Embedding Reconstruction Error

The first and simplest way of comparing two embeddings X and $\tilde{X}$ is to measure the reconstruction error $\Vert X-\tilde{X}{\Vert }_F.$ Note that in order to be able to use this measure of quality, $X\text{and}\tilde{X}$ must have the same dimension.

Pairwise Inner Product (PIP) Loss

Given $X{X}^T\text{and}\tilde{X}{\tilde{X}}^T,$ the Gram matrices of the uncompressed and compressed embeddings, the Pairwise Inner Product (PIP Loss) [41] is defined as ${\Vert X{X}^T-\tilde{X}{\tilde{X}}^T\Vert }_F.$ This measure of quality was recently proposed to explain the existence of an optimal dimension for word embeddings, in terms of a bias-variance trade-off for the PIP loss.

Spectral Approximation Error

A symmetric matrix A is defined [42] to be a (Δ₁, Δ₂)-spectral approximation of another symmetric matrix B if it satisfies $\left(1-{\mathrm{\Delta }}_1\right)B\underset{-}{\prec }A\underset{-}{\prec }\left(1+{\mathrm{\Delta }}_2\right)B$ (in the semidefinite order). Zhang et al. [42] show that if $\tilde{X}{\tilde{X}}^T+\text{λ}I\text{is a}\left({\mathrm{\Delta }}_1,{\mathrm{\Delta }}_2\right)$-spectral approximation of $X{X}^T+\text{λ}I$ for small values of Δ₁ and Δ₂, then the linear model trained using $\tilde{X}$ and regularization parameter λ will attain similar generalization performance to the model trained using X. Avron et al. [3] use a single scalar Δ in place of Δ₁ and Δ₂, and use this scalar as a measure of approximation error, while Zhang et al. [42] consider Δ₁ and Δ₂ independently, and use the quantity ${\mathrm{\Delta }}_{\max }:=\max (\frac{1}{1-{\mathrm{\Delta }}_1},{\mathrm{\Delta }}_2).$

2.3. Motivation: Two Surprising Empirical Observations

We now present two surprising empirical observations which illustrate the need to better understand the downstream performance of models trained using compressed embeddings. In these experiments we compare the downstream performance of the methods introduced in Section 2.1, and attempt to use the measures of compression quality from Section 2.2 to explain the relative performance of these compression methods. Our observations reveal that explaining the downstream performance of compressed embeddings is challenging. We now provide an overview of these two observations; for a more thorough presentation of these results, please see Section 4.

• First, we observe that the downstream performance of embeddings compressed using the various methods from Section 2.1 cannot be satisfactorily explained in terms of any of the existing measures of compression quality described in Section 2.2. For example, in Figure 1 (left) we see that on GloVe embeddings [29], the uniform quantization method with compression rate 32× can have over 1.3× higher PIP loss than dimensionality reduction with compression rate 6×, while attaining better downstream performance by over 2.5 F1 points on the Stanford Question Answering Dataset (SQuAD) [33]. Furthermore, the PIP loss and the two spectral measures of approximation error Δ and Δ_max only achieve Spearman correlations (absolute value) of 0.49, 0.46, and 0.62 with the question answering test F1 score, respectively (Table 1). These results show that existing measures of compression quality correlate poorly with downstream performance.

• Our second observation is that the simple uniform quantization method matches or outperforms the more complex DCCL and k-means compression methods across a number of tasks, embedding types, and compression ratios. For example, we see in Figure 1 (right) that with a compression ratio of 32×, uniform quantization attains an average F1 score 0.47% absolute below the uncompressed GloVe embeddings on the Stanford Question Answering Dataset [33], while the DCCL method [34] is 0.43% below.

Figure 1: Two motivating empirical observations.

Left: Existing measures of compression quality do not satisfactorily explain the relative downstream performance of different compression methods. For example, compressed embeddings with higher PIP loss can perform better on question answering than those with lower PIP loss. Right: Across different compression rates, a simple uniform quantization method can compete with more complex methods such as DCCL and k-means.

Table 1: Spearman correlation between measures of compression quality and downstream performance.

For each measure of compression quality, we show the absolute value of its Spearman correlation with downstream performance, on the SQuAD (question answering), SST-1 (sentiment analysis), MNLI (natural language inference), and QQP (question pair matching) tasks. We see that the eigenspace overlap score $\mathcal{E}$ attains stronger correlation than the other measures.

Dataset	SQuAD		SST-1		MNLI	QQP
Embedding	GloVe	fastText	GloVe	fastText	BERT WordPiece	BERT WordPiece
PIP loss	0.49	0.34	0.46	0.25	0.45	0.45
Δ	0.46	0.31	0.33	0.29	0.44	0.36
Δmax	0.62	0.72	0.51	0.60	0.86	0.86
1 − $\mathcal{E}$	0.81	0.91	0.75	0.73	0.92	0.93

These two observations suggest the need to better understand the downstream performance of compressed embeddings. Toward this end, we focus on finding a measure of compression quality with the properties that (i) we can directly relate it to generalization performance, and (ii) we can use it to analyze the performance of uniformly quantized embeddings.

3. A New Measure of Compression Quality

To better understand what properties of compressed embeddings determine their downstream performance, and to help explain the surprising empirical observations above, we introduce the eigenspace overlap score, and show that it satisfies the two desired properties described above. In Section 3.1, we present generalization bounds for compressed embeddings in the context of linear and logistic regression, in terms of the eigenspace overlap score between the compressed and uncompressed embeddings. In Section 3.2 we show that uniformly quantized embeddings in expectation attain high eigenspace overlap scores, helping to explain their strong downstream performance. Based on the connection between the eigenspace overlap score and downstream performance, in Section 3.3 we propose using this score as a way of efficiently selecting among different compressed embeddings.

3.1. The Eigenspace Overlap Score and Generalization Performance

We begin by defining the eigenspace overlap score, which measures how well a compressed embedding approximates an uncompressed embedding. We then present our theoretical result relating the generalization performance of compressed embeddings to their eigenspace overlap scores.

3.1.1. The Eigenspace Overlap Score

We now define the eigenspace overlap score, and discuss the intuition behind this definition.

Definition 1.

Given two embedding matrices $X\in {ℝ}^{n\times d},\tilde{X}\in {ℝ}^{n\times k},$ whose Gram matrices have eigendecom-positions $X{X}^T=U\mathrm{\Lambda }{U}^T,\tilde{X}{\tilde{X}}^T=\tilde{U}\tilde{\mathrm{\Lambda }}{\tilde{U}}^{T}forU\in {ℝ}^{n\times d},\tilde{U}\in {ℝ}^{n\times k},$ we define the eigenspace overlap score $\mathcal{E}(X,\tilde{X}):=\frac{1}{\max (d,k)}{\Vert U^T\tilde{U}\Vert }_F^2.$

This score measures the degree to which the span of the eigenvectors with nonzero eigenvalue of $\tilde{X}{\tilde{X}}^T$ agrees with that of $X{X}^T.$ In particular, assuming $k\le d,$ it measures the ratio between the squared Frobenius norm of U before and after being projected onto $\tilde{U}.$ Computing this score takes time O(n max(d, k)²), as it requires computing the singular value decompositions (SVDs) of $X\text{and}\tilde{X}.$ As is clear from the definition, the eigenspace overlap score only depends on the left singular vectors of the two embedding matrices. To better understand why this is a desirable property, consider two embedding matrices $X\text{and}\tilde{X}$ with the same left singular vectors. It follows that the output of any linear model over X can be exactly matched by the output of a linear model over $\tilde{X};$ if we consider the SVDs $X=US{V}^T,\tilde{X}:=U\tilde{S}{\tilde{V}}^T,$ then for any parameter vector $w\in {ℝ}^d\text{over}X,\tilde{w}:=\tilde{V}{\tilde{S}}^{-1}S{V}^{T}w\text{gives}Xw=\tilde{X}\tilde{w}.$ This observation shows how central the left singular vectors of an embedding matrix are to the set of models which use this matrix, and thus why it is reasonable for the eigenspace overlap score to only consider the left singular vectors. In Appendix B.3 we discuss this score’s robustness to perturbations, while in Appendix B.4 we discuss the connection between this score and a variant of embedding reconstruction error.

3.1.2. Generalization Bound

We now present our theoretical result bounding the difference in generalization performance between models trained on compressed vs. uncompressed embeddings, in terms of the eigenspace overlap score. For this bound, we consider an average-case analysis in the context of fixed design linear regression; note that we consider the fixed design setting for ease of analysis, as it has a closed-form expression for generalization performance. Before presenting our bound in Theorem 1, we will briefly review fixed design linear regression, and discuss the average-case setting we consider.

In fixed design linear regression, we observe a set of labeled points ${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^n$ where the observed labels $y_i={\overline{y}}_i+{ϵ}_i\in ℝ$ are perturbed from the true labels ${\overline{y}}_i$ with independent zero-mean noise ϵ_i (variance σ²). If we let $x_i\in {ℝ}^d$ denote the i^th row of the matrix $X=US{V}^T,\text{and let}y\text{and}\overline{y}\text{in}{ℝ}^n$ denote the perturbed and true label vectors, it is easy to show that the expected error of the optimal linear regressor¹ $f_{X,ϵ}$ trained on this perturbed dataset is equal to ${\mathcal{R}}_{\overline{y}}(X):={\mathbb{E}}_{ϵ}\left[\frac{1}{n}\sum _{i=1}^n{\left(f_{X,ϵ}\left(x_i\right)-{\overline{y}}_i\right)}^2\right]=\frac{1}{n}\left(\Vert \overline{y}{\Vert }^2-{\Vert U^T\overline{y}\Vert }^2+d{\sigma }^2\right).$ For the derivation, see Appendix A.2.

We consider average-case bounds for two reasons: First, in the setting where one would like to use the same compressed embedding across many tasks (i.e., different label vectors $\overline{y}$), an average-case bound describes the average performance across these tasks. Second, for both empirical and theoretical reasons we argue worst-case bounds are too loose to adequately explain our empirical observations. Empirically, we observe that compressed embeddings with large values of Δ₁ and Δ₂ can attain strong generalization performance (Appendix E.6), even though these values imply large worst-case bounds on the generalization error [42]. From a theoretical perspective, worst-case bounds must account for all possible label vectors, including those chosen adversarially. For example, if there exists a single direction in span(U) orthogonal to span $(\tilde{U})$ (which always occurs when $\dim (\tilde{U})<\dim (U))$ the label vector $\overline{y}$ can simply be equal to this direction, resulting in large generalization error for $\tilde{X}$ and small generalization error for X. Thus, we consider an average-case analysis in which we assume y is a random label vector in span(U). We consider this setting because we are most interested in the situation where we know the uncompressed embedding matrix X performs well (in this case, ${\mathcal{R}}_{\overline{y}}(X)=d{\sigma }^2/n),$ and we would like to understand how well $\tilde{X}$ can do.² We now present our bound.

Theorem 1.

Let $X=US{V}^T\in {ℝ}^{n\times d}$ be the singular value decomposition of a full-rank embedding matrix $X.Let\tilde{X}\in {ℝ}^{n\times k}$ be another full-rank embedding matrix, and let $\overline{y}=Uz\in {ℝ}^n$ denote a random label vector in span(U), where z is random with zero mean and identity covariance matrix. It follows that

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]=\frac{d}{n}\cdot (1-\mathcal{E}(X,\tilde{X}))-\frac{d-k}{n}{\sigma }^2.$$ (1)

See Appendix B for the proof, where we consider the more general setting of z having zero mean and arbitrary covariance. This theorem reveals that a larger eigenspace overlap score results in better expected generalization performance for the compressed embedding. Note that if we focus on the low-dimensional and low-noise setting, where $d\ll n,{\sigma }^2<\frac{d}{n}=\frac{1}{n}\mathbb{E}[{\sum }_{i=1}^n{\overline{y}}_i^2],$ we can effectively ignore the term $\frac{d-k}{n}{\sigma }^2=O(d^2/n^2),$ and the generalization performance is determined by the eigenspace overlap score.

Although here we analyze regression, in Appendix B we show the eigenspace overlap score also plays an important role in generalization bounds for Lipschitz-continuous loss functions (e.g., logistic regression).

3.2. The Eigenspace Overlap Score and Uniform Quantization

To help explain the strong downstream performance of uniformly quantized embeddings, in this section we present a lower bound on the expected eigenspace overlap score for uniformly quantized embeddings. Combining this result with Theorem 1 directly provides a guarantee on the performance of the uniformly quantized embeddings. This result further demonstrate how the eigenspace overlap score can be used to better understand the performance of compressed embeddings.

To prove this bound on the eigenspace overlap score, we use the Davis-Kahan $\sin (\mathrm{\Theta })$ theorem from matrix perturbation analysis [8]. Note that we assume unbiased stochastic rounding is used for the uniform quantization (see [14] or Appendix A.1). We now present the result (proof in Appendix C):

Theorem 2.

Let $X\in {ℝ}^{n\times d}$ be a bounded embedding matrix with $X_{ij}\in [-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}}]$³ and smallest singular value ${\sigma }_{\min }=a\sqrt{n/d},fora\in (0,1].$⁴ Let $\tilde{X}$ be an unbiased stochastic uniform quantization of X, where b bits are used per entry. Then for n ≥ max(33, d), we can lower bound the expected eigenspace overlap score of $\tilde{X},$ over the randomness of the stochastic quantization, as follows:

$$\mathbb{E}[1-\mathcal{E}(X,\tilde{X})]\le \frac{20}{{\left(2^b-1\right)}^2{a}^4}.$$

A consequence of this theorem is that with only a logarithmic number of bits $b\ge {\log }_2(\frac{\sqrt{20}}{a^2\sqrt{ϵ}}+1),$ uniform quantization can attain an expected eigenspace overlap score of at least 1 − ϵ. This helps explain the strong downstream performance of uniform quantization at high compression rates.

In Appendix C.2 we empirically validate that the scaling of the eigenspace overlap score with respect to the quantities in Theorem 2 matches the theory; we show $1-\mathcal{E}(X,\tilde{X})$ drops as the precision b and the scalar a are increased, and is relatively unaffected by changes to the vocabulary size n and dimension d.

3.3. The Eigenspace Overlap Score as a Selection Criterion

Due to the theoretical connections between generalization performance and the eigenspace overlap score, we propose using the eigenspace overlap score as a selection criterion between different compressed embeddings. Specifically, the algorithm we propose takes as input an uncompressed embedding along with two or more compressed versions of this embedding, and returns the compressed embedding with the highest eigenspace overlap score to the uncompressed embedding. Ideally, a selection criterion should be both accurate and robust. For each downstream task, we consider accuracy as the fraction of cases where a criterion selects the best-performing embedding on the task. We quantify the robustness as the maximum observed performance difference between the selected embedding and the one which performs the best on a downstream task. In Section 4.3, we show across extensive experiments that the eigenspace overlap score is a more accurate and robust criterion than existing measures of compression quality.

4. Experiments

We empirically validate our theory relating the eigenspace overlap score with generalization performance, our analysis on the strong performance of uniform quantization, and the efficacy of the eigenspace overlap score as an embedding selection criterion. We first demonstrate that this score correlates better with downstream performance than existing measures of compression quality in Section 4.1. We then demonstrate in Section 4.2 that uniform quantization consistently matches or outperforms the compression methods to which we compare, both in terms of the eigenspace overlap score and downstream performance. In Section 4.3, we show that the eigenspace overlap score is a more accurate and robust selection criterion than other measures of compression quality.

Experiment setup

We evaluate compressed versions of publicly available 300-dimensional fastText and GloVe embeddings on question answering and sentiment analysis tasks, and compressed 768-dimensional WordPiece embeddings from the pre-trained case-sensitive BERT_BASE model [10] on tasks from the General Language Understanding Evaluation (GLUE) benchmark [38]. We use the four embedding compression methods discussed in Section 2: DCCL, k-means, uniform quantization, and dimensionality reduction.⁵ For the tasks, we consider question answering using the DrQA model [5] on the Stanford Question Answering Dataset (SQuAD) [33], sentiment analysis using a CNN model [19] on all the datasets used by Kim [19], and language understanding using the BERT_BASE model on the tasks in the GLUE benchmark [38]. We present results on the SQuAD dataset, the largest sentiment analysis dataset (SST-1 [35]) and the two largest GLUE tasks (MNLI and QQP) in this section, and include the results on the other sentiment analysis and GLUE tasks in Appendix E. Across embedding types and tasks, we first compress the pre-trained embeddings, and then train the non-embedding model parameters in the standard manner for each task, keeping the embeddings fixed throughout training. For the GLUE tasks, we add a linear layer on top of the final layer of the pre-trained BERT model (as is standard), and then fine-tune the non-embedding model parameters.6 For more details on the various embeddings, tasks, and hyperparameters we use, see Appendix D.

4.1. The Eigenspace Overlap Score and Downstream Performance

To empirically validate the theoretical connection between the eigenspace overlap score and downstream performance, we show the eigenspace overlap score correlates better with downstream performance than the existing measures of compression quality discussed in Section 2. Thus, even though our analysis is for linear and logistic regression, we see the eigenspace overlap score also has strong empirical correlation with downstream performance on tasks using neural network models.

In Figure 2 we present results for question answering (SQuAD) performance for compressed fastText embeddings, and natural language inference (MNLI) performance for compressed BERT WordPiece embeddings, as a function of the various measures of compression quality. In each plot, for each combination of compression rate and compression method, we plot the average compression quality measure (x-axis) and the average downstream performance (y-axis) across the five random seeds used. If the ranking based on the measure of compression quality was identical to the ranking based on downstream performance, we would see a monotonically decreasing sequence of points. As we can see from the rightmost plots in Figure 2, the downstream performance decreases smoothly as the eigenspace overlap score decreases; the downstream performance does not align as well with the other measures of compression quality.

Figure 2: Downstream performance vs. measures of compression quality.

We plot the performance of compressed fastText embeddings on the SQuAD question answering task (top), and compressed BERT WordPiece embeddings on the MNLI language inference task (bottom), as a function of different measures of compression quality. The eigenspace overlap score $\mathcal{E}$ demonstrates better alignment with downstream performance across compression methods than the other measures. We quantify the degree of alignment using the Spearman correlation ρ, and include ρ in the plot titles.

To quantify how well the ranking based on the quality measures matches the ranking based on downstream performance, we compute the Spearman correlation ρ between these quantities. In Table 1 we can see that the eigenspace overlap score gets consistently higher correlation values with downstream performance than the other measures of compression quality. Note that Δ_max also attains relatively high correlation values, though the eigenspace overlap score still outperforms Δ_max by 0.06 to 0.24 on the tasks in Table 1. See Appendix E.5 for similar results on other tasks.

4.2. Downstream Performance of Uniform Quantization

We show across tasks and compression rates that uniform quantization consistently matches or outperforms the other compression methods, in terms of both the eigenspace overlap score and downstream performance. These empirical results validate our analysis from Section 3.2 showing that uniformly quantized embeddings in expectation attain high eigenspace overlap scores, and are thus likely to attain strong downstream performance. In Figure 3, we plot the eigenspace overlap score and downstream performance averaged over five random seeds for different compression methods and compression rates; we visualize the standard deviation with error bars. In particular, we show the results for the fastText embeddings on question answering (SQuAD, left) and BERT WordPiece embeddings on natural language inference (MNLI, right). Our primary conclusion is that the simple uniform quantization method consistently performs similarly to or better than the other compression methods, both in terms of the eigenspace overlap score and downstream performance.⁷ Given the connections between downstream performance and the eigenspace overlap score, the high eigenspace overlap scores attained by uniform quantization help explain its strong downstream performance. For results with the same trend on the GLUE and sentiment tasks, see Appendices E.1, E.4.⁸

Figure 3: The eigenspace overlap score and downstream performance of compressed embeddings.

Uniform quantization can match or outperform the more complex k-means and DCCL compression methods, both in terms of the eigenspace overlap score $\mathcal{E}$ and downstream performance. We present results for fastText embeddings on question answering (SQuAD, left) and for BERT WordPiece embeddings on natural language inference (MNLI, right).

4.3. Compressed Embedding Selection with the Eigenspace Overlap Score

We now show that the eigenspace overlap score is a more accurate and robust selection criterion for compressed embeddings than the existing measures of compression quality. In our experiment, we first enumerate all the embeddings we compressed using different compression methods, compression rates, and five random seeds, and we evaluate each of these embeddings on the various downstream tasks; we use the same random seed for compression and for downstream training. We then consider for each task all pairs of compressed embeddings, and for each measure of compression quality report the selection error rate—the fraction of cases where the embedding with a higher compression quality score attains worse downstream performance. We show in Table 2 that across different tasks the eigenspace overlap score achieves lower selection error rates than the PIP loss and the spectral distance measures Δ and Δ_max, with 1.3× to 2× lower selection error rates than the second best measure. To demonstrate the robustness of the eigenspace overlap score as a criterion, we measure the maximum difference in downstream performance, across all pairs of compressed embeddings discussed above, between the better performing embedding and the one selected by the eigenspace overlap score. We observe that this maximum performance difference is 1.1× to 5.5× smaller for the eigenspace overlap score than for the measure of compression quality with the second smallest maximum performance difference. See Appendix E.8 for more detailed results on the robustness of the eigenspace overlap score as a selection criterion.

Table 2: The selection error rate of each measure of compression quality as a selection criterion.

Across all pairs of compressed embeddings from our experiments, we measure for each task the fraction of cases when a quality measure selects the worse performing embedding. We observe that the eigenspace overlap score $\mathcal{E}$ achieves lower error rates than other compression quality measures.

Dataset	SQuAD		SST-1		MNLI	QQP
Embedding	GloVe	fastText	GloVe	fastText	BERT WordPiece	BERT WordPiece
PIP loss	0.32	0.37	0.32	0.40	0.31	0.32
Δ	0.34	0.58	0.39	0.57	0.32	0.33
Δmax	0.28	0.22	0.30	0.27	0.15	0.16
1 − $\mathcal{E}$	0.17	0.11	0.19	0.20	0.10	0.10

5. Related Work

Compressing machine learning models is critical for training and inference in resource-constrained settings. To enable low-memory training, recent work investigates using low numerical precision [22, 9] and sparsity [36, 11, 25]. To compress a model for low-memory inference, Han et al. [15] investigate pruning and quantization for deep neural networks.

Our work on understanding the generalization performance of compressed embeddings is also closely related to work on understanding the generalization performance of kernel approximation methods [39, 32]. In particular, training a linear model over compressed word embeddings can be viewed as training a model with a linear kernel using an approximation to the kernel matrix. Recently, there has been work on how different measures of kernel approximation error relate to the generalization performance of the model trained using the approximate kernels, with Avron et al. [3] and Zhang et al. [42] proposing the spectral measures of approximation error which we consider in this work.

6. Conclusion and Future Work

We proposed the eigenspace overlap score, a new way to measure the quality of a compressed embedding without requiring training for each downstream task of interest. We related this score to the generalization performance of linear and logistic regression models, used this score to better understand the strong empirical performance of uniformly quantized embeddings, and showed this score is an accurate and robust selection criterion for compressed embeddings. Although this work focuses on word embeddings, for future work we hope to show that the ideas presented here extend to other domains—for example, to other types of embeddings (e.g., graph node embeddings [13]), and to compressing the activations of neural networks. We also believe that our work can help understand the performance of any model trained using compressed or perturbed features, and to understand why certain proposed methods for compressing neural networks succeed while others fail. We hope this work inspires improvements to compression methods in various domains.

Figure 9: Downstream performance vs. compression rate: task-specific embeddings.

We plot the average BLEU4 test performance for compressed task-specific embeddings on the IWSLT’14 German-to-English translation task across five random seeds (standard deviations indicated with error bars). note that because for some random seeds the TT method attains very low BLEU scores, in this plot we report the best performance for the TT method across the five random seeds. We observe that the uniform quantization and k-means compression methods generally achieve better BLEU4 test performance than the TT method for compression rates up to 128×.

A. Background

A.1. Uniform Quantization

A b-bit uniform quantization of a real number $x\in [-r,r]$ is computed as follows: First, the interval $[-r,r]$ is divided into 2^b − 1 sub-intervals of equal size. Then, x is rounded to either the top or bottom of the sub-interval $[\underset{\_}{x},\overline{x}]$ containing x, where $\underset{\_}{x}=r+j\frac{2r}{2^b-1}\text{and}\overline{x}=r+(j+1)\frac{2r}{2^b-1},\text{for}j\in \{0,1,\dots ,2^b-2\}.$ Given this rounded value, one can simply store the b-bit integer j or j + 1 in place of the real-valued x, depending on whether x was rounded to $\underset{\_}{x}\text{or}\overline{x}$ respectively. In this work, we will consider a deterministic rounding scheme which rounds x to the nearest value (denoted by Q_b,r(x)), as well as an unbiased stochastic rounding scheme (denoted by ${\tilde{Q}}_{b,r}(x).$ More details below). Note that our analysis will focus on the stochastic rounding scheme, while our experiments will focus on deterministic quantization; however, for completeness, in Appendix E.9, we show that stochastic quantization also performs quite well empirically.

We now define unbiased stochastic uniform quantization more formally. We will denote by ${\tilde{Q}}_{b,r}(x)$ the b-bit unbiased stochastic uniform quantization of a real number $x\in [-r,r].$ More formally, if $x\in [\underset{\_}{x},\overline{x}]$ for $\underset{\_}{x}=r+j\frac{2r}{2^b-1}\mathrm{and}\overline{x}=r+(j+1)\frac{2r}{2^b-1},\text{for}j\in \left\{0,1,\dots ,2^b-2\right\},\mathbb{P}\left[{\tilde{Q}}_{b,r}(x)=\underset{\_}{x}\right]=\frac{\overline{x}-x}{\overline{x}-\underset{\_}{x}}\text{and}\mathbb{P}\left[{\overline{Q}}_{b,r}(x)=\overline{x}\right]=\frac{x-\underset{\_}{x}}{\overline{x}-\underset{\_}{x}}.$ Note that $\mathbb{E}[{\tilde{Q}}_{b,r}(x)]=x\text{and}\mathbb{V}\mathbb{A}ℝ[{\tilde{Q}}_{b,r}(x)]\le \frac{r^2}{{(2^b-1)}^2}={\delta }_b^2{r}^2\text{for}{\delta }_b^2:=\frac{1}{{(2^b-1)}^2}.$ We bound the variance using the fact that a bounded random variable in an interval of length c has variance at most $c^2/4$ by Popoviciu’s inequality on variances [31] (in our case, $c=\frac{2r}{2^b-1}$).

Using the above definition of ${\tilde{Q}}_{b,r},$ we define the b-bit stochastic uniform quantization of a matrix X:

Definition 2.

For a bounded embedding matrix $Xwith{X}_{ij}\in [-r,r],$ we define a b-bit stochastic uniform quantization of X to be a matrix $\tilde{X}$ such that ${\tilde{X}}_{ij}={\tilde{Q}}_{b,r}(X_{ij}).$

For details on how we use uniform quantization to compress word embeddings, please see Algorithm 1 and the associated discussion in Appendix D.3.

A.2. Fixed Design Linear Regression

We derive here the close form expression for the risk of fixed design linear regression. In this setting we observe a set of labeled points ${\{(x_i,y_i)\}}_{i=1}^n$ where the observe labels $y_i={\overline{y}}_i+{ϵ}_i\in ℝ$ are perturbed versions of the true label $\overline{y}$ with independent zero-mean noise ϵ_i (with variance σ²). In other words, $y=\overline{y}+ϵ$ with $ϵ$ being a n-dimensional zero-mean random variable with covariance ${\sigma }^2{I}_n.\text{Let}X\in {ℝ}^{n\times d}$ be the feature matrix. The weight vector w^* of the optimal linear regressor $f_{X,ϵ}(x)=\langle x,w^{*}\rangle$ is computed by minimizing the least square loss:

$$w^{*}=\underset{w\in d}{\arg \min }\frac{1}{n}\Vert Xw-y{\Vert }^2.$$

From the normal equation, we know that $w^{*}={(X^{T}X)}^{-1}{X}^{T}y.$ The risk, or expected error, of the optimal linear regressor f_X,ϵ trained on data matrix X and label vector $y=\overline{y}+ϵ$ is defined as

$$\begin{gathered} {\mathcal{R}}_{\overline{y}}(X):={\mathbb{E}}_{ϵ}\left[\frac{1}{n}\sum _{i=1}^n{(f_{X,ϵ}(x_i)-{\overline{y}}_i)}^2\right] \\ ={\mathbb{E}}_{ϵ}\left[\frac{1}{n}{\Vert X{w}^{*}-\overline{y}\Vert }^2\right]. \end{gathered}$$

Proposition 3.

If the feature matrix $X\in {ℝ}^{n\times d}$ is full-rank and has the SVD decomposition $X=US{V}^T,$ then the risk of the optimal linear regressor, in the fixed design linear regression problem with noise variance σ², is

$${\mathcal{R}}_{\overline{y}}(X)=\frac{1}{n}\left(\Vert \overline{y}{\Vert }^2-{\Vert U^T\overline{y}\Vert }^2+d{\sigma }^2\right).$$

Proof. From the normal equation,

$$w^{*}={(X^{T}X)}^{-1}{X}^{T}y={(V{S}^2{V}^T)}^{-1}VS{U}^{T}y=V{S}^{-2}{V}^{T}VS{U}^{T}y=V{S}^{-2}S{U}^{T}y=V{S}^{-1}{U}^{T}y.$$

Substituting this expression into the definition of the risk, we obtain

$$\begin{gathered} {\mathcal{R}}_{\overline{y}}(X)=\frac{1}{n}{\mathbb{E}}_{ϵ}\left[\Vert Xw-\overline{y}{\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\Vert US{V}^{T}V{S}^{-1}{U}^{T}y-\overline{y}\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\Vert U{U}^{T}y-\overline{y}\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\Vert U{U}^T(\overline{y}+ϵ)-\overline{y}\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\Vert \left(U{U}^T-I_n\right)(\overline{y}+ϵ)+ϵ\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\Vert \left(I_n-U{U}^T\right)(\overline{y}+ϵ)-ϵ\Vert }^2\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[\Vert A(\overline{y}+ϵ)-ϵ{\Vert }^2\right]\left(\text{letting}A:=I_n-U{U}^T\right) \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{(\overline{y}+ϵ)}^T{A}^2(\overline{y}+ϵ)-{(\overline{y}+ϵ)}^{T}Aϵ-{ϵ}^{T}A(\overline{y}+ϵ)+{ϵ}^Tϵ\right] \\ =\frac{1}{n}{\mathbb{E}}_{ϵ}\left[{\overline{y}}^{T}A\overline{y}-{ϵ}^{T}Aϵ+{ϵ}^Tϵ\right]\left(\text{using}{A}^2=A,\text{and}{\mathbb{E}}_{ϵ}\left[{ϵ}^{T}A\overline{y}\right]={\mathbb{E}}_{ϵ}\left[{\overline{y}}^{T}Aϵ\right]=0\right) \\ =\frac{1}{n}\left({\overline{y}}^T(I_n-U{U}^T)\overline{y}+{\mathbb{E}}_{ϵ}[-{ϵ}^T\left(I_n-U{U}^T\right)ϵ+{ϵ}^Tϵ]\right) \\ =\frac{1}{n}\left(\Vert \overline{y}{\Vert }^2-{\Vert U^T\overline{y}\Vert }^2+{\mathbb{E}}_{ϵ}\left[{ϵ}^{T}U{U}^Tϵ\right]\right) \\ =\frac{1}{n}\left(\Vert \overline{y}{\Vert }^2-{\Vert U^T\overline{y}\Vert }^2+d{\sigma }^2\right), \end{gathered}$$

where the last step follows from

$${\mathbb{E}}_{ϵ}[{ϵ}^{T}U{U}^Tϵ]={\mathbb{E}}_{ϵ}[\mathrm{tr}(U{U}^Tϵ{ϵ}^T)]=\mathrm{tr}(U{U}^T{\mathbb{E}}_{ϵ}[ϵ{ϵ}^T])=\mathrm{tr}(U{U}^T{\sigma }^2{I}_n)={\sigma }^2\Vert U{\Vert }_F^2=d{\sigma }^2.$$

□

B. The Eigenspace Overlap Score: Theory and Extensions

B.1. Proof of Theorem 1: Average Case Analysis for Fixed Design Linear Regression

We present the proof of Theorem 1, relating the generalization performance and eigenspace overlap score in the context of fixed design linear regression. The true label $\overline{y}$ is assumed to be randomly distributed in the span of U, of the form $\overline{y}=Uz$ for some zero-mean d-dimensional random variable z. We prove a more general version of Theorem 1, for a general covariance matrix Σ of z.

Theorem 1 (Generalized).

Let $X=US{V}^T\in {ℝ}^{n\times d}$ be the singular value decomposition of a full-rank embedding matrix $X.Let\tilde{X}\in {ℝ}^{n\times k}$ be another full-rank embedding matrix with k ≤ d, and let $\overline{y}=Uz\in {ℝ}^n$ denote a random label vector in span(U), where z has mean zero and covariance matrix Σ. Let λ_min(Σ) be the smallest eigenvalue of Σ. Then the gap in expected risk of fixed design linear regression trained on X and $\tilde{X},$ with label noise variance σ², is:

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]\le \frac{\mathrm{tr}\mathrm{\Sigma }-d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X})}{n}-\frac{d-k}{n}{\sigma }^2.$$

If the random variable z has identity covariance matrix, then

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]=\frac{d}{n}\cdot (1-\mathcal{E}(X,\tilde{X}))-\frac{d-k}{n}{\sigma }^2.$$

Proof. Since $\overline{y}=Uz,$ we have

$$\begin{gathered} {\mathbb{E}}_{\overline{y}}[{\Vert U^T\overline{y}\Vert }^2]={\mathbb{E}}_z[{\Vert U^{T}Uz\Vert }^2]={\mathbb{E}}_z[\Vert z{\Vert }^2]={\mathbb{E}}_z[\mathrm{tr}(z^{T}z)] \\ ={\mathbb{E}}_z[\mathrm{tr}(z{z}^T)]=\mathrm{tr}({\mathbb{E}}_z[z{z}^T])=\mathrm{tr}(\mathrm{\Sigma }). \end{gathered}$$

Similarly,

$$\begin{gathered} {\mathbb{E}}_{\overline{y}}[{\Vert {\tilde{U}}^T\overline{y}\Vert }^2]={\mathbb{E}}_z[{\Vert {\tilde{U}}^{T}Uz\Vert }^2]={\mathbb{E}}_z[\mathrm{tr}(z^T{U}^T\tilde{U}{\tilde{U}}^{T}Uz)]={\mathbb{E}}_z[\mathrm{tr}(U^T\tilde{U}{\tilde{U}}^{T}Uz{z}^T)] \\ =\mathrm{tr}(U^T\tilde{U}{\tilde{U}}^{T}U{\mathbb{E}}_z[z{z}^T])=\mathrm{tr}(U^T\tilde{U}{\tilde{U}}^{T}U\mathrm{\Sigma })=\mathrm{tr}({\mathrm{\Sigma }}^{1/2}{U}^T\tilde{U}{\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}) \\ ={\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2, \end{gathered}$$

where Σ^1/2 is the positive semidefinite (PSD) matrix such that (Σ^1/2)² = Σ. From Proposition 3, the risks and ${\mathcal{R}}_{\overline{y}}(X)=\frac{1}{n}(\Vert \overline{y}{\Vert }^2-{\Vert U^T\overline{y}\Vert }^2+d{\sigma }^2)$ and ${\mathcal{R}}_{\overline{y}}(\tilde{X})=\frac{1}{n}(\Vert \overline{y}{\Vert }^2-{\Vert {\tilde{U}}^T\overline{y}\Vert }^2+k{\sigma }^2).$ We thus obtain:

$$\begin{gathered} {\mathbb{E}}_{\overline{y}}\left[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)\right]=\frac{1}{n}({\mathbb{E}}_y[{\Vert U^{T}y\Vert }^2]-{\mathbb{E}}_y[{\Vert {\tilde{U}}^{T}y\Vert }^2]-(d-k){\sigma }^2) \\ =\frac{1}{n}(\mathrm{tr}(\mathrm{\Sigma })-{\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2)-\frac{d-k}{n}{\sigma }^2. \end{gathered}$$ (2)

We can lower bound ${\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2$ in terms of the smallest eigenvalue of Σ and the eigenspace overlap score of $X\text{and}\tilde{X}.$ Specifically, we now show that ${\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2\ge d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X}),\text{where}{\lambda }_{\min }(\mathrm{\Sigma })$ is the smallest eigenvalue of Σ. We will use the fact that $\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d$ is PSD, and so ${\left(\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right)}^{1/2}$ exists. We now prove the above inequality:

$$\begin{gathered} {\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2-d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X})={\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2-{\text{λ}}_{\min }(\mathrm{\Sigma }){\Vert {\tilde{U}}^{T}U\Vert }_F^2 \\ =\mathrm{tr}\left({\mathrm{\Sigma }}^{1/2}{U}^T\tilde{U}{\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\right)-{\text{λ}}_{\min }(\mathrm{\Sigma })\mathrm{tr}\left(U^T\tilde{U}{\tilde{U}}^{T}U\right) \\ =\mathrm{tr}\left(U^T\tilde{U}{\tilde{U}}^{T}U\mathrm{\Sigma }\right)-\mathrm{tr}\left(U^T\tilde{U}{\tilde{U}}^{T}U{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right) \\ =\mathrm{tr}\left(U^T\tilde{U}{\tilde{U}}^{T}U\left(\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right)\right) \\ =\mathrm{tr}\left({\left(\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right)}^{1/2}{U}^T\tilde{U}{\tilde{U}}^{T}U{\left(\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right)}^{1/2}\right) \\ ={\Vert {\tilde{U}}^{T}U{\left(\mathrm{\Sigma }-{\text{λ}}_{\min }(\mathrm{\Sigma })I_d\right)}^{1/2}\Vert }_F^2 \\ \ge 0. \end{gathered}$$

Thus, we have shown that ${\mathbb{E}}_{\overline{y}}[{\Vert {\tilde{U}}^T\overline{y}\Vert }^2]={\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2\ge d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X}).$ Substituting this lower bound into Equation (2) yields

$${\mathbb{E}}_{\overline{y}}\left[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)\right]\le \frac{1}{n}\left(\mathrm{tr}(\mathrm{\Sigma })-d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X})\right)-\frac{d-k}{n}{\sigma }^2.$$

In the case where $\mathrm{\Sigma }=I_d,$ we obtain $\mathrm{tr}(\mathrm{\Sigma })=d\text{and}{\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2={\Vert {\tilde{U}}^{T}U\Vert }_F^2=d\mathcal{E}(X,\tilde{X}).$ Thus from Equation (2), we obtain

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]=\frac{d}{n}(1-\mathcal{E}(X,\tilde{X}))-\frac{d-k}{n}{\sigma }^2.$$

□

B.2. Average Case Analysis for Lipschitz-Continuous Loss Function

We now consider the fixed design setting with a Lipschitz-continuous loss function, and discuss how the average risk of training on $\tilde{X}$ can be bounded in terms of the average risk of training on X and the eigenspace overlap score $\mathcal{E}(X,\tilde{X}).$

Let $\mathcal{l}:ℝ\times ℝ\to ℝ$ be a non-negative loss function which is L-Lipschitz in both its first and second arguments, $X\in {ℝ}^{n\times d}$ be a fixed data matrix with $\text{SVD}X=US{V}^T,x_i\in {ℝ}^d$ be the i^th row of X, and $y\in {ℝ}^n$ be a label vector. We assume that arg ${\min }_{v^{\prime }}\mathcal{l}\left(v^{\prime },v\right)=v\text{for all}v\in ℝ.$ We will consider a linear model f(x) = x^Tw parameterized by some weight vector w, such that the loss function for each data point x_i under this model is $\mathcal{l}(x_i^{T}w,y_i).$

Similar to the fixed design linear regression setting, we assume that y is generated from the true label $\overline{y}\in {ℝ}^n$ by adding zero-mean independent noise: $y=\overline{y}+ϵ$ where $ϵ={[{ϵ}_1,\dots ,{ϵ}_n]}^T\in {ℝ}^n$ and the ϵ_i are independent with zero mean and variance σ². We also assume that the true label $\overline{y}$ is randomly distributed in the span of U, of the form $\overline{y}=Uz$ for some zero-mean d-dimensional random variable z, with covariance matrix Σ. Note that because $\frac{1}{n}\mathbb{E}[\Vert \overline{y}{\Vert }^2]=\frac{1}{n}{\mathbb{E}}_z[\Vert Uz{\Vert }^2]=\frac{1}{n}{\mathbb{E}}_z[\Vert z{\Vert }^2]=\frac{1}{n}\mathrm{tr}(\mathrm{\Sigma }),$ it makes sense for the variance σ² of the noise we add to each entry of y to scale as ${\sigma }^2=O\left(\frac{1}{n}\mathrm{tr}(\mathrm{\Sigma })\right).\text{Lastly, let}\tilde{X}=\tilde{U}\tilde{S}{\tilde{V}}^T\in {ℝ}^{n\times k}$ be any matrix, which for our purposes will represent a compressed version of X.

We now define the optimal weight vectors $w^{*}\text{and}{\tilde{w}}^{*}$ trained on $X\text{and}\tilde{X}$ respectively, along with their corresponding vectors of predictions $u\text{and}\tilde{u}:$

$$w^{*}=\underset{w}{\arg \min }\sum _{i=1}^n\mathcal{l}\left(x_i^{T}w,y_i\right),u:=X{w}^{*}$$

$${\tilde{w}}^{*}=\underset{\tilde{w}}{\arg \min }\sum _{i=1}^n\mathcal{l}({\tilde{x}}_i^T\tilde{w},y_i),\tilde{u}:=\tilde{X}{\tilde{w}}^{*}.$$ (3)

Note that $u\text{and}\tilde{u}$ depend on ϵ and $\overline{y}$, which are random.

The risks, or expected errors (expectation taken over ϵ, for a fixed $\overline{y}$, for the models trained with X and $\tilde{X}$ respectively, are defined as

$${\mathcal{R}}_{\overline{y}}(X):={\mathbb{E}}_{ϵ}\left[\frac{1}{n}\sum _{i=1}^n\mathcal{l}\left(x_i^T{w}^{*},{\overline{y}}_i\right)\right].$$ (4)

$${\mathcal{R}}_{\overline{y}}(\tilde{X}):={\mathbb{E}}_{ϵ}\left[\frac{1}{n}\sum _{i=1}^n\mathcal{l}\left({\tilde{x}}_i^T{\tilde{w}}^{*},{\overline{y}}_i\right)\right].$$ (5)

We are now ready to present our Theorem for the case of average-case generalization performance with L-Lipschitz continuous loss functions. Note that we will use the assumption we stated above that ${\sigma }^2=O\left(\frac{1}{n}\mathrm{tr}(\mathrm{\Sigma })\right).$

Theorem 4.

$LetX=US{V}^T\in {ℝ}^{n\times d}$ be the singular value decomposition of a full-rank embedding matrix X. $Let\tilde{X}\in {ℝ}^{n\times k}$ be another full-rank embedding matrix with k ≤ d, and $let\overline{y}=Uz\in {ℝ}^n$ denote a random label vector in span(U), where z has mean zero and covariance matrix Σ. $Let{\text{λ}}_{\min }(\mathrm{\Sigma })$ be the smallest eigenvalue of Σ. Then the gap in expected risk of the linear models trained on $X\text{and}\tilde{X},$ with a loss function $\mathcal{l}:ℝ\times ℝ\to ℝ$ which is non-negative and L-Lipschitz continuous and satisfies arg ${\min }_{v^{\prime }}\mathcal{l}\left(v^{\prime },v\right)=v\forall v\in ℝ,$ with label noise variance ${\sigma }^2=\frac{a^2}{n}\mathrm{tr}(\mathrm{\Sigma })$ for a scalar $a\in ℝ,$ satisfies:

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]\le \frac{L}{\sqrt{n}}\sqrt{\mathrm{tr}(\mathrm{\Sigma })-d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X})}+\frac{2\mathit{La}\sqrt{\mathrm{tr}(\mathrm{\Sigma })}}{\sqrt{n}}.$$

If the random variable z has identity covariance matrix, then

$${\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]\le \frac{L\sqrt{d}}{\sqrt{n}}(\sqrt{1-\mathcal{E}(X,\tilde{X})}+2a).$$

Proof. Let $f(v,y)=\frac{1}{n}{\sum }_{i=1}^n\mathcal{l}\left(v_i,y_i\right)$ be the average loss on the training set, given predictions v, and let $f(v,\overline{y})=\frac{1}{n}{\sum }_{i=1}^n\mathcal{l}\left(v_i,{\overline{y}}_i\right)$ be the average test loss. Note that $f\text{is}L/\sqrt{n}$-Lipschitz in its first argument, because ${\Vert {\nabla }_{u}f(v,y)\Vert }^2\le {\sum }_{i=1}^n{(L/n)}^2=L^2/n.$ Similarly, $f\text{is}L/\sqrt{n}$-Lipschitz in its second argument. The training loss on X is then $f(u,y),$ and the risk is ${\mathbb{E}}_{ϵ}[f(u,\overline{y})].$ Similarly, the training loss on $\tilde{X}\text{is}f(\tilde{u},y)$ and the risk is ${\mathbb{E}}_{ϵ}[f(\tilde{u},\overline{y})].$

We can bound the difference in the average risk (average over $\overline{y}$) when training on $X\text{and}\tilde{X}$ in terms of the eigenspace overlap score. We do this in three steps: First, we lower bound ${\mathcal{R}}_{\overline{y}}(X).$ Second, we upper bound ${\mathcal{R}}_{\overline{y}}(\tilde{X}).$ Third, we used the bounds from the first two steps to upper bound the expectation over $\overline{y}$ of the difference between ${\mathcal{R}}_{\overline{y}}(X)\text{and}{\mathcal{R}}_{\overline{y}}(\tilde{X}).$ We now go through these steps one at a time:

• Step 1: We show that ${\mathcal{R}}_{\overline{y}}(X)\ge f(\overline{y},\overline{y}).$

  $${\mathcal{R}}_{\overline{y}}(X)={\mathbb{E}}_{ϵ}[f(u,\overline{y})]\ge {\mathbb{E}}_{ϵ}[f(\overline{y},\overline{y})]=f(\overline{y},\overline{y}).$$

  Here, we used the fact that $\overline{y}=\arg {\min }_{v}f(v,\overline{y})$ (which follows from our assumption on the loss function $\mathcal{l}$).
• Step 2: We show that for all $\tilde{w}\in {ℝ}^k,{\mathcal{R}}_{\overline{y}}(\tilde{X})\le f(\tilde{X}\tilde{w},\overline{y})+2L\sigma .$

  $$\begin{gathered} {\mathcal{R}}_{\overline{y}}(\tilde{X})={\mathbb{E}}_{ϵ}[f(\tilde{u},\overline{y})] \\ \le {\mathbb{E}}_{ϵ}\left[f(\tilde{u},y)+\frac{L}{\sqrt{n}}\Vert y-\overline{y}\Vert \right](f\text{is}L/\sqrt{n}-\text{Lipschitsz}) \\ ={\mathbb{E}}_{ϵ}\left[f\left(\tilde{X}{\tilde{w}}^{*},y\right)\right]+{\mathbb{E}}_{ϵ}\left[\frac{L}{\sqrt{n}}\Vert ϵ\Vert \right] \\ \le {\mathbb{E}}_{ϵ}[f(\tilde{X}\tilde{w},y)]+{\mathbb{E}}_{ϵ}\left[\frac{L}{\sqrt{n}}\Vert ϵ\Vert \right](\text{by Equation}(3)) \\ \le {\mathbb{E}}_{ϵ}\left[f(\tilde{X}\tilde{w},\overline{y})+\frac{L}{\sqrt{n}}\Vert \overline{y}-y\Vert \right]+{\mathbb{E}}_{ϵ}\left[\frac{L}{\sqrt{n}}\Vert ϵ\Vert \right](f\text{is}L/\sqrt{n}-\text{Lipschitsz}) \\ =f(\tilde{X}\tilde{w},\overline{y})+\frac{2L}{\sqrt{n}}{\mathbb{E}}_{ϵ}[\Vert ϵ\Vert ] \\ \le f(\tilde{X}\tilde{w},\overline{y})+2L\sigma \left(\text{by}{\mathbb{E}}_{ϵ}{[\Vert ϵ\Vert ]}^2\le {\mathbb{E}}_{ϵ}\left[\Vert ϵ{\Vert }^2\right]=n{\sigma }^2\right). \end{gathered}$$
• Step 3: We bound the expected difference, over the randomness in the label vector $\overline{y}$, between ${\mathcal{R}}_{\overline{y}}(\tilde{X})\text{and}{\mathcal{R}}_{\overline{y}}(X),$ leveraging the results from steps 1 and 2 above.

  $$\begin{gathered} {\mathbb{E}}_{\overline{y}}[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)]\le {\mathbb{E}}_{\overline{y}}[f(\tilde{X}\tilde{w},\overline{y})+2L\sigma -f(\overline{y},\overline{y})](\text{by steps}1\text{and}2) \\ \le {\mathbb{E}}_{\overline{y}}[\frac{L}{\sqrt{n}}\Vert \tilde{X}\tilde{w}-\overline{y}\Vert +2L\sigma ](f\text{is}L/\sqrt{n}\text{-Lipschitsz}). \end{gathered}$$

To get the tightest bound, we can minimize $\Vert \tilde{X}\tilde{w}-\overline{y}\Vert$ over $\tilde{w}\in {ℝ}^k.$ But this is exactly the least squares problem, with solution $\tilde{w}={({\tilde{X}}^T\tilde{X})}^{-1}{\tilde{X}}^T\overline{y}=\tilde{V}{\tilde{S}}^{-1}{\tilde{U}}^T\overline{y},$ and minimum value $\sqrt{\Vert \overline{y}{\Vert }^2-{\Vert {\tilde{U}}^T\overline{y}\Vert }^2}$ (by proof of Proposition 3). We can substitute this bound into the above inequalities and continue:

$$\begin{gathered} {\mathbb{E}}_{\overline{y}}\left[{\mathcal{R}}_{\overline{y}}(\tilde{X})-{\mathcal{R}}_{\overline{y}}(X)\right]\le \frac{L}{\sqrt{n}}{\mathbb{E}}_{\overline{y}}[\sqrt{\Vert \overline{y}{\Vert }^2-{\Vert {\tilde{U}}^T\overline{y}\Vert }^2}]+2L\sigma \\ \le \frac{L}{\sqrt{n}}\sqrt{{\mathbb{E}}_{\overline{y}}\left[\Vert \overline{y}{\Vert }^2-{\Vert {\tilde{U}}^T\overline{y}\Vert }^2\right]}+2L\sigma (\text{by}\text{Jensens}\text{inequality}) \\ \le \frac{L}{\sqrt{n}}\sqrt{\mathrm{tr}(\mathrm{\Sigma })-d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X})}+2L\sigma , \end{gathered}$$

where this last step follows from ${\mathbb{E}}_{\overline{y}}[\Vert \overline{y}{\Vert }^2]={\mathbb{E}}_z[z^T{U}^{T}Uz]={\mathbb{E}}_z[z^{T}z]=\mathrm{tr}(\mathrm{\Sigma }),\text{and}{\mathbb{E}}_{\overline{y}}[{\Vert {\tilde{U}}^T\overline{y}\Vert }^2]\ge d{\text{λ}}_{\min }(\mathrm{\Sigma })\mathcal{E}(X,\tilde{X}),$ which we show in the proof of Theorem 1 (Generalized). Using the assumption ${\sigma }^2=\frac{a^2}{n}\mathrm{tr}(\mathrm{\Sigma }),\text{and thus}\sigma =\frac{a\sqrt{\mathrm{tr}(\mathrm{\Sigma })}}{\sqrt{n}},$ completes the proof. □

Note that in the case of logistic regression where we observe the noisy logits,⁹ the loss is 1-Lipschitz in the first argument. If we assume that the weight vector w has bounded norm (say, because of L2 regularization), and that the data matrix X is bounded, then the loss function is also Lipschitz in the second argument. We can think of z_i as being the optimal logits such that $P(y_i=1)=\sigma (z_i)$ (one can think of these $z_i=x_i^{T}w$ as the parameters of the generative model which generated the data). Just like in the linear regression case, we see that the overlap $\mathcal{E}(X,\tilde{X})=\frac{1}{d}{\Vert {\tilde{U}}^{T}U\Vert }_F^2$ gives an upper bound on the maximum possible expected difference in the loss functions when training on $X\text{vs}.\tilde{X}.$

B.3. Robustness of the Eigenspace Overlap Score to Perturbations

For a measure of compression quality to correlate strongly with downstream performance, a necessary condition is for it to be robust to embedding perturbations which are unlikely to significantly affect generalization performance. Here, we give an example of an embedding perturbation which has minimal effect on the eigenspace overlap score and on average-case generalization performance, while having a much larger impact on the other measures of compression quality. We consider the following simple perturbation: if $X={\sum }_{i=1}^d{\sigma }_i{U}_i{V}_i^T$ is the singular `value decomposition of X, we consider setting it’s largest singular value to 0, resulting in the perturbed matrix $\tilde{X}:={\sum }_{i=2}^d{\sigma }_i{U}_i{V}_i^T.$ Assuming a label vector $y=Uz,\tilde{X}$ would have generalization error of $\Vert y{\Vert }^2-{\Vert {\tilde{U}}^{T}y\Vert }^2=z_1^2.$ If we assume that $z_1^2\ll {\sum }_{i=2}^d{z}_i^2$ (as would be expected in our average-case analysis), then $\tilde{X}$ would perform similarly to X.

In Table 3, we show the impact of the above perturbation on the various measures of compression quality we have discussed. At a high-level, we observe that this perturbation can have a dramatic effect of the previously proposed measures, while having minimal effect on the eigenspace overlap score. For example, the eigenspace overlap score after this perturbation is equal to $\mathcal{E}(X,\tilde{X})=\frac{d-1}{d},$ relative to the maximum possible overlap of 1. In contrast, this perturbation results in a Δ₁ value very close to 1 if $\lambda \ll {\sigma }_1$ (note that 1 is the maximum possible value for Δ₁, and that Zhang et al. [42] show generalization bounds scale with $\frac{1}{1-{\mathrm{\Delta }}_1}$) This makes sense, because Δ₁ can be used to attain a worst-case generalization bound for the perturbed embeddings, and there exist cases where setting the largest singular value to 0 can significantly harm the generalization performance of the embeddings $(\text{e.g., if}{z}_1^2\approx \Vert z{\Vert }^2).$ Thus, while the Δ₁ measure is important for understanding the worst-case performance of the compressed embeddings, it is generally an overly pessimistic measure. The eigenspace overlap score, on the other hand, is generally unable to provide worst-case guarantees, but aligns nicely with the expected performance of the compressed embeddings in the average-case setting.

Table 3: Effect of perturbation on measures of compression quality.

In this table, we consider the effect of perturbing an embedding matrix X by setting its largest singular value to 0 on the various measures of compression quality discussed above. As we can see, setting the largest singular value of X to 0 can have a disproportionately large effect on the relative reconstruction error $\left(\frac{\Vert X-\tilde{X}{\Vert }_F}{\Vert X{\Vert }_F}\right),$ relative PIP loss $\left(\frac{{\Vert X{X}^T-\tilde{X}{\tilde{X}}^T\Vert }_F}{{\Vert X{X}^T\Vert }_F}\right),{\mathrm{\Delta }}_1,\mathrm{\Delta },\text{and}{\mathrm{\Delta }}_{\max }$ measures (values can approach 1), while having a modest effect on the eigenspace overlap score (value of 1/d always).

Compression quality measure	Measure after perturbation
Rel. reconstruction error	${\sigma }_1/\sqrt{{\sum }_{i=1}^d{\sigma }_i^2}$
Rel. PIP loss	${\sigma }_1^2/\sqrt{{\sum }_{i=1}^d{\sigma }_i^4}$
Δ1	${\sigma }_1^2/\left({\sigma }_1^2+\text{λ}\right)$
Δ2	0
Δ	${\sigma }_1^2/\left({\sigma }_1^2+\text{λ}\right)$
Δmax	$\left({\sigma }_1^2+\text{λ}\right)/\text{λ}$
$1-\mathcal{E}(X,\tilde{X})$	$\frac{1}{d}$

B.4. Relating the Eigenspace Overlap Score to Embedding Reconstruction Error

We now define a variant of embedding reconstruction error which we show is closely related to the eigenspace overlap score. As we mention in Section 2.2, the definition of embedding reconstruction error $\Vert X-\tilde{X}{\Vert }_F$ is only applicable when $X\in {ℝ}^{n\times d}\text{and}\tilde{X}\in {ℝ}^{n\times k}$ have the same dimensions (d = k). To get around this limitation, we define the projected embedding reconstruction error as ${\min }_{P\in {ℝ}^{k\times d}}\Vert \tilde{X}P-X{\Vert }_F^2.$ It is easy to show that the matrix P minimizing the above expression is $P^{\star }:={({\tilde{X}}^T\tilde{X})}^{-1}{\tilde{X}}^{T}X.$ Letting $X=US{V}^T\text{and}\tilde{X}=\tilde{U}\tilde{S}{\tilde{V}}^T$ be the singular value decompositions of $X\text{and}\tilde{X},$ we can simplify the expression for the projected embedding reconstruction error as follows:

$$\begin{gathered} \underset{P\in {ℝ}^{k\times d}}{\min }\Vert \tilde{X}P-X{\Vert }_F^2={\Vert \tilde{X}{({\tilde{X}}^T\tilde{X})}^{-1}{\tilde{X}}^{T}X-X\Vert }_F^2 \\ ={\Vert \tilde{U}\tilde{S}{\tilde{V}}^T(\tilde{V}{\tilde{S}}^{-2}{\tilde{V}}^T)\tilde{V}S{\tilde{U}}^{T}X-X\Vert }_F^2 \\ ={\Vert \tilde{U}{\tilde{U}}^{T}X-X\Vert }_F^2 \\ =\mathrm{tr}({(\tilde{U}{\tilde{U}}^{T}X-X)}^T(\tilde{U}{\tilde{U}}^{T}X-X)) \\ =\Vert X{\Vert }_F^2-{\Vert {\tilde{U}}^{T}X\Vert }_F^2 \\ =\Vert X{\Vert }_F^2-{\Vert {\tilde{U}}^{T}US{V}^T\Vert }_F^2 \\ =\Vert X{\Vert }_F^2-{\Vert {\tilde{U}}^{T}US\Vert }_F^2 \end{gathered}$$

Thus, the projected embedding reconstruction error is equal to a term $\left(\Vert X{\Vert }_F^2\right)$ which is constant in $\tilde{X},$ minus a term ${\Vert {\tilde{U}}^{T}US\Vert }_F^2={\sum }_{i=1}^d{\sigma }_i^2{\Vert {\tilde{U}}^T{U}_i\Vert }_2^2.$ Note that this second term is simply a version of the eigenspace overlap score $\frac{1}{\max (d,k)}{\Vert {\tilde{U}}^{T}U\Vert }_F^2=\frac{1}{\max (d,k)}{\sum }_{i=1}^d{\Vert {\tilde{U}}^T{U}_i\Vert }_2^2$ which weights the projections of the different singular vectors $U_i\text{of}X\text{onto}\tilde{U}$ according to the singular values of X. In Section B.1 we show that in the case where the random label vector $\overline{y}=Uz$ where z is a zero mean random variable in ${ℝ}^d$ with covariance Σ, the expected error depends on a term ${\Vert {\tilde{U}}^{T}U{\mathrm{\Sigma }}^{1/2}\Vert }_F^2.$ Thus, the projected embedding reconstruction error is directly related to the expected error when z is sampled with covariance matrix Σ = S².

In Table 4 we show that the projected embedding reconstruction error, like the eigenspace overlap score, attains high Spearman correlation with downstream performance.

Table 4: Spearman correlation between projected embedding reconstruction error and downstream performance.

In this table, we show that the Spearman correlation ρ between the projected embedding reconstruction error and downstream performance is relatively similar to the Spearman correlation between the eigenspace overlap score and downstream performance. We show results on the SQuAD question answering task, and the SST-1 sentiment analysis task, for both GloVe and fastText embeddings. In each table entry, we present the correlation absolute values as “GloVe |ρ| | fastText |ρ|.”

	SQuAD		SST-1
Projected embed. reconst. error	0.82	0.86	0.75	0.64
1 − $\mathcal{E}$	0.81	0.91	0.75	0.73

C. The Eigenspace Overlap Score of Uniformly Quantized Embeddings

This Appendix focuses on the eigenspace overlap score of uniformly quantized embeddings. In Appendix C.1 we prove our result on the expected eigenspace overlap score of uniformly quantized embeddings (Theorem 2). In Appendix C.2 we validate that the empirical scaling of the eigenspace overlap score with respect to the vocabulary size, embedding dimension, compression rate, and smallest singular value of the embedding matrix, matches the scaling predicted by the theory. Lastly, in Appendix C.3, we demonstrate that choosing the clipping value for uniform quantization is crucial for attaining a high eigenspace overlap score, and that choosing the clipping threshold with lowest reconstruction error is very similar to choosing the clipping threshold with highest eigenspace overlap score. Additionally, we demonstrate that the optimal clipping thresholds for deterministic and stochastic quantization are very similar, and that deterministic quantization attains slightly higher eigenspace overlap scores than stochastic quantization.

C.1. Theorem 2 Proof

We now prove Theorem 2, which bounds the expected eigenspace overlap scores for uniformaly quantized embeddings. The core of our proof is an application of the Davis-Kahan $\sin (\mathrm{\Theta })$ theorem [8]. We now review this classic theorem, and then prove our result.

Theorem 5.

(Davis-Kahan $\sin (\mathrm{\Theta })$ Theorem (adapted)) $LetK=U_0{S}_0{U}_0^T+U_1{S}_1{U}_1^T$ be the eigendecomposition of K such that $U_0\in {ℝ}^{n\times d}$ are the first d eigenvectors of $K=US{U}^T,S_0$ the first d eigenvalues, U₁, S₁ the rest. Similarly, let $\tilde{K}=V_0{R}_0{V}_0^T+V_1{R}_1{V}_1^T$ be the equivalent eigendecomposition for $\tilde{K}=K+H.$ If the eigenvalues of S0 are contained in the interval (a₀, a₁), and the eigenvalues of R₁ are excluded from the interval $(a_0-\delta ,a_1+\delta )$ for some δ > 0, then

$$\Vert V_1^T{U}_0\Vert \le \frac{\Vert V_1^{T}H{U}_0\Vert }{\delta }$$ (6)

for any unitarily invariant norm $\Vert \cdot \Vert .$

To prove Theorem 2, we will apply the Davis-Kahan sin(Θ) theorem to the setting where K is the Gram matrix of an uncompressed matrix X, and $\tilde{K}$ is the gram matrix of a b-bit stochastic uniform quantization $\tilde{X}$ of X (See Definition 2). We now present and prove Theorem 2.

Theorem 2.

Let $X\in {ℝ}^{n\times d}$ be a bounded embedding matrix with $X_{ij}\in \left[-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}}\right]$ and smallest singular value ${\sigma }_{\min }=a\sqrt{n/d},\mathit{\text{for}}a\in (0,1].$¹⁰ Let $\tilde{X}$ be a b-bit stochastic uniform quantization of X. Then for n ≥ max(33, d), we can lower bound the expected eigenspace overlap score of $\tilde{X}$, over the randomness of the stochastic quantization, as follows:

$$\mathbb{E}[1-\mathcal{E}(X,\tilde{X})]\le \frac{20}{{\left(2^b-1\right)}^2{a}^4}.$$

Proof. We will denote the Gram matrices of X and $\tilde{X}\text{by}K=X{X}^T=US{U}^T\text{and}\tilde{K}=\tilde{X}{\tilde{X}}^T=(X+C){(X+C)}^T=VR{V}^T.$ Here, C is a stochastic matrix satisfying $\mathbb{E}[C_{ij}]=0\text{and}\mathbb{V}\mathbb{A}ℝ[C_{ij}]\le {\delta }_b^2/d\forall i,j,\text{for}{\delta }_b^2:=\frac{1}{{(2^b-1)}^2}$ (see Appendix A.1). In our application of the Davis-Kahan $\sin (\mathrm{\Theta })$ theorem, we will use $a_0={\sigma }_{\min }(K),a_1=\infty ,\delta ={\sigma }_{\min }(K).$ Note also the $H=\tilde{K}-K=(X+C){(X+C)}^T-X{X}^T=X{C}^T+C{X}^T+C{C}^T.$ We will let a ∈ [0, 1] be the scalar such that ${\sigma }_{\min }(X)=a\sqrt{\frac{n}{d}}$ (equivalently, ${\sigma }_{\min }(K)=a^2\frac{n}{d}$).

Using the Davis-Kahan $\sin (\mathrm{\Theta })$ theorem, along with Lemma 6 (below), we can show the following:

$$\begin{gathered} {\Vert V_1^T{U}_0\Vert }_F\le \frac{{\Vert V_1^{T}H{U}_0\Vert }_F}{{\sigma }_{\min }(K)} \\ =\frac{{\Vert V_1^T\left(X{C}^T+C{X}^T+C{C}^T\right)U_0\Vert }_F}{{\sigma }_{\min }(K)} \\ \le \frac{{\Vert V_1^T\Vert }_2{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F{\Vert U_0\Vert }_2}{{\sigma }_{\min }(K)}(\text{using}\Vert AB{\Vert }_F\le \Vert A{\Vert }_2\Vert B{\Vert }_F\text{twice.}) \\ \le \frac{{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F}{{\sigma }_{\min }(K)}(\text{using}{\Vert V_1^T\Vert }_2={\Vert U_0\Vert }_2=1.) \\ \Rightarrow \frac{1}{d}{\Vert V_1^T{U}_0\Vert }_F^2\le \frac{{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2}{d\cdot {\sigma }_{\min }{(K)}^2} \\ \iff 1-\frac{1}{d}{\Vert V_0^T{U}_0\Vert }_F^2\le \frac{{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2}{d\cdot {\sigma }_{\min }{(K)}^2}\left(\text{using}{\Vert V_0^T{U}_0\Vert }_F^2+{\Vert V_1^T{U}_0\Vert }_F^2={\Vert U_0\Vert }_F^2=d\right) \\ \iff 1-\mathcal{E}(X,\tilde{X})\le \frac{{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2}{d\cdot {\sigma }_{\min }{(K)}^2} \end{gathered}$$

$$\begin{gathered} \Rightarrow \mathbb{E}[1-\mathcal{E}(X,\tilde{X})]\le \mathbb{E}\left[\frac{{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2}{d\cdot {\sigma }_{\min }{(K)}^2}\right] \\ =\frac{\mathbb{E}\left[{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2\right]}{d\cdot {\sigma }_{\min }{(K)}^2} \\ \le \frac{\frac{20n^2{\delta }_b^2}{d}}{d\cdot {\sigma }_{\min }{(K)}^2}(\text{by Lemma}6). \\ =\frac{20n^2{\delta }_b^2}{d^2{a}^4\left(n^2/d^2\right)} \\ =\frac{20{\delta }_b^2}{a^4} \end{gathered}$$

□

We now present and prove Lemma 6.

Lemma 6.

Let $X\in {ℝ}^{n\times d}$ be a bounded embedding matrix with $X_{ij}\in \left[-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}}\right].\text{Let}\tilde{X}=X+C$ be a b-bit stochastic uniform quantization of X. Then for n ≥ max(33, d), it follows that

$$\mathbb{E}\left[{\Vert X{C}^T+C{X}^T+C{C}^T\Vert }_F^2\right]\le \frac{20n^2{\delta }_b^2}{d}.$$ (7)

Proof. We will let $H:=X{C}^T+C{X}^T+C{C}^T.$ To bound $\mathbb{E}[\Vert H{\Vert }_F^2]={\sum }_{i,j=1}^n\mathbb{E}[H_{ij}^2],$ we will consider two cases: $H_{ij}\text{for}i\ne j\text{and}{H}_{ij}\text{for}i=j.$ We will let $x_i,c_i\in {ℝ}^d$ denote the i^th rows of X and C respectively.

1. Case 1: $i\ne j$

   $$\begin{gathered} \mathbb{E}\left[H_{ij}^2\right]=\mathbb{E}\left[{\left(x_i^T{c}_j+c_i^T{x}_j+c_i^T{c}_j\right)}^2\right] \\ =\mathbb{E}\left[{\left(x_i^T{c}_j\right)}^2+{\left(c_i^T{x}_j\right)}^2+{\left(c_i^T{c}_j\right)}^2\right] \\ =\mathbb{E}\left[{\left(\sum _{k=1}^d{x}_{ik}{c}_{jk}\right)}^2\right]+\mathbb{E}\left[{\left(\sum _{k=1}^d{c}_{ik}{x}_{jk}\right)}^2\right]+\mathbb{E}\left[{\left(\sum _{k=1}^d{c}_{ik}{c}_{jk}\right)}^2\right] \\ =\mathbb{E}\left[\left(\sum _{k=1}^d{x}_{ik}^2{c}_{jk}^2\right)\right]+\mathbb{E}\left[\left(\sum _{k=1}^d{c}_{ik}^2{x}_{jk}^2\right)\right]+\mathbb{E}\left[\left(\sum _{k=1}^d{c}_{ik}^2{c}_{jk}^2\right)\right] \\ =\sum _{k=1}^d{x}_{ik}^2\mathbb{E}\left[c_{jk}^2\right]+\sum _{k=1}^d\mathbb{E}\left[c_{ik}^2\right]x_{jk}^2+\sum _{k=1}^d\mathbb{E}\left[c_{ik}^2\right]\mathbb{E}\left[c_{jk}^2\right] \\ \le \frac{{\delta }_b^2}{d}\cdot \sum _{k=1}^d{x}_{ik}^2+\frac{{\delta }_b^2}{d}\cdot \sum _{k=1}^d{x}_{jk}^2+\sum _{k=1}^d{\left(\frac{{\delta }_b^2}{d}\right)}^2 \\ \le \frac{{\delta }_b^2}{d}\cdot {\Vert x_i\Vert }^2+\frac{{\delta }_b^2}{d}\cdot {\Vert x_j\Vert }^2+\sum _{k=1}^d{\left(\frac{{\delta }_b^2}{d}\right)}^2 \\ \le \frac{2{\delta }_b^2+{\delta }_b^4}{d}(\text{using}{\Vert x_i\Vert }^2\le 1) \\ \le \frac{3{\delta }_b^2}{d}(\text{using}{\delta }_b\le 1). \end{gathered}$$
2. Case 2: $i=j$

   $$\begin{gathered} \mathbb{E}\left[H_{ii}^2\right]=\mathbb{E}\left[{\left(x_i^T{c}_i+c_i^T{x}_i+c_i^T{c}_i\right)}^2\right] \\ =\mathbb{E}\left[{\left(2\sum _{k=1}^d{x}_{ik}{c}_{ik}+\sum _{l=1}^d{c}_{il}^2\right)}^2\right] \\ =\mathbb{E}\left[4{\left(\sum _{k=1}^d{x}_{ik}{c}_{ik}\right)}^2+4\left(\sum _{k=1}^d{x}_{ik}{c}_{ik}\right)\cdot \left(\sum _{l=1}^d{c}_{il}^2\right)+{\left(\sum _{l=1}^d{c}_{il}^2\right)}^2\right] \\ =\mathbb{E}\left[4\sum _{k=1}^d{x}_{ik}^2{c}_{ik}^2+4\sum _{k,l=1}^d{x}_{ik}{c}_{ik}{c}_{il}^2+\sum _{k,l=1}^d{c}_{il}^2{c}_{ik}^2\right] \\ =4\sum _{k=1}^d{x}_{ik}^2\mathbb{E}\left[c_{ik}^2\right]+4\sum _{k=1}^d{x}_{ik}\mathbb{E}\left[c_{ik}^3\right]+\sum _{k,l=1}^d\mathbb{E}\left[c_{il}^2{c}_{ik}^2\right] \\ \le 4\cdot \frac{{\delta }_b^2}{d}\cdot \sum _{k=1}^d{x}_{ik}^2+4\sum _{k=1}^d\frac{1}{\sqrt{d}}{\left(\frac{2}{\sqrt{d}\left(2^b-1\right)}\right)}^3+\sum _{k,l=1}^d{\left(\frac{2}{\sqrt{d}\left(2^b-1\right)}\right)}^4 \\ =4\cdot \frac{{\delta }_b^2}{d}\cdot {\Vert x_i\Vert }^2+4d\cdot \frac{8}{d^2{\left(2^b-1\right)}^3}+d^2\cdot \frac{16}{d^2{\left(2^b-1\right)}^4} \\ \le \frac{4{\delta }_b^2}{d}+\frac{32}{d{\left(2^b-1\right)}^3}+\frac{16}{{\left(2^b-1\right)}^4} \\ =\frac{4{\delta }_b^2+32{\delta }_b^3}{d}+16{\delta }_b^4 \\ \le \frac{36{\delta }_b^2}{d}+16{\delta }_b^4(\text{using}{\delta }_b\le 1). \end{gathered}$$

Now we can combine the above results:

$$\begin{gathered} \sum _{i,j=1}^n\mathbb{E}\left[H_{ij}^2\right]\le \sum _{i\ne j}\left(\frac{3{\delta }_b^2}{d}\right)+\sum _{i=1}^n\left(\frac{36{\delta }_b^2}{d}+16{\delta }_b^4\right) \\ =n(n-1)\left(\frac{3{\delta }_b^2}{d}\right)+n\left(\frac{36{\delta }_b^2}{d}+16{\delta }_b^4\right) \\ =\frac{3n^2{\delta }_b^2-3n{\delta }_b^2+36n{\delta }_b^2}{d}+16n{\delta }_b^4 \\ =\frac{3n^2{\delta }_b^2+33n{\delta }_b^2}{d}+16n{\delta }_b^4 \\ \le \frac{4n^2{\delta }_b^2}{d}+16n{\delta }_b^4(\text{assuming}n\ge 33.) \\ \le \frac{4n^2{\delta }_b^2}{d}+\frac{16n^2{\delta }_b^2}{d}(\text{assuming}n\ge d.) \\ =\frac{20n^2{\delta }_b^2}{d} \end{gathered}$$

□

C.2. Empirical Validation of Theorem 2 Scaling

We now validate Theorem 2 empirically by showing the impact of the precision (b), the scalar (a), the vocabulary size (n), and the embedding dimension (d) on the eigenspace overlap score $\mathcal{E}(X,\tilde{X})$ of uniformly quantized embeddings matrices. As predicted by the theory, we will show in Figure 4 that $1-\mathcal{E}(X,\tilde{X})$ drops as b and a are increased, and is relatively unaffected by changes in n and d.

Figure 4: Empirical Validation of Theorem 2.

We measure the eigenspace overlap score $\mathcal{E}$ of uniformly quantized embeddings with the uncompressed embedding for various precisions, values of a, vocabulary sizes n, and dimensions d. We observe that 1 − $\mathcal{E}$ decays as the precision b and scalar a grow, and that 1 − $\mathcal{E}$ is largely unaffected by the vocabulary size n and embedding dimension d.

We now describe our experimental protocol for studying the impact of each of these parameters on the eigenspace overlap score:

• Precision (b), Figure 4(a): We randomly generate a 10⁴ × 10 matrix, with entries drawn uniformly from $\left[-\frac{1}{\sqrt{10}},\frac{1}{\sqrt{10}}\right].$ We uniformly quantize this matrix with precisions $b\in \{1,2,4,8,16\}$, and compute the eigenspace overlap score between the quantized matrix and the original matrix. As one can see, $1-\mathcal{E}(X,\tilde{X})$ drops rapidly as the precision is increased.

• Scalar (a), Figure 4(b): We randomly generate a 10⁴ × 10 matrix, with entries drawn uniformly from $\left[-\frac{1}{\sqrt{10}},\frac{1}{\sqrt{10}}\right].$ We then multiply this matrix on the right by diagonal matrices with diagonal entries spaced logarithmically between 1 and $\{1,0.1,.01,.001,.0001\},$ thus generating matrices with increasingly small values of the scalar a. We uniformly quantize each of these matrices with precisions $b\in \{1,2,4\},$ and compute the eigenspace overlap score between the quantized matrices and the original matrices. As one can see, $1-\mathcal{E}(X,\tilde{X})$ drops as the scalar a increases.

• Vocabulary size (n), Figure 4(c): We randomly generate n × 10 matrices for $n\in \{{10}^2,3\times {10}^2,{10}^3,3\times {10}^3,{10}^4,3\times {10}^4,{10}^5\},$ with entries drawn uniformly from $\left[-\frac{1}{\sqrt{10}},\frac{1}{\sqrt{10}}\right].$ We uniformly quantize these matrices with precisions $b\in \{1,2,4\},$ and compute the corresponding eigenspace overlap scores. As one can see, the vocabulary size n has minimal impact on the eigenspace overlap score.

• Embedding dimension (d), Figure 4(d): We randomly generate 10⁴ × d matrices for $d\in \{10,30,100,300,1000\}$ with entries drawn uniformly from $\left[-\frac{1}{\sqrt{d}},\frac{1}{\sqrt{d}}\right].$ We uniformly quantize these matrices with precisions $b\in \{1,2,4\},$ and compute the corresponding eigenspace overlap scores. As one can see, the embedding dimension d has minimal impact on the eigenspace overlap score.

An important thing to mention about Theorem 2 is that this bound can be vacuous when the embedding matrix has a quickly decaying spectrum, and thus a small value of a. This is a consequence of the proof of the Davis-Kahan $\sin (\mathrm{\Theta })$ theorem, which uses the smallest eigenvalue of $X{X}^T$ to lower bound a matrix multiplication; this inequality is relatively tight when the spectrum of $X{X}^T$ decays slowly, but is quite loose if it doesn’t.

C.3. Impact of Clipping and Deterministic vs. Stochastic Quantization on the Eigenspace Overlap Score

As shown in Algorithm 1 (described in Section D.3), clipping is the first step in the uniform quantization method we use for compressing word embeddings. Here, we show that clipping is important because it can significantly improve the eigenspace overlap scores of the compressed embeddings, compared to uniform quantization without clipping. Specifically, we compute the eigenspace overlap score of $Q_{b,r}\left({\mathrm{clip}}_r(X)\right)(\text{and}{\tilde{Q}}_{b,r}\left({\mathrm{clip}}_r(X)\right))$ with X, for a range of clipping values $r\in [0,\max (|X|)],$ using the publicly available 300-dimensional pre-trained GloVe embeddings as X (see Appendix D.2 for embedding details). Recall that $Q_{b,r}\text{and}{\tilde{Q}}_{b,r}$ are the deterministic and stochastic b-bit uniform quantization functions for the interval $[-r,r],$ respectively (defined in Section A.1). In Figure 5(a), we plot the eigenspace overlap scores attained by both quantization methods as a function of the clipping value r, for precisions $b\in \{1,2,4\}.$ We observe that choosing the value of r appropriately is crucial for attaining high eigenspace overlap scores. We also observe that deterministic quantization typically attains slightly higher eigenspace overlap scores than stochastic quantization. This result helps explain our empirical observation in Appendix E.9 that deterministic quantization often attains slightly better downstream performance than stochastic quantization.

Figure 5: The impact of clipping and deterministic vs. stochastic quantization on the eigenspace overlap score.

(a) We plot the eigenspace overlap score as a function of the clipping threshold, for precisions $b\in \{1,2,4\}$ and for both stochastic and deterministic quantization. We observe that choosing the value of r appropriately is crucial for attaining high eigenspace overlap scores, and that deterministic quantization generally gives slightly higher eigenspace overlap scores than stochastic quantization. (b) For each precision b, we plot the clipping thresholds which give the highest eigenspace overlap scores and embedding reconstruction errors, for both stochastic and deterministic quantization. We observe that for both types of quantization, the optimal clipping threshold chosen according to embedding reconstruction error is very similar to the optimal clipping threshold chosen according to the eigenspace overlap score.

In Algorithm 1, we choose the clipping threshold $r^{*}$ which minimizes the embedding reconstruction error of the clipped and quantized embeddings. In Figure 5(b), we show that choosing the clipping threshold based on the embedding reconstruction error gives very similar results to choosing the clipping threshold based on the eigenspace overlap score, for both deterministic and stochastic quantization. This helps explain the strong downstream performance of the embeddings compressed using Algorithm 1.

D. Experiment Details

We now discuss in detail the protocols we used for all our experiments. In Appendix D.1, we describe the model architectures and datasets we use for each downstream task, including the train/development/test splits for each dataset. We then discuss in Appendix D.3 the details of the different compression methods we use. In Appendix D.4 we discuss the training details for each of the downstream tasks, including the hyperparameter grids we use to tune our models.

D.1. Task Details

Question Answering

For the question answering task, we use the DrQA model [5] trained and evaluated on the Stanford Question and Answering Dataset (SQuAD) [33]. For this task, given a paragraph and a corresponding question in natural language, the model must predict the start and end position, within the paragraph, of the answer to the question. We use the default train and development set splits for the SQuAD dataset, and report all results on the development set, as the test set is not publicly available. The DrQA model consists of a three-layer bidirectional LSTM model with 128-dimensional hidden units on top of a pretrained word embedding. We train the DrQA model on the SQuAD-v1.1 training set, and report the F1 score on the SQuAD-v1.1 development set. We use the implementation of the DrQA model from the Facebook Research DrQA repository.¹¹

Sentiment Analysis

For the sentiment analysis tasks, we use the convolutional neural network (CNN) architecture proposed by Kim [19], and evaluate performance on the datasets used in that work (see Section 3 of that paper for dataset details). We use the data released as part of the Harvard NLP group’s sentiment analysis repository.¹² For the datasets which are pre-split into train/development/test (SST-1, SST-2), we use these dataset splits. For the datasets which are pre-split into train/test (TREC), we take a random 10% of the training set as a development set. For the datasets which have no pre-specified splits (MR, Subj, CR, MPQA), we take a random 10% of the data as a test set, and a random 10% of the remaining data as a development set; the rest of the data is used as the training set. We tune hyperparameters (learning rate) on the development sets, and report results on the test sets. The CNN architecture we use for this task has one convolutional layer with multiple filters, followed by a ReLU non-linearity and a max-pooling layer. The convolutional layer uses filter windows of size 3, 4, and 5, each with 100 feature maps. As we use PyTorch [27] for all our experiments, we reimplemented this model architecture in PyTorch, using the original Theano implementation as a template.¹³.

GLUE Tasks

The General Language Understanding Evaluation (GLUE) benchmark [38] is a collection of nine natural language understanding tasks. We summarize these tasks in Table 5, along with the evaluation metric used for each task. We use the default train and development set splits for each of these tasks. We tune hyperparameters (learning rate) on the development sets, and also report results on the development sets, as the test sets are not publicly available. For each task, we use the standard approach of adding a linear layer on top of the pre-trained BERT model, and then fine-tuning the model using the data for that task. To evaluate the performance of compressed embeddings on these tasks, we compress the WordPiece [40] embeddings in the pre-trained case-sensitive BERT_BASE model, and then fine-tune all the non-embedding model parameters, keeping the embeddings frozen during training. We use a third-party implementation of the BERT model, and of the fine-tuning procedure.¹⁴ We run experiments on all the GLUE tasks except WNLI. We skip the WNLI dataset because this is a dataset on which it is very difficult to outperform the trivial model which always outputs the majority class. This trivial model attains 65.1% accuracy, and only two of the contributors to the GLUE leaderboard¹⁵ have outperformed this model, as of this writing.

Table 5: The GLUE datasets, along with the evaluation metric used for each dataset.

For the MRPC and QQP datasets, the average of the F1 score and accuracy on the development set is used. For the STS-B dataset, the average of the Pearson and Spearman correlations on the development set is used. For the MNLI dataset, the average of the accuracies on the matched and mismatched development sets is used.

Datasets	Evaluation Metrics
The Corpus of Linguistic Acceptability (CoLA)	Matthew’s Correlation
The Stanford Sentiment Treebank (SST-2)	Accuracy
Microsoft Research Paraphrase Corpus (MRPC)	F1 / Accuracy
Semantic Textual Similarity Benchmark (STS-B)	Pearson-Spearman Correlation
Quora Question Pairs (QQP)	F1 / Accuracy
Multi-Genre Natural Language Inference (MNLI)	Accuracy (matched/mismatched)
Question Natural Language Inference (QNLI)	Accuracy
Recognizing Textual Entailment (RTE)	Accuracy
Winograd Natural Language Inference (WNLI)	Accuracy

D.2. Word Embedding Details

For the GloVe embeddings, we use publicly available embeddings pre-trained on the Wikipedia 2014 and Gigaword 5 corpora.¹⁶ These are available for dimensions $d\in \{50,100,200,300\};$ we use the 300-dimensional embeddings for all our experiments, except for our GloVe dimensionality reduction experiments, where we use the lower-dimensional embeddings. For the fastText embeddings, we use the publicly available 300-dimensional embeddings trained on the Wikipedia 2017 corpus, the UMBC webbase corpus, and the statmt.org news dataset.¹⁷ For the WordPiece embeddings [40], we use the embeddings which are part of the pre-trained case-sensitive BERT_BASE model, available through the Hugging Face BERT repository.¹⁸

D.3. Compression Method Details

Uniform Quantization

In Algorithm 1 we show how we use uniform quantization to compress word embeddings. The input to the algorithm is an embedding matrix $X\in {ℝ}^{n\times d},$ where n is the size of the vocabulary, and d is the dimension of the embeddings. We define the function ${\mathrm{clip}}_r(x)=\max (\min (x,r),-r)$ for any non-negative r; when matrices are passed in as inputs to this function, it clips the entries in an element-wise fashion. Given an input embedding and a desired numbers of bits to use per entry of the compressed embedding matrix, the uniform quantization method operates in two steps:

• Step 1: We find the value of $r\in [0,\max (|X|)]$ which minimizes the reconstruction error of the quantized embeddings after X is clipped to $[-r,r].$ More formally, we let $r^{*}:=\arg {\min }_{r\in [0,\max (|X|)}\Vert Q_{b,r}\left({\mathrm{clip}}_r(X)\right)-X{\Vert }_F,$ and use this value r^* to clip X. In our experiments, we find r^* to within a specified tolerance ϵ = 0.01 using the golden-section search algorithm [18]. To avoid stochasticity impacting the search process for the clipping threshold, we always use deterministic rounding in the search for r^*, regardless of whether we use stochastic rounding or deterministic nearest rounding in the final quantization after clipping the extremal values.

• Step 2: We quantize the clipped embeddings to b bits per entry with $Q_{b,r}.$

In all of our main experiments on the downstream performance (question answering, sentiment analysis, GLUE tasks) of compressed word embeddings, we use the deterministic quantization function $Q_{b,r}$ introduced in Appendix A.1 for both steps of this algorithm. However, in Appendix E.9 we use the stochastic quantization function ${\tilde{Q}}_{b,r}$ for the second step of this compression algorithm, and show that it performs similarly to deterministic quantization on downstream tasks.

K-means

The k-means clustering method can be used to compress embeddings as follows: First, the one-dimensional k-means clustering algorithm is run on all the scalar entries in the full-precision embedding matrix X. Then, each entry in X is replaced by the centroid to which it is closest. If 2^b centroids are used during the clustering step, then for each entry of the compressed embedding matrix, only the integer $j\in \{0,1,\dots ,2^b-1\}$ of the corresponding centroid needs to be stored; this requires b bits per entry. In our experiments, we use the Scikit Learn [28] implementation of k-means. We use the default configuration from

Algorithm 1.

Uniform quantization for word embeddings

1:	Input: Embedding $X\in {ℝ}^{n\times d};$ quantization func. $Q_{b,r};$ clipping func. clipr: $ℝ\to [-r,r].$
2:	Output: Quantized embedding $\tilde{X}\in {ℝ}^{n\times d}.$
3:	$r^{*}:=\arg {\min }_{r\in [0,\max (|X|)]}{\Vert Q_{b,r}\left({\mathrm{clip}}_r(X)\right)-X\Vert }_F.$
4:	Return: $Q_{b,r^{*}}({\mathrm{clip}}_{r^{*}}(X)).$

Scikit Learn, which runs for a maximum of 300 iterations and can early stop if the relative decrease of the loss function is smaller than 10⁻⁴.

Deep Compositional Code Learning (DCCL)

We give an overview of the DCCL method [34] in Section 2.1. The important hyperparameters for this method include the learning rate η of the Adam optimizer [20], the number of dictionaries m, the size k of each dictionary, the temperature parameter τ for Gumbel sampling, and the mini-batch size. To select the learning rate η and the dictionary size k for each compression rate, we perform a grid search using the Cartesian product of $\eta \in \{0.00001,0.00003,0.0001,0.0003,0.001\}\text{and}k\in \{2,4,8,16\}$ for each uncompressed embedding type (GloVe, fastText, BERT WordPiece embeddings) and compression rate. Note that given a compression rate and a dictionary size k, this uniquely determines the number of dictionaries m to use. We select the combination of learning rate and dictionary size which minimizes the reconstruction error of the compressed embeddings. When compressing BERT WordPiece embedding, we extended the dictionary size grid to $k\in \{2,4,8,16,32,64,128,256\}$ to avoid the optimal dictionary size touching the boundary of the grid. We provide the optimal learning rates and dictionary sizes in Table 6 for reproducibility. For the temperature parameter τ, we follow Shu and Nakayama [34] and consistently use τ = 1.0. For all our experiments we use a mini-batch size of 64, which is the default value in the DCCL repository.¹⁹

Table 6:

The optimal learning rates η and dictionary sizes k for DCCL.

Embedding	GloVe			fastText			BERT WordPiece
Compression rate	8×	16×	32×	8×	16×	32×	8×	16×	32×
k	8	4	4	8	4	8	128	64	32
η	0.0003	0.0003	0.0003	0.0001	0.0001	0.0001	0.0003	0.0003	0.0003

Dimensionality Reduction

The two dimensionality reduction methods we consider are (1) using pre-trained lower-dimensional embeddings, and (2) principal component analysis (PCA). For the GloVe embeddings, we use the publicly available lower-dimensional embeddings described in Appendix D.2. These embeddings are available for dimensions $d\in \{50,100,200,300\},$ where we consider the 300-dimensional embeddings to be the “uncompressed” embeddings. For our experiments with fastText and BERT WordPiece embeddings, we use PCA to reduce the dimension of the embeddings, as these embeddings are not publicly available in lower dimensions. When we compress the 300-dimensional fastText and GloVe embeddings with dimensionality reduction, we use compression rates in $\{1\times ,1.5\times ,3\times ,6\times \}.$ For the 768-dimensional BERT WordPiece embeddings, we use compression rates in $\{1\times ,2\times ,4\times ,8\times \}.$

We now give details on how we implement the PCA dimensionality reduction method. For an embedding $X\in {ℝ}^{n\times d}$ with vocabulary size n and dimension d, let $X=US{V}^T\text{be the SVD of}X\text{with}U=[U_1,\dots U_d],S=\mathrm{diag}\left([s_1,\dots s_d]\right),\text{and}V=[V_1,\dots V_d].$ If we let $U_{(k)}:=[U_1,\dots ,U_k],S_{(k)}:=\mathrm{diag}\left([s_1,\dots s_k]\right),\text{and}{V}_{(k)}:=[V_1,\dots ,V_k]$ then we use $\tilde{X}:=U_{(k)}{S}_{(k)}$ as the k-dimensional compressed embedding. Note that for the GLUE tasks, we instead use $\tilde{X}:=U_{(k)}{S}_{(k)}{V}_{(k)}^T$ to ensure that these compressed embeddings are compatible with the parameters of the pre-trained BERT model; because the dimension k of these compressed embeddings is small compared to the vocabulary size n, storing $V_{(k)}^T$ requires a relatively small amount of additional memory.

D.4. Training Details

We now discuss the training details for the different tasks we consider, focusing on how we tune the hyperparameters.

Question Answering

We use the default hyperparameters from the Facebook Research DrQA implementation for all our question answering experiments, as these are tuned for the SQuAD dataset.²⁰ We summarize these hyperparameters in Table 7.

Table 7:

Training hyperparameter for DrQA on the SQuAD dataset.

Hyperparameter	Value
Optimizer	Adamax
Decay rates for 1st moment β1	0.9
Decay rates for 2nd moment β2	0.999
Adamax ϵ	10−8
Learning rate	2 × 10−3
Batchsize	32
Training epochs	40
Dropout	0.4

Sentiment Analysis

We tune the learning rate for each of the sentiment analysis datasets using the grid {10⁻⁶, 10⁻⁵, 10⁻⁴, 10⁻³, 10⁻², 10⁻¹, 1.0}. For this tuning process, we use the uncompressed embedding for each dataset and embedding type (GloVe, fastText), and pick the learning rate which attains highest average accuracy on the development set across five random seeds. This learning rate is then used to train the models that use the uncompressed embeddings, as well as the embeddings compressed using uniform quantization, k-means, and DCCL. Note that we tune the learning rate individually for each embedding compressed using dimensionality reduction (for both GloVe and fastText). We do this to ensure that the lower dimensionality of these compressed embeddings does not result in the learning rate being improperly tuned. We list the hyperparameters shared across datasets in Table 8 and list the optimal learning rate for each dataset and embedding type in Table 9.

Table 8:

Training hyperparameter shared across sentiment analysis datasets.

Hyperparameter	Value
Optimizer	Adam
Decay rates for 1st moment β1	0.9
Decay rates for 2nd moment β2	0.999
Adam ϵ	10−8
Batchsize	32
Training epochs	100
Dropout	0.5

Table 9:

The optimal learning rate η for different sentiment analysis datasets.

Datasets	MR	SST-1	SST-2	Subj	TREC	CR	MPQA
GloVe uncompressed	0.001	0.001	0.001	0.001	0.001	0.001	0.001
GloVe dim. red. 1×	0.001	0.001	0.001	0.001	0.001	0.001	0.001
GloVe dim. red. 1.5 ×	0.001	0.001	0.001	0.001	0.001	0.001	0.001
GloVe dim. red. 3×	0.0001	0.001	0.001	0.001	0.001	0.001	0.001
GloVe dim. red. 6×	0.001	0.001	0.001	0.001	0.001	0.001	0.001
fastText uncompressed	0.001	0.001	0.001	0.001	0.001	0.001	0.001
fastText dim. red. 1×	0.0001	0.001	0.001	0.001	0.001	0.001	0.001
fastText dim. red. 1.5 ×	0.0001	0.001	0.001	0.001	0.001	0.001	0.001
fastText dim. red. 3×	0.0001	0.001	0.001	0.001	0.001	0.001	0.001
fastText dim. red. 6×	0.001	0.0001	0.001	0.001	0.001	0.001	0.001

GLUE Tasks

We tune the learning rate for each of the GLUE tasks using the grid {10⁻⁵, 2 × 10⁻⁵, 3 × 10⁻⁵, 5 × 10⁻⁵, 10⁻⁴}. When tuning the learning rate, we use the uncompressed WordPiece embeddings, and we fine-tune the entire model, without freezing the embedding parameters. For each task we pick the learning rate which gives the best average performance (according to the metrics in Table 5) on the development set, across five random seeds. The optimal learning rates are listed in Table 10 for all the GLUE tasks we run. We use the default values (from both the Google Research TensorFlow BERT repository²¹ and the Hugging Face PyTorch BERT repository²²) for the other hyperparameters. Specifically, we fine-tune the model for 3 epochs using the Adam optimizer with a mini-batch size of 32, and a weight decay strength of 0.01 (weight decay is not applied to the layer norm layers or to the bias parameters). We use a linear learning rate warm-up for the first 10% of training (learning rate grows linearly from 0× to 1× the specified learning rate), and then a linear learning rate decay for the remaining 90% of training (learning rate decays linearly from 1× to 0× the specified learning rate).

Table 10:

The optimal learning rate η for different GLUE tasks

Tasks	MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE
η	3 × 10−5	3 × 10−5	3 × 10−5	3 × 10−5	10−5	2 × 10−5	2 × 10−5	2 × 10−5

D.5. Infrastructure Details

We run our experiments using AWS p2.xlarge instances, which have NVIDIA Tesla K80 GPUs. We use Python 3.6 for our experiments. For compatibility with the DrQA repository (which had not been ported to PyTorch 1.0 when we began our experiments), we use PyTorch 0.3.1 for the question answering and sentiment analysis tasks. For the GLUE tasks we use PyTorch 1.0.

E. Extended Empirical Results

We now provide a more complete version of the empirical results included in the main body of the paper, as well as a number of additional experiments validating claims related to our work. More specifically:

• In Appendix E.1 we present extended results comparing the downstream performance of the different compression methods across a range of compression rates for the GloVe, fastText, and BERT WordPiece embeddings, on question answering, sentiment analysis, and GLUE tasks. We show that uniform quantization can consistently match or outperform the other compression methods across these settings.

• In Appendix E.2 we present experiments comparing the performance of the different compression methods when applied to compressing task-specific embeddings which have been trained end-to-end for a translation task. Though our main focus in this paper is compressing task-agnostic embeddings (e.g., GloVe, fastText), we show that uniform quantization can effectively compete with a recently proposed tensorized factorization [17] of the embedding matrix designed for the task-specific setting.

• In Appendix E.3 we study whether, under a fixed memory budget, it is better to use low-dimensional high-precision embeddings, or high-dimensional low-precision embeddings. We show that under a wide range of memory budgets, one can attain large improvements in downstream performance on the SQuAD question answering task by using high-dimensional low-precision embeddings in place of lower dimensional high-precision embeddings.

• In Appendix E.4 we present extended results comparing the eigenspace overlap scores of the different compression methods, for different compression rates and embeddings types. We show that uniform quantization can attain comparable or higher eigenspace overlap scores relative to the other compression methods, helping to explain the strong empirical performance of this compression method.

• In Appendix E.5 we present extended results on the correlations between downstream performance and the different measures of compression quality. We show that across the question answering, sentiment analysis, and GLUE tasks we consider, the eigenspace overlap score consistently attains higher Spearman correlation with downstream performance than the other measures of compression quality (PIP loss, Δ, Δ_max).

• In Appendix E.6 we show that the eigenspace overlap score also correlates better with downstream performance than the Δ₁ and Δ₂ compression quality metrics, across a range of tasks.

• In Appendix E.7 we show that our claim that the eigenspace overlap score correlates better with downstream performance than Δ and Δ_max is robust to the choice of the parameter λ used when computing the values of Δ and Δ_max.

• In Appendix E.8 we show that the eigenspace overlap score is a more robust selection criterion for choosing between pairs of compressed embeddings than the other measures of compression quality.

• In Appendix E.9 we compare the downstream performance of embeddings compressed using deterministic vs. stochastic uniform quantization. We show these methods perform similarly, though the deterministic quantization performs slightly better at precision b = 1.

We present all these results in more detail below.

E.1. Downstream Performance vs. Compression Rate: Pre-Trained Embeddings

In Figures 6 (GloVE), 7 (fastText), and 8 (BERT), we show the downstream performance of the embeddings compressed using different compression methods, across question answering, sentiment analysis, and GLUE tasks. We show that the simple uniform quantization method can match or outperform the other compression methods across these tasks. We also observe that for the GLUE tasks (Figure 8), freezing the WordPiece embeddings during the BERT model fine-tuning does not observably hurt downstream performance.

Figure 6: Downstream performance vs. compression rate for compressed GloVe embeddings.

We evaluate the downstream performance of the different compression methods on question answering and sentiment analysis tasks, across different compression rates. For question answering, we use the SQuAD dataset, and for sentiment analysis we use the MR, SST-1, SST-2, Subj, TREC, CR and MPQA datasets. We show average performance across five random seeds, with error bars indicating standard deviations.

Figure 7: Downstream performance vs. compression rate for compressed fastText embeddings.

We evaluate the downstream performance of the different compression methods on question answering and sentiment analysis tasks, across different compression rates. For question answering, we use the SQuAD dataset, and for sentiment analysis we use the MR, SST-1, SST-2, Subj, TREC, CR and MPQA datasets. We show average performance across five random seeds, with error bars indicating standard deviations.

Figure 8: Downstream performance vs. compression rate for compressed BERT WordPiece embeddings.

We evaluate the downstream performance of the different compression methods on all GLUE tasks except WNLI (as discussed in Appendix D.1), across different compression rates. In these plots, the horizontal dashed pink line marks the performance of the BERT model fine-tuned with uncompressed and unfrozen WordPiece embeddings. We show average performance across five random seeds, with error bars indicating standard deviations.

E.2. Downstream Performance vs. Compression Rate: Task-Specific Embeddings

The main focus of our work is on understanding the downstream performance of NLP models trained using compressed pre-trained word embeddings. Recently, Khrulkov et al. [17] proposed compressing word embedding matrices by parameterizing them as a product of tensors, and then learning the entries of these tensors jointly with the downstream NLP model in a task-specific, end-to-end fashion; they call this method a Tensor Train (TT) decomposition of the embedding matrix. In this section, we show that we can apply uniform quantization to compressing task-specific word embeddings, and attain competitive downstream performance with the TT method.

Task details

We consider the IWSLT’14 German-to-English translation task [4]. We use a six-layer Transformer [37] based translation model for this task, and use the Fairseq [26] implementation of this model. In our experiments, across all compression rates and compression methods, we train for 50000 steps, and use the same model size with a 512-dimensional transformer hidden layer; thus, the uncompressed embeddings are 512 dimensional. We use the default training and inference hyperparameters for this German-to-English translation task in the Fairseq repository; we list the values of these hyperparameters in Table 12. To be compatible with the Fairseq implementation, we run these experiments using PyTorch 1.0.

Table 12:

The hyperparameters we use for our experiments on the IWSLT’14 German-to-English translation task.

Hyperparameter	Value
Optimizer	Adam
Adam decay rates for 1st moment β1	0.9
Adam decay rates for 2nd moment β2	0.999
Adam ϵ	10−8
Training steps	50000
Learning rate schedule	$$\begin{gathered} {10}^{-7}+\left(5*{10}^{-4}-{10}^{-7}\right)n/4000\text{for step}n<=4000 \\ 5*{10}^{-4}*\sqrt{4000/n}\text{for step}n>4000 \end{gathered}$$
Warmup initial learning rate	10−7
Dropout	0.3
Weight decay	0.0001
Beam search width	5
Transformer hidden dimension	512

Compression method details

We now provide details on how we apply the different compression methods in this task-specific setting. Note that because TT can achieve compression rates greater than 32×, we run experiments both above and below this compression rate.

• Dimensionality reduction: We randomly initialize lower-dimensional embeddings, and train the parameters of these embeddings jointly with the rest of the model.

• Uniform quantization: We jointly train the full-precision embedding matrix and the transformer model for the first half of the training steps; we then compress this embedding matrix with uniform quantization (Algorithm 1), and keep the embedding parameters fixed for the remainder of training. To attain a compression rate c > 32, we perform the first half of training using lower-dimensional embeddings (compression rate c/32), and then apply uniform quantization to these lower-dimensional embeddings with compression rate 32.

• K-means: We use the same protocol as we do for uniform quantization, but apply the k-means compression method in place of uniform quantization.

• DCCL: As we do for uniform quantization and k-means, we jointly train the full-precision embedding matrix and the transformer model for the first half of the training steps; we then compress the embeddings with DCCL, and perform the rest of training with the embedding parameters fixed. We grid search the dictionary size k ∈ {2, 4, 8, 16, 32, 64} and the learning rate η ∈ {0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.01} for DCCL, and pick the combination of values which minimizes the embedding reconstruction error with respect to the embeddings generated in the first half of training. We show the optimal hyperparameters for each compression rate in Table 11.

• Tensor Train: We use the TT method in the manner described in the original paper [17]. For each compression rate, there are two hyperparameters that must be tuned—the number of tensor factors and the “TT-rank” of these factors. We consider 3 and 4 as the number of factors, following the values used in the paper [17], and pick the one which gives the lowest validation perplexity. Given the number of factors, the TT-rank of these factors is automatically determined for a given compression rate.

Table 11:

The optimal learning rates η and dictionary sizes k for DCCL for compressing the task-specific embeddings for the IWSLT’14 translation task.

Compression rate	8×	16×	32×	64×	128×	256×
k	16	16	4	4	4	2
η	0.0003	0.0003	0.001	0.001	0.001	0.001

Results

In Figure 10 we plot the average test BLEU4 score across five random seeds for the compression methods described above, at a wide range of compression rates; because for some random seeds the TT method attains very low BLEU scores, for the TT method we plot the BLEU4 score of the seed which performs best. We observe that the uniform quantization and k-means methods generally achieve better BLEU4 score than the TT method up to compression rate 128×, and that the dimensionality reduction method performs significantly worse than the other methods beyond compression rate 8×. These observations suggest that uniform quantization and k-means can be effectively applied to compress task-specific embeddings.

Figure 10: Dimension vs. precision trade-off.

We plot the downstream performance on question answering (SQuAD, left) and sentiment analysis (SST-2, right) of GloVe embeddings of dimensions d ∈ {25, 50, 100, 200, 400} compressed with uniform quantization with precisions b ∈ {1, 2, 4, 8, 16, 32}, as a function of the memory occupied by the embeddings. We observe that embeddings compressed with 1-bit precision typically demonstrate the best downstream performance under a range of memory budgets. We show average performance across five random seeds, with error bars indicating standard deviations.

E.3. Dimension vs. Precision Trade-Off

We show that in the memory constrained setting, using low-precision high-dimensional embeddings typically outperforms using high-precision low-dimensional embeddings which occupy the same memory. To demonstrate this, we train GloVe embeddings (details below) of dimensions d ∈ {25, 50, 100, 200, 400}, and then compress each of these embeddings using uniform quantization with precisions b ∈ {1, 2, 4, 8, 16, 32} (32 bits represents no compression). We then train DrQA models [5] using all of these embeddings on the SQuAD dataset [33], and CNN models [19] on the SST-1 sentiment analysis dataset. In Figure 10 we present the downstream performance of all of these models (y-axis) in terms of the memory occupied by the embeddings (x-axis). As we can see, across a range of memory budgets, it is optimal to use low-precision (1 bit) high-dimensional embeddings, as this allows for using the largest dimension possible under that memory budget.

GloVe embedding training details

We train GloVe embeddings on a full English Wikimedia dump on December 4, 2017 which was pre-processed by a fastText script ²³ while keeping the letter cases and digits. We use the GloVe Github repository²⁴ for embedding training. We use a vocabulary size of 400000, a window size of 15, a learning rate of 0.05, and train for 50 epochs.

E.4. Eigenspace Overlap Score vs. Compression Rate

In Figure 11, we plot the eigenspace overlap scores attained by the different compression methods at different compression rates for GloVe, fastText, and BERT WordPiece embeddings. We observe that uniform quantization consistently attains higher or matching eigenspace overlap scores than the other compression methods. Based on the theoretical connection between the eigenspace overlap score and downstream performance, this empirical observation helps explain the strong downstream performance of embeddings compressed with uniform quantization.

Figure 11: Eigenspace overlap score vs. compression rate.

We plot the eigenspace overlap scores attained by different compression methods at different compression rates, for Glove, fastText, and BERT WordPiece embeddings. Across embedding types and compression rates, we observe that uniform quantization attains similar or higher eigenspace overlap scores than the other compression methods. We show average eigenspace overlap scores across five random seeds, with error bars indicating standard deviations.

E.5. Downstream Performance vs. Measures of Compression Quality

We show across tasks and embedding types that the eigenspace overlap score correlates better with downstream performance than the other measures of compression quality. In Figures 12, 13, and 14, we plot the downstream performance (y-axis) of the compressed Glove, fastText, and BERT WordPiece embeddings (respectively) on a variety of tasks, as a function of the different measures of compression quality (x-axis). For GloVe and fastText, we show performance on question answering (SQuAD) and on the largest sentiment analysis dataset (SST-1). For BERT, we show performance on MNLI and QQP, the two largest GLUE datasets. We see in these plots that the eigenspace overlap score generally aligns quite well with downstream performance, while the other measures of compression quality often do not. To quantify this observation, we measure the Spearman correlations between the downstream performances of the embeddings we compressed, and the various measures of compression quality. We include these correlations for all the sentiment analysis tasks for the Glove and fastText embeddings in Table 13, and for all the GLUE tasks for the BERT WordPiece embeddings in Table 14. From these results, we can see that across different tasks and embedding types, the eigenspace overlap score generally correlates better with downstream performance than the other measure of compression quality.

Figure 12: Downstream performance vs. measures of compression quality (GloVE embeddings).

We plot the performance of compressed GloVe embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the different measures of compression quality for these embeddings. We can see that the eigenspace overlap score $\mathcal{E}$ generally aligns better with downstream performance than the other measures of compression quality. To quantify this, in the title of each plot we include the Spearman correlation ρ between downstream performance and the measure of compression quality for that plot. We can see that the eigenspace overlap score attains the strongest correlations with downstream performance, as it has the largest values for |ρ|.

Figure 13: Downstream performance vs. measures of compression quality (fastText embeddings).

We plot the performance of compressed fastText embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the different measures of compression quality for these embeddings. We can see that the eigenspace overlap score $\mathcal{E}$ generally aligns better with downstream performance than the other measures of compression quality. To quantify this, in the title of each plot we include the Spearman correlation ρ between downstream performance and the measure of compression quality for that plot. We can see that the eigenspace overlap score attains the strongest correlations with downstream performance, as it has the largest values for |ρ|.

Figure 14: Downstream performance vs. measures of compression quality (BERT WordPiece embeddings).

We plot the performance of compressed BERT WordPiece embeddings on the two largest GLUE datasets (MNLI, left column; QQP, right column) in terms of the different measures of compression quality for these embeddings. We can see that the eigenspace overlap score $\mathcal{E}$ generally aligns better with downstream performance than the other measures of compression quality. To quantify this, in the title of each plot we include the Spearman correlation ρ between downstream performance and the measure of compression quality for that plot. We can see that the eigenspace overlap score attains the strongest correlations with downstream performance, as it has the largest values for |ρ|.

Table 13: Spearman correlations ρ between compression quality measures and sentiment analysis performance.

Within each entry of the table, the correlations are presented in terms of ‘GloVe |ρ| / fastText |ρ|.’ Note that we present the absolute values of the correlation coefficients, with higher absolute values indicating stronger correlation.

	MR	SST-1	SST-2	Subj	TREC	CR	MPQA
PIP loss	0.36/0.03	0.46/0.25	0.29/0.21	0.30/0.13	0.14/0.05	0.10/0.03	0.49/0.18
Δ	0.29/0.39	0.33/0.29	0.39/0.40	0.26/0.16	0.33/0.29	0.11/0.34	0.41/0.40
Δmax	0.39/0.41	0.51/0.60	0.41/0.62	0.32/0.49	0.23/0.30	0.12/0.40	0.60/0.39
1 − $\mathcal{E}$	0.29/0.59	0.75/0.73	0.72/0.83	0.27/0.58	0.49/0.32	0.40/0.55	0.60/0.55

Table 14: Spearman correlations ρ between compression quality measures and GLUE task performance.

Note that we present the absolute values of the correlation coefficients, with higher absolute values indicating stronger correlation.

	MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE
PIP loss	0.45	0.45	0.43	0.18	0.32	0.41	0.28	0.22
Δ	0.44	0.36	0.36	0.25	0.25	0.43	0.23	0.12
Δmax	0.86	0.86	0.85	0.67	0.75	0.84	0.59	0.58
1 − $\mathcal{E}$	0.92	0.93	0.92	0.83	0.86	0.87	0.62	0.66

E.6. Downstream Performance vs. 1/(1 − Δ₁) and Δ₂

We show two main results: First, we show examples of compressed embeddings that have large values of $\frac{1}{1-{\mathrm{\Delta }}_1}\text{or}{\mathrm{\Delta }}_2,$ but which still attain strong downstream performance; because large values of $\frac{1}{1-{\mathrm{\Delta }}_1}\text{or}{\mathrm{\Delta }}_2$ imply large worst-case bounds on the generalization error of the embeddings [42], these observations demonstrate that the worst-case bounds are too loose to explain the empirical results. Second, we show that the eigenspace overlap score generally attains stronger correlation with downstream performance than both 1/(1 − Δ₁) and Δ₂.

For the first result, we can see in Figure 15 that there are points with large $\frac{1}{1-{\mathrm{\Delta }}_1},$ for example, but where the downstream performance is still quite close to the full-precision embedding performance. For the second result, we show in Table 15 that the eigenspace overlap score attains higher Spearman correlation with downstream performance than 1/(1 − Δ₁) and Δ₂ across a range of tasks.

Figure 15: Downstream performance vs. 1/(1 − Δ₁) and Δ₂ (GloVe embeddings).

We plot the performance of compressed GloVe embeddings on question answering (SQuAD, left column) and sentiment analysis (SST-1, right column), in terms of the 1/(1 − Δ₁) and Δ₂ measures of compression quality. We can see that there are compressed embeddings with large values $\frac{1}{1-{\mathrm{\Delta }}_1},$ but where the downstream performance is still quite close to the full-precision embedding performance. Additionally, we can see that from a visual perspective, these compression quality measures do not align very well with downstream performance. For example, the dimensionality reduction embeddings with compression ratio 6× attain smaller 1/(1 − Δ₁), but worse F1 score on SQuAD, than uniform quantization embeddings with compression ration 32×.

Table 15: Spearman correlations between compression quality measures and downstream performance.

On the SQuAD (question answering), SST-1 (sentiment analysis), MNLI (natural language inference), and QQP (question pair matching) tasks, the eigenspace overlap score $\mathcal{E}$ attains higher Spearman correlation (absolute value) with downstream performance than 1/(1 − Δ₁) and Δ₂.

Dataset	SQuAD		SST-1		MNLI	QQP
Embedding	GloVe	fastText	GloVe	fastText	BERT WordPiece	BERT WordPiece
1/(1 − Δ1)	0.62	0.80	0.52	0.65	0.87	0.87
Δ2	0.46	0.48	0.33	0.44	0.30	0.20
1 − $\mathcal{E}$	0.81	0.91	0.75	0.73	0.92	0.93

E.7. Downstream Performance vs. Δ_max and Δ with different λ values

In Section 4, we showed across numerous tasks and embedding types that the eigenspace overlap score typically attains stronger correlation with downstream performance than the other measures of compression quality, including Δ_max and Δ. For these results, we computed Δ_max and Δ with the parameter λ being the smallest non-zero eigenvalue of the Gram matrix of the uncompressed embeddings (see Section 2.2 for a review of how λ is used when calculating these measures). We now show these results are robust to the choice of λ. Specifically, in Table 16 we show the Spearman correlations attained by Δ_max and Δ with different λ values. Letting λ_min and λ_max be the smallest and largest eigenvalues of the uncompressed embedding Gram matrix, we consider $\text{λ}\in \{{\text{λ}}_{\min }/100,{\text{λ}}_{\min }/10,{\text{λ}}_{\min },{\text{λ}}_{\min }\times 10,{\text{λ}}_{\min }\times 100,{\text{λ}}_{\max }\}$ for this table. We observe that the eigenspace overlap score attains stronger correlation with downstream performance across the tasks and embedding types in this table than Δ and Δ_max, across all the λ values listed above.

Table 16: Spearman correlations between Δ and Δ_max and downstream performance, for different λ values.

On the SQuAD (question answering), SST-1 (sentiment analysis), MNLI (natural language inference), and QQP (question pair matching) tasks, the eigenspace overlap score $\mathcal{E}$ attains higher Spearman correlation (absolute value) with downstream performance than Δ_max and Δ computed with different λ values.

Dataset		SQuAD		SST-1		MNLI	QQP
Embedding		GloVe	fastText	GloVe	fastText	BERT WordPiece	BERT WordPiece
Δmax,	λ = λmin/100	0.66	0.71	0.54	0.63	0.47	0.56
Δmax,	λ = λmin/10	0.65	0.73	0.54	0.61	0.47	0.57
Δmax,	λ = λmin	0.62	0.72	0.51	0.60	0.38	0.56
Δmax,	λ = λmin × 10	0.61	0.65	0.53	0.51	0.35	0.43
Δmax,	λ = λmin × 100	0.25	0.49	0.18	0.43	0.13	0.36
Δmax,	λ = λmax	0.15	0.08	0.22	0.03	0.49	0.08
Δ,	λ = λmin/100	0.41	0.31	0.27	0.30	0.51	0.05
Δ,	λ = λmin/10	0.41	0.32	0.27	0.30	0.51	0.05
Δ,	λ = λmin	0.46	0.31	0.33	0.29	0.57	0.04
Δ,	λ = λmin × 10	0.42	0.00	0.28	0.05	0.55	0.28
Δ,	λ = λmin × 100	0.70	0.32	0.60	0.27	0.87	0.26
Δ,	λ = λmax	0.35	0.01	0.31	0.03	0.10	0.02
	1 − $\mathcal{E}$	0.81	0.91	0.75	0.73	0.92	0.93

E.8. The Robustness of the Measures of Compression Quality as Selection Criteria

In Section 4.3 we argued that the eigenspace overlap score is a more accurate and robust selection criterion for choosing between compressed embeddings than the other measures of compression quality. We showed in Table 2 the selection error rates attained by the various measures of compression quality across different tasks and embeddings types. Here we provide detailed results on the robustness of the various measures of compression quality when used as selection criteria. To quantify the robustness of each measure of compression quality as a selection criterion, we measure for each task the maximum difference in performance, across all pairs of compressed embeddings from our experiments, between the embedding which performs best and the one which is selected by the measure of compression quality. We report these results in Table 17 for GloVe and fastText embeddings on the question answering (SQuAD) and sentiment analysis (SST-1) tasks, and for BERT WordPiece embeddings on the language infernece (MNLI) and question pair classification (QQP) tasks. We observe that the eigenspace overlap score can attain 1.1× to 5.5× lower maximum performance differences than the next best measures of compression quality.

Table 17: The robustness of each measure of compression quality as a selection criterion.

Across all pairs of compressed embeddings from our experiments, we measure for each task the maximum difference in performance between the embedding selected by each measure of compression quality and the one which performs best on the task. We report these results in the table below, and observe that the eigenspace overlap score $\mathcal{E}$ attains lower maximum performance differences than the other measures of compression quality.

Dataset	SQuAD		SST-1		MNLI	QQP
Embedding	GloVe	fastText	GloVe	fastText	BERT WordPiece	BERT WordPiece
PIP loss	0.03	0.08	0.11	0.08	0.04	0.02
Δmax	0.03	0.03	0.11	0.05	0.02	0.02
Δ	0.03	0.08	0.11	0.09	0.03	0.02
1 − $\mathcal{E}$	0.01	0.01	0.04	0.03	0.01	0.01

E.9. Stochastic vs. Deterministic Uniform Quantization

Thus far, all the uniform quantization experiments we have presented on question answering, sentiment analysis, and GLUE tasks have used deterministic rounding. However, our theoretical analysis on the expected eigenspace overlap score of uniformly quantized embeddings assumed unbiased stochastic quantization is used. In this section, we show that (1) stochastic and deterministic uniform quantization perform similarly on downstream tasks, and that (2) the eigenspace overlap score still correlates well with downstream performance when using stochastic quantization instead of deterministic quantization. In Figure 16, we compare the downstream performance of deterministic and stochastic quantization on the SQuAD question answering task and on the SST-1 sentiment analysis task. We can observe that uniform and deterministic quantization perform similarly, although at 1-bit precision deterministic quantization performs slightly better than stochastic quantization. We then show in Figure 17 that regardless of whether we use deterministic or stochastic quantization, the eigenspace overlap score correlates better with downstream performance across compression methods than the other measures of compression quality.

Figure 16: Downstream performance vs. compression rate for deterministic vs. stochastic uniform quantization.

We can observe, using compressed GloVe embeddings on both the SQuAD question answering task and the SST-1 sentiment analysis task, that stochastic uniform quantization (left plots) performs similarly to deterministic uniform quantization (right plots).

Figure 17: Downstream performance vs. measures of compression quality for deterministic vs. stochastic uniform quantization.

We can see that regardless of whether stochastic (left plots) or deterministic (right plots) quantization is used, the eigenspace overlap score correlates better with downstream performance than the other measures of compression quality (as quantified by the Spearman correlations ρ in the plot titles).