Training Complex Models with Multi-Task Weak Supervision

Abstract

As machine learning models continue to increase in complexity, collecting large hand-labeled training sets has become one of the biggest roadblocks in practice. Instead, weaker forms of supervision that provide noisier but cheaper labels are often used. However, these weak supervision sources have diverse and unknown accuracies, may output correlated labels, and may label different tasks or apply at different levels of granularity. We propose a framework for integrating and modeling such weak supervision sources by viewing them as labeling different related sub-tasks of a problem, which we refer to as the multi-task weak supervision setting. We show that by solving a matrix completion-style problem, we can recover the accuracies of these multi-task sources given their dependency structure, but without any labeled data, leading to higher-quality supervision for training an end model. Theoretically, we show that the generalization error of models trained with this approach improves with the number of unlabeled data points, and characterize the scaling with respect to the task and dependency structures. On three fine-grained classification problems, we show that our approach leads to average gains of 20.2 points in accuracy over a traditional supervised approach, 6.8 points over a majority vote baseline, and 4.1 points over a previously proposed weak supervision method that models tasks separately.

1. Introduction

One of the greatest roadblocks to using modern machine learning models is collecting hand-labeled training data at the massive scale they require. In real-world settings where domain expertise is needed and modeling goals change frequently, hand-labeling training sets is prohibitively slow, expensive, and static. For these reasons, practitioners are increasingly turning to weak supervision techniques wherein noisier, often programmatically-generated labels are used instead. Common weak supervision sources include external knowledge bases [24; 37; 8; 31], heuristic patterns [14; 27], feature annotations [23; 36], and noisy crowd labels [17; 11]. The use of these sources has led to state-of-the-art results in a range of domains [37; 35]. A theme of weak supervision is that using the full diversity of available sources is critical to training high-quality models [27; 37].

The key technical difficulty of weak supervision is determining how to combine the labels of multiple sources that have different, unknown accuracies, may be correlated, and may label at different levels of granularity. In our experience with users in academia and industry, the complexity of real world weak supervision sources makes this integration phase the key time sink and stumbling block. For example, if we are training a model to classify entities in text, we may have one available source of high-quality but coarse-grained labels (e.g. “Person” vs. “Organization”) and one source that provides lower-quality but finer-grained labels (e.g. “Doctor” vs. “Lawyer”); moreover, these sources might be correlated due to some shared component or data source [2; 33]. Handling such diversity requires addressing a core technical challenge: estimating the unknown accuracies of multi-granular and potentially correlated supervision sources without any labeled data.

To overcome this challenge, we propose MeTaL, a framework for modeling and integrating weak supervision sources with different unknown accuracies, correlations, and granularities. In MeTaL, we view each source as labeling one of several related sub-tasks of a problem—we refer to this as the multi-task weak supervision setting. We then show that given the dependency structure of the sources, we can use their observed agreement and disagreement rates to recover their unknown accuracies. Moreover, we exploit the relationship structure between tasks to observe additional cross-task agreements and disagreements, effectively providing extra signal from which to learn. In contrast to previous approaches based on sampling from the posterior of a graphical model directly [28; 2], we develop a simple and scalable matrix completion-style algorithm, which we are able to analyze by applying strong matrix concentration bounds [32]. We use this algorithm to learn and model the accuracies of diverse weak supervision sources, and then combine their labels to produce training data that can be used to supervise arbitrary models, including increasingly popular multi-task learning models [5; 29].

Compared to previous methods which only handled the single-task setting [28; 27], and generally considered conditionally-independent sources [1; 11], we demonstrate that our multi-task aware approach leads to average gains of 4.1 points in accuracy in our experiments, and has at least three additional benefits. First, many dependency structures between weak supervision sources may lead to non-identifiable models of their accuracies, where a unique solution cannot be recovered. We provide a compiler-like check to establish identifiability—i.e. the existence of a unique set of source accuracies—for arbitrary dependency structures, without resorting to the standard assumption of non-adversarial sources [11], alerting users to this potential stumbling block that we have observed in practice. Next, we provide sample complexity bounds that characterize the benefit of adding additional unlabeled data and the scaling with respect to the user-specified task and dependency structure. While previous approaches required thousands of sources to give non-vacuous bounds, we capture regimes with small numbers of sources, better reflecting the real-world uses of weak supervision we have observed. Finally, we are able to solve our proposed problem directly with SGD, leading to over 100× faster runtimes compared to prior Gibbs-sampling based approaches [28; 26], and enabling simple implementation using libraries like PyTorch.

We validate our framework on three fine-grained classification tasks in named entity recognition, relation extraction, and medical document classification, for which we have diverse weak supervision sources at multiple levels of granularity. We show that by modeling them as labeling hierarchically-related sub-tasks and utilizing unlabeled data, we can get an average improvement of 20.2 points in accuracy over a traditional supervised approach, 6.8 points over a basic majority voting weak supervision baseline, and 4.1 points over data programming [28], an existing weak supervision approach in the literature that is not multi-task-aware. We also extend our framework to handle unipolar sources that only label one class, a critical aspect of weak supervision in practice that leads to an average 2.8 point contribution to our gains over majority vote. From a practical standpoint, we argue that our framework represents an efficient way for practitioners to supervise modern machine learning models, including new multi-task variants, for complex tasks by opportunistically using the diverse weak supervision sources available to them. To further validate this, we have released an open-source implementation of our framework.¹

2. Related Work

Our work builds on and extends various settings studied in machine learning.

Weak Supervision:

We draw motivation from recent work which models and integrates weak supervision using generative models [28; 27; 2] and other methods [13; 18]. These approaches, however, do not handle multi-granularity or multi-task weak supervision, require expensive sampling-based techniques that may lead to non-identifiable solutions, and leave room for sharper theoretical characterization of weak supervision scaling properties. More generally, our work is motivated by a wide range of specific weak supervision techniques, which include traditional distant supervision approaches [24; 8; 37; 15; 31], co-training methods [4], pattern-based supervision [14; 37], and feature-annotation techniques [23; 36; 21].

Crowdsourcing:

Our approach also has connections to the crowdsourcing literature [17; 11], and in particular to spectral and method of moments-based approaches [38; 9; 12; 1]. In contrast, the goal of our work is to support and explore settings not covered by crowdsourcing work, such as sources with correlated outputs, the proposed multi-task supervision setting, and regimes wherein a small number of labelers (weak supervision sources) each label a large number of items (data points). Moreover, we theoretically characterize the generalization performance of an end model trained with the weakly labeled data.

Multi-Task Learning:

Our proposed approach is motivated by recent progress on multi-task learning models [5; 29; 30], in particular their need for multiple large hand-labeled training datasets. We note that the focus of our paper is on generating supervision for these models, not on the particular multi-task learning model being trained, which we seek to control for by fixing a simple architecture in our experiments.

Our work is also related to recent techniques for estimating classifier accuracies without labeled data in the presence of structural constraints [26]. We use matrix structure estimation [22] and concentration bounds [32] for our core results.

3. Programming Machine Learning with Weak Supervision

As modern machine learning models become both more complex and more performant on a range of tasks, developers increasingly interact with them by programmatically generating noisier or weak supervision. These approaches of effectively programming machine learning models have recently been formalized by the following pipeline [28; 27]: First, users provide one or more weak supervision sources, which are applied to unlabeled data to generate a set of noisy labels. These labels may overlap and conflict; we model and combine them via a label model in order to produce a final set of training labels. These labels are then used to train some discriminative model, which we refer to as the end model. This programmatic weak supervision approach can utilize sources ranging from heuristic rules to other models, and in this way can also be viewed as a pragmatic and flexible form of multi-source transfer learning.

In our experiences with users from science and industry, we have found it critical to utilize all available sources of weak supervision for complex modeling problems, including ones which label at multiple levels of granularity. However, this diverse, multi-granular weak supervision does not easily fit into existing paradigms. We propose a formulation where each weak supervision source labels some sub-task of a problem, which we refer to as the multi-task weak supervision setting. We consider an example:

Example 1

A developer wants to train a fine-grained Named Entity Recognition (NER) model to classify mentions of entities in the news (Figure 2). She has a multitude of available weak supervision sources which she believes have relevant signal for her problem—for example, pattern matchers, dictionaries, and pre-trained generic NER taggers. However, it is unclear how to properly use and combine them: some of them label phrases coarsely as PERSON versus ORGANIZATION, while others classify specific fine-grained types of people or organizations, with a range of unknown accuracies. In our framework, she can represent them as labeling tasks of different granularities—e.g. Y₁ = {Person, Org}, Y₂ = {Doctor, Lawyer, N/A}, Y₃ = {Hospital, Office, N/A}, where the label N/A applies, for example, when the type-of-person task is applied to an organization.

Figure 2:

An example fine-grained entity classification problem, where weak supervision sources label three sub-tasks of different granularities: (i) Person vs. Organization, (ii) Doctor vs. Lawyer (or N/A), (iii) Hospital vs. Office (or N/A). The example weak supervision sources use a pattern heuristic and dictionary lookup respectively.

In our proposed multi-task supervision setting, the user specifies a set of structurally-related tasks, and then provides a set of weak supervision sources which are user-defined functions that either label each data point or abstain for each task, and may have some user-specified dependency structure. These sources can be arbitrary black-box functions, and can thus subsume a range of weak supervision approaches relevant to both text and other data modalities, including use of pattern-based heuristics, distant supervision [24], crowd labels, other weak or biased classifiers, declarative rules over unsupervised feature extractors [33], and more. Our goal is to estimate the unknown accuracies of these sources, combine their outputs, and use the resulting labels to train an end model.

4. Modeling Multi-Task Weak Supervision

The core technical challenge of the multi-task weak supervision setting is recovering the unknown accuracies of weak supervision sources given their dependency structure and a schema of the tasks they label, but without any ground-truth labeled data. We define a new algorithm for recovering the accuracies in this setting using a matrix completion-style optimization objective. We establish conditions under which the resulting estimator returns a unique solution. We then analyze the sample complexity of our estimator, characterizing its scaling with respect to the amount of unlabeled data, as well as the task schema and dependency structure, and show how the estimation error affects the generalization performance of the end model we aim to train. Finally, we highlight how our approach handles abstentions and unipolar sources, two critical scenarios in the weak supervision setting.

4.1. A Multi-Task Weak Supervision Estimator

Problem Setup

Let X ∈ $\mathcal{X}$ be a data point and Y = [Y₁, Y₂,...,Y_t]^T be a vector of categorical task labels, Y_i ∈ {1,...,k_i}, corresponding to t tasks, where (X, Y) is drawn i.i.d. from a distribution $\mathcal{D}$ (for a glossary of all variables used, see Appendix A.1).

The user provides a specification of how these tasks relate to each other; we denote this schema as the task structure G_task. The task structure expresses logical relationships between tasks, defining a feasible set of label vectors $\mathcal{Y}$, such that Y ∈ $\mathcal{Y}$. For example, Figure 2 illustrates a hierarchical task structure over three tasks of different granularities pertaining to a fine-grained entity classification problem. Here, the tasks are related by logical subsumption relationships: for example, if Y₂ = DOCTOR, this implies that Y₁ = PERSON, and that Y₃ = N/A, since the task label Y₃ concerns types of organizations, which is inapplicable to persons. Thus, in this task structure, Y = [PERSON, DOCTOR, N/A]^T is in $\mathcal{Y}$ while Y = [PERSON, N/A, HOSPITAL]^T is not. While task structures are often simple to define, as in the previous example, or are explicitly defined by existing resources—such as ontologies or graphs—we note that if no task structure is provided, our approach becomes equivalent to modeling the t tasks separately, a baseline we consider in the experiments.

In our setting, rather than observing the true label Y, we have access to m multi-task weak supervision sources s_i ∈ S which emit label vectors λ_i that contain labels for some subset of the t tasks. Let 0 denote a null or abstaining label, and let the coverage set τ_i ⊆ {1,...,t} be the fixed set of tasks for which the ith source emits non-zero labels, such that ${\mathbf{\lambda }}_i$ ∈ $\mathcal{Y}$_τi. For convenience, we let τ₀ = {1,...,t} so that $\mathcal{Y}$_τ0 = $\mathcal{Y}$. For example, a source from our previous example might have a coverage set τ_i = {1, 3}, emitting coarse-grained labels such as ${\mathbf{\lambda }}_i$ = [PERSON, 0, N/A]^T. Note that sources often label multiple tasks implicitly due to the constraints of the task structure; for example, a source that labels types of people (Y₂) also implicitly labels people vs. organizations (Y₁ = PERSON), and types of organizations (as Y₃ = N/A). Thus sources tailored to different tasks still have agreements and disagreements; we use this additional cross-task signal in our approach.

The user also provides the conditional dependency structure of the sources as a graph G_source = (V, E), where V = {Y, ${\mathbf{\lambda }}_1$, ${\mathbf{\lambda }}_2$,...,${\mathbf{\lambda }}_m$} (Figure 3). Specifically, if (${\mathbf{\lambda }}_i$, ${\mathbf{\lambda }}_j$) is not an edge in G_source, this means that ${\mathbf{\lambda }}_i$ is independent of ${\mathbf{\lambda }}_j$ conditioned on Y and the other source labels. Note that if G_source is unknown, it can be estimated using statistical techniques such as [2]. Importantly, we do not know anything about the strengths of the correlations in G_source, or the sources’ accuracies.

Figure 3:

An example of a weak supervision source dependency graph G_source (left) and its junction tree representation (right), where Y is a vector-valued random variable with a feasible set of values, Y ∈ $\mathcal{Y}$. Here, the output of sources 1 and 2 are modeled as dependent conditioned on Y. This results in a junction tree with singleton separator sets, Y. Here, the observable cliques are O = {${\mathbf{\lambda }}_1$, ${\mathbf{\lambda }}_2$, ${\mathbf{\lambda }}_3$, ${\mathbf{\lambda }}_4$, {${\mathbf{\lambda }}_1$, ${\mathbf{\lambda }}_2$}} ⊂ $\mathcal{C}$.

Our overall goal is to apply the set of weak supervision sources S = {s₁,...,s_m} to an unlabeled dataset $\mathcal{X}$_U consisting of n data points, then use the resulting weakly-labeled training set to supervise an end model $f_w:\mathcal{X}\mapsto \mathcal{Y}$ (Figure 1). This weakly-labeled training set will contain overlapping and conflicting labels, from sources with unknown accuracies and correlations. To handle this, we will learn a label model P_μ(Y|λ), parameterized by a vector of source correlations and accuracies μ, which for each data point X takes as input the noisy labels λ = {${\mathbf{\lambda }}_1$,...,${\mathbf{\lambda }}_m$} and outputs a single probabilistic label vector $\tilde{\mathbf{Y}}.$ Succinctly, given a user-provided tuple ($\mathcal{X}$_U, S, G_source, G_task), our key technical challenge is recovering the parameters μ without access to ground truth labels Y.

Figure 1:

A schematic of the MeTaL pipeline. To generate training data for an end model, such as a multi-task model as in our experiments, the user inputs a task graph G_task defining the relationships between task labels Y₁,…Y_t; a set of unlabeled data points X; a set of multi-task weak supervision sources s_i which each output a vector ${\mathbf{\lambda }}_i$ of task labels for X; and the dependency structure between these sources, G_source. We train a label model to learn the accuracies of the sources, outputting a vector of probabilistic training labels $\tilde{\mathbf{Y}}$ for training the end model.

Modeling Multi-Task Sources

To learn a label model over multi-task sources, we introduce sufficient statistics over the random variables in G_source-Let $\mathcal{C}$ be the set of cliques in G_source, and define an indicator random variable for the event of a clique C ∈ $\mathcal{C}$ taking on a set of values y_C:

$$\psi \left(C,y_C\right)=\mathbb{1}\left\{{\cap }_{i\in C}{V}_i={\left(y_C\right)}_i\right\},$$

where (y_C)_i ∈ $\mathcal{Y}$_τi. We define $\psi (C)\in {\{0,1\}}^{{\mathrm{\Pi }}_{i\in C}}{}^{\left(\left|{\mathcal{Y}}_{{\tau }_i}\right|-1\right)}$ as the vector of indicator random variables for all combinations of all but one of the labels emitted by each variable in clique C—thereby defining a minimal set of statistics—and define ψ(C) accordingly for any set of cliques C ⊆ $\mathcal{C}$. Then $\mu =\mathbb{E}[\psi (\mathcal{C})]$ is the vector of sufficient statistics for the label model we want to learn.

We work with two simplifying conditions in this section. First, we consider the setting where G_source is triangulated and has a junction tree representation with singleton separator sets. If this is not the case, edges can always be added to G_source to make this setting hold; otherwise, we describe how our approach can directly handle non-singleton separator sets in Appendix A.3.3.

Second, we use a simplified class-conditional model of the noisy labeling process, where we learn one accuracy parameter for each label value ${\mathbf{\lambda }}_i$ that each source s_i emits. This is equivalent to assuming that a source may have a different accuracy on each different class, but that if it emits a certain label incorrectly, it does so uniformly over the different true labels Y. This is a more expressive model than the commonly considered one, where each source is modeled by a single accuracy parameter, e.g. in [11; 28], and in particular allows us to capture the unipolar setting considered later on. For further details, see Appendix A.3.4.

Our Approach

The chief technical difficulty in our problem is that we do not observe Y. We overcome this by analyzing the covariance matrix of an observable subset of the cliques in G_source, leading to a matrix completion-style approach for recovering μ. We leverage two pieces of information: (i) the observability of part of Cov [ψ($\mathcal{C}$)], and (ii) a result from [22] which states that the inverse covariance matrix Cov [ψ($\mathcal{C}$)]⁻¹ is structured according to G_source, i.e., if there is no edge between ${\mathbf{\lambda }}_i$ and ${\mathbf{\lambda }}_j$ in G_source, then the corresponding entries are 0.

We start by considering two disjoint subsets of $\mathcal{C}$: the set of observable cliques, O ⊆ $\mathcal{C}$—i.e., those cliques not containing Y—and the separator set cliques of the junction tree, $\mathcal{S}$ ⊆ $\mathcal{C}$. In the setting we consider in this section, $\mathcal{S}$ = {Y} (see Figure 3). We can then write the covariance matrix of the indicator variables for O ∪ $\mathcal{S}$, Cov [ψ(O ∪ $\mathcal{S}$)], in block form, similar to [6], as:

$$\mathbf{\text{Cov}}[\psi (O\cup \mathcal{S})]\equiv \mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_O& {\mathrm{\Sigma }}_{O\mathcal{S}}\\ {\mathrm{\Sigma }}_{O\mathcal{S}}^T& {\mathrm{\Sigma }}_{\mathcal{S}}\end{array}\right]$$ (1)

and similarly define its inverse:

$$K={\mathrm{\Sigma }}^{-1}=\left[\begin{array}{cc}{K}_O& K_{O\mathcal{S}}\\ K_{O\mathcal{S}}^T& K_{\mathcal{S}}\end{array}\right]$$ (2)

Here, ${\mathrm{\Sigma }}_O=\mathbf{\text{Cov}}[\psi (O)]\in {ℝ}^{d_O\times d_O}$ is the observable block of Σ, where $d_O={\sum }_{C\in O}{\prod }_{i\in C}\left(\left|{\mathcal{Y}}_{{\tau }_i}\right|-1\right).\text{Next,}{\mathrm{\Sigma }}_{O\mathcal{S}}=\mathbf{\text{Cov}}[\psi (O),\psi (\mathcal{S})]$ is the unobserved block which is a function of μ, the label model parameters that we wish to recover. Finally, ${\mathrm{\Sigma }}_{\mathcal{S}}=\mathbf{Cov}[\psi (\mathcal{S})]=\mathbf{Cov}[\psi (\mathbf{Y})]$ is a function of the class balance P(Y).

We make two observations about Σ_{$\mathcal{S}$}. First, while the full form of Σ_{$\mathcal{S}$} is the covariance of the |$\mathcal{Y}$| − 1 indicator variables for each individual value of Y but one, given our simplified class-conditional label model, we in fact only need a single indicator variable for Y (see Appendix A.3.4); thus, Σ_s is a scalar. Second, Σ_{$\mathcal{S}$} is a function of the class balance P(Y), which we assume is either known, or has been estimated according to the unsupervised approach we detail in Appendix A.3.5. Thus, given Σ_O and Σ_{$\mathcal{S}$}, our goal is to recover the vector Σ_{O$\mathcal{S}$} from which we can recover μ.

Algorithm 1.

Source Accuracy Estimation for Multi-Task Weak Supervision

Input: Observed labeling rates $\widehat{\mathbb{E}}[\psi (O)]$ and covariance ${\widehat{\mathrm{\Sigma }}}_O;$ class balance $\widehat{\mathbb{E}}[\psi (\mathbf{Y})]$ and variance Σ$\mathcal{S}$; correlation sparsity structure Ω

$$\begin{gathered} \widehat{z}\leftarrow {\mathrm{argmin}}_z{\Vert {\widehat{\mathrm{\Sigma }}}_O^{-1}+z{z}^T\Vert }_{\mathrm{\Omega }} \\ \widehat{c}\leftarrow {\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\left(1+{\widehat{z}}^T{\widehat{\mathrm{\Sigma }}}_O\widehat{z}\right),{\widehat{\mathrm{\Sigma }}}_{O\mathcal{S}}\leftarrow {\widehat{\mathrm{\Sigma }}}_O\widehat{z}/\sqrt{\widehat{c}} \\ {\widehat{\mu }}^{\prime }\leftarrow {\widehat{\mathrm{\Sigma }}}_{O\mathcal{S}}+\widehat{\mathbb{E}}[\psi (\mathbf{Y})]\widehat{\mathbb{E}}[\psi (O)] \end{gathered}$$

return ExpandTied $\left({\widehat{\mu }}^{\prime }\right)$

Applying the block matrix inversion lemma, we have:

$$K_O={\mathrm{\Sigma }}_O^{-1}+c{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1},$$ (3)

where $c={\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1}\in {ℝ}^{+}.\text{Let}z=\sqrt{c}{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}};$ we can then express (3) as:

$$K_O={\mathrm{\Sigma }}_O^{-1}+{zz}^T$$ (4)

The right hand side of (4) consists of an empirically observable term, ${\mathrm{\Sigma }}_O^{-1},$ and a rank-one term, zz^T, which we can solve for to directly recover μ. For the left hand side, we apply an extension of Corollary 1 from [22] (see Appendix A.3.2) to conclude that K_O has graph-structured sparsity, i.e., it has zeros determined by the structure of dependencies between the sources in G_source. This suggests an algorithmic approach of estimating z as a matrix completion problem in order to recover an estimate of μ (Algorithm 1). In more detail: let Ω be the set of indices (i, j) where (K_O)_i,j = 0, determined by G_source, yielding a system of equations,

$$0={\left({\mathrm{\Sigma }}_O^{-1}\right)}_{i,j}+{\left(z{z}^T\right)}_{i,j}\text{for}(i,j)\in \mathrm{\Omega },$$ (5)

which is now a matrix completion problem. Define ||A||_Ω as the Frobenius norm of A with entries not in Ω set to zero; then we can rewrite (5) as ${\Vert {\mathrm{\Sigma }}_O^{-1}+z{z}^T\Vert }_{\mathrm{\Omega }}=0.$ We solve this equation to estimate z, and thereby recover Σ_{O$\mathcal{S}$}, from which we can directly recover the label model parameters μ algebraically.

Checking for Identifiability

A first question is: which dependency structures G_source lead to unique solutions for μ? This question presents a stumbling block for users, who might attempt to use non-identifiable sets of correlated weak supervision sources.

We provide a simple, testable condition for identifiability. Let G_inv be the inverse graph of G_source; note that Ω is the edge set of G_inv expanded to include all indicator random variables ψ($\mathcal{C}$). Then, let M_Ω be a matrix with dimensions |Ω| × d_O such that each row in M_Ω corresponds to a pair (i, j) ∈ Ω with 1’s in positions i and j and 0’s elsewhere.

Taking the log of the squared entries of (5), we get a system of linear equations M_Ωl = q_Ω, where l_i = log $\left(z_i^2\right)$ and $q_{(i,j)}=\log \left({\left({\left({\mathrm{\Sigma }}_O^{-1}\right)}_{i,j}\right)}^2\right).$ Assuming we can solve this system (which we can always ensure by adding sources; see Appendix), we can uniquely recover the $z_i^2,$ meaning our model is identifiable up to sign.

Given estimates of the $z_i^2,$ we can see from (5) that the sign of a single z_i determines the sign of all other z_j reachable from z_i in G_inv. Thus to ensure a unique solution, we only need to pick a sign for each connected component in G_inv. In the case where the sources are assumed to be independent, e.g., [10; 38; 11], it suffices to make the assumption that the sources are on average non-adversarial; i.e., select the sign of the z_i that leads to higher average accuracies of the sources. Even a single source that is conditionally independent from all the other sources will cause G_inv to be fully connected, meaning we can use this symmetry breaking assumption in the majority of cases even with correlated sources. Otherwise, a sufficient condition is the standard one of assuming non-adversarial sources, i.e. that all sources have greater than random accuracy. For further details, see Appendix B.1.

Source Accuracy Estimation Algorithm

Now that we know when a set of sources with correlation structure G_source is identifiable, yielding a unique z, we can estimate the accuracies μ using Algorithm 1. We also use the function ExpandTied, which is a simple algebraic expansion of tied parameters according to the simplified class-conditional model used in this section; see Appendix A.3.4 for details. In Figure 4, we plot the performance of our algorithm on synthetic data, showing its scaling with the number of unlabeled data points n, the density of pairwise dependencies in G_source, and the runtime performance as compared to a prior Gibbs sampling-based approach. Next, we theoretically analyze the scaling of the error $\Vert \widehat{\mu }-{\mu }^{*}\Vert .$

Figure 4:

(Left) Estimation error $\Vert \widehat{\mu }-{\mu }^{*}\Vert$ decreases with increasing n. (Middle) Given G_source, our model successfully recovers the source accuracies even with many pairwise dependencies among sources, where a naive conditionally-independent model fails. (Right) The runtime of MeTaL is independent of n after an initial matrix multiply, and can thus be multiple orders of magnitude faster than Gibbs sampling-based approaches [28].

4.2. Theoretical Analysis: Scaling with Diverse Multi-Task Supervision

Our ultimate goal is to train an end model using the source labels, denoised and combined by the label model $\widehat{\mu }$ we have estimated. We connect the generalization error of this end model to the estimation error of Algorithm 1, ultimately showing that the generalization error scales as $n^{-\frac{1}{2}},$ where n is the number of unlabeled data points. This key result establishes the same asymptotic scaling as traditionally supervised learning methods, but with respect to unlabeled data points.

Let $P_{\widehat{\mu }}(\tilde{\mathbf{Y}}|\mathbf{\lambda })$ be the probabilistic label (i.e. distribution) predicted by our label model, given the source labels λ as input, which we compute using the estimated $\widehat{\mu }.$ We then train an end multi-task discriminative model $f_w:\mathcal{X}\mapsto \mathcal{Y}$ parameterized by w, by minimizing the expected loss with respect to the label model over n unlabeled data points. Let $l(w,X,\mathbf{Y})=\frac{1}{t}{\sum }_{s=1}^t{l}_t\left(w,X,{\mathbf{Y}}_s\right)$ be a bounded multi-task loss function such that without loss of generality l(w, X, Y) ≤ 1; then we minimize the empirical noise aware loss:

$$\widehat{w}={\mathrm{argmin}}_w\frac{1}{n}\sum _{i=1}^n{\mathbb{E}}_{\tilde{\mathbf{Y}}~P_{\widehat{\mu }}(\cdot |\mathbf{\lambda })}\left[l\left(w,X_i,\tilde{\mathbf{Y}}\right)\right],$$ (6)

and let $\tilde{w}$ be the w that minimizes the true noise-aware loss. This minimization can be performed by standard methods and is not the focus of our paper; let the solution $\widehat{w}$ satisfy $\mathbb{E}\left[\Vert \widehat{w}-\tilde{w}{\Vert }^2\right]\le \gamma .$ We make several assumptions, following [28]: (1) that for some label model parameters μ*, sampling $(\mathbf{\lambda },\mathbf{Y})~P_{{\mu }^{*}}(\cdot )$ is the same as sampling from the true distribution, (λ, Y) ~ $\mathcal{D}$; and (2) that the task labels Y_s are independent of the features of the end model given λ sampled from $P_{{\mu }^{*}}(\cdot ),$ that is, the output of the optimal label model provides sufficient information to discern the true label. Then we have the following result:

Theorem 1

Let $\tilde{w}$ minimize the expected noise aware loss, using weak supervision source parameters $\widehat{\mu }$ estimated with Algorithm 1. Let $\widehat{w}$ minimize the empirical noise aware loss with $\mathbb{E}\left[\Vert \widehat{w}-\tilde{w}{\Vert }^2\right]\le \gamma ,w^{*}={\min }_{w}l(w,X,\mathit{Y}),$ and let the assumptions above hold. Then the generalization error is bounded by:

$$\mathbb{E}\left[l(\widehat{w},X,\mathit{Y})-l\left(w^{*},X,\mathit{Y}\right)\right]\le \gamma +4\left|\mathcal{Y}\right|\Vert \widehat{\mu }-{\mu }^{*}\Vert .$$

Thus, to control the generalization error, we must control $\Vert \widehat{\mu }-{\mu }^{*}\Vert ,$ which we do in Theorem 2:

Theorem 2

Let $\widehat{\mu }$ be an estimate of μ* produced by Algorithm 1 run over n unlabeled data points. Let $a\text{ :=}{\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}+{\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}\right)}^2{\text{λ}}_{\mathit{max}}\left(K_O\right)\right)}^{\frac{1}{2}}\mathit{\text{and}}b:=\frac{{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2}{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}.$ Then, we have:

$$\mathbb{E}\left[\Vert \widehat{\mu }-{\mu }^{*}\Vert \right]\le 16(r-1)d_O^2\sqrt{\frac{32\pi }{n}}ab{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\left(3\sqrt{d_O}a{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)+1\right)\left(\kappa \left({\mathrm{\Sigma }}_O\right)+{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)\right).$$

Interpreting the Bound

We briefly explain the key terms controlling the bound in Theorem 2; more detail is found in Appendix B. Our primary result is that the estimation error scales as $n^{-\frac{1}{2}}.$ Next, ${\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right),$ the largest singular value of the pseudoinverse $M_{\mathrm{\Omega }}^{+},$ has a deep connection to the density of the graph G_inv. The smaller this quantity, the more information we have about G_inv, and the easier it is to estimate the accuracies. Next, λ_min(Σ_O), the smallest eigenvalue of the observed covariance matrix, reflects the conditioning of Σ_O; better conditioning yields easier estimation, and is roughly determined by how far away from random guessing the worst weak supervision source is, as well as how conditionally independent the sources are. λ_max(K_O), the largest eigenvalue of the upper-left block of the inverse covariance matrix, similarly reflects the overall conditioning of Σ. Finally, ${\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min },$ the smallest entry of the inverse observed matrix, reflects the smallest non-zero correlation between source accuracies; distinguishing between small correlations and independent sources requires more samples.

4.3. Extensions: Abstentions & Unipolar Sources

We briefly highlight two extensions handled by our approach which we have found empirically critical: handling abstentions, and modeling unipolar sources.

Handling Abstentions.

One fundamental aspect of the weak supervision setting is that sources may abstain from labeling a data point entirely—that is, they may have incomplete and differing coverage [27; 10]. We can easily deal with this case by extending the coverage ranges $\mathcal{Y}$_τi of the sources to include the vector of all zeros, $\overrightarrow{0},$ and we do so in the experiments.

Handling Unipolar Sources.

Finally, we highlight the fact that our approach models class conditional source accuracies, in particular motivated by the case we have frequently observed in practice of unipolar weak supervision sources, i.e., sources that each only label a single class or abstain. In practice, we find that users most commonly use such unipolar sources; for example, a common template for a heuristic-based weak supervision source over text is one that looks for a specific pattern, and if the pattern is present emits a specific label, else abstains. As compared to prior approaches that did not model class-conditional accuracies, e.g. [28], we show in our experiments that we can use our class-conditional modeling approach to yield an improvement of 2.8 points in accuracy.

5. Experiments

We validate our approach on three fine-grained classification problems—entity classification, relation classification, and document classification—where weak supervision sources are available at both coarser and finer-grained levels (e.g. as in Figure 2). We evaluate the predictive accuracy of end models supervised with training data produced by several approaches, finding that our approach outperforms traditional hand-labeled supervision by 20.2 points, a baseline majority vote weak supervision approach by 6.8 points, and a prior weak supervision denoising approach [28] that is not multi-task-aware by 4.1 points.

Datasets

Each dataset consists of a large (3k-63k) amount of unlabeled training data and a small (200–350) amount of labeled data which we refer to as the development set, which we use for (a) a traditional supervision baseline, and (b) for hyperparameter tuning of the end model (see Appendix C). The average number of weak supervision sources per task was 13, with sources expressed as Python functions, averaging 4 lines of code and comprising a mix of pattern matching heuristics, external knowledge base or dictionary lookups, and pre-trained models. In all three cases, we choose the decomposition into sub-tasks so as to align with weak supervision sources that are either available or natural to express.

Named Entity Recognition (NER):

We represent a fine-grained named entity recognition problem—tagging entity mentions in text documents—as a hierarchy of three sub-tasks over the OntoNotes dataset [34]: Y₁ ∈ {Person, Organization}, Y₂ ∈ {Businessperson, Other Person, N/A}, Y₃ ∈ {Company, Other Org, N/A}, where again we use N/A to represent “not applicable”.

Relation Extraction (RE):

We represent a relation extraction problem—classifying entity-entity relation mentions in text documents—as a hierarchy of six sub-tasks which either concern labeling the subject, object, or subject-object pair of a possible or candidate relation in the TACRED dataset [39]. For example, we might label a relation as having a Person subject, Location object, and Place-of-Residence relation type.

Medical Document Classification (Doc):

We represent a radiology report triaging (i.e. document classification) problem from the OpenI dataset [25] as a hierarchy of three sub-tasks: Y₁ ∈ {Acute, Non-Acute}, Y₂ ∈ {Urgent, Emergent, N/A}, Y₃ ∈ {Normal, Non-Urgent, N/A}.

End Model Protocol

Our goal was to test the performance of a basic multi-task end model using training labels produced by various different approaches. We use an architecture consisting of a shared bidirectional LSTM input layer with pre-trained embeddings, shared linear intermediate layers, and a separate final linear layer (“task head”) for each task. Hyperparameters were selected with an initial search for each application (see Appendix), then fixed.

Core Validation

We compare the accuracy of the end multi-task model trained with labels from our approach versus those from three baseline approaches (Table 1):

• Traditional Supervision [Gold (Dev)]: We train the end model using the small hand-labeled development set.

• Hierarchical Majority Vote [MV]: We use a hierarchical majority vote of the weak supervision source labels: i.e. for each data point, for each task we take the majority vote and proceed down the task tree accordingly. This procedure can be thought of as a hard decision tree, or a cascade of if-then statements as in a rule-based approach.

• Data Programming [DP]: We model each task separately using the data programming approach for denoising weak supervision [27].

Table 1: Performance Comparison of Different Supervision Approaches.

We compare the micro accuracy (avg. over 10 trials) with 95% confidence intervals of an end multi-task model trained using the training labels from the hand-labeled development set (Gold Dev), hierarchical majority vote (MV), data programming (DP), and our approach (MeTaL).

	NER	RE	Doc	Average
Gold (Dev)	63.7 ± 2.1	28.4 ± 2.3	62.7 ± 4.5	51.6
MV	76.9 ± 2.6	43.9 ± 2.6	74.2 ± 1.2	65.0
DP [28]	78.4 ± 1.2	49.0 ± 2.7	75.8 ± 0.9	67.7
MeTaL	82.2 ± 0.8	56.7 ± 2.1	76.6 ± 0.4	71.8

In all settings, we used the same end model architecture as described above. Note that while we choose to model these problems as consisting of multiple sub-tasks, we evaluate with respect to the broad primary task of fine-grained classification (for subtask-specific scores, see Appendix). We observe in Table 1 that our approach of leveraging multi-granularity weak supervision leads to large gains—20.2 points over traditional supervision with the development set, 6.8 points over hierarchical majority vote, and 4.1 points over data programming.

Ablations

We examine individual factors:

Unipolar Correction:

Modeling unipolar sources (Sec 4.3), which we find to be especially common when fine-grained tasks are involved, leads to an average gain of 2.8 points of accuracy in MeTaL performance.

Joint Task Modeling:

Next, we use our algorithm to estimate the accuracies of sources for each task separately, to observe the empirical impact of modeling the multi-task setting jointly as proposed. We see average gains of 1.3 points in accuracy (see Appendix).

End Model Generalization:

Though not possible in many settings, in our experiments we can directly apply the label model to make predictions. In Table 6, we show that the end model improves performance by an average 3.4 points in accuracy, validating that the models trained do indeed learn to generalize beyond the provided weak supervision. Moreover, the largest generalization gain of 7 points in accuracy came from the dataset with the most available unlabeled data (n=63k), demonstrating scaling consistent with the predictions of our theory (Fig. 5). This ability to leverage additional unlabeled data and more sophisticated end models are key advantages of the weak supervision approach in practice.

Figure 5:

In the OntoNotes dataset, end model accuracy scales with the amount of available unlabeled data.

6. Conclusion

We presented MeTaL, a framework for training models with weak supervision from diverse, multi-task sources having different granularities, accuracies, and correlations. We tackle the core challenge of recovering the unknown source accuracies via a scalable matrix completion-style algorithm, introduce theoretical bounds characterizing the key scaling with respect to unlabeled data, and demonstrate empirical gains on real-world datasets. In future work, we hope to learn the task relationship structure and cover a broader range of settings where labeled training data is a bottleneck.

Figure 6:

Using the label model (LM) predictions directly versus using an end model trained on them (EM).

A. Problem Setup & Modeling Approach

In Section A, we review our problem setup and modeling approach in more detail, and for more general settings than in the body. In Section B, we provide an overview, additional interpretation, and the proofs of our main theoretical results. Finally, in Section C, we go over additional details of our experimental setup.

We begin in Section A.1 with a glossary of the symbols and notation used throughout this paper. Then, in Section A.2 we present the setup of our multi-task weak supervision problem, and in Section A.3 we present our approach for modeling multi-task weak supervision, and the matrix completion-style algorithm used to estimate the model parameters. Finally, in Section A.4, we present in more detail the subcase of hierarchical tasks considered in the main body of the paper.

A.1. Glossary of Symbols

Table 2:

Glossary of variables and symbols used in this paper.

Symbol	Used for
X	Data point, X ∈ $\mathcal{X}$
n	Number of data points
Ys	Label for one of the t classification tasks, Ys ∈ {1,...,ks}
t	Number of tasks
Y	Vector of task labels Y = [Y1, Y2,...,Yt]T
r	Cardinality of the output space, r = |$\mathcal{Y}$|
G task	Task structure graph
$\mathcal{Y}$	Output space of allowable task labels defined by Gtask, Y ∈ $\mathcal{Y}$
$\mathcal{D}$	Distribution from which we assume (X, Y) data points are sampled i.i.d.
si	Weak supervision source, a function mapping X to a label vector
${\mathbf{\lambda }}_i$	Label vector ${\mathbf{\lambda }}_i$ ∈ $\mathcal{Y}$ output by the ith source for X
m	Number of sources
λ	m × t matrix of labels output by the m sources for X
$\mathcal{Y}$ 0	Source output space, which is $\mathcal{Y}$ augmented to include elements set to zero
τi	Coverage set of ${\mathbf{\lambda }}_i$- the tasks si gives non-zero labels to; for convenience, τ0 = {1,…,t}
$\mathcal{Y}$ τi	The output space for ${\mathbf{\lambda }}_i$ given coverage set τi
${\mathcal{Y}}_{{\tau }_i}^{\text{min}}$	The output space ${\mathcal{Y}}_{{\tau }_i}$ with all but the first value, for defining a minimal set of statistics
G source	Source dependency graph, Gsource = (V, E), V = {Y, ${\mathbf{\lambda }}_1$,...,${\mathbf{\lambda }}_m$}
$\mathcal{C}$	Cliqueset (maximal and non-maximal) of Gsource
$\tilde{\mathcal{C}},\mathcal{S}$	The maximal cliques (nodes) and separator sets of the junction tree over Gsource
Ψ(C, yC)	The indicator variable for the variables in clique C ∈ $\mathcal{C}$ taking on values yC, (yC)i ∈ $\mathcal{Y}$τi
μ	The parameters of our label model we aim to estimate; $\mu =\mathbb{E}[\psi ]$
O	The set of observable cliques, i.e. those corresponding to cliques without Y
Σ	Generalized covariance matrix of O ⊆ $\mathcal{S}$, Σ ≡ Cov [Ψ(O ⊆ $\mathcal{S}$)]
K	The inverse generalized covariance matrix K = Σ−1
dO, d $\mathcal{S}$	The dimensions of O and $\mathcal{S}$ respectively
G aug	The augmented source dependencies graph Gaug = (Ψ, Eaug)
Ω	The edge set of the inverse graph of Gaug
P	Diagonal matrix of class prior probabilities, P(Y)
Pμ (Y, λ)	The label model parameterized by μ
$\tilde{\mathbf{Y}}$	The probabilistic training label, i.e. Pμ(Y|λ)
fw (X)	The end model trained using (X, $\tilde{\mathbf{Y}}$)

A.2. Problem Setup

Let X ∈ $\mathcal{X}$ be a data point and Y = [Y₁, Y₂,...,Y_t]^T be a vector of task labels corresponding to t tasks. We consider categorical task labels, Y_i ∈ {1,..., k_i} for i ∈ {1,...,t}. We assume (X, Y) pairs are sampled i.i.d. from distribution $\mathcal{D}$; to keep the notation manageable, we do not place subscripts on the sample tuples.

Task Structure

The tasks are related by a task graph G_task. Here, we consider schemas expressing logical relationships between tasks, which thus define feasible sets of label vectors $\mathcal{Y}$, such that Y ∈ $\mathcal{Y}$. We let r = |$\mathcal{Y}$| be the number of feasible task vectors. In section A.4, we consider the particular subcase of a hierarchical task structure as used in the experiments section of the paper.

Multi-Task Sources

We now consider multi-task weak supervision sources s_i ∈ S, which represent noisy and potentially incomplete sources of labels, which have unknown accuracies and correlations. Each source s_i outputs label vectors ${\mathbf{\lambda }}_i$, which contain non-zero labels for some of the tasks, such that ${\mathbf{\lambda }}_i$ is in the feasible set $\mathcal{Y}$ but potentially with some elements set to zero, denoting a null vote or abstention for that task. Let $\mathcal{Y}$₀ denote this extended set which includes certain task labels set to zero.

We also assume that each source has a fixed task coverage set τ_i, such that (${\mathbf{\lambda }}_i$)_s ≠ 0 for s ∈ τ_i, and (${\mathbf{\lambda }}_i$)_s = 0 for s ∉ τ_i; let $\mathcal{Y}$_τi ⊆ $\mathcal{Y}$₀ be the range of ${\mathbf{\lambda }}_i$ given coverage set τ_i. For convenience, we let τ₀ = {1,...,t} so that $\mathcal{Y}$_τ0 = $\mathcal{Y}$. The intuitive idea of the task coverage set is that some labelers may choose not to label certain tasks; Example 2 illustrates this notion. Note that sources can also abstain for a data point, meaning they emit no label (which we denote with a symbol $\overrightarrow{0},$); we include this in $\mathcal{Y}$_τi. Thus we have s_i : $\mathcal{X}$ ↦ $\mathcal{Y}$_τi, where, again, ${\mathbf{\lambda }}_i$ denotes the output of the function s_i.

Problem Statement

Our overall goal is to use the noisy or weak, multi-task supervision from the set of m sources, S = {s₁,...,s_m}, applied to an unlabeled dataset $\mathcal{X}$_U consisting of n data points, to supervise an end model $f_w:\mathcal{X}\mapsto \mathcal{Y}$. Since the sources have unknown accuracies, and will generally output noisy and incomplete labels that will overlap and conflict, our intermediate goal is to learn a label model $P_{\mu }:\mathbf{\lambda }\mapsto {[0,1]}^{|\mathcal{Y}|}$ which takes as input the source labels and outputs a set of probabilistic label vectors, $\tilde{\mathbf{Y}},$ for each X, which can then be used to train the end model. Succinctly, given a user-provided tuple ($\mathcal{X}$_U, S, G_source, G_task), our goal is to recover the parameters μ.

The key technical challenge in this approach then consists of learning the parameters of this label model—corresponding to the conditional accuracies of the sources (and, for technical reasons we shall shortly explain, cliques of correlated sources)—given that we do not have access to the ground truth labels Y. We discuss our approach to overcoming this core technical challenge in the subsequent section.

A.3. Our Approach: Modeling Multi-Task Sources

Our goal is to estimate the parameters μ of a label model that produces probabilistic training labels given the observed source outputs, $\tilde{\mathbf{Y}}=P_{\mu }(\mathbf{Y}|\mathbf{\lambda }),$ without access to the ground truth labels Y. We do this in three steps:

1. We start by defining a graphical model over the weak supervision source outputs and the true (latent) variable Y, (${\mathbf{\lambda }}_1$,...,${\mathbf{\lambda }}_m$, Y), using the conditional independence structure G_source between the sources.

2. Next, we analyze the generalized covariance matrix Σ (following Loh & Wainwright [22]), which is defined over binary indicator variables for each value of each clique (or specific subsets of cliques) in G_source. We consider two specific subsets of the cliques in G_source, the observable cliques O and the separator sets $\mathcal{S}$, such that:

   $$\mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_O& {\mathrm{\Sigma }}_{O\mathcal{S}}\\ {\mathrm{\Sigma }}_{O\mathcal{S}}^T& {\mathrm{\Sigma }}_{\mathcal{S}}\end{array}\right]{\mathrm{\Sigma }}^{-1}=K=\left[\begin{array}{cc}{K}_O& K_{O\mathcal{S}}\\ K_{O\mathcal{S}}^T& K_{\mathcal{S}}\end{array}\right],$$

   where Σ_O is the block of Σ that we can observe, and Σ_{O$\mathcal{S}$} is a function of μ, the parameters (corresponding to source and clique accuracies) we wish to recover. We then apply a result by Loh and Wainwright [22] to establish the sparsity pattern of K = Σ⁻¹. This allows us to apply the block-matrix inversion lemma to reformulate our problem as solving a matrix completion-style objective.

3. Finally, we describe how to recover the class balance P(Y); with this and the estimate of μ, we then describe how to compute the probabilistic training labels $\tilde{Y}=P_{\mu }(\mathbf{Y}|\mathbf{\lambda }).$

We start by focusing on the setting where G_source has a junction tree with singleton separator sets; we note that a version of G_source where this holds can always be formed by adding edges to the graph. We then discuss how to handle graphs with non-singleton separator sets, and finally describe different settings where our problem reduces to rank-one matrix completion. In Section B, we introduce theoretical results for the resulting model and provide our model estimation strategy.

Figure 7:

A simple example of a weak supervision source dependency graph G_source (left) and its junction tree representation (right). Here Y is as a vector-valued variable with a feasible set of values, Y ∈ |$\mathcal{Y}$|, and the output of sources 1 and 2 are modeled as dependent conditioned on Y. This results in a junction tree with singleton separator sets Y. Here, the observable cliques are O = {${\mathbf{\lambda }}_1$, ${\mathbf{\lambda }}_2$, ${\mathbf{\lambda }}_3$, ${\mathbf{\lambda }}_4$, {${\mathbf{\lambda }}_1$, ${\mathbf{\lambda }}_2$}} ⊂ $\mathcal{C}$.

A.3.1. Defining a Multi-Task Source Model

We consider a model G_source = (V, E), where V = {Y, ${\mathbf{\lambda }}_1$,...,${\mathbf{\lambda }}_m$}, and E consists of pairwise interactions (i.e. we consider an Ising model, or equivalently, a graph rather than a hypergraph of correlations). We assume that G_source is provided by the user. However, if G_source is unknown, there are various techniques for estimating it statistically [2] or even from static analysis if the sources are heuristic functions [33]. We provide an example G_source with singleton separator sets in Figure 7.

Augmented Sufficient Statistics

Finally, we extend the random variables in V by defining a matrix of indicator statistics over all cliques in G_source, in order to estimate all the parameters needed for our label model P_μ. We assume that the provided G_source is chordal, meaning it has no chordless cycles of length greater than three; if not, the graph can easily be triangulated to satisfy this property, in which case we work with this augmented version.

Let $\mathcal{C}$ be the set of maximal and non-maximal cliques in the chordal graph G_source. We start by defining a binary indicator random variable for the event of a clique C ∈ $\mathcal{C}$ in the graph G_source = (V, E) taking on a set of values y_C:

$$\psi \left(C,y_C\right)=\mathbb{1}\left\{{\cap }_{i\in C}{V}_i={\left(y_C\right)}_i\right\},$$

where ${\left(y_C\right)}_i\in {\mathcal{Y}}_{{\tau }_i}^{\min }\text{and}{\mathcal{Y}}_{{\tau }_i}^{\min }$ contains all but one values of $\mathcal{Y}$_τi, thereby leading to a minimal set of statistics. Note that in our notation, V₀ = Y, $\mathcal{Y}$_τ0 = $\mathcal{Y}$, and V_i>0 = ${\mathbf{\lambda }}_i$. Accordingly, we define $\psi (C)\in \{0,1\}{\prod }_{i\in C}\left(\left|{\mathcal{Y}}_{{\tau }_i}\right|-1\right)$ as the vector of indicator random variables for all combinations of all but one of the labels emitted by each variable in clique C, and define Ψ(C) accordingly for any set of cliques C ⊆ $\mathcal{C}$, Then $\mu =\mathbb{E}[\psi (\mathcal{C})]$ is the vector of sufficient statistics for the label model we want to learn. Our model estimation goal is now stated simply: we wish to estimate μ, without access to the ground truth labels Y.

A.3.2. Model Estimation without Ground Truth Using Inverse Covariance Structure

Our goal is to estimate $\mu =\mathbb{E}[\psi (\mathcal{C})];$ this, along with the class balance P(Y) (which we assume we know, or else estimate using the approach in Section A.3.5), is sufficient information to compute P_μ(Y|λ). If we had access to a large enough set of ground truth labels Y, we could simply take the empirical expectation $\widehat{\mathbb{E}}[\psi ];$ however in our setting we cannot directly observe this. Instead, we proceed by analyzing a sub-block of the covariance matrix of Ψ($\mathcal{C}$), which corresponds to the generalized covariance matrix of our graphical model as in [22], and leverage two key pieces of information:

• A sub-block of this generalized covariance matrix is observable, and

• By a simple extension of Corollary 1 in [22], we know the sparsity structure of the inverse generalized covariance matrix Σ⁻¹, i.e. we know that it will have elements equal to zero according to the structure of G_source.

Since G_source is triangulated, it admits a junction tree representation [19], which has maximal cliques (nodes) $\tilde{\mathcal{C}}$ and separator sets $\mathcal{S}$. Note that we follow the convention that $\mathcal{S}$ includes the full powerset of separator set cliques, i.e. all subset cliques of separator set cliques are also included in $\mathcal{S}$. We proceed by considering two specific subsets of the cliques of our graphical model G_source: those that are observable (i.e. not containing Y), O = {C | Y ∉ C, C ∈ C}, and the set of separator set cliques (which will always contain Y, and thus be unobservable).

For simplicity of exposition, we start by considering graphs G_source which have singleton separator sets; given our graph structure, this means that $\mathcal{S}$ = {{Y}}. Note that in general we will write single-element sets without braces when their type is obvious from context, so we have $\mathcal{S}$ = {Y}. Intuitively, this corresponds to models where weak supervision sources are correlated in fully-connected clusters, corresponding to real-world settings in which sources are correlated due to shared data sources, code, or heuristics. However, we can always either (i) add edges to G_source such that this is the case, or (ii) extend our approach to many settings where G_source does not have singleton separator sets (see Section A.3.3).

In this singleton separator set setting of $\mathcal{S}$ = {Y}, we now have:

$$O=\left\{C\right|\mathbf{Y}\notin C,C\in \mathcal{C}\}\mathcal{S}=\{\mathbf{Y}\}.$$

where Ψ(O) and Ψ(Y) are the corresponding vectors of minimal indicator variables. We define corresponding dimensions d_O and d_{$\mathcal{S}$}:

$$d_O=\sum _{C\in O}\prod _{i\in C}\left(\left|{\mathcal{Y}}_{{\tau }_i}\right|-1\right)d_{\mathcal{S}}=r-1.$$

We now decompose the generalized covariance matrix and its inverse as:

$$\mathbf{\text{Cov}}[\psi (O\cup \mathcal{S})]\equiv \mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_O& {\mathrm{\Sigma }}_{O\mathcal{S}}\\ {\mathrm{\Sigma }}_{O\mathcal{S}}^T& {\mathrm{\Sigma }}_{\mathcal{S}}\end{array}\right]{\mathrm{\Sigma }}^{-1}=K=\left[\begin{array}{cc}{K}_O& K_{O\mathcal{S}}\\ K_{O\mathcal{S}}^T& K_{\mathcal{S}}\end{array}\right],$$ (7)

This is similar to the form used in [6], but with several important differences: we consider discrete (rather than Gaussian) random variables and have additional knowledge of the graph structure. Here, Σ_O is the observable block of the generalized covariance matrix Σ, and Σ_{O$\mathcal{S}$} is the unobserved block which is a function of μ, the parameters (corresponding to source and source clique accuracies) we wish to recover. Note that with the singleton separator sets we are considering, Σ_{$\mathcal{S}$} is a function of the class balance P(Y), which we assume is either known, or has been estimated according to the unsupervised approach we detail in Section A.3.5. Therefore, we assume that Σ_{$\mathcal{S}$} is also known. Concretely then, our goal is to recover Σ_{O$\mathcal{S}$} given Σ_O, Σ_{$\mathcal{S}$}.

We start by applying the block matrix inversion lemma to get the equation:

$$K_O={\mathrm{\Sigma }}_O^{-1}+{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}{\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}.$$ (8)

Next, let $J{J}^T={\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1}.$ We justify this decomposition by showing that this term is positive semidefinite. We start by applying the Woodbury matrix inversion lemma:

$${\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1}={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\left({\mathrm{\Sigma }}_O+{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T\right)}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}.$$ (9)

Now, note that Σ_O and Σ_{$\mathcal{S}$} are both covariance matrices themselves and are therefore PSD. Furthermore, from [22] we know that Σ⁻¹ must exist, which implies that Σ_O and Σ_{$\mathcal{S}$} are invertible (and thus in fact positive definite). Therefore we also have that ${\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T\succ 0\Rightarrow {\left({\mathrm{\Sigma }}_O+{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T\right)}^{-1}\succ 0,$ and therefore (9) is positive definite, and can therefore always be expressed as JJ^T for some J. Therefore, we can write (8) as:

$$K_O={\mathrm{\Sigma }}_O^{-1}+{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}J{J}^T{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}.$$

Finally, define $Z={\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}J;$ we then have:

$$K_O={\mathrm{\Sigma }}_O^{-1}+Z{Z}^T.$$ (10)

Note that $Z\in {ℝ}^{d_O\times d_H},$ where d_H = r − 1, and therefore ZZ^T is a rank-(r − 1) matrix. Therefore, we now have a form (10) that appears close to being a matrix completion-style problem. We complete the connection by leveraging the known sparsity structure of K_O.

Define G_aug = (Ψ, E_aug) to be the augmented version of our graph G_source. In other words, let i = (C₁, y_C1) and j = (C₂, y_C2) according to the indexing scheme of our augmented indicator variables; then, (i, j) ∈ E_aug if C₁, C₂ are subsets of the same maximal clique in G_source. Then, let G_inv-aug = (Ψ, Ω) be the inverse graph of G_aug, such that $(i,j)\in E_{\text{aug}}=>(i,j)\notin \mathrm{\Omega }$ and vice-versa.

We start with a result that extends Corollary 1 in Loh & Wainwright [22] to our specific setting where we consider a set of the variables that contains all observable cliques, O, and all separator sets $\mathcal{S}$ (note that this result holds for all $\mathcal{S}$, not just $\mathcal{S}$ = {Y}):

Corollary 1

Let U = O ∪ $\mathcal{S}$. Let Σ_U be the generalized covariance matrix for U. Then ${\left({\mathrm{\Sigma }}_U^{-1}\right)}_{i,j}=0$ whenever i, j correspond to cliques C₁, C₂ respectively such that C₁, C₂ are not subsets of the same maximal clique.

Proof: We partition the cliques $\mathcal{C}$ into two sets, U and W = $\mathcal{C}$ \ U. Let Σ be the full generalized covariance matrix (i.e. including all maximal and non-maximal cliques) and Γ = Σ⁻¹. Thus we have:

$$\mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_U& {\mathrm{\Sigma }}_{UW}\\ {\mathrm{\Sigma }}_{UW}^T& {\mathrm{\Sigma }}_W\end{array}\right]{\mathrm{\Sigma }}^{-1}=\mathrm{\Gamma }=\left[\begin{array}{cc}{K}_U& K_{UW}\\ K_{UW}^T& K_W\end{array}\right].$$

By the block matrix inversion lemma we have:

$${\mathrm{\Sigma }}_U^{-1}=K_U-K_{UW}{K}_W^{-1}{K}_{UW}^T.$$

We now follow the proof structure of Corollary 1 of [22]. We know K_U is graph structured by Theorem 1 of [22]. Next, using the same argument as in the proof of Corollary 1 of [22], we know that K_w, and therefore $K_W^{-1},$ is block-diagonal. Intuitively, because the set U contains all of the separator set cliques, and due to the running intersection property of a junction tree, each clique in W belongs to precisely one maximal clique-leading to block diagonal structure of K_W. We thus need only to show that the following quantity is zero for two cliques C_i, C_j that are not subsets of the same maximal clique, with corresponding indices i, j:

$${\left(K_{UW}{K}_W^{-1}{K}_{UW}^T\right)}_{i,j}=\sum _B{\left(K_{UW}\right)}_{i,B}{\left(K_W^{-1}\right)}_{B,B}{\left(K_{UW}^T\right)}_{B,j},$$

where B are the indices corresponding to the blocks in $K_W^{-1},$ which correspond to maximal cliques. Our argument follows again as in Corollary 1 of [22]: since U contains the separator sets, if the two cliques C₁, C₂ are not subsets of the same maximal clique, then for each B, either (K_UW)_i,B or ${\left(K_{UW}^T\right)}_{B,j}$ must be zero, completing the proof.

Now, by Corollary 1, we know that K_i,j = 0 if (i, j) ∈ Ω. Let A_Ω denote a matrix A with all entries (i, j) ∉ Ω masked to zero. Then, we have:

$${\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\mathrm{\Omega }}+{\left(Z{Z}^T\right)}_{\mathrm{\Omega }}=0.$$ (11)

Thus, given the dependency graph G_source, we can solve for Z as a rank-(r − 1) matrix completion problem, with mask Ω. Defining the semi-norm $\Vert A{\Vert }_{\mathrm{\Omega }}={\Vert A_{\mathrm{\Omega }}\Vert }_F,$ we can solve:

$$\widehat{Z}={\mathrm{argmin}}_Z{\Vert {\mathrm{\Sigma }}_O^{-1}+Z{Z}^T\Vert }_{\mathrm{\Omega }}.$$ (12)

Now, we have an estimate of Z. Note that at this point, we can only recover Z up to orthogonal transformations. We proceed by considering a reduced rank-one model, detailed in Section A.3.4, and in Section B.1 establish concrete conditions under which this model is uniquely identifiable.

We denote this rank-one setting by switching to writing Z as $z\in {ℝ}^{d_O\times 1},$ in which case we now have:

$$\widehat{z}={\mathrm{argmin}}_z{\Vert {\mathrm{\Sigma }}_O^{-1}+z{z}^T\Vert }_{\mathrm{\Omega }}.$$ (13)

Once we have recovered z uniquely (see Section B.1), we next need to recover ${\mathrm{\Sigma }}_{O\mathcal{S}}=c^{-\frac{1}{2}}{\mathrm{\Sigma }}_{O}z.$ We use the fact that $c={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\left(1+z^T{\mathrm{\Sigma }}_{O}z\right),$ which we can confirm explicitly below, starting from the definition of c:

$$\begin{gathered} c={\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1} \\ ={\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\left(c^{-\frac{1}{2}}{\mathrm{\Sigma }}_{O}z\right)}^T{\mathrm{\Sigma }}_O^{-1}\left(c^{-\frac{1}{2}}{\mathrm{\Sigma }}_{O}z\right)\right)}^{-1} \\ ={\left({\mathrm{\Sigma }}_{\mathcal{S}}-c^{-1}{z}^T{\mathrm{\Sigma }}_{O}z\right)}^{-1} \end{gathered}$$

$$\Rightarrow c^{-1}={\mathrm{\Sigma }}_{\mathcal{S}}-c^{-1}{z}^T{\mathrm{\Sigma }}_{O}z$$

$$\Rightarrow c^{-1}\left(1+z^T{\mathrm{\Sigma }}_{O}z\right)={\mathrm{\Sigma }}_{\mathcal{S}}$$

$$\Rightarrow c={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\left(1+z^T{\mathrm{\Sigma }}_{O}z\right)$$

Thus, we can directly recover an estimate of Σ_{O$\mathcal{S}$} from the observed Σ_O, known Σ_{$\mathcal{S}$}, and estimated z. Finally, we have:

$${\mathrm{\Sigma }}_{O\mathcal{S}}+\mathbb{E}[\psi (O)]\mathbb{E}{[\psi (\mathcal{S})]}^T=\mathbb{E}\left[\psi (O)\psi {(\mathcal{S})}^T\right].$$ (14)

Here, we can clearly observe $\mathbb{E}[\psi (O)]$, and given that we know the class balance P(Y), we also have $\mathbb{E}\left[\psi (\mathcal{S})\right]$ therefore we can compute $\mathbb{E}\left[\psi (O)\psi {(\mathcal{S})}^T\right]$. Our goal now is to recover the columns $\mathbb{E}\left[\psi (O)\psi ({\mathbf{Y}}_i)\right]$, which together make up μ; we can do this based on the constraints of our rank-one model (Section A.3.4), thus recovering an estimate of μ, which given the uniqueness of $\hat{z}$ (Section B.1) is also unique. The overall procedure is described in the main body, in Algorithm 1.

A.3.3. Handling Non-Singleton Separator Sets

Now, we consider the setting where G_source has arbitrary separator sets. Let $d_S={\sum }_{S\in \mathcal{S}}{\prod }_{i\in S}\left(\left|{\mathcal{Y}}_{{\tau }_i}\right|-1\right).$ We see that we could solve this using our standard approach—this time, involving a rank-d_S matrix completion problem—except for the fact that we do not know Σ_{$\mathcal{S}$}, as it now involves terms besides the class balance.

Note first of all that we can always add edges between sources to G_source such that it has singleton separator sets (intuitively, this consists of “completing the clusters”), and as long as our problem is still identifiable (see Section B.1), we can simply solve this instance as above.

Instead, we can also take a multi-step approach, wherein we first consider one or more subgraphs of G_source that contain only singleton separator sets, and contain the cliques in $\mathcal{S}$. We can then solve this problem as before, which then gives us the needed information to identify the elements of Σ_{$\mathcal{S}$} in our full problem, which we can then solve. In particular, we see that this multi-step approach is possible whenever the graph G_source has at least three components that are disconnected except for through Y.

A.3.4. Rank-One Settings

We now consider settings where we can estimate the parameters of our label model, μ, involving only a rank-one matrix completion problem.

First, in the simplest setting of a single-task problem with binary class variable, Y ∈ {0,1} and G_source with singleton separator sets, d_H = r − 1 = 1 and our problem is directly a rank-one instance.

Next, we consider the setting of general Y, with |$\mathcal{Y}$| = r and G_source with singleton separator sets. By default, our problem now involves a rank-(r − 1) matrix completion problem. However, we can reduce this to involving only a rank-one matrix completion problem by adding one simplifying assuption to our model: namely, that sources emit different incorrect labels with uniform conditional probability. Concretely, we add the assumption that:

$${\left({\mathbf{\lambda }}_C\right)}_i=\mathbf{Y}\iff {\left({\mathbf{\lambda }}_C^{\prime }\right)}_i=\mathbf{Y}\forall i\in C\Rightarrow P\left({\mathbf{\lambda }}_C|\mathbf{Y}\right)=P\left({\mathbf{\lambda }}_C^{\prime }|\mathbf{Y}\right)$$ (15)

Note that this is the same assumption as in the main body, but expressed more explicitly with respect to a clique C. For example, under this assumption, $P\left({\mathbf{\lambda }}_i=y^{\prime }|\mathbf{Y}=y\right)$ is the same for all y′ such that y′ ≠ y. As another example, $P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y^{\prime }|\mathbf{Y}=y\right)$ is the same for all y′ such that y′ ≠ y. Intuitively, under this commonly-used model, we are not modeling the different class-wise errors a source makes, but rather just whether it is correct or not given the correctness of other sources it is correlated with. The idea then is that with assumption (15) even though |H| = r − 1 (and thus ΣO$\mathcal{S}$ has r − 1 columns), we only actually need to solve for a single parameter per element of O.

We can operationalize this by forming a new graph with a binarized version of Y, Y_B ∈ {0,1}, such that the r classes are mapped to either 0 or 1. We see that this new variable still results in the same structure of dependencies G_source, and still allows us to recover the parameters α_y (and thus μ). We now have:

$$\mathcal{S}=\left\{{\mathbf{Y}}_B\right\}$$

We now solve in the same rank-one way as in the binary Y case. Now, for singleton cliques, {${\mathbf{\lambda }}_i$, Y}, given that we know P(Y), we can directly recover P(${\mathbf{\lambda }}_i$ = y |Y = y′) for all y′, given our simplified model.

For non-singleton cliques {λ_C, Y}, note that we can directly recover $P({\cap }_{i\in C}{\mathbf{\lambda }}_i=y|\mathbf{Y}=y^{\prime })$ in the exact same way. From these, computed for all cliques, we can then recover any probability in our model. For example, for y′ ≠ y:

$$\begin{gathered} P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y^{\prime }|\mathbf{Y}=y\right)=P\left({\mathbf{\lambda }}_i=y|\mathbf{Y}=y\right)-\sum _{y^{\prime \prime }\ne y^{\prime }}P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y^{\prime \prime }|\mathbf{Y}=y\right) \\ =P\left({\mathbf{\lambda }}_i=y|\mathbf{Y}=y\right)-P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y|\mathbf{Y}=y\right)- \\ \times (r-2)P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y^{\prime }|\mathbf{Y}=y\right) \end{gathered}$$

$$=>P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y^{\prime }|\mathbf{Y}=y\right)=\frac{1}{r-1}\left(P\left({\mathbf{\lambda }}_i=y|\mathbf{Y}=y\right)-P\left({\mathbf{\lambda }}_i=y,{\mathbf{\lambda }}_j=y|\mathbf{Y}=y\right)\right).$$

In this way, we can recover all of the parameters μ while only involving a rank-one matrix completion problem. Note that this also suggests a way to solve for the more general model, i.e. without (15), using a hierarchical classification approach.

A.3.5. Recovering the Class Balance P & Computing P (Y|λ)

We now turn to the task of recovering the class balance P(Y), for Y ∈ $\mathcal{Y}$. In many practical settings, P(Y) can be estimated from a small labeled sample, or may be known in advance. However here, we consider using a subset of conditionally independent sources, s₁,...,s_k to estimate P(Y). We note first of all that simply taking the majority vote of these sources is a biased estimator.

Instead, we consider a simplified version of the matrix completion-based approach taken so far. Here, we consider a subset of the sources s₁,...,s_k such that they are conditionally independent given G_source, i.e. ${\mathbf{\lambda }}_i\perp {\mathbf{\lambda }}_j|\mathbf{Y},$ and consider only the unary indicator statistics. Denote the vector of these unary indicator statistics over the conditionally independent subset of sources as ϕ, and let the observed overlaps matrix between sources i and j be $A_{i,j}=\mathbb{E}\left[{\varphi }_i{\varphi }_j^T\right].$ Note that due to the conditional independence of ${\mathbf{\lambda }}_i$ and ${\mathbf{\lambda }}_j$, for any k, l we have:

$$\begin{gathered} {\left(A_{i,j}\right)}_{k,l}=\mathbb{E}\left[{\left({\varphi }_i\right)}_k{\left({\varphi }_j\right)}_l\right] \\ =P\left({\mathbf{\lambda }}_i=y_k,{\mathbf{\lambda }}_j=y_l\right) \\ =\sum _{y\in \mathcal{Y}}P\left({\mathbf{\lambda }}_i=y_k,{\mathbf{\lambda }}_j=y_l|\mathbf{Y}=y\right)P(\mathbf{Y}=y) \\ =\sum _{y\in \mathcal{Y}}P\left({\mathbf{\lambda }}_i=y_k|\mathbf{Y}=y\right)P\left({\mathbf{\lambda }}_j=y_l|\mathbf{Y}=y\right)P(\mathbf{Y}=y). \end{gathered}$$

Letting B_i be the $\left|{\mathcal{Y}}_{{\tau }_i}\right|\times |\mathcal{Y}|$ matrix of conditional probabilities, ${\left(B_i\right)}_{j,k}=P\left({\mathbf{\lambda }}_i=y_j|\mathbf{Y}=y_k\right),$ and P be the diagonal matrix such that $P_{i,i}=P\left(\mathbf{Y}=y_i\right),$ we can re-express the above as:

$$A_{i,j}=B_{i}P{B}_j^T.$$

Since P is composed of strictly positive elements, and is diagonal (and thus PSD), we re-express this as:

$$A_{i,j}={\tilde{B}}_i{\tilde{B}}_j^T,$$ (16)

where ${\tilde{B}}_i=B_i\sqrt{P}.$ We could now try to recover P by decomposing the observed A_i,j to recover the ${\tilde{B}}_i,$ and from there recover P via the relation:

$$P=\mathrm{diag}{\left({\tilde{B}}_i^T\overrightarrow{1}\right)}^2,$$ (17)

since summing the column of ${\tilde{B}}_i$ corresponding to label Y is equal to $\sqrt{P(\mathbf{Y})}{\sum }_{y\in {\mathcal{Y}}_i}P\left({\mathbf{\lambda }}_i=y|\mathbf{Y}\right)=\sqrt{P(\mathbf{Y})}$ by the law of total probability. However, note that ${\tilde{B}}_{i}U$ for any orthogonal matrix U also satisfies (16), and could thus lead to a potentially infinite number of incorrect estimates of P.

Class Balance Identifiability with Three-Way View Constraint

A different approach involves considering the three-way overlaps observed as A_i,j,k. This is equivalent to performing a tensor decomposition. Note that above, the problem is that matrix decomposition is typically invariant to rotations and reflections; tensor decompositions have easier-to-meet uniqueness conditions (and are thus more rigid).

Specifically, we apply Kruskal’s classical identifiability condition for unique 3-tensor decomposition. Consider some tensor

$$T=\sum _{r=1}^R{X}_r\otimes Y_r\otimes Z_r,$$

where X_r, Y_r, Z_r are column vectors that make up the matrices X, Y, Z. The Kruskal rank k_X of X is the largest k such that any k columns of X are linearly independent. Then, the decomposition above is unique if k_X + k_Y + k_Z ≥ 2R + 2 [20; 3]. In our case, our triple views have R = |Y|, and we have

$$A_{i,j,k}={\tilde{B}}_i\otimes {\tilde{B}}_j\otimes {\tilde{B}}_k.$$ (18)

Thus, if $k_{{\tilde{B}}_i}+k_{{\tilde{B}}_j}+k_{{\tilde{B}}_k}\ge 2|\mathcal{Y}|+2,$ we have identifiability. Thus, it is sufficient to have the columns of each of the ${\tilde{B}}_i’\text{s}$ be linearly independent. Note that each of the $B_i’\text{s}$ have columns with the same sum, so these columns are only linearly dependent if they are equal, which would only be the case if the sources were random voters.

Thus, we can use (18) to recover the ${\tilde{B}}_i$ in a stable fashion, and then use (17) to recover the P(Y).

Figure 8:

Example task hierarchy G_task for a three-task classification problem. Task Y₁ classifies a data point X as a PERSON or BUILDING. If Y₁ classifies X as a PERSON, Y₂ is used to distinguish between DOCTOR and NON-DOCTOR. Similarly, if Y₂ classifies X as a BUILDING, Y₃ is used to distinguish between HOSPITAL and NON-HOSPITAL. Tasks Y₂, Y₃ are more specific, or finer-grained tasks, constrained by their parent task Y₁.

A.3.6. Predicting Labels with the Label Model

Once we have an estimate of μ, we can make predictions with the label model—i.e. generate our probabilistic training labels P_μ(Y| λ)—using the junction tree we have already defined over G_source. Specifically, let $\tilde{\mathcal{C}}$ be the set of maximal cliques (nodes) in the junction tree, and let $\mathcal{S}$ be the set of separator sets. Then we have:

$$P_{\mu }(\mathbf{Y},\mathbf{\lambda })=\frac{{\prod }_{C\in \tilde{\mathcal{C}}}P\left(V_C\right)}{{\prod }_{S\in \mathcal{S}}P\left(V_S\right)}=\frac{{\prod }_{C\in \tilde{\mathcal{C}}}\mu \left(C,\left(\mathbf{Y},{\mathbf{\lambda }}_C\right)\right)}{{\prod }_{S\in \mathcal{S}}\mu \left(S,\left(\mathbf{Y},{\mathbf{\lambda }}_S\right)\right)},$$

where again, $V_C={\left\{V_i\right\}}_{i\in C},\text{where}{V}_0=\mathbf{Y}\text{and}{V}_{i>0}={\mathbf{\lambda }}_i.$ Thus, we can directly compute the predicted labels P_μ(Y| λ) based on the estimated parameters μ.

A.4. Example: Hierarchical Multi-Task Supervision

We now consider the specific case of hierarchical multi-task supervision, which can be thought of as consisting of coarser- and finer-grained labels, or alternatively higher- and lower-level labels, and provides a way to supervise e.g. fine-grained classification tasks at multiple levels of granularity. Specifically, consider a task label vector Y= [Y₁,...,Y_t]^T as before, this time with Y_s ∈ {N/A, 1,...,k_s}, where we will explain the meaning of the special value N/A shortly. We then assume that the tasks Y_s are related by a task hierarchy which is a hierarchy G_task = (V, E) with vertex set V = {Y₁, Y₂,..., Y_t} and directed edge set E. The task structure reflects constraints imposed by higher level (more general) tasks on lower level (more specific) tasks. The following example illustrates a simple tree task structure:

Example 2

Let Y₁ classify a data point X as either a PERSON (Y₁ = 1) or BUILDING (Y₁ = 2). If Y₁ = 1, indicating that X represents a PERSON, then Y₂ can further label X as a DOCTOR or NON-DOCTOR. Y₃ is used to distinguish between HOSPITAL and NON-HOSPITAL in the case that Y₁ = 2. The corresponding graph G_task is shown in Figure 8. If Y₁ = 2, then task Y₂ is not applicable, since Y₂ is only suitable for persons; in this case, Y₂ takes the value N/A. In this way the task hierarchy defines a feasible set of task vector values: Y = [1, 1, N/A]^T, [1, 2, N/A]^T, [2, N/A, 1]^T, [2, N/A, 2]^T are valid, while e.g. Y = [1,1,2]^T is not.

As in the example, for certain configurations of Y’s, the parent tasks logically constrain the one or more of the children tasks to be irrelevant, or rather, to have inapplicable label values. In this case, the task takes on the value N/A. In Example 2, we have that if Y₁ = 1, representing a building, then Y₂ is inactive (since X corresponds to a building). We define the symbol N/A (for incompatible) for this scenario. More concretely, let $\mathcal{N}$(Y_i) = {Y_j : (Y_j, Y_j) ∈ E} be the in-neighborhood of Y_i. Then, the values of the members of $\mathcal{N}$(Y_i) determine whether Y_i = N/A, i.e., $\mathbb{1}$ {Y_i = N/A} is deterministic conditioned on $\mathcal{N}$(Y_i).

Hierarchical Multi-Task Sources

Observe that in the mutually-exclusive task hierarchy just described, the value of a descendant task label Y_d determines the values of all other task labels in the hierarchy besides its descendants. For example, in Example 2, a label Y₂ = 1 ⇒ (Y₁ = 1, Y₃ = N/A); in other words, knowing that X is a DOCTOR also implies that X is a PERSON and not a BUILDING.

For a source ${\mathbf{\lambda }}_i$ with coverage set t_i, the label it gives to the lowest task in the task hierarchy which is non-zero and non-N/A determines the entire label vector output by ${\mathbf{\lambda }}_i$. E.g. if the lowest task that ${\mathbf{\lambda }}_i$ labels in the hierarchy is Y₁ = 1, then this implies that it outputs vector [1,0, N/A]^T. Thus, in this sense, we can think of each sources ${\mathbf{\lambda }}_i$ as labeling one specific task in the hierarchy, and thus can talk about coarser- and finer-grained sources.

Reduced-Rank Form: Modeling Local Accuracies

In some cases, we can make slightly different modeling assumptions that reflect the nature of the task structure, and additionally can result in reduced-rank forms of our model. In particular, for the hierarchical setting introduced here, we can divide the statistics μ into local and global subsets, and for example focus on modeling only the local ones to once again reduce to rank-one form.

To motivate with our running example: a finer-grained source that labels DOCTOR versus NON-DOCTOR probably is not accurate on the building type subtask; we can model this source using one accuracy parameter for the former label set (the local accuracy) and a different (or no parameter) for the global accuracy on irrelevant tasks. More specifically, for cliques involving ${\mathbf{\lambda }}_i$, we can model P(${\mathbf{\lambda }}_i$, Y) for all Y with only non-N/A values in the coverage set of ${\mathbf{\lambda }}_i$ using a single parameter, and call this the local accuracy; and we can either model μ for the other Y using one or more other parameters, or simply set it to a fixed value and not model it, to reduce to rank one form, as we do in the experiments. In particular, this allows us to capture our observation in practice that if a developer is writing a source to distinguish between labels at one sub-tree, they are probably not designing or testing it to be accurate on any of the other subtrees.

B. Theoretical Results

In this section, we focus on theoretical results for the basic rank-one model considered in the main body of the paper. In Section B.1, we start by going through the conditions for identifiability in more detail for the rank-one case. In Section B.2, we provide additional interpretation for the expression of our primary theoretical result bounding the estimation error of the label model. In Section B.3, we then provide the proof of Theorem 1, connecting this estimation error to the generalization error of the end model; and in Section B.4, we provide the full proof of the main bound.

B.1. Conditions for Identifiability

We consider the rank-one setting as in the main body, where we have

$$-{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\mathrm{\Omega }}={\left(z{z}^T\right)}_{\mathrm{\Omega }},$$ (19)

where Ω is the inverse augmented edge set, i.e. a pair of indices (i, j), corresponding to elements of Ψ($\mathcal{C}$), and therefore to cliques A, B ∈ $\mathcal{C}$, is in Ω if A, B are not part of the same maximal clique in G_source (and therefore (K_O)i,j = 0). This defines a set of |Ω| equations, which we can encode using a matrix M_Ω, where if (i, j) is the (r − 1)th entry in Ω, then

$${\left(M_{\mathrm{\Omega }}\right)}_{r,s}=\{\begin{array}{ll}1\hfill & s\in \{i,j\},\hfill \\ 0\hfill & \text{else}.\hfill \end{array}$$ (20)

Let $l_i=\log \left(z_i^2\right)\text{and}{q}_{(i,j)}=\log \left(\left({\left({\mathrm{\Sigma }}_O^{-1}\right)}_{i,j}\right)\right);$ then by squaring and taking the log of both sides of 19, we get a system of linear equations:

$$M_{\mathrm{\Omega }}l=q_{\mathrm{\Omega }}.$$ (21)

Thus, we can identify z (and therefore μ) up to sign if the system of linear equations (21) has a solution.

Notes on Invertibility of M_Ω

Note that if the inverse augmented edge graph consists of a connected triangle (or any odd-numbered cycle), e.g. Ω = {(i, j), (j, k), (i, k)}, then we can solve for the z_i up to sign, and therefore M_Ω must be invertible:

$$z_i^2=\frac{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{i,j}{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{i,k}}{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{j,k}},$$

and so on for z_j, z_k. Note additionally that if other z_i are connected to this triangle, then we can also solve for them up to sign as well. Therefore, if Ω contains at least one triangle (or odd-numbered cycle) per connected component, then M_Ω is invertible.

Also note that this is all in reference to the inverse source dependency graph, which will generally be dense (assuming the correlation structure between sources is generally sparse). For example, note that if we have one source ${\mathbf{\lambda }}_i$ that is conditionally independent of all the other sources, then Ω is fully connected, and therefore if there is a triangle in Ω, then M_Ω is invertible.

Identifying the Signs of the z_i

Finally, note that if we know the sign of one z_i, then this determines the signs of every other z_j in the same connected component. Therefore, for z to be uniquely identifiable, we need only know the sign of one of the z_i in each connected component. As noted already, if even one source ${\mathbf{\lambda }}_i$ is conditionally independent of all the other sources, then Ω is fully connected; in this case, we can simply assume that the average source is better than random, and therefore identify the signs of z without any additional information.

B.2. Interpreting the Main Bound

We re-state Theorem 2, which bounds the average error on the estimate of the label model parameters, providing more detail on and interpreting the terms of the bound.

Theorem 2

Let $\widehat{\mu }$ be an estimate of μ* produced by Algorithm 1 run over n unlabeled data points. Let $a:={\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}+{\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}\right)}^2{\text{λ}}_{\mathit{max}}\left(K_O\right)\right)}^{\frac{1}{2}}\mathit{\text{and}}b:=\frac{{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2}{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}.$ Then, we have:

$$\mathbb{E}\left[\Vert \widehat{\mu }-{\mu }^{*}\Vert \right]\le 16(r-1)d_O^2\sqrt{\frac{32\pi }{n}}ab{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\left(3\sqrt{d_O}a{\lambda }_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)+1\right)\left(\kappa \left({\mathrm{\Sigma }}_O\right)+{\lambda }_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)\right).$$

Influence of ${\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)$ the largest singular value of the pseudoinverse $M_{\mathrm{\Omega }}^{+}.$ Note that ${\Vert M_{\mathrm{\Omega }}^{+}\Vert }^2={\left({\text{λ}}_{\min }\left(M_{\mathrm{\Omega }}^T{M}_{\mathrm{\Omega }}\right)\right)}^{-1}.$ As we shall see below, ${\text{λ}}_{\min }\left(M_{\mathrm{\Omega }}^T{M}_{\mathrm{\Omega }}\right)$ measures a quantity related to the structure of the graph G_inv. The smaller this quantity, the more information we have about G_inv, and the easier it is to estimate the accuracies. The smallest value of ${\Vert M_{\mathrm{\Omega }}^{+}\Vert }^2$ (corresponding to the largest value of the eigenvalue) is ~ $\frac{1}{\sqrt{m}};$ the square of this quantity in the bound reduces the m² cost of estimating the covariance matrix to m.

It is not hard to see that

$$M_{\mathrm{\Omega }}^T{M}_{\mathrm{\Omega }}=\mathrm{diag}\left(\mathrm{deg}\left(G_{\text{inv}}\right)\right)+\mathrm{Adj}\left(G_{\text{inv}}\right).$$

Here, deg(G_inv) are the degrees of the nodes in G_inv and Adj(G_inv) is its adjacency matrix. This form closely resembles the graph Laplacian, which differs in the sign of the adjacency matrix term: $\mathcal{L}$(G) = diag(deg(G)) − Adj(G). We bound

$${\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\le {\left(d_{\min }+{\text{λ}}_{\min }\left(\mathrm{Adj}\left(G_{\text{inv }})\right)\right)\right)}^{-1},$$

where d_min is the lowest-degree node in G_inv (that is, the source s with fewest appearances in Ω). In general, computing λ_min(Adj(G_inv))) can be challenging. A closely related task can be done via Cheeger inequalities, which state that

$$2h_G\ge {\text{λ}}_{\min }(\mathcal{L}(G))\ge \frac{1}{2}{h}_G^2,$$

where λ_min($\mathcal{L}$(G)) is the smallest non-zero eigenvalue of $\mathcal{L}$(G) and

$$h_G=\underset{X}{\min }\frac{|E(X,\overline{X})|}{\min \left\{{\sum }_{x\in X}{d}_x,{\sum }_{y\in \overline{X}}{d}_y\right\}}$$

is the Cheeger constant of the graph [7]. The utility of the Cheeger constant is that it measures the presence of a bottleneck in the graph; the presence of such a bottleneck limits the graph density and is thus beneficial when estimating the structure in our case. Our Cheeger-constant like term ${\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)$ acts the same way.

Now, in the easiest and most common case is that of conditionally independent sources [9; 38; 9; 17]., Adj(G_inv) has 1’s everywhere but the diagonal, and we can compute explicitly that

$${\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)=\frac{1}{\sqrt{m-2}}.$$

In the general setting, we must compute the minimal eigenvalue of the adjacency matrix, which is tractable, for example, for tree structures.

Influence of λ_min(Σ_O) the smallest eigenvalue of the observed matrix. This quantity reflects the conditioning of the observed (correlation) matrix; the better conditioned the matrix, the easier it is to estimate Σ_O.

Influence of (${\mathrm{\Sigma }}_O^{-1}$)_min the smallest entry of the inverse observed matrix. This quantity contributes to Σ⁻¹, the geenralized precision matrix that we centrally use; it is a measure of the smallest non-zero correlation between source accuracies (that is, the smallest correlation between non-independent source accuracies). Note that the tail bound of Theorem 2 scales as exp(−((${\mathrm{\Sigma }}_O^{-1}$)_min)² This is natural, as distinguishing between small correlations and independencies requires more samples.

B.3. Proof of Theorem 1

Let $\mathcal{D}$ be the true data generating distribution, such that (X, Y) ~ $\mathcal{D}$. Let P_μ(Y|λ) be the label model parameterized by μ and conditioned on the observed source labels λ. Furthermore, assume that:

1. For some optimal label model parameters ${\mu }^{*},P_{{\mu }^{*}}$(λ, Y) = P(λ, Y);

2. The label Y is independent of the features of our end model given the source labels λ

That is, we assume that (i) the optimal label model, parameterized by μ*, correctly matches the true distribution of source labels λ drawn from the true distribution, (s(X), Y) ~ $\mathcal{D}$; and (ii) that these labels λ provide sufficient information to discern the label Y. We note that these assumptions are the ones used in prior work [28], and are intended primarily to illustrate the connection between the estimation accuracy of $\widehat{\mu },$ which we bound in Theorem 2, and the end model performance.

Now, suppose that we have an end model parameterized by w, and that to learn these parameters we minimize a normalized bounded loss function l(w, X, Y), such that without loss of generality, l(w, X, Y) ≤ 1. Normally our goal would be to find parameters that minimize the expected loss, which we denote w*:

$$L(w)={\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}[l(w,X,\mathbf{Y})]$$ (22)

However, since we do not have access to the true labels Y, we instead minimize the expected noise-aware loss, producing an estimate $\widehat{w}:$

$$L_{\mu }(w)={\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}[{\mathbb{E}}_{\tilde{\mathbf{Y}}~P_{\mu }(\cdot |\mathbf{\lambda }(X))}[l(w,X,\tilde{\mathbf{Y}})]].$$ (23)

In practice, we actually minimize the empirical version of the noise aware loss over an unlabeled dataset U = {X⁽¹⁾,...,X⁽ⁿ⁾}, producing an estimate $\widehat{w}:$

$${\widehat{L}}_{\mu }(w)=\frac{1}{n}\sum _{i=1}^n{\mathbb{E}}_{\tilde{\mathbf{Y}}~P_{\mu }\left(\cdot |\mathbf{\lambda }\left(X^{(i)}\right)\right)}\left[l\left(w,X^{(i)},\tilde{\mathbf{Y}}\right)\right].$$ (24)

Let w* be the expected loss L, let $\tilde{w}$ be the minimizer of the noice-aware loss for estimated label model parameters μ, L_μ, and let $\widehat{w}$ be the minimize of the empirical noice aware loss ${\widehat{L}}_{\mu }.$ Our goal is to bound the generalization risk- the difference between the expected loss of our empirically estimated parameters and of the optimal parameters,

$$L(\widehat{w})-L\left(w^{*}\right).$$ (25)

Additionally, since analyzing the empirical risk minimization error is standard and not specific to our setting, we simply assume that error $\left|L_{\mu }(\tilde{w})-L_{\mu }(\widehat{w})\right|\le \gamma (n),$ where γ(n) is a decreasing function of the number of unlabeled data points n.

To start, using the law of total expectation first, followed by our assumption (2) about condtional independence, and finally using our assumption (1) about our optimal label model ${\mu }^{*},$ we have that:

$$\begin{gathered} L(w)={\mathbb{E}}_{\left(X^{\prime },{\mathbf{Y}}^{\prime }\right)~\mathcal{D}}[L(w)] \\ ={\mathbb{E}}_{\left(X^{\prime },{\mathbf{Y}}^{\prime }\right)~\mathcal{D}}\left[{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[l\left(w,X^{\prime },\mathbf{Y}\right)|X=X^{\prime }\right]\right] \\ ={\mathbb{E}}_{\left(X^{\prime },{\mathbf{Y}}^{\prime }\right)~\mathcal{D}}\left[{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[l\left(w,X^{\prime },\mathbf{Y}\right)|s(X)=s\left(X^{\prime }\right)\right]\right] \\ ={\mathbb{E}}_{\left(X^{\prime },{\mathbf{Y}}^{\prime }\right)~\mathcal{D}}\left[{\mathbb{E}}_{(\mathbf{\lambda },\tilde{\mathbf{Y}})~{\mu }^{*}}\left[l\left(w,X^{\prime },\tilde{\mathbf{Y}}\right)|\mathbf{\lambda }=s\left(X^{\prime }\right)\right]\right] \\ =L_{{\mu }^{*}}(w). \end{gathered}$$

Now, we have:

$$\begin{gathered} L(\widehat{w})-L\left(w^{*}\right)=L_{{\mu }^{*}}(\widehat{w})+L_{\mu }(\widehat{w})-L_{\mu }(\widehat{w})+L_{\mu }(\tilde{w})-L_{\mu }(\tilde{w})-L_{{\mu }^{*}}\left(w^{*}\right) \\ \le L_{{\mu }^{*}}(\widehat{w})+L_{\mu }(\widehat{w})-L_{\mu }(\widehat{w})+L_{\mu }\left(w^{*}\right)-L_{\mu }(\tilde{w})-L_{{\mu }^{*}}\left(w^{*}\right) \\ \le \left|L_{\mu }(\widehat{w})-L_{\mu }(\tilde{w})\right|+\left|L_{{\mu }^{*}}(\widehat{w})-L_{\mu }(\widehat{w})\right|+\left|L_{\mu }\left(w^{*}\right)-L_{{\mu }^{*}}\left(w^{*}\right)\right| \\ \le \gamma (n)+2\underset{w^{\prime }}{\max }\left|L_{{\mu }^{*}}\left(w^{\prime }\right)-L_{\mu }\left(w^{\prime }\right)\right|, \end{gathered}$$

where in the first step we use our result that L = L_μ* as well as add and subtract terms; and in the second step we use the fact that $L_{\mu }(\tilde{w})\le L_{\mu }\left(w^{*}\right).$ We now have our generalization risk controlled primarily by $\left|L_{{\mu }^{*}}\left(w^{\prime }\right)-L_{\mu }\left(w^{\prime }\right)\right|,$ which is the difference between the expected noise aware losses given the estimated label model parameters μ and the true label model parameters μ*. Next, we see that, for any w′:

$$\begin{gathered} \left|L_{{\mu }^{*}}\left(w^{\prime }\right)-L_{\mu }\left(w^{\prime }\right)\right|=\left|{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[{\mathbb{E}}_{\tilde{\mathbf{Y}}~P_{{\mu }^{*}}(\cdot |\text{λ})}[l(w,X,\tilde{\mathbf{Y}})]-{\mathbb{E}}_{\tilde{\mathbf{Y}}~P_{\mu }(\cdot |\mathbf{\lambda })}[l(w,X,\tilde{\mathbf{Y}})]\right]\right| \\ =\left|{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[\sum _{{\mathbf{Y}}^{\prime }\in \mathcal{Y}}l\left(w,X,{\mathbf{Y}}^{\prime }\right)\left(P_{{\mu }^{*}}\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)-P_{\mu }\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)\right)\right]\right| \\ \le \sum _{{\mathbf{Y}}^{\prime }\in \mathcal{Y}}{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[\left|P_{{\mu }^{*}}\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)-P_{\mu }\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)\right|\right] \\ \le \left|\mathcal{Y}\right|{\max }_{{\mathbf{Y}}^{\prime }}{\mathbb{E}}_{(X,\mathbf{Y})~\mathcal{D}}\left[\left|P_{{\mu }^{*}}\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)-P_{\mu }\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)\right|\right], \end{gathered}$$

where we have now bounded $\left|L_{{\mu }^{*}}\left(w^{\prime }\right)-L_{\mu }\left(w^{\prime }\right)\right|$ by the size of the structured output space |$\mathcal{Y}$|, and a term having to do with the difference between the probability distributions of μ and μ*.

Now, we use the result from [16] (Lemma 19) which establishes that the log probabilities of discrete factor graphs with indicator features (such as our model P_μ(λ, Y)) are (l_∞, 2)-Lipschitz with respect to their parameters, and the fact that for $x,y\text{s.t.}|x|,|y|\le 1,|x-y|\le |\log (x)-\log (y)|,$ to get:

$$\begin{gathered} \left|P_{{\mu }^{*}}\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)-P_{\mu }\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)\right|\le \left|\log P_{{\mu }^{*}}\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)-\log P_{\mu }\left({\mathbf{Y}}^{\prime }|\mathbf{\lambda }\right)\right| \\ \le \left|\log P_{{\mu }^{*}}\left(\mathbf{\lambda },{\mathbf{Y}}^{\prime }\right)-\log P_{\mu }\left(\mathbf{\lambda },{\mathbf{Y}}^{\prime }\right)\right|+\left|\log P_{{\mu }^{*}}(\mathbf{\lambda })-\log P_{\mu }(\mathbf{\lambda })\right| \\ \le 2{\Vert {\mu }^{*}-\mu \Vert }_{\infty }+2{\Vert {\mu }^{*}-\mu \Vert }_{\infty } \\ \le 4\Vert {\mu }^{*}-\mu \Vert , \end{gathered}$$

where we use the fact that the statement of Lemma 19 also holds for every marginal distribution as well. Therefore, we finally have:

$$L(\widehat{w})-L\left(w^{*}\right)\le \gamma (n)+4\left|\mathcal{Y}\right|\Vert {\mu }^{*}-\mu \Vert .$$

B.4. Proof of Theorem 2

Proof: First we briefly provide a roadmap of the proof of Theorem 2. We consider estimating $\tilde{\mu }$ with our procedure in the rank-one setting, and we seek a tail bound on $\Vert \tilde{\mu }-\mu \Vert .$ The challenge here is that the observed matrix ${\mathrm{\Sigma }}_O$ we see is itself constructed from a series of observed i.i.d. samples $\psi {(O)}^{(1)},\dots ,\psi {(O)}^{(n)}.$ We bound (through a matrix concentration inequality) the error ${\mathrm{\Delta }}_O={\tilde{\mathrm{\Sigma }}}_O-{\mathrm{\Sigma }}_O,$ and view Δ_O as a perturbation of Σ_O. Afterwards, we use a series of perturbation analyses to ultimately bound $\Vert {\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}\Vert ,$ and then use this directly to bound $\Vert \tilde{\mu }-\mu \Vert ;$ each of the perturbation results is in terms of Δ_O.

We begin with some notation. We write the following perturbations (note that all the terms written with Δ are additive, while the S term is relative)

$$\begin{array}{r}{\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}}={\mathrm{\Sigma }}_{O\mathcal{S}}+{\mathrm{\Delta }}_{O\mathcal{S}},\\ {\tilde{\mathrm{\Sigma }}}_O={\mathrm{\Sigma }}_O+{\mathrm{\Delta }}_O,\\ \tilde{\mathcal{l}}=\mathcal{l}+{\mathrm{\Delta }}_{\mathcal{l}},\\ \tilde{z}=\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z.\end{array}$$

Now we start our perturbation analysis:

$$\begin{gathered} {\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}}=\frac{1}{\sqrt{\tilde{c}}}{\tilde{\mathrm{\Sigma }}}_O\tilde{z}=\frac{1}{\sqrt{\tilde{c}}}\left({\mathrm{\Sigma }}_O+{\mathrm{\Delta }}_O\right)\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z \\ =\frac{1}{\sqrt{\tilde{c}}}\left({\mathrm{\Sigma }}_{O}z+{\mathrm{\Sigma }}_O\mathrm{diag}\left({\delta }_z\right)z+{\mathrm{\Delta }}_O\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z\right). \end{gathered}$$

Subtracting ${\mathrm{\Sigma }}_{O\mathcal{S}}=\frac{1}{\sqrt{c}}{\mathrm{\Sigma }}_{O}z$, we get

$${\mathrm{\Delta }}_{O\mathcal{S}}=(\frac{1}{\sqrt{\tilde{c}}}-\frac{1}{\sqrt{c}}){\mathrm{\Sigma }}_{O}z+\frac{1}{\sqrt{\tilde{c}}}\left({\mathrm{\Sigma }}_O\mathrm{diag}\left({\delta }_z\right)z+{\mathrm{\Delta }}_O\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z\right).$$ (26)

The rest of the analysis requires us to bound the norms for each of these terms.

Left-most term.

We have that

$$\Vert \left(\frac{1}{\sqrt{\tilde{c}}}-\frac{1}{\sqrt{c}}\right){\mathrm{\Sigma }}_{O}z\Vert =|\frac{\sqrt{c}}{\sqrt{\tilde{c}}}-1|\Vert \frac{1}{\sqrt{c}}{\mathrm{\Sigma }}_{O}z\Vert =|\frac{\sqrt{c}}{\sqrt{\tilde{c}}}-1|\Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert \le \sqrt{d_O}|\frac{\sqrt{c}}{\sqrt{\tilde{c}}}-1|\le \sqrt{d_O}|\tilde{c}-c|.$$

Here, we bounded $\Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert \text{by}\sqrt{d_O},\text{since}{\mathrm{\Sigma }}_{O\mathcal{S}}\in {[-1,1]}^{d_O}.$ Then, note that $c={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\left(1+z^T{\mathrm{\Sigma }}_{O}z\right)\ge 0,\text{since}{\mathrm{\Sigma }}_{\mathcal{S}}<1\text{and}{\mathrm{\Sigma }}_O\succcurlyeq 0\Rightarrow z^T{\mathrm{\Sigma }}_{O}z\ge 0,$ so therefore $c,\tilde{c}\ge 1.$ In the last inequality, we use this to imply that $|\sqrt{c}/\sqrt{\tilde{c}}-1|\le |\sqrt{c}-\sqrt{\tilde{c}}|\le |\tilde{c}-c|.$ Next we work on bounding $|\tilde{c}-c|.$ We have

$$\begin{gathered} |\tilde{c}-c|=\left|{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right|\left|{\tilde{z}}^T{\tilde{\mathrm{\Sigma }}}_O\tilde{z}-z^T{\mathrm{\Sigma }}_{O}z\right| \\ =\left|{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right|\left|z^T{\left(I+\mathrm{diag}\left({\delta }_z\right)\right)}^T\left({\mathrm{\Sigma }}_O+{\mathrm{\Delta }}_O\right)\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z-z^T{\mathrm{\Sigma }}_{O}z\right| \\ =\left|{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right|\left|z^T{\mathrm{\Sigma }}_O\mathrm{diag}\left({\delta }_z\right)z+z^T{\mathrm{\Delta }}_O\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z+z^T\mathrm{diag}{\left({\delta }_z\right)}^T\left({\mathrm{\Sigma }}_O+{\mathrm{\Delta }}_O\right)\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z\right| \\ \le \left|{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right|\Vert z{\Vert }^2\left(\Vert {\mathrm{\Sigma }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2\right)+\Vert {\mathrm{\Delta }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2+1\right)\right) \\ \le \Vert z{\Vert }^2\left(\Vert {\mathrm{\Sigma }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2\right)+\Vert {\mathrm{\Delta }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2+1\right)\right). \end{gathered}$$

Thus,

$$\Vert \left(\frac{1}{\sqrt{\tilde{c}}}-\frac{1}{\sqrt{c}}\right){\mathrm{\Sigma }}_{O}z\Vert \le \sqrt{d_O}\Vert z{\Vert }^2\left(\Vert {\mathrm{\Sigma }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2\right)+\Vert {\mathrm{\Delta }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2+1\right)\right).$$ (27)

Bounding c.

We will need a bound on c to bound z. We have that

$$c={\left({\mathrm{\Sigma }}_{\mathcal{S}}-{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\right)}^{-1}.$$

Applying the Woodbury matrix inversion lemma, we have:

$$c={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T{\left({\mathrm{\Sigma }}_O-{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T\right)}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}$$

Now, by the blockwise inversion lemma, we know that

$$K_O={\left({\mathrm{\Sigma }}_O-{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T\right)}^{-1}$$

So we then have:

$$c={\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}^T{K}_O{\mathrm{\Sigma }}_{O\mathcal{S}}{\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\le {\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\left({\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right)}^2{\Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert }^2\Vert K_O\Vert$$

Bounding z.

We’ll use our bound on c, since $z=\sqrt{c}{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}.$

$$\begin{gathered} \Vert z\Vert =\Vert \sqrt{c}{\mathrm{\Sigma }}_O^{-1}{\mathrm{\Sigma }}_{O\mathcal{S}}\Vert \\ \le {\left({\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\left({\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right)}^2{\Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert }^2\Vert K_O\Vert \right)}^{\frac{1}{2}}\Vert {\mathrm{\Sigma }}_O^{-1}\Vert \Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert \\ \le {\left({\mathrm{\Sigma }}_{\mathcal{S}}^{-1}+{\left({\mathrm{\Sigma }}_{\mathcal{S}}^{-1}\right)}^2{d}_O\Vert K_O\Vert \right)}^{\frac{1}{2}}\Vert {\mathrm{\Sigma }}_O^{-1}\Vert \sqrt{d_O} \\ =\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}{\left(\frac{{\mathrm{\Sigma }}_{\mathcal{S}}}{d_O}+{\lambda }_{\max }\left(K_O\right)\right)}^{\frac{1}{2}}{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right) \end{gathered}$$

In the last inequality, we used the fact that ${\Vert {\mathrm{\Sigma }}_{O\mathcal{S}}\Vert }^2\le d_O.$ Now we want to control $\Vert {\mathrm{\Delta }}_{\mathcal{l}}\Vert .$

Perturbation bound.

We have the perturbation bound

$$\Vert {\mathrm{\Delta }}_{\mathcal{l}}\Vert \le \Vert M_{\mathrm{\Omega }}^{+}\Vert \Vert {\tilde{q}}_S-q_S\Vert .$$ (28)

We need to work on the term $\Vert {\tilde{q}}_S-q_S\Vert .$ To avoid overly heavy notation, we write $P={\mathrm{\Sigma }}_O^{-1},\tilde{P}={\tilde{\mathrm{\Sigma }}}_O^{-1},$ and ${\mathrm{\Delta }}_P=P-\tilde{P}.$ Then we have:

$$\begin{gathered} {\Vert {\tilde{q}}_S-q_S\Vert }^2=\sum _{(i,j)\in S}{\left(\log \left({\tilde{P}}_{i,j}^2\right)-\log \left(P_{i,j}^2\right)\right)}^2 \\ =4\sum _{(i,j)\in S}{\left(\log \left(\left|{\tilde{P}}_{i,j}\right|\right)-\log \left(\left|P_{i,j}\right|\right)\right)}^2 \\ =4\sum _{(i,j)\in S}{\left(\log \left(\left|P_{i,j}+{\left({\mathrm{\Delta }}_P\right)}_{i,j}\right|\right)-\log \left(\left|P_{i,j}\right|\right)\right)}^2 \\ \le 4\sum _{(i,j)\in S}{\left[\log \left(1+\left|\frac{{\left({\mathrm{\Delta }}_P\right)}_{i,j}}{P_{i,j}}\right|\right)\right]}^2 \\ \le 8\sum _{(i,j)\in S}{\left(\frac{\left|{\left({\mathrm{\Delta }}_P\right)}_{i,j}\right|}{\left|P_{i,j}\right|}\right)}^2 \\ \le \frac{8}{P_{\min }^2}\sum _{(i,j)\in S}{\left({\mathrm{\Delta }}_P\right)}_{i,j}^2 \\ \le \frac{8{\Vert {\tilde{\mathrm{\Sigma }}}_O^{-1}-{\mathrm{\Sigma }}_O^{-1}\Vert }^2}{{\left({\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }\right)}^2}. \end{gathered}$$

Here, the second inequality uses ${(\log (1+x))}^2\le x^2,$ and the fourth inequality sums over squared values. Next, we use the perturbation bound $\Vert {\tilde{\mathrm{\Sigma }}}_O^{-1}-{\mathrm{\Sigma }}_O^{-1}\Vert \le {\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2\Vert {\mathrm{\Delta }}_O\Vert ,$ so that we have

$$\Vert {\tilde{q}}_S-q_S\Vert \le \frac{2\sqrt{2}{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2\Vert {\mathrm{\Delta }}_O\Vert }{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}.$$

Then, plugging this into (28), we get that

$$\Vert {\mathrm{\Delta }}_{\mathcal{l}}\Vert \le {\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\frac{2\sqrt{2}{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2\Vert {\mathrm{\Delta }}_O\Vert }{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}.$$ (29)

Bounding δ_z.

Note also that ${\Vert {\mathrm{\Delta }}_{\mathcal{l}}\Vert }^2={\sum }_{i=1}^m\left(\log \left({\tilde{z}}_i^2\right)-\log \left(z_i^2\right)\right).$ We have that

$$\begin{gathered} {\Vert {\mathrm{\Delta }}_{\mathcal{l}}\Vert }^2=\sum _{i=1}^m\log \left(\frac{{\tilde{z}}_i^2}{z_i^2}\right) \\ =2\sum _{i=1}^m\log \left(\frac{\left|{\tilde{z}}_i\right|}{\left|z_i\right|}\right) \\ =2\sum _{i=1}^m\log \left(1+\left|{\left({\delta }_z\right)}_i\right|\right), \\ \ge 2\sum _{i=1}^m{\left({\delta }_z\right)}_i^2 \\ =2{\Vert {\delta }_z\Vert }^2, \end{gathered}$$

where in the fourth step, we used the bound $\log (1+a)\ge a^2$ for small a. Then, we have

$$\Vert {\delta }_z\Vert \le \frac{\sqrt{2}{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2\Vert {\mathrm{\Delta }}_O\Vert }{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right).$$ (30)

Putting it together.

Using (26), we have that

$$\begin{gathered} \Vert {\mathrm{\Delta }}_{O\mathcal{S}}\Vert =\Vert \left(\frac{1}{\sqrt{\tilde{c}}}-\frac{1}{\sqrt{c}}\right){\mathrm{\Sigma }}_{O}z+\frac{1}{\sqrt{\tilde{c}}}\left({\mathrm{\Sigma }}_O\mathrm{diag}\left({\delta }_z\right)z+{\mathrm{\Delta }}_O\left(I+\mathrm{diag}\left({\delta }_z\right)\right)z\right)\Vert \\ \le \Vert \left(\frac{1}{\sqrt{\tilde{c}}}-\frac{1}{\sqrt{c}}\right){\mathrm{\Sigma }}_{O}z\Vert +\left(\Vert {\mathrm{\Sigma }}_O\mathrm{diag}\left({\delta }_z\right)\Vert +\Vert {\mathrm{\Delta }}_O\left(I+\mathrm{diag}\left({\delta }_z\right)\right)\Vert \right)\Vert z\Vert \\ \le \sqrt{d_O}\Vert z{\Vert }^2\left(\Vert {\mathrm{\Sigma }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2\right)+\Vert {\mathrm{\Delta }}_O\Vert \left(2\Vert {\delta }_z\Vert +{\Vert {\delta }_z\Vert }^2+1\right)\right) \\ +\Vert {\mathrm{\Sigma }}_O\Vert \Vert {\delta }_z\Vert \Vert z\Vert +\Vert {\mathrm{\Delta }}_O\Vert \Vert z\Vert \left(1+\Vert {\delta }_z\Vert \right) \\ \le \sqrt{d_O}\Vert z{\Vert }^2\left(3\Vert {\mathrm{\Sigma }}_O\Vert \Vert {\delta }_z\Vert +3\Vert {\mathrm{\Delta }}_O\Vert \Vert {\delta }_z\Vert +\Vert {\mathrm{\Delta }}_O\Vert \right) \\ +\Vert {\mathrm{\Sigma }}_O\Vert \Vert {\delta }_z\Vert \Vert z\Vert +\Vert {\mathrm{\Delta }}_O\Vert \Vert z\Vert \left(1+\Vert {\delta }_z\Vert \right) \\ \le \Vert z\Vert (3\sqrt{d_O}\Vert z\Vert +1)\left(\left(\Vert {\mathrm{\Sigma }}_O\Vert +\Vert {\mathrm{\Delta }}_O\Vert \right)\Vert {\delta }_z\Vert +\Vert {\mathrm{\Delta }}_O\Vert \right) \end{gathered}$$

Where in the first inequality, we use the triangle inequality and the fact that $\tilde{c}>1,$ and in the third inequality, we relied on the fact that we can control $\Vert {\delta }_z\Vert \left(\text{through}\Vert {\mathrm{\Delta }}_O\Vert \right)$ so that we can make it small enough and thus take ${\Vert {\delta }_z\Vert }^2\le \Vert {\delta }_z\Vert .$ Now we can plug in our bounds on $\Vert z\Vert \text{and}\Vert {\delta }_z\Vert$ from before:

$$\begin{gathered} \Vert {\mathrm{\Delta }}_{OS}\Vert \le \left(\frac{d_O}{{\mathrm{\Sigma }}_S}{\left(\frac{{\mathrm{\Sigma }}_S}{d_O}+{\text{λ}}_{\max }\left(K_O\right)\right)}^{\frac{1}{2}}{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)\right)\left(3\sqrt{d_O}\left(\frac{d_O}{{\mathrm{\Sigma }}_S}{\left(\frac{{\mathrm{\Sigma }}_S}{d_O}+{\text{λ}}_{\max }\left(K_O\right)\right)}^{\frac{1}{2}}{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)\right)+1\right) \\ \times \left(\left(\Vert {\mathrm{\Sigma }}_O\Vert +\Vert {\mathrm{\Delta }}_O\Vert \right)\left(\frac{\sqrt{2}{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2\Vert {\mathrm{\Delta }}_O\Vert }{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\right)+\Vert {\mathrm{\Delta }}_O\Vert \right) \end{gathered}$$

For convenience, we set $\Vert {\mathrm{\Delta }}_O\Vert =t.$ Recall that

$$a={\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}+{\left(\frac{d_O}{{\mathrm{\Sigma }}_{\mathcal{S}}}\right)}^2{\text{λ}}_{\max }\left(K_O\right)\right)}^{\frac{1}{2}}$$

and

$$b=\frac{{\Vert {\mathrm{\Sigma }}_O^{-1}\Vert }^2}{{\left({\mathrm{\Sigma }}_O^{-1}\right)}_{\min }}.$$

Then, we have

$$\Vert {\mathrm{\Delta }}_{OS}\Vert \le (3\sqrt{d_O}a{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)+1)\left(\sqrt{2}ab\kappa \left({\mathrm{\Sigma }}_O\right){\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)t+\sqrt{2}ab\frac{{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)}{{\text{λ}}_{\min }\left({\mathrm{\Sigma }}_O\right)}{t}^2+a{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)t\right).$$

Again we can take t small so that t² ≤ t. Simplifying further, we have

$$\Vert {\mathrm{\Delta }}_{OS}\Vert \le \left(3\sqrt{d_O}a{\text{λ}}_{\min }^{{}^{-1}}\left({\mathrm{\Sigma }}_O\right)+1\right)\left(\sqrt{2}ab{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\left[\kappa \left({\mathrm{\Sigma }}_O\right)+{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)\right]+a{\text{λ}}_{\min }^{{}^{-1}}\left({\mathrm{\Sigma }}_O\right)\right)t.$$

Finally, since the $a{\text{λ}}_{\min }^{{}^{-1}}\left({\mathrm{\Sigma }}_O\right)$ is smaller than the left-hand term inside the parentheses, we can write

$$\Vert {\mathrm{\Delta }}_{O\mathcal{S}}\Vert \le \left(3\sqrt{d_O}a{\text{λ}}_{\min }^{{}^{-1}}\left({\mathrm{\Sigma }}_O\right)+1\right)\left(2\sqrt{2}ab{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\left[\kappa \left({\mathrm{\Sigma }}_O\right)+{\text{λ}}_{\min }^{{}^{-1}}\left({\mathrm{\Sigma }}_O\right)\right]\right)t.$$ (31)

Concentration bound.

We need to bound t = ||Δ_O ||, the error when estimating Σ_O from observations $\psi {(O)}^{(1)},\dots ,\psi {(O)}^{(n)}$ over n unlabeled data points.

To start, recall that O is the set of observable cliques, $\psi (O)\in {\{0,1\}}^{d_O}$ is the corresponding vector of minimal statistics, and ${\mathrm{\Sigma }}_O=\mathbf{\text{Cov}}[\psi (O)].$ For notational convenience, let $R=\mathbb{E}\left[\psi (O)\psi {(O)}^T\right],r=\mathbb{E}[\psi (O)],\text{and}{r}_k=\psi {(O)}^{(k)},\text{and}{\mathrm{\Delta }}_r=\frac{1}{n}{\sum }_{i=1}^n{r}_k-r.$ Then we have:

$$\begin{gathered} \Vert {\mathrm{\Delta }}_O\Vert =\Vert {\mathrm{\Sigma }}_O-{\tilde{\mathrm{\Sigma }}}_O\Vert =\Vert \left(R-r{r}^T\right)-\left(\frac{1}{n}\sum _{i=1}^n{r}_i{r}_i^T-\left(r+{\mathrm{\Delta }}_r\right){\left(r+{\mathrm{\Delta }}_r\right)}^T\right)\Vert \\ \le \underset{{\mathrm{\Delta }}_R}{\underbrace{\Vert R-\frac{1}{n}\sum _{i=1}^n{r}_i{r}_i^T\Vert }}+\underset{{\mathrm{\Delta }}_r}{\underbrace{\Vert r{r}^T-\left(r+{\mathrm{\Delta }}_r\right){\left(r+{\mathrm{\Delta }}_r\right)}^T\Vert }}. \end{gathered}$$

We start by applying the matrix Hoeffding inequality [32] to bound the first term, ${\mathrm{\Delta }}_R.\text{Let}{S}_k=\frac{1}{n}\left(R-R_k\right),$ and thus clearly $\mathbb{E}\left[S_k\right]=0.$ We seek a sequence of symmetric matrices $A_k\text{s.t.}{S}_k^2\preccurlyeq A_k^2.$ First, note that, for some vectors x, v,

$$x^T\left(\Vert v{\Vert }^{2}I-v{v}^T\right)x=\Vert v{\Vert }^2\Vert x{\Vert }^2-{\langle x,v\rangle }^2\ge 0$$

using Cauchy-Schwarz; therefore $\Vert v{\Vert }^{2}I\succcurlyeq v{v}^T,$ so that

$$d_O^{2}I\succcurlyeq {\Vert r_k\Vert }^{4}I\succcurlyeq {\Vert r_k\Vert }^2{r}_k{r}_k^T={\left(r_k{r}_k^T\right)}^2.$$

Next, note that ${\left(r_k{r}_k^T+R\right)}^2\succcurlyeq 0.$ Now, we use this to see that:

$${\left(n{S}_k\right)}^2={\left(r_k{r}_k^T-R\right)}^2\preccurlyeq {\left(r_k{r}_k^T-R\right)}^2+{\left(r_k{r}_k^T+R\right)}^2=2\left({\left(r_k{r}_k^T\right)}^2+R^2\right)\preccurlyeq 2\left(d_O^{2}I+R^2\right).$$

Therefore, let $A_k^2=\frac{2}{n^2}\left(d_O^{2}I+R^2\right),$ and note that $\Vert R^2\Vert \le \Vert R{\Vert }^2\le {\left(d_O\Vert R{\Vert }_{\mathit{max}}\right)}^2=d_O^2.$ We then have

$${\sigma }^2=\Vert \sum _{k=1}^n{A}_k^2\Vert \le \frac{2}{n}\left(d_O^2+\Vert R^2\Vert \right)\le \frac{4d_O^2}{n}.$$

And thus,

$$P\left(\Vert {\mathrm{\Delta }}_R\Vert \ge \gamma \right)\le 2d_O\exp \left(-\frac{n{\gamma }^2}{32d_O^2}\right).$$ (32)

Next, we bound Δ_r. We see that:

$$\begin{gathered} \Vert {\mathrm{\Delta }}_r\Vert =\Vert r{r}^T-\left(r+{\mathrm{\Delta }}_r\right){\left(r+{\mathrm{\Delta }}_r\right)}^T\Vert \\ =\Vert r{\mathrm{\Delta }}_r^T+{\mathrm{\Delta }}_r{r}^T+{\mathrm{\Delta }}_r{\mathrm{\Delta }}_r^T\Vert \\ \le \Vert r{\mathrm{\Delta }}_r^T\Vert +\Vert {\mathrm{\Delta }}_r{r}^T\Vert +\Vert {\mathrm{\Delta }}_r{\mathrm{\Delta }}_r^T\Vert \\ \le 2\Vert r\Vert \Vert {\mathrm{\Delta }}_r\Vert +{\Vert {\mathrm{\Delta }}_r\Vert }^2 \\ \le 3\Vert r\Vert \Vert {\mathrm{\Delta }}_r\Vert \\ \le 3\Vert r{\Vert }_1{\Vert {\mathrm{\Delta }}_r\Vert }_1 \\ \le 3d_O^2\left|{\mathrm{\Delta }}_r^{\prime }\right|, \end{gathered}$$

where ${\mathrm{\Delta }}_r^{\prime }$ is the perturbation for a single element of ψ(O). We can then apply the standard Hoeffding’s bound to get:

$$P\left(\Vert {\mathrm{\Delta }}_r\Vert \ge \gamma \right)\le 2\exp \left(-\frac{2n{\gamma }^2}{3d_O^2}\right),$$

Combining the bounds for ||Δ_R|| and ||Δ_r||, we get:

$$P\left(\Vert {\mathrm{\Delta }}_O\Vert \ge \gamma \right)=P(t\ge \gamma )\le 3d_O\exp \left(-\frac{n{\gamma }^2}{32d_O^2}\right).$$ (33)

Final steps

Now, we use the bound on t in (31) and the concentration bound above to write

$$\begin{gathered} P\left(\Vert {\mathrm{\Delta }}_{O\mathcal{S}}\Vert \ge t^{\prime }\right)\le P\left(Vt\ge t^{\prime }\right) \\ =P\left(t\ge \frac{t^{\prime }}{V}\right) \\ \le 2d_O\exp \left(-\frac{n{t}^{\prime 2}}{32V^2{d}_O^2}\right), \end{gathered}$$

where $V=\left(3\sqrt{d_O}a{\text{λ}}_{\min }^{-1}\left({\mathrm{\Sigma }}_O\right)+1\right)\left(2\sqrt{2}ab{\sigma }_{\max }\left(M_{\mathrm{\Omega }}^{+}\right)\left[\kappa \left({\mathrm{\Sigma }}_O\right)+\frac{1}{{\text{λ}}_{\min }(\mathrm{\Sigma }o)}\right]\right).$

Given ${\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}},$ we recover ${\tilde{\mu }}_1={\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}}+\mathbb{E}[\psi (H)]\widehat{\mathbb{E}}[\psi (O)].$ We assume $\mathbb{E}[\psi (H)]$ is known, and we can bound the error introduced by $\mathbb{E}[\psi (H)]\hat{\mathbb{E}}[\psi (O)]$ as above, which we see can be folded into the looser bound for the error in ${\tilde{\mathrm{\Sigma }}}_{O\mathcal{S}}.$

Finally, we expand the rank-one form ${\tilde{\mu }}_1\text{into}\tilde{\mu }$ algebraically, according to our weight tying in the rank one model we use. Suppose in the rank one reduction (see Section A.3.4), we let ${\mathbf{Y}}_B=\mathbb{1}\left\{\mathbf{Y}=y_1\right\}.$ Then each element of μ₁ that we track corresponds to either the probability of being correct, ${\alpha }_{C,y}=P\left({\cap }_{i\in C}\left\{{\mathbf{\lambda }}_i=y\right\},\mathbf{Y}=y\right)$ or the probability of being incorrect, $\frac{1}{r-1}\left(1-{\alpha }_{C,y}\right),$ for each source clique C and label output combination y_C, and this value is simply copied r − 1 times (for the other, weight-tied incorrect values), except for potentially one entry where it is multiplied by (r − 1) and then subtracted from 1 (to transform from incorrect to correct). Therefore, $\Vert {\mathrm{\Delta }}_{\mu }\Vert =\Vert \mu -\tilde{\mu }\Vert \le 2(r-1)\Vert {\mu }_1-{\tilde{\mu }}_1\Vert .$ Thus, we have:

$$P\left(\Vert {\mathrm{\Delta }}_{\mu }\Vert \ge t^{\prime }\right)\le 4(r-1)d_O\exp \left(-\frac{n{t}^{\prime 2}}{32V^2{d}_O^2}\right),$$

where V is defined as above. We only have one more step:

$$\begin{gathered} \mathbb{E}[\Vert \tilde{\mu }-\mu \Vert ]={\int }_0^{\infty }P(\Vert \tilde{\mu }-\mu \Vert \ge \gamma )d\gamma \\ \le {\int }_0^{\infty }4(r-1)d_O\exp \left(-\frac{n}{32V^2{d}_O^2}{\gamma }^2\right)d\gamma \\ =\frac{4(r-1)d_O\sqrt{\pi }}{2\sqrt{\frac{n}{32V^2{d}_O^2}}} \\ =4(r-1)d_O^2\sqrt{\frac{32\pi }{n}}V. \end{gathered}$$

Here, we used the fact that ${\int }_0^{\infty }\exp \left(-a{\gamma }^2\right)d\gamma =\frac{\sqrt{\pi }}{2\sqrt{a}}.$ □

C. Experimental Details

C.1. Data Balancing and Label Model Training Procedure

For each application, rebalancing was applied via direct subsampling to the training set in the manner that was found to most improve development set micro-averaged accuracy. Specifically, we rebalance with respect to the median class for OpenI (i.e., removing examples from majority class such that none had more than the original median class), the minimum class for TACRED, and perform no rebalancing for OntoNotes. For generative model training, we use stochastic gradient descent with a step size, step number, and ${\mathcal{l}}_2$ penalty listed in Table 3 below. These parameters were found via 10-trial coarse random search, with all values determined via maximum micro-averaged accuracy evaluated on the development set.

C.2. End Model Training Procedure

Before training over multiple iterations to attain averaged results for reporting, a 10-trial random search over learning rate and ${\mathcal{l}}_2$ regularization with the Adam optimizer was performed for each application based on micro-averaged development set accuracy. Learning rate was decayed by an order of magnitude if no increases in training loss improvement or development set accuracy were observed for 10 epochs, and the learning rate was frozen during the first 5 epochs. Models are reported using early stopping, wherein the best performing model on the development set is eventually used for evaluation on the held-out test set, and maximum epoch number is set for each application at a point beyond which minimal additional decrease in training loss was observed.

C.3. Dataset Statistics

We give additional detail in here (see Table 4) on the different datasets used for the experimental portion of this work. All data in the development and test sets is labeled with ground truth, while data in the training set is treated as unlabeled. Each dataset has a particular advantage in our study. The OntoNotes set, for instance, contains a particularly large number of relevant data points (over 63k), which enables us to investigate empirical performance scaling with the number of unlabeled data points. Further, the richness of the TACRED dataset allowed for the creation of an 8-class, 7-sub-task hierarchical classification problem, which demonstrates the utility of being able to supervise at each of the three levels of task granularity. Finally, the OpenI dataset represents a real-world, non-benchmark problem drawn from the domain of medical triage, and domain expert input was directly leveraged to create the relevant supervision sources. The fact that these domain expert weak supervision sources naturally occurred at multiple levels of granularity, and that the could be easily integrated to train an effective end model, demonstrates the utility of the MeTaL framework in practical settings.

Table 3:

Model architecture and training parameter details.

	OntoNotes	TACRED	OpenI
Label Model Training
Step Size	5e-3	1e-2	5e-4
${\mathcal{l}}_2$ Regularization	1e-4	4e-4	1e-3
Step Number	50	25	50
End Model Architecture
Embedding Initialization	PubMed	FastText EN	Random
Embedding Size	100	300	200
LSTM Hidden Size	150	250	150
LSTM Layers	1	2	1
Intermediate Layer Dimensions	200, 50	200, 50, 25	200, 50
End Model Training
Learning Rate	1e-2	1e-3	1e-3
${\mathcal{l}}_2$ Regularization	1e-4	1e-4	1e-3
Epochs	20	30	50
Dropout	0.25	0.25	0.1

Table 4:

Dataset split sizes and sub-task structure for the three fine-grained classification tasks on which we evaluate MeTaL.

	# Train	#Dev	# Test	Tree Depth	# Tasks	# Sources/Task
OntoNotes (NER)	62,547	350	345	2	3	11
TACRED (RE)	9,090	350	2174	3	7	9
OpenI (Doc)	2,630	200	378	2	3	19

C.4. Task Accuracies

For clarity, we present in Table 5 the individual task accuracies of both the learned MeTaL model and MV for each experiment. These accuracies are computed from the output of evaluating each model on the test set with ties broken randomly.

C.5. Ablation Study: Unipolar Correction and Joint Modeling

We perform an additional ablation to demonstrate the relative gains of modeling unipolar supervision sources and jointly modeling accuracies across multiple tasks with respect to the data programming (DP) baseline [27]. Results of this investigation are presented in Table 6. We observe an average improvement of 2.8 points using the unipolar correction (DP-UI), and an additional 1.3 points from joint modeling within MeTaL, resulting in an aggregate gain of 4.1 accuracy points over the data programming baseline.

Table 5:

Label model task accuracies for each task for for both our approach and majority vote (MeTaL/MV)

	OntoNotes	TACRED	OpenI
Task 1
MV	93.3	74.2	83.9
MeTaL	91.9	80.5	84.1
Task 2
MV	73.3	46.2	77.8
MeTaL	7S.6	6S.9	83.7
Task 3
MV	71.4	74.9	61.7
MeTaL	74.1	74.8	61.7
Task 4
MV	-	34.4	-
MeTaL	-	60.2	-
Task 5
MV	-	36.2	-
MeTaL	-	40.2	-
Task 6
MV	-	56.3	-
MeTaL	-	49.9	-
Task 6
MV	-	36.8	-
MeTaL	-	56.3	-

Table 6: Effect of Unipolar Correction.

We compare the micro accuracy (avg. over 10 trials) with 95% confidence intervals of a model trained using data programming (DP), data program with a unipolar correction (DP-UI), and our approach (MeTaL).

	OntoNotes (NER)	TACRED (RE)	OpenI (Doc)	Average
DP [28]	78.4 ± 1.2	49.0 ± 2.7	75.8 ± 0.9	67.7
DP-UI	81.0 ± 1.2	54.2 ± 2.6	76.4 ± 0.5	70.5
MeTaL	82.2 ± 0.8	56.7 ± 2.1	76.6 ± 0.4	71.8