Robustifying Likelihoods by Optimistically Re-weighting Data

Abstract

Likelihood-based inferences have been remarkably successful in wide-spanning application areas. However, even after due diligence in selecting a good model for the data at hand, there is inevitably some amount of model misspecification: outliers, data contamination or inappropriate parametric assumptions such as Gaussianity mean that most models are at best rough approximations of reality. A significant practical concern is that for certain inferences, even small amounts of model misspecification may have a substantial impact; a problem we refer to as brittleness. This article attempts to address the brittleness problem in likelihood-based inferences by choosing the most model friendly data generating process in a distance-based neighborhood of the empirical measure. This leads to a new Optimistically Weighted Likelihood (OWL), which robustifies the original likelihood by formally accounting for a small amount of model misspecification. Focusing on total variation (TV) neighborhoods, we study theoretical properties, develop estimation algorithms and illustrate the methodology in applications to mixture models and regression.

1. Introduction

When the likelihood is correctly specified, there is arguably no substitute for likelihood-based statistical inferences (see e.g. Zellner, 1988; Royall, 2017). However, if the likelihood is misspecified, inferences may be flawed (Huber, 1964; Tsou and Royall, 1995; Huber and Ronchetti, 2009; Hampel et al., 2005; Miller and Dunson, 2019). This has motivated methods for model comparison and goodness-of-fit assessment (see e.g. Huber-Carol et al., 2012; Claeskens and Hjort, 2008). A key is to verify that the assumed likelihood is consistent with the data at hand. Yet, even with substantial care in model assessment, some amount of model misspecification is inevitable.

Unfortunately, even slight model misspecification can have dire consequences in certain settings; a problem we refer to as brittleness. Brittleness can occur in various applications, including in high dimensional problems (e.g. Bradic et al., 2011; Bradic, 2016), in the presence of outliers and contaminating distributions (e.g. Huber, 1964; Huber and Ronchetti, 2009; Hampel et al., 2005), or for mixture models (e.g. Markatou, 2000; Chandra et al., 2023; Liu and Moitra, 2023; Cai et al., 2021). This article is focused on robustifying likelihoods to avert such brittleness.

Figure 1 illustrates brittleness and our proposed solution in the setting of model-based clustering with kernel mixture models. Here, most of the data are perfectly modeled by a mixture of two well-separated Gaussians. However, a small fraction of the data have been corrupted, and are instead drawn uniformly from an interval between the two modes. As the left panel demonstrates, the maximum likelihood estimate (MLE) is sensitive to the corrupted data. Our proposal is to instead maximize an optimistically weighted likelihood (OWL), which slightly perturbs the data to be more consistent with the assumed model. The left panel shows that maximizing the OWL reduces brittleness, while the right panel shows the corresponding optimistically re-weighted data.

Figure 1:

An example illustrating failures of maximum likelihood estimation (MLE) for misspecified models. The data consist of $n=1000$ observations randomly generated from an equally-weighted mixture of two Gaussians with means −2.5 and 2.5, and with standard deviation 1/ 4. 5% of the data were corrupted and drawn i.i.d uniform (−1,1). Left: The green dashed line denotes the MLE found using expectation maximization. The orange dashed line denotes the solution found by maximizing our optimistically weighted likelihood based on ${\epsilon }_0=0.05$. Right: Depicts the optimistically re-weighted data that were used in place of the observed data to compute the MLE.

Although in principle one can address the issue of misspecification by designing more flexible models, e.g. via mixture or nonparametric models (Taplin, 1993; Beath, 2018), such flexibility often comes at the expense of interpretability, identifiability, and efficiency, both computational and statistical. Moreover, there is always the risk that one’s model is not flexible enough and is therefore subject to the brittleness concerns outlined above. These issues motivate the core approach of the field of robust estimation: rather than designing a complicated model that fits all aspects of the data generating process, one instead fits a model that is only approximately correct, albeit with a different methodology. At the heart of many robust estimation methods is the concept of the weighted likelihood.

For a likelihood function $f$, the weighted likelihood replaces $L\left(x_{1:n}\mid \theta \right)=\prod _{i=1}^{n}f\left(x_i\mid \theta \right)$ by the weighted counterpart

$$L_{\mathbf{w}}\left(x_{1:n}\mid \theta \right)=\prod _{i=1}^{n}f{\left(x_i\mid \theta \right)}^{w_i}$$ (1)

for a collection of weights $\mathbf{w}={\left\{w_i\right\}}_{i=1}^n$ that typically depend on $\theta$ and the data. Estimation of $\theta$ and $\mathbf{w}$ often proceeds via iteratively reweighted maximum likelihood estimation (IRW-MLE) by repeating the following steps until convergence: a) maximize likelihood $\theta \mapsto L_{\mathbf{w}}\left(X_{1:n}\mid \theta \right)$ based on the current $\mathbf{w}$ and, b) update $\mathbf{w}$ using the current $\theta$ (Avella Medina and Ronchetti, 2015; Maronna et al., 2019). A variety of classical robust methods (Huber and Ronchetti, 2009; Hampel et al., 2005) fall in this framework, providing weights measuring the influence of each observation (e.g., Figure 8 in Mancini et al., 2005). Typical practice assumes $w_i=w\left(x_i,\theta \right)$ with $w(\cdot )$ chosen so that $w_i\approx 0$ for observations having very low likelihood under the presumed model; e.g., $w(x,\theta )$ is a function of the density $f(x\mid \theta )$ (Windham, 1995) or its cumulative distribution function (Field and Smith, 1994; Dupuis and Morgenthaler, 2002). See See Supplement S1 for more discussion about these methods.

Typical robust estimation methods focus on sensitivity to outliers, while our interest is in broader types of misspecification. In Figure S1 from Supplement S2, we illustrate how misspecification in the shape of the kernel can lead to errors in selecting the number of components in mixture models. We aim to develop a general weighted likelihood methodology that allows for model misspecification beyond outliers. We focus on robustness to a small amount of misspecification with respect to an appropriate metric $\mathit{d}$ in the space of probability distributions.

A well-studied approach to achieving robustness is minimum distance estimation (Wolfowitz, 1957; William, 1981), which attempts to find a model that minimizes the distance to the generating distribution with respect to the metric $\mathit{d}$. While this method can be shown to be robust to misspecification in $\mathit{d}$ (Donoho and Liu, 1988), it is often challenging to compute (e.g. Yatracos, 1985) beyond suitable $\varphi$-divergences (Lindsay, 1994). In continuous spaces, when $\mathit{d}$ is a $\varphi$-divergence with smooth $\varphi$, minimum distance estimation can be computed using weighted likelihood equations with $w(x,\theta )=\frac{A\left(\delta \right(x\mid \theta \left)\right)-A(-1)}{\delta (x\mid \theta )+1}$ where $A$ is the residual adjustment function corresponding to $\varphi$ and $\delta (x\mid \theta )=\hat{p}\left(x\right)/f(x\mid \theta )-1$ is the Pearson residual based on a density estimate $\hat{p}$ of the observed data (Basu and Lindsay, 1994; Basu et al., 1998; Markatou et al., 1998; Greco and Agostinelli, 2020). Unfortunately, most $\varphi$-divergences outside the Hellinger and total-variation (TV) cases are not metrics, making interpretation difficult. Moreover, the important case of TV distance is excluded by this work since the corresponding $\varphi$ is not smooth. Recent work (Cherief-Abdellatif and Alquier, 2020; Briol et al., 2019; Bernton et al., 2019) has provided tools for computing minimum distance estimators when $\mathit{d}$ is the maximum mean discrepancy (MMD) or Wasserstein distance, however this approach relies on generative models and does not make use of the likelihood.

Here, we develop a weighted likelihood methodology that can perform approximate minimum distance estimation, providing a recipe for robustifying Bayesian and frequentist approaches by re-weighting the likelihood (see Supplement S2). For non-negative weights $\mathbf{w}={\left\{w_i\right\}}_{i=1}^n$ that sum to $n$, our key observation is that the weighted likelihood (1) is an ordinary (i.e. un-weighted) likelihood for perturbed data represented by the weighted empirical distribution ${\hat{Q}}_{\mathbf{w}}=n^{-1}\sum _{i=1}^n{w}_i{\delta }_{x_i}$, where ${\delta }_x$ denotes the Dirac measure at $x$. Thus IRW-MLE simultaneously finds: i) an optimal data perturbation ${\hat{Q}}_{{\mathbf{w}}^{\mathrm{*}}}$, and ii) the ordinary maximum likelihood estimate $\hat{\theta }$ based on the perturbed data ${\hat{Q}}_{{\mathbf{w}}^{\mathrm{*}}}$. For approximate minimum distance estimation, we require the perturbations ${\hat{Q}}_{\mathbf{w}}$ to lie in a $(\mathit{d},ϵ)$ neighborhood around the empirical distribution of the observed data $\hat{P}=n^{-1}\sum _{i=1}^n{\delta }_{x_i}$ for a choice of $ϵ>0$ given by the user.

Our optimal weights ${\mathbf{w}}^{\mathrm{*}}$ are determined using the principle of optimism: in the absence of information other than constraint $\mathit{d}({\hat{Q}}_{\mathbf{w}},\hat{P})\le ϵ$, we choose the perturbation ${\hat{Q}}_{\mathbf{w}}$ closest to the model family in the Kullback Leibler (KL) sense. If ${\left\{P_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$ denotes our model family and KL is an estimator for the KL divergence, we (Algorithm 1) adapt the IRW-MLE procedure to solve the following Optimistic Kullback Leibler (OKL) minimization:

$$\underset{\theta \in \mathrm{\Theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{\mathbf{w}:\mathit{d}({\hat{Q}}_{\mathbf{w}},\hat{P})\le ϵ}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathrm{K}\mathrm{L}({\hat{Q}}_{\mathbf{w}}\mid P_{\theta }).$$ (2)

When $ϵ=0$, OKL minimization is equivalent to the usual MLE, and the choice $ϵ>0$ reflects the degree of misspecification. If $P_0$ is the true distribution generating $x_1,\dots ,x_n$, and $ϵ$ is greater than the degree of misspecification ${\epsilon }_0\doteq {\mathrm{m}\mathrm{i}\mathrm{n}}_{\theta \in \mathrm{\Theta }}\mathit{d}\left(P_0,P_{\theta }\right)$, then any global minimizer $\hat{\theta }$ of (2) can be asymptotically shown to lie in the set $\left\{\theta :\mathit{d}\left(P_0,P_{\theta }\right)\le ϵ\right\}$ (Proposition 1). Thus when $ϵ$ is tuned to approximate ${\epsilon }_0$ (Section 2.2.3), $\hat{\theta }$ will be a minimum distance estimator.

In this work, we show that the OKL is intimately related to the concept of the coarsened likelihood (Miller and Dunson, 2019) – a genuine likelihood that acknowledges small misspecification in terms of $\mathit{d}$. While even evaluating the coarsened likelihood requires the computation of a high-dimensional integral, we use techniques from large deviation theory (Dembo and Zeitouni, 2010) to show that the coarsened likelihood is asymptotically equivalent to the OKL. Thus, the OWL methodology can be seen as a bridge, connecting minimum distance estimation and weighted likelihood methods on one side and the coarsened likelihood approach on the other.

When $\mathit{d}(Q,P)=\int \mathrm{l}\mathrm{o}\mathrm{g}\frac{dQ}{dP}dQ$ is the KL divergence, the Lagrangian formulation of our OKL minimization problem corresponds to minimizing a Cressie-Read power-divergence between the observations and the model family (Ghosh and Basu, 2018). Since different computational strategies may be required across metrics, we focus on total variation (TV) distance for $\mathit{d}$, which is easy to interpret, is a proper metric, and is robust to outliers and contamination (Donoho and Liu, 1988). Further, TV based estimators have been shown to attain minimax estimation rates under Huber’s contamination model (Chen et al., 2016).

Coarsened likelihoods (Miller and Dunson, 2019) in general, and the principle of optimism used here in particular, are related to statistical learning from fuzzy or imprecise data (Hüllermeier and Cheng, 2015; Hüllermeier, 2014); the key idea is to treat observed data as imprecise to facilitate robust learning (Lienen and Hüllermeier, 2021). Among various approaches to robust inference in regression (e.g. She and Owen, 2011; Rousseeuw and Leroy, 2005; Avella-Medina and Ronchetti, 2018), Bondell and Stefanski (2013) implements weighted least squares with weights learned within an empirical likelihood framework subject to a target weighted squared residual error. While OKL minimization (2) finds weights in a fashion similar to this and other empirical likelihood approaches (e.g. Choi et al., 2000), our optimization problem arises from large deviation formulas and leads to weights that are constrained to lie within a small total-variation $\left({\ell }_1\right)$ distance of the uniform weight vector.

The remainder of this paper is structured as follows: Section 2 presents the OWL methodology, the associated alternating optimization scheme, and discusses some formal robustness guarantees. Section 3 demonstrates the asymptotic connection between OWL and coarsened inference. Section 4 presents a suite of simulation experiments for the OWL methodology in both regression and clustering tasks. Section 5 uses OWL to infer the average intent-to-treat effect in a micro-credit study (Angelucci et al., 2015), whose inference using ordinary least squares (OLS) was shown to be brittle to the removal of an handful of observations (Broderick et al., 2023). Finally, we provide a robust clustering application for single-cell RNAseq data in Supplement S12.

2. Optimistically Weighted Likelihoods

As Figure 1 shows, maximum likelihood estimation can be brittle when the data generating distribution $P_0$ lies outside, albeit suitably close, to the model family ${\left\{P_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$. To assuage the problem of brittleness under misspecification, we propose an Optimistically Weighted Likelihood (OWL) approach that iterates between (1) an optimistic (or model-aligned) re-weighting of the observed data points and (2) updating the parameter estimate by maximizing a weighted likelihood based on the current data weights.

In Section 2.1, we study this parameter inference methodology at the population level where we formally allow $P_0$ to be misspecified as long as it is close to the model family in the total-variation (TV) distance. Here we introduce the population level Optimistic Kullback Leibler (OKL) function with parameter $ϵ\in \left[\mathrm{0,1}\right]$ (Definition 3), and show that its minimizer will be a parameter $\theta$ for which $P_{\theta }$ is $ϵ$-close to $P_0$ in TV distance. We provide formal robustness guarantees for these minimizers by analyzing a bias-distortion curve.

Motivated by this population analysis, in Section 2.2, we derive the OWL-based parameter estimation methodology when only samples $x_1,\dots ,x_n$ from $P_0$ are available.

2.1. Population level Optimistic Kullback Leibler minimization

Let $𝒫\left(𝒳\right)$ denote the space of probability distributions on the data space $𝒳$. In principle, our methodology can accommodate misspecification in terms of a variety of probability metrics (e.g. MMD or Wasserstein) on $𝒫\left(𝒳\right)$, but here we mainly focus on the total variation (TV) distance for concreteness and interpretability. Let ${\mathrm{d}}_{\mathrm{T}\mathrm{V}}(P,Q)=\underset{A\subseteq 𝒳}{\mathrm{s}\mathrm{u}\mathrm{p}}|P(A)-Q(A)|$ denote the TV metric between two probability distributions $P,Q\in 𝒫(𝒳)$, where the supremum ranges over all measurable subsets $A$ of $𝒳$. Given a model family ${\left\{P_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}\subseteq 𝒫(𝒳)$, we assume that the data generating distribution $P_0\in 𝒫(𝒳)$ for the data population in question satisfies ${\mathrm{d}}_{\mathrm{T}\mathrm{V}}(P_0,P_{{\theta }^{\mathrm{*}}})\le ϵ$ for a known value $ϵ\ge 0$ and some unknown ${\theta }^{\mathrm{*}}\in \mathrm{\Theta }$. In other words, we make the following assumption (satisfied under Huber’s $ϵ$-contamination model (Huber, 1964) given by $P_0=(1-ϵ)P_{{\theta }^{\mathrm{*}}}+ϵC$ for any $C\in 𝒫(𝒳)$).

Assumption 2.1.

Given $ϵ\ge 0$ and the true data distribution $P_0$, the set of parameters ${\mathrm{\Theta }}_I=\left\{\theta \in \mathrm{\Theta }\mid {\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,P_{\theta }\right)\le ϵ\right\}$ is non-empty.

In terms of Liu and Lindsay (2009), Assumption 2.1 requires that our model be adequate for the data at level $ϵ$ in terms of the TV neighborhood. While Liu and Lindsay (2009) develop a likelihood-ratio test for this assumption, here we focus on estimating parameters by maximizing a natural likelihood (Section 3), leading to a different optimization problem.

In general, under Assumption 2.1, it may only be possible to identify the set ${\mathrm{\Theta }}_I$, rather than any particular ${\theta }^{\mathrm{*}}\in \mathrm{\Theta }$. Although such indeterminacy may be inherent, it is practically insignificant whenever $ϵ$ is sufficiently small so that the distinction between two elements from ${\mathrm{\Theta }}_I$ is irrelevant (Huber, 1964). In line with this insight, the goal throughout the rest of the paper will be to identify some parameter in ${\mathrm{\Theta }}_I$. Indeed, selecting $ϵ$ to approximate ${\mathrm{m}\mathrm{i}\mathrm{n}}_{\theta }{\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,P_{\theta }\right)$ (see Section 2.2.3) will recover a minimizer of the map $\theta \mapsto {\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,P_{\theta }\right)$.

At the population level, maximum likelihood parameter estimation amounts to minimizing the Kullback Leibler (KL) function $\theta \mapsto \mathrm{K}\mathrm{L}\left(P_0\mid P_{\theta }\right)$ on the parameter space $\mathrm{\Theta }$. Even under small amounts of misspecification, KL minimizers are very brittle since any minimizer of the KL function must place sufficient probability mass wherever $P_0$ does, including on outliers. In contrast, TV distance is far less sensitive to outliers and slight shifts in the shape of the distribution. Hence one may minimize $\theta \mapsto {\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,P_{\theta }\right)$ as a robust alternative, particularly under Assumption 2.1. However, direct minimization of TV distance over the parameter space $\mathrm{\Theta }$ is difficult to implement in practice due to the lack of suitable optimization primitives (e.g. maximum likelihood estimators) and the non-convex and non-smooth nature of the optimization problem (see e.g. Yatracos, 1985).

Instead, we modify the KL loss by minimizing its first argument over the $ϵ$-neighborhood of $P_0$ in the TV distance. The resulting function, which we term the Optimistic Kullback Leibler (OKL), is defined as follows.

Definition 2.1.

(Optimistic Kullback Leibler) Given $P_0$ and $ϵ>0$, the OKL function $I_{ϵ}:\mathrm{\Theta }\to [0,\mathrm{\infty }]$ is defined as:

$$I_{ϵ}(\theta )={\mathrm{i}\mathrm{n}\mathrm{f}}_{Q\in {ℬ}_{ϵ}\left(P_0\right)}\mathrm{K}\mathrm{L}\left(Q\mid P_{\theta }\right),$$ (3)

where ${ℬ}_{ϵ}\left(P_0\right)=\left\{Q\in 𝒫(𝒳)\mid {\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,Q\right)\le ϵ\right\}$ is the TV ball of radius $ϵ$ around $P_0$. If $I_{ϵ}(\theta )<\mathrm{\infty }$, the underlying optimization over ${ℬ}_{ϵ}\left(P_0\right)$ has a unique minimizer $Q^{\theta }$ called the I-projection (Csiszár, 1975).

The OKL function $\theta \mapsto I_{ϵ}(\theta )$ measures the fit of a model $P_{\theta }$ to the data $P_0$, allowing for a degree $\epsilon$ of data re-interpretation in the TV distance before assessing model fit. More precisely, since $I_{ϵ}(\theta )=\mathrm{K}\mathrm{L}(Q^{\theta }\mid P_{\theta })$, the OKL is equal to the KL divergence between the optimistic (i.e. most model aligned) distribution $Q^{\theta }\in {ℬ}_{ϵ}\left(P_0\right)$ in the neighborhood of $P_0$ and the model $P_{\theta }$. Here, $ϵ\ge 0$ regulates the permitted degree of re-interpreting the data by controlling the neighborhood size.

The OKL function can be used to find a parameter from the set ${\mathrm{\Theta }}_I$. Indeed, as illustrated in Figure 2 (right), the minimum value of zero for the OKL function is attained exactly on the set ${\mathrm{\Theta }}_I$ since $Q^{\theta }=P_{\theta }$ if and only if $\theta \in {\mathrm{\Theta }}_I$. However, the OKL can be non-convex as a function of $\theta$, so that calculating the global minimizer of OKL may not be straightforward. Fortunately, the OKL lends itself to a feasible alternating optimization scheme (Figure 2; left) that is guaranteed to decrease the OKL objective or reach a saddle point under regularity conditions.

Figure 2:

Population level description of OWL (left) and the OKL function (right). The point $P_{\theta }$ is labeled using $\theta \in \mathrm{\Theta }$. The inference problem is to find a point in ${\mathrm{\Theta }}_I\subseteq \mathrm{\Theta }$ where the model family intersects the $ϵ$-neighborhood ${ℬ}_{ϵ}$ of the data distribution $P_0$. The set ${\mathrm{\Theta }}_I$ is the set of minimizers of the OKL function $\theta \mapsto I_{ϵ}(\theta )$ (right). Starting from initial point ${\theta }_1\in \mathrm{\Theta }$, the OWL procedure finds a saddle point of the OKL function by iterating ${\theta }_{t+1}={argmin}_{\theta \in \mathrm{\Theta }}\mathrm{K}\mathrm{L}(Q^{{\theta }_t}\mid P_{\theta })$ for $t=\mathrm{1,2},\dots$, until convergence, where $Q^{\theta }={argmin}_{Q\in {ℬ}_{ϵ}}\mathrm{K}\mathrm{L}\left(Q\mid P_{\theta }\right)$ denotes the information projection (Amari (2016)) of the model $P_{\theta }$ on the total variation (TV) neighborhood ${ℬ}_{ϵ}$ of $P_0$. Iterations alternate between I-projection and weighted likelihood estimation steps, illustrated via solid and dashed lines.

We now summarize formal robustness guarantees for the global minimizers of the OKL function against perturbations of $P_0$ in small TV-neighborhoods around the model family; details can be found in Supplement S4. We call the mapping from $P_0$ to the global minimizers of the OKL function as the OWL functional ${𝒯}_{ϵ}:𝒫(𝒳)\to 2^{\mathrm{\Theta }}$, defined as ${𝒯}_{ϵ}\left(P_0\right)\doteq arg{min}_{\theta \in \mathrm{\Theta }}{I}_{ϵ}(\theta )$ where $2^{\mathrm{\Theta }}$ denotes the power-set of $\mathrm{\Theta }$. Motivated by the robustness theory for minimum distance functionals (Donoho and Liu, 1988), when $P_0=P_{{\theta }^{\mathrm{*}}}$ lies on the model family, we bound the growth of a bias-distortion (BD) curve $\delta \mapsto {\mathrm{B}\mathrm{D}}_{ϵ}\left(\delta \mid P_0\right)$ defined as ${\mathrm{B}\mathrm{D}}_{ϵ}\left(\delta \mid P_0\right)\doteq \underset{\begin{array}{c}P\in 𝒫(𝒳)\\ {\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P,P_0\right)\le \delta \end{array}}{\mathrm{s}\mathrm{u}\mathrm{p}}\mathrm{H}\mathrm{a}\mathrm{u}\mathrm{s}\left({𝒯}_{ϵ}(P),{𝒯}_{ϵ}\left(P_0\right)\right)$, where Haus denotes the Hausdorff distance on $2^{\mathrm{\Theta }}$. Indeed, classically important robustness indicators like sensitivity and qualitative-robustness are properties about the growth of a suitable BD curve when $\delta$ is close to zero, and the breakdown-point is the point at which the BD curve has a vertical asymptote (Donoho and Liu, 1988, Figure 1). Given this setup, under suitable regularity conditions, our two main results in Supplement S4 show that the OWL functional is qualitatively-robust with finite sensitivity (corollary of Lemma S8) and has a breakdown-point of at least $ϵ$ whenever $ϵ$ is less than half of the best breakdown-point for the model family among all Fisher consistent functionals (Lemma S8).

2.2. Optimistically Weighted Likelihood (OWL) estimation

Here we extend the population level methodology from Section 2.1 to handle the practical case when samples $x_1,\dots ,x_n\stackrel{\mathit{i.i.d}.}{~}{P}_0$ are available, providing a computable approximation to the alternating procedure in Figure 2. Particularly, the I-projection step is now approximated by a convex optimization problem over weight vectors constrained to lie within the intersection of the $n$-dimensional probability simplex and the ${\ell }_1$ ball of radius $2ϵ$ around the uniform probability vector; these optimal weights can be interpreted as an optimistic re-weighting of the original data points $x_1,\dots ,x_n$ to match the current model estimate. Given these weights, a new parameter estimate is then found by maximizing the weighted likelihood. We call this the Optimistically Weighted Likelihood (OWL) method.

2.2.1. Approximating OKL by a finite dimensional optimization problem

Given observed data $x_1,\dots ,x_n\stackrel{\mathit{i.i.d}.}{~}{P}_0$, we start by approximating the OKL function $I_{ϵ}(\theta )$ in terms of a finite dimensional optimization problem. Henceforth, let us assume that the model family ${\left\{P_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$ and measure $P_0$ have densities ${\left\{p_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$ and $p_0$ with respect to a common measure $\lambda$. We will focus on two cases of interest: when $𝒳$ is a discrete space and $\lambda$ is the counting measure, and when $𝒳={\mathbb{R}}^d$ and $\lambda$ is the Lebesgue measure.

When $𝒳$ is discrete, we look to solve the optimization problem in eq. (3) over data re-weighting $Q={\sum }_{i=1}^n{w}_i{\delta }_{x_i}$ as the weight vector $w=\left(w_1,\dots ,w_n\right)$ varies over the $n$-dimensional probability simplex ${\mathrm{\Delta }}_n$ and satisfies the TV constraint $\frac{1}{2}\Vert w-o\Vert 1\le ϵ$ where $o=(1/n,\dots ,1/n)\in {\mathrm{\Delta }}_n$. Formally, our finite space OKL approximation is given by

$${\hat{I}}_{ϵ,\text{fin}}(\theta )=\underset{\begin{array}{c}w\in {\mathrm{\Delta }}_n\\ \frac{1}{2}\Vert w-o{\Vert }_1\le ϵ\end{array}}{\mathrm{i}\mathrm{n}\mathrm{f}}\sum _{i=1}^n{w}_i\mathrm{l}\mathrm{o}\mathrm{g}\frac{n{w}_i{\hat{p}}_{\text{fin}}\left(x_i\right)}{p_{\theta }\left(x_i\right)},$$ (4)

where ${\hat{p}}_{\text{fin}}(y)=\frac{\left|\left\{i\in \left[n\right]\mid x_i=y\right\}\right|}{n}$ is the histogram estimator for $p_0$ based on observed data. An application of the log sum inequality (Cover and Thomas, 2006, Theorem 2.7.1) shows that the weights that solve eq. (4) have the appealing and natural property that $w_i=w_j$ whenever $x_i=x_j$. Moreover, when the support of $p_0$ is finite and contains the support of $p_{\theta },{\hat{I}}_{ϵ,\text{fin}}(\theta )$ converges to $I_{ϵ}(\theta )$ at rate $n^{-1/2}$, as demonstrated by the following result (Supplement S5). A modified proof can establish the weaker result of consistency of ${\hat{I}}_{ϵ,\text{fin}}(\theta )$ even when the support of $p_0$ is (countably) infinite, provided $\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_{\theta }\right)\subseteq \mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)$. The latter condition is mild and holds whenever the model family has a fixed support (e.g. exponential family) and $p_0$ is a Huber’s contamination from the model.

Theorem 1.

Suppose that $I_{{ϵ}_0}(\theta )<\mathrm{\infty }$ for some ${ϵ}_0>0$ and pick $\delta >0$ and $ϵ>{ϵ}_0$. If $\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_{\theta }\right)\subseteq \mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)$, $\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)$ is finite, and $x_1,\dots ,x_n\stackrel{\mathit{i.i.d.}}{~}{p}_0$, then with probability at least $1-\delta$,

$$|I_{ϵ}(\theta )-{\hat{I}}_{ϵ,\text{fin}}(\theta )|\le O\left(\frac{\left|\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)\right|}{ϵ-{ϵ}_0}\sqrt{\frac{1}{n}\log \frac{\left|\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)\right|}{\delta }}\right),$$

where $\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}(p)=\{x\in 𝒳:p(x)>0\}$.

When $𝒳={\mathbb{R}}^d$, the above approximation strategy needs to be modified, since $\mathrm{K}\mathrm{L}\left(Q\mid P_{\theta }\right)$ is unbounded whenever $P_{\theta }$ is a continuous distribution and $Q=\sum _{i=1}^n{w}_i{\delta }_{x_i}$ is a discrete distribution. However, using a suitable kernel $\kappa :𝒳\times 𝒳\to [0,\mathrm{\infty })$ (e.g. ${\kappa }_h(x,y)=\frac{1}{(\sqrt{2\pi }h{)}^d}{e}^{-\frac{{\Vert x-y\Vert }^2}{2h^2}}$) one can establish the continuous space approximation (Supplement S6)

$${\hat{I}}_{ϵ}(\theta )=\underset{\begin{array}{c}w\in {\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n\\ \frac{1}{2}{\Vert w-o\Vert }_1\le ϵ\end{array}}{\mathrm{i}\mathrm{n}\mathrm{f}}\sum _{i=1}^n{w}_i\mathrm{l}\mathrm{o}\mathrm{g}\frac{n{w}_i\hat{p}\left(x_i\right)}{p_{\theta }\left(x_i\right)}$$ (5)

where $\hat{p}$ is a density estimator for $p_0$ based on $x_1,\dots ,x_n,A$ is an $n\times n$ matrix with entries $A_{ij}=\frac{\kappa \left(x_i,x_j\right)}{n\hat{p}\left(x_i\right)}$, and ${\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n=A{\mathrm{\Delta }}_n$ is the image of the $n$-dimensional probability simplex under linear operator $A$. We will typically take $\hat{p}(\cdot )=\frac{1}{n}\sum _{j=1}^n\kappa \left(\cdot ,x_j\right)$ to be the kernel-density estimate based on the same kernel $\kappa$, in which case $A$ is the stochastic matrix obtained by normalizing the rows of the kernel matrix $K={\left(\kappa \left(x_i,x_j\right)\right)}_{i,j\in \left[n\right]}$ to sum to one.

The continuous space approximation in eq. (5) yields the finite space approximation in eq. (4) as a special case when $\kappa (x,y)=I\{x=y\}$ is taken to be the indicator kernel. The weights vectors in ${\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n$ are always non-negative and approximately sum to one for large values of $n$, since $\sum _{i=1}^n{A}_{ij}=\frac{1}{n}\sum _{i=1}^n\frac{\kappa \left(x_i,x_j\right)}{\hat{p}\left(x_i\right)}\approx \int \frac{\kappa \left(x,x_j\right)}{\hat{p}(x)}{p}_0(x)dx\approx \int \kappa \left(x,x_j\right)dx=1$.

Now we briefly describe the large sample convergence of ${\hat{I}}_{ϵ}(\theta )$ to $I_{ϵ}(\theta )$ shown in Supplement S6.. Proving this exact statement is technically challenging, as it requires proving uniform-convergence of the objective in eq. (5) when the weights are allowed to vary over the entire range ${\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n$. To get around this, we restrict the optimization domain to ${\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n^{\beta }=A{\mathrm{\Delta }}_n^{\beta }$ where ${\mathrm{\Delta }}_n^{\beta }=\{\left(v_1,\dots ,v_n\right)\in {\mathrm{\Delta }}_n:v_i\in [\frac{\beta }{n},\frac{1}{n\beta }]\}$ for a suitably small constant $\beta >0$. Thus, the estimator that we theoretically study is given by

$${\hat{I}}_{ϵ,\beta }(\theta )=\underset{\begin{array}{c}w\in {\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n^{\beta }\\ \frac{1}{2}{\Vert w-o\Vert }_1\le ϵ\end{array}}{\mathrm{i}\mathrm{n}\mathrm{f}}\sum _{i=1}^n{w}_i\mathrm{l}\mathrm{o}\mathrm{g}\frac{n{w}_i\hat{p}\left(x_i\right)}{p_{\theta }\left(x_i\right)}.$$ (6)

Given this change, we can show the following result.

Theorem 2.

Suppose $\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_0\right)=\mathrm{s}\mathrm{u}\mathrm{p}\mathrm{p}\left(p_{\theta }\right)=𝒳$ is a compact subset of ${\mathbb{R}}^d$ and there exists a constant $\gamma >0$ such that $p_0(x),p_{\theta }(x)\in [\gamma ,1/\gamma ]$ for all $x\in 𝒳$. Suppose that we use the probability kernel $\kappa (x,y)=\frac{1}{h^d}\varphi (|x-y|/h)$ having bandwidth $h>0$, where $\kappa$ is positive semi-definite and $\varphi$ is a bounded function with exponentially decaying tails. Assume that $p_0$ and $\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\theta }$ are $\alpha$-Hölder smooth over $𝒳$, and suppose that we use the clipped density estimator $\hat{p}(x)=\mathrm{m}\mathrm{i}\mathrm{n}\left(\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{n}\sum _{i=1}^n\kappa \left(x_i,x\right),\gamma \right),1/\gamma \right)$. Then for any constant $0<\beta \le {\gamma }^2/4$

$$\left|{\hat{I}}_{ϵ,\beta }\left(\theta \right)-I_{ϵ}\left(\theta \right)\right|\le \tilde{O}\left(n^{-\frac{1}{2}}{h}^{-d}+h^{\frac{\alpha }{2}}+\psi \left(\sqrt{h}\right)\right),$$

with probability at least $1-1/n$, where $\psi \left(r\right)=\frac{\lambda \left(𝒳\backslash {𝒳}_{-r}\right)}{\lambda \left(𝒳\right)},\lambda (\cdot )$ is the d-dimensional Lebesgue measure, ${𝒳}_{-r}=\{x\in 𝒳:B(x,r)\subseteq 𝒳\}$, and $\tilde{O}(\cdot )$ hides constants and logarithmic factors.

Here $\psi (r)$ measures the fraction of the volume of $𝒳$ that is contained in the envelope of width $r$ closest to the boundary. For well-behaved sets, we expect $\psi (r)$ to decrease to 0 as $r\to 0$. For example, if $𝒳$ is a $d$-dimensional ball of radius $r_0$, then $\psi (r)=1-{(1-\frac{r}{r_0})}^d$.

We prove our theory with the truncated estimator ${\hat{I}}_{ϵ,\beta }(\theta )$ instead of ${\hat{I}}_{ϵ}(\theta )$ for technical reasons. By a suitable version of the sandwiching lemma (Supplement S3.2), under the assumptions of Theorem 2, we conjecture that the optimal weights in (5) lie in the set ${\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}}_n^{\beta }$ for a small enough constant $\beta >0$, in which case we will have ${\hat{I}}_{ϵ,\beta }(\theta )={\hat{I}}_{ϵ}(\theta )$.

Given the large sample convergence of ${\hat{I}}_{ϵ}(\theta )$ to $I_{ϵ}(\theta )$ for every $\theta \in \mathrm{\Theta }$, one expects that any estimator $\hat{\theta }\in \mathrm{\Theta }$ that is a global minimizer of ${\hat{I}}_{ϵ}$ will approach the set ${\mathrm{\Theta }}_I$ (Assumption 2.1) as $n\to \mathrm{\infty }$. A qualitative version of this result, proved in Supplement S7, is shown next.

Proposition 1.

Suppose that (i) $𝒳$ and $\mathrm{\Theta }$ are compact metric spaces, and $(x,\theta )\mapsto \mathrm{l}\mathrm{o}\mathrm{g}{p}_{\theta }(x)$ is a continuous function from $𝒳\times \mathrm{\Theta }\to \mathbb{R}$. Let $ϵ\ge 0$ be such that (ii) Assumption 2.1 is satisfied. Let $x_1,\dots ,x_n\stackrel{\mathit{i.i.d.}}{~}{p}_0$, and suppose that (iii) for any fixed $\theta \in \mathrm{\Theta }$, we have ${\hat{I}}_{ϵ}(\theta )\stackrel{P}{\to }{I}_{ϵ}(\theta )$ as $n\to \mathrm{\infty }$. If an estimator $\hat{\theta }\in \mathrm{\Theta }$ satisfies $\left|{\hat{I}}_{ϵ}(\hat{\theta })-{\mathrm{m}\mathrm{i}\mathrm{n}}_{\theta \in \mathrm{\Theta }}{\hat{I}}_{ϵ}(\theta )\right|\stackrel{P}{\to }0$, then for every $\delta >0$

$$\underset{n\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}\mathbb{P}\left({\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_{\hat{\theta }},P_0\right)>ϵ+\delta \right)=0.$$

Theorem 1 and Theorem 2 (with the conjectured $\beta =0$) provide sufficient conditions for condition (iii) in Proposition 1 to hold. Condition (i) is primarily used to show that the convergence in (iii) is uniform, i.e. $\underset{\theta \in \mathrm{\Theta }}{\mathrm{s}\mathrm{u}\mathrm{p}}|{\hat{I}}_{ϵ}(\theta )-I_{ϵ}(\theta )|\stackrel{P}{\to }0$. Finally, condition (ii) is satisfied when the data $P_0=(1-ϵ)P_{{\theta }^{\mathrm{*}}}+ϵC$ is a contamination. Proposition 1 and triangle inequality then bound the model estimation error: ${\mathrm{d}}_{\mathrm{T}\mathrm{V}}(P_{\hat{\theta }},P_{{\theta }^{\mathrm{*}}})\le 2ϵ+o_P(1)$.

2.2.2. OWL Methodology

Based on the computable approximation ${\hat{I}}_{ϵ}$ (or ${\hat{I}}_{ϵ,\mathrm{f}\text{in}}$) to the OKL function, Algorithm 1 uses an alternating optimization to minimize ${\hat{I}}_{ϵ}(\theta )$ over $\theta$. The procedure simultaneously estimates a minimizer $\hat{\theta }$ and optimistic weights $w_1,\dots ,w_n\ge 0$ that sum to $n$.

In Supplement S8, we expand further on the computational details for the $\theta$-step and $w$-step in Algorithm 1. In particular, the $w$-step is a convex optimization problem which we can solve by using the consensus ADMM algorithm (Boyd et al., 2011; Parikh and Boyd, 2014) based on proximal operators that can be efficiently computed. Further, the $\theta$-step corresponds to maximizing a weighted likelihood, which can be performed for many models through simple modifications of procedures for the corresponding maximum likelihood estimation. As shown in Supplement S8, the latter computation is easy to perform exactly for exponential families, while for mixture models we can use a weighted variant of the ‘hard EM’ algorithm (Samdani et al., 2012).

In Supplement S10, we show that the iterates ${\left\{{\theta }_t\right\}}_{t\ge 1}$ of Algorithm 1 will decrease the objective value $\theta \mapsto {\hat{I}}_{ϵ}(\theta )$ at each step, and that Algorithm 1 is an instance of the well-studied Majorization-Minimization (MM) (Hunter and Lange, 2004) and Difference of Convex (DC) (Le Thi and Pham Dinh, 2018) class of algorithms. We use this to analyze the convergence of ${\left\{{\theta }_t\right\}}_{t\ge 1}$ when ${\left\{p_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$ is an exponential family (see Remark S10.1).

Algorithm 1.

OWL Methodology

Input: Model ${\left\{p_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}$, radius $ϵ\ge 0$, kernel $\kappa$, initial ${\theta }_1\in \mathrm{\Theta }$, and iteration limit $T$. for $t=1,\dots ,T$ do $w$-step: Find $w_t=\left(w_{t,1},\dots ,w_{t,n}\right)\in {\mathbb{R}}_{\ge 0}^n$ that minimizes eq. (5) for $\theta ={\theta }_t$. $\theta$-step: Find ${\theta }_{t+1}$ that maximizes the weighted likelihood $\theta \mapsto \sum _{i=1}^n{w}_{t,i}\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\theta }\left(x_i\right)$. end for Output: The robust parameter estimate ${\theta }_T$ and the data weights $n{w}_T$.

In practice, when the data lie in a continuous space, we often avoid using the kernel-based estimator eq. (5) to determine the weights in the $w$-step of Algorithm 1 because it greatly slows down the computation (see Supplement S8.1.3). Instead, we often perform the $w$-step by solving the unkernelized optimization problem using $\kappa (x,y)=I\{x=y\}$. We demonstrate via simulations (Section 4) that the unkernelized version of the OWL procedure has equally good performance compared to the kernelized version with a suitably tuned bandwidth. The theoretical impact of this approximation can be explored by studying the limiting behavior of the OKL estimator for suitably fixed choices of $\kappa$ as $n\to \mathrm{\infty }$.

2.2.3. Setting the corruption fraction $\mathit{ϵ}$

So far, we have assumed that the parameter $ϵ\in (\mathrm{0,1})$, which can be interpreted as the fraction of corrupted samples in the population distribution, is fixed at a known value that satisfies Assumption 2.1. Now let us see how the population level analysis (Section 2.1) can inform our choice of $ϵ$. Assumption 2.1 is satisfied as long as $ϵ\ge {\epsilon }_0$, where

$${\epsilon }_0=\underset{\theta \in \mathrm{\Theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}{\mathrm{d}}_{\mathrm{T}\mathrm{V}}\left(P_0,P_{\theta }\right)=\left\{ϵ\in [\mathrm{0,1}]:\underset{\theta \in \mathrm{\Theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}{I}_{ϵ}(\theta )=0\right\}.$$

Hence, in principle, we could set $ϵ={\epsilon }_0$ to use OWL to perform minimum-TV estimation (Yatracos, 1985), which has the following advantages: (1) while directly minimizing TV distance is computationally intractable, the OWL methodology decomposes this problem into alternating convex optimization and weighted MLE steps, both of which are standard problems that often tend to be well-behaved, and (2) the OWL methodology provides us with weight vectors that can indicate outlying observations and relates minimum TV-estimation to likelihood based inference.

In order to choose $ϵ\approx {\epsilon }_0$ in practice, we define the function $\hat{g}(ϵ)={\hat{I}}_{ϵ}({\hat{\theta }}_{ϵ})$, where ${\hat{\theta }}_{ϵ}$ is the parameter estimate computed by the OWL procedure for a given $ϵ$. At the population level, the corresponding function $g(ϵ)={\mathrm{m}\mathrm{i}\mathrm{n}}_{\theta \in \mathrm{\Theta }}{I}_{ϵ}(\theta )$ is monotonically decreasing in $ϵ$ until $ϵ={\epsilon }_0$, at which point it remains at 0. This introduces a kink, or elbow, at ${\epsilon }_0$ that we hope to identify in the sample estimate $\hat{g}$. Thus, our $ϵ$ search procedure is to compute $\hat{g}$ over a fixed grid of $ϵ$-values, smooth the resulting grid, and then select amongst the points of largest curvature (computed numerically), where the curvature of a twice-differentiable function $f$ at a point $x$ is given by $f^{\prime \prime }(x)/{(1+f^{\prime }(x{)}^2)}^{1.5}$ (Satopaa et al., 2011). Despite the various approximations involved, our simulation results (Section 4) show that the OWL procedure with such a tuned value of $ϵ$ provides almost identical performance when compared with the OWL procedure with the true value of $ϵ$.

2.3. OWL extension to non-identically-distributed data

While the population level analysis and theoretical results for the OKL estimator were derived under the assumption that data are generated i.i.d. from a distribution $P_0$, the OWL procedure can be adapted to robustify likelihood based inference in the setting where the data are conditionally independent, but not necessarily identically distributed.

Suppose data $z_1,\dots ,z_n\in 𝒵$ are conditionally independent, with the likelihood having the product form $p_{\theta }\left(z_{1:n}\right)=\prod _{i=1}^n{p}_{\theta ,i}\left(z_i\right)$, for known functions ${\left\{p_{\theta ,i}\right\}}_{i=1}^n$. For example, if $z_i=\left(y_i,x_i\right)\in \mathbb{R}\times 𝒳$ for $i=1,\dots ,n$, this includes the case of regression models ${\left\{q_{\theta }(y\mid x)\right\}}_{\theta \in \mathrm{\Theta }}$ under the setup $p_{\theta ,i}\left(z_i\right)=q_{\theta }\left(y_i\mid x_i\right)$. Another example of this setup includes mixture models if we expand the parameter space to also include cluster assignments (see Supplement S8.2).

To robustify inference based on the product likelihood $p_{\theta }\left(z_{1:n}\right)=\prod _{i=1}^n{p}_{\theta ,i}\left(z_i\right)$, we can replace the $w$- and $\theta$- steps in Algorithm 1 by analogous steps in the product likelihood case. In particular, the modified $w$-step is given by

$$w_t=\underset{\begin{array}{c}w\in {\mathrm{\Delta }}_n\\ \frac{1}{2}{\Vert w-o\Vert }_1\le ϵ\end{array}}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\{-{\sum }_{i=1}^n{w}_i\mathrm{l}\mathrm{o}\mathrm{g}{p}_{{\theta }_t,i}\left(x_i\right)+{\sum }_{i=1}^n{w}_i\mathrm{l}\mathrm{o}\mathrm{g}{w}_i\}$$

and the modified $\theta$-step is given by

$${\theta }_{t+1}={argmin}_{\theta \in \mathrm{\Theta }}\sum _{i=1}^n{w}_{t,i}\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\theta ,i}\left(x_i\right).$$

Despite our lack of theory in the non-identically-distributed case, we continue to see good empirical performance of OWL (see Section 4, Section 5, and Supplement S12).

3. Asymptotic connection to coarsened inference

The development of the OWL methodology in Section 2 followed from a presumed form of misspecification given by Assumption 2.1. An alternative way to frame and address such misspecifications in a probabilistic framework was proposed by Miller and Dunson (2019) who introduced a Bayesian methodology centered around the concept of a coarsened likelihood defined as

$$L_{ϵ}\left(\theta \mid x_{1:n}\right)\doteq {\mathbb{P}}_{\theta }\left(\mathit{d}({\hat{P}}_{Z_{1:n}},{\hat{P}}_{x_{1:n}})\le ϵ\right),$$ (7)

where $\mathit{d}$ is a suitably chosen discrepancy between empirical probability measures. Here, ${\hat{P}}_{x_{1:n}}=n^{-1}\sum _{i=1}^n{\delta }_{x_i}$ denotes the empirical distribution of data $x_{1:n}$, and the probability is computed under ${\mathbb{P}}_{\theta }$—the distribution underlying the artificial data $Z_1,\dots Z_n\stackrel{\mathit{i.i.d}.}{~}{P}_{\theta }$ from which the random measure ${\hat{P}}_{Z_{1:n}}=n^{-1}\sum _{i=1}^n{\delta }_{Z_i}$ is constructed. The coarsened likelihood implicitly captures the likelihood of a probabilistic procedure in which idealized data are first generated by some model ${\mathbb{P}}_{\theta }$ in the model class under consideration, but are then corrupted in such a way that the discrepancy between empirical measures of the idealized data and the observed data is bounded by $ϵ$.

When $\mathit{d}$ is an estimator for the KL-divergence and an exponential prior is placed on $ϵ$, Miller and Dunson (2019) showed that the Bayes posterior based on $L_{ϵ}\left(\theta \mid x_{1:n}\right)$ could be approximated by raising the likelihood to a power less than one in the formula for the standard posterior. However, to obtain a robustified alternative to maximum likelihood estimation, one may wish to maximize $\theta \mapsto L_{ϵ}\left(\theta \mid x_{1:n}\right)$ directly for a choice of $\mathit{d}$ that guarantees robustness (e.g. Maximum Mean Discrepancy or the TV distance). Such an approach would in general be quite challenging since evaluating eq. (7) corresponds to computing a high-dimensional integral.

In this section, we show that for large $n$, the coarsened likelihood can be approximately maximized via the OWL methodology when $\mathit{d}$ is an estimator for the TV distance. Specifically, if the observed data $x_1,\dots ,x_n$ are generated i.i.d. from some distribution $P_0$ and $\mathit{d}$ satisfies some regularity conditions, then the quantity $-\frac{1}{n}\mathrm{l}\mathrm{o}\mathrm{g}{L}_{ϵ}\left(\theta \mid x_{1-n}\right)$ asymptotically converges as $n\to \mathrm{\infty }$ to a variant of $I_{ϵ}(\theta )$ based on $\mathit{d}$. Hence, the OWL methodology asymptotically maximizes the coarsened likelihood $\theta \mapsto L_{ϵ}\left(\theta \mid x_{1:n}\right)$.

We state this asymptotic connection first for finite spaces and then for continuous spaces. These results rely on Sanov’s theorem from large deviation theory (Dembo and Zeitouni, 2010) and are proved in Supplement S9.

3.1. Asymptotic connection in finite spaces

Let $𝒳$ be a finite set and denote the space of probability distributions on $𝒳$ by the simplex ${\mathrm{\Delta }}_{𝒳}\doteq \{q\in {[\mathrm{0,1}]}^{𝒳}\mid \sum _{\chi \in 𝒳}q(x)=1\}$. Let ${\left\{p_{\theta }\right\}}_{\theta \in \mathrm{\Theta }}\subseteq {\mathrm{\Delta }}_{𝒳}$ denote the collection of model distributions, and $p_0\in {\mathrm{\Delta }}_{𝒳}$ denote the true data generating distribution. To connect the OKL with the coarsened likelihood in eq. (7), we take $\mathit{d}(p,q)=\frac{1}{2}\Vert p-q\Vert 1$.

Given this setting, we can show that $-\frac{1}{n}\mathrm{l}\mathrm{o}\mathrm{g}{L}_{ϵ}\left(\theta \mid x_{1:n}\right)$ converges in probability to the OKL function $I_{ϵ}(\theta )$ at a rate of $n^{-1/2}$, as demonstrated by the following theorem.

Theorem 3.

Suppose that $I_{{ϵ}_0}(\theta )<\mathrm{\infty }$ for some ${ϵ}_0>0$ and let $\delta >0$. If $ϵ>{ϵ}_0$ and $x_1,x_2,\dots ,x_n\stackrel{\mathit{i.i.d.}}{~}{p}_0$, then with probability at least $1-\delta$,

$$\left|I_{ϵ}(\theta )+\frac{1}{n}\mathrm{l}\mathrm{o}\mathrm{g}{L}_{ϵ}\left(\theta \mid x_{1:n}\right)\right|\le O\left(\frac{\left|𝒳\right|}{ϵ-{ϵ}_0}\sqrt{\frac{1}{n}\log \frac{1}{\delta }}+\frac{\left|𝒳\right|}{n}\left(\left|𝒳\right|+\log \left(n\right)+\frac{1}{ϵ-{ϵ}_0}\right)\right).$$

Theorem 1 and Theorem 3 together show that, in the large sample limit, the OWL methodology and coarsened likelihood philosophy are two sides of the same coin: they both provide approximations of the OKL and, in turn, must approximate each other.

3.2. Asymptotic connection in continuous spaces

Suppose $𝒳$ is a Polish space (e.g. $𝒳={\mathbb{R}}^d$). Similar to the finite case, we can establish the following asymptotics for the coarsened likelihood for a suitable class of discrepancies $\mathit{d}$, which includes the Wasserstein distance, Maximum Mean Discrepancy with suitable choice of kernels (Simon-Gabriel and Schölkopf, 2018), and the smoothed TV distance (Definition S9.1 in Supplement S9.3).

Theorem 4.

Suppose $I_{{ϵ}_0}(\theta )<\mathrm{\infty }$ for some ${ϵ}_0>0$ and $\mathit{d}:𝒫(𝒳)\times 𝒫(𝒳)\to [0,\mathrm{\infty })$ is a pseudometric that is convex in its arguments and continuous with respect to the weak convergence topology on $𝒫(𝒳)$. If $ϵ>{ϵ}_0$ and $x_1,\dots ,x_n\stackrel{\mathit{i.i.d.}}{~}{P}_0$, then

$$-\frac{1}{n}\log L_{ϵ}\left(\theta \mid x_{1:n}\right)\stackrel{P}{\to }\underset{\begin{array}{c}Q\in 𝒫\left(𝒳\right)\\ \mathbf{d}\left(Q,P_0\right)\le ϵ\end{array}}{\mathrm{i}\mathrm{n}\mathrm{f}}\mathrm{K}\mathrm{L}\left(Q\mid P_{\theta }\right)\mathrm{a}\mathrm{s}n\to \mathrm{\infty }.$$

Recall that the limiting expression in the above theorem has the same form as that of the OKL function given in eq. (3). However, in order to establish connection between the OKL function and the coarsened likelihood, unlike in the finite case, we cannot merely take the discrepancy $\mathit{d}$ in the coarsened likelihood to be the TV distance, since the TV distance between the two empirical distributions in eq. (7) will almost surely be equal to one. Instead, we will take $\mathit{d}$ to be a smoothed version of TV distance calculated by first convolving the empirical measures with a smooth kernel function $K_h:𝒳\times 𝒳\to [0,\mathrm{\infty })$ indexed by a bandwidth parameter $h>0$. Further details can be found in Supplement S9.3.

4. Simulation Examples

We now demonstrate the OWL methodology in simulated examples with artificially injected corruptions. In each simulation, we considered two methods for choosing the points to corrupt: (i) max-likelihood corruption where we fit a maximum likelihood estimate to the uncorrupted data and select the points with the highest likelihood; and (ii) random corruption where we choose the points to corrupt uniformly at random. We ran each simulation with 50 initial seeds, plotting the mean performance and its 95% confidence band. For clarity and space, we present only the results for max-likelihood corruptions in this section, and we defer the results for randomly-selected corruptions to Supplement S11.

In all comparisons, OWL refers to our methodology with the data-based choice of the corruption fraction $ϵ$ as described in Section 2.2.3, while OWL ($ϵ$ known) refers to our methodology with $ϵ$ equal to the true level of corruption in the data. MLE refers to the standard maximum likelihood estimate. The Pearson residuals baseline is the method from Markatou et al. (1998) based on the Hellinger residual adjustment function: a weighted-likelihood based method aimed at performing minimum Hellinger-distance estimation. For settings where the outcomes were continuous, the Pearson residual density estimate was computed via kernel density estimation, with the bandwidth parameter selected to minimize the empirical Hellinger distance between the final model and the density estimate. For the linear regression baselines, ridge regression was performed with L2 penalty chosen via cross validation, and Huber regression used the commonly-accepted Huber penalty of 1.345 (Huber, 1964). For the logistic regression baselines, L2-regularized MLE selected the L2 penalty via cross validation. For both regression settings, Random Sample Consensus (RANSAC) utilized the ground-truth corruption fraction.

Gaussian modeling.

Our first simulation fit a multivariate normal distribution to data generated from a $p$-dimensional spherical normal distribution. The ground truth mean was drawn uniformly from $[-10,10{]}^p$, and the corrupted data points were also drawn uniformly from $[-10,10{]}^p$. The total number of points was 200, and we measured the dimension-normalized mean-squared error (MSE) of ground-truth mean parameter recovery.

Figure 3a shows the results of the Gaussian simulation. In particular, we see that both the kernelized and the unkernelized versions of OWL had almost identical performance in this example, achieving substantial robustness to corruptions over the sample mean. In light of this similarity of performance, the rest of our simulation examples utilize the much more computationally efficient unkernelized version of OWL. We also observed that the kernelized version of OWL was not particularly sensitive to the choice of bandwidth parameter (Figure S3). We further observe that on this example, the Pearson residuals method outperforms OWL, although the difference in performance is small relative to the gains of all methods over MLE.

Figure 3:

Simulation results for max-likelihood corruptions. (a) Mean parameter reconstruction for a multivariate normal model. (b) Test MSE for linear regression. (c) Test accuracy for logistic regression. (d) Mean parameter reconstruction for mixture models. In all figures, the dashed black line denotes median performance of MLE on full uncorrupted training set, and the shaded regions denote bootstrapped 95% confidence intervals over 50 random seeds.

Linear regression.

We considered a homoscedastic model with normally distributed errors. We considered two datasets. The first is a simulated dataset with 10-dimensional i.i.d. standard normal covariates. The ground-truth regression coefficients were drawn independently from $𝒩(0,4)$, the intercept was set to 0, and the residual standard deviation was $\sigma =1/4$. The training set consisted of 1,000 data points. For the test set, we drew 1,000 new data points and computed the MSE on the underlying mean response.

The second dataset was taken from a quantitative structure activity relationship (QSAR) dataset (Olier, 2020) compiled by Olier et al. (2018) from the ChEMBL database. It consists of 5012 chemical compounds whose activities were measured on the epidermal growth factor receptor protein erbB1. The activities were recorded as the negative log of the chemical concentration that inhibited 50% of the protein target, i.e. the pIC₅₀ value. Each compound had 1024 associated binary covariates, corresponding to the 1024-dimensional FCFP4 fingerprint representation of the molecule (Rogers and Hahn, 2010). We used PCA to reduce the dimension to 50. For every random seed, we computed a random 80/20 train/test split. The test MSE on this dataset is the standard MSE over the test responses. In both datasets, for each data point selected to be corrupted, we corrupted the responses by fitting a least squares solution and observing the residuals: if the residual is positive, we set the response to be 3v where $v$ is the largest absolute value observed value in the training set responses, otherwise setting it to $-3v$.

Figure 3b shows the results of the linear regression simulations for the max-likelihood corruptions. Across both datasets, we see that OWL is competitive with the best of the robust regression methods, whether that method is RANSAC or Huber regression.

Logistic regression.

For the logistic regression setting, we have parameters $b\in \mathbb{R},w\in {\mathbb{R}}^p$ and observations $\left(x_i,y_i\right)\in {\mathbb{R}}^{p+1}$ assumed to follow the distribution

$$y_i~\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left(\frac{1}{1+\exp \left(-⟨x_i,w⟩-b\right)}\right).$$

We considered three datasets. The first is a simulated dataset using the same parameters as the linear regression setting. The training labels are created according to the generative model. For test accuracy, we computed accuracy against the true sign-values, i.e. $I\left\{⟨w^{\star },x⟩\ge 0\right\}$.

The second dataset is taken from the MNIST handwritten digit classification dataset (LeCun et al., 1998). We considered the problem of classifying the digit ‘1’ v.s. the digit ‘8,’ resulting in a dataset with 14702 data points and 784 covariates, representing pixel intensities. The third dataset is a collection of 5172 documents from the Enron spam classification dataset, preprocessed to contain 5116 covariates, representing word counts (Metsis et al., 2006). For both the MNIST and the Enron spam datasets, we reduced the dimensionality to 10 via PCA and used a random 80/20 train/test split.

Figure 3c shows the results of the logistic regression simulations. Across all datasets, OWL again outperforms the other approaches in the presence of corruption.

Gaussian mixture models.

Recall the standard Gaussian mixture modeling setup: there are a collection of means ${\mu }_1,\dots ,{\mu }_K\in {\mathbb{R}}^p$, standard deviations ${\sigma }_1,\dots ,{\sigma }_K>0$, and mixing weights $\pi \in {\mathrm{\Delta }}^K$. Data points $x_i\in {\mathbb{R}}^p$ are drawn i.i.d. according to

$$X_i~\sum _{k=1}^K{\pi }_k𝒩\left({\mu }_k,{\sigma }_k^2{I}_p\right).$$

For our simulations, we generated a synthetic dataset of 1000 points in ${\mathbb{R}}^{10}$ by first drawing $K=3$ means ${\mu }_1^{\star },\dots ,{\mu }_K^{\star }$ whose coordinates are i.i.d. Gaussian with standard deviation 2. We set ${\sigma }_1={\sigma }_2={\sigma }_3=1/2$ and ${\pi }_1={\pi }_2={\pi }_3=1/3$. To corrupt a data point, we randomly selected half of its coordinates and set them randomly to either a large positive value or a large negative value (here, 5 and −5). As a metric, we measured the average mean squared Euclidean distance between the means of the fitted model and the ground truth model.

For all methods, we used random restarts, choosing the final model based on the method’s criterion: likelihood for MLE, the OKL estimate for OWL, and empirical Hellinger distance for Pearson residuals. We see that OWL remains robust against varying levels of corruptions, whereas both MLE and Pearson residuals perform significantly worse (left panel of Figure 3d).

Bernoulli product mixture models.

Consider the following model for $p$-dimensional binary data: there are a collection of probability vectors ${\lambda }_1,\dots ,{\lambda }_K\in [\mathrm{0,1}{]}^p$ and mixing weights $\pi \in {\mathrm{\Delta }}^K$. Each data point $x_i$ is drawn i.i.d. according to the process

$$z_i~\mathrm{C}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{g}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}(\pi )$$

$$x_{ij}~\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left({\lambda }_{z_{i}j}\right)\text{for}j=1,\dots ,p.$$

For our simulations, we generated a synthetic dataset of 1000 points in {0,1}¹⁰⁰ by first drawing $K=3$ means ${\lambda }_1^{\star },\dots ,{\lambda }_K^{\star }$ whose coordinates are i.i.d. from a $\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}(1/\mathrm{10,1}/10)$ distribution. The mixing weights were chosen to be uniform over the components. To corrupt a data point, we flipped each zero coordinate with probability 1/2. As a metric, we measured the average ${\ell }_1$-distance between the $\lambda$ parameters of the fitted model and the ground truth model.

The right panel of Figure 3d shows the results of the Bernoulli mixture model simulations. We see that OWL remains robust against varying levels of corruptions, whereas both MLE and Pearson residuals perform significantly worse.

5. Application to micro-credit study

In this section we apply OWL to data from a micro-credit study for which maximum likelihood estimation (MLE) is shown to be brittle. In Angelucci et al. (2015), the authors worked with one of the largest micro-lenders in Mexico to randomize their credit rollout across 238 geographical regions in the Sonora state. Within 18–34 months after this rollout, the authors surveyed $n=16,560$ households for various outcome measures.

Following Broderick et al. (2023), here we focus on the Average Intention to Treat effect (AIT) of the rollout on household profits. For $i\in \{1,\dots ,n\}$, let $Y_i$ denote the profit of the ith household during the last fortnight (in USD PPP units), and let $T_i\in \left\{\mathrm{0,1}\right\}$ be a binary variable that is one if the household $i$ falls in the geographical region where the rollout happened. The AIT on household profits is defined as the coefficient ${\beta }_1$ in the model:

$$Y_i={\beta }_0+{\beta }_1{T}_i+{\epsilon }_i,{\epsilon }_i\stackrel{\mathit{i.i.d}.}{~}N\left(0,{\sigma }^2\right),i\in \{1,\dots ,n\}.$$ (8)

In Supplement S13.1, we reproduce the brittleness in estimating ${\beta }_1$ using the MLE as demonstrated in Broderick et al. (2023) by removing an handful of observations.

Here we compare OWL to the above data deletion approach. We fit the model (8) to the full data set using 50 log₁₀ -spaced $ϵ$-values between 10⁻⁴ and 10⁻¹, and used the tuning procedure in Section 2.2.3 to select the value ${ϵ}_0=0.005$ where the minimum-OKL versus epsilon plot (Supplement S13, Figure S9) has its most prominent kink. We also calculate the MLE, which corresponds to OWL with $ϵ=0$. The AIT on household profit estimated by OWL as a function of $ϵ$ can be seen in the left panel of Figure 4. For values of $ϵ$ below ${ϵ}_0$, the AIT estimates change rapidly as $ϵ$ changes, while for values of $ϵ$ above ${ϵ}_0$, the AIT estimates are quite stable with changes in $ϵ$. This is due to OWL automatically down-weighting the outlying observations, as seen in the right panel of Figure 4. We quantify uncertainty in our estimates by using a variant of the Bootstrap (Supplement S13.2).

Figure 4:

Estimating the Average Intention to Treat (AIT) effect on household profits in the micro-credit study Angelucci et al. (2015) in the presence of outliers. Left: the AIT estimates using OWL for various values of $ϵ$ along with 90% bootstrap vertical confidence bands. The vertical line is drawn at the value ${ϵ}_0=0.005$ obtained by the tuning procedure in Section 2.2.3, and roughly coincides with the $ϵ$ beyond which the AIT estimates stabilize and the size of the confidence bands shrinks (see Supplement S13). Right: shows that the weights estimated by OWL procedure at $ϵ={ϵ}_0$ down-weight roughly 1% of the households that have outlying profit values (for visual clarity, we omit a down-weighted household with profit less that $-40K$ USD PPP); see also Figure S10 in Supplement S13.

In summary, the OWL procedure chose to down-weight roughly 1% of the households with extreme profit values and estimated an AIT of ${\beta }_1=0.6$ USD PPP per fortnight based on the selected value of ${ϵ}_0=0.005$. The value $ϵ={ϵ}_0$, tuned using the procedure in Section 2.2.3, roughly coincides with the point at which the AIT estimates become stable with respect to $ϵ$ and also with the point at which the 90% confidence bands for AIT become narrower — both suggesting that OWL with the choice $ϵ={ϵ}_0$ has identified and down-weighted outliers that may be causing brittleness in estimating AIT.

6. Discussion

In this paper, we introduced the optimistically weighted likelihood (OWL) methodology, motivated by brittleness issues arising from misspecification in statistical methodology based on standard likelihoods. On the theoretical side, we established the consistency and robustness of our approach and showed its asymptotic connection to the coarsened inference methodology. We also proposed a feasible alternating optimization scheme to implement the methodology and demonstrated its empirical utility on both simulated and real data.

The OWL methodology opens up several interesting future directions. One practical open problem is how to scale to larger datasets. As a weighted likelihood method, OWL requires solving for a weight vector whose dimension is the size of the dataset. While we can solve the resulting convex optimization problem for thousands of data points, the procedure becomes significantly more complicated when the size of the dataset exceeds computer memory. How do we maintain a feasible solution, i.e. one that lies in the intersection of the simplex and some probability ball, when the entire vector cannot fit in memory?

Another practical question is how to apply the OWL approach in more complex models; for example, involving dependent data. This may be relatively straightforward for models in which the likelihood can still be written in product form due to conditional independence given random effects. This would open up its application to nested, longitudinal, spatial and temporal data, as random effects models are routinely used in such settings.

Finally, it will be fruitful to implement the OWL method for a choice of $\mathit{d}$ other than the TV distance. Indeed, as reflected in Assumption 2.1 and our robustness results, the choice of $\mathit{d}$ controls the nature of robustness offered by OWL. An important choice for $\mathit{d}$ may be the Wasserstein distance because it metrizes the weak convergence topology on bounded spaces (Huber and Ronchetti, 2009, Section 2.4) and, as argued in Chapter 1 of Huber and Ronchetti (2009), can capture robustness to both small changes in all observations (e.g. due to rounding or grouping) and large changes in a few observations (e.g. due to contamination or outliers). Indeed, while several of our theoretical results already hold for the Wasserstein metric, appropriate modifications to the OKL estimator and the OWL algorithm will be needed to make this approach feasible.

Supplementary Material

The accompanying supplementary materials contain a real data application to cluster single-cell RNAseq data, along with additional details (figures, lemmas, definitions, etc. with names starting with the letter ‘S’) referenced in the article.