Adversarial Monte Carlo Meta-Learning of Optimal Prediction Procedures

Abstract

We frame the meta-learning of prediction procedures as a search for an optimal strategy in a two-player game. In this game, Nature selects a prior over distributions that generate labeled data consisting of features and an associated outcome, and the Predictor observes data sampled from a distribution drawn from this prior. The Predictor’s objective is to learn a function that maps from a new feature to an estimate of the associated outcome. We establish that, under reasonable conditions, the Predictor has an optimal strategy that is equivariant to shifts and rescalings of the outcome and is invariant to permutations of the observations and to shifts, rescalings, and permutations of the features. We introduce a neural network architecture that satisfies these properties. The proposed strategy performs favorably compared to standard practice in both parametric and nonparametric experiments.

1. Introduction

1.1. Problem Formulation

Consider a dataset consisting of $n\ge 2$ observations $\left(X_1,Y_1\right),\dots ,\left(X_n,Y_n\right)$ drawn independently from a distribution $P$ belonging to some known model $𝒫$, where each $X_i$ is a continuously distributed feature with support contained in $𝒳≔{\mathbb{R}}^p$ and each $Y_i$ is an outcome with support contained in $𝒴≔\mathbb{R}$. This dataset can be written as $\mathit{D}≔(\mathit{X},\mathit{Y})$, where $\mathit{X}$ is the $n\times p$ matrix for which row $i$ contains $X_i$ and $\mathit{Y}$ is the $n$-dimensional vector for which entry $i$ contains $Y_i$. The support of $\mathit{D}$ is contained in $𝒟≔{𝒳}^n\times {𝒴}^n$. The objective is to develop an estimator of the regression function ${\mu }_P$ that maps from $x_0$ to ${\mathrm{E}}_P\left[Y\mid X=x_0\right]$. An estimator $T$ belongs to the collection $𝒯$ of operators that take as input a dataset $\mathit{d}≔(\mathit{x},\mathit{y})$ and output a prediction function $T\left(\mathit{d}\right):𝒳\to \mathbb{R}$, where here and throughout we use $\mathit{d}=(\mathit{x},\mathit{y})$ to denote a possible realization of the random variable $\mathit{D}=(\mathit{X},\mathit{Y})$. Examples of estimators include the generalized linear models (Nelder and Wedderburn, 1972), random forests (Breiman, 2001), and gradient boosting machines (Friedman, 2001). We will also refer to estimators as prediction procedures. We focus on the case that the performance of an estimator is quantified via the standardized mean-squared error (MSE), namely

$$R(T,P)≔{\mathrm{E}}_P\left[\int \frac{{\left[T\left(\mathit{D}\right)\left(x_0\right)-{\mu }_P\left(x_0\right)\right]}^2}{{\sigma }_P^2}d{P}_X\left(x_0\right)\right],$$ (1)

where the expectation above is over the draw of $\mathit{D}$ under sampling from $P,P_X$ denotes the marginal distribution of $X$ implied by $P$, and ${\sigma }_P^2$ denotes the variance of the error ${ϵ}_P≔Y-{\mu }_P\left(X\right)$ when $(X,Y)~P$. Note that ${ϵ}_P$ may be heteroscedastic. Throughout we assume that, for all $P\in 𝒫,{\mathrm{E}}_P\left[Y^2\right]<\mathrm{\infty }$ and ${ϵ}_P$ is a continuous random variable. Note that the continuity of ${ϵ}_P$ implies that $Y$ is continuous and that ${\sigma }_P^2>0$.

In practice, the distribution $P$ is not known, and therefore the risk $R(T,P)$ of a given estimator $T$ is also not known. We now describe three existing criteria for judging the performance of $T$ that do not rely on knowledge of $P$. The first criterion is the maximal risk ${\mathrm{s}\mathrm{u}\mathrm{p}}_{P\in 𝒫} R(T,P)$. If $T$ minimizes the maximal risk over $𝒯$, then $T$ is referred to as a minimax estimator (Wald, 1945). Minimax estimators optimize for the worst-case scenario wherein the distribution $P$ is chosen adversarially in such a way that the selected estimator performs as poorly as possible. The second criterion is Bayesian in nature, namely the average of the risk $R(T,P)$ over draws of $P$ from a given prior $\Pi$ on $𝒫$. Specifically, this Bayes risk is defined as $r(T,\Pi )≔{\mathrm{E}}_{\Pi }\left[R\right(T,P\left)\right]$ (Robert, 2007). A $\Pi$-Bayes estimator optimally incorporates the prior beliefs encoded in $\Pi$ with respect to the Bayes risk $r(\cdot ,\Pi )$ — more concretely, an estimator $T$ is referred to as a $\Pi$-Bayes estimator if it minimizes the Bayes risk over $𝒯$. Though the optimality property of Bayes estimators is useful in settings where $\Pi$ only encodes substantive prior knowledge, its utility is less clear otherwise. Indeed, as the function $r(\cdot ,\Pi )$ generally depends on the choice of $\Pi$, it is possible that a $\Pi$-Bayes estimator $T$ is meaningfully suboptimal with respect to some other prior ${\Pi }^{\prime }$, that is, that $r(T,\Pi )\gg {\mathrm{i}\mathrm{n}\mathrm{f}}_{T^{\prime }} r\left(T^{\prime },{\Pi }^{\prime }\right)$. This phenomenon can be especially common when the sample size is small or the model is nonparametric. In fact, in the latter case, Bayes estimators against particular priors $\Pi$ can easily be inconsistent even though consistent frequentist estimators are available (Ghosal and Van der Vaart, 2017) — for such priors, Bayes estimators perform poorly even when the sample size is large. Therefore, in settings where there is no substantive reason to favor a particular choice of $\Pi$, it is sensible to seek another approach for judging the performance of $T$. A natural criterion is the worst-case Bayes risk of $T$ over some user-specified collection $\Gamma$ of priors, namely ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r(T,\Pi )$. This criterion is referred to as the $\Gamma$-maximal Bayes risk of $T$. The collection $\Gamma$ may be restricted to contain all priors that are compatible with available prior information, such as knowledge about the smoothness of a regression function, while being left large enough to acknowledge that prior knowledge may be too vague to encode within a single prior distribution (see Section 3.6 of Robert, 2007, for more possible forms of vague prior information). If $T$ is a minimizer of the $\Gamma$-maximal Bayes risk, then $T$ is referred to as a $\Gamma$-minimax estimator (Berger, 1985). Such estimators can be viewed as the optimal strategy in a sequential two-player game between a Predictor and Nature, where the Predictor selects an estimator and Nature then selects a prior in $\Gamma$ at which the Predictor’s chosen estimator performs as poorly as possible in terms of Bayes risk. Notably, in settings where $\Gamma$ contains all distributions with support in $𝒫$, the $\Gamma$-maximal Bayes risk is equivalent to the maximal risk. Consequently, in this special case, an estimator is $\Gamma$-minimax if and only if it is minimax. In settings where $\Gamma =\left\{\Pi \right\}$, an estimator is $\Gamma$-minimax if and only if it is $\Pi$-Bayes. Therefore, by allowing for a choice of $\Gamma$ as large as the unrestricted set of all possible distributions or as small as a singleton set, $\Gamma$-minimaxity provides a means of interpolating between the minimax and Bayesian criteria.

Though $\Gamma$-minimax estimators represent an appealing compromise between the Bayesian and minimax paradigms, they have seen limited use in practice because they are rarely available in closed form. In this work, we aim to overcome this challenge in the context of prediction by providing an iterative strategy for learning $\Gamma$-minimax prediction procedures. Due to the potentially high computational cost of this iterative scheme, a key focus of our work involves identifying conditions under which we can identify a small subclass of $𝒯$ that still contains a T-minimax estimator. This then makes it possible to optimize over this subclass, which we show in our experiments can dramatically improve the performance of our iterative scheme given a fixed computational budget.

Hereafter we refer to $\Gamma$-minimax estimators as ‘optimal’, where it is to be understood that this notion of optimality relies on the choice of $\Gamma$.

1.2. Overview of Our Strategy and Our Contributions

Our strategy builds on two key results, each of which will be established later in this work. First, under conditions on $𝒯$ and $\Gamma$, there exists a $\Gamma$-minimax estimator in the subclass ${𝒯}_e\subset 𝒯$ of estimators that are equivariant to shifts and rescalings of the outcome and are invariant to permutations of the observations and to shifts, rescalings, and permutations of the features. Second, under further conditions, there is an equilibrium point $\left(T^{\star },{\Pi }^{\star }\right)\in {𝒯}_e\times \Gamma$ such that

$$\underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r\left(T^{\star },\Pi \right)=r\left(T^{\star },{\Pi }^{\star }\right)=\underset{T\in {𝒯}_e}{\mathrm{i}\mathrm{n}\mathrm{f}} r\left(T,{\Pi }^{\star }\right).$$ (2)

Upper bounding the right-hand side by ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } {\mathrm{i}\mathrm{n}\mathrm{f}}_{T\in {𝒯}_e} r(T,\Pi )$ and applying the max-min in-equality shows that $T^{\star }$ is $\Gamma$-minimax. To find an equilibrium numerically, we propose to use adversarial Monte Carlo meta-learning (AMC) (Luedtke et al., 2020) to iteratively update an estimator in ${𝒯}_e$ and a prior in $\Gamma$. AMC is a form of stochastic gradient descent ascent (e.g., Lin et al., 2019) that can be used to learn optimal statistical procedures in general decision problems.

We make the following contributions:

• In Section 2, we characterize several equivariance properties of optimal estimators for a wide range of $(𝒯,\Gamma )$.

• In Section 3, we present a general framework for adversarially learning optimal prediction procedures.

• In Section 4, we present a novel neural network architecture for parameterizing estimators that satisfy the equivariance properties established in Section 2.

• In Section 5, we apply our algorithm in two settings and learn estimators that outperform standard approaches in numerical experiments. In Section 6, we also evaluate the performance of these learned estimators in data experiments.

All proofs for the results in the above sections can be found in Section 7. Section 8 describes possible extensions and provides concluding remarks.

To maximize the accessibility of our main theoretical results, we do not use group theoretic notation when presenting them in Sections 2 through 4. However, when proving these results, we will heavily rely on tools from group theory; consequently, we adopt this notation in Section 7.

1.3. Related Works

The approach proposed in this work is a form of meta-learning (Schmidhuber, 1987; Thrun and Pratt, 1998; Vilalta and Drissi, 2002), where here each task is a regression problem. Most existing works in this area pursue a task-distribution strategy to meta-learning (Hospedales et al., 2020), where the objective is to minimize the average loss (risk) across draws of tasks from some specified distribution. As we will now show, the objective function employed in such strategies in fact corresponds to a Bayes risk. In regression problems, each task is a tuple containing a dataset $\mathit{d}$ and a task-dependent loss $ℒ:𝒟\times 𝒯\to \mathbb{R}$. For a given prior $\Pi$, a draw from the task distribution can be obtained by first sampling $P~\Pi$, next sampling a dataset $\mathit{D}$ of independent observations from $P$, drawing an evaluation point $X_0~P_X$, and finally defining the loss by $ℒ(\mathit{d},T)={\left[T\left(\mathit{d}\right)\left(X_0\right)-{\mu }_P\left(X_0\right)\right]}^2/{\sigma }_P^2$ or some related loss, such as a squared error loss that does not standardize by ${\sigma }_P^2$. The objective function is then equal to $T\mapsto E\left[ℒ\right(\mathit{D},T\left)\right]$, where the expectation is over the draw of $(\mathit{D},ℒ)$ from the task distribution. This objective function is exactly equal to the Bayes risk function $T\mapsto r(T,\Pi )$. Hence, existing meta-learning approaches for regression problems whose objective functions take this form can be viewed as optimizing a Bayes risk.

We now review existing meta-learning strategies, starting with those that parameterize $𝒯$ as a neural network class. Hochreiter et al. (2001) advocated parameterizing $𝒯$ as a collection of long short-term (LSTM) networks (Hochreiter and Schmidhuber, 1997). More recent works have advocated using memory-augmented neural networks (Santoro et al., 2016) or conditional neural processes (CNPs) (Garnelo et al., 2018) rather than LSTMs in meta-learning tasks. There have also been other works on the meta-learning of supervised learning procedures that are parameterized as neural networks (Bosc, 2016; Vinyals et al., 2016; Ravi and Larochelle, 2017). Compared to these works, we adversarially learn a prior $\Pi$ from a collection $\Gamma$ of priors, and we also formally characterize equivariance properties that will be satisfied by any optimal prediction procedure in a wide variety of problems. This characterization leads us to develop a neural network architecture designed for the prediction settings that we consider.

Model-agnostic meta-learning (MAML) is another popular meta-learning approach (Finn et al., 2017). In our setting, MAML aims to initialize the weights of a regression function estimate (parameterized as a neural network, for example) in such a way that, on any new task, only a limited number of gradient updates are needed. More recent approaches leverage the fact that, in certain settings, the initial estimate can instead be updated using a convex optimization algorithm (Bertinetto et al., 2018; Lee et al., 2019). To run any of these approaches, a prespecified prior over tasks is required. In our setting, these tasks take the form of data-generating distributions $P$. In contrast, our approach adversarially selects a prior from $\Gamma$.

Two recent works (Yin et al., 2018; Goldblum et al., 2019) developed meta-learning procedures that are trained under a different adversarial regime than that studied in the current work, namely under adversarial manipulation of one or both of the dataset $\mathit{d}$ and evaluation point $x_0$ (Dalvi et al., 2004). This adversarial framework appears to be most useful when there truly is a malicious agent that aims to contaminate the data, which is not the case that we consider. In contrast, in our setting, the adversarial nature of our framework allows us to ensure that our procedure will perform well regardless of the true value of $P$, while also taking into account prior knowledge that we may have.

Our approach is also related to existing works in the statistics and econometrics literatures on the numerical learning of minimax and $\Gamma$-minimax statistical decision rules. In finite-dimensional models, early works showed that it is possible to numerically learn minimax rules (Nelson, 1966; Kempthorne, 1987) and, in settings where $\Gamma$ consists of all priors that satisfy a finite number of generalized moment conditions, $\Gamma$-minimax rules (Noubiap and Seidel, 2001). Other works have studied the $\Gamma$-minimax case where $\Gamma$ consists of priors that only place mass on a pre-specified finite set of distributions in $𝒫$, both for general decision problems (Chamberlain, 2000) and for constructing confidence intervals (Schafer and Stark, 2009). Defining $\Gamma$ in this fashion modifies the statistical model $𝒫$ to only consist of finitely many distributions, which can be restrictive. A recent work introduced a new approach, termed AMC, for learning minimax procedures for general models $𝒫$ (Luedtke et al., 2020). In contrast to earlier works, AMC does not require the explicit computation of a Bayes estimator under any given prior, thereby improving the feasibility of this approach in moderate-to-high dimensional models. In their experiments, Luedtke et al. (2020) used neural network classes to define the sets of allowable statistical procedures. Unlike the current work, none of the aforementioned studies identified or leveraged the equivariance properties that characterize optimal procedures. As we will see in our experiments, leveraging these properties can dramatically improve performance.

1.4. Notation

We now introduce the notation and conventions that we use. For a function $f:𝒫\to 𝒫$, we let $\Pi \circ f^{-1}$ denote the pushforward measure that is defined as the distribution of $f\left(P\right)$ when $P~\Pi$. For any dataset $\mathit{d}=(\mathit{x},\mathit{y})$ and mapping $f$ with domain $𝒟$, we let $f(\mathit{x},\mathit{y})≔f\left(\mathit{d}\right)$. We take all vectors to be column vectors when they are involved in matrix operations. We write $\odot$ to mean the entrywise product and $a^{\odot 2}$ to mean $a\odot a$. For an $m_1\times m_2$ matrix $a$, we let $a_{i\mathrm{*}}$ denote the $i^{\text{th}}$ row, $a_{\mathrm{*}j}$ denote the $j^{\text{th}}$ column, $\bar{a}≔\frac{1}{m_1}{\sum }_{i=1}^{m_1} a_{i\mathrm{*}}$, and $s(a{)}^2≔\frac{1}{m_1}{\sum }_{i=1}^{m_1} {\left(a_{i\mathrm{*}}-\bar{a}\right)}^{\odot 2}$. When we standardize a vector $a$ as $[a-\bar{a}]/s\left(a\right)$, we always use the convention that 0/0 = 0. We write $[a\mid b]$ to denote the column concatenation of two matrices. For an $m_1\times m_2\times m_3$ array $a$, we let $a_{i\mathrm{*}\mathrm{*}}$ denote the $m_2\times m_3$ matrix with entry $(j,k)$ equal to $a_{ijk},a_{i\mathrm{*}k}$ denote the $m_2$-dimensional vector with entry $j$ equal to $a_{ijk}$, etc. For $a\in \mathbb{R}$ and $b\in {\mathbb{R}}^k$, we write $a+b$ to mean $a{1}_k+b$.

2. Characterization of Optimal Procedures

2.1. Optimality of Equivariant Estimators

We start by presenting conditions that we impose on the collection of priors $\Gamma$. Let $𝒜$ denote the collection of all $n\times n$ permutation matrices, and let $ℬ$ denote the collection of all $p\times p$ permutation matrices. We suppose that $\Gamma$ is preserved under the following transformations:

• P1.Permutations of features: $\Pi \in \Gamma$ and $B\in ℬ$ implies that $\Pi \circ f_1^{-1}\in \Gamma$, where $f_1\left(P\right)$ is the distribution of $(BX,Y)$ when $(X,Y)~P$.
• P2.Shifts and rescalings of features: $\Pi \in \Gamma ,a\in {\mathbb{R}}^p$, and $b\in {\left({\mathbb{R}}^{+}\right)}^p$ implies that $\Pi \circ f_2^{-1}\in \Gamma$, where $f_2\left(P\right)$ is the distribution of $(a+b\odot X,Y)$ when $(X,Y)~P$.
• P3.Shift and rescaling of outcome: $\Pi \in \Gamma$ and $\tilde{a}\in \mathbb{R}$ and $\tilde{b}>0$ implies that $\Pi \circ f_3^{-1}\in \Gamma$, where $f_3\left(P\right)$ is the distribution of $(X,\tilde{a}+\tilde{b}Y)$ when $(X,Y)~P$.

The above conditions implicitly encode that $f_1\left(P\right),f_2\left(P\right)$, and $f_3\left(P\right)$ all belong to $𝒫$ whenever $P\in 𝒫$. Section 7.1 provides an alternative characterization of P1, P2, and P3 in terms of the preservation of $\Gamma$ under a certain group action.

Condition P1 ensures that permuting the features during preprocessing will not impact the collection of priors considered. This condition is reasonable in settings where there is only a limited prior understanding of each individual feature under consideration or, if such information is available, there is little anticipated benefit from including it in the analysis. Most commonly supervised machine learning algorithms similarly do not incorporate specific prior information about individual features, and are instead designed to work across a variety of settings — this is the case, for example, for commonly used implementations of random forests, extreme gradient boosting, and penalized linear models (Pedregosa et al., 2011; Chen and Guestrin, 2016). It is worth noting, however, that P1 still allows information on the features to be incorporated should it be available — for example, prior beliefs on the multivariate feature distribution, such as the number of modes that it has, or the regression function, such as its level of sparsity, can be imposed in the collection $\Gamma$ of prior distributions. Conditions P2 and P3 are imposed to ensure that the $\Gamma$-maximal risk criterion captures the possibility that the data may be preprocessed via affine transformations, such as prestandardization or a change of the unit of measure (Fahrenheit to Celsius, say), before being supplied to the prediction algorithm. By having $\Gamma$ be large enough to ensure that P2 and P3 are satisfied, the $\Gamma$-minimax risk reflects performance in an adversarial setting wherein affine transformations are applied to the features and outcome in such a way as to make the (Bayes) risk as large as possible for a given prediction algorithm. Because it minimizes this adversarial criterion, a $\Gamma$-minimax estimator should be robust to such adversarial transformations, thereby ensuring satisfactory performance regardless of the chosen unit of measure or prestandardization scheme.

We also assume that the signal-to-noise ratio (SNR) is finite — this condition is important in light of the fact that the MSE risk that we consider standardizes by ${\sigma }_P^2$.

• P4.Finite SNR : ${\mathrm{s}\mathrm{u}\mathrm{p}}_{P\in 𝒫} {\mathrm{v}\mathrm{a}\mathrm{r}}_P\left({\mu }_P\left(X\right)\right)/{\sigma }_P^2<\mathrm{\infty }$.

We now present conditions that we impose on the class of estimators $𝒯$. In what follows we let ${𝒟}_0≔\left\{\left(\mathit{d},x_0\right)\in 𝒟\times 𝒳:s\left(\mathit{y}\right)\ne 0,s\left(\mathit{x}\right)\ne 0_p\right\}$. For $\left(\mathit{d},x_0\right)\in {𝒟}_0$, we let

$$z\left(\mathit{d},x_0\right)≔\left(\frac{\mathit{x}-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)},\frac{\mathit{y}-\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)},\frac{x_0-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)},\frac{\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)},\frac{\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)},\mathrm{l}\mathrm{o}\mathrm{g}s\left(\mathit{x}\right),\mathrm{l}\mathrm{o}\mathrm{g}s\left(\mathit{y}\right)\right),$$

where $\mathrm{l}\mathrm{o}\mathrm{g}s\left(\mathit{x}\right)$ is the vector where log is applied entrywise and where we abuse notation and let $\frac{\mathit{x}-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)}$ represent the $n\times p$ matrix for which row $i$ is equal to $\left[x_i-\stackrel{-}{\mathit{x}}\right]/s\left(\mathit{x}\right)$, and similarly for $\stackrel{-}{\mathit{x}}/s\left(\mathit{x}\right)$. We let $Ƶ≔\left\{z\left(\mathit{d},x_0\right):\left(\mathit{d},x_0\right)\in {𝒟}_0\right\}$. When it will not cause confusion, we will write $\mathit{z}≔z\left(\mathit{d},x_0\right)$. Fix $T\in 𝒯$. Let $S_T:𝒵\to \mathbb{R}$ denote the unique function that satisfies

$$T\left(\mathit{d}\right)\left(x_0\right)=\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)S_T\left(\mathit{z}\right)\text{for}\text{all}\left(\mathit{d},x_0\right)\in {𝒟}_0.$$ (3)

The uniqueness arises because $s\left(\mathit{y}\right)\ne 0$ on ${𝒟}_0$. Because we have assumed that $X$ and $Y$ are continuous random variables under sampling from any $P\in 𝒫$, it follows that, for all $P\in 𝒫$, the class $𝒮≔\left\{S_T:T\in 𝒯\right\}$ uniquely characterizes the functions in $𝒯$ up to their behavior on subsets of $𝒟\times 𝒳$ of $P$-probability zero. In what follows, we will impose smoothness constraints on $𝒮$, which in turn imposes constraints on $𝒯$. The first three conditions suffice to show that $𝒮$ is compact in the space $C(Ƶ,\mathbb{R})$ of continuous $Ƶ\to \mathbb{R}$ functions equipped with the compact-open topology.

• T1.$𝒮$ is pointwise bounded: For all $\mathit{z}\in \mathrm{Ƶ},{\mathrm{s}\mathrm{u}\mathrm{p}}_{S\in 𝒮} \left|S\right(\mathit{z}\left)\right|<\mathrm{\infty }$.
• T2.$𝒮$ is locally Hölder: For all compact sets $𝒦\subset Ƶ$, there exists an $\alpha \in \left(\mathrm{0,1}\right)$ such that

$$\underset{S\in 𝒮,\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒦}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}_2^{\alpha }}<\mathrm{\infty },$$

where $\parallel \cdot {\parallel }_2$ denotes the Euclidean norm. We take the supremum to be zero if $𝒦$ is a singleton or is empty.

• T3.$𝒮$ is sequentially closed in the topology of compact convergence: If ${\left\{S_j\right\}}_{j=1}^{\mathrm{\infty }}$ is a sequence in $𝒮$ and $S_j\to S$ compactly in the sense that, for all compact $𝒦\subset Ƶ,{\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{z}\in 𝒦} \left|S_j\left(\mathit{z}\right)-S\left(\mathit{z}\right)\right|\to 0$, then $S\in 𝒮$.

The following conditions ensure that $𝒮$ is invariant to certain preprocessings of the data, in the sense that, for any function $S\in 𝒮$, the function that first preprocesses the data in an appropriate fashion and then applies $S$ to this data is itself in $𝒮$. When formulating these conditions, we write $z\left(\mathit{d},x_0\right)$ to mean an element of $Ƶ$. Because $z$ is a bijection between ${𝒟}_0$ and $Ƶ$, it is possible to recover $\left(\mathit{d},x_0\right)$ from $z\left(\mathit{d},x_0\right)$. Below we use this fact to abuse notation and define functions with domain $Ƶ$ like $z\left(\mathit{d},x_0\right)\mapsto g\left(\mathit{d},x_0\right)$ for functions $g$ with domain ${𝒟}_0$, without explicitly introducing notation for the inverse of $z$.

• T4.Permutations: For all $S\in 𝒮,A\in 𝒜$, and $B\in ℬ,z\left(\mathit{d},x_0\right)\mapsto S\left(z\left((A\mathit{x}B,A\mathit{y}),B^{\mathrm{\top }}{x}_0\right)\right)$ is in $𝒮$.
• T5.Shifts and rescalings: For all $S\in 𝒮,a\in {\mathbb{R}}^p,b\in {\left({\mathbb{R}}^{+}\right)}^p,\tilde{a}\in \mathbb{R}$, and $\tilde{b}>0$, the function $z\left(\mathit{d},x_0\right)\mapsto S\left(z\left(\left({\mathit{x}}^{a,b},\tilde{a}+\tilde{b}\mathit{y}\right),a+b\odot x_0\right)\right)$ is in $𝒮$, where ${\mathit{x}}^{a,b}$ is the $n\times p$ matrix with row $i$ equal to $a+b\odot {\mathit{x}}_{i\mathrm{*}}$.

In Appendix B, we provide two examples of classes $𝒮$ that satisfy Conditions T1-T5. One of these classes is finite-dimensional and the other is infinite-dimensional. The infinite-dimensional class takes a particularly simple form. In particular, for some $c,\alpha >0$ and $F:Ƶ\to {\mathbb{R}}^{+}$some function that is invariant to permutations, shifts, and rescalings, we consider the class $𝒮$ to be the collection of all the collection of all $S:Ƶ\to \mathbb{R}$ such that $\left|S\right(\mathit{z}\left)\right|\le F\left(\mathit{z}\right)$ and $\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|\le c{∥\mathit{z}-{\mathit{z}}^{\prime }∥}_2^{\alpha }$ for all $\mathit{z},{\mathit{z}}^{\prime }\in Ƶ$.

Let ${𝒯}_e\subseteq 𝒯$ denote the class of estimators that are equivariant to shifts and rescalings of the outcome and are invariant to permutations of the observations and to shifts, rescalings, and permutations of the features. Specifically, ${𝒯}_e$ consists of functions in $𝒯$ satisfying the following properties for all pairs $\left(\mathit{d},x_0\right)$ of datasets and features in ${𝒟}_0$, permutation matrices $A\in 𝒜$ and $B\in ℬ$, shifts $a\in {\mathbb{R}}^p$ and $\tilde{a}\in \mathbb{R}$, and rescalings $b\in {\left({\mathbb{R}}^{+}\right)}^p$ and $\tilde{b}>0$ :

$$T\left(A\mathit{x}B,A\mathit{y}\right)\left(B^{\top }{x}_0\right)=T\left(\mathit{d}\right)\left(x_0\right),$$ (4)

$$T\left({\mathit{x}}^{a,b},\tilde{a}+\tilde{b}\mathit{y}\right)\left(a+b\odot x_0\right)=\tilde{a}+\tilde{b}T\left(\mathit{d}\right)\left(x_0\right),$$ (5)

The following result shows that the $\Gamma$-maximal risk is the same over $𝒯$ and ${𝒯}_e\subseteq 𝒯$.

Theorem 1. Under P1-P4 and T1-T5,

$$\underset{T\in 𝒯}{\mathrm{i}\mathrm{n}\mathrm{f}} \underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r(T,\Pi )=\underset{T\in {𝒯}_e}{\mathrm{i}\mathrm{n}\mathrm{f}} \underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r(T,\Pi ).$$

The above does not rule out the possibility that there exists a non-equivariant $\Gamma$-minimax estimator, that is, a $\Gamma$-minimax estimator that belongs to $𝒯\setminus {𝒯}_e$. Rather, when paired with additional conditions that ensure that the infimum over ${𝒯}_e$ above is achieved (see Theorem 3), the above implies that ${𝒯}_e$ contains at least one $\Gamma$-minimax estimator.

Theorem 1 is a variant of the Hunt-Stein theorem (Hunt and Stein, 1946). Our proof, which draws inspiration from Le Cam (2012), consists in showing that our prediction problem is invariant to the action of an amenable group and subsequently applying Day’s fixed-point theorem (Day, 1961) to show that, for all $T\in 𝒯$, the collection of $T^{\prime }$ for which ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T^{\prime },\Pi \right)\le {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r(T,\Pi )$ has nonempty intersection with ${𝒯}_e$.

This theorem has a natural analogy to the translation equivariance that is enjoyed by convolutional neural networks in object detection problems, where the goal is to classify and draw a bounding box around objects in an image (Russakovsky et al., 2015). To simplify the discussion, here we focus on the special case where there is only one object class of interest (e.g., humans), so that the goal is simply to draw a bounding box around each object that is contained in the image. In object detection settings, a key insight is that an objecťs class does not change even if its position is shifted. Given this insight, it seems reasonable to expect that any sufficiently rich collection of candidate detectors will be such that, given any object detector $V$, the collection will contain a translation equivariant detector with equal or superior performance to that of $V$. For this to be true, certain requirements are also generally needed of the loss function used to measure performance. In particular, the error accrued by incorrectly bounding or failing to bound an object should not depend on the position of that object in the image — this condition is satisfied by many loss functions that are commonly used in this setting. In our setting, conditions P1-P3, which say that a prior still belongs to $\Gamma$ even after certain transformations are applied to the distributions drawn from that prior, are the analogues of the translation invariance property of an objecťs class (“a human remains a human if they are shifted to the left, and the pushforward of a prior in $\Gamma$ remains in $\Gamma$ even if features and outcomes are permuted, shifted, or rescaled”); conditions T4 and T5 are the analogues of the requirement that the collection of detectors be sufficiently rich; and the fact that the standardized squared error ${\left[T\left(\mathit{d}\right)\left(x_0\right)-{\mu }_P\left(x_0\right)\right]}^2/{\sigma }_P^2$ does not depend on the particular ordering of the features or the centering or scaling of the features or outcomes is analogous to the translation invariance of the loss functions used in object detection.

2.2. Focusing Only on Distributions with Standardized Predictors and Outcome

Theorem 1 suggests restricting attention to estimators in ${𝒯}_e$ when trying to learn a $\Gamma$-minimax estimator. We now show that, once this restriction has been made, it also suffices to restrict attention to a smaller collection of priors ${\Gamma }_1$ when identifying a least favorable prior. In fact, we show something slightly stronger, namely that the restriction to ${\Gamma }_1$ can be made even if optimal estimators are sought over the richer class ${\widetilde{𝒯}}_e\supseteq {𝒯}_e$ of estimators that satisfy the equivariance property (5) but do not necessarily satisfy (4).

We now define ${\Gamma }_1$. Let $h\left(P\right)$ denote the distribution of

$$\left({\left(\frac{X_j-{\mathrm{E}}_P\left[X_j\right]}{{\mathrm{v}\mathrm{a}\mathrm{r}}_P{\left(X_j\right)}^{1/2}}\right)}_{j=1}^p,\frac{Y-{\mathrm{E}}_P\left[Y\right]}{{\sigma }_P}\right)$$

when $(X,Y)~P$. Note that here, and here only, we have written $X_j$ to denote the $j^{\text{th}}$ feature rather than the $j^{\text{th}}$ observation. Also let ${\Gamma }_1≔\left\{\Pi \circ h^{-1}:\Pi \in \Gamma \right\}$, which is a collection of priors on ${𝒫}_1≔\left\{h\right(P):P\in 𝒫\}$.

Theorem 2. If $P_2$ and $P3$ hold and all $T\in 𝒯$ satisfy (5), then $T^{\star }$ is $\Gamma$-minimax if and only if it is ${\Gamma }_1$-minimax.

We conclude by noting that, under P2 and P3,${𝒫}_1$ consists precisely of those $P\in 𝒫$ that satisfy:

$${\mathrm{E}}_P\left[X\right]=0_p,{\mathrm{E}}_P\left[X^{\odot 2}\right]=1_p,{\mathrm{E}}_P\left[Y\right]=0,{\sigma }_P^2=1.$$ (6)

2.3. Existence of an Equilibrium Point

We also make the following additional assumption on $𝒮$.

• T6.$𝒮$ is convex: $S_1,S_2\in 𝒮$ and $\delta \in \left(\mathrm{0,1}\right)$ implies that $\mathit{z}\mapsto \delta S_1\left(\mathit{z}\right)+(1-\delta )S_2\left(\mathit{z}\right)$ is in $𝒮$.

The two examples in Appendix B also satisfy T6.

We also impose the following condition on the size of the collection of distributions ${𝒫}_1$ and the collection of priors ${\Gamma }_1$, which in turn imposes restrictions on $𝒫$ and $\Gamma$.

• P5.There exists a metric $\rho$ on ${𝒫}_1$ such that (i) $\left({𝒫}_1,\rho \right)$ is a complete separable metric space, (ii) ${\Gamma }_1$ is tight in the sense that, for all $\epsilon >0$, there exists a compact set $𝒦$ in $\left({𝒫}_1,\rho \right)$ such that $\Pi \left(𝒦\right)\ge 1-\epsilon$ for all $\Pi \in {\Gamma }_1$, and (iii) for all $T\in {𝒯}_e,P\mapsto R(T,P)$ is upper semi-continuous and bounded from above on $\left({𝒫}_1,\rho \right)$.

In Appendix C, we give examples of parametric and nonparametric settings where P5 is applicable.

So far, the only conditions that we have required on the $\sigma$-algebra $𝒜$ of $𝒫$ are that $h$ and $R(T,\cdot ),T\in 𝒯$, are measurable. In this subsection, and in this subsection only, we add the assumptions that P5 holds and that $𝒜$ is such that $\left\{A\cap {𝒫}_1:A\in 𝒜\right\}$ equals ${ℬ}_1$, where ${ℬ}_1$ is the collection of Borel sets on $\left({𝒫}_1,\rho \right)$.

We will also assume the following two conditions on ${\Gamma }_1$.

• P6.${\Gamma }_1$ is closed in the topology of weak convergence: if ${\left\{{\Pi }_j\right\}}_{j=1}^{\mathrm{\infty }}$ is a sequence in ${\Gamma }_1$ that converges weakly to $\Pi$, then $\Pi \in {\Gamma }_1$.
• P7.${\Gamma }_1$ is convex: for all ${\Pi }_1,{\Pi }_2\in \Gamma$ and $\alpha \in \left(\mathrm{0,1}\right)$, the mixture distribution $\alpha {\Pi }_1+(1-\alpha ){\Pi }_2$ is in $\Gamma$.

Under Conditions P5 and P6, Prokhorov’s theorem (Billingsley, 1999) can be used to establish that ${\Gamma }_1$ is compact in the topology of weak convergence. This compactness will be useful for proving the following result, which shows that there is an equilibrium point under our conditions.

Theorem 3. If T1-T3, T6, and P2-P7 hold, then there exists $T^{\star }\in {𝒯}_e$ and ${\Pi }^{\star }\in {\Gamma }_1$ such that, for all $T\in {𝒯}_e$ and $\Pi \in {\Gamma }_1$, it is true that $r\left(T^{\star },\Pi \right)\le r\left(T^{\star },{\Pi }^{\star }\right)\le r\left(T,{\Pi }^{\star }\right)$.

Combining the above with Lemma 10 in Section 7.2.3 establishes (2), that is, that the conclusion of Theorem 3 remains valid if $\Pi$ varies over $\Gamma$ rather than over ${\Gamma }_1$.

3. AMC Meta-Learning Algorithm

We now present an AMC meta-learning strategy for obtaining a $\Gamma$-minimax estimator within some class $𝒯$. Here we suppose that $𝒯=\left\{T_t:t\in \tau \right\}$, where each $T_t$ is an estimator indexed by a finite-dimensional parameter $t$ that belongs to some set $\tau$. We note that this framework encapsulates: model-based approaches (e.g., Hochreiter et al., 2001), where $T_t$ can be evaluated by a single pass of $\left(\mathit{d},x_0\right)$ through a neural network with weights $t$; optimization-based approaches, where $t$ are the initial weights of some estimate that are subsequently optimized based on $\mathit{d}$ (e.g., Finn et al., 2017); and metric-based approaches, where $t$ indexes a measure of similarity ${\alpha }_t$ that is used to obtain an estimate of the form ${\sum }_{i=1}^n {\alpha }_t\left(x_i,x_0\right)y_i$ (e.g., Vinyals et al., 2016).

We suppose that all estimators in $𝒯$ satisfy the equivariance property (5), which can be arranged by prestandardizing the outcome and features and then poststandardizing the final prediction — see Algorithm 2 for an example. Since all $T\in 𝒯$ satisfy (5), Theorem 2 shows that it suffices to consider a collection ${\Gamma }_1$ of priors with support on ${𝒫}_1$, that is, so that, for all $\Pi \in {\Gamma }_1,P~\Pi$ satisfies (6) almost surely. To ensure that the priors are easy to sample from, we parameterize them via generator functions $G_g$ (Goodfellow et al., 2014) that are indexed by a finite-dimensional $g$ that belongs to some set $\gamma$. Each $G_g$ takes as input a source of noise $U$ drawn from a user-specified distribution ${\nu }_u$ and outputs the parameters indexing a distribution in $𝒫$ (Luedtke et al., 2020). Though this form of sampling limits to parametric families $𝒫$, the number of parameters indexing this family may be much larger than the sample size $n$, which can, for all practical purposes, lead to a nonparametric estimation problem. For each $g$, we let ${\Pi }_g$ denote the distribution of $G_g\left(U\right)$ when $U~{\nu }_u$. We then let ${\Gamma }_1=\left\{{\Pi }_g:g\in \gamma \right\}$. It is worth noting that classes ${\Gamma }_1$ that are defined in this way will not generally satisfy the conditions P5-P7 used in Theorem 3. To iteratively improve the performance of the prior, we require the ability to differentiate realized datasets through the parameters indexing the prior. To do this, we assume that, for each $P\in 𝒫$, the user has access to a generator function $H_P:𝒱\to \mathbb{R}$ such that $H_P\left(V\right)$ has the same distribution as $(X,Y)~P$ when noise $V$ is drawn from a user-specified distribution ${\nu }_v$. We suppose that, for all realizations of the noise $u$ in the support of ${\nu }_u$ and $v$ in the support of ${\nu }_v$, the function $g\mapsto H_{G_g\left(u\right)}\left(v\right)$ is differentiable at each parameter value $g$ indexing the prior.

The AMC learning strategy is presented in Algorithm 1. The algorithm takes stochastic gradient steps on the parameters indexing an estimator and prior generator to iteratively reduce and increase the Bayes risk, respectively. All gradients in the algorithm can be computed via backpropagation using standard software — in our experiments, we used Pytorch for this purpose (Paszke et al., 2019). Note that, when computing ${\nabla }_g$ Loss, the dependence of Loss on $g$ is tracked through the dependence of $P$ on $g$ on line 5, the dependence of $X_0$ and $\mathit{D}={\left(X_i,Y_i\right)}_{i=1}^n$ on $P$ on lines 6 and 7, and the the dependence of Loss on $P,X_0$, and $\mathit{D}$ on line 8. We caution that, when the outcome or some of the features are discrete, ${\nabla }_g$ Loss will not generally represent an unbiased estimate of the gradient of $g\mapsto r\left(T_t,{\Pi }_g\right)$, which can cause Algorithm 1 to perform poorly. To handle these cases, the algorithm can be modified to instead obtain an unbiased gradient estimate using the likelihood ratio method (Glynn, 1987).

Though studying the convergence properties of the minimax optimization in Algorithm 1 is not the main focus of this work, we now provide an overview of how results from Lin et al. (2019) can be used to provide some guarantees for this algorithm. When doing so, we focus on the special case where there exists some $\ell <\mathrm{\infty }$ such that, for all $g,t\mapsto r\left(T_t,G_g\right)$ is differentiable with $\ell$-Lipschitz gradient and, for some finite (but potentially large) collection ${𝒫}_D≔\left\{P_1,\dots ,P_D\right\}\subset 𝒫,\Gamma$ is the collection of all mixtures of distributions in ${𝒫}_D$. We also suppose that the parameter $g$ indexing the generator $G_g$ takes values on the $D-1$ simplex and that this generator is parameterized in such a way that ${\nu }_u\circ G_g^{-1}$ has the same distribution as the mixture of distributions in ${𝒫}_D$ that places mass $g_j$ on distribution $P_j,j=1,\dots ,D$. In this case, provided the learning rates ${\eta }_1$ and ${\eta }_2$ are chosen appropriately, Theorem 4.5 in Lin et al. (2019) gives guarantees on the number of iterations required to return an $ϵ$-stationary point $T_{t_K}$ (idem, Definition 3.7) within a specified number of iterations — this stationary point is such that there exists a $t^{\prime }$ near $t_K$ at which the function $t\mapsto {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T_t,\Pi \right)$ has at least one small subgradient (idem, Lemma 3.8, for details). If, also, $t\mapsto T_t\left(\mathit{d}\right)$ is convex for all $\mathit{d}$, then this also implies that $T_{t_K}$ is nearly $\Gamma$-minimax. If, alternatively, the prior update step in Algorithm 1 (line 13) is replaced by an oracle optimizer such that, at each iteration, $g$ is defined as a true maximizer of the Bayes risk $g\mapsto r\left(T,{\Pi }_g\right)$, then Theorem E.4 of Lin et al. (2019) similarly guarantees that an $ϵ$-stationary point will be reached within a specified number of iterations.

Alternatives to Algorithm 1 are possible. As one example, the stochastic gradient descent ascent optimization scheme could be replaced by an extragadient method (Korpelevich, 1976), which has been shown to perform well in generative adversarial network settings (Gidel et al., 2018). As another example, the prior distribution could, in principle, be specified via its density rather than as the pushforward distribution ${\nu }_u\circ G_g^{-1}$ defined by the generator. While this density-based parameterization may make it easier to relate the specified priors to commonly used probability distributions, it may also lead to challenges since sampling from a distribution specified by its density is generally a hard problem that necessitates the use of numerical approaches such as Markov chain Monte Carlo methods (Hastings, 1970; Geman and Geman, 1984). Because the prior is updated at each of the $K$ iterations, it seems that many instances of these numerical sampling schemes would need to be run before theb termination of the AMC algorithm. Identifying a means to expedite the convergence of this density-based approach is an interesting area for future work.

4. Proposed Class of Estimators

4.1. Equivariant Estimator Architecture

Algorithm 2 presents our proposed estimator architecture, which relies on four modules. Each module $k$ can be represented as a function $m_k$ belonging to a collection ${ℳ}_k$ of functions mapping from ${\mathbb{R}}^{a_k}$ to ${\mathbb{R}}^{b_k}$, where the values of $a_k$ and $b_k$ can be deduced from Algorithm 2. For given data $\mathit{d}$, a prediction at a feature $x_0$ can be obtained by sequentially calling the modules and, between calls, either mean pooling across one of the dimensions of the output or concatenating the evaluation point as a new column in the output matrix.

We let ${𝒯}_{ℳ}$ represent the collection of all prediction procedures described by Algorithm 2, where here ${\left(m_k\right)}_{k=1}^4$ varies over ${\prod }_{k=1}^4 {ℳ}_k$. We now give conditions under which the proposed architecture yields an equivariant estimator.

• M1)$m_1(AvB{)}_{\mathrm{*}\mathrm{*}\ell }=A\left[m_1(v{)}_{\mathrm{*}\mathrm{*}\ell }\right]B$ for all $m_1\in {ℳ}_1,A\in 𝒜,B\in ℬ,v\in {\mathbb{R}}^{n\times p\times 2}$, and $\ell \in \left\{1,\dots ,o_1\right\}$.
• M2)$m_2\left(Bv\right)=B{m}_2\left(v\right)$ for all $m_2\in {ℳ}_2,B\in ℬ$, and $v\in {\mathbb{R}}^{p\times o_1}$.
• M3)$m_3\left(Bv\right)=B{m}_3\left(v\right)$ for all $m_3\in {ℳ}_3,B\in ℬ$, and $v\in {\mathbb{R}}^{p\times o_2}$.

Theorem 4. If M1-M3, then all $T\in {𝒯}_{ℳ}$ satisfy (4) and (5).

4.2. Neural Network Parameterization

In our experiments, we choose the four module classes ${ℳ}_k,k=\mathrm{1,2},\mathrm{3,4}$, indexing our estimator architecture to be collections of neural networks. For each $k$, we let ${ℳ}_k$ contain the neural networks consisting of $h_k$ hidden layers of widths $w_k^1,w_k^2,\dots ,w_k^{h_k}$, where the types of layers used depends on the module $k$. When $k=1$, multi-input-output channel equivariant layers as defined in Hartford et al. (2018) are used. In particular, for $j=1,\dots ,h_1+1$, we let ${ℒ}_1^j$ denote the collection of all such layers that map from ${\mathbb{R}}^{n\times p\times w_1^{j-1}}$ to ${\mathbb{R}}^{n\times p\times w_1^j}$, where we let $w_1^0=2$ and $w_1^{h_1+1}=o_1$. For each $j$, each member $L_1^j$ of ${ℒ}_1^j$ is equivariant in the sense that, for all $A\in 𝒜,B\in ℬ$, and $v\in {\mathbb{R}}^{n\times p\times w_1^{j-1}},L_1^j(AvB{)}_{\mathrm{*}\mathrm{*}\ell }=A{L}_1^j(v{)}_{\mathrm{*}\mathrm{*}\ell }B$ for all $\ell =1,\dots ,o_1$. When $k=\mathrm{2,3}$, multi-input-output channel equivariant layers as described in Eq. 22 of Zaheer et al. (2017) are used, except that we replace the sum-pool term in that equation with a mean-pool term (see the next subsection for the rationale). In particular, for $j=1,\dots ,h_k+1$, we let ${ℒ}_k^j$ denote the collection of all such equivariant layers that map from ${\mathbb{R}}^{p\times w_k^{j-1}}$ to ${\mathbb{R}}^{p\times w_k^j}$. For each $j$, each member $L_k^j$ of ${ℒ}_k^j$ is equivariant in the sense that, for all $B\in ℬ$ and $v\in {\mathbb{R}}^{p\times w_k^{j-1}}$, $L_k^j\left(Bv\right)=B{L}_k^j\left(v\right)$. When $k=4$, standard linear layers mapping from ${\mathbb{R}}^{w_4^{j-1}}$ to ${\mathbb{R}}^{w_4^j}$ are used for each $j=1,\dots ,h_4+1$, where $w_4^0=o_3$ and $w_4^{h_4+1}=1$. For each $j$, we let ${ℒ}_4^j$ denote the collection of all such layers. For a user-specified activation function $q$, we then define the module classes as follows for $k=\mathrm{1,2},\mathrm{3,4}$ :

$${ℳ}_k≔\left\{v\mapsto q\circ L_k^{h_k+1}\circ q\circ L_k^{h_k}\circ \dots \circ q\circ L_k^1\left(v\right):L_k^j\in {ℒ}_k^j,j=\mathrm{1,2},\dots ,h_k+1\right\}.$$

Notably, ${ℳ}_1$ satisfies M1 (Ravanbakhsh et al., 2017; Hartford et al., 2018), and ${ℳ}_2$ and ${ℳ}_3$ satisfy M2 and M3, respectively (Ravanbakhsh et al., 2016; Zaheer et al., 2017). Each element of ${ℳ}_4$ is a multilayer perceptron.

The proposed architecture bears some resemblance to CNPs (Garnelo et al., 2018). Like our proposed architecture, CNPs are invariant to permutations of the observations. Nevertheless, CNPs fail to satisfy the other properties imposed on ${𝒯}_e$, namely invariance to shifts, rescalings, and permutations of the features and equivariance to shifts and rescalings of the outcome. Moreover, a decision-theoretic rationale for making CNPs invariant to permutations of the observations has not yet been provided in the literature, for example, via a Hunt-Stein-type theorem.

4.3. Pros and Cons of Proposed Architecture

A benefit of using the proposed architecture in Algorithm 2 is that Modules 1 and 2 can be evaluated without knowing the feature $x_0$ at which a prediction is desired. As a consequence, these modules can be precomputed before making predictions at new feature values, which can lead to substantial computational savings when the number of values at which predictions will be made is large. Another advantage of the proposed architecture is that it can be evaluated on a dataset that has a different sample size $n$ than did the datasets used during meta-training. In the notation of Eq. 4 from Hartford et al., this corresponds to noting that the weights from an ${\mathbb{R}}^{N\times M\times k}\to {\mathbb{R}}^{N\times M\times o}$ multi-input-output channel layer can be used to define an ${\mathbb{R}}^{N^{\prime }\times M\times k}\to {\mathbb{R}}^{N^{\prime }\times M\times o}$ layer for which the output $Y_{n,m}^{⟨o⟩}$ is given by the same symbolic expression as that displayed in Eq. 4 from that work, but now with $n$ ranging over $1,\dots ,N^{\prime }$. We will show in our upcoming experiments that procedures trained using 500 observations can perform well even when evaluated on datasets containing only 100 observations. It is similarly possible to evaluate the proposed architecture on datasets containing a different number of features than did the datasets used during meta-training — again see Eq. 4 in Hartford et al. (2018), and also see Eq. 22 in Zaheer et al. (2017), but with the sum-pool term replaced by a mean-pool term. The rationale for replacing the sum-pool term by a mean-pool term is that this will ensure that the scale of the hidden layers will remain fairly stable when the number of testing features differs somewhat from the number of training features.

A disadvantage of the proposed architecture is that it currently has no established universality guarantees. Such guarantees have been long available for standard multilayer perceptrons (e.g., Cybenko, 1989; Hornik, 1991), and have recently also become available for certain invariant architectures (Maron et al., 2019). In future work, it would be interesting to see if the arguments in Maron et al. (2019) can be modified to provide universality guarantees for our architecture. Establishing such results may also help us to overcome a second disadvantage of our architecture, namely that the resulting neural network classes will not generally satisfy the convexity condition T6 used in Theorem 3. If a network class ${𝒯}_{ℳ}$ that we have proposed can be shown to satisfy a universality result for some appropriate convex class ${𝒯}_c$, and if ${𝒯}_{ℳ}$ is itself a subset of ${𝒯}_c$, then perhaps it will be possible to invoke Theorem 3 to establish an equilibrium result over the class of estimators ${𝒯}_c$, and then to use this result to establish an (approximate) equilibrium result for ${𝒯}_{ℳ}$. To ensure that conditions T1-T3 are satisfied, such an argument will likely require that the weights of the networks in ${𝒯}_{ℳ}$ be restricted to belong to some compact set.

5. Numerical Experiments

5.1. Overview

In this section, we present the results from two sets of numerical experiments, with the first corresponding to benchmarks from the meta-learning literature and the second consisting of settings designed to evaluate the performance of our method relative to that of analytically-derived estimators that are commonly used in practice for which theoretical performance guarantees are available. In each example, the collection of estimators $𝒯$ is parameterized as the network architecture introduced in Section 4.2 with $o_1=o_2=50,o_3=10,h_1=h_3=10,h_2=h_4=3$, and, for $k=\mathrm{1,2},\mathrm{3,4},w_k=100$. For each module, we use the leaky ReLU activation $q\left(z\right)≔\mathrm{m}\mathrm{a}\mathrm{x}\{z,0\}+0.01\mathrm{m}\mathrm{i}\mathrm{n}\{z,0\}$. At the end of this section, we report the results of an ablation study that evaluates the extent to which imposing invariance to permutations of the observations and features improves performance.

All experiments were run in Pytorch 1.0.1 on Tesla V100 GPUs using Amazon Web Services. The code used to conduct the experiments can be found at https://github.com/alexluedtke12/amc-meta-learning-of-optimal-prediction-procedures. Further experimental details can be found in Appendix D.

5.2. Meta-Learning Benchmarks

5.2.1. Preliminaries

We now evaluate the performance of AMC on widely used meta-learning benchmarks. As described in the Introduction, existing meta-learning algorithms tend to be Bayesian in nature, where the goal during meta-training is to learn an estimator with small Bayes risk under a specified prior $\Pi$. Consequently, when adjudicating performance in this study, we will primarily focus on the evaluation of each learned estimator $T$ in terms of its Bayes MSE against this fixed prior $\Pi$, defined as $\int {\mathrm{E}}_P\left[\int {\left\{T\left(\mathit{D}\right)\left(x_0\right)-{\mu }_P\left(x_0\right)\right\}}^{2}d{P}_X\left(x_0\right)\right]d\Pi \left(P\right)$.

Because our method is designed to learn adversarially over a collection of priors $\Gamma$ that satisfies the invariance properties P1, P2, and P3, we define the collection $\Gamma$ used when training our method as the smallest collection of priors that satisfies these three properties and contains $\Pi$. It can be verified that ${\Gamma }_1$ is a singleton in this case, so that the generator is a constant function and is never updated in these benchmark settings. Though this simplified meta-training may make it appear that AMC will not be robust to an adversarial choice of prior, it is worth noting that the learned estimator in fact is robust to such a choice in the sense that the Bayes risk of the learned estimator will be invariant under permutations of the features and also under shifts and rescalings of the outcomes and features. The main motivation for using a small $\Gamma$ when comparing to these benchmarks is that doing so will help inform on the performance of the estimator architecture that we proposed in Section 4 even in Bayesian settings for which existing meta-learning approaches are tailor-made.

We compare the performance of AMC to that of two popular meta-learning methods for which code is readily available: MAML (Finn et al., 2017) and CNPs (Garnelo et al., 2018). Because these algorithms do not prestandardize the features and outcomes, they may have large standardized Bayes MSEs (the Bayes risk derived from Eq. 1) if these quantities are simply shifted or rescaled. To ensure that possible discrepancies in performance between AMC and MAML or CNPs are not solely due to prestandardization, we also compare our method to natural variants of MAML and CNPs that, like AMC, are robust to such shifts and rescalings. For each method, these variants prestandardize the features and outcomes, and then, in an analogous fashion to line 9 of Algorithm 2, scale the final output by the sample standard deviation of the original training outcomes and shift by their sample mean. These algorithms, which we refer to as MAML-Eq and CNP-Eq, are invariant to shifts and rescalings of the features and equivariant to shifts and rescalings of the outcomes. Details on the MAML and CNP implementations used can be found in Appendix D.1.

5.2.2. Sinusoidal Regression

We start with a benchmark few-shot regression setting from that is commonly used in the meta-learning literature. The prior $\Pi$ is defined as follows. The feature is 1-dimensional and is Unif (−5, 5) distributed, and the regression function ${\mu }_P$ takes the form $x\mapsto a\mathrm{s}\mathrm{i}\mathrm{n}(x-b)$, where the parameters $a$ and $b$ are drawn independently from a Unif (0.1, 5.0) and Unif $(0,\pi )$ distribution, respectively (Finn et al., 2017). Following related meta-learning benchmarks (Finn et al., 2018; Vuorio et al., 2018), the error ${ϵ}_P$ added to the signal ${\mu }_P\left(X\right)$ is distributed as $N\left(0,{0.3}^2\right)$. We use the same sample sizes as were used in Finn et al. (2017), namely $n=\mathrm{5,10}$, and 20.

We now report on the performance of the various meta-learning approaches in this setting. In Table 1a, we can see that MAML and CNPs consistently outperform their equivariant counterparts, namely MAML-Eq and CNP-Eq, in this setting. Nevertheless, as we noted earlier, MAML and CNPs are non-robust in that their standardized MSE can be made large by simply shifting or rescaling the outcomes or features. In Figure S5 in the appendix we provide evidence that this is indeed the case. As a particularly striking example, when $n=5$, scaling the feature down by a factor of 5 leads to 24-fold and 149-fold increases in the MSEs of MAML and CNPs, respectively. The degradation of performance worsens with sample size. Indeed, when $n=20$, the same rescaling leads to 144-fold and 487-fold increases in the MSEs of these two methods. Consequently, even seemingly innocuous preprocessings of the data, such as applying an affine transformation to change the unit of measurement, can have a dramatic impact on the performance of MAML and CNPs. In contrast, the standardized MSE performance of MAML-Eq and CNP-Eq is invariant to such preprocessings of the data.

Table 1:

Bayes MSEs of meta-learning approaches in the meta-learning benchmark experiments, where the Bayes MSE is defined as the squared difference between the predictions and true underlying regression function, averaged across draws of the data-generating distribution from the prior and the feature from the feature distribution. Standard errors all <0.005 in the sinusoid experiment and < 0.001 in the Gaussian process experiments.

(a) Sinusoid				(b) Gaussian process
					1d feature		5d feature
	n=5	10	20		n=5	50	5	50
MAML*	0.22	0.10	0.03	MAML*	0.85	0.13	1.00	1.00
CNP*	0.05	0.02	0.01	CNP*	0.47	0.04	0.95	0.73
MAML-Eq	2.06	0.47	0.07	MAML-Eq	0.93	0.13	1.22	1.02
CNP-Eq	1.13	0.13	0.04	CNP-Eq	0.56	0.04	1.12	0.73
AMC (ours)	0.89	0.09	0.03	AMC (ours)	0.56	0.03	1.11	0.66

^*As these two algorithms do not prestandardize the features or outcomes, their standardized MSEs can be made large by simply shifting or rescaling the features and outcomes. See Figure S5 for more information.

Table 1a also displays results for AMC. AMC consistently outperforms the robust versions of existing algorithms, namely MAML-Eq and CNP-Eq. When compared with the non-robust variants, AMC is outperformed by MAML when $n=5$, outperforms MAML when $n=10$, and has about the same performance as MAML when $n=20$. CNPs perform better than MAML and AMC, though this difference begins to diminish as the sample size increases.

5.2.3. Gaussian Process Regression

We next consider a benchmark Gaussian process regression setting. We consider two cases for the prior. The first is the same as that considered in Garnelo et al. (2018), except that they considered the noise-free case where ${ϵ}_P=0$ almost surely, whereas we consider the noisy case where the errors ${ϵ}_P$ are homoscedastic and distributed as $N\left(0,{0.3}^2\right)$. Considering a noisy case where ${ϵ}_P$ is non-degenerate is necessary for the standardized MSE that we consider to be well-defined, and also better reflects real-world regression scenarios where observed outcomes are rarely, if ever, deterministic functions of the features considered. Following Garnelo et al. (2018), the feature is 1-dimensional and follows a Unif (−2, 2) distribution, and the regression function ${\mu }_P$ is drawn from a mean-zero Gaussian process with a squared exponential kernel with lengthscale 0.4 and variance 1. We also use the same sample sizes as were used in that work, namely $n=5$ and 50. The second case that we consider is the same as the first except that the feature $X$ is 5-dimensional, where the entries of $X$ are independent Unif(−2, 2) random variables, and the lengthscale is taken to be equal to 1.2.

Table 1b displays the performance of the various methods in this setting. Adversarial Monte Carlo noticeably outperforms MAML and MAML-Eq across all settings except the 5-dimensional, $n=5$ case, where MAML performs slightly better than does AMC. The ordering between AMC and the CNP-based methods varies by sample size. At the smaller sample size considered $(n=5)$, AMC outperforms the robust CNP-based method, namely CNP-Eq, but is outperformed by the non-robust method, namely CNP. In the larger sample size considered $(n=50)$, AMC outperforms both CNP and CNP-Eq. The fact that AMC outperforms CNP in this setting is notable given that CNPs are designed to mimic the desirable properties of Gaussian process regression procedures (Garnelo et al., 2018).

5.3. Comparing to (Regularized) Empirical Risk Minimizers

5.3.1. Preliminaries

We now compare the performance of our approach to that of existing estimators that are commonly used in practice for which theoretical performance guarantees are available. The examples differ in the definitions of the model $𝒫$ and the collection $\Gamma$ of priors on $𝒫$. In each case, $\Gamma$ satisfies the invariance properties P1, P2, and P3. By the equivariance of the estimators in $𝒯$, Theorem 2 shows that it suffices to consider a collection of priors ${\Gamma }_1$ with support on ${𝒫}_1$. Hence, it suffices to define the collection ${𝒫}_1\subset 𝒫$ of distributions $P$ satisfying (6). By P2 and P3, we see that $𝒫={\cup }_{P\in {𝒫}_1}𝒫\left(P\right)$, where $𝒫\left(P\right)$ consists of the distributions of $(a+b\odot X,\tilde{a}+\tilde{b}Y)$ when $(X,Y)~P$; here, $a,b,\tilde{a}$, and $\tilde{b}$ vary over ${\mathbb{R}}^p,{\left({\mathbb{R}}^{+}\right)}^p,\mathbb{R}$, and ${\mathbb{R}}^{+}$, respectively. In each setting, the submodel ${𝒫}_1$ takes the form

$${𝒫}_1≔\left\{P:{\mu }_P\in ℛ,P_X\in {𝒫}_X,{ϵ}_P\mid X\stackrel{P}{~}N\left(\mathrm{0,1}\right)\right\}$$

and the $p=10$ dimensional feature $X$ is known to be drawn from a distribution in the set ${𝒫}_X$ of $N\left(0_p,\Sigma \right)$ distributions, where $\Sigma$ varies over all positive-definite $p\times p$ covariance matrices with diagonal equal to $1_p$. The collections $ℛ$ of regression functions differ in the examples and are detailed in the coming subsections. These collections are indexed by a sparsity parameter $\mathfrak{s}$ that specifies the number of features that may contribute to the regression function ${\mu }_P$. In each setting, we considered all four combinations of $\mathfrak{s}\in \left\{\mathrm{1,5}\right\}$ and $n\in \left\{\mathrm{100,500}\right\}$, where $n$ denotes the number of observations in the datasets $\mathit{d}$ used to evaluate the performance of the final learned estimators. For each $n$, we evaluated the performance of AMC meta-trained with datasets of size $n_{mt}=100$ observations (AMC100) and $n_{mt}=500$ observations (AMC500).

5.3.2. Sparse Linear Regression

We next considered the setting where ${\mu }_P$ belongs to a sparse linear model and the feature is $p=10$ dimensional. In this setting,

$$ℛ≔\left\{x\mapsto {\beta }^{\mathrm{\top }}x:\parallel \beta {\parallel }_0\le \mathfrak{s},\parallel \beta {\parallel }_1\le 5\right\},$$ (7)

where $\parallel a{\parallel }_0≔\#\left\{j:a_j\ne 0\right\}$ and $\parallel a{\parallel }_1≔{\sum }_{j=1}^p \left|a_j\right|$. The collection $\Gamma$ is described in Appendix D.

For each sparsity level $\mathfrak{s}\in \left\{\mathrm{1,5}\right\}$, we evaluated the performance of the prediction procedure trained at sparsity level $\mathfrak{s}$ using two priors. Both priors sample the covariance matrix of the feature distribution $P_X$ from the Wishart prior ${\Pi }_X$ described in Appendix D.2.1 and let $\beta =(\alpha ,0)$ for a random $\alpha$ satisfying $\parallel \alpha {\parallel }_1\le 5$. They differ in how $\alpha$ is drawn. Both make use of a uniform draw $Z$ from ${\ell }_1$ ball $\left\{a\in {\mathbb{R}}^{\mathfrak{s}}:\parallel a{\parallel }_1=5\right\}$. The first sets $\alpha =Z$, whereas the second sets $\alpha =UZ$ for $U~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(\mathrm{0,1}\right)$ drawn independently of $Z$. We will refer to the two settings as ‘boundary’ and ‘interior’, respectively. We refer to the $\mathfrak{s}=1$ and $\mathfrak{s}=5$ cases as the ‘sparse’ and ‘dense’ settings, respectively. Further details can be found in Appendix D.2.2.

In this example, AMC leverages knowledge of the underlying sparse linear regression model by generating synthetic training data from distributions $P$ for which $E_P[Y\mid X=\cdot ]$ belongs to the class $ℛ$ defined in Eq. 7 (see line 5 of Algorithm 1). Therefore, we aimed to compare AMC’s performance to that of estimators that also take advantage of this linearity. Ideally, we would compare AMC’s performance to that of the true $\Gamma$-minimax estimator. Unfortunately, as is the case in most problems, the form of this estimator is not known in this sparse linear regression setting. Therefore, we instead compared AMC’s performance to ordinary least squares (OLS) and lasso (Tibshirani, 1996) with tuning parameter selected by 10-fold cross-validation, as implemented in scikit-learn (Pedregosa et al., 2011).

Table 2a displays performance for the sparse setting. We see that AMC outperformed OLS and lasso for the boundary priors, and was outperformed for the interior priors. Surprisingly, AMC500 outperformed AMC100 for the interior prior when $n=100$ observations were used to evaluate performance. The fact that AMC100 was trained specifically for the $n=100$ case suggests that a suboptimal equilibrium may have been reached in this setting. Table 2b displays performance for the dense setting. Here AMC always performed at least as well as OLS and lasso when $n_{mt}=n$, and performed comparably even when $n_{mt}\ne n$.

Table 2:

MSEs based on datasets of size $n$ in the linear regression settings. Standard errors all < 0.001.

(a) Sparse signal
	Boundary		Interior
	n=100	500	100	500
OLS	0.12	0.02	0.12	0.02
Lasso	0.06	0.01	0.06	0.01
AMC100 (ours)	0.02	<0.01	0.11	0.09
AMC500 (ours)	0.02	<0.01	0.07	0.04
(b) Dense signal
	Boundary		Interior
	n=100	500	100	500
OLS	0.13	0.02	0.13	0.02
Lasso	0.11	0.02	0.09	0.02
AMC100 (ours)	0.10	0.04	0.08	0.02
AMC500 (ours)	0.09	0.02	0.09	0.02

5.3.3. Fused Lasso Additive Model

We next considered the setting where $P$ belongs to a variant of the fused lasso additive model (FLAM) (Petersen et al., 2016) and the feature is $p=10$ dimensional. This model enforces that ${\mu }_P$ belong to a generalized additive model, that only a certain number of the components can be different from the zero function, and that the sum of the total variations of the remaining components is not too large. We recall that the total variation $V\left(f\right)$ of $f:\mathbb{R}\to \mathbb{R}$ is equal to the supremum of ${\sum }_{\ell =1}^k \left|f\left(a_{\ell +1}\right)-f\left(a_{\ell }\right)\right|$ over all ${\left(a_{\ell }\right)}_{\ell =1}^{k+1}$ such that $k\in \mathbb{N}$ and $a_1<a_2<\dots <a_{k+1}$ (Cohn, 2013). Let $v\left(\mu \right)≔{\left(V\left({\mu }_j\right)\right)}_{j=1}^p$. Writing $x_j$ to denote feature $j$, the model we considered imposes that ${\mu }_P$ falls in

$$ℛ≔\left\{x\mapsto \sum _{j=1}^p {\mu }_j\left(x_j\right):\parallel v\left(\mu \right){\parallel }_1\le M,\parallel v\left(\mu \right){\parallel }_0\le \mathfrak{s}\right\}.$$

We take $M=10$ in the experiments in this section. The collection $\Gamma$ is described in Appendix D.

In this example, we preprocessed the features before supplying them to the estimator. In particular, we replaced each entry with its rank statistic among the $n$ observations so that, for each $i\in \{1,\dots ,n\}$ and $j\in \{1,\dots ,p\}$, we replaced ${\mathit{x}}_{ij}$ by ${\sum }_{k=1}^n I\left\{{\mathit{x}}_{ij}\ge {\mathit{x}}_{kj}\right\}$ and $x_{0j}$ by ${\sum }_{k=1}^n I\left\{x_{0j}\ge {\mathit{x}}_{kj}\right\}$. This preprocessing step is natural given that the FLAM estimator (Petersen et al., 2016) also only depends on the features through their ranks. An advantage of making this restriction is that, by the homoscedasticity of the errors and the invariance of the rank statistics and total variation to strictly increasing transformations, the learned estimators should perform well even if the feature distributions do not belong to a Gaussian model, but instead belong to a much richer Gaussian copula model.

We evaluated the performance of the learned estimators using variants of simulation scenarios 1–4 from Petersen et al. (2016). The level of smoothness varies across the settings (see Fig. 2 in that work). In the variants we considered, the true regression function either contains ${\mathfrak{s}}_0=1$ (‘sparse’) or ${\mathfrak{s}}_0=4$ (‘dense’) nonzero components. In the sparse setting, we evaluated the performance of the estimators that were meta-trained at sparsity level $\mathfrak{s}=1$, and, in the dense setting, we evaluated the performance of the estimators that were meta-trained at $\mathfrak{s}=5$. Further details can be found in Appendix D.2.3.

Figure 2:

Improvement of AMC estimators over existing estimators, in terms of differences of cross-validated MSEs of FLAM and AMC FLAM (x-axis) and Lasso and AMC Linear (y-axis). Positive values indicate that AMC outperformed the comparator. AMC performed similarly to or better than existing estimators in settings where the number of features in the dataset was the same as were used in meta-training. As expected, the performance was somewhat worse for datasets that had fewer features than were used during meta-training, though, surprisingly, it was still sometimes better than that of existing methods.

Similarly to as in the previous example, AMC leverages knowledge of the possible forms of the regression function that is imposed by $ℛ$ — in this case, the model for the regression function is nonparametric but does impose that this function belongs to a particular sparse generalized additive model. Though there does not exist a competing estimator that is designed to optimize over $ℛ$, the FLAM estimator (Petersen et al., 2016) optimizes over the somewhat larger, non-sparse model where $\mathfrak{s}=p$. We, therefore, compared the performance of AMC to this estimator as a benchmark, with the understanding that AMC is slightly advantaged in that it has knowledge of the underlying sparsity pattern. Nevertheless, we view this experiment as an important proof-of-concept, as it is the first, to our knowledge, to evaluate whether it is feasible to adversarially meta-learn a prediction procedure within a nonparametric regression model.

To illustrate the kinds of functions that AMC can approximate, Fig. 1 displays examples of AMC500 fits from scenario 3 when $(n,\mathfrak{s})=\left(\mathrm{500,1}\right)$. Table 3 provides a more comprehensive view of the performance of AMC and compares it to that of FLAM. Table 3a displays performance for the sparse setting. The AMC procedures meta-trained with $n_{mt}=n$ observations outperformed FLAM for all of these settings. Interestingly, AMC procedures meta-trained with $n_{mt}\ne n$ also outperformed FLAM in a majority of these settings, suggesting that learned procedures can perform well even at different sample sizes from those at which they were meta-trained. In the dense setting (Table 3b), AMC500 outperformed both AMC100 and FLAM in all but one setting (scenario 4, $n=100$), and in this setting both AMC100 and AMC500 dramatically outperformed FLAM. The fact that AMC500 also sometimes outperformed AMC100 when $n=100$ in the linear regression setting suggests that there may be some benefit to training a procedure at a larger sample size than that at which it will be evaluated. We leave an investigation of the generality of this phenomenon to future work.

Figure 1:

Examples of AMC500 fits (thin blue lines) based on $n=500$ observations drawn from distributions at sparsity level $\mathfrak{s}=1$ with four possible signal components (thick black lines). Predictions obtained at different signal feature values with all 9 other features set to zero.

Table 3:

MSEs based on datasets of size $n$ in the FLAM settings. Standard errors for FLAM all < 0.04 and for AMC all < 0.01.

(a) Sparse signal
	Scenario 1		Scenario 2		Scenario 3		Scenario 4
	n=100	500	100	500	100	500	100	500
FLAM	0.44	0.12	0.47	0.17	0.38	0.11	0.51	0.19
AMC100 (ours)	0.34	0.20	0.18	0.08	0.27	0.14	0.17	0.08
AMC500 (ours)	0.48	0.12	0.19	0.06	0.35	0.10	0.23	0.08
(b) Dense signal
	Scenario 1		Scenario 2		Scenario 3		Scenario 4
	n=100	500	100	500	100	500	100	500
FLAM	0.59	0.17	0.65	0.24	0.53	0.16	0.76	0.36
AMC100 (ours)	1.20	0.91	0.47	0.39	0.87	0.57	0.30	0.30
AMC500 (ours)	0.58	0.15	0.37	0.08	0.46	0.12	0.36	0.09

5.4. Ablation Study to Evaluate the Performance of Permutation Invariance

We numerically evaluated the utility of imposing invariance in the architecture in Algorithm 2. To do this, we repeated the $n=n_{mt}=100$ and $n=n_{mt}=500$ FLAM settings, separately modifying the architecture to remove invariance to permutations of the observations and the features. In the case where the architecture was not invariant to permutations of the observations, we weakened M1 to the condition that $m_1(vB{)}_{\mathrm{*}\mathrm{*}\ell }=\left[m_1(v{)}_{\mathrm{*}\mathrm{*}\ell }\right]B$ for all $m_1\in {ℳ}_1,B\in ℬ,v\in {\mathbb{R}}^{n\times p\times 2}$, and $\ell =1,\dots ,o_1$. We used the same architecture as was used in our earlier experiment, except that each layer in Module 1 was replaced by a multi-input-output channel layer that is equivariant to permutations of the $p$ features (Zaheer et al., 2017), and the output of the final layer was of dimension ${\mathbb{R}}^{p\times o_1}$ so that the subsequent mean pooling layer could be removed. In the case where the architecture was not invariant to permutations of the features, we removed conditions M2 and M3 and also weakened M1 to the condition that $m_1(Av{)}_{\mathrm{*}\mathrm{*}\ell }=A\left[m_1(v{)}_{\mathrm{*}\mathrm{*}\ell }\right]$ for all $m_1\in {ℳ}_1$, $A\in 𝒜,v\in {\mathbb{R}}^{n\times p\times 2}$, and $\ell =1,\dots ,o_1$. We used the same architecture as in our earlier experiment except that Modules 2 and 3 were replaced by multilayer perceptrons and each layer in Module 1 was replaced by a multi-input-output channel layer that is equivariant to permutations of the $n$ observations.

Table 4 displays the results. In every setting considered, removing invariance to permutations of the observations led to a marked increase in the MSE of the estimator, with the degradation of performance tending to be worse at the larger sample size. In the most extreme scenario, the MSE of the non-invariant estimator was 38 times higher than that of the invariant estimator. Removing invariance to permutations of the features also tended to worsen performance, sometimes by a factor of 2 or 3, though there were a few settings where performance improved slightly (no more than 5%). Taken together, these results suggest that a priori enforcing that the estimator is invariant to permutations of the features and observations can dramatically improve performance.

Table 4:

Fold-change in MSEs for modifications of AMC in the FLAM settings with $n=100$, as compared to the performances of FLAM listed in Table 3. Standard errors all ≤ 0.03 times the fold-change in the MSE.

(a) Sparse signal
		Scenario 1		Scenario 2		Scenario 3		Scenario 4
		n=100	500	100	500	100	500	100	500
Not invariant to	observations	6.98	38.29	5.82	29.93	5.03	27.58	4.29	13.08
permutations of:	features	1.01	0.95	1.16	1.09	1.02	0.98	1.01	0.99
(b) Dense signal
		Scenario 1		Scenario 2		Scenario 3		Scenario 4
		n=100	500	100	500	100	500	100	500
Not invariant to	observations	1.86	14.68	1.69	8.60	1.97	14.20	1.51	4.70
permutations of:	features	1.05	2.55	0.99	1.98	1.09	3.02	1.04	1.67

6. Data Experiments

We also used real datasets to evaluate the performance of AMC100 estimators meta-trained in sparse linear regression settings (Section 5.3.2) or fused lasso additive model settings (Section 5.3.3). We compared the performance of our estimators to the estimators from our numerical experiments, namely, the OLS, lasso, and FLAM estimators. These estimators are natural comparators because they assume the same or similar models as do our AMC estimators; consequently, comparing to these estimators allows us to focus our discussion on differences in the performance of existing estimation strategies as compared to that of new meta-learned strategies, rather than on differences in underlying assumptions that could potentially be resolved by training a new AMC estimator in a different model.

Because the implementations of lasso and FLAM that we compared to both use 10-fold cross-validation to select tuning parameters, we also used 10-fold cross-validation to select tuning parameters for the AMC100 estimators. The first of these estimators, which we refer to as “AMC Linear”, selects a tuning parameter $\mathfrak{s}\in \{\mathrm{1,2},\dots ,10\}$ by finding the value of $\mathfrak{s}$ for which the cross-validated MSE of an AMC100 estimator trained in the sparse linear regression setting with sparsity level $\mathfrak{s}$ is minimal. The final prediction then corresponds to that returned by the AMC100 estimator trained in the model with this selected value of $\mathfrak{s}$. The second, which we refer to as “AMC FLAM”, selects two tuning parameters, one of which reflects the sparsity level $\mathfrak{s}$ of the problem and the other of which corresponds to the bound $M$ on the sum of the variation norms of the ${\mu }_j$ components in the fused lasso additive model. In particular, the tuning parameters $(\mathfrak{s},M)\in \{\mathrm{1,2},\dots ,10\}\times \left\{\mathrm{5,10,20}\right\}$ are chosen to be those that minimize the cross-validated MSE of an AMC100 estimator trained in the fused lasso additive model with parameters $(\mathfrak{s},M)$. Notably, each candidate estimator considered by AMC Linear and AMC FLAM only has access to 90, rather than 100, observations when selecting tuning parameter values using 10 -fold cross-validation on a dataset of size $n=100$. This does not pose a problem because, as was noted in Section 4.3, the trained estimators can be evaluated at different sample sizes than those at which they were trained.

In settings where both AMC-trained estimators and other estimators are available, it is natural to wonder whether there is a way to capitalize on the availability of both types of methods. Ensemble algorithms provide a natural means to do this, with stacked ensembles representing an especially appealing option given theoretical guarantees that adding base learners will not typically degrade performance (Van der Vaart et al., 2006; Van der Laan et al., 2007) and existing experiments showing that they often outperform all included base learners (e.g., Polley and Van der Laan, 2010). We, therefore, evaluate the performance of three stacked ensembles in these experiments. The first includes only the AMC Linear and AMC FLAM estimators as base learners. The second only includes the OLS, lasso, and FLAM estimators. The third includes all five of these estimators. Predictions of the base learners were combined using 10-fold cross-validation. Following the recommendation of Breiman (1996), we employed a non-negative least squares estimator for this combination step.

Our experiments make use of ten datasets. Six of these datasets are available through the University of California, Irvine (UCI) Machine Learning Repository (Dua and Graff, 2017), three were used to illustrate supervised learning machines in popular statistical learning textbooks (Friedman et al., 2001; James et al., 2013), and one was used as an illustrative example in the paper that introduced FLAM (Petersen et al., 2016). All of these datasets contain more than 100 observations. Five of them have at least 10 features and the others have fewer (5, 6, 6, 7, and 9). All outcomes are standardized to have empirical variance 1 so that, for each dataset, the cross-validated MSE performance of a sample mean for predicting the outcome is approximately 1. Further details on these datasets can be found in Appendix E.1.

We evaluated our learned estimators in three settings. First, we considered the case where the number of features in the datasets matched the number that they saw during training, namely 10. In particular, we evaluated the performance of AMC Linear and AMC FLAM in the 5 datasets that have 10 or more features by randomly selecting 100 observations and 10 features from each dataset and evaluating MSE on the held out observations. This and all other Monte Carlo evaluations of MSE described in what follows were repeated 200 times and averaged across the replications. Second, we evaluated the robustness of our learned estimators to a key assumption used during training. In particular, we evaluated the performance of our estimators (a) Datasets with same number of features as used during meta-training (b) Datasets with fewer features than used during meta-training on the 5 datasets that have fewer features than the 10 used during meta-training, again sampling 100 observations and evaluating MSE on the held out observations. Third, we evaluated the relative performance of our estimators at varying levels of signal sparsity for each of the ten datasets. In particular, for each training-test split of the data, we selected $\mathfrak{s}$ total features from the dataset, removed the remaining features, and then included $(10-\mathfrak{s})$ Gaussian noise features so that the dimension of the feature was always $p=10$.

We first discuss performance on datasets with the same number of features as were used during meta-training. Complete numerical results for estimator performance can be found in Table S5 in Appendix E.2. Here, we focus on graphical summaries of performance to communicate the key trends that we saw. Figure 2a shows that AMC FLAM performed similarly to or better than FLAM across all settings, and AMC Linear performed similarly to lasso across all settings. We have compared AMC Linear to lasso as a baseline in this figure because lasso performed similarly to or better than OLS across all settings. Figure 3a shows that stacking all available base learners consistently yielded better performance than did only stacking only the existing estimators or the AMC estimators. This stacked ensemble also outperformed all base learners considered. These results suggest that incorporating AMC estimators into regression pipelines (a) Datasets with same number of features as used during meta-training (b) Datasets with fewer features than used during meta-training can reliably lead to improved predictions even in settings where performant learners are already available.

Figure 3:

Improvement of the stacked ensemble algorithm that includes all base learners over those which only include a subset (existing learners or AMC learners), in terms of differences of cross-validated MSEs. Including both AMC and existing estimators as base learners always outperformed only including a subset when the dataset contained the same number of features as were used during training. Adding AMC base learners did not tend to improve performance when the dataset had fewer features than were used during meta-training, though any degradation in performance was minimal.

We now discuss performance on datasets with fewer features than were used during meta-training. Figure 2b displays performance on datasets that have fewer features than were used during meta-training. Unsurprisingly, performance was somewhat less desirable than it was for datasets with the same number of features as were used during meta-training. AMC FLAM tended to be somewhat outperformed by FLAM, though did outperform FLAM in one setting. AMC Linear continued to perform similarly to lasso across all settings. Figure 3 shows that stacking all available base learners outperformed stacking only AMC estimators, and performed similarly to stacking existing estimators.

We conclude by discussing the performance of the estimators when we induce varying levels of signal sparsity. Figure 4 shows that AMC FLAM outperformed FLAM for the vast majority of datasets and sparsity patterns. The only exception to this trend occurred for the yacht dataset and the LAozone dataset for denser signals (7, 8, or 9 signal features), where AMC FLAM was slightly outperformed by FLAM. Figure S6 in the appendix shows that AMC Linear consistently outperformed OLS and performed comparably to or slightly better than lasso in most settings.

Figure 4:

Performance of FLAM and AMC FLAM at different sparsity levels. For each training-validation split of the data, between 1 and $q$ features are selected at random from the original dataset (x-axis), where $q$ is the minimum of 10 and the total number of features in the dataset, and Gaussian noise features are then added so that there are 10 total features. Therefore, the signal is expected to become denser and stronger as the x-axis value increases. AMC FLAM outperforms FLAM in most settings.

Figure S7 shows that there was not a major difference between the cross-validated MSE of the three stacking algorithms. Nevertheless, it is worth noting that stacking all available base learners did outperform the other two stacking schemes in 53% of the 83 dataset-sparsity settings considered, with the stacking scheme that only included AMC algorithms performing best in 39% of the settings and the scheme that only included existing algorithms performing best in only 8% of these settings. Thus, we again see evidence that including AMC base learners in a stacked ensemble can improve performance, even when other learners are already available.

7. Proofs

7.1. A Study of Group Actions that are Useful for Our Setting

To prove Theorem 1, it will be convenient to use tools from group theory to describe and study the behavior of our estimation problem under the shifts, rescalings, and permutations that we consider. For $k\in \mathbb{N}$, let $\mathrm{S}\mathrm{y}\mathrm{m}\left(k\right)$ be the symmetric group on $k$ symbols. Let $\mathbb{R}⋊{\mathbb{R}}^{+}$ be the semidirect product of the real numbers with the positive real numbers with the group multiplication

$$\left(a_1,b_1\right)\left(a_2,b_2\right)=\left(a_1+b_1{a}_2,b_1{b}_2\right).$$

Define ${𝒢}_0≔\left(\mathbb{R}⋊{\mathbb{R}}^{+}\right)\times \left[{\left(\mathbb{R}⋊{\mathbb{R}}^{+}\right)}^p⋊\mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)\right]\times \mathrm{S}\mathrm{y}\mathrm{m}\left(n\right)$. Let ${𝒪}_n≔\left\{a\in {\mathbb{R}}^n:\bar{a}=0,s\left(a\right)=1\right\}$. Throughout we equip ${𝒢}_0$ with the product topology.

We note that the quantity $Ƶ$ defined in Section 2.1 writes as

$$𝒵={𝒪}_n^p\times {𝒪}_n\times {\mathbb{R}}^p\times {\mathbb{R}}^p\times \mathbb{R}\times {\mathbb{R}}^p\times \mathbb{R}.$$ (8)

Denote the generic group element $g=\left({\left(g^{j+},g^{j\times }\right)}_{j=0}^p,{\tau }_g,{\eta }_g\right)$ where $\left(g^{j+},g^{j\times }\right)\in \mathbb{R}⋊{\mathbb{R}}^{+},{\tau }_g\in \mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)$, and ${\eta }_g\in \mathrm{S}\mathrm{y}\mathrm{m}\left(n\right)$. Denote the generic element $\mathit{z}\in Ƶ$ by

$$\mathit{z}=\left({\left(z^{x,1,j},\cdots ,z^{x,n,j}\right)}_{j=1}^p,\left(z^{y,1},\cdots ,z^{y,n}\right),{\left(z^{x,0,j}\right)}_{j=1}^p,{\left(z^{\bar{x},j}\right)}_{j=1}^p,z^{\bar{y}},{\left(z^{s\left(x\right),j}\right)}_{j=1}^p,z^{s\left(y\right)}\right).$$

For $g_1=\left({\left(g_1^{j+},g_1^{j\times }\right)}_{j=0}^p,{\tau }_1,{\eta }_1\right),g_2=\left({\left(g_2^{j+},g_2^{j\times }\right)}_{j=0}^p,{\tau }_2,{\eta }_2\right)$, two arbitrary elements in ${𝒢}_0$, define the group multiplication as

$$g_1{g}_2=\left(g_1^{0+}{g}_2^{0\times }+g_1^{0\times }{g}_2^{0+},g_1^{0\times }{g}_2^{0\times },{\left(g_1^{j+}{g}_2^{{\tau }_1^{-1}\left(j\right)\times }+g_1^{j\times }{g}_2^{{\tau }_1^{-1}\left(j\right)+},g_1^{j\times }{g}_2^{{\tau }_1^{-1}\left(j\right)\times }\right)}_{j=1}^p,{\tau }_1{\tau }_2,{\eta }_1{\eta }_2\right).$$

Define the group action ${𝒢}_0\times Ƶ\to Ƶ$ by

$$\begin{array}{ll}(g\cdot \mathit{z}{)}^{x,i,j}& =z^{x,{\eta }_g^{-1}\left(i\right),{\tau }_g^{-1}\left(j\right)}\\ (g\cdot \mathit{z}{)}^{y,i}& =z^{y,{\eta }_g^{-1}\left(i\right)}\\ (g\cdot \mathit{z}{)}^{x,0,j}& =z^{x,0,{\tau }_g^{-1}\left(j\right)}\\ (g\cdot \mathit{z}{)}^{\bar{x},j}& =\frac{g^{j+}}{g^{j\times }}+z^{\bar{x},{\tau }_g^{-1}\left(j\right)}\\ (g\cdot \mathit{z}{)}^{\bar{y}}& =\frac{g^{0+}}{g^{0\times }}+z^{\bar{y}}\\ (g\cdot \mathit{z}{)}^{s\left(x\right),j}& =\mathrm{l}\mathrm{o}\mathrm{g}{g}^{j\times }+z^{s\left(x\right),{\tau }_g^{-1}\left(j\right)}\\ (g\cdot \mathit{z}{)}^{s\left(y\right)}& =\mathrm{l}\mathrm{o}\mathrm{g}{g}^{0\times }+z^{s\left(y\right)},\end{array}$$

where $i\in \{\mathrm{1,2},\cdots ,n\}$ and $j\in \{\mathrm{1,2},\cdots ,p\}$.

We make use of the below result without statement in the remainder of this section.

Lemma 1. The map defined above is a left group action.

Proof. The identity axiom, namely that $e\cdot \mathit{z}=\mathit{z}$ when $e$ is the identity element of ${𝒢}_0$, is straightforward to verify and so we omit the arguments. Fix $g_1,g_2\in {𝒢}_0$ and $\mathit{z}\in Ƶ$. We establish compatibility by showing that $g_1{g}_2\cdot \mathit{z}=g_1\cdot \left(g_2\cdot \mathit{z}\right)$. To see that this is indeed the case, note that, for all $i\in \{1,\dots ,n\}$ and $j\in \{1,\dots ,p\}$:

$${\left(g_1{g}_2\cdot \mathit{z}\right)}^{y,i}=z^{y,{\left({\eta }_1{\eta }_2\right)}^{-1}\left(i\right)}=z^{y,{\eta }_2^{-1}{\eta }_1^{-1}\left(i\right)}={\left(g_2\cdot \mathit{z}\right)}^{y,{\eta }_1^{-1}\left(i\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{y,i}$$

$$\begin{array}{ll}{\left(g_1{g}_2\cdot \mathit{z}\right)}^{x,i,j}& =z^{x,{\eta }_2^{-1}{\eta }_1^{-1}\left(i\right),{\tau }_2^{-1}{\tau }_1^{-1}\left(j\right)}={\left(g_2\cdot \mathit{z}\right)}^{x,{\eta }_1^{-1}\left(i\right),{\tau }_1^{-1}\left(j\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{x,i,j}\\ {\left(g_1{g}_2\cdot \mathit{z}\right)}^{x,0,j}& =z^{x,0,{\tau }_2^{-1}{\tau }_1^{-1}\left(j\right)}={\left(g_2\cdot \mathit{z}\right)}^{x,0,{\tau }_1^{-1}\left(j\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{x,0,j}\\ {\left(g_1{g}_2\cdot \mathit{z}\right)}^{\bar{x},j}& =\frac{g_1^{j+}{g}_2^{{\tau }_1^{-1}\left(j\right)\times }+g_1^{j\times }{g}_2^{{\tau }_1^{-1}\left(j\right)+}}{g_1^{j\times }{g}_2^{{\tau }_1^{-1}\left(j\right)\times }}+z^{\bar{x},{\tau }_2^{-1}{\tau }_1^{-1}\left(j\right)}=\frac{g_1^{j+}}{g_1^{j\times }}+{\left(g_2\cdot z\right)}^{\bar{x},{\tau }_1^{-1}\left(j\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{\bar{x},j}\\ {\left(g_1{g}_2\cdot \mathit{z}\right)}^{\bar{y}}& =\frac{g_1^{0+}{g}_2^{0\times }+g_1^{0\times }{g}_2^{0+}}{g_1^{0\times }{g}_2^{0\times }}+z^{\bar{y}}=\frac{g_1^{0+}}{g_1^{0\times }}+{\left(g_2\cdot \mathit{z}\right)}^{\bar{y}}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{\bar{y}}\\ {\left(g_1{g}_2\cdot \mathit{z}\right)}^{s\left(x\right),j}& =\mathrm{l}\mathrm{o}\mathrm{g}\left(g_1^{j\times }{g}_2^{j\times }\right)+z^{s\left(x\right),{\tau }_2^{-1}{\tau }_1^{-1}\left(j\right)}=\mathrm{l}\mathrm{o}\mathrm{g}{g}_1^{j\times }+{\left(g_2\cdot \mathit{z}\right)}^{s\left(x\right),{\tau }_1^{-1}\left(j\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{s\left(x\right),j}\\ {\left(g_1{g}_2\cdot \mathit{z}\right)}^{s\left(y\right)}& =\mathrm{l}\mathrm{o}\mathrm{g}\left(g_1^{0\times }{g}_2^{0\times }\right)+z^{\left(s\right(y)}=\mathrm{l}\mathrm{o}\mathrm{g}{g}_1^{0\times }+{\left(g_2\cdot \mathit{z}\right)}^{s\left(y\right)}={\left(g_1\cdot \left(g_2\cdot \mathit{z}\right)\right)}^{s\left(y\right)}.\end{array}$$

□

We now introduce several group actions that we will make heavy use of in our proof of Theorem 1 and in the lemmas that precede it. We first define ${𝒢}_0\times 𝒮\to 𝒮$. For $S\in 𝒮$ and $g\in {𝒢}_0$, define $g\cdot S$ to be $(g\cdot S)\left(\mathit{z}\right)=S(g\cdot \mathit{z})$. Conditions T4 and T5 can be restated as $g\cdot S\in 𝒮$ for all $g\in {𝒢}_0$ and $S\in 𝒮$. It can then readily be shown that, under these conditions, the defined map is a left group action. For $T\in 𝒯$, we will write $g\cdot T$ to denote the $𝒟\to (𝒳\to \mathbb{R})$ operator defined so that

$$(g\cdot T)\left(\mathit{d}\right):x_0\mapsto \left\{\begin{array}{ll}\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)\left(g\cdot S_T\right)\left(z\left(\mathit{d},x_0\right)\right),& \text{if}\left(\mathit{d},x_0\right)\in {𝒟}_0,\\ 0,& \text{otherwise}\end{array}\right.$$

It is possible that $g\cdot T$ does not belong to $𝒯$ due to its behavior when $\left(\mathit{d},x_0\right)\notin {𝒟}_0$, and therefore that the defined map is not a group action. Nonetheless, because ${𝒟}_0$ has $P$-probability one for any $P\in 𝒫$, this fact will not pose any difficulties in our arguments.

We now define the group action ${𝒢}_0\times (𝒴\times 𝒳)\to (𝒴\times 𝒳)$. For $(y,x)\in \mathbb{R}\times {\mathbb{R}}^p$, define $g\cdot (y,x)$ as

$$g\cdot (y,x)=\left(g^{0+}+g^{0\times }y,{\left(g^{i+}+g^{i\times }{x}^{{\tau }_g^{-1}\left(i\right)}\right)}_{i=1}^p\right).$$

Similar arguments to those used to prove Lemma 1 show that the map defined above is a left group action. We now define the group action ${𝒢}_0\times 𝒫\to 𝒫$. For $P\in 𝒫,g\in {𝒢}_0$, define $g\cdot P=P\circ g^{-1}$ by $(g\cdot P)\left(U\right)=P\left(g^{-1}\left(U\right)\right)$, where

$$g^{-1}\left(U\right)=\left\{(y,x)\in {\mathbb{R}}^{p+1}:g\cdot (y,x)\in U\right\}.$$

Under P1, P2, and P3, which, as noted in the Section 2.1, implicitly encode that $P\circ g^{-1}\in 𝒫$, it can readily be shown that the defined map is a left group action. Finally, we define the group action ${𝒢}_0\times \Gamma \to \Gamma$. For $\Pi \in \Gamma ,g\in {𝒢}_0$, define $g\cdot \Pi =\Pi \circ g^{-1}$ by $(g\cdot \Pi )\left(U\right)=\Pi \left(g^{-1}\left(U\right)\right)$ where

$$g^{-1}\left(U\right)=\{P\in 𝒫:g\cdot P\in U\}.$$

We can restate P1, P2, and P3 as $\Pi \circ g^{-1}\in \Gamma$ for all $\Pi \in \Gamma ,g\in {𝒢}_0$. Under these conditions, it can be shown that the defined map is a left group action.

We now show that ${𝒢}_0$ is amenable — see Appendix A for a review of this concept. Establishing this fact will allow us to apply Day’s fixed point theorem (Theorem S1 in Appendix A) in the upcoming proof of Theorem 1.

Lemma 2. ${𝒢}_0$ is amenable.

Proof. Because $\mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)$ and $\mathrm{S}\mathrm{y}\mathrm{m}\left(n\right)$ are finite groups, they are compact, and therefore amenable. Because $\mathbb{R}$ and ${\mathbb{R}}^{+}$are Abelian, they are also amenable. By Theorem S19, group extensions of amenable groups are amenable. □

7.2. Proofs of Theorems 1 through 4

This section is organized as follows. Section 7.2.1 introduces three general lemmas that will be useful in proving the results from the main text. Section 7.2.2 proves several lemmas, proves the variant of the Hunt-Stein theorem from the main text (Theorem 1), and concludes with a discussion of the relation of this result to those in Le Cam (2012). Section 7.2.3 establishes a preliminary lemma and then proves that, when the class of estimators is equivariant, it suffices to restrict attention to priors in ${\Gamma }_1$ when aiming to learn a $\Gamma$-minimax estimator (Theorem 2). Section 7.2.4 establishes several lemmas, including a minimax theorem for our setting, before proving the existence of an equilibrium point (Theorem 3). Section 7.2.5 establishes the equivariance of our proposed neural network architecture (Theorem 4).

In this section, we always equip $C(Ƶ,\mathbb{R})$ with the topology of compact convergence and, whenever T2 holds so that $𝒮\subset C(Ƶ,\mathbb{R})$, we equip $𝒮$ with the subspace topology. For a fixed compact $𝒦\subset Ƶ$ and a function $h\in C(Ƶ,\mathbb{R})$, we also let $\parallel h{\parallel }_{\mathrm{\infty },𝒦}≔{\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{z}\in 𝒦} \left|h\right(\mathit{z}\left)\right|$.

7.2.1. Preliminary lemmas

We now prove three lemmas that will be used in our proofs of Theorems 1 and 3.

Lemma 3. $C(\mathbb{Ƶ},\mathbb{R})$ with the compact-open topology is metrizable.

Proof. See Example IV.2.2 in Conway (2010). □

As a consequence of the above, we can show that a subset of $C(Ƶ,\mathbb{R})$ is closed by showing that it is sequentially closed, and we can show that a subset of $C(Ƶ,\mathbb{R})$ is continuous by showing that it is sequentially continuous.

Lemma 4. If T1, T2, and T3 hold, then $𝒮$ is a compact subset of $C(\mathbb{Ƶ},\mathbb{R})$.

Proof. By T1, $𝒮$ is pointwise bounded. Moreover, the local Hölder condition T2 implies that $𝒮$ is equicontinuous, in the sense that, for every $ϵ>0$ and every $z\in Ƶ$ there exists an open neighborhood $𝒰\subset Ƶ$ of $\mathit{z}$ such that, for all $S\in 𝒮$ and all ${\mathit{z}}^{\prime }\in 𝒰$, it holds that $\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|<ϵ$. Hence, by the Arzelà-Ascoli theorem (see Theorem 47.1 in Munkres, 2000 for a convenient version),$𝒮$ is a relatively compact subset of $C(Ƶ,\mathbb{R})$. By T3, $𝒮$ is closed, and therefore $𝒮$ is compact. □

We now show that the group action ${𝒢}_0\times 𝒮\to 𝒮$ is continuous under conditions that we assume in Theorem 1. Establishing this continuity condition is necessary for our use of Day’s fixed point theorem in the upcoming proof of that result.

Lemma 5. If T2, T4, and T5 hold, then the group action ${𝒢}_0\times 𝒮\to 𝒮$ is continuous.

Proof. By T4 and T5, ${𝒢}_0\times 𝒮\to 𝒮$ is indeed a group action. Also, by T2 and Lemma 3,$𝒮$ is metrizable. Recall the expression for $Ƶ$ given in (8) and that

$${𝒢}_0≔\left(\mathbb{R}⋊{\mathbb{R}}^{+}\right)\times \left[{\left(\mathbb{R}⋊{\mathbb{R}}^{+}\right)}^p⋊\mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)\right]\times \mathrm{S}\mathrm{y}\mathrm{m}\left(n\right).$$

The product topology is compatible with semidirect products, and so the fact that each multiplicand is a metric space implies that ${𝒢}_0$ is a metric space. Hence, it suffices to show sequential continuity. Let ${\left\{\left(g_k,S_k\right)\right\}}_{k=1}^{\mathrm{\infty }}$ be a sequence in ${𝒢}_0\times 𝒮$ such that $\left(g_k,S_k\right)\to (g,S)$, where $(g,S)\in {𝒢}_0\times 𝒮$. By the definition of the product metric, $g_k\to g$ and $S_k\to S$. Let ${𝒦}_1\subseteq {𝒪}_n^p,{𝒦}_2\subseteq {𝒪}_n,{𝒦}_3\subset {\mathbb{R}}^p,{𝒦}_4\subset {\mathbb{R}}^p,{𝒦}_5\subset \mathbb{R},{𝒦}_6\subset {\mathbb{R}}^p$, and ${𝒦}_7\subset \mathbb{R}$ be compact spaces. Since each compact space $𝒦\subset 𝒵$ is contained in such a ${\prod }_{i=1}^7 {𝒦}_i$, it suffices to show that

$$\underset{\mathit{z}\in \prod _{i=1}^7 {𝒦}_i}{\mathrm{s}\mathrm{u}\mathrm{p}} \left|\left(g_k\cdot S_k\right)\left(\mathit{z}\right)-(g\cdot S)\left(\mathit{z}\right)\right|={∥g_k\cdot S_k-g\cdot S∥}_{\mathrm{\infty },\prod _{i=1}^7 {𝒦}_i}\to 0$$

for arbitrary compact sets ${𝒦}_1,\cdots ,{𝒦}_7$. To show this, we will use the decomposition $g_k=\left(g_{k,1},g_{k,2},g_{k,3},g_{k,4}\right)$, where $g_{k,1}\in \mathbb{R}⋊{\mathbb{R}}^{+},g_{k,2}\in {\left(\mathbb{R}⋊{\mathbb{R}}^{+}\right)}^p,g_{k,3}\in \mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)$, and $g_{k,4}\in \mathrm{S}\mathrm{y}\mathrm{m}\left(n\right)$. We similarly use the decomposition $g=\left(g_1,g_2,g_3,g_4\right)$. For all $N$ large enough, all of the statements are true for all $k>N:g_{k,3}=g_3,g_{k,4}=g_4,g_{k,1}$ is contained in a compact neighbourhood $C_1$ of $g_1$, and $g_{k,2}$ is contained in a compact neighbourhood $C_2$ of $g_2$.

Since permutations are continuous, $g_4{𝒦}_1{g}_3≔\left\{g_{4}w{g}_3:w\in {𝒦}_1\right\},g_4{𝒦}_2≔\left\{g_{4}w:w\in {𝒦}_2\right\}$, and ${𝒦}_j{g}_3≔\left\{w{g}_3:w\in {𝒦}_j\right\},j=\mathrm{3,4},6$, are compact. In the following we use the decomposition $g^{\prime }≔\left(g_1^{\prime },g_2^{\prime },g_3^{\prime },g_4^{\prime }\right)$ for an arbitrary element $g^{\prime }\in 𝒢$. Since addition and multiplication are continuous, $C_2\odot \left({𝒦}_3{g}_3\right)≔\left\{g_2^{\prime }\cdot w:g_2^{\prime }\in C_2,w\in {𝒦}_3{g}_3\right\},C_2\odot \left({𝒦}_4{g}_3\right)≔\left\{g_2^{\prime }\cdot w:g_2^{\prime }\in C_2,w\in \right.\left.{𝒦}_4{g}_3\right\},C_1\odot {𝒦}_5≔\left\{g_1^{\prime }\cdot w:g_1^{\prime }\in C_1,w\in {𝒦}_5\right\},C_2\odot \left({𝒦}_6{g}_3\right)≔\left\{g_2^{\prime }\cdot w:g_2^{\prime }\in C_2,w\in {𝒦}_6{g}_3\right\}$, and $C_1\odot {𝒦}_7≔\left\{g_1^{\prime }\cdot w:g_1^{\prime }\in C_1,w\in {𝒦}_7\right\}$ are compact. Define ${𝒦}^{\circ }$ to be the compact set

$${𝒦}^{\circ }=g_4{𝒦}_1{g}_3\times g_4{𝒦}_2\times C_2\odot \left({𝒦}_3{g}_3\right)\times C_2\odot \left({𝒦}_4{g}_3\right)\times C_1\odot {𝒦}_5\times C_2\odot \left({𝒦}_6{g}_3\right)\times C_1\odot {𝒦}_7$$

Then,

$${∥g_k\cdot S_k-g\cdot S∥}_{\mathrm{\infty },\prod _{i=1}^7 {𝒦}_i}\le {∥S_k-S∥}_{\mathrm{\infty },{𝒦}^{\circ }}\to 0.$$

7.2.2. Proof of Theorem 1

We begin this subsection with four lemmas and then we prove Theorem 1. Following this proof, we briefly describe how the argument relates to that given in Le Cam (2012). In the proof of Theorem 1, we will use notation that we established about the group ${𝒢}_0$ in Section 7.1. We refer the reader to that section for details.

Lemma 6. For any $g\in {𝒢}_0,T\in 𝒯$, and $P\in 𝒫,R(g\cdot T,P)=R(T,g\cdot P)$

Proof. Fix $T\in 𝒯$ and $P\in 𝒫$, and let $S≔S_T$, where $S_T$ is defined in (3). By the change-of-variables formula,

$$R(g\cdot T,P)={\mathrm{E}}_P\left[\int {\sigma }_P^{-2}{\left\{\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S(g\cdot \mathit{Z})-{\mu }_P\left(x_0\right)\right\}}^{2}d{P}_X\left(x_0\right)\right]={\mathrm{E}}_{P\circ g^{-1}}\left[\int {\sigma }_P^{-2}{\left\{g^{-1}\cdot \stackrel{-}{\mathit{Y}}+s\left(g^{-1}\cdot \mathit{Y}\right)S\left(\mathit{Z}\right)-{\mu }_P\left(g^{-1}\cdot x_0\right)\right\}}^{2}d\left(P_X\circ g^{-1}\right)\left(x_0\right)\right].$$

Plugging the fact that $g^{-1}\cdot \mathit{y}=\left(\mathit{y}-g^{0+}\right)/g^{0\times }$ and that

$${\mu }_P\left(g^{-1}\cdot x_0\right)={\text{E}}_P\left[Y\mid X_0=g^{-1}\cdot x_0\right]={\text{E}}_P\left[Y\mid g\cdot X_0=x_0\right]=\frac{{\text{E}}_P\left[g\cdot Y\mid g\cdot X_0=x_0\right]-g^{0+}}{g^{0\times }}=\frac{{\mu }_{P\circ g^{-1}}\left(x_0\right)-g^{0+}}{g^{0\times }}$$

into the right-hand side of the preceding display yields that

$$\begin{array}{ll}& R(g\cdot T,P)\\ & ={\mathrm{E}}_{P\circ g^{-1}}\left[\int {\sigma }_P^{-2}{\left\{\frac{\stackrel{-}{\mathit{Y}}-g^{0+}}{g^{0\times }}+s\left(\frac{\stackrel{-}{\mathit{Y}}-g^{0+}}{g^{0\times }}\right)S\left(\mathit{Z}\right)-\frac{{\mu }_{P\circ g^{-1}}\left(x_0\right)-g^{0+}}{g^{0\times }}\right\}}^{2}d\left(P_X\circ g^{-1}\right)\left(x_0\right)\right]\\ & ={\mathrm{E}}_{P\circ g^{-1}}\left[\int {\sigma }_P^{-2}{\left\{\frac{\stackrel{-}{\mathit{Y}}}{g^{0\times }}+s\left(\frac{\stackrel{-}{\mathit{Y}}-g^{0+}}{g^{0\times }}\right)S\left(\mathit{Z}\right)-\frac{{\mu }_{P\circ g^{-1}}\left(x_0\right)}{g^{0\times }}\right\}}^{2}d\left(P_X\circ g^{-1}\right)\left(x_0\right)\right].\end{array}$$

By the shift and scale properties of the standard deviation and variance, the above continues as

$$\begin{array}{ll}& ={\mathrm{E}}_{P\circ g^{-1}}\left[\int {\sigma }_P^{-2}{\left\{\frac{\stackrel{-}{\mathit{Y}}}{g^{0\times }}+\frac{s\left(\stackrel{-}{\mathit{Y}}\right)}{g^{0\times }}S\left(\mathit{Z}\right)-\frac{{\mu }_{P\circ g^{-1}}\left(x_0\right)}{g^{0\times }}\right\}}^{2}d\left(P_X\circ g^{-1}\right)\left(x_0\right)\right]\\ & ={\mathrm{E}}_{P\circ g^{-1}}\left[\int {\sigma }_{P\circ g^{-1}}^{-2}{\left\{\stackrel{-}{\mathit{Y}}+s\left(\stackrel{-}{\mathit{Y}}\right)S\left(\mathit{Z}\right)-{\mu }_{P\circ g^{-1}}\left(x_0\right)\right\}}^{2}d\left(P_X\circ g^{-1}\right)\left(x_0\right)\right]\\ & =R(T,g\cdot P).\end{array}$$

□

Lemma 7. For any $g\in {𝒢}_0,T\in 𝒯$, and $\Pi \in \Gamma$, it holds that $r(g\cdot T,\Pi )=r(T,g\cdot \Pi )$.

Proof. This result follows quickly from Lemma 6. Indeed, for any $g\in {𝒢}_0,T\in 𝒯$, and $\Pi \in \Gamma$,

$$\begin{gathered} r(g\cdot T,\Pi )=\int R(g\cdot T,P)d\Pi \left(P\right)=\int R(T,g\cdot P)d\Pi \left(P\right) \\ =\int R(T,P)d\left(\Pi \circ g^{-1}\right)\left(P\right)=r(T,g\cdot \Pi ). \end{gathered}$$

Let ${𝒮}_e≔\left\{S\in 𝒮:g\cdot S=S\right.$ for all $\left.g\in {𝒢}_0\right\}$ consists of the ${𝒢}_0$-invariant elements of $𝒮$. The following fact will be useful when proving Theorem 1, and also when proving results in the upcoming Section 7.2.3.

Lemma 8. It holds that ${𝒮}_e=\left\{S_T:T\in {𝒯}_e\right\}$.

Proof. Fix $S\in {𝒮}_e$ and $g\in {𝒢}_0$. By the definition of $𝒮≔\left\{S_T:T\in 𝒯\right\}$, there exists a $T\in 𝒯$ such that $S=S_T$. For this $T$, the fact that $S_T\left(\mathit{z}\right)=S_T(g\cdot \mathit{z})$ implies that

$$\begin{gathered} T(g\cdot \mathit{z})=\left(g^{0+}+g^{0\times }\stackrel{-}{\mathit{y}}\right)+g^{0\times }s\left(\mathit{y}\right)S_T(g\cdot \mathit{z})=\left(g^{0+}+g^{0\times }\stackrel{-}{\mathit{y}}\right)+g^{0\times }s\left(\mathit{y}\right)S_T\left(\mathit{z}\right) \\ =g^{0+}+g^{0\times }\left[\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)S_T\left(\mathit{z}\right)\right]=g^{0+}+g^{0\times }T\left(\mathit{z}\right). \end{gathered}$$

As $g$ was arbitrary, $T\in {𝒯}_e$. Hence, ${𝒮}_e\subseteq \left\{S_T:T\in {𝒯}_e\right\}$. □

Now fix $T\in {𝒯}_e$ and $g\in {𝒢}_0$. Note that $S_T\left(\mathit{z}\right)=\left[T\right(\mathit{z})-\stackrel{-}{\mathit{y}}]/s\left(\mathit{y}\right)$. Using that $T\in {𝒯}_e$ implies that $T(g\cdot \mathit{z})=g^{0+}+g^{0\times }T\left(\mathit{z}\right)$, we see that

$$\begin{gathered} S_T(g\cdot \mathit{z})=\frac{T(g\cdot \mathit{z})-g^{0+}-g^{0\times }\stackrel{-}{\mathit{y}}}{s(g\cdot \mathit{y})}=\frac{T(g\cdot \mathit{z})-g^{0+}-g^{0\times }\stackrel{-}{\mathit{y}}}{g^{0\times }s\left(\mathit{y}\right)} \\ =\frac{g^{0+}+g^{0\times }T\left(\mathit{z}\right)-g^{0+}-g^{0\times }\stackrel{-}{\mathit{y}}}{g^{0\times }s\left(\mathit{y}\right)}=\frac{T\left(\mathit{z}\right)-\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)}=S_T\left(\mathit{z}\right). \end{gathered}$$

As, $g$ was arbitrary, $S_T\in {𝒮}_e$, and so ${𝒮}_e\supseteq \left\{S_T:T\in {𝒯}_e\right\}$. □

We define $r_0:𝒮\times \Gamma \to [0,\mathrm{\infty })$ as follows:

$$r_0(S,\Pi )≔\int {\mathrm{E}}_P\left[{\int }_{x_0:\left(\mathit{D},x_0\right)\in {𝒟}_0} \frac{{\left\{\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S\left(z\left(\mathit{D},x_0\right)\right)-{\mu }_P\left(x_0\right)\right\}}^2}{{\sigma }_P^2}d{P}_X\left(x_0\right)\right]d\Pi \left(P\right).$$ (9)

Because ${𝒟}_0$ occurs with $P$-probability one (for any $P\in 𝒫$ ), it holds that $r(T,\Pi )=r_0\left(S_T,\Pi \right)$ for any $T\in 𝒯$.

Lemma 9. Fix $\Pi \in \Gamma$. If T1, T2, and P4 hold, then $r_0(\cdot ,\Pi ):𝒮\to \mathbb{R}$ is lower semicontinuous.

Proof. Fix $\Pi \in \Gamma$. For any compact $𝒦\subset Ƶ$, we define $f_{𝒦}:𝒮\to \mathbb{R}$ by

$$f_{𝒦}\left(S\right)≔\int {\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} {\sigma }_P^{-2}{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^{2}d{P}_X\left(x_0\right)\right]d\Pi \left(P\right),$$

where here and throughout in this proof we let $\mathit{Z}≔z\left(\mathit{D},x_0\right)$ and ${𝒳}_{\mathit{D},𝒦}≔\left\{x_0:\left(\mathit{D},x_0\right)\in \right.\left.{𝒟}_0,z\left(\mathit{D},x_0\right)\in 𝒦\right\}\subseteq 𝒳$. Recalling that there exists an increasing sequence of compact subsets ${𝒦}_1\subset {𝒦}_2\subset \cdots$ such that ${\bigcup }_{j=1}^{\mathrm{\infty }} {𝒦}_j=𝒵$, we see that ${\mathrm{s}\mathrm{u}\mathrm{p}}_{j\in \mathbb{N}} f_{{𝒦}_j}(\cdot )=r_0(\cdot ,\Pi )$ by the monotone convergence theorem. Moreover, as suprema of collections of continuous functions are lower semicontinuous, we see that $f$ is lower semicontinuous if $f_{𝒦}$ is continuous for every $𝒦$. In the remainder of this proof, we will show that this is indeed the case.

By Lemma 3, it suffices to show that $f_{𝒦}$ is sequentially continuous. Fix $S_1,S_2\in 𝒮$. By Jensen’s inequality,

$$\begin{array}{ll}& \left|f_{𝒦}\left(S_1\right)-f_{𝒦}\left(S_2\right)\right|\\ & =\mid \int {\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} {\sigma }_P^{-2}\left({\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_1\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\begin{array}{c}\left.\left.-{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_2\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\right)d{P}_X\left(x_0\right)\right]d\Pi \left(P\right)\mid \\ \le \int {\sigma }_P^{-2}{\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} \mid {\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_1\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\right.\\ \left.-{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_2\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\mid d{P}_X\left(x_0\right)\right]d\Pi \left(P\right).\end{array}\right.\right.\end{array}$$ (10)

In what follows, we will bound the right-hand side above by some finite constant times ${∥S_1-S_2∥}_{𝒦,\mathrm{\infty }}$. We start by noting that, for any $\left(\mathit{d},x_0\right)\in {𝒟}_0$ such that $z\left(\mathit{d},x_0\right)\in 𝒦$,

$$\begin{array}{ll}& \left|{\left[\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)S_1\left(\mathit{z}\right)-{\mu }_P\left(x_0\right)\right]}^2-{\left[\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)S_2\left(\mathit{z}\right)-{\mu }_P\left(x_0\right)\right]}^2\right|\\ & =\left|s\left(\mathit{y}\right)\left[2\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)\left\{S_1\left(\mathit{z}\right)+S_2\left(\mathit{z}\right)\right\}-2{\mu }_P\left(x_0\right)\right]\left[S_1\left(\mathit{z}\right)-S_2\left(\mathit{z}\right)\right]\right|\\ & \le {∥S_1-S_2∥}_{\mathrm{\infty },𝒦}s\left(\mathit{y}\right)\left|2\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)\left\{S_1\left(\mathit{z}\right)+S_2\left(\mathit{z}\right)\right\}-2{\mu }_P\left(x_0\right)\right|\\ & \le {∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\left(s(\mathit{y}{)}^2\left[{∥S_1∥}_{𝒦,\mathrm{\infty }}+{∥S_2∥}_{𝒦,\mathrm{\infty }}\right]+2s(\mathit{y})\left|\stackrel{-}{\mathit{y}}-{\mu }_P\left(x_0\right)\right|\right)\\ & \le {∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\left(s(\mathit{y}{)}^2\left[{∥S_1∥}_{𝒦,\mathrm{\infty }}+{∥S_2∥}_{𝒦,\mathrm{\infty }}\right]+2s(\mathit{y})\left|\stackrel{-}{\mathit{y}}-{\mathrm{E}}_P\left[Y\right]\right|+2s(\mathit{y})\left|{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right|\right)\\ & \le 2{∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\left(C_{1}s(\mathit{y}{)}^2+s(\mathit{y})\left|\stackrel{-}{\mathit{y}}-{\mathrm{E}}_P\left[Y\right]\right|+s(\mathit{y})\left|{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right|\right),\end{array}$$

where $C_1≔{\mathrm{s}\mathrm{u}\mathrm{p}}_{S\in 𝒮} \parallel S{\parallel }_{𝒦,\mathrm{\infty }}$ is finite by T1 and T2. Integrating both sides shows that

$$\begin{array}{c}{\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} \left|{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_1\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2-{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_2\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\right|d{P}_X\left(x_0\right)\right]\\ \le 2{∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\left(C_1{\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} s(\mathit{Y}{)}^{2}d{P}_X\left(x_0\right)\right]+{\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} s\left(\mathit{Y}\right)\left|\stackrel{-}{\mathit{Y}}-{\mathrm{E}}_P\left[Y\right]\right|d{P}_X\left(x_0\right)\right]\right.\\ \left.+{\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} s\left(\mathit{Y}\right)\left|{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right|d{P}_X\left(x_0\right)\right]\right)\\ \le 2{∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\left(C_1{\mathrm{E}}_P\left[s(\mathit{Y}{)}^2\right]+{\mathrm{E}}_P\left[s\left(\mathit{Y}\right)\left|\stackrel{-}{\mathit{Y}}-{\mathrm{E}}_P\left[Y\right]\right|\right]\right.\\ \left.+{\mathrm{E}}_P\left[s\left(\mathit{Y}\right)\int \left|{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right|d{P}_X\left(x_0\right)\right]\right).\end{array}$$ (11)

We now bound the three expectations on the right-hand side by finite constants that do not depend on $S_1$ or $S_2$. All three bounds make use of the bound on the first expectation, namely ${\mathrm{E}}_P\left[s(\mathit{Y}{)}^2\right]=\frac{n-1}{n}{\mathrm{V}\mathrm{a}\mathrm{r}}_P\left(Y\right)\le \frac{n-1}{n}{C}_2{\sigma }_P^2$, where $C_2≔{\mathrm{s}\mathrm{u}\mathrm{p}}_{P\in 𝒫} {\mathrm{V}\mathrm{a}\mathrm{r}}_P\left(Y\right)/{\sigma }_P^2$. We note that (P4) can be used to show that $C_2<\mathrm{\infty }$. Indeed,

$${\mathrm{E}}_P\left[{\mathrm{V}\mathrm{a}\mathrm{r}}_P(Y\mid X)\right]={\mathrm{E}}_P\left[{\mathrm{V}\mathrm{a}\mathrm{r}}_P\left({ϵ}_P\mid X\right)\right]={\mathrm{E}}_P\left[{ϵ}_P^2\right]={\sigma }_P^2,$$

and so, by the law of total variance and (P4), $C_2=1+{\mathrm{s}\mathrm{u}\mathrm{p}}_{P\in 𝒫} {\mathrm{V}\mathrm{a}\mathrm{r}}_P\left({\mu }_P\left(X\right)\right)/{\sigma }_P^2<\mathrm{\infty }$. By Cauchy-Schwarz, the second expectation on the right-hand side of (11) bound as

$${\mathrm{E}}_P\left[s\left(\mathit{Y}\right)\left|\mathit{Y}-{\mathrm{E}}_P\left[Y\right]\right|\right]\le {\mathrm{E}}_P{\left[s(\mathit{Y}{)}^2\right]}^{1/2}{\mathrm{E}}_P{\left[{\left\{\mathit{Y}-{\mathrm{E}}_P\left[Y\right]\right\}}^2\right]}^{1/2}={\mathrm{E}}_P{\left[s(\mathit{Y}{)}^2\right]}^{1/2}{\sigma }_P=\sqrt{\frac{n-1}{n}}\sqrt{C_2}{\sigma }_P^2,$$

and the third expectation bounds as

$$\begin{gathered} {\mathrm{E}}_P\left[s\left(\mathit{Y}\right)\left|{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right|\right] \\ \le {\mathrm{E}}_P{\left[s(\mathit{Y}{)}^2\right]}^{1/2}{\mathrm{E}}_P{\left[\int {\left\{{\mu }_P\left(x_0\right)-{\mathrm{E}}_P\left[Y\right]\right\}}^{2}d{P}_{X_0}\right]}^{1/2} \\ \le {\mathrm{E}}_P{\left[s(\mathit{Y}{)}^2\right]}^{1/2}{\mathrm{V}\mathrm{a}\mathrm{r}}_P(Y{)}^{1/2}\le \sqrt{\frac{n-1}{n}}\sqrt{C_2}{\sigma }_P{\mathrm{V}\mathrm{a}\mathrm{r}}_P(Y{)}^{1/2} \\ \le \sqrt{\frac{n-1}{n}}{C}_2{\sigma }_P^2. \end{gathered}$$

Plugging these bounds into (11), we see that

$$\begin{array}{ll}& {\mathrm{E}}_P\left[{\int }_{{𝒳}_{\mathit{D},𝒦}} \left|{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_1\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2-{\left[\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S_2\left(\mathit{Z}\right)-{\mu }_P\left(x_0\right)\right]}^2\right|d{P}_X\left(x_0\right)\right]\\ & \le 2{∥S_1-S_2∥}_{\mathrm{\infty },𝒦}{\sigma }_P^2\sqrt{\frac{n-1}{n}}{C}_2^{1/2}\left(C_1{C}_2^{1/2}\sqrt{\frac{n-1}{n}}+C_2^{1/2}+1\right).\end{array}.$$

Plugging this into (10), we have shown that

$$\left|f_{𝒦}\left(S_1\right)-f_{𝒦}\left(S_2\right)\right|\le 2{∥S_1-S_2∥}_{\mathrm{\infty },𝒦}\sqrt{\frac{n-1}{n}}{C}_2^{1/2}\left(C_1{C}_2^{1/2}\sqrt{\frac{n-1}{n}}+C_2^{1/2}+1\right).$$

We now conclude the proof by showing that the above implies that $f_{𝒦}$ is sequentially continuous at every $S\in 𝒮$, and therefore is sequentially continuous on $𝒮$. Fix $S$ and a sequence $\left\{S_j\right\}$ such that $S_j\to S$ compactly. This implies that ${∥S_j-S∥}_{\mathrm{\infty },𝒦}\to 0$, and so the above display implies that $f_{𝒦}\left(S_j\right)\to f_{𝒦}\left(S\right)$, as desired. □

We now prove Theorem 1.

Proof of Theorem 1. Fix $T_0\in 𝒯$ and let $S_0≔S_{T_0}\in 𝒮$. Let $𝒦$ be the set of all elements $S\in 𝒮$ that satisfy

$$\underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r_0(S,\Pi )\le \underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r_0\left(S_0,\Pi \right).$$

For fixed ${\Pi }_0\in \Gamma$, the set of $S\in 𝒮$ that satisfy $r_0\left(S,{\Pi }_0\right)\le {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r_0\left(S_0,\Pi \right)$ is closed due to the lower semicontinuity of the risk function (Lemma 9) and contains $S_0$. The intersection of such sets is closed and contains $S_0$ so that $𝒦$ is a nonempty closed subset of the compact Hausdorff set $𝒮$, implying that $𝒦$ is compact. By the convexity of $x\mapsto {\left(\frac{x-a}{b}\right)}^2$, the risk function $S\mapsto r_0(S,\Pi )$ is convex. Hence, $𝒦$ is convex. If $S\in 𝒦$, then Lemma 7 shows that, for any $g\in {𝒢}_0$,

$$r_0\left(g\cdot S,{\Pi }_0\right)=r_0\left(S,g\cdot {\Pi }_0\right)\le \underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r_0\left(S_0,\Pi \right).$$

Thus, $g\cdot S\in 𝒦$ and ${𝒢}_0\times 𝒦\to 𝒦$ is an affine group action on a nonempty, convex, compact subset of a locally compact topological vector space. Combining this with the fact that ${𝒢}_0$ is amenable (Lemma 2) shows that we may apply Day’s fixed point theorem (Theorem S1) to see that there exists an $S_e\in 𝒮$ such that, for all $g\in {𝒢}_0,g\cdot S_e=S_e$ and

$$\underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r_0\left(S_e,\Pi \right)\le \underset{\Pi \in \Gamma }{\mathrm{s}\mathrm{u}\mathrm{p}} r_0\left(S_0,\Pi \right).$$

The conclusion is at hand. By Lemma 8, there exists a $T_e\in {𝒯}_e$ such that $S_e=S_{T_e}$. Furthermore, as noted below (9),$r_0\left(S_{T_e},\Pi \right)=r\left(T_e,\Pi \right)$ and $r_0\left(S_{T_0},\Pi \right)=r\left(T_0,\Pi \right)$ for all $\Pi \in \Gamma$. Recalling that $S_0≔S_{T_0}$, the above shows that ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T_e,\Pi \right)\le {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T_0,\Pi \right)$. As $T_0\in 𝒯$ was arbitrary and $T_e\in {𝒯}_e$, we have shown that ${\mathrm{i}\mathrm{n}\mathrm{f}}_{T_e\in {𝒯}_e} {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T_e,\Pi \right)\le {\mathrm{i}\mathrm{n}\mathrm{f}}_{T_0\in 𝒯} {\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T_0,\Pi \right)$. □

The proof of Theorem 1 is inspired by that of the Hunt-Stein theorem given in Le Cam (2012). Establishing this result in our context required making meaningful modifications to these earlier arguments. Indeed, Le Cam (2012) uses transitions, linear maps between L-spaces, to characterize the space of decision procedures. This more complicated machinery makes it possible to broaden the set of procedures under consideration. Indeed, with this characterization, it is possible to describe decision procedures that cannot even be represented as randomized decision procedures via a Markov kernel, but instead come about as limits of such decision procedures. Despite the richness of the space of decision procedures considered, Le Cam is still able to show that this space is compact by using a coarse topology, namely the topology of pointwise convergence. Unfortunately, this topology appears to generally be too coarse for our Bayes risk function $r_0(\cdot ,\Pi )$ to be lower semi-continuous, which is a fact that we used at the beginning of our proof of Theorem 1. Another disadvantage to this formulation is that it makes it difficult to enforce any natural conditions or structure, such as continuity, on the set of estimators. It is unclear whether it would be possible to implement a numerical strategy optimizing over a class of estimators that lacks such structure. In contrast, we showed that, under appropriate conditions, it is indeed possible to prove a variant of the Hunt-Stein theorem in our setting even once natural structure is imposed on the class of estimators. To show the compactness of the space of estimators that we consider, we applied the Arzelà-Ascoli theorem.

7.2.3. Proof of Theorem 2

We provide one additional lemma before proving Theorem 2. The lemma relates to the class ${\widetilde{𝒯}}_e$ of estimators in $𝒯$ that satisfy the equivariance property (5) but do not necessarily satisfy (4). Note that ${𝒯}_e\subseteq {\widetilde{𝒯}}_e\subseteq 𝒯$

Lemma 10. If $P2$ and $P3$ hold, then, for all $T\in {\widetilde{𝒯}}_e$,

$$r(T,\Pi )=r\left(T,\Pi \circ h^{-1}\right)\text{for}\text{all}\Pi \in \Gamma ,$$

and so ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r(T,\Pi )={\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in {\Gamma }_1} r(T,\Pi )$.

Proof of Lemma 10. Let $e$ be the identity element in $\mathrm{S}\mathrm{y}\mathrm{m}\left(n\right)\times \mathrm{S}\mathrm{y}\mathrm{m}\left(p\right)$. For each $P\in 𝒫$, define $g_P\in {𝒢}_0$ to be

$$g_P≔\left(-\frac{E_P\left[Y\right]}{{\sigma }_P},\frac{1}{{\sigma }_P},{\left(-\frac{E_P\left[X_j\right]}{\sqrt{{\mathrm{V}\mathrm{a}\mathrm{r}}_P\left(X_j\right)}}\right)}_{j=1}^p,{\left(\frac{1}{\sqrt{{\mathrm{V}\mathrm{a}\mathrm{r}}_P\left(X_j\right)}}\right)}_{j=1}^p,e\right).$$

It holds that

$$\begin{gathered} R\left(T,\Pi \circ h^{-1}\right)=\int R(T,P)d\left(\Pi \circ h^{-1}\right)\left(P\right) \\ =\int R\left(T,P\circ g_P^{-1}\right)d\Pi \left(P\right)\text{by}\text{the}\text{definition}\text{of}h \\ =\int R\left(g_P\cdot T,P\right)d\Pi \left(P\right)\text{by}\text{Lemma}6 \\ =\int R(T,P)d\Pi \left(P\right)=r(T,\Pi )\text{since}T\in {\widetilde{𝒯}}_e. \end{gathered}$$

□

We conclude by proving Theorem 2.

Proof of Theorem 2. Under the conditions of the theorem, ${\widetilde{𝒯}}_e=𝒯$. Recalling that ${\Gamma }_1≔\left\{\Pi \circ h^{-1}:\Pi \in \Gamma \right\}$, Lemma 10 yields that, for any $T\in 𝒯,{\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r(T,\Pi )={\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r\left(T,\Pi \circ h^{-1}\right)={\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in {\Gamma }_1} r(T,\Pi )$. Hence, an estimator $T\in 𝒯$ is $\Gamma$-minimax if and only if it is ${\Gamma }_1$-minimax. □

7.2.4. Proof of Theorem 3

In this subsection, we assume (without statement) that all $\Pi \in \Gamma$ are defined on the measurable space $(𝒫,𝒜)$, where $𝒜$ is such that $\left\{A\cap {𝒫}_1:A\in 𝒜\right\}$ equals ${ℬ}_1$, where ${ℬ}_1$ is the collection of Borel sets on the metric space $\left({𝒫}_1,\rho \right)$ described in P5. Under P2 and P3, which we also assume without statement throughout this subsection, it then follows that each ${\Pi }_1\in {\Gamma }_1$ is defined on the measurable space $\left({𝒫}_1,{ℬ}_1\right)$, where ${ℬ}_1$ is the collection of Borel sets on $\left({𝒫}_1,\rho \right)$. Let ${\Gamma }_0$ denote the collection of all distributions on $\left({𝒫}_1,{ℬ}_1\right)$. For each $A\in {ℬ}_1$, define the $ϵ$-enlargement of $A$ by $A^{ϵ}≔\left\{P\in {𝒫}_1:\exists P^{\prime }\in A\right.$ such that $\left.\rho \left(P,P^{\prime }\right)<ϵ\right\}$. Further let $\xi$ denote the Lévy-Prokhorov metric on ${\Gamma }_0$, namely

$$\xi \left(\Pi ,{\Pi }^{\prime }\right)≔\mathrm{i}\mathrm{n}\mathrm{f}\left\{ϵ>0:\Pi \left(A\right)\le {\Pi }^{\prime }\left(A^{ϵ}\right)+ϵ\text{and}{\Pi }^{\prime }\left(A\right)\le \Pi \left(A^{ϵ}\right)+ϵ\text{for}\text{all}A\in {ℬ}_1\right\}.$$

Lemma 11. If $P5$ and $P6$, then $\left({\Gamma }_1,\xi \right)$ is a compact metric space.

Proof of Lemma 11. By Prokhorov’s theorem (see Theorem 5.2 in van Gaans, 2003 for a convenient version, or see Theorems 1.5.1 and 1.6.8 in Billingsley, 1999), P5 implies that ${\Gamma }_1$ is relatively compact in $\left({\Gamma }_0,\xi \right)$. The fact that ${\Gamma }_1$ is closed (P6) implies the result. □

We now define $r_1:{𝒮}_e\times {\Gamma }_1\to [0,\mathrm{\infty })$, which is the analogue of $r_0:𝒮\times \Gamma \to [0,\mathrm{\infty })$ from Section 7.2.2:

$$r_1(S,\Pi )≔\int {\mathrm{E}}_P\left[{\int }_{x_0:\left(\mathit{D},x_0\right)\in {𝒟}_0} {\left\{\stackrel{-}{\mathit{Y}}+s\left(\mathit{Y}\right)S\left(z\left(\mathit{D},x_0\right)\right)-{\mu }_P\left(x_0\right)\right\}}^{2}d{P}_X\left(x_0\right)\right]d\Pi \left(P\right).$$ (12)

Note that, because each distribution in $𝒫$ is continuous, each distribution in ${𝒫}_1$ is also continuous. Hence, ${𝒟}_0$ occurs with $P$-probability one for all $P\in {𝒫}_1$, and so the definition of $r_1$ combined with Lemma 8 shows that $r(T,\Pi )=r_1\left(S_T,\Pi \right)$ for any $T\in {𝒯}_e$ and $\Pi \in {\Gamma }_1$.

Lemma 12. If $P5$, then, for each $S\in {𝒮}_e,r_1(S,\cdot )$ is upper semicontinuous on $\left({\Gamma }_1,\xi \right)$.

Proof of Lemma 12. Fix $S\in {𝒮}_e$, and note that, by Lemma 8, there exists a $T\in {𝒯}_e$ such that $S=S_T$. Let ${\left\{{\Pi }_j\right\}}_{j=1}^{\mathrm{\infty }}$ be such that ${\Pi }_j\stackrel{k\to \mathrm{\infty }}{⟶}\Pi$ in $\left({\Gamma }_1,\xi \right)$ for some $\Pi \in {\Gamma }_1$. Because $\xi$ metrizes weak convergence (Theorem 1.6.8 in Billingsley, 1999), the Portmanteau theorem shows that $\mathrm{l}\mathrm{i}\mathrm{m}{\mathrm{s}\mathrm{u}\mathrm{p}}_{k\to \mathrm{\infty }} {\mathrm{E}}_{{\Pi }_j}\left[f\right(P\left)\right]\le {\mathrm{E}}_{\Pi }\left[f\right(P\left)\right]$ for every $f:{𝒫}_1\to \mathbb{R}$ that is upper semicontinuous and bounded from above on $\left({𝒫}_1,\rho \right)$. By part (iii) of P5, we can apply this result at $f:P\mapsto R(T,P)$ to see that $\mathrm{l}\mathrm{i}\mathrm{m}{\mathrm{s}\mathrm{u}\mathrm{p}}_{k\to \mathrm{\infty }} r\left(T,{\Pi }_j\right)\le r(T,\Pi )$. As ${\left\{{\Pi }_j\right\}}_{j=1}^{\mathrm{\infty }}$ was arbitrary, $r(T,\cdot )$ is upper semicontinuous on $\left({\Gamma }_1,\xi \right)$. Because $r(T,\cdot )=r_1\left(S_T,\cdot \right)$ and $S=S_T$, we have this shown that $r_1(S,\cdot )$ is upper semicontinuous on $\left({\Gamma }_1,\xi \right)$. □

Lemma 13. Under the conditions of Lemma 4, ${𝒮}_e$ is a compact subset of $C(\mathbb{Ƶ},\mathbb{R})$.

Proof. By Lemma 4$,{𝒮}_e\subset 𝒮$ is relatively compact. Hence, it suffices to show that ${𝒮}_e$ is closed. By Lemma 3, a subset of $C(\mathrm{Ƶ},\mathbb{R})$ is closed in the topology of compact convergence if it is sequentially closed. Let ${\left\{S_j\right\}}_{j=1}^{\mathrm{\infty }}$ be a sequence on ${𝒮}_e$ such that $S_j\to S$ compactly. Because ${𝒮}_e\subset 𝒮$ and $𝒮$ is closed by T3, we see that $S\in 𝒮$. We now wish to show that $S\in {𝒮}_e$. Fix $\mathit{z}\in 𝒵$ and $g\in {𝒢}_0$. Because the doubleton set $\{\mathit{z},g\cdot \mathit{z}\}$ is compact, $S_j\left(\mathit{z}\right)\to S\left(\mathit{z}\right)$ and $S_j(g\cdot \mathit{z})\to S(g\cdot \mathit{z})$, and thus $S_j\left(\mathit{z}\right)-S_j(g\cdot \mathit{z})\to S\left(\mathit{z}\right)-S(g\cdot \mathit{z})$. Moreover, because $S_j\in {𝒮}_e,S_j(g\cdot \mathit{z})=S_j\left(\mathit{z}\right)$ for all $j$. Hence, $S_j\left(\mathit{z}\right)-S_j(g\cdot \mathit{z})\to 0$. As these two limits must be equal, we see that $S\left(\mathit{z}\right)=S(g\cdot \mathit{z})$. Because $\mathit{z}\in 𝒵$ and $g\in {𝒢}_0$ were arbitrary, $S\in {𝒮}_e$. □

Lemma 14. Fix $\Pi \in {\Gamma }_1$. If T1, T2, and P4 hold, then $r_1(\cdot ,\Pi ):{𝒮}_e\to \mathbb{R}$ is lower semicontinuous.

Proof. The proof is similar to that of Lemma 9 and is therefore omitted. □

Lemma 15. If $T6$, then ${𝒮}_e$ is convex.

Proof. Fix $S_1,S_2\in {𝒮}_e$ and $\delta \in \left(\mathrm{0,1}\right)$. For any $\mathit{z}\in 𝒵$ and $g\in {𝒢}_0$,

$$g\cdot \left(\delta S_1+[1-\delta ]S_2\right)\left(\mathit{z}\right)=\delta S_1(g\cdot \mathit{z})+[1-\delta ]S_2(g\cdot \mathit{z})=\delta S_1\left(\mathit{z}\right)+[1-\delta ]S_2\left(\mathit{z}\right),$$

where the latter equality holds since $S_1,S_2\in {𝒮}_e$. Hence, $g\cdot \left(\delta S_1+[1-\delta ]S_2\right)=\delta S_1+[1-\delta ]S_2$ for all $g\in {𝒢}_0$. By T6, $\delta S_1+[1-\delta ]S_2\in 𝒮$. Hence, $\delta S_1+[1-\delta ]S_2\in {𝒮}_e≔\{S\in 𝒮:g\cdot S=S$ for all $\left.g\in {𝒢}_0\right\}$.

Lemma 16 (Minimax theorem). Under the conditions of Theorem 3,

$$\underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} \underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} r_1(S,\Pi )=\underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} \underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} r_1(S,\Pi ).$$ (13)

Proof of Lemma 16. We will show that the conditions of Theorem 1 in Fan (1953) are satisfied. By Lemma 3, $C(\mathrm{Ƶ},\mathbb{R})$ is metrizable by some metric ${\rho }_0$. By Lemma 13, $\left({𝒮}_e,{\rho }_0\right)$ is a compact metric space. Moreover, by Lemma 11, $\left({\Gamma }_1,\xi \right)$ is a compact metric space. As all metric spaces are Hausdorff, $\left({𝒮}_e,{\rho }_0\right)$ and $\left({\Gamma }_1,\xi \right)$ are Hausdorff. By Lemma 12, for each for each $S\in {𝒮}_e,r_1(S,\cdot )$ is upper semicontinuous on $\left({\Gamma }_1,\xi \right)$. By Lemma 14, for each $\Pi \in {\Gamma }_1,r_1(\cdot ,\Pi )$ is lower semicontinuous on $\left({𝒮}_e,{\rho }_0\right)$. It remains to show that $r_1$ is concavelike on ${\Gamma }_1$ (called “concave on” ${\Gamma }_1$ by Fan) and that $r_1$ is convexlike on ${𝒮}_e$ (called “convex on” ${𝒮}_e$ by Fan). To see that $r_1$ is concavelike on ${\Gamma }_1$, note that ${\Gamma }_1$ is convex (P7), and also that, for all $S\in {𝒮}_e,r_1(S,\cdot )$ is linear, and therefore concave, on ${\Gamma }_1$. Hence, $r_1$ is concavelike on ${\Gamma }_1$ (page 409 of Terkelsen, 1973). To see that $r_1$ is convexlike on ${𝒮}_e$, note that ${𝒮}_e$ is convex (Lemma 15), and also that, for all $\Pi \in {\Gamma }_1,r_1(\cdot ,\Pi )$ is convex on ${𝒮}_e$. Hence, $r_1$ is convexlike on ${𝒮}_e$ (ibid.). Thus, by Theorem 1 in Fan (1953), (13) holds. □

We conclude by proving Theorem 3.

Proof of Theorem 3. We follow arguments given on page 93 of Chang (2006) to show that, under the conditions of this theorem, (13) implies that there exists an $S^{\star }\in {𝒮}_e$ and a ${\Pi }^{\star }\in {\stackrel{\mathrm{‾}}{\Gamma }}_1$ such that

$$\underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} r_1\left(S^{\star },\Pi \right)=r_1\left(S^{\star },{\Pi }^{\star }\right)=\underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} r_1\left(S,{\Pi }^{\star }\right).$$ (14)

Noting that pointwise maxima of lower semicontinuous functions are themselves lower semicontinuous, Lemma 14 implies that ${\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r_1(\cdot ,\Pi )$ is lower semicontinuous. Because ${𝒮}_e$ is compact (Lemma 13), there exists an $S^{\star }\in {𝒮}_e$ such that

$$\underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} r_1\left(S^{\star },\Pi \right)=\underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} \underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} r_1(S,\Pi ).$$

Similarly, Lemma 12 implies that ${\mathrm{m}\mathrm{i}\mathrm{n}}_{S\in {𝒮}_e} r_1(S,\cdot )$ is upper semicontinuous on $\left({\Gamma }_1,\xi \right)$. Because $\left({\Gamma }_1,\xi \right)$ is compact (Lemma 11), there exists a ${\Pi }^{\star }\in {\Gamma }_1$ such that

$$\underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} r_1\left(S,{\Pi }^{\star }\right)=\underset{\Pi \in {\Gamma }_1}{\mathrm{m}\mathrm{a}\mathrm{x}} \underset{S\in {𝒮}_e}{\mathrm{m}\mathrm{i}\mathrm{n}} r_1(S,\Pi ).$$

By Lemma 16, the above two displays show that ${\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r_1\left(S^{\star },\Pi \right)={\mathrm{m}\mathrm{i}\mathrm{n}}_{S\in {𝒮}_e} r_1\left(S,{\Pi }^{\star }\right)$. Combining this result with the elementary fact that ${\mathrm{m}\mathrm{i}\mathrm{n}}_{S\in {𝒮}_e} r_1\left(S,{\Pi }^{\star }\right)\le r_1\left(S^{\star },{\Pi }^{\star }\right)\le {\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r_1\left(S^{\star },\Pi \right)$ shows that (14) holds.

Recall from below (12) that $r_1\left(S_T,\Pi \right)=r(T,\Pi )$ for all $\Pi \in {\Gamma }_1$ and $T\in {𝒯}_e$. Moreover, since ${𝒮}_e=\left\{S_T:T\in {𝒯}_e\right\}$ (Lemma 8), there exists a $T^{\star }\in {𝒯}_e$ such that $S=S_{T^{\star }}$. Combining these observations shows that (i) ${\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r_1\left(S^{\star },\Pi \right)={\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r_1\left(S_{T^{\star }},\Pi \right)={\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Gamma }_1} r\left(T^{\star },\Pi \right)$; (ii) $r_1\left(S^{\star },{\Pi }^{\star }\right)=r_1\left(S_{T^{\star }},{\Pi }^{\star }\right)=r\left(T^{\star },{\Pi }^{\star }\right)$; and (iii) ${\mathrm{m}\mathrm{i}\mathrm{n}}_{S\in {𝒮}_e} r_1\left(S,{\Pi }^{\star }\right)={\mathrm{m}\mathrm{i}\mathrm{n}}_{T\in {𝒯}_e} r_1\left(S_T,{\Pi }^{\star }\right)={\mathrm{m}\mathrm{i}\mathrm{n}}_{T\in {𝒯}_e} r\left(T,{\Pi }^{\star }\right)$. Hence, by (14), ${\mathrm{m}\mathrm{a}\mathrm{x}}_{\Pi \in {\Pi }^{\star }} r\left(T^{\star },\Pi \right)=r_1\left(T^{\star },{\Pi }^{\star }\right)={\mathrm{m}\mathrm{i}\mathrm{n}}_{T\in {𝒯}_e} r_1\left(T,{\Pi }^{\star }\right)$. Equivalently, for all $T\in {𝒯}_e$ and $\Pi \in {\Gamma }_1,r\left(T^{\star },\Pi \right)\le r\left(T^{\star },{\Pi }^{\star }\right)\le r\left(T,{\Pi }^{\star }\right)$. □

7.2.5. Proof of Theorem 4

Proof of Theorem 4. Fix $T\in ℳ$, and let $\left(m_1,m_2,m_3,m_4\right)\in {\prod }_{k=1}^4 {ℳ}_k$ be the corresponding modules. Recall from Algorithm 2 that, for a given $\left(\mathit{d},x_0\right),x_0^0≔\frac{x_0-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)}$ and ${\mathit{d}}^0\in {\mathbb{R}}^{n\times p\times 2}$ is defined so that ${\mathit{d}}_{i\mathrm{*}1}^0=\frac{x_i-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)}$ for all $i=1,\dots ,n$ and ${\mathit{d}}_{\mathrm{*}j2}^0=\frac{\mathit{y}-\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)}$ for all $j=1,\dots ,p$. Now, for any $\left(\mathit{d},x_0\right)\in {𝒟}_0$,

$$T\left(\mathit{d}\right)\left(x_0\right)=\stackrel{-}{\mathit{y}}+s\left(\mathit{y}\right)m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid x_0^0\right]\right)}_{j\mathrm{*}}\right),$$

and so $S_T$ takes the form

$$S_T\left(z\left(\mathit{d},x_0\right)\right)=m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid x_0^0\right]\right)}_{j\mathrm{*}}\right).$$

Because $S_T$ does not depend on the last four arguments of $z\left(\mathit{d},x_0\right)$, we know that $T$ satisfies (5), that is, is invariant to shifts and rescalings of the features and is equivariant to shifts and rescalings of the outcome. It remains to show permutation invariance, namely (4). By the permutation invariance of the sample mean and sample standard deviation, it suffices to establish the analogue of this property for $S_T$, namely that $S_T\left(z\left(A\mathit{d}B,B{x}_0\right)\right)=S_T\left(z\left(\mathit{d},x_0\right)\right)$ for all $\left(\mathit{d},x_0\right)\in {𝒟}_0,A\in 𝒜$, and $B\in ℬ$. For an array $M$ of size ${\mathbb{R}}^{n\times p\times o}$, we will write $AMB$ to mean the ${\mathbb{R}}^{n\times p\times o}$ array for which $(AMB{)}_{\mathrm{*}\mathrm{*}\ell }=A{M}_{\mathrm{*}\mathrm{*}\ell }B$ for all $\ell =\mathrm{1,2},\dots ,o$. Note that

$$\begin{array}{ll}& S_T\left(z\left(A\mathit{d}B,B{x}_0\right)\right)=m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left(A{\mathit{d}}^{0}B\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid B^{\mathrm{\top }}{x}_0^0\right]\right)}_{j\mathrm{*}}\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n {\left(A{m}_1\left({\mathit{d}}^0\right)B\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid B^{\mathrm{\top }}{x}_0^0\right]\right)}_{j\mathrm{*}}\right)\left(\text{by M}1\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(B^{\mathrm{\top }}\frac{1}{n}\sum _{i=1}^n {\left(A{m}_1\left({\mathit{d}}^0\right)\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid B^{\mathrm{\top }}{x}_0^0\right]\right)}_{j\mathrm{*}}\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(B^{\mathrm{\top }}\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid B^{\mathrm{\top }}{x}_0^0\right]\right)}_{j\mathrm{*}}\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[B^{\mathrm{\top }}{m}_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid B^{\mathrm{\top }}{x}_0^0\right]\right)}_{j\mathrm{*}}\right)\left(\text{by M}2\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(B^{\mathrm{\top }}\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid x_0^0\right]\right)}_{j\mathrm{*}}\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p {\left(B^{\mathrm{\top }}{m}_3\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid x_0^0\right]\right)\right)}_{j\mathrm{*}}\right)\left(\text{by M}3\right)\\ & =m_4\left(\frac{1}{p}\sum _{j=1}^p m_3{\left(\left[m_2\left(\frac{1}{n}\sum _{i=1}^n m_1{\left({\mathit{d}}^0\right)}_{i\mathrm{*}\mathrm{*}}\right)\mid x_0^0\right]\right)}_{j\mathrm{*}}\right)\\ & =S_T\left(z\left(\mathit{d},x_0\right)\right).\end{array}$$

Hence, $T$ satisfies (4). □

8. Extensions and Discussion

We have focused on a particular set of invariance properties on the collection of priors $\Gamma$, namely P1-P3. Our arguments can be generalized to handle other properties. As a simple example, suppose P3 is strengthened so that $\Gamma$ is invariant to nonzero (rather than only nonnegative) rescalings $\tilde{b}$ of the outcome – this property is in fact satisfied in all of our experiments. Under this new condition, the results in Section 2 remain valid with the definition of the class of equivariant estimators ${𝒯}_e$ defined in (4) and (5) modified so that $\tilde{b}$ may range over $\mathbb{R}\setminus \left\{0\right\}$. Moreover, for any $T$, Jensen’s inequality shows that the $\Gamma$-maximal risk of the symmetrized estimator that averages $T(\mathit{x},\mathit{y})\left(x_0\right)$ and negative $T(\mathit{x},-\mathit{y})\left(x_0\right)$ is no worse than that of $T$. To assess the practical utility of this observation, we numerically evaluated the performance of symmetrizations of the estimators learned in our experiments. Symmetrizing improved performance across most settings (see Appendix F). We, therefore, recommend carefully characterizing the invariance properties of a given problem when setting out to meta-learn an estimator.

Much of this work has focused on developing and studying a framework for meta-learning a $\Gamma$-minimax estimator for a single, prespecified collection of priors $\Gamma$. In some settings, it may be difficult to a priori specify a single such collection that is both small enough so that the $\Gamma$-minimax estimator is not too conservative while also being rich enough so that the priors in this collection actually place mass in a neighborhood of the true data-generating distribution. Two approaches for overcoming this challenge seem to warrant further consideration. The first would be to employ an empirical Bayes approach (Efron and Morris, 1972), wherein a large dataset from a parallel situation can be used to inform about the possible forms that the prior might take; this, in turn, would also inform about the form that the collection $\Gamma$ should take. Recent advances also make it possible to incorporate knowledge about the existence of qualitatively different categories of features when performing empirical Bayes prediction (Nabi et al., 2020). The second approach involves using AMC to approximate $\Gamma$-minimax estimators over various choices of $\Gamma$, and then to use a stacked ensemble to combine the predictions from these various base estimators. In our data experiments, we saw that a simple version of this ensemble that combined four base AMC estimators consistently performed at least as well as the best of these base estimators.

In this work, we have focused on the case where the problem of interest is a supervised learning problem and the objective is to predict a continuous outcome based on iid data. While the AMC algorithm generalizes naturally to a variety of other sampling schemes and loss functions (see Luedtke et al., 2020), our characterization of the equivariance properties of an optimal estimator was specific to the iid regression setting that we considered. In future work, it would be interesting to characterize these properties in greater generality, including in classification settings and inverse reinforcement learning settings (e.g., Russell, 1998; Geng et al., 2020).

Appendices

A. Review of amenability

In this appendix, we review the definition of an amenable group, an important implication of amenability, and also some sufficient conditions for establishing that a group is amenable. This material will prove useful in our proof of Theorem 1 (see Section 7.2.2). We refer the reader to Pier (1984) for a thorough coverage of amenability.

Definition 1 (Amenability). Let $𝒢$ be a locally compact, Hausdorff group and let $L^{\mathrm{\infty }}\left(𝒢\right)$ be the space of Borel measurable functions that are essentially bounded with respect to the Haar measure. A mean on $L^{\mathrm{\infty }}\left(𝒢\right)$ is defined as a linear functional $M\in L^{\mathrm{\infty }}(𝒢{)}^{\mathrm{*}}$ such that $M\left(\lambda \right)\ge 0$ whenever $\lambda \ge 0$ and $M\left(1_{𝒢}\right)=1$. A mean $M$ is said to be left invariant for a group $𝒢$ if and only if $M\left({\delta }_g\mathrm{*}\lambda \right)=M\left(\lambda \right)$ for all $\lambda \in L^{\mathrm{\infty }}\left(𝒢\right)$, where $\left({\delta }_g\mathrm{*}\lambda \right)\left(h\right)=\lambda \left(g^{-1}h\right)$. The group $𝒢$ is said to be amenable if and only if there is a left invariant mean on $L^{\mathrm{\infty }}\left(𝒢\right)$.

We now introduce the fixed point property, and subsequently present a result showing its close connection to the definition given above. Throughout this work, we equip all group actions $𝒢\times 𝒲\to 𝒲$ with the product topology

Definition 2 (Fixed point property). We say that a locally compact, Hausdorff group $𝒢$ has the fixed point property if, whenever $𝒢$ acts affinely on a compact convex set $𝒦$ in a locally convex topological vector space $E$ with the map $𝒢\times 𝒦\to 𝒦$ continuous, there is a point in $x_0\in 𝒦$ fixed under the action of $𝒢$.

Theorem S1 (Day’s Fixed Point Theorem). A locally compact, Hausdorff group $𝒢$ has the fixed point property if and only if $𝒢$ is amenable.

Proof. See the proof of Theorem 5.4 in Pier (1984). □

The following results are useful for establishing amenability.

Lemma S17. Any compact group is amenable.

Proof. Take the normalized Haar measure as an invariant mean. □

Lemma S18. Any locally compact Abelian group is amenable.

Proof. See the proof of Proposition 12.2 in Pier (1984). □

Lemma S19. Let $𝒢$ be a locally compact group and $𝒩$ a closed normal subgroup of $𝒢$. If $𝒩$ and $𝒢/𝒩$ are amenable, then $𝒢$ is amenable.

Proof. Assume that a continuous affine action of $𝒢$ on a nonempty compact convex set $𝒦$ is given. Let ${𝒦}^{𝒩}$ be the set of all fixed points of $𝒩$ in $𝒦$. Since $𝒩$ is amenable, Theorem S1 implies that ${𝒦}^{𝒩}$ is nonempty. Since the group action is continuous, ${𝒦}^{𝒩}$ is a closed subset of $𝒦$ and hence is compact. Since the action is affine, ${𝒦}^{𝒩}$ is convex. Now, note that, for all $x\in {𝒦}^{𝒩}$, $g\in 𝒢$, and $n\in 𝒩$, the fact that $g^{-1}ng\in 𝒩$ implies that $g^{-1}ngx=x$ which implies $ngx=gx$. Hence, ${𝒦}^{𝒩}$ is preserved by the action of $𝒢$. The action of $𝒢$ on ${𝒦}^{𝒩}$ factors to an action of $𝒢/𝒩$ on ${𝒦}^{𝒩}$, which has a fixed point $x_0$ since $𝒢/𝒩$ is amenable. But then $x_0$ is fixed by each $g\in 𝒢$. Hence, $𝒢$ is amenable. □

B. Examples of collections $𝒮$ where T1-T6 hold

B.1. Infinite-dimensional class

We start by presenting an infinite-dimensional class $𝒮$ that satisfies T1-T6, and then we subsequently present a finite-dimensional class. To define this class, we fix $c,\alpha >0$ and a function $F:Ƶ\to {\mathbb{R}}^{+}$some function that is invariant to permutations, shifts, and rescalings, in the sense that both of the following hold:

• F1.
  Permutations: For all $\left((\mathit{x},\mathit{y}),x_0\right)\in {𝒟}_0,A\in 𝒜$ and $B\in ℬ$, it holds that

  $$F\left(z\left((A\mathit{x}B,A\mathit{y}),B^{\mathrm{\top }}{x}_0\right)\right)=F\left(z\left((\mathit{x},\mathit{y}),x_0\right).\right.$$
• F2.Shifts and rescalings: For all $\left((\mathit{x},\mathit{y}),x_0\right)\in {𝒟}_0,a\in {\mathbb{R}}^p,b\in {\left({\mathbb{R}}^{+}\right)}^p,\tilde{a}\in \mathbb{R}$, and $\tilde{b}>0$, it holds that $F\left(z\left(\left({\mathit{x}}^{a,b},\tilde{a}+\tilde{b}\mathit{y}\right),a+b\odot x_0\right)\right)=F\left(z\left((\mathit{x},\mathit{y}),x_0\right)\right.$, where ${\mathit{x}}^{a,b}$ is the $n\times p$ matrix with row $i$ equal to $a+b\odot {\mathit{x}}_{i\mathrm{*}}$.

These conditions bear some resemblance to T4 and T5. One example of a function $F$ satisfies the above conditions is a constant function.

The infinite-dimensional class of $Ƶ\to \mathbb{R}$ functions that we consider is defined as

$${𝒮}_{F,\alpha ,c}≔\left\{S:\forall \mathit{z}\in 𝒵,\left|S\right(\mathit{z}\left)\right|\le F\left(\mathit{z}\right),\underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c\right\}.$$

We will now show that this class satisfies T1-T6. Conditions T1 and T2 follow immediately from the definition of ${𝒮}_{F,\alpha ,c}$. We now show that T3 holds. Because $C(Ƶ,\mathbb{R})$ is complete, it suffices to show that, if $S_n\to S$ converges compactly and $S_n\in {𝒮}_{F,\alpha ,c}$, then $S\in {𝒮}_{F,\alpha ,c}$. Let $S_n\to S$ compactly. To see that $\left|S\right(\mathit{z}\left)\right|\le F\left(\mathit{z}\right)$, note that

$$\left|S\right(\mathit{z}\left)\right|\le \left|S_n\left(\mathit{z}\right)-S\left(\mathit{z}\right)\right|+\left|S_n\left(\mathit{z}\right)\right|\le F\left(\mathit{z}\right)+\left|S_n\left(\mathit{z}\right)-S\left(\mathit{z}\right)\right|$$

and then take the limit as $n\to \mathrm{\infty }$. To see that $S$ satisfies the Hölder condition, note that, for any $\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵$,

$$\frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le \frac{\left|S\left(\mathit{z}\right)-S_n\left(\mathit{z}\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}+\frac{\left|S_n\left(\mathit{z}\right)-S_n\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}+\frac{\left|S_n\left({\mathit{z}}^{\prime }\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}$$

and again take the limit as $n\to \mathrm{\infty }$. Hence, $\frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c$ for each $\mathit{z}\ne {\mathit{z}}^{\prime }$, and so ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c$. Hence $S\in {𝒮}_{F,\alpha ,c}$, and thus T3 holds. We now show that T4 and T5 hold. To do this, we will use the group theoretic notation defined in Section 7.1. As noted in that section, T4 and T5 are equivalent to the condition that $g\cdot S\in {𝒮}_{F,\alpha ,c}$ for all $g\in {𝒢}_0$ and $S\in {𝒮}_{F,\alpha ,c}$. We will therefore fix $S\in {𝒮}_{F,\alpha ,c}$ and $g\in {𝒢}_0$ and show that $g\cdot S\in {𝒮}_{F,\alpha ,c}$. For $\mathit{z}\in 𝒵$, we have that

$$\left|\right(g\cdot S\left)\right(\mathit{z}\left)\right|=\left|S\right(g\cdot \mathit{z}\left)\right|\le F(g\cdot \mathit{z})=F\left(\mathit{z}\right),$$

where the inequality holds since $S\in {𝒮}_{F,\alpha ,c}$. Note that for any $\mathit{z},{\mathit{z}}^{\prime }\in 𝒵,∥g\cdot \mathit{z}-g\cdot {\mathit{z}}^{\prime }∥=∥\mathit{z}-{\mathit{z}}^{\prime }∥$. Hence,

$$\underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|(g\cdot S)\left(\mathit{z}\right)-(g\cdot S)\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}=\underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|(g\cdot S)\left(\mathit{z}\right)-(g\cdot S)\left({\mathit{z}}^{\prime }\right)\right|}{{∥g\cdot \mathit{z}-g\cdot {\mathit{z}}^{\prime }∥}^{\alpha }}=\underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c,$$

where the inequality holds since $S\in {𝒮}_{F,\alpha ,c}$. Hence, $g\cdot S\in {𝒮}_{F,\alpha ,c}$, and so T4 and T5 hold. It remains to show T6. To see that this holds, fix $S_1,S_2\in {𝒮}_{F,\alpha ,c}$ and $\delta \in \left(\mathrm{0,1}\right)$ and let $S=\delta S_1+(1-\delta )S_2$. By the triangle inequality and the fact that $S_1,S_2\in {𝒮}_{F,\alpha ,c}$, we have the following two displays for any $\mathit{z},{\mathit{z}}^{\prime }\in 𝒵$ :

$$\begin{array}{ll}& \left|S\right(\mathit{z}\left)\right|=\left|\delta S_1\left(\mathit{z}\right)+(1-\delta )S_2\left(\mathit{z}\right)\right|\le \delta \left|S_1\left(\mathit{z}\right)\right|+(1-\delta )\left|S_2\left(\mathit{z}\right)\right|\le F\left(\mathit{z}\right),\\ & \underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le \delta \underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S_1\left(\mathit{z}\right)-S_1\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}+(1-\delta )\underset{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \frac{\left|S_2\left(\mathit{z}\right)-S_2\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c.\end{array}$$

Hence, $S\in {𝒮}_{F,\alpha ,c}$, and so T6 holds.

B.2. Finite-dimensional class

B.2.1. Overview

For an explicit representation of $𝒵$, we have

$$𝒵={𝒪}_n^p\times {𝒪}_n\times {\mathbb{R}}^p\times {\mathbb{R}}^p\times \mathbb{R}\times {\mathbb{R}}^p\times \mathbb{R},$$

where ${𝒪}_n=\left\{a\in {\mathbb{R}}^n\mid \bar{a}=0,s\left(a\right)=1\right\}$. For ease of communication, we will abbreviate

$$z_a=\left(\frac{\mathit{x}-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)},\frac{\mathit{y}-\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)}\right)\in {𝒪}_n^p\times {𝒪}_n,$$

$$z_t=\frac{x_0-\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)}\in {\mathbb{R}}^p$$

$$z_m=\left(\frac{\stackrel{-}{\mathit{x}}}{s\left(\mathit{x}\right)},\frac{\stackrel{-}{\mathit{y}}}{s\left(\mathit{y}\right)}\right)\in {\mathbb{R}}^p\times \mathbb{R}$$

$$z_s=\left(\text{log}s\left(x\right),\text{log}s\left(y\right)\right)\in {\mathbb{R}}^p\times \mathbb{R},$$

so that $z=\left(z_a,z_t,z_m,z_s\right)$. Here, $z_a$ stands for the angular component, $z_t$ stands for the test point, $z_m$ stands for the mean, $z_s$ stands for the standard deviation.

To define our parametric example for $𝒮$, we can use separation of variables to consider the coordinates of $\mathit{z}$ separately. We will consider estimators belonging to the class $𝒮$ of all $S$ such that

$$S\left(\mathit{z}\right)=S_a\left(z_a\right)S_t\left(z_t\right)S_g\left(z_m,z_s\right),S_a\in {𝒮}_a,S_t\in {𝒮}_t,S_g\in {𝒮}_g.$$

We refer to ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ as the angular part, test point part, and group part of $𝒮$, respectively. In what follows, we will describe conditions on ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ that make it so that T1-T6 hold. We will then describe interesting collections ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ that satisfy these conditions.

First note that we have the following inequality:

$$\begin{array}{ll}\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|& =\left|S_a\left(z_a\right)S_t\left(z_t\right)S_g\left(z_m,z_s\right)-S_a\left(z_a^{\prime }\right)S_t\left(z_t^{\prime }\right)S_g\left(z_m^{\prime },z_s^{\prime }\right)\right|\\ & \le \left|S_t\left(z_t\right)\right|\left|S_g\left(z_m,z_s\right)\right|\left|S_a\left(z_a\right)-S_a\left(z_a^{\prime }\right)\right|+\left|S_a\left(z_a^{\prime }\right)\right|\left|S_g\left(z_m,z_s\right)\right|\left|S_t\left(z_t\right)-S_t\left(z_t^{\prime }\right)\right|+\left|S_a\left(z_a^{\prime }\right)\right|\left|S_t\left(z_t^{\prime }\right)\right|\left|S_g\left(z_m,z_s\right)-S_g\left(z_m^{\prime },z_s^{\prime }\right)\right|.\end{array}$$

Thus if $S_a,S_t$, and $S_g$ were uniformly bounded by $M^{1/3}$ and each of their global Hölder constant was less than or equal to $\frac{c}{3M^{2/3}}$, then ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{z}\in 𝒵} \left|S\right(\mathit{z}\left)\right|\le M$ and ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{z}\ne {\mathit{z}}^{\prime }\in 𝒵} \frac{\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|}{{∥\mathit{z}-{\mathit{z}}^{\prime }∥}^{\alpha }}\le c$. Hence, if ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ are such that functions in these collections are uniformly bounded by $M^{1/3}$ and are $\frac{c}{3M^{2/3}}$-Hölder, then $𝒮\subseteq {𝒮}_{M,\alpha ,c}$. In that case, conditions T1 and T2 hold. Since every compact subset of $𝒵$ can be written as a subset of a product of compact sets $K=K_1\times K_2\times K_3$, $K_1\subseteq {𝒪}_n^{p+1},K_2\subset {\mathbb{R}}^p,K_3\subset {\mathbb{R}}^{2p+2}$, for condition T3 to hold, it suffices to show ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ are closed. Condition T4 holds if ${𝒮}_a$ is closed under rotations with respect to the $n$ observations and if ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ are closed under permutations with respect to the $p$ features. The latter can be done by letting ${𝒮}_a,{𝒮}_t$, and ${𝒮}_g$ be $p$-fold tensor products of an identical space of functions. Condition T5 is satisfied when ${𝒮}_g$ is closed under shifts. Finally, condition T6 holds when ${𝒮}_a$, ${𝒮}_t$, and ${𝒮}_g$ are convex since the projected tensor product of convex sets is convex.

B.2.2. Angular Part $\left({𝒮}_{\mathit{a}}\right)$

We define ${𝒮}_a$ by truncating an orthonormal basis for the tensor product space $L^2{\left({𝒪}_n\right)}^{\otimes (p+1)}$ to a specified finite number of terms and then taking the subset of the span of those basis vectors that are contained in $S_{M^{1/3},\alpha ,c/(3M^{2/3})}$ for some $c$ and $M$. Note that $O_n\cong S^{n-2}$, where “≅” denotes an isomorphic relation and ${\mathbb{S}}^{n-2}$ is the $(n-2)$-dimensional unit sphere. Let 1 be the $n$-dimensional vector of 1's, and note that ${𝒪}_n$ can be expressed in the following form:

$${𝒪}_n=\left\{w\in {\mathbb{R}}^n\mid n^{-1/2}{w}^{T}1=0,n^{-1}{w}^{T}w=1\right\}.$$

Let $U\in O\left(n\right)$, the orthogonal group, be such that $n^{-1/2}U1=e_n$, the $n$th elementary basis vector. Such a $U$ exists because $∥n^{-1/2}1∥=1$. Then,

$${𝒪}_n=\left\{\sqrt{n}{U}^{T}v\mid v\in {\mathbb{R}}^n,v_n=0,\parallel v{\parallel }^2=1\right\}$$

We have the isomorphism $\zeta :L^2\left({𝒪}_n\right)\to L^2\left({\mathbb{S}}^{n-2}\right),\zeta \left(f\right)\left(v\right)=f\left(\sqrt{n}{U}^{T}v\right)$. Thus, if we have an orthonormal basis for $L^2\left({\mathbb{S}}^{n-2}\right)$, we may use the operator ${\zeta }^{-1}$ to obtain an orthonormal basis for $L^2\left({𝒪}_n\right)$. Let ${\mathbf{H}}_{\ell }$ be the space of harmonic polynomials of degree $\ell$ in $(n-1)$-dimensions. By the Stone-Weierstrass theorem, the direct sum ${⨁}_{\ell =0}^{\mathrm{\infty }} {\mathbf{H}}_{\ell }$ is dense in $L^2\left({\mathbb{S}}^{n-2}\right)$. We can truncate the series and stop at a prespecified point $q_a$, so that

$${𝒮}_a={\left(\left(\underset{\ell =0}{\overset{q_a}{⨁}} {\mathbf{H}}_{\ell }\right)\circ \left(\sqrt{n}{U}^T\right)\right)}^{\otimes (p+1)}\bigcap {𝒮}_{M^{1/3},\alpha ,c/\left(3M^{2/3}\right)}\left({𝒪}_n^{p+1}\right).$$ (S1)

We use the orthonormal basis $\left\{{𝒴}_{l_1,l_2,\cdots ,l_{n-2}}:\left|l_1\right|\le l_2\le \cdots \le l_{n-2}\right\}$ for the spherical harmonics introduced in Higuchi (1987) (replacing “$Y$” in their notation by “$𝒴$” to avoid notational overload), where an explicit expression for this basis is provided in that work. Let $N(n,p,q)={\left({\sum }_{\ell =0}^q \mathrm{d}\mathrm{i}\mathrm{m}{\mathbf{H}}_{\ell }\right)}^{p+1}$ and

$${𝒞}_a=\left\{A\in {\mathbb{R}}^{N\left(n,p,q_a\right)}:\prod _{j=0}^p \sum _{\left|l_1\right|\le l_2\le \cdots \le l_{n-2}\le q_a} A_{l_1,\cdots ,l_{n-2}}{𝒴}_{l_1,\cdots ,l_{n-2}}\left(\sqrt{n}{U}^T{z}^{x,\cdot ,j}\right)\in {𝒮}_a\right\},$$

where $z^{x,i,0}=z^{y,i}$. The set ${𝒞}_a$ is the coefficient space of the basis expansion in ${𝒮}_a$ and is convex and compact if and only if ${𝒮}_a$ is convex and compact. The set ${𝒮}_a$ is closed under rotations in the $n$ observations since the spherical harmonics for any given degree is closed under rotations. It is also closed under permutations due to the $(p+1)$-fold tensor product form. As an intersection of closed convex sets, it is closed and convex.

B.2.3. Test Point Part $\left({𝒮}_{\mathit{t}}\right)$

Similarly to ${𝒮}_a,{𝒮}_t$ is defined by truncating an orthonormal basis for $L^2\left({\mathbb{R}}^p\right)$. Let ${\left\{{\psi }_k\right\}}_{k=0}^{\mathrm{\infty }}$ be the normalized Hermite functions. They form an orthonormal basis of $L^2\left(\mathbb{R}\right)$ and so their $p$-fold tensor product is an orthonormal basis of $L^2\left({\mathbb{R}}^p\right)$. We can take

$${𝒮}_t={\left(\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\left\{{\psi }_k\mid k\in \left\{\mathrm{0,1},\cdots ,q_t\right\}\right\}\right)}^{\otimes p}\cap {𝒮}_{M^{1/3},\alpha ,c/\left(3M^{2/3}\right)}\left({\mathbb{R}}^p\right).$$

We can similarly define the coefficient space ${𝒞}_t$ :

$${𝒞}_t=\left\{A\in {\mathbb{R}}^{q_{t}p}:\left(z_t\mapsto \prod _{j=1}^p \sum _{k=0}^{q_t} A_{jk}{\psi }_k\left(z_{t,j}\right)\right)\in {𝒮}_t\right\}.$$

Similarly to ${𝒮}_a$, the $p$-fold tensor product form and it being an intersection of closed and convex sets show all of the necessary conditions are satisfied.

B.2.4. Group Part $\left({𝒮}_{\mathit{g}}\right)$

The ${𝒮}_g$ that we will define imposes that the functions are periodic in each dimension, in the sense that, if $S_g\in {𝒮}_g$ and $z_g-z_g^{\prime }=\pm e_i$ for some elementary basis vector $e_i$, then $S_g\left(z_g\right)=S_g\left(z_g^{\prime }\right)$. In other words, we will be dealing with functions on the $(2p+2)$-dimensional torus, ${\mathbb{T}}^{2p+2}={\left(S^1\right)}^{2p+2}$. Since the torus is a product of 1 -spheres, we can use the same process as described when defining the angular part $\left({𝒮}_a\right)$, namely letting

$${𝒮}_g={\left(\underset{\ell =0}{\overset{q_g}{⨁}} H_{\ell }\right)}^{\otimes (2p+2)}\bigcap {𝒮}_{M^{1/3},\alpha ,c/\left(3M^{2/3}\right)}\left({\mathbb{T}}^{2p+2}\right)$$ (S2)

In this case, $H_{\ell }=\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}\{\mathrm{c}\mathrm{o}\mathrm{s}(2\pi \ell x),\mathrm{s}\mathrm{i}\mathrm{n}(2\pi \ell x\left)\right\}$ and translations can be dealt with by the sum and difference formulas for sine and cosine. Translations under periodicity are the same as rotation, and since it is known that spherical harmonics are rotationally invariant, ${𝒮}_g$ is closed under translations. Similarly, the tensor product form of ${𝒮}_a$ and its being an intersection of closed and convex sets implies that the rest of the sufficient conditions described at the end of Section B.2.1 are satisfied.

C. Examples of collections $\mathbf{\Gamma }$ where P5 hold

We now describe settings where P5 is often applicable. We will specify ${𝒫}_1$ in each of these settings, and the model $𝒫$ is then defined by expanding ${𝒫}_1$ to contain the distributions of all possible shifts and rescalings of a random variate drawn from some $P_1\in {𝒫}_1$. The first class of models for which P 5 is often satisfied is parametric in nature, with each distribution $P_{\theta }\in {𝒫}_1$ indexed smoothly by a finite dimensional parameter $\theta$ belonging to a subset $\Theta$ of ${\mathbb{R}}^k$. We note here that, because the sample size $n$ is fixed in our setting, we can obtain an essentialy unrestricted model by allowing $k$ to be large relative to $n$. In parametric settings, $\rho$ can often be defined as $\rho \left(P_{\theta },P_{{\theta }^{\prime }}\right)={∥\theta -{\theta }^{\prime }∥}_2$, where we recall that $\parallel \cdot {\parallel }_2$ denotes the Euclidean norm. If ${\Gamma }_1$ is uniformly tight, which certainly holds if $\Theta$ is bounded, then P5 holds provided $\theta \mapsto R\left(T,P_{\theta }\right)$ is upper-semicontinuous for all $T\in {𝒯}_e$. For a concrete example where the conditions of P5 are satisfied, consider the case that $\Theta =\left\{\theta :\parallel \theta {\parallel }_0\le {\mathfrak{s}}_0,\parallel \theta {\parallel }_1\le {\mathfrak{s}}_1\right\}$ for sparsity parameters ${\mathfrak{s}}_0$ and ${\mathfrak{s}}_1$ on $\parallel \theta {\parallel }_0≔\#\left\{j:{\theta }_j\ne 0\right\}$ and $\parallel \theta {\parallel }_1≔{\sum }_j \left|{\theta }_j\right|$, and $P_{\theta }$ is the distribution for which $X~N\left(0_p,{\mathrm{I}\mathrm{d}}_p\right)$, and $Y\mid X~N\left({\theta }^{\mathrm{\top }}X,1\right)$. This setting is closely related to the sparse linear regression example that we study numerically in Section 5.3.2.

Condition P5 also allows for nonparametric regression functions. Define ${\varphi }^p$ to be the $p$-dimensional standard Gaussian measure. Define $L_0^2\left({\varphi }^p\right)=\left\{f\in L^2\left({\varphi }^p\right)\mid \int f\left(x\right)d{\varphi }^p\left(x\right)=0\right\}$. Let $ℱ\subset L_0^2\left({\varphi }^p\right)$ satisfy the following conditions:

1. $ℱ$ is bounded. ${\mathrm{s}\mathrm{u}\mathrm{p}}_{f\in ℱ} \parallel f{\parallel }_{L^2\left({\varphi }^p\right)}<\mathrm{\infty }$.

2. $ℱ$ is uniformly equivanishing. ${\mathrm{l}\mathrm{i}\mathrm{m}}_{N\to \mathrm{\infty }} {\mathrm{s}\mathrm{u}\mathrm{p}}_{f\in ℱ} {∥f{1}_{B(0,N{)}^c}∥}_{L^2\left({\varphi }^p\right)}=0$.

3. $ℱ$ is uniformly equicontinuous. ${\mathrm{l}\mathrm{i}\mathrm{m}}_{r\searrow 0} {\mathrm{s}\mathrm{u}\mathrm{p}}_{f\in ℱ} {\mathrm{s}\mathrm{u}\mathrm{p}}_{y\in B(0,r)} {∥{\tau }_{y}f-f∥}_{L^2\left({\varphi }^p\right)}=0$ where ${\tau }_y$ is the translation by $y$ operator.

4. $ℱ$ is closed in $L^2\left({\varphi }^p\right)$.

5. There exists $q^{\prime }>2$ such that $ℱ\subset L^{q^{\prime }}\left({\varphi }^p\right)$.

By a generalization of the Riesz-Kolmogorov theorem as seen in Guo and Zhao (2019), $ℱ$ is compact under assumptions (i) through (iv). Let $c>0,\alpha \in \left(\mathrm{0,1}\right]$. We suppose that $𝒮={𝒮}^0$ where ${𝒮}^0$ is the set of all functions $S:Ƶ\to \mathbb{R}$ such that $\left|S\right(\mathit{z}\left)\right|\le F\left(\mathit{z}\right),\left|S\left(\mathit{z}\right)-S\left({\mathit{z}}^{\prime }\right)\right|\le c{∥\mathit{z}-{\mathit{z}}^{\prime }∥}_2^{\alpha }$ for all $\mathit{z},{\mathit{z}}^{\prime }\in 𝒵$. Assume further that $F$ is bounded, i.e.

$$\underset{z\in 𝒵}{\mathrm{s}\mathrm{u}\mathrm{p}} \left|F\right(z\left)\right|=B_{{𝒮}^0}<\mathrm{\infty },$$ (S3)

and also that $F$ is constant in the orbits induced by the group action on $𝒵$ defined in Section 7.1.

For each $f\in ℱ$, let $P_f$ denote the distribution of $X~N\left(0,{\mathrm{I}\mathrm{d}}_p\right),Y\mid X~N\left(f\right(X),1)$. Suppose that ${𝒫}_1=\left\{P_f\mid f\in ℱ\right\}$. With the metric $\rho (f,g)=\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)},\left({𝒫}_1,\rho \right)$ is a complete separable compact metric space. We also see that $P\mapsto R(T,P)$ is continuous.

Lemma S20. For all $T\in {𝒯}_e,P\mapsto R(T,P)$ is continuous in this example.

Proof. To ease presentation, we introduce some notation. For $f\in ℱ$, let $f\left(\mathit{x}\right)≔{\left(f\left(x_i\right)\right)}_{i=1}^n$, $\bar{f}\left(\mathit{x}\right)≔\frac{1}{n}{\sum }_{i=1}^n f\left(x_i\right),s_f\left(\mathit{d}\right)≔s(\mathit{y}+f(\mathit{x}\left)\right)$, and $\bar{y}\left(\mathit{y}\right)≔\stackrel{-}{\mathit{y}}$. Let $S_{T,f}$ denote the map $\left(\mathit{d},x_0\right)\mapsto S_T\left(z_f\left(\mathit{d},x_0\right)\right)$, where $z_f\left(\mathit{d},x_0\right)$ takes the same value as $z_f\left(\mathit{d},x_0\right)$ except that the entry $\frac{\mathit{y}-\stackrel{-}{\mathit{y}}}{s\left(y\right)}$ is replaced with $\frac{\mathit{y}+f\left(\mathit{x}\right)-\stackrel{-}{\mathit{y}}-\bar{f}\left(\mathit{x}\right)}{s_f}$. Also let ${\varphi }^{\star }≔{\varphi }^{p(n+1)+n}$. For $q\in [1,\mathrm{\infty })$ and a function $f:𝒟\times 𝒳$, we let $\parallel f{\parallel }_{L^q\left({\varphi }^{\star }\right)}≔{\left[\int {\left|f\left(\mathit{x},\mathit{y},x_0\right)\right|}^q{\varphi }^{\star }\left(d\mathit{x},d\mathit{y},d{x}_0\right)\right]}^{1/q}$. We let $\parallel f{\parallel }_{L^{\mathrm{\infty }}\left({\varphi }^{\star }\right)}≔\mathrm{i}\mathrm{n}\mathrm{f}\{c\ge 0:f\left(\mathit{x},\mathit{y},x_0\right)\le c{\varphi }^{\star }$-a.s. $\}$. For $f:𝒟\to \mathbb{R}$, we write $\parallel f{\parallel }_{L^q\left({\varphi }^{\star }\right)}$ to mean ${∥\left(\mathit{d},x_0\right)\mapsto f\left(\mathit{d}\right)∥}_{L^q\left({\varphi }^{\star }\right)}$, and follow a similar convention for functions that only take as input $\mathit{x},x_i,\mathit{y}$, or $x_0$. We will write ≲ to mean inequality up to a positive multiplicative constant that may only depend on $𝒮$ or $ℱ$.

Fix $\epsilon \in \left(\mathrm{0,1}\right)$ and $T\in {𝒯}_e$. Now, for any $f\in ℱ$, a change of variables shows that

$$\begin{gathered} R\left(T,P_f\right) \\ =E_{P_f}\left[\int {\left[T(\mathit{X},\mathit{Y})\left(x_0\right)-f\left(x_0\right)\right]}^{2}d{\varphi }^p\left(x_0\right)\right] \\ =\int {\left[T(\mathit{x},\mathit{y})\left(x_0\right)-f\left(x_0\right)\right]}^2(2\pi {)}^{-\frac{n}{2}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-\frac{1}{2}\sum _{i=1}^n {\left\{y_i-f\left(x_i\right)\right\}}^2\right]{\varphi }^{p(n+1)}\left(d\mathit{x},d{x}_0\right)d\mathit{y} \\ =\int {\left[T(\mathit{x},\mathit{y}+f(\mathit{x}\left)\right)\left(x_0\right)-f\left(x_0\right)\right]}^2{\varphi }^{\star }\left(d\mathit{x},d{x}_0,d\mathit{y}\right) \\ =\int {\left[\stackrel{-}{\mathit{y}}+s(\mathit{y}+f(\mathit{x}\left)\right)S_{T,f}\left(\mathit{d},x_0\right)+\bar{f}\left(\mathit{x}\right)-f\left(x_0\right)\right]}^2{\varphi }^{\star }\left(d\mathit{x},d{x}_0,d\mathit{y}\right). \end{gathered}$$

Hereafter we write $d{\varphi }^{\star }$ to denote ${\varphi }^{\star }\left(d\mathit{x},d{x}_0,d\mathit{y}\right)$.

Fix $f,g\in ℱ$. Most of the remainder of this proof will involve establishing that $\left(T,P_f\right)-R\left(T,P_g\right)\lesssim {\epsilon }^{-2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}+\epsilon$. By symmetry, it will follow that $\left|R\left(T,P_f\right)-R\left(T,P_g\right)\right|\le {\epsilon }^{-2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}+\epsilon$.

In what follows we will use the notation $(g-f)\left(x_0\right)$ to mean $g\left(x_0\right)-f\left(x_0\right),(\bar{g}-\bar{f})\left(\mathit{x}\right)$ to mean $\bar{g}\left(\mathit{x}\right)-\bar{f}\left(\mathit{x}\right)$, etc. The above yields that

$$\left.{R(T,P}_f\right)-R\left(T,P_g\right)=\int \left[{\left(\bar{f}\left(\mathit{x}\right)-f\left(x_0\right)\right)}^2-{\left(\bar{g}\left(\mathit{x}\right)-g\left(x_0\right)\right)}^2\right]d{\varphi }^{\star }$$ (S4)

$$+2\int \stackrel{-}{\mathit{y}}\left[(g-f)\left(x_0\right)-(\bar{g}-\bar{f})\left(\mathit{x}\right)\right]d{\varphi }^{\star }$$ (S5)

$$+2\int \stackrel{-}{\mathit{y}}\left[s_f\left(\mathit{d}\right)S_{T,f}\left(\mathit{d},x_0\right)-s_g\left(\mathit{d}\right)S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star }$$ (S6)

$$+\int \left[s_f^2\left(\mathit{d}\right)S_{T,f}{\left(\mathit{d},x_0\right)}^2-s_g^2\left(\mathit{d}\right)S_{T,g}{\left(\mathit{d},x_0\right)}^2\right]d{\varphi }^{\star }$$ (S7)

$$+2\int \left[\left(\overline{f}\left(\mathit{x}\right)-f\left(x_0\right)\right)s_f\left(\mathit{d}\right)S_{T,f}\left(d,{\mathit{x}}_0\right)-\left(\overline{g}\left(x\right)-g\left(x_0\right)\right)s_g\left(\mathit{d}\right)S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star }.$$ (S8)

We bound the labeled terms on the right-hand side separately. After some calculations, it can be seen that (S4) and (S5) are bounded by a constant multiplied by $\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}$. These calculations, which are omitted, involve several applications of the triangle inequality, the Cauchy-Schwarz inequality, and condition (i).

The integral in (S6) bounds as follows:

$$\begin{gathered} \int \stackrel{-}{\mathit{y}}\left[s_f\left(\mathit{d}\right)S_{T,f}\left(\mathit{d},x_0\right)-s_g\left(\mathit{d}\right)S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star } \\ =\int \stackrel{-}{\mathit{y}}{S}_{T,f}\left(\mathit{d},x_0\right)\left[s_f\left(\mathit{d}\right)-s_g\left(\mathit{d}\right)\right]d{\varphi }^{\star }+\int \stackrel{-}{\mathit{y}}{s}_g\left(\mathit{d}\right)\left[S_{T,f}\left(\mathit{d},x_0\right)-S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star }\le {∥\bar{y}{S}_{T,f}\left[s_f-s_g\right]∥}_{L^1\left({\varphi }^{\star }\right)}+{∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]∥}_{L^1\left({\varphi }^{\star }\right)}. \end{gathered}$$ (S9)

We start by studying first term of the right-hand side above. Note that, by (S3) and the assumption that $\left|S\right(\mathit{z}\left)\right|\le F\left(\mathit{z}\right)$ for all $\mathit{z}\in Ƶ$ and $S\in 𝒮$, we have that $\left|S_{T,f}\left(\mathit{d},x_0\right)\right|\le B_{{𝒮}^0}$. Combining this with Cauchy-Schwarz, the first term on the right-hand side above bounds as

$${∥\bar{y}{S}_T\left[s_f-s_g\right]∥}_{L^1\left({\varphi }^{\star }\right)}\le B_{{𝒮}_0}\parallel \bar{y}{\parallel }_{L^2\left({\varphi }^{\star }\right)}{∥s_f-s_g∥}_{L^2\left({\varphi }^{\star }\right)}.$$ (S10)

To continue the above bound, we will show that ${∥s_f-s_g∥}_{L^2\left({\varphi }^{\star }\right)}\lesssim \parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{1/2}$. Noting that

$$s_f^2\left(\mathit{d}\right)-s_g^2\left(\mathit{d}\right)=\frac{1}{n}\sum _{i=1}^n\left[f{\left(x_i\right)}^2-g{\left(x_i\right)}^2+2\left(y_i-\overline{\mathit{y}}\right)\left[f\left(x_i\right)-g\left(x_i\right)+\overline{g}\left(\mathit{x}\right)-\overline{f}\left(\mathit{x}\right)\right]\right.\left.+2\left[g\left(x_i\right)\overline{g}\left(\mathit{x}\right)-f\left(x_i\right)\overline{f}\left(\mathit{x}\right)\right]+\overline{f}{(\mathit{x})}^2-\overline{g}{(\mathit{x})}^2\right]$$

we see that, by the triangle inequality and the Cauchy-Schwarz inequality,

$${∥s_f^2-s_g^2∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim \parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}.$$

For $a>0,b>0,|\sqrt{a}-\sqrt{b}|\le \sqrt{|a-b|}$, and so $\left|s_f\left(\mathit{d}\right)-s_g\left(\mathit{d}\right)\right|\le \sqrt{\left|s_f^2\left(\mathit{d}\right)-s_g^2\left(\mathit{d}\right)\right|}$, which implies that ${\left|s_f\left(\mathit{d}\right)-s_g\left(\mathit{d}\right)\right|}^2\le \left|s_f^2\left(\mathit{d}\right)-s_g^2\left(\mathit{d}\right)\right|$, which in turn implies that ${∥s_f-s_g∥}_{L^2\left({\varphi }^{\star }\right)}^2\le {∥s_f^2-s_g^2∥}_{L^1\left({\varphi }^{\star }\right)}$. Combining this with the above and taking square roots of both sides gives the desired bound, namely

$${∥s_f-s_g∥}_{L^2\left({\varphi }^{\star }\right)}\lesssim \parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{1/2}.$$ (S11)

Recalling (S10), we then see that the first term on the right-hand side of (S9) satisfies

$${∥\bar{y}{S}_{T,f}\left[s_f-s_g\right]∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim \parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{1/2}.$$

We now study the second term in (S9). Before beginning our analysis, we note that, for all $\mathit{d}$,

$$1\le 1_{\left\{s_g\left(\mathit{d}\right)\le \epsilon \right\}}+1_{\left\{s_g\left(\mathit{d}\right)>\epsilon \right\}\cap \left\{\left|s_g\left(\mathit{d}\right)-s_f\left(\mathit{d}\right)\right|<\epsilon /2\right\}}+1_{\left\{\left|s_g\left(\mathit{d}\right)-s_f\left(\mathit{d}\right)\right|\ge \epsilon /2\right\}}.$$ (S12)

Combining the above with the triangle inequality, the second term in (S9) bounds as:

$$\begin{gathered} {∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]∥}_{L^1\left({\varphi }^{\star }\right)}\le \\ {∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{\left\{s_g\le \epsilon \right\}}∥}_{L^1\left({\varphi }^{\star }\right)} \\ +{∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{\left\{s_g>\epsilon \right\}\cap \left\{\left|s_f-s_g\right|<\epsilon /2\right\}}∥}_{L^1\left({\varphi }^{\star }\right)} \\ +{∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{\left\{\left|s_g-s_f\right|\ge \epsilon /2\right\}}∥}_{L^1\left({\varphi }^{\star }\right)}. \end{gathered}$$ (S13)

In the above normed quantities, expressions like $1_{\left\{s_g\le \epsilon \right\}}$ should be interpreted as functions, e.g. $1_{\left\{s_g(\cdot )\le \epsilon \right\}}$. By (S3), the first term on the right-hand side bounds as

$${∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{s_g\le \epsilon }∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim \epsilon .$$

For the second term, we start by noting that

$$\begin{array}{ll}& {∥z_f\left(\mathit{d}\right)-z_g\left(\mathit{d}\right)∥}_2\\ & ={∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)}{s_g\left(\mathit{d}\right)s_f\left(\mathit{d}\right)}(\mathit{y}-\stackrel{-}{\mathit{y}})+\frac{1}{s_f\left(\mathit{d}\right)s_g\left(\mathit{d}\right)}\left[s_f\left(\mathit{d}\right)(f-g+\bar{g}-\bar{f})\left(\mathit{x}\right)+\left(s_g-s_f\right)\left(\mathit{d}\right)(f-\bar{f})\left(\mathit{x}\right)\right]∥}_2.\end{array}$$

Using that $(a+b+c{)}^{\kappa }\le a^{\kappa }+b^{\kappa }+c^{\kappa }$ whenever $a,b,c>0$ and $\kappa \in \left(\mathrm{0,1}\right]$, this then implies that

$$\begin{gathered} {∥z_f\left(\mathit{d}\right)-z_g\left(\mathit{d}\right)∥}_2^{\alpha }\le \\ {∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)}{s_g\left(\mathit{d}\right)s_f\left(\mathit{d}\right)}(\mathit{y}-\stackrel{-}{\mathit{y}})∥}_2^{\alpha }+{∥\frac{(f-g+\bar{g}-\bar{f})\left(\mathit{x}\right)}{s_g\left(\mathit{d}\right)}∥}_2^{\alpha } \\ +{∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)(f-\bar{f})\left(\mathit{x}\right)}{s_f\left(\mathit{d}\right)s_g\left(\mathit{d}\right)}∥}_2^{\alpha }, \end{gathered}$$

where above $\alpha$ is the exponent from the Hölder condition satisfied by ${𝒮}^0$. Combining the Hölder condition with the above, we then see that

$$\begin{array}{ll}\left|S_{T,f}\left(\mathit{d},x_0\right)-S_{T,g}\left(\mathit{d},x_0\right)\right|\lesssim & {∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)}{s_g\left(\mathit{d}\right)s_f\left(\mathit{d}\right)}(\mathit{y}-\stackrel{-}{\mathit{y}})∥}_2^{\alpha }+{∥\frac{(f-g+\bar{g}-\bar{f})\left(\mathit{x}\right)}{s_g\left(\mathit{d}\right)}∥}_2^{\alpha }\\ & +{∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)(f-\bar{f})\left(\mathit{x}\right)}{s_f\left(\mathit{d}\right)s_g\left(\mathit{d}\right)}∥}_2^{\alpha }.\end{array}$$

Multiplying both sides by $\left|\stackrel{-}{\mathit{y}}{s}_g\left(\mathit{d}\right)1_{\left\{s_g\left(\mathit{d}\right)>\epsilon ,\left|\left(s_f-s_g\right)\left(\mathit{d}\right)\right|<\epsilon /2\right\}}\right|$, we then see that

$$\begin{array}{ll}& \left|\bar{y}{s}_g\left(\mathit{d}\right)\left[S_{T,f}\left(\mathit{d},x_0\right)-S_{T,g}\left(\mathit{d},x_0\right)\right]1_{\left\{s_g\left(\mathit{d}\right)>\epsilon ,\left|\left(s_f-s_g\right)\left(\mathit{d}\right)\right|<\epsilon /2\right\}}\right|\\ & \lesssim \left|\stackrel{-}{\mathit{y}}\right|s_g\left(\mathit{d}\right){∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)}{s_g\left(\mathit{d}\right)s_f\left(\mathit{d}\right)}(\mathit{y}-\stackrel{-}{\mathit{y}})∥}_2^{\alpha }{1}_{\left\{s_g\left(\mathit{d}\right)>\epsilon ,\left|\left(s_f-s_g\right)\left(\mathit{d}\right)\right|<\epsilon /2\right\}}\\ & +\left|\stackrel{-}{\mathit{y}}\right|s_g\left(\mathit{d}\right){∥\frac{(f-g+\bar{g}-\bar{f})\left(\mathit{x}\right)}{s_g\left(\mathit{d}\right)}∥}_2^{\alpha }{1}_{\left\{s_g\left(\mathit{d}\right)>\epsilon ,\left|\left(s_f-s_g\right)\left(\mathit{d}\right)\right|<\epsilon /2\right\}}\\ & +\left|\stackrel{-}{\mathit{y}}\right|s_g\left(\mathit{d}\right){∥\frac{\left(s_g-s_f\right)\left(\mathit{d}\right)(f-\bar{f})\left(\mathit{x}\right)}{s_f\left(\mathit{d}\right)s_g\left(\mathit{d}\right)}∥}_2^{\alpha }{1}_{\left\{s_g\left(\mathit{d}\right)>\epsilon ,\left|\left(s_f-s_g\right)\left(\mathit{d}\right)\right|<\epsilon /2\right\}}\\ & \lesssim {\epsilon }^{-\alpha }\left|\stackrel{-}{\mathit{y}}\right|s_g(\mathit{d}{)}^{1-\alpha }\parallel \mathit{y}-\stackrel{-}{\mathit{y}}{\parallel }_2^{\alpha }{\left|\left(s_g-s_f\right)\left(\mathit{d}\right)\right|}^{\alpha }\\ & +\left|\stackrel{-}{\mathit{y}}\right|s_g(\mathit{d}{)}^{1-\alpha }\parallel (f-g+\bar{g}-\bar{f}\left)\right(\mathit{x}){\parallel }_2^{\alpha }\\ & +{\epsilon }^{-\alpha }\left|\stackrel{-}{\mathit{y}}\right|s_g^{1-\alpha }\parallel (f-\bar{f})\left(\mathit{x}\right){\parallel }_2^{\alpha }{\left|\left(s_g-s_f\right)\left(\mathit{d}\right)\right|}^{\alpha }.\end{array}$$

The inequality above remains true if we integrate both sides against ${\varphi }^{\star }$. The resulting three terms on the right-hand side can be bounded using Hölder’s inequality. In particular, we have that

$$\begin{array}{l}\begin{array}{c}{\epsilon }^{-\alpha }{\Vert \left|\overline{y}{|}^{\alpha }\Vert y-\overline{y}\Vert {}_2^{\alpha }{\left|s_g-s_f\right|}^{\alpha }\right|\overline{y}{|}^{1-\alpha }{s}_g^{1-\alpha }\Vert }_{L^1\left({\varphi }^{\star }\right)}\le {\epsilon }^{-\alpha }{\Vert \overline{y}\Vert y-\overline{y}\Vert {}_2\left(s_g-s_f\right)\Vert }_{L^1\left({\varphi }^{\star }\right)}^{\alpha }{\Vert \overline{y}{s}_g\Vert }_{L^1\left({\varphi }^{\star }\right)}^{1-\alpha }\end{array}\\ \begin{array}{c}\lesssim {\epsilon }^{-\alpha }{\Vert f-g\Vert }_{L^2\left({\varphi }^p\right)}^{\alpha /2},\end{array}\\ \begin{array}{c}{\Vert \overline{y}{s}_g^{1-\alpha }{\Vert \left(f-g+\overline{g}-\overline{f}\right)\left(x\right)\Vert }_2^{\alpha }\Vert }_{L^1\left({\varphi }^{\star }\right)}\le {\Vert \overline{y}{s}_g\Vert }_{L^1\left({\varphi }^{\star }\right)}^{1-\alpha }{\Vert \overline{y}\Vert \left(f-g+\overline{g}-\overline{f}\right)\left(x\right)\Vert {}_2\Vert }_{L^1\left({\varphi }^{\star }\right)}^{\alpha }\end{array}\\ \lesssim \parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{\alpha /2},\\ {\epsilon }^{-\alpha }{\Vert \overline{y}{s}_g^{1-\alpha }{\Vert \left(f-\overline{f}\right)\left(x\right)\Vert }_2^{\alpha }\Vert s_g-s_f\Vert {}^{\alpha }\Vert }_{L^1\left({\varphi }^{\star }\right)}\le {\epsilon }^{-\alpha }{\Vert \overline{y}{s}_g\Vert }_{L^1\left({\varphi }^{\star }\right)}^{1-\alpha }{\Vert {\Vert \left(f-\overline{f}\right)\left(x\right)\Vert }_2\left|s_g-s_f\right|\Vert }_{L^1\left({\varphi }^{\star }\right)}^{\alpha }\\ \lesssim {\epsilon }^{-\alpha }{\Vert f-g\Vert }_{L^2\left({\varphi }^p\right)}^{\alpha /2}.\end{array}$$

Hence, we have shown that the second term on the right-hand side of (S13) satisfies

$${∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{s_g>\epsilon ,\left|s_g-s_f\right|<\epsilon /2}∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim {\epsilon }^{-\alpha }\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{\alpha /2}.$$

We now study the third term on the right-hand side of (S13). We start by noting that, by Markov’s inequality and (S11),

$$\begin{array}{ll}{P}_{{\varphi }^{\star }}\left(\left|s_g\left(\mathit{D}\right)-s_f\left(\mathit{D}\right)\right|\ge \frac{\epsilon }{2}\right)=& P\left({\left|s_g\left(\mathit{D}\right)-s_f\left(\mathit{D}\right)\right|}^2\ge \frac{{\epsilon }^2}{4}\right)\\ & \le \frac{4}{{\epsilon }^2}{∥s_f-s_g∥}_{L^2\left({\varphi }^{\star }\right)}^2\lesssim {\epsilon }^{-2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}.\end{array}$$

Moreover, by the generalized Hölder’s inequality with parameters (4, 2, ∞, 4), we see that

$$\begin{array}{ll}& {∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]1_{\left\{\left|s_g-s_f\right|\ge \epsilon /2\right\}}∥}_{L^1\left({\varphi }^{\star }\right)}\\ & \le \parallel \bar{y}{\parallel }_{L^4\left({\varphi }^{\star }\right)}{∥s_g∥}_{L^2\left({\varphi }^{\star }\right)}{∥S_{T,f}-S_{T,g}∥}_{L^{\mathrm{\infty }}\left({\varphi }^{\star }\right)}{∥1_{\left\{\left|s_g-s_f\right|\ge \epsilon /2\right\}}∥}_{L^4\left({\varphi }^{\star }\right)}\\ & \le 2\parallel \bar{y}{\parallel }_{L^4\left({\varphi }^{\star }\right)}{∥s_g∥}_{L^2\left({\varphi }^{\star }\right)}{B}_{{𝒮}_0}P{\left(\left|s_g-s_f\right|\ge \epsilon /2\right)}^{1/4}\\ & \lesssim {\epsilon }^{-1/2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{1/4}\end{array}.$$

Combining our bounds for the three terms on the right-hand side of (S13), we have shown that

$${∥\bar{y}{s}_g\left[S_{T,f}-S_{T,g}\right]∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim \epsilon +{\epsilon }^{-\alpha }\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{\alpha /2}+{\epsilon }^{-1/2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}^{1/4}.$$ (S14)

The above provides our bound for the (S6) term from the main expression.

We now study the (S7) term from the main expression. We start by decomposing this term as

$$\int \left[s_f^2{S}_{T,f}^2-s_g^2{S}_{T,g}^2\right]d{\varphi }^{\star }=\int S_{T,f}^2\left(s_f^2-s_g^2\right)d{\varphi }^{\star }+\int s_g^2\left[S_{T,f}^2-S_{T,g}^2\right]d{\varphi }^{\star },$$

where for brevity, we have suppressed the dependence on $s_f,s_g,S_{T,f}$, and $S_{T,g}$ on their arguments. By (S11), the first term is bounded by a constant times $\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}$. For the second term, we note that the uniform bound on $S_{T,f}$ and $S_{T,g}$ shows that

$${∥s_g^2\left[S_{T,f}^2-S_{T,g}^2\right]∥}_{L^1\left({\varphi }^{\star }\right)}\lesssim {∥s_g^2\left[S_{T,f}-S_{T,g}\right]∥}_{L^1\left({\varphi }^{\star }\right)}$$

Similarly to as we did when studying (S6), we can use (S12) and the triangle inequality to write

$${∥s_g^2\left[S_{T,f}-S_{T,g}\right]∥}_{L^1\left({\varphi }^{\star }\right)}\le {∥s_g^2\left[S_{T,f}-S_{T,g}\right]1_{\left\{s_g\le \epsilon \right\}}∥}_{L^1\left({\varphi }^{\star }\right)}+{∥s_g^2\left[S_{T,f}-S_{T,g}\right]1_{\left\{s_g>\epsilon ,\left|s_f-s_g\right|<\epsilon /2\right\}}∥}_{L^1\left({\varphi }^{\star }\right)}+{∥s_g^2\left[S_{T,f}-S_{T,g}\right]1_{\left\{\left|s_g-s_f\right|\ge \epsilon /2\right\}}∥}_{L^1\left({\varphi }^{\star }\right)}.$$

The first term on the right upper bounds by a constant times ${\epsilon }^2$. The analyses of the second and third terms are similar to the analysis of the analogous terms from (S6). A minor difference between the study of these terms and that of (S6) is that, when applying Hölder’s inequality to separate the terms in each normed expression, we use (v) to ensure that ${∥s_g∥}_{L^{q^{\prime }}\left({\varphi }^{\star }\right)}<\mathrm{\infty }$ for some $q^{\prime }>2$. This helps us deal with the fact that $s_g^2$, rather than $s_g$, appears in the normed expressions above. Due to the similarity of the arguments to those given for (S6), the calculations for controlling the second and third terms are omitted. After the relevant calculations, we end up showing that, like (S6), (S7) is bounded by a constant times the right-hand side of (S14).

To study (S8) from the main expression, we rewrite the integral as

$$\begin{array}{ll}& \int [\left.\left(\bar{f}\left(\mathit{x}\right)-f\left(x_0\right)\right)s_f\left(\mathit{d}\right)S_{T,f}\left(\mathit{d},x_0\right)-\left(\bar{g}\left(\mathit{x}\right)-g\left(x_0\right)\right)s_g\left(\mathit{d}\right)S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star }\\ & =\int s_f\left(\mathit{d}\right)S_{T,f}\left(\mathit{d},x_0\right)\left[\bar{f}\left(\mathit{x}\right)-\bar{g}\left(\mathit{x}\right)+f\left(x_0\right)-g\left(x_0\right)\right]d{\varphi }^{\star }\\ & +\int S_{T,f}\left(\mathit{d},x_0\right)\left(\bar{g}\left(\mathit{x}\right)+g\left(x_0\right)\right)\left(s_f-s_g\right)\left(\mathit{d}\right)d{\varphi }^{\star }\\ & +\int s_g\left(\mathit{d}\right)\left(\bar{g}\left(\mathit{x}\right)+g\left(x_0\right)\right)\left[S_{T,f}\left(\mathit{d},x_0\right)-S_{T,g}\left(\mathit{d},x_0\right)\right]d{\varphi }^{\star }.\end{array}$$

Each of the terms in the expansion can be bounded using similar techniques to those used earlier in this proof. Combining our bounds on (S4) through (S8), we see that

$$\left|R\left(T,P_f\right)-R\left(T,P_g\right)\right|\lesssim {\epsilon }^{-2}\parallel f-g{\parallel }_{L^2\left({\varphi }^p\right)}+\epsilon .$$

As $f,g$ were arbitrary, we see that, for any sequence $\left\{f_k\right\}$ in $ℱ$ such that $f_k\to f$ in $L^2\left({\varphi }^p\right)$ as $k\to \mathrm{\infty }$, it holds that $\mathrm{l}\mathrm{i}\mathrm{m}{\mathrm{s}\mathrm{u}\mathrm{p}}_k \left|R\left(T,P_{f_k}\right)-R\left(T,P_f\right)\right|\lesssim \epsilon$. As $\epsilon \in \left(\mathrm{0,1}\right)$ was arbitrary, this shows that $R\left(T,P_{f_k}\right)\to R\left(T,P_f\right)$ as $k\to \mathrm{\infty }$. Hence, $P\mapsto R(T,P)$ is continuous in this example. □

D. Further details on numerical experiments

D.1. Meta-Learning Benchmarks

We implemented MAML via the learn2learn python package (Arnold et al., 2020), which in turn makes use of the Torchmeta package (Deleu et al., 2019) when generating the sinusoid functions. We trained MAML on a total of 10⁶ datasets with a batch size of 25 datasets and used the same learning rates and number of adaptation steps as were used in learn2learn/examples/maml_sine.py. We tried two network architectures, namely the same two-hidden layer perceptron architecture that was used in the sinusoid experiments in Finn et al. (2017) and a larger network whose hidden layers contained the same number of nodes (40) but that used a total of five hidden layers. For each of the three regression settings considered (sinusoid, Gaussian process with a 1-dimensional feature, and Gaussian process with a 5-dimensional feature), we reported results for the architecture that performed best across the sample sizes considered. This ended up corresponding to reporting results for the smaller network architecture across all three settings.

For the Gaussian process example with a 1-dimensional feature, we used the implementation of CNPs provided by Jiang (2021), which corresponds to a Pytorch implementation of the code from Garnelo et al. (2018). We also modified this code so that it could apply to the sinusoidal regression example and the Gaussian process example where the feature is 5-dimensional. The CNPs were updated over the same number of iterations and using the same batch size as AMC, namely 10⁶ and 25, respectively. We tried two network architectures for the CNPs, namely the same architecture as was used in Garnelo et al. (2018), with the input size modified in one of the Gaussian process settings to account for the 5-dimensional feature, and also a deeper architecture that has a similar number of hidden layers as does the architecture used for AMC. In particular, the encoder and decoder in this larger architecture each had nine hidden layers consisting of 100 nodes. Similar to as we did for MAML, for each of the three regression settings considered, we reported results for the architecture that performed best across the sample sizes considered. This corresponded to reporting CNP results for the smaller architecture for the Gaussian process with a 5-dimensional feature, and the larger architecture for the Gaussian process with a 1-dimensional feature and the sinusoidal regression.

D.2. Comparing to Analytically-Derived Estimators with Known Theoretical Performance Guarantees

D.2.1. Preliminaries

We now introduce notation that will be useful for defining ${\Gamma }_1$ in the two examples. In both examples, all priors in ${\Gamma }_1$ imply the same prior ${\Pi }_X$ over the distribution $P_X$ of the features. This prior ${\Pi }_X$ imposes that the $\Sigma$ indexing $P_X$ is equal in distribution to $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left(W^{-1}\right)}^{-1/2}{W}^{-1}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left(W^{-1}\right)}^{-1/2}$, where $W$ is a $p\times p$ matrix drawn from a Wishart distribution with scale matrix $2{\mathrm{I}\mathrm{d}}_p$ and 20 degrees of freedom, and $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(W^{-1}\right)$ denotes a matrix with the same diagonal as $W^{-1}$ and zero in all other entries. The expression for $\Sigma$ normalizes by $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}{\left(W^{-1}\right)}^{-1/2}$ to ensure that the diagonal of $\Sigma$ is equal to $1_p$, which we require of distributions in ${𝒫}_X$. We let ${\Gamma }_{\mu }$ be a collection of Markov kernels $\kappa :{𝒫}_X\to ℛ$, so that, for each $\kappa$ and $P_X\in {𝒫}_X,\kappa \left(\cdot ,P_X\right)$ is a distribution on $ℛ$. The collections ${\Gamma }_{\mu }$ differ in the two examples, and will be presented in the coming subsections. Let $\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(ℬ\right)$ denote a uniform distribution over the permutations in $ℬ$. For each $\kappa \in {\Gamma }_{\mu }$, we let ${\Pi }_{\kappa }$ represent a prior on ${𝒫}_1$ from which a draw $P$ can be generated by sampling $P_X~{\Pi }_X,\mu \mid P_X~\kappa \left(\cdot ,P_X\right)$, and $B\mid P_X,\mu ~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(ℬ\right)$, and subsequently returning the distribution of $\left(X,\mu \left(BX\right)+{ϵ}_P\right)$, where $X~P_X$ and ${ϵ}_P~N\left(\mathrm{0,1}\right)$ are independent. We let ${\Gamma }_1≔\left\{{\Pi }_{\kappa }:\kappa \in {\Gamma }_{\mu }\right\}$. For a general class of estimators $𝒯$, enforcing that each draw $P$ has a regression function ${\mu }_P$ of the form $x\mapsto \mu \left(Bx\right)$ for some permutation $B$ is useful because it allows us to restrict the class ${\Gamma }_{\mu }$ so that each function in this class only depends on the first $\mathfrak{s}$ coordinates of the input, while yielding a regression function ${\mu }_P$ that may depend on any arbitrary collection of $\mathfrak{s}$ out of the $p$ total coordinates. For the equivariant class that we consider (Algorithm 2), enforcing this turns out to be unnecessary - the invariance of functions in $𝒯$ to permutations of the features implies that the Bayes risk of each $T\in 𝒯$ remains unchanged if the random variable $B$ defining ${\Pi }_{\kappa }\in {\Gamma }_1$ is replaced by a degenerate random variable that is always equal to the identity matrix. Nonetheless, allowing $B$ to be a random draw from $\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(ℬ\right)$ allows us to ensure that our implied collection of priors $\Gamma$ satisfies P1, P2, and P3, thereby making the implied $\Gamma$ compatible with the preservation conditions imposed in Section 2.

Figure S5:

Bayesian standardized MSE $\left(E_{\Pi }\left[R\right(T,P\left)\right]\right.$, where $R$ is defined in Eq. 1) of the five meta-learning algorithms considered in the sinusoidal regression example when the feature $x$ or the outcome $y$ is scaled down by a multiplicative factor (left two columns) or when $x$ or $y$ is shifted by an additive factor (right two columns). For reference, the numbers reported in Table 1 in the main text are equal to the standardized MSE reported on the far-left side of each facet times the variance of the error (0.09). The three equivariant procedures (MAML-Eq, CNP-Eq, and AMC) have constant standardized MSE under the shifts and rescalings considered. The non-equivariant procedures, namely MAML and CNPs, are sensitive even to small shifts or rescalings of $x$, and CNPs are also sensitive to small shifts in $y$.

We now use the notation of Kingma and Ba (2014) to detail the hyperparameters that we used. In all settings, we set $\left({\beta }_2,ϵ\right)=\left(0.999,{10}^{-8}\right)$. Whenever we were updating the prior network, we set the momentum parameter ${\beta }_1$ to 0, and whenever we were updating the estimator network, we set the momentum parameter to 0.25. The parameter $\alpha$ differed across settings. In the sparse linear regression setting with $\mathfrak{s}=1$, we found that choosing $\alpha$ small helped to improve stability. Specifically, we let $\alpha =0.0002$ when updating both the estimator and prior networks. In the sparse linear regression setting with $\mathfrak{s}=5$, we used the more commonly chosen parameter setting of $\alpha =0.001$ for both networks. In the FLAM example, we chose $\alpha =0.001$ and $\alpha =0.005$ for the estimator and prior networks, respectively.

The learning rates were of the estimator and prior networks were decayed at rates $t^{-0.15}$ and $t^{-0.25}$, respectively. Such two-timescale learning rate strategies have proven to be effective in stabilizing the optimization problem pursued by generative adversarial networks (Heusel et al., 2017). As noted in Fiez et al. (2019), using two-timescale strategies can cause the optimization problem to converge to a differential Stackelberg, rather than a differential Nash, equilibrium. Indeed, under some conditions, the two-timescale strategy that we use is expected to converge to a differential Stackelberg equilibrium in the hierarchical two-player game where a prior $\Pi$ is first selected from $\Gamma$, and then an estimator $T$ is selected from $𝒯$ to perform well against $\Pi$. An optimal prior ${\Pi }^{\star }$ in this game is called $\Gamma$-least favorable, in the sense that this prior maximizes ${\mathrm{i}\mathrm{n}\mathrm{f}}_{T\in 𝒯} r(T,\cdot )$ over $\Gamma$. For a given $\Gamma$-least favorable prior ${\Pi }^{\star }$, an optimal estimator $T^{\star }$ in this game is a Bayes estimator against ${\Pi }^{\star }$, that is, an estimator that minimizes $r\left(\cdot ,{\Pi }^{\star }\right)$ over $𝒯$. This $T^{\star }$ may not necessarily be a $\Gamma$-minimax strategy, that is, $T^{\star }$ may not minimize ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\Pi \in \Gamma } r(\cdot ,\Pi )$ over $𝒯$. Nevertheless, we note that, under appropriate conditions, the two notions of optimality necessarily agree. Though such a theoretical guarantee is not likely to hold in our experiments given the neural network parameterizations that we use, we elected to use this two-timescale strategy because of the improvements in stability that we saw.

In all settings, the prior and estimator were updated over 10⁶ iterations using batches of 100 datasets. For each dataset, performance is evaluated at 100 values of $x_0$.

D.2.2. Sparse linear regression

We now introduce notation that will be useful for presenting the collection ${\Gamma }_{\mu }$ in the sparse linear regression example. For a function $G:\mathbb{R}\to \mathbb{R}$ and a distribution $P_X\in {𝒫}_X$, we let ${\kappa }_G\left(\cdot ,P_X\right)$ be equal to the distribution of

$$x\mapsto {\left(U_0\frac{\left(e^{G\left(U_1\right)},\dots ,e^{G\left(U_{\mathfrak{s}}\right)},0,\dots ,0\right)}{\sum _{j=1}^{\mathfrak{s}} e^{G\left(U_j\right)}}\right)}^{\mathrm{\top }}x,$$

where $U_0~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}(-\mathrm{5,5})$ and $\left(U_1,\dots ,U_{\mathfrak{s}}\right)~N\left(0_{\mathfrak{s}},{\mathrm{I}\mathrm{d}}_{\mathfrak{s}}\right)$ are drawn independently. Notably, here ${\kappa }_G\left(\cdot ,P_X\right)$ does not depend on $P_X$. We let ${\Gamma }_{\mu }≔\left\{{\kappa }_G:G\in 𝒢\right\}$, where $𝒢$ takes different values when $\mathfrak{s}=1$ and when $\mathfrak{s}=5$. When $\mathfrak{s}=1,𝒢$ consists of all four-hidden layer perceptrons with identity output activation, where each hidden layer consists of forty leaky ReLU units. When $\mathfrak{s}=5,𝒢$ consists of all four-hidden layer neural networks with identity output activation, but in this case each layer is a multi-input-output channel equivariant layer as described in Eq. 22 of Zaheer et al. (2017). Each hidden layer is again equipped with a ReLU activation function. The output of each such network is equivariant to permutations of the $\mathfrak{s}=5$ inputs.

In each sparse linear regression setting considered, we initialized the estimator network by pretraining for 5,000 iterations against the initial fixed prior network. After these 5,000 iterations, we then began to adversarially update the prior network against the estimator network.

Five thousand Monte Carlo replicates were used to obtain the performance estimates in Table 2.

D.2.3. Fused lasso additive model

When discussing the FLAM example, we will write $x_j$ to denote the $j^{\text{th}}$ feature, that is, we denote a generic $x\in 𝒳$ by $x=\left(x_1,\dots ,x_p\right)$. We emphasize this to avoid any notational confusion with the fact that, elsewhere in the text, $X_i\in 𝒳$ is used to denote the random variable corresponding to the $i^{\text{th}}$ observation.

In the FLAM example, each prior ${\kappa }_G$ in ${\Gamma }_{\mu }$ is indexed by a function $G:{\mathbb{R}}^{\mathfrak{s}+2}\to [0,\mathrm{\infty }{)}^{\mathfrak{s}}$ belonging to the collection of four-hidden layer perceptrons with identity output activation, where each hidden layer consists of forty leaky ReLU units. Specifically, ${\kappa }_G\left(\cdot ,P_X\right)$ is a distribution over generalized additive models $x\mapsto {\sum }_{j=1}^p {\mu }_j\left(x_j\right)$ for which each component ${\mu }_j$ is piecewise-constant and changes values at most 500 times. To obtain a draw ${\mu }_P$ from ${\kappa }_G\left(\cdot ,P_X\right)$, we can first draw 500 iid observations from $P_X$ and store these observations in the matrix $\tilde{\mathit{X}}$. Each component ${\mu }_j$ can only have a jump at the 500 points in ${\tilde{\mathit{X}}}_{\mathrm{*}j}$. The magnitude of each jump is defined using the function $G$ and the sign of the jump is defined uniformly at random. More specifically, these increments are defined based on the independent sources of noise $\left(H_{jk}:j=1,\dots ,p;k=\right.1,\dots ,500)$, which is an iid collection of Rademacher random variables, and $\left(U_k:k=1,\dots ,500\right)$, which is an iid collection of $N\left(0_{\mathfrak{s}+2},{\mathrm{I}\mathrm{d}}_{\mathfrak{s}+2}\right)$ random variables. The component ${\mu }_j$ is chosen to be proportional to the function $f_j\left(x_j\right)={\sum }_{k=1}^{500} H_{jk}G{\left(U_k\right)}_{j}I\left\{x_j\ge {\tilde{\mathit{X}}}_{kj}\right\}$. The proportionality constant $c≔{\sum }_{j=1}^p {\sum }_{k=1}^{500} G{\left(U_k\right)}_j$ is defined so that the function ${\mu }_P\left(x\right)=c^{-1}{\sum }_{j=1}^p f_j\left(x_j\right)$ saturates the constraint $\parallel v\left(\mu \right){\parallel }_1\le M$ that is imposed by $ℛ$. To recap, the random draw ${\mu }_P$ from ${\kappa }_G\left(\cdot ,P_X\right)$ can be obtained by independently drawing $\tilde{\mathit{X}},\left(H_{j,k}:j,k\right)$, and $\left(U_k:k\right)$, and subsequently following the steps described above to define the corresponding proportionality constant $c$ and components $f_j,j=1,\dots ,p$.

We evaluated the performance of the learned prediction procedures using a variant of the simulation scenarios 1–4 from the paper that introduced FLAM (Fig. 2 in Petersen et al., 2016). As presented in that work, the four scenarios have $p$ independent Unif (−2.5, 2.5) features, with the components corresponding to ${\mathfrak{s}}_0=4$ of these features being nonzero. These scenarios offer a range of smoothness settings, with scenarios 1–4 enforcing that the components be (1) piecewise constant, (2) smooth, (3) a mix of piecewise constant and smooth functions, and (4) constant in some areas of its domain and highly variable in others. To evaluate our procedures trained with ${∥v\left({\mu }_P\right)∥}_0\le 5$, we used the $\mathrm{R}$ function sim. data in the flam package (Petersen, 2018) to generate training data from the scenarios in Petersen et al. (2016) with $p=10$ features. We then generated new outcomes by rescaling the regression function by a positive multiplicative constant so that ${∥v\left({\mu }_P\right)∥}_1=10$, and subsequently added standard Gaussian noise. To evaluate our procedures trained at sparsity level $\mathfrak{s}=1$ in a given scenario, we defined a prior over the regression function that first randomly selects one of the four signal components, then rescales this component so that it has total variation equal to 10, and then sets all other components equal to zero. Outcomes were generated by adding Gaussian noise to the sampled regression function. We compared our approach to the FLAM method as implemented in the flam package when, in the notation of Petersen et al. (2016), $\alpha =1$ and $\lambda$ was chosen numerically to enforce that the resulting regression function estimate $\hat{\mu }$ satisfied $\parallel v\left(\hat{\mu }\right){\parallel }_1\approx 10$. Choosing $\lambda$ in this fashion is reasonable in light of the fact that ${∥v\left({\mu }_P\right)∥}_1=10$ for all settings considered.

Two thousand Monte Carlo replicates were used to obtain the performance estimates in Table 3.

E. Additional details and results for data experiments

E.1. Datasets

We start by describing the six datasets that we considered that are available through the UCI Machine Learning Repository (Dua and Graff, 2017). The first dataset (“abalone”) contains information on 4177 abalones. The objective is to predict their age based on 7 features, namely length, diameter, height, whole weight, shucked weight, viscera weight, and shell weight (Nash et al., 1994). The second dataset (“airfoil”) is from the National Aeronautics and Space Administration (NASA) that contains information on 1,503 airfoils at various wind tunnel speeds and angles of attack (Brooks et al., 1989). The objective is to estimate the scaled sound level in decibels. Five features are available, namely frequency, angle of attack, chord length, free-stream velocity, and suction side displacement thickness. The third dataset (“fish”) was originally used to develop quantitative structure-activity relationship (QSAR) models to predict acute aquatic toxicity towards the fathead minnow. This dataset contains 908 total observations, each of which corresponds to a distinct chemical. The outcome is the LC₅₀ for that chemical, which represents the concentration of the chemical that is lethal for 50% of test fish over 96 hours. Six features that describe the molecular characteristics of the chemical are available — see the UCI Machine Learning Repository and Cassotti et al. (2015) for details. The fourth and fifth datasets contain information on 1,599 red wines (“wine-red”) and 4,898 white wines (“wine-white”) (Cortez et al., 2009). The objective is to predict wine quality score based on 11 available features — see the UCI Machine Learning Repository and Cassotti et al. (2015) for details. The sixth dataset (“yacht”) contains information on 308 sailing yachts. The objective is to learn to predict a ship’s performance in terms of residuary resistance. Six features describing a ship’s dimensions and velocity are available, namely: the longitudinal position of the center of buoyancy, the prismatic coefficient, the length-displacement ratio, the beam-draught ratio, the length-beam ratio, and the Froude number. See Gerritsma et al. (1981) for more information on these features.

The seventh and eighth of our datasets that we considered were used to illustrate regression procedures in James et al. (2013). They are available through the ISLR R package (James et al., 2017). One of these datasets (“college”) consists of information on 777 colleges in the United States. The objective is to predict out-of-state tuition based on 16 available continuous features. The second of these datasets (“hitters”) contains information on 322 baseball players. The objective is to predict salary based on the 16 available continuous features. The ninth dataset (“LAozone”) was used to illustrate regression procedures in (Friedman, 2001). It consists of 330 daily meteorological measurements in the Los Angeles basin in 1976. The objective is to predict ozone levels based on 9 available features. The final dataset that we considered (“happiness”) was used in the paper that introduced the FLAM to illustrate the performance of the method (Petersen et al., 2016). This dataset consists of information about 109 countries. The objective is to predict the national happiness level via 12 country-level features.

E.2. Additional results for data experiments

Table S5 displays the cross-validated MSEs across the ten datasets in numerical form. Figure S6 shows the performance of the individual linear algorithms considered at different sparsity levels, and Figure S7 shows the same results but for the stacking algorithms.

Table S5:

Cross-validated MSEs on the ten datasets. The first 5 datasets had the same number of features the same as were used during meta-training (10), whereas the others had fewer. For each of the three categories (linear estimators, FLAM estimators, and stacked estimators) and each dataset, the algorithm with the lowest Monte Carlo MSE is emphasized in bold. There was no clear ordering between the performance of AMC Linear and the existing estimators (OLS and lasso). AMC FLAM tended to outperform FLAM when the number of features was the same as were used during meta-training, and be slightly outperformed otherwise. When the number of features was the same as were used during meta-training, stacking the existing and AMC estimators consistently outperformed all other approaches. When there were fewer features than were used during meta-training, stacking all available learners performed similarly to stacking only the existing algorithms and still outperformed all individual learners.

	Features	OLS	Lasso	AMC Linear (ours)	FLAM	AMC FLAM (ours)	Stacked Existing	Stacked AMC (ours)	Stacked Both (ours)
college	10	0.414	0.397	0.377	0.392	0.395	0.358	0.354	0.348
happiness	10	0.270	0.277	0.275	0.315	0.311	0.280	0.261	0.256
hitters	10	0.667	0.660	0.662	0.626	0.619	0.602	0.615	0.585
wine-red	10	0.768	0.737	0.746	0.826	0.776	0.737	0.737	0.731
wine-white	10	0.833	0.814	0.824	0.899	0.860	0.809	0.815	0.802
LAozone	9	0.341	0.335	0.337	0.335	0.367	0.310	0.320	0.309
abalone	7	0.559	0.546	0.540	0.709	0.675	0.539	0.538	0.537
fish	6	0.471	0.475	0.480	0.544	0.554	0.464	0.476	0.468
yacht	6	0.381	0.372	0.350	0.019	0.035	0.015	0.029	0.015
airfoil	5	0.524	0.525	0.528	0.617	0.701	0.516	0.523	0.520

F. Performance of symmetrized estimators in experiments

We now present the additional experimental results that we alluded to in Section 8. These results were obtained by symmetrizing the meta-learned AMC100 and AMC500 estimators whose performance was reported in Section 5. In particular, we symmetrized a given AMC estimator $T$ as

$$T^{\mathrm{s}\mathrm{y}\mathrm{m}}(\mathit{x},\mathit{y})\left(x_0\right)≔\frac{1}{2}\left[T(\mathit{x},\mathit{y})-T(\mathit{x},-\mathit{y})\left(x_0\right)\right].$$

When reporting our experimental results, we refer to the symmetrized estimator derived from the meta-learned AMC100 and AMC500 estimators as ‘symmetrized AMC100' and ‘symmetrized AMC500', respectively. We emphasize that these symmetrized estimators are derived directly from the AMC100 and AMC500 fits that we reported in Section 5 – we did not rerun our AMC meta-learning algorithm to obtain these estimators.

Figure S6:

Performance of OLS, lasso, and AMC Linear at different sparsity levels. For each training-validation split of the data, between 1 and $q$ features are selected at random from the original dataset (x-axis), where $q$ is the minimum of 10 and the total number of features in the dataset, and Gaussian noise features are then added so that there are 10 total features. Therefore, the signal is expected to become denser and stronger as the x-axis value increases. AMC Linear consistently outperformed OLS and performed similarly to or better than lasso in most settings (54% of all sparsity-dataset pairs).

Table S6 reports the results for the linear regression example. In many settings, the two approaches performed similarly. However, in the sparse setting, the improvements that resulted from symmetrization sometimes resulted in the MSE being cut in half. In one setting (dense, interior, $n=100$), AMC100 outperformed symmetrized AMC100 slightly – though not deducible from the table, we note here that the difference in MSE in this case was less than 0.003, and it seems likely that this discrepancy is a result of Monte Carlo error. Table S6 reports the results for the fused lasso additive model example. Symmetrization led to a reduction in MSE in most settings. In all other settings, the MSE remained unchanged.

Figure S7:

Performance of the three stacking algorithms at different sparsity levels. For each training-validation split of the data, between 1 and $q$ features are selected at random from the original dataset (x-axis), where $q$ is the minimum of 10 and the total number of features in the dataset, and Gaussian noise features are then added so that there are 10 total features. Therefore, the signal is expected to become denser and stronger as the x-axis value increases. Though all algorithms performed similarly, the stacking algorithm that combined all available algorithms (Stacked Both) performed slightly better than the others in a majority of the settings (53% of all sparsity-dataset pairs), and Stacked AMC performed best in most other settings (39% of all sparsity-dataset pairs).

Table S6:

MSEs based on datasets of size $n$ in the linear regression settings. All Monte Carlo standard errors are less than 0.001. Symmetrized AMC100 entries appear in bold when they had lower MSE (rounded to the nearest hundredth) than the corresponding AMC100 entry, and vice versa. Similarly, symmetrized AMC500 entries appear in bold when they had lower MSE than the corresponding AMC500 entry, and vice versa.

(a) Sparse signal
	Boundary		Interior
	n=100	500	100	500
OLS	0.12	0.02	0.12	0.02
Lasso	0.06	0.01	0.06	0.01
AMC100 (ours)	0.02	<0.01	0.11	0.09
Symmetrized AMC100 (ours)	0.02	<0.01	0.06	0.04
AMC500 (ours)	0.02	<0.01	0.07	0.04
Symmetrized AMC500 (ours)	0.02	<0.01	0.06	0.03
(b) Dense signal
	Boundary		Interior
	n=100	500	100	500
OLS	0.13	0.02	0.13	0.02
Lasso	0.11	0.02	0.09	0.02
AMC100 (ours)	0.10	0.04	0.08	0.02
Symmetrized AMC100 (ours)	0.09	0.03	0.09	0.02
AMC500 (ours)	0.09	0.02	0.09	0.02
Symmetrized AMC500 (ours)	0.09	0.02	0.09	0.02

Table S7:

MSEs based on datasets of size $n$ in the FLAM settings. The Monte Carlo standard errors for the MSEs of FLAM and (symmetrized) AMC are all less than 0.04 and 0.01, respectively. Symmetrized AMC100 entries appear in bold when they had lower MSE (rounded to the nearest hundredth) than the corresponding AMC100 entry, and vice versa. Similarly, symmetrized AMC500 entries appear in bold when they had lower MSE than the corresponding AMC500 entry, and vice versa.

(a) Sparse signal
	Scenario 1		Scenario 2		Scenario 3		Scenario 4
	n=100	500	100	500	100	500	100	500
FLAM	0.44	0.12	0.47	0.17	0.38	0.11	0.51	0.19
AMC100 (ours)	0.34	0.20	0.18	0.08	0.27	0.14	0.17	0.08
Symmetrized AMC100 (ours)	0.32	0.18	0.18	0.08	0.26	0.13	0.16	0.08
AMC500 (ours)	0.48	0.12	0.19	0.06	0.35	0.10	0.23	0.08
Symmetrized AM5100 (ours)	0.43	0.12	0.17	0.05	0.32	0.09	0.21	0.07
(b) Dense signal
	Scenario 1		Scenario 2		Scenario 3		Scenario 4
	n=100	500	100	500	100	500	100	500
FLAM	0.59	0.17	0.65	0.24	0.53	0.16	0.76	0.36
AMC100 (ours)	1.20	0.91	0.47	0.39	0.87	0.57	0.30	0.30
Symmetrized AMC100 (ours)	1.16	0.84	0.45	0.37	0.83	0.52	0.29	0.30
AMC500 (ours)	0.58	0.15	0.37	0.08	0.46	0.12	0.36	0.09
Symmetrized AM5100 (ours)	0.55	0.15	0.36	0.08	0.43	0.11	0.34	0.09