Geometric Analysis of Uncertainty Sampling for Dense Neural Network Layer

Abstract

For model adaptation of fully connected neural network layers, we provide an information geometric and sample behavioral active learning uncertainty sampling objective analysis. We identify conditions under which several uncertainty-based methods have the same performance and show that such conditions are more likely to appear in the early stages of learning. We define riskier samples for adaptation, and demonstrate that, as the set of labeled samples increases, margin-based sampling outperforms other uncertainty sampling methods by preferentially selecting these risky samples. We support our derivations and illustrations with experiments using Meta-Dataset, a benchmark for few-shot learning. We compare uncertainty-based active learning objectives using features produced by SimpleCNAPS (a state-of-the-art few-shot classifier) as input for a fully-connected adaptation layer. Our results indicate that margin-based uncertainty sampling achieves similar performance as other uncertainty based sampling methods with fewer labelled samples as discussed in the novel geometric analysis.

I. Introduction

Recently, deep neural networks have been commonly used for hypothesis learning in the context of various regression and classification problems. These models require large labeled data sets to achieve good generalization performance [1]. Obtaining large labeled data sets is costly; several approaches exist to overcome this limitation. For example, model adaptation with few samples (informally referred to as zero-, one-, few-shot learning) may enable transformation of a hypothesis model (e.g., a classifier) trained for one specific task to a hypothesis model suitable for another task through minimal changes to its structure and parameters [1], [2]. Model adaptation for deep neural networks between classification tasks is typically achieved by adjusting the last few layers [3], [4]. Active learning is crucial in cases where rapid adaptation is required with limited data labeling resources [5].

Related Work:

As described in Settle’s work [6], active learning objectives often combine the following measures: uncertainty in probability space to select ambiguous samples [7], density in data space (e.g. N nearest neighbors [8]), expected model change (e.g. absolute sum of the gradients [9]), and expected error reduction (e.g. maximizing mutual information [10]). Other proposed methods include influence functions [11], and representer point based selection [12]. In the existing active learning literature, uncertainty sampling methodologies (Entropy Sampling (ES) [13], Confidence Sampling (CS) [14], and Margin Sampling (M) [15]) are often used as baseline comparisons [16], [17] due to their low computational overhead. Existing work usually reports the best performing uncertainty sampling method based on the test performance results, but performance comparison across (ES), (CS) and (M) and justification of the performance differences of these uncertainty methods across different datasets are omitted. The fundamental differences across (ES), (CS) and (M) are studied in the literature with extensive experimentation in different domains [18], [19]. Uncertainty sampling has also been comprehensively studied in parameter estimation for logistic regression models [20]. We suggest that the performance differences across (ES), (CS) and (M) across different testing datasets occur due to the sampling behaviour differences in these uncertainty sampling methods during actively updating the models. However, none of the existing work provide an explanation to these differences. Here, we aim to explain the sampling behavior differences for uncertainty based methods from an information geometric perspective, specifically through the geometrical analysis of the unlabeled sample predictions. We consider a fully connected neural network layer and demonstrate that by design (M) performs better in selecting riskier samples (i.e., the samples that are geometrically located in highly uncertain locations in the probability simplex) that enables it to achieve similar performances to (ES) and (CS) with fewer samples.

Contributions:

As mentioned above, uncertainty sampling methods have been commonly used and they were compared among each other and against other methods always through their performances on different datasets [19], [21]. We propose here a novel analytical approach to compare these methods, such an analytical approach does not exist in the literature. Specifically, (i) We identify conditions under which uncertainty-based methods have equal performance. (ii) We use information geometry to demonstrate the behavior of sample selection for (ES), (CS) and (M) and highlight that (M) selects samples from highly uncertain locations in the probability simplex. (iii) We validate our analysis in a few-shot learning scenario.

Preliminaries:

Let $(\mathcal{X},\mathcal{Y})$ denote the domain of data and labels over C different classes $\mathcal{H}=\left\{{\mathcal{H}}_1,{\mathcal{H}}_2,\cdots ,{\mathcal{H}}_C\right\}$. Let $\left(x\in {ℝ}^d,y\right)$ be a single data example and its corresponding one-hot label vector. To estimate the class label, we fit a parametric model $h_{\theta }:\mathcal{X}\to {\Delta }_C$ where Δ_C is the C-dimensional simplex:

$${\Delta }_C=\left\{\left(p_1,p_2,\cdots ,p_C\right)\in {ℝ}^C\mid p_i>0\forall i,\sum _i{p}_i=1\right\}$$

Model parameters θ are fit by minimizing a loss function $L:\left(h_{\theta }(\mathcal{X}),\mathcal{Y}\right)\to ℝ$, the model parameters are optimized appropriately. In this paper, we specifically focus on the following parameterized model;

$$\begin{gathered} h_{\theta }(x)=\mathrm{softmax}\left(\theta \left[\begin{array}{l}x\hfill \\ 1\hfill \end{array}\right]\right)\text{where}\theta \in {ℝ}^{C\times (d+1)} \\ \mathrm{softmax}{\left(a\right)}_i=\frac{e^{a_i}}{{\sum }_{j=1}^d{e}^{a_j}}\text{where}a\in {ℝ}^d \end{gathered}$$ (1)

Here θ is the parameter matrix and softmax : ${ℝ}^n\to {\Delta }_n$ denotes the operator that maps the linearly transformed x to the probability simplex [22]. The hypothesis class in (1) is a linear logistic regression function, i.e., a fully connected neural network layer followed by softmax nonlinearity.

In many applications, hypothesis learning can be achieved using a combination of labelled and unlabelled data s.t. $(\mathcal{X},\mathcal{Y})=\left(\left\{{\mathcal{X}}_L,{\mathcal{X}}_U\right\},\left\{{\mathcal{Y}}_L,{\mathcal{Y}}_U\right\}\right)$ where L and U denote subsets of labelled and unlabelled data respectively; Y_U is not available to the learner. We focus on the empirical risk minimization framework, in which optimal parameters for the model h are computed as:

$${\widehat{\theta }}_L=\arg \underset{\theta }{\min }\frac{1}{\left|{\mathcal{X}}_L\right|}\sum _{(x,y)\in \left({\mathcal{X}}_L,{\mathcal{Y}}_L\right)}L\left(h_{\theta }(x),y\right)$$

For the training of the last fully connected layer of a classification neural network, minimization of average cross entropy loss is considered $L\left(h_{\theta }(x),y\right)=H\left(h_{\theta }(x),y\right)$ [23]:

$$\begin{gathered} \arg \underset{\theta }{\min }H\left(\overrightarrow{y},h_{\theta }(x)\right)=\arg \underset{\theta }{\min }-\sum _i{y}_i\log \left({\left[h_{\theta }(x)\right]}_i\right) \\ =\arg \underset{\theta }{\min }-\log \left({\left[h_{\theta }(x)\right]}_{\widehat{i}}\right)\text{where}\widehat{i}=\arg \underset{j}{\max }{y}_j \\ =\arg \underset{\theta }{\min }{\left[h_{\theta }(x)\right]}_{\widehat{i}}\text{where}\widehat{i}=\arg \underset{j}{\max }{y}_j \end{gathered}$$ (2)

Actively learning a model includes an agent that selects anchor samples $x_a\in {\mathcal{X}}_U$ according to a sampling objective f, and receives the corresponding label y_a from an oracle to update the model:

$$\begin{gathered} x_a=\arg \underset{x\in {\mathcal{X}}_U}{\max }f\left(x,{\mathcal{X}}_L,{\mathcal{Y}}_L,h_{\theta }\right) \\ {\widehat{\theta }}_{L\cup a}=\arg \underset{\theta }{\min }\frac{1}{\left|{\mathcal{X}}_L\right|}\sum _{(x,y)\in \left({\mathcal{X}}_L\cup x_a,{\mathcal{Y}}_L\cup y_a\right)}H(\overrightarrow{y},h_{\theta }(x)) \\ \left({\mathcal{X}}_L,{\mathcal{Y}}_L\right)\leftarrow \left({\mathcal{X}}_L,{\mathcal{Y}}_L\right)\cup \left(x_a,y_a\right) \\ \left({\mathcal{X}}_U,{\mathcal{Y}}_U\right)\leftarrow \left({\mathcal{X}}_U,{\mathcal{Y}}_U\right)\backslash \left(x_a,y_a\right) \end{gathered}$$ (3)

In (3), f is designed to decrease the loss value as fast as possible by selecting meaningful samples. For f, we consider (ES), (CS) and (M) objectives presented in Table I and the geometry of each method based on their objectives is illustrated in Figure 1.

Table I:

Uncertainty sampling methods that form the basis of active learning methodologies.

Identifier	Root	Selection Method	Ref
(R)	random	xa = random(χU)	–
(ES)	entropy	xa = argmaxxϵχU − ∑i[hθ(x)]i log[hθ(x)]i	[13]
(CS)	confidence	xa = argminxϵχU [hθ(x)]i s.t. i = argmaxc [hθ(x)]c	[14]
(M)	margin	xa = argminxϵχU [hθ(x)]i − [hθ(x)]j s.t. i = argmaxc[hθ(x)]c j = argmaxc≠i[hθ(x)]c	[15]

Figure 1:

Geometry of the given methods in Δ₃. Figures from left to right represent the values of the objective functions presented in Table I (ES), (CS) and (M) respectively.

II. Analysis

The performance differences among (ES), (CS) and (M) selection objectives arise from the sequence of anchor samples x_a selected. Through Proposition 1, we show that these objectives have the same performance for 2-class classification. Further analyses then discuss the performance differences among the methods.

Proposition 1.

Given 2 class case $\mathcal{H}=\left\{{\mathcal{H}}_1,{\mathcal{H}}_2\right\}$ and the set $\mathcal{X}$ with $h_{\theta }(x)=p\in {\Delta }_2$ then;

$$\begin{gathered} x_a=\arg \underset{p\in h_{\theta }(\mathcal{X})}{\max }H(p)(\text{ES}) \\ =\arg \underset{p}{\min }\underset{i}{\max }{p}_i(\text{CS}) \\ =\arg \underset{p}{\min }\underset{j\ne i}{\max }\underset{i}{\max }{p}_i-p_j(\text{M}) \end{gathered}$$

Proof.

Let us denote the anchor sampling methods; (i) argmax_p H(p), (ii) argmin_p max_i p_i, (iii) $\arg {\min }_p{\max }_{j\ne i}{\max }_i{p}_i-p_j\cdot p\in {\text{Δ}}_2\Rightarrow p_i=1-p_{\ne i}$.

$$\arg \underset{p}{\min }\underset{j\ne i}{\max }\underset{i}{\max }{p}_i-p_j=\arg \underset{p}{\min }\underset{i}{\max }{p}_i-p_{\ne i}$$

$$=\arg \underset{p}{\min }\underset{i}{\max }2p_i-1=\arg \underset{p}{\min }\underset{i}{\max }{p}_i\Rightarrow (\text{ii})\equiv (\text{iii})$$

Similarly,

$$\begin{gathered} \arg \underset{p}{\max }H(p)=\arg \underset{p}{\max }-\sum _i{p}_i\log \left(p_i\right) \\ =\arg \underset{p}{\max }-p_i\log \left(p_i\right)-p_{\ne i}\log \left(p_{\ne i}\right) \\ =\arg \underset{p}{\max }-p_i\log \left(p_i\right)-\left(1-p_i\right)\log \left(1-p_i\right) \end{gathered}$$

WLOG assume then, p_i ≥ p_≠i then, 1 ≥ p_i ≥ 0.5 and −p_ilog(p_i)−(1 − p_i) log(1 − p_i) is monotonically decreasing wrt. p_i → given p, q ϵ Δ₂ max_i p_i > max_i q_i ⟹ H(p) < H(q) ⟹ argmax_p H(p) = argmin_p max_ip_i ⟹ (i) ≡ (ii). Then (i) ≡ (ii) ≡ (iii). □

In a C-class classification problem, model h_θ makes a correct decision for a sample and label tuple $(x,y)\in (\mathcal{X},\mathcal{Y})$ if $\arg {\max }_i{y}_i=\arg {\max }_i{\left[h_{\theta }(x)\right]}_i$. Therefore ∀hθ(x) ∈ Δ_n the critical boundary for a class i is formed where coordinate i is tied with a single other coordinate j. In other words, ∃j s.t. ${\left[h_{\theta }(x)\right]}_i={\left[h_{\theta }(x)\right]}_j\ge {\left[h_{\theta }(x)\right]}_{k\ne i,j}\forall k$.

Assume a three class classification as illustrated in Figure 2 where the labels are denoted with a,b,c. Then in order to correctly classify a for example, the hypothesis should result a probability vector where a has the highest probability mass (blue highlighted area). Hence all possible probability vector with a having the max probability contribute to the true selection. In the figure the decision boundary for class a is also visualized with the blue-bold lines. Given p = h_θ(x), these lines satisfy ∃j s.t. p_i = p_j ≥ p_k ∀k. In other words, the model is uncertain between two competitors. Points $x\in \mathcal{X}$ that result in p = h_θ(x) close to these decision boundaries are riskier samples.

Figure 2:

A three class decision boundaries on Δ₃. The highlighted area corresponds to where the probability mass of a in a given probability vector ∈ Δ₃ is the highest. Therefore a hypothesis resulting a probability within that area correctly classifies a. Dashed red line represent an equi-probability contour of a logistic normal distribution that passes through the uniform distribution.

(ES) has a frail confidence assessment:

Let $\{x1,x2\}\subset {\mathcal{X}}_{\mathcal{U}}$ where argmax_i y1_i = argmax_i y2_i = 1 and $p=h_{\widehat{\theta }}(x1)$, $q=h_{\widehat{\theta }}(x2)\in {\text{Δ}}_{10}$ with $p=[.6,.4-8ϵ,ϵ,\cdots ]$ where 0 < ϵ << 1 and $q=[.7,.0\overline{3},\cdots ]$. It is apparent that q is more confident on the true class however due to 0.97 ≈ H(p) < H(q) ≈ 1.82 (ES) selects q over p. For this example (M) and (CS) captures the notion of confidence which is determined by the maximum posterior probability and selects the sample with lesser confidence.

(CS) makes decisions using single value:

Let $\{x1,x2,x3,x4\}\subset {\mathcal{X}}_{\mathcal{U}}$ with $p1=h_{\widehat{\theta }}(x1)$, $p2=h_{\widehat{\theta }}(x2)\in {\text{Δ}}_{10}$, $q1=h_{\widehat{\theta }}(x3)$, $q2=h_{\widehat{\theta }}(x4)\in {\text{Δ}}_{10}$ with $p1=[.5,.5-8ϵ,ϵ,\cdots ],p2=[.6,.4-8ϵ,ϵ,\cdots ]$ where 0 < ϵ << 1 and $q1=[.5,0.0\overline{5},\cdots ]$, $q2=[.6,0.0\overline{4},\cdots ]$ and we compare ps then qs. Since confidence level on a particular class differs the same amount .1 (CS) treats p and q cases the same, for the ps the 2^nd best class is still a legitimate competitor, whereas for qs there was no other competitor. However, (M) makes a distinction between these cases by incorporating another element to select the sample closer to decision boundary (p).

Sample Behavior:

Ultimately, the differences in performance occur due to the different geometric locations of the data in the simplex (i.e., h_θ(x) ∈ Δ_n ) compared to the geometric selection regions of different objectives. Let us observe the different outcome possibilities from the fully connected layer. Consider Figure 3 for Δ₃ for three scenarios of h_θ outcomes. Given $\left(x^l,y^l\right)\in \left({\mathcal{X}}_L,{\mathcal{Y}}_L\right)$ and $\left(x^u,y^u\right)\in \left({\mathcal{X}}_U,{\mathcal{Y}}_U\right)$ if ∃θ s.t.: (s) separation $\left(\forall \left(x^l,y^l\right)\arg {\max }_i{\left[h_{\theta }\left(x^l\right)\right]}_i=\arg {\max }_i{y}_i^l\right)$, (qs) quasi-separation $(\forall \left(x^l,y^l\right)\arg {\max }_i{\left[h_{\theta }\left(x^l\right)\right]}_i=\arg {\max }_i{y}_i^l$ and ∃(x^l,y^l) s.t. |argmax_i h_θ(x^l)| ≥ 2. e.g. h_θ(x^l) = [0.5,0.5,0,···], y^l = [1,0,···]), (o) overlap (∃(x^l,y^l) where $\arg {\max }_i{\left[h_{\theta }\left(x^l\right)\right]}_i\ne \arg {\max }_i{y}_i^l)$ then a stationary point $\widehat{\theta }=\arg {\inf }_{\theta }\left(1/\left|{\mathcal{X}}_L\right|\right){\sum }_{(x,y)\in \left({\mathcal{X}}_L,{\mathcal{Y}}_L\right)}H\left(\overrightarrow{y},h_{\theta }(x)\right)$ only exists for (o). We refer the reader to Albert’s work [24] for the detailed proof showing (s) and (qs) project samples to a two class decision as shown in Figure 3. In Proposition 1 we already discussed if samples reside on a line which is the case for (s) and (qs), the sampling methodologies behave the same and hence we further investigate the case of (o). Note that it is empirically known that the differences in the performances among (CS), (ES) and (M) increase as the training progresses. As also discussed above, the performances differ only in the case of (o). With the following proposition, we show that during the training, the probability of achieving the case of (o) as the outcome of h_θ in a dataset increases as more data is incorporated into the labeled pool ${\mathcal{X}}_L$ for training;

Figure 3:

Separated (s), quasi-separated (qs) and overlapping (o) labelled data ${\mathcal{X}}_L$ is used to determine a stationary point $h_{\widehat{\theta }}$ Observe that (s), (qs) yields unlabelled samples on the lines that force exact same results in (ES), (CS) and (M). (o) on the other hand allows selection methods operate in 2D.

Proposition 2.

Let h_θ be an arbitrary model and $(\mathcal{X},\mathcal{Y})=\left\{(x,y)\mid x~f_y\right\}$ where f_y is an arbitrary distribution identified with the label y. Let $\left({\mathcal{X}}_1,{\mathcal{Y}}_1\right)\subseteq \left({\mathcal{X}}_2,{\mathcal{Y}}_2\right)\subseteq (\mathcal{X},\mathcal{Y})$ then probability of (o) in $\left({\mathcal{X}}_2,{\mathcal{Y}}_2\right)$ is greater or equal than probability of (o) in $\left({\mathcal{X}}_1,{\mathcal{Y}}_1\right)$.

Proof.

Let A := ∃ overlap $\in \left({\mathcal{X}}_1,{\mathcal{Y}}_1\right)$, B := ∃overlap ∈ $\left({\mathcal{X}}_2,{\mathcal{Y}}_2\right)\backslash \left({\mathcal{X}}_1,{\mathcal{Y}}_1\right)\Rightarrow A+B:=\exists$ overlap $\in \left({\mathcal{X}}_2,{\mathcal{Y}}_2\right)$. Trivially p(A+B) ≥ p(A)+p(B) ⟹ p(A+B) ≥ p(A). □

In summary, Proposition 2 states that as the training progresses, the probability of the case (o) increases which implies that for active learning the (CS), (ES) and (M) will start to have different performances.

Special case with Gaussian data:

For the case of (o), consider samples originating from multi-variate Gaussian distributions for a C-class classification s.t. $\forall (x,y)\in (\mathcal{X},\mathcal{Y})$, $x=\mathcal{N}\left({\mu }_y,{\Sigma }_y\right)$. Note that the equi-probability contours are not centered around the center of the Δ_C (simplex) but the corners. Since x is normally distributed, linear combinations with a scalar shift of the random variables also follow a normal distribution. Moreover, the inverse of softmax operator is well approximated as central logratio transform clr(.) [22], [25]:

$$\mathrm{clr}(p)=\left[\frac{p_1}{g(p)},\cdots ,\frac{p_C}{g(p)}\right]\text{where}g(p)={\left(\prod _i{p}_i\right)}^{1/C}$$

Hence the outputs of the model defined in (1) follow a logistic-normal distribution for which the pdf is;

$$\begin{gathered} f_{{\text{Δ}}_C}(p;m,s)=J_C\frac{1}{|2\pi s{|}^{\frac{1}{2}}}\exp (-\frac{1}{2}{\left(p^{\prime }-m\right)}^T \\ {s}^{-1}\left(p^{\prime }-m\right))\text{where}{p}^{\prime }=p/g(p) \end{gathered}$$ (4)

As discussed in [22], it is possible to find equi-probability contours within Δ_C. WLOG assume we are interested in ${\mathcal{H}}_1$ with an ideal m = [1 − ε, ε,··· ,ε] with 0 < ε << 1. In Figure 2 we visualize a sample equi-probability contour with the dashed red line. Hence in terms of the probability geometry, the assessment of riskier samples are not centric as guided by the entropy but they follow a pattern that are analogous with the critical boundaries and hence a perspective that is centered around the corners is more preferable. Comparison between Figures 1 and 2 show that (M) captures these equi-probable regions but not (CS) or (ES).

III. Experiments and Results

Paradigm:

Using a few-shot learning scenario, we compare (ES), (CS) and (M). Also we consider the random selection case, see Table I. In few-shot learning approaches, model parameters are updated starting from a checkpoint [1], [2]. In the cases where adaptation needs to be fast, especially in deep feature models, a backbone that is already trained on a large labelled dataset (e.g. image classification ResNet [26]) is kept constant and an adaptation layer is further adjusted to generalize across an unseen task. Specifically, as the backbone, in our experiments we use the model presented in [3] and later simplified in [4] which is by design learned to output features that follow Gaussian distribution. We then actively learn a fully connected layer during adaptation.

Dataset and Experiment:

We use Meta-dataset [27], a benchmark for few-shot learning and image classification that comprises the following labelled image datasets: ILSVRC-2012 [28], Omniglot [29], FGVC-Aircraft [30], CUB-200–2011 [31], Describable Textures [32], QuickDraw [33], FGVCx Fungi, VGG Flower [34], Traffic Signs [35] and MSCOCO [36]. We follow the train-test splits provided by Meta-dataset in our experiments. We define ‘root-acc’; as the test set accuracy achieved using all training labels set is available. We evaluate active learning methods by comparing what percentage of the training set must be queried before achieving the root-acc. In our experiments, we initialize the system by providing labels for 5% of the training samples and training an initial model: $\left|{\mathcal{X}}_U\right|=0.05\times \left|{\mathcal{X}}_{U\cup L}\right|$. At each iteration, we select a batch of 10-samples to be labelled according to the objectives presented in Table I. At each iteration, the models are also updated to a stationary point.

Results:

Results are presented in Table II. In this Table, the rows represent the results for different datasets and the average of all is presented as the final row. The table is divided into 4 grouped-columns, where each group denotes a model reaching a pre-determined performance value close to root-acc. For example, where root-acc 85%, the column root-acc−15% represents a model achieving a performance ≥ 70. Each column in each group presents the results of a different sample selection method from Table I. We present the mean and standard deviation for the percentage of the dataset used to achieve the pre-determined performance value. Lower mean values indicate that the method requires less data to match the performance of the other sampling methods. The results show that (M) picks the samples that result in faster increase in performance. Moreover, we observe that the gap between methodologies increases as labelled data size increases (to achieve higher performance more data is required) which is also stated in Prop. 2.

Table II:

A fully connected layer on top of the backbone network architecture is used. The average number of training samples are ≈ 355 where number of test samples are ≈ 90. In the table root-acc and max-acc refer to accuracy achieved in test-set if all training-set is used and maximum accuracy achieved in any point of active learning respectively. We report the ratio of training samples used to cardinality of the entire training set to reach a neighborhood of root-accuracy. Therefore, a smaller number represents fewer samples used and hence the less the better. It is apparent that margin sampling (M) outperforms other selection methods and achieves the confidence range using less percentage of the training set labelled.

		root-acc −15%				root-acc −10%				root-acc −5%				root-acc −1%
data-set	root-acc (%)	(R)	(ES)	(CS)	(M)	(R)	(ES)	(CS)	(M)	(R)	(ES)	(CS)	(M)	(R)	(ES)	(CS)	(M)
aircraft	83	0.24±0.17	0.29±0.20	0.24±0.17	0.23±0.17	0.29±0.17	0.40±0.22	0.27±0.17	0.26±0.17	0.43±0.20	0.62±0.22	0.38±0.18	0.33±0.17	0.64±0.24	0.80±0.18	0.56±0.20	0.47±0.20
cifar-10	78	0.24±0.16	0.37±0.22	0.21±0.16	0.24±0.15	0.31±0.16	0.49±0.23	0.25±0.16	0.24±0.15	0.47±0.20	0.70±0.19	0.36±0.17	0.33±0.16	0.68±0.22	0.85±0.14	0.52±0.21	0.47±0.21
cu-birds	76	0.24±0.17	0.29±0.20	0.24±0.17	0.23±0.17	0.29±0.17	0.40±0.22	0.27±0.17	0.26±0.17	0.43±0.19	0.62±0.22	0.38±0.18	0.33±0.16	0.64±0.24	0.80±0.17	0.56±0.20	0.48±0.20
fungi	40	0.50±0.14	0.50±0.14	0.50±0.14	0.50±0.14	0.50±0.14	0.50±0.14	0.50±0.14	0.50±0.14	0.53±0.15	0.54±0.17	0.52±0.14	0.51±0.14	0.62±0.20	0.65±0.21	0.57±0.16	0.58±0.16
ms-coco	48	0.38±0.12	0.39±0.12	0.38±0.11	0.39±0.12	0.40±0.12	0.44±0.15	0.39±0.12	0.39±0.12	0.52±0.17	0.58±0.20	0.45±0.13	0.44±0.13	0.73±0.20	0.76±0.22	0.62±0.18	0.58±0.17
omniglot	91	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.76±0.15	0.83±0.11	0.83±0.11	0.80±0.12	0.78±0.12
quickdraw	76	0.44±0.10	0.44±0.11	0.44±0.11	0.44±0.11	0.44±0.10	0.46±0.12	0.44±0.10	0.44±0.10	0.52±0.13	0.60±0.17	0.50±0.11	0.48±0.11	0.78±0.17	0.84±0.15	0.66±0.14	0.63±0.14
traffic-sign	77	0.39±0.12	0.39±0.13	0.38±0.12	0.39±0.12	0.39±0.12	0.41±0.16	0.39±0.12	0.39±0.12	0.42±0.14	0.50±0.24	0.39±0.12	0.39±0.12	0.50±0.19	0.63±0.30	0.41±0.13	0.40±0.12
vgg-flower	91	0.29±0.21	0.30±0.21	0.29±0.21	0.29±0.21	0.30±0.21	0.36±0.25	0.30±0.21	0.29±0.21	0.42±0.20	0.61±0.28	0.33±0.20	0.32±0.20	0.65±0.23	0.85±0.20	0.44±0.20	0.41±0.19
average	-	0.38	0.41	0.38	0.38	0.40	0.47	0.40	0.39	0.50	0.61	0.45	0.43	0.67	0.78	0.57	0.53

IV. Conclusion

In this work we provide an analysis of actively adapting a fully connected final layer in a network architecture in a model-adaption setting. Specifically, we focused on uncertainty sampling methods that are widely used as a sanity check in active learning tasks. We have shown that geometrically, fully connected layer behavior and sample positioning by fact strengthens margin sampling over other uncertainty based approaches. Empirically, we validated the claims in a few-shot learning setting where a fully connected adaptation layer exists. With that knowledge, it is possible to propose proxy gradient methods that leverage margin instead of selection based on mutual information surrogates.