. Author manuscript; available in PMC: 2023 Aug 2.

Published in final edited form as: Adv Neural Inf Process Syst. 2022 Dec;35(DB):29776–29788.

Table 1: Performance comparison:

For each model, we report the accuracy and number of tasks with accuracy above 80%. SES is the Sample Efficiency Score; it favors models with high performance in low data regimes and consistent accuracy across regimes. SES and AUC are computed over the 20–1000 data regimes. OOD generalization results are provided in the SI.

N train samples			20		50		100		200		500		1000		SES	AUC	10000
rand-init	ind	ResNet-50[15]	28.0	1	31.1	1	32.5	3	34.0	6	38.7	12	44.8	24	33.7	34.9	-	-
		ViT-small[11]	28.6	1	30.1	4	30.9	4	31.9	4	33.8	4	35.1	7	31.3	31.7	-	-
		SCL[40]	26.9	0	30.0	1	30.3	2	30.0	2	31.4	2	33.4	5	29.9	30.3	-	-
		WReN[[3]	30.0	0	32.0	2	32.9	2	34.1	3	36.3	6	39.0	15	33.4	34.1	-	-
		SCL-ResNet 18	31.4	1	37.3	9	37.8	9	39.6	15	42.7	21	48.3	26	38.4	39.5	-	-

	joint	ResNet-50	27.5	0	28.2	0	29.9	2	33.9	6	52.1	29	59.2	34	36.0	38.4	93.7	93
		ViT-small	27.3	1	27.8	2	28.0	1	28.1	1	29.9	2	31.4	3	28.4	28.7	58.7	37
		SCL	25.8	0	25.8	0	28.3	1	34.1	3	43.2	22	46.2	27	32.2	33.9	56.9	34
		WReN	26.8	0	27.6	0	28.5	0	30.1	0	36.4	9	42.3	20	30.9	32.0	64.5	43
		SCL-ResNet 18	26.4	0	28.4	0	31.6	4	40.7	13	51.4	32	64.0	42	37.6	40.4	78.9	73

SSL	ind	ResNet-50	40.5	13	47.3	18	52.9	29	56.8	34	61.9	42	67.7	50	52.4	54.5	-	-
	ind	ViT-small	46.7	16	51.6	24	54.8	29	57.5	38	62.0	44	65.5	46	54.9	56.4	-	-

	joint	ResNet-50	44.3	16	50.3	24	55.3	30	59.5	42	68.9	49	79.2	59	57.0	59.6	93.1	97
	joint	ViT-small	39.3	15	39.5	13	40.8	14	44.1	16	53.3	30	60.7	41	44.7	46.3	81.6	67

IN	joint	ResNet-50	32.0	2	35.1	5	39.0	9	43.8	13	57.7	48	69.5	48	43.4	46.2	-	-
IN	joint	ViT-small	27.9	2	28.2	1	28.6	2	30.0	2	35.6	5	47.2	24	31.7	32.9	-	-

CLIP	joint	ResNet-50	28.7	0	32.0	2	40.8	11	46.9	18	59.7	40	74.4	53	43.7	47.1	-	-
CLIP	joint	ViT-base	31.1	1	37.4	7	43.9	14	56.0	30	68.9	48	78.8	62	48.9	52.7	-	-