. 2022 Aug 17;28(4):377–431. doi: 10.1007/s10732-022-09495-3

Table 5.

Comparison of SAT-based MaxSAT approaches versus CALOT for the Covering Array Number problem

Instance	ACTS		CALOT CCX a.0		CALOT CCX a.1		RC2-B CCX a.0		RC2-B CCX a.0 wpm		linear CCX a.0
	Size	Time	Size	Time	Size	Time	Size	Time	Size	Time	Size	Time
Cohen et al. (2008)
1	48.00	0.83	38.00	1688.39	38.00	1470.88	$-$ 1.00	$-$ 1.00	37.00	7146.52	37.00	5367.02
2	32.00	1.08	30.00*	2.80	30.00*	3.57	30.00*	9.85	30.00*	10.81	30.00*	2.91
3	19.00	1.21	18.00*	0.09	18.00*	0.12	18.00*	0.59	18.00*	0.73	18.00*	0.13
4	22.00	1.04	20.00*	0.52	20.00*	0.95	20.00*	2.58	20.00*	3.19	20.00*	0.74
5	54.00	1.31	47.00	1955.92	48.00	241.15	$-$ 1.00	$-$ 1.00	48.00	378.46	47.00	1693.29
6	25.00	1.11	24.00*	1.18	24.00*	1.46	24.00*	4.46	24.00*	5.47	24.00*	1.48
7	12.00	1.13	9.00	0.06	9.00	0.08	$-$ 1.00	$-$ 1.00	9.00	0.61	9.00	0.13
8	47.00	1.01	38.00	1403.51	37.00	4727.41	$-$ 1.00	$-$ 1.00	37.00	3407.87	38.00	547.77
9	22.00	1.21	20.00*	0.57	20.00*	0.84	20.00*	2.36	20.00*	2.66	20.00*	0.49
10	47.00	1.28	44.00	6860.63	44.20	5250.31	$-$ 1.00	$-$ 1.00	44.00	4759.95	44.00	1343.51
11	47.00	1.26	41.00	2641.09	41.00	4167.69	$-$ 1.00	$-$ 1.00	42.00	6172.69	42.00	1916.85
12	43.00	1.29	36.00*	14.13	36.00*	24.01	36.00*	36.81	36.00*	49.94	36.00*	20.66
13	40.00	1.15	36.00*	6.78	36.00*	9.80	36.00*	24.42	36.00*	29.91	36.00*	6.73
14	39.00	0.83	36.00*	3.76	36.00*	5.25	36.00*	13.81	36.00*	15.56	36.00*	4.80
15	32.00	1.05	30.00*	1.02	30.00*	1.49	30.00*	4.73	30.00*	4.88	30.00*	1.30
16	25.00	0.84	24.00*	1.36	24.00*	1.94	24.00*	5.59	24.00*	6.60	24.00*	1.65
17	41.00	1.17	36.00*	10.16	36.00*	13.60	36.00*	30.57	36.00*	35.75	36.00*	10.98
18	52.00	1.30	42.00	128.65	42.00	201.42	$-$ 1.00	$-$ 1.00	42.00	177.28	42.00	202.10
19	51.00	1.43	46.00	178.56	45.00	118.23	$-$ 1.00	$-$ 1.00	46.00	265.76	46.00	128.89
20	60.00	1.53	57.00	129.90	57.00	224.31	$-$ 1.00	$-$ 1.00	57.00	236.47	56.00	1092.85
21	39.00	1.33	36.00*	2.59	36.00*	3.86	36.00*	9.99	36.00*	9.95	36.00*	3.40
22	37.00	1.08	36.00*	1.75	36.00*	2.70	36.00*	7.74	36.00*	8.57	36.00*	2.17
23	14.00	0.75	12.00*	0.05	12.00*	0.09	12.00*	0.42	12.00*	0.41	12.00*	0.08
24	48.00	1.40	44.40	4172.55	45.00	185.06	$-$ 1.00	$-$ 1.00	44.00	41.40	45.00	13.58
25	52.00	1.28	52.00	100.56	52.00	117.86	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	52.00	423.91
26	34.00	1.37	27.00	49.99	27.00	31.54	$-$ 1.00	$-$ 1.00	27.00	112.13	27.00	551.57
27	37.00	1.32	36.00*	1.14	36.00*	1.68	36.00*	5.35	36.00*	6.70	36.00*	1.33
28	57.00	1.54	53.00	95.96	53.00	78.94	$-$ 1.00	$-$ 1.00	53.00	330.79	53.00	120.36
29	29.00	1.15	25.00*	5.63	25.00*	8.77	25.00*	20.73	25.00*	26.38	25.00*	7.81
30	22.00	0.81	16.00*	1.38	16.00*	1.75	16.00*	4.53	16.00*	5.98	16.00*	1.56
apache	33.00	1.25	30.00*	8.96	30.00*	13.68	30.00*	33.44	30.00*	38.18	30.00*	11.28
bugzilla	19.00	1.03	16.00*	0.35	16.00*	0.61	16.00*	1.72	16.00*	1.79	16.00*	0.36
gcc	23.00	0.93	15.00	13.09	15.00	19.06	$-$ 1.00	$-$ 1.00	15.00	47.85	15.00	22.14
spins	26.00	1.22	19.00*	0.13	19.00*	0.22	19.00*	0.56	19.00*	0.70	19.00*	0.16
spinv	45.00	0.84	33.00	5.28	32.00	188.97	$-$ 1.00	$-$ 1.00	32.00	55.75	33.00	21.39
Segall et al. (2011)
Banking1	15.00	0.85	13.00*	0.01	13.00*	0.01	13.00*	0.20	13.00*	0.29	13.00*	0.08
Banking2	11.00	0.46	10.00*	0.01	10.00*	0.02	10.00*	25.56	10.00*	22.95	10.00*	0.03
CommProtocol	19.00	1.05	16.00*	0.02	16.00*	0.03	16.00*	0.19	16.00*	0.22	16.00*	0.03
Concurrency	6.00	0.49	5.00*	0.00	5.00*	0.00	5.00*	0.08	5.00*	0.08	5.00*	0.01
Healthcare1	30.00	0.55	30.00*	0.03	30.00*	0.03	30.00*	0.24	30.00*	0.25	30.00*	0.05
Healthcare2	16.00	0.60	14.00	0.02	14.00	0.07	$-$ 1.00	$-$ 1.00	14.00	0.21	14.00	0.07
Healthcare3	38.00	0.58	34.00*	0.48	34.00*	0.75	34.00*	2.14	34.00*	2.39	34.00*	0.68
Healthcare4	49.00	0.59	46.00*	1.77	46.00*	2.33	46.00*	5.88	46.00*	7.24	46.00*	2.61
Insurance	527.00	0.31	527.00*	16.82	527.00*	18.56	527.00*	44.42	527.00*	45.47	527.00*	21.08
NetworkMgmt	112.00	0.58	110.00*	214.18	110.00*	511.52	110.00*	180.82	110.00*	233.46	110.00*	1100.79
ProcessorComm1	29.00	0.52	21.00	67.53	22.00	0.47	$-$ 1.00	$-$ 1.00	22.00	1.14	21.00	28.65
ProcessorComm2	32.00	0.77	25.00*	0.42	25.00*	0.55	25.00*	1.42	25.00*	1.70	25.00*	0.42
Services	106.00	1.20	100.00*	28.65	100.00*	29.89	100.00*	40.55	100.00*	37.37	100.00*	26.69
Storage1	17.00	0.75	17.00*	0.00	17.00*	0.01	17.00*	0.11	17.00*	0.11	17.00*	0.01
Storage2	18.00	0.25	18.00*	0.01	18.00*	0.01	18.00*	0.12	18.00*	0.12	18.00*	0.02
Storage3	50.00	0.66	50.00*	0.14	50.00*	0.28	50.00*	11.02	50.00*	1.86	50.00*	0.26
Storage4	136.00	0.60	130.00*	2.49	130.00*	3.24	130.00*	7.24	130.00*	9.28	130.00*	1.69
Storage5	218.00	0.87	215.00	68.48	215.00	389.33	$-$ 1.00	$-$ 1.00	215.00	140.68	215.00	811.43
SystemMgmt	17.00	0.57	15.00*	0.01	15.00*	0.03	15.00*	0.16	15.00*	0.18	15.00*	0.02
Telecom	32.00	0.58	30.00*	0.04	30.00*	0.05	30.00*	0.31	30.00*	0.37	30.00*	0.06
Yu et al. (2015)
RL-A	155.00	2.88	153.00	14.02	153.00	25.95	$-$ 1.00	$-$ 1.00	153.00	28.52	153.00	18.89
RL-B	767.00	170.61	760.00	4471.65	763.00	5862.61	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	764.00	5633.81

Instance	loandra CCX a.0		loandra CCX a.1		tt-open-wbo-inc CCX a.0		tt-open-wbo-inc CCX a.1		tt-open-wbo-inc CCX a.0 wpm		tt-open-wbo-inc CCX a.1 wpm
	Size	Time	Size	Time	Size	Time	Size	Time	Size	Time	Size	Time
Yamada et al. (2016)
Company2	81.00	9.37	72.00	1.80	72.00	1.88	$-$ 1.00	$-$ 1.00	72.00	3.33	72.00	2.48
Cohen et al. (2008)
1	37.00	1718.97	37.00	426.06	37.00	902.46	38.00	133.26	37.00	4724.61	38.00	1196.04
2	30.00*	7.19	30.00*	7.68	30.00	0.81	30.00	1.42	30.00	0.92	30.00	1.70
3	18.00*	0.36	18.00*	0.47	18.00	0.06	18.00	0.08	18.00	0.05	18.00	0.07
4	20.00*	1.34	20.00*	2.24	20.00	0.30	20.00	0.29	20.00	0.31	20.00	0.45
5	46.00	1629.29	45.80	1795.81	46.00	1624.87	46.00	1348.31	46.20	1803.99	46.00	4655.60
6	24.00*	2.98	24.00*	3.51	24.00	0.33	24.00	0.48	24.00	0.34	24.00	0.47
7	9.00	30.98	9.00	30.57	9.00	0.06	9.00	0.06	9.00	0.05	9.00	0.06
8	36.80*	2849.90	37.00	3731.63	37.00	1178.58	36.60	2948.62	37.00	2333.83	37.00	5133.82
9	20.00*	1.22	20.00*	2.30	20.00	0.18	20.00	0.27	20.00	0.19	20.00	0.41
10	41.80	1235.36	41.00	1891.73	41.00	2720.18	41.00	970.10	41.00	5823.09	41.00	641.10
11	39.00	6308.22	40.00	4323.47	40.00	741.96	40.00	1086.24	39.00	1628.02	39.00	2892.01
12	36.00*	36.16	36.00*	42.45	36.00	8.18	36.00	10.46	36.00	10.75	36.00	11.37
13	36.00*	14.91	36.00*	25.04	36.00	3.46	36.00	3.31	36.00	3.23	36.00	3.60
14	36.00*	8.84	36.00*	11.51	36.00	1.89	36.00	2.48	36.00	1.51	36.00	4.28
15	30.00*	2.91	30.00*	3.70	30.00	0.47	30.00	0.66	30.00	0.90	30.00	0.77
16	24.00*	4.30	24.00*	4.46	24.00	0.56	24.00	0.60	24.00	0.57	24.00	0.62
17	36.00*	22.72	36.00*	31.18	36.00	5.88	36.00	7.48	36.00	7.31	36.00	6.94
18	41.00	637.62	41.00	734.99	41.00	134.63	41.00	626.83	41.00	2943.99	41.00	1871.62
19	44.00	2056.11	43.80	2638.11	44.00	755.22	44.00	756.79	44.00	2892.85	44.00	578.78
20	54.80	1422.09	54.60	1736.92	55.00	32.47	55.00	500.42	55.00	437.94	55.00	176.33
21	36.00*	7.08	36.00*	8.88	36.00	1.06	36.00	1.38	36.00	1.72	36.00	1.94
22	36.00*	5.25	36.00*	6.78	36.00	0.65	36.00	0.88	36.00	0.72	36.00	1.04
23	12.00*	0.22	12.00*	0.30	12.00	0.03	12.00	0.05	12.00	0.06	12.00	0.09
24	42.00	1693.19	41.00	4304.36	42.00	1465.17	43.00	59.40	42.00	884.17	42.00	926.51
25	50.00	2069.85	49.80	1133.70	51.00	136.11	50.00	2101.71	50.00	5892.75	50.00	263.99
26	27.00	103.46	27.00	57.66	26.00	6740.38	26.00	790.74	27.00	210.04	27.00	141.21
27	36.00*	2.96	36.00*	3.56	36.00	0.39	36.00	0.61	36.00	0.45	36.00	0.62
28	51.20	333.93	51.00	1236.07	51.00	3112.51	51.00	607.78	52.00	61.99	49.00	3329.73
29	25.00*	12.69	25.00*	17.48	25.00	3.05	25.00	4.02	25.00	3.55	25.00	4.63
30	16.00*	3.57	16.00*	3.94	16.00	0.68	16.00	0.83	16.00	0.77	16.00	1.08
apache	30.00*	21.65	30.00*	32.14	30.00	5.17	30.00	6.70	30.00	5.82	30.00	5.57
bugzilla	16.00*	0.89	16.00*	1.19	16.00	0.14	16.00	0.19	16.00	0.14	16.00	0.46
gcc	15.00	63.99	15.00	68.18	15.00	9.44	15.00	12.76	15.00	11.82	15.00	29.62
spins	19.00*	0.28	19.00*	0.28	19.00	0.13	19.00	0.21	19.00	0.20	19.00	0.21
spinv	32.00	61.37	32.00	97.82	32.00	26.18	32.00	82.41	32.00	53.36	32.00	199.38
Segall et al. (2011)
Banking1	13.00*	0.05	13.00*	0.05	13.00	0.09	13.00	0.08	13.00	0.06	13.00	0.06
Banking2	10.00*	19.30	10.00*	10.62	10.00	0.02	10.00	0.02	10.00	0.02	10.00	0.02
CommProtocol	16.00*	0.15	16.00*	0.09	16.00	0.05	16.00	0.03	16.00	0.02	16.00	0.02
Concurrency	5.00*	0.03	5.00*	0.03	5.00	0.03	5.00	0.01	5.00	0.01	5.00	0.01
Healthcare1	30.00*	0.13	30.00*	0.15	30.00	0.17	30.00	0.03	30.00	0.02	30.00	0.03
Healthcare2	14.00	30.55	14.00	30.06	14.00	0.21	14.00	0.33	14.00	0.05	14.00	0.35
Healthcare3	34.00*	1.63	34.00*	1.44	34.00	0.26	34.00	0.32	34.00	0.37	34.00	0.45
Healthcare4	46.00*	3.64	46.00*	3.71	46.00	0.73	46.00	0.82	46.00	1.83	46.00	2.22
Insurance	527.00*	45.23	527.00*	46.40	527.00	5.29	527.00	10.47	527.00	5.08	527.00	11.25
NetworkMgmt	110.00*	45.65	110.00*	34.19	110.00	344.68	110.00	302.44	110.00	408.88	110.00	276.77
ProcessorComm1	21.00	468.17	21.00	107.68	21.00	610.20	21.00	1690.90	22.00	0.22	21.00	1015.43
ProcessorComm2	25.00*	0.78	25.00*	1.22	25.00	0.23	25.00	0.39	25.00	0.26	25.00	0.41
Services	100.00*	3.94	100.00*	3.97	100.00	2.43	100.00	2.44	100.00	2.42	100.00	1.46
Storage1	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	17.00	0.01	17.00	0.01	17.00	0.01	17.00	0.01
Storage2	18.00*	0.05	18.00*	0.05	18.00	0.02	18.00	0.01	18.00	0.01	18.00	0.01
Storage3	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	50.00	0.09	50.00	0.11	50.00	0.08	50.00	0.12
Storage4	130.00*	5.16	130.00*	6.65	130.00	0.55	130.00	0.84	130.00	0.58	130.00	0.83
Storage5	215.00	85.07	215.00	90.98	215.00	17.01	215.00	16.83	215.00	45.27	215.00	46.54
SystemMgmt	15.00*	0.08	15.00*	0.09	15.00	0.02	15.00	0.02	15.00	0.02	15.00	0.02
Telecom	30.00*	0.18	30.00*	0.22	30.00	0.03	30.00	0.04	30.00	0.03	30.00	0.04
Yu et al. (2015)
RL-A	153.00	58.49	153.00	63.11	153.00	3.25	153.00	3.77	153.00	12.04	153.00	15.40
RL-B	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	$-$ 1.00	727.00	966.65	$-$ 1.00	$-$ 1.00	727.00	5809.65
Yamada et al. (2016)
Company2	72.00	32.15	72.00	32.96	72.00	0.68	72.00	0.75	72.00	0.99	72.00	1.29

Bold values represent the best results. In case of ties in size, the best time is marked. For sizes with a star the optimum has been certified in at least one of the five seeds executed