. Author manuscript; available in PMC: 2017 Jan 12.

Published in final edited form as: J Chem Inf Model. 2016 May 17;56(6):1063–1077. doi: 10.1021/acs.jcim.5b00523

Table 2.

Docking and scoring results for Phase 2 were evaluated by Spearman ρ for relative ranking across all the ligands for a target and by the median, symmetry-corrected RMSD for the top-pose ligands (based on all ligands with crystal structures available). Numbers in bold denote good performance (median RMSD ≤ 2.0Å or ρ ≥ 0.5). For docking, methods with all median RMSD ≤ 2Å are listed in bold with blue background, and italics with green backgrounds mark those with all median RMSD ≤ 3Å. For ranking, methods with at least one ρ ≥ 0.5 are listed in italics with green background, and those in bold with blue background have a second system with ρ ≥ 0.4. The statistical significance is discussed at length in the text.

ID^a	TrmD		SYK		FXa		Evaluation^b		Docking Method	Scoring/Ranking Method (More than one scoring method can be applied to the same docked poses)
ID^a	Median RMSD	ρ	Median RMSD	ρ	Median RMSD	ρ	Dock	Rank	Docking Method
R	1.56	0.59	2.00	0.52	1.62	−0.15	9	4	SHAFTS⁷⁰ + MDock¹	ITScore^1,10(plus flexible ligand term)¹¹
B-1	0.95	0.47	0.55	0.60	1.43	−0.13	9	3	Glide-XP²	XP DockScore^2,16
B-2	0.95	0.34	0.55	0.78	1.43	0.00	repeat	Bl2	Repeat of B-1	XP DockScore²+ ROCS¹⁵shape similarity¹⁶
S-2	0.90	0.64	0.50	0.28	1.62	0.15	9	2	Glide-XP²	MMGBSA^2,100,101
I	1.08	0.64	0.81	0.25	1.81	−0.13	9	2	PLANTS³	Consensus (MedusaScore⁷⁵X-score⁷⁴DSX- DrugScore⁷³+ ChemPLP³)
G-2	1.21	0.51	1.25	0.09	1.32	0.15	9	2	Wilma⁴	SIE-FISH^77,105
C	1.34	0.47	1.59	0.30	1.73	−0.14	9	1	Gold⁵	GoldScore^5,78
S-1	0.90	0.20	0.50	0.38	1.62	−0.05	repeat	0	Repeat of S-2	XP DockScore²
X-5	1.32	0.29	0.95	0.27	1.74	0.34	9	0	SMINA⁶– Any Xtal^c	Autodock-Vina^12,13
J cross- dock	1.22	0.36	1.73	0.08	1.77	0.12	9	0	Glide XP²/PELE⁷	PELE⁷ + GB solvation energy⁸³ + ligand-strain term + conformational entropy term⁸⁴
G-1	1.21	0.06	0.61	0.10	1.31	0.13	9	0	Wilma⁴	SIE^4,77
V	1.07	0.23	1.67	0.54	—	—	8.5	2	FlexX⁸	HYDE¹⁴
N	—	—	1.75	−0.03	1.57	0.00	8.5	0	MedusaDock⁹	Consensus model of Random Forest + SVM¹⁰⁶
E	2.09	0.59	0.91	0.04	1.85	0.25	8	2	Unknown^d	Unknown^d
A-1	1.51	−0.31	2.17	−0.06	1.74	−0.06	8	0	SOL^98,99	FLM^98,99
A-2	1.84	0.41	2.13	0.29	2.07	0.26	7	1	SOL^98,99	SOL^98,99
D	2.22	0.61	2.23	0.12	2.15	0.19	6.5	2	Unknown^d	Unknown^d
K	2.21	0.18	1.16	0.31	2.75	0.11	6.5	0	IMG Dock¹⁰⁷	Customized empirical⁶⁰
X-1	1.23	0.41	0.78	0.63	4.77	0.25^e	6	3	Built in by hand ^g – Most Sim ^f	Autodock-Vina¹²+ visual inspection¹³
X-2	1.23	0.41	0.78	0.63	4.77	0.27^e	repeat	3	Repeat of X-1	Autodock-Vina^12,13
H	1.10	0.51	1.89	0.40	6.65	0.13^e	6	3	MedusaDock⁹	MedusaScore^75,79
X-4	1.32	0.54	4.46	0.07^e	1.74	0.36	6	2	SMINA⁶ – Most Sim ^f	Autodock-Vina^12,13
X-8	1.23	−0.18	0.78	0.32	4.77	0.23^e	repeat	0	Repeat of X-1	Autodock-Vina¹² divided by size¹³
X-3	1.10	0.18	1.07	0.04	4.69	0.28^e	6	0	Built in by hand ^g – Any Xtal ^c	Autodock-Vina^12,13
U-3	1.54	0.38	7.99	0.40^e	1.88	0.16	6	0	MDock¹	ITScore^1,10 (refit with Phase 1 complexes added to the refined PDBbind 2012 training data)⁸¹
Q	10.34	0.52^e	1.57	0.23	1.67	−0.02	6	0	Autodock-Vina¹²	ConvexPL⁷⁶
U-4	1.61	0.58	6.27	0.40^e	2.22	0.15	5	2	Autodock-Vina¹²	Autodock-Vina^12,81
U-1	1.42	0.67	9.20	0.31^e	2.29	−0.04	5	2	MDock¹	ITScore^1,10,81
U-2	1.41	0.16	3.73	0.54^e	2.27	0.11	5	0	MDock¹	ITScore^1,10 (refit to refined PDBbind 2012)⁸¹
F	2.22	0.61	2.61	0.08	2.50	0.32	4.5	2	Unknown^d	Unknown^d
L	1.95	0.60	6.02	−0.01^e	2.72	0.20	4	2	Autodock-Vina¹²	Autodock-Vina¹²
P	1.71	0.39	3.48	0.18^e	7.98	0.20^e	3	0	Rosetta Ligand^92,93	Rosetta talaris2013^92,93
W	—	—	2.41	−0.47	—	—	3	0	Gold⁵	Chemscore (kinase parameters)^108,109
T	2.61	0.25	4.12	−0.13^e	2.55	−0.08	2	0	Autodock⁶¹	Hybrid Autodock⁶¹/Autodock-Vina¹²
M cross- dock	2.73	0.13	3.83	−0.17^e	6.03	0.07^e	1	0	Pharmer⁷¹	Autodock-Vina¹² (in SMINA⁶) + fit to receptor pharmacophore model⁷²
M-2	—	—	3.83	0.21^e	—	—	repeat	0	Repeat of M-1	Fit to receptor pharmacophore model⁷²
Y cross- dock	6.14	0.46^e	5.73	0.13^e	8.92	−0.06^e	0	0	SPA Dock⁹⁵	SPA⁹⁵
O	—	0.37	—	0.23	—	−0.19	—	0	Not applicable	QSAR (unknown design^d)
X-7	—	0.05	—	0.05	—	0.29	—	0	Not applicable	Ligand-based (Support Vector Machine)¹³
X-6	—	−0.03	—	−0.06	—	0.19	—	0	Not applicable	Ligand-based (K-nearest neighbors)¹³
MW		0.39		0.21		0.12				Null model for ranking¹⁸
SlogP		0.44		−0.10		0.13				Null model for ranking¹⁸

Dashed numbers denote alternate methods from the same group. Many participants submitted more than one set of predictions to compare different approaches and parameters.

The overall evaluation of docking was calculated by adding points for each median RMSD across all three targets: 3 points for ≤2Å, 2 points for ≤2.4Å, 1 point for ≤3Å, or 0 for >3Å. The overall evaluation of ranking was calculated by adding points for each ρ across all three targets: 2 points for ≥0.5, 1 point for ≥0.4, or 0 for <0.4. Adjustments were made 1) for participants who submitted results for only two systems (groups V and N are not penalized), 2) for methods where docking was less than optimal for all three targets but none had RMSD >3Å (groups D, K, and F), and 3) for good rankings when the median RMSD was >3Å (no points for group U on SYK, no points for groups Q and Y for TrmD).

The crystal structure used for docking each ligand was randomly chosen from among those in the set provided to the applicants (compare to ^f below).

Some participants were unable to provide details about their methods.

ρ values are italicized when coupled with a median RMSD > 3Å. In these cases, ρ values are suspect.

Each ligand in the set was specifically docked to the receptor of the crystal structure with the most similar ligand bound.

The docked pose was built into the binding site using substructure alignment and minimization in SMINA(Vina).