Table 7.
Learn to rank on training data
| Learn to rank performance | Individual system contribution | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Top-K | Strategy | Model | P | R | F1 | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 |
| Top-1 | Union | SVMRank | 0.1513 | 0.5960 | 0.2413 | 0.25 | 0.04 | 0.45 | 0.01 | 0.00 | 0.01 | 0.00 | - | 0.23 |
| ListNet | 0.1153 | 0.5880 | 0.1928 | 0.73 | 0.03 | 0.21 | 0.02 | - | 0.00 | - | - | - | ||
| RankNet | 0.0924 | 0.5206 | 0.1570 | 1.00 | - | - | - | - | - | - | - | - | ||
| RankBoost | 0.1296 | 0.6125 | 0.2139 | 0.46 | 0.05 | 0.50 | - | - | 0.01 | 0.28 | 0.28 | 0.28 | ||
| Oracle | SVMRank | 0.1513 | 0.5960 | 0.2413 | 0.25 | 0.04 | 0.45 | 0.01 | 0.00 | 0.01 | 0.00 | - | 0.23 | |
| ListNet | 0.1153 | 0.5880 | 0.1928 | 0.73 | 0.03 | 0.21 | 0.02 | 0.00 | - | - | - | |||
| RankNet | 0.0924 | 0.5206 | 0.1570 | 1.00 | - | - | - | - | - | - | - | - | ||
| RankBoost | 0.1791 | 0.6113 | 0.2770 | 0.27 | 0.04 | 0.49 | - | - | 0.01 | 0.18 | 0.00 | - | ||
| Top-2 | Union | SVMRank | 0.1122 | 0.6426 | 0.1911 | 0.58 | 0.07 | 0.66 | 0.03 | 0.00 | 0.02 | 0.00 | 0.23 | 0.40 |
| ListNet | 0.0996 | 0.6566 | 0.1730 | 0.94 | 0.06 | 0.88 | 0.10 | 0.01 | 0.01 | 0.00 | - | - | ||
| RankNet | 0.0989 | 0.6477 | 0.1716 | 1.00 | - | 1.00 | - | - | - | - | - | |||
| RankBoost | 0.1084 | 0.6469 | 0.1857 | 0.55 | 0.09 | 0.61 | 0.01 | 0.00 | 0.28 | 0.76 | 0.76 | - | ||
| Oracle | SVMRank | 0.2340 | 0.6316 | 0.3415 | 0.09 | 0.00 | 0.67 | 0.01 | 0.00 | 0.01 | - | - | 0.23 | |
| ListNet | 0.2390 | 0.6439 | 0.3486 | 0.19 | 0.00 | 0.75 | 0.05 | 0.01 | 0.00 | - | - | - | ||
| RankNet | 0.2533 | 0.6363 | 0.3624 | 0.17 | - | 0.83 | - | - | - | - | - | - | ||
| RankBoost | 0.2385 | 0.6359 | 0.3469 | 0.11 | 0.00 | 0.57 | - | - | 0.02 | 0.29 | 0.00 | - | ||
| Top-3 | Union | SVMRank | 0.1051 | 0.6545 | 0.1811 | 0.61 | 0.18 | 0.68 | 0.07 | 0.02 | 0.03 | 0.24 | 0.40 | 0.77 |
| ListNet | 0.0921 | 0.6761 | 0.1621 | 0.97 | 0.60 | 0.96 | 0.40 | 0.03 | 0.03 | 0.01 | - | - | ||
| RankNet | 0.0943 | 0.6486 | 0.1647 | 1.00 | 1.00 | 1.00 | - | - | - | - | - | |||
| RankBoost | 0.1048 | 0.6532 | 0.1806 | 0.56 | 0.11 | 0.63 | 0.29 | 0.23 | 0.75 | 0.97 | 0.97 | - | ||
| Oracle | SVMRank | 0.2469 | 0.6409 | 0.3565 | 0.07 | 0.00 | 0.62 | 0.02 | 0.01 | 0.01 | 0.00 | - | 0.28 | |
| ListNet | 0.2716 | 0.6596 | 0.3848 | 0.13 | 0.00 | 0.72 | 0.14 | 0.01 | 0.00 | - | - | - | ||
| RankNet | 0.2536 | 0.6367 | 0.3627 | 0.17 | 0.00 | 0.83 | - | - | - | - | - | - | ||
| RankBoost | 0.2553 | 0.6397 | 0.3650 | 0.10 | 0.00 | 0.56 | - | 0.09 | 0.02 | 0.23 | 0.00 | - | ||
Macro precision, recall and F1 at different top K levels. The highest scoring system F1 for each level (both union and oracle strategies) is shown in bold. The table also shows the individual contribution of the systems to the final score where italics scores indicate the highest contributing individual system(s) to each ensemble.