Table 5.
Evaluation results on several datasets and several metrics
| Dataset | System | MAP | MRR | nDCG | # Best Rel vs IrRel |
|---|---|---|---|---|---|
| TREC PM 2017 | KeywordScore | 0.0006 | 0.066 | 0.426 | 2 |
| PubMed | 0.0008 | 0.056 | 0.585 | 5 | |
| VIST MTL | 0.0003 | 0.051 | 0.238 | 20* | |
| VIST SVM | 0.0008 | 0.095 | 0.458 | 20* | |
| Tumorboard | KeywordScore | 0.0082 | 0.011 | 0.115 | - |
| PubMed | 0.0489 | 0.070 | 0.230 | - | |
| VIST MTL | 0.0242 | 0.035 | 0.103 | - | |
| VIST SVM | 0.0579 | 0.081 | 0.220 | - | |
| UserStudy | KeywordScore | 0.0631 | 0.296 | 0.645 | 2 |
| PubMed | 0.0847 | 0.236 | 0.580 | 3 | |
| VIST MTL | 0.0571 | 0.239 | 0.407 | 9* | |
| VIST SVM | 0.1874 | 0.650 | 0.933 | 9* |
Low values are due to a small number of known PMIDs for individual queries. “# best Rel vs IrRel”: Number of queries for which the corresponding system has the best “Rel vs IrRel” score (27 queries for TREC PM 2017, 14 queries for UserStudy). *VIST SVM and VIST MTL are compared separately with KeywordScore and PubMed. KeywordScore is the ranking provided in the default settings of Solr