Table 4.
Performance of the two-tier extraction system: our rule-based system in the first tier
| Min #Inst. per Relation | #Uniq. Relation | #Instances | Methods | Per-Relation | Per-Instance | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pos | Neg | Pos | Neg | TP | FP | Precision | Recall | F-Score | TP | FP | Precision | Recall | F-Score | ||
| ours | 132 | 7 | 0.950 | 0.214 | 0.349 | 153 | 10 | 0.939 | 0.153 | 0.263 | |||||
| +hybrid | 477 | 326 | 0.594 | 0.772 | 0.671 | 701 | 555 | 0.558 | 0.701 | 0.621 | |||||
| +SST-PT | 231 | 112 | 0.673 | 0.374 | 0.481 | 287 | 167 | 0.632 | 0.287 | 0.395 | |||||
| 1 | 618 | 2,312 | 1,000 | 4,834 | +SVM | 152 | 44 | 0.776 | 0.246 | 0.374 | 172 | 67 | 0.720 | 0.172 | 0.278 |
| +NB | 247 | 254 | 0.493 | 0.400 | 0.442 | 285 | 410 | 0.410 | 0.285 | 0.336 | |||||
| +DT | 155 | 40 | 0.795 | 0.251 | 0.382 | 180 | 67 | 0.729 | 0.180 | 0.289 | |||||
| +kNN | 215 | 226 | 0.488 | 0.348 | 0.406 | 242 | 310 | 0.438 | 0.242 | 0.312 | |||||
| ours | 80 | 3 | 0.964 | 0.406 | 0.571 | 101 | 5 | 0.953 | 0.174 | 0.294 | |||||
| +hybrid | 183 | 151 | 0.548 | 0.929 | 0.689 | 395 | 310 | 0.560 | 0.682 | 0.615 | |||||
| +SST-PT | 127 | 62 | 0.672 | 0.645 | 0.658 | 186 | 102 | 0.646 | 0.321 | 0.429 | |||||
| 2 | 197 | 695 | 579 | 2,827 | +SVM | 94 | 36 | 0.723 | 0.477 | 0.575 | 116 | 63 | 0.648 | 0.200 | 0.306 |
| +NB | 113 | 110 | 0.507 | 0.574 | 0.538 | 151 | 196 | 0.435 | 0.261 | 0.326 | |||||
| +DT | 90 | 21 | 0.811 | 0.457 | 0.585 | 112 | 45 | 0.713 | 0.193 | 0.304 | |||||
| +kNN | 129 | 138 | 0.483 | 0.655 | 0.556 | 163 | 235 | 0.410 | 0.282 | 0.334 | |||||
| ours | 51 | 2 | 0.962 | 0.573 | 0.718 | 67 | 3 | 0.957 | 0.185 | 0.310 | |||||
| +hybrid | 88 | 94 | 0.484 | 0.989 | 0.650 | 257 | 222 | 0.537 | 0.708 | 0.611 | |||||
| +SST-PT | 70 | 38 | 0.648 | 0.787 | 0.711 | 129 | 58 | 0.690 | 0.355 | 0.469 | |||||
| 3 | 89 | 314 | 363 | 1,835 | +SVM | 58 | 22 | 0.725 | 0.652 | 0.687 | 78 | 32 | 0.709 | 0.215 | 0.330 |
| +NB | 59 | 54 | 0.522 | 0.663 | 0.584 | 86 | 89 | 0.491 | 0.237 | 0.320 | |||||
| +DT | 53 | 5 | 0.914 | 0.596 | 0.722 | 70 | 8 | 0.897 | 0.193 | 0.318 | |||||
| +kNN | 66 | 85 | 0.437 | 0.742 | 0.550 | 97 | 153 | 0.388 | 0.267 | 0.316 | |||||
| ours | 35 | 2 | 0.946 | 0.686 | 0.795 | 50 | 3 | 0.943 | 0.201 | 0.331 | |||||
| +hybrid | 51 | 63 | 0.447 | 1 | 0.618 | 171 | 150 | 0.533 | 0.687 | 0.600 | |||||
| +SST-PT | 45 | 27 | 0.625 | 0.882 | 0.732 | 92 | 43 | 0.681 | 0.369 | 0.479 | |||||
| 4 | 51 | 181 | 249 | 1,349 | +SVM | 38 | 15 | 0.717 | 0.745 | 0.731 | 55 | 25 | 0.688 | 0.221 | 0.335 |
| +NB | 42 | 43 | 0.494 | 0.824 | 0.618 | 67 | 86 | 0.438 | 0.269 | 0.333 | |||||
| +DT | 36 | 5 | 0.878 | 0.706 | 0.783 | 54 | 8 | 0.871 | 0.217 | 0.347 | |||||
| +kNN | 44 | 56 | 0.440 | 0.863 | 0.583 | 74 | 104 | 0.416 | 0.297 | 0.347 | |||||
| ours | 23 | 1 | 0.958 | 0.821 | 0.884 | 37 | 1 | 0.974 | 0.236 | 0.380 | |||||
| +hybrid | 28 | 40 | 0.412 | 1 | 0.584 | 108 | 88 | 0.551 | 0.688 | 0.612 | |||||
| +SST-PT | 25 | 14 | 0.641 | 0.893 | 0.746 | 57 | 22 | 0.722 | 0.363 | 0.483 | |||||
| 5 | 28 | 107 | 157 | 984 | +SVM | 25 | 12 | 0.676 | 0.893 | 0.769 | 40 | 18 | 0.690 | 0.255 | 0.372 |
| +NB | 24 | 21 | 0.533 | 0.857 | 0.657 | 40 | 40 | 0.500 | 0.255 | 0.338 | |||||
| +DT | 23 | 4 | 0.852 | 0.821 | 0.836 | 37 | 8 | 0.822 | 0.236 | 0.367 | |||||
| +kNN | 26 | 31 | 0.456 | 0.929 | 0.612 | 45 | 51 | 0.469 | 0.287 | 0.356 | |||||