Table 3.
Automatic limitation recognition results with varying training data composition and size. The results shown are for the SVM classifier. POS: NEG ratio of 1: 1 indicates a balanced dataset. The SEED-TEST split we used to obtain the results in Table 2 is shown in italics (33.3).
| Method | Precision | Recall | F1 score | Accuracy |
|---|---|---|---|---|
| Undersampling NEG instances | ||||
| POS: NEG Ratio | ||||
| 1: 1 | 62.4 | 89.4 | 73.5 | 87.0 |
| 1: 2 | 71.2 | 79.2 | 75.0 | 89.4 |
| 1: 3 | 75.7 | 73.9 | 74.8 | 90.0 |
| Training split size as a proportion of the annotated dataset (SEED+TEST) | ||||
| Training Pct. | ||||
| 10 | 74.5 | 67.6 | 70.9 | 86.9 |
| 20 | 75.8 | 69.4 | 72.5 | 87.5 |
| 30 | 78.6 | 71.3 | 74.8 | 88.6 |
| 33.3 | 77.8 | 71.3 | 74.4 | 88.4 |
| 40 | 78.0 | 72.2 | 75.0 | 88.6 |
| 55 | 81.8 | 75.0 | 78.3 | 90.2 |
| 70 | 83.5 | 70.4 | 76.4 | 89.7 |
| 80 | 83.3 | 69.4 | 75.8 | 89.5 |