Table 7.
Evaluation outcomes of the different implementations and datasets
| Named-entity recognition | Attribute relation detection | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Implementation | Set | Approach | Outcome | Lung cancera | Non-small cell lung cancerb | Recurrencec | Recurrent lung cancerd | |||
| Present | Absent | Present | Absent | Present | Present | Absent | ||||
| 1 | Development set (n = 30) | Relaxed | Detected | 15 | 1 | 11 | 1 | 0 | ||
| cTAKES with UMLS 2011 | Not detected | 12 | 2 | 6 | 12 | 15 | ||||
| Strict | Detected | 15 | 1 | 11 | 1 | |||||
| Not detected | 8 | 6 | 6 | 12 | ||||||
| Test set (n = 68) | Relaxed | Detected | 35 | 0 | 29 | 1 | 0 | |||
| Not detected | 16 | 17 | 7 | 31 | 30 | |||||
| Strict | Detected | 33 | 2 | 29 | 1 | |||||
| Not detected | 7 | 26 | 2 | 36 | ||||||
| 2 | Development set (n = 30) | Relaxed | Detected | 10 | 0 | 5 | 0 | 0 | ||
| cTAKES with UMLS 2016 | Not detected | 17 | 3 | 12 | 13 | 15 | ||||
| Strict | Detected | 10 | 0 | 5 | 0 | |||||
| Not detected | 13 | 7 | 12 | 13 | ||||||
| Test set (n = 68) | Relaxed | Detected | 29 | 0 | 18 | 1 | 0 | |||
| Not detected | 22 | 17 | 18 | 31 | 30 | |||||
| Strict | Detected | 29 | 0 | 18 | 1 | |||||
| Not detected | 11 | 28 | 13 | 36 | ||||||
| 3 | Development set (n = 30) | Relaxed | Detected | 23 | 0 | 13 | 0 | 15 | ||
| cTAKES with UMLS 2016 and custom dictionary | Not detected | 4 | 3 | 4 | 13 | 0 | ||||
| Strict | Detected | 23 | 0 | 13 | 0 | |||||
| Not detected | 0 | 7 | 4 | 13 | ||||||
| Test set (n = 68) | Relaxed | Detected | 36 | 0 | 22 | 1 | 30 | |||
| Not detected | 15 | 17 | 14 | 31 | 0 | |||||
| Strict | Detected | 36 | 0 | 22 | 1 | |||||
| Not detected | 4 | 28 | 9 | 36 | ||||||
| 4 | Development set (n = 30) | Relaxed | Detected | 23 | 0 | 17 | 0 | 15 | 6 | 0 |
| DIRECT cTAKES with UMLS 2016, custom dictionary, and post-processing | Not detected | 4 | 3 | 0 | 13 | 0 | 0 | 10 | ||
| Strict | Detected | 23 | 0 | 17 | 0 | |||||
| Not detected | 0 | 7 | 0 | 13 | ||||||
| Test set (n = 68) | Relaxed | Detected | 36 | 0 | 28 | 1 | 29 | 9 | 0 | |
| Not detected | 15 | 17 | 8 | 31 | 1 | 3 | 18 | |||
| Strict | Detected | 36 | 0 | 28 | 1 | |||||
| Not detected | 4 | 28 | 3 | 36 | ||||||
aSNOMED CT concept 93880001 | Primary malignant neoplasm of lung (disorder)
bSNOMED CT concept 254637007 | Non-small cell lung cancer (disorder)
cSNOMED CT concept 255227004 | Recurrent (qualifier value)
dRelationship between three SNOMED CT concepts: 93880001 | Primary malignant neoplasm of lung (disorder) |: 263502005 | Clinical course (attribute) | = 255227004 | Recurrent (qualifier value)