Table 3.
Precision, recall and calculated F-scores from the evaluation outcomes
| Development set (n = 30) | Test set (n = 68) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Implementation | Algorithm | Concept | Approach | Precision | Recall | F-score | Precision | Recall | F-score |
| UMLS2011 | Named-entity recognition | Lung cancera | Implied | 0.938 | 0.556 | 0.698 | 1.000 | 0.686 | 0.814 |
| cTAKES with UMLS 2011 | Strict | 0.938 | 0.652 | 0.769 | 0.943 | 0.825 | 0.880 | ||
| Non-small cell lung cancerb | Implied | 0.917 | 0.647 | 0.759 | 0.967 | 0.806 | 0.879 | ||
| Strict | 0.917 | 0.647 | 0.759 | 0.967 | 0.935 | 0.951 | |||
| Recurrencec | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |||
| UMLS2016 | Named-entity recognition | Lung cancera | Implied | 1.000 | 0.370 | 0.541 | 1.000 | 0.569 | 0.725 |
| cTAKES with UMLS 2016 | Strict | 1.000 | 0.435 | 0.606 | 1.000 | 0.725 | 0.841 | ||
| Non-small cell lung cancerb | Implied | 1.000 | 0.294 | 0.455 | 0.947 | 0.500 | 0.655 | ||
| Strict | 1.000 | 0.294 | 0.455 | 0.947 | 0.581 | 0.720 | |||
| Recurrencec | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |||
| UMLS2016 + Dict. | Named-entity recognition | Lung cancera | Implied | 1.000 | 0.852 | 0.920 | 1.000 | 0.706 | 0.828 |
| cTAKES with UMLS 2016 and custom dictionary | Strict | 1.000 | 1.000 | 1.000 | 1.000 | 0.900 | 0.947 | ||
| Non-small cell lung cancerb | Implied | 1.000 | 0.765 | 0.867 | 0.957 | 0.611 | 0.746 | ||
| Strict | 1.000 | 0.765 | 0.867 | 0.957 | 0.710 | 0.815 | |||
| Recurrencec | 1.000 | 1.000 | 1.000 | 0.882 | 1.000 | 0.938 | |||
| DIRECT | Named-entity recognition | Lung cancera | Implied | 1.000 | 0.852 | 0.920 | 1.000 | 0.706 | 0.828 |
| cTAKES with UMLS 2016, custom dictionary, and additional processing | Strict | 1.000 | 1.000 | 1.000 | 1.000 | 0.900 | 0.947 | ||
| Non-small cell lung cancerb | Implied | 1.000 | 1.000 | 1.000 | 0.966 | 0.778 | 0.862 | ||
| Strict | 1.000 | 1.000 | 1.000 | 0.966 | 0.903 | 0.933 | |||
| Recurrencec | 1.000 | 1.000 | 1.000 | 0.879 | 0.967 | 0.921 | |||
| Attribute relationship detection | Recurrent lung cancerd | 1.000 | 1.000 | 1.000 | 1.000 | 0.750 | 0.857 | ||
aSNOMED CT concept 93880001 | Primary malignant neoplasm of lung (disorder)
bSNOMED CT concept 254637007 | Non-small cell lung cancer (disorder)
cSNOMED CT concept 255227004 | Recurrent (qualifier value)
dRelationship between three SNOMED CT concepts: 93880001 | Primary malignant neoplasm of lung (disorder) |: 263502005 | Clinical course (attribute) | = 255227004 | Recurrent (qualifier value)