Table 4.
Clinical IE-related NLP shared tasks.
| Shared Task | Year | Brief Description | No. of Partici pants | Best Participant Performance (F-measure) | Website |
|---|---|---|---|---|---|
| i2b2 de-identification and smoking challenge [103, 104] | 2006 | Automatic de-identification of personal health information and identification of patient smoking status. | 15 | De-identification: 0.98; Smoking identification: 0.90. | https://www.i2b2.org/NLP/DataSets/ |
| i2b2 obesity challenge [105] | 2008 | Identification of obesity and its co- morbidities. | 30 | 0.9773 | |
| i2b2 medication challenge [106] | 2009 | Identification of medications, their dosages, modes (routes) of administration, frequencies, durations, and reasons for administration in discharge summaries. | 20 | Durations identification: 0.525; Reason identification: 0.459. | |
| i2b2 relations challenge [107] | 2010 | Concept extraction, assertion classification and relation classification. | 30 | Concept extraction: 0.852; Assertion and relation classification: 0.936. | |
| i2b2 coreference challenge [108] | 2011 | Coreference resolution. | 20 | 0.827 | |
| i2b2 temporal relations challenge [109] | 2012 | Extraction of temporal relations in clinical records including three specific tasks: clinically significant events, temporal expressions and temporal relations. | 18 | Event: 0.92; Temporal expression: 0.90; Temporal relation: 0.69. | |
| i2b2 de-identification and heart disease risk factors challenge [110, 111] | 2014 | Automatic de-identification and identification of medical risk factors related to coronary artery disease in the narratives of longitudinal medical records of diabetic patients. | 10 | De-identification: 0.9586; Risk factor: 0.9276. | |
| CLEF eHealth shared task 1 [112] | 2013 | Named entity recognition in clinical notes. | 22 | 0.75 | https://sites.google.com/site/shareclefehealth/ |
| CLEF eHealth shared task 2 [113] | 2014 | Normalization of acronyms or abbreviations. | 10 | Task 2a: 0.868 (accuracy); Task 2b: 0.576 (F-measure). | https://sites.google.com/site/clefehealth2014/task-2 |
| CLEF eHealth shared task 1b [114] | 2015 | Clinical named entity recognition from French medical text. | 7 | Plain entity recognition: 0.756; Normalized entity recognition: 0.711; Entity normalization: 0.872. | https://sites.google.com/site/clefehealth2015/task-1/task-1b |
| CLEF eHealth shared task 2 [115] | 2016 | Clinical named entity recognition from French medical text. | 7 | Plain entity recognition: 0.702; Normalized entity recognition: 0.529; Entity normalization: 0.524. | https://sites.google.com/site/clefehealth2016/task-2 |
| SemEval task 9 [116] | 2013 | Extraction of drug-drug interactions from biomedical texts. | 14 | Recognition of drugs: 0.715; Extraction of drug-drug interactions: 0.651. | https://www.cs.york.ac.uk/semeval-2013/task9.html |
| SemEval task 7 [117] | 2014 | Identification and normalization of diseases and disorders in clinical reports. | 21 | Identification: 0.813; Normalization: 0.741 (accuracy). | http://alt.qcri.org/semeval2014/task7/index.php?id=task-description |
| SemEval task 14 [118] | 2015 | Named entity recognition and template slot filling for clinical texts. | 16 | Named entity recognition: 0.757; Template slot filling: 0.886 (accuracy); Disorder recognition and template slot filling: 0.808 (accuracy). | http://alt.qcri.org/semeval2015/task14/ |
| SemEval task 12 [119] | 2016 | Temporal information extraction from clinical texts including time expression identification, event expression identification and temporal relation identification. | 14 | Time expression identification: 0.795; Event expression identification: 0.903; Temporal relation identification: 0.573. | http://alt.qcri.org/semeval2016/task12/ |