Table 1. Clinical NLP Challenges, the tasks they posed, and the number of participating teams, since 2015, ordered by data sensitivity.
| Category | Year | Challenge name | Task description | Data type | Data source | teams type | |||
|---|---|---|---|---|---|---|---|---|---|
| Academia | Industry | Joint | Total | ||||||
| 2015 | TREC Clinical Decision Support (CDS) 9 | Paticnt-ccntcrcd information retrieval | Medical case narratives | Synthetic, PubMed | 33 | 3 | 0 | 36 | |
| TREC Precision Medicine 11 | |||||||||
| Synthetic | 2017 | > Track 1 > Track 2 |
Patient-centered literature article retrieval Patient-centered clinical trials retrieval | Semi-structured cases | Synthetic, PubMed, ClinicalTrials.gov | 27 | 5 | 0 | 32 |
| 2016 | CLEF cHealth 12 | Information extraction | Nursing handover notes | NICTA synthetic nursing handover notes | 4 | 0 | 0 | 4 | |
| Text Analysis Conference (TAC) Adverse Drug Reaction Extraction from Drug Labels (ADR) 18 | |||||||||
| > Track 1 | ADR mentions and modifiers extraction | ||||||||
| Prescription drug labels | 2017 | > Track 2 > Track 3 > Track 4 |
Relation extraction Positive ADR filtering Positive ADR normalization | Drug labels | Drugs-Library.com | 6 | 3 | 1 | 10 |
| 2015 | CLPsych: Depression and PTSD on Twitter 22 | Binary classification of depression and PTSD users | Social media | 3 | 0 | 0 | 3 | ||
| Social Media Mining (SMM) 24 | |||||||||
| 2016 | > Track 1 > Track 2 > Track 3 |
ADR classification Information extraction Concept normalization | Social media | 9 | 2 | 0 | 11 | ||
| Online social data | 2017 |
Social Media Mining for Health Applications (SMM4HA)
29
> Track 1 > Track 2 > Hack 3 |
ADR classification Classification of medication intake Concept normalization | Social media | 12 | 1 | 0 | 13 | |
| 2016 | CLPsych: Triaging content in online peer-support forums 33 | Classification of mental health severity in 4 levels | Forum | RcachOut | 13 | 1 | 1 | 15 | |
| 2017 | CLPsych: Triaging content in online peer-support forums 35 | Classification of mental health severity in 4 levels | Forum | RcachOut | 12 | 2 | 1 | 15 | |
| 2017 | NTCIR-13 MedWeb 36 | 8-class classification of diseases and symptoms | Multilingual Social media | 7 | 1 | 1 | 9 | ||
| Analysis of Clinical Text (ACT) 39 | |||||||||
| 2015 | > Track 1 > Track 2a > Track 2b |
Disorder NER and normalization Template slot filling (given gold spans) Disorder recognition and template slot filling (end-to-end) |
Clinical notes | ShARc corpus (MIMIC) | 18 | 3 | 0 | 21 | |
| 2016 | TREC Clinical Decision Support (CDS) 43 | Paticnt-ccntcrcd IR | Nursing admission notes | MIMIC, PubMed | 21 | 5 | 0 | 26 | |
| Medication and Adverse Drug Events (MADE1.0) | |||||||||
| 2017 | > Track 1 > Track 2 |
Medication, ADE, sign and symptom identification Relation extraction | Clinical notes | UMass Memorial Medical Center | |||||
| Clinical TempEval 45 | |||||||||
| > Track 1 | Time expression extraction | ||||||||
| 2015 | > Track 2 > Track 3 |
Event extraction Relation extraction (wrt DCT) Relation extraction (wrt narrative containers) |
Pathology reports | Mayo Clinic | 3 | 0 | 0 | 3 | |
| Clinical data | Clinical TempEval 46 > Track 1 | Time expression extraction | |||||||
| 2016 | > Track 2 > Track 3 |
Event extraction Relation extraction (wrt DCT) Relation extraction (wrt narrative containers) |
Pathology reports | Mayo Clinic | U | 3 | 0 | 14 | |
| Clinical TempEval 48 | |||||||||
| 2017 | > Track 1 > Track 2 |
Time expression extraction (cross-domain) Event extraction (cross-domain) | Pathology reports, Clinical notes | Mayo Clinic | 9 | 2 | 0 | 11 | |
| > Track 3 | Relation extraction (wrt DCT) Relation extraction (wrt narrative containers) |
||||||||
| Centers for Excellence in Genomics N-GRJD (CEGS-NGR1D) 51 | |||||||||
| 2016 | > Track la > Track lb > Track 2 |
De-identification (cross-domain) Dc-identification Psychiatric Symptom Severity Prediction |
Psychiatric evaluation records | Partners Healthcare and Harvard Medical School | 23 | 5 | 3 | 31 | |