Table 2. List of shared tasks with data source, data size, sub-tasks descriptions, and best-performance score (metrics differ per challenge). The table also contains information about data availability after the challenge, whether the data have been de-identified, and whether they require a DUA to be signed.

Category	Year	Challenge name	Task description	Data type	Data source	Data size	De-identification / anonymization	DUA	Currently Available?	Best Performance	Measure
	2015	TREC Clinical Decision Support (CDS) 9	Patient-centered information retrieval	Medical case narratives	Synthetic, PubMed	30 topics, 730K articles	no	no	yes	38.21%	infNDCG
		TREC Precision Medicine 11
Synthetic	2017	> Track 1 > Track 2	Patient-centered literature article retrieval Patient-centered clinical trials retrieval	Semi-structured cases	Synthetic, PubMed, ClinicalTrials.gov	30 topics, 27M abstracts, 241K trials	no	no	yes	63.10% 44.29%	P@10 P@10
	2016	CLEF cHcalth 12	Information extraction	Nursing handover notes	NICTA synthetic nursing handover notes	300 notes	no	no	yes	38.20%	Fl (macro avg.)
		Text Analysis Conference (TAC) Adverse Drug Reaction Extraction from Drug Labels (ADR) 18
Prescription drug labels	2017	> Track 1 > Track 2 > Track 3 > Track 4	ADR mentions and modifiers extraction Relation extraction Positive ADR filtering Positive ADR normalization	Drug labels	Drugs-Library.com	2309 labels	no	no	yes	82.48% 49.00% 82.19% 85.33%	Fl Fl Fl (macro avg.) Fl (macro avg.)
	2015	CLPsych: Depression and PTSD on Twitter 22	Binaty classification of depression and PTSD users	Social media	Twitter	7.8M tweets	yes	yes	yes	80.00%	Avg. Precision
		Social Media Mining (SMM) 24
	2016	> Track 1 > Track 2 > Track 3	ADR classification Information extraction Concept normalization	Social media	Twiner	10,882 tweets	no	no	yes	41.95% 61,10% -	Fl Fl
		Social Media Mining for Health Applications (SMM4HA) 29
Online social data	2017	> Track 1 > Track 2 > Track 3	ADR classification Classification of medication intake Concept normalization	Social media	Twitter	15,777 tweets	no	no	yes	43.50% 69.30% 88.50%	Fl Fl (micro avg.) Accuracy
	2016	CLPsych: Triaging content in online peer-support forums 33	Classification of mental health severity in 4 levels	Forum	ReachOut	65,024 (1,227 annotated)	yes	yes	yes, on request	42.00%	Fl (macro avg.)
	2017	CLPsych: Triaging content in online peer-support forums 35	Classification of mental health severity in 4 levels	Forum	ReachOut	157,963 posts (1,588 annotated)	yes	yes	yes, on request	46.70%	Fl (macro avg.)
	2017	NTCIR-13 MedWeb 36	8-class classification of diseases and symptoms	Multilingual Social media	Twitter	2560 tweets	yes	yes	yes, on request	-
		Analysis of Clinical Text (ACT) 39
	2015	> Track 1 > Track 2a > Track 2b	Disorder NER and normalization Template slot filling (given gold spans) Disorder recognition and template slot filling (end-to-end)	Clinical notes	ShARc corpus (MIMIC)	531 summaries	yes	yes	yes	75.70% 88.60% 80.80%	Fl (strict) Fl * weighted acc. Fl * weighted acc.
	2016	TREC Clinical Decision Support (CDS) 43	Patient-centered IR	Nursing admission notes	MIMIC, PubMed	30 notes, 1.25M abstracts				40.33%	P@10
		Medication and Adverse Drug Events (MADE1.0)
	2017	> Track 1 > Track 2	Medication, ADE, sign and symptom identification Relation extraction	Clinical notes	UMass Memorial Medical Center	1092 records	yes	yes	no	-
		Clinical TempEval 45
	2015	> Track 1 > Track 2 > Track 3	Time expression extraction Event extraction Relation extraction (wrt DCT) Relation extraction (wrt narrative containers)	Pathology reports	Mayo Clinic	600 notes	yes	yes	yes, on request	72.50% 87.50% 70.20% 12.30%	Fl Fl Fl Fl
Clinical data		Clinical TempEval 46
	2016	> Track 1 > Track 2 > Track 3	Time expression extraction Event extraction Relation extraction (wrt DCT) Relation extraction (wrt narrative containers)	Pathology reports	Mayo Clinic	600 notes	yes	yes	yes, on request	79.50% 90.30% 75.60% 47.90%	Fl Fl Fl Fl
		Clinical TempEval 48
	2017	> Track 1 > Track 2 > Track 3	Time expression extraction (cross-domain) Event extraction (cross-domain) Relation extraction (wrt DCT) Relation extraction (wrt narrative containers)	Pathology reports, Clinical notes	Mayo Clinic	1216 notes	yes	yes	yes, on request	57.00% 72.00% 59.00% 32.50%	Fl Fl Fl Fl
		Centers for Excellence in Genomics N-GRID (CEGS-NGRID) 51
	2016	> Track la > Track lb > Track 2	De-identification (cross-domain) Dc-identification Psychiatric Symptom Severity Prediction	Psychiatric evaluation records	Partners Healthcare and Harvard Medical School	1000 records	yes	yes	yes, on request	79.85% 91.43% 86.30%	Fl Fl INMAE^M