Table 3.

NLP models for MIDD

Model	Full form	Pretrained on	Architecture	Built on	Performance	Year
BioBERT62,63	Bio-Bidirectional Encoder Representations from Transformers	PubMed and PMC	Transformer	BERT	Outperforms state-of-the-art (SOTA) for named entity recognition, relation extraction, question answering	September 19
SciBERT64,65	Science—Bidirectional Encoder Representations from Transformers	Semantic Scholar	Transformer	BERT	Outperforms SOTA for named entity recognition, relation extraction, patient enrollment task	November 19
ClinicalBERT66,67	Clinical Bidirectional Encoder Representations from Transformers	MIMIC III	Transformer	BERT	Outperforms deep language model for clinical prediction	November 20
BioClinicalBERT68,69	Bio-Clinical Bidirectional Encoder Representations from Transformers	MIMIC III	Transformer	BioBERT	Outperforms BERT and BioBERT on named entity recognition and natural language inference	June 19
BioMed-RoBERTa70,71	BioMedical Robustly optimized Bidirectional Encoder Representations from Transformers	Semantic Scholar	Transformer	RoBERTa	Outperforms RoBERTa on text classification, relation extraction and named entity recognition	May 20
Bio Discharge Summary BERT69,72	Bio Discharge Summary Bidirectional Encoder Representations from Transformers	MIMIC III discharge summaries	Transformer	BioBERT	Outperforms BERT and BioBERT on named entity recognition and natural language inference	June 19
BioALBERT73	Bio-A Lite Bidirectional Encoder Representations from Transformers	PubMed, PMC, MIMIC III	Transformer	ALBERT	Outperforms SOTA for named entity recognition, relation extraction, question answering, sentence similarity, document classification	July 21
ChemBERTa74,75	Chem-Bidirectional Encoder Representations from Transformers	PubChem	Transformer	RoBERTa	Outperforms baseline on one task of molecular property prediction	October 20