Table 7.

The lenient F1 scores for named entity recognition of single and state-of-the-art ensemble models compared with our best model. The lenient F1 scores for relation extraction of state-of-the-art ensemble models with and without rules, compared with our best model.

NERa					Relation extraction
Entity type	BiLSTM-CRFb [38]	Committee [38]	Knowledge-joint	Relation type		Committee + CNN-RNNc [38]	Committee + CNN-RNN + Rules [38]	Knowledge-joint
Drug	0.955	0.956	0.960	N/Ad		N/A	N/A	N/A
Strength	0.982	0.983	0.980	Drug−strength		0.964	0.972	0.975
Form	0.958	0.958	0.958	Drug−form		0.940	0.952	0.954
Frequency	0.974	0.975	0.971	Drug−frequency		0.941	0.958	0.962
Route	0.956	0.956	0.949	Drug−route		0.930	0.942	0.937
Dosage	0.943	0.948	0.946	Drug−dosage		0.923	0.935	0.937
Duration	0.856	0.862	0.846	Drug−duration		0.740	0.786	0.779
ADEe	0.422	0.530	0.535	Drug−ADE		0.475	0.476	0.490
Reason	0.680	0.675	0.727	Drug−reason		0.572	0.579	0.650
Overall (micro)	0.933	0.935	0.935	Overall (micro)		0.879	0.891	0.895

^aNER: named entity recognition.

^bBiLSTM-CRF: bidirectional long short-term memory–conditional random field.

^cCNN-RNN: convolutional neural network–recurrent neural network.

^dNot applicable.

^eADE: adverse drug event.