Deep learning in clinical natural language processing: a methodical review

Abstract

Objective

This article methodically reviews the literature on deep learning (DL) for natural language processing (NLP) in the clinical domain, providing quantitative analysis to answer 3 research questions concerning methods, scope, and context of current research.

Materials and Methods

We searched MEDLINE, EMBASE, Scopus, the Association for Computing Machinery Digital Library, and the Association for Computational Linguistics Anthology for articles using DL-based approaches to NLP problems in electronic health records. After screening 1,737 articles, we collected data on 25 variables across 212 papers.

Results

DL in clinical NLP publications more than doubled each year, through 2018. Recurrent neural networks (60.8%) and word2vec embeddings (74.1%) were the most popular methods; the information extraction tasks of text classification, named entity recognition, and relation extraction were dominant (89.2%). However, there was a “long tail” of other methods and specific tasks. Most contributions were methodological variants or applications, but 20.8% were new methods of some kind. The earliest adopters were in the NLP community, but the medical informatics community was the most prolific.

Discussion

Our analysis shows growing acceptance of deep learning as a baseline for NLP research, and of DL-based NLP in the medical community. A number of common associations were substantiated (eg, the preference of recurrent neural networks for sequence-labeling named entity recognition), while others were surprisingly nuanced (eg, the scarcity of French language clinical NLP with deep learning).

Conclusion

Deep learning has not yet fully penetrated clinical NLP and is growing rapidly. This review highlighted both the popular and unique trends in this active field.

INTRODUCTION

Technical research is changing rapidly. Deep learning (DL) techniques have begun to dominate because of their simplicity (no need for handcrafted features), efficient processing (assuming dedicated, massively parallelized hardware), and state-of-the-art results (on a plethora of tasks). Meanwhile, the widespread adoption of electronic health records (EHRs) has produced massive amounts of digital text concerning patients, and the medical informatics community has invested substantial effort to make use of clinical text via natural language processing (NLP). Furthermore, research manuscripts themselves are coming under greater scrutiny for rigor and reproducibility,^1–3 yet they are simultaneously being generated on preprint servers with more momentum and less oversight than ever before.

This work aims to characterize the relationship between DL techniques and the field of clinical NLP, in today’s wider landscape of technical research, through a methodical review of the literature. We seek to answer the following research questions:

1. RQ1: Methods. What deep learning methods are being contributed or applied?

2. RQ2: Scope. What types of problems are addressed and solved?

3. RQ3: Context. How do these articles fit into the wider research context?

To answer these questions and draw out other insights, our study methodically considers 25 variables across 212 articles from a variety of venues, mostly published between 2014 and April 2019. Notable data-based observations from this study include:

• Publications on DL in clinical NLP are more than doubling each year, through 2018.

• The majority of this literature uses existing DL models on well-known information extraction tasks in English clinical notes, but there are many exceptions.

• There is growing acceptance of DL as the baseline for NLP research and of DL-based NLP in the medical community.

Working definitions

Deep learning is a modern, popular paradigm for machine learning (ML) heralded for avoiding the extensive manual feature engineering that was common in traditional methods. For the purposes of this study, deep learning consists of neural network-based or -inspired methods that utilize modern optimization techniques and training objectives. For example, recurrent neural networks (RNNs) are frequently employed to model sequential data such as language; convolutional neural networks (CNNs) are often used to model signals such as images. We include embeddings—dense, data-driven vectorial representations of, for example, a word—under the deep learning umbrella. Embeddings serve as the input layer of most modern neural networks; some embeddings are directly created as part of larger neural networks, while other embedding methods have none of the nonlinearities that are characteristic of neural networks. Finally, though we do not consider older neural networks (eg, multi-layered perceptron) to be “deep learning,” we nonetheless include them in our study.

We take a broad view of natural language processing techniques, namely, any work that computationally represents, transforms, or utilizes text (or speech) and its derivatives. Thus, diverse tasks can be viewed as NLP activities; from producing dependency parses, to text-based event prediction, to image classification via captions. However, our study only considers manuscripts with NLP in the specific clinical setting of electronic health records—digital profiles of patients’ health, primarily authored by health professionals and administrators. Other health-related settings such as social media, web forums, and messaging platforms differ vastly from EHRs in linguistic profile and data availability and were thus excluded.

Finally, unlike some recent work,^4–6 we have titled this work a methodical review, by which we mean that our work follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines closely, but not completely. Because this work (and with it, most computing literature) does not qualify as a traditional Cochrane-style “systematic review,” we have chosen to consider this a “methodical” review.

Related work

While this work emphasizes methods, especially in deep learning, we do not thoroughly discuss the underlying models. Instead, we refer the reader to other treatments thereof, such as Shickel et al’s taxonomy of DL methods⁷ or classic works on embeddings,^8,9 RNNs,^10,11 CNNs,^12,13 attention,^14,15 and adversarial learning.^16,17

Deep learning in a clinical NLP is an active and multidisciplinary area of research, and has thus spawned numerous other review articles, as shown in Table 1. Of note, Dreisbach et al had a similar technical focus but overviewed symptom extraction techniques and used patient-authored texts, rather than our focus of all NLP tasks on clinician-authored texts in the EHR. Shickel et al wrote in the context of EHR data, and as such also provided an informative enumeration of important clinical NLP tasks such as representation learning. In contrast with their methodical conceptual survey, our work offers a PRISMA-like review with quantitative analysis and focuses exclusively on NLP. Wang et al performed a thorough methodical review on NLP from EHRs. However, their eligible articles only included work up to September 6, 2016, before DL was really adopted as mainstream in the informatics community (see Figure 5a). Their work is also exclusively focused on applications of NLP, whereas this work also considers primarily methodological contributions. Xiao et al also methodically reviewed DL literature in EHRs up until January 30, 2018, categorizing the tasks involved, deep learning techniques, and the associated challenges. Our work is similar but provides a narrower focus on NLP, more quantitative analysis on a larger number of updated articles through 2019, and consideration of contemporary factors, such as preprints and scientific rigor.

Table 1.

Comparison of review articles related to this one, with their methods and scope

Year	Authors	PRISMA Review?	DL?	NLP?	EHR?
2019	This paper	✓	✓	✓	✓
2018	Al-Aiad et al18	–	✓	Broader	✓
2018	Ching et al19	–	–	Broader	Broader: biology
2019	Dreisbach et al4	✓	✓	Narrower: IE	Patient-authored; Narrower: symptoms
2019	Esteva et al20	–	–	Broader	✓
2017	Gonzalez et al21	–	–	✓	Broader: with social media
2016	Liu et al22	–	–	Narrower: IE	Broader: biology
2018	Névéol et al23	Selection	–	✓	Broader: with social media
2018	Névéol et al24	–	–	✓	Non-English; Broader
2019	Sheikhalshahi et al5	✓	–	✓	Narrower: chronic diseases
2017	Shickel et al7	Methodical	✓	Broader	✓
2018	Velupillai et al25	–	–	✓	✓ with health outcomes
2018	Wang et al26	✓	–	Narrower: IE	✓
2018	Xiao et al6	✓	✓	Broader	✓
2018	Zeng et al27	–	–	✓	Narrower: computational phenotyping

Abbreviations: DL, deep learning; EHR, electronic health record; NLP, natural language processing.

Figure 5.

(a) Number of papers from different research communities and their publication types (Journal, Conference) over time; overall, the informatics community produced the highest volume of literature (48.6%), followed by CS (22.6%), followed by NLP (18.4%), followed by Others (5.66%) and Medical (4.7%). (b) Types of contributions for each community.

Abbreviations: CS, computer science; NLP, natural language processing.

MATERIALS AND METHODS

Our review adheres as closely as possible to the PRISMA guidelines with most analyses considering categorical variables rather than the results of the component articles. The overall workflow is shown in Figure 1 and described below.

Figure 1.

PRISMA flowchart for including articles in our study, with example search string (for Scopus) and primary reasons for exclusion.

Eligibility, sources, and search

Articles eligible for inclusion in our study were characterized by: 1) natural language processing, 2) deep learning or neural networks, and 3) clinical domain tasks using EHR data. These criteria were approximated through librarian-assisted development of queries for each database (the string for Scopus is in Figure 1). Ovid MEDLINE, EMBASE, and Scopus, and the Association for Computing Machinery (ACM) Digital Library were searched on April 10, 2019. We subsequently included articles from “Other Sources.” Most notably, we searched the Association for Computational Linguistics (ACL) Anthology with EHR- and DL-related keywords, retrieving 61 articles before deduplication. “Other Sources” also included free suggestions from all authors, who were given a 1-week span to submit relevant works that should be included, preserving the recall-centric style of searching.

Duplicates were removed. The remaining papers were used for screening according to our inclusion/exclusion criteria.

Individual papers’ risks of bias were not a concern for our primarily count-based aggregate analyses of categorical variables. Rather, there is the typical risk of selection bias in these articles, since indexes such as MEDLINE or Scopus are not exhaustive, and search strings may have left out relevant articles.

Study selection

Title-abstract screening

During the screening stage, 2 randomly assigned co-authors (and 1 adjudicator) screened articles, excluding 1 487 for the ordered list of reasons shown in Figure 1. Thus, the 366 (under “Excluded at screening”) that were not typical research manuscripts were already considered to not be duplicates.

Note that 7 preprint articles (suggested by authors) were excluded, despite the fact that some such articles (eg, BioBERT²⁸) are influential in the research community at the time of this writing. A preprint is a full draft of a research paper that is shared publicly before it has been peer reviewed. Preprints can bring broad and instant visibility to research and have been widely utilized by research communities, though not without controversy. We explore the impact of preprints in RQ3.

Full-text screening

A full-text screening step, performed in conjunction with data collection, further ruled out 38 papers. Here, aside from criteria that were missed in other steps, we also excluded abstract-only publications (8 references), works with insufficient NLP or DL (6 and 16, respectively), or those that were poor in quality (1 reference). Note that we excluded out-of-the-box algorithms, which accounted for 4 of the 6 that were labeled “no NLP,” and 12 of the 16 labeled “no DL”—these had no retraining of models, variation in architectures, or systematic evaluation.

Data collection

Nine co-authors extracted 25 variables regarding DL Techniques, Embeddings, Tasks, Data, Experimental Setup, Comparisons, Contributions, and Venue & Timing (see Supplementary Appendix B) from each of 212 included articles (see Supplementary Appendix A), according to a data collection form. These data were normalized, validated, and analyzed by team members.

For all analyses, note that 2019 was a partial year including an imbalanced sampling of underlying data; for example, computer science and NLP conferences tend to occur in the summer, after April 10, of any given year. Also, many of our variables allowed a single paper to have multiple responses; in the text, numbers and percentages represent portions of the 212 included articles unless otherwise indicated, while figures and tables at times utilize relative percentages.

RESULTS

RQ1: What deep learning methods are being contributed or applied?

Methods: Deep learning architectures

Figure 2 illustrates the rise and fall of broad categories of DL architectures (see Supplementary Material Table 1 for more fine-grained categories and mappings). We focus on RNNs, CNNs, feed forward neural networks, and Embeddings-only articles, where Embeddings-only papers used embeddings without neural networks in a similarity function or in a more traditional ML classifier, for example, support vector machine. Note that Figure 2 intentionally double-counts papers that combine multiple methods (n = 41, 19.3%).

Figure 2.

Growth of broad architectures in deep learning over the years. Percentages are relative to the number of studies published in that year. Overall, RNN variants were the most common (n = 129, 60.8%), CNNs were second (n = 83, 39.2%), traditional feed-forward networks were third (n = 22. 10.4%), and embeddings-only were fourth (n = 21, 9.9%).

Abbreviations: CNN, convolutional neural networks; RNN, recurrent neural networks.

The volume of DL publications is currently increasing quickly each year (>200% through 2018), and the same is true of each type of architecture (with the possible exception of “Other DL”). Across years, RNN variants were the most common (60.8%), split among memory units, such as long short-term memory (LSTM, 52.8%), gated recurrent unit (7.5%), and the “vanilla” RNN (2.4%). The second-most common architecture type was CNNs (n = 83, 39.2%). Both show consistent growth in the last few years.

Interestingly, traditional neural networks (largely multi-layer perceptrons) also grew in percentage usage (from 5% in 2017 to 9% in 2018), perhaps as a byproduct of excitement over neural methods. On the other hand, Embeddings-only papers (n = 21, 9.9%) decreased in percentage after initial interest, similar to general domain NLP. Other DL architectures employed in this study included autoencoders,^29–32 residual neural networks,^33–36 deep belief networks,³⁷ capsule networks,³⁸ memory networks,³⁹ seq2seq extensions,⁴⁰ and the attention-based Transformer via BERT.⁴¹

Attention mechanisms (in Figure 2, the line and right-hand axis), which can increase predictive performance and—debatably⁴²—improve model explainability, were often combined with other methods and are increasingly popular (overall n = 26, 12.3%). We anticipate a significant future uptick in attention mechanisms due to BERT.⁴³ Adversarial learning, while typically popular and even motivational in natural language generation tasks,³¹ was surprisingly only used by 2 papers.^38,44

Though the norm was to repurpose or combine existing models for clinical NLP tasks, a few unique architectures were introduced that reflected their clinical task and domain particularly well. For example, Xie et al⁴⁴ used a tree-of-sequences LSTM to fit the tree structure of ICD codes, along with adversarial learning, isotonic constraints for ordering, and attention-matching. A second case is the deep averaging network used by Dligach & Miller,⁴⁵ whose input is concept-level embeddings that are simply averaged and passed to a downstream layer. This allows the model to robustly focus on sets of conditions for a variety of phenotyping tasks without over-emphasizing specific context.

Methods: embeddings

Table 2 details the embedding techniques in our study. Overall, the most prominent embedding model is word2vec (74.1%, combining values in Table 2a and Table 2b), followed by GloVe (9.9%). To avoid out-of-vocabulary problems in representing clinical words, many studies combine character embeddings with word-level embeddings (13.7%), or utilize the fastText subword model (3.8%).

Table 2.

Embedding techniques among the included articles. Due to the use of multiple approaches in individual papers, percentages overlap and may not add up

(a) Popular embedding techniques
Method	# papers
word2vec only	89 (42.0%)
+ character	26 (12.3%)
+ syntax	14 (6.6%)
+ position	8 (3.8%)
+ semantics	5 (2.4%)
+ 2 more features	6 (2.8%)
GloVe	17 (8.0%)
+ syntax	1 (0.5%)
+ character	3 (1.4%)
fastText	8 (3.8%)
(b) Embedding comparisons
Methods	Reference
word2vec vs GloVe	46–50
word2vec vs fastText	51–54
(c) Less common embeddings
Method	Task
Collobert55	NER,56 Abbrev. Disambiguation57
RNNLM58	De-identification59
Starspace60	Representation learning61
VecMap62	Cross-lingual concept extraction63
ELMo64	NER65–67
BERT43	NER,65 pretrained resource41

Abbreviation: NER, named entity recognition.

Especially for the tasks of concept and relation extraction, a number of studies explored additional lexical features combined with word embeddings: syntax embeddings, such as parts-of-speech^68,69 and dependency trees;^70–72 semantic embeddings, such as dictionary features,^35,73–76 controlled unclassified information from Unified Medical Language System (UMLS),^77–79 and semantic role labels;⁷⁸ and position embeddings, such as word^{34,38,70,75,80–84} and section⁸⁵ positions. The number of embedding methods combined with different input features is shown in Table 2a. A few studies (Table 2b) compared multiple word embedding algorithms in terms of extrinsic, downstream tasks, finding no significant differences between the popular methods.

Less common embedding models are reported in Table 2c, including Collobert’s ranking-based embeddings;⁵⁵ the early RNN-based language model RNNLM;⁵⁸ Wu et al’s⁶⁰ generalization of fastText, StarSpace; the VecMap⁶² framework for learning cross-lingual word embeddings; contextual word embeddings ELMo;⁶⁴ and the recent language model, BERT.⁴³

Large, unlabeled data sources are often used to train effective of word embeddings. Among the 63% of articles that reported using pre-trained resources, clinical word embeddings are built on clinical notes (29%), like MIMIC-III; health-related text from biomedical literature (25%), like PubMed; and healthcare websites (5%), like WebMD. The rest (46%) have randomly initialized the word embeddings and are trained on specific target data. Few works directly compared the effectiveness of pretraining on different resources, yet large pretraining data does not guarantee effective word embeddings for clinical NLP. Notably, in 1 study,⁸⁶ embeddings concatenating domain-specific with domain-agnostic embeddings yielded the best results.

Methods: medical knowledge

Clinical NLP has traditionally relied heavily on medical-specific knowledge resources, such as the UMLS.⁸⁷ Interestingly, only 38 papers (17.9%) used external (ie, not entirely derived from the training samples) medical knowledge. Of these 38, only 12 papers (5.7%) incorporated that knowledge into the deep learning architecture itself. With only 1 exception,⁴⁴ these 12 produced and incorporated knowledge-resource embeddings (eg, concatenating a UMLS controlled unclassified information embedding with a word embedding). Further detail on popular knowledge resources is in the Supplementary Material Table 2.

Methods: implementation

Many papers did not clearly report the use of DL libraries (only 53.4%) or NLP tools (only 36.8%). The most popular DL libraries in our study are TensorFlow and Keras, with increasing trends starting from 2015. The most frequently used NLP tools were Gensim (mostly for word-level or paragraph-level embeddings), cTAKES, NLTK, MetaMap, Stanford CoreNLP, and Jieba (a word segmentation toolkit for Chinese). Further details on the DL and NLP implementation tools can be found in Supplementary Material Table 3 and Figure 2.

We also note that the vast majority had underspecified experimental setups and did not report the type of hardware used (172 of 212, or 81.1%). Of those that did report computational equipment, a few used commercially available resources (3 of 40, 7.5%), some used local CPUs (3 of 40, 7.5%), but most used local GPUs (34 of 40, 85.0%).

RQ2: What types of problems are addressed and solved?

Scope: tasks

Figure 3a illustrates the NLP and clinical tasks addressed among the 212 papers. We categorized the main NLP tasks in each paper into 4 types: Text Classification (40.5%), NER (34.0%), Relation Extraction (13.7%), and Others (10.8%). Note that the Text Classification category is composed primarily of document-level tasks (64 of 86, 74%), but sentence-level (12 of 86, 14%) and Other (10 of 86, 12%) tasks are also present (not pictured). Figure 3a further subdivides clinical domain-specific tasks. The top clinical tasks are clinical concept extraction (ie, the extraction of common clinical concepts, such as problem, lab test, treatment, time expressions, and events); phenotyping (ie, the broad characterization of patients’ conditions); and clinical relation extraction (ie, the identification of relations between the common clinical concepts). The figure also reveals a “long tail” of papers on more variegated tasks, include Information Retrieval,^86,88–90 Natural Language Generation,^31,40,91 Abbreviation Disambiguation,^57,68 Section Boundary Detection,^92,93 Shallow Parsing,⁹⁴ Machine Translation,⁹⁵ Spelling Correction,⁹⁶ and others. Overall, many common Clinical NLP tasks are present (eg, negation/attribute identification,^33,72,97,98 event detection,^99–101 and adverse drug event relation extraction^{38,81,102,103}) but other tasks are infrequent or absent (sentence detection, part-of-speech tagging, and text simplification) from our study, presumably because DL techniques have yet to be applied on these tasks.

Figure 3.

(a) Tasks and their prevalence in our study, from the NLP and the clinical perspective. (b) Deep learning architectures for major task groupings.

Abbreviation: NLP, natural language processing.

Figure 3b shows the DL algorithm distribution for each task, confirming widely held associations in the community: CNNs are the most common approach to the Text Classification task, and RNNs are the most common approach to NER and Relation Extraction.

Scope: data sources

Table 3a shows the source of corpora for the included studies. Almost half of the studies used private datasets (n = 104; 49.06%), which are rarely shared or replicated due to patient privacy concerns. Though at times used in conjunction with private datasets, publicly available data sources were employed by more than half of the papers in our study. Excluding counts of multiple public corpora in Table 3a, this is 54.7% of 212 papers, whether from research challenges (i2b2/n2c2, CCKS, SemEval, etc; 34.0%) or otherwise contributed by the research community (MIMIC, THYME, MEDLINE, etc; 20.8%). Considering the publicly available corpora used for each task, the most popular were: MIMIC data for text classification (15 of 86 papers); i2b2 challenges for NER/concept extraction (19 of 74 papers); i2b2 challenges for Relation Extraction (5 of 17 papers); and SemEval challenges for temporal events and relations (4 of 7 papers).

Table 3.

(a) Types of labeled corpora used among the included articles, and their availability for 3rd party researchers. Each percentage uses 212 papers as its denominator, but, due to the use of multiple corpora in individual papers, percentages do not add up within any grouping. (b) The languages for labeled corpora, in comparison with Névéol et al’s¹⁰⁴ reviewed papers. In 4 of our cases of non-English corpora, an English corpus was also used

(a) Source of Labeled Corpus
Availability	Corpus			Count	Percent
Private	Institutional (proprietary) dataset			104	49.1%
Challenge	i2b2 challenges			34	16.0%
	CCKS			12	5.7%
	SemEval challenges			8	3.8%
	CEGS N-GRID challenge			7	3.3%
	MADE challenge			7	3.3%
	Other			11	5.2%
Public	MIMIC data			21	9.9%
	THYME			5	2.4%
	MEDLINE case reports			2	0.9%
	MedlinePlus			2	0.9%
	Other			20	9.4%
Other	∼Not reported∼			3	1.4%
(b) Language of Labeled Corpus
		Count		Rank
Language		This study	Névéol et al	This study	Névéol et al
English	71.2%	151	–	–	–
Chinese	19.8%	42	11	1	5
Spanish	2.4%	5	17	2	3
Japanese	1.9%	4	8	3	6
Finnish	1.9%	4	6	3	7
French	0.9%	2	36	5	1
Italian	0.9%	2	3	5	9
Dutch	0.5%	1	5	7	8
Thai	0.5%	1	–	7	–
German	0.5%	1	19	7	2
Swedish	0.5%	1	12	7	4
Not reported	0.5%	2	–	–	–

Table 3b presents some statistics on the languages of these corpora. As expected, most of the studies used datasets in English (n = 151; 71.23%), but a significant proportion utilized Chinese corpora (n = 42; 19.81%). Datasets in all the other languages (such as Spanish, Japanese, and Finnish) were used by 5 (2.36%) or fewer studies. Interestingly, this differed drastically from Névéol et al’s²⁴ recent review of non-English Clinical NLP, especially with the reversed roles of Chinese and French. Of course, Névéol et al cannot be considered head-to-head with this work, because it had an eligibility cutoff date while DL was still nascent in the field (January 2017). However, our data suggest that deep learning-related articles were published sooner in Chinese (2016) than French (2018), and the disparity in volume is growing (see Supplementary Material Table 4).

Further details on datasets available for specific tasks, size of corpora, and subdomain of data are available in the Supplementary Material Table 5 and data-related subsections.

Scope: contributions

RQ1 (Methods) asks what types of contributions are being made. Thus, we judged each of our study articles for its contribution type: Application (to a new dataset, new domain, or setting); Methods (new DL architecture, new embedding method, new NLP task approach, or a variant of existing methods); Resource; or Evaluation; 44 articles made multiple contributions.

Pairing this with RQ3 (Context), we are interested in the novelty of articles, which is typically quite subjective. Thus, we defined a low bar of “novelty” as attempting methodological contributions that included new DL architecture, new embedding method, or new NLP Task approach (grouped together in the bottom center of Figure 4), while excluding “variant of existing method” Methods contributions. While there were contributions of each kind for each task, Text Classification papers were overrepresented in Applications (43% vs 36% in Methods); NER papers were overrepresented in Methods variants (40% vs 24% in new Methods and 30% in Applications); and Relation Extraction was overrepresented in new Methods (17% vs 7% in Methods variants and 8% in Applications).

Figure 4.

Number of papers on various NLP tasks per contribution category. Articles tagged with multiple contributions and/or Tasks appear more than once.

Abbreviations: EN, Entity Normalization; IE, Other Information Extraction; IR, Information Retrieval; NER, Named Entity Recognition; NLG, Natural Language Generation; NLP, natural language processing; RE, Relation Extraction; TC, Text Classification; TE, Temporal Expressions; WSD, Word Sense Disambiguation.

Overall, there were 46 contributions of “novel” methods (excluding duplicates, this is 20.8% of papers), most introducing new DL architectures of some kind, for example, concatenating 2 new types of embeddings. The 5 papers proposing a new approach to an NLP task were related to temporal extraction/relation/inference,¹⁰⁵ spelling/grammar error identification and recovery,⁹⁶ computational semantics,¹⁰⁶ information retrieval (IR),⁸⁸ and NER.⁶⁹ The 2 papers proposing new embedding methods were associated with text classification¹⁰⁷ and word sense disambiguation.⁵⁷ One paper was marked for having both DL and NLP innovation: Cai et al’s⁶⁹ self-attention to label part-of-speech tags and named entities.

Considering the temporal trends of these novel contributions (see Supplementary Material Figure 3), the new embeddings Methods were the earliest (2015 and 2016), whereas Methods contributions proposing a new DL architecture began in 2016; Methods papers did not take new approaches to NLP tasks until 2017. Overall, “novel” Methods contributions are perhaps the slowest growing contribution type in recent history (from 10 to 17, from 2017 to 2018). Methods variants are the fastest-growing type of contribution (an increase from 18 to 52, over the same period), Applications and Resource papers show intermediate rates of increase (from 22 to 51; and from 3 to 5, respectively).

RQ3: How do these articles fit into the wider research context?

Context: research communities

In Figure 5a, we categorized each publication venue as being a Conference or Journal, and arising from 1 of 5 communities:

1. Computer science (CS, 22.6%), such as NeurIPS, or AAAI;

2. Informatics (48.6%), such as AMIA, JAMIA, or JBI;

3. Medical (4.7%), such as Radiology, Drug Safety, or Nature Medicine;

4. NLP (18.4%), such as BioNLP, EMNLP, or ACL; or

5. Other (5.7%), such as PloS One, or IEEE Transactions on NanoBioscience.

The NLP community saw the earliest push (in 2016) in DL-related papers, but the CS and Informatics communities followed quickly (in 2017). Interestingly, medical venues seem to exhibit an increasing acceptance (beginning 2018) of deep learning, despite its reputation as a “black box” which clinical experts might hesitate to use.

Overall, more articles have been published in conferences than journals, but the differences per community demonstrate implicit community preferences; the computer science and NLP communities tend to prefer conferences, which boast quicker turnaround times and higher standards of novelty. In contrast, medical conferences are not considered full publications, so only journals are present for that community. The informatics community is a multidisciplinary blend, and has thus far also produced the largest volume of literature using DL for Clinical NLP.

Figure 5b shows what types of contributions each community has made. This illustrates relatively similar contributions from the Informatics and CS venues, though resources and evaluation are stronger in the Informatics community. As might be expected, new Methods are overrepresented in the NLP venues but underrepresented in Medical venues.

Context: preprint status

We checked the preprint status of each article to see whether it had been posted on the preprint servers arXiv (https://arxiv.org/) or bioRxiv (https://www.biorxiv.org/). Out of 212 papers, 35 were posted on arXiv, and 0 on bioRxiv. Of the 35, 9 were ultimately published in peer-reviewed journals, including 8 informatics-related journals (eg, JAMIA, JBI, BMC Medical Informatics and Decision Making). The number of preprints is also increasing per year.

Context: scientific rigor

A renewed emphasis on scientific rigor has permeated the academic establishment,^108,109 with replicability and reproducibility studies (eg, from ACM’s definitions^110,111) showing how much of the literature has fallen short of its scientific goals. While a direct study on the reproducibility and replicability of our 212 articles is beyond the scope of this work, we labeled each article for methodological implementation details that may contribute to later studies on scientific rigor. First, regarding the type of contributed software, we categorized papers as Open-source vs Restricted (the software exists but requires, for example, a commercial license) vs Not provided. Second, regarding hyperparameters (eg, number of layers in the model, the size of dimensions for each layer, the learning rate, the optimizer, dropout rate), we tagged each paper with Not provided, Partially provided, and Present for the 212 papers.

The results in Table 4 demonstrate that although most papers did not provide software (91.51%), many of these nonetheless offered enough hyperparameters and details (55.19%) so that other researchers can potentially reimplement their methods. An encouraging majority of papers (63.68%, summing the values in bold and/or italic in Table 4) have contributed software or offered hyperparameters.

Table 4.

Availability of rigor-related implementation details (count and proportion)

		Hyperparameters
		Not provided	Partial	Present	Total (row)
Software	Not provided	48 (22.64%)	29 (13.68%)	117 (55.19%)	194 (91.51%)
	Restricted	1 (0.47%)	0 (0.00%)	3 (1.42%)	4 (1.89%)
	Open-source	2 (0.94%)	3 (1.42%)	9 (4.25%)	14 (6.60%)
	Total (col)	51 (24.05%)	32 (15.1%)	129 (60.85%)

Bolded entries indicate that articles identified the software in some way. Italicized entries indicate that articles gave a sufficient set of hyperparameters.

In addition, we find that the papers whose contributions were tagged as Method type tend to provide more implementation details (either contributed software or presented enough hyperparameters) (66.43% = 93/140) than papers with types of Application and Resources (58.33% = 42/72).

Context: comparisons with traditional machine learning

Figure 6 shows that the percentage of studies comparing DL against traditional ML decreased from 2016 (70%) to 2017 (63%) to 2018 (55%) and 2019 (47%). This is likely due to the increasing acceptance of other DL algorithms as baseline models. Of the 212 papers, just over half compared their proposed methods with traditional ML methods (n = 108, 50.9%). Within these 108 studies, the majority (72%) proposed DL methods that outperformed the traditional ML methods, though there were also some negative results (11% of DL algorithms were worse than traditional ML).

Figure 6.

Comparisons between deep learning techniques and traditional machine learning, over time.

DISCUSSION

Growing acceptance

There is evidence that in addition to growing in volume, DL for Clinical NLP is becoming more widely accepted. This acceptance is demonstrated in the fact that deep learning approaches are increasingly considered the baseline technique, with no need for comparison with traditional ML. Additionally, despite their genesis in the CS and NLP communities, DL-based NLP approaches have thoroughly permeated the informatics community and penetrated reputable clinical journals.

The implications are that informaticians and clinicians will increasingly be willing to adopt DL technologies in clinical settings as it becomes more familiar and widespread. This is both an opportunity and a hazard, so health professionals need to both accept and discern the associated risks appropriately.

Substantiating vs surprising results

A number of commonly held assumptions were substantiated in our data analysis. CNNs have dominated the Text Classification task because of early, successful CNN-based methods.¹¹² A similar effect was noted for LSTMs with NER (typically cast as a sequence labeling problem). Also, as expected, early adopters of deep learning DL for clinical NLP were publishing in NLP venues rather than informatics or medical venues; there is a lag for adoption of cutting-edge techniques in the informatics community, and an even longer one for the medical community.

However, other analyses were surprising when compared with conventional wisdom. Whereas previous approaches to clinical NLP seemed to extensively utilize knowledge resources, only 17.9% of DL-based approaches did so. Whereas earlier work²⁴ found French to be the most prolific non-English language in clinical NLP, our study presents a previously unreported fact about non-English clinical NLP: when dealing with deep learning, Chinese has more representation than French. Whereas some have assumed that cutting-edge DL research is entirely contained on preprint servers, we found that only 16.5% of the papers in our study were posted as preprints prior to their publication.

Projections

This review has suggested some potential future trends in DL for clinical NLP. Because of the cost of annotating clinical corpora and the privacy concerns with sharing in-domain training data, domain adaptation and transfer learning strategies are important. However, there has been little systematic analysis on this issue from a deep learning perspective, perhaps in part due to the lack of convincing results. With the rise of successful pretrained models like BERT, we expect that the use and refinement of transfer learning will rise in popularity quickly.

Despite this, we also believe medical knowledge resources have been underutilized. Though the mantra of deep learning has been to “let the weights determine what’s important” rather than to handcraft features, DL architectures and inputs still need human input, as evidenced by a recent push to consider inductive bias (eg, gender biases found in word embeddings). Knowledge resources may provide calculable and objective means to guide data-driven DL algorithms, and the medical domain is uniquely equipped with such resources.

Further, it is clear from other subfields involving ML for clinical tasks that deep learning is not always successful.^113,114 In this we mean that oftentimes it fails to outperform basic models, such as logistic regression. In the current set of articles included in this study, however, there were no in-depth analyses of the limitations or failures of DL methods for clinical NLP. In the few cases (n = 12) where DL failed to outperform traditional ML, there was no investigation into the underlying causes of the failure. Missing are investigations into the relative merits and pitfalls of deep learning based on data size, data quality, language, medical specialty, informatics task, and, to our previous point, the inclusion of knowledge resources. Therefore, we assert that there is a critical need for empirical investigations into the limitations of DL methods for clinical NLP tasks. For example, accuracy improvements of DL methods often rely on large computational resources that consume substantial amounts of energy, which can be detrimental to the environment.¹⁰⁴

Overall, we would argue that at this point it is dangerous to only compare DL-based methods, especially in regard to applications about to be deployed to the clinic. However, this paper also demonstrates that it is scientifically naïve to not compare to some kind of DL baseline as well. Additionally, emerging DL practice suggests some instability surrounding initialization and hyperparameter selection. Few works presented here experimented with re-initialization of random seeds, small adjustments to hyperparameters, etc; 1 example of simple stability characterizations was Tourille et al,¹¹⁵ who repeated each experiment 30 times and plotted the results. We recommend that as best practices emerge in the NLP community (and these practices still are in the nascent stages), the clinical NLP community quickly adopt these and report results in a consistent manner.

Finally, we suggest that truly novel core methodological contributions in this field may eventually plateau or even decrease, but the application and tweaking of deep learning to many (potentially new) clinical tasks will continue beyond any such plateau.

Limitations

This methodical review has a number of limitations. First, there is the possible selection bias inherent to the search methods used. This includes bias both in the types of searches we performed as well as the underlying limitations of those search engines. For instance, despite the fact that the studies had already been published, our ACL Anthology search missed both emrQA¹¹⁶ and CliCR¹¹⁷ whereas a current search with the exact same criteria on the ACL Anthology would have returned those studies. Similarly, we consider that a great variety of relevant papers don’t explicitly contain the keywords of our method in their titles, therefore additional papers were manually added based on our authors’ research experiences. During the process, we noticed that specific DL models mentioned in the title were probably missed based on the current search and can be considered as keywords in the future—for instance, condensed memory networks,³⁹ graph-based models, BERT⁴¹, etc. In addition, some conventional clinical tasks relying heavily on NLP are also mentioned in the title, while missed in the previous search, such as de-identification,⁵⁹ automatic ICD-9 coding,⁴⁴ diagnostic inference,³⁹ and patient representation learning.⁴⁵ If anything, the trend towards universality of DL methods for NLP means that simple keyword searches such as “deep learning” and “neural networks” will increasingly miss relevant papers as more and more of these methods are the assumed default and not an element of novelty.

Second, while our review attempted to define mostly objective criteria for data collection, some data elements (eg, Application vs Evaluation papers) were not precisely defined and still had an inherent element of subjectivity. Related to that, third, many of the data collection elements and the normalization to broader categories were difficult to judge and agree upon, especially as papers are diverse in structure and style, and we allowed multiple tags for many of the data collection elements. All authors who participated in data collection (9) were tested for inter-annotator agreement on 11 of the 212 papers; in brief, this revealed that most data elements’ responses could be normalized, but fundamental disagreements between the 9 annotators were still found in virtually every category.

Finally, regarding scope, we found it infeasible to address the very interesting question of comparing clinical NLP vs open-domain NLP. To do so, we would have needed to narrow our focus (eg, only LSTMs or only text classification), which would have been a very different scope than the current study.

CONCLUSION

We have reported quantitative and qualitative analyses on our methodical review of 212 papers regarding DL in clinical NLP, finding an active research area with multiple vibrant communities making diverse contributions. We quantitatively observed some widely held associations in the community, regarding methods (eg, CNNs tend to be used on classification) as well as preferences (eg, NLP community prefers conferences, medical community prefers journals). We expect deep learning to continue and extend its leading role in the wider research context before other technological paradigms supplant it.

FUNDING

This work was supported in part by NIH Grants U01TR002062 and R00LM012104, the UTHealth Innovation for Cancer Prevention Research Training Program Pre-doctoral Fellowship (CPRIT grant #RP160015), and PCORI grant ME-2018C1-10963.

AUTHOR CONTRIBUTIONS

SW and HX conceived the study, then designed it along with KR. SW oversaw the screening, data collection, analysis, drafting, and editing. SW, KR, SD, JD, ZJ, YS, SS, QW₁, QW₂, and BZ completed initial screening. SW, KR, SD, JD, ZJ, YS, SS, QW₂, and YX performed data collection. Analysis, figures, and drafting was split among algorithms (KR), embeddings (YS), tasks (ZJ), data (SS), contributions (SD), venues (JD), experiments (YX), and comparisons (QW₂). All authors reviewed the manuscript.