Learning regular expressions for clinical text classification

Abstract

Objectives

Natural language processing (NLP) applications typically use regular expressions that have been developed manually by human experts. Our goal is to automate both the creation and utilization of regular expressions in text classification.

Methods

We designed a novel regular expression discovery (RED) algorithm and implemented two text classifiers based on RED. The RED+ALIGN classifier combines RED with an alignment algorithm, and RED+SVM combines RED with a support vector machine (SVM) classifier. Two clinical datasets were used for testing and evaluation: the SMOKE dataset, containing 1091 text snippets describing smoking status; and the PAIN dataset, containing 702 snippets describing pain status. We performed 10-fold cross-validation to calculate accuracy, precision, recall, and F-measure metrics. In the evaluation, an SVM classifier was trained as the control.

Results

The two RED classifiers achieved 80.9–83.0% in overall accuracy on the two datasets, which is 1.3–3% higher than SVM's accuracy (p<0.001). Similarly, small but consistent improvements have been observed in precision, recall, and F-measure when RED classifiers are compared with SVM alone. More significantly, RED+ALIGN correctly classified many instances that were misclassified by the SVM classifier (8.1–10.3% of the total instances and 43.8–53.0% of SVM's misclassifications).

Conclusions

Machine-generated regular expressions can be effectively used in clinical text classification. The regular expression-based classifier can be combined with other classifiers, like SVM, to improve classification performance.

Introduction

A regular expression is a sequence of characters describing a pattern of text. It is a standard technique supported by most programming languages (eg, .NET, Java, Python, Perl, Ruby, AWK, and Tcl). Many prior natural language processing (NLP) studies in the clinical domain have used regular expressions in designing their NLP solutions.^1–12 These regular expressions are typically created by software developers working with domain experts. Since there is no standard way to generate or test regular expressions, their maintenance and extension is a challenge.

Given the wide use of regular expressions, computer algorithms have been developed to automatically discover them from training text samples. A key challenge in learning regular expressions is the huge search space of candidates since each text string may be matched by numerous expressions. To reduce the search space and select the valid expressions, two main learning approaches have been employed (figure 1). The top-down approach often starts with domain experts supplying seed patterns; they are then transformed or modified until satisfying predefined evaluation metrics have been achieved. This approach has been used by Xie et al, Li et al, Babbar and Singh, and Brauer et al to learn email,^{13 14} URL,^{13 14} software name,^{13 15} notebook name,¹⁵ course number,¹³ phone number,^{13 15 16} invoice number,¹⁵ and date–time¹⁶ patterns.

Figure 1.

Top-down versus bottom-up approaches in automated regular expression generation.

The bottom-up approach starts by finding similarities or patterns among textual data that are then generalized to build regular expressions (figure 1). For example, Prasse et al¹⁷ used the bottom-up approach to discover regular expressions from batches of email and campaign message templates. They tested the method on predefined input batches (5–50 instances per batch). Meng et al¹⁸ have employed multiple sequence alignment algorithms to detect the sequential patterns in clinical text. The identified patterns could potentially be used for text classification, although the study itself did not perform classification procedures.

One major issue with existing regular expressions learning algorithms is the generalizability. The performance of the top-down approach is dependent on seed patterns, the creation of which is not automated. In the example of figure 1, patterns like ‘quit smoking’ or ‘smokes \d+packs a day’ are manually generated. The bottom-up approach is totally automated, though the algorithms described in the literature are still task-specific. For instance, the sample data in figure 1 are not pre-clustered batches as the data in Prasse et al's study.¹⁷ Ideally, a generalizable expression-based classifier would behave just like feature-based classifiers that learn directly from annotated data and generate labels for unseen samples.

We present a general purpose regular expression learning algorithm in this paper, called the regular expression discovery (RED). The algorithm employs a bottom-up approach that first conducts pairwise alignment to identify sequential similarity and then constructs regular expressions.

We created two classifiers that use the RED generated expressions. The RED algorithm was tested on two different clinical classification tasks. In addition, we compared the RED performance with the popular support vector machine (SVM) algorithm using bag of word (BOW) features.

Materials and methods

Datasets

Two annotated clinical datasets were used in this project. Both datasets were obtained from the US Department of Veterans Affairs’ (VA) electronic medical records. The first dataset, SMOKE, was generated for the automated classification of smoking status. Smoking is a risk factor for a number of diseases, including cardiovascular disease, which is particularly prevalent among the veteran population. Multiple prior NLP tools have attempted smoking status detection with good results.^{10 19–21} Our team had previously trained a smoking detection module on Partners Healthcare System's records. However, when tested on VA records, we observed that it often failed on templates and pseudo-structured texts (eg, questions and corresponding check boxes). In addition, the VA is a nationwide healthcare system and the VA medical documents are particularly diverse.

To more accurately classify the smoking status for cardiac risk prediction using VA data, we randomly sampled 1091 text snippets containing certain keywords (‘tab*’, ‘cigar*’, or ‘smok*’) from the records of 1 million VA patients who had answered a 2001 survey containing questions regarding smoking. In this project, we define a text snippet as a sequence of printable characters that provides semantic information for the text classification of interest. As such, a snippet may be a sentence, a fragment of a sentence, multiple sentences, or a fragment of template text. For instance, ‘He smoked heavily in the past’ and ‘Smoking? No’ are both considered snippets. These text snippets were then independently annotated by two reviewers into the categories of ‘current smoker’, ‘past smoker’, ‘non-smoker’, or ‘unknown’. Some snippets do not provide sufficient information on the smoking status and are thus labeled ‘unknown’. From the retrieval point of view, they can be viewed as false positives. The ‘unknown’ label was assigned smoking-related instances that are hypothetical, non-patient specific, and/or empty templates, for example ‘If you smoke, quit’, ‘Patient advised of the health risks of tobacco and urged to quit ? [ ] yes [ ] no’, and ‘Patient reinforced about the danger of smoking when oxygen in use’. A κ of 92% was obtained. The disagreements were adjudicated by a third reviewer.

The second dataset, PAIN, was generated to identify the pain status of patients in order to assess the effectiveness of complementary and alternative (CAM) therapy. The VA has begun offering CAM therapies to patients for pain management. However, the comparative effectiveness, as well as the cost effectiveness, of these therapies requires further research. Pain as a symptom is often documented as one of the vital signs in VA medical records, yet the documentation is neither consistent nor complete. To study the effect of CAM therapies, such as acupuncture, it is necessary for us to track patients’ pain status over time. Therefore, we set out to extract this information from patient notes to supplement the structured data. To create the dataset we randomly selected 100 documents, which contain references of ‘pain’ and ‘acupuncture’, from the nationwide VA clinical data repository and we manually extracted 702 text snippets related to ‘pain’. Two reviewers performed annotation on the entire dataset, independently classifying the snippets into ‘have pain’, ‘no pain’, or ‘other’. The ‘other’ label was assigned to text snippets which are hypothetical, non-patient specific, and/or empty templates, e.g. ‘Pan scale: 0 1 2 3 4 5 6 7 8 9 10’, ‘some pain medications cause constipation’. A κ of 94% was reached and then disagreements between reviewers were resolved through discussion, by consensus.

RED algorithm overview

In the context of the RED algorithm, a text snippet is defined as a sequence of printable characters that provides semantic information for the text classification of interest. A snippet contains tokens which are defined as words, numbers, or symbols. We refer to a sequence of consecutive tokens as a phrase. An ordered list of phrases is referred to as a key. Please note that individual phrases in the key are separated by one or more tokens. Examples of snippets, tokens, phrases, and keys are shown below. Please note these are just examples not an exhaustive list of keys or phrases:

Snippets: ‘He has been suffering from lower back pain for 3 years’.

Tokens: ‘He’, ‘has’, ‘been’, ‘suffering’, ‘from’, ‘lower’, ‘back’, ‘pain’, ‘for’, ‘3’, ‘years’.

Phrases: ‘He’, ‘has been’, ‘suffering’, ‘from’, ‘lower back pain’, ‘for’, ‘3 years’.

Keys: [‘has been’, ‘suffering’, ‘from’] [‘has been’, ‘lower back pain’, ‘for’].

The input of the RED algorithm is an annotated set of text snippets. Annotated datasets may have n classes or labels. RED generates regular expressions for each class, treating the instances in one class as positive and the rest of the instances as negative. In other words, RED classifier transforms multi-classification tasks into n binary tasks. First, RED preprocesses all strings. To find the pattern through the bottom-up approach, RED performs pairwise alignment at the token level. Based on the alignment results, keys are extracted. The regular expressions are then generated using the keys. These expressions are tested on the positive and negative instances and filtered using a predefined threshold of precision. Figure 2 provides a demonstration of the RED steps using a small set of text snippets.

Figure 2.

RED steps demonstrated using a small set of smoking data. RED, regular expression discovery; SVM, support vector machine.

The pseudo code of the RED is presented in figure 3.

Figure 3.

Pseudo code of the RED algorithm.

Preprocessing

The dataset was first transformed into lowercase and tokenized using Penn Treebank (PTB) tokenizer implemented by the Stanford NLP group.^21a Comparing to whitespace-based tokenizer and statistical OpenNLP tokenizer,^21b the Stanford tokenizer produces more fined-grained tokens. For example, it split ‘(no)’ into three tokens—‘(’, ‘no’, and ‘)’—rather than treating it as a single token. Duplicated strings are merged to prevent redundant processing.

Pairwise alignment

We adopted a pairwise alignment algorithm, the Smith–Waterman (SW) algorithm, which is widely used in bioinformatics research. The SW algorithm is a local alignment algorithm which is known its ability to find regions of similarity between even dissimilar sequences. In comparison, the global alignment algorithms (eg, Needleman–Wunsch) are more appropriate for closely matched sequences. Since text snippets are not always very similar in words or length, SW was chosen.

RED pairs up one snippet (primary snippet) with every other string (secondary snippet) and uses SW to identify matching segments of token (ie, phrases). It assigns positive scores to the matching segments and negative scores to the mismatch, insertion, or deletion. In the example below, to align ‘he has smoked for years’ and ‘he has been smoking for years’, we consider ‘he’, ‘has’, ‘for’, and ‘years’ as matches while ‘smoked’ and ‘been’ are mismatches and ‘smoking’ and ‘3’ are insertions. In our implementation, matching tokens are given a score equal to the character length, and mismatch, insertion, or deletion tokens are scored as −1. The similarity score is calculated as the sum of the match, mismatch, insertion, or deletion scores. In the example below, the similarity score is 10. There are often multiple ways to align two snippets. SW selects the highest scoring one for output (table 1).

Table 1.

An example of the pairwise alignment and the calculation of similarity score

	Token-level alignment result
Snippet A	he	has	smoked		for		years
Snippet B	he	has	been	smoking	for	3	years
Score	+2	+3	−1	−1	+3	−1	+5	10 (total)

If a string contains numeric patterns, a second alignment is performed where consecutive digits are replaced by a ‘\d+’ pattern. For instance, the terms ‘5/10’, ‘3/10’, and ‘7/10’ have the same pattern representation \d+/\d+and are aligned.

Key extraction

RED utilizes the alignment results keys. Please note that we refer to a sequence of consecutive and aligned tokens as a phrase, and an ordered list of aligned phrases as a key. Following the alignment example above, the aligned phrases are ‘he has’, ‘for’, and ‘years’. One of the keys is [‘he has’ ‘for’ ‘years’]. Given a set of aligned phrases, a number of keys may be generated, for example, [‘he has’], [‘for’ ‘years’], [‘he has’ ‘years’].

The keys generated using aligned phrases are passed on to the next steps to create regular expressions. Some of those regular expressions are then filtered out depending on their matching performance on the training data.

If none of the expressions generated using the aligned phrases survived filtering, key extraction

will iteratively add unaligned tokens. For example, assuming keys generated using ‘he has’, ‘for’, and ‘years’ did not produce any successful expressions, we will iteratively add by unaligned words ‘smoked,’ ‘been’, ‘smoking’, ‘3’, or ‘\d+’. A couple of the new keys will be [‘he has smoked’ ‘for’ ‘years’] and [‘he has been smoking’ ‘for’].

Using unaligned and aligned phrases together, there can be an exponential growth of keys. The total number of potential keys is , (U is the total number of unaligned tokens, P is the total number of phrases after adding unaligned tokens, ). In the RED implementation we set adjustable limits to control the maximum number of keys.

Regular expression generation

For each distinct key, RED produces two regular expressions, with and without distance control. The none distance control (NDC) version uses phrases from keys to develop regular expression and uses the (\s+\S+)* pattern to permit any number of tokens in between phrases.. The second, distance control (DC) version restricts the number of tokens in between phrases. It uses the pattern (\s+\S+){n,m}, where \S+ is any group of non-whitespace characters representing a word, n is the minimum token length required, and m is the maximum token length allowed. RED sets n and m based on the number of tokens between the phrase in the original alignment. The DC version has a lower chance to match unseen snippets, but is less likely to match false positive snippets. Continuing with the example we used for alignment and key extraction, for the key [‘he has smoked’ ‘for’ ‘years’], an NDC version of the expression is ‘he has smoked(\s+\S+)*for(\s+\S+)*years’, while the DC version is ‘he has smoked(\s+\S+){0,1}for(\s+\S+){0,1}years’. The NDC version can match a sentence such as ‘he has smoked 1 pack/per day for 6 or 7 years’, while the DC version can not.

Filtering using training sample

Each new regular expression generated using a key is tested against the rest of the training set. The recall, precision, and F-measure metrics are calculated for each individual expression. The regular expressions are filtered based on a precision threshold. The default precision is set at 100%, because human developers typically create expressions that have 100% precision and address the sensitivity issue with more expressions. The high precision threshold removes the majority of the regular expressions generated from keys. We have observed that the DC expressions are more likely to survive this test than the NDC expressions.

Regular expression classifiers

We implemented two RED-based classifiers. Both classifiers first attempt to match text with RED-generated expressions. Since many regular expressions can match a snippet of text, we rank the expressions using a formula we refer to as the predicting score. This score favors patterns including longer phrases. An extra one is added to the square root of the DC version, favoring DC over NDC if two versions performed the same. To break ties in predicting scores, the F-measure is used. In the formulas below, n is the number of phrases in a key that is used to generate the expression and t is the number of tokens in a phrase.

This score favors patterns including longer phrases. An extra one is added to the square root of the DC version, favoring DC over NDC if two versions performed the same. To break ties in predicting scores, the F-measure is used.

Both RED-based classifiers first assign a label based on the highest-scoring matching expression. For instance ‘he has not smoked for years’ may match different expressions for two patterns: ‘he has…for years’ which associates with Current Smoker, and ‘has not smoked’ which associates with Past Smoker. Assuming the expression for the latter pattern has a higher predicting score, RED will pick Past Smoker as the label.

Sometimes, no regular expression matches a text snippet (which occurred 10–15% of the time in our experiment). The alignment classifier and SVM classifier are used as the backup classifiers to classify unmatched snippets. In the RED+ALIGN classifier, the same SW algorithm described in the alignment sections is used as the backup. An input snippet is assigned the same label as its most similar snippet, that is, the snippet with the highest similarity score, in the training data. The RED+SVM classifier uses a SVM classifier as the backup. To train the SVM classifier, we used the sequential minimal optimization (SMO) algorithm with BOW features which are generated using the PTB tokenizer, treating each token as a feature. Each feature's value is set to the token's frequency. The SMO algorithm was implemented by the data mining software WEKA with a linear kernel as well as other WEKA default parameters.

Evaluation

First, we applied the RED+ALIGN and RED+SVM to the SMOKE and PAIN datasets. Ten-fold cross validation was performed and accuracy, recall, precision, and F-measure were calculated. Please note that during the training, we do calculate these performance measures for each individual expression for filtering and selection purposes, but only use the train data. The 10-fold cross validation results are produced using testing data. For comparison, we then tested the same SVM classifier used in the RED+SVM in 10-fold cross validation. Lastly, to better understand the learning behavior of the RED classifiers, we created learning curves for the three classifiers. To create the learning curves, we used 50 instances as the initial sample size and increased the sample sizes by a batch size of k=50 until we reached the total number of annotated instances. At each sample size we performed 10-fold cross validation 50 times and calculated the average accuracies. We also performed paired the t test between RED+SVM vs SVM and RED+ALIGN vs SVM at each different sample size.

Results

On average, in each cross validation, RED generated 8955 unique regular expressions from the SMOKE dataset and 25 433 unique expressions from the PAIN dataset. RED expressions recognized 84.6% of the SMOKE text and yielded 84.8% precision. On the PAIN dataset, RED expressions matched 90.9% of the test data, with 83.1% precision. Please note that these precisions are only on matched strings. Examples of correct matching and incorrect matching classification are shown in table 2. We observe that patterns with higher predicting score and F-measure from the training data tend to provide correct classification on the testing data as well.

Table 2.

Examples of correct and incorrect regular pattern matching classification in the SMOKE and PAIN dataset

Regular expressions and matched strings	Predicting score	F-measure (%)
[S]\s+a\.\s+current\s+smoker\s+\[\s+x\s+\]\s+\[\s+\]\s+\[\s+\]\s+b\.\s+lifetime\s+non-smoker\s+\[\s+\]\s+\[(\s+\S+){0,1}\s+\]\s+\[\s+\]\s+ [S] a. current smoker [ x ] [ ] [ ] b. lifetime non-smoker [ ] [ ] [ ]	16.31	0.7
[S]\s+smoking(\s+\S+)*\s+\[\s+x\s+\]\s+\[\s+\]\s+\[\s+\]\s+ [P] c. past history of smoking [ x ] [ ] [ ]	7.07	1.3
[P]\s+he\s+quit\s+smoking\s+\d+\s+years\s+ago(\s+\S+){0,1}\s+ [P] he states he quit smoking 50 years ago	6.08	1.9
[P](\s+\S+)\s+yes\s+\[\s+\]\s+no\s+\[(\s+\S+){0,1}\s+\]\s+\S+){3,6}\s+\[\s+x\s+\]\s+\S+){0,1}\s+ [P] educated on smoking yes [ ] no [ ] congratulated on quitting [ x ]	6	2.9
[P](\s+\S+){1}\s+smoking(\s+\S+){0,1}\s+\d+\s+months\s+ago\s+ [S] pt relates that he has recently taken up smoking 8 months ago.	3.32	3.7
[HP]\s+the\s+patient\s+rates\s+his\s+pain\s+a\s+\d+/\d+\s+per\s+numeric\s+rating\s+scale\s+ [HP] the patient rates his pain a 3/10 per numeric rating scale	11	0.9
[HP]\s+a\s+\d+-year-old\s+male\s+with\s+chief\s+complaint\s+of\s+neck\s+pain\s+ [HP] a 55-year-old male with chief complaint of neck pain	9	0.9
[NP]\s+yes\s+\(\s+\)\s+no\s+\(\s+x\s+\)\s+ [NP] chest pain yes () no (x)	7	8.4
[HP](\s+\S+){2}\s+tab\s+take\s+one\s+tablet\s+by\s+mouth(\s+\S+){1,2}\s+active(\s+\S+){1,7}\s+for\s+pain\s+ [HP] methadone hcl 5 mg tab take one tablet by mouth every active 8 h for pain—next rx due 4/27/10 (mail)	6.48	0.9
[O](\s+\S+){0,2}\s+what\s+makes\s+the\s+pain(\s+\S+){0,2}\s+worse(\s+\S+){1}\s [HP] what makes the pain worse : activity	4.24	2.5

For each matching instance, the table presents a regular expression pattern, a matched string, as well as the predicting score, and F-measure of the pattern. The match is considered correct if the label of pattern is identical to the label of test string, incorrect if those are not identical. SMOKE labels: S, current smoker; N, nonsmoker; P, previous smoker; U, unknown. PAIN labels: HP, have pain; NP, no pain; O, other.

The classification performances of RED+ALIGN, RED+SVM, and SVM are summarized in table 3. On the SMOKE data, the RED+SVM classifier achieved the best accuracy of 83% (vs RED+ALIGN 81.5% and SVM 80.5%). On the PAIN set, the RED+ALIGN classifier provided the best accuracy of 81.2% (vs RED+SVM 80.9% and SVM 79.2%). In both cases, RED-related classifiers outperformed the SVM classifier alone, using BOW features.

Table 3.

Summary of a 10-fold cross validation classification performance

	SMOKE												PAIN
	RED+ALIGN				RED+SVM				SVM				RED+ALIGN			RED+SVM			SVM
	S	N	P	U	S	N	P	U	S	N	P	U	HP	NP	O	HP	NP	O	HP	NP	O
Class distribution	492	205	231	163	492	205	231	163	492	205	231	163	53	190	459	53	190	459	53	190	459
Classification	524	201	233	133	540	194	232	125	522	213	225	131	41	179	482	38	177	487	37	183	482
Accuracy (%)	81.5				83.0				80.5				81.2			80.9			79.2
Recall (%)	86.4	82.4	80.1	67.5	89.6	83.4	81.8	64.4	87.0	82.9	78.4	60.7	56.6	74.2	89.5	50.9	72.1	89.8	49.1	71.1	89.5
Precision (%)	81.1	84.1	79.4	82.7	81.7	88.1	81.5	84.0	82.0	79.8	80.4	75.6	73.2	78.8	85.3	71.1	77.4	84.6	70.3	73.8	85.3
F-measure (%)	83.7	83.3	79.7	74.3	85.5	85.7	81.6	72.9	84.4	81.3	79.4	67.3	63.8	76.4	87.4	59.3	74.7	87.1	57.8	72.4	87.4

Number of classified instances, accuracy, recall, precision, and F-measure are calculated for each label in the SMOKE and PAIN datasets. Bold font marks the best performance among the three classifiers: RED+ALIGN, RED+SVM, and SVM.

SMOKE labels: S, current smoker; N, nonsmoker; P, previous smoker; U, unknown. PAIN labels: HP, have pain; NP, no pain; O, other.

The learning progress of three classifiers is shown in figure 4. Clearly, all classifier performances are following the inverse log pattern of machine learning curves. A larger sample size benefits all classifiers, however the learning behavior of classifiers may differ. In the SMOKE dataset, the RED+SVM classifier performs almost the same as SVM until the training size reaches 400 samples. Then the RED+SVM curve surpasses SVM. The RED+ALIGN curve also surpasses the SVM curve after 800 samples. On the PAIN dataset, the RED+ALIGN and the RED+SVM curves mostly overlap since most of the classifications (90.9%) are performed by RED. Those classifiers start outperforming SVM at approximately 200 training samples. Overall, RED classifiers appear to perform better than SVM at larger sample sizes, but not at smaller sample sizes. The improvement is relatively small (1.3–1.4% on PAIN dataset, 1.4–3.0% on SMOKE dataset), but very consistent (p<0.0001) at larger sample sizes (see online supplementary appendix A).

Figure 4.

The learning curve shows the relationship between classification accuracy and training size. RED, regular expression discovery; SVM, support vector machine.

Analyzing the differences between RED and SVM further, we compared the classifications of RED+ALIGN with SVM. Their classifications overlap, but not completely (see figure 5). An important finding is that RED+ALIGN correctly classified a fair number of instances that were misclassified by the SVM classifier, which account for 8.1–10.3% of the total instances and 43.8–53.0% of SVM's misclassifications. In other words, the RED+ALIGN complements SVM.

Figure 5.

Comparing RED+ALIGN vs SVM, based on percentage of correct and incorrect classifications.

Discussion

Significance

We designed and tested RED, a new regular expression learning algorithm. This regular expression and sequence alignment-based algorithm is an alternative to the traditional, feature-based machine learning algorithms. Although their performance difference is relatively small (1.0–2.0%), further analysis showed that the RED+ALIGN classifier can uniquely classify 8.1–10.3% of the total test samples correctly, which accounts for 43.8–53.0% of SVM's misclassifications. At the same time, the SVM classifier uniquely classified 6.7–9.3% of the samples correctly (figure 5). This suggests that an ensemble approach could potentially combine the two classifiers in order to further improve the accuracy performance.

By our observation, RED is more effective on template text than SVM. At the same time, it requires a sufficiently large sample size to learn from since the RED learning curves were beneath or overlapped the SVM curves at lower sample sizes. One reason that RED-based classification performs differently than SVM is that the key we used for the expressions was generated by alignment and testing. We are more likely to identify meaningful tokens this way than by simple BOW. Additionally, the expressions are able capture the sequential relationship among tokens (eg, ‘pain: no (X) yes ()’ can be differentiated from ‘pain: no () yes (X)’). Ngrams (eg, unigram, bi-gram, tri-gram^22–24) could be used to capture those relationships as well. Nevertheless, certain relationships (eg, ‘quit smoking \d+ years ago’ ‘begun smoking \d+ years ago’) would require a much longer ngram than typically used.

Implication

Human-crafted regular expressions tend to score very high in precision and lower in recall. In this study, the RED-generated regular expressions achieved a recall rate of 84.6–90.9%, and precision rate was 83.1–84.8%. We deem the recall rates to be relatively satisfactory, while the precision rates may be improved. This could be accomplished by adding more constraints to the testing and filtering step (eg, we could limit the number of positive instances, the predicting score, etc). Boosting precision, however, may negatively impact recall.

Text classification techniques can be applied to text snippets (words, phrases, or sentences), as well as to paragraphs or documents. Similar to the genre classification of news stories, various types of clinical documents (clinical notes,^{23 25} radiology reports,²⁶ incident reports,²² pathology reports,²⁷ etc) have been classified according to their content. Paragraph-level classification deals with smaller chunks of text, such as online messages²⁸ or posts.²⁹ Snippets are even smaller chunks of text. Paragraph- and snippet-level classification sometimes requires an additional text extraction step, carried out either manually³⁰ or automatically³¹ to identify text spans of relevance. We tested RED on snippet-level classification since it is not typical to develop regular expressions for the paragraph- or document-level. The snippet-level results, however, may be further aggregated for document-level or patient-level classification tasks.

Limitations

On the PAIN and SMOKE datasets, none of the classifiers reached their plateau points. Thus we do not have a complete picture of RED learning behavior in comparison with SVM. In this study we focused on the description and testing of RED. For comparison, we only used SVM with BOW, as it is a popular and effective approach in text mining.^22–24 In future studies we plan to conduct more testing of RED with other classifiers and features.

Text classification performance is typically influenced by overall sample size, number of classes, and class distribution. The tasks in our experiments involve multiple classes. The RED algorithm does convert them into binary tasks to generate regular expressions. Nevertheless, smaller classes tend to have lower performance. We did not calculate AUC for each class mostly because neither of the two RED classifiers produces a single value which could be used as threshold. Regular expression matching could output the matching pattern's F-measure. However, unmatched snippets are processed through the alignment algorithm or SVM, which does not give us the same F-measure. In future work, we will explore the calculation of AUC for RED classifiers.

To minimize the search space and reduce running time, RED only makes use of a subset of regular expression syntax (non-white space, digits; whitespace; 0 or more times, 1 or more times, m to n times). It is possible though to generate more levels of abstractions, for example, distinguishing lower and upper case letters, differentiating symbols, or introducing stemming. Indeed, we have experimented with some of these abstractions. Depending on the use case, the benefits vary. On the other hand, the computation time increase exponentially with the levels of abstraction. On average, RED took 51 s to train on the 1091 smoking instances, and took 134 s to train on the 702 pain instances (i7 Intel core CPU, 3.4 GHz). The training speed largely depends on the number of unique regular expressions that are generated and evaluated.

Future work

We plan to optimize RED for information extraction tasks and validate it on use cases that require document-level or patient-level aggregation. We are also developing ensemble approaches that will combine RED+ALIGN with SVM and other classifiers, such as Naive Bayes, or conditional random field.

Conclusion

In this paper, we present a new regular expression algorithm that we have applied to two different clinical text classification tasks. The evaluations’ results suggest that the two classifiers using RED patterns were slightly better than SVM in performance and were able to correctly recognize many of the instances misclassified by SVM. We believe that using regular expressions to tap into the sequential relationships among salient words is a promising approach to improve text classification performance.