Identification of Gout Flares in Chief Complaint Text Using Natural Language Processing

Abstract

Many patients with gout flares treated in the Emergency Department (ED) often do not receive optimal continuity of care after an ED visit. Thus, developing methods to identify patients with gout flares in the ED and referring them to appropriate outpatient gout care is required. While Natural Language Processing (NLP) has been used to detect gout flares retrospectively, it is much more challenging to identify patients prospectively during an ED visit where documentation is usually minimal. We annotate a corpus of ED triage nurse chief complaint notes for the presence of gout flares and implement a simple algorithm for gout flare ED alerts. We show that the chief complaint alone has strong predictive power for gout flares. We make available a de-identified version of this corpus annotated for gout mentions, which is to our knowledge the first free text chief complaint clinical corpus available.

Introduction

Gout affects over 9 million Americans^1,2 and is the most common form of inflammatory arthritis in men with a prevalence rate over 5%^1,3. Gout is associated with poor quality of life⁴ and the frequency of gout is increasing worldwide, with prevalence rates estimated to be as high as 7% in older men^2,5,6. Gout leads to work absenteeism, loss of productivity, increased healthcare utilization, and premature death^7–12. The U.S. National ED Sample (NEDS) reports over 200,000 visits annually with gout as the primary diagnosis, accounting for 0.2% of ED visits and over $280 million in annual billable charges¹³. In addition, NEDS data demonstrate a marked increase in the number of ED visits for gout in the U.S. from 168,000 in 2006 to 214,000 in 2014, with more than 1.7 million people being seen in an ED for acute gout¹⁴. Gout patients are prescribed opioids in over 50% of ED encounters, potentially attributable to ED physicians de-prioritizing gout specific management in favor of treatment of more severe conditions¹⁵.

The ED at our academic medical center evaluates over 700 patients a year with gout. Older age, male sex, lower income, and African American race have shown to be associated with increased ED gout visits and greater gout-related expenses^13,14. In an aging population that is living longer with more serious chronic comorbid conditions, the societal burden posed by gout is growing exponentially and will substantially burgeon over the coming decades^14,16. Based on this epidemiology, the ED is a natural place to improve gout care. Most patients seen in an ED will be discharged home; many with inadequate follow-up to outpatient gout care. Often, at the time of initial gout symptoms, patients have accumulated significant deposits of urate, leading to life-long intermittent, if not chronic, symptoms without treatment. Gout is also quite frequently accompanied or preceded by other chronic diseases^17–22. Thus, providing continuity of care after the ED visit for an acute flare is the cornerstone for improving outcomes of patients with gout and reducing healthcare-related costs^23–25. Because gout is a chronic condition which develops in the presence of hyperuricemia, achieving low serum urate concentrations using long-term urate lowering therapy (ULT) is essential for improving outcomes^26–29.

Identification of Gout Flares using Natural Language Processing

Defining gout flare in clinical notes is challenging. Despite the difficulty, gout flares have been detected in administrative claims data^30,31 using a combination of diagnosis or billing codes (ICD-9-CM) and prescriptions to identify patients. More recently, a gout flare detection algorithm combining Natural Language Processing (NLP) with Machine Learning (ML) has been used to find gout flares in patient notes retrospectively³² and has been applied successfully to identify risk factors for gout flares³³. This algorithm³² identified gout flares in clinical notes using an existing diagnosis of gout and a prescription for ULT with a recall (i.e., sensitivity) of 82.1%, precision (i.e., positive predictive value) of 77.9% and a F-Score of 87%. Using the output of the NLP process as inputs to a ML algorithm the investigators were able to identify more than twice as many gout flare cases in patients using NLP, 18,869 cases versus 7,861 using only claims data. Clinical notes from 200 patients were annotated and split evenly into a training and testing data set to develop the algorithm and to our knowledge, this work represents the state of the art in gout flare detection. Unfortunately, neither the implementation or a de-identified, labelled set of those notes for training is publicly available for download to facilitate a local implementation. Moreover, retrospective studies are not a feasible strategy to identify patients while receiving care in the ED since gout billing codes are not reliable and such data is available too late to allow for patient contact in the ED and prompt referral to outpatient gout care. Additionally, patients without a previous diagnosis of gout who are experiencing a gout flare or not taking ULT cannot be captured by this method. Finally, while NLP and ML approaches offer the best approach currently³² to identify gout flares, they generally offer limited applicability in an Electronic Health Record (EHR) alerting environment. For example the Cerner Command Language (CCL) used in the alerting environment at our institution does not support the loading or development of pre-trained deep learning NLP frameworks such as BERT³⁴. This level of support from EHR vendors would be ideal, given ”the field of natural language processing has been propelled forward by an explosion in the use of deep learning models” like BERT and other transformer styled derivatives³⁵.

To promote continuity of care for patients with gout and extend others’ work to the ED setting another approach is needed, one that can allow patient identification in the ED and efficient referral to outpatient gout care without further burdening the ED clinical care team. Given the delayed nature of billing and the potential lack of available ULT therapy information on patients who are initially diagnosed with gout in the ED, one potential avenue for identification of acute gout ED patients is NLP of a patient’s chief complaint (CC). Processing of the CC dates at least as far back as Johnson and Friedman³⁶, who processed the CC section of discharge summaries in 1996. The use of CC was later investigated for detection of asthma visits by McLung³⁷ (achieving 37% sensitivity and 97% specificity). At around the same time the importance of the CC to identify possible bioterrorism attacks also became apparent in the post 9-11 discussion of the 2001 AMIA Roundtable on Bioterrorism Detection³⁸. The CC in its own right was not a subject of analysis of NLP studies until 2002, when Travers and Bodenreider³⁹ created a CC ED vocabulary. Subsequent work⁴⁰ extended the use of the free-text CC to syndromic surveillance by classifying free-text CC into 7 syndromic categories. However the primary focus for NLP has been biosurveillance, as reviewed by Chapman⁴¹ since emerging diseases will likely contain biosurveillance information of interest accessible to NLP long before such information had a SNOMED or ICD-10-CM diagnostic code. Despite the relevance of the CC, a dedicated CC containing de-identified corpus has not been made publicly available, nor has any previous work been published on the utility of the CC in the detection of gout flares.

In this study we develop a corpus of ED triage nurse chief complaints that are annotated for the presence of gout flare. We use them to develop NLP algorithms to identify gout flares in the ED, including a rule-based algorithm deployed in Cerner^TM environment and a BERT³⁴-based algorithm to compare the relative effectiveness of a state of the art deep learning approach versus a rule-based approach. Finally, we assess the utility of ICD-10-CM gout flare codes in identifying instances of gout flare and make all of our data and implementation details available to the public.

Method: Development of GOUT-CC-2019-CORPUS and CC-2020-CORPUS

Similar to Zheng et al.³² we looked at keywords for the initial (Beta) development of our NLP algorithm and corpus. In our case, many gout keywords mentioned in their work including ’podagra’, ’tophaceous’, ’tophi’ and ’tophus’ were not found to be in current use in chief complaints (CCs) at our institution. Therefore our gout flare enriched corpus GOUT-CC-2019-CORPUS was generated using only a single keyword, ”gout”, to identify 300 CCs for the initial development of the algorithm, including any needed allowance for oversampling of the rare gout flare class.

Each CC was annotated to indicate whether the CC (and only the CC) was indicative of a gout flare, not indicative of a gout flare, or unknown with respect to gout flare. This data set was double-annotated by a practicing rheumatologist (MID) and a PhD informatician. Thereafter, a full manual chart review was performed by one rheumatologist (MID) and a post-doctoral fellow (GR) to determine gout flare status for 197 of the 300 ED encounters. A sample of chief complaints with consensus annotations is shown in Table 2.

Table 2:

GOUT-CC-2019-CORPUS Examples

Chief Complaint Text	Predicted*	Actual**
AMS, lethargy, increasing generalized weakness over 2 weeks. Hx: ESRD on hemodialysis at home, HTN, DM, gout, neuropathy	No	No
”I started breathing hard” hx-htn, gout, anxiety,	No	No
R knee pain x 8 years. pmh: gout, arthritis	Unknown	No
Doc N Box DX pt w/ R hip FX on sat. Pt states no falls or injuries. PMH: gout	Unknown	No
out of gout medicine	Yes	Yes
sent from boarding home for increase BP and bilateral knee pain for 1 week. Hx of HTN, gout.	Yes	Yes

^*Consensus predicted gout flare status determined by annotator examination of CC

^**Gout flare status determined by chart review.

A final held out set of 8042 chief complaints from 2020 (CC-2020-CORPUS) was selected without regard to the presence of any gout related keywords (thus containing a distribution of alerts representative of that time period). A total of 5 CCs were removed from the original 8042 CC since 4 of them contained distinguishing information on suicide attempts, assaults or description of incidents that could allow for identification of the CC patient by those familiar with the incident. An additional CC was removed because the encounter could no longer be found in the EHR leaving a corpus of 8037 CCs. Similar to the earlier process, two annotators (JDO and AM) screened for the presence of gout flares using CC, double annotating a set of 300 mentions to compute annotator agreement. MID and GR performed an ED clinical note review on all CCs identified as either indicative or unknown, using the criteria for a gout flare as described by Gaffo et al⁴² as guide. Additionally, a sample of 100 patients screened as negative for gout flare by CC alone had their chart reviewed by MID to verify negative gout flare status. The negative cases for review were selected by the presence of gout related keywords from the GOUT-GAZETEER and LOCATION-GAZETEER to strengthen our confidence that no gout flares were left unannotated. The gazeteers are shown in Table 1. Annotator agreement was calculated separately for this corpus and all annotator agreements are presented in Table 3.

Table 1:

Gazeteer Description

GOUT-GAZETEER Case Insensitive	LOCATION-GAZETEER Case Insensitive	LOCATION-GAZETEER Case Sensitive	PMH-GAZETEER Case Insensitive
gout	toe   knee	LLE	pmh
pain	ankle   wrist	RLE	hx
tender	foot   feet	LUE	pmxh
swelling	thumb   finger	RUE	pmx
swollen	arm   elbow	UE
stiff	shoulder   hip	LE
	leg   hand

Table 3:

Annotator Agreement in GOUT-CC-2019-CORPUS and CC-2020-CORPUS

Corpus Name	GF* POS	GF NEG	GF UNK	Review**	Agreement	Cohen’s κ
GOUT-CC-2019-CORPUS	93	194	13	CC	0.883	0.825
GOUT-CC-2019-CORPUS	70	118	9	Chart	0.849	0.774
CC-2020-CORPUS	14	7992	129	CC	0.977	0.965
CC-2020-CORPUS	25	232	7	Chart	0.904	0.856

^*GF (Gout Flare)

^**Review refers to the process that was made to determine gout status. Chart indicates a full chart review, CC indicates that only the chief complaint was used to predict whether a patient was presenting with a gout flare.

Method: De-identification and Distribution of Corpus

Two authors (JDO and TO) performed an initial manual review for personal information in 2000 CCs from the CC-2020-CORPUS to gain insight into the dataset. We then used the fine-tuned BERT³⁴ and ALBERT⁴³ models using Flair⁴⁴ by training on the i2b2 2014 de-identification dataset⁴⁵ to identify PHI using IOB sequence tagging. Fine-tuning was done over 70 epochs with a batch size of 32 with an initial learning rate of 0.1 and hidden size of 256 for both the BERT base uncased model and ALBERT base v2 model. We created regular expressions for matching date, addresses, medical locations, and for additional terms specific to the dataset including names/abbreviations of police departments, ambulance services, local medical care centers. The resulting tagged text was converted into BRAT ”.ann” format and uploaded as CC documents to a local BRAT server⁴⁶ for manual verification of PHI removal for the remaining 6142 CC documents. Data for this project was collected under IRB-300004156 and IRB-300001664; Improving Care for Gout in the Southeast Enhancing Gout Minority Patients Care and Participation in Gout Clinical Research. The data is approved for distribution per the U-BRITE Deidentified Translational Data Repository for Research and Education (IRB-300002212) with a data use agreement requiring the recipient to not redistribute or try to re-identify the data. The corpus is being made available under the PhysioNet Credentialed Health Data License 1.5.0 Physionet on Physionet (https://physionet.org/) as the ”Gout Emergency Room Chief Complaint Corpus”.

Method: Development of Gout Alert

The alert was developed in Cerner Command Language (CCL) using Discern Visual Developer and utilized the CC as its primary data source to identify gout flares in ED patients. The initial algorithm was developed in Jupyter Notebook (https://jupyter.org/) with data analysis and visualization using Pandas⁴⁷ and MatPlotLib⁴⁸ to develop an algorithm capable of being translated to the Cerner^TM alert environment at UAB that supports CCL based alerts only. Gazeteers as shown in Table 1 were developed as part of this analysis on the GOUT-CC-2019-CORPUS in conjunction with discussion with all authors. The algorithm was developed iteratively using the enriched GOUT-CC-2019-CORPUS until an acceptable performance was achieved. The algorithm works on simple Boolean logic and regular expressions and will identify a gout flare based on the presence of ”gout” to the left of any of the PMH-GAZETEER words OR if there is both a location from the LOCATION-GAZETEER and the ”gout” keyword in the past medical history. Other gout related keywords in GOUT-GAZETEER are not used in this algorithm. A python implementation that could be rewritten for vendor software is available for download on Github (https://github.com/ozborn/gout_chief_complaint_alert). We refer to this regular expression and list based algorithm for the detection of gout flares as SIMPLE-GF. We also developed a high recall algorithm termed NAIVE-GF which predicts every mention of gout as a gout flare case.

Method: Development of Chief Complaint BERT-based Gout Flare Detection

We trained both a 2019 and a 2020 BERT³⁴ model to detect gout flares from chief complaints using stratified 10-fold validation, where 1 fold was held out as the development or validation set and 1 fold as the test set. Test set labels for the GOUT-CC-2019-CORPUS were derived only from ED encounters that had a full manual chart review performed by MID and GR in the GOUT-CC-2019-CORPUS, when there was disagreement we chose the label given by the experienced rheumatologist (MID).

Training labels for GOUT-CC-2019-CORPUS were derived not from chart review, but from the assessment of gout status using CC only done by MID and JDO where MID’s annotation took precedence.

For training and testing on the CC-2020-CORPUS, we combined the U label and the Y label due to the data class imbalance which is reflective of the overall low number of ER visits related to gout. Single annotated training data was derived by gout flare classification using only the chief complaints by JDO and AM. A consensus label for the test set in CC-2020-CORPUS was derived from chart review by GR and MID, with discrepancies in the double annotated test data set resolved by discussion. To further increase the gout flare positive class for training purposes we over-sampled this class by adding the gout flare enriched GOUT-CC-2019-CORPUS to the training data.

For both the 2019 and 2020 models we used ”BERT-base-uncased” word embeddings and a document rnn embedding with 1 layer with a hidden size of 512, which re-projected words into 256 dimensions, with a batch size of 32 and initial learning rate of 0.1. All models were trained for 50 epochs. We refer to this BERT based algorithm in the paper as BERT-GF.

Results: Characteristics of Chief Complaint Gout Corpus

The CC itself for the larger CC-2020-CORPUS is on average 108 characters and is densely packed with extensive use of abbreviations and acronyms. A total of 7996 negative, 129 unknown and 14 positive annotations were determined by CC review. The inclusion of prior medial history (PMH) is widespread with approximately 80% of CC having some mention of PMH appearing at the end of the CC. Cohen’s kappa was calculated on both corpora for gout flare determination based on CC and based on chart review as shown in Table 3. The number of positive, negative or unknown gout flares in Table 3 corresponds to the consensus annotation for each data set.

Results: ICD-10-CM Billing Codes for Gout Flare Diverge from Chart Review Determined Gout Flare Status

The use of ICD-9-CM codes to represent the diagnosis of gout flares is presumed to be less reliable^32,49, although we are unaware of a published quantitative evaluation for ICD-10-CM. We show in Table 4 that ICD-10-CM M10 prefixed gout flare codes poorly predict the underlying gout flares as determined by a rheumatologist chart review in GOUT-CC-2019-CORPUS. Our results suggest that ICD-10-CM gout billing codes over-represent the ED gout flare burden.

Table 4:

ICD-10-CM M10 Prefixed Codes vs Chart Reviewed Gout Flare Status

Code Types	Precision	Recall	F1 Score
Primary and Secondary Code	0.43	1.00	0.60
Primary Code	0.56	1.00	0.72

The vast majority (over 91%) of gout flare codes used at our academic ED in 2019 are represented by M10.9 (Gout, unspecified). The remaining codes have a widely distributed range of M10 coding suffixes, with only M10.071 and M10.072 (Idiopathic gout, left and right foot respectively) and M10.00 (idiopathic gout, site unspecified, used to indicate a diagnosis for reimbursement purposes) with a frequency higher than 1%.

Results: Chief Complaint is Highly Informative of Gout Flare Status as Determined by Chart Review

We show in Table 5 that CC is highly informative as to gout flare status, providing a high level recall in both data sets when examined by annotators for the presence of gout flares. The results for precision (positive predictive value), recall (sensitivity) and F-score are shown for the gout flare positive class only for both GOUT-CC-2019-CORPUS and CC-2020-CORPUS. Human annotator recall is better estimated on the GOUT-CC-2019-CORPUS since a higher proportion of the gout flare negative class was subject to chart review as described in the methods.

Table 5:

Prediction of Gout Flare by Chief Complaint

	GOUT-CC-2019-CORPUS			CC-2020-CORPUS
Annotator or Algorithm	Precision	Recall	F1 Score	Precision	Recall	F1 Score
MID	0.88	0.92	0.90	NA	NA	NA
JDO	0.84	0.97	0.90	0.56	1.0	0.71
AM	NA	NA	NA	0.67	1.0	0.80
Human Average	0.86	0.95	0.90	0.62	1.0	0.76
NAIVE-GF	0.23	1.00	0.38	0.28	0.56	0.37
SIMPLE-GF	0.44	0.84	0.58	0.37	0.40	0.38
BERT-GF	0.71	0.48	0.56	0.79∗	0.47∗	0.57∗

^∗BERT-GF for the CC-2020-Corpus used a combined dataset of both corpi, due to the heavy class-imbalance in CC-2020-Corpus.

The NAIVE-GF algorithm (which simply searches for mentions of ”gout”) performs slightly better on CC-2020-CORPUS due to the relative rarity of gout mentions. However this algorithm fails to distinguish between a patient’s past medical history (PMH) of gout versus a current flare which is a problem when PMH is found in approximately 80% of gout notes. References to a PMH of gout in the CC may be particularly useful in identifying gout patients. These references may be relatively more reliable than PMH in other types of clinical notes because the past medical history field may not always be kept up to date in the EHR. Moreover, the PMH mentioned in the CC can include elements of PMH discovered in the EHR by the ED triage nurse as well as any current representation of a patient’s account of their past medical problems.

A distribution of gout-associated body location mentions from that corpus is shown in Figure 1 using CC from all patients from CC-2020-CORPUS and chart review identifying gout flare respectively. The SIMPLE-GF uses these keywords to identify gout flares, privileging recall over precision for mentions like ”arm” even though most such mentions are not gout-related.

Figure 1:

Distribution of LOCATION-GAZETEER Body Locations in CC of GOUT-CC-2019-CORPUS

Discussion

Our results indicate that despite the distinct and numerous M10 prefixed ICD-10-CM codes for gout flare, such codes are a poor proxy for the presence of a gout flare in the ED. It is unclear whether the high preference for M10.9 prefixed codes represents an incorrect assessment of gout flare occurrence among coders or miss-classification versus a more specific or appropriate M10 prefixed code such as M10.00. Regardless of assessment accuracy, these codes are assigned post-visit and therefore no use for alerting.

Our results also indicate it is possible for non-physicians to make reasonable predictions (F1 score of 0.76 to 0.90 for CC-2020-CORPUS and GOUT-CC-2019-CORPUS respectively) using only information available in the CC. This is even more notable considering this F1 score is only slightly worse than the F-Scores of a full chart review (0.88 and 0.93)³². One possible interpretation of this performance similarity may be that the ED triage nurse does an informal review for that patient at time of ED visit, thus already incorporating such information into the CC. An alternative interpretation may be that EHR notes may be more difficult to process given the amount of information they contain compared to the terse, free-text represented in a CC. The latter interpretation is in line with the utility of other short text resources like Twitter, which has been shown to be useful for non-clinical event extraction⁵⁰. Our results suggest that the CC captured at our institution show promise for identification of gout flare cases during the ED visit and could be utilized to activate referral pathways to outpatient gout care or to clinical research studies.

Finally, although the language used to describe gout is ”marked by ambiguity and imprecision”⁵¹ in the scientific literature, in the context of an ED visit, a CC with a mention of gout and an involved body part is highly informative. Even discounting whitespace, the 4 characters comprising ”gout” are surprisingly specific in the English language and the only ”gout” mention not referring to gout we found in a CC was the spelling mistake ”througout” or mentions of pseudogout, a disease that has similar clinical presentation to gout. Body locations referencing common sites of gout flares included in expressions such as ”knee surgery” and ”drop foot” were more tedious to rule out. Nonetheless, we believe this preliminary work could be bolstered with more data, particularly more instances of gout flares that lack explicit gout mentions. The performance of BERT-GF on CC-2020-CORPUS is a likely result of this lack of training data that our over-sampling approach could not fully remedy.

Limitations

Our findings should be interpreted in the light of some limitations. Our algorithm was developed and tested at a single academic ED and due to lack of available public data, we could not systematically compare the composition of our institutional CC data to the composition of CC for ED visits other institutions. Common practices such as documenting prior medical history in the CC of patients presenting to ED or other quality or writing standards in place at at our medical center may not generalize to other institutions. We also lack a comparison of gout flare prediction versus a retrospective chart review which would better measure the performance of our BERT based algorithm versus previous published gout flare detection algorithms.

Conclusion

Our results show that ICD-10-CM gout flare codes are a poor indicator for gout flares. We make available via Github^TM an ED algorithm for the detection of gout flares in the CC with performance similar to modern machine learning approaches on gout enriched corpora, that can be deployed in Cerner^TM and should be feasible to implement with other major EHR vendors. We show that the CC can be strongly informative for gout flares and generate 2 de-identified CC corpora available for download, GOUT-CC-2019-CORPUS and CC-2020-CORPUS. We also demonstrate that ED detection of gout flares is possible with CC, but that vendor limitations may make deployment challenging. In the future, we plan to test the ability of the NLP ED algorithm to activate efficient referral pathways to outpatient care and to clinical research studies to improve outcomes of patients with gout.

Acknowledgement

This publication was supported by funding from NIH grant 3P50AR60772-08S1 and a NVidia^TM grant of a Titan XP GPU used for machine learning.