Using Natural Language Processing of Clinical Notes to Predict Outcomes of Opioid Treatment Program

Abstract

Potential of natural language processing (NLP) in extracting patient’s information from clinical notes of opioid treatment programs (OTP) and leveraging it in development of predictive models has not been fully explored. The goal of this study was to assess potential of NLP in identifying legal, social, mental, medical and family environment-based determinates of distress from clinical narratives of patients with opioid addiction, and then using this information in predicting OTP outcomes. Around 63% of patients reported improvements after completing OTP. We compared the results of Logistics Regression and Random Forest for predictive modeling. Random Forest model performed slightly better than logistic regression (75% F1 score) with 74% accuracy.

Clinical Relevance—

Psychiatric and medical disorders, social, legal and family-based distress are important determinants of distress in patients enrolled in OTP. These information are often recorded in clinical notes. Extraction of this information and their utilization as features fed to the models will lead to the enhancement of the performance of the OTP outcome predictive models.

I. Introduction

In recent years, opioid use disorder has been recognized as a significant public health problem leading to thousands of death in the United States. Opioid prescription and deaths due to heroin overdose have been increasing since 1999 [1]. In addition to high risk of contracting infection diseases, opioid abusers are exposed to the development of mental illnesses and increase in financial costs [2]. Prescription and illicit opioid abuse annual costs were estimated just over $55 billion in 2009 [3]. Methadone and buprenorphine has been reported as effective treatments for opioid dependence, and their widespread use could mitigate the negative health and societal effects of opioid use disorder [4]. Opioid Treatment Programs (OTPs) are among the licensed providers of medication for opioid abusers that usually require patients to take medication at a clinic.

Patient’s information in OTP are recorded in either structure format (e.g., admission and discharge date, diagnosis codes, medications and patient’s demographic data) or an unstructured format (e.g., clinical text notes in the form of discharge summaries and progress notes). Clinical text notes are documented by the treatment providers through recording the information told by their patients. These notes contain vast amounts of information about the patient that cannot be found in structured data. This information includes prescribed medications, detailed physical and mental health conditions, as well as indications of social, legal or family-based distress. Since this information has unstructured nature, it needs to be extracted and then categorized for further utilization and analysis in daily healthcare settings and research [5]. Recently, natural language processing (NLP) has been widely used to extract those information as much accurate and efficient as possible [6].

On the other hand, while significant amount of studies have been conducted on extend, prevention and treatment of the opioid addiction [7–8], research on the effectiveness of OTP and prediction of its outcomes is limited. Among these studies, very sparse numbers have used information extracted from clinical text documentations in their analysis. Untreated psychiatric and medical disorders, social or family environment, and legal distress (due to for example incarceration) have been considered as risk factors for opioid misuse are among important patient’s information that as above mentioned are recorded in clinical notes. Utilization of these information may enhance the accuracy of outcome prediction models [9–10].Various NLP tools exist to extract information from clinical notes, including Clinical Text Analysis and Knowledge Extraction system (cTAKES) [11], MataMap/MetaMap [12] and Clinical Language Annotation, Modeling and Processing (CLAMP) [13]. While MetaMap and cTAKES are general purpose NLP systems, CLAMP provides an integrated development environment with GUIs for the users who need to build customized NLP pipelines for their individual applications. In this study we used CLAMP to extract information from the clinical notes.

The aim of this study is to explore the application of NLP strategy for identifying determinates of distress from clinical narratives of patients with opioid addiction, and then using this information to predict OTP outcomes. The rest of this paper is organized as follows: Section II focuses on describing the corpus used in this paper and presenting our data cleaning pipeline developed to only keep the most informative notes for the analysis. This section also describes our adopted strategy in generating lexicons that contain the terms that are indication of five determinats of distress. Moreover, this section briefly introduces the identity extraction tools utilized in this work, and describes our procedure in defining the effectivenss of the OTP and in selecting the variables to construct the predictive models. The results of descriptive statistics and performance of the developed predictive models are presented in Section III, followed by discussion and conclusion in Sections IV and V, respectively.

II. Methods

A. Dataset

A dataset used in this study contained a harmonized aggregation of several relevant sources including notes taken by different providers (such as nurses, medical doctors and counsellors) in OTP, information on admission, transfer, follow-up and discharge records from OTP in the New York City area since 1960s. The dataset also contains patient’s demographic information, medication, daily intake logs and drug screening information. Data was collected from the New York State Office of Addiction Service and Support’s (OASAS) opioid treatment program for patients who received treatment at Mount Sinai Health System (MSHS) in New York City. This data includes admission records from May 1965 to March 2021. It contains 48,249 records of 31,685 unique patients. Around 11,000 patients were admitted to the program more than once. Out of 31,685 unique patients enrolled in OTP, 9511 have records of the notes documented by 10 general disciplines listed in Table I. This table also lists for each discipline the percentage of patients who have records of notes from the corresponding discipline. According to this table, the least common author types were assistant supervisor, clinic manager, financial counsellor and social work intern. The average number of documents per patient was 54 with maximum number of 780 and minimum number of 1 documents).

TABLE I.

GENERAL STATISTIC OF THE NOTES

Discipline	Number of patients	Frequency
Counsellor	6674	70%
Nurse	6339	67%
Physician assistant	6019	63%
Medical doctor	4353	46%
Social worker	2767	29%
Vocational rehab counsellor	1221	13%
Clinic manager	91	0.9%
Assistant supervisor	59	0.6%
Financial counsellor	1	0.01%
Social work intern	91	0.9%

B. NLP Tool

As mentioned earlier, there are several identity extraction (IE) systems developed to process the clinical text such as MetaMap, cTAKES and CLAMP. However, MetaMap and cTAKES are general-purpose type of IE systems and studies have shown that end users need to take substantial effort to adopt these NLP systems [14]. Moreover, it has been reported that the performance of existing general-purpose IE systems had often been reduced when they were applied without customization beyond their original purpose [15]. While CLAMP follows pipeline-based architecture composed of multiple NLP components that provides end users with a GUI in order to help them build their own customized NLP pipelines for their individual applications [16]. In this work we considered CLAMP’s default clinical pipeline. The list of this pipeline’s components are as follow: sentence boundary detection, tokenizer, part-of-speech tagger, section header identifier, abbreviation reorganization and disambiguation, named entity recognition, UMLS encoder and rule engine. The output of CLAMP contains the start and the end point of the word (or sequence of words) detected as entity within the text, the semantic tag associated with it, Concept Unique Identifier (CUI) number (along with RX-Norm code for the entities tagged as drug), assertion, and the actual text extracted as entity (Table II shows a screenshot from CLAMP output). The semantic tag is divided into 20 categories as follow: ‘temporal’, ‘treatment’, ‘problem’, ‘history’, ‘drug’, ‘test’, ‘strength’, ‘route’, ‘frequency’, ‘body location’, ‘course’, ‘duration’, ‘subject’, ‘condition’, ‘generic’, ‘lab value’, ‘form’, ‘dosage’, ‘severity’. The assertion can be “present” or “absent” (in case of negation). Information regarding each category can be found in CLAMP official site [17].

TABLE II.

Screenshot of clamp output

Start	End	Semantic	CUI	Assertion	Entity
138	143	temporal	null	null	daily
224	232	drug	C0028040, RxNorm=[7407]	present	nicotine
475	479	subject	null	null	wife
494	497	problem	null	present	ill
503	516	problem	null	absent	throat

As it can be seen in Table I, less than 1% of the patients have notes taken by the last four disciplines. This encouraged us to check how informative the notes from these disciplines are and how many informative terms they contain. To find the answer, we randomly selected 500 patients. For each patient we extracted the notes from all disciplines. For each discipline, we integrated the notes from all patients to a single large documents. We then used CountVectorizer from scikit-learn library in python to extract for each discipline the list of tokens and to count the frequency of each token. The input parameters min_df and ngram_range are set to be 2 and (1, 2), respectively. To prevent the CountVectorizer tool from considering the stop words, we constructed a list of stop words and passed it as the next input parameter to the tool [18]. The tokens were converted to lowercase and low frequency tokens and punctuation marks were discarded. Fig. 1 and 2 show the appearance frequency of the most frequent tokens (only the first 20) in notes taken by medical doctor and assistant supervisor. According to these figures, compared to the medical doctor the notes from assistant supervisor contain only 8 tokens (after deleting the stop words) which none of them are informative. The distribution for financial counsellor, clinic manager and social network intern were the same. As a result, in the next analyses we only considered the notes from the first six authors in Table I. For each patient, the notes from the first six disciplines were aggregated into a single large document and then fed as an input to the CLAMP. Next, the output of CLAMP for every patient were converted to CSV file and lastly all CSV files were appended together to generate one single large CSV file containing the entire extracted entities from the notes of 500 patients. This output was used to develop lexicons in the next step.

Figure 1.

Medical doctor.

Figure 2.

Assistant supervisor

C. Development of Lexicons

As mentioned earlier, some patient’s important information is not reflected in structured format. Information such as existence and the type of mental, medical, legal, social and family environment-based distress are often documented in clinical notes. This information can be extracted and used as new features to be fed to the predictive models in order to enhance their performance. In this study, we decided to recognize every word/term which are indication of mental, medical, legal, social and family environment-based distress, as five important determinants of distress, in OTP patient’s life. To do so, we first needed to generate separate lexicon for each determinant of distress. Since documented standard and data collection strategies for theses determinant of distress in the EHR are in an early stage, generating lexicon for each determinant was challenging. The initial list of terms were collected using standard terminologies in SNOMED-CT. On the other hand, since clinical notes are often documented as natural language, examining standard terminologies alone may lead to miss important information embedded in clinical notes. In order to include any variant of standard terms into the lexicons, we queried every term in every lexicons against extracted entities of the CLAMP’s output in the previous section. This resulted in incorporation of any spelling variant of each term and finding any relevant lexical representations. For example, the terms “low income” and “loss of income” are both indication of social distress but former is a standard term in SNOMED-CT while the term “loss of income” has been extracted through search into CLAMP output using regular expression (RegEx) patterns. An screenshot of CLAMP output can be seen in Table II.

Next, we used two domain expert’s assessment to construct the final lexicons that appropriately represents social, legal, mental, medical and family environment-based distress. The terms indicating mental distress included “anxiety”, “depression”, “adhd”, “insomnia”, “psychiatric disorders”, “borderline personality disorder”, “ptsd disorders”, “substance induced psychological disorders”, “dissociative identity disorder”, “multiple personality disorder”, “panic disorder”. The terms indicating social distress included “low income”, “loss of income”, “financial issues”, “immigration issues”, “loss of housing”, “homelessness”, “homelessness issues”, “job loss”, and “work related issues”. The terms that are indication of medical problem included “diabetes”, “uncontrolled high blood pressure”, “heart attack”, “surgery in left knee”, “obesity” and “asthma”. Regarding the terms indicating distresses rooted in family environment we assumed that the appearance of the terms in the note is a direct indication of family-related distress in the patient’s life and the validity of this assumption were manually assessed by two domain experts. The terms included “wife”, “dead wife”, “brother”, “husband”, “interpersonal relationship issues”, “family issues”, “marital issues”, “family stress”, “increased family stress”. Lastly, the terms indication legal distress included “legal problems”, “legal related issues”, “criminal legal issues”, “second arrest”, “multiple arrests”, “recent release from prison”, “drug related arrest”, “multiple drug related incarcerations”, “recent incarceration”. The size of lexicons is as follow: 143, 38, 236, 66 and 31 for respectively mental, legal, health, family and social distress lexicon. The work [19–20] had used the same process of development lexicons as we utilized in this work. Programming language for all analysis was Python 3.8.

D. Study Design

The first step in predicting the OTP outcome is to define a metric as an outcome. Literature have considered variety of metrics as treatment outcomes including mortality, retention and abstinence rates, continuing illicit opioid use and change in mental health [21–22]. To determine patients’s treatment effectiveness, in this study we focused on discharge records and the variable named as “discharge status”. This variable recorded different unique reasons of discharge from the program among which we focused on four cases as follow: “completed treatment: all treatment goals met”, “completed treatment: half or more goals met”, “treatment not Complete: some goals met” and “treatment not complete: no goals met”.

The patients with the first three discharge status were considered as the patients whom treatment was somehow successful, while the patients with the last status identified as failed treatment cases. Out of 31,065 unique patients discharged from the program, 19,453 patients belong to the successful treatment while 9,524 patients met no goals of the treatment. Out of 19,453 patients with successful treatment, only 2,955 had notes available. We extracted notes from randomly selected 150 patients and put them in a folder called as “succeeded”. While out of 9,524 patients with failed treatment, only 1,678 had notes available out of which we extracted notes from 150 random patients and put them in a folder called as “failed”. For each patient in each group, the notes from the first 6 disciplines in Table I were aggregated into a single large note.

In order to develop a model predicting the effectiveness of the treatment in OTP through leveraging features extracted from the notes, we first extracted entities for each patient’s large note in each group using CLAMP. Next, we counted the number of words out of five lexicons that appeared as extracted entity for each patient across both groups. Table III shows a screenshot of this analysis for “succeeded” group. It should be noted that two domain expert reviewers manually assessed the notes and confirmed that the existence of any term from the five lexicons is indication that the patient actually experiences that specific distress in his/her life. We further added a new feature as “geometric mean” which calculated by raising the product of word counts for all determinants of distress to the inverse of the total length of determinants (i.e. 5). Since different patients may have different dimensions of distress affected, we use geometric mean to compare overall distress level between different patients.

TABLE III.

Screenshot of frequency count in succeeded group

Patient ID	Mental distress	Social distress	Legal distress	Medical distress	Family distress	Geometric mean
Patient 1	23	1	1	155	45	10.99
Patient 2	1	1	1	199	1	2.88
Patient 3	1	1	1	45	1	2.14

E. Variables

In predictive modeling, we not only used the features from the notes (Table III), but also leveraged the information from discharge form. We used patient’s socio-economic status, living situation, education level and employment status. The variable “age” was defined using the first admission date and date of birth. Number of arrestment, number of days in detox, and secondary and third discharge substances were considered as input variables to the model as well. A new variable “length of enrollment” has been added too. This variable was defined using first admission date and the date of discharge. All variables were preprocessed as follow: all categorical variables were converted to numerical variables and then were normalized before feeding to the model. As mentioned earlier, to determine treatment effectiveness we focused on discharge status of the patients and defined two groups/classes of succeeded and failed patients. The patients in succeeded group were labeled as 1 while the patients in the other group labeled as 0.

In training phase, 70% of the data were used and the rest were assigned for testing (the split was random). We adopted logistic regression with regularization (λ=0.01) and random forest (number of estimators was set to 100) and compared the result of these models. Evaluation metrics were “accuracy” and “f1-score”. It should be noted that this project has been approved by institutional ethics board.

III. Results

The average age in succeeded group was 45 years old while it was 41 in failed group. Around 22% and 32% of patients were female in failed and succeeded group, respectively. Around 98% of patients in both group listed heroin as their primary abuse substance. Around 26% of patients who failed in treatment are Black or African American, 29% White. While the distribution of Black or African American and White race among patients with successful treatment is: 19% and 36%, respectively. Percentage of patients with Hispanic ethnicity is around 43% and 33% in failed and succeeded group, respectively. Most of the patients in both groups consume no secondary substance. Moreover, just over 27% of patients in failed group consume cocaine as their secondary substance while it is around 18% in the other group. Succeeded patients with high school diploma are 48% against 36% in failed group. The percentage of patients failed in treatment without high school diploma (41%) is about 14% more compared to the patients in succeeded group. 18% of succeeded patients are employed while employment rate is less than 9% in failed group. The average value for variable “length of enrollment” in failed group is around 1.9 years old with standard deviation value of 4.3 years old, while in succeeded group the mean and standard deviation for the same variable are 3.3 and 6.7 years old, respectively.

Both random forest and logistic regression performed well in predicting the effectiveness of the treatment with 75% and 73% F1-score, respectively.

IV. Discussion

In comparison to the patients who met treatment goals, patients with unsuccessful treatment are younger with higher percentage of Black African American and Hispanic, higher rate of unemployment, lower education and higher rate of cocaine consumers. The age, employment and education status, race and the type of secondary substance can be considered as factors that may contribute to opioid abuse. Moreover, patients in successful treatment represented by the longest duration in OTP. After incorporation of various variables, random forest performed slightly better, in terms of F1-score and accuracy, in comparison with logistic regression in predicting the effectiveness of the treatment in OTP.

V. Conclusion

Part of the data recorded in OTP are in the form of narrative text notes without structured format. While they feature a gold mine of information. This information must be extracted and converted to a structured format to be later used for quality improvement and also enhancing the performance of predictive models. In this paper, CLAMP as an automated system was used in order to parse information from clinical notes. In specific, we used NLP for identifying legal, social, mental and medical determinates of distress along with source of distress rooted in family environment from clinical narratives of patients with opioid addiction, and then used this information to predict OTP outcomes. Five lexicons were generated for all five determinants of distress. The lexicons generated in this study combine standard concepts and domain expert knowledge. Additional variables such as socio-economic status, age, living situation, education level and employment status and length of enrollment in OTP were incorporated in developing the predictive models. Two predictive models were developed: logistic regression and random forest. Random forest outperformed logistic regression by 2% for F1-score. The age, employment and education status, race, the type of secondary substance and longer enrollment in OTP can be considered as factors that may contribute to opioid abuse.

TABLE IV.

PREDICTIVE MODELS RESULTS

	F1-Score	Accuracy
Logistic Regression	0.73	0.73
Random Forest	0.75	0.74