Natural Language Processing with Machine Learning to Predict Outcomes after Ovarian Cancer Surgery

Abstract

Objective:

To determine if natural language processing (NLP) with machine learning of unstructured full text documents (a preoperative CT scan) improves the ability to predict postoperative complication and hospital readmission among women with ovarian cancer undergoing surgery when compared with discrete data predictors alone.

Methods:

Medical records from two institutions were queried to identify women with ovarian cancer and available preoperative CT scan reports who underwent debulking surgery. Machine learning methods using both discrete data predictors (age, comorbidities, preoperative laboratory values) and natural language processing of full text reports (preoperative CT scans) were used to predict postoperative complication and hospital readmission within 30 days of surgery. Discrimination was measured using the area under the receiver operating characteristic curve (AUC).

Results:

We identified 291 women who underwent debulking surgery for ovarian cancer. Mean age was 59, mean preoperative CA125 value was 610 U/ml and albumin was 3.9 g/dL. There were 25 patients (8.6%) who were readmitted and 45 patients (15.5%) who developed postoperative complications within 30 days. Using discrete features alone, we were able to predict postoperative readmission with an AUC of 0.56 (0.54 – 0.58, 95% CI); this improved to 0.70 (0.68–0.73, 95% CI) (p<0.001) with the addition of NLP of preoperative CT scans.

Conclusions:

Natural language processing with machine learning improved the ability to predict postoperative complication and hospital readmission among women with ovarian cancer undergoing surgery.

INTRODUCTION

Ovarian cancer is treated with radical surgical debulking, which is associated with high rates of postoperative complications and hospital readmissions. Rates of complications can be as high as 50% in randomized trials of radical surgery [1]. These major postoperative complications lead directly to morbidity, and subsequently to decreased survival secondary to delays in chemotherapy [2]. Predicting which patients will develop complications or hospital readmissions is useful for resource allocation, counseling, and treatment decisions. Additionally, patients who undergo upfront cytoreductive surgery have consistently higher rates of postoperative complication than those treated with neoadjuvant chemotherapy [1, 3, 4]. Thus, the use of neoadjuvant chemotherapy prior to surgery could be used for those patients at the highest risk of postoperative complications.

Our current efforts to predict postoperative complications and hospital readmissions have focused on the association between traditional surgical risk factors, such as age or preoperative albumin, and postoperative outcomes [5–7]. In addition to risk factors in discrete data fields, we now have access to the rich free text data within the electronic medical record; however, harnessing these data can be difficult. The use of machine learning and other artificial intelligence techniques have emerged as potential solutions. Machine learning refers to a broad collection of statistical techniques and algorithms that make inferences from data automatically and with minimum human input [8]. Instead of the traditional research approach in which a hypothesis is pre-specified and tested, in machine learning, the data is given to the computer and the algorithm makes inferences about the relationships between exposures and outcomes without human guidance. Although these modeling techniques are novel, they are still limited to discrete data predictors.

To obtain access to more data and predictors, we can utilize natural language processing, which is a specific form of artificial intelligence used to extract structured information from unstructured text notes. It is plausible that full text documents, such as preoperative CT scan reports, could contain information that could be associated with either postoperative complication or postoperative hospital readmission, as the location of the disease may be associated with the radicality of the operation. The objective of this study was to determine if natural language processing with machine learning of unstructured full text documents (a preoperative CT scan) improves the ability to predict postoperative complication and hospital readmission among women with ovarian cancer undergoing surgery when compared with discrete data predictors alone. Additionally, we sought to identify novel predictors of postoperative complication and hospital readmission from these full text documents.

METHODS

Cohort:

Using the Northwestern Medicine Enterprise Data Warehouse (NM-EDW), a database that collects and integrates data from the EHR at Northwestern Medicine Healthcare (NMHC) system practice settings, we identified ovarian cancer patients hospitalized at Northwestern Memorial Hospital or Lake Forest Hospital between January 1, 2011 and December 31, 2017. We defined ovarian cancer by ICD-9 codes 83.xx. We included women who were age greater than 18 years-old and underwent surgery during their hospitalization. We excluded patients who underwent surgery for ovarian cancer recurrence. The IRB at Northwestern University approved this study (STU00206617).

Data Extraction:

We obtained discrete structured variables and unstructured free-form text-based clinical notes from the EHR (Cerner, Kansas City, MO) pertaining to the ovarian cancer hospitalization for all patients meeting study criteria from the EDW.

For discrete variables, we recorded demographics (age, race, ethnicity, insurance type), medical history and comorbidities based on ICD-9/10 codes. Additionally, we recorded laboratory values in the 60 days prior to surgery. This included values for CA125, white blood cell count, hemoglobin, albumin, creatinine, and platelet count. In the event that there were multiple values for a given patient, the mean of the values was calculated and used as a single predictor. Comorbidities were obtained by extracting ICD-9 codes linked to encounters in the 60 days prior to surgery including the surgery encounter. For non-discrete variables (e.g., text), a data analyst extracted the notes from the EDW. We included preoperative CT scan reports which had to include the abdomen and pelvis and be performed within 30 days prior to surgery. We combined the raw text data with the discrete data, linking by a common identifier.

Feature Selection:

We built different feature sets, or groups of predictor variables, for our predictive models. First, we compiled discrete data which were used as the first feature set. Variables chosen as discrete data predictors were selected based on previous associations with postoperative complication in this population (Table 1). These variables were then abstracted from the EDW.

Table 1:

Demographic Characteristics and Preoperative Laboratory Values

Characteristic	Values (n=291)
Mean age in years (SD)	59 (12.8)
Race, n (%)
White	208 (71.5)
Black or African American	39 (13.4)
Hispanic or Latino	4 (1.4)
Asian	15 (5.2)
Unknown/not recorded	25 (8.6)
Insurance status
Private	175 (60.1)
Medicare	91 (31.3)
Medicaid	17 (5.8)
Not available	8 (2.7)
Platelet count	306 (108)
Albumin	3.9 (0.47)
White blood cell count	7.2 (2.8)
Creatinine	0.81 (0.33)
CA125	610 (1723)
Hemoglobin	12.0 (1.7)

All laboratory values were obtained within 60 days prior to surgery.

Data is presented as n(%) or mean (SD) as appropriate.

Next, we constructed three different types of natural language processing (NLP) features from the unstructured clinical notes. To do that, we first pre-processed the notes to remove language abnormalities and make it usable for feature extraction. Specifically, we lowercased the text, removed punctuations, stop words (e.g. the, at, a) and non-alphanumeric words. We aggregated all the reports for each patient and then aggregated reports from all the patients to create a large corpus. We then created a token dictionary of all of the unique important terms from the corpus. We experimented with unigrams, bigrams, trigrams and noun phrases; however, we found the combination of unigrams and bigrams to work best.

For our first set of NLP features, using the token dictionary, we transformed the corpus to a patient-token matrix in which each token (unigram or bigram) is represented by term-frequency-inverse document frequency (tf-idf). Next, we used logistic regression with “l1” penalty (LASSO) to reduce the large dimensionality of features [9]. The LASSO method puts a constraint on the sum of the parameter coefficient and applies shrinking (regularization) to penalize the coefficient of non-essential features to zero. We filtered all the non-zero coefficient features and used them as our second set of features.

For second set of features, on the patient-token matrix, we applied principal component analysis (PCA) [10] and constructed a graph of the variance by cumulative number of principal components. This graph provided us with the most effective number of principal components that explained the most variance in the data set. We then selected these principal components to form our third set of features.

For the final set of features, we ran word2vec [11] on the text corpus to learn word vectors for each token in our dictionary. We used genism [12] package and continuous bag of words approach with standard parameters for running word2vec algorithm. Next, to construct a patient vector we summed all the individual token vectors for each token present in each patient’s report. Doing this, each patient is then represented by a single vector, which formed our fourth and final set of features.

Definition of Outcomes:

We built models for two outcomes: major postoperative complication and hospital readmission. Major complication was defined as a grade 3 or higher complication on the Clavien Dindo scale that occurred within 30 days postoperatively [13]. Readmission was defined as any unplanned inpatient hospitalization in the 30 days after surgery. We excluded planned or scheduled inpatient encounters, emergency department visits without admission, and observation admissions. Women who died within 30 days were included for the postoperative complication analysis but excluded for the hospital readmission analysis as they would not be eligible to be readmitted.

Predictive models:

We developed models to predict two outcomes: 30-day major complications and 30-day unplanned hospital readmission. For each of these outcomes, we trained different predictive models and compared them with each other. In addition, we also used different types of features for each of the predictive models as discussed above. Thus, our study not only evaluates the performance of different predictive algorithms, but also the added value of different types of features. We trained a number of different base predictive models as well as several hierarchical predictive models to enhance predictive performance. The base models included logistic regression [14], random forests [15], support vector machines [16], gradient boosting machines [17], and finally extreme gradient boosting (XGBoost) [18]. We trained each of these models for each of the feature types and compared the performance across multiple models.

Validation and Evaluation:

To avoid over-fitting, we performed 5-fold cross-validation [19]. Cross-validation, also called rotation estimation, is a technique to evaluate predictive models by partitioning the original sample into a training set to train the model and a validation set to evaluate it. In k-fold cross-validation, the original sample is randomly partitioned into k equal size subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k-1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds can then be averaged (or otherwise combined) to produce a single estimation. We also performed hyper-parameter tuning for our base model within each fold using “hyperopt” python package [20]. For each constructed model we used the area under the receiver operator characteristic curve with a corresponding 95% confidence interval to measure the discrimination of the model, where 0.5 is no ability to discriminate between observations with the outcome and without and 1.0 is perfect discrimination.

RESULTS

We identified 291 patients with ovarian cancer who underwent primary surgical treatment from 2011–2017 who also had preoperative CT scan reports available in our system in the 30 days prior to surgery (Table 1). The mean age of the cohort was 59 years. The majority of the cohort was white (208, 71.5%) and had private insurance (175, 60.1%). The mean preoperative CA125 value was 610 U/ml and the mean albumin was 3.9 g/dL. In terms of co-morbidities, 28% of patients carried a diagnosis of hypertension, 14% had a nutritional or metabolic disorder and 12% had a cardiac dysrhythmia (Table 2). There were 25 patients (8.6%) who were readmitted within 30 days and 45 patients (15.5%) who developed major postoperative complications within 30 days.

Table 2:

Comorbid Conditions

Comorbid condition (ICD-9 code)	N=291
Diabetes mellitus without complication	25 (8.6)
Diabetes mellitus with complications	5 (1.7)
Fluid and electrolyte disorders	50 (17.2)
Other nutritional; endocrine; and metabolic disorders	42 (14.4)
Essential hypertension	81 (27.8)
Hypertension with complications and secondary hypertension	5 (1.7)
Coronary atherosclerosis and other heart disease	5 (1.7)
Pulmonary heart disease	15 (5.2)
Cardiac dysrhythmias	34 (11.7)
Congestive heart failure; nonhypertensive	2 (0.7)
Phlebitis; thrombophlebitis and thromboembolism	21 (7.2)
Chronic obstructive pulmonary disease and bronchiectasis	9 (3.1)
Asthma	14 (4.8)
Intestinal obstruction without hernia	40 (13.7)
Acute and unspecified renal failure	10 (3.4)
Chronic kidney disease	5 (1.7)
Anxiety disorders	32 (11.0)
Mood disorders	29 (10.0)

All comorbid conditions were recorded as ICD-9 codes linked to an encounter in the 60 days prior to surgery, including the surgical encounter.

For 30-day readmission, our baseline logistic regression model using discrete features (age, laboratory values, CA125, comorbidities) obtained an AUC of 0.56 (0.54 – 0.58, 95% CI). Using NLP-based embedding features obtained from preoperative CT scan reports, we obtained a significant increase in discrimination with an AUC of 0.69 (0.66–0.72, 95% CI). When we combined the discrete data predictors with the NLP of the CT scan reports, we obtained a marginal increase with AUC of 0.70 (0.68–0.73, 95% CI). Overall, there was a 25% improvement in discrimination with the addition of NLP to traditionally used discrete data predictors (p<0.01).

We obtained similar results for prediction of 30-day major postoperative complication. The baseline model of logistic regression with discrete features as predictors (age, laboratory values, CA125, comorbidities) obtained an AUC of 0.57 (0.55–0.59, 95% CI). Using NLP embedded features within preoperative CT scan reports, we obtained an 18% improvement with an AUC of 0.67 (0.65–0.7, 95% CI). When we combined the NLP features with discrete features, our best model obtained an AUC of 0.68 (0.66–0.7).

On further inspection of the NLP models, we found words and phrases that were predictive of readmission and postoperative complication. These included phrases related to the location of ovarian cancer such as “liver lesion”, “left adnexal mass”, “omental”, “lymphadenopathy”, “bilateral ovarian lesions”, “hydronephrosis”, and “ascites”. Additional predictive words or phrases included those that described the appearance of the cancer. These included things such as “stranding”, “exophytic”, “ill-defined soft”, “bulky”, “high density” and “contiguous”. A final group of predictors included words such as “fracture” and “compression fracture” that may be more related to the underlying patient health status. A more comprehensive set of features weighted by contribution to prediction is depicted in the form of a wordcloud (Figure 1).

Figure 1: Word Cloud of Words and Phrases Associated with Postoperative Complication.

This word cloud depicts words and phrases associated with postoperative complication. Words which are more strongly associated are given a higher feature importance in the machine learning models and hence appear larger. The colors are merely artistic and do not indicate an association.

DISCUSSION

In this study, we evaluted the use of machine learning with natural language processing to attempt to improve upon current methods for prediction of postoperative complication and readmission among women with ovarian cancer. We found that machine learning models, such as random forest or XGBoost, led to very minimal improvements in the discrimination of the models over traditionally-used logistic regression, and thus the ability to predict postoperative events. In contrast, the addition of natural language processing of preoperative CT scans, which allows for the conversion of full text data into discrete data that can then be placed into a predictive model, resulted in an approximately 20–25% improvement in the ability to predict both postoperative complications and postoperative readmissions.

Artificial intelligence, which refers to the use of computer systems to perform tasks that normally require human intelligence, is finding increasing applications in medicine. Within artificial intellingence, we used two techniques: machine learning and natural language processing. Machine learning automates analytical model building and allows for the computer or “machine” to find insights or relationships hidden within the data rather than to rely on pre-specified hypotheses. Natural language processing allows for the computer to take a free text document and transform the words or phrases within that document into variables or vectors that can then be analyzed by an analytical model. These techniques been used to improve accurate diagnosis in mammography and to screen for diabetic retinopathy [21, 22]. They have also been used to enhance human decision making and improve prediction of events. This includes applications such as predicting survival from cancer and predicting complications among patients admitted to ICUs after cardiothoracic surgery [23].

Predicting complications and hospital readmissions after surgery for ovarian cancer is useful for many reasons. Patient counseling can be tailored and expectations can be set. Resources can be directed at those at highest risk through enhanced postoperative monitoring using both traditional and novel telehealth methods. Additionally, patients at highest risk of postoperative complications can be identified and their treatment plans adjusted accordingly. Multiple randomized trials have shown that neoadjuvant chemotherapy decreases the rates of postoperative complications when compared with primary debulking surgery [1, 3, 4]. Furthermore, postoperative complications can lead to delays in initiation of adjuvant chemotherapy [2]. For patients at the highest risk, beginning with neoadjuvant chemotherapy may be a reasonable strategy to try to mitigate the effects of postoperative complications on adjuvant treatment and survival.

Additional findings of interest from this study include the words and phrases that the machine learning algorithms found to be predictive of postoperative complication and hospital readmission. Some are words which clinically we already think of as describing the extent of disease present, such as “ascites”, “bulky lymphadenopathy” or “liver lesion” and may not come as a surprise. However, others are words which we do not tend to associate with high morbidity surgery or postoperative complication, such as “ill-defined”, “stranding”, “lobulated”, and “exophytic”. It is possible that these words, which refer to the appearance or composition of the tumor itself, indicate a more invasive tumor that requires more morbid surgery, or some other characteristic associated with complication being conveyed by these descriptive words. An additional group of words focused on the health of the patient, such as “compression fracture,” which is indicative of the frailty of the patient and their ability to withstand the surgical insult.

Strengths of this study include the use of novel methods which have not been applied in gynecologic oncology or ovarian cancer surgery to date. We also chose predictors that could be obtained without direct chart review to enhance the ability to scale this predictive model. Limitations include that this study was performed with medical records from two institutions; thus generalizability may be limited and the models may suffer from over-fitting. Specifically with natural language processing of CT scan reports there may be regional or local differences in words that are used which limits genealizability outside the two studied institutions. Additionally, although this study found significant improvements with the use of natural language processing of free text information, the models still have an overall discrimination that was fair, indicating that there is more work to be done to improve the use of these techniques before they can be used in this clinical setting. Examining predictors found in other places within the electronic medical record could be considered, such as within operative reports, pathology reports, or clinic notes. Additionally, using the CT images themselves, rather than reports, as predictors could also improve the discrimination of these models. However, despite these limitations and future directions, our results indicate that artifical intelligence, specifically machine learning methods using natural language processing, may improve the ability to predict postoperative complications and readmission after ovarian cancer surgery. Incorporating these methods will be important for future research in gynecologic oncology.

Highlights.

• Machine learning methods improved the ability to predict postoperative complication after ovarian cancer surgery.

• Natural language processing of preoperative CT scan reports was particularly useful in predicting complication.

• Words in CT reports such as ascites, ill-defined, and stranding, were predictive of complication.