Use of Topic Modeling for Recommending Relevant Education Material to Diabetic Patients

Abstract

The need for and challenges of educating and informing patients are well known and these are even greater for patients with low levels of literacy. Furthermore, as the population ages and with the increase in prevalence of chronic diseases where patient self-management is essential to holding disease in abeyance, patient education becomes increasingly important. With the advent of electronic medical records, there is an opportunity for automated tools to assist in addressing these challenges. In this paper we report on one approach to recommending relevant educational articles to patients. We attempt to infer the patient’s information needs from his/her electronic medical records and use topic modeling to identify and match topics. A manual evaluation of the articles recommended by the proposed method showed that these articles are significantly more relevant (p < 0.01) to the patient’s disease state than articles selected at random from within the same disease domain.

Introduction

Literature suggests that in chronic diseases such as diabetes, 90% of the care is self-management¹. Improvements in self-management can potentially lead to better outcomes, increased satisfaction, and reduction in costs^2,3. Research on patient education has recognized that it is ultimately the patients who make decisions, take actions, and face consequences⁴. As such patient information needs tend to be driven by and specific to their own disease contexts. Thus, the information that facilitates self-management should be patient and disease stage-specific, in addition to being understandable and actionable.

While there are a large number of educational articles designed for patients, clinical records contain the most detailed description of a patient’s condition and the most specific and current treatment plan for the patient. When educational information is provided based on the patient’s problems and instructions at a particular point of time, they become more relevant and can enable actions. For instance, detailed instructions on foot care could be handed out along with a referral to a podiatrist.

With the large number of health texts currently available online, it would be useful to categorize education material by topic, so that an appropriate text could be chosen for a patient with a particular disease at a particular stage and/or with particular complications. However, human indexing of content by topic is costly and labor-intensive. For example MEDLINE⁵, a large collection of bibliographic records of articles in the biomedical domain, includes about 18 million references which have been manually indexed. The costs for indexing amount to several million dollars per year and the MEDLINE database continues to grow at a fast rate, with the increasing availability of documents in electronic form⁶. In contrast, most patient education material is not rigorously indexed. As such, automated methods to detect topics in texts are necessary.

In this paper, we describe a method for matching patient educational material to patient’s clinic notes through the use of topic modeling. This is part of a larger effort to create computer-generated comprehensible, relevant and actionable information for patients. This work is intended to support patients in being more effective in self-management tasks, which are particularly relevant for chronic diseases, including diabetes. In particular, we are planning to produce an after visit summary (AVS) that physicians can provide to patients to summarize their action plan. As part of the AVS, we hope to recommend a small number of educational articles that are appropriate for the patient and fulfill his/her information needs.

Background

A topic can be defined as a collection of words that occur together frequently and are related to a common subject. Topic modeling provides a method for learning, in an unsupervised way, the topics contained in a large volume of text. Using surrounding text, topic modeling can identify words with similar meanings and distinguish between uses of words with multiple meanings. For a general introduction to topic modeling, see for example Probabilistic Topic Models by Steyvers and Griffiths⁷.

MALLET (Machine Learning for Language Toolkit)⁸ is a JAVA based open-source toolkit that implements methods for statistical natural language processing, document classification, cluster analysis, information extraction, and other machine learning applications, including text topic modeling. In the current study, to model topics, we have used the efficient, sampling-based implementations of Latent Dirichlet Allocation (LDA)^9,10 available in MALLET. LDA is a generative model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word’s existence is attributable to one of the document’s topics.

Topic modeling has been used in several applications. Steyvers and Griffiths¹¹, used this approach to analyze abstracts from the Proceedings of the National Academy of Sciences by using Bayesian model selection to establish the number of topics. They showed that the extracted topics captured meaningful structure in the data, consistent with the class designations provided by the authors of the articles. It has also been used in a seminar recommendation system¹², where topic modeling was used to generate the topic compositions of a corpus of about 5000 seminars announcements.

Similarly, Term Frequency–Inverse Document Frequency (TF-IDF)^13,14, a statistical measure used in information retrieval and text mining to evaluate the importance of a word to a document in a corpus, has been used for classification tasks. The importance increases with the frequency of the word in the specific document but decreases with the frequency of the word in the whole corpus. It has been used, for instance, in assigning papers to reviewers¹⁵.

In the health domain, the PERSIVAL¹⁶ system was designed to accept queries from both medical personnel and lay people and return personalized results from large medical digital libraries that contain text and audiovisual content. This system includes a summarization component, Centrifuser¹⁷ that summarizes documents that match the user query. The Centrifuser module uses the patient’s health records to define the context and discovers topics in a collection of consumer documents by building a ‘document tree’ which is a combination of the ‘headers’ in the document and selected content. One advantage of this approach is that parts of a document can be identified as relevant to a particular query. In the process of discovering topics, the module also attempts to separate ‘main’ topics from low-level ‘intricate’ topics. A small evaluation study (13 subjects) of the system showed that these summarizations enabled lay people to quickly identify high-level information while directing further knowledge discovery with indicative differences and navigation links to related queries and representative documents.

For our study we chose to omit the patient query and estimate the patient’s needs based on the patient’s health record because this record is authoritative for the state of the patient’s health and relevant to the actions recommended to the patient. By further coupling our recommendations with an after visit summary or discharge instructions, caregivers can review our recommendations for relevancy.

Material and Methods

Patient Education Material (PEM)

198 patient education documents related to diabetes, its treatment, symptoms and complications, available though public access websites of MedlinePlus, MayoClinic, National Institute of Diabetes and Digestive and Kidney Diseases, American Diabetes Association^a, and news items published in the New York Times, TIME, Guardian etc.^b were collected.

Using the MALLET toolkit, these documents were processed to extract a set of topics describing the documents. The result was that each document was described as a mixture of topics. The input to the toolkit is the number of topics the user expects the documents to cover. We varied the number of topics from 10 to 400 and a set of 150 topics was used in the current study as this seemed to provide reasonable results.

EMR Notes

A corpus of 70000 free-text electronic medical records (EMR) from the enterprise data warehouse (EDW) at the University of Utah Health Sciences Center was extracted¹⁸. The EDW is an aggregate of clinical data generated from inpatient and outpatient settings, patient demographic, visit and scheduling information and billing information. The extracted EMRs included several types of notes including clinic notes (inpatient and outpatient), discharge summaries, discharge instructions, surgery notes, etc.

The top twenty terms for each topic defined by the MALLET model were extracted resulting in 1229 unique terms. With the help of a regular expression module in the Health Information Extraction Tool (HITEx)^19,20, we looked for these terms in the notes and all occurrences of these terms along with their negation status²¹ were extracted. This information is used to calculate the number of occurrences of terms in each note.

Method

From the topic modeling done on the PEM corpus, the following values are retrieved:

• A weight for each [term, topic] pair that denotes the relative importance of the term to the topic. Let W_tei,toj denote the normalized weight of term te_i in topic to_j.

• A weight for each [topic, PEM document] pair that denotes the relative coverage of the topic in the document. Let P_toj,dk denote the coverage of topic to_j in document d_k

Given an EMR note n for which we want to find appropriate patient education material, we compute:

• $F_{{\mathit{te}}_i,n}=\frac{C_{{\mathit{te}}_i,n}}{{\sum }_{\mathit{te}\in \hspace{0.17em}n\hspace{0.17em}}{C}_{\mathit{te},n}}*\mathit{log}\left(\frac{\left|N\right|}{\left|\left\{n|{\mathit{te}}_i\in n\right\}\right|}\right)$, the TF-IDF frequency of term te_i in the note n. C_tei,n is the number of occurrences of te_i in n, |N| the cardinality of the set of notes N, and |{n|te_i ∈ n}| is the number of notes in N that contain te_i.

• $F_{{\mathit{to}}_j,n}={\sum }_{{\mathit{te}}_i}\left(F_{{\mathit{te}}_i,n}*\frac{W_{{\mathit{te}}_i,{\mathit{to}}_j}}{{\sum }_{{\mathit{to}}_j}{W}_{{\mathit{te}}_i,{\mathit{to}}_j}}\right)$, the topic frequency of each topic to_j in the note n. Here, W_tei,toj is a measure of the relative importance of te_i in to_j. We believe that terms that occur in a lot of topics are less important than terms that occur in a few topics; hence, W_tei,toj is weighed down by the sum of weights of te_i in all topics. A high F_toj,n indicates that topic to_j is well represented in the given note.

• $M_{d_k,n}={\sum }_{{\mathit{to}}_j}\left(F_{{\mathit{to}}_j,n}*\frac{P_{{\mathit{to}}_j,d_k}}{{\sum }_{d_k}{P}_{{\mathit{to}}_{j,}{d}_k}}\right)$, the relative match of document d_k to the note n. Similar to the computation of F_toj,n, the relative importance of the topic to_j to document d_k (P_toj,dk), is weighed down with the sum of weights of to_j in all PEM documents. A higher value of M_dk,n indicates a better match.

The PEM documents are ordered by their M_dk,n and the top two documents are recommended as appropriate education material for the given note.

Validation

A registered nurse (RN) was asked to review a small sample of 50 EMR notes, along with their recommendations. For each note, the top two recommended PEMs and a random PEM (as control) were selected for review. The second-best recommendation was included in the review to help understand the relative usefulness of the recommendations. The relative order of the recommended PEMs and the identity of the control were withheld from the reviewer. Each of the three PEMs was assigned a rating on a Likert²² scale of 1–5, with 1 indicating a recommendation of low relevance and 5 indicating a highly relevant recommendation. Since all documents are in the domain of diabetes and all patients are diabetic, it can be argued that all documents are somewhat relevant to all of the patients. However, the focus is on the document’s relevance to a patient’s specific conditions and treatments.

The average rating of the two recommended PEMs and the control PEMs is computed and reported. It is assumed that statistically significant higher ratings for the recommended PEMs compared to those of the control PEM, would indicate that the proposed method is effective in identifying articles relevant to the patient.

Results

A topic model was built using MALLET to detect 150 topics and Table 1 lists a small set of topics with the terms in each topic ordered by their relative weights. It can be seen that the assignment of weights succeeds in grouping terms that are related to the same/similar concept(s). For example, terms related to acute renal failure (nephropathy) which could be one of the complications in diabetic patients, have been grouped into a single topic, even though the model cannot infer what the topic pertains to.

Table 1.

Sample topics modeled by MALLET. Only the top 20 most prominent terms in each topic are shown. The column labeled ‘Topic’ is not an output of MALLET but a guess by the authors of the topic being modeled.

ID	Terms	Topic
4	blood heart disease risk pressure high people stroke prevent damage vessels smoking healthy cholesterol problems addition increase vessel smoke	Complications
9	kidney disease urine failure protein kidneys microalbuminuria dialysis function diabetic waste damage filtering nephropathy diet body filter creatinine drugs	Renal failure
15	pregnancy baby women gestational pregnant risk birth health normal babies defects delivery good pregnancies weeks pounds care healthy hormones	Pregnancy
24	eat foods healthy food eating weight diet make fruits choose family vegetables plan calories amount grains include lose daily	Diet
45	diabetes type risk american gestational developing weight factors impaired overweight increased higher tolerance condition african asian program family lower	Risks
139	feet foot shoes skin problems diabetic toes sores legs feeling wear infection dry blood socks check make cuts sore	Foot care

An analysis of the topic distribution among PEM documents show that 30% of the topics are covered in two or fewer documents. A very small percentage of topics (1.5%) are covered in twenty or more documents and are topics in which terms such as ‘insulin’, ‘glucose’, ‘blood’ – terms that occur very often in diabetes education material - have high weights. Appendix 1^c is a sub-graph that shows the mapping between topics (represented by circles) and documents (represented by squares). An edge between a topic and a document shows that the topic is represented in the document and the color/thickness of the edge denotes the strength of this representation. To make the graph manageable only the top 10% of the edges have been shown. As can be seen in the figure, a few topics (T63, T94, T132) are represented in a large number of documents while most of the topics have less prominent representation.

Validation

Figure 1 shows a histogram of the average F_to,n for all the topics in the 50 notes used for validation. The distribution suggests that a small percentage of topics (topics with high average F_to,n) are discussed in a significant number of notes and some of the topics are irrelevant (low average F_to,n) to this set of notes. The most commonly mapped topic is related to kidney failure {kidney failure kidneys chronic acute blood substances bladder end risk injury severe obstruction renal waste vascular bloodstream infection decreased} and the least commonly mapped topic is related to an apparent ambiguous topic {ndi gene inherited water result aqp acquired mutations genetic males mutated females forms levels lithium receptor chromosome identified genes}.

Figure 1.

Histogram of the average F_to,n of the topics in the validation note set.

To assess how well the proposed method can differentiate between the PEM documents and to test whether it chooses different PEM documents for different EMRs, we computed the median rank for each document. A PEM that has been relevant to a good number of notes would have a low median rank. Figure 2 shows a histogram of the median ranks of the PEM documents for the reviewed notes. For example, 3% of the PEM documents had a median rank less than 20 i.e. these documents are likely to be in the top 20 of the most suitable recommendations for a majority of the notes. Similarly about 5% of the documents had very high median rank i.e. they were very rarely recommended as their topics were being better represented in other PEMs. This suggests that some of the documents in the PEM corpus can be removed without affecting the recommendations. An article related to acute kidney failure^d was most often the highest scoring recommendation (highest M_d,n) and a document about the prevalence of diabetes and pre-diabetes^e was very often the least scoring document (lowest median rank). This is probably because terms/concepts related to prevalence are rarely encountered in the clinic notes and hence the topics represented in this document are rarely found to be important. As many notes did not include a comprehensive medication list, documents that contain drug information were also found to have low M_d,n.

Figure 2.

Histogram of the median rank of the PEM documents in the validation note set.

An analysis of the reviewer’s rating for the recommended PEMs of the 50 notes showed that the average rating for the top recommended PEM was 2.62 (std. dev = 1.46) and that of the second best recommendation was 2.38(std. dev = 1.35) whereas the average rating for the control PEM was lower at 1.78(std. dev = 0.99). The average rating of the best of the two recommendations was found to be 2.98(std. dev = 1.44).

Figure 3 shows the distribution of the reviewer assigned rating for the three recommendations. Half of the recommendations from the control group were assigned a rating of 1 whereas only 1 document was rated 5. Although some of the recommendations from the algorithm get good ratings, a significant number were rated 1. In 6 of the notes, the control PEM had a better rating than both the recommended PEMs. A paired t-test between the ratings of the top recommendation and the control showed the difference in means to be statistically significant (p < 0.001). Similar results were observed between the second best PEM and control (p < 0.01).

Figure 3.

Likert rating assigned by a RN as an estimate of the suitability of a recommended PEM to a note.

Examples of very relevant recommendations include: a PEM about preventing/taking care of foot ulcers when the patient’s assessment/plan section detailed management of the foot ulcer in consultation with a podiatrist; a PEM about renal failure and dialysis for a patient whose assessment/plan section stated “will consult Renal in the a.m. and will arrange for a dialysis catheter […], will make the patient n.p.o. at midnight and check coagulations and then initiate dialysis per Renal”.

Discussion

We have described a method that can help select education material relevant to a given patient. A limited evaluation of the recommendations resulting from the application of this method showed that the identified education content is more relevant to a patient’s inferred information needs than a random diabetes-related PEM. With an average rating of about 2.6 on a 1–5 scale (from least to most relevant), the approach of providing individualized recommendations based on topic modeling can be considered to be marginally successful.

Keyword or full text matches are alternatives to the method described here. However, most clinical notes and education documents are not indexed with keywords and in measuring full text overlap, functional words can overshadow the few important topic words. Since our method maps educational material to patient’s notes based on topics, we believe it can address some of the deficiencies of approaches that use narrower term-level matching. The method can be useful in providing supplementary information to after visit notes and/or discharge summaries and hence, can potentially reduce the time and effort required of clinicians, nurses and patient educators for manually selecting relevant education material. Additionally, the method eliminates the need for the user to formulate need-based queries.

However, the overall relevance of the recommendations suggests that there is room for improvement. In only 13 instances, the recommendations received a relevance rating of 5. We hypothesize that this is probably due to the limited size of the PEM corpus and the method can benefit from a larger, more comprehensive corpus. It is also important to note that the review criteria used to manually assess the relevance of the PEMs is not free of ambiguity. For instance, it is hard to assess the relevance of any content to a diabetic patient with Alzheimer’s or manic depression or for a non-English speaker. A more robust evaluation with multiple reviewers, using a consensus built review criteria, could be beneficial in identifying the deficiencies of the recommendation system.

Additionally, while there is a large body of research indicating the beneficial effects of personalized recommendations related to health conditions^{23,24,25,26,27}, not all applications that provide personalized health information have produced positive impact on patients^28,29. An evaluation scenario that includes patients is necessary to not only judge the relevance of the recommended PEM to the patients’ information needs but also to analyze the overall effect on his/her information seeking and decision making.

Our analysis of the results also suggests that it is important to consider temporal information when inferring information needs. A patient who has had his/her feet amputated in the past does not need to learn about foot care, nor about amputation as a procedure. Our method is not intelligent enough to differentiate between a procedure performed in the past versus one planned for the future. The use of the section header information in the clinical notes (history section vis-a-vis assessment/plan section) can improve the inference of patient needs.

An interesting observation we have come across in our analysis of the topic-PEM mapping (Appendix 1) is that when only the top 10% of the high weighted edges are considered, the resulting graph is not a connected graph. For example, the sub-graph in Figure 3 is not connected (no shared edges) to the main component shown in Appendix 1.

This indicates that some topics are represented, to a significant extent, in only a small subset of documents which in turn cover very specific topics. Although the number and size of these unconnected sub-graphs can change with the inclusion of more edges (for instance, the graph would be connected when the top 50% of the edges are used), the documents thus identified to be topic-specific could be recommended directly to patients interested in those topics, without the use of formula-based mapping.

The topic modeling with MALLET was performed with the number of required topics preset to 150. We are not aware of an established method to identify the optimum number of topics for a given corpus size. We however examined the results of the recommendation system with fewer topics and the resulting recommendations were found to be not as good as those obtained with 150 topics. Although, the evaluation with the 50 notes identified some topics that can possibly be removed from consideration, a more systematic way to determine an optimum number of topics needs to be explored. Additionally, although we have used a stop word list available is MALLET to remove functional words before modeling topics, this list does not include stop words specific to the health domain (for example: doctor, patient, clinic) that should also have been removed. Use of a more extensive stop word list and stemming³⁰ of the words could result in a more well-defined set of topics.

A related factor to be considered in determining appropriate education material is the mismatch between the literacy level of the patient and the readability of the educational material. For example, for a patient who reads at the sixth-grade level the relevance of the recommended material is immaterial if s/he finds the material hard to understand. A content recommendation system should take this into consideration but the method described here does not. However, we have been working on tools to automatically assess the readability of health content and tools to make health content more readable through the application of a few semantic and syntactic simplification methods. We hope that some of the methods developed for these tools could be incorporated into the recommendation system to address the literacy-readability disparity issue.

In our future research, we intend to refine the existing method with more in-depth processing of the clinical notes and with a larger education document collection. We would also like to test the method with a larger set of clinical notes.

Figure 4.

An example sub-graph with topic-specific documents

Appendix 1.

A graph representing the mapping between the topics identified by MALLET and the PEM documents in the corpus.