Unsupervised Large Language Models to Identify Topics in Cancer Center Patient Portal Messages

Abstract

Background:

The increasing use of patient portal messages has enhanced patient-provider communication. However, the high volume of these messages has also contributed to physician burnout.

Methods:

Patient-generated portal messages sent to a single cancer center from 2011 to 2023 were extracted. BERTopic, a natural language processing topic modeling technique based on large language model, was optimized. For further categorization, the topic words were labeled using GPT-4, followed by review by two oncologists. Uniform Manifold Approximation and Projection was employed for dimensionality reduction and visualizing topics. Message volume changes over time were assessed using a Student’s t-test.

Results:

A total of 2,280,851 messages were analyzed. The monthly average number of messages increased from 2,071 in 2012 to 43,430 in 2022 (p<0.001). There was a significant rise in message volume following the COVID-19 pandemic, with a posterior probability of a causal effect of 96.4% (p=0.04). Scheduling-related messages were the most frequent across departments, while symptoms and health concerns were second or third most common topics. In medical oncology and surgical oncology, topics on prescriptions and medications were more common compared to radiation oncology and gynecological oncology. Despite concurrent institutional changes in self-scheduling systems, scheduling-related messages did not decrease over time.

Conclusions:

The substantial increase in patient portal messages, particularly scheduling-related inquiries, underscores the need for streamlined communication to reduce the burden on healthcare providers. These findings highlight the need for strategies to manage message volume and mitigate physician burnout, laying groundwork for artificial intelligence-driven future triage systems to improve message management and patient care.

Introduction

The use of patient portal messages has been rapidly increasing.¹ Patient portal messages enhance patient-provider communication², relationships with patients³ and enable patients to remotely report significant symptom changes or adverse events so that medical providers can take measures promptly. The impact is most significant for patients with limited physical access to healthcare facilities due to their condition or distance from the hospital.⁴ It can also help facilitate a multidisciplinary approach and prevent treatment discontinuation by allowing various specialists to access patient-reported information, enabling timely responses to patient needs and reducing the risk of interrupted treatment.⁵ The COVID-19 pandemic and the accompanying increase in remote clinical care further accentuated its importance, significantly accelerating growth.⁶

However, perceived patient expectations to respond to the patient portal message have created new stressors³, and the high volume of patient messages are associated with clinician burnout.⁷ This not only adversely affects the well-being of physicians themselves but also has negative consequences for patients. Physician burnout is well-documented to be associated with medical errors^8–10 and lower patient satisfaction.¹¹

To achieve a balance and optimize the use of patient portal message, it is imperative to identify message subject matter and devise a strategy for the most efficient management. Topic modeling is a powerful natural language processing (NLP) method for text mining, clustering, and classification. It may uncover topics by automating analysis in the tremendous expanse of patient portal messages and reveal hidden themes by identifying patterns in word choice and linking documents with similar patterns.¹² In particular, large language models (LLMs) such as Bidirectional Encoder Representations from Transformers (BERT) and its derivatives, like sentence transformer, can be applied to enhance these analyses.¹³

In the current study, we aimed to identify the most frequent topics from patient portal messages to cancer center medical providers with NLP-based topic modeling, with the goal of providing foundational data to inform the development of automated message triage systems.

Methods

Data collection

This retrospective study adhered to the principles of the Declaration of Helsinki and was approved by the institutional review board (IRB) of the University of California San Francisco (UCSF) (IRB number: 19–29686). The need for informed consent was waived due to the retrospective nature of the study, as approved by the ethics board. Patient portal messages generated from October 2011 to April 2023 in UCSF Cancer Center were retrieved. Basic information such as created date, time, recipient identifier, and department were collected. This study adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines to ensure comprehensive and transparent reporting of the research methodology and findings.

Topic modeling

Messages were first pre-processed, including removing message headers, quoted bounce-back messages from delivery failure, and stopwords. Stopwords such as “thank you”, “thanks”, “thank”, “thankyou”, “perfect”, “great”, “dr”, “hi”, “ok”, “sounds good”, “okay”, “yes”, “thx” and frequently mentioned names of medical providers were removed in addition to the frozenset “English” in Countvectorizer. To create embeddings, sentence transformer all-MiniLM-L6-v2 was used.¹⁴

For department-wise analysis, a total of 40 departments in the cancer center were condensed into 6 categories: medical oncology, surgical oncology, supportive care, gynecology oncology, radiation oncology and others. Detailed classification information can be found in Supplemental Table 1. Topic modeling was performed in the medical oncology, surgical oncology, gynecology oncology, and radiation oncology categories because ‘supportive’ or ‘others’ could include a vast variety of services (e.g., psycho-oncology, nutrition, exercise) and may not directly pertain to a specific oncologic treatment, warranting a separate, dedicated analysis. Considering the characteristics and diversity of messages from each department, topic modeling was conducted on a department-by-department basis.

BERTopic was used for topic modeling. BERTopic is a variation of BERT, which is a language representation model designed to pre-train for fine-tuning.¹⁵ BERTopic leverages transformers and c-TF-IDF for clustering and interpretable topic modeling.¹⁶

Topic number

The determination of the final number of topics was a multi-step process combining quantitative metrics with qualitative assessment. We first evaluated models with various topic counts (e.g., 50, 100, 150, 200) using topic coherence (NPMI) and diversity scores (OCTIS v1.13.1).¹⁷ As these metrics often present a trade-off, we then performed a qualitative review of the resulting topics at each threshold. This involved examining representative documents and topic words to identify the point at which increasing the topic count began to produce overly granular or redundant themes (e.g., splitting a single “appointment request” topic into multiple, nearly identical sub-topics). The determined topics were distilled to their four most representative topic words and 30 representative documents. To label the categories, the topic words were anonymized and initially labeled using GPT-4. The grouping process was further reviewed by two board-certified oncologists.

Message volume analysis

The causal impact of COVID-19 on the increase in message volume was analyzed with a Bayesian structural model (tfcausalimpact v0.0.13).¹⁸ Changes in message volume over time were evaluated using a Student’s t-test to determine statistical significance.

Results

From October 2011 to April 2023, 23,977,703 message lines across 2,280,851 messages were received at our cancer center. The monthly average number of messages in 2012 was 2,071, surging to 43,430 messages per month in 2022 (p<0.001) (Supplemental Figure 1-A). An analysis by department revealed significant variation in message volume, as visualized in Supplemental Figure 1-B. Medical oncology and surgical oncology consistently accounted for the vast majority of messages. While all departments experienced substantial growth over the study period, these two departments remained the dominant recipients. Each physician received a median of 7.93 messages per month (interquantile range, 2.39–29.17). In the causal impact analysis, a notable increase in the rate of growth was observed following the issuance of a stay-at-home order by California after the COVID-19 outbreak on March 19, 2020. The analysis revealed a posterior probability of a causal effect of 96.4% (p=0.04).

After conducting a comparative analysis of NPMI, topic diversity scores, and multiple trial runs, the number of topics was determined as follows (Supplemental Data 1), factoring in the relative volume and diversity of messages in each specialty: 100 for medical oncology, 100 for surgical oncology, 50 for radiation oncology, and 50 for gynecology oncology. A bar chart representing topics and topic words in order of top frequency in the medical oncology category is depicted in Figure 1.

Figure 1.

Frequency Distribution of the Top 16 Topics and Topic Words in the Medical Oncology Category

These topics were further categorized after anonymizing the topic words using GPT-4 and reviewed by two oncologists (JHC and LS): 34 for medical oncology, 26 for surgical oncology, 19 for radiation oncology and 21 for gynecology oncology. The agreement between the GPT-4 categorization and the evaluators was as follows: 0.48 for medical oncology, 0.55 for surgical oncology, 0.7 for radiation oncology and 0.82 for gynecology oncology. In the messages sent to all department categories, scheduling-related topics were found to be the most frequent. The categorization of topics for the medical oncology and surgical oncology departments is presented in Table 1 and Table 2, while the categorization for the radiation oncology and gynecologic oncology departments is detailed in Supplemental Table 2. In medical oncology, communications primarily focus on appointments and scheduling (38.0%), prescriptions and medications (14.4%), and symptoms and health concerns (12.0%). Similarly, surgical oncology discussions revolve around appointments and scheduling (40.2%), symptoms and health concerns (12.9%), and prescriptions and medications (10.4%). In radiation oncology, the main topics include scheduling and test results (48.9%), gratitude and well wishes (11.6%), and symptoms and health concerns (11.5%). Lastly, gynecology oncology communications center on appointments and scheduling (29.7%), symptoms and health concerns (15.5%), and test orders and results (14.3%).

Table 1.

The topic lists of medical oncology department following topic modeling and categorization

Medical Oncology	Number of Messages	Percentage (%)
Appointments and Scheduling	468663	38.0
Prescriptions and Medications	177364	14.4
Symptoms and Health Concerns	147759	12.0
Gratitude and Well Wishes	130843	10.6
Forms and Documents	75550	6.1
Simple Communication	66446	5.4
File Attachment	26218	2.1
Contact information	20687	1.7
Location	17945	1.5
Test Orders and Results	17313	1.4
Medical Records and Results	18972	1.5
Contacting on behalf of someone	8653	0.7
Asking for clarification	4508	0.4
Outliers	4506	0.4
Communications about “numbers”	3874	0.3
Equipment and Supplies	3760	0.3
Primary care	3711	0.3
Supportive Care for Symptoms	3542	0.3
Appointment delays or Transportation	3499	0.3
COVID/Vaccine	3390	0.3
Test Preparation	3336	0.3
Weight	2824	0.2
Carcinogen and Prognosis	2605	0.2
Insurance and Payment	2569	0.2
Health and Lifestyle Management	2559	0.2
Log in issue	2494	0.2
System messages	1870	0.2
Drain	1806	0.1
Non-English	1578	0.1
Misclassified due to initial letter grouping	1490	0.1
Procedure	799	0.1
Dental	790	0.1
Video meeting	697	0.1
Cord Blood	550	0.0
Total	1233170	100.0

Table 2.

The topic lists of surgical oncology department following topic modeling and categorization

Surgical Oncology	Number of Messages	Percentage (%)
Appointments and Scheduling	291541	40.2
Symptoms and Health Concerns	93298	12.9
Prescriptions and Medications	75285	10.4
Gratitude and Well Wishes	68121	9.4
Forms and Documents	36892	5.1
Simple Communication	49431	6.8
Asking for clarification	18054	2.5
Location	16369	2.3
File Attachment	15108	2.1
Equipment and Supplies	9710	1.3
Contacting on behalf of someone	8529	1.2
Test Preparation	7157	1.0
Test Orders and Results	6926	1.0
Health and Lifestyle Management	6911	1.0
Nutrition	4371	0.6
Questions about contrasts	3501	0.5
Procedures	2906	0.4
Outlier	2257	0.3
Personal Information	1588	0.2
Non-English	1269	0.2
Insurance and Payment	1207	0.2
Medical Records and Results	1102	0.2
System messages	1081	0.1
Interpreter Request	850	0.1
Patient education	616	0.1
Log in Issue	427	0.1
Total	724507	100

Discussion

Effective communication is crucial for all medical conditions, but it holds particular significance for cancer patients as they undergo intense longitudinal treatments that require frequent communication with multiple departments regarding treatments, symptoms, appointments, and diagnostic results. Therefore, the advantages of patient portal messages may be further emphasized in cancer patients. However, at the same time, oncologists are at a higher risk of experiencing burnout due to their care of seriously ill patients, demanding workloads, and the growing reliance on electronic medical record system, resulting in a higher prevalence.^{19, 20} Cancer care involves not only decision-making about treatment but also the need for regular visits and monitoring of test results over an extended period of time. Additionally, this complex process includes the provision of emotional support. To alleviate the burden on healthcare providers and focus their efforts, it is necessary to examine how resources are being allocated. Therefore, in the current study, we conducted a topic analysis of patient portal messages with a specific focus on cancer care. The results revealed a significant number of messages regarding scheduling, highlighting a persistent inefficiency in communication that lays the groundwork for developing future automatic triage systems.

To contextualize these findings, our institution uses a hybrid messaging system where messages are triaged by care teams or sent directly to physicians. This dual system creates a significant burden from both complex escalated cases and high-volume unfiltered inquiries. The prevalence of administrative topics we found suggests an automated routing system could manage many of these messages, thereby alleviating the workload for the entire care team.

We utilized the widely employed topic modeling tool, BERTopic, which is currently prevalent in medical research as well. BERTopic stands as one of the outstanding topic modeling models alongside others such as Top2Vec, latent Dirichlet allocation (LDA), and non-negative matrix factorization (NMF). BERT, an LLM, serves as the basis for BERTopic’s embedding, enabling it to capture the contextual meanings of words more effectively, which is a key differentiator from other models. Additionally, the application of c-TF-IDF allows for precise topic clustering, further enhancing its strength and uniqueness compared to other models. This allows BERTopic to handle dynamic updates better and provide consistent topic coherence, making it particularly suitable for large-scale text datasets.^{16, 21} Many medical-related studies utilizing topic modeling, including BERTopic, have predominantly focused on data from social media platforms such as Twitter^22–24 and Reddit^{25, 26}, providing insights into what concerns and experiences patients actually share within the community. In contrast, this study presented a different perspective by exploring the topics and characteristics of patients’ inquiries directed towards medical providers using BERTopic.

Furthermore, our research findings demonstrated a significant increase in the number of messages, particularly following the outbreak of COVID-19. While messages offer convenience, the widespread accessibility of patient portal messages can lead to significant work burden and burnout among healthcare providers.²⁷ As one solution, some hospitals have implemented billing systems to reduce the absolute volume of messages. However, even after implementation, the billing rate remains low, with only 1.4% of all messages being billed at UCSF since November 2020.⁶ Peppercorn argued for the minimal financial impact on care sustainability and highlighted potential barriers to care if patients avoid messaging due to cost concerns.²⁸ They proposed better solutions, such as automating workflows to route specific inquiries to billed phone calls or virtual visits, utilizing templates for common questions, and specifying acceptable message topics. Our study investigates the proportion of message topics suitable for templates or automation within the same context.

Recent studies have been conducted on using LLMs to generate response drafts and Epic has already implemented the use of GPT-4 to draft communications with patients in the In Basket.²⁹ In one study, it was announced that this approach led to a reduction in task load and emotional exhaustion.³⁰ However, automation bias remains a significant concern. Chen et al. pointed out that while there were improvements in efficiency and interphysician agreement, there were also risks associated with LLM-generated responses, with some potentially leading to severe harm or even death, highlighting the need for continuous monitoring by physicians.³¹

Various attempts have been made to classify patient portal messages; however, the majority of them have employed manual analysis or supervised learning methods.^32–35 While establishing predefined categories and classifying messages can be an effective approach, considering the substantial message volume and the diverse mix of senders and providers, an unsupervised approach appears to be a more pragmatic choice from a research perspective, as opposed to a supervised method. The results demonstrated that across departments, scheduling messages were the most prevalent. The next most frequent topics were “Symptoms and Health Concerns” and “Prescriptions/Medications”. While the ranking differed slightly across departments, in medical oncology, surgical oncology, and radiation oncology, these three topics accounted for over 60% of all messages. While symptom-related inquiries align with the original purpose of patient portal messages – facilitating patient communication – scheduling and medication-related messages present opportunities for system optimization. By streamlining these processes, a significant portion of these inquiries could be resolved without human intervention, alleviating the increasing burden on healthcare providers. This optimization would not only reduce the sheer volume of messages but also streamline the delivery process, minimizing unnecessary steps and ultimately decreasing stress for medical professionals. UCSF’s scheduling practices have evolved asynchronously, prompting an investigation into whether this influences the temporal changes in topic frequency. Upon examining the topics over time, it was found that there was no difference in schedule-related topic frequency over time. This suggests there are still factors hindering easy access for patients, highlighting the need for further simplification.

Additionally, all departments received approximately 10% of messages expressing gratitude and well wishes, indicating a positive impact on building rapport between patients and medical providers. Positive feedback has been reported to reduce physician burnout.³⁶ Therefore, although patient portal messages may increase workload, they may also offer the advantage of providing positive experiences.³⁷

This study possesses several inherent limitations. Firstly, as a single-center study within an academic cancer center, the findings may not be generalizable to other healthcare settings with differing patient populations, workflows, and technology adoption, such as community hospitals or non-oncology specialties.³⁸ Secondly, despite utilizing BERTopic and quantitative metrics, the process of determining the number of topics and interpretation inherently involves subjectivity, potentially introducing bias. Finally, given that the analysis considered entire patient messages, instances of multiple topics within a single message were observed. Consequently, our annotation process likely prioritized the most salient concept, which can oversimplify multifaceted messages. This approach, however, was consistent with our primary objective of first identifying the most frequent, dominant topics to build a foundation for an effective routing system.

While acknowledging its limitations, the current study comprehensively analyzed over two million patient messages accumulated over a decade, offering a rich and unprecedented dataset of patient-provider communication within a large academic cancer center using NLP. It is also important to acknowledge that this is the first study to investigate the topics present in patient messages sent to a cancer center. The study focuses on identifying broad topical categories, a crucial step towards effective message routing and triage, supports the level of granularity achieved in outlining these categories as adequately informative and appropriate for this stage of investigation. Future work will focus on assessing the accuracy of topic labels for automatic triage to ensure that messages are routed efficiently and correctly, minimizing the workload on healthcare providers while maintaining high-quality patient care, as similar machine learning approaches have already been shown to reduce rates of acute care visits.³⁹

In conclusion, this study analyzed over two million patient portal messages from the UCSF Cancer Center, revealing significant insights into communication patterns between patients and healthcare providers. The findings highlighted a substantial increase in message volume, particularly during the COVID-19 pandemic, with scheduling-related inquiries being the most prevalent across departments. Despite advancements in self-scheduling systems, these messages persist, suggesting ongoing barriers to patient access. These insights underscore the need for continued efforts to streamline communication processes in order to reduce the burden on healthcare providers and improve patient access to care.

Context summary:

Key objective:

How can unsupervised topic modeling using large language model identify common themes in patient portal messages to guide strategies for improving communication efficiency and reducing clinician workload?

Knowledge generated:

The monthly message volume experienced a more than 20-fold increase between 2012 and 2022. Across all oncology specialties, communications regarding appointments and scheduling consistently represented the largest topic (over 30% of messages), followed by inquiries about symptoms and medications.

Relevance (written by Dr. Umit Topaloglu):

The study highlights that the massive increase in patient portal messages, especially for scheduling, is a major contributor to physician burnout. Therefore, developing efficient strategies, potentially including AI-driven triage systems, is clinically essential to manage this communication volume and improve provider well-being and patient care.

Data Availability Statement

The data supporting the findings of this study contain sensitive personal information and, as such, cannot be shared publicly or made available upon request. Due to the confidentiality and privacy concerns associated with the data, we are unable to provide access to the dataset used for this research. We have adhered to all relevant ethical guidelines and institutional policies to ensure the protection of participants’ privacy.