Using telecommunication dialogue and nursing documentation to predict the risk of emergency room visit in a web-based telehealth programme

Abstract

Aims

The effectiveness of telehealth care programmes in reducing mortality among patients with chronic conditions has been well established. Valuable insights into patients’ conditions can be gleaned through daily telecommunication between patients and nurse case managers. We hypothesized that using natural language processing can predict acute deterioration in patients with chronic conditions in telehealth care programme based on the nursing records and speech dialogues occurring during daily telecommunication.

Methods and results

We conducted a retrospective study utilizing audio recording transcripts from telecommunication sessions between patients and nurse case managers at our telehealth care centre, along with nursing notes as input data. Pre-trained transformer-based neural network models were constructed to predict emergency room (ER) visits within a 2-week timeframe. The case group included 94 patients with 585 speech recordings and nursing records, while the control group included 36 patients with 396 speech recordings and nursing records. Our results showed that employing transcripts and a bidirectional encoder representations from transformers (BERT)-base model with a sliding window for predicting ER visits yielded moderate accuracy 0.75 (interquartile range: 0.742, 0.773). The inclusion of long short-term memory in the model did not significantly enhance accuracy. Notably, combining nursing records and transcripts as inputs exhibited superior performance, achieving an overall accuracy of 0.892 (interquartile range: 0.891, 0.893) by the six models.

Conclusion

Our study demonstrates the feasibility of predicting ER visits using telehealth dialogue transcripts and nursing notes with pre-trained transformer models. The incorporation of nursing notes significantly enhances the model’s performance, providing a valuable method for improving predictive accuracy in telehealth care.

Graphical Abstract

Graphical Abstract.

Introduction

By using digital technology and communication tools, telehealth care can deliver medical service and consultation remotely.¹ The power of telehealth care has been demonstrated in the Coronavirus disease 2019 (COVID-19) era.² Synchronized telehealth care has been demonstrated to reduce all-cause mortality in the care of heart failure in the Telemedical Interventional Management in Heart Failure II (TIM-HF2) study.³ Our prior studies also revealed a reduction in mortality and a dominant cost effectiveness, as reported in TIM-HF2 study.⁴ One of the essential components of the synchronized telehealth care is the communication between patient and medical staff in telehealth centre.

The deterioration of chronic diseases like heart failure can be insidious, often manifesting gradually and subtly, even before an individual seeks medical attention through an emergency room (ER) visit. For example, in the Goal-directed Afterload Reduction in Acute Congestive Cardiac Decompensation (GALACTIC) study, which enrolled patients with acute decompensated heart failure, the median time from onset of dyspnoea to ER visit was 6 days.⁵ The gradual onset of deterioration presents us with a valuable opportunity for timely intervention. The personnel in telehealth centre must determine if an acute condition develops and if the patients have to go to ER based on the vital signs and the information derived from telecommunication. If an algorithm can recognize potential acute events via telecommunication and notify medical staff for prompt intervention, it may lessen the burden on medical personnel and enable early intervention.

By harnessing the capabilities of natural language processing (NLP), we can facilitate the extraction of secondary information from unstructured medical records. It has been shown that the unstructured nursing notes can be used to predict clinical outcomes in intensive care units.⁶ However, the raw speech dialogues between nursing managers and patients during telecommunication remain largely unexplored in predicting clinical events. Considering this gap in knowledge, we designed a pilot study to ascertain the feasibility of utilizing telehealth dialogues to construct an artificial intelligence model by using NLP to extract the information from daily speech. This model aims to predict whether a patient is at risk of experiencing an emergency medical event that necessitates a visit to the ER within a two-week time frame.

Methods

Ethics

This study was approved by the Institutional Review Board of National Taiwan University Hospital (IRB no. 202206092RIPD, NTUH, Taipei, Taiwan). All analytic procedures were performed in compliance with relevant laws and institutional guidelines. The informed consents were obtained from all participants for voice recording and analysis in this study.

Study design

We designed a retrospective study by using the audio recordings during telecommunications between patients and nurse case managers in our telehealth care centre. The study design is shown in Figure 1.

Figure 1.

Study design. This study uses telecommunication transcripts (and nursing notes) as input to train pre-trained transformer models to predict any emergency room visit within 2 weeks of the telecommunication in a telehealth care program. ER, emergency room.

National Taiwan University Hospital telehealth care centre and nursing note

Our hospital’s fourth-generation telehealth programme is a synchronized and integrated remote case management programme.⁷ Our programme provides services including daily transfer of physiologic data, daily communication with nurse case managers, 24/7 telehealth care, and discussions with primary care physicians for long-term medication management during and after acute events. The programme focuses on medication adherence, medical instruction, and early clinical deterioration detection.⁴

Approximately 82% of nurse case managers in our centre have achieved a Level 2 or higher out of the four levels, with an average age of 34.1 and job tenure of 8.9 years. In Taiwan, nurses of Level 2 or above have completed two years of clinical work and successfully passed the N2 evaluation. They are qualified to participate in the care of critically ill patients. (see Supplementary material online, Appendix  S1).⁸ They participate in in-service education focusing on managing common medical conditions such as chest pain, abdominal pain, hypertension, and more before working in our centre. Nurse case managers communicate with patients daily and make nursing notes after finishing the telecommunication. Any abnormalities in vital signs monitored at home will be thoroughly reviewed during telecommunication. The emergence of new symptoms, particularly those indicative of clinical decline or potential adverse effects of medications, will also be addressed. These nursing notes have been categorized into six predefined categories, including vital signs, clinical conditions, operation process, self-care, patient visit and equipment. The structure of our nursing notes generally included three parts: problem summary, evaluation, and management. Of the six categories, only the ‘equipment’ was excluded from the study. We believe that the equipment problem might not be relevant to the prediction in this study. Three samples of nursing notes are provided in Supplementary material online, Appendix  S2.

Data collection

The speech recording (in .wav format) and the corresponding nursing records (in .txt format) from 2013 to 2014 in our telehealth centre were used. We selected data from 2013 onwards because our telephone recording system became more refined after 2013. The speech recording from patients with ER visited within 2 weeks of the recordings were selected as cases; while the speech recording from patients without ER visit within 2 weeks were selected as control. The speech recording on the day of ER visit was not included to avoid bias. The 2-week time window chosen in our study was somehow arbitrary. In patients with chronic heart failure, it is estimated that earliest detectable alterations in biophysiological parameters might occur 10–20 days prior to an onset of clinical deterioration.⁹ Therefore, we selected a two-week time window in this pilot study.

Data preprocessing

Speech to text transformation

The speech recordings of telecommunications are transcribed verbatim by professional transcriptionists using headphones. Filler words, stutters and repetition were also retained. Transcription was performed entirely manually without the assistance of software. The transcripts were used for model building. A sample of dialogue transcript is provided in Supplementary material online, Appendix  S2.

Pre-trained transformer-based neural network models

To handle the textual clinical transcripts, we employed bidirectional encoder representations from transformers (BERT) as a foundational model for natural language understanding. Bidirectional encoder representations from transformers is an NLP tool that can ‘understand’ the context of words in a sentence by analysing them in both directions (left-to-right and right-to-left). Bidirectional encoder representations from transformers can be fine-tuned on specific tasks and datasets, enabling it to adapt to particular applications and deliver even greater accuracy. Five different BERT-based models and one pre-trained transformer model (GTP2-base-Chinese) were employed, each tailored for the Chinese language, to extract and process the nursing notes. The selected encoder models included:

1. BERT-base-Chinese: A classic bidirectional transformer model, adept at capturing contextual information. It has been pre-trained specifically for Chinese, containing rich contextual semantic information. It performs stably in semantic understanding of medium to short texts but may encounter bottlenecks when dealing with long texts (with over 256 tokens).

2. Chinese-xlnet-base: Compared with BERT, XLNet integrates autoregressive and bidirectional properties, making it better at handling complex word orders. It is suitable for modelling dependencies in long texts (256–512 tokens).

3. Ckiplab-base-Chinese: A model optimized by the CKIP Lab team for traditional Chinese tasks, trained with data more aligned to the characteristics of the Chinese language. It demonstrates strong generality in traditional Chinese language processing tasks.

4. Albert-base-Chinese: Albert is a lightweight version of BERT, with fewer parameters and faster inference speed. The performance of Albert-base-Chinese model may be suboptimal in complex tasks.

5. Text2vec-base-Chinese: A model focused on generating semantic embedding vectors, suitable for tasks such as semantic matching, similarity comparison, and vector retrieval for sentences or text fragments. It is optimized to align text in the embedding space, demonstrating excellent performance in sentence-level semantic modeling.

6. GPT2-base-Chinese: An autoregressive generative model that excels at producing natural and fluent text content. In Chinese environments, it is suitable for generating language content such as reports, summaries, or task responses.

These transformer encoders allowed us to capture rich contextual information and linguistic nuances specific to the Chinese language. We did not choose BioBERT or ClinicalBERT in our study as their focus on biomedical datasets and predominant English clinical records making them less aligned with our primarily Chinese conversational data.

Bidirectional encoder representations from transformers feature extraction

The WordPiece tokenizer was used in Bert-base Chinese, Ckiplab-base Chinese, Albert-base-Chinese and Text2vec-base-Chinese model; while SetencePiece and BPE tokenizer were used in Chinese-xlnet base model and Gpt2-base-Chinese model, respectively. For each segment of transcripts processed by the sliding window technique, BERT embeddings were generated. These embeddings captured the contextual information, enabling us to represent the transcript in a numerical format suitable for machine learning.

Sliding window

BERT has proven its effectiveness in capturing contextual information from text. However, a limitation of BERT is its fixed token length of 512. To overcome this constraint and efficiently process lengthy nursing notes, we applied a sliding window technique. We split the transcripts into overlapping segments of 512 tokens, allowing us to capture a comprehensive representation of the text while retaining the temporal structure of the data (Figure 2).

Figure 2.

Pre-trained transformer architecture, with sliding window. The architecture of model development by pre-trained transformer (bidirectional encoder representations from transformers as an example) with sliding window. The sliding windows, shown as the blue and red frames, are used to overcome the limitation of 512 tokens of bidirectional encoder representations from transformers model.

Long short-term memory for temporal relationships

The clinical transcripts contain time information that is crucial for understanding the temporal aspect of patient health. To model the sequential and temporal dependencies in the data, long short-term memory (LSTM) networks were utilized in part of our experiments. Long short-term memories are well-suited for processing sequences of data, making them an appropriate choice for handling time-series information in the clinical transcripts. Text embeddings are primarily obtained by extracting paragraphs using a sliding window. Each segment obtained from the sliding window served as text embeddings to be further fed into the LSTM network, enabling the model to capture temporal relationships between the various nursing notes (Figure 3).

Figure 3.

Pre-trained transformer architecture, with long short-term memory and sliding window. The architecture of model development by pre-trained transformer (bidirectional encoder representations from transformers as an example) with sliding window and long short-term memory. LSTM, long short-term memory.

Incorporating nursing notes

In addition to the telecommunication transcript, we integrated nursing notes as an additional input into our combination model. Nursing notes offer a valuable source of structured information that complements the unstructured nature of clinical notes. The inclusion of nursing notes aimed to capture patient-related data and enhance the model’s predictive accuracy. In the experiments involving nursing notes, we combined the speech transcripts and nursing records into a single long document, organized in chronological order.

Training and evaluation

The model was trained on a labelled dataset, where each instance was associated with the patient’s clinical history and an indicator of whether an ER visit occurred within the subsequent 2 weeks. Asynchronous successive halving algorithm was used for hyperparameter optimization.¹⁰ We employed standard machine learning metrics such as precision, recall, F1-score, and area under the receiver operating characteristic curve (AUC-ROC) to evaluate the model’s performance. We use accuracy and AUC-ROC as the main indicators for model’s performance. Cross-validation was not conducted due to the time sequence effect, as the model trained on the latest data may have incorporated outcome information from the earlier data.

Prediction

Patients with any acute condition requiring the visit of ER with or without subsequent hospitalization within two weeks from the clinical recording were the target of prediction in this study. Acute conditions include any reasons for emergency visits, encompassing unstable vital signs, high-risk symptoms such as unrelieved chest pain, and acute exacerbations of chronic diseases, regardless of whether hospitalization ultimately occurs. To prevent predictions influenced by irrelevant information, transcripts or nursing notes on the same day of ER visit were excluded from entering model training.

Experiments

We conducted five experiments using all the six BERT-based models in each experiment to predict the risk of acute event within 2 weeks. These experiments were different in the input and data processing techniques. The baseline experiment used telecommunication dialogue transcript as input (without using sliding window technique or LSTM) for the development of each model. Experiment 1 used telecommunication dialogue transcript as input and sliding window technique to deal with the transcript. Experiment 2 used telecommunication dialogue transcript as input and sliding window technique and LSTM to deal with the transcript. Experiment 3 used telecommunication dialogue transcript and nursing record as input and sliding window technique to deal with the transcript. Experiment 4 used telecommunication dialogue transcript and nursing records as input and sliding window technique and LSTM to deal with the transcript (Table 1).

Table 1.

The inputs and data processing techniques used in baseline experiment and Experiment 1–4

Experiment	Input		Data processing techniques
	Telecommunication dialogue transcript	Nursing record	Sliding window, to overcome the limitation of token length of 512	Long short-term memory (LSTM) for temporal relationships
Baseline experiment	V	—	—	—
Experiment 1	V	—	V	—
Experiment 2	V	—	V	V
Experiment 3	V	V	V	—
Experiment 4	V	V	V	V

Statistical analysis

The metrics used in this study to evaluate the performance of our models are presented as follows (true positive (TP), true negative (TN), false positive (FP), false negative (FN):

• Accuracy = (TP + TN)/(TP + TN + FP + FN)

• Recall (0) (Specificity) = TN/(TN + FP)

• Recall (1) (Sensitivity) = TP/(TP + FN)

• Precision (0) = TN/(TN + FN)

• Precision (1) = TP/(TP + FP)

• F1-score (0) = 2×Precision (0)×Recall (0)/(Precision (0) + Recall (0))

• F1-score (1) = 2×Precision (1)×Recall (1)/(Precision (1) + Recall (1))

To evaluate the general performance of each model, we used accuracy as the primary evaluation metric in this study. The overall performance of six models in each experiment was presented by median and interquartile range (IQR). We use the Wilcoxon signed-rank test To compare the overall performance of six models between five experiments, adjusted by the Bonferroni test for multiple comparisons.

Results

A total of 4.48 GB of .wav files or 1857 recordings were included in this study. There were 94 patients in the case group, with 585 speech recordings and nursing records; while 36 patients in the control group, with 396 speech recordings and nursing records. The dataset was divided into three parts: training, testing and validation, with 79, 10, and 9 patients, and 779, 132, and 70 recordings in each part, respectively.

Baseline experiment: telecommunication transcript as input, using pre-trained transformer models

The overall accuracy of the six models is 0.72 (IQR 0.705, 0.758). Among the models, Bert-base-Chinese demonstrated the highest overall accuracy (0.773) and sensitivity (0.949), indicating its ability to correctly identify patients at risk of an ER visit (Table 2). It was followed closely by Text2vec-base-Chinese, which displayed notable sensitivity (0.856), although with a slightly lower accuracy (0.758). In contrast, Albert-base-Chinese achieved perfect specificity but poor specificity in this experiment. The F1-scores provided a balanced assessment of model performance, where Bert-base-Chinese achieved the highest F1-score for the positive class (F1-score (1) = 0.86).

Table 2.

Baseline experiment—the performance of pre-trained transformer models; using transcripts as input

Model	Accuracy	Specificity	Sensitivity	Precision	F1-score (0)	F1-score (1)
Bert-base-Chinese	0.773	0.286	0.949	0.786	0.4	0.86
Chinese-Xlnet-base	0.705	0.4	0.814	0.79	0.418	0.802
Ckiplab-base-Chinese	0.697	0.486	0.773	0.807	0.46	0.79
Albert-base-Chinese	0.735	0	1	1	0	0.847
Text2vec-base-Chinese	0.758	0.486	0.856	0.822	0.515	0.838
GPT2-base-Chinese	0.705	0.457	0.794	0.802	0.451	0.798
Overall: median (IQR)	0.720(0.705–0.758)	0.428(0.286–0.486)	0.835(0.794–0.949)	0.804(0.790–0.822)	0.435(0.400–0.460)	0.820(0.798–0.847)

Experiment 1: transcript as input, using pre-trained transformer models and sliding window

In the context of our study, the utilization of sliding window techniques with various encoder models yielded distinct performance outcomes (Table 3). The overall accuracy of the six models is 0.75 (IQR: 0.742, 0.773). Notably, Bert-base-Chinese demonstrated the highest accuracy (0.7727) and sensitivity (0.9072), reflecting its effectiveness in correctly identifying patients at risk of an ER visit. This model also achieved a F1-score for the positive class [F1-score (1) = 0.854], showing its balance between precision and recall. Chinese-xlnet-base exhibited the second-highest accuracy (0.78) and sensitivity (0.928), indicating its strong predictive capabilities. It achieved a competitive F1-score for the positive class [F1-score (1) = 0.861], showing an effective balance between precision and recall. Additionally, Ckiplab-base-Chinese and Albert-base-Chinese delivered notable accuracy and sensitivity. However, the overall performance of these models, as measured by accuracy and F1-scores, varied. Text2vec-base-chinese and GPT2-base-chinese, while offering good sensitivity, showed lower overall accuracy. Despite this, their respective F1-scores for the positive class indicated a balanced approach to precision and recall, demonstrating their potential utility in identifying patients at risk of an ER visit.

Table 3.

Experiment 1—the performance of pre-trained transformer models with sliding window; using transcripts as input

Model	Accuracy	Specificity	Sensitivity	Precision	F1-score (0)	F1-score (1)
Bert-base-Chinese	0.773	0.4	0.907	0.807	0.483	0.854
Chinese-Xlnet-base	0.78	0.371	0.928	0.804	0.473	0.861
Ckiplab-base-Chinese	0.735	0.4	0.856	0.798	0.444	0.826
Albert-base-Chinese	0.758	0.286	0.928	0.783	0.385	0.849
Text2vec-base-Chinese	0.742	0.314	0.897	0.784	0.393	0.837
GPT2-base-Chinese	0.742	0.2	0.938	0.765	0.292	0.843
Overall: median (IQR)	0.750(0.742–0.773)	0.343(0.286–0.400)	0.917(0.897–0.928)	0.791(0.783–0.804)	0.419(0.385–0.473)	0.846(0.837–0.854)

Experiment 2: transcript as input, using pre-trained transformer models with long short-term memory

In our study, we employed BERT-based models in combination with LSTM for predicting the risk of an ER visit over various time frames (Table 4). The overall performances of the six models using different time interval data were: Day 1: 0.755 (IQR 0.731, 0.769), Day 1–3: 0.731 (IQR 0.731, 0.74), Day 1–7: 0.712 (IQR 0.702, 0.75), Day 1–14: 0.734 (IQR 0.712, 0.75). The overall performance of the six models using these four different time interval data showed no significant difference (all paired comparison P > 0.05, Wilcoxon Signed-Rank Test, adjusted by Bonferroni test). The results showed that different models exhibited varying levels of accuracy at different time intervals: For the initial day (Day1), Chinese-Xlnet-base demonstrated the highest accuracy (0.779), followed by GPT2-base-chinese, text2vec-base-chinese, Ckiplab_base-chinese, and Albert-base-Chinese. Over a 3-day timeframe (Day1–3), Chinese-Xlnet-base performed better in accuracy (0.808), with Bert-base-Chinese and Text2vec-base-chinese also showing competitive performance. Over the course of the first week (Day1–7), Bert-base-Chinese, Chinese-Xlnet-base, and Text2vec-base-chinese achieved balanced accuracy levels of 0.75, highlighting their reliability in short-term prediction scenarios.

Table 4.

Experiment 2—the performance of pre-trained transformer models with sliding window and long short term memory, using transcripts as input

Model	Day 1 accuracy	Day 1–3 accuracy	Day 1–7 accuracy	Day 1–14 accuracy
Bert-base-Chinese	0.721	0.731	0.75	0.75
Chinese-Xlnet-base	0.779	0.808	0.789	0.75
Ckiplab-base-Chinese	0.75	0.731	0.712	0.721
Albert-base-Chinese	0.731	0.721	0.702	0.75
Text2vec-base-Chinese	0.76	0.74	0.615	0.702
GPT2-base-Chinese	0.769	0.731	0.712	0.712
Overall: median (IQR)	0.755(0.731–0.769)	0.731(0.731–0.740)	0.712(0.702–0.750)	0.736(0.712–0.750)

Experiment 3: transcripts and nursing records as input, using pre-trained transformer models with sliding window

Incorporating both transcripts and nursing records into BERT-based models with sliding window techniques enhanced predictive accuracy for ER visits (Table 5). The overall accuracy of the six models is 0.892 (IQR: 0.891, 0.893). Notable results include high specificity (e.g. Text2vec-base-chinese with 0.9671) and balanced F1-scores [e.g. Bert-base-Chinese with F1-score (0) = 0.9390, F1-score (1) = 0.4677]. Models exhibited varying sensitivities, offering flexibility for specific project requirements. We tested the distribution of positive result prediction in the six different models in this Experiment 3, adjusted by Tukey HSD for multiple comparisons. The result showed some different behaviours between different models, with GPT2-base-Chinese model predicted significantly fewer positive events compared with other five models (see Supplementary material online, Appendix  S3).

Table 5.

Experiment 3—the performance of pre-trained transformer models with sliding window; using transcripts and nursing records as input

Model	Accuracy	Specificity	Sensitivity	Precision	F1-score (0)	F1-score (1)
Bert-base-Chinese	0.891	0.963	0.383	0.6	0.939	0.468
Chinese-Xlnet-base	0.892	0.977	0.3	0.654	0.941	0.411
Ckiplab-base-Chinese	0.893	0.983	0.269	0.693	0.942	0.387
Albert-base-Chinese	0.892	0.976	0.304	0.645	0.94	0.413
Text2vec-base-Chinese	0.892	0.967	0.37	0.618	0.94	0.463
GPT2-base-Chinese	0.896	0.977	0.326	0.673	0.942	0.439
Overall: median (IQR)	0.892(0.891–0.893)	0.977(0.967–0.977)	0.315(0.300–0.370)	0.649(0.618–0.673)	0.940(0.940–0.942)	0.426(0.411–0.463)

Experiment 4: transcripts and nursing records as input, using pre-trained transformer models with long short-term memory

Incorporating both transcripts and nursing records into BERT-based models with LSTM yielded varying levels of predictive accuracy (Table 6). The overall performances of the six models using different time interval data were: Day 1: 0.881 (IQR 0.872, 0.89), Day 1–3: 0.872 (0.873, 0.88), Day 1–7: 0.872 (IQR 0.872, 0.872), Day 1–14: 0.872 (IQR 0.872, 0.872). The overall performance of the six models using these four different time interval data showed no significant difference (all paired comparison P > 0.05, Wilcoxon Signed-Rank Test, adjusted by Bonferroni test). Models like Text2vec-base-chinese and Bert-base-Chinese demonstrated accuracy ranging from 0.8716 to 0.8898. Chinese-Xlnet-base excelled over a one-week period with an accuracy above 0.8869. Across a two-week timeframe, all models consistently maintained accuracy levels around 0.8716. The the overall accuracy of Experiment 3 (median 0.892, IQR 0.891–0.893), and overall accuracy of Experiment 4 on Day 1 (median 0.881, IQR 0.871–0.890) was similar (P > 0.05, Wilcoxon signed-rank test with Bonferroni adjustment, Figure 4). Compared with Experiment 3, Experiment 4 needed 2.7% more memory space, 11.7% time for training on central processing unit or 225% time for training on graphics processing unit.

Table 6.

Experiment 4—the performance of pre-trained transformer models with sliding window and long short term memory; using transcripts and nursing records as input

Model	Day 1 accuracy	Day 1–3 accuracy	Day 1–7 accuracy	Day 1–14 accuracy
Bert-base-Chinese	0.89	0.874	0.872	0.872
Chinese-Xlnet-base	0.872	0.872	0.887	0.884
Ckiplab-base-Chinese	0.893	0.872	0.872	0.872
Albert-base-Chinese	0.89	0.88	0.872	0.872
Text2vec-base-Chinese	0.872	0.892	0.872	0.872
GPT2-base-Chinese	0.872	0.872	0.872	0.872
Overall: median (IQR)	0.881(0.872–0.890)	0.873(0.872–0.880)	0.872(0.872–0.872)	0.872(0.872–0.872)

Figure 4.

The comparisons of accuracy between different experiments. *P < 0.05, adjusted by the Bonferroni test for multiple comparisons. There are significant differences in accuracy between Experiment baseline 1 or 2 vs. Experiment 3 or 4. There are no significant differences in accuracy between Experiment baseline 1 and 2. There are no significant differences in accuracy between Experiment 3 and 4.

Discussion

Main findings

The results of our study provide insights into predictive modelling for clinical outcomes by using telecommunication dialogue and nursing notes. Firstly, our findings demonstrate the feasibility of utilizing transcripts for predicting clinical outcomes, showcasing the potential for leveraging textual data in healthcare applications. Secondly, the use of pre-trained transformer models with sliding window techniques has proven to be a practical and effective approach, offering a framework for capturing contextual information and temporal relationships in the data. Thirdly, the significant improvement in predictive accuracy observed when incorporating nursing notes into the input underscores the importance of utilizing structured data in conjunction with unstructured textual data, resulting in a marked enhancement in model performance. These results collectively highlight the promise of employing advanced NLP techniques and telecommunication records for predictive modelling in telehealth care contexts. This predictive model is helpful in telehealth care for nursing staff to identify patients with higher risk for adverse events. This model may also be applicable to other scenarios, such as hospital-at-home care, where it can help identify patients whose changing conditions necessitate admission to a traditional brick-and-mortar hospital. This study also paves the way for future virtual nursing assistants. AI-based virtual nursing assistants will be able to communicate with patients and pick up the high-risk signals from dialogue.

Although incorporating nursing records into the input substantially improved accuracy and specificity, this came at the cost of a reduction in sensitivity. This trade-off suggests that incorporating nursing records was effective at ruling out low-risk patients but may miss some who are truly at risk. This behaviour may be favourable in certain operational contexts, such as reducing alarm fatigue or optimizing downstream resource use. When the models in Experiment 3 flag a patient as high-risk, clinicians should take the alarm seriously and consider further clinical evaluation.

Studies using clinical notes to predict outcomes

There is a study to analyse the trend of telehealth calls and COVID-19 ER visit 3 days later.¹¹ However, no study has been performed to analyse the relationship between the content of telecommunication and future ER visit. Several studies have explored the utilization of NLP and electronic clinical notes to predict various clinical outcomes and events.^12–15 These studies employ NLP techniques to extract valuable information from textual patient records, contributing to improved healthcare decision-making. In 2018, Google and researchers from University of San Francisco published a study by using electronic health record (including text and non-text) and deep learning method from 216 221 patients to predict in-hospital mortality, length-of-stay and discharge diagnoses. The models developed in their study outperformed traditional prediction models with accuracy up to 0.94 for in-hospital mortality. This study showed that deep learning methods can outperform traditional prediction models, achieving high levels of accuracy.¹² From the electronic health records among 101 196 patients admitted to ICU, Marafino et al. extracted vital signs, laboratory data and clinical free-text notes to develop a model to predict in-hospital mortality with an accuracy up to 0.922 (95% CI, 0.916–0.924; P < 0.001).¹³ In a later methodology study, authors from the same group reported that the preprocessing strategies have impact on the model performance. Clinical notes preprocessed with term frequency-inverse document frequency or 1–2 n-gram methods have higher AUROC compared with raw text, cleaned text or cleaned and stemmed text. This study highlights the importance of data preparation in achieving better results in this task.¹⁴ Brann et al. using NLP to process nursing triage notes to predict sepsis with high accuracy (AUC 0.97 at 12 h).¹⁵ These studies showed that using NLP to process nursing notes is feasible to predict acute events. Our study is the first to use the raw dialogue between nurse care managers and patients as the input to predict acute events. However, our data also showed that by adding the nursing record into raw dialogue can improve the prediction performance.

Sliding window in this study

The BERT model has a limitation in processing text beyond 512 tokens, which can be a challenge when dealing with long clinical transcripts or nursing records in our study. To address this, we implement the sliding window technique with a window size of 512 tokens. This approach allows us to systematically segment lengthy text into overlapping partitions. By using the sliding window technique, we maximize the utilization of relevant details from patient transcripts and nursing records, preserving contextual continuity across overlapping partitions. This method may also enhance model performance by ensuring that important medical terms, symptoms, and patient conditions remain within the model. This technique reduces the risk of missing key information that might otherwise be excluded if only the first 512 tokens were considered.

Medical dialogue to predict outcome

Currently, there is no research in the literature that uses the conversation during medical interview speech to predict clinical events. Studies in the field of medical conversation have primarily concentrated on automated speech recognition and converting speech to text.^16,17 A Chinese medical dialogue dataset, ReelMedDial, collected 2637 medical dialogues short-video clips of real medical consultations. The aim of this dataset is for NPL tasks including medical response generation, department routing, and doctor recommendation.¹⁶ Another telemedicine dialogue dataset, IMC-21, has been published recently. The dataset collected medical consultation records on 10 paediatric diseases. Five tasks including named entity recognition, dialogue act classification, symptom label inference, medical report generation and diagnosis-oriented dialogue policy and a set of benchmark results have been reported in the study.¹⁷ These studies showed that the importance of medical dialogue has been recognized. Our result provides a new potential usage of telemedicine dialogue.

Nursing notes recorded during hospitalization to predict readmissions

In a hospital setting, nursing notes are a rich source of structured data, containing information on a patient’s vital signs, medications, nursing assessments, and care interventions. The application of NLP in nursing notes is an emerging research field, with most of the studies conducted within hospital settings.¹⁸ Our result showed that by including nursing record into the input of the model increased the specificity of the prediction. This result implied that our nursing records may highlight more symptoms or signs for low-risk conditions compared with simple dialogues. In a recent study, Topaz et al. used NLP to identify seven common symptoms (including anxiety, depressed mood, and pain) narrative nursing notes automatically in 89 825 patients admitted to a large homecare agency in the USA. They found that NLP can identify the symptoms accurately in narrative nursing notes. The more identified symptoms were associated with higher risk of ER visit.¹⁹ However, these seven common symptoms were prespecified but not derived from the data. This approach may limit its generalizability and accuracy. In contrast, our study did not pre-determine the symptoms or medical concepts to be associated with risk for ER visit. Further studies to compare these two approaches may be warranted.

Pre-trained transformer models in the use of medical field; vs. other large language model (LLM)

A transformer is a type of deep learning model used for NLP and other sequential data tasks. It relies on a self-attention mechanism to process input data in parallel, capturing long-range dependencies and contextual information effectively. Models like BERT are based on the transformer architecture and have been pre-trained on vast text corpora, making them proficient at various NLP tasks, such as text classification, translation, and text generation. Recently, large language models have been demonstrated with unprecedented ability in NLP and text generation. State-of-the-art model such as GPT-4 can even exceeds the passing score on the United States Medical Licencing Examination by 20 points.^20,21 Recently, LLM has also been used in virtual assistants in health care domain, and can be fine-tuned for specific health care purpose.²² While GPT-4-like LLM is very useful in text generation or question-answering tasks, the primary strength of BERT lies in understanding the context of language. Therefore, we choose BERT for our base model. However, future researches involving other NLP techniques, including large language models, is encouraged within this subject area.²²

Nurse-led patient education to prevent emergency room visit

Case management is an effective intervention to prevent unnecessary admission and ER visit.²³ The effectiveness of case management programme for patients with chronic conditions is partly through nurse-led patient education.^24,25 Nurse-led education empowers patients with the knowledge and skills needed to make healthcare decisions, manage their health effectively, and detect deterioration earlier, and may have substantial legacy effect on reducing ER visits. The legacy effect has been observed in several chronic condition management including glycaemic control, blood pressure control and lipid-lowering therapy.²⁵ In the recent E-INH (extended—Interdisciplinary Network Heart Failure) trial, which employed telephone-based remote patient management for patients with heart failure, observed that various clinical outcomes exhibited statistically significant differences between the treatment and control groups specifically in the post-intervention follow-up period, rather than during the intervention itself.²⁶ It is possible that our nurse-led patient education via daily telecommunication has a legacy effect on patient outcomes.

Limitations

In this study, several limitations warrant consideration. Firstly, our research adopted a retrospective design, which inherently introduces the risk of recall bias. Furthermore, the sample size of patients included in our analysis was relatively small, which may limit the generalizability of our findings. Additionally, our study utilized the pre-trained transformer models as a foundation for NLP tasks, while a larger language model could have potentially yielded more nuanced results. The choice of BERT, though widely used and effective, may have constrained the depth and breadth of the semantic understanding of text data. Meanwhile, our results were not validated in other independent, external datasets. The absence of external validation may limit the generalizability of our model, particularly in populations speaking different languages. It is possible that these six models may have different compatibility with different languages. However, we believe that leveraging dialogue to predict future acute events remains a valuable avenue for exploration, as it reflects the cognitive processes of healthcare professionals in managing patients with chronic illnesses. Further studies incorporating larger patient cohorts across diverse populations are warranted to enhance the model’s validity and generalizability. Finally, the training data were not fully balanced, which may have some effects on the performance of the final model. These limitations should be acknowledged in the interpretation of our results and may provide avenues for future research to explore these topics in more depth.

Conclusion

By using BERT-based NLP, our pilot study demonstrates the feasibility to predict ER visits with telehealth dialogue transcript and nursing notes in a telehealth care programme. The incorporation of nursing notes significantly enhances the model’s performance. Further studies are required to evaluate the model’s performance in a prospective setting.

Lead author biography

Wu Hui-Wen graduated with Master of Science in nursing at National Taipei University of Nursing and Health Sciences. She is currently the associate director of Telehealth care center in National Taiwan University. Her main area of interest is the management of telehealth care programme.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health.

Author contributions

H-.W.W: Conceptualization, investigation, methodology, supervision, writing—original draft. C-.S.H.: Data curation, formal analysis, methodology, writing—original draft. Y-.H.C.: Conceptualization, data curation, funding acquisition, resources. C-.C.H.: Conceptualization, formal analysis, funding acquisition, resources. J-.K.L.: Investigation, methodology, project administration, software, supervision. S-.T.H.: Conceptualization, project administration, supervision. Y.L.H.: Conceptualization, funding acquisition, resources, supervision, writing-original draft, writing—review & editing.

Funding

The study was supported by the National Science and Technology Council, Taiwan (NSTC 111-2622-8-002-026 –IE; NSTC 112-2634-F-002-003). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data availability

The data underlying this article cannot be shared publicly due to the privacy of individuals that participated in the study. The data will be shared on reasonable request to the corresponding author.