Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Feb 16.
Published in final edited form as: Nurs Res. 2022 Feb 16;71(4):285–294. doi: 10.1097/NNR.0000000000000586

Detecting Language Associated with Home Healthcare Patient’s Risk for Hospitalization and Emergency Department Visit

Jiyoun Song 1, Marietta Ojo 2, Kathryn H Bowles 3, Margaret V McDonald 4, Kenrick Cato 5, Sarah Collins Rossetti 6, Victoria Adams 7, Sena Chae 8, Mollie Hobensack 9, Erin Kennedy 10, Aluem Tark 11, Min-Jeoung Kang 12, Kyungmi Woo 13, Yolanda Barrón 14, Sridevi Sridharan 15, Maxim Topaz 16
PMCID: PMC9246992  NIHMSID: NIHMS1776783  PMID: 35171126

Abstract

Background:

About one in five patients receiving home health care (HHC) services are hospitalized or visit an emergency department (ED) during a home care episode. Early identification of patients at risk can prevent these negative outcomes. However, risk indicators, including language in clinical notes that indicate a concern about a patient, are often hidden in narrative documentation throughout their HHC episode.

Objective:

To develop an automated natural language processing (NLP) algorithm to identify concerning language indicative of HHC patient risk of hospitalizations or ED visits.

Methods:

This study used the Omaha System—a standardized nursing terminology that describes problems/signs/symptoms that can occur in the community setting. First, five HHC experts iteratively reviewed the Omaha System and identified concerning concepts indicative of HHC patient risk of hospitalizations or ED visits. Next, we developed and tested an NLP algorithm to identify these concerning concepts in HHC clinical notes automatically. The resulting NLP algorithm was applied on a large subset of narrative notes (2.3 million notes) documented for 66,317 unique patients (n = 87,966 HHC episodes) admitted to one large HHC agency in the Northeast United States between 2015 and 2017.

Results:

A total of 160 Omaha System signs/symptoms were identified as concerning concepts for hospitalizations or ED visits in HHC. These signs/symptoms belong to 31 of the 42 available Omaha System problems. Overall, the NLP algorithm showed good performance in identifying concerning concepts in clinical notes. More than 18% of clinical notes were detected as having at least one concerning concept, and more than 90% of HHC episodes included at least one Omaha System problem. The most frequently documented concerning concepts were pain, followed by issues related to neuro-musculoskeletal function, circulation, mental health, and communicable/infectious conditions.

Conclusion:

Our findings suggest that concerning problems or symptoms that could increase the risk of hospitalization or ED visit were frequently documented in narrative clinical notes. NLP can automatically extract information from narrative clinical notes to improve our understanding of care needs in HHC. Next steps are to evaluate which concerning concepts identified in clinical notes predict hospitalization or ED visit.

Keywords: clinical deterioration, home health care, natural language processing, nursing informatics, Omaha System, risk assessment

Since the early 1970s, the health care system has drastically changed, with an increasing number of clinically complex patients treated in post-acute care settings (e.g., long-term care hospitals, inpatient rehabilitation facilities, skilled nursing facilities, and home health care agencies; Hardin & Mason, 2019; Jarvis, 2001). Home health care (HHC) has emerged as one of the many types of post-acute care, represented by intermittent home visits conducted by skilled health care providers (e.g., registered nurses, physical therapists, social workers). Currently, approximately 3.4 million adults receive HHC from more than 12,200 HHC agencies across the United States (Medicare Payment Advisory Commission, 2019). Previous studies support the critical role of HHC in demonstrating improved patient outcomes and reduced costs (Han et al., 2013; Howard et al., 2019; Weeks et al., 2018). HHC services are expected to increase due to longer life expectancy and a higher prevalence of chronic diseases among the elderly (Landers et al., 2016; Mitzner et al., 2009).

Recent reports show that about one in five patients is hospitalized or visit an emergency department (ED) during HHC services (Medicare Payment Advisory Commission, 2019). Despite national and local quality improvement efforts focusing on acute and chronic ambulatory care sensitive conditions (i.e., health conditions for which adequate management, treatment, and interventions delivered in the ambulatory care setting could potentially prevent hospitalization; Agency for Healthcare Research and Quality, 2018), these numbers have not improved over the last several years (Centers for Medicare & Medicaid Services, 2019). HHC providers have an opportunity to apply recent evidence, mainly from the hospital setting, to reduce hospitalization risk via early patient risk detection and notification (Fu et al., 2020). Given that up to 40% of hospitalizations are preventable with early or timely intervention (Morganti et al., 2013), the early identification of the patients at risk for clinical deterioration is emphasized as one of the key objectives to improve the quality of HHC.

The utilization of clinical nursing narrative documentation in predicting clinical outcomes is an area that has been coming into the limelight because notes often contain more information on patients’ medical conditions than structured data in electronic health records or through standardized assessment tools (Song et al., 2021). In addition to standardized assessments, identifying risk factors in free-text data can further improve the prediction of health care utilization. Natural language processing (NLP) techniques used to extract the information in clinical notes can leverage free-text data more effectively and accurately. In the hospital settings, NLP was used to mine clinical notes written by nurses and other health providers to extract important symptom information (Koleck et al., 2021), identify socio-behavioral risk factors (e.g., substance or alcohol abuse; Navathe et al., 2018), find patients with depression (Zhou et al., 2015), predict patient’s hospital mortality (Collins et al., 2013), etc. Several recent systematic reviews support the influence of NLP on a diverse range of medical fields, such as mental health (Le Glaz et al., 2021) and radiology (Sorin et al., 2020).

In HHC nursing, several studies demonstrated that nurses’ expressions in free-text clinical notes were valid as an indicator for predicting negative outcomes such as deterioration (Topaz et al., 2020a, 2021). However, those studies had a limited focus on a small number of risk factors (e.g., symptoms) when examining information from the clinical notes. Given that HHC is a community-based program providing services in patients’ homes, inclusion of a larger set of personal, social, and environmental factors may improve identification of hospitalization or ED visit risk.

To address these knowledge gaps, this study aimed to develop a comprehensive NLP algorithm that can be used to create an early warning score system based on a set of lexicon categories for concerning concepts indicative of patient risk of hospitalizations or ED visits in HHC. Specifically, the aims were to (a) identify HHC concerning concepts that are indicative of an increased risk for hospitalization or ED visits using a standardized nursing terminology (the Omaha System), (b) create and validate an NLP algorithm to extract concerning concepts from narrative clinical notes, and (c) estimate the prevalence of concerning concepts in the clinical notes.

Method

Study Data Set and Population

The study data set included all patients admitted to HHC services at a large urban HHC organization in the Northeastern United States between January 1, 2015, and December 31, 2017. Patients could have more than one HHC episode; there were 87,966 HHC episodes for 66,317 unique patients. Episode was defined as all services provided within a specific period of time (i.e., initiated an admission, ended by a discharge). Approximately 2.3 million HHC clinical notes were extracted for this patient cohort. Two types of clinical notes were included:

  1. Visit notes, which are used to document the patient’s status and the care provided during an HHC visit (total n = 1,029,535); and

  2. care coordination notes, which are used to document communication between health care providers and to note other administrative care-related activities (total n = 1,292,442).

The institutional review boards approved the study of the participating institutions.

Identifying Concerning Concepts for Hospitalization or ED Use in HHC Patients

With 2.3 million notes to examine, the team used the Omaha System, a widely used standardized terminology for documentation of clinical information in community-based care (https://www.omahasystem.org/) to identify a subset of possible concerning terms (Martin, 2005). The Omaha System was chosen as the tool to examine documentation in this study because the system’s standard terminology (i.e., ontology) comprehensively and holistically describes health factors to communicate and improve clinical practice (Monsen, 2018). The Omaha System includes 42 problems (e.g., income, neglect, health care supervision, circulation), organized in four domains (environmental, psychosocial, physiological, and health-related behavior) aligned with 335 unique signs/symptoms (e.g., uninsured medical expenses, lacks adequate physical care, inconsistent source of health care, irregular heart rate).

In this study, we used the signs/symptoms and problem labels of the Omaha System to identify the corpus of terms used in NLP to detect concerning concepts documented in the nurse notes. The signs/symptoms and problem labels in the Omaha System associated with risk of hospitalizations or ED visits in HHC were defined as concerning concepts.

Concerning Concepts

Concerning concepts were identified by five experts in HHC and informatics (JS, MT, KHB, AT, and VA) based on the literature, expert’s opinion, and/or their clinical experience. All had extensive clinical or research experience in HHC, one had a master’s degree, and the other four had PhDs in nursing. The experts were presented with 335 Omaha System signs and symptoms and asked to indicate, “What signs/symptoms, if documented in the patient records, would cause concern for risk of hospitalizations or ED visits for HHC patients 65 years of age or older?”

The experts scored each sign/symptom on a scale from 1 to 3 (1 – usually not concerning; 2 – occasionally concerning; 3 – usually concerning). The signs/symptoms that received a score of 3 (usually concerning) from the majority of experts (3 or more) were regarded as concerning signs/symptoms potentially associated with hospitalizations or ED visits in HHC. The study team discussed signs/symptoms that received a score of 2 from most experts for further consideration. Signs/symptoms that received a score of 1 from the majority of experts (3 or more) were considered as “not concerning” (see Supplemental Digital Content [SDC]). The inter-rater agreement between experts was fair (Fleiss’ kappa = 0.32; McHugh, 2012). All discrepancies, including the sign/symptom with the difference between the maximum and the minimum score of 2, were resolved in several consensus group meetings.

NLP Algorithm Creation and Validation

The process of the NLP algorithm creation and validation is depicted in Figure 1 and consists of six steps:

Figure 1.

Figure 1.

Open in a new tab

The process of workflow for natural language prcoessing (NLP) algorithm creation and validation

1. Create preliminary list of terms for concerning concepts using a vocabulary of standardized health terms.

The initial steps to extract specific data require comprehensive lexicon development. We identified a list of synonyms for each concerning the Omaha System concept using a large vocabulary of standardized health terms (Unified Medical Language System [UMLS]; Sinha et al., 2013). For example, for the Omaha System sign/symptom of inadequate medication regimen under the problem of medication regimen, we identified UMLS synonym expressions such as “uses less medication than prescribed,” “inconsistency with medication regimen,” “no medication list,” and “unsure of medication history.”

Five of the research team members (JS, MO, MT, VA, and MH) reviewed the preliminary list independently then validated and finalized concerning signs/symptoms of the Omaha System. They were HHC nurses, PhD-level research scientists in nursing informatics, and a nursing PhD student.

2. Language model creation: Word embedding model (Word2Vec).

Language models are statistical representations of a certain body of text. A specific language model called word embedding (Word2Vec) was used (Mikolov et al., 2013). Word2Vec learns word associations from a large corpus of text, then once trained, it can help detect synonymous words for terms of interest based on a specific domain.

In this study, a publicly available NimbleMiner NLP software (Topaz et al., 2019) was used to identify synonyms for the concerning signs/symptoms present in a large body of HHC clinical notes available for the study period. The system can be downloaded from http://github.com/mtopaz/NimbleMiner under General Public License v3.0.

3. Interactive vocabulary explorer to identify synonymous concerning signs/symptoms.

An interactive rapid vocabulary explorer (part of the NimbleMiner software) was implemented by three coauthors (JS, MO, and MT), experts in HHC and informatics, to discover large vocabularies of synonyms. The researchers interacted with the software to input a query term of interest (e.g., sign/symptom of bruising under Omaha System problem of skin), and the system output included a list of potential synonyms (e.g., “bruise noted,” “bruised area,” “ecchymosis”). We started the vocabulary expansion process using prepopulated lists of synonyms extracted from the UMLS for each concerning signs/symptoms in the Omaha System and then expanded the list of synonymous expressions using the interactive rapid vocabulary explorer. Relevant terms were selected and saved by clicking on them in the interactive vocabulary explorer user interface. The NLP software has a “negation module”; “negation” refers to words indicating negated synonyms (for example: “denies,” “no,” “not,” “ruled out”). Hence, our NLP software can distinguish and exclude negated terms.

4. Label assignment and review.

In the final stage of NLP algorithm creation, the system used terms selected and saved during the previous stage to assign labels to clinical notes. Assigning a “positive label” means that the clinical note includes concerning signs/symptoms. The negation module distinguished the positive or negative sense of the concept of concern. For example, if a clinical note mentioned “severe dyspnea,” it was considered a positive label. In contrast, clinical notes with a negation, such as “no dyspnea” or “denies dyspnea,” were not assigned a positive label. We mapped the concerning signs/symptoms to a broader Omaha System problem category for labeling to enable human interpretability of the results and enable NLP system performance evaluation.

5. Development of an expert generated reference standard for NLP algorithm validation.

To validate whether the NLP algorithm could identify Omaha System problems, including the concerning signs/symptoms with accuracy similar to a clinical expert, we generated a “gold standard” testing set of 1,000 randomly extracted clinical notes. This gold standard testing set was manually annotated using Microsoft Excel by three HHC experts (JS, MO, and MT) for presence of concerning signs/symptoms at the level of the Omaha System problem. The observed inter-rater agreement was high (Fleiss’ kappa = 0.89), and all discrepancies were resolved through discussion.

6. Evaluation of NLP algorithm performance.

Lastly, the NLP algorithm was applied on the gold standard testing set to calculate precision (defined as the number of true positives out of the total number of predicted positives), recall (the number of true positives out of the actual number of positives), and F-score (the weighted harmonic means of the precision and recall) for each Omaha System problem.

Applying the NLP Algorithm

The finalized NLP algorithm was applied to the entire data set, including approximately 2.3 million HHC clinical notes. Descriptive statistics were used to calculate the proportion of notes that include concerning signs/symptoms. The differences in note length and type of note between the notes that had the concerning signs/symptoms and those that did not include the concerning signs/symptoms were compared using student t-test or chi-square tests. For all analyses, p-values less than 0.05 were regarded as the indicator of statistical significance. Odds ratios (OR) for the type of notes were calculated with 95% confidence intervals (CI). Lastly, the frequency of each Omaha System problem was calculated at the episode level. All analyses were implemented using R software version 4.1.0 (R Foundation of Statistical Computing, Vienna, Austria).

Results

Identifying Concerning Signs and Symptoms for Hospitalizations and ED Visits in HHC

Of 335 signs/symptoms of the Omaha System presented to the experts, 131 were initially identified as concerning, with 29 signs/symptoms added after reviewing the “occasionally concerning” category through group consensus. This resulted in a list of 160 signs/symptoms identified as concerning signs/symptoms for hospitalizations or ED visits in HHC. These signs/symptoms belong to 31/42 (73.8%) of available Omaha System problems. In several Omaha System problem categories (e.g., abuse), virtually all signs/symptoms were considered concerning signs/symptoms. In other problem categories, only selected subsets of signs/symptoms were selected (e.g., circulation, skin, or income). For instance, under the skin problem, the signs/symptoms of excessively dry, excessively oily, or hypertrophy of nails were not considered concerning concepts for the risk of unplanned hospitalization or ED visits. Further, no signs/symptoms from several problem categories (e.g., postpartum or pregnancy) were selected concerning signs/symptoms since they did not apply to the older adult HHC patient population of interest for the study. The detailed concerning signs/symptoms of the Omaha System problem is presented in the SDC.

NLP Algorithm Performance

Among 1,000 clinical notes randomly selected for gold-standard manual review, 393 (39.3%) included at least one Omaha System problem, and a total of 496 Omaha System problems were detected. The most common Omaha System problem documented in the clinical notes was neuro-musculoskeletal function (16%) which included signs/symptoms of decreased balance and gait/ambulation disturbance, followed by pain and circulation problems as the next most common documented problems (14% and 9%, respectively). The overall NLP algorithm’s performance in identifying the Omaha System problem, including the concerning signs/symptoms, was good (average F-score = 0.84), with best results for the social contact, hearing, income, and abuse problems (F-score = 1) and poorest results for the nutrition problem (F-score = 0.62; see Table 1).

Table 1.

Evaluation of NLP Algorithm Performance via Gold-Standard Manual Review (Total n = 1,000 Clinical Notes)

The Omaha System problems Total frequency and proportion of documentation [%(n)] Precision Recall F-score
Neuro-musculo-skeletal function 16% (78) 0.99 0.76 0.86
Pain 14% (68) 0.84 0.95 0.89
Circulation 9% (47) 0.94 0.80 0.86
Mental health 9% (47) 0.96 0.75 0.84
Skin 9% (46) 0.93 0.80 0.86
Health care supervision 7% (35) 1.00 0.61 0.76
Cognition 7% (34) 0.97 0.89 0.93
Respiration 6% (32) 0.97 0.78 0.86
Communicable infectious condition 4% (18) 0.94 0.81 0.87
Social contact 3% (17) 1.00 1.00 1.00
Digestion hydration 3% (14) 0.93 0.62 0.74
Medication regimen 2% (9) 0.88 0.68 0.77
Bowel function 2% (8) 1.00 0.89 0.94
Genito urinary function 2% (8) 1.00 0.73 0.84
Nutrition 2% (8) 1.00 0.44 0.62
Neglect 1% (5) 1.00 0.45 0.63
Communication with community resources 1% (4) 1.00 0.57 0.73
Speech and language 1% (4) 1.00 0.8 0.89
Hearing 1% (3) 1.00 1.00 1.00
Income 1% (3) 1.00 1.00 1.00
Residence 1% (3) 1.00 0.70 0.82
Consciousness 0% (2) 1.00 0.71 0.83
Abuse 0% (1) 1.00 1.00 1.00
Interpersonal relationship 0% (1) 0.50 1.00 0.67
Personal care 0% (1) 1.00 0.67 0.80
Total 100% (496)
Average 0.95 0.78 0.84
Open in a new tab

Note: Only categories where concerning concepts found in gold-standard references were presented. Therefore, Neighborhood/workplace safety, Oral health, Sanitation, Sexuality, Sleep and rest patterns, and Substance use were not presented in the table.

Applying NLP Algorithm on a Large Set of Clinical Notes

On average, 26.4 (standard deviation [SD] = 27.8) clinical notes were documented for each episode. In general, visit notes were longer than care coordination notes (mean length [SD] = 246.8 [241.4] characters vs. 99 [98.5] characters, p < .05).

When applying the NLP algorithm on the full sample of clinical notes, 18.1% (419,517 of 2,321,977) notes were detected as having at least one concerning signs/symptoms. Among notes that included at least one concerning signs/symptoms, 81% (347,652 of 419,517) visited notes, and the rest were care coordination notes. Statistically, concerning signs/symptoms were almost nine times more likely to be documented in visit notes than in care coordination notes (OR, 8.7 [95% CI, 8.6 - 8.8]). In addition, the average length of notes that included concerning signs/symptoms was significantly longer than notes that did not have concerning signs/symptoms (mean length [SD] = 359.4 characters [263.6] vs. 121.6 characters [138.4]; all p < .05). Below are example sentences that include concerning signs/symptoms:

Example 1: “… VN (visiting nurse) reported to MD (medical doctor) swelling (concerning signs/symptoms under the problem of circulation) and bruise (concerning signs/symptoms under the problem of skin) noted on pts (patient) lt (left) lower leg and foot.”

Example 2: “… PT (patient) is homebound due to weakness on both legs (concerning signs/symptoms under the problem of neuro-musculoskeletal function) and difficulty walking (concerning signs/symptoms under the problem of neuro-musculoskeletal function).”

Example 3: “… MD (medical doctor) saw patient earlier in week and patient had congestion. Reviewed patient’s diet: patient is not compliant with low salt diet (concerning signs/symptoms under the problem of health care supervision).”

A total of 289,324 Omaha System problems were detected during the study period. Figure 2 shows the frequency of episodes with documented Omaha System problems in the clinical notes. At the episode level, the most frequently reported problem was pain, followed by neuro-musculoskeletal function, circulation, mental health, and communicable/infectious conditions (48.3%, 46.1%, 35.3%, 32.8%, and 25.9%, respectively). Overall, an average of 3.2 (SD = 2.3, median = 3) Omaha System problems were detected per episode. Among the episodes having at least one Omaha System problem, an average of 3.6 (SD = 2.2, median = 3) problems were detected. Over 90% of the home health episodes had at least one Omaha System problem (i.e., with concerning signs/symptoms) documented (79,865 out of 87,966).

Figure 2.

Figure 2.

Open in a new tab

Frequency of HHC episodes containing documentation of concerning signs and symptoms within Omaha System problems.

Discussion

To our knowledge, this is the first study to identify concerning concepts potentially associated with hospitalization or ED visits in HHC using the Omaha System. Compared to other standardized terminologies such as International Classification of Diseases, 9 Revision, Clinical Modification (ICD-9-CM) that encompasses mostly medical diagnoses, the Omaha System was developed to describe health care provided in community settings, like HHC and public health. Previously, the Omaha System has been identified as a better candidate for predicting hospitalization in HHC patients compared to ICD-9-CM (Monsen et al., 2012).

Our results highlight the uniqueness of the HHC environment. Compared to risk factors identified in a similar study in the hospital setting (Kang et al., 2020), we found additional unique signs/symptoms under the health care supervision problem (e.g., fails to obtain routine/preventive care or inconsistent source of health care) and the medication regimen problem (e.g., not follow recommended dosage/schedule or unable to take medications without help). A recent systematic review showed that most of the previous studies of HHC patient risk of hospitalization and ED visit only used a small fraction of potential risk factors at the level of signs/symptoms of the Omaha System identified in this study (Ma et al., 2018). For example, concerning concepts such as signs/symptoms within health care supervision or medication regimen of the Omaha System, problems were infrequently, if ever, identified in previous studies. To develop a more comprehensive picture for modeling HHC patient risk, more inclusive terms, such as those in the Omaha System, are warranted.

Our results are supported by a recent study using a different form of NLP to identify risk factors associated with HHC patient risk for hospitalizations and ED visits (Topaz et al., 2020b). This previous study found several broad categories of words and expressions HHC nurses use to indicate increased patient’s risk, including “clinical factors” (expressions like “nausea,” “dehydrated,” “vomited”), “coordination/communication” (expressions like “plan discussed,” “MD called,” “supplies ordered”), and “social/environmental factors” (expressions like “lives alone,” “PT lives with,” “private insurance”). Virtually all of these broad categories are represented by the Omaha System problems identified in this study. Our results represent a more complete list of risk factors than the previous research.

This study aimed to develop and test an NLP algorithm to identify factors related to the risk of hospitalization or ED visits in HHC. The NLP algorithm achieved good performance on a subset of randomly selected HHC clinical notes. Our findings suggest that the risk of hospitalization or ED visits was frequently documented in narrative clinical notes. NLP can extract data from narrative clinical notes to improve our understanding of care needs in HHC. This finding also highlights the value of HHC clinical documentation in improving identification of patients at risk for hospitalization or ED visits (Song et al., 2021; Topaz et al., 2020b).

In addition, in developing the NLP algorithms, our team used clinician-validated lexicons rather than unsupervised machine learning techniques. Although sometimes machine learning techniques can capture hidden patterns that are hard to identify through human review (Wang et al., 2019), the lexicons created by clinical experts are transparent, comprehensible, and intuitive (Weng et al., 2017). Our study adds to the research conducted by others that successfully used the Omaha System in text mining applications (Bjarnadottir et al., 2020; Monsen et al., 2018). These results can inform the potential development of an early warning scoring system to help reduce negative outcomes in HHC by identifying a patient’s concerns, signs, and symptoms.

In this study, 90% of HHC episodes had at least one documented Omaha System problem. This number is higher than reported in previous NLP studies. For example, recent studies examined HHC clinical notes for presence of six common symptoms (i.e., pain, fatigue, sleeping difficulty, breathing difficulty, depressed mood, and anxiety) and found that two thirds of HHC episodes had one of these symptoms (Patel et al., 2019; Topaz et al., 2021). As the Omaha System implemented in this study included the above six symptom categories and many additional problems, our study expands the previous findings. The presence of problems in many HHC episodes is also an expected clinical finding; to be eligible for HHC services, patients need to present one or more problems that require intervention by skilled HHC providers. Concerning signs/symptoms were more frequently documented in visit notes than in care coordination notes. This is an expected difference; visit notes often include information about the patient’s signs and symptoms (the focus of this study), while care coordination notes often describe issues related to care management, communication, and care plans.

Among the Omaha System problems, pain was most frequently documented, followed by neuro-musculoskeletal function, circulation, mental health, and communicable/infectious conditions. The prevalence of problems noted in our study was similar to that of other studies. For example, other studies report that one half of a community-dwelling elderly population experienced acute/chronic pain (Hunt et al., 2015; Patel et al., 2019); our findings show a similar percentage (48.3%). Similarly, our results suggest that mental health issues, including fatigue, anxiety, and depressed mood, were documented for over one third of HHC episodes. This is consistent with the rate in the community-dwelling elderly who suffered from those issues, as reported in other studies (de Rekeneire et al., 2014; Zengarini et al., 2015). On the other hand, problems of sexuality, neighborhood/workplace safety, and sanitation were rarely documented. This study was limited to one agency; perhaps HHC clinicians in other geographic locations would encounter different concerns. This finding needs to be explored further using qualitative and quantitative methods.

Our findings can inform several further studies. First, we plan to conduct a large retrospective study that will examine the association between the documentation of risk factors—including the type of concerning signs/symptoms, along with the combination of and intensity of the documented concerning signs/symptoms—and health outcomes (i.e., hospitalization or ED visits in HHC). Our early results in other studies support strong associations between documented symptoms and patient outcomes in HHC (Topaz et al., 2020a, 2021; Wang et al., 2019). We expect to find similar correlations with concerning signs/symptoms. Our NLP system can also be used as a foundation for further development of an early warning system. Specifically, the suggested early warning system will automatically scan HHC clinical notes to identify concerning signs/symptoms. Once concerning signs/symptoms are identified, they will be communicated in automated electronic alerts with the HHC nurse who is managing the patient and potentially HHC clinical managers or primary care providers when necessary. As shown in hospital settings (Lee et al., 2020), alerts generated by such early warning systems can help provide timely interventions to reduce patients’ risk for negative outcomes.

Lastly, recent studies (Moy et al., 2021), including in HHC (Hobensack et al., 2021), show that clinicians experience high documentation burden with up to 50% of clinical time spent on documentation. Some documentation is redundant as some concerning signs/symptoms are documented in a standardized form (e.g., drop-down list of several types of patient’s breathing issues) and in clinical notes. Our results show that HHC clinical notes are a rich source for concerning signs/symptoms. Hypothetically, some redundant standardized form documentation can be removed or shortened to reduce clinical documentation burden. However, further research is needed to understand the completeness of different documentation types (e.g., standardized vs. clinical notes) and to examine the amount of time it takes to document using one type of documentation versus the other.

Limitations

This study has several limitations. First, we identified the signs/symptoms of the Omaha System indicative of the risk for hospitalization and ED visits in HHC through expert reviews. Therefore, additional external validation is required to integrate the classification into a real-world clinical decision support system (e.g., early warning score system). Second, as this study was conducted using data from a single organization, there is a possibility that organization-specific clinical language and jargon patterns limit the generalizability of the NLP algorithms. Third, the data were collected in 2015–2017, which may not appropriately reflect documentation trends in more recent data.

Conclusion

This study is the first to develop and test an NLP algorithm to identify concerning concepts related to the risk of hospitalization or ED visits in HHC using standardized health terminology (Omaha System). Our findings suggest that information concerning signs and symptoms that may indicate the risk of hospitalization or ED visits was frequently documented in narrative clinical notes. NLP can extract information from narrative clinical notes to improve our understanding of care needs in HHC. The study findings provide a foundation for developing an early warning scoring system to help reduce adverse outcomes in HHC.

Supplementary Material

JS_revision_Supplemental Data File

Acknowledgements

This study was funded by Agency for Healthcare Research and Quality [AHRQ] (R01 HS027742), “Building risk models for preventable hospitalizations and emergency department visits in homecare (Homecare-CONCERN).” The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Ms. Hobensack is supported by the National Institute for Nursing Research training grant Reducing Health Disparities through Informatics (RHeaDI) (T32NR007969) as a predoctoral trainee. Ms. Kennedy is supported by the National Institute of Nursing Research Ruth L. Kirschstein National Research Service Award training program Individualized Care for at Risk Older Adults (T32NR009356) as a predoctoral trainee.

Footnotes

All authors report no conflicts of interest relevant to this article.

Ethical Conduct of Research

This study was approved by the Columbia University and Visiting Nurse Service of New York Institutional Review Boards.

Clinical Trial Registration

Not applicable

Contributor Information

Jiyoun Song, Columbia University School of Nursing, New York City, NY.

Marietta Ojo, Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York, NY.

Kathryn H. Bowles, Professor and vanAmeringen Chair in Nursing Excellence, University of Pennsylvania School of Nursing, Department of Biobehavioral Health Sciences, Philadelphia, PA; Vice President and Director of the Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York City, NY.

Margaret V. McDonald, Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York, NY.

Kenrick Cato, Columbia University School of Nursing, New York City, NY; Emergency Medicine, Columbia University Irving Medical Center, New York, NY.

Sarah Collins Rossetti, Columbia University School of Nursing, New York City, NY; Columbia University, Department of Biomedical Informatics, New York City, NY.

Victoria Adams, Quality Care Management, Visiting Nurse Service of New York, New York, NY.

Sena Chae, College of Nursing, University of Iowa, Iowa City, IA.

Mollie Hobensack, Columbia University School of Nursing, New York City, NY.

Erin Kennedy, University of Pennsylvania School of Nursing, Department of Biobehavioral Health Sciences, Philadelphia, PA.

Aluem Tark, College of Nursing, University of Iowa, Iowa City, IA.

Min-Jeoung Kang, College of Nursing, Catholic University of Korea, Seoul, Korea.

Kyungmi Woo, The Research Institute of Nursing Science, College of Nursing, Seoul National University, Seoul, Korea.

Yolanda Barrón, Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York, NY.

Sridevi Sridharan, Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York, NY.

Maxim Topaz, Columbia University School of Nursing, New York City, NY; Data Science Institute, Columbia University, New York City, NY; Center for Home Care Policy & Research, Visiting Nurse Service of New York, New York City, NY.

References

  1. Agency for Healthcare Research and Quality. (2018). Potentially avoidable hospitalizations. Retrieved April 5, 2021, from https://www.ahrq.gov/research/findings/nhqrdr/chartbooks/carecoordination/measure3.html
  2. Bjarnadottir RI, Martin CL, Maganti S, Hoffman M, & Monsen KA (2020). Discovering the value of the Omaha System for knowledge representation and data classification: A semantics-powered text mining case study. SAGE Publications. 10.4135/9781529730555 [DOI] [Google Scholar]
  3. Centers for Medicare & Medicaid Services. (2019). Home health quality measures. Retrieved April 5, 2021, from https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HomeHealthQualityInits/Home-Health-Quality-Measures.html
  4. Collins SA, Cato K, Albers D, Scott K, Stetson PD, Bakken S, & Vawdrey DK (2013). Relationship between nursing documentation and patients’ mortality. American Journal of Critical Care, 22, 306–313. 10.4037/ajcc2013426 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. de Rekeneire N, Leo-Summers L, Han L, & Gill TM (2014). Epidemiology of restricting fatigue in older adults: The precipitating events project. Journal of the American Geriatrics Society, 62, 476–481. 10.1111/jgs.12685 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Fu L-H, Schwartz J, Moy A, Knaplund C, Kang M-J, Schnock KO, Garcia JP, Jia H, Dykes PC, Cato K, Albers D, & Rossetti SC (2020). Development and validation of early warning score system: A systematic literature review. Journal of Biomedical Informatics, 105, 103410. 10.1016/j.jbi.2020.103410 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Han SJ, Kim HK, Storfjell J, & Kim MJ (2013). Clinical outcomes and quality of life of home health care patients. Asian Nursing Research, 7, 53–60. 10.1016/j.anr.2013.03.002 [DOI] [PubMed] [Google Scholar]
  8. Hardin L, & Mason DJ (2019). Bringing it home: The shift in where health care is delivered. JAMA, 322, 493–494. 10.1001/jama.2019.11302 [DOI] [PubMed] [Google Scholar]
  9. Hobensack M, Ojo M, Bowles K, McDonald M, Song J, & Topaz M (2021). Home healthcare clinicians’ perspective on electronic health records: A qualitative study. Studies in Health Technology and Informatics, 284, 426–430. 10.3233/SHTI210763 [DOI] [PubMed] [Google Scholar]
  10. Howard J, Kent T, Stuck A, Crowley C, & Zeng FEN (2019). Improved cost and utilization among Medicare beneficiaries dispositioned from the ED to receive home health care compared with inpatient hospitalization. American Journal of Accountable Care, 7, 10–16. [Google Scholar]
  11. Hunt LJ, Covinsky KE, Yaffe K, Stephens CE, Miao Y, Boscardin WJ, & Smith AK (2015). Pain in community-dwelling older adults with dementia: Results from the National Health and Aging Trends Study. Journal of the American Geriatrics Society, 63, 1503–1511. 10.1111/jgs.13536 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Jarvis WR (2001). Infection control and changing health-care delivery systems. Emerging Infectious Diseases, 7, 170–173. https://wwwnc.cdc.gov/eid/article/7/2/70-0170_article [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Kang M-J, Dykes PC, Korach TZ, Zhou L, Schnock KO, Thate J, Whalen K, Jia H, Schwartz J, Garcia JP, Knaplund C, Cato KD, & Rossetti SC (2020). Identifying nurses’ concern concepts about patient deterioration using a standard nursing terminology. International Journal of Medical Informatics, 133, 104016. 10.1016/j.ijmedinf.2019.104016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Koleck TA, Tatonetti NP, Bakken S, Mitha S, Henderson MM, George M, Miaskowski C, Smaldone A, & Topaz M (2021). Identifying symptom information in clinical notes using natural language processing. Nursing Research, 70, 173–183. 10.1097/NNR.0000000000000488 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Landers S, Madigan E, Leff B, Rosati RJ, McCann BA, Hornbake R, MacMillan R, Jones K, Bowles K, Dowding D, Lee T, Moorhead T, Rodriguez S, & Breese E (2016). The future of home health care: A strategic framework for optimizing value. Home Health Care Management & Practice, 28, 262–278. 10.1177/1084822316666368 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Le Glaz A, Haralambous Y, Kim-Dufor D-H, Lenca P, Billot R, Ryan TC, Marsh J, DeVylder J, Walter M, Berrouiguet S, & Lemey C (2021). Machine learning and natural language processing in mental health: Systematic review. Journal of Medical Internet Research, 23, e15708. 10.2196/15708 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Lee J-R, Kim E-M, Kim S-A, & Oh EG (2020). A systematic review of early warning systems’ effects on nurses’ clinical performance and adverse events among deteriorating ward patients. Journal of Patient Safety, 16, e104–e113. 10.1097/pts.0000000000000492 [DOI] [PubMed] [Google Scholar]
  18. Ma C, Shang J, Miner S, Lennox L, & Squires A (2018). The prevalence, reasons, and risk factors for hospital readmissions among home health care patients: A systematic review. Home Health Care Management & Practice, 30, 83–92. 10.1177/1084822317741622 [DOI] [Google Scholar]
  19. Martin KS (2005). The Omaha System: A key to practice, documentation, and information management. Elsevier Saunders. [Google Scholar]
  20. McHugh ML (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22, 276–282. [PMC free article] [PubMed] [Google Scholar]
  21. Medicare Payment Advisory Commission. (2019, March). Report to the Congress: Medicare payment policy. Retrieved from https://www.medpac.gov/wp-content/uploads/import_data/scrape_files/docs/default-source/reports/mar19_medpac_entirereport_sec_rev.pdf
  22. Mikolov T, Sutskever I, Chen K, Corrado G, & Dean J (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 3111–3119. Retrieved April 20, 2021, from https://papers.nips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf [Google Scholar]
  23. Mitzner TL, Beer JM, McBride SE, Rogers WA, & Fisk AD (2009). Older adults’ needs for home health care and the potential for human factors interventions. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53, 718–722. SAGE Publications. 10.1177/154193120905301118 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Monsen KA (2018). The Omaha system as an ontology and meta-model for nursing and healthcare in an era of Big Data. Kontakt, 20, e109–e110. 10.1016/j.kontakt.2018.03.001 [DOI] [Google Scholar]
  25. Monsen KA, Maganti S, Giaquinto RA, Mathiason MA, Bjarnadottir RI, & Kreitzer MJ (2018). Use of the Omaha System for ontology-based text mining to discover meaning within CaringBridge social media journals. Kontakt, 20, e210–e216. 10.1016/j.kontakt.2018.03.002 [DOI] [Google Scholar]
  26. Monsen KA, Swanberg HL, Oancea SC, & Westra BL (2012). Exploring the value of clinical data standards to predict hospitalization of home care patients. Applied Clinical Informatics, 3, 419–436. 10.4338/aci-2012-05-ra-0016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Morganti KG, Bauhoff S, Blanchard JC, Abir M, Iyer N, Smith A, Vesely JV, Okeke EN, & Kellermann AL (2013). The evolving role of emergency departments in the United States. Rand Health Quarterly, 3, 3. [PMC free article] [PubMed] [Google Scholar]
  28. Moy AJ, Schwartz JM, Chen R, Sadri S, Lucas E, Cato KD, & Rossetti SC (2021). Measurement of clinical documentation burden among physicians and nurses using electronic health records: A scoping review. Journal of the American Medical Informatics Association, 28, 998–1008. 10.1093/jamia/ocaa325 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Navathe AS, Zhong F, Lei VJ, Chang FY, Sordo M, Topaz M, Navathe SB, Rocha RA, & Zhou L (2018). Hospital readmission and social risk factors identified from physician notes. Health Services Research, 53, 1110–1136. 10.1111/1475-6773.12670 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Patel KV, Guralnik JM, Phelan EA, Gell NM, Wallace RB, Sullivan MD, & Turk DC (2019). Symptom burden among community-dwelling older adults in the United States. Journal of the American Geriatrics Society, 67, 223–231. 10.1111/jgs.15673 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Sinha PK, Sunder G, Bendale P, Mantri M, & Dande A (2013). Unified medical language system. In Electronic health record: Standards, coding systems, frameworks, and infrastructures (pp. 145–152). John Wiley & Sons. 10.1002/9781118479612.ch16 [DOI] [Google Scholar]
  32. Song J, Woo K, Shang J, Ojo M, & Topaz M (2021). Predictive risk models for wound infection-related hospitalization or ED visits in home health care using machine-learning lgorithms. Advances in Skin & Wound Care, 34, 1–12. 10.1097/01.Asw.0000755928.30524.22 [DOI] [PubMed] [Google Scholar]
  33. Sorin V, Barash Y, Konen E, & Klang E (2020). Deep learning for natural language processing in radiology—Fundamentals and a systematic review. Journal of the American College of Radiology, 17, 639–648. 10.1016/j.jacr.2019.12.026 [DOI] [PubMed] [Google Scholar]
  34. Topaz M, Adams V, Wilson P, Woo K, & Ryvicker M (2020a). Free-text documentation of dementia symptoms in home healthcare: A natural language processing study. Gerontology and Geriatric Medicine, 6, 2333721420959861. 10.1177/2333721420959861 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Topaz M, Koleck TA, Onorato N, Smaldone A, & Bakken S (2021). Nursing documentation of symptoms is associated with higher risk of emergency department visits and hospitalizations in homecare patients. Nursing Outlook, 69, 435–446. 10.1016/j.outlook.2020.12.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Topaz M, Murga L, Bar-Bachar O, McDonald M, & Bowles K (2019). NimbleMiner: An open-source nursing-sensitive natural language processing system based on word embedding. CIN: Computers, Informatics, Nursing, 37, 583–590. 10.1097/cin.0000000000000557 [DOI] [PubMed] [Google Scholar]
  37. Topaz M, Woo K, Ryvicker M, Zolnoori M, & Cato K (2020b). Home healthcare clinical notes predict patient hospitalization and emergency department visits. Nursing Research, 69, 448–454. 10.1097/nnr.0000000000000470 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Wang Y, Sohn S, Liu S, Shen F, Wang L, Atkinson EJ, Amin S, & Liu H (2019). A clinical text classification paradigm using weak supervision and deep representation. BMC Medical Informatics Decision Making, 19, 1. 10.1186/s12911-018-0723-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Weeks LE, Macdonald M, Martin-Misener R, Helwig M, Bishop A, Iduye DF, & Moody E (2018). The impact of transitional care programs on health services utilization in community-dwelling older adults: A systematic review. JBI Database of Systematic Reviews and Implementation Reports, 16, 345–384. 10.11124/jbisrir-2017-003486 [DOI] [PubMed] [Google Scholar]
  40. Weng W-H, Wagholikar KB, McCray AT, Szolovits P, & Chueh HC (2017). Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach. BMC Medical Informatics and Decision Making, 17, 155. 10.1186/s12911-017-0556-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Zengarini E, Ruggiero C, Pérez-Zepeda MU, Hoogendijk EO, Vellas B, Mecocci P, & Cesari M (2015). Fatigue: Relevance and implications in the aging population. Experimental Gerontology, 70, 78–83. 10.1016/j.exger.2015.07.011 [DOI] [PubMed] [Google Scholar]
  42. Zhou L, Baughman AW, Lei VJ, Lai KH, Navathe AS, Chang F, Sordo M, Topaz M, Zhong F, Murrali M, Navathe S, & Rocha RA (2015). Identifying patients with depression using free-text clinical documents. Studies in Health Technology and Informatics, 216, 629–633. 10.3233/978-1-61499-564-7-629 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

JS_revision_Supplemental Data File
NIHMS1776783-supplement-JS_revision_Supplemental_Data_File.docx (172.6KB, docx)

RESOURCES