Abstract
The aim of this article was to describe a novel methodology for transforming complex nursing care plan data into meaningful variables to assess the impact of nursing care. We extracted standardized care plan data for older adults from the electronic health records of 4 hospitals. We created a palliative care framework with 8 categories. A subset of the data was manually classified under the framework, which was then used to train random forest machine learning algorithms that performed automated classification. Two expert raters achieved a 78% agreement rate. Random forest classifiers trained using the expert consensus achieved accuracy (agreement with consensus) between 77% and 89%. The best classifier was utilized for the automated classification of the remaining data. Utilizing machine learning reduces the cost of transforming raw data into representative constructs that can be used in research and practice to understand the essence of nursing specialty care, such as palliative care.
Keywords: nursing, nursing informatics, electronic health records, standardized nursing terminology, machine learning
INTRODUCTION
Hospitalized patients are primarily cared for by nurses, who ensure the interdisciplinary team’s desired outcomes are achieved. Given their key role as front-line providers, nurses routinely capture a large volume of patient data in their documentation in electronic health records (EHR).1–3 Nurses’ care plans are a particularly rich source of documented information that dynamically captures the judgments and decisions nurses make about a patient’s care and the outcomes achieved. The challenge is to convert data gathered for a specific purpose, such as care plans, into a representation that parsimoniously and holistically conveys the meaning of the data.
In recent years, there has been an increasing emphasis on the standardization of EHR data to enable their use and reuse to answer important research questions related to care quality and costs, productivity, and patient outcomes.1,2 Standardization typically requires that data be coded and structured using standards and terminologies to allow interoperability.4 Though the availability of standardized data makes synthesizing a large set of diverse elements collected in the EHR (eg, medical diagnosis, procedures data, laboratory results, and nursing data elements) possible, it is also necessary to develop methods that support the analysis and presentation of these data in valid and easily understood formats.5–7 In this study, we describe an innovative methodology, using palliative care as an exemplar, to transform EHR-standardized care plan data into variables that allow us to answer important research questions about nursing care. The transformed variables described were ultimately used in a subsequent study published elsewhere that examined the relationship between palliative care provided by nurses to older adults with and without cognitive impairment and the impact of such care on hospital length of stay.8 There is a compelling need to better understand the impact of palliative care particularly for patients nearing the end of life when the cost of health care soars and the quality of life diminishes.
MATERIALS AND METHODS
Care plan data
From a larger dataset of standardized care plan data collected over a 3-year period on 9 medical-surgical adult units located in 4 Midwestern hospitals in the United States,9 we extracted deidentified admission nursing care plans for 4354 hospitalized older adults (aged 85 and older). A care plan consists of nursing diagnoses, interventions, and outcomes (DIOs) pertinent for moving a patient to discharge (see Figure 1). Each nursing diagnosis is linked to at least one outcome, which in turn is linked to at least one intervention, coded, respectively, with NANDA International,10 Nursing Outcomes Classification,11 and Nursing Interventions Classification,12 commonly known as a set (NNN). The combination of one nursing diagnosis, one intervention, and one outcome is referred to as a DIO, the basic care plan unit. Each care plan typically contains multiple DIOs that collectively represent the nurses’ foci of care, the treatments they provide, and the outcomes being monitored.
Figure 1.
An example of a care plan documented by nurses in the HANDS system and 4 combinations of nursing diagnoses, interventions, and outcomes (DIOs) from it. The blue squares represent nursing diagnoses (patients’ problems), the green circle is nursing outcomes (patients’ goals), and the purple triangles are the nursing interventions. The dotted lines show the linkages (combinations) among the care plan data elements. The numbering between parenthesis shows the current and expected nursing outcome ratings. The data elements with strikethrough lines are the ones no longer active in the care plan. MAR, medical record; q, every; NOC, Nursing Outcomes Classification, for nursing outcomes; NIC, Nursing Interventions Classification, for nursing interventions. Copyright © 2021 HANDS Research Team.
Figure 1 is an illustration of a nursing care plan that contains 9 unique DIOs. The first 4 DIOs are outlined in the figure. A summary interpretation of the meaning of the 4 DIOs is as follows: nurses are performing 4 different nursing interventions to address the patient’s Death Anxiety (nursing diagnosis). Two of the nursing interventions are directed at reducing the patient’s Anxiety Level (nursing outcome) while 2 other nursing interventions are directed at enhancing the patient’s Hope (nursing outcome).
Palliative care framework
We developed our palliative care framework for the purpose of mapping care plan data of older adults into parsimonious variables fully representing the nursing palliative care provided. It was derived from 3 sources: (1) the 8 domains described in the National Consensus Project (NCP) Clinical Practice Guidelines for Quality Palliative Care,13 (2) Ferrell’s 8 domains for end-of-life care,14 and (3) von Krogh, Dale, and Nåden’s framework for the integration of standardized nursing terminologies in EHRs.15 Our palliative care framework consists of 8 categories that comprehensively cover the types of nursing care an older patient with a life-limiting illness would receive: (1) family, (2) well-being, (3) mental comfort, (4) physical comfort, (5) mental, (6) safety, (7) functional, and (8) physiological.8 The definitions of these 8 categories and rules for mapping DIOs into these categories can be found in Figure 2.
Figure 2.
Novel palliative care framework for the classification of unique nursing diagnoses, interventions, and outcomes (DIOs) combinations found in care plan data documented by nurses extracted from electronic health records.8
Manual classification of a DIO subset
Using the palliative care framework and rules of classification, 2 experts in standardized terminologies and nursing data (TGRM and GMK) independently classified a subset of unique DIOs into 1 of the 8 framework categories. Disagreements were resolved through a consensus process.
Machine learning
The manually classified DIOs were used to train machine learning classifiers. We utilized random forest classifiers, which are tree-based classifiers that successively partition feature space and are capable of capturing complex interaction structures within the data. In our previous work16,17 applying various predictive modeling approaches (decision trees, k-nearest neighbors, support vector machines, Näive-Bayes, and logistic regression) to end-of-life patients’ care plans, we found that decision trees consistently out-performed the other strategies in predicting patient outcomes from care plans. Their performance, however, can be limited by noisiness. The random forest approach reduces the variance of basic tree classifiers by averaging over a large number of trees and is among the best-performing methods for small-to-medium-sized datasets.18
The size of our training dataset was small, and it was necessary to preprocess the nursing diagnosis, interventions, and outcomes to reduce the dimensions of the feature space for machine learning to perform well. For this, we leveraged the 3-level structures (domain-class-concept) of the NNN terminologies. Each domain consists of multiple classes, which in turn consists of multiple concepts (nursing DIOs; see Figure 3 for an NANDA International terminology example). These structures were systematically developed for purpose of organizing the terms and reflecting the similarities and differences among them. In general, the finer the granularity of the input feature (concept > class > domain), the higher the ceiling of performance but also the larger training dataset needed to approach the ceiling. With our limited dataset, we explored using domain and class as features for our classifiers.
Figure 3.
An example of one domain from the 2018–2020 NANDA International terminology version, its classes, and the concepts (ie, nursing diagnoses) within each class.
In addition, based on their expertise with terminologies and our work with other modeling strategies,16,17 our nursing experts identified 3 keywords (family, self-care assistance, and monitoring) as potential features. Therefore, we assessed the performance of 4 random forest classifiers utilizing the following sets of predictors respectively: (1) domains from NANDA International (n = 12), Nursing Outcomes Classification (n = 7), and Nursing Interventions Classification (n = 7), (2) domains + keywords, (3) consolidated classes from NANDA International (n = 15), Nursing Outcomes Classification (n = 21), and Nursing Interventions Classification (n = 21), and (4) consolidated classes + keywords.
Note that our dataset contained 37 NANDA International classes, 31 Nursing Outcomes Classification classes, and 28 Nursing Interventions Classification classes. The consolidated classes were obtained by grouping the classes not frequently present into the category “other” to further reduce the dimensionality of our feature space with minimal information loss.
To assess the performance of classifiers, we randomly split the nursing DIOs manually classified by the experts into a training and a validation set using a standard 2/1 training/validation ratio19 with stratification to balance the distribution of classes.20 Utilizing the randomForest package with the default hyperparameter,21 we trained the classifiers on the training set and then applied them to the validation set for comparison with the human consensus. This process was repeated with 100 different random splits and the means and standard deviations of the performance metrics (accuracy, precision, recall, and f-measure) were obtained. All analyses were performed using R, version 3.6.3.22
RESULTS
We identified 3277 unique DIOs in our dataset, consisting of 93 unique nursing diagnoses, 191 unique outcomes, and 286 unique interventions. Two expert raters manually classified a subset of 1000 DIOs and achieved 78% agreement (Kappa = 0.70).
Model performance
Table 1 contains descriptive statistics (mean and SD) of the performance metrics for all 4 classifiers. Model 1 (domains) has an overall accuracy (percentage of agreement with the expert consensus) of around 77% (SD = 1%), Model 2 (domains + keywords) 85% (SD = 1%), Model 3 (classes) 83% (SD = 1%), and Model 4 (classes + keywords) 89% (SD = 1%).
Table 1.
Classification performance for the random forest classifiersa
| Classifiers | Measures | Palliative care categories |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Family | Well-being | Mental comfort | Physical comfort | Mental | Safety | Functional | Physiological | Overall | ||
| Model 1: NNN Domains | Accuracy | 96% (4%) | 64% (7%) | 88% (4%) | 92% (2%) | 89% (3%) | 93% (2%) | 72% (2%) | 83% (1%) | 77% (1%) |
| Precision | 95% (5%) | 94% (23%) | 73% (7%) | 93% (3%) | 85% (5%) | 79% (5%) | 75% (4%) | 72% (2%) | ||
| Recall | 92% (8%) | 29% (15%) | 79% (8%) | 86% (4%) | 81% (6%) | 88% (4%) | 52% (4%) | 90% (2%) | ||
| F-measure | 93% (5%) | 43% (18%) | 75% (6%) | 89% (3%) | 83% (4%) | 83% (3%) | 62% (3%) | 80% (1%) | ||
| Model 2: NNN Domains + keywordsb | Accuracy | 96% (4%) | 61% (7%) | 90% (4%) | 93% (2%) | 90% (3%) | 94% (2%) | 86% (1%) | 90% (1%) | 85% (1%) |
| Precision | 100% | 86% (34%) | 75% (6%) | 94% (3%) | 85% (5%) | 79% (5%) | 82% (2%) | 86% (2%) | ||
| Recall | 92% (8%) | 23% (14%) | 82% (8%) | 86% (4%) | 81% (6%) | 91% (4%) | 79% (3%) | 90% (2%) | ||
| F-measure | 95% (4%) | 36% (19%) | 78% (5%) | 90% (3%) | 83% (4%) | 84% (3%) | 81% (2%) | 88% (1%) | ||
| Model 3: NNN Classesc | Accuracy | 96% (4%) | 73% (8%) | 96% (2%) | 97% (1%) | 94% (2%) | 95% (2%) | 82% (2%) | 86% (1%) | 83% (1%) |
| Precision | 92% (7%) | 100% | 79% (6%) | 96% (2%) | 96% (2%) | 82% (5%) | 76% (3%) | 81% (2%) | ||
| Recall | 92% (8%) | 47% (17%) | 95% (3%) | 95% (2%) | 90% (4%) | 92% (4%) | 73% (4%) | 83% (2%) | ||
| F-measure | 92% (6%) | 62% (16%) | 86% (4%) | 95% (1%) | 93% (2%) | 86% (3%) | 74% (3%) | 82% (2%) | ||
| Model 4: NNN Classes + keywords | Accuracy | 94% (4%) | 74% (7%) | 98% (1%) | 97% (1%) | 96% (2%) | 95% (1%) | 89% (1%) | 92% (1%) | 89% (1%) |
| Precision | 100% | 100% | 77% (6%) | 97% (1%) | 98% (1%) | 82% (5%) | 87% (2%) | 89% (1%) | ||
| Recall | 88% (9%) | 48% (15%) | 97% (2%) | 95% (2%) | 92% (4%) | 93% (3%) | 83% (3%) | 91% (2%) | ||
| F-measure | 93% (5%) | 63% (14%) | 86% (4%) | 96% (1%) | 95% (2%) | 87% (3%) | 85% (2%) | 90% (1%) | ||
Random splitting of the 1000 linkages was repeated 100 times and the percentages of the runs were aggregated to generate the mean and standard deviation of the computer classifiers.
Keywords: Family, Self-Care Assistance, Monitoring.
Thirty-seven NANDA-International (for nursing diagnoses) classes were consolidated into 15, 31 Nursing Outcomes Classification (for outcomes) classes were consolidated into 21, and 28 Nursing Interventions Classification (for nursing interventions) classes were consolidated into 21 categories.
NNN: NANDA International, Nursing Interventions Classification, Nursing Outcomes Classification.
The accuracy was lowest in the well-being category, with the classifiers agreeing with the experts only 64–74% of the time on whether a DIO signifies well-being care. Close examination reveals that the classifiers achieved good precision (86–100%), meaning that the DIOs classified by the computer as well-being were classified by the experts as such vast majority of the time. On the other hand, all classifiers had a poor recall (23–48%), meaning that they missed over half of the DIOs classified as well-being by the experts.
Of the 4 classifiers, Model 4 (classes + keywords) consistently performed best, achieving 89% (SD = 1%) overall accuracy and except for the category of well-being, achieving good precision and recall. Note that 2 independent classifiers with 89% accuracy would agree with each other 79% of the time, comparable to the agreement rate (78%) of the experts.
Application to remaining data
Model 4, trained with all 1000 manually classified DIOs, was then applied to the remaining 2277 DIOs. In Table 2, we show the distribution of the DIOs among the 8 palliative care categories.
Table 2.
Distribution of classifier outputs
| First 1000 (human), % | First 1000 (computer), % | Remaining 2277 (computer), % | |
|---|---|---|---|
| Family | 3 | 3 | 3 |
| Well-being | 2 | 1 | 1 |
| Mental comfort | 5 | 6 | 9 |
| Physical comfort | 10 | 10 | 11 |
| Mental | 8 | 7 | 6 |
| Safety | 8 | 10 | 7 |
| Functional | 27 | 25 | 27 |
| Physiological | 37 | 38 | 36 |
DISCUSSION
We utilized machine learning for the automated transformation of standardized nursing care plan data into palliative care variables, leveraging the hierarchical structures (domain and classes) of standardized nursing terminologies. The transformed data can be easily interpreted by clinicians and used in research and practice to provide insight for the care of populations in great need, such as older adults with life-limiting illnesses. The machine learning classifiers achieved high performance, indicating that this approach has the potential to reduce the cost of the important task of mapping care plan and other EHR data.
In a study published elsewhere,8 we utilized these transformed variables to find that older adults with cognitive impairment were undertreated for symptoms of physical suffering (eg, pain, nausea) and to find an association between cognitive impairment and increased hospital length of stay. These results are a practical representation of the knowledge generated through the analysis of meaningful variables. The application of the machine learning algorithm and palliative care framework developed in this study helped transform a complex dataset into meaningful and parsimonious representation,23 which when analyzed returned actionable, reliable, and clinically valid research results.24
A limitation of our models was the classification of the nursing DIOs combinations into the well-being category. We observed that the 4 models achieved a moderate accuracy and had poor recall performance for this palliative care category. A closer examination of the subset with 1000 nursing DIOs used in the training and testing of the models revealed that only 20 of the DIOs were classified by the experts as well-being. This finding is consistent with the reality of current nursing practice, which is driven by a medical model mostly focused on addressing time-limited health problems,25,26 failing to provide holistic care.8 To improve classifier performance in this category, future studies can consider manual selection of additional features relevant to this category as well as oversampling of this category during the construction of the training dataset.
A strength of our approach is its reproducibility. Our approach is applicable to EHR-care plan data documented with any of the Unified Medical Language System (UMLS)27 compatible nursing terminologies that have similar 3-level taxonomic structure. In this study, we have focused on admissions care plans, but the methodology can also be applied to longitudinal care plan data to examine nursing care delivered from admission to hospital discharge.
CONCLUSION
The application of the machine learning algorithm built in this study reduced the time and financial cost of the classification task. In addition, the strength of the machine learning algorithm developed is its reproducibility. The method described offers a viable strategy for transforming standardized nursing care plan data into meaningful representation useful in examining important research questions pertaining to the impact of nursing care. Our approach can be used to transform care plan data coded with other classification systems and also adapted to transform similarly structured EHR data (such as longitudinal observations).
FUNDING
This work was supported by the National Institutes of Health (NIH), National Institute of Nursing Research (NINR) grant number R01NR018416 and the National Institute on Aging (NIA) grant number R21AG072265. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH, NINR, or NIA. The final peer-reviewed manuscript is subject to the NIH Public Access Policy.
AUTHOR CONTRIBUTIONS
TGRM, GMK, and YY conceived of the main conceptual idea. TGRM and GMK conducted the manual classification of the data. TGRM and YY applied the machine learning modeling. TGRM wrote the paper, with all authors contributing significant edits to all versions of the work. All authors approved the final version to be published.
CONFLICT OF INTEREST STATEMENT
None declared.
DATA AVAILABILITY
The data underlying this article will be shared on reasonable request to the corresponding author.
REFERENCES
- 1. Vuokko R, Mäkelä-Bengs P, Hyppönen H, et al. Impacts of structuring the electronic health record: results of a systematic literature review from the perspective of secondary use of patient data. Int J Med Inform 2017; 97: 293–303. [DOI] [PubMed] [Google Scholar]
- 2. Macieira TGR, Chianca TCM, Smith MB, et al. Secondary use of standardized nursing care data for advancing nursing science and practice: a systematic review. J Am Med Inform Assoc 2019; 26 (11): 1401–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Fennelly O, Grogan L, Reed A, et al. Use of standardized terminologies in clinical practice: a scoping review. Int J Med Inform 2021; 149: 104431. [DOI] [PubMed] [Google Scholar]
- 4.Healthcare Information and Management Systems Society. Interoperability in healthcare. https://www.himss.org/resources/interoperability-healthcare Accessed August 22, 2021.
- 5. Navar AM. Electronic health record data quality issues are not remedied by increasing granularity of diagnosis codes. JAMA Cardiol 2019; 4 (5): 465. [DOI] [PubMed] [Google Scholar]
- 6. Cirillo D, Núñez-Carpintero I, Valencia A.. Artificial intelligence in cancer research: learning at different levels of data granularity. Mol Oncol 2021; 15 (4): 817–29. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Häyrinen K, Saranto K, Nykänen P.. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008; 77 (5): 291–304. [DOI] [PubMed] [Google Scholar]
- 8. Macieira TGR, Yao Y, Smith MB, et al. Nursing care for hospitalized older adults with and without cognitive impairment. Nurs Res 2020; 69 (2): 116–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Keenan G, Yakel E, Yao Y, et al. Maintaining a consistent big picture: meaningful use of a web-based POC EHR system. Int J Nurs Knowl 2021; 23 (3): 119–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Herdman TH, Kamitsuru S, eds. NANDA International Nursing Diagnoses: Definitions & Classification, 2018–2020. 11th ed. New York, NY: Thieme; 2018. [Google Scholar]
- 11. Moorhead S, Johnson M, Maas M, Swanson E, eds. Nursing Outcomes Classification (NOC). 6th ed. St. Louis, MO: Elsevier; 2018. [Google Scholar]
- 12. Butcher H, Bulechek GM, Dochterman JMM, Wagner C, eds. Nursing Interventions Classification (NIC). 7th ed. St. Louis, MO: Elsevier; 2018. [Google Scholar]
- 13.National Consensus Project for Quality Palliative Care. Clinical practice guidelines for quality palliative care (3rd ed.). 2013. https://www.nationalcoalitionhpc.org/ncp- guidelines-2013/ Accessed March 4, 2019.
- 14. Ferrell BR. Overview of the domains of variables relevant to end-of-life care. J Palliat Med 2005; 8 (Suppl. 1): S22–9. [DOI] [PubMed] [Google Scholar]
- 15. von Krogh G, Dale C, Nåden D.. A framework for integrating NANDA, NIC, and NOC terminology in electronic patient records. J Nurs Scholarsh 2005; 37 (3): 275–81. [DOI] [PubMed] [Google Scholar]
- 16. Lodhi MK, Ansari R, Yao Y, et al. Predictive modeling for comfortable death outcome using electronic health records. Proc IEEE Int Congr Big Data 2015; 2015: 409–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Lodhi MK, Stifter J, Yao Y, et al. Predictive modeling for end-of-life pain outcome using electronic health records. Adv Data Min 2015; 9165: 56–68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Hastie T, Tibshirani R, Friedman J.. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York, NY: Springer Science & Business Media; 2009. [Google Scholar]
- 19. Raschka S. Model evaluation, model selection, and algorithm selection in machine learning. 2018. https://arxiv.org/abs/1811.12808 Accessed May 17, 2021.
- 20. Kuhn M. caret: classification and regression training. R package version 6.0-88. https://CRAN.R-project.org/package=caret Accessed May 28, 2021.
- 21. Liaw A, Wiener M.. Classification and regression by randomForest. R News 2002; 2 (3): 18–22. https://CRAN.R-project.org/doc/Rnews/ Accessed May 28, 2021. [Google Scholar]
- 22.R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2020. https://www.R-project.org/ Accessed August 3, 2021.
- 23. Ferrão JC, Oliveira MD, Janela F, et al. Preprocessing structured clinical data for predictive modeling and decision support. A roadmap to tackle the challenges. Appl Clin Inform 2016; 7 (4): 1135–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Khodyakov D, Mendoza-Graf A, Berry S, et al. Return of value in the new era of biomedical research-one size will not fit all. AJOB Empir Bioeth 2019; 10 (4): 265–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Calnan M, Tadd W, Calnan S, et al. I often worry about the older person being in that system’: exploring the key influences on the provision of dignified care for older people in acute hospitals. Ageing Soc 2013; 33: 465–85. [Google Scholar]
- 26. de Almeida Tavares JP, da Silva AL, Sá-Couto P, et al. Portuguese nurses’ knowledge of and attitudes toward hospitalized older adults. Scand J Caring Sci 2015; 29 (1): 51–61. [DOI] [PubMed] [Google Scholar]
- 27.National Library of Medicine (U.S.). Metathesaurus. UMLS® Reference Manual [Internet]. http://www.ncbi.nlm.nih.gov/books/NBK9684/ Accessed March 8, 2020.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data underlying this article will be shared on reasonable request to the corresponding author.