Evidence of Progress in Making Nursing Practice Visible Using Standardized Nursing Data: a Systematic Review

Abstract

Nursing care documentation in electronic health records (EHRs) with standardized nursing terminologies (SNTs) can facilitate nursing’s participation in big data science that involves combining and analyzing multiple sources of data. Before merging SNTs data with other sources, it is important to understand how such data are being used and analyzed to support nursing practice. The main purpose of this systematic review was to identify studies using SNTs data, their aims and analytical methods. A two-phase systematic process resulted in inclusion and review of 35 publications. Aims of the studies ranged from describing most popular nursing diagnoses, outcomes, and interventions on a unit to predicting outcomes using multi-site data. Analytical techniques varied as well and included descriptive statistics, correlations, data mining, and predictive modeling. The review underscored the value of developing a deep understanding of the meaning and potential impact of nursing variables before merging with other sources of data.

Introduction

The main frontline providers of care are nurses who also represent the largest category of health workers in the hospital setting. Among the 2.8 million registered nurses currently working in the United States (U.S.), 61% work in hospitals¹ whereas 19% of 297,100² pharmacists and 41.9% of 854,698 physicians in practice work in hospitals.^3,4

Nurses are responsible 24 hours each day for continuously identifying care issues, implementing and adjusting care prescribed by themselves and other providers to achieve desired patient outcomes. To date, however, it has been difficult to effectively evaluate the impact of nursing on patient outcomes. The growing use of electronic health records (EHRs) to document care now offers the opportunity to use the data captured in practice for discovering knowledge to transform health care. Thus, the documentation entered by nurses into EHRs, for the first time ever, is a potential source for discovering the impact of nursing care on patient outcomes and using the knowledge to improve care. In this article, we report our systematic review of studies that utilized nursing EHRs data to answer a variety of research questions from describing nursing care for a specific population to predicting patient outcomes. The publications reviewed provide a foundation for identifying future paths of inquiry involving nursing and other data retrievable from EHRs.

The use of standardized nursing terminologies (SNTs) to document nursing care enables the easy retrieval and analysis of nursing data while also representing the nurse’s clinical reasoning.⁵ The integration of nursing data into large datasets requires the frequent and rapid input of new valid information from EHRs.⁶ These can be achieved through the use of controlled vocabularies in EHRs, which helps overcome the major challenges of aggregation, processing and analysis associated with unstructured text data.^7–8 In nursing, SNTs are controlled vocabularies that represent nursing care as nursing diagnoses, interventions and outcomes.⁸ The SNT coded data retrieved from EHRs can be analyzed alone or merged with other EHRs data. The use of SNTs to document nursing practice is a big step toward supporting the aggregation of nursing data to large datasets and big data science.

Different sets of SNTs are used to document nursing care. The American Nurses Association (ANA) recognizes and supports the use of certain nursing terminologies to guide and document care if those have clear and unambiguous concepts, are coded with a unique identifier per concept, and if those terminologies were tested for reliability, clinical usefulness and validity.⁹ The following nursing terminologies are recognized by ANA: NANDA-International (NANDA-I)¹⁰; Nursing Interventions Classification (NIC)¹¹; Nursing Outcomes Classification (NOC)¹²; International Classification for Nursing Practice (ICNP)¹³; Omaha System¹⁴; Clinical Care Classification (CCC)¹⁵; and the Perioperative Nursing Data Set (PNDS).¹⁶ While ICNP, Omaha System, CCC and PNDS sets contain diagnoses, interventions and outcomes terms; NANDA-I (diagnoses), NIC (interventions) and NOC (outcomes) are three separate terminologies. Since NANDA-I, NIC and NOC are very often used together, we will refer to them as a terminology set (NNN).

Systematically reporting and analyzing studies that used SNTs nursing data retrieved from EHRs is important to understand the analytic issues related to the complexity and richness of data generated from the use of these terminologies. Given the growing emphasis on using existing health care datasets, there is a need for new statistical, computational, and visualization methods to analyze EHRs data, given their complexity and volume.⁶ For nursing to join this effort, an important first step is to identify and examine the studies that analyze EHR data coded with SNTs.¹⁷

To date, there are few reviews that report secondary analysis of SNTs nursing data. One recently published systematic review¹⁸ described study focus, sample characteristics and frequency of publications that studied SNTs. The authors also identified a limited set of studies in which SNTs nursing data were being analyzed.¹⁸ However, substantial evaluation and discussion on the analysis of SNTs nursing data were not performed. Another review¹⁹ described characteristics of nursing research data (ranging from patient demographics, social history, medical history, medications, among others) and evaluated if a specific index metadata system represents sufficiently nursing data. Our systematic review differentiates from the previous investigations as we focused only on studies that analyzed nursing data coded with SNTs retrieved from EHRs. The earlier literature review¹⁹ included all types of unstructured nursing data, and different types of studies, such as controlled clinical trials. The authors also did not restrict their selection criteria for nursing data retrieved from EHRs.¹⁹

The systematic review presented below was conducted to describe and critically analyze the body of studies in which secondary analyses of data coded with the ANA recognized SNTs was performed. These nursing data were documented during the delivery of nursing care and retrieved from EHRs. We believe that the findings from the present systematic review will emphasize the importance of coded nursing data and will encourage a wider use of SNTs that will allow greater participation of nursing in big data science initiatives.

Objective

The objective of this systematic review was to uncover the state of the science related to the use of standardized nursing data (coded with SNTs) retrieved from EHRs to answer research questions, describe the analytical techniques employed, and outline the lessons learned applicable big data science and nursing.

Methods

Search strategy

A comprehensive literature search was conducted to identify publications in which secondary analysis was performed on data extracted from EHRs and documented using terms from the ANA recognized nursing terminologies (or sets). There were two phases used in this process: 1) study selection and 2) data collection process.

Building from the work of Tastan et al.¹⁸ that reported secondary uses of SNTs documentation in studies up to 2011, the databases PubMed and CINAHL were searched using keywords encompassing all ANA recognized SNTs, along with “nursing” and “electronic health records” from 2010 to 2017. Keywords were defined for each database with the help of a librarian, who was a specialist in Consumer Health, Nursing and Health Education and Behavior. The limiters since 2010, abstract available and published in English were used. Potential publications identified in both databases were downloaded into a reference management program (EndNote X7, Thompson Reuters ISI ResearchSoft), in which duplicates were deleted and abstracts were reviewed. Finally, grey literature search was conducted using Google Scholar to identify possible publications not captured by the traditional search methods, including relevant publications not in PubMed or CINAHL, but in computer science databases like IEEE. Name of authors of publications already reviewed and included in this study’s sample were individually searched in Google Scholar. Publications pertinent to the subject were reviewed and included.

Study selection

The studies identified using the specified search strategy were submitted to abstract review according to the following inclusion criteria: 1) publications that conducted secondary analysis of nursing data retrieved from EHRs, 2) nursing data were coded with one of the ANA recognized nursing terminologies and documented at the point-of-care, as part of the institution’s routine, and 3) publications were published in English. Following PRISMA guidelines²⁰, Figure 1 shows the number of publications first identified, retained after selection from abstracts, and reviewed as full text.

Figure 1.

PRISMA flow diagram illustrating the selection process of publications.

The team utilized a two-stage selection process. In the first stage, two doctoral students in nursing informatics (TM, ND) independently conducted the literature search in the databases using the same keywords and found the same number of studies. A guide created by the authors was used to systematically review the abstracts. The reviewers first identified if the article contained at least one of the ANA recognized SNTs. If the abstract described the use of a SNT, the reviewer would continue to verify if the data analyzed were retrieved from EHRs. In the case of an affirmative answer, the reviewers would further analyze the abstract to assess if the SNT coded nursing data were documented as part of routine clinical practice and not solely for research purposes. Publications selected by one reviewer but not the other were discussed until agreement was reached. In the second stage, TM and MS (doctoral students) conducted a full-text review for further eligibility screening.

Data collection process

Data from the eligible publications were independently collected by TM and MS. Tables were created to summarize and analyze the content of the articles. The following information was extracted: study foci; sample characteristics (number of health records); variables (dependent, independent, descriptive); and statistical analysis. Data extracted from the selected articles were also reviewed by two faculty members, one specialist in nursing informatics and one statistician. Consensus agreement was reached. The present study aimed to report the diverse statistical analysis methods utilized and studies foci addressed through the use of SNTs coded nursing data. We did not intend to report the results of those publications.

Quality appraisal

Quality appraisal of publications is common in systematic reviews, however, it is challenging when the publications being reviewed are of the secondary data analysis nature. For the purpose of this review, quality appraisal of articles was evaluated regarding completeness of reporting of information. An adapted version of the STROBE Statement-checklist for observational studies²¹ was used, including the following items from the original checklist: study size, clear definition of variables, and description of all statistical methods used. Clear definition of variables was scored as 1 (not defined), 2 (partially defined), and 3 (well defined). These ratings considered if the variables in the publications were described with sufficient clarity to be replicated (collected) in further studies. As the previous stages of the review, authors independently (TM, MS) evaluated the completeness of the items mentioned in rounds until agreement was achieved.

Results

Abstracts of 1,809 publications were examined and 75 publications were retained according to the inclusion criteria. In the data collection process (second stage), of the 75 publications remaining, 40 were excluded due to different reasons as described in Figure 1. A total of 35 publications were included in our systematic review.

The studies included represent considerable diversity in terms of sample sizes, ranging from 29 to 379,601 health records. The largest sample sizes represent more than one unit in a single hospital or multiple hospitals, and more than one primary health care center. In these cases, the same EHR and SNTs were implemented across the institutions. Types of records portrayed in the studies encompass daily entries, or shift entries of nursing information into the EHRs at point-of-care. Episodes of care represent the entire history of a hospitalization, typically consisting of several entries for a unique patient hospitalization. For the purpose of this study, health records were defined as representing unique patients and health records entries were defined as daily/shift documentation of care.

In this review, we found studies for only three of the five terminology sets. There were no studies found for CCC and PNDS. There were 21 NNN articles, 10 Omaha System articles, three articles on ICNP, and one article that evaluated both NANDA-I and ICNP. Although all articles were published in English, we identified studies conducted in countries where English is not an official language. All articles on ICNP were conducted in South Korea, while all articles using the Omaha System were conducted in the US. The studies focusing on NNN were from Spain and U.S.

Descriptive, dependent and independent variables of each study were collected. In studies of descriptive nature, the variables analyzed were the nursing diagnoses, interventions and outcomes relevant to each set of SNTs. For studies in which statistical tests for associations were conducted, the majority of independent variables were the sets of SNTs, along with other factors related to delivery of care and the healthcare environment. Dependent variables in those studies varied significantly, including for example, occurrence of falls²², pain^23–24, gender, age and medical diagnoses²⁵, nursing cost^26–27, among others.

Research focus

Research foci encompassed different questions that could be answered by SNT coded nursing data. Twelve studies focused on characterization of units and patients regarding the most common nursing diagnoses, nursing interventions and nursing outcomes^28–39 (Table 1). In two studies, authors described potential flaws in the terminologies regarding their content^40–41 (Table 1).

Table 1.

Research foci.

SNT	Research focus
	To describe availability of plans-of-care data34
	To describe most prevalent NNN of an SNT-based electronic nursing documentation37
	To describe the most frequent NNN documented for hospitalized older patients with a primary discharge diagnosis-related group of pneumonia30
	To describe the most frequent NNN documented for hospitalized older patients with a discharge primary diagnosis of heart failure33
	To report pain care from admission to discharge or death for end-of-life patients24
	To determine statistically significant changes in mean scores of outcomes from admission to discharge for specific medical diagnoses45
	To determine changes in outcomes scores from admission to discharge, and describe interventions for pediatric patients with dehydration44
	To measure the cost of delivering high surveillance, among hospitalized elders at risk for falling27
	To measure association among patient characteristics, unit characteristics, medical, pharmacy, and nursing interventions, and falls for older adults22
NNN	To measure the association between high surveillance and low surveillance and the occurrence of failure to rescue48
	To establish association between degree of severity of problems and psychiatric diagnosis, number of nursing diagnoses, age, gender25
	To determine associations between the variables nursing characteristics and nursing cost per acute care episode26
	To identify patient and nurse-related factors associated with nursing outcomes for end-of-life patients with death anxiety51
	To report of NOC scores linked with the most frequent nursing interventions, age and length of stay36
	To describe the difference between plans-of-care with NNN and plans-of-care without NNN in relation to intermediate health outcomes38
	To measure association between the variable nurse continuity and the occurrence of pressure ulcers56 Describe data mining techniques to predict whether a patient would meet the expected pain related outcomes23
	To develop predictive models that show how patient and nursing variables impact comfortable death outcome in end-of-life patients52
	To predict current NOC rating for each nurse shift based on previous shift ratings and factors related to nurse shift53
	To create predictive models that determine whether a patient with pain problems will be re-admitted to a hospital55
	To describe infomarkers that mark a shift in pain outcomes from standard care to palliative care among patients that died during hospitalization54
	To describe client problems, interventions, categories and targets, along with outcomes documented for patients in a low-birth weight program28
	To describe all Omaha System data elements across multiple homecare settings using two different EHR vendors29
	To describe client risk index, family home visiting interventions between groups of low- and high-risk31
	To describe free text entry for the category Signs and Symptoms of the Omaha Classification40
OS	To describe free text entries associated with Omaha System on a software41
	To describe ratings of knowledge, behavior, and status on admission and discharge for Latina adolescent and adult mothers with mental problems43
	To demonstrate differences between.p-values and Cohen’s d in describing clinically meaningful changes in scores for Omaha System problems46
	To determine which group of interventions predicts the outcome hospitalization for frail and non-frail elders47
	To measure association between patient, support system factors and nursing interventions and improvement in urinary and bowel incontinence49
	To determine the association between nurses, Omaha System interventions and client characteristics, and variability in health literacy50
	To describe variability in nursing intervention to prevent and treat pressure ulcer patients32
ICNP	To describe nursing preventive pressure-ulcer interventions and to compare with measures from two published guidelines35
	To describe narrative nursing statements of cancer patients treated with cisplatin-based chemotherapy35
	To test different computerized search strategies, to analyze the incidence and clinical characteristics of contrast media hypersensitivity42
N/ICNP	To describe key nursing diagnoses from the two terminologies for patients with heart failure39
Key = NNN (NANDA-I, NOC, NIC); OS (Omaha System); N/ICNP (NANDA-I and ICNP).

Study of the nursing care through the use of SNTs included differences between standardized and non-standardized plans-of-care³⁸; comparison of nursing care provided and standard nursing care from guidelines³²; and testing different search strategies using nursing interventions to analyze the incidence of a medical complication.⁴² The benefits of using SNTs in EHRs to increase availability, validity, and reliability of data for statistical analyses and other research purposes were also described.³⁴

SNTs NOC and Omaha System were used across different studies, measuring changes in outcomes from admission to discharge.^43–45 Both terminologies have a grading scale to be used by nurses daily to rate the nursing outcomes for each patient. In the studies^43–45, authors compared ratings given at admission and discharge to patients as a way of measuring effectiveness of care. Association of those ratings with other variables from EHR was also performed.⁴³ One study focused on demonstrating differences between two statistical methods in describing changes in ratings for the Omaha System terminology.⁴⁶

NOC and NIC were used to estimate nursing cost.^26–27 Jenkins and Welton²⁶ used the NOC rating scale as a measure of nursing intensity, and in turn, nursing cost. A different measure of nursing cost was used by Shever et al.²⁷ In their study, NIC was used to measure the cost of delivering the nursing intervention surveillance. Surveillance was used as a measure of intensity of treatment, based on the average number of times a day that it was delivered (low or high). The number of times the intervention was performed was associated with the total hospital cost for the population under study.²⁷

In two studies the researchers examined relationships between nursing interventions and outcomes.^47–48 Other (n=4) studies in the sample focused on establishing associations among a broader number of variables, such as patient characteristics, support system factors; nursing characteristics, nurse continuity, degree of severity of problems; medical diagnoses, pharmacy and nursing interventions, and nursing outcomes.^{22,25,50–51} Lodhi et al.⁵¹, Lodhi et al.⁵², and Lodhi et al.⁵³ identified patient and nurse-related factors associated with meeting expected nursing outcomes.

Almasalha et al.²³ focused on the use of SNTs to predict whether a patient would meet the expected outcomes by the end of a hospitalization period. Yao et al.⁵⁴ identified specific elements of SNT related to changes in outcomes that could be used as an indicator for the adjustment from standard nursing care to palliative nursing care. Lastly, Lodhi et al.⁵⁵ explored the creation of different predictive models for re-admission of patients that were diagnosed with pain during their last hospitalization.

Statistical analysis

A total of 15 studies used only descriptive statistics to analyze their data. The following statistical analysis methods were present among the remaining studies: propensity scores^26,48; logistic regression and other regression methods^{25–27,47–48,52,55–56}; generalized estimating equations^22,27; data mining techniques^{22,47,51–53,55}; chi-square^{23–24,51–53}; t-test^{23,39,44–46}; ANOVA^35,54; Pearson correlation^25,53; Tukey’s post-hoc test⁵⁴; logistical mixed-effects model⁵⁰; effect sizes (Cohen’s d)⁴⁶; general linear mixed methods models⁴³; and known-group comparative analysis.⁵⁴

Data from same institutions were used across different publications to answer a diverse number of research questions. Table 2 shows the progression of science and knowledge in nursing across the years. Publications are grouped by dataset (each group of institutions used the same EHR). It can be seen (see Table 2) that earlier publications used mostly descriptive statistics and advanced to generalized estimating equations, logistic regressions, and data mining procedures. Sample sizes varied across publications and more specific and complex statistical analysis were related to smaller parts of the datasets.

Table 2.

Progression of nursing science and knowledge.

Key = * health records; ** health records entries

Quality of publications included

Among the 35 publications included in this systematic review, all described their study size, which was represented by number of health records or health records entries, and listed statistical analyses used. Definitions of variables were scored for the 35 publications; 74% of the publications^{22–25,27–33,37,39–40,44–48,51–56} scored a 3 (variables well defined), 23% of the pubications^{26,34–35,38,41,42,49–50} scored a 2, and 3% of the publications⁴³ scored a 1. Overall, the studies were rated as good for completeness of reporting of information.

Discussion

The present systematic review provides important information about the types of studies that have been conducted on SNTs coded nursing data retrieved from EHRs. The 35 studies were identified and reviewed through a comprehensive systematic process. The SNTs NANDA-I, NOC, NIC and Omaha System were those implemented most often across different EHRs. Specifically, NNN was represented more in the hospital setting, while Omaha System was present in the primary level of healthcare. Worth noting however is that both of these terminologies can be used interchangeably between hospitals and primary health care centers.

There was great diversity in the questions addressed in each study and the variables analyzed. Characterization of nursing care through the most common nursing diagnoses, interventions and outcomes was the aim of some studies. Others focused on determining significant changes in outcomes from admission to discharge, measure cost of nursing interventions, establishing associations using variables inherent to nursing care and other variables found in EHRs. The creation and use of predictive outcomes models were also studied. This range of study foci show the potential of obtaining and translating new knowledge from nursing data coded with SNTs and the holistic approach of nursing care. It also brings the attention of researchers to the possibility of asking new questions aimed at improving patient outcomes, nursing practice and the healthcare system.

Among the studies included in the sample, most investigators either utilized descriptive analysis (mean, standard deviation, frequency, percentage, etc.) to describe terminology usage in different healthcare settings or relied on classical tests (e.g., ANOVA, Tukey’s Post Hoc, Chi-square, t-test, Mann Whitney U test, correlation test) to determine bivariate associations. Although of limited nature, descriptive studies are a first step to understand a dataset and the target population under study. Descriptive statistics are also very efficient and helpful in the identification of predictors with low variability that could be excluded in further analysis.⁵⁷ The descriptive studies included in this systematic review provide a foundation for the nursing scientific community to move forward to the use of more sophisticated statistical methods.

The association analyses were those between the adoption of SNTs and patients’ outcomes, between nursing diagnosis and episode length, and between nursing interventions and patient outcomes. A number of articles conducted multivariate analysis utilizing regression analysis (e.g., generalized linear mixed regression, generalized estimating equations, logistic regression, etc.) incorporating techniques such as propensity score and clustering to examine the effects of patient, nurse, and care characteristics on patient outcomes. Finally, there were a few studies^{51–53, 56} using data mining methods to identify hidden patterns among nursing care data or to construct predictive models for patient outcomes. Interesting results were reported in these studies. For example, the pain management, medication, management, and positioning interventions were collectively associated with pain relief among end-of-life patients.²³ Continuity of nursing care was not associated with occurrence of pressure ulcers⁵⁶ and predictive modeling on large EHR SNTs data can be used as foundation in nursing decision support.⁵³

Our systematic review revealed strengths of using SNTs to code nursing data. Data stored in digital format that are unstructured or not standardized pose difficult challenges to data processing and analysis.^19,57 Unstructured data may obscure linkages among elements, adversely impact the validity of the data collected, and inadvertently fuel misunderstandings and errors. Words that describe care might have different meanings from unit to unit, and between institutions, due to the fact that standards of care and cultures can vary considerably.⁵⁷ The processing of unstructured data is also more time’consuming and labor’intensive when compared to SNT coded nursing data.

Nursing data coded with SNTs facilitates comparisons of patient care within and across institutions.¹⁷ The evidence generated in turn can enhance the decision-making process¹⁷ and demonstrate the impact of nursing care making nursing practice visible.⁵⁷ The use of valid standardized nursing terminologies makes extraction of nursing data easier⁵⁷, reduces the cost of reporting quality measures⁵⁸, and produces interoperable data.¹⁸

There are some limitations of this systematic review. The search focused on two databases, which excluded research studies that could have been published in books or sources not indexed by the selected databases, although some articles from other databases were identified through grey literature search. Also, the performed search strategies and the search terms used to uncover the literature may have excluded publications on the subject that used different search terms or keywords. That the focus of our review was on study methods rather that study results limits the overall conclusions that are informed by this review and point to the need for a review of study results.

A limitation identified among the studies included in this systematic review is the scarce integration and analysis of SNT coded nursing data with other parts of EHRs. In our sample, only four studies included other variables from EHRs in their analysis, such as medical diagnoses, pharmaceutical treatments and pressure ulcer rating scale.^22,25,27,32 To be a part of big data science, nursing data not only need to be coded with SNTs, but also should be combined with other parts of the EHRs in order to enable big data analysis. An example of big data analysis in the healthcare field would be to examine relationships among nursing diagnoses, medical conditions and laboratory results and their impact on patients’ outcomes.¹⁰

A main lesson learned for nursing and big data science is the extraordinary potential of coding nursing care with SNTs and using the data to understand the impact of nursing and continuously expand nursing science.¹⁰ Our systematic review highlights the importance of SNTs to enable faster retrieval and processing of nursing data and the application of big data analysis. Big data science can benefit from the integration of nursing data to other datasets. The care provided by nurses is unique and an important contribution to patient outcomes. Thus, including the nursing component in analyses involving health care outcomes is essential to learning the improvements nurses can make to enhance the overall patient outcomes achieved. In the absence of analyzable nursing care data, the impact of nursing will continue to be elusive. This review provided a lens into the uses and potential benefits of SNTs now and in the future.

Conclusion

The use of SNTs to code nursing data enables the aggregation of this fundamental part of EHRs to big data datasets. Secondary data analyses of nursing data using methods such as data mining and clustering techniques should be used more often as a way of finding more meaningful results that could change positively the nursing practice and impact directly the care received by patients. This systematic review underscored the value of developing a deep understanding of the meaning and potential impact of nursing variables before merging with other sources of data. Further research is needed to analyze nursing data along with other parts of the EHRs, revealing possible associations among care variables from different health professions.