Abstract
Objective
To analyze the nursing diagnostic concordance among users of a clinical decision support system (CDSS), The Electronic Documentation System of the Nursing Process of the University of São Paulo (PROCEnf-USP®), structured according to the Nanda International, Nursing Intervention Classification and Nursing Outcome Classification (NNN) Taxonomy.
Materials and Methods
This pilot, exploratory-descriptive study was conducted from September 2017 to January 2018. Participants were nurses, nurse residents, and nursing undergraduates. Two previously validated written clinical case studies provided participants with comprehensive initial assessment clinical data to be registered in PROCEnf-USP®. After having registered the clinical data in PROCEnf-USP®, participants could either select diagnostic hypotheses offered by the system or add diagnoses not suggested by the system. A list of nursing diagnoses documented by the participants was extracted from the system. The concordance was analyzed by Light’s Kappa (K).
Results
The research study included 37 participants, which were 14 nurses, 10 nurse residents, and 13 nursing undergraduates. Of the 43 documented nursing diagnoses, there was poor concordance (K = 0.224) for the diagnosis “Ineffective airway clearance” (00031), moderate (K = 0.591) for “Chronic pain” (00133), and elevated (K = 0.655) for “Risk for unstable blood glucose level” (00179). The other nursing diagnoses had poor or no concordance.
Discussion
Clinical reasoning skills are essential for the meaningful use of the CDSS.
Conclusions
There was concordance for only 3 nursing diagnoses related to biological needs. The low level of concordance might be related to the clinical judgment skills of the participants, the written cases, and the sample size.
Keywords: nursing diagnosis, clinical decision support system, concordance
INTRODUCTION
Decision-making is an important aspect of clinical practice. Nurses make clinical decisions every 30 s.1 Usually, nurses have to consider a variety of competing factors while making clinical decisions, besides balancing time constraints, heavy workloads, and increasing patient acuity.1 In this context, clinical decision support systems (CDSS) are essential to assist nurses in making accurate decisions when: dealing with a high volume of complex clinical data, they are less experienced, and time constraints are a challenge to decision-making.
A CDSS is a valuable tool for health professionals to support their decision-making process by providing information intelligently filtered and presented in a timely manner.2,3 In nursing, a CDSS should support nurses in making decisions throughout the Advanced Nursing Process, which consists of a problem-solving approach to decision-making rooted in science-based nursing diagnosis, intervention, and outcome classifications.4
Ideally, CDSSs used by nurses should incorporate SNLs to allow nurses to describe and communicate the central elements of nursing practice and, consequently, to extract, analyze, and exchange data on nursing practice.5–7 The use of SNLs, such as International Classification Nursing Practice (ICNP), Community Health System (OMAHA), Clinical Care Classification (CCC), Nanda International (NANDA-I), Nursing Intervention Classification (NIC), and Nursing Outcome Classification (NOC), has been shown to improve communication among healthcare providers, enhance documentation of nursing care, and promote evidence-based practice. They also have been associated with improved patient outcomes such as reduced length of hospital stay, decreased mortality rates, and improved patient satisfaction.8–11
A systematic review of the literature found that nursing diagnoses predict patient outcomes such as quality of life and mortality, and organizational outcomes such as length of stay, hospital costs, amount of nursing care, and willingness to discharge. Patient care plans based on nursing diagnoses improved sleep quality, quality of life, and glycemic control.9 Another prospective observational study found that the number of nursing diagnoses is an independent predictor of length of stay in the hospital (β=.15; P<.001) and longer-than-expected length of stay (β=.19; P<.001) after controlling for patients’ sociodemographic characteristics, all patient refined diagnostic related group, disease severity morbidity indexes, and organizational hospital variables. These findings show that nursing diagnoses describe the complexity of the patient more comprehensively and can be used to improve the efficiency of hospital management.10
The Electronic Documentation System of the Nursing Process of the University of São Paulo (PROCEnf-USP®) is a nursing CDSS pending integration in the electronic medical record. It was developed by nursing faculty from the School of Nursing (EEUSP) and nurses working at the University Hospital of the University of São Paulo (HU-USP).12 The technical quality and functional performance of PROCEnf-USP® were evaluated in a study including nursing undergraduates, computer specialists, faculty, and staff nurses using the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 25010 Quality Model.13 This model is significant because it provides consistent terminology for specifying, measuring, and evaluating system and software product quality. Additionally, it presents a range of quality attributes that can be used as a benchmark for ensuring that all quality requirements are met.14 In the study, PROCEnf-USP® achieved more than 70% positive responses in most of the quality characteristics evaluated, such as functional adequacy, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability.13
PROCEnf-USP® also complies with the recommendations of the Nursing Process-Clinical Decision Support System (NP-CDSS) standard,5,15 which is a guide to developing nursing CDSSs. According to the NP-CDSS, the Advanced Nursing Process should be the centerpiece of the system, as it recommends that the NP-CDSS should suggest research-based, predefined nursing diagnoses and correct linkages between diagnoses, evidence-based interventions, and patient outcomes.5,15
PROCEnf-USP® is structured according to the Advanced Nursing Process steps (assessment, diagnosis, planned outcomes, interventions, and outcome evaluation), and the harmonization of the NANDA International Classification (NANDA-I), Nursing Intervention Classification (NIC), and Nursing Outcomes Classification (NOC), also known as NNN taxonomy.16 It has 2 environments: professional and academic. The professional environment is specifically used to document nursing care provided to patients, whereas the academic environment is used for educational and research purposes.12 Both environments work in the same way. The user enters data from the assessment by answering branched questionnaires organized according to the NNN harmonized taxonomic structure.12 Then, based on the documented assessment data, PROCEnf-USP® suggests diagnostic hypotheses which can be selected by the user. Besides selecting the diagnoses, the user is offered possible nursing outcomes and interventions. In addition to selecting diagnoses, outcomes, and interventions sensitized by PROCEnf-USP®, the user may add others that were not suggested by the CDSS, even though corresponding assessment data were not registered in the system.12Figures 1–4 illustrate how the system operates.
Figure 1.
Access screen to PROCEnf-USP® and selection of academic or professional environment.
Figure 2.
Screen for inserting patient assessment data in PROCEnf-USP®.
Figure 3.
Selection screen and add diagnoses.
Figure 4.
Documented nursing diagnostics screen.
This particular characteristic of PROCEnf-USP® may influence how users at different levels of expertise use the system and, ultimately, the accuracy of their clinical decisions concerning the nursing diagnoses. According to the from novice to expert theory, experience and education are the basis for the acquisition of skills, qualified performance, and development of nurses’ clinical knowledge.17 This theory was proposed by Benner and explains how a learner (student, new, or experienced nurses) develops skills and understanding of a practice situation over time. Learners move through 5 stages of competency, from novice to expert in a nonlinear process. In other words, learners do not necessarily move from one stage to the next systematically, rather they may move from one stage to another repeatedly while learning new knowledge and skills. Moreover, at the novice stage, nurses tend to use more the logical reasoning process to make decisions and follow all the steps within PROCEnf-USP®; at more advanced stages, nurses are more likely to use intuition17 and recognize patients’ patterns of response. As a result, more experienced nurses may jump to clinical decisions without registering assessment data in PROCEnf-USP®. A CDSS would be expected to account for the lack of experience of novice nurses when the right data are registered in the system. Thus, a high diagnostic concordance for highly accurate or “correct” diagnoses would be expected among users at different levels of expertise.
In fact, nursing undergraduates are able to make diagnostic decisions at a higher level of accuracy when using PROCEnf-USP® in relation to manual documentation.18 In addition, researchers found that PROCEnf-USP® supports clinical decision-making when comparing users at different levels of education and professional experience in nursing to select diagnoses with high and moderate accuracy.19
Thus, it is plausible to think that without the CDSS experienced nurses, due to their knowledge, clinical experience, and more developed clinical reasoning skills, would perform better than nonexperienced nurses. However, with the aid of the CDSS, the less experienced nurses would perform better and more similar to the more experienced nurses, leading to concordance. Concordance refers to the ability of different users to make similar decisions about the same phenomenon.20 Reaching high levels of concordance among nurses with varied levels of education and expertise when using a CDSS is important to ensure patient safety and quality of care, and inappropriate variations in processes of care or management.21,22 Inasmuch, when nurses agree on caring for the patient based on highly accurate nursing diagnoses, it is likely that better patient outcomes will be achieved. On the other hand, low accuracy interpretations can lead to omission of care and unnecessarily prolonged treatment.23
Analyzing the diagnostic concordance among CDSS users is relevant to understand the behavior of decision support systems when they are used by users with different educational levels and professional experience. To this date, it remains unclear whether CDSSs such as PROCEnf-USP® lead to high levels of concordance regarding decisions about nursing diagnoses when comparing users at different levels of expertise. The hypothesis of this study is that the use of a CDSS will lead to high concordance among users with different levels of education and experience when selecting nursing diagnoses based on the set of clinical data. This study aims to evaluate nursing diagnostic concordance among medical-surgical nurses, resident nurses, and nursing undergraduates using PROCEnf-USP®.
MATERIALS AND METHODS
This is a pilot, exploratory-descriptive study carried out from September 2017 to January 2018.
Sample and participants
The sample included all medical-surgical nurses (N = 27) and all adult and older adult health nurse residents (N = 12) in HU-USP, nursing undergraduates taking part in the internship program at HU-USP (N = 58). The HU-USP is an academic hospital and the main site for EEUSP undergraduate students and nurse residents. Currently, PROCEnf-USP® is used only in this hospital.
All nurses working on the medical-surgical floors were eligible to participate in this study. These units were chosen because nurses have been using the CDSS system for the longest time, since 2009. Resident nurses were eligible if they had taken the nursing theory and process course, a didactic course which conveys the use of nursing theories and Advanced Nursing Process to guide clinical decision-making in common situations experienced by adults and older adults. The nursing residency program is a type of clinical specialist program characterized by in-service education.24 Nursing undergraduates were eligible to participate in the study if they had attended the integrative conceptual matrix course, which provides concepts, models, and theories in nursing, as well as clinical reasoning and Advanced Nursing Process to guide clinical experiences and reflexive practice. Participants not performing academic or professional activities during the data collection period or those who did not register any data in the assessment component in PROCEnf® were excluded.
Data collection
The principal investigator met with all potential participants in person to invite them to participate in this study. Those who agreed to participate provided an email address to receive the consent form, the data collection instrument consisting of demographics, the 2 case studies, login number, and password to access PROCenf-USP® academic environment.
Participants were provided with detailed clinical data from 2 validated case studies23 to be entered into PROCEnf-USP®. The cases were related to medical-surgical clinical scenarios and aligned with the context in which the data was gathered. Authorization to use the case studies was obtained from the original authors. The case studies were previously validated by experts, who established the gold standard nursing diagnoses. We tested the cases using PROCEnf-USP® and found that the system suggested the gold standard diagnoses when all clinical data was entered into the system.
Case study 1 reported data from a 57-year-old female patient admitted to the medical unit of a general hospital, with a history of obesity, type II diabetes mellitus, and arterial hypertension. Clinical data were consistent with myocardial ischemia. The patient was awake, calm, and reported chest pain, shortness of breath on exertion, weakness, impaired vision, and worry about her husband. Physical examination and vital signs were within normal limits. The gold-standard nursing diagnoses for this case were: Imbalanced nutrition: more than body requirements (00001), Ineffective control of therapeutic regimen (00078), Activity intolerance (00092), Risk for impaired skin integrity (00047), and Risk for physical trauma (00038).
Case study 2 reported data from a 58-year-old female patient admitted to the medical unit of a general hospital, presenting signs and symptoms consistent with pneumonia. The patient reported a history of smoking, anxiety, productive cough, shortness of breath on exertion, chest pain, and pain in lower limbs due to fibromyalgia. Physical examination revealed decreased vesicular murmurs in the left pulmonary base, tachypnea and tachycardia. The gold-standard nursing diagnoses for this case study were: Acute pain (00132), Activity intolerance (00092), Imbalanced nutrition: more than body requirements (00001), and Ineffective breathing pattern (00032).
Both cases provided the participants with clinical data to be registered in PROCEnf-USP® for an initial patient assessment. Participants were able to access the system academic environment from any HU-USP computer, at any time (before, during, or after work or internship). They had a total of 2 h to complete the 2 cases studies over a 15-day period and were allowed to log in and log off as often as they wished during this period. After including the clinical data provided in the case studies, participants could either select any hypotheses they deemed relevant or add diagnoses that had not been suggested by the system. It was not possible to retrieve the time each participant spent completing the case studies on the system.
A list of diagnoses documented by the participants was obtained from the system by the HU-USP information technology department. The data collected for this study were stored in Excel® software spreadsheets. The spreadsheet was composed of characterization data of the participants and the nursing diagnoses selected and added for each case (1 and 2) by each participant. In this study, participants used the NANDA-I 2015-2017 version.25
Data analyses
Data were analyzed using the software R 3.4.2®. Descriptive statistics were used to present the demographic data and describe the nursing diagnoses documented by the participants. Continuous variables were described by central tendency (mean and median) and dispersion measures (standard deviation—DP, interquartile interval—IIQ, and minimum and maximum values). Categorical variables were described by means of absolute and relative frequencies.26 The normality in distribution and differences among groups of continuous variables was verified under the Shapiro-Wilks test.27 Differences among groups in relation to categorical variables were tested by means of Fisher’s exact test.26,27
The concordance was analyzed by Light’s Kappa, which measures the concordance between 2 or more judges when they are classifying objects into categories. Light’s Kappa is the Cohen’s Kappa average for all possible pairs of participants. The interpretation of concordance according to the Kappa values are: Total concordance (0.8 ≤ K ≤ 1), High concordance (0.6 ≤ K < 0.8), Moderate concordance (0.4 ≤ K < 0.6), Weak concordance (0.2 ≤ K < 0.4), Poor concordance (0 ≤ K < 0.2), and No concordance (K < 0).23–28
Ethical aspects
The study was approved by the EEUSP (process number 2.037.980) and HU-USP (process number 2.026.662) Ethics Committees.
RESULTS
Out of 95 eligible individuals, 39 agreed to participate, a recruitment rate of 41%. However, one nurse resident on maternity leave and another nurse resident who withdrew from the program during the data collection period were excluded. Therefore, the attrition rate was 5.1% (n = 2). Thus, 37 participants (14 nurses, 10 nurse residents, and 13 nursing undergraduates) completed case 1. Three nursing undergraduates and 3 nurses did not complete case 2. Therefore, the total sample for case 2 consisted of 31 participants (11 nurses, 10 nurse residents, and 10 nursing undergraduate).
Table 1 describes the demographics, academic, and professional characteristics of the sample according to group categories.
Table 1.
Demographics, academic, and professional characteristics of the sample according to participants’ groups
| Variables of characteristics | Nursing undergraduate (n = 13) | Nurse resident (n = 10) | Nurse (n = 14) | P-value |
|---|---|---|---|---|
| Age | ||||
| Mean (DP) | 24.0 (2.6) | 25.3 (1.5) | 40.7 (8.5) | <.001a |
| Median [Q1; Q3] | 24.0 [22.0; 25.5] | 26.0 [24.3; 26.0] | 39.0 [34.3; 48.5] | |
| Min–Max | 21.0–30.0 | 23.0–27.0 | 27.0–54.0 | |
| Years of professional experience in nursing | ||||
| Mean (DP) | NA | 1.6 (2.0) | 16.2 (6.6) | <.001a |
| Median [Q1; Q3] | NA | 1.3 [0.2; 2.0] | 15.0 [11.5; 21.5] | |
| Min.–Max. | NA | 0.0; 7.0 | 4.0; 29.0 | |
| Years of professional experience in adult and older adult health | ||||
| Mean (DP) | NA | 1.6 (2.0) | 15.1 (6.4) | <.001a |
| Median [Q1; Q3] | NA | 1.3 [0.1; 1.9] | 14.0 [10.2; 20.0] | |
| Min.–Max. | NA | 0.0; 7.0 | 4.0; 26.0 | |
| Experience in nursing process applied to clinical practice | ||||
| High | 0 (0%) | 2 (20%) | 7 (50%) | <.001b |
| Considerable | 6 (46.1%) | 5 (50%) | 7 (50%) | |
| Low | 5 (38.4%) | 3 (30%) | 0 (0%) | |
| None | 2 (15.3%) | 0 (0%) | 0 (0%) | |
| Experience in nursing diagnosis applied to clinical practice | ||||
| High | 0 (0%) | 3 (30%) | 6 (42.8%) | .002b |
| Considerable | 8 (61.5%) | 6 (60%) | 8 (57.1%) | |
| Low | 3 (23%) | 1 (10%) | 0 (0%) | |
| None | 2 (15.3%) | 0 (0%) | 0 (0%) | |
| Experience using PROCEnf-USP® | ||||
| High | 0 (0%) | 3 (30%) | 7 (50%) | .001b |
| Considerable | 8 (61.5%) | 4 (40%) | 7 (50%) | |
| Low | 3 (23%) | 3 (30%) | 0 (0%) | |
| None | 2 (15.3%) | 0 (0%) | 0 (0%) | |
| Highest academic degree | ||||
| Doctorate (PhD) | 0 (0%) | 0 (0%) | 2 (14.2%) | <.001b |
| Specialization | 0 (0%) | 2 (20%) | 7 (50%) | |
| Undergraduate/Bachelor | 13 (100%) | 8 (80%) | 1 (7.1%) | |
| Master | 0 (0%) | 0 (0%) | 4 (28.6%) | |
DP: standard deviation; Q1: 1st quartile; Q3: 3rd quartile; Min.: minimum; Max.: maximum; NA: not applicable.
Kruskal–Wallis test.
Fisher’s exact test.
Regarding case study 1, participants documented 32 different nursing diagnoses as shown in Table 2.
Table 2.
Nursing diagnoses documented for case study 1 according to participants’ groups
| Nursing diagnoses | Nursing undergraduates | Nurse residents | Nurses | Total |
|---|---|---|---|---|
| Case 1 | (n = 13) | (n = 10) | (n = 14) | (n = 37) |
| % | % | % | % | |
| Nursing diagnoses suggested by PROCEnf-USP® | ||||
| Activity intolerance (00092)a | 6 (46.15%) | 10 (100%) | 9 (64.28%) | 25 (67.57%) |
| Risk for falls (00155) | 5 (38.46%) | 2 (20%) | 6 (42.85%) | 13 35.14 |
| Acute pain (00132) | 5 (38.46%) | 1 (10%) | 4 (28.57%) | 10 (27.03%) |
| Risk for impaired skin integrity (00047)a | 5 (38.46%) | 2 (20%) | 3 (21.42%) | 10 (27.03%) |
| Impaired tissue integrity (00044) | 2 (15.89%) | 2 (20%)_ | 4 28.57 | 8 (21.62%) |
| Fatigue (00093) | 4 (30.77%) | 0 (0%) | 3 (21.43%) | 7 (18.92%) |
| Impaired skin integrity (00046) | 2 (15.39%) | 3 (30%) | 2 (14.28%) | 7 (18.92%) |
| Decreased cardiac output (00029) | 3 (23.8%) | 1 (10%) | 1 (7.14%) | 5 (13.51%) |
| Imbalanced nutrition: more than body requirements (00001)a | 1 (7.69%) | 1 (10%) | 3 (21.43%) | 5 (13.51%) |
| Sedentary lifestyle (00168) | 3 (23.08%) | 0 (0%) | 1 (7.14%) | 4 (10.81%) |
| Ineffective breathing pattern (00032) | 2 (15.38%) | 0 (0%) | 2 (14.29%) | 4 (10.81%) |
| Deficient knowledge (00126) | 0 (0%) | 2 (20%)_ | 1 (7.14%) | 3 (8.11%) |
| Ineffective control of therapeutic regime (00078)a | 0 (0%) | 1 (10%) | 1 (7.14%) | 2 (5.41%) |
| Chronic pain (00133) | 1 (7.69%) | 0 (0%) | 1 (7.14%) | 2 (5.41%) |
| Impaired walking (00088) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Impaired airway clearance (00031) | 0 (0%) | 0 (0%) | 1 (7.14%) | 1 (2.70%) |
| Disturbed body image (00118) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Risk for activity intolerance (00094) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Impaired gas exchange (00030) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Impaired spontaneous ventilation (00033) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Deficient fluid volume (00027) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Risk for imbalanced fluid volume (00025) | 1 (7.69%) | 0 (0%) | 0 (0%) | 1 (2.70%) |
| Nursing diagnoses added by the participants | ||||
| Risk for unstable blood glucose level (00179) | 10 (76.92%) | 7 (70%) | 12 (85.71) | 29 (78.38%) |
| Ineffective tissue perfusion (00024) | 6 (46.15%) | 1 (10%) | 9 (64.28%) | 16 (43.24%) |
| Risk for ineffective peripheral tissue perfusion (00228) | 2 (15.39%) | 1 (10%) | 2 (14.29%) | 5 (13.51%) |
| Risk for infection (00004) | 3 (23.08%) | 1 (10%) | 0 (0%) | 4 (10.81%) |
| Anxiety (00146) | 0 (0%) | 1 (10%) | 2 (14.29%) | 3 (8.11%) |
| Bathing/hygiene self-care deficit (00108) | 1 (7.69%) | 0 (0%) | 2 (14.29%) | 3 (8.11%) |
| Risk for decreased cardiac tissue perfusion (00200) | 1 (7.69%) | 1 (10%) | 1 (7.14%) | 3 (8.11%) |
| Impaired physical mobility (00085) | 0 (0%) | 0 (0%) | 1 (7.14%) | 1 (2.70%) |
| Risk for peripheral neurovascular dysfunction (00086) | 0 (0%) | 1 (10%) | 0 (0%) | 1 (2.70%) |
%: percentage of participants who documented the nursing diagnosis.
Gold-standards.
Concerning case study 2, participants documented 30 different nursing diagnoses as shown in Table 3.
Table 3.
Nursing diagnoses documented for case study 2 according to participants’ groups
| Nursing diagnoses | Nursing undergraduates | Nurse residents | Nurses | Total |
|---|---|---|---|---|
| Case 2 | (n = 10) | (n = 10) | (n = 11) | (n = 31) |
| % | % | % | % | |
| Nursing diagnoses suggested by PROCEnf-USP® | ||||
| Ineffective breathing pattern (00032)a | 7 (70%) | 4 (40%) | 4 (36.36%) | 15 (48.39%) |
| Ineffective airway clearance (00031) | 2 (20%) | 5 (50%) | 6 (54.55%) | 13 (40.63%) |
| Activity intolerance (00092)a | 3 (30%) | 3 (30%) | 5 (45.45%) | 11 (35.48%) |
| Acute pain (00132)a | 3 (30%) | 2 (20%) | 3 (27.27%) | 8 (25.00%) |
| Risk for falls (00155) | 3 (30%) | 1 (10%) | 2 (18.18%) | 6 (19.35%) |
| Ineffective health maintenance (00099) | 2 (20%) | 1 (10%) | 2 (18.18%) | 5 (16.67%) |
| Ineffective tissue perfusion (00024) | 1 (10%) | 0 (0%) | 3 (27.27%) | 4 (12.90%) |
| Hyperthermia (00007) | 0 (0%) | 0 (0%) | 2 (18.18%) | 3 (9.68%) |
| Impaired sensory perception (00122) | 0 (0%) | 1 (10%) | 2 (18.18%) | 3 (9.68%) |
| Impaired gas exchange (00030) | 2 (20%) | 0 (0%) | 1 (9.09%) | 3 (9.68%) |
| Impaired walking (00088) | 2 (20%) | 0 (0%) | 0 (0%) | 2 (6.45%) |
| Impaired swallowing (00103) | 1 (10%) | 0 (0%) | 1 (9.09%) | 2 (6.45%) |
| Impaired physical mobility (00085) | 0 (0%) | 1 (10%) | 1 (9.09%) | 2 (6.45%) |
| Imbalanced nutrition: more than body requirements (00001)a | 1 (10%) | 0 (0%) | 1 (9.09%) | 2 (6.45%) |
| Risk for constipation (00015) | 2 (20%) | 0 (0%) | 0 (0%) | 2 (6.45%) |
| Risk for powerlessness (00152) | 1 (10%) | 1 (10%) | 0 (0%) | 2 (6.45%) |
| Risk for deficient fluid volume (00028) | 1 (10%) | 1 (10%) | 0 (0%) | 2 (6.45%) |
| Impaired dentition (00048) | 1 (10%) | 0 (0%) | 1 (9.09%) | 2 (6.45%) |
| Chronic low self-esteem (00119) | 1 (10%) | 0 (0%) | 0 (0%) | 1 (3.23%) |
| Decreased cardiac output (00029) | 1 (10%) | 0 (0%) | 0 (0%) | 1 (3.23%) |
| Fatigue (00093) | 1 (10%) | 0 (0%) | 0 (0%) | 1 (3.23%) |
| Imbalanced nutrition: less than body requirements (00002) | 0 (0%) | 0 (0%) | 1 (9.09%) | 1 (3.23%) |
| Risk for activity intolerance (00094) | 1 (10%) | 0 (0%) | 0 (0%) | 1 (3.23%) |
| Chronic sadness (00137) | 0 (0%) | 1 (10%) | 0 (0%) | 1 (3.23%) |
| Excess fluid volume (00026) | 0 (0%) | 0 (0%) | 1 (9.09%) | 1 (3.23%) |
| Nursing diagnoses added by the participants | ||||
| Chronic pain (00133) | 9 (90%) | 7 (70%) | 9 (81.82%) | 25 (80.65%) |
| Anxiety (00146) | 2 (20%) | 3 (30%) | 4 (36.36%) | 9 (29.03%) |
| Risk for infection (00004) | 3 (30%) | 2 (20%) | 0 (0%) | 5 (16.13%) |
| Impaired oral mucous membrane (00045) | 0 (0%) | 0 (0%) | 4 (36.36%) | 4 (12.90%) |
| Bathing/hygiene self-care deficit (00108) | 0 (0%) | 1 (10%) | 1 (9.09%) | 2 (6.45%) |
%: percentage of participants who documented the nursing diagnosis.
Gold-standards.
For both case studies, 43 different nursing diagnoses were documented; 18 nursing diagnoses were selected by participants in both case studies, 14 were selected only for case 1, and 11 only for case 2. Table 4 shows the overall diagnostic concordance among participants and according to participants’ groups.
Table 4.
Light’s Kappa values for the diagnostic concordance among the overall sample and according to participants’ groups
| Nursing diagnoses for both cases in alphabetic order | Nursing undergraduates | Nurse residents | Nurses | Total |
|---|---|---|---|---|
| Anxiety (00146)a | 0.012 | 0.016 | −0.054 | 0.040 |
| Chronic low self-esteem (00119) | −0.053 | NE | NE | −0.017 |
| Deficient knowledge (00126) | NE | 0.016 | −0.048 | 0.020 |
| Ineffective control of therapeutic regimen (00078) | NE | −0.059 | −0.048 | 0.001 |
| Impaired walking (00088) | −0.089 | NE | NE | −0.028 |
| Decreased cardiac output (00029) | −0.111 | NE | −0.048 | −0.028 |
| Bathing/hygiene self-care deficit (00108)a | NE | −0.059 | −0.081 | −0.034 |
| Impaired swallowing (00103) | −0.053 | NE | −0.048 | 0.001 |
| Impaired dentition (00048) | −0.053 | NE | −0.048 | 0.001 |
| Ineffective airway clearance (00031) | 0.012 | 0.196 | 0.313 | 0.224 |
| Disturbed body image (00118) | −0.053 | NE | NE | 0.017 |
| Acute pain (00132)b | −0.099 | −0.100 | −0.100 | −0.034 |
| Chronic pain (00133) a | 0.600 | 0.438 | 0.662 | 0.591 |
| Sedentary lifestyle (00168) | 0.012 | NE | NE | 0.001 |
| Fatigue (00093) | −0.089 | NE | 0.010 | −0.001 |
| Hyperthermia (00007) | −0.053 | NE | 0.010 | 0.020 |
| Impaired skin integrity (00046) | 0.012 | 0.016 | 0.010 | 0.080 |
| Impaired tissue integrity (00044) | 0.012 | 0.016 | 0.144 | 0.125 |
| Activity intolerance (00092) b | −0.065 | 0.591 | −0.091 | 0.080 |
| Ineffective health maintenance (00099) | 0.012 | −0.059 | 0.010 | 0.060 |
| Impaired physical mobility (00085) | −0.100 | −0.059 | NE | −0.028 |
| Impaired oral mucous membrane (00045)a | NE | NE | 0.144 | 0.039 |
| Imbalanced nutrition: more than body requirements (00001)b | −0.111 | −0.059 | −0.081 | −0.022 |
| Imbalanced nutrition: less than body requirements (00002) | NE | NE | −0.048 | −0.017 |
| Ineffective breathing pattern (00032)b | 0.169 | 0.196 | −0.054 | 0.126 |
| Impaired sensory perception (00122) | 0.012 | −0.059 | −0.006 | 0.040 |
| Ineffective tissue perfusion (00024)a | 0.194 | −0.059 | 0.127 | 0.126 |
| Risk for constipation (00015) | 0.012 | NE | NE | 0.001 |
| Risk for imbalanced fluid volume (00025) | −0.053 | NE | NE | −0.017 |
| Risk for unstable blood glucose level (00179) a | 0.487 | 0.438 | 1.000 | 0.655 |
| Risk for infection (00004)a | NE | −0.111 | −0.100 | −0.032 |
| Risk for impaired skin integrity (00047) | 0.012 | 0.016 | 0.010 | 0.080 |
| Risk for activity intolerance (00094) | −0.111 | NE | NE | −0.034 |
| Risk for decreased cardiac tissue perfusion (00200) | NE | −0.059 | NE | −0.017 |
| Risk for ineffective peripheral tissue perfusion (00228)a | NE | −0.059 | −0.048 | 0.001 |
| Risk for falls (00155) | −0.111 | −0.059 | −0.054 | −0.019 |
| Risk for powerlessness (00152) | −0.053 | −0.059 | NE | 0.001 |
| Risk for deficient fluid volume (00028) | −0.053 | −0.059 | NE | 0.001 |
| Chronic sadness (00137) | NE | −0.059 | NE | −0.017 |
| Impaired gas exchange (00030) | 0.012 | NE | −0.048 | 0.020 |
| Impaired spontaneous ventilation (00033) | −0.053 | NE | NE | −0.017 |
| Deficient fluid volume (00027) | −0.053 | NE | NE | −0.017 |
| Excess fluid volume (00026) | NE | NE | −0.048 | −0.017 |
%: percentage of participants who documented the nursing diagnosis; NE: not estimable. In bold: diagnoses that had concordance.
Diagnosis added by the participants.
Gold-standards.
For 40 diagnostic labels there was either no overall concordance or poor concordance among study participants. There was high overall concordance (Kappa 0.655) for “Risk for unstable blood glucose level” (00179), moderate (Kappa 0.591) for “Chronic Pain” (00133), and weak (Kappa 0.224) for the diagnosis of “Ineffective airway clearance” (00031). Also, there was moderate concordance (Kappa 0.591) among nurse residents concerning the nursing diagnosis “Activity intolerance” (00092).
DISCUSSION
To the best of our knowledge, this pilot study was the first to evaluate the diagnostic concordance between nurses, resident nurses, and nursing undergraduates who used the CDSS entitled PROCEnf-USP®. We expected that PROCEnf-USP® users even at different levels of education and experience would agree on their decisions about the nursing diagnoses while using PROCEnf-USP® since they all had the same clinical data and the support of decision-making software. Findings from another study suggested that PROCEnf-USP® offers great clinical decision support to less experienced users.18 The diagnostic accuracy determined by undergraduates through PROCEnf-USP® compared to the manual (paper) documentation was verified that nursing graduates determine diagnoses at a high degree of accuracy when using PROCEnf-USP®.18 Another study found that the PROCEnf-USP® supports decision-making, which leads users at different levels of education and professional experience in nursing to select nursing diagnoses with high and moderate accuracy.19
However, our data did not support our hypothesis. We found that for most of the nursing diagnoses there was poor or no concordance among the users. Only biological nursing diagnoses showed concordance among participants, and only one gold-standard, biological, nursing diagnosis (activity intolerance) had moderate concordance among nurse residents. Interestingly, 2 diagnoses with high (risk for unstable blood glucose level) and moderate (chronic pain) overall concordance were not suggested by the system but added by the participants. In fact, “Risk for unstable blood glucose level” (00179) did not exist when the gold standards for the case studies were created. However, at the time of this study, the system was using a more up-to-date version of the NANDA-I classification.25
PROCEnf-USP® and support to clinical decision-making
One explanation that might help understand the low concordance among our participants is the way PROCEnf-USP® operates. While using the system, the user might decide that s/he does not need the support of the system to make a clinical decision. In such a case, clinical data are not provided to the system, which in turn does not suggest diagnostic hypotheses. However, the user documents his/her clinical decision in the system. Therefore, there might be a discrepancy between the suggested diagnostic hypotheses and those nursing diagnoses in fact documented by the user.15 A study that evaluated the concordance of caregivers and nurses, who used a CDSS for the guidance of children undergoing chemotherapy, showed that nurses had greater concordance with the guidelines generated with the aid of CDSS than nurses who did not use the software because of the facility for standardizing clinical procedures on the care of children undergoing chemotherapy.29 Another study analyzed the concordance of nursing diagnoses documentation by nurses using an Electronic Health Records (EHR) compared to paper documentation, found greater concordance among nurses using the EHR.30
It is important to emphasize that all the gold standard diagnoses would have been suggested by the system, had the users provided it with all the necessary clinical information. Our results show that undergraduate students tended to select gold-standard diagnoses more often. This was an unexpected finding that might be explained by the more linear process of clinical reasoning used by less experienced nurses and students.17 Following this linear process, it is possible that the undergraduate students read the written case studies more closely, paying more attention to relevant clues, whereas more experienced users jumped into clinical conclusions without thinking through the entire case.
The possible influence of clinical reasoning skills also has to be considered when interpreting our results. Clinical reasoning is the application of knowledge and experience to a clinical situation. The basis of a nursing diagnosis is clinical reasoning. Clinical reasoning to establish a diagnosis depends on cognitive functions, including short-term memory, which is normally able to handle up to 7 pieces of information simultaneously.31 A previous study aimed at evaluating the effect of the educational program “Guided Clinical Reasoning” and the introduction of an intelligent electronic nursing documentation system on the quality of the nursing process concluded that intelligent systems can support nurses throughout the nursing process, but for meaningful use of the system, clinical reasoning is essential.32 Clinical reasoning should be taught and encouraged in nursing undergraduates, nurse residents, and nurses as a basis for the practice of the Advanced Nursing Process.
Considering the steps of the Advanced Nursing Process (assessment, diagnosis, planned outcomes, interventions, and outcome evaluation), the diagnosis step is fundamental because it determines what will be done in the following steps.4,25 Thus, we can consider that the CDSS supported the diagnostic decision of less experienced users and with fewer years of health education. In order to encourage the use of all functionalities of a CDSS, the system must be fast and avoid an excessive number of clicks.33 A common complaint of PROCEnf-USP® users is that the system is sometimes too slow. Although a number of improvements have been made over the years,34,35 such perception might negatively affect how users choose to use the system, which might result in the selection of less accurate nursing diagnoses. However, it is important to emphasize that PROCEnf-USP® was developed in a collaborative effort among nursing faculty, nursing staff, and informatics experts to ensure that it comprises clinical data relevant to the identification of the central elements of nursing care (diagnosis, intervention, and results).5 In the future, developers of the system should consider integrating PROCEnf-USP® into the electronic health record, as well as integrating voice recognition and artificial intelligence software to optimize PROCEnf’s usability and decision-making support.
Despite our findings, the ability of PROCEnf-USP® to support clinical decision-making has been suggested in the literature. A study that evaluated the decision-making process of nurses in the real world using PROCEnf-USP® revealed that the diagnosis suggestions made by the system were accepted in 70% of cases by the users.36 This finding shows that the system provides structure to support the clinical decision and does not require the nurses to agree on the suggested diagnostic hypotheses, it gives them the opportunity to choose other options, even if the clinical data has not been suggested. Ultimately, this is important because the nurse is the one who decides and takes responsibility for the clinical decision.
Although the level of education and expertise has been found as an important factor in nursing diagnostic accuracy, we found that the concordance even within the same group of participants was low. These findings might be explained by the type of analysis conducted in this study. Light’s Kappa uses coefficients that do not analyze unbalanced cases as many more no’s than yes’s. Light’s Kappa also does not allow analyzing the total agreement for the participants who did not respond to the 2 cases, which is why some nursing diagnoses appear in Table 4 as not estimated (NE).28 In future studies about diagnostic concordance, nurses and residents from different specialties should be included in the sample, as well as more case studies should be used. In addition, the use of other statistical analysis approach is encouraged to account for unbalanced cases.
Limitations
This study has limitations which are worthy of mention. The small sample size (2 cases) is an important limitation of this study. The sample size may interfere with interpretation if the degree of Kappa concordance exceeds the coincidence of randomized assessments.20,28 Although there is no theory developed for the sample sizing of studies with several judges, studies using the Kappa test use larger samples. However, we only used 2 cases because they were the cases validated by experts.
The other limitation in this study is the use of written clinical cases. Written cases did not enable the participants to contact the patients to clarify or complement information, which is oftentimes necessary during the collection of data in the nursing process. The idea that diagnoses with no and low concordance were determined by the inability to contact the patients to clarify or collect more data cannot be refuted. On the other hand, written studies ensure that case information is the same for everyone, which can contribute to concordance among participants.
CONCLUSIONS
There was concordance for only 3 nursing diagnoses related to biological needs. The low level of concordance may be related to the clinical judgment skills and educational level of the participants, the written cases, the sample size, and also the type of test used in the statistical analysis.
Future research should be conducted to better understand the reasons for poor or no concordance on psychosocial and health promotion nursing diagnoses as well as for a clinical reasoning process when health professionals use a decision-making support system.
The development and use of the PROCEnf-USP® standardized language system represent a breakthrough for nursing and serve to demonstrate the contribution of nursing to the health of individuals, allowing the measurement of the result of the nursing care provided and, thus, developing the practice of evidence-based nursing.
Contributor Information
Regina Célia dos Santos Diogo, School of Nursing, University of São Paulo, São Paulo, Brazil.
Rita de Cassia Gengo e Silva Butcher, Florida Atlantic University Christine E Lynn College of Nursing, Boca Raton, Florida, USA; Graduate Program in Adult Health Nursing (PROESA), School of Nursing, University of São Paulo, São Paulo, Braz, il.
Heloísa Helena Ciqueto Peres, School of Nursing, University of São Paulo, São Paulo, Brazil.
FUNDING
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
AUTHOR CONTRIBUTIONS
All the authors have contributed for project conception, data collection and analysis, manuscript preparation, and revision of the final version.
CONFLICT OF INTEREST STATEMENT
None declared.
DATA AVAILABILITY
The dataset used to support the conclusions of this study is available upon reasonable request to the corresponding author.
REFERENCES
- 1. Nibbelink CW, Brewer BB.. Decision-making in nursing practice: an integrative literature review. J Clin Nurs 2018; 27 (5–6): 917–28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Dagliati A, Tibollo V, Sacchi L, et al. Big data as a driver for clinical decision support systems: a learning health systems perspective. Front Digit Humanit 2018; 5: 8. [Google Scholar]
- 3. Musen M, Middleton B, Greenes R.. Clinical decision-support systems. In: Shortliffe E, CIMINO J. eds. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. New York: Springer; 2014: 643–74. [Google Scholar]
- 4. Müller-Staub M, Abt J, Brenner A, Hofer B.. Expert Report on the Responsibility of Nursing. Bern: Swiss Association for Nursing Science ANS; 2015. https://www.vfp-psi.ch/fileadmin/user_upload/Dokumente/Expertenbericht_Englisch.pdf. Accessed June 2, 2023. [Google Scholar]
- 5. Müller-Staub M, de Graaf-Waar H, Paans W.. An internationally consented standard for nursing process-clinical decision support systems in electronic health records. Comput Inform Nurs 2016; 34 (11): 493–502. [DOI] [PubMed] [Google Scholar]
- 6. D’Agostino F, Zeffiro V, Cocchieri A, et al. Impact of an electronic nursing documentation system on the nursing process accuracy. Adv Intell Syst Comput 2019; 804: 247–52. [Google Scholar]
- 7. Kang MJ, Dykes PC, Korach TZ, et al. Identifying nurses’ concern concepts about patient deterioration using a standard nursing terminology. Int J Med Inform 2020; 133 (1): 104016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Marcotullio A, Caponnetto V, La Cerra C, Toccaceli A, Lancia L.. NANDA-I, NIC, and NOC taxonomies, patients’ satisfaction, and nurses’ perception of the work environment: an Italian cross-sectional pilot study. Acta Biomed 2020; 91 (6-S): 85–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Sanson G, Vellone E, Kangasniemi M, Alvaro R, D’Agostino F.. Impact of nursing diagnoses on patient and organizational outcomes: a systematic literature review. J Clin Nurs 2017; 26 (23–24): 3764–83. [DOI] [PubMed] [Google Scholar]
- 10. D’Agostino F, Vellone E, Cocchieri A, et al. Nursing diagnoses as predictors of hospital length of stay: a prospective observational study. J Nurs Scholarsh 2019; 51 (1): 96–105. [DOI] [PubMed] [Google Scholar]
- 11. Barioni EMS, Nascimento CdSd, Amaral TLM, Ramalho Neto JM, Prado PRd.. Clinical indicators, nursing diagnoses, and mortality risk in critically ill patients with COVID-19: a retrospective cohort. Rev Esc Enferm USP 2022; 56: e20210568. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Peres HHC, Cruz DALM, Lima AFC, et al. Development electronic systems of nursing clinical documentation structured by diagnosis, outcomes and interventions. Rev Esc Enferm USP 2009; 43 (spe2): 1149–55. [Google Scholar]
- 13. Oliveira NBD, Peres HHC.. Evaluation of the functional performance and technical quality of an electronic documentation system of the nursing process. Rev Lat Am Enferm 2015; 23 (2): 242–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Hussain A, Mkpojiogu EOC.. An application of the ISO/IEC 25010 standard in the quality-in-use assessment of an online health awareness system. Jurnal Teknologi 2015; 77 (5): 9–13. [Google Scholar]
- 15. Oliveira NB, Peres HHC.. Quality of the documentation of the nursing process in clinical decision support systems. Rev Latino-Am Enferm 2021; 29:e3426. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Dochterman JM, Jones DA, Hahn K.. Unifying nursing languages: the harmonization of NANDA, NIC, and NOC. Int J Nurs Knowl 2004; 15 (2): 34. [Google Scholar]
- 17. Benner P. From Novice to Expert: Excellence and Power in Clinical Nursing Practice. Addison‐Wesley: Publishing Company; 1984. [Google Scholar]
- 18. Peres HHC, Jensen R, Martins TdC. Assessment of diagnostic accuracy in nursing: paper versus decision support system. Acta Paul Enferm 2016; 29 (2): 218–24. [Google Scholar]
- 19. Diogo RCS, Gengo And Silva Butcher RC, Peres HHC.. Evaluation of the accuracy of nursing diagnoses determined by users of a clinical decision support system. J Nurs Scholarsh 2021; 53 (4): 519–26. [DOI] [PubMed] [Google Scholar]
- 20. Miot HA. Agreement analysis in clinical and experimental trials. J Vasc Bras 2016; 15 (2): 89–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Lucena AdF, Laurent MdCR, Reich R, et al. Nursing diagnosis risk for bleeding as an indicator of quality of care for patient safety. Rev Gaúcha Enferm 2019; 40 (spe): e20180322. [DOI] [PubMed] [Google Scholar]
- 22. Mebrahtu TF, Bloor K, Ledward A, et al. ; Cochrane Effective Practice and Organisation of Care Group. Effects of computerised clinical decision support systems (CDSS) on nursing and allied health professional performance and patient outcomes. Cochrane Database Syst Rev 2021; 2021 (3): CD014699. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Matos FGOA, Cruz DALM.. Development of an instrument to evaluate diagnosis accuracy. Rev Esc Enferm USP 2009; 43 (spe): 1088–97. [Google Scholar]
- 24. Peres HHC, Lima AFC, Cruz DALM, Follador NN.. Programas de Residência em Saúde e em Enfermagem. 1st ed. Rio de Janeiro: Atheneu; 2019: 113–30. [Google Scholar]
- 25. Heardman TH, Kamitsuru S.. NANDA International Nursing Diagnoses: Definitions & Classification, 2015–2017. Oxford: Wiley-Blackwell; 2014. [Google Scholar]
- 26. Bussab WO, Morettin PA.. Estatística Básica. São Paulo: Ed. Saraiva; 2017. [Google Scholar]
- 27. Agresti A. An Introduction to Categorical Data Analysis. New York: John Wiley and Sons; 2019. [Google Scholar]
- 28. Landis JR, Koch GG.. The measurement of observer agreement for categorical data. Biometrics 1977; 33 (1): 159–74. [PubMed] [Google Scholar]
- 29. Lopes VJ, Shmeil MAH.. Evaluation of computer-generated guidelines for companions of pediatric patients undergoing chemotherapy. Rev Gaúcha Enferm 2016; 37 (esp): 67407. [DOI] [PubMed] [Google Scholar]
- 30. Graaf-Waar H, Paans W, Müller-Staub M.. Agreement, correctness and accuracy of nursing diagnoses: a randomized trial. ACENDIO 2015 Proceedings. J Nurs Scholarsh 2015: 398–400. https://www.researchgate.net/publication/275541433 [Google Scholar]
- 31. Elstein AS, Schwartz A.. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ 2002; 324 (7339): 729–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Bruylands M, Paans W, Hediger H, Müller-Staub M.. Effects on the quality of the nursing care process through an educational program and the use of electronic nursing documentation. Int J Nurs Knowl 2013; 24 (3): 163–70. [DOI] [PubMed] [Google Scholar]
- 33. Clarke S, Wilson ML, Terhaar M.. Using dashboard technology and clinical decision support systems to improve heart team efficiency and accuracy: review of the literature. Stud Health Technol Inform. 2016; 225: 364–6. [PubMed] [Google Scholar]
- 34. Peres HHC, Cruz DALM, Tellez M, et al. Implementation of improvements in an electronic documentation nursing process system structured on NANDA-I, NOC and NIC (NNN) classification. Stud Health Technol Inform 2016; 225: 1082–3. [PubMed] [Google Scholar]
- 35. Diogo R, Peres HHC, Silva RCG, Ortiz DCF, Maia FOM. Electronic documentation system of the nursing process: software versioning modeling method. In: Anais do Encontro Internacional do Processo de Enfermagem, 2017. Electronic Anais.Campinas: Galoá; 2017. 10.17648/enipe-2017-60463. [DOI] [Google Scholar]
- 36. Peres HHC, Cruz DALM, Tellez M, et al. Usability testing of PROCEnf-USP: a clinical decision support system. Stud Health Technol Inform 2015; 216: 247–50. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The dataset used to support the conclusions of this study is available upon reasonable request to the corresponding author.