Evidence for the Existing American Nurses Association-Recognized Standardized Nursing Terminologies: A Systematic Review

Abstract

Objective

To determine the state of the science for the five standardized nursing terminology sets in terms of level of evidence and study focus.

Design

Systematic Review.

Data sources

Keyword search of PubMed, CINAHL, and EMBASE databases from 1960s to March 19, 2012 revealed 1,257 publications.

Review Methods

From abstract review we removed duplicate articles, those not in English or with no identifiable standardized nursing terminology, and those with a low-level of evidence. From full text review of the remaining 312 articles, eight trained raters used a coding system to record standardized nursing terminology names, publication year, country, and study focus. Inter-rater reliability confirmed the level of evidence. We analyzed coded results.

Results

On average there were 4 studies per year between 1985 and 1995. The yearly number increased to 14 for the decade between 1996–2005, 21 between 2006–2010, and 25 in 2011. Investigators conducted the research in 27 countries. By evidence level for the 312 studies 72.4% were descriptive, 18.9% were observational, and 8.7% were intervention studies. Of the 312 reports, 72.1% focused on North American Nursing Diagnosis-International, Nursing Interventions Classification, Nursing Outcome Classification, or some combination of those three standardized nursing terminologies; 9.6% on Omaha System; 7.1% on International Classification for Nursing Practice; 1.6% on Clinical Care Classification/Home Health Care Classification; 1.6% on Perioperative Nursing Data Set; and 8.0% on two or more standardized nursing terminology sets. There were studies in all 10 foci categories including those focused on concept analysis/classification infrastructure (n = 43), the identification of the standardized nursing terminology concepts applicable to a health setting from registered nurses’ documentation (n = 54), mapping one terminology to another (n = 58), implementation of standardized nursing terminologies into electronic health records (n = 12), and secondary use of electronic health record data (n = 19).

Conclusions

Findings reveal that the number of standardized nursing terminology publications increased primarily since 2000 with most focusing on North American Nursing Diagnosis-International, Nursing Interventions Classification, and Nursing Outcome Classification. The majority of the studies were descriptive, qualitative, or correlational designs that provide a strong base for understanding the validity and reliability of the concepts underlying the standardized nursing terminologies. There is evidence supporting the successful integration and use in electronic health records for two standardized nursing terminology sets: (1) the North American Nursing Diagnosis-International, Nursing Interventions Classification, and Nursing Outcome Classification set; and (2) the Omaha System set. Researchers, however, should continue to strengthen standardized nursing terminology study designs to promote continuous improvement of the standardized nursing terminologies and use in clinical practice.

Introduction

Nursing terminologies, a body of standardized terms for the practice and science of nursing, are essential to capture, represent, access, and communicate nursing practice data. Terminologies are also critically important to discover practice-based knowledge and to conduct research related to the quality and effectiveness of nursing care. Around the world, there are many formal and informal nursing terminologies, but five nursing terminology sets (each set consists of diagnosis, intervention, and outcome terms typically used together) have been recognized by the American Nurses Association for more than two decades (McGonigle and Mastrian, 2012). Despite the longstanding availability of these terminology sets, we found only one article with frequency counts for publications related to these terminology sets (Anderson et al., 2009) and no articles with a systematic review of the scientific base for standardized nursing terminology sets. A review of the research that summarizes the strengths and weaknesses of the data-driven evidence base for the standardized nursing terminology sets can help inform decisions relative to development of nursing content in practice and provide direction for research to address the remaining gaps. The purpose of this article is to present a systematic review of the data-based literature for the five American Nurses Association recognized standardized nursing terminology sets.

Now, electronic health records are mandated in many countries and although nursing tasks are well represented in current electronic health records, the intellectual component of nursing care (nursing diagnoses or problems, interventions, and outcomes) is typically missing. Barriers to inclusion of these vital data are the oral communication traditions between nurses and insufficient use of standardized nursing terminologies in clinical practice. As electronic health record use expands across the globe, efforts have increased to capture the work of nurses in a computerized format, which signifies now as an opportune time to ensure that standardized nursing terminology sets are properly integrated into electronic health records. Computerization of standardized nursing terminologies could improve the consistency, content, and format of nursing communication and by so doing enhance the effectiveness and efficiency of the information shared among nurses, other healthcare providers, the public, and third-party payers. Nurse leaders and decision makers are finding it difficult to adopt standardized nursing terminology sets (Meyer et al., 2007) due to the number available and the absence of clear selection criteria and directions for use (Lundberg et al., 2008). There currently is no systematic review of standardized nursing terminology research available in the literature, and we believe that such a review would help administrators and clinical nurses make appropriate decisions about standardized nursing terminologies and speed the adoption and appropriate use of standardized nursing terminology sets in practice. standardized nursing terminology implementation is costly and difficult to reverse when insufficient evidence was used to guide the selection and use. This article is the first step to provide insights about the research evidence available on the five American Nurses Association recognized standardized nursing terminology sets (1) NANDA-I (North American Nursing Diagnosis-International), NIC (Nursing Interventions Classification), and NOC (Nursing Outcome Classification) (together known as NNN), (2) ICNP (International Classification for Nursing Practice), (3) Omaha System, (4) CCC/HHCC (Clinical Care Classification/Home Health Care Classification), and (5) PNDS (Perioperative Nursing Data Set).

Literature Review

Nursing has a long history of developing standardized nursing terminologies, almost four decades. In North America, the American Nurses Association operates a process for this standardization through its Committee for Nursing Practice Information Infrastructure. American Nurses Association-recognized standardized nursing terminologies by set appear in Table 1. Use of standardized nursing terminologies is expected to increase the visibility of nursing and generate the data needed to demonstrate the impact of nursing care on patient outcomes (Keenan et al., 2008). Implementation of standardized nursing terminologies improves the quality of documented nursing diagnoses, nursing interventions, and patient outcomes. If collected and stored systematically such data can be used to improve the quality and safety of care by enabling the identification and diffusion of best practices and elimination of poor practices (Kautz and Van Horn, 2008, Muller-Staub, 2009).

Table 1.

Characteristics of American Nurses Association-Recognized Standardized Nursing Terminologies (2010)

Standardized Nursing Terminology Sets	Year initially recognized	Nursing problems	Nursing outcomes	Nursing interventions	Population Focus
NANDA-I+	1992	X			Comprehensive
NOC+	1997		X		Comprehensive
NIC+	1992			X	Comprehensive
ICNP	2000	X	X	X	Comprehensive
OMAHA System	1992	X	X	X	Home care / Community based
CCC/HHCC*	1992	X	X	X	Comprehensive
PNDS	1999	X	X	X	Perioperative care

Note: + the five standardized nursing terminology sets include seven since NANDA-I, NOC and NIC are recognized as separate terminologies and but part of the single standardized nursing terminology set -- NNN;

^*HHCC was changed to the CCC System in 2003 and originally focused on home care

Communication between members of the health care team is necessary for continuity of care and patient safety. Nurses are central to care planning and communicating vital care information (Keenan and Yakel, 2005). Using the standardized nursing terminologies in the care planning process provides a framework for documenting nursing care that enhances communication among all members of the health care team (Scherb and Weydt, 2009). Using standardized nursing terminologies in direct documentation helps promote and sustain a shared understanding of care and progress toward desired patient outcomes (Keenan and Yakel, 2005).

Time management, work complexity assessment, and economic analyses can be positively affected by the use of standardized nursing terminologies. The specificity in the outcome indicators of the NOC standardized nursing terminology accounted for a decrease in documentation time (Moorhead et al., 1998). Consequently, nurses are able to spend more time with patients instead of documentation (Moorhead et al., 1998). Use of standardized nursing terminology interventions and outcomes in a work complexity assessment was also associated with improved patient satisfaction (Scherb and Weydt, 2009).

Standardized nursing terminologies have been used to calculate staffing ratios (Lundberg et al., 2008) and to capture nursing care costs (Saba and Arnold, 2004). However, economic studies using standardized nursing terminologies are rare, due to the complexity of analysis and the paucity of nursing data collected and properly coded with standardized nursing terminologies and stored in electronic health records (Stone et al., 2004). Stone et al. (2004) nonetheless argue for the implementation of standardized nursing terminologies as they make it possible to examine the economic value of nursing by providing data in a format that can show the impact of nursing care on patient outcomes (e.g., as measured by the standardized nursing terminology NOC).

Unfortunately, we were not able to find a published review of the research on standardized nursing terminologies. One article (Anderson et al, 2009), investigators of a bibliometric count of all articles on one or more of the American Nurses Association recognized standardized nursing terminologies reported that the vast majority were written about NANDA-I, NOC, and NIC and that the most prolific authors of NNN articles had the deepest and broadest co-author networks within and across standardized nursing terminology sets. We believe that a systematic review of the research available on the five American Nurses Associated recognized standardized nursing terminology sets is an important step to promoting the appropriate selection and use of standardized nursing terminologies that will achieve the desired benefits. The specific aim of this study was to determine the research base for the five American Nurses Association-recognized standardized nursing terminology sets (NNN, ICNP, Omaha System, CCC/HHCC, & PNDS), including the component of the standardized nursing terminology studied, the level of evidence generated by the study, the focus of the study, and characteristics of the study sample.

METHODS

Search Strategy

We queried three databases (PubMed, CINAHL, EMBASE) to obtain the initial list of potential articles for systematic review. Since development of S standardized nursing terminologies began in the 1960s, our searches included articles published in that decade through March 19, 2012. Table 2 lists the initial search terms for each database and terminology. From consultation with a librarian information science specialist we identified specific terms in accord with the list of key words for each database to assure that we identified the research-based literature for all the American Nurses Association approved standardized nursing terminologies. We identified and downloaded 1,257 references into EndNote X4 (Thompson Reuters ISI ResearchSoft, 2010), a reference management program. We deleted duplicate articles and those without abstracts; we retained 796 articles (Figure 1) (the list of references is available online at http://www.tneel.uic.edu/download.asp).

Table 2.

Search Terms Used to Identify Articles for the Study

Terminology	Database
	Pubmed	CINAHL	EMBASE
NANDA-I	MeSH Term: Nursing diagnosis and NANDA: and Journal Article	CINAHL Heading Major concept “NANDA nursing diagnoses” and Advanced search Research+Journal article	Emtre “NANDA and “nursing diagnosis” and Article
NOC	Text Term: “nursing outcomes classification”: and Journal Article	CINAHL Heading Major concept “Iowa Nursing Outcomes Classification”  And Advanced search Research+Journal article:	Emtre “nursing outcomes classification”  and Article
NIC	Text Term: “nursing interventions classification” And Journal Article	CINAHL Heading Major concept “Iowa Nursing Interventions Classification”   and Advanced search Research+Journal article	Emtre “nursing interventions classification” and Article
ICNP	All field Mesh ‘nursing diagnosis’: and Journal Article	CINAHL Heading Major concept “International Classification for Nursing Practice”  And Advanced search Research+Journal article	Emtre “International Classification for Nursing Practice”  and Article
OMAHA system	Text Term Mesh ‘nursing diagnosis’: and Journal Article	CINAHL Heading Major concept “OMAHA system” and Advanced search Research+Journal article	Emtre “OMAHA system ” And Article
CCC	Text Term: Mesh ‘nursing diagnosis’, and Journal Article	CINAHL Heading Major concept “Saba Clinical Care Classification”  and Advanced search Research+Journal article	Search; “Clinical Care Classification ” And Article
HHCC (became CCC in 2003)	Text Term Mesh ‘nursing diagnosis’: And Journal Article	CINAHL Heading: 0 Search; “HHCC” and Advanced search Research+Journal article	Emtre “HHCC” and Article
PNDS	All field Mesh ‘nursing diagnosis’ and Journal Article	Search; “Perioperative Nursing Data Set” and Advanced search Research+Journal article	Search; “PNDS and Article

Figure 1.

Flow diagram illustrating the selection process for publications reviewed

Selection Criteria

We conducted a two-stage review, first of abstracts and then of full text articles. In stage one as indicated in Figure 1, we examined the 796 English-language abstracts and retained 609 articles in any language, that appeared to include at least one of the five American Nurses Association recognized standardized nursing terminology sets, and evidence level Ia to VI, based on Polit’s and Beck’s (2012) criteria (Table 3). We excluded abstracts with evidence level VII, which we defined as the opinions of authorities, case studies, and non-systematic and non-integrated literature reviews.

Table 3.

Levels of Evidence, Definitions, and Frequency by the American Nurses Association-Recognized Standardized Nursing Terminologies^*

Level of Evidence	Definition	NANDA-I+	NOC+	NIC+	NNN+	ICNP	Omaha	CCC/ HHCC	PNDS	Combo
I. a. Systematic review of Randomized Clinical Trials	Integrated findings from multiple Randomized Clinical Trial studies using rigorous methods.	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0
I. b. Systematic review of non-Randomized Clinical Trials	Findings from multiple non-randomized trials using rigorous methods.	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0
II. a. Single Randomized Clinical Trial	Study that involve an intervention, randomization, and control of an independent variable.	2 (1)	1 (0)	1 (0)	1 (1)	0 (0)	0 (0)	0 (0)	0 (0)	0
II. b. Single Non-Randomized Clinical Trial	Quasi-experimental study with intervention but lacks randomization.	4 (1)	3 (0)	4 (1)	3 (3)	0 (0)	0 (0)	0 (0)	0 (0)	0
II. c. Single Non-Randomized Clinical Trial w/o control	As II. B, but also lacks control.	10 (6)	4 (0)	6 (1)	5 (5)	0 (0)	7 (7)	1 (1)	0 (0)	0
III. Systematic review of non-randomized studies	Integrated findings from multiple non-randomized intervention studies using rigorous methods	1 (1)	1 (1)	0 (0)	0 (0)	0 (0)	1 (1)	0 (0)	1 (1)	0
IV. Single correlational	Studies that explore the interrelationships among variables of interest without researcher intervention.	29 (23)	12 (9)	12 (7)	5 (5)	2 (1)	9 (8)	1 (0)	0 (0)	2
V. Systematic review of descriptive or qualitative studies	Systematic review of the multiple descriptive or qualitative Systematic review for development of new term	3 (1)	2 (0)	3 (1)	2 (1)	2 (1)	1 (0)	1 (0)	0 (0)	1
VI. Single descriptive or qualitative	Study focus to observe, describe and document aspects of a phenomenon as it naturally occurs without control of an independent variable.	120 (74)	38 (15)	73 (35)	38 (33)	29 (20)	25 (14)	14 (4)	8 (4)	22
Total*		169 (107)	61 (25)	99 (45)	54 (48)	33 (22)	43 (30)	17 (5)	9 (5)	25

^*The total outside the parentheses exceeds 312 because articles covering multiple standardized nursing terminology sets are counted multiple times; the total number inside the parentheses represents the number of articles exclusive to the designated standardized nursing terminology and across the standardized nursing terminology sets is equal to 312; + represents one standardized nursing terminology set, but listed separately because each component of the set (NANDA-I, NIC, NOC) can be studied by itself; NNN refers to NANDA-I, NOC, and NIC; Combo refers to studies that examine aspects of more than one standardized nursing terminology set.

For the second stage, we reviewed 381 articles that were retained from stage one, written in English, and available in electronic format. Based on the full-text review, we retained the articles that unequivocally contained at least one of the five American Nurses Association-recognized standardized nursing terminology sets and represented a Ia to VI level of evidence. Excluded were articles with evidence level VII. A total of 312 of the 381 articles met these criteria (Figure 1) (the list of the 312 references is available online at http://www.tneel.uic.edu/download.asp).

Inter-rater Reliability

Abstract Reviews

As explained above, the first stage of our process based on the abstract, was to determine whether the evidence level of an article was higher than VII and whether it contained at least one of the five American Nurses Association-recognized standardized nursing terminology sets.

A group of eight Doctor of Philosophy prepared faculty members or doctoral students in nursing informatics received training to classify the abstracts and were retrained based on their inter-rater reliability. Specifically, we first created an abstract training set using a random selection process and provided training sessions to assist reviewers to reliably classify the abstracts by level of evidence and the standardized nursing terminology(s) studied. For subsequent trainings, we selected those that represented difficult coding examples and were different than previously used for training.

Following the training for the stage one reviews, each reviewer rated the abstracts for 24 articles, selected randomly from the database. The reviewers were asked to determine, based on the abstracts, the levels of evidence presented and the standardized nursing terminologies included in the articles. We found moderate agreement for level VII vs all other levels of evidence (Fleiss’ Kappa = .52; positive agreement 89%, negative agreement 63%) reported in each of the studies. The inter-rater reliability was fair (Fleiss’ Kappa = .38; positive agreement 80%, negative agreement 58%) for absence of a standardized nursing terminology.

The research group met to discuss the difficulties and disagreements that arose regarding rating some of the abstracts and received retraining. The content expert selected 12 abstracts for a second round of inter-rater reliability testing. Every reviewer evaluated the same 12 abstracts. After retraining, the Kappa was .62 (positive agreement 95%, negative agreement 67%) for evidence level rating and .32 (positive agreement 92%, negative agreement 40%) for the standardized nursing terminology rating. The raters then coded the 796 abstracts (Figure 1). For each abstract, two raters were chosen randomly from eight to determine whether the article included a standardized nursing terminology, and if so, whether the article evidence level was Ia to VI, meaning it was a descriptive research design or higher. The Kappa for the standardized nursing terminology ratings was .80 with positive agreement 96% and negative agreement 84%. The Kappa for the evidence level ratings was 0.68 with positive agreement 91% and negative agreement 76%. These numbers indicate good inter-rater reliability and low chance of excluding articles by mistake based on the evidence level or the standardized nursing terminology ratings. To further reduce the chance of mistaken exclusion, two raters were asked to review all abstracts with at least one “no standardized nursing terminology” or one “Evidence level VII” rating and an abstract was excluded only if both raters agreed.

Article Reviews

Two senior faculty experts reviewed a randomly selected set of 50 articles and coded each on standardized nursing terminologies included, evidence level, and study focus. The definitions of the criteria for study focus appear in Table 4. We compared the consensus of the two raters with the original ratings. On standardized nursing terminologies included, the agreement rate was 96%, indicating that the original ratings on standardized nursing terminologies were highly reliable. On evidence level, the agreement rate was 82%, indicating good reliability of the original ratings on evidence level. On study focus, the agreement rate was 74%. We found the reliability for the three focus areas related to the implementation and use of standardized nursing terminologies in practice (areas 6–8) was low with an agreement rate of 18%. Excluding the 43 articles coded as focus area 6–8, the agreement rate for the remaining articles was 83%.

Table 4.

Definitions of the Criteria for Study Focus

Concept Analysis (evaluation of a single concept, e.g., NNN) / Classification Infrastructure (infrastructure = organization of all terms into meaningful categories) Typical question: What are the defining attributes or indicators, or activities that characterize term X – May or may not provide rigorous procedures reliability and validity Examples: Determine domain +/or classes to classify terms or evaluate completeness –Use of theory to create taxonomy Assessment of reliability and/or validity for set of standardized nursing terminology terms or measures or instruments Examples: IRR, internal consistency, diagnostic accuracy, term meaning reliabilities, construct/criterion validities Instrument example: validation of QDIO (Quality of Diagnosis, Interventions, and Outcomes) instrument for use in examining RN (registered nurse) documentation of standardized nursing terminologies RNs perceptions of potential/actual term usage (full or partial standardized nursing terminology set) Example: Surveys distributed in health care setting, by professional organization, or internet survey asking RNs about CCC term usage for cardiac patients Identification of standardized nursing terminology terms that apply to a health setting from RN documentation 4a1. Standardized nursing terminology terms were in documentation; or 4a2. Documentation was recoded into standardized nursing terminologies 4b1. Paper documentation or 4b2. Electronic documentation or 4b3. Documentation mode not indicated Mapping one terminology (or more) to another terminology Examples: NANDA-I to Omaha; NNN to ICNP to show likenesses/gaps, natural language of documentation to CCC Implementation of 1 or more terminologies into PAPER documentation system Example: Author describes training and processes/content of standardized nursing terminology implementation and measures used to evaluate success Implementation of 1 or more taxonomies into ELECTRONIC documentation system Example: Author describes training and processes/content of standardized nursing terminology implementation and measures used to evaluate success Evaluation of USE or diffusion of standardized nursing terminology into practice Examples: % of NNN Content in leading textbooks, observation of RN documentation with QDIO Evaluation of differences/similarities between/among standardized nursing terminologies Examples: ICNP has definitions no other content, NNN has defining attributes, ICNP is multi-axial NNN is not Use of point of care standardized nursing terminology documentation for secondary uses Examples: HANDS data mining, determining workload & continuity of care, studies that use parts of standardized nursing terminologies to evaluate NON standardized nursing terminology defined intervention/s (may collect standardized nursing terminology measure only for study

Copyright © 2012 HANDS Workgroup, used with permission

RESULTS

The 312 research studies were conducted in 27 countries (Figure 2). The number of publications increased steadily starting between 1985 and 1995 and averaging four studies per year (Figure 3). The yearly number of publications more than tripled to 14 for the decade between 1996 and 2005, increased further to 21 from 2006 to 2010, and increased to 25 in 2011 (Figure 3,2012 is not included because the search included less than 3 months of that year). By evidence level for the 312 studies, 72.4% were descriptive, 18.9% were observational, and 8.7% were intervention studies (Table 3).

Figure 2.

Number of articles per country (n=27) for the 312 reviewed articles

Figure 3.

Average number of articles published per year for the 312 reviewed articles

Frequency of Studies by Standardized Nursing Terminologies

All seven of the standardized nursing terminologies were studied at least once. There were 169 NANDA-I articles, 99 on NIC, 61 on NOC, 54 on NNN (any 2 or all 3 of NANDA-I, NOC and NIC), 9 on PNDS, 33 on ICNP, 17 on CCC/HHCC, and 43 on Omaha System. The frequency of standardized nursing terminologies by the levels of evidence appears in Table 3.

Study Focus

Table 5 presents the frequency of study focus by the standardized nursing terminologies. The focus of the studies was diverse and included development and implementation of the standardized nursing terminologies.

Table 5.

Frequency of Study Focus by the Standardized Nursing Terminology Sets

Study Focus	NANDA-I+	NOC+	NIC+	NNN	ICNP	Omaha	CCC/HHCC	PNDS	Combo
Concept analysis/classification infrastructure	32 (30)	5 (3)	5 (3)	2 (2)	5 (5)	0 (0)	0 (0)	0 (0)	0
Assessment of reliability or validity for set of terms	29 (23)	12 (8)	7 (3)	5 (5)	2 (1)	4 (4)	1 (1)	2 (2)	1
Mapping one terminology (or more) to another terminology	21 (9)	2 (1)	17 (9)	5 (3)	20 (13)	11 (3)	10 (1)	4 (2)	17
Evaluation of differences/similarities between standardized nursing terminologies	2 (0)	1 (0)	1 (0)	1 (0)	2 (0)	1 (0)	1 (0)	0 (0)	2
RN perceptions of potential/actual term usage (full or partial standardized nursing terminology set)	20 (14)	11 (5)	23 (16)	8 (7)	1 (1)	1 (0)	0 (0)	1 (1)	1
ID of standardized nursing terminology terms that apply to a health setting from RN documentation	32 (18)	10 (2)	20 (7)	13 (13)	1 (1)	12 (12)	0 (0)	1 (0)	1
Evaluation of use or diffusion of standardized nursing terminology into practice	16 (6)	6 (1)	12 (2)	10 (9)	2 (1)	3 (2)	2 (1)	1 (0)	1
Implementation into paper documentation system	4 (3)	1 (0)	1 (0)	1 (1)	0 (0)	4 (3)	0 (0)	0 (0)	1
Implementation into electronic documentation system	7 (1)	7 (1)	6 (1)	6 (6)	0 (0)	3 (3)	0 (0)	0 (0)	0
Secondary uses of point of care standardized nursing terminology documentation	6 (3)	6 (4)	7 (4)	3 (2)	0 (0)	4 (3)	3 (2)	0 (0)	1

Note: + represents one standardized nursing terminology set, but listed separately because each component of the set (NANDA-I, NIC, NOC) can be studied by itself; NNN refers to a combinations of NANDA-I, NOC, and NIC; numbers in parentheses represents the frequency exclusive to the designated standardized nursing terminology; Combo refers to combination of multiple terminologies; ID refers to identification

Development

Most of the 312 articles focused on standardized nursing terminology development issues. There were 43 studies in which the primary focus was on a category we named concept analysis (to define the attributes, indicators, or activities of a specific term) / classification infrastructure (organization of all terms into meaningful categories). A total of 48 studies focused on assessment of validity and reliability for a set of standardized nursing terminology terms, measures or instruments. Such articles typically addressed internal consistency, diagnostic accuracy, reliabilities for the meaning of terms, or construct/criterion validity of the set of terms. Related to standardized nursing terminology development were studies of concept mapping (n=58) the standardized nursing terminology or natural language to another standardized nursing terminology (e.g., mapping natural language to the standardized nursing terminology, translation to another language, or mapping the standardized nursing terminology to other standardized terminologies).

Investigators conducted studies (n=45) of registered nurses’ perceptions of potential or actual use of terms (full or partial standardized nursing terminology set). Typically, these studies were surveys distributed in a health care setting, by professional organization, or via Internet asking registered nurses about the use of terms for a specific group of patients.

Other investigators conducted studies (n=54) to identify standardized nursing terminology terms applied to a health setting from registered nurses’ documentation and when doing so the standardized nursing terminology terms were in the actual documentation record (n=36) or the documentation was recoded into standardized nursing terminologies (n=18). For those studies, where the standardized nursing terminology terms were documented in the actual record, the documentation was paper-based (n=12), electronic (n=17), or not indicated (n=7).

Implementation

A total of 64 studies focused on implementation. Studies focused on implementation of one or more of the standardized nursing terminologies into a paper documentation system (n=8) or an electronic documentation system (n=12). Investigators conducted studies to evaluate the use or diffusion of standardized nursing terminology into practice (n=23). Additionally, investigators conducted studies (n=19) of the use of point-of-care standardized nursing terminology documentation for secondary uses, such as data mining for knowledge generation, determining workload and continuity of care, or studies that used parts of a standardized nursing terminologies to evaluate a non standardized nursing terminology defined intervention. Finally, two articles focused on comparison of different terminologies: one compared NANDA-I with ICNP, and the other one contrasted NANDA-I-NOC-NIC with ICNP, Omaha, and CCC/HHCC.

Except for the mapping study focus, there was very little overlap across different terminologies. Studies with a focus on implementation, evaluation of use or diffusion, and secondary use of standardized nursing terminology gathered data primarily included NANDA-NOC-NIC or Omaha as the standardized nursing terminologies. In these studies, there was substantial overlap between NANDA, NOC, and NIC, meaning that the study reported in many of the articles included all three terminologies.

Sample Characteristics

The 312 articles represented considerable diversity in terms of sample characteristics. Regarding the study settings, 6 involved health systems (e.g., Kaiser, Advocate, District); 143 involved hospitals; 38 involved ambulatory centers including nurse managed or community centers; 12 involved long-term care facilities; 28 involved home care facilities; 56 involved units (within the previous settings); and 119 involved other settings (e.g., local, regional or national survey by a professional organization).

The sample types included the entire population (n=56), a randomly selected sample (n=35), a purposive sample (n=73), a convenience sample (n=158), or not reported (n=24). The total number of subjects ranged from 1 to 530,218, with a mean = 2,035, SD = 29,199, median = 78; the 25% quantile was 30 and the 75% quantile was 258. Since the second largest sample was 13,760 and much smaller than the largest sample of 530,218, we also computed the descriptive statistics excluding the largest sample: mean = 429, SD = 1,439, median = 77; the 25% quantile was 30 and the 75% quantile was 258. The quantiles provide additional information on the center and spread since the distribution of sample sizes is heavily skewed.

The source of data was patient records (registered nurse/student nurse used standardized nursing terminologies at point of care) (n=107); patient records (actual documentation coded to standardized nursing terminology by another person) (n=35); patients (e.g., surveyed, interviewed, observed) (n=32); registered nurses (e.g., surveyed, interviewed, observed) (n=98); students (n=6); or other (e.g., articles, concepts/terminologies, surveys of directors/deans, patient messages, statements from town hall meetings, family members of patients, nurse experts, patient/care giver pairs, characteristics of body image) (n=75).

Discussion

We are the first to report the results of a systematic review of the world’s English-language research on the five existing American Nurses Association-recognized standardized nursing terminology sets. A substantial number of articles was revealed by our standardized nursing terminology search strategies. Annually, the number of standardized nursing terminology publications has increased primarily since 2000; NANDA-I-NOC-NIC had the most, especially increasing since the early 1990s, and PNDS had the least. In fact NANDA-I-NOC-NIC had nearly nine times more publications than Omaha, the standardized nursing terminology with the second highest frequency. The vast majority (72%) of the studies were descriptive designs; nearly 20% were observational studies, and nearly 9% were intervention studies. We found evidence ranging from validation of the standardized nursing terminology concepts to the successful integration and use of standardized nursing terminologies in the electronic health record. Specifically, both the NANDA-I-NOC-NIC and Omaha standardized nursing terminology sets have been studied as they were implemented at the point of care with documentation in paper-based medical records or electronic health records. No evidence was found supporting use of the remaining three standardized nursing terminology sets in electronic health records. There were only two randomized clinical trials of any standardized nursing terminology set and only five controlled intervention studies that lacked randomization. Although there were mapping studies comparing one standardized nursing terminology to another, no evidence was found to indicate that any standardized nursing terminology was more effective than the others. Findings indicate that researchers should continue to strengthen standardized nursing terminology study designs to promote continuous improvement of the standardized nursing terminologies and to further improve the scientific base that supports these nursing-practice focused terminologies.

From the escalating number of standardized nursing terminology research publications, 25 per year in 2011, it is apparent that a large number of researchers are studying standardized nursing terminologies around the world. We had resources to review only the English publications, but the list of 796 publications (online resource available http://www.tneel.uic.edu/download.asp) is a useful resource for others to contribute to literature synthesis of those that we were not able to review.

The majority of the study designs were descriptive, qualitative, or correlational. The development work provides a strong base for understanding the validity and reliability of the concepts underlying the standardized nursing terminologies, including studies of more than one standardized nursing terminology set. These findings are consistent with the nursing discipline’s strong value for assuring validity and reliability in research. As well, there are some reviews that synthesize the non-experimental research on standardized nursing terminologies. The findings of this literature synthesis show that an important strength of the standardized nursing terminology research is the careful attention that scientists have given to the foundation of the standardized nursing terminologies that began in the 1960s, grew steadily for several decades and now has escalated in the 21^st century.

Findings also show a dearth of intervention studies involving the implementation of standardized nursing terminologies (only 27 of the 312 articles), especially well controlled studies with random assignment to experimental conditions. As well, the lack of experimental studies influences the possibility of literature synthesis reports, which to date is non-existent. The nature of research focused on implementing standardized nursing terminologies likely requires a healthcare system change that will not occur if randomized clinical trial research is required to justify the change. Creativity in study design is necessary to advance the science of standardized nursing terminology research. Given that well designed observational studies often reveal findings consistent with randomized clinical trials (Benson and Hartz, 2000, Ioannidis et al., 2001), perhaps well designed studies that take advantage of natural experiments can provide a sufficient level of evidence to advance knowledge of the benefits and challenges of standardized nursing terminologies used at the point-of-care. The findings of this literature synthesis show that an important weakness of the standardized nursing terminology research is the lack of evidence to support the influence of standardized nursing terminology use on patient outcomes or other important healthcare related outcomes. Extensive work has occurred to carefully develop the standardized nursing terminologies and to initiate clinical use of some of the standardized nursing terminologies, and now a logical extension of this work is to expand systematic use of standardized nursing terminologies in practice settings to identify best practices and carefully evaluate the impact on healthcare-related outcomes.

From this review, we note a critical gap that brings opportunity. A key benefit of the clinical use of standardized nursing terminologies to nursing science as a whole is the potential for comparisons of patient data, including outcomes, across settings. This benefit, however, has yet to be realized, perhaps in part because of the number of different standardized nursing terminologies and the rarity of their actual and consistent use in practice across health care settings. Extensive research has occurred to carefully develop the standardized nursing terminologies and to initiate clinical use of some of the standardized nursing terminologies, but now there is an urgent need to link the use of standardized nursing terminologies to patient outcomes that allow nurses to demonstrate the effectiveness of our interventions. One major problem noted in this review is that researchers tend to design and study standardized nursing terminology implementation for single or limited sites, which adversely affects the generalizability of the findings. To ultimately use standardized nursing terminology coded data to evaluate the effectiveness of nursing care on patient outcomes, standardized nursing terminologies must be integrated in ways that produce interoperable data. To produce interoperability, the standardized nursing terminologies must be integrated into user interfaces and databases consistently across healthcare organizations and not uniquely tailored to each setting. Since randomized clinical trials are infeasible for this purpose, researchers need to create novel ways of testing standardized nursing terminology implementations so that the results are broadly generalizable. Keenan et al (2012), published after we initiated this review, is one example of a study design that allowed the cost effective evaluation of an electronic plan of care system that used NNN deployed identically across eight diverse units located in four different hospitals in one Midwestern state in the United States.

This review is not without limitations. Because of our sampling strategy, we did not review research that could have been published in books or sources not indexed by the selected databases. Although we used an extensive set of search terms and strategies to uncover the associated standardized nursing terminology literature, it is possible that other search terms would have identified other publications. We believe this issue did not affect our study because our broad search strategy was specific to each standardized nursing terminology and to each database and was informed by librarian information science expertise to assure that we found all the relevant literature. The massive number of studies involved with this literature synthesis allowed us to provide an overview of the level of evidence supporting standardized nursing terminologies in the English-language literature. Our inter-rater reliability on study focus was 74%, which means that there could be some error in our findings, especially for the implementation related focus areas with low reliability (6–8). The expensive training and retraining that was required to achieve a sufficient level of inter-rater reliability in the review emphasizes the need for additional and more focused review of selected bodies of this research. Before we initiated this project, we had hoped to provide a greater synthesis of the knowledge provided by the body of research. Once we learned the extent of the literature, we refocused our goal to achieve a reliable and meaningful overview of the strengths and weaknesses of the research. We recommend that future literature synthesis is needed both within and between standardized nursing terminologies to extend the knowledge synthesis in focused areas that are beyond the scope of this review. Our online resources, however, should provide a valuable resource to others who seek to extend this work, particularly to synthesize the research published in languages other than English.

In conclusion, academic and clinical nurse leaders should consider the evidence supporting the use of both NNN and Omaha standardized nursing terminology sets, especially as nurse leaders design and implement content for electronic health records. Other standardized nursing terminologies may be appropriate for electronic records but research is needed to demonstrate effective use of these in an electronic health record. Use of standardized terminologies may allow comparison of practice across settings, but the number of different standardized nursing terminology sets and variety of methods of integrating standardized nursing terminology sets into electronic health records are currently impeding that comparison. Clearly, this lack of consistency in database architecture and methods of capturing items coded with standardized nursing terminologies needs immediate attention to enable meaningful data comparisons across settings. As such, researchers should continue to strengthen standardized nursing terminology study designs to promote continuous improvement in the use of the standardized nursing terminology sets for nursing documentation. More research is needed to inform the decision about which standardized nursing terminology is most appropriate for an electronic health record in a particular practice setting. Substantial research funding is needed to move this research forward so that electronic health records will adequately document the intellectual contribution of nurses to patient outcomes.

What is already known about the topic?

• -Currently seven American Nurses Association-approved standardized nursing terminology sets can be used to represent nursing diagnoses, interventions, and outcomes in medical records.
• -A bibliometric review of all articles (all types) appearing in the literature on at least one of the American Nurses Association-recognized standardized nursing terminologies (Anderson et al, 2009) reported that the vast majority were written about NANDA-I, NOC, and NIC and that the most prolific authors of NNN articles had the deepest and broadest co-author networks within and across standardized nursing terminology sets.
• -Extensive research that has been conducted since the 1960s underpins these standardized nursing terminology sets, but no synthesis of all this literature exists to assist administrators and electronic health record vendors as they make decisions about inclusion of nursing documentation in the paper-based medical record or the electronic health record

What this paper adds?

• -We found evidence of the pattern of evolutionary development of standardized nursing terminology science ranging from the creation and iterative refinement of the standardized nursing terminology structures, concept development (for diagnosis, outcome, and intervention terms and measures) and validation, designating terms for practice areas, the successful integration into practice documentation (paper and electronic) and the use of data coded with standardized nursing terminologies to describe practice and the impact on outcomes achieved.
• -Both the NANDA-I-NOC-NIC and Omaha standardized nursing terminology sets have been studied as they were implemented at the point of care with documentation in paper-based medical records or electronic health records.
• -Given the potential of standardized nursing terminologies to enable nursing effectiveness research, we believe the review indicates an imperative to conduct standardized nursing terminology implementation studies that cross multiple institutions and electronic health records and by so doing to produce generalizable results. Such findings are critically needed to guide standardized nursing terminology set selection and integration decisions that will produce interoperable nursing data.