Trends in Publication of Nursing Informatics Research

Abstract

We analyzed 741 journal articles on nursing informatics published in 7 biomedical/nursing informatics journals and 6 nursing journals from 2005 to 2013 to begin to understand publication trends in nursing informatics research and identify gaps. We assigned a research theme to each article using AMIA 2014 theme categories and normalized the citation counts using time from publication. Overall, nursing informatics research covered a broad spectrum of research topics in biomedical informatics and publication topics seem to be well aligned with the high priority research agenda identified by the nursing informatics community. The research themes with highest volume of publication were Clinical Workflow and Human Factors, Consumer Informatics and Personal Health Records, and Clinical Informatics, for which an increasing trend in publication was noted. Articles on Informatics Education and Workforce Development; Data Mining, NLP, Information Extraction; and Clinical Informatics showed steady and high volume of citations.

Introduction

Information technologies were identified as key factors in achieving improved patient safety and quality of care and the published literature is starting to confirm this role^(1–3). Biomedical informatics, including nursing informatics, is a fast moving field that is heavily influenced by healthcare policy and clinical practice. Professional and/or academic organizations have been dedicating substantial effort to facilitate nursing informatics research and training that address current needs. For example, many nursing education programs offer training opportunities specialized in nursing informatics and also mandate an introductory nursing informatics class to all students as part of their core curriculum^(4–7). The Nursing Informatics Working Group (NIWG) at AMIA addresses various issues in policy, research, and educational aspects of nursing informatics. The NIWG also offers two award programs to recognize noteworthy research presented in the annual AMIA symposium to encourage participation of nurse scholars ⁽⁸⁾.

Scientific journals are a venue for dissemination of research to a large community of readers. The proportion of peer-reviewed articles published in a certain area of research (and their citations) can provide interesting insights on topic trends in informatics^(9,10). Although it may not constitute a perfect measure, the number of citations is considered a de facto standard for measuring the impact of a scientific publication. Based on this idea, we have previously analyzed the publication and citation volumes in biomedical informatics that were published relatively recently (2009–2012) in the J Amer Med Inform Assoc (JAMIA) to better understand research topic trends for biomedical informatics in general.

We conducted a similar analysis to describe the trends in published nursing informatics research by analyzing the articles published on selected major informatics and nursing journals during the past 9 years. We also compared the active research areas reflected in these articles against the nursing informatics research agenda proposed by nursing informatics leaders^(11–13). Through this analysis, we aimed at (1) checking the trajectory of published nursing informatics research for the past 9 years, and (2) identifying potential gaps in particular research areas.

Background

In 1993, the National Institute of Nursing Research (NINR) of National Institutes of Health (NIH) sponsored a group of nursing informatics scholars in the investigation of research needs in nursing informatics with the purpose of developing a nursing informatics research agenda⁽¹⁴⁾. Healthcare and biomedical research are changing rapidly due to the availability of advanced technologies to collect and analyze large volumes of data.

Many nursing informatics leaders have proposed an updated research agenda over the past decade. In 2007, the AMIA NIWG proposed comprehensive nursing informatics agenda in nursing practice, education, and research ⁽¹³⁾. A year later, Bakken et, al. identified the areas that nursing informatics research needed to expand further to better accommodate fast moving biomedical sciences⁽¹²⁾. In 2012, the Nursing Informatics International Research Network (NIIRN) conducted an international survey of 468 nursing informatics researchers across the globe soliciting opinions on research topics that should be prioritized⁽¹¹⁾. The research agenda identified in these works is summarized in Table 1.

Table 1.

Nursing informatics research agenda

AMIA-NIWG (2007)(13)	Bakken, et al. (2008)(12)	NIIRN (2012)*(11)
• Secondary use of clinical data • Use of aggregated & de-identified data • Data privacy and confidentiality • Data security and related technical infrastructure • Population health • Development and use of standards	• User friendly tools for data analysis visualization, and modeling • Evaluation methods of human and organizational factors on health IT • Consumer empowerment • Use of genomics and environmental data • Reengineering of nursing practice	• Clinical information systems that provide real-time feedback • Evaluation of the impact of health IT system on patient outcome • Nursing decision support systems • Evaluation of the impact of health IT system on nursing workflow • Management of nursing data for research and patient care • Training nurses in health IT • Identifying nursing outcomes that are important to patients

^*Only the top 7 highest priority items out of 20 are listed.

Recently, Carrington et, al. reported on informatics topics that were most actively researched, the types of research, and research settings by reviewing 69 peer-reviewed articles with a nurse as the first author, published between Aug 2011 and Aug 2012⁽¹⁵⁾. They identified three broad topics of research: (1) clinical informatics research that deals with various clinical information applications such as Electronic Medical Records (EMR) and bar-code medication administration, (2) human factors such as human computer interaction and communication, and (3) data interoperability such as terminology/standardization and care transition/handoffs⁽¹⁵⁾. This work provided an important snapshot view of the research trends in that given year and provided valuable insights on the status of nursing informatics research.

Materials and Methods

Article collection

We retrieved articles relevant to nursing informatics published between 2005 and 2013 using ISI’s Web of Science⁽¹⁶⁾. Here, the “articles relevant to nursing informatics” was defined according to the disciplines covered by the selected journals, keywords found in the articles, and authors’ affiliations. Articles published in nursing informatics journals were considered relevant. Articles published in general biomedical informatics journals were considered relevant when at least one of the authors was affiliated with a nursing institution (e.g., nursing schools, nursing research institutes, centers of nursing excellence, etc.). Additionally, articles published in non-informatics nursing journals were considered relevant if they contained the keyword “informatics” in the title or abstract.

Nursing informatics research is published in a large number of biomedical informatics and/or engineering journals. We included one nursing informatics journal (CIN: Computers, Informatics, Nursing) and seven general biomedical informatics journals (Journal of Medical Internet Research, Journal of American Medical Informatics Association, Medical Decision Making, International Journal of Medical Informatics, Journal of Biomedical Informatics, BMC Medical Informatics and Decision Making, Method of Informatics in Medicine) in this analysis. These journals encompass a wide range of informatics topics and are among the top 20 journals with highest 5-year impact factors in the category “Medical Informatics” in ISI’s Web of Science ⁽¹⁶⁾. Four¹ of these eight journals were also recognized in a prior work as the journals in which the most nursing informatics works were published ⁽¹⁵⁾.

As many nursing informatics scholars publish informatics relevant works on non-informatics journals, we also included six nursing journals (i.e., International Journal of Nursing Studies, Journal of Advanced Nursing, Nursing Research, Journal of Nursing Scholarship, Nursing Outlook, Journal of Nursing Administration) that publish general clinical nursing research as well as administrative, and managerial topics. These journals are among the top 20 nursing journals with highest 5-year impact scores under the category “Nursing” of ISI’s Web of Science⁽¹⁶⁾. This journal selection approach undoubtedly has limitations, which are described in the discussion section.

We retrieved citation information of selected articles from the ISI’s Web of Science⁽¹⁶⁾ using its citation report function. A total of 741 articles were included in this analysis, which excluded white papers, opinion papers, duplicated articles, and articles without citation reports. The largest number of articles (N=406) was collected from the nursing informatics journal, 306 from biomedical informatics journals, and 29 from nursing journals. The paper selection process is summarized in Figure 1.

Figure 1.

Article selection process

Citation score normalization

The citation report obtained from the ISI’s Web of Science⁽¹⁶⁾ shows the annual citation counts of an article for the indicated time period. Using the same method reported in the prior work^(9,10) we produced normalized citation scores to each paper to minimize the bias introduced by the differences in publication dates. For instance, an article published in January of 2006 has more chances of being read and cited than an article published in December of 2011 as it has been available to research communities for a longer period of time. Therefore we used a unit of measure that reflects the number of citation per month (CPM) since publication, as adopted in the prior work ^(9,10).

The CPM scores were calculated separately for each year (i.e., annual CPM = number of citation in a given year / number of months an article was available in a given year) as well as for the entire period (i.e., overall CPM = total number of citations / total number of months an article was available). The two calculations were performed to account for the fact that articles that are old accrue a relatively small number of citations per month after several years of publication, and articles that are recent also accrue a relatively small number of citations in the first two years after publication. Hence articles published in the middle of the accounted period (i.e, 2009–2010) would account for higher overall CPMs than articles at the extremes. However, they should result in comparable annual CPM when the number of years after publication is taken into account.

Research theme assignment

We assigned a research topic category to each article using the AMIA themes proposed for the 2014 annual symposium⁽¹⁷⁾. Although the articles provide a list of keywords and/or are indexed with the terms from Medical Subject Heading (MeSH), they may not provide a single category label that represents the main topic of the study with sufficient expressivity for subdomains of informatics. We decided to use the AMIA 2014 themes after investigating various potential alternatives, such as chapter headings from biomedical informatics textbooks and subjects listed in informatics training curricula^(18,19). Although AMIA themes may focus on the contemporary topics of biomedical informatics research, they were deemed viable options for this work, as they are expressive labels designed to capture a single main topic of a research work for the purposes of directing a diverse set of attendees to the presentations that are more relevant to their interests.

We also developed annotation rules and a process shown in Figure 2 to maximize annotation consistency and accuracy. The theme categories that were deemed straightforward to use (i.e., Public Health Informatics and Biosurveillance, Translational Bioinformatics and Bioinformatics, Global eHealth, Imaging Informatics) are not shown in Figure 2 to simplify the display of the process. In addition, we did not use the theme category Meaningful Use as it can directly relate to many other informatics themes and was not relevant to older articles.

Figure 2.

Topic assignment process

Two authors (HK, XJ) independently annotated the main research topics of randomly selected 100 articles using the AMIA 2014 themes. The annotation results were collaboratively reviewed and the discrepancies were resolved by reaching consensus. Also, the theme assignment rules and process were refined as necessary. One author (HK) who is trained in nursing informatics assigned themes to the remaining 641 articles following the refined rules and process. Metadata for the 741 articles with themes are available at https://idash-data.ucsd.edu/folder/504.

Trend analysis

We generated descriptive statistics on distributions of the research topics and their citation scores. We also tested whether these distributions differ significantly among journal types and whether they changed significantly over the years. SPSS (version 22) and Matlab were used to analyze the data.

Results

Among 741 articles collected, 701 were research articles and 40 were review articles. A total of 117 articles reported on work that targeted specific clinical information systems. The most frequently studied areas referred to Electronic Medical Records (EMR, N=61), Clinical Decision Support System (CDSS, N=33), Computerized Provider Order Entry (CPOE, N=10), and bar-coding medication administration (N=10).

Distribution of research themes

Most of the themes were utilized in the categorization of articles, except for the themes Imaging Informatics and Translational Bioinformatics and Biomedicine for which there were no articles. No article was left unassigned. The research theme most frequently appearing for the past 9 years was Clinical Workflow and Human Factors. Global eHealth was the least frequent (Figure 3). We analyzed the trend of publication frequencies for each research theme over the years using a parametric method. The article frequencies were used as the predictor variables and the publication year was used as the response variable to fit a linear regression model. Two parameters (constant and slope) were calculated and we conducted a t-test to check whether they were statistically significant. In Figure 3, the starred (*) themes are the ones that show statistically significant positive trends (i.e., increment of the article counts).

Figure 3.

Annual article volume and Citation per Month (CPM), and overall CPM by Themes

Publications focusing on Consumer Informatics and PHR and Mobile Health have been continuously growing. In addition to these two themes, Clinical Informatics and Clinical Workflow and Human Factors are the areas where the high volume of nursing informatics articles has been published for the past 3 years. Also it is noteworthy that the nursing informatics articles on Biomedical Data Visualization and Global eHealth started showing up relatively recently.

The articles published in informatics journals represented the broadest theme areas. Two most frequent themes from the nursing informatics journal were Informatics in Health Professional Education, and Clinical Workflow and Human Factor. Consumer Informatics and PHRs, and Clinical Workflow and Human Factor are the themes most frequently appearing in general biomedical informatics journals. Half of the articles from nursing journals were on Informatics Education and Workforce Development (Figure 4). The differences in theme distributions among journal types were statistically significant when tested with the Pearson Chi-Square test (p<0.005).

Figure 4.

Overall Citations Per Month (CPM) distribution by journal type

Research impact

The top 10 most highly cited articles (based on overall CPM) are about Informatics Education and Workforce Development (ranked 1^st)⁽²⁰⁾, Biomedical Data Visualization (2^nd)⁽²¹⁾, Clinical Workflow and Human Factor (3^rd)⁽²²⁾ and Consumer Informatics and PHRs (4^th–10^th). The first ranked article was published in a non-informatics nursing journal and the other 9 articles were published in general biomedical informatics journals.

Informatics Education and Workforce Development; Data Mining, NLP, Information Extraction; and Clinical Informatics were the research themes that were associated with the consistently high annual CPMs (Figure 3). However, Consumer Informatics and PHRs came out as having the highest overall CPM, followed by Global eHeath and Informatics Education and Workforce Development. It is noteworthy that the articles on Global eHealth and Biomedical Data Visualization themes showed quite high overall CPM scores in spite of their relatively short article available time. This indicates that these two themes are fastest growing research areas in nursing informatics but this observation needs to be revalidated when more data are accumulated. The overall CPMs for Data Mining, and NLP, Information Extraction and Clinical Informatics still remain high.

Alignment with the nursing informatics research agenda

We first aligned the nursing informatics research agenda items presented in Table 1 with the AMIA themes. We then checked the number of articles published in those themes and the overall CPM and the most recent stable annual CPM scores as proxy information that reflects how active research in a particular agenda item has been. A research agenda item was assigned with only one theme even if it can be related to multiple themes. We excluded the agenda item “reengineering nursing practice” as it encompasses a full spectrum of research themes.

The agenda items were related to nine of the seventeen AMIA themes considered in this study. The eight of the nine mapped themes covered more than 60% of the articles analyzed in this study. The average overall CPM score of the articles on these eight themes was 0.102 (s.d. = 0.033). Of note, the average overall CPM score of the seventeen themes used in this study was 0.010 (s.d. = 0.027) (Table 2).

Table 2.

Distribution of research themes by journal types

AMIA Themes	Nursing Informatics Journal	Nursing Journal	Biomedical Informatics Journal	Total
Clinical workflow and human factor	76	2	47	125
Consumer informatics and PHR	46	1	68	115
Clinical informatics	60	3	44	107
Mobile health	59	0	36	95
Informatics in health profession education	76	5	4	85
Informatics education and workforce development	22	15	5	42
Data interoperability and information exchange	21	0	14	35
Simulation and modeling	10	1	23	34
Clinical research informatics	12	0	14	26
Terminology and standards ontologies	5	1	14	20
Interactive system	10	0	6	16
Data mining, NLP, information extraction	5	0	9	14
Biomedical data visualization	2	0	7	9
Policy and ethical issues	0	1	8	9
Public health informatics and bio-surveillance	1	0	6	7
Global ehealth	1	0	1	2
Total	406	29	306	741

Discussion

We observed that the nursing informatics articles covered a broad spectrum of topics in biomedical informatics over a 9-year window. Highest volumes of articles were associated with Clinical Workflow and Human Factor, Consumer Informatics and PHRs, and Clinical Informatics, for which an increasing trend in publication was also noted. An increasing publication trend was also observed for the themes like Mobile Health, Informatics in Health Profession Education, Simulation and Modeling, and Biomedical Data Visualization.

In terms of citation counts, themes related to informatics training, clinical informatics and data mining dominated, possibly because they are established areas of research. When we adjust citation counts with publication dates (i.e., overall CPM), Consumer Informatics and PHRs and Global eHealth stood out as the most highly cited themes. However, the overall CPM of Global eHealth was calculated based on the 2 articles published only 2–3 years ago. Therefore, this high overall CPM may indicate temporary attention caused by the paucity of relevant publications thus a longer period of observation is needed to substantiate this trend.

We found that education topics published in non-informatics nursing journals are very highly cited compared to those published in informatics journals. In addition, we found that the articles published in the nursing informatics journal have relatively lower overall CPMs than the articles published in general biomedical informatics journals and non-informatics nursing journals. (Figure 5). This is probably related to the size of readership (the audience pool), which is much smaller for nursing informatics as it is for nursing in general, or for informatics in general.

Although most of the high priority areas of nursing informatics research were well covered, areas that may require more attention – related to public health informatics and bioinformatics – were also noted.

It is noteworthy that Clinical Workflow and Human Factor is among the most frequently published and cited themes in nursing informatics research. This is not surprising considering the large amount of time that nurses spend working with clinical information system, especially EMRs⁽²³⁾. This research trend may also imply that nurses can make significant contributions to mitigating various EMR usability issues. In July of 2013, the Office of National Coordinator (ONC) hosted a meeting with stakeholders to discuss usability issues in EMRs. Representatives from a healthcare informatics professional group, an EMR certification agency, and the American Medical Association were invited and had a chance to provide testimonials. Common criticism was that designs and workflows of many EMRs largely focused on meeting the Meaningful Use (MU) requirements rather than usability^(24–26). These testimonials addressed the usability issues commonly faced by various healthcare professionals. Also not many MU core/menu objectives are directly related to nursing documentation. Nonetheless it would be beneficial to actively seek nursing perspectives on the usability issues, considering that nurses are the major workforce for documenting and generating clinical data, as well as being heavily involved in the research on usability and workflow aspects of clinical information systems.

The findings of this study may not reflect the most comprehensive and accurate trends of nursing informatics research due to several limitations. Firstly, the operational definitions of nursing informatics research we adopted were arbitrary: research related to informatics published in nursing journals OR research done by a researcher affiliated with a nursing institution and published in an informatics journal. Ideally, nursing informatics research should be defined as any scholarly work that investigated informatics aspects relevant to nursing research, practice and education. However, operational definitions that could be readily applicable to literature search were unavailable. Use of crude definitions of nursing informatics research may have caused either the inclusion of non-nursing informatics articles or exclusion of nursing informatics articles.

Secondly, we collected articles from a small subset of informatics and nursing journals. For example, we excluded conference papers with no citation information and specialty nursing journals, where many practical applied informatics articles are published, as they can be most beneficial to practicing nurses. Thirdly, although we used CPM to minimize the bias introduced by publication dates, CPM is still not completely free from the bias given that trends captured in the early age of articles might drastically change later. Finally, even if we found AMIA themes were robust and clear enough to label the articles collected for this study, there were cases that the authors struggled with assigning proper themes. Like in many studies that involve with human annotation of concepts, this study also carries subjectivity and inconsistency issues in theme assignment.

Conclusion

We conducted a descriptive analysis of nursing informatics articles published in the past 9 years to understand overall trends in publication of nursing informatics research in the peer-reviewed literature and to identify the research areas that may be relatively under-represented. Overall, we found that published nursing informatics research covered a broad spectrum of research topics in biomedical informatics and is aligned with high priority research agenda items advocated by the nursing informatics community. Despite limitations related to relatively narrow coverage of journals, the findings allow us to initiate a dialogue with journal editors and the nursing informatics community at large about the differences in maturity of research in each of the themes, and potentially guide authors and readers on which venue currently provides the highest number of articles related to a given theme.

Table 2.

Volume of articles, overall CPM, and most recent annual CPM (2012) by research agenda

Agenda items (proposed by)	Themes	Articles N (%)	CPMs: overall (2012)
Clinical information systems that provide real-time feedback (NIIRN)	Clinical Informatics	107 (14.44)	0.105 (0.130)
Nursing decision support system (NIIRN)
Secondary use of clinical data (AMIA-NIWG)
User friendly tools for data analysis visualization, and modeling (Bakken et, al.)
Evaluation of the impact of health IT system on patient outcome (NIIRN)
Evaluation methods of human and organizational factors on health IT (Bakken et, al.)	Clinical Workflow and Human Factor	125 (16.87)	0.109 (0.112)
Evaluation of the impact of health IT system on nursing workflow (NIIRN)
Use of aggregated & de-identified data (AMIA-NIWG)	Clinical Research Informatics	26 (3.51)	0.069 (0.061)
Data privacy and confidentiality (AMIA-NIWG)
Data security & related technical infrastructure (AMIA-NIWG)
Management of nursing data for research patient care (NIIRN)
Consumer empowerment (Bakken et, al.)	Consumer Informatics and PHRs	115 (15.52)	0.148 (0.087)
Identifying nursing outcomes important to patients (NIIRN)
Population health (AMIA-NIWG)	Public Health Informatics and Biosurveillance	5 (0.94)	0.054 (0.028)
Use of genomics and environmental data (Bakken et, al.)	Translational Bioinformatics & Biomedicine	0	NA
Development and use of standards (AMIA-NIWG)	Terminology and Standards Ontologies	20 (2.70)	0.086 (0.070)
	Data interoperability and information exchange	35 (4.72)	0.059 (0.063)
Training nurses in Health IT (NIIRN)	Informatics Education and Workforce Development	27 (5.67)	0.137 (0.153)