Models of Collaboration and Dissemination for Nursing Informatics Innovations in the 21^st Century

Abstract

Nursing informatics innovations are constantly adapting to a rapidly changing healthcare environment. Four case studies of informatics projects led by nurse scientists were described and analyzed through the lens of the Informatics Research Organizing Model (IROM) which was modified to include policy and interoperability contexts. The comparison analysis examined dynamic relationships between processes and constructs in nursing informatics interventions and also highlighted the scientific, intellectual property, technical, and policy challenges encountered among the four case studies. The analysis provided implications for future intervention development and implementation in consideration of multiple contexts.

Introduction

The American Nurses Association (ANA) defined nursing informatics as a “specialty that integrates nursing science, computer science, and information science to manage and communicate data, information, knowledge, and wisdom in nursing practice. Nursing informatics supports consumers, patients, and providers in their decision‐making in all roles and settings. This support is accomplished through the use of information structures, information processes, and information technology.” (ANA, 2008) While this definition remains relevant today, the policy drivers for nursing informatics specialists have evolved to meet the demands of the 21^st century. Accordingly, research in nursing informatics is constantly adapting. In this paper, we first review key policy drivers in the 21^st century with an emphasis on nursing informatics research. Then, we describe and analyze four informatics research projects led by nurse scientists using the lens of the Informatics Research Organizing Model (IROM) as a strategy to compare and contrast technical and scientific approaches. We selected the IROM from among the many informatics frameworks because of its explicit handling of context, which we modified by including policy considerations. Based upon this analysis, we describe key lessons for nurse scientists conducting informatics research in the current policy context.

Key Policy Drivers

Health Information Technology for Economic and Clinical Health Act

The enactment of the Health Information Technology for Economic and Clinical Health (HITECH) Act in 2009 spurred health institutions and practices throughout the nation to implement health information technology (HIT) (The U.S. Department of Health & Human Services, 2009). For example, over 4800 hospitals have received payment incentives for participation in the Medicare and Medicaid Electronic Health Record (EHR) Incentive Program (Centers for Medicare and Medicaid Services, 2016). These developments in the HIT landscape have been supported by regulations to guide the use and deployment of technology with the aim of achieving real, measurable benefits for health institutions, professionals, and patients.

Meaningful Use, Federal HIT Strategic Plan 2015 – 2020 and ONC’s A Shared Nationwide Interoperability Roadmap (2015)

Meaningful Use Regulations, a program led by the Centers for Medicare and Medicaid Services (CMS) and the Office of the National Coordinator (ONC) for HIT were developed to set requirements and deadlines for the implementation and use of certified EHRs (Centers for Medicare and Medicaid Services, 2018). The latest effort, stage 3 of Meaningful Use, seeks to achieve the interoperability of health information by ensuring that certified EHRs are connected to be able to support the electronic exchange of health information. Stage 3 requires that this integration be able to demonstrate improvements in the quality of care for patients while maintaining adequate privacy and security. The development of local and national health information exchange (HIE) is one initiative promoted to achieve these goals.

More recently, the Federal HIT Strategic Plan (2015–2020) from the ONC outlines how the federal government aims to use HIT to leverage lower costs while achieving higher quality care and greater engagement from the population (ONC, 2015). The ONC’s shared nationwide interoperability roadmap further highlights the importance of integrating patient-generated health data into care delivery and research (ONC, 2018). Nurses, regarded as the most honest and ethical profession by the public, are in an ideal position to use HIT tools and patient-generated health data to engage patients in their own care and promote connected health and care.

21^st Century Cures Act

In 2016, the 21^st Century Cures Act was signed into law to accelerate innovations in research and medical products to help patients in need of faster access and efficiency (The 114th United States Congress, 2016). In 2017, ONC engaged with stakeholders in support of the 21^st Century Cures Act Trusted Exchange Framework and Common Agreement that prioritized the cohesiveness and interoperability of HIT. However, in the clinical implementation of HIT, a developer or researcher can encounter significant barriers related to intellectual property and governance. In cases where an innovation is shown to be effective in one EHR system, considerable reprogramming and retrofitting may be required for it to function in another. One possible solution to this issue is the adoption of HIT standards, including common data models. Yet, the implementation of such projects is subject to internal administrative decision-making and technology customization, which may be out of the control of nurse informaticians.

Modified IROM

To plan, design, and implement HIT effectively, the social and organizational context of care must be considered. Although several frameworks exist in informatics to guide system development, few take account of context. Effken (2003) The IROM is a comprehensive conceptual model developed at a high level of abstraction to guide many types of nursing informatics research (Effken, 2003; Figure 1). The IROM incorporates the elements of the Systems Development Life Cycle (SDLC) within the broader framework of the Systems Research Organizing Model (Effken, 2003). The IROM’s concepts of client, context, outcomes, and action focus (adapted to nursing informatics innovation in the IROM) reflects the reciprocal nature of the concepts (Brewer, Verran, & Stichler, 2008). The SDLC, which is widely used in HIT project management, guides the design, implementation, and evaluation process, creating many opportunities to develop and validate research hypotheses. The SDLC is a unidirectional cycle that directs the planning, analysis, design, implementation, and maintenance of a nursing informatics innovation within the IROM. The IROM incorporates evaluation as an iterative step, implying that evaluation is continuous and diffused at many stages in the innovation process. Outcomes evaluated with the IROM can include the information flow, knowledge, quality of decisions, and the effect of actions taken to improve quality, safety, satisfaction, and cost (Effken, 2003). Evaluation can be carried out using logic models, program evaluation methods, or formal research studies. Figure 1 operationalizes the concepts of the IROM for the projects conducted individually by the authors, and adding policy under “Context”.

Figure 1.

Modified Informatics Research Organizing Model (Modified from Effken, 2003)

Study Purpose

In light of the policy context and the need to develop, test and deploy informatics for nursing our study purpose was to: 1) analyze four informatics research projects from the lens of the IROM, and 2) describe lessons learned and decision making to inform design choices for future informatics projects. Issues around interoperability, intellectual property, decisions to build a new system or adapt an existing system, and the potential impact of decisions on patient outcomes are presented.

Methods

The constructs of the IROM were used as a lens to complete a cross-case analysis of 4 different research projects. Use of the IROM was selected to highlight how each case study addressed different stages of the SDLC and to identify lessons learned. These projects addressed unique populations and were conducted across varied healthcare settings (e.g., urban, rural, intensive care, primary care, and outpatient patient education). Variations in the approaches to technology design, intellectual property management, and project implementation are presented. While Institutional Review Board (IRB) approval was obtained by each author for their respective projects, this cross-case analysis did not involve human subjects. Hence, IRB approval was not necessary for this project.

Over 10 months, face-to-face and electronic meetings were conducted to collaboratively identify and analyze the application of the IROM elements across the cases using a multiple case study approach (Eisenhardt, 2007; Stake, 2013). This cross-case approach mirrored one taken when applying community-based participatory research principles to engage underserved populations in informatics research (Unertl et al., 2016).

The four cases varied in terms of study procedures, populations, metrics captured, and analytic approaches. Hence, merging their results was not possible. However, applying a cross-case qualitative comparison was used to identify a set of key ideas that others may find helpful as they create, implement, and disseminate technologies. For that reason, the collection and analysis of data for the cross-case comparison followed a descriptive qualitative approach. In initial meetings, the author team described their individual projects and the components thereof using the IROM concepts (Table 1). Their stage of development according to the SDLC was identified (Table 2), and then development decisions were described (Table 3). Together, the tables served as data collection structures to facilitate the cross-case qualitative analysis. The authors reviewed and discussed study details and examined patterns across the cases to identify key lessons learned. Group discussions held via conference calls allowed deeper explorations of decisions and approaches and identified patterns across cases in addition to individual differences.

Table 1.

Components of the Informatics Research Organizing Model by Case

Project	Context	Client	Informatics Intervention	Evaluation/Outcomes
Connect Diabetes (Jing Wang, Chu, Li, Hayes, & Siminerio, 2018; J. Wang, Coleman, Kanter, Ummer, & Siminerio, 2018; Wang J, 2015)	Cultural (organizational culture): Paper-based food diaries that can be tracked by educators to set behavioral goals Economic: Patients are increasingly using apps and wearable devices for tracking diet and physical activity and prefer various brands Social: Increase patient-educator communication; patient-centered medical home Physical (implementation): Diabetes education programs in primary care and other ambulatory settings urban and rural Houston, TX, and Pittsburgh, PA Policy: None of the major hospitals or clinics were willing to connect mobile data to EHR as of 2013 when the project started. Diabetes educators typically use two systems: EHR for general patient records and Chronicle system for diabetes education specific documentation. Interoperability of these two systems	Users of the system: diabetes educators Intervention target: Type 2 diabetes patients Data: Diet and physical activity behaviors	Patients: Behavioral self-monitoring using mobile app and wearable fitness tracker Diabetes educators: Tailored diabetes selfmanagement education using mobile selfmonitoring data Content: Calendar view in two tabs on patient self-monitoring of diet and activity data Structure: This connected interface is housed in an existing platform used by diabetes educators for documentation. Flow of information: From patients to educators Technology characteristics: Web-based electronic system for diabetes education documentation Proprietary or not: Interoperability Patients: commercial apps or fitness trackers that are compatible with the Validic system. Diabetes educators: freely available for ADA-recognized education programs once activated.	Usability: of connected interface, System usability scale think aloud, cognitive walkthrough of connected interface prior to the design of wireframe, after the wireframe, and iterative evaluation, real-world usability evaluation in a 2-week pilot and 3-month feasibility trial. Feasibility: Capacity to use this tool to facilitate diabetes education sessions to be more personalized Acceptability: Patients and diabetes educators’ views on the acceptability of this connected interface Patient/clinical outcomes: Efficacy Patient A1 c change and weight loss Effectiveness Future studies in evaluating this tool.
ml SMART(J. A. Mallow et al., 2015; J. A. Mallow, Theeke, L. A., Walls, R., Theeke, E., & Mallow, B. K., 2016a, 2016b)	Cultural: Patients experience multiple chronic conditions and health disparity. Providers need to provide frequent, intensive intervention that decreases burden for patients. Physical: Free primary care clinic in rural Appalachia Economic: Low-income patients who still have access to internet and mobile devices. Social: Increase patient and provider communication and patient centered care. Policy: Policies for federal payers were improving and Medicare pays for services that provide live, interactive videoconferencing at the time of development. Regionally, no system was willing to connect mobile data to EHR due to potential legal considerations. Locally, Most of the care provided at the clinic is by nurse practitioners. Interoperability: Providers have to use log into multiple systems to provide comprehensive care to patients.	Users of the system: Rural patients and healthcare providers. Intervention target: Patients with multiple chronic conditions and health disparities Data: Blood glucose, blood pressure, pulse, and weight from mHealth monitors; error logs from application; page hits and time in application for patients and providers	Patients: Access patient facing application that enables primary care from a distance. track diagnoses, medications, lab results, receive reminders for self-management, perform self-monitoring for multiple chronic conditions, obtain feedback in real time, engage in education, and attend eVisits (video conferencing) Providers: Access provider facing application to deliver education, prescribe selfmanagement via mHealth devices, monitor patient self-monitoring results, provide feedback and intervention, perform assessment and intervention via video conferencing. Content: Track diagnoses, medications, laboratory results, receive reminders for selfmanagement, perform self-monitoring, obtain feedback in real time, engage in education, and attend video visits Structure: Web based platform to display results of self-management results using mHealth devices Flow of information: two way between patients and healthcare providers. Technology characteristics: Web-based system for primary care delivery and access. Proprietary or Not: Proprietary	Feasibility: Ability to receive and transmit health information wirelessly in rural areas. Ability to complete primary care visits without travel. Acceptability: Evaluated cost, quality, safety, interface usability, and patient satisfaction Patient/clinical outcomes: Adherence to treatment/Delivery of guideline-recommended care via technology Efficacy: Self-management ability, quality of life, depression, blood glucose, blood pressure, and body weight. Effectiveness: Future studies.
NEC-Zero Clinical Decision Support (Gephart et al., 2017; Gephart, Wyles, & Canvasser, 2018)	Cultural: Directed at clinicians and not specifically addressing cultural factors for patients. Fit to clinician workflow and designed to align with their clinical decision-making approaches. Economic: Goal to reduce barriers to use of the technology across systems but did not want it to become the property of a single EHR. It is costly to recreate CDS across systems. Social: Distributed decisionmaking in the NICU so that clinicians can identify decisions made by others. Physical: Neonatal intensive care unit Policy: 21st Century Cures Act, MACRA and HITECH prioritize interoperable clinical decision support to support high quality care and lower cost. Local CDs and clinician governance groups involved with designing the CDS. Local policies that support or limit access to FHIR apps have affected their readiness to implement.	Users: Neonatal clinicians making decisions Intervention target: Very low birth weight neonates within the first 2 months of life Data: Clinician role, neonate risk, feeding, lab, medications, and care factors from the EHR. Data is accessed via API request using FHIR.	Patients: Premature infants weighing < 1500 grams at birth. Clinicians: Neonatologists, neonatal nurse practitioners, nurses, pharmacists, physician assistants Content: Text-based and dashboard displays of guidance and clinical information to support clinical decisions that align with the NEC-Zero evidence-based intervention. NEC-Zero includes early, exclusive access to human milk; use of a standardized feeding protocol; avoiding prolonged early antibiotics and any antacids; and a structured risk assessment with early warning score for NEC. Structure: in-line with EHR using a CDS hook to link from within the EHR to the dashboard which displays outside of the EHR. The CDS architecture is separate from the data that is generated within the EHR. Flow of information: EHR-generated data used to populate the dashboard. Rules trigger guidance within the dashboard based on data that is fed in from the EHR. Technology characteristics: Service-oriented architecture that lays over the EHR within a hospital’s firewall with CDS engine separated from EHR. Proprietary or not: Connecting platform (designed to work across platforms-e.g. Cerner, GE Centricity, Sorian) is proprietary with components (i.e. dashboard) available open source.	Feasibility: FHIR appropriateness and need for extensions to accommodate the neonatal intensive care data elements. Case examples of NICUs using the knowledge in NEC-Zero to reduce NEC. Acceptability: Adherence score deemed acceptable by clinicians, data used to populate display identified via clinician focus groups. Clinician satisfaction and usability assessment planned. Efficacy: Testing is in progress to evaluate effect on patient and CDS outcomes. Effectiveness: Future studies
EnTICE3 (Arcia, 2018; Unertl et al., 2016)	Cultural: Latino research subjects Physical: community-based data collection restricted to 5 ZIP codes in Washington Heights and Inwood communities in NYC Economic: Mainly Medicaid/Medicare recipients Social: Low levels of education and health literacy Policy: RESTful Web service architecture supports interoperability	User: Researcher Target population: Latino research participants in WICER project Data: Participantgenerated via face-to-face survey (sociodemographics, PROMIS measures, health behaviors), blood pressure, BMI	Tailored infographics Content: Visual display of individual or aggregate with comparison to criterion, community, or national standards on blood pressure, physical activity, PROMIS physical and mental health, etc. Structure: Standalone module using the RESTful Web service architecture Flow of information: Participant data to EnTICE3, output as tailored infographic infographics Technology platform: Web Proprietary or not: Non-proprietary (open source in GitHub)	Software: EnTICE3 Feasibility: Demonstrated capacity to create tailored visualizations at scale Visualization output component of EnTICE3 Usability: Phase 1: Participatory design - designs perceived to be understandable, acceptable, and actionable Phase 2: Tailored designs created with EnTICE3 rated as useful and important by >100 participants in a town hall setting who received their individual data Phase 3: Comprehensive laboratory study (in progress)

ADA: American Diabetes Association; EHR: electronic health record; mI SMART: Mobile Improvement of Self-Management Ability through Rural Technology; NEC: necrotizing enterocolitis; FHIR: Fast Health Interoperability Resources app; NICU: neonatal intensive care unit; CDS: clinical decision support; MACRA: Medicare Access and CHIP Reauthorization Act; HITECH: Health Information Technology for Economic and Clinical Health Act; API: application program interface; EnTICE³: Electronic Tailored Infographics for Engagement, Education, and Empowerment system; WICER: Washington Heights/Inwood Informatics Infrastructure for Comparative Effectiveness Research; RESTful: representational state transfer; PROMIS: Patient-Reported Outcomes Measurement Information System; BMI: body mass index

Table 2.

Development Decisions

Project	Decision	Rationale
		Scientific Rationale	Intellectual Property Considerations	Technical Capacity	Policy Considerations
Connect Diabetes	Build a connected interface linking mobile self-monitoring data into Chronicle Diabetes, a nationally available electronic diabetes education system used by diabetes education programs and focus on usability of the connected interface, without real integration into EHRs	Diabetes educators demand integration of mobile self-monitoring data into Chronicle Diabetes system to facilitate goal setting and follow-up for diabetes selfmanagement education and support. Patients and diabetes educators will benefit from this with less time spent on doing dietary recalls, and more time on individualized education and support. Focusing on usability of the connected interface is critically important before implementing connected systems nationwide.	Chronicle Diabetes system is owned by a company and the University of Pittsburgh. The ADA has free license to use it to benefit the diabetes educator community. A number of commercially available mobile tools can engage patients in diet and physical activity self-monitoring, with open APIs to link to other systems. Connecting Chronicle Diabetes with a single mobile app or fitness tracker company does not allow patients to choose their preferred brands. The decision was made to use a 3rd party intermediary platform to enable patient choices of different mobile and wearable tools, yet standardized its connection with EHRs.	Chronicle Diabetes is a nationally used system specifically designed for and by diabetes educators on documenting diabetes selfmanagement education, a perfect location to receive patient self-monitoring data. The 3rd party intermediary platform is a HIPPA-compliant system experienced in connecting different mobile and wearable tools, and standardizing language to connect with EHRs.	Local/System: No local hospital system was willing to connect mobile data into their EHRs. National: The ADA is giving the Chronicle Diabetes system to all of its recognized diabetes education programs to use for free as an EHR system for tracking diabetes education practice.
ml SMART	Use technology to overcome the known health disparity of distance to clinic for low-income individuals in a rural state living with multiple chronic conditions.	No existing technology tool or intervention was able to treat/monitor multiple chronic conditions for individuals with social determinants of health disparity such as low socioeconomic status, education, poor physical environment, employm ent challenges, lack of social support networks, and access to health care low literacy.	Components of the mI SMART intervention are published and freely usable by interested individuals. However, the intellectual property related implementation of the mI SMART intervention into workflow of individual systems and customization prior to implementation remains proprietary. This decision was made by the team to insure burden of using the technology intervention for patients and providers is minimized. Customization takes time and monetary resources.	The mI SMART intervention is a web-based application that sits outside of individual EHRs. This allows allowing for future expansion without the need for a specific app operating systems. However, this requires patients and providers to log into a separate system outside the EHR.	Local/System: The local free clinic uses a free EHR. Integration into this EHR would limit the scalability of the intervention to clinics using this specific EHR. While, Application Programing Interfaces are available to developers, the local large medical center was not willing to allow the integration of the intervention into the nationally used EHR system.
NEC-Zero	Build a SMART on FHIR app using standards and open source licensing for CDS to help clinicians make evidence-based decisions about preventing neonatal complications.	To test effectiveness in the future, the CDS must be efficiently scaled to apply to 20+ NICUs at one time. Recreating it in each system would prohibit the feasibility of conducting a pragmatic trial in the future.	Needed to have the freedom to test it across NICUs without recreating the CDS. Did not want one vendor to “own” or “limit” our ability to share it or scale it.	Every neonatal intensive care unit clinician could benefit from the knowledge base for NEC-Zero and it was an ethical decision to make it open source.	Local: Hospital needs to allow and use the FHIR server. National: The 21st Century Cures act made it mandatory for EHR vendors to allow open APIs that enable linking into the EHR via SMART on FHIR (Mandl & Kohane, 2017).
EnTICE3	Build a visualization module suitable for interfacing with applications	Returning research results through tailored information visualizations has the potential to improve understanding of healthcare data, promote engagement, and motivate selfmanagement activities.	Open-source approach	No existing tool met project requirements for a module that could take input data from a variety of sources (e.g., survey) and output tailored information visualizations to a variety of applications (e.g., paper, community portal, EHR)	Local: Institutional policy prevented release of infographic data to research cohort via community portal due to a recent HIPAA violation fine.

EHR: electronic health record; HIPAA: Health Insurance Portability and Accountability Act of 1996; API: application program interface; ADA: American Diabetes Association; mI SMART: Mobile Improvement of Self-Management Ability through Rural Technology; NEC: necrotizing enterocolitis; FHIR: Fast Health Interoperability Resources app; CDS: clinical decision support; NICU: neonatal intensive care unit; EnTICE³: Electronic Tailored Infographics for Engagement, Education, and Empowerment system

Table 3.

Current and Future Informatics Innovation Decisions Targeting Individual, Group, or Population Levels

Project	Individual	Group	Population
Connect Diabetes	Connected system allowed diabetes educators to look at individual patient self-monitoring of diet and physical activity data in a calendar format, daily, weekly average, and monthly options	Current: Diabetes educators can select a subgroup of individuals based on their A1c, for example, and send a letter or email to those with A1c>7. Future: Diabetes educators will be able to send an email to those who do not monitor their diet for >7 days or before the visit.	Current: System evaluates and selects a specific parameter (e.g., percentage of people with A1c >7) Future: % under dietary target, or % under certain self-monitored blood glucose levels
ml SMART	A web-based system to allow the bi-directional flow of information between individuals and the healthcare system through the following modalities: 1) Web app personalized reminders for self-management (medications, finger sticks, etc.), 2) mHealth sensors send self-monitoring results to the health system 3) Web app provides automatic personalized self-management feedback 4) Web app provides critical values to provider in real time 5) secure messaging or phone call feedback from providers for critical values 6) Web app/EHR communication to provide active medication list, diagnosis list, lab results 7) video conferencing for healthcare visits and personalized education 8) Web app provides recorded educational videos.	Providers can look at results on self-monitoring data from individuals or subgroups of patients.	Future: Use predictive analysis to identify trends in data to improve overall population health in West Virginia, Medicare and Medicaid and employees of health system
NEC-Zero	Direct CDS at the individual level (clinician taking care of one baby) that includes progress toward NEC-Zero adherence, alerts for non-adherence and NEC early warning score integration. Displays clinical variables relevant to NEC in a dashboard.	Planned: Group-level adherence to the NEC prevention practices and quality outcome visualizations. Shows the overall NICU performance to meet quality drivers and how they align with meeting process targets.	Current: Able to assimilate data for a group of NICUs to provide population level quality measurement for participating NICUs. Future: Apply machine learning to learn from population-level data to enhance early recognition dashboard.
EnTICE3	Creates tailored visualizations based on individual measures	Individual measures are compared to aggregate measures (e.g., PROMIS anxiety) or criteria (e.g., CDC fruit and vegetable recommendations) at WICER cohort/community/ZIP code level depending upon specific measure.	Individual measures are aggregated at WICER population level.

Results

Analysis of Four Case Studies Using the Modified IROM

An overview of the four cases is presented in Supplemental File 1. Table 1 summarizes our analysis of the four case studies using the 4 IROM constructs (client, context, outcomes, and nursing informatics intervention).

Context

In the original IROM, context is defined as a complex, multifaceted concept that exists within the cultural, economic, social, and physical environment of the intervention. As informatics interventions are heavily influenced by current key policies within the context, we expanded the original definition of context to include policy at both the local and/or national level. This decision was made in light of the enormous impact local policies have on the ability to implement interventions at the system level. Such policies include the willingness of local systems or clinical partners to implement nursing informatics innovations that are beyond their technical capacity.

Cultural context.

The cultural context includes the traditional cultural factors in the study settings and specific populations that shaped the four informatics projects. The cultural context in the 4 studies included the burden related to managing multiple chronic conditions, difficulty tracking behavioral goals with paper diaries for diabetes educators, shared decision making in the neonatal intensive care unit (NICU), and reporting research results to Latinos with varied levels of health literacy and limited English proficiency.

Economic context.

The economic context includes the financial affordability of target users (i.e., patients, providers, or researchers). Funding resources for developing the informatics tools played an important role in the economic context as well. Connect Diabetes and ml SMART both targeted patients (low-income and others) who owned mobile devices and providers who were part of a limited pool that had access to the system. NEC-Zero considered economic factors that could reduce financial barriers to accessing the informatics system. EnTICE³ was used by researchers to create tailored infographics with the initial target population of the visualizations being primarily Medicare/Medicaid recipients; there were no technology requirements for the recipients. To date, EnTICE³ use has been funded by grants. Both NEC-Zero and EnTICE³ were primarily funded by federal resources. All of the authors considered economic context with the goal of scaling the intervention in the future.

Social context.

Connect Diabetes and ml SMART aimed to increase patient-provider communication to facilitate patient-centered care. The social context of both studies facilitated the connection of patient data from mobile devices to providers. NEC-Zero aimed for distributed decision making in the NICU, which supports the development of its decision support tool. EnTICE³ aimed to improve the communication of research results from the researcher to the research participant.

Physical context.

The physical context of the settings included NICUs, specialized diabetes education clinics, and the community. One common physical context was that the application of informatics tools all aimed to support an underserved or vulnerable population.

Policy context.

Policy context both facilitated and limited the scope of the projects. Both Connect Diabetes and ml SMART encountered barriers posed by institutional policies. Neither local system was willing to connect data from mobile devices directly to the EHR. Both healthcare systems cited legal, security, or cost considerations around the time of implementation (2013–2015). NEC-Zero was able to capitalize its momentum on the 21st Century Cures and HITECH Acts, prioritizing interoperable CDS and involving CDS experts and clinician groups in designing its tool.

Client

The ANA position statement on nursing informatics explicitly states that target user groups of the innovation should support consumers, patients, and other providers in their decision‐making in all roles and settings. Our 4 case studies reflected this wide range of targeted users indicated in the ANA definition. The target population of the 4 case studies included patients or community residents from various settings; however, the targeted users of these proposed systems were clinicians in 2 of the studies, with EnTICE³ focusing on researchers and mlSMART targeting both patients and clinicians.

Informatics Intervention

The original IROM outlines content, structure, flow of information, and technology characteristics. We further outlined the content component to specifically include informatics intervention content for each type of targeted user if there were multiple targeted user groups and to characterize the technology as proprietary or non-proprietary.

Content.

Connect Diabetes and mlSMART both focused on connecting patient self-monitoring values, diet, and physical activity to clinicians. While Connect Diabetes focused on diabetes patients, ml SMART focused on individuals with multiple chronic conditions, including diabetes, obesity, hypertension, and depression, who resided in rural areas. Both mlSMART and NEC-Zero gave clinicians easy access to clinical practice guidelines. NEC-Zero’s tool focused on CDS in text and dashboard guidance. Patients in the ml SMART trial also receive educational videos prescribed by their clinicians. EnTICE³’ input content was participant-generated research data (e.g., BP, BMI, physical activity, fruit and vegetable consumption, Patient-Reported Outcomes Measurement Information System [PROMIS] depression score), which were processed and output as tailored infographics.

Structure.

Both the Connect Diabetes and ml SMART projects used a proprietary web-based platform. NEC-Zero’s structure is in line with EHR with components but designed to integrate from outside of the EHR, and EnTICE³’s structure is a standalone module using the Representational State Transfer (RESTful) web service architecture.

Flow of information.

The diabetes educators in the Connect Diabetes study are receiving information from patients, however, patients are not receiving feedback from the connected system. The tool was developed to support face-to-face or telephone communication between patients and diabetes educators. The ml SMART platform enabled two-way communication through its web-based system. Participant data are input to EnTICE³, which processes the data to produce output as tailored infographics.

Technology characteristics.

Three of the four studies used web-based platforms, while NEC-Zero, with a focus on implementation and dissemination, used a service-oriented architecture that overlays the EHR, and separates the data source (EHR) from the CDS engine and dashboard.

Proprietary or non-proprietary:

While EnTICE³ and NEC-Zero were designed as open-source platforms, the informatics tools developed in the other 2 studies are mostly proprietary. NEC-Zero code is available at Bitbucket.

Outcomes and Evaluation Plan

All 4 case studies had a component of feasibility/usability/acceptability evaluation to ensure that the developed informatics innovations were perceived to be usable, acceptable, and useful by the respective users and clients. Evaluations for usability, feasibility, acceptability, efficacy and effectiveness are described in Table 1.

Decision Points/Rationale for Different Decisions

While all four case studies applied scientific rationale to develop informatics innovations to address a significant gap in science, different decisions were made owing to considerations related to intellectual property, technical capacity, and policy constraints (Table 2).

In summary, when there is no existing tool or system framework that meets the desired design system, new informatics tools are designed from scratch (mlSMART and EnTICE³). When there was an existing tool to meet certain needs, but there was a lack of function in the existing tool to support the desired specification, the decision was to expand the existing tool (Connect Diabetes) or develop an engine that will run with the existing tool (NEC-Zero). The ml SMART team developed a web-based system outside the EHR to have the flexibility of two-way communication between patients and providers. Connect Diabetes used a third-party intermediary platform to enable the patient to choose from various mobile and wearable tools, yet standardized its connection with EHR systems. NEC-Zero focused on implementation and dissemination research, so it is one step ahead in that the SMART on the FHIR app was developed to enable easy integration with any EHR. The other three studies planned to integrate with EHRs in their next steps. Connect Diabetes and EnTICE³ focused on usability testing of the developed systems in their studies as the initial focus, while noting the potential to output to the EHR. EnTICE³ was designed to integrate with a variety of applications other than EHRs, including community portals.

In addition to the national and local policy context that impacted decisions, funding support for the development of some of these informatics tools was primarily from the federal government, and this impacted the final decision to make the tools open source. ml SMART was developed as a proprietary platform because part of the implementation of the mI SMART system involves the assessment of the workflow of individual systems and customization prior to implementation. NEC-Zero’s in-line structure can operate within the hospital’s firewall, but the investigators tried to make the component of the CDS engine available in an interoperable way via SMART on FHIR. EnTICE³ is open-source, and Connect Diabetes was limited to a pool of national users with its agreement between the original developer and the American Diabetes Association.

Comparison of Case Studies Working at Individual, Group, and Population Levels

In the original IROM as depicted in Figure 1, three 2-directional arrows connecting the four key constructs indicated that nursing informatics interventions can work at individual, group, and population levels. Data can be collected at the individual level and aggregated later at the group or population level to measure outcomes or be part of the informatics intervention to provide CDS. Table 3 highlights the current informatics innovations in the four case studies targeting each of the three levels as well as future and planned informatics innovations that can be built upon existing efforts. All four case studies targeted individual level data in the initial trial. An expansion of this discussion is available in supplementary file 1.

Discussion

Our comparative analysis provided insights into the application of a comprehensive framework to guide nursing informatics research and system development by mapping 4 health informatics case studies to the modified IROM, which was expanded by adding policy context. The significance of the IROM lies in its incorporation of the SROM and the SDLC; the former takes into account the care, client, informatics intervention, and outcomes, and the iterative, unidirectional SDLC allows for the planning, analysis, design, implementation, and maintenance of the health informatics innovation (Effken, 2003). The successful application of the IROM and cross-case qualitative comparison analysis in this study resulted in the identification of tangible benefits, such as an examination of dynamic relationships between processes and constructs in nursing informatics interventions and the standardization of the creation, implementation, and dissemination of future HIT based on IROM concepts. Our analysis also highlighted scientific rationales and challenges encountered among the four case studies based on the SDLC. These findings facilitated development decisions and examined the trajectory of such decisions according to individual, group, and population level needs. The broad applicability of the model enables the consideration of such factors as nursing informatics researchers institute their own projects in tandem with IROM principles.

Nurses, researchers, and informaticists must prioritize the cultural, economic, social, physical, and policy contexts of nursing informatics technologies for the successful implementation of such projects. Aligning with studies by Effken (2003) and (Brennan & Bakken, 2015), our cross-case qualitative analysis confirmed the contextual basis of health informatics innovations as an important relationship and a critical component of the IROM that must be taken into consideration during planning, design, and implementation stages. A modified IROM was suggested with the inclusion of policy as an additional contextual basis, given that informatics interventions are influenced, and often limited, by institutional, local, and federal policies on interoperability and meaningful use. Taking context into account while evaluating intervention outcomes transforms clinical research into systems-oriented research (Brennan & Bakken, 2015). Our study found that the inclusion of evaluation in the IROM assures continuous improvement and enables the direction of informatics innovations toward definite outcomes, such as desired efficacy and effectiveness. Furthermore, sustained engagement with target users is maintained by focusing on usability, feasibility, and acceptability as fundamental IROM components. Our findings also support the active identification of the target users, content, structure, flow of information, and technology characteristics of the nursing informatics intervention. Brewer et al. (2008) noted that the discovery and interrelationships between all these constructs may not apply in individual research studies; however, testing such links signifies mature research processes with supporting data from multiple studies. Accordingly, we found that addressing most of these factors allows for uniformity, emphasizes structure, and enables the discovery of new features and limitations that need to be established among nursing informatics research and interventions.

We also analyzed the application of the 4 informatics interventions on individual, group, and population level as it relates to the design of informatics interventions in these four projects as well as future capacity that can be built upon existing infrastructure to expand into the population level. Three of the studies focused on current efforts at an individual level with plans to expand into population health, while one study started with the population level in mind and provided informatics solutions to address all levels. As nurses play an increasingly important role (Mason, 2016) in primary care and population health, nursing informatics tools should consider group and population level application from the beginning.

Decisions about the future development and improvement of the interventions were made based on scientific merit, intellectual property, technical capacity considerations, and policy limitations. Scientific rationales addressed current limitations in health education, care coordination, and delivery that the interventions proposed to address. A distinguishing perspective in our cross-case analysis was building technical capacity by locating inefficiencies and expanding the functionality of existing tools and system frameworks alongside the development of new systems based on care needs. Secondary application integration, particularly with EHRs, was of substantial concern for care coordination. While being aware of its potential for integration with EHR our comparative analysis also noted that focusing on EHR integration tended to stall innovative intervention development, especially in the initial planning and design stages. Our research accordingly promotes the need for nursing inquiry to be informed by multiple data sources that go beyond the confines of the EHR (Brennan & Bakken, 2015; Westra et al., 2017). Policy constraints were centered on enhancing accessibility through open source systems and the scalability of interventions. The structuring of these features helped guide decisions about current individual, group, and population level outcomes and will inform future innovation development. The National Consortium for Data Science (Ahalt, 2014) advocates a data-driven economy that uses digital data to accelerate discovery and improve critical decision-making processes. We thus conceptualized data-driven decisions that focused on augmenting the flow of information between patients and providers, evidence-based CDS and adherence, efficient data visualization based on provider needs, and statistical means for trend analysis and prediction. In particular, enabling the data flow between patients and providers to address the needs of incorporating patient-generated health data into clinical workflow and giving patients access to longitudinal health records requires advancing research agendas on common data fields and standards. Particular attention should be paid to enhancing usability (Wang J, 2015) and reducing inefficiencies or interruptions to clinical work flow and clinician burnout (Shanafelt et al., 2016) related to the use of connected systems to bring more patient-generated health data into clinical care.

Our findings also support the notion that thoughtful considerations in intellectual property strategy in combination with academic industry collaboration can move an informatics program forward. Developing innovation programs and solutions in academic settings and seeking industry collaboration to commercialize or make the created intellectual property open source are key. The use of Small Business Innovation Research Small Business Technology Transfer grants (US Small Business Administration, 2018) as a funding mechanism may offer opportunities for such a collaboration and will play a key role in moving projects forward. In addition, these collaborations with small businesses could offer a unique perspective relevant to consumers.

While our study offers perspectives that may be useful to nurse informaticians and others developing and designing technology solutions, our approach had a few limitations. The 4 case studies were not selected to represent the whole spectrum of nursing informatics studies. The authors coalesced around this topic because of our common interests in nursing informatics innovations with a focus on different aspects of interoperability and our insights when working together in a leadership training program. Thus, these 4 case studies are not meant to be representative. The IROM was applied to the studies after they were conducted and that analysis is portrayed here as an organizing framework to explore similarities and differences across the studies.

Conclusion

By comparing four informatics innovations that were conducted by different nurse scientists and addressed different clinical challenges, this paper demonstrates the utility of applying the IROM across nursing informatics and describes key development decisions that were driven by policy contexts. As informatics grows and the specialty of nursing informatics expands its reach, nursing science will grapple with more meaningful ways to harness, learn from, study, and apply big data to clinical questions. This paper offers a framework for designing and evaluating such innovations. In the future, nursing informatics will require less focus on developing for specific brands and devices (e.g., EHR vendors, stand-alone systems) and will emphasize the need to enable patient engagement and patient autonomy in connected health and care. This will need to be done in ways that maximize interoperability, reduce interruptions in clinical workflow, handle the complexity of integrating multiple devices, and address investments in time and iterative development to ensure usability and leverage diverse data streams.

Highlights.

1. Within a policy context of national initiatives focused on data science, precision health, and broad technology integration, nursing informatics innovations need to be adaptive.

2. The design, adaptation, and implementation of informatics innovations can be informed through the lens of the Modified Informatics Research Organizing Model with policy context added.

3. Informatics development decisions were influenced by a combination of scientific, intellectual property, technical, and policy factors, as well as commercialization potential and stakeholder needs.

4. Issues around interoperability, intellectual property, decisions to build a new system or adapt an existing system, and the potential impact of decisions on patient outcomes can impact design choices for future informatics projects.

5. Designing to scale to the group and population level in the future facilitates broader technology integration and long-term dissemination and implementation.