Affirming Proposed Variable Relationship Patterns in a Conceptual Nursing Model by Converting Qualitative Data to Causal Loop Diagrams

Abstract

Even with decades of use, there is minimal understanding about the impact that the use of Health Information Technology has on nursing work and workarounds. Reliance on quantitative methods has to some degree constrained our understanding by viewing phenomena from only one perspective. This multimethods research used qualitative data to develop causal loop diagrams and inform a Health Information Technology Workaround model. This approach can play an important role in generating an improved understanding of nursing clinical workflow and workarounds. This research strategy has not been identified in nursing literature to date, but perhaps will encourage future exploration and paradigm crossing. Investigating the use of causal loop diagrams and systems modelling in nursing can create an opportunity to enrich our insights and encourage scientific dialogue about the complexity of clinical workflow and the integration of Health Information Technology.

Introduction

The purpose of this paper is to describe a multimethod approach to model development and workflow diagramming. In informatics research there is a tendency to use a quantitative approach to investigations under the assumption that this is the way to reach significant conclusions. A flaw in this perspective is that it relies on linear, structured relationships to build knowledge about clinical care, a phenomenon that does not operate according to straight path algorithms. The study of complex adaptive systems (CAS) however is framed by dynamic interactions and complexity. It often addresses multiple levels and units of analysis; using multimethods to tell a story about how and why phenomenon evolved. Unfortunately, as a newer science, there are not many examples of CAS analysis in nursing.¹

This paper describes the transformation of qualitative data into causal loop diagrams (CLD”s) in order to map the proposed variable relationships in a Health Information Technology Workaround (HITW) Model (See Figure 1) depicting health information technology (HIT) workarounds. This work was part of a larger dissertation study exploring HIT workarounds being used by nurses in intensive care. This study generated a multi-level conceptual framework through the specification of relevant variables and relationships essential for understanding what is actually happening with the use of HIT at the bedside. One aspect of analysis was the creation of causal loop diagrams (CLDs) in order to integrate feedback loops with the proposed HITW Model. This paper describes the development of these diagrams. The CLD’s are models that portray the behavior of variables in a system, presented as causal relationships and feedback loops.² The premise of this work is that the feedback loops of a system should be understood in order to understand the system.³

Figure 1.

Health information technology workaround (HITW) model

The proposed HITW model is presented in Figure 1. This is a three level model with the micro level representing the patient, mezzo representing the nurse and macro the organization. This model is based on Arthur Stinchcombe’s constructing social theories work for functional theoretical explanations.⁴ The dependent variable in this work is the homeostatic variable (H). The variables (causes) that disrupt homeostasis are tensions (T) and the variable that tries to compensate for this and return the system to homeostasis is structure (S). Stinchcombe’s work allows the review of functional theory as an easily understood pattern of relationships between variables.⁴

Systems Thinking

The complexities of healthcare can be overwhelming, yet in order to simplify workflow models we may be missing the real-world experiences of clinicians and the true complexity of care. For example, cosigning high risk medication should result in better patient safety, however as one nurse described, there may be unintended consequences such as altered or missing documentation: “I will not list all my IV medication titration changes that I made just to keep from having to beg someone to put in their password several times for each change”.

Quantitative approaches tend to break processes down, focusing on small numbers of linear relationships. Conversely, systems thinking looks at interactions and relationships between variables and expands our view of the ever-changing landscape of the phenomena.⁵ Systems are active, dynamic processes with multiple, variable interactions that will vary with time and environment. For instance, the same variable can act as an independent and dependent variable simultaneously in a systems thinking approach. Negative and positive feedback loops are a fundamental point of focus and can significantly influence systems behavior.^{6, 7, 8}

There is a formal process for systems thinking, systems dynamics, that blends modeling and simulation research with qualitative methods.² This multimethod research is generally broken down into five parts. The first step is to identify variables from the qualitative data and secondly to develop a causal loop diagram (CLD) using those identified variables. From the CLD, stock and flow diagrams are constructed. The fourth stage is the development of the mathematical equations formulating the problem as designed in the stock and flow diagram. Finally, model simulation is performed using the predefined formulas.^{9, 10} The scope of this work led only as far as to consider the presence and general implications of feedback loops with interactions between safe patient care, turbulence, workload, barriers, workarounds and HIT protocols. In order to explore relationships between variables, the methodology for the development of causal loop diagrams (steps one and two) from the coding of qualitative data was followed ^{2, 10, 11}

Methods

This study was conducted in collaboration with The American Association of Critical Care Nurses and approved by The University of Texas Health Science Center IRB. A sample of 307 Registered Nurses voluntary responded to an email survey consisting of two qualitative open ended questions followed by quantitative items measuring nurse characteristics, elements of nursing work, HIT problems, and patient safety. Multiple sources of data were used to compare, refine, and elaborate findings from both methods. The qualitative/ modeling techniques have been described in systems dynamics research over the last decade, yet they are still poorly specified.¹¹ In this case the procedures described by Kapmeier¹⁰ and Yearworth and White² were followed. As a strategy to help ensure rigor, the research outline offered by Kapmeier was followed.¹⁰ To enhance the validity of the multimethod research, legitimation process and checks guided each stage of the research.¹²

The initial HITW model was designed using literature and personal experience. An internet survey was selected to collect data from the critical care nurses. The survey asked the nurse to describe, in their own words, the problem they experienced with HIT and the workaround used. Quantitative variables alone (from the literature) would have limited the data collected, and variables we knew to be important (such as workarounds) had no associated measures. It was clear that qualitative inquiry would be essential in modeling this complex system. The ability to follow up with participants was considered and rejected because assuring complete confidentiality was imperative to gaining truthful responses.

We were cautious in determining the a priori sample size because of concern over the amount of narrative transcripts we would receive. Nunally recommends sampling at least 10 times as many subjects as variables, and so the sample size was determined from quantitative methods and set at 290.¹³ Unlike true grounded theory, the qualitative analysis was conducted after the literature work, but before the quantitative analysis. Using a qualitative data analysis software, new behaviors or problems were identified and quantified. Themes and concepts were explored across texts to identify relationships. After reducing the volume of data during the coding process, categories were generated that could be linked to the concepts presented in the literature. ^{2, 10, 11}

The work was divided into six exercises:

1. Qualitative data was coded from the survey transcripts. Specific qualitative variables (i.e., types of workarounds and frequency measures) were quantified and imported to IBM Corp. SPSS.¹⁴

2. Reliability procedures were performed with each scale and exploratory factor analysis performed on the scales for turbulence and HIT problems. Inter-rater reliability was performed by different groups of experts to determine the internal validity of qualitative items, and to confirm agreement between quantitative items and qualitative descriptions. To confirm the agreement of preliminary workaround definitions, inter-rater reliability analysis was conducted by a panel of critical care nurses.

3. Quantitative correlation analysis of variable relationships and identification of relational patterns.¹⁵

4. Qualitative code relations browsers were utilized to produce variable relationship matrices using MAXQDA software.¹⁶ The strength and direction of the relationships were informed by the narratives and quantitative analysis.

5. Development of multilevel causal loop diagrams representing the micro/ mezzo level and the mezzo/ macro level.¹⁷

Results

The respondents were 87% female and 13% male. Fifty-eight percent of the nurses were 45 years old or greater. Almost 50% of the nurses had a bachelor’s degree in nursing, 20.6% an associate degree and 19.9% had a master’s degree. Nurse experience was midway between a proficient and expert. Intensive care specialties included adult, pediatric and neonatal. Patient acuity was reported as: 61.8% critical, 28.7% guarded and 9.2% stable. Workload of the nurse was reported as heavy (40%) and moderate (58%). There was a wide range of software represented including: KBMA (Allscripts), Carefusion, Cerner, Epic, Meditech, McKesson, Soarian, eICU, EndoTool and GlucoStabilizer.

Key variable definitions were confirmed using quantitative approaches; factor structures were identified and the reliability of new scales assessed. The key variables and associated survey items were: workload, turbulence, patient safety hazard, and HIT barriers/ problems. In previous work on the HITW model, the variable workload did not fully describe the amount and type of work nurses were performing. The variable turbulence was created to measure additional, unanticipated work that nurses perform.¹⁸ Examples of turbulence attributes include distractions, interruptions, missing equipment and loss of information. The alpha coefficient for the 15 items was. 751, suggesting acceptable internal consistency. Turbulence is connoted as the delta symbol in the HITW model. Four workaround variables developed during a previous pilot were confirmed and their specific attributes identified. These workarounds variables are: problem solving and intuitive workaround, and informal and formal communication.¹⁹ All proposed variable definitions and attributes were supported.

Once qualitative coding was completed, binary code matrices were produced (Table 1). These matrices allowed identification of relationship patterns and represented a starting point for investigation of mutual causality and feedback. This approach helped identify co-occurring concepts within and across groups. Table 1 identifies, for example, no relationships between patient safety risks, workarounds, and the lack of a unit secretary. There is also no relationship between the workload attributes of admissions, transfers and discharges and patient safety risk. The matrix did however identify relationships between turbulence (technology response time), patient safety risk and workarounds. Some interesting relationships were suggested. For instance, technology response time and ethical/ moral conflict were the only two items associated with workarounds creating a patient safety hazard. On the other hand, the creation of a patient safety hazard caused by the primary problem was associated with 9 turbulence items such as loss of information, equipment issues and information overload.

Table 1.

Binary code matrix; turbulence and workaround types

WA= Workaround Commu= Communication	Safety event 2nd Problem	Safety event 2nd WA	WA Improves safety	WA Safety risk	Problem safety risk	WA Formal Commu	WA Informal Commu	WA Problem Solving
Ethical/ Moral Drift	0	0	1	1	1	1	1	1
Workload ADT	0	0	0	0	0	0	0	0
Turbulence\Technology Response Time	0	0	0	1	1	0	1	1
Turbulence\lnadequate Training	0	0	1	0	1	1	0	1
Turbulcnce\Communication Technology	1	0	1	0	1	1	1	1
Turbulence\Adminljtrative or Regulatory Demands	0	0	0	0	0	1	1	0
Turbulence\lnterpersonal Distractions	0	0	0	0	0	0	0	0
Turbulence\Noise	0	0	0	0	0	0	0	0
Turbulence\Loss of Information: Handoff	0	0	0	0	1	0	0	1
Turbulence\Student/ Preceptee	0	0	0	0	0	0	0	0
Turbulence\Staff off Unit	0	0	0	0	0	0	0	0
Turbulcnce\No Secretary	0	0	0	0	0	0	0	0
Turbulence\Equipment & Supply Issues	0	0	0	0	1	1	1	1
Turbulence\lnfo Overload	0	0	0	0	1	0	0	1
Turbulence\lnterruptlon	0	0	1	0	1	1	1	1
Turbulence\Distraction	0	0	1	0	1	1	0	1
Turbulence\Communication breakdowns	0	0	1	0	1	1	0	1

Turning to the quantitative items, correlations examined between workaround variables and demographic data (i.e., nurse characteristics) demonstrated no significant relationships. Quantitative variable analysis revealed that direct relationships exist between nursing workload, turbulence, the HIT barrier (problem) and patient safety hazards. For example, there were significant positive relationships between the HIT barrier (problem) and workload (r =. 32, N= 293, p=. 000) and HIT barrier and turbulence (r =. 33, N= 293, p=. 000). Turbulence was positively correlated with safety hazards (r =. 41, N= 293, p=. 000) and workload (r =. 48, N= 293, p=. 000). The weakest association was between workload and safety hazards (r =. 16, N= 29, p=. 005). There was overall agreement between the quantitative correlation findings and the qualitative binary matrix.

The correlational analysis supported a number of positive loops. An example of four positive reinforcing loops suggested by the data is presented in Figure 2 Positive reinforcing feedback loops are labeled as R in a CLD.⁷ As the number of HIT problems increase, so does workload. As turbulence increases, workload increases and as HIT problems increase, turbulence increases. Finally, as turbulence increases, risks of a patient safety hazard increases.

Figure 2.

Positive (reinforcing) and negative (balancing) feedback loops

The analysis also supported a number of negative loops. Examples of four negative balancing loops are presented in Figure 2. These negative balancing feedback loops are labeled as B in a CLD.⁷ In these examples, as intuitive and informal communication workarounds increase, stress decreases. As workarounds increase, inefficiency decreases and as informal communication workarounds increase, complexity decreases. The correlation results provided direct and inverse relationship data to derive proposed feedback loops, however for quantitative data to be useful and accurate a large amount of data is required.²⁰ The qualitative data resulted in more representative and detailed matrices.

To begin modeling of the CLDs, qualitative matrices were produced for the micro and mezzo variables. Since the relationships can be in either direction, the nature of the relationships (direction) is determined by the researcher, but assessment of the strength of the category relationships can be determined by the number of times two categories are linked. In many cases, the quantitative correlational data supported the findings and suggested the direction of the relationship (direct or inverse). The more frequently the category is linked in the matrix result, then the more evidence there is that the two categories are related.^{2, p.154} The qualitative matrix example in Table 2 suggests, for example, that as the stress associated with HIT problems increase so does the patient safety risk (n=17). The strongest relationship in this table showed that as efficiency improved with the use of workarounds patient safety improved (n=38).

Table 2.

Qualitative Matrix Example

WA= Workaround	Turbulence Commu.	WA decreases Stress	Problem Increases Stress	Problem reduce efficiency	WA reduce efficiency	WA Improve efficiency	WA Improve Safety	Problem Safety Risk	Added Tasks
Turbulence Communication	1	16	1	2	1	21	12	4	3
WA decreases stress	16	0	13	1	0	4	1	3	2
Problem increases stress	1	13	0	10	1	3	13	17	5
Problem reduce efficiency	2	1	10	1	0	5	9	12	2
WA reduces efficiency	1	0	1	0	0	0	0	2	3
WA improves efficiency	21	4	3	5	0	0	38	14	12
WA improve safety	12	1	13	9	0	38	0	15	18
Problem safety risk	4	3	17	12	2	14	15	0	7
Added Tasks/ Extra steps	3	2	5	2	3	12	18	7	1

A CLD of the micro and mezzo interfaces was developed based on the preliminary data produced from the qualitative matrix report and supported by the quantitative analysis (Figure 3). This CLD provides evidence that protocols are interacting with barriers, workload and turbulence creating positive and negative feedback loops that also interact with safe patient care. This is by no means complete, but does give a sense of the complexity of the balancing and reinforcing loops and some of the primary relationships that were identified.

Figure 3.

Causal loop diagram micro and mezzo levels

Some of the primary relationships identified in the micro/ mezzo causal loop diagram included:

1. Balancing loop 1 (B1): As turbulence increases, workarounds increase and problems decrease.

2. Balancing loop 2 (B2): As patient safety risks increases, workarounds increase. When workarounds increase, stress and inefficiency decrease and so safety risks decrease.

3. Balancing loop 3 (B3): As problems increase, workarounds increase. Workarounds act to improve the performance of HIT, decreasing process steps. Decreased process steps result in decreased problems.

4. Reinforcing Loop 1 (R1): As patient safety risks increase turbulence increases. Increased turbulence results in increased workload and that increases the patient safety risks.

5. Reinforcing Loop 2 (R2): As problems increase, turbulence increases. As turbulence increases so does workload. Increased turbulence and workload is associated with increases patient safety risk; increased risks contribute to an increase in problems.

Some of the primary relationships identified in the mezzo/ macro casual loop diagram (Fig. 4) included:

1. Balancing loop 1(B1): As formal communication increases, process steps increase. As added steps increase there is a reduction in adherence to the protocols which forces an increase in formal communication.

2. Balancing loop 2 (B2): As protocol aherence increases so do the work process mismatches. These mismatches cause an increase in delays, increasing workarounds, and a reduction in adherence to protocols.

3. Balancing loop 3 (B3): As adherence to hospital protocols decrease, hospital reimbursement also decreases. When reimbursement decreases so does job security/ salary. As job security decreases, workarounds increase and are used not necessarily to increase accurate protocol use but to increase the perception that protocols are being followed.

4. Balancing loop 4 (B4): As care delivery delay increases, lost time increases. With an increase in lost time, there is an increase in workarounds. As workarounds increase the care delivery delays decrease.

5. Reinforcing loop 1 (R1): As workarounds increase, meeting the intent of the protocol increases. As compliance with protocols increase (or at least the appearance of compliance) then job security increases.

6. Reinforcing loop 2 (R2): As formal communication workarounds increase, added process steps also increase. As the steps increase so do the delays. When delays increase, the workarounds increase and this increases the use of formal communication workarounds.

7. Reinforcing loop 3 (R3): As workload increases, turbulence increases. An increase in turbulence causes an increased safety risk which then causes an increased errors. Increased errors cause an increase in workload.

Figure 4.

Causal loop diagram mezzo and macro levels

Discussion

The use of qualitative modeling can provide an opportunity to consider situations that might not have been considered before. In quantitative inquiry the variables explored are pre-determined while causal loop diagramming allows for consideration of new behaviors during the study. For example, a number of nurses described that their year-end evaluations were directly tied to a compliance audit of scanning percentages and documentation. As this was explored, it was realized that nurses who could not perform the behavior were using workarounds to give the appearance that the protocol had been met.

The qualitative data supported the idea that nurses try to use HIT and the associated protocols to deliver safe patient care. It was also evident that nurses were attempting to comply with the HIT protocols even when the technology, added steps and work process mismatches made it difficult or impossible. When attempting to utilize Structure variables to achieve Homeostasis, nurses often encountered problems and turbulence which disrupted care delivery. In turn, nurses used workarounds as solutions for efficiency, complexity and time problems in an attempt to achieve Homeostatic outcomes.

Increased workload and reduced staffing were confirmed to add additional complexity, time pressure and inefficiencies further threatening safety and efficiency. Workarounds acted to mediate the relationships between turbulence, workload and patient safety. This was substantiated by balancing feedback loops between turbulence and workarounds and the time saved by use of a workaround in the absence of a problem. Current thinking is that patient safety risk increases when workarounds are utilized, but this analysis suggests that the opposite may also be true; workarounds are being used when a nurse recognizes a patient safety threat. It was also determined that workarounds were being used to comply with protocols. For example, it was reported that one protocol required that barcodes on all blood products be scanned prior to administration as part of the safety checks. When the blood bank had to split the blood product between 2 bags, there was no barcode on the second bag. The nurse entered another order for the same blood product in order to generate a barcode, and then canceled the order. Using this workaround, the nurse could follow the policy and scan the barcodes on both bags.

The CLD’s helped identify additional processes that the primary relationships might contribute to. For instance, formal communication workarounds are defined as written or oral communication that occurs through designated channels of the organization to address HIT systems barriers and disseminate protocol variations.¹⁸ Correlation analysis and CLD’s identified positive relationships between formal communication, extra process steps and care delivery delay. The CLD also identified that the administrative sanctioned workarounds (formal communication) indirectly reduced a nurses’ adherence to protocols. (Fig. 4) This had not been apparent from the qualitative analysis alone. Finally, the development of CLD’s from code matrices also permitted us to see quite rapidly where relationships did not exist. For instance, in nursing it is often assumed that inadequate HIT training is somehow associated with increased patient safety risk and workarounds. In fact, it is not uncommon to find mandatory education sessions following safety events. The code matrix (Table 1) did not identify a relationship between these variables.

Figure 5 presents an overlay of the CLD’s with the micro-mezzo HITW model. The feedback loops confirmed and clarified the relationships in the HITW model. The relationship between workload and adherence to HIT protocols was confirmed as bi-directional. Problems with S are filtering directly back to T (i.e., HIT performance, additional steps, staffing) further increasing workload and Tension. Turbulence interaction with model variables was apparent at the anticipated interfaces. The CLD’s supported the HITW model proposal that barriers (problems) and turbulence interfere with a nurses’ ability to utilize HIT to protect the patient from safety risk.

Figure 5.

Causal loop diagrams and HITW model at the micro-mezzo level

One criticism of qualitative system dynamic modeling is the concern that wrong inferences might be attained. Causal loop structures can quickly become overly large and complex, masking the primary model behavior with excessive detail.²⁰ It is recommended that a system archetypal model be maintained summarizing the essence of the model. In this case, the HITW model summarizes the relationships between HIT protocols, patient safety and workload and denotes the primary feedback loops derived from the quantitative and qualitative findings (Figure 5).

Implications

Qualitative inquiry allows identification and incorporation of variables that were perhaps never even considered at the onset of study. For example, documentation audits and the influence of nurses’ job evaluations on the use of workarounds was unexpected and not considered at the start of the study when quantitative variables were being designed. The qualitative analysis allowed the identification of this relationship during coding, inclusion in the matrix evaluations and timely integration of this interaction into the study.

Feedback loops occur when the output of a process becomes the input of another. Resulting behaviors, such as unintended consequences can be very hard to predict. The development of CLD’s assists in developing a view of the intertwining of problems and solutions and helps to anticipate the possible consequences of system or process use and changes. At the very least, interdisciplinary CLD development sparks a planning dialogue.

The CLD’s, although valuable, are only a first step. The next consideration must be the integration of time and time delay into the models. Integrating time into the analysis allows for understanding how much of a delay one variable can cause for the entire system. Matrices can be developed for time, and will assist in determining which variables are appropriate for intervention.²¹ This would be a most valuable asset for nursing and informatics planners.

Arising from the qualitative CLD development that displays links between cause and effect, are the stock and flow diagrams. The diagrams are more detailed and precise, relying on mathematical functions. and is the development of stock and flow diagrams. From stock and flow diagrams evolves the ability to run systems dynamic simulation and many simulation software products are evolving for precisely this type of use.²

Although CLD’s are used as a stepping stone to more advanced analysis, there are some underlying truths and patterns that can be identified and utilized. CLD’s can be used to better predict consequences of change resulting from feedback loops. For example, CLD’s with a majority of negative feedback loops are more likely to display systemic resistance to changes or disturbances. Alternately, a system predominated by positive feedback loops can be highly unstable. Change should be approached cautiously and monitored for the emergence of new feedback loops. Different approaches to change management are recommended in each situation.²² Finally, as big-data and data analytics become more common place, the integration of system dynamics with data mining can more rapidly expand the knowledge discovery in databases.

Conclusions

Critical care nursing and healthcare in general is in a significant transformation. Rapid adoption of cutting edge technologies, new reimbursement systems and changes in the characteristics of nursing and physician jobs can stress an already complex healthcare system. The complexity of healthcare creates barriers to care delivery and, along with rapid change and poor workflow design, can inhibit a nurses’ ability to comply with the desired protocols envisioned by policymakers. When integrating HIT into clinical care and nursing workflow, a lack of theoretical underpinnings and a reliance on linear models has limited the ability to anticipate unintended consequences and consider the possible adaptations that nurses might make when using HIT.

This research offered one approach to integration of qualitative data into workflow analysis in order to explore variable relationships in a HITW model by creating causal loop diagrams. The foundational assumption in this work is that the nursing environments we are studying are dynamic complex adaptive systems that are constantly changing. The approach presented here offers one alternative to the structured linear methodology of nursing research by combining different methods in an attempt to better describe and understand the complexity of HIT use in critical care. The causal loop diagrams developed here, although elementary, do provide insights into nurses HIT workarounds in critical care. The research approach described in this paper is just the first step to more advanced analysis and the development of simulation models. Continuing to view our acute care environments as the complex adaptive systems that they are will require new approaches to research design and methods. This paper was one attempt to offer alternative ideas to the utilization of qualitative inquiry and discovery in nursing informatics research.