Understanding Barriers and Facilitators to the use of Clinical Information Systems for Intensive Care Units and Anesthesia Record Keeping: A Rapid Ethnography

Abstract

Objective

This study evaluated the current use of commercial-off-the-shelf Clinical Information Systems (CIS) for intensive care units (ICU) and Anesthesia Record Keeping (ARK) for operating rooms and post-anesthesia care recovery settings at three Veterans Affairs Medical Centers (VAMCs). Clinicians and administrative staff use these applications at bedside workstations, in operating rooms, at nursing stations, in physician’s rooms, and in other various settings. The intention of a CIS or an ARK system is to facilitate creation of electronic records of data, assessments, and procedures from multiple medical devices. The US Department of Veterans Affairs (VA) Office of the Chief of Nursing Informatics sought to understand usage barriers and facilitators to optimize these systems in the future. Therefore, a human factors study was carried out to observe the CIS and ARK systems in use at three VAMCs in order to identify best practices and suggested improvements to currently implemented CIS and ARK systems.

Methods

We conducted a rapid ethnographic study of clinical end-users interacting with the CIS and ARK systems in the critical care and anesthesia care areas in each of three geographically distributed VAMCs. Two observers recorded interactions and/or interview responses from 88 CIS and ARK end-users. We coded and sorted into logical categories field notes from 69 shadowed participants. The team transcribed and combined data from key informant interviews with 19 additional participants with the observation data. We then integrated findings across observations into meaningful patterns and abstracted the data into themes, which translated directly to barriers to effective adoption and optimization of the CIS and ARK systems.

Results

Effective optimization of the CIS and ARK systems was impeded by: (1) integration issues with other software systems; (2) poor usability; (3) software challenges; (4) hardware challenges; (5) training concerns; (6) unclear roles and lack of coordination among stakeholders; and (7) insufficient technical support. Many of these barriers are multi-faceted and have associated sub-barriers, which are described in detail along with relevant quotes from participants. In addition, regionalized purchases of different CIS and ARK systems, as opposed to enterprise level purchases, contributed to some of the identified barriers. Facilitators to system use included (1) automation and (2) a dedicated facility-level CIS-ARK coordinator.

Conclusions

We identified barriers that explain some of the challenges with the optimization of the CIS and ARK commercial systems across the Veterans Health Administration (VHA). To help address these barriers, and evolve them into facilitators, we categorized report findings as (1) interface and system-level changes that vendors or VA healthcare systems can implement; (2) implementation factors under VA control and not under VA control; and (3) factors that may be used to inform future application purchases. We outline several recommendations for improved adoption of CIS and ARK systems and further recommend that human factors engineering and usability requirements become an integral part of VA health information technology (HIT) application procurement, customization, and implementation in order to help eliminate or mitigate some of the barriers of use identified in this study. Human factors engineering methods can be utilized to apply a user-centered approach to application requirements specification, application evaluation, system integration, and application implementation.

1. INTRODUCTION

Successful implementation and adoption of health information technology (HIT) in healthcare environments is an ongoing challenge. For intensive care units (ICUs) and other acute care environments, acceptance of HIT can be especially difficult to achieve [1], where the complexity of care and presence of critically ill patients often demand rapid decisions and actions [2]. For a healthcare system like the Veterans Health Administration (VHA), use of a commercial-off-the-shelf application can add an additional layer of complexity because it must be integrated with the VA’s existing, internally developed HIT systems. The VHA uses an electronic health record (EHR) known as the Computerized Patient Record System (CPRS). CPRS, through a graphical user interface, integrates multiple Veterans Health Information Systems and Technology Architecture (VistA) software applications designed to allow clinicians to order medications, laboratory tests, consultations, and document actions [3]. Clinicians use Clinical Information Systems (CIS) and Anesthesia Record Keeping (ARK) at bedside workstations, in operating rooms, at nursing stations, in physician’s rooms, and in other various settings. CIS and ARK systems are commercial off-the-shelf (COTS) HIT applications and intended for use in coordination with an EHR, such as CPRS/VistA. A CIS or an ARK system is designed to facilitate creation of electronic records of data, assessments, and procedures from multiple medical devices, thereby eliminating manual entry or use of traditional critical care and anesthesia paper documentation. CIS and ARK systems can provide information to also produce analytics for reports and analysis to potentially improve patient care.

For a facility with as-is CPRS/VistA and paper flowsheets, the CIS and ARK systems are meant to enable the following after implementation: clinical documentation in one electronic application, rather than on a paper chart or in CPRS templates; chart data transferred from CIS to CPRS/VistA, making it available for all clinical, administrative, and/or business staff external to the critical care unit; data from monitors and medical devices recorded automatically by the application, rather than transcribed by nurses onto paper charts; single sign-on to CIS/ARK and CPRS/VistA; and lab results, medication orders, and other care data from CPRS/VistA viewable in CIS or ARK. Each of the 23 regional Veterans Integrated Service Networks (VISNs) purchased separate CIS and ARK systems from a selection of vendors. At the time of this study, 90% of the VISNs had purchased and/or deployed a CIS system and 74% had purchased and/or deployed an ARK system. However, inconsistent adoption of these systems across the VHA suggested the presence of barriers to their effective use. Therefore, we designed a study to identify barriers, facilitators, and suggested improvements to currently implemented CIS and ARK systems at a sample of VA Medical Centers (VAMCs).

We framed our study with sociotechnical systems theory, which has been used and discussed prominently in the medical informatics literature [4–14]. Although there is no one “sociotechnical approach” [4], studies that rely on a sociotechnical perspective all have a high-level recognition that organizational and health systems at large have a substantial influence in shaping HIT and that the technology and context are intertwined [5]. For example, two studies examined the interrelation of the organizational environment and technical subsystems for the implementation of computerized provider order entry (CPOE) systems [6, 13], resulting in unintended consequences [6] and implementation failure [13]. Berg and others note that “neutral” HIT applications in work practices evolve within the specific socio-political contexts of these practices [7, 8]. One example is the “forgotten” power of paper as EHRs began to take shape; that is, how the paper record evolved within the sociotechnical work practices and how many of these nuances were lost with the introduction of the EHR [9]. A clear recognition of the social and technical subcomponents, including the recursive relationships within the day to day practice of an organization, is particularly well-articulated in some recent studies of HIT [10–12], where there is often a risk of technology-driven design and implementation without sufficient regard to the social subsystem [12]. Westbrook and others summarize that HIT problems are complex sociotechnical problems and that each subsystems within the sociotechnical system must be considered in relation to each other since optimization of one may negatively impact the other [14]. These integrated parts of a sociotechnical work system can be identified as: social, technical, and environmental subsystems (including internal and external) [15, 16]. Within human factors engineering, this sociotechnical view is sometimes referred to as a macroergonomics approach [15].

For CIS and ARK applications in the VHA, the social subsystem is comprised of the clinicians who use the applications for patient care, staff who use the applications for administrative purposes, and the patients whose care the applications are designed to support. The technical subsystem includes the CIS and ARK applications, the related computing systems that host the applications, as well as related HIT tools, including the VA’s CPRS/VistA. Physical environmental and local contextual factors are part of the internal environment. The external environment includes any external influences on the CIS and ARK applications, such as vendor support and national VA performance measurement. Ideally, the social and technical subsystems, as well as the internal environment, are balanced to meet the demands of the external environment [16]. Work practices in the sociotechnical system are networks comprised of elements from each of the subsystems. We describe our findings in the context of sociotechnical systems theory.

2. METHODS

2.1 Study Design Overview

We used a rapid ethnographic approach [17, 18], with data-collection methods that included field observations, semi-structured interviews with ‘key informants’, and opportunistic interviews. We chose rapid ethnography, as opposed to/classical ethnography’, primarily because of the time-constraints for this study; VA stakeholders expected our study results to inform rapid improvement of currently implemented CIS and ARK applications, as well as inform near-term product purchases. Moreover, we needed to minimize potential disruption to the participating clinicians, who were consumed with patient-care activities. The techniques employed for rapid ethnography allow us to understand the work system as a whole, from the larger sociotechnical components down to the computer interface level [19]. This is advantageous since the CIS and ARK applications exist in a complex sociotechnical system, where factors beyond the computer interface level, such as external organizational influences, have a direct impact on the usability of the applications. End users of CIS and ARK systems with various training backgrounds (e.g., nurses, anesthesia clinicians, and respiratory therapists) and levels of clinical experience were invited to participate in the observations or key-informant interviews by a host at each of the study sites a week or two prior to our site visits. Data-collection sites included three geographically distributed VAMCs, each located in a different VISN. These sites were specifically chosen based on the existence of reported issues with CIS and ARK, as well as the presence of a strong advocate to assist the data collection team. The purposeful selection of the three study sites was also intended to provide a representative sample of four of the five vendor systems currently implemented by VAMC’s. We did not include the fifth vendor because the study timeline made visiting additional sites prohibitive. Although each of the three study sites was chosen, in part, because of reported issues with the CIS and ARK applications, we believe these sites to be representative of a typical VA Medical Center with recent deployment of CIS and ARK systems. At each of the study sites, data collection occurred over the course of three days. Our objective was to find generalizable barriers and facilitators to the effective use of the CIS and ARK systems across the different vendors represented at the study sites.

2.2 Ethnographic Observation with Opportunistic Interviews

Ethnographic observations and opportunistic interviews were conducted with 69 of 88 CIS and ARK end-users at the three study sites during the months of April and May, 2014. During observations, two members of the data collection team separately shadowed participants as they interacted with the CIS and ARK systems during an actual work shift. Each observer shadowed a participant for at least an hour before moving on to another participant. Table 1 shows the breakdown of participants by background, area, and site.

Table 1.

Number of participants across study sites. The specific backgrounds of the key informants are not provided to protect their anonymity. RN = Registered Nurse, RT = Respiratory Technician; CRNA = Certified Registered Nurse Anesthetist; CNS = Clinical Nurse Specialist; MSA = Medical Support Assistant.

	Observation (with opportunistic interviewing)	Key Informant Interviews
VA Medical Center 1	18	3
(Observer 1)	2 RNs, 1 RT (CIS) 4 RNs, 1 CRNA, 1 Anesthesiologist (ARK)
(Observer 2)	4 RNs, 2 RTs, 1 Dietician (CIS) 1 RN, 1 CRNA (ARK)
VA Medical Center 2	23	7
(Observer 1)	6 RNs, 1 CNS, 1 RT (CIS) 3 CRNAs, 1 MSA, 1 Anesthesiologist (ARK)
(Observer 2)	9 RNs (CIS) 1 RN (ARK)
VA Medical Center 3	28	9
(Observer 1)	1 RN, 1 RT, 1 Physician (CIS) 3 CRNAs, 1 RN, 1 Anesthesiologist (ARK)
(Observer 2)	14 RNs, 2 RTs, 1 Physician (CIS)
-	2 RNs, 1 Physician Assistant (ARK)

Although it is important to include an observational method of end-users working with CIS and ARK in real-life situations, we did not rely solely on ethnographic observation since the relative time an end-user spends interacting with the CIS and ARK systems was unknown, making opportunistic interviewing an important supplementary data collection method during the ethnographic observation. Therefore, we conducted opportunistic interviews with the clinicians we observed to better understand the observational data and to obtain specific feedback from them about their use of the CIS and ARK systems. These were done as time permitted, so that the natural workflow of the clinician was not interrupted, such as when the clinicians had short breaks or could answer our questions concurrently while performing their clinical work. The interview discussions focused on the clinician-s perspective on the use of the CIS and ARK systems; specifically barriers and facilitators, and suggestions for improving the application. This opportunistic feedback was recorded with the observation notes and analyzed with the observational data. This feedback supplemented and aided understanding of the corresponding observations.

The observations and the opportunistic interview responses were recorded using handwritten notes, capturing observable activities and verbalizations. This strategy has been successfully used in prior studies of HIT [19, 20]. Clinician name identifiers were not recorded in the observation notes. All participants who were shadowed received a study information sheet prior to data collection and their patients were verbally informed of the purpose of our presence, as appropriate. Participants for the key informant interviews received a similar study information sheet. We did not record patient identifying information.

2.3 Key-informant Interviews

Semi-structured interviews were conducted with 19 key informants across the three study sites. These key informants were intended to be clinical managers or higher-lever administrators (not necessarily end-users) who had knowledge of the implementation of the CIS and ARK systems at the facility. The specific backgrounds of the key informants are not provided to protect their anonymity but include representatives from relevant stakeholder groups, including Biomedical Engineering, Information Technology, the VISN administrators, physician and nurse administrators. Semi-structured interviews allowed us to further understand differences and potential challenges in CIS and ARK application usage among clinicians and facilities. The semi-structured interview approach provided flexibility and gave participants an opportunity to identify and explain important information that might not have surfaced otherwise [21]. At the same time, responses were comparable across interviews, since the same core questions were asked of each participant. Unlike the opportunistic interviews, these interviews were scheduled in advance for 30 minutes, although some of the interviews lasted for an hour or more based on the willingness of the participant to extend the interview. Interviews (15) were audio recorded, as permitted by the participants, and professionally transcribed by a transcription service. One interview was not audio recorded because the participant chose not to allow us to record. Three interviews were not audio-recorded because they occurred over the phone. The complete set of core questions from the Key Informant Interview Guide are in Appendix A.

2.4 Analysis

Data analysis followed a process of abstraction of qualitative field observations, in which details that are specific to the setting were replaced by the underlying strategies and performance criteria relevant across settings [22, 23]. In other words, the data were represented at a higher level of abstraction such that observations could be integrated across cases to show patterns of behavior and recurrent themes related to the use of the CIS and ARK systems. The handwritten observations were typed promptly after each site visit, and a coding scheme applied to permit tracking of observer, site, and day. Two team members independently coded a sample of the data to develop an initial coding scheme. During a consensus meeting, the two team members resolved differences in coding to develop a unified coding strategy. This consensus coding scheme appears in Appendix B. Then the two team members used the agreed upon coding scheme to recode the initial sample of data as well as the rest of the data. Next, the coded observations were sorted into relevant categories from the coding scheme to establish patterns across participants and study sites. Findings were integrated across observations into meaningful patterns and data were abstracted into emerging themes, such as recurrent strategies for use of the CIS and ARK systems. Results from the observational analysis were combined with the semi-structured interview data to converge evidence from two different data sources and methodologies, thereby creating a “thick” description of the phenomenon [24].

3. RESULTS

We identified seven barriers and two facilitators to the effective use of CIS and ARK systems through our inductive analysis, many of which contain related subcategories. Note that many of the barriers can be framed as a facilitator and vice versa. For example, a full-time, dedicated facility level CIS-ARK Coordinator was a facilitator at VAMC 3; the lack of a comparable CIS-ARK Coordinator at the other study sites was a barrier.

3.1 Barrier 1: Integration Issues with other Software Systems

3.1.1 Integration with VistA and CPRS

The need for improved integration of the CIS and ARK systems with VistA and CPRS was a consistent theme across all three study sites. Frustrations related to: the CIS notes were not well integrated within CPRS; certain required documentation could be done in CPRS and not in CIS, which resulted in toggling between the two applications; and the inpatient medical-surgical wards having access only to CPRS and not to the CIS or ARK systems. At VAMC 1, Respiratory Service expressed frustration that not all of their patients could be found in the CIS and needed to be manually added. There was a desire to be able to “bridge” and “browse” automatically to CPRS when using the CIS or ARK systems and a general perception that it does not make sense to have a second system (CIS or ARK) that is not integrated closely with CPRS. At VAMC 3, this theme extended to actual charting inaccuracies where the CIS record was overwritten by VistA for date of surgery in certain scenarios (e.g., critical care transferring a patient back to the operating room). Better integration of the CIS or ARK system with VistA and CPRS could result in improved clinical work flow and increased adoption and effective use of the CIS and ARK systems.

3.1.2 Transmission of CIS or ARK record to VistA and CPRS

At all three study sites, a Portable Document File (PDF) is generated from CIS and ARK for the patient’s record when the patient is ready for discharge or transfer from the critical care or anesthesia area. This PDF is then transmitted to and saved in VistA Imaging for subsequent viewing through CPRS. Persistent errors for the transmission of the CIS or ARK PDF to VistA Imaging were apparent at all three study sites. At VAMC 1, participants noted that some data is missing from the PDF during the transfer of an ARK record to VistA Imaging. Further, several participants expressed a lack of confidence that records were transferring to CPRS and wanted additional automated confirmation of successful transfer. At VAMC 2, one participant noted that two or three times a month, only half the chart from ARK is transmitted and saved to VistA Imaging, without explanation. Similar issues were present at VAMC 3, where some of the PDFs from the CIS and ARK systems intermittently failed to transfer to VistA Imaging. However, VAMC 3 had a full time and fully dedicated CIS-ARK Coordinator on site to monitor these transmissions on a daily basis and find any PDFs that failed to transfer every 24-hour period. A relevant quote was: “Right now, especially for the ARK system, there’s no notification. So if QM [Quality Management] or someone who’s checking the records notice that it’s not in VistA imaging, that’s when -- that would be our first notification it didn’t make it across. And that could be a week, two weeks, a month later.” (VAMC 1). The elimination or minimization of errors in the transmission of the CIS or ARK records to VistA Imaging/CPRS may contribute to end-user confidence in the CIS and ARK systems.

3.1.3 Use of multiple applications

Another barrier to the use of CIS and ARK systems at all three sites was end-user frustration caused by the need to use multiple existing applications. In addition to the CIS or ARK system, many end users need to access CPRS, the Bar Code Medication Administration (BCMA) system, and other existing applications. Study participants expressed a desire for a smaller number or better integration of applications to increase efficiency of clinical work. More than one end user noted that having to work with multiple applications introduced the possibility of forgetting something and losing information. Physicians at VAMC 2 expressed frustration with having to go to multiple applications in order to get a comprehensive view of the patient: CPRS for physician orders, notes, and labs; CIS for overnight vitals and events; BCMA for medications. Similarly, at VAMC 3, a physician noted that s/he used to be able to quickly understand what was going on with patients by viewing the nursing note in CPRS. Now, s/he is referred to the CIS record and cannot see the same day CIS record remotely from home. The use of multiple applications also created a need for multiple sign-ins and a desire to streamline the log-on verification process. Finally, inconsistent time stamps (i.e., the time was not synced across devices and monitors) was another issue related to the use of multiple applications, especially with regard to the ARK systems (e.g., manually correcting times related to infusions). Integration of multiple applications to the clinical work flow may improve efficiency and reduce end-user frustration.

3.1.4 Duplicate documentation

There were cases of duplicate documentation with CIS or ARK systems and CPRS at each of the three study sites. However, duplicate documentation only seemed to be a consistent theme at the VAMC 2, likely due to the specific CIS vendor product in use at that site. For example, one RN at VAMC 2 noted, “You can’t just rely on CIS for charting. For narrative charting, you need CPRS. CIS is only good for hourly charting”. Skin assessment was consistently noted as a case of duplicate charting. An RN noted, “In CPRS you need the skin assessment every 12 hours; if I do skin assessment in CIS, it’s just double charting”. A CRNA noted, “Sometimes you have to write two different notes. Have to document using both systems [ARK and CPRS].” In many ways, duplicate documentation relates to the findings for integration of CIS and ARK with CPRS as well as use of multiple applications. Better integration of these applications may reduce the need for duplicate documentation and contribute to greater efficiencies in clinical work.

3.2 Barrier 2: Poor Usability

3.2.1 Customization

Inability to better customize any of the CIS and ARK systems was a consistent finding across all three study sites and was especially prevalent at VAMC 3. Participants noted that the CIS and ARK systems were not customizable to individual use, individual cases, or even customizable to their service’s needs. Feedback ranged from ‘not enough choices’ to ‘too many choices’ with regard to drop down lists, fields, and other components of the interface. In addition, lack of ability to customize the PDF output from ARK was a source of frustration at the VAMC 3. Factors affecting the ability to customize the CIS and ARK systems include: the contract with the vendor, vendor responsiveness, and the need for standard collection of data. Although a degree of standardization is necessary to support the roll-up of data from individual VAMCs to the national level, the lack of ability to customize the CIS and ARK systems for service-specific needs, in a timely manner, is a barrier to effective use of these systems for end-users. Clinicians expressed a strong desire for enhanced options with the vendor’s interface and records format in order to work more efficiently and effectively.

3.2.2 Data organization

The organization of the data on the user interface of the CIS and ARK systems was a major and consistent theme across each of the study sites. Participants noted that some fields seemed randomly placed or were redundant and could be organized in a more thoughtful manner. A key theme was a need to be able to quickly visualize the data to gain a high-level view. Organization of the PDF content generated data from CIS or ARK at VAMC 3 was heavily criticized by end-users. Criticisms included: an inability to highlight important information and locate it “front and center”; presentation of the data in difficult to read, narrow columns; the inclusion of empty cells; the inclusion of extraneous information; and the excessive number of pages for the PDF. As one nurse said: “When you look at the PDF [generated from CIS] it doesn’t make sense.” (VAMC 3). Clinical data that is well organized and respects workflow is a critical component of usability.

3.2.3 Ease of use and efficiency

The evidence was mixed in terms of the CIS and ARK systems facilitating or hindering ease of use and efficiency. End users, however, regarded ease of use and efficiency as priorities. In some cases participants felt that the CIS and/or ARK systems showed an over-reliance on user knowledge and memory rather than recognition (i.e., the end-user must remember information to complete a dialog or field). Some participants also noted the need for excessive scrolling and clicking. At VAMC 3, efficiency of ARK usage was questioned in the operating room for shorter-duration patient cases. The initial set-up of ARK for a patient case can be time-consuming, but is advantageous for longer stays in terms of efficiency and continuous, automated monitoring.

3.3 Barrier 3: Software Challenges

3.3.1 Analytics

Software challenges unrelated to usability impacted effective use of the CIS and ARK systems. One challenge was the inability to run reports and leverage the analytics capabilities of these systems. This finding was consistent across all three study sites and especially a concern at the VAMC 2. Nursing administration expressed a great deal of frustration about not having the ability to run reports from the CIS system. One RN noted that although s/he was told that certain reports could be run before the CIS system was deployed, this capability still had not been implemented. S/he noted that the nurses are spending a great deal of time documenting and entering data in the CIS system and they are not able to “get anything out of it” in terms of reports and analytics. The ability to leverage the analytics as an output of the CIS and ARK systems may increase the potential benefit of using these systems.

3.3.2 Reliability and trustworthiness of the data

Study participants shared perceived concerns about the reliability, quality, or trustworthiness of the data being entered in to the CIS and ARK systems. This was especially apparent at VAMCs 2 and 3. One RN at the VAMC 2 noted that with the CIS system there is “canned data”; that through the use of drop down menus end-users are selecting same things and as a result, you don’t get the “flavor” of the patient (i.e., patients all appear similar). One pharmacist noted discrepancies between what s/he sees in CIS and what the nurses are telling her/him. At the VAMC 3, an anesthesiologist noted that Anesthesiology staff members feel a need to scrutinize the ARK record carefully to have confidence that the record is accurate for each patient. We also observed a case of erroneous charting in real-time. During an active surgery case, anesthesiology staff discovered the patient’s record in ARK had erroneous charting in it. There were notes in the patient’s chart from a provider who was not in the room or even in the facility at the time of the surgery. Staff implemented the contingency plan and switched to paper flow sheet charting for the duration of the surgery. The cause of the erroneous charting incident was related to the previous end user not signing out of the application. Increased data reliability and data trustworthiness may result in increased confidence of end-users in using the CIS and ARK systems.

3.4 Barrier 4: Hardware Challenges

Hardware-related challenges pose barriers to the use of the CIS and ARK systems. At each study site, we noted examples of cabling or connectivity issues between devices and the CIS and ARK systems, which impacted the ability to send data accurately or at all. For example, at VAMC 2, an RN in the surgical intensive care unit (SICU) noted that there were frequent problems associated with the system not sending reliable data for various reasons (e.g., the patient moves and hardware becomes dislodged). Because the RN is busy s/he cannot easily readjust everything: “I have time to either deal with the patient or deal with the technology.” Other hardware-related issues included: not enough workstations, not enough data entry ports, and an ARK workstation unit being down for an extended period of time. Some hardware issues may be related to the specific vendor for the CIS and ARK systems. For example:

“[vendor 1’s product] is just a box of software. [vendor 2’s product] is a perfect turnkey solution because they insist that they install all their own [hardware] equipment and that they maintain that...” (VAMC 2)

“It [vendor 1’s product] was really hard to implement just because there’s so many pieces. All the different computers and wires and patch panels and wall ports and the com cables. And I could show you like how some of them are set up. It’s a little bit jerry-rigged, unfortunately.” (VAMC 3)

3.5 Barrier 5: Training Concerns

The use of ‘super-users’ is a training technique employed at all three study sites. Super-users are clinicians selected from each service or care area for extended training on the CIS or ARK systems. These super-users are then available to answer questions and assist other staff in their care areas. The degree of super-user training and roles of the super-users varied across the study sites. At VAMC 1, super-users received training over the course of three days for CIS and their role was then to train the end users in their care areas. VAMC 2 provided an eight hour CIS class for staff nurses plus an additional four hours of training for super-users. Super-users at VAMC 3 received eight hours of training on CIS and other staff received four hours. Not all staff received training outside of their care area and instead “learned on the job” (VAMC 2). Complicating factors for the super-user training strategy included turnover (some super-users had left the VA), as well as delayed roll-out of the CIS system (VAMC 1). That is, the time between training and the actual roll-out of a system was longer than expected and impacted retention of what was learned during the super-user training (reported at the VAMCs 1 and 3). Other complicating factors associated with training in general included commitment from administration to allow staff to attend training (VAMC 1) or training longer than 4 hours (VAMC 3), especially when adequate staffing levels were a concern for patient care. Inadequate training can often be used as an excuse to hide poor usability [25]. However, an adequate level of training is clearly necessary for complex critical care HIT systems. Finding the right balance between delivering training and improving the system design is the key, as illustrated by the following quote: “…they kept being told by the VISN, you need to educate the staff more, you need to educate the staff, you need to educate the staff. And so you know, at a certain point, you don’t need to educate the staff, you need to change the system.” (VAMC 3).

3.6 Barrier 6: Unclear Roles and Lack of Coordination among Stakeholders

The roles and coordination between the CIS-ARK Administrator, vendor, Biomedical Engineering, and Office of Information & Technology (OI&T) is often complex and confusing. Biomedical Engineering service has technical responsibility for the CIS and ARK systems because the systems are classified as medical devices. However, OI&T, the CIS-ARK Coordinators, and the vendor all have critical roles in supporting the CIS and ARK systems. The decision to classify the CIS and ARK systems as biomedical equipment has extensive implications. At each of the study sites, the relationships and coordination among Biomedical Engineering and the other stakeholders could be enhanced to improve deployment and support of these systems. The following are examples of less than ideal coordination among the stakeholders:

“…there was actually a cable missing [from the ARK system]… they looked right at us and said, well, go buy it. Yeah. And it had to be made. And it delayed the deployment. … You would think Biomed would know that, but you’d also think the vendor would know that.” (VAMC 1)

“So the vendor will get called, and we’ll try to get OI&T and Biomed again - that circuits [brings them] together to own who is going to clean and back up and do maintenance -- preventative maintenance so that the system doesn’t crash. And it usually only takes about three tries to do that. You never successfully get any one of those people to take that responsibility. On every project we have done, it has been, you know, who’s going to take responsibility, who’s really going to take responsibility, and then who’s really going to do it.” (VAMC 1)

Increased commitment and clearly defined roles and coordination among the various stakeholders may reduce problems with the successful implementation, support, and use of the CIS and ARK systems.

3.7 Barrier 7: Insufficient Technical Support

In addition to the routine technical support needed to maintain a complex critical care or anesthesia system, weak coordination and commitment among the stakeholders extend to and impact technical support for the CIS and ARK systems. Technical support from the vendors is also critical for the successful use of the CIS and ARK systems. We found a breadth of internal and external technical support frustrations at each of the study sites. These technical support issues mainly related to the lack of promptness of the requested support. In addition, there was a strong sense, especially at VAMC 2, that OI&T should be involved during the vendor selection process to ensure applications being considered were compatible with VA HIT infrastructure from a technical support perspective. Furthermore, regionalized purchases of different CIS and ARK systems, as opposed to enterprise level purchases for the entire VHA, seemed to be a substantial factor in the lack of sufficient technical support at the study sites. Increasing the sufficiency and speed of technical support, both internal to the VA, as well as external with the CIS and ARK vendors, may positively impact the application s reliability and end-user satisfaction.

3.8 Facilitator 1: Automation

At all three study sites, end-users appreciated the automation aspect of the CIS and ARK systems. The continuous feed of vitals and other data, without the need for manual entry or re-entry of data, was highly valued. A cautionary note, however, is that carrying data forward across time increments needs sufficient verification from the end-user to ensure that potential errors are not propagated. Also, an RN at the VAMC 1 noted that the increased automation may lead to a loss of situation awareness. This potential loss of situation awareness can be mitigated by ensuring sufficient feedback and verification cues and checks are designed into the interface. Finding ways to increase auto-population of data and automate additional sources of data may further increase end-user satisfaction.

3.9 Facilitator 2: Dedicated Facility Level CIS-ARK Coordinator

A key facilitator present at VAMC 3 was the designation of a 100% full-time employee devoted to CIS-ARK administration at the facility level. In contrast, the CIS Coordinators at VAMCs 1 and 2 were performing this role in addition to their other full-time responsibilities. The benefits of having a full-time dedicated CIS and/or ARK Coordinator were apparent:

“If [X] leaves tomorrow, we’re dead in the water. And it’s not even [his or her] job.” (VAMC 2)

“So for the facilities, we - the leadership - identified a CIS and an ARK coordinator. So they were -- they were still responsible for doing their regular day-to-day job in addition to supporting the CIS/ARK system. So there was no 100% dedication to any of the systems. And that -- that really caused a problem. And it still causes a problem because you can see -- when you went to the PACU [post-anesthesia care unit], we do have a system that’s down.” (VAMC 1)

The success of the CIS and ARK systems is impacted by the amount of time and effort formally designated to a facility-level CIS and ARK administrator. The facility level CIS and ARK administrator roles are essential for coordinating the support needed for applications of this level of complexity. A person with substantial time and effort devoted to these roles is likely to be more effective in coordinating the support needed from the vendor, Biomedical Engineering, OI&T, and other stakeholders for improved end-user experience.

4. DISCUSSION

CIS and ARK systems exist in a complex sociotechnical system, where factors beyond the computer interface level have a direct impact on system usability. These contextual, environmental, and organizational influences affect the ability of applications end-users to perform clinical work effectively, efficiently, and safely. We frame our findings in the context of sociotechnical systems theory, which distinguishes three integrated parts of a work system: social, technical, and environmental (internal and external) subsystems. Many informatics tools are technology-driven designs with little attention to the social and external influences on the system during design or acquisition and deployment. The study findings are evidence of less than ideal integration between the social and the technical subsystems. Ideally, the social and technical subsystems, as well as the internal environment, are balanced, or jointly optimized, to meet the demands of the external environment [15, 16]. Many of the reported barriers to the effective use of CIS and ARK systems are a result inadequate support of the work practices and misalignment of social and technical subsystems. Table 2 maps each of the barriers and facilitators from our analysis directly to the social, technical, and external environmental subsystems within sociotechnical systems theory. The unit of analysis for the sociotechnical system in this case is a VAMC. Thus, factors such as VISN-level funding, vendor technical support, and training provided by the VISN or vendor, would be considered part of the external environmental subsystem.

Table 2.

Study findings mapped to the subsystems that comprise sociotechnical systems theory.

Barriers and Facilitators	Sociotechnical subsystems	Implications
Barrier 1: Integration Issues with other Software Systems	Technical	Better integration with other software applications may lead to increased efficiency and confidence within the social subsystem.
Barrier 2: Poor Usability	Technical and External environment (vendor controlled)	Greater customization and organization of the data may result in greater satisfaction within the social subsystem.
Barrier 3: Software Challenges	Technical	Some software issues impact data reliability and data trustworthiness; reversing this may result in increased confidence within the social subsystem.
Barrier 4: Hardware Challenges	Technical	Some hardware issues impact deployment and use of the CIS and ARK systems; improving these issues will likely lead to better integration between the technical and social subsystems.
Barrier 5: Training Concerns	Social and External environment	An appropriate level of training may contribute to joint- optimization between the social and technical subsystems.
Barrier 6: Unclear Roles and Lack of Coordination among Stakeholders	Social and External environment	Increased commitment and clearly defined roles and coordination among the various stakeholders may contribute to joint-optimization between the social and technical subsystems.
Barrier 7: Insufficient Technical Support	Social (internal support) and External environment (vendor support)	Increasing the sufficiency and speed of technical support, both internal to the VA, as well as external with the CIS and ARK vendors, may positively impact the technical subsystem.
Facilitator 1: Automation	Technical	Finding ways to increase automation with the CIS and ARK applications may further increase end-user satisfaction within the social subsystem.
Facilitator 2: Dedicated Facility Level CIS-ARK Coordinator	Social and External environment (funding)	Provision of a full-time employee devoted to CIS-ARK administration at the facility level is likely to positively impact each of the other findings and increase the likelihood of achieving joint- optimization between the social and technical subsystems.

A number of the barriers identified in this study can be addressed with cooperation and collaboration with the vendors and changes to VistA to include critical care and anesthesia documentation. These include the ability to better integrate the CIS and ARK systems with VistA/CPRS and other VA applications; most usability issues, including a great deal of the customization needs; and better organization of the data displays, as well as other software challenges. Hardware challenges and external technical support may also necessitate the cooperation and collaboration of the vendor, depending on the specific vendor/system. However, some barriers are mostly under VA control and include those related to training, roles and communication between stakeholders, and internal technical support. A dedicated facility level CIS-ARK Coordinator is required, as well as ensuring that Biomedical Engineering and OI&T are sufficiently represented during the vendor selection process.

For any system, human factors engineering methods and/or best practices can be applied to early requirements specification, formative usability assessment (prior to implementation), system implementation, and summative usability assessment (post-deployment) as part of a holistic user-centered design framework. As a result of this study, we have strongly recommended that human factors engineering and usability requirements become an integral part of future applications purchases and implementation for VA as another strategy to help eliminate or mitigate some of the barriers identified in this study. In addition, we have recommended adding usability support language to the contracts for VA selection, deployment, and maintenance of commercial systems. To support the selection process, we recommend applications demonstrations by each of the prospective vendors to showcase applications features by walking through a series of use case scenarios. This would enable, for each use case scenario, vendor applications to be evaluated on usability of the user interface, interoperability with other VA systems, and support or enhancement of clinical workflow. For deployment, we recommend contract language that requires the vendor to work with a VA team to implement a published user-centered design process for deploying the system to VAMCs. For maintenance support, we recommend contract requirements that require the vendor to receive, maintain, and manage a list of known usability and technical problems and change requests for the deployed system. Further, we recommend contract language that requires evaluations of intended modifications to the system prior to implementation, as well as requirements to monitor the impact of changes to the system and ensure that user training on the application reflects any and all modifications made to the system. We believe these recommendations will have substantial impact on the ultimate success of an application in the VA clinical environment.

4.1 Limitations

The primary limitation of this study was convenience sampling of the CIS and ARK end-users. However, this is a standard technique employed when access to busy professionals is limited. While we visited only three of approximately150 VAMCs, the sample VAMCs were geographically distributed across three different VISNs and systems (i.e., each VAMC had deployed systems from different vendors). The three-day study duration at each site was also a limitation in that we did not observe many of the critical incidents that participants described as especially problematic with HIT system incompatibility or poor usability. In these cases we relied on our key informant interviews and opportunistic interviews with end-users. Another potential limitation is that one of the VAMCs had used a different vendor application for over 10 years, and the current vendor choice, had a very different front end design. This is a potential limitation since many of the clinicians at that study site may have been biased by their extensive experience with the previous application, as compared to sites were a CIS or ARK application was introduced for the first time.

4.2 Conclusion

Although this study was originally carried out for internal VA purposes, these findings are likely broadly relevant to other healthcare institutions. Many health care systems face integration of newer systems, whether commercial or in-house developed, with existing legacy systems, and without usability evaluation prior to purchase or development, significant usability issues can emerge post-implementation which result in a direct impact to workflow. Several mitigation strategies are being addressed directly with the vendors. Resolution of some of the barriers will require changes in the legacy applications. Due to the limitations and evaluations being conducted within the first year of implementation when systems were changing and evolving, a follow up study has been requested.

SUMMARY TABLE.

What is already known on this subject?

• Successful implementation and adoption of health information technology (HIT) in healthcare environments is an ongoing challenge.

• For intensive care units and other and other acute care environments, where rapid decisions and actions are often required, acceptance of HIT can be especially difficult to achieve.

• Formative usability assessment prior to implementation can be essential.

What is already known on this subject?

• We identified barriers and facilitators to the effective deployment and adoption of commercial-off-the-shelf Clinical Information Systems (CIS) for intensive care units and Anesthesia Record Keeping (ARK) for operating rooms and post-anesthesia care recovery settings.

• The use of rapid ethnography can result in rich data that can inform future applications purchases.

• We have recommended usability support contract language for vendor selection, deployment, and maintenance of commercial systems.

HIGHLIGHTS.

• We evaluated applications for intensive care units and anesthesia care settings.

• We identified barriers that explain challenges with adoption of these applications.

• Facilitators included automation and a dedicated facility-level CIS-ARK coordinator.

Appendix A: CIS-ARK Semi-structured Interview Guide

Note: These types of questions are meant serve as starting points for more in-depth, unstructured exploration of topics.

1. Were you involved in the vendor selection process for the CIS and ARK systems? How was a decision made to go with the current vendors?

   1. Have there been issues with having two different vendors for CIS and ARK?

2. What challenges do you or staff experience when using CIS and/or ARK?

3. Are you confident in the ability of CIS and/or ARK to alert staff of patient changes and emergencies? Why or why not?

   1. Tell me about the ability to tailor the system parameters - for example, change when staff are alerted about a patient’s blood pressure value?

4. Is there a need for staff to be able to access patient information from locations outside this VAMC related to the use of CIS and/or ARK? Do you feel staff are able to easily do this? Why or why not?

5. Does staff have a need to perform clinical documentation with CIS or ARK? Are staff able to document efficiently with the CIS and ARK systems? Why or why not?

6. Describe your process for obtaining technical support for the CIS and ARK systems when you need support.

7. What type of training did you and staff receive on the CIS and ARK systems? Do you feel it was adequate?

8. What impact, if any, has CIS and/or ARK had on workload to the end user?

   1. Was additional full-time equivalent employe0s (FTEE) put into place related to CIS or ARK?

   2. Has the implementation and use of CIS and ARK impacted positive or negative on resources?

9. What advice would you give someone that’s just starting to work with CIS/ARK?

10. What aspects of the system are the most difficult to learn in your opinion?

11. What aspects of the system could be best improved upon?

12. In your opinion, what would kind of changes would improve the CIS or ARK systems?

13. How have CIS and ARK facilitated your ability to provide patient care?

14. Have CIS and ARK presented any barriers to providing patient care? How have you responded to these barriers?

15. Were you here when CIS and ARK were implemented? What do you remember from that process?

16. (Time permitting): Any final comments about CIS or ARK at your facility and your experience with them?

Appendix B: CIS-ARK Study Coding Scheme

*Note, if more than one code clearly applies, assign more than one code to the row of data

*If no codes apply but the data is relevant to the study, assign the code ‘Other’

*If the data is irrelevant to the study objectives, leave blank or assign ‘No Code’

1. Usability

Examples

• Both negative and positive usability issues (e.g., increased automation)

• Interface fields

• Interface design

• Font size

• Warnings

2. Crossover

Examples

• Multiple applications

• Connection with VistA

3. Software issues

Examples

• Unrelated to usability

• System updates

• Reliability of the software

• Erroneous charting

• Data issues, legal concerns

• Analytics, ability to run reports

4. Hardware/cabling issues

Examples

• Connection with devices (e.g., ventilator)

• Not enough workstations

5. Technical Support

Examples

• Super-users

• OI&T

• Biomed

• CIS-ASK coordinator

• Deployment issues related to technical configuration

6. Training

Examples

• Super-users

• Non-use

• Deployment issues related to training

7. Workarounds

Examples

• Paper charting

• Computer-based workarounds

8. Vendor

Examples

• Vendor contract

• Vendor selection

9. Workflow

Examples

• Integration of CIS or ARK into clinical workflow

• Flow of documentation/software organization of charting

10. Other

• Allow for the emergence of new codes when no other codes apply