Appendix A. Summary of Studies on HFE-Based Healthcare System Redesign.

Projects~	Setting	Healthcare system being redesigned	HFE issues*	Work system elements related to intervention	Phases of redesign process				Impact
					Analysis	Design	Implementation	Evaluation	Care process and patient outcome	Other outcomes
1	A 118-bed hospital with 150 radiation therapists and 16 radiation treatment machines (Canada)	User interface of radiotherapy treatment delivery system	C	Tools and technologies	-- Analysis of the radiotherapy treatment delivery process 30 hours of field observation of radiation therapists performing their regular tasks conducted by an HFE expert Workflow analysis to identify tasks regularly performed by radiation therapists and areas associated with high likelihood of incidents Heuristic evaluation by 1 radiation therapist and 2 HFE experts to identify HFE issues of the system and evaluate their severity	-- Redesign of the existing radiotherapy treatment delivery system Redesign of the system based on HFE design principles 2 focus groups with experienced radiation therapists to collect feedback on the redesigned system Refining the system based on results of focus groups		-- Comparison of the redesigned and the existing radiotherapy treatment delivery systems User testing with 16 radiation therapy students using four scenarios related to typical treatment delivery tasks	-- Impact of redesigned system on care processes Decreased error rates for overlooking an important note and changes in approval dates Reduced mean task completion time	-- Impact of redesigned system on employee outcomes Improved user satisfaction
2	A 1200-bed academic hospital (Canada)	CPOE order set system	C	Tools and technologies	-- Analysis of existing paper admission order sets to Identify tasks for completing an admission order set Identify necessary functionalities of CPOE order set system	-- Design and redesign of the CPOE order set system Configuration of the system based on the content of existing paper order sets Heuristic evaluation by 3 HFE experts and 1 physician to identify HFE issues of the system Iterative redesign of the system based on HFE design principles		-- Comparison of the existing paper order sets, the original CPOE order set system and the redesigned CPOE order set system User testing with 27 end-user representatives	-- Impact of redesigned CPOE order set system on care processes Reduced mean task completion time No need for assistance No significant differences in number of errors
3	A 929-bed quaternary and tertiary referral teaching hospital, a 600-bed tertiary referral teaching hospital, a 330-bed community hospital, and a 80-bed referral center (Australia)	Paper medical observation chart used by hospital staff to document physiological observations over time for an individual patient	P, C	Tools and technologies	-- Development of usability heuristics for observation charts Informal task analysis carried out by an HFE expert to examine how charts are filled out and used by a range of clinicians Review and adaptation of usability heuristics from the software and web design domains -- Analysis of existing observation charts Heuristic evaluation of 25 general observation charts that were in use by 5 evaluators to identify usability problems and develop design guidelines	-- Design of a new observation chart Creation of a new observation chart using the newly developed design rules Heuristic evaluation of the newly developed observation chart to confirm that usability problems had been addressed as much as possible		-- Comparison of the redesigned observation chart and 4 existing observation charts User testing with experienced and novice chart users to examine the impact of the observation charts on the ability of health professionals to recognize patient deterioration	-- Impact of redesigned observation chart on care processes Reduced response time for detecting abnormal observations Reduced error rate for detecting abnormal observations
4	A major referral center handling 140,000 outpatient visits and 14,000 surgical cases annually (Netherlands)	OR floor marking	P	Environment	-- Benchmarking with aviation Site visits of airport conducted by a multidisciplinary team to learn experience of runway operators regarding marking, position of materials, traffic flows, safety rules and regulations, and incident management Field observation of ORs by the multidisciplinary team to learn surgical workflow and workplace layout	-- Decision made by the multidisciplinary team to use floor marking to facilitate consistency in the correct positioning of surgical devices and to minimize infection risks Integration of field experience from ORs with knowledge learned from aviation	-- Implementation of floor marking in a step-by-step fashion Temporary floor marking implemented in two of four ORs for six months Temporary floor marking implemented in all ORs for six months Permanent floor marking implemented in all ORs	-- Evaluation of system redesign Observation to evaluate compliance with positioning of surgical devices within the clean airflow Interviews with ophthalmic surgeons, surgical and anesthesia nurses, and managers	-- Impact of floor marking on care processes Increased compliance with recommended positioning of surgical devices in the clean airflow	-- Impact of floor marking on employee outcomes Enhanced safety awareness among surgical staff
5	Design and pilot testing: a 1,002-bed academic hospital Implementation and evaluation: 11 academic hospitals (6 intervention hospitals and 5 control hospitals) (Netherlands)	Checklist for surgical pathway from admission to discharge	C, O	Tools and technologies	-- Analysis of the surgical pathway and related critical safety risks Literature review to describe surgical pathway into distinct stages and to identify critical safety risks in each of the stages	-- Development of a checklist prototype based on the results of analysis and HFE literature on checklist design-- Validation of the checklist First set of observations of 41 surgical procedures by an independent researcher to identify process deviations Creation of an observation form based on the first set of observation Second set of observation of 130 surgical procedures to evaluate the extent of agreement between process deviations occurring in reality and items on the checklist Revision of the checklist based on results of observation	-- Pilot testing of the checklist with 350 surgical procedures during a 5-month period Instructive presentations about the checklist given to all users Introduction of the checklist in daily practice Structured interviews with 21 surgeons and 17 anesthesiologists at the end of the pilot test to assess usability of the checklist and to identify barriers to using the checklist Revision of the checklist based on the results of interviews -- Implementation of the checklist Formation of an implementation team consisting of a surgeon, an anesthesiologist and a quality control officer in each participating hospital Implementation of the checklist in a step-by-step fashion in the surgical wards and operating rooms	-- Evaluation of the impact of checklist on patient outcomes using a controlled, multi-center, prospective design Review of data on complication rate and in-hospital mortality collected during a 3-month period both pre- and post-intervention Review of data on compliance with the checklist -- Retrospective analysis of the effect of the checklist on care process (timing of antibiotic prophylaxis administration) in a single hospital	-- Impact of checklist on care processes Better compliance with regard to the timing of antibiotic prophylaxis administration -- Impact of the checklist on patient outcomes Decreased complication rate Complication rate positively associated with rate of compliance Decreased in-hospital mortality
6	Acute stroke unit of two public, tertiary hospitals (Australia)	Booklet for providing information to stroke patients	C	Tools and technologies, tasks, organization	-- Analysis of current practice in providing written information Interviews with 57 stroke patients and 12 care providers to (1) assess the content and design characteristics of existing written materials, (2) compare the reading level of existing written materials with the reading ability of stroke patients, and (3) assess information needs of stroke patients	-- Identification of system design considerations Literature review to identify design principles for health education materials A multidisciplinary focus group to identify factors that influence design and adoption of the computer system -- Design of the graphical user interface and associated database using a HFE approach Focus groups to identify graphical user interface requirements Interface development to incorporate all system and human functions Usability evaluation using feedback from clinicians Final design of the computer system	-- Pilot test of the computer system with 8 stroke patients and their care providers Interview with pilot test participants to evaluate their satisfaction with the system and to identify opportunities for improvement	-- Evaluation of the impact of the computer system Randomized control trial to evaluate the effects of providing stroke patients with computer-generated tailored written information compared to generic written information Randomized control trial to evaluate the effects of an education package (including computer-generated tailored written information, verbal reinforcement of information pre-discharge, and telephone contact post-discharge) on the knowledge, health, psychosocial and satisfaction outcomes of stroke patients	-- Impact of providing computer-generated tailored written information on patient outcomes Higher self-efficacy for accessing stroke information More positive of being informed Higher satisfaction with information received
7	A 594-bed academic hospital (USA)	Redesign of family history-tracking and pedigree drawing program for assessing genetic risk of hereditary cancer syndrome	C	Tools and technologies	-- Analysis of the original application Open-ended interviews with end users to determine tasks Hierarchical task analysis conducted by reviewing interview data with end users through scenarios Heuristic evaluation and user testing to uncover usability problems and the discrepancies between the users’ and the designers’ conceptual model -- User analysis of the redesigned application Survey to determine needs of potential users Comparative analysis of three commercial products available on the market Functional analysis of the redesigned interface	-- Creation of prototypes Paper prototypes built based on the results of analysis -- Small-scale usability studies Iterative heuristic evaluation based on usability principles and guidelines User testing with think-aloud method to validate interface design decisions and test alternative interfaces -- Modification of prototypes based on results of usability studies		-- Comparison between new and old systems using a controlled experiment Objective measures and questionnaire to determine whether the redesigned application decreases error rate, increases productivity, and increases user satisfaction	-- Impact of redesigned system on care processes Reduced mean task completion time No usability problems encountered	-- Impact of redesigned system on employee outcomes Improved user satisfaction
8	An adult ED in a 719-bed regional referral center and level 1 trauma center (USA)	Paper-based ED charting system	P, C	Tools and technologies	-- Analysis of existing charting system Observation to determine incidence of chart binder placement into the wrong chart rack slot and frequency of inappropriate chart binder transposition into an adjacent clinical unit Identification of deficiencies of the existing charting system, including inconsistent and small font labeling, a single-color scheme, and absence of HFE cues	-- Redesign of charting system Use of HFE principles and input from clinical providers and HFE experts to (1) improve labeling of chart binder for rapid identification of assigned patient room and clinical area, (2) enable easy distinction of a chart binder from others in proximity, and (3) highlight the misplacement of charts through design features that relied on intuitive pattern recognition -- Iterative redesign of charting system with ad hoc discussions with clinician end users	-- Implementation of redesigned charting system System phased in 86 individual patient care spaces in the ED over a 1-month period	-- Evaluation of the impact of redesigned charting system Comparison of pre- and post-implementation data on incidence of chart binder placement into the wrong chart rack slot and frequency of inappropriate chart binder transposition into an adjacent clinical unit	-- Impact of redesigned charting system on care processes Reduced number of chart binder location problems
9	Two 16-bed adult ED units in a 719-bed academic regional referral hospital (USA)	Interface of ED telemetry system	P, C, O	Tools and technologies, environment, organization	-- Assessment of baseline ED telemetry system performance in detecting life-threatening cardiac arrhythmias using on-site simulation -- Development of HFE knowledge base to define pre-intervention system state (e.g., hardware, task, process, user, organizational and environmental factors and issues) and HFE objectives Literature review Hardware inventory Functions diagnostic Real-time clinical use observation End-user survey of needs analysis Informal discussions with small user groups and institutional experts	-- Identification and grouping of design specifications for ED telemetry system in HFE categories (physical, cognitive and organizational HFE) Modified Delphi process involving investigators and ED stakeholders Iterative discussions with institutional biomedical engineers, device manufacturer and end users to verify feasibility, processes and value of the design specifications -- Integration of design specifications into a multi-element intervention to improve (1) system accessibility, (2) system relevance with enhanced signal/noise ratio and system utility for real-world ED practice, and (3) organizational processes for system sustainment	-- Incrementally implementation of the intervention over a period of 17 months Announcement of study conduct and intervention at ED personnel meetings Study simulation sessions Group training and on-shift training of ED personnel	-- Interim and post-intervention assessment of ED telemetry system performance in detecting life-threatening cardiac arrhythmias using on-site simulation -- Review of live environment alarm log records -- Collection of unsolicited anecdotal provider reports of system utility -- Informal survey of end users to assess matching of post-intervention system functions with user needs and to collect suggestions and feedback for future improvements	-- Impact of redesigned ED telemetry system on care processes Increased number of cardiac arrhythmias (e.g., ventricular tachycardia and sinus bradycardia) detected
10	A 230-bed tertiary care academic hospital (USA)	Computerized decision support system to improve adherence with deep venous thrombosis (DVT) prophylaxis	C, O	Tools and technologies	-- Assembly of a knowledge base on DVT prophylaxis Literature review on published clinical guidelines Collection of protocols from academic affiliates Discussion with subject matter experts -- Analysis of DVT prophylaxis process Observation with think-aloud method to understand the cognitive tasks, ordering patterns, and workflow of users Description of clinical workflow and design requirements	-- Design of computerized decision support system Creation of mock-ups of computerized decision support system Usability testing of mock-ups with surgeons Development of specialty-specific order menus based on feedback from surgeons	-- First implementation cycle Release of the new order menus Presentations to residents, pharmacists and nurses to describe the purpose and design of the new menus Informational flyers posted in all surgical staff and resident working areas and break rooms -- Second implementation cycle Observation, semi-structured interviews, focus groups and chart reviews of the prophylaxis orders to understand sociotechnical barriers to user acceptance of the system Reconfiguration of the order menus Training during resident orientation Engagement of clinical champions to support in-service trainees	-- Evaluation of the use of new order menus and the impact of new order menus on compliance with prophylaxis recommendations -- Evaluation of potential relationship between use of new order menus and organizational performance	-- Impact of new order menus on care processes Low acceptance and use of the new order menus during the first implementation cycle Improved acceptance and use of the new order menus during the second implementation cycle Improved compliance with prophylaxis recommendations after reconfiguring the order menus -- Impact of new order menus on patient outcomes Improved medical center indicator performance after reconfiguring order menus
11	A 404-bed academic hospital (Canada)	Interface of a patient-controlled analgesia (PCA) pump	C	Tools and technologies	-- Cognitive task analysis to identify information requirements of programming PCA pump Field observation and interviews with experienced nurses to collect feedback on the current device Bench tests to identify characteristics of the device that make its operation prone to error Assessment of the current device based on HFE design principles	-- Redesign of PCA pump Development of a new prototype interface based on HFE principles and identified information requirements		-- Evaluation of redesigned interface Experiment with nurse students and experienced nurses to compare redesigned interface and current device	-- Impact of redesigned system on care processes Reduced programming time Fewer errors	-- Impact of redesigned system on employee outcomes Lower mental workload ratings
12	Not specified	Hospital code cart medication drawer	P, C	Tools and technologies	-- Analysis of current medication drawer User testing with nurses, pharmacists and nursing managers to identify usability problems	-- Redesign of medication drawer Development of several modified drawers based on comments and suggestions from user testing participants User testing of modified drawers until the final prototype drawer was created		-- Evaluation of redesigned medication drawer User testing to compare medication retrieval time and number of wasteful actions associated with current and redesigned drawers Questionnaire to examine user experience with redesigned medication drawer	-- Impact of redesigned medication drawer on care processes Lower medication retrieval time Fewer wasteful actions Improved perceptions of medication drawer visibility, usability and organization

^~The projects’ numbers refer to the list in Table 3.

^*HFE issues. P: physical; C: cognitive; O: organizationa