Human factors analysis, design, and evaluation of Engage, a consumer health IT application for geriatric heart failure self-care

Abstract

Human factors and ergonomics (HFE) and related approaches can be used to enhance research and development of consumer-facing health IT systems, including technologies supporting the needs of people with chronic disease. We describe a multiphase HFE study of health IT supporting self-care of chronic heart failure by older adults. The study was based on HFE frameworks of “patient work” and incorporated the three broad phases of user-centered design: study or analysis; design; and evaluation. In the study phase, data from observations, interviews, surveys, and other methods were analyzed to identify gaps in and requirements for supporting heart failure self-care. The design phase applied findings from the study phase throughout an iterative process, culminating in the design of the Engage application, a product intended for continuous use over 30 days to stimulate self-care engagement, behavior, and knowledge. During the evaluation phase, we identified a variety of usability issues through expert heuristic evaluation and laboratory-based usability testing. We discuss the implications of our findings regarding heart failure self-care in older adults and the methodological challenges of rapid translational field research and development in this domain.

1. Introduction

1.1. Human factors/ergonomics (HFE) and health IT

Human factors/ergonomics (HFE) approaches and methods are more prevalent in healthcare than ever before (Xie & Carayon, 2015). Given the prominence of information technology (IT) in contemporary healthcare delivery, it is not surprising that a growing area of human factors in healthcare is the application of HFE methods such as user-centered design and usability evaluation to health IT (Carayon et al., 2013; Patel & Kannampallil, 2014; Zahabi, Kaber, & Swangnetr, 2015). Major US reports recommend attention to human-computer interaction when designing or deploying health IT (Institute of Medicine, 2012; Stead & Lin, 2009) and others provide guidance on specific HFE considerations and methods (Middleton et al., 2013; Schumacher & Lowry, 2010). HFE is part of the curriculum of medical and nursing informatics (Kulikowski et al., 2012; Staggers & Thompson, 2002) and an element of the new board certification in clinical informatics (Finnell & Dixon, 2015).

Traditionally, health IT has meant clinician-facing systems such as electronic health record (EHR) or clinical decision support (CDS) systems used by physicians, nurses, pharmacists, and other healthcare professionals (HCPs). Today, health IT is also designed for and used by patients, family members, community groups, and other non-HCPs. Sometimes these individuals are called healthcare “consumers” or, more simply, “citizens” (Doherty & Mendenhall, 2006). With growing computer literacy, IT ownership, and Internet access, such patient- or consumer-facing health IT (CHIT) has received increasing attention in the US and has been addressed in regulations such as Meaningful Use and initiatives such as the Blue Button campaign (Ricciardi, Mostashari, Murphy, Daniel, & Siminerio, 2013). Importantly, there is national recognition of the value of HFE methods for ensuring usable, effective, and acceptable CHIT. For example, a US Agency for Healthcare Research and Quality (AHRQ) report recommends that designers of CHIT “engage human factors experts in the design team” (Agarwal, Anderson, Crowley, & Kannan, 2011, p. 63). Reflecting on AHRQ-funded CHIT projects, Zayas Cabán and Dixon (2010) concluded:

Human factors and ergonomics should be incorporated early and iteratively into the design of consumer health IT. If human factors and ergonomics considerations are incorporated at the end of the development process, any redesign work may be resource-intensive and significantly impair technology acceptance… Therefore, developers should include human factors professionals in the multidisciplinary design team to adequately inform the concept, needs assessment and development process. (p.i66)

1.2. Health IT for chronic disease management.

Supporting the needs of people with chronic disease has been a particularly fruitful avenue of health IT research and practice. Various forms of CHIT systems have been studied, from patient portals, to self-management ‘apps’ for mobile devices, to websites and online communities (Finkelstein et al., 2012). Reviews demonstrate overall efficacy of these systems for managing chronic illnesses such as chronic heart disease (e.g., heart failure, hypertension), lung disease (e.g., asthma, chronic obstructive pulmonary disease), and diabetes (Finkelstein et al., 2012; Gibbons et al., 2009). However, not all are successful, particularly with respect to usability, integration of health IT with daily life, and addressing critical issues such as security, trust in IT, and added burden (Jimison et al., 2008). As many people with chronic disease are older, health IT must address age-related needs as well as the integration of informal caregivers such as the family members who assist older adults (Ji et al., 2010; Dyer, Kansagara, Mclnnes, Freeman, & Woods, 2012; U.S. Department of Health and Human Services, 2012; Nunes & Fitzpatrick, 2015). This has not been fully achieved, according to a systematic review of CHIT for older adults (Vedel, Akhlaghpour, Vaghefi, Bergman, & Lapointe, 2013) and national reports (National Research Council, 2011; U.S. Department of Health and Human Services, 2012). We therefore identify a need for applying HFE methods to improve the design and value of CHIT for older adults with chronic disease.

1.3. HFE study of CHIT for geriatric heart failure management.

In response to this need, we performed a multiphase project using HFE study, design, and evaluation methods to produce a new CHIT system for older adults with heart failure. The study was performed on the premise that HFE methods and a user-centered design approach enhance the ease of use and effectiveness of CHIT, ensure better integration of CHIT into daily life, and promote acceptance and use (Holden & Karsh, 2009; Marquard & Zayas Cabán, 2012; Zayas Cabán & Dixon, 2010).

Chronic heart failure (CHF, also known as congestive heart failure) is a costly, debilitating chronic disease affecting 5.7 million Americans and 12% of older adults (Mozaffarian et al., 2016). In heart failure, the heart’s pumping or filling function is impaired, resulting in worsened delivery of oxygen and expulsion of waste, including water. The accumulation of water in the body causes symptoms such as swelling and shortness of breath; it is life threatening when localized in the lungs (called pulmonary edema) and is one of the top reasons for emergency room visits and hospitalizations among older adults (Chen, Normand, Wang, & Krumholz, 2011). Managing heart failure involves a number of self-care behaviors, including taking medications, daily weighing and vitals assessment to detect physiological changes, self-monitoring for symptoms, exercise to strengthen the heart muscle and lose weight, sodium-restricted diet, and often fluid restriction (Riegel et al., 2009). People with heart failure must also attend clinic visits, communicate with HCPs, seek additional care as needed, and sometimes receive medical interventions such as device implantation (Yancy et al., 2013). In short, heart failure is a serious disease with multiple requirements from the patient to perform routine and situational behaviors that help manage both long-term health and acute conditions (Riegel, Lee, & Dickson, 2011).

2. HFE conceptual frameworks

Three frameworks concerning HFE, healthcare, and health IT guided this study and connect it to the broader practice of user-centered design (see Figure 1). The first is SEIPS 2.0 (SEIPS stands for Systems Engineering Initiative for Patient Safety), an organizing framework for HFE in healthcare (Holden et al., 2013). SEIPS 2.0 states that both HCPs and non-HCPs (e.g., patients, families) perform health-related work, defined as the “exertion of effort and investment of time on the part of patients or family members to produce or accomplish something” (Strauss, 1993, pp. 64–65). Work performed by HCPs without active patient or family involvement is called “professional work,” whereas work performed by non-professionals without HCP involvement is called “patient work.” In reality, most health-related processes are of a third kind of work, “collaborative patient-professional work,” involving the active participation of both professionals and nonprofessionals. Examples include communication during a clinic visit and the long-term management of a chronic disease. The role of patients and families in this work is usually uncompensated and may be unacknowledged (Ancker et al., 2015), making it what sociologists call “invisible work” (Daniels, 1987). According to the SEIPS 2.0 model, health-related work occurs within and is shaped by a sociotechnical work system that produces work processes, which subsequently shape outcomes (Figure 1a). The work system includes the interacting components: person(s); tasks; tools or technologies; organization; and internal and external environments. SEIPS 2.0 also describes the multiple outcomes produced by work processes and the feedback loop representing planned and unplanned adaptation of work structures and processes over time. Overall, the SEIPS 2.0 model describes (1) how dynamic configurations of sociotechnical systems produce health-related work over time, (2) how patients, families, HCPs, and others within the system engage in health-related work either separately or collaboratively, and (3) how the system adapts and evolves in planned and unplanned ways through a feedback mechanism.

Figure 1.

The HFE conceptual frameworks that guided this study. (a) SEIPS 2.0 model depicting the sociotechnical work system, performance processes, and outcomes with adaptation and feedback mechanism. (b) Different lenses for designing consumer health IT, with the broadest being the patient work lens. (c) Three broad, iterative phases of user-centered design and example activities in each. (Image used with permission from Richard J. Holden, myows.com license #97820.)

The second conceptual framework is the Patient Work Lens for CHIT (Valdez, Holden, Novak, & Veinot, 2015b). It applies the above work system and patient work concepts to the domain of CHIT. The framework highlights the broader context in which patient work occurs and argues that successful CHIT is “Designed to Align” (p.2) with this context. The framework integrates with standard user-centered design processes to expand beyond an individual and account for activities and contexts within which intended users are embedded (Figure 1b). A key premise is that health work such as chronic disease management is not merely biological or psychological, but rather biopsychosocial. The Patient Work Lens framework also distinguishes between clinical technologies, consumer technologies, and collaborative technologies, based on the SEIPS 2.0 distinctions in types of work (Valdez, Holden, Novak, & Veinot, 2015a).

The last framework outlines a cycle of three phases for user-centered design of CHIT: study (or analysis); design; and evaluation (Holden, Voida, Savoy, Jones, & Kulanthaivel, 2016). The framework describes an iterative process wherein practitioners (i) seek to understand the users, their tasks, goals, different aspects of the surrounding environment, and broader contexts, (ii) design abstract representations or more traditional artifacts such as wireframes or user interface prototypes, and (iii) evaluate the designs against initial understandings of users and goals (Figure 1c). We refer interested readers to a variety of supporting literature, as the framework reiterates a ubiquitous and fairly standard approach in design science and practice (Scapin, 1990; Bagor et al., 2008; Beyer & Hotzblatt, 1998; Buxton, 2007; Card et al., 1983; Carayon, 2012; Gennari & Reddy, 2000; Greenberg et al., 2011; John et al., 1996; Johnson et al., 2005; Karsh et al., 2006; Lindgaard et al., 2006; Lim et al., 2008; Nielsen, 1993; Norman, 1986; Preece et al., 2002; Polson et al., 1992; Pruitt & Adlin, 2006; Snyder, 2003).

In the following sections, we report the methods and results of each phase of the user-centered design framework (Holden, Voida, et al., 2016) applied in our development of CHIT for older adults with heart failure.

3. Study (analysis) phase

3.1. Methods

Between 2012–2014, a core set of data were collected from and about 63 older adults and 35 informal caregivers (family members) in the Southeast US. About half of the patients were recruited from outpatient cardiology and heart failure clinics of a large academic medical center. The rest were recruited from a larger study of adults discharged from the same medical center. All patients were aged 65 or older, diagnosed with CHF with mild, moderate, or severe CHF functional status, and lived within a 200-mile radius in a region including rural and urban areas of two US states. About half had diastolic heart failure (with preserved left ventricular ejection fraction), a quarter had systolic heart failure (reduced ejection fraction), and a quarter had systolic/diastolic CHF. Their mean age was 73.3 (median = 72, SD = 6.7, range 65–86); 51% were male; 74% were White non-Hispanic; 31% lived alone, 54% lived with a spouse, and 15% lived with another family member; 90% were retired; 61% had an annual household income less than $50,000 (31% less than $25,000); 34% had a high school degree, 15% did not graduate high school, and 51% had completed some college or had a college degree. A majority had other chronic conditions besides heart failure: high cholesterol (82%); high blood pressure (90%); and diabetes (60%). They took on average about 17 medications per day.

Typical data collection for a patient and his/her informal caregiver included observations of scheduled outpatient clinic visits, standardized surveys (97% response rate), electronic medical record review and abstraction, and either a 30-minute interview followed by a 90-minute follow-up interview or one extended 1.5–2 hour interview. Interviews and surveys focused on the characteristics of patients and informal caregivers, their self-care tasks, technologies used for self-care, and the context of self-care. A variety of specific probes and questionnaires were used, for example, assessing general health, self-care knowledge, perceived control, health literacy, self-care adherence, self-care workflow, support from others, physical environment, and household obligations. Because the main objective of data collection was to understand the heart failure patient work system, processes, and outcomes, topics were chosen based on the SEIPS 2.0 model and Patient Work Lens framework. For the patients recruited post-discharge, we received consent to analyze data from extensive interviewer-administered surveys collected for a parallel study during the patient’s hospital stay and at approximately two, 30, and 90 days post-discharge (Meyers et al., 2014). An additional set of data was also collected using a structured researcher-administered questionnaire with an additional 31 patients in the US and 35 in Singapore presenting to comparable urban emergency departments with suspected acute heart failure.

All participants, including patients, informal caregivers, and observed clinicians provided informed consent. Individuals who consented to videorecording indicated whether they wished their faces blurred and how researchers were permitted to use the video. Patients and caregivers were paid up to $65 per patient; clinicians were not compensated. The study was approved by the Vanderbilt University Institutional Review Board and Human Research Protection Program.

Data were professionally transcribed or, for medical record and survey data, coded by researchers. Various analyses have been performed and their findings are summarized below. Although a total of 129 patients participated, analyses have been performed on subsets ranging from n=21 (e.g., Cornet, Holden, & Voida, unpublished) to n=61 patients (e.g., Mickelson, Unertl, & Holden, 2016), with the typical analyzed data set of about n=30 patients (e.g., Holden et al, 2015). Informal caregiver data were included when analyzing the corresponding patient’s data. For most qualitative analyses we used NVivo software and applied descriptive qualitative content analysis or model-based deductive techniques with iterative theme development. Analyses were used to address a variety of questions and therefore differed in their analytic models. Several analyses used a work system model like SEIPS 2.0 to identify and describe characteristics of the work system (Holden et al., 2015b; Holden et al, 2015c; Holden et al., in press). Others focused on a specific element of the patient work system, such as tools and technologies (Mickelson et al., 2015) and context (Ye & Holden, 2015) or on work processes and outcomes (Mickelson et al., 2016). In most cases, multiple coders performed analyses using an agreed-upon analytic model and codebook developed after an initial analysis phase, during which coders analyzed the same set of transcripts. Following this, distinct subsets of data were assigned to individual coders and regular coding discussions were held, during which coding was discussed, disagreements were resolved, the codebook and code definitions were updated, and analysts recoded their assigned datasets. This process is described further elsewhere (Holden, Schubert, & Mickelson, 2015) and is a common approach to ensuring analytic convergence (Barry, Britten, Barber, Bradley, & Stevenson, 1999). Quantitative data were analyzed using a combination of Microsoft Excel, SPSS, and MINITAB. Methods, including distinct analysis approaches, are described more fully in publications listed in Table 1 or can be obtained from author RJH.

Table 1.

Summary of findings from observations, surveys, interviews, and medical record reviews

Analysis	Summary of findings	Major themes (see text)
Patient work system (Holden, Schubert, & Mickelson, 2015)	Describes the work system of patients with CHF, including the person(s), tasks, tools and technologies, and contextual factors shaping self-care performance. Identifies task complexity and difficulty as an important factor in poor performance but highlights the interactive nature of patient work system factors, as opposed to single-factor causation.	Suboptimal information work; Inadequate tools; Collaborative self-care; Disengaged patients; Knowledge gaps; Integration requirements
Macroergonomic elements of patient work systems (Holden, Valdez, Schubert, Thompson, & Hundt, 2016)	Describes the physical, social, and organizational context of chronic disease self-care, at household and community levels. Highlights the multiple obligations and roles of patients and their social networks. Identifies workload and routines as well as routine disruptions as important phenomena.	Collaborative self-care; Integration requirements
Information work model*	Describes the nature of and breakdowns in the information management work performed in the course of CHF self-care, in particular, the monitoring and detection of symptoms and events, their interpretation, and responsive actions. Also identifies information logging and communication activities.	Suboptimal information work; Inadequate tools; Collaborative self-care; Disengaged patients; Knowledge gaps
Analysis	Summary of findings	Major themes
Systems barriers to self-care (Holden, Schubert, Eiland, et al., 2015)	Identifies some of the self-reported reasons for patients’ difficulties performing self-care, including many related to the person (e.g., lack of knowledge) and some related to disruptions of routine, lack of resources, and other contextual factors. Highlights the interaction of biopsychosocial factors shaping self-care.	Disengaged patients; Knowledge gaps; Integration requirements
Distribution of medication management across people (Mickelson & Holden, 2013)	Assesses how medication management is distributed across people, time, place, and objects/tools. Identifies formal and informal caregivers as key actors in “self”-care. Actors include family members, community members and friends, various physicians, nurses and nurse practitioners, community pharmacists, case managers, and more. Demonstrates that artifacts and physical environments are used to support memory, organization, medication administration, and other processes as well as to communicate between actors.	Inadequate tools; Collaborative self-care; Integration requirements
Cognitive artifacts used for medication management and self-care (Mickelson & Holden, 2015; Mickelson, Willis, & Holden, 2015)	Analyzes the use of cognitive artifacts in the performance of medication management, including assessment of benefits and gaps. Identifies major gaps between patients, informal caregivers, and clinicians not bridged by technologies or exacerbated by the use of misaligned tools.	Suboptimal information work; Inadequate tools; Collaborative self-care; Knowledge gaps; Integration requirements
Analysis	Summary of findings	Major themes
Geospatial analysis of self-care (Ye & Holden, 2015)	Identifies and ranks by frequency of mention the places where self-care occurs, as well as ways in which “place matters.” Demonstrates that self-care takes place in the home, in clinics and hospitals, in transit, in public places in the community, in schools and at work, and elsewhere; common effects of place include social influence through place-related norms; multipurpose nature of space (e.g., as social and instrumental); and misalignment of geospatial design with patient needs.	Integration requirements
Workflow of self-care and medication management*	Maps the workflow of self-care and medication management as both task sequences and high-level macrocognitive functions such as sensemaking, planning, monitoring, decision making, and coordination. Notable findings include that the most frequent source of information is one’s social network, followed by clinicians; during decision making, patients have difficulty applying appropriate rules; and expected their electronic health records to track data for them.	Suboptimal information work; Inadequate tools; Collaborative self-care; Disengaged patients; Knowledge gaps; Integration requirements
Patient-centered communication during clinic visits*	Assesses patient encounters with physicians for elements of patient-centered care, defined as elicitation and responsiveness to patients’ preferences, needs, and values.	Suboptimal information work; Collaborative self-care
Analysis	Summary of findings	Major themes
Activity theory analysis of patient-clinician-caregiver interaction*	Analyzes clinician, patient, and informal caregiver work and interactions using the activity theory formalism. Distinguishes between “self-reliant,” “patient as innovator,” “physician as coach and coordinator,” “clinician as scientist,” “building trust,” “distributed care,” “sensemaking,” and “advocacy” subtypes of self-reliance, patient-clinician cooperation, and patient-support network cooperation.	Suboptimal information work; Collaborative self-care; Disengaged patients; Knowledge gaps
Safety and resilience in medication management*	Applies a dynamic systems model of safety to 70 medication non-adherence events classified as lapses, slips, mistakes, and violations. Depicts the boundaries (goals) of performance and performance shaping factors related to non-adherence events. Identifies performance shaping factors such as elements of the person or team (e.g., patient limitations), task (e.g., complexity), tools and technologies (e.g., tool quality), and organizational, physical, and social context (e.g., resources, support, social influence).	Suboptimal information work; Inadequate tools; Collaborative self-care; Disengaged patients; Knowledge gaps; Integration requirements
Analysis	Summary of findings	Major themes
Biopsychosocial personas*	Performs quantitative and qualitative analyses to extract up to six personas (or archetypes) of heart failure patients, accounting for personal and technological characteristics, as well as aspects of their healthcare systems and social environments.	Integration requirements
Transitions from hospital to home-based self-care*	Analyzes data from heart failure patients who were discharged from a hospital for acute heart failure decompensation. Identifies themes related to: discordant patient vs. clinician understanding of the post-discharge reality; the challenge of constructing or reconstructing routines in the post-discharge period; and changes in workload experienced by patients following discharge.	Inadequate tools; Integration requirements

^*Unpublished or under review

3.2. Results

Analyses in this phase produced a variety of relevant findings reported fully in other publications and summarized in Table 1. Across these analyses, there were six major themes that were particularly instrumental to the subsequent design stage. These themes are reported for each analysis in Table 1 and described below.

1. Suboptimal information work. Patients perform information work, but not optimally. Information work has a core sequence of monitoring, interpreting, and acting (see also Riegel, Dickson, & Topaz, 2013), as well as the secondary processes of logging and communicating data with. Not all perform each part of the sequence in each case, as for example, a person might notice shortness of breath, interpret it as caused by fluid retention, but take no action. Further, some of the information work is “outsourced” to others, shifting the locus of control away from patients; for example, a patient might let a physician or family member monitor their ankle swelling, or a patient who experiences fatigue might wait until the next clinic visit to get the physician’s interpretation.

2. Inadequate tools. Existing tools are inadequate to support patients’ information work, particularly interpreting data and problem solving, keeping meaningful data logs, and communicating data to others. Few who log data receive adequate feedback from clinicians and, not surprisingly, few sustain optimal logging behavior.

3. Collaborative self-care. The self-care process can generally be described as collaborative patient-professional work distributed across time, place, artifacts, and a network of actors, including patients, informal caregivers, community members, and clinicians.

4. Disengaged patients. Many patients are not engaged or motivated to sustain self-care or disease management – with the extreme result being the “outsourcing” of self-care mentioned above. Others engage with self-care occasionally, for a few days at a time. In many cases, patients’ motivation is not enough to overcome the burden of self-care.

5. Knowledge gaps. Major gaps in self-care knowledge are related to functional associations and knowing how to implement general instructions (see also Granger, Sandelowski, Tahshjain, Swedberg, & Ekman, 2009). Many have difficulty making connections between, for example, their medications, symptoms, heart function, and behaviors such as exercise and diet. The typical individual has a partial mental model of these interactions; for example, one might understand that a diuretic medication was related to removing fluid from the body, but not that the diuretic compensates for a weakened heart. Examples of knowledge gaps related to implementing instructions are not being able to tell the severity of symptoms or not knowing how to perform recommended daily exercise when outdoor weather was poor or the patient was in pain.

6. Integration requirements. There were several requirements for integrating a new technology or process into a patient work system: minimal additional workload; embedment of the intervention into existing routines; portability; support for collaboration and coordination; making explicit information and needs that are often implicit; and customizability.

3.3. Discussion

In sum, we identified several gaps in and requirements for self-care performance. The three that most influenced subsequent design were our observations of: (1) lack of support for information work, including logging and communication of data about symptoms and behaviors; (2) psychological and behavioral disengagement from self-care and disease management; and (3) lack of relational and applied knowledge. Our analyses also identified important conditions to consider during design such as workload and collaboration.

4. Design phase

4.1. Methods

Design was performed by a team with rotating personnel, led by the study’s principal investigator (author RJH), a PhD-trained human factors scientist with expertise in HCI, health informatics, and social-cognitive psychology. Other team members were three PhD-trained experts in HCI and computer science and Master’s and PhD students of HCI and health informatics with graduate degrees in HCI, computer science, information science, and geographic information systems. Our team consulted with clinical experts including nurses, cardiologists, and cardiovascular nurse practitioners and informal caregivers at various points in the design process. During the design phase, the team’s goal was to address needs identified during the study phase (see section 3.3) and to create a product to support heart failure self-care by older adults and informal caregivers. A reduced set of design requirements (Table 2) derived from the study phase (see point 6, section 3.2) was used to promote the design of a system that is (i) viewable by the patient (primary user), potentially extending to be viewed by informal caregivers and clinicians, (ii) minimally burdensome, (iii) aligned with the patient’s routines and preferences, and (iv) personalizable.

Table 2.

Reduced design requirements

Full requirements list based on findings (see section 3.2)	Reduced set
Minimal additional workload Embedment of the intervention into existing routines Portability Support for collaboration and coordination Making explicit information and needs that are often implicit Customizability	Viewable by the patient (primary user), potentially extending to be viewed by informal caregivers and clinicians Minimally burdensome Aligned with the patient’s routines and preferences, Personalizable

We followed a multi-step process beginning with educating the research team on the problem through multiple presentations, examples from the data, articles, and question-and-answer sessions. Next, the team performed divergent brainstorming and conceptualization of various solutions before converging on and further developing the leading solution. During divergent conceptualization, many ideas were elicited, but the team converged on four design challenges and candidate solutions for each, described in Table 3. A design challenge was essentially a restatement of a smaller set of specific gaps in self-care identified during the study phase. Several design solutions based on concepts such as serious games and gamification, social networking, use of sensors or wearable devices, and other approaches were discussed but not incorporated in further design work.

Table 3.

Summary of four design challenges and candidate solutions, reproduced verbatim from design notes

Motivating finding from study phase	Design challenge	Candidate solution
Disengaged patients; Inadequate tools	Can we motivate people to log data over a continuous period of time and, if so, what benefits can be gained from it?	“30-day challenge” for logging health data to support self-care behavior over a short period of time.
Suboptimal information work; Inadequate tools; Knowledge gaps	Can we modify knowledge and behavior by supporting cause-effect simulation and experimentation to depict the functional relationships between the different parts of human body and behavior?	The “Hearty Humunculus,” a simulation of the effect of different self-care behaviors to support learning and testing of dynamic functional relationships.
Disengaged patients; Inadequate tools	Can we use data logging to support goal setting and problem solving?	A tool to promote behavior by setting goals and helping identify and resolve barriers to their goals.
Suboptimal information work; Inadequate tools; Collaborative self-care	Can we structure and improve patient/caregiver-clinician communication regarding heart failure patient data?	A structured clinic visit tool to support and enhance communication and collaboration between patients and clinicians before and during clinic visits.

In the convergent phase of design, the first two solutions in Table 3 received the most interest in the design team and were further elaborated. Each received roughly equal support, leading to the principal investigator making an executive decision, based on the heart failure interventions literature, to pursue the “30-day challenge” design. As described in Table 3, the 30-day challenge refers to the notion that a successfully designed intervention could engage a patient in 30 days of continuous, self-reflective logging and monitoring of heart failure related data. Because the goal of such design would be to address the finding that some patients are disengaged from self-care, at this point, this concept was renamed to “Engage.” Elements of the other three solutions, for example knowledge and goal-setting components, were later integrated into Engage.

Subsequent design meetings focused on designing the overall Engage concept, device platform, content, functionality, information architecture, navigation, and user interface. After several design sessions, the team split to work separately on back-end and front-end development. Front-end design and development was supported by whiteboard and paper sketches, Microsoft PowerPoint wireframes and mockups, and lastly interactive prototypes in Axure (v7) prototyping software. Design meetings comprised of three to six team members were regularly held to make decisions and refine products. In the rest of this paper, we describe mainly the front-end user interface development and testing because of the sophistication of the back-end logic, data models, server, and security features.

Throughout the design phase, the principal investigator made design recommendations and revisions with implicit or explicit reference to HFE design principles for older adults (e.g., Fisk, Rogers, Charness, Czaja, & Sharit, 2009; Pak & McLaughlin, 2011).

4.2. Results

The initial concept of Engage had patients monitor and track symptoms, status, and a host of other things, which could be interpreted and acted upon by the patient, clinician, or other outside person who had permission to view logged data. Although sensors were discussed as the basis for data collection and logging, Engage was designed as a manual data entry tool. Besides data acquisition, other primary goals of Engage were to instill and motivate good self-care practices that could last over a longer term and provide an opportunity for patients to gain self-care knowledge by engaging in self-care behavior in parallel with the delivery of informational content. To realize these goals, Engage initially included content in the form of textbook knowledge with follow-up quizzes, as well as goal setting and daily action planning to encourage the assimilation of self-care routines.

In later iterations, Engage was designed to support three core activities (sketched in Figure 2 under “Engage”):

Figure 2.

A slide used during design, combining whiteboard sketches and notes, to depict Engage’s core functionalities and purpose. (Image used with permission from Richard J. Holden, myows.com license #97821.)

• 1)“LOG” or logging information on symptom data (e.g., weight) and self-care behavior (e.g., taking medication);
• 2)“HINT” or reading of practical hints related to heart failure and performing heart failure self-care; and
• 3)“GOAL” or action planning daily behaviors based on longer-term goals.

Engage was designed for short-term use, because of the disengagement identified in the study phase, as well as known challenges of sustaining self-care documentation among older patients with heart failure (Holden et al., 2015c; Miclelson & Holden, 2015; Mickelson et al., 2015). While the logging function was designed to support information work, goal setting and action planning were explicitly intended to improve engagement. The hint function was meant to address self-care knowledge deficits, particularly by suggesting practical ways to relate and apply knowledge. We used the metaphor of a “deck of cards,” with daily deals over the course of a month-long period, comprised of LOG, HINT, and GOAL cards. The finding that self-care was distributed and collaborative, led to using two feedback loops (Figure 3): one corresponding to the patient’s direct use of Engage (reducing over-reliance on clinicians and supporting distribution of work over time), and the other corresponding to clinician intervention based on the observation of data collected, filtered, and communicated via Engage. Accordingly, we engineered the back-end logic and server communication protocols so that prior behavior (e.g., trends in logged data) or outside intervention (e.g., by a clinician) could influence the cards selected or dealt over time.

Figure 3.

Representation of the concept of feedback loops in Engage, taken from a design summary document. Loop 1 represents the patient’s clinician-independent use of Engage and Loop 2 depicts a clinician’s involvement in patient’s self-care based on data patients log in Engage.

Given the identified need for customization, Engage was designed to be flexible with respect to the content of the cards and to accommodate both a starter set of default cards as well as the opportunity to create custom card decks (e.g., per patient, per diagnosis, per clinic, etc.). A motivational incentive was added in the form of virtual coins that are earned with each card played and could optionally be redeemed for tangible rewards.

4.3. Discussion

We followed an iterative, multi-step process of diverging and converging to produce a candidate solution based on principal findings from the study phase. We initially identified four design challenges and four candidate solutions, one of which became the Engage application. Of note, over time, Engage incorporated elements of two of the three remaining solutions. In the end, Engage could be seen as a case of “design as hypothesis” (Woods, 1998): the explicit operationalization of the research team’s assumptions and understandings about collaborative heart failure self-care and older adult interaction with self-care IT. These assumptions were subjected to empirical testing in the evaluation phase, presented next, in which Engage was subjected to using expert heuristic evaluation and laboratory-based usability testing.

5. Evaluation phase

5.1. Methods

Owing to the multiple, rotating HFE and human-computer interaction (HCI) experts involved with Engage design (see section 4.1), evaluation of Engage for usability and compatibility with user abilities and limitations was performed continuously during the design process. In addition, we performed one formal expert (heuristic) evaluation and a series of laboratory-based usability tests. The formal heuristic evaluation was performed during the early prototyping period by one of the team’s HCI experts using the heuristics from Nielsen and Molich (1990). Expert-identified usability issues were then classified as: i) cosmetic and thus do not need to be fixed unless time permits; ii) minor and thus lower priority for fixing; (iii) major and thus should be fixed; or (iv) catastrophic and must be fixed (Nielsen, 1993).

Two phases of laboratory-based usability testing were performed, with a period of redesign in between. As the second phase is ongoing, only findings from the first preliminary usability test, performed January–February 2016, are reported here. Five participants diagnosed with mild or moderate heart failure were recruited from the Indiana University Health urban outpatient heart failure clinic: three males and two females, with a mean age of 61.2 (SD=4.97). Each participant gave informed consent and received a $40 gift card. The usability test was approved by the Indiana University Institutional Review Board.

Usability tests lasted 60 to 90 minutes. First, participants completed a brief interview about their self-care management practices (e.g., do you pay attention to and write down or record your weight?) and experience using technology (e.g., have you used a tablet before?). Next, participants received a brief tutorial on interacting with a mobile tablet device, for example, touching and tapping the display screen. The tutorial also included a high-level description of the Engage application.

Participants then interacted with a high fidelity, dynamic prototype of Engage, administered on a 7-inch Asus tablet. They were first instructed to explore and become familiar with the application. Participants then completed a series of tasks using Engage while thinking aloud, that is, performing concurrent verbalization of their naturally occurring thoughts. Participants performed seven types of tasks: 1) logging into Engage; 2) recording information; 3) setting goals; 4) checking if a goal was met; 5) reviewing long-term goals; 6) skipping self-management activities; and 7) reviewing a disease-related fact. Usability sessions were video-recorded and facilitated by three researchers: one administered the session and the other two made observer notes and ensured the video recording system was functioning. At the end of each session, participants responded to questions on projected long-term use of Engage and perceived usability, measured using questions adapted from the 10-item System Usability Scale (Bangor et al., 2008; Brooke, 1996). To identify usability issues to be corrected in the next design iteration, we applied the coding scheme of Kushniruk and Patel (2004) to the transcripts from the video recorded usability sessions.

5.2. Results

Expert heuristic evaluation on annotated mockups of Engage’s user interface identified 45 usability flaws, 49% of which were cosmetic, 38% minor, 13% major, and 0% catastrophic. Corrections were implemented during subsequent high-fidelity prototyping.

Of the five participants in the usability tests of high-fidelity prototypes, four (80%) owned a computer, three (60%) were mobile phone users, and two (40%) had used a tablet before. Two individuals owned and used smartphones and one other owned and used a tablet device, mostly for solving crossword puzzles.

Table 3 reports the participants’ attention to recording and sharing of elements related to their heart failure. A varied percentage of participants reported paying attention to the different characteristics of their health, with some recording or logging information to be shared with clinicians (Table 4). All participants were able to learn to use the testing tablet and Engage following the brief tutorial and initial exploration.

Table 4.

Participant responses to background questions about health management.

Health characteristic	Pay attention (% of n=5)	Record/log information	Information shared with clinicians at clinic visits (% of n=5)
Weight	100%	Only one individual logged weight in a notebook	100%*
Blood Pressure	80%	Two individuals recorded their blood pressure either using a store-bought blood pressure monitoring system or on paper	100%*
Heart Rate	40%	No individual recorded heart rate	100%*
Diet	100%	Only one individual logged dietary information on paper	40%
Overall health	60%	Two individuals recorded their overall wellbeing either as a mental note or on paper	80%

^*participants reported that these vitals are most often measured and recorded in the patient’s electronic health record during their visit; they are rarely shared outside of the clinical setting.

Table 5 summarizes the results from the analysis of usability test sessions. Overall, participants had difficulty understanding the concept of using Engage on a daily basis. This resulted in not understanding that completing a set of cards is related to routine, self-management actions. The following were frequently identified usability issues, which resulted in errors such as unintended interactions or inadvertently skipping screens:

Table 5.

Code with sample excerpts and participant behavior

Code	Example excerpt and behavior	Usability issue	Implication for re-design
Navigation	(while completing task on viewing Fact card) I would go to facts (taps on Facts menu item)(sees feedback). hmmm…	Hidden menu	Replace the traditional menu comprising of a faceted approach with a linear approach
Graphics	Graphs are always useful…only that I don’t understand what this means (points to the y axis on graph)…	Interpreting graphs	Include actual numbers in the visual graph in addition to appropriate labels for every information
Layout/screen organization	I don’t know how to enter the passcode…where is the keypad? I don’t know where to tap…I have no idea…	Password and login	Make the keyboard visible by default
Color	Well I don’t know what the colors indicate (while setting a goal)	Labels	Provide more information on how the color guide helps when a patient is setting a goal
Resolution	Even without glasses I can see the text very well…
Meaning of labels	This screen is about facts about medicines that I am on and symptoms of the medication. Logs is a reference to the past, if I want facts on my daily weight I will hit this one. I don’t know what status means (points to third card)…	Labels	Rename the labels. For instance, rename the label “FACT” as “See a hint about”
Understanding of system instructions/error messages	If I skipped a card, I would lose a coin there. Man that is pretty harsh..	Similarity of options	Clearly depict the difference between skipping and snoozing a card
Consistency of operations	It says walk here…there is no bike here…I don’t have a clue how to change…	Hidden menu	Make all the physical activity selection items visible
Overall ease of use	There’s weight. June 1st through 6th, so I guess I could look at blood pressure too. See this could be very helpful though once the person got the hang of everything…it’s really nice…
Response time	It’s just going too slow…
Visibility of system status	To me right now it is all confusing, I just don’t understand what is going on	Hidden menu	Display information on a scrollable page with hyperlinks to prevent people from getting lost

• Password and login – problems due to the keypad not being visible by default.

• Menu – problems due to hidden menu items, similar labels in the menu bar and main screen (e.g., goal menu item vs. goal card).

• Similarity of options – confusion of discrete but similarly named actions, e.g., “Ask me later” vs. “Skip.”

• Labels – misunderstanding of the meaning of decontextualized labels, e.g., “Fact.”

• Interpreting graphs – difficulty processing graphed longitudinal data.

The mean overall SUS score was 66.2. All the participants commented that they saw value in Engage following an initial learning period.

5.3. Discussion

Expert feedback on Engage’s usability during the initial stages and subsequent heuristic evaluations helped guide our design decisions as Engage progressed from static images to an interactive prototype. End-user evaluation provided ample additional information for subsequent redesign, identifying usability issues that were not addressed during design despite the team’s HCI and HFE expertise and reference to aging-related principles for design. Although patients acknowledged the potential value of Engage, the overall SUS score of 66.2 is “below average” (Brooke, 1996) and specific usability issues resulted in errors and confusion. The two evaluations promoted several major redesigns, described next. We note that post-test re-designs were largely intended to overcome interaction and usability issues, not to address the original design requirements, which were deemed to be adequately addressed during iterative cycles of design prior to testing.

Simplifying flow (Figure 4).

Figure 4.

Screenshots of Engage that were reviewed by an expert (left), then refined for usability testing (middle). Following usability testing, Engage was redesigned for simpler, more linear flow, by removing the menu bar (right).

Although user control is a principle of usable design (Nielsen, 1993), the multiple, sometimes conflicting options in the original designs were confusing to older adults participants. This was the case even after attempts were made to use clear labeling and separation of the navigation menu from the main screen (Figure 4 left vs. middle). After usability testing, we therefore simplified and linearized the flow of using Engage by removing the tab menu in favor of a separate main menu and decluttering the screens (Figure 4, right).

Increasing visibility (Figure 5).

Figure 5.

Screenshots of Engage that were reviewed by an expert (left), then refined for usability testing (middle). Following usability testing, Engage was redesigned to replace hidden menus with immediately visible content (right).

Several usability problems were associated with hidden menus and accordions with collapsible panels. Some of these were identified during expert evaluation and corrected for the usability test (Figure 5, left vs. middle), but had to be further redesigned following testing (Figure 5, right). In the report section of Engage, accordions were replaced by links and a scrollable page, to avoid users having to uncover hidden content.

Use of familiar metaphors and simplification (Figure 6).

Figure 6.

Screenshots of Engage that were reviewed by an expert (left), then refined for usability testing (middle). Following usability testing, Engage was redesigned to replace the complex data displays with a scrollable summary report with hyperlinks (right).

Subsequent to expert evaluation, we created a fairly complex and data-rich feature with reports of patient data adjacent to their goals and including a combination of long-term values, short-term values, and longitudinal trend data (Figure 6, left vs. middle). This level of complexity turned out to be less rather than more helpful to individuals and increased apparent workload and confusion. The post-test redesign of Engage created a single, consolidated report mimicking the standard scrollable, text-heavy, summary report typical of visit or discharge summaries (Figure 6, right). Although this could create separate usability issues and might not be considered optimal design, we chose this approach for two reasons: 1) the secondary nature of reports in this application; and 2) to avoid critical errors and health implications due to hidden data values. A decision was made to use the familiar summary report metaphor until it could be improved in subsequent versions.

Other changes (color, labeling).

Additional changes were made to improve the color coding of content and action buttons, enhance redundancy whenever color was used, and overall less use of color and increase of the font size of word labels. Furthermore, labels were renamed, including use of sentences rather than single words, renaming of ambiguous words, and provision of additional instructions.

6. General discussion

Self-care for chronic disease is a complex set of processes with multiple goals, embedded in a complex, interacting sociotechnical system, and associated with multiple outcomes. Chronic disease management work is collaborative in that it is performed by a collective, with agents including patients, informal caregivers, and healthcare professionals. In many cases, however, there are multiple breakdowns and barriers, which can be addressed by technologies or other interventions. Human factors frameworks such as SEIPS 2.0, the patient work lens, and user-centered design and associated HFE methods help formalize the understanding of this instance of patient work and develop technologies to support it. As the SEIPS 2.0 and patient work frameworks are macroergonomic (Hendrick & Kleiner, 2002) in nature, their use encouraged the design team to “think systems” and “think big(ger)” (Holden, Rivera, & Carayon, 2015), which in this case means consideration of patients’ long-term goals, overall workload, and integration of self-care recommendations into daily life. These considerations may not have arisen with the use of narrower HFE or HCI frameworks such as GOMS (Card, Newell, & Moran, 1983), which focus on cognitive architecture but not the user’s environment.

A significant finding from our work is the lack of patients’ engagement or motivation to perform self-care behavior over a sustained period of time. Combined with the finding of potentially high levels of workload associated with activities such as information work, this encourages interventions that: overcome the inertia of disengagement; promote regular engagement, even for a short period of time such as 30 days; and scaffold the subsequent routinization or habituation of behaviors. These design goals are different from ones such as “increase adherence” and “increase knowledge,” both of which assume baseline levels of engagement. This notwithstanding, overcoming disengagement alone may not produce adequate outcomes. Design solutions for heart failure self-care among older adults should also address partial mental models and other knowledge deficits, gaps in collaboration between members of the self-care collective, and decision making about responding to symptoms and other changes.

6.1. Methodological discussion

This study used multiple methods, including a variety of data collection approaches during the study phase as well as two evaluation methods. In contrast, recent reviews have noted the tendency for user-centered design research in consumer-facing health IT to use only one or two methods (Zapata, Fernández-Alemán, Idri, & Toval, 2015) and to prefer certain methods (e.g., interviews) over others (e.g., observations) (Hakobyan, Lumsden, & O’Sullivan, 2015). Related to this, a unique contribution of this paper is its presentation of the multiple phases of user-centered design in one publication. There are few other examples of this in the literature on CHIT (Siek et al., 2011), despite calls for such multiphase, iterative approaches (van Gemert-Pijnen et al., 2011). We suspect this is an artifact of publishing conventions and space constraints, as much as if not more than the nature of actual research being performed.

Additional methodological considerations for each study phase are as follows.

1. Study phase: The study phase involved data collection and analysis from a relatively large sample but restricted to one region of the United States and one hospital in Singapore. (Moreover, most analyses were performed on the US sample). It focused on self-care in older adults with a particular condition, chronic heart failure. In addition, although patient and informal caregiver perspectives of self-care were well represented, the study collected minimal data from healthcare professionals. A strength of the study phase was the multiple analyses performed, allowing both confirmation of recurring patterns of findings and exploration of different aspects of self-care. However, the combined analyses ranged over a period of several years and therefore only some could be used during the design phase. Nevertheless, continued analyses provide input for future design cycles.

2. Design phase: The design team by necessity included a restricted set of skills and disciplines, primarily computer science, HFE and HCI. Clinical experts were consulted at various stages leading up to and during design, but neither patients nor healthcare professionals had stable roles on the design team. The design approach was not participatory; the decision to not involve older adults as co-designers of Engage was made largely because of the perceived logistics challenges and lack of experience involving older adults as designers. Subsequent to this, we have begun to experiment with involving older adults in the design process, based on recent work articulating principles and strategies for engaging older adults in design of health IT (Hakobyan et al., 2015; Lindsay, Jackson, Schofield, & Olivier, 2012; Massimi & Baecker, 2006). Lastly, our team followed a “waterfall” approach, performing analysis, design, and evaluation in relatively long and sequential stages, as opposed to the agile approach of many cycles of rapid product development and user feedback. This resulted in delays and some instances of redesigning elements of Engage that should have been tested with end-users prior to the usability evaluation phase. Even when an agile approach is not explicitly used, there are good examples of rapid-cycle testing of CHIT that have avoided the time delays experienced in the present study (e.g., Gustafson Jr. et al., 2016).

3. Evaluation phase: The user evaluation session was conducted in a controlled, laboratory environment, with de-contextualized scenarios and a focus on performing discrete tasks using Engage, rather than prolonged product use in natural or naturalistic settings. Consequently, participants in these tests had difficulty understanding the longitudinal aspects of the application. For this reason, and to increase the sample size for the usability evaluation phase, we are presently performing additional phases of testing with a refined prototype and testing protocol. While we acknowledge the importance of performing agile usability testing in natural settings with older chronically ill adults, we have also identified implementation barriers to performing this kind of rapid translational field research, from miscommunication with health system partners to cancelations and other scheduling discrepancies (Holden, Bodke, et al., 2016). In general, our research demonstrates that there are considerable challenges for performing field research with older adults in community settings (Holden, McDougald Scott, Hoonakker, Hundt, & Carayon, 2015), although several strategies can be used (Valdez & Holden, in press).

7. Conclusion

HFE is critical to the creation of effective, usable, and satisfying interventions for improving health and healthcare. An HFE approach to the design and testing of health information technologies for clinicians and patients is a complex, multiphase effort, as we have described above. More studies should fully detail how HFE is applied across multiple phases of a technology design study in order to demonstrate to other disciplines the process and value of HFE. Doing so will help combat fallacies that HFE merely entails preliminary work, is an add-on applied at the end of product development, or constitutes an art rather than an applied science. Healthcare, in particular, would benefit from convincing illustrations that the HFE approach is scientific, rigorous, data-driven, and therefore highly efficacious.

Biography

Preethi Srinivas is a Senior User Experience Designer at Regenstrief Institute Inc. She received a PhD in Human-Computer Interaction from Indiana University. Her research is at the intersection of HCI, ubiquitous computing, computer supported and co-operative work, with particular interest in technology support for healthcare, health and well-being.

Victor P. Cornet is a PhD candidate in Human Computer Interaction at the Indiana University School of Informatics and Computing. He received a Master’s degree in Computer Engineering from ESIEE Paris in 2014. His research explores the intersection of mental health and opportunities and challenges brought by new technologies.

Richard J. Holden is assistant professor of BioHealth Informatics at the Indiana University School of Informatics and Computing. He received a joint PhD in industrial engineering and psychology from the University of Wisconsin-Madison in 2009. Dr. Holden’s research applies human factors engineering and psychology to study and improve the work performance of patients, informal caregivers, and clinicians.