Abstract
Telemedicine has the potential to transcend geographic and socio-cultural barriers to the delivery of high quality health care to the medically underserved populations. However, there are significant cognitive and usability barriers. This paper presents a multifaceted cognitive evaluation of the IDEATel diabetes education and telemedicine program. The evaluation included a cognitive walkthrough analysis to characterize task complexity and identify potential problems as well as field usability testing in patients’ homes. The study revealed dimensions of the interface that impeded optimal access to system resources. In addition, we found significant obstacles corresponding to perceptual-motoric skills, mental models of the system, and health literacy. The objective of this work is to contribute to a design framework so that participants with a wide range of skills can better manage their chronic illnesses.
INTRODUCTION
Patient-centered telemedicine research has produced generally positive results in terms of clinical outcomes and patient acceptability, although the research has been limited in scope1. However, there are cognitive and usability barriers to productive use of computer-mediated technology and this problem may be especially acute with regards to seniors. Although seniors are increasingly using computers and the internet, the gap remains relatively wide as compared to other demographic age groups2. The elderly are also more likely to be less affluent and have less education than younger adults, factors which are also associated with the digital divide. Understanding the dimensions of this divide is essential for meaningfully engaging seniors in computer-mediated healthcare initiatives.
An expanded role for patient self-management is increasingly seen as a necessary part of chronic disease care3. Chronic illness affects over 100 million individuals in the United States, and many of them are limited in their daily activities by their condition4. Home telemedicine is a medium with the potential to transcend social, economic and geographic barriers1. With the rapid growth of the Internet and related technologies, telemedicine may serve to bridge significant gulfs of accessibility in the delivery of quality healthcare. In inner urban ci12ties, populated largely by minorities, the obstacles include language, low educational attainment, and lack of social support for health-related activities. Telemedicine affords the possibility of breaking down these barriers to improve access and thereby contribute to reductions in disparities among socio-demographic groups in access to care, quality of care and health outcomes5
There is currently a lack of cognitive human-computer interaction research that addresses the challenges seniors’ confront in learning to negotiate the Internet6. There is however, a growing body of cognitive aging research that can inform design and health-care interventions for older adults6. There are age-related declines in psychomotor skills, especially in dexterity and hand-eye coordination. There is some evidence that these physical limitations can impair individuals from learning to use a keyboard and mouse. Age reduces working memory. Older adults are more affected by distracting context and this limits their ability to selectively attend to relevant screen features and perform concurrent tasks (e.g., work on a computer and hold a conversation) 6.
This paper presents a cognitive evaluation of a large scale telemedicine project targeted at elderly diabetics. The objective of the cognitive evaluation research is to analyze different facets of the telemedicine system as a set of cognitive tasks with a focus on usability and learnability. We are particularly interested in understanding how to facilitate the design of home telecare systems for older patients with chronic health conditions. In addition, a focal point of this research is in characterizing the barriers to productive, efficient, and safe use of these systems towards the goal of sustainable autonomous self-management by patients. The focus on interface features and patients’ competencies represents the cornerstone of a cognitive framework for elucidating and eliminating barriers to home telecare for seniors.
METHOD
IDEATel Telemedicine System
The IDEATel project is a large-scale multi-institution randomized control study designed to assess the efficacy of a comprehensive home-based telemedicine system5. The target population for the IDEATel intervention is Medicare patients living in medically underserved areas in rural Upstate New York and in New York City region of Northern Manhattan and the Bronx. The Upstate population consists mostly of English speakers and. the New York City population is predominantly Hispanic. The focal point of the intervention is the home telemedicine unit (HTU) which provides the following functions: 1) synchronous video-conferencing, 2) electronic transmission of fingerstick glucose and blood pressure readings, 3) email to a physician and nurse case manager, 4) review of one’s clinical data and 5) access to web-based educational materials. The system is designed to be accessible to elderly novice computer users. All components of the system and related services are available in both English and Spanish.
Cognitive Walkthrough
The cognitive walkthrough (CW) is a usability inspection method7 which has been applied to the study the usability and learnability of several distinct medical information technologies8. The purpose of a CW is to evaluate the cognitive processes of users performing a task. The method involves identifying sequences of actions and goals needed to accomplish a given task. The specific aims of the CW procedure are to determine whether the user’s background knowledge and the cues generated by the interface are sufficient to produce the correct goal-action sequence required to perform a task. The method is intended to identify potential usability problems that may impede the successful completion of a task. To perform a CW analysis, an experimenter/analyst performs a task simulation, ‘walking through’ the sequence of actions (both behavioral and cognitive) necessary to achieve a goal. The principal assumption underlying this method is that a given task has a particular generic goal-action structure (i.e., the ways in which a user’s objectives can be translated into the particular action sequences).
The CW analysis also provides us with substantial insight into the cognitive demands of a task. For example, tasks that require the user to execute lengthy sequences of actions or require movement between different screens make heavier demands on a user’s working memory. Similarly, a graphical representation or display that necessitates extensive perceptual processing and rapidly consumes limited attentional resources is not going to be an effective tool. An important consideration in carrying out a walkthrough is an understanding of the target population. In this context, the elderly users of this system are likely to have a lower tolerance for excessive memory or attentional demands. One of the most desirable properties of the walkthrough is that it yields a theory of competent performance.
Usability Testing: Subjects and Procedure
We conducted field usability testing in 25 subjects’ homes, including 14 subjects in the New York City (NYC) area and 11 in Upstate New York. Table 1 summarizes certain patient characteristics. A notable difference is in the years of education. The mean number of years of education was 12.1 for the Upstate subjects and 8.5 for the NYC area participants. In addition, 12 out of 14 NYC subjects were Spanish speaking, whereas all of the Upstate patients were English speaking. We selected both subjects who had been using the system to access the web with some regularity and those who had not.
Table 1.
Subject Characteristics (means ± SD or %)
| New York City N=14 | Upstate N=11 | |
|---|---|---|
| Age (years) | 69.6 (6) | 73.7 (8) |
| Education | 8.5 (5) | 12.1 (3) |
| Level (years) | Range: 0–14 | Range: 7–16 |
| Living with Diabetes (years) | 10.4 (10) | 12.1 (9) |
| Language | Spanish 86% | English 100% |
| Web site use* | 36% | 45% |
Use of IDEATel Web Site Prior to Usability Study
Subjects were asked a series of questions about their use of computers and experience with diabetes. We then asked them to perform a series of tasks including: measuring blood pressure, uploading results, accessing the Diabetes Manager (Siemens Medical Solutions, Malvern, PA) web page, reviewing patient data, and generating and interpreting a table of blood glucose results. Subjects were asked to think aloud throughout the task and offer comments on each screen (and screen transition). Whenever the subjects experienced difficulty, the experimenter provided as much guidance as necessary. The entire procedure took about 60 minutes on average. All sessions were audio and video taped. In addition, screens were captured to a digital camera via a VGA to TV Scan Converter.
RESULTS AND DISCUSSION
Cognitive Walkthrough Evaluation
The complete IDEATel program enables 15 distinct superordinate tasks. We can define a task functionally as that which orients a user towards achieving an objective related to diabetes health care. Some of the integrated tasks support multiple constituent tasks. For example, tracking in the diabetes diary enables the patient to view both their blood pressure and glucose. In addition, a patient can view his or her record over the course of a day over a variable period of time such as a week or a month.
The glucose and blood pressure monitoring tasks employed tightly coupled goal action sequences and were reasonably easy to execute. A tightly coupled sequence is one in which an action transparently flows from a goal and the user can readily perceive that the system has responded thereby signaling the next subgoal and action sequence. A partial walkthrough of the glucose task is illustrated below.
Task/Goal: Measure Blood Glucose Level
1. Subgoal: Begin Measurement
Action: Press Blue Power Button
System Response: Meter Displays Last Blood Glucose Result
2. Subgoal: Obtain a Blood Sample
Subgoal: Use Sterile/Sharp Lancet
Action: Replace Lancet (if necessary)
Subgoal: Draw Blood using Instrument
Action: Pierce Finger with Lancet
3. Subgoal: Apply blood to test strip
Subgoal: Locate Pink Test Area
Action: Dab Finger/Touch Strip
Potential Problem: Missing Test Area Applying Excessive Blood
4. Subgoal: Take Measurement with Device
Subgoal: Determine readiness of the device
Action: Look for flashing test strip on meter
System Response: Code 4 and Flashing test strip
In total, the walkthrough of the glucose monitoring device necessitated 5 subtasks, 9 subgoal action pairings, 12 actions, and 5 device/screen transitions. Familiarity with the device components and related objects (e.g., meters, lancets and test strips) very likely facilitate the relative ease in which patients execute the task. This is in contrast to some of the web-based tasks. Accessing the web necessitates 9 actions and 8 screen transitions. The transitions include a series of displays with connecting messages and security related screens. For the most part, the screens (e.g., security screens) are not meaningful to the participants and may encourage passive responding. The transitions are likely to be a source of considerable confusion to beginner users and in fact usability testing appears to bear this out.
The Diabetes Manager screen represents the command center of the IDEATel system. Most of the tasks supported by the system can be initiated from this page. The screen is somewhat cluttered and tasks are not well segregated. In addition, there are labeled links such as glucose and blood pressure that appear twice but reference somewhat different functions. There is also an abundance of text and some of it appears in small lettering which proved to be difficult for some of the participants to read. The goal-action sequences and affordances across several of the subtasks are relatively consistent (e.g., using dialogue boxes for retrieving values) which supports and reinforces the learning process. However, this is somewhat offset by the immense complexity of the screen.
Many of the patients are rather elderly and vision and dexterity are significant issues. In addition, many of the widgets (e.g., scroll bars) present unique problems for this population of users. The problem appears to be both of one familiarity and the necessity of fine eye-hand coordination. The following task involving tracking blood pressure illustrates some of these problems. The tracking BP application enables an individual to perform arrange of tasks in view to monitor one’s values over a certain period of time. The following partial CW analysis pertains to changing the dates to view one’s values.
Subgoal: Select Date
Subgoal: Change Month in “From” Field
Action: Click on diamond on Pull Down Menu
System Response: Pull Down Menu Unfolds
Action: Scroll Down to Correct Month
System Response: Selected Month is Highlighted
The calendar widget is common to a range of applications and it is relatively straightforward from a cognitive vantage point necessitating the repetition of the aforementioned goal-action sequence applied to each of the date fields. However, the narrow scroll bars, especially when the date was not visible on the menu, necessitating additional scrolling. Elderly users experienced considerable difficulty with many of the widgets that necessitated fine eye-hand coordination.
Usability Testing
The cognitive walkthrough highlighted areas of complexity in which the demands of the system were likely to present challenges for this target population and to a large extent this was supported by the usability testing. However, the field testing also revealed a range of both cognitive and noncognitive barriers that were unanticipated by the CW analysis. The focus in this presentation is on the new users and the challenges they confronted. These subjects routinely used the glucose and blood pressure monitoring devices without difficulty and were rather satisfied with the video-visit component. However, we observed a range of problems in their use of the web components. Many of these problems can be subsumed by three categories: 1) perceptual-motoric skills, especially in relation to the use of the mouse, 2) mental models which refer to a basic understanding of system navigation, and 3) health literacy, including basic literacy and numeracy.
Fifteen of the participants had minimal or no computer experience and mastery of the mouse turned out to be challenging and even formidable for some of the subjects. There are a range of skills involved in mouse mastery including: proper grip, fluid movement, coordinated movement and simultaneous click, cursor and pointer recognition, precision of clicking on links, and latency/sensitivity of click. Two subjects showed substantial improvement over the course of the session, 10 of the subjects were able to develop minimal proficiency and three subjects evidenced little to no improvement in developing mouse skills.
Mental models are dynamically constructed knowledge structures that enable one to both explain the occurrence of events and anticipate changes. In this context, it refers to a basic knowledge of how the system works. A basic understanding is necessary to negotiate screen transitions, plan extended-goal action sequences (e.g., necessary to complete a task), and engage in elementary “troubleshooting” to explain why an intended action never materialized. Web browsers can provide a “direct manipulation” style of interaction and can potentially enable novice users to learn the basic functionality of systems reasonably quickly. However, all of the novice subjects experienced some difficulty in developing a coherent mental model of the system and were frustrated by their inability to navigate screen transitions and construct goal-action mappings. This may be partly due to the complexity of the system. For example better transitions between screens and more salient perceptual cues would foster development of system models. It is also likely that seniors need more time to develop basic mastery. Most of the more experienced users exhibited rather robust mental models of the systems and demonstrating considerable competency in using the system. In general, the lack of a coherent mental model was a less formidable obstacle than mouse skill. We can anticipate that subjects will improve with increased experience using the system.
Health literacy emerged as a serious problem for 3 of the subjects and was a source of concern for several other participants. The subjects need to have a basic ability to comprehend relevant materials to productively participate in a self-management regimen. This involves monitoring ones glucose and blood pressure values and making the necessary adjustments in one’s lifestyle (e.g., diet and exercise). One of the most striking findings corresponded to problems with numeracy and representational fluency (e.g., ability to read a table or understand a chart). The core skills involve an abstract understanding of covariation and how it can be expressed as a functional relationship (i.e., cells and rows in a table), establish correspondence between monitoring device (e.g., blood pressure) and presentation in tabular format (systolic/diastolic) as illustrated in Figure 2.
Figure 2.
Mapping Values between Monitor and Table
Three subjects were unable to interpret a table. One subject, who had only two years of schooling, experienced considerable difficulty making sense of the information on the screen. Although she was functionally literate and had no difficulty reading the materials, she had a relatively low level of numeracy and could not make sense of a table of blood pressure values. The numbers and cells of the table appeared to her as discrete values rather than as a system of columns and rows that represented BP values recorded on a given date. In addition, there was no sense that the values corresponded to the measures that she recorded on a daily basis on the BP monitoring device (see Figure 2). Clearly, she was lacking basic skills necessary to monitor her readings in this format and understand the events that may have precipitated them. Two other subjects experienced similar difficulties. Most other participants were more literate, but several had problems recognizing abnormal results (especially BP), discerning patterns over bounded periods of time (e.g., comparing values over a week) and drawing appropriate inferences that could lead to appropriate modifications of lifestyle.
CONCLUSIONS
This research identified barriers to effective home telemedicine use in this population that were in part related to dimensions of system complexity and in part related to essential competencies for self-management of a chronic illness such as diabetes. Much of society has embraced the web and the presumably intuitive graphical user interfaces and direct manipulation interaction style. Nevertheless, the findings suggest that seniors who are new to computing are less likely to perceive system affordances in quite the same way. In addition, health literacy may be a surprisingly important issue pertaining to the use of home telecare technology in relation to chronic care management. Diabetic patients with inadequate health literacy are more likely to have poor glycemic control and associated complications such as retinopathy9. Although telemedicine affords such individuals an opportunity to increase their health literacy, a minimum level of proficiency is necessary before users can effectively use the system resources to manage their illness.
The findings of this research have contributed to an iterative design process, modification of a patient tutorial and a training program. The ultimate objective of this work is to develop a comprehensive design and evaluation framework for enabling seniors to more effectively participate in Internet-mediated health care initiatives. One of the primary reasons that seniors use the Internet is to seek health information2 and this may motivate the participants in programs such as IDEATel. However, there exist significant cognitive, perceptual-motoric, scientific literacy and innumeracy barriers that preclude some patients from fully exploiting the benefits of web-based telemedicine. An in-depth understanding of these barriers is a prerequisite for tapping into the vast potential of such innovative interventions
Figure 1.
Tracking Blood Pressure Interface
Acknowledgments
This research was supported by Cooperative Agreement Centers for Medicare and Medicaid Services (95-C-90998). We are grateful to all the participants who graciously gave of their time to participate in the study. In addition to the authors, the IDEATel Research Consortium includes the following: Columbia University and New York Presbyterian Hospital (NYC): Walter Palmas, MD, Lesley Field, RN, MSN, Rafael Lantigua, MD, George Hripcsak, MD, Soumitra Sengupta, PhD, Edward Shortliffe, MD, PhD, Mohammed Ashraf, Sean Mota, Catherine Tuck, MD*, Patricia Kringas, RN, Jessica Rivera, RN, Rene Bachmann, RN, CDN, CDE; Harlem Hospital Center (NYC): Linnea Capps, MD; St. Luke’s-Roosevelt Hospital Center: Alejandro Prigollini, MD; Hebrew Home for the Aged at Riverdale (Bronx, NY): Douglas Holmes, PhD, Jeanne Teresi, EdD, PhD, Stephanie Silver, MPH; Jian Kong, MS, Eva Petkova, PhD, Lucja Orzechowska, MA, EdM, Joseph P. Eimicke, BS; SUNY Upstate Medical University at Syracuse (Syracuse, NY): Roberto Izquierdo, MD, Paul E. Knudson, MD, L. Thomas Wolff, MD, Suzanne Meyer, RN, CDE, Suzan M. Carusone, RN, CCRC, Fraser G. Tudiver, MD**, Teresa J. Wagner, MS, Carina Lagua, MA, RD, CDE, Susan Fox, RN, Susan West, RN, MSN, CDE, JoAnn Shupe, AOS, Donald R. Roller, BS, Joseph Ziemba, MBA, David P. Thomson, MD, Michele Malone, AOS, Kelly Duncan BSW, CCRC, Wendy VanRiper, Sarah Doolittle, BS, CCRC, Joanne Kearns, MA, RD, CDE, Deborah Bowerman, RN, CDE, Phyllis T. Bazen, MSN, ACNP-C, Pamela Suddaby, RN, BS, Kristine M. Brindak, RN, Fronda Benson, LPN; A. L. Lee Memorial Hospital, Inc. (Fulton, NY): Shirley Seabury, MHA; Arnot-Ogden Medical Center (Elmira, NY): John F. Carroll, MD, MBA, Frederic P. Skinner, MD, PhD, Terri L. Ennis, RN; Bassett Healthcare (Cooperstown, NY): Kathleen Sellers, RN, PhD, Barbara Leo, BA, BS, RN, Elsie Haus, MEd, FNP, Catherine Cooney, BS, MA, RN; Delaware Valley Hospital (Walton, NY): Michael J. Freeman, DO, MPH; Family Health Network of Central New York, Inc. (Cortland, NY): Susan Ulrich, MS, RN, CPNP; Guthrie Healthcare System (Sayre, PA): Brian D. Cassetta, MD; Hudson Headwaters Health Network (Glens Falls, NY): Patricia Auer, MSN, FNP, W. Emily Hammond, AAS, RN, Cairenn L. Young, MSW, Mary M. Beadnell, LPN; Lourdes Hospital (Binghamton, NY): Timothy C. Howland, MD; Olean General Hospital (Olean, NY): Robert Catalano, MD, Galvin Anderson, MD, Terri N. Kolasinski, RN; Samaritan Medical Center (Watertown, NY): Anil Shah, MD, Kate R. Fraczek, RN, Terry Flynt, BSN, RN; State University of New York at Buffalo (Buffalo, NY): John Taylor, MBA, Lurene R. Tumiel, PhD, Marc Shilling, MA, MPA, Chester Fox, MD, United Health Services (Binghamton, NY): David Kwiatkowski, MD; American Telecare, Inc. (Eden Prairie, MN): C.R. Abbruscato, MSEE, Karen Boril. *Deceased. **Current address: E. Tennessee State University (Johnson City, TN).
REFERENCES
- 1.Bashshur R, Reardon T, Shannon G. Telemedicine: A new health care delivery system. Annu Rev Public Health. 2000;21:613–637. doi: 10.1146/annurev.publhealth.21.1.613. [DOI] [PubMed] [Google Scholar]
- 2.U. S. Department of Commerce. (2002). A Nation Online: How Americans Are Expanding Their Use of the Internet. Washington, DC
- 3.Wagner E, Groves T. Care for chronic diseases. BMJ. 2002;2002;325:913–914. doi: 10.1136/bmj.325.7370.913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.National Academy on an Aging Society Report, 1999. Chronic Conditions: A Challenge for the 21st Century.
- 5.Shea S, Starren J, Weinstock R, et al. Columbia University’s Informatics for Diabetes Education and Telemedicine (IDEATel) Project: Rationale and Design. JAMIA. 2002;9(1):49–60. doi: 10.1136/jamia.2002.0090049. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hawthorn D. Possible implications of aging for interface designers. Interacting with Computers. 2002;12(5):507–528. [Google Scholar]
- 7.Polson P, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies. 1992;36:741–773. [Google Scholar]
- 8.Kushniruk A, Kaufman D, Patel V, et al. Assessment of a computerized patient record system: A cognitive approach to evaluating an emerging medical technology. MD Computing. 1996;13:406–415. [PubMed] [Google Scholar]
- 9.Schillinger D, Grumbach K, Piette J, et al. Association of health literacy with diabetes outcomes. JAMA. 2002;288:475–82. doi: 10.1001/jama.288.4.475. [DOI] [PubMed] [Google Scholar]