The Usability of Electronic Personal Health Record Systems for an Underserved Adult Population

Abstract

Objective

The goals of this study were to identify the demands associated with using electronic personal health records (PHRs) and to evaluate the ability of adults of lower socioeconomic status and low health literacy to use PHRs to perform health management activities.

Background

PHRs are proliferating in clinical practices and health care organizations. These systems offer the potential of increasing the active involvement of patients in health self-management. However, little is known about the actual usability of these tools for health consumers.

Method

We used task analysis and health literacy load analysis to identify the cognitive and literacy demands inherent in the use of PHRs and evaluated the usability of three currently available PHR systems with a sample of 54 adults. Participants used the systems to perform tasks related to medication management, interpretation of lab/test results, and health maintenance. Data were also gathered on the participants’ perception of the potential value of using a PHR.

Results

The results indicated that a majority of the participants had difficulty completing the tasks and needed assistance. There was some variability according to task and PHR system. However, most participants perceived the use of PHRs as valuable.

Conclusions

Although considered a valuable tool by consumers, the use of PHR systems may be challenging for many people. Strategies are needed to enhance the usability of these systems, especially for people with low literacy, low health literacy, or limited technology skills.

Application

The data from this study have implications for the design of PHRs.

INTRODUCTION

Chronic diseases, such as heart disease, cancer, and diabetes, account for 70% of deaths in the United States and cause major limitations in daily living for almost 1 out of 10 Americans (Centers for Disease Control and Prevention [CDC], 2013). One fifth of middle-aged adults have a chronic condition (Freid, Bernstein, & Bush, 2012), and the number of people managing chronic conditions is likely to increase given the growth in the older population. Health care costs in the United States have also been rising. In 2011, health care costs in the United States reached $2.7 trillion or $8,680 per person (Centers for Medicare and Medicaid Services [CMS], 2011). Effective management of chronic diseases and disease prevention requires greater consumer participation in health education, prevention, and treatment activities. This participation in turn requires that consumers have access to good, reliable, trusted health information and the ability to comprehend and use this information (Institute of Medicine, 2004).

Online medical information tools, such as patient medical records, have the potential to facilitate the ability of providers and patients to meet these challenges. For example, electronic health records (EHRs) and personal health records (PHRs) are proliferating in clinical practices and health care delivery systems. In fact, deployment of health information technology tools, such as EHRs, is a major government initiative. As part of the American Recovery and Reinvestment Act, the Health Information Technology for Economic and Clinical Health (HITECH) Act appropriated $19.2 billion to encourage the adoption of EHRs, and the associated regulations requires the adoption of EHRs by 2014 for 70% of the primary care provider population (athenahealth, 2009).

On the consumer side, PHRs are Internet-based tools that allow individuals to access, manage, and share their health information (Connecting for Health, 2003). There is a wide variety of PHRs available, ranging from commercially available stand-alone systems to those connected to EHRs. Although the functionality of PHRs varies, they usually allow individuals to communicate electronically with providers, schedule appointments, request renewal of prescriptions, access health management information, and review and track health summary information and test results (Tang, Ash, Bates, Overhage, & Sands, 2006). Some systems also include personal record-keeping features and links to other sources of health information (e.g., Medline Plus).

In theory, PHRs offer several potential benefits to consumers, such as greater access to a wide array of health information, information on care plans, ability to track health status and conditions, and more efficient mechanisms for communicating with providers. Some studies have shown that use of a PHR may be a key component of new models of care and that use of PHRs can result in improvements in health care utilization and/or chronic disease control through better care coordination, access to care, communication, and patient empowerment (Green et al., 2008; Ralston et al., 2009; Simon et al., 2011). In addition, patients who have used PHRs have reported positive effects, such as knowing more about their health care, being able to ask their doctors new questions, and taking steps to improve their health (Undem, 2010). In a focus group study with underserved patient populations, we found that the participants were highly enthusiastic about the potential of being able to access their personal health information online (Zarcadoolas, Vaughon, Czaja, Levy, & Rockoff, 2013).

However, despite the potential benefits of PHRs, widespread adoption of these systems has been relatively slow (Gibbons, 2011; Liu, Shih, & Hayes, 2011; Markle Foundation, 2011). Reported barriers to adoption include concerns about privacy, skepticism about their usefulness, lack of effective promotion by providers, lack of Internet access and computer skills, literacy issues, and usability problems (Hilton et al., 2012; Kim et al., 2009; Liu et al., 2011; Luque et al., 2012; Sarkar et al., 2011; Segall et al., 2011).

Relatively little attention has been paid to the usability issues and human factors engineering challenges associated with PHRs, especially among underserved populations who already experience health disparities. Further, of the studies that have been done, many have been physician oriented (Archer, Fevrier-Thomas, Lokker, McKibbon, & Straus, 2011; Gibbons, 2011). Achieving broad public acceptance and adoption of PHR systems requires attention to consumers’ diverse needs and circumstances, their experiences with health information and technology, and their differing capacities for health management (U.S. Department of Health and Human Services [US DHHS], 2006). This attention is particularly warranted for people with chronic illnesses or disabilities, elderly persons, and underserved populations who are more likely to need to use PHRs. Unless barriers to meaningful access to these systems are remediated, the widespread deployment of PHRs may exacerbate health care disparities for large segments of consumers.

Study Objectives

The objectives of this study were to identify the demands associated with using PHRs and to evaluate the ability of adults of lower socioeconomic (SES) status and low health literacy to use PHRs to perform health management activities. The activities were related to common functions available on PHRs: finding health information, medication management, lab/test results, health maintenance/preventive care, and communication/appointment setting. We paid special attention to literacy demands as health materials, especially web-based information, typically have high literacy demands (e.g., 10th to 12th grade; Zarcadoolas, Pleasant, & Greer, 2006). We also gathered information on users’ perceptions of the potential value of PHRs for health self-management. Our sample included lower-SES adults people as lower-SES populations generally have a higher incidence of chronic diseases and are more likely to have lower health literacy (CDC, 2013).

METHOD

Overview

We used a mixed-methods approach that included task and health literacy load analyses and usability testing. We evaluated three currently available electronic PHR systems: a widely used commercial product, customized by a federally qualified health center (System A); a widely used off-the-shelf commercial product (System B); and a provider-developed system (System C) designed to have an emphasis on patient-centered preventive care. All of the systems are tethered to medical records; however, for this study, given concerns about privacy and security, we populated the systems with fictitious patient data. We used versions of the portals available at the time of the study. We describe the systems and present the results of the task and health literacy load analyses and the usability testing.

General Description of the PHR Systems

System A had broad functionality. The home page included a left sidebar menu with categories related to activities such as medical record, appointments, and billing/insurance. Clicking on a category produced a sublist of activities related to that category, such as tests results, medications, preventive care, and so on. The sidebar also contained a search box for Medline Plus. The center of the page included boxes to select tasks, such as “ask a medical question,” “schedule an appointment,” and “view your health summary.” Thus participants were able to access information for some tasks, such as viewing health summary information, either by using the sidebar menu or by clicking on the Health Summary box. The home page also included links to Medline Plus for information on topics such as diabetes. The font size was adjustable. Navigation through the system involved clicking on links or on specific information. For example, within the initial lab/test result page, the user clicked on a particular row within a table to view specific lab/test results, which were then presented on a subsequent page and included the participant’s value (e.g., cholesterol), the “standard range,” and measurement units of the value (e.g., mg/dL). The participant had to interpret if his or her value was in the standard range. The participant could also click on a link for historical results for a particular test and was taken to another page that showed results in a graphic format for a particular date range.

System B also had broad functionality. The home page included a navigation bar and a left sidebar menu with categories and related subcategories. The categories were messages, account information, intake forms, review, appointment, and requests; and the subcategories were related to tasks such as refill requests, appointments, and diagnostic reports. Patient reminder messages were displayed in the center of the page. The center of the page also contained a link to the participant’s PHR and a section for “Clinical Decision Support Alerts” with the name of the alert, status, and due date (e.g., vaccines). The home page was fairly cluttered and contained a great deal of information, requiring the user to scroll to view all of the information. Navigating through the site involved clicking on links. Results for lab/test results could be viewed in alternate forms, a simplified page that presented information only for a specific category, such as lipid profile, or a more detailed page that also included information on any conditions related to that diagnostic category (e.g., hypertension) and medications.

System C did not include functionality related to medication management or communication/appointments. The home page included a toolbar with tabs to “your summary,” “your health information,” “resources,” and “feedback.” There was also a menu on the left side, “What You Need Now,” that had messages, such as “your last cholesterol was too high,” or general health information, such as “quit smoking,” with links to related information. The page also included a box with information related to status of preventive cancer screenings with links within the box for information related to health behaviors, heart care, and vaccines. There was also a box with clickable links called “Your Key Values” that included basic health indices (e.g., body mass index) with a red or green feedback indicator. There was also a box on the left with clickable links containing information for “Your Key Dates” that listed the names and dates of preventive services (e.g., pap smear). There were also boxes with clickable links at the bottom of the page: “Update Your Health Information,” “Follow Your Conditions,” and “Follow Your Abnormal Results.” Clicking on links to the results of a particular test (cholesterol) brought up a page with summary information for that test, basic information about the test, the benefits of preventive care, recommended next steps, and information to help complete the steps.

The home page was cluttered, included redundant information in different formats, and required the participant to scroll to view the page contents. Participants also had to learn that “clickable links” on some of the pages were related to the color of the link.

Task Analysis

To develop the task problems for the usability testing, we conducted a task analysis (Drury, 1983) of the functions available in each PHR: (a) finding basic health information, (b) medication management, (c) interpreting lab/test results, (d) health maintenance preventive care, and (e) communication/appointment setting. On the basis of these analyses, we developed 17 task problems (Table 1) for the study.

TABLE 1.

Description of the Task Problems

Functions and Task Problems

Finding health information 1AFind the place where you can find out about the medical problems or conditions this patient has. 1BWhat are this patient’s health problems? Medication management 2AFind the medicines this patient is taking. 2BHow should the patient take [selected medication]? 2CCould this patient get a refill of [selected medication] using this patient portal? 2DIf the patient wanted to get more information about the medicine, show me how the patient could get to that information about the medicine. Lab/test results 3AGo to the place in this portal where this patient could find information about his or her lab test results. 3BWhat tests did the patient have? 3CWhat are this patient’s numbers for cholesterol? 3DAccording to the patient portal, are this patient’s numbers normal or abnormal (not normal)? 3EAre these numbers/test results something this patient should be concerned about? Why? 3FShow me where this patient could go to see his or her past test numbers. 3GHow have these numbers changed over time? Health promotion/disease prevention 4AShow me any cancer screening tests that this patient should have. 4BTell me if this patient has had a [selected test] before. 4CDoes this patient need to get a [selected test] in the future? Communication and appointment setting 5AWhere could this patient go in this patient portal to send an e-mail to his or her doctor? 5BWhere could this patient go in this patient portal to make an appointment to see his or her doctor?

System A supported all of the tasks, and System B supported all of the tasks except 2D, 3F, and 3G. System C did not include functionality for Functions 2 and 5. If a function or task was not available on a system, it was not performed during performance testing using that system.

We also identified the user demands inherent in the tasks. Our focus was on the cognitive demands and numeracy, given the importance of these abilities to the use of PHRs (Taha, Czaja, Sharit, & Morrow, 2013). We used four numeracy ability categories: (a) basic—understanding quantitative data with no manipulation of numbers; (b) computational—simple manipulations of numbers; (c) analytical—higher level concepts, such as inference and estimation; and (d) statistical—comparing information presented on different measurement scales and analyzing information related to constructs such as life expectancy (Golbeck, Ahlers-Schmidt, Paschal, & Dismuke, 2005). Tables 2 through 4 present examples of the results to illustrate our approach.

TABLE 2.

Example of a Task Analysis for Finding Medication Information (System A)

Subtasks/Steps	Sensory/Perceptual Demands	Cognitive Demands	Response Demands
Access page that provides access to medication information	Vision and perceptual recognition: Locate the appropriate link on the welcome page	Selective attention, text and health literacy, working and long-term memory: Discriminate among the links and ignore irrelevant information Read and understand the labels of the links Identify/remember the appropriate link to the needed information (e.g., view your health summary) Understand and remember basic mouse and window operations	Fine motor skills/dexterity: Position cursor on link Select link via mouse clicking
Access page with medication information	Vision and perceptual recognition: Locate/identify link on the sidebar	Selective attention, text and health literacy, working and long-term memory: Discriminate among the links and ignore irrelevant information Read and understand the labels on the links Remember how to select link	Fine motor skills/dexterity: Position cursor on link Select link via mouse clicking
Access correct medication information	Vision and perceptual recognition: Locate/identify medication Locate dosage information	Selective attention, text and health literacy, comprehension, working and long-term memory, basic numeracy: Read the labels and discriminate among the types of medications Ignore irrelevant information on the page Understand where the dosage information is located Comprehend the dosage information	Fine motor skills/dexterity: Scroll on the page
Access correct medication information	Vision and perceptual recognition: Locate medication instruction information	Selective attention, text and health literacy, comprehension, working and long-term memory, basic numeracy, computational numeracy: Ignore irrelevant information Discriminate between the dosage and instruction information Discriminate among the various medications Read and understand the medication instructions Compute and use simple number manipulations
Select correct medication	Vision and perceptual recognition: Locate/identify the medication	Selective attention, text and health literacy, comprehension, working and long-term memory: Discriminate among the links Ignore irrelevant information Read and understand the names of the medications	Fine motor skills, manual dexterity: Position cursor on box next to correct medication Select via mouse clicking

TABLE 4.

Example of a Task Analysis: Health Management/Disease Prevention for Breast Cancer (System C)

Subtasks/Steps	Sensory/Perceptual Demands	Cognitive Demands	Demands
Access health information	Vision and perceptual recognition: Locate/recognize “See Your Summary” button on page	Selective attention, comprehension, working memory, long-term memory: Discriminate among buttons on tool bar Read and comprehend labels on buttons on tool bar Remember location of your summary health information Select and click on button Understand and remember basic mouse and window operations	Fine motor skills/dexterity: Cursor positioning Scrolling Clicking
Access information for cancer screenings	Vision and perceptual recognition: Locate/recognize “Cancer Screenings” box on page	Selective attention, text literacy/comprehension, working memory, long-term memory: Discriminate among information on screen and ignore irrelevant information Read and comprehend text on screen Remember and comprehend that box labeled “Cancer Screenings” has needed information Understand and remember basic mouse and window operations	Fine motor skills/dexterity: Cursor positioning Scrolling Clicking
Identify breast cancer screening test requirements information and information related to due dates	Vision and perceptual recognition: Locate the needed information on the page	Selective attention, text literacy, comprehension, basic numeracy, working and long-term memory: Ignore irrelevant information Read and understand the names of the tests Read and understand the concepts of due date, status, and next steps Remember the meaning of color codes Identify time and date of due date Understand and remember basic mouse and window operations, e.g., that information in light blue is clickable
Find and interpret guidelines related to mammograms	Vision and perceptual recognition: Locate the link related to mammogram Locate the information related to guidelines within the page	Selective attention, text literacy, comprehension, computational numeracy, working and long- term memory: Ignore irrelevant information and discriminate among cancer screening links Read and understand the labels of links Read and understand the text related to breast cancer screening and guidelines Understand guidelines on the basis of a woman’s age Remember how to return to summary page Understand and remember basic mouse and window operations	Fine motor skills/dexterity: Cursor positioning Scrolling Clicking

As illustrated, all of the PHRs placed demands on cognitive abilities, such as perceptual recognition skills, selective attention, memory, and processing speed. Completion of the tasks also required health and text literacy; basic, computational, and analytic numeracy; and basic computer, mouse, and window skills.

Literacy Load Analysis

We also conducted a literacy load analysis to examine the demands associated with recognizing and interpreting the content presented in the PHRs. A literacy load analysis involves a structural and semantic analysis. The structural analysis focuses on the vocabulary, length, and complexity of phrases and sentences; the amount of repetition and reinforcement; and the coherence across sentence and paragraphs. The semantic level focuses on identifying concepts such as risk or dose response and if the language should be interpreted literally or metaphorically (e.g., your immune system surrounds the enemy cells; Zarcadoolas, 2010).

Overall, all of the systems had high barrier elements or text and concepts that were likely to be difficult for our participants to read and understand. For example, the use of complex medical terminology was ubiquitous across the systems. Participants had to read and interpret the highly technical names of lab and screening tests, such as “basic metabolic panel” or “sigmoidoscopy,” and health conditions, such as “pure hypercholesterolemia.” They also had to read and interpret strings of complex medical terms used to describe health conditions (e.g., “diabetes mellitus without mention of complication, type II or unspecified type not stated as uncontrolled”).

Many sentences were also complex and assumed knowledge about sophisticated health or mathematical concepts. For example, the following sentences use words and math concepts (data/data range) that are likely difficult for those with low literacy or numeracy: “Adjust the date range to display more or less data. Enter the dates in mm/dd/yy format. Use the buttons below the data to see different views of the data.”

The content also included sentences that were grammatically simple but were difficult to interpret because they assumed health literacy concepts that may be unfamiliar to an average consumer. For example, the vocabulary in the sentence “Balance the calories you eat with the calories you burn” is simple but, without some understanding of what calories are and how they are “burned,” can be difficult to interpret. Participants also had to identify values of lab results (e.g., cholesterol) and determine if a value was in the normal range or a cause for concern. This determination involved both an understanding of sophisticated phrases, like “your value” and “standard range,” and having the numeracy skills to recode the numerical string into propositional statements as well as interpret numerical symbols (e.g., “>”).

Labeling also presented barriers. In many cases, words or phrases used to label a category of information or to indicate a link to information were not commonly used terms or not logically linked to the associated content. For example, in System B, the category label “Clinical Decision Support Alerts” was not directly linked to the subcategories listed within that category, such as the status and due date of vaccines. In addition, in the sidebar, the “Account Information” category had links to “Personal Information” and “Additional Information,” causing potential confusion regarding the information contained within the links. The labels of links to information/activities were also inconsistent. For example, in System A, the home page contained a link for “My Medical Record” and for “View Your Health Summary.” In System C, the home page included a box labeled “Cancer Screening,” which included the status/date information for “colon cancer testing” and a box for “Preventive Services” regarding status/date information for “Colonoscopy.”

USABILITY TESTING

Sample

The usability testing was conducted at three sites designated as medically underserved and as primary care health professional shortage areas by the US HDDS, Health Resources and Service Administration (HRSA): the Mount Sinai Medical Center, East Harlem; the Union Settlement Association, East Harlem; and the Queens Library, Long Island City branch. The designation of medical underservice is based on the Index of Medical Underservice, which is based on four variables: ratio of primary medical care physicians per 1,000 population, infant mortality rate, percentage of the population below the poverty level, and age 65+ years (HRSA, 2014). The sites provided access to adults in different neighborhoods. The Institutional Review Board of Mount Sinai’s School of Medicine approved the study.

Participants were recruited via flyers and word of mouth. Interested participants contacted the study coordinator, who administered a telephone prescreening to ensure eligibility. Inclusion criteria included being a New York City resident, being between 21 and 75 years old, being able to read and speak English, and having a high school education (or equivalent) or below. We chose this educational cutoff to help ensure recruitment of lower-SES participants.

Our sample included 54 adults ranging in age from 22 to 62 years. Twenty-six percent of the sample was under age 30, 42% were ages 30 to 49, and the remaining 32% were ages 50 and older. Table 5 presents additional demographic information. The majority of the participants (94%) had computer/Internet experience, and of those, most (75%) used a computer at least a few times a week and had Internet access at home (63%). The most common reasons for using the Internet included e-mail (41%) or other social media (53%), searching for information related to employment (38%), reading the news (25%), and shopping (23%). Most participants indicated they typically got health information from a variety of sources. Eighty-nine percent indicated that they had gotten health information from the Internet, but only 20% indicated that they regularly used the Internet to get health information, and only 15% always used the Internet for this purpose. All participants provided informed consent and were compensated $50 and received a $4.50 Metro Card for participation.

TABLE 5.

Sample Description (N = 54)

Variable	M (SD)	n (%)
Age (range 22–62)	40.96 (11.65)
New vital signs (0–6)	2.48 (1.23)
Gender
Male		16 (30%)
Female		38 (70%)
Education
Less than high school		26 (48%)
High school		28 (52%)
Race/ethnicity
Hispanic		21 (39%)
Non-Hispanic Black/African		30 (56%)
Non-Hispanic Other		3 (5%)
Annual household income
Less than $20,000		42 (78%)
$20,000 to $50,000		6 (11%)
Don’t know		6 (11%)
Employment status
Work for pay		13 (24%)
Student		6 (11%)
Other		35 (65%)
General health
Fair/Poor		11 (20%)
Good		24 (44%)
Excellent/Very good		19 (35%)
Used the Internet to communicate with provider or view health information		2 (4%)

Materials and Equipment

Background questionnaire

A background questionnaire captured demographic information, self-reported ratings of health, prior computer/Internet experience, and questions related to sources of health information.

Newest Vital Sign

With this screening tool for health literacy (Weiss et al., 2005), individuals are shown a specially designed ice cream nutrition label and are asked six questions related to the label, which can be referred to while answering the questions. A point is given for each correct answer. Scores can range from 0 to 6; a score of 4 or greater generally indicates adequate health literacy.

System ratings questions

Participants were asked to rate the overall difficulty of completing the tasks, navigating through the system, and ease of understanding the language. They responded using a 5-point Likert-type scale (e.g., very difficult to very easy).

PHR rating scale

An eight-item questionnaire assessed perceptions of potential value of using a PHR and the likelihood of using a PHR. Participants were asked if they would use a PHR if available (yes/no), reasons for using a PHR (e.g., make an appointment), reasons for not using (e.g., concerns about privacy), and four questions related to potential value, which were rated on a 5-point Likert-type scale (e.g., agree to disagree).

Procedure

Usability testing occurred on an individual basis with a trained observer. Participants were given a description of the protocol, provided informed consent, and then completed the background questionnaire and the Newest Vital Sign test. They then received basic computer and Internet training and completed a basic practice search problem.

Following this procedure, they were then given a basic overview of PHRs and were informed that they would be assuming the role of a fictitious patient and performing some tasks using the PHRs. To accommodate the study’s low-literacy participants, all tasks instructions and system rating questions were read aloud by the observer. Participants were informed that the material could be repeated or further explained if needed.

To minimize practice/learning effects, the fictitious patient created for each system had a slightly different profile. The order of PHR presentations was counterbalanced across participants. The observer coded participants’ performance of the tasks on an observation form. All computer/Internet training and usability testing was conducted on a Dell Latitude E5510 laptop. The laptop was equipped with Morae usability software tools (Version 3.2.1; TechSmith Corporation), which recorded all screen activity during the session.

After using a system to complete the tasks, participants were asked the system rating questions. After using all three PHRs, they completed the PHR rating scale.

RESULTS

Scoring of the Performance Data

The performance data for each task within each function (e.g., medication management) were coded during data collection using the following scoring scheme: 1 = unable to complete the task after three attempts or gave up; 2 = needed assistance from the observer or completed the task with difficulty; 3 = completed the task without difficulty. Videotapes of the session were reviewed and coded twice. Any discrepancies were resolved with an independent rater. For tasks that required identifying information, such as a name of a medication, the participant’s performance was scored using a rating of 2 = accurate and 1 = inaccurate. A performance score was computed for each function (e.g., lab/test results) for each PHR by summing the scores for each task within that function. In addition, an overall performance score was computed by summing performance scores across function for each PHR.

We computed descriptive statistics for the performance data and, when possible, did comparative analyses of performance across PHRs using the Friedman test. Post hoc tests for significant effects were performed using the Wilcoxon signed ranks test. For the tasks that were scored as accurate or inaccurate, we used the Cochran test and the McNemar test for post hoc analyses. We chose these tests given that our data required nonparametric analyses and there were repeated measures across PHRs. We chose an alpha level of p < .01 given the number of comparisons.

Task Performance

Overall, although there was some variability across the systems, the data indicated that on average, performance was relatively low for many of the functions and tasks (Tables 6 and 7). Examination of performance differences across system for overall functions revealed there were significant differences for finding medical information, χ²(2) = 36.41, p ≤ .001, and interpreting lab/test results, χ²(2) = 30.301, p ≤ .001. In both cases, participants had less difficulty using System A than System B or System C and using System C than System B (all ps < .01). There was also performance variability between systems for medication management, χ²(1) = 10.80, p ≤ .001; on average, performance was better for System A as compared to System B and for health promotion and disease prevention, χ²(2) = 34.07, p ≤ .001. For this task, performance was better using both Systems B and C as compared to System A.

TABLE 6.

Mean Task Performance by Task Function

Task Function	System A	System B	System C	Significance
Finding health information (0–4)	3.72 (0.45)	2.57 (1.06)	2.80 (1.47)	A > B*** A > C***
Medication management (0–4)	3.63 (0.62)	3.19 (0.62)	—	A > B***
Lab/test results (0–7)	6.46 (1.16)	5.87 (1.33)	5.19 (1.83)	A > B** A > C*** B > C**
Health promotion/disease prevention (0–4)	0.83 (1.50)	2.43 (1.80)	2.91 (1.38)	B > A*** C > A***
Communication/appointment setting (0–4)	3.17 (1.10)	2.98 (1.20)	—
Total score (System A, 0–35; System B, 0–23; System C, 0–18)	20.87 (4.14)	16.89 (4.09)	12.33 (3.72)

Note. Standard deviations shown in parentheses.

^**p < .01.

^***p < .001.

TABLE 7.

Performance Data for the Individual Tasks

	n (%)
Task	System A	System B	System C	Overall
Locating and identifying health condition information
Identifying where to find out about medical problems or conditions
Unable to complete/needs assistance	13 (24%)	38 (70%)	26 (48%)	77 (48%)
Locate patient health problem
Unable to complete/needs assistance	2 (4%)	9 (17%)	9 (19%)	20 (13%)
Medication management
Find the medicine
Unable to complete/needs assistance	5 (9%)	16 (30%)	—	21 (20%)
How to take Lisinopril
Inaccurate	3 (6%)	4 (7%)	—	7 (7%)
Get refill of Lisinopril
Inaccurate	10 (19%)	19 (36%)	—	29 (27%)
Able to find more information about the medicine
Unable to complete/needs assistance	26 (50%)	—	—	26 (50%)
Lab/test results
Find lab test results
Unable to complete/needs assistance	5 (9%)	6 (11%)	29 (54%)	40 (25%)
Able to name tests
Unable to complete/needs assistance	3 (6%)	10 (19%)	15 (29%)	28 (18%)
Number for cholesterol
Inaccurate	0	1 (2%)	0	1 (1%)
Determine level of normality
Inaccurate	6 (11%)	17 (32%)	9 (17%)	32 (20%)
Provide reason of concern
Inaccurate	10 (19%)	20 (38%)	19 (35%)	49 (30%)
Go to past test numbers
Unable to complete/needs assistance	27 (50%)	—	38 (73%)	65 (61%)
Numbers change over time
Inaccurate	11 (21%)	—	1 (3%)	12 (14%)
Health promotion/disease prevention
Show cancer screening tests
Unable to complete/needs assistance	45 (85%)	25 (47%)	14 (26%)	84 (53%)
Had a previous cancer screening test
Inaccurate	1 (8%)	3 (9%)	14 (30%)	18 (19%)
Need to get a cancer screening test in the future
Inaccurate	1 (8%)	1 (3%)	7 (15%)	9 (10%)
Communication and appointment setting
Where to go to send e-mail
Unable to complete/needs assistance	19 (35%)	15 (35%)	—	34 (35%)
Where to go to make an appointment
Unable/needs assistance	5 (10%)	5 (10%)	—	10 (10%)

Examination of the performance data for the individual tasks (Table 7) indicated that locating information was difficult for many participants. For example, overall, 47% of the participants were unable to find or had difficulty finding information related to the patient’s medical history. Participants also had difficulty refilling prescriptions and using links to find additional information about medications, such as side effects.

Tasks related to lab/test results were also challenging. Twenty-five percent of the participants were unable to find information related to lab/test results. Of those who found the information, 20% were unable to determine if the lab/test results were in the normal range, 30% did not accurately comprehend if the results were a matter of concern, and more than half (61%) were unable to find information related to past results. Participants also had difficulty with health maintenance/disease prevention tasks, such as finding information related to upcoming cancer screenings.

Ratings of the PHR Systems and Perceived Value of PHRs

Participants’ rating of the difficulty of using the PHRs varied across the systems, χ²(2) =20.96, p < .001. Overall, the participants perceived that it was easier to use System A than Systems B or C and that System B was easier to use than System C (all ps < .01). There was also a difference in ratings of navigational ease, χ²(2) =23.48, p < .001, and difficulty understanding the language, χ²(2) = 22.79, p < .001. Participants found it easier to navigate System A and to understand the information provided by System A.

However, overall, the participants had positive perceptions of PHRs and thought that a PHR would be useful for them and would help them manage health activities (Table 8).

TABLE 8.

Usability Questionnaire

Questionnaire Item	n (%)
A patient portal would be useful for me
Agree	47 (87%)
Somewhat agree	7 (13%)
A patient portal would help me to be healthier
Agree	32 (59%)
Somewhat agree	20 (37%)
A patient portal would make it easier for me to get information about my health
Agree	49 (91%)
Somewhat agree	5 (9%)
I would trust information I found on a patient portal
Agree	38 (70%)
Somewhat agree	13 (24%)
Reasons for using a patient portal (N_= 51)
To make appointments	46 (90%)
To get information about my health problems	46 (90%)
To review my lab/test results	45 (88%)
To manage my medication	44 (86%)
To learn about preventive health screenings	44 (86%)
To communicate with my doctors	42 (82%)
Reasons for not using a patient portal (N = 1)
I have concerns about privacy	1 (100%)

DISCUSSION

Information technology is assuming a central role in our health care systems, and the deployment of PHRs is rapidly increasing within health care settings. Unfortunately, limited attention has been given to human factors issues associated with use of these systems, especially for vulnerable populations. In this study, we evaluated the usability of three currently available PHR systems in a sample of lower-SES, low-literate adults. We also gathered information on user demands inherent in the use of PHRs and user perceptions of the potential value of these systems for the performance of health self-management activities.

Overall, similar to findings from other studies (e.g., Taha et al., 2013; Zarcadoolas et al., 2013), the majority of our participants found potential value in the use of PHRs and felt that use of a PHR would make it easier for them to get information related to their health. However, despite these positive perceptions, our sample had difficulty performing routine health management tasks using any of the three systems. These difficulties were most pronounced for the tasks related to correctly interpreting the labels/links when searching for a category of information, interpreting data for lab/test results, and using health maintenance information, such as cancer screening information.

These performance difficulties were likely due to the design features of the systems. The task analysis and health literacy load analysis indicated that the systems placed high demands on fundamental and health literacy. For example, the use of highly technical terms and complex terminology was ubiquitous across the systems. In addition, menu labels and terminology were often inconsistent within the systems, and menu labels were not always linked to menu content. Further, in many instances, the text used assumes the user has knowledge of medical concepts, which likely created barriers for our participants as they had low health literacy. Clearly, to make these systems usable, it is critically important that consideration is given to literacy demands and that clear and accessible language is used for the content. Recent data (Kirsch, Jungeblut, Jenkins, & Kolstad, 1993; Kutner, Greenberg, Jin, & Paulsen, 2006) indicate that fewer than half of the adult population in the United States have adequate literacy and that approximately 50% of adults in the United States have limited or below-basic health literacy (National Network of Libraries of Medicine, 2013). Low health literacy is especially prevalent among older adults, those with less education, and minorities. In general, people with low health literacy are more likely to use health care services and thus are more likely to need to use a PHR (US DHHS, 2014).

Consistent with the findings of others (e.g., Taha et al., 2013), we also found that use of PHRs place a high demand on numeracy and cognitive skills. For example, the PHRs evaluated in our study required recognition of numeric symbols, such as “>”; interpretation of math concepts, such as standard range, dosage instructions, and measurement units; and the ability to interpret graphic information. These requirements can be problematic for many consumers as large proportions of the adults in the United States have poor numeracy skills (Goodman, Finnegan, Mohadjer, Krenzke, & Hogan, 2013). The systems evaluated in our study also placed demands on cognitive abilities, such as attention and memory. Thus people with lower cognitive abilities, such as many older adults, or those with cognitive impairments are likely to have difficulties using these systems.

There were some differences in performance across the PHR systems. For the most part, the results showed that performance was better for System A for most of the tasks. These findings may have been related to the design features of the systems. The screens in System B and especially System C were rather cluttered and crowded with a variety of different types of information. In many cases, the labels for categories of information and icons were somewhat vague with respect to content, and there was inconsistency in use of referent terms. These results were consistent with the participants’ ratings of usability. Overall, the participants indicated that it was easier to complete the tasks using System A, easier to navigate System A, and easier to understand the information presented in System A. The sidebar menu of System A was very task oriented, which may have allowed the user to find needed information in an easier manner.

However, despite differences among the systems, the overall finding that our participants had difficulties performing the tasks is especially significant given that the majority (94%) of our participants had computer/Internet experience and used computers, the Internet, or mobile devices (smartphones) on a regular basis. One can speculate that use of these systems would be even more challenging for those with limited computer or Internet experience.

Overall, our findings underscore the need to adopt a user-centered design approach when designing PHR systems and to include diverse user groups in the system evaluation. Information on the usability of these systems for diverse user groups is essential to the design and deployment of useful and usable PHR systems. This consideration is essential to ensuring meaningful adoption of these systems and decreasing the potential for health disparities among vulnerable populations, such as older adults or those with low health literacy or education, who are users of health care services.

CONCLUSIONS

Our findings indicate that use of PHRs may be difficult for many health consumers. This difficulty is problematic given the increasing deployment of these systems within health care settings. Clearly, this area would benefit from the application of human factors engineering. Our findings highlight areas for improvement for PHR systems, such as reducing the literacy, numeracy, and cognitive demands of these systems and enhancing the ease of navigation. There are numerous human factors guidelines related to interface and web design that are applicable to the design of PHRs and guidelines related to the use of clear and simple language in health care communications (e.g., CMS, 2011).

It should be noted that the study had some limitations. Our sample was relatively small and restricted to younger and middle-aged adults with low levels of education and low health literacy. Clearly, the usability of PHR systems needs to be evaluated on larger, more diverse user groups. Our participants may also have had a positive response bias when rating the potential value and use of PHRs as they volunteered to participate in a study related to this topic. Further, even though the systems evaluated represented currently available and widely used systems, the task scenarios and information contained in the systems were related to “fictitious patients,” which may have influenced performance. Finally, the usability testing occurred during one session; thus experience using the systems was limited. It should also be noted that PHR systems undergo constant updating and design modifications, and the systems included may have undergone some design changes since the time of this study.

TABLE 3.

Example of the Task Analysis for Interpreting Lab Test Results for Cholesterol (System B)

Subtasks/Steps	Sensory/Perceptual Demands	Cognitive Demands	Response Demands
Access information on lab test results	Vision and perceptual recognition: Locate/recognize “Request your PHR” button on portal home page or Locate/recognize “Lab/Diagnostic Reports” from “Review” subheading on the sidebar	Selective attention, text and health literacy/comprehension, working memory, long-term memory: Discriminate among information on page and ignore irrelevant information Read and comprehend text on page Remember and comprehend that box labeled “Request Your PHR” has lab information Read and comprehend text on side bar Remember/comprehend that “Lab/Diagnostic Reports” contains lab/test result information Understand and remember basic mouse and window operations	Fine motor skills/dexterity: Cursor positioning Scrolling Clicking
Access the correct lab/test results: cholesterol value	Vision and perceptual recognition: Locate/identify cholesterol	Selective attention, text and health literacy/comprehension, working memory, basic numeracy: Read and comprehend labels in results box Discriminate among the types of test values presented Ignore irrelevant information on the page Identify numbers	Fine motor skills/dexterity: Cursor positioning Scrolling Clicking
Determine if value within normal range	Vision and perceptual recognition: Locate/identify the “result (normal range)” information	Selective attention, text and health literacy/comprehension, working memory, long-term memory, basic numeracy, analytical numeracy: Read and comprehend labels Discriminate among the type of value information presented (e.g., attribute vs. result) Compare values (your value vs. your normal range) Understand meaning of mathematical symbols—parentheses Understand and make equivalences— does value fall within higher range

Note. PHR = personal health record.

KEY POINTS.

• Personal health records (PHRs) are rapidly being deployed in health care settings.

• Our study participants, adults of lower socioeconomic status and low health literacy, perceive the value in these systems and would be willing to use them to perform health management tasks.

• Unless attention is paid to the usability of PHRs, many health consumers, particularly, those with low health literacy, are likely to encounter difficulties using these systems to perform health management tasks.

• Usability challenges may also influence user adoption of and willingness to use PHRs.

• Unless PHRs are designed using a user-centered design approach, the deployment of these systems may contribute to health care disparities.

• It is also important to ensure that all health care consumers have the necessary computer and Inter-net skills and access to PHRs.

Biographies

Sara J. Czaja is Leonard M. Miller Professor at the University of Miami Miller School of Medicine. She received a PhD in industrial engineering/human factors from the State University of New York at Buffalo in 1980.

Christina Zarcadoolas is a professor at the CUNY School of Public Health at Hunter College. She received a PhD in sociolinguistics from Brown University in 1982.

Wendy L. Vaughon is a doctoral student at the CUNY School of Public Health at Hunter College. She received an MPH from Mount Sinai School of Medicine in 2009.

Chin Chin Lee is a senior project manager at the University of Miami Miller School of Medicine. She received an MSPH from the University of Miami in 2008.

Maxine L. Rockoff is an adjunct associate research scientist at Columbia University. She received a PhD in mathematics from the University of Pennsylvania in 1964.

Joslyn Levy is a president at Joslyn Levy & Associates, LLC. She received an MPH from Columbia University in 1982.