Online patient websites for electronic health record access among vulnerable populations: portals to nowhere?

Abstract

Objective: With the rapid rise in the adoption of patient portals, many patients are gaining access to their personal health information online for the first time. The objective of this study was to examine specific usability barriers to patient portal engagement among a diverse group of patients and caregivers.

Materials and Methods: We conducted interviews using performance testing and think-aloud methods with 23 patients and 2 caregivers as they first attempted to use features of a newly launched patient portal.

Results: In navigating the portal, participants experienced basic computer barriers (eg, difficulty using a mouse), routine computer barriers (eg, mistyping, navigation issues), reading/writing barriers, and medical content barriers. Compared to participants with adequate health literacy, participants with limited health literacy required 2 additional minutes to complete each task and were more likely to experience each type of navigational barrier. They also experienced more inaccuracies in interpreting a test result and finding a treatment plan within an after-visit summary.

Discussion: When using a patient portal for the first time, participants with limited health literacy completed fewer tasks unassisted, had a higher prevalence of encountering barriers, took longer to complete tasks, and had more problems accurately interpreting medical information.

Conclusion: Our findings suggest a strong need for tailored and accessible training and support to assist all vulnerable patients and/or caregivers with portal registration and use. Measuring the health literacy of a patient population might serve as a strong proxy for identifying patients who need the most support in using health technologies.

BACKGROUND AND SIGNIFICANCE

Coupled with the rising uptake of electronic health record (EHR) technology,^1–3 many health systems worldwide are implementing patient portals,^4,5 websites tethered to EHRs that grant patients and health care proxy agents secure access to summaries of medical visits, test results, and other features.⁶ Potential benefits of portal use include improvements in health knowledge, self-efficacy, health behaviors, patient satisfaction, and communication.⁷ Although evidence is still limited, use of portal features has been linked to improved outcomes for diabetes, hypertension, and depression.^7–9 While other countries are in earlier stages of promoting portal uptake, the financial incentives of the Meaningful Use program¹⁰ as part of US health care reform¹¹ have spurred a rapid uptake of portals across health systems nationwide.

Although there have been early adopters in the field,^12,13 many safety net health systems, systems that provide a significant level of care to low-income and vulnerable patients,¹⁴ are in the midst of implementing patient portals for the first time. Among safety net health systems, patient portals carry great potential to provide additional support to patients disproportionately affected by chronic diseases and the complications of these conditions.¹⁵ Despite this potential, there is well-established evidence among early adopters of portals that racial/ethnic minorities and patients with limited health literacy, income, and education are significantly less likely to use portals.^16–21 Studies of patient portals within safety net settings have found that there are persistent barriers to their use, including perceived threats to security and a lack of technological access and skills.^22–24

In particular, health literacy, the capacity to obtain, process, and understand health information to make health decisions,²⁵ may play an important role in portal use. As the use of health information technology has exploded, the construct of health literacy has expanded to include consumers’ ability to use technology²⁶ to meet the demands of the changing landscape of health communication.²⁷ As health systems implement patient portals, it will be necessary to take steps to better accommodate individuals with limited health literacy.²⁸ Although a few previous studies have explored how health literacy is related to overall portal use,¹⁶ the rates of basic task completion on existing portal websites,^29–32 and the ability to interpret medical information delivered via portals,^33–35 there is a need to understand the mechanisms by which barriers arise to make portals more accessible to vulnerable patient populations.³⁶ By including diverse patients in usability testing of health technologies like portals, we can improve the design of these platforms to improve accessibility and usability for everyone.³⁷

OBJECTIVE

The objectives of this study were to (1) determine the barriers to engaging with a patient portal website by diverse patients and caregivers in a safety net setting, and (2) identify whether limited health literacy manifested in the specific barriers to navigating and interpreting features of the website, and how it did so.

MATERIALS AND METHODS

We employed usability testing methods to provide rich information about how patients and caregivers perceived and interacted with an online patient portal.^38,39 Think-aloud interviews and survey methodology,⁴⁰ as employed in this study, allow researchers to gain deeper insight into a system’s usability not only by listening to participants’ perceptions of using a technology but also by directly observing users as they navigate through the system and identifying where and why they face obstacles.

Research setting

The study was conducted at the Zuckerberg San Francisco General Hospital and Trauma Center, a safety net hospital in the San Francisco Health Network system. This system serves a predominantly Medicaid or uninsured patient population, most of whom are non-white; nearly 40% have been shown to have limited health literacy.⁴¹ From February to July 2015, we recruited 23 chronic disease patients and 2 caregivers with varied experience using a computer and the Internet to manage their health. Participants were recruited from the Richard H. Fine People’s Clinic (RFPC), a primary care clinic serving more than 6500 patients, and nurse-led diabetes support and education group sessions to which RFPC patients were referred. RFPC began exclusively using an EHR in June 2013. The systemwide patient portal, MYSFHEALTH, was launched in January 2015 using eClinicalWorks.

Sampling procedure

We recruited participants through an electronic query of patients with upcoming clinic or diabetes group appointments. Participants were eligible for the study if they (1) spoke English, (2) did not have a diagnosis of cognitive impairment, (3) were diagnosed with ≥1 chronic diseases or were the caregiver of a patient with ≥1 chronic diseases, and (4) had not signed up for, seen, or used the newly launched patient portal. We defined a caregiver as someone other than a patient or medical provider playing a role in the management of a patient’s health. We limited participants to English speakers since the portal under study is only available in English. We focused on patients with chronic disease because portal use can support ongoing self-management. Following provider referrals of known caregivers and patients from an electronic query, research staff approached potential participants during clinic or group appointments, explained the purpose and procedures of the study, and obtained contact information for interested individuals who met the eligibility criteria. Study staff contacted interested individuals at a later date to confirm interest and schedule in-person sessions to obtain written informed consent and complete an interview. The institutional review board of the University of California, San Francisco, approved the study.

Data collection

Survey Administration

During recruitment, we administered a short questionnaire to gather information on demographics (race/ethnicity, age, and gender); chronic disease diagnoses (heart disease, diabetes, high blood pressure, heart failure, asthma/chronic obstructive pulmonary disease, and/or chronic kidney disease); self-reported interest in using the Internet to help manage health care (none, low, neutral, some, high, do not know/need more information); average frequency of current Internet use (daily, weekly, biweekly, monthly or less, or never); and health literacy status.

We administered a modified version of the System Usability Scale to measure participants’ overall ratings of the portal after engaging with its main features (Appendix 2).^42,43 This scale has been widely used to measure the perceived usability of technology interventions, and we sought to determine how well it related to participants’ performance on the observational tasks outlined below.

Think-aloud interview and performance testing process

To gain an in-depth understanding of usability barriers in our system, we conducted 25 observational interviews using performance measures and a think-aloud protocol with 23 patients and 2 caregivers to observe their experiences as they used a patient portal test account. We followed generally accepted principles of usability testing for think-aloud interviews in this study.⁴⁰ The think-aloud process was explained in detail at the beginning of the interview; before system tasks were given, participants completed a practice exercise by “thinking aloud” as they imagined the waiting room of their primary care clinic; and study staff intervened only when participants asked questions, stopped talking, or stopped using the system (Appendix 1, Interview Guide). Interviews were video-recorded using Camtasia,⁴⁴ capturing the computer screen and participants’ faces.

Portal Navigation: To assess participants’ ability to effectively navigate a patient portal, we asked them to complete 5 archetypal tasks on a test patient portal account on a computer: (1) logging in (with a username and password for the test account), (2) viewing a prototypical visit summary (for a diabetes continuity visit), (3) viewing specifically prescribed health education (a page with information about the medication Warfarin), (4) viewing a test result (for a fasting glucose test), and (5) looking up general health information (about blood glucose) in an online health education library linked to the portal. Participants were allowed a maximum of 2 independent attempts to complete each task, which we defined as an effort to complete a task before assistance was given. We gave assistance in the next step of the process if participants asked questions, noted that they were stuck, or gave up.

Health Interpretation: As participants navigated through the portal, we asked them to complete tasks to assess their ability to interpret health information. First, we asked participants to review an after-visit summary, which listed information regarding the reason for the visit, vitals, allergies, diagnoses, medications, notes from the provider with next steps after the visit, immunizations, other medical conditions, and smoking status. Within the after-visit summary, we asked participants to identify where in the summary a treatment plan was located and state what the doctor and patient decided as next steps following the visit as written. Second, we asked participants to review a fasting glucose test result, which included the name of the test, numerical value of the result, reference range, American Diabetes Association classifications for fasting glucose test results (normal, impaired, provisional diagnosis of diabetes), and a written summary of the result. After viewing the test results, we asked participants if they would interpret the result as a good or bad health outcome and if they would ask their provider for further explanation or clarification.

Primary predictor of interest: health literacy

We explored usability outcomes by health literacy status because we hypothesized that health literacy would be strongly linked to task performance. To measure health literacy, we administered a previously validated single-item health literacy item measuring confidence filling out medical forms on one’s own (not at all, a little bit, somewhat, quite a bit, extremely).^45,46 We classified participants who reported any lack of confidence (ie, quite a bit confident or less) as having “limited health literacy” because we felt that the health literacy demands of using an online patient portal were sufficiently complex to warrant this cutoff, and because prior studies have used this cutoff.^16,23 We conducted sensitivity analyses using a secondary cutoff (ie, somewhat confident or less), termed “very limited health literacy” hereafter, to determine whether there was a gradient in our health literacy findings.

Data Analysis

First, the entire analytic team used a deductive coding scheme based on the research questions to code 1 interview. The team iterated and finalized the coding scheme in collaborative group meetings. Then 2 members of the analytic team independently coded the same 2 interviews to finalize the coding scheme and calculate the interrater reliability. Since we determined that the IRR was adequate (0.88), we subsequently divided the remaining interviews between coders to complete the analysis.

Usability outcomes of interest

We examined 3 major categories of usability outcomes. First, we assessed whether participants could complete each navigational task independently, how many attempts they needed to complete it (up to 2), if they needed assistance to complete the tasks or if they gave up, and the time it took to either complete the task or reach the end of a maximum of 2 unsuccessful attempts. Second, we coded the frequency of barriers leading to failed attempts to complete navigational tasks. Each participant could experience multiple types of barriers per task. Finally, we assessed barriers in health interpretation by coding whether participants were able to accurately find and state the treatment plan within the after-visit summary and interpret the fasting glucose test results without assistance. We stratified our results by health literacy status.

Statistical analysis

We described participant characteristics and prevalence of usability outcomes using descriptive statistics. In addition, we performed a sensitivity analysis to test for differences in performance measures across the 3 levels of health literacy. We used the Mann-Whitney rank sum test to test for differences in the mean number of tasks completed without assistance and Fisher’s exact test to test for differences in the proportion of participants experiencing navigation and interpretation barriers.

Finally, to assess the overall ratings of usability of the portal, we calculated participants’ composite System Usability Scale scores (0–100). We used a threshold of below 70 to represent unacceptable usability and above 85 to represent exceptional usability, based on findings from a thorough review of studies that employed this scale.⁴²

RESULTS

Enrollment

A total of 58 individuals were approached about the study. Of those, 14 individuals declined or were ineligible and 44 expressed initial interest in the study. Of the 44 who expressed initial interest, 9 subsequently could not be reached, 2 had already used the patient portal, 1 subsequently declined, 3 were deemed ineligible for the study, and 4 did not show up for scheduled interview appointments. We enrolled 25 participants in the study.

Description of Sample

We enrolled 23 patients and 2 caregivers in the study. The sample was racially and ethnically diverse (76% non-white), and was predominantly African American (36%) and female (68%). Over two-thirds of sample participants expressed some or high interest in using the Internet to help manage their health care or the health care of a patient (68%) and used the Internet at least weekly (72%). A majority of participants had limited health literacy (60%). Among the 15 participants with limited health literacy, a subset of 4 possessed very limited health literacy (Table 1).

Table 1.

Participant characteristics

Variable	Overall, n = 25	Adequate Health Literacy, n = 10	Limited Health Literacy, n = 15	Very Limited Health Literacy, n = 4
Age (mean, SD)	58 (7.2)	61 (4.9)	57 (8.1)	55 (5.3)
Gender (n, %)
Male	8 (32.0)	3 (30.0)	5 (33.3)	0 (0)
Female	17 (68.0)	7 (70.0)	10 (66.7)	4 (100.0)
Race/Ethnicity
Black or African  American	9 (36.0)	4 (40.0)	5 (33.3)	3 (75.0)
White or Caucasian	6 (24.0)	2 (20.0)	4 (26.7)	0 (0)
Hispanic/Latino	2 (8.0)	0 (0)	2 (13.3)	0 (0)
Asian or Pacific  Islander	5 (20.0)	2 (20.0)	3 (20.0)	0 (0)
Other/Mixed	3 (12.0)	2 (20.0)	1 (6.7)	1 (25.0)
Chronic Condition
Diabetes	14 (56.0)	7 (70.0)	7 (46.7)	2 (50.0)
Hypertension	13 (52.0)	4 (40.0)	9 (60.0)	2 (50.0)
Asthma or COPD	6 (24.0)	2 (20.0)	4 (26.7)	1 (25.0)
Heart Disease	6 (24.0)	2 (20.0)	4 (26.7)	1 (25.0)
Heart Failure	4 (16.0)	2 (2.0)	2 (13.3)	0 (0)
Chronic Kidney  Disease	1 (4.0)	0 (0)	1 (6.7)	0 (0)
Health Literacy
Adequate	10 (40.0)	10 (100)	N/A	N/A
Limited	15 (60.0)	N/A	5 (100)	4 (100)
Interest Using Internet for Health Care
High Interest	14 (56.0)	8 (80.0)	6 (40.0)	0 (0)
Some Interest	3 (12.0)	1 (10.0)	2 (13.3)	0 (0)
Neutral	3 (12.0)	1 (10.0)	2 (13.3)	1 (25.0)
No Interest	1 (4.0)	0 (0)	1 (6.7)	0 (0)
Do not know, need  more info	4 (16.0)	0 (0)	4 (26.7)	3 (75.0)
Frequency of Internet Use
Daily	11 (44.0)	7 (70.0)	4 (26.7)	0 (0)
Weekly	7 (28.0)	2 (20.0)	5 (33.3)	1 (25.0)
Every 2–3 Weeks	3 (12.0)	1 (10.0)	2 (13.3)	1 (25.0)
Monthly or Less	1 (4.0)	0 (0)	1 (6.7)	0 (0)
Never	3 (12.0)	0 (0)	3 (20.0)	2 (50.0)

COPD: chronic obstructive pulmonary disease

Although 80% of participants with adequate health literacy expressed high interest in using the Internet to manage their health care, only 40% and 0% with limited and very limited health literacy, respectively, did so. One-fifth of participants (3/15) with limited health literacy and half of participants (2/4) with very limited health literacy had never used the Internet (Table 1).

Portal Usability

Overall Usability

There was a significant burden of barriers to portal navigation and interpretation of health information, regardless of health literacy status. However, participants with limited health literacy were more likely to require assistance (Figure 1), took on average 2 minutes longer to complete individual tasks (specific time values in Table 2), and were more likely to experience navigational barriers (Table 3). On average, participants with limited health literacy could complete few tasks without assistance (Table 3).

Figure 1.

Need for assistance to complete portal navigation tasks.

Table 2.

Average time needed to complete^a portal navigation tasks

Task	Adequate Health Literacy (min:sec)	Limited Health Literacy (min:sec)	Very Limited Health Literacy (min:sec)
Task 1: Login	3:04	6:17	7:53
Task 2: Find Visit Summary	3:29	4:26	4:56
Task 3: Find Provider Education Page	2:06	5:07	3:43
Task 4: Find Lab Results	0:34	1:40	2:02
Task 5: Search Health Education	2:22	4:13	3:42
Total (All Tasks)	11:35	21:43	22:16

^aFor participants who did not successfully complete tasks within a maximum of 2 attempts, we used the time required to reach the end of 2 unsuccessful attempts to complete a task.

Table 3.

Health literacy status and barriers to portal navigation and interpretation of health information

Domain	Variable	Adequate Health Literacy	Limited Health Literacy	Very Limited Health Literacy	P-value
Portal Navigation	Tasks Completed Without Assistance (mean)	4.2	1.27	0.25	<.001
	Basic Computer Barrier, n (%)	1 (10)	11 (73)	4 (100)	.003
	Routine Computer Barrier, n (%)	9 (90)	14 (93)	3 (75)	.546
	Reading/Writing Barrier, n (%)	3 (30)	5 (33)	2 (50)	.725
	Medical Content Barrier, n (%)	1 (10)	4 (26)	0 (0)	.326
Health Interpretation	Difficulty Finding Treatment Plan, n (%)	0 (0)	5 (41)	2 (50)	.067
	Difficulty Interpreting Lab Results, n (%)	1 (10)	6 (40)	3 (75)	.187
	Seeking Explanation From Provider, n (%)	9 (90)	14 (93)	4 (100)	1.000

Barriers to Portal Navigation

Overall, participants with limited health literacy experienced a greater number of barriers to completing all tasks (Figure 2), the majority of which were computer-related. Participants experienced barriers in 4 categories: (1) basic computer (n = 12), (2) routine computer (n = 13), (3) reading and writing (n = 8), and (4) medical content (n = 5). Examples and representative quotes of each type of barrier are presented in Table 4.

Figure 2.

Frequency of barriers to portal navigation. Note. This figure shows the frequency of barriers experienced by participants in each task, stratified by health literacy status. Per task, each participant could experience multiple barriers within each category and barriers from multiple categories. LHL: Limited health literate participants AHL: Adequate health literacy participants

Table 4.

Examples of barriers to portal navigation and representative quotes

Barrier Leading to Task Incompletion	Description	Example	Representative Quote
Basic Computer	Lack of basic computer knowledge or skills, including typing, how to use the Internet, etc.	Participant did not know how to type an exclamation point during login.	“These things right here [special characters on the keyboard]. I don't know how to use those things. Do you have to push it twice?” (Male patient, age 56–60 years, black or African American, with limited health literacy)
		Participant was unfamiliar with how to use a search bar to look up health terms within the health education tools.	“For the search, I did not know that you didn't need to put ‘www.' before the search. All you [do is] put the word in.” (Female patient, age 51–55 years, black or African American, with very limited health literacy)
Routine Computer	Computer barriers experienced by any user with at least basic knowledge of how to use a computer or the Internet, including typos, navigation issues, etc.	Participant became confused when the main website URL was redirected.	“It’s the wrong website. What I am reading in the bar is epp.uc.wcloud.com. It is nothing to do with MySFHealth. It does not concern me, but it confuses me.” (Male patient, age 66–70 years, white or Caucasian, with adequate health literacy)
		Participant had trouble finding the locations of specific features within the portal interface.	“I did not know that visit summary and lab results was under medical records. … It says Dashboard, but then there is a picture of a home there.” (Female patient, age 51–55 years, black or African American, with adequate health literacy)
Reading/Writing	Difficulty reading, understanding, and writing language.	Participant had trouble entering in the portal URL.	“It was easy [to log in] but my spelling is not very good.” (Female patient, age 66–70 years, Asian or Pacific Islander, with adequate health literacy)
Medical Content	Difficulty knowing, recognizing, and applying medical information.	Participant did not know where to go when prompted to find health education on the medication Warfarin.	“Warfarin … what means warfarin? No, I don’t understand. Can you show me please?” (Male patient, age 41–45 years, Asian or Pacific Islander, with limited health literacy)
		Participant had difficulty differentiating HbA1c test and fasting glucose test.	“There’s two of them! January 16 and January 16. Wait a minute …” (Female patient, age 51–55 years, black or African American, with very limited health literacy)

Basic Computer Barriers

Basic computer barriers included obstacles that were clearly related to a lack of previous computer experience, including difficulty typing special characters or capital letters, inexperience using a mouse, and inexperience using search bars or uniform resource locators (URLs). Participants with adequate health literacy were much less likely to face basic computer barriers (10%) than those with limited health literacy (69%). All 4 participants with very limited health literacy faced a basic computer barrier while completing at least 1 task.

Participants experienced the greatest number of basic computer barriers in the initial step of logging in to the portal website, mostly due to difficulty accurately typing in the URL, username, or password.

Routine Computer Barriers

Routine computer barriers were common barriers that could be experienced by any user with at least basic technology experience, or were related to navigation of the portal website rather than lack of experience using technology. These included URL or password mistyping, difficulty finding specific features within the website interface, and confusion due to redirected URLs. For participants with adequate health literacy, routine computer barriers were the predominant or only cause of task incompletion.

Reading and Writing Barriers

Barriers to reading and writing were clear instances of basic literacy challenges. These included difficulty spelling generic search terms while using the health education library and difficulty understanding non-health portal terms. About one-third of participants experienced a barrier to reading and writing, predominantly when asked to use the health education tool to search for information.

Medical Content Barriers

Medical content barriers were specific challenges to understanding medical terminology presented on the portal, including inability to identify the name of a medication, inability to distinguish between different types of laboratory tests, and lack of knowledge of acronyms for health terms. More participants with limited health literacy (25%) experienced medical content barriers than those with adequate health literacy (10%).

Barriers to Interpretation of Health Information

When asked to evaluate or interpret health information, participants with limited or very limited health literacy experienced more difficulties (Table 3).

Identifying a Treatment Plan

While all participants with adequate health literacy were able to find the treatment plan within the after-visit summary, over a third of participants with limited health literacy and half of participants with very limited health literacy experienced difficulty with the task, either noting they were unable to find it or misidentifying the medication list as the treatment plan.

Interpreting Test Results

After viewing the results of a fasting glucose test, 10% of participants with adequate health literacy had difficulty interpreting the results correctly, compared to 40% of participants with limited health literacy. Three of the 4 participants with very limited health literacy (75%) inaccurately interpreted or were unable to interpret the lab result.

“I do not know what this means… . Maybe when I come [to the clinic], they explain. Now I do not know.” (Female patient, age 46–50 years, Asian or Pacific Islander, with limited health literacy.)

The vast majority of participants, including all participants with very limited health literacy, noted that they would seek further explanation or clarification from their provider after viewing the fasting glucose test result.

“My doctor always sits there and explains it to me. … If there is something I need to know, I already ask her.” (Female patient, age 61–65 years, black or African American, with very limited health literacy.)

Even among those who interpreted the laboratory test result accurately, participants noted the importance of discussing with a provider how the results of the test specifically applied to the patient’s health.

“If I wanted to know how this applies to me, [my provider] would be the person to go to. I would say, ‘This is what I saw on the health portal, can you explain these symptoms on my record… how the normal fasting glucose applies to me?’” (Female patient, age 66–70 years, Hispanic or Latino, with limited health literacy.)

Statistical analysis

Across the 3 categories of health literacy, we found significant differences in the mean number of tasks completed without assistance (P < .001) and the proportion of participants experiencing basic computer barriers (P = .003) (Table 3).

Ratings of system usability

Overall, when asked to rate the usability of the portal using the System Usability Scale, participants rated the patient portal favorably (mean composite score of 80). There was little difference between mean composite scores among participants with adequate (81) and limited (79) health literacy. However, among the subset of participants noted as having very limited health literacy, ratings (mean composite score of 63) appeared to reflect a greater burden of barriers to using the portal.

DISCUSSION

Patients are increasingly tasked with comprehending digital communication in the management of their health.^26,27 Within a safety net setting, we identified significant barriers to navigating and interpreting health information presented within an existing patient portal using usability testing. Notably, patients and caregivers with limited health literacy could complete fewer tasks unassisted, had a higher prevalence of encountering barriers, took longer to complete tasks, and had more problems interpreting test results and treatment plans. Our findings are consistent with previous literature showing that patients face numerous challenges in navigating and locating key features of patient portals,^30,32 but ours is one of the first that has used an in-depth think-aloud approach to elucidate the relationship between self-reported health literacy status and usability outcomes across multiple domains.

While past studies have examined the usability of patient portals,^29–34 to our knowledge, only one other study has assessed the usability of personal health records within a safety net setting.³⁵ Our findings build upon the previous literature by classifying and describing the barriers experienced by potential portal users in the safety net. The significant burden of routine computer barriers among our sample may indicate that improvements in portal design and navigation can greatly improve usability. However, particularly for participants with limited health literacy, we observed a significant burden of basic computer barriers in which participants had difficulty using basic features of a computer, demonstrating that many potential portal users lack the basic digital literacy skills to effectively navigate a patient portal.^47–49 Perhaps most importantly, the health literacy status of vulnerable patient populations seems to predict whether individuals would be able to use the portal, even more for basic computer issues than for finding and interpreting medical content on the website. Our findings may also be applicable to other countries with diverse populations as they similarly embark on portal implementation.

Consistent with previous studies, participants in our study experienced difficulties with interpreting the information presented on a patient portal.^30,32–35 This was particularly evident among participants with limited health literacy, who made up the majority of our sample, suggesting that portal users within safety net settings may require assistance or improvements in the clarity of presented information to accurately interpret their health information.³⁵ Even among participants who interpreted test results accurately, the vast majority of those in our sample stated that they would seek further explanation from their health care provider. While in some cases the use of patient portals may facilitate electronic communication or provide information that allows patients to forgo clinic visits, this finding supports the previous literature noting that patients wish to preserve existing in-person communication with their providers and use portals to extend those relationships between visits.^23,50–52

Using the System Usability Scale, we found little difference in the way participants with adequate and limited health literacy rated the usability of the test patient portal account. Using published thresholds for system acceptability,⁴² only patients with very limited health literacy rated the portal website as unacceptable, despite a significant burden of barriers across health literacy levels. Previous studies have also documented consistently high satisfaction with portal use.^29,31,53 Our findings indicate that the System Usability Scale is useful in differentiating larger differences in usability, but it is likely an overestimate of participants’ ability to meaningfully use technology, at least in safety net settings or among individuals without extensive prior computer use. Our findings suggest that patient reports of usability alone are insufficient to evaluate the usability of health technologies, but may best be used as a complement to objective and observed usability testing.

There are several limitations to note. First, our measure of health literacy, although well validated in our patient population, was brief and may not have captured an even greater spectrum of health literacy barriers. Second, our findings may be affected by task ordering effects. Due to the linking of information between tasks (eg, finding an educational page on Warfarin, a medication listed in the visit summary from the previous task), we felt that it would be most logical to keep the order of tasks consistent. Third, because we focused our sample on 1 urban safety net clinic, our findings may not be generalizable to larger health systems or integrated care settings. It may be possible that those who ultimately enrolled in the study were more likely to be interested in computer use, limiting our ability to generalize our findings. To address this, we included a larger size to incorporate diverse patient perspectives, including participants expressing no interest in or having no prior experience using the Internet. Finally, the think-aloud approach may have affected task performance, but we believe that the tasks were simple enough to minimize these effects.

CONCLUSIONS

Patient portals will likely be among the first health technologies to systematically reach diverse patient populations worldwide. Our findings illustrate the need for health systems to employ strategies to provide training not only in health literacy, but in basic digital literacy skills to effectively navigate a portal. Measuring the health literacy of the patient population might serve as a strong proxy for identifying the patients who need the most support and assistance in being able to use health technologies, as literacy levels seem to correlate with both basic computer skills and the ability to effectively interpret and act upon medical information.

FUNDING

This work was supported by the Agency for Healthcare Research and Quality (R00HS022408, R24HS022047, 1K08HS022561) and the National Institute for Diabetes and Digestive and Kidney Diseases (P30-DK092924), National Library of Medicine (G08LM012166, R01LM012355), and National Center for Advancing Translational Sciences (KL2TR000143) at the National Institutes of Health. These funding sources had no role in the preparation, review, or approval of the manuscript.

COMPETING INTERESTS

The authors declare that they have no competing interests.

CONTRIBUTORs

L.T. guided study design, data acquisition, and data analysis and drafted the manuscript. D.S., U.S., N.R., and J.D.R. guided study design and edited and approved the manuscript. M.H. and K.H. performed data analysis and edited and approved the manuscript. C.R.L. conceptualized the study; guided study design, data acquisition, and data analysis; and edited and approved the manuscript. All authors have fulfilled the criteria for authorship established by the International Committee of Medical Journal Editors and approved submission of the manuscript.