A randomized trial to train vulnerable primary care patients to use a patient portal

Abstract

Background

Patient portals are becoming ubiquitous. Previous research has documented substantial barriers, especially among vulnerable patient subgroups such as those with lower SES or limited health literacy (LHL). We tested the effectiveness of delivering online, video-based portal training to patients in a safety net setting.

Methods

We created an online video curriculum about accessing the San Francisco Health Network portal, and then randomized 93 English-speaking patients with 1+ chronic diseases to receive: 1) an in-person tutorial with a research assistant or 2) link to view the videos on their own. We also examined a third, non-randomized usual care comparison group. The primary outcome was portal log-in (yes/no) 3–6 months post-training, assessed via the EHR. Secondary outcomes were self-reported attitudes and skills collected via baseline and follow-up surveys.

Results

Mean age was 54, 51% had LHL, 60% were non-white, 52% were female, 45% reported fair/poor health, and 76% reported daily Internet use. At follow-up, 21% logged into the portal, with no differences by arm (p=0.41), but this was higher than the overall clinic rate of 9% (p<0.01) during the same time period. We found significant pre-post improvements in self-rated portal skills (p=0.03) and eHealth literacy (p<0.01). Those with LHL were less likely to log in post-training (p<0.01).

Conclusions

Both modalities of online training were comparable, and neither mode enabled a majority of vulnerable patients to use portals, especially those with LHL. This suggests that portal training will need to be more intensive or portals need improved usability to meaningfully increase use among diverse patients.

Introduction

Online patient portal use has expanded exponentially because of financial incentives of the Meaningful Use program.¹ These websites allow patients online access to medical records through features such as lab results and secure messaging² – representing patient-centered efforts to improve efficiency and convenience of care. Some studies suggest portal websites may help improve patient outcomes,^3–5 such as improving processes that ultimately affect health behaviors (e.g., simpler refill processes leading to better medication adherence).^6–8

However, one of the biggest challenges for widespread portal use has been the lack of uptake among diverse patient populations.⁹ Across the overwhelming majority of studies, patients from racial/ethnic backgrounds and with limited health literacy have been significantly less likely to be portal users,^10–15 even when accounting for Internet use in everyday life. Furthermore, usability studies of existing portal interfaces have shown considerable barriers, especially among those with limited health literacy.^16–19

Therefore, there remains a large implementation gap in research and clinical practice to engage a broader set of patients in portal use. This goal of increased portal use among diverse patients is relevant for both improving access to care as well as potentially mitigating future healthcare disparities as systems continue to deliver more care electronically.²⁰ Because previous research specifically identified training and additional support as a primary means to reduce barriers to use,^21,22 we designed and evaluated an online portal training program for patients in a safety net healthcare setting. In a randomized trial, we evaluated the impact of implementing different modes of training on subsequent portal use rates.

Methods

Study Setting

The San Francisco Health Network provides primary care to over 63,000 patients/year as the city’s only public healthcare delivery system. The majority of patients are non-White and on Medicaid/uninsured.²³ Launched in January 2015, the online patient portal (called MYSFHEALTH) allows patients to access their visit summaries, lab results, and health education materials online. To register, patients initiate sign-up in-person at their clinic and receive an email to activate their account on their own. At the time of the study, 35% of primary care patients had initiated sign-up for MYSFHEALTH.

MYSFHEALTH Training Content

Informed by documented usability barriers to portal use in our setting,¹⁷ we worked in close consultation with the library at Zuckerberg San Francisco General Hospital (ZSFG) as well as with 2 advisory boards to iterate both the content and format of our training curriculum. Our patient advisory board was comprised of 10 members who were primary care patients or caregivers, and our project advisory board included 9 experts from health technology research, adult learning and literacy, and clinical administration/operations. Over the course of 4 in-person meetings, we created a final portal training curriculum with simple instructions and 11 how-to videos for accessing MYSFHEALTH (patient story, getting started, signing up, signing in, creating a username, creating a password, accessing the homepage, accessing a visit summary, reviewing lab results, using the online health library, and message from a healthcare provider – all videos available upon request). Participants could watch all videos sequentially, or skip to relevant topics as preferred. Our iterations simplified the content to better match the digital and health literacy levels of the population. In particular, these videos used audio (with captions) for explaining the portal functionality as well as screenshots of how to access each feature. When completed, we put the training content onto an online learning platform, LearnerWeb,²⁴ which allowed us to track individual access of the videos during the study. At the conclusion of the study, we moved all video content to the Network’s website for public access.

Trial Recruitment

From June-October 2016, we recruited participants from 2 primary care clinics, one based on the campus of ZSFG and one community-based clinic. Through an electronic query of the electronic health record (EHR), we generated a list of patients who had clinic visits prior to the study (April-July 2016) meeting the following criteria: (1) English-speaking (as the portal was only available in English in our setting), (2) age 18 or older, and (3) diagnosed with a chronic condition (as previous studies have shown that portal use is highest among those managing a chronic illness²⁵). Providers reviewed the lists and excluded individuals with cognitive or visual impairment, severe mental health conditions, or other barriers to enrollment. Through phone screening, we further excluded individuals without email addresses (as this was necessary for portal registration) and those who already self-reported using MYSFHEALTH.

Randomization

We randomized patients during an in-person session to receive: a) an in-person tutorial with a trained research assistant versus b) a link to access the online tutorial on their own. During this in-person enrollment session, all participants received an informational pamphlet that was disseminated within the general clinic population outlining key features of the patient portal, and were guided through the steps of signing up for a LearnerWeb account to access the training materials. For participants randomized to the in-person training arm, a trained research assistant prompted participants to log into the learning platform and guided them in accessing the training materials. The staff member provided further explanation or clarification if participants had questions about the training material. Participants in the take-home arm were given a paper handout with a link to the training materials and an outline of the steps for accessing the training curriculum. The research assistants delivering the intervention were blinded to the randomization allocation until after the consent process was complete.

The trial is registered at clinicaltrials.gov as .

Access of Training Curriculum

Using LearnerWeb, we measured whether participants accessed the training videos, categorized as yes/no and number of videos watched (from 0–11).

Portal Sign-up and Use

Portal outcomes were assessed via EHR chart review as 1) initiating sign-up for portal access (yes/no), and 2) logging into the portal after the sign-up process was complete (yes/no and total number of log-ins). Our primary outcome was the binary assessment of portal log-on in the follow-up period of 3–6 months post-training (mean 112 days, range 82–192 days), with an estimated sample size of 100 to detect detect a 25% difference in portal use (10% vs. 35%) with 80% power. Because portal enrollment is tied to an in-person sign-up process, we examined portal use data after the participant’s next visit whenever possible, using 3–6 months as a timeframe in which most patients with chronic diseases were generally scheduled for a follow-up appointment.

Clinic Comparison Group

To obtain comparison outcomes in a third usual care cohort, we performed an additional, non-randomized EHR data pull of all patients who had visited the 2 primary care clinics during the recruitment time period (April-July 2016) and followed them through the same follow-up period (July-December 2016). We used the same portal use outcome ascertainment in this group.

Survey Measures

We also collected patient survey measures at baseline (in-person) and follow-up (via phone). Participants could receive in-person support from the research assistant at any time to complete their written baseline survey. Demographics included age, gender, race/ethnicity, and highest education completed. We assessed health literacy status using a screener about confidence in filling out medical forms independently.²⁶ We categorized participants noting any lack of confidence filling out medical forms as having limited health literacy based on previous work.^10,11,17,27 We asked participants how well they spoke English and categorized less than “very well” as limited English proficiency. Participants also reported how often they used the Internet/email (daily, weekly, or less), as well as their specific chronic condition(s).

To assess portal interest/attitudes and skills, we also asked a series of self-reported items at both baseline and follow up. Interest and attitude items included 1) usefulness of Internet for making health decisions (5-point Likert from not at all to very useful), 2) importance of getting medical information electronically (3-point Likert from not at all to very important)²⁸, 3) confidence in safeguards for online medical records (3-point Likert from not at all to very confident),²⁸ 4) interest in using the MYSFHEALTH portal website to see their medical record (5-point Likert from no interest to high interest), and 5) interest in using specific potential portal features (4-point Likerts from not at all to very interested²⁸). Next, we assessed their self-reported skills by asking about: 1) self-rated skills to use a website to manage healthcare (5-point Likert strongly disagree to strongly agree), 2) confidence in logging into MYSFHEALTH without help (scored 1 to 10), 3) confidence in using MYSFHEALTH to improve their health (scored 1 to 10), and 4) self-reported eHealth literacy using 4 items of a validated scale (such as “I know how to use the Internet to answer my health questions”²⁹).

Next, we assessed patient perceptions of their healthcare that we hypothesized could be impacted by portal use, either because of increased convenience or access to specific healthcare information. This included the Patient Assessment of Chronic Illness Care³⁰ and a modified version of the Self-Efficacy for Managing Chronic Disease Scale³¹ (adjusted from 11^th to 4^th grade Flesch-Kincaid readability level).

The follow-up surveys also included an open-ended question about the reason(s) for portal non-use among those without any log-ins documented in the EHR.

Participants received $25 gift cards to a major retailer for completing the baseline and follow-up portions of the study.

Analysis

We summarized participant baseline characteristics by trial arm. Next, we examined the portal use outcomes using chi-squared tests by study arm. We completed intent-to-treat analyses among all participants, as well as a per-protocol examination comparing participants exposed vs. not exposed to the online training materials.

Because there were no statistical differences or consistent patterns by study arm, we collapsed all participants for the remaining secondary statistical analyses. We first compared the overall portal sign-up and use in the trial to the total rates in usual care during the same time period, using 2-sample tests of proportions. We also examined changes over time in self-reported survey measures, using paired t-tests for continuous variables and McNemar’s test for categorical assessments. In addition, we looked at unadjusted differences in portal use in the follow-up period by participant characteristics using chi-squared tests. We performed all analyses using Stata 14.2 (College Station, TX).

Finally, to analyze participant-reported reasons for portal nonuse at follow-up, two members of the analytic team categorized the reasons into distinct categories, meeting in person to establish consensus.

Results

Figure 1 [Insert Figure 1] displays the recruitment and follow-up of the trial sample.³² We enrolled 93 participants in the trial, collecting portal use outcomes on 88 individuals (95% follow-up overall; 94% take-home and 95% in-person). Follow-up surveys were collected on 75 individuals (81% follow-up overall; 78% take-home and 84% in-person).

Figure 1.

Recruitment of Participants (Adapted from CONSORT Flow Diagram)

Participant Characteristics

The mean age of the sample was 54 years, and 51% had self-reported limited health literacy; 61% were non-white, 52% were female, 45% were in fair/poor health (Table 1) [Insert Table 1]. A quarter of the sample reported limited English proficiency, despite completing the phone screening process in English. A majority used the Internet and email daily (76% and 65%, respectively). Participants were balanced between trial arms, with the exception of anxiety.

Table 1.

Baseline Participant Characteristics, Overall and by Trial Arm

Variable	Overall (n=93)	Take-Home Training Arm (n=49)	In-Person Training Arm (n=44)	p-value (t-test or chi-square test)
Age (mean, SD)	54.3 (13)	52.5 (14)	56.3 (10)	0.14
Gender, n (%)				0.48
Female	48 (52)	25 (51)	23 (52)
Race/Ethnicity, n (%)				0.79
White or Caucasian	35 (39)	17 (35)	18 (41)
African American or Black	26 (29)	15 (27)	11 (30)
Hispanic/Latino	11 (12)	7 (14)	4 (9)
Asian or Pacific Islander	12 (14)	6 (12)	6 (14)
Other/Mixed	5 (6)	4 (8)	1 (2)
Health Literacya, n (%)				0.84
Limited	47 (51)	25 (52)	22 (50)
Adequate	45 (49)	23 (48)	22 (50)
English Proficiencyb, n (%)				0.56
Limited	23 (25)	12 (25)	10 (23)
Advanced	69 (75)	36 (74)	33 (75)
Self-Rated Health Status, n (%)				0.73
Fair or poor	41 (45)	21 (43)	20 (47)
Chronic Condition, n (%)
Hypertension	50 (58)	28 (57)	22 (50)	0.49
Depression	36 (42)	20 (41)	16 (36)	0.66
Diabetes	33 (38)	16 (33)	17 (39)	0.55
Anxiety	31 (36)	22 (45)	9 (21)	0.01
Asthma or COPD	21 (23)	11 (22)	10 (23)	0.97
Heart Disease	8 (9)	5 (10)	3 (7)	0.56
Heart Failure	6 (6)	4 (8)	2 (5)	0.48
Chronic Kidney Disease	6 (7)	2 (4)	4 (9)	0.42
Frequency of Internet Use, n (%)				0.57
Daily	71 (76)	36 (75)	35 (80)
Weekly	14 (15)	9 (19)	5 (11)
Every 2–3 weeks or less	7 (8)	3 (6)	4 (9)

^aLimited health literacy categorized as any self-reported lack of confidence (less than extremely confident) filling out medical forms on own

^bLimited English proficiency categorized as any self-reported limitation in speaking English (less than very well)

RCT Findings: Portal Sign-up and Use

The main trial results are shown in Table 2 [Insert Table 2]. By design, everyone in the in-person training arm watched the videos at least once, compared to 43% in the take-home arm (p<0.001). The average number of video lessons viewed (out of 11) was 3.4 (s.d.=3.6, median=2, range 0–11), with 1.3 average views in the take-home arm and 5.7 average views in the in-person arm. The most watched lesson topics were for accessing lab results (46%) and signing up (43%), while the least watched video topics were creating usernames (16%) and passwords (15%).

Table 2.

Primary Trial Outcomes: Portal Initiation and Use at 3–6 Month Follow-Up Assessed via Electronic Health Record

Intention to Treat Analysis:Comparison by Trial Arm, n (%)
	In-Person Training Arm (n=44)	Take-home Training Arm (n=49)	p-value (chi-squared tests)
Accessed Training Curriculum	44 (100)	21 (43)	<0.001
Initiated Sign-up for Portal in Clinic	8 (19)	9 (20)	0.9
Logged into Portal Website	9 (21)	9 (20)	0.8
Per Protocol Analysis: Comparison by Exposure to Portal Training, n (%)
	Accessed Training (n=65)	Did Not Access Training (n=28)	p-value (chi-squared tests)
Initiated Sign-up for Portal in Clinic	12 (20)	5 (19)	1.0
Logged into Portal Website	14 (23)	4 (15)	0.4

^aOverall participants: n=93, including access to online training curriculum

^bPortal use outcomes (initiation and log-ins): n=88

Overall, 18 participants (21%) logged in to the portal at least once during the follow-up period and another 17 (20%) were newly web-enabled during a clinic visit to initiate the portal sign-up process. For the intention to treat analysis, the proportion of patients logging into the portal did not differ by in-person vs. take-home arm (p=0.8). The mean number of log-ins to the portal website was 1.3 overall (s.d.=3.1, range 0–15) and 4.7 among those who logged in at least once (s.d.=4.4, range 1–15), and this also did not differ by study arm (p=0.54 and p=0.53, respectively). Similarly, there was no difference in portal sign-up rates at follow-up by study arm (p=0.9). In the per protocol analysis, those who had watched at least one of the videos (vs. none) had a slightly higher proportion of logging into the portal (23% vs. 15%), but this difference was not statistically different. There were no significant differences in portal outcomes by clinic site (data not shown).

Usual Care Comparison

When comparing portal use to the general clinic rates over the same time (Figure 2) [Insert Figure 2], rates of portal sign-up and login were over twice as high: 20% in the trial compared to 8% in usual care for initiating portal sign-up process (p<0.001), and 21% in the trial compared to 9% in usual care for portal log-ins (p<0.001). The mean number of portal logins within the usual care comparison sample was also lower at 0.49 (s.d.=4.2).

Figure 2.

Portal initiation and use during 3–6 month follow-up period, comparing all study participants to a non-randomized usual care comparison group

Participant Attitudes/Skills and Perceptions of Healthcare

We found a significant changes in patients’ self-reported skills in using the portal website from the baseline to follow-up surveys (Table 3) [Insert Table 3], with the proportion reporting that they agreed they had sufficient ability to use the website increasing from 63% to 78% (p=0.03). Similarly, there was a significant increase in the eHealth literacy scale over time (14.4 to 16.2, p<0.001). However, we found a decrease in self-reported interest in using the portal, from 53% to 39% (p=0.01).

Table 3.

Changes in Participants’ Self-Reported Survey Measures from Baseline to Follow-Up at 3–6 Months Post-Training

Variable	Baseline Survey: N (%) or Mean (s.d.)	Follow-up Survey: N (%) or Mean (s.d.)	p-value (paired t-test or McNemar’s test)
Interest/Attitudes related to Portal Use
Internet is useful for health decisions*	47 (65)	54 (75)	0.18
Important to get personal medical information electronically†	71 (95)	70 (93)	1.00
Confident in PHR safeguards/privacy‡	60 (80)	57 (76)	0.44
High interest in using MYSFHEALTH portal to see personal medical record§	53 (72)	39 (53)	0.01
High interest in potential portal features||
Appointment reminders	55 (73)	56 (75)	0.82
General health information	36 (48)	42 (56)	0.22
Medication reminders	43 (57)	43 (56)	0.83
Lab/test results	62 (83)	59 (79)	0.44
Skills for Portal Use
Self-reported skills to use portal website¶	46 (63)	57 (78)	0.03
High confidence in using MYSFHEALTH without help#	47 (67)	54 (77)	0.11
High confidence in using MYSFHEALTH to improve health#	44 (63)	41 (58)	0.53
eHealth Literacy (eHeals**)	14.4 (3.7)	16.2 (2.4)	<0.001
Self-reported Chronic Disease Management
Patient Assessment of Chronic Illness Care	3.2 (1.0)	3.2 (0.9)	0.72
Self-Efficacy for Managing Chronic Disease‡	6.3 (1.9)	6.7 (2.0)	0.10

^aSample size is n=75 with complete data at both time points

^*Usefulness categorized as very useful/useful vs. neutral/not useful/not at all useful

^†Importance categorized as very/somewhat vs. not at all important

^‡Confidence in safeguards categorized as very/somewhat vs. not confident

^§Interest in portal categorized as high interest vs. low/neutral/moderate/none/need more information

^||Interest in portal features categorized as high interest vs. somewhat/a little/not at all interested

^¶Skills categorized as strongly agree/agree vs. neutral/disagree/strongly disagree

^#Confidence in using portal without help and to improve health categorized as 8–10 vs. 1–7

^**eHeals score calculated using 4 of 8 eHeals questions; range 4–20 with higher score noting higher perceived eHealth literacy

^††Patient Assessment of Chronic Illness Care Survey; range 1–5 with higher score noting higher ratings of chronic illness scare

^‡Self-Efficacy for Managing Chronic Disease Scale; range 1–10 with higher score noting higher self-efficacy

Participant Demographics and Portal Use

In a secondary analysis, we examined portal use during follow-up by relevant participant characteristics (Figure 3) [Insert Figure 3]. Participants who were 60 or older were more likely to have logged in to the portal than their younger counterparts (32% vs. 13%, p=0.03). In addition, portal use was higher among participants with adequate compared to limited health literacy (35% vs. 7%, p<0.01), and among participants with 1 vs. with 2+ conditions (35% vs. 15%, p=0.03).

Figure 3.

Overall Portal Use during 3–6 Month Follow-up Period, by Selected Participant Characteristics

Reasons for Nonuse

Among 38 participants who reported reasons for not using the portal at follow-up, the most common reasons were: being too busy or occupied with personal issues (24%), not having the need to use it (18%), being concerned about security (13%), needing more help to sign up (8%), misplacing written information from the training (8%), and having limited access to a computer/Internet (8%).

Discussion

We found an online video-based portal training resulted in moderate use of the portal in subsequent months. Participants’ self-reported confidence in using the website and eHealth digital literacy appeared to be the most malleable to improvement post-training. The mode of the training (in-person vs. take-home) did not affect ultimate portal use, suggesting that any one-on-one session highlighting the website and providing general information about enrollment may prompt some level of increased patient use. Therefore, the scalability of our training may be promising, especially among systems with a way to deliver a brief in-person educational session. A secondary examination of overall portal use during the same time period suggests that the trial participants had substantially higher use than usual care.

Yet our training was not successful in getting a majority of patients to become portal users. Our portal enrollment process at the San Francisco Health Network protects patients’ security by requiring in-person verification to sign up, but this represents a high bar for widespread adoption. Furthermore, previous studies have shown that providers/staff and system implementation directly influence individual-level patient portal use,^33–35 and our system was early in the clinic workflow development process to fully support patient use. While we were not able to harness specific patient-provider or patient-staff discussions about portal use to time the delivery of training in our study, future work capitalizing on clinic and provider support for portal use will be critical. Finally, the existing usability of existing portal interfaces has been shown to be subpar,^16–19,36 which represents an additional barrier to patient log-in. In fact, patients in our study reported less interest in using the portal at follow-up, perhaps suggesting that our system’s implementation and enrollment process and current portal functionality may be less appealing than they had envisioned.

One previous study evaluated a nurse-led portal training for primary care patients enrolled in an ongoing chronic disease care coordination program, reporting pre-post improvements in functional status and reduced emergency department visits.³⁷ This study found no improvement in chronic disease self-efficacy, similar to our findings, but much higher rates of portal use among the participants (perhaps due to higher levels of patient educational attainment).

It is important to note that a large proportion of patients in our Network were not eligible for this study, given the high number of non-English speakers and lower rates of email use in our healthcare setting. The lack of portal accessibility in multiple languages, as well as the need for a higher level of existing digital literacy skills to sign up, represent substantial barriers to use among diverse populations.³⁸ Even within our sample that was purposefully screened for these issues, we delivered training to patients reporting limited English proficiency and a lack of regular Internet and/or email use – two groups who likely need more intensive assistance to meaningfully access portal websites.

Overall, our findings suggest a need for more computer/digital literacy training and support, especially given reported barriers related to technology at follow-up among eligible participants. More vulnerable patients, particularly those with limited health literacy and the sickest patients, were less likely to be portal users, similar to previous work.^9,11,17,39 Patients with limited health literacy had particularly low rates of portal use, underscoring a need for tailored training or support for this sub-population. Interestingly, those over age 60 were more likely to be portal users post-training (which is in contrast to other work^34,39); this population might have specfiically benefited from the brief training to orient them to the website.

Our study was limited by a modest sample size. In addition, our work was conducted within clinics in the same urban safety net system, which may limit generalizability, especially to more resourced healthcare settings with different enrollment strategies and more affluent patients. We also used self-reported health literacy which could differ from direct assessment of health literacy status such as the Short Test of Functional Health Literacy in Adults (S-TOFHLA). Finally, our pre-post changes in the self-reported patient survey outcomes lack an a equivalent control group and could be subject to regression to the mean.

Moving forward, we cannot forget the basic training needs of diverse patients to be able to use existing health technologies. In contrast to a “build it and they will come” approach, portals will not be universally effective without sufficient training and support for broader uptake. In addition, health literacy (and likely English proficiency, which we could not study fully given our lack of non-English functionality) continues to present barriers to use. We need better research and implementation strategies focused on patients with communication barriers to address this issue more explicitly.