Using Electronic Health Record Portals to Improve Patient Engagement: Research Priorities and Best Practices

Abstract

Ninety percent of health care systems now offer patient portals to access electronic health records (EHRs) in the United States, but only 15% to 30% of patients use these platforms. Using PubMed, the authors identified 53 studies published from September 2013 to June 2019 that informed best practices and priorities for future research on patient engagement with EHR data through patient portals, These studies mostly involved outpatient settings and fell into 3 major categories: interventions to increase use of patient portals, usability testing of portal interfaces, and documentation of patient and clinician barriers to portal use. Interventions that used one-on-one patient training were associated with the highest portal use. Patients with limited health or digital literacy faced challenges to portal use. Clinicians reported a lack of workflows to support patient use of portals in routine practice. These studies suggest that achieving higher rates of patient engagement through EHR portals will require paying more attention to the needs of diverse patients and systematically measuring usability as well as scope of content. Future work should incorporate implementation science approaches and directly address the key role of clinicians and staff in promoting portal use.

Electronic health records (EHRs) were developed to manage clinical information, not to engage patients. However, patient access to their EHR data through online portals or mobile applications represents a potential tool for improving patient engagement (1). As the landscape expands with the growth of application programming interfaces to increase bidirectional data flow with patients (2) and greater patient access to medical data, such as clinical notes (3), the potential impact of patient engagement with these platforms will grow in parallel.

Currently, approximately 90% of U.S. health care systems and providers offer patients online portal access to their EHR data (4), largely supported by the over $30 billion in financial incentives from the meaningful use program (5). Common features of online patient portals include the ability to view visit summaries, test results, and immunization and allergy lists, in addition to secure messaging, appointment scheduling, and medication renewals (6). Despite a robust patient portal infrastructure across many U.S. health care systems, only 15% to 30% of patients use even a single portal feature (4), and portal use is largely confined to a specific setting, such as outpatient care in integrated delivery systems (7).

To date, there is limited evidence linking clinical outcomes to portal use (8), but there is substantial demand from patients and their caregivers to access EHR data and communicate electronically with health systems. The strongest evidence supporting the importance of portal use is related to extremely high patient interest (8) and potential to improve patient satisfaction, convenience, and self-management (9).

We examined recent studies of patient engagement with EHR data through patient portals to identify research priorities and best practices.

Methods

Adhering to the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) recommendations for scoping reviews (10), we searched PubMed from 1 September 2013 through 4 June 2019 to identify published studies on use of online patient portals (Appendix Table 1, available at Annals.org). Search terms represented the constructs of self-management, engagement, and uptake of patient portals. We searched only PubMed because of the specific biomedical and health care focus of papers published on online portal use and our focus on implementation in U.S. settings. The beginning search date was chosen to exclude studies before meaningful use requirements, when the absence of regulation made the functionality of these systems very heterogeneous.

We excluded articles in which no original data were presented (for example, reviews, protocols, commentaries), reported studies were done outside the United States, and reports focused on stakeholder or clinician perspectives without patient input.

Two coauthors (C.R.L. and A.G.C.) completed all data extraction and conducted full-text review of all articles that met initial selection criteria; this was because studies on portal implementation were not always evident in the abstract alone. Full-text review enabled us to identify and exclude studies that did not examine an intervention or program to engage patients in portal use, and studies that did not examine portal use or barriers to use as a primary outcome. A third coauthor (U.S.) reviewed a subset of the full-text results to ensure reliability in the final inclusion process. We did not hand-search the reference list of included papers to find additional studies, but we did determine whether any of the included studies evaluated the same inter vention or program, combining studies in our summarization as appropriate.

We chose to exclude observational portal use and outcome studies that did not have an interventional component because of the well-established body of literature examining patterns of portal use by patient age (11, 12), race/ethnicity (10, 13, 14), socioeconomic status (12), language (14, 15) presence of chronic health conditions (12), digital literacy or access (13, 16), and provider influence or recommendation (10, 12). These observational studies do not shed light on implementation issues that may explain why some patient subgroups use portals less than others.

We grouped included articles according to the major purpose of the research, to separate studies focused on usability or barriers to portal use from studies of interventions aimed to evaluate or increase portal use. We abstracted information from each study on the methods and sample, primary objective and intervention, and major findings reported.

We used the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework (17) to specifically note whether an included study provided relevant findings related to implementation and dissemination and, qualitatively, identified exemplar interventional studies. RE-AIM is an evaluation framework used in implementation science to promote wider adoption and sustained implementation of effective programs and interventions. Specifically, we examined included studies by using the explicit RE-AIM framework domains of reach (proportion and representativeness of patients using the portal or responding to a specific engagement approach), adoption (proportion and representativeness of the sites offering portals or portal engagement strategies), implementation (consistency, costs, and barriers in patient engagement approaches), and maintenance (the ability to sustain the portal engagement strategies over time). We did not focus on the effectiveness domain of RE-AIM because other recent review articles have specifically focused on portal use and subsequent effectiveness on clinical or behavioral outcomes (18, 19).

Results

Of 283 articles initially identified, 53 met inclusion criteria (Appendix Figure, available at Annals.org). We evaluated 3 major categories of articles separately: studies on use and outcomes, usability studies, and studies on barriers to use.

Use and Outcomes Studies

Twenty-four articles described a type of portal intervention or program, either delivery of an intervention through a portal (20–27) or evaluation of a program to increase portal use (28–43) (Appendix Table 2, available at Annals.org). The majority (18 studies) targeted adults in outpatient primary or specialty care, and 6 evaluated portal enrollment and use among pediatric patients and their parents or caregivers (26, 29, 31, 35, 40, 41). Only 7 studies examined portal use among inpatients, often engaging caregivers in use of the portal (28, 31, 34, 38, 39, 42, 43). Overall, there was low to moderate uptake of portal use in most studies.

Among the 8 studies that delivered interventions via an existing portal, the type of intervention varied widely. For example, studies prompted patients to complete such tasks as advance care planning (21) and cancer screening decision-making modules (22, 27) on the portal before a visit. Other studies examined tasks (such as patient-reported outcome completion) after office visits (20), and others were subgroup analyses of broader programs in which portals were only one component of an intervention (23, 24). Overall, the studies with lowest rates of use passively delivered the intervention via the portal without a broader engagement plan.

Of the 15 studies that evaluated specific interventions or programs to increase portal use, the types of outcome measures assessed included portal enrollment and registration (30, 32, 33, 35, 41, 42); portal logins immediately after training (28, 31, 34, 39); sustained portal use postintervention over a follow-up period (36–38, 40); and use of specific features, such as secure messaging within the portal (28, 31, 39). Five were randomized trials or quasi-experimental designs; the remainder were pre–post or cross-sectional analyses. Among the randomized and quasi-experimental studies, 48% to 81% of patients used the portal postintervention (34, 36, 37, 39, 42). The pre–post or cross-sectional studies generally evaluated broader system-or clinic-wide quality improvement efforts to increase portal use and reported more variation in rates of portal use ranging from 8% to 77% of patients (22, 30, 32, 33, 38). Of note, the studies with high rates of use often employed dedicated staff to enroll and assist patients.

Usability Studies

Twelve usability studies evaluated the functionality of portal systems related to design, layout, format, and content (Appendix Table 3, available at Annals.org). These studies included 10 observational approaches or pilots of existing platforms (44–53) and 2 user surveys assessing ease of use, usefulness, usability, and satisfaction (54, 55). The studies covered the overall functionality of the portal (such as having participants complete multiple tasks in sequence) as well as more focused testing (such as improving test result reporting in an iterative manner).

Direct observation studies demonstrated that patients with limited health literacy and numeracy skills, along with those without extensive previous computer experience and older adults, faced substantial barriers in using the portal interfaces (46, 49–51, 53, 54). However, despite these observed challenges in usability, several self-report studies noted that patients rated the systems highly and expressed strong interest in using portals (45, 46). These observations highlight the differences in usability when outcomes are direct observations versus patient ratings. Studies also underscored the need for better functionality of the platforms, with specific needs for simplified log-ins, consistency across platforms, and greatly improved navigation (50, 53, 55). In addition, acceptability to patients appeared to be better in studies that used audiovisual features or improved graphical representations of the portal content (47, 48, 52).

Barriers to Use

Seventeen studies specifically outlined barriers to portal use (Appendix Table 4, available at Annals.org). Six of these studies included clinicians in their sample along with patients (56–61), whereas the remaining studies focused on patients or caregivers (62–72). Ten of the studies were qualitative, and 7 studies used quantitative approaches.

The most common concerns or barriers among patients centered on the need for support in understanding the medical content presented (44, 56, 59, 60, 63, 64, 66, 69, 71) and the need for improved digital skills and confidence (62, 66, 67, 69, 70). Other barriers included preferences for and satisfaction with current in-person communication (66–69), security/privacy concerns (62, 68, 70, 72), and desire for more ways to contribute their own feedback or data back into the EHR (59, 63). Included studies focusing on clinicians identified concerns about overwhelming patients with information (34, 59), EHR information creating patient anxiety (56, 59), and the potential time burden of secure messaging due to a lack of existing workflows to support electronic communication in their practices (34, 59, 61).

Exemplar Studies of Implementation

Many studies documented low use of portals across various health care settings, but several studies went further in addressing the RE-AIM concept of reach (extent to which the intervention reached the intended population). For example, Arcia (45) examined real-world portal uptake by documenting a tailored portal outreach program, completing the work in both English and Spanish and using the prenatal care period to deliver educational content. Similarly, Ramsey and colleagues (35) outlined an outreach program that utilized medical students to meet adolescent patients before their visits to review the purpose of the portal and assist with enrollment, and collected patient surveys to understand current digital use and preferences for electronic health care communication.

Other studies highlighted findings related to the implementation domain of RE-AIM. McCleary and associates (33) and Kamo and coworkers (30) targeted patients and clinicians or staff simultaneously to promote portal use. For example, Kamo and coworkers adapted their telephone triage system to complement patient portal use in their setting and expended substantial effort to develop workflows that could deliver secure messages sent via the portal to the appropriate staff member (in a team-based rather than physician-focused approach). McCleary and associates developed staff educational programming about patient portal use and rolled out programs to support staff with patient portal enrollment.

Less evidence was provided on comparing different portal outreach approaches across clinics or sites, or long-term implementation approaches. Therefore, available evidence reveals little about the adoption and maintenance constructs of the RE-AIM framework.

Discussion

Available studies generally found low to modest uptake of patient portals. Studies that used broader implementation strategies to engage patients, caregivers, clinicians, or staff generated higher rates of patient engagement. In addition, studies that directly addressed usability of these systems by diverse users shed light on functionality challenges that could be improved to achieve broader portal uptake. Patient-facing digital health technology encounters barriers to uptake and sustained engagement (73, 74), so our findings have relevance beyond patient portals.

Current evidence suggests 4 best practices for future research studying patient use of EHRs and portals (Table).

Table.

Summary Recommendations on Research and Reporting for Understanding Patient Engagement in Their Health Care Using Electronic Health Records

Research	Examples
Measure patients’ skills and interests in using digital health tools.	Understanding health and digital literacy oftarget users is critical to interpreting study results.
Use mixed-methods studies to determine usability and uptake.	User-centered design of portals will require gathering qualitative data about perceived barriers as well as observational evidence of navigational challenges.
Tailor portal interface and outreach to reach broad groups of patients, families, and caregivers.	Consider using audiovisual formats for data presentation to improve comprehension across literacy levels; translate tools into languages other than English.
Study the combination of digital tools with human support.	Initial in-person training and ongoing support can increase digital tool use; greater integration into existing provider and staff workflows will also reinforce patient support in using the tools.
Reporting
Incorporate implementation frameworks.	Frameworks such as RE-AIM may be particularly suited to understand processes and contextual factors that influence patient and caregiver use as well as provider and staff use.
Emphasize external validity alongside internal validity.	Report how and whether the approach could be tested in new or different populations.
Report effectiveness and use from multiple viewpoints.	Consider investigating a broader set of stakeholders in digital health research, including caregivers, peers, providers, staff, and system leadership.

RE-AIM = reach, effectiveness, adoption, implementation, and maintenance.

1. Studies of patient engagement via EHR portals should directly measure patient skills and interest, including digital health literacy, digital access, health literacy, and numeracy.

Available studies identify patient subgroups that face barriers to portal use. Paying attention to digital literacy, device ownership, and broadband access—along with health literacy and numeracy—is critical to improving engagement via patient portals. Not only do these factors influence portal use, but they also often coexist with increased health needs (75). Some studies suggest a correlation between health literacy and digital literacy (49, 51) and underscore the growing evidence that a digital divide persists among a sizable proportion of the U.S. population, in particular for low-income and older adults (76). Research examining portal use, or delivering interventions via the portal, should explicitly measure these factors.

2. Interventions to promote patient engagement via EHR portals should be designed to address the usability barriers identified to date.

Available studies, particularly those examining usability and barriers to use, highlight the need for greatly improved usability of portal platforms, almost all of which appeared to be vendor-designed portals without many modifications. Cumbersome, fragmented, and complex interfaces dampen the enthusiasm of new users and disadvantaged patients who struggle to find relevant information and might never return to the platform.

Future research should incorporate audiovisual and improved graphical representation of medical data, because studies that used these features reported better patient uptake and satisfaction (47, 48). Only 2 studies recruited non–English speakers (45, 58), suggesting a need to study portal-based interventions available in languages other than English. Finally, future studies should consider combining quantitative use or survey data with direct observation of portal use. Studies reporting observation provided a richer understanding of patient experiences by uncovering underlying beliefs, concerns, and gaps in functionality.

3. Interventions to promote patient engagement via EHR portals should incorporate in-person support for patients.

We observed more favorable portal uptake in studies that incorporated in-person enrollment and training for patients (28, 36). This suggests an opportunity in future research to combine technological outreach (such as communication delivered via portals) with human support, which is a burgeoning area of investigation. Studies that use adaptive designs to understand the best combination of technological and human interaction could be fruitful, especially if we can automate certain processes while systematically investigating the best timing for human follow-up, support, and reinforcement.

The role of caregivers, family, friends, and peers remains a critically underutilized and underevaluated aspect of patient engagement that warrants much more attention in future studies. Some patients cannot use portals themselves, but their care partners or proxy may be able and willing to do this for them (28).

4. Studies of patient engagement via EHR portals should use the principles of implementation science.

Patient engagement problems represent implementation problems. Using RE-AIM, as well as other implementation frameworks, can promote standard ways to report on portal use that take key stakeholders into account. Several of the studies in this review dealt critically with the construct of reach to understand who was offered and currently using portals (including the representativeness of the user groups). Future work should shift more to the adoption, implementation, and maintenance components to spread best practices. This will include understanding key steps in a patient’s health care journey and specific health behaviors for which portal engagement is most relevant (combined with appropriate, established behavioral change approaches [77]), and understanding and designing workflows for clinicians and staff to support patients’ use (78).

Moreover, patient subgroups with the greatest barriers to use could be specifically targeted by using implementation science approaches with a goal of increasing health equity. In particular, high-cost, high-need patients are those with the most to gain from more regular health care communication and information about their conditions, and yet most systems have not targeted portal engagement efforts on the basis of their potential impact.

To directly address some of these recommendations, promising work is currently under way. With respect to usability, tools have been developed that can be overlaid or integrated with the EHR to improve the comprehensibility of medical content, such as hyperlinks to plain-language descriptions and tailored educational materials (7, 75, 79). In addition, to specifically address the health and digital literacy barriers of diverse patients, a growing body of work is linking patients to community-based resources, such as libraries and nonprofit organizations, to foster digital inclusion (80).

Finally, related to the need for greater attention to implementation strategies, efforts are under way that have generated knowledge about patient engagement tied to learning health system approaches (81, 82) and implementation of comprehensive marketing and promotion of patient portals (83).

Our study has limitations. First, our search terms could have excluded relevant articles that used different key words, as well as articles published in journals not indexed in PubMed. Second, we were unable to combine outcome measures or directly compare the quality of the findings across articles in a systematic way because of the heterogeneous nature of the included studies. Third, we did not focus on the clinical impact of portal use because there are existing review articles on that topic. Finally, small sample sizes for certain types of studies, such as those engaging both families and caregivers, precluded us from making more specific recommendations in some areas.

In conclusion, available studies suggested low patient use of portals made available to them and identified barriers to use. Moving forward, research on improving patient engagement using EHR portals should measure a broader array of patient skills, access, and interest; address usability barriers identified to date; incorporate in-person support for patients; and apply the principles of implementation science. Widespread adoption and routine use of EHR portals by patients and caregivers will require multifaceted approaches to encourage uptake.

Appendix Figure 1.

Study flow diagram.

Appendix Table 1.

Search Strategy

Date	Database Searched	Search Strategy	Results, n
1 September 2013 to 4 June 2019	PubMed	(“self management”[tiab] OR engaged[tiab] OR engagement[tiab] OR engages[tiab] OR engage[tiab] OR engaging[tiab] OR “user uptake”[tiab] OR “self help”[tiab] OR “Patient Participation”[Mesh]) AND (“patient portal”[tiab] OR “patient portals”[tiab] OR “portal use”[tiab] OR “online portal”[tiab] OR “online portals”[tiab]) AND (“2013/09/01”[PDat] : “2019/06/04”[PDat])	283

Appendix Table 2.

Studies of Patient Portal Use and Related Outcomes

Study, Year (Reference)	Design (Sample Size)*	Setting and Population	Intervention	Findings
Interventions delivered via the portal
Bell et al, 2018 (20)	Quantitative (n = 17 133)  EHR and use data	Single multispecialty academic, adult orthopedic practice in Philadelphia	Portal utilization: patient-reported outcomes	Use  82% of patients logged in at least once during the 30-mo study period Intervention participation  42% to 52% completed patient-reported outcomes, and 30% sent at least one message  Lower uptake among older patients, but improved uptake in all age groups on mobile platform versus browser
Bose-Brill et al, 2018 (21)	Quantitative (n = 419)  Prospective, quasi-experimental analysis of EHR and use data	Older patients (≥50 y) in 2 Columbus, Ohio, clinics	Portal utilization: advanced care planning module	Intervention participation  20% of intervention patients responded, one half of whom then had advance care planning documentation rates and improved quality at follow up (significantly higher than control group)
Krist et al, 2017 (22); Woolf et al, 2018 (27)	Quantitative (n = 11 458)  EHR and use data	12 primary care practices in northern Virginia	Portal utilization: cancer screening modules	Use  About one fifth of the unique portal users had to make a decision about 1 of the 3 selected screenings Intervention participation  21% started the module, and 8% of these patients completed the decision module  41% of them felt the module made their appointment more productive  Lower use among women, those without prior cancer screening, Hispanic patients, Asian patients, non-English-speaking patients, and uninsured patients
Pecina et al, 2017 (23)	Quantitative (n = 1769)  Subgroup analysis of larger collaborative care intervention EHR and use data	Large, academic medical center in Minnesota	Portal utilization: communication with depression care providers	Use  Higher use among younger, married, and female users Intervention participation  15% of patients analyzed used the portal to communicate with their care managers  Portal users were more likely to complete the depressive screening at 6 mo (76% vs. 66%)
Quinn et al, 2018 (24)	Quantitative  Subgroup analysis of RCT (107 patients with diabetes) Qualitative  Secure messages (4109 messages)	26 primary care clinics in Maryland	Portal utilization: secure messaging with diabetes educator	Intervention participation  77% of intervention group participants messaged during the intervention period  Many messages were about glucose monitoring and self-care Other outcomes  Better glycemic control among those sending messages versus those who did not
Raghu et al, 2015 (25)	Quantitative (n = 18 702)  EHR and use data	Large outpatient clinics in Arizona	Portal utilization: updating medication lists via secure message	Intervention participation  Around 50% responded to either telephone or secure message requests, but demographic patterns of who was using portals differed
Thompson et al, 2018 (26)	Quantitative (n = 257)  Surveys: patients with visits to clinic	Academic clinic in Florida primarily serving adolescents with Medicaid	Portal utilization: survey delivered via portal	Intervention participation  Only 3 participants fully completed either survey, but more than one quarter of both groups read the survey e-mail
Programs to improve portal use
Dalal et al, 2016 (28)	Quantitative  Use data (120 patient-caregiver dyads; 239 total)  System usability and satisfaction survey (10 patients and 8 caregivers) Qualitative  Patients’ goals, preferences, concerns and content of secure messages	Inpatient MICU and oncology unit (patients and caregivers) Large academic hospital in Boston	Portal education (inpatient) and portal implementation evaluation	Use  66% of participants used the goal setting and/or the secure messaging features Other outcomes  Positive usability and satisfaction ratings  Specific suggestions for portal functionality improvement (e.g., highlight abnormal test results and increase clinician portal knowledge)  Digital and logistical barriers to adoption
Dykes et al, 2017 (43)	Quantitative  (2105 admissions [1030at baseline, 1075 during intervention])  EHR and use data	MICU patients at a large tertiary care center	Portal education and implementation evaluation (inpatient)	Use  18% of patients admitted during the intervention enrolled onto the portal Other outcomes  Higher enrollment among patients who were white, younger, and privately insured  Aggregate rate of adverse events decreased by 29% during the intervention period  Patient satisfaction increased from 72% to 93%  Care partner satisfaction increased from 84% to 90%
Fiks et al, 2016 (29)	Quantitative  Use data (n = 237)  Monthly surveys via portal Qualitative  Interviews (n = 22)  Focus groups (n = 10)	Pediatric (6–12 years) asthma patients’ parents/guardians 20 primary care sites in 11 states	Portal implementation evaluation	Use  2.6% of invited families used the portal, most because of a mailed letter invitation Other outcomes  Use of portal was associated with more medication changes and primary care visits at follow-up
Greysen et al, 2018 (39)	Quantitative  Prospective, randomized intervention  Use data, observations, and patient satisfaction (n = 97)	Adult patients admitted to hospitalist service at large academic medical center in San Francisco	Patient education (inpatient randomization to usual care vs. brief training) and assistance with registration/log-in via tablets at the bedside	Use  58% logged into the portal at least once within 7 d of discharge; no differences between study groups Other outcomes  Observed ability to view provider messaging higher in training group (92% vs. 77%)  88% were satisfied or very satisfied
Kamo et al, 2017 (30)	Quantitative  Use data and cohort analysis (n = 189 723)  Patient satisfaction surveys (n = 465)	Integrated urban health care delivery system in the Pacific Northwest	Portal implementation evaluation	Use  39% of patients seen within a 12-mo period had enrolled onto the portal Other outcomes  Patient portal enrollees were likely to be urban, white, younger, female patients  41% of survey respondents rated their portal experience as excellent
Kelly et al, 2017 (31)	Quantitative  Use data (n = 296)  Parent survey (n = 90)	Preteen inpatients and their families at a tertiary children’s hospital inWisconsin	Portal implementation evaluation (inpatient)	Use  90% of parents offered the acute care portal used it  176 requests (29% of patients) and 36 messages (5% of patients) Other outcomes  89% of survey respondents felt the portal reduced errors in their child’s health care  60% said the portal improved communications with their child’s provider
Kidwell et al, 2018 (40)	Quantitative  Use data (n = 44) Qualitative  Patient ratings of ease of use and usefulness	Adolescents and young adults with sickle cell disease in clinics throughout the Midwest	Portal education program (homework for patients to complete via portal over time)	Use  All patient logged on at least once, with 46% continuing to use the portal after 2 mo Other outcomes  90% rated portal as high quality, 77% rated it as very easy to use, and 81% agreed it was useful or very useful  No clear effects on portal use on medical decision making or patient-provider communication
Krist et al, 2014 (32)	Quantitative  Use data (n = 28 910)  Clinician and patient surveys Qualitative  Staff training session transcripts (n = 7)  Exit interviews	Eight primary care practices in Virginia	Portal implementation evaluation	Use  26% of patients seen within the study period created a portal account, with consistent rates across all clinic sites after broad implementation efforts  Higher uptake among older patients and patients with multiple comorbidities, and lower uptake among underrepresented minorities Other outcomes  Clinicians’ and staffs’ previous negative experiences with informatics tools were a barrier to implementing the portal successfully
McCleary et al, 2018 (33)	Quantitative  Patient surveys (n = 1019) Qualitative  Focus groups with patients and staff (n = 25)	Ambulatory oncology practices at academic cancer treatment center in Boston	Portal implementation evaluation and staffing education/support	Use  Intervention increased patient portal enrollment by 6% to 53% over 2 mo Other outcomes  Barriers reported in terms of computer access, difficulty signing up, and lack of awareness of the benefits
O’Leary et al, 2016 (34)	Quantitative  Site-randomized portal intervention  Use data (100 intervention participants and 102 control participants)  Patient satisfaction questionnaires Qualitative  Structured patient and provider interviews (n = 100)	Large academic hospital in Chicago	Portal education (inpatient) and effectiveness evaluation	Use  57% used the portal more than once daily Other outcomes  A higher percentage of intervention unit patients could identify their care team and roles than the control group  No difference in knowledge about procedures, tests, or medications  No difference in patient activation  Patients often allow surrogates to use their portal account
Ramsey et al, 2018 (35)	Quantitative  Survey data (n = 96)	Adolescent practice in urban Maryland	Portal implementation evaluation	Use  88% of approached patients enrolled in the patient portal  High enrollment rates among black patients that reflect the overall clinic population
Ratliff-Schaub and Valleru, 2017 (41)	Quantitative  Pre-post analysis (total clinicsize, 1700 children annually)	Pediatric clinic serving patients with chronic illness in Columbus, Ohio	Portal implementation evaluation (targeting staff commitment, workflow, and family awareness)	Use  Percentage of patient visits increased from 2% to 30% over the quality improvement cycle work and was maintained for 16 months
Shaw et al, 2017 (38)	Quantitative  Utilization survey data at 3 points in time (n = 14)	Admitted patients undergoing a cardiac procedure	Portal education program with training delivered by nurse navigators	Use  36% of patients self-reported using a portal feature  Almost one fifth of patients watched an assigned health video via the portal
Sorondo et al, 2016 (36)	Quantitative  Prospective, quasi-experimental analysis (n = 96)	5 primary care sites in eastern Maine	Portal education and effectiveness evaluation	Use  79% of enrolled patients used the portal during the follow-up study period Other outcomes  No change in self-efficacy or health perceptions, but potential improvement in functional status and emergency visits
Stein et al, 2018 (42)	Quantitative  Prospective, randomized analysis intervention of portal education on subsequent portal use (n = 70)	Public hospital in Seattle	Portal education to train hospitalized patients to access discharge summaries	Use  48% of trained patients and 11% of the control group registered for the portal Other outcomes  Only 43% of eligible patients had working e-mail addresses to be able to be included in this study  80% to 85% of patients in both study groups preferred hospitals with access to an online patient portal
Toscos et al, 2016 (37)	Quantitative  Prospective, quasi-experimental analysis (n = 200)	Large cardiology practice in Indiana	Portal education and effectiveness evaluation	Use  81% of patients logged in ≥4 times over 12 mo Other outcomes  No change in patient activation between groups  High portal users had lower hemoglobin A1c values at follow-up

EHR = electronic health record; MICU = medical intensive care unit; RCT = randomized controlled trial.

^*Unless otherwise specified, sample sizes are the number of patients.

Appendix Table 3.

Studies of Portal Usability

Study, Year (Reference)	Design (Sample Size)	Setting and Population	Intervention	Findings
Observational testing or piloting
Alpert et al, 2016 (44)	Qualitative  Recall interviews with patients (n = 31)  Two provider focus groups (n = 13)	2 primary care practices in Virginia	Overall portal evaluation	73% of recalled incidents were negative Negative patient findings: lack of personalization, need for more functionality, need for more knowledge to understand laboratory data Provider negative findings: lack of feedback, increase on workload, inappropriate patient use Patients liked instant and clear information, and providers liked potential for patient empowerment and increased efficiency
Arcia, 2017 (45)	Quantitative  Use data (n = 12)  Usability/satisfaction surveys (n = 16) Qualitative  4 focus groups (n = 16)	Safety-net clinics in New York City Spanish-speaking and English-speaking pregnant women enrolled in Medicaid	Secure messages with educational content	Use  75% of participants had logged into the portal during the 4-mo study period Portal feedback  Satisfaction and usability were rated highly  Spanish speakers needed more assistance in navigating e-mail, logging in  Participants overall desired easier portal log-in without passwords
Czaja et al, 2015 (46)	Observational interviews (n = 54)	3 safety-net clinics in New York City	Overall portal evaluation; 3 different platforms	Diverse patients faced barriers to using the system, especially in terms of complex navigation and medical terminology for those with limited health literacy High ratings and interest in portals overall
Martinez et al, 2018 (52)	Observational interviews (n = 14 over 3 rounds)	Vanderbilt University adult primary care clinic	Design and evaluation of a diabetes dashboard embedded within the portal	Computer system usability improved from initial to final prototype rounds Specific changes included examples such as increasing font size, as well as adding reminders and star ratings to the dashboard for personalization
Morrow et al, 2017 (47)	Observational interviews (n = 12 and n = 24 in 2 rounds)	Older adults patients in Indiana	Secure message content displaying laboratory results	Patients understood video messages well and were satisfied with synthetic voice used to deliver test result information
Nystrom et al, 2018 (48)	Observational interviews and usability questionnaires (n = 14)	Participants recruited via e-mail (location not specified)	Laboratory test displays	Iterative rounds of feedback on test result graphical displays (such as out-of-range lipid levels), with corresponding changes in system usability scale with design changes Final version had improved clickability and navigation, along with simpler/less confusing layout related to normal versus abnormal results
Portz et al, 2019 (53)	Observational interviews (n = 24, 15 of which were portal users and 9 were nonusers)	Older adults with multiple chronic conditions at Kaiser Permanente Colorado	Overall portal evaluation	Observed barriers to portal tasks  Overall digital literacy anxiety and lack of awareness of features  Nonusers identified problems with font size and colors  Users identified problems with registration, logging in, and scheduling appointments
Taha et al, 2014 (51)	Observational interviews and questionnaire (n = 51)	Community sample of older adults in Miami	Overall portal evaluation	Observed barriers to many portal tasks, especially for those with limited numeracy and lack of previous Internet experience
Tieu et al, 2017 (49)	Observational interviews and usability questionnaire (n = 25)	San Francisco primary care public health care setting	Overall portal evaluation	Those with limited health literacy were significantly more likely to need assistance in completing portal tasks and faced basic computer barriers High interest in portals was expressed by patients overall
Yen et al, 2018 (50)	Observational interviews (n = 19)	Midwestern academic medical center with 6 hospitals	Overall portal evaluation	Most difficulty in exiting a specific section of the Web site and finding the right tab with the appropriate medical information Highest number of errors among the oldest participants
Patient reports
Mackert et al, 2016 (54)	Surveys (n = 4974)	Online national sample of respondents from an existing research platform	Assessed ease of use and usefulness of apps, trackers, and portals	Patients with limited health literacy associated with decreased use and lower ease of use and usefulness of portals, along with higher misperceptions about privacy
Nazi et al, 2018 (55)	Surveys (n = 200 624)	Veterans Affairs patients using the portal nationwide	Assessed current user experiences with pop-up survey	Patients liked tracking of prescription refills, but did not like lack of session time-out warnings and overall poor navigation Future decisions under way to increase proxy portal access

Appendix Table 4.

Studies of Barriers to Portal Use

Study, Year (Reference)	Design (Sample Size)*	Setting and Population	Intervention	Participant Concerns and Desires
Studies assessing patient experiences
Colorafi et al, 2018 (62)	40 patients  Quantitative  Survey Qualitative  Visit observation  Interviews	Older adults from 2 urban cardiac clinics in Arizona	Discussion of AVS	Digital literacy and access Privacy and security
Giardina et al, 2015 (64)	Qualitative  Interviews (n = 13)	Patients and caregivers in the Houston VA system	Discussion of abnormal test results	Support to interpret medical information Timeliness of information
Gerard et al, 2017 (63)	Qualitative  Open-ended responses within online platform (n = 260)	Primary care patients at an academic hospital system in Boston	Discussion of visit note and care plan	Want to contribute own data and share data with others
Haun et al, 2017 (65)	Qualitative (n = 48)  Focus groups  Simulations	Patients and caregivers from the VA health systems in Bedford, Massachusetts, and Tampa, Florida	Discussion of the overall VA HIT system	New features Virtual visits Better functionality and standardization Security and privacy More education and training needed
Hefner et al, 2019 (71)	Qualitative  Three focus groups (n = 17)	Patients with a cardiopulmonary condition at a large academic medical center in the Midwest	Discussion of experiences using portal secure messaging	Digital literacy/access More education/training needed Concern about provider engagement
Irizarry et al, 2017 (66)	Quantitative  Surveys (n = 100) Qualitative  4 focus groups (n = 23)	Community-based sample in Pittsburgh with varying health literacy and portal use experience	Discussion on overall portal interest and usefulness	Digital literacy and access Preference for in-person communication More education/training needed
Kim and Fadem, 2018 (69)	Qualitative  Focus groups (n = 17)	Convenience sample of older adults in New Jersey	Discussion on overall portal interest and specific features	Preference for in-person communication More education/training needed Concern about provider engagement
Mishuris et al, 2015 (67)	n = 19 Qualitative  In-depth interviews Quantitative  Survey	Home-based primary care patients, caregivers, and staff in the Boston VA system	Discussion on overall portal interest and usefulness	Digital literacy and access More education/training needed Satisfied with current care delivery methods Want to share data with others
Price-Haywood et al, 2017 (72)	Quantitative  Cross-sectional survey (n = 247)	Older adults with hypertension or diabetes at a large, integrated health delivery system	Discussion on overall interest and experiences with portals	Digital literacy/access Need for simpler interface Need for increased awareness
Sadasivaiah et al, 2019 (70)	Mixed methods (n = 16 507)  Overall interest in portal registration (yes/no)  Documentation and coding of specific reasons for noninterest	Inpatients at a large public hospital in San Francisco	Specific documentation of interest and noninterest in portal use documented in the EHR among nurses	Low interest Digital literacy and access Physical or mental barriers Security and privacy
Tieu et al, 2015 (68)	Qualitative  In-depth interviews (n = 16)	Patients and caregivers in the San Francisco safety-net system	Discussion on overall portal interest and usefulness	Digital literacy and access Health literacy Security and privacy Preference for in-person communication Want to share data with others Better functionality and standardization
Studies assessing patient and provider experiences
Alpert et al, 2018 (56)	Qualitative  Interviews (35 patients and 13 oncologists)	National Cancer Center in central Virginia	Participants provide feedback about portal usefulness and communication practices	Digital literacy and access Health literacy
Black et al, 2015 (57)	Qualitative  Interviews (10 patients) 6 focus groups (21 patients and 13 providers)	Asthma clinics in urban Philadelphia	Participants review AVS features and portal	More education/training needed Better functionality and standardization Digital access and literacy
Ochoa et al, 2017 (58)	Quantitative  Surveys (400 patients and 59 providers)	Safety-net health care system in Los Angeles	Participants provide feedback about portal adoption	Digital access and literacy (among a subset) Limited interest from providers
O’Leary et al, 2016 (59)	Qualitative  Interviews (18 patients) 3 focus groups (21 providers)	Large academic hospital in Chicago	Participants provide feedback about portal usefulness	Digital access and literacy Need for new features
Pillemer et al, 2016 (60)	Quantitative  Use data  Surveys (n = 6368) Qualitative  Interviews (13 patients)	Large, integrated delivery system in Western Pennsylvania	Participants provide feedback about their experience with the portal	Increased patient anxiety
Sieck et al, 2017 (61)	Qualitative  Interviews (29 patients and 13 providers)	Primary care offices at a large academic medical center in Ohio	Participants provide feedback about portal usefulness	More education and training are needed

AVS = after-visit summary; EHR = electronic health record; HIT = health information technology; VA = Veterans Affairs.

^*Unless otherwise specified, sample sizes are the number of patients.