Abstract
Background
Health care systems have entered a new era focused on patient engagement. Patient portals linked to electronic health records are recognized as a promising multifaceted tool to help achieve patient engagement goals. Achieving significant growth in adoption and use requires agile evaluation methods to complement periodic formal research efforts.
Objective
This paper describes one of the implementation strategies that the Department of Veterans Affairs (VA) has used to foster the adoption and sustained use of its patient portal, My HealtheVet, over the last decade: an ongoing focus on user-centered design (UCD). This strategy entails understanding the users and their tasks and goals and optimizing portal design and functionality accordingly. Using a case study approach, we present a comparison of early user demographics and preferences with more recent data and several examples to illustrate how a UCD can serve as an effective implementation strategy for a patient portal within a large integrated health care system.
Methods
VA has employed a customer experience analytics (CXA) survey on its patient portal since 2007 to enable ongoing direct user feedback. In a continuous cycle, a random sample of site visitors is invited to participate in the Web-based survey. CXA model questions are used to track and trend satisfaction, while custom questions collect data about users’ characteristics, needs, and preferences. In this case study, we performed analyses of descriptive statistics comparing user characteristics and preferences from FY2008 (wherein “FY” means “fiscal year”) to FY2017 and user trends regarding satisfaction with and utilization of specific portal functions over the last decade, as well as qualitative content analysis of user’s open-ended survey comments.
Results
User feedback has guided the development of enhancements to core components of the My HealtheVet portal including available features, content, interface design, prospective functional design, and related policies. Ten-year data regarding user characteristics and portal utilization demonstrate trends toward greater patient engagement and satisfaction. Administration of a continuous voluntary Web-based survey is an efficient and effective way to capture veterans’ voices about who they are, how they use the patient portal, needed system improvements, and desired additional services.
Conclusions
Leveraging “voice-of-the-customer” techniques as part of patient portal implementation can ensure that such systems meet users’ needs in ways that are agile and most effective. Through this strategy, VA has fostered significant adoption and use of My HealtheVet to engage patients in managing their health.
Keywords: patient portal, user-centered design, eHealth, veteran
Introduction
Background
Health care systems have entered a new era focused on patient engagement [1-3] described by enthusiasts as “the holy grail of health care” [4] and the “blockbuster drug of the century” [5]. Patient engagement strategies are designed to empower patients to play a more active role in their health care and make informed decisions, improve the patient experience, increase patient satisfaction, and achieve better health outcomes. Patient portals linked to electronic health records (EHRs) are recognized as a promising multifaceted tool to help achieve these patient engagement goals [6-9]. However, the adoption and sustained use of portals has generally fallen short of initial optimism [10-13] even in light of the significant growth in EHRs and tethered patient portals incentivized by Meaningful Use [14]. Positive benefits of portal use have been demonstrated [15-19], and the OpenNotes movement [20] has promoted patient engagement through health records transparency by enabling patient access to provider notes. Evidence indicates that such access improves communication and trust, patient safety, and, potentially, patient outcomes [21-25]. Two large integrated health care systems that launched tethered patient portals in 2003 with significant patient adoption and sustained use are Kaiser Permanente (KP) and the Department of Veterans Affairs (VA). KP’s portal, My Health Manager, is used by more than 5 million members, representing about 70% of adult KP members [26]. VA’s patient portal, My HealtheVet, has more than 4 million registered users (69% of VA patients receiving health care services in FY 2017 [wherein “FY” means “fiscal year”]), with 2.5 million authenticated Premium accounts (42% of VA patients receiving health care services) required for access to all portal features [27]. To better understand what elements are driving this adoption and sustained use, an implementation case study approach is warranted. One of the implementation strategies that are critical to foster the adoption and sustained use of patient portals is an ongoing focus on user-centered design (UCD). This is often accomplished as part of periodic research studies; however, more timely and agile methods are needed to design and evaluate patient portals.
User-Centered Design
UCD is a design philosophy and evaluation process that focuses on the end user’s characteristics, needs, preferences, and limitations throughout the design process and development lifecycle [28]. The emphasis of UCD is on understanding the end users and their tasks and goals and optimizing the product to enable the users to fulfill these, rather than requiring users to adapt to the designer’s preferences [29]. UCD of eHealth applications, such as patient portals, necessitates ongoing assessment of user characteristics and preferences and incorporation of assessment insights into ongoing portal development and enhancements. This process includes focusing on what features are considered to be most essential by users [30]. Published compilations of implementation strategies have called for the further development of processes like UCD as a means to obtain and use patient or consumer feedback to support the adoption of innovations and practice change efforts in health care and other settings [31].
VA has used various methods over the last decade to achieve UCD for My HealtheVet; among them, the principal method has been a continuous, voluntary, and anonymous survey of end users. As a complement to periodic formal research studies [32,33], this ongoing assessment offers the advantage of rapid continuous feedback, which is part of a cyclical process for improvement that entails understanding users, eliciting their input, identifying changes or future design implications, deploying enhancements, and then obtaining feedback to evaluate these enhancements. This method enables VA to obtain ongoing direct feedback from veterans, which can then be leveraged to improve the patient experience.
About the Department of Veterans Affairs’ Patient Portal
VA is the largest integrated health care system in the United States and has been a pioneer in enabling patients to access and download their VA medical record data using the Blue Button feature [34,35]. This includes OpenNotes, which are known in My HealtheVet as VA Notes and contain both clinical and mental health providers’ notes [36]. The My HealtheVet patient portal [37] is tethered to the VA EHR and provides a suite of Web-based tools. Veterans self-register to create a basic account and can then self-enter information into their personal health record and access health education resources. VA patients who are matched by the system via the Master Veteran Index are automatically upgraded to an Advanced account and can request VA prescription refills. Patients who complete a one-time process of identity authentication (in person or Web-based) are upgraded to a Premium account and can then access all portal features, including access to health record information and Secure Messaging with VA health care professionals.
Use of My HealtheVet continues to grow. In fiscal year (FY)2017, portal user activity demonstrated significant increases compared with that in FY2016, including a 20.7% increase in Web-based prescription refills, a 33.9% increase in Secure Messaging exchanges between VA patients and their health care team, and a 38.7% increase in use of the VA Blue Button feature [27].
In this paper, we examine one of the implementation strategies that VA has used to foster adoption and sustained use of its patient portal over the last decade: an ongoing focus on UCD. This includes iterative use of survey and operational data with user interface redesign to meet the needs and preferences of veteran users. We describe the organization’s implementation strategy for agile UCD and present unique 10-year data on user adoption, characteristics, and utilization to demonstrate trends toward greater patient engagement and satisfaction. Following an initial analysis of portal users and their preferences in 2007 [38], we compared the characteristics of patient portal users one decade later and used a case study approach to present several examples of how user preferences and continuous feedback have informed the evolution of VA’s patient portal.
Methods
Since 2007, VA has used the ForeSee customer experience analytics (CXA) survey tool for the direct measurement of customer satisfaction and prioritization of enhancements. The CXA survey is a standardized method of measuring and monitoring customer satisfaction based on the American Customer Satisfaction Index [39]. The survey methodology uses a psychometric “voice-of-the-customer” technique to assess consumer drivers of satisfaction (look and feel, navigation, site information, site performance, and task processes) and prioritize areas of improvement. In the CXA model, scores are based on data from randomized voluntary Web-based surveys and are reported on a scale of 0 to 100, indicating less to more customer satisfaction. Multiple item measures are combined algorithmically to compile a satisfaction index each time an adequate quantity of data has been collected through completed surveys [40]. The survey tool for the My HealtheVet portal includes standard questions, to allow for trend analysis of core components such as overall satisfaction, and user experience of navigation. The inclusion of custom questions on an as-needed basis further enables the collection of rich data about user demographics, needs, and preferences to address specific and time-sensitive evaluation topics and to inform ongoing design and development efforts.
The CXA survey is conducted with all veterans using My HealtheVet and is, therefore, a nationwide sample of veteran My HealtheVet users. The survey is implemented on the My HealtheVet portal as a Web-based pop-up browser window inviting a random sample of site visitors to participate. A persistent cookie prevents site visitors who received the survey invitation from being invited again for 90 days. When visitors accept the invitation, the survey presents when they leave the site. The loyalty factor, currently 4 pages, ensures that respondents have experienced multiple pages on the site before being prompted to participate in the survey. The sampling percentage, set at 13% in FY2008 and later changed to 4% in FY2010 due to the large amount of data being collected and increasing survey completion rates, ensures that a minimum number of site visitors are surveyed in order to reduce respondent burden while enabling the collection of adequate data.
This paper presents selected analyses of the CXA survey data collected over a course of 10 years, including a comparison of data collected early in the implementation of My HealtheVet (FY2008) to more recent data (FY2017), to examine the characteristics of patient portal users and their preferences.
Data analysis is primarily descriptive and based on forced-choice responses. Analysis of open-ended comments includes a combination of traditional qualitative techniques [41,42] along with keyword clustering to group related comments for further analysis. A variety of strategies are used to then translate insights into iterative improvements, including ongoing data reviews, requirement elaboration, design sessions with key stakeholders, and review of user feedback after deployment of enhancements.
Results
Overview
We first present a recent summary of user demographics and characteristics and patterns of portal use and relevant comparisons to previous data. Following our case study approach, we then provide selected examples from the My HealtheVet evaluation program to illustrate how different assessments that capture the voice of the customer have directly informed the evolution of the portal and the addition of new functionality. For FY2008 (October 1, 2007-September 30, 2008), of the surveys presented to site visitors, 17.1% (100,069/585,039) were completed. For FY2017 (October 1, 2016-September 30, 2017), of the surveys presented to site visitors, 68.9% (100,555/146,023) were completed. As completion rates increased over the last decade, the sampling rate was reduced in FY2010 from 13% to 4% in order to minimize respondent burden.
User Demographics and Characteristics
Table 1 provides a comparison of user demographics and characteristics for all survey respondents in FY2017 and FY2008. In FY2017, 97% (97,538/100,555) of respondents were veterans compared with 93% (93,064/100,069) in FY2008. Respondents reported having completed higher levels of education, with 40% (39,990/99,974) being college graduates, completing some postgraduate school, or having a graduate or professional degree in FY2017 compared with 34% (732/2152) in FY2008. The proportion of male respondents increased slightly to 93% (90,507/97,319) in FY2017. In FY2017, respondents were generally older, with 64% (59,819/93,467) in the age range of 60-74 years compared with 47% (14,563/30,984) in FY2008; furthermore, 17% (15,889/93,467) of respondents in FY2017 were older than 75 years. This shift in age is also shown in Figure 1.
Table 1.
Type | FY2017 | FY2008 | |
Rolea, n (%) | 100,555 | 100,069 | |
|
Veteran | 97 | 93 |
|
Family member | 3 | 5 |
|
Veteran Service Organization | 1 | 1 |
|
National Guard or Reserve | 1 | N/A |
|
General public | 1 | <1 |
|
Other role | 1 | 1 |
|
VA employee | 1 | 1 |
|
Non-VA federal employee | <1 | 1 |
|
Caregiver (other than family) | <1 | N/A |
|
State or local government | <1 | N/A |
|
Active duty | <1 | <1 |
|
News Media | N/A | <1 |
Highest level of education, n (%) | 99,974 | 2154 | |
|
Did not complete high school | 3 | 2 |
|
High school graduate | 13 | 17 |
|
Some college or vocational school | 44 | 44 |
|
College graduate | 21 | 19 |
|
Some postgraduate school | 6 | 5 |
|
Graduate or professional degree | 13 | 10 |
Health conditionsa, n (%) | 22,795 | N/A | |
|
High blood pressure | 66 | N/A |
|
High cholesterol | 62 | N/A |
|
Arthritis of any kind | 60 | N/A |
|
Chronic pain | 47 | N/A |
|
Diabetes | 37 | N/A |
|
Stomach or gastrointestinal problems | 37 | N/A |
|
Heart problems | 37 | N/A |
|
Mental health or psychiatric condition | 34 | N/A |
|
Cancer of any kind | 29 | N/A |
|
Lung problems (including asthma) | 25 | N/A |
|
Neurological disorders | 13 | N/A |
|
Other | 12 | N/A |
|
Prefer not to answer | 2 | N/A |
Age, n (%) | 93,467 | 30,984 | |
|
Under 20 | <1 | <1 |
|
20-24 | <1 | <1 |
|
25-29 | <1 | <1 |
|
30-34 | <1 | 1 |
|
35-39 | 1 | 2 |
|
40-44 | 1 | 4 |
|
45-49 | 3 | 6 |
|
50-54 | 5 | 10 |
|
55-59 | 8 | 18 |
|
60-64 | 13 | 26 |
|
65-69 | 28 | 14 |
|
70-74 | 23 | 7 |
|
75-79 | 9 | 5 |
|
80-84 | 5 | 3 |
|
85 or older | 3 | 1 |
Gender, n (%) | 97,319 | 31,020 | |
|
Male | 93 | 91 |
|
Female | 7 | 9 |
Self-reported health status, n (%) | 98,007 | 40,315 | |
|
Excellent | 4 | 5 |
|
Very good | 21 | 18 |
|
Good | 41 | 38 |
|
Fair | 27 | 29 |
|
Poor | 7 | 10 |
Self-reported internet ability, n (%) | 53,725 | 55,658 | |
|
Beginner | 4 | 4 |
|
Intermediate | 36 | 29 |
|
Advanced | 60 | 68 |
aMultiple categories may be selected.
While 60% (60,042/100,069) of users in FY2008 reported their period of military service as the Vietnam War, this increased to 67% (67,372/100,555) of users in FY2017. Fewer users self-reported their internet ability as advanced in FY2017 (32,235/53,725, 60%) than in FY2008 (37,848/55,658, 68%), whereas more users reported it as intermediate in FY2017 (19,341/53,725, 36%) than in FY2008 (16,141/55,658, 29%). A greater proportion of respondents reported better health in FY2017, with 34% (33,323/98,007) reporting fair or poor health in FY2017 compared with 39% (15,723/40,315) in FY2008. Although the FY2008 survey did not ask users about health conditions, responses in FY2017 revealed a high prevalence of chronic conditions, including high blood pressure (15,045/22,795, 66%), high cholesterol (14,133/22,795, 62%), arthritis (13,677/22,795, 60%), chronic pain (10,714/22,795, 47%), diabetes (8434/22,795, 37%), stomach or gastrointestinal problems (8434/22,795, 37%), and heart problems (8434/22,795, 37%).
Portal Access Patterns and Usage
Table 2 provides a summary of survey respondents’ self-reported portal access patterns and usage. While the proportion of survey respondents who use VA health care services remained the same from FY2008 to FY2017 (96%), in FY2017 50% (47,066/94,132) of respondents stated that they also use a community non-VA provider. When asked about travel time to the nearest VA facility, 32% respondents reported it to be less than 30 minutes in both FY2008 and FY2017; however, a greater proportion noted fewer minutes of travel time in FY2017 than in FY2008. For example, 37% (19,902/53,788) respondents in FY2008 reported a travel time greater than 60 minutes to the nearest VA facility compared with 24% (12,075/50,313) in FY2017. The proportion of respondents who have a Premium account, offering them access to all portal services, increased significantly from 60% (56,884/94,806) in FY2008 to 77% (73,001/94,806) in FY2017. While a greater number of respondents were first time users in FY2008 (12,074/100,617, 12%) than in FY2017 (4022/100,555, 4%), respondents reported using the portal more frequently in FY2017, with 46% (45,255/100,555) using it about once a month and 29% (29,161/100,555) using it about once a week. When asked about the length of use in FY2017, 63% (63,349/100,5455) respondents reported having used My HealtheVet for more than 2 years.
Table 2.
Respondent Characteristic | FY2017 | FY2008 | |
Use VA health care services, n (%) | 98,007 | 29,528 | |
|
Yes | 96 | 96 |
|
No | 3 | 4 |
|
Not sure | 1 | N/A |
Use community non-VA providers, n (%) | 94,132 | N/A | |
|
Yes | 50 | N/A |
|
No | 47 | N/A |
|
Not sure | 3 | N/A |
Premium My HealtheVet accounta, n (%) | 94,806 | 100,617 | |
|
Yes | 77 | 60 |
|
No | 9 | 24 |
|
Not sure | 15 | 15 |
|
Not applicable | N/A | 1 |
Travel time to nearest VA facilitya, n (%) | 50,313 | 53,788 | |
|
Less than 30 min | 32 | 32 |
|
30-60 min | 43 | 32 |
|
61-90 min | 14 | 20 |
|
91 min to 2 h | 6 | 9 |
|
Over 2 h | 4 | 8 |
|
Not sure | N/A | 1 |
Frequency of use, n (%) | 100,555 | 100,617 | |
|
Daily or more than once a day | 5 | 5 |
|
About once a week | 29 | 25 |
|
About once a month | 46 | 49 |
|
About every 6 mo | 9 | 5 |
|
Less than every 6 mo | 4 | 3 |
|
First time | 4 | 12 |
|
Not sure or Do not recall | 2 | N/A |
Length of use, n (%) | 100,555 | N/A | |
|
Less than 6 mo | 9 | N/A |
|
6 mo-less than 1 y | 6 | N/A |
|
1-2 y | 19 | N/A |
|
More than 2 y | 63 | N/A |
|
Not sure or Do not recall | 3 | N/A |
aPercentages do not add to 100 due to rounding.
User Preferences and Responsive Design
In keeping with the UCD process, VA has used direct veteran feedback about preferences obtained via the CXA survey to shape the identification and prioritization of portal improvements. In this section, we describe how different types of user feedback have directly informed enhancements to the core components of the system including available features, interface design, content, policy, and prospective functional design of a new feature.
Additional Services Desired
UCD principles focus on identifying what features users consider to be essential. One survey question that has been crucial in getting feedback to prioritize portal enhancements over the last decade has been “What additional services would you like to see on My HealtheVet?” As shown in Table 3, additional services desired by users in FY2008 included the ability to view (79,892/92,160, 87%) or schedule (68,395/92,160, 74%) VA Appointments, access information from the VA medical record (67,714/92,160, 73%), and Web-based secure communication with my doctor (58,878/92,160, 64%). Each of these features was subsequently added to the portal (Table 4).
Table 3.
Service | n (%) | |
FY2017 | 88,308 | |
|
Schedule or change my VA appointments | 45,695 (52) |
|
View a list of my VA health care providers and their contact information | 38,489 (44) |
|
Check to determine if my different medications are safe taken together | 22,710 (26) |
|
View or pay my VA bills or copayments | 21,768 (25) |
|
Use a mobile app for My HealtheVet | 13,823 (16) |
|
Advance check-in for my VA clinic visits | 12,677 (14) |
|
Authorize sharing information with my Non-VA health care provider | 11,467 (13) |
|
Authorize sharing information with my VA health care team | 8851 (10) |
|
Authorize sharing information with other people (eg, family, caregiver) | 7584 (9) |
|
Other | 6573 (7) |
|
More Web-based educational programs | 5396 (6) |
|
Join a Web-based forum to discuss health issues with other veterans | 3831 (4) |
FY2008 | 92,160 | |
|
View my upcoming appointments | 79,892 (87) |
|
Schedule or change my appointments | 68,395 (74) |
|
Look at information in my VA medical record | 67,714 (73) |
|
Web-based, secure communication with my doctor | 58,878 (64) |
|
Checking that different medications I take are safe when used together | 45,986 (50) |
|
Reminders of preventive care I need (eg, shots, cancer screening) | 34,707 (38) |
|
Notification of new content or features on the site | 32,418 (35) |
|
Advance check-in for my VA clinic visits | 31,863 (35) |
|
Monthly email newsletter | 24,186 (26) |
|
Share information that I have stored in My HealtheVet with other people | 23,088 (25) |
|
Advanced directive (eg, living will, durable power of attorney) | 20,418 (22) |
|
Educational programs | 18,800 (20) |
|
Information about the quality of VA health care | 11,231 (12) |
|
Other | 8791 (10) |
Table 4.
Year | Milestone |
1999 |
|
2003 |
|
2004 |
|
2005 |
|
2006 |
|
2007 |
|
2008 |
|
2009 |
|
2010 |
|
2011 |
|
2012 |
|
2013 |
|
2014 |
|
2015 |
|
2016 |
|
2017 |
|
Secure Messaging implementation began in 2008, which enabled secure Web-based communication with VA health care teams, with the full national release to all VA primary care providers in 2012. Veterans could then also use Secure Messaging to request VA Appointments. The ability to view upcoming VA Appointments was deployed in 2011, with appointment email reminders added in 2015. Building on early access to VA Medication History, VA incrementally expanded the types of information from the VA medical record available in My HealtheVet, for example, preventative Wellness Reminders (2009), VA Chemistry or Hematology Lab Results (2011), VA Immunizations (2012), VA Notes including mental health notes (2013), a more comprehensive Medication List that includes patient-reported non-VA medications (2016), Surgical and Clinical Procedure Notes (2017), and VA Medical Images and Reports (2017).
Additional services desired by users in FY2017 included the ability to schedule or change VA Appointments directly (52%), a list of health care providers and their contact information (44%), a tool to determine whether different medications are safe when taken together (26%), and the ability to view and pay VA bills or copayments (25%). The ability to schedule or change VA Appointments directly was piloted in FY2017 and is being rolled out to all VA facilities in FY2018. The enhanced VA Health Summary (2017) provides VA patients with a list of their primary health care providers, which will be expanded in FY2018 to include their contact information. Although VA has not yet invested in the development of tools to check medications for potential interactions; this enhancement is being given further consideration in FY2018. In addition, the ability to view a VA Patient Statement and remit payment is also being developed and scheduled for pilot testing in FY2018.
Patient-Identified Main Improvements
In addition to eliciting user feedback on additional services desired, the CXA survey also invites open-ended comments in response to the question: “What is the main improvement that you would suggest for the My HealtheVet website?” Below we offer examples of how these comments have led to user-directed improvements.
With the expansion of Lab and Test Results and the addition of VA Notes in January 2013, one theme that surfaced in the ensuing months was veterans’ desire for more timely access to this information. These comments were crucial in driving VA policy change to reduce the hold period for lab results and progress notes from 7 calendar days after verification to 3 calendar days. This policy change was implemented in June 2013.
To complement the prioritization of known desired additional services by users, open-ended main improvement comments also allow veterans to suggest needed functional enhancements in their own words. In October 2013, thematic analysis of free-text comments identified the need for multiple functional enhancements including the ability to track delivery of the filled prescriptions, the desire to be notified before automatic log out when the user session was nearing time-out, and the need for improved navigation to complete common tasks. The ability to track delivery of mailed prescriptions by opting-in to receive an email notification was deployed in 2015. Other functional enhancements (session time-out warning, improved navigation, and reduced number of steps to complete common tasks) became core requirements for a major website redesign project. The session time-out warning and ability to extend the session time was deployed as VA migrated to a content management system in October 2016. The incremental deployment of website redesign in October 2016 and September 2017 was significantly informed by veteran main improvement comments:
Publishing labs and notes within 24 hours of a lab or health visit. Waiting a week for lab results, or a week for Dr and nurse notes is absurd, given that the health problem is “right now,” not right now + seven days, especially when Dr's notes are also instructions for post visit procedures, such as when and how much meds to take, or “If it hasn't improved in three days” see me. Not everyone is “present” at the end of a visit due mostly to anxieties surrounding the visit.
Why does it take so long for results of lab work, radiology, notes from Drs to show up? It can take a week or more. My Dr already called with the results yesterday but I still can’t see it here. Also saw GI Dr 3 days ago and not notes here. I wish we could access our information sooner.
Eliminate wait period to view VA Notes, Results, etc. Once the provider has entered the note or viewed the results of test, they should be made available for viewing by the veteran.
Aligning Content With Patient-Suggested Topics of Interest
Periodically, an open-ended question is added to the survey asking users about topics of interest for portal content, such as feature articles to ensure that content is directly aligned with veterans’ needs and preferences. An editorial calendar is created to provide articles throughout the year focused on these topics. Topics are also highlighted in a subscription-based monthly electronic newsletter that was developed in 2015 as a user-desired additional service (see Table 3), with more than 500,000 subscribers in FY2017. Examples of topic clusters for August 2009 (N=1809), August 2013 (N=3300), and August 2017 (N=1189) are shown in Figure 2. In 2017, the top user-suggested topics included Health (“general health, age concerns, pre-existing medical issues”), Diabetes (“articles on diabetes and feet or hand or finger neuropathy”), Care (“information on special health care programs for specific conditions”), and Agent Orange (“need more information on Agent Orange exposure and health issues”).
Prospective Functional Design of a New Feature
Veteran feedback has also driven the functional design of new features. One feature that is currently being developed is the ability for the users to assign a delegate who can access their account. For example, a spouse or caregiver who may be assisting a veteran patient in managing his or her health. In October 2014, VA convened key stakeholders and subject matter experts to define the business requirements for this feature; however, there was a lack of consensus on a key functional requirement: whether “read-only” access should allow or restrict a delegate’s ability to also print and download data. Using the CXA survey, veterans were asked “If you approve read access for another person to help you manage your personal health information, what would you want that person to be able to do?”
Of those veterans with a preference to delegate read access to another person, 75% (8194/11,006) would want such access to include print and download capability, while 14% (1541/11,006) would want a delegate to be able to read or view their information on the screen, but not print or download it. With this direct veteran input on desired functional design, requirements were prospectively aligned with user preferences. Data were also collected to assess patient preferences regarding delegating access to health information [43], use of My HealtheVet to transfer information [44], how veterans with non-VA providers use the Blue Button feature to share information with their non-VA providers [45], and the veteran experiences with access to their VA Notes [36].
Website Redesign and Satisfaction Trends
Analysis of CXA data over the course of the last decade has been an integral part of the recent My HealtheVet website redesign initiative by enabling a deeper understanding of the end users and their tasks and goals, in keeping with UCD principles. As shown in Table 5, while 75% (75,241/100,617), 24% (23,923/100,617), and 18% (17,899/100,617) users in FY2008 accessed the portal to request a prescription refill, view their medication history, and look up information about a medication, respectively, user goals and tasks in FY2017 have shifted and expanded. Although prescription refill requests remained a predominant task (53,193/100,555, 53%), users also accessed the portal to view their VA Appointments (38,664/100,555, 38%), communicate with their health care team using Secure Messaging (28,952/100,555, 29%), track the delivery status of their medication refills (23,884/100,555, 24%), view their lab or test results (19,382/100,555, 19%), and access their VA health records (11,966/100,555, 12%). An important goal of the culminating website redesign was to improve navigation and usability for these specific core features, and the overall customer satisfaction index score was used as a performance indicator.
Table 5.
Reason for visit or goal trying to accomplisha | FY2017 (N=100,555), n (%) | FY2008 (N=100,617), n (%) |
Request a prescription refill | 53,193 (53) | 75,241 (75) |
View my VA Appointments | 38,664 (38) | N/A |
Use Secure Messaging to communicate with my VA health care team | 28,952 (29) | N/A |
Track the status of my prescription refill delivery | 27,516 (27) | N/A |
View my medication history | 23,884 (24) | 23,923 (24) |
View my lab or other test results | 19,382 (19) | N/A |
Access my VA health records or Blue Button or VA Health Summary | 11,966 (12) | N/A |
View my VA Notes (written by my health care team) | 11,058 (11) | N/A |
Look up information about a health condition or medication | 9393 (9) | N/A |
Learn more about features that are available | 9149 (9) | N/A |
Look up information about a medication | N/A | 17,899 (18) |
Find information about VA benefits | 9111 (9) | 6246 (6) |
Enter or keep track of personal information | 5695 (6) | 14,507 (14) |
Other | 5101 (5) | 9198 (9) |
Enter or keep track of personal health care information (eg, blood pressure) | 3202 (3) | 13,125 (13) |
Use the Veterans Health Library (Research a health condition) | 2648 (3) | 6367 (6) |
Enter information about my non-VA medications or supplements | 2288 (2) | N/A |
Find a VA facility | 1646 (2) | 2206 (2) |
Complete a HealtheLiving Assessment | 1533 (2) | N/A |
aMultiple categories may be selected.
Historical customer satisfaction trends are shown in Figure 3. From October 2007 to October 2015, the aggregate average CXA score was 74, based on 945,480 completed surveys. The average for the 12 months that followed was stable at 76 (N=139,934). While multiple factors impacted customer satisfaction over the last decade, including a period of system performance issues in 2014 that was resolved by improving system architecture, the overall trend toward greater customer satisfaction is evident.
In October 2016, as part of an incremental website redesign, a dashboard was added to the portal home page to enhance user access to the core features (Figure 4).
As anticipated, the introduction of changes to the website resulted in an initial decrease in satisfaction (72), followed by satisfaction recovery (75), and subsequent increase to a new high of 79 (Figure 5). A similar pattern was observed with the deployment of additional website redesign changes in September 2017. Satisfaction initially decreased (77), but then recovered to previous levels (79). Satisfaction continued to increase in January 2018 (80).
Discussion
Principal Findings
The literature on adoption and use of patient portals highlights the need for health care organizations to employ UCD approaches to ensure that portals align with end users’ characteristics, needs, preferences, and goals, and, ultimately, help advance portal implementation. In its commitment to UCD, one method that VA has used to accomplish the above is a continuous survey to elicit a direct feedback from a random sample of veterans who use VA’s patient portal, My HealtheVet. In combination with other methods, such as targeted research studies, the CXA survey has enabled a deeper understanding of portal users and directly informed changes in portal features, functions, policies, and processes. By incorporating the results of this systematic evaluation of the user experience into the portal redesign, VA aims to continue to enhance the ability of My HealtheVet to engage and activate veterans in managing their health.
Patient Portal Users
This study compared the characteristics and behaviors of users during the early period of patient portal implementation, 2008, with that of later adopters. This provided a trajectory of how portal use has evolved over a decade. Many aspects remained stable, while others showed clear trends toward portal adoption by populations believed less likely to use patient-facing health technologies. While only 13.4% (89,780/670,000) portal users in FY2008 were VA patients with a Premium account, by FY2017, this increased to 62.5% (2.5 million/4 million) users. Despite early assumptions about older users not adopting and using patient portals [46,47], leading to a gray digital divide [48], the VA experience reveals an increasingly elderly population of users. Within the veteran population, research has shown that VA patients tend to be older and more socioeconomically disadvantaged than veterans who do not rely on VA for care [49]. Although the survey indicates that the majority of users have one or more chronic health conditions and access the portal with increasing frequency, the survey results also suggest a trend toward those with less internet ability and better health also accessing the patient portal. This trend may be a result of the portal expanding the types of transactional services that users find convenient, based on direct veteran input. It also suggests that the portal is engaging a broader segment of the veteran population. Although the proportion of female veterans responding to the survey decreased slightly in FY2017 (from 9% to 7%), the overall population of female veterans was estimated to be 9.4% in 2015. However, only 22.4% used VA health care services [50], which is a key driver for accessing the patient portal. Portal users in FY2017 also tended to have completed higher levels of education than those in FY2008. This may be reflective of changes in the veteran population overall, with the enhanced provision of educational support programs for separating service members. Given that half of the survey respondents in FY2017 reported that they also use community non-VA care providers, VA will need to continue to develop tools that enable effective information sharing across settings of care. Portal functions that support consumer-mediated health information exchange are currently in early field testing [51]. These patient portal user trends align with similar trends for the VA patient population overall in terms of gender (91% male), age (median age of male VA patients, 64 years), and increasing use of VA education benefits [52].
Incremental Portal Redesign
Based on user self-report about goals and tasks, a significant redesign of the website was undertaken to enhance navigation to the features aligned with the most common user tasks and to decrease the number of steps to accomplish these. After an initial period of satisfaction decline, anticipated due to the phenomenon of change aversion [53], the satisfaction index recovered and increased. Once users adjusted to the change, they were more satisfied with the new design as measured using the CXA satisfaction index. Looking ahead, there are additional improvements and enhancements that will be important to address.
Limitations
It is important to note that the results of the CXA survey reflect the characteristics and perspectives of a random sample of portal users who are invited and opt to participate in the survey and may not be fully generalizable to the larger population. More broadly, the respondent sample represents patient portal users; therefore, other methods are also needed to elicit input from veterans who are not portal users to understand their characteristics and preferences and identify barriers that may exist to system access and use. VA is in the process of adding questions to its patient experience survey, administered to veterans who had a recent medical encounter, to help fill this gap, and ongoing research about veteran preferences for digital tools and services provides complementary insights [32,33]. There may also be data that were not collected in the survey that could be important. Since the survey is anonymous, there is no opportunity to follow up with respondents for more information or clarification. Despite limitations inherent to an anonymous survey, it has the benefit of enabling a continuous flow of direct feedback. While the findings from our case study may not be fully generalizable to other patient populations, the principle of using agile approaches to employ UCD has potential to be a promising implementation strategy for other health care organizations.
Conclusions
By leveraging UCD principles, VA has continued to enhance its patient portal and supported its continued implementation, achieving significant growth in adoption and use over the last decade. While quantitative and qualitative research studies are an important component of patient portal evaluation, more agile methods are also needed to complement formal research efforts. As illustrated through this case study, we have found the ongoing administration of a continuous voluntary Web-based survey as an efficient and effective way to capture veteran’s voices about who they are, how they use the patient portal, what improvements are needed, and what additional services are desired. This approach, together with others intended to explore the perspectives of veterans who are not portal users, will help ensure that VA’s health information technology services are developed and enhanced to optimize the benefits to all VA patients. With impending changes to VA’s EHR platform, capturing veteran’s voices is more crucial than ever. More broadly, developing patient portals as an effective patient engagement strategy will require that UCD principles are employed to foster adoption and sustained use. In an era of finite resources, leveraging the “voice-of-the-customer” techniques helps ensure that the portal continues to meet patients’ needs in ways that enhance full participation in their own health care.
Acknowledgments
The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. KMN was affiliated with the Department of Veterans Affairs during the time of this study but recently retired and is now an independent consultant specializing in Sociology, Communication, and Consumer Health Informatics.
Abbreviations
- CXA
customer experience analytics
- EHR
electronic health record
- KP
Kaiser Permanente
- UCD
user-centered design
- VA
Department of Veterans Affairs
Footnotes
Authors' Contributions: KMN was responsible for data collection. All authors (KMN, CLT, DMK, and TPH) contributed to the drafting of the manuscript and revising critically for intellectual content.
Conflicts of Interest: None declared.
References
- 1.Barello S, Graffigna G, Vegni E. Patient engagement as an emerging challenge for healthcare services: mapping the literature. Nurs Res Pract. 2012;2012:905934. doi: 10.1155/2012/905934. doi: 10.1155/2012/905934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Gephart S, Effken J. Online Journal of Nursing Informatics (OJNI) 2013. [2018-03-15]. Using Health Information Technology to Engage Patients in their Care http://ojni.org/issues/?p=2848 .
- 3.James J, Hibbard J, Agres T. Health Affairs. 2013. [2018-06-27]. Health policy brief: patient engagement https://www.healthaffairs.org/do/10.1377/hpb20130214.898775/full/
- 4.Wilkins S. KevinMD.com. 2012. Jan 27, [2018-03-14]. Patient engagement is the holy grail of health care https://www.kevinmd.com/blog/2012/01/patient-engagement-holy-grail-health-care.html .
- 5.Chase D. Forbes. 2012. Sep, [2018-03-14]. Patient engagement is the blockbuster drug of the century https://www.forbes.com/sites/davechase/2012/09/09/patient-engagement-is-the-blockbuster-drug-of-the-century/
- 6.Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, DeVito DA. Patient Portals as a Tool for Health Care Engagement: A Mixed-Method Study of Older Adults With Varying Levels of Health Literacy and Prior Patient Portal Use. J Med Internet Res. 2017 Mar 30;19(3):e99. doi: 10.2196/jmir.7099. http://www.jmir.org/2017/3/e99/ v19i3e99 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Nazi K, Hogan T, Woods S. Consumer health informatics: engaging and empowering patients and families. In: Finnell J, Dixon B, editors. Clinical Informatics Study Guide. Switzerland: Springer International Publishing; 2016. pp. 459–500. [Google Scholar]
- 8.Ricciardi L, Mostashari F, Murphy J, Daniel JG, Siminerio EP. A national action plan to support consumer engagement via e-health. Health Aff (Millwood) 2013 Feb;32(2):376–384. doi: 10.1377/hlthaff.2012.1216.32/2/376 [DOI] [PubMed] [Google Scholar]
- 9.Sorondo B, Allen A, Fathima S, Bayleran J, Sabbagh I. Patient Portal as a Tool for Enhancing Patient Experience and Improving Quality of Care in Primary Care Practices. EGEMS (Wash DC) 2016;4(1):1262. doi: 10.13063/2327-9214.1262. http://europepmc.org/abstract/MED/28203611 .egems1262 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fraccaro P, Vigo M, Balatsoukas P, Buchan IE, Peek N, van DVSN. Patient Portal Adoption Rates: A Systematic Literature Review and Meta-Analysis. Stud Health Technol Inform. 2017;245:79–83. [PubMed] [Google Scholar]
- 11.Goldzweig CL, Orshansky G, Paige NM, Towfigh AA, Haggstrom DA, Miake-Lye I, Beroes JM, Shekelle PG. Electronic patient portals: evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med. 2013 Nov 19;159(10):677–687. doi: 10.7326/0003-4819-159-10-201311190-00006.1770672 [DOI] [PubMed] [Google Scholar]
- 12.Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res. 2015;17(6):e148. doi: 10.2196/jmir.4255. http://www.jmir.org/2015/6/e148/ v17i6e148 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient Portals: An Underused Resource for Improving Patient Engagement. Chest. 2016 Jan;149(1):272–277. doi: 10.1378/chest.14-2559.S0012-3692(15)00110-5 [DOI] [PubMed] [Google Scholar]
- 14.Centers for Medicare & Medicaid Services (CMS) HHS Federal Register. 2015. Oct 16, [2018-06-28]. Medicare and Medicaid programs: electronic health record incentive programs. Final rule https://www.gpo.gov/fdsys/pkg/FR-2015-10-16/pdf/2015-25595.pdf .
- 15.Henry SL, Shen E, Ahuja A, Gould MK, Kanter MH. The Online Personal Action Plan: A Tool to Transform Patient-Enabled Preventive and Chronic Care. Am J Prev Med. 2016 Jul;51(1):71–77. doi: 10.1016/j.amepre.2015.11.014.S0749-3797(15)00780-1 [DOI] [PubMed] [Google Scholar]
- 16.Lau M, Campbell H, Tang T, Thompson DJS, Elliott T. Impact of patient use of an online patient portal on diabetes outcomes. Can J Diabetes. 2014 Feb;38(1):17–21. doi: 10.1016/j.jcjd.2013.10.005.S1499-2671(13)01356-7 [DOI] [PubMed] [Google Scholar]
- 17.Nazi KM, Hogan TP, McInnes DK, Woods SS, Graham G. Evaluating patient access to Electronic Health Records: results from a survey of veterans. Med Care. 2013 Mar;51(3 Suppl 1):S52–S56. doi: 10.1097/MLR.0b013e31827808db.00005650-201303001-00011 [DOI] [PubMed] [Google Scholar]
- 18.Sarkar U, Lyles CR, Parker MM, Allen J, Nguyen R, Moffet HH, Schillinger D, Karter AJ. Use of the refill function through an online patient portal is associated with improved adherence to statins in an integrated health system. Med Care. 2014 Mar;52(3):194–201. doi: 10.1097/MLR.0000000000000069. http://europepmc.org/abstract/MED/24374412 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wade-Vuturo AE, Mayberry LS, Osborn CY. Secure messaging and diabetes management: experiences and perspectives of patient portal users. J Am Med Inform Assoc. 2013 May 01;20(3):519–525. doi: 10.1136/amiajnl-2012-001253. http://europepmc.org/abstract/MED/23242764 .amiajnl-2012-001253 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.OpenNotes. [2018-05-12]. https://www.opennotes.org/
- 21.Delbanco T, Walker J, Bell SK, Darer JD, Elmore JG, Farag N, Feldman HJ, Mejilla R, Ngo L, Ralston JD, Ross SE, Trivedi N, Vodicka E, Leveille SG. Inviting patients to read their doctors' notes: a quasi-experimental study and a look ahead. Ann Intern Med. 2012 Oct 02;157(7):461–470. doi: 10.7326/0003-4819-157-7-201210020-00002. http://europepmc.org/abstract/MED/23027317 .1363511 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry. 2016;38:89–93. doi: 10.1016/j.genhosppsych.2015.08.001.S0163-8343(15)00191-7 [DOI] [PubMed] [Google Scholar]
- 23.Gerard M, Fossa A, Folcarelli PH, Walker J, Bell SK. What Patients Value About Reading Visit Notes: A Qualitative Inquiry of Patient Experiences With Their Health Information. J Med Internet Res. 2017 Jul 14;19(7):e237. doi: 10.2196/jmir.7212. http://www.jmir.org/2017/7/e237/ v19i7e237 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Leveille SG, Walker J, Ralston JD, Ross SE, Elmore JG, Delbanco T. Evaluating the impact of patients' online access to doctors' visit notes: designing and executing the OpenNotes project. BMC Med Inform Decis Mak. 2012 Apr 13;12:32. doi: 10.1186/1472-6947-12-32. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-12-32 .1472-6947-12-32 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Wright E, Darer J, Tang X, Thompson J, Tusing L, Fossa A, Delbanco T, Ngo L, Walker J. Sharing Physician Notes Through an Electronic Portal is Associated With Improved Medication Adherence: Quasi-Experimental Study. J Med Internet Res. 2015 Oct 08;17(10):e226. doi: 10.2196/jmir.4872. http://www.jmir.org/2015/10/e226/ v17i10e226 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Garrido T, Raymon B, Wheatley B. Health Affairs. 2016. Apr, [2018-06-28]. Lessons learned from more than a decade in patient portals https://www.healthaffairs.org/do/10.1377/hblog20160407.054362/full/
- 27.Veterans and Consumers Health Informatics Office Department of Veterans Affairs. 2018. [2018-06-28]. https://connectedcare.va.gov/
- 28.Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Bainbridge W, editor. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications; 2004. [Google Scholar]
- 29.Katz-Haaz R. Ten guidelines for user-centered web design. Usability Interface. 1998;5(1):1–3. [Google Scholar]
- 30.Morales RM, Casper G, Brennan PF. Patient-centered design. The potential of user-centered design in personal health records. J AHIMA. 2007 Apr;78(4):44–46. [PubMed] [Google Scholar]
- 31.Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 Feb 12;10:21. doi: 10.1186/s13012-015-0209-1. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0209-1 .10.1186/s13012-015-0209-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Haun JN, Chavez M, Nazi K, Antinori N, Melillo C, Cotner BA, Hathaway W, Cook A, Wilck N, Noonan A. Veterans' Preferences for Exchanging Information Using Veterans Affairs Health Information Technologies: Focus Group Results and Modeling Simulations. J Med Internet Res. 2017 Oct 23;19(10):e359. doi: 10.2196/jmir.8614. http://www.jmir.org/2017/10/e359/ v19i10e359 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Haun JN, Chavez M, Nazi KM, Antinori N. Developing a Health Information Technology Systems Matrix: A Qualitative Participatory Approach. J Med Internet Res. 2016 Oct 06;18(10):e266. doi: 10.2196/jmir.6499. http://www.jmir.org/2016/10/e266/ v18i10e266 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Hogan TP, Nazi KM, Luger TM, Amante DJ, Smith BM, Barker A, Shimada SL, Volkman JE, Garvin L, Simon SR, Houston TK. Technology-assisted patient access to clinical information: an evaluation framework for blue button. JMIR Res Protoc. 2014;3(1):e18. doi: 10.2196/resprot.3290. http://www.researchprotocols.org/2014/1/e18 .v3i1e18 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Nazi K. VAntage Point. 2010. Dec 08, [2018-03-14]. VA’s Blue Button: Empowering People with Their Data https://www.blogs.va.gov/VAntage/866/vas-blue-button-empowering-people-with-their-data/
- 36.Nazi KM, Turvey CL, Klein DM, Hogan TP, Woods SS. VA OpenNotes: exploring the experiences of early patient adopters with access to clinical notes. J Am Med Inform Assoc. 2015 Mar;22(2):380–389. doi: 10.1136/amiajnl-2014-003144.amiajnl-2014-003144 [DOI] [PubMed] [Google Scholar]
- 37.My HealtheVet. [2018-05-12]. https://www.myhealth.va.gov .
- 38.Nazi KM. Veterans' voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users' characteristics, needs, and preferences. J Am Med Inform Assoc. 2010;17(2):203–211. doi: 10.1136/jamia.2009.000240. http://europepmc.org/abstract/MED/20190065 .17/2/203 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Wood FB, Siegel ER, Feldman S, Love CB, Rodrigues D, Malamud M, Lagana M, Crafts J. Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey. J Med Internet Res. 2008 Feb 15;10(1):e4. doi: 10.2196/jmir.944. http://www.jmir.org/2008/1/e4/ v10i1e4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Fornell C, Johnson MD, Anderson EW, Cha J, Bryant BE. Cornell University. 1996. Oct, [2018-03-14]. The American Customer Satisfaction Index: nature, purpose, and findings https://scholarship.sha.cornell.edu/cgi/viewcontent.cgi?article=1427&context=articles .
- 41.Corbin J, Strauss A. Basics of Qualitative Research. Thousand Oaks: Sage Publications; 2008. Techniques and Procedures for Developing Grounded Theory. [Google Scholar]
- 42.Green J, Thorogood N. Qualitative Methods for Health Research. Thousand Oaks: Sage Publications; 2009. [Google Scholar]
- 43.Zulman DM, Nazi KM, Turvey CL, Wagner TH, Woods SS, An LC. Patient interest in sharing personal health record information: a web-based survey. Ann Intern Med. 2011 Dec 20;155(12):805–810. doi: 10.7326/0003-4819-155-12-201112200-00002.155/12/805 [DOI] [PubMed] [Google Scholar]
- 44.Turvey C, Zulman D, Nazi K, Wakefield B, Woods S, Hogan T, Weaver F, McInnes K. Transfer of information from personal health records: a survey of veterans using My HealtheVet. Telemed J E Health. 2012 Mar;18(2):109–14. doi: 10.1089/tmj.2011.0109. [DOI] [PubMed] [Google Scholar]
- 45.Turvey C, Klein D, Fix G, Hogan TP, Woods S, Simon SR, Charlton M, Vaughan-Sarrazin M, Zulman DM, Dindo L, Wakefield B, Graham G, Nazi K. Blue Button use by patients to access and share health record information using the Department of Veterans Affairs' online patient portal. J Am Med Inform Assoc. 2014;21(4):657–663. doi: 10.1136/amiajnl-2014-002723.amiajnl-2014-002723 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc. 2015 Jul;22(4):888–895. doi: 10.1093/jamia/ocv025. http://europepmc.org/abstract/MED/25914099 .ocv025 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The Digital Divide and Patient Portals: Internet Access Explained Differences in Patient Portal Use for Secure Messaging by Age, Race, and Income. Med Care. 2016 Dec;54(8):772–779. doi: 10.1097/MLR.0000000000000560. [DOI] [PubMed] [Google Scholar]
- 48.Morris A. E-literacy and the grey digital divide: a review with recommendations. JIL. 2007 Dec 13;1(3):13–28. doi: 10.11645/1.3.14. [DOI] [Google Scholar]
- 49.RAND Corporation Centers for Medicare and Medicaid (CMS) 2015. Sep 01, [2018-03-14]. A product of the CMS Alliance to modernize healthcare federally funded research and development center https://www.va.gov/opa/choiceact/documents/assessments/Assessment_A_Demographics.pdf .
- 50.National Center for Veterans Analysis and Statistics Department of Veterans Affairs. 2017. [2018-03-14]. Women Veterans Report: the past, present, and future of Women Veterans https://www.va.gov/vetdata/docs/SpecialReports/Women_Veterans_2015_Final.pdf .
- 51.Williams C, Mostashari F, Mertz K, Hogin E, Atwal P. From the Office of the National Coordinator: the strategy for advancing the exchange of health information. Health Aff (Millwood) 2012 Mar;31(3):527–536. doi: 10.1377/hlthaff.2011.1314. http://content.healthaffairs.org/cgi/pmidlookup?view=long&pmid=22392663 .31/3/527 [DOI] [PubMed] [Google Scholar]
- 52.National Center for Veterans Analysis and Statistics Department of Veterans Affairs. 2017. [2018-06-28]. VA Utilization Profile https://www.va.gov/vetdata/docs/Quickfacts/VA_Utilization_Profile.pdf .
- 53.Sedley A. GV Library. 2012. Apr 23, [2018-03-14]. Change aversion: why users hate what you launched (and what to do about it) https://library.gv.com/change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766 .