Qualitative and Quantitative Analysis of Patients' Perceptions of the Patient Portal Experience with OpenNotes

Abstract

Background  Access to medical encounter notes (OpenNotes) is believed to empower patients and improve the quality and safety of care. The impact of such access is not well understood beyond select health care systems and notes from primary care providers.

Objectives  This article analyzes patients' perceptions about the patient portal experience with access to primary care and specialist's notes and evaluates free-text comments as an improvement opportunity.

Materials and Methods  Patients at an academic health care system who accessed the patient portal from February 2016 to May 2016 were provided a link to complete a 15-item online survey. Those who had viewed at least one note were asked about patient characteristics, frequency of note access, note usefulness, note understanding, and if any action was taken after accessing the note. Free-text comments were associated with nine questions which were analyzed using qualitative methods.

Results  A total of 23% (1,487/6,439) of patients who viewed the survey in the portal, participated. Seventy-six percent (1,126/1,487) knew that the notes were available on the portal, and of those, 957 had viewed at least one note to continue the survey. Ninety percent of those were older than 30 years of age, and 90% had some college education. The majority (83%) thought OpenNotes helped them take better care of themselves, without increasing worry (94%) or contacting the physician after reading the note (91%). The qualitative analysis of free-text responses demonstrated multiple positive and negative themes, and they were analyzed for potential improvement opportunities.

Conclusion  Our survey confirms that patients who choose to access their primary care and specialists' online medical records perceive benefits of OpenNotes. Additionally, the qualitative analysis of comments revealed positive benefits and several potential patient portal improvement opportunities which could inform implementation of OpenNotes at other health systems.

Background and Significance

In 2001, the Institute of Medicine (IOM; now The National Academy of Medicine) published “Crossing the Quality Chasm: A New Health System for the 21st Century” that espoused the need for “patient-centered care” defined as care that is “respectful of and responsive to individual patient preferences, needs, and values, and ensuring that patient values guide all clinical decisions.” 1 This philosophy was the beginning of the patient engagement movement by the U.S. health care system.

Over the past several decades much has happened to support patient engagement. Not only have multiple organizations such as the IOM endorsed this movement, but the emergence of ubiquitous electronic health records (EHR) and widespread patient portal availability have encouraged patient access to medical records. Additionally, the Health Insurance Portability and Accountability Act of 1996 stated the patients have the right to inspect, review, and receive copies of their medical records. 2

In 2010, three medical systems (Beth Israel Deaconess Medical Center, Geisinger Health System, and Harborview Medical Center) launched the OpenNotes study that involved 105 primary care physicians (PCPs) and approximately 20,000 patients. For the first time, patients were given access to outpatient physician encounter notes via the patient portal, if their PCP agreed to provide them access. Results were published in 2012 and demonstrated that patients did take the time to read their notes, were not more anxious after reading their notes, and were more engaged in the control of their health. Importantly, physicians did not experience an increased workload, and all suggested that the study be continued. 3 Following the OpenNotes initial study, the participating institutions have expanded OpenNotes to specialty clinics. 4 Since 2010, over 27 million patients have had access to physician notes in the United States because of the OpenNotes initiative and the movement has been extended internationally. 5

In 2017, the National Cancer Institute (NCI) conducted the Health Information Trend Survey that analyzed national patient access to online medical records. The survey reported that 52% of patients surveyed were offered access to their online medical record and of that 53% accessed it at least once in the past year. Eighty-two percent found access easy to understand and useful. Laboratory results were most widely available at 92%, while clinical notes were found by only 51%. 6

Following the original OpenNotes article, multiple studies have been reported, but primarily from the same three health care systems and from the perspective of PCPs and their patients. This article will report the results of a large-scale implementation of OpenNotes consisting of patients under the care of PCPs and specialists, not related to the initial three institutions.

Objectives

1. To summarize patients' perceptions about the patient portal experience, to include access to primary care and specialist's notes (OpenNotes).

2. To compare patients' perceptions in this study with prior OpenNotes studies.

3. To evaluate free-text qualitative comments as an improvement opportunity for an academic health care system.

Materials and Methods

Setting

Virginia Commonwealth University Health System (VCUHS) is an urban health care system located in Richmond, Virginia, United States. VCUHS is staffed by 750 physicians in 200 specialties. The tertiary health care system includes a level 1 trauma center and an NCI-designated cancer center. In 2017, VCUHS had over 700,000 outpatient visits by more than 165,000 patients. 7

The Intervention

OpenNotes concept was presented to the medical directors in January 2015 and request for comments by outpatient medical staff followed. OpenNotes was adopted and initiated in June 2015.

Survey Instrument

Postinitiation of OpenNotes, a panel of experts convened in 2015 to determine how to study portal users of OpenNotes. Most of the questions developed followed themes from the initial OpenNotes study. The survey instrument selected was SurveyMonkey, 8 and data were collected from February 24, 2016 to May 2, 2016. Table 1 lists the questions asked in the survey. Because OpenNotes is part of the patient portal, survey questions apply to both. Questions 1 to 11 and 13 to 14 were close-ended questions, whereas question 12 was open-ended.

Table 1. Survey questions.

Question	Answer choices
Q1 Did you know you could see your visit (outpatient appointment) notes from your provider in the portal?	Yes, No
Q2 Have you looked at any of your visit notes in the portal?	Yes, No
Q3 How often have you looked at your notes?	Once after every visit, Many times after a visit, Only if I have questions about my visit, Other
Q4 Do you find them helpful?	Always, Sometimes, Never
Q5 What have you found to be helpful in the note (select all that apply)?	What we discussed, Medications, When I should come back, Test results, Other
Q6 Have you contacted your provider after reading a note?	Yes, No
Q7 Does reading your note change how often you contact your provider?	I now contact more, I now contact less, I contact about the same
Q8 Do you ever contact your provider to make a correction to the note?	Yes, No
Q9 Did you think seeing your note helps you take better care of yourself?	Yes, No
Q10 How much of what was in the note do you understand?	All of it, Most of it, Some of it, None of it
Q11 How does reading a note affect how worried you are about something?	Makes me less worried, Makes me more worried, No change
Q12 Is there anything else you would like to tell us about the ability to see your notes in the portal?	Free text
Q13 How old are you	18–29, 30–59, 60 or older, Other
Q14 How much education have you had?	Less than 12 th grade, GED, Some college, College graduate, Graduate degree
Q15 Are you ok if we use your answers for research and publication?	Yes, No

Abbreviation: GED, General Educational Development.

Patient Recruitment

The Web-based survey was made available during the survey period (February 24, 2016–May 2, 2016) to everyone registered to the patient portal (56,000). Patients could participate in the survey only if they logged into the portal. Twenty-three percent (1,487/6,239) of those who logged in during this period agreed to participate in the survey. Patients had the option to skip all the questions. Individual question response rates ranged from 19 to 100% (average 60%). The response rate of 19% was noted for question 12 that was the only open-ended question.

Participants who answered question #2 (Have you looked at any of your visit notes in the portal?) were those who either answered yes (1,126) to question #1 (Did you know you could see your visit notes from your provider in the portal?) or skipped the question (6). Those who answered no to question #2 were not allowed to proceed. Therefore, the remainder of the survey questions answered were by the 957 people who claimed to have seen at least one note. Two questions in the survey determined patient characteristics and not validated by demographic data in the EHR. Survey respondent characteristics were analyzed using descriptive statistics and chi-squares.

No patient identifying data were collected. The study was reviewed and approved by the university's institutional review board.

Patient Portal

The patient portal (HealtheLife) used was part of the Cerner EHR system and had been enabled since 2011. The Web-based tool includes patient education, secure messaging, medication refills, laboratory results, pathology results, radiology results, and provider notes. Portal instructions are in English only. 9 Portal viewing/downloading of notes increased from 2,312/1,600 in September 2015 to 6,439/5,239 in February 2016.

Tables 2 and 3 provide the response rates and frequencies for questions 1 to 11 and 13 to 14. Questions were grouped into four quantitative themes: respondent characteristics, note availability and access, the usefulness of OpenNotes, and note comprehension.

Table 2. Respondent characteristics ^a .

	N (%)
Q 13. How old are you? (y)	872 (91)
18–29	72 (8)
30–59	437 (50)
> 60	347 (40)
Other	16 (2)
Q14. How much education have you had?	866 (90)
Less than 12th	27 (3)
GED	42 (5)
Some college	278 (32)
College graduate	314 (36)
Graduate degree	205 (24)

Abbreviation: GED, General Educational Development.

^a The frequency of the responses by age revealed that only question #4 showed a statistically significant interaction by age, chi-square (2) = 10.9, p  = 0.004. The frequency of the responses by education revealed that only question #10 showed a statistically significant interaction by education, chi-square (4) = 31.98 p  = 0.001.

Table 3. Results of questions 1 to 11 reported.

	N (%)
Q1. Did you know you could see your visit (outpatient appointment) notes from your provider in the portal?	1,481 (99)
Yes	1,126 (76)
No	355 (24)
Q2. Have you looked at any of your visit notes in the portal?	1,122 (75)
Yes	957 (85)
No	165 (15)
Q3. How often have you looked at your notes?	896 (94)
Once after every visit	334 (37)
Many times after a visit	223 (25)
Only if I have questions about my visit	265 (30)
Q 4. Do you find them helpful?	889 (93)
Always	563 (63)
Sometimes	309 (35)
Never	17 (2)
Q5. What have you found to be helpful in the note (select all that apply)?	886 (93)
What we discussed (the plan)	587 (66)
Medications	398 (45)
When I should come back	387 (44)
Test results	703 (79)
Q6 Have you ever contacted your provider after reading a note?	887 (93)
Yes	350 (39)
No	537 (61)
Q7 Does reading your note change how often you contact your provider?	855 (89)
I now contact more	82 (10)
I now contact less	167 (20)
I contact about the same	606 (71)
Q8. Do you contact your provider to make a correction to the note?	872 (91)
Yes	79 (8)
No	793 (92)
Q9. Do you think seeing your note helps you take better care of yourself?	878 (92)
Yes	726 (83)
No	152 (17)
Q10. How much of what was in the note did you understand?	884 (92)
All of it	492 (56)
Most of it	287 (32)
Some of it	92 (10)
None of it	13 (1)
Q11. How does reading a note affect how worried you are about something?	869 (91)
Makes me less worried	377 (43)
Makes me more worried	49 (6)
No change	443 (51)

Qualitative Analysis

Questions 4 to 11 included a comment section related to specific questions. Question 12 was open-ended and asked, “Is there anything else you would like to tell us about the ability to see your notes in the portal?” We first tagged (coded) these responses with positive, negative, and neutral tags. Two reviewers independently tagged the responses and a third arbitrated any differences. Concordance for positive, negative, and neutral tags for question 12 was close ( k  = 0.78).

Based on a “thematic analysis” approach, positive and negative themes were developed based on the initial tags for question 12. 10

The positive themes were: general, convenience, accessibility, communication, and education related. The general theme meant that the comment was positive, short, and without detail, such as “the portal is great.” The education theme was intended to be broad and include the attribute of OpenNotes being a resource for reminding patients about any aspect of their care. The negative themes were: technical, understanding, communication, and information related. The information theme was also broad and included comments dealing with the lack of information or incorrect information. After selecting themes for the comments related to question 12, the free-text comments related to questions 4 to 11 were analyzed similarly. Table 4 reports the frequency of positive, negative, and neutral comments (tags) related to questions 4 to 12. The frequency of positive and negative themes is reported in Tables 5 and 6 along with related quotations from respondents.

Table 4. Tags by question comments.

Question	Positive (%)	Negative (%)	Neutral (%)	Number
4 Do you find them helpful?	18 (25)	44 (61)	10 (14)	72
5 What have you found to be helpful in the note (select all that apply)?	70 (71)	17 (17)	11 (12)	98
6 Have you ever contacted your provider after reading a note?	19 (25)	21 (28)	35 (47)	75
7 Does reading your note change how often you contact your provider?	10 (24)	6 (14)	26 (62)	42
8 Do you contact your provider to make a correction to the note?	2 (3)	20 (34)	37 (63)	59
9 Do you think seeing your note helps you take better care of yourself?	113 (88)	3 (2)	13 (10)	129
10 How much of what was in the note did you understand?	22 (46)	9 (19)	17 (35)	48
11 How does reading a note affect how worried you are about something?	25 (42)	6 (11)	28 (47)	59
12 Is there anything else you would like to tell us about the ability to see your notes in the portal?	108 (41)	54 (21)	100 (38)	262
Total	387 (46)	180 (21)	277 (33)	844

Table 5. Summary of positive themes.

Themes	Number	Frequency	Quotes
General	247	64%	“I found the portal helpful”
Convenience	4	1%	“It allows me to be more involved in my health care without the need to drive to or sit in the office of or take up the time of my physician”
Accessibility	7	3%	“I love being able to access my personal medical records not just for myself but also for other physicians and organizations which may require information concerning medical health”
Communication	24	6%	“I am glad we have this communication device to ask questions when needed”
Education	101	26%	“I think being able to read the notes helps me to take my time and better understand what is happening”
Total	383	100%

Table 6. Summary of negative themes.

Themes	Number	Frequency	Quotes
Technical	26	14%	“You need to add a feature where a patient can print their medical records and lab works....and the later should be able to be downloaded to a PDF or spreadsheet so to do a comparison of earlier results”
Understanding	14	8%	“I don't always understand lab results. What can I do?”
Communication	16	9%	“I have not received any answers to the messages I have sent to my providers in the Portal yet”
Information	126	69%	“How can I correct wrong info about my meds....dosages, in particular?”
Total	182	100%

There was discussion whether a request for a change or addition to OpenNotes represented a neutral or negative response. Most were treated as negative responses and tagged as an “improvement opportunity” along with those that were tagged as neutral but also a potential improvement opportunity. The same themes adopted for negative responses were used to code improvement opportunities. Table 7 reports the themes related to improvement opportunities, along with their frequencies.

Table 7. Improvement opportunities.

Theme	Number	Frequency	Quotes
Technical	32	19%	“I would like to be able to reply to more than one physician”
Understanding	7	3%	“I would like to see & understand what some of the abbreviations mean from my labs”
Communication	13	8%	“I think there should be a place for general questions: I would like to ask about the GENERAL preliminary instructions before an echocardiogram, but I can find no lead to that”
Information	120	70%	“If written education materials are not attached to the visit summary, it would be helpful to have common topics available through the CHOR website or portal for patients to access (i.e. breastfeeding education, guidelines for starting solids with infants, criteria for sick visits, etc.)”
Total	172	100%

Results

Participant characteristics : Of the 957 participants who had viewed at least one note and proceeded further in the survey, 872 provided data. Table 2 displays the results that show 90% (784) of the responders were above 30 years of age with 8% between 18 and 29 years; 40% were above 60 years and 50% were between 30 and 59 years. One participant stated that they accessed the portal for a minor and another as a family surrogate. Out of 866 individuals who responded regarding their education level, 92% (797) responded that they had a college degree or greater, while 24% (205) had a graduate degree. Only 1% (12) individual had education less than 12th grade. Frequency analysis showed that individuals in the 18 to 29 and 60 and older age groups were more likely to find the notes helpful. We assumed that younger patients were more comfortable with portal technology and older patients used the portal due to more chronic disease. Greater note comprehension correlated with greater education. Subgroup analysis of noncollege compared with college participants showed that noncollege participants were more likely to access notes “many times” than college participants ( p  = 0.015).

Knowledge of Note Availability and Access

Note that 99.6% of patients responded to this question. A total of 1,126 (76%) patients reported knowing that outpatient clinical notes could be seen through the patient portal, while 355 (24%) were not aware. Also, 957 (85%) out of 1,122 individuals reported that they reviewed at least one clinic visit note. Out of 896 individuals, 37% (334) looked at their notes once after every visit, 25% (223) looked at notes many times after a visit, and 30% (265) of individuals looked at visit notes selectively only if they had questions about the visit.

Usefulness of Open Notes

Out of the 889 participants who responded to question 4, 63% (563) found clinical notes to be always helpful, whereas 35% (309) individuals found clinical notes useful sometimes, and 2% never found them useful. Sixty-six percent (587) of 886 participants found the clinical plan to be useful, whereas 45% (398) found medications in the note to be helpful. Forty-four percent (387) found it useful to remind them when they should come back. Seventy-nine percent (703) found laboratory test results to be useful. Overwhelmingly, 83% (726) of individuals thought OpenNotes helped them take care of themselves better. Ninety-four percent (820) noted no change or improvement in their worry level.

Understanding of Provider Note

Out of the 884 individuals, 56% reported that they understood clinical notes completely, whereas 32% (287) of individuals understood most of it. Ninety-two individuals (10%) understood “some” of the clinical notes, whereas 13 (1%) claimed they understood “none of it.”

Qualitative Analysis

Table 4 displays the number of positive, negative, and neutral free-text responses for questions 4 to 12. Because questions 4 to 11 were close-ended, the comments made were related to the question and varied greatly regarding the frequency of positive, negative, and neutral comments.

Table 5 displays the positive theme frequency and exemplary quotations. The majority (64%) of positive responses were general in nature and complementary but rarely included suggestions for improvement. The next most common positive theme was education. A recurrent subtheme of education was the ability of OpenNotes to remind and refresh the memory of patients for details of their medical care. The next most common positive theme was communication that reinforced the concept that OpenNotes enhanced patient–provider engagement and communication.

Table 6 displays the negative theme frequency and exemplary quotations. The information theme was most common with many negative comments about missing or incorrect information included in the note. The next most common negative theme was technical due to issues with the portal, appointments, and the inability to print results. Participants did complain that their providers did not always respond back (communicate) promptly. Lastly, a minority expressed frustration with understanding abbreviations and laboratory data.

Improvement Opportunities

Neutral and negative comments for questions 4 to 12 were additionally tagged as improvement opportunities and assigned themes reported in Table 7 . The open-ended question 12 yielded the majority of patient responses with concerns about information being most common. Potential solutions will be discussed in the “Discussion” section.

Discussion

A quantitative and qualitative analysis of a patient survey from a university-based health care system using OpenNotes is reported. Both closed and open-ended questions were analyzed. Only a small percentage of the total portal enrolled population in the health care system participated in this survey, and they were highly educated, thus not reflecting the typical health care system population. As pointed out in several studies, portal use correlates with the level of education. 11 12

Compared with the Veterans Affairs OpenNotes study, our patient population was more aware of notes being available (76% vs. 33.5%), and more patients had viewed their notes at least once (85% vs. 23.5%). 12 In the initial OpenNotes study, 47 to 84% of participants had opened at least one note. 4

While the design of the original OpenNotes study and ours were quite different, there were similarities. For example, our study reported that the OpenNotes initiative made 83% of respondents feel like they take better care of themselves, similar to 77 to 87% reported in the study by Delbanco et al. 3 Eight percent of the participants in our study contacted the office to report a correction, similar to 7% reported by Bell et al. 13

The qualitative review of OpenNotes by Esch et al included an analysis of 576 free-text comments, as well as 13 patient interviews. They included responses only from patients who had access to at least six office notes during the study period. Five themes were extracted: improved understanding, improved relationship, improved quality, and improved self-care. 14 Improved understanding was similar to our theme of education, as was improved self-care. OpenNotes tended to improve understanding (education) and refresh the memory of patients. Our survey did not examine an improved relationship between physician and patient, nor did it evaluate any quality measures. In another qualitative review of OpenNotes and a reporting tool by Gerard et al, they reported on eight themes: “confirm and remember next steps,” “quicker access and results,” “positive emotions” and “sharing information with care partners,” “accuracy and correcting mistakes,” “partnership and engagement,” “bidirectional communication and education,” and “importance of feedback.” 15 These were similar to our positive themes of convenience, communication, and education.

Unlike most OpenNotes studies reported, this study was conducted by a health care organization unrelated to the original three organizations. In addition, unlike the three organizations that reported data solely from patients under the care of PCPs, this study reported experiences from patients under the care of specialists and PCPs. Also, the original OpenNotes study design invited patient participants only if their PCP agreed to offer them access to office notes. Lastly, to the best of our knowledge, this is the largest qualitative analysis of free-text comments related to OpenNotes.

Lessons Learned and Improvement Opportunities

Our investigation revealed several improvement opportunities associated with both negative and neutral comments. The most common technical complaint was that notes and laboratory results could not be copied or printed. Respondents requested email notification when any new content appeared on the panel and the ability to message multiple providers at the same time. Multiple respondents requested providers not to use acronyms and specify the location of the appointment. The most common improvement opportunity in the understanding theme was comprehension of laboratory results, particularly associated with unknown terms or abbreviations, such as anion gap and SPEP (serum protein electrophoresis). The recurrent issue with the communication theme was that a minority of respondents did not hear back after messaging their provider. Many patients noted inaccurate information regarding their problem list, medications, and medical history which were frequently carried forward from the previous encounter and was not updated by providers.

The present patient portal supports the print option, and an opportunity exists in making the portal interface more user-friendly and making the print option more explicit and evident. A study by Mafi et al showed that email alerts are important to remind patients to review notes. 16 Unfortunately, email notification is not a feature of the portal currently. The portal has since been modified to allow multiprovider messaging. Appointment acronyms are related to the legacy system which stored appointment location in the form of acronyms. On the basis of feedback given by patients, a project is underway to revamp nomenclature of appointment location to be more apparent and easily readable. Although there is a link for laboratory interpretation, more could be done to make it more visible, relevant, and inside the content. The findings of this survey can be useful in providing additional education for providers using OpenNotes so that they might be more aware of patient concerns. Klein et al reported some helpful tips for clinicians who use OpenNotes to improve their performance. 17 Based on feedback from patients, we have expanded OpenNotes to include discharge summaries and eventually plan to include mental health notes, as others have done. Furthermore, a portal-based patient feedback tool as reported by several authors would be of value for reporting a variety of patient issues on an ongoing basis. 15 18

Study Limitations

The survey was available for only a short period and not actively marketed to the patients which may have contributed to the low response rate. The only demographics available in the survey were age and education, and we did not collect patient-identifying information. It is not known if gender or racial data would have been insightful. The survey did not include any questions about data security in the portal. In the study by Vodicka et al, they reported that one-third of their participants expressed privacy and security concerns. 19 This was not expressed by any participants in the comment sections of our survey. Most of the survey respondents were well educated so results might not be generalizable to all populations. Additionally, because we did not survey portal nonparticipants, it is likely that our participants were influenced by “response bias.” 20

The Future

The OpenNotes initiative would benefit from evidence that patient access to clinical notes improves clinical outcomes. Medication adherence has been shown to subjectively improve, and in one study there was an objective improvement in antihypertensive but not antilipid medication adherence. 4 21 More evidence-based studies are needed. Because portal adoption is higher in the well-educated, there need to be strategies for increased enrollment of a more diverse population. OpenNotes should be implemented in more clinical settings, by more specialties, and with different care partners. 22 Bell et al have written about the potential of OpenNotes to support patient safety and quality initiatives by the patient reporting errors and inconsistencies in the notes. 13 23 24

Conclusion

One of the most recent innovations to promote patient engagement is the OpenNotes movement. Our survey reported that those who accessed OpenNotes believed it made them take better care of themselves without increasing anxiety. Given the high level of education of participants, our results may not be generalizable to other populations.

Clinical Relevance Statement

Our survey confirms that for those enrollees who choose to access their online medical records, there is a very good acceptance level of most portal features, to include clinician notes. Our results are skewed due to the high educational level of participants. The majority of portal enrollees choose not to access their records for reasons not elicited in our survey.

Multiple Choice Questions

1. OpenNotes help patients? Select two.

   1. Take care of themselves better.

   2. Makes them more worried.

   3. Makes them less worried.

   4. Patients do not understand medical notes.

   Correct Answer: Options a and c are the correct answers. Most of the patients understand medical notes and they are less worried after reading the note.

2. Most of the patients who participate in patient portal have

   1. Education less than 12th grade.

   2. Some college or college graduate.

   3. General Educational Development.

   4. None of the above.

   Correct Answer: The correct answer is option b. About 90% of respondents had some college education.