A patient and family reporting system for perceived ambulatory note mistakes: experience at 3 U.S. healthcare centers

Abstract

Objective

The study sought to test a patient and family online reporting system for perceived ambulatory visit note inaccuracies.

Materials and Methods

We implemented a patient and family electronic reporting system at 3 U.S. healthcare centers: a northeast urban academic adult medical center (AD), a northeast urban academic pediatric medical center (PED), and a southeast nonprofit hospital network (NET). Patients and families reported potential documentation inaccuracies after reading primary care and subspecialty visit notes. Results were characterized using descriptive statistics and coded for clinical relevance.

Results

We received 1440 patient and family reports (780 AD, 402 PED, and 258 NET), and 27% of the reports identified a potential inaccuracy (25% AD, 35% PED, 28% NET). Among these, patients and families indicated that the potential inaccuracy was important or very important in 58% of reports (55% AD, 55% PED, 71% NET). The most common types of potential inaccuracies included description of symptoms (21%), past medical problems (21%), medications (18%), and important information that was missing (15%). Most patient- and family-reported inaccuracies resulted in a change to care or to the medical record (55% AD, 67% PED, data not available at NET).

Discussion

About one-quarter of patients and families using an online reporting system identified potential documentation inaccuracies in visit notes and more than half were considered important by patients and clinicians, underscoring the potential role of patients and families as ambulatory safety partners.

Conclusions

Partnering with patients and families to obtain reports on inaccuracies in visit notes may contribute to safer care. Mechanisms to encourage greater use of patient and family reporting systems are needed.

INTRODUCTION

Adverse events and medical errors in ambulatory care are not uncommon.¹ For example, approximately 12 million individuals are estimated to experience a diagnostic error in ambulatory care in the United States each year.² However, opportunities for error identification may be particularly challenging in ambulatory care (compared with inpatient settings) because visits only provide clinicians a brief snapshot of the patient’s care, diagnoses may evolve in the space between visits, where they are difficult to see, and routine quality improvement mechanisms that can capture such problems historically have not been as well developed as in-patient systems.³ As a result, established mechanisms to identify errors or adverse events such as voluntary reporting by providers, chart reviews, and trigger tools only capture a small fraction of all ambulatory errors and safety events, and do not consistently include patients and families.^4–6

Partnering with patients, families, and informal caregivers is urged by experts to improve safety and quality of care, but proven systematic mechanisms for doing so are few.^7–10 A recent evidence review highlighted the potential role for sharing ambulatory visit notes with patients and families through a patient portal (open notes) to identify documentation and diagnostic errors.¹¹ OpenNotes is a nonprofit international academic research and advocacy initiative that seeks to empower and engage patients and families in their health care by promoting access to ambulatory visit notes through the patient portal. Starting with 20,000 patients in a proof-of-concept study, open notes are now accessible to over 40 million patients and families in every state across the nation.¹² In addition to providing patients and families ready access to health information, it enhances engagement and trust, and creates an opportunity for patients and families to participate in efforts to improve safety.^13–15

Evidence of frequent errors in electronic health records (EHRs) suggest that systematic patient and family review may yield substantial findings.^16,17 For example, in an assessment of both medication adherence and EHR accuracy, one report found that more than half of over 1000 patients showed discrepancies between blood levels of medications and those prescribed in the EHR.¹⁸ In a pilot program using a patient reporting tool to collect patient feedback on potential inaccuracies in visit notes, 21% of the reports submitted by adult primary care patients identified a potential inaccuracy and the majority of these reports resulted in a change to the medical record.¹⁹

Although not all potential inaccuracies discovered by patients and families are necessarily medical errors or lead to direct patient harm, identifying such potential hazards in visit notes resonates with general patient safety principles aiming to identify risk before harm reaches the patient. For example, wrong medication doses may not create harm for an index patient but still present risk not only to that patient but also to future patients who may be vulnerable to the same underlying EHR, provider, or communication error. Outdated family histories, inaccurate symptom reporting, or concerning test results that have not been addressed with the patient also may represent risk of future harm, especially if they lead to diagnostic delay.^7,9,19

To date, no reports have explored patient and family reporting systems at the individual-note level in pediatric or adult subspecialty patient populations. As health information transparency grows, organizations will need a systematic way to capture and respond to patient feedback about EHR errors. Understanding the known limitations of patient reporting systems (primarily low uptake)²⁰, we aimed to aggregate and assess patient and family reports and characterize perceived note inaccuracies across ambulatory primary care and subspecialty visit notes including medicine, surgery, adult care and pediatrics within 3 diverse healthcare institutions in the United States. We anticipate that such information would augment clinician safety reporting and organizational learning, and be helpful for other organizations considering patient and family reporting mechanisms for perceived errors in open notes, adapted in their own settings.

MATERIALS AND METHODS

Settings

Three healthcare institutions implemented a patient-and-family reporting system for open notes: a northeast U.S. academic adult hospital (AD), a northeast U.S. freestanding academic pediatric hospital (PED), and a southeast U.S. nonprofit hospital network that includes multiple hospitals in several states (NET) (Table 1).

Table 1.

Organizational characteristics

Site	AD	PED	NET
Licensed beds	673	404	7200
Estimated ambulatory visits/year	643 975	557 000	5 980 000
Estimated portal registrants	136 483	75 000	405 792
Formal parent/proxy registrant, %	0	89	0
Patient registrant, %	100	11	100
Clinics participating in patient/family reporting system	Primary care, OBGYNa	All clinics with open notesa	All clinics with open notesa
Adult vs pediatric ambulatory visits available	Adult	Both	Both
U.S. geographic location	Northeast	Northeast	Southeast

AD: adult academic medical center; NET: hospital network; PED: pediatric academic medical center;

^aAt AD, participating clinics included 2 primary care clinics and all obstetrics and gynecology clinics using open notes; at PED, participating clinics included all primary care and subspecialty clinics with the exception of Adolescent Clinic, Gynecology Clinic, Psychiatry Clinic, Adolescent Substance Abuse Clinic and Child Protection team Clinic; at NET, participating clinics included all primary care and subspecialty clinics.

Patients and participants

AD participants included adult patients using an online patient portal who had access to an eligible clinic note from August 2014 to March 2017. Eligible clinic notes included those written by 2 teams of clinicians at the hospital-based primary care practice (from August 2014 to March 2017) and 1 additional pilot primary care site, as well as obstetrics and gynecology clinics except family planning (from April 2016 to March 2017).

PED participants included adolescent (≥13-17 years of age) and adult patients who held their own patient portal account, and parents or medical guardians who had access to an eligible clinic note from June 2016 to March 2017. Eligible ambulatory clinic notes included those from all primary care and specialty clinics with the exception of 5 clinics that did not share their notes on the portal (adolescent clinic, adolescent substance abuse clinic, child protection clinic, psychiatry clinic, and gynecology clinic.) These clinics produced <5% of ambulatory clinic notes.²¹

Participants at NET included adult patients and parents or medical guardians of pediatric patients who used an online patient portal and had access to an eligible clinic note from September 13 to October 30, 2017. Eligible clinic notes included those from all primary care, urgent care and subspecialty clinics, as well as emergency department notes across the network of healthcare facilities.

For simplicity, we refer to respondents as “patients and families” at all sites. As part of routine open notes implementation policy at all 3 sites, providers could “hide” individual clinic notes, such as notes containing confidential information, so that they did not appear in the portal. At AD, <1% of notes were hidden,¹⁹ while at PED, approximately 5% of all ambulatory notes were hidden.²¹ NET implementation of open notes was new and data regarding hidden notes was not available.

Patient and family reporting system development

The open notes patient reporting tool was created by a multidisciplinary team of stakeholders at AD including patients and families, physicians, nurses, and quality improvement (QI) experts; patient relations personnel; and health information management (HIM) and information systems representatives. Details of the process from the pilot period (August 2014-2015) have been previously published.¹⁹ Based on feedback from the pilot project, and in collaboration with stakeholders at AD, PED, and NET, the tool was further refined and more broadly implemented in 2016.

The reporting tool consists of 10 core questions, with some follow-up questions based on branching logic soliciting feedback about patient and family understanding of the note and the care plan, potential inaccuracies in the visit note, characterization of inaccuracies, and challenges related to the care plan (Supplementary Appendix 1). To balance the type of feedback captured by the questions, respondents were given the opportunity to provide positive feedback about their providers, their notes, and on the reporting tool itself. Identical reporting tools were deployed at both AD (beginning April 2016) and PED (beginning June 2016). The reporting tool at NET was deployed in September 2017 with slight modifications to account for administrative data collection and local preferences (Supplementary Appendix 2).

Patient and family reporting system implementation

Participants at each site accessed the reporting system via the patient portal, but given technical requirements at each site, reporting tool implementation varied. This enabled us to test the reporting system in real-world circumstances that reflect different organizational practices and portal functionality related to sharing notes, including different EHR and patient portal platforms. We therefore intentionally allowed for modifications to meet site-specific requirements, while maintaining the core patient and family reporting function (Supplementary Appendix 2).

QI review and case coding

AD and PED defined a potential safety concern in 3 ways: 1) a “no” response to the questions, “Did you understand the care plan?” or “Do you plan to follow the care plan?” 2) a “yes” or “unsure” response to the question, “Did you find a possible inaccuracy?” or 3) participants reporting a concerning test result that they had not discussed with their provider. NET defined a potential safety concern as: 1) a “no” response to the question, “Did you understand the plan for your care” or 2) a “yes” or “unsure” response to the question, “Did you find anything in your note that you thought was a mistake?” The test result questions were not asked at NET.

The process for reviewing the reporting tool patient and family responses was tailored to individual institutional resources and workflows. At AD and PED, the reports flowed through a previously published QI algorithm for real-time responses to patient and family feedback.¹⁹ Reports were reviewed by patient relations experts, who contacted patients, caregivers, parents, or providers after confirming a possible safety concern. Outcomes for each potential safety concern were coded using an established coding scheme (Supplementary Appendix 3). In addition, a team including a doctor, a QI specialist, and a patient relations expert reviewed each potential safety concern to determine whether it represented an actual, possible, or no safety concern. The determination was based on the group’s consensus over whether the reported concern could directly affect care now or in the future. For example, an incorrect prescription medication dose or an abnormal test result that was not discussed with the patient and family was considered an actual safety concern. Instances in which there was not enough information and the patient could not be reached for follow-up, but the team concluded that the information in the patient report could potentially lead to a change of care, were coded as possible safety concerns. Reports that provided information but were not urgent and would not likely lead to a change of care, such as an inaccurate social history, were coded as no safety concern. At NET, patients and families who reported a potential safety concern were asked if they would like to be contacted. Those responding “yes” were contacted by a member of the HIM team.

Analysis

We used descriptive statistics to report responses at each site. We avoided statistical comparison because we anticipated that the response rate to a voluntary patient and family reporting system would be low based on the literature,²⁰ the reporting system was designated quality improvement work at 2 sites, and our tolerance for site-specific differences in implementation limited statistical comparisons. We used administrative data for organizational characteristics, but identification of note-reading rates was not possible due to technical factors at 2 sites. Beyond comprehensive review of patient or family described errors as detailed subsequently, a systematic qualitative analysis of other open-ended questions was beyond the scope of this project. However, some quotes representing common comments are included to provide real examples of patient and family reports.

The project was reviewed by AD and PED institutional review boards and determined to be a QI initiative focused on real time responses on a note-by-note basis to improve care. NET obtained institutional review board approval for the project as a research study with a limited intervention period and included collection of additional patient data (Supplementary Appendix 2).

RESULTS

Respondents and note perceptions

Characteristics of the 3 healthcare systems are listed in Table 1. A total of 1440 reports were submitted by patients and families during the evaluation period (AD = 780, PED = 402, NET = 258) (Table 2).

Table 2.

Characteristics of patient reporting system responses

Reporting tool item	AD	PED	NET
Number of completed reports	780	402	258
Understood the note	734 (94)	363 (90)	215 (87)a
Understood the care plan	739 (95)	377 (94)	225 (92)b
Expected to follow the care plan	313 (98)c,d	367 (97)	N/A
Potential documentation inaccuracy	193 (25)	139 (35)	66 (28)f
Yes	175 (22)	123 (30)	55(23)
Not sure	18 (2)	16 (4)	11 (5)
Concerning results identified	51(15)c	57 (14)	N/A
Results not previously discussed with provider	7 (14)	12 (21)	N/A
Bothersome words noted	30 (9)c	33 (8)	N/A
Positive feedback volunteered	528 (68)	170 (42)	N/A

Values are n or n (%).

AD: adult academic medical center; N/A: not available; NET: hospital network; PED: pediatric academic medical center.

^aThere were 10 nonrespondents (n = 248).

^bThere were 14 nonrespondents (n = 244).

^cQuestion was only asked after 2016 (total at AD after 2016 was 339, compared with total at AD of 780 during 2014-2017).

^dThere were 20 nonrespondents (total = 319).

^eIncludes respondents who indicated “yes” or “not sure.”

^fThere were 18 nonrespondents (total = 240).

The majority (≥97%) of reports at AD and NET were submitted by patients, while at PED, most reports were submitted by parents of minor patients (94%). The majority of reports at all sites indicated that participants were able to understand the visit note (94% AD, 90% PED, 87% NET) and the care plan (95% AD, 94% PED, 92% NET). At AD and PED, >97% reports indicated that patients and families expected to follow the care plan. Some patients and families reported the presence of concerning test results (15% AD and 14% PED), though the majority of concerning results had already been discussed with their provider (86% AD, 79% PED).

Documentation inaccuracies

Overall, 398 (27%) patient and family reports at all 3 sites (193 of 780 [25%] AD, 139 of 402 [35%] PED, and 66 of 240 [28%] NET) identified a total of 557 possible documentation inaccuracies, with several reports noting multiple inaccuracies (Table 3).

Table 3.

Types of documentation inaccuracies reported by patients and families

	AD (n = 193)	PED n = 139)	NET (n = 66)	ALL (n = 398)	Examples
Type of perceived documentation inaccuracy
Symptoms	31 (16)	40 (29)	14 (21)	85 (21)	“In the note, it says [patient] has afebrile seizures and unprovoked seizures. This is not true, all his seizures have come from fevers, mostly when they spike. We have not seen any unprovoked seizures at this time.”
					“It mentions that I have urine retention. It does not mention the 35 times a day that I am going to the bathroom. In addition to the urgency, pain and leakage I am having.”
Health problems	38 (20)	33 (24)	13 (20)	84 (21)	“Date and type of transplant was not correct”
					“…my son does not have conversion disorder. He has convergence related to his eyesight. Huge difference”
Medications	40 (21)	24 (17)	10 (15)	74 (18)	“He is not currently on 6MP. He started Abatacept/Orencia at the end of Aug.”
					“Amitiza missing, famotadine missing, magnesium missing”
Something important was missing	29 (15)	23 (16)	6 (9)	58 (15)	“No mention of recent dental surgery and prescribed high doses of antibiotic, … and [I] suspect [the effect] on digestive system [was related to antibiotics]”
					“It excluded important information from the parent and only included it from the child which was not reflective of the true picture.”
Family history	17 (9)	18 (13)	5 (8)	40 (10)	“Our family history was listed incorrectly as it omitted that [patient’s] maternal aunt has a thyroid condition, his mother is the other person allergic to sulfa and his father is allergic to erthyromycin, penicillin and amoxicillin [allergy visit note]”
					“My brother has something other than atrial fibrillation, and is having surgery this month to help correct. I can't recall the name of his diagnosed heart condition, though symptoms tend to mimic a-fib.”
Physical exam	15 (8)	13 (9)	7 (11)	35 (9)	“The exam listed was not performed to the extent noted-especially the anal examination which did not take place. This paragraph looks like it was a cut and paste from another record.”
Social history	8 (4)	5 (4)	7 (11)	20 (5)	“One thing that is really important is that I want to (have to) quit smoking…. I have asked repeatedly over the past couple years if there is a support group addressing this. There is not.”
					“She wrote that I would have 10 drinks [which is double what I actually drink]. 5 of this and 5 of that. Perhaps the use of the word OR would help.”
Names of healthcare providers	7 (4)	15 (11)	2 (3)	24 (6)	“It stated that I identify PCP as Dr.X. This is inaccurate, my PCP is Dr Y, I have never met Dr X”
Appointment scheduling	2 (1)	6 (4)	2 (3)	10 (2)	“We were never able to make an appointment with Dr. X …Also, the notes say that we should follow up in one month but during our appointment Dr. Y told us to come back in 2-3 months.”
Other	52 (27)	38 (27)	37 (56)	127 (32)	“Note refers to right knee at first which is correct then switches to the left knee which is incorrect”
					“This note was filed under the wrong child's record”

Values are n (%).

^aSome reports listed multiple inaccuracies.

The most common types of reported inaccuracies at all 3 sites included inaccurate description of symptoms or reasons for the visit (21%), other health problems (21%), and medication list (18%). Fifteen percent of reports indicated that important information was missing from the note. Nine percent of reports noted inaccurate description of the physical exam, including parts of the exam that patients and families reported were documented but not performed. Examples of “other” inaccuracies included laterality (right or left side) errors; wrong age, sex, or name; and notes filed in the wrong patient’s record. Overall, 58% of patients and families who identified a possible documentation inaccuracy reported that it was important or very important (55% AD, 55% PED, 71% NET).

Bothersome language

Relatively few reports indicated bothersome language in the note (9% AD, 8% PED; not available for NET). Examples of words or phrases that respondents reported as bothersome are available in Supplementary Appendix 5.

Positive feedback

At AD and PED, 698 of 1182 (59%) of patient and family reports included voluntary positive feedback (68% AD, 42% PED). Many of those same respondents also reported potential inaccuracies. Some respondents detailed their appreciation for having the opportunity to provide feedback through the reporting tool and described personal benefits from reading clinic notes (Supplementary Appendix 4). At NET, 70 of 237 (29%) indicated that reading their note had affected the way they think about their provider; among these, 46 of 69 (67%), of the comments were positive.

QI review and case coding

Among all patient reports at AD and PED, 350 of 1182 (30%) reports were identified as potential safety concerns, which included possible documentation inaccuracies, not understanding or not intending to follow the care plan, and concerning test results that were not previously discussed with a provider (206 of 780 [26%] AD and 144 of 402 [36%] PED). These were reviewed by patient relations and the QI team and coded for outcomes.

Among these 350 patient and family reports, no further action was required in 26 of 206 (13%) of cases at AD and 25 of 144 (17%) at PED. In 2 cases at AD follow up was not available (patients no longer with the healthcare system). Patient relations reached out to patients and families in all remaining cases but achieved no response in 41 of 178 (23%) of cases at AD and 12 of 119 (10%) of cases at PED, leaving 244 cases, 137 at AD and 107 at PED (Table 4). Among these, 44 of 137 (32%) at AD and 9 of 107 (8%) at PED were resolved in conversation between patient relations and the patients and families, while 76 of 137 (55%) at AD and 72 of 107 (67%) at PED resulted in a change to the medical record or to patient care. Examples of changes to patient care included a clinician reaching out to the patient to clarify complicated medication regimens that the patient did not understand (eg, a steroid taper), correct a discrepancy between a medication dosage in the note as compared with a prescription, or enter a missing test requisition for a stated plan for cardiac testing. In each of these instances, patients and families identified the potential safety problem after reading the note, prompting a change that would otherwise not have taken place in a timely manner. Further intervention was declined in a few instances by the patient and family: 3 of 137 (2%) at AD and 3 of 107 (3%) at PED. Healthcare providers declined to make changes in 4 of 137 (4%) of cases at AD and 14 of 107 (13%) of reports at PED. Very few patients and families requested a formal amendment to the medical record: 2 of 137 (2%) at AD and none at PED, though no patients at AD ultimately filed the formal amendment request. A minority of reports (8 of 137 [6%] at AD and 9 of 107 [8%] at PED) were deferred to the patient relations team, because either they had a previous relationship with the respondent or the content of the report extended beyond the purpose of the reporting tool. After review of all patient and family reports qualifying as potential safety concerns, including those requiring no action and those rated as not important by patients and families, 122 of 206 (59%) at AD and 78 of 144 (54%) at PED were identified as actual or possible safety concerns by the QI team.

Table 4.

Outcomes of safety concerns

	AD	PED
Total safety concernsa	137	107
Resolved in conversation	44 (32)	9 (8)
Declined intervention	3 (2)	3 (3)
Change made to medical record or to patient’s care	76 (55)	72 (67)
Request for amendment declined	4 (3)	14 (13)
Formal amendment request initiated	2 (2)	0
Defer to patient relations	8 (6)	9 (8)

Values are n or n (%).

AD: adult academic medical center; PED: pediatric academic medical center.

^aCases in which patients did not respond (AD 41, PED 12), follow-up was not available (AD 2), or no action was needed (AD 26, PED 25) were excluded from this analysis.

At NET, 66 of 258 (26%) of patient and family reports were flagged as a potential safety concern, including identification of a possible note mistake and not understanding the care plan; among these, 49 of 66 (74%) indicated a desire to be contacted. Outcomes of the outreach made by HIM to the patients and families were not systematically coded.

DISCUSSION

Our assessment of over 1400 patient and family reports at 3 diverse U.S. healthcare institutions is the largest, to our knowledge, that examines patient and family experiences and safety concerns related to ambulatory visit notes. The findings add to growing literature that indicates that patients and families can be valuable partners in quality improvement efforts.^22,23 Across the 3 sites, an average of 90% of participants reported that they understood their notes. About one-quarter of the reports submitted by patients and families identified potential documentation inaccuracies; the most common being the description of symptoms, medications, and health problems, very similar to our prior experience in primary care.¹⁹ Among these, patients and families rated 58% of the possible inaccuracies they identified as important or very important. At AD and PED, the majority of cases deemed by patient relations to require further evaluation resulted in an amendment to the medical record or a change in practice, and the majority were considered actual or possible safety concerns on QI review.

Medication errors and diagnostic inaccuracies are common causes of ambulatory safety hazards,^24–26 and participants in this QI project identified perceived inaccuracies in notes that could affect both. Medication errors in medical records are common,^27–29 and engaging patients and families to review visit notes may help to reduce errors, adverse events, and perhaps even preventable readmissions. Few studies have examined patient and family contributions to identifying inaccuracies of chief complaint or history of present illness in ambulatory care.^24,26 However, these were commonly noted by respondents in our project. While some reported inaccuracies were relatively minor, such as slight variations in duration of symptoms, others were more substantial, such as major symptom errors, wrong reason for the visit, incorrect cancer screening preferences, inaccurate family histories including heritable malignancies, and concerning test results that were not previously discussed with patients and families. Each of these could affect the diagnostic or treatment process. Furthermore, inaccuracies related to laterality of symptoms or physical exam findings (a common error reported in the safety literature),^30–32 as well as inaccuracies related to demographic information (eg, wrong patient name, age, or sex) and notes filed in the wrong patient’s chart, could lead to other medical errors and reflect poorly on clinicians.

One common disadvantage of EHRs includes copy or paste errors and the ease with which templates or macros can be inserted into the text.³³ Safety consequences of this practice include inaccurate documentation of the physical exam and the continued propagation of incorrect information.^33,34 Participating patients and families detected these unintended consequences in our reporting system. Consequently, their feedback may help curb some of these practices and diminish the further proliferation of inaccurate information, although focused research on this topic is needed.

Our QI initiative highlights the possibility that the use of multiple reporting sources, including both clinicians and patients and families, may further enhance the ability to effectively capture potential safety issues.^35,36 In addition, aggregate patient reports can identify vulnerabilities in organizational processes.³⁷ As one example based on this experience, organizations may want to ensure accurate use of imported physical exam templates to reflect actual events at the visit and avoid misleading interpretations by subsequent clinicians. They may also want to consider further research examining whether patient and family reports of inaccurate symptoms, which were among the most frequent patient-reported inaccuracies in our experience, are related to diagnostic errors.

Despite concerns that inviting patients and families to identify potential inaccuracies would impact the therapeutic relationship between the provider and patient and family member, more than half of all respondents volunteered to report positive feedback about their providers. This finding resonates with prior studies that suggest that transparent notes can improve trust in patient-doctor relationships.^15,38,39 Also, comments provided by patients and families reflected their desire to help improve the accuracy of the medical record. Many respondents who noted a possible mistake also provided positive feedback, suggesting that they were not using the system unilaterally to voice concerns. A minority of patients and families used the tool to voice their disapproval with a provider or to report other negative feedback about care. At AD and PED, most of the patients and families who reported negative feedback were already known to the patient relations teams, which facilitated early outreach and ongoing communication to address their concerns.

A relatively small percentage of patients and families at AD and PED reported that they found bothersome language in their notes. In our experience with the dissemination of open notes over the last 8 years and in conducting focus groups with clinicians, many clinicians do not expect to change the way they document in order for patients to read notes. However, providing clinicians with guidance around language or terminology that may be particularly bothersome to patients and families may help alleviate potential distress or confusion. Our findings further resonate with prior reports suggesting that most patients and families who read notes report that they understand them as they are currently written, and that sharing notes offers a tangible and scalable strategy for engaging and partnering with patients.^10,40,41

Several limitations apply to this work. Similar to most patient and family event reporting systems, the feedback reporting tool is likely capturing only a small fraction of all inaccuracies in clinic notes due to its reliance on patients and families to voluntarily complete the report. At the same time, the proportion of reports documenting a possible mistake does not necessarily reflect the proportion of documentation errors in all medical records because patients who found errors may have been more likely to use the reporting system than those who did not, although other types of feedback were also encouraged. Owing to technology constraints, we could not track note-reading rates at each site to determine the proportion of respondents who used the reporting system. The overall response rate was low, as expected, although comparable to that observed for other patient reporting tools²⁰ and other clinician online ambulatory reporting systems.¹⁹ Nevertheless, despite limited uptake, the reporting system identified actionable clinical information. Patients and families who used the reporting system may represent a self-selection bias, as they had to be registered for the portal to participate by definition, as notes were read through the portal. While more patients and families are enrolling and gaining access to their health information, studies have shown disparities in portal use by level of health literacy, education, race, and primary language.^42–45 In addition, some patients and families may not have the time, interest, or literacy level to use the reporting system, while others may fear negative repercussions.⁴⁰ These factors may bias our findings in unpredictable directions, and outreach to engage broader patient populations on the patient portal is needed. Although reports were validated at AD and PED with systematic QI review including chart review where appropriate, outreach at NET was guided by patient and family request for follow-up, and some reports included as “potential safety concerns” may have been omissions or misunderstandings. Furthermore, although the wording was very similar, some of the questions asked at AD and PED were not strictly identical to the ones used at NET and may have led to some differences in responses. Because the reporting tool was a QI intervention at 2 sites, the experience of patients and families is not intended to be generalizable to other organizations or geographic areas, and use of the reporting system would need to be modified to address local culture and practices. Based on the experiences at these 3 organizations, research with more diverse patient populations is needed.

CONCLUSION

Implementation of an open notes patient and family reporting system at 3 different healthcare institutions, across patients and families, and different visit types yielded results that were overall similar, with identification of perceived documentation inaccuracies in about one-quarter of reports. Altogether, findings suggest that patients and families can provide valuable feedback on the accuracy of clinic notes, which may help to improve the quality of care. However, greater outreach is needed to register diverse patients on the patient portal and to encourage patient and family reporting. Future researchers should examine the effect of such reporting on error prevention and patient outcomes.

FUNDING

This work was supported by the CRICO/Risk Management Foundation of the Harvard Medical Institutions, the Robert Wood Johnson Foundation, the Gordon and Betty Moore Foundation, the Peterson Center on Healthcare, and Cambia Health Foundation.

CONTRIBUTIONS

FCB contributed to the conception and design of the work, data collection, data analysis and interpretation, drafting the article, critical revision of the article, and final approval of the version to be published. AF contributed to the data collection, data analysis and interpretation. MG contributed in the design of the work, data collection, data analysis and interpretation. MED contributed in the design of the work, data collection, data analysis and interpretation and critical revision of the article. YJT contributed in the design of the work, data collection, and critical revision of the article. CDC contributed in the data collection and interpretation, and critical revision of the article. TV contributed in the data collection and critical revision of the article. AM contributed in the design of the work, and critical revision of the article. MDS contributed in the design of the work, and critical revision of the article. PF contributed in the design of the work, data interpretation, and final approval of the article. SKB contributed in the conception and design of the work, data collection, data analysis and interpretation, drafting the article, critical revision of the article, and final approval of the version to be published.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of interest statement

None declared.