The benefits and harms of open notes in mental health: A Delphi survey of international experts

Charlotte Blease; Anna Kharko; Maria Hägglund; Stephen O’Neill; Deborah Wachenheim; Liz Salmi; Kendall Harcourt; Cosima Locher; Catherine M DesRoches; John Torous

doi:10.1371/journal.pone.0258056

. 2021 Oct 13;16(10):e0258056. doi: 10.1371/journal.pone.0258056

The benefits and harms of open notes in mental health: A Delphi survey of international experts

Charlotte Blease ^1,^*, Anna Kharko ², Maria Hägglund ³, Stephen O’Neill ^1,⁴, Deborah Wachenheim ¹, Liz Salmi ¹, Kendall Harcourt ¹, Cosima Locher ^5,⁶, Catherine M DesRoches ^1,⁴, John Torous ^4,⁷

Editor: Umberto Volpe⁸

PMCID: PMC8513879 PMID: 34644320

Abstract

Importance

As of April 5, 2021, as part of the 21^st Century Cures Act, new federal rules in the U.S. mandate that providers offer patients access to their online clinical records.

Objective

To solicit the view of an international panel of experts on the effects on mental health patients, including possible benefits and harms, of accessing their clinical notes.

Design

An online 3-round Delphi poll.

Setting

Online.

Participants

International experts identified as clinicians, chief medical information officers, patient advocates, and informaticians with extensive experience and/or research knowledge about patient access to mental health notes.

Main outcomes, and measures

An expert-generated consensus on the benefits and risks of sharing mental health notes with patients.

Results

A total of 70 of 92 (76%) experts from 6 countries responded to Round 1. A qualitative review of responses yielded 88 distinct items: 42 potential benefits, and 48 potential harms. A total of 56 of 70 (80%) experts responded to Round 2, and 52 of 56 (93%) responded to Round 3. Consensus was reached on 65 of 88 (74%) of survey items. There was consensus that offering online access to mental health notes could enhance patients’ understanding about their diagnosis, care plan, and rationale for treatments, and that access could enhance patient recall and sense of empowerment. Experts also agreed that blocking mental health notes could lead to greater harms including increased feelings of stigmatization. However, panelists predicted there could be an increase in patients demanding changes to their clinical notes, and that mental health clinicians would be less detailed/accurate in documentation.

Conclusions and relevance

This iterative process of survey responses and ratings yielded consensus that there would be multiple benefits and few harms to patients from accessing their mental health notes. Questions remain about the impact of open notes on professional autonomy, and further empirical work into this practice innovation is warranted.

Introduction

As of April 5, 2021, new federal rules in the U.S. mandate that all health providers offer patients online access to their clinical notes [1,2]. These new information sharing rules resulting from the 21st Century Cures Act of 2016 require rapid and full online access to test results, medication lists, referral information, and progress notes (so-called ‘open notes’). Psychotherapy notes are exempt, and “information blocking” is permitted, if doing so “…will substantially reduce the risk of harm” where this is understood as physical harm to a patient or to another person or if there is a privacy exception (§ 171.201(a) p. 704) [1]. Licensed health professionals can decide what constitutes a substantial risk when working “…in the context of a current or prior clinician-patient relationship” (p. 702).

Sharing access to mental health notes is controversial, and in countries that have already opened notes to patients, many psychiatric organizations have resisted implementing the practice [3]. In surveys, mental health clinicians worry that patients will become anxious, confused or upset by what they read; many also anticipate access will lead to increased work burdens [4–6]. Findings drawn predominantly from primary care suggest that mental health patients may derive benefits from accessing their clinical notes including feeling more in control of their care, better remembering their care plan, and better understanding the rationale for medications [7,8]. However, sharing mental health notes could present more challenges than in other clinical specialties. Currently, only a few pilot surveys have examined patients’ experiences in specialized mental health settings, and while the results are encouraging, at least some patients reported feeling disrespected or judged by what they read [9–12]. In existing studies, small sample sizes, the exclusion of patients with serious mental illnesses (SMI) or personality disorders, and the possibility of clinician and patient response biases also limit the potential for informative inferences about psychiatric care [13].

In light of limitations, clinicians may be uncertain about when it is appropriate to redact information. For example, it is unclear whether access may exacerbate harms among mental health patients. On the other hand, blocking information may deny patients important opportunities to better understand, manage, and become engaged in their care. Considering the pressing need for greater clarity in light of the new U.S. healthcare regulations taking effect in April 2021, and while awaiting further empirical research to inform best practice, our goal was to establish expert-led consensus about the potential benefits and harms of sharing notes in mental healthcare including among persons with serious and complex needs. Therefore, we set out to conduct the first Delphi poll in the field, encompassing a range of stakeholders including patient advocates [14].

Methods

Background

First developed by the Rand Corporation in the 1950s, Delphi polls are now an established qualitative methodology for exploring the consensus views of experts in an emerging field [15,16]. Delphi polls rely on non-probability sampling and studies show they offer more accurate predictions than other forecasting techniques [17–19]. The approach is designed to anonymously pool the views and predictions of a purposive sample of identified experts who are invited to answer a series of open-ended questions on a focused topic. Participants are next required to reassess their initial judgements, in light of aggregate group feedback, until consensus is obtained. While some Delphi polls employ the use of face-to-face meetings to help establish areas of agreement, more robust approaches employ anonymous, iterative techniques to avoid the influences of group think or dominant personalities [15,18]. Delphi polls therefore have clear advantages over focus groups owing to participants’ anonymity and have been used extensively in healthcare policy and management, including in psychiatry [18,20,21].

Approach

We used a modified Delphi technique structured around three discrete online surveys [22,23]. While there is no universal agreement about the sample size for Delphi polls, a key objective of Delphi is to maximize the response rate between each round which has been demonstrated to improve the accuracy of consensus opinions and forecasts [15,18,22]. In line with this approach, in Round 1, questions were open-ended requiring free-text responses which were then aggregated and translated into survey items (see S1 Appendix: Round One Survey for questions). In Round 2, experts were provided with the list of survey items and asked to rate their level of agreement using Likert scales. To reduce survey fatigue, those statements that reached a predefined level of consensus were omitted for Round 3, and remaining items recirculated. In this final round, participants were reminded of their previous judgment of each of the remaining items and furnished with the median response of the other experts and invited to preserve or revise their previous response.

The expert panel

Using purposive sampling, the research team compiled a list of prospective participants with expertise on sharing mental health notes. We defined expertise as individuals with experience as: clinicians sharing mental health notes with patients; patients with mental health diagnoses, including patient advocates with lived experience or knowledge of the practice; chief medical information officers or directors of divisions of health organizations that offered open notes to mental health patients; and/or, academic informaticians, who had published significant contributions within the field of health informatics and patient access to clinical records including authors of publications skeptical about the innovation. The research team was mindful to ensure gender, age, race, and international diversity, and specifically strove to include representation from countries and health systems where open notes have been implemented. To ensure a diversity of representation we also employed a snowballing technique by inviting chief medical information officers of health organizations that had opened psychiatric notes to assist us by providing names and contact details of mental health clinicians with experience of the practice. The final list of experts included 92 prospective participants. The study received approval from the Beth Israel Deaconess Medical Center Institutional Review Board in April 2020 (Protocol # 2020P000218) and the University of Plymouth, UK (Protocol # 19/20-1331). Invited participants were advised that the survey was voluntary and unpaid, that their responses would be anonymous to other participants, and that their identity was restricted to members of the research team. Identified experts were contacted via email in August 2020 with an invitation and link to the web-based survey. Invitees were informed that an adequate response time would be given between rounds and that they could choose to withdraw at any time. All respondents gave informed consent before participating.

The questionnaire

An electronic survey was created using Qualtrics (Qualtrics, Provo, UT). Since all prospective participants were fluent English speakers, the survey was only administered in this language. The poll incorporated the three-step modified Delphi technique, and data collection ran from August to November 2020. Participants were sent up to 4 reminders, 1 week apart, and given 4 to 5 weeks to respond to each round (see Fig 1, Flowchart of Delphi Poll). The first round comprised demographic questions and requested information about the nature of participants’ expertise with open notes. This was followed by six open-ended questions on sharing mental health notes plus one additional open-ended question allowing participants to comment on the survey or submit additional responses (see S1 Appendix: Round One Survey). The present Delphi study was based on respondents’ answers to two questions on the potential benefits and harms to patients of reading mental health notes. Participants were asked: “What, in your opinion, are the benefits–if any–of sharing mental health notes with patients?” and “What, in your opinion are the harms–if any–of sharing mental health notes with patients?” Answers to other questions formed the basis of a separate qualitative study that has been published elsewhere.

Following the closure of Round 1, descriptive content analysis was used to transform responses into lists of statements [24]. Owing to the limitations of the data-set (brief comments or sentence fragments), full thematic coding was not applicable [25]. Coding was conducted by CB and independently reviewed by JT and MH and subsequent refinements were made. These individuals were selected for their diverse epistemological backgrounds: CB is a philosopher of medicine and healthcare ethicist from the UK, JT is a psychiatrist with experience of sharing mental health notes with patients in the US, and MH is a healthcare informaticist and implementation scientist based in Sweden. Comments that were unclear or deemed irrelevant to the question were excluded. Statements were then transformed into survey items, and where possible participants’ exact phrasing was preserved. Survey items relating to benefits, and harms of sharing mental health notes formed two separate sections of the survey circulated in Round 2, and participants were requested to respond to each question using a variety of predefined 7-point Likert scales: 1 = strongly disagree, 2 = moderately disagree, 3 = slightly disagree, 4 = neutral, 5 = slightly agree, 6 = moderately agree, 7 = strongly agree (see S2 Appendix: Round Two Survey). Prior to analyzing responses to Rounds 2 and 3, consensus to items was set at an interquartile range of ≤ 1 [15,26]. After analysis of Round 2 responses, items that did not reach consensus were recirculated in Round 3. In Round 3, each participant received a bespoke survey in which they were reminded of their responses to items in Round 2 and provided with the median response of all participants. After competing closed-ended survey questions in each round, participants were also invited to provide any additional feedback. All survey items that reached consensus were collated into themes, and subthemes by CB and JT.

Results

Overview

A total of 70 of 92 (76%) participants from 6 countries responded to Round 1. Among respondents, 35 of 70 (50%) identified as male, and the mean age was 50 (see Table 1). Forty-seven of 70 (67%) reported currently working in clinical practice, and 7 of 70 (10%) reported lived experience of illness. The mean number of years of experience working as a clinician, an open notes researcher, or as a patient advocate was 16 years. All respondents left comments (5,760 words) which were typically brief (1 or 2 sentences or sentence fragments. Replies were shorter when discussing benefits (2,639 words) than harms (3,121 words). As a result of descriptive analysis of free-text responses, the survey was expanded into 88 items, 42 items on the potential benefits of sharing mental health notes with patients and 46 on potential harms (see S2 Appendix: Round Two Survey).

Table 1. Characteristics of survey respondents by round.

	No. (%) of Respondents
Characteristic	Round 1 (n = 70)	Round 2 (n = 56 of 70 [80.0%])	Round 3 (n = 52 of 56 [92.8%])
Gender
Male ^a	35 (50)	28 (50)	26 (50)
Age range, years
20–29	1 (1)	-	-
30–39	19 (27)	17 (30)	17 (33)
40–49	14 (20)	11 (20)	10 (19)
50–59	19 (27)	15 (27)	14 (27)
≥ 60	17 (24)	13 (23)	11 (21)
Ethnicity
Asian	6 (9)	4 (7)	4 (8)
Black/African/Caribbean	1 (1)	1 (2)	1 (2)
White	59 (84)	49 (88)	46 (88)
Other	2 (3)	1 (2)	-
Declined to answer	2 (3)	1 (2)	1 (2)
Country of residence
Canada	3 (4)	2 (4)	2 (4)
Estonia	1 (1)	1 (2)	1 (2)
Norway	3 (4)	3 (5)	3 (6)
Sweden	12 (17)	10 (18)	10 (19)
UK	4 (6)	4 (7)	4 (8)
USA	47 (67)	36 (64)	32 (62)
Currently works in clinical practice	45 (64)	34 (61)	31 (60)
Psychiatry	15 (21)	10 (18)	9 (17)
Primary care	10 (14)	8 (14)	8 (15)
Clinical psychology/Psychotherapy	7 (10)	6 (11)	6 (12)
Psychiatric Nursing	4 (6)	4 (7)	4 (8)
Paediatrics	3 (4)	3 (5)	2 (4)
Social work	3 (4)	1 (2)	-
Palliative care/Home hospice	2 (3)	1 (2)	1 (2)
Peer support	2 (3)	2 (4)	2 (4)
Hospitalist	1 (1)	1 (2)	1 (2)
Radiology	1 (1)	1 (2)	1 (2)
Occupation or expertise related to health/open notes informatics research/patient advocacy ^c
Clinician	46 (66)	35 (63)	32 (62)
Researcher	25 (31)	21 (38)	21 (40)
Chief Information Officer/Portal Director/Medical Director	14 (20)	12 (21)	9 (17)
Patient Advocate/Person with Lived Experience	5 (7)	4 (7)	4 (8)
Social Worker	1 (1)	1 (2)	1 (2)

Open in a new tab

^a Including one transgender male.

^b Including Adult, Child and Adolescent Psychiatry.

^c Numbers & percentages may add up greater than the total n, as some participants reported multiple expertise.

In Round 2, 56 of 70 (80%) experts participated in the online survey and reached consensus on 22 of 88 items (25%) (see S3 Appendix: De-identified Raw Data Results of Round Two). The mean age was 50 years old, and the mean number of years of experience working as a clinician, open notes researcher, or as a patient advocate was 17 years. In Round 3, 52 of 56 (93%) experts responded, reaching consensus on a further 43 items. In total, consensus was achieved for 65 of 88 items (74%). Among Round 3 respondents, the mean age was 49, and the mean number of years of experience working as a clinician, an open notes researcher, or as a patient advocate was 17 years. Table 2 presents the items that reached consensus after each round. Following thematic analysis, all items that reached consensus were categorized into four themes; items in shaded boxes in Table 2 represent perceived benefits of open notes in mental healthcare.

Table 2. Items on the benefits and harms of open notes in mental healthcare that reached consensus.

Item	Mean (SD) Rating	Median–Consensus ^a
Effects on Patient Understanding, Recall and Empowerment (20 items)
Access to mental health notes will improve patient recall about what was communicated during the visit.	6.3 (.89)	7 –Strongly agree
Access to mental health notes will improve patient recall especially among patients with memory or cognitive problems.	6.2 (.77)	6 –Moderately agree
Access to mental health notes will improve patient recall of homework between visits.	6.02 (.96)	6 –Moderately agree
Access to mental health notes will improve patient understanding about their care plan.	6 (.86)	6 –Moderately agree
Access to mental health notes will improve patient understanding about their diagnosis/mental health condition.	5.85 (.89)	6 –Moderately agree
Access to mental health notes will improve patient understanding about the rationale for their treatments.	5.75 (.95)	6 –Moderately agree
Access to mental health notes will improve patient understanding about their health changes over time.	5.71 (1.07)	6 –Moderately agree
Access to mental health notes will improve patient awareness over current medications.	5.77 (.96)	6 –Moderately agree
Access to mental health notes will improve patient preparation for visits.	5.46 (1.16)	6 –Moderately agree
Access to mental health notes will increase the likelihood patients will ask relevant questions at visits.	5.38 (1.14)	6 –Moderately agree
Patient access to mental health notes will improve patient-clinician communication.	5.75 (1.03)	6 –Moderately agree
Patient access to mental health notes will help to demystify psychotherapy.	5.37 (1.17)	6 –Moderately agree
Access to mental health notes will improve patient sense of control of their health.	5.75 (.99)	6 –Moderately agree
Patient access to mental health notes will prompt patients to research more about their healthcare.	5.27 (1.01)	5 –Somewhat agree
Access to mental health notes will improve patient sense of privacy over their health information.	4.6 (1.19)	5 –Somewhat agree
Access to mental health notes will improve patient responsibility for their healthcare.	5.18 (.99)	5 –Somewhat agree
Patient access to mental health notes will improve family/friend/caregiver recall about patient appointments.	5.45 (1.06)	5 –Somewhat agree
When accessing their mental health notes patients will be confused by clinical abbreviations.	5.25 (1.56)	5 –Somewhat agree
When accessing their mental health notes patients will be confused by psychiatric terms.	4.65 (1.31)	5 –Somewhat agree
Patients in domestic abuse situations will be less candid in visits.	4.65 (1.1)	5 –Somewhat agree
Relational Effects (17 items)
Patient access to mental health notes will improve mutual understanding between patients and clinicians.	5.79 (.98)	6 –Moderately agree
Patient access to mental health notes will improve patient-clinician goal-alignment.	5.75 (1.05)	6 –Moderately agree
Access to mental health notes will increase adoption of patient-centered language in clinicians’ documentation.	5.67 (1.13)	6 –Moderately agree
Patient access to mental health notes will improve the therapeutic alliance.	5.45 (1.11)	6 –Moderately agree
Access to mental health notes will diminish the use of disrespectful or derogatory language in notes.	5.96 (.95)	6 –Moderately agree
Access to mental health notes will diminish patient anxiety about what clinicians write about them.	5.6 (1.11)	6 –Moderately agree
Access to mental health notes will increase shared decision-making.	5.52 (.95)	6 –Moderately agree
Access to mental health notes will increase the depth of patient-clinician dialogue during visits.	5.13 (1.07)	5 –Somewhat agree
Access to mental health notes will improve patient trust in clinicians.	5.55 (1.01)	6 –Moderately agree
Access to mental health notes will improve patient satisfaction with care.	5.23 (1.08)	5 –Somewhat agree
Patient access to mental health notes will improve family/friend/caregiver trust in clinicians.	5 (1.01)	5 –Somewhat agree
As a result of reading their mental health notes, a minority of patients will feel ashamed.	4.71 (1.07)	5 –Somewhat agree
As a result of reading their mental health notes, a minority of patients will feel insulted.	4.80 (1.29)	5 –Somewhat agree
As a result of reading their mental health notes, a minority of patients will feel alienated by clinical language.	4.86 (1.26)	5 –Somewhat agree
Perceived misunderstandings in mental health notes will cause patients to be frustrated.	5.11 (.93)	5 –Somewhat agree
Perceived misunderstandings in mental health notes will cause patients to be angry.	4.84 (.95)	5 –Somewhat agree
Access to mental health notes will be detrimental to the therapeutic alliance.	2.5 (1.36)	2 –Moderately disagree
Quality of Care, Outcomes, and Patient Safety (22 items)
Hiding mental health/psychotherapy notes will lead to greater patient stigmatization.	5.56 (.98)	6 –Moderately agree
Patient access to mental health notes will help to close the feedback loop on care.	5.56 (.98)	6 –Moderately agree
Access to mental health notes will help patients to correct errors in notes.	5.37 (1.58)	6 –Moderately agree
Access to mental health notes will help patients to correct clinician misinterpretations	5.35 (1.62)	6 –Moderately agree
Patient access to mental health notes will improve quality of care.	5.34 (1.15)	5 –Somewhat agree
Patient access to mental health notes will improve treatment processes.	5.34 (1.16)	5 –Somewhat agree
Patient access to mental health notes will improve clinical outcomes.	5.02 (1.06)	5 –Somewhat agree
Access to mental health notes will improve patient adherence to treatment plans.	5.24 (1.07)	5 –Somewhat agree
Access to mental health notes will improve patient adherence to medications.	5.12 (.96)	5 –Somewhat agree
As a result of opening mental health notes there will be harms if notes are written without intention to share with patients.	5.39 (1.04)	5 –Somewhat agree
As a result of opening mental health notes there will be harms if patients lack guidance on reading notes.	4.52 (1.32)	5 –Somewhat agree
As a result of reading their mental health notes, a minority of patients will be distressed.	5.16 (1.02)	5 –Somewhat agree
Access to mental health notes will diminish patient feelings of stigmatization.	4.75 (1.19)	5 –Somewhat agree
As a result of reading their mental health notes, a minority of patients will experience cyberchondria.	4.43 (1.26)	4 –Neutral
Patients in domestic abuse situations will be at increased harm.	4.27 (1.05)	4 –Neutral
Too much detail in notes will re-traumatize patients.	3.48 (1.21)	4 –Neutral
Access to mental health notes will be detrimental to patient adherence to treatment plans.	2.29 (1.14)	2 –Moderately disagree
Patients with the following conditions/symptoms will be harmed from accessing their notes: bipolar disorder disorders.	2.31 (1.08)	2 –Moderately disagree
Patients with the following conditions/symptoms will be harmed from accessing their notes: personality disorders.	2.6 (1.32)	2 –Moderately disagree
Patients with the following conditions/symptoms will be harmed from accessing their notes: patients who are suicidal.	2.35 (1.08)	2 –Moderately disagree
Patients with the following conditions/symptoms will be harmed from accessing their notes: patients with obsessive conditions.	2.35 (1.12)	2 –Moderately disagree
Patients with the following conditions/symptoms will be harmed from accessing their notes: eating disorders.	2.33 (1.2)	2 –Moderately disagree
Effects on Professional Autonomy & Healthcare Efficiencies (6 items)
Clinicians will be less detailed/accurate in documenting negative aspects of patient relationship.	4.93 (1.14)	5 –Somewhat agree
Clinicians will be less detailed/accurate in documenting patients’ personalities.	4.77 (1.25)	5 –Somewhat agree
As a result of opening mental health notes there will be an increase in patients demanding changes to their notes.	4.73 (1.29)	5 –Somewhat agree
Access to mental health notes will increase efficiency in patient care.	4.56 (1.3)	5 –Somewhat agree
Clinicians will spend less time in patient visits.	2.69 (.98)	3 –Somewhat disagree
Access to mental health notes will be detrimental to clinician honesty in therapy sessions.	2.69 (1.08)	3 –Somewhat disagree

Open in a new tab

Abbreviations: IQR–interquartile range.

^a Response scale included the following levels: 1 –Strongly disagree, 2 –Moderately disagree, 3 –Somewhat disagree, 4 –Neutral, 5 –Somewhat agree, 6 –Moderately agree, 7 –Strongly agree.

Shaded items represent perceived benefits of opening mental health notes.

Themes

Effects on patient understanding, recall, and empowerment

Panelists agreed that open notes could aid patient recall about what was communicated in visits, and that access to documentation could improve patient-clinician communication, including patient understanding about their diagnosis, prescribed medications, and the rationale for treatments. They also agreed that access could improve mental health patients’ sense of control over their healthcare, insight about their health, and help them better prepare for visits.

Relational effects

Reflecting on whether access could affect the patient-clinician relationship, there was agreement among experts that open notes could improve trust, enhance the therapeutic alliance, increase patient-clinician goal-alignment, and strengthen shared decision-making. Panelists also believed that patient access would increase the adoption of patient-centered language and diminish the use of disrespectful or derogatory language in documentation. There was also agreement that access to clinical notes could diminish patients’ anxiety about what clinicians might write about them.

Quality of care, outcomes, and patient safety

Commenting on the potential for adverse effects, panelists disagreed that open notes would be harmful to patients with serious mental health diagnoses, specifically including persons with bipolar disorders, major depressive disorders, personality disorders, eating disorders, obsessive conditions, and individuals who are suicidal. Panelists agreed that access could help patients to correct errors in notes, including clinician misinterpretations, and that offering access could help to close the feedback loop on care. Furthermore, there was consensus that hiding mental health or psychotherapy notes might lead to greater patient stigmatization or harm. However, participants “somewhat agreed” that harms could arise if clinicians wrote notes without the intention that they might be read by patients, or if patients lacked guidance on how to read their notes. Panelists also “somewhat agreed” that access could improve treatment, including medication adherence.

Effects on professional autonomy and healthcare efficiencies

Expert panelists “somewhat agreed” that as a result of access, there could be an increase in patients demanding changes to their clinical notes. Commenting on the effects on professional autonomy, panelists also “somewhat agreed” that mental health clinicians would be less detailed/accurate in documenting negative aspects of the patient relationship, details about patients’ personalities, or symptoms of paranoia in patients. However, experts predicted that patient access to mental health notes could increase efficiency in care delivery.

Discussion

Currently, limited attention has been paid to the benefits and harms of reading mental health notes among patients with psychiatric diagnoses, including persons with SMI. In view of the lack of large-scale studies into the effects on mental health patients’ experiences, we used a rigorous modified Delphi approach to establish the consensus views about this practice innovation among an international stakeholder group of clinicians, patients, chief medical information officers, and informaticians. The final results comprise the first consensus-driven statement on the benefits and harms of online access to mental health notes.

In this Delphi poll there was consensus that offering online access to mental health notes could enhance patients’ understanding about their diagnosis, care plan, and rationale for treatments. These views contrast with findings of small scale surveys in the U.S. and Sweden in which mental health clinicians anticipated most patients would find notes more confusing than helpful [4,5]. There was also consensus that access could enhance patient recall and sense of empowerment about their care plan. The expert panel also agreed that patient access to mental health notes could strengthen multiple relational benefits of care. The panel believed that open notes present no special harms to patients with SMI.

Despite clear consensus on the potential benefits to patient autonomy questions were raised about the effects on professional autonomy. Experts anticipated changes to the detail and accuracy of records with respect to information patients might perceive as negative. Notably, while experts expect patients to request changes to their notes, they also believed that patient feedback could improve documentation quality. Finally, there was consensus that blocking mental health notes could lead to greater harms including increased feelings of stigmatization. Small scale qualitative studies in psychotherapy suggest that at least some patients experience negative feelings as a consequence of being denied access to their notes [11,12].

The present study invites a number of important unresolved questions. It was not established whether the panelists believed that any potential changes in the detail or accuracy of records might diminish the quality of care. Perhaps reflecting lack of research on key issues, expert panelists held no strong views about whether patients who disagreed with their diagnosis would be less likely to attend visits or whether too much detail might re-traumatize patients. Our expert panelists also reported “neutral” consensus opinions on whether clinicians would be less detailed/accurate in writing differential diagnoses or in documenting substance abuse disorders. These and other issues, warrant further, more nuanced empirical examination and we strongly recommend that research investigate patients’ experiences with accessing mental health notes across a range of settings, including outpatient and inpatient care, and encompassing a wide range of patient populations with different mental health diagnoses. Focused empirical research, including randomized controlled trials, are required to better understand whether access influences objective health outcomes and attendance at visits, and whether patients feel judged or offended by what they read [27]. There is a need for data among patients with serious mental illness, including schizophrenia, personality disorders, or active suicidality to avoid overgeneralizing current evidence to these populations and use cases. Larger scale studies are needed to examine psychiatric clinicians’ experiences including emergent concerns with the practice, and potential sources of patient-clinician disagreement. It will be important to determine objective changes to documentation as a result of patient access [28]. Combined, empirical research should aim to inform how the innovation might be optimized for its new dual purpose–both as a detailed aide memoire for clinicians and as a tool for communication with patients. Finally, it remained unresolved when redaction of aspects of mental records from patients might be appropriate [29], and future research should elaborate on how best this can be practically and ethically managed for both patients and clinicians [30].

Limitations

This study has several limitations many of which are inherent to the Delphi methodology. As with all Delphi polls, there are no standard guidelines for identifying expertise. Although we selected participants to represent a diverse spectrum comprising healthcare professionals, patients with lived experiences of mental illnesses, and health informatics researchers, the reliability of consensus opinions in this survey is dependent on the specialist knowledge and experiences of those who participated. While our expert panel represented a diversity of expertise from countries and health organizations where patients are currently offered access to their notes, it is likely that the survey would have benefited from greater ethnic and socio-economic diversity. Notably, challenges related to portal access, and the digital divide, received less attention, and this may have been a consequence of the demographic composition of panelists. In addition, we cannot exclude the possibility that, even though clinicians and chief medical information officers had extensive knowledge and experience of opening mental health notes, some individuals might already have been more positively disposed to the practice, influencing results. Notwithstanding these limitations, the study benefited from high response rates between each round.

Conclusions

In the era of health information transparency, both patient and professional autonomy in mental health contexts must be balanced with the potential benefits and risks to patient care. Experts in this Delphi poll anticipated multiple benefits and few harms of patient access to mental health notes. Further empirical inquiry is required to explore the impact on both patients and clinicians of psychiatric patients of reading their notes, and how both patients and clinicians can become better prepared and supported for documentation transparency in mental healthcare.

Supporting information

S1 Appendix. Round one survey.

(PDF)

Click here for additional data file.^{(1.8MB, pdf)}

S2 Appendix. Round two survey.

(PDF)

Click here for additional data file.^{(1.4MB, pdf)}

S3 Appendix. De-identified raw data results of round two.

(CSV)

Click here for additional data file.^{(19.8KB, csv)}

Acknowledgments

The authors express their gratitude to the experts who participated in this survey.

Data Availability

Data cannot be shared publicly because the Delphi poll is confidential to other respondents but the participants' responses were not anonymous to the survey team. This is because Delphi polls use iterative techniques that involve recirculating bespoke surveys which include the earlier responses of each individual participant together with the aggregate response of the group.

Funding Statement

Keane Scholar Award: CB Beyond Implementation of eHealth (2020-01229) funded by FORTE – the Swedish Research Council for Health, Working Life and Welfare: MH, CB, CD, KH. NORDeHEALTH (100477) funded by NordForsk: MH Gordon & Betty Moore Foundation: CD Cambia Health: CD

References

1.Health and Human Services Department, USA. 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program. Federal Register; 2020. Available: https://www.govinfo.gov/content/pkg/FR-2020-05-01/pdf/2020-07419.pdf.
2.Salmi L, Blease C, Hägglund M, Walker J, DesRoches CM. US policy requires immediate release of records to patients. British Medical Journal Publishing Group; 2021. [DOI] [PubMed] [Google Scholar]
3.Blease CR, O’Neill S, Walker J, Hägglund M, Torous J. Sharing notes with mental health patients: balancing risks with respect. The Lancet Psychiatry. 2020. [cited 29 Sep 2020]. Available: 10.1016/S2215-0366(20)30. [DOI] [PubMed] [Google Scholar]
4.Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA mental health clinician experiences and attitudes toward OpenNotes. General hospital psychiatry. 2016;38: 89–93. doi: 10.1016/j.genhosppsych.2015.08.001 [DOI] [PubMed] [Google Scholar]
5.Petersson L, Erlingsdóttir G. Open notes in swedish psychiatric care (part 1): survey among psychiatric care professionals. JMIR mental health. 2018;5: e11. doi: 10.2196/mental.9140 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Petersson L, Erlingsdóttir G. Open Notes in Swedish Psychiatric Care (Part 2): Survey Among Psychiatric Care Professionals. JMIR mental health. 2018;5: e10521. doi: 10.2196/10521 [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Klein JW, Peacock S, Tsui JI, O’Neill SF, DesRoches CM, Elmore JG. Perceptions of primary care notes by patients with mental health diagnoses. The Annals of Family Medicine. 2018;16: 343–345. doi: 10.1370/afm.2287 [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Blease C, Dong Z, Torous J, Walker J, Hagglund M, DesRoches C. Association of patients reading clinical notes with perception of medication adherence among persons with serious mental illness. JAMA Network Open. 2021. doi: 10.1001/jamanetworkopen.2021.2823 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Peck P, Torous J, Shanahan M, Fossa A, Greenberg W. Patient access to electronic psychiatric records: a pilot study. Health policy and technology. 2017;6: 309–315. [Google Scholar]
10.Strudwick G, Booth RG, McLean D, Leung K, Rossetti S, McCann M, et al. Identifying indicators of meaningful patient portal use by psychiatric populations. Informatics for Health and Social Care. 2020; 1–13. doi: 10.1080/17538157.2020.1776291 [DOI] [PubMed] [Google Scholar]
11.O’Neill S, Chimowitz H, Leveille S, Walker J. Embracing the new age of transparency: mental health patients reading their psychotherapy notes online. Journal of Mental Health. 2019;28: 527–535. doi: 10.1080/09638237.2019.1644490 [DOI] [PubMed] [Google Scholar]
12.Cromer R, Denneson LM, Pisciotta M, Williams H, Woods S, Dobscha SK. Trust in mental health clinicians among patients who access clinical notes online. Psychiatric Services. 2017;68: 520–523. doi: 10.1176/appi.ps.201600168 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Blease CR, O’Neill SF, Torous J, DesRoches CM, Hagglund M. Patient access to mental health notes: Motivating evidence-informed ethical guidelines. The Journal of nervous and mental disease. 2021;209: 265–269. doi: 10.1097/NMD.0000000000001303 [DOI] [PubMed] [Google Scholar]
14.PCORI (Patient-Centered Outcomes Research Institute). Influencing the Culture of Research. 1 May 2020 [cited 31 Mar 2021]. Available: https://www.pcori.org/engagement/influencing-culture-research.
15.Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Practical assessment, research & evaluation. 2007;12: 1–8. [Google Scholar]
16.Dalkey N. An experimental study of group opinion: the Delphi method. Futures. 1969;1: 408–426. [Google Scholar]
17.Linstone HA, Turoff M. The Delphi Method: Techniques and Applications, AddisonWesley. Reading, MA. 1975. [Google Scholar]
18.Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing. 2000;32: 1008–1015. [PubMed] [Google Scholar]
19.Rowe G, Wright G, McColl A. Judgment change during Delphi-like procedures: The role of majority influence, expertise, and confidence. Technological Forecasting and Social Change. 2005;72: 377–399. [Google Scholar]
20.Lieff SJ, Clarke D. What factors contribute to senior psychiatry residents’ interest in geriatric psychiatry? a Delphi study. The Canadian Journal of Psychiatry. 2000;45: 912–916. doi: 10.1177/070674370004501006 [DOI] [PubMed] [Google Scholar]
21.Vijayaraghavan L, Krishnamoorthy ES, Brown RG, Trimble MR. Abulia: a delphi survey of British neurologists and psychiatrists. Movement disorders: official journal of the Movement Disorder Society. 2002;17: 1052–1057. doi: 10.1002/mds.10194 [DOI] [PubMed] [Google Scholar]
22.Powell C. The Delphi technique: myths and realities. Journal of advanced nursing. 2003;41: 376–382. doi: 10.1046/j.1365-2648.2003.02537.x [DOI] [PubMed] [Google Scholar]
23.Custer RL, Scarcella JA, Stewart BR. The Modified Delphi Technique–A Rotational Modification. Journal of vocational and technical education. 1999;15: 50–58. [Google Scholar]
24.Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3: 77–101. [Google Scholar]
25.Joffe H, Yardley L. Content and thematic analysis. Research methods for clinical and health psychology. 2004;56: 68. [Google Scholar]
26.Ludwig B. Predicting the future: Have you considered using the Delphi methodology. Journal of extension. 1997;35: 1–4. [Google Scholar]
27.Fernández L, Fossa A, Dong Z, Delbanco T, Elmore J, Fitzgerald P, et al. Words Matter: What Do Patients Find Judgmental or Offensive in Outpatient Notes? Journal of General Internal Medicine. 2021; 1–8. doi: 10.1007/s11606-020-06432-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Blease CR, Torous J, Hägglund M. Opinion: Does patient access to clinical notes change documentation? Frontiers in Public Health. 2020;8: 578. [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Blease C, Salmi L, Rexhepi H, Hägglund M, DesRoches CM. Patients, clinicians and open notes: information blocking as a case of epistemic injustice. Journal of Medical Ethics. 2021. doi: 10.1136/medethics-2021-107275 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Blease C, Torous J, Kharko A, DesRoches CM, Harcourt K, O’Neill S, et al. Preparing Patients and Clinicians for Open Notes in Mental Health: Qualitative Inquiry of International Experts. JMIR mental health. 2021;8: e27397. doi: 10.2196/27397 [DOI] [PMC free article] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0258056.r001

Decision Letter 0

Umberto Volpe

6 Aug 2021

PONE-D-21-12428

The Benefits and Harms of Open Notes in Mental Health: A Delphi Survey of International Experts

PLOS ONE

Dear Dr. Blease,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 20 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Umberto Volpe

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include a caption for figure 1.

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

4. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

As highlighted by referees, minor corrections should be applied in a revised version.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a great paper and goes a long way to answer questions that have been raised by clinicians and IT leads for many years. Interestingly in a UK family practice that has 80% of patients able to access their records the highest percentage of patients accessing their records when we looked at this 5 years or so ago were patients with mental health problems. This suggests that patients with mental health issues d want t see their notes. The results of the questionnaires from an international group of specialists within countries that already practice patient access to records seem very reassuring and valuable. A small non peer reviewed study of ours https://bjgp.org/content/68/suppl_1/bjgp18X697133 showed that patients increased their knowledge of their own medical history when tested before and after seeing their notes. The benefits and harms of pen Notes in mental health paper is very timely. The very positive feelings of the interviewed experts would suggest that it might be unethical not to undertake a controlled trial of patients with mental health accessing their mental health records and measuring the patients response to the access against the 80 or so tested fields in the questionnaire.

Reviewer #2: This is an interesting paper and seems to me worthy of publication after some revision. The main problem for me was the abstract – which seemed lacking in actual results. In the abstract the setting seems to me to be methods. Should there not be a methods section in Abstract?. Setting = online.

Results in the Abstract is purely on the numbers responding rather than the content of the statements- benefits and harms. The only results come in the conclusions. Is there not space to put results in results? For example, that 88 items (42 potential benefits, 46 potential harms) were identified is a step more towards ‘results’- but better still to know what these were – summarising Table 2.

Approach (p5)

This is written in present tense – assuming that this is what was actually done, it should be in past tense- otherwise you are left with a feeling of doubt as to whether or not this is what the methods actually were. An indication earlier in the text that the questions were shown in Appendix 1 would be useful – I was left wondering what they were until later in the paper.

Results

It would help the reader if some ways was found of making where there was agreement on benefit and where on harm in Table 2. Given that some statements are worded in the positive and some in the negative, and then there is agreement or disagreement, it is not immediately clear where there was agreement on benefit or harm. You come on to this on page 15, but I have already been trying to work it out from the table – so a clearer table would help.

(p17) I find this sentence a bit odd – “Whether the panel believed any such changes might diminish the quality of care, however, was not established.” Isn’t it the case that there was no consensus? Ie opinion was divided? So isn’t it clearer to say (something like) Opinions of the panel were divided on whether quality of care might be diminished’. And so is this not just another of the ‘unresolved questions’ ie it should go into the next paragraph.

Reviewer #3: I found this paper interesting to read but would have liked to have seen more detail on the methodology. I also have concerns that the expert panel was not representative in that it consists of experts more likely to hold positive views about open notes. I wonder if the findings would have been very different if a more representative expert panel had been used.

Minor points

P1, lines 7, 9 and 11 – I would put a comma after ‘round 1’, ‘round 2’, and ‘round 3’.

P3, para 2, line 3 – I would put a comma after ‘In surveys’.

P4, line 4 – I wonder if the term ‘redact’ might come across as less judgemental than ‘hide’?

P5, para 2, line 3 – replace ‘clinicians of sharing mental health notes’ with ‘clinicians sharing mental health

notes’

P6, para 2, line 2. Would languages other than English have been made available if required?

P9, Table 1. Could “Age range, y” be replaced with “Age range, years” ?

P15, 4th line from bottom – as before would ‘redacting’ come across as less judgemental than ‘hiding’?

P16, line 2 – replace “medication, adherence” with “medication adherence”

Specific concerns

P5, ‘The expert panel’ – The authors describe how individuals with expertise included clinicians with experience of sharing mental health notes with patients and CMIOs of organizations that offer open notes to patients. This would seem to have introduced a bias into the sample in that clinicians and CMIOs who may have actively decided not to offer open notes were not represented?

P11, last sentence. The authors mention that thematic analysis was employed, on p7 the authors reference Braun and Clarke. I would like to know a little more information about the epistemological perspective taken by the authors and to see some evidence of reflexivity. I feel there could be more detail on the thematic analysis – the authors mention that 2 individuals were responsible for identifying themes and subthemes – how was this done and what procedures were in place when these individuals disagreed?

Pages 12-14. Whilst I can see the value of a thematic analysis of the questions generated by the expert panel, the data would seem to lend itself quite well to a factor analysis. Did the authors consider a factor analysis and if so, what were their reasons for not conducting one?

P16 – It would be good to see some consideration of the difficulties inherent in having two versions of a record when elements of the record have been redacted for the patient view. How can this be managed in practical terms so that clinicians can still see the full record?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Richard Fitton

Reviewer #2: Yes: Ray Jones

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Oct 13;16(10):e0258056. doi: 10.1371/journal.pone.0258056.r002

Author response to Decision Letter 0

19 Aug 2021

PONE-D-21-12428

The Benefits and Harms of Open Notes in Mental Health: A Delphi Survey of International Experts

PLOS ONE

Dear PLOS ONE,

Thank you for inviting a revised submission of our paper. We detail our responses to the reviewers below, and our grateful for their valuable feedback.

Yours

C. Blease, PhD

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

________________________________________

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

________________________________________

3. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

________________________________________

4. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

________________________________________

5. Review Comments to the Author

Reviewer #1:

This is a great paper and goes a long way to answer questions that have been raised by clinicians and IT leads for many years. Interestingly in a UK family practice that has 80% of patients able to access their records the highest percentage of patients accessing their records when we looked at this 5 years or so ago were patients with mental health problems. This suggests that patients with mental health issues d want t see their notes. The results of the questionnaires from an international group of specialists within countries that already practice patient access to records seem very reassuring and valuable. A small non peer reviewed study of ours https://bjgp.org/content/68/suppl_1/bjgp18X697133 showed that patients increased their knowledge of their own medical history when tested before and after seeing their notes. The benefits and harms of pen Notes in mental health paper is very timely. The very positive feelings of the interviewed experts would suggest that it might be unethical not to undertake a controlled trial of patients with mental health accessing their mental health records and measuring the patients response to the access against the 80 or so tested fields in the questionnaire.

Response: We thank the reviewer for these kind comments and feedback. The reviewer raises an excellent point about RCTs which we have now added in the Discussion on pages 18-19:

“Focused empirical research, including randomized controlled trials, are required to better understand whether access influences objective health outcomes and attendance at visits, and whether patients feel judged or offended by what they read.[27]”

Reviewer #2:

This is an interesting paper and seems to me worthy of publication after some revision. The main problem for me was the abstract – which seemed lacking in actual results. In the abstract the setting seems to me to be methods. Should there not be a methods section in Abstract?. Setting = online. Results in the Abstract is purely on the numbers responding rather than the content of the statements- benefits and harms. The only results come in the conclusions. Is there not space to put results in results? For example, that 88 items (42 potential benefits, 46 potential harms) were identified is a step more towards ‘results’- but better still to know what these were – summarising Table 2.

Response: We thank the reviewer for this helpful feedback. We have now amended the Abstract (which is indeed restricted by the word count). We have now amended Setting to “Online” and added more detailed information on the actual results:

“A total of 70 of 92 (76%) experts from 6 countries responded to Round 1. A qualitative review of responses yielded 88 distinct items: 42 potential benefits, and 48 potential harms. A total of 56 of 70 (80%) experts responded to Round 2, and 52 of 56 (93%) responded to Round 3. Consensus was reached on 65 of 88 (74%) of survey items. There was consensus that offering online access to mental health notes could enhance patients’ understanding about their diagnosis, care plan, and rationale for treatments, that access could enhance patient recall and sense of empowerment. Experts also agreed that blocking mental health notes could lead to greater harms including increased feelings of stigmatization. However, panelists predicted there could be an increase in patients demanding changes to their clinical notes, and that mental health clinicians would be less detailed/accurate in documentation.”

Approach (p5)

Response: The Approach and Methods section have now amended to past tense, and reference to Appendix 1 has also been signposted earlier in the text.

Results

Response: We do agree that the table is, in some respects, not intuitive. However, we followed other Delphi studies in the layout which lists items according to strength of opinion reached in the consensus. In addition, as with other Delphi polls, we have preserved the survey items, as far as possible, according to the wording used by the participants in Round 1. Nonetheless, the the reviewer raises an important point and in order for items about harms and benefits to be clearer in Table 2, and more intuitive for the reader, we have now shaded those items that represent perceived benefits of opening mental health notes. Please see pages 13-14.

Response: We have now moved this sentence to the next paragraph. In fact, the original phrasing was accurate and unavoidably nuanced since it was not established (one way or the other) from the Delphi poll whether participants did believe that any changes to documentation might ultimately diminish quality of care.

Reviewer #3:

I found this paper interesting to read but would have liked to have seen more detail on the methodology. I also have concerns that the expert panel was not representative in that it consists of experts more likely to hold positive views about open notes. I wonder if the findings would have been very different if a more representative expert panel had been used.

Response: We thank the reviewer for his or her helpful comments. We have now included more detail on the Delphi methodology on p4 including new references:

“First developed by the Rand Corporation in the 1950s, Delphi polls are now an established qualitative methodology for exploring the consensus views of experts in an emerging field.[15,16] Delphi polls rely on non-probability sampling and studies show they offer more accurate predictions than other forecasting techniques.[17–19] The approach is designed to anonymously pool the views and predictions of a purposive sample of identified experts who are invited to answer a series of open-ended questions on a focused topic. Participants are next required to reassess their initial judgements, in light of aggregate group feedback, until consensus is obtained. While some Delphi polls employ the use of face-to-face meetings to help establish areas of agreement, more robust approaches employ anonymous, iterative techniques to avoid the influences of group think or dominant personalities.[15,18] Delphi polls therefore have clear advantages over focus groups owing to participants’ anonymity and have been used extensively in healthcare policy and management, including in psychiatry.[18,20,21]”

We note that, because Delphi polls using purposive sampling techniques, representativeness is neither intended nor assured. In addition, our experts all had knowledge and experience of the practice of open notes in mental healthcare. However, we do agree that there is a question about representativeness and have amended the text to reflect this limitation. On page 5 we added the clarification:

“We defined expertise as individuals with experience as: […] and/or, academic informaticians, who had published significant contributions within the field of health informatics and patient access to clinical records including authors of publications skeptical about the innovation.”

On page 19 we added:

“In addition, we cannot exclude the possibility that, even though clinicians and chief medical information officers had knowledge and experience of opening mental health notes, some individuals might already have been more positively disposed to the practice, influencing results.”

Minor points

P1, lines 7, 9 and 11 – I would put a comma after ‘round 1’, ‘round 2’, and ‘round 3’.

Response: This has been deleted (see response to Reviewer 2).

P3, para 2, line 3 – I would put a comma after ‘In surveys’.

Response: Now amended.

P4, line 4 – I wonder if the term ‘redact’ might come across as less judgemental than ‘hide’?

Response: Now amended.

P5, para 2, line 3 – replace ‘clinicians of sharing mental health notes’ with ‘clinicians sharing mental health notes’

Response: Now amended.

P6, para 2, line 2. Would languages other than English have been made available if required?

Response: No, the survey was only made available in English as it was predicted that all prospective experts on sharing mental health notes (who are based mostly in the USA and Sweden, as well as some in England) would be fluent in the language.

P9, Table 1. Could “Age range, y” be replaced with “Age range, years” ?

Response: Now amended.

P15, 4th line from bottom – as before would ‘redacting’ come across as less judgemental than ‘hiding’?

Response: We retained this use of “hiding” since the participants used this phrasing in the open responses. We concede that this wording might prove challenging depending on the perspective of, for example, clinicians versus patients, since “redacting” might also sound jargonistic from a patient perspective. To balance this, we have included “redact” in the Introduction of the paper and retained “hiding” to reflect the wording of the participants, in the Results and Discussion.

P16, line 2 – replace “medication, adherence” with “medication adherence”

Response: Now amended.

Specific concerns

Response: This is a good point: please see response to comment above, and amendments in the text.

Response: Owing to the limitations of the data-set (brief comments, or sentence fragments) full thematic analysis was not applicable but we have now added further elaboration and a reference to that effect. We have also offered details of the background of the authors involved in coding. Finally, the sentence referring to disagreement was inaccurate and has been elided. We have now added on page 7:

“Owing to the limitations of the data-set (brief comments or sentence fragments), full thematic coding was not applicable.[25] Coding was conducted by CB and independently reviewed by JT and MH and subsequent refinements were made. These individuals were selected for their diverse epistemological backgrounds: CB is a philosopher of medicine and healthcare ethicist from the UK, JT is a psychiatrist with experience of sharing mental health notes with patients in the US, and MH is a healthcare informaticist and implementation scientist based in Sweden.”

Response: We did not include factor analysis since this does not form part of Delphi methodology which is a form of qualitative research and we have now offered an enhanced description of Delphi in response to the reviewer’s previous comment.

Response: While we note that the panelists did not raise the issue of shadow records or redaction of aspects of the record, the reviewer makes an excellent point. We have now added this issue to the discussion. On page 19 we added:

“Finally, it remained unresolved when redaction of aspects of mental records from patients might be appropriate, and future research should elaborate on how best this can be practically and ethically managed for both patients and clinicians.”

________________________________________

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Richard Fitton

Reviewer #2: Yes: Ray Jones

Reviewer #3: No

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(27.7KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0258056.r003

Decision Letter 1

Umberto Volpe

17 Sep 2021

The Benefits and Harms of Open Notes in Mental Health: A Delphi Survey of International Experts

PONE-D-21-12428R1

Dear Dr. Blease,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Umberto Volpe

Academic Editor

PLOS ONE

Additional Editor Comments (optional): none

Reviewers' comments:

PLoS One. doi: 10.1371/journal.pone.0258056.r004

Acceptance letter

Umberto Volpe

30 Sep 2021

PONE-D-21-12428R1

The Benefits and Harms of Open Notes in Mental Health: A Delphi Survey of International Experts

Dear Dr. Blease:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Umberto Volpe

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Appendix. Round one survey.

(PDF)

Click here for additional data file.^{(1.8MB, pdf)}

S2 Appendix. Round two survey.

(PDF)

Click here for additional data file.^{(1.4MB, pdf)}

S3 Appendix. De-identified raw data results of round two.

(CSV)

Click here for additional data file.^{(19.8KB, csv)}

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(27.7KB, docx)}

Data Availability Statement

[pone.0258056.ref001] 1.Health and Human Services Department, USA. 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program. Federal Register; 2020. Available: https://www.govinfo.gov/content/pkg/FR-2020-05-01/pdf/2020-07419.pdf.

[pone.0258056.ref002] 2.Salmi L, Blease C, Hägglund M, Walker J, DesRoches CM. US policy requires immediate release of records to patients. British Medical Journal Publishing Group; 2021. [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref003] 3.Blease CR, O’Neill S, Walker J, Hägglund M, Torous J. Sharing notes with mental health patients: balancing risks with respect. The Lancet Psychiatry. 2020. [cited 29 Sep 2020]. Available: 10.1016/S2215-0366(20)30. [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref004] 4.Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA mental health clinician experiences and attitudes toward OpenNotes. General hospital psychiatry. 2016;38: 89–93. doi: 10.1016/j.genhosppsych.2015.08.001 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref005] 5.Petersson L, Erlingsdóttir G. Open notes in swedish psychiatric care (part 1): survey among psychiatric care professionals. JMIR mental health. 2018;5: e11. doi: 10.2196/mental.9140 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref006] 6.Petersson L, Erlingsdóttir G. Open Notes in Swedish Psychiatric Care (Part 2): Survey Among Psychiatric Care Professionals. JMIR mental health. 2018;5: e10521. doi: 10.2196/10521 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref007] 7.Klein JW, Peacock S, Tsui JI, O’Neill SF, DesRoches CM, Elmore JG. Perceptions of primary care notes by patients with mental health diagnoses. The Annals of Family Medicine. 2018;16: 343–345. doi: 10.1370/afm.2287 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref008] 8.Blease C, Dong Z, Torous J, Walker J, Hagglund M, DesRoches C. Association of patients reading clinical notes with perception of medication adherence among persons with serious mental illness. JAMA Network Open. 2021. doi: 10.1001/jamanetworkopen.2021.2823 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref009] 9.Peck P, Torous J, Shanahan M, Fossa A, Greenberg W. Patient access to electronic psychiatric records: a pilot study. Health policy and technology. 2017;6: 309–315. [Google Scholar]

[pone.0258056.ref010] 10.Strudwick G, Booth RG, McLean D, Leung K, Rossetti S, McCann M, et al. Identifying indicators of meaningful patient portal use by psychiatric populations. Informatics for Health and Social Care. 2020; 1–13. doi: 10.1080/17538157.2020.1776291 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref011] 11.O’Neill S, Chimowitz H, Leveille S, Walker J. Embracing the new age of transparency: mental health patients reading their psychotherapy notes online. Journal of Mental Health. 2019;28: 527–535. doi: 10.1080/09638237.2019.1644490 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref012] 12.Cromer R, Denneson LM, Pisciotta M, Williams H, Woods S, Dobscha SK. Trust in mental health clinicians among patients who access clinical notes online. Psychiatric Services. 2017;68: 520–523. doi: 10.1176/appi.ps.201600168 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref013] 13.Blease CR, O’Neill SF, Torous J, DesRoches CM, Hagglund M. Patient access to mental health notes: Motivating evidence-informed ethical guidelines. The Journal of nervous and mental disease. 2021;209: 265–269. doi: 10.1097/NMD.0000000000001303 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref014] 14.PCORI (Patient-Centered Outcomes Research Institute). Influencing the Culture of Research. 1 May 2020 [cited 31 Mar 2021]. Available: https://www.pcori.org/engagement/influencing-culture-research.

[pone.0258056.ref015] 15.Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Practical assessment, research & evaluation. 2007;12: 1–8. [Google Scholar]

[pone.0258056.ref016] 16.Dalkey N. An experimental study of group opinion: the Delphi method. Futures. 1969;1: 408–426. [Google Scholar]

[pone.0258056.ref017] 17.Linstone HA, Turoff M. The Delphi Method: Techniques and Applications, AddisonWesley. Reading, MA. 1975. [Google Scholar]

[pone.0258056.ref018] 18.Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing. 2000;32: 1008–1015. [PubMed] [Google Scholar]

[pone.0258056.ref019] 19.Rowe G, Wright G, McColl A. Judgment change during Delphi-like procedures: The role of majority influence, expertise, and confidence. Technological Forecasting and Social Change. 2005;72: 377–399. [Google Scholar]

[pone.0258056.ref020] 20.Lieff SJ, Clarke D. What factors contribute to senior psychiatry residents’ interest in geriatric psychiatry? a Delphi study. The Canadian Journal of Psychiatry. 2000;45: 912–916. doi: 10.1177/070674370004501006 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref021] 21.Vijayaraghavan L, Krishnamoorthy ES, Brown RG, Trimble MR. Abulia: a delphi survey of British neurologists and psychiatrists. Movement disorders: official journal of the Movement Disorder Society. 2002;17: 1052–1057. doi: 10.1002/mds.10194 [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref022] 22.Powell C. The Delphi technique: myths and realities. Journal of advanced nursing. 2003;41: 376–382. doi: 10.1046/j.1365-2648.2003.02537.x [DOI] [PubMed] [Google Scholar]

[pone.0258056.ref023] 23.Custer RL, Scarcella JA, Stewart BR. The Modified Delphi Technique–A Rotational Modification. Journal of vocational and technical education. 1999;15: 50–58. [Google Scholar]

[pone.0258056.ref024] 24.Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3: 77–101. [Google Scholar]

[pone.0258056.ref025] 25.Joffe H, Yardley L. Content and thematic analysis. Research methods for clinical and health psychology. 2004;56: 68. [Google Scholar]

[pone.0258056.ref026] 26.Ludwig B. Predicting the future: Have you considered using the Delphi methodology. Journal of extension. 1997;35: 1–4. [Google Scholar]

[pone.0258056.ref027] 27.Fernández L, Fossa A, Dong Z, Delbanco T, Elmore J, Fitzgerald P, et al. Words Matter: What Do Patients Find Judgmental or Offensive in Outpatient Notes? Journal of General Internal Medicine. 2021; 1–8. doi: 10.1007/s11606-020-06432-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref028] 28.Blease CR, Torous J, Hägglund M. Opinion: Does patient access to clinical notes change documentation? Frontiers in Public Health. 2020;8: 578. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref029] 29.Blease C, Salmi L, Rexhepi H, Hägglund M, DesRoches CM. Patients, clinicians and open notes: information blocking as a case of epistemic injustice. Journal of Medical Ethics. 2021. doi: 10.1136/medethics-2021-107275 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0258056.ref030] 30.Blease C, Torous J, Kharko A, DesRoches CM, Harcourt K, O’Neill S, et al. Preparing Patients and Clinicians for Open Notes in Mental Health: Qualitative Inquiry of International Experts. JMIR mental health. 2021;8: e27397. doi: 10.2196/27397 [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

The benefits and harms of open notes in mental health: A Delphi survey of international experts

Charlotte Blease

Anna Kharko

Maria Hägglund

Stephen O’Neill

Deborah Wachenheim

Liz Salmi

Kendall Harcourt

Cosima Locher

Catherine M DesRoches

John Torous

Roles

Abstract

Importance

Objective

Design

Setting

Participants

Main outcomes, and measures

Results

Conclusions and relevance

Introduction

Methods

Background

Approach

The expert panel

The questionnaire

Fig 1. Flowchart of Delphi poll.

Results

Overview

Table 1. Characteristics of survey respondents by round.

Table 2. Items on the benefits and harms of open notes in mental healthcare that reached consensus.

Themes

Effects on patient understanding, recall, and empowerment

Relational effects

Quality of care, outcomes, and patient safety

Effects on professional autonomy and healthcare efficiencies

Discussion

Limitations

Conclusions

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Umberto Volpe

Roles

Author response to Decision Letter 0

Decision Letter 1

Umberto Volpe

Roles

Acceptance letter

Umberto Volpe

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases