Impacts of a web-based educational program for veterans who read their mental health notes online

Abstract

Objective

This study evaluates whether a web-based educational program for patients who read their mental health notes online improves patient-clinician communication and increases patient activation.

Methods

The web-based educational program, developed with end-user input, was designed to educate patients on the content of mental health notes, provide guidance on communicating with clinicians about notes, and facilitate patients’ safe and purposeful use of their health information. Eligible patients were engaged in mental health treatment (≥1 visit in the prior 6 months) and had logged into the Veterans Health Administration (VHA) patient portal at least twice. Participants completed measures of patient activation, perceived efficacy in healthcare interactions, patient trust in their clinicians, and patient assessment of the therapeutic relationship before and after participating in the program. A total of 247 participants had complete data and engaged with the program for 5 minutes or more, comprising the analytic sample. Multivariate analysis using mixed effects models were used to examine pre-post changes in outcomes.

Results

In bivariate analyses, patient activation, perceived efficacy in healthcare interactions, and trust in clinicians increased significantly between pre- and post-training assessments. In fully adjusted models, changes in patient activation [b = 2.71 (1.41, 4.00), P < 0.01] and perceived efficacy in healthcare interactions [b = 1.27 (0.54, 2.01), P < 0.01)] remained significant.

Conclusions

Findings suggest that this educational program may help empower mental health patients who read their notes online to be active participants in their care, while also providing information and tools that may facilitate better relationships with their clinicians.

INTRODUCTION

Nearly a decade ago, the Health Information Technology for Economic and Clinical Health (HITECH) Act called for increased, purposeful usage of electronic health records among providers and health care systems in the United States.¹ Specifically, the HITECH Act emphasized the need for increased patient access to health information and transparency in patient-provider interactions.^1–3 To this end, the international OpenNotes initiative⁴ encourages healthcare systems to provide patients online access to detailed visit information written in their clinical progress notes. The U.S. Veterans Health Administration (VHA) is unique in that it is one of a few healthcare systems that allow patients to access all of their progress notes written after January 1, 2013—including all mental health notes—via VHA’s own OpenNotes initiative.

Research investigating the impacts of patient access to clinical progress notes suggests that reading notes may increase patients’ engagement in care and help them feel more in control of their healthcare experiences.^5–8 However, some mental health clinicians report apprehension to patients reading their notes online.^9–11 Clinicians fear that reading mental health notes may cause unintentional harm for their patients by increasing worry and confusion. Additionally, VHA mental health clinicians have reported making changes to the content of their notes in response to the rollout of OpenNotes, such as decreasing the amount of diagnosis-related details.⁹ In qualitative interviews, VHA mental health clinicians worried that patient online access to progress notes could hinder patient-provider trust and therapeutic rapport.¹⁰ Qualitative interviews with veterans receiving mental health care in VHA reinforced the idea that strong patient-clinician relationships were essential to the therapeutic process and that reading their notes could sometimes strain their relationship with their clinicians.¹² Survey data on patient responses to OpenNotes found that most veterans who read their mental health notes online frequently experienced benefits, but a few also frequently felt upset from reading their notes.¹³

To reduce the likelihood of unintended harms to the patient-clinician relationship and enhance patient benefit in the form of increased engagement in their care, we developed a web-based educational program on OpenNotes for veterans receiving VHA mental health care. Based on findings from research described above, the program was designed to educate patients on the content of mental health notes, provide guidance on communicating with clinicians about their notes, and facilitate patients’ safe and purposeful use of information contained in their notes. In this study, we evaluate whether the program improved patient-clinician communication and increased patient activation.

METHODS

This study was reviewed and approved by the Institutional Review Board of the participating VHA medical center. All participants provided informed consent prior to study activities.

Setting

In 2003, VHA launched My HealtheVet, an online patient portal that VHA patients can use to manage their healthcare and securely communicate with their providers. In January 2013, the Blue Button feature in My HealtheVet was updated to provide patients the opportunity to view their clinical notes online. Now, all notes written after January 1, 2013, are available for viewing 3 days after the note is complete.

This study was conducted at a large VHA medical center in the Pacific Northwest region of the United States. The medical center serves approximately 100 000 veterans at 11 urban and rural sites throughout the region. More than 200 mental health clinicians (physicians, psychologists, social workers, nurse practitioners) provide mental health services to approximately 16 000 veterans. Veterans who receive mental health services at this medical center are most frequently white, non-Hispanic (82%), and male (88%), with an average age of 53. They seek care for post-traumatic stress disorder (PTSD; 28%), depression (26%), substance use disorder (12%), schizophrenia/bipolar spectrum disorders (9%), and personality disorder (3%). When this study was initiated (July 2016), approximately half had signed up to access their clinical notes online.

Web-based educational program

The study team worked closely with an experienced instructional designer and a contracted team of developers to design and build a web-based educational program on OpenNotes for VHA patients receiving mental health care. The program was designed to familiarize patients with OpenNotes, educate patients on how to read and use their notes, provide recommendations regarding discussing notes with their clinicians and what to do when they had questions about their notes, as well as help patients protect their health information when using OpenNotes (Figure 1). Content for the program was derived from interviews with 28 mental health clinicians and 28 patients receiving VA mental health care. Interviews discussed their experiences, concerns, and potential benefits regarding OpenNotes (for further description, see Pisciotta et al.¹⁴). Content for the web-based educational program is available in a Supplementary Appendix.

Figure 1.

Learning objectives of the web-based educational program.

Procedures

We recruited participants between February and July of 2016. Eligible patients had completed at least 1 in-person VHA mental health appointment during the 6 months prior, had logged into My HealtheVet at least twice during this period, and were authenticated My HealtheVet users (allowing them to access their clinical notes). We excluded patients with documented dementia or cognitive disorder diagnoses. We used VHA Patient Care Databases to identify eligible patients from a range of mental health services (eg, substance abuse treatment program, PTSD clinic) to obtain a diverse sample representing a range of mental health conditions. We mailed recruitment letters to 3381 patients and received 598 responses indicating interest in participating. After describing the study and confirming eligibility, we enrolled 407 participants. Fourteen patients withdrew prior to survey administration, resulting in 393 participants receiving the baseline assessment in August 2016.

This study used a switching replications design to capture differences between pre-training assessments and post-training assessments that might be attributable to secular changes in outcomes (ie, changes in outcomes due to increased familiarity with OpenNotes that might occur naturally over time). Participants completed assessments at 3 timepoints and viewed the web-based program once. All participants received the assessments on the same schedule: month 0 (baseline), month 5, and month 10. Wave 1 participants received the educational program at month 1 (between the first and second assessment), and wave 2 participants received the educational program at month 6 (between the second and third assessments). The educational program was available only to study participants. Overall retention was good; 358 participants returned the first assessment (91%), 334 returned the second assessment (85%), and 329 returned the third assessment (84%). Content of the assessments are described below. All participants received instructions for using My HealtheVet to access their clinical notes at the beginning of the study and a $5 gift card for each completed study activity.

Data sources

Healthcare utilization and diagnostic data

Data on healthcare utilization and mental health diagnoses (depression, PTSD, substance use disorder, anxiety, schizophrenia, bipolar disorder, delusional disorder, and personality disorder) during the year prior to the survey were extracted from VHA’s Patient Care Database. Schizophrenia, bipolar disorder, and delusional disorders were combined into 1 category labeled “psychotic or bipolar spectrum disorders.”

Assessments

The assessments contained questions on demographic characteristics (baseline only) and several validated measures assessing patient engagement in care and patient perceptions of the patient-clinician relationship. The eHealth Literacy Scale (eHEALS)¹⁵ was administered at baseline to assess perceived ability to use technology or web-based tools to access appropriate health-related information. Outcome constructs were chosen based on the overall aims of the program, which reflected the goals of the OpenNotes initiative as well as previous research on mental health clinician and veteran patient concerns and experiences with OpenNotes.^4,9,10,12,14 Specifically, we were interested in improving patient engagement in their health care and reducing the potential for fissures in the therapeutic relationship. The Patient Activation Measure for Mental Health (PAM-MH) was used to assess the extent to which patients actively participate in their treatment.¹⁶ The PAM-MH is a widely used version of the PAM¹⁷ and has been tested in a clinical sample.¹⁶ To assess patients’ perceived efficacy in healthcare interactions, we used the Perceived Efficacy in Patient-Physician Interactions (PEPPI-10).¹⁸ Patients’ feelings of trust in their clinicians was measured using 5 items from the Cultural Competence item set of the Consumer Assessment of Healthcare Providers and Systems (CAHPS); these were developed as standardized items to assess information from patients about the care they received.¹⁹ Finally, patients’ overall impression of the therapeutic relationship with their clinicians was measured using the Scale to Assess the Therapeutic Relationship-Patient (STAR-P), which captures 3 aspects of the therapeutic relationship from the patient perspective: positive collaboration, positive clinician input, and non-supportive clinician input.²⁰ These items were administered at all 3 timepoints to capture change in these measures over time.

Analysis

We restricted our analytic sample to participants who spent at least 5 minutes in the web-based educational program and had complete data on outcomes and covariates of interest (n = 247). Using t tests for continuous variables and chi-square tests for categorical variables, we compared our analytic sample to participants with complete data on outcomes and covariates who spent less than 5 minutes with the educational program (n = 53). Participants who spent at least 5 minutes in the program were slightly older, more often male, and more frequently diagnosed with a psychotic or bipolar spectrum disorder in the prior year (Table 1) than those who spent less than 5 minutes in the program. However, participants included in the analytic sample more closely resembled patients seeking mental health care at the medical center.

Table 1.

Characteristics of the sample population by time spent in the training course

	≥5 minutes (n = 247) n (%)	<5 mins (n = 53) n (%)	Pa
Age, mean (sd)	59.4 (±11.1)	53.0 (±13.9)	<0.01
Male	206 (83.4)	36 (67.9)	0.01
Non-Hispanic white	212 (85.8)	42 (79.3)	0.23
Education			0.53
High school degree or less	23 (9.3)	4 (7.6)
Some college or degree	157 (63.6)	38 (71.7)
More than college degree	67 (27.1)	11 (20.8)
Employment status			0.63
Employed	47 (19.0)	14 (26.4)
Unemployed	20 (8.1)	5 (9.4)
Disabled, unable to work	89 (36.0)	17 (32.1)
Retired	91 (36.8)	17 (32.1)
eHeals score, mean (sd)	28.7 (±6.3)	30.2 (±6.2)	0.12
Number of MH visits, mean (sd)b	14.9 (±20.5)	18.0 (±20.1)	0.32
Diagnosesc
Anxiety or panic disorder	26 (10.5)	6 (11.3)	0.87
Major depression	149 (60.3)	30 (56.7)	0.62
PTSD	92 (37.3)	23 (43.4)	0.40
Psychotic or bipolar spectrum disorder	35 (14.2)	2 (3.8)	0.04

Note: eHeals = eHealth Literacy Scale [15].

^at tests comparisons for continuous variables, chi-square tests for categorical variables.

^bSpecialty mental health visits in the previous 12 months; ^cin the previous 12 months.

We assessed bivariate changes in outcomes by calculating the mean pre- and post-training scores and by running paired t tests comparing the mean differences between the 2 timepoints. Next, we conducted a multivariate analysis using mixed effects models to account for our longitudinal research design. After examining the effect of wave assignment on model outcomes, it was determined that no significant differences existed between waves (suggesting pre-post changes were not attributable to increased familiarity with OpenNotes that would naturally occur over time). Therefore, wave assignment was removed from analyses; we analyzed assessments immediately before and immediately after participation in the education program. Non-missing former/latter assessment data were filled in forwards/backwards for 12 participants. All mixed models were a priori adjusted for age, education level, number of mental health visits in the previous 12 months, eHealth literacy score, anxiety or panic disorder, major depression, PTSD, and schizophrenia or bipolar disorder. Distributions for the outcome measures tended to be negatively skewed, with scores clustering in the higher ends of the scale. Therefore, we ran mixed models as both linear and tobit distributions, and used visual model diagnostics to determine which method was a better fit for each outcome measure. Model diagnostics demonstrated that linear mixed models were a better fit for the PAM score, STAR-P score, and PEPPI score models, and a tobit mixed model was a better fit for the CAHPS score model due to its heavy skew toward its maximum value. An additional sensitivity analysis was conducted by restricting the sample to participants who completed at least 10 minutes with the educational program; we found no substantial or pattern of differences between the subgroups, except that change in CAHPS score over time gained magnitude and significance in the restricted group. Consequently, we decided to retain the full group with 5 or more minutes with the educational program. All statistical analyses were conducted using Stata 15 (StataCorp, College Station, TX).

RESULTS

Participants included in the analyses (n = 247) had a mean age of 59 (±11.1) years old, were mostly male (n = 206; 83%), non-Hispanic white (n = 212; 86%), and had at least some college (n = 224; 91%) (Table 1). Participants had a range of documented mental health diagnoses, with major depression being the most common diagnosis in the group (n = 149; 60%).

In t test analyses (Table 2), significant improvements were observed post-training for patient activation (t[246]= 4.10, P<.01), perceived efficacy in healthcare interactions (t[246]= 3.39, P<.01), and patient trust in their clinician (t[246]= 1.98, P=.049). No significant change was observed in patient assessment of the therapeutic relationship.

Table 2.

Minimum, maximum, average, and change in outcome scores (n = 247)

	Min-max score	Pre-training score Mean (sd)	Post-training score Mean (sd)	Change	Pa
PAM	22-100	55.5 (±13.4)	58.2 (±14.5)	+2.71	<0.01
STAR-P	14-48	36.1 (±8.2)	36.6 (±8.0)	+0.47	0.20
PEPPI	13-50	37.3 (±8.4)	38.5 (±8.4)	+1.27	<0.01
CAHPS	10-100	76.6 (±25.5)	79.4 (±24.2)	+2.83	0.049

PAM=Patient Activation Measure [16].

Abbreviations:STAR-P= Scale to Assess the Therapeutic Relationship-Patient [20]; PEPPI=Perceived Efficacy in Patient-Physician Interactions [18]; CAHPS=Consumer Assessment of Healthcare Providers and Systems [19]; higher scores are better for all measures.

^aPaired t test.

In fully adjusted mixed models (Table 3), patient activation score and perceived efficacy in healthcare interactions increased significantly between the pre- and post-training assessments [PAM: pre-post score change b = 2.71 (1.41, 4.00), P < 0.01; PEPPI: pre-post score change b = 1.27 (0.54, 2.01), P < 0.01)]. Several covariates were also significant in fully adjusted models. Older age, higher number of mental health visits, higher baseline eHealth literacy, and recent schizophrenia/bipolar disorder diagnosis were significantly associated with higher patient activation scores (PAM: all P < 0.05). Older age and higher baseline eHealth literacy were significantly associated with a stronger therapeutic relationship (STAR-P: all P < 0.01). Likewise, older age, higher baseline eHealth literacy, and recent schizophrenia/bipolar disorder diagnosis were significantly associated with higher perceived efficacy in healthcare interactions (PEPPI: all P < 0.01). Last, older age and recent schizophrenia/bipolar disorder diagnosis were independently significantly associated with higher perceived efficacy in healthcare interactions (CAHPS: all P < 0.03).

Table 3.

Fully adjusted model results (n = 247)

	PAM score		STAR-P score		PEPPI score		CAHPS scorea
	beta (95% CI)	P	beta (95% CI)	P	beta (95% CI)	P	beta (95% CI)	P
Pre-post score change	2.71 (1.41, 4.00)	<0.01	0.47 (-0.25, 1.19)	0.20	1.27 (0.54, 2.01)	<0.01	3.61 (-0.75, 7.96)	0.11
Age	0.32 (0.19, 0.44)	<0.01	0.17 (0.08, 0.25)	<0.01	0.15 (0.07, 0.23)	<0.01	0.68 (0.30, 1.07)	<0.01
Education
≤ High school degree	Ref	–	Ref	–	Ref	–	Ref	–
Some college/degree	−0.40 (-5.84, 5.04)	0.89	−2.16 (-5.96, 1.63)	0.26	−1.16 (-5.07, 2.74)	0.56	−4.22 (-22.77, 14.34)	0.66
≥ College degree	0.67 (-5.44, 6.77)	0.83	−0.96 (-5.01, 3.08)	0.64	0.03 (-4.18, 4.25)	0.99	0.02 (-20.42, 20.95)	0.99
Number of MH visitsb	−0.05 (-0.10, <-0.01)	0.04	−0.01 (-0.05, 0.03)	0.53	−0.02 (-0.05, 0.02)	0.42	0.12 (-0.15, 0.40)	0.39
eHeals	0.93 (0.70, 1.15)	<0.01	0.29 (0.14, 0.43)	<0.01	0.52 (0.37, 0.66)	<0.01	0.50 (-0.24, 1.24)	0.18
Diagnoses
Anxiety/panicc	−1.00 (-6.14, 4.14)	0.70	2.33 (-0.44, 5.09)	0.10	0.30 (-2.89, 3.49)	0.85	6.37 (-5.84, 18.59)	0.31
Major depressionc	−1.62 (-4.51, 1.27)	0.27	0.26 (-1.59, 2.10)	0.79	−0.14 (-1.99, 1.72)	0.89	−3.65 (-12.56, 5.26)	0.42
PTSDc	−2.46 (-5.35, 0.44)	0.10	0.62 (-1.23, 2.47)	0.51	0.62 (-1.23, 2.48)	0.51	1.78 (-6.81, 10.36)	0.69
Schizophrenia/bipolarc	5.08 (0.62, 9.54)	0.03	2.23 (-0.74, 5.20)	0.14	3.21 (0.92, 5.49)	<0.01	16.21 (1.54, 30.88)	0.03

PAM=Patient Activation Measure [16].

Abbreviations: STAR-P= Scale to Assess the Therapeutic Relationship-Patient [20]; PEPPI=Perceived Efficacy in Patient-Physician Interactions [18]; CAHPS=Consumer Assessment of Healthcare Providers and Systems [19]; eHeals = eHealth Literacy Scale [15].

^aTobit mixed model was a better fit to the data than the linear mixed model; tobit model presented here.

^bSpecialty mental health visits in the previous 12 months.

^cDiagnosis in the previous 12 months.

DISCUSSION

In this study, we examined the impact of a web-based educational program on the patient-clinician relationship and patient engagement in care among a group of patients receiving mental health care who read their progress notes online. Bivariate analyses indicated significant improvements in patient activation, perceived efficacy in healthcare interactions, and patient trust in clinicians over time, suggesting that our web-based educational program on OpenNotes helps patients who read their mental health notes feel more empowered in their healthcare and may improve their perceptions of the relationships they have with their clinicians. As such, this web-based program could be a useful tool in mental health settings that provide patients online access to their progress notes, where clinicians may be concerned about the potential for online progress notes to negatively impact their relationships with patients.^9,10 In adjusted models, post-educational program improvements in patient activation and perceived efficacy in healthcare interactions remained significant. Since patients report feeling hesitant to talk about their notes with their clinicians,¹² this increase in patient activation and healthcare efficacy may help patients get the most out of OpenNotes and improve communication about notes with their clinicians. However, in adjusted models, the positive change in patient trust in clinicians was no longer significant, suggesting that more may be needed to positively affect change in the patient-clinician relationship in the context of OpenNotes for some patients. More work is needed to investigate these ideas.

Meanwhile, several covariates were independently significantly associated with study outcomes that might be worth exploring further. Older age appeared to be positively associated with all of the outcomes, suggesting that older patients who read their mental health notes may be more highly activated and have stronger relationships with their clinicians than younger patients. Older patients may have less experience and familiarity with web-based health tools, as veterans who are older are less likely to use the internet.²¹ As such, older veterans who read their notes online may be those who are already especially activated and engaged in their healthcare. However, there are also data showing that older veterans are equally likely to use the VA’s patient health portal and view their notes online as younger veterans.^21,22 Not surprisingly, higher baseline eHealth literacy scores were also frequently positively associated with study outcomes, namely, patient activation, perceived efficacy in healthcare interactions, and perceptions of the therapeutic relationship. Finally, a prior-year diagnosis of schizophrenia or bipolar disorder was significantly associated with increased patient activation, perceived efficacy in healthcare interactions, and trust in clinicians. In prior research on clinician concerns about OpenNotes, clinicians have reported being especially concerned about disruptions in therapeutic rapport with their patients diagnosed with these conditions.^9,10 The current analysis is consistent with other work suggesting that patients with these diagnoses are not more or less likely to experience negative consequences from reading their progress notes online.¹³

Several limitations should be considered in this study. Our sample was derived from one VA medical center in the Pacific Northwest region of the United States. Although this medical center serves veterans across a wide geographic region, at 11 urban and rural sites, our findings may not be generalizable to veterans in other parts of the United States. We also saw significant demographic differences between participants who engaged with the program for at least 5 minutes compared to those who did not. However, our analytic sample was demographically similar to veterans receiving mental health care at the medical center and similar to the general veteran population with regard to race and gender.²³ Further, our findings may not generalize to patients in other healthcare systems, as several aspects of the VA healthcare system are unique (eg, VHA’s service connected disability status ratings). As this was not a randomized trial, it is possible that changes in study outcomes are due to other variables not measured here. Similarly, since participants self-selected into the study, it is possible that selection bias played some role in our study’s findings; patients who desire to be more engaged in their care may have been more likely to participate in this study. Finally, the learning management software limited the type of usage data we could collect, so we were unable to examine outcomes associated with specific sections of the program.

Increased patient access to their medical record progress notes through online patient portals has the potential to increase the meaningful use of health information, but worries some clinicians—especially those in mental health settings—about the potential for negative consequences. Content for our web-based educational program was developed with input from patients and clinicians in mental health settings to help patients optimize the potential benefits of having access to their progress notes online, while reducing the likelihood of negative outcomes. Our findings suggest that this educational program may further empower patients reading their notes online to be active participants in their care, while also providing information and tools that facilitate better relationships with their clinicians. As such, health systems that provide patients online access to their progress notes may wish to offer similar educational opportunities to enhance patient benefit from viewing their notes online.

FUNDING

This material is based upon work supported by the Department of Veterans Affairs, Veterans Health Administration, and Health Services Research and Development Service Project IIR 13-347. Dr Denneson is a Core Investigator at the VA HSR&D Center to Improve Veteran Involvement in Care at the VA Portland Health Care System. The funders did not give input on study design; the collection, analysis, or interpretation of data; the writing of the article; or the decision to submit it for publication. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or United States government.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONTRIBUTORS

Conception and design of study: Dobscha, SK; Denneson, LM; Pisciotta, M

Acquisition of data: Pisciotta, M; Denneson, LM

Analysis and/or interpretation of data: Hooker, ER; Denneson, LM

Drafting the manuscript: Denneson, LM; Trevino, A; Hooker, ER

Revising the manuscript: Denneson, LM; Trevino, A; Pisciotta, M; Hooker, ER; Dobscha, SK

Approval of the version of the manuscript to be published: Denneson, LM; Trevino, A; Pisciotta, M; Hooker, ER; Dobscha, SK

Conflict of interest statement. None declared.