Engaging patients and family members in the evaluation of a mental health patient portal: protocol for a mixed-methods study

Abstract

Introduction

Twenty per cent of Canadians will experience a mental illness in any year. Mental health patient portals have been developed to support these individuals in taking more control over their own mental health and care. This may be done through electronic access to their health records and other supportive functions like completion of online self-assessments. To date, there has been limited research into the value that these portals may provide within mental health contexts. This study will identify what value mental health patient portals may offer to patients and their family members.

Methods and analysis

This study will use a mixed-methods design. Patients will complete a survey consisting of validated instruments at the time of enrolment in the portal, and at 3 and 6 months of portal use. Patient and family member focus groups will be conducted. Portal usage data will be collected to identify if there are differences in outcomes based on usage. The study will be done at Canada’s largest mental health and addiction teaching hospital, and will be conducted using a patient and family-oriented research approach, engaging these important representatives in all stages of the research process. The primary data analysis for the survey portion of the study will be done using linear mixed-effect models, assessing the differences between patients with different portal usage levels. A thematic analysis will be conducted of the focus group transcripts.

Ethics and dissemination

Approval from the study site’s Research Ethics Board has been obtained. The dissemination of findings of this study will be done through presentations at conferences, as well as a formal peer-reviewed journal article. Additionally, the research team will work with a group of patients and family members to identify opportunities to complete knowledge translation and dissemination activities in non-traditional venues.

Strengths and limitations of this study.

• Patient and family member representatives have been engaged in the development of the study protocol, and will be involved in all aspects of data collection, analysis and dissemination.

• This is the first known study to evaluate patient-identified outcomes of patient portal use over a period of time within a mental health context.

• This study will be done using a specific patient portal technology at a single site, and thus the generalisability of the findings is unknown.

Introduction

By the age of 40, half of Canadians have or previously have had a mental illness.¹ It is therefore not surprising that mental illness is the leading cause of disability in Canada and the top priority of numerous organisations across the country.² Despite the large number of individuals afflicted by mental illness, the Mental Health Commission of Canada has stated that ‘… using technology to control, detect, screen, or treat an illness is seemingly common. But not for mental health problems or mental illness. Technology in this area is not as widely used or invested in’.³ Due to the limited use of technology in mental health organisations, there is also a lack of research into the value that these technologies may play in supporting mental health. The limited examination of this topic further contributes to mental health organisations not having the evidence to be able to justify the expenses associated with implementing such technologies, which has resulted in mental health organisations falling behind their counterparts with regards to technology adoption.^{3 4}

One technology that could have benefits for people suffering from mental illness is a mental health patient portal (MHPP).^5–8 A MHPP is a secure online website that allows patients access to their mental health clinical information from a particular healthcare organisation or system. Commonly, a MHPP is tethered to an electronic health record system where numerous health professionals have documented clinical notes. Information included in a MHPP may include (1) lab results; (2) health professional clinical notes, for example, discharge instructions; (3) medication information; (4) results from other tests; and (5) a list of care team members. MHPPs may also have other functions such as being able to (1) send a message to a health professional, (2) request a prescription renewal, (3) book an appointment, (4) be reminded of an appointment, (5) update personal contact information and (6) answer questionnaires to support health monitoring and care delivery.^9–11

At times, patients may want a family member to help them with their appointments or with accessing care.¹⁰ MHPPs often allow patients to give permission for a family member and/or caregiver to access their portal, enabling the family member to see some, or all, of the patient’s personal health information.

As of 2018, there are 19 known health organisations in the world that share mental health clinical notes with patients, and only 2 of these organisations are located in Canada.^{12 13} Further, there is limited evidence to date to identify the value that these MHPPs may provide for patients and their family members. Findings from the few studies of portal implementations have suggested that MHPPs may support improved mental health recovery,⁶ improved sense of empowerment,⁶ enhanced trust with health professionals and better communication with health professionals.^{12 13} However, sufficient evidence to support these potential benefits is lacking. Patient portals used in non-mental health contexts have shown value for patients in managing symptoms such as those related to taking medications¹⁴; however, it has been argued that improvements in functioning could be a more meaningful outcome than addressing symptoms for people suffering from mental illness.¹⁵ In addition, there is a growing body of literature highlighting the important role that families can play in supporting someone with mental illness.^{15 16} No known studies have been conducted to explore whether and how MHPPs offer families value in doing so. This study aims to address these current gaps in the literature.

Methods and analysis

Study objectives

The overall objective of this study is to determine what value MHPPs offer to patients and their family members. Primary, secondary and exploratory objectives are listed below.

Primary objectives

1. Determine if MHPPs are associated with improved mental health outcomes.

   1. Identify if improved functioning is associated with the use of a MHPP.
   2. Identify if mental health recovery is associated with the use of a MHPP.

Secondary objective

• 2. Identify if patient perceptions of empowerment, trust and communication with health professionals change following the use of a MHPP.

Exploratory objective

• 3. Describe patient and family member perceptions of whether and how a MHPP offers them value.

Frameworks

Two frameworks have been used to inform this study: (1) the Value-Based Health Care (VBHC) Delivery Framework has informed the conceptualisation of this study¹⁷ and (2) the Strategy for Patient-Oriented Research (SPOR) Patient Engagement Framework from the Canadian Institutes of Health Research has informed the methodological approach for this study.¹⁸ The central tenet of VBHC is that improving health outcomes must come from improving the value of care delivery.¹⁷ Value, defined as outcomes over cost, becomes the overarching goal that unites all stakeholders involved in care delivery for a patient’s medical condition.¹³ Outcomes include needs, wishes and expectations of individual patients based on their unique contexts.¹⁹ While the VBHC approach has been applied in primary care for patient medical conditions such as knee replacement surgery, it has not yet been operationalised for mental healthcare. Given the number of individuals impacted by mental illness, and the resource coordination and costs associated with the current delivery system, redesigning mental healthcare delivery using a VBHC approach could be of enormous benefit to society. In this study, the VBHC framework is complemented with the SPOR Patient Engagement Framework to (1) inform the makeup of the current research team (including collaborators), (2) determine ways for a committee made up of patient and family representatives to meaningfully contribute to this study and (3) engage a Peer Support Worker (a mental health worker with lived experience) in the various aspects of data collection.

Design and approach

The proposed study will use a sequential explanatory mixed-methods design consisting of a series of surveys, MHPP usage data obtained from the software and focus groups with patients and their family members. Patients enrolled in this study will complete a survey consisting of validated instruments at the following three time periods: (1) time of initial enrolment in the MHPP (baseline, T0); (2) 3 months of MHPP use (T1) and (3) 6 months of MHPP use (T2). These time periods were selected based on the literature that suggests that there may be an increased sense of ‘hype’ in the first couple of months when technologies are implemented and thus patients and their family members may have a different level of MHPP use during this time frame than they would otherwise.²⁰ As well, it may take a number of months for there to be a noticeable change in the selected mental health outcomes measured in this study.⁶ Individual usage data of the MHPP will be collected on each participant from the portal software outlining the extent (frequency) and nature of their usage from the time they started using the portal until 6 months. Additionally, focus groups will be conducted with patients and family members who have used the MHPP for a minimum of 3 months. This study will take place over a 2-year time period. A summary of the design and approach is shown in table 1. A Patient and Family Advisory Committee will be engaged during each stage of the research process (planning, execution and dissemination) in order to ensure the relevance, meaningfulness and feasibility of the study. This committee will consist of two patient representatives and two family member representatives.

Table 1.

Overview of data source, variables and timeline

Data source	Variables
	T0 (baseline)	T1 (3 months)	T2 (6 months)
Survey	Functioning Mental Health Recovery Empowerment Trust with health professionals Communication with health professionals Demographic data	Functioning Mental Health Recovery Empowerment Trust with health professionals Communication with health professionals	Functioning Mental Health Recovery Empowerment Trust with health professionals Communication with health professionals
Focus groups		Patient and family focus groups
Usage data	Number of accesses per month; functions of the patient portal accessed

Setting

This study will be conducted at Canada’s largest mental health and addiction teaching hospital located in Toronto, Ontario.²¹ The study site employs physicians, nurses, occupational therapists, social workers, pharmacists, recreation therapists, personal support workers, behavioural therapists, peer support workers and a variety of other health professional groups relevant to mental health clinical care. Patients served at the organisation range from children to the elderly, and vary in terms of their mental health diagnosis (eg, depression, schizophrenia, schizoaffective disorder, concurrent disorders, etc). Mental health services are offered through inpatient, outpatient and partial hospital programmes. The organisation has the only stand-alone mental health emergency department in Canada.

The study site implemented a comprehensive electronic health record in 2013 with computerised provider order entry, clinical decision support, electronic medication administration with bar code technology, clinical documentation, electronic care planning and laboratory results reporting and viewing.²² The organisation obtained stage 7 on the Healthcare Information Management Systems Society Electronic Health Record Adoption Model in 2017.²³ In late 2017, the organisation initiated efforts to introduce a patient portal tethered to the electronic health record, with a phased-in approach to the various portal functions such as access to documentation, self-assessments and viewing test results.

Sample and sample size

Patient participants will be eligible to participate in the survey portion of this study if they (1) have enrolled in the MHPP at the study site and (2) have had access to the MHPP for <2 weeks. Patient participants will be eligible to participate in the focus groups if they completed surveys at both baseline/T0 and T1. Family member participants will be eligible if their family member is registered in a MHPP at the study site. All participants in the study will be over the age of 16.

Sample size

For the primary objective (objective 1a), a sample of 68 participants provides 80% power to detect a drop of 30% in the total WHO Disability Assessment Scale (WHODAS) V.2.0²⁴ 12-item score from baseline/T0 to T2, using a small to medium effect size (Cohen’s d=0.27). This power calculation also assumes a paired comparison in a pre-test post-test design with a correlation between baseline/T0 and T2 WHODAS V.2.0 score of 0.7, with a confidence level of 0.05 and two-tailed tests. Based on a previous MHPP study, a 30% dropout rate is expected between T0 and T2, and therefore a minimum of 97 participants should be recruited to obtain 68 participants who complete all surveys at the various time points. To be conservative, this study will aim to obtain 100 participants. The change from baseline/T0 in per cent and the power calculation was based on the data means and SD for patients with mental health conditions²⁵ conducted with G*Power V.3.1.9.2.²⁶ Likewise, a change of 8% in the Mental Health Recovery Measure (MHRM) scale can be detected with such sample size and 80% power, based on these data.²⁷

For the exploratory objective (objective 3), four focus groups will be conducted with approximately 6–10 participants in each, totalling 24–40 participants. This number has been shown in past research to be an adequate sample size to obtain meaningful data for patient portal research.²⁸

Recruitment

Participants will be recruited to participate in the survey portion of this study through the following three ways:

1. When participants are provided with a pamphlet describing how to enrol in the MHPP, a recruitment flyer will be attached to the back of this pamphlet.

2. As part of the registration process for the MHPP, patients will be emailed a registration link. At the bottom of the email with the registration link, recruitment information will be present.

3. When MHPP users sign on to their portal, the homepage will contain recruitment information for the study.

Once participants indicate they are interested in participating in the surveys, a registration link will be sent to them by a research assistant. This registration link contains a study information letter, consent information and data fields to collect the minimum necessary personal identification information to conduct the surveys, such as email address, and name of patient as it would appear on their medical record.

At the end of the survey at T1, participants will be asked if they are interested, or their family member may be interested, in participating in a focus group. Participants enrolled through this approach will be provided with the logistical information so that they can decide which focus group to register for should they wish to do so. Recruitment methods for the survey portion of the study will also be used to recruit family members to the study.

Data sources and collection procedures

The primary and secondary study objectives (objectives 1a, 1b and 2) will be measured using a survey at three time points (baseline/T0, T1, T2) using validated instruments. The survey will be self-administered online through a secure survey website, or by a trained Peer Support Worker if participants would prefer the survey be administered over the phone. If a Peer Support Worker administers the survey, he or she will input the responses into the secure survey accordingly.

Participants will be emailed an electronic gift card by the research assistant each time they complete a survey to thank them for their time. Participants will have the opportunity to decide which gift card they would like to receive from a coffee shop, grocery store or movie theatre. The options for the gift cards were determined by patient and family members who were consulted in the design of this study. If preferred by participants, physical gift cards will also be made available.

Demographic information (age, ethnicity, education, sex) will be collected during the baseline survey (T0). Other variables in the study will be measured using instruments, which are considered ‘gold standard’ in the field. Specifically, the primary and secondary objectives will be measured through a survey at all three time points (baseline/T0, T1 and T2). For primary objective 1a, functioning will be measured using the WHODAS V.2.0 12-item version.²⁴ The reported Cronbach’s alpha for this measure ranges from 0.82 to 0.97.²⁴ For primary objective 1b, mental health recovery will be measured using the MHRM 30-item version.²⁹ The reported Cronbach’s alpha for this measure range from 0.86 to 0.94.^{29 30} For the secondary objective (objective 2), empowerment will either be measured using the Consumer Evaluation of Mental Health Services—Original Version³¹ or a subscale of the MHRM. Trust with health professionals will be measured using the Health Care Relationship Trust Scale—Revised Version,³² and communication with health professionals will be measured using Health Care Communication Questionnaire.³³ Acceptable Cronbach’s alphas have been reported from these measures in previous studies. In addition, the relationship between functioning and mental health recovery and individual patient portal usage will be evaluated. Permission has been obtained to use all instruments in this study.

Monthly usage data will be collected from the study site’s MHPP software for the duration of the study. Specifically, the number of times a patient participant has accessed their record each month, and the functions that the patient uses, will be collected.

The exploratory objective (objective 3) will be measured through two 60–90 min focus groups with patients, and two 60–90 min focus groups with family members once they have used the portal for a number of months. Each focus group will consist of between 6 and 10 participants, be audio recorded and be facilitated by a trained Peer Support Worker using a semistructured interview guide. All focus group participants will be provided with a cash honorarium as a thank you for participating in the focus group, and funds to cover public transportation costs.

Data analysis

Survey

Quantitative data analysis will be completed for the primary and secondary objectives (objectives 1a, 1b and 2), which were to identify how functioning, mental health recovery, empowerment, trust and communication with health professionals may change following the use of a MHPP. Analysis will begin with a description of the sample at T0 and at T2, and will be conducted for all metrics that are relevant to the study. A description of the demographics at T0 will also be conducted. A comparison between participants who dropped out and those who completed the surveys will be made using Fisher’s exact test for categorical variables, and the Mann-Whitney U test for continuous variables. Participants will be classified into groups according to the frequency that they used the portal (low, medium and high) using the MHPP monthly usage data, with a focus on forming three groups of similar sizes. The comparison of these groups will provide additional evidence for how or if the use of a MHPP may influence certain outcomes. The statistical analysis of the primary objectives 1a and 1b will be done using linear mixed-effect models, where individual participants will be treated as random effects and the main effect of the portal will be estimated by the fixed effect of time. Time will enter the model as a categorical variable with three levels (baseline/T0, T1 and T2), and a linear contrast will be used to test the change in the outcomes (WHODAS V.2.0 and MHRM) from baseline/T0 to T2.

For the secondary objective (objective 2), the differences between baseline/T0 and T1, and between T1 and T2 will also be tested. All quantitative data analyses will be conducted using SAS V.9.2 (SAS Institute).

Focus groups

A qualitative data analysis will be completed for the exploratory objective (objective 3), which was to describe patient and family member perceptions of whether and how a MHPP offers them value. This will be done using thematic analysis drawing on Braun and Clarke’s six steps: (1) familiarising yourself with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes and (6) producing the report.^34–36 All audio transcriptions from the focus groups will be transcribed verbatim and uploaded into NVivo V.11 Pro (QSR International, Burlington, Massachusetts, USA) for data analysis. To enhance the trustworthiness of the analysis, a member of the research team and a research assistant will independently complete data analysis of the focus groups.^{37 38} They will then meet to compare their initial codes (including coding hierarchy); identifying similarities and differences among results and tracking these analytical findings in study memos. Inter-rater reliability will be calculated using Cohen’s kappa. If there is significant disagreement regarding the themes, the two participants will jointly recode sections of the transcripts to resolve thematic differences.

Patient and public involvement

The conceptualisation of this study, which includes both the determination of the research questions and methods described, was done in collaboration with a patient and family member representative prior to the submission for research funding. The study planning and execution involves a patient and family advisory committee with two patient and two family member representatives.

Ethics and dissemination

This study has received ethical approval by the study site’s Research Ethics Board. For the survey portion of this study, several decisions have been made to ensure that data collection is carried out in an ethical manner. When participants enrol in the survey portion of the study, they will have an opportunity to take as much time as they need to read an electronic study information letter, and ask questions of the research team before beginning the survey. The voluntary nature of the study will be communicated in this letter. Once participants have begun to fill in the survey, they can decide to stop at any point without penalty. Participant responses to the survey will be collected via an online survey platform. All data in the online survey are stored on a secure server at the study site which enhances the security of participant data.

With regards to the focus group portion of the study, informed consent will be obtained prior to the focus groups beginning. Participants will receive a copy of the study information letter and informed consent via email in advance of the focus group. Hard copies will also be available when they arrive for the focus group. Participants will have as much time as they need to read the document and ask any questions before the focus group begins. During the focus group, participants will be asked not to use any identifying information. If someone accidentally uses identifying information, it will not be transcribed from the audio. A pseudonym will be used instead in any transcriptions and any reporting of the study results. Focus group transcriptions will also be kept in a secure research drive at the study site with access only being provided to the research team. Hard copies of consent forms from the focus groups will be kept in a locked filing cabinet in a locked room (study principal investigator’s office) at the study site.

A multipronged approach will be used to disseminate the findings of this study with relevant audiences. The committee of patient and family representatives will identify appropriate venues and types of materials for knowledge translation and dissemination activities. In addition, this group will advise the development of these materials so that they are relevant to the target audience.

Traditional dissemination strategies will also be used. The research team will share the findings of this study in an international field-specific peer-reviewed journal and will present the findings at relevant local, national and international conferences as appropriate. Additionally, to target mental healthcare administrators with technology decision-making responsibilities, an article may be written in a trade publication such as Canadian Healthcare Technology.

Conclusion

Once completed, this study will provide insights from patients and family members into the value that MHPPs may provide for these groups. The findings will specifically identify if use of a MHPP is associated with certain outcomes. As portal technology may be expensive and complex to implement within the mental health context, this study will provide some initial findings for organisations to consider when deciding whether they should implement and adopt MHPPs. By having patient and family member representatives in all stages of study operationalisation, both the relevance and feasibility of the research will be enhanced. This research is a first step in understanding the potential outcomes of technology use within mental health settings.