Impact of digital literacy training on outcomes in serious mental illness in community and inpatient settings

Abstract

Objective:

Focusing on people with serious mental illness, we explored how one digital literacy program, Digital Outreach for Obtaining Resources and Skills (DOORS), may be able to improve self-reported functional skills and clinical outcomes.

Methods:

The eight-week program was offered to participants in mental health community centers (n=113) and an inpatient psychiatric unit (n=74). Pre and post self-report measurements were collected. Descriptive statistics and two-tailed t-Tests were used for analysis.

Results:

Improvements in 27 of the 29 self-reported functional skills measuring changes in digital literacy were seen. Seven of these were statistically significant. Although community center participants reported larger improvements in clinical outcomes than inpatient participants, no statistically significant changes in symptoms were seen in either setting.

Conclusion:

Digital skills training is necessary to increase access to care through technology. DOORS can improve self-reported digital literacy, but further research is necessary to determine their immediate impact on symptoms.

Introduction:

The need for digital literacy training has become more apparent around COVID-19 and the increased reliance on technology in all facets of care. While access to technology remains an issue for some, the second digital divide of knowledge, skills, and confidence is now greater. Ensuring that all patients are able to engage with digital health is thus not only a matter of ensuring access to healthcare but also equity. This paper explores a digital literacy program aimed toward patients with mental illness in both a community mental health and inpatient setting.

As more healthcare services are offered virtually, digital literacy has become recognized as a social determinant of health [1]. Patients with limited digital literacy access telemedicine services at lower rates [2]. The issue is pressing in mental health as telehealth and virtual visits expand, and likely will become a permanent facet of care, because of changes sparked by COVID-19 [3]. For patients with serious mental illness (SMI) who often need the most services, but on average have less education and may experience cognitive impairment, the ability to access digital health is a critical issue.

Access to digital health for patients with mental illness is limited most by digital literacy. While computers are often associated with telepsychiatry, mobile devices like smartphones are the primary modality through which patients are accessing digital health services [4, 5]. It is already well established that people experiencing mental illness own smartphones at rates nearly as high as the general population [6, 7]. This trend continues today; for example, a 2021 study of digital skills among people with schizophrenia and bipolar disorder found that over 85% of participants owned a digital device. However, 42% lacked foundational skills around using those devices as measured by the Essential Digital Skills framework. [8].

This lack of digital literacy has increasingly been recognized as a primary barrier to mental health acquisition. Despite higher clinical needs due to the COVID-19 pandemic, reports suggest that individuals with schizophrenia are now attending fewer appointments [9] and that only 5% of the population connected to mental health services for the first time during the pandemic [10]. While access to care is a complex and multifaceted issue, gaps in technology literacy are large contributors [11].

As one solution, even before COVID-19, our team has offered digital literacy training for mental health through a program called Digital Outreach for Obtaining Resources and Skills (DOORS) [12]. DOORS offers eight-weeks of group-based digital skills training with the goal of teaching participants skills facilitated through their smartphones [13]. During COVID-19, the program expanded and we created an online version (skills.digitalpsych.org).

In seeking to further improve DOORS, we sought to research its impact with the primary goal of assessing changes in self-reported digital literacy with an exploratory aim around self-reported functional skills. We also sought to assess if learning digital literacy skills was associated with transdiagnostic improvements around problem solving, feelings of control, anxiety, and mood-related symptoms. Thus, in this paper, we explore how targeted digital skills training impacts functional and clinical outcomes.

Methods:

DOORS was offered at in-person groups led by our team of trained digital navigators who completed a 10-hour training [14] that ensures they are able to facilitate and lead DOORS groups effectively.

All DOORS sessions in this paper were conducted in facilities in Boston, MA from July to November 2021. The program was offered in two settings: 1) outpatient mental health community centers known as clubhouses and 2) an inpatient psychiatric unit (IPU). Clubhouses received the traditional eight-week curriculum, while IPU participants received a modified curriculum which involved only one lesson on a weekly repeating basis due to the high rate of patient turnover from discharges. The clubhouse DOORS sessions were each 90 minutes in length, while the IPU sessions were 45-minutes long to accommodate the workflow of the unit. Personally identifiable information was not collected and in line with guidance from site leaders at the community and inpatient sites. This study did not collect written informed consent, as this is an anonymous survey that collects no personal healthcare information apart from age. Verbal consent was obtained.

Measurement tools:

Past iterations of DOORS have focused on either wellness goals or functional outcomes [12, 13]. We adapted these survey tools to assess for changes in skill acquisition, confidence, knowledge, and mental health related outcomes (see online supplement). The survey questions used to measure the mental health related clinical changes were informed from scales utilized in single-session intervention studies [15]. These scales measured social functioning, negative thought patterns, hope, mood, problem-solving, and anxiety with the goal of assessing symptoms as well as potential underlying factors related to symptoms (e.g., problem-solving). Participants were instructed to answer clinical survey questions based on their current state. Functional survey questions were adapted from our previous survey measurement tool [13].

Pre- and post-survey data were analyzed through descriptive statistics and two-tailed t-Tests. The lack of a gold standard for measuring digital literacy in SMI led us to rely on our research with self-reported scales for determining thresholds. We set digital literacy skill deficiency to be below 50% of the mean, sufficiency to be within one standard deviation of deficiency, and proficiency to be higher than one standard deviation from deficiency. This threshold was determined by our team because, to our knowledge, there are no clear standards or validated metrics outlining digital literacy proficiency in patients experiencing SMI.

Results:

113 pre-surveys and 87 post-surveys were collected from the clubhouse sessions, while 74 pre-surveys and 52 post-surveys were collected from the IPU sessions. The clubhouse study cohort had an average age range of 45–54 years old, while the IPU age average was 35–44 years old.

Clinical

The participants from the IPU self-reported significantly higher initial anxiety, depression, and stress scores as compared to the clubhouse group (p < 0.05). But differences in initial clinical survey questions regarding secondary control, problem-solving, and internal motivation did not reach significance.

Overall, there were no statically significant differences in any clinical outcomes when measured before and after each session for either the clubhouse or IPU participants. As shown in the online supplement, clubhouse participants did report feeling that they were more motivated to use their smartphone as part of their recovery, better able to solve problems, and more optimistic. Average survey results from IPU participants revealed an increase in optimism. Both clubhouse and IPU participants had an average decrease in anxiety, anhedonia, and feeling that things are outside of their control. Further details on pre- and post-averages for these clinical outcome variables in each setting are available online. The greatest difference in change in survey scores between the IPU and clubhouses was seen in questions regarding stress and problem solving, as shown in the online supplement.

Functional

Clubhouse participants responded to 3–4 unique functional survey questions at each of the 8 sessions. Changes in pre- and post-survey averages for this population are shown below in Figure 1. The average pre- and post-survey scores for the statistically significant functional survey questions are available online. Of the 29 functional survey questions measuring changes in digital skills literacy, 27 questions showed overall improvements, with statistically significant (p < 0.05) improvements visible in seven of 27 the questions. Session 7 of the curriculum, which educates participants on how to navigate smartphones safely, showed the most improvement in functional skills as compared to any other session. The three skills include the ability to ‘find an app’s reviews and ratings’, to ‘find an apps privacy policy’, and to ‘differentiate between apps that protect their data and those that do not.’

Fig 1.

Changes in Digital Literacy Functional Outcomes in Clubhouses. Statistically significant differences with a p < 0.05 is indicated with an asterisk.

Clubhouse participants initially scored poorly (i.e., displayed deficiencies) on 24 of the 29 (83%) functional skills taught, indicating how urgent the need for digital skills training is to achieve the core competencies needed for technology implementation into care. After offering our digital literacy curriculum to the clubhouse participants, 25 of the 29 (86%) functional skills were assessed as sufficient and of those 25 skills, 12 (48%) were assessed as proficient.

IPU participants, however, showed no deficiencies for functional skills, with all pre-survey averages possessing a value of greater than 6.5. Although overall improvements were seen between pre- and post-survey functional questions, no statistically significant differences were observed, as shown in a table available online.

Discussion:

As the need for digital literacy training in mental health expands, programs like DOORS offer a ready-to-utilize solution. Our results suggest that DOORS can improve these functional digital literacy skills in patients with SMI and IPU patients. While our results do not support that DOORS sessions can convey clinical benefit as a single session intervention, they do suggest feasibility with trends that should be explored in larger powered studies. Future studies should consider other design methods, as this immediate pre-post design to measure improvements in clinical outcomes proved difficult to detect.

Our results highlight the importance of digital literacy training. Clubhouse participants demonstrated substantial deficiencies in their initial digital skills knowledge. Given the flexible nature of DOORS, we can use these results to adapt the program to make it more effective. Our results also suggest that some functional skills, such as adding a contact or making a phone call may be too rudimentary to teach. This, however, presents a dilemma as any single skill may be most important to that individual and this cannot be captured in averages. One solution we plan to explore is a “key to DOORS” course to teach those fundamental skills separately to those who need them while covering more advanced skills like downloading apps through the main programming.

Our findings around functional skills for the IPU participants must be interpreted differently as we taught one session in a repeated fashion to match the flow of patients on and off the unit. These IPU participants had a high pre-survey average for all functional skills and thus their lack of improvement may in part be related to a ceiling effect. This suggests that teaching more advanced skills than those taught in the clubhouse groups may be appropriate. However, the short lengths of stay and patients often not being able to attend the entirety of the session poses a challenge.

Some results are more challenging to understand. For example, certain survey results revealed a decrease in reported digital skill comfort and knowledge post-session. Although we cannot determine what factors alone caused the perception of skillset to change, potential attributions include 1) an initial overestimation of skills, 2) challenges with group learning, or 3) external disturbances that pulled participants away from the session. This highlights the need for better assessment tools for digital literacy beyond the self-report methods utilized here.

Our study also has several weaknesses that must be acknowledged. Since the post-survey was conducted immediately following the educational session, long-term knowledge retention and real-world impact or benefit were not measured. Of note, we developed our own survey scales and thresholds due to the lack of validated scales and metrics for digital literacy in patients experiencing SMI. The need for the creation of these scales and metrics is evident. Another limitation is the lack of a control group. Participants having to leave groups during the session resulted in some missing post-survey data and may have affected outcomes. Finally, this study was not designed to be powered.

Despite these limitations, our approach has several strengths. The material to run DOORS groups is publicly available and easy to customize. The real-world use case of DOORS reported in this study, done even with the challenges related to the pandemic, suggests the flexibility necessary to offer the program in diverse settings. Future iterations of this study will aim to add control groups completing the curriculum online to assess its utility compared to the in-person sessions.

Conclusion:

The DOORS digital literacy curriculum can help reduce this digital divide and improve functional skills in community mental health settings for people with SMI.

Highlights:

• Digital literacy training can lead to improvements in functional digital skills for patients experiencing serious mental illness

• Functional skills related to navigating technology safely were among the skills that had the largest improvement in patients

• Participants showed sufficient knowledge in 86% of functional skills taught