Development and Evaluation of an Online Mental Health Program for Medical Students

Abstract

Objective:

Medical school presents a time of psychological distress for many students, who are less likely than the general population to seek mental health treatment due to multiple treatment barriers. Internet-based cognitive behavioral therapy (iCBT) programs may be an acceptable option for medical students. This study aims to assess acceptability and usability of ThinkFeelDo, an iCBT program, and to examine the impact of the program on perceived stress, quality of life, and the development of cognitive and behavioral coping skills.

Method:

Fourteen medical students (M age = 25.4 years, 50% female) participated in the 6-week ThinkFeelDo program and completed baseline and end of treatment assessments.

Results:

ThinkFeelDo was used (login M = 11.9, SD = 9.8) and was rated as somewhat useful. Participants requested further refinement of lessons to better fit the typical narrative of a medical student, and endorsed interest in the program being offered at the beginning of medical school. At end of program, participants increased the frequency with which they used cognitive and behavioral coping skills, t(10) = −3.400, p = .007.

Conclusions:

Results of this study indicate that medical students are willing to utilize online mental health programs and may receive benefit. However, the sample was small, self-selected, and without a comparison group. Feedback collected through this study provides insight on how to effectively integrate iCBT programs into the medical school experience.

Medical school presents a period of psychological distress for physicians-in-training who experience significantly higher rates of depression and anxiety compared to the general population [1, 2]. Despite having seemingly easier access to care, medical students are less likely than the general population to received appropriate treatment [3] and report high levels of barriers to mental health treatment [1, 4]. Untreated depression presents a public health concern not just for trainees, but for the general population as it has been associated with increased burnout, poor quality patient care, and a decline in the physician workforce [5, 6].

Internet-based cognitive behavioral therapy (iCBT) programs may reduce barriers to treatment for medical students. These programs can offer increased flexibility and privacy for users relative to traditional face-to-face treatments, as they can be accessed any time of day or night, from wherever the user feels comfortable accessing the internet. Over the years, iCBT has repeatedly demonstrated efficacy for treating a wide range of mental health concerns [7].

Here, we present a single arm pilot study of an internet-based cognitive behavioral therapy (iCBT) program for medical students. The primary aims of this study were to gather information regarding the acceptability and usability of the program, and to gather user feedback to inform program development. Secondary aims of this study were to examine the impact of the program on perceived stress, quality of life, and the development of cognitive and behavioral coping skills.

Methods:

Recruitment

Participants were recruited from the University of Illinois medical school campuses, and this study was approved by the University of Illinois at Chicago Institutional Review Board. Recruitment was conducted through the medical school’s student government. All University of Illinois medical students were eligible to participate.

Procedure

Students participated in a 6-week iCBT program. Participants completed an online consent form prior to participation, and completed online questionnaires at baseline and end of treatment. Study data were collected using Research Electronic Data Capture (REDCap), a secure web-based survey tool hosted at the University of Illinois at Chicago [8]. Participants also completed an end of treatment semi-structured interview with authors EGL and JLD. To ensure privacy and confidentiality, all participant data was labeled with ID numbers rather than names, and only study staff had access to these materials.

The ThinkFeelDo Med Student Intervention

The ThinkFeelDo Med Student program is a mobile-accessible web intervention that consisted of 14 lessons and five tools. The ThinkFeelDo program [9] was developed at the Center for Behavioral Intervention Technologies at Northwestern University. Lessons, on topics such as behavioral activation, cognitive restructuring, and managing anxiety, included text and video and required approximately 10 minutes to complete. The site included five interactive tools designed to support implementation of cognitive behavioral skills. Tools included activity scheduling and monitoring (Activity Tracker), cognitive restructuring (Thought Tool), goal setting, relaxation, and mood tracking.

Outcomes

Site use outcomes include frequency and duration of overall site use, in addition to frequency of specific components within the site. Usability and user feedback outcomes were assessed with the USE questionnaire [10] and a 9-item semi-structured interview conducted over the telephone. The Perceived Stress Scale (PSS) [11] was used to measure subjective experiences of life stress. The Medical Student Well-Being Index (MSWBI) [12] was used to measure quality of life indicators. The Cognitive and Behavioral Response to Stress Scale (CB-RSS) [13] was used to measure frequency and usefulness of cognitive and behavioral coping skills.

Statistical Analyses

Descriptive statistics were performed on website use data and the USE questionnaire [10] to examine program usage and usability. Paired samples t-tests were run to examine changes on the PSS [11] and CB-RSS [13] from baseline to end of treatment. McNemar tests were run to examine changes in the proportion of participants responding “yes” and “no” to items on the MSWBI [12]. As the analyses of psychosocial measures were exploratory analyses on a prototype program, we a priori considered a type I error of 0.2, rather than a more stringent level of 0.05. This alpha level was set to not dismiss findings that should be explored in a larger study of this program.

Results:

Study sample

A total of 16 students enrolled in the study and completed a baseline assessment. Two participants were lost to follow-up prior to program start, resulting in a final sample of 14 students. The sample was racially diverse and nearly evenly split between males (n = 6) and females (n = 8).

Site Use

During the course of the 6 week trial, participants logged in an average of 11.9 times (SD = 9.8, range: 3 – 35), completed an average of 10 of the 14 didactic lessons (SD = 4, range: 3 – 14), and used at least one tool. Participants tended to log a few thoughts in the Thought Tool (M = 6.64, SD = 5.72, range: 0–16), monitor several activities in the Activity Tracker tool (M = 34.21, SD = 20.82, range: 0–64), and plan at least a few activities in the Activity Tracker tool (M = 5.71, SD = 6.28, range: 0–21). Most participants (n = 12, 85.71%) logged their mood in the mood-rating tool at least once, but repeated use of this tool was low (M = 4.57, SD = 6.44, range: 0–24). Half of the participants (n = 7) used the goal-setting tool, with a mean use of 1.1 times (SD = 1.38, range: 0–4). Only 2 participants accessed the relaxation tool.

Usability and user feedback

Feedback from the USE questionnaire [10] indicated that the program was generally useable in its current form. On a scale of 1–7, in which 1 indicates strongly agree and 7 indicates strongly disagree, the mean usefulness rating was 2.90 (SD = 1.32, range: 1–5) and the mean satisfaction rating was 3.40 (SD = 2.17, range: 1–7). Participants were asked to rate the statement “I would recommend the program to a friend” on the same 1–7 scale, and the mean score was 2.90 (SD = 1.73, range: 1–6).

Eleven of the fourteen participants completed a telephone-based feedback interview. Participants noted ways in which lessons directly related to future clinical work, including gaining first-hand experience with techniques such as progressive muscle relaxation and learning how to be more precise and descriptive when talking about thoughts, feelings, and behaviors. Most participants interviewed (n = 10) reported that the program would be useful to students just beginning medical school. Many noted personal tendencies to default to old habits when feeling busy, and stated potential benefits of the program being used to establish healthy habits early in training. Participants were asked to rate their interest in content to include for future iterations of the program. We observed the strongest interest for lessons on time management and burnout, and many students provided additional requests for student-specific situations, such as taking tests, interviewing, and networking.

Participants also shared ideas about how to encourage program use. Many (n = 7) endorsed preference for some expectation that all students “check out” the program as part of a message from the medical school promoting the student mental health. One student noted “If it was required, that means I would do it and that would be the best for me in the long run.”

Psychosocial Outcomes

Participants endorsed a high degree of perceived stress at baseline (M = 32, SD = 5.88) which remained high, yet was significantly lower at end of program (M = 28.75, SD = 6.70), t(11) = 2.120, p = .058, d = .534. Participants increased the frequency with which they used cognitive and behavioral coping skills, (Baseline: M = 33.73, SD = 7.75; End of program: M = 41.45, SD = 8.86; t(10) = −3.400, p = .007, d = −.923). At baseline, participants found the cognitive and behavioral coping skills to be useful (M = 43.36, SD = 8.19), and no change was observed in how useful participants found those skills, t(10) = −.939, p = .37, d = −.182, from baseline to end of program assessment (M = 45.45, SD = 11.93). Quality of life measures remained fairly stable. However, at end of program, fewer participants (n= 2, 18.18%) reported feeling burned out from medical school compared to baseline (n = 7, 63.64%), p = .063.

Discussion:

Results highlight the potential utility of ThinkFeelDo as an online mental health resource for medical students and provide insight on how to effectively integrate iCBT programs into the medical school experience. Program use was higher than is typical with a mean number of logins at 11.9. Past studies of stand-alone iCBT programs for general adult audiences, typically delivered over longer periods of time, have had an average of 2 logins, whereas coached iCBT programs have had approximately 9 logins per user [14, 15]. All participants returned to the site multiple times with minimal support from the research team, suggesting that medical students, as a subpopulation, may use and benefit from module-based online mental health programs. Participants began the intervention at the start of an academic semester, and we anticipated that levels of stress and measures of quality of life, such as burnout, would increase at mid-semester follow-up [16]. However, moderate reductions in stress, large increases in the frequency with which cognitive and behavioral coping skills were used, and significant reductions in the number of students endorsing feelings of burnout were observed – indicating potential value in the further development of a medical student version of ThinkFeelDo. As this was a single arm trial, further research is needed to determine if these changes are associated with program use, and these changes should be interpreted with caution, as there is no gold standard for assessing these constructs.

Participants all began the program at the same time in the calendar year. Due to differences in individuals’ schedules, the varying time commitments required for specific tests and clinical rotations presented barriers to program use for some participants, highlighting the potential value of flexible start times or open availability based on student needs. The program was rated as somewhat usable in its current form with mean scores on each of the USE subscales being higher than 4 (indicating a neutral response). Several participants commented that they might have used the program more frequently if prompted to do so from their mobile phones, rather than accessing the program exclusively via a computer. Consistent with findings by Renton and colleagues [17], computer accessibility may have served as an access barrier. This barrier could be ameliorated by more clearly making users aware of their ability to access the program via their mobile phones in the initial instructions. Participants in this trial endorsed desire for further refinement of standard cognitive-behavioral lessons to better fit the typical narrative of a medical student, such as more in-depth discussion of anxiety management techniques for common stressors (e.g., taking tests, delivering presentations, interviewing, and networking). The use of this type of targeting in intervention design can support continued engagement and benefit [18].

A key question to guide further work was “When, and how, can this program be useful to students?” Due to a known tendency to default to familiar habits, when feeling stressed and busy, the majority of participants suggested the program be offered at the beginning of medical school. In line with the known importance of “the hidden curriculum” in medical education, which includes how students are socialized to the field of medicine [19], many participants suggested that an expectation be set that all students begin the program, framed as a message from administration emphasizing the value of good mental health throughout training.

While this study demonstrates feasibility for an iCBT program for medical students and suggested clinical benefit, the sample was small, self-selected and did not include a control condition. There may have been a selection bias, in that participants may have opted into this study for reasons that make them different in some way from a general medical student sample. To determine the efficacy of this program, a larger randomized controlled trial of an optimized version of ThinkFeelDo for medical students will need to be conducted and additional needs-assessment should be done to determine if this program is best suited as a preventive intervention or treatment program. Future iterations of iCBT programs for medical students may benefit from: further tailoring to include a medical student narrative, designing the website for easy use from multiple devices (e.g., computers, tablets, smartphones), and granting students access to the program early in their medical school education.