Development and Implementation of a Virtual Clinical Skills Examination in General Psychiatry

Abstract

Objective

During the COVID-19 pandemic, psychiatry programs have administered the Clinical Skills Evaluation (CSE) through videoconferencing. The authors evaluated the feasibility and appropriateness of administering virtual CSEs.

Methods

Virtual CSEs were administered to 11 general psychiatry residents on March 16, 2021. Teleconference software was used to connect faculty at work sites, residents at a simulation center, and volunteer patients at home. Before and after the CSE, residents and faculty were surveyed with Likert scale questions to evaluate their perceptions and experience.

Results

All virtual CSEs were completed successfully. Nine residents (82%) and 12 faculty (92%) responded to both surveys. Most participants (range, 67–83%) indicated that the virtual CSE was appropriate for assessing patient health and resident skills. Most participants (range, 56–100%) reported that the opening and closing of the interview, informational and affective cues, and rapport were adequately assessed. All participants agreed that suicidal and homicidal risks could be adequately assessed. Most faculty and residents (76%) believed that unique skills were required for telehealth interviews. Before the CSE, more faculty than residents believed that they received adequate training for the virtual CSE (P=.02); afterward, most participants thought that training was adequate (P=.46). More faculty than residents reported increased convenience with virtual assessments (both surveys, P<.01).

Conclusion

Virtual CSEs were deemed feasible and appropriate. Further research is needed to identify the specific skills required to perform a virtual CSE and to clarify the potential limitations and benefits of this format.

The COVID-19 pandemic resulted in a dramatic shift in psychiatry residents’ curriculum and clinical training toward virtual options. Many medical educators saw this period as an opportunity to adjust [1] and modernize [2] training, and suggestions from trainees across the world have been published [3, 4].

To accommodate these unique circumstances, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Psychiatry and Neurology (ABPN) allowed the Clinical Skills Evaluation (CSE) to be completed virtually. The completion of 3 CSEs has been a requirement for residency graduation and ABPN board eligibility since 2008. In April 2020, the ABPN allowed trainees who completed virtual CSEs to sit for the initial ABPN certification examination. This policy remained in effect at least through June 30, 2022 [5].

The risks and opportunities of the increase in telehealth during medical training have not been thoroughly investigated. Telehealth skills (e.g., telepresence) include not only technical skills but also the ability to use verbal and nonverbal communication skills to offset the lack of touch and other sensory experiences during a telehealth visit. A lack of telehealth skills could negatively affect resident performance on a virtual CSE. The virtual CSE has not yet been validated as being equivalent to an in-person CSE. Therefore, the virtual CSE may have limitations compared with an in-person examination for trainees and examiners.

The current study investigated the appropriateness and feasibility of a virtual (video-based) CSE. Virtual CSEs potentially can be used to evaluate several telehealth competencies through direct observation, and they provide trainees with opportunities to receive formative feedback. Therefore, implementation of virtual CSEs could also contribute to development of telehealth skills.

Methods

The study was approved by the Mayo Clinic Institutional Review Board (study ID: 21-001711) and by the Mayo Clinic Education Review Committee. The reporting of this study is in compliance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement [6]. We conducted a prospective cohort study by examining attitudes and experience of trainees and faculty examiners before and after participating in a virtual CSE. Trainees were general psychiatry residents in their second, third, or fourth year of training. Examiners were board-certified Mayo Clinic faculty and adjunct faculty from other institutions. All psychiatry trainees and examiners scheduled to participate in the virtual CSE on March 16, 2021, were included. There were no exclusion criteria. Participants received a consent form via email and oral consent information during the pretest orientation. Participants indicated their consent by completing the pretest survey.

The Mayo Clinic general psychiatry residency program had previously established a standardized simulated CSE experience. The simulated examination is consistent with part II of the ABPN national certification examination, which existed before the residency-based CSE was developed. The primary goals for developing the virtual CSE were to maintain fidelity to the in-person CSE and to ensure the safety and privacy of all participants. The Zoom communication platform (Zoom Video Communications, Inc) was selected for the virtual CSE because it had already been established as the primary Health Insurance Portability and Accountability Act–compliant telehealth delivery platform across the Mayo Clinic enterprise.

Residents and faculty examiners attended orientation sessions before the examination. Sessions consisted of a brief presentation that outlined the purpose of the examination and expectations of conduct. No video visit training was provided during orientation because all faculty examiners and trainees had prior telehealth training within the year preceding the study as part of the transition to virtual care during the pandemic. Residents were assigned to individual rooms in the simulation center to conduct the examination. Technical support personnel managed the virtual meeting spaces to maintain patient privacy and provide technical assistance. The residents presented the case for up to 20 min, and faculty examiners provided verbal feedback on their performance via Zoom. Examiners completed and returned an electronic version of the ABPN-approved grading forms.

Patients were recruited from a pool of volunteers who had previously participated in CSEs at the simulation center. For the virtual CSE, patients remained in their homes and used their personal device for the CSE. Residents interviewed patients from the simulation center, and 2 faculty examiners per resident connected synchronously from offices located elsewhere on campus or at another site. Faculty were instructed to turn off their cameras during the patient interview.

Trainee and faculty participants were emailed the surveys 1 h before the examination and within 2 h after completing the CSE. Participants were sent several reminders to complete the survey for 5 days, after which the survey was closed. The pretest survey had 9 items and the posttest survey had 16 items. Surveys asked for demographic information, level of training, and prior experience with the CSE and telehealth. Both surveys used questions with 5-point Likert scales (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree). Scoring for negatively worded questions was reversed.

Survey questions focused on the appropriateness of virtual CSEs (e.g., the resident’s ability to examine a patient, the examiner’s ability to assess the resident). The posttest also included questions regarding specific elements of the ABPN CSE grading form. Four open-ended questions were used to assess positive and negative views, to document technical difficulties, and to assess whether unique skills were required to conduct a telehealth visit. Both surveys provided areas for free-text comments. To improve survey completion rates and to reduce response bias, the survey was designed to take less than 10 min to complete. Participants were allotted dedicated time after the orientation to complete the survey before starting the CSE. Participants were reassured that individual results were confidential and de-identified and would in no way affect their participation or grading of the CSE.

Data were collected and managed using the REDCap (Research Electronic Data Capture) electronic data capture tools hosted at Mayo Clinic. REDCap is a secure, web-based software platform designed to support data capture for research studies, providing multiple benefits for audit trails, data capture, manipulation, integration, export, and analysis [7, 8]. The settings within REDCap were selected to prompt the participant to complete all questions and to not allow duplicate entries [7, 8].

The Likert questions that were asked in both surveys were evaluated for intergroup differences. The Wilcoxon signed-rank test was used to detect differences in the summed values between repeated measurements. Differences in survey responses between trainees and faculty were evaluated with the Mann-Whitney-Wilcoxon test. Statistical analyses were conducted using the R programming language [9] and RStudio [10]. Answers to the 4 open-ended questions were reviewed to identify keywords and characterize themes. Keywords and themes were then quantified (frequency counts).

Results

Eleven trainees and 13 faculty participated in the virtual CSE; of these, 9 trainees and 12 faculty (88%) completed both surveys. Six residents (67%) were in the second year of training, 2 (23%) were in the third year, and 1 (11%) was in the fourth year. Faculty were from Mayo Clinic (Rochester, MN) (n=8 [67%]), Mayo Clinic Health System (Wisconsin) (n=2 [17%]), and other medical institutions in the community (n=2 [17%]). The mean (SD) percentage of clinical time spent on telehealth visits was 36.3% (27.7%), and the percentage was higher for faculty than for trainees. The majority of trainees (n=8 [89%]) had never completed a CSE in the simulation center, whereas 9 faculty (75%) had previously served as an examiner for the in-person CSE at the simulation center.

All 11 virtual CSEs were successfully completed. Technical challenges were reported by 4 trainees (44%) and 4 faculty (33%). The most common technical problem was the patient having initial difficulty connecting to the telehealth video platform (reported by 6 participants [29%]).

Neither survey showed a statistically significant difference in responses between faculty and trainees regarding the appropriateness of the virtual CSE (Fig. 1). The majority of both groups indicated that a virtual CSE was appropriate in the context of COVID-19 restrictions and would remain appropriate after the pandemic. In the posttest survey, there was an increase in participants who indicated that a virtual CSE was not appropriate after COVID restrictions (1 trainee [11%], 2 faculty [25%]). Faculty more than residents reported that the virtual format was more convenient, both before (P<.01) and after (P<.01) the examination.

Fig. 1.

Responses to select survey questions from resident trainees and faculty examiners. Left panel, Survey administered before the Clinical Skills Evaluation (CSE). Right panel, Survey administered after the CSE

Before the CSE, more faculty than residents reported that they had received adequate training for the virtual CSE (P=.02); however, this difference was not significant after the examination (P=.46). Most participants believed that unique skills were required for telehealth interviews (14 [67%] before and 16 [76%] after the examination).

Most participants reported that they were able to adequately complete or assess specific CSE items from the ABPN grading form, but only 4 trainees (44%) agreed that mental status could be adequately assessed (Table 1). All respondents agreed that suicidal and homicidal risks could be adequately assessed through the virtual CSE.

Table 1.

Clinical skills evaluation grading items question: I was able to adequately assess or complete these items via Zoom for the virtual CSE (equivalent to as if it was in person)

Clinical skill	Trainees (N=9), n (%)			Faculty (N=12), n (%)
	Agree	Neutral	Disagree	Agree	Neutral	Disagree
Opening and closing	5 (56)	0 (0)	4 (44)	9 (75)	0 (0)	3 (25)
Informational cues	6 (67)	2 (22)	1 (11)	11 (92)	1 (8)	0 (0)
Affective cues	5 (56)	1 (11)	3 (33)	10 (83)	0 (0)	2 (17)
Communication style and rapport	7 (78)	0 (0)	2 (22)	12 (100)	0 (0)	0 (0)
Questioning techniques	8 (89)	1 (11)	0 (0)	11 (92)	1 (8)	0 (0)
Control and direction of interview	5 (56)	2 (22)	4 (44)	11 (92)	1 (8)	0 (0)
Assessment of suicidal risk	9 (100)	0 (0)	0 (0)	12 (100)	0 (0)	0 (0)
Assessment of homicidal risk	9 (100)	0 (0)	0 (0)	12 (100)	0 (0)	0 (0)
Mental status examination	4 (44)	3 (33)	2 (22)	10 (83)	2 (17)	0 (0)

Residents more commonly indicated that virtual CSEs were inadequate for assessing clinical skills. In the posttest survey, 3 residents (33%) and 2 faculty (17%) agreed or strongly agreed that a virtual CSE could not adequately assess all components of the examination. For example, 4 residents (44%) disagreed that they could adequately control and direct the interview with the virtual format, and 3 (33%) indicated that they could not adequately assess affective cues (Table 1).

Significantly more faculty than trainees indicated that a virtual CSE made it more convenient for them to participate (P<.01 for both surveys) (Fig. 1). In the free-text comments, the most common themes regarding the benefits of the virtual CSE were comfort and convenience for participants and patients (n=16), decreased anxiety or feeling less intimidated (n=7), assessment of telehealth skills (n=6), and increased access to faculty and patients (n=5). However, numerous comments also described technologic problems, such as patients having difficulty connecting (n=16). Other common concerns were difficulty building rapport (n=6), difficulty discerning emotional cues or subtle findings (n=7), and challenges in assessing gait and motor function (n=6).

Discussion

This study examined the feasibility and appropriateness of using a video platform to administer the CSE in a general psychiatry residency program. Our findings suggest that a video-based format is feasible, as indicated by the successful completion of all examinations. Technical difficulties occurred but did not prevent completion of the CSEs. Faculty and trainees saw particular benefits of this format in the setting of COVID restrictions and noted its potential benefits in providing an experience that reflects the frequent use of telehealth in their clinic training. Notably, faculty reported that a virtual CSE made it more convenient for them to participate. Although most participants thought that the virtual format was acceptable, the considerable minority expressed concerns about virtual examinations that should not be overlooked. Trainees more commonly than faculty reported concerns about completing elements of the CSE by video, which may be due to the trainees having less experience with telehealth in clinical settings. Experts on telehealth competencies have suggested adjusting the mental status examination or replacing items if conducting a semistructured examination [11].

Although the ABPN has not indicated whether virtual CSEs will be accepted on a long-term basis, this study highlights several potential benefits of incorporating virtual formats into assessment, including convenience, increased access to patients and faculty, and the ability to evaluate telehealth-specific skills. In particular, the remote format could facilitate greater diversity among faculty examiners (including faculty from other programs or from nonacademic centers) and involve examiners who are not familiar with the trainees. Some have suggested that objectivity is diminished when a CSE is evaluated by examiners who know the trainees well [12]. This study also highlights the challenges and limitations of the virtual format, including technical difficulties. Additional research is needed to explore how to mitigate these challenges if a virtual CSE option continues to be offered.

Although telehealth has been used routinely during the pandemic in most psychiatry training programs, telehealth competencies are not part of the ACGME program requirements or Milestones. Most participants in this study indicated that telehealth assessments require unique skills and recognized that a virtual CSE provided an opportunity to evaluate those skills.

Institutional competencies for telehealth, including video care [13] and asynchronous care [14], have been suggested. Trainees must develop the appropriate attitudes, knowledge, and skills to be proficient providers of telehealth services, and formal assessment of a trainee’s ability to provide competent telehealth services is a sensible next step. One method of promoting telehealth competencies during residency education is to require at least 1 virtual CSE for board eligibility.

There are several limitations to this study. First, the sample size was small and derived from only 1 training institution; therefore, the results cannot necessarily be generalized to other programs. Second, survey responses represent the subjective experiences of the faculty and trainee participants, and this study lacked a more formal qualitative analysis or objective measures. Third, the relatively small number of survey items potentially limited the depth and breadth of the investigation. More rigorous methods, such as a multisite design with a control group and the use of third-party observers to rate the validity and fidelity of the virtual format, would strengthen future findings. Finally, this study did not include any survey data from patient volunteers, so the effects of the virtual format on patient experiences could not be assessed.

In summary, this preliminary study suggests that the virtual CSE is a feasible method for assessing psychiatry trainees, with potential benefits and notable limitations. With telehealth continuing to be an important mode of health care delivery, the virtual CSE provides an opportunity for residents to build necessary telehealth skills for their future practice.

Declarations

Ethics Approval

The study was approved by the Mayo Clinic Institutional Review Board (study ID: 21-001711) and by the Mayo Clinic Education Review Committee.

Disclosures

On behalf of all authors, the corresponding author states that there is no conflict of interest.