Neurology Clerkship: Predictors of Objective Structured Clinical Examination and Shelf Performance

Abstract

Background:

We sought to determine whether the following factors are associated with stronger performance on the medical school neurology clerkship: (1) structure of the outpatient rotation (working with a single general neurologist or multiple subspecialists), (2) dedicated shelf exam preparation, and (3) clerkships completed prior to neurology rotation.

Methods:

A total of 439 Feinberg medical students between 2014 and 2016 were analyzed based on the 3 variables of interest listed above. Student performance was evaluated using the National Board of Medical Examiner shelf exam and Objective Structured Clinical Examination/standardized evaluation scores. Univariate and multivariate analyses were conducted.

Results:

The format of the 2-week outpatient rotation did not significantly affect shelf examination (P = .59), or standardized evaluation (P = .34) scores. Taking a shelf pre-test correlated with overall higher standardized evaluation scores (P < .01), and higher shelf examination scores (P < .01). No individual clerkship correlated with better performance; however, the total number of core clerkships was associated with higher shelf examination scores (P = .007). Each additional core clerkship taken prior to neurology was associated with 0.72 points greater shelf examination score.

Conclusions:

Greater attending continuity did not appear to be associated with stronger performance perhaps due to a difference in types of cases observed. Students who took a practice shelf exam did better on both their shelf exam and standardized evaluation, suggesting that acquisition of knowledge translates to a better clinical performance. No individual clerkship offers an advantage, but rather it is the total number of clerkships that is correlated with stronger performance.

Background

The American Academy of Neurology (AAN) estimates that there is a current shortage of neurologists in the United States, a trend that is likely to worsen in the coming years with a projected shortfall of 20% by the year 2025.¹ National Resident Matching Program (NRMP) data suggest that 2% to 3% of US medical graduates apply into neurology, a percentage which has been stagnant over the last 10 years.² With an increasing demand for neurologists, and a relatively stable supply of providers, it has never been more important to provide optimal neurology education to the overwhelming majority of medical students who do not enter neurology practice.

The neurology clerkship is required in over 90% of US medical schools, is most often done during the third year, and is the primary means for providing structured neurology education to medical students.³ At our institution, students are evaluated at the end of their rotation with a combination of the National Board of Medical Examiners (NBME) clinical subject exam and an Objective Structured Clinical Examination (OSCE). There are several modifiable factors which may be associated with performance on these measures: (1) clerkship structure, including the format of the outpatient portion of the rotation; (2) dedicated preparation for end-of-rotation shelf exam; and (3) clerkship sequence order.

We hypothesized that the students who had greater continuity with outpatient general neurology attendings, those who had taken practice shelf pre-tests in preparation for the NBME shelf test, and those who had done internal medicine or psychiatry rotations prior to neurology would have better performance on these two outcomes.

Methods

This project was submitted to the Northwestern University Institutional Review Board and was deemed exempt from review.

Students at Northwestern University Feinberg School of Medicine who completed the neurology rotation between the 2014 and 2016 academic years were studied. The neurology clerkship at Feinberg consists of 2 weeks of outpatient neurology and 2 weeks of inpatient neurology rotations (divided among stroke service, general neurology service, neurology consult service, or emergency neurology service).

Students are evaluated at the end of the clerkship with the NBME shelf examination and the OSCE clinical assessment test. The NBME shelf exam may be used as a surrogate to evaluate for specialty-specific knowledge and is used by a majority of neurology clerkships in US medical schools. It typically accounts for 20% to 25% of the overall final grade (personal communication). The OSCE is an institutionally developed end of clerkship examination that includes a standardized patient assessment, as well as written short cases and multiple choice questions, which evaluates for specialty-specific clinical skill. The OSCE has been previously shown to serve as a reliable and predictable measure of clinical performance.⁴ In the standardized patient assessment, students collect a patient history, perform a complete neurologic physical examination, and complete a post encounter note to document their encounter with the standardized patients. The OSCE score ranges from 0 to 35, and the NBME shelf exam score ranges from 0 to 99. Students were analyzed based on the 3 variables of interest: outpatient structure, shelf exam pre-test, and core clerkships done prior to the neurology rotation (Table 1).

Table 1.

Independent variables.

Outpatient type
General neurology (single attending)	255 students	58%
Subspecialty neurology (multiple attendings)	184 students	42%
Shelf examination pre-test
No test	242 students	55%
NeuroSAE test	94 students	21%
NBME test	103 students	24%
Rotation done prior to neurology clerkship
Internal medicine	216 students	49%
Psychiatry	210 students	48%
Pediatrics	228 students	52%
Surgery	194 students	44%
Primary care	260 students	59%
Obstetrics-gynecology	233 students	53%

Abbreviation: NBME, National Board of Medical Examiners.

The majority of students worked with a single general neurologist during their 2 weeks of outpatient (n = 255, 58%), whereas the remainder rotated directly with multiple neurology subspecialty faculty members (n = 184, 42%). Students who rotated through the subspecialty clinics typically spent 1 to 3 days each in one of the following clinics: epilepsy, neuroimmunology, neuromuscular, sleep medicine, cognitive neurology, neuro-oncology, or movement disorders. Students were allowed to provide preference for their outpatient rotation; however, the majority of students did not indicate one and were randomly assigned to 2 to 3 different clinics during their 2-week rotation. In addition to these clinical encounters, all students, regardless of their rotation format, participated in 2 simulated patient encounters, at the beginning of their rotation and halfway through their clerkship. These sessions aimed to provide additional clinical experience to fill any gaps of knowledge not covered by the actual rotation.

There were two types of practice shelf exams; from 2014 to 2015, the Self-Assessment Exam (NeuroSAE)⁵ was offered, and from 2015 to 2016, the NBME⁶ test was offered. Each student was only offered one type of official test based on the timing of the clerkship. Students were directed to other possible testing resources, such as clerkship preparation books; however, no data were collected regarding the utilization of these unofficial resources. Approximately half of the students elected not to take a practice shelf exam (n = 242, 55%), with the remainder of students taking either the NeuroSAE test (n = 94, 21%) or the NBME test (n = 103, 24%).

Each of the core clerkships (internal medicine, psychiatry, pediatrics, surgery, primary care, and obstetrics-gynecology) at the Feinberg School of Medicine was evaluated as an independent variable. In addition, the total number of clerkships taken prior to the neurology rotation was also included in the analysis.

Students were pooled from all 3 years (2014-2016) and were collectively analyzed. There did not appear to be any significant difference among the years in terms of number of students from each year, percentage of students assigned to various rotations, or ratio of third-year versus fourth-year students. The United States Medical Licensing Examination (USMLE) Step 1 scores gradually increased over the course of 3 years, reflecting the increase seen in national average. There was no significant difference among classes in terms of shelf exam scores. Individual analysis from each year was not performed.

Statistical analysis was conducted using SAS 9.4 (Cary, NC). The Pearson correlation coefficient was used to test the correlation between total OSCE grade and shelf examination score. LOESS, a nonparametric scatterplot-smoothing technique, was used to display the association between total OSCE grade and shelf exam score. Univariate analyses of normally distributed continuous variables were conducted using the Student 2-sample t-test; for non-normally distributed continuous variables, the Wilcoxon rank sum test was used. Multivariate analyses were conducted using linear regression. The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Results

There was a total of 439 students evaluated, with the majority of students being third-year medical students (n = 396, 90%) and the remainder being fourth-year medical students (n = 43, 10%). Students who rotate in the fourth year have typically deferred their neurology rotation in favor of another elective, such as ophthalmology or radiology. Although they were a small portion of the total group, these students were included in the analysis to minimize the effect of selection bias.

The format of the 2-week outpatient rotation (subspecialty clinic vs general neurology) did not significantly affect end of clerkship shelf examination scores (80.59 ± 7.36 vs 80.17 ± 6.83, P = .59), or OSCE scores (30.14 ± 2.21 vs 29.91 ± 2.52, P = .34). Taking a shelf examination pre-test (either NeuroSAE or NBME test) correlated with overall small, but significantly higher OSCE scores (no test: 29.54 ± 2.47 vs NeuroSAE: 30.30 ± 2.39 and NBME: 30.82 ± 1.95, P = .01 and P < .0001) and higher clerkship shelf examination scores (no test: 78.63 ± 7.13 vs NeuroSAE: 80.66 ± 7.17 and NBME: 84.10 ± 5.04, P < .01 and P < .0001). This effect was seen in both pre-test groups; however, the effect was greater in the cohort which took the NBME test by over 5 points on the shelf exam and 1 point on the OSCE. Both pre-test groups were compared with the cohort that elected not to take any practice test. No direct comparison was made between the NBME group and the NeuroSAE group as only one of these tests was offered to students based on the timing of their clerkship. Moreover, because the number of fourth-year students overall was so small, we did not compare outcomes between third- and fourth-year students.

There was no effect of taking a prior core clerkship on final OSCE scores (Table 2). However, each of the core clerkships, except for psychiatry, correlated positively with higher neurology shelf examination scores when analyzed independently (Table 3). There was no statistically significant correlation between the total number of core clerkships performed prior to neurology and total OSCE grade (Figure 1A); however, there was a significantly positive correlation between total core clerkships and shelf examination score (Figure 1B).

Table 2.

Effect of prior clerkships on OSCE score.

Clerkship	With clerkship prior to neurology	Without clerkship prior to neurology	Difference	P-value
Pediatrics	29.76 ± 2.48	30.27 ± 2.28	−0.51	.02
Surgery	29.97 ± 2.49	30.04 ± 2.32	−0.07	.76
Primary care	30.00 ± 2.58	30.00 ± 2.12	0	>.9
Obstetrics-gynecology	29.96 ± 2.46	30.06 ± 2.33	−0.10	.65
Medicine	30.42 ± 2.00	29.78 ± 2.48	0.64	.07
Psychiatry	30.03 ± 2.22	29.78 ± 2.48	0.25	.39

Abbreviation: OSCE, Objective Structured Clinical Examination.

Table 3.

Effect of prior clerkship on shelf score.

Clerkship	With clerkship prior to neurology	Without clerkship prior to neurology	Difference	P-value
Pediatrics	81.27 ± 6.90	79.35 ± 7.09	1.92	.006
Surgery	81.77 ± 6.91	79.22 ± 6.98	2.55	<.01
Primary care	81.23 ± 6.92	79.06 ± 7.07	2.17	.004
Obstetrics-gynecology	81.23 ± 6.87	79.34 ± 7.14	1.89	.007
Medicine	81.35 ± 6.32	78.63 ± 7.21	2.72	.01
Psychiatry	79.54 ± 7.65	78.63 ± 7.21	0.91	.22

Figure 1.

(A) Relationship between total number of clerkship and OSCE score. (B) Relationship between total number of clerkship and shelf score.

When controlling for the total number of core clerkships students had taken prior to neurology, the effect between any of the individual core clerkships and the overall shelf examination score was lost. There were only 2 factors that had a positive effect on outcomes (Table 4). The groups of students who had taken a practice shelf exam still had overall higher shelf and OSCE scores compared with the cohort that did not take a practice test. In addition, after adjusting for the outpatient and pre-test type, only the total number of core clerkships was positively associated with shelf examination score (P = .007). Each additional core clerkship taken prior to neurology was associated with a 0.72 ± 0.27 points greater shelf examination score.

Table 4.

Fully adjusted model predicting effect of variable on shelf score.

Outpatient type
	Difference compared to*	P-value
General neurology (single attending)	−0.25	.69
Subspecialty neurology (multiple attendings)	—	—
Shelf examination pre-test
	Difference compared to*	P-value
No test	—	—
NeuroSAE test	2.06	.01
NBME test	5.51	<.001
Rotation done prior to neurology clerkship
	Effect on shelf score	P-value
Internal medicine	1.29	.08
Psychiatry	0.37	.59
Pediatrics	−0.08	.92
Surgery	1.45	.05
Primary care	0.79	.27
Obstetrics-gynecology	0.92	.23
Total number of clerkships	0.72	.007

Abbreviation: NBME, National Board of Medical Examiners.

USMLE Step 1 exam scores were also evaluated for each group as a surrogate for inherent test-taking ability, which could be a potential confounder in the interpretation of shelf exam differences. There was no statistically significant difference in Step 1 scores between the groups segmented by structure of outpatient rotation (P = .15), or in the groups who had done internal medicine (P = .73) or psychiatry (P = .15) prior to neurology. The group of students who had taken the NBME practice test did have slightly higher Step 1 scores (P = .01); however, the group who had taken the NeuroSAE practice test did not (P = .07).

Discussion

This study was the first to look at modifiable characteristics of a neurology clerkship which may be associated with stronger performance and acquisition of knowledge on a 4-week rotation. The results show that there are 2 factors which seem to have a positive effect on outcome: practice shelf exam prior to the end of clerkship test, and total number of core clerkships completed, regardless of the specialty of the rotations.

Structure of outpatient rotation

We also had clinical equipoise as to whether there were greater benefits from being placed with either subspecialists or with general neurologists. We postulated that students who spend their 2 weeks of outpatient neurology with a single attending in a general neurology clinic would perform better, based on prior studies which demonstrated possible educational benefits from less specialized rotations with greater longitudinal faculty contact.^7–9 In our study, this was not true for either of our outcome measures. There has been conflicting data on the relative importance of volume and breadth of cases that students are exposed to in their neurology rotation.^10–12 Students who rotate with a single general neurology attending may be able to develop a stronger longitudinal relationship with their faculty to facilitate their clinical skills and confidence, which may or may not translate into better performance on the shelf and OSCE. However, this may come at the expense of seeing less complex cases with potentially narrower pathology base. Alternatively, students who rotate through more specialized clinics may get more in-depth (and possibly breadth depending on the number of different subspecialties through which they rotate) exposure to specific pathologies. However, they may see fewer overall patients due to the complexity of patients seen in subspecialized clinics, and may not encounter “bread and butter” neurology. Thus, any possible benefit from a greater depth of exposure may be negated by lower volume of patients seen and less continuity with a single attending.

Shelf examination pre-test

There has been relatively little data on the effect of dedicated shelf preparation on final shelf examination performance, with most small studies suggesting that it is a not a primary focus of clerkship directors and that students rely heavily on their own question books.^13,14 Although shelf exam score does not necessarily translate into being a qualified physician, shelf preparation remains a large focus of students, is used in the calculation of their final overall clerkship grade, and can be reflective of a student’s test-taking ability. Thus, while performance on the shelf may not be helpful in predicting a student’s clinical acumen or ability, it does impact a student’s clerkship grade, which may impact their chances at matching into a particular residency.

When controlling for all other factors, students who elected to take a shelf pre-test scored better on both the OSCE and their final shelf exam. It is possible that the NBME practice test had a more positive impact than the NeuroSAE because NBME practice test is written by the same test writers as the actual shelf exam. However, since there was also a statistically significant effect with the NeuroSAE test, which is written by a different organization, the impact seems to lie more with the act of taking a practice test. Those students who elected to take a practice test also did better on the OSCE exam, suggesting that the acquisition of knowledge translates to an overall better performance. It is also possible that students who elected to take the practice test were independently more motivated learners or better test-takers. To address this, we used Step 1 scores of each group as a surrogate for inherent test-taking ability, as there have been several studies showing a positive correlation between USMLE and shelf performance.^15,16 We found that there was only a slightly higher average Step 1 score in the group that took the NBME practice test.

Rotations completed prior to neurology

The effect of rotations taken prior to the clerkship has not previously been assessed in neurology. There have been multiple studies demonstrating that clerkship sequence has an impact on final shelf performance, with internal medicine and surgery often having a more robust impact than other rotations.^17–20 In our study, each core clerkship (except psychiatry) independently correlated with higher shelf exam scores, with internal medicine and surgery having the strongest impact. This effect was not seen with OSCE scores. Thus, it is fathomable that having taken a prior shelf exam facilitates test-taking ability, but not clinical skills acquisition. It is also possible that any clinical skill difference present before taking the rotation is negated by completing the rotation, suggesting that all students are appropriately acquiring the necessary neurological clinical skills equally from the rotation. However, when controlling for the total core clerkships taken, this individual core clerkship effect on shelf scores was no longer evident for any rotation. Rather it was the total number of prior clerkships that mattered, with students who had taken a higher number of clerkships prior to neurology performing better, a fact which has been supported by other studies as well.

Limitations

Our study has several limitations. First, we did not assess exact types of cases seen by students in their outpatient weeks, and thus cannot determine whether the volume or breadth of cases was significantly different between the two groups. This could easily be assessed with case logs and could provide further clarity as to the importance of faculty continuity in overall outcomes. The second limitation is the inherent self-selection bias in the students who elected to take a practice shelf exam. It is possible that the students who elected to take either practice test were already motivated to do well, or were better test-takers and would have scored higher regardless of this factor. As described earlier, we tried to address at least the second component of this using Step 1 scores as a surrogate of test-taking ability as described above. The third limitation is the focus on outpatient rotation type. The 2-week inpatient rotation is also a source of variability, as students who rotate on a neurovascular service are typically exposed to less breadth of inpatient pathology, compared with those students who rotate on a general ward or consult service. Controlling for and assessing this variable would be an interesting addition to consider as part of a future project. Finally, we did not assess the exact order of when certain clerkships were taken prior to neurology, or whether a certain combination of clerkships may have provided some additive effect in outcome. This would likely require a higher volume of student results to achieve enough statistical power and could be done with additional years’ worth of data.

Conclusions

Students who take a practice shelf exam prior to the end of their neurology rotation do significantly better on both the OSCE and the end of clerkship shelf exam, regardless of the effect of outpatient type, pre-test type, and any rotations prior to the clerkship. The total number of core clerkships taken prior to neurology is positively and significantly associated with shelf examination score, with each additional core clerkship taken prior to neurology being associated with an expected 0.72 points greater shelf examination score. There is no effect seen with individual rotations, or with the structure of the outpatient component of the clerkship.

This information can be useful to clerkship directors who may be seeking to redesign the neurology curriculum, or who are looking for ways to improve shelf exam preparation for their students. It can also be helpful to residency program directors in their interpretation of clerkship performance based on the timing of the rotation in relation to their clerkship schedule. Similarly, these results can be used when counseling students about how to schedule their third-year clerkships, specifically for those who are interested in pursuing a career in neurology.