Abstract
Purpose
This study quantified the impact of clinical clerkships on medical students’ disciplinary knowledge using the Comprehensive Clinical Science Examination (CCSE) as a formative assessment tool.
Methods
This study involved 155 third-year medical students in the College of Human Medicine at Michigan State University who matriculated in 2016. Disciplinary scores on their individual Comprehensive Clinical Science Examination reports were extracted by digitizing the bar charts using image processing techniques. Segmented regression analysis was used to quantify the differences in disciplinary knowledge before, during, and after clerkships in five disciplines: surgery, internal medicine, psychiatry, pediatrics, and obstetrics and gynecology (ob/gyn).
Results
A comparison of the regression intercepts before and during their clerkships revealed that, on average, the participants improved the most in ob/gyn (11.193, p.0001), followed by psychiatry (10.005, p.001), pediatrics (6.238, p.0001), internal medicine (1.638, p.30), and improved the least in surgery (−2.332, p.10). The regression intercepts of knowledge during their clerkships and after them, on the other hand, suggested that students’ average scores improved the most in psychiatry (7.649, p.008), followed by ob/gyn (4.175, p.06), surgery (4.106, p.007), and pediatrics (1.732, p.32).
Conclusions
These findings highlight how clerkships influence the acquisition of disciplinary knowledge, offering valuable insights for curriculum design and assessment. This approach can be adapted to evaluate the effectiveness of other curricular activities, such as tutoring or intersessions. The results have significant implications for educators revising clerkship content and for students preparing for the United States Medical Licensing Examination Step 2.
Keywords: Disciplinary clerkships, clinical knowledge, longitudinal assessment, progress examination, segmented regression
Introduction
Assessing the impact of clerkship rotation requires standardized assessment tools and testing procedures, either before and after the clerkship or across multiple repeated measures [1]. However, since clinical knowledge is multidimensional, it is challenging to quantify the learning growth from the clerkship experience. Without validated and reliable assessment tools or methods, meaningful measurement of growth is not possible. Moreover, scheduling every student’s clerkship rotation at the same time is not administratively possible, further complicating the assessment of the impact of clerkship programs on the growth of clinical knowledge.
This study is grounded in experiential learning theory, which posits that learning is a cyclical process of concrete experience, reflective observation, abstract conceptualization, and active experimentation [2]. During clinical clerkships, medical students engage in this cycle, applying their theoretical knowledge to real-world patient encounters, observing expert clinicians, and reflecting on their experiences to refine their understanding and clinical decision-making. The experiential learning framework suggests that the varied clinical experiences encountered during clerkships can have differential impacts on students’ disciplinary knowledge development.
Formative assessment is an evaluative process used to monitor student learning and provide ongoing feedback that instructors can use to improve their teaching and students to improve their learning [3]. The Comprehensive Clinical Science Examination (CCSE) is a formative assessment tool that gives students feedback on their progress in clinical domains, such as internal medicine, surgery, and pediatrics. By analyzing students’ CCSE scores before, during, and after their clerkship experiences, we aimed to elucidate the differential effects of various clinical rotations on acquiring disciplinary knowledge.
Experiential learning theory [2] and its application are commonly used in clinical rotations and clerkships. Building on the foundation of Dewey, Lewin, Piaget, and Knowles, Kolb conceptualized the work of learning from experience with four different abilities: (1) concrete experience, (2) reflective observation, (3) abstract conceptualization, and (4) active experimentation [4]. These four experiential learning abilities formed a cycle and greatly influenced current clerkship teaching and curriculum.
The current medical school curriculum reform trend includes early clerkship experience and longitudinal integrated clerkships [5,6]. Since the problem-based curriculum was implemented in the early 1980s, medical schools’ curriculum models have necessitated that knowledge be evaluated in tandem with clerkship learning. Nevertheless, few empirical studies have looked at how students benefit from their clerkship learning experience, or how clerkship content might be better aligned with their cognitive learning styles. Previous studies have shown that students’ performance improves incrementally, both on multiple-choice examinations of relevant knowledge [7] for clerkships, and on self-assessments of competency [8] after their clerkship rotation [7,9–11]. However, medical schools vary widely in the delivery and sequencing of their disciplinary clerkship rotations and the evaluation of students during their third-year rotations [12,13], making it challenging to determine which clerkships provide the most significant positive impact on the development of future clinicians. Additionally, grading rubrics across different clerkships even within an individual school are not necessarily comparable [14], further complicating the evaluation of any given clerkship’s effectiveness.
Standardized assessment plays a vital role in progress testing [15]. Established and stable reliability and validity of examination scores minimize the chances that the impact of assessments gets confounded [16–18]. Especially when the assessment is measured multiple times, the sources of variation can include form-difficulty variations, mode effects, and/or item-difficulty variations.
The National Board of Medical Examiners (NBME) CCSE is a standardized assessment tool that can serve as a reliable and valid measure of progress in competency-based medical education [19]. The CCSE is designed to provide students with formative feedback on their progress in clinical domains, such as internal medicine, surgery, and pediatrics. By analyzing students’ CCSE scores before, during, and after their clinical clerkship experiences, researchers can elucidate the differential impacts of various rotations on the growth of students’ disciplinary knowledge and clinical competencies.
Using a well-established, standardized assessment like the CCSE offers several advantages. First, the CCSE has been extensively validated, and its psychometric properties are well-documented, ensuring that the scores reflect meaningful and reliable measures of clinical knowledge and skills. Second, the CCSE’s consistent format and content across multiple administrations allows for the longitudinal tracking of student performance, enabling researchers to identify patterns and trends in the development of disciplinary knowledge throughout clinical clerkships. Finally, the CCSE’s comprehensive coverage of key clinical domains aligns with the multidimensional nature of clinical knowledge, providing a more holistic assessment of student learning compared to narrowly focused, single-subject examinations. Thus, CCSE can serve as such a tool in competence-based progress tests.
Drawing on experiential learning theory in medical education, which emphasizes a cyclical process of concrete experience, reflective observation, abstract conceptualization, and active experimentation, this study delves into the impact of clinical clerkships on the growth of medical students’ disciplinary knowledge. Experiential learning theory posits that during clinical clerkships, students engage in a dynamic cycle of applying theoretical knowledge in real-world settings, observing expert clinicians, reflecting on experiences, and actively refining their clinical decision-making skills [20–23]. This framework suggests that the diverse clinical encounters students experience during clerkships can have varying effects on the development of their disciplinary knowledge [22]. This study analyzes students’ performance on the CCSE before, during, and after their clerkship experiences to unveil the differential impacts of different clinical rotations on acquiring disciplinary knowledge. Integrating experiential learning theory in medical education underscores the importance of hands-on experiences and reflective practice in shaping students’ clinical competencies and understanding.
This study aimed to quantify the impact of clinical clerkships on the growth of medical students’ disciplinary knowledge. Using the Comprehensive Clinical Science Examination (CCSE) as a formative assessment tool, we examined students’ performance before, during, and after their clerkship experiences to evaluate the differential effects of various rotations on the acquisition of disciplinary knowledge.
Methods
Study participants
This study’s participants were 155 third-year medical students in the College of Human Medicine at Michigan State University who matriculated in 2016. CCSE is required when the students are in their third year (Fall 2018, Spring 2019, and Summer 2019) in the MD program, so every student is a study participant.
Measurements
The CCSE was used to assess the participants’ disciplinary clinical knowledge trajectory. They were required to take it twice per semester over the three semesters of the third year of medical school, which count toward their grades each semester. Numeric scores are not available in individual CCSE reports. Disciplinary scores were digitized and extracted from the bar interval charts of the individual CCSE reports using image processing techniques. An example of a bar interval chart can be seen in Figure 1. Specifically, each bar interval chart was extracted as an image and read in as an array of pixels that stored brightness, color, and distance. We extracted the coordinates of the lower and upper ends of the bars in the array and used their middle points as the CCSE disciplinary scores in the study. The first author wrote the image processing technique using R language, and the digitized numbers were validated with the unregistered version of the software PlotDigitizer [24]. To aid understandability, the ends of the chart were scaled such that the disciplinary scores range from 0 to 100. The same digital extraction procedure was repeated for all students and across all six measures.
Figure 1.
An example of a bar interval graph.
Except for the surgery clerkship, which lasts eight weeks, the major disciplinary clerkships—internal medicine, psychiatry, pediatrics, and obstetrics and gynecology (ob/gyn)—are four weeks each. All students must undertake all five clerkships after completing the United States Medical Licensure Examination (USMLE) Step 1 at the end of their second year but do so in different orders.
Statistical methods
Because the students’ rotation schedules through their clerkships were not the same, we defined a time scale oriented around each clerkship separately: i.e. as consisting of Phase 1, Phase 2, and Phase 3, indicating students’ performance in disciplinary clinical knowledge before, during, and after the relevant disciplinary clerkship, respectively.
Segmented regression analysis (also called piecewise regression or broken-stick regression) was then used to quantify the pairwise differences in disciplinary knowledge among the three phases [25,26]. The impacts of the five major disciplinary clerkships were analyzed separately, and each impact was modeled and quantified by the differences in the regression intercept of each phase [27]. For model simplicity, we specified the same growth instead of phase-specific growth across the three phases. The disciplinary scores were digitized from the CCSE reports using the png package, and the segmented-regression models were built using the lmerTest package [28], in R version 3.6.3. All statistical analyses were conducted using R, a language and environment for statistical computing developed by the R Core Team and supported by the R Foundation for Statistical Computing (Vienna, Austria) [29].
This study was conducted using de-identified data, which exempts it from informed consent requirements under the U.S. Department of Health and Human Services Common Rule (45 CFR 46.104(d)(4)) and the HIPAA Privacy Rule (45 CFR 164.514), both of which allow for the use of de-identified information without obtaining consent from individuals. A designated honest broker is used to deidentify curricular and student evaluation data collected as a normal part of the medical school’s educational programs. According to the Michigan State University’s Human Research Protection Program’s determination, these data are not considered human subject data (IRB# STUDY 00007478).
To justify the sample size for our segmented regression analysis, we conducted a power calculation using a simulation-based approach in R. We used the intercept change (= 2) as the effect size and fixed the breakpoint at the midpoint of the measuring points. Each subject provided six measuring points, resulting in 930 observations (n = 155 subjects). We simulated the data under the alternative hypothesis, where there was a change in the intercept at the breakpoint while the slope remained constant. Random noise with a standard deviation of five was added to the simulated data to reflect variability. For each of the 1000 simulations, we fitted a linear regression model with an indicator variable for the time after the breakpoint. The power of the test was estimated by calculating the proportion of simulations where the p-value for the intercept change was less than the significance level (α = 0.05). The results indicated an estimated power of 0.821, suggesting that our study design has a high probability of detecting the specified effect size at the given significance level.
Results
Table 1 shows the descriptive statistics across the six measures. The average scores in all disciplines showed increasing trends, but those in psychiatry were generally the highest.
Table 1.
Descriptive statistics by specialty and time point.
| Specialty | Time | Mean | SD | Max | Min |
|---|---|---|---|---|---|
| Internal medicine | Time 1 | 21.65 | 15.55 | 68.18 | 3.59 |
| Time 2 | 23.41 | 16.67 | 73.28 | 3.57 | |
| Time 3 | 27.61 | 16.23 | 78.58 | 3.59 | |
| Time 4 | 29.30 | 15.09 | 67.28 | 3.84 | |
| Time 5 | 36.18 | 18.49 | 76.63 | 3.71 | |
| Time 6 | 44.56 | 18.87 | 87.78 | 4.94 | |
| Ob/Gyn | Time 1 | 22.30 | 14.17 | 61.34 | 3.71 |
| Time 2 | 27.20 | 16.85 | 64.70 | 3.69 | |
| Time 3 | 31.94 | 18.23 | 81.99 | 3.71 | |
| Time 4 | 33.38 | 17.24 | 74.95 | 4.32 | |
| Time 5 | 41.16 | 15.92 | 77.47 | 7.03 | |
| Time 6 | 46.04 | 17.04 | 81.70 | 10.88 | |
| Pediatrics | Time 1 | 25.54 | 13.71 | 65.61 | 3.71 |
| Time 2 | 28.28 | 15.95 | 75.85 | 3.57 | |
| Time 3 | 31.84 | 16.76 | 78.56 | 3.71 | |
| Time 4 | 35.43 | 16.87 | 85.39 | 4.99 | |
| Time 5 | 39.19 | 15.95 | 74.07 | 9.14 | |
| Time 6 | 44.45 | 17.14 | 75.74 | 8.94 | |
| Psychiatry | Time 1 | 31.98 | 15.47 | 63.04 | 3.59 |
| Time 2 | 32.15 | 20.13 | 83.70 | 3.43 | |
| Time 3 | 39.49 | 19.94 | 83.02 | 4.21 | |
| Time 4 | 45.32 | 18.42 | 83.47 | 3.74 | |
| Time 5 | 45.31 | 16.80 | 80.45 | 6.76 | |
| Time 6 | 47.85 | 20.35 | 83.23 | 10.30 | |
| Surgery | Time 1 | 21.87 | 13.64 | 65.61 | 3.71 |
| Time 2 | 22.89 | 17.33 | 81.85 | 3.45 | |
| Time 3 | 27.07 | 16.63 | 70.87 | 3.71 | |
| Time 4 | 29.12 | 17.67 | 79.23 | 3.72 | |
| Time 5 | 39.31 | 16.09 | 73.21 | 4.75 | |
| Time 6 | 46.47 | 18.83 | 86.49 | 8.59 |
Table 2 presents the regression discontinuity estimates for each disciplinary clerkship. Figure 2 is a plot of the observed growth and the modeled piecewise regression line for each discipline.
Table 2.
Regression discontinuity estimates in each disciplinary clerkships.
| Phase 2 vs. Phase 1 |
Phase 3 vs. Phase 2 |
Phase 3 vs. Phase 1 |
||||
|---|---|---|---|---|---|---|
| Estimate | p-Value | Estimate | p-Value | Estimate | p-Value | |
| Internal medicine | 1.638 | 0.30 | 3.012 | 0.06 | 4.650 | <0.0001 |
| OB/Gyn | 11.193 | <0.0001 | 4.175 | 0.06 | 15.368 | <0.0001 |
| Pediatrics | 6.238 | <0.0001 | 1.732 | 0.32 | 7.970 | <0.0001 |
| Psychiatry | 10.005 | 0.001 | 7.649 | 0.008 | 17.653 | <0.0001 |
| Surgery | −2.332 | 0.10 | 4.106 | 0.007 | 1.774 | 0.34 |
Figure 2.
The individual observed curve (in gray) and the modeled piecewise regression curve (in black) for each discipline.
Phase 2 vs. Phase 1
To capture change in disciplinary knowledge right after the start of the disciplinary rotation, we compared the regression intercepts before a given clerkship and during it (i.e. Phase 2 vs. Phase 1 in Table 2). This revealed that students’ average scores increased the most in ob/gyn (11.193, p < .0001), followed by psychiatry (10.005, p = .001), pediatrics (6.238, p < .0001), internal medicine ( 1.638, p = .30), and surgery (2.332, p = .10). However, the increases were only statistically significant in the first three disciplines; knowledge changes in internal medicine and surgery were not significantly different from zero (p > .05).
Phase 3 vs. Phase 2
When we compared the regression intercepts of knowledge during a clerkship and after it (i.e. Phase 3 vs. Phase 2 in Table 2), we found that the students’ average scores improved the most in psychiatry (7.649, p = .008), followed by ob/gyn ( 4.175, p = .06), surgery ( 4.106, p = .007), and pediatrics (1.732, p = .32). However, the observed changes were only statistically significant for psychiatry and surgery.
Phase 3 vs. Phase 1
The regression intercepts of knowledge difference between Phase 3 and Phase 1 indicated that disciplinary knowledge increased significantly (p < 0.0001) pre- to post-clerkship in all disciplines except surgery. As compared to their CCSE disciplinary scores from before the relevant clerkship, students’ average post-clerkship score gains were 4.65 points in internal medicine (p < .0001), 7.97 points in pediatrics (p < .0001), 15.37 points in ob/gyn (p < .0001), and 17.65 points in psychiatry (p < .0001). Students’ surgery scores only increased 1.77 points after their surgery clerkships (p = 0.34), which was not statistically different from zero.
The impact of the rotation was found to be incremental. Students’ growth in disciplinary clinical knowledge scores before and after their clerkship also reflected their learning curves in the third year of medical school. For internal medicine, although the students’ changes between Phase 2 vs. Phase 1 and Phase 3 vs. Phase 2 were not statistically significant from zero, the change between Phase 3 and Phase 1, 4.65 points, was a statistically significant improvement. Ob/gyn’s score-change trajectory was similar to that of pediatrics: i.e. a large increase occurred when students started the rotation, which implied the impact of ob/gyn and pediatrics rotations on the corresponding clinical knowledge. Interestingly, however, the change in surgery scores across phases differed from the other four disciplines. On average, students’ surgical knowledge had a non-significant drop after they started the rotation, but then returned to their original pre-rotation level. This suggests that the surgery clerkship did not significantly affect the students’ surgical-knowledge growth. The underlying reasons for this are discussed in the next section.
The ‘step’ shown in Figure 2 between Phases 1 and 3 clearly shows the impact of the psychiatry rotation. Except in psychiatry, the sampled students’ performance steadily grew between Phase 1 and Phase 3. In psychiatry, though the regression intercept difference between Phase 1 and Phase 3 showed a significant increase, growth within these two phases was zero.
Discussion
Several formative assessment methods with good psychometric properties [25,30–33], have been proposed for use in the clerkship settings. Given that curriculum reform in medical schools is focused on the transition to an integrated curricular structure, assessment and evaluation methods need to be changed accordingly. This study has proposed and demonstrated the utility of segmented regression analysis for quantifying the impact of clerkships on students’ clinical knowledge in various medical disciplines and examining their knowledge growth before and after the clerkship-rotation period, using formative assessment data.
The results provide helpful information for medical schools regarding how their students acquire clinical knowledge while preparing for the USMLE Step 2 clinical knowledge examination [34]. The implications of its findings are particularly important to instructors and proctors investigating and seeking to revise the content of clerkship activities. These findings suggest that certain clerkship experiences may be more effective than others in promoting the acquisition of clinical knowledge. Medical schools could use this information to optimize their clerkship structures and activities to better support student learning. Additionally, the analytical approach used in this study could be adapted to evaluate the effectiveness of other curricular interventions, such as tutoring programs or intersession activities. The methods we have presented can be readily applied to other similar purposes; this approach could be utilized to measure the effectiveness of curriculum activities, such as tutoring or intersessions [35]. Before conducting this study, we performed pair-comparison and sequence analyses to examine the order effect of the rotation schedule on disciplinary knowledge. Similar to the previous study [36], the results showed that students’ disciplinary knowledge was not affected by their rotation order. This finding led us to design separate analyses for each discipline. However, if it were to later emerge that the rotation order or rotation schedule did affect students’ disciplinary knowledge, we would recommend using multivariate segmented regression analysis with disciplinary order included as an interaction term.
We also conducted linear mixed segmented regression analysis with phase-specific growth to find the best model. The results did not indicate that phase-specific growth rates significantly differed across the three phases. For simplicity’s sake, we, therefore, specified the growth rate as the same across the phases. The observed steady growth in clinical knowledge in the third-year contrasts sharply with the varying growth rate in students’ medical knowledge generally observed in the first two years of medical school. To our knowledge, it has not been the topic of any previous literature. As such, the learning trajectory we identified can serve as validation evidence for further research.
Our evaluation results showed the impact that the clerkships can have on the students’ assessment performance. This may be varied due to medical schools’ clerkship content, schedules, workloads, lengths, and designs. However, because the methods used in this research have no causal-inference implications, the reasons behind the observed growth and declines in clinical knowledge should be discussed further with the students and the educators involved. For example, our finding that students’ CCSE surgery scores dropped during their surgery rotation could have been because that clerkship’s learning was not aligned with the examination content or because its workload was so heavy that students did not have enough time to prepare for the exam; or because this batch of students’ interest in acquiring surgery-related clinical knowledge was low. Another interesting result involved the flatness of students’ growth rate in psychiatry in Phase 1 and Phase 3. This indicated that, although the clerkship experience helped improve their psychiatry exam scores, it might be the only source of psychiatry knowledge in their third-year medical education. Again, such a finding warrants further qualitative or program-evaluation research among curriculum developers, clinical educators, clerkship communities, medical students, psychometricians, and school leaders.
Quantifying the effectiveness of rotation training is not intended as a criticism of the conduct of disciplinary clerkships or to examine a specific clinical task [30,37–40], but rather to assess the relationship between such clerkships and students’ disciplinary clinical knowledge over time [41–44]. Clinical knowledge is a multidimensional latent construct and assessing it on a discipline-specific basis has always been challenging for medical schools. Carrying out research using our methodology could also allow schools to initiate conversations with the clerkship community on preparing students to be better future physicians in practice.
The paper assessed and quantified clerkships’ impact using segmented regression analysis. However, it should be noted that it only used CCSE disciplinary scores to measure students’ clinical learning outcomes. What clerkship experience brings to students is beyond what CCSE can cover and measure; indeed, most of its benefits may be unmeasurable. As part of the current trend of ‘data booming’, we can expect a warning system to be built that will provide an in-depth, dynamic evaluation of students’ learning outcomes in clerkship activities. Such a system would allow early detection of students’ difficulties and thus facilitate prompt, appropriate assistance to bridge the gap between clerkship experience and learning outcomes.
During periods of crisis, such as the COVID-19 pandemic, young healthcare students and residents face additional challenges that can impact their knowledge advancement. The increased stress, anxiety, and workload can detract from their ability to focus on learning and retaining clinical knowledge. Moreover, disruptions to clinical rotations and educational activities can result in missed learning opportunities and reduced hands-on experience. These factors may lead to uneven knowledge acquisition and a potential decline in clinical competence. However, the post-pandemic era offers an opportunity to integrate adaptive measures permanently into medical education. These measures include leveraging digital tools like virtual patient simulations and telemedicine practices, which proved instrumental during the pandemic, to complement traditional clinical experience. Instructors and curriculum designers can also focus on fostering resilience and adaptability by incorporating training on coping strategies and mental health resources into medical education programs. Such innovations can mitigate the impact of future crises and ensure the continued growth of medical students’ clinical competencies and readiness for practice. For instance, Moldovan et al. highlights the significant impact of the COVID-19 pandemic on orthopedic residents in Romania, emphasizing the need for adaptive measures in medical training during such times [45].
Limitations of the study
A key limitation of this study is its reliance on standardized test scores as the sole measure of student learning. While these scores objectively assess disciplinary knowledge, they do not capture other important domains of clinical competence, such as communication skills and professionalism. Future research should consider incorporating multiple assessment methods to gain a more comprehensive understanding of the impact of clerkships on overall clinical development.
Another limitation is the potential impact of the COVID-19 pandemic on the study participants’ learning experiences and knowledge acquisition. The disruptions to clinical rotations and educational activities during this time may have influenced the observed results, and further research is needed to understand the long-term implications of the pandemic on medical education.
Beyond the challenges posed by the pandemic, additional limitations of the research design warrant consideration, particularly those related to the observational nature of the study and its reliance on a single institutional cohort. A primary limitation of this study lies in its observational design, which inherently limits causal inferences. The absence of randomization or a control group to account for potential confounding factors, such as differences in clerkship sequence or student-specific characteristics, reduces the ability to isolate the effects of clerkships on disciplinary knowledge. Additionally, while segmented regression analysis offers robust insights into phase-specific changes, it assumes linear trends within each phase, potentially oversimplifying non-linear growth patterns in clinical knowledge acquisition. Another limitation is the reliance on a single cohort of students from a single institution, which may limit the generalizability of the findings to other medical schools with different curricular structures, demographic profiles, or educational environments.
Future research should address these limitations by employing study designs incorporating randomization or matched comparisons to control for confounding variables and strengthen causal inferences. Expanding the study to include multiple institutions with diverse student populations and curricular models would enhance the generalizability of the findings. Additionally, integrating mixed-method approaches, such as combining quantitative assessments with qualitative interviews, could provide a more holistic understanding of how clerkships impact clinical knowledge and other competencies, such as communication skills, professionalism, and teamwork. Longitudinal studies tracking students’ clinical performance post-graduation could also shed light on the long-term effects of clerkship experiences. Finally, examining innovative assessment tools, such as virtual simulations and competency-based evaluations, could further refine our understanding of how clerkships contribute to medical education outcomes.
Conclusion
This study evaluated the effectiveness of clinical clerkships on the growth of medical students’ disciplinary knowledge using their CCSE scores as a formative assessment [46]. The results indicate that the impact of clerkships varied across different disciplines, with students showing the greatest knowledge gains in obstetrics and gynecology, psychiatry, and pediatrics, and the least gains in surgery.
Overall, this study contributes valuable insights into the varying effectiveness of clinical clerkships across different medical disciplines. The findings can inform efforts to enhance the design and implementation of clerkship experiences to better support the growth of medical students’ clinical knowledge and competencies.
Acknowledgements
The authors would like to thank Ann Schultz in the Office of Medical Education Research and Development for her support of data cleaning and the audiences for comments and feedback when the original manuscript was presented at the annual conference of the 2021 Central Group on Educational Affairs.
Funding Statement
No funding was received.
Ethical approval
The activity described in this submission was determined not to involve ‘human subjects’ as defined by the U.S. Department of Health and Human Services (DHHS) regulations for the protection of human research subjects. Thus, consent forms are not needed. A designated honest broker is used to deidentify curricular and student evaluation data collected as a normal part of the medical school’s educational programs. According to the Michigan State University’s Human Research Protection Program’s determination, these data are not considered human subject data (IRB# STUDY 00007478). Documentation concerning the honest broker program can be found at https://omerad.msu.edu/research/honest-broker-for-educational-scholarship.
Author contributions
CC conceived and designed the experiments, analyzed the data, prepared figures and/or tables, and authored and reviewed drafts of the paper. HLF conceived and designed the experiments, performed the experiments, and authored and reviewed drafts of the paper. JM provided interpretation of the data and reviewed drafts of the paper. CP reviewed drafts of the paper. DS conceived and designed the experiments and reviewed drafts of the paper. All authors have read and approved the final manuscript.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Disclaimer
None.
Previous presentations
This paper is extended from the conference paper presented at the Annual Conference of the Central Group on Educational Affairs on 21 April 2021.
Data availability statement
The participants of this study did not give written consent for their data to be shared publicly, so due to the sensitive nature of the research, supporting data is not available.
References
- 1.Wagner AK, Soumerai SB, Zhang F, et al. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27(4):299–309. doi: 10.1046/j.1365-2710.2002.00430.x. [DOI] [PubMed] [Google Scholar]
- 2.Kolb DA, Boyatzis RE, Mainemelis C.. Experiential learning theory: previous research and new directions. In: Perspectives on thinking, learning, and cognitive styles. New York, NY: Routledge; 2014. p. 227–248. [Google Scholar]
- 3.Black P, Wiliam D.. Assessment and classroom learning. Assess Educ Princ Policy Pract. 1998;5(1):7–74. doi: 10.1080/0969595980050102. [DOI] [Google Scholar]
- 4.Merriam SB, Caffarella RS, Baumgartner LM.. Learning in adulthood: a comprehensive guide. NJ: John Wiley & Sons Inc; 2007. [Google Scholar]
- 5.Teherani A, Irby DM, Loeser H.. Outcomes of different clerkship models: longitudinal integrated, hybrid, and block. Acad Med. 2013;88(1):35–43. doi: 10.1097/ACM.0b013e318276ca9b. [DOI] [PubMed] [Google Scholar]
- 6.McLaughlin K, Bates J, Konkin J, et al. A comparison of performance evaluations of students on longitudinal integrated clerkships and rotation-based clerkships. Acad Med. 2011;86(10 Suppl):S25–S29. doi: 10.1097/ACM.0b013e31822a6eb6. [DOI] [PubMed] [Google Scholar]
- 7.Jacques LB. Incremental change in student knowledge during a third-year family medicine clerkship. Fam Med. 1997;29(7):477–482. [PubMed] [Google Scholar]
- 8.Porter-Williamson K, Gunten CF, von Garman K, et al. Improving knowledge in palliative medicine with a required hospice rotation for third-year medical students. Acad Med. 2004;79(8):777–782. doi: 10.1097/00001888-200408000-00013. [DOI] [PubMed] [Google Scholar]
- 9.Norris TE, Schaad DC, DeWitt D, et al. Longitudinal integrated clerkships for medical students: an innovation adopted by medical schools in Australia, Canada, South Africa, and the United States. Acad Med. 2009;84(7):902–907. doi: 10.1097/ACM.0b013e3181a85776. [DOI] [PubMed] [Google Scholar]
- 10.Poncelet A, Bokser S, Calton B, et al. Development of a longitudinal integrated clerkship at an academic medical center. Med Educ Online. 2011;16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Zaidi NLB, Kreiter CD, Castaneda PR, et al. Generalizability of competency assessment scores across and within clerkships. Acad Med. 2018;93(8):1212–1217. doi: 10.1097/ACM.0000000000002262. [DOI] [PubMed] [Google Scholar]
- 12.Riese A, Rappaport L, Alverson B, et al. Clinical performance evaluations of third-year medical students and association with student and evaluator gender. Acad Med. 2017;92(6):835–840. doi: 10.1097/ACM.0000000000001565. [DOI] [PubMed] [Google Scholar]
- 13.Hemmer P, Pangaro L.. The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med. 1997;72(7):641–643. doi: 10.1097/00001888-199707000-00018. [DOI] [PubMed] [Google Scholar]
- 14.Kreiter C, Ferguson K, Lee W, et al. A generalizability study of a new standardized rating form used to evaluate students’ clinical clerkship performances. Acad Med. 1998;73(12):1294–1298. doi: 10.1097/00001888-199812000-00021. [DOI] [PubMed] [Google Scholar]
- 15.Vleuten CPMVD, Verwijnen GM, Wijnen WHFW.. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18(2):103–109. doi: 10.3109/01421599609034142. [DOI] [Google Scholar]
- 16.Bernal JL, Cummins S, Gasparrini A.. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46(1):348–355. doi: 10.1093/ije/dyw098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Setyonugroho W, Kennedy KM, Kropmans T.. Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: a systematic review. Patient Educ Couns. 2015;98(12):1482–1491. doi: 10.1016/J.PEC.2015.06.004. [DOI] [PubMed] [Google Scholar]
- 18.Tavares W, Eva K.. Validity of OSCEs using multi-facet Rasch measurement. Acad Med J Assoc Am Med Coll. 2016;91(9):1305–1310. [Google Scholar]
- 19.National Board of Medical Examiners . Comprehensive Clinical Science Examination (CCSE); 2024. https://www.nbme.org/subject-exams/comprehensive/clinical-science [DOI] [PubMed]
- 20.Kolb DA. Experiential learning: experience as the source of learning and development. Hoboken, New Jersey: Prentice-Hall; 1984. [Google Scholar]
- 21.Kolb DA. Experiential learning: experience as the source of learning and development. 2nd ed. London, England: Pearson Education; 2014. [Google Scholar]
- 22.Mahmoud MA. Experiential learning and its application in clinical rotations and clerkships; 2015. Available from: https://qatar-weill.cornell.edu/continuing-professional-development/topic-of-the-month/archive/experiential-learning-and-its-application-in-clinical-rotations-and-clerkships
- 23.Meyer EG, Battista A, Sommerfeldt JM, et al. Experiential learning cycles as an effective means for teaching psychiatric clinical skills via repeated simulation in the psychiatry clerkship. Acad Psychiatry. 2021;45(2):150–158. doi: 10.1007/s40596-020-01340-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.PlotDigitizer Team . PlotDigitizer: version 3; 2021. [Google Scholar]
- 25.Holmboe ES, Huot S, Chung J, et al. Construct validity of the MiniClinical Evaluation Exercise (MiniCEX). Acad Med. 2003;78:826–830. doi: 10.1097/00001888-200308000-00018. [DOI] [PubMed] [Google Scholar]
- 26.Taljaard M, McKenzie JE, Ramsay CR, et al. The use of segmented regression in analysing interrupted time series studies: an example in pre-hospital ambulance care. Implement Sci. 2014;9(1):77. doi: 10.1186/1748-5908-9-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Zhang X, Wu K, Pan Y, et al. Introducing activation functions into segmented regression model to address lag effects of interventions. BMC Med Res Methodol. 2023;23(1):277. doi: 10.1186/s12874-023-02098-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Kuznetsova A, Brockhoff PB, Christensen RHB.. {lmerTest} package: tests in linear mixed effects models. J Stat Soft. 2017;82(13):1–26. doi: 10.18637/jss.v082.i13. [DOI] [Google Scholar]
- 29.R Core Team . R: a language and environment for statistical computing. R Foundation for Statistical Computing; 2024. Available from: https://www.R-project.org/
- 30.Norcini J, Burch V.. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29:855–871. doi: 10.1080/01421590701775453. [DOI] [PubMed] [Google Scholar]
- 31.Norcini JJ, Blank LL, Arnold GK, et al. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123:795–799. doi: 10.7326/0003-4819-123-10-199511150-00008. [DOI] [PubMed] [Google Scholar]
- 32.Norcini JJ, Blank LL, Duffy FD, et al. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003;138:476–481. [DOI] [PubMed] [Google Scholar]
- 33.Zhang J. Psychometric tools for formative classroom assessment: test construction and item pool design based on cognitive diagnostic models. 2019. [Google Scholar]
- 34.Ilkjær C, Nielsen KJS, Kjær LB, et al. Clinical clerkship with or without scheduled patient consultations: does it make a difference to medical students’ experiences of learning? Med Sci Educ. 2024;34:1101–1110. doi: 10.1007/s40670-024-02160-3. [DOI] [Google Scholar]
- 35.Zhang X, Wu K, Pan Y, et al. Optimized segmented regression models for the transition period of intervention effects. Glob Health Res Policy. 2023;8(1):29. doi: 10.1186/s41256-023-00312-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Peterson LN, Rusticus SA, Ross LP.. Comparability of the National Board of Medical Examiners comprehensive clinical science examination and a set of five clinical science subject examinations. Acad Med. 2015;90(5):684–690. doi: 10.1097/ACM.0000000000000643. [DOI] [PubMed] [Google Scholar]
- 37.Wallsten TS, Pleskac TJ, Lejuez WC.. Modeling behavior in a clinically diagnostic sequential risk-taking task. Psychol Rev. 2005;112(4):862–880. doi: 10.1037/0033-295X.112.4.862. [DOI] [PubMed] [Google Scholar]
- 38.Arain SA, Alhadid DA, Rasheed S, et al. Perceived effectiveness of learning methods among preclinical medical students-role of personality and changes over time t. Pak J Med Sci. 2021;37(7):1854–1859. doi: 10.12669/pjms.37.7.4355. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Ahluwalia T, Toy S, Gutierrez C, et al. Feasible and effective use of a simulation-based curriculum for post-graduate emergency medicine trainees in India to improve learner self-efficacy, knowledge, and skills. Int J Emerg Med. 2021;14(1):42. doi: 10.1186/s12245-021-00363-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Al-Ageely TA, Ismail M, Al-Araji ZA, et al. Effectiveness of mentorships and early exposure in neurosurgery. World Neurosurg. 2023;178:e773–e776. doi: 10.1016/j.wneu.2023.08.010. [DOI] [PubMed] [Google Scholar]
- 41.Collard A, Mélot F, Bourguignon JP.. Longitudinal assessment of progress in reasoning capacity and relation with self-estimation of knowledge base. Assess Eval High Educ. 2015;40(1):74–88. doi: 10.1080/02602938.2014.888394. [DOI] [Google Scholar]
- 42.von Davier M, Xu X, Carstensen CH.. Measuring learning and change in a longitudinal large-scale assessment with a general latent variable model. Psychometrika. 2011;76(2):318–336. doi: 10.1007/s11336-011-9202-z. [DOI] [Google Scholar]
- 43.Ament Giuliani Franco C, Franco RS, Cecilio-Fernandes D, et al. Added value of assessing medical students’ reflective writings in communication skills training: a longitudinal study in four academic centres. BMJ Open. 2020;10(11):e038898. doi: 10.1136/bmjopen-2020-038898. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Isbej L, Cantarutti C, Fuentes-Cimma J, et al. The best mirror of the students’ longitudinal performance: portfolio or structured oral exam assessment at clerkship? J Dent Educ. 2022;86(4):383–392. doi: 10.1002/jdd.12823. [DOI] [PubMed] [Google Scholar]
- 45.Moldovan F, Gligor A, Moldovan L, et al. The impact of the COVID-19 pandemic on the orthopedic residents: a pan-Romanian survey. Int J Env Res Public Health. 2022;19(15):9176. doi: 10.3390/ijerph19159176. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Kontopantelis E, Doran T, Springate DA, et al. Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ. 2015;350:h2750. doi: 10.1136/bmj.h2750. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The participants of this study did not give written consent for their data to be shared publicly, so due to the sensitive nature of the research, supporting data is not available.


