Using the Progress Test Medizin (PTM) for evaluation of the Medical Curriculum Munich (MeCuM)

Abstract

Aims: The Medical Curriculum Munich (MeCuM) has been implemented since 2004 and was completely established in 2007. In this study the clinical part of MeCuM was evaluated with respect to retention of the knowledge in internal medicine (learning objectives of the 6th/7th semester).

Methods: In summer of 2009 and winter of 2009/2010 1065 students participated in the Progress Test Medizin (PTM) from Charité Medical School Berlin. Additionally the students answered a questionnaire regarding the acceptance and rating of the progress test and basic demographic data.

Results: The knowledge of internal medicine continuously increases during the clinical part of the medical curriculum in Munich. However, significant differences between the sub-disciplines of internal medicine could be observed. The overall acceptance of the PTM was high and increased further with the study progress. Interestingly, practical experiences like clinical clerkships positively influenced the test score.

Conclusions: The PTM is a useful tool for the evaluation of knowledge retention in a specific curriculum.

Introduction

The Problem

The goal of medical education is to impart practical skills and knowledge enduringly [1]. In contrast to summative examinations relevant to passing examinations, the formative Medicine Progress Test (MPT) examines spontaneously retrievable, and thus enduringly stored knowledge, because the students do not prepare [2]. The main objective of progress tests is individual feedback to students [3]. But the data can also be analysed with respect to curriculum evaluation. Progress testing is a reliable tool that is used across institutions and countries [4]. Applied longitudinally (analysing knowledge growth curves), progress tests can compare various curricula or curricular changes [5], [6], [7]. Here, longitudinal growth of knowledge is a more stable and reliable parameter than measuring of individual points of reference [8]. In previous studies, a longitudinal survey was carried out and growth of knowledge measured [5], [6], [7], [8]. As part of this project at the Ludwig-Maximilians-Universität (LMU) Munich, a cross-sectional analysis was performed and the permanence of acquired knowledge in specific technical topics checked. Here we analyse an example from “conservative medicine”. In the clinical part of the Medical Curriculum Munich (MeCuM), after a share semester on the basics of clinical medicine in the 6^th semester, students are divided into two streams, one on conservative and one on surgical medicine and swap in the 7th semester. Since the abolition of the State Examination before the start of the Practical Year, once students have received their certificates in internal medicine and surgery, they will not be examined in these subjects until the Part 2 of the Medical Examinations at the end of their undergraduate studies and received no feedback regarding these subjects. The Medicine Progress Test used by the Charité Berlin was introduced at the LMU Munich in the 2009 summer semester. The curriculum planners and co-managers of MeCuM until then had not known how knowledge of the individual subjects and specialist disciplines was progressing, whether knowledge was regressing, was maintained or even increased.

Goals

The aim of this study was to evaluate the permanence of knowledge gained in the “Conservative Medicine” MeCuM module.

Problem

How big is the gain of spontaneously accessible knowledge in relation to the learning targets of the “Conservative medicine” module students at the LMU acquire and how enduring is this knowledge in the subsequent course of their studies? Is MPT helpful for the evaluation of a medical curriculum?

Methods

Medicine Progress Test at LMU Munich

Figure 1 (Fig. 1) shows the rotation of student cohorts in MeCuM and the measurement time points. Students from Module 1 are distributed between Module 2 and 3, then swap in the following semester. The following semester, students are distributed among Modules 4 and 5, then swap the following semester. In the first week of the 2009 summer semester and 09/10 winter semester, the the PTM (http://ptm.charite.de/) was introduced obligatorily at three points in the clinical part of undergraduate studies: T1: start of Module 2 (“Conservative Medicine”, M2; cohorts A and B), T2: Start of Module 4 (“Nervous System and Sensorium”, M4; cohorts C + D) and T3: during the Practical Year. In the PY cohort (T3; cohorts E + F), only those students in the Practical Year or module 6 were examined who were not abroad and were registered for the State Examination Repetitorium (LMU-StaR). LMU-Star is an optional blended learning concept which accompanies the PY and is used by the majority of students in the PY. An analysis of the basic data (age, gender, school leaving grade, grade of the First Medical Examination) showed that the LMU-Star participants did not differ in a statistically significant way from the rest of PY students. Over the two measurement time points, all students in the clinical stage of studies beyond Module 1 were tested once only (N=1065; Summer Sem. 09: T1=249, T2=202, T3=156; Winter Sem. 09/10: T1=197, T2=190, T3=71). For the evaluation, the MPT questions were assigned to the MeCuM modules, in particular Module 2 “Conservative Medicine”, by the organiser of Module 2. The internal medicine sub-disciplines (endocrinology, gastroenterology, infectious diseases, cardiology, nephrology, oncology, pneumology, rheumatology) were categorised according to the learning targets of Module 2. Immediately after completing the MPT, students filled in an evaluation sheet. This included both questions on the MPT and other questions about personal data. The evaluation response rates were T1=94%, T2=92%, T3=94% (Summer Sem. 09), T1=79%, T2=82%, T3=89% (Winter Sem. 09/10 ), with a mean total of 88.3%.

Figure 1. Student cohorts and modular structure of MeCuM.

Definition of the various student cohorts in the 2009 summer semester and the 2009/2010 winter semester and the measurement points of the progress test.

Statistics

The statistical analysis and generation of the graphics was done in the Open Source statistical package R (http://www.r-project.org/), Version 2.9.0, and using Microsoft Excel. As a normal distribution could be shown for the PTM results, the calculation of correlation coefficients was done according to Pearson. For most collected variables, mean and standard deviation (SD) were also determined. The p-values listed were determined, given the normal distribution, using a t-test, otherwise using a Wilcoxon rank-sum test. To calculate the correlation of MPT performance with items of the questionnaire (leaving grades, First State Examination, self-assessment, etc.), the rank correlation coefficient Spearman’s rho was used. The box plots show the range between the 25% and 75% percentile, the line within the box indicates the median. If the notches of two groups intersect, the difference is significant.

Results

Implementation, Results and Acceptance of the MPT at the LMU

The implementation of the MPT was, in accordance with the guidelines of the Charité - University Medicine Berlin, without problem from an organisational point of view. Regarding overall performance, the LMU students with an average of 46.3 points (SD=21.7) at the beginning of the 6th semester, 61.7 points (SD=21.8) at the beginning of the 8^th semester and 77.9 (SD=24.9) at the beginning of the 10^th semester are comparable with the students of the control programme in Berlin (36.8, 57.7 and 71.2 points, SD=18.2, 22.6 and 27.2) and the students at the University of Witten/Herdecke (45.8 or 55.3, SD=20.7 and 17.4, no data available for the 10^th semester). The acceptance of the MPT by LMU students was very high. The question on “I consider individual feedback on my performance important [Likert scale: 1=strongly agree, 6=strongly disagree]” was answered, on average, with 1.73 (N=894, SD=0.91). This desire fore feedback increases as studies progress and reaches a value of 1.20 at the beginning of the PY (T3). The MPT was globally assessed (“I rate the Medicine Progress Test with the following grade:”) with the average grade of 2.27 (N=917: SD=0.92). This leads into an overwhelming endorsement of the solid implementation of the MPT at the LMU (“The Medicine Progress Test should be introduced at the LMU. [Likert scale: 1=strongly agree, 6=strongly disagree]”). Accordingly, in particular the T3 cohort with 67% argues for conducting it once per semester (“How often do you think the Medicine Progress Test should be carried out during an undergraduate degree?”, [Each semester - once a year - less often - never]). Only 4% of students of this cohort suggested “less often” or “never” (T1: 22% T2: 12%).

Knowledge Progression in “Conservative Medicine” in the MeCuM

Using the data of the large cross-sectional analysis, an evaluation of Module 2 “Conservative Medicine” was carried out. Based on subjective judgements by professors at the LMU and in analogy to a large analysis of knowledge on internal medicine amongst practicing physicians [9] which demonstrated a significant inverse correlation between the test result and time elapsed since the certification examination, it was investigated if knowledge in “conservative medicine” decreases in the subsequent progression of studies. Moreover, the curriculum committee assumed and had the subjective experience during Module 3 “Medical Surgery”, many learning targets of Module 2 “Conservative Medicine” were also taught. Through such non-targeted redundancies, serious inhomogeneities are created in the prior knowledge of students, making it teaching in small-groups difficult (seminars, problem-oriented learning tutorials).

Figure 2 (Fig. 2) shows that the knowledge of “Conservative Medicine” (Module 2) increases in Module 2 and 3 (T2 versus T1 comparison) but contrary to initial expectations, continues to increase in the subsequent course of study until the beginning of the Practical Year (T3). The results of the 2009 summer semester and the 09/10 winter semester were pooled and the result is shown as a percentage of points achieved. The number of questions about “Conservative Medicine” and therefore the absolute number of achievable points are different in each semester. Figure 3 (Fig. 3) shows that in respect to Module 3, a significant increase in knowledge of “Conservative Medicine” occurs (A vs. B). After completion of both modules, the results return to an identical level (C, D) and continue to increase up until T3 (E/F) although this increase is not statistically significant. The learning targets in Module 2 “Conservative Medicine” are divided in the various sub-specialisations of internal medicine. It was therefore also investigated whether there are differences in the knowledge gain between these specialisations. Figure 4 (Fig. 4) shows that subjects with higher knowledge growth (for example cardiology and rheumatology) can be distinguished from subjects in which knowledge regarding specific learning targets tested in the Progress Test is virtually unchanged (for example infectious diseases and nephrology). Further analysis shows (see Figure 5 (Fig. 5)) that prior knowledge from the preclinical stage of studies and Module 1 “Clinical Basics” is low internal medicine with respect to the specific learning targets. Cardiology aside, only endocrinology, oncology and pneumology show comparably high initial levels. Regardless of this prior knowledge, there is a marked increase in knowledge in Module 2 in disciplines such as cardiology or rheumatology. These show virtually no further increase in the further course of studies. On the other hand, students make up for their lack of knowledge in nephrology and infectious diseases as their studies progress. Here the significant growth occurs only after completion of Module 2. In an interdisciplinary subject such as infectious diseases, this is can be easily understood.

Figure 2. Knowledge growth on “Conservative Medicine” in the MeCuM.

Y-axis: Share of correct answers; x-axis: measurement points (Definition T1, T2 and T3 at Figure 1).

Figure 3. Knowledge growth on “Conservative Medicine” in the MeCuM.

Results of individual cohorts (see Figure 1 for definition).

Figure 4. Knowledge growth in sub-disciplines of internal medicine in the MeCuM.

Y-axis: Knowledge growth (T2 - T1) in % of achievable maximum points.

Figure 5. Knowledge growth in sub-disciplines of internal medicine in the MeCuM.

Y-axis: Share (correct-wrong) questions in the listed sub-disciplines. X-axis: Measurement points (see Figure 1 for definition).

Influencing Factors for MPT Results

The measured cohort was checked for the actual qualities the MPT was measuring for validity testing or more specifically, the participants’ parameters which influenced the test results.

Performance

When looking at the knowledge of Module 2 “Conservative Medicine”, a significant correlation (rho=-0.38) can be seen between the performance in the Module 2 exam and the MPT points in internal medicine. The Module 2 exam is a summative written examination to be held in two parts (at the middle and end of Module 2) and contains both multiple-choice (MCQs) and open questions. Also, the achieved MPT points in internal medicine also correlated significantly with school leaving grade (rho=-0.26) or the First Part of the Medical Examination (rho=-0.37). The gender of the participants had no influence on the result.

Motivation

Another factor that was checked was motivation to participate. As a surrogate parameter, the subjective global assessment of the MPT (“grading”, see above) and the desire for its introduction at the LMU (“implementation”, see above) was used. Here better subjective evaluation results is linked to better performance. A significant correlation (rho=-0.29) can be observed here, as well as for the (rho=-0.36) correlation between self-assessment (“How do you assess your results in the Medicine Progress Test/Internal Medicine overall? In my year, I belong to the...” [Likert scale (1-5): very good, good, average, poor, very poor participants]) and the MPT result. This correlation is reflected almost identically for the items “Conservative Medicine” but irrespective of whether the students had already taken Module 2 “Conservative Medicine”.

Experience

According to self-evaluation, the MPT displays the cross-section of the knowledge level, which is expected of a graduate on their first day at work. Many of the items contain a clinical case vignette. It should therefore be investigated whether, in addition to individual performance (reflected by the school leaving grade or the examination at the end of the pre-clinical phase), the clinical experience of students also affects the MPT test result. A significant correlation (p <0.0001) between the completion of the 4 month work placement (Famulatur) and the overall MPT result. The improved results in “Conservative Medicine” through the PY tertial in internal medicine at the end of undergraduate studies turned out not to be significant (n=60).

Discussion

The Medicine Progress Test is a formative feedback tool for students, which checks the knowledge that a doctor on their first day should possess [2]. If one brings together all students of a university at a certain point in time and only examines particular area, then the progression of expertise during the course of their studies and thus the curriculum can be evaluated using the same test. The MPT so far has only been specifically used for curriculum evaluation in a small study in 2004 in the reformed degree course at the Charité University Berlin Medical School. In a UK study, final year students were tested on their knowledge on musculoskeletal topics [10]. The most comparable study comes from the Netherlands. In this study the growth of knowledge in psychiatry and behavioural sciences was analysed in a problem-based, student-centred curriculum [11]. Here curricular changes were because no further knowledge growth could be detected in the last two years of study as would have been the expectation in a PBL curriculum. Since the students were largely self-selecting the date of when they tackled learning targets, an analysis of the permanence of acquired knowledge was not possible here. The German Medicine Progress Test, Charité University Berlin Medical School has already been used for curriculum evaluation [12] but mainly for interdepartmental comparison and not to analyse individual components of a specific curriculum.

Of course inferences on the quality of the curriculum can only be drawn very indirectly through the evaluation method described in this study. However, the permanent retention of knowledge is an important outcome parameter of a degree program. Communication and practical skills in this approach, however, are almost completely ignored. The significant impact of the 4 month work placement could either point to the fact that the largely case-based German MPT to a relevant extent tests clinically relevant practical knowledge or that an increased amount of factual knowledge is learned during the 4 month work placement. Holding the 4 month work placement early could, however, be evidence of an above-average motivation of this group of students. While the school leaving grade and the subjective self-assessment correlate with the test result, attitude towards MPT is not a major factor. Being critical towards our own work, it must be noted that for organisational reasons only those students could participate at point T3 who were taking part in the Munich State Examination Repetitorium LMU-StaR. However, in terms of basic data (age, gender, school leaving grade, grade of the First Part of the Medical Examination) this subgroup did not differ in a statistically significantly way from the other students, meaning the selection bias is probably low. Future investigations should give more weight to examining motivational aspects in this regard. Overall, a longitudinal analysis appears to be the appropriate method to for answering this question, however, there is something attractive about getting results quick. In longitudinal studies, curricular reform could onlz take place after several years and its effectiveness again would be measured after several years.

The aim of this cross-sectional analysis of all students in the clinical part of their studies at the LMU was the evaluation of Module 2 “Conservative Medicine” and clarification of whether the Medicine Progress Test can be used as an evaluation instrument for such a purpose. The MeCuM is organised into autonomous modules in which basic teaching in internal medicine and surgery is taught soon after the clinical core subjects in Module 1. There always has been, and still is, the fear amongst teachers and students that this knowledge is mostly lost by the start of the PY. Our cross-sectional analysis using MPT showed, however, that knowledge of “Conservative Medicine” continues to expand even after completion of Module 2 “Conservative Medicine”. Comparable results of knowledge increase of 16.6% in absolute terms in the permanence test after eight months were found in a Mexican study of 584 medical students in their 3rd year of training in pharmacology [13]. Here, the students had deliberately prepared for the exam and were examined again after eight months without preparation. This setting broadly corresponds to the situation under investigation at the LMU. Interestingly, these results from Mexico could only be found in a newly-designed pharmacology course; the old, traditional course did not display knowledge gain. Similar results were also found [14] in a North American study. Other analyses, mainly on basic biomedical knowledge, showed between 2.9% and 35% of knowledge loss after 15 to 21 months [15], [16], [17]. In a well planned current study, three subjects (immunology, physiology, and neuroanatomy) were compared [18], showing dramatic loss of knowledge of 17.6%, 19.4% and 52.7% after 10-11 months. The differences were not related to the results in the primary test or the acceptance of the courses by students. The acquisition of permanent knowledge is influenced by active learning and the nature of the teaching material (procedural vs. declarative, general vs. specific). Following a multi-disciplinary surgery course on mesorectal excision in Canada, there was no significant loss of knowledge even after a year [19]. Due to the different curriculum structures and sequence of the implementation of learning targets, a direct comparison between the universities investigated in Munich (LMU), Berlin and Witten/Herdecke is not possible. However, when applying the LMU learning target definition to the curricula in Berlin and Witten/Herdecke, it can be shown that there is also a continuously rising curve of knowledge in “Conservative Medicine”. Ultimately it cannot be clarified whether this is a general phenomenon in the field of internal medicine, or whether all three universities foster a particular type of knowledge retention due to their specific educational concepts. In the LMU-specific analysis, it was particularly interesting that Module 2 “Conservative Medicine” there was virtually no increase in knowledge in some sub-disciplines of internal medicine although they are not differentiated in the curriculum from other forms sub-disciplines of internal medicine as regards their teaching and examination methods. Most likely, these are down to inconsistencies between Module 2 learning targets and the examination objectives of the MPT. This feedback can lead to improvements in Module 2. Moreover, the cross-sectional analysis identify inhomogeneities in the prior knowledge of students, cause by the curriculum (especially through different module sequence, see Figure 1 (Fig. 1)) which is perceived as disturbing by students and teaching staff. This data supports the planned curriculum reform at the LMU (an organ-centred, interdisciplinary, surgical-internal medicine base year).

In summary, this analysis shows that the introduction of the MPT can contribute to the evaluation of a curriculum regarding permanence of subject knowledge. Using the example of internal medicine, it has been shown that permanent knowledge increase occurs continuously and depends on the practical experiences of students in the clinical part of their studies. Elsewhere, deficits of the curriculum could be documented. Using such cross-sectional analyses repeatedly, the effects of successful curriculum change can be evaluated.

Acknowledgements

This article was part of a project in the Master of Medical Education (MME) program of the Medical Faculty Day at the Medical Faculty of the University of Heidelberg.

Competing interests

The authors declare that they have no competing interests.