Distance Learning Improves Attainment of Professional Milestones in the Early Years of Surgical Training

A surgical sciences e-learning program designed to support academic development of trainees in the early years of surgical training was associated with improved success in surgical trainees' professional examination, positive student feedback, and significant academic attainment.

Abstract

Objectives:

To assess the impact of a surgical sciences e-learning programme in supporting the academic development of surgical trainees during their preparation for professional examination.

Background:

In 2007, a 3-year online part-time Master of Surgical Sciences (MSc) degree programme was launched, utilizing an innovative platform with virtual case scenarios based on common surgical conditions addressed by the curriculum relating to the Membership Examination of the Royal Colleges of Surgeons (MRCS). Multiple-choice questions with feedback and discussion boards facilitated by expert clinical tutors provided formative assessment. Summative assessment comprised written examination at the end of each of the first 2 years (equivalent to MRCS level), culminating in submission of a research dissertation in year 3 toward an MSc.

Methods:

Students' age, gender, and level at entry to the programme were documented. Anonymized student feedback from 2008 to 2012 was examined using online questionnaires, and performance in the MSc programme was compared to MRCS examination outcomes for students who had consented to release of their results.

Results:

A total of 517 surgical trainees from 40 countries were recruited over the 6-year period, and 116 MSc students have graduated to date. Of 368 students, 279 (76%) were foundation doctors (interns) and had not commenced formal surgical training on enrolling in the MSc programme. However, level at entry did not influence performance (P > 0.05 across all 3 years). Average pass rates since the programme launched, for those students completing all of the required assessments, were 84% ± 11% in year 1, 85% ± 10% in year 2, and 88% ± 7% in year 3 of the MSc programme. MSc students had significantly higher MRCS pass rates than nonenrolled trainees (67% vs 51%, P < 0.01, n = 352). There was a significant correlation between MRCS examination performance and overall performance in the MSc (R² = 58%; P < 0.01, n = 37). Of 248 respondents, 202 (81%) considered that the MSc would improve their chances of gaining a surgical training post, and 224 (90%) would recommend the programme to their peers.

Conclusions:

The online MSc programme supports academic development of trainees in the early years of surgical training, is well received by students, and is associated with improved success in their professional examination.

Recent years have seen major changes in surgical training worldwide, with reduced working hours enforced under European law,1 and the introduction of run-through and core training programmes in the United Kingdom, which no longer expose the surgical trainee to all the common surgical conditions and operations that would have formed part of traditional surgical training. Trends to reduce working hours are also evident in North America2 and Australasia.3 In addition, there has been a lack of teaching provision of knowledge and consistent educational support as the trainee has approached the major compulsory examination of their early professional career, Membership examination of the Royal College of Surgeons (MRCS), normally taken during the first 2 years of basic surgical training. In response to these changes, a jointly awarded modular qualification, the Edinburgh Surgical Sciences Qualification (ESSQ—http://www.essq.rcsed.ac.uk), was launched in 2007 by the University of Edinburgh and the Royal College of Surgeons of Edinburgh (RCSEd). The programme was based on the MRCS curriculum but was tailored to be equally applicable to the early years of surgical training programmes worldwide.

The online ESSQ provides learning material tailored to the professional needs of surgical trainees, delivered in a flexible and readily accessible manner that enables the student to measure their progress through the individual components of the MRCS curriculum, while providing peer support and senior mentorship during this critical phase of their professional development. The online clinical case scenarios have specific objectives that underpin the curriculum in the applied surgical sciences of anatomy, radiology, physiology, pathology and bacteriology, with modules in the assessment and care of the surgical patient, research, and communication skills.

The aim of this study was to determine the impact of the ESSQ e-learning programme in supporting the early academic development of surgical trainees in their preparation for professional examination. The success of trainees in completing their MSc degree and in securing their professional examination (MRCS) was assessed and the quality of the programme determined by securing direct student feedback.

METHODS

The ESSQ programme is intended for doctors in the early stages of their careers (pre-MRCS), and therefore those who already possessed the MRCS or Fellowship of the Royal College of Surgeons (FRCS) were ineligible to enroll. Applications were accepted only from trainees in possession of a medical degree from their country of origin (MBChB or equivalent) and who were based within a training environment. The programme provides academic recognition of trainees' acquisition of knowledge and development; the ESSQ leads progressively to a Certificate (year 1—basic science relevant to surgery), Diploma (year 2—clinical surgery) and an MSc in Surgical Sciences (year 3—research component) over a 3-year period of part-time study that runs alongside and complements full-time clinical training. Students in the first 2 years of the programme have access to e-tutor–assessed collaborative discussion boards, where they are expected to lead debate and facilitate interaction with others and bring original material to support their arguments (http://demo.essq.ed.ac.uk/Surgical_Oncology_Discussion.asp), in addition to completion of multiple-choice questions (MCQs) and mini essays, with formative feedback provided. Summative assessment of knowledge is undertaken through an end-of-year examination comprising single best answer questions and extended matching item questions in a similar format to the MRCS examination. In addition, there is an in-course assessment score reflecting the student's performance in various aspects of the course throughout the year. During the second year, students are also introduced to the basic concepts relating to their academic development in preparation for the final year, in which students are afforded the opportunity to undertake a period of novel research leading to a dissertation and the award of MSc in surgical sciences. Although the primary objective of the third year is to consolidate academic principles and learning, many individuals go on to publish their work in peer-reviewed journals.

Academic data regarding performance and assessment (end-of-year scores) were sourced from student records relating to academic years 2007–2008 to 2011–2012 of the MSc in Surgical Sciences programme at the University of Edinburgh, in partnership with the Royal College of Surgeons of Edinburgh. Demographic data collected included age, gender, and the training post held at the time of entry to the programme. In years 1 and 2, students were assessed on the basis of completion of in-course MCQ, discussion board contributions, essays (year 2 only), and final examination marks. The relative weighting of in-course assessment increased over the 3 years: 25% in year 1, 30% in year 2, and 50% in year 3. In year 3, students completed a research project proposal and summary, an e-poster, and a final e-dissertation, all of which contributed to the end-of-year score. All examination questions were quality assured (including Angoff standard setting) and the examination and in-course assessment results were ratified by a programme Board of Examiners at the University of Edinburgh, which included 2 external examiners, both consultant surgeons with academic appointments at other UK institutions, which included responsibilities for postgraduate clinical teaching. The selection of external examiners and conduct of the board of examiners were done in accordance with the University's Taught Assessment Regulations and Guidelines.

Anonymized student feedback was examined over the same period using archived responses to online questionnaires. Such feedback was sought on agreed pro forma that were circulated to students at the end of each academic year. A search of PubMed and Web of Knowledge was undertaken using ESSQ student name, supervisor name, and research topic to identify published research articles and abstracts resulting from the dissertation topic.

The MRCS is designed especially for surgical trainees in Great Britian and Ireland and is normally taken during the first 24 months of surgical training. Part A of the MRCS is a written examination on generic surgical sciences and applied knowledge, whereas the Part B Objective Structured Clinical Examination integrates basic surgical scientific knowledge and its application to clinical surgery. An e-mail was sent to all students who had completed both years 1 and 2 of the ESSQ, requesting their consent for the programme team to obtain MRCS Part A examination scores. Where consent was given, MRCS results were obtained from the “Assessment and Quality Assurance Examinations” section of the RCSEd. In addition, pass rates (of 100%) were obtained from the RCSEd for all MRCS examinations held during 2008 and 2012. Data were separated into pass rates for ESSQ students and non-ESSQ candidates. Immediately after their correlation with performance in the MSc programme, all data relating to individuals' MRCS examination scores were anonymized.

Statistical analysis was performed using SPSS Statistics 19 and GraphPad Prism 3.0 software. Differences between group means were identified using Student t test for 2-group comparisons, and 1-way analysis of variance for multiple group comparisons. The influence of age, gender, and level at entry on end-of-year scores was assessed using the Cox proportional hazards regression model. Linear regression analysis was carried out to determine the relationship between ESSQ performance and MRCS examination scores. Data are expressed as mean ± SD unless otherwise stated. Statistical significance was accepted when P < 0.05.

RESULTS

A total of 517 surgical trainees from 40 countries (Fig. 1) were admitted to the programme over the 6-year (2007–2012) period observed. Of the 205 trainees who entered the 3-year programme during 2007–2009, 116 completed successfully and have graduated with an MSc degree (57%), 36 (18%) completed at Diploma level, 26 (13%) completed at Certificate level, 4 (2%) have returned from an interruption of study, and 23 (11%) withdrew early with no award. The recruitment and overall progression rates of the ESSQ programme over the 5 completed academic years are shown in Table 1.

FIGURE 1.

The online distance-learning ESSQ/MSc in Surgical Sciences attracts students from 40 different countries.

TABLE 1. ESSQ Recruitment and Progression Data.

The majority of students, 279 of the 368 (76%) whose level of work experience was known, entered as foundation doctors (interns) within 2 years of graduation from medical school and before commencing formal surgical training. Of the remaining 89 trainees, 68, 18, and 3 students were in the first, second, and third years, respectively, of a surgical training programme (Fig. 2). Level of work experience at entry to programme was not found to influence performance across each of the 3 years of the programme (Fig. 3). On multivariate analysis, age, gender, and level of work experience at entry were not found to be predictors of end-of-year scores (Table 2).

FIGURE 2.

Surgical training structure in the United Kingdom. The trainee has to demonstrate the acquisition of knowledge and technical skills in the workplace and negotiate examinations of the Royal College of Surgeons (Membership, MRCS, and Fellowship, FRCS), which are mandatory requirements in demonstrating competency in surgery.

FIGURE 3.

The influence of level at entry to the ESSQ programme and academic performance (2007–2012). Individual marks (with mean values) shown for students at the end of (A) year 1 (n = 279), (B) year 2 (n = 188), and (C) year 3 (n = 117) reveal that there is no significant effect of level of training upon entering the programme and end-of-year results (P = 0.26, 0.12, and 0.48 for years 1, 2, and 3, respectively; 1-way analysis of variance). FY1/2 = Foundation year doctors (year 1 or 2 after graduation); ST1/2 = first-/second-year surgical trainee, equivalent to first-/second-year core trainee (CT1/2); ST3 = third-year surgical trainee.

TABLE 2. Multivariate Analysis of the Influence of Gender, Age, and Training Level at Entry, on Academic Performance on the ESSQ Programme.

Average pass rates since the programme launched were 84% ± 11% in year 1, 85% ± 10% in year 2, and 88% ± 7% in year 3. MSc students had significantly higher MRCS pass rates than nonenrolled trainees (Fig. 4), for both parts A and B of the examination. Comparison of the groups sitting the MRCS examinations revealed that candidates were similar in terms of age and expected level of clinical experience: ESSQ and non-ESSQ candidates both had an average age of 26 years. Furthermore, there was a significant correlation between MRCS examination performance and overall performance in the first 2 years of the MSc programme (Fig. 5).

FIGURE 4.

Pass rate in 2008-2012 for students enrolled (ESSQ) or not enrolled (non-ESSQ) in the ESSQ. (A) Combined or overall examination pass rate (mean ± SD; *P < 0.01, n = 352; unpaired student t test) and (B) separate part A and B examination pass rate (mean ± SD; *P < 0.05, n = 223; †P < 0.01, n = 129; unpaired Student t test).

FIGURE 5.

Regression analysis showing the association between student performance in the first 2 years of the ESSQ and MRCS part A examination outcome (y = 0.6505x + 34.859; R² = 58%; P < 0.01, n = 37) (95% confidence intervals).

Feedback from online, anonymous questionnaires revealed that 224 (90%) of 248 respondents would recommend the programme to their peers and the same percentage reported that they had a better understanding of the basic and applied surgical sciences than they would have otherwise had if they had not enrolled on the ESSQ programme. The majority of students (81%) considered that the ESSQ would improve their chances of gaining a surgical training post. In addition, 61% found the level of the MCQs within the ESSQ programme to be more difficult than those in the MRCS examination. On average, year 1 students felt that they needed to spend 10 ± 6 hours per week engaged in ESSQ activities to keep up-to-date with the timetable. Respondents consistently reported that they found the ESSQ an enjoyable experience. Search of PubMed and Web of Knowledge revealed that there have been 26 full research papers and 27 abstracts published to date relating to ESSQ research dissertations submitted May 2010 onwards.

DISCUSSION

The ESSQ/MSc in Surgical Sciences was established in 2007 as a response to changes in employment law in Europe, which reduced clinical exposure in the workplace.1 Contemporaneously, changes in surgical training were implemented with the aim of streamlining progression through training milestones and shortening the duration of training. The entire ESSQ programme, based on the MRCS curriculum, is taught through self-directed learning at a distance. It is delivered via a virtual learning environment which, allows students easy access to internal and external coursework, on an “anytime, anywhere” basis via the Internet on computer or mobile devices. The learning style provides students the options of a problem-oriented entry or systematic learning illustrated through subsequent problem solving, delivered in modules scheduled in blocks of time that accommodate and complement the student's clinical training demands.

Since its inception, the ESSQ has consistently exceeded its recruitment target with the highest student numbers of any postgraduate programme at the University of Edinburgh. Although targeted specifically at surgical trainees based in the United Kingdom, it has attracted more than 500 students from 40 countries in the last 6 years. The programme appears, therefore, to have a growing international reach with applicability to the early years of surgical training in many countries. This may be related to the current general reduction in working experience for doctors; the requirement to reduce working hours in Europe, North America, and Australasia1–3 has diminished considerably the trainee's exposure to both common and less frequently encountered surgical conditions.4,5 There is, therefore, a need to consolidate the applied knowledge that may be compromised by reduced exposure to the assessment of the patient in both the elective and emergency setting.6,7 We view distance-learning surgical educational programmes as an approach different from and complementary to traditional vocational methods, and which serve to enhance the learning experience of surgeons in training, not to replace but to complement experience in the workplace. The high recruitment to the programme and positive feedback from students would endorse this view.

The design of the ESSQ programme is underpinned by the pedagogical principles of e-learning, whereby students are supported as members of an online community with clearly defined learning objectives and use of materials that are active, engaging, collaborative, and relevant.8,9 Ease of navigation around the virtual learning environment and regular support from the programme team are critical in achieving successful online learning.10 The e-learning platform developed in-house (eeSURG, http://learning.essq.ed.ac.uk) allows students to navigate resources easily, follow the programme curriculum, access course material, and interact with course tutors and other students. Central to course design, Blanchard and Frasson11 emphasize the importance of roleplay to enhance constructivist learning. The activity model within the eeSURG platform is provided by Labyrinth (http://labyrinth.mvm.ed.ac.uk), a tool for authoring and delivering case narratives. Labyrinth was designed in-house with the goal of supporting development of richly engaging, narrative medical cases that invite users to take control of their own decisions—and, by extension, their own learning—and develop the critical analysis skills to face effectively the consequences of those decisions.12 It is noted that there was an appreciable “dropout” rate from the programme each year (10%–15%) but this was less than that observed in other distance learning masters programmes; online courses are associated with higher attrition rates than on-campus equivalents,13 with several studies reporting dropout rates for online masters programmes exceeding 40%.14

Although the online ESSQ/MSc in Surgical Sciences is designed to be studied part-time, alongside full-time clinical training in a hospital setting, the absence of a significant association between students' training level and academic performance on the ESSQ programme suggests that the content of the programme stands alone and is not necessarily dependent on previous surgical experience. Greater experience held by more advanced trainees may be offset by a longer period away from formal academic study, conferring an advantage to recent graduates who are more accustomed to intensive study. Furthermore, it is recognized that each student within a particular level of foundation (intern) or surgical training can have a very different profile of skills and experience compared with others at the same level, accounting for the lack of a relationship between performance and level at entry to the programme.

Given that one of the principal aims of the ESSQ programme was to prepare the surgical trainee better for the individual components of the MRCS examination that were covered in the first 2 years to Diploma level, monitoring pass rates for those students who have presented themselves for MRCS examination in the United Kingdom has provided a valuable assessment of outcome. The increased average pass rate for ESSQ students (∼15% higher) than non-ESSQ students add credence to the positive appraisal of the programme to date and will contribute to future student engagement and motivation. The correlation between performance in the ESSQ programme and MRCS examination scores and the positive feedback from students further demonstrates that the programme is fit-for-purpose. It is accepted that a potential limitation of the current analysis is that the ESSQ may not improve MRCS outcomes per se, but rather the ESSQ students are a self-selected group of highly motivated students who were more likely to be successful in the MRCS.

The programme, however, was also intended to support the academic development of the surgical trainee at an early stage, and it is noteworthy that 116 students who enrolled in the programme since its launch in 2007 continued to complete a postgraduate degree. This has provided them with a strong academic foundation irrespective of their final career intentions. It may encourage some to pursue an additional period of research training, whereas for others, the academic exposure may replace the traditional approach, which has required surgical trainees in the United Kingdom to pursue 2 or 3 years out of surgical programme training to secure a postgraduate qualification that may make them competitive to progress thought-advanced surgical training. Although, at the present time, there is insufficient follow-up of MSc postgraduates to determine whether their participation will impact on their future surgical training plans or academic development, it is encouraging to note that 26 full research articles have been published directly from content relating to the MSc dissertation project.

There is considerable potential for distance learning programmes to have a major impact on the future delivery of surgical training. The data from this study are relatively immature to determine whether it has contributed to a major change, but it has been demonstrated that the online ESSQ programme supports the academic development of trainees in the early years of surgical training, is well received by students, and is associated with improved success in their professional examination.

DISCUSSANTS

I. Popescu (Bucharest, Romania):

First of all thank you very much for the honor of reviewing this paper. Second, congratulations for the nice work in a very important area; that of education. As a comment, in this paper, the authors have explored the impact of a surgical sciences e-learning program in supporting the academic development of surgical trainees for their professional examination. Thus, it has showed that an e-learning platform is associated with improved outcome in professional examination, master in surgical sciences. Students also had significantly higher MRCS pass rates than nonenrolled trainees; however, it is not clear how the surgical trainees were recruited in the program. Did they voluntarily enroll in the program and did they have to pass a prior examination? One might speculate that surgical trainees who were voluntarily enrolled (with or without a prior examination) could be a priori selected students (the most motivated ones and with the best performances of their generation).

Taking into consideration the aforementioned factors, and including the fact that the questions were similar to the MRCS questions, that may bias the results along with the fact that master in surgical science students have gained added knowledge and expertise within the program. So, in my opinion, it is not quite a surprise that those students have had higher MRCS pass scores.

So, finally a question, I would like to ask the authors if there was any significant differences at MRCS pass scores between the surgical trainees that graduated, the highest level with master in surgical science degree, and the other types of enrolled students. Thank you very much.

Response from O.J. Garden (Edinburgh, UK):

Thank you. You make a number of important points and the first is in terms of which trainees were allowed to enter the program. Clearly, they must have a medical degree, normally be within 5 years of graduation and be based in a supportive training environment. We do accept your point that the format of part of the assessment of our program is similar to the MCQ component of the MRCS examination. Inevitably, this is going to prepare the student better for that examination. I guess that is part of the aim, but we would also like to think that this also results from their engagement with the program over a prolonged period of time through year 1 and year 2. We have not looked specifically at the performance of the students who have gone on to complete the 3 years of the program. I think the numbers are too small, but I suspect that we will find that these are actually top performing students and I would be surprised if they did not perform well in the MRCS examination.

DISCUSSANTS

R.P. O'connell (Dublin, Ireland):

James, this is greatly to be welcomed because any investment in the education of our trainees, such as you have done, is greatly welcomed. But following the last question, it is a bit of a self-fulfilling prophecy that if you invest in the education, and your endpoint is how they do in the membership examination, clearly they are going to do better. My question for you is how do you know they are better clinical scientists as was the objective because it is no longer fashionable to do clinical or laboratory research? What you want to do is to produce clinician scientists and how are you going to measure that outcome?

Response from O.J. Garden (Edinburgh, UK):

I think it is too early in the program to actually assess that parameter specifically. You will appreciate that it is a contentious issue as to whether or not we should be discouraging some trainees from taking time out to go through the traditional route of academic development, and I know that our own trainees in the southeast of Scotland generally do take time out to undertake a period of full-time study toward an MD or a PhD but not all of them go on to become academic surgeons. I think that there needs to be a way of trying to support their academic development, without taking them out of the clinical training program while providing them with a quality-assured academic qualification. Clearly the students are voting with their feet in saying that that is what we actually want. On the contrary, some of the trainees who have been through this program have actually used this as a stepping-stone to full-time research. Only time will tell as to how the program is going to best fit the needs of the modern surgical trainee.

DISCUSSANTS

J.V. Lanschot (Rotterdam, The Netherlands):

Thank you very much. As I recently learned, the scope of the College in Edinburgh is really global. Some of the residents in The Netherlands have tried to participate in your programs and at the end of the day during the examinations they have to really learn what is the correct answer in the UK to some questions, and what the correct answer would have been in The Netherlands. So, they are in a split, they learn things that they should not learn at least in The Netherlands. My question is what percentage of your program is really global and independent of local knowledge and what percentage is in fact more or less useless to international students?

Response from O.J. Garden (Edinburgh, UK):

I am surprised to find out that the surgical evidence base in The Netherlands is different from the evidence base in Great Britain and Ireland, but you do make an important point. This examination, this academic qualification, was very much structured around training in Great Britain and Ireland, and we have been taken quite by surprise that there has been so much interest internationally, but most of the trainees outside of Great Britain and Ireland obviously feel that the content of the program is actually fit for their purpose. It is not delivered in a very didactic way, it is there to support their reflective learning and as we know, from this very meeting, there are many views sometimes on how a particular condition should be managed, when a particular operation should be performed. So, we like to think that what we are trying to do is to develop that academic ethos, to look more critically at the evidence that is there.

DISCUSSANTS

N. Senninger (Munster, Germany):

With reference to control mechanisms for your testing, I am convinced that you are going to be successful but you will have to prove it. I would be interested to see what happened, concerning examination performance, to the so-called dropouts or to the ones who were not permitted to the test. What happened to them in comparison to the ones who passed the test successfully?

Response from O.J. Garden (Edinburgh, UK):

That again is a very important area to try and pursue. The withdrawal rate is lower than many distance learning programs and the predominant reason for students withdrawing is for either financial reasons or a conscious decision to change career such that the programme becomes less relevant to them. Unfortunately, a lot of students who do drop out do not really come back to us when we are looking for information as to why they have left the program and what they have then gone on to do.