Abstract
Objectives
The medical field is facing a clinician-scientist shortage. Medical schools could foster the clinician-scientist workforce by offering students research opportunities. Most medical schools offer elective research programmes. Subsequently, a subset of doctors graduates without any research experience. Mandatory research projects may be more sufficient to develop clinician-scientist, but take more supervision and curricular time. There is limited insight in the scientific outcomes of mandatory research experiences. This study aims to examine publication rates of a mandatory research experience, identify factors associated with publication, and includes postgraduate research engagement.
Design and setting
Prospective follow-up study involving 10 cohorts of medical students’ mandatory research projects from Leiden University Medical Center.
Participants
All medical students who conducted their research project between 2008 and 2018 (n=2329) were included.
Main outcome measure
Publication rates were defined as peer-reviewed scientific publications, including research papers, reviews, and published meeting abstracts. Postgraduate research engagement was defined as research participation and dissemination of research at scientific conferences or in journals.
Results
In total, 644 (27.7%) of all mandatory research experiences resulted in publication, with students mainly as first (n=984, 42.5%) or second author (n=587, 25.3%) and above world average citation impact (mean normalised journal score 1.29, mean normalised citation score 1.23). Students who conducted their research in an academic centre (adjusted OR 2.82; 95% CI 2.10 to 3.77), extended their research (adjusted OR 1.73; 95% CI 1.35 to 2.20), were involved in an excellency track (adjusted OR 2.08; 95% CI 1.44 to 3.01), or conducted clinical (adjusted OR 2.08; 95% CI 1.15 to 3.74) or laboratory (adjusted OR 2.16; 95% CI 1.16 to 4.01) research published their research more often. Later as junior doctors, this group significantly more often disseminate their research results at scientific conferences (adjusted OR 1.89; 95% CI 1.11 to 3.23) or in journals (adjusted OR 1.98; 95% CI 1.14 to 3.43).
Conclusions
Our findings suggest that a significant subset of hands-on mandatory research projects with flexible learning pathways result in tangible research output with proper impact and that such successful experiences can be considered as diving board towards a research-oriented career.
Keywords: MEDICAL EDUCATION & TRAINING, EDUCATION & TRAINING (see Medical Education & Training), STATISTICS & RESEARCH METHODS
Strengths and limitations of this study.
This is the first prospective cohort study that bibliometrically reports scientific outcomes (publications) of a hands-on mandatory research experience including postgraduate research engagement in 10 cohorts with over 2000 medical students in total.
Insight in scientific outcomes (publications) of mandatory research programmes fills a gap in the literature since previous studies mainly focus on elective research outcomes with a subset of students graduating without any research experience.
Our study identified student and project factors associated with publication of a mandatory research project, thereby providing insight how to reach high academic levels among medical students.
Insight in postgraduate research engagement is limited due to loss to follow-up and non-response.
Publication rate is subjected to an underestimation of actual published papers due to publication delay and false negative cases.
Introduction
All doctors should be able to critically appraise and use research in clinical practice to keep up to date and apply evidence-based practice within their field of expertise.1 2 Additionally, society needs doctors to conduct research and contribute to new developments and knowledge.3 Clinician-scientists, that is, doctors with research expertise and engagement, do not only conduct research, but also play significant roles in directly translating clinical observations to the bench and in moving research findings into everyday practice. Thereby they contribute importantly to the development of tomorrow’s healthcare as newly invented medical solutions and developments will reach patients sooner.4 5 The adoption of this scholarly competency in frameworks as the US Accreditation Council for Graduate Medical Education and the Canadian Medical Education Directives for Specialists reflects the importance of doctors who conduct research.6 7 Despite this recognition, the number of clinician-scientists globally declined over the past few decades resulting in a shortage.8–12
A solution to overcome the clinician-scientist shortage is to engage medical students in research endeavours during medical school. Efforts concentrated on the research engagement of medical students and consisted of extracurricular or intracurricular research activities, the latter either mandatory or elective research programmes.9 13–17 To date, most medical schools only offer elective or extracurricular research programmes, such as summer schools and scholarly concentration programmes, mostly aimed at excellent or highly motivated students.9 18–20
Several studies demonstrated that these undergraduate research experiences (voluntary as well as mandatory) enhance research skills such as searching and critically appraising evolving medical literature, designing research, data analysis, academic writing and presenting.9 16 17 19 21–23 Furthermore, they foster research self-efficacy, positive research perceptions, motivation for research,24–27 and, on the long term, the ambition to pursue an academic career.9 11 14 17 19 28–30 Even more, some research experiences result in peer-reviewed publications, often assumed as an objective measure and a proxy for the ultimate learning experience of research programmes, and suggested to be one of the factors related to persistence within academic medicine.31–33 Considering these positive effects, one may argue that every medical student should engage in hands-on research. However, as current research experiences are mostly voluntarily, about 30%–70% medical students graduate without any hands-on research experience.17 20 34 Some of these students initially may lacked interest and motivation, while others did not participate in research due to time pressure, a lack of supervision, and/or opportunities.35–37
It may well be that elective programmes involve above average motivated and committed students.37 As such, previously described beneficial outcomes of elective research experience may differ from mandatory research experiences. Furthermore, given the limited curricular time, the benefits of mandatory research projects must outweigh the efforts of compressing an already tight learning schedule. In addition, proper supervision of mandatory research projects may demand substantial efforts from scientists and faculty, which might be justified if these research projects result in at least some publications. To the best of our knowledge, however, a large cohort analysis of medical students’ mandatory research output has not yet been conducted. This may prove useful to medical schools with established mandatory research programmes or others considering the introduction of a mandatory research experience. It can provide insight into the effects of mandatory research and help to influence policy around the introduction of mandatory research experiences and the enhancement of research-oriented careers among medical students. Therefore, this 10-year cohort study aims to investigate the scientific output based on number of publications resulting from mandatory research projects and identify key factors associated with these publications. In addition, we explore scientific engagement after medical school including the residency period and early clinical careers.
Methods
Setting
In the Netherlands, all eight medical schools educational programmes are based on the Dutch National Blueprint for Medical Education. The programme consists of a 3-year bachelor programme and a 3-year master programme. Individual mandatory research projects are longstanding part of each Master of Medicine and were already incorporated in all Dutch medical curricula even before 1970. Students have 4 to 6 months for a full-time, authentic, and hands-on research experience. They go through the phases of the empirical cycle by conducting their own research and develop research skills such as searching and critically appraising literature, designing research, analysing and interpreting data, academic writing and presenting. During this project students have much autonomy, for example, in arranging their internship at a health institute and department of preference, in choosing a research domain (eg, laboratory research, clinical research, public health research). In addition, students are free to choose the timing to conduct their research (ie, before or after clerkships) and to extend their research project with 5 or 10 weeks. During the research project, students fulfil the role of the primary investigator and receive input from one or few supervisors. Supervision is carried out by faculty staff members, that is, (clinician-)scientists or PhD candidates. As final products, students write a research report and orally present their findings at the department.
Materials and definitions
Publication rates and factors associated with publication
This follow-up study included all medical students from Leiden University Medical Centre, who started their mandatory research internship between 1 January 2008 and 1 January 2018. The latter cut-off was to allow for lag time between project completion and peer-reviewed publication. We extracted names and initials of all students together with the name of the supervisor(s) from course registration systems, together with other student factors (eg, participation in an excellency track) and project factors (eg, planned duration of the research project). Scientific output is operationalised as peer-reviewed publication rates of research projects. We included the following publications: research articles, meeting abstracts, and reviews, as these are described as most common measures for research success.9 Letters to the editor, editorial materials, corrections and news items were excluded. Within the publications, we looked at author position of the student, year of publication, and impact. For the latter, we used the mean normalised citation score (MNCS) as impact ratio of research articles, compared with the world citations average in the subfields in which the research unit is active, as well as the mean normalised journal score (MNJS) as impact ratio of the journal in which a research unit has published (the research unit’s journal selection), compared with the world citations average in the subfields covered by these journals.38 39
Postgraduate research engagement
For postgraduate scientific engagement, we developed a questionnaire (online supplemental appendix A) regarding research activity after graduation (other than accomplishing publication(s) of the research project). We defined conducting research as postgraduate participation in research, whether or not in the form of a PhD programme, next to disseminating research results, that is, publishing articles in journals or provide oral presentations at scientific conferences. This questionnaire was part of an institutional questionnaire about different postgraduate career pathways. Those who graduated before May 2019 were sent a questionnaire for postgraduate (ie, after medical school) follow-up.
bmjopen-2021-056053supp001.pdf (54.4KB, pdf)
Procedure
Publication rates and factors associated with publication
To identify mandatory research projects that resulted in a peer-reviewed publication, we searched full names of the students and supervisor(s) together with filters based on department and year of research project using validated bibliometric methods. Bibliometric methods enables to track scientific output of individuals strengthened by mapping individual hits to larger sets of publications (ie, author clustering), with more robust bibliometric scores of citation impact as a result. Author clustering algorithms are more accurate when more information is available, as publications can be clustered even when the initials do not match exactly.40 41 Consequently, students are more susceptible for false positive results due to a minimal oeuvre compared with their prolific supervisor(s). Therefore, as a first step, between December 2019 and January 2020, we searched names of all supervisor(s) in the in-house database in of one of the most comprehensive and widely used publisher-independent global citation database, Web of Science (WoS), at the Centre for Science and Technology Studies using a validated algorithm. This bibliometric search resulted in a list of clustered oeuvres of the supervisors. Second, we searched publications that also included the students’ name they supervised and considered these papers as publication that resulted from the research project. Common problems in such searches are false positive or negative assignments of papers, due to common Dutch names, forgotten initials or spelling errors.42 This problem is applicable for the bibliometric search to identify the oeuvre of the supervisor, as well as searching students’ names within this oeuvre. Therefore, we checked all included publications to distinct if the published paper matched the topic of the research project, department, and institute. Some false negatives are inevitable as a subset of students published in journals that are not indexed in the WoS-database (eg, a Dutch-language journal or English-language journals not processed for WoS), or because of spelling errors, missing initials, changed names, or changed initials. Complementary to bibliometric analysis, we performed a sensitivity analysis by manual assignment on a random sample of 150 research projects. By searching key words based on research title, next to students’ together with supervisors’ names on Google Scholar, PubMed, LinkedIn, and ResearchGate, 12% (n=18) false negatives and no false positives were identified. We critically studied these publications to identify explanations for being false negative in order to improve our search and added the publications to our dataset.
Postgraduate research engagement
After graduation, the Alumni Office registers medical graduates of whom 80% agreed to receive questionnaires. To identify long-term scientific engagement, we invited medical graduates from 2008 up until May 2019 by email with a link to the online questionnaire. Participants received information on the study and an informed consent form.
Analysis
Publication rates and factors associated with publication
We used descriptive statistics to describe demographic variables. We grouped the population into a publisher and non-publisher group to analyse factors associated with publication. An unpaired t-test was used to compare group differences (eg, age and gender) between the publishers and non-publishers. To identify what student and project factors are associated with publication, we used logistic regressions, both crude and adjusted for possible confounding variables. Additionally, regarding publications, impact score, author position and mean publication delay were analysed.
Postgraduate research engagement
For sensitivity analysis to identify possible (non-)response bias, we performed unpaired t-tests to disclose any differences between features (eg, age, publication) of responders and non-responders of the alumni questionnaire. To identify postgraduate outcomes associated with publication of the research project as student, we used multiple logistic and linear regressions. We adjusted for age, gender, and previous participation in an excellency track (ie, Honours programme) as possible confounders.43
We used a 95% CI to determine statistical significance. We analysed our data using IBM SPSS Statistics V26.0.
Patient and public involvement
No patients involved.
Results
Publication rates and factors associated with publication
Between 2008 and 2018, 2329 medical students had started their research internship. These students were 20 to 39 years (M=24.3, SD=2.0). Of all 2329 students, 1561 (67.0%) were female. In total, 644 students (27.7%) had one or more publication(s) as a result of their research project. Within the group that had published their research project, 57% has published one article, 15% has published two articles, 8% has published three articles, and 20% has published four or more articles related to their research project. Publishers and non-publishers did not differ in gender. However, they did differ in age with a mean difference of 0.46 years (95% CI 0.29 to 0.63). Further demographics are shown in table 1.
Table 1.
Demographics of student and project factors of mandatory research projects (n=2329)
| Variable | Mean (SD) or no (%) of students |
| Student factors | |
|
24.3 (SD 2.0) |
|
1561 (67.0) |
|
125 (5.4) |
| Project factors | |
|
1167 (40.9) |
|
|
| 5 weeks 10 weeks |
636 (27.3) 523 (22.5) |
|
|
| Clinical research Laboratory research Public health research Other |
1547 (66.4) 422 (18.1) 259 (11.1) 101 (4.3) |
|
1731 (75.5) |
|
216 (9.3) |
| Low-income country Middle-income country High-income country |
24 (11.1) 25 (11.6) 167 (77.3) |
Students who (1) were involved in an excellency track, (2) voluntary extended their research project with 10 weeks, (3) conducted their research in an academic medical centre, or (4) conducted clinical or laboratory research published their research project more often (table 2). After adjustment for potential confounding variables, effects of timing of the research project and doing research abroad lost significance. When looking at research abroad more closely, 24 projects were conducted in a low-income country, whereof one was published (4.2%) and 25 projects conducted in middle-income countries had no associated publications. Projects conducted abroad in high-income countries (n=136) resulted in 31 publications (18.6%).
Table 2.
Student and project factors associated with published research projects
| Factors associated with publication | Crude OR | 95% CI | Possible confounders adjusted for | Adjusted OR | 95% CI |
| Participated in an excellency track | |||||
|
1.00 2.31* |
1.60 to 3.32 | Age, gender | 1.00 2.08* |
1.44 to 3.01 |
| Timing of research project | |||||
|
1.00 0.89 |
0.74 to 1.07 | Age, gender, participation in an excellency track, type of institute, project duration, research type | 1.00 1.16 |
0.92 to 1.46 |
| Project duration | |||||
|
1.00 1.20 1.47* |
0.96 to 1.49 1.17 to 1.84 |
Age, gender, participation in an excellency track, type of institute, project duration, research type, timing of the research project | 1.00 1.20 1.73* |
0.95 to 1.51 1.35 to 2.20 |
| Research type | |||||
|
1.00 1.34 2.10* 2.24* |
0.72 to 2.47 1.21 to 3.61 1.26 to 3.97 |
Age, gender, participation in an excellency track, project duration, type of institute, timing of research project | 1.00 1.35 2.08* 2.16* |
0.69 to 2.61 1.15 to 3.74 1.16 to 4.01 |
| Type of institute | |||||
|
1.00 2.60* |
1.97 to 3.45 | Participation in an excellency track, project duration, research type, timing of the research project | 1.00 2.82* |
2.10 to 3.77 |
| Country | |||||
|
1.00 0.41* |
0.27 to 0.61 | Age, gender, participation in an excellency track, type of institute, project duration, research type, timing of the research project | 1.00 0.47 |
0.47 to 1.40 |
*P<0.05
Sensitivity analysis showed comparable results during the study period, publication rates excepted. The latter declined in the last 3 years (figure 1). Of all 2182 publications, 1451 (66.5%) were research papers, followed by 609 (27.9%) meeting abstracts and 122 (5.6%) reviews. Of all students who published their research project, almost half of them (46.0%) had at least two types of publications (eg, research paper and meeting abstract). When distinguishing research papers and reviews from meeting abstracts, over two-thirds (69.7%) of students with a meeting abstract had a research paper and/or review as well. Students were first author of 984 publications (42.5%), followed by second author of 587 publications (25.3%), third author of 349 publications (15.1%), and fourth author of 398 publications (17.2%). Publications were cited with an average of 17.8 citations per publication. The MNJS in which students published was 1.29, with an MNCS of 1.23. The average lag time between the research project and first publication of this project was 2.4 years.
Figure 1.
Published research projects per year.
Postgraduate research engagement
In total, 250 alumni (11% of all included students) participated in the survey. Table 3 shows main findings.
Table 3.
Postgraduate research engagement after publication of the undergraduate research project
| Association between publication and postgraduate research engagement | Crude OR | Crude 95% CI | Adjusted OR* | Adjusted 95% CI |
| Postgraduate research participation | 1.26 | 0.76 to 2.09 | 1.12 | 0.66 to 1.90 |
| Postgraduate research publication(s) | 2.11† | 1.25 to 3.54 | 1.98† | 1.14 to 3.43 |
| Participation in a PhD programme | 1.84† | 1.10 to 3.08 | 1.69 | 0.98 to 2.90 |
| Postgraduate research conference contribution | 1.99† | 1.19 to 3.34 | 1.89† | 1.11 to 3.23 |
*Adjusted for the following confounders: age, gender, participation in an excellency track
†P<0.05
The mean time between graduation and participation was 5.1 years (SD 2.7, median 4.5 years). We found no significant differences between the responder and non-responder group in gender, Honours programme participation, timing of research project or year in which the research project was started. The groups significantly differed in publication rates, with more publications of the mandatory research project in the responder group (mean difference −0.18, 95% CI −0.25 to 0.12).
Students who had published their undergraduate research project were more likely to publish (adjusted OR 1.98, 95% CI 1.14 to 3.43) after medical school or to share their research at a scientific conference (adjusted OR 1.89, 95% CI 1.11 to 3.23). Logistic regression showed a crude association between publication of the student research project and later enrolment in a PhD programme as medical doctor (OR 1.95, 95% CI 1.16 to 3.29). After adjusting for participation in an excellency track as possible confounder, this effect became marginally smaller (OR 1.74, 95% CI 1.01 to 3.00), as shown in table 3.
Discussion
The integration of scientific research projects into medical school programmes to develop scholarly doctors or even clinician-scientists is a widely discussed topic. Our study revealed that more than one in four medical students publishes findings of their mandatory research project in a peer-reviewed paper, mainly as first or second author. These papers are apparently of good quality as they passed peer review procedures as well with impact scores above world citation average, even though these students can be considered as relatively young researchers. Students who were younger, participated in an excellency track, conducted their research in an academic medical centre, and voluntary extended their project with 10 weeks by using elective weeks for the research project were more likely to publish their undergraduate research project. Timing or type of research did not impact publication rates.
Only few other studies have focused on the scientific output (publications) of mandatory research experiences. Three of these studies were conducted in private schools, with small amounts of graduates every year, which limits the generalisability, usability and applicability for education systems of public.44–46 Two other studies conducted in public schools reported publication rates of 11% and 17%, however, these were outdated or included less than 230 research projects.47 48 One other study conducted at a Dutch single institute included 551 research projects and describes a publication rate of 27%, in line with our results.49 Studies on elective or extracurricular research experiences reported publication rates between 14% and 75%, with limited numbers of students included usually selected on excellence or prior research interest.9 31 44 50 Although publication rates vary, at best 75%, this concerns a subset of an already pre-selected group representing the minority of all students. From this perspective, the scientific output based on number of publications of mandatory research experiences found in this study is relatively high and indicates actual scholarly development of medical students when research is imposed on them. It is important to note that comparisons of measured output reported in other studies should be done carefully, as variability in publication rates in the literature is likely attributable to differences in objective output for example, including or excluding meeting abstracts or oral presentations (the number of confirmed publications is, as expected, lower). Moreover, our study shows that student and project factors (eg, duration) are associated with publication rates of mandatory research experiences and might vary between institutes.
Student and project factors associated with publication provide insight in how faculties can optimise research experiences to foster the future clinician-scientist workforce. In line with Möller and Shoshan, we have found no gender difference regarding publication rates.51 Other studies are inconclusive and reasons behind a potential gender difference regarding publication rates remain unclear.26 50 52 53 While timing of the research projects apparently does not affect publication rates, extended duration results in higher publication rates. Half of all students are motivated to spend their elective weeks on extension of their research project. More time for research evidently leads to more mature research products with increased publication rates, which is also described by Dyrbye et al.44 Lastly, this study showed that projects conducted in an academic medical centre more often resulted in a publication. This might be attributed to the supervisor. Perhaps, projects conducted in an academic medical centre are more ‘publishable’ than others, as they are supervised by (clinician-)scientists working in an academic environment. This academic environment is highly research oriented as it includes research departments (ie, department of statistics and department of epidemiology) and facilitates, for example, journal clubs and research courses. Another explanation could be that these supervisors are more experienced in publishing research, as most clinicians in academic hospitals are involved in academic activities next to clinical care. Indeed, Alamri et al found that students with academic supervisors publish more often than those with non-academic clinicians as supervisor.50
Previous research has not demonstrated that mandatory research in medical school leads to a more productive academic career.9 22 This study provides a first insight in scientific engagement in the first years as medical doctor. It seems that graduates after publication of their research project tend to be more often involved in research and doctoral programmes, but this was not significant. However, when they did, they had significantly more scientific impact as they were two times as likely to disseminate their knowledge via peer-reviewed publications or presentations at scientific conferences. Perhaps, as (pre-)resident, these doctors are more scientifically literate and conduct high(er) quality research, which in turn might lead to more accepted published papers and orals at conferences. Another explanation is the power of success experiences for self-efficacy levels.26 Published student research projects might comfort students about publication issues and the dissemination of scientific knowledge and fosters future publications.31 This is an interesting outcome, as dissemination of research findings is essential for translating scientific outcomes to clinical practice and enhance evidence based patient care, considered as the most important aspect of clinician-scientists.
Furthermore, there is also the aforementioned selection effect for research opportunities to preferably hiring medical graduates who have published before.54 As a result, we cannot firmly state that the association with postgraduate research engagement is regardless or if because they had a greater interest in research, and whether the publication of their scientific work had directly benefitted postgraduate research opportunities. At the same time, unknown makes unloved; one may argue that there is a subset of students who on beforehand do not have the ability to take on extracurricular activities next to the overcrowded formal curriculum, hold inaccurate perceptions, or, perhaps, even do not have initial interest in research at all.35 36 This seems undesirable, as other studies showed that a significant subset (30%–70%) of students graduates without any research experiences,17 20 34 next to the clinician-scientist shortage. A mandatory research experience can provide them with an opportunity to explore how much fun it is, and an experience of success when they successfully fulfil their own research project, or even publish their first paper. As this is assumable, but cannot be drawn from our data, it would be worthwhile to explore if undergraduate mandatory research experiences positively affect research motivation, perceptions, and self-efficacy, and thus can foster future clinician-scientists who perhaps would have missed out on future research engagement when a first research experience would not have been imposed on them.
Our study has several limitations. First, although bibliometric methods are widely accepted and used for large-scale analysis of scientific output, false positive and negative results might occur. Sensitivity analysis resulted in 12% suspected false negatives and 0% false positives, suggesting that the observed 27.7% publication rate may underestimate the actual rate. Further adaption of our bibliometric search strategy risks the inclusion of false positives. Additionally, it is likely that conference proceedings were under-recognised in our study, as we included conference presentations as evidenced by publication of the associated abstract; however, many conferences do not publish abstracts. Therefore, we have to accept that our result is subjected to an underestimation of the number of actual publications. This is further strengthened by publication delay. Most publications appeared in the literature 2.4 years after research completion, a lag-period that is in line with findings from other studies and is especially applicable to research projects that have started at the final phase of our inclusion period.47 50 55 This might explain the decrease observed in figure 1, when looking at the last years of the analysis, as papers might still be in the process of getting published (‘the pipeline’).
As second limitation, we conducted this study at a single institution. However, van Eyk et al showed very small differences between Dutch medical schools’ scientific training regarding timing, duration and European Credits, as well as students’ publication rates during medical school.56 Therefore, we assume that our results are representative for other medical schools with similar mandatory research training.
A third limitation is that postgraduate responses were voluntary and despite the exact response rate is not known, 11% of all students were included for long-term follow-up. Although this is low, it does not substantially deviate from response rates of medical education surveys elsewhere in the literature. As a result, response bias might occur, as perhaps ‘publishers’ are more motivated to participate in our survey.
Conclusions
To our best knowledge, this is the first study investigating objectively verified publications rates as a result of undergraduate mandatory research experiences, together with associated factors and postgraduate outcomes in over 2000 medical students. Next to all students having experienced an authentic hands-on research project before becoming a clinician, a significant proportion of authentic undergraduate mandatory research experiences have great scientific value, judged by an overall publication rate of at least 27.7% of all medical students, with mainly first or second author positions and above world average citation impact. Especially when medical schools provide the opportunity to conduct research in an academic environment and facilitates flexible pathways regarding the duration and curricular position with respect to clerkships, for those who are willing to invest more. After experiencing such high levels of scholarly achievement during medical school, as young doctor, this group also more often disseminates their scientific findings with the field, enhancing the translation of research to clinical care, considered as one of the unique and distinctive aspects of clinician-scientists. As such, mandatory research experiences not only equip all future doctors with basic research knowledge and skills, but can also serve as breeding ground for potential clinician-scientists and can be perceived as worth it when countering the current decline in clinician-scientists.
Supplementary Material
Acknowledgments
The authors wish to thank all members of the research group of the Centre for Innovation in Medical Education at Leiden University Medical Centre for their critical appraisal of the manuscript.
Footnotes
Contributors: CRdB (guarantor) was together with BWCO, FWD and AJdB, responsible for the initial study design, data collection and data analyses. TNvL joined them and contributed to refinement of bibliometric data collection and analyses. CRdB drafted the manuscript and revised the manuscript before submission. BWCO, FWD and AJdB provided critical feedback on the study design, conduct and reporting of the study at several stages during the study period. All authors contributed to and approved the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
Ethics statements
Patient consent for publication
Not applicable.
Ethics approval
This study involves human participants and was approved by the Educational Institutional Review Board of Leiden University Medical Center approved this study (reference number OEC/ERRB/20191112/1). Participants gave informed consent to participate in the study before taking part.
References
- 1.de Beaufort AJ, de Goeij AFPM, AFPM de G. Academic and scientific education in medical curricula in the Netherlands: a programme director’s view. Perspect Med Educ 2013;2:225–9. 10.1007/s40037-013-0076-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Frank J, Snell L, Sherbino J. CanMEDS 2015 physician competency framework, 2015. [Google Scholar]
- 3.Laidlaw A, Aiton J, Struthers J, et al. Developing research skills in medical students: AMEE guide No. 69. Med Teach 2012;34:754–71. 10.3109/0142159X.2012.704438 [DOI] [PubMed] [Google Scholar]
- 4.Stewart PM. Academic medicine: a faltering engine. action is needed to respond to growing need and opportunities. Europe PMC 2002;324:437–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Woolf SH. The meaning of translational research and why it matters. JAMA 2008;299:211–3. 10.1001/jama.2007.26 [DOI] [PubMed] [Google Scholar]
- 6.Frank JR, Snell L, Sherbino J, eds. CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada, 2015. [Google Scholar]
- 7.ACGME . Accreditation Council for graduate medical education common program requirements (residency), 2018. Available: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2019.pdf [Accessed 13 Aug 2020].
- 8.Milewicz DM, Lorenz RG, Dermody TS, et al. Rescuing the physician-scientist workforce: the time for action is now. J Clin Invest 2015;125:3742–7. 10.1172/JCI84170 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Chang Y, Ramnanan CJ. A review of literature on medical students and scholarly research: experiences, attitudes, and outcomes. Acad Med 2015;90:1162–73. 10.1097/ACM.0000000000000702 [DOI] [PubMed] [Google Scholar]
- 10.Sklar DP. We must not let clinician-scientists become an endangered species. Acad Med 2017;92:1359–61. 10.1097/ACM.0000000000001870 [DOI] [PubMed] [Google Scholar]
- 11.Lopes J, Ranieri V, Lambert T, et al. The clinical academic workforce of the future: a cross-sectional study of factors influencing career decision-making among clinical PHD students at two research-intensive UK universities. BMJ Open 2017;7:e016823. 10.1136/bmjopen-2017-016823 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Sheridan DJ. Reversing the decline of academic medicine in Europe. The Lancet 2006;367:1698–701. 10.1016/S0140-6736(06)68739-4 [DOI] [PubMed] [Google Scholar]
- 13.Rosenkranz SK, Wang S, Hu W. Motivating medical students to do research: a mixed methods study using Self-Determination theory. BMC Med Educ 2015;15:95. 10.1186/s12909-015-0379-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ley TJ, Rosenberg LE. The physician-scientist career pipeline in 2005: build it, and they will come. JAMA 2005;294:1343-51. 10.1001/jama.294.11.1343 [DOI] [PubMed] [Google Scholar]
- 15.Weaver AN, McCaw TR, Fifolt M, et al. Impact of elective versus required medical school research experiences on career outcomes. J Investig Med 2017;65:942–8. 10.1136/jim-2016-000352 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Conroy MB, Shaffiey S, Jones S, et al. Scholarly research projects benefit medical students' research productivity and residency choice: outcomes from the University of Pittsburgh school of medicine. Acad Med 2018;93:1727–31. 10.1097/ACM.0000000000002328 [DOI] [PubMed] [Google Scholar]
- 17.Amgad M, Man Kin Tsui M, Liptrott SJ, et al. Medical student research: an integrated mixed-methods systematic review and meta-analysis. PLoS One 2015;10:e0127470. 10.1371/journal.pone.0127470 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.George P, Green EP, Park YS, et al. A 5-year experience with an elective scholarly concentrations program. Med Educ Online 2015;20:29278. 10.3402/meo.v20.29278 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Havnaer AG, Chen AJ, Greenberg PB. Scholarly concentration programs and medical student research productivity: a systematic review. Perspect Med Educ 2017;6:216–26. 10.1007/s40037-017-0328-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Wolfson RK, Arora VM. More on promoting medical student scholarly research. Acad Med 2016;91:159. 10.1097/ACM.0000000000001038 [DOI] [PubMed] [Google Scholar]
- 21.Griffin MF, Hindocha S. Publication practices of medical students at British medical schools: experience, attitudes and barriers to publish. Med Teach 2011;33:e1–8. 10.3109/0142159X.2011.530320 [DOI] [PubMed] [Google Scholar]
- 22.Parsonnet J, Gruppuso PA, Kanter SL, et al. Required vs. elective research and in-depth scholarship programs in the medical student curriculum. Acad Med 2010;85:405–8. 10.1097/ACM.0b013e3181cccdc4 [DOI] [PubMed] [Google Scholar]
- 23.Healey M, Jordan F, Pell B, et al. The research–teaching nexus: a case study of students’ awareness, experiences and perceptions of research. Innovations in Education and Teaching International 2010;47:235–46. 10.1080/14703291003718968 [DOI] [Google Scholar]
- 24.O'Sullivan PS, Niehaus B, Lockspeiser TM, et al. Becoming an academic doctor: perceptions of scholarly careers. Med Educ 2009;43:335–41. 10.1111/j.1365-2923.2008.03270.x [DOI] [PubMed] [Google Scholar]
- 25.Brancati FL, et al. Early predictors of career achievement in academic medicine. JAMA 1992;267:1372. 10.1001/jama.1992.03480100078035 [DOI] [PubMed] [Google Scholar]
- 26.Ommering BWC, van Blankenstein FM, van Diepen M, et al. Academic success experiences: promoting research motivation andSelf-Efficacy beliefs among medical students. Teach Learn Med 2021;33:423–33. 10.1080/10401334.2021.1877713 [DOI] [PubMed] [Google Scholar]
- 27.Boyle SE, Cotton SC, Myint PK, et al. The influence of early research experience in medical school on the decision to intercalate and future career in clinical academia: a questionnaire study. BMC Med Educ 2017;17:245. 10.1186/s12909-017-1066-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Siemens DR, Punnen S, Wong J, et al. A survey on the attitudes towards research in medical school. BMC Med Educ 2010;10:4. 10.1186/1472-6920-10-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. The Lancet 2010;376:1923–58. 10.1016/S0140-6736(10)61854-5 [DOI] [PubMed] [Google Scholar]
- 30.Jain MK, Cheung VG, Utz PJ, et al. Saving the Endangered Physician-Scientist - A Plan for Accelerating Medical Breakthroughs. N Engl J Med 2019;381:399–402. 10.1056/NEJMp1904482 [DOI] [PubMed] [Google Scholar]
- 31.Waaijer CJF, Ommering BWC, van der Wurff LJ, et al. Scientific activity by medical students: the relationship between academic publishing during medical school and publication careers after graduation. Perspect Med Educ 2019;8:223–9. 10.1007/s40037-019-0524-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Reinders JJ, Kropmans TJB, Cohen-Schotanus J. Extracurricular research experience of medical students and their scientific output after graduation. Med Educ 2005;39:237. 10.1111/j.1365-2929.2004.02078.x [DOI] [PubMed] [Google Scholar]
- 33.Wootton R, A simple WR. A simple, generalizable method for measuring individual research productivity and its use in the long-term analysis of departmental performance, including between-country comparisons. Health Res Policy Syst 2013;11:2. 10.1186/1478-4505-11-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Young BK, Cai F, Tandon VJ, et al. Promoting medical student research productivity: the student perspective. R I Med J 2014;97:50–2. [PubMed] [Google Scholar]
- 35.Almasry M, Kayali Z, Alsaad R, et al. Perceptions of preclinical medical students towards extracurricular activities. Int J Med Educ 2017;8:285–9. 10.5116/ijme.5973.297a [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Stockfelt M, Karlsson L, Finizia C. Research interest and activity among medical students in Gothenburg, Sweden, a cross-sectional study. BMC Med Educ 2016;16:226. 10.1186/s12909-016-0749-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Ommering BWC, van Blankenstein FM, Wijnen-Meijer M, et al. Fostering the physician-scientist workforce: a prospective cohort study to investigate the effect of undergraduate medical students' motivation for research on actual research involvement. BMJ Open 2019;9:e028034. 10.1136/bmjopen-2018-028034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Waltman L, van Eck NJ, van Leeuwen TN, et al. Towards a new crown indicator: some theoretical considerations. J Informetr 2011;5:37–47. 10.1016/j.joi.2010.08.001 [DOI] [Google Scholar]
- 39.Waltman L, van Eck NJ, van Leeuwen TN, et al. Towards a new crown indicator: an empirical analysis. Scientometrics 2011;87:467–81. 10.1007/s11192-011-0354-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Moed HF, De Bruin RE, Van Leeuwen TN. New bibliometric tools for the assessment of national research performance: database description, overview of indicators and first applications. Scientometrics 1995;33:381–422. 10.1007/BF02017338 [DOI] [Google Scholar]
- 41.Moed HF, Visser MS. Developing bibliometric indicators of research performance in computer science: An exploratory study. CWTS report. 1, 2007. [Google Scholar]
- 42.van Leeuwen TN. Modelling of bibliometric approaches and importance of output verification in research performance assessment. Res Eval 2007;16:93–105. 10.3152/095820207X227529 [DOI] [Google Scholar]
- 43.Ommering BWC, van den Elsen PJ, van der Zee J, et al. Using an extracurricular Honors program to engage future physicians into scientific research in early stages of medical training. Med Sci Educ 2018;28:451–5. 10.1007/s40670-018-0565-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Dyrbye LN, Davidson LW, Cook DA. Publications and presentations resulting from required research by students at Mayo medical school, 1976-2003. Acad Med 2008;83:604–10. 10.1097/ACM.0b013e3181723108 [DOI] [PubMed] [Google Scholar]
- 45.McPherson JR, Mitchell MM. Experience with providing research opportunities for medical students. Academic Medicine 1984;59:865–8. 10.1097/00001888-198411000-00004 [DOI] [PubMed] [Google Scholar]
- 46.Laskowitz DT, Drucker RP, Parsonnet J, et al. Engaging students in dedicated research and scholarship during medical school: the long-term experiences at Duke and Stanford. Academic Medicine 2010;85:419–28. 10.1097/ACM.0b013e3181ccc77a [DOI] [PubMed] [Google Scholar]
- 47.Al-Busaidi IS, Alamri Y. Publication rates and characteristics of undergraduate medical theses in New Zealand. N Z Med J 2016;129:46–51. [PubMed] [Google Scholar]
- 48.Salmi LR, Gana S, Mouillet E. Publication pattern of medical theses, France, 1993-98. Med Educ 2001;35:18–21. 10.1046/j.1365-2923.2001.00768.x [DOI] [PubMed] [Google Scholar]
- 49.van Wijk IJ, Daelmans HEM, Wouters A, et al. Exploring the timing of medical student research internships: before or after clerkships? BMC Med Educ 2018;18:259. 10.1186/s12909-018-1367-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Alamri Y, Currie W, Magner K, et al. Publication rates of, and attitudes toward, summer research projects: 10-year experience from a single institution in New Zealand. Adv Med Educ Pract 2019;10:263–71. 10.2147/AMEP.S198789 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Möller R, Shoshan M. Medical students’ research productivity and career preferences; a 2-year prospective follow-up study. BMC Med Educ 2017;17:51. 10.1186/s12909-017-0890-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Funston G, Piper RJ, Connell C, et al. Medical student perceptions of research and research-orientated careers: an international questionnaire study. Med Teach 2016;38:1041–8. 10.3109/0142159X.2016.1150981 [DOI] [PubMed] [Google Scholar]
- 53.Salgueira A, Costa P, Gonçalves M, et al. Individual characteristics and student's engagement in scientific research: a cross-sectional study. BMC Med Educ 2012;12:95. 10.1186/1472-6920-12-95 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Husnoo N, Goonoo MS. How to boost your specialty training application. BMJ 2016;352:i1392. 10.1136/bmj.i1392 [DOI] [Google Scholar]
- 55.Wells CI, Wallace HB, McLaughlin SJP. Rate and predictors of publication by medical and health science summer research students: a 14-year analysis from Auckland, New Zealand. MedEdPublish 2016;3:43. 10.15694/mep.2016.000129 [DOI] [Google Scholar]
- 56.van Eyk HJ, Hooiveld MHW, Van Leeuwen TN, et al. Scientific output of Dutch medical students. Med Teach 2010;32:231–5. 10.3109/01421591003596592 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2021-056053supp001.pdf (54.4KB, pdf)
Data Availability Statement
All data relevant to the study are included in the article or uploaded as online supplemental information.

