Abstract
Around 100,000 medical conferences are organized all over the world and hence, they form an integral part of a medical professional’s life. Oral presentations, especially award sessions, are judged by a panel of faculty judges who score individual presentations on various aspects including content, delivery and submission. Our objective was to compare the scores given by student-judges and faculty-judges for scientific presentations using the standardized score sheet. An objective, structured score-sheet was designed using existing literature. Five presentations, all made using PowerPoint, were judged using the structured score-sheet by seven student-judges and two-faculty judges. The mean score of all the score-sheets (n = 45) was 38.5 + 5.4 (out of a maximum score of 50). There was no statistical difference between mean scores assigned by students or faculty (p = 0.2). Thus, an objective, structured score sheet like ours, when used to judge scientific presentations, gave uniform results even when judges hailed from different levels of the medical hierarchy.
Keywords: Conferences, Medical education, Presentations, Bias
Introduction
Academic conferences are organized by medical societies all over the world. Such conferences have become an integral part of a medical professional’s life. It has been estimated that approximately 100,000 medical conferences are organized in a year worldwide [1].
In most conferences, a paper session is conducted where selected abstracts are presented by participating delegates. Oral presentations, especially award sessions, are judged by a panel of faculty judges who score individual presentations on various aspects including content, delivery, and submission.
Students often participate in these conferences and compete for prizes. The subject content in student conferences can be highly variable per session; frequently, no validated score sheet is used, thus, judging such events is a challenge.
Studies have shown that structured score sheets promote uniformity in scoring [2]. We sought to develop and evaluate a structured score sheet to judge scientific presentations in an international medical students’ conference. Our objective was to compare the scores given by student judges and faculty judges for scientific presentations using the standardized score sheet. Our hypothesis was that student judges and faculty judges, when use an objective, structured score sheet, would assign comparable scores.
Methods
This was a prospective, cross-sectional study conducted in April 2013 during MEDICON, the 6th International Medical Students’ Conference, held at University College of Medical Sciences, Delhi.
Designing the Score Sheet
An objective, structured score-sheet (Fig. 1) was designed using existing literature [3, 4]. The score sheet was designed under two broad heads—scientific content and presentation of the content. Both parts of the score sheet were given equal weight, since clear and logical delivery of ideas and scientific results is as important for a successful scientific presentation as the scientific content itself [5].
Fig. 1.
Score sheet used to judge the presentations
Scientific content was assessed through five components which were based on a previous recommended score sheet [3]. These were research question, scientific thought, methods, discussion/conclusion, and preparedness to answer questions. Each component was given equal importance.
To evaluate presentation of content, the remainder of the score was distributed among eight components. These components were adapted and modified from a published score sheet that was not specific for medical presentations [4]. The score given to each component was different and depended on their relative importance as assessed through discussion between the authors.
PowerPoint was given the highest weightage in this part of the score sheet. The five basic parameters to judge the PowerPoint were clarity, text, visual aid, consistency, and simplicity. These five parameters have been described as key for effective use of PowerPoint [6].
Definition of scientific terms for nonspecialized audience was considered important because the audience is often varied and may comprise of healthcare professionals of different specialties and knowledge fields. Grammar was also given significance as poor grammar implies that the speaker is uninformed; the audience is likely to lose faith in the content [7]. Speech also was considered essential with respect to clarity of speech and use of appropriate pauses and pitch. Body language of the speaker conveys trust and confidence [8], and thus we considered posture and eye contact when judging the speaker.
The presenter was given an advantage for sticking to time as most conferences are extremely short of time. Thus, exceeding the time limit resulted in a lower score; however, even a shorter-than-recommended presentation was penalised.
The practice of acknowledging sources of support fulfils a certain social function and has cognitive significance [9]. Thus, when presenters overlooked this aspect, they were allotted a score of zero. Finally, professional attitude, important in a medical professional or student, was also considered when judging the presentation.
After Institutional Review Board approval and informed consent, this score sheet was pilot tested in a scientific session held in 2012, and feedback from faculty judges and student judges was taken; modifications were made based objectively on the score sheets submitted at the end of the sessions and on the basis of qualitative feedback. The subsequently validated score sheet was used during MEDICON 2013.
Seven medical students from semesters 5th to 9th were recruited from among the audience half an hour before the presentation. These student judges, and two faculty judges recruited by the organizers of the conference, were informed about the study and the score sheet was introduced to them.
Five presentations, all made using PowerPoint, were judged using the structured score sheet. After the presentations were adjudicated, the score sheets were retrieved from the judges. The data was stored and handled as per the Declaration of Helsinki and coded to avoid identification of individual judges or participants.
Statistical Analysis
SPSS version 20.0 was used to analyse the collected data; the data was tested for normalcy. As the data was normally distributed, unpaired t test was used to compare scores assigned by students and faculty.
Results
Thirty-five score sheets were retrieved from student judges and 10 from facuty judges. The mean score of all the score sheets (n = 45) was 38.5 ±5.4 (out of a maximum score of 50). Table 1 shows the scores by faculty judges and student judges in two broad areas—scientific content and presentation style.
Table 1.
Scores by faculty and students to each participant
| Participant 1 | Participant 2 | Participant 3 | Participant 4 | Participant 5 | ||
|---|---|---|---|---|---|---|
| Research (max. = 28) | Student judges | 23.5 (±2.9) | 20 (±4.7) | 19.7 (±4.3) | 24.3 (±2.2) | 20.6 (±6) |
| Faculty judges | 25 (±4.2) | 17 (±1.4) | 19.5 (±0.7) | 20.5 (±0.7) | 14 (±2.8) | |
| Presentation (max. = 22) | Student judges | 16.4 (±1.9) | 19.6 (±1.5) | 17.3 (±2.5) | 17.3 (±0.8) | 16 (±3.5) |
| Faculty judges | 15.5 (±2.1) | 18.5 (±0.7) | 17 (±0) | 18.5 (±0.7) | 15 (±0) | |
| Complete score (total = 50) | Student judges | 40 (±3.3) | 39.6 (±5.6) | 37 (±5.9) | 41.5 (±2.8) | 36.6 (±8.0) |
| Faculty judges | 40.5 (±6.4) | 35.5 (±2.1) | 36.5 (±0.7) | 39 (±1.4) | 29 (±2.8) |
There was no statistical difference between mean scores assigned by students or faculty, i.e., p > 0.05 (Table 2). This means that the scoring by the student judge and faculty judge was similar statistically.
Table 2.
Comparison of scores given by students and faculty to the five participants
| Judge | Mean | Std. error | t 2 | p value |
|---|---|---|---|---|
| Student (35 score sheets) | 38.97 | 0.924 | 1.17 | 0.2 |
| Faculty (10 score sheets) | 36.7 | 1.60 |
Discussion
The consistency between scores given by student judges and faculty judges in our study was excellent. We demonstrated that this structured score sheet, with detailed descriptions of each scoring criteria, could serve well as a standardised scoring tool for scientific presentations in medical conferences. Additionally, the score sheet could be used by a judge at any level of medical hierarchy and still yield the same result. This is highly desirable for judging award sessions.
Some limitations of our study include a small sample size; the study was conducted during a single session at a single conference and may not give as consistent results at other medical conferences. Nevertheless, the study has important potential implications. Even though peer review, sometimes blinded, of scientific abstracts for presentation in medical conferences is now usually employed, the ability of a judge to correctly evaluate the merits and demerits of a short abstract without any bias has been questioned [1, 10]. The availability of a standardised score sheet such as ours, to judge presentations, could be a great tool to neutralize “reviewer biases”.
Secondly, a standardised scoring tool would help in more efficient and holistic evaluation of presentations by judges. A structured score sheet might save time and energy expended in arbitrarily scoring a presentation without defined parameters; it would avoid the potential to miss scoring crucial aspects of the scientific presentation.
Finally, a standardised scoring of presentations, such as ours, could inspire faith in the presenters, eliminating any doubts or controversies regarding the judging. The organisers could be assured of delivering a superior academic experience to participants. Awards based on such uniform scoring could promote healthy competition, especially amongst student researchers.
To the best of our knowledge, this is the first study to evaluate the use of a standardised, objective, and structured score sheet for judging scientific presentations. Our study demonstrates the validity of using such a score sheet, promoting uniform results from both students and faculty alike. Further studies are required incorporating a larger sample size and diverse presenters to establish the use of such a scoring tool. We plan to continue the use of our scoring tool at medical conferences in order to refine and standardise it further and to study its other properties.
Conclusion
This objective, structured score sheet, when used to judge scientific presentations, gave uniform results even when judges hailed from different levels of the medical hierarchy.
References
- 1.Ioannidis JPA. Are medical conferences useful? And for whom? JAMA. 2012;307(12):1257–1258. doi: 10.1001/jama.2012.360. [DOI] [PubMed] [Google Scholar]
- 2.Watts CM, Knaus WA. The case for using objective scoring systems to predict intensive care unit outcome. Crit Care Clin. 1994;10(1):73–89. [PubMed] [Google Scholar]
- 3.Junior Science and Humanities Symposia. Judging score sheet. www.jshs.org/forms/Judscr.doc
- 4.Free World U. Oral presentation score sheet. www.freeworldu.org/textpassages/JIOPScoreSheet.pdf
- 5.Bourne PE. Ten simple rules for making good oral presentations. PLoS Comput Biol. 2007;3(4):e77. doi: 10.1371/journal.pcbi.0030077. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hesson G (University of O. Power Pointers: issues to consider when designing a presentation [Internet]. 2012. p. 1–7. http://slc.otago.ac.nz/files/2012/10/power-pointers-2013.pdf
- 7.Bovee CL, Thill JV (2007) Writing business messages. In: Bovee CL, Thill JV (eds). Bus. Commun. Today. 9th edn. 76–101
- 8.How your body language conveys confidence, intelligence and trust … or a lack there of . http://www.sixwise.com/newsletters/05/10/19/how-your-body-language-conveys-confidence-intelligence-and-trust--or-a-lack-there-of.htm
- 9.Tiew WS, Sen BK, National HK. Acknowledgement patterns in research articles: a Bibliometric study based on Journal of Natural Rubber Research 1986-1997. Malays J Libr Inf Sci. 2002;7(1):43–56. [Google Scholar]
- 10.Ross JS, Gross CP, Desai MM, Hong Y, Grant AO, Daniels SR, et al. Effect of blinded peer review on abstract acceptance. JAMA. 2006;295(14):1675–1680. doi: 10.1001/jama.295.14.1675. [DOI] [PubMed] [Google Scholar]