Teaching and Assessing Reflecting Skills among Undergraduate Medical Students Experiencing Research

Abstract

Introduction

Reflection is the integral component of lifelong learning. Hence, there is a need for incorporating opportunities for students in the curriculum, to develop these skills.

Aim

To evaluate the feasibility of incorporating teaching-learning activity on reflection early in the undergraduate medical curriculum using research experience as a context, and, to determine whether the reflective skills of students improve upon training.

Materials and Methods

The study was experimental with test and control groups and was conducted at Melaka Manipal Medical College, India. Senior batch of medical students in the second year of the course, about to complete their research project were considered as the test group and subsequent junior batch which was in middle of the research activity was the control. The test group was provided with a teaching-learning activity on reflection. Following this, students were asked to write reflective summary on experience of doing research. The control group who did not receive any training on reflection were also requested to write reflective summaries. Reflective summaries were graded by two authors independently using a newly developed rubric. Later, the grades were designated with scores. Perspective regarding this teaching-learning activity was collected from the test group. Feasibility was examined during teaching-learning activity and assessment.

Mean reflective summary scores of control and test groups were expressed as mean±standard deviation and compared using independent samples t-test. A p-value of <0.05 was considered as statistically significant. Inter-rater reliability of the rubric was analyzed using Kappa statistics.

Results

The teaching-learning activity lasted for two hours. It took an average of five minutes for researchers to assess each reflective summary. There was a statistically significant (p<0.001) difference in the mean reflective summary scores between control (26.45±9.43) and test (51.66±6.56) groups. Kappa for inter-rater reliability was 0.784 denoting substantial agreement between two raters. Perceptions revealed acceptability of module (median 4, scale 1-5).

Conclusion

Teaching and assessing reflecting skills among students using research experience as context was feasible. This study demonstrated that students acquire better reflecting skills after undergoing training.

Introduction

Today’s medical students who will become tomorrow’s doctors should be trained to confront the various dilemmas and intricacies awaiting their professional lives. In this regard, reflection serves as a model for life-long learning [1–6] by engaging students in thinking what they learnt in a given context and how they will apply that learning in future. Critical reflection is the “process of analyzing, questioning, and reframing an experience in order to make an assessment of it for the purposes of learning (reflective learning) and/or to improve practice (reflective practice)” [7]. The deeper critical reflection moves beyond merely seeking an alternate plan for future similar experiences or identifying reasons for the outcome. It involves questioning underlying conceptual frameworks [8]. The ability to reflect is considered as one of the outcomes of undergraduate and postgraduate medical education [7–10]. Educators who have attempted to teach reflection to health care professional students have echoed the need for incorporating opportunities for students in the curriculum, to develop these skills [11]. Recently, researchers have demonstrated reflection as an attribute for students’ academic performance, where students’ reflection had significant effect on the quality of clinical case solving [12]. Reports suggest that reflective learning may improve professionalism, clinical reasoning, critical thinking and health care [7,13–16].

For the reflection to be effective it has to be guided and appropriate educational approaches are to be designed [2]. Literature suggests a diversity of practical approaches for effective implementation of reflection in undergraduate curriculum [7,8,10,17,18]. However, there is no definite evidence on the best way to teach reflection.

The context used for the reflection exercises should be complex which cannot be resolved by previous problem solving strategies [19]. As engaging in research, in an already ‘packed curriculum’ is a challenging context for undergraduate medical students, reflecting on those experiences may help them to understand their learning process better. For teachers, students’ reflections may serve as a useful route to give feedback to students on their attitude, learning goals and reflective skills. To the best of our knowledge, a systematic attempt for inculcating reflective skills among undergraduate medical students using research as a context has not been reported.

The positive evidence that reflection, like any other skill can be taught [20] and the growing literature proclaiming the irrefutable impact of incorporating reflective skills in medical education [1,21,22] has motivated us to venture on determining the feasibility of imparting this skill to undergraduate medical students in our medical school. In this connection, by setting the Mentored Student Project (MSP) program at our institution as the context for enhancing reflective skills, the present study was conducted with the aim to evaluate the feasibility of incorporating a teaching-learning activity on reflective skills of undergraduate medical students and to determine whether the reflective skills of students improve upon training.

Materials and Methods

Context

At Melaka Manipal Medical College (MMMC), Manipal Campus, Manipal University, India, there are twice a year admission in March and September. The MSP program was blended in the undergraduate medical curriculum (Bachelor of Medicine and Bachelor of Surgery; MBBS) from September 2007, with an intention of developing research skills in students. In MSP, second year students in groups (n=3 to 5) undertook a research project under the guidance of their faculty mentors. In addition, students wrote a scholarly report and presented their work as a poster. Institutional MSP coordination committee guided students with timely orientations pertaining to research methodology, protocol writing, ethical guidelines, project report writing and poster presentation [23,24]. Apart from these, reflective writing on students’ research experience was introduced as a new initiative from the year 2015 onwards. Two MSP coordinators with Foundation for Advancement of Medical Education and Research (FAIMER) fellowship degree, who had reflection as a part of their training program were involved in the development and teaching of reflection to medical students.

Study Design

The study design was experimental and was conducted after obtaining Institutional Ethics Committee permission. Senior batch of medical students in the second year of the course who were at the end of their MSP process were considered as the test group and subsequent junior batch which was in middle of the MSP process was the control. Convenience sampling was done for the control group and students who volunteered to take part in the study were included. Out of 155 students in junior batch, 38 participated voluntarily whereas, all students in the senior batch participated as a part of their research training. The test group (n = 173) was oriented regarding the process of reflection in a lecture class while they undertook their research projects in groups. Students were posed relevant questions during orientation which enabled them to reflect. Following this, students were provided an opportunity to write a reflective summary by themselves, as part of training using project title selection phase of MSP as a context. A few students (n=10) volunteered to share their reflection with the class. Subsequently, constructive feedback on the quality of reflection was given to all students based on this. Students were asked to write reflective summary on experience of doing research in 500 words to measure the impact of training on their reflective skills. The control group (n= 38) who did not receive any training on reflection were also requested to write reflective summaries. Reflective summaries were later analyzed by two authors independently using a newly developed rubric to identify level of reflection. Perspective regarding this teaching-learning activity was collected from the test group using a questionnaire. The items in the questionnaire denoted students’ perceptions regarding the quality of content and instructional method, authenticity in grading and the impact of the activity on their attitude regarding the relevance of reflection. The content validity of the questionnaire was checked prior to its administration, by two medical education experts of the institution who had an experience in validation methods.

Development of Module

The goal of the teaching session was to make students aware of the phases and indicators of critical reflection which will help them to integrate past with present and present with their future research experiences.

The module was designed and implemented by the MSP coordinators of the institution as per the tips given by Aronson L [7]. Students were provided information regarding critical reflection [7] and the benefits of reflection [7,8,17,19]. The different phases of reflection were described using eclectic model of Koole S [10]. This model describes reflection in three phases: A. Reviewing the experience, B. Critical analysis, and C. Reflective outcome. In addition, each phase has been described using indicators, which in turn directs the reflectors to put the model into practice [10]. The module was reviewed by two other faculty members with Foundation for Advancement of Medical Education and Research (FAIMER) fellowship degree for content validity.

Development of Rubric to Assess Reflective Summaries

To grade students’ reflective summaries, a rubric was developed based on the grading criteria proposed by Association for Medical Education in Europe guide adopted from Moon 2004 for reflection [7,8] [Table/Fig-1]. Though we used Koole’s indicators to describe the process of reflection, for grading students’ reflection, grading system developed by Moon was used as we could easily incorporate the indicators in it. Two experts with FAIMER fellowship degree reviewed the rubric independently and the content validity was established. Based on their feedback, the rubric was suitably revised and approved by the researchers of this study.

[Table/Fig-1]:

Reflective summary score sheet showing alignment of teaching (Koole’s elective model of reflection) [10] with assessment.

Eclectic model (phases and indicators) used for teaching reflection	Assessment
	Grades	Poor (1)	Good (2)	Very Good (3)	Excellent (4)
Phase 3: Reflective Outcome 1. The ability to draw conclusions. 2. The ability to describe concrete learning goals and plans for future action.	Grade A: explains how experiencing MSP has changed how he/she experiences research projects in future or explains how he may wish to change how he responds to MSP in future.	Realisation that next experience should be better and something need to be learnt	Explaining how it should be better	Explaining what need to be learnt for better experience (goals)	Explaining plans to reach goals
Phase 2: Critical analysis The ability to answer searching questions and being aware of the frames of references in use	Grade B: Involves judgement – what went well, or less well and why.	Poor description of what went well/less well	Description of what went well and less well	Inadequate description of reasons for what went well and less well	Good description for both
Phase 2: Critical analysis The ability to ask searching questions.	Grade C: Describing MSP – recognising how it affects his feelings, attitudes and beliefs and/or questioning what has been learnt and comparing it to previous experience.	Poor description of what has been learnt	Good description of what has been learnt	Poor comparison of present learning with past experience	Adequately compared the present learning with past experience
Phase 1: Reviewing the experience	Grade D: Describing MSP – recognising that something is important but not explaining why.
1. The ability to describe an event/situation adequately.	Grade E: Describing MSP – repeating the details without offering any interpretation.
2. The ability to identify essential elements and to describe own thoughts and feelings	Grade F: Describing MSP – poor description of an event.

MSP: Mentored Student Project

Each reflective summary was graded by first and the second authors of this study as A to F based on the level of reflection [Table/Fig-1]. The grades were designated with scores as follows: A= 60; B = 50; C = 40; D = 30; E = 20; F=10. Construct validity was tested by comparing the scores of students who are anticipated to have less skills and those with more skills. If the rubric measures students’ reflective skills accurately then there should be differences in the level of reflection between students of test and control groups. Inter-rater reliability of the rubric was established by comparing the scores given by two examiners for each reflective summary.

Statistical Analysis

Mean reflective summary scores of control and test groups were expressed as mean±standard deviation. Inter-rater reliability was analyzed using Kappa statistics using SPSS, version 16, statistical analysis programme (SPSS, Inc., Chicago, IL). Students’ perception was first expressed as median and interquartile range. Frequency analysis of responses was done and expressed as a cumulative percentage of ‘agree’ and ‘strongly agree’ responses.

Results

Quality of reflective summaries, construct validity and reliability of rubric: Out of 173 students in the test group, 55 (32%) had A grade which depicted vision and plans for future research. Ninety two (53%) had B grade which showed judgement on present research experience. Twenty six students (15%) had C grade which described lessons learned during the research experience. Whereas, in the control group (n=38), five students had F grade, while 11 students each scored C, D and E grades. There was a statistically significant (p<0.001) difference in the mean reflective summary scores between control (26.45±9.43) and test (51.66±6.56) groups which reflects the construct validity of the rubric. Inter-rater reliability for the raters was found to be Kappa = 0.78 (p <0.001), 95% confidence interval (0.84, 0.72) denoting substantial agreement among two authors. Retrospective analysis with the study sample size revealed the power for 5% significance as 100%.

Feasibility data and students perceptions: The initial orientation on reflection for the test group lasted for an hour and the practice session followed by feedback for one more hour. Orientation and practice session was conducted as a large group teaching-learning session encouraging student interaction by posing relevant questions. Reflective summaries written by students during practice sessions were graded by their peers, in 10 minutes using the newly developed rubric. All students in the test group wrote a reflective summary and submitted it before the deadline. It took around five minute for the authors to grade each reflective summary. While grading, only the level of reflection was considered and not the quality of English and creative writing skills. Perceptions revealed acceptability of module (median 4, scale 1-5) [Table/Fig-2].

[Table/Fig-2]:

Students’ perception (median and interquartile range (IQR)) regarding the module on reflecting skills.

Item No.	Items	Median and IQR	Cumulative percentage of agree and strongly agree responses
1.	Allocated time for teaching-learning session was adequate	4 (2,4)	75
2.	Module needs to be revised	2 (2,3)	25
3.	Task asked to perform was consistent with teaching objectives	4 (3,5)	70
4.	Timely feedback was given	4 (3,5)	65
5.	Feedback highlighted my areas of weakness	4 (4,5)	85
6.	Explanation given during teaching- learning session was clear	4 (4,5)	80
7.	Reflective summary score reflected my performance	4 (4,4)	75
8.	I have not been given deserved score	3 (2,3)	26
9.	This skill should be taught to all medical students	4 (3,4)	65
10.	I am confident in writing reflective summaries in the future	3 (2,4)	55
11.	This skill is relevant to my future career	4 (4,5)	75

Discussion

Inculcating skills and attitude for life-long learning is an imperative requirement of Malaysian Medical Council to which MMMC is accredited. Reflective practices can boost learners’ self-directed learning skills, which in turn is an essential attribute of life-long learning [11]. This study showed that it is feasible to use research as a context to help students to experience the process of reflection and thereby gain reflective skills in order to enhance their self-directed learning skills. The use of reflective summaries to facilitate reflective practice is well documented [25–28]. Literature reports the benefits of written reflection as an instructional method, as it guides students’ critical thinking of their own learning process [7]. In the present study, the control group could not engage in a systematic analysis of how and what they learnt from the research experience. Whereas, the test group’s improved reflective capacity was reflected by the streamlined analysis of their research experience. More of such exercises may inculcate high quality reflecting skills in students.

The module was highly commended by experts in medical education with respect to its relevant content and the superior quality of the rubric which reflected the intended learning outcomes. This established its face and content validity. While establishing construct validity, students with minimal reflective skills would be anticipated to score less than those with some training. In our study the control group was not exposed to any type of training in reflection and probably led to their low scores compared to the test group who had undergone training and had high scores. This implies that students acquire better reflecting skills after undergoing training and the criteria for construct validity therefore seems to be fulfilled.

Even though studies report the use of rubrics in patient care [29], the best of our knowledge, there are no rubrics reported till date, which can measure reflective writing regarding research experience. Hence, there was need for a rubric which could be used for assessing reflective writing on research experience. Though the content of the reflective summary was context specific, the extent to which a student goes into deeper reflection depends on their reflecting ability [8,12]. Hence, the grades obtained, manifested students’ generic reflective skills. The grading system differentiated summaries with mere habitual actions, from critical reflection, and helped us to grade the summaries objectively based on quality. Our study subjects did not have prior experience with reflective skills pertaining to their exposure to neither patient nor research. As a curricular component, students were not exposed to the process of reflection. Hence we assume that students’ grade, reflected the reflective skills they gained through the training. The high ICC value of scores given by two assessors independently, demonstrated that the rubric used is a reliable instrument. The grade also reflected the core processes used by students for the reflection.

It was encouraging to note that all students had submitted their assignment on reflective summary. The finding that majority (53%) of students had grade B, and 15% having grade C [Table/Fig-1] affirms that, reflective writing as a first time experience was a creditable skill which they accomplished. A similar study done on physiotherapy students during clinical internship with patient care as context had shown similar results with majority of students achieving descriptive- analytical and analytical levels of reflection which is comparable to A and B grades of reflection in our study [11]. In a study done on first year medical students who had not received any form of systematic training on reflection, demonstrated low level of reflection which is comparable to the reflection level of control group in our study [30]. Despite the notification about prizes for 10 best reflective summaries, intended to motivate them, only 32 % of students had A grade. This shows that factors like interest, fluency in English language, writing skills etc., might have influenced students’ performance, even though they were well informed that these factors will not be taken into consideration while grading. In addition, only 55% of students opined that they are confident in writing reflective summaries in future [Table/Fig-2]. This module was the first of its kind and more practice sessions and repeated exercises along with constructive feedback from mentors may inculcate deep reflecting skills leading to more fruitful transformative learning in students.

We found it challenging to fit the module on reflection into already packed teaching schedule. In addition, students were not serious about the relevance of the module as their grade in reflective writing was not considered during summative assessment. To tackle this issue, we are in the process of considering students’ grade in their reflective summaries as a mandatory requirement of completion of MSP. There was a gross disparity in the sample size between control (n=38) and test group (n=173). With the S.D of 9.5, power of 80% and the minimum difference in the reflective score between test and control as 15, the estimated sample size is found to be 7 in control group and 21 in test group. Hence, we are confident that sample size was appropriate to make the study conclusion. A follow up study could not be conducted to investigate whether repeated exercises inculcate deep reflection in more number of students because our students undertake the research project in second year (one year duration) following which they enter the clinical phase of six months. They complete the succeeding two and a half years in Malaysia. So it becomes a practical difficulty for us to conduct a follow up study.

Conclusion

This study describes a module on reflection that was acceptable to students. In future, the rubric developed in this study for a summative purpose could be used formatively to help students in developing deeper level of reflection. Our model of teaching reflection is feasible in a curriculum where undergraduate students are exposed to research. This model can also be generalized to other context with suitable modifications.

Financial or Other Competing Interests

None.