Use of Multi-Response Format Test in the Assessment of Medical Students’ Critical Thinking Ability

Abstract

Introduction

To evaluate students critical thinking skills effectively, change in assessment practices is must. The assessment of a student’s ability to think critically is a constant challenge, and yet there is considerable debate on the best assessment method. There is evidence that the intrinsic nature of open and closed-ended response questions is to measure separate cognitive abilities.

Aim

To assess critical thinking ability of medical students by using multi-response format of assessment.

Materials and Methods

A cross-sectional study was conducted on a group of 159 undergraduate third-year medical students. All the participants completed the California Critical Thinking Skills Test (CCTST) consisting of 34 multiple-choice questions to measure general critical thinking skills and a researcher-developed test that combines open and closed-ended questions. A researcher-developed 48-question exam, consisting of 8 short-answers and 5 essay questions, 19 Multiple-Choice Questions (MCQ), and 16 True-False (TF) questions, was used to measure critical thinking skills. Correlation analyses were performed using Pearson’s coefficient to explore the association between the total scores of tests and subtests.

Results

One hundred and fifty-nine students participated in this study. The sample comprised 81 females (51%) and 78 males (49%) with an age range of 20±2.8 years (mean 21.2 years). The response rate was 64.1%. A significant positive correlation was found between types of questions and critical thinking scores, of which the correlations of MCQ (r=0.82) and essay questions (r=0.77) were strongest. The significant positive correlations between multi-response format test and CCTST’s subscales were seen in analysis, evaluation, inference and inductive reasoning. Unlike CCTST subscales, multi-response format test have weak correlation with CCTST total score (r=0.45, p=0.06).

Conclusion

This study highlights the importance of considering multi-response format test in the assessment of critical thinking abilities of medical students by using both open and closed-ended response questions.

Introduction

In the 21^st century, the main goal of education is not the acquisition of more information, but it is rather to strengthen students’ critical thinking skills which enable them to analyse and then apply the existing information [1]. The importance of this issue increases when promotion of the ability to think critically has been identified as a considerable factor in the professional success of medical students [2]. Many university faculty members believe that critical thinking should be the main purpose of college education [3], and several organisations have called for critical thinking development in medical education [4]. As listed in reports released by medical universities such as Aberdeen, Dundee and McGill, critical thinking has been viewed as a key competency to be cultivated and assessed in medical students [5,6]. In addition to Western universities, recent reforms in medical undergraduate curriculum in Iran have as well advocated the improvement of critical thinking skills in medical students [7]. Despite the vested interest developed among medical colleges in improving critical thinking as a core competency, evidences indicate that assessing critical thinking has entailed many difficulties and challenges [8].

The difficulties associated with critical thinking assessment are diverse. One of the obstacles is lack of consensus regarding an obvious and operational description of critical thinking. Notwithstanding this most researchers concur on the importance of critical thinking competency, and its being nurtured and honed among medical students [9-11]. Then, with varied definitions of critical thinking presented so far, its assessment still remains indefinite [12-14]. In other words, there has not been a consensus on proper assessment that objectively and accurately computes medical students’ critical thinking. Another stumbling block is that critical thinking is considered as a complex and multivariate concept which contains both cognitive and dispositional components [15]. Consequently, a test should be designed in a way that can more validly measure it.

The assessment of a student’s ability to think critically is a constant challenge, and yet there is considerable debate on the best assessment methods [16,17]. There is evidence that closed and open-ended response questions measure separate cognitive capabilities, with their respective constraints looming [18,19]. Cox M et al., mentioned that no single method of assessment is better than other and possibly a valid test needs a combination of different assessment methods [20]. Another study recommends that multiple test measures be used in order to assess changes in students’ critical thinking skills [21].

The aim of the present study was to contribute further to methodological gaps in the assessment of critical thinking as a major outcome of medical education by utilising multi-response format of assessment. This study was of paramount importance as it provided a multi-response format test for the assessment of critical thinking using surrogate measures which had never been used before at any medical school.

Materials and Methods

This cross-sectional study was conducted at Tehran University of Medical Sciences (TUMS), School of Medicine, one of Iran’s largest and oldest medical schools, in 2013-2014. The undergraduate medical curriculum at TUMS is divided into three phases; including two and half years of studies in basic sciences, one year of pathophysiology, and three and half years of clinical phase. Since 2006, TUMS commenced to develop and implement a newly revised curriculum for delivering undergraduate medical education. One main feature of the revised curriculum is to pay more attention to the integration of critical thinking as cross-cutting theme for training and assessing medical students [22].

In this study, data were collected from a group of medical students involved in the completion of California Critical Thinking Skills test (CCTST) [23], to measure general critical thinking ability and a researcher-developed test which consists of both open and closed-ended questions. The initial convenience sample was 159 third-year students enrolled in medical undergraduate curriculum at TUMS. Third-year medical students were chosen to participate in this study because they had completed the mandatory critical thinking course. Out of these 159 students, 102 completed both the tests. The reason of having incomplete data was voluntary participation and exclusion of participants who did not complete all tests.

Instruments

Multi-response format test: A researcher-developed 48-question exam (The Persian version), consisting of short answers and essay questions, Multiple-Choice Questions (MCQ), and True-False (TF) questions, was used to measure critical thinking skills of third-year medical students. While the content of the questions was focused on medical-related subject matter, these were developed to measure a range of critical thinking abilities (assumptions, analysis, inference, evaluation, cognitive biases etc.,). Once complete, the research team reviewed the questions and then sent them to the medical education specialists and critical thinking experts for review of content. Additionally, internal consistency was examined using Cronbach’s alpha (0.69) computed for the total score. We administered the test assessments to medical students. Respondents, with an aim of determining evidence of critical thinking ability in relation to a medical context, using multi-response format test, answered the test consisting of open-ended questions (8 short answer questions and 5 essays) and closed-ended questions (19 MCQ which was followed by 4 or 5 options and 16 TF questions). In a bid to motivate students to participate, a battery of questions was designed which were related to the students’ background, promising them to provide feedback on the results in due course.

California critical thinking skills test [24]: The CCTST questionnaire (The Persian version of CCTST-form B) was used to evaluate general critical thinking skills of medical students. The CCTST contains 34 multiple-choice questions (analysis, inference, evaluation, deductive reasoning, and inductive reasoning) with a correct answer (0-1 score) that targets those general critical thinking skills, considered to be essential elements in higher education [Table/Fig-1]. Reliability and validity of the CCTST assessment has been reported in previous publications [23]. Each correct response was assigned one score and total score on CCTST ranged from a minimum of 0 to a maximum of 34.

[Table/Fig-1]:

Possible ranges of instrument total scores and subscales.

Instrument and Subscale	Range of Possible Scores
CCTST total score	0-34
Multi-response format Test total score	0-58
Essay questions (each question 0-3 score)	0-15
Short answer questions	0-8
Multiple choice questions	0-19
True-false questions	0-16

Procedure

The multi-response format test was completed in about 75 minutes. Medical and health problems were chosen due to their multifaceted nature, and the fact that they were related to the students’ background. Students took the test in an exam hall, and each put forth the effort to write down his/her responses. There were tight invigilation rules, and communication and collusion between candidates (by copying, whispering or any kind of signal, exchange of paper or objects) was not allowed. To establish the degree of content independent critical thinking ability, CCTST (Form B) was conducted two weeks later. It took students about 45 minutes to complete the instrument.

Ethical Consideration

This paper is a part of a thesis submitted for the degree of PhD in Medical education. Research Ethical Committee of Tehran University of Medical Sciences approved the study. Students received a full explanation prior to participation which was voluntary and anonymity was maintained and guaranteed.

Statistical Analysis

For all statistical analyses SPSS 22.0 was used. Descriptive statistics (means, standard deviations, and frequencies) were used to describe the sample. Reliability of the researcher-developed test was examined using Cronbach’s alpha. Pearson correlation co-efficient was employed to explore the association between the total scores of tests and subtests.

Results

One hundred and fifty nine students were included in the study which comprised 81 females (51%) and 78 males (49%) with an age range of 20±2.8 years (mean 21.2 years). One hundred two medical students completed both the CCTST and multi-response format test. Data from matched pairs were analysed. The response rate was 64.1%. The possible range for instrument scores is shown in [Table/Fig-1].

Pearson correlation was computed to explore the association between the CCTST and multi-response format test subscales. [Table/Fig-2,3] present the correlation between multi-response format test and CCTST scores and subtests scores. While there were positive correlation with all multi-response format test subtests, the relationship of MCQ (r=0.82) and essay questions (r=0.77) was strongest. CCTST total score correlated best with total scores for MCQ questions (r=0.25) [Table/Fig-2].

[Table/Fig-2]:

Correlation of total scores on multi-response format test and CCTST with different format questions was computed. a and b indicate statistical significance at confidence levels of p<0.05 and p<0.01 respectively

Instrument and Subscale	Multi-Response Format Test Total Score	CCTST Test Total Score
Essay questions	0.77b	0.18
MCQ	0.82b	0.25a
Short answer question	0.42b	0.05
TF	0.30a	0.06

California Critical Thinking Test (CCTST), Multiple Choice Question (MCQ), True-False (TF)

[Table/Fig-3]:

Correlation of scores on subtests’ CCTST with multi-response format test was computed. a and b indicate statistical significance at confidence levels of p<0.05 and p<0.01 respectively

Instrument and Subscale	Multi-Response Format Test Total Score	CCTST Test Total Score
Analysis	0.25a	0.56b
Evaluation	0.31a	0.64b
Inference	0.24a	0.73b
Deductive Reasoning	0.17	0.70b
Inductive Reasoning	0.25a	0.74b

California Critical Thinking Test (CCTST)

The significant positive correlations between multi-response format test and CCTST’s subscales were seen in analysis, evaluation, inference and inductive reasoning. Significant correlations were found between CCTST total score and scores for nearly all its subscales. The correlations between CCTST score and its subscales scores were between 0.56 and 0.74 [Table/Fig-3].

In general, descriptive analysis of CCTST and multi-response format test total scores are presented in [Table/Fig-4]. There was a weak positive correlation between CCTST with multi-response format test total scores, which was not significant (r=0.22, p=0.06).

[Table/Fig-4]:

Medical students’ critical thinking total scores

Instrument and Subscale	Mean	SD	Min	Max
Multi-Response Format Test	35.21	5.7	18.5	49
CCTST Test Total Score	18.25	6.05	11	27

Discussion

The combination of two-response format (open and closed-ended questions) into one test is viewed as the current trend in the assessment of critical thinking. Ku KY has mentioned that any measurement of critical thinking that utilises a single-response format is neither sufficient in reflecting students’ true critical thinking ability, nor compatible with the conceptualisation of critical thinking [25]. In this study, closed and open-ended questions were utilised to indicate a better understanding of medical students’ critical thinking abilities while facing problems in the field of medicine.

The results showed changes in the assessment of medical students’ critical thinking by multi-response format test consisting of MCQ, essay, TF and short-answer questions. Our results showed that students’ scores of MCQ and essay questions significantly correlated with total scores of medical students’ critical thinking abilities. This does confirm that well-constructed MCQ can also assess higher levels of cognitive skills of medical students [26]. These findings that MCQ and essay can be used to assess critical thinking were similar to those found in previous studies. The Halpern Critical Thinking Assessment Using Everyday Situations (HCTAES) is a general test that incorporates both multiple-choice and essay questions into a single test [19]. Stein and Haynes developed a measurement tool including standardised multiple-choice tests, essay tests, and faculty-developed rubrics for evaluating student work [27].

In current the study, the mean CCTST score for medical students was 18.25. Athari Z et al., and Haghani F et al., have reported marginally lower means CCTST in medical students [28,29]. Although, the mean score gained in the area of critical thinking proved not to be high enough, by the medical students in our study, it was higher than the national scores as compared to the similar international studies. This can reflect the effects of critical thinking training program in renewal of medical curriculum.

In addition, a positive correlation was observed between total scores of CCTST and multi-response format test during the study, though being quite small. These findings indicate that performance on the multi-response format test in medicine is little related to general measures of critical thinking skills. In our study, the CCTST was used to measure critical thinking as a general competency [30], while the multi-response format test was used to measure critical thinking in medicine discipline. This can reflect that medical students’ multi-response test score is not explained by these other critical thinking tests, which rely on multiple-choice questions [27].

Performance on the multi-response format test generally correlates with scores on analysis, evaluation, inference, and inductive reasoning of CCSTS. However, the relative low correlation between the multi-response format test and subtests’ CCTST scores might suggest that the two tests assess different components of critical thinking via different test contents.

Limitation

Several limitations of this study prevent cautious transfer of the findings to other contexts. The time needed to complete the two tests was long. This may have caused the unwillingness of some medical students to answer the tests properly. Future researches are necessary to validate the findings of this work.

Conclusion

This study highlights the importance of considering the right format of response in critical thinking assessment. An important feature of the multi-response format test is that it relies primarily on different types of methods to assess critical thinking, unlike many standardised tests that rely on multiple-choice questions. It consists of MCQ, essay, short answer and true-false questions for evaluating medical students’ critical thinking. The results showed that medical students response to essay and well-constructed MCQ questions reveal their critical thinking when encountering medical problems.

Financial or Other Competing Interests

None.