Psychometric characteristics of Clinical Reasoning Problems (CRPs) and its correlation with routine multiple choice question (MCQ) in Cardiology department

Abstract

Introduction:

Clinical reasoning is one of the most important skills in the process of training a medical student to become an efficient physician. Assessment of the reasoning skills in a medical school program is important to direct students’ learning. One of the tests for measuring the clinical reasoning ability is Clinical Reasoning Problems (CRPs). The major aim of this study is to measure psychometric qualities of CRPs and define correlation between this test and routine MCQ in cardiology department of Shiraz medical school.

Methods:

This study was a descriptive study conducted on total cardiology residents of Shiraz Medical School. The study population consists of 40 residents in 2014. The routine CRPs and the MCQ tests was designed based on similar objectives and were carried out simultaneously. Reliability, item difficulty, item discrimination, and correlation between each item and the total score of CRPs were all measured by Excel and SPSS software for checking psycometeric CRPs test. Furthermore, we calculated the correlation between CRPs test and MCQ test. The mean differences of CRPs test score between residents’ academic year [second, third and fourth year] were also evaluated by Analysis of variances test (One Way ANOVA) using SPSS software (version 20)(α=0.05).

Results:

The mean and standard deviation of score in CRPs was 10.19 ±3.39 out of 20; in MCQ, it was 13.15±3.81 out of 20. Item difficulty was in the range of 0.27-0.72; item discrimination was 0.30-0.75 with question No.3 being the exception (that was 0.24). The correlation between each item and the total score of CRP was 0.26-0.87; the correlation between CRPs test and MCQ test was 0.68 (p<0.001). The reliability of the CRPs was 0.72 as calculated by using Cronbach's alpha. The mean score of CRPs was different among residents based on their academic year and this difference was statistically significant (p<0.001).

Conclusion:

The results of this present investigation revealed that CRPs could be reliable test for measuring clinical reasoning in residents. It can be included in cardiology residency assessment programs.

Introduction

Clinical reasoning is defined as the skill to categorize through features presented by a patient and precisely assign a diagnosis, and development of a proper treatment plan for the patient (1). Clinical reasoning is an essential skill for the practice of medicine, and is recognized to lead patient safety (2). Clinical reasoning skills will be learned by gathering a bulk of knowledge and by observing skillful and expert clinicians. Nowadays it has been assumed that training in this regard ought to be considered in the curriculum of medical schools (3).

The first stage of clinical reasoning is collecting accurate and reliable information from the patient. The patient is a source of information to be well looked into by a physician who tries to gather relevant data for purposes of making diagnostic hypotheses through delving into his/her knowledge and experience (4).

After nearly three decades of attempts to map the nature of clinical reasoning, we have now found a better perception of the factors associated with clinical reasoning; still, there is considerable controversy among researchers about some of the key features of clinical reasoning (5).

Assessment of the reasoning skills in a medical school program is important to direct students’ learning about reasoning. In the setting of medical education, some new investigations has investigated clinical reasoning development through the practice of Clinical Reasoning Exercises (CREs), Clinical Reasoning Tests (CRT) and Clinical Reasoning Problems (CRPs) (6).

CRPs were for the first time, presented by Michele Grooves (7). In Iran it is used to measure clinical reasoning skills for talented undergraduate medical students participating in National Medical Science Olympiads (8-9).

Investigation about errors that may happen in the clinical reasoning development requires, first, a method that is able to separate and assess different clinical reasoning steps. The clinical CRPs are a method being precisely designed to measure these three steps in clinical reasoning: identification, interpretation of related information and hypothesis generation. Each problem contains a small clinical scenario including patient present problem, history and physical examination. Based on the data provided, students are asked to suggest the two most likely possible diagnoses. They are then asked to incline the features of the case that they considered important in expressing their diagnoses, to specify whether these features existed positively or negatively predictive. Completed problems are scored by means of an outline designed to reveal clinical reasoning ability more than diagnostic accuracy (10-11).

Considering the importance of assessing clinical reasoning using CRPs, and the need for residents to become familiar with it and since it is a new approach in assessment, in this research we decided to compare the results of CRPs with routine MCQ tests in the cardiovascular department. The reliability of the CRPs, item difficulty, item discrimination and the correlation between CRPs and MCQ test was measured in this study. To our knowledge this is the first study on CRPs in cardiology residents.

Methods

This study was a descriptive study conducted on total cardiovascular residents of Shiraz medical school that were 40 residents (year 2014). First, they answered 15 questions of MCQ and then completed 10 questions of CRPs; in fact, the CRPs questions were compatible with routine multiple-choice questions and to design CRPs questions they used the board questions of the same year. For holding the test, initially necessary arrangements were made with cardiac ward manager. Then, the residents were informed about objectives and about how to answer the questions.

Principles of CRPs Questions

A CRP contains a clinical scenario including a short case presentation, history and physical examination. In this study each CRP was designed and checked for accuracy and realism by an expert cardiologist that had also Master of Science degree in medical education. Based on the information given to residents, they are asked to choose the two most likely diagnoses and to list the features of the case that they regarded important in regard to their diagnoses, and then mention whether these features were positively or negatively related to diagnosis. Each scenario was designed to provide more than one possible diagnosis. An example of a CRP is provided in Appendix 1. In the study each item has 4 parts. Part 1 & 3 were relevant to diseases diagnoses and part 2 & 4 were relevant to choose just 5 clinical finding.

10 CRPs was designed based on the above mentioned criteria. To certify content validity, each CRP was revised by an expert panel consists of 10 cardiologists.

Scoring CRPs

The CRP tests are graded based on binary method in which the correct answer has one score and the wrong one has zero. Then by using summative method, the total score (sum of the scores of questions) will be obtained. In this study, each test of CRP has 30 scores; for every correct diagnosis [(10 scores for each correct diagnoses) (20 scores for part 1 & 3)]. Students receive 5 scores and they are allowed to choose just 5 clinical findings among them by using positive or negative mark; they will receive one score for correct clinical finding selection and zero for wrong one. If they choose more than 5 clinical findings, they will lose one point for every extra selection 7,8,10,11 (10 scores for part 2 & 4). Finally the scoring of CRPs and MCQ tests was calculated out of 20 (a sample of CRP is in the Table 1).

Table 1.

CRPs Test Example

A 45 year old female with fever, fatigue, pain, muscle weakness and weight loss has been admitted, pain and discoloration occurs when the patient put her fingers in cold water. In recent months, the patient hospitalized twice with acute pulmonary edema and has a history of TIA. She complains of Orthopnea and PND, especially when sleeping on her left side. In Trans thoracic echocardiography, she has high pulmonary pressure.

A. Which of the following diagnoses match the patient problems? (Choose only one)

1. Atrial myxoma

2. Scleroderma

3. PAN

4. Factor V laiden Syndrome

B. Among the below clinical finding choose 5 clinical findings from the list and mark it (+) if its in favore of your diagnosis and mark it (-) if it is against it. (Note that do not choose more than 5 otherwise you will receive a negative point)

1. Fever and fatigue

2. Muscular pain

3. Hands' color changing in cold water

4. Acute pulmonary edema

5. Weight loss

6. PND on the left hand

7. Transient edema

8. Female gender

9. High lung pressure

10. Positive TIA history

C. Which of the following diagnoses match the patient problems? (Choose only one)

1. Atrial myxoma

2. Scleroderma

3. PAN

4. Factor V laiden Syndrome

D. Among the below clinical finding choose 5 clinical findings from the list and mark it (+) if its in favore of your diagnosis and mark it (-) if it is against it. (Note that do not choose more than 5 otherwise you will receive a negative point)

1. Fever and fatigue

2. Muscular pain

3. Hands' color changing in cold water

4. Acute pulmonary edema

5. Weight loss

6. PND on the left hand

7. Transient edema

8. Female gender

9. High lung pressure

10. Positive TIA history

Examinees

The examinees in this study were 40 residents of cardiovascular department of Shiraz Medical School, consisting of 12 second year residents, 8 third year residents and 20 senior residents.

Analysis

Descriptive statistics (mean, standard deviation, (Pearson correlation coefficient and One Way ANOVA were used to determine the CRP mean differences between 3 residents group. using SPSS software. Item difficulty and item discrimination for CRP tests were measured by using Whitney and Sabers’ method by Excel software (12).

Results

Total cardiology residents participated in this study including 18 female and 22 male residents and they answered 10 questions of CRP exam within 60 minutes. The whole score for the exam was 300. The total score was calculated based on the 4 parts scores. Each part has pointed score: [first diagnosis (10 points), choosing five positive and negative data based on the first diagnosis (5 points), second diagnosis (10 points), choosing five positive and negative data based on the second diagnosis (5 points)]. So the whole score for each item was 30. Finally the scoring of CRPs and MCQ tests was calculated from 20. Descriptive statistics for CRPs and MCQ was shown in Table 2.

Table 2.

Descriptive table of CRP test, MCQ test

	N	Max	Min	Mean±SD
CRP test	40	15.73	3.86	10.19±3.39
MCQ test	40	18.66	4	13.15±3.81

The reliability of both tests (CRP, MCQ) was measured by Cronbach's alpha, which was 0.72 for CRP test and 0.71 for MCQ test.

Discrimination index determines the power of the question in identifying between the strong group and the weak group of students, which means that the question as to how much can identify a strong group from the weak group. Its values vary between +1 and -1 and closer to +1 is better.

The Table 3 shows discrimination index measured based on the method of Whitney and Sabers (12) (calculated by Excel software). All of the CRPs attained positive discrimination index. It must be noticed that the questions that obtain negative coefficient should be deleted or revised properly. Zero coefficients display that the question could not differentiate the high level and low level participants (Table 3).

Table 3.

Item Discrimination of CRP test questions

Questions	1	2	3	4	5	6	7	8	9	10
Item discrimination	0.71	0.32	0.24	0.39	0.38	0.37	0.35	0.30	0.75	0.46

Difficulty index also was calculated for CRPs by using the method of Whitney and Sabers (12). The amount of this coefficient was acceptable for all questions (0.3-0.7) except question 5. When the difficulty index is near to 1, the question is considered easier and when it is far from 1 it will be considered harder. Maximum Difficulty Index was 0.72 for question 7 and the minimum was 0.27 for question 4 (Table 4).

Table 4.

Difficulty index of each CRP test

Questions	1	2	3	4	5	6	7	8	9	10
Item discrimination	0.45	0.36	0.69	0.63	0.27	0.57	0.72	0.48	0.37	0.39

In this study, the correlations between each item score and the total examination score were calculated using Pearson correlation coefficient. Three items (3, 4 and 7) has no correlation (p>0.05) (Table 5).

Table 5.

Item total correlation for CRPs

Questions	1	2	3	4	5	6	7	8	9	10
Correlation Coefficient	0.87	0.53	0.26	0.27	0.52	0.60	0.26	0.46	0.84	0.63
P	0.01	0.01	0.10	0.08	0.001	0.01	0.10	0.002	0.01	0.01

The item discrimination, item difficulty and correlation coefficient showed three CRPs (3, 4 & 7) are necessary to modify.

Pearson correlation coefficient between CRPs and MCQ was measured. Results showed that there is moderate correlation between CRPs and MCQ test [Pearson correlation=0.68 (p<0.001)].

In this study, we measured mean and standard deviation of CRP test for each academic year separately, as shown in Table 6.

Table 6.

Comparison of CRP test results for each academic year separately

CRP test score	N	Mean±SD	P
Residents
Year 2	12	7.40±2.59	0.001
Year 3	8	7.49±1.71
Year 4	20	13.02±1.36

One Way ANOVA & Tukey tests showed mean difference between second and third year were no significant but they were different with fourth year (p<0.001).

Discussion

One of the main parts of clinical education in which problem solving has an undeniable role is clinical reasoning. Assessment should not only assess the students’ knowledge but also should measure higher levels of thinking such as creative problem solving and reasoning. By clinical reasoning tests, the students' ability in solving problems, finding right diagnosis and treating clinically ill patients are evaluated. Promotion, improvement and progress in the clinical reasoning will help the students to reduce medical errors, thereby reducing the mortality rate, reducing the costs and increasing patient and staff’s satisfaction (13).

The purpose of the present study was to investigate the use of CRPs for measuring clinical reasoning and decision-making skills in cardiology residents. Our results showed that the reliability of CRPs was acceptable (r=0.72). This reliability was higher than other studies on reliability of CRPs. An earlier reliability study that was done by Groves et al, showed values ranged from 0.61 to 0.83 (7). In another study by groves et al in 2013, reliability of CRPs was reported 0.61 to 0.68 (14). Our previous study showed reliability of CRPs was 0.71 (8).

Item discrimination of CRP was calculated between 0.3 to 0.75.It seems that these questions could differentiate between good students and weak ones well, with the exception of question number 3 that was 0.24 which is needed to be reconsidered.

Correlations between the total examination score and score for each item were calculated between 0.27-0.87 with an acceptable significance level. This proved that the questions can determine good and weak students (with the exception of Question 3,4 and 7).

Item difficulty was obtained between 0.27-0.72; in our previous study the item difficulty of CRPs was reported between 0.4 and 0.69 (8).

Pearson correlation coefficients were calculated 0.68 between the two tests of CRP and MCQ, which is significantly meaningful; this result indicates a moderate significant relationship between CRP and MCQ test which is more than the result of our pervious study (8).

In the present study, mean, standard deviation and variance of CRP test for each academic year were measured separately. Results show that fourth year residents obtained more scores and this difference is statistically significant. The results suggest that clinical reasoning will be increased with increasing clinical experience during training. Other studies in the field of clinical reasoning showed similar results (4).

These positive results, together with content validation of the tests, heightened the validity of the CRPs examination. Acceptable correlation between each of the CRPs and total grade was an indicator for construct validity of the whole examination.

Research in the field of clinical reasoning is one of the research priorities in Eastern Mediterranean Region and Iran (15). Results of some of the other studies on clinical reasoning test in Iran showed acceptable reliability of other clinical reasoning tests (16-18).

One of the strengths of the present study was that CRPs and MCQ test was done on residents simultaneously. Strength was choosing adequate number of expert panels. The major limitation of the study is that this study was a cross sectional study and it was not possible to have a cohort of residents and follow them based on their scores.

Conclusions

The purpose of teaching clinical reasoning is to train individuals who use clinical reasoning and higher thinking skills for proper diagnosis and choosing perfect treatment approaches. The results of this study and its relationship with other studies suggest that CRPs could be a good alternative for measuring clinical reasoning skills. To include the clinical reasoning tests in routine assessments of medical schools, appropriate educational programs should be designed. Using teaching and assessing methods such as clinical reasoning tests for medical students and paramedical fields can be a good way to measure higher skills and abilities.