Evaluation of Clinical Reasoning in Basic Emergencies Using a Script Concordance Test

Abstract

Objectives

To develop and assess the reliability of a script concordance test (SCT) to evaluate pharmacy students' clinical reasoning when facing basic emergency situations.

Design

A first aid course was designed that consisted of 8 weekly instructional sessions (4 on internal medicine, including life threatening situations; 2 on pediatrics; and 2 on trauma) in which the instructor presented case studies in a small-group format. In the first and final sessions of the course, a practice SCT was administered to familiarize students with the test format.

Assessment

A 66-question SCT examination was administered to the 68 third-year pharmacy students enrolled in the first aid course. The students' mean score was 68.5% ± 9.8% and panel members' mean score was 86.5% ± 4.2%. Twenty students were selected randomly to complete a course survey and 85% indicated they were satisfied with using the SCT.

Conclusions

A first aid SCT was found to be both a practical and reliable testing instrument for assessing the clinical reasoning of pharmacy students in basic emergency situations.

INTRODUCTION

In community practice, the pharmacy is a place where the public seeks counselling from a pharmacist for various medical problems, including basic emergencies.^1-3 However, pharmacists need guidelines and training to improve their patient counselling skills, especially those needed for basic emergencies.⁴ No articles published from 1995-2009 were found in PubMed that matched the following search criteria: pharmacy students, emergencies/emergency situations, and training/education. However, 2 articles addressed the important role of the pharmacist in the urgent health care chain for general practitioners and emergency personnel. The general practitioner sees pharmacists as experts in nonprescription drugs and as filters to identify customers/patients with potentially serious illnesses. Emergency personnel see pharmacists as care providers who could manage an estimated 8% of the adults who instead report unnecessarily to the emergency department for care.^5,6

In our review of the pharmacy education literature, we found several articles discussing education of pharmacy students to care for patients with specific diseases or conditions such as asthma, diabetes, hypertension, or to provide care in specific circumstances, ie, disaster preparedness or emergency contraception.^7-11 These articles deal more with chronic disease follow-up, ie, asthma, diabetes, or hypertension, than management of emergency situations.^7-9

Since 2007 the Université catholique de Louvain has provided a first aid course for pharmacy students during the first semester of their third year. The aim of the course is to develop pharmacy students' clinical reasoning skills so they are able to provide the best response to patients' health problems in an emergency, keeping in mind that pharmacists often have to act alone and that the emergency problems they encounter in a pharmacy setting have a high degree of uncertainty. Until the introduction of this course in the curriculum, students were neither exposed to nor trained in providing basic emergency care. A needs analysis was performed among community pharmacists via a 2-part online survey. In the first part, pharmacists were asked to rate the frequency of the medical problems they encountered. In the second part, pharmacists were asked to specify, using a scale of interest, the emergency topics that would be important to include in a clerkship of their choice. The first aid course was built taking all of their feedback into account, and thus covers the most frequent pathologies encountered in pediatrics, trauma, and internal medicine.

Situations in which first aid is needed usually involve a degree of uncertainty. For instance, patients are often anonymous, their past medical history is usually unknown, and decisions have to be made based on relatively little information. The Script Concordance Test (SCT) allows assessment of providers' patient care skills in uncertain situations.¹² The SCT, which is based on cognitive psychology script theories, is useful for assessment of reasoning skills in medicine, but its effectiveness in pharmacy has not been documented. The tool allows assessment in ambiguous or uncertain situations and thus allows testing on real-life situations that are not adequately measured with current tests.¹³ This led us to use the SCT instead of multiple-choices questions or open-ended short-answer questions for pharmacy student assessment.

The use of SCT has been validated in domains such as urology, intensive care, and emergency medicine^14-16 and among pharmacists in Thailand to assess pharmaceutical diabetic care.¹⁷ Advantages of using the SCT are that it provides an indication of a student's clinical reasoning ability in real-life situations where there is uncertainty.^13,18,19 The objectives of this study were to develop and assess pharmacy students' clinical reasoning in basic emergency situations using the SCT.

DESIGN

The 2-hour elective first aid course was offered during the first 10 weeks of the fall semester. The aim of the course was to develop the clinical reasoning skills of pharmacy students so they would be able to provide the best response and action to patients in a basic emergency. The course was composed of 10 sessions, the first and last of which were used to administer practice tests. In the other sessions, students were presented with the most frequently encountered pathologies in the following specialities: pediatrics, trauma, and internal medicine.

In the first session, the SCT format was presented to students, they were informed that they would be taking part in a research study, and objectives for the course were presented. The precourse SCT was then administered. The following sessions (2-9) comprised the theoretical part of the course: 4 sessions on internal medicine (including life threatening situations), 2 sessions on pediatrics, 2 sessions on trauma. During each session, the instructor presented different clinical cases to small groups of students and then made the theoretical synthesis. For instance, the teacher described several cases of patients presenting with chest pain and asked students to identify the commonalities and differences of the various clinical presentations of chest pain and the ideal approach to take in addressing each. Then, the teacher summarized the clinical presentation, management options, and attitude to take toward handling this clinical situation.

In the last session, the postcourse test was administered and students' questions were answered. Before the final examination, each student received his precourse and postcourse results.

EVALUATION AND ASSESSMENT

The SCT was constructed following published guidelines and composed of vignettes based on actual clinical cases, each followed by a series of 3 questions.²⁰ The vignette did not contain all the information needed for the student to make an unequivocal treatment decision as would be needed in a real life emergency situation. Thus, the students had to apply knowledge learned in class to decide on the best course of action to take. The clinical vignettes were based on the main objective of the course (to develop the clinical reasoning skills of pharmacy students so they would be able to provide the best response and action to patients in a basic emergency).^13,20

The SCT differed somewhat depending on the assessment objective (hypothesis or attitude), and for each vignette, the questions that followed were ordered by format (eg, some items on hypothesis, followed by some items on advice or treatment options).²⁰ Each test item (question) consisted of 3 parts.²⁰ The first part was a hypothesis, advice, or a treatment option that was relevant to the situation described in the vignette. The second part presented new information (eg, a sign, condition) that might have an effect on the student's hypothesis or attitude. The third part was a 5-point Likert-type scale on which the student rated his/her reaction to the information presented in that item. An example of a SCT vignette is presented in Figure 1. It also was specified clearly in the instructions to the participants that, within each vignette, each of the 3 questions was independent of the other 2 questions. The goal of each question was not to determine the additive effect of a series of clinical information, but to determine the effect of an isolated element of clinical information on the student's hypothesis or attitude. ²⁰

Figure 1.

Example of clinical vignettes developed for a script concordance examination used in a first aid course for third-year pharmacy students.

The SCT was composed of 66 questions, constituting a certifying examination, and based on the main objectives of the first aid course (Figure 1). Twenty-six questions concerned clinical hypothesis and 40 questions were related to advice or treatment options. The 66 questions were distributed as follows: allergic reaction (3), basic life support (3), chest pain (6), diabetes (3), ocular pathology (3), gynecology (3), neurology (12), paediatrics (9), respiratory system (9), toxicology (6) and trauma including burns (9). Questions were reviewed for face and content validity by the 4 lecturers (physicians) as well as by a pharmacist.

The scoring of the SCT is based on an aggregate method, described by Norman and Norcini that takes into account the variability of responses of experienced clinicians to particular clinical situations.^21,22 Thus, for the construction of the answer key for the SCT, input was sought from a panel of 9 experienced practitioners.^20,23 The practitioners had to answer the SCT in the same timing conditions as students, using only their medical knowledge and in conditions close to those a pharmacist would have, ie, without specific drug available and without medical equipment other than what a pharmacist can use. The scoring process used for an SCT is based on the principle that any answer given by an expert reflects the opinion of an expert, and just because there is not unanimous agreement among the experienced practitioners does not mean an answer should be discarded.²⁴ In other words, any answer given by an expert has an intrinsic value, even if other experienced practitioners do not agree with it.¹⁴

For each item, the credit given to each possible answer depended on the number of experienced practitioners who gave that response (Table 1). The maximum credit for a given item was 1 point. Other responses were assigned a fraction of a point, and the responses that were not chosen by any of the experienced practitioners received no credit. Scores obtained on each question were added to obtain a total score for the test. This number was then divided by the number of questions and multiplied by 100 to get a percentage score.²⁰

Table 1.

Scoring Methods

Optimization of the test (question optimization) was performed by calculating the corrected item to total item correlation and then eliminating in a stepwise manner all items for which the correlation was less than 0.10.²⁰ Those questions were eradicated from the examination. This process ensured the maximal internal consistency of the final scale (Cronbach α).

The examination was administrated to 68 pharmacy students. Prior to the examination, students received 2 practice sessions answering an SCT. The SCT used for both sessions was composed of similar yet different cases than those used in the final examination. Students had 90 minutes to complete the test. The students' responses to each question were compared to those of the reference panel.

The 66-question test was completed by all students and within the allocated 90 minutes. The Cronbach α of the 66-question examination was 0.68. After optimization, 11 questions were deleted, the final test was reduced to 55 questions, with a Cronbach α of 0.82.

The students' mean score was 68.5% (SD 9.8) and the emergency physicians' mean score was 86.5% ± 4.2. Students' results ranged from 35% to 85.5%. As seen in Figure 2, the distribution of the students' scores deviated to the right of a Gaussian distribution plot and the distribution is flattened on the left (mean = 68.5%; median = 70.3%), with interquartile ranges from 62.2% to 75.6%.

Figure 2.

Distribution of pharmacy students' scores on a script concordance test used as the final examination in a first aid course (N = 68).

Four months after the final examination, a course survey using a 5-point Likert-type scale was administered to 20 randomly selected students to determine their perceptions of the new assessment tool (Table 2). Seventy percent of the students were satisfied or highly satisfied with the SCT type of examination, 75% agreed or strongly agreed that the examination was adequate for the course, 90% accepted that it showed real situations encountered in community pharmacy, and 75% indicated that the SCT helped prepare them for their future practice.

Table 2.

Satisfaction Survey Conducted Among 20 Pharmacy Students (Out of a Cohort of 68 Students) Following the SCT-Examination

DISCUSSION

The main objective of this study was to develop a reliable assessment tool to measure clinical reasoning in third-year pharmacy students after completing a first aid course. The course was created to improve students' ability to manage basic emergency situations that they might encounter in real practice. In view of the high degree of uncertainty linked to emergency situations, the authors chose to develop a SCT specific to first aid, which was then used to test the cohort of 68 pharmacy students. The SCT had an overall good reliability but could be optimized by further reducing the number of questions as the Cronbach alpha increased from 0.68 to 0.82 when the SCT was reduced from 66 to 55 questions.

For the 2 first years of the first aid course, the authors used traditional methods of evaluation: multiple-choice questionnaires and open-ended short-answer questions. However, those tools did not take 2 characteristics of emergency situations into consideration: the uncertainty and the specificity of clinical reasoning needed by the health care provider.²⁵

While a multiple-choice test can correctly assess factual knowledge through objective scoring and a high level of fidelity, it cannot assess clinical reasoning well²⁵ and may encourage superficial learning rather than problem solving.¹⁸ Although a multiple-choice test with a rich context might better explore clinical reasoning skills, students' expertise levels could not be differentiated as they can using an SCT.^16,25 Furthermore, because questions need to be unambiguous and not contain cueing effects, an effective multiple-choice test for clinical reasoning could be difficult to construct.²⁵

The open-ended question is a traditional assessment tool, but its scoring system is inconsistent from one examiner to the next.²⁵ In comparison with the other assessment methods used prior to the study, the SCT allowed the best alignment of course objectives, course content, and assessment tools. As detailed above, because the SCT reflects the response of each panel member, which gives it an intrinsic value, it allows the development and assessment of clinical reasoning in the context of uncertainty.^18,13

As the SCT evaluates links between knowledge rather than the actual amount of knowledge, we suggest that using this kind of tool actually improved students' clinical reasoning. This is particularly the case when facing emergency situations, as the pharmacist is rarely confronted with a clearly defined clinical entity (eg, myocardial infarction) but rather ill-defined problems (eg, chest pain).¹⁶ Scripts regarding cognitive psychology are networks of knowledge adapted to goals of clinical tasks.^12,24 Furthermore, as described in the literature, results on SCTs improve with experience.^24,26 While students have low or nonexistent clinical reasoning skills, experts in the field have enhanced clinical reasoning. Therefore, the SCT appears well suited not only to assess student learning in this first aid course but also to monitor students' progress.

Because, in our educational system, physicians teach courses such as first aid to pharmacy students, a medical panel was recruited to develop the SCT. However, this was not an easy task. It would be interesting in a future study to compare the resulting SCT created by a panel of pharmacists to that by a panel of physicians for the same course assessment. Students were generally satisfied using the SCT as a certifying examination. They felt that the clinical vignettes represented real situations encountered in community pharmacy, and that it could help them prepare for both their courses and their future professional practice.

There were several limitations to the present study. The study involved only 1 offering of the first aid course and 1 group of students at a single point in the program. It would be interesting to study a progression of students during the course of their training, eg, before and after the clerkship. Also, the study was conducted only once in a single center. It would be useful to offer the same assessment in other pharmacy schools.

A third limitation was the number of experienced practitioners (9) in the panel composition; it would have been preferable statistically to have 10 and ideally 15 experienced practitioners.²³ Also, the SCT was the sole method of evaluation used. In a further study, the SCT results should be compared with results using a test containing multiple-choice or open-ended short-answer questions. Finally, there were only 20 respondents to the satisfaction survey, which is a relatively small number. Students' interviews would be an interesting way to better explore students' perception about this type of assessment tool.

CONCLUSIONS

The developed first aid SCT was a practical and reliable assessment tool for third-year pharmacy students. Our findings support its construct validity for assessing pharmacy students' clinical reasoning in the face of uncertain basic emergency situations. Pharmacy students were satisfied using the SCT as a certifying examination and felt it prepared them well for real practice. These results open the avenue for further studies in the field of script concordance testing, such as comparative studies using expert panels composed of either physicians or pharmacists, or even conducting repeat SCTs after practical internships in order to monitor students' progress or serve as a feedback tool.