Association of Health Sciences Reasoning Test Scores With Academic and Experiential Performance

Abstract

Objectives. To assess the association of scores on the Health Sciences Reasoning Test (HSRT) with academic and experiential performance in a doctor of pharmacy (PharmD) curriculum.

Methods. The HSRT was administered to 329 first-year (P1) PharmD students. Performance on the HSRT and its subscales was compared with academic performance in 29 courses throughout the curriculum and with performance in advanced pharmacy practice experiences (APPEs).

Results. Significant positive correlations were found between course grades in 8 courses and HSRT overall scores. All significant correlations were accounted for by pharmaceutical care laboratory courses, therapeutics courses, and a law and ethics course.

Conclusion. There was a lack of moderate to strong correlation between HSRT scores and academic and experiential performance. The usefulness of the HSRT as a tool for predicting student success may be limited.

INTRODUCTION

Critical thinking has been defined as “the process of purposeful, self-regulatory, judgment [which] gives reasoned consideration to evidence, context, conceptualizations, methods, and criteria.”¹ Ongoing advances in technology and changes in necessary workplace skills have placed a growing emphasis on the need for pharmacists to possess critical-thinking skills in order to solve complex patient-care problems and work in multidisciplinary interprofessional teams. As a result, significant interest exists among pharmacy educators concerning the measurement and development of these skills. This interest was highlighted by: the 2009 American Association of Colleges of Pharmacy (AACP) Curricular Change Summit, which identified the ability to think critically as one of the key outcomes that pharmacy graduates should possess; the 2009-2010 AACP Academic Affairs Standing Committee, which published an environmental scan of critical thinking and problem solving in pharmacy schools; and the Summit to Advance Experiential Education in Pharmacy, which emphasized the importance of enhancing admissions criteria and screening tools to include problem solving and critical thinking.^2-4

Critical-thinking skills can significantly impact student outcomes. College graduates with lower gains in critical thinking, for example, have higher unemployment rates, amass higher credit card debt, and report lower lifestyle satisfaction than their more accomplished peers.⁵ A meta-analysis of studies evaluating critical-thinking test scores and academic success of health professions trainees, including 5 studies from pharmacy education, found that critical thinking and academic success were moderately correlated (r=0.31).⁶ In pharmacy education, a study by Allen and Bond⁷ found that the California Critical Thinking Skills Test (CCTST) was a predictor of practice-related courses and clerkship success, while Kidd and Latif⁸ found the CCTST to be a predictor of performance in APPEs. When looking at specific coursework, Miller⁹ found a correlation between CCTST scores and grades in a drug literature evaluation course and Adamcik¹⁰ reported strong correlations between the Watson-Glaser Critical Thinking Appraisal (WGCTA) and performance in therapeutics coursework. Given the importance of critical thinking to pharmacy practice and student outcomes, identifying measures and tools that effectively capture the critical-thinking ability of students at the point of admissions could significantly improve the ability of colleges and schools of pharmacy to identify qualified students capable of excelling in a pharmacy curriculum and ultimately meeting the complex needs of 21^st century health care.

The HSRT is a validated critical-thinking skills test with questions stated in a health sciences context. It is intended for use in health sciences educational programs and with professional practitioners in health sciences fields. It was developed by Insight Assessment, who also developed the CCTST and California Critical Thinking Disposition Inventory. The HSRT consists of 33 multiple-choice questions from 5 critical-thinking domains categorized to match the constructs of the American Philosophical Association Delphi Consensus Definition of critical thinking: analysis, inference, evaluation, induction, and deduction.¹¹ The test is usually administered over a 50-minute period, and no previous knowledge of science or health sciences is required, nor is it beneficial in answering the questions. An overall score of 15 to 20 is indicative of moderate critical-thinking abilities, 21 to 25 is indicative of strong critical-thinking abilities, and ≥26 is indicative of superior critical-thinking abilities. Subscale scores above 5 are considered strong for analysis, inference, and evaluation, and subscale scores above 8 are considered strong for induction and deduction.¹²

A limited number of studies could be found in the literature evaluating the association of HSRT scores and course grades (didactic or experiential) for health sciences students. An association was not found between scores on the HSRT and academic performance in the first and second year of classroom-based coursework within the pharmacy curriculum at the University of Mississippi.¹³ While the HSRT may provide insight into the critical-thinking skills of students at the point of admissions,¹⁴ it is unclear whether this test may be useful for predicting student performance within a pharmacy curriculum. The objective of our study was to assess the association of scores on the HSRT with classroom and APPE performance in the PharmD curriculum. If an association were found, the HSRT could serve as an additional admissions tool to select students who may excel in coursework and pharmacy practice experiences that rely heavily on critical-thinking skills. This study is an extension of a previous study that explored the correlation between typical cognitive admission criteria and HSRT scores.¹⁴

METHODS

The HSRT test administration methods have been described in a previous paper exploring the correlation of the HSRT with student admission variables.¹⁴ The HSRT was administered to 329 of 459 (71.6%) P1 students enrolled at the UNC Eshelman School of Pharmacy from fall 2007 to fall 2009 (Table 1). Response rates were 87% for the class of 2011 (135 of 155), 72% for the class of 2012 (108 of 151), and 56% for the class 2013 (86 of 153). The test was administered during the students’ first year of the curriculum in either the fall or spring semester.

Table 1.

Demographic Data for First-Year Pharmacy Students Who Completed the Health Sciences Reasoning Test

Following completion of the test, an overall HSRT score and 5 subscale scores (analysis, inference, evaluation, deductive reasoning, and inductive reasoning) were calculated for each participant. Each student’s final course grades, measured on a scale from 1 to 4 (4=A, 3=B, 2=C, 1=F ), were obtained for all 29 classroom-based courses offered during the first 3 years of the curriculum. In addition, each student’s performance on APPEs, measured on a scale of 0 to 100, was obtained. For the purposes of analysis, each APPE was coded as 1 of 6 types: ambulatory care, advanced community, advanced hospital, adult acute care, clinical specialty, and elective. To interpret the findings, each course was categorized as basic science (Pharmaceutics I and II, Biochemistry I and II, Medicinal Chemistry I, II and III, and Physiology), therapeutics, pharmaceutical care laboratories, or other (Nonprescription Drugs and Self-Care, Pharmacy Practice Management, Law and Ethics, Pharmacokinetics I and II, Pharmacodynamics, Health Systems, Immunizations, and Drug Literature and Evaluation). This study was a continuation of a previous study and considered exempt by the University of North Carolina at Chapel Hill Institutional Review Board.

All data were de-identified prior to statistical analysis. Characteristics of study participants are presented using descriptive statistics (eg, means, standard deviations, percentages). Pearson correlation was used to determine the relationship between HSRT scores and all continuous variables, including student performance in each of the 29 classroom-based courses and on each of the 6 types of APPEs. Any -0.20 < r_p <0.20 was considered to demonstrate no relationship or negligible relationship. The t test for independent samples was used to compare groups (ie, gender, class). Continuous data were presented as mean and standard deviation. Statistical significance was established at α=0.05. All quantitative data analysis was conducted in SPSS for Windows, version 20 (IBM Corp, Armonk, NY).¹⁵

RESULTS

Table 1 details the characteristics of the study participants. The HSRT was completed by 329 of 459 (71.6%) P1 students over 3 years. One hundred thirty-five students from the class of 2011 participated in fall 2007, 108 students from the class of 2012 participated in fall 2008, and 86 students from the class of 2013 participated in fall 2009 and spring (January) 2010.

The average overall HSRT score for all participants was 24.4±3.5 (of 33 possible points). Average scores on the subscales were 4.9±1.0 points on HSRT analysis, 4.0±1.3 points on HSRT inference, 5.2±0.9 points on HSRT evaluation, 8.1±1.2 points on HSRT induction, and 7.9±1.7 points on HSRT deduction. Female students scored significantly higher on the inference subscale than male students (4.2±1.2 vs 3.8±1.4, p=0.046). There was no significant difference on the overall HSRT score or the other 5 subscales for gender or racial-ethnic group. There were no significant differences in any scores between each class year.

Nine courses demonstrated a significant relationship with the HSRT and its subscales (Table 2). Significant correlations with r_p≥0.20 were found between the HSRT overall score and 2 (out of 7) therapeutics courses, 4 (out of 5) pharmaceutical care laboratories (PCLs), Law & Ethics, and the Nonprescription Drugs & Self-Care course. While some of these courses were also significantly correlated with the HSRT analysis, evaluation, induction, and deduction subscales, no courses or APPEs presented significant associations with the HSRT inference subscale. The strongest significant correlations were associated with Law & Ethics and PCL 3 (the third laboratory course in the PCL sequence) (Table 3). The strongest 5 correlations for the HSRT overall and its subscales were between Law & Ethics and HSRT overall (r_p=0.27), PCL 3 and HSRT overall (r_p=0.26), PCL 3 and HSRT deduction (r_p=0.26), Law & Ethics and HSRT deduction (r_p=0.25), and PCL 3 and HSRT induction (r_p=0.23). There were no significant correlations found between HSRT scores and grades from basic science courses or from APPEs. All significant correlations found were positive and weak (r_p<0.3).

Table 2.

Number of Courses Significantly Correlated with Health Sciences Reasoning Test Scores^a

Table 3.

Correlation of Health Sciences Reasoning Test Overall Score with Course Grades^a

DISCUSSION

This study examined the association between performance on the HSRT and academic performance at the University of North Carolina at Chapel Hill Eshelman School of Pharmacy. This is one of the first studies in pharmacy education to examine the relationship between the HSRT and classroom and APPE grades. The major finding was that 9 courses were significantly correlated with HSRT scores: 4 pharmaceutical care laboratory courses, 3 therapeutics courses, Law and Ethics, and Nonprescription Drugs & Self-Care course. However, because of the lack of moderate to strong correlations between HSRT scores and academic performance, the usefulness of the HSRT as an admissions instrument may be limited.

Significant correlations between the HSRT and courses were found in more applied courses, as opposed to foundational courses, although the correlations were weak. Law and Ethics and PCL 3 presented the strongest associations with the HSRT and its subscales. This may be because of the activities and content of these courses. For example, neither of these courses are traditional lecture-based courses and both promote strategies that have been shown to facilitate critical thinking. Traditional lecture is not as effective a method to teach critical thinking compared with other strategies such as problem-based learning, reading assignments, reflection, self-assessment, and experiential learning.^16-18 Although the PharmD curriculum has changed over the past few years to include teaching and learning strategies other than traditional lectures, most of the classroom-based courses examined in this study were conducted in a traditional lecture format. It is surprising then that the HSRT was derived from the same study format as the CCTST, which makes it likely that the HSRT does measure critical thinking.¹⁹ Research by Cox and colleagues¹⁴ suggests that the HSRT is likely measuring something absent in standard cognitive admission criteria.

This study found only weak correlations between HSRT and academic performance in some of the courses and no association in the remaining courses. While these findings support Baseck’s¹³ study of HSRT and classroom grades in the first and second years of a pharmacy curriculum, they are in direct contrast to numerous studies demonstrating strong and significant relationships between academic performance and critical-thinking scores from other tests like the CCTST and WGCTA.^7-10 A possible explanation for the discrepancies among these studies is the institutional and course-level variability associated with grading. While grades are often considered a proxy for critical thinking, they are comprised of varying criteria influenced, in part, by course content, format, assignments, learning objectives, and instructor preferences. Taken together, these studies highlight the challenges associated with measuring critical thinking and using those measures to inform pharmacy curricula in a meaningful way.

At the point of admissions, other tools and instruments may be useful in capturing constructs or skills that are more strongly correlated with success in the pharmacy curriculum. While critical thinking is clearly a vital skill for pharmacy students, our students are also graded on other critical constructs, including communication, collaboration, content mastery, and professionalism. Using approaches in admissions that account for the knowledge, skills, and abilities on which our students are assessed within the curriculum, including critical thinking, may be a more effective approach to identifying students who will excel in the curriculum and pharmacy practice. As a result, our institution is reevaluating our admissions model and incorporating more measures of noncognitive as well as cognitive constructs in the process. One instrument that may be helpful in identifying noncognitive traits, including critical thinking and problem solving, is the multiple-mini interview.²⁰ This tool enables schools to evaluate multiple skills at the point of admissions. Additionally, institutions may want to admit students who demonstrate strong critical-thinking skills upon admission as changes in critical-thinking skills over the course of study have been variable in previous studies and often do not improve dramatically.^21-25

Given the emphasis on critical thinking and problem solving in pharmacy education, institutions should consider systematically and explicitly assessing critical thinking at regular intervals throughout the curriculum using valid, reliable measures. Critical thinking may be most effectively practiced and measured in courses like PCL, which rely heavily on applied activities that integrate learning across the curriculum. However, the usefulness of the HSRT as a tool for predicting student success may be limited.

The limitations of this study included the sample size and time of test. The HSRT was administered during a professional development course, which had variable attendance and resulted in differences in response rates between the 3 years of test administration. Informed consent was used with no incentives offered to students who participated in the study. Recruitment strategies did not differ between the years. While the test was not administered to the entire cohort of students within each year, the results were analyzed as a single cohort because there were no significant differences between the cohorts. This resulted in a 71.6% response rate, which was felt to be sufficiently high. In addition, the sample for this study was limited to a single institution. Future studies examining the relationship between the HSRT and academic performance of pharmacy students should be extended to include other institutions. Additionally, the HSRT was administered to some students early in the spring semester of the P1 year as opposed to the fall semester upon entrance to the program. Critical-thinking ability could have varied depending on the point in the P1 year when it was administered; however, this likely did not significantly impact the results because the administration times of the HSRT during the first year only differed by 4 months at most. Two studies exploring changes in critical-thinking skills throughout 4 years of a pharmacy curriculum showed only small changes. A smaller time period may not lead to changes in critical thinking or detectable differences in critical-thinking assessment.^9,23

CONCLUSION

All associations examined between the HSRT and academic performance in the current PharmD program were weak or negligible. While in theory the HSRT is an attractive instrument to predict academic performance, its use may be limited by the lack of moderate to strong correlation between scores and course performance in the first through the third years and with APPEs. This is likely because of the multiple factors that contribute to academic and experiential success and the lack of critical-thinking assessment in current evaluation tools. Given the emphasis on critical thinking and problem solving in pharmacy education, institutions should consider systematically and explicitly assessing critical thinking at regular intervals throughout their pharmacy curriculum using valid, reliable measures; however, other approaches for determining student qualifications and strengths at the point of admissions should be considered.