Correlation of the Health Sciences Reasoning Test With Student Admission Variables

Abstract

Objectives. To assess the association between scores on the Health Sciences Reasoning Test (HSRT) and pharmacy student admission variables.

Methods. During the student admissions process, cognitive data, including undergraduate grade point average and Pharmacy College Admission Test (PCAT) scores, were collected from matriculating doctor of pharmacy (PharmD) students. Between 2007 and 2009, the HSRT was administered to 329 first-year PharmD students. Correlations between HSRT scores and cognitive data, previous degree, and gender were examined.

Results. After controlling for other predictors, 3 variables were significantly associated with HSRT scores: percentile rank on the reading comprehension (p<0.001), verbal (p<0.001), and quantitative (p<0.001) subsections of the PCAT.

Conclusions. Scores on the reading comprehension, verbal, and quantitative sections of the PCAT were significantly associated with HSRT scores. Some elements of critical thinking may be measured by these PCAT subsections. However, the HSRT offers information absent in standard cognitive admission criteria.

INTRODUCTION

There are many definitions of critical thinking in the literature. Critical thinking has been defined as “reasonable reflective thinking that is focused on deciding what to believe or do.” ¹ The American Philosophical Association defined critical thinking as “purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based.”² Although definitions may differ somewhat, the consensus is that critical thinking skills are vital in the health professions because of the information-rich and constantly changing health professions environment.³

Pharmacist responsibilities have evolved in the healthcare system from the historic role of compounding and dispensing medication to a role of optimizing pharmaceutical care. This evolving nature places greater emphasis on problem-solving abilities and critical-thinking skills. As a result, healthcare professionals and accrediting bodies have called for pharmacy educators to train pharmacists who are able to optimize pharmaceutical care by using critical-thinking skills.⁴ To achieve this goal, colleges and schools of pharmacy have an interest in admitting students who already possess strong critical-thinking skills or have a disposition toward critical thinking. However, pharmacy program applicants are evaluated based on common criteria, such as performance in prerequisite courses, overall grade point average (GPA), scores on the Pharmacy College Admissions Test (PCAT), participation in extracurricular activities, and an interview to assess noncognitive attributes (eg, professionalism, empathy).

Critical-thinking assessment is often excluded from the normal battery of measures within the admissions process because traditional admissions criteria tend to correlate with academic success in the classroom.^5-8 Much like in students’ previous educational training, classroom success is typically measured by grades resulting from examinations of content knowledge. The correlation between traditional admission criteria and success may be invalid when success is measured by clinical performance in an experiential setting instead of academic performance in the classroom. Cognitive measures, such as the PCAT and GPAs, have mediocre to poor predictive ability for clinical performance.⁹ Clinical decision-making and performance during pharmacy practice experiences is more challenging to measure and predict because of variability in types of clerkships, critical-thinking ability of students, relative subjectivity of assessment measures by the evaluators, and the difficulty of assessing noncognitive skills, such as integrity, empathy, and professionalism. Predicting both classroom success and clinical performance may depend on a combination of traditional admissions criteria and measures of other qualities, such as critical thinking.

Pharmacy colleges and schools have begun to evaluate the critical-thinking skills of applicants, most commonly using the California Critical Thinking Skills Test (CCTST), the California Critical Thinking Disposition Inventory, or the Watson-Glaser Critical Thinking Skills Assessment (WGCTSA).¹⁰ The CCTST can predict academic success when used during the admission process. An examination of PCAT scores, CCTST scores, interview scores, and GPAs found that the PCAT and CCTST scores strongly predicted success in pharmacy courses.¹⁰ Additionally, the CCTST positively correlated with PCAT scores, suggesting that there may be some overlap in the abilities measured by the CCTST and PCAT.

The Health Sciences Reasoning Test (HSRT) is a 33-question, validated, multiple-choice test designed to assess critical-thinking skills in health sciences students. It uses the same critical-thinking subscales as the CCTST (ie, analysis, evaluation, inference, deduction, induction, and reasoning skills overall) and was designed to be an appealing instrument for measuring baseline critical-thinking skills based on the test questions being written in health-related and professional practice contexts. However, prior knowledge of health topics or lack of this knowledge, has not been found to affect the outcome of the HSRT.¹¹ This test was investigated in reference to admission criteria in 2 different cohorts of pharmacy students. In one study, the HSRT failed to predict first-year students’ GPAs, but scores were correlated with scores on the WGCTSA.¹² In the other study, scores on the HSRT significantly correlated with PCAT composite percentile scores but did not correlate with GPA.¹³ The equivocal evidence regarding critical-thinking tests and admissions provides an opportunity to further explore the relationship between information provided by the HSRT and cognitive admissions information provided by other tests, particularly the PCAT.

The purpose of this study was to assess the association between performance on the HSRT and cognitive student admission criteria, primarily PCAT scores, used at the University of North Carolina at Chapel Hill Eshelman School of Pharmacy based on the theory that the HSRT may be an additional tool that can be used to measure critical thinking, which is not captured by cognitive assessment instruments currently used in the admissions process.

METHODS

The study population consisted of 329 first-year PharmD students enrolled at the UNC Eshelman School of Pharmacy from fall 2007 to fall 2009. The HSRT was administered to students during their first year of the curriculum in either the fall semester or early in the spring semester. There were 135 students in the class of 2011, 108 in the class of 2012, and 86 in the class of 2013 who completed the assessment. Performance on the HSRT was compared with data collected during the admissions process, including gender, presence of a 4-year degree, undergraduate GPA, PCAT composite percentile rank, PCAT chemistry percentile rank, PCAT quantitative percentile rank, PCAT biology percentile rank, PCAT verbal percentile rank, and PCAT reading comprehension percentile rank. This study was approved by the University of North Carolina at Chapel Hill Institutional Review Board.

Data were de-identified prior to statistical analysis. Characteristics of study participants are presented using descriptive statistics (eg, means, standard deviations, and percentages). Bivariate relationships between HSRT scores and predictor variables were examined using Pearson correlations for numerical predictors (ie, GPA, PCAT scores, undergraduate GPA) and independent group t tests for categorical variables (ie, gender, class, receipt of a 4-year college degree prior to entering the PharmD program). Multiple linear regression was conducted using SAS (SAS Institute, Inc., Cary, NC) to assess the relationship between predictor variables and HSRT scores after controlling for other variables.

RESULTS

The majority of students were female (70.7%), and had a 4-year college degree (64.7%). Table 1 shows the sample mean and standard deviation for undergraduate GPA, percentile rank on the individual components of the PCAT, percentile rank on the composite PCAT, and HSRT scores. Also shown are the correlations between undergraduate GPA, PCAT percentile ranks, and the HSRT. Scores on the HSRT did not vary as a function of gender or class. Students who had a 4-year college degree prior to entering the PharmD program scored slightly lower on the HSRT than did students who entered the program without a previous 4-year degree (mean=24.1 vs 24.9, p<0.05).

Table 1.

Sample Characteristics of Student Applicants to the Doctor of Pharmacy Program

Table 2 presents the results of multivariate analyses predicting HSRT scores. The full model explained 27.4% of the variance in HSRT scores. After controlling for other predictors, 3 variables were significantly associated with HSRT scores: scores on the reading comprehension, verbal, and quantitative sections of the PCAT. These results confirm the results of the bivariate analyses; that is, even after controlling for other variables, HSRT scores were positively correlated with scores on the 3 sections of the PCAT listed above. In the multivariate analyses, HSRT scores were not significantly associated with having a prior 4-year degree, undergraduate GPA, or scores on the biology or chemistry sections of the PCAT.

Table 2.

Linear Regression Model Predicting Health Sciences Reasoning Test Scores

DISCUSSION

This study is one of the first in the pharmacy literature to examine the relationship between a critical thinking assessment and other standard cognitive admissions criteria. The major finding is that typical admission criteria, GPA, and PCAT performance explain a relatively small fraction of the variance (27.4%) in scores on the HSRT. This finding suggests that the HSRT is capturing something beyond standard admission information—most likely, students’ skills related to critical thinking.

The construct validity of the HSRT was derived by the same study format as the California Critical Thinking Skills Test, which makes it likely that the HSRT, like the CCTST, assesses critical-thinking skills.¹⁴ Several studies examined use of the HSRT compared with other measures that may relate to skills associated with critical thinking. A study of physical therapists used the HSRT to compare the performance of novices with those of experts. Experts scored better on the HSRT overall and in subsections related to analysis and deduction, but there was no difference between expert and novice performance in the other 3 subsections (induction, inference, and evaluation).¹⁴ A potential relationship between clinical skills and critical-thinking assessments has been found in nurses. In a study of the association between student nurse diagnoses and performance on both the CCTDI and the HSRT, investigators found no significant differences in the accuracy of nurse diagnoses by students scoring low on the CCTDI and those scoring high.¹⁵ Conversely, when examining student nurse diagnoses in relationship to their HSRT scores, more accurate diagnoses were associated with higher scores on the analysis domain of the HSRT. No other significant differences were found between diagnostic accuracy and scores on other subcategories of the HSRT. A limitation of this nursing study was that it was conducted using a younger nurse population. When the CCTDI was used in a mixed-age population, higher CCTDI scores were found among older and more experienced nurses than among their younger comparators.^16,17 In pharmacy, 1 study found that the HSRT failed to predict first-year GPA, but HSRT scores were correlated with scores on the validated critical-thinking instrument, WGCTSA.¹²

One of the challenges of assessing critical-thinking skills is potential covariates. Within the current study, there was a significant positive relationship between PCAT reading comprehension and HSRT scores. This finding may suggest a potential limitation for students with underdeveloped reading comprehension skills or those for whom English is a second language. Within the current sample, there were few students for whom English was a second language. This also may indicate an alignment between critical-thinking skills and the ability to dissect written material. Roy and colleagues found that critical-thinking skills were lower in medical students given a video-based patient case than in medical students given a text-based patient case. While there are several possible reasons for this effect, familiarity with text material and a high degree of reading comprehension cannot be ruled out.

Limitations of this study include sample size, time of test administration, and multiple PCAT attempts by students. Although the assessment was administered over 3 years, it did not capture the entire cohort of students within each year. The HSRT was administered during a professional experience program course, which had variable attendance. This half-credit course orients students to various aspects of the professional experience, such as various practice sites and settings. Although the HSRT was administered during students’ first year of the pharmacy program, it was not administered at a consistent point in the first year (ie, either in the fall semester or early spring semester). Students’ critical-thinking ability could change over time from admission and could vary depending on when during the first year the HSRT was administered. These factors, however, are not expected to have significantly impacted the results, considering that the HSRT administration times during the first year differed by only a few weeks to 4 months. For example, 2 studies demonstrated a small but significant change in students’ critical-thinking skills after completing 4 years of a pharmacy curriculum, suggesting that changes in critical thinking or detectable differences in critical thinking assessment are unlikely to occur over a much smaller time period.^19,20 Finally, a high percentage of students had multiple PCAT scores. In these cases, the highest composite PCAT score was used for the analysis. Using the lowest or average PCAT score may have produced a different result.

CONCLUSION

Scores on the reading comprehension, verbal, and quantitative sections of the PCAT were significantly associated with HSRT scores. Some elements of critical thinking may be measured by these PCAT sections. However, the HSRT offers additional information that is not provided by standard cognitive assessment measures used in the admission process. Further work is needed to identify whether student success, particularly in clinical environments such as pharmacy practice experiences, can be predicted by HSRT scores.