The Predictive Utility of Nontraditional Test Scores for First-Year Pharmacy Student Academic Performance

Abstract

Objectives

To determine the value of employing the Learning and Study Strategies Inventory (LASSI), Defining Issues Test (DIT), and Watson-Glaser Critical Thinking Appraisal (WGCTA) in predictive models for first-year pharmacy student academic performance.

Methods

Six years of pharmacy student admission and progression data were evaluated. Additional predictive validity offered by these variables over a model of prepharmacy grade point average and pharmacy college admission test (PCAT) score was examined.

Results

None of the 3 measures offered the ability to predict first-semester or first-year academic performance over and above GPA and PCAT.

Conclusions

The LASSI, DIT, and WGCTA do not appear to assess abilities that are directly related to academic performance; however, these instruments may be useful in assessing other student attributes that are highly desirable for the practice of pharmacy.

INTRODUCTION

Colleges and schools of pharmacy use multiple techniques to deal with a student's failure to progress. Seventy percent of reporting institutions allowed students to retake an entire year of course work, 54% allowed them to retake a semester, and 24% provided a retest.¹ All of these options require the time and energy of faculty members and administration. The primary purpose of a college or school of pharmacy is to educate students and provide the community at large with able pharmacy practitioners. In fact, it is this responsibility to develop able practitioners that has led some pharmacy schools to adopt abilities-based education.²

The purpose of abilities-based education is to produce graduates who possess those abilities that are important in the practice of pharmacy.² In a simplified model of academic progression, students are admitted to the program, educated in the curriculum, and graduate with these abilities. Opportunities exist in both the application process and the monitoring process to evaluate a student's ability to learn and perform the specified abilities.

Some schools of pharmacy have added both quantitative and qualitative criteria to admission and progression assessments. For example, at The University of Mississippi, the School of Pharmacy assesses employment history, leadership, citizenship, and performance on the Watson-Glaser Critical Thinking Appraisal (WGCTA) as part of its admission formula.³ The School also collects data for the purpose of evaluating ability acquisition. These measures, including the Learning and Study Strategies Inventory (LASSI) and the Defining Issues Test (DIT), have been added to the tests administered as assessment tools at the school since 1996.^4,5 These measures served only to inform the faculty members and administration of abilities acquired by students and did not serve as a basis for decisions regarding admission or progression.

This study evaluated the potential of these nontraditional assessments to predict students’ progression through the pharmacy curricula. While colleges and schools of pharmacy have begun to adopt abilities-based curricula, the predominant admission formula remains based heavily on quantitative measures. The WGCTA, LASSI, DIT, and other nontraditional measures may add to the quantitative gold standards of GPA and PCAT.

The most common evaluation of admission and progression criteria has centered on quantitative measures such as GPA and PCAT.^6-25 Munson and Bourne evaluated prepharmacy GPA and PCAT scores and found both to be viable predictors of first-year pharmacy GPA.⁶ This trend has continued to the recent work of Chisholm and colleagues, who found prepharmacy math/science GPA, prior degree, and certain PCAT components to be predictors of first-year pharmacy GPA.⁷

The exploration of nontraditional measures of assessment for the purpose of predicting progression has been limited. Most commonly, the assessment of demographic variables has been investigated to expand the model beyond GPA and PCAT.

Liao and Adams added demographic variables to the quantitative predictors of PCAT and academic performance. Variables included were gender, age, residency (ie, in-state vs. out of state), and number of universities attended. The study found no significant differences in first-year GPA based on student demographics.⁸

Cox and Teat included gender and age in their analysis of 3 classes of pharmacy students at Campbell University. Their analysis showed the best correlation for gender, but it resulted in a low correlation (r = 0.16). Neither of the demographic variables were significant predictors.⁹

The variables that Jacoby and colleagues explored were age, years in college, total hours in math/science courses, origin (transfer school), individual course grades, prepharmacy GPA, and PCAT score. Only once, for 1974 students, did age emerge as a significant predictor of pharmacy student progression, accounting for 6% of the variance.¹⁰

Palmieri derived a model for the prediction of success within the transfer student population at the University of Wyoming School of Pharmacy. The research included PCAT score, chemistry GPA, transfer GPA, total hours on transcript, type of college attended before transfer, major, and student age. The final model included age as a predictor of performance.¹¹ Thus, the literature shows a history of nontraditional predictors of academic performance showing marginal predictive capabilities at best.

The use of measures other than GPA and PCAT to make admission decisions is increasing. For example in addition to prepharmacy GPA and PCAT, The University of Mississippi School of Pharmacy collects other data from entering students. Among these is the Watson-Glaser Critical Thinking Appraisal.³ The WGCTA is a standardized test administered formerly by The Psychological Corporation (now by Harcourt Assessment) that is used to measure attitudes of inquiry, knowledge of the nature of valid inferences, and the ability to apply those attitudes and knowledge.³ This assessment reports a single score reflective of 5 components of critical thinking, inference, recognition of assumptions, deduction, interpretation, and evaluation of arguments.²⁶

The University of Mississippi School of Pharmacy also collects data using measures that are not currently included in the admission calculation. Two of these are the Learning and Study Skills Inventory and the Defining Issues Test. These measures are distinct from the WGCTA and other assessments.

The LASSI is a test designed to assess students’ learning and study strategy methods. The LASSI is intended for the identification of areas of potential improvement in individual student's study skills and to develop a prescriptive educational plan in the event of remediation. The LASSI has 10 scales and 80 total items measuring attitude, motivation, time management, anxiety, concentration, information processing, selecting main ideas, study aids, self-testing, and test strategies.⁴ This measure is diagnostic in nature and the scores on individual scales can be used to identify areas for improvement in learning techniques for students. As a result, each of the 10 scales is independent. However, combinations of these scales comprise 3 components of strategic learning. Information processing, selecting main ideas, and test strategies comprise the Skill Component. Attitude, motivation, and anxiety comprise the Will Component. Concentration, time management, and self-testing and study aids comprise the Self-Regulation Component. Currently, there are over 2000 institutions that use the LASSI.²⁷

The DIT assesses how people deal with issues of right and wrong in social situations. In the DIT administration, respondents are presented with a hypothetical situation and asked to provide a recommendation, rate the important issues in the short story, and identify the most important items.⁵ The responses to the vignettes are multiple choice.

Research Using the DIT, LASSI, and WGCTA

The DIT has been used to examine the professionalism of pharmacy students and is also considered as an admission criterion.^12,13 Latif suggested that the DIT should be considered as a criterion for applicant and student evaluation because of the importance of moral reasoning in clinical decision making.¹³ Latif also recently found that the moral reasoning scores using the DIT of first-year pharmacy students in a nationwide sample were not different from the scores of third-year pharmacy students.²⁸ The LASSI and the WGCTA have been studied less than the DIT. Pringle and Lee examined the LASSI in relation to student success or failure on part I of the National Board of Chiropractic Examiners (NBCE) Test and found a high correlation between LASSI and NBCE test scores.²⁹ Other anecdotal reports have shown relationships between the LASSI motivation and concentration subscores and student success.³⁰ The relationship between the WGCTA and medical student performance during clinical clerkships seems to be relatively weak. Overall the WGCTA was found to have weak relationships with performance in the clerkship with no correlations greater than 0.34 and some closer to 0.31 To date, the role of these measures in the academic performance of pharmacy students has not been assessed. With the typical contribution of the PCAT and GPA scores to the variance in first-year pharmacy student GPA being less than 50% shared variance, it is important to evaluate nontraditional measures for their role in the prediction of academic progression. It is possible that these nontraditional measures may also aid in the prediction of academic performance.

Other than prepharmacy GPA and PCAT scores, research has failed to identify consistently significant contributors to the prediction of the academic progression of pharmacy students. While many additional variables have been explored, the value of those variables in the prediction of academic performance has not been consistent. This may be due in part to small sample sizes. Prediction based on more years of data might yield more stable predictors. The usefulness of additional assessments of students’ abilities, ethical disposition, and background, such as the WGCTA, the LASSI, and the DIT, in the prediction of academic performance remains unexplored and might provide value to a predictive model. Nevertheless, the motive for their inclusion would likely be the selection of students based on traditional academic and nontraditional criteria. This led to the following hypothesis: the Watson-Glaser Critical Thinking Appraisal, the Learning and Study Skills Inventory, and the Defining Issues Test contribute significantly to the variance accounted for by a traditional academic performance model of prepharmacy GPA and PCAT.

METHODS

This project used the in-house data collected at the time of admission and the database created for the monitoring of student progression at a single southern US school of pharmacy. A sample of 6 consecutive years’ of data (1997 to 2002 inclusive) was drawn from the records and database at The University of Mississippi School of Pharmacy. The selection of a 6-year time frame coincided with the inclusion of the LASSI, DIT, and WGCTA as assessment tools and resulted in 405 student records in the sample.

The University of Mississippi School of Pharmacy has an early assurance program which allows for the admission of freshman-level students into the pharmacy curriculum. These students apply to the school through a different process, and the course work sequence for their progression is different from that followed by students admitted though the traditional mechanism. As a result, these early assurance students were excluded from the analysis.

To enable comparison to previously conducted research, the dependent variables of interest were the first-professional semester pharmacy GPA and first-professional year pharmacy GPA. These variables served as the dependent variables for the regression equations. Independent variables extracted from the records and database included prepharmacy math/science course GPA, LASSI scores, DIT scores, WGCTA scores, and PCAT composite scores. The use of prepharmacy math/science GPA allowed for an enhanced ability for other institutions to replicate this study and is consistent with the work of Chisholm et al.⁷

RESULTS

The research hypothesis was analyzed using a hierarchical multiple regression model. In the base model, the traditional variables of prepharmacy math/science GPA and PCAT were entered first. Subsequently, the nontraditional assessments, LASSI, DIT, and WGCTA were entered. Separate models were performed for first-semester GPA and first-year GPA.

The base models of prepharmacy math/science GPA and PCAT were significant for both first-semester and first-year pharmacy GPAs. The 2 predictors resulted in adjusted R² values of 0.342 for the first-semester GPA and 0.344 for the first-year pharmacy GPA. For both base models, the prepharmacy math/science GPA and the PCAT scores were significant predictors in the regression equations:

• First-semester pharmacy GPA = 0.512 (prepharmacy math/science GPA) + 0.007 (PCAT percentile score) + 1.145

• First-year pharmacy GPA = 0.464 (prepharmacy math/science GPA) + 0.006 (PCAT percentile score) + 1.3

Following the computation of the base models, the nontraditional assessments were added to each model. In neither the first-semester pharmacy GPA model nor the first-year pharmacy GPA models were the nontraditional assessments significant predictors of GPA (Tables 1 and 2).

Table 1.

First-Semester GPA Regression Model Used to Determine the Utility of Nontraditional Test Scores in Predicting First-Year Pharmacy Student Academic Performance

Model 1- Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA

Model 2 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile

Model 3 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile, WG Score

Model 4 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile, WG Score, LASSI1, LASSI8, LASSI4, LASSI3, LASSI6, LASSI7, LASSI9, LASSI5, LASSI2, LASSI10

Table 2.

First-Year GPA Regression Model Used to Determine the Utility of Nontraditional Test Scores in Predicting First-Year Pharmacy Student Academic Performance

Model 1- Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA

Model 2 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile

Model 3 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile, WG Score

Model 4 - Predictors: (Constant), PCAT Percentile Score, Prepharmacy Math/Science GPA, DIT Percentile, WG Score, LASSI1, LASSI8, LASSI4, LASSI3, LASSI6, LASSI7, LASSI9, LASSI5, LASSI2, LASSI10

Based on these results, the research hypothesis is not supported. The LASSI, DIT, and WGCTA do not contribute significantly to regression models based on prepharmacy math/science GPA and PCAT for the prediction of first-semester pharmacy GPA or first-year pharmacy GPA. In part this is due to shared variance between the LASSI, DIT, and WGCTA measures and the PCAT and prepharmacy GPA measures.

On their own, some of the LASSI scores are significantly correlated with first-year GPA (LASSI2, motivation = 0.20; LASSI4, anxiety = 0.19; LASSI5, concentration skills = 0.12; LASSI7, selecting main ideas = 0.17; LASSI10, test strategies = 0.22), as is the WGCTA (r = 0.25, p < 0.01). However, none of these correlations exceeded r = 0.25 and all of these relationships were nonsignificant in the regression analyses with the base model of PCAT and prepharmacy GPA.

DISCUSSION

This study assessed the utility of the LASSI, the DIT, and the WGCTA in predicting academic performance (first-semester and first-year pharmacy GPA). None of the measures significantly increased the predictability performance over the traditional prepharmacy math/science GPA and PCAT scores. While these nontraditional assessments are not significant predictors of progression over and above currently used predictors, one should not conclude that they are not appropriate for the assessment of ability acquisition.

Not surprisingly, the Defining Issues Test did not add to the prediction of performance. This measure assesses how people deal with issues of right and wrong in social situations.¹³ As a measure of moral reasoning, it assesses an ability that is not central in GPA acquisition; in fact, it has the potential to be negatively correlated with performance. That is, the less scrupulous student may be more willing to behave in an ethically questionable manner to acquire improved grades. As was noted, some schools have adopted the DIT as part of their admission criteria. This decision can still be considered appropriate in that the goal of DIT inclusion is to admit students of a certain ethical caliber rather than to aid in the prediction of academic success.

Unlike the DIT, the Learning and Study Skills Inventory is a test designed to assess students’ learning and study strategy methods.¹² While it is interesting that none of the individual scores on the LASSI increased the ability to predict academic performance, this set of scales is useful still in identifying those students with learning methods that may affect the ability to perform well academically. In this light, the LASSI has been used as a diagnostic test to determine which students may be more at risk of remediation. This value of the LASSI in pharmacy education has not been diminished by the findings of this study.

The WGCTA, a standardized test administered by Harcourt Assessment and used to measure attitudes of inquiry, knowledge of the nature of valid inferences, and the ability to apply those attitudes and knowledge, also failed to contribute to the prediction of academic performance. It may be that critical thinking and the reasoning and inferences associated with critical thinking are higher order concepts that are not captured within a GPA or individual grade. Another possibility is that early curriculum does not involve enough critical thinking for this measure to be useful as a predictor variable. The ability to apply reason and problem solve as measured by the WGCTA is distinct from the ability to perform in pharmacy courses and the WGCTA is likely to be a measure of at least some of those abilities desired in pharmacy school graduates. Therefore, while these tests may also assess abilities separate from those captured within a grade point average, it was not illogical to hypothesize that the measures would be significant predictors of performance.

This study is based on 6 years of data. While this is a significant amount, the data are all from one school which limits the generalizability of the findings. The years included in this study correspond with the inception of the use of the LASSI, DIT, and WGCTA at this institution, making it possible that implementation and timing issues affected the tests’ ability to predict performance accurately. At best, this study is a true assessment of the measures’ capability in progression prediction. At worst, the effect of these measures is diminished. Further study of the value of the LASSI and WGCTA as admissions criteria is warranted.

CONCLUSION

This study found that scores on the LASSI, DIT, and WGCTA do not account for variance in pharmacy school GPA. These measures were intended to assess different abilities that may not be captured within a GPA, and their use in that manner may indeed be appropriate. In fact, their lack of correlation with academic performance is potential evidence that the LASSI, DIT, and WGCTA are measuring their intended concepts. Based on these findings, however, their use in the prediction of first-year pharmacy student academic performance is questionable at best.