Predicting Performance in the First-Year of Pharmacy School

Abstract

Objectives. To determine the predictive validity of the Pharmacy College Admission Test (PCAT) scores and other variables for students’ success in the first year of doctor of pharmacy (PharmD) programs.

Methods. PCAT scores, entering grade-point averages (GPAs), demographic characteristics, and GPAs for the first year of pharmacy curriculum were collected from the Pharmacy College Application Service (PharmCAS) and from 22 pharmacy programs.

Results. PCAT scores and entering GPAs were positively correlated with subsequent GPAs after the first year. Regression analyses showed the contribution of PCAT scores and entering GPAs in predicting first-year pharmacy GPAs.

Conclusions. PCAT scores and prepharmacy GPAs both showed moderate predictive validity in indicating candidates likely to succeed in the first year of the pharmacy program. These findings are consistent with those of previous similar studies.

INTRODUCTION

The Pharmacy College Admission Test (PCAT) consists of 5 multiple-choice subtests (verbal ability, biology, chemistry, reading comprehension, and quantitative ability) and a writing subtest. Scaled scores ranging from 200-600 are reported for each multiple-choice subtest, along with a composite score (an unweighted average of the 5 subtest scaled scores). Separate conventions of language and problem-solving scores are reported for the writing subtest. According to the American Association of Colleges of Pharmacy, nearly 79% of the PharmD programs in the United States currently include PCAT scores in their admission requirements.¹

An important method of evaluating high-stakes standardized admission tests such as the PCAT is to examine its criterion-related or predictive validity to determine how well the scores predict later academic performance. Studies conducted since the introduction of the PCAT in 1974 have found the test to be a moderate-to-strong predictor of subsequent performance, with predictive validity statistics comparable to those of other standardized admission tests commonly used by graduate and professional schools.^2,3 However, many older studies were conducted before the PharmD became the required degree for professional pharmacy practice, and relatively few considered demographic characteristics as predictor variables.⁴ Most of the data for these studies were collected prior to the introduction of the current PCAT scaled scores in 2004 and the writing subtest in June 2007.

The purpose of the current study was to examine data collected from PharmD programs to determine the value of the current PCAT scaled scores, PCAT writing scores, and entering GPAs in predicting subsequent GPAs during the first year of pharmacy school. The study also addressed the issue of differing PCAT mean scores for various demographic groups by including demographic characteristics as predictor variables. The study examined whether a combination of PCAT scores and prepharmacy GPA constitute a better predictor of first-year pharmacy school GPA than either variable alone, and whether various demographic characteristics are related to first-year pharmacy school GPA.

METHODS

The 22 colleges and schools of pharmacy submitting data for this study included the following institutional types and program lengths: 12 public institutions; 10 private institutions; seventeen 2+4 programs (2 prerequisite years followed by 4 professional years); three 3+4 programs (3 prerequisite years followed by 4 professional years); and two 2+3 programs (2 prerequisite years followed by 3 accelerated professional years). The participating programs also represented the following regional representation (by US census regions): Middle Atlantic, 7; South Atlantic, 6; East North Central, 3; East South Central, 2; West North Central, 2; West South Central, 1; Mountain, 1.

For each participating pharmacy college or school, PharmCAS provided spreadsheets containing data for candidates who applied during the year prior to the fall 2008 term, including each candidate's most recent PCAT scores; previously earned cumulative, math, and science GPAs; date of birth, citizenship status, native language, sex, racial/ethnic identity, and parents’/guardians’ highest level(s) of education; the candidate's level (2- or 4-year college) and type (public or private) of school most recently attended, and previous degrees earned. Following the 2008-2009 academic year, each pharmacy college and school submitted performance data for each student who matriculated into a PharmD program for the fall of 2008, including GPAs earned during the first year of pharmacy study and each student's enrollment status at the end of the first year.

Analyses of the sample data included descriptive, correlation, and multiple regression analyses. All analyses involving candidate and student data were conducted using the SAS, version 9.1.3 (SAS Institute Inc., Cary, NC). To accommodate for differences in the distributions of first-year GPAs among pharmacy colleges and schools, GPA data for each school were converted to z scores for all data analyses. Correlation coefficients for each school were converted into Fisher's z values, weighted proportionate to the total number of cases, and averaged to determine an overall Fisher's z value and overall correlation. This method increased the reliability of overall correlations. Pearson product-moment correlation coefficients were calculated to determine the degree to which PCAT scaled scores, writing scores, and entering GPAs related to subsequent first-year GPAs in the pharmacy programs.

Both uncorrected and corrected correlations for PCAT scaled scores were determined. Corrected correlations were calculated using a formula to adjust for range restriction.⁵ These adjustments were made to compensate for using a select sample composed of a subset of individuals who had taken the PCAT and matriculated into a pharmacy program. This restriction of range tends to result in underestimates of the actual relationships. Neither the PCAT writing scores nor the entering GPAs were similarly adjusted because neither represented standardized values.

Multiple regression also was used to analyze the contributions of PCAT scores, previously earned GPA, and demographic variables in predicting subsequent GPAs for students during their first year of pharmacy school. Regression coefficients and standardized coefficient estimates were calculated to allow comparisons among predictor variables.

In the regression models involving PCAT scores and entering GPAs, the regression coefficients indicate either the amount of change in first-year pharmacy GPA predicted for each 1-point increase in a variable (multiple regression, unstandardized coefficients, or the degree of effect on first-year pharmacy GPA predicted for each variable [multiple regression, standardized coefficient estimates]). In the multiple-regression model that used demographic factors as predictor variables, the regression coefficients represent the degree to which each characteristic influenced the outcome variable.

RESULTS

Admissions information collected from the participating schools indicated that the greatest number of prepharmacy credit requirements were in the biological sciences and chemistry, followed by credits in the humanities/liberal arts/general education areas, and English/speech, with fewer required in math and physics and the social/behavioral sciences. The first-year curricula in the pharmacy programs varied somewhat in the types of credits required for the first year, with most of the required credits in the biomedical, pharmaceutical, and clinical sciences. One explanation for some of this variance is that 2 of the participating colleges and schools of pharmacy were 3-year accelerated programs, which typically have different course lengths and may offer some courses during the first year that are not offered until later in traditional programs. Such program differences may have complicated comparisons of student GPA data obtained for this study.

Comparisons of PCAT scores and entering GPAs for candidates applying through PharmCAS to the 22 participating pharmacy colleges and schools during the year prior to fall 2008 with those of students who matriculated into these programs in fall 2008 showed that matriculated students earned higher mean PCAT scores and entering GPAs than did the total group of applicants. Comparisons of mean PCAT scores, entering GPAs, and first-year pharmacy GPAs for all matriculated students by demographic category showed differences in performance between some of the groups.

Table 1 shows the sample sizes and mean PCAT scaled scores, entering GPAs, and first-year pharmacy GPAs by status at the end of the 2008-2009 academic year. These data indicate that the majority of students were still enrolled and were in good academic standing. Table 1 also shows that students who were continuing in good academic standing earned the highest PCAT scores for every subtest except problem solving (writing) and earned the highest entering GPAs and first-year pharmacy GPAs. Conversely, students who discontinued enrollment prior to the second year earned the lowest mean PCAT scores for every subtest except quantitative ability and the lowest first-year pharmacy school GPAs, but earned higher entering GPAs than students continuing on academic probation. Consideration of these comparisons should take into account the small sample sizes for the academic probation and discontinued groups.

Table 1.

Mean PCAT Scores and GPAs by Status at the End of the First Year of Pharmacy Study for Students Entering the Participating Programs in Fall Semester 2008

Academic Characteristics	Still Enrolled		Discontinued Prior to Second Year
	Good Academic Standing	Academic Probation
PCAT Verbal Ability SS	412.3	403.7	399.6
PCAT Biology SS	417.5	406.4	402.8
PCAT Reading Comprehension SS	412.1	404.2	401.3
PCAT Quantitative Ability SS	414.5	403.9	405.4
PCAT Chemistry SS	417.2	404.2	401.5
PCAT Composite SS	414.7	404.6	402.1
PCAT Writing: Conventions of Language	3.03	2.86	2.81
PCAT Writing: Problem Solving	2.95	2.98	2.84
Entering GPA: Cumulative	3.5	3.3	3.4
Entering GPA: Math	3.5	3.2	3.3
Entering GPA: Science	3.4	3.1	3.3
First-Year Pharmacy GPA	3.3	2.4	2.1
n (%)	2,161 (96.3)	57 (2.5)	26 (1.2)

Abbreviations: PCAT = Pharmacy College Admission Test; SS = scaled score; GPA = grade point average; entering GPA = earned prior to admission to pharmacy program; first-year pharmacy GPA = GPA earned during the first year of a doctor of pharmacy program; mean GPA for all first-year students = 3.2 (n = 2244)

Table 2 shows both uncorrected correlations and correlations corrected for restriction of range between independent variables (PCAT scores and entering GPAs) and first-year pharmacy GPA. For the study sample, uncorrected correlations between first-year pharmacy GPAs and PCAT subtest scaled scores ranged from 0.13 for verbal ability to 0.29 for chemistry, with 0.32 for composite and 0.06 for the 2 writing scores. Correlations for entering cumulative GPA, entering math GPA, and entering science GPA were 0.44, 0.28, and 0.44, respectively. When the PCAT scaled score correlations were adjusted for range restriction, the corrected correlations ranged from 0.16 for verbal ability to 0.35 for biology, 0.36 for chemistry, and 0.44 for composite. The r² values indicate the percentage of first-year pharmacy GPA that is explainable by each variable, and thus may represent more intuitive indications of correlation than the r values.

Table 2.

Correlations Between Predictors and GPA in First Year of Pharmacy Program (N = 2,244)

Variable	First Year GPA
	Uncorrected		Corrected
	r	r2	r*	r*2
PCAT Verbal Ability SS	0.13	0.02	0.16	0.03
PCAT Biology SS	0.26	0.07	0.35	0.12
PCAT Reading Comprehension SS	0.17	0.03	0.24	0.06
PCAT Quantitative Ability SS	0.21	0.05	0.28	0.08
PCAT Chemistry SS	0.29	0.09	0.36	0.13
PCAT Composite SS	0.32	0.10	0.44	0.20
PCAT Writing: Conventions of Language	0.06	0.003
PCAT Writing: Problem Solving	0.06	0.003
Entering GPA: Cumulative	0.44	0.19
Entering GPA: Math	0.28	0.08
Entering GPA: Science	0.44	0.19

Abbreviations: SS = scaled score; GPA = grade point average; r = correlation coefficient; r² = the percent of one variable explainable by another variable

All correlations are weighted means of the correlations for the 22 schools combined based on Fisher's z transformations; the corrected correlations were done using the formula of Cohen et al.⁵

Table 3 shows the results of multiple-regression analyses for the sample of enrolled students and the degree to which combined independent variables predicted first-year pharmacy GPAs (R²) and the degree of contribution made by each independent variable (regression coefficients). The data in Table 3 show that when PCAT scores and entering GPAs were considered together (Model 3), they accounted for 25% (R² = 0.25) of the variance in first-year pharmacy GPAs, compared with 10% (R² = 0.10) for PCAT scores alone (Model 1) and 19% (R² = 0.19) for entering GPAs alone (Model 2). The predictor variable regression coefficients indicate which variables had a significant effect. For Model 1, all PCAT scores made significant positive contributions, except for verbal ability, which had a negative effect (ie, predicted a lower first-year pharmacy GPA), and writing scores, which had no significant effect. For Model 2, both GPA variables made positive significant contributions. For Model 3, most variables made significant positive contributions toward predicting first-year pharmacy GPA, except for PCAT verbal ability, quantitative ability, and writing, which did not contribute.

Table 3.

Multiple Regression Analyses for PCAT Scores, Entering GPAs, and Demographic Variables as Predictors of First-Year Pharmacy GPAS

Predictor Variables	Regression Models
	1. PCAT Scores		2. Entering GPAs		3. PCAT Scores + Entering GPAs		4. PCAT Scores + Entering GPAs + All Demo.
	Unst.	Stand.	Unst.	Stand.	Unst.	Stand.	Unst.	Stand.
PCAT Verbal Ability SS	-0.003	-0.07			*	*	*	*
PCAT Biology SS	0.01	0.12			0.01	0.12	0.01	0.12
PCAT Reading Comprehension SS	0.01	0.12			0.01	0.13	0.01	0.11
PCAT Quantitative Ability SS	0.004	0.08			*	*	*	*
PCAT Chemistry SS	0.01	0.17			0.004	0.09	0.01	0.11
PCAT Writing: CL	*	*			*	*	*	*
PCAT Writing: PS	*	*			*	*	*	*
Entering GPA: Math			0.19	0.10	0.19	0.10	0.19	0.10
Entering GPA: Science			0.97	0.37	0.87	0.34	0.95	0.37
Age							-0.02	-0.08
Candidate Earned: At Least Bachelor's							0.28	0.12
All Other Demographic Variables							*	*
R2	0.10		0.19		0.25		0.27

Abbreviations: PCAT = Pharmacy College Admission Test; R² = coefficient of determination; SS = scaled score; CL = conventions of language; PS = problem solving; GPA = grade point average; Age = candidate's age on Aug. 1, 2008; All Other Demographic Variables = student's sex, ethnicity, native language, citizenship, parents' education, previous level and type of school attended; PCAT Scores = all subtest scores combined; Unst. = unstandardized regression coefficients; Stand. = standardized coefficients estimates; All Demo. = all demographic variables available; * = no significant contribution

Writing CL and PS scores were dummy variables set as 1 when greater than or equal to 3.0 (otherwise 0). All demographic variables except age were dummy variables (either 1 or 0).

As shown in Table 3, the combined variables in Model 4 accounted for 27% (R² = 0.27) of the variance. Results for Model 4 also show that PCAT biology, reading comprehension, chemistry, and both entering GPA variables made significant positive contributions. However, the only demographic variables making significant contributions in Model 4 were student age (a negative effect) and whether a student had previously earned a bachelor's degree (a positive effect). These results highlight the value of considering PCAT scores together with entering GPAs and suggest that the only demographic characteristic adding predictive value is whether a student had earned at least a bachelor's degree prior to matriculation.

DISCUSSION

The results of this study generally support and expand on earlier findings regarding predictive validity of PCAT scores. The uncorrected correlation values between PCAT composite scores and first-year pharmacy GPAs shown in Table 2 compare favorably with those from a study that found similar uncorrected correlation values of about 0.30 between the Graduate Record Examination (GRE) general test and first-year graduate school GPAs, and to the 90% credibility interval of 0.32-0.68 determined for PCAT total in a meta-analysis of PCAT predictive validity studies.^1,6

However, these correlations are somewhat lower than those found in a previous study conducted by the publisher of the PCAT.⁷ The higher mean PCAT scores, entering GPAs, and first-year pharmacy GPAs found in the current study may reflect higher admission standards, which would result in greater range restriction and somewhat lower correlations between entering variables and first-year GPAs. Another possible explanation may be found in the results of a 20-year longitudinal study, in which the researchers speculated that progressively lower correlations between PCAT scores and pharmacy school GPAs may be attributable to grade inflation.⁸

The uncorrected and corrected correlations shown in Table 2 for PCAT biology, chemistry, and composite and the uncorrected correlations for entering GPAs all are moderately positive, suggesting the continuing usefulness of each of these variables, particularly the PCAT composite scores and entering cumulative and science GPAs. The relatively higher correlations for PCAT biology and chemistry are not surprising considering the high proportion of credit requirements in the basic biological, pharmaceutical, and clinical sciences for the first year. The lower correlations for the other PCAT subtests may result from the abilities measured by these subtests not being as directly applicable to course content typically offered during the first year. The corrected correlations shown in Table 2 represent the best estimates of how PCAT scores relate to pharmacy GPA across the full range of applicant abilities. While the uncorrected correlations suggest how well PCAT scores predict subsequent GPA among matriculates, the corrected correlations estimate how well the test scores predict the subsequent performance of admission candidates.

Multiple Regression Analyses

As shown in Table 3, the R² values for PCAT scores, entering GPAs, and PCAT scores combined with entering GPAs indicate that 10%, 19%, and 25% of the variance in pharmacy school GPAs is explainable by each of these variables, respectively. These data suggest that PCAT scores combined with entering GPAs have more predictive value than either variable alone.

The R² value shown in Model 1 for PCAT scores compares favorably to the findings of a study that found a multivariate R value of 0.30 (R² = 0.09) between the current version of the Medical College Admission Test (MCAT) and GPAs for the first 2 years of medical school.⁹ The difference in R² values between PCAT scores and entering GPAs is similar to the increase in explained variance (incremental variance) found for the GRE and for other professional and school admission tests, which is typically less than 10%.^6,10

The unstandardized regression coefficients shown in Table 3 indicate the amount of change in first-year pharmacy GPA predicted for each 1-point increase in a variable. The significant positive contributions of PCAT biology and chemistry scores in Models 1 and 3 and the stronger contribution of entering science GPA in Models 2 and 3 are consistent with the correlation findings presented in Table 2, which show that scores on the science-oriented PCAT subtests and the entering science GPAs were the factors most related to subsequent first-year pharmacy GPAs. Although this conclusion is further supported by the focus of the first-year pharmacy course requirements on basic biomedical, pharmaceutical, and clinical sciences, the relatively strong contribution of PCAT reading comprehension scores in Models 1 and 3 also suggests the importance of reading skills in first-year pharmacy school performance.

The standardized regression coefficient estimates shown in Table 3 suggest the comparable effects that variables had on predicting first-year pharmacy GPA. These findings are generally consistent with the unstandardized regression coefficient results shown in Table 3, as well as with the correlation results shown in Table 2, suggesting that science-related variables, a bachelor's degree, and strong reading skills are important influences on first-year pharmacy GPA.

With respect to demographic variables, the regression coefficients shown for Model 4 indicate that students’ age seemed to have a negative influence on first-year pharmacy school GPA, while students’ having a bachelor's degree appeared to have a positive influence. The finding regarding age is consistent with the finding regarding mean first-year pharmacy GPA data for the study sample, which showed that students aged 28 years and older earned slightly lower mean GPAs compared with those in other age groups. The significant positive influence of students’ having a bachelor's degree is consistent with the slightly higher mean first-year pharmacy GPAs for these students than for students without a bachelor's degree. However, the similarities between regression coefficients for PCAT scores and entering GPAs seen in Models 3 and 4 suggest that demographic variables explain little of the variation in first-year pharmacy students’ GPAs.

Some previous studies examining issues related to demographic variables have found sex, ethnicity, and native-language to be associated with performance.^11-14 However, other studies have found comparable performances among students regardless of previous educational background, age, sex, or ethnicity.^15,16 Studies of the influence of educational background have found that the competitiveness and level of college attended prior to matriculation into a PharmD program added predictive value to PCAT scores and entering GPAs only for students’ performance in the fourth year of pharmacy school.^17,18 Several other studies have found that pharmacy students with bachelor's degrees earned higher GPAs during their first year than did students with less prior education. ^19-24

The findings reported in Table 3 are generally consistent with these previous studies, especially for students with a bachelor's degree, except with respect to the negative influence of student age. Regarding educational background data, neither candidates’ last previous attendance at a 4-year college or a private school prior to matriculation nor parents’ attainment of a bachelor's degree were significant factors in predicting first-year pharmacy students’ GPAs.

When interpreting regression statistics, it is important to consider that such information tends to be rather abstract, unintuitive, and easy to misinterpret.⁶ Table 3 shows that the addition of PCAT scores to entering GPAs represents an incremental variance of 6%. While accurate, such conventional portrayals of incremental variance may appear misleadingly small and may not clearly suggest the value that PCAT scores add to entering GPAs. Because the variance figures in Table 3 are uncorrected for range restriction, these results are less about the ability of PCAT and entering GPA to predict whether students will succeed in pharmacy school than they are about predicting which admitted students will do better. Considering that few matriculates fail in pharmacy programs (Table 1), the regression findings shown in Table 3 are less relevant than the corrected PCAT score correlations shown in Table 2 with respect to predicting the likely success of applicants for admission.

This study had several limitations. Although the types of colleges and schools participating in this study were fairly evenly divided (12 public and 10 private institutions), regional representation was less evenly distributed, with 13 from Atlantic coastal areas, one from a Mountain state, and none from Pacific coast areas. This may limit the generalizability of the findings somewhat for schools in Western regions. Further, the participating colleges and schools represented a self-selected sample of programs that use the PharmCAS service rather than a random sample from among all pharmacy programs, which may further limit the generalizability of the findings.

A second limitation relates to possible differences in grading practices among professors and institutions. Because a reasonable alternative approach could not be determined, the data analyses for this study assumed equivalent grading practices within and between institutions with respect to both previously earned GPAs and GPAs earned during the first year of pharmacy study.

A third and persistent limitation in studies such as this one relates to restriction of range. The recent trend among colleges and schools of pharmacy to tighten admission standards has resulted in even less variance in the academic characteristics of pharmacy school matriculates than in the past, which makes unadjusted correlation results less than ideally reliable. Though the researchers did include corrected PCAT scaled score correlations, it was not feasible to adjust PCAT writing scores and entering GPAs to allow for adequate comparisons.

Despite these limitations, the results of this study contribute to the existing literature on the predictive validity of the PCAT by supporting the findings of previous studies and by expanding upon most studies by including analyses involving PCAT writing scores and important demographic characteristics as predictor variables.

CONCLUSIONS

The results of this study are comparable in many ways to previous research findings, showing the moderate validity of PCAT scores in predicting GPAs during the first year of pharmacy school, particularly when considered in conjunction with entering GPAs. This study also found that several demographic characteristics seem relatively insignificant in terms of predicting first-year pharmacy GPAs. This latter point is especially important considering the continuing observation of differing PCAT mean scores and previous GPAs for different demographic groups. This study suggests that PCAT writing scores are not significant as factors in predicting first-year pharmacy GPAs, though they may serve as useful indicators of candidates’ language and communication skills for admission decisions. Small proportions of students discontinued or continued on academic probation after the first year of study. If success in pharmacy school is defined as maintaining good academic standing following the first year, these findings suggest that the criteria being used in admission decisions are appropriate and effective.