Examining the Association of GPA and PCAT Scores on Objective Structured Clinical Examination Scores

Abstract

Objective. To examine the association between certain demographic and admission measures and Objective Structured Clinical Examination (OSCE) performance in a cohort of pharmacy students.

Methods. A retrospective review of demographic characteristics, admissions data [cumulative and science admission Grade Point Average (GPA), Pharmacy College Admissions Test (PCAT) scores], and OSCE scores was performed for the Class of 2017 at the University of Tennessee College of Pharmacy.

Results. Female students scored significantly higher than male students on the Warfarin OSCE – Standardized Patient (SP) rated General Communication Skills and on the Warfarin OSCE – Faculty rated Patient Interviewing Skills. Age was significantly, inversely correlated with Warfarin OSCE – Faculty rated Therapeutic Knowledge score. Warfarin OSCE – SP rated General Communication Skills score was significantly, positively related to PCAT composite score and PCAT reading comprehension score. PCAT composite score was significantly, inversely correlated to Warfarin OSCE – Faculty rated Patient Interviewing Skills score. Warfarin OSCE – Faculty rated General Communication Skills score was significantly, positively related to cumulative admission GPA and admission science GPA.

Conclusion. Eight statistically significant correlations were found between demographic and admissions measures and specific OSCE scores. Regression models were significant but explained a low percentage of the variance in OSCE scores, suggesting other factors not included in the study have a greater effect on scores. Such factors may include knowledge acquired in pharmacy school courses such as the therapeutics course series.

INTRODUCTION

The Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards and Key Elements for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree (Standards 2016) outline that admissions processes should identify “students who have the potential for success in the professional degree program and the profession.”¹ The American Association of Colleges of Pharmacy Special Committee on Admissions recommends a holistic approach to pharmacy admissions, using measurements of both cognitive and non-cognitive abilities in the admissions process.² To assess pre-pharmacy competency, Pharmacy College Admission Test (PCAT) scores and admission grade point averages (GPAs) continue to be the most frequently used measures.² Prior studies have found that PCAT scores and admission GPAs may be moderate to strong predictors of student performance in PharmD didactic course grades and GPA, as well as on the North American Pharmacist Licensure Examination (NAPLEX).^3-5 Pharmacy programs that use Multiple Mini-Interviews (MMI) as part of their holistic admissions process have found a correlation between MMI performance and academic difficulty in the PharmD curriculum.⁶ These factors may also have value in predicting clinical performance, as indicated by Objective Structured Clinical Examinations (OSCEs).

While there is no set standard for measuring success in the pharmacy profession after professional licensure, successful practice of the pharmacy profession may be heavily influenced by a student’s performance in the clinical activities as part of the pharmacy curriculum. OSCEs serve as one type of clinical performance measure and are standardized patient interactions designed to assess student competence in therapeutic knowledge as well as non-cognitive skills necessary for success in practice. Meszaros and colleagues found that OSCE performance during the pharmacy curriculum correlated with performance on advanced pharmacy practice experiences (APPEs).⁷

If a relationship exists between standard cognitive admissions measures, demographics and OSCE performance, admissions committees may use this information in their holistic admissions processes to assist in selecting students who are likely to perform well in APPE experiences as well as in clinical practice. Research evaluating this relationship between standard cognitive admissions measures or a student’s demographic parameters and OSCE performance in pharmacy students is lacking. In one of the few available published studies, McLaughlin and colleagues found only weak relationships between admissions data and OSCE and APPE scores.⁸ The medical literature in this area is also inconclusive. Kulatunga-Moruzi and Norman found undergraduate GPAs to have the most utility in predicting academic and clinical performance in the 20 station OSCE portion of the Medical Council of Canada’s Licensing Examination in medical students, while MCAT verbal reasoning scores were shown to be effective in predicting communication skills in the same OSCEs.⁹ However, Peksun and colleagues found scores from a 10 station OSCE for second year medical students were not correlated with cognitive admissions variables.¹⁰

To address this noted gap in the pharmacy education literature, the purpose of this study was to examine the association between standard cognitive admissions measures, student demographic parameters and OSCE performance in a cohort of pharmacy students.

METHODS

A retrospective review of official student records was conducted for the Class of 2017 at the University of Tennessee College of Pharmacy. The study was approved by the University of Tennessee Health Science Center Institutional Review Board.

During Spring 2016, the following data were collected from student records by a study coordinator (as a quality control measure, data entry was checked by a study investigator): demographic characteristics (gender, race/ethnicity, age); admissions data (cumulative admission GPA, admission science GPA, composite and individual subset PCAT scores); and communication and therapeutic knowledge scores, as rated by faculty and standardized patients (SPs), from OSCEs. Cumulative admission GPA and admissions science GPA were calculated by PharmCAS (American Association of Colleges of Pharmacy, Arlington, VA) and represent all coursework that a student had completed at the time of application. The admissions data included in this study represent the data reflective of academic ability routinely reviewed by the admissions committee upon application.

The scores for two OSCEs were collected. The first OSCE was conducted during an Interprofessional Education and Clinical Simulation course (IPECS) in the P2 year (Fall 2014). This OSCE was focused on warfarin treatment and communication with a difficult patient. Students were rated on their ability to provide in-depth counseling on a newly prescribed high-risk medication to a hurried patient. In this OSCE, faculty scored students on their general communication skills (0-40 points), patient interviewing skills (0-10 points), and therapeutic knowledge (0-24 points), while SPs scored students on their general communication skills (0-15 points). SPs evaluated students from the patient’s perspective, while faculty evaluated students from the content expert perspective.

The second OSCE was conducted during an IPECS course in the P3 year (Fall 2015) and was focused on medication therapy management (MTM) in a patient with several comorbid conditions and multiple medications. Prior to this assessment, the OSCEs consisted of patients with one or two medical conditions and only two or three medications. Students were rated on their ability to perform a comprehensive medication review using strategies used in the Pharmacists’ Patient Care Process (PPCP).¹¹ In this OSCE, faculty scored students on their general communication skills (0-24 points) and therapeutic knowledge (0-22 points).

The rubric used for the warfarin case was adapted from the American Pharmacists Association rubric used for the National Patient Counseling Competition for assessment of communication and patient interviewing skills.¹² The MTM OSCE rubric was created using principles outlined in the PPCP.¹¹ Total points available for each section were determined by the main focus of each OSCE and the faculty coordinating. The warfarin OSCE was more heavily focused on communication, as compared to therapeutic knowledge, since it takes place early during the third didactic semester and students have had less therapeutics content in the curriculum. Whereas the MTM OSCE was equally focused on communication skills and therapeutic knowledge as it occurs later in the fifth didactic semester after most of the therapeutics courses sequence was completed. Standardized patients were trained by a single faculty member on the blueprint of the patient profile for each OSCE case. Standardized patients did not participate in the scoring of this OSCE due to the nature of the assessment and the dual use of the OSCE as part of the requirements toward completion of an MTM training certificate program. Faculty were provided a standardized evaluation rubric and to decrease the risk for inter-rater reliability, they collectively graded a few students and discussed the evaluation process as a group. A pre-brief outlining the objectives, timing and activities to take place was conducted prior to each OSCE to adequately prepare students for the OSCEs. The warfarin OSCE was 10 minutes in length or less, whereas the MTM OSCE was 20 minutes in length due to the nature of the case being a simulated MTM session.

Statistical analyses were performed using IBM SPSS Statistics 22.0 (Armonk, NY). Categorical variables were summarized using frequencies and continuous variables were summarized using means, standard deviations, and medians. Independent samples t-tests were conducted to examine the relationships between gender, race/ethnicity, and OSCE scores. Pearson’s r correlations were calculated to determine the associations between age, PCAT composite and subset scores, cumulative undergraduate GPA, science GPA, and OSCE scores. Standard multiple linear regression analysis was conducted for those OSCE score variables significantly associated with two or more independent variables in the bivariate analysis. Gender, race/ethnicity, age, PCAT subset scores, and cumulative admission GPA were the independent variables in each regression analysis (PCAT composite score and admission science GPA were not included due to multicollinearity). To conduct each standard regression analysis, variables were entered in one block simultaneously. The a priori significance level was .05.

RESULTS

Of 170 students admitted to the Class of 2017, data were collected for 166 (97.6%). Of the four students excluded from the analysis: one foreign transfer student was excluded because admissions data were not available; one student was moved to the Class of 2018 due to poor academic performance before having the opportunity to take the OSCEs under study; one student was dismissed prior to Fall 2014; and one student withdrew prior to Fall 2014. The mean age of the students was 26.1 years of age. The majority of students were female (61.4%) and white (67.5%). Table 1 displays a summary of students’ admissions factors. Mean cumulative admission GPA for the Class of 2017 was 3.36 (0.35) and mean composite PCAT score was 65.2 (19.5).

Table 1.

Summary of Class of 2017 (N=166) Admission Factors

Table 2 displays a summary (means, standard deviations, and medians) of the six OSCE scores. Statistically significant differences based on gender were noted on two of the six OSCE scores. Female students scored significantly higher than male students on the Warfarin OSCE – SP rated General Communication Skills [14.4 (1.3) versus 13.6 (2.2), respectively; p=.02]. Female students also scored significantly higher than male students on the Warfarin OSCE – Faculty rated Patient Interviewing Skills [9.4 (1.6) versus 8.9 (1.7), respectively; p=.03]. No statistically significant differences were noted based on race/ethnicity for any of the six OSCE scores.

Table 2.

Summary of Objective Structured Clinical Examination (OSCE) Scores for the Class of 2017 (N=166)

In the correlational analysis (Table 3), age was significantly, inversely correlated with Warfarin OSCE – Faculty rated Therapeutic Knowledge score (Pearson’s r=-.18, p=.02). Warfarin OSCE – SP rated General Communication Skills score was significantly, positively related to PCAT composite score (Pearson’s r=.16, p=.05) and PCAT reading comprehension score (Pearson’s r=.19, p=.02). PCAT composite score was significantly, inversely correlated to Warfarin OSCE – Faculty rated Patient Interviewing Skills score (Pearson’s r=-.16, p=.04). Warfarin OSCE – Faculty rated General Communication Skills score was significantly, positively related to cumulative admission GPA (Pearson’s r=.22, p=.01) and admission science GPA (Pearson’s r=.20, p=.01). No other significant correlations were found between age, admissions factors, and OSCE scores.

Table 3.

Correlations Between Age, Pharmacy College Admission Test (PCAT) Scores, Grade Point Average (GPA), and Objective Structured Clinical Examination (OSCE) Scores

Regression analysis was conducted for the following OSCE variables: Warfarin OSCE – SP rated General Communication Skills score, Warfarin OSCE – Faculty rated Patient Interviewing Skills score, and Warfarin OSCE – Faculty rated General Communication Skills score. The final model for Warfarin OSCE – SP rated General Communication Skills score included gender and PCAT reading comprehension score (F=7.4, p<.01), and explained 7.2% (adjusted R²=.072) of the variance in SP rated OSCE score (gender alone explained 4% of the variance). In the multiple linear regression analysis for Warfarin OSCE – Faculty rated Patient Interviewing Skills score, no independent variables were significant in the final model (p>.05). In the multiple linear regression analysis for Warfarin OSCE – Faculty rated General Communication Skills score, only cumulative undergraduate GPA was included in the final model (F=8.1, p=.01), explaining 4.2% (adjusted R²=0.042) of the variance in the Faculty rated General Communication Skills score. Refer to Table 4 for a summary of the final regression models for Warfarin OSCE – SP rated General Communication Skills score and Warfarin OSCE – Faculty rated General Communication Skills score.

Table 4.

Multiple Linear Regression Final Model Summaries for Objective Structured Clinical Examination (OSCE) Scores

DISCUSSION

In the current study, the SP rated General Communication Skills score for the Warfarin OSCE was found to have two positive correlations, between the PCAT composite score and reading comprehension subscore. Female students were also noted to have significantly higher scores. The faculty rated General Communication Skills score on the same OSCE was found to have two positive correlations, between the cumulative admission GPA and admission science GPA. Additionally, female students had significantly higher scores on faculty rated patient interview skills on the same OSCE. These results suggest that certain admissions factors, namely PCAT composite score, PCAT reading subscore, cumulative admission GPA and admission science GPA, as well as gender, may predict success on communication and interviewing OSCE scores. These findings support the work by McLaughlin and colleagues and together suggest the need to explore these correlations further with additional student cohorts.⁴ Significantly higher OSCE scores in female pharmacy students compared to male pharmacy students are a new finding that also warrant further exploration in pharmacy student cohorts.

Interestingly, two significant but inverse correlations were found in this study: one between age and the Warfarin – Faculty rated OSCE score specific to therapeutic knowledge and the other between PCAT composite score and the Warfarin – Faculty rated OSCE score specific to patient interviewing skills. Many different reasons may exist for these findings. Currently, there are no published studies that demonstrate a strong correlation between age and OSCE scores. Perhaps older students have had less exposure to active learning activities before pharmacy school and are less comfortable with OSCE type assessments. Similarly, there are no published studies that demonstrate a strong correlation, either positive or negative, between PCAT scores and OSCE scores. Patient interviewing skills can be considered a non-cognitive trait, whereas PCAT scores are normally considered to be a better measurement of cognitive traits. It would not be surprising then to see no correlation between these measurements. The finding of a negative correlation may suggest that some students who score lower on cognitive measurements may score higher on non-cognitive measurements. Evaluation of larger datasets than those used in this study may help identify other causes for these inverse correlations. Additionally, it would be interesting to see if correlations exist between OSCE scores and student performance on MMI in programs that use MMI as part of their admissions process.

Of note, all significant correlations found involved the Warfarin OCSE. There were no significant correlations found in the analysis of OCSE scores from the MTM case. This result may be interpreted as the Warfarin OSCE served as a better differentiator of student performance than the MTM OSCE did. Students overall may have found the pharmacist/patient dynamics in the MTM OSCE to be “easier” than that of the Warfarin OSCE due to the cooperation of the SP in the MTM OSCE. Alternatively, this result may be interpreted as a need in our curriculum to spend more time on motivational interviewing techniques as the uncooperative patient may have affected the students’ ability to collect information to make their assessment.

In our study, regression analysis modules explained only 7.2% of the variance in SP rated OSCE score and 4.2% of the variance in the Faculty rated General Communication Skills score. Although significant, the models explain a low percentage of the variance in score. This suggests other factors not included in the study have a greater effect on OSCE scores. Such factors may include knowledge acquired in pharmacy school courses such as therapeutics. Future studies should explore additional factors, including both cognitive and non-cognitive variables that may impact OSCE scores. A better understanding of such variables may assist pharmacy programs as they prepare students for clinical/experiential aspects of the curriculum in the short-term and pharmacy practice in the long-term.

This study has limitations. Despite the training for faculty evaluators, there is always a risk of low inter-rater reliability. Because this study relied on retrospective data, causality cannot be established. Further, generalizability of the findings is limited as this study was conducted at one college of pharmacy. However, demographics of the Class of 2017 were similar to US pharmacy Class of 2017 enrollment as of Fall 2016, providing support for more widespread consideration of the results.¹³ Additionally, each OSCE included in our study consisted of only one encounter, not multiple encounters. The results found from our use of this formative assessment may not be generalizable to programs that use multiple encounter OSCEs or use OSCEs as high-stakes summative assessments.

CONCLUSION

As admissions committees work to identify students who have the potential for success in their pharmacy programs, it continues to be important to understand how traditional cognitive admission measurements translate into actual performance in the pharmacy curriculum. The frequency of OSCE use in the pharmacy curricula varies widely among programs, and the correlations between cognitive admission measurements and OSCE performance are still weak. Our study suggests that a correlation exists, although the regression analysis shows that the criteria studied only accounted for small percentages of the variances observed. Additional analysis with larger student cohorts may better define these correlations. Until strong correlations are proven, admissions committees should not consider those cognitive admission measurements to be strong predictors of OSCE scores in the pharmacy curriculum. While our study did not evaluate correlations between non-cognitive admission measurements, such as MMI, and OSCE scores, further work in this area may provide additional useful information to predict future success on OSCEs in the pharmacy curriculum.