Assessment of Communications-related Admissions Criteria in a Three-year Pharmacy Program

Abstract

Objective. To determine if there is a correlation between TOEFL and other admissions criteria that assess communications skills (ie, PCAT variables: verbal, reading, essay, and composite), interview, and observational scores and to evaluate TOEFL and these admissions criteria as predictors of academic performance.

Methods. Statistical analyses included two sample t tests, multiple regression and Pearson’s correlations for parametric variables, and Mann-Whitney U for nonparametric variables, which were conducted on the retrospective data of 162 students, 57 of whom were foreign-born.

Results. The multiple regression model of the other admissions criteria on TOEFL was significant. There was no significant correlation between TOEFL scores and academic performance. However, significant correlations were found between the other admissions criteria and academic performance.

Conclusion. Since TOEFL is not a significant predictor of either communication skills or academic success of foreign-born PharmD students in the program, it may be eliminated as an admissions criterion.

INTRODUCTION

The increasing number of pharmacy students whose native language is not English emphasizes the need to assess proficiency of the English language as part of the admissions process. As of 2012, of the 61 275 students enrolled in a US doctor of pharmacy (PharmD) program, 2.8% were foreign students.¹ Although not all foreign students necessarily have difficulty with English, it can present a communication barrier. Studies demonstrate that students whose second language is English significantly misunderstand essential and commonly used health-related and pharmacy-related vocabulary.^2,3 This can affect their ability to learn pharmacology and to speak and write clearly, and lead to miscommunication.^2,3

Diaz-Gilbert found that students whose native language was not English lacked essential standard English writing skills.⁴ This could affect students’ academic performance and, ultimately, academic success. However, Wu-Pong et al found that English as a second language was not a factor in predicting academic success.⁵ Moreover, all of these studies were from 4-year pharmacy programs. This project does not compare English communication skills of foreign-born students and those U.S.-born students. Instead, this study explores the adequacy of the Pharmacy College Admission Test (PCAT) variables (verbal, reading, essay, and composite) in place of the Test of English as Foreign Language (TOEFL) as an instrument to assess communication skills.

The University of Maryland Eastern Shore (UMES) is a land-grant, historically black university. The UMES School of Pharmacy (SOP) uses a 3-year modular professional curriculum. There are 2 major components of the program: didactic and experiential. The didactic component consists primarily of classroom experiences in the first 2 professional years (P1 and P2). Each module is divided into 2-week blocks, and students take an assessment on the Friday at the end of the block. The information students must learn in the 2 weeks before an assessment can be challenging, especially if the student’s native language is not English.

The experiential component begins in the P1 year, with an 8-hour introductory pharmacy practice experience (IPPE) every other week throughout the P1 and P2 years. The concentrated IPPEs (CIPPEs) occur in December and/or the summer between the P1 and P2 year. Advanced pharmacy practice experiences (APPEs) occur during the P3 year. The experiential component requires that students be able to generate complex documents (eg, care plans, presentations, article critiques) and are thus expected to be able to communicate in writing effectively.

With the trend of moving from a medication-centered role to a more patient-centered one, more responsibility has been placed on pharmacists to be effective communicators with patients, families of patients, and other health care professionals. Written and verbal communication skills are important components of classroom and clinical practice experiences, a further challenge for students whose native language is not English. Further, the Accreditation Council for Pharmacy Education (ACPE) recommends that communication skills be assessed as part of the admissions process, as stated in Guideline 17.3.⁶

In most US colleges and universities, a standardized English language assessment examination, such as TOEFL, is used to assess the English proficiency of foreign-born applicants. English proficiency scores are sometimes used beyond their intended role of evaluating the ability of non-native speakers of English to use and understand English.⁴ The TOEFL score may be misused as an indicator of academic success in the absence of uniform ranking tools. Even in cases where TOEFL is not used as a primary indicator of potential academic success, TOEFL score minimums are frequently imposed as a condition for acceptance. Pharmacy schools also may use the standardized PCAT scores to assess English proficiency.

The PCAT was developed by a joint subcommittee of the American Association of Colleges of Pharmacy (AACP) and The Psychological Testing Corporation in an effort to measure verbal, quantitative, and scientific abilities of students interested in pursuing pharmacy careers.⁷ It was designed to predict academic success during the first professional year.⁸ Several studies have been conducted to determine the correlation of PCAT subtest scores with students’ academic performance, sometimes with contradicting results.^5,9-17 More specifically, PCAT subtest scores, (ie, verbal, reading, and writing scores) are used to evaluate a student’s communication skills. Most of these studies are based on 4-year pharmacy programs.^4,8-16 However, data collected from 22 different pharmacy programs, including two 2+3 programs (2 prerequisite years followed by 3 accelerated professional years), showed that PCAT writing and verbal scores were not significant predictors of first-year GPA.¹⁴ On the other hand, PCAT reading comprehension was found to be a significant factor in predicting first-year GPA, suggesting the importance of reading skills in first-year pharmacy school performance.¹⁵ Many PharmD programs, including the UMES program, require the PCAT as a measure of general academic ability and scientific knowledge necessary for the commencement of pharmaceutical education, regardless of country of origin.¹⁸

At the UMES School of Pharmacy, the TOEFL is used to gauge the communication skills of all students born outside of the United States, and is an admissions requirement for all foreign-born applicants regardless of their native language. It is requested that students submit their TOEFL scores before the application deadline. However, applicants who meet the eligibility criteria but have not yet taken TOEFL, may be invited for a campus interview, a half-day process that includes written and analytical assessment and individual interviews by 4 faculty members. If students interview well and are extended an offer, they must submit the TOEFL results before starting the program. If TOEFL scores are received after the interview, a competitive TOEFL score may be an additional deciding factor in cases where the student is waitlisted. Waitlisted candidates are students who were not considered to be the most ideal candidates after their interview and whose offers of admission were withheld pending further consideration. Additionally, the PCAT is an admissions requirement for all applicants at the school. The biology, chemistry, and composite scores (including verbal, reading, and writing) are used to determine interview eligibility. Unlike TOEFL, which is an English-proficiency examination, the PCAT tests proficiency in areas required for pharmacy.

The cost of TOEFL can range from $160 to $250, and the cost for PCAT is approximately $200.^19,20 Feedback from foreign-born applicants suggested that the costs associated with TOEFL may be a prohibitive factor for applicants choosing to apply to the UMES program vs schools without this requirement. Since TOEFL is a requirement for higher studies in the United States for foreign-born students, it is an initial unavoidable expense. However, TOEFL results are only valid for a limited period of time, and the elimination of the TOEFL requirement at the school is an attempt to relieve students from having to retake it and incur additional expenses. The TOEFL is also potentially culturally biased (ie, a student with a knowledge of American culture has a decided advantage, both practically and psychologically).¹⁸ However, this bias was disproved by a comprehensive study using data from 2 groups of examinees, those who tested in the United States (ie, consisted of people who lived in the country for more than one year) and those who tested in their native countries. Test items included no specific references to a particular culture.²¹ The objectives of our study were to determine if there was a correlation between TOEFL and other admissions criteria that assess communications skills (ie, PCAT variables: verbal, reading, essay, and composite), interview, and observational scores and to evaluate TOEFL and the these admissions criteria as predictors of academic performance.

METHODS

This investigation used a retrospective record review of the first three class years of PharmD students at the school (classes of 2013, 2014, and 2015). Data were from admissions records collected during the student application process for the three classes. Variables used in this study were scores from the TOEFL, PCAT , interviews, as well as observational scores. Measurements of academic performance included cumulative biweekly assessment, IPPE, CIPPE, and APPE raw scores. The raw score for didactic and experiential performance was reported on a numerical scale from 0 to 100 rather than as a letter grade. Student data were entered into an Excel spreadsheet and then converted for analysis using SPSS,v22. Basic descriptive statistics including means, standard deviations, and percentages were calculated. The statistical analyses for parametric variables (ie, two sample t test, multiple regression, and Pearson’s correlations) and for nonparametric variables (ie, Mann-Whitney U test) were conducted at a 95% confidence interval. The students were excluded if they had incomplete data for the analyses conducted. This study was approved by the UMES Institutional Review Board.

Didactic assessments were given at the end of a 2-week block. The content of the biweekly assessment was proportional to the time spent on a particular subject during that 2-week block. Individual assessments were followed with a team assessment using the same questions. An additional 5 percentage points were added to an individual’s assessment score if his or her team scores at least 95% on the team assessment. To provide an accurate interpretation of results, the individual first attempt scores of the biweekly assessments were used without team assessment scores.

For practice experiences, students were evaluated by preceptors using the IPPE, CIPPE, and APPE evaluation forms. During the P1 year, students took IPPEs I and II and CIPPEs I and II. For the P2 year, students took IPPEs III and IV. The IPPEs linked key concepts in the P1 and P2 curriculum with contemporary pharmacy practice. The students spent an 8-hour day in a pharmacy setting every other week to experience patient-care activities and pharmacy operations. During the alternating week, students participated in guided discussions about the previous week’s learning objectives to strengthen the learning experience. Students took the CIPPEs between the fall and spring terms, and during the summer between the P1 and P2 years. The CIPPEs were designed to help students make the successful transition from the classroom to the practice environment via practical application of concepts learned. The integration of classroom knowledge and experiential training serves as the cornerstone of students’ education, while instilling professionalism and ensuring competency in the provision of pharmaceutical care.

Concentrated IPPEs reviewed the basic technical and distributive functions of pharmaceutical care in the institutional and retail environments by providing students with opportunities to expand their knowledge base, practice their skills, and develop professional attitudes in an actual pharmacy setting. The experiential component culminated in APPEs in the P3 year. The APPE rotations started in the summer after the P2 year and comprised the third academic year. Students took eight 5-week rotations: 4 were required, and 4 were elective courses. For the study, final scores from the preceptor evaluations were used.

The PCAT is composed of 240 multiple-choice questions and 2 writing assignments that must be completed in 4 hours. The PCAT is divided into 7 subtests, issued in the following order: writing (part I), verbal ability, biology, chemistry, writing (part II), reading comprehension, and quantitative ability. The writing responses are judged in terms of problem-solving skills and the effective use of language conventions. The PCAT composite score for the combined multiple-choice subtests includes reading, verbal, chemistry, biology, and quantitative.

From June 2007 through January 2011, PCAT writing scores were reported for conventions of language (essay) and problem solving, along with mean scores indicating the averages of all writing scores earned by candidates during a given test administration.¹⁸ The PCAT essay score provides insight into the student’s advanced skill in sentence formation, usage, or mechanics. The PCAT problem-solving score provides insights into the student’s effective composition, clear organization, clear development of ideas, persuasive presentation of ideas, and advanced critical-thinking skills. We were interested in students’ ability to compose an original essay and, thus, only considered the PCAT essay score and not the PCAT problem-solving score for the writing assessment. Percentile ranks were used for PCAT reading, verbal, and composite. An earned score assigned by trained scorers was used for the PCAT essay (1-6).²²

The TOEFL measures the ability of non-native English speakers to use and understand the English language as it is heard, spoken, read, and written in the university classroom. The test is given in English and administered electronically. There are 4 sections: listening, reading, speaking, and writing. Scores are based on the student’s performance and are reported as 4 scaled section scores and a total score. For the study, students’ total score (0-120) was used. Individual pharmacy schools set their own score requirements for acceptability.²⁰

The last 2 scores were interview and observational. Student candidates are invited to the campus for a half-day interview, which includes an individual interview by 4 interviewers, comprised of faculty members, staff, and/or preceptors. Students are evaluated using an interview rubric. The interview questions assess students’ ability to meet the school’s communication technical standards. Once completed, students receive an average normalized interview score (0-25). The campus interview also includes an ethical dilemma presentation for which students will receive an average observational score from two raters, comprised of faculty members and/or staff. The student’s observational score is determined using an ethical dilemma scoring rubric. The score is based on presentation skills and is highly subjective. For the ethical dilemma activity, students are grouped as a team. The team discusses an ethical dilemma situation and comes up with a solution to the problem, followed by a group presentation of their work. Each student typically will have a designated section to present (eg, statement of the problem), which is then followed by a question-and-answer component. For the study, the observation score (1-5) was used.²³

RESULTS

Table 1 shows the demographic characteristics of foreign-born and US-born students in the study. A majority of the foreign-born students’ race was black (65%). Foreign-born black students were from the African countries Cameroon, Nigeria, Ethiopia, Ghana, and Kenya. A fourth of the foreign-born students were classified as “other” (ie, Asian or Hispanic) (26%), and a small percentage was white (9%). Almost half of US-born students were white (47%); a third were black (33%) and a fifth were classified as other (ie, Asian, Hispanic, or Native American) (20%).

Table 1.

Demographic Data of Foreign-born and US-born Students

Table 2 provides a comparison of the foreign-born and US-born students’ standardized test scores interview scores, observational scores, and academic performance. After performing a 2-sample t test on the parametric variables, foreign-born students were found to have significantly lower PCAT reading and verbal scores. Further, after performing a Mann-Whitney U test on the nonparametric variables (PCAT essay and observational scores), observational scores were found to be significantly lower for the foreign-born students. There was no significant difference in the didactic and experiential scores between the 2 groups. The relationship between the 6 independent variables (PCAT variables: verbal, reading, essay, and composite, interview, and observational scores) and TOEFL was examined, and the results are summarized in Table 3. This model was significant (p=0.001).

Table 2.

Two Sample t Test and Mann-Whitney U Test Comparing Variables between Foreign-born and U.S.-born Students

Table 3.

Multiple Regression Analysis for Predictors on the TOEFL

Table 4 summarizes results from the analysis that examined the influence of the admissions variables on didactic scores. Pearson’s correlation coefficient r and p values were identified. For foreign-born students, there was a significant correlation between PCAT reading (r=0.29, p<0.05) and composite scores (r=0.33, p <0.05), with P1 didactic scores. There was also a significant correlation between PCAT composite and P2 didactic scores (r=0.35, p<0.01). For US-born students, there were significant correlations between PCAT composite score with both P1 (r=0.24, p<0.05) and P2 didactic scores (r=0.23, p<0.05). Similarly, when the 2 groups were combined, there were significant correlations between PCAT composite score with both P1 (r=0.25, p<0.01) and P2 didactic scores (r=0.24, p<0.01).

Table 4.

Correlations Between Admissions Variables And Didactic Scores

Table 5 summarizes results from the analysis that examined the influence of the admissions variables on experiential scores. For foreign-born students, there was a significant correlation between PCAT essay score and P2 scores (r=-0.39, p<0.05). For US-born students, there were significant correlations between interview scores and P1 scores (r=0.32, p<0.01), observational scores and P1 scores (r=0.35, p<0.01), PCAT composite and P2 scores (r=-0.25, p<0.05), PCAT essay and P2 scores (r=0.24, p<0.05), interview scores with P2 scores (r=0.25, p<0.05) and with P3 scores (r=0.31, p<0.01). When the 2 groups were combined, there were significant correlations between interview score and P1 scores (r=0.26, p<0.01), PCAT composite and P2 scores (r= -0.23, p<0.05), and interview score and P3 scores (r=0.23, p<0.05). The PCAT essay and PCAT composite had a negative correlation with experiential performance among foreign-born and US-born students, respectively.

Table 5.

Correlations Between Admissions Variables And Experiential Scores

DISCUSSION

Assessment of Communications-Related Admissions Criteria

Students’ communication abilities are essential for success in a pharmacy program and, eventually, as professional pharmacists. For example, in a survey report from 78 pharmacy schools Chesnut and Phillips indicated oral and written communication skills were important preprofessional attributes of pharmacy students.²⁴ Further, students should be able to read at a level necessary to comprehend the complex textbooks and literature required in pharmacy school. The reading ability levels required for pharmacotherapy textbooks have been shown to be higher than students’ reading abilities. Filler et al found that the reading ability of P3 students in a 4-year program was lower than the reading level of the textbook.²⁵ Therefore, it is important that schools include in their admissions criteria methods to assess students’ communication abilities, particularly with regard to the increasing number of foreign-born students applying to pharmacy schools.

As indicated in other studies, English can pose communication issues for non-native speakers, but it may not be a factor in predicting their academic success.^2-4 To assess the school’s admissions criteria for communication abilities, several analyses were conducted to suit the data for valuable interpretations. The PCAT reading and verbal scores were significantly different between the 2 groups of students (Table 2).

For foreign-born students, the PCAT reading and verbal scores were significantly lower than native speakers’ scores. However, foreign-born students performed equally well in the didactic and experiential curriculum. Carroll and Garavalia suggested higher levels of intrinsic and extrinsic motivation may compensate for the lower verbal and reading comprehension skills, allowing nonwhites to be equally successful academically. Nonwhites in their study included black, Asian, African, Hispanic, Middle-Eastern, and Native American.²⁶

There was no significant difference in PCAT essay and interview scores between the 2 groups. However, the observational scores between the groups were significantly different, with a lower score observed for foreign-born students. The variability in the observational scores may be attributed to students’ presentation skills, level of confidence in communicating in English, proficiency of English, and/or accent. For the interview scores, the interview rubric did not quantitatively measure verbal communication skills. The students were rated based on the nature of their answers. However, students’ communications skills may have affected their rating on each question item on the interview rubric. Further, if students had excellent verbal communication skills, it was usually noted in the comments box as a strength. The evaluation for the interview score was subjective and dependent on the interviewer’s interaction with the student. For the academic performance of the groups, there was no significant difference between the didactic and experiential scores.

Since the school uses multiple standardized tests and interview scores to evaluate the students’ communication skills, an analysis was conducted to examine the relationship of the PCAT variables and interview scores on the TOEFL, which was chosen as a dependent variable because this standardized test is not specific to the pharmacy school. The multiple regression analysis (Table 3) indicated that PCAT, observational, and interview scores, are associated with TOEFL (p=0.001). This indicates that the measure of a student’s overall communication abilities may be assessed collectively by PCAT (verbal, reading, essay, and composite), ethical dilemma (observational score), and interview, as well as by TOEFL. This result further indicates that the PCAT results could be used to assess communication skills, and that the TOEFL administration is possibly redundant as an admissions criterion.

Predicting Didactic and Experiential Performance

The utility of the 7 variables in predicting academic performance in the didactic and experiential components of the curriculum was evaluated using correlations analyses. The PCAT composite score was the only variable that positively correlated with all didactic scores for all students. Particularly for the first year, this result may have been a reflection of the science-based nature of the P1 curriculum. For example, the first course offered in the P1 year, incorporates organic chemistry, fundamentals of drug action, principles of cell biology, and biochemistry. Hence, as an admission criterion, PCAT composite is a strong predictor of students’ academic success at the school. The PCAT reading score positively correlated with only P1 didactic scores for foreign-born students. Since the PCAT reading score correlated with TOEFL, PCAT reading scores could replace TOEFL as a predictor of the foreign-born students’ P1 didactic performance. This result was consistent with Meagher et al, who demonstrated the importance of reading skills in first-year didactic performance.¹⁵

However, when considering all students, all other variables were not significantly correlated with P1 didactic performance. The measure of academic performance is the didactic assessment scores. The school’s assessments use multiple-choice examinations, which may not be the best method for students to demonstrate their verbal and written communication skills. Additionally, our results indicated that TOEFL was not significantly correlated with P1 didactic performance for foreign-born students. This provides additional support that PCAT composite and reading scores are significantly more accurate predictors for didactic performance than TOEFL scores.

On the other hand, the PCAT composite score had no significant correlation with the P2 experiential performance of foreign-born students and had a negative correlation with the P2 experiential performance of the US-born students (Table 5). The PCAT composite score was not a significant predictor of the P3 experiential performance for either group. The effect on P3 experiential performance is consistent with findings by Kidd et al for a 4-year pharmacy program.¹³ Additionally, Meagher et al reported that the predicting validity of PCAT composite scores decreased for each subsequent program year.¹⁵ This may be a result of the increasing clinical nature of the curriculum, and few clinical content areas are assessed by PCAT.¹⁵ The practice experience is not merely knowledge recall, but also application, which is not assessed by PCAT.²³

The PCAT composite score had no significant correlation with P1 experiential performance for either group. Similarly, there were no significant correlations between PCAT reading and verbal, and experiential performance. This is consistent with Meagher et al, who found that PCAT verbal was not a significant predictor of first-year to fourth-year GPAs in a 4-year program.¹⁵ The PCAT essay score had a negative correlation with P2 experiential performance for foreign-born students, indicating that lower PCAT essay scores may result in lower P2 experiential performance. However, the PCAT essay positively correlated with P2 experiential performance for US-born students. When all students were considered, PCAT reading, verbal, and essay scores showed no significant correlation with experiential scores. In addition, the TOEFL score was not significantly correlated with either experiential or didactic performance.

The observational score was only significant in predicting P1 experiential performance of US-born students. However, when all students were considered, observational scores did not have predictive validity on the experiential scores. The interview scores were not significant predictors of experiential performance for foreign-born students. However, interview scores were significant predictors of P1, P2, and P3 experiential performance for US-born students. When all students were considered, the interview scores maintained its predictive validity on P1 and P3 experiential performance. This result suggests that interview scores can predict which students will likely be successful in the experiential component of the curriculum. A highly subjective tool may be used, therefore, to determine qualified students and to identify students who may potentially be successful in the program, particularly in the experiential component. This result is consistent with the findings by Hardigan et al.¹²

Our results suggested that TOEFL was not a significant predictor of didactic and experiential success for either group. Further, the PCAT variables were not significant predictors of experiential success for all students. However, the interview score was a significant predictor of P1 and P3 experiential success for all students. None of the PCAT variables, interview, or observational scores were significant predictors of experiential success of foreign-born students. However, PCAT composite and reading were significant predictors of didactic performance of the foreign-born students.

Because most of the school’s foreign-born students are of African descent, further studies evaluating other factors that may predict academic performance of this population are needed. Kellow and Jones suggested African-American students generally performed poorly on standardized tests compared with their white counterparts, which may explain our findings for the foreign-born students.²⁷ Moreover, other studies show standardized tests, such as the Law School Admissions Test and the Graduate Record Examination, may underestimate intellectual ability of African-American students.^28,29 We found that TOEFL was not a predictor of academic performance and could be eliminated as an admissions criterion at the UMES School of Pharmacy.

Future Research and Study Limitations

An area for future research could be on additional factors (eg, extracurricular activities, age) that may predict overall academic performance of foreign-born students, particularly those of African descent. Another possibility would be to survey views and perceptions of pharmacy students and faculty members on the utility of the TOEFL. Further investigation on the impact of the interview process on the students’ experiential success also is needed.

Most foreign-born students were from countries where English is commonly used, even though it may not be the native language. The assumption that US-born and foreign-born students raised in the United States have a good command of the English language was a limitation. The findings may not be readily generalized to other institutions with respect to student demographics and program requirements. The assessment scores used were based on multiple-choice questions and limit an evaluation of the students’ verbal and writing abilities.

CONCLUSION

Verbal, reading, essay, and composite PCAT scores, as well as interview and observational scores significantly relate to TOEFL scores, suggesting that the PCAT is an adequate tool to evaluate students’ communications skills. However, the interview score is a significant predictor of experiential success. No significant correlation was found between TOEFL scores and academic and experiential performance. In particular, TOEFL was not a significant predictor of foreign-born students’ academic success in the school’s program. Therefore, PCAT results may be used to assess communication skills, making TOEFL results redundant as an admission criterion.