Pharmacy Students' Perceptions of Their Preparedness to Provide Pharmaceutical Care

Abstract

Objective

To assess students' perceptions of their preparedness to perform advanced pharmacy practice competencies.

Design

The Preparedness to Provide Pharmaceutical Care (PREP) survey was modified and administered to each class at a Midwestern university from 2005-2008. Factor analysis and 1-way ANOVA with multiple comparisons were applied to assess the effectiveness of changes made in the pharmacy curriculum.

Assessment

Factor analysis yielded patterns similar to those reported in the literature. Students rated themselves highest on the psychological aspects and lowest on the administrative aspects of care. Perceived pharmaceutical care skills grew as students progressed through the curriculum, and changes in coursework were reflected in the competencies.

Conclusion

Students’ perceived competencies (ie, communication, psychological, technical, administrative) were similar to those at other institutions and perceptions of competencies increased in a manner consistent with actual program outcomes.

INTRODUCTION

Assessment of student learning outcomes is a high priority for colleges of pharmacy and is mandated by Accreditation Council for Pharmacy Education (ACPE) Accreditation Standard 15 as essential for quality improvement of the curriculum.^1-5 Assessments must be employed systematically and sequentially throughout the curriculum, be both formative and summative in nature, and be employed in both didactic and experiential curricula. Data must be collected from a variety of sources including comprehensive knowledge examinations, licensure examination pass rates, direct observation of students in laboratory settings, objective structured clinical examinations (OSCEs), and student performance in introductory or advanced pharmacy practice experiences (IPPEs/APPEs).²

Aggregate survey data collected from individual students over time can be useful as one form of feedback, even if the data reflect indirect assessments.⁶ In some cases, researchers have used annual surveys of graduates to effectively detect trends over time in student perceptions of the curriculum⁷ and the adequacy of preparation in a problem-based learning curriculum.⁸ Perhaps the most detailed and rigorous analysis of this type was conducted by Ried et al, who developed an instrument called Perceptions of Preparedness to Provide Pharmaceutical Care (PREP). This survey asked students to provide their opinions regarding their ability to perform advanced pharmacy practice competencies at the end of each academic year.⁹ The instrument can be used to examine differences between pharmacy classes within the same academic year, or follow a single cohort over the 4-year professional curriculum. Most importantly, it can examine whether increases in students PREP scores occur at the expected times for a given curriculum.

To date, few colleges of pharmacy have used the PREP survey in a systematic manner for evaluating their curricula. Still fewer, if any, have tied those evaluations to the timing of specific curricular changes. North Dakota State University (NDSU) has a unique opportunity to utilize the PREP survey in both of these manners. Starting in 2003, the NDSU College of Pharmacy initiated several curricular changes, including the formation of a concept pharmacy (practice laboratory), and IPPE and APPE.The PREP survey was subsequently administered for several years, allowing for the collection and analysis of both cross-sectional and longitudinal outcomes data.

The first 2 years of NDSU's PharmD program are focused on the pharmaceutical (basic) sciences, including biochemistry, immunology, pathophysiology, pharmaceutics, pharmacokinetics, and pharmacodynamics (pharmacology, medicinal chemistry). Pharmacy practice coursework, including pharmacotherapy courses and pharmacy law, is concentrated in the third year of the program. The fourth year comprises the APPEs.

Although NDSU adopted the PharmD curriculum in 1989, the curriculum continues to undergo revision as new concepts and materials are added, and other material is consolidated or dropped from the curriculum. Several major changes to the curriculum were initiated in 2003-2004. The first major change was the deletion of the pharmaceutical sciences compounding laboratory and the addition of the concept pharmacy for the first 3 years (P1 through P3, respectively) of the curriculum. This change became operational with the 2003 cohort. In the concept pharmacy, students are exposed to dispensing, compounding, sterile products, selected nonprescription products, dispensing software, several automated dispensing machines, telepharmacy, and patient counseling. The concept pharmacy is an onsite pharmacy that is registered and licensed by the North Dakota Board of Pharmacy.

A second major change implemented in 2003 was the pharmaceutical care sequence over the first 3 years of the PharmD curriculum. This 6-course sequence integrated health care delivery, socio-behavioral models, nonprescription drugs, dietary supplements, herbal therapy, communication and counseling skills, ethics, immunization, cardiopulmonary resuscitation (CPR), and pharmaceutical care. The content in the concept pharmacy was integrated with that covered in the pharmaceutical care sequence concept pharmacy and the pharmaceutical care sequence and also to integrate IPPE into the curriculum.

The third major change occurred in 2006, with the addition of an IPPE to each of the first 3 years. Initially, students were required to complete 28 hours of IPPEs per year. In 2008, the IPPE was expanded to the full 100 yearly hours for P1s in 2008, with the P2s in 2009, and P3s in 2010.

In 2005, the pharmacy practice management course (P2) year was revised to place greater emphasis on basic financial management, human resources, and communications skills. Further refinements have been added since then; for example, a business plan on pharmaceutical care was added in 2007.

By 2005, students had participated in the course sequence for the first 3 years (P1-P3), and thus were able to evaluate longitudinally the effectiveness of the curriculum in enhancing their pharmaceutical care skills. As such, an opportunity existed to use the PREP survey to evaluate, both cross-sectionally and longitudinally, the effectiveness of those changes from the perspective of NDSU pharmacy students. The objectives of this study were to:

• (1) Apply factor analysis to the NDSU PREP data and compare the factor structure with Ried et al's PREP survey results.⁹ This represents an initial attempt to generalize the results beyond their original population.

• (2) Within the same academic year, examine differences in all 4 professional classes on the same competency to examine how students’ perceptions of skills vary based on their current experience and location in the curriculum. The study also examined student cohorts by year and professional class to determine whether improvement in students’ PREP occurred at expected times, given the curricular content (ie, pharmaceutical care course sequence, Concept Pharmacy).

DESIGN

Survey Design

A study was conducted to understand better the impact that the pharmaceutical care course sequence was having on students, particularly how this experience prepared them to perform advanced pharmacy practice competencies. To achieve this, the investigators used a slightly modified version of Ried's Preparedness to Perform Pharmaceutical Care (PREP) survey.⁹ Ried's survey items were based on the CAPE curricular outcomes for the PharmD degree developed by AACP. He also added 2 items to reflect the research mission of the University of Florida College of Pharmacy, and these items were retained in the NDSU survey. The modified NDSU PREP survey contained 38 items, comprising 5 socio-demographic items and 33 items covering various aspects of pharmaceutical care, all of which were drawn directly from the Ried survey. Of the 33 pharmaceutical care items, 14 items cover the technical aspects of pharmaceutical care, 9 items cover psychosocial aspects, 4 items address communications aspects, 4 items cover administrative aspects, and 2 items address research aspects.

The original survey instrument contained 41 items on various aspects of pharmaceutical care; however, 8 items did not load on a factor or loaded on multiple factors. These items were excluded from the remainder of the empirical analyses and were deleted in 2003 from the NDSU PREP instrument. These 8 items were: evaluation of the legitimacy of a prescription; integration of the basic and clinical sciences for dispensing decision; provision of professional, ethical, compassionate care; provision of emergency care; promotion of public awareness of health; compliance with federal laws; use of data/computers in professional practice; and, assumption of leadership positions. Instead of the 7-item Likert scale that ranged from poor (1) to excellent (7) preparation used by Ried, the NDSU form used a 5-item Likert scale that ranged from poor (1) to excellent (5) preparation, so that survey responses could be recorded and aggregated using a standard Scantron (Eagan, Minnesota) form.

Hypotheses and A Priori Expections

Because the curriculum changes were new, and because the PREP survey was confined to self-reported perceptions of pharmacy students from a different demographic profile than those used by Ried, there was little a priori information upon which to base hypotheses. As such, the approach used in this analysis was exploratory in nature. There are no prior expectations about how the survey items will load in the factor analysis, nor are there any prior expectations about whether and how those latent factors vary by class or by year. As such, factor analysis is applied to the entire data set. These factors are subsequently analyzed under the null hypothesis that no differences exist by year or by class. If the null hypotheses are rejected, it is possible to go further using post hoc analyses to characterize these differences and by extension, examine some of the ramifications of the NDSU college of pharmacy's curriculum changes.

Survey Administration

The NDSU PREP instrument was successfully pilot-tested on the P1 and P4 classes in 2004. Based on these results, the assessment committee decided to expand the administration of the PREP survey to all years of the curriculum and to monitor students longitudinally as they progressed through the new curriculum. In 2005, the P3 and P4 students completed the survey instrument near the end of the spring semester and the P1 and P2 students completed the survey instrument early in the fall semester. Between 2006-2008, all classes (P1, P2, P3, and P4 students) completed the survey instrument annually, near the end of the spring semester. A summary of these scores can be found in the NDSU College of Pharmacy Assessment Committee Annual Technical Report for 2005-2008, available from the lead author upon request. Since individual responses were not identifiable, the lead author applied for, and was granted, exempt status from North Dakota State University's Institutional Review Board.

Data Analysis

All data was scanned and entered into an Excel spreadsheet, and the data analysis was conducted using SPSS (Chicago, Illinois) version 16.0. The a priori probability for significance was p < 0.05. Mean scores and standard deviations were determined for each item. Exploratory factor analysis was applied to all 33 items using principal component extraction with varimax rotation.^9-11 Items in the rotated factors were considered significant and included in scales using an a priori value of 0.40. Student responses for the 4 years of assessment results were analyzed cumulatively for all responses, and subsequently for each class within each year. For the latter, a one-way analysis of variance (ANOVA) with multiple comparisons using the Scheffe post hoc test was utilized.

EVALUATION AND ASSESSMENT

Table 1 summarizes the respondent characteristics. The data span 4 professional classes of 85 students per class over 4 years, resulting in 340 potential observations per professional year and 1,360 total potential observations. Thus, the 974 responses represent a 71.6% response rate. For the first 3 years of the curriculum, there was a similar number of students that completed the survey instrument, with average participation rates in excess of 75%. However, the number of P4 students who completed the survey was significantly lower than the other classes (149 responses out of 340, or 43.8%). Having a suitable time and date to adminster the survey to students who were on APPEs accounted for this problem. Given that the P4 response rate was still within an acceptable range (ie, above 40 %) and the exploratory nature of the analysis (which intends to analyze differences across the factors by year and class, and thus disaggregates P4 students from the remainder of the sample), it is reasonable to include the P4 students in the analysis. Nonetheless, the discrepancy in response rates is a limitation of this study, and future work should be conducted in this area.

Table 1.

Characteristics of Pharmacy Students Who Participated in a Longitudinal Survey to Determine Their Preparedness to Deliver Pharmaceutical Care

More females (61.8%) than males (38.2%) completed the survey instrument, which was consistent over the 4 years, and consistent with enrollment demographics (64.1% female and 35.9% male) over the same time period. Most of the class completed prepharmacy requirements in 2 years (54.8%), followed by 3 years (29.3%) prepharmacy, and 14.4% had a bachelor's degree before admission to the PharmD program. Many students worked in a pharmacy setting and the proportion of these students increased as they proceeded through the curriculum. About a third listed experience in a chain pharmacy (32.4%), followed by independent (22.2%), and hospital (16.1%) settings. Approximately a quarter (23.5%) of students reported not working in a pharmacy. Students were also asked in which pharmacy setting they planned to work in the future. Most reported chain pharmacy (39.9%), followed by independent (25.7%), hospital (22.8%), and other pharmacy career paths (10.4%). Only 1.2% reported an expected career path outside of pharmacy (eg, attorney, medicine).

Table 2 reports students’ ratings (means and standard deviations) of their perceived preparation to perform pharmaceutical care. Comparing the 4 factors, students rated themselves highest on the psychological aspects and lowest on the administrative aspects. Within the psychological area, the highest rated item was “impact of values in professional interactions, followed by “apply ethical theories to professional interactions.” All of the administrative area items were rated low, and the lowest rated item was “manage fiscal and human resources.”

Table 2.

Aggregated Students Rating of Their Preparedness to Perform Pharmaceutical Care (PREP) (n=974)

^aScale: 1 = poor preparation to 5 = excellent preparation

Table 3 provides the results obtained from applying factor analysis to the entire data set (objective 1). As indicated by the heuristic measures at the bottom of the table, the data were amenable to factor analysis. The ratio of the sample size to the number of variables was 29.52, implying that a sufficient number of observations existed to compute all necessary correlations and extract the latent factors (ie, eigenvalues). The Kaiser Meyer Olkin (KMO) measure was 0.97 (acceptable measures range from a minimum of 0.7 to a maximum of 1.0) and Bartlett's test of spericity (statistic value = 29,055.694, p < 0.001) rejected the null hypothesis of no joint correlation across the variables at a 5% significance level.

Table 3.

Factor Analysis of Students’ Preparedness to Perform Pharmaceutical Care (PREP)

^aItems excluded from a factor because they loaded on multiple factors.

Table 3 summarizes the loadings identified by the factor analysis. The results were similar to those reported by Ried, except for the 2 research items that loaded onto multiple factors (in Ried's data these items loaded solely onto the technical factor). As described by Ried, the first factor comprised 14 items that focused on the technical aspects of pharmaceutical care including “recommend appropriate drug therapy,” “integrate knowledge for pharmacotherapy,” and “provide counseling to patients.” The second factor included 9 items with an emphasis on the psychosocial aspects of care such as “impact of values in professional interactions,” “apply ethical theories to professional decisions,” “gather information to resolve a problem,” and “understand practice related to changing societal expectations.” Composed of 4 items, the third factor focused on the communcation aspects of care, particularly “respond to an information request from a patient,” “communicate medical record information to patient,” and “communicate medical record information to a health professional.” The fourth factor included 4 items that focused on the administrative aspects of pharmacy including “evaluate, select, and purchase pharmaceuticals,” “manage fiscal and human resources,” and “perform drug use evaluations and formulary.” Taken jointly, these 4 factors explain 70% of the variation in the data set.

Following established methodology, these latent factors were characterized by taking a simple average (mean) across all items that load significantly onto a single factor, and thus created factor scales. Because the research items exhibited consistent and significant factor loading values, but did not load highly onto a single factor, the researchers decided to average these 2 values and identify research as a distinct latent process. Doing so allows interested readers to make inferences about student perceptions regarding research without reducing the generality of the analysis, since under normal circumstances these variables would be excluded from the creation of any latent factor. Thus, readers can simply ignore the results for the research factor if they choose. Cronbach's alpha scores for the technical (0.970), psychological (0.917), communication (0.916), administrative (0.911), and research (0.868) scales all indicate a high degree of internal consistency across each of the survey items contributing to the creation of that specific factor scale.

Table 4 reports the average score for preparation of PREP scale factors at the end of each professional year (objective 2). Overall, the psychological and communication areas were tied for the highest total average, followed by the technical, research and adminstrative areas. Graduating (P4) students’ highest total average self-assessed level of competence was in the area of communication, followed by psychological, technical, research, and administrative competencies. Graduating P4s scored on a Likert scale (1 = poor preparation to 5 = excellent preparation) were typically in the high 3 (average) to 4 (good) range for most items. The lowest rated area was administrative aspects and suggests that management should be a targeted area for improvement. At the end of the first pharmacy year (P1), the psychological area was rated highest, followed by communication, technical, research, and administrative. The upward trend of scores were reported from the beginning to the end of the curriculum for each factor and also for the year within each factor. As expected, the P1 competency scores were substantially below those for the P4 students and represent a preparation level at a poor or fair level in most areas. Additionally, the levels of preparation improved for most items from the P1 year through the P4 year. As the students in each class continued their coursework and experiential training, their self-reported competency levels also improved.

Table 4.

One-way ANOVA and Scheffe Post Hoc Tests of PREP Factors

Abbreviations: P1 = first professional year; P2 = second professional year; P3 = third professional year; P4 = fourth professional year.

Table 4 also reports on the one-way ANOVA with multiple comparisons of the PREP factors. The F-test for each of the 5 factors (technical aspect F-statistic = 107.7; psychological aspect F-statistic = 18.7; communication aspect F-statistic = 22.4; management aspect F-statistic = 23.6; research aspect F-statistic = 31.6) was significant (p ≤ 0.0001). As measured by the Scheffe post hoc test, most of the mean differences were also significant (p ≤ 0.0001). The overall mean difference from the P1 to the P4 year was significant for most years within factors. The 3 exceptions were the psychological aspects (2005 and 2008) and the administrative aspects (2006 class). Overall, there was a significant increase in PREP scores over the 4 years of the curriculum. This table also reported the mean differences from year to year, and for most of these comparisons, the mean differences were not significant. For 2 of the 4 years (2005, 2007), technical aspects had a significant improvement from the P1 through P3 years. During 1 year (2008), communication aspects had a significant improvement from the P1 year to P2 and from the P3 to P4 year.

The previous paragraphs present results based on a typical respondent's class standing in the program (P1-P4) or the year in which a response was provided, and thus are solely cross-sectional or time-series in nature. The results in Table 4 can also be used to follow specific cohorts of students (by reading the diagonal elements in Table 4) as they progressed through the 4 years of the program. This allows for a cohort (or panel) analysis of findings (the corollary to objective 2). The majority of these cohorts reported similar perceived competency levels, indicating that students exhibited similar perceptions, and that the formation of these perceptions changed in a stable and predictable fashion as each cohort moved through the program. Two notable exceptions are the cohorts that were P1 and P2 students in 2005. These cohorts exhibited lower levels of perceived technical competencies, which subsequently “caught up” to other cohorts’ perceived competencies as they reached the P3 and P4 years. To a lesser extent the 2005 P2 cohort exhibited lower values for the psychological, communication, and administrative aspects of pharmaceutical care as well. As with the technical competencies, these perceptions matched those of other cohorts by the P3 year. In all likelihood, these latter findings were attributable to discrepancies in the survey's administration. In 2005, P1 and P2 students were surveyed early in the fall, instead of near the end of the spring semester. As such, the lower perceptions were due likely to the fact that students in 2005 had not completed as much coursework as other cohorts, and thus did not have as much (perceived) mastery over each of these competencies.

DISCUSSION

Two major objectives were included in this analysis. The first objective was to replicate Ried's study⁹ to determine whether (and how) the formation of self-assessed pharmaceutical care competencies for students at NDSU differed from those at other universities, and more specifically, at the University of Florida. The formation of PREP was consistent across the 2 colleges. For example, in both colleges the graduating students’ perceptions improved over the 4 years of the professional curriculum. In both colleges, student perceptions generally improved where expected, given the placement of course content. Thus, the PREP was found to measure consistently these examples and others across both colleges. Consequently, limited evidence exists to support the generalizability and validity of the instrument. The 1 small difference that exists concerns perceptions of research-related competencies. In the Ried study, these items loaded on the technical factor, while in this study these items loaded relatively evenly on the technical and psychological factors, and ultimately were treated as distinct from these factors. This discrepancy is attributable to differences in the culture and mission across the 2 schools. The University of Florida's College of Pharmacy is a larger institution, with a greater variety of graduate programs and a greater emphasis on research and scholarship. Concomitantly, the NDSU College of Pharmacy is a much smaller school with greater emphasis on the professional program and a less intense research mission. As a result, students may have less exposure to research-based competencies – either directly through a paucity of research-based courses or indirectly through faculty members who place less emphasis on scholarly activities and less interaction with other graduate/professional students – and thus perceive their research competencies differently than those at larger schools.

Our second objective was to examine whether these latent factors and perceived student competencies exhibited significant cross-sectional variation based on a student's year (P1 through P4) in the PharmD program. The results suggest that student perceptions of competency generally increased over the course of the program; P4 students generally had higher perceived competencies than P1 students in most instances. To the extent that perceptions mirror reality, this implies that the curriculum was successful at teaching students pharmaceutical care skills. Moreover, students appeared to gain perceived competencies in a manner that coincides with the nature of the curriculum and its recent changes. For example, P1 and P2 students exhibited low mean technical competency scores, and those scores increased dramatically during the P3 and P4 years, when students shifted from basic science to applied pharmaceutical care content. Similarly, while students showed gains in administrative competencies over each of the 4 years, those gains were not as great in the P3 and P4 years. This is consistent with the nature of the coursework, since students take their only required course in pharmacy management during their second year.

A corollary to the second objective was to determine whether there was any longitudinal variation in perceived competencies which corresponds with the changes the college of pharmacy made in its curriculum. Again, some evidence was found to support this. For example, the concept pharmacy was initiated in 2003, and its course work was further refined in 2006. The 2006 academic year also marked the introduction of IPPEs. In both cases, the intent was to allow students to practice and enhance their compounding, dispensing, and counseling skills, especially for P1 and P2 students with limited knowledge of pharmaceutical care processes. Between 2005 and 2008, mean technical perceived competencies increased among P1 and P2 students. Moreover, for P1 students, perceived competencies in each of the 5 factors appeared to rise between 2005 and 2006, and fall thereafter. P2 students also exhibited mean gains in perceived psychological and communication competencies, especially between 2005 and 2006. After this time, the gains appeared to level off. P3 and P4 students did not exhibit significant perceived gains over time (Table 4). While these changes may appear discouraging, they indicate that the curriculum changes were effective because P1 and P2 students take their basic pharmaceutical science coursework, and only have limited exposure to the practice of pharmaceutical care. Thus, P1 and P2 students may think that they are more knowledgeable about the practice of pharmacy than they actually are. The introduction of the concept pharmacy coursework and IPPE may help cure students of these false pretenses, thereby reducing perceived competencies. Once students begin to learn actual competencies, their perceptions begin to improve as they progress through the curriculum. Additionally, it was noted earlier that the P1 and P2 students were surveyed early in the fall of 2005, and then again near the end of the 2006 academic year. Thus, these students had a longer timeframe than the other cohorts (approximately 18 months versus 12 months) during which to gain these perceived competencies.

Unfortunately, the longitudinal assessment also identified a curriculum change that was not effective at increasing perceived competencies. Students consistently ranked administrative competencies as the lowest of the 4 perceived competencies (or 5, if research is included as a distinct competency). Students exhibited a reasonable gain between the P1 and P2 years, but this gain did not rise consistently between the P1 and P3 years, and in 1 case (2006) between the P1 and P4 years. Moreover, this pattern was relatively consistent between 2005 and 2008, despite changes to the Pharmacy Management coursework. Thus, more assessment, and potentially more course content modification, must be done to ensure that students’ actual and perceived competencies in this functional area are improved.

The most glaring limitation of this analysis was the emphasis on perceived, as opposed to actual, pharmaceutical care competencies. Boyce reported that self-assessments have not correlated well with performance,⁶ and students commonly rate themselves higher than they are able to perform,¹² particularly those with lower performance, or when assessing patient communication and interaction abilities.¹³ Self-assessment information is also used for program assessment. Pharmacy work programs have asked students to self-assess their ability to meet curricular outcomes during and at the end of the academic program.^6,9,13 These self-assessments can be aggregated to determine which curricular outcomes are perceived to be met or not met and where these curricular outcomes are developed.^9,13 While self-assessed perceptions are certainly important – students must understand and have confidence in their pharmaceutical care skills – it is equally important to match what students think they know about their skills with what they actually know about pharmaceutical care. Much work remains to be done in this area, both by pharmaceutical assessment researchers in general, and the NDSU COP in particular.

Another limitation was the nature of the data collection process. While response rates among P1 through P3 students were quite high, response rates among P4 students were much lower, which could potentially skew the study's findings. Moreover, P1 and P2 students in 2005 were surveyed in a different (longer) timeframe than the others, which could potentially impact findings. Replications of this study at NDSU, as well as other comparable institutions that have a balanced set of responses by professional year, would allow for a broader, more comprehensive, and unbiased analysis of perceived pharmaceutical care competencies.

A third limitation is that the researchers modified Reid's survey instrument to facilitate its administration and encourage higher response rates. The NDSU survey did not contain 8 items that were originally included in Reid's study, but were later excluded by him, because they did not load significantly on any factor. The scales in the NDSU survey were also shortened from a 7-point Likert scale to a 5-point scale to allow the responses to be recorded on a standard Scantron form. Future studies using the PREP survey (whether the form used by Reid or that used at NDSU) are necessary to ensure that the modifications used in this study do not impact the generalizability of its findings.

A final limitation is that analysis suggests that not all curriculum changes were effective. Much work needs to be done to strengthen students’ perceived and actual administrative competencies. Course content and the timing of that content may have impacted this study. The college may want to consider adding more administrative content to the P3 and/or P4 years, either through additional required content, additional electives, or possibly through additional APPE opportunities.

CONCLUSION

This study examined the formation of perceived pharmaceutical care competencies of students in a public PharmD program in the upper Midwestern United States. An attempt was made to tie these perceptions to those of other PharmD programs, and to actual program outcomes and curriculum changes at the college in question. Students’ perceived competencies were found to be similar to those at other institutions. Perceived pharmaceutical care skills grow in a logical fashion as students complete their coursework, and changes in that coursework can impact both actual and perceived pharmaceutical care competencies.