Assessment of Critical Thinking in Pharmacy Students

Abstract

Objective

To determine whether changes occur over 1 academic year in pharmacy students' critical thinking skills and disposition to think critically.

Methods

First, second, third, and fourth-year pharmacy students completed the California Critical Thinking Skills Test (CCTST) and the California Critical Thinking Disposition Inventory (CCTDI) at the beginning and end of 1 academic year.

Results

One hundred thirty-seven students completed the study. No significant changes occurred over the year in total scores on either instrument. However, scores in 3 of 12 subscale scores changed significantly and several significant correlations were found.

Conclusion

Pharmacy students' scores on 2 critical thinking instruments showed no major improvements over 1 academic year but most scores were above average. Some areas of possible weakness were identified. Additional studies comparing scores over a longer period of time (eg, admission to graduation) are needed.

INTRODUCTION

Critical thinking has been identified as an essential outcome of a pharmacy school education.

The Commission to Implement Change in Pharmaceutical Education in 1994, stated: “Entry level graduates must be able to examine issues rationally, logically, and coherently…since much of professional practice is problem-solving, students need to develop analytical skills to make decisions in both familiar and unfamiliar circumstances.”¹

Norris and Ennis have described critical thinking as “reasonable and reflective thinking that is focused upon deciding what to believe or do.”² Kurfiss added that “in critical thinking all assumptions are open to question, divergent views are aggressively sought, and the inquiry is not biased in favor of a particular outcome.”³ The adequacy of critical thinking training in pharmacy schools' curricula has been questioned. An American College of Clinical Pharmacy (ACCP) White Paper called for renewed attention to outcomes such as critical thinking and their integration into the training of future pharmacists.⁴

Many colleges and schools of pharmacy have responded by implementing changes to better promote desired outcomes. One noteworthy example has been the implementation of problem-based learning (PBL) into many curricula,⁵ which some believe fosters critical thinking skills.⁶ The ability to assess whether desired curricular outcomes are being achieved provides important feedback.

Consenting students enrolled in the Campbell University School of Pharmacy (CUSOP) completed assessment instruments designed to measure critical thinking skills and the disposition to use critical thinking. The primary objective of this study was to determine whether changes in critical thinking scores occurred between the beginning and end of the 2005-2006 academic school year. Secondary objectives included determining whether: (1) differences existed among the 4 student subject groups in scores, (2) correlations existed between instrument total scores and measures of achievement eg, selected Pharmacy College Admissions Test (PCAT) scores and grade point averages, and (3) the use of these instruments might be of value in the future for assessment. This study was significant in that it provided an evaluation of critical thinking using surrogate measures which had previously never been used at the School.

Campbell University is a private institution in North Carolina. The School of Pharmacy was established in 1986 as a doctor of pharmacy (PharmD) program. For analysis the students were placed in subject groups based on their year in the program, ie, first year (P1), second year (P2), third year (P3), or fourth year (P4).

METHODS

This research was approved by the Campbell University Review Board, Copernicus Group IRB. Informed consent was required of all subjects. The study period was the 2005-2006 academic year. A prospective, pretest/posttest design was utilized. The population for this study were the CUSOP students.

Two previously validated instruments were used in this study: the California Critical Thinking Skills Test, 2000 (CCTST) and the California Critical Thinking Disposition Inventory (CCTDI). ^7-11 These instruments have been used in several pharmacy studies.^12-17 The CCTST measures critical thinking skills and provides a total score and scores for 5 subscales (Analysis, Evaluation, Inference, Deductive Reasoning, and Inductive Reasoning). The CCTDI measures the willingness or disposition of an individual to engage in critical thinking. This instrument provides a total score and scores for 7 subscales (Truth-seeking, Open-Mindedness, Analyticity, Systematicity, Critical Thinking Self-Confidence, Inquisitiveness, and Cognitive Maturity).^8-12,14

The 2 instruments were administered initially to consenting P1, P2, and P3 students at the beginning of the fall 2005 semester. However, P4 students first completed the instruments in May 2005, immediately before the majority of these students left campus for the beginning of advanced pharmacy practice experiences (APPEs). The final administration of the instruments for all 4 student groups took place at the end of the spring semester 2006.

Due to scheduling conflicts, the instruments could not be administered at the same time; however, all 4 subject groups completed the CCTST and CCTDI within a 2-week period. Most subjects required 45-60 minutes to complete both instruments.

PCAT scores and school grades were obtained through the Office of the Dean with consent of the subjects. All PCAT scores used in this study were scaled scores, not percentiles. PCAT scores from test administration prior to the change in the PCAT norms were recalculated using formulas obtained from Harcourt Assessment (San Antonio, Texas) to represent the adjusted scales.

At CUSOP, the composite, biology, and chemistry PCAT scores are given emphasis by the Admissions Committee. As a result, these scores were used to study possible correlations with instrument total scores along with grade point average (GPA). SPSS 16.0 for Windows (SPSS, Inc., Chicago, IL) was used for data analysis. Because of the differences in times at which the subjects took the PCAT, correlation coefficients were calculated for each group separately rather than for all the subjects together. The predetermined alpha level was 0.05.

RESULTS

The baseline data of the participants are presented in Table 1. One hundred fifty-four students provided informed consent and began the study. The majority of subjects were female with no significant differences in gender distribution between subject groups and their respective classes (p<0.05). Overall participation in the study ranged from 51.3% of the P1 class to 24.7% of the P4 class. More P3 subjects had bachelor's degrees than would have been predicted from the number in their respective class. One hundred thirty-seven students completed both phases of the administration of the CCTST and CCTDI. Data from matched pairs were used for analysis.

Table 1.

Baseline Information on Pharmacy Students Participating in a Study to Determine Changes in Critical Thinking Over One Academic Year

^aNo significant differences (p < 0.05) between groups and their respective classes in age or prior years of college.

^bNo significant differences between the P1, P2, and P4 groups and their respective classes in attainment of prior college degree (p < 0.05). A significant difference was found between the P3 group and their respective class.

^cPrior GPA for P1 students = prepharmacy GPA; prior GPA for P2, P3, and P4 students = pharmacy GPA immediately prior to study period. Cumulative GPA = cumulative pharmacy GPA at end of study period. No significant difference was found among the study groups or between the study groups and their respective classes in GPAs (p < 0.05)

No significant differences were found between the subject groups and their respective classes in GPA prior to the study period or cumulative GPA at the end of the study period.

The possible ranges for instrument scores are reported in Table 2. Paired t test analyses are presented in Table 3 (total scores) and Table 4 (subscale scores). No significant changes in scores between the beginning and end of the academic year occurred in total scores for either instrument. Of 12 possible subscale scores, significant (p < 0.05) changes occurred in 3 (Table 3): Inference (P3 subjects' scores decreased), Open-Mindedness (P4 subjects' scores decreased) and Self-Confidence (P2 subjects' scores increased). Inference is a subscale of the CCTST. Open-mindedness and Self-Confidence are subscales of the CCTDI.

Table 2.

Possible Ranges of Instrument Total Scores and Subscales⁹,¹¹

Abbreviation: CT = critical thinking

Table 3.

Changes in Pharmacy Students' Critical Thinking Total Scores^a,^b

Abbreviations: CCTST = California Critical Thinking Skills Test; CCTDI = California Critical Thinking Disposition Inventory

^aAlpha = 0.05; none of the comparisons were significant (p > 0.05).

^bOne way ANOVA detected no significant difference (p > 0.05) among the 4 subject groups for initial or final CCTST and CCTDI scores.

Table 4.

Changes in CCTST and CCTDI Subscale Scores

One-way ANOVA revealed no significant differences among the 4 subject groups in either the initial or final total test scores for the 2 instruments. However, 4 significant CCTST subscale differences were detected among groups and are noted in Table 4. No significant differences were found among subject groups in CCTDI subscale scores.

Several significant correlations are noted in Table 5. A consistent finding was correlation between composite PCAT and CCTST scores. Composite PCAT scores were correlated (p < 0.05) with both initial and final CCTST total scores in the P1, P2, and P4 groups and with the initial CCTST total score in the P3 group. The CCTDI scores had fewer significant correlations.

Table 5.

Pearson Correlation Coefficients of Pharmacy Students' CCTST and CCTDI Scores With Demographic Variables

DISCUSSION

Although no significant changes occurred in total scores during the study, the majority of scores were favorable. CCTST total scores from subject groups were comparable to mean scores of 17-21 reported by others.^12,14,17 The mean CCTST score from an aggregate sample of 4-year college students (non-health science majors) was 16.8 (median 16).¹¹ Miller¹² and Phillips et al¹⁴ both reported that median subject scores in their studies exceeded the college comparative group median of 16. For both CCTST initial and final scores, 70% of all CUSOP subjects scored greater than 16.

Miller¹² and Phillips et al¹⁴ reported significant increases in CCTST total scores in their studies when the same cohorts of students were followed from admission to graduation. In our study, scores were compared before and after 1 academic year only. Miller¹² did present yearly data for several classes but no significant yearly changes were indicated. However, a trend toward higher scores was found as students proceeded through the curriculum.

The influence of a “ceiling effect” limiting changes in CCTST is a possible factor. Phillips et al¹⁴ found that students who scored at or above 16 on an initial administration of CCTST did not experience a significant change in their subsequent score; however, those students who initially scored below 16 had a significant increase in their second score. The possibility of a limiting or ceiling effect in studies of medical students and residents also has been described.^18,19

Some similarities exist between Phillips et al¹⁴ findings and the CUSOP study. The mean initial total score of CUSOP subjects whose scores were already at or above 16 decreased slightly over the study period: 22.3 vs. 21.9 (p = 0.326, n = 104). Subjects whose pretest scores were below 16 had an increase in their second score, though the change was not significant: 13.5 vs. 14.5 (p = 0.084, n = 33).

No changes in CCTDI total scores over 1 academic year were found with CUSOP subjects, though significant increases were found by Phillips et al¹⁴ but not by Miller.¹² Over 75% of all CCTDI total scores at CUSOP were above a value of 280. Scores below 280 are considered indicative of overall weakness in willingness to use critical thinking.⁹ CUSOP total scores were in the same range as mean scores, which have been reported by others.^12,14,17

CUSOP subscale scores overall were also comparable to scores reported in other studies and to comparative norms.^9,12,14 All but one of the CCTDI subscale mean scores were in the 40-49 range in this study and in others.^12,14 This range (40-49) is considered positive. The Truthseeking subscale scores though were in the 30-39 range for all 4 CUSOP subject groups. Truthseeking is the willingness to pursue the truth even if it might be different than one's own preconceived beliefs.⁹ The 30-39 range is indicative of ambivalence.

Similar ranges for CCTDI subscales have now been found in at least 3 different pharmacy studies of students at different stages of a pharmacy curriculum. All but the Truthseeking scale have been in the 40-49 range. The Truthseeking scores might be “normal” for pharmacy students. Perhaps curricula are limited in the extent to which this trait can be influenced. Individual maturity and professional experience may be more significant factors in influencing this score. Or it may be an indication of curricular weakness in developing this trait. As more experiences with these instruments are published, a better picture can be obtained of what is reasonable to expect from students and from a curriculum. The availability of test scores from practicing pharmacists may also provide an additional view of critical thinking and a valuable source of data for comparison with students' scores. It is difficult to discern the significance of the subscale changes. The majority of subscale scores (9 of 12) did not change significantly, and most changes in scores that did occur were positive. The 3 changes occurred in 3 different subject groups. Future studies with these instruments hopefully can provide more insight.

If a presumption is that students' critical thinking scores as measured by these instruments should steadily improve each academic year, then a conclusion of this study might be that the CUSOP curriculum in some way may be deficient. This possibility cannot be ignored and continuous curriculum assessment is critical. However, similar research which found significant changes compared scores over a longer time period.^12,14

If the effect of a curriculum on critical thinking is cumulative, perhaps there may be delayed effects on scores from instruments such as the CCTST and CCTDI. Instrument scores after 1 semester or 1 academic year may not truly reflect influences on critical thinking by the curriculum experienced during that study period. A longer study period would allow more time for any influence of the curriculum to become apparent. The interactions between and among courses, faculty members, and students, including the school culture, may have an impact on critical thinking and other outcomes. These influences are difficult to control for and measure and may require a longer period of time to effect instrument scores.

A possible exception might be during times of significant curriculum change or for specific interventions such as PBL or critical thinking classes.

The CCTST was associated with several significant correlations. The correlations between the composite PCAT and CCTST total scores suggest that elements of critical thinking are indeed being measured by the composite PCAT. Miller also found similar correlations.¹² In the case of the correlations involving the P4 subjects, the PCAT was taken several years prior to this study. The relationship between the composite PCAT and CCTST scores seems to extend deeper into the pharmacy curriculum than just the initial first or second year. This may have implications in admissions decisions. The lack of many correlations with CCTDI is not unexpected because of the differences between the instruments. However, one significant finding was the correlation between P4 final GPA and their CCTDI total scores. Kidd and Latif¹⁷ found that both CCTST and CCTDI were among a group of factors which seemed useful in predicting curricular success.

Some CUSOP subjects did have CCTST and CCTDI total scores below average or in a range indicative of possible weakness. Reports of the significance of low instrument scores on academic progression in pharmacy have not been found. Following these students to evaluate any type of difficulties encountered in the curriculum may provide valuable feedback on the meaning and value of the instrument scores. Overall our findings provide useful information for curriculum and admissions discussions. A picture of the CUSOP subjects was obtained that was not previously available.

Continuous use of these instruments has not taken place since the time of this study. However, we plan to administer the instruments again to new P1 subjects and follow this same cohort through the curriculum and retest again at least during their P4 year.

There is an expense associated with the use of these instruments; thus, yearly testing of every class may not be cost-effective or labor-effective for some schools.

Results of this study support the need for attention to the critical thinking abilities of CUSOP pharmacy students and have provided at least a snapshot of the extent of these skills in the subjects as measured by these instruments. This study provided a baseline to which future use of these instruments, as well as other measures of assessment, can be compared.

The use of the CCTST or the CCTDI is one part of an overall assessment plan which can present a unique picture of the students and the curriculum. These instruments along with other assessment measures may provide the triangulation necessary for the best possible evaluation of student critical thinking abilities and the effectiveness of a curriculum. As Henderson and Hawthorne²⁰ have described, a complete curriculum picture requires several “lenses” to view the curriculum through. CCTST and CCTDI can provide two of those additional lenses. Overall the results of this study have established a foundation which can serve as a starting point for continued research here.

Limitations

Several limitations are recognized. Sample size was small and a significant limiting factor. The subsequent low power may have impaired the detection of significant differences. This type of research had not been conducted at CUSOP prior to the study. Hopefully, continued educational research will result in students becoming more agreeable to participate in such studies. Because of IRB requirements students could not be required to participate; thus, self-selection may have biased the results. The P1 subject group was the only group that represented at least 50% of their respective class. Although a few differences among the groups were noticed (Table 1), differences in other characteristics, such as motivation, may have influenced the results and differentiated the groups. The influence of a ceiling effect limiting changes in scores might be possible. Future studies may provide more information regarding this influence.

A better picture of critical-thinking changes and influences of the curriculum might be seen with longer study periods. Significant changes in critical thinking scores with these instruments may require a longer study period. Future studies are being planned to compare scores at least between admission and graduation for a cohort of students.

The time needed to complete the instruments was longer than the typical 50-minute class period for many students. This may have contributed to the lack of willingness of some students to complete both phases of the study. The lack of incentives available to the students, as well as the pressure of school work in general, may have reduced willingness to participate. The experience of participating in this study in their early years in pharmacy school may have a positive influence on students' willingness to participate in future research.

CONCLUSIONS

There were no significant changes in P1, P2, P3, and P4 students' scores on the CCTST and CCTDI over 1 academic year. The majority of students' scores were still above average for college students/young adults and were comparable to scores of pharmacy students reported in other studies. Some student scores were below average and indicated possible weakness in the area of critical thinking. An understanding of the significance of both higher and lower scores on these instruments as students progress through the curriculum would be valuable.