Achieving Equivalent Academic Performance Between Campuses Using a Distributed Education Model

Abstract

Objectives

To demonstrate that students in competency-based anatomy and pharmaceutical calculations courses performed similarly whether enrolled in the classes through distance education or face-to-face lectures.

Methods

Student outcomes data including module examination scores, final course grades, and student demographics data were collected, merged, and analyzed.

Results

Mean module examination final scores and final course grades did not significantly differ between students at the lecture site and students at the remote site.

Conclusions

The competency-based anatomy and pharmaceutical calculations courses, whether remote or at the lecture site, provided equitable learning opportunities and roughly equivalent learning outcomes for students.

INTRODUCTION

Distributed education methods have been widely implemented within US pharmacy schools. A variety of distance education programs exist, including programs with courses that are totally Internet-based and hybrid courses that blend face-to-face and online components.^1,2 The Texas Tech University Health Sciences Center School of Pharmacy first initiated distance education courses in the third year of the doctor of pharmacy program in 1999. MacLaughlin and colleagues evaluated differences in learning outcomes in these third-year pharmacotherapy courses between local and distant students.³ Learning outcomes were not significantly different for any of these pharmacotherapy courses between the host site (face-to-face learning in a classroom setting) and remote sites.

During the 2007-2008 academic year, Texas Tech School of Pharmacy implemented a unique distributed education model for first-year students. Of the 130 students entering their first year of the PharmD program at Texas Tech, 90 students attended a campus in Amarillo and 40 students attended a campus in Abilene. All 130 students participated in the same courses, though separated by a distance of 260 miles. For classroom-based instruction, lectures originated in either Amarillo or Abilene. Students in the host classroom participated in face-to-face instruction, while students in the remote classroom participated in a virtual face-to-face environment created with videoconferencing technologies.

Two of the first-year courses offered in the distributed format at Texas Tech were designed based on the principles of Keller's Personalized System of Instruction (PSI). Keller's PSI, known as the Keller method, was developed to overcome limitations of the traditional lecture-based method of instruction.^4-6 Some of these limitations are: (1) all students must proceed through a course at the same pace, (2) there is frequently limited feedback on examinations, and (3) accommodations are lacking for student's differing levels of preparedness and learning skills. With the Keller method, course content is broken into discrete modules, instructional materials are provided for each module, learning objectives are clearly defined, students are allowed to test on each module repeatedly until competency is achieved, and immediate, personal feedback on module examinations is provided. The 2 courses grounded in the Keller method were anatomy and pharmaceutical calculations; these 1-semester courses were developed for use within a WebCT (Blackboard Inc., Washington, DC) courseware system.

Weaker, less-prepared students tend to benefit the most from participation in Keller-method courses.⁷ Furthermore, minorities have unique challenges in achieving academic preparedness for professional degree programs.⁸ Accordingly, courses based on the Keller method provide the opportunity for all students to achieve competency while accommodating students' differing learning styles and backgrounds. Further, the Keller method offers the potential for reducing academic disparities for minority students.

While the competency-based courses described in this study were developed with the objective of optimizing student learning outcomes, it was unknown whether students in the remote environment (Abilene) might have different outcomes than students in the host environment (Amarillo), and whether student outcomes might differ with respect to race/ethnicity.

This research study focused on quantitatively assessing the impact of the unique learning environment on student outcomes in the 2 Keller-method courses, anatomy and pharmaceutical calculations. The primary research question of this study is “does student performance as measured by mean module examination scores and final course grades in Keller-method, competency-based pharmaceutical calculations and anatomy courses differ by location based upon the distributed education model implemented at the Texas Tech School of Pharmacy?” A secondary question is “does student race/ethnicity impact outcomes based upon the distributed education model?” The principal objective of the study was to demonstrate that students enrolled in the Keller method courses are not differentially impacted by the distributed education method.

METHODS

Two pharmacy courses offered at the Texas Tech School of Pharmacy were designed using the Keller method similar to previous methodology⁶ and implemented in a distributed learning environment. Of the 130 students enrolled in the courses, 90 attended the campus in Amarillo and 40 attended the campus in Abilene. Approximately 80% of the students regularly attended lectures; though attendance declined when major examinations were scheduled for other courses. The lecture portion of the courses was used for instructional and motivational purposes. If a lecture was delivered in a classroom in Amarillo, students in Amarillo were able to interact with the instructor face-to-face. Through the use of technologies, including large display screens and speakers, students in Abilene received the lecture audio and video synchronously in their classroom, and they were able to ask questions orally using microphones. Similarly, the lecture could be delivered in a classroom in Abilene, with the students in Amarillo participating in the course via the use of instructional technologies. In an effort to assure that the distributed education model did not diminish the learning process for students in the remote location, faculty members were present in both locations during all courses sessions. Although faculty members were available during course sessions at both locations, faculty members at the remote location served principally as facilitators; they did not lecture. For these Keller-method courses, the location from which the lecture originated (Amarillo or Abilene) could vary from day to day. However, instructional teams and team leadership were located in Amarillo, with 85% of all lectures originating from Amarillo. Thus, Abilene students were generally participating from a remote setting.

Two courses, anatomy and pharmaceutical calculations, were included in this study. The courses were hybrid; they contained both online and on-campus components. Furthermore, the competency-based courses were grounded in the Keller method. The anatomy course was designed with 23 learning modules; the pharmaceutical calculations course had 9 learning modules. For each module in both courses, learning objectives were established, instructional materials were developed, and module examinations with random selection of questions from a comprehensive test bank were created.

Students progressed through the instructional modules in a self-paced manner and they were allowed to repeatedly take module examinations. Examination questions were open-entry, requiring the student to enter correctly spelled anatomical terms and correct numerical calculations (allowing for rounding error). Formats such as multiple choice and true-false were not used. The students received immediate computer-generated feedback comprised of correct answers and explanations on each examination question. Since students were allowed to repeatedly take module examinations, performance on any specific attempt was of less consequence than if the students had been participating in a course that allowed only one attempt per examination (ie, high stakes testing). As such, student test preparation and test-taking behaviors, as well as progression through the instructional modules, may differ from that of traditional courses.

Students were required to achieve competency, as defined by a minimum score of 70%, on each module examination. Students could continue to retake module examinations after achieving competency in an effort to raise their module examination scores, though all testing had to be completed by the end of the scheduled semester. For final grade calculations in both courses, the student's best score on each module was used. This provided students with an incentive to progress beyond minimal competency on the instructional modules. Final course grades were based on performance on module examinations as well as on traditional mid-semester and final examinations. Though instructional materials and module examinations were provided in an online, WebCT format, students were expected to attend and participate in classroom-based sessions each week.

Student demographic data are provided in Table 1. Students in Amarillo (n = 90) and Abilene (n = 40) did not differ significantly with respect to prepharmacy grade point average (GPA), Pharmacy College Admission Test (PCAT) composite scores, completion of a baccalaureate degree, or age. The percentage of female students in Abilene was significantly greater than in Amarillo (68% vs 42%, p = 0.008). Abilene also had a larger percentage of white students than Amarillo (70% vs. 48%, p = 0.019).

Table 1.

Student Demographics in Two Competency-Based Pharmacy Courses

^aChi square or student's t

Half a letter grade (5 points on a 100-point scale) difference in mean module examination scores and final grades between student groups was designated as academically significant for this study. For a priori power calculations, the distribution of module examination scores was assumed to have a standard deviation of 10. Using Cohen's d to calculate the effect size, these estimates represent a moderate effect of d = 0.5.⁹ Given 90 students in Amarillo and 40 students in Abilene, the sample size was sufficient to achieve power of 80%.

Student outcomes data, including module examination scores for each module attempt and final course grades, were transferred from WebCT to SPSS 16.0. The student outcomes data were then merged with student demographics data from the institutional data warehouse. For quality assurance, the data were reviewed and cleaned, and student identifiers were removed to assure confidentiality. The study was designated exempt from review by Texas Tech University Health Sciences Center's Institutional Review Board.

This quantitative 2-group cohort study empirically characterized the association of student location with student learning outcomes as measured by mean module examination scores and final grades. All data analyses were performed using SPSS 16.0 (SPSS Inc., Chicago, IL). Descriptive statistics included percentages, means, and standard deviations. Independent-samples t tests were conducted to assess differences in means of student outcomes variables for the 2 groups. Multiple regression models were developed to determine the association of student location and race/ethnicity with student outcomes while controlling for covariates.^10,11 For the multiple regression models, categorical race/ethnicity data were dummy coded with white as the reference category. Variance inflation factors and residuals were reviewed to assure that multiple regression assumptions were met. A 2-way MANOVA with module examination scores and final grades as dependent variables was used to test an interaction effect for student location by race/ethnicity.¹² The level of significance for all analyses was set at 0.05.

RESULTS

This study assessed the impact of a distributed education environment on student learning outcomes as measured by mean module examination scores and final grades in 2 courses grounded in the Keller method. The 2 student groups were defined as those in Amarillo (the host location) and in Abilene (the remote location). The 2 groups were roughly equivalent with respect to education levels, prepharmacy GPA, and PCAT composite scores (Table 1); the Abilene group had a higher percentage of female and white students.

Each student in the Amarillo and Abilene groups (N = 90, N = 40, respectively) was allowed to complete module examinations for 32 learning modules (23 for anatomy, 9 for pharmaceutical calculations), yielding an upper bound of 4160 (130 x 32) cases of students by modules. Three students dropped the courses during the semester, leaving 4,066 cases of students by modules (Table 2). Abilene students had higher mean first attempt module examination scores (baseline scores) than Amarillo students (59.6 vs. 57.0; p = 0.031). Both groups' means fell below the required 70% competency standard on module first attempts. However, through the remediation and retesting opportunities inherent in the Keller method, both Amarillo and Abilene student groups achieved excellence in mean best scores on module examinations (95.3, 94.5, respectively). Mean best scores on module examinations did not differ in a significant manner. Amarillo students had larger mean gain scores, defined as the difference between best scores and baseline scores, than Abilene students (38 vs. 35, p = 0.004). Amarillo students also had a higher mean number of examination attempts per module than Abilene students (4.1 vs. 3.6; p < 0.001). Although students at the host location (Amarillo) underperformed at baseline compared to students at the remote location (Abilene), through remediation and retesting as provided by the Keller method, both groups improved their scores to roughly equivalent levels and both groups achieved excellence (mean module examination scores ≥ 90) in learning outcomes.

Table 2.

Module Examination Outcomes in Two Competency-Based Pharmacy Courses

^an = students by modules. 130 students initially enrolled in 2 courses which had a total of 32 learning modules. The upper bound for N is 130 x 32 = 4,160. Three students dropped the courses, yielding a total N of 4,066.

^bStudent's t.

The multivariate analysis revealed that after controlling for covariates, student location is not a significant predictor of gain scores, though race/ethnicity is. Hispanic students had lower gains than whites (p = 0.003), while Asians had higher gains (p = 0.045). Black students did not differ significantly from whites. Other independent variables that were significant include prepharmacy GPA (p < 0.001), PCAT composite score (p < 0.001), student age (p = 0.004), time per module examination attempt (p < 0.001), number of examination attempts per module (p < 0.001), and module first attempt scores (p < 0.001). Student gender was not a significant predictor of gain scores.

Of all students' first attempts at completing all module examinations (4,066 first attempts), students demonstrated competency with a score of 70 or greater 2116 times, and did not demonstrate competency 1,950 times. Multivariate models were developed to predict gain scores for these 2 groups (students who demonstrated competency on first attempts and students who did not demonstrate competency). Student location was not a significant predictor for either of the 2 groups. However, for those students who did not achieve competency on first attempts, race/ethnicity was a predictor. Hispanic students had lower gains than whites (mean difference = -2.3; p = 0.024) and blacks had higher gains than whites (mean difference = 4.7; p = 0.011). Additionally, male students had lower gains than female students (mean difference = -1.4; p = 0.040).

Final course grades were based upon module examination scores as well as other factors such as laboratory grades and final examinations. In the anatomy course, Amarillo students had final grades of 92.2 ± 2.7, while Abilene students had final grades of 91.7 ± 3.5. In the pharmaceutical calculations course, Amarillo students had final grades of 88.2 ± 6.2, while Abilene students had final grades of 88.3 ± 5.7. For both courses, final grades did not differ in a significant manner based upon whether students were at the host location or the remote location (Table 3).

Table 3.

Students Learning Outcomes by Course, Mean (SD)

^aStudent's t

The analyses described above reveal that student location was not associated with student learning outcomes in the 2 Keller method courses, though student race/ethnicity had some bearing on learning outcomes. To test if the interaction of student location and student race/ethnicity had an impact on student outcomes, a MANOVA analysis was conducted with module examination scores and final grades as the dependent variables and student location (Amarillo, Abilene) and student race/ethnicity (white, Asian, Hispanic, black, other) as fixed factors. The interaction term (student location by race/ethnicity) was not significant.

DISCUSSION

The primary goal of this study was to demonstrate that students enrolled in pharmacy courses grounded in the Keller method were not impacted by implementation of a distributed education environment. More specifically, the study focused on quantitatively assessing the impact of students' locations within the distributed education environment on student learning outcomes as measured by module examination scores and final course grades.

Students in this study were assigned to 2 campuses. Ninety students were assigned to the host location (Amarillo) and 40 to the remote location (Abilene). Students at the 2 locations were roughly equivalent with respect to degree attainment, prepharmacy GPAs, and PCAT composite scores. During the semester, students at the host location (Amarillo) earned lower mean module first attempt scores (baseline scores) than students at the remote location (Abilene). However, the remediation and retesting aspects of the Keller method provided an opportunity for students who underperformed initially to catch up with others. In this study, the Amarillo students who underperformed at baseline remediated and retested more frequently than the Abilene students. This allowed the Amarillo students to reach parity with the Abilene students. Inferential statistical analyses revealed that mean module examination best scores (final scores) did not significantly differ between students at the host location (Amarillo) and students at the remote location (Abilene). Similarly, mean learning gain scores and final course grades did not differ significantly between students at the host location and the remote location. Collectively, these findings suggest that students participating in these courses were not favorably or adversely impacted by location (host vs. remote). This helps to validate the effectiveness of the distributed education environment for competency-based courses.

The finding that students are neither favorably nor adversely impacted by location may be of interest to a variety of stakeholders. Accrediting agencies are interested in empirical evidence that distributed learning environments do not adversely impact positive learning outcomes.² Similarly, students and parents may wish to know that parity regardless of student location is achieved. Finally, faculty members and administrators wish to assure that student learning is effectively and equitably implemented. This study demonstrates that equal learning outcomes were attained in Keller-method pharmacy courses for students at both the host and remote locations. Additionally, this study demonstrated benefits of the Keller method, including the notable finding that student knowledge as assessed by module examinations scores increased substantially for all students with mean module examination best scores exceeding 90 for both courses and at both locations.

A secondary finding was that race/ethnicity was associated with learning outcomes, particularly for those students who did not achieve competency on module examination first attempts. Hispanics achieved lower learning gains than whites, and this was most evident among students who did not demonstrate competency on first attempts at module examinations. Thus, race/ethnicity is a significant differentiator of learning gains. However, the multivariate analysis that controlled for covariates suggested that the difference in mean gain scores between whites and Hispanics should be about 1.6 points on the 100-point grading scale. Though the 1.6 point difference was significant, it was not academically significant (ie, 5 points on the 100-point scale) when using the a priori standard identified in the Methods section. Furthermore, both Hispanics and whites earned mean module examination best scores greater than 90 (91.8 and 94.9, respectively). Although Hispanics underperformed in a significant manner relative to whites, the Keller method provided the opportunity for both race/ethnicity groups to achieve excellence in learning outcomes, with differences being of limited practical importance. The study also demonstrated that there was not a significant interaction effect for race/ethnicity and student location. This finding shows that the 2 Keller-method pharmacy courses provided appropriate learning opportunities and produced roughly equivalent learning outcomes for students regardless of race/ethnicity or location. However, the findings provide some basis for asserting that special attention to some racial/ethnic groups may be warranted, especially for minorities who underperform at baseline.

Though Amarillo and Abilene student groups were roughly equivalent with respect to salient variables (Table 1), this retrospective study was not a controlled experiment. Variables other than those included in this study may have impacted student outcomes. Caution should be exercised when generalizing the findings from this study to dissimilar student populations. Use of the Keller method in clinical courses may not yield results comparable to those achieved in the anatomy and pharmaceutical calculations courses. The Keller method was the only instructional method involved in this study; the findings may not be applicable to other methods. Some of the student data such as race/ethnicity was self-eported; these data were not validated.

CONCLUSION

This study assessed the impact of a distributed education model on student learning outcomes in hybrid, competency-based anatomy and pharmaceutical calculations courses that were grounded in the Keller method. The distributed education model did not differentially impact students based upon whether they were taking the courses at the host location or a remote location. Although student outcomes differed by race/ethnicity, the differences were small, and all student groups ultimately achieved excellence as measured by mean module examination scores regardless of race/ethnicity. Thus, students had roughly equivalent learning opportunities and outcomes regardless of their location or race/ethnicity. Accordingly, this study serves to validate the effectiveness of the distributed education model for competency-based pharmacy courses.