Using Performance-based Assessments to Evaluate Parity Between a Campus and Distance Education Pathway

Abstract

Objectives

To compare the performance of campus-based students with that of distance students during the first 2 years of a doctor of pharmacy program to evaluate parity between the pathways.

Methods

Twelve cases were created for each year of the program along with performance criteria. The cases were converted into computer-based simulations for programmatic assessment at the end of the 2002-2003 and 2003-2004 school years. All first-professional year (P1) and second-professional year (P2) students participated in the assessments. Overall class means were calculated and used to compare student performances between campus and distance education pathways.

Results

Overall scores for the 2003 P1 class were 56.4% for the campus-based students and 62.4% for the distance students, (p = 0.002); overall scores for the 2003 P2 class were 48.8% and 55.5%, respectively (p < 0.0001). The 2004 overall scores for P1 campus and distance students were 59.0% and 65.7%, respectively, (p = 0.001); and for 2004 P2 scores the results were51.8% and 56.5%, respectively (p = 0.049).

Conclusions

Students receiving their pharmacy education via a distance pathway scored higher on performance-based assessments compared with students receiving their pharmacy education via the traditional campus-based pathway. This indicates that distance students are receiving at least an equivalent curricular experience in the P1 and P2 years compared to that received by campus-based students.

INTRODUCTION

The aging population and corresponding increase in prescriptions has created a current and future demand for pharmacists.¹ This workforce demand has in turn produced an unprecedented period of growth in pharmacy education. Universities are opening new schools of pharmacy, while existing schools and colleges are developing innovative ways to increase their class sizes. They are graduating more practitioners by developing satellite campuses, and employing distance education technology.²

Our school addressed this shortage of pharmacists through the development of a distance-based doctor of pharmacy degree pathway in which all didactic courses, with the exception of 2 laboratory-based courses and the clinical rotations, are delivered via the Internet. A more complete description of this pathway and its development is available in a previous publication.³

This increase in enrollment was not without challenges such as ensuring that we effectively increase class size without sacrificing quality. The courses are identical in content to the campus-based pathway, and when possible, are delivered at the same time via distance technology. But not all courses are taught simultaneously or by the same instructors. Some of the distance courses are taught out of sequence and by different instructors. Also, differences in examination administration exist with the differences in pathways. For example, the distance-based pathway may use QuestionMark to administer examinations electronically, while in the campus-based course, a paper-based format is used. Therefore, we required a controlled testing condition to account for potential course-to-course variations in examinations. The purpose of this project was to provide a standardized testing environment to compare the performance of campus-based students to that of distance-based students for the first 2 years of the program to evaluate parity between the pathways.

METHODS

The development of test methods for this project has been described in detail in a previous publication.⁴ Our program adopted an abilities-based curriculum and established 12 terminal educational ability-based outcomes. These ability-based outcomes became the basis for comparing the 2 curricular pathways. The first step was case development based on these ability-based outcomes. Twelve faculty teams were formed: 1 team per ability-based outcome. When developing the faculty teams, care was taken to select faculty members who taught courses in which the outcomes were listed in the course syllabus as a primary focus. Faculty teams took the terminal educational ability-based outcomes and defined expected outcomes for student performance by each year in the curriculum. In other words, the terminal ability-based outcome (Appendix 1) was defined by producing student performance expectations or tasks for each year in the curriculum. Twenty-four different cases (12 for P1 and 12 for P2) were authored for the computer-based clinical simulations; each computer-based clinical simulation assessment that students completed consisted of 12 case stations. Each station was composed of a case, directions to the student, case performance criteria, and any references deemed necessary to complete the problem or clinical task(s) required in the case.

The faculty members were instructed to use their professional/clinical experience to describe a problem that focused on the ability-based outcome level in question. This real-life problem then constituted the case scenario. For each case brief descriptions of the context of the problem were provided to ensure congruence between students' perception and actual case construct. These directions allowed the students to quickly identify the tasks required to complete the case. An example of student directions for a P1-level case is provided in Appendix 2. Case writers defined the case scenario, directions to the student, performance criteria used for evaluating student performance, and identified any references necessary for solving the case. The performance criteria were in the form of rating scales or checklists, which are proven means of evaluating clinical skills in health professional education.^5,6 These checklists consisted of performance indicators (ie, items) or observable behaviors that could be scored by a faculty grading panel.

Validations of case content and checklists were based on a previously described method.⁷ Briefly, the case content, rating scale, and directions were reviewed by a faculty validation panel. The panel was composed of 5 pharmacy faculty members from different pharmacy schools: 2 representatives from the East Coast (Pittsburgh, Shenandoah), 1 from the Midwest (Drake), 1 from the West Coast (UCSF), and 1 from the South (Houston). Two of the faculty members were employed in private pharmacy schools (Shenandoah and Drake) while the remainder came from state schools. The purpose of the validation was twofold. First, to ensure that each case appropriately tested the specific outcome in question (ie, construct) and that the case was true to the context in which it was set (ie, problem or clinical condition). Second, the panel addressed the generalizability of the cases to their institution and region of the country. Comments from the panel were used to produce the final cases. The cases were then developed into online computer simulations and pilot tested at the pharmacy school by P4 clerkship students. Twenty minutes were allotted for each case. Case references and props (eg, prescription bottles) were case-dependent and could not be moved; thus, students physically rotated from one computer to another until all cases were completed. The students provided feedback and the assessment procedures were amended based on their comments. The revised assessments were then administered to both campus and distance students. Wired classrooms were utilized for access to computers at each case.

All student responses were scored by a 2-member faculty grading panel according to the performance criteria indicated in the checklists. Each item on a rating scale was scored dichotomously (ie, “yes, the student completed/preformed the item successfully” or “no, the student did not”). The number of items varied by case, and each case was treated as an independent unit (question) of the entire performance-based assessment since each case assessed a different outcome. The 2 scores generated by the grading panel were averaged and this mean represented the students' score.

Overall class means were calculated and used to compare student performances for both campus and distance pathways using a Mann-Whitney U test. This project was reviewed by the Institutional Review Board for Research and deemed to be exempt.

RESULTS

All P1 and P2 students in the program completed the assessments. Data were similar with no statistical difference found between the groups in entering grade point averages (Table 1). Overall scores for the 2003 academic year are listed in Table 2. The mean score for P1 campus students' was 56.4% and for P1 distance students, 62.4%, (p = 0.002). The mean score for P2 campus students was 48.8% and for P2 distance students, 55.5% (p < 0.0001). Table 3 lists the overall scores for the 2003-2004 academic year. Scores for P1 campus students compared to those of distance students were 59.0% vs. 65.7%, respectively (p = 0.001). Scores for P2 campus students compared to those of distance students were 51.8% vs. 56.5%, respectively (p = .049). Course-grade comparisons between the 2 pathways for P1 and P2 are provided in Tables 4 and 5, respectively.

Table 1.

Demographics of Pharmacy Students Enrolled in Distance and Campus Pathways

P1 = first-year students; P2 = second-year students; GPA = grade point average

Table 2.

Comparison Results for of 2002-2003 P1 and P2 Campus and Distance Students

Table 3.

Comparison of Results for 2003-2004 P1 and P2 Campus and Distance Students

Table 4.

Comparison of Grades Between P1 Students Enrolled in Campus and Distance Pathways

*Mean based on the following scale A = 4.0, B+ = 3.5, B = 3.0, C+ = 2.5, C = 2.0, D = 1, F = 0

†p < .05; significance determined using a Mann-Whitney U analysis

Table 5.

P2 Student Grade Comparison Between the Campus and Distance Pathways

*Mean based on the following scale A = 4.0, B+ = 3.5, B = 3.0, C+ = 2.5, C = 2.0, D = 1, F = 0

†p < .05; significance determined using a Mann-Whitney U analysis

DISCUSSION

Students receiving their pharmacy education via a distance-based pathway scored higher on performance-based assessments compared with students receiving their pharmacy education via the traditional campus-based pathway. This indicates that distance students are receiving at least an equivalent curricular experience in the P1 and P2 years compared to that received by campus-based students.

Several reasons exist that justify the need for this study. An examination of the course grade comparisons between the 2 pathways is provided in Tables 4 and 5 and is difficult to interpret. Knowing that there is some variation in course sequence, course instructor, and minor differences in testing conditions, some controlled means of assessing parity between the pathways is necessary. Also, since our program has an abilities based curriculum, and not all tests utilized are performance-based, it is programmatically essential to identify how well students are achieving or performing the program's ability-based outcomes.

The use of performance assessments to compare student outcomes between the 2 pathways was important. The Accreditation Council for Pharmacy Education (ACPE) and the American Association of Colleges of Pharmacy (AACP) encourage the adoption of an abilities-based curriculum.^8,9 Our school has adopted an abilities-based curriculum, and therefore when evaluating such a program it would be appropriate to use performance assessments as they require students to demonstrate or produce evidence of learning.⁴

Our data indicate that learning occurred; that students were able to demonstrate competency of the abilities. This is unique in that one may assume student experiences were different based on the pathway. That is, learning was clearly documented even though the learning experiences may have differed based on the pathway.

Even though the distance students scored significantly higher than the campus students in our study, neither of the groups performed well on the assessments. One possible explanation for the low scores may be that our students are not primarily assessed via performance-based methods during the first 2 years of the program. This may account for the discrepancy between expectations (ability-based outcomes are performance-based) and scores.

The data from this analysis only included students who were in the first- and second-professional years of the program. To get a clearer and more overall picture of students' abilities, we must also look at the assessment data from the third- and fourth-professional years. The P3 and P4 assessments were conducted using standardized participants that included the Objective Structured Clinical Examination (OSCE) approach.⁴ These data are currently being analyzed. In addition, looking at individual course scores, overall GPA, and NAPLEX results may give additional information to assess parity between the 2 pathways. These analyses are underway.

Once all of our performance assessment data are analyzed, it will be used in aggregate to determine the effectiveness of the curriculum's content in terms of preparing students to meet the program's educational ability-based outcomes.⁴ The combination of student performance data regarding specific programmatic outcomes may be used to monitor overall effectiveness, demonstrate improvement, and provide accountability of student achievement to all constituents.^4,10 The data can then be related back to the instructional methods and used for program planning, such as the incorporation of increased active-learning strategies in the classroom.⁴ Further, the information in this study may be useful to other schools of pharmacy in making decisions regarding adopting a distance-based pharmacy pathway or to model the performance-based assessment approach we have taken with their own abilities-based curriculum.

Limitations

Because the distance students take their didactic courses online, the distance students may be more adept at online assessments than the campus students. This may confound our results. Also, because different instructors were used in a few of the courses, our results may reflect differences in pedagogical effectiveness versus absolute student learning. Finally, because this is the first large performance-based assessment of hundreds of students, no previous data exist with which to compare. Therefore, it is difficult to interpret the low scores. To determine whether other students would perform similarly or better, more research is needed.

CONCLUSIONS

The performance-based assessment data indicate that our distance pathway provides a similar learning experience in the P1 and P2 years to that provided by the campus pathway based on student performance. This study provides data to help support the evaluation of parity between the 2 pathways to date based on student performance.

Appendix 1. Ability-based educational outcomes for graduates.

Pharmaceutical Care Abilities

1. Patient Assessment - The student shall contribute to the database of information about the patient by: a) performing a medication history, review of systems, and physical assessment; b) recommending and interpreting laboratory tests; c) assessing medical, psychosocial, behavioral, and economic status; and d) identifying potential drug-related problems.

2. Pharmaceutical Care Plan Development - The student shall develop pharmaceutical care plans^a that maximize the patients' response to drug therapy and prevent or resolve a drug-related problem(s) in order to ensure positive outcome(s). The student shall appropriately document the implementation of and outcomes related to the care plan. The pharmaceutical care plan shall include medical devices, as needed, and educational information (e.g., nutrition, lifestyle, etc.) intended to promote general health and prevent or minimize disease progression.

3. Drug Therapy Evaluation - The student shall assess and monitor the patient's drug therapy, including a consideration of the chemical, pharmaceutical, pharmacokinetic, and pharmacological characteristics of the administered medications.

4. Pharmacotherapy Decision-Making - The student shall make pharmacotherapy decisions and support those decisions based on knowledge of biomedical, pharmaceutical, administrative, and clinical sciences.^b The student shall recommend patient use of prescription and nonprescription medications, as well as nondrug therapy.

5. Medication Preparation, Distribution, and Administration –The student shall compound and/or dispense drug products consistent with patient needs and in harmony with the law. The student shall demonstrate the ability to accurately interpret the prescription, select the appropriate dosage form, route and method of administration, and appropriately package and label the product. The student shall demonstrate the ability to administer medications, when appropriate.

6. Systems Management - The student shall use and evaluate acquisition, inventory control and distribution systems, while documenting and maintaining quality. The student shall plan, organize, direct and control pharmaceutical care systems and human, material, and financial resources, utilizing management theories and information technology.

General Education Abilities

• 7. Communication Skills - The student shall read, write, speak, listen and use multimedia to communicate effectively. The student shall counsel and educate patients, as well as communicate with other healthcare professionals.

• 8. Critical Thinking - The student shall acquire, comprehend, apply, analyze, synthesize, and evaluate information. The student shall integrate these abilities to identify, resolve, and prevent problems and make appropriate decisions. The student shall understand the research process.

• 9. Professional Ethics and Responsibility - The student shall represent the profession in an ethical manner. The student shall identify, analyze, and resolve ethical problems.

• 10. Social Interaction, Citizenship, Leadership, Professionalism - The student shall demonstrate appropriate interpersonal behaviors. The student shall provide service to the profession, as well as the community. The student shall be proactive in collaborating with other health care professionals.

• 11. Life-long Learning - The student shall continuously strive to expand his or her knowledge to maintain professional competence.

• 12. Information Management – The student shall apply technology to pharmacy practice and science. The student shall demonstrate the ability to interpret and evaluate data for the purpose of assessing the suitability, accuracy, and reliability of information from reference sources.

^aThe pharmaceutical care plan shall include subjective and objective patient information, an assessment of that information, and a plan to resolve and monitor any drug-related problems that were identified.

^bPharmacotherapy decisions determine what, why, when, where, and how drug therapy is provided. The making of pharmacotherapy decisions is the foremost expressions of the professional knowledge, responsibility and authority of pharmacists.

Appendix 2. Directions to the Student

Directions

You are a pharmacist in the clinic pharmacy serving a medical building containing physicians from many different practices. Dr. Baker just prescribed Avandamet for a 34-Year-old woman just diagnosed with type 2 diabetes today. Please review the prescription and clarify any problems using the resources available. If you find any problems, please document them on the form provided.

The following is a list of tertiary sources; please check which would be pertinent to determine if problems exist in this case (Check all that apply)

• ○ Briggs Drugs and Pregnancy

• ○ Facts and Comparisons

• ○ Micromedex

• ○ Pharmacotherapy (old edition)

• ○ American Hospital Formulary Service Drug Information

Using only Micromedex determine if problems exist with the following prescription.