Effect of Incentives on Student Performance on Milemarker Examinations

Abstract

Objectives

The objective of this study was to evaluate the effect of incentives on student performance on comprehensive cumulative examinations administered at the College of Pharmacy, University of Houston.

Methods

This study is a retrospective longitudinal study conducted over a period of 6 years, from 2000 to 2005. Passing rates on the cumulative examinations administered during the first 3 years of the doctor of pharmacy curriculum were obtained. These cumulative examinations, known as the Milemarker assessments, involve 3 examinations: Milemarker I, II and III, each offered after completion of each progressive year. Milemarker I and II examinations were phased in throughout the years with various incentives to increase student performance. Incentives for these examinations included books, achievement letters, bonus points, and remediation exercises. Incentives with respect to Milemarker III examination was determination of students' progression into the experiential year of the curriculum and did not change over the study period. Passing rates were compared for these examinations before and after the implementation of these incentives.

Results

Passing rates for Milemarker I increased significantly by 185% from 2003 to 2004 when incentives were changed from awards such as books and achievement letters from the Dean's office to bonus points towards future examinations and a remediation process. Similar results were seen for Milemarker II, where the passing rates increased by 590% during the same period for similar incentives. However, passing rates were much higher throughout the time period for Milemarker III due to the high-stakes incentive of stops on progression to the next year.

Conclusion

Appropriate incentives may be effective in changing student performance on comprehensive cumulative examinations.

INTRODUCTION

Researchers have shown that introducing incentives can enhance performance. This is true with respect to any setting whether it is a healthcare institution, industry, office, or school.^1-5 Incentives can serve several functions such as initiating action, changing or suggesting goals and intentions, and ensuring commitment.^6-11 In the public sector, incentives can effectively enhance performance when provided in combination with goals and feedback.⁸

One model explaining the relationship between incentives and performance is Victor Vroom's expectancy model. According to this model, a worker's motivation for success is a function of the worker's expectancy of success (whether the added effort produces the desired success), value of success (whether better performance or success results in any reward), and need to succeed (value of the incentive or reward).^8,9 Incentives are usually high powered in private sectors, whereas they are usually low-powered or lacking in the public sector.¹⁰ High-powered incentives are usually monetary, such as bonuses, and are highly correlated with output. Low-powered incentives depend on output as well as effort, such as salaries or hourly wages.¹¹ Locke et al have proposed that the effect of incentives on performance is controlled by desired goals.¹² However, Tolchinsky and King⁶ have stated that monetary incentives and goal setting can influence performance independently, provided monetary incentives are substantial.

The scenario is similar in academics. Attendance incentive plans, in which teachers are required to use their personal days for sick leave, are believed to reduce teacher absenteeism.¹³ Monetary incentives for schools and teachers may have a causal effect on improving students' achievements.¹⁴ Moreover, introducing monetary incentives for teachers as a reward for improvement in students' performance is more cost-effective than allotting extra instruction time and may be as effective as rewarding students with cash bonuses for their performance.¹⁵ Students have varying enthusiasm and interest for schoolwork or the education process based on incentives offered.¹⁶ Educational institutions have to determine which incentive structures should be used in order to improve student performance.¹⁷

Incentives provided to teachers have helped improve student academic outcomes.^14-18 Few studies have been conducted to examine the effect of incentives provided directly to students, based on their performance. Further, there is lack of knowledge with respect to the role of incentives and their effect on students' performance on comprehensive examinations in health professions, especially in pharmacy education.

Milemarker assessments are cumulative and comprehensive examinations used to assess student knowledge and retention of information from didactic courses. A detailed description of the Milemarker assessments process can be viewed in a previously published article.^19,20 Briefly, there are a total of 3 Milemarker examinations, each “administered”? after completion of a professional year. Milemarker I and II are formative, use 5 cases, and are useful for students to identify their areas of strengths and weakness. Milemarker III is summative, uses 6 cases, and is used to determine progression into the experiential year of the curriculum.

The objective of this study was to evaluate the effect of incentives on the performance of PharmD students on the comprehensive cumulative Milemarker assessments.

METHODS

A retrospective longitudinal study was conducted over a 6-year period from 2000 to 2005. Each year, incentives were introduced for students to enhance their performance on the Milemarker examinations.

When the Milemarker assessments were first introduced, the first 2 examinations were only considered as formative examinations with no consequences for a poor performance. However, Milemarker III was introduced as a summative examination which determined progression to the experiential portion of the curriculum. Thus, the incentive for students to study for Milemarker III may be considered as a “high-stakes” incentive. Students who fail Milemarker III are prevented from starting clinical rotations until they have passed it. Students who do not pass on their first attempt (Milemarker IIIa) can retake the examination (Milemarker IIIb) prior to the start of their rotations. If they do not pass the second attempt, they will miss their first rotation and their graduation may be delayed. They then have an opportunity to take Milemarker IIIc, which is administered within 6 weeks, and if necessary Milemarker IIId, administered an additional 6 weeks later. If they still do not pass Milemarker IIId, they must wait 1 semester before attempting to take the examination again. The passing score on the Milemarker assessments is based on a predetermined score which is obtained using a modified Angoff process described previously.¹³

From 2000 to 2003, incentives for passing Milemarker I and II consisted of providing reference books as awards to students scoring in the top 3 of their class and certificates of achievement from the Dean's office to others who performed well and passed these examinations. During this period, students who failed these assessments had an option to retake the Milemarker to improve their performance or to simply use them as a practice examination in preparation for Milemarker III. In 2004, incentives offered to students for passing Milemarker I and II examinations were changed to credits/carryover points (bonus points) that would be counted towards their overall score on Milemarker IIIa. Credits to be carried over to Milemarker IIIa were calculated as follows:

1. Milemarker I: (Percentage score on Milemarker I) × (0.05 × total possible score of Milemarker III)

2. Milemarker II: (Percentage score on Milemarker II) × (0.05 × total possible score of Milemarker III)

However, in either case, the maximum number of points awarded from Milemarker I and II would be 10 points each. In addition, students who did not meet the minimum competency on Milemarker I and II had to complete an appropriate remediation assignment based upon determined area(s) of weakness, and would not receive points towards Milemarker IIIa. Bonus point would not carry beyond Milemarker IIIa if students failed to pass Milemarker III in their first attempt.

Data were obtained from the Office of Assessment at the College of Pharmacy and included the number of PharmD students that took and passed the Milemarker comprehensive assessments at the College of Pharmacy from 2000 to 2005. The data did not include any identifier or student information. Passing rates were calculated and compared for each Milemarker, before and after each intervention (incentive of carryover points/remediation after year 2003). The study was approved by the Committee for the Protection of Human Subjects at the institution. Descriptive analyses, as well as t tests were performed to evaluate the effect of incentives on student performance.

RESULTS

The number of students who attempted the Milemarker examination over the 6-year period were 645 (Milemarker I), 509 (Milemarker II), and 390 (Milemarker III). The number of students taking the examination and passing rates for all Milemarker examinations can be viewed in Table 1.

Table 1.

Passing Rates for Milemarker I, II, and III Examinations

^*Milemarker IIIresults only include information on Milemarker IIIa (first attempt)

Passing rates on the Milemaker I examination increased steadily from 2000 to 2003 at a rate of 4.2% per year, indicating that incentives had little effect in changing student behavior and performance. The passing rate jumped 184.9% from 2003 to 2004 and 258.6% from 2003 to 2005. The results of a t test comparing the mean passing rates before and after 2003 indicated a significant difference and can be seen in Figure 1. The difference in the 2 mean passing rates was 64.3%, indicating that incentives with carryover points/remediation assignments worked better in changing student performance than gifts or recognition.

Figure 1.

Student Performance on Milemarker I before and after intervention. (* indicates significant difference; t-statistic = 82.9, p < 0.05).

The passing rate trend for Milemarker II was the opposite of that for Milemarker I during the initial phase (2001-2003). Passing rates decreased from 2001 to 2003 by 46.1%, indicating the effect of books and certificates as incentives was decreasing over the years. The introduction of incentives, in the form of carryover points/remediation assignments, caused passing rates to increase significantly (p < 0.05) after 2003. This trend was similar to that of Milemarker I as the passing rate for Milemarker II jumped 589.8% from 2003 to 2004 and 910.1% from 2003 to 2005. The results of a t test comparing the mean passing rates before and after 2003 indicated a significant difference and can be seen in Figure 2. The difference in the 2 mean passing rates was 48.2% indicating similar results of incentives as seen with Milemarker I.

Figure 2.

Student Performance on Milemarker II before and after intervention. (* indicates significant difference; t-statistic = 8.7, p < 0.05).

Passing rates for Milemarker III were consistently higher and increased gradually from 2002 to 2005 by 56.2%. Student performance on Milemarker III examinations was significantly higher (p < 0.05) as compared to that of Milemarker I and II, respectively, during the initial phase, indicating that the high stakes incentive (the stick) had a greater effect than books/certificates (the carrot). Further, Milemarker III passing rates were higher as compared to Milemarker I and II after 2003, indicating, that high-stakes incentives are the best alternatives to changing student behavior and performance. These results can be seen in Figures 3 and 4.

Figure 3.

Comparison of student performance in Milemarker I and Milemarker III before and after the introduction of incentives in 2003. (* indicates significant difference; t-statistic = 18.8, p<0.05)

Figure 4.

Comparison of student performance in Milemarker II and Milemarker III, before and after the introduction of incentives in 2003. (* indicates significant difference between groups; t-statistic = 20.4, p < 0.05; # indicates significant difference between groups; t-statistic = 6.6, p < 0.05).

DISCUSSION

This study evaluated the role of incentives, in the form of rewards or punishment, on pharmacy students' performance on cumulative examinations. This study demonstrated an increasing trend in passing rates after the introduction of carryover points and remediation assignments. Moreover, Milemarker III had a significantly higher passing rate as compared to Milemarker I or II because there were serious repercussions if they failed Milemarker III. Thus, we can say that a high-stakes incentive or a “big stick” is more effective in improving student performance compared to just a “carrot” or a “carrot” along with a small “stick.”

A study published by Oliver²¹ has shown how rewards and punishments may be used as components of collective action. According to that study, selective incentives, in the form of rewards and punishments, may be used to reward those who comply with the required actions and punish those who are noncompliant. Oliver further states that rewards are usually effective when the number of cooperators is small, whereas punishments are effective when unanimous cooperation is desired.²¹ Our study had a combination of both positive and negative incentives in the form of carryover points and remediation assignments, respectively. Furthermore, it validated the results indicated by Oliver that high-stakes incentives work much more efficiently and quickly.

A major goal of this study was to establish an incentive structure to enhance student performance with respect to cumulative examinations. In a study conducted by Sansgiry et al²⁰ to determine the perceptions of PharmD students, the authors concluded that in spite of the increasing importance of cumulative examinations, students had a negative attitude about these assessments. The authors also emphasized the need for intervention strategies to create a positive feeling for students towards these assessments.²⁰ Our study provides evidence of a successful intervention to enhance student performance on cumulative examinations.

Several indirect interventions may enhance student performance. One example of such indirect intervention would be incentives for teachers to improve student outcomes. Eberts et al¹⁸ compared the effects of performance-based compensation systems and merit-based pay systems provided to teachers based on student outcomes. According to this study by Eberts and colleagues, although merit-based pay systems may increase student retention in class, they have no effect on student grade point averages, reduced attendance, or increased student failure.¹⁸ Thus, colleges and schools of pharmacy must introduce appropriate incentives because inappropriate incentives may have negative or no effects.

Our study suggests that student performance has improved over the years. However, more research is required to determine other factors influencing student performance. In a previous study, the authors concluded that students had a slightly negative attitude towards the Milemarker assessments and the novelty of these examinations could have been a factor influencing student attitude.²⁰ The authors of this study suggest that the introduction of incentives may have caused students to take these assessments seriously and may have eventually changed their attitude towards these examinations as well. Future research should evaluate the role of faculty member incentives on student performance.

Limitations

Student performance rates were only followed for 6 years and a longer study period may be required to elucidate the effects of incentives on student performance in order to determine whether the effect remains constant or is reduced. Also, absence of any control group prevented us from evaluating the effects of covariates such as adoption over time on passing rates. This study evaluated the effect of incentives on student performance with respect to only one type of assessment and results may vary with other types of examinations. Moreover, the study only compared passing rates for PharmD students and results may be different for students at different academic levels or in different programs of study.

CONCLUSIONS

High-stake incentives may be effective in improving pharmacy students' performance in cumulative examinations. Introduction of direct incentives may be a better way of improving student performance than indirect incentives for college students. Further research is required to study students' attitudes towards various incentives.