Progress Examinations in Pharmacy Education

Abstract

Interest in the use of the progress examination has grown in the current culture of accountability in higher education. The Accreditation Council for Pharmacy Education's (ACPE's) Standards 2007 calls for comprehensive, knowledge- and performance-based examinations as part of a school or college of pharmacy's evaluation and assessment of student learning. Progress examinations have been used primarily in medical education. The purpose of this paper is to provide a brief overview of the literature on progress examinations and considerations for their potential use within an effective assessment plan.

INTRODUCTION

Anderson and colleagues suggested that assessment should have the dual purpose of accountability and the improvement of student learning as well as the 3 overarching characteristics, “…(1) it consists of a systematic and continuous process; (2) it emphasizes student learning, with the cornerstone being what students can do; and (3) it focuses on the improvement of educational programs.”¹ Assessment measures should be used for continuous curricular improvement and employ a variety of valid and reliable systematic measures that are both summative and formative in nature.^2-3 Outcomes assessment involves collecting information on a desired outcome and comparing it to previously established mission statements, goals, and objectives.⁴ Outcomes assessment is a critical component of accreditation in that it has strengthened accreditation and accreditation has likewise sustained the assessment movement, thus creating a synergistic relationship.⁵ There has been an increased focus on the accountability component of assessment and accreditation in higher education as evidenced by the publications from the US Department of Education and the National Association of State Universities and Land-Grant Colleges (NASULGC).^6-8 Within this increased culture of accountability, there has been a push for greater standardization, and within this movement towards standardization is the issue of the use of progress examinations. ACPE Standards 2007, Guideline 15.1 states that a school or college of pharmacy's evaluation of student learning should, “…incorporate periodic, psychometrically sound, comprehensive, knowledge-based and performance-based formative and summative assessments, including nationally standardized assessments…that allow comparisons and benchmarks with all accredited and peer institutions.”² While there are several key foundational resources available within pharmacy education that address the issue of assessment, there is little within these resources on the use and implications of progress examinations.^3,4,9-11 The purpose of this paper is to provide an overview of the literature on progress examinations within the health sciences and considerations for their potential use within an effective assessment plan.

A progress examination can be defined as a method of assessing both the acquisition and retention of knowledge at one or more points in the curriculum relative to curricular goals and objectives.¹² Progress examinations have been suggested for a variety of uses: peer-comparisons among schools (also referred to as benchmarking in the educational literature), comparisons among students to identify those who could benefit from remediation, formative assessment as part of an overall assessment plan, high-stakes assessment to determine progression in the curriculum, low-stakes assessment, and as an adjunct to program evaluation.

OVERVIEW OF THE LITERATURE

In undertaking any assessment endeavor, such as progress examinations, there are numerous elements that must be considered, chief of which are reliability and validity.^13-14 A glossary of terms related to reliability and validity is provided in Appendix 1. While Standards 2007 have brought the issue of progress-type examinations back into the forefront, the Basic Pharmaceutical Sciences Exam (BPSE) developed by Pharmat, Inc, was used by some schools and colleges of pharmacy in the 1980s as a progress examination prior to entry into the final year of the curriculum.¹⁵ The BPSE which covered pharmacology, toxicology, pharmaceutics, and medicinal chemistry, was designed to measure pharmacy student knowledge relative to both the content and goals in the preclinical curriculum.¹⁵ Fassett and Campbell examined the BPSE as a potential predictor of student performance during clinical training but found there was no correlation between scores on the BPSE and performance in the “clinical” coursework or with performance on experiential rotations.¹⁵ However, the authors concluded that the BPSE appeared to serve as a comprehensive evaluation of basic science knowledge in that there was a strong correlation with BPSE scores and basic pharmaceutical sciences course grades (Spearman's rho = 0.75, p = 0.001) and therefore could be used for national comparison among institutions in aggregate. The BPSE fell out of use during the 1980s and is no longer available for use by colleges and schools of pharmacy.

Kirschenbaum and colleagues examined programmatic curricular outcomes assessment at schools and colleges of pharmacy in the United States and Puerto Rico and found that 4 institutions used an end-of-semester comprehensive written examination that did not affect course grades, 13 used a “high-stakes” end-of-year examination, and 17 used a “low-stakes” examination to assess curricular outcomes (N = 68).¹⁶ Locally developed examinations have been used at individual schools and colleges pharmacy such as the Milemarker Assessment at the University of Houston, which is a case-based, multiple-choice progress examination administered at the conclusion of each didactic year in the professional curriculum.¹⁷

Use of Progress Examinations

Progress examinations have been primarily used in medicine to assess knowledge. Newble and Jaeger examined the effect of assessments and examinations on learning in medical students at the University of Adelaide in Australia following a change in their curriculum to include both an experiential component in the curriculum and low-stakes clinically based assessments in the final year that did not influence progression in addition to the existing year-end high stakes multiple-choice examinations.¹⁸ The year-end high-stakes multiple-choice examination served as the basis for pass or fail decisions in the final year of the curriculum. They cautioned that, “Should the examination system be seen by the students to require predominantly recall of factual information then they will tend to adopt a surface-level or rote-learning approach.”¹⁸ Based on questionnaires sent to recent graduates, they found that the study habits of students did indeed change to conform to the format of the year-end multiple-choice progress examination (ie, attempting to study for the examination through memorization) which had items written at the recall level of knowledge rather than spending time in clinical experiences offered to students. This was found to be an unintended consequence of using the high-stakes examination, which in essence competed with a low-consequence but high-learning situation of clinically-based assessments. A mismatch between the educational objectives and the operationalization of the assessment program can result in a hidden curriculum based on the assessment.^18-19

Blake et al examined the psychometric properties of the progress examination introduced into McMaster University's medical curriculum in Canada as well as the effect on student learning.²⁰ The progress examination was intended primarily for formative student self-assessment rather than as a peer-comparison measure with other schools or as a high-stakes assessment. The progress examination consisted of the same 180 multiple-choice items administered 3 times per year to all classes. To assess the psychometric properties of the progress examination, the researchers examined the reliability across multiple administrations as well as construct validity. The reliability estimates increased the more times a student had taken the examination suggesting a potential test-retest threat to internal validity. The stated overriding construct of the progress examination was that it was, “…capable of demonstrating consistent progress over time, reflecting increased knowledge of students.”²⁰ However, this was based on the assumption that the total score really represented knowledge versus increased test-taking savvy or other sources of construct-irrelevant variance. A 6-item questionnaire was administered to students to ascertain their perceptions on how, if at all, the progress examination affected their approach to learning. Students gave a low rating to the progress examination as a factor in changing their approach to learning.

Van der Vleuten et al described the use of a progress examination designed specifically for problem-based learning curriculum at Maastricht medical school in the Netherlands.¹⁹ They suggested that progress examinations should share some common features: (1) the material covered is sufficiently comprehensive so that students cannot study specifically for the examination; and (2) student assessment is based on successive overall performance on progress examinations rather than performance on a single examination. The examination consisted of 250 true/false items written by faculty members and included an additional selection choice of “I don't know” intended to minimize guessing. A new examination was constructed with items compiled from faculty members, for each of the 4 administrations during an academic year. The examination was administered to all students regardless of year in the curriculum. To compare across examinations, scores were expressed as percentages since the selection of the “I don't know” option did not count in the calculation of the total score which was calculated based on the number correct minus the number incorrect. Since the expression of scores as a percentage is not test equating in the measurement sense, it was not possible to determine whether the various iterations of the examination were truly equivalent in terms of difficulty and content. In a related study, researchers at Maastricht medical school attempted to establish the potential role and value of knowledge-based progress examinations in medical education and examined the convergent validity of the progress examination relative to clinically based assessments.²¹ Acknowledging the dissonance between the level of learning addressed in a progress examination (knowledge/recall) and the educational goals or instructional methods of the curriculum, the progress test correlated with a clinical reasoning test.

Remmen and colleagues suggested that a written progress examination could be used as an alternative to a more time- and personnel-intensive performance-based Objective Structured Clinical Examination (OSCE).²² This study compared the performance of 106 medical students on a 132-item true/false/“don't know” progress examination with their performance on an OSCE consisting of 12 stations that were each 13 minutes in duration and staffed by medical school faculty members. The OSCE was graded using both a checklist and an overall global rating. Both assessments covered basic physical diagnostic and therapeutic skills. The correlation between the performance on the written progress examination (broken down by school) and the performance on the OSCE (broken down by grading method) ranged from 0.35 to 0.48 which the researchers then corrected for attenuation with a resulting range of 0.64 to 0.87. No p values were reported for any of the correlations between the 2 types of assessments. The authors suggested that the reported reliabilities from the written progress examination of 0.32 for Antwerp students and 0.58 for Ghent students were too low for summative assessment but could be improved through adding more items. The Spearman Brown Prophesy formula reveals that the effect of doubling the number of items to 264 items only predicts an increase in the reliability from 0.32 to 0.48 for 1 of the schools. Thus, the relative gains in reliability by increasing the number of items tends to be modest. The results of this study also need to be considered in light of their small sample size.

The use of progress examinations in combination with OSCEs for formative student self-assessment has been described as an adjunct to program evaluation at the Ohio University College of Osteopathic Medicine.¹² The 6-hour progress examination consisted of 330 multiple-choice items with 5 response choices administered twice annually across the entire 7-year curriculum. While all students in all years take the same examination each time it is administered, a new set of examination items are used for each administration. The OSCE, administered only to fourth-year medical students, was not used for formal grading or advancement determination purposes. At the time of publication, the progress examination has been administered twice and the OSCE once. They concluded that using the 2 different forms of assessment together provided more valuable feedback to students as well as to the institution, with the written progress examination providing information on accumulated knowledge and the OSCE providing information on accumulated clinical and interpersonal skills.

Researchers at Utrecht medical school in the Netherlands proposed a progress test using short answer questions as an alternative to true/false examinations.²³ The short answer examination consisted of 40 cases with both clinical and basic science components administered 3 times a year during the final 2 years. The examination was based on the concept of mastery learning where the goal is to have all students achieve some predetermined level, in this case a score of 80% on at least 3 separate administrations of the examination. The examination was developed using a blueprint to establish content validity and then assessed for face validity using a committee of experts. Since the examination was based on mastery learning, students could sit for the examination as many times as necessary over the 6 possible administrations until mastery was achieved. The primary disadvantage to this form of progress testing is the time required to grade the responses and develop psychometrically sound items.

Use of Progress Examinations for Benchmarking

Medical schools in the Netherlands have also been involved in using progress examinations in benchmarking among institutions as well as collaboration among institutions in the development of the examinations.^24-25 The Maastricht Progress Test was piloted as a method of international benchmarking across medical schools in Europe.²⁴ The Maastricht Progress test consisted of 250 true/false/don't know items administered 4 times annually.^19,24-25 For international benchmarking, the method of administration varied among schools so that for some students the examination was voluntary without any credit awarded, for other students the examination was voluntary but credit was awarded, and for yet other students the examination was mandatory.²⁴ This led to construct-irrelevant variance, limiting the interpretation of the comparisons. Led by the Maastricht medical school, a partnership was established among 3 medical schools, with a fourth medical school purchasing the examination for the development and administration of a progress examination to achieve some economy of scale.²⁵ The authors noted that, unlike the United States, the Netherlands does not have a national medical licensure examination so this collaboration could be considered a partial move towards a national examination, but one where schools themselves retained ownership and control over the process.²⁵ A centralized review committee that included students was established to evaluate test items with the goal of each institution contributing 300 items. It took several test cycles before there was parity among the institutions in the contribution of usable items. Plans for the future included moving away from a true/false format in favor of multiple-choice in addition to less emphasis on factual knowledge as well as the need for expertise in methodologies to equate tests across administrations since differences in difficulty levels was a concern. While not mentioned in this paper, item response theory could be used to place different forms of an examination on the same scale to allow for comparisons among students of the similar overall ability such as those in the same professional year (horizontal equating) and students of different abilities such as those in different professional years (vertical equating).²⁶

DISCUSSION

There are several potential advantages to using progress type examinations. One advantage is the opportunity to have students review material more often (eg, reviewing materials prior to progressing to the next professional year with progress examination given annually).²⁷ A related potential advantage is that progress examinations can provide implicit emphasis on the cumulative nature of pharmacy education.²⁷ Progress examinations can also help colleges and schools identify students requiring remediation when used for formative assessment, as well as for assessing the curriculum overall.²⁷

While progress tests have many potential advantages in an assessment plan, they are not without disadvantages. One major concern with the use of progress examinations is the issue of consequential validity: the consideration of the potential effect the assessment in question can have on learning.^18-19,21 Knowledge-based examinations written at the factual or recall level of learning can potentially encourage memorization over higher levels of learning as students are not required to integrate, apply, synthesize, or evaluate information. There is also the potential to create a disconnect between having a knowledge-based examination while promoting lifelong learning. Whether a sufficiently comprehensive examination prevents studying to the examination is debated.

There are numerous sources of construct-irrelevant variance such as the difficulty in controlling conditions of administration and differences in curricular sequencing across institutions.²⁸ Controlling conditions of administration is of special concern when the progress examination is being considered for use in benchmarking with other institutions when there are different consequences for students at various institutions, which will affect how a student approaches the examination (eg, credit awarded for participation in the examination versus an examination required for progression). Different curricular sequencing such as year round versus traditional semester programs can also affect the ability to use the examination for benchmarking across institutions, reflecting opportunity to learn issues and potential recency effects.

Item format also needs to be considered with any examination administration since there are advantages and disadvantages with each item type.^29-34 True-false items, for example, tend to focus exclusively on recall of factual knowledge, can potentially reinforce retention of false information that is then difficult to unlearn, and are highly prone to guessing.³¹ There is debate over the value of attempting to control for guessing as potential gains in reliability and validity are modest and are based on the assumption that guessing is “blind” which is often not the case.³⁵ Numerous additional questions should also be considered prior to implementing a new assessment plan as presented in Table 1.^33-34 Each type of validity evidence should be considered prior to the implementation of a progress examination as part of an assessment plan.

Table 1.

Different Types of Validity Evidence for Educational Assessments^*

^*Adapted from Nitko³³ and Messick³⁴

As the Academy considers the use of progress examinations for assessment purposes, the emphasis these examinations receive within an assessment plan is of particular concern. Making a progress examination high-stakes could be seen as a move towards a United States Medical Licensing Examination (USMLE) Step 1 type of examination. The USMLE Step 1 examination, sponsored by the Federation of State Medical Boards and the National Board of Medical Examiners, is used by medical schools to assess the first 2 years of the medical curriculum which concentrate on the basic sciences and a passing score is required for progression.³⁶ The USMLE Step 1 examination has been criticized as dictating the content of the first 2 years of medical school curricula so that students are prepared for the material on the examination, thus limiting the amount of innovation possible in curricular structure. The cost associated with a multi-stage licensure process similar to that used in medicine would also be a major consideration of the ability of students to shoulder an increased financial burden. Use of multiple assessments such as knowledge/content/recall examinations in addition to clinical skills evaluations such as OSCEs may collectively provide more information on student learning, educational methods, and institutional comparisons. The use of multiple assessments must be balanced with regard to the benefit of the information provided versus the increased cost of administering different assessment. The use of multiple assessments must be balanced: the benefit of the information provided must be weighed against the increased cost of administering different assessments.

Appendix 1. Glossary*