Table 3.

Strengths and challenges in the test development process

Strengths
Project group	Consisted of professionals with profound content knowledge, a medical educationalist and a statistician with experience in test-development.
Test content	Based on nationally defined learning objectives, which generated relevant and coverable test content.
Test blueprint	Predefined and based on nationally developed learning objectives.
Test format	MCQ’s, which can test more than simple facts, is suitable for large groups and time- and cost effective. Assess competences at the two lower levels of Millers triangle, knows and knows how.
Language	Predefined spelling and abbreviations ensured consistency in wordings and terms.
Proofreading	Several proofreaders. Proofreading of content, language and structure/format.
Pilot test participants	A large sample representing in part the intended test-takers.
Pilot testing	Written and verbal feedback gave insight into the pilot participants’ thought processes during testing.
Standard setting	An acknowledged method was used. The passing score was adjusted to minimize false-positive values and was validated on initial test responses.
Psychometric properties	Evaluated on both pilot test responses and the responses from the real test-takers.
Test-takers	A high number of participants enabled the use of advanced statistical analyses such as Rasch analyses.
No. of options in each item	Three or four options were chosen dependent on the numbers of plausible distractors.
Challenges
Test format	A written assessment cannot assess competences on the two higher levels of Millers triangle, shows how and does (i.e. clinical performance).
Number of items	More items would expectedly have increased reliability and would have allowed for the development of an item bank.
Item difficulty	Items of a higher difficulty would expectedly have increased reliability and entailed a more challenging test.
Pilot test participants	Medical and midwifery students did not represent the intended test-takers and lowered the percentage of correct answers.
Relations to other variables	There was no test available for comparison.
Context	The context of pilot testing and real testing differed; pilot participants did not attend a one-day teaching course prior to testing and the test was therefore more challenging than in the real setting.
Time devoted for assessment	More items and items with a higher difficulty require more time devoted for assessment in an education program.