Table 3.
Performance of assessment items (faculty content validation, psychometric scores, recommendations)
| AO Principles in Small Animal Fracture Management - Pre |
Faculty difficulty index (Dif I) and rating (n=36) |
Participants Psychometric tests |
Quality and Recommendations |
||||||
|---|---|---|---|---|---|---|---|---|---|
| Dif I | % Good | % OK | Dif I | Dis I | NFD | ||||
| C1-Q1 | 93% | 89% | 8% | 73% | 0.24 | 1 | Easy item with 1 NFD – possibly revise | ||
| C1-Q2 | 95% | 85% | 15% | 73% | 0.18 | 0 | |||
| C2-Q1 | 100% | 57% | 34% | 69% | 0.25 | 0 | |||
| C2-Q2 | 94% | 77% | 20% | 74% | 0.15 | 1 | Easy item with 1 NFD - possibly revise | ||
| C3-Q1 | 92% | 82% | 12% | 47% | 0.21 | 1 | Moderately difficult item with 1 NFD - possibly revise | ||
| C3-Q2 | 97% | 88% | 9% | 32% | 0.19 | 0 | |||
| C4-Q1 | 75% | 54% | 29% | 50% | 0.24 | 0 | |||
| C4-Q2 | 92% | 74% | 16% | 42% | 0.16 | 1 | Moderately difficult item with 1 NFD – possibly revise | ||
| C5-Q1 | 89% | 84% | 13% | 87% | 0.13 | 1 | Easy item, similar Dif I of faculty and participants - revise | ||
| C5-Q2 | 80% | 70% | 21% | 25% | 0.24 | 0 | Difficult item - keep | ||
|
Average |
91% |
76% |
18% |
57% |
0.20 |
- |
|
||
|
AO Principles in Small Animal Fracture Management - Post |
Faculty difficulty index (Dif I) and rating (n=23) |
Participants Psychometric tests |
|
||||||
|
Dif I |
% Good |
% OK |
Dif I |
Dis I |
NFD |
|
|||
| C1-Q1 | 80% | 78% | 13% | 91% | -.01 | 2 | Easy item, negative Dis I and 2 NFDs - revise | ||
| C1-Q2 | 92% | 88% | 12% | 71% | 0.16 | 1 | Moderately easy item with 1 NFD – possibly revise | ||
| C2-Q1 | 96% | 96% | 4% | 89% | 0.21 | 2 | Easy item with 2 NFDs - revise | ||
| C2-Q2 | 100% | 92% | 8% | 96% | 0.21 | 3 | Easy item with 3 NFDs - revise | ||
| C3-Q1 | 96% | 91% | 9% | 51% | 0.10 | 0 | |||
| C3-Q2 | 92% | 63% | 29% | 91% | 0.08 | 3 | Easy item, similar Dif I of faculty and participants, poor Dis I, 3 NFDs - revise | ||
| C4-Q1 | 79% | 79% | 21% | 63% | 0.05 | 0 | Average item with poor Dis I - revise | ||
| C4-Q2 | 70% | 83% | 8% | 94% | 0.15 | 2 | Easy item, participants Dif I higher that faculty, 2 NFDs - revise | ||
| C5-Q1 | 96% | 91% | 9% | 55% | 0.02 | 1 | Average item with poor Dis I - revise | ||
| C5-Q2 | 71% | 64% | 27% | 45% | 0.14 | 0 | |||
|
Average |
87% |
83% |
14% |
75% |
0.11 |
- |
|
||
|
AO Management of Facial Trauma - Pre |
Faculty difficulty index (Dif I) and rating (n=138) |
Participants Psychometric tests |
|
||||||
|
Dif I |
% Good |
% OK |
Dif I |
Dis I |
NFD |
|
|||
| C1-Q1 | 95% | 84% | 13% | 74% | 0.14 | 2 | Moderately easy item with 2 NFDs - revise | ||
| C1-Q2 | 68% | 78% | 18% | 45% | 0.02 | 1 | Moderately difficult item with poor Dis I, 1 NFD - revise | ||
| C2-Q1 | 71% | 69% | 20% | 47% | 0.12 | 0 | |||
| C2-Q2 | 91% | 74% | 19% | 66% | 0.25 | 0 | |||
| C3-Q1 | 95% | 87% | 10% | 82% | 0.17 | 2 | Easy item with 2 NFDs - revise | ||
| C3-Q2 | 91% | 83% | 13% | 63% | 0.13 | 2 | Average item with 2 NFDs - revise | ||
| C4-Q1 | 81% | 73% | 17% | 50% | 0.15 | 0 | |||
| C4-Q2 | 79% | 74% | 17% | 57% | 0.16 | 1 | Average item with 1 NFD – possibly revise | ||
| C5-Q1 | 47% | 42% | 35% | 27% | 0.14 | 1 | Difficult item with 1 NFD – possibly revise | ||
| C5-Q2 | 51% | 50% | 28% | 26% | 0.10 | 1 | Difficult item, poor Dis I, 1 NFD - revise | ||
| C6-Q1 | 82% | 85% | 13% | 58% | 0.26 | 1 | Average item with 1 NFD – possibly revise | ||
| C6-Q2 | 76% | 81% | 15% | 42% | 0.18 | 0 | |||
| Average | 77% | 73% | 18% | 53% | 0.15 | - | |||
C1-Q1 = Competency 1 - Question 1
Good = It’s good (clear and fair) - keep it, OK = It’s average - keep it or make minor changes, Dif I = difficulty index, Dis I = discrimination index, NFD = non-functioning distractor
Easy items (Dif I >80%), moderately easy (Dif I 71%-80%), average (difficulty index, Dif I 50%-70%), moderately difficult (Dif I 30-50%) and difficult (Dif I <30%).