Table 3.
Summary of the measures used by included studies, methods of analysis and results of comparisons
| Study | Proxy measure | Direct Measure (DM) | Analysis | ||||||||
|
Description 1. Method V = Clinical vignette (No. of case simulations) CI/Q = Clinician interview/questionnaire MR = Medical Record review PI/Q = Patient interview/questionnaire 2. Timing |
Clinician self report (SR) | Medical Record Review (MR) | Patient report (PR) |
Description 1. Method SP = Simulated Patients DO = Direct Observation VR = Video recording AR = Audio recording 2. Timing |
SP Training reported | Psychometrics (IRR) | Compared Item by Item | Compared Summary Scores |
Agreement between measures: Co-efficient r; kappa (k); Structural equation modelling (SEM); Sensitivity (Sens) & Specificity (Spec) Difference between mean scores: ANOVA; T-test |
P | |
|
Stange [5] 1998 |
1. MR; PQ 2. At end of consultation |
√ | √ | DO | 0.39 to 1.00 (kappa) | √ | MR Sens = 8% (diet advice) – 92% (Lab tests) Spec = 83% (social history) – 100% (counselling services, physical exam, lab tests) k = 0.12 to 0.92 (79 comparisons) PR Sens = 17% (mammogram) – 89% (Pap test) Spec = 85% (in-office referral) – 99% (immunisation, physical exam, lab tests) k = 0.03 to 0.86 (53 comparisons) |
NR | |||
|
Flocke [6] 2004 |
1. PQ 2. At end of consultation (24%) or postal return (76%) |
√ | DO | NR | √ | Sens* = 11% (substance use) – 76% (smoking cessation) | NA | ||||
|
Wilson [7] 1994 |
1. MR; PQ 2. At end of consultation |
√ | √ | AR | 0.79 to 1.00 | √ | MR Sens = 31%, Spec* = 99% 28.6 (Alcohol) Sens = 29%, Spec* = 100% 83.3 (BP) Sens = 83%, Spec* = 93% % agreement between DM & MR: 45.5 (Smoking) PR Sens = 74%, Spec* = 94% 75.0 (Alcohol) Sens = 75%, Spec* = 94% 100 (BP) Sens = 100%, Spec* = 90% % agreement between DM & PR: 81.8 (Smoking) |
NA | |||
|
Ward [8] 1996 |
1. PQ 2. Questionnaire mailed to patient within 2 days of consultation |
√ | AR | 0.74 to 0.94 (kappa) | √ | Sens = 93% (smoking status) Spec = 79% Sens = 92% (cessation advice) Spec = 82% |
NA | ||||
|
Zuckerman [9] 1975 |
1. MR 2. At end of consultation |
√ | AR | NR | √ | Sens* = 0% (side effects) – 100% (Diagnosis) Spec* = 9% (Diagnosis) – 100% (side effects) |
NA | ||||
|
Luck [10] 2000 |
1. MR 2. At end of consultation |
√ | SP (27) each role-playing 1 of 8 case simulations | √ | NR | √ | √ | ANOVA (4-way) Necessary care: Sens = 70%, Spec = 81% Unnecessary care: Sens = 65%' Spec = 64%. |
<0.0001 NA |
||
|
Page [11] 1980 |
1. V (4) 2. Upto 6 weeks before or 3 weeks after SP visit |
√ | SP (4) each role-playing 1 case simulation | √ | 0.76 | √ | √ | r = .56 & .68 r = .26 & .37 "Must do" actions Sens* = 97%, Spec* = 33% "Must not do" actions Sens* = 30%, Spec* = 98% |
>0.05 <0.05 |
||
|
Gerbert [12] 1988 |
1. CI; MR; PI 2. At end of consultation |
√ | √ | √ | √R | NR | √ | k = 0.67 (SR) k = 0.54 (MR) k = 0.50 (PR) |
<0.001 <0.001 <0.001 |
||
|
Pbert [13] 1999 |
1. CI; PI 2. At end of consultation |
√ | √ | AR. | NR | √ | √ | r = 0.77 (SR) r = 0.67 (PR) |
<0.0001 <0.0001 |
||
|
Gerbert [14] 1986 |
1. CI; MR; PI 2. At end of consultation |
√ | √ | √ | √R | 0.52 to 0.93 (kappa) | √ | Median % agreement (All categories): 0.84 (SR) 0.88 (MR) 0.86 (PR) |
NA | ||
|
Dresselhaus [15] 2000 |
1.V (8); MR 2. NR |
√ | √ | SP (4) each role-playing a simple and complex case presentation | √ | NA | √ | ANOVA (3-way) | <0.01 | ||
|
Rethans [16] 1987 |
1. V (1). 2. Completed 2 months after SP visit |
√ | SP (3) each role-playing same case simulation | √ | 0.78 to 1.0 (kappa) | √ | √ | T-test: Overall "Obligatory" "Intermediate" "Superfluous" |
ns <0.005 <0.05 <0.05 |
||
|
Rethans [17] 1994 |
1. MR 2. Charts reviewed two years after SP visit. |
√ | SP (4) each role-playing 1 of 4 case simulations | √ | 0.93 (kappa) | √ | √ | r = 0.54 (Overall) r = 0.17 (History taking) r = 0.45 (Physical exam) r = 0.75 (Lab exam) r = 0.50 (Advice) r = 0.43 (Medication) r = -0.04 (Follow-up) |
<0.05) ns ns <0.01 <0.05 ns ns |
||
|
Peabody [18] 2000 |
1. V (8); MR 2. Completed "several weeks" after SP visit |
√ | √ | SP (4) each role-playing a simple and complex case presentation | √ | NA | √ | ANOVA (4-way) | <0.001 | ||
|
O'Boyle [19] 2001 |
1. % time practiced hand hygiene 2. Up to one month prior to observation period |
√ | DO Nurses observed for 2 hours or until 10 indications for handwashing had occurred |
0.94 to 0.98 | √ | r = 0.21 SEM = 0.201 |
<0.05 <0.05 |
||||
* Calculated by authors NA = Not applicable NR = Not reported ns = non-significant