Table 3.

Methods of validation for the USP cases

Domain	Indicator	Data collection		Statistical analysis
		Phase	Method
Content validity	Content Validity Index (CVI)	USP case review	Expert panel review of SP cases, measured by a 4-point Likert scale (1=lowest, 4=highest).	CVI for SP case and for specific USP, where CVI=number of raters giving a rating of 3 or 4 divided by the total number of raters.
Face validity	Authenticity of SP role-play	Validation study	Transcripts of the recording of the USP–clinician encounter to be assessed by a member of the project team for accuracy of portraying the clinical case by a 5-point Likert scale (1=100% inaccurate, 5=100% accurate).	Accuracy score=per cent of positive evaluations (ie, evaluation ≥4).
	Detection ratio		Clinicians receiving an SP visit to complete a ‘detection form’ afterwards to report any suspected USP visits: 0=not suspected; 1=somehow suspected; 2=suspected with certainty).	Detection ratio=number of detected USP visit divided by the total number of USP visits (for case-specific detection ratio and all-case detection ratio, respectively). Detection ratio of 10% and less is considered acceptable.
Criterion validity	Lin’s concordance correlation coefficient (rc); kappa statistic	Validation study	SP-completed checklist against that by a clinician based on the transcript of the visit (ie, the clinician rating as the ‘reference standard’).	The concordance of the quality scores based on SP-completed checklist against that based on the reference standard. rc used for continuous process quality scores, and kappa for dichotomous diagnoses and treatment and management measures.
Test–retest reliability	Lin’s concordance correlation coefficient (rc); kappa statistic	Validation study.	The same SP to score his own recorded encounter in a month.	The concordance to be examined by rc for continuous process quality scores, fees charged (yuan) and time spent (min), and kappa for dichotomous diagnoses and treatment and management measures.
Inter-rater reliability			Multiple SPs to complete the checklist for the same recorded transcript.

SP, standardised patient; USP, unannounced standardised patient.