Table 2.

Main validation domains of the study

Domain	Indicator	Data collection		Statistical analysis
		Phase	Method
Content validity	Content validity index	VP case review	Evaluations by an expert panel after reviewing VP cases, measured by a 4-point Likert scale (1=lowest, 4=highest).	CVI for VP case and for specific VP domain will be computed, where CVI=number of raters giving a rating of 3 or four divided by the total number of raters.
Feasibility	Willingness to participate; Adherence rate	Feasibility study	The subsample of clinicians’ interactions with the two VP cases will be recorded by the online assessment	Willingness to Participate=clinicians taking the VP tests divided by the percentage of clinician selected Adherence rate=clinicians completed two VP cases divided by the percentage of clinicians taking VP tests
Face validity	Satisfying score		Clinicians’ subjective attitude towards the VP test experience measured by a 5-point Likert scale (1=most negative, 5=most positive).	Satisfying score for VP case and for specific aspects (eg, usability, accessibility, etc) will be computed, where satisfying score=frequency multiply by positive evaluations (3–5) and scores≥1.5 are considered acceptable.
Criterion validity	Concordance correlation coefficient (rc); Kappa statistic	Validation study	The same clinician receives a USP visit and a VP test for a matching condition. The USP-clinician interaction is evaluated by the USP using the checklist, including fees and time per visit; while VP-clinician interaction is graded by the system.	The concordance of VP-test scores against USP-test score (gold standard) or two-repeated VP-tests will be examined by rc for continuous process quality scores, fees charged (yuan) and time spent (min) and Kappa for dichotomous diagnoses and treatment and management measures.
Test-retest reliability			Repeat VP-tests on the same clinician in a month
Internal consistency	Cronbach’s alpha coefficient (α)		VP-test scores on a single occasion	Intercorrelation of scores for process quality indicators with α>0.7 is acceptable.

CVI, content validity index; VP, virtual patient.