Table 2.

Agreement analysis between LLMs and human performance.

	Kappa (95% CI)
ChatGPT-4 vs. Human Examinees (Correctness)	0.34 (0.26, 0.43)
Claude 3.7 Sonnet vs. Human Examinees (Correctness)	0.39 (0.3, 0.48)
ChatGPT-4 First vs. Second Attempt	0.66 (0.59, 0.73)
Claude 3.7 Sonnet First vs. Second Attempt	0.68 (0.61, 0.76)
ChatGPT-4vs Human Majority (Answer Choice)	0.50 (0.45, 0.56)
Claude 3.7 Sonnet vs. Human Majority (Answer Choice)	0.63 (0.57, 0.68)

Cohen’s kappa coefficients with 95% CI. Correctness agreement compares binary correct/incorrect classifications. Answer choice agreement compares selected options with the human majority response. Self-consistency measured agreement between LLM attempts.

CI confidence interval, LLM large language model.