TABLE 3.

Rubric for “How does a vaccine work?” prompt with excerpts from student responses that contain that component, along with raw agreement levels and Cohen’s kappa

	Criteriona	Raw agreement	Cohen’s kappab
Basic knowledge
Pathogen-likeness	A vaccine contains something that is part of or is shaped like the pathogen, including the pathogen itself or a weakened or modified version of it.“A certain vaccine contains a small dosage of the pathogen and it is injected into our bodies.”—PH student	95%	0.90
Immune activation	A vaccine stimulates an immune response.“I think vaccines are something like a weakened strain of a pathogen or their protein markers or something. The immune system detects them and starts making white blood cells and antibodies to combat it.” —PH student	94%	0.85
Prevention	Vaccines function mostly to prevent disease caused by a pathogen or lessen the disease’s severity in the future.“I guess the body familiarizes itself with it so if you face it again it won’t affect you.” —EBM student	90%	0.76
Accuracy
Accurate	None of the ideas present are contradicted by authoritative sources on how vaccines work.	91%	0.69

^aFor the completeness criteria, the parts of the prompt that fulfill the criterion are bolded.

^bCohen’s kappas of 0.60–0.80 are considered to be “substantial” agreement” and 0.80–1 to be “excellent” agreement (Landis and Koch, 1977).