Table 4.

Guideline evolution: overall agreement (Kappa) by guideline sequence

Guideline version	Batch	Kappa (p-value)	Guideline revision based on adjudication
0 (none)	1A	0.161* (p = 0.045)	Distinguished between a qualified/not-qualified applicant resumes Defined transferrable skills Defined key terms: clinical role/setting, clinical experience, clinical research experience Set requirements for clinical experience and academic preparation by level
1	2	0.453 (p < 0.001)	Revised years of clinical experience by level Categorized academic degree as scientific/health-related field or nonscientific/non-health-related field Refined clinical research coordinator (CRC) 2, 3, 4-degree requirements, years of clinical research, and clinical experience requirements Required clinical research certification for a CRC 4
2	3	0.407 (p < 0.001)	Updated glossary examples of clinical settings, clinical experience by job titles, and clinical research experience Revised years of experience in clinical roles and years of clinical research experience Developed categorization of foreign-trained MD and bench (laboratory) research personnel
3	4	0.555 (p < 0.001)	Removed clinical role/setting experience requirement from CRC 2–4 and required a minimum of 1-year clinical research experience for those with a scientific degree Revised laboratory (bench) scientist requirements Removed academic experience as a research assistant unless it involved over 1000 contact hours. Rationale: academic experience was typically data entry or participation in a component of the research project but not full engagement in a project
4	5	0.608 (p <0.001)	Increased the number of years required for a laboratory (bench) researcher
5	1B	0.593 (p < 0.001)	Final guidelines

^*Kappa computed for Raters 1 and 2 for Guidelines 0 (Batch 1A), Kappa computed for 3-Raters for remainder of development.