Table 3.

Study to Explore Early Development quality control procedures and results

Study contacts and instruments	Type of quality control assessment(s) and requirements	Specific quality control training requirementsa	Ongoing quality control (frequency)	Results ongoing quality controlb
Invitation telephone call, including eligibility screener and Social Communication Questionnaire	Semiqualitative call rating form—a priori criteria established for acceptable score.c	Acceptable scores on 4 role-playing (mock) calls and first 2 “live” calls.	5% per interviewer	88% acceptable rating scores overall, with improvements over time; >90% after first 6 months.
Follow-up call, including structured pregnancy dates questionnaire for caregiver interview	Semiqualitative call rating form—a priori criteria established for acceptable score.c	Acceptable scores on 3 role playing (mock) calls.	5% per interviewer	93% acceptable rating scores overall, with improvements over time; >90% after first 6 months and 100% after first 12 months.
Caregiver interview	Semiqualitative call rating form—a priori criteria established for acceptable score.c Quantitative inter-rater reliability assessment of selected interview items. Acceptable score is ≥95% concordance.	Acceptable scores for both assessments on 3 role-playing (mock) interviews and first 2 “live” calls.	5% per interviewer	99% acceptable call rating scores; 98% acceptable inter-rater reliability assessments throughout study.
Questionnaire packets I, II, and III (self-administered forms)d	No specific systematic QC requirements, but continual supervisor oversight and all forms reviewed for missing or illegible data or contradictory entries. Participants are recontacted as needed.	None. General training provided on forms and appropriate responses to participant queries.	NA	NA
Autism Diagnostic Observation Schedule (ADOS)	Intersite: Supervising clinicians establish reliability by scoring the same ADOS exam videotapes. Acceptable score is ≥80% concordance on algorithm items. Intrasite: All clinicians establish reliability with supervising clinician. Acceptable score is ≥80% concordance on algorithm items	Both intersite and intrasite reliability established in advance of study start.	Quarterly intersite and intrasite reliability exercises	Intersite: 99% acceptable scores on “first pass” quarterly exercises and 100% acceptable scores on “second pass” Intrasite: 99% acceptable scores on “first pass” quarterly exercises and 100% acceptable scores on “second pass”.
Autism Diagnostic Interview-Revised (ADI-R)	Intersite: Supervising clinicians establish reliability by scoring the same ADI-R interview videotapes. Acceptable score is ≥90% concordance on algorithm items. Intrasite: All clinicians establish reliability with supervising clinician. Acceptable score is ≥90% concordance on algorithm items.	Both intersite and intrasite reliability established in advance of study start.	Quarterly intersite and intrasite reliability exercises.	Intersite: 99% acceptable scores on “first pass” quarterly exercises and 100% acceptable scores on “second pass”. Intrasite: 87% acceptable rating scores on “first pass” quarterly exercises and 100% acceptable scores on “second pass”.
Mullen Scales of Early Learning	No specific systematic QC requirements, but continual supervisor oversight and all forms reviewed for missing or illegible data or contradictory entries.	5 clinicians monitor initial assessments until competency determined.	NA	NA
Vineland Adaptive Behavioral Scales	No specific systematic QC requirements, but continual supervisor oversight and all forms reviewed for missing or illegible data or contradictory entries.	None. Supervising site clinicians monitor initial assessments until competency determined.	NA	NA
Dysmorphology physical examination: examination, photography, and anthropometric measurements.	Intersite: Examiners review common set of photos and compare measurements. Acceptable score is ≥80% concordance. Intrasite: Examiners certified as meeting acceptable levels of reliability (80% or higher depending on component) on performance standards in several areas.	Acceptable intrasite scores in all areas on 5 practice examinations and first 3 “live” examinations.	Intersite: 1 exercise /month Intrasite: 10% per examiner	Intersite: 100% acceptable scores on monthly exercises. Intrasite: 95% acceptable scores.
Biologic specimens: buccal swabs and blood specimens (child, mother, father) and hair specimen (child).	All: Central laboratory staff processes specimens upon receipt and performs preliminary QC (gross visual inspection). Sample of participants: Second blood specimen obtained for duplicate processing and analysis	None. Extensive staff training on study protocol for obtaining and processing biologic specimens	2% sample of duplicate blood specimens	Will assess analyte concordance (e.g., genotypes) among duplicates
Medical record abstraction (4 forms: prenatal, labor and delivery, neonatal, pediatric).	Intersite: Common reliability abstraction exercises, each focusing on different sections/ forms. Acceptable score is ≥90% concordance across sites. Intrasite: Quantitative inter-rater reliability assessment of selected items on each form. Acceptable score is ≥90% concordance.	Acceptable intrasite scores on first 2 “live” abstractions for each form type (8 total abstractions).	Intersite: Quarterly reliability exercises. Intrasite: 5% per abstractor (across form types)	Intersite: Exercises revealed minor inconsistencies in abstraction process; no major substantive differences noted. Intrasite: 91–100% acceptable rating scores across the 4 form types

^aTraining QC requirements consisted of requirement for staff to pass formal reliability or other QC assessment on mock exercises in advance of “live” field work and initial QC requirement on first instruments/examinations once in the field.

^bFor each instrument, if a study staff member did not meet criteria for acceptable score during ongoing QC, retraining and training QC requirements were instituted.

^cSemiqualitative call rating forms for invitation, follow-up, and caregiver interview calls included items such as use of call script, coverage of essential points, ability to respond to participant questions, probing on unclear or neutral responses, professionalism, and delivery and response recording for applicable study instruments (Social Communications Questionnaire, pregnancy reference form, or caregiver interview). For each item, QC supervisor rated interviewer as “good”, “fair”, or “poor”. Criteria for acceptable score consisted of: no item rated as “poor” and 20% or fewer rated as “fair”; and mandatory ratings of “good” for select items (dependent on type of call).

^dQuestionnaire packet I included maternal medical history form, paternal medical history form, family autoimmune disease history form, paternal occupational exposure questionnaire, child services and treatments questionnaire, child early development questionnaire, and child gastrointestinal function questionnaire. Questionnaire packet II included Child Behavior Checklist, Carey Temperament Scale, Child Sleep Habits questionnaire, Child Social Responsiveness Scale, and Parent Social Responsiveness Scale. Questionnaire packet III included child diet diary and child stool diary.