Novice
An individual with no experience rating behaviors according to standardized assessments, and no subject matter expertise that is relevant to the domain under study.
|
Generally inexpensive to recruit. Scheduling is often flexible. May be less vulnerable to profession-based stereotype effects (i.e., where ratings are based on the learners’ inclusion in a group, such as medical student or nurse, rather than their own unique behaviors; 15).
|
May be challenging to train and retain (e.g., if the rater does not have sufficient knowledge or commitment related to the goals of the assessment). May not be ideal for obtaining accurate, nuanced, and reliable ratings depending on the purpose of the assessment Case examples of this can be found in the following studies (19–22).
|
Expert/Experienced
An individual with moderate to extensive experience rating behaviors according to standardized assessments (preferably specific ones used in your assessment paradigm), but no subject matter expertise that is relevant to the domain under study.
|
Typically, they are less expensive to recruit and retain compared to SMEs. Have demonstrated superiority to novice raters, even when the novice raters were acclimated through basic frame-of-reference training (20).
|
Often require more resources to recruit and retain compared to novice raters. This type of expertise may not be sufficient to establish reliable and valid ratings, particularly if the assessment is highly domain specific and may require subject matter expertise to effectively differentiate behaviors.
|
Subject Matter Expert (SME)
An individual who has subject matter expertise relevant to the domain under study (e.g., a medical degree), but typically does not have prior experience rating behaviors according to standardized assessments.
|
May be more apt at discriminating nuanced behaviors relevant to their domain (21). More readily able to grade the accuracy of behaviors that require domain expertise, such as whether an appropriate diagnosis and treatment was selected, which may be challenging for non-SMEs to recognize even when given extensive training (15).
|
Typically, they are very expensive to recruit, train, and retain. Schedules are often highly restrictive. Will likely still require training, as evidence demonstrates that even among SMEs, those who are also experienced raters with training in using specific observational tools are more consistent in assessments relative to SMEs who are not experienced raters (19, 22).
|
