| • Assess multiple tools (e.g., file conversion tool, annotation software) for a given task and select according to task-specific performance. |
| • Aim for a diverse sample on which to develop the annotation guideline and schema. |
| • Text file data should be cleaned and parsed prior to annotation to prevent disagreements based on differences in annotation boundaries. |
| • Collect rationale for each annotator's decisions, particularly during the schema development phase and when an annotator assigns the indeterminate tag. Document resolution. |
| • A single annotator may not be sufficient to identify instances of a given phenomenon amidst lexical variability, ambiguity of meaning, and complexity of the task. Using more than one annotator reduces errors of missingness and enables discussions around boundaries and rules for annotation. |
