Table 3.
Usability issues identified in each UCD iteration’s usability testing.
| Iteration | Theme | Usability issues | Nielsen’s 10 usability heuristics |
|---|---|---|---|
| 1 | Unclear information presentation | Hard to find the guideline button | Recognition rather than recall |
| Incomprehensible concepts | Hard to understand some concepts used in STAT (e.g., entity, relation, and semantic triple) | Help and documentation | |
| Lack of functionality | Could not create a new relationship if no relation has been identified | Flexibility and efficiency of use | |
| Unsure how to create complex triplets when there are multiple associations presented | Flexibility and efficiency of use | ||
| The annotation box appears every time by default; and there is no way to disable it | Flexibility and efficiency of use | ||
| Cannot change the size and format of the text | Flexibility and efficiency of use | ||
| 2 | Unclear information presentation | Hard to find the menu button, because it was placed close to the STAT logo | Recognition rather than recall |
| Hard to see existing (defined) semantic triples | Recognition rather than recall | ||
| Tagging “Previous Abstract” and “Next Abstract” is redundant and somewhat bothering the participants to focus on the annotation work | Aesthetic and minimalist design | ||
| Hard for a user to find the setting option for the “normalization popup window” | Recognition rather than recall | ||
| Lack of functionality | Could not change the “entity” or “relation” type in the popped window | Flexibility and efficiency of use | |
| Difficult to find relations and put them into the semantic triple form when there were more than two entities being related | Flexibility and efficiency of use | ||
| The font size of the entities that have been annotated will not change after adjusting the font size setting | Consistency and standards | ||
| 3 | Unclear documentation and annotation guideline | Confusion between entity and relation | Help and documentation |
| Lack of an overall cohesive explanation of the point of the task explained in very simple terms | Help and documentation | ||
| The normalization step was unclear | Help and documentation | ||
| Learning of the functionalities was a little hard | Help and documentation | ||
| Users were not certain whether performing the task correctly, despite reviewing the video and guidelines several times | Help and documentation | ||
| Lack of functionality | Hard to remove a term normalized in error | Help users recognize, diagnose, and recover from errors | |
| Bugs | The definition box froze and needed to refresh the page | Error prevention | |
| 4 | Incomplete functionality | Hard for users to highlight things precisely since they would include the leading or trailing space when highlighting things. | Flexibility and efficiency of use |
| Unclear documentation / guidelines | Lack of examples | Help and documentation | |
| Hard to understand what terms are fitting for criteria | Help and documentation | ||
| Hard to construct a relationship | Help and documentation | ||
| Hard to understand the basic term (e.g., entity and relation) | Help and documentation | ||
| Unclear information presentation | Finished triples were not visible properly | Flexibility and efficiency of use | |
| Bugs | The triples in the queue are not added or removed when moving to the next sentence | Error prevention |