Table 5:
Six Factors in Cognitive Task Analysis Quality45
| Trait | Description | Evaluation of our HTA |
|---|---|---|
| Valid | • Construct
validity: appropriate choice of methodology, reliance on
domain practitioners as a source of information (either as the focus of
analysis, or as members of the analysis team), observation of work
practices and inspection of tools and artifacts in the work domain
itself • Face validity: whether the relationship between the goals of the analysis and the analysis methods is apparent. |
• Construct
validity: demonstrated by using HTA as the appropriate
methodological approach for the goals of this research, recruiting
participants that were older adults themselves, and collecting data in
their natural work environment, which was studied closely in pilot work
while developing the Senior Section intervention.13,22 • Good face validity: the goals of this analysis (evaluating the Senior Section) and the analysis method (analyzing older adults as experts who need to use the Senior Section) align |
| Reliable | • Redundancy - did the analyst interview enough people, see enough tasks performed, watch activities under enough conditions that he or she started hearing and seeing the same things repeated, without novel information coming to light? | • This HTA is reliable: the analysis stopped only once data saturation was achieved and two coders participated in the analysis |
| Generalizable | • Did the analyst explicitly describe the systems aspects to which the analytic results generalize, and support that argument by making links from the participants and settings in which the analysis took place, to those of interest? | • This HTA is generalizable to older adults that live in this region and shop for OTC medications in a community pharmacy chain. Future work is needed to apply this HTA to other groups of older adults to ensure this output is generalizable to all older adults. |
| Scope | • Is scope of the analysis appropriate for the design goals? | • The scope is appropriate: the HTA covered the entire decision-making process from experience symptoms to selecting an OTC to treat the medication. We are able to generalize the scope of the HTA diagram to the entire shopping experience surrounding the Senior Section. |
| Complete | • It is usually impossible to
guarantee completeness of coverage – not every variable,
function, task, component, decision, etc. is going to be identified or
addressed. • Consider the constraints and challenges inherent in the work domain, and the knowledge, skills, and strategies that expert practitioners bring to bear on those problems. |
This HTA output is complete: variety of goals and subgoals have been identified, recognizing that some goals may have been missed because we did not recruit participants that consider alternative decision processes (e.g., demographically different participants than that of our study) |
| Useful | • Did the analysis result in
information that is useful for the design of a new decision aid or
training program? • Does this analysis support knowledge elicitation from and participation of subject matter experts in the analysis process? |
This HTA is useful: used to evaluate the impact of the Senior Section intervention on older adult OTC medication decision making and can be applied prospectively to future iterations of the Senior Section and its implementation in pharmacy store chains. |