Use of self-service query tools varies by experience and research knowledge

Abstract

The lack of understanding of user experience with self-service query tools is a barrier to designing effective query tools and propelled this study. User actions were documented and transformed into networks of actions for qualitative analysis. Proficient use of self-service query tools requires significant technical experience. To decrease the user learning curve additional user education is necessary for novice users.

Introduction

A major challenge to clinical and translational research resides in facilitaing access to EHR data for reserachers [1]. Self-service query tools (SSQT) have been developed to meet this need among diverse users [2]. Understanding how diverse users interact with SSQTs can inform effective query tool designs in the future. Therefore, this study reports user experiences across SSQT expereince and research knowledge.

Materials and Methods

Eight semi-structured interviews and user observations were performed at four academic institutions. Users (physcians, clinical reserachers, EHR data analysts) were asked to perform a query to resolve thier real-world information need using “think aloud” protocols. All observations were videotaped to capture the actions and thoughts of the user. When the user completed the query, an exit interview was performed. A user-action schema was iteratively developed and pruned for a video annotation. For each video, user actions were annotated with this schema by a single annotator and repeated for quality control. Users were divided into two sub-groups, experts and novices, based on having greater than 2 years of experience with research and the SSQT. For each expert and novice group across research knowledge and SSQT expereince, normalized directed network graphs were produced and user-action tables were created.

Results

We identified four user actions: browse, enter, review, and select, which had three, four, five, and ninetine subtasks, respectively. As shown in Table 1, expert and novice users had similar frequency distribtions among actions. Expert reserachers extensively used the action “Enter ‘Search Criteria’” while novice reserachers rarely did.

Table 1.

User-Actions by SSQT expereince and research knowledge

User-action	Research Knowledge		SSQT Expereince
	Expert (n=5)	Novice (n=3)	Expert (n=3)	Novice (n=5)
Browse	7	11	8	9
Enter	10	9	12	8
Review	4	5	4	4
Select	33	54	42	40

The directed network graphs show patterns of user-actions within our groups. Most notably SSQT experts exhibited a more organized flow of user actions. They added data elements rather than removed them after reviewing the query build, and tend to reformulate their quieres after obtaining a result set. SSQT novices seemed to augment their queires by adding or removing data elements after reviewing their query build. Interestingly, research knowledge did not display varying patterns of user-actions. Both expert and novice researchers used the SSQT in a simimilar pattern.

Conclusions

As expected, SSQT experts seem to be more efficient with their actions when completing a query, implying that an imporved user expierince may be related to user education of the SSQT functionalities and uses. Additional, SSQT experts seem to augment their query more often after reviewing query results. Interestingly, there seemed to be minimal differences between research experts and novices’ user-action pattern. However, research experts frequently performed data element searches rather than browsing.