Clinical Data Warehouse Query and Learning Tool Using a Human-Centered Participatory Design Process

Abstract

BMI Investigator (BMII) is an interactive web-based tool with a learning knowledge base, which provides a way for researchers to query structured, unstructured, genomic and image data contained in a data warehouse. We demonstrate how development of an efficient, usable, and learnable web interface for a diverse group of research stakeholders benefits from an iterative human- centered participatory design process utilizing a team of clinicians, students, programmers, and informatics experts.

1. Introduction

In the medical domain, health researchers interact with digital information systems every day. Ensuring the delivery of safe and ethical research with secondary clinical data can be a complex process. This process needs to take into account the ability of researchers to not only interact efficiently with healthcare tools, but also be able to easily use them for their intended purpose. In order to make research more accessible to stakeholders with diverse technological and research skills, namely clinical informatics fellows, residents, medical students, medical researchers, and biomedical informatics graduate students, an interactive web-based tool that facilitates rapid exploratory retrospective studies was created with a searchable knowledge base for educational purposes and support. Biomedical Informatics Investigator (BMII) allows the user to query clinical information from a database of electronic health records. However, across the design, implementation, and evaluation of tools like BMII, challenges can arise if there is a failure to understand and engage end-users in technology adoption and use in context [1–3]. Participatory design bridges the relationship between stakeholders’ tacit knowledge and researchers’ analytic knowledge to engage end-users in the design of better products [4,5]. To develop BMII into an effective, efficient, and learnable web interface for its intended diverse group of stakeholders, an iterative human-centered design process using participatory design was undertaken.

2. Methods

BMII allows one query to rapidly access both structured and unstructured data from the electronic health record (EHR). Structured components are transformed into relevant database coding schemes and placed in relational tables in OMOP common data model format. Unstructured notes and reports are coded using the High Throughput Phenotyping Natural Language Processing (HTP-NLP) scheme, producing SNOMED-CT annotated text stored in a graph database and Berkeley DBs [6]. Diagnoses, procedures, demographics, and laboratory values can be queried from different sections of the EHR. Additional data types including medical images and genetic data are currently being added to BMII with relevant searchable parameters. In the development of the BMII web-interface, human-centered design principles using participatory design, an iterative development process that relies on feedback from stakeholders at all stages, were implemented (ISO 9241–210:2010) [7]. These standards stress that solutions and commentary from users should be able to be fed back into the design process early in the development process [7]. Treating stakeholders as co-designers of the interface, a multi-disciplinary team was created consisting of five clinical fellows, a PhD candidate and statistician, programmers, and natural language processing (NLP) expert. To encompass a diverse representation of typical stakeholders, the cohort of clinical fellows had varying degrees of computer science knowledge and medical backgrounds (internist, pathologist, pediatrician, surgeon, and anesthesiologist). The project followed the iterative design process defined by Turner et al. and Spinuzzi, with the following three phases: design, implementation, and evaluation [1,4]. Team members were given continual access to the prototype at all stages, and a bug/issue tracking system using GitHub was implemented to facilitate feedback for the tool. A series of collaborative design sessions involving team members were held. Field notes were dictated and coded to give theoretical insights on the positive and negative components of the design. Session notes were hosted on the Github and available to all team members. Initial design and implementation were done by the principle investigator, who is a clinician, researcher, and NLP expert, and a programmer. The evaluation stage, which incorporated the full participatory design team, featured structured questionnaires to assess researchers’ needs and the usability of the tool for research queries. Stakeholders were asked to participate in simulations using BMII to query case studies to reveal challenges in the design, usability, and functionality of the web interface for each of the clinician team members. After analysis of outcomes from the initial design, mock-ups of the new web-interface incorporating key outcomes and analysis from current well-known interfaces, were distributed and analyzed by the group. Additional design aspects and tools built in to the web interface following the second iteration were shown to team members by the programmers, who then elicited further analysis and feedback. In parallel with the second design, a third design iteration began with a focus on using the query tool as a research education tool for a diverse group of stakeholders beyond clinical informatics fellows based on stakeholders’ evaluation of the mock-ups and current implementations. Educational systems for medical students, residents, clinical fellows, and biomedical informatics graduate students were assessed and key factors of research design for secondary use of clinical data were abstracted from these systems. A semi-structured questionnaire was distributed to biomedical informatics medical students, clinical fellows, and biomedical informatics graduate students assessing their knowledge prior to affiliation with the program and aspects of informatics education that could be enhanced through a learning querying tool. Using this information, implementation began on the use of a help-desk and knowledge base to disseminate pertinent clinical informatics research to the end-user.

3. Results

The first design prototype of the web interface allowed the user to create compound Boolean queries using SNOMED-CT concept codes or descriptions and demographic variables. The user was able to choose a variety of inclusion/exclusion criteria and queries could be saved, imported and joined (Figure 1). During the first design evaluation, stakeholders primarily produced case studies involving identifying cohorts with certain medications, diagnoses, and demographic restrictions (i.e. morphologic subtypes of non-small cell lung cancer broken down by age and gender, comparison of treatment rates for depression in white and non-white pregnant women). During the case simulation on diabetes, stakeholders stated “the measure of HbAlc has an upper level of 9.5, but in BMI investigator the scope for entering a range for blood levels is missing” and it was not intuitive on how to “use or select lab values or cut-off values for lab values.” In addition, for a case study of cases with ADHD between 70 and 100 years old, one stakeholder used the visualization function and realized that the chart gave counts instead of percentages, stating “raw numbers tell me scale, but visualization should tell me proportion.” Key requirements addressed in the collaborative design sessions and case simulations for web-interface modifications were improved visual representation, case counts including proportion of the population, easier navigation, reproducibility of past searches, and data transparency. These requirements were fed back into the iterative design process to produce mock- ups (Figure 1).

Figure 1.

Iterative Design 1 Working Prototype (left); Example of a Mock-up (right)

For the second design, mock-ups were created based on frequently used research query tools using Pencil Project. The stakeholders agreed that clinical researchers’ familiarity with these tools would make the learning curve and usability easier. In addition, constructing an interface incorporating social media and widely used application designs was discussed. The team decided that the goals of the tool would benefit from the simple design aspects of research tools, such as Medline and PubMed, over applications that host variable data uses. In Figure 1 (right), you can see familiar research tool aspects, such as query list, query builder, and a ‘Filter By’ demographics, values, and time. Stakeholders evaluated the Figure 1 mock-up with 80% of fellows suggesting a help-desk icon linking to a knowledge query database of frequently asked questions. The stakeholders liked the search bar, which would allow the system to decide whether a code, concept description, or synonym was inserted.

For design 3, the semi-structured questionnaire had a response rate of 78.6% (11 of 14). The majority of responders (90.8%) had been involved in the Department of Biomedical Informatics for at least six months. At the beginning of their training, 90.9% rated 3 or less on a 5-point Likert scale with 1 being ‘not at all familiar’ and 5 being ‘familiar’ for knowledge of research design methodology, and 72.7% were ‘not at all familiar’ with ontology or terminology. After analysis of the survey and learning objectives, stakeholders valued learning statistical analysis, research methodology, the creation of structured query language queries for data research, and the concepts of terminology and ontology. These learning objectives and frequently asked questions will be hosted on the searchable knowledge base with links to articles, learning materials, and Jupyter notebooks.

4. Conclusion

A participatory design process can avoid some of the complications related to poor health research interface design by incorporating key stakeholder opinions early in the process. A creative secondary purpose of the tool emerged from this process: to provide a knowledge base and support center for key clinical informatics training objectives, such as research methodology, programming, and ontology. (8) Future work includes testing this comprehensive knowledge base on the user population, since stakeholders all target population types. The findings in the stakeholder questionnaire and design sessions allowed us to build a BMII that is able to provide stakeholders with efficient, effective, and useful methods for querying a data warehouse and needed clinical informatics knowledge.