Integrating human-centered design in public health data dashboards: lessons from the development of a data dashboard of sexually transmitted infections in New York State

Abstract

Objective

The increased availability of public data and accessible visualization technologies enhanced the popularity of public health data dashboards and broadened their audience from professionals to the general public. However, many dashboards have not achieved their full potential due to design complexities that are not optimized to users’ needs.

Material and Methods

We used a 4-step human-centered design approach to develop a data dashboard of sexually transmitted infections for the New York State Department of Health: (1) stakeholder requirements gathering, (2) an expert review of existing data dashboards, (3) a user evaluation of existing data dashboards, and (4) an usability evaluation of the prototype dashboard with an embedded experiment about visualizing missing race and ethnicity data.

Results

Step 1 uncovered data limitations and software requirements that informed the platform choice and measures included. Step 2 yielded a checklist of general principles for dashboard design. Step 3 revealed user preferences that influenced the chart types and interactive features. Step 4 uncovered usability problems resulting in features such as prompts, data notes, and displaying imputed values for missing race and ethnicity data.

Discussion

Our final design was accepted by program stakeholders. Our modifications to traditional human-centered design methodologies to minimize stakeholders’ time burden and collect data virtually enabled project success despite barriers to meeting participants in-person and limited public health agency staff capacity during the COVID-19 pandemic.

Conclusion

Our human-centered design approach and the final data dashboard architecture could serve as a template for designing public health data dashboards elsewhere.

INTRODUCTION

Although public health data dashboards have become more common, many have not reached their full potential due to design complexities that are not optimized to their users’ needs.^1–4 For example, a review of 2 dashboards for displaying influenza surveillance data shows that although these tools provide useful information, some of their design features (eg, use of acronyms and jargon and lack of granular data limit the usefulness of these applications).⁴ Data dashboards have become more popular due to the increased availability of publicly available data and accessible visualization technologies such as Tableau.⁵ The adoption of dashboards in different domains broadened both the form and content of the original dashboards. Traditionally, data dashboards comprised single-view representations of key performance measures for executive decision-makers in organizational contexts.⁶ In contrast, new dashboards usually include multiple data views with interactive interfaces and are used to facilitate learning, communication, and motivation in different areas, including public health.⁵ Moreover, using data dashboards for communication and education expanded their audiences from executives to decision-makers at all levels and the general public.⁵ Public health data dashboards allow these diverse users to interact with public health data, thereby promoting data-driven decision-making, encouraging new research, and supporting advocacy work.^7,8

The human-centered design has been useful in developing data dashboards and other information visualization tools. Taking this approach, previous studies have included users in the design process to ensure products’ ease of use and usefulness for the intended audience.^9–11 It is critical that new information visualization tools and technologies be evaluated by users because tool development is mostly conducted by computer scientists and engineers who are trained in statistical techniques and data analytics methods and therefore have a different perception of visualizations than the general population.^12,13 Despite the usefulness of human-centered design, its application in developing public health data dashboards has been limited. One reason is the diversity of target users and use cases¹⁴; consequently dashboards cannot meet the needs of all users and it is critical to identify a clear focus and purpose. For example, while certain graphical features might improve quantitative reasoning, other features may more effectively encourage behavior changes or perceptions in the target audience such as willingness to be vaccinated or satisfaction with healthcare decisions related to breast cancer treatment.¹⁵ Moreover, dashboard usability might differ among user groups with heterogeneous computer and visualization literacy. For example, in a usability study of a “Treatment as Prevention” intervention related to HIV prevention in South Africa, task completion and accuracy varied between scientific and operational staff versus community advisory board members and nursing staff.¹⁶ Employing human-centered design requires that dashboard designers have explicit goals and recruit participants who are similar to target users. Another challenge of integrating human-centered design in public health data dashboards is that traditional methodologies, such as observation of users’ activities, are time-consuming and require heavy investment from users.^10,17

We collaborated with the New York State Department of Health AIDS Institute (hereafter, AIDS Institute) to develop a prototype data dashboard for sexually transmitted infections (STIs) using a human-centered design approach. New York State was an early leader in publicizing open health data¹⁸ and integrating stakeholder feedback into dashboards such as the New York State Prevention Agenda dashboard¹⁹ and the Ending the (HIV) Epidemic Dashboard.^20,21 The focus on STIs was a useful case for US public health practice because of the high morbidity (estimated STI prevalence, 67.6 million cases in 2018), high health-related costs (estimated at $15.9 billion in 2018), and increasing rates of gonorrhea, syphilis, and congenital syphilis.^22–24 The new National STI Strategic Plan set a goal to increase accessibility and use of STI data to inform decisions,²⁵ and data dashboards could provide an opportunity for education, increasing community awareness, monitoring trends, and evaluating programs. Moreover, the general knowledge about STIs is low in the US population²⁶ and there is a gap in provider communication about STIs,²⁷ making it especially important to make reliable sources of information available in an accessible format.

METHODS

Overview

This project comprised 4 steps, which are described in detail in Table 1. The University at Albany Institutional Review Board determined the study was exempt from human subjects review.

Table 1.

Steps involved in the development and evaluation of the data dashboard

Steps	Detailed procedure
Step 1. Stakeholder requirements gathering and establishing the partnership	Purpose: Understand stakeholders’ goals, the organizational context, and establish the partnership. Three initial meetings were conducted with 2 AIDS Institute Office of Sexual Health and Epidemiology staff to discuss: (1) their goals for a data dashboard, (2) feedback on our proposed human-centered design approach to developing the data dashboard, and (3) feedback on evaluation design details such as sampling, recruitment, and the rubric. Two initial meetings were conducted with 3 staff at the AIDS Institute Office of Planning and Community Affairs to: (1) explain our human-centered design approach and solicit feedback on recruitment strategies and (2) provide a flyer, registration form, and slides for presentation and recruitment at one of their advisory board meetings. A site visit was conducted at the AIDS Institute Office of Sexual Health to learn about the data structure and data limitations, meet staff involved in surveillance system maintenance, and understand the organizational context. A meeting series was established with 2 key staff at the Office of Sexual Health to keep the AIDS Institute closely engaged throughout the project lifecycle.
Step 2. Expert review of current US states’ STI data dashboards to produce a usability checklist	Purpose: Produce a checklist to prevent common usability problems when designing dashboards.a Publicly available data dashboards for STIs on state health department websites were identified. A set of usability principles was derived from the information visualization literature. Six reviewers with usability knowledge and diverse domain expertise used the rubric to examine the dashboards.b Experts’ usability scores and their textual comments were analyzed to produce a usability checklist.
Step 3. Pairwise user study with domain experts to understand needs and preferences of target users	Purpose: Elicit needs and preferences of target users. A data dashboard that received high usability scores in Step 2 was chosen for the user study with domain experts. Twenty participants were recruited with the assistance of the AIDS Institute Office of Planning and Community Affairs that had different familiarity with data dashboards and were primarily from the community-based organizations that serve the population at high-risk for STIs.b A pairwise user study was conducted, including a monitored user activity, usability and user experience survey, and an exit interview. Summary statistics were used to analyze the survey results, and an inductive thematic analysis method was used to analyze the transcripts.
Step 4. Development and usability evaluation of the prototype data dashboard for STIs in New York State	Purpose: Integrate findings from the prior steps into the design, development, and evaluation of the prototype dashboard. A prototype dashboard was developed using the NYS STI surveillance data (data preparations in R and data visualizations in Tableau). Seventeen participants were recruited with the assistance of the AIDS Institute Office of Planning and Community Affairs, most of whom had participated in Step 3.b A usability evaluation was conducted, including a free exploration of the dashboard that was monitored for potential usability problems, and an usability survey to learn about the user experience. An experiment was embedded in the usability evaluation to test the methods to visualize missing data. Participants completed tasks using 2 experimental conditions, one displaying the missing values as a separate category (common practice on data dashboards) and one showing the imputed values added to other categories. A comparison of responses in the 2 conditions was tested with a paired t test, and a qualitative synthesis of usability observations were conducted to find generalizable insights. Findings of the experiment and the usability evaluation informed the final prototype.

^aThis checklist is now published²⁸ and can be used in addition to other published work for evaluating data dashboards’ usability²⁹ to avoid usability issues without the effort of conducting usability testing.

^bFor the expert review (Step 2), one of the reviewers was an AIDS Institute Office of Sexual Health staff member. All participants in the pairwise user study (Step 3) were from the community, and represented a mix of direct service providers, healthcare administrators, and volunteers. The Step 4 participants were similar to Step 3 participants, with no AIDS Institute staff included.

Step 1. Stakeholder requirements gathering and establishing the partnership

Three preliminary meetings with the data owners (AIDS Institute Office of Sexual Health and Epidemiology) and an in-person site visit were used to establish relationships, secure buy-in, and discuss topics such as data dashboard goals, the intended audience, available resources, data limitations, software requirements, and other relevant organizational contexts. Additionally, 2 meetings with staff from the AIDS Institute Office of Planning and Community Affairs, which is responsible for policy development, strategic planning, budgetary and legislative analysis, and community were used to expand the organization’s engagement with the project and finalize the participant recruitment plan. A meeting series was established to keep staff engaged throughout the project lifecycle.

Step 2. Expert review of current US states’ STI data dashboards to produce a usability checklist

An expert review of all US state health department STI data dashboards (N = 13, as of June 1, 2021) was used to develop a usability checklist for data dashboard designers to conform to usability principles.²⁸ The selection of dashboards was limited to state (rather than local) health departments because, although exceptions exist, local departments usually have limited resources and staff to prepare complex data dashboards, making their websites less comparable to state departments. The included states were Arizona, Colorado, District of Columbia, Florida, Georgia, Hawaii, Iowa, Kansas, Maryland, New York, North Carolina, Tennessee, and Texas. Six expert evaluators evaluated the STI data dashboards, representing domain experts and nondomain experts. Per the requirements of expert review,³⁰ all evaluators had knowledge of usability and data visualization. The domain expert evaluators comprised a state health department employee, a faculty member with public health expertise, and a doctoral student with health informatics expertise. The nondomain expert evaluators comprised a faculty member with expertise in digital government, a faculty member with expertise in human–computer interaction, and a doctoral student with expertise in human–computer interaction. The usability checklist was derived from a qualitative synthesis of evaluators’ comments regarding major and minor usability problems. The usability evaluation was accompanied by an analysis of the data dashboards’ characteristics such as available visualizations, interactive features, and use of filtering or faceting to show the breakdown of the represented data.

Step 3. Pairwise user study with domain experts to understand needs and preferences of target users

A pairwise user study was conducted with practitioners who primarily worked in community-based organizations that serve populations at high risk for STIs. These community champions were ideal participants because they could represent the needs of target users (community-based organizations and patients) and had established relationships with the AIDS Institute, making them relatively easy to access. Participants were paired to minimize researcher intrusion, and increase the chance of revealing usability problems through the process of helping each other work on the specified activity.³¹ During virtual meetings, pairs of participants collaborated to complete several tasks on Arizona’s STI dashboard, which was selected from the data dashboards that were evaluated in Step 2. The Arizona STI data dashboard had different visualizations with interactive features and was scored highly in Step 2, making it an ideal choice for a user study to determine if users and expert opinions diverged. The tasks included exploration, pattern identification, association, comparison, verification, policy development, and dissemination, and were designed based on common tasks in visual analytics of public health data.³² All meetings were held online and the participants were given 30–45 min to complete the tasks. Details of the data collection meetings are available in Supplementary Appendix S1. Usability problems and user preferences were identified from the moderator’s direct observations of their task completion, a thematic analysis of audio transcripts, and a brief survey of their user experiences.³³

Step 4. Development and usability evaluation of the prototype data dashboard for STIs in New York State

A prototype dashboard was developed using the NYS STI surveillance data. Multiple data preparations steps were conducted in R software, including merging with Census population data to produce STI rates among different populations and redistributing missing values. The data dashboard was created in Tableau using a dashboard layout and interactive charts. All findings from the first 3 steps were integrated in the prototype data dashboard’s design. Each of the prior 3 steps informed the design. For example, the data limitations and the software requirements from Step 1 were used in choosing the platform and selecting appropriate measures for presentation. The user preferences from Step 3 were considered when selecting the chart types and adding interactive features. The checklist produced in Step 2 was used as a general guide for designing the layout, text, and charts.

Participants from Step 3 were recruited for a Zoom-based usability evaluation of the prototype, in which participants were asked to freely explore each page, use interactive features, and narrate what they saw. In addition, the participants were asked to use the data dashboard to complete some tasks so potential usability problems would emerge. Details of the data collection meetings are available in Supplementary Appendix S2. When working with the data dashboard, participants used their pointing device to point to where they were looking on their screen and verbalized their thoughts in a sentence such as, “The red button is the first thing that caught my eyes. I am wondering what this button will do, and I’m going to click to find out.” The moderator monitored users’ activities and specific usability problems encountered and took notes of usability issues as well as the way participants interacted with the data dashboard (eg, the first thing to which they pointed on each page, and if they read the guides or used the interactive features at any time during their interaction). The moderator synthesized her notes to identify common usability issues and generalizable insights for other data dashboards.

One specific issue that emerged in Step 3 was how to present missing race and ethnicity data, which are common in chlamydia and gonorrhea surveillance data.³⁴ Two versions of a visualization of cases stratified by race and ethnicity, using different methods to present missing demographic information, were included in the prototype to assess users’ understanding and preferences. The findings of the experiment and the usability evaluation were incorporated in the final prototype data dashboard.

RESULTS

Key findings from usability evaluations

Table 2 summarizes key findings that influenced the design. The stakeholder requirements gathering (Step 1) revealed important considerations about the data structure and limitations, organizational context, compatible software, and resources. For example, the AIDS Institute preferred Tableau for consistency with other projects and easier maintenance. Another finding was that the final data dashboard would need to be hosted on an external organization’s website due to state government agency “executive clearance” requirements that each website update be reviewed and approved before release to the public.

Table 2.

Key findings from evaluations that influenced the data dashboard design

	Key findings
Step 1 findings	Tableau was the desired platform for consistency with the AIDS Institute’s other projects and easier maintenance. The final data dashboard had to be hosted on an external organization’s website due to organizational rules.
Step 2 findings	Using a usability checklist could inform the design of layout, text, and chart to prevent some of the common usability problems on STI data dashboards. When showing granular data (eg, by sex, age, region), filters and faceting should be used to produce charts that are useful for domain experts and understandable for non-expert users.
Step 3 findings	Simple and familiar visualizations (eg, bar charts and line charts) should be used to focus users’ attention on the content rather than the visualization technique. Maps should be used with caution to avoid stigmatizing highly impacted regions. The reason for missing data and its effects on interpretations should be explained to improve users’ trust in the data. Up-to-date terminology and clear explanations should be used for classifying population groups based on sex, race and ethnicity, and other demographic variables. Guidance should be provided to help users navigate and learn about interaction possibilities.
Step 4 findings	On multi-page data dashboards, functionality and interactivity should be consistent across pages, so users can learn once and use their knowledge throughout. Prompts should direct users to notes and explanations because some users only focus on the interactive features and ignore the text on the data dashboard. Notes should discuss both what is included as well as what is excluded (eg, diseases for which reporting is not required by law and surveillance data are unavailable for inclusion should be explained). The data visualization principle of “removing extraneous ink” should be checked with end-users to ensure that charts are comprehensible. Data analysis terminology (eg, filter, cluster, aggregate) should be avoided to prevent confusion among users without data analysis expertise. The visualization with the added imputed values improved accuracy in interpretation and confidence in the data.

The evaluation of existing STI data dashboards (Step 2) uncovered a tension between complexity and usability. Complex data dashboards, with multiple visualizations and interactive features, were more suitable for public health experts interested in specific populations but received lower usability scores because of their limited usefulness for novice users. In contrast, simple data dashboards received higher usability scores but were less useful for expert users. When we examined filtering or faceting, 2 common methods to deal with complexity in data visualization,³⁵ data dashboards that used both methods simultaneously achieved a good balance of usability and complexity that would be useful for expert and novice users.

The user study (Step 3) identified important preferences among target users. First, participants were more attentive to the content with the use of simple visualization techniques such as bar charts and line charts. Second, some participants were concerned about the potential misinterpretation of disease maps. They shared that showing highly impacted regions in dark colors could be stigmatizing. Third, participants found charts showing the number of STI cases stratified by race and ethnicity with a “missing” category (including cases for whom the race and ethnicity information was not available) to be confusing and raised concerns about the trustworthiness of the data dashboard. Fourth, participants raised concerns about the use of outdated terminology for population groups (eg, transgender male to female). Fifth, navigation and interaction possibilities were not immediately clear to all participants, and many needed guidance.

The usability evaluation and the embedded experiment in Step 4 had several findings. First, participants repeatedly discussed how consistency in layouts and interaction possibilities helped them learn and use that knowledge throughout the dashboard. Second, participants exhibited different reading patterns, with some reading all guides and notes before interacting with the charts and others ignoring the text and using the interactive charts exclusively, which led to later confusion. Third, participants frequently asked about the excluded diseases, regions, and populations. For example, human papillomavirus was not included because data were not available, but participants found it confusing that some STIs were omitted from the data dashboard. Fourth, following the data visualization principle of “removing extraneous ink,”²⁸ some items were removed (eg, axis titles that were repetitive of page titles). However, these choices reduced chart comprehension among some participants. Fifth, the initial terminology used to explain interactivity options (eg, filter, cluster, aggregate) was unclear to users, and they requested more common terms, such as drop-down menus. Finally, in the embedded experiment, the visualization with the added imputed values improved users’ accuracy in interpretation and their confidence in the data.

Final data dashboard prototype

Figure 1 shows one page of the final prototype data dashboard, which incorporates changes made in response to the Step 4 feedback. A multipage format was selected to help users digest each piece of information and encourage them to assess information from different perspectives. Arrow “a” points to the use of contrast in the navigation bar to help users see their current location and other pages available for their exploration. Arrow “b” points to the reading and interaction guides for each page to support novice users. Arrow “c” points to the filters for expert users to drill down into specific population groups. Each filter had a drop-down menu and a prompt to encourage users to read the footnotes. Arrow “d” points to the faceting technique to produce simpler charts for nonexperts. Arrow “e” shows the extensive data footnotes.

Figure 1.

Screenshot of one data dashboard page demonstrating the features included based on the usability evaluations. Note: Data are unvalidated and should not be used for program planning or other purposes. Arrows indicate features included in response to issues identified during the usability evaluations.

Figure 2 shows another data dashboard page that displays geo-faceting as a solution for a concern raised in the usability evaluation about inadvertently stigmatizing highly impacted counties in standard maps with higher-morbidity counties shaded in a darker color. Moreover, equal-sized boxes allowed each county to have the same prominence regardless of their relative population size or land area. The layout and available interactions were consistent with prior pages for improved usability. The final user study found that the geographical resemblance between the layout of the county charts and New York State was not clear to all users. The final prototype included bold and underlined font in the reading guide to communicate the positioning to users, county names on each chart, and a zooming function (zooming each box when hovered over) to provide additional guidance.

Figure 2.

Screenshot of one data dashboard page demonstrating the use of geo-faceting to map infection trends in the counties of the New York State. Note: Data are unvalidated and should not be used for program planning or other purposes.

DISCUSSION

The prototype design was accepted by the AIDS Institute and is currently in the process of being implemented as a public-facing data dashboard. Several research methodology choices enabled the successful use of the human-centered design approach. First, traditional human-centered design methods were modified to be more pragmatic, which was critical during the COVID-19 pandemic (a highly contagious respiratory disease caused by the SARS-CoV-2 virus) when public health staff were incredibly busy. The pragmatic modifications included learning from existing data dashboards instead of starting from scratch and conducting virtual meetings instead of observing user behavior in their workplace. Also, we recruited participants from practitioners who already had close relationships with the AIDS Institute as community-based public health champions. These participants were knowledgeable about STIs but most of them had little previous experience with data dashboards. Therefore, we did not expect them to understand data dashboards better than a layperson. Moreover, they work directly with healthcare providers and patients, and could share their own professional experiences with the dashboard and represent the needs of community members.

A second critical factor contributing to project success was the engagement of the AIDS Institute throughout the project lifecycle to increase the likelihood that research findings would be implemented. In Step 1, multiple meetings with the AIDS Institute were conducted to understand the organizational context, their goals for the data dashboard, and the data presented on the data dashboard. In Step 2, one AIDS Institute staff member participated as an expert evaluator. In Step 3, AIDS Institute staff identified a participant pool from its existing workgroups, and community-based listservs assisted with recruitment. In Step 4, early findings from the usability evaluations were discussed with AIDS Institute staff, and they were invited to contribute ideas for solutions for the found usability problems. Collaborating with practitioners enhanced the usefulness of findings for stakeholders and ensured buy-in and collective ownership of the final design.

Our methods have several contributions to usability evaluation practices. First, traditional human-centered methods suggest starting with the users’ current technology and improving areas in which the current technology does not meet the users’ needs.³⁶ In this study, we initially considered New York State dashboards such as the influenza tracker but decided to go beyond the state health department to review all STI data dashboards on other comparable health department websites. The reviewed data dashboards had different levels of complexity, which we found was related to the number of identified usability problems. These findings led us to explicitly consider an appropriate balance between complexity and usability throughout our data dashboard design process. Second, for understanding users’ needs and preferences, 2 traditional human-centered methods are contextual inquiry (observation of user activities in their work environment) and domain knowledge workshops (conducting workshops with users to discuss their work practices in detail). These methods need heavy investment from domain experts, which is not realistic in many situations.^10,11 Contextual inquiry requires the acceptance of researchers in users’ work environments and their interruptions for inquiries Domain knowledge workshops require users to attend lengthy workshops to discuss their work practices. Both were impractical in our context, between staff being diverted to the COVID-19 response (thereby severely constraining their time) and staff working in confidential spaces due to the sensitive data (thereby making it difficult for a researcher team to gain security clearances to observe workflows directly). Our virtual meetings in Steps 3 and 4 may have been limited compared to the traditional methods because the data collection was not in the users’ work environment, and we could not collect some potentially useful data such as eye movements or facial expressions. However, our meetings required 2 h of participants’ time (outside their working hours), was easier to schedule as virtual meetings with 1 or 2 participants (vs intensive long workshops with many participants), and did not require us to obtain special approvals to observe their interactions with the technologies.

Using the human-centered design approach helped us prevent some of the usability problems that were found in previous usability evaluations of public health data dashboards. For example, one of the usability problems found in previous dashboard evaluations is that data were not available at a granular level.⁴ We provided multiple filters (with dropdown menus) on each page of the dashboard to enable access to granular data by county, by sex, by age, and by race and ethnicity. Moreover, we discussed the levels of these variables with participants to ensure the output would be useful for them. For example, STI clinics sometimes focus on a specific age group (eg, adolescents) or a specific population group (eg, Black women). We used our participants’ comments about target audiences to ensure these groups were included in our dropdown menus. Another usability problem found in previous dashboard evaluations is that the terminology (eg, surveillance, prevalence, or antiviral resistance) was not suitable for those unfamiliar with health-related information.⁴ Our participants had experience directly working with general consumers and helped us avoid terminology that would not resonate with laypersons and develop footnotes to facilitate data interpretation.

This project had several limitations. First, focusing on one disease area and state might limit the generalizability of findings. Second, although the evaluators and participants comprised diverse groups, their opinions may not reflect all stakeholders’ opinions. Third, the data collection of this project was conducted virtually because of the concerns regarding COVID-19 transmission and the difficulties of scheduling time with busy public health experts during the pandemic. Virtual meetings limited the ability to collect additional user data such as body expressions and eye movements. Finally, based on the human-centered design framework, a long-term and consistent evaluation of the data dashboard is necessary, which was not yet possible because the dashboard is currently being implemented. However, this is a promising area for future research to investigate the emerging usability problems as technology changes and the user base grows.

FUNDING

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

Both authors conceived the study. Ansari directed the research, collected data, conducted the analysis, built the dashboard, and wrote the initial draft. Martin reviewed the methods and results and revised the manuscript for intellectual content.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

The authors have no competing interests to declare.

DATA AVAILABILITY

The data underlying this article will be shared on reasonable request to the corresponding author.