Abstract
Human-computer interaction and related areas of user experience (UX) research, such as human factors, workflow evaluation, and data visualization, are thus essential to presenting data in ways that can further the analysis of complex data sets such as those used in patient-centered research. However, a review of available data on the state of UX research as it relates to patient-centered research demonstrates a significant underinvestment and consequently a large gap in knowledge generation. In response, this report explores trends in funding and research productivity focused on UX and patient-centered research and then presents a set of recommendations to advance innovation at this important intersection point. Ultimately, the aim is to catalyze a community-wide dialogue concerning future directions for research and innovation in UX as it applies to patient-centered research.
Keywords: Biomedical Research, Health Services Research, Comparative Effectiveness Research, Quality Improvement, Informatics, Data Display
Introduction
Numerous reports have described the benefits afforded by leveraging unique human cognitive capabilities in the areas of pattern recognition and higher-order reasoning to interpret complex data sets, such as those commonly encountered in health care.1–7 Such benefits include improved abilities to identify important patterns in data, more rapid hypothesis discovery and testing, and enhanced support for collaborative or team science models. However, the achievement of such benefits often requires the optimization of the ways in which humans interact with and use computers to view, manipulate, and analyze data. As such, the appropriate use of user experience (UX) principles in the context of biomedical computing represents an open and ongoing area of research and innovation. In the domains of patient-centered research, the need to support effective human-computer interaction paradigms is amplified as a consequence of the heterogeneity of frequently encountered data.3,5,8 In such use cases, researchers often need to collect, aggregate, and analyze data sets from a variety of sources, including clinical, research, and patient-facing information systems, to identify important patterns or associations.6,9,10
While statistical and machine learning methods are useful for supporting the data analysis needs of patient-centered research programs, they are frequently insufficient to identify more complex patterns in large or heterogeneous data sets and thus demand the interactive involvement of humans in the data analysis and pattern recognition process.11,12 Fortunately, a robust scientific and technical discipline concerned with UX provides a ready basis for the design, selection, and use of strategies that can support and enable human-centric interactivity with research data. The discipline of UX research and practice draws on or includes several complementary areas, including but not limited to human factors, workflow evaluation, human-computer interaction, and data visualization.1
Funding and Publication Activity at the Intersection of UX and Patient-Centered Research
Given the potential benefits of UX research to support and enable information needs for patient-centered research, we conducted a review of available data sets to understand current trends in supporting patient-centered research. Included in the review was National Institutes of Health (NIH) data describing numbers and amounts of research awards (accessed via NIH RePORTER) and peer-reviewed publications indexed in the National Library of Medicine’s (NLM) Medline bibliographic database (accessed via the PubMed portal). The review focused on evaluating potential relationships between the availability of funding and “downstream” research productivity in term of publications and other indicators of contributions to the relevant knowledge base. Specifically, the review undertook the following analyses:
A comparison of the distribution of absolute funding provided by the NIH for UX-relevant, clinical research–relevant, and other application areas over a five-year period from 2007–2011 (Figure 1)
A comparison of the relative growth rate for funding provided by the NIH for both UX-relevant and clinical research–relevant projects and of overall extramural funding over a five-year period from 2007–2011 (Figure 2)
A review of the abstracts of indexed and peer-reviewed literature found in Medline and assigned Medical Subject Heading (MESH)terms corresponding to both “clinical research” and some combination of “computer-human interaction,” “usability,” or “data visualization” over a three-year period from 2010–2012 (Table 1) (A three-year period reflected contemporary rather than historical research trends in the literature.)
Figure 1.
Distribution of absolute funding provided by the NIH for UX-relevant, clinical research—relevant, and other application areas over a five-year period from 2007 through 2011
Figure 2.
Relative growth rate for funding provided by the NIH for both UX-relevant and clinical research—relevant projects as well as for overall extramural funding over a five-year period from 2007 through 2011
Notes: In this analysis, we used funding as of 2007 as a baseline. The trends reflect an overall decline in NIH funding for clinical research and associated areas (including UX innovation with applications in the clinical research domain) beginning in 2010 and coinciding with policy and funding initiatives related to the NIH roadmap for clinical and translational science
Table 1.
Summary of literature search and abstraction results for manuscripts with curated MESH terms corresponding to both “clinical research” and some combination of “computer-human interaction, “usability,” or “data visualization,” published between 2010 and 2012
| Manuscript Type | UX Focus | ||||
|---|---|---|---|---|---|
| Evaluation Studies | Framework or Methods Development | Review or Perspectives | Primary | Secondary | |
| 2010 (n = 128) | 121 (94.5%) | 2 (1.6%) | 5 (3.9%) | 5 (3.9%) | 123 (96.1%) |
| 2011 (n = 168) | 162 (96.4%) | 1 (<1%) | 5 (3%) | 8 (4.8%) | 160 (95.2%) |
| 2012 (n = 147) | 138 (93.9%) | 1 (<1%) | 8 (5.4%) | 16 (10.9%) | 131 (89.1%) |
| 2010–2012 (n = 443) | 418 (94.4%) | 4 (<1%) | 18 (4.1%) | 31 (6.5%) | 457 (95.6%) |
An assessment of the findings generated by the analyses yields several interesting findings. First, funding of UX-relevant research has remained consistently on the order of 2 percent of the total NIH extramural portfolio on an annual basis. However, when compared to the relative growth rate for overall NIH extramural funding, both UX-relevant research and clinical research–relevant funding has expanded at a rate higher than that of the total NIH award portfolio. Nevertheless, when the two areas are compared directly, with the exception of 2010 (for which it could be argued the effects of American Recovery and Reinvestment Act [ARRA] funding may serve as a confounder), the rate of growth for UX-relevant funding did not keep pace with the rate of growth for clinical research–relevant funding. Second, publication of UX-relevant research with applications in the patient-centered research domain is uncommon (443 out of 29,806 indexed publications, or 1.5 percent, pertained to patient-focused health care research). Furthermore, most of the studies (95.6 percent) do not have a primary focus on UX-related topics but instead include them as a secondary focus. Finally, fewer than 1 percent of such publications over the previously defined three-year period described the development of the UX framework or methods.
It is difficult to argue that there are direct and attributable relationships between funding rates and the “downstream” impact of a body of research-related activities. However, these relationships are worth noting in light of a growing body of literature showing that even marginal funding increases in specific research domains can have a positive impact on increased research productivity (as measured in terms of publications or intellectual property disclosures.13–15 Perhaps more important, reports that have described the relationship between funding and research productivity have consistently shown that increased financial support for areas of investigation that exhibit smaller or less well-developed knowledge bases produce more demonstrable and dramatic impacts as compared to more well-developed areas of research and innovation.13–15
While many of the findings and assessments discussed in this report represent areas of concern or unmet opportunity, it should be noted that some of the potential challenges associated with funding rates and publication activities at the intersection of UX and patient-centered research may be exacerbated by issues that extend beyond scientific and technical barriers. For example, a small number of published reports identified during the course of the targeted literature review describe a number of people and organizational issues, including (1) a common emphasis among information technology and informatics practitioners on engineering-focused approaches to problem solving that tend to emphasize function over form,16–19 and (2) a prevailing sentiment that UX-related investigation is not needed because UX is a “solved problem.” In contrast, a body of literature found during the course of this study describes unresolved research questions focused on identifying and understanding optimal ways to present and interact with multidimensional data, information, and knowledge.1–4,7,8,18,19 Such research questions inquire about (1) the use of visualization technologies to enable interactive mining of large-scale and/or heterogeneous data sets;2,3 (2) the design and application of icons and other metaphor-graphic–based models for the delivery of semantically complex information;5,7,12 and (3) the incorporation of issues surrounding human cognitive limits, particularly in distributed or information-intensive environments, relative to the design of Electronic Health Record (EHR) user interfaces that may be used for both clinical documentation and research-focused data reuse.8,16,20,21
Current State of Knowledge and Practice Surrounding UX and Patient-Centered Research
It is important to note that, while limited in scope, major contributions continue to be made to the state of knowledge and practice in the general area of UX research and development. Such contributions, undertaken by members of the UX community in the computer, information, and cognitive science domains, have generated a variety of theories and methods related to both (1) qualitative and quantitative system-focused usability testing,19 and (2) cognitive analyses of the impact of technology and barriers to system adoption or use.11 In addition, the health care–focused UX community is devoting attention to new lines of investigation related to (1) mobile devices and applications; (2) patient and health care consumer empowerment; and (3) the creation of ecosystems of data and interactivity (e.g., leveraging ubiquitous computing and sensor technologies).16,18–20 Finally, the complementary and highly interrelated data visualization community is making similar and relevant advances by developing, for example,3,12 (1) visualization and interaction toolkits for complex and information-dense (e.g., multidimensional) data sets, such as those in the biomolecular domain; (2) the visualization of “big data” by using distributed or a high-performance computing infrastructure; and (3) the visualization of continuous data streams from sensor technologies. The scale and scope of these examples are not, however, necessarily commensurate with corresponding growth in the broad clinical research domain. These efforts represent glimpses of the types of innovation and practice possible at the intersection of UX and patient-centered research and may point to the potential benefits that such work may produce.
The general trend in innovation is reflected in a smaller scope at the intersection of UX and patient-centered research. While a comprehensive review of innovative projects is beyond the scope of this report, several representative examples follow:
Network visualization for literature synthesis. Using a human-centered and iterative design process, an application known as the Action Science Explorer enables humans to interact with and synthesize complex literature corpora during the course of planning a given research program.22 A key aspect of this tool is the use of both semantic reasoning and visualization approaches to facilitate the identification of indirectly related and relevant literature from areas outside an original search request, thus increasing the quality and completeness of the resultant literature reviews.
Supporting temporal summarization of longitudinal patient or population-based data. A major challenge in attempts to identify and understand important trends in clinical data derived from EHRs (at the patient or population level) is the temporal “normalization” and alignment of such information in order to support pattern recognition. The EventFlow application employs a hybrid algorithmic and human-centered visualization approach to refine iteratively and improve the temporal alignment of EHR-derived data and the downstream identification of important patterns in those data.23 Evaluations of the tool have shown that EventFlow’s combined algorithmic and human-centered workflow outperforms conventional quantitative methods for addressing information needs related to the temporal summarization of EHR-derived phenotypes.
Ubiquitous research data collection. In many patient-centered research contexts, investigators need to be able to collect data from participants in settings outside the clinic or laboratory. With the increasing use of ubiquitous computer devices, such as smart phones, opportunities abound for data collection. Abowd and colleagues have generated a number of examples of ubiquitous research data-capture tools that exploit computer technology.24 Through an iterative and highly user-centric process, the data capture instruments have been refined to balance the simultaneous needs of ensuring end-user acceptance and supporting high-fidelity data capture for research purposes. The resulting technologies provide a novel way to empower research participants and their families or caregivers to become integral parts of the overall data-capture “fabric.”
An assessment of the current state of research and innovation surrounding UX in general and UX as applied to patient-centered research gives rise to several conclusions. In particular, the application of UX theories and methods may improve our ability to understand the types of complex and heterogeneous data commonly encountered in the patient-centered research use cases. It can also be argued that increased investment in UX research and development, particularly as pertaining to the broad clinical research domain, has the potential to increase “downstream” research productivity and expand the resultant knowledge base, with a low marginal cost given current rates of investment. However, barriers to realizing such a vision persist, including people and organizational factors such as misperceptions about the role of UX in biomedical research computing and priorities that emphasize function over form. At the same time, examples of the opportunities that lie at the intersection of UX and patient-centered research may guide potentially productive areas of expanded research support as suggested above. Specific examples of innovations in the UX domain that can contribute to the design and conduct of patient-centered research include: 1) support for human-centric hypothesis discovery and data visualization,2,3 improved protocol monitoring tools afforded by “dash-boarding” approaches to data presentation5,7 and the optimal integration of computational tools into research workflow through both human-computer interaction and cognitive science theories and methods.8
Conclusions and Recommendations
If the community of investigators with interests and expertise relevant to the intersection of UX innovation and patient-centered research can define and pursue a vigorous and systematic research agenda, advocate for funding, and increase the rate of knowledge generation relevant to UX and patient-centered research, significant benefits may accrue to the broad research community and to stakeholders with an interest in patient-centered research (e.g., patients, their families, and their communities). Such benefits could include both foundational improvements in researchers’ ability to interpret and act on complex data sets in a timely and resource-efficient manner and more indirect and positive impacts relative to the quality and uptake of resulting research products informed by human-centric data analytics.
More specifically and drawing on this report’s assessment of the state of the art and practice of innovation and knowledge generation at the intersection of UX and patient-centered research, we conclude that significant and sustained efforts are needed to advance and invigorate UX research that will create and enable a compelling end-user experience for comparative effectiveness research (CER), patient centered outcomes research (PCOR), and quality improvement (QI) investigators and practitioners. Such efforts will support the realization of a broader conceptual model for CER, PCOR, and QI research,10 as summarized in Figure 3.
Figure 3.
Conceptual models for CER, PCOR, and QI, illustrating data, information, and knowledge resources, analytic methods, and involved or otherwise interested stakeholders
Notes: The figure is based on a model originally presented in (10)
In this model, researchers and practice teams employ and adopt protocols and study designs that leverage or adapt both computational and human analytic methods supported and enabled by a diverse collection of data, information, and knowledge.
It is important that expanded research and innovation efforts in the UX and patient-centered research domains are informed (and used) by a community of interested decision makers and consumers with the ultimate objective of driving improvements in clinical care and public health. Achieving this aim will require both consensus building and a strong community orientation aim in order to overcome potential barriers to knowledge dissemination and “uptake” in real-world settings. While several bodies of work address various aspects of a stake-holder driven approach to research and innovation,5,6,9,10,16,17 the pertinent UX research and practice knowledge base do not adequately address the human analyses component of the conceptual models for CER, PCOR, and QI presented in Figure 3 and thus represent an area in need of further exploration. To address this gap, two major initiatives are essential.
Pursuing Critical Research Foci on UX
As noted in recent reports on the information needed essential for conducting CER, PCOR, and QI research programs,6,7,9–11,16 several critical types of data resources must be leveraged. They include but are not limited to (1) EHR-derived clinical phenotypes; (2) patient-reported data; (3) biomolecular phenotypes; and (4) supporting sources of knowledge (such as those found in public data sets or bibliographic databases). Therefore, it is necessary to undertake UX-focused research and development to identify optimal and well validated methods, technologies, and best practices that address all of information needs associated with the ability to effectively use the preceding data resources for patient-centered research, with an emphasis on facilitating:
The presentation of and interaction with structured, semistructured, and unstructured data in EHR, Personal Health Record (PHR), and Clinical Research Management Systems (pertaining to clinical phenotyping) from provider, researcher, and patient perspectives;
The visualization of complex and heterogeneous data, information, and knowledge resources spanning the biomolecular, clinical, and patient-reported data domains; and
The provision of systematic and reproducible toolkits and platforms that enable staff to construct and use data integration, presentation, and interaction “pipelines,” with an emphasis on pattern recognition and hypothesis generation.Ideally, the research products generated in the aforementioned areas would provide for a more robust and readily adoptable “knowledge base” by which the designers and implementers of computational systems targeted to the patient-centered research domain could quickly and easily employ state-of-the-art UX approaches in the design of their tools. Such a scenario could then lead to improved usability and end-user benefits relative to the speed and accuracy of data-centric tasks, especially as related to human-centric data analytics.
Enhancing Community, Policy, and Funding Support for UX
As demonstrated by the preceding analyses, investments in UX-related research to facilitate patient-centered research are failing to “keep pace” with the growth in user needs for such methods and technologies as well as with the demand for a more efficient and effective research enterprise that will yield broad social benefits. Such a trend is evident in the peer-reviewed literature as reflected in the dearth of grants and contracts targeted to UX-related research and the dearth of downstream contributions to the state of knowledge. A potential explanation for the current state of affairs is a lack of understanding among stakeholders as to the critical need for and benefits of UX research with direct applications to patient-centered research. The lack of understanding, in turn, likely results from a host of factors, including low penetration of relevant knowledge into the broader clinical research community, a limited number of well-known and empirically based studies that demonstrate the value of such work, and the emphasis among decision makers and practitioners on software development practices that emphasize function over form.
Ultimately, generating new knowledge to advance the understanding of UX is likely to be a highly multidisciplinary undertaking that will require efforts at the community level to create a UX research “roadmap” that encompasses the diverse and complementary needs of decision-making, funding, research, and application leaders. The roadmap should also provide recommendations on how to reinvigorate innovation as it pertains to health care writ large. It is likely that achieving these goals will also require us to mobilize pertinent professional organizations to create “homes” for investigators. The hope is that this commentary will catalyze the pursuit of such agenda-setting activities and, more broadly, the creation of a community-wide dialogue concerning the issues introduced herein.
Acknowledgments
This work was supported by a contract from the Electronic Data Methods (EDM) Forum. The EDM Forum is supported by the Agency for Healthcare Research and Quality (AHRQ) through the American Recovery & Reinvestment Act of 2009, Grant U13 HS19564-01.
References
- 1.Boehm-Davis DA. Discoveries and developments in human-computer interaction. Human factors. 2008 Jun;50(3):560–4. doi: 10.1518/001872008X288529. [DOI] [PubMed] [Google Scholar]
- 2.Fox P, Hendler J. Changing the equation on scientific data visualization. Science. 2011 Feb 11;331(6018):705–8. doi: 10.1126/science.1197654. [DOI] [PubMed] [Google Scholar]
- 3.Kehrer J, Hauser H. Visualization and Visual Analysis of Multi-faceted Scientific Data: A Survey. IEEE transactions on visualization and computer graphics. 2012 Apr 10; doi: 10.1109/TVCG.2012.110. [DOI] [PubMed] [Google Scholar]
- 4.Pavelin K, Cham JA, de Matos P, Brooksbank C, Cameron G, Steinebck C. Bioinformatics Meets User-Centered Design: A Perspective. PLoS Comput Biol. 2012;8(7):e1002554. doi: 10.1371/journal.pcbi.1002554. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Payne PR, Embi PJ, Johnson SB, Mendonca E, Starren J. Improving Clinical Trial Participant Tracking Tools Using Knowledge-anchored Design Methodologies. Appl Clin Inform. 2010;1(2):177–96. doi: 10.4338/ACI-2010-02-RA-0012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Payne PR, Johnson SB, Starren JB, Tilson HH, Dowdy D. Breaking the translational barriers: the value of integrating biomedical informatics and translational research. J Investig Med. 2005 May;53(4):192–200. doi: 10.2310/6650.2005.00402. [DOI] [PubMed] [Google Scholar]
- 7.Starren JB, Payne PR, Kaufman DR. Human computer interaction issues in Clinical Trials Management Systems. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2006:1109. [PMC free article] [PubMed] [Google Scholar]
- 8.Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of biomedical informatics. 2004 Feb;37(1):56–76. doi: 10.1016/j.jbi.2004.01.003. [DOI] [PubMed] [Google Scholar]
- 9.Embi PJ, Payne PR. Clinical research informatics: challenges, opportunities and definition for an emerging domain. J Am Med Inform Assoc. 2009 May-Jun;16(3):316–27. doi: 10.1197/jamia.M3005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Embi PJ, Payne PR. The Role of Biomedical Informatics in Facilitating Outcomes Research: Current Practice and Future Directions. Circulation. 2009;120:2393–9. doi: 10.1161/CIRCULATIONAHA.108.795526. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Myneni S, Patel VL. Organization of Biomedical Data for Collaborative Scientific Research: A Research Information Management System. Int J Inf Manage. 2010;30(3):256–64. doi: 10.1016/j.ijinfomgt.2009.09.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Shneiderman B. Designing the User Interface: Strategies for Effective Human Computer Interaction. 3rd ed. Reading, MA: Addison Wesley Longman; 1998. [Google Scholar]
- 13.Auranen O, Nieminen M. University research funding and publication performance—An international comparison. Research Policy. 2010;39(6):822–34. [Google Scholar]
- 14.Hicks D. Performance-based university research funding systems, Research Policy. Research Policy. 2012;41(2):251–61. [Google Scholar]
- 15.Payne AA, Siow A. Does Federal Research Funding Increase University Research Output? Advances in Economic Analysis and Policy. 2003;3(1) [Google Scholar]
- 16.Ash JS, Anderson NR, Tarczy-Hornoch P. People and Organizational Issues in Research Systems Implementation. Journal of the American Medical Informatics Association. JAMIA. 2008 Feb 28; doi: 10.1197/jamia.M2582. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Lorenzi NM, Novak LL, Weiss JB, Gadd CS, Unertl KM. Crossing the Implementation Chasm: A Proposal for Bold Action. J Am Med Inform Assoc. 2008;15(3):290–6. doi: 10.1197/jamia.M2583. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Sittig DF, Singh H. A New Socio-technical Model for Studying Health Information Technology in Complex Adaptive Health-care Systems. Qual Saf Health Care. 2011;19(Suppl 3):i68–i74. doi: 10.1136/qshc.2010.042085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Yen P, Wantland D, Bakken S. Development of a Customizable Health IT Usability Evaluation Scale. AMIA Annual Symposium proceedings / AMIA Symposium AMIA Symposium. 2010:917–21. [PMC free article] [PubMed] [Google Scholar]
- 20.Searl MM, Borgi L, Chemali Z. It is time to talk about people: a human-centered healthcare system. Health Res Policy Syst. 2010;8(35) doi: 10.1186/1478-4505-8-35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Card Moran Newell The Psychology of Human-Computer Interaction. 1983 [Google Scholar]
- 22.Dunne C, Shneiderman B, Gove R, Klavans J, Dorr B. Rapid understanding of scientific paper collections: integrating statistics, text analytics, and visualization. JASIST: Journal of the American Society for Information Science and Technology. 2012 [Google Scholar]
- 23.Monroe M, Lan R, Morales del Olmo J, Shneiderman B, Plaisant C, Milstein J. ACM Conference on Human-Computer Interaction (CHI 2013) Paris, France: ACM; 2013. The Challenges of Specifying Intervals and Absences in Temporal Queries: A Graphical Language Approach. [Google Scholar]
- 24.Iachello G, Truong KN, Abowd GD, Hayes GR, Stevens M. ACM Conference on Human-Computer Interaction (CHI 2006) Montreal, Canada: ACM; 2006. Prototyping and sampling experience to evaluate ubiquitous computing privacy in the real world. [Google Scholar]