Abstract
For time-sensitive treatment of a patient with malignant melanoma, physicians must obtain a rapid overview of the patient’s status. This study aimed to analyze context-specific features and processes at the point of care to derive requirements for a dashboard granting more straightforward access to information. The Think-Aloud method, contextual inquiries, and interviews were performed with physicians from the Department of Dermatology at the University Hospital Essen in Germany. The user statements and observations that were obtained were grouped and categorized using an affinity diagram. Based on the derived subjects, requirements were defined, confirmed, and prioritized. The resulting affinity diagram revealed four topics of importance at the point of care. These topics are “Identifying and Processing the Important”, a comprehensive “Patient Record”, tasks and challenges in the “Clinical Routine”, and interactions and experiences with the available “Systems”. All aspects have been reflected in 135 requirements for developing context- and indication-specific patient dashboards. Our work has elucidated the most important aspects to consider when designing a dashboard that improves patient care by enabling physicians to focus on the relevant information. Furthermore, it has been demonstrated that the aspects most often mentioned are not context-specific and can be generalized to other medical contexts.
Keywords: Melanoma, Dashboard, Requirements, User experience, Electronic health records, Context exploration
Subject terms: Health care, Skin cancer, Data integration, Software
Introduction
During the treatment of patients diagnosed with melanoma (black skin cancer), it is crucial for dermato-oncologists (dermatologists specialized in cancer) to obtain a complete understanding of the patient’s status immediately. Therefore, accessibility and timeliness of the relevant data are in focus. In recent years, various medical information systems have been used increasingly in the EU to support the process of treatment and documentation at the point of care1; however, the usability of many of these systems has been generally poor2–7. Usability is defined by the accuracy and completeness with which the physician can reach the desired goal (effectiveness), the effort that is needed for task completion (efficiency), and how enjoyable the interaction is throughout the process (enjoyment of use)8. Lack of usability can lead to errors that can be life-threatening to patients3,9,10 and numerous kinds of workarounds, where physicians use the information system differently, than it was intended, to reach their desired goal10. Without addressing the issue of optimizing software for the context in which it will be used, these problems will remain11.
To counteract this well-known problem, methodologies such as user experience12 that take the user, the setting, and the situation of a software use into the focus of the implementation have been introduced. Such a user-centered design can address the problem of high mental load, especially in situations in which many different types of information need to be gathered, combined, and compared under time pressure13,14. Early involvement of users in the development process and an initial, thorough context exploration can lead to the creation of systems with easier accessibility that support physicians in clinical routine and specific tasks15,16, thus increasing productivity14. However, this step is often skipped during the development of most systems17.
Prior research has also revealed common factors that influence the acceptance or rejection of medical information systems2,11,18–20. For the acceptance of a system, users have specified ease of use as a key factor. The ability of the system to integrate into a context-specific workflow and its helpfulness in achieving a set goal have also been labeled as essential. Privacy issues20,21, poor design, long system downtimes, and long response times have led medical staff to reject systems5,19,20.
Initiating software development by placing a focus on context exploration has proven to prevent developers from deriving (defining) inadequate and incomplete requirements22,23. Granting users influence on an application through participation in its development process24 leads to higher acceptance25 and can create enriched designs by including different user insights26. Such designs often focus especially on user-relevant factors and therefore reduce mental load and information loss27. By overcoming the challenges of time constraints in clinical routines and recruiting a sufficient number of participants to represent the context28, user-centered design can improve software acceptance if the identified key factors are interwoven into the solution29.
To exploit the advantages of a user-centered design, our approach involved a combination of methods from the field of user research to explore the context and workflows in which the software would be used. For this purpose, user statements and insights were gathered and combined to gain contextual knowledge. The derived requirements were phrased and ranked according to their impact on the user’s ability to achieve their objective. Following this approach, a complete set of requirements has been established based directly on observations of the context or user statements. These requirements, which form the basis of dashboard development, consider user-related factors, place a focus on data relevant to the treatment of melanoma patients, and offer contextual knowledge to enable workflow integration.
Our study aimed to define a reproducible starting point for context-sensitive dashboard development using early user involvement. Therefore, in this paper, we introduce a set of functional (FR) and non-functional requirements (NFR) for the context of melanoma treatment. Furthermore, we uncovered the underlying user’s intention related to the defined requirements by linking them with the important aspects of this context through an affinity diagram (AD). To address real-world problems in the context, we introduce the unique combination of using the insights gathered from the think-aloud method (TA) and contextual inquiry (CI) in an AD to define requirements.
Methods
Ethics statement
This study was approved by the Ethics Committee of the Medical Faculty (approval number 22-10814-BO for CI and survey) and the Faculty of Engineering (approval number 2111SPKA9493 for TA) of the University of Duisburg-Essen. All procedures were performed in accordance with the ethical standards of the institutional research committees, the 1964 Helsinki Declaration, and its later amendments or comparable ethical standards. All subjects provided their informed consent for inclusion on the TA in written form. The CI and survey protocols did not include any personal data.
Approach
For the definition of requirements, user phrases obtained by the TA method, CI, and continuous feedback regarding derived context descriptions were collected to serve as foundation of an AD revealing the topics most important to the context (Fig. 1). The requirements were then drafted and subsequently ranked regarding their priority by three experts. Physicians with various levels of expertise from the Department of Dermatology at the University Hospital Essen (UME) in Germany participated in these qualitative methods. Senior physicians, specialists, and residents who had just begun working in this department contributed their domain knowledge and personal input (Table 1). Finally, the requirements were reviewed with regard to their relationships to the hospital and excluded if not generally applicable.
Table 1.
Method | Purpose in information processing | Focus of exploration | Number of participants | Data collection method |
---|---|---|---|---|
Think-Aloud | Context exploration | Preparation of tumor board | 10 | Screen recording, voice recording |
Contextual inquiry | Ambulant treatment | 2 | Protocol | |
Initial presentation | 2 | Protocol | ||
Interviews | Drafts of context models | 2 | Protocol | |
Survey | Evaluation of derived requirements | Prioritization of derived requirements | 2 | Online survey |
Think-Aloud method
The TA30,31 method is common in user-centered approaches and has been used frequently to reveal problem areas and factors important to an inspected context32. It is conducted by asking participants to fulfill a task and at the same time express their thoughts verbally. These actions and thoughts are recorded and analyzed to depict the context and build designs based on this knowledge. This method is applicable to any scenario33, has been shown to be highly effective, and allows modeling and design based on voices of real users rather than external interpretations33.
The TA method was conducted with 10 participants from the UME in the field of melanoma treatment. The explored task was the preparation of a fictitious tumor board for a patient in a possibly metastatic setting. Task execution was captured via computer screen and voice recording. To simulate real-world circumstances, this process occurred in the clinic using real patient data from the hospital information systems. The time was limited to 15 min, and no interaction with the conductors occurred other than reminders to think aloud.
The TA provided deep insights into the interactions with the software, the relevant data, how they relate to one another, and how physicians translate the information into decisions. Nevertheless, the time required is very high, especially in this time-critical context. The setting and the task were as realistic as possible, and the participants were informed in advance that the study focused on their interaction with the software and not on their medical decisions. Despite this, some physicians were nervous, felt uncomfortable expressing their professional thoughts out loud, and maybe filtered their statements.
The data gathered was anonymized by voice alteration and transcription of the recorded voices. Afterwards, relevant phrases were extracted from the transcripts and used anonymously as part of the AD.
Contextual inquiry
CI16 is a participatory method in user research, which involves the user in the development process and takes an entire context into focus. By accompanying physicians in conducting the tasks supported by software and observing their interactions, the nature of situations can be revealed. With this approach, direct insights into the setting, intents of usage, and utilization of available IT systems can be gathered2. This knowledge enables the design of software that uses context-specific wording and helps to identify standard solutions that might cause conflicts in this context. Even facts that are subconscious and difficult to articulate can be revealed using this method17. Revealing true needs is a strong basis for deriving requirements based on real insights rather than interpretation by external factors28. Conducting a CI is relevant for a holistic view of the context15,34,35.
CI was conducted with two participants each for the tasks of outpatient care and the initial presentation of new patients. The information gathered was documented in prepared protocols that focused on the setting, interruptions, and interactions with IT systems or colleagues. No personal information regarding the participating physicians was documented.
The CI gave an unfiltered view of the observed tasks, usual settings, and interactions with the medical staff and patients without impacting the physician’s time. The physicians were used to introducing new colleagues to their tasks and being accompanied, which made the CI very realistic and unvaried. Observations were only limited if the patient felt uncomfortable with an observer, or the observer did not want to accompany the medical treatment.
To facilitate empathy during the AD modeling process, user phrases were extracted from the protocols by rephrasing the insights in the first person. In addition, any questions that arose during the modeling phase were collected and discussed in semi-structured interviews with two dermato-oncologists. These insights were similarly extracted from the protocols and inserted into the AD.
Affinity diagram
As part of contextual design, the AD35 was developed to reveal the topics important to an explored context. Building an AD begins with sorting user phrases and insights gained from qualitative methods according to their content. Obtaining a maximum number of statements per group leads to critical reflection, discovery of deeper insights, reorganization, and rephrasing of the groups. Afterwards, these groups are clustered into sub-topics, which again are grouped into topics. The specified groups, sub-topics, and topics are called affinities, from which the name of the diagram originates. During the modeling process, all groups, sub-topics, and topics are given descriptions in I-perspective, which supports empathy during the design process. This structuring of messages leads to the identification and clarification of areas of interest separated into manageable groups36,37 and serves as a solid foundation to derive user-founded requirements38.
Requirements
In computer science, it is important to formulate a need that software must fulfill under given circumstances and known limitations39 in order to ensure the quality of the implementation and determine the purpose of the task. These needs and limitations are defined as requirements and exist in a hierarchical order with more specific sub-requirements. A distinction can be made between FRs, which define a functionality a software must provide, and NFRs, which describe characteristics a software must integrate. To differentiate their importance, a requirement can be prioritized into three levels. Functionalities or characteristics that are necessary to fulfill a task supported by the defined software have the highest priority and shall be implemented. Useful features that are not essential for task completion are requirements that should be included. Aspects facilitating work but not necessary for reaching the set goal may be supported by the software40.
Using these user-centered methods, we have derived a set of requirements for patient dashboard development and optimization in the context of melanoma treatment.
Deriving requirements
To derive requirements based on qualitative data, the transcriptions of TA sessions and CI protocols were split into shorter phrases containing only one message. The user statements were then grouped into three hierarchies and labeled in I-perspective regarding their shared content by modeling the AD. Requirements were determined based on the resulting topics, sub-topics, and groups. Starting with the most abstract level (topics) of the AD, each level down to the very practical user statements was considered individually. The core message of the affinity was extracted and its influence on the dashboard was evaluated. In the next step, this insight was phrased as a new requirement or assigned to an existing one. Afterward, all requirements were examined to determine whether they described a functionality or a feature. Finally, the requirements were grouped by their focus in a hierarchical order, with more specific sub-requirements. The majority of requirements were derived from sub-topics and groups. In very few cases, general requirements were linked to single phrases or specialized sub-requirements referred to a topics.
Evaluation of derived requirements
The derived requirements were evaluated for generalizability; those directly tied to the organization and its specific workflows were excluded. Furthermore, during a survey, the requirements were either prioritized or excluded to ensure correct interpretation of the user phrases by the derived requirements. In this process, two physicians and an IT specialist rated the requirements regarding their prioritization. The following four options were available to classify the requirements, the corresponding values of which are given in brackets: Functionalities or characteristics necessary to fulfill a task supported by the defined software have the highest priority and shall (3) be implemented. Useful features that are not essential for task completion are requirements that should (2) be included. Aspects facilitating work but not necessary for reaching the set goal may (1) be supported by the software. Participants could also mark a requirement as not necessary (0) to check for or add missing aspects. After adding the priorities, the arithmetic mean of the prioritization was finally applied to the list of requirements.
Overall, this study combined the TA method and CI to collect user-statements during the work process. These statements were subsequently sorted in an AD regarding their shared topics. These topics and statements were used to derive requirements that were directly related to the context. To prevent misinterpretation of these statements within the requirements, they were finally ranked regarding their prioritization or filtered out by conducting a survey.
Results
Affinity diagram
Overall, 507 user phrases were extracted from the qualitative data (text from transcripts and protocols) and combined in an AD. The resulting topics were then used to define 135 requirements. Considering the results side by side, it revealed not only required functionalities and features but also underlying user intentions.
Analysis of 507 user phrases organized in an AD revealed four main topics relevant to the context (Fig. 2). These topics included identification of important data related to treatment (“Identifying and Processing the Important”), the importance of a comprehensive patient record (“Patient Record”), integration with clinical routine and experience (“Clinical Routine”), and interaction with available systems (“Systems”). The following section only focuses on AD’s first top hierarchies. The complete set of affinities is included in the supplement (excel sheet tab: “1-overview-affinities”).
Containing 214 sub-elements, “Identifying and Processing the Important” was the most-often mentioned aspect of the task of treating melanoma patients. Outlined by eight sub-topics, it encompasses the challenges of finding the proper starting point, collecting data, and searching for details important to the case even under time pressure. All the information gathered must be evaluated for pertinence (behavior over time) and combined with data from other sources. While maintaining a focused view of the current case, the disease’s progression must also be considered.
A comprehensive “Patient Record” displaying all the required information was the second-most frequently referenced aspect. Information that must be included in a melanoma-specific dashboard was identified by 203 sub-elements in six sub-topics: staging, adverse events, internal and external reports, therapies, determining the patient’s status, and details on the melanoma itself.
The subject of “Clinical Routine” was mentioned in 179 sub-elements. The six sub-topics consider the setting and the level of work experience of the physicians. Having the patient in focus reflects the responsibility of physicians towards their patients, the aim of personalizing treatment, and communication between both physicians and colleagues and physicians and patients. Furthermore, the importance of side tasks during treatment was emphasized.
The last main topic relates to the “Systems” available and interactions with them. In five sub-topics containing 107 sub-elements, pain points (pitfalls) in software interaction, the influence of personal preferences and habits, data quality, and the possibilities of reusing data already known by the system(s) were in focus. This topic also includes references to the dashboard as a new system.
Requirements
A total of 426 of 507 user phrases were reflected in 135 requirements for a dashboard supporting the treatment of melanoma patients, whereas 81 user statements could not be used to define requirements for a dashboard, and three requirements were excluded due to their strong relationship to the workflow at the hospital itself.
The 135 derived requirements could be further classified into 97 which defined functional aspects, and 38 which referred to non-functional characteristics that must be considered. Through prioritization, 39 of the 97 functionalities were marked as indispensable (shall) for the treatment of melanoma patients. To facilitate the process, 52 more of the 97 requirements should be implemented, and six may be an opportunity to simplify the workflow and information retrieval further. The functionalities were grouped into 13 main requirements, which are shown in Table 2 in conjunction with the affinities from which they were derived. In the following description, only the top hierarchy level of requirements with the most frequent assignments to affinities has been presented in detail. The complete catalog of requirements and sub-requirements has been included in the supplement (excel sheet tab: “2-overview-requirements”).
Table 2.
Main requirement ID | Requirement “The dashboard shall/ should/may include…” | Count affinity per requirement group | Affinity-1st sub-topics |
---|---|---|---|
FR00 | An all-encompassing view of all a patient’s data | 69 | Determine patient’s status, usage of data available, responsibility, side tasks, personalization of medicine, level of work experience, data collection, identifying important aspects, combining information, searching for details, focused view |
FR01 | The current staging | 22 | Staging |
FR02 | Ensuring access to external and internal findings | 51 | Internal and external reports, responsibility, side tasks |
FR03 | The master data of the patient | 15 | Data quality |
FR04 | The latest tumor board protocols | 12 | Determine patient’s status, starting points |
FR05 | The medication | 6 | Determine patient’s status, side tasks |
FR06 | The current laboratory values | 16 | Determine patient’s status |
FR07 | Progress documentation | 29 | Determine patient’s status |
FR08 | The histology of the tumor | 9 | Determine patient’s status |
FR09 | An overview of all melanoma-specific aspects | 45 | Melanoma |
FR10 | The therapy | 42 | Therapy |
FR11 | The reference to external documents needed for the fulfillment of the tasks | 24 | Communication, level of work experience |
FR12 | Searching in and filtering lists | 55 | Searching for details, treatment factor time |
NFR00 | Enable the user to get an overview even when time is short | 111 | Information processing |
NFR01 | Meet the requirements for data protection | 10 | Data quality |
NFR02 | Limit the respective view to the use case‐dependent essential information | 29 | Focused view |
NFR03 | Arrange elements based on the logic of melanoma treatment | 15 | Focused view |
NFR04 | Optimize work in any task‐appropriate usage setting | 24 | Personal factors, communication |
Requirements have been represented by their ID and their specific requirement text. The affinities have been reflected in the counts of affinities associated with the requirement group (main requirement with sub-requirements), alongside assigned first sub-topics.
Based on the number of user phrases associated with the functional aspects, four key factors could be identified. The first factor (FR00), associated with 69 user phrases, states that an all-encompassing view of all a patient’s data, with straightforward access, is the most important functionality. A second consideration is the importance of sorting and filtering functionalities to search available data for necessary information and to reflect the patient’s status in chronological order. This aspect was mentioned in 55 user statements (FR12). Mentioned by 51 users, access to original external and internal reports ranked third. These reports are main sources of information and are useful for identifying information regarding documentation of progress and fulfillment of side tasks related to patients (FR02). The fourth aspect, mentioned 45 times by participants, refers to obtaining an overview of the melanoma itself (FR09). This overview includes the details of the primary tumor, in particular its location, tumor diagnosis, classification, lymph node status, location and confirmation of metastases, and the development of the tumor over time (Table 3).
Table 3.
Requirement ID | Priority | Functionality | Count statement per requirement |
---|---|---|---|
FR09.01 | SHALL | The current status of the primary tumor | 5 |
FR09.01.01 | SHALL | The localization of the primary tumor | 1 |
FR09.02 | SHALL | The tumor diagnosis | 7 |
FR09.02.01 | SHOULD | The confirmation status of the diagnosis | 4 |
FR09.03 | SHALL | The TNM classification | 7 |
FR09.03.01 | SHALL | The tumor stage | 3 |
FR09.04 | SHOULD | The change of the tumor stage over time | 8 |
FR09.05 | SHOULD | The current status of the lymph nodes | 4 |
FR09.06 | SHALL | Information regarding possible metastases | 2 |
FR09.06.01 | SHOULD | The type and location of the metastasis | 1 |
FR09.06.02 | SHOULD | The status of the confirmation of metastasis findings | 2 |
The user phrases have been reflected in the counts of statements associated with each requirement.
Apart from the functionalities required, 38 NFRs were also derived to describe the characteristics that a dashboard should exhibit. According to the prioritization by two domain experts and one IT-expert, 18 of these attributes are indispensable (shall), 18 could simplify (should), and two may enrich the process of melanoma patients’ treatment. The characteristics have been grouped into five main requirements, along with their associations with the affinities from which they were derived.
Mentioned most often, with 111 assigned affinities, is the requirement (NFR00) for support of information processing through an overview that not only contains all necessary information but also is especially easy to interpret. Assigned to 29 user phrases, a view adapted to the task at hand can assist in obtaining a focused view of the relevant information (NFR02). The adaptation of the software to different settings was mentioned in 24 affinities (NFR04). Such adaptation includes flexibility with personal preferences and habits and support for communication. Optimizing a view while considering melanoma domain knowledge can provide experts with a focused and more easily interpretable view (NFR03), as highlighted by 15 user statements. According to 10 affinities, a dashboard must also ensure data protection, quality, and reproduction true to the original information (NFR01).
Discussion
Our work has provided a set of important requirements at the point of care in the treatment of melanoma patients. These requirements were derived from user phrases using an AD to identify and clarify the most important aspects. Using CI and the TA method, insights were gained by observing the working routine rather than making assumptions.
In contrast to other publications that have employed user-centered approaches only for final evaluation, this approach emphasized the value of early involvement and thorough context exploration before designing and implementing the system. Although the benefits of direct insights into context have been shown previously, user-centered methods have remained sparsely used in the field of medicine, especially at the initial stages of the development process. For example, CI has been used to gain insights into the workflows of telecardiology and derive requirements based on these41. The AD has been used as a basis for determining topics that need to be investigated in further detail for daily morning surgical rounds in intensive care units (ICUs)42. Another publication has described two studies that applied a combination of CI and AD to derive requirements for the dictation process of physicians and a nursing documentation system43. By bringing these approaches to the field of melanoma treatment, unique requirements for this context were derived in the present study. To our knowledge, no other work has combined the TA, CI, and AD in one study to derive requirements for the development of a patient dashboard.
The results reported here demonstrate the strength of this approach. They have highlighted generalizable aspects of dashboard development and have also provided detailed insights into domain-specific needs. The factors mentioned most often refer to general aspects, such as gaining a comprehensive view even under time pressure, which would be transferable to other dashboard projects. Additionally, detailed insights into the medical aspects of melanoma have been highlighted. This includes specific prioritized data and a comprehensive view of the patient’s information that is needed due to the high impact of cancer on the patient’s entire body. Different types of cancer have different markers and values, such as typical genetic alterations, which need to be considered in diagnosis and treatment. However, similar diagnostic procedures are often used, resulting in the same data types. This implies that the very context-specific requirements are also highly reusable for other types of cancer. Further research will have to evaluate the generalizability of information-related requirements as well as those referring to general aspects. Addressing the related contexts of (skin) cancer treatment will give first valuable insights into the scalability of the requirements.
One limitation of this approach is that each context must be investigated separately to obtain context-specific insights. Context exploration must be performed for each discipline and does not offer a generally valid solution for every application. Different contexts might involve different settings the software will be used in, like mobile usage or limited software interaction possibilities as usual in operation rooms. Different diseases will lead to different data being prioritized during decision-making. However, the generalizable aspects identified can be used to focus primarily on domain-specific details in future projects. Requirements referring to getting an overview of the patient’s status or searching and filtering functions are most likely to be transferable even to other disciplines. More specialized aspects, like the information prioritized, might be partly reusable for other cancers and highly reusable for other skin cancers. Those differences will always need to be evaluated in different settings.
A further limitation is that these results were derived based on a single hospital. Whether our requirements can be implemented directly in other hospitals needs to be evaluated. Differences in workflows and related data acquisition might as well be hindering the transferability of the requirements, as might differences in the user group regarding the working experience and the user’s technical affinity. Those aspects might lead to different data needed during the treatment and shift focus on their prioritization. As treatment for melanoma follows guidelines, the transferability of the requirements might strongly depend on the implementation of the guidelines in different hospitals.
Overall, the applicability of the TA method and especially CI have proven to be valuable in this time-critical context. Combining, on the one hand, the TA, which has a specific task in a staged setting in focus, with the CI, where physicians are accompanied during working routines, has opened a broad view of the context from several directions. They have demonstrated utility in gaining deep insights into this complex context, making it accessible even for people not directly involved in that context, such as developers, through an AD.
The derived requirements can serve as a foundation for iteratively incorporating these needs into a dashboard design, allowing the developer to reflect on the design with the user’s intentions in mind. Deriving requirements with a user-centered approach was only the first step in the development of a context-, patient- and user-optimized dashboard for the treatment of melanoma. Future research needs to explore how the relevant data can be termed, grouped44,45, and visualized45,46 while addressing different, even contradictory personal preferences47,48. Data coming from different clinical information systems might be, duplicated, incomplete, and contradictory. The challenge of preventing the user from overload while evaluating the combined information will need to be addressed49.
Varying software interfaces of the different information systems can cause problems in the merging of data. It needs to be explored how interoperability between these systems can be established while ensuring data quality20,49. Combining this research with continuous user feedback and testing for developed concepts, prototypes, and the end product, can ensure the correct implementation15 and the usability of medical systems can be sustainably improved.
In summary, this study made two main contributions to the field of medical informatics. On one hand, it identified critical requirements for melanoma patient treatment at the point of care through user-centered methods. Through early user involvement and thorough context exploration, our approach provided valuable insights into workflows and specific needs. The highlighted generalizable aspects can, furthermore, be used as a foundation for dashboards in different medical domains. On the other hand, the study demonstrated the utility of CI and TA in understanding complex contexts, making the knowledge gained accessible to developers through an AD and reusable by the definition of requirements.
It points out a combination of methods that are proven to be applicable in the time-critical and complex context of medical treatment to define an implementation instruction for a dashboard that is aware of the patient’s context and the user. This user-centered focus in development ensures that the dashboard will address real-world needs and counteract errors related to missing context integration. Thereby, making the access to data easier for the physicians allows them to focus only on the important aspects and make decisions based on them.
Supplementary Information
Acknowledgements
The authors thank the physicians of the Department of Dermatology (University Hospital Essen) for their dedicated participation in the usability studies. They also thank Prof. Dr. Peter Haas (University of Applied Sciences and Arts Dortmund) and Dr. Sylvia Nürnberg (WisPerMed) for their comments on the manuscript and supportive discussions. They thank Dr. Katharina Zeiner (Siemens, Munich) for her mentoring and fruitful discussions regarding the user experience methodology. All figures in this manuscript were created by the first author Eva Hartmann using Miro board online (https://miro.com/) for Fig. 1 and Excel—Office 365 version 2405 (https://www.microsoft.com/en-us/microsoft-365/excel) for Fig. 2.
Author contributions
EMH: conceptualization, methodology, investigation, data curation, visualization, formal analysis, software, writing—original draft preparation. AK: data curation, methodology, investigation, formal analysis, writing—review, and editing. GCL, EL, and DS: conceptualization, resources, writing—review, and editing. JS and CLB: methodology, data curation, writing—review, and editing. SS: supervision, conceptualization, methodology, writing—review, and editing.
Funding
Open Access funding enabled and organized by Projekt DEAL. This work was funded by Ph.D. grants from the DFG Research Training Group 2535, “Knowledge- and data-based personalization of medicine at the point of care (WisPerMed)”.
Data availability
The authors confirm that the data supporting the findings of this study is available within the article and its supplementary materials or upon reasonable request to the corresponding author.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
The online version contains supplementary material available at 10.1038/s41598-024-67857-2.
References
- 1.European Commission, RAND Europe, Open Evidence, BDI Research. Benchmarking Deployment of eHealth Among General Practitioners: Final Report (Publications Office, 2018). [Google Scholar]
- 2.Viitanen, J. et al. National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. Int. J. Med. Inform.80(10), 708–725. 10.1016/j.ijmedinf.2011.06.010 (2011). 10.1016/j.ijmedinf.2011.06.010 [DOI] [PubMed] [Google Scholar]
- 3.Nielsen Norman Group. Medical Usability: How to Kill Patients Through Bad Design.https://www.nngroup.com/articles/medical-usability/ (Accessed 7 December 2023).
- 4.Viitanen, J., Tyllinen, M., Tynkkynen, E. & Lääveri, T. Usability of information systems: Experiences of outpatient physicians, outpatient nurses, and open care social welfare professionals from three large cross-sectional surveys in Finland. Int. J. Med. Inform.165, 104836. 10.1016/j.ijmedinf.2022.104836 (2022). 10.1016/j.ijmedinf.2022.104836 [DOI] [PubMed] [Google Scholar]
- 5.Hudson, D., Kushniruk, A., Borycki, E. & Zuege, D. J. Physician satisfaction with a critical care clinical information system using a multimethod evaluation of usability. Int. J. Med. Inform.112, 131–136. 10.1016/j.ijmedinf.2018.01.010 (2018). 10.1016/j.ijmedinf.2018.01.010 [DOI] [PubMed] [Google Scholar]
- 6.Topaz, M. et al. Nurse informaticians report low satisfaction and multi-level concerns with electronic health records: Results from an International Survey. AMIA Annu. Symp. Proc.2016, 2016–2025 (2016). [PMC free article] [PubMed] [Google Scholar]
- 7.Vainiomäki, S. et al. Better usability and technical stability could lead to better work-related well-being among physicians. Appl. Clin. Inform.8(4), 1057–1067. 10.4338/ACI-2017-06-RA-0094 (2017). 10.4338/ACI-2017-06-RA-0094 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ergonomics of Human–System Interaction: Part 11: Usability: Definitions and Concepts, 9241-11:2018(en) (International Organization for Standardization, 2018).
- 9.Kushniruk, A. W., Triola, M. M., Borycki, E. M., Stein, B. & Kannry, J. L. Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application. Int. J. Med. Inform.74(7–8), 519–526. 10.1016/j.ijmedinf.2005.01.003 (2005). 10.1016/j.ijmedinf.2005.01.003 [DOI] [PubMed] [Google Scholar]
- 10.Blijleven, V., Hoxha, F. & Jaspers, M. Workarounds in electronic health record systems and the revised sociotechnical electronic health record workaround analysis framework: Scoping review. J. Med. Internet Res.24(3), e33046. 10.2196/33046 (2022). 10.2196/33046 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kaipio, J. et al. Usability problems do not heal by themselves: National survey on physicians’ experiences with EHRs in Finland. Int. J. Med. Inform.97, 266–281. 10.1016/j.ijmedinf.2016.10.010 (2017). 10.1016/j.ijmedinf.2016.10.010 [DOI] [PubMed] [Google Scholar]
- 12.Norman, D., Miller, J. & Henderson, A. What you see, some of what’s in the future, and how we go about doing it. In Conference Companion on Human Factors in Computing Systems—CHI’95 155 (1995).
- 13.Belden, J. L. et al. Designing a medication timeline for patients and physicians. J. Am. Med. Inform. Assoc.26(2), 95–105. 10.1093/jamia/ocy143 (2019). 10.1093/jamia/ocy143 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems 9241-210:2019 (International Organization for Standardization, 2019).
- 15.van Gemert-Pijnen, J. E. W. C. et al. A holistic framework to improve the uptake and impact of eHealth technologies. J. Med. Internet Res.13(4), e111. 10.2196/jmir.1672 (2011). 10.2196/jmir.1672 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Karen, H. & Sandra, J. Contextual inquiry: A participatory technique for system design. In Participatory Design: Principles and Practices 1st edn (eds Schuler, D. & Namioka, A.) 177–210 (CRC, 1993). [Google Scholar]
- 17.Kip, H., Beerlage-deJong, N. & Wentzel, J. The contextual inquiry. In eHealth Research, Theory, Development: A Multi-disciplinary Approach (eds van Gemert-Pijnen, L. et al.) 168–186 (Taylor and Francis, 2018). [Google Scholar]
- 18.Bleich, H. L. & Slack, W. V. Reflections on electronic medical records: When doctors will use them and when they will not. Int. J. Med. Inform.79(1), 1–4. 10.1016/j.ijmedinf.2009.10.002 (2010). 10.1016/j.ijmedinf.2009.10.002 [DOI] [PubMed] [Google Scholar]
- 19.Huryk, L. A. Factors influencing nurses’ attitudes towards healthcare information technology. J. Nurs. Manag.18(5), 606–612. 10.1111/j.1365-2834.2010.01084.x (2010). 10.1111/j.1365-2834.2010.01084.x [DOI] [PubMed] [Google Scholar]
- 20.Schopf, T. R., Nedrebø, B., Hufthammer, K. O., Daphu, I. K. & Lærum, H. How well is the electronic health record supporting the clinical tasks of hospital physicians? A survey of physicians at three Norwegian hospitals. BMC Health Serv. Res.19(1), 934. 10.1186/s12913-019-4763-0 (2019). 10.1186/s12913-019-4763-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Steininger, K. & Stiglbauer, B. EHR acceptance among Austrian resident doctors. Health Policy Technol.4(2), 121–130. 10.1016/j.hlpt.2015.02.003 (2015). 10.1016/j.hlpt.2015.02.003 [DOI] [Google Scholar]
- 22.Schön, E.-M., Thomaschewski, J. & Escalona, M. J. Agile requirements engineering: A systematic literature review. Comput. Stand. Interfaces49, 79–91. 10.1016/j.csi.2016.08.011 (2017). 10.1016/j.csi.2016.08.011 [DOI] [Google Scholar]
- 23.Maguire, M. Using human factors standards to support user experience and agile design. In Lecture Notes in Computer Science, Vol. 8009, Universal Access in Human-Computer Interaction: 7th International Conference, UAHCI 2013, Held as Part of HCI International 2013, Las Vegas, NV, USA, July 21–26, 2013; Proceedings (eds. Stephanidis, C. & Antona, M.) 185–194 (Springer, 2013).
- 24.Martikainen, S., Kaipio, J. & Lääveri, T. End-user participation in health information systems (HIS) development: Physicians’ and nurses’ experiences. Int. J. Med. Inform.137, 104117. 10.1016/j.ijmedinf.2020.104117 (2020). 10.1016/j.ijmedinf.2020.104117 [DOI] [PubMed] [Google Scholar]
- 25.Groeneveld, S. W. M., den Ouden, M. E. M., van Gemert-Pijnen, J. E. W. C., Verdaasdonk, R. M. & van Os-Medendorp, H. Underestimated factors regarding the use of technology in daily practice of long-term care: Qualitative study among health care professionals. JMIR Nurs.6, e41032. 10.2196/41032 (2023). 10.2196/41032 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Harris, M. A. & Weistroffer, H. R. A new look at the relationship between user involvement in systems development and system success. Commun. Assoc. Inf. Syst.24(1), 442. 10.17705/1CAIS.02442 (2009). 10.17705/1CAIS.02442 [DOI] [Google Scholar]
- 27.Norman, D. A. Things That Make Us Smart: Defending Human Attributes in the Age of the Machine (Basic Books, 1993). [Google Scholar]
- 28.Kushniruk, A. & Nøhr, C. Participatory design, user involvement and health IT evaluation. Stud. Health Technol. Inform.222, 139–151 (2016). [PubMed] [Google Scholar]
- 29.Bano, M., Zowghi, D. & da Rimini, F. User satisfaction and system success: An empirical exploration of user involvement in software development. Empir. Softw. Eng.22(5), 2339–2372. 10.1007/s10664-016-9465-1 (2017). 10.1007/s10664-016-9465-1 [DOI] [Google Scholar]
- 30.Simon, H. A. & Newell, A. Human problem solving: The state of the theory in 1970. Am. Psychol.26, 145–159 (1971). 10.1037/h0030806 [DOI] [Google Scholar]
- 31.Lewis, C. Using the “Thinking-Aloud” Method in Cognitive Interface Design. https://dominoweb.draco.res.ibm.com/reports/RC9265.pdf (Accessed 7 December 2023) (1982).
- 32.Maramba, I., Chatterjee, A. & Newman, C. Methods of usability testing in the development of eHealth applications: A scoping review. Int. J. Med. Inform.126, 95–104. 10.1016/j.ijmedinf.2019.03.018 (2019). 10.1016/j.ijmedinf.2019.03.018 [DOI] [PubMed] [Google Scholar]
- 33.Jacobsen, J. & Meyer, L. Praxisbuch Usability und UX: Was alle wissen sollten, die Websites und Apps entwickeln 3rd edn. (Rheinwerk Verlag, 2022). [Google Scholar]
- 34.Kip, H. et al. Methods for human-centered ehealth development: Narrative scoping review. J. Med. Internet Res.24(1), e31858. 10.2196/31858 (2022). 10.2196/31858 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Holtzblatt, K. & Beyer, H. Contextual Design: Defining Customer-Centered Systems 1st edn. (Morgan Kaufmann, 1997). [Google Scholar]
- 36.Simon, R. W. & Canacari, E. G. A practical guide to applying lean tools and management principles to health care improvement projects. AORN J.95(1), 85–100. 10.1016/j.aorn.2011.05.021 (2012). 10.1016/j.aorn.2011.05.021 [DOI] [PubMed] [Google Scholar]
- 37.Beyer, H. & Holtzblatt, K. Contextual design. Interactions6(1), 32–42. 10.1145/291224.291229 (1999). 10.1145/291224.291229 [DOI] [Google Scholar]
- 38.Coble, J. M., Maffitt, J. S., Orland, M. J. & Kahn, M. G. Contextual inquiry: Discovering physicians’ true needs. In Proceedings. Symposium on Computer Applications in Medical Care 469–473 (1995). [PMC free article] [PubMed]
- 39.International Standard—Systems and Software Engineering—Life Cycle Processes—Requirements Engineering 29148 (ISO/IEC/IEEE).
- 40.Bradner, S. Key Words for Use in RFCs to Indicate Requirement Levels (1997).
- 41.Gil-Rodríguez, E. P. et al. Organizational, contextual and user-centered design in e-health: Application in the area of telecardiology. In HCI and Usability for Medicine and Health Care: Third Symposium of the Workgroup Human–Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2007 Graz, Austria, November, 22, 2007. Proceeding 69–82 (2007).
- 42.Tripathi, S., Naevor, A. J., Henrekin, L. L. & Welke, K. F. Design and development of daily morning surgical rounds in ICU by quality function deployment. Pediatr. Qual. Saf.4(3), 171. 10.1097/pq9.0000000000000171 (2019). 10.1097/pq9.0000000000000171 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Viitanen, J. Contextual inquiry method for user-centred clinical IT system design. In User Centred Networked Health Care 965–969. https://ebooks.iospress.nl/volumearticle/14314 (Accessed 7 December 2023) (IOS Press, 2011). [PubMed]
- 44.Charbonneau, D. H. & James, L. N. FluView and FluNet: Tools for influenza activity and surveillance. Med. Ref. Serv. Q.38(4), 358–368. 10.1080/02763869.2019.1657734 (2019). 10.1080/02763869.2019.1657734 [DOI] [PubMed] [Google Scholar]
- 45.Tory, M. & Möller, T. Human factors in visualization research. IEEE Trans. Vis. Comput. Graph.10(1), 72–84. 10.1109/TVCG.2004.1260759 (2004). 10.1109/TVCG.2004.1260759 [DOI] [PubMed] [Google Scholar]
- 46.Padilla, L. M., Creem-Regehr, S. H., Hegarty, M. & Stefanucci, J. K. Decision making with visualizations: A cognitive framework across disciplines. Cogn. Res. Princ. Implic.3, 29. 10.1186/s41235-018-0120-9 (2018). 10.1186/s41235-018-0120-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Conati, C. & Maclaren, H. Exploring the role of individual differences in information visualization. In Proc. Working Conference on Advanced Visual Interfaces 199–206 (2008).
- 48.Janssen, A. et al. Electronic health records that support health professional reflective practice: A missed opportunity in digital health. J. Healthc. Inform. Res.6(4), 375–384. 10.1007/s41666-022-00123-0 (2022). 10.1007/s41666-022-00123-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Alami, H. et al. Rethinking the electronic health record through the quadruple aim: Time to align its value with the health system. BMC Med. Inform. Decis. Mak.20(1), 32. 10.1186/s12911-020-1048-9 (2020). 10.1186/s12911-020-1048-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The authors confirm that the data supporting the findings of this study is available within the article and its supplementary materials or upon reasonable request to the corresponding author.