Usability evaluation of an innovative virtual care platform with remote monitoring functionalities for residential aged care homes: A cognitive walkthrough study

Abstract

Background

Virtual care (VC) can improve timely access to general practitioners (GPs) for residents in residential aged care homes (RACHs). However, despite rapid implementation, little empirical work has examined the learnability and task-level usability of these systems in aged care settings, where staff operate under time pressure and residents often have complex needs.

Objective

This study aimed to evaluate the usability of a virtual care platform used to connect residents in RACHs with GPs, using a cognitive walkthrough (CW) approach to examine system learnability and safety-critical task performance within GP and nursing workflows.

Methods

Together with a doctor, nurse and an older adult advisor, two human factors experts walked through 10 clinician tasks (GP=4, nurse=6) and 44 subtasks (GP=13, nurse=31) using the VC technologies, and identified problems, directed by four established CW prompts. Using the problems list, we rated problem severity based on clinician workflow prior to descriptive analysis.

Results

A total of 45 usability problems were identified, over half affecting task learnability. Most learnability problems occurred during nursing tasks associated with vital-sign capture, where required actions and system feedback were frequently unclear to users. Several problems were rated as major or catastrophic, with implications for nursing workflow reliability, task interruption and patient safety. Additional issues related to inadequate error prevention and recovery by design.

Conclusion

This study provides one of the first applications of cognitive walkthrough to evaluate a virtual care platform in a residential aged care context. By revealing safety-critical learnability and error-prone design issues that are often missed by satisfaction-based evaluations, the findings demonstrate the value of task-level usability assessment for supporting safe and effective virtual care delivery in aged care settings. This research has implications for improving user experience, preventing unintended consequences, and enhancing the quality-of-care delivery.

Introduction

A changing healthcare landscape, increasing use of health technologies and the occurrence of the COVID-19 pandemic have contributed to the significant attention virtual care delivery has received in recent times. Virtual models of care involve the use of technologies for tele/videoconferencing, remote monitoring, and clinical decision support.^1,2 Within this context, virtual consultations can be delivered using synchronous (e.g. telephone consults, video, text/image messaging) and asynchronous (e.g. Email consultations) channels. ³ The adoption of a virtual model of care can impact all six domains of quality of care, as defined by the Institute of Medicine i.e. timeliness, efficiency, patient-centredness, effectiveness, safety, and equity. ⁴ For example, existing studies have shown that virtual care can improve primary care efficiency by facilitating the remote triage of patients, identifying those who require urgent face-to-face care from those who can be managed virtually.^5,6

Despite the widespread uptake of virtual care services, evaluation has lagged behind implementation, as a systematic review of 16 studies on the usability of virtual care monitoring solutions, identified very limited studies focusing on evaluating usability. ⁷ Of the limited studies, the review found that many users of virtual care monitoring solutions reported a complex interface, difficulty in using the display, absence of instructions and a lack of digital literacy expertise. ⁷ Virtual care implementation continues to expand and health service organisations are now considering how to translate and scale virtual care models to settings and cohorts where needs and benefits are likely to be greatest. ⁸ Residential aged care homes (RACHs) are one such setting, where virtual care has the potential to enable timely and efficient access to primary care providers for residents.^9,10 However, there are many reports of challenges associated with virtual care technology adoption including barriers related to the technology (e.g. usability, workflow integration), users (e.g. physical, cognitive, or sensory impairment in older adults, lack of motivation among providers), organization (e.g. staff turnover), and broader context (e.g. legislative issues).^9,11 In RACH settings, where clinicians work under time pressure and residents often have complex needs, poor usability and limited learnability of virtual care technologies may not only hinder adoption but also disrupt workflows, increase cognitive load, and introduce risks to patient safety and quality of care.

Effective and regular training on virtual care technologies is often recommended to overcome some of these adoption barriers in RACH settings.^12,13 However, ongoing staff training is challenging due to the characteristics of the RACH context.^9,14 For example, ⁹ RACH settings have been reported to experience high job strain and staff overwork, alongside high staff turnover, and a growing workforce shortage.^15–17 Furthermore, there are a high proportion of staff working in RACH settings with limited qualifications, typically trained overseas, and with low digital literacy skills.^12,17 In this context, systems that are difficult to learn pose particular challenges, as reliance on ongoing training is often unrealistic in RACH settings characterised by high workload, staff turnover, and variable digital literacy. Given these unique contextual and user factors, rather than relying solely on the level of digital literacy of RACH staff, virtual care technologies should be easy to learn. Yet, little empirical work has examined whether virtual care platforms used in residential aged care are learnable by novice users performing routine, safety-critical tasks.

For a virtual care technology to be considered fully usable, it has been suggested that they should demonstrate Nielsen’s five attributes of usability: learnability, efficiency, memorability, errors and satisfaction. ¹⁸ However, the extent to which virtual care technologies used in RACH settings have these five usability attributes remains unclear. The aim of this study was to evaluate the usability of a virtual care platform used to connect residents in residential aged care homes with primary care providers.

Methods

Study design

We employed cognitive walkthrough (CW) to assess usability, addressing a gap identified in a systematic review of 16 studies on the usability of virtual care monitoring solutions which found that none of the studies used CW to evaluate usability. ⁷ Cognitive walkthrough is a structured task-based method of evaluation that assesses ease of learning.^19,20 Through this method, evaluators (e.g. designers, software developers, target users, and human factors experts) simulate the target user’s cognitive processes when following a sequence of actions to perform specified tasks. ²¹ This process examines the ease of performing each step for novice users. ²² Existing digital health evaluations have often focused on adoption and satisfaction measures, with comparatively less attention to task-level usability issues such as learnability and error recovery.^7,23 The CW method was selected because it focuses specifically on learnability and novice users’ ability to successfully complete task sequences, making it well suited to evaluating virtual care technologies in RACH settings where staff have varying levels of digital literacy and limited opportunity for ongoing training. Consistent with established usability engineering practice, this cognitive walkthrough was conducted as a formative inspection intended to identify potential learnability and safety risks, rather than to validate performance or quantify real-world error rates.

The virtual care platform and where it was implemented

The suite of virtual care technologies includes a platform with video conferencing functions, secure document/image sharing, a 6-in-1 remote monitoring device for measuring vital signs (e.g. temperature and oxygenation), and other remote technologies that connect to the 6-in-1 device via Bluetooth technology (e.g. digital stethoscope, otoscope, throat scope, urine analyser). The virtual care platform also provides integration with some RACH and GP clinical information systems. For this study, our evaluation was focused on the virtual care platform.

The suite of virtual care technologies was implemented at the Sydney North Health Network (SNHN) which spans a large geographic area (approx. 900km ² ) covering nine local government areas. It includes 103 RACHs with around 9000 beds, and 288 general practices with 756 GPs. The virtual care service was implemented to improve the level of service from GPs into RACHs as a low proportion (18.1%) of those aged 75+ years received a health assessment from their GP in the 12 months from January 2022. Across the RACHs that opted-in to the service, SNHN provided a suite of hardware, software and education in 2024.

Target users

General practitioners and nurse practitioners in primary care settings that deliver care to RACHs were the intended users of the platform, as were nurses who care for residents in RACHs.

Evaluators

As existing evidence indicates that 3-5 evaluators are sufficient to perform a CW evaluation, five evaluators with complementary expertise participated in this study to ensure coverage of GP, nursing and resident-related domains.^22,24 The evaluators were all part of the research team, who co-designed the study together, and these included two human factors experts, one GP, one RACH nurse and one older person (older adult advisor aged 65+ in a RACH, enabling the inclusion of age-related usability considerations relevant to the target population). The team of evaluators consisted of expertise in qualitative research, human factors, implementation science, psychology, information technology, medicine, and nursing. The human factors experts served as the primary evaluators and conducted all steps of the cognitive walkthrough, including task execution and problem identification. The GP and nurse acted as clinical domain representatives and participated only in severity rating of identified problems, while the simulated resident contributed resident-perspective insights and assisted in identifying a subset of usability issues relevant to the resident interface. All evaluators were novice users of the virtual care technologies.

Data collection and analysis

Data collection occurred at the University of Sydney’s simulation laboratory on the 23^rd of July 2024 and lasted 3 hours. The CW procedure was conducted across three steps, based on existing literature^25,26: 1) evaluation preparation, 2) evaluation execution, 3) analysing problems.

Step 1: Evaluation preparation

Together with a RACH nurse and a GP, scenarios, tasks, action sequences, target users, and documents were developed based on the most frequent and routine tasks undertaken by system users. See Box 1 for the scenarios, Table 1 for number of tasks and sub-tasks in each scenario, and Appendices A and B for a complete list of all tasks and actions.

Box 1. GP and nurse scenarios

Scenario 1: A GP scenario on conducting a virtual care consult with a RACH resident

A RACH resident requires a consult with a GP. The nurse uses the Health Teams platform to generate a link and sends it to the general practice’s reception. The GP clicks on the link to the telehealth call from their appointment system. During the telehealth appointment, the GP views the resident’s information and vital signs, and undertakes the consultation with the resident, staff and carer (if available). The nurse documents key components of the consultation in the Health Teams platform and the GP simultaneously creates notes and, if required, prescriptions in their practice’s clinical information system.

Scenario 2: A RACH Nurse scenario on facilitating a virtual care consult between a GP and RACH resident

A RACH resident requires a consult with a GP. The RACH nurse needs to create a request for consultation with the relevant GP via the Health Teams dashboard. The RACH nurse needs to capture the resident’s vitals and document the relevant information in the resident’s profile. At the appointed time for the consult, the RACH nurse connects with the GP and then the resident and family member (if applicable).

Table 1.

Number of tasks and sub-tasks in each scenario.

Scenario number	Scenario description (see Box 1)	Tasks	Sub-tasks
1	GP scenario	4	13
2	Nursing scenario	6	31

Step 2: Evaluation execution

Executing these scenarios using the virtual care technologies required the completion of multiple sub-tasks. Using the scenarios, tasks and sub-tasks identified in step 1, we developed a checklist to evaluate the execution of the sub-tasks and their actions when using the technologies. This checklist included the tasks, sub-tasks, and the four questions recommended to be asked for the evaluation of each action in cognitive walkthroughs (see Tables 2 and 3). ²¹

Table 2.

Cognitive walkthrough evaluation questions.

Question number	Question	Descriptive theme
1	Will the user try to achieve the right effect? Do users understand that the action (step) at hand is needed to reach their larger goal?	Notifying of actions at hand
2	Will the user notice that the correct action is available? Is the interactive element that achieves the step visible or easily findable?	Existence of necessary controls
3	Will the user associate the correct action with the effect that the user is trying to achieve? Perhaps the right button is visible, but will users understand the label and will they know to engage with it?	Visibility and clarity of controls for users
4	If the correct action is performed, will the user see that progress is being made toward the solution of the task? Based on what occurs after the action is taken, will users know that this action was correct and helped them make progress toward their larger goal?	User perception of the work process in the system

Table 3.

An example of the checklist.

Task	Sub-task	Questions
		Q1	Q2	Q3	Q4
1) Clicking the link to the telehealth call	1a) Open email and click the “join the consult” link	​	​	​	​
	1b) Accept “terms of use” and click “join”	​	​	​	​
	1c) Click “join the call”	​	​	​	​

Two evaluators executed each task together and completed the checklist. The answer to the checklist questions were “yes” if no problem was detected, and “no” if a problem was identified. For each problem detected, the evaluators entered a description and location of the problem into the checklist. In addition to working through the cognitive walkthrough evaluation questions for each sub-task, the evaluators reviewed the platform to identify other usability problems, not directly related to learnability. Once the evaluation process was completed, the evaluators reviewed the list of usability problems to ensure that problems were accurately described. Although evaluators were members of the research team, potential confirmation bias was mitigated through the use of novice users, a structured cognitive walkthrough protocol, and paired task execution to support cross-checking of observations.

Step 3: Analysing the problems

Usability problems were classified based on Nielsen’s five attributes: learnability (based on the CW process), efficiency, memorability, errors and satisfaction ¹⁸ (see definitions in Appendix C). To determine the severity of the usability problems identified, we used Nielsen’s ²⁷ severity rating method and assigned a score from 0 to 4 to each problem (0 = not a problem. 1 = cosmetic problem, does not need to be fixed. 2 = minor problem, low priority for fixing 3 = major problem, high priority for fixing. 4 = usability catastrophe, imperative to be fixed). Severity was broadly determined by three factors (i.e. frequency, impact, persistence) that have been established in the literature^25,28: 1) the proportion of users who might encounter the problem; (2) the impact of the usability problem on users’ interaction with the system and on residents; and, (3) whether the problem tended to be temporary or long-lasting over the course of using the system. These factors were considered qualitatively and in combination during severity rating, consistent with Nielsen’s severity assessment approach, rather than being independently scored. Severity judgments reflected anticipated impact on routine clinical workflow and patient safety and were determined through role-based consensus discussion. This meant that a persistent problem with a major impact that most users would experience was assigned the highest severity rating. To determine severity, we prioritised the voices of the GP for GP workflow and the nurse for the nursing workflow, who were the target users of the system. Severity ratings were determined through a role-based consensus process. For the GP workflow, the GP met with two human-factors experts, providing clinical reasoning regarding frequency, impact, and persistence of each problem while the human-factors experts ensured consistent application of Nielsen’s severity criteria. For this purpose, the problems list with a screenshot of the problem location in the system was provided to the GP in a 2-hour meeting to discuss the severity of each problem based on the GP workflow. The same process was followed for the nursing workflow, with the RACH nurse completing severity ratings together with the same two human-factors experts. During each session, any initial differences in interpretation were resolved through discussion until consensus was reached. As workflows were rated in separate role-specific sessions, no cross-role pooling or averaging was conducted. All evaluators executed tasks directly on the platform; screenshots were used solely to support structured discussion during severity-rating consensus meetings. In total, the evaluation session and two severity rating sessions lasted 7 hours. Data were analysed descriptively.

Ethics approval and consent

This study was a cognitive walkthrough usability inspection, which is an expert-based evaluation method and does not involve human participants as research subjects. All individuals involved in the evaluation were members of the research team or acted in advisory roles contributing professional or experiential expertise to the design assessment. No participant recruitment, patient interaction, or collection of personal or health data occurred. On this basis, the study did not meet the definition of human research requiring review by a Human Research Ethics Committee. As no human research participants were involved, written informed consent was not required.

Results

Overall, the identified usability problems clustered around three areas, with learnability-related issues dominating, particularly in nursing workflows. These were most evident in tasks involving remote vital sign capture, alongside problems related to error prevention and recovery, with several high severity issues occurring in safety-critical nursing tasks. Although evaluators generally found the virtual care technologies easy to use, a total of 45 usability problems were identified, including 21 learnability problems (based on the CW evaluation questions) and 24 other problems related to effectiveness (n=2), efficiency (n=2), error (n=8), memorability (n=6) and satisfaction (n=6) as shown in Figure 1. The most critical barriers to system use were high-severity learnability problems in nursing tasks and failures in error prevention and recovery affecting safety-critical workflows.

Figure 1.

Severity distribution across 45 usability problems mapped to Nielsen’s usability categories.

While usability problems were identified in both GP and nursing workflows, their distribution and severity differed markedly by role, with nurses experiencing a higher concentration of learnability and high-severity issues in routine, safety-critical tasks.

Category of problems based on evaluation problems

Learnability problems were most frequently associated with users not being adequately prompted about required actions or the availability of necessary controls, with many of these problems rated as minor to major in severity. Of the four questions, ‘notifying of actions at hand’ and ‘existence of necessary controls’ (see Table 1) had the highest proportion of learnability problems (43% and 33% respectively). For example, a problem identified was that: users were often unable to recognise that clicking ‘disconnected’ was required to connect the remote monitoring device via Bluetooth. Also, users were often unable to recognise they were required to click on the serial number of the device found by Bluetooth. The list of learnability problems based on the evaluation questions is shown in Appendix D and the severity of the problems is shown in Table 4.

Table 4.

Counts and percentages of learnability problems based on evaluation questions and mean problem severity.

Note. Percentages represent number of problems as a proportion of total number of identified learnability problems (n=21).

Category of learnability problems based on tasks

Learnability problems were predominantly observed in the nursing workflow, particularly in tasks involving resident vital sign capture and joining residents to telehealth calls, whereas learnability issues in the GP workflow were limited and task-specific. Of the four tasks for the GP workflow, ‘viewing current and previous vital signs for a resident’ was the only task where learnability problems were found (n=3, mean severity=2.3). Of the six tasks for the nursing workflow, ‘capturing resident’s vitals (temperature, heart rate and oxygenation)’ was the task with the highest number of learnability problems (47.6%) but the majority of these problems were major or minor in severity, with a mean severity of 2.5. One of the catastrophic problems related to the ‘joining a resident to a call’ task i.e. back button ended the telehealth call from the nurse’s end, but not the GP’s end. The list of learnability problems based on tasks is shown in Appendix D and the severity of the problems is shown in Table 5.

Table 5.

Counts and percentages learnability problems by tasks and mean problem severity.

Note. The number of sub-tasks per task does not correspond directly to the number of problems identified; multiple sub-tasks may yield no problems, while individual sub-tasks may yield more than one problem.

Other usability problems

Most of the other usability problems were categorised as ‘error’ problems (n=8). Half of these problems were rated as having major severity (all within the GP workflow) and half as catastrophic severity (all within the nursing workflow). Error-related usability problems differed by role: GP-related errors were primarily associated with documentation and navigation inefficiencies, whereas nursing-related errors more often affected call management, device connectivity, and record handling, several of which were rated as catastrophic. Table 6 summarises non-learnability usability problems and their corresponding actionable design recommendations identified by evaluators. Design recommendations were formulated by mapping identified usability problems to established usability principles, primarily Nielsen’s heuristics, to ensure recommendations were specific and actionable.

Table 6.

Non-learnability usability problems and actionable design recommendations.

Nielsen’s usability category	Usability problem identified by evaluators	Workflow	Severity rating (1-4)	Evaluators’ recommendation
Effectiveness	Lots of scrolling down on the overview page to see patient record	GP	2	Increase visibility and affordance of the scroll bar to ensure users recognise additional information below the fold. Consider whether it is necessary to have the resident’s screen if not in a call.
	In scenarios where the nurse is taking and reading out vital signs to the GP, the GP cannot add patient’s vital signs data during the call.	GP	3	Provide ‘take vitals now’ button/functionality for GPs during a call
Efficiency	Patient records need to be closed. Link is unique to resident, so the GP cannot jump to another resident from one call	GP	2	Once a GP has joined a consult using the link for a resident, it would be more efficient if they can see the list of their patients in that home, and start a consult from there rather than needing a new link to be generated.
	There is nothing to indicate that the GP has joined the call and is in a waiting room	Nurse	2	Provide information to indicate that the GP or any other invited party is in the waiting room
Error	Notes are not saved automatically if you move across tabs	GP	3	Warn users that they need to save before moving across tabs. GP notes should be auto-saved.
	Save note button needs scrolling otherwise user will not see it	GP	3	GP notes should be auto-saved.
	Scroll bar is hard to use when writing notes	GP	3	Improve scrolling and navigation behaviour in the notes section to maintain visibility of action controls (e.g., save)
	Resident’s face not visible on all data entry pages. It is very easy to click into another resident’s file when looking at clinical data and wrong notes may be inadvertently entered.	GP	3	Resident’s face should be visible on all pages - where vital signs or notes can be entered
	One iPad needs to be muted so there will be no feedback	Nurse	4	Implement system-level audio state management to automatically detect and suppress feedback loops, reducing reliance on manual user intervention, and provide instructions to help users recover from errors
	Error message received stating “remote participant issue”, user restarted the call again but it did not work with GP starting the call, the RN must start the call	Nurse	4	Implement robust call-state management and recovery mechanisms, including explicit system feedback on call status (e.g., who must initiate the call) and guided recovery actions following call-connection failures.
	Cannot edit GP name and details once entered. This can inadvertently lead to creating a duplicate doctor and subsequently selecting the wrong doctor’s account	Nurse	4	Allow GP details to be updated by the RN
	There should be no opportunity to delete a consult that has been created - for medico-legal reasons.	Nurse	4	Remove delete button for each created consult
Memorability	Headings in clinical overview are inconsistent with the tabs e.g. Upcoming appointments listed as part of clinical overview is not a tab	GP	2	Ensure tabs are consistent with what’s in the clinical overview. Add upcoming appointments tab.
	Placement of + icon lacks consistency	Nurse	1	Place + icon consistently in the same location on the interface
	Purpose of heart rate symbol on the dashboard in the waiting room is unclear	Nurse	1	Use a label to clarify purpose upon mouse hover
	It is not clear how to refresh the page. This is needed if a nurse wants to see notes written by the GP. The only way to do this right now is for users to a) leave the notes tab and come back to see notes written by the GP or b) use the back button which inadvertently ends the call	Nurse	3	Add refresh icon for others to see users notes
	Headings are too complex in the nurse’s template	Nurse	2	Allow free text and remove structured template
	The view on desktop is very different from that of the iPad	Nurse	3	​
Satisfaction	Unclear why there is another “join the call button”. Unnecessary additional step.	GP	2	Resident and nurse may need waiting room, GP does not need additional step. Terms used are misleading. GP email should say “click here to view the patient’s records and join the call”, or ‘go to telehealth site’ from email, and ' agree'.
	The description of some fields for collection of vital signs are unclear. E.g. ear, cough, throat and lungs. Are those vital signs?	Nurse	1	Provide clarity on vital signs data collection fields
	User can click on 2 refresh buttons even though its the same device and the vital signs are taken at the same time. Clicking on one of the refresh buttons works for both vital signs at the same time.	Nurse	1	Remove extra refresh button. Move one of the refresh buttons up - next to the heart rate label
	The purpose of the icons at the top or left side of the screen is unclear. It is also unknown whether they can be used without being in a call	Nurse	1	Remove unused icons in the “waiting room”. Also, label upon mouse hover.
	Resident’s view of the interface is too cluttered	Nurse	2	Remove unnecessary elements on the screen for the resident’s view. Residents need only the end call, add family, mute/unmute, video on/off functions.
	Chat between GP and RN is visible to all team members, although patient can’t see what has been typed, and when they click on chat they cannot see the conversation.	Nurse	4	Provide resident with access to full chat functionality

Table 6 shows that non-learnability usability issues were predominantly related to error prevention and recovery, with 8 error-related problems identified, many of which were rated as major or catastrophic, particularly within the nursing workflow. Several issues posed clear patient safety and medico-legal risks, including the inability to edit GP details, the ability to delete safeguards for consult records, and unresolved call or audio failures. In contrast, problems related to efficiency, memorability and satisfaction were mostly rated as minor to moderate, reflecting usability challenges, inconsistency, or confusion rather than direct safety threats. Across findings, interface issues primarily affected usability and user experience, whereas error-prone system design was associated with higher severity risks, particularly for nurses facilitating care in RACH settings.

Discussion

We sought to assess the usability of a virtual care platform used to connect residents in residential aged care homes with primary care providers. Overall, this innovative virtual care platform was considered to be easy to use by evaluators, often aligning with user expectations and supporting decision-making in the RACH setting, broadly consistent with established interface design guidelines. Upon evaluation, our study identified 45 usability problems, almost half related to the learnability of the virtual care platform. Of the learnability problems found, the most frequent problem related to the system not making it clear to users that they needed to perform an action to get the task done. The majority of the learnability problems were identified in the nursing workflow, particularly when nurses performed the task of capturing resident’s vitals (temperature, heart rate and oxygenation). Our study also revealed a high number of problems where errors were inadequately prevented by design, with half identified as catastrophic within the nursing workflow. For the GP workflow, all problems relating to inadequate prevention of errors were considered to be of major severity and the only learnability problem occurred when viewing current and previous vital signs for a resident on the virtual care platform. Taken together, these findings suggest that the usability (and learnability) of the virtual care platform was limited.

In our study, we found that it was difficult to identify the relationship between what to do (i.e. user action), system controls (e.g. icons, buttons) and the task at hand reflecting poor visibility of system status and weak mapping between actions and outcomes. This result is in line with our previous study that assessed work system barriers and facilitators for detecting and managing deterioration when virtual care technologies are used in a virtual hospital. ²⁹ In that study, clinicians reported that it was difficult to differentiate between completed and pending tasks. ²⁹ Existing research indicates that a key reason why users are unable to identify the relationship between what to do (i.e. user action), system controls (e.g. icons, buttons) and the task at hand is poor interface design such as improper placement of icons and buttons, inconsistent fonts, improper placement of control.^22,30,31 Also, evidence suggests that a lack of instructions on how to use the system or the lack of cues to guide users can increase cognitive burden and result in dissatisfaction. ³² Within a RACH setting, where ongoing staff training is challenging, this finding indicates the importance of improving design using meaningful symbols, cues and/or clear instructions when using virtual care technologies.

Our study identified that almost half of the learnability problems identified were associated with the task of capturing resident’s vital signs using remote monitoring technologies in connection with the virtual care platform. This task included 12 sub-tasks, which represented the highest proportion of sub-tasks when compared to all other tasks in both the nursing and GP workflows. Existing studies have reported that tasks that require more sub-tasks or actions result in more user problems as they increase cognitive load and working-memory demands during task execution, particularly under time pressure. ²⁵ . For example, a study that used cognitive walkthrough analysis to assess five scenarios in a nursing information system found that the scenario with the highest number of sub-tasks or actions (i.e. manual entry of 29 data elements) had the most severe usability problems than other scenarios. ²² Although the technologies connected to the virtual care platform via Bluetooth in this study transmitted data automatically, supporting complex tasks and data, users were required to perform many actions on the platform to facilitate data capture. In the context of a RACH setting, with high workload, our findings underscore the need to minimise the number of steps needed to conduct what would be a frequently performed task, that of capturing vital signs with remote monitoring technologies. While some CW studies report metrics such as problems per task or sub-task, direct numerical comparison across studies is limited by differences in task scope, system complexity, and severity anchoring. Accordingly, we position our findings analytically rather than numerically.

Furthermore, our study uncovered a high number of errors that could have been prevented, particularly within the nursing workflow. Several of these errors were rated as catastrophic and related to core system functions, including call management and audio handling. These failures have the potential to interrupt consultations, delay escalation of care, and compromise continuity during time-critical clinical interactions, particularly because the system provides limited support for error prevention and recovery. For example, we found that a nurse is unable to edit the GP’s name and contact details once entered into the virtual care platform. This could inadvertently lead to the creation of a duplicate doctor, selecting the wrong doctor’s account, sharing patient data with the wrong doctor and subsequently resulting in logistical delays and delayed access to care for the patient. As a key goal of virtual care is to enable timely and efficient access to primary care providers for residents,^9,10 error-prone design can prevent this benefit from being realised. This finding aligns with existing literature that used heuristic evaluation to identify the design features that have the highest impact on the quality of care and to evaluate user interfaces generally used for virtual visits. ³³ In that study, they found that ‘error prevention and recovery’ had the highest impact on the safety and efficiency domains of the six domains of quality of care as defined by the Institute of Medicine. ³³ Therefore, careful design that prevents errors from occurring and helps to easily recover from errors is imperative for virtual care technologies to be usable and to prevent unintended consequences.

From the resident’s perspective, the busy resident-facing interface may be particularly challenging for residents with cognitive impairment, as unnecessary on-screen elements can increase cognitive load, reduce attentional focus, and limit residents’ ability to meaningfully participate in virtual consultations.

At the level of implementation and scale, these findings indicate that virtual care systems deployed in RACHs must prioritise inherent learnability, streamlined task design, and robust error prevention. Usability weaknesses in routine nursing workflows are likely to be amplified when systems are scaled across settings characterised by high workload, staff turnover, and variable digital literacy.

Taken together, these findings highlight how breakdowns in cognitive load management, visibility of system status, and support for error recovery can undermine the safe and effective use of virtual care technologies in high-pressure aged care settings. By demonstrating how task-level usability evaluation can reveal safety-critical learnability and error-related issues in routine clinical workflows, this study provides a foundation for future digital health research to prioritise rigorous, human centred usability methods that better reflect real world care delivery.

Strengths and limitations

To the best of our knowledge, this is the first time the CW method was used to assess the usability of a virtual care platform implemented in a residential aged care setting. Also, this study was conducted with five evaluators, which is the maximum number of evaluators recommended for a CW evaluation. As evaluators were members of the research team, some risk of evaluator bias cannot be excluded. However, the use of novice users, a structured cognitive walkthrough protocol, paired evaluation, and role-based clinician involvement helped mitigate this risk, although future evaluations could benefit from the inclusion of independent external evaluators. In addition, as CW focuses on system learnability, the severity of problems was assessed by real users of the virtual care platform. Also, this study focused on system learnability rather than learning over time and therefore did not assess changes in performance with repeated use or training. While evaluators executed tasks directly using the platform, severity ratings were resolved through structured, role-based consensus discussions supported by screenshots, which may influence the interpretation of severity judgments.

This evaluation focused on a single virtual care platform; however, many of the usability problems identified reflect broader human factors challenges commonly reported in digital health systems, including high cognitive load, limited visibility of system status, and inadequate support for error prevention and recovery. These issues arise from fundamental interactions between users, tasks, and system design, suggesting relevance to other virtual care platforms used in RACH settings. Differences in platform functionality, implementation models, and organisational workflows may influence how these problems occur in practice, and therefore the findings are best interpreted as transferable to similar settings rather than universally generalisable.

Although the study was conducted in a simulation laboratory rather than a real-world residential aged care home, an older adult aged 65+ participated as the “older adult advisor”, allowing us to incorporate age-related sensory, cognitive and physical considerations into the usability evaluation. However, we acknowledge that simulated environments cannot fully replicate the complexity, interruptions, and contextual variability of routine aged care practice. As such, while the inclusion of an older adult mitigates some concerns around external validity, the findings should be interpreted with caution. Although severity ratings were informed by expert judgment, they were developed through a role-based consensus process (i.e. clinician and two human factors experts), which reduces individual rater bias. Nonetheless, as only one clinician per role participated, future studies could strengthen reliability by involving multiple clinicians within each role before consensus.

Also, we acknowledge that the CW method follows a logical and stepwise process, and was conducted in a controlled simulated setting. Given the absence of real-world interruptions, noise, and time pressure, the severity ratings reported here should be interpreted as conservative estimates of risk under routine RACH conditions. Consequently, our study may not have fully captured exploratory behaviour, non-linear navigation, and interruptions that users might exhibit in a real RACH setting. Additionally, the impact of system integration with existing clinical information systems on documentation burden and workflow efficiency could not be assessed and should be examined in future real-world implementations.

Conclusion

This study provides one of the first task-level usability evaluations of a virtual care platform implemented in residential aged care using a cognitive walkthrough approach. By examining routine GP and nursing workflows, we identified that usability problems were dominated by learnability issues, particularly in safety-critical nursing tasks such as remote vital sign capture, alongside failures in error prevention and recovery. These findings demonstrate that even systems perceived as generally easy to use can harbour design weaknesses that disrupt clinical workflows and introduce risks to safe care.

Importantly, the results show that usability in virtual care is not merely a matter of user experience, but a critical determinant of effective implementation and scale up in residential aged care environments characterised by high workload, staff turnover, and limited opportunities for ongoing training. The disproportionate burden of usability problems on nursing workflows highlights the need for virtual care platforms to prioritise inherent learnability, streamlined task design, and robust error prevention by design. At a theoretical level, the findings demonstrate how learnability breakdowns emerge from the interaction between task complexity, distributed work across people and devices, and constrained care environments. By illustrating how cognitive walkthrough can uncover safety critical usability issues that are often missed by satisfaction-based evaluations, this study offers clear direction for future digital health research and design to better align virtual care technologies with the realities of aged care practice.

Consent to participate

There are no human participants in this article and informed consent is not required.

ORCID iD

Adeola Bamgboje-Ayodele https://orcid.org/0000-0002-5629-1236

Data Availability Statement

The data that support the findings of this study are available upon reasonable request from the corresponding author.*