Integrated health system to assess and manage frailty in community dwelling: Co-design and usability evaluation

Abstract

Objective

We aimed to co-create and evaluate an integrated system to follow-up frailty in a community dwelling environment and provide a multi-modal tailored intervention. Frailty and dependency among the older population are a major challenge to the sustainability of healthcare systems. Special attention must be paid to the needs and particularities of frail older persons as a vulnerable group.

Methods

To ensure the solution fits all the stakeholders’ needs, we performed several participatory design activities with them, such as pluralistic usability walkthroughs, design workshops, usability tests and a pre-pilot. The participants in the activities were older people; their informal carers; and specialized and community care professionals. In total, 48 stakeholders participated.

Results

We created and evaluated an integrated system consisting of four mobile applications and a cloud server, which has been evaluated through a 6-months clinical trial, where secondary endpoints were both usability and user experience evaluation. In total, 10 older adults and 12 healthcare professionals participated in the intervention group using the technological system. Both patients and professionals have positively evaluated their applications.

Conclusion

Both older adults and healthcare professionals have considered the resulted system easy to use and learn, consistent and secure. In general terms, they also would like to keep using it in the future.

Introduction

Designing usable systems for vulnerable population is not an easy task, and it is extremely important when these systems are aimed at tackling health problems, since lack of treatment adherence can hinder the effectiveness of the proposed interventions. Even if the patient/user is aware of the long-term expected benefits of a change in lifestyle and habits, a decline in motivation over time can lead to treatment abandonment or inadequate compliance with the prescribed interventions. Then, effective implementation of home care systems requires a careful consideration of the patient's needs, capabilities and motivation, the promotion of empowerment, and the adaptation of the interventions to the current stage in the behavior change process, as critical countermeasures to fight against abandonment and lack of compliance. Co-creation ¹ and participatory design ² are needed while new technological innovations are proposed in general, but particularly when involving older users, ³ in order to guarantee high usability and user experience. ⁴

This research work focuses on a specific target vulnerable group: pre-frail and frail older persons. In an ageing society like ours, which is challenged in terms of how to provide the health care services required by older people while maintaining the sustainability of the system, improving adoption is crucial to maximize the odds that the designed interventions have the expected impact.

Clinicians may be reluctant to adopt eHealth technology in general, and home care systems in particular, as they are usually worried about liability issues and the burden of their work. ⁵ This is especially relevant among community care professionals of public healthcare systems, who usually have only a few minutes to attend a patient, and they perceive that using a new technology directly translates into an increased workload. In this case, change management is essential, being crucial to make the system adapt to their needs, and not the opposite. Care professionals are willing to adopt new technologies, including home care systems, as long as they enrich face-to-face visits instead of replacing them. ⁶

The World Health Organization (WHO) defines Healthy Ageing as the process of developing and maintaining the functional ability that enables well-being in older age. ⁷ This definition emphasizes that healthy ageing is more than just the absence of disease. For older people, the maintenance of functional ability is of upmost importance. Functional ability is highly related to quality of life and an excellent predictor of health outcomes. ⁷ This means that, opposed to the current disease-centric approach, a holistic and patient-centered approach is required. ⁸ Finally, it is now well recognized that the main risk factor for both incident disability and death, is function, rather than the different diseases suffered by the older population. ⁹ Latest efforts in this domain have converged in the emerging of the concept of frailty, defined as a state of increased vulnerability to adverse outcome due to a reduction in the ability to respond to stressors. ¹⁰ The most important aspect is that frailty can be prevented and reverted. ¹¹

According to the currently available scientific knowledge, effective interventions on frailty are strongly related to daily routines and, therefore, require some extent of self-management at home and other preferred locations. ¹² Home-care systems for self-management of chronic diseases have been a hot topic for decades now. ¹³ However, they have not had a significant impact on clinical practice. On the one hand, patient engagement in self-management is a prerequisite for success ¹⁴ ; however, it is difficult for general practitioners to get their patients motivated to comply with a new lifestyle. ⁶ Patients are asked to change their habits, but they do not perceive any immediate or short-term benefit. A similar situation is to be expected for frailty screening.

The content of this manuscript is framed mainly in the context of the POSITIVE project (funded by EIT-Health ¹⁵ ), which aimed at creating a technological ecosystem co-created involving all relevant stakeholders, including primary and specialized care professionals, older adults, informal caregivers, User Experience (UX) designers and researchers on technology applied to ageing.

The main purpose of POSITIVE was to enable a new organizational model that allows providing care to older persons in a coordinated way, involving all relevant actors in the process but placing the older person in the center, based on the integrated care model proposed by Sanchez et al., ¹⁶ and incorporating the active participation of community care professionals.

The implementation of this model allows early detection of functional decline and thus timely interventions aimed at preventing disability and dependency. This improves the health condition of the older persons and reduces the use of healthcare resources by, for instance, limiting the number of face-to-face visits to the doctor or nurse and thus increasing efficiency. In that sense, the co-creation activities have been essential not only to define the functional requirements of the system and to adapt its design to the users’ needs, but also to analyze and better specify how to implement this new organizational model.

Given the context of the problem with its multiple perspectives, the main objective of this research work is to provide further evidence on the impact of co-creation strategies when designing for vulnerable population (frail older adults in this case). We hypothesize that involving relevant actors, in our case, frail older users and their different careers in the creation process, will have a positive impact on the latter evaluation of the resulted system. To that aim, the resulting system after the co-creation process has been evaluated from a usability and user experience perspective.

The paper is structured as follows. First, the applied co-creation methodology is described as well as the methods used to carry out the evaluation process. Second, the resulting co-created system and subsystems are outlined. Afterwards, those results related to the usability and UX evaluation of the system are presented and discussed. Finally, some conclusions are drawn, and future work is sketched out.

Methods

Co-creation and human centered design

We followed a co-creation approach to design, develop and evaluate a home care system for frailty monitoring.^17,18 These co-creation activities resulted in the design of an integrated system that includes technological solutions for the different actors: older persons, healthcare professionals, and informal caregivers.

To maximize both the usability and the user experience, we have followed the so-called Human-Centered Design approach ¹⁹ to guarantee acceptability, adoption, and adherence. This approach allowed us to incrementally specify, refine and improve our design, always ensuring that it fits the users’ needs and expectations.

To reach this goal, we have performed several co-creation activities with the different stakeholders of our system: older users ( + 70), community care professionals (physicians and nurses), specialized care professionals (geriatricians, physiotherapists, and nutritionists), technological researchers, and UX researchers. Co-creation is a new approach where stakeholders are involved in the design process, allowing them to have an active role in the creation of the new product. ²⁰ This approach is very beneficial both for designers and developers as it allows them to capture the real requirements and needs of the potential users, who are involved during the whole creation process, and then can intervene and validate all decisions taken along it. On the other hand, participatory design activities with real potential users ensure that the final product will fulfil their needs, and it also improves their level of satisfaction with the resulting product, as they feel they have been heard and have partly created it. ²¹ In addition, participatory design is essential to make sure that a software design is properly designed and provides high levels of usability and a good UX. ²² This is vitally important when the adoption and acceptance of new technologies by the potential users is low. In these cases, perfectly adapting the design of a software to the mental model of the potential users is crucial to create a usable, and then, successful software system.

We have used three participatory design methods: design workshops, pluralistic usability walkthroughs and usability tests. At the beginning of the project, two design workshops were carried out to specify all the relevant aspects related to the users and the system's context of use and how to translate them into a system's design. For this purpose, two focus groups were performed with all the stakeholders of the system (including users of any role, but also the software system and interaction designers and developers). A focus group is a qualitative research technique ²³ that is specified as a meeting of potential users where participants give their opinion about one or more topics. The most interesting output of this meeting is the debate that emerges between the participants, as this forces them to defend, argue and justify their ideas, and then to conceptualize and externalize their real mental model. This verbalization helps understanding the individual user requirements, but also the existence of patterns shared by users of different profiles. Detecting these patterns is essential to define and design the communications between users and the interaction with the system.

Once the main requirements of the system were defined, a first prototype was created. Using it as reference, a pluralistic usability walkthrough session was carried out. Pluralistic usability walkthrough ²⁴ is a usability inspection method where the stakeholders revise a software design. This helps specifying more in detail the tasks that must be developed as different perspectives from different roles are brought together, and usability problems affecting the learnability, effectiveness, effectivity, satisfaction, and memorability of the system are identified.

Usability and UX evaluation

Usability testing has been used to assess the usability and UX of the system and to identify problems and issues to be corrected. ²⁵ This technique is essential in a user-centered design approach, as the system is designed and developed incrementally and iteratively based on the results obtained in the usability tests. In our case, we have evaluated the usability three times: first, the prototype has been modified and improved several times to address the usability problems found during the pluralistic usability walkthrough sessions and the usability tests; then, we have assessed the usability of the real system with older adults who participated in a pre-pilot; and finally, during the clinical trial, we have also evaluated the usability and UX of the patient's and professional's applications.

Usability, defined as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use, ¹⁹ has been assessed using the System Usability Scale (SUS).^26,27 SUS is a short 10-item Likert questionnaire that provides a measure of people's subjective perceptions of the usability of a system (see supplementary material, section A). Concretely, SUS focuses on learnability and usability, which are indeed correlated. ²⁸ These 10 items can be evaluated from ‘1-Strongly disagree’ to ‘5-Strongly agree’. It is important to point out that SUS odd questions are redacted in a positive way, whereas even questions are negative. Thus, to obtain a total score, answers are normalized from 0 to 4, and then the sum is multiplied by 2.5 to obtain a score from 0 to 100. Although SUS is a simple tool, a study carried out by Tullis and Stetson, ²⁹ who compared the effectiveness and accuracy of five questionnaires for assessing usability across different sample sizes, reached the conclusion that it is a reliable scale. A SUS above 71.4 is considered good, and above 85.5 is considered as excellent. ³⁰

UX has been assessed with the short version of the User Experience Questionnaire (UEQ-S), ³¹ consisting of eight pairs of antonym adjectives (see supplementary material, section B). Each pair of adjectives is evaluated using a semantic scale of seven levels, ranging from −3 to 3. In its short version, UEQ provides three scores; two are related to dimensions “pragmatic” and “hedonic”, whereas the third provides an overall score. The score for each category is calculated by averaging the levels selected in the items associated with each dimension. UEQ-S was included as an evaluation tool to complement the domains addressed by the SUS.

SUS and UEQ-S for professionals were assessed at the initial, intermediate, and final visits (after 2 weeks, 3 months and 6 months of use, respectively). With regards to the older persons, SUS was collected at the initial, intermediate, and final visits, as with professionals. However, UEQ-S was only collected at the final visit to reduce their overload.

Descriptive statistics are used within this manuscript to analyze obtained results in section 4. This methodology is a useful tool for summarizing and describing data such as those collected in this study.

Resulting system

This section presents the results achieved after the co-creation process, namely the POSITIVE technological ecosystem.

Between February and December 2019, 18 co-creation sessions were conducted: 6 design workshops, 3 pluralistic usability walkthrough, 8 usability tests, and one pre-pilot. We incorporated relevant stakeholders: that is, community and specialized health care, patients’ dwelling, and sociocultural. In total, 11 older users (aged 70–96), 7 community care professionals (physicians and nurses), 8 specialized care professionals (geriatricians, physiotherapists, and nutritionists), 6 technological researchers, and 4 UX researchers participated. Later, in June 2020, seven older adults (aged 64–74) participated in a 4-weeks pre-pilot, facilitated by five UX researchers, where the technical viability, usability, UX and acceptability were evaluated. Most of these participants, except the older adults, were involved in more than one co-creation session. The concrete details of each session, including the date, type of activity performed, participants involved, and purpose are detailed in the supplementary material, section C. All stakeholders who participated in the co-creation activities received, read, and signed a written informed consent before starting the research.

The resulting system relies on the participation and interaction of three loops of care that coexist and complement each other, as illustrated in Figure 1.

Figure 1.

New organizational model proposed in POSITIVE.

Figure 2 illustrates the conceptual architecture of the POSITIVE integrated home care ecosystem, conditioned by the results of the co-creation process. The system consists of five main components: Patient subsystem, Caregiver subsystem, Health Care Professionals subsystem (consisting of the Community Care and Specialized Care subsystems), and Cloud subsystem.

Figure 2.

Description of the integrated home care system.

Patient subsystem

This subsystem includes a frailty Home Monitoring Kit able to detect changes in the three most informative variables, namely, gait speed, ³² power in the lower limbs,^33,34 and involuntary weight loss. Thus, the kit consists of a gait speed sensor, a sensor to indirectly measure power in lower-limbs throughout the chair stand test, ³⁵ and a commercial smart weight scale. Furthermore, a mobile app is used as a data concentrator, as a guiding element for the older person to comply with their prescription, and as data input point that enables older users using the sensors and completing a set of questionnaires to enrich the information handled by the clinical professionals. The mobile application is installed in a 10.1″ tablet to facilitate the reading and interaction, using both big font sizes and big components. The questionnaires presented to older adults to assess their health and functional status are adapted versions of the Frailty Phenotype criteria, ¹⁰ Mini Nutritional Assessment (MNA), ³⁶ Barthel Index, ³⁷ Frail Scale, ³⁸ and the Functional Activities Questionnaire (FAQ). ³⁹ The worsening of any of these remotely collected variables (gait speed, lower-limb strength, weight, and health and functional status) could reflect a deterioration in the patient's health status in terms of frailty.

The POSITIVE technological ecosystem aims to encourage older adults to take control of their functional status, turning them into active and co-responsible actors who can monitor their state of health when desired; to perform the physical activity plan prescribed by the health care professionals; to commit to their personalized nutritional advice; and to ease communication with their community care professionals through an asynchronous communication channel. On top of this, the challenge is that older adults perform these tasks in a simple, intuitive, and comfortable manner.

Initial design

The patient's application created in a previous project, FACET, ⁴⁰ was used as the starting point to design the POSITIVE patient app. This reference app only allowed to perform the home sensor-based tests, namely gait speed, lower-limb strength and weight, and some standard questionnaires like Linda Fried or Barthel.

Based on this and the requirements emerged during the first co-creation activities, an initial functional prototype was created in Figma for the following co-creation sessions. Figure 3 illustrates some of the screens of this initial prototype. The full design of the initial prototype is presented in the supplementary material, section F.

Figure 3.

Some screens of the initial prototype of the patient's app.

In this design, several challenges had to be addressed. First, the great number of functionalities had to be accommodated in the user interface, but making sure the screen was not too cluttered and that the components were big enough to facilitate their visualization and interaction. As a golden rule, clean and simple interfaces have been conceived to avoid overloading the users. Green has been chosen as the primary color since it transmits well-being, health, and healthy life. In addition, interaction has been simplified by using only large enough buttons and text to be read and used by users who potentially have fine-grain manipulation and vision problems.

For that reason, we decided to divide the main menu into two screens, keeping in one of them the functionalities related to monitoring and intervention (home tests, questionnaires and Vivifrail exercises), and the accessory ones in the second menu screen (nutrition and exercise recommendations, visualization of the evolution in the tests, and messages exchanged with the primary care professional).

As a general principle, designers must consider some heuristics to ensure the usability of the systems, like always informing about the status of the system, ensuring that the user always maintains control, allowing him/her undoing any kind of action resulting in unexpected effects, or providing alternative ways of performing an action. However, based on our previous experience, in the very particular case of older users, the compliance with these heuristics can be sometimes counterproductive, since they may feel overwhelmed and lost. This occurs especially with the navigation, as it is one of the most difficult and confusing interactions for older adults. In general terms, providing freedom to the user to navigating through alternative paths is a good practice to allow each user selecting the option that better matches his/her mental model. However, we have detected that this freedom, in fact, supposes a huge overload for older adults, who do not understand the alternative, which ends up generating them confusion and misunderstanding. To avoid this, a very simple and straightforward navigation was designed, consisting of only allowing the user to navigate to the following step in the process. To ensure the consistency and memorability of the interaction, this was always done by clicking on a big button that was always located in the same position on all the screens that were part of a task flow. Then, navigation is simplified at the cost of losing freedom, which is one of the heuristics from Nielsen. ⁴¹

With respect to the tests performed at home, based on our experience evaluating the FACET application for patients, a directed process was performed to increase the learnability of the app, which is a must with older adults. Instructions were provided through synchronized text and audio, and, in addition, an explanatory video was included to provide the instructions in a more illustrative manner facilitating the reproduction of the actions to be done. ⁴² Finally, in order to ensure the user is always informed of the system status, the end of the tests was always illustrated through a final screen using the primary color of the app as background color, and providing an icon of a cup, which was considered a very familiar and recognizable visual metaphor for older adults. The specific result of the test was provided on this screen through a short and simple text.

The design of the questionnaires was a huge challenge, as during the co-creation activities, we have identified that older persons react negatively to textual instructions that are too long. Then, we were obliged to rephrase all questionnaires included in the app (Linda Fried, Frail Scale, MNA, Barthel, and FAQ) (and also all the instructions provided in the application) to make them simpler and more readable. The readability was applied both to the questions and to the answers. As a result, most questions are stated in terms of a comparison, using always a very simple and plain language, and then two or three possible answers were provided (worse, equal, better). Additionally, a structural change was implemented with respect to the FACET application. In the latter, questions were displayed as part of a questionnaire. However, the name of these questionnaires was unfamiliar to them the users, which made them feel unsecure and doubtful. For that reason, the belonging to one or another questionnaire became transparent to the older adults, who only were asked a set of questions, that were later categorized internally by the system to assess the different questionnaires.

In the case of the Vivifrail exercises to be performed by the users, one screen was designed for each exercise, including a drawing of the exercise and an explanatory video. In this case, however, a big difference was implemented with respect to the rest of the app: users have the possibility to navigate both to following and previous exercise. This was done as users were not obliged to perform the exercises in a given order, and then we had to provide the possibility to fully navigate through them. The exercise screen also included a button to indicate the exercise has been made.

Finally, for the complementary functionalities, simple text-based recommendations were provided related to nutrition and physical exercise and communication with the primary care professional was implemented as a regular mobile chat. With respect to the evolution of the results of the home tests and Vivifrail exercises, during the co-creation activities, we discovered that the visualization of these results cannot be done through graphs or diagrams, since these elements are usually unknown to the older adults. Instead, it was detected that it was better to simply list the last results of each test indicating only the date of the test and the value obtained.

Final design

After performing all the co-creation activities, the design was iteratively and continuously refined until the final version was created, used to perform a final evaluation in a clinical pilot.

The first main change was implemented in the main menu. It was detected that the division of the menu into two screens was absolutely confusing for the older adults, making it almost impossible for them to understand how to access the complementary functionalities. The solution was to keep the menu in a single screen, which obliged us to reorganized and group the functionalities. In parallel, older adults also complained about the terminology used in the menu items, as it was not natural for them. Our proposal, based on this, was to organize the functionalities, and by extension the menu, based on when they occur in the regular interaction with the care professionals: on the one hand, users must answer a set of questions and some tests are administered to them to diagnose the problem; then, based on this diagnosis, a treatment is provided. Separately, the communication functionality is considered a first-level functionality, as it may occur at any moment.

Following this scheme, the menu item “My progress” includes the diagnostic activities, namely questionnaires, home sensor-based tests, and the history of results of both. On the other hand, “My treatment” encompasses the performance of the physical exercises of the Vivifrail program, together with the recommendations, divided into nutrition and general recommendations. This is another result stemming from the co-creation activities, where it has been identified that nutrition recommendations must be individualized since they are based on their nutritional profile, defined by the healthcare professional.

The second structural change is related to the navigation. Even if the navigation has to be as simple and straightforward as possible, during the co-creation activities we have identified that, in many cases, users remained captive on screens that they have accessed unintentionally or by mistake. This is especially relevant with older adults, who are more prone to making mistakes, due either to dexterity problems (basic tremors, loss of sensitivity, difficulty handling fine grain, blurred vision…) or to lack of experience with technology, which makes them more prone to exploration. Additionally, we have been told that the navigation was not internally consistent, as in all cases it was done through a button at the bottom of the screen, except in the Vivifrail exercises, where there were two bottoms, which also were not at the bottom of the screen. To solve both problems, we designed a unique navigation scheme, regardless the task and the screen, consisting of three buttons, always located at the bottom of the screen: Previous (step or screen), Home (emergency exit that leads to the main menu), and Next (step or screen).

Improvements were also performed in the home tests activities. On the one hand, the drawing illustrating the test was substituted a picture of a real user. This simple change was important, as older adults may have problems (more than other users) to recognize visual metaphors that are too abstract or non-familiar, and then, using a real picture, ensure recognition better than recalling the test by the name (which is another of the Nielsen heuristics). ⁴³ With respect to the instructions, they were still provided through text, voice, and video. However, the visualization of the video was also adapted to maintain the navigation frame. This was essential as users claimed that interacting with video playback platforms was too complicated, and this generally resulted in the patients not being able to abandon that screen.

In the case of the questionnaires, co-creation activities helped us realize that the answers were too simplified (Better, Same, Worse), obliging the users to remember exactly what they were been asked. As a solution, we proposed to write clearer and more specific questions, but maintaining the three levels. Thus, as an example, the question “Do you feel more or less appetite than in the last few days?”, could be answered with the answers “More appetite”, “Same appetite”, or “Less appetite”.

The design of the screens implemented to perform physical exercises (Vivifrail) also suffered several changes based on co-creation activities feedback. First, the drawing was substituted by a picture of a real user performing the exercise. To maintaining the consistency with the home tests, we also included textual instructions, who specifically were adapted from the original instructions to make them short and simple, using a plain language. Also, instead of having to click over the drawing to reproduce the video, a button has been added, as it happens in the home tests. This was made to maintain the internal consistency, but basically because it was not intuitive for older adults to, first, recognize that over the drawing there was a symbol of play, and second, to understand that this symbol was interactive and had to be clicked to watch the video. Also, we detected that in the initial prototype, the screens of the physical exercises were too cluttered, which was overwhelming and confusing for the users. For that reason, we created a new screen in which the user had to indicate if he/she has performed the exercise. This screen follows the same scheme of the questionnaires’ screens.

As mentioned before, the nutrition recommendations have gained importance, since now not only general recommendations are provided through text, but also users can watch specific videos for their nutritional profile, which include, for example, diabetes or hypertension. The remaining functionalities (evolution and communication) remain almost the same, due to their simplicity, and the good opinion of the users.

Figure 4 shows some examples of the final design of the older person's app. The rest of the screens are included in the supplementary material, section G.

Figure 4.

Some screens of the final design of the patient's app.

Healthcare professional subsystem

During the co-creation activities, we learnt community care professionals are considered by patients as their reference contact with the healthcare system. The closeness and frequency of visits make it possible to create a relationship of trust towards these professionals.

On the contrary, patients only visit a specialized care professional when they are referred to by the community care professional, not on demand. Then, community care professionals usually act as the medical hub of the patient, as they make the general and initial diagnosis of the patient, and based on it, they decide if they prescribe a treatment to patient and/or ask for a more specific consultation by a specialized care professional (e.g., geriatricians, cardiologists, rheumatologists, etc.). The specialized care professional may vary from one visit to the other, but the community care professional is always the same.

Because of this, one of the main purposes of the integrated health system is to facilitate and improve this interaction between patients and healthcare professionals. To do so, a mobile application has been designed to allow healthcare professionals monitoring the functional status and evolution of the older persons they follow up, and to intervene when required to prevent deterioration and potential disability and dependency.

Initial design

One of the main outcomes of the co-creation activities with healthcare professionals was that, given the limited time doctors have to attend to their patients, it is essential for the system to be very efficient and helpful to them, and not a burden. In relation to this, professionals stated that they do not have the ability to actively monitor all their patients, so having a system that automatically monitors patients and alerts them if something happens would be very helpful. To solve this problem, an alert engine has been incorporated into the system to warn clinicians about potential worsening of the intrinsic capacity of their patients depending on the information that is being collected at home. The rules implemented in the alert engine, provided in the supplementary material, section D, have been defined during the co-creation activities. This continuous monitoring enables triggering early interventions aimed at preventing disability and dependency, without obliging care professionals to actively monitor each of their patients.

To ensure the efficiency and learnability of the app, simplicity was used as the main design rule, but ensuring that the relevant information required by the professional to diagnose and administer treatment is accessible. In particular, the main screen shows the list of patients, together with the notifications of alerts and messages that have still not been treated, since these are the two events that clearly require the attention of the professional. Additionally, to provide the same information in an aggregated way, the main screen also shows the list of alerts and messages that have been generated by all the patients. By doing this, we provide professionals the possibility to visualize the information individualized for each patient, or in an aggregated manner.

Similarly, from the main screen, the professional can navigate to a specific screen where the list of existing alerts is displayed, or to the screen allowing the professional to receive and send messages to his/her patients.

Focusing on the specific information of the patients, the strategy of designing a simple interface that offers the most relevant information in a summarized but informative way continued to be maintained. From the first moment, the professionals made it clear that they did not need or want the system to have an excess of information and detail since they already have that in their current system, into which our system would be integrated. Therefore, one of the objectives of the first co-creation activities was to identify with the professionals what information they considered relevant enough to appear in the app. In general terms, four main categories were identified:

1. Patient's data, including personal data, personal history, history of medical consultations and reason of the last consultation.

2. Current treatment, including pharmacology, exercise, and nutrition prescriptions.

3. Evolution of the results from tests performed at home.

4. Basal situation of the patient, including physical, mental, and social information.

Additionally, the main screen of each patient displays the alerts and messages that may have been generated by him/her.

One of the learnings acquired during the co-creation activities was that the medical consultation is the nuclear component of the process, as it is where the care professionals reflect on the diagnosis of the patient after analyzing the results of the tests, in addition to the treatment that must be followed to prevent, reverse, or alleviate the worsening of the patient's frailty state.

This is why the flow of actions performed within the “New consultation” task is implemented in a separate screen. With respect to the information that needs to be stored in the system after a medical consultation, healthcare professionals expressed the need for a system that facilitates and supports the quick input of information, preventing mistakes from being made. Also, they asked that the system could guide them through the process, giving them recommendations or help, but always keeping control and decision-making.

To match these requirements, a divide-and-conquer approach has been implemented, clustering the information to be fulfilled for a new consultation into eight categories that emerged during the co-creation activities:

1. Reason for the consultation.

2. Current history, where professional describes the patient's symptoms.

3. Physical exploration, where all the data related to physical aspects are inputted (weight, height, BMI, blood pressure, state of the skin or abdomen, reflexes…).

4. Comprehensive Geriatric Assessment, that includes co-morbidity, personal history, allergies, current pharmacotherapy, results of tests evaluating the functional, mental, and nutritional status, social situation, and sensory information.

5. Other tests, where results from blood or urine tests, electrocardiograms, x-rays, densitometry or bioimpedance can be provided.

6. Evolution, where the professional can reflect on the progress of the patient with respect to the previous consultation(s).

7. Clinical judgement, allowing the professional to indicate the diagnosis of the patient.

8. Treatment plan, where professional indicates which tests must be taken, the new pharmacotherapy, the Vivifrail exercise plan to be performed and the nutrition recommendations to be followed.

In all cases, it was essential not only to determine which concepts should appear in each category, but also to use the specific terminology that is known and used by the professionals, to make the application easy-to-use and easy-to-learn.

Error prevention and guidance through the process was also implemented by using specific components wherever it was possible (data pickers, dropdown list, checkboxes, radio buttons…). A special mention should be made of the completion of the pharmacological treatment, in which an incremental search field for drugs was implemented based on the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10), ⁴⁴ which ensures consistency and validity of the pharmacological prescription. In addition, the STOPP/START criteria ⁴⁵ were implemented in the system to support healthcare professionals when establishing said treatment, avoiding the concurrence of medications whose joint prescription is inappropriate for the older adults.

Considering all these learnings and requirements, an initial functional prototype was created in Figma, which was incrementally created and modified through the different co-creation activities. Figure 5 displays two screens of the healthcare professional application. The full design of the initial prototype is presented in the supplementary material, section H.

Figure 5.

Some screens of the initial design of the app of the healthcare professional.

Final design

As for the patient's app, the healthcare app was iteratively and continuously refined until the final version was created and then assessed in the clinical pilot.

One of the main changes that were implemented during the co-creation process was that it was clearly identified that community and specialized care professionals cannot use the same application, since their functions, tools and objectives are different. For this reason, it was decided to create two different applications, both designed following the same principles and using the initial prototype as a reference, but customizing them to the needs of each healthcare professional profile:

• Specialized care professionals should not manage the alerts, given that daily monitoring and follow-up are carried out by community care professionals.

• Specialized care professionals should only receive messages from primary care professionals, not from patients.

• The Comprehensive Geriatric Assessment (CGA) should only be administered by specialized care professionals, as it includes specialized tests that cannot be performed by community care professionals. However, there is a subset of the tests included in the CGA that are administered by community care professionals that must be maintained in their app.

• Community care professionals must be able to refer a patient to a specialized care professional.

Apart from this, there were several aspects that were modified and improved throughout the co-creation activities.

One of the main changes was the inclusion of a new concept: the technical incidents. During the co-creation activities, it was detected that the patients could inadvertently perform some of the home tests incorrectly, producing a value that could erroneously reflect a worsening of their functional status. This would happen if, for example, when measuring gait speed, the cat crossed in front of the sensors during the test, or if the weight that was collected was that of the granddaughter, and not that of the older person. In the same way, it could also happen that there was a failure in the system or in the sensors, generating an incorrect value, despite the fact that the older person did the test correctly. To solve this problem, it was agreed that the community care professional should be able to indicate in the system, after checking the validity of the data with the patient, that a given value is actually erroneous or the consequence of a technical failure, causing it to stop being shown in the history and to be taken into account by the alert engine. As for the alerts, this functionality is only accessible by community care professionals, since they are the ones who do the continuous follow-up of the patient.

The second main change was related to the visualization of the historical information of both the tests carried out at home, as well as the community and specialized care visits. Instead of showing everything mixed up on the same screen, it was decided to separate the information based on its origin. To do this, a series of screens were created, accessible through visual metaphors of a home, a community care center, and a hospital. In the same way, and to maintain consistency, in the case of the primary application, access to the patient's alerts, messages and technical incidents also happened through visual metaphors. This separation also allowed more information to be displayed, such as the history of the test results or consultation reports, without overloading the user interface, and therefore maintaining the simplicity of the design.

With respect to the new possibility of referring a patient to the specialized care professional, it was proposed to include in the system a Decision Support System, in order to help the community care professional deciding when to refer a patient. To do so, the system checks if the results of the tests performed at the consultation comply with a series of rules, defined during the co-creation activities. In case any of these rules is violated, the system notifies the professional so that he/she can decide whether to refer the patient to the specialized care professional or not. The complete definition of the rules defined in the Decision Support System is presented in supplementary material, section E.

The last main change, as explained before, was carried out in the “new consultation” functionality, where the type and amount of information has been specified and personalized for each of the healthcare professional profiles.

Finally, with respect to the main screen, based on the healthcare comments during the co-creation activities, it was simplified by deleting the aggregated information about alerts and messages, and displaying only the list of patients, together with their notifications in the case of the community care professionals’ app. Also, the whole terminology has been revised and adapted to ensure it fits with the one used by healthcare professionals based on their profile.

Figures 6 and 7 show some screens of the final design of the community and specialized care professionals’ app, respectively. The rest of the screens are included in supplementary material, sections I and J.

Figure 6.

Some screens of the final design of the app of the community care professionals.

Figure 7.

Some screens of the final design of the app of the specialized care professionals.

Caregiver subsystem

A specific application was developed for the older persons’ informal caregivers so that they can be an active part of the care loop and track the health status of their loved ones. This is essential to reduce the uncertainty and potential burden that caregivers may feel and may positively influence in the adoption of the technology by the older adults.

A web-based application has been developed to make sure it can be used on any device with connection to the Internet and a web browser, like any current mobile device or computer.

The design of the application is also simple and clean, but in this case, the main color is blue. This has allowed us to distinguish between the older person's and caregiver's app, as the caregiver may potentially need to interact at some level with both (e.g., in case the older person needs help). In addition, blue transmits calm, confidence, and reliability, that are aspects sought by caregivers when using the application.

Using this application, caregivers can monitor the health status of their cared older person by visualizing the results of the tests as well as the commitment to the physical activity plan. Due to its simplicity and similarity with two of the complementary functionalities of the patient's app, the caregiver's app implements the same design decisions taken in the latter, and then has not been specifically co-designed. Designed screens are provided in the supplementary material, section K.

Cloud subsystem

The cloud subsystem is the server that allows integrating all the components of the POSITIVE ecosystem. This subsystem is responsible for the exchange and storage of all the information generated by the different subsystems, and its subsequent analysis. Then, the different mobile applications only act as senders and receivers of information, but they do not manage it. This structure is essential to ensure the consistency of the information. The centralization of the information also allows including the rules and logic of the alert engine and the decision support system in the cloud server.

The cloud subsystem stores, manages, and transmits to the older persons’ app the personalized monitoring and intervention plans specifically designed by the healthcare professionals through their apps. When prescribing medication in the intervention plan, the server is also in charge of checking the STOPP/START criteria and informing health care professionals about any potential negative drug interaction.

The server also contains a set of educational contents that are accessible to the older users through their application. These contents are structured as recommendations to maintain a healthy life. For instance, contents related to nutrition for healthy ageing have been designed by a nutritionist following the WHO and Tufts University recommendations. ⁴⁶

Finally, the cloud subsystem acts as the communication hub between the different actors.

Usability and UX evaluation of the resulting system

The resulting co-created system was later validated in a multicentric (Spain, Sweden, and Poland) randomized clinical trial (RCT; ClinicalTrials.gov Identifier: NCT04592146) to primarily test the clinical impact of the tool. Nevertheless, this study also pursued (as a secondary endpoint) evaluating the usability and UX. In this work, we present this evaluation conducted in Spain, concretely in the Region of Madrid.

Professionals participating in this study belonged to the Primary Care Services of SERMAS, Madrid's Region Healthcare Service. Older persons were recruited from the community; participation inclusion criteria were being at least 70 years old; living at home; having an informal caregiver; obtaining a Barthel index ⁴⁷  ≥ 90; and being pre-frail or frail according to the frailty phenotype criteria. ¹⁰

Additionally, it was set that older adults should meet three more criteria for their results being considered in the final analysis of the usability and UX:

• The application must be used at least half of the weeks of the trial, that is 12 weeks.

• The older person must complete the whole trial.

• The older person must be the main user of the application (and not the caregiver).

If any of these criteria occurred, the patient was excluded from the analysis, as we considered the evaluation was not a consequence of the real use of the system, in the terms defined in the trial.

All older adults and healthcare professionals who participated in the clinical trial received, read, and signed a written informed consent before starting the research.

We incorporated in the usability evaluation those users that met additional criteria of actual usage of the system, which were 10 older persons with an average age of 75.7 years old, and a female percentage of 60%. In addition, 12 professionals participated (six primary care doctors and six nurses), and out of them, 75% were female with an average age of 43.30 years old. Table 1 shows the SUS score and UX results obtained in the different sampling points carried out during the validation process.

Table 1.

Usability test results.

			Older Persons (average (standard deviation))	Healthcare Professionals (average (standard deviation))
SUS	Baseline (SD)		81.94 (13.04)	71.46 (15.90)
	M3 (SD)		78.50 (15.82)	73.41 (14.89)
	M6 (SD)		90.00 (6.96)	66.11 (12.75)
UEQ-S	Baseline	Pragmatic	-	1.48
		Hedonic	-	1.54
		Overall	-	1.51
	M3	Pragmatic	-	1.59
		Hedonic	-	1.93
		Overall	-	1.76
	M6	Pragmatic	2.25	1.44
		Hedonic	1.79	2
		Overall	2.02	1.72

In all cases, the systems obtained a SUS score above what is considered “good” (over 71.4). It is worth mentioning that older users rated the system better than professionals, giving a score that is considered as “excellent”, and in the last SUS it is almost 90.9 that is considered “best imaginable”.

Studying individual answers to the questionnaire from older adults, in all questions, more than half of the participants have answered positively (Agree and Strongly Agree) to positive questions (odd questions) and negatively (Disagree and Strongly Disagree) to negative questions (even questions). The remaining older participants used the neutral answer, meaning that they were neither satisfied nor dissatisfied with the system; only a few of the respondents clearly assess the usability of the system as bad.

From the healthcare professionals’ point of view, results indicate that they feel they would need help from a technician to use the system, and that they do not feel confident when using all the services. In a post-trial interview, professionals stated that the problem was due to how alerts were managed, as it was quite cumbersome and confusing, generating distrust and detachment. Being the alerts one of the core functionalities for the professionals, since they allow for early detecting potential worsening of the functional status of the older persons, this may have affected the global perception, and therefore how healthcare professionals have assessed usability and UX. Complete results of SUS questionnaire of both older adults and healthcare professional are included in supplementary material, section L.

To better analyze the results of the usability evaluation, we grouped SUS questions into categories related to usability attributes: general perception (Q1), ease of use (Q2, Q3 & Q8), learnability (Q4, Q7 & Q10), security (Q9), and consistency (Q5 & Q6). Even questions that are formulated as negative questions are transformed into positive statements.

Figure 8 clearly shows that all categories have been assessed very positively at baseline and after 6 months of use, but their evaluation has been considerably lower in month 3, especially in terms of ease of use, learnability, and consistency. This result contrasts with the evolution of the average SUS score, that, even though it decreases from baseline to month 3, it does not do it in a significant quantity. This diminution is common in this type of trials, where participants start the pilot very motivated with really high technology adoption. But as the study progresses, participants feel more tired and may not see the benefits of using the technology, which sometimes is difficult to perceive. This affects their motivation and adherence to the technology. In our case, this diminished perception at month 3 may be because the default configuration established older persons had to do many activities during the week (data collection, physical activity, etc.). Also, it is important to remark that the trial was conducted during the COVID-19 pandemic, which especially affected older adults, whose state of mind to try new technology might not be the most appropriate.

Figure 8.

Rate of older adults answering positively to questions, grouped by categories.

In the case of the healthcare professional, Figure 9 illustrates that their overall perception has improved over the time, so that almost 90% of the participants had a positive perception of the system. Ease of use and consistency suffered a decrease in their evaluation at month 3 but recovered an acceptable level after 6 months of use (around 70% and 78%, respectively). However, even after 6 months of use, the learnability and security of the system have been rated negatively (below 50%) due to the problems with the alerts management, as discussed above.

Figure 9.

Rate of healthcare professionals answering positively to questions, grouped by categories.

With respect to the UX evaluation, Figure 10 shows that older adults have evaluated as Excellent the three dimensions assessing UX (Pragmatic, Hedonic, and Overall), but it must be highlighted that the Hedonic dimension has a lower evaluation than the Pragmatic. This confirms the previous statement, as participants clearly recognize the utility of the system to help them monitor and improve their health status, but their lack of motivation due to the pandemic and the overload of weekly activities have slightly weighed down their perception in terms of feelings and sensations.

Figure 10.

UEQ-S benchmarking after 6 months of use (older persons).

In the case of the healthcare professional, Figure 11 reflects that their results are in line with those of the older adults, as the Pragmatic dimension is just rated as above average, as professionals felt that the system was not useful in their daily work since the alert management system was not intuitive. On the contrary, the Hedonic dimension has been rated as Excellent as they felt the system was innovative and had a great potential to help them in their work, provided the alert management meets their expectations and needs.

Figure 11.

UEQ-S benchmarking after 6 months of use (healthcare professionals).

These results do not seek statistical validity, as both questionnaires (SUS and UEQ-S) are qualitative, and then large number are not sought. Additionally, even if the number of participants in the trial is limited, their distribution as realistic as it represents quite accurately the actual population in Spain: 53% of the population 70  +  years old are women (in our study 60% of the participants are women), and 75% of the healthcare professionals are women (in our study 75% of the participants are women). ⁴⁸

Discussion

Main results

As shown by the results of the evaluation in terms of usability and UX of the POSITIVE system, following a co-creation strategy has resulted in a technological solution that not only allowed older users to follow the intervention during 6 months, but also it has been perceived as a usable, easy to use and satisfactory solution by older users and their healthcare professionals.

The COVID-19 pandemic has affected how the ecosystem has been used and thus perceived by older users and healthcare professionals. We believe older users may have perceived the system more useful than before the pandemic situation. On the contrary, during the pandemic, the exorbitant workload suffered by healthcare professionals has prevented them from participating in the trial as they wished. This has also been one of the reasons that explain the low performance of the alerts management system, which was designed to be checked on a weekly basis. Not being able to meet this do it due to the increased workload caused the alerts to accumulate week after week, meaning that every time professionals accessed the application, there were a huge number of alerts pending to be processed.

Data on the usability and UX evaluation of the caregivers’ application have not been presented since its use has been residual. This is because, in most cases, older persons have carried out the tests in the presence of their caregivers, since the latter visited them frequently. In a subsequent interview, older people were asked why they did the tests and measurements with their caregivers, and they all mentioned that they preferred to do them with their caregivers, not because they could not or did not know how to use the application on their own, but because they felt more comfortable this way. They also indicated that caregivers themselves preferred to be with them as well to be sure everything runs smoothly.

The literature has documented the main barriers and facilitators factors that exist when it comes to co-designing computer systems with older people. ⁴⁹ In this work, we have paid especial attention to them, ensuring the existence of group discussion during the activities, engaging the participants from the really beginning and all along the process, recruiting all the stakeholders implied in the system, paying especial attention to recruitment to ensure the representativity and objectivity of the population, and using familiar and accessible locations to perform the activities.

Additionally, in line with previous work, we created a relaxed, calm, and trusting environment with older people, which is of utmost importance to ensure their successful participation in co-creation activities. ⁵⁰ However, co-design does not ensure by itself the achievement of high levels of usability and user experience. ⁵¹ Therefore, we have enlarged the co-design activities to overcome these barriers and empower the facilitating factors. ⁵²

Limitations

One thread to the validity of the results is that not all participants (older adults and healthcare professionals), have evaluated the system in the three sampling points. In the case of the older adults, it was impossible to contact one of the patients at baseline, since he/she had gone on vacation and was ignoring his/her cell phone. On the other hand, in month 6, one of the participants refused to answer the questionnaires as he/she considered there were too many questions and that it was going to take too much time. In the case of the professionals, the variation in the number is due to the COVID-19 situation, as many of them have been unable to dedicate enough time to the trial, and therefore to answering the questionnaires.

Even if specialized care professionals have their own application, which is similar but not equal to the community care one, we have not evaluated it as only two geriatricians participated in the trial, but no patients were referred to them, so they had no opportunity to really use all the services.

Conclusions and future work

An integrated system has been designed to monitor remotely the functional status of older adults and deliver a multi-modal tailored intervention to prevent disability. The new organizational model includes older persons (in the center of care), their informal caregivers, and both community and specialized healthcare professionals.

To maximize the adoption and adherence to this new technology, a user-centered design process has been followed. More specifically, the system has been co-created by 48 older adults, healthcare professionals, technologists and UX researchers in 18 sessions. Also, 10 older adults and 12 healthcare professionals participated in a 6-month clinical trial, where the usability and UX of the system were assessed as secondary endpoints. Results are very promising, despite the enormous difficulties we had to face due to the COVID-19 pandemic. In fact, at the end of the trial, 88.9% of the participants, both older adults and healthcare professionals, stated that they would like to continue using the system. Even if our users (older adults and healthcare professionals) suffered the hardest hit by the pandemic, the results show that the POSITIVE technological ecosystem is highly usable and generates a good UX. Nonetheless, we have identified that some functionalities such as the alert management system need to be re-designed to be more flexible and adapt to the particular needs of the professionals. Furthermore, other aspects not directly related to how the system is designed and conceived, namely the default configuration of the monitoring and intervention plan for older persons, would need to be revisited to enhance user perception of the proposed solution.

In the short term, our plan is to refine the system to correct the identified usability issues. It is also planned to perform a formal evaluation of the system by specialized care professionals.

Regarding accessibility, we plan to perform a detailed expert evaluation following the Web Content Accessibility Guidelines from the W3 Consortium to ensure that all potential older users may benefit from the resulted ecosystem.

Finally, future work encompasses the clinical impact analysis of the trial, and the latter integration of the solution in a real environment, and its future transferability into a commercial solution.