Avatars and virtual agents – relationship interfaces for the elderly

Abstract

In the Digital Era, the authors witness a change in the relationship between the patient and the care-giver or Health Maintenance Organization's providing the health services. Another fact is the use of various technologies to increase the effectiveness and quality of health services across all primary and secondary users. These technologies range from telemedicine systems, decision making tools, online and self-services applications and virtual agents; all providing information and assistance. The common thread between all these digital implementations, is they all require human machine interfaces. These interfaces must be interactive, user friendly and inviting, to create user involvement and cooperation incentives. The challenge is to design interfaces which will best fit the target users and enable smooth interaction especially, for the elderly users. Avatars and Virtual Agents are one of the interfaces used for both home care monitoring and companionship. They are also inherently multimodal in nature and allow an intimate relation between the elderly users and the Avatar. This study discusses the need and nature of these relationship models, the challenges of designing for the elderly. The study proposes key features for the design and evaluation in the area of assistive applications using Avatar and Virtual agents for the elderly users.

1. Introduction

The Digital Era changed and challenged healthcare relationship and created transformation in several layers in respect to healthcare services especially in issues relating to communication and interaction of humans and machines in the health/care scenario. To understand these transformations, we must mention the following three historical developments and their role for both healthcare providers and patients: the change in users’ requirements, the market forces dynamics, and most importantly the technological advancements and availability. Shepherd [1] claims that Digital Era is characterised by technology which increases the speed and breadth of knowledge turnover within the economy and society. ‘The Digital Era can be seen as the development of an evolutionary system in which knowledge turnover is not only very high, but also increasingly out of the control of humans, making it a time in which our lives become more difficult to manage’ furthermore she claims that ‘the Digital Era has changed the way we live and work by creating a society and economy that is geared to knowledge, whether that knowledge is content-laden and therefore scientifically factual or instead is content-free and therefore reliant on emotions or indeed any combination in between’ ([1], p. 1).

In this era, people belong to active social and economic communities and the effect on all members of these economies is their aspiration to gain as much knowledge as possible. As technological functionality becomes more knowledge-based, we become more depended on knowledge retrieval technologies and online devices. Thus, understanding the Digital Era in terms of evolution means enrolling technologies to provide service and knowledge in every area and healthcare has been an excellent subject of this revolution.

Social economics indicates that ‘elderly population’ is growing massively creating a new balance in market forces of the healthcare domain. The engagement with older population requires the adaptation of new care methodologies and technologies. The attention to elderly's support and care has grown and the services the healthcare provider is providing to this population has grown enormously especially in trends of prevention and companionship.

Preventive medicine and service has developed to be a game changer and technologies that deal with predictive medicine are developed to assist in prevention. This trend has changed the lifestyle not only of patients but also of care givers as well.

Interaction in the digital, mobile and big data era suggests that the healthcare segment is indeed not much different from any other line of service and that the health provider offers a service to the patient, now can also be defined as a customer for all aspects of service. This customer is now demanding a friendly and efficient customer experience when having a health interaction in every channel of interaction offered to him. On the other hand, this customer is also expected to be able to interact freely and to provide information as asked. This is a great change of attitude in both sides.

In fact, let us coin the expression HaaS-Health as a Service (based on the cloud platform services). Health providers are offering HaaS as part of their digital strategy directly or through third parties and patients are eager to accept and try new venues of communication. The HaaS trend is also a result of the availability of supporting technologies like: internet, and cloud services, mobile networking, multimedia, storage and transfer and computing.

This, coupled with user interface technologies, creates an easy to use interaction by every user from children to elderly population. Such services are divided into several categories such as: information services, data collection, Q&A and consulting, diagnostics and emergency assistance.

2. Patient avatars and virtual agents in health care services

An example of such deployed technology is Avatars and Virtual Agents; (A&VA), create a ‘live’-like sensation and interaction experience thanks to the correct and smart fusion of the multimodal capabilities. In fact, the more sophisticated the interface is and the more knowledgeable, the better the interaction between users and their Avatars. Virtual Agents are able to get the user to ‘task completion home run’ if they can be personalised to the right extent and, no less important, if they are efficient. Avatars and Virtual Agents; a new tool used to create a unique customer experience. It fits the needs of the customer and personalised to his requirements and measures. In fact, A&VA are used to upgrade the quality of service, availability and accessibility, accommodating the growing needs of patients for information and interaction. To be more specific; Avatars are used in a more personalised environment as an extension of the patient for prevention and data recording applications and self-service. Virtual Agents are used by the health provider and Health Maintenance Organizations (HMOs) to give general information services to their customers, collect data and provide answers and first aid medical diagnostics. Virtual Agents may be used as a first step before being transferred to a live agent or medical professional.

One example is ‘Fredrick’ [2] a medical Avatar designed for mobile platform built to assist Diabetes patients with day-to-day routines. The Avatar is designed as a virtual representation of a medical assistant with whom the user can communicate using graphical or voice interface. Fredrick multimodal application includes the necessary components to manage the basic routines of the patient, for instance: (i) track blood sugar level, (ii) set reminders, (iii) watch blood sugar level statistics over time, (iv) share the data with external reviewers, and more. All of these are provided through a user-friendly multimodal interface including voice input and out puts, text and touch capabilities. The Avatar itself is oriented to accommodate various users using simple, very clear language with a carefully designed Neuro-linguistic programming (NLP) components (lexicons and grammars). The dialogue manager controls the logic component in addition to a communication module between the interfaces and the databases creating a single, synchronous application (Fig. 1).

Fig. 1.

Fredrick medical avatar application screens

3. Human vs. human machine relationship

The need of the elderly population for automatise assistance and companionship, is well established in several researches [3–5], all of them claiming that the elderly users will benefit from a technology that will enable them to stay autonomous in their home environment and to have a self-determined way of living for as long as possible has been marked as one of the most important societal goals of the future. According to Yaghoubzadeh et al. [6] technology could offer some of the needed support but often suffers from a lack of ease of use and, consequently, an acceptance barrier with these special user groups. Thus, it is not yet totally clear what are the key factors to a successful interaction of elderly users and their Virtual Assistant or Avatar. It is the human-like behaviour which is the key element? This will be further discussed in Sections 5 and 6.

Cassell [7] claims that, where social collaborative behaviour is a key factor, representing a system as a human is the correct interface. Her term for A&VAs is Embodied Conversational Agents (ECA). She argues that an ECA is an interface in which the system is represented as a person and in which information is conveyed to human users via multiple modalities such as voice and hand gestures and so on. Casswell [7] was actually able to translate the rules of engagement into a table of conversational functions and their behaviour realisation which, in turn, could be used for ECA-based systems so that the user interface will be as similar as possible to human-to-human interaction.

This is in fact a challenging task. Some of these implementations require a high order of artificial intelligence technology to support them, such as NLP, machine learning and so on. The ability of an Avatar to generate face-to-face communication between real and virtual persons allows a much richer communication channel. It enables multimodal communication through both verbal and nonverbal channels such as gaze, gesture, spoken intonation and body posture [8].

In his paper ‘Four models of physician-patient relationship’, Emanuel and Emanuel [9] suggested four structures to describe the interaction and the change in the relationship: ‘Paternalistic model’ – the physician decides on the treatment and ‘calls the shots’. The patient has little autonomy and decision in regards to his health choices. Informative model – the physician tells the patients about the possible treatments and data but patients make the final choice in full autonomy. Interpretive models – physician helps the patients to explore their own values, not only health related but choices that has to do with their private and social or moral considerations. Deliberative model – physician helps the patient to explore health related values and they choose together.

It is clear that in todays’ Digital Era patients are more involved in the discovery and information gathering process, exploring for data even before they meet with the physician. They come ready with lists and questions, suggestion and case studies and success stories. They are often overloaded with inaccurate data that they gathered in their exploration over the internet. The fact is that huge amount of data is available; some is structured and some manipulated, some is based in personal views and social media or advertisement. All in all the balance relationships between the physician and the patient has change.

When we come to analyse human machine relationship such as with AV&A, our basic working assumption is that an interaction is taking place and the aspiration is to achieve a dialogue that enables task completion whatever the user task is.

In their work, Rich et al. [10] discuss the theory of collaboration, which is basically intended for human-to-human interactions, and apply it to human-machine interaction. ‘Collaboration is a process in which two or more participants coordinate their actions towards achieving shared goals’ ([10], p. 2). They introduce the concept of an agent independent of the application that communicates with the user in a three-way communication (agent, app and user). The machine has been personified and ‘collaboration’ is now taking place between two users: one virtual and one human.

In our examination of theories of collaboration, we see that we can directly relate them to the concept of team work and we find three perspectives as described by [11]: The humanistic view (teams with human-only interaction), the mechanistic view (in AI, for teams of machines) and the human-machine view (joint teams). If we accept the definition that team work means being responsive to each other's needs and mutually supportive to succeed in the joint plan, then we definitely get a sense of relationship that is being created between users, in our case between the Avatars and Virtual Agents and the elderly user.

When comparing the four physician-patient models mentioned by Emanuel and Emanuel [9] to the three types of interaction relationships between A&VA and users describes in [12], we find similarity between the models. We find that in order for a dialogue to be beneficial some form of relationship, empathy and collaboration must be established between the parties of the interaction be it humans or virtual. The same way different models exist for humans, different relationship types exist for humans & A&VA.

4. Avatars & virtual relationship model

Lomanowska and Guitton [13] surveys the vast research done about the usage of digital technologies for intimate social interactions via Internet and mobile. According to them there is a growing interest and popularity so that the presence of well detailed human-like avatars coupled with advanced animation technologies enhances intimate connection in virtual interaction compared to other online media ‘Interestingly, virtual relationships can be as meaningful as those established in the real world, and they can often transition from the virtual to the real world or be maintained in parallel across these two spheres.’ ([13], p. 1–2)

Shaked and Artelt [12] propose three main relationships types between human users and Avatars & Virtual Agents:

4.1. Type 1 – the avatar as virtual me

In this relationship type the Avatar is a mirror image of the user, a persona used to function as an extension for various purposes (playing a virtual game, personal assistance, medical avatar to be used in health applications, and a reflection of fitness and health application). The relationship is self-contained; personal activities are tracked without the need for external intervention. This type of Avatar is highly personalised according to the profile of the user and his preferences, with the ability to learn patterns and behaviours and the relationship can take on a high degree of intimacy. The preferred modalities will be inherent to the user and his profile and will not change very much.

An avatar for elderly people called GeriJoy [14] is a caregiving companion, built to address many of the unique challenges faced by seniors and their families. It is an example how simple solutions partly human and partly virtually based can prove to be extremely beneficial for elderly patients with dementia or Alzheimer's. Dementia patients, in particular, struggle with loneliness, and daily human interaction, communication improves mental function and mood. Thus, GeriJoy companion avatar is designed to be a supportive friend and caregiver. It is able to listen to and remember what the user is saying, like names of grandchildren, favourite places, TV shows. It monitors the emotional states of the elderly such as feeling lonely or confused, and can provide engaging and supportive conversation.

4.2. Type 2 – interaction with a personalised/specialised A&VA

This relationship type refers to the use of a specialised Virtual Agent that specialises in a certain type of activities such as: banking and financial services, HMO health agents’, education, and government assistance. On the one hand these virtual agents represent a company, an organisation or a service provider but, on the other hand, they are not random. They have prior knowledge of the user as their customer, sometimes on a routine basis. They know the profile of the user, his history and habits. They may be not as good as the personal avatar but in their specific line of business they can make predictions and next best offers. They can also connect generally available data with personal data to maximise service.

Going back to the idea of HaaS (Health as a service) e-health services provided by HMO are growing and are directly connected to a multichannel approach whereby users can access their personal records and information and perform actions from various digital channels such as Internet, mobile app, telephone, chat and so on. Having an agent that is common to all channels and familiar with the patient's profile and habits is an advantage that is leveraged to create a holistic, seamless customer experience across every interaction.

4.3. Type 3 – me and a random avatar

Type 3 is defined as a relationship between the user and a Virtual Agent with whom he is not familiar, a random interaction with a virtual representative who contacts user while he surfs the internet surfing or through an online segmentation and targeted advertisement campaign.

Here the user is mostly in a passive and less cooperative mode. For design purposes these agents must be very interesting, articulate and very engaging to get user attention and response.

Essentially these random virtual agents can be defined as virtual personas answering real-time questions asked on a website, at click speed, without pauses, 24 hours a day, completely automatically. These VAs can engage in a dialogue and help customers make a decision.

Virtual Agents of this kind are also called ‘Chatbots’. There are many good examples of Chatbots being used in marketing or customer service that offer more than just information. According to statistics from the World Health Organization the world faces a deficit of some 13 million doctors and health care professionals over the next 20 years Deloitte [15]. That means that service application like health bots are growing in demand. In 2015 the Chinese search engine Baidu launched a medical Chatbot designed to make diagnosing illnesses easier. The conversational bot is named Melody (http://www.theverge.com/2016/10/11/13240434/baidu-medical-chatbot-china-melody) and comes built into the company's iOS and Android Baidu Doctor app. Baidu Doctor allows users to contact local doctors, book appointments, and ask questions, with the Chatbot intended to speed up this process.

In the interaction with elderly users, which of these suggested types has the best chance of creating the right relationship that will enhance the users’ success and experience encouraging a natural and positive engagement? Moreover, what are the necessary design elements crucial for success?

5. Design for the elderly – challenges

Yaghoubzadeh et al. [6] researched Virtual Agents as daily assistants for the elderly and conducted studies on acceptance and interaction feasibility. They were concerned with two issues: ‘(i) are virtual agents accepted as assistants by these user groups, and which system design would be particularly preferred? (ii) How can the interaction between the agent and such users be made feasible, i.e. sufficiently robust and effective?’ ([6], p. 2).

Yaghoubzadeh et al [6] also mention two projects which support the elderly inclination to use Avatars and Virtual Agents and interact in a spoken dialog. The GUIDE project [16] which using focus groups, found out that spoken language was the most preferred interaction modality for elderly users unfamiliar with technology. They also noticed the preference of users to converse with Avatars if there is a task problem while in a regular fluent interaction they should disappear appearing automatically when a task problem was detected. With regard to Virtual Agents as potential social companions, Vardoulakis et al. [17] used Wizard-of-Oz (WOZ) evaluation and found high acceptance ratings for a relational agent dialogue system perceived as a social companion for older people, who were free to choose topics for the conversations. These experiments are done in controlled environment using focus groups and WOZ methodology not real time systems and thus will lack some real time interactive factors and some system errors that are bound to happen in production environment. In terms of the inclination to use virtual assistant or companion, it is clear that there is a need from both the side of the elderly patient as well as the care givers. In addition, according to these studies the elderly users have needs but also have expectations from the systems.

According to Olsson [18] user expectations are a factor affecting the actual user experience design in human-computer interaction. There are four different layers of expectations constituting a framework which is providing an understanding of the spectrum of user expectations from technologies. Surveying elderly users about their expectations, values, needs, and requirements from these emerging technologies can certainly shed light on which designs may succeed and how people conceptualise the interfaces. In [19] we established the connection between technology readiness and user needs and requirements. We claimed that the interaction technologies that underlie user interfaces are greatly dependent on and influenced by user expectations and experience. This is crucial to the success of interface design in general, and is even more important in complex Mobile Multimodal interfaces where multiple factors can influence the user experience. To create a multimodal holistic experience that meets user needs and expectations we must first methodically understand all the layers of user expectations.

Going back to the relationship types detailed in Section 4, the claim is that type-1; the avatar which is a personal image functioning as an extension of the user or a close virtual assistance, will be most successful in creating the intimate relationship and the companionship so needed by the elderly users. The possible personalisation of this Avatar and the ability to understand the user patterns and interaction choices is critical in the case of elderly users who need guidance and who feel lonely many hours of the day as well as requiring some type of monitoring health or social. The following section will outline the required features which influence the interface design.

6. Evaluation of relationship

Following the work in [12] a selected list of features where found to be most important for the elderly in the interaction with A&VA.

Ease of Use: This feature is concerned with how friendly and easy it is to navigate the interface, and if it requires a sophisticated learning effort. This feature is crucial to accommodate the elderly cognitive and physical abilities. Visualisation: This feature deals with the external appearance of the A&VA in the application — whether there is a human-like agent, an animal, an image or just an app user interface. How likeable is it and how well it represents the user, will determine the elderly's level of cooperation. As shown in the case of [14] and others, animals have created a sense of cooperation and warmth that appealed especially to the elderly users. This is directly connected to the important choice of the Persona; the Avatar and its personality which is conveyed through visual, speech, facial and emotional gestures. Personalisation relates to how important it is that the A&VA will be able to offer an array of services specifically personalised to the elderly daily tasks and needs. To establish Type 1 relationship, personalised tasks are critically important. Data Collection and Analytics also play an important role in the process of personalisation. The Analytics facilitate the recognition of user behaviour patterns and the prediction of future user behaviour which create a sense of familiarity and intimacy for the elderly user.

Error recovery is the ability of the A&VA to gracefully recover from a dialogue mistake or misunderstanding rather than create an infinite loop and a bad interaction experience. Error recovery is not easy and requires NLP module as well as data analytics technologies, but it is highly important especially in a dialogue with users that are prone to mistakes and confusion. Failure to create a smooth interaction is a problem. Another feature important to user experience is Latency. i.e., the delay between input into the system and the appearance of the desired output. Latency greatly affects the quality of the interaction as well as the flow of the dialog. As a result, high latency (the reaction in less than a few seconds per interaction) may be perceived in the eyes of the elderly user as inability to understand him and create a sense of confusion.

7. Other supporting technologies

Other supporting technologies also determine the quality of the interaction and the level of the user experience for the elderly. We have already mentioned the importance of Multimodal Interaction Technologies in A&VA design. The ability to incorporate the largest number of input/output technologies into the dialogue enables the user to choose which interface is preferable and accommodate the elderly capabilities. Gamification, i.e., adding gaming components and rewards for task completion and improvement into the A&VA-user dialog, is highly recommended for Type 1 Avatars because it support continuity and familiarity with the user. Machine learning algorithms and AI capabilities are necessary for an A&VA to interact in a natural, human-like way, leveraging the user experience and generating cooperation from the side of the user. To achieve this, the interaction must be monitored continuously over time, with the ability to collect, store and analyse the data and then run Machine learning AI algorithms to process the interaction and learn to improve it.

8. Conclusion

In a new demanding health environment, Avatars and Virtual Agents assistance for the elderly is required and seem to prove efficient. The opportunities to create helpful and friendly interfaces is challenging. The success is highly dependent on creating a relationship between the human and the virtual so that a correct evaluation matrix can be met. The feature matrix described above gives us a glimpse into the considerations and determination of the quality issues of A&VA. The matrix includes visual, performance and ambient features as well as trust and entrainment aspects to establish the relationship between the elderly users and their Avatar. If so, the contribution to healthcare providers, as well as to patients and care giver is highly valuable.

10. Conflict of interest

None declared.