Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Jul 1.
Published in final edited form as: Interact Comput. 2010 Jul 1;22(4):289–298. doi: 10.1016/j.intcom.2009.12.001

Response to a Relational Agent by Hospital Patients with Depressive Symptoms

Timothy W Bickmore a, Suzanne E Mitchell b, Brian W Jack b, Michael K Paasche-Orlow b, Laura M Pfeifer a, Julie ODonnell b
PMCID: PMC2901553  NIHMSID: NIHMS169315  PMID: 20628581

Abstract

Depression affects approximately 15% of the US population, and is recognized as an important risk factor for poor outcomes among patients with various illnesses. Automated health education and behavior change programs have the potential to help address many of the shortcomings in health care. However, the role of these systems in the care of patients with depression has been insufficiently examined. In the current study, we sought to evaluate how hospitalized medical patients would respond to a computer animated conversational agent that has been developed to provide information in an empathic fashion about a patient’s hospital discharge plan. In particular, we sought to examine how patients who have a high level of depressive symptoms respond to this system. Therapeutic alliance—the trust and belief that a patient and provider have in working together to achieve a desired therapeutic outcome— was used as the primary outcome measure, since it has been shown to be important in predicting outcomes across a wide range of health problems, including depression. In an evaluation of 139 hospital patients who interacted with the agent at the time of discharge, all patients, regardless of depressive symptoms, rated the agent very high on measures of satisfaction and ease of use, and most preferred receiving their discharge information from the agent compared to their doctors or nurses in the hospital. In addition, we found that patients with symptoms indicative of major depression rated the agent significantly higher on therapeutic alliance compared to patients who did not have major depressive symptoms. We conclude that empathic agents represent a promising technology for patient assessment, education and counseling for those most in need of comfort and caring in the inpatient setting.

Keywords: Depression, Relational Agent, Embodied Conversational Agent, Hospital Discharge, Therapeutic Alliance

1. Introduction

Depression is a common, debilitating mental health disorder and is one of the leading causes of disability among men and women of all ages worldwide (World Health Organization, 2001). Although depression is treatable, stigma associated with mental health problems in general, and depression in particular, represents a significant barrier to treatment (Sirey et al., 2001). In addition, access to mental health care can also present a formidable barrier to depression treatment, particularly in rural areas of the country (U.S. Dept of Health and Human Services, 2004). Furthermore, once patients are in treatment for depression, relapses due to non-adherence are common.

Computerized systems hold the promise of increasing the reach and efficacy of depression screening and treatment interventions. Several of these systems have now been developed and successfully evaluated, demonstrating efficacy for the identification and treatment of depression (Fann et al., 2009; Marks et al., 2003; J Wright et al., 2005). Central to the effective treatment of depression is the therapeutic alliance - the collaborative bond between patient and mental health provider (Krupnick et al., 2006), and we would expect that the therapeutic alliance is equally important to the efficacy of computerized depression treatment programs (Timothy Bickmore, Gruber, & Picard, 2005). Yet, existing automated systems for detecting and treating depression have not focused on the development of therapeutic alliance.

Over the last four years we have been developing and evaluating a computer animated conversational agent, designed to provide information and counseling to hospital patients at the time of hospital discharge. The agent has been designed with many verbal and nonverbal behaviors intended to foster the development of a therapeutic alliance with patients, such as empathy and social dialogue. In the current study, we sought to explore how patients with major depressive symptoms would react to the agent. Specifically, we investigated whether they would find it more or less acceptable and usable, and whether they would form a stronger or weaker therapeutic alliance with the agent, compared to patients without major depressive symptoms. Positive findings would indicate that animated conversational agents with relational behavior could provide an effective medium for delivering automated depression screening and treatment.

1.1 Depression

Depression affects 13–16% of people in the US, and is an important risk factor for functional impairment and poor outcomes among patients with various chronic illnesses such as diabetes, coronary artery disease and multiple sclerosis (Hasin, Goodwin, Stinson, & Grant, 2005; Parashar et al., 2006; Scherer & Herrmann-Lingen, 2009). Depression is associated with both physiological and behavioral factors affecting health outcomes for chronic disease patients including poor self-care management, reduced treatment plan adherence, and certain medical conditions such as chronic inflammatory states and hypercoagulability.

Despite the availability of depression treatment, less than one third of depressed patients receive treatment (Pratt & Brody, 2008). While depressed patients often express interest in treatment they also report barriers including time constraints, stigma, childcare conflicts, lack of transportation, and poor access to mental health services particularly in rural areas (Goodman, 2009). Studies also show that when physicians face a burden of competing clinical priorities for a given medical encounter, depression often goes untreated. Other barriers cited by physicians include fragmented mental health systems, lack of insurance coverage, patient resistance and difficulty making the diagnosis of depression.

1.2 Health Literacy and Mental Health

Health literacy is also a central interest of ours in developing agents for inpatient education and counseling, given the low levels of health literacy in our patient population (Paasche-Orlow, Parker, Gazmararian, Nielsen-Bohlman, & Rudd, 2005). Health literacy is the ability to perform the basic reading and numerical tasks required to function in the healthcare environment (American Medical Association Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, 1999). Health literacy is not simply the ability to read; it also requires a complex set of analytical and decision-making skills, and the ability to apply these skills to health situations. Fully 36% of American adults have limited health literacy skills, with even greater prevalence among patients with chronic diseases, those who are older, and those who have lower levels of education (Paasche-Orlow et al., 2005). Among indigent and minority patients in urban areas this number rises to over 80% (Williams et al., 1995).

A handful of studies have evaluated the relationship between health literacy and mental illness (Francis, Weiss, Senf, Heist, & Hargraves, 2007; Gazmararian, Baker, Parker, & Blazer, 2000; Lincoln et al., 2008; Lincoln et al., 2006; Morris, MacLean, & Littenberg, 2006; Weiss, Francis, Senf, Heist, & Hargraves, 2006). In most of these studies limited health literacy has been associated with higher rates of depressive symptoms. For example, one report demonstrated that health literacy has an important longitudinal relationship with the course of depressive symptoms among adults with addiction (Lincoln et al., 2006). Given this high degree of association between depression and health literacy, interventions targeted at assessing and treating depression must be designed to accommodate patients with limited health literacy.

1.3 Relational Agents in Mental Health

Relational agents—animated conversational agents designed to establish trust and therapeutic alliance with users over time—represent a potentially powerful technology for delivering health care services to patients with mental illness. Figure 1 shows the relational agent interface used in our work. These interfaces use the easy-to-understand format of face-to-face conversation, making them less intimidating and more accessible to patients with a wide range of computer, reading and health literacy skills, and a low-pressure environment where patients are free to take as much time as they need. In prior work, this interface has been found to be easily used by a wide range of participants, including those with no prior experience with computers (Bickmore, Caruso, Clough-Gorr, & Heeren, 2005). In addition, these agents use many verbal and nonverbal behaviors designed to establish a therapeutic alliance with patients, including displays of empathy, close proximity, more frequent eye gaze and attentiveness, and social dialogue and humor (Bickmore, Gruber, & Picard, 2005). Finally, many studies have shown that patients are more honest with a computer than a human clinician when disclosing potentially stigmatizing behaviors such as alcohol consumption and HIV risk behavior (Ahmad et al., 2009; Card & Lucas, 1981; Ghanem, Hutton, Zenilman, Zimba, & Erbelding, 2005; Kissinger et al., 1999; Newman et al., 2002). Individuals with depression may thus find a relational agent more approachable than a clinician in many situations, making it more effective at depression screening and counseling.

Figure 1.

Figure 1

Open in a new tab

Relational Agent Interface for Virtual Hospital Discharge Nurse Application

1.4 Overview

The development of the relational agent followed work over the last decade on a new standardized hospital discharge protocol designed to reduce re-hospitalizations. A key element of this protocol involves a human nurse spending half an hour with each patient, ensuring they understand their post-discharge self-care regimen before they leave the hospital. While this protocol was shown to be effective in a previous clinical trial (Jack et al., 2009), the additional nursing time required made it impractical to disseminate, motivating the development of the relational agent to perform this task. A more detailed description of the agent development methodology and pilot test results was presented in (Bickmore, Pfeifer, & Jack, 2009).

In the rest of this paper we first review related work, then briefly review the design of our relational agent for hospital patient education. We then describe a new study assessing the acceptance of the agent by hospitalized medical patients with and without symptoms of depression, as part of a new, ongoing clinical trial.

2. Related Work

2.1 Technology in Mental Health

Internet-based technology has been explored extensively for opportunities to expand access to mental health education and services. Mental health internet sites are now used to provide information, screen for mental health and mood disorders, assist in the delivery of treatment, and provide social and therapeutic support. Web-based interventions are effective for a range of mental health disorders including depression, panic, post-traumatic stress disorder, perceived stress in schizophrenia, stress, insomnia, and eating disorders. Overall, computer-based mental health services are well received by patients and clinically useful in feasibility trials, although an actively depressed mood can be a barrier to use.

2.2 Technologies for Depression Intervention & Treatment

Investigators have reported mixed results from clinical trials of internet-based depression screening and treatment programs (Helen Christensen, Griffiths, & Jorm, 2004; Clarke et al., 2005; Clarke et al., 2002). In general, computer-based technology has been used to substitute or augment traditional face-to-face therapeutic contact and the delivery of self-help materials. Studies using internet technology for depression support groups have shown reduced social isolation among users and perceived benefit from participation (Hill, Weinert, & Cudney, 2006; Weinert, Cudney, & Hill, 2008). However, some reports indicate that adolescents, who participated in internet support groups, had reduced interpersonal communication with family members. Screening for depression using internet-based self-assessment tools has proved successful, although minority and elderly persons are less likely to engage in web-based depression screening programs (Houston et al., 2001). Internet-based Cognitive Behavior Therapy and some self-help depression interventions are effective in relieving depression symptoms in mild to moderately depressed patients, and in some cases even more effective with severely depressed patients (Houston et al., 2001). Among factors positively correlated with computer use for mental health treatment are perceived usefulness of the treatment, preference for anonymity, ease of contact and ease of use (Lai, Larson, Rockoff, & Bakken, 2008). However, according to one study, when intervention programs are accessed outside the context of a randomized research trial, attrition rates are high.i This suggests that contact with a mental health provider may be necessary.

While currently available internet-based depression treatment can improve access to care, new technologies that serve as third parties to therapeutic interventions are on the horizon to enhance patient engagement in mental health care. For example, Coyle and Doherty described a 3D computer game designed to enhance adolescent engagement in therapeutic discussion (Doherty & Coyle, 2009). In their study, therapists reported overall acceptance of the game therapy by patients, however no evidence on patient preference or attitudes toward the technology was available from the subjects themselves.

Lisetti used animated anthropomorphic conversational agents to address social phobias such as public speaking anxiety, panic disorder and agoraphobia, in which the patient interacts with the agent in the reenactment of a fear-inducing situation (Lisetti, 2008). Patients participating in internet or computer-based depression interventions also face some challenges. Some of the barriers reported by users include time constraints, lack of motivation, technical or computer-access problems, physical illness, the lack of face-to-face contact, preference for taking medication, perceived lack of treatment effectiveness, and burden of the program (H Christensen, Griffiths, & Farrer, 2009). One commonly encountered barrier to computer interaction for depression across trials is active depressive symptoms (Doherty & Coyle, 2009). For example, in the Tailored Interventions for Management of DEpressive Symptoms (TIDES) program, a computer-based education program on self-care strategies for depressive symptoms in persons living with HIV/AIDS, was rated as easy to use and useful, but computer anxiety and depressive symptoms were negatively correlated with intention to use. Klein et al reported similar difficulties in a trial of cognitive behavior therapy for panic disordered patients (Klein, Richards, & Austin, 2006).

2.3 Conversational Agents in Mental Health

Some of the earliest dialogue systems developed in healthcare were designed for psychotherapy applications. The ELIZA system was developed in 1966 to simulate the behavior of a Rogerian psychotherapist, in which the patient and the computer exchanged typed text messages (Weizenbaum, 1966). Although ELIZA was not intended to be used for actual therapy, similar systems have been proven effective for therapy in which the system is essentially prompting a patient to think aloud and work through his or her own problems (Slack, 2000). In these applications, significant errors in understanding user input or in producing incoherent system output can often be tolerated, as the primary function of the system is just to keep the user engaged in the interaction.

Colby developed an ELIZA-like system that was designed to use Cognitive Behavioral Therapy to treat individuals with depression. In addition to providing typed text counseling with patients, the system provided text-based educational materials about depression (Colby, 1995). While Colby reported that the program was accepted by patients, evaluations by other researchers indicate the typed text medium confused some patients and the only comparative evaluation in the literature indicates that the system did not work as well as clinician-administered therapy (Wright, 2004).

2.4 Relational Agents in Mental Health

Bickmore developed a relational agent to promote medication adherence among patients with schizophrenia. The agent tracks each patient’s medication taking behavior for a single antipsychotic medication taken by mouth in pill or capsule form based on self-report, and also promotes physical activity (walking), and talking to the agent every day. For each of these three behaviors, the agent first asks for a self report of behavior, provides feedback on the behavior, and negotiates a behavioral goal. Intervention on each behavior is started and terminated according to a schedule for a 30-day intervention. Several elements were incorporated into the system to address the needs of individuals with schizophrenia, including extended orientation and termination of the therapeutic relationship, use of concrete language, and certain nonverbal behaviors. A 30-day quasi-experimental pilot study involving 20 patients indicated high levels of acceptance, usability, and self-reported adherence (Bickmore & Pfeifer, 2008).

3. Design of a Relational Agent for Patient Education at Hospital Discharge

We have developed an automated system that teaches hospital patients about their post-discharge self-care regimen, including medications, follow-up appointments, exercise and diet regimens, and pending lab tests. The system is designed to be used by patients while they are still in their hospital beds. In order to make the system as acceptable and effective as possible, we designed the interface to incorporate an animated virtual nurse agent who embodies best practices in health communication for patients with inadequate health literacy. The agent is deployed on a wheeled kiosk with a touch screen display attached to an articulated arm that can be positioned in front of patients while they are in bed (Figure 1). The agent is designed to interact with patients once every day they are in the hospital, but the primary interaction is just prior to hospital discharge. At this time, the patient spends approximately half an hour using the system, reviewing the layout and contents of a personalized “After Hospital Care Plan” (AHCP) booklet that is produced for them and contains their post-discharge self-care instructions. The paper booklet is given to patients before their conversation with the agent, and the agent reviews a digital version of the booklet in the interface, so that patients can follow along with the agent’s explanation in their paper booklets.

3.1 Development Methodology

Our multi-disciplinary design team comprised HCI researchers, doctors and nurses, a health literacy expert, and programmers and animators. The design process, from project start to completion of the user studies, lasted three years.

We used a multi-faceted approach to designing a system that would effectively teach patients, including those with inadequate health literacy, about their hospital discharge instructions. We began our design process with an ethnographic study of a re-engineered hospital discharge intervention that was currently underway in the hospital. Members of the design team visited hospital rooms, attended rounds with the medical team, observed discharge sessions in which nurses taught patients about their AHCP booklets, and interviewed the nurses who were performing this task. From these activities we learned about the stakeholders and basic workflow requirements of the system.

In addition to these “big picture” activities, we investigated the micro-behavior of expert nurses during discharge consultations with patients. We videotaped several mock discharge interactions in which one of the nurses explained an AHCP booklet either to a member of the research staff or a participant recruited from the community. We conducted discourse analyses of the videotaped interactions to characterize the verbal and nonverbal behavior used by the nurses while explaining a booklet to a patient. We also conducted two rounds of user testing of an agent explaining discharge instructions to users in our HCI lab (Bickmore, Pfeifer, & Yin, 2008).

3.2 Implementation

The agent was developed using an existing framework for ECA-based health counseling (Bickmore & Picard, 2005), extended with a computational model for the explanation of documents (Bickmore et al., 2008). In this interface, the agent speaks, using a synthetic voice, and displays animated nonverbal behavior (hand gestures, posture shifts, facial expressions, etc.) in synchrony with the speech. User contributions to the conversation are made by touching utterance option buttons on a touch screen display that are dynamically updated for each user speaking turn (Figure 1). We considered using speech recognition as the input modality rather than the touch screen, but the hospital room can be a very noisy environment, and a significant portion of the patient population speaks English as a second language with many accents that would be problematic for commercial speech recognition products.

Dialogues are scripted, using a custom hierarchical transition network-based scripting language, and a visual dialogue design tool. The final system contains 550 dialogue states including 322 unique medication scripts covering 2254 medicines, along with 48 scripts for diagnoses.

The importance of caring, empathy and good “bedside manner” is widely recognized in healthcare as a key factor in improving not only patient satisfaction, but treatment outcomes across a wide range of health care disciplines (Garrity, 1981), and particularly in nursing (Sourial, 1997). Given this, prior successful implementations of empathic computer agents (Bickmore & Picard, 2005), and the need for the agent to maintain patients’ attention for the hour it may take to relate all of the information in their discharge plans, we augmented the agent’s informational dialogue with relational dialogue. Following earlier work on relational agents, we integrated a range of relational behavior into the agent dialogue, including appropriate forms of address (calling the patient by name), social chat at the beginning of every interaction, meta-relational communication, appropriate humor, appropriate feedback at every empathic opportunity, and references to information discussed in past interactions to give a sense of continuity (Bickmore et al., 2005). The agent also offers patients the opportunity to take breaks at several points during the interaction in order to sustain attention and engagement.

A fragment of a typical conversation is shown in Figure 2. The agent proceeds through the AHCP booklet linearly, describing each section before moving on to the next. Conversations generally consist of: 1) a greeting and social chat (Figure 3); 2) orientation to the virtual nurse and the discharge process; 3) introduction of the AHCP (Figure 4); 4) review of medications, including comprehension tests (Figure 5); review of appointments, including comprehension tests; review of recommended diet and physical activity; “patient activation”, in which the patient is urged to keep track of any questions or issues they want to discuss with their primary care provider (Figure 6); and their primary diagnosis (Figure 7). Following the interaction with the system, typically lasting 30–40 minutes, a report is printed for the human nurse that describes issues raised by the program that require follow up, such as questions about medications that the virtual nurse could not answer.

Figure 2.

Figure 2

Open in a new tab

Sample Dialogue - Relational Aspects Highlighted

Figure 3.

Figure 3

Open in a new tab

Greeting and Social Chat with Virtual Nurse

Figure 4.

Figure 4

Open in a new tab

Introduction of After Hospital Care Plan by Virtual Nurse

Figure 5.

Figure 5

Open in a new tab

Review of Medications by Virtual Nurse

Figure 6.

Figure 6

Open in a new tab

Patient Activation by Virtual Nurse

Figure 7.

Figure 7

Open in a new tab

Review of Primary Diagnosis by Virtual Nurse

3.3 Pilot Studies

Two rounds of pilot studies were conducted to assess acceptance, usability and satisfaction with the system (Bickmore et al., 2009). Results indicate that patients found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or nurse. Patients also expressed appreciation for the time and attention provided by the virtual nurse, and felt that it provided an additional authoritative source for their medical information.

4. Acceptance of Relational Agent by Patients with Depressive Symptoms

We sought to explore how hospital patients with major depressive symptoms would react to the relational agent, by investigating whether they would find it more or less acceptable and usable, and whether they would form a stronger or weaker therapeutic alliance with the agent, compared to patients without major depressive symptoms.

4.1 Methods

A secondary analysis was performed using data from 139 English-speaking, hospitalized adults from an ongoing randomized controlled trial conducted at an urban academic safety-net hospital. The parent study, currently ongoing, is a two-armed evaluation study of the impact of the agent and other improvements to the hospital discharge process on 30-day hospital readmissions.

4.1.1 Study Setting

The deployment site for the relational agent system is Boston Medical Center, a 547 bed safety net hospital that serves an urban, 84% minority, traditionally underserved population. Approximately 58% of this population has inadequate health literacy (see Section 4.1.3 for the measure used).

4.1.2 Participants

Participants in the study were English-speaking adult patients, 18 years or older, admitted to the teaching service of Boston Medical Center, a large urban safety-net hospital with an ethnically diverse patient population. Three hundred and forty-seven subjects were enrolled and randomized between October 15, 2008 and June 20, 2009. Patients were required to have a telephone, be able to comprehend study details and the consent process in English, and have plans to be discharged to a US community. Patients were not enrolled if they were admitted from a skilled nursing facility or other hospital, transferred to a different hospital service prior to enrollment, admitted for a planned hospitalization, on hospital precautions, on suicide watch, deaf or blind.

Of the 347 subjects enrolled into the parent study, 173 were randomized into the relational agent intervention arm of the study. Of these, 131 completed all measures necessary for our analyses.

4.1.3 Measures

Depressive Symptoms

The primary independent variable of interest was depressive symptoms, defined as a positive score for major depression on the validated PHQ-9 depression screening tool. A dichotomized variable was created using a standardized scoring system to determine the screening cut-off for major depressive symptoms (score ≥ 10 on a possible score range of 0 to 27) (Kroenke, Spitzer, & Williams, 2001).

Therapeutic Alliance

Patient perception of therapeutic alliance with the agent was assessed using the Bond subscale of the Working Alliance Inventory, a 12-item questionnaire assessing the emotional dimension of a patient’s trust and belief that they can work together with their provider to achieve desired therapeutic outcomes (Horvath & Greenberg, 1989).

Health Literacy

We assessed health literacy using the 66-word version of the Rapid Estimate of Adult Literacy in Medicine (REALM) (Davis et al., 1993). We defined limited health literacy as a reading level of 8th grade and below and adequate health literacy as 9th grade and above for our analyses, as prior authors have done (Lincoln et al., 2006; Lindau, Basu, & Leitsch, 2006; Mancuso & Rincon, 2006; Sudore et al., 2006).

Attitudes Towards the Agent

In addition to therapeutic alliance, we assessed additional patient attitude towards and satisfaction with the agent using single item, scale response questions, shown in Table 1.

Table 1.

Self-report Measures of Attitudes Towards the Relational Agent

Measure Question Anchor 1 Anchor 7
SATISFACTION How satisfied were you with Elizabeth? Not at all Very satisfied
USABILITY How easy was talking to Elizabeth? Easy Difficult
CONTINUE How much would you like to continue working with Elizabeth? Not at all Very much
RELATIONSHIP How would you characterize your relationship with Elizabeth? Complete stranger Close Friend
PREFERENCE Would you rather have talked to your doctor or nurse than Elizabeth? Definitely prefer doctor or nurse Definitely prefer Elizabeth
ADHERENCE How likely is it that you will follow Elizabeth’s advice? Not at all likely Very likely
Open in a new tab
Questions Asked

All patient interactions with the agent were logged for subsequent analysis. From these logs we counted the number of times each patient selected a response that provided additional information when offered as an option.

4.1.4 Procedure

Following enrollment, collection of background, demographic and depressive symptom information and randomization, intervention patients have their post-discharge self-care information entered into a workstation by a study nurse. This information is used to generate the AHCP booklet, and is also downloaded to a mobile kiosk that is then wheeled into the patient’s room (Figure 1). After the patient is given their paper booklet and provided with a brief training session on how to use the touch-screen interface, they are left to conduct their conversation with the agent. At the end of this interaction, any unresolved patient questions or issues are displayed for a human nurse to follow up with the patient, and results of the session are uploaded to a database. At this time, self-report measures covering attitudes towards the agent and therapeutic alliance are collected. All self-report measures are verbally collected to accommodate patients with limited literacy.

4.2 Results

All variables were tested for normality. Therapeutic alliance was found to have a significant negative skew and was logarithmically transformed (after reflection) to normalize the distribution. Questions Asked had a positive skew but was not transformed due a floor effect and the nature of the measure (non-parametric tests were used).

Of the 131 patients analyzed, 19 (14.5%) were classified as having major depressive symptoms according to the PHQ-9. Table 2 compares patient demographic and other characteristics by depression status. Patients with major depressive symptoms were similar to other patients on all characteristics except for computer literacy, with the depressive group scoring significantly lower.

Table 2.

Patient Demographics by Depression Status

Characteristics No Major Depressive
Condition n=112		 Major Depressive
Condition n= 19		 p-value
Gender, n (%)
   Male 62 (56) 8 (42) 0.32
   Female 49 (44) 11 (58)
Age, mean (SD) 49 (13) 46 (13) 0.28
Education level, n (%)
   Less than high school 4 (4) 2 (11) 0.41
   Some high school 23 (21) 6 (32)
   HS graduate or GED 39 (35) 7 (37)
   Some college 24 (21) 2 (11)
   4-yr college graduate or above 19 (17) 2 (11)
Health literacy level, n (%) 0.09
   Grade 3 and below 13 (11) 5 (26)
   Grade 4 to 6 10 (9) 4 (21)
   Grade 7 to 8 31 (28) 3 (16)
   Grade 9 and above 58 (52) 7 (37)
English primary language at home, n (%) 99 (90) 17 (89) 0.94
Patient married, n (%) 21 (19) 1 (5) 0.14
Computer literacy, mean (SD) 2.45 (0.87) 2.0 (1.05) <0.001
Attitude towards computers, mean (SD) 2.59 (0.81) 2.63 (0.96) 0.13
Have you ever been told by a doctor or
therapist that you have depression?		 34 (30) 15 (79) <0.001
Have you ever been prescribed medication for
depression?(only if “yes” to above –
diagnosed with depression at some point).		 29 (26) 14 (74) <0.001
Do you take medication for depression now?
(only if “yes” to above = diagnosed
with depression at some point).		 12 (11) 9 (47) <0.001
Length of hospital stay in days, mean (SD) 2.16 (2.99) 2.84 (1.46) 0.13
Open in a new tab

Table 3 shows self-report ratings of the agent. Overall ratings of satisfaction and ease of use were very high, and only 24% of patients said they would have preferred receiving their discharge information from their doctor or nurse (40% were neutral, 36% said they definitely preferred the agent). Patients with major depressive symptoms stated a significantly greater desire to continue interacting with the agent, p<.05 by Mann-Whitney test (mean of 6.6 vs. 5.6).

Table 3.

Self-report Assessments of the Relational Agent p-values from Mann-Whitney test

Measure No Major Depressive Condition Major Depressive Condition p-value
Mean SD Mean SD
SATISFACTION 6.50 1.07 6.59 1.18 0.37
USABILITY 1.99 1.82 1.71 1.83 0.49
CONTINUE 5.58 1.78 6.56 1.03 0.03
RELATIONSHIP 4.80 1.73 5.24 1.89 0.30
PREFERENCE 4.16 2.03 4.73 1.67 0.29
ADHERENCE 6.22 1.21 6.73 0.59 0.13
Open in a new tab

Table 4 shows correlations among continuous measures. There are significant positive correlations between therapeutic alliance, number of questions asks, satisfaction with the overall virtual nurse system, desire to continue using the system, preference for the virtual nurse over a human doctor or nurse, and stated expectation of following the agent’s advice.

Table 4.

Correlations Among Continuous Measures

Measure Alliance Literacy Questions SATISFY USABILITY CONTINUE RSHIP PREF ADHERE
Alliance 1 −.123 .208* .317** −.249** .408** .393** .394** .319**
Literacy −.123 1 −.193* −.093 −.072 −.073 −.069 −.090 −.053
Questions .208* −.193* 1 .039 .103 .208* .217* .128 .154
SATISFY .317** −.093 .039 1 −.306** .430** .253** .198* .118
USABILITY −.249** −.072 .103 −.306** 1 −.166 −.087 −.023 −.223*
CONTINUE .408** −.073 .208* .430** −.166 1 .497** .310** .297**
RSHIP .393** −.069 .217* .253** −.087 .497** 1 .280** .171
PREF .394** −.090 .128 .198* −.023 .310** .280** 1 .198*
ADHERE .319** −.053 .154 .118 −.223* .297** .171 .198* 1
Open in a new tab
*

p< .05 level

**

p<.01

Patients with major depressive symptoms scored the agent significantly higher on therapeutic alliance compared to patients classified as not having major depressive symptoms (6.2 vs. 5.5, before transformation), t(108)=2.02, p<.05, d=0.58.

Patients classified as having inadequate health literacy scored significantly higher on therapeutic alliance compared to patients with adequate health literacy (5.9 vs. 5.4, before transformation), t(116)=2.56, p<.05, d=0.47. Patients with inadequate health literacy also asked the agent significantly more questions, compared to patients with adequate health literacy, p<.05 by Mann-Whitney test.

4.3 Discussion

Self-report ratings of satisfaction, ease of use, and attitudes towards the agent were high for all patients, with only 24% of patients indicating they would have preferred receiving their discharge information from their doctor or nurse. This result is similar to that found in our pilot studies, in which patients stated that they appreciated the amount of information given to them by the agent, the amount of time that the agent spent with them, and that the agent did not talk down to them as many providers do (Bickmore et al., 2009). As one patient reported:

“It was just like a nurse, actually better, because sometimes a nurse just gives you the paper and says ‘Here you go.’ Elizabeth explains everything.”

The primary finding of the study is that patients with symptoms indicative of major depression rated the agent significantly higher on therapeutic alliance compared to patients who did not have major depressive symptoms. In combination with their greater stated desire to continue working with the agent, this indicates that a relational agent is not only acceptable to patients with major depressive symptoms, but that these patients feel they have established a stronger emotional bond with the agent compared to patients without depressive symptoms.

Patients with inadequate health literacy also reported a significantly greater therapeutic alliance with the agent compared to patients with adequate health literacy. This effect was independent of depression. Given that both health literacy and depression represent barriers to healthcare, this indicates that automated patient education systems incorporating relational agent technology could help reduce disparities in access to care.

4.3.1 Limitations

There are several important limitations to our preliminary study. First, our results may not be generalizable to populations other than those served by urban safety net hospitals or other populations excluded from the parent study. Second, our results are correlational, so we do not know the direction of the associations between major depressive symptoms and other measures reported. Finally, our results may be due to the fact that patients with depressive symptoms react more positively to computer-based interventions, or any novel intervention. This would be consistent with reports from evaluations of many other computer interventions for patients with mental health problems (Wright, 2004). Our observation that patients with inadequate health literacy also had higher therapeutic alliance may serve to partially mitigate this concern.

5. Conclusion and Future Work

Depressive symptoms are a risk factor for many adverse health conditions and early hospital readmission, thus the ability to identify hospitalized patients for depression is of great interest to the healthcare community. A relational agent that can screen and develop a therapeutic alliance may also enhance the likelihood of successful outpatient depression treatment following hospital discharge. Further, the deleterious effects of active depression on treatment response and disease severity may be ameliorated with prompt identification and initiation of treatment at the time of hospitalization for patients with depressive symptoms.

Low health literacy contributes to health disparities across diverse realms of healthcare, including access to care, medical decision-making, medication adherence, and preparation for a diagnostic test. The significantly greater therapeutic alliance reported by patients of low health literacy may be related to the agent’s ability to deliver information in a nonjudgmental manner at a patient-centered pace of learning, alleviating the stigma felt by patients with low health literacy. Developing similar patient counseling applications in other healthcare settings where health literacy impacts disease outcomes could have a positive impact on health disparities.

Our future work in this area involves developing and evaluating relational agents for depression screening and treatment, to determine if the results found in this study do in fact imply the promise of the technology for reaching and treating depressed individuals in the inpatient setting.

Acknowledgements

This work was supported by grants from the NIH National Heart, Lung and Blood Institute and the Agency for Health Research and Quality. Thanks to the many members of the Re-Engineered Discharge project team at Boston Medical Center and the Relational Agents Group at Northeastern University for their contributions to this work.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. American Medical Association Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs. Health literacy: report of the Council on Scientific Affairs. JAMA. 1999;281(6):552–557. [PubMed] [Google Scholar]
  2. Ahmad F, Hogg-Johnson S, Stewart D, Skinner H, Glazier R, Levinson W. Computer-assisted screening for intimate partner violence and control: a randomized trial. Ann Intern Med. 2009;151(2):93–102. doi: 10.7326/0003-4819-151-2-200907210-00124. [DOI] [PubMed] [Google Scholar]
  3. Bickmore T, Caruso L, Clough-Gorr K, Heeren T. "It's just like you talk to a friend" - Relational Agents for Older Adults. Interacting with Computers. 2005;17(6):711–735. [Google Scholar]
  4. Bickmore T, Gruber A, Picard R. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Education & Counseling. 2005;59(1):21–30. doi: 10.1016/j.pec.2004.09.008. [DOI] [PubMed] [Google Scholar]
  5. Bickmore T, Pfeifer L. Relational Agents for Antipsychotic Medication Adherence Proceedings of the CHI'08 Workshop on Technology in Mental Health; Florence, Italy: 2008. [Google Scholar]
  6. Bickmore T, Pfeifer L, Jack BW. Taking the Time to Care: Empowering Low Health Literacy Hospital Patients with Virtual Nurse Agents; Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI); Boston, MA: 2009. [Google Scholar]
  7. Bickmore T, Pfeifer L, Yin L. The Role of Gesture in Document Explanation by Embodied Conversational Agents. International Journal of Semantic Computing. 2008;2(1):47–70. [Google Scholar]
  8. Bickmore T, Picard R. Establishing and Maintaining Long-Term Human-Computer Relationships. ACM Transactions on Computer Human Interaction. 2005;12(2):293–327. [Google Scholar]
  9. Card W, Lucas R. Computer Interrogation in Medical Practices. International Journal of Man-Machine Studies. 1981;14:49–57. [Google Scholar]
  10. Christensen H, Griffiths K, Farrer L. Adherence in Internet Interventions for Anxiety and Depression. J Med Internet Res. 2009;11(2):e13. doi: 10.2196/jmir.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: randomised controlled trial. British Medical Journal. 2004:328. doi: 10.1136/bmj.37945.566632.EE. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Clarke G, Eubanks D, Kelleher C, O'Connor E, DeBar LL, Lynch F, et al. Overcoming Depression on the Internet (ODIN) (2): A Randomized Trial of a Self-Help Depression Skills Program With Reminders. J Med Internet Res. 2005;7(2) doi: 10.2196/jmir.7.2.e16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Clarke G, Reid E, Eubanks D, O'Connor E, DeBar LL, Kelleher C, et al. Overcoming Depression on the Internet (ODIN): A Randomized Controlled Trial of an Internet Depression Skills Intervention Program. J Med Internet Res. 2002;4(3) doi: 10.2196/jmir.4.3.e14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Colby K. A computer program using cognitive therapy to treat depressed patients. Psychiatric Services. 1995;46:1223–1225. doi: 10.1176/ps.46.12.1223. [DOI] [PubMed] [Google Scholar]
  15. Davis TC, Long SW, Jackson RH, Mayeaux EJ, George RB, Murphy PW, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993;25(6):391–395. [PubMed] [Google Scholar]
  16. Doherty G, Coyle D. Clinical Evaluations and Collaborative Design: Developing new technologies for mental health interventions; Proceedings of CHI 2009; Boston, MA: 2009. [Google Scholar]
  17. Fann J, Berry D, Wolpin S, Austin-Seymour M, Bush N, Halpenny B, et al. Depression screening using the Patient Health Questionnaire-9 administered on a touch screen computer. Psychooncology. 2009;18(1):14–22. doi: 10.1002/pon.1368. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Francis L, Weiss B, Senf J, Heist K, Hargraves R. Does literacy education improve symptoms of depression and self-efficacy in individuals with low literacy and depressive symptoms? A preliminary investigation. J Am Board Fam Med. 2007;20(1):23–27. doi: 10.3122/jabfm.2007.01.060058. [DOI] [PubMed] [Google Scholar]
  19. Garrity T. Medical compliance and the clinician-patient relationship: a review. Soc Sci Med. 1981;15E:215–222. doi: 10.1016/0271-5384(81)90016-8. [DOI] [PubMed] [Google Scholar]
  20. Gazmararian J, Baker D, Parker R, Blazer D. A multivariate analysis of factors associated with depression: evaluating the role of health literacy as a potential contributor. Arch Intern Med. 2000;160(21):3307–3314. doi: 10.1001/archinte.160.21.3307. [DOI] [PubMed] [Google Scholar]
  21. Ghanem K, Hutton H, Zenilman J, Zimba R, Erbelding E. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sex Transm Infect. 2005;81(5):421–425. doi: 10.1136/sti.2004.013193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Goodman J. Women’s attitudes, preferences and perceived barriers to treatment for perinatal depression. Birth. 2009;36(1):60–69. doi: 10.1111/j.1523-536X.2008.00296.x. [DOI] [PubMed] [Google Scholar]
  23. Hasin D, Goodwin R, Stinson F, Grant B. Epidemiology of major depressive disorder: results from the National Epidemiologic Survey on Alcoholism and Related Conditions. Arch Gen Psychiatry. 2005;62(10):1097–1106. doi: 10.1001/archpsyc.62.10.1097. [DOI] [PubMed] [Google Scholar]
  24. Hill W, Weinert C, Cudney S. Influence of a computer intervention on the psychological status of chronically ill rural women: preliminary results. Nurs Res. 2006;55(1):34–42. doi: 10.1097/00006199-200601000-00005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Horvath A, Greenberg L. Development and Validation of the Working Alliance Inventory. Journal of Counseling Psychology. 1989;36(2):223–233. [Google Scholar]
  26. Houston TK, Cooper LA, Vu HT, Kahn J, Toser J, Ford DE. Screening the public for depression through the Internet Psychiatr Serv. 2001;52:362–367. doi: 10.1176/appi.ps.52.3.362. [DOI] [PubMed] [Google Scholar]
  27. Jack BW, Chetty VK, MD DA, Greenwald JL, PharmD GMB, Johnson AE, et al. The Re-Engineered Hospital Discharge Program to Decrease Rehospitalization: A Randomized, Controlled Trial. Annals of Internal Medicine. 2009;150(178–187) doi: 10.7326/0003-4819-150-3-200902030-00007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kissinger P, Rice J, Farley T, Trim S, Jewitt K, Margavio V, et al. Application of computer-assisted interviews to sexual behavior research. Am J Epidemiol. 1999;149(10):950–954. doi: 10.1093/oxfordjournals.aje.a009739. [DOI] [PubMed] [Google Scholar]
  29. Klein B, Richards J, Austin D. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry. 2006;37(3):213–238. doi: 10.1016/j.jbtep.2005.07.001. [DOI] [PubMed] [Google Scholar]
  30. Kroenke K, Spitzer R, Williams J. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Krupnick J, Sotsky S, Elkin I, Simmens S, Moyer J, Watkins J, et al. The Role of the Therapeutic Alliance in Psychotherapy and Pharmacotherapy Outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. J. Consulting and Clinical Psychology. 2006;64:532–539. doi: 10.1037//0022-006x.64.3.532. [DOI] [PubMed] [Google Scholar]
  32. Lai T, Larson E, Rockoff M, Bakken S. User Acceptance of HIV TIDES—Tailored Interventions for Management of Depressive Symptoms in Persons Living with HIV/AIDS. J Am Med Inform Assoc. 2008;15:217–226. doi: 10.1197/jamia.M2481. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Lincoln A, Espejo D, Johnson P, Paasche-Orlow M, Speckman J, Webber T, et al. Limited literacy and psychiatric disorders among users of an urban safety-net hospital's mental health outpatient clinic. J Nerv Ment Dis. 2008;196(9):687–693. doi: 10.1097/NMD.0b013e31817d0181. [DOI] [PubMed] [Google Scholar]
  34. Lincoln A, Paasche-Orlow M, Cheng D, Lloyd-Travaglini C, Caruso C, Saitz R, et al. Impact of health literacy on depressive symptoms and mental health-related: quality of life among adults with addiction. J Gen Intern Med. 2006;21(8):818–822. doi: 10.1111/j.1525-1497.2006.00533.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Lindau S, Basu A, Leitsch S. Health literacy as a predictor of follow-up after an abnormal Pap smear: a prospective study. J Gen Intern Med. 2006;21(8):829–834. doi: 10.1111/j.1525-1497.2006.00534.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Lisetti C. Embodied Computer Agents for psychotherapy. Paper presented at the CHI 2008 Workshop on Technology in Mental Health.2008. [Google Scholar]
  37. Mancuso C, Rincon M. Impact of health literacy on longitudinal asthma outcomes. J Gen Intern Med. 2006;21(8):813–817. doi: 10.1111/j.1525-1497.2006.00528.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Marks IM, Mataix-Cols D, Kenwright M, Cameron R, Hirsch S, Gega L. Pragmatic evaluation of computer-aided self-help for anxiety and depression. The British Journal of Psychiatry. 2003;183:57–65. doi: 10.1192/bjp.183.1.57. [DOI] [PubMed] [Google Scholar]
  39. Morris N, MacLean C, Littenberg B. Literacy and health outcomes: a cross-sectional study in 1002 adults with diabetes. BMC Fam Pract. 2006;7:49. doi: 10.1186/1471-2296-7-49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Newman J, Des Jarlais D, Turner C, Gribble J, Cooley P, Paone D. The Differential Effects of Face-to-Face and Computer Interview Modes. Am J Public Health. 2002;92(2):294–297. doi: 10.2105/ajph.92.2.294. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. World Health Organization. Geneva, Switzerland: World Health Organization; The world health report 2001 - Mental Health: New Understanding, New Hope. 2001
  42. Paasche-Orlow MK, Parker RM, Gazmararian JA, Nielsen-Bohlman LT, Rudd RR. The prevalence of limited health literacy. J Gen.Intern.Med. 2005;20(2):175–184. doi: 10.1111/j.1525-1497.2005.40245.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Parashar S, Rumsfeld J, Spertus J, Reid K, Wenger N, Krumholz H, et al. Time course of depression and outcome of myocardial infarction. Arch Intern Med. 2006;166:2035–2043. doi: 10.1001/archinte.166.18.2035. [DOI] [PubMed] [Google Scholar]
  44. Pratt L, Brody D. Depression in the United States Household Population. NCHS Data Brief, 2005–2006. 2008 Sep 7;:1–8. [PubMed] [Google Scholar]
  45. Scherer M, Herrmann-Lingen C. Single item on positive affect is associated with 1-year survival in consecutive medical inpatients. J Gen Hosp Psych. 2009;31:8–13. doi: 10.1016/j.genhosppsych.2008.09.020. [DOI] [PubMed] [Google Scholar]
  46. U.S. Department of Health and Human Services. Rockville, Maryland: DHHS; The President's New Freedom Commission on Mental Health, Subcommittee on Rural Issues: Background paper. 2004
  47. Sirey J, Bruce M, Alexopoulos G, Perlick D, Friedman S, Meyers B. Stigma as a barrier to recovery: Perceived stigma and patient-rated severity of illness as predictors of antidepressant drug adherence. Psychiatr Serv. 2001;52(12):1615–1620. doi: 10.1176/appi.ps.52.12.1615. [DOI] [PubMed] [Google Scholar]
  48. Slack W. Patient-Computer Dialogue: A Review. Yearbook of Medical Informatics 2000. 2000:71–78. [PubMed] [Google Scholar]
  49. Sourial S. An analysis of caring. J Adv Nurs. 1997;26:1189–1192. doi: 10.1111/j.1365-2648.1997.tb00812.x. [DOI] [PubMed] [Google Scholar]
  50. Sudore R, Yaffe K, Satterfield S, Harris T, Mehta K, Simonsick E, et al. Limited literacy and mortality in the elderly: The health, aging, and body composition study. Journal of General Internal Medicine. 2006;21(8):806–812. doi: 10.1111/j.1525-1497.2006.00539.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Weinert C, Cudney S, Hill W. Rural women, technology, and self-management of chronic illness. Can J Nurs Res. 2008;40(3):114–134. [PMC free article] [PubMed] [Google Scholar]
  52. Weiss B, Francis L, Senf J, Heist K, Hargraves R. Literacy education as treatment for depression in patients with limited literacy and depression: a randomized controlled trial. J Gen Intern Med. 2006;21(8):823–828. doi: 10.1111/j.1525-1497.2006.00531.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Weizenbaum J. Eliza - a computer program for the study of natural language communication between man and machine. Communications of the ACM. 1996;9(1):36–45. [Google Scholar]
  54. Williams M, Parker R, Baker D, Parikh N, Pitkin K, Coates W, et al. Inadequate Functional Health Literacy Among Patients at Two Public Hospitals. JAMA. 1995;274(21):1677–1720. [PubMed] [Google Scholar]
  55. Wright J. Computer-Assisted Cognitive-Behavior Therapy. In: Wright J, editor. Review of Psychiatry: Cognitive-Behavior Therapy. Vol. 23. Washington, DC: American Psychiatric Publishing; 2004. pp. 55–82. [Google Scholar]
  56. Wright J, Wright A, Albano A, Basco M, Goldsmith L, Raffield M, et al. Computer-Assisted Cognitive Therapy for Depression: Maintaining Efficacy While Reducing Therapist Time. Am J Psychiatry. 2005;162:1158–1164. doi: 10.1176/appi.ajp.162.6.1158. [DOI] [PubMed] [Google Scholar]

RESOURCES