User-acceptability of an automated telephone call for post-operative follow-up after uncomplicated cataract surgery

Abstract

Background

Innovative technology is recommended to address the current capacity challenges facing the NHS. This study evaluates the patient acceptability of automated telephone follow-up after routine cataract surgery using Dora (Ufonia Limited, Oxford, United Kingdom), which to our knowledge is the first AI-powered clinical assistant to be used in the NHS. Dora has a natural-language, phone conversation with patients about their symptoms after cataract surgery.

Methods

This is a prospective mixed-methods cohort study that was conducted at Buckinghamshire Healthcare NHS Foundation Trust. All patients who were followed up using Dora were asked to give a Net Promoter Score (NPS), and 24 patients were randomly selected to complete the validated Telephone Usability Questionnaire (TUQ) as well as extended semi-structured interviews that underwent thematic analysis.

Results

A total of 170 autonomous calls were completed. The median NPS score was 9 out of 10. The TUQ (scored out of 5) showed high rates of acceptability, with an overall mean score of 4.0. Simplicity, time saving, and ease of use scored the highest with a median of 5, whilst ‘speaking to Dora feels the same as speaking to a clinician’ scored a median of 3. The main themes extracted from the qualitative data were ‘I can see why you’re doing it’, ‘It went quite well actually’, ‘I just trust human beings I suppose’.

Conclusion

We found high levels of patient acceptability when using Dora across three acceptability measures. Dora provides a potential solution to reduce pressure on hospital capacity whilst also providing a convenient service for patients.

Introduction

The demand for the healthcare system is rising, with an ever-increasing mismatch between the capacity of the trained workforce and the demand for care [1]. Cataract surgery, the most commonly performed elective surgical procedure, is one of many pathways facing significant capacity concerns.

The NHS Long Term Plan considers digital advancements as a key part of the solution to capacity concerns [2] and Ophthalmology has already been at the forefront of digital innovation over the last two decades with deep learning-based artificial intelligence (AI) solutions demonstrating expert-level performance for clinical tasks such as diabetic retinal screening and predicting conversion to wet age-related macular degeneration [3, 4].

More recently, the first AI-driven clinical assistant capable of delivering cataract surgery follow-up calls has been developed, Dora. There is already a body of evidence demonstrating the safety and efficacy of clinician delivered follow-up calls after routine cataract surgery over the telephone [5–7]. Dora was designed to automate these often-stereotyped conversations, in turn freeing the time of skilled clinicians. Automating such conversations has the potential to help reduce outpatient face-to-face follow-up significantly, in line with guidance from RCOphth and the NHS [2, 8].

Despite the adoption of conversational voice agents in the UK and globally [9], these agents remain novel within healthcare, and as such, many patients are unfamiliar with engaging with automation as part of their care. In light of the Medical Research Council’s framework for evaluating complex interventions, we aimed to assess the patient acceptability of using Dora to follow up with patients over the phone after routine cataract surgery in order to inform the ongoing development of Dora [10].

Methods

This study is a prospective mixed-methods cohort study that was conducted at Buckinghamshire Healthcare NHS Foundation Trust (BHT) between June and September 2021.

Population

At BHT, patients having routine, uncomplicated cataract surgery usually have a nurse-led telephone follow-up call approximately 3–4 weeks postoperatively. For this study, suitability for a Dora call was broadly equivalent to suitability for a nurse-led telephone-based follow-up, with inclusion and exclusion criteria outlined in Appendix 1.

Intervention

Follow-up calls were delivered by Dora, a UKCA mark Class 1 AI-powered clinical assistant. All patients were provided with written information about Dora either at discharge, or via the post prior to their follow-up call. At the start of this call, suitable patients were offered enrolment in the study and provided informed consent on the call (outlined in Appendix 2). Consented participants had an autonomous Dora telephone call instead of the standard clinician call.

The Dora call initially checks the patient’s identity, then asks key symptom questions to ascertain the presence of significant symptoms—red eye, painful eye, change in vision, floaters, and flashing lights. These questions were adapted from a number of studies examining the safety of telephone follow-up for cataract surgery [5–7, 11], and included follow-up questions to help clarify the significance of symptoms. The call then provides an opportunity for patients to ask questions and discuss further surgery and the next steps of their care (Fig. 1).

Fig. 1. Dora call overview summary diagram.

Boxes represent conversation ‘modules’, patients experience this as a back-and-forth conversation.

All Dora calls in this trial were supervised in real-time by an ophthalmologist (EL, AH or GM) through a web-based supervisor interface. The supervisor contacted the patient to clarify any concerns and arrange further review if clinically required.

Assessment of acceptability

Net Promoter Score SM

Patient acceptability was assessed at two timepoints. Firstly, the Net Promoter Score SM (NPS®) [12] was captured by Dora who asks every participant at the end of the call, “On a scale of 1 to 10, how likely would you be to recommend this automated system to a friend or a colleague?” An overall NPS score is calculated by taking the promoters (scores 9 or 10) minus the detractors (scores below 7) and dividing by the total number of responses. Scores equal to and above 50% are deemed excellent with companies that garner world-class loyalty receiving NPS scores of 75–80% [13].

Extended interviews

A total of 15% of participants were randomly selected to participate in a structured telephone interview to allow for a deeper exploration of their experience. Interviews were conducted by SK. A list of consenting participants for a qualitative interview was randomly selected by listing their alphanumeric codes in excel and applying the randomisation tool. They were then contacted via telephone until the target recruitment of 24 interviews was reached.

The interview consisted of two parts, initially the validated Telehealth Usability Questionnaire (TUQ) that uses a series of Likert scale responses to 16 questions was used to assess patient acceptability [14]. Subsequently, patients were asked a series of semi-structured interview questions from a topic guide (Appendix 3). The topic guide was developed based on the Theoretical Framework of Acceptability to assess patient acceptability before, during and after their Dora call [15]. The topic guide ensured that discussions covered the same basic issues with each interviewee but with sufficient flexibility to allow the exploration of new issues of importance to patients. Interviews lasted up to 40 min and were recorded before being formally transcribed.

Data analysis

The NPS was analysed for normality and then reported using standard methods of averages and distribution.

The qualitative data from the semi-structured interviews was analysed using thematic analysis as described by Nowell et al. [16]. MAXQDA (Berlin, Germany) software was used to track and facilitate data analysis. The process of thematic analysis involved researchers (1) familiarising themselves with the data, (2) creating initial codes (using inductive method), (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and then producing the report. All four reviewers performed inductive coding of the data set independently. After familiarising themselves once with the data, a codebook was developed based on the initial codes. Team meetings were held during the iterative process, and a final comprehensive thematic framework was agreed upon when data saturation was reached. The qualitative study was developed and recorded according to the COREQ guidance [17].

The study was conducted according to the tenets of the Declaration of Helsinki. Ethics approval was obtained from the Manchester Research Ethics Committee, and the study was registered with the ClinicalTrial.gov website (registration no: NCT04885868).

Results

A total of 184 calls were made with Dora to 177 patients (7 patients received Dora calls also for their second eye surgery). In all, 96% (177/184) of calls reached the clinical decision part of the conversation, and 92% (170/184) of calls were completed in full. 24 patients completed the TUQ and 21 underwent further semi-structured interviews. The 3 participants who dropped out of the interviews attributed it to time constraints. Baseline demographic information is shown in Table 1.

Table 1.

Demographic details of patients consenting to Dora calls and further semi-structured interviews.

	All Dora calls	Semi-structured interviews
Number of participants	177	24a
Age range	41–98	56–86
Median (IQR)	76 (10)	76 (10)
Male:Female ratio	(1:1.3)	(1: 1.4)

^aThree patients opted out of full-length interviews and so their results have not been included in the thematic analysis.

Net Promoter Score (NPS)

Across 170 calls, the NPS score out of ten was a mean of 8.6, the mode 10 and median 9. The distribution of these scores is displayed in Fig. 2. An overall NPS score of 45 was calculated.

Fig. 2. Net Promoter Score (NPS) distribution.

Patients were asked the question by Dora: “On a scale of 1 to 10, how likely would you be to recommend this automated system to a friend or a colleague?”. A total of 56% of patients were ‘promoters’ of the system (score of 9–10), 33% ‘neutral’ (score of 7–8), and 11% ‘detractors’ (score of 1–6).

Telephone Usability Questionnaire

The TUQ analysis showed high rates of acceptability, with an overall mean score of 4.0. Simplicity, time saving, and ease of use scored the highest with a median of 5, whilst ‘speaking to Dora feels the same as speaking to a clinician’ scored a median of 3 (Fig. 3).

Fig. 3. Telehealth Usability Questionnaire (TUQ) results.

Patients on average, strongly agreed with statements that Dora highly on time saving from travelling, ease of use and being able to hear Dora clearly.

Thematic analysis

Three main themes were extracted from the semi-structured interviews as well as sub-themes and concepts that captured minor themes (Table 2).

Table 2.

The main themes, sub-themes, concepts and quotes from extended interviews.

	Main themes	Sub-themes	Concepts	Quotes
Theme 1	I can see why you’re doing it	Saving time and money for the NHS	Higher value activities, staffing issues	“we haven’t got the staff or the money to speak to a nurse”. U039 “ …that’s saving time and money for the National Health, my heart beats for the National Health”. U078 “I’m quite happy with Dora, I can see how it will save time for more important things”. U062
		Convenience for patients	Increased accessibility, no parking required, no driving required, no physical appointment required	“everytime I (would) choose Dora because it seemed not to be a waste of time. [It is a] wholly unnecessary exercise to get up and get in the car, aim to be at the hospital at a certain time, find a parking and all that goes with all that, just to see at the doctor’s office that no side effects at all, and you say ‘okay thank you very much’.. so Dora seems an ideal way as I can see it eliminating the vast majority of calls which will be from people saying “yes I’m fine, I’ve got no side effects”’. U034 “well it’s less time - it’s more convenient for both the health service and for me, because I didn’t have to go anywhere to have that done“. U078 “I don’t have to go and meet anybody, park up and things like that. You know, when you go and meet somebody, you need to find parking spaces and travel and because I don’t drive now it’s not convenient, so I’m quite happy to do Dora”. U064 “[the benefit is] I didn’t have to go along to the hospital appointment”. U079
Theme 2	It went quite well actually	Acceptance of new technology	Comparison with automated telephone banking services, moving with the times, information given about Dora in advance of the call is reassuring	“you know like other places when you phone and you have an automated thing, you’re thinking ‘oh my gosh!’… initially that’s what I thought it was going to be you know…but the voice is pleasant to listen to which is good ‘cause you want someone pleasant to listen to when you go into things like that, so that was good, yeah it was a positive experience”. U038 “I think generally for people that are a bit switched on it would be fine with them”. U039 “I don’t have strong feelings at all because a lot of the paperwork…was…explaining that there would be a follow-up call, it didn’t surprise me or phase me at all”. U034 “I mean it’s been happening before for me so it makes no difference”. U063 “[I understand it] pretty well, it’s the same as phoning the bank”. U069 “I didn’t find any questions a problem and it is easy enough to follow’. U071 “[A] very adequate substitute for a routine follow up”. U034 “[Dora] just spoke clear English and that’s automatic I can understand it”. U044 “we cannot look backwards, we have to look forward to new technologies”. U078
		Apprehension of new technology	Feeling unsure before the call, not being good with technology, belief that new technology is not appropriate for their age, acceptable for them but apprehension for peers who might not cope	”I mean I’m not very good with technology but it was easy…it just followed through”. U062 “I wasn’t sure what I would feel, it’s new technology”. U078 “I’ m probably not the best person in the world for dealing with machines…I don’t believe in making people of my generation have smartphones”. U083 “having looked after an elderly gentleman across the road I know that he wouldn’t have coped with it, you know along with Zoom and iPads and things… I don’t think that he’d even know who or what Dora was or why Dora was calling”. U039 “I was a bit worried because, you know, you’re not talking to a person, until it happened. I was alright once it happened, but when you’re talking to an automated thing you’re not too sure are you”. U064 “I didn’t find any questions a problem, and is it easy enough to follow, but…if I have a slight query over something at the end… I don’t think Dora would have answered that for me”. U071 “I was concerned because I didn’t quite know what questions were going to be fired at me by Dora”. U084 “I’m just a bit concerned that perhaps some other people who didn’t quite grasp the fact that this was an automated person would chat on aimlessly”. U084
Theme 3	I just trust human beings I suppose	Missing social interaction	Trust, personal quality of humans, ability to express to a human, feeling comfortable with a human	“I much prefer to talk to a human being…it just seems more personal…I just trust human beings I suppose and not a machine”. U082 “Well I prefer to speak to a nurse because I liked that interaction”. U039 “I’m not really comfortable talking to machines, I’d rather talk to a person… I know I can express myself”. U025 “you get to know the doctor…he knows your history, he knows exactly what you’re going to say… I feel comfortable…he listens and you know [I] come away feeling better just having seen him”. U035
		Inability to have a discussion with a clinician	Preference for discussion rather than more focused questions, inability to give detail, unsure if Dora can answer questions	“[I would have] preferred to actually have spoken to the gentleman… than just answering automated questions… if you want to discuss something you can’t talk to an automated voice really”. U031 “I couldn’t answer using the details I wanted to. I could only say yes or no“. U041 “For me it wasn’t difficult. I’m quite good at giving concise answers but for a lot of patients I think that would be, you know, quite a difficult thing for them to do”. U062 “I wasn’t sure how I would be able to interact if I had a question. So if I had a question that she hadn’t asked me… I didn’t quite know how she was going to react”. U084

Theme 1: I can see why you’re doing it…

The first theme extracted shows that patients have an inherent understanding of the utility of the innovation in the context of post-cataract follow-up surgery. Patients felt that Dora was efficient and could help the NHS as well as being more convenient for themselves. The convenience of not having to drive to the hospital and find parking spaces were shared concepts amongst some patients.

Theme 2: It went quite well actually…

This theme demonstrates how patient acceptability was affected by prior expectations and experiences with automated systems, or pre-call information. Most patients had broader acceptance of automation, and some felt prepared to speak to Dora simply based on the information provided prior to the call. A feeling of ‘moving with the times’ was also a common sentiment. Most patients felt Dora worked well for themselves, and cited ease of use. This was reflected in the high overall rates of acceptability from the TUQ scores. Some patients remained apprehensive about using new technology. Age was referred to as a perceived barrier, with some patients extrapolating their experience to suggest that it may be acceptable to them, but not to other users. A number of patients were apprehensive at the idea that there was not a clinician on the other end of the line.

Theme 3: I just trust human beings I suppose…

The third theme extracted was that Dora was referred to as a machine and compared directly to a human. Although patients found Dora convenient and simple to use, many of them stated they would prefer to speak to a human. Patients explicitly referred to missing human interaction. Some referred to the implicit nuances that human interaction provides such as trust and being able to express themselves. The inability to have a dialogue with a clinician who can answer their questions was the second subtheme extracted. Of these patients, even though Dora is able to understand more complex language, many felt they could not have a detailed conversation and were unsure if Dora would be able to answer their questions.

Discussion

This study examines patient acceptability of a conversational agent delivering a like-for-like replacement of a post-operative follow-up call after routine cataract surgery. Overall, we found high levels of patient acceptability across three acceptability measures of telephone usability, thematic analysis, and NPS scores.

We have previously assessed the patient acceptability of a nurse-led, telephone follow-up call for cataract surgery at this site [18]. Both studies used the TUQ questionnaire and demonstrated similar patterns for acceptability between Dora and nurse-led telephone follow-up. Similar to that of nurse-led follow-up, we found that patients, on average, rated Dora highly on ease of use, simplicity, and convenience due to time saved travelling to a hospital. Conversely, we also found that patients on average, neither agree nor disagree with the statement that the calls were the same as an in-person visit.

Beyond this cohort, a number of authors have already demonstrated high levels of patient acceptability in a nurse-led model of telephone follow-up for routine cataract surgery in a UK and East Asian population [5, 7, 19, 20]. In some of these studies, a significant number of patients actually preferred nurse-led telephone follow-up to in-person. TUQ has also been used in previous studies using the TUQ for human telephone follow-up in general surgery and head and neck cancer. Comparing the results of these studies to those in this study with Dora, they both show a similarly high level of acceptability across most domains [21, 22].

Through detailed thematic analysis in this study, we found that in common with nurse-led telephone follow-up of other surgical procedures, patients appreciated the convenience, availability, and accessibility of Dora. In contrast, our thematic analysis revealed that whilst many patients found Dora pleasant and easy to speak to, some expressed a preference to speak to a human clinician. Reasons cited included valuing the relationship with a clinician and trust. For conversational agents to be widely used, they need to be able to handle the complex needs and long-term monitoring of chronic conditions, which make up a significant portion of outpatient appointments. The ability of patients to form relationships and build trust through repeated interactions with automated systems needs to be further explored.

Finally, the NPS is a single metric focused on ‘recommendation’ of a healthcare service, and has some limitations in assessing acceptability [18]. However, its simplicity has led it, and adapted forms of it, to be used to assess patient satisfaction in a variety of care programmes [23–25]. The well-known NHS ‘Friends and Family Test’ is a simplified version of the NPS 21. Dora’s score of 45 is comparable to industry averages for smartphones (NPS:44) and laptop computers (NPS:43), and outperforms health insurance (NPS:13) and pharmacies/drug stores (NPS:28) [26]. Interestingly, NPS has also been found to correlate with validated patient-reported outcomes in other elective surgical use cases [27].

The findings of this study are contextualised against the backdrop of a number of wider trends. The COVID pandemic has catalysed the adoption of telemedicine, with patients’ and clinicians’ attitudes towards telemedicine changing across the healthcare sector [28]. Conversational assistants have also been shown as an increasingly promising tool to address some of the demand pressures facing health services [29, 30]. To date, most studies examining the role of conversational assistants in healthcare tend to focus on ‘supportive’ use cases such as supporting the elderly at home, prompting behavioural modification or medication adherence [31]. Reflecting this, actual adoption by healthcare institutions has largely been around sign-posting, providing healthcare tips and guidelines, or general support.

Within Ophthalmology, there is interest around voice as it represents a ‘natural’ user interface, and promises to be more accessible than a device or application which may be visually based [32]. This is particularly true for older patients who comprise the majority of cataract patients. This is a group often thought to be at risk of being digitally disenfranchised due to low degrees of internet access and smartphone ownership [33]. Recent research into smartphone-based home-vision monitoring for macular degeneration progression has even identified that increasing age is a risk factor for non-compliance with monitoring [34]. Overall, the results of this study are promising, especially given that this is a group of patients with a median age of 76. Whilst barriers to implementation remain, this paper suggests that telephone-based follow-up could represent a more accessible means of providing follow-up after routine cataract surgery.

Implications for implementation

Patient’s prior experiences with conversational agents, such as with banking systems were often cited as reasons for both positive and negative experiences. Similarly, how thoroughly participants read the patient information leaflets seemed also to affect their expectations of how easy it would be to use the systems. This was notable, as a number of patients remarked how ‘surprisingly’ simple it was to speak to Dora, highlighting the importance of robust post-op education in aligning patients’ expectations of the follow-up call [35]. Information could reassure patients around common concerns associated with conversational agents, such as—that no technology is needed, that the calls are reviewed, and that any concerns that cannot be addressed will be escalated to the clinical teams.

The importance of discharge education has also been emphasised by the recent nationwide push towards Patient-Initiated Follow Up (PIFU) [36]. Whilst PIFU has been shown in some patient groups to be a relatively safe model of care [37], evidence exists that especially in an elderly cohort, telephone-based interventions are appreciated by patients and may improve treatment adherence [38], or reduce anxiety [39]. Conversely, recent work in the NHS Enhanced Recovery After Surgery programme has shown that some patients want to avoid disturbing the staff, with the authors of this work suggesting that it is crucial that clinical teams initiate the phone call [40].

We suggest that pathways using autonomous conversational agents can provide a form of PIFU where asymptomatic patients are discharged. This also introduces predictability into post-op care pathways for capacity planning and could reduce unplanned burdens on emergency or community services. The conversations can also act as an accessible means for patient concerns to be captured and be a potential platform for clinical teams to gather feedback, refraction or other patient-reported outcome measure data. Looking beyond elective surgery follow-up, with virtual clinics and deep learning systems becoming more commonplace [41] and natural-language processing continuing to mature, such scalable communication channels will become increasingly important.

Limitations

One limitation of this study is the consent process’ susceptibility to selection bias. Although all suitable patients were offered a Dora call, it is possible that patients with queries or concerns may have been less likely to consent. The study protocol did not capture the percentage of the pathway that was consented, and the reasons for non-consent which will be crucial to understanding broader acceptability. Another limitation is that the consent process itself included the requirement for patients to speak to a human clinician for consent prior to speaking to Dora. This could have affected perceptions of the call and may not be representative of the fully autonomous interaction in a more ‘real-world’ condition. Finally, ethnic and socioeconomic data that have been shown to affect engagement with care were not collected which limits the generalisability of our findings [34]. Further work is currently being done as part of a prospective multi-site trial to explore acceptability, safety and real-world feasibility of the Dora system in a more diverse patient cohort [42].

Conclusion

Automation of routine healthcare tasks represents an opportunity to provide a reliable service for patients at their convenience. We demonstrate high levels of patient acceptability for an automated conversational voice agent for cataract surgery follow-up in this patient group.

Summary

What was known before

• Many new artificial intelligence–supported systems are being integrated into clinical care to support routine clinical tasks.

• Recent advances in natural-language processing have meant that conversational agents have the potential to soon automate clinical conversations.

• However, the patient acceptability of such conversational agents has not been examined in the context of cataract surgery follow-up—the highest volume elective procedure in the world.

What this study adds

• Follow-up with an automated telephone-based clinical conversational assistant is an acceptable form of follow-up for an elderly group of patients who have undergone routine cataract surgery.

• Patients understood the inherent value of such a service, namely convenience and ease of use, but missed speaking to a human clinician.

Author contributions

GM was the principal investigator of this study. EL, AH and GM supervised patient calls. SK conducted the patient interviews. SK, EL, AH and GM performed data collection, interpretation and analysis. SK authored the initial manuscript along with EL and AH who provided revisions for subsequent drafts. Figures by EL. All co-authors were actively involved in reviewing the manuscript.

Competing interests

At the time of this work, GM and NdeP were employees of Ufonia Limited. NdeP is a shareholder at Ufonia Limited. SK has been an employee of Ufonia Limited since August 2021. EL and AH were clinical fellows at Imperial College London and Oxford University Hospitals funded by an NIHR AI in Health and Care Award, of which Ufonia Limited is the grant holder. Since February 2022 both are Ufonia employees.

Supplementary information

The online version contains supplementary material available at 10.1038/s41433-022-02289-8.