Abstract
This study evaluates patient engagement and satisfaction with Everyday Medical Monitoring Ally (E.M.M.A), a purpose-trained artificial intelligence (AI)-powered health coach delivered via WhatsApp. E.M.M.A integrates AI-driven symptom tracking, personalized medical information, and lifestyle guidance to support eye health management in a real-world clinical setting. This pilot study enrolled patients from a UK-based ophthalmology clinic managing chronic eye conditions. Patient interactions with E.M.M.A were logged over a 2-month period. Data collected included chat frequency, temporal patterns, and interaction modality (text or audio). A post-study satisfaction survey was conducted. Human validation of AI-generated question–answer pairs was performed independently by two glaucoma consultants. Ninety-one patients consented to participate, of whom 83 interacted with E.M.M.A, generating 446 analyzed chat sessions. Audio interactions accounted for 28% of chats. A subgroup of 39 users (42%) generated nearly 75% of all chats, while 52% of users were repeat users. Most patient queries related to general eye health and symptoms, comprising 39% of questions. Symptom-related content accounted for 26% of response classifications. The satisfaction survey was completed by 66 users, with 88% reporting being satisfied or very satisfied. A total of 65% would definitely recommend E.M.M.A, while 9% reported mild to moderate anxiety during use. Human expert validation scores exceeded 95% across all assessed domains. Qualitative feedback indicated acceptance of E.M.M.A as a reassuring and valuable adjunct when immediate clinical support was unavailable. AI-driven health coaches such as E.M.M.A may help bridge gaps in ophthalmic care between clinical visits by providing continuous, personalized, and medical record-informed patient support.
Keywords: AI Health coach, ophthalmology, self-management
Artificial Intelligence (AI) technologies are increasingly transforming healthcare by supporting patient self-management, particularly in chronic conditions. In ophthalmology, AI has demonstrated potential in disease forecasting,[1] diagnostics, patient education,[2] and clinical decision support.[3] Most existing research has focused on theoretical applications or clinician-facing tools, with limited evaluation of AI-powered patient-facing solutions.[4] The adoption of generative AI in healthcare has also raised concerns regarding safety and efficacy,[5] including dissemination of misinformation,[6] inappropriate emotional responses,[7] and hallucinations or incomplete answers,[8] all of which may adversely affect patient trust and safety.
The Everyday Medical Monitoring Ally (E.M.M.A) is an AI-powered health coach designed to support patients in managing chronic ophthalmic conditions. Delivered via WhatsApp, E.M.M.A integrates AI-driven symptom tracking, personalized medical information, and lifestyle guidance to provide continuous support between clinical visits. By leveraging advanced large language models (LLMs), E.M.M.A offers real-time, tailored responses and actionable insights to enhance treatment adherence and patient understanding.[9] This study evaluates patient engagement with E.M.M.A and overall satisfaction in a real-world clinical setting while addressing key risks associated with LLM use in healthcare.
Methods
Study population
Patients attending a UK-based general ophthalmology clinic were invited to participate if they had at least one consultation with a UK ophthalmology consultant. General ophthalmology clinics in the UK primarily manage chronic conditions; urgent cases and pediatric patients are managed separately and were excluded. Included diagnostic categories were glaucoma, dry eye disease, and age-related macular degeneration. Prior clinic letters were uploaded into E.M.M.A to enable personalized responses. The study adhered to the Declaration of Helsinki, received approval from the Worcestershire Glaucoma Clinic Ethics Committee, and obtained informed consent from all participants.
E.M.M.A integration
E.M.M.A provided AI-driven support through WhatsApp, offering symptom tracking, personalized health information, and lifestyle coaching. Patients interacted with E.M.M.A through text or voice-based chats. A Retriever-Augmented Generation (RAG) framework powered by state-of-the-art language models (LLMs) was used to address patients’ questions. This framework allowed matching the patient’s questions with the curated data and selected the most relevant subset of the curated data useful to answer the user’s question. The curated data included a corpus of medical articles approved by the medical domain experts. The relevant subset was provided to the AI model along with the prior conversation as context to generate an appropriate response to the user’s question. Further, once the response is generated, two sets of guardrails were used to evaluate and modify (if necessary) the generated response. Guardrails included another base AI model to detect any hallucination and evaluate the generated response for relevancy, used alongside hard coded guardrails ensuring E.M.M.A’s response remains informational, avoiding deep clinical advice.
An agentic workflow was used, involving more than one AI base model each working on narrow tasks in collaboration to accomplish the overall goal. The conversation data collected through the pilot project helped to improve the overall workflow including curated data, prompts, guardrails definitions, content, and tone in the responses, as opposed to altering the base AI models.
Data collection
Patient interactions with E.M.M.A were recorded over 2 months. Data included chat frequency, temporal patterns, interaction modality (text or audio), and question–answer content. All interactions were reviewed and categorized. A satisfaction survey was distributed at study completion, assessing overall experience, quality of information, and likelihood of recommending E.M.M.A. The questionnaire included numerical ratings (from 1 to 10), Likert-scale items, and free-text responses.
Data analysis
Patient queries were categorized using topic modeling into five domains: (1) general eye health and symptoms, (2) surgery and procedures, (3) eye drops and treatments, (4) glaucoma and medication, and (5) lifestyle and diet. Responses were similarly categorized across symptom-, disease-, drug-, lifestyle-, and procedure-related domains, with responses permitted to span multiple categories.
After excluding greetings, 321 question–answer pairs were independently evaluated by two glaucoma consultants across accuracy, completeness, and empathy, each scored on a 5-point Likert scale. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Statistical analyses were performed using Python-based tools. A chat was defined as a topic-specific exchange, and repeat users were those returning after a minimum 24-hour interval.
Results
Patient engagement
A total of 110 patients were invited, of which 91 accepted and 83 unique patients interacted with E.M.M.A over a period of 2 months, generating 507 chats. The mean age of the patients was 74.8 years, and the diagnoses were 80% (%) having primary open angle glaucoma, 60% had dry eye, 30% had age-related macular degeneration, and 20% had ocular hypertension. After removing symptom trackers and image submissions, 446 chats were analyzed. Of the 446 chats, 141 were audio interactions (28%) and 23 users out of 83 (28%) engaged in audio chats. The median response time of E.M.M.A across 446 chats was 7 seconds, which coincides with normal human conversation response rates. 39 (42%) patients were responsible for 75% of the chats, indicating a core group of highly engaged users, while another 45% engaged in 204 chats [Fig. 1]. On temporal analysis, it was determined that 43 (52%) of 83 users were repeat users; that is, they revisited E.M.M.A after a break of 24 hours to either continue their chat or initiate a new chat. The average chat duration was 8 minutes and 20 seconds. The average number of questions asked was 9 per chat.
Figure 1.
Chat engagement with AI assistants by patients with ophthalmic conditions
Topic modeling and expert validation
A majority, 179 questions (39%), were related to general eye health and symptoms such as “Why does daylight make everything a lighter shade?” A quarter (25%, n = 127) of the questions were about glaucoma and medications, 67 questions (15%) were about eye drops and treatments, and 55 (12%) about surgery and procedures [Fig. 2]. Lifestyle and Diet-related questions constituted a minority being only 7% of all.
Figure 2.
Topic modeling questions asked to AI assistants by patients with ophthalmic conditions
Topic modeling responses to questions by E.M.M.A AI yielded a total 713 total classifications as each response to a question can fall in more than one response category [Fig. 3]. Analysis revealed that the majority of 185 responses (26%) were related to symptoms, 143 (20%) were related to disease, 131 (18%) to drugs, 130 (18%) to lifestyle and diet, and 124 (17%) related to procedures. These included guidance on recognizing and responding to acute symptoms, such as pain, vision loss, or ocular redness (e.g. “If you experience severe pain, vision loss, or increased redness, contact your ophthalmologist immediately.”), focusing on managing comorbid conditions and understanding the impact of systemic diseases on eye health, medication adherence and safety, preventive care through healthy behaviors (e.g. “Maintain a healthy lifestyle, including a balanced diet and regular exercise, which benefits overall eye health.”), and explaining common ophthalmic procedures and postoperative care.
Figure 3.
Topic modeling of responses given by AI assistants to patients with ophthalmic conditions: The number of responses for answer categories is greater than the corresponding question categories, while answer categories do not directly align with question categories. Although the number of answers matches the number of questions, answers fall into multiple categories (as sorted by the independent reviewers), which is one reflection of the holistic answers given by E.M.M.A
A total of 321 question–answer pairs generated by E.M.M.A were independently evaluated by two glaucoma specialists using a 5-point Likert scale (1 = poor, 5 = excellent) across accuracy, completeness, and empathy. Inter-rater reliability was assessed using ICCs. ICC values ranged from 0.82 to 0.91, which indicated a high level of agreement between reviewers. The mean expert ratings were high across all domains: accuracy 4.82 ± 0.52 (95% CI: 4.76–4.88), completeness 4.87 ± 0.45 (95% CI: 4.82–4.92), and empathy 4.96 ± 0.20 (95% CI: 4.94–4.98), indicating precise and consistent evaluations.
A post-hoc power analysis was performed for the expert validation of E.M.M.A’s responses. For the 321 independently rated question–answer pairs, the mean scores for accuracy (4.82 ± 0.52), completeness (4.87 ± 0.45), and empathy (4.96 ± 0.20) were substantially higher than the neutral midpoint of the 5-point Likert scale (score = 3). Using a one-sample t-test framework and the observed effect sizes (Cohen’s d ranging from approximately 3.5 to 9.8 across domains), the achieved statistical power exceeded 99% (α = 0.05) for all three dimensions. This indicates that the study was more than adequately powered to detect high expert ratings relative to a neutral benchmark.
Patient satisfaction
The end of study survey was completed by 66 out of 83 users (response rate = 82%). Overall, 32 (49%) were very satisfied with using E.M.M.A, 26 (39%) were satisfied, and 8 (12%) were neither satisfied nor dissatisfied. As to the likelihood of recommending E.M.M.A, 43 (65%) responded as definitely, 21 (32%) as probably, and only 2 (3%) as probably would not recommend. Little to moderate anxiety while using E.M.M.A was experienced by 6 (9%).
Qualitative summary analysis
This qualitative summary is intended to provide contextual insight into patient experiences and was not designed as a formal qualitative analysis with quantified theme frequencies.
A qualitative content analysis of patient feedback reveals a nuanced view of the AI system E.M.M.A as a supplementary tool in the self-management of chronic eye conditions, particularly glaucoma. Across the responses, several key themes emerged, reflecting the diversity of patient experiences and perspectives. These included:
Supplementary Role in Healthcare: Patients widely viewed E.M.M.A as a supportive adjunct to traditional healthcare, particularly between specialist visits. E.M.M.A was frequently used to ask follow-up questions and address concerns that arose outside of clinical settings, such as at home or when trying to recall or ask follow-up questions after a medical appointment. Additionally, E.M.M.A was seen as a helpful resource when new concerns arise or when patients are influenced by information from external sources.
Accessibility and Inclusivity: While some patients were experienced in seeking evidence-based information through platforms like TRIP or reputable websites, others highlighted their self-perception of lacking skills and confidence to do so. In this context, E.M.M.A was seen to bridge this gap, offering accessible support for those without formal training in health information seeking.
Communication Modality Preferences: There was a clear contrast between the use of verbal and text input. Initially attempting to engage E.M.M.A through speech, the patient reported limited understanding and delayed response times. As a result, they shifted to text-based interaction, which proved to be more effective and aligned with their communication preference. Delays in voice responses can disrupt conversational flow, increase cognitive load, and lead to frustration, especially among older adults who may have reduced working memory or hearing-related challenges. Similarly, inaccuracies or instability in speech recognition can disproportionately affect elderly users due to age-related changes in speech patterns, accent variation, or lower vocal intensity. This finding suggests a potential area for system improvement in speech recognition and processing.
Quality and Safety of Responses: Patients expressed a high level of satisfaction with the quality and depth of E.M.M.A’s responses. Answers were described as comprehensive and well-considered. A particularly valued feature was the frequent safety-netting, in which E.M.M.A reminded users of appropriate next steps, including when to seek direct consultation with a healthcare professional. This feature enhanced the user’s sense of reassurance and safety.
Usage Frequency and Expectations: Although many patients used E.M.M.A infrequently, reporting around four interactions over the course of 2 months, they still found the tool beneficial. This suggests that low-frequency use can offer meaningful support, particularly for chronic, stable conditions that do not require constant monitoring.
Recommendations for Evaluation: Patients supported the continued development and evaluation of AI in healthcare. Patients were more comfortable with a staged approach to research: beginning with questionnaire-based studies to capture user experiences and perceptions, followed by long-term evaluations to assess clinical outcomes. This reflects a patient-endorsed call for evidence-based integration of AI tools in healthcare.
The collective patient feedback portrays E.M.M.A as a valuable and reassuring tool for managing chronic eye conditions, particularly when clinical support is not immediately available. While technical limitations were noted in verbal communication, text-based use was preferred by many and provided clear, detailed responses. E.M.M.A’s emphasis on safety and professional referral was seen as a strength, and patients advocated for continued evaluation to determine the AI’s broader clinical value and limitations.
Some sample comments from E.M.M.A users
“I think E.M.M.A is a wonderful idea you can ask a question and get an instant response.”
“Useful to have audio questions and answers for those with poor eyesight.”
“It’s a great service. I would definitely continue to use!”
“I have not used an AI tool before, but it’s clearly a very useful way of getting quick responses to straightforward questions.”
“Prefer to have contact with a real person.”
Discussion
This study demonstrates the success of E.M.M.A in patient engagement and alludes to the future potential of use of patient-facing AI devices, particularly with enhancing the self-management of ophthalmic patients. In this study, 83 unique patients engaged in a total of 446 analyzed chats, underscoring the demand for real-time, accessible health support beyond clinical visits. The engagement metrics reveal a subset of highly active users—42% of participants generated 75% of the chats—indicating that E.M.M.A was particularly effective for patients requiring ongoing support. This finding suggests that AI-powered tools such as E.M.M.A may support patient engagement in their healthcare journey, particularly in the context of chronic conditions that require ongoing conservative management.
Expert Validation results were particularly compelling, demonstrating E.M.M.A’s efficacy in providing accurate and empathetic responses. Use of the retrieval-augmented generation framework was appropriate for this health coach in addressing and preventing hallucinations.[9] This framework’s use was combined with a thoroughly trained retriever encoder to improve the quality of outputs,[10] which was rated highly by reviewers.
A total of 321 question–answer pairs were reviewed by two independent glaucoma specialists. The average scores across all three dimensions—accuracy (4.82 ± 0.52), completeness (4.87 ± 0.45), and empathy (4.96 ± 0.20)—were notably high. These validation scores affirm that E.M.M. A’s responses are not only factually correct but also comprehensive and empathetically communicated. The importance of empathy in healthcare interactions, particularly when mediated by AI, cannot be overstated as it contributes to building trust and improving patient satisfaction. The nearly perfect empathy score highlights the potential of AI to mimic the nuanced communication that patients often seek in clinical settings.
The patient satisfaction survey also underscores the effectiveness of E.M.M.A in delivering a positive user experience. Among the 68 respondents (82% response rate), 88% of patients reported being satisfied or very satisfied with E.M.M.A, while 65% stated they would recommend E.M.M.A, and a further 32% would probably recommend it. These results indicate a high level of acceptance and trust in the AI-based platform, reinforcing its role as a reliable adjunct to traditional care. Notably, only 9% of respondents experienced any degree of anxiety while using E.M.M.A, suggesting that, for most patients, interacting with AI posed little to no psychological discomfort. A systematic review of eight trials of AI-powered chatbot intervention for managing chronic diseases has also shown favorable acceptance by users.[11]
Chat modality preferences also provide interesting insights into how patients interact with E.M.M.A. While text-based chats constituted most of the interactions, 28% of users engaged with E.M.M.A via audio, indicating the importance of multimodal communication for accessibility, particularly in a population dealing with visual impairments. This flexibility in engagement styles likely contributed to the high satisfaction ratings as patients could choose their preferred method of interaction. Voice interface of LLMs has been identified as being potential to help individuals with visual impairments to obtain health information through conversation.[12]
The repeat user rate of 52% further highlights the platform’s utility as more than half of the patients returned to E.M.M.A after an initial interaction. This statistic, combined with the high percentage of users generating multiple chats, suggests that E.M.M.A effectively addresses ongoing patient needs, thus supporting sustained patient engagement and management of ophthalmic conditions.
There is a scarcity of ophthalmology educators, and conventional educational approaches are limited in catering to personalized patient needs. LLMs have been reported to be helpful in enhancing understandability of medical reports, enhancing patient understanding of online material, and can facilitate patient comprehension, and answers have been rated high on empathy.[13,14,15,16] Tools like E.M.M.A go above and beyond conventional LLMs by giving a more personalized experience. A systematic review of LLMs in ophthalmology reported variable performance across subdomains in responding to patient queries with suboptimal responses to certain conditions like vernal keratoconjunctivitis and lacrimal drainage disorders and a lack of studies exploring AI-powered chatbots for general ophthalmological conditions was apparent with disease diagnosis and report interpretation comprising most use cases. It also flagged ethical concerns, especially data hallucination.[13] Our study adds to the body of evidence supporting the use of AI-powered personal health coach within built guardrails to prevent incorrect responses tailored to ophthalmological practice.
Limitations
Through its success with improving patient engagement, the E.M.M.A health coach gives reason to be optimistic about the future potential of using AI as a medium for appointment pre-preparation, patient monitoring, surgical prehabilitation, and rehabilitation. The absence of a control or comparator group limits interpretation of engagement and satisfaction outcomes, although this is acceptable for a pilot study. Additionally, the small sample size, single-center design, and focus on an elderly UK-based cohort restrict the generalizability of the findings.
Further study including randomized trials is required to comment on whether patient-facing AI tools like E.M.M.A can be used to improve clinical and patient reported outcomes. The qualitative summary analysis was exploratory in nature and did not include formal coding with quantification of theme frequencies, which limits the ability to assess the relative prominence of individual themes. As such, the qualitative findings should be interpreted as descriptive and hypothesis-generating, and future studies should incorporate structured qualitative methodologies with quantified thematic analysis. Further study is also required over a longer period of study to assess for long-term continued use and with a more diverse study population in a variety of clinical environments to explore and help define the broader role of patient-facing AI devices in the healthcare setting.
Conclusions
This study demonstrates that E.M.M.A is an effective AI-powered health coach for supporting patients with chronic ophthalmic conditions. High expert validation scores and strong patient satisfaction indicate that E.M.M.A delivers accurate, empathetic, and reliable information. AI-driven tools such as E.M.M.A may help bridge gaps in care between clinical visits, offering scalable, patient-centered support. Further research in diverse populations with bigger sample sizes and longer-term studies is warranted to establish their role in improving patient-reported and clinical outcomes.
Conflicts of interest
There are no conflicts of interest.
Funding Statement
Nil.
References
- 1.Huang X, Raja H, Madadi Y, Delsoz M, Poursoroush A, Kahook MY, et al. Predicting glaucoma before onset using a large language model Chatbot. Am J Ophthalmol. 2024;266:289–9. doi: 10.1016/j.ajo.2024.05.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Eid K, Eid A, Wang D, Raiker RS, Chen S, Nguyen J. Optimizing ophthalmology patient education via ChatBot-generated materials: Readability analysis of AI-generated patient education materials and the american society of ophthalmic plastic and reconstructive surgery patient brochures. Ophthalmic Plast Reconstr Surg. 2024;40:212–6. doi: 10.1097/IOP.0000000000002549. [DOI] [PubMed] [Google Scholar]
- 3.Tailor PD, Dalvin LA, Chen JJ, Iezzi R, Olsen TW, Scruggs BA, et al. A comparative study of responses to retina questions from either experts, expert-edited large language models, or expert-edited large language models alone. Ophthalmol Sci. 2024;4:100485. doi: 10.1016/j.xops.2024.100485. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Biswas S, Davies LN, Sheppard AL, Logan NS, Wolffsohn JS. Utility of artificial intelligence-based large language models in ophthalmic care. Ophthalmic Physiol Opt. 2024;44:641–71. doi: 10.1111/opo.13284. [DOI] [PubMed] [Google Scholar]
- 5.Reddy S. Generative AI in healthcare: An implementation science informed translational path on application, integration and governance. Implement Sci. 2024;19:27. doi: 10.1186/s13012-024-01357-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Monteith S, Glenn T, Geddes JR, Whybrow PC, Achtyes E, Bauer M. Artificial intelligence and increasing misinformation. Br J Psychiatry. 2024;224:33–5. doi: 10.1192/bjp.2023.136. [DOI] [PubMed] [Google Scholar]
- 7.Mlonyeni P. M. Personal AI, deception, and the problem of emotional bubbles. AI Soc. 2024;40:1927–38. [Google Scholar]
- 8.Hatem R, Simmons B, Thornton JE. A call to address AI “Hallucinations” and how healthcare professionals can mitigate their risks. Cureus. 2023;15:e44720. doi: 10.7759/cureus.44720. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Choudhury A, Chaudhry Z. Large language models and user trust: Consequence of self-referential learning loop and the deskilling of health care professionals. J Med Internet Res. 2024;26:e56764. doi: 10.2196/56764. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Zhang G, Xu Z, Jin Q, Chen F, Fang Y, Liu Y, et al. Leveraging long context in retrieval augmented language models for medical question answering. NPJ Digit Med. 2025;8:239. doi: 10.1038/s41746-025-01651-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare. 2025;13:603. doi: 10.3390/healthcare13060603. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kurniawan MH, Handiyani H, Nuraini T, Hariyati RTS, Sutrisno S. A systematic review of artificial intelligence-powered (AI-powered) chatbot intervention for managing chronic illness. Ann Med. 2024;56:2302980. doi: 10.1080/07853890.2024.2302980. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Yang Z, Wang D, Zhou F, Song D, Zhang Y, Jiang J, et al. Understanding natural language: Potential application of large language models to ophthalmology. Asia Pac J Ophthalmol. 2024;13:100085. doi: 10.1016/j.apjo.2024.100085. [DOI] [PubMed] [Google Scholar]
- 14.Wong M, Lim ZW, Pushpanathan K, Cheung CY, Wang YX, Chen D, et al. Review of emerging trends and projection of future developments in large language models research in ophthalmology. Br J Ophthalmol. 2024;108:1362–70. doi: 10.1136/bjo-2023-324734. [DOI] [PubMed] [Google Scholar]
- 15.Strika Z, Petkovic K, Likic R, Batenburg R. Bridging healthcare gaps: A scoping review on the role of artificial intelligence, deep learning, and large language models in alleviating problems in medical deserts. Postgrad Med J. 2024;23(101):4–16. doi: 10.1093/postmj/qgae122. [DOI] [PubMed] [Google Scholar]
- 16.Chiam J, Lim A, Nott C, Mark N, Teredesai A, Shinde S. Co-pilot for health: Personalized algorithmic AI nudging to improve health outcomes (arxiv.org; abs) Computer Science: Human-Computer Interaction. doi: 10.48550/arXiv.2401.1. [Google Scholar]