AI‐Powered Chatbot as a Health Literacy Tool for Enhancing Oral Cancer Awareness: Expert Feedback

ABSTRACT

Artificial intelligence (AI)‐powered chatbots have emerged as potentially effective tools for delivering personalized, interactive, and culturally sensitive health education. Avoidable factors majorly cause oral and oropharyngeal cancers, and usually present with confusable symptoms, often leading to delayed diagnoses and poorer outcomes. Traditional oral health education methods have limitations in addressing the unique accessibility needs of groups that are most impacted by these outcomes. The primary aim of this study was to evaluate the usability and accuracy of an AI‐powered chatbot prototype as an approachable, trustworthy, and engaging educational resource to enhance oral cancer awareness. A mixed‐methods evaluation was conducted with six experts in behavioral science, oncology, dentistry, and AI. Experts interacted with the chatbot via a web interface and provided feedback via surveys rating usability and accuracy. Qualitative insights were gathered through open‐ended survey questions and analyzed using thematic analysis, and quantitative data were collected through REDCap. The experts provided positive and constructive feedback, recognizing the chatbot's potential as an educational tool aiming to improve oral cancer‐related information. They particularly praised its ease of access and the reliability of the information provided. However, experts identified areas for enhancement, including user interface changes, simplifying medical terminology, providing clearer initial guidance for users, and improving visual accessibility. Additionally, recommendations included integrating supplementary educational resources and clearer medical definitions to support deeper understanding and user engagement. This expert evaluation will guide the refinement for broader implementation in the project's next phase, evaluating acceptability and efficacy among non‐expert users.

Study Highlights

• What is the current knowledge on the topic?

  • ○
    Oral and oropharyngeal cancers rank as one of the top 15 most common malignancies worldwide, and survival is strongly tied to stage at diagnosis. Traditional educational materials (e.g., pamphlets) often fail to meet diverse literacy and personalized needs. AI‐powered chatbots have been used in multiple health education contexts, but evidence varies by setting and design.
• What question did this study address?

  • ○
    Can an AI‐powered chatbot, built on an open‐source large language model with a retrieval augmented generation framework and American Cancer Society content, provide consistent oral cancer‐related information with acceptable usability and informational accuracy based on expert evaluation?
• What does this study add to our knowledge?

  • ○
    The study identified four themes needed to be concentrated on that are critical to improve the usability of digital health education tools, which are system introduction, simplified terminology, multimedia integration, and enhanced accessibility. These insights provide researchers in similar fields with a blueprint of actionable design principles for developing and evaluating AI‐powered educational chatbots.
• How might this change clinical pharmacology or translational science?

  • ○
    By outlining a staged, expert‐informed process for developing AI based educational interventions, this work informs the development, design, and evaluation of next generation patient‐facing digital health tools. Documenting expert‐identified usability, safety, and accessibility considerations supports responsible translation of digital health innovations and technology‐driven educational tools into clinical and public settings and guides subsequent testing prior to broader implementation, in addition to providing guidance for similar future studies.

1. Introduction

Oral and oropharyngeal cancers are the 8th and 14th most common cancers by incidence and the 9th and 17th leading cause of cancer‐related deaths worldwide and in the U.S., respectively [1, 2, 3, 4]. Survival rates for oral and oropharyngeal cancers are significantly impacted by stage at diagnosis, with patients diagnosed with localized disease having roughly 1.3‐times the 5‐year survival of those with regional disease and about 2.4‐times the survival of those with distant disease, highlighting the benefit of early detection [5].

Major causes of delayed presentation include limited knowledge about the disease and how to spot symptoms, lack of resources, and low screening/check‐up rates [6, 7, 8]. Oral cancer screening in different communities in the U.S. is neither well surveilled nor researched [9]. Poverty, an indicator of low socioeconomic status, is theorized to mediate oral cancer outcomes through inadequate knowledge and attitudes and behaviors that promote risk factors associated with increased incidence and severity and delayed diagnosis [10]. Tobacco and alcohol, the most significant risk factors for oral cancer, account for approximately 75% of lip, oral cavity, and pharyngeal cancer cases in the U.S. [11, 12]. Smoking cessation reduces risk by 35% within 1–4 years and about 80% by 20 years [13]. These factors are avoidable, and improvements in health literacy through understanding the harms of these behaviors increases the likelihood of their successful cessation [14, 15].

Improving knowledge about oral cancer could reduce risk behaviors and increase screening rates, leading to lower incidence rates, earlier diagnosis, and improved survival. This can be done in various ways, but not without challenges. In the clinical setting through one‐on‐one interactions with healthcare providers, challenges include time constraints, implicit biases, and the inability to reach underserved populations [16]. Traditional educational methods like pamphlets have been used to disseminate health information. However, they are static and often fail to address the diverse literacy levels and specific needs of target populations. This limits their impact on knowledge and health behaviors, particularly in communities with lower health literacy levels [16, 17].

The growing field of artificial intelligence (AI) has opened new opportunities for improving health literacy. AI‐powered chatbots are interactive tools that can provide personalized, user‐oriented educational experiences, offering scalability, customizability, real‐time interaction, and adaptability to individual needs, making them a promising alternative to traditional approaches. Rule‐based chatbots have been used in various health domains, including cancer‐related applications, assisting in screening, prevention, treatment guidance, risk stratification, monitoring, lifestyle changes, patient support, and patient education, showing efficacy and a high degree of user satisfaction [18, 19, 20]. In a study, reviewers preferred ChatGPT answers to patient questions about 79% of the time, where they scored much higher for quality and empathy than human answers, suggesting chatbots can outperform clinicians at answering patient questions [21]. Despite their potential, few studies have explored their use in the context of oral cancer education. Moreover, many chatbots used in healthcare have relied on outdated or unregulated information sources, raising concerns about accuracy and reliability [22, 23].

This is the first phase of a 2‐phase project aiming to address critical gaps in knowledge by evaluating an AI‐powered chatbot as an educational tool for oral and oropharyngeal cancers. This phase focuses on evaluating the usability and accuracy of information provided by the chatbot prototype, while phase 2 will be a randomized controlled trial assessing the implementation metrics as well as the efficacy of this tool and comparing it to a pamphlet [24]. This project has the potential to expand health education strategies, reduce disparities, and improve outcomes.

2. Methods

2.1. Study Design

This study employed a mixed‐methods approach combining quantitative and qualitative methods to evaluate the usability and accuracy of an AI‐powered chatbot developed to provide health education about oral and oropharyngeal cancers. As illustrated in Figure 1, this phase focused on chatbot development and expert evaluation of usability and informational accuracy. Feedback from this phase will inform the refinement of the chatbot prototype used in Phase 2.

FIGURE 1.

Planned stages of the multi‐phase project this study is a part of. This study's phase is marked by a red square.

2.2. Tool Design and Technological Implementation

The chatbot leverages a publicly accessible web interface developed by “oobabooga” and stored in a GitHub repository [25]. On the backend, the chatbot is powered by Llama 2, an open‐source large language model (LLM) developed by Meta [26, 27]. Because LLMs can generate inaccurate responses, the system was designed to minimize hallucination risk and reduce the risk of misinformation through a Retrieval‐Augmented Generation (RAG) framework which enables it to retrieve relevant content from a predefined dataset. The code for the web interface was modified to accommodate the RAG module developed specifically for this project, and the modified version was stored in another repository [28]. For this study, the American Cancer Society's patient education materials titled “Oral Cavity (Mouth) and Oropharyngeal (Throat) Cancer” were chosen as the reference dataset [29, 30].

The RAG framework's design limits the chatbot's responses to information retrieved from the reference dataset based on cosine semantic similarity scores to the user's query. After retrieval, a secondary LLM reorders the retrieved content based on the relevance of the information before formulating responses in order to confirm their relevance to the user's query [31]. The model is instructed to answer only when relevant source content meets a similarity threshold, and if no suitable content is retrieved, the system is configured to decline responding and prompt the user to ask an on‐topic question.

Designed to be modular and adaptable, the chatbot supports various LLMs and datasets. For the purposes of this study, the chatbot was hosted locally on the Virginia Commonwealth University (VCU) High Performance Research Computing Center cluster that offers the hardware resources necessary to load the LLMs while complying with institutional protocols. Users accessed the chatbot through a secure URL using the publicly available interface without modifications.

2.3. Participants, Sampling, and Expert Panel Composition

Purposive sampling was used to recruit six expert participants affiliated with VCU. The expert evaluation panel comprised six experienced professionals from VCU, each contributing specialized knowledge essential to the chatbot's assessment. Experts representing behavioral science and oncology provided insights into patient education needs and clinical relevance. The dentistry expert offered expertise in oral cancer specifics and dental health literacy, providing feedback on information accuracy. Two experts with backgrounds in AI and health informatics assessed technological architecture, responsiveness, and adaptability, while the last expert provided their expertise in clinical research. Feedback was collected anonymously.

2.4. Evaluation Procedure

Experts accessed the chatbot via a secure link and engaged with it using queries of their own choice after being informed of the purpose of the chatbot. Following their interaction, participants completed an online evaluation survey hosted on REDCap [32, 33]. The survey used an adapted subset of the Guideline Compliance Scale, which is designed for human–machine interaction evaluation and composed of Likert‐scale questions with answers ranging from 1 (Very Poor) to 10 (Excellent) [34]. Items were selected for relevance to the current chatbot and to cover the two prespecified domains of interest: Usability (10 items; e.g., adequacy of features for the tool's goal, novice user usability, language appropriateness and consistency, text legibility, tool performance, and error handling) and Accuracy (3 items; e.g., relevance of answers, factual accuracy, and consistency across responses). These domains were chosen to reach the study goal of evaluating the tool for implementation for non‐experts. As the scale was adapted, internal consistency of the 2 domains was calculated using Cronbach's α. The usability domain's Cronbach's α was 0.90 while the accuracy domain's was 0.92. Qualitative feedback was collected through 4 open‐ended questions that explored subjective experiences, perceived efficacy, and suggestions for improvement.

2.5. Data Collection and Analysis

Quantitative and qualitative data were collected anonymously via REDCap. Quantitative analysis was conducted using R software (v.4.5.0), calculating descriptive statistics, including medians and interquartile ranges (IQR) for usability and accuracy measures [35]. A composite score for each measure was created by calculating the median of the scores given by the experts per question. Qualitative responses underwent manual thematic analysis. Two of the authors who had training and experience in qualitative analysis independently familiarized themselves with the data, developed a coding framework, and iteratively refined the codes through discussions between them. Data were organized into themes, and representative quotes were selected to provide contextual richness.

2.6. Ethical Considerations

The project was conducted for improving the quality and usability of the chatbot. As responses were collected anonymously and only to refine the tool, the project does not qualify as human subjects research. The main study aiming to assess the feasibility of the tool was deemed by VCU Institutional Review Board (IRB) to meet the criteria under 45 CFR 46.104(d) for exemption from IRB review under the IRB ID HM20030494.

3. Results

3.1. Quantitative Usability Evaluation

Quantitative analysis of usability revealed consistently satisfactory ratings across most assessed chatbot components, ranging from 2.5 to 9 out of 10 and receiving a composite median score of 6.65 (IQR = 2.25) (Figure 2). Direct and easy access to the chatbot using a web address sent to the experts through email received the highest median score of 9 (IQR = 2.75), while text and content legibility followed with a score of 8 (IQR = 3). Experts also gave high scores for the ability of the chatbot interface functions to meet user goals, the consistency of terms used by the answers provided by the chatbot, and the performance of the interface in terms of processing speed. Recommendations from quantitative findings suggested that further improvements were strongly needed for the interface's starting page, which received the lowest score of 2.5 (IQR = 4.75).

FIGURE 2.

Box plot showing the distribution of scores with the median given by the reviewers in response to the questions evaluating chatbot usability.

3.2. Quantitative Accuracy Evaluation

Quantitative assessments of accuracy indicated strong overall performance, with a total median composite score of 7.67 (IQR = 2.17) (Figure 3). Both accuracy of the answers provided by the chatbot and consistency of information provided across different questions received the highest composite score of 8 (IQR = 2.25 and 1.5, respectively). Feedback regarding relevance of the answers provided based on input questions was also positive, receiving a median score of 7.5 (IQR = 3.25).

FIGURE 3.

Box plot showing the distribution of scores with the median given by the reviewers in response to the questions evaluating chatbot accuracy.

3.3. Qualitative Thematic Analysis

A comprehensive thematic analysis of the qualitative expert feedback revealed four core themes: (a) System Introduction and Guidance; (b) Language and Terminology; (c) Content Accessibility and Educational Tools; and (d) User Interaction and System Responsiveness. Each theme provided important insights that are valuable for improving the design and utility of the chatbot.

System Introduction & Guidance emerged as a primary concern, with experts underscoring the importance of orienting users at the outset of their interaction. A consistent recommendation was to include a structured opening dialogue that clearly outlines the chatbot's purpose, capabilities, and intended use. One expert emphasized this need, stating: “The introduction chat needs to introduce the system and its purpose, as well as its domain‐specific knowledge and any instructions to the user. Also, consider providing example topics to give the user a direction as they may not know where to start,” while another recommended “having a paragraph of text at the top explaining parameters of kinds of questions the chatbot is designed to answer.” One expert explained this perspective by writing that “users don't know what they don't know so they might not be able to ask the right questions.” This feedback highlights the importance of onboarding in order to reduce user uncertainty and make utilization of the tool easier, particularly when engaging with new or complicated educational tools. Including suggested conversation starters or prompts was suggested as a helpful strategy to guide users who may be unfamiliar with chatbot interfaces or unsure how to initiate a conversation.

The second theme, Language & Terminology, pointed to the need for simplifying complex medical language. Experts strongly advised that all terminology be tailored to a lower reading level to ensure accessibility across a wide user base. The feedback stressed that technical terms should not be used without appropriate, user‐friendly definitions. As one expert stated, “[the chatbot] should not use terminology that it cannot define.” What prompted this feedback is that after the chatbot used the term “squamous cells” in its response to a question about what oral cancer is, it failed to provide a clear definition when clarification was requested by the expert. One expert suggested that the RAG database should include comprehensive information to define terms. This insight suggests a need for either improving the prompts received by the LLM in the backend or expanding the RAG database by including thorough definitions for commonly used terms.

Content Accessibility & Educational Tools was the third major theme identified. Experts advocated for the inclusion of multimedia elements such as images, videos, and diagrams to reinforce text‐based explanations, particularly in describing symptoms, treatments, or oral anatomy. Visual aids could be important for users with more limited health literacy, enhancing understanding through multiple learning modalities. Additional suggestions included integrating links to reliable health sources, enabling chat log downloads by the user for future reference, and offering printable or savable educational materials. These enhancements would support users in revisiting and sharing information outside the live chatbot interaction.

The last theme of User Interaction & System Responsiveness focused on ensuring the chatbot delivers a seamless, responsive, and inclusive user experience. Experts emphasized accessibility considerations, with suggestions to adopt a white background for better readability, offer adjustable text sizes, and incorporate audio features for users with visual impairments. These refinements would expand the target population and broaden the chatbot's reach. An issue with the conversational flow that was mentioned by an expert was the tool's confusion of which topics it can discuss, where “When asked it said it was a healthcare AI, however it knows nothing about diabetes or covid. Then I asked what healthcare topics it could tell me about and it said diabetes was one of them,” therefore affecting the smoothness of the conversation. A suggestion by another expert to improve user interaction through conversational flow is to add a feature that allows the chatbot to suggest following questions for users to ask.

4. Discussion

The study represents Phase 1 of a multistage development and evaluation process for an AI‐powered educational chatbot focused on oral cancer. The purpose of this phase was to obtain expert feedback on usability and accuracy prior to broader testing. Findings from this evaluation are being used to refine the chatbot interface, onboarding guidance, terminology, conversational flow, and accessibility features. Subsequent phases of this project will include evaluation among non‐expert users to assess usability, acceptability, and feasibility in comparison with a pamphlet.

AI‐powered chatbots have the potential to deliver accessible, independent, and interactive health information and services, significantly enhancing the reach and effectiveness of health education tools [36, 37]. This prototype was strongly praised for its ease of access, a fundamental determinant of the accessibility of this type of educational tool leading to improved intention to use and satisfaction [38, 39]. Additionally, perceived information quality and the ability to provide correct and customizable responses to user queries further enhance efficacy [38]. It was also commended for correctly addressing user queries and providing accurate content aligned with current knowledge based on the experts' answers to the quantitative question asking about this domain; however, qualitative analysis showed how there is room for improvement. Adding a clear introduction and explanation of the chatbot's purpose was proposed to enhance user experience and set user expectations, as recommended by the expert panel. When users first interact with a chatbot, a clear introduction that explains what the chatbot can and cannot do helps prevent confusion or frustration and guides users to ask relevant questions.

When it came to off‐topic queries, the chatbot received positive feedback about managing them. Several approaches have been recommended for managing off‐topic queries prompted by users to avoid confusion and misleading answers and ensuring that only on‐topic conversations proceed. Studies showed that a major limitation to chatbot acceptance is the user's expectations on what it can do, and by instructing it to clearly explain its capability to answer only within a defined topic and to politely decline off‐topic questions by providing polite fallback and clarification responses, this limitation can be mitigated [40]. For example, when asked an off‐topic question, the chatbot responds, “I can only answer questions about <topic>” thereby guiding users back to the intended subject area. When a chatbot handles off‐topic queries by acknowledging limitations and providing clear guidance, it builds trust and demonstrates competence, both of which enhance user satisfaction [41, 42]. This feature was concentrated on when developing the chatbot, and the use of RAG gave it the ability to avoid answering off‐topic questions.

Consistent accuracy of information across different responses was identified as a positive feature in the chatbot from the scores given by the experts. One common cause identified in the literature that prompted inconsistent responses in chatbots was semantically similar questions asked by users. One method proposed to tackle this before the emergence of LLMs was semantic similarity measures that quantify the relatedness of two text segments based on meaning rather than exact word matching, which is a great advantage brought on by the rise of LLMs as a tool for natural language processing (NLP) [43]. In addition, due to the RAG framework restricting information retrieval to a predefined dataset, consistency of information is maintained across prompts with similar semantic scores.

Enhanced user experience, being the main determinant of success of chatbots, was also achieved through AI‐chatbots that offered free‐flow conversations rather than constrained ones through providing non‐predefined answers in the user‐initiated conversation [44]. This particular area, however, was one that experts highlighted as an area in need of improvement in the prototype, where smooth conversational flow was impacted by the users not knowing what the next steps in the conversation should be and some discrepancies in the chatbot's responses to off‐topic prompts, which will be solved in future versions by suggesting prompts for the users to use to continue the conversation and by improving the chatbot persona to better handle off‐topic questions.

Previous studies have highlighted the need for AI‐chatbots to have the ability to provide answers to users based on their background, which helps establish appropriate rapport to improve user engagement [44, 45]. This is a major benefit of using LLMs for NLP, as they offer the flexibility of easily customizing the interaction based on user attributes. While this prototype was not integrated with the ability to consider user attributes, a brief questionnaire collecting information such as age, educational background, and literacy can be used to optimize the customization of every conversation for different users.

Adding accessibility features is a crucial step in enabling the chatbot to ensure equal access by individuals of all age groups and capabilities. In the past few years, it has been shown that the elderly have been more and more engaged with technology, with 63% of adults aged 65 years and older using the internet to obtain health information [46]. Additionally, according to the World Health Organization, 16% of the world's population experience significant disability and require certain accessibility features to be able to connect digitally [47]. Such accessibility features could include visual aids, text‐to‐speech or speech‐to‐text, and font and color adjustments.

Another feature that is essential to ensure accessibility by individuals from different backgrounds and age groups is the chatbot's ability to simplify medical terminology, at which this prototype did not perform perfectly. It is especially important as simplifying medical terminology addresses differences in health literacy, making healthcare information more accessible to people of different ages, education levels, and backgrounds [48]. This is especially valuable for vulnerable populations who may otherwise struggle to understand complex medical concepts. AI chatbots that translate or simplify medical language significantly improve patients' understanding of their diagnoses, treatment options, and medical reports, as shown in studies evaluating AI models like ChatGPT‐4 [48]. Patients become more actively engaged in their care and are more likely to follow medical advice when presented with their medical information in simpler language [49].

Chatbots have the potential to complement healthcare services by reducing educational demands and addressing resource constraints within healthcare systems. They also have the potential of making information that is already available more accessible by the community by simplifying complex concepts or even the readability of educational materials, as shown in previous studies [50]. While these implications were not evaluated in the current phase, they provide important context for future development.

Building trust is a critical requirement for the adoption of AI powered educational tools in healthcare settings. Trust can be supported through governance of data sources and consistent testing of whether outputs align with the reference materials. In this project, trust was reinforced through the use of a RAG framework that constrained responses to curated content and through explicit refusal to answer queries outside the defined domain based on the numerical similarity scores between the users' question and the pulled pieces of information. While these procedures do not fully eliminate the risk of hallucination, they are one step closer to reaching more transparent and explainable AI‐powered tools.

4.1. Limitations

While this study is, to our knowledge, the first to assess expert feedback on an LLM‐powered chatbot using RAG as an educational tool, several limitations should be acknowledged. Some technical limitations include the small size of the document used to retrieve relevant information using RAG, resulting in the inability of the chatbot to answer some questions. However, this was an intentional choice, as the same document will be used for the second phase of the project, maintaining the amount of information similar between study arms. The RAG framework designed for this project can use bigger datasets than the one used for this study.

The expert panel consisted of a small, purposively sampled group from a single academic institution, which limits the generalizability of findings. Additionally, while expert feedback provided valuable insights, future research involving end‐users from target populations will be essential to fully evaluate the chatbot's effectiveness and usability in real‐world settings. Finally, as the chatbot is still in the prototype phase, certain functionalities and response capabilities are in ongoing development, which may have influenced expert evaluations. Some capabilities suggested by the experts, such as visual and auditory aids, could be difficult to apply to the chatbot as they necessitate the utilization of extensive resources.

5. Conclusion and Future Work

This study presents the design, implementation, and expert evaluation of an LLM‐powered chatbot that uses a RAG framework to provide information about oral cancer. Among six experts, the chatbot received positive usability and accuracy scores with the biggest strengths being ease of access, performance, and consistency of terms and answers. Qualitative analysis of the feedback revealed major areas needing improvement, including the need for clearer onboarding to guide users, medical terminology, and incorporation of visual aids. These findings, while demonstrating the promise of LLM‐powered chatbots, highlight important modifications that are necessary to improve scalability and acceptability of the tool and will guide the changes for the next stage of the project. Future research comparing the quality of responses generated by different NLP techniques including other LLMs or Small Language Models (domain‐specific models that use less resources) is needed and could make similar tools more scalable.

Author Contributions

N.A., A.M.N., R.R., A.L.O., and T.S.G. wrote the manuscript. N.A. and T.S.G. designed the research. N.A. performed the research. N.A. and A.M.N. analyzed the data.

Funding

This project and its subsequent phases are in part supported by the American Association for Cancer Education through the Research, Education, Advocacy, and Direct Service READS Grants Program. Additional support was provided in part by the Clinical and Translational Science Award program through award No. UM1TR004360 from the National Center for Advancing Translational Sciences. The study's content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.

Conflicts of Interest

The authors declare no conflicts of interest.

Abosamak N., Namoos A. M., Ramadan R., Olex A. L., and Gal T. S., “ AI‐Powered Chatbot as a Health Literacy Tool for Enhancing Oral Cancer Awareness: Expert Feedback,” Clinical and Translational Science 19, no. 2 (2026): e70499, 10.1111/cts.70499.