Supporting patient understanding of cervical and ovarian cancer: how well does AI perform?

Abstract

Background

As the public increasingly interacts with artificial intelligence (AI) chatbots, we compared the answers from five AI chatbots to standardized questions that patients might ask about ovarian and cervical cancer.

Methods

ChatGPT 3.5, Google Gemini 2.0, Reddit Answers, Bootcamp, and DeepSeek were queried with 15 frequently asked questions (FAQs) on cervical cancer and 11 on ovarian cancer. In a blinded, randomized survey, each deidentified response was independently assessed by three gynecologic oncologists using a 4-point scale (1, accurate and comprehensive; 2, accurate but inadequate; 3, accurate but outdated or inaccurate; 4, completely inaccurate). Readability (Flesch-Kincaid grade level) and word count were recorded.

Results

For cervical cancer FAQs, ChatGPT 3.5, Google Gemini 2.0, Reddit Answers, Bootcamp, and DeepSeek received scores of 1.4, 1.6, 2.7, 1.5, and 1.5, respectively. For ovarian cancer FAQs, average scores were 1.3, 1.4, 2.5, 1.2, and 1.4, respectively. All AI chatbot responses were written at a reading level above 11th grade, making them generally difficult for the average American to read.

Conclusions

Although generally rated as accurate and adequate, the answers were frequently off-topic and not generalizable. Healthcare providers should be aware of unintentionally generated misinformation to better counsel patients.

Patients increasingly use artificial intelligence (AI) chatbots to seek health information, including cancer-related guidance. As with traditional patient education materials, clinicians should be aware that chatbot-generated information may not be written at a reading level that most Americans can easily understand. Prior studies have repeatedly shown that patient education materials are often written well above recommended reading levels, which can limit comprehension and widen disparities.^1–3

In addition, incorrect or misleading health information, whether from online platforms or AI-generated responses, can influence patients’ beliefs and health decisions. Studies have documented the prevalence and impact of online health misinformation, including effects on decision-making and health-related attitudes.^4–6 Despite these concerns, AI chatbots may offer benefits, including rapid, on-demand access to general health information and assistance in navigating the healthcare system. However, evidence supporting clinical effectiveness is still evolving, and limitations require careful evaluation before these tools are relied upon in routine patient care.⁷

This study aimed to evaluate the quality and educational utility of responses from five AI chatbots to questions about cervical and ovarian cancer by assessing their accuracy, comprehensiveness, and readability. To our knowledge, this is the first study to compare five different AI chatbots as well as the readability of the generated answers.

METHODS

This study was modeled after the study of McGrath et al (2024), which compared genetic counseling responses between the paid and free versions of ChatGPT.⁸ We evaluated AI-generated responses to the American College of Obstetricians and Gynecologists’ (ACOG) frequently asked questions (FAQs) on cervical and ovarian cancer. Fifteen cervical cancer questions and 11 ovarian cancer questions were selected (Table 1).

Table 1.

Frequently asked questions on cervical and ovarian cancer as listed by the American College of Obstetricians and Gynecologists

Cancer	Questions
Cervical	What is cervical cancer? How long does it take for cervical cancer to develop? What is the main cause of cervical cancer? Does HPV always lead to cervical cancer? How can I prevent HPV infection? When should people get the HPV vaccine? Who is at risk of cervical cancer? Is there a screening test for cervical cancer? What are some of the symptoms of cervical cancer? How is cervical cancer diagnosed? How is invasive cervical cancer treated? What is involved in surgical treatment for cervical cancer? What is involved in radiation therapy for cervical cancer? What is involved in chemotherapy for cervical cancer? Is special follow-up required after treatment?
Ovarian	What is ovarian cancer? What is the most common type of ovarian cancer? What are the risk factors for ovarian cancer? How does age affect my risk of ovarian cancer? What are the symptoms of ovarian cancer? Is there a screening test for ovarian cancer? How is ovarian cancer diagnosed? How is ovarian cancer treated? What type of follow-up is needed after treatment? How can I reduce my risk of ovarian cancer? What should I know if I am at high risk of ovarian cancer?

Each question was entered into five platforms: ChatGPT, Google Gemini, Reddit Answers, Bootcamp, and DeepSeek. Questions were entered verbatim and without follow-up or prompting. A new session was used for each question to minimize the influence of prior prompts on subsequent answers. All responses were then deidentified, randomized, blinded, and independently scored by three practicing gynecologic oncologists. Responses were rated using a 4-point accuracy scale (1 = accurate and comprehensive; 2 = accurate but inadequate; 3 = mixed accurate and inaccurate/outdated; 4 = completely inaccurate). Ratings reflected each reviewer’s assessment of clinical accuracy, completeness, and applicability to patient counseling.

Readability was assessed using the Flesch-Kincaid grade level (FKGL) in Microsoft Word.⁹ FKGL estimates the reading grade level based on word, sentence, and syllable counts using the following formula: FKGL = 0.39 × (words/sentences) + 11.8 × (syllables/words) − 15.59. It has previously been reported that the average American reads at approximately an 8th-grade level.¹⁰ Data from the National Center for Education Statistics in 2023 demonstrated that 28% of US adults performed at level 1 or below in literacy, indicating limited ability to interpret longer or more complex written materials.^11,12

Attending ratings and FKGL scores were analyzed using descriptive statistics, and an analysis of variance (ANOVA) was conducted to assess significant differences in performance among chatbots. Descriptive statistics were used to analyze attending ratings and FKGL scores, and ANOVA was performed to assess differences in performance among the AI chatbots.

RESULTS

Across all platforms, mean accuracy scores were 1.7 for cervical cancer responses and 1.6 for ovarian cancer responses. Cervical cancer questions received more inaccurate responses than ovarian cancer questions (P = 0.02). Reddit produced the least accurate responses for both cervical and ovarian cancer question sets (P < 0.001) (Figure 1). For cervical cancer, approximately 70% of responses generated by ChatGPT and DeepSeek were rated “accurate and comprehensive” (score 1), whereas 66% of Reddit responses were rated as “outdated/inaccurate” or “completely inaccurate” (scores 3–4). For ovarian cancer, ChatGPT and Bootcamp had the highest proportion of “accurate and comprehensive” responses (80%), while 66% of Reddit responses were again rated as scores 3–4. Readability levels were high across platforms. The mean FKGL was 12.3 for cervical cancer responses and 12.5 for ovarian cancer responses, corresponding to at least a 12th-grade reading level.

Figure 1.

Ratings of answers to cancer FAQs by all five AI platforms (by percentage) by each AI chatbot on the 4-point scale for (a) cervical cancer and (b) ovarian cancer. When compared to the other AI platforms, Reddit answers scored statistically significantly worse based on post hoc Tukey’s HSD test (P < 0.001).

DISCUSSION

A key finding was that AI chatbot responses were written at approximately a 12th-grade reading level, substantially higher than recommended readability targets for patient-facing materials. This mirrors longstanding concerns that online and institutional patient education materials often exceed the recommended 6th- to 8th-grade level, potentially limiting comprehension and exacerbating disparities, particularly for patients with lower health literacy or limited English proficiency.¹

AI chatbot answers were also frequently limited by variable clinical applicability. Although average accuracy scores fell between “accurate and comprehensive” and “accurate but inadequate,” reviewers commonly observed that responses were overly broad, included extraneous information, and did not consistently reflect the nuance required for individualized counseling. These findings align with a growing body of evidence showing that AI chatbots can perform reasonably well on general medical questions but may degrade in quality when questions require guideline-specific recommendations, contextual tailoring, or clinically precise framing.¹³

Reddit AI produced the least accurate responses in both cancer domains. A plausible explanation is that Reddit-style outputs are often grounded in user-generated narratives and community posts, which may prioritize personal experience, emotional support, or anecdotal advice over evidence-based guidance. In oncology, where counseling often depends on staging, comorbidities, pathology, fertility goals, and evolving guidelines, such narratives can be incomplete or misleading when interpreted as medical advice. Importantly, the frequent presence of peer-support resources through links to social groups on the platform suggests Reddit may function better as an adjunct emotional support channel than as a medically reliable educational tool.¹⁴ This distinction matters clinically as patients may arrive with high confidence in information that feels relatable yet lacks the precision needed for safe counseling decisions.

Reviewers’ observations that AI chatbots performed better on “static” information (e.g., definitions, basic science, staging concepts) than on counseling-oriented questions are consistent with published work in gynecologic oncology and cervical cancer–focused evaluations, where performance is often stronger for general FAQs and weaker for guideline-dependent or context-sensitive recommendations.¹³ The drawbacks of AI chatbots are not necessarily completely incorrect answers, but rather partially correct answers that lack the nuance patients need. Examples of these drawbacks include omitting contraindications, overstating certainty, or failing to specify when a clinician must individualize recommendations. These issues overlap with known limitations of AI chatbots, including “hallucinations” (fabricated or incorrect statements), sensitivity to prompt phrasing and question context, and vulnerability to outdated or misleading training data.¹⁵

Rather than viewing AI chatbots as replacements for counseling, current evidence supports a narrower role of drafting, triage, and educational support under clinician oversight.¹⁶ A study of AI-drafted replies integrated into patient messaging workflows suggests potential efficiency gains but also emphasizes the need for human review and quality standards before clinical use.¹⁷ In gynecologic oncology specifically, the combination of high-stakes decisions and high patient anxiety makes reliability and readability especially consequential.¹⁸ AI chatbot content that is accurate but difficult to understand may still fail patients, particularly when medical terminology inherently raises the reading level and when responses do not distinguish general education from what requires individualized physician input. Ultimately, AI chatbots should only bolster, not replace, clinical counseling by physicians.

This study has several limitations. First, chatbot outputs can change over time due to model updates, so results are best interpreted as time-stamped snapshots. Second, although each prompt was asked in a new session to reduce the influence of prior answers, responses may still be affected by hard-to-control factors, such as the device, region, or safety settings. Therefore, the use of new sessions should not be interpreted as complete independence between prompts. Future work should repeat prompts across separate accounts, devices, browsers, IP addresses, and internet networks to better evaluate whether chatbot responses vary by user profile or technical environment. To assess response variability, future work should repeat each prompt across multiple independent sessions and on multiple dates, while documenting the chatbot model and version, as well as the test date, to track changes over time.

Third, the 4-point scale is practical but may introduce interrater variability, particularly when distinguishing “accurate but inadequate” from “accurate and comprehensive.” Although the responses were scored independently by practicing gynecologic oncologists, the small number of evaluators may increase susceptibility to individual reviewer bias. Future studies should include a larger and more diverse evaluator panel and assess inter-rater reliability.

Although the use of FAQs published by ACOG allowed for standardization, clinical relevance, and reproducibility across platforms, these FAQs likely represent a best-case testing environment. Patients often ask questions using incomplete, ambiguous, emotionally charged, or nonmedical language, and they may combine multiple concerns in a single prompt. These are important considerations, as a prior study demonstrated that chatbot response quality varied by prompt structure.¹⁹ Future studies should evaluate prompts derived from real-world patient behavior, including commonly searched questions, patient portal messages, and online forum posts. Expanding prompts could improve generalizability. Reliability could be strengthened by adding raters, using brief rater training, and using clearer scoring anchors or a more detailed rubric. Additionally, the United States-centric and English-only nature of this study may limit the generalizability of our findings to non-US populations and non–English-speaking individuals.

In conclusion, at present, AI chatbots should be considered adjuncts rather than substitutes for clinician counseling, with particular caution warranted for platforms that rely on user-generated narratives. Health systems and clinicians should anticipate chatbot-informed patient questions and proactively guide patients toward evidence-based resources and context-specific clinical evaluation. Future evaluation frameworks should move beyond accuracy alone and assess whether chatbot responses are understandable and actionable, reproducible across technical conditions, and responsive to how patients actually ask health-related questions.

Disclosure statement/Funding

The authors report no funding or conflicts of interest. The data that support the findings of this study are available from the corresponding author upon reasonable request.